Is your Elastic Cluster Stable and Production Ready?
3 likes1,005 views
So you are deployed to production (or soon to be) with Elasticsearch running and powering important application features. Or maybe used for centralized logging for effective debugging. Was your Elastic cluster deployed correctly? Is it stable? Can it hold the throughput you expect it to? How did you do capacity planning? How to tell if the cluster is healthy and what to monitor? How to apply effective multi-tenancy? and what would be an ideal cluster topology and data ingestion architecture?
Ad
More Related Content
What's hot (19)
Similar to Is your Elastic Cluster Stable and Production Ready? (20)
Ad
More from DoiT International (19)
Ad
Recently uploaded (19)
Is your Elastic Cluster Stable and Production Ready?
- 1. Is your Elasticsearch Cluster Production Ready? Itamar Syn-Hershko http://code972.com | @synhershko http://BigDataBoutique.co.il
- 3. What does it take? • Cluster deployed using best practices • Thorough monitoring • Inspect. Fix. Repeat. • Good capacity planning • Memory management • Indexing and sharding strategy • Security
- 4. Cluster Topology Master-eligible nodes (3) Data nodes (sizing by data) Client nodes, aka coordinating nodes (scalable, sizing by traffic)
- 5. Deployments • Prefer immutable images & scripted deployments • For AWS see https://github.com/synhershko/elasticsearch- cloud-deploy/ • GCP coming soon
- 6. Backups • Very efficient • Very important • Several storages supported • To a shared file system • HDFS • Azure / GCP / AWS repositories via plugins
- 7. What to monitor (on the cluster, per host)? • CPU load • Memory utilization • Heap utilization • GC time • Disk utilization • Disk IOPs • Merges • Deleted docs • Requests per sec (indexing, search) • Load average < number of cores • Network in / out • Thread pool rejections • Number of nodes • Cache sizes • Cache evictions • Cluster state / health • Number of shards per type
- 8. X-Pack monitoring (aka Marvel)
- 9. Grafana dashboards • More fine-grained, cluster-wide view • Provided with metrics polling script (Python) https://github.com/synhershko/elasticsearch-grafana-monitoring
- 10. Monitoring Destination • To the same cluster • To a different cluster (Recommended) • External systems (e.g. graphite) – only if already in org • X-Pack subscribers can now send metrics to Elastic Cloud
- 11. Typical garbage collection sawtooth
- 12. CPU monitoring
- 13. Correlating metrics • Shards on the same node have issues? • During merges? • CPU and GC • HTTP traffic and indexing or search operations
- 15. Boosting slow operations • Search or Indexing heavy? • Measure operations also from applications side! • Slow searches • Queries need optimization • Scoring (not using filters) • Numeric ranges pre-5 • Scripts • Slow indexing • Sharding strategy • Use bulk indexing (optimize for 10-15MB of data, regardless of number of documents / operations) • Slow analyzers affects both! (e.g. n-grams)
- 16. Don’t use NGrams! • Being used for “contains” search • You ain’t gonna need it, use WordDelimiter Token Filter instead • Useful for fuzzy search / auto-correction • Best used via Elasticsearch’s Suggesters • Useful for languages without spaces, or with compound words • min_gram , max_gram
- 17. Caches • Query cache • Request cache • Measure evictions rate & cache usage
- 18. Memory Allocation • ES_HEAP_SIZE • DocValues used? • Fielddata usage • Query cache (for queries in filter context) • Request cache (for aggregations and count queries) • Never over 32GB! • Default cache sizes not always fit usage • Set appropriate static configs in elasticsearch.yml • At least 50% of memory to file-system cache • Usually more
- 19. Server Sizing • Master nodes • 1-2 cores, 2-4 GB memory, 50% ES_HEAP_SIZE • Data nodes • > 4 cores, measure and preserve disk/mem ratio (can start with 1/24) • ES_HEAP_SIZE as per previous slide • Client nodes • CPU and network heavy, 4GB memory should be enough for most use cases
- 20. Index Management Patterns • A Monolith Index • Search façade on top of your data • Record linkage • Anomaly detection • Rolling indexes (time based events) • Centralized logging • Auditing • IoT logs-2016.11.20 logs-2016.11.21 logs-2016.11.22 logs-2016.11.23logs-2016.11.19
- 21. Optimal shard size • Few millions in document size, for search performance • A bit more if only doing aggregations • 5-8GB on disk max, for startup times and network reallocation • doc_values are enabled by default, turn off for non-aggs fields to save space
- 22. Sharding • Index Shards • Resharding / auto-sharding not supported • Index-level sharding • Avoid using types (deprecated > 6.x) • Multi-tenancy • Rollover API (> 5.x) • Cluster level • Cluster per project • Cross-cluster search capability
- 23. Multitenancy • Silos – Every tenant get their own index • Index sizes vary • Potentially wasting resources • Pool – All tenants are in one big index • Sharding isn’t dynamic • Effects on tf/idf, aggregations, throughput • Hybrid – Big tenants in their own index, pool(s) for small ones
- 24. Use Explicit Mapping (aka Avoid Schemaless) • In one of two ways: • Disable dynamic mapping in settings (index.mapper.dynamic: false). Will refuse indexing. • Create catch-all dynamic template with enabled:false mapping • Why? • Avoids hundreds of fields by mistake • Saves effort on indexing and disk space • Defaults are bad anyhow, don’t rely on them • Prefer using index templates (especially for rolling indices)
- 25. Re-balancing is your enemy • Lock down shard rebalancing • cluster.routing.rebalance.enable • none • cluster.routing.allocation.enable • primaries • new_primaries • none
- 26. More safe configs • action.disable_delete_all_indices: true • action.auto_create_index: false
- 27. Deep paging (don’t!) • Don’t from-size • search_after (> 5.x) • Scroll and sliced-scroll (> 5.x) • Not for normal operation
- 28. Deletions • Deletions have an overhead • Slow searches • Segmentation • More work on segment merging • Non-exact tf/idf • Every document update is a deletion • No need to avoid it completely, just design accordingly
- 29. Geographic Distribution • Never with the same cluster! • Cross-cluster search (formerly Tribe Node) • For geographic sharding • Different indexes in different regions • xDCR for HA / DR • Can be solved by infra – replicating queues (Kafka), DBs • Solution coming in X-Pack
- 30. Your ingestion architecture? • Favor external ingestion, relieve Elastic from that responsibility • Upgrade Logstash to 5.x • Consider using FileBeat instead of logstash for log-tailing • Prefer logstash machines over ingest nodes • Use queues (Kafka, Redis) to protect against surges
- 31. Security
- 32. Protecting your cluster • Don’t bind to a public IP • Use only private IP/DNSs, preferably in subnets (e.g. AWS VPC) • network.host in elasticsearch.yml • Proxy all client requests to ES • Disable HTTP where not needed • + Don’t use default ports • Secure publicly available client nodes • Access via VPN only • At the very least SSL + authentication if VPN not an option • Disable dynamic scripting (pre-5.x)
- 33. Securing Indexes and Documents • Heavy Kibana user? • Authentication and authorization • Index, Document and Field level security • Requires X-Pack Security • Application level authentication and authorization • Application filtering of content (fields, documents) • Index level (e.g. index per tenant) • Document level (using permissions) • Inter-node comms, encryption at rest (X-Pack only)
- 34. Upcoming in ES land • Elasticsearch 6 • Machine Learning • Anomaly detection on time series data • Enterprise Cloud • Elastic Cloud deployed on-premise • Any plugin authors in the crowd?
- 35. Elasticsearch Training Elasticsearch for Developers & Maintaining Elasticsearch in Production • September (10,11,17/9) • November (12,13,16/11) http://bdbq.co.il/courses Consultancy and Development services http://bdbq.co.il/services/elasticsearch
- 36. Questions? @synhershko on social (Twitter, github, …) Blog at http://code972.com Training and consultancy at http://BigDataBoutique.co.il