Elasticsearch Arcihtecture & What's New in Version 5
Download as PPTX, PDF0 likes386 views
General architectural concepts of Elasticsearch and what's new in version 5? Examples are prepared with our company business therefore these are excluded from presentation.
![Multi Dimensional Points
• Based k-d tree (Solution of range search and nearest neighbor search)
• Support for byte[], IPv6, BigInteger, BigDecimal, 2D .. And higher.
• Allowing 8D (versus 1) points and 16bytes (versus 8bytes) limit per dimension.
• %36 faster at querying, %71 faster at indexing, %66 less disk and %85 less memory consumption.
• !!! New half_float and scaled_float](https://image.slidesharecdn.com/presentation-170214122153/85/Elasticsearch-Arcihtecture-What-s-New-in-Version-5-4-320.jpg)
Ad
More Related Content
What's hot (19)
Viewers also liked (6)
Ad
Similar to Elasticsearch Arcihtecture & What's New in Version 5 (20)
Ad
Recently uploaded (20)
Elasticsearch Arcihtecture & What's New in Version 5
- 1. ELASTICSEARCH ARCHITECTURE & WHAT’S NEW IN VERSION5 H. BURAK TUNGUT SOFTWARE ARCHITECT 03.02.2017
- 2. WHAT’S NEW IN ELASTICSEARCH 5 • New Data Structures • Indexing Performance • Ingest Node • Painless Scripting
- 3. NEW DATA STRUCTURES • Multi Dimensional Points • Text & Keyword
- 4. Multi Dimensional Points • Based k-d tree (Solution of range search and nearest neighbor search) • Support for byte[], IPv6, BigInteger, BigDecimal, 2D .. And higher. • Allowing 8D (versus 1) points and 16bytes (versus 8bytes) limit per dimension. • %36 faster at querying, %71 faster at indexing, %66 less disk and %85 less memory consumption. • !!! New half_float and scaled_float
- 5. k-d Tree
- 6. NEW DATA STRUCTURES • Multi Dimensional Points • Text & Keyword
- 7. Text & Keyword • Causing problem in case of using different use-cases on same field. • Splitted to text and keyword on same field. • Wanna do full-text search? Use foo path. • Wanna do exact match or aggregation? Use foo.keyword path.
- 8. Indexing Performance • Concurrent update performance improvements • Reduced locking when fsync and translog • Async fsync support • %25 - %80 indexing improvement depends on use-case
- 9. Ingest Node • %{IP:CLIENT} %{WORD:METHOD} %{URIPATHPARAM:REQUEST} %{NUMBER:BYTES} %{NUMBER:DURATION}
- 10. Painless Scripting • New scripting langauge Painless • Promoted as fast, safe, secure and enabled by default • 4 times fast as compared Groovy, Javascript and Python • With Reindex API and Ingest Node powerful way to manipulate documents
- 11. Parent Child vs Nested • Parent/child types are good at normalization and updating • Child docs can be searched without parent • Nested types good at searching performance Use nested types, if data can be duplicated, it is efficent way Use parent/child types, for real independently updateable documents
- 12. Architecture
- 13. Hierarchy •Cluster •Node • Index • Types • Document
- 14. Sharding • About scaling and failover • Primary Shards (one lucene instance) • Default 5 per index • Executes simultaneously • Replica Shards (duplication) • Default 1 per primary shard • A use case example with 1000 documents with more than one PS and just one PS
- 15. DevOps
- 16. Memory Optimization • Default heap size is 1GB, it must be changed! • More is better? We have 64GB RAM, should we give 64GB to Elasticsearch? • More RAM = More in-memory caching = better performance, it is accepted! • But we can get in trouble with Lucene! • Lucene segments are stored in individual files, they are immutable. Ready for caching everytime. • Most of case shows that Lucene deserves %50 of available total memory, like ES. • (Case of using aggs on analyzed string field)
- 17. Do not cross with 32GB • JVM has a feature that called compressed oops (ordinary object pointers) • We know that objects are allocated in heap and pointers linked to these area block’s • In 32 bit systems • The heap size is limited to 4GB (2^32 bytes) • We need more! Compressed oops • In 64 bit systems • The heap size is limited to 16 exabytes • It is enough. But the bandwith and CPU cache is not enough for that.
- 18. Build and Run ES in Docker • docker network create es-net • docker run --rm -p 9200:9200 -p 9300:9300 --name=es0 --network=es-net elasticsearch:latest -E cluster.name=burak -E network.host=172.18.0.2 -E node.name=node0 -E discovery.zen.ping.unicast.hosts="172.18.0.3:9300 • docker run --rm -p 9201:9200 -p 9301:9300 --name=es1 --network=es-net elasticsearch:latest -E cluster.name=burak -E network.host=172.18.0.3 -E node.name=node1
- 19. Thread Pool • Types • Fixed • Scaling • Size • Queue Size • Processor limits • Generic : scaling • Index : #availableprocessor thread, 200 queue size • Search : (3*#availableprocessor)/2 + 1 thread, 1000 queue size • Get : #availableprocessor thread, 1000 queue size • ...
- 20. Shard Allocation • Not detailed in this presentation • CLUSTER.ROUTING.ALLOCATION.NODE_CONCURRENT_INCOMING_RECOVERIES • CLUSTER.ROUTING.ALLOCATION.NODE_CONCURRENT_OUTGOING_RECOVERIES • CLUSTER.ROUTING.ALLOCATION.DISK.WATERMARK.LOW • CLUSTER.INFO.UPDATE.INTERVAL • ...
- 21. Monitoring • http://localhost:9200/_cluster/stats • http://localhost:9200/_nodes/stats • http://localhost:9200/product_season/_stats • Mervel | XPack
- 22. Query Examples
- 23. Full Text Search • Match • Match Phrase • Match Phrase Prefix • Match All • Common Terms (https://www.elastic.co/guide/en/elasticsearch/reference/current/query-dsl-common-terms-query.html) • Q.String (https://www.elastic.co/guide/en/elasticsearch/reference/current/query-dsl-query-string-query.html)
- 24. Term Level Queries • Term • Range • Prefix • Wildcard • Regexp • Fuzziness (Levenshtein distance)
- 25. Compound Queries • Constant score • Bool query (must-should-should with boosting) • Function score (sum, multiply, max | min_score)
- 26. Joining Queries • Nested Query • Child / Parent Queries
Editor's Notes
- #10: Pipeline - processor