Finding relevant results faster with Elasticsearch

1
Finding relevant hits
faster with Elasticsearch
Adrien Grand
Tech lead, Elasticsearch

2
This presentation and the accompanying oral presentation contain forward-looking statements, including statements
concerning plans for future offerings; the expected strength, performance or benefits of our offerings; and our future
operations and expected performance. These forward-looking statements are subject to the safe harbor provisions
under the Private Securities Litigation Reform Act of 1995. Our expectations and beliefs in light of currently
available information regarding these matters may not materialize. Actual outcomes and results may differ materially
from those contemplated by these forward-looking statements due to uncertainties, risks, and changes in
circumstances, including, but not limited to those related to: the impact of the COVID-19 pandemic on our business
and our customers and partners; our ability to continue to deliver and improve our offerings and successfully
develop new offerings, including security-related product offerings and SaaS offerings; customer acceptance and
purchase of our existing offerings and new offerings, including the expansion and adoption of our SaaS offerings;
our ability to realize value from investments in the business, including R&D investments; our ability to maintain and
expand our user and customer base; our international expansion strategy; our ability to successfully execute our
go-to-market strategy and expand in our existing markets and into new markets, and our ability to forecast customer
retention and expansion; and general market, political, economic and business conditions.
Additional risks and uncertainties that could cause actual outcomes and results to differ materially are included in
our filings with the Securities and Exchange Commission (the “SEC”), including our Annual Report on Form 10-K for
the most recent fiscal year, our quarterly report on Form 10-Q for the most recent fiscal quarter, and any
subsequent reports filed with the SEC. SEC filings are available on the Investor Relations section of Elastic’s
website at ir.elastic.co and the SEC’s website at www.sec.gov.
Any features or functions of services or products referenced in this presentation, or in any presentations, press
releases or public statements, which are not currently available or not currently available as a general availability
release, may not be delivered on time or at all. The development, release, and timing of any features or functionality
described for our products remains at our sole discretion. Customers who purchase our products and services
should make the purchase decisions based upon services and product features and functions that are currently
available.
All statements are made only as of the date of the presentation, and Elastic assumes no obligation to, and does not
currently intend to, update any forward-looking statements or statements relating to features or functions of services
or products, except as required by law.
Forward-Looking Statements

Outline
• What Elasticsearch does for you.
– Inverted indexes.
– Block-Max WAND.
– rank_feature.
– distance_feature.
• What you can do to help Elasticsearch.

4
What Elasticsearch
does for you

Inverted indexes
• "Inverted indexes power search engines."
• "Here is an inverted index."
• [invert, index, power, search, engine]
• [here, invert, index]
• engine: [0
• here: [1
• index: [0, 1
• invert: [0, 1
• power: [0
• search: [0

Block-max WAND
• New in 7.0
• "Elasticsearch OR Kibana"
• Maximum possible scores for each term are:
– Elasticsearch: 3
– Kibana: 2
• Score of 10th best document so far: 3.5
• The query can actually be rewritten to "Elasticsearch AND Kibana"

rank_feature
• New field type + query in 7.0
• Encode your features directly in the index
GET my-index/_search
{
"query": {
"bool": {
"must": {
"match": { "body": "Elasticsearch" }
},
"should": {
"rank_feature": { "field": "pagerank" }
}
}
}
}
POST my-index/_doc
{
"body": "Elasticsearch is a
search engine",
"pagerank": 12
}

distance_feature
• New query in 7.0
GET my-index/_search
{
"query": {
"bool": {
"must": {
"match": { "body": "Elasticsearch" }
},
"should": {
"distance_feature": {
"field": "@timestamp",
"origin": "now",
"pivot": "7d"
}
}
}
}
}
POST my-index/_doc
{
"body": "Elasticsearch is a
search engine",
"@timestamp": "2020-09-05"
}

9
What you can do to
help Elasticsearch

Best practices
• Keep the number of searched fields to a minimum
• Keep the number of features to a minimum
• CCR
• Model as many queries as possible as term queries
• index_prefixes / index_phrases
• track_total_hits

Hardware
• Ds you have fast disks?
• Do you have fast CPUs?
• Do you have enough nodes?
• Look into using index.preload to preload parts of the index in
memory

Rescoring
• Simple base query with 12 features
• More expensive logic at rescoring time

Optimize cache hit rates
• Use preference=${session_id}
– Subsequent user requests often reuse parts of the request
• Increasing the number of replicas doesn't always improve search
eﬃciency
– Hurts the hit rate of the FS cache

Use index sorting
• Enables early termination when sorting by field
– ≠ relevance
• But also enables clustering doc IDs, which makes queries more
efficient!
– Web search: sort by reverse host name then path, e.g.
"org.wikipedia.en/Elasticsearch"
– E-commerce: sort by category

Why doc ID ordering helps
6 x
5 x x
4
3 x
2 x
1 x
0 x
Doc ID Elasticsearch Kibana
6 jumps

Now doing a permutation of doc IDs
6 x
5 x x
4
3 x
2 x
1 x
0 x
6 jumps
0
4
1
2
6
3
5

Now doing a permutation of doc IDs
6
5 x
4 x
3 x x
2 x
1 x
0 x
4 jumps - 33% less

Finding relevant results faster with Elasticsearch

Recommended

More Related Content

What's hot (20)

Similar to Finding relevant results faster with Elasticsearch (20)

More from Elasticsearch (20)

Recently uploaded (20)

Finding relevant results faster with Elasticsearch