MongoDB .local Toronto 2019: MongoDB Atlas Search Deep Dive
0 likes568 views
The document presents a deep dive into MongoDB Atlas Full-Text Search, highlighting its features such as native text indexing, powerful query operators, and flexible scoring mechanisms. It discusses the architecture, relevance in search results, and the process of creating and querying full-text indexes. Additionally, it outlines the capabilities of the search engine using MongoDB's aggregation pipeline and details future roadmap features and improvements.
![#MDBLocal
{
“name”: “Will Chow”,
“MongoDB”: {
“positions”: [“Senior Solutions Architect”],
“since” : new Date (“2018-10”),
},
“Pre-MongoDB”: {
“positions”: [“Systems Engineer”, “Architect”, “Developer”],
“companies”: [ “Cloudera”, “IBM”, “DataMirror”]
},
“likes”: [“running”, “renovating” ]
}
About Me](https://image.slidesharecdn.com/mongodb-191119185704/85/MongoDB-local-Toronto-2019-MongoDB-Atlas-Search-Deep-Dive-2-320.jpg)
![#MDBLocal
Why didn’t
[thing I want]
show up first?](https://image.slidesharecdn.com/mongodb-191119185704/85/MongoDB-local-Toronto-2019-MongoDB-Atlas-Search-Deep-Dive-5-320.jpg)
![#MDBLocal
Searching with Text Search - $text
db.movies.find
({$text: {$search: "chikago"}},
{score: {$meta: "textScore"},
_id : 0,
title: 1,
fullplot: 1
}).sort
({score:{$meta:"textScore"}}).pretty()
[] (empty set)](https://image.slidesharecdn.com/mongodb-191119185704/85/MongoDB-local-Toronto-2019-MongoDB-Atlas-Search-Deep-Dive-8-320.jpg)
![#MDBLocal
Searching with Full Text Search - $searchBeta
db.movies.aggregate([
{$searchBeta: { term: {
path: "title",
query: "chikago",
fuzzy: { "maxEdits": 1}},
highlight: { path: "fullplot"}}},
{$project: {
_id : 0,
title: 1,
score: { $meta: 'searchScore' },
highlights: { $meta: "searchHighlights"}}},
{$limit : 1}]).pretty()
{"title" : "Chicago",
"score" : 4.210028171539307,
highlights" : [{
path" : "fullplot",
"texts" : [
{"value" : "Murderess Velma Kelly finds herself on
death row in 1920s ",
"type" : "text"},
{"value" : "Chicago",
"type" : "hit"}],
"score" : 0.863726019859314}]}](https://image.slidesharecdn.com/mongodb-191119185704/85/MongoDB-local-Toronto-2019-MongoDB-Atlas-Search-Deep-Dive-16-320.jpg)
![#MDBLocal
db.movies.aggregate([
{ $match": {
"title": "The Godfather"
}},
{ $sort: { title: 1 } }
])
$searchBeta
db.movies.aggregate([
{ $searchBeta": {
"search": { query: "The Godfather",
path: "title",
score: { boost: {value: 3}}}}
])
Sort occurs after filter Sort occurs during filter
Query-time Scoring](https://image.slidesharecdn.com/mongodb-191119185704/85/MongoDB-local-Toronto-2019-MongoDB-Atlas-Search-Deep-Dive-17-320.jpg)
![#MDBLocal
Inverted Indices and Analyzers
“_id”: 3,
“title” : “Planes, Trains & Automobiles”
{
“planes” : [3, …],
“trains” : [3, …],
“automobiles” : [3, … ]
},
lucene.simple lucene.english
{
“plane” : [3, …],
“train” : [3, …],
“automobile” : [3, …],
}
lucene.keyword
{
“Planes, Trains &
Automobiles” : [3, …]
},](https://image.slidesharecdn.com/mongodb-191119185704/85/MongoDB-local-Toronto-2019-MongoDB-Atlas-Search-Deep-Dive-23-320.jpg)
![#MDBLocal
Query Lifecycle
Wire Protocol
(over internet to
MongoDB Atlas) aggregate([ {$searchBeta: {
search: {
path: “name”,
query: “star wars”
}
}}]
mongod
Wire protocol
(localhost)
Lucene booleanQuery:
(should
(term(“name”, “star”),
term(“name”, wars”))
search: {
path: “name”,
query: “star wars”
}
[ { _id: “123”,score:
1.23,highlights: […] },
{…}]
{ “name” :
{ “star” : [123,124],
“wars” : [123,125,…]
}
}
Lookup([{_id: “123”],
{…}])
mongot
db.col.aggregate([
{$searchBeta: {
search: {
path: “name”,
query: “star wars”
}
}}])
app
[ { _id: “123”, title:
“Star Wars”}, {…}]
MongoDB Atlas Host](https://image.slidesharecdn.com/mongodb-191119185704/85/MongoDB-local-Toronto-2019-MongoDB-Atlas-Search-Deep-Dive-28-320.jpg)
MongoDB .local Toronto 2019: MongoDB Atlas Search Deep Dive
- 1. #MDBlocal Atlas Search Deep Dive Will Chow Senior Solutions Architect, MongoDB TORONTO
- 2. #MDBLocal { “name”: “Will Chow”, “MongoDB”: { “positions”: [“Senior Solutions Architect”], “since” : new Date (“2018-10”), }, “Pre-MongoDB”: { “positions”: [“Systems Engineer”, “Architect”, “Developer”], “companies”: [ “Cloudera”, “IBM”, “DataMirror”] }, “likes”: [“running”, “renovating” ] } About Me
- 3. #MDBLocal AGENDA Why Full Text Search? Atlas Search Features Queries + How They Work Indexes and Analyzers Architecture
- 4. Why Full-Text Search? Search is a requirement for most applications
- 5. #MDBLocal Why didn’t [thing I want] show up first?
- 6. #MDBLocal Relevance is Complicated… Developer: “This result doesn’t match the most terms.” Marketing: “Why isn’t the promo on top?” Product: “If I type in an exact product name, just skip the search.” UX: “Users need categories instead.” CEO: “Change the top result right NOW!” Customer: “Je ne trouve pas ce que je cherche …”
- 8. #MDBLocal Searching with Text Search - $text db.movies.find ({$text: {$search: "chikago"}}, {score: {$meta: "textScore"}, _id : 0, title: 1, fullplot: 1 }).sort ({score:{$meta:"textScore"}}).pretty() [] (empty set)
- 10. #MDBLocal Atlas Search Features Native Powered by Performant Text Indexes Powerful Query Operators Configurable Language Analyzers Built-in Highlighting Flexible Scoring
- 11. #MDBLocal Atlas Search Features Native Powered by Score is the Measure of Relevance Flexible Scoring
- 12. #MDBLocal So, build the best search for your application Atlas Search Full Text Search Indexes $searchBeta
- 13. #MDBLocal How do I use it? Create a cluster on MongoDB Atlas 4.2 (M30+) Create a database and collection Create a full text index on that collection in Atlas Query via $searchBeta aggregation pipeline
- 14. #MDBLocal Full Text Search Index Creation Dynamic Mapping Static Mapping
- 16. #MDBLocal Searching with Full Text Search - $searchBeta db.movies.aggregate([ {$searchBeta: { term: { path: "title", query: "chikago", fuzzy: { "maxEdits": 1}}, highlight: { path: "fullplot"}}}, {$project: { _id : 0, title: 1, score: { $meta: 'searchScore' }, highlights: { $meta: "searchHighlights"}}}, {$limit : 1}]).pretty() {"title" : "Chicago", "score" : 4.210028171539307, highlights" : [{ path" : "fullplot", "texts" : [ {"value" : "Murderess Velma Kelly finds herself on death row in 1920s ", "type" : "text"}, {"value" : "Chicago", "type" : "hit"}], "score" : 0.863726019859314}]}
- 17. #MDBLocal db.movies.aggregate([ { $match": { "title": "The Godfather" }}, { $sort: { title: 1 } } ]) $searchBeta db.movies.aggregate([ { $searchBeta": { "search": { query: "The Godfather", path: "title", score: { boost: {value: 3}}}} ]) Sort occurs after filter Sort occurs during filter Query-time Scoring
- 18. DEMO A Movie Search Engine using Atlas Search Will Chow
- 20. #MDBLocal MongoDB + • Pre-existing functionality • Highlights, Fuzzy-matching, Query-time scoring and more • Analyzers • Language support • Western languages: English, French, etc. • Eastern Languages: CJK, Hindi, Thai, etc. • Inverted index structure = fast searches
- 21. #MDBLocal Inverted Index { _id: 1, S: “The quick brown fox jumped over the lazy dog” } { _id: 2, S: “Quick brown foxes leap over lazy dogs in summer” } TERM DOC The 1 Quick 2 brown 1, 2 fox 1 foxes 2 jumped 1 leap 2 TERM DOC the 1 quick 1, 2 brown 1, 2 fox 1, 2 in 2 jump 1, 2 dog 1, 2 STEMMING,SYNONYMS
- 22. #MDBLocal Real-world Inverted Index: A Concordance
- 23. #MDBLocal Inverted Indices and Analyzers “_id”: 3, “title” : “Planes, Trains & Automobiles” { “planes” : [3, …], “trains” : [3, …], “automobiles” : [3, … ] }, lucene.simple lucene.english { “plane” : [3, …], “train” : [3, …], “automobile” : [3, …], } lucene.keyword { “Planes, Trains & Automobiles” : [3, …] },
- 24. #MDBLocal Querying and Analyzers search: { query: “planes trains”, path: “title” } term(“plane”) OR term(“train”) term: { query: “planws”, path: { value: “title”, multi: ”simple”}, fuzzy: {maxEdits: 1 } } Fuzzy(“planws”, maxEdits: 1) term: { query: “Planes(.*)”, path: { value: “title”, multi: ”keyword”}, regex: true} } Regex(“Planes(.*)”) lucene.keyword lucene.simple lucene.english
- 25. #MDBLocal MongoDB Atlas FTS components mongod mongos mongot (NEW!) ● $searchBeta aggregation pipeline stage ● Talks mongodb wire protocol to mongot ● Shard aware implementation ● scatter-gather queries ● Based on Apache Lucene 8 ● Integrated into MongoDB Atlas ● Separate java process from mongod ● collocated with mongod
- 26. #MDBLocal FTS Indexing: Initial Sync MongoDB Atlas Query Index Definition Automation Agent mongotmongod Collection Scan Complete!
- 27. #MDBLocal FTS Indexing: Steady State Documents mongotmongod changestream MongoDB Atlas (per node) mongot watches the changestream continuously and updates the search index
- 28. #MDBLocal Query Lifecycle Wire Protocol (over internet to MongoDB Atlas) aggregate([ {$searchBeta: { search: { path: “name”, query: “star wars” } }}] mongod Wire protocol (localhost) Lucene booleanQuery: (should (term(“name”, “star”), term(“name”, wars”)) search: { path: “name”, query: “star wars” } [ { _id: “123”,score: 1.23,highlights: […] }, {…}] { “name” : { “star” : [123,124], “wars” : [123,125,…] } } Lookup([{_id: “123”], {…}]) mongot db.col.aggregate([ {$searchBeta: { search: { path: “name”, query: “star wars” } }}]) app [ { _id: “123”, title: “Star Wars”}, {…}] MongoDB Atlas Host
- 29. #MDBLocal Sharding • Merge • Sort by score mongos request result per shard per shard • Scatter/gather (each shard) mongot, mongod mongot, mongod Primary Secondaries Query
- 31. #MDBLocal • Expanded data type support • Nums, dates, geo • Synonyms • Improved operators/syntax • Architecture/performance improvements 2019 Roadmap
- 32. #MDBLocal Feedback Visit cloud.mongodb.com Please provide feedback using the link on the FTS page. We are listening!
- 34. Key Takeaways Ø Apache Lucene 8 Ø Uses MongoDB Query Language Ø Wide variety of query operators – fuzzy, wildcard Ø Flexible Scoring and Highlights Ø Configurable Indexes Ø Saves you time!!!
- 35. THANK YOU
- 36. #MDBlocal Every session you rate enters you into a drawing for a gift card! https://www.surveymonkey.com/r/C8TFLRC MongoDB Atlas Full-Text Search Deep Dive