SlideShare a Scribd company logo

Tips and Tricks for Avoiding Common Query Pitfalls Christian Kurze

MongoDB
MongoDB

The document discusses strategies for optimizing MongoDB queries, focusing on issues such as blocking stages, the use of the $or operator, and the importance of indexing for performance. It describes the challenges faced by stakeholders at Acme Games, Inc. regarding a new strategy game, emphasizing how query tuning can lead to significant efficiency improvements. Specific examples of query performance improvements through indexing are provided, along with insights into the differences between $and and $or query performance.

Software
1 of 133
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
Tips and Tricks for Avoiding Common Query Pitfalls Christian Kurze
CHRISTIAN KURZE Senior Solutions Architect MongoDB
Roadmap • Motivation • Three items to be aware of: • Blocking stages • Using the $or operator • Case insensitivity
The Power of Query Optimization • Query tuning results in: • Improved performance • Reduced resource utilization • This may lead to: • Improved stability and predictability • A smaller hardware footprint Not uncommon to observe efficiency improvements greater than 99%
The Challenge @ Acme Games, Inc. Charlie • DBA at Acme Game, Inc. • MongoDB Champion Stakeholders • Developers • Leadership • RDBMS History
Acme Games Introduces… The Peanuts Brand new strategy game Launching November 15th
Stakeholder Concerns • Game nearly complete • Developers have learned a lot from Charlie
Stakeholder Concerns • Game nearly complete • Developers have learned a lot from Charlie Indexes support the efficient execution of queries in MongoDB
Stakeholder Concerns • Game nearly complete • Developers have learned a lot from Charlie Indexes support the efficient execution of queries in MongoDB
Stakeholder Concerns • Game nearly complete • Developers have learned a lot from Charlie Indexes support the efficient execution of queries in MongoDB Ace Bob Sue … …
Stakeholder Concerns • Game nearly complete • Developers have learned a lot from Charlie • App being stress tested
Stakeholder Concerns • Game nearly complete • Developers have learned a lot from Charlie • App being stress tested • Concerned over current performance
Stakeholder Concern #1 Developers created index db.games.createIndex({ gamerTag: 1 })
Stakeholder Concern #1 Developers created index db.games.createIndex({ gamerTag: 1 }) This query takes several seconds to execute: db.games.find( { gamerTag: "Ace" } ).sort({score:-1})
Stakeholder Concern #1 Developers created index db.games.createIndex({ gamerTag: 1 }) This query takes several seconds to execute: db.games.find( { gamerTag: "Ace" } ).sort({score:-1}) Adding the index on score does not help! db.games.createIndex({ score: -1 })
Stakeholder Concern #1 Developers created index db.games.createIndex({ gamerTag: 1 }) This query takes several seconds to execute: db.games.find( { gamerTag: "Ace" } ).sort({score:-1}) Adding the index on score does not help! db.games.createIndex({ score: -1 }) “Clearly MongoDB is not web-scale!”
Blocking Operations
Blocking Operations ● Formally: ■ “An operation which must process all input before it can begin to produce any output.” ● Opposite of the often desirable “fully pipelined” plan which can stream results back as soon as they are found. ● Commonly observed when a sort is added to a query
Sorting with blocking
Sorting with blocking
Sorting with blocking
Sorting with blocking
Sorting with blocking
Sorting with blocking
Sorting with blocking
Sorting with blocking
Sorting with blocking
Sorting with blocking
Sorting with blocking
Sorting with blocking
Sorting with blocking
Sorting with blocking
Sorting with blocking
Sorting with blocking
Sorting with blocking
Sorting with blocking
Sorting with blocking
Sorting with blocking
Sorting with blocking
Sorting with blocking
Sorting with blocking
Sorting with blocking
Sorting without blocking
Sorting without blocking
Sorting without blocking
Sorting without blocking
Blocking Stages • $sort • In aggregation and find • $group • $bucket • $count • $facet Are there any other blocking operations?
Working with blocking stages For sorting: • Add a supporting index • Worth the overhead in almost all circumstances For other stages: • Do you need the blocking stage? • Offload to secondary member
Stakeholder Concern #1 Performance of db.games.find( { gamerTag: "Ace" } ).sort({score:-1}) “Clearly MongoDB is not web-scale!”
Stakeholder Concern #1 Performance of db.games.find( { gamerTag: "Ace" } ).sort({score:-1}) db.games.createIndex({ gamerTag: 1, score:-1 })
Stakeholder Concern #1 Performance of db.games.find( { gamerTag: "Ace" } ).sort({score:-1}) db.games.createIndex({ gamerTag: 1, score:-1 }) "That’ll work great!”
Stakeholder Concern #2 The $and version of a query returns quickly: db.games.find({ $and : [ { gamerTag: "Ace" }, { score: {$gt: 9000} } ] })
Stakeholder Concern #2 The $and version of a query returns quickly: db.games.find({ $and : [ { gamerTag: "Ace" }, { score: {$gt: 9000} } ] }) But the $or version is slow: db.games.find({ $or : [ { gamerTag: "Ace" }, { score: {$gt: 9000} } ] })
Stakeholder Concern #2 The $and version of a query returns quickly: db.games.find({ $and : [ { gamerTag: "Ace" }, { score: {$gt: 9000} } ] }) But the $or version is slow: db.games.find({ $or : [ { gamerTag: "Ace" }, { score: {$gt: 9000} } ] }) We just created an index with both those fields… Can’t it be used?
$or
$and example Query on games: db.games.find({ $and : [ { gamerTag: "Ace" }, { score: {$gt: 9000} } ] }) Matching games: { gamerTag: "Ace", score: 9500 } Non-matching games: { gamerTag: "Ace", score: 500 }, { gamerTag: "Bob", score: 9500 }, { gamerTag: "Bob", score: 500 }
Groups of documents { gamerTag: "Ace", score: 9500 } { gamerTag: "Ace", score: 500 } { gamerTag: "Bob", score: 9500 } { gamerTag: "Bob", score: 500 }
Groups of documents { gamerTag: "Ace", score: 9500 } { gamerTag: "Ace", score: 500 } { gamerTag: "Bob", score: 9500 } { gamerTag: "Bob", score: 500 } gamerTag: "Ace"
Groups of documents { gamerTag: "Ace", score: 9500 } { gamerTag: "Ace", score: 500 } { gamerTag: "Bob", score: 9500 } { gamerTag: "Bob", score: 500 } gamerTag: "Ace" score: {$gt: 9000}
$and Venn Diagram (logical) { gamerTag: "Ace", score: 9500 } { gamerTag: "Ace", score: 500 } { gamerTag: "Bob", score: 9500 } { gamerTag: "Bob", score: 500 } db.games.find({ $and : [ { gamerTag: "Ace" }, { score: {$gt: 9000} } ] })gamerTag: "Ace" score: {$gt: 9000}
$and Venn Diagram (logical) { gamerTag: "Ace", score: 9500 } { gamerTag: "Ace", score: 500 } { gamerTag: "Bob", score: 9500 } { gamerTag: "Bob", score: 500 } db.games.find({ $and : [ { gamerTag: "Ace" }, { score: {$gt: 9000} } ] })gamerTag: "Ace" score: {$gt: 9000}
$and Venn Diagram (logical) db.games.find({ $and : [ { gamerTag: "Ace" }, { score: {$gt: 9000} } ] })gamerTag: "Ace" score: {$gt: 9000}
$and Venn Diagram (logical) db.games.find({ $and : [ { gamerTag: "Ace" }, { score: {$gt: 9000} } ] })gamerTag: "Ace" score: {$gt: 9000}
$and Venn Diagram (logical) db.games.find({ $and : [ { gamerTag: "Ace" }, { score: {$gt: 9000} } ] })gamerTag: "Ace" score: {$gt: 9000}
$and Venn Diagram (logical) db.games.find({ $and : [ { gamerTag: "Ace" }, { score: {$gt: 9000} } ] })gamerTag: "Ace" score: {$gt: 9000}
$and Venn Diagram (logical) db.games.find({ $and : [ { gamerTag: "Ace" }, { score: {$gt: 9000} } ] })gamerTag: "Ace" score: {$gt: 9000}
$and Venn Diagram (logical) db.games.find({ $and : [ { gamerTag: "Ace" }, { score: {$gt: 9000} } ] }) Ace Bob 500 950 0 500 950 0 {gamerTag:1 , score:-1}
$and Venn Diagram (logical) db.games.find({ $and : [ { gamerTag: "Ace" }, { score: {$gt: 9000} } ] }) Ace Bob 500 950 0 500 950 0 "indexBounds" : { "gamerTag" : [ "["Ace", "Ace"]" ], "score" : [ "[inf.0, 9000.0)" ] } {gamerTag:1 , score:-1}
$and Venn Diagram (logical) db.games.find({ $and : [ { gamerTag: "Ace" }, { score: {$gt: 9000} } ] }) Ace Bob 500 950 0 500 950 0 "indexBounds" : { "gamerTag" : [ "["Ace", "Ace"]" ], "score" : [ "[inf.0, 9000.0)" ] } {gamerTag:1 , score:-1}
$and Venn Diagram (logical) db.games.find({ $and : [ { gamerTag: "Ace" }, { score: {$gt: 9000} } ] }) Ace Bob 500 950 0 500 950 0 "indexBounds" : { "gamerTag" : [ "["Ace", "Ace"]" ], "score" : [ "[inf.0, 9000.0)" ] } {gamerTag:1 , score:-1}
$and Venn Diagram (logical) db.games.find({ $and : [ { gamerTag: "Ace" }, { score: {$gt: 9000} } ] }) Ace Bob 500 950 0 500 950 0 "indexBounds" : { "gamerTag" : [ "["Ace", "Ace"]" ], "score" : [ "[inf.0, 9000.0)" ] } {gamerTag:1 , score:-1}
$or example Query on games: db.games.find({ $or : [ { gamerTag: "Ace" }, { score: {$gt: 9000} } ] }) Matching games: { gamerTag: "Ace", score: 9500 }, { gamerTag: "Ace", score: 500 }, { gamerTag: "Bob", score: 9500 } Non-matching games: { gamerTag: "Bob", score: 500 }
$or Venn Diagram (logical) { gamerTag: "Ace", score: 9500 } { gamerTag: "Ace", score: 500 } { gamerTag: "Bob", score: 9500 } { gamerTag: "Bob", score: 500 } db.games.find({ $or : [ { gamerTag: "Ace" }, { score: {$gt: 9000} } ] })gamerTag: "Ace" score: {$gt: 9000}
$or Venn Diagram (logical) { gamerTag: "Ace", score: 9500 } { gamerTag: "Ace", score: 500 } { gamerTag: "Bob", score: 9500 } { gamerTag: "Bob", score: 500 } db.games.find({ $or : [ { gamerTag: "Ace" }, { score: {$gt: 9000} } ] })gamerTag: "Ace" score: {$gt: 9000}
$or Venn Diagram (logical) { gamerTag: "Ace", score: 9500 } { gamerTag: "Ace", score: 500 } { gamerTag: "Bob", score: 9500 } { gamerTag: "Bob", score: 500 } db.games.find({ $or : [ { gamerTag: "Ace" }, { score: {$gt: 9000} } ] })gamerTag: "Ace" score: {$gt: 9000}
$or Venn Diagram (logical) { gamerTag: "Ace", score: 9500 } { gamerTag: "Ace", score: 500 } { gamerTag: "Bob", score: 9500 } { gamerTag: "Bob", score: 500 } db.games.find({ $or : [ { gamerTag: "Ace" }, { score: {$gt: 9000} } ] })gamerTag: "Ace" score: {$gt: 9000}
$or Venn Diagram (logical) { gamerTag: "Ace", score: 9500 } { gamerTag: "Ace", score: 500 } { gamerTag: "Bob", score: 9500 } { gamerTag: "Bob", score: 500 } db.games.find({ $or : [ { gamerTag: "Ace" }, { score: {$gt: 9000} } ] })gamerTag: "Ace" score: {$gt: 9000}
$or Venn Diagram (logical) db.games.find({ $or : [ { gamerTag: "Ace" }, { score: {$gt: 9000} } ] })gamerTag: "Ace" score: {$gt: 9000}
$or Venn Diagram (logical) db.games.find({ $or : [ { gamerTag: "Ace" }, { score: {$gt: 9000} } ] })gamerTag: "Ace" score: {$gt: 9000}
$or Venn Diagram (logical) db.games.find({ $or : [ { gamerTag: "Ace" }, { score: {$gt: 9000} } ] })gamerTag: "Ace" score: {$gt: 9000}
$or Venn Diagram (logical) db.games.find({ $or : [ { gamerTag: "Ace" }, { score: {$gt: 9000} } ] })gamerTag: "Ace" score: {$gt: 9000}
$or Venn Diagram (logical) db.games.find({ $or : [ { gamerTag: "Ace" }, { score: {$gt: 9000} } ] })gamerTag: "Ace" score: {$gt: 9000}
$or (single) Index visualization Ace Bob {gamerTag:1 , score:-1} 500 950 0 500 950 0 db.games.find({ $or : [ { gamerTag: "Ace" }, { score: {$gt: 9000} } ] })
$or (single) Index visualization Ace Bob {gamerTag:1 , score:-1} 500 950 0 500 950 0 db.games.find({ $or : [ { gamerTag: "Ace" }, { score: {$gt: 9000} } ] }) Expected Index Bounds: "indexBounds" : { "gamerTag" : [ "["Ace", "Ace"]" ], "score" : [ "[inf.0, 9000)" ] }
$or (single) Index visualization Ace Bob {gamerTag:1 , score:-1} 500 950 0 500 950 0 db.games.find({ $or : [ { gamerTag: "Ace" }, { score: {$gt: 9000} } ] }) Expected Index Bounds: "indexBounds" : { "gamerTag" : [ "["Ace", "Ace"]" ], "score" : [ "[inf.0, 9000)" ] }
$or (single) Index visualization Ace Bob {gamerTag:1 , score:-1} 500 950 0 500 950 0 db.games.find({ $or : [ { gamerTag: "Ace" }, { score: {$gt: 9000} } ] }) Expected Index Bounds: "indexBounds" : { "gamerTag" : [ "["Ace", "Ace"]" ], "score" : [ "[inf.0, 9000)" ] }
$or (single) Index visualization Ace Bob {gamerTag:1 , score:-1} 500 950 0 500 950 0 db.games.find({ $or : [ { gamerTag: "Ace" }, { score: {$gt: 9000} } ] }) Expected Index Bounds: "indexBounds" : { "gamerTag" : [ "["Ace", "Ace"]" ], "score" : [ "[inf.0, 9000)" ] }
$or (single) Index visualization Ace Bob {gamerTag:1 , score:-1} 500 950 0 500 950 0 db.games.find({ $or : [ { gamerTag: "Ace" }, { score: {$gt: 9000} } ] }) Expected Index Bounds: "indexBounds" : { "gamerTag" : [ "["Ace", "Ace"]" ], "score" : [ "[inf.0, 9000)" ] }
$or (single) Index visualization Ace Bob {gamerTag:1 , score:-1} 500 950 0 500 950 0 db.games.find({ $or : [ { gamerTag: "Ace" }, { score: {$gt: 9000} } ] }) Expected Index Bounds: "indexBounds" : { "gamerTag" : [ "["Ace", "Ace"]" ], "score" : [ "[inf.0, 9000)" ] }
$or (single) Index visualization Ace Bob {gamerTag:1 , score:-1} 500 950 0 500 950 0 db.games.find({ $or : [ { gamerTag: "Ace" }, { score: {$gt: 9000} } ] }) Expected Index Bounds: "indexBounds" : { "gamerTag" : [ "["Ace", "Ace"]" ], "score" : [ "[inf.0, 9000)" ] } Actual (Hinted) Index Bounds: "indexBounds" : { "gamerTag" : [ "[MinKey, MaxKey]" ], "score" : [ "[MaxKey, MinKey]" ] }
$or (single) Index visualization Ace Bob {gamerTag:1 , score:-1} 500 950 0 500 950 0 db.games.find({ $or : [ { gamerTag: "Ace" }, { score: {$gt: 9000} } ] }) Expected Index Bounds: "indexBounds" : { "gamerTag" : [ "["Ace", "Ace"]" ], "score" : [ "[inf.0, 9000)" ] } Actual (Hinted) Index Bounds: "indexBounds" : { "gamerTag" : [ "[MinKey, MaxKey]" ], "score" : [ "[MaxKey, MinKey]" ] } So is there anything we can do to improve the performance of this query?
Recommendations Use multiple indexes! db.data.createIndex({gamerTag: 1}) db.data.createIndex({score: 1})
$or (multiple) Index visualization db.games.find({ $or : [ { gamerTag: "Ace" }, { score: {$gt: 9000} } ] })
$or (multiple) Index visualization Ace db.games.find({ $or : [ { gamerTag: "Ace" }, { score: {$gt: 9000} } ] })
$or (multiple) Index visualization Ace Bob db.games.find({ $or : [ { gamerTag: "Ace" }, { score: {$gt: 9000} } ] })
$or (multiple) Index visualization Ace Bob {gamerTag:1} db.games.find({ $or : [ { gamerTag: "Ace" }, { score: {$gt: 9000} } ] })
$or (multiple) Index visualization Ace Bob {gamerTag:1} 500 db.games.find({ $or : [ { gamerTag: "Ace" }, { score: {$gt: 9000} } ] })
$or (multiple) Index visualization Ace Bob {gamerTag:1} 500 950 0 db.games.find({ $or : [ { gamerTag: "Ace" }, { score: {$gt: 9000} } ] })
$or (multiple) Index visualization Ace Bob {gamerTag:1} 500 950 0 {score:1} db.games.find({ $or : [ { gamerTag: "Ace" }, { score: {$gt: 9000} } ] })
$or (multiple) Index visualization Ace Bob {gamerTag:1} 500 950 0 {score:1} "indexBounds" : { "gamerTag" : [ "["Ace", "Ace"]" ] } "indexBounds" : { "score" : [ "(9000.0, inf.0]" ] } db.games.find({ $or : [ { gamerTag: "Ace" }, { score: {$gt: 9000} } ] })
$or (multiple) Index visualization Ace Bob {gamerTag:1} 500 950 0 {score:1} db.games.find({ $or : [ { gamerTag: "Ace" }, { score: {$gt: 9000} } ] }) "indexBounds" : { "gamerTag" : [ "["Ace", "Ace"]" ] } "indexBounds" : { "score" : [ "(9000.0, inf.0]" ] }
$or (multiple) Index visualization Ace Bob {gamerTag:1} 500 950 0 {score:1} db.games.find({ $or : [ { gamerTag: "Ace" }, { score: {$gt: 9000} } ] }) "indexBounds" : { "gamerTag" : [ "["Ace", "Ace"]" ] } "indexBounds" : { "score" : [ "(9000.0, inf.0]" ] }
$or (multiple) Index visualization Ace Bob {gamerTag:1} 500 950 0 {score:1} db.games.find({ $or : [ { gamerTag: "Ace" }, { score: {$gt: 9000} } ] }) "indexBounds" : { "gamerTag" : [ "["Ace", "Ace"]" ] } "indexBounds" : { "score" : [ "(9000.0, inf.0]" ] }
$or (multiple) Index visualization Ace Bob {gamerTag:1} 500 950 0 {score:1} db.games.find({ $or : [ { gamerTag: "Ace" }, { score: {$gt: 9000} } ] }) "indexBounds" : { "gamerTag" : [ "["Ace", "Ace"]" ] } "indexBounds" : { "score" : [ "(9000.0, inf.0]" ] }
$or (multiple) Index visualization Ace Bob {gamerTag:1} 500 950 0 {score:1} db.games.find({ $or : [ { gamerTag: "Ace" }, { score: {$gt: 9000} } ] }) "indexBounds" : { "gamerTag" : [ "["Ace", "Ace"]" ] } "indexBounds" : { "score" : [ "(9000.0, inf.0]" ] }
Recommendations Use multiple indexes! db.data.createIndex({gamerTag: 1}) db.data.createIndex({score: 1})
Recommendations Use multiple indexes! db.data.createIndex({gamerTag: 1}) db.data.createIndex({score: 1}) We already have the {gamerTag:1, score:-1} index, do we need both of these new ones?
Recommendations Use multiple indexes! db.data.createIndex({gamerTag: 1}) db.data.createIndex({gamerTag: 1, score: -1}} db.data.createIndex({score: 1}) We already have the {gamerTag:1, score:-1} index, do we need both of these new ones?
Recommendations Use multiple indexes! db.data.createIndex({gamerTag: 1}) db.data.createIndex({gamerTag: 1, score: -1}} db.data.createIndex({score: 1}) Rule of thumb for compound indexes: equality, sort, range
Recommendations Use multiple indexes! db.data.createIndex({gamerTag: 1}) db.data.createIndex({gamerTag: 1, score: -1}} db.data.createIndex({score: 1}) Rule of thumb for compound indexes: equality, sort, range Works with sorting Generate a SORT_MERGE plan
Stakeholder Concern #2 db.games.find({ $or : [ { gamerTag: "Ace" }, { score: {$gt: 9000} } ] }) Having the right index is critical
Stakeholder Concern #2 db.games.find({ $or : [ { gamerTag: "Ace" }, { score: {$gt: 9000} } ] }) Having the right index is critical
Stakeholder Concern #2 db.games.find({ $or : [ { gamerTag: "Ace" }, { score: {$gt: 9000} } ] }) Having the right index is critical "Super!!”
Stakeholder Concern #3 “Wait wait wait, we can’t even FIND the gamers!” A basic search on gamerTag takes several seconds already: db.games.find({gamerTag: /Ace/i}) “This query is SLOWER with the index than it is without it!”
Case Insensitive
Case Sensitive db.games.find({ gamerTag: /^Ace$/ }) //equivalent to db.games.find({ gamerTag: “Ace” }) Matching games: { gamerTag: "Ace", score: 9500 }, Non-matching games: { gamerTag: "ACE", score: 500 }, { gamerTag: "aCe", score: 9500 }, { gamerTag: "ace", score: 0 }, { gamerTag: "Bob", score: 500 }, { gamerTag: "acxyz", score: 9500 }, { gamerTag: "Ace mdb", score: 9500 }
Case Sensitive ace aCe acxy z Ace Ace mdb ACE Bob "indexBounds" : { "gamerTag" : [ "["Ace", "Acf")", "[/^Ace$/, /^Ace$/]" ] } Matching games: { gamerTag: "Ace", score: 9500 }, Non-matching games: { gamerTag: "ACE", score: 500 }, { gamerTag: "aCe", score: 9500 }, { gamerTag: "ace", score: 0 }, { gamerTag: "Bob", score: 500 }, { gamerTag: "acxyz", score: 9500 }, { gamerTag: "Ace mdb", score: 9500 }
Case insensitive db.games.find({ gamerTag: /^Ace$/i }) //equivalent to db.games.find({ gamerTag: { $regex: “^Ace$”, $options: “i” } }) //equivalent to db.games.find({ gamerTag: “Ace”}) .collation({locale:’en’, strength:2}) Matching games: { gamerTag: "Ace", score: 9500 }, { gamerTag: "ACE", score: 500 }, { gamerTag: "aCe", score: 9500 }, { gamerTag: "ace", score: 0 } Non-matching games: { gamerTag: "Bob", score: 500 }, { gamerTag: "acxyz", score: 9500 }, { gamerTag: "Ace mdb", score: 9500 }
Case insensitive db.games.find({ gamerTag: /^Ace$/i }) //equivalent to db.games.find({ gamerTag: { $regex: “^Ace$”, $options: “i” } }) //equivalent to db.games.find({ gamerTag: “Ace”}) .collation({locale:’en’, strength:2}) Matching games: { gamerTag: "Ace", score: 9500 }, { gamerTag: "ACE", score: 500 }, { gamerTag: "aCe", score: 9500 }, { gamerTag: "ace", score: 0 } Non-matching games: { gamerTag: "Bob", score: 500 }, { gamerTag: "acxyz", score: 9500 }, { gamerTag: "Ace mdb", score: 9500 } Would a $text search be the same as well?
Case INsensitive ace aCe acxy z Ace Ace mdb ACE Bob "indexBounds" : { "gamerTag" : [ "["", {})", "[/^Ace$/i, /^Ace$/i]" ] } Matching games: { gamerTag: "Ace", score: 9500 }, { gamerTag: "ACE", score: 500 }, { gamerTag: "aCe", score: 9500 }, { gamerTag: "ace", score: 0 } Non-matching games: { gamerTag: "Bob", score: 500 }, { gamerTag: "acxyz", score: 9500 }, { gamerTag: "Ace mdb", score: 9500 }
Recommendations Case insensitive index! Collations available since 3.4
Recommendations Case insensitive index! Collations available since 3.4 db.games.createIndex( { gamerTag: 1}, { collation: { locale: 'en', strength: 2 } } )
Recommendations Case insensitive index! Collations available since 3.4 db.games.createIndex( { gamerTag: 1}, { collation: { locale: 'en', strength: 2 } } ) > db.games.find( { gamerTag: "Ace"}).collation( { locale: 'en', strength: 2 } )
Recommendations Case insensitive index! Collations available since 3.4 db.games.createIndex( { gamerTag: 1}, { collation: { locale: 'en', strength: 2 } } ) > db.games.find( { gamerTag: "Ace"}).collation( { locale: 'en', strength: 2 } ) { "_id" : ObjectId("5b29dbee6c7d4f531bf73b5d"), "gamerTag" : "Ace", "score" : 9500 } { "_id" : ObjectId("5b29dbee6c7d4f531bf73b5e"), "gamerTag" : "ACE", "score" : 500 } { "_id" : ObjectId("5b29dbee6c7d4f531bf73b5f"), "gamerTag" : "aCe", "score" : 9500 } { "_id" : ObjectId("5b29dbee6c7d4f531bf73b60"), "gamerTag" : "ace", "score" : 0 }
Recommendations Case insensitive index! Collations available since 3.4 db.games.createIndex( { gamerTag: 1}, { collation: { locale: 'en', strength: 2 } } ) Store a transformed (eg toLower()) copy of the string
Stakeholder Concern #3 db.games.find({gamerTag: “Ace”}) .collation({locale:'en', strength:2})
Stakeholder Concern #3 db.games.find({gamerTag: “Ace”}) .collation({locale:'en', strength:2})
Stakeholder Concern #3 db.games.find({gamerTag: “Ace”}) .collation({locale:'en', strength:2}) “Wow, MongoDB can do anything!!!”
Summary
Work Smarter Not Harder • Understand the business logic • Index appropriately • Is it the right index to support the query? • Be aware of: • Blocking Stages • Usage of $or • Case sensitivity • Leverage the Performance Advisor
Work Smarter Not Harder • Understand the business logic • Index appropriately • Is it the right index to support the query? • Be aware of: • Blocking Stages • Usage of $or • Case sensitivity • Leverage the Performance Advisor
Countdown to The Peanuts Powered by an optimized MongoDB environment, The Peanuts is sure to be a hit!
Tips and Tricks for Avoiding Common Query Pitfalls Christian Kurze

More Related Content

What's hot (11)

PDF
MongoDB World 2019: How to Keep an Average API Response Time Less than 5ms wi...
MongoDB
 
PPT
Heroku Waza 2013 Lessons Learned
Simon Bagreev
 
PPTX
How to win $10m - analysing DOTA2 data in R (Sheffield R Users Group - May)
Paul Richards
 
PDF
MongoDB .local Toronto 2019: Tips and Tricks for Effective Indexing
MongoDB
 
PDF
MongoDB .local Paris 2020: La puissance du Pipeline d'Agrégation de MongoDB
MongoDB
 
PDF
Search Engines: How They Work and Why You Need Them
Toria Gibbs
 
PDF
New opportunities for connected data
Neo4j
 
PPT
MySQLConf2009: Taking ActiveRecord to the Next Level
Blythe Dunham
 
PDF
Python WATs: Uncovering Odd Behavior
Amy Hanlon
 
PDF
Firebase_not_really_yohoho
Roman Sachenko
 
PPTX
Firebase not really_yohoho
DA-14
 
MongoDB World 2019: How to Keep an Average API Response Time Less than 5ms wi...
MongoDB
 
Heroku Waza 2013 Lessons Learned
Simon Bagreev
 
How to win $10m - analysing DOTA2 data in R (Sheffield R Users Group - May)
Paul Richards
 
MongoDB .local Toronto 2019: Tips and Tricks for Effective Indexing
MongoDB
 
MongoDB .local Paris 2020: La puissance du Pipeline d'Agrégation de MongoDB
MongoDB
 
Search Engines: How They Work and Why You Need Them
Toria Gibbs
 
New opportunities for connected data
Neo4j
 
MySQLConf2009: Taking ActiveRecord to the Next Level
Blythe Dunham
 
Python WATs: Uncovering Odd Behavior
Amy Hanlon
 
Firebase_not_really_yohoho
Roman Sachenko
 
Firebase not really_yohoho
DA-14
 

Similar to Tips and Tricks for Avoiding Common Query Pitfalls Christian Kurze (20)

PPTX
MongoDB.local Dallas 2019: Tips & Tricks for Avoiding Common Query Pitfalls
MongoDB
 
PDF
MongoDB .local London 2019: Tips and Tricks++ for Querying and Indexing MongoDB
MongoDB
 
PDF
MongoDB .local London 2019: Tips and Tricks++ for Querying and Indexing MongoDB
Lisa Roth, PMP
 
PDF
MongoDB .local Houston 2019:Tips and Tricks++ for Querying and Indexing MongoDB
MongoDB
 
PDF
MongoDB .local Bengaluru 2019: Tips and Tricks++ for Querying and Indexing Mo...
MongoDB
 
PDF
MongoDB .local San Francisco 2020: Tips and Tricks++ for Querying and Indexin...
MongoDB
 
PDF
MongoDB .local Munich 2019: Tips and Tricks++ for Querying and Indexing MongoDB
MongoDB
 
PDF
MongoDB Performance Tuning
MongoDB
 
KEY
MongoDB - Introduction
Vagmi Mudumbai
 
PPTX
Query Optimization in MongoDB
Hamoon Mohammadian Pour
 
PDF
MongoDB World 2016: Deciphering .explain() Output
MongoDB
 
PDF
Full metal mongo
Israel Gutiérrez
 
PPTX
MongoDB_ppt.pptx
1AP18CS037ShirishKul
 
PDF
Latinoware
kchodorow
 
PDF
MongoD Essentials
zahid-mian
 
KEY
Thoughts on MongoDB Analytics
rogerbodamer
 
PPTX
MongoDB Aggregations Indexing and Profiling
Manish Kapoor
 
PPTX
MongoDB is a document database. It stores data in a type of JSON format calle...
amintafernandos
 
PDF
Mongo db
Toki Kanno
 
PPT
Fast querying indexing for performance (4)
MongoDB
 
MongoDB.local Dallas 2019: Tips & Tricks for Avoiding Common Query Pitfalls
MongoDB
 
MongoDB .local London 2019: Tips and Tricks++ for Querying and Indexing MongoDB
MongoDB
 
MongoDB .local London 2019: Tips and Tricks++ for Querying and Indexing MongoDB
Lisa Roth, PMP
 
MongoDB .local Houston 2019:Tips and Tricks++ for Querying and Indexing MongoDB
MongoDB
 
MongoDB .local Bengaluru 2019: Tips and Tricks++ for Querying and Indexing Mo...
MongoDB
 
MongoDB .local San Francisco 2020: Tips and Tricks++ for Querying and Indexin...
MongoDB
 
MongoDB .local Munich 2019: Tips and Tricks++ for Querying and Indexing MongoDB
MongoDB
 
MongoDB Performance Tuning
MongoDB
 
MongoDB - Introduction
Vagmi Mudumbai
 
Query Optimization in MongoDB
Hamoon Mohammadian Pour
 
MongoDB World 2016: Deciphering .explain() Output
MongoDB
 
Full metal mongo
Israel Gutiérrez
 
MongoDB_ppt.pptx
1AP18CS037ShirishKul
 
Latinoware
kchodorow
 
MongoD Essentials
zahid-mian
 
Thoughts on MongoDB Analytics
rogerbodamer
 
MongoDB Aggregations Indexing and Profiling
Manish Kapoor
 
MongoDB is a document database. It stores data in a type of JSON format calle...
amintafernandos
 
Mongo db
Toki Kanno
 
Fast querying indexing for performance (4)
MongoDB
 
Ad

More from MongoDB (20)

PDF
MongoDB SoCal 2020: Migrate Anything* to MongoDB Atlas
MongoDB
 
PDF
MongoDB SoCal 2020: Go on a Data Safari with MongoDB Charts!
MongoDB
 
PDF
MongoDB SoCal 2020: Using MongoDB Services in Kubernetes: Any Platform, Devel...
MongoDB
 
PDF
MongoDB SoCal 2020: A Complete Methodology of Data Modeling for MongoDB
MongoDB
 
PDF
MongoDB SoCal 2020: From Pharmacist to Analyst: Leveraging MongoDB for Real-T...
MongoDB
 
PDF
MongoDB SoCal 2020: Best Practices for Working with IoT and Time-series Data
MongoDB
 
PDF
MongoDB SoCal 2020: MongoDB Atlas Jump Start
MongoDB
 
PDF
MongoDB .local San Francisco 2020: Powering the new age data demands [Infosys]
MongoDB
 
PDF
MongoDB .local San Francisco 2020: Using Client Side Encryption in MongoDB 4.2
MongoDB
 
PDF
MongoDB .local San Francisco 2020: Using MongoDB Services in Kubernetes: any ...
MongoDB
 
PDF
MongoDB .local San Francisco 2020: Go on a Data Safari with MongoDB Charts!
MongoDB
 
PDF
MongoDB .local San Francisco 2020: From SQL to NoSQL -- Changing Your Mindset
MongoDB
 
PDF
MongoDB .local San Francisco 2020: MongoDB Atlas Jumpstart
MongoDB
 
PDF
MongoDB .local San Francisco 2020: Aggregation Pipeline Power++
MongoDB
 
PDF
MongoDB .local San Francisco 2020: A Complete Methodology of Data Modeling fo...
MongoDB
 
PDF
MongoDB .local San Francisco 2020: MongoDB Atlas Data Lake Technical Deep Dive
MongoDB
 
PDF
MongoDB .local San Francisco 2020: Developing Alexa Skills with MongoDB & Golang
MongoDB
 
PDF
MongoDB .local Paris 2020: Realm : l'ingrédient secret pour de meilleures app...
MongoDB
 
PDF
MongoDB .local Paris 2020: Upply @MongoDB : Upply : Quand le Machine Learning...
MongoDB
 
PDF
MongoDB .local Paris 2020: Les bonnes pratiques pour sécuriser MongoDB
MongoDB
 
MongoDB SoCal 2020: Migrate Anything* to MongoDB Atlas
MongoDB
 
MongoDB SoCal 2020: Go on a Data Safari with MongoDB Charts!
MongoDB
 
MongoDB SoCal 2020: Using MongoDB Services in Kubernetes: Any Platform, Devel...
MongoDB
 
MongoDB SoCal 2020: A Complete Methodology of Data Modeling for MongoDB
MongoDB
 
MongoDB SoCal 2020: From Pharmacist to Analyst: Leveraging MongoDB for Real-T...
MongoDB
 
MongoDB SoCal 2020: Best Practices for Working with IoT and Time-series Data
MongoDB
 
MongoDB SoCal 2020: MongoDB Atlas Jump Start
MongoDB
 
MongoDB .local San Francisco 2020: Powering the new age data demands [Infosys]
MongoDB
 
MongoDB .local San Francisco 2020: Using Client Side Encryption in MongoDB 4.2
MongoDB
 
MongoDB .local San Francisco 2020: Using MongoDB Services in Kubernetes: any ...
MongoDB
 
MongoDB .local San Francisco 2020: Go on a Data Safari with MongoDB Charts!
MongoDB
 
MongoDB .local San Francisco 2020: From SQL to NoSQL -- Changing Your Mindset
MongoDB
 
MongoDB .local San Francisco 2020: MongoDB Atlas Jumpstart
MongoDB
 
MongoDB .local San Francisco 2020: Aggregation Pipeline Power++
MongoDB
 
MongoDB .local San Francisco 2020: A Complete Methodology of Data Modeling fo...
MongoDB
 
MongoDB .local San Francisco 2020: MongoDB Atlas Data Lake Technical Deep Dive
MongoDB
 
MongoDB .local San Francisco 2020: Developing Alexa Skills with MongoDB & Golang
MongoDB
 
MongoDB .local Paris 2020: Realm : l'ingrédient secret pour de meilleures app...
MongoDB
 
MongoDB .local Paris 2020: Upply @MongoDB : Upply : Quand le Machine Learning...
MongoDB
 
MongoDB .local Paris 2020: Les bonnes pratiques pour sécuriser MongoDB
MongoDB
 
Ad

Recently uploaded (20)

PDF
Automate Cybersecurity Tasks with Python
VICTOR MAESTRE RAMIREZ
 
PDF
HiHelloHR – Simplify HR Operations for Modern Workplaces
HiHelloHR
 
PDF
TheFutureIsDynamic-BoxLang witch Luis Majano.pdf
Ortus Solutions, Corp
 
PPTX
Home Care Tools: Benefits, features and more
Third Rock Techkno
 
PPTX
Finding Your License Details in IBM SPSS Statistics Version 31.pptx
Version 1 Analytics
 
PPTX
Agentic Automation Journey Series Day 2 – Prompt Engineering for UiPath Agents
klpathrudu
 
PPTX
Foundations of Marketo Engage - Powering Campaigns with Marketo Personalization
bbedford2
 
PDF
MiniTool Partition Wizard Free Crack + Full Free Download 2025
bashirkhan333g
 
PDF
Generic or Specific? Making sensible software design decisions
Bert Jan Schrijver
 
PDF
Digger Solo: Semantic search and maps for your local files
seanpedersen96
 
PDF
AI + DevOps = Smart Automation with devseccops.ai.pdf
Devseccops.ai
 
PDF
유니티에서 Burst Compiler+ThreadedJobs+SIMD 적용사례
Seongdae Kim
 
PDF
Alexander Marshalov - How to use AI Assistants with your Monitoring system Q2...
VictoriaMetrics
 
PDF
Download Canva Pro 2025 PC Crack Full Latest Version
bashirkhan333g
 
PDF
iTop VPN With Crack Lifetime Activation Key-CODE
utfefguu
 
PDF
Why Businesses Are Switching to Open Source Alternatives to Crystal Reports.pdf
Varsha Nayak
 
PPTX
OpenChain @ OSS NA - In From the Cold: Open Source as Part of Mainstream Soft...
Shane Coughlan
 
PDF
Top Agile Project Management Tools for Teams in 2025
Orangescrum
 
PPTX
Empowering Asian Contributions: The Rise of Regional User Groups in Open Sour...
Shane Coughlan
 
PPTX
Why Businesses Are Switching to Open Source Alternatives to Crystal Reports.pptx
Varsha Nayak
 
Automate Cybersecurity Tasks with Python
VICTOR MAESTRE RAMIREZ
 
HiHelloHR – Simplify HR Operations for Modern Workplaces
HiHelloHR
 
TheFutureIsDynamic-BoxLang witch Luis Majano.pdf
Ortus Solutions, Corp
 
Home Care Tools: Benefits, features and more
Third Rock Techkno
 
Finding Your License Details in IBM SPSS Statistics Version 31.pptx
Version 1 Analytics
 
Agentic Automation Journey Series Day 2 – Prompt Engineering for UiPath Agents
klpathrudu
 
Foundations of Marketo Engage - Powering Campaigns with Marketo Personalization
bbedford2
 
MiniTool Partition Wizard Free Crack + Full Free Download 2025
bashirkhan333g
 
Generic or Specific? Making sensible software design decisions
Bert Jan Schrijver
 
Digger Solo: Semantic search and maps for your local files
seanpedersen96
 
AI + DevOps = Smart Automation with devseccops.ai.pdf
Devseccops.ai
 
유니티에서 Burst Compiler+ThreadedJobs+SIMD 적용사례
Seongdae Kim
 
Alexander Marshalov - How to use AI Assistants with your Monitoring system Q2...
VictoriaMetrics
 
Download Canva Pro 2025 PC Crack Full Latest Version
bashirkhan333g
 
iTop VPN With Crack Lifetime Activation Key-CODE
utfefguu
 
Why Businesses Are Switching to Open Source Alternatives to Crystal Reports.pdf
Varsha Nayak
 
OpenChain @ OSS NA - In From the Cold: Open Source as Part of Mainstream Soft...
Shane Coughlan
 
Top Agile Project Management Tools for Teams in 2025
Orangescrum
 
Empowering Asian Contributions: The Rise of Regional User Groups in Open Sour...
Shane Coughlan
 
Why Businesses Are Switching to Open Source Alternatives to Crystal Reports.pptx
Varsha Nayak
 

Tips and Tricks for Avoiding Common Query Pitfalls Christian Kurze

  • 1. Tips and Tricks for Avoiding Common Query Pitfalls Christian Kurze
  • 3. Roadmap • Motivation • Three items to be aware of: • Blocking stages • Using the $or operator • Case insensitivity
  • 4. The Power of Query Optimization • Query tuning results in: • Improved performance • Reduced resource utilization • This may lead to: • Improved stability and predictability • A smaller hardware footprint Not uncommon to observe efficiency improvements greater than 99%
  • 5. The Challenge @ Acme Games, Inc. Charlie • DBA at Acme Game, Inc. • MongoDB Champion Stakeholders • Developers • Leadership • RDBMS History
  • 6. Acme Games Introduces… The Peanuts Brand new strategy game Launching November 15th
  • 7. Stakeholder Concerns • Game nearly complete • Developers have learned a lot from Charlie
  • 8. Stakeholder Concerns • Game nearly complete • Developers have learned a lot from Charlie Indexes support the efficient execution of queries in MongoDB
  • 9. Stakeholder Concerns • Game nearly complete • Developers have learned a lot from Charlie Indexes support the efficient execution of queries in MongoDB
  • 10. Stakeholder Concerns • Game nearly complete • Developers have learned a lot from Charlie Indexes support the efficient execution of queries in MongoDB Ace Bob Sue … …
  • 11. Stakeholder Concerns • Game nearly complete • Developers have learned a lot from Charlie • App being stress tested
  • 12. Stakeholder Concerns • Game nearly complete • Developers have learned a lot from Charlie • App being stress tested • Concerned over current performance
  • 13. Stakeholder Concern #1 Developers created index db.games.createIndex({ gamerTag: 1 })
  • 14. Stakeholder Concern #1 Developers created index db.games.createIndex({ gamerTag: 1 }) This query takes several seconds to execute: db.games.find( { gamerTag: "Ace" } ).sort({score:-1})
  • 15. Stakeholder Concern #1 Developers created index db.games.createIndex({ gamerTag: 1 }) This query takes several seconds to execute: db.games.find( { gamerTag: "Ace" } ).sort({score:-1}) Adding the index on score does not help! db.games.createIndex({ score: -1 })
  • 16. Stakeholder Concern #1 Developers created index db.games.createIndex({ gamerTag: 1 }) This query takes several seconds to execute: db.games.find( { gamerTag: "Ace" } ).sort({score:-1}) Adding the index on score does not help! db.games.createIndex({ score: -1 }) “Clearly MongoDB is not web-scale!”
  • 18. Blocking Operations ● Formally: ■ “An operation which must process all input before it can begin to produce any output.” ● Opposite of the often desirable “fully pipelined” plan which can stream results back as soon as they are found. ● Commonly observed when a sort is added to a query
  • 47. Blocking Stages • $sort • In aggregation and find • $group • $bucket • $count • $facet Are there any other blocking operations?
  • 48. Working with blocking stages For sorting: • Add a supporting index • Worth the overhead in almost all circumstances For other stages: • Do you need the blocking stage? • Offload to secondary member
  • 49. Stakeholder Concern #1 Performance of db.games.find( { gamerTag: "Ace" } ).sort({score:-1}) “Clearly MongoDB is not web-scale!”
  • 50. Stakeholder Concern #1 Performance of db.games.find( { gamerTag: "Ace" } ).sort({score:-1}) db.games.createIndex({ gamerTag: 1, score:-1 })
  • 51. Stakeholder Concern #1 Performance of db.games.find( { gamerTag: "Ace" } ).sort({score:-1}) db.games.createIndex({ gamerTag: 1, score:-1 }) "That’ll work great!”
  • 52. Stakeholder Concern #2 The $and version of a query returns quickly: db.games.find({ $and : [ { gamerTag: "Ace" }, { score: {$gt: 9000} } ] })
  • 53. Stakeholder Concern #2 The $and version of a query returns quickly: db.games.find({ $and : [ { gamerTag: "Ace" }, { score: {$gt: 9000} } ] }) But the $or version is slow: db.games.find({ $or : [ { gamerTag: "Ace" }, { score: {$gt: 9000} } ] })
  • 54. Stakeholder Concern #2 The $and version of a query returns quickly: db.games.find({ $and : [ { gamerTag: "Ace" }, { score: {$gt: 9000} } ] }) But the $or version is slow: db.games.find({ $or : [ { gamerTag: "Ace" }, { score: {$gt: 9000} } ] }) We just created an index with both those fields… Can’t it be used?
  • 55. $or
  • 56. $and example Query on games: db.games.find({ $and : [ { gamerTag: "Ace" }, { score: {$gt: 9000} } ] }) Matching games: { gamerTag: "Ace", score: 9500 } Non-matching games: { gamerTag: "Ace", score: 500 }, { gamerTag: "Bob", score: 9500 }, { gamerTag: "Bob", score: 500 }
  • 57. Groups of documents { gamerTag: "Ace", score: 9500 } { gamerTag: "Ace", score: 500 } { gamerTag: "Bob", score: 9500 } { gamerTag: "Bob", score: 500 }
  • 58. Groups of documents { gamerTag: "Ace", score: 9500 } { gamerTag: "Ace", score: 500 } { gamerTag: "Bob", score: 9500 } { gamerTag: "Bob", score: 500 } gamerTag: "Ace"
  • 59. Groups of documents { gamerTag: "Ace", score: 9500 } { gamerTag: "Ace", score: 500 } { gamerTag: "Bob", score: 9500 } { gamerTag: "Bob", score: 500 } gamerTag: "Ace" score: {$gt: 9000}
  • 60. $and Venn Diagram (logical) { gamerTag: "Ace", score: 9500 } { gamerTag: "Ace", score: 500 } { gamerTag: "Bob", score: 9500 } { gamerTag: "Bob", score: 500 } db.games.find({ $and : [ { gamerTag: "Ace" }, { score: {$gt: 9000} } ] })gamerTag: "Ace" score: {$gt: 9000}
  • 61. $and Venn Diagram (logical) { gamerTag: "Ace", score: 9500 } { gamerTag: "Ace", score: 500 } { gamerTag: "Bob", score: 9500 } { gamerTag: "Bob", score: 500 } db.games.find({ $and : [ { gamerTag: "Ace" }, { score: {$gt: 9000} } ] })gamerTag: "Ace" score: {$gt: 9000}
  • 62. $and Venn Diagram (logical) db.games.find({ $and : [ { gamerTag: "Ace" }, { score: {$gt: 9000} } ] })gamerTag: "Ace" score: {$gt: 9000}
  • 63. $and Venn Diagram (logical) db.games.find({ $and : [ { gamerTag: "Ace" }, { score: {$gt: 9000} } ] })gamerTag: "Ace" score: {$gt: 9000}
  • 64. $and Venn Diagram (logical) db.games.find({ $and : [ { gamerTag: "Ace" }, { score: {$gt: 9000} } ] })gamerTag: "Ace" score: {$gt: 9000}
  • 65. $and Venn Diagram (logical) db.games.find({ $and : [ { gamerTag: "Ace" }, { score: {$gt: 9000} } ] })gamerTag: "Ace" score: {$gt: 9000}
  • 66. $and Venn Diagram (logical) db.games.find({ $and : [ { gamerTag: "Ace" }, { score: {$gt: 9000} } ] })gamerTag: "Ace" score: {$gt: 9000}
  • 67. $and Venn Diagram (logical) db.games.find({ $and : [ { gamerTag: "Ace" }, { score: {$gt: 9000} } ] }) Ace Bob 500 950 0 500 950 0 {gamerTag:1 , score:-1}
  • 68. $and Venn Diagram (logical) db.games.find({ $and : [ { gamerTag: "Ace" }, { score: {$gt: 9000} } ] }) Ace Bob 500 950 0 500 950 0 "indexBounds" : { "gamerTag" : [ "["Ace", "Ace"]" ], "score" : [ "[inf.0, 9000.0)" ] } {gamerTag:1 , score:-1}
  • 69. $and Venn Diagram (logical) db.games.find({ $and : [ { gamerTag: "Ace" }, { score: {$gt: 9000} } ] }) Ace Bob 500 950 0 500 950 0 "indexBounds" : { "gamerTag" : [ "["Ace", "Ace"]" ], "score" : [ "[inf.0, 9000.0)" ] } {gamerTag:1 , score:-1}
  • 70. $and Venn Diagram (logical) db.games.find({ $and : [ { gamerTag: "Ace" }, { score: {$gt: 9000} } ] }) Ace Bob 500 950 0 500 950 0 "indexBounds" : { "gamerTag" : [ "["Ace", "Ace"]" ], "score" : [ "[inf.0, 9000.0)" ] } {gamerTag:1 , score:-1}
  • 71. $and Venn Diagram (logical) db.games.find({ $and : [ { gamerTag: "Ace" }, { score: {$gt: 9000} } ] }) Ace Bob 500 950 0 500 950 0 "indexBounds" : { "gamerTag" : [ "["Ace", "Ace"]" ], "score" : [ "[inf.0, 9000.0)" ] } {gamerTag:1 , score:-1}
  • 72. $or example Query on games: db.games.find({ $or : [ { gamerTag: "Ace" }, { score: {$gt: 9000} } ] }) Matching games: { gamerTag: "Ace", score: 9500 }, { gamerTag: "Ace", score: 500 }, { gamerTag: "Bob", score: 9500 } Non-matching games: { gamerTag: "Bob", score: 500 }
  • 73. $or Venn Diagram (logical) { gamerTag: "Ace", score: 9500 } { gamerTag: "Ace", score: 500 } { gamerTag: "Bob", score: 9500 } { gamerTag: "Bob", score: 500 } db.games.find({ $or : [ { gamerTag: "Ace" }, { score: {$gt: 9000} } ] })gamerTag: "Ace" score: {$gt: 9000}
  • 74. $or Venn Diagram (logical) { gamerTag: "Ace", score: 9500 } { gamerTag: "Ace", score: 500 } { gamerTag: "Bob", score: 9500 } { gamerTag: "Bob", score: 500 } db.games.find({ $or : [ { gamerTag: "Ace" }, { score: {$gt: 9000} } ] })gamerTag: "Ace" score: {$gt: 9000}
  • 75. $or Venn Diagram (logical) { gamerTag: "Ace", score: 9500 } { gamerTag: "Ace", score: 500 } { gamerTag: "Bob", score: 9500 } { gamerTag: "Bob", score: 500 } db.games.find({ $or : [ { gamerTag: "Ace" }, { score: {$gt: 9000} } ] })gamerTag: "Ace" score: {$gt: 9000}
  • 76. $or Venn Diagram (logical) { gamerTag: "Ace", score: 9500 } { gamerTag: "Ace", score: 500 } { gamerTag: "Bob", score: 9500 } { gamerTag: "Bob", score: 500 } db.games.find({ $or : [ { gamerTag: "Ace" }, { score: {$gt: 9000} } ] })gamerTag: "Ace" score: {$gt: 9000}
  • 77. $or Venn Diagram (logical) { gamerTag: "Ace", score: 9500 } { gamerTag: "Ace", score: 500 } { gamerTag: "Bob", score: 9500 } { gamerTag: "Bob", score: 500 } db.games.find({ $or : [ { gamerTag: "Ace" }, { score: {$gt: 9000} } ] })gamerTag: "Ace" score: {$gt: 9000}
  • 78. $or Venn Diagram (logical) db.games.find({ $or : [ { gamerTag: "Ace" }, { score: {$gt: 9000} } ] })gamerTag: "Ace" score: {$gt: 9000}
  • 79. $or Venn Diagram (logical) db.games.find({ $or : [ { gamerTag: "Ace" }, { score: {$gt: 9000} } ] })gamerTag: "Ace" score: {$gt: 9000}
  • 80. $or Venn Diagram (logical) db.games.find({ $or : [ { gamerTag: "Ace" }, { score: {$gt: 9000} } ] })gamerTag: "Ace" score: {$gt: 9000}
  • 81. $or Venn Diagram (logical) db.games.find({ $or : [ { gamerTag: "Ace" }, { score: {$gt: 9000} } ] })gamerTag: "Ace" score: {$gt: 9000}
  • 82. $or Venn Diagram (logical) db.games.find({ $or : [ { gamerTag: "Ace" }, { score: {$gt: 9000} } ] })gamerTag: "Ace" score: {$gt: 9000}
  • 83. $or (single) Index visualization Ace Bob {gamerTag:1 , score:-1} 500 950 0 500 950 0 db.games.find({ $or : [ { gamerTag: "Ace" }, { score: {$gt: 9000} } ] })
  • 84. $or (single) Index visualization Ace Bob {gamerTag:1 , score:-1} 500 950 0 500 950 0 db.games.find({ $or : [ { gamerTag: "Ace" }, { score: {$gt: 9000} } ] }) Expected Index Bounds: "indexBounds" : { "gamerTag" : [ "["Ace", "Ace"]" ], "score" : [ "[inf.0, 9000)" ] }
  • 85. $or (single) Index visualization Ace Bob {gamerTag:1 , score:-1} 500 950 0 500 950 0 db.games.find({ $or : [ { gamerTag: "Ace" }, { score: {$gt: 9000} } ] }) Expected Index Bounds: "indexBounds" : { "gamerTag" : [ "["Ace", "Ace"]" ], "score" : [ "[inf.0, 9000)" ] }
  • 86. $or (single) Index visualization Ace Bob {gamerTag:1 , score:-1} 500 950 0 500 950 0 db.games.find({ $or : [ { gamerTag: "Ace" }, { score: {$gt: 9000} } ] }) Expected Index Bounds: "indexBounds" : { "gamerTag" : [ "["Ace", "Ace"]" ], "score" : [ "[inf.0, 9000)" ] }
  • 87. $or (single) Index visualization Ace Bob {gamerTag:1 , score:-1} 500 950 0 500 950 0 db.games.find({ $or : [ { gamerTag: "Ace" }, { score: {$gt: 9000} } ] }) Expected Index Bounds: "indexBounds" : { "gamerTag" : [ "["Ace", "Ace"]" ], "score" : [ "[inf.0, 9000)" ] }
  • 88. $or (single) Index visualization Ace Bob {gamerTag:1 , score:-1} 500 950 0 500 950 0 db.games.find({ $or : [ { gamerTag: "Ace" }, { score: {$gt: 9000} } ] }) Expected Index Bounds: "indexBounds" : { "gamerTag" : [ "["Ace", "Ace"]" ], "score" : [ "[inf.0, 9000)" ] }
  • 89. $or (single) Index visualization Ace Bob {gamerTag:1 , score:-1} 500 950 0 500 950 0 db.games.find({ $or : [ { gamerTag: "Ace" }, { score: {$gt: 9000} } ] }) Expected Index Bounds: "indexBounds" : { "gamerTag" : [ "["Ace", "Ace"]" ], "score" : [ "[inf.0, 9000)" ] }
  • 90. $or (single) Index visualization Ace Bob {gamerTag:1 , score:-1} 500 950 0 500 950 0 db.games.find({ $or : [ { gamerTag: "Ace" }, { score: {$gt: 9000} } ] }) Expected Index Bounds: "indexBounds" : { "gamerTag" : [ "["Ace", "Ace"]" ], "score" : [ "[inf.0, 9000)" ] } Actual (Hinted) Index Bounds: "indexBounds" : { "gamerTag" : [ "[MinKey, MaxKey]" ], "score" : [ "[MaxKey, MinKey]" ] }
  • 91. $or (single) Index visualization Ace Bob {gamerTag:1 , score:-1} 500 950 0 500 950 0 db.games.find({ $or : [ { gamerTag: "Ace" }, { score: {$gt: 9000} } ] }) Expected Index Bounds: "indexBounds" : { "gamerTag" : [ "["Ace", "Ace"]" ], "score" : [ "[inf.0, 9000)" ] } Actual (Hinted) Index Bounds: "indexBounds" : { "gamerTag" : [ "[MinKey, MaxKey]" ], "score" : [ "[MaxKey, MinKey]" ] } So is there anything we can do to improve the performance of this query?
  • 93. $or (multiple) Index visualization db.games.find({ $or : [ { gamerTag: "Ace" }, { score: {$gt: 9000} } ] })
  • 94. $or (multiple) Index visualization Ace db.games.find({ $or : [ { gamerTag: "Ace" }, { score: {$gt: 9000} } ] })
  • 95. $or (multiple) Index visualization Ace Bob db.games.find({ $or : [ { gamerTag: "Ace" }, { score: {$gt: 9000} } ] })
  • 96. $or (multiple) Index visualization Ace Bob {gamerTag:1} db.games.find({ $or : [ { gamerTag: "Ace" }, { score: {$gt: 9000} } ] })
  • 97. $or (multiple) Index visualization Ace Bob {gamerTag:1} 500 db.games.find({ $or : [ { gamerTag: "Ace" }, { score: {$gt: 9000} } ] })
  • 98. $or (multiple) Index visualization Ace Bob {gamerTag:1} 500 950 0 db.games.find({ $or : [ { gamerTag: "Ace" }, { score: {$gt: 9000} } ] })
  • 99. $or (multiple) Index visualization Ace Bob {gamerTag:1} 500 950 0 {score:1} db.games.find({ $or : [ { gamerTag: "Ace" }, { score: {$gt: 9000} } ] })
  • 100. $or (multiple) Index visualization Ace Bob {gamerTag:1} 500 950 0 {score:1} "indexBounds" : { "gamerTag" : [ "["Ace", "Ace"]" ] } "indexBounds" : { "score" : [ "(9000.0, inf.0]" ] } db.games.find({ $or : [ { gamerTag: "Ace" }, { score: {$gt: 9000} } ] })
  • 101. $or (multiple) Index visualization Ace Bob {gamerTag:1} 500 950 0 {score:1} db.games.find({ $or : [ { gamerTag: "Ace" }, { score: {$gt: 9000} } ] }) "indexBounds" : { "gamerTag" : [ "["Ace", "Ace"]" ] } "indexBounds" : { "score" : [ "(9000.0, inf.0]" ] }
  • 102. $or (multiple) Index visualization Ace Bob {gamerTag:1} 500 950 0 {score:1} db.games.find({ $or : [ { gamerTag: "Ace" }, { score: {$gt: 9000} } ] }) "indexBounds" : { "gamerTag" : [ "["Ace", "Ace"]" ] } "indexBounds" : { "score" : [ "(9000.0, inf.0]" ] }
  • 103. $or (multiple) Index visualization Ace Bob {gamerTag:1} 500 950 0 {score:1} db.games.find({ $or : [ { gamerTag: "Ace" }, { score: {$gt: 9000} } ] }) "indexBounds" : { "gamerTag" : [ "["Ace", "Ace"]" ] } "indexBounds" : { "score" : [ "(9000.0, inf.0]" ] }
  • 104. $or (multiple) Index visualization Ace Bob {gamerTag:1} 500 950 0 {score:1} db.games.find({ $or : [ { gamerTag: "Ace" }, { score: {$gt: 9000} } ] }) "indexBounds" : { "gamerTag" : [ "["Ace", "Ace"]" ] } "indexBounds" : { "score" : [ "(9000.0, inf.0]" ] }
  • 105. $or (multiple) Index visualization Ace Bob {gamerTag:1} 500 950 0 {score:1} db.games.find({ $or : [ { gamerTag: "Ace" }, { score: {$gt: 9000} } ] }) "indexBounds" : { "gamerTag" : [ "["Ace", "Ace"]" ] } "indexBounds" : { "score" : [ "(9000.0, inf.0]" ] }
  • 107. Recommendations Use multiple indexes! db.data.createIndex({gamerTag: 1}) db.data.createIndex({score: 1}) We already have the {gamerTag:1, score:-1} index, do we need both of these new ones?
  • 108. Recommendations Use multiple indexes! db.data.createIndex({gamerTag: 1}) db.data.createIndex({gamerTag: 1, score: -1}} db.data.createIndex({score: 1}) We already have the {gamerTag:1, score:-1} index, do we need both of these new ones?
  • 109. Recommendations Use multiple indexes! db.data.createIndex({gamerTag: 1}) db.data.createIndex({gamerTag: 1, score: -1}} db.data.createIndex({score: 1}) Rule of thumb for compound indexes: equality, sort, range
  • 110. Recommendations Use multiple indexes! db.data.createIndex({gamerTag: 1}) db.data.createIndex({gamerTag: 1, score: -1}} db.data.createIndex({score: 1}) Rule of thumb for compound indexes: equality, sort, range Works with sorting Generate a SORT_MERGE plan
  • 111. Stakeholder Concern #2 db.games.find({ $or : [ { gamerTag: "Ace" }, { score: {$gt: 9000} } ] }) Having the right index is critical
  • 112. Stakeholder Concern #2 db.games.find({ $or : [ { gamerTag: "Ace" }, { score: {$gt: 9000} } ] }) Having the right index is critical
  • 113. Stakeholder Concern #2 db.games.find({ $or : [ { gamerTag: "Ace" }, { score: {$gt: 9000} } ] }) Having the right index is critical "Super!!”
  • 114. Stakeholder Concern #3 “Wait wait wait, we can’t even FIND the gamers!” A basic search on gamerTag takes several seconds already: db.games.find({gamerTag: /Ace/i}) “This query is SLOWER with the index than it is without it!”
  • 116. Case Sensitive db.games.find({ gamerTag: /^Ace$/ }) //equivalent to db.games.find({ gamerTag: “Ace” }) Matching games: { gamerTag: "Ace", score: 9500 }, Non-matching games: { gamerTag: "ACE", score: 500 }, { gamerTag: "aCe", score: 9500 }, { gamerTag: "ace", score: 0 }, { gamerTag: "Bob", score: 500 }, { gamerTag: "acxyz", score: 9500 }, { gamerTag: "Ace mdb", score: 9500 }
  • 117. Case Sensitive ace aCe acxy z Ace Ace mdb ACE Bob "indexBounds" : { "gamerTag" : [ "["Ace", "Acf")", "[/^Ace$/, /^Ace$/]" ] } Matching games: { gamerTag: "Ace", score: 9500 }, Non-matching games: { gamerTag: "ACE", score: 500 }, { gamerTag: "aCe", score: 9500 }, { gamerTag: "ace", score: 0 }, { gamerTag: "Bob", score: 500 }, { gamerTag: "acxyz", score: 9500 }, { gamerTag: "Ace mdb", score: 9500 }
  • 118. Case insensitive db.games.find({ gamerTag: /^Ace$/i }) //equivalent to db.games.find({ gamerTag: { $regex: “^Ace$”, $options: “i” } }) //equivalent to db.games.find({ gamerTag: “Ace”}) .collation({locale:’en’, strength:2}) Matching games: { gamerTag: "Ace", score: 9500 }, { gamerTag: "ACE", score: 500 }, { gamerTag: "aCe", score: 9500 }, { gamerTag: "ace", score: 0 } Non-matching games: { gamerTag: "Bob", score: 500 }, { gamerTag: "acxyz", score: 9500 }, { gamerTag: "Ace mdb", score: 9500 }
  • 119. Case insensitive db.games.find({ gamerTag: /^Ace$/i }) //equivalent to db.games.find({ gamerTag: { $regex: “^Ace$”, $options: “i” } }) //equivalent to db.games.find({ gamerTag: “Ace”}) .collation({locale:’en’, strength:2}) Matching games: { gamerTag: "Ace", score: 9500 }, { gamerTag: "ACE", score: 500 }, { gamerTag: "aCe", score: 9500 }, { gamerTag: "ace", score: 0 } Non-matching games: { gamerTag: "Bob", score: 500 }, { gamerTag: "acxyz", score: 9500 }, { gamerTag: "Ace mdb", score: 9500 } Would a $text search be the same as well?
  • 120. Case INsensitive ace aCe acxy z Ace Ace mdb ACE Bob "indexBounds" : { "gamerTag" : [ "["", {})", "[/^Ace$/i, /^Ace$/i]" ] } Matching games: { gamerTag: "Ace", score: 9500 }, { gamerTag: "ACE", score: 500 }, { gamerTag: "aCe", score: 9500 }, { gamerTag: "ace", score: 0 } Non-matching games: { gamerTag: "Bob", score: 500 }, { gamerTag: "acxyz", score: 9500 }, { gamerTag: "Ace mdb", score: 9500 }
  • 122. Recommendations Case insensitive index! Collations available since 3.4 db.games.createIndex( { gamerTag: 1}, { collation: { locale: 'en', strength: 2 } } )
  • 123. Recommendations Case insensitive index! Collations available since 3.4 db.games.createIndex( { gamerTag: 1}, { collation: { locale: 'en', strength: 2 } } ) > db.games.find( { gamerTag: "Ace"}).collation( { locale: 'en', strength: 2 } )
  • 124. Recommendations Case insensitive index! Collations available since 3.4 db.games.createIndex( { gamerTag: 1}, { collation: { locale: 'en', strength: 2 } } ) > db.games.find( { gamerTag: "Ace"}).collation( { locale: 'en', strength: 2 } ) { "_id" : ObjectId("5b29dbee6c7d4f531bf73b5d"), "gamerTag" : "Ace", "score" : 9500 } { "_id" : ObjectId("5b29dbee6c7d4f531bf73b5e"), "gamerTag" : "ACE", "score" : 500 } { "_id" : ObjectId("5b29dbee6c7d4f531bf73b5f"), "gamerTag" : "aCe", "score" : 9500 } { "_id" : ObjectId("5b29dbee6c7d4f531bf73b60"), "gamerTag" : "ace", "score" : 0 }
  • 125. Recommendations Case insensitive index! Collations available since 3.4 db.games.createIndex( { gamerTag: 1}, { collation: { locale: 'en', strength: 2 } } ) Store a transformed (eg toLower()) copy of the string
  • 126. Stakeholder Concern #3 db.games.find({gamerTag: “Ace”}) .collation({locale:'en', strength:2})
  • 127. Stakeholder Concern #3 db.games.find({gamerTag: “Ace”}) .collation({locale:'en', strength:2})
  • 128. Stakeholder Concern #3 db.games.find({gamerTag: “Ace”}) .collation({locale:'en', strength:2}) “Wow, MongoDB can do anything!!!”
  • 130. Work Smarter Not Harder • Understand the business logic • Index appropriately • Is it the right index to support the query? • Be aware of: • Blocking Stages • Usage of $or • Case sensitivity • Leverage the Performance Advisor
  • 131. Work Smarter Not Harder • Understand the business logic • Index appropriately • Is it the right index to support the query? • Be aware of: • Blocking Stages • Usage of $or • Case sensitivity • Leverage the Performance Advisor
  • 132. Countdown to The Peanuts Powered by an optimized MongoDB environment, The Peanuts is sure to be a hit!