MongoDB .local Bengaluru 2019: Aggregation Pipeline Power++: How MongoDB 4.2 Pipeline Empowers Queries, Updates, and Materialized Views

#MDBlocal
How MongoDB 4.2 Pipeline is
Powering Queries, Updates and Views
Sanjay Gianchandani
AGGREGATION POWER++

#MDBW17
Analytics with MongoDB Aggregation Framework
@asya999 by Asya Kamsky,
Lead MongoDB Maven
PIPELINE POWER

#MDBLocal
ps ax | grep mongod | head 1
*nix command line pipe
PIPELINE

#MDBLocal
$match $group | $sort|
Input stream {} {} {} {} Result {} {} ...
PIPELINE
MongoDB document pipeline

Stage 1 Stage 2 Stage 3 Stage 4
{} {} {} {}
{} {} {} {}
DATA PIPELINE
{} {} {} {}
{"$stage":{ ... }}
START
Collection
View
Special stage
STAGES

{title: "The Great Gatsby",
language: "English",
subjects: "Long Island"}
subjects: "New York"}
subjects: "1920s"}
subjects: [
"Long Island",
"New York",
"1920s"] },
{"$match":{"language":"English"}}
$match
{ _id:"Long Island",
count: 1 },
$group
{ _id: "New York",
count: 2 },
$unwind
{ _id: "1920s",
count: 1 },
$sort $skip$limit $project
{"$unwind":"$subjects"}
{"$group":{"_id":"$subjects", "count":{"$sum:1}}
{ _id: "Harlem",
count: 1 },
{ _id: "Long Island",
count: 1 },
{ _id: "New York",
count: 2 },
{ _id: "1920s",
count: 1 },
{title: "Open City",
subjects: [
"New York"
"Harlem" ] }
{ title: "The Great Gatsby",
subjects: [
"Long Island",
"New York",
"1920s"] },
{ title: "War and Peace",
language: "Russian",
subjects: [
"Russia",
"War of 1812",
"Napoleon"] },
{ title: "Open City",
subjects: [
"New York",
"Harlem" ] },
subjects: "Harlem"}
{ _id: "Harlem",
count: 1 },
{"$sort:{"count":-1} {"$limit":3}
{"$project":...}

language: "English".
subjects: "Long Island"}
subjects: "1920s"}
subjects: [
"Long Island",
"New York",
"1920s"] },
{"$match":{"language":"English"}}
$match
{ _id:"Long Island",
count: 1 },
$group
{ _id: "New York",
count: 2 },
$unwind
{ _id: "1920s",
count: 1 },
$sort $skip$limit $project
{"$unwind":"$subjects"}
{"$group":{"_id":"$subjects", "count":{"$sum:1}}
{ _id: "Harlem",
count: 1 },
{ _id: "Long Island",
count: 1 },
{ _id: "New York",
count: 2 },
{ _id: "1920s",
count: 1 },
subjects: [
"New York"
"Harlem" ] }
{ title: "The Great Gatsby",
subjects: [
"Long Island",
"New York",
"1920s"] },
{ title: "War and Peace",
language: "Russian",
subjects: [
"Russia",
"War of 1812",
"Napoleon"] },
{ title: "Open City",
subjects: [
"New York",
"Harlem" ] },
subjects: "Harlem"}
{ _id: "Harlem",
count: 1 },
{"$sort:{"count":-1} {"$limit":3}
{"$project":...}
$group $sort
1

#MDBLocal
INPUT STAGE RESULTSSTAGE
STREAMING RESOURCE USE
Each document is streamed through in RAM

#MDBLocal
INPUT STAGE RESULTSSTAGE
BLOCKING RESOURCE USE
Everything has to be kept in RAM (or spill)

5 minute review
https://github.com/asya999/mdbw17

PREVIOUSLY ...
... 2017 ... 2018

#MDBLocal
THE FUTURE OF AGGREGATION
Better performance & optimizations
More stages & expressions
More options for output
Compass helper for aggregate
Unify different languages

#MDBLocal

#MDBLocal
THE PRESENT OF AGGREGATION

#MDBLocal
Unify Different Languages

#MDBLocal
{children: [
{name:"Max", dob:"1994-12-01", dep:true},
{name:"Sam", dob:"1997-09-28", dep:true},
{name:"Kim", dob:"2000-02-29", dep:true}
]}
AGGREGATION

#MDBLocal
{children: [
]}
AGGREGATION
db.c.aggregate([
{$addFields:{
numChildren:{$size:"$children"},
numDependents:{$size:{
$filter:{
input:"$children.dep",
cond: "$$this"
}
}}
}},
...
])

#MDBLocal
{children: [
]}
AGGREGATION
FIND
db.c.aggregate([
{$addFields:{
numChildren:{$size:"$children"},
numDependents:{$size:{
$filter:{
input:"$children.dep",
cond: "$$this"
}
}}
}},
...
])

#MDBLocal
{children: [
]}
AGGREGATION
FIND
db.c.find (
{$expr:{
$lt:[
{$size:{$filter:{
input: "$children.dep",
cond: "$$this"
}}},
2
]
}}
)

#MDBLocal
{children: [
]}
AGGREGATION
FIND
UPDATE
db.c.find (
{$expr:{
$lt:[
{$size:{$filter:{
input: "$children.dep",
cond: "$$this"
}}},
2
]
}}
)

#MDBLocal
{children: [
]}
AGGREGATION
FIND
UPDATE
db.c.update(
{$expr:{
$anyElementTrue:{$map:{
input:"$children",
in: {$and:[
{$lt:["$$this.dob","1997-01-22"]},
"$$this.dep"
]}
}}
}},
{$set:{ audit:true }}
)

#MDBLocal
Update
db.coll.update(
<query>,
<update>,
<options>
)

#MDBLocal
Update
db.coll.update(
<query>,
<update>,
<options>
)
<update>

#MDBLocal
Update
{
f1: <value>,
f2: <value>,
...
}
{
$set: { },
$inc: { },
$...
}
<update>

#MDBLocal
Update in 4.2
{ } OR [ ]
<update>

#MDBLocal
Update in 4.2
{ <same> } [ ]
<update>

#MDBLocal
Update in 4.2
{ <same> } [ <aggregation-pipeline> ]
<update>

Updates Using Aggregation
Pipeline

#MDBLocal
{ $addFields: { } }
{ $project: { } }
{ $replaceRoot: { } }
{ $set: { } }
{ $unset: [ ] }
{ $replaceWith: { } }

#MDBLocal
db.coll.update({_id:1},
{$inc:{a:1}},
{upsert:true})
{ _id: 1 }
{ _id: 1, a: 10 }
{ _id: 1, a: 100 }
---
{ _id: 1, a: "10" }
{ _id: 1, a: 1 }
{ _id: 1, a: 11 }
{ _id: 1, a: 101 }
{ _id: 1, a: 1 }
"errmsg" : "Cannot apply to a value of
non-numeric type."

#MDBLocal
{ _id: 1 }
{ _id: 1, a: 10 }
{ _id: 1, a: 100 }
---
{ _id: 1, a: "10" }
{ _id: 1, a: 1 }
{ _id: 1, a: 11 }
{ _id: 1, a: 101 }
{ _id: 1, a: 1 }
{ _id: 1, a: 1 }
[ {$set:{a:{$sum:["$a",1]}}} ],
{upsert:true})

#MDBLocal
{ _id: 1 }
{ _id: 1, a: 10 }
{ _id: 1, a: 100 }
---
{ _id: 1, a: "10" }
{ _id: 1, a: 1 }
{ _id: 1, a: 11 }
{ _id: 1, a: 101 }
{ _id: 1, a: 1 }
"errmsg" : "$add only supports
numeric or date types, not string"
[ {$set:{a:{$add:["$a",1]}}} ],
{upsert:true})

#MDBLocal
{ _id: 1 }
{ _id: 1, a: 10 }
{ _id: 1, a: 100 }
---
{ _id: 1, a: "10" }
{ _id:1, a: 21 }
{ _id: 1, a: 11 }
{ _id: 1, a: 101 }
{ _id:1, a: 21 }
[ {$set:{a:{$ }} ],
{upsert:true})

#MDBLocal
{ _id: 1 }
{ _id: 1, a: 10 }
{ _id: 1, a: 100 }
---
{ _id: 1, a: "10" }
{ _id:1, a: 21 }
{ _id: 1, a: 11 }
{ _id: 1, a: 101 }
{ _id:1, a: 21 }
db.coll.update({_id:1}, [ {$set:{a:{$cond:{
if: ,
then: , else: }}}}], {upsert:true})

#MDBLocal
{ _id: 1 }
{ _id: 1, a: 10 }
{ _id: 1, a: 100 }
---
{ _id: 1, a: "10" }
{ _id:1, a: 21 }
{ _id: 1, a: 11 }
{ _id: 1, a: 101 }
{ _id:1, a: 21 }
if: {$eq:[{$type:"$a"},"missing"]},
then: , else: }}}}], {upsert:true})

#MDBLocal
{ _id: 1 }
{ _id: 1, a: 10 }
{ _id: 1, a: 100 }
---
{ _id: 1, a: "10" }
{ _id:1, a: 21 }
{ _id: 1, a: 11 }
{ _id: 1, a: 101 }
{ _id:1, a: 21 }
then: 21, else: {$sum:["$a",1]} }}}}], {upsert:true})

#MDBLocal
{ _id: 1 }
{ _id: 1, a: 10 }
{ _id: 1, a: 100 }
---
{ _id: 1, a: "10" }
{ _id:1, a: 21 }
{ _id: 1, a: 11 }
{ _id: 1, a: 100 }
{ _id:1, a: 21 }
then: 21, else: {$sum:["$a",1]} }}}}], {upsert:true})

#MDBLocal
{ _id: 1 }
{ _id: 1, a: 10 }
{ _id: 1, a: 100 }
---
{ _id: 1, a: "10" }
{ _id:1, a: 21 }
{ _id: 1, a: 11 }
{ _id: 1, a: 100 }
{ _id:1, a: 21 }
db.coll.update({_id:1}, [ {$set:{a:{$min:[100, {$cond:{
then: 21, else: {$sum:["$a",1]} }}]} }}], {upsert:true})

#MDBLocal
{ _id: 1 }
{ _id: 1, a: 10 }
{ _id: 1, a: 100 }
---
{ _id: 1, a: "10" }
{ _id:1, a: 21 }
{ _id: 1, a: 11 }
{ _id: 1, a: 100 }
{ _id:1, a: 21 }
{ _id:1, a: 1 }
then: 21, else: {$sum:["$a",1]} }}]} }}], {upsert:true})

#MDBLocal
{ _id: 1 }
{ _id: 1, a: 10 }
{ _id: 1, a: 100 }
---
{ _id: 1, a: "10" }
{ _id:1, a: 21 }
{ _id: 1, a: 11 }
{ _id: 1, a: 100 }
{ _id:1, a: 21 }
{ _id:1, a: 1 }
then: 21, else: {$sum:["$a",1]} }}]}, prev_a:"$a" }}],
{upsert:true})

#MDBLocal
{ _id: 1 }
{ _id: 1, a: 10 }
{ _id: 1, a: 100 }
---
{ _id: 1, a: "10" }
{ _id:1, a: 21 }
{ _id: 1, a: 11, prev_a: 10 }
{ _id: 1, a: 100, prev_a: 100 }
{ _id:1, a: 21 }
{ _id:1, a: 1, prev_a: "10" }
then: 21, else: {$sum:["$a",1]} }}]}, prev_a:"$a" }}],
{upsert:true})

#MDBLocal
Set Defaults
{_id: 1, a: 5, b: 12}
{_id: 2, a: 15, c: "abc"}
{_id: 3, b: 99, c: "xyz"}
If a or b are missing, set to 0, if c is missing -> "unset"

#MDBLocal
Set Defaults
{_id: 1, a: 5, b: 12}
{_id: 2, a: 15, c: "abc"}
{_id: 3, b: 99, c: "xyz"}
db.coll.update({}, [
{$replaceWith:{
}}
], {multi:true})

#MDBLocal
Set Defaults
{_id: 1, a: 5, b: 12}
{_id: 2, a: 15, c: "abc"}
{_id: 3, b: 99, c: "xyz"}
{$replaceWith:{$mergeObjects:[
]}}
], {multi:true})

#MDBLocal
Set Defaults
{_id: 1, a: 5, b: 12}
{_id: 2, a: 15, c: "abc"}
{_id: 3, b: 99, c: "xyz"}
{ a:0, b:0, c:"unset" },
"$$ROOT"
]}}
], {multi:true})

#MDBLocal
Set Defaults
{_id: 1, a: 5, b: 12}
{_id: 2, a: 15, c: "abc"}
{_id: 3, b: 99, c: "xyz"}
{ a:0, b:0, c:"unset" },
"$$ROOT"
]}}
], {multi:true})
{_id: 1, a: 5, b: 12, c: "unset"}
{_id: 2, a: 15, b: 0, c: "abc"}
{_id: 3, a: 0, b: 99, c: "xyz"}

#MDBLocal
{ id: 1,
d: ISODate("2019-06-04T00:00:00"),
h: [
{ hour:"11", value: 296 },
{ hour:"12", value: 300 }
]}
id: X, d:Y, hour:Z, value: VAL
db.coll.update({id:X, d:Y},
[ {$set:{h:{$cond:{
if:
then:
else:
}}}}],
{upsert:true})

#MDBLocal
{ id: 1,
d: ISODate("2019-06-04T00:00:00"),
h: [
{ hour:"11", value: 296 },
{ hour:"12", value: 300 }
]}
id: X, d:Y, hour:Z, value: VAL
db.coll.update({id:X, d:Y},
[ {$set:{h:{$cond:{
if: {$in:[Z,{$ifNull:["$h.hour",[]]}]},
then:{$map:{
input:"$h",
in: {$cond:{ if:{$ne:["$$this.hour",Z]}, then:"$$this",
else: {hour: Z, value: {$sum:[ "$$this.value", VAL]}}
}}}},
else:{$concatArrays:[{$ifNull:["$h",[]]},[{hour:Z,value:VAL}]]}
}}}}],
{upsert:true})
if:
then:
else:

#MDBLocal
Recap:
Updates can be specified with aggregation pipeline
All fields from existing document can be accessed
Slightly slower, but a lot more powerful

#MDBLocal

#MDBLocal
More Options for Output

#MDBLocal
Prior to MongoDB 4.2
$out
coll
new_coll
$out

#MDBLocal
$out
coll
new_coll
$out
db.coll.aggregate( [ {pipeline}, ...
{$out: "new_coll"} ]);

#MDBLocal
$out
coll
new_coll
$out
db.coll.aggregate( [ {pipeline}, ...
{$out: "new_coll"} ]);
new_coll
○ must be unsharded
○ overwrites existing

New $merge stage
in aggregation pipeline

#MDBLocal
MongoDB 4.2
$merge
coll
coll2
$merge

#MDBLocal
MongoDB 4.2
$merge
db.coll.aggregate( [
{pipeline}, ...,
{$merge: { ... }
]);
coll
coll2
$merge

#MDBLocal
MongoDB 4.2
$merge
db.coll.aggregate( [
{pipeline}, ...,
{$merge: { ... }
]);
coll2
can exist
same or different 'db'
can be sharded
coll
coll2
$merge

#MDBLocal
coll
coll2
$merge
{ } { } { } { }
{ } { } { } { }
MongoDB 4.2

#MDBLocal
{
$merge: {
into: <target>
}
}
$merge syntax

#MDBLocal
{$merge: "collection2"}
$merge syntax
{
$merge: {
into: <target>
}
}

#MDBLocal
{$merge: {into: {db: "db2", coll: "collection2"}}
$merge syntax
{
$merge: {
into: <target>
}
}

#MDBLocal
{
$merge: {
into: <target>,
on: <fields>
}
}
on: "_id"
on: [ "_id", "shardkey(s)" ]
must be unique
$merge syntax

#MDBLocal
{
$merge: {
into: <target>,
on: <fields>
}
}
$merge syntax

#MDBLocal
Actions
source target

#MDBLocal
Actions
nothing matched:
source target

#MDBLocal
Actions
nothing matched: usually insert
source target

#MDBLocal
Actions
document matched:
source target

#MDBLocal
Actions
document matched: overwrite? update? ???
source target

#MDBLocal
Actions
document matched: update
source target

#MDBLocal
Actions
document matched: update (merge)
source target

#MDBLocal
$merge syntax
{
$merge: {
into: <target>,
whenNotMatched:
whenMatched:
}
}

#MDBLocal
$merge syntax
{
$merge: {
into: <target>,
whenNotMatched:"insert",
whenMatched:
}
}

#MDBLocal
$merge syntax
{
$merge: {
into: <target>,
whenNotMatched:"insert",
whenMatched:"merge"
}
}

#MDBLocal
$merge syntax
{
$merge: {
into: <target>,
whenNotMatched:"insert"|"discard"|"fail",
whenMatched:"merge"
}
}

#MDBLocal
$merge syntax
{
$merge: {
into: <target>,
whenNotMatched:"insert"|"discard"|"fail",
whenMatched:"merge"|"replace"|"keepExisting"|"fail"|[...]
}
}

#MDBLocal
$merge syntax
{
$merge: {
into: <target>,
whenMatched:[...]
}
}

#MDBLocal
$merge syntax
{
$merge: {
into: <target>,
whenMatched:[<custom pipeline>]
}
}

#MDBLocal
$merge example
{
$merge: {
into: <target>,
whenMatched:[
{$addFields:{
}}
]
}
}

#MDBLocal
$merge example
{
$merge: {
into: <target>,
whenMatched:[
{$addFields:{
total:{$sum:["$total","$$new.total"]}
}}
]
}
}

#MDBLocal
$merge example
{
$merge: {
into: <target>,
whenMatched:[
{$set:{
}}
]
}
}

#MDBLocal
$merge example
{
$merge: {
into: <target>,
whenMatched:[
{$set:{
}}
]
}
}
Incoming Target
{
_id: "37",
total: 64,
f1: "x"
}
{
_id: "37",
total: 245,
f1: "yyy"
}
Result:
{
}

#MDBLocal
$merge example
{
$merge: {
into: <target>,
whenMatched:[
{$set:{
}}
]
}
}
Incoming Target
{
_id: "37",
total: 64,
f1: "x"
}
{
_id: "37",
total: 245,
f1: "yyy"
}
Result:
{
_id: "37",
total: 309,
f1: "yyy"
}

#MDBLocal
$merge example 2
{
$merge: {
into: <target>,
whenMatched:[
"$$new",
{total:{$sum:["$$new.total", "$total"]}}
]}}
]
}
}

#MDBLocal
$merge example 2
{
$merge: {
into: <target>,
whenMatched:[
"$$new",
]}}
]
}
}
Incoming Target
{
_id: "37",
total: 64,
f1: "x"
}
{
_id: "37",
total: 245,
f1: "yyy"
}
Result:
{
}

#MDBLocal
$merge example 2
{
$merge: {
into: <target>,
whenMatched:[
"$$new",
]}}
]
}
}
Incoming Target
{
_id: "37",
total: 64,
f1: "x"
}
{
_id: "37",
total: 245,
f1: "yyy"
}
Result:
{
_id: "37",
total: 309,
f1: "x"
}

#MDBLocal
$merge syntax
{
$merge: {
into: <target>,
let: { ... },
whenMatched:[ ...]
}
}

#MDBLocal
$merge syntax
{
$merge: {
into: <target>,
let: {new: "$$ROOT"},
whenMatched:[ ...]
}
}

#MDBLocal
{
$merge: {
into: <target>,
whenMatched:[
{$set:{
}}
]
}
}

#MDBLocal
{
$merge: {
into: <target>,
let: {itotal: "$total"},
whenMatched:[
{$set:{
total:{$sum:["$total","$$itotal"]}
}}
]
}
}
{
$merge: {
into: <target>,
whenMatched:[
{$set:{
}}
]
}
}

EXAMPLES
APPEND from TEMP collection

#MDBLocal
temp
real
data
real
Using $merge to append loaded and
cleansed records loaded into db

#MDBLocal
aggregate 'temp' and append valid records to 'data'
db.temp.aggregate( [
{ ... } /* pipeline to massage and cleanse data in temp */,
{$merge:{
into: "data",
whenMatched: "fail"
}}
]);

#MDBLocal
aggregate 'temp' and append valid records to 'data'
db.temp.aggregate( [
{ ... } /* pipeline to massage and cleanse data in temp */,
{$merge:{
into: "data",
whenMatched: "fail"
}}
]);
Similar to SQL's INSERT INTO T1 SELECT * from T2

#MDBLocal
mflix
users
users
mfriendbook
users
sv
Using $merge to populate/update
user fields from other services

#MDBLocal
mflix
users
users
mfriendbook
users
sv
Using $merge to populate/update
user fields from other services
sv.users
{
_id: "user253",
dob: ISODate(...),
f1: "yyy"
}

#MDBLocal
$merge updates fields from mflix.users collection into
sv.users collection. Our "_id" field is unique username
mflix_pipeline = [
{ "$project" : {
"_id" : "$username",
"mflix" : "$$ROOT"
}},
{ "$merge" : {
"into" : {
"db": "sv",
"collection" : "users"
},
"whenNotMatched" : "discard"
}}
]
(in mflix)
sv.users
{
_id: "user253",
dob: ISODate(...),
f1: "yyy"
}

#MDBLocal
$merge updates fields from mflix.users collection into
mflix_pipeline = [
{ "$project" : {
"mflix" : "$$ROOT"
}},
{ "$merge" : {
"into" : {
"db": "sv",
},
}}
]
(in mflix) db.users.aggregate(mflix_pipeline)
sv.users
{
_id: "user253",
dob: ISODate(...),
f1: "yyy",
mflix: { ... }
}

#MDBLocal
$merge updates fields from mfriendbook.users collection into
mfriendbook_pipeline = [
{ "$project" : {
"mfriendbook" : "$$ROOT"
}},
{ "$merge" : {
"into" : {
"db": "sv",
},
}}
]
(in mfriendbook)
sv.users
{
_id: "user253",
dob: ISODate(...),
f1: "yyy",
mflix: { ... }
}

#MDBLocal
$merge updates fields from mfriendbook.users collection into
mfriendbook_pipeline = [
{ "$project" : {
"mfriendbook" : "$$ROOT"
}},
{ "$merge" : {
"into" : {
"db": "sv",
},
}}
]
(in mfriendbook) db.users.aggregate(mfriendbook_pipeline)
sv.users
{
_id: "user253",
dob: ISODate(...),
f1: "yyy",
mflix: { ... },
mfriendbook: { ... }
}

EXAMPLES
Populate ROLLUPS into summary table

registrations
real
regsummary
real
Using $merge to incrementally
update periodic rollups in summary

#MDBLocal
$merge to create/update periodic
rollups in summary collection (for all days)
db.regsummary.createIndex({event:1, date:1}, {unique: true});

#MDBLocal
db.registrations.aggregate([
{$match: {event_id: "MDBW19"}},
{$group:{
_id:{$dateToString:{date:"$date",format:"%Y-%m-%d"}},
count: {$sum:1}
}},
{$project: {_id:0,event:"MDBW19",date:"$_id",total:"$count"}},
{$merge: {
into: "regsummary",
on: ["event", "date"]
}}
])

#MDBLocal
{$match: {event_id: "MDBW19"}},
{$group:{
_id:{$dateToString:{date:"$date",format:"%Y-%m-%d"}},
count: {$sum:1}
}},
{$project: {_id:0,event:"MDBW19",date:"$_id",total:"$count"}},
{$merge: {
into: "regsummary",
}}
])
{ "event" : "MDBW19", "date" : "2019-05-19", "total" : 33 }

#MDBLocal
$merge to incrementally update periodic rollups in
summary collection (for single day)

#MDBLocal
{$match: {
event_id: "MDBW19",
date:{$gte:ISODate("2019-05-22"),$lt:ISODate("2019-05-23")}
}},
{$count: "total"},
{$addFields: {event:"MDBW19", "date":"2019-05-22"}},
{$merge: {
into: "regsummary",
}}
])

MongoDB .local Bengaluru 2019: Aggregation Pipeline Power++: How MongoDB 4.2 Pipeline Empowers Queries, Updates, and Materialized Views

https://github.com/asya999/mdbw17

MongoDB .local Bengaluru 2019: Aggregation Pipeline Power++: How MongoDB 4.2 Pipeline Empowers Queries, Updates, and Materialized Views

Recommended

More Related Content

What's hot (20)

Similar to MongoDB .local Bengaluru 2019: Aggregation Pipeline Power++: How MongoDB 4.2 Pipeline Empowers Queries, Updates, and Materialized Views (20)

More from MongoDB (20)

Recently uploaded (20)

MongoDB .local Bengaluru 2019: Aggregation Pipeline Power++: How MongoDB 4.2 Pipeline Empowers Queries, Updates, and Materialized Views