Webinar: Schema Patterns and Your Storage Engine
2 likes2,156 views
How do MongoDB’s different storage options change the way you model your data? Each storage engine, WiredTiger, the In-Memory Storage engine, MMAP V1 and other community supported drivers, persists data differently, writes data to disk in different formats and handles memory resources in different ways. This webinar will go through how to design applications around different storage engines based on your use case and data access patterns. We will be looking into concrete examples of schema design practices that were previously applied on MMAPv1 and whether those practices still apply, to other storage engines like WiredTiger. Topics for review: Schema design patterns and strategies, real-world examples, sizing and resource allocation of infrastructure.
![Why is Schema Design Important ?
• Defines our applications interactions
• How we defined data
• Can considerably impact performance
• Impact DBA’s sleep!
{
first_name: ‘Paul’,
surname: ‘Miller’,
cell: 447557505611,
city: ‘London’,
location: [45.123,47.232],
Profession: [banking, finance, trader],
cars: [
{ model: ‘Bentley’,
year: 1973,
value: 100000, … },
{ model: ‘Rolls Royce’,
year: 1965,
value: 330000, … }
]
}
Fields can contain an
array of sub-documents
Typed field values
Fields can contain arrays](https://image.slidesharecdn.com/norbertowebinarversion-161026165222/85/Webinar-Schema-Patterns-and-Your-Storage-Engine-10-320.jpg)
![db.collection.update(
{a: 1},
{ $set: {
b: 1,
c: {$inc:10},
d: {$push: ["hello"]
})
$set Operator
MMAPv1 / Schema Design Best Practices](https://image.slidesharecdn.com/norbertowebinarversion-161026165222/85/Webinar-Schema-Patterns-and-Your-Storage-Engine-30-320.jpg)
![db.collection.insert(
{
_id: 1,
name: 'Norberto',
parking_tickets: {},
eat_outs:[
null,
null,
null,
null
]
}
)
Document
Pre-allocation
db.collection.update(
{a: 1},
{ $set: {
b: 1,
c: {$inc:10},
d: {$push: ["hello"]
})
$set Operator
MMAPv1 / Schema Design Best Practices](https://image.slidesharecdn.com/norbertowebinarversion-161026165222/85/Webinar-Schema-Patterns-and-Your-Storage-Engine-31-320.jpg)
![db.collection.insert(
{
_id: 1,
name: 'Norberto',
parking_tickets: {},
eat_outs:[
null,
null,
null,
null
]
}
)
Document
Pre-allocation
db.collection.update(
{a: 1},
{ $set: {
b: 1,
c: {$inc:10},
d: {$push: ["hello"]
})
$set Operator
db.movies.update(
{_id: "b872ad6e34"},
{$push:{
reviews: {
$each:
["great", "5*","awful"],
$slice: 10
}
}})
Keep Documents
Small
MMAPv1 / Schema Design Best Practices](https://image.slidesharecdn.com/norbertowebinarversion-161026165222/85/Webinar-Schema-Patterns-and-Your-Storage-Engine-32-320.jpg)
![Document Level
Concurrency
Control
Cache Size Compression
WiredTiger/ Operational Best Practices
Disk
RAM
cacheSizeGB
Heap
OS mongod
Collection
Collection
Collection
Doc1
Doc2
Doc3
Doc4
Doc5
Doc6
DocX
RAM
{
"message": "hello
MongoDB World",
"user": "Norberto",
"channel": "twitter",
"comments": [
{"text":"howdy",
"user":"Ross"}]
}](https://image.slidesharecdn.com/norbertowebinarversion-161026165222/85/Webinar-Schema-Patterns-and-Your-Storage-Engine-43-320.jpg)
![WiredTiger/ Schema Design - Compression
{
"product_id": "b872ad6e34",
"name": "MongoDB Atlas",
"company": "MongoDB",
"description": "MongoDBAAS",
"comments": [
{"text":"Beautiful"}
]
}](https://image.slidesharecdn.com/norbertowebinarversion-161026165222/85/Webinar-Schema-Patterns-and-Your-Storage-Engine-45-320.jpg)
![WiredTiger/ Schema Design - Compression
{
"product_id": "b872ad6e34",
"name": "MongoDB Atlas",
"company": "MongoDB",
"description": "MongoDBAAS",
"comments": [
{"text":"Beautiful"}
]
}
MMAPv1
{
"pid": "b872ad6e34",
"n": "MongoDB Atlas",
"c": "MongoDB",
"d": "MongoDBAAS",
"cs": [
{"t":"Beautiful"}
]
}](https://image.slidesharecdn.com/norbertowebinarversion-161026165222/85/Webinar-Schema-Patterns-and-Your-Storage-Engine-46-320.jpg)
![WiredTiger/ Schema Design - Compression
{
"product_id": "b872ad6e34",
"name": "MongoDB Atlas",
"company": "MongoDB",
"description": "MongoDBAAS",
"comments": [
{"text":"Beautiful"}
]
}
WiredTiger](https://image.slidesharecdn.com/norbertowebinarversion-161026165222/85/Webinar-Schema-Patterns-and-Your-Storage-Engine-47-320.jpg)
Ad
More Related Content
What's hot (20)
Similar to Webinar: Schema Patterns and Your Storage Engine (20)
Ad
More from MongoDB (20)
![MongoDB .local San Francisco 2020: Powering the new age data demands [Infosys]](https://cdn.slidesharecdn.com/ss_thumbnails/315pminfosysfinalsfoversionvocalpart1-200120221508-thumbnail.jpg?width=560&fit=bounds)
Ad
Recently uploaded (20)
Webinar: Schema Patterns and Your Storage Engine
- 1. RSVP: mongodb.com/events What’s coming in 3.4? San Francisco | 11/1
- 3. Schema Patterns and Your Storage Engine
- 4. Agenda Schema Design 101 MMAPv1 Wired Tiger Examples Update Process
- 6. Why do we have so some many options? MMAP V1WT 3rd Party Available 3.2 Your own? In-memoryEncrypted
- 7. Example of Document Definition • App1 - Realtime dashboard - Ad hoc queries - User profiles • App2 - Heavy writes batch process - Analytics workload - Multi-tenant application
- 9. Why is Schema Design Important ?
- 10. Why is Schema Design Important ? • Defines our applications interactions • How we defined data • Can considerably impact performance • Impact DBA’s sleep! { first_name: ‘Paul’, surname: ‘Miller’, cell: 447557505611, city: ‘London’, location: [45.123,47.232], Profession: [banking, finance, trader], cars: [ { model: ‘Bentley’, year: 1973, value: 100000, … }, { model: ‘Rolls Royce’, year: 1965, value: 330000, … } ] } Fields can contain an array of sub-documents Typed field values Fields can contain arrays
- 11. Document Model class Item(Object): def __init__(self, name, car ,date): self.name = name self.car = car self.date = date class Car(Object): def __init__(self, brand, manufactor, date): self.brand = brand self.manufactor = manufactor self.date = date Natural Representation Flexible Schema Aligned with Development { _id: 1, 'greetings':'hello' } { _id: 2, 'k':'greetings', 'v': 'hello' } { _id: 1, 'k':'greetings', 'v': 'hello' } { _id: 2, 'car':{ 'manufactor': 'somecarmaker', 'brand': 'somebrand', 'date': ISODate("2016-06-27") } 'date': ISODate("2016-06-27") }
- 12. What to Consider Change Rate ConcurrencyData Structure Data Lifecycle
- 13. What to Consider Data Structure users accounts reviews products
- 14. What to Consider Data Structure accounts users products reviews
- 15. What to Consider Data Structure users accounts reviews products
- 16. What to Consider Data Structure Client1 Client2 Client3 Client4
- 17. What to Consider { _id: 1, model : "ford" year: 2014, picture: BinData(2, "afGGAF677..."), } { _id: 1, model : "ford" date: {year: 2014, month: 5, day: 1}, picture: "AAFF123BEFC...", } { _id: 1, model : ObjectId("21213312"), date: ISODate("20140501"), picture: "AAFF123BEFC...", } Change Rate
- 18. What to Consider { _id: 1, model : "ford" year: 2014, picture: BinData(2, "afGGAF677..."), } { _id: 1, model : "ford" date: {year: 2014, month: 5, day: 1}, picture: "AAFF123BEFC...", } { _id: 1, model : ObjectId("21213312"), date: ISODate("20140501"), picture: "AAFF123BEFC...", } Different data types Change Rate
- 19. What to Consider { _id: 1, model : "ford" year: 2014, picture: BinData(2, "afGGAF677..."), } { _id: 1, model : "ford" date: {year: 2014, month: 5, day: 1}, picture: "AAFF123BEFC...", } { _id: 1, model : ObjectId("21213312"), date: ISODate("20140501"), picture: "AAFF123BEFC...", } Different field structure Change Rate
- 20. What to Consider { "product_id": "b872ad6e34f87102cf866fead4e10e29", "purchases":{ "2016": { "06": 2823, "05": 5535, ... "total": 1222312 }, "2015": {...}, "2014": {...}, "2013": {...}, } } Data Lifecycle
- 21. What to Consider { "product_id": "b872ad6e34f87102cf866fead4e10e29", "purchases":{ "2016": { "06": 2823, "05": 5535, ... "total": 1222312 }, "2015": {...}, "2014": {...}, "2013": {...}, } } What we actually read and write Data Lifecycle
- 22. What to Consider { "product_id": "b872ad6e34f87102cf866fead4e10e29", "purchases":{ "2016": { "06": 2823, "05": 5535, ... "total": 1222312 }, "2015": {...}, "2014": {...}, "2013": {...}, } } What we don't anymore Data Lifecycle
- 27. MMAPv1
- 28. Disclaimer! WiredTiger is our default Storage Engine - 3.2 onwards
- 29. MMAPv1 / Basics • Data is Mapped into virtual Memory for Fast access • Documents pointers are request per access • If in Memory = fast • If not = Disk seek • Indexes follow the same structure • Allocation based on Database per file
- 30. db.collection.update( {a: 1}, { $set: { b: 1, c: {$inc:10}, d: {$push: ["hello"] }) $set Operator MMAPv1 / Schema Design Best Practices
- 31. db.collection.insert( { _id: 1, name: 'Norberto', parking_tickets: {}, eat_outs:[ null, null, null, null ] } ) Document Pre-allocation db.collection.update( {a: 1}, { $set: { b: 1, c: {$inc:10}, d: {$push: ["hello"] }) $set Operator MMAPv1 / Schema Design Best Practices
- 32. db.collection.insert( { _id: 1, name: 'Norberto', parking_tickets: {}, eat_outs:[ null, null, null, null ] } ) Document Pre-allocation db.collection.update( {a: 1}, { $set: { b: 1, c: {$inc:10}, d: {$push: ["hello"] }) $set Operator db.movies.update( {_id: "b872ad6e34"}, {$push:{ reviews: { $each: ["great", "5*","awful"], $slice: 10 } }}) Keep Documents Small MMAPv1 / Schema Design Best Practices
- 33. Wired Tiger
- 34. WiredTiger / Basics • MVCC Storage Engine • Compression • Document Level Concurrency Control • Better resource allocation WiredTiger Engine Schema & Cursors Python API C API Java API Database Files Transactions Page read/write Logging Column storage Block management Row storage Snapshots Log Files Cache
- 35. WiredTiger/ Schema Design Best Practices
- 36. Cache Size WiredTiger/ Schema Design Operational Best Practices Disk RAM
- 37. Cache Size WiredTiger/ Operational Best Practices Disk RAM cacheSizeGB
- 38. Cache Size WiredTiger/ Operational Best Practices Disk RAM cacheSizeGB Heap OS
- 39. Document Level Concurrency Control Cache Size WiredTiger/ Operational Best Practices Disk RAM cacheSizeGB Heap OS mongod Collection Collection Collection
- 40. Document Level Concurrency Control Cache Size WiredTiger/ Operational Best Practices Disk RAM cacheSizeGB Heap OS mongod Collection Doc1 Doc2 Doc3 Doc4 Doc5 Doc6 DocX
- 41. Document Level Concurrency Control MMAPv1/ Operational Best Practices mongod Collection1 doc1 Collection2 doc2 Collection2 doc3 Collection4 doc4 Collection5 doc5 Collection6 doc6 Collection7 doc7 CollectionX docX
- 42. Document Level Concurrency Control MMAPv1/ Operational Best Practices mongod Collection1 doc1 Collection2 doc2 Collection2 doc3 Collection4 doc4 Collection5 doc5 Collection6 doc6 Collection7 doc7 CollectionX docX
- 43. Document Level Concurrency Control Cache Size Compression WiredTiger/ Operational Best Practices Disk RAM cacheSizeGB Heap OS mongod Collection Collection Collection Doc1 Doc2 Doc3 Doc4 Doc5 Doc6 DocX RAM { "message": "hello MongoDB World", "user": "Norberto", "channel": "twitter", "comments": [ {"text":"howdy", "user":"Ross"}] }
- 44. Document Level Concurrency Control Cache Size Compression WiredTiger/ Operational Best Practices Disk RAM cacheSizeGB Heap OS mongod Collection Collection Collection Doc1 Doc2 Doc3 Doc4 Doc5 Doc6 DocX RAM gzip(0x001231400122 2321)
- 45. WiredTiger/ Schema Design - Compression { "product_id": "b872ad6e34", "name": "MongoDB Atlas", "company": "MongoDB", "description": "MongoDBAAS", "comments": [ {"text":"Beautiful"} ] }
- 46. WiredTiger/ Schema Design - Compression { "product_id": "b872ad6e34", "name": "MongoDB Atlas", "company": "MongoDB", "description": "MongoDBAAS", "comments": [ {"text":"Beautiful"} ] } MMAPv1 { "pid": "b872ad6e34", "n": "MongoDB Atlas", "c": "MongoDB", "d": "MongoDBAAS", "cs": [ {"t":"Beautiful"} ] }
- 47. WiredTiger/ Schema Design - Compression { "product_id": "b872ad6e34", "name": "MongoDB Atlas", "company": "MongoDB", "description": "MongoDBAAS", "comments": [ {"text":"Beautiful"} ] } WiredTiger
- 48. WiredTiger/ Schema Design – Index Prefix Compression { "user": "Norberto", "country": "Portugal", "last_comment": ”European Champions!!!", }
- 49. WiredTiger/ Schema Design – Index Prefix Compression { "user": "Norberto", "country": "Portugal", "last_comment": "Iceland beat England :-)", } db.users.createIndex( {"country": 1} ) Australia-Zimbabwe A-N N-Z N-Q Q-Z… record_id: 0x12311
- 50. WiredTiger/ Schema Design – Index Prefix Compression { "user": "Norberto", "country": "Portugal", "last_comment": "Iceland beat England :-)", } db.users.createIndex( {"country": 1} ) Australia-Zimbabwe A-N N-Z N-Q Q-Z… { key: "Por", record_id: 0x12311 } record_id: 0x12311 { key: "Por", record_id: 0x123BB } { key: "Pol", record_id: 0xF23CB }
- 51. Picking a Storage Engine Matters ?
- 52. In-place Updates Memory { country: "Portugal", star: "Cristiano" } { country: "Iceland", star: "Hannes Magnusson" } { country: "Netherlands", star: "Arthur Viegers" } { country: "Portugal", star: "Crestiano" }
- 53. In-place Updates Memory { country: "Portugal", star: "Cristiano" } { country: "Iceland", star: "Hannes Magnusson" } { country: "Netherlands", star: "Arthur Viegers" } db.teams.update({country: "Portugal"}, {"$set": { "star": "Cristiano Ronaldo" }}) { country: "Portugal", star: "Cristiano Ronaldo" } MMAPv1
- 54. In-place Updates Memory { country: "Portugal", star: "Crestiano" } db.teams.update({country: "Portugal"}, {"$set": { "star": "Cristiano Ronaldo" }}) { country: "Portugal", star: "Cristiano Ronaldo" } WiredTiger version 1 version 2
- 55. In-place Updates Memory { country: "Portugal", star: "Crestiano" } { country: "Portugal", star: "Cristiano Ronaldo" } WiredTiger version 1version 2
- 56. Insert Heavy server db.pool.insert( { "subject": "Euro 2016", "winner": "Portugal" } ) mongod
- 57. Insert Heavy Server 1 mongod db.pool.insert( { "subject": "Euro 2016", "winner": "Portugal" } ) MMAPv1 Server 2 mongod Server 3 mongod Sharding
- 58. Insert Heavy server mongod db.pool.insert( { "subject": "Euro 2016", "winner": "Portugal" } ) mongod mongod MMAPv1 Micro-Sharding
- 60. Insert Heavy server db.pool.insert( { "subject": "Euro 2016", "winner": "Portuaal" } ) mongod WiredTiger CPU availability
- 61. Insert Heavy server mongod db.pool.insert( { "subject": "Euro 2016", "winner": "Portuaal" } ) WiredTiger server mongod server mongod Sharding
- 62. Buckets { "product_id": "b872ad6e34f87...", "visits":{ "store1": { "Jan_31": 2823, "Jan_30": 5535, ... "total": 1222312 }, "store_2": { "Jan_31": 2823, "Jan_28": 5535, ... "total": 1222312 }, } }
- 63. Buckets { "product_id": "b872ad6e34f87...", "visits":{ "store1": { "Jan_31": 2823, "Jan_30": 5535, ... "total": 1222312 }, "store_2": { "Jan_31": 2823, "Jan_28": 5535, ... "total": 1222312 }, } } db.visits.update( {"product_id": "b872ad6e34f87...."}, {"$inc": { "store_2.March_19": 10, "store_1.April_1": 68, "stores.total": 78 } )
- 64. Buckets { "product_id": "b872ad6e34f87...", "visits":{ "store1": { "Jan_31": 2823, "Jan_30": 5535, "April_1": 10, "total": 1222312 }, "store_2": { "Jan_31": 2823, "Jan_28": 5535, "March_19": 68, "total": 1222312 }, } } In-place update MMAPv1
- 65. Buckets { "product_id": "b872ad6e34f87...", "visits":{ "store1": { "Jan_31": 2823, "Jan_30": 5535, ... "total": 1222312 }, "store_2": { "Jan_31": 2823, "Jan_28": 5535, ... "total": 1222312 }, } } New version { "product_id": "b872ad6e34f87...", "visits":{ "store1": { "Jan_31": 2823, "Jan_30": 5610, "April_1": 10, "total": 1222312 }, "store_2": { "Jan_31": 2823, "Jan_28": 5545, "March_19": 68, "total": 12229032 }, } } WiredTiger
- 67. What to Look For Bucketing Read/Write RatioIn-place Update
- 68. Update Process • Apply changes to your schema design • Check if there's any performance regression • Make sure you have CPU resources available • Swap secondary nodes • Swap primary • Enjoy the increased performance!
- 69. Example of Document Definition • App1 - Realtime dashboard - Ad hoc queries - User profiles • App2 - Heavy writes batch process - Analytics workload - Multi-tenant application
- 70. Example of Document Definition • App1 - Realtime dashboard - Ad hoc queries - User profiles • App2 - Heavy writes batch process - Analytics workload - Multi-tenant application WiredTiger or MMAPv1
- 71. Example of Document Definition • App1 - Realtime dashboard - Ad hoc queries - User profiles • App2 - Heavy writes batch process - Analytics workload - Multi-tenant application WiredTiger or MMAPv1 WiredTiger
- 72. Market Size $36 Billion Partners 1,000+ International Offices 15 Global Employees 575+ Downloads Worldwide 15,000,000+ Make a GIANT Impact www.mongodb.com/careers http://grnh.se/pj10su
- 73. https://university.mongodb.com/courses/M310/about M310: MongoDB Security
- 74. Feel free to reach out! Twitter: @nleite [email protected] #mongodbeurope
- 75. Obrigado