Sizing Your MongoDB Cluster
Download as pptx, pdf6 likes2,734 views
Jay Runkel presented a methodology for sizing MongoDB clusters to meet the requirements of an application. The key steps are: 1) Analyze data size and index size, 2) Estimate the working set based on frequently accessed data, 3) Use a simplified model to estimate IOPS and adjust for real-world factors, 4) Calculate the number of shards needed based on storage, memory and IOPS requirements. He demonstrated this process for an application that collects mobile events, requiring a cluster that can store over 200 billion documents with 50,000 IOPS.
Sizing Your MongoDB Cluster
- 1. FEBRUARY 15, 2018 | BELL HARBOR #MDBlocal Sizing MongoDB Clusters
- 2. #MDBlocal Master Solutions Architect Jay Runkel MongoDB @jayrunkel
- 3. #MDBlocal MongoDB: 4 years NoSQL: 9 years Solution Architect: engineer assigned to sales Significant experience sizing MongoDB clusters About me
- 5. #MDBlocal Do I need to shard? What size servers should I use? What will my monthly Atlas/AWS/Azure/Google costs be? When will I need to add a new shard or upgrade my servers? How much data can my servers support? How many queries can my servers support? Will we be able to meet our query latency requirements? Sizing
- 6. #MDBlocal • Large coffee chain: PlanetDollar • Collect mobile app performance • Every tap, click, gesture will generate an event • 2 Year History • Perform analytics • Historical • Near real-time (executive dashboards) • Support usage • 3000 – 5000 events per second Your boss comes to you… I need a budget for the monthly Atlas costs?
- 7. #MDBlocal • Build a prototype • Run performance tests using actual data and queries on hardware with specs similar to production servers The only accurate way to size a cluster • EVERY OTHER APPROACH IS A GUESS • Including the one I am presenting today
- 8. #MDBlocal • Early in project, but • Need to order hardware • Estimate costs to determine “Go/No Go” decision • Schema design • Compare the hardware requirements for different schemas Sometimes, it is necessary to guess ☹
- 9. #MDBlocal MongoDB Clusters Look Like This Config Config Config Application Driver Primary Secondary Secondary
- 10. #MDBlocal • # of shards • Specifications of each server • CPU • Storage • Size • Performance: IOPS • Memory • Network Our Solution Will Consist Of
- 11. #MDBlocal • Sizing Objective ✓ • IOPS, Query Processing, Working Set • Sizing Methodology with an Example Agenda
- 13. #MDBlocal • # of shards • Specifications of each server • CPU • Storage • Size • Performance: IOPS • Memory • Network Our Solution Will Consist Of
- 14. #MDBlocal • IOPS – input output units per second • Throughput • Random access • Most workloads “randomly” access documents IOPS collection
- 15. #MDBlocal Storage Performance Type IOPS 7200 rpm SATA ~ 75 – 100 15000 rpm SAS ~ 175 – 210 RAID-10 (24 x 7200 RPM SAS) 2000 Amazon EBS 250 – 500 Amazon EBS Provisioned IOPS 10000 - 20000 SSD 50000 Flash Storage 100K – 400K (or more) http://en.wikipedia.org/wiki/IOPS
- 16. #MDBlocal • How many IOPS do we need? • Want the real answer, run a test • How to estimate? Hardest Part of Sizing is IOPS
- 17. #MDBlocal Processing a Query Select Index Load relevant index entries from disk Identify documents using index Retrieve documents from disk Filter documents Return Documents
- 18. #MDBlocal Processing a Query IO Select Index Load relevant index entries from disk Identify documents using index Retrieve documents from disk Filter documents Return Documents
- 19. #MDBlocal But MongoDB Has a Cache File System indexes collections CPU Memory indexes documents Disk access is only necessary if indexes or documents are not in cache Select Index Load relevant index entries from disk Identify documents using index Retrieve documents from disk Filter documents Return Documents
- 20. #MDBlocal Working Set Working Set = indexes plus frequently accessed documents If RAM greater than working set then reduced IO Select Index Load relevant index entries from disk Identify documents using index Retrieve documents from disk Filter documents Return Documents File System indexes collections CPU Memory indexes documents
- 21. #MDBlocal This is all great, but how do we estimate IOPS?
- 22. #MDBlocal Assume • Working Set < RAM < Data Size • Memory contains indexes only MongoDB Simplified Model File System indexes collections CPU Memory indexes documents
- 23. #MDBlocal Assume appropriate indexes To resolve find: • Navigate in-memory indexes • Retrieve document from disk 1 IOP per document returned Find Queries With Simplified Model File System indexes collections CPU Memory indexes documents
- 24. #MDBlocal Assume appropriate indexes To resolve find: • Navigate in-memory indexes • Retrieve document from disk 1 IOP per document returned Find Queries With Simplified Model File System indexes collections CPU Memory indexes documents
- 25. #MDBlocal To resolve insert: • Write document to disk • Update each index file IOPS = 1 + # of indexes Inserts With Simplified Model File System indexes collections CPU Memory indexes documents
- 26. #MDBlocal To resolve delete: • Navigate in-memory indexes • Mark document deleted • Update each index file IOPS = 1 + # of indexes Deletes With Simplified Model File System indexes collections CPU Memory indexes documents
- 27. #MDBlocal To resolve delete: • Navigate in-memory indexes • Mark document deleted • Insert new document version • Update each index file IOPS = 2 + # of indexes Updates With Simplified Model File System indexes collections CPU Memory indexes documents
- 28. #MDBlocal • Working Set • Checkpoints • Document size relative to block size • Indexed Arrays • Journal, Log The Simplified Model is too simplistic
- 29. #MDBlocal • WiredTiger write process: 1. Update document in RAM (cache) 2. Write to journal (disk) 3. Periodically, write dirty documents to disk (checkpoint) • 60 seconds or 2 GB (whichever comes first) Checkpoints Checkpoint 1 Checkpoint 2 Checkpoint 3 B C A A C A 3 writes 3 documents written 3 writes 2 documents written
- 30. #MDBlocal • Estimate total requirements (using simplified model): • RAM • CPU • Disk Space • IOPS • Adjust based upon working set, checkpoints, etc. • Design (sharded) cluster that provides these totals How are we going to get there?
- 32. #MDBlocal Methodology Application Requirements Cluster Sizing • Number of shards • Server specs Magic Happens
- 33. #MDBlocal 1.Collection Size 2.Working Set 3.Queries -> IOPS 4.Adjust based upon working set, checkpoints, etc. 5.Calculate # of shards 6.Review, iterate, repeat Methodology (cont.) Build a spread sheet Multiple iterations may be required
- 34. #MDBlocal 1.Assumptions 1.Data Size 1.Working Set • Index Size • Frequently Accessed Documents 1.Queries – IOPS 1.Shard Calculations Sizing Spreadsheet
- 35. #MDBlocal 1.Assumptions 1.Data Size 1.Working Set • Index Size • Frequently Accessed Documents 2. Queries – IOPS 1. Shard Calculations Sizing Spreadsheet
- 38. #MDBlocal Sizing Spreadsheet 1.Assumptions 1.Data Size 1.Working Set • Index Size • Frequently Accessed Documents 2. Queries – IOPS 1. Shard Calculations
- 39. #MDBlocal • # of documents • Data size • Index size • WT compression Collection Analysis
- 40. #MDBlocal Calculate The Number of Documents Application Description # of Documents in Collection There will be 20M documents in the collection by the end of 2017 20,000,000 We expect to insert 10K documents per day with 1 year retention period 365*10,000 = 3,655,000 We have 3000 devices each producing 1 event per minute and we need to keep a 90 day history 3000 * 60 * 24 * 90 = 388,800,000
- 41. #MDBlocal Calculate The Number of Documents Application Description # of Documents in Collection There will be 20M documents in the collection by the end of 2017 20,000,000 We expect to insert 10K documents per day with 1 year retention period 365*10,000 = 3,655,000 We have 3000 devices each producing 1 event per minute and we need to keep a 90 day history 3000 * 60 * 24 * 90 = 388,800,000 PlanetDollar: 2 year history. Each day 5000 inserts per second for 5 hours and 3000 inserts per second for 19 hours 2*365*(5000*5*3600 + 3000*19*3600) = 215,496,000,000
- 42. #MDBlocal • Data Size = # of documents * Average document size • This information is available in db.stats(), Compass, Ops Manager, Cloud Manager, Atlas, etc. Calculate the Data Size
- 43. #MDBlocal • Write some code • Programmatically generate a large data set • 5-10% of expected size • Measure • Collection size • Index size • Compression What if there aren’t any documents?
- 44. #MDBlocal • Use db.collection.stats() • Take data size, index size and extrapolate to production size • Calculate compression ratio db.collection.stats() { count: 10000 size: 70,388,956 avgObjSize: 7038 storageSize: 25341952 … totalIndexSize: 147456 } Determine collection and data size Parameter Formula Value # of documents 2.5B avgObjSize 7038 Collection Size =2.5B * 7038 1.760E13 Bytes WT Compression = 25341952/70388956 .36 Collection Storage =2.5B * 7038 * .36 6.33E12 Bytes Index Size Per Doc = 147456 / 10000 15 Bytes Collection Index Size =2.5B * 15 /1024^3 35 GB
- 45. #MDBlocal PlanetDollar - Collection Size
- 46. #MDBlocal Sizing Spreadsheet 1.Assumptions 1.Data Size 1.Working Set • Index Size • Frequently Accessed Documents 1.Queries – IOPS 1.Shard Calculations
- 47. #MDBlocal WorkSet = Indexes plus the set of documents accessed frequently We know the index size from previous analysis Estimate the frequently accessed documents Given the queries What are the frequently accessed docs? Working Set File System collections indexes CPU Memory indexes documents
- 48. #MDBlocal Query Analysis Dashboards – last minute of data Customer support - last hour of data Reports (run once per day) inspect last years worth of data Active Documents = 1 hours worth of data PlanetDollar Working Set
- 49. #MDBlocal PlanetDollar - Working Set
- 50. #MDBlocal 1.Assumptions 1.Data Size 1.Working Set • Index Size • Frequently Accessed Documents 1.Queries – IOPS 1.Shard Calculations Sizing Spreadsheet
- 51. #MDBlocal + # of documents returned per second + # of documents updated per second + # of indexes impacted by each update + # of inserts per second + # of indexes impacted by each insert + # of deletes per second (x2) + # of indexes impacted by each delete - Multiple updates occurring within checkpoint - % of find query results in cache Total IOPS IOPS Calculation
- 52. #MDBlocal • 5000 inserts per second • 5000 deletes per second Dashboards (aggregations: 100 per minute) • Total events per minute across all users (current minute) • Total events per minute per region (current minute) • Total events per store per minute (current minute) Debugging Tool (ad hoc – 5 per second) • Find all events for a user in last 60 minutes (100 events returned, on average) PlanetDollar Queries
- 53. #MDBlocal • Each insert: • Update collection • Update each index (3 indexes) • Each Delete: • Update collection • Update each index (3 indexes) • 5000 inserts/sec • 5000 deletes/sec IOPS for inserts and deletes 4 IOPS 5 IOPS (4 * 5000) + (5 * 5000) = 45000 IOPS
- 54. #MDBlocal • Example: Total events per minute across all users (current minute) • How many documents will be read from disk? IOPS for PlanetDollar Aggregations 05000 per second * 60 seconds = 300,000 Most data in cache Some IOPS will likely be require
- 55. #MDBlocal • Find all events for a user in last 60 minutes • 5 per second • 100 documents per query • # IOPS = 5 * 100 = 500 IOPS IOPS For Find
- 57. #MDBlocal • CPU utilized for: • Compress/decompress • Encrypt/Decrypt • Aggregation queries • General query processing • In most cases, RAM requirements → large servers → many cores • Possible exception: aggregation queries • One core per query • # cores >> # of simultaneous aggregation queries How Many CPUs Do I Need?
- 58. #MDBlocal 1.Assumptions 1.Data Size 1.Working Set • Index Size • Frequently Accessed Documents 1.Queries – IOPS 1.Shard Calculations Sizing Spreadsheet
- 59. #MDBlocal • At this point you have: 1. Required storage capacity 2. Working Set Size 3. IOPS Estimate 4. Some idea about class of server (or VM) the customer plans to deploy • Determine number of required shards Shard Calculations
- 60. #MDBlocal • Sum of disk space across shards greater than required storage size Disk Space: How Many Shards Do I Need? Example Data Size = 9 TB WiredTiger Compression Ratio: .33 Storage size = 3 TB Req Disk: 6 TB Server disk capacity = 2 TB 3 Shards Required Recommend providing 2X the compressed data size in disk
- 61. #MDBlocal RAM: How Many Shards Do I Need? Example Working Set = 428 GB Server RAM = 128 GB 428/128 = 3.34 4 Shards Required
- 62. #MDBlocal IOPS: How Many Shards Do I Need? Example Require: 50K IOPS AWS Instance: 20K IOPS 3 Shards Required
- 63. #MDBlocal PlanetDollar - Shard Calculations
- 64. #MDBlocal 1. Calculate: • Collection size • Index size 2. Estimate Working Set 3. Use simplified model to estimate IOPS 4. Revise (working set coverage, checkpoints, etc.) 5. Calculate shards Sizing Summary