Capacity Planning For Your Growing MongoDB Cluster
Download as PPTX, PDF8 likes3,493 views
This document discusses capacity planning for deploying MongoDB. It defines capacity planning as planning for requirements like availability, throughput, and responsiveness by determining necessary resources like CPU, memory, storage, and network capacity. It emphasizes starting capacity planning before launch to avoid downtime. Key aspects of capacity planning for MongoDB include estimating working memory set size, storage I/O needs based on data size and access patterns, using tools like IOStat and MongoDB Management Service for monitoring and automation, and conducting iterative testing and deployments. Failure occurs if planned resources cannot meet requirements.
Ad
More Related Content
What's hot (20)
Viewers also liked (16)
Ad
Similar to Capacity Planning For Your Growing MongoDB Cluster (20)
Ad
More from MongoDB (20)
![MongoDB .local San Francisco 2020: Powering the new age data demands [Infosys]](https://cdn.slidesharecdn.com/ss_thumbnails/315pminfosysfinalsfoversionvocalpart1-200120221508-thumbnail.jpg?width=560&fit=bounds)
Recently uploaded (20)
Capacity Planning For Your Growing MongoDB Cluster
- 1. Solution Architect, MongoDB Sam Weaver Capacity Planning: Deploying MongoDB #mongodb
- 2. Capacity Planning • Why is it important? • What is it? • When is it important? • How is it actually done?
- 3. Why?
- 4. Prepping for launch • You’ve written your application • The code is good • You’re looking to launch soon • How do I deploy?
- 5. Questions to ask yourself • Instance types – Standalone? – Replica set? – Sharded? • Architecture • Size of machines – Machines cost money – Size of machines may affect instance types required
- 6. • What are the consequences of not planning? Why does it matter?
- 7. Why • Once we launch, we don't want to have avoidable down time due to poorly selected HW • As our success grows we want to stay in front of the demand curve • We want to meet business' and users' expectations • We want to keep our jobs
- 8. What?
- 9. What is Capacity Planning? Requirement s Resources
- 10. Requirements • Availability – Planning for a crash – Planning for binary upgrades – Planning for hardware maintenance • Throughput – X many users at any one time – Bulk loads vs. random access • Responsiveness – SLAof x ms per page load – Amazon, Google study
- 11. How?
- 12. CPU • Non-indexed Data • Sorting • Aggregation – Map/Reduce – Framework • Data – Fields – Nesting – Arrays/Embedded-Docs
- 13. Network • Latency – WriteConcern – ReadPreference – Batching • Throughput – Update/Write Patterns – Reads/Queries
- 14. Understand memory usage for MongoDB • Data & indexes memory mapped into virtual address space • Data accessed is paged into RAM • OS evicts least recently used page • More frequently used pages stay in RAM
- 15. Identify your working set Number of active users on the system at any one time Number of distinct pages accessed per second =
- 16. Working Set
- 17. Working Set 4 distinct pages per second RAM Disk
- 18. Working Set 4 distinct pages per second
- 19. Working Set 4 distinct pages per second Worst case 4 disk accesses
- 20. Working Set 6 distinct pages per second
- 21. Working Set 6 distinct pages per second
- 22. Working Set 6 distinct pages per second
- 23. Working Set 6 distinct pages per second Worst case disk access on every op
- 25. Memory • Working set affected by –Sorting –Aggregation –Connections SORTS Connections Aggregations
- 26. Working Set Estimator "workingSet" : { "note" : "thisIsAnEstimate", "pagesInMemory" : <num>, "computationTimeMicros" : <num>, "overSeconds" : num } Number of unique pages the server needed in the last 15 minutes. Use this to see if you are growing out RAM
- 27. Storage • Different storage have different IOPs – Spinning disk • 7,500k SATA 75-100 IOPs – SSD • 9,000-120,000 IOPs – EBS • 100 IOPs – Provisioned EBS • 2,000 IOPs • Work out how much data you need to write per time frame. • MongoDB writes to a journal and datafiles flush to disk. • Replication adds oplog considerations
- 28. Using this information • Plan hardware to hold the working set + indexes • Allow room to grow • If working set is larger than RAM and you can’t reasonably add more resources, then shard – Don’t shard too early – Lots of little instances vs. a few big instances • Think about architecture – Local disk or central storage – Don’t be surprised with x copies of data with x number of nodes
- 29. Development to production • Don’t be surprised by: – More data = more/larger indexes – Indexes make your working set bigger • Replication adds a network overhead • Journal has different access patterns
- 30. What tools are there to help me?
- 31. IOStat
- 32. MongoStat
- 33. MongoPerf • Measure amount of data written to device per second
- 34. MongoDB Management Service • Free Cloud or On-Premise based management tool – Monitoring – Automation – Backup
- 36. Scaling for capacity – MMS automation
- 37. When?
- 38. Capacity Planning: When • When? – Before it's too late! – Iterative process Start Launch Version 2
- 39. Repeat (continuously) • Repeat Testing • Repeat Evaluations • Repeat Deployment
- 40. What is failure? • We have failed at Capacity Planning when our resources don’t meet our requirements • Because our requirements can have many dimensions, we may exceed our requirements in one characteristic but not meet them in another • This means that we can spend many $$$ and still fail!
- 41. Models • Load/Users – Response Time/TTFB • System Performance – Peak Usage – Min Usage
- 42. Starter Questions • What is the working set? – How does that equate to memory – How much disk access will that require • How efficient are the queries? • What is the rate of data change? • How big are the highs and lows?
- 43. Questions?
- 44. Solution Architect, MongoDB Sam Weaver Thank You #mongodb