High performance Infrastructure Oct 2013
0 likes728 views
The document discusses high performance infrastructure for Server Density which includes 150 servers that have been running since June 2009 and migrated from MySQL to MongoDB. It stores 25TB of data per month. Key aspects of performance discussed are using fast networks like 10 Gigabit Ethernet on AWS, ensuring high memory, using SSDs over spinning disks for performance, and factors like replication lag based on location. The document also compares options like using cloud, dedicated servers, or colocation and discusses monitoring, backups, dealing with outages, and other operational aspects.
Ad
More Related Content
Viewers also liked (20)
Similar to High performance Infrastructure Oct 2013 (20)
Ad
Recently uploaded (20)
Ad
High performance Infrastructure Oct 2013
- 5. Server Density Infrastructure •150 servers
- 6. Server Density Infrastructure •150 servers • June 2009 - 4yrs
- 7. Server Density Infrastructure •150 servers • June 2009 - 4yrs •MySQL -> MongoDB
- 8. Server Density Infrastructure •150 servers • June 2009 - 4yrs •MySQL -> MongoDB •25TB data per month
- 9. Performance • Fast network Picture is unrelated! Mmm, ice cream.
- 10. Performance • Fast network EC2 10 Gigabit Ethernet - Cluster Compute - High Memory Cluster - Cluster GPU - High I/O - High Storage - Network cards - VLAN separation
- 11. Performance • Fast network Workload: Read/Write? What is being stored? Result set size - Read / write: adds to replication oplog - Images? Web pages? Tiny documents? - What is being returned? Optimised to return certain fields?
- 12. Performance • Fast network Use Network Throughput Normal 0-100Mb/s Replication (Initial Sync) Burst +100Mb/s Replication (Oplog) 0-100Mb/s Backup Initial Sync + Oplog
- 13. Performance • Fast network Inter-DC LAN - Latency
- 14. Performance • Fast network Inter-DC LAN Cross USA Washington, DC - San Jose, CA
- 15. Performance • Fast network Location Ping RTT Latency Within USA 40-80ms Trans-Atlantic 100ms Trans-Pacific 150ms Europe - Japan 300ms Ping - low overhead Important for replication
- 17. Failover •Replication •Master/slave - One master accepts all writes - Many slaves staying up to date with master - Can read from slaves
- 18. Failover •Replication •Master/slave •Min 3 nodes Minimum of 3 nodes to form a majority in case one goes down. All store data. Odd number otherwise != majority Arbiter
- 19. Failover •Replication •Master/slave •Min 3 nodes •Automatic failover Drivers handle automatic failover. First query after a failure will fail which will trigger a reconnect. Need to handle retries
- 20. Performance •Replication lag Location Within USA 40-80ms Trans-Atlantic 100ms Trans-Pacific 150ms Europe - Japan - Replication lag Ping RTT Latency 300ms
- 21. Replication Lag 1. Reads: eventual consistency
- 22. Replication Lag 1. Reads: eventual consistency 2. Failover: slave behind
- 23. Eventual Consistency Stale data Not what the user submitted?
- 24. Eventual Consistency Stale data Inconsistent data Doesn’t reflect the truth
- 25. Eventual Consistency Stale data Inconsistent data Changing data Could change on every page refresh
- 26. Eventual Consistency Use Case Needs consistency? Graphs No User profile Yes Statistics Depends Alert config Yes Statistics - depends on when they’re updated
- 27. Replication Lag 1. Reads: eventual consistency 2. Failover: slave behind
- 28. Slave behind Failover: out of date master Old data Rollback
- 29. MongoDB WriteConcern • Safe by default >>> from pymongo import MongoClient >>> connection = MongoClient(w=int/str) Value 0 1 2 3 wtimeout - wait for write before raising an exception Meaning Unsafe Primary Primary + x1 secondary Primary + x2 secondaries
- 31. Performance • Fast network •More RAM Picture is unrelated! Mmm, ice cream.
- 32. http://www.slideshare.net/jrosoff/mongodb-on-ec2-and-ebs No 32 bit No High CPU RAM RAM RAM.
- 35. Performance More RAM = expensive x2 4GB RAM 12 month Prices
- 39. Performance SSD vs Spinning SSDs are better at buffered disk reads, sequential input and random i/o.
- 40. Performance SSD vs Spinning However, CPU usage for SSDs is higher. This may be a driver issue so worth testing your own hardware. Tests done using Bonnie.
- 41. Cloud?
- 44. •Elastic workloads •Demand spikes •Unknown requirements Cloud?
- 45. Dedicated?
- 49. Colo?
- 52. •Hardware spec/value •Total cost •Internal skills? Colo?
- 53. •Hardware spec/value •Total cost •Internal skills? •More fun?! Colo?
- 55. Colo experiment • Build master (buildbot): VM x2 CPU 2.0Ghz, 2GB RAM – $89/m • Build slave (buildbot): VM x1 CPU 2.0Ghz, 1GB RAM – $40/m • Staging load balancer: VM x1 CPU 2.0Ghz, 1GB RAM – $40/m • Staging server 1: VM x2 CPU 2.0Ghz, 8GB RAM – $165/m • Staging server 2: VM x1 CPU 2.0Ghz, 2GB RAM – $50/m • Puppet master: VM x2 CPU 2.0Ghz, 2GB RAM – $89/m Total: $473/m
- 56. Colo experiment •Dell 1U R415 •x2 8C AMD 2.8Ghz •32GB RAM
- 57. Colo experiment •Dell 1U R415 •x2 8C AMD 2.8Ghz •32GB RAM •Dual PSU, NIC
- 58. Colo experiment •Dell 1U R415 •x2 8C AMD 2.8Ghz •32GB RAM •Dual PSU, NIC •x4 1TB SATA hot swappable
- 60. Colo: Networking •10-50Mbps: £20-25/Mbps/m •51-100Mbps: £15/Mbps/m •100+Mbps: £13/Mbps/m
- 61. Colo: Metro •100Mbps: £300/m •1000Mbps: £750/m
- 62. •£300-350/kWh/m •4.5A = £520/m •9A = £900/m Colo: Power
- 64. Tips: rand() •Field names -Field names take up space
- 65. Tips: rand() •Field names •Covered indexes - Get everything from the index
- 66. Tips: rand() •Field names •Covered indexes •Collections / databases - Dropping collections faster than remove() - Split use cases across databases to avoid locking - Put databases onto different disks / types e.g. SSDs
- 67. Backups What is the use case?
- 68. Backups What is the use case? Fixing user errors? Point in time restore? Disaster recovery?
- 69. Backups •Disaster recovery Offsite - What kind of disaster? - Store backups offsite
- 70. Backups •Disaster recovery Offsite Age How log do you keep the backups for? How far do they go back? How recent are they?
- 71. Backups •Disaster recovery Offsite Age Restore time Latency issue - further away geographically, slower the transfer time Partition backups to get critical data restored first
- 72. david@asriel ~: scp david@stelmaria:~/local/local.11 . local.11 100% 2047MB 6.8MB/s 05:01 Restore time - Needed to resync a database server across the US - Take too long; oplog not large enough - Fast internal network but slow internet
- 74. Backups Frequency Consistency Verification - How often? - Backing up cluster at the same time - data moving around - Can the backups be restored?
- 75. Monitoring •System Disk i/o Disk use www.flickr.com/photos/daddo83/3406962115/ Disk i/o % util Disk space usage
- 76. david@pan ~: df -a Filesystem /dev/sda1 proc none none none binfmt_misc david@pan ~: df -ah Filesystem /dev/sda1 proc none none none 1K-blocks Used Available Use% Mounted on 156882796 148489776 423964 100% / 0 0 0 - /proc 0 0 0 - /dev/pts 2097260 0 2097260 0% /dev/shm 0 0 0 - /proc/sys/fs/ Size 150G 0 0 2.1G 0 - Needed to upgrade a machine - Resize = downtime - Resyncing finished just in time Used Avail Use% Mounted on 142G 415M 100% / 0 0 - /proc 0 0 - /dev/pts 0 2.1G 0% /dev/shm 0 0 - /proc/sys/fs/binfmt_
- 77. Monitoring •System Disk i/o Disk use Swap www.flickr.com/photos/daddo83/3406962115/ Disk i/o % util Disk space usage
- 79. Monitoring tools Run yourself Ganglia So Server Density is the tool my company produces but if you don’t like it, want to run your own tools locally or just want to try some others, then that’s fine.
- 81. Dealing with humans On-call - Sharing out the responsibility Determining level of response: 24/7 real monitoring or first responder 24/7 real monitoring for HA environments, real people at a screen at all times First responder: people at the end of a phone
- 82. Dealing with humans On-call 1) Ops engineer - During working hours our dedicated ops engineers take the first level - Avoids interrupting product engineers for initial fire fighting
- 83. Dealing with humans On-call 1) Ops engineer 2) All engineers - Out of hours we rotate every engineer, product and ops - Rotation every 7 days on a Tuesday
- 84. Dealing with humans On-call 1) Ops engineer 2) All engineers 3) Ops engineer - Always have a secondary - This is always an ops engineer - Thinking is if the issue needs to be escalated then it’s likely a bigger problem that needs additional systems expertise
- 85. Dealing with humans On-call 1) Ops engineer 2) All engineers 3) Ops engineer 4) Others - Support from design / frontend engineering - Have to press a button to get them involved
- 86. Dealing with humans Off-call - Responders to an incident get next 24 hours off-call - Social issues to deal with
- 87. Dealing with humans On-call CEO - I receive push notifications + e-mails for all outages
- 88. Dealing with humans Uptime reporting - Weekly internal report on G+ - Gives visibility to entire company about any incidents - Allows us to discuss incidents to get to that 100% uptime
- 89. Dealing with humans Social issues - How quickly can you get to a computer? Are they out drinking on a Friday? What happens if someone is ill? What if there’s a sudden emergency: accident? family emergency? Do they have enough phone battery? Can you hear the ringtone?
- 90. Dealing with humans Backup responder - Backup responder Time out the initial responder Escalate difficult problems Essentially human redundancy: phone provider, geographic area, internet connectivity
- 91. Dealing with outages Expected - Outages are going to happen, especially at the beginning - Costs money for redundancy - How you deal with them
- 92. Dealing with outages Communication Externally - Telling people what is happening - Frequently - Dependent on audience - we can go into more detail because our customers are techies - Github do a good job of providing incident writeups but don’t provide a good idea of what is happening right now - Generally Amazon and Heroku are good and go into more detail
- 93. Dealing with outages Communication Internally - Open Skype conferences between the responders - Usually mostly silence or the sound of the keyboard, but simulates being in the situation room - Faster than typing
- 94. Dealing with outages Really test your vendors - Shows up flaws in vendor support processes Frustrating when waiting on someone else You want as much information as possible Major outage? Everyone will be calling them
- 95. Dealing with outages Simulations - Try and avoid unncessary problems - Do servers come back up from boot? - Can hot spares handle the load? - Test failover: databases, HA firewalls - Regularly reboot servers - Wargames can happen at another stage: startups are usually too focused on building things first