Cassandra Performance and Scalability on AWS
61 likes41,837 views
Summary of past Cassandra benchmarks performed by Netflix and description of how Netflix uses Cassandra interspersed with a live demo automated using Jenkins and Jmeter that created two 12 node Cassandra clusters from scratch on AWS, one with regular disks and one with SSDs. Both clusters were scaled up to 24 nodes each during the demo.
![[WhaTap DevOps Day] ์ธ์
5 : ๊ธ์ต Public ํด๋ผ์ฐ๋/ Devops ๊ตฌ์ถ ์ฌ์ ](https://cdn.slidesharecdn.com/ss_thumbnails/whatapdevopsdaydevops-220705054116-6ce35c67-thumbnail.jpg?width=560&fit=bounds)
Ad
More Related Content
What's hot (20)
Viewers also liked (20)
Ad
Similar to Cassandra Performance and Scalability on AWS (20)
Ad
More from Adrian Cockcroft (14)
Recently uploaded (20)
Cassandra Performance and Scalability on AWS
- 1. Cassandra Performance and Scalability on AWS August 8th, 2012 Adrian Cockcroft @adrianco #netflixcloud #cassandra12 http://www.linkedin.com/in/adriancockcroft
- 2. Things we donโt do
- 3. Things we do do. Run benchmarks. Now.
- 4. YOLO
- 5. Screenshots from Live Demo Backup slides from pre-runs of the demo with some updates to show what actually happened
- 6. Asgard cass_perf apps, with no instances running
- 11. Asgard Iinitial set of cass instances up and running
- 12. Back to Presentation While the load gets goingโฆ.
- 14. Scalability from 48 to 288 nodes on AWS http://techblog.netflix.com/2011/11/benchmarking-cassandra-scalability-on.html Client Writes/s by node count โ Replication Factor = 3 1200000 1099837 1000000 800000 Used 288 of m1.xlarge 600000 4 CPU, 15 GB RAM, 8 ECU 537172 Cassandra 0.86 400000 Benchmark config only 366828 existed for about 1hr 200000 174373 0 0 50 100 150 200 250 300 350
- 15. Blah Blah Blah (Iโm skipping all the cloud intro etc. Netflix runs in the cloud, if you hadnโt figured that out already you arenโt paying attention and should go read slideshare.net/netflix)
- 16. โSome people skate to the puck, I skate to where the puck is going to beโ Wayne Gretzky
- 17. Cassandra on AWS The Past The Future โข Instance: m2.4xlarge โข Instance: hi1.4xlarge โข Storage: 2 drives, 1.7TB โข Storage: 2 SSD volumes, 2TB โข CPU: 8 Cores, 26 ECU โข CPU: 8 HT cores, 35 ECU โข RAM: 68GB โข RAM: 64GB โข Network: 1Gbit โข Network: 10Gbit โข IOPS: ~500 โข IOPS: ~100,000 โข Throughput: ~100Mbyte/s โข Throughput: ~1Gbyte/s โข Cost: $1.80/hr โข Cost: $3.10/hr
- 18. Cassandra Disk vs. SSD Benchmark Same Throughput, Lower Latency, Half Cost
- 19. Live Demo Workload โข Jenkins automation โ Jmeter load driver โ Asgard provisioning โ Priam instance management โข Traffic โ Reading/writing whole 100 column rows โ Randomly selected from 25M row keys โ Run for 10minutes, then double ring size
- 20. The Netflix Streaming Service
- 21. Major Front End Services โข Non-member Web Site www.netflix.com โ Marketing driven, sign up flow, SOX/PCI scope โข Member Web Site movies.netflix.com โ Personalization driven โข CDNs for delivering bulk video/audio โ Netflix CDN: openconnect.netflix.com โข API for external and device user interfaces โ Mostly private APIs, public API docs at developer.netflix.com โข API for controlling video playback โ DRM, QoS management, Bookmarks
- 22. Netflix Deployed on AWS 2009 2009 2010 2010 2010 2011 Content Logs Play WWW API CS Content S3 International DRM Sign-Up Metadata Management Terabytes CS lookup EC2 Diagnostics & EMR CDN routing Search Device Config Encoding Actions S3 Movie TV Movie Customer Call Hive & Pig Bookmarks Petabytes Choosing Choosing Log Business Social Logging Ratings CS Analytics Intelligence Facebook CDNs ISPs Terabits Customers
- 23. Cassandra Instance Architecture Linux Base AMI (CentOS) Priam Cassandra Manager Token Java7 Management, Bac kups, Autoscaling Tomcat/Java7 AppDynamics Monitoring appagent monitoring Cassandra 1.09 Log rotation AppDynamics GC and thread machineagent dump logging Etc.
- 24. Priam โ Cassandra Automation Available at http://github.com/netflix โข Netflix Platform Tomcat Code โข Zero touch auto-configuration โข State management for Cassandra JVM โข Token allocation and assignment โข Broken node auto-replacement โข Full and incremental backup to S3 โข Restore sequencing from S3 โข Grow/Shrink Cassandra โringโ
- 25. Astyanax Available at http://github.com/netflix โข Features โ Complete abstraction of connection pool from RPC protocol โ Fluent Style API โ Operation retry with backoff โ Token aware โข Recipes โ Distribute row lock (without zookeeper) โ Multi-DC row lock โ Uniqueness constraint โ Multi-row uniqueness constraint โ Large file storage
- 26. Scale Up Return to live demo to watch new nodes coming online
- 27. Kiklos Clusters growing from 12 to 24 in-service, bootstrapping, garbage-collecting, cass-down โข http://explorers.us-east- 1.dyntest.netflix.net:7001/jr/cassandradashbo ard
- 28. Kiklos Clusters growing from 12 to 24 in-service, bootstrapping, garbage-collecting, cass-down
- 29. Asgard Showed 24 nodes per cluster, but didnโt get a screen shot
- 30. Back to Presentation While jenkins/jmeter collects graphs and shuts down the systems
- 31. Cassandra on AWS A highly available and durable deployment pattern
- 32. High Availability โข Cassandra stores 3 local copies, 1 per zone โ Synchronous access, durable, highly available โ Read/Write One fastest, use for fire and forget โ Read/Write Quorum 2 of 3, use for read-after-write โข AWS Availability Zones โ Separate buildings โ Separate power etc. โ Fairly close together
- 33. โTraditionalโ Cassandra Write Data Flows Single Region, Multiple Availability Zone, Not Token Aware Cassandra โขDisks 2โขZone A 2 4 2 1. Client Writes to any Cassandra 3 3Cassandra If a node goes Cassandra Node โขDisks5 โขDisks 5 offline, hinted handoff 2. Coordinator Node โขZone C 1 โขZone B completes the write replicates to nodes when the node comes Non Token and Zones back up. 3. Nodes return ack to Aware coordinator Clients Requests can choose to 4. Coordinator returns Cassandra Cassandra wait for one node, a ack to client โขDisks โขDisks quorum, or all nodes to 5. Data written to โขZone B โขZone C ack the write internal commit log 3 disk (no more than Cassandra SSTable disk writes and 10 seconds later) โขDisks 5 compactions occur โขZone A asynchronously
- 34. Astyanax - Cassandra Write Data Flows Single Region, Multiple Availability Zone, Token Aware Cassandra โขDisks โขZone A 1. Client Writes to Cassandra 2 2Cassandra If a node goes nodes and Zones โขDisks3 โขDisks 3 offline, hinted handoff 2. Nodes return ack to โขZone C 1 โขZone B completes the write client Token when the node comes 3. Data written to back up. internal commit log Aware disks (no more than Clients Requests can choose to 10 seconds later) Cassandra Cassandra wait for one node, a โขDisks โขDisks quorum, or all nodes to โขZone B โขZone C ack the write 2 Cassandra SSTable disk writes and โขDisks 3 compactions occur โขZone A asynchronously
- 35. Data Flows for Multi-Region Writes Token Aware, Consistency Level = Local Quorum 1. Client writes to local replicas If a node or region goes offline, hinted handoff 2. Local write acks returned to completes the write when the node comes back up. Client which continues when Nightly global compare and repair jobs ensure 2 of 3 local nodes are everything stays consistent. committed 3. Local coordinator writes to remote coordinator. 100+ms latency Cassandra Cassandra 4. When data arrives, remote โข Disks โข Zone A โข Disks โข Zone A coordinator node acks and Cassandra 2 2 Cassandra Cassandra 4Cassandra 6 โข Disks โข Disks 6 3 5โข Disks6 4 Disks6 copies to other remote zones โข Zone C 1 โข Zone B โข Zone C โข โข Zone B 4 5. Remote nodes ack to local US EU coordinator Clients Clients Cassandra 2 Cassandra Cassandra Cassandra 6. Data flushed to internal โข Disks โข Zone B โข Disks โข Zone C 6 โข Disks โข Zone B โข Disks โข Zone C commit log disks (no more Cassandra 5 6Cassandra โข Disks than 10 seconds later) โข Zone A โข Disks โข Zone A
- 36. Extending to Multi-Region Added production UK/Ireland support with no downtime Minimize impact on original cluster using bulk backup move 1. Create cluster in EU Take a Boeing 737 on a domestic flight, upgrade it to a 747 by adding more engines, fuel and bigger wings 2. Backup US cluster to S3 and fly it to Europe without landing it on the wayโฆ 3. Restore backup in EU 4. Local repair EU cluster 5. Global repair/join Cassandra 100+ms latency Cassandra 1 โข Disks โข Disks โข Zone A โข Zone A Cassandra Cassandra Cassandra Cassandra โข Disks โข Disks โข Disks โข Disks โข Zone C โข Zone B โข Zone C โข Zone B US 5 EU Clients Clients Cassandra Cassandra Cassandra Cassandra โข Disks โข Disks โข Disks โข Disks โข Zone B โข Zone C โข Zone B โข Zone C Cassandra Cassandra โข Disks โข Disks โข Zone A 3 โข Zone A 4 2 S3
- 37. Cassandra Backup โข Full Backup Cassandra Cassandra Cassandra โ Time based snapshot โ SSTable compress -> S3 Cassandra Cassandra โข Incremental S3 Backup Cassandra Cassandra โ SSTable write triggers compressed copy to S3 Cassandra Cassandra โข Archive Cassandra Cassandra โ Copy cross region A
- 38. ETL for Cassandra โข Data is de-normalized over many clusters! โข Too many to restore from backups for ETL โข Solution โ read backup files using Hadoop โข Aegisthus โ http://techblog.netflix.com/2012/02/aegisthus-bulk-data-pipeline-out-of.html โ High throughput raw SSTable processing โ Re-normalizes many clusters to a consistent view โ Extract, Transform, then Load into Teradata
- 39. Netflix Open Source Strategy โข Release PaaS Components git-by-git โ Source at github.com/netflix โ we build from itโฆ โ Intros and techniques at techblog.netflix.com โ Blog post or new code every few weeks โข Motivations โ Give back to Apache licensed OSS community โ Motivate, retain, hire top engineers โ โPeer pressureโ code cleanup, external contributions
- 40. Open Source Projects and Posts Legend Github / Techblog Priam Exhibitor Servo and Autoscaling Cassandra as a Service Zookeeper as a Service Scripts Apache Contributions Astyanax Honu Curator Techblog Post Cassandra client for Log4j streaming to Zookeeper Patterns Java Hadoop Coming Soon EVCache CassJMeter Circuit Breaker Memcached as a Cassandra test suite Robust service pattern Service Cassandra Asgard Eureka / Discovery Multi-region EC2 AutoScaleGroup based Service Directory datastore support AWS console Aegisthus Archaius Chaos Monkey Hadoop ETL for Dynamics Properties Robustness verification Cassandra Service
- 41. Chaos Monkey http://techblog.netflix.com/2012/07/chaos-monkey-released-into-wild.html โข Computers (Datacenter or AWS) randomly die โ Fact of life, but too infrequent to test resiliency โข Test to make sure systems are resilient โ Allow any instance to fail without customer impact โข Chaos Monkey hours โ Monday-Friday 9am-3pm random instance kill โข Application configuration option โ Apps now have to opt-out from Chaos Monkey
- 42. Asgard http://techblog.netflix.com/2012/06/asgard-web-based-cloud-management-and.html โข Replacement for AWS Console at Scale โ Groovy/Grails/JVM based โ Supports all AWS regions on a global basis โ Specific to AWS feature set โข Hides the AWS credentials โ Use AWS IAM to issue restricted keys for Asgard โ Each Asgard instance manages one account โ One install each for test, prod, audit accounts
- 43. Roadmap for 2012 โข More resiliency and improved availability โข More automation, orchestration โข โHardeningโ the platform, code clean-up โข Lower latency for web services and devices โข IPv6 โ running now, see techblog for details โข More open sourced components โข Las Vegas in November - AWS Re:Invent
- 44. Back to Live Demo
- 45. Disclaimers โข We didnโt have time to tune the demo โข These are the plots from the live demo run โข Runโs need to be longer to get to steady state โข Data size only reached around 5GB per node โข Plenty of โI wonder why it did thatโ remains โข Itโs a fair comparison, but not the best absolute performance possible for this workload and configuration โข When you remove the IO bottleneck, the next few bottlenecks appearโฆ
- 46. Activity during the talk 10:30-11:30 Custom AppDynamics dashboard showing CPU and IOPS per node
- 47. Jmeter Plots โข Plots are the output of the Jenkins build โข Each instance has its own set of plots โข Each availability zone has its own summary plots โข One of the three zone summary plots is compared for each metric โข Plot collection is currently duplicated as we are transitioning from โEpicโ to โAtlasโ
- 48. Jenkins Collected results and graphs after job has completed
- 49. The past m2.4xlarge Instances per zone The future hi1.4xlarge
- 50. The past m2.4xlarge Transactions per zone, same as total client transactions The future hi1.4xlarge
- 52. The past m2.4xlarge Thousands of Microseconds The future hi1.4xlarge
- 56. Takeaway Netflix has built and deployed a scalable global platform based on Cassandra and AWS. Key components of the Netflix PaaS are being released as Open Source projects so you can build your own custom PaaS. If you like lots of SSDโs come and work for usโฆ. http://github.com/Netflix http://techblog.netflix.com http://slideshare.net/Netflix http://www.linkedin.com/in/adriancockcroft @adrianco #netflixcloud #cassandra12
Editor's Notes
- #26: Complete connection pool abstractionQueries and mutations wrapped in objects created by the Keyspace implementation making it possible to retry failed operations.ย This varies from other connection pool implementations on which the operation is created on a specific connection and must be completely redone if it fails.Simplified serialization via method overloading.ย The low level thrift library only understands data that is serialized to a byte array.ย Hector requires serializers to be specified for nearly every call.ย Astyanax minimizes the places where serializers are specified by using predefined ColumnFamiliy and ColumnPath definitions which specify the serializers.ย The API also overloads set and get operation for common data types.The internal library does not log anything.ย All internal events are instead ... calls to a ConnectionPoolMonitor interface.ย This allows customization of log levels and filtering of repeating events outside of the scope of the connection poolSuper columns will soon be replaced by Composite column names. As such it is recommended to not use super columns at all and to use Composite column names instead. There is some support for super columns in Astyanax but those methods have been deprecated and will eventually be removed.