Dynomite: A Highly Available, Distributed and Scalable Dynamo Layer--Ioannis Papapanagiotou, Shailesh Birari, Jason Cacciatore, Netflix
6 likes3,919 views
Dynomite is a thin, distributed dynamo layer for different storage engines and protocols. Currently at Netflix, we are focusing on using Redis as the storage engine. Dynomite supports multi-datacenter replication and is designed for high availability. In the age of high scalability and big data, Dynomite’s design goal is to turn single-server datastore solutions into peer-to-peer, linearly scalable, clustered systems while still preserving the native client/server protocols of the datastores, e.g., Redis protocol. In this talk, we are going to present Dynomite recent features, and the Dyno client. Both projects are open source and available to the community.
Ad
More Related Content
What's hot (20)
Viewers also liked (20)
Ad
Similar to Dynomite: A Highly Available, Distributed and Scalable Dynamo Layer--Ioannis Papapanagiotou, Shailesh Birari, Jason Cacciatore, Netflix (20)
Ad
More from Redis Labs (20)
Recently uploaded (20)
Dynomite: A Highly Available, Distributed and Scalable Dynamo Layer--Ioannis Papapanagiotou, Shailesh Birari, Jason Cacciatore, Netflix
- 1. Cloud Database Engineering Making Non-Distributed Databases, Distributed - Shailesh Birari - Ioannis Papapanagiotou, PhD
- 2. Dynomite Ecosystem ● Dynomite ● Dynomite-manager ● Dyno client
- 3. Cloud Database Engg (CDE) Team ● Develop and operate data stores in AWS - Cassandra, Dynomite, Elastic Search, RDS, S3 ● Ensure availability, scalability, durability and latency SLAs ● Database expertise, client libraries, tools and best practices
- 4. ● Cassandra not a speed demon (reads) ● Needed a data store: o Scalable & highly available o High throughput, low latency o Active-active multi datacenter replication ● Usage of Redis increasing: o Netflix use case is active-active, highly available o Does not have bi-directional replication o Cannot withstand a Monkey attack Problems & Observations
- 5. What is Dynomite? ● A framework that makes non-distributed data stores, distributed. o Can be used with many key-value storage engines like Redis, Memcached, LMDB, etc. o Focus: performance, cross-datacenter active-active replication and high availability o Features: node warmup (cold bootstrapping), tunable consistency, S3 backups/restores
- 6. Dynomite @ Netflix ● Running around 1.5 years in PROD ● ~1000 customer facing nodes ● 1M OPS at peak ● Largest cluster: 6TB ● Quarterly upgrades in PROD
- 7. Dynomite Overview ● Layer on top of a non-distributed key value data store ○ Peer-peer, Shared Nothing ○ Auto Sharding ○ Multi-datacenter ○ Linear scale ○ Replication(Encrypted) ○ Gossiping
- 8. ● Each rack contains one copy of data, partitioned across multiple nodes in that rack ● Multiple Racks == Higher Availability (HA) Topology
- 9. Replication ● A client can connect to any node on the Dynomite cluster when sending requests. o If node owns the data, ▪ data are written in local data-store and asynchronously replicated. o If node does not own the data ▪ node acts as a coordinator and sends the data in the same rack & replicates to other nodes in other racks and DC.
- 10. The Dynomite Ecosystem RESP = Redis Serialization Protocol
- 11. Consistency ● DC_ONE o Reads and writes are propagated synchronously only to the node in local rack and asynchronously replicated to other racks and data centers ● DC_QUORUM o Reads and writes are propagated synchronously to quorum number of nodes in the local region and asynchronously to the rest ● Consistency can be configured dynamically for read or write operations separately (cluster-wide)
- 12. Performance Setup ● Instance Type: ○ Dynomite: r3.2xlarge (1Gbps) ○ Pappy/Dyno: m2.2xls (typical of an app@Netflix) ● Replication factor: 3 ○ Deployed Dynomite in 3 zones in us-east-1 ○ Every zone had the same number of servers ● Demo app used simple workloads key/value pairs ○ Redis: GET and SET ● Payload ○ Size: 1024 Bytes ○ 80%/20% reads over writes
- 13. Performance (Dynomite Speed) ● Throughput scales linearly with number of nodes. ● Dynomite can reach >1Million Client requests with ~24 nodes.
- 14. Performance (Latency - average/P50) ● Dynomite’s latency on average is 0.16ms. ● Client side latency is 0.6ms and does not increase as the cluster scales up/down
- 15. Performance (Latency - P99) ● The major contributor to latency at P99 is the network. ● Dynomite affects <10%
- 16. Dynomite-manager ● Token management for multi-region deployments ● Support AWS environment ● Automated security group update in multi-region environment ● Monitoring of Dynomite and the underlying storage engine ● Node cold bootstrap (warm up) ● S3 backups and restores ● REST API
- 17. Dynomite-manager: warm up 1. Dynomite-manager identifies which node has the same token in the same DC 2. Sets Redis to “Slave” mode of that node 3. Checks for peer syncing a. difference between master and slave offset 4. Once master and slave are in sync, Dynomite is set to allow write only 5. Dynomite is set back to normal state 6. Checks for health of the node - Done!
- 18. Warm up (node terminated)
- 20. Warm up (node with same token)
- 21. Warm up (Redis replication)
- 22. Warm up (Streaming data)
- 23. Warm up (Nodes in sync)
- 24. Dynomite: S3 backups/restores ● Why? o Disaster recovery o Data corruption ● How? o Redis dumps data on the instance drive o Dynomite-manager sends data to S3 buckets ● Data per node are not large so no need for incrementals. ● Use case: o clusters that use Dynomite as a storage layer o Not enabled in clusters that have short TTL or use Dynomite as a cache
- 25. Dynomite S3 backups (operation) 1. Perform backup a. Dynomite-manager performs it on a pre-defined interval b. Dynomite-manger REST call: i. curl http://localhost:8080/REST/v1/admin/s3backup 2. Perform a Redis BGREWRITEAOF or BGSAVE. a. Check the size of the persisted file. If the size is zero, which means that there was an issue with Redis or no data are there, then we do not perform S3 backups 3. S3 backup key: backup/region/clustername-ASG/token/date
- 26. Dynomite S3 restores 1. Perform restore: a. Dynomite-manager performs once it starts if configuration is enabled b. Dynomite-manger REST call: i. curl http://localhost:8080/REST/v1/admin/s3backup 2. Stop Dynomite process: a. We perform this to notify Discovery that Dynomite is not accessible b. Stop Redis process 3. Restore the data from a specific date a. provided in the configuration 4. Start Redis process and check if the data has been loaded. 5. Start Dynomite and check if process is up
- 27. Dyno Client - Java API ● Connection Pooling ● Load Balancing ● Effective failover ● Pipelining ● Scatter/Gather ● Metrics, e.g. Netflix Insights
- 28. Dyno Load Balancing ● Dyno client employs token aware load balancing. ● Dyno client is aware of the cluster topology of Dynomite within the region, can write to specific node using consistent hashing.
- 29. Dyno Failover ● Dyno will route requests to different racks in failure scenarios.
- 30. Roadmap ● Multi-threaded support for Dynomite ● Data reconciliation & repair v2 ● Dynomite-spark connector ● Investigation for persistent stores ● Async Dyno Client ● Others….
- 31. More information ● Netflix OSS: o https://github.com/Netflix/dynomite o https://github.com/Netflix/dyno ● Contact: o email: {sbirari, ipapapanagiotou}@netflix.com