Redis Day TLV 2018 - 10 Reasons why Redis should be your Primary Database
3 likes598 views
Redis should be the primary database for three key reasons: 1. It is extremely fast and scales linearly even as more nodes are added. Performance remains highly consistent across scaling tests. 2. It is highly available with techniques like quorum-based replication across nodes rather than shards, in-memory replication for faster performance, and watchdogs to monitor the cluster. 3. It uses CRDTs to provide strong eventual consistency for active-active replication across replicas in under 1 millisecond, solving the conflicts that arise much faster than other approaches.
Ad
More Related Content
What's hot (20)
Similar to Redis Day TLV 2018 - 10 Reasons why Redis should be your Primary Database (20)
Ad
More from Redis Labs (20)
Ad
Recently uploaded (20)
Redis Day TLV 2018 - 10 Reasons why Redis should be your Primary Database
- 1. 10 Reasons why Redis should be your Primary Database YIFTACH SHOOLMAN CO-FOUNDER AND CTO @ REDIS LABS
- 2. The Database Market 2 NOSQL 35.0% CAGR 2016-21 RELATIONAL 7.5% CAGR 2016-21 Source: 451 Research Total Data Market Monitor
- 4. Docker Hub: The World’s Most Popular Database # of containers launched as of Feb 2018 630M+ (1.87M/day, 78K/hr, 1.28K/min) 308M+ 263M+ 24M+ 4
- 5. Redis Enterprise 5 DBaaS • Available since mid 2013 • 8,100+ enterprise customers Software • Available since early 2015 • 300+ enterprise customers 550K+ databases managed worldwide • 6 of top Fortune 10 companies • 3 of top 5 communications companies Customers • 3 of top 4 credit card issuers • 3 of top 5 healthcare companies
- 6. 6 It’s Fast (Extremely Fast) and Scales Linearly 1
- 7. Redis Enterprise Cluster Node 1 Node 2 Node N (odd number) 7 Uneven number of symmetric nodes
- 8. Redis Enterprise Cluster Node 1 Node 2 Node N (odd number) 8 Single master database M
- 9. Redis Enterprise Cluster Node 1 Node 2 Node N (odd number) 9 An HA database SM
- 10. Redis Enterprise Cluster Node 1 Node 2 Node N (odd number) 10 A Clustered Database M1 M2 M3
- 11. Redis Enterprise Cluster Node 1 Node 2 Node N (odd number) 11 A Clustered Database M1 M2 M3S3 S1 S2
- 12. Redis Enterprise Node Cluster Manager Enterprise Layer Open Source Layer REST API Zero latency proxy Redis Shards
- 13. Redis Enterprise: Shared Nothing Symmetric Architecture Node 1 Node 2 Node N (odd number) Redis Shards & Proxies Data Path 13 Data-Path and Control/Management Path Separation
- 14. Redis Enterprise: Shared Nothing Symmetric Architecture Cluster Management Path Node Watchdog Cluster Watchdog Node 1 Node 2 Node N (odd number) Redis Shards & Proxies Data Path 14 Data-Path and Control/Management Path Separation
- 17. 17 Seamless Resharding Proxy 1 Bring trimmed slaves S1 S2 ½ of the dataset ½ of the dataset M S
- 18. 18 Seamless Resharding Proxy 2 Start draining S1 S2 Until no outstanding requests (~msec) M S
- 19. 19 Seamless Resharding Proxy 3 MàM1, S->M2 & stop draining S1 S2 M1 M2
- 20. 20 Seamless Resharding Proxy 4 Trim M1, M2 S1 S2 M1 M2
- 21. 21 Scaling Out / In Scaling out, Resharding & Rebalancing Scale out Rebalancing Resharding
- 22. x2 faster Application 1 2 3 4 5 6 7 1 2 3 4 5 6 7 Multiplexing & pipelining With Proxy Proxy How Does The Proxy Work ?
- 23. Single-proxy – dense policy Multi-proxy – sparse policy Scale-Out Proxy Single Database Endpoint Single Database Endpoint
- 24. Scaling Linearly with OSS Cluster API Node 1 Proxy 1 Shard 1 Shard 2 Shard n/m Node 2 Proxy 2 Shard n/m+1 Shard n/2mShard n/m+2 Node M Proxy M Shard n-m+1 Shard nShard n-m+2 M nodes Redis Enterprise Cluster with N shards database Client 1 Client 2 Client k HS ranges = {ranges #1, ranges #2, ranges #n/m} HS ranges = {ranges #n/m+1, ranges #n/m+2, ranges #n/2m} HS ranges = {ranges #n-m+1, ranges #n-m+2, ranges #n}
- 25. 6 nodes, 120 shards cluster 12 nodes, 240 shards cluster 18 nodes, 360 shards cluster True Linear Scalability
- 26. 6 nodes, 120 shards cluster 12 nodes, 240 shards cluster 18 nodes, 360 shards cluster True Linear Scalability
- 27. 6 nodes, 120 shards cluster 12 nodes, 240 shards cluster 18 nodes, 360 shards cluster Sub-millisecond latency is maintained across all the tests True Linear Scalability
- 29. 29 Lessons learned m4.large a quorum node with no-data Redis Enterprise Node 3 • 5+ years in production • 550K+ database created • 50+ data-centers/zones • 2000+ node failure events • 100+ complete data-center outages
- 30. 30 HA Concept #1– Quorum by Nodes, not by Shards 3 replicas Redis 90GB r4.4xlarge m4.large a quorum node with no-data Redis Enterprise 90GB r4.4xlarge 90GB r4.4xlarge 90GB r4.4xlarge 90GB r4.4xlarge M1 S1 S2 Node 1 Node 2 Node 3 Node 1 Node 2 Node 3 M1 S1
- 31. 31 HA Concept #1 – Quorum by Nodes, not by Shards 3 replicas Redis 90GB r4.4xlarge m4.large a quorum node with no-data Redis Enterprise 90GB r4.4xlarge 90GB r4.4xlarge 90GB r4.4xlarge 90GB r4.4xlarge M1 S1 S2 Node 1 Node 2 Node 3 Node 1 Node 2 Node 3 M1 S1
- 32. 32 HA Concept #1 – Quorum by Nodes, not by Shards 3 replicas Redis 90GB r4.4xlarge m4.large a quorum node with no-data Redis Enterprise 90GB r4.4xlarge 90GB r4.4xlarge 90GB r4.4xlarge 90GB r4.4xlarge M1 S1 S2 Node 1 Node 2 Node 3 Node 1 Node 2 Node 3 M1 S1 • ~30% infrastructure cost savings • Less network traffic • Easy to manage
- 33. 33 HA Concept #2 – Pure In-Memory Replication Disk-based Replication (OSS default) M S OSS Diskless Replication M S Pure In-Memory Replication M S 1 2 3 1 2 1
- 34. 34 HA Concept #2 – Pure In-Memory Replication Disk-based Replication (OSS default) M S OSS Diskless Replication M S Pure In-Memory Replication M S 1 2 3 1 2 1
- 35. 35 HA Concept #2 – Pure In-Memory Replication Disk-based Replication (OSS default) M S OSS Diskless Replication M S Pure In-Memory Replication M S 1 2 3 1 2 1 x2 faster
- 36. 36 HA Concept #3 – Watchdogs are Part of the Cluster
- 37. 37 HA Concept #3 – Watchdogs are Part of the Cluster
- 38. HA Concept #4 – How to deploy a Multi-AZ/Rack Cluster 1. At least 3 AZs/Racks 2. Distance between Azs/Racks < 10msec 3. Master and Slave of the same shard must be deployed on different AZs/Rack 4. For every i,j,k: #_of_node ( AZi +AZj) > #_of_node AZk
- 39. Redis Failover Benchmark %oftimesdatawaslost Averagetimetorecoverinseconds Redis Enterprise AWS ElastiCache Heroku Redis Compose (IBM) Redis Azure Redis Cache <5sec
- 41. Data Loss SSD - persistent and ephemeral data Failed Instance SSD - persistent and ephemeral data New Empty Instance Data-Persistence - The Wrong Way Uses Network Attached Persistent Storage, not Ephemeral Discuss SQL Strategy
- 42. Data Loss SSD - persistent and ephemeral data Failed Instance SSD - persistent and ephemeral data New Empty Instance Data-Persistence - The Wrong Way No Data Loss SSD - ephemeral data Failed Instance SSD - ephemeral data New Populated Instance Persistent Storage AOF, Snapshot Data-Persistence - The Right Way AOF, Snapshot Dataload Uses Network Attached Persistent Storage not Ephemeral Discuss SQL Strategy
- 43. Tunable Data Persistence Configuration Non-Replicated M
- 44. Tunable Data Persistence Configuration Tuned for Speed Data-Persistence at the slave M S Non-Replicated M
- 45. Tunable Data Persistence Configuration Tuned for Speed Data-Persistence at the slave M S Tuned for Reliability Data-Persistence at the master & slave M S Non-Replicated M AOF-every-sec, AOF-every-write, Snapshot (RDB)
- 46. • Redis performance during AOF rewrite • Data-persistence when multiple Redis instances reside on the same node Two Main Challenges with Redis Data-Persistence
- 47. Redis Enterprise Enhanced Storage Engine Redis Labs Proprietary and Confidential Information
- 51. 51 Redis-ML RediSearch ReJSON Redis-Timeseries Redis-Graph Rebloom Custom ? Rate Limiter Multi-Model – based on Open Core with Modules
- 52. 52 And It’s still Fast (extremely fast) with Modules RedisSearch – x5
- 53. 53 And It’s still Fast (extremely fast) with Modules ReBloom – x17RedisSearch – x5
- 54. 54 msec msec RedisSearch – x5 Redis-ML – x2000 And It’s still Fast (extremely fast) with Modules ReBloom – x17
- 55. 55 msec msec Redis-ML – x2000 And It’s still Fast (extremely fast) with Modules Redis-Graph Wait for Redisconf: Pier 27, San-Francisco April 26-88 RedisSearch – x5 ReBloom – x17
- 56. 56 It Uses a Different Approach for Active-Active 5
- 57. App Eventual Consistency à 100msec Active-Active: Existing Approaches are just Slow App App
- 58. App Active-Active: Existing Approaches are just Slow App App App App App Eventual Consistency à 100msec Strong Consistency à 200msec
- 59. We Need Something Faster than the Speed of Light Light > 20msec RTT Network > 70msec RTT Redis < 1msec RTT
- 60. Conflict Resolution is Hard • Application level solution à too complex to write • LWW (Last Write Wins) à doesn’t work for many of the Redis use cases, e.g.: • Counters • Sets • Sorted Sets • Lists • Modules’ new datatypes
- 61. CRDT • Years of academic research • Based on consensus free protocol • Strong eventual consistency • Built to resolve conflicts with complex data types
- 62. The CRDT Approach App App App Strong Eventual Consistency à 1 msec
- 63. Solving Conflicts – Counters c = 500 Replica A c = 500 Replica B c = 500 Replica C 63
- 64. Solving Conflicts – Counters c = 500 INCRBY 200 Replica A c = 500 Replica B c = 500 Replica C 64
- 65. Solving Conflicts – Counters c = 500 INCRBY 200 Replica A c = 500 DECRBY 300 Replica B c = 500 Replica C 65
- 66. Solving Conflicts – Counters c = 500 INCRBY 200 Replica A c = 500 DECRBY 300 Replica B c = 500 INCRBY 1000 Replica C 66
- 67. Convergence Function (commutative): 500 + ∑c(i) = 500 +200 -300 +1000 = 1400 Solving Conflicts – Counters c = 500 INCRBY 200 Replica A c = 500 DECRBY 300 Replica B c = 500 INCRBY 1000 Replica C 67
- 68. Solving Conflicts – Sets S = {A, B, C} Replica A S = {A, B, C} Replica B S = {A, B, C} Replica C 68
- 69. Solving Conflicts – Sets S = {A, B, C} SADD D Replica A S = {A, B, C} Replica B S = {A, B, C} Replica C 69
- 70. Solving Conflicts – Sets S = {A, B, C} SADD D Replica A S = {A, B, C} SADD A Replica B S = {A, B, C} Replica C 70
- 71. Solving Conflicts – Sets S = {A, B, C} SADD D Replica A S = {A, B, C} SADD A Replica B S = {A, B, C} SREM A Replica C 71
- 72. Convergence Function (associative): • S = S + D + A - A = {A, B, C, D} • Observed Removed + Add Wins Solving Conflicts – Sets S = {A, B, C} SADD D Replica A S = {A, B, C} SADD A Replica B S = {A, B, C} SREM A Replica C 72
- 74. This is NOT Causal Consistency Replica A S = {A} Replica B Replica C 74 SADD A SADD A
- 75. This is NOT Causal Consistency Replica A S = {A} Replica B Replica C 75 SADD A S = {A, B} SADD B SADD A S = {B, A}
- 76. This is Casual Consistency Replica A S = {A} Replica B Replica C 76 SADD A S = {A, B} SADD B SADD A S = {A, B}
- 77. Active-Active: Comparison App App App App App App Eventual Consistency Strong Consistency App App App Strong Eventual Consistency + Causal Consistency 100msec 200msec <1msec
- 78. 78 It Saves You $$ 6
- 79. Multi-Tenant from Day One • Single tenant multiple shards/DBs
- 80. • Multi-tenant multiple shards/ DBs • Customer B • Customer A • Customer N Multi-Tenant from Day One • Single tenant multiple shards/DBs • OR
- 81. #1 200GB #2 200GB #50 200GB 50 x r3.8xlarge instances #51 200GB #51 200GB #100 200GB 1st replica for HA #101 200GB #102 200GB #150 200GB 2nd replica for quorum Total cost (reserved instances) = $2,132,250/yr 10TB Deployment on AWS with 2 Replicas
- 82. #1 200GB #2 200GB #50 200GB 50 x r3.8xlarge instances #51 200GB #51 200GB #100 200GB 1 replica for HA #101 15GB Total cost (reserved instances) = $1,421,500/yr Savings = $710,750/yr 1 quorum server 10TB Deployment on AWS with 1 Replica + a Quorum Server
- 83. Redis on Flash – Built for a Tiered Memory Architecture Persistent Storage: Entire Dataset AOF, Snapshot SSD: Cold Values DRAM: Keys & Hot Values Cluster Node 83
- 84. RoF - Designed for the New Persistent Memory Technology 84 NVMe vs. SATA
- 85. RoF - Designed for the New Persistent Memory Technology 85 Optane (3DXP) vs. NVMe
- 86. $2,200,162/yr $1,772,606/yr $766,096/yr $232,875/yr $0/yr $500,000/yr $1,000,000/yr $1,500,000/yr $2,000,000/yr $2,500,000/yr Other Redis Provider RCP RAM Cloud-Based NoSQL RCP Flash <1 msec <10 msec <1 msec DBaaS Price Comparison 2TB Dataset with HA @ 100k ops/sec (on-demand pricing) Up to 89% savings! 86 Other Redis Provider < 1msec Redis Enterprise VPC < 1msec Cloud-Based NoSQL < 10 msec RoF < 1 msec
- 87. RoF by Numbers • GA zero touch – 11/2017 • Quite a few customers • All of them are using it as a primary data-store • Database size 0.5TB à 10TB+
- 90. All Verticals Financial Services AdvertisingMedia Technology Communications EducationGaming Banks E-commerce Business Services Social Travel 90
- 91. Device Devices • Raspberry Pi support • A single OSS Redis instance (3MB footprint) • Persistent • Redis Streams Client
- 92. Edge Devices • Raspberry Pi support • A single OSS Redis instance (3MB footprint) • Persistent • Redis Streams Client • RPi/x86 nodes (4 cores/2GB RAM /50GB SSD) • RedisEnterprise Cluster • Redis Streams Server & Client • Persistent & HA • Redison Flash • Modules: ‒ Search, JSON, Graph, Time-Series, ReBloom, ML Edge
- 93. Everywhere Devices • Raspberry Pi support • A single OSS Redis instance (3MB footprint) • Persistent • Redis Streams Client • RPi/x86 nodes (4 cores/2GB RAM /50GB SSD) • RedisEnterprise Cluster • Persistent & HA • Redison Flash • Redis Streams Server & Client • Modules: ‒ Search, JSON, Graph, Time-Series, ReBloom, ML Edge • Large RedisEnterpriseCluster(s) • Multi-cloud/DBaaS/Self-managed • Multi-master geo-replication • Redis Streams Server • Persistent & HA (multi-az) • Redison Flash • Modules: ‒ Search, JSON, Graph, Time-Series, ReBloom, ML, Cloud
- 94. 94 It Simplifies Data Services 8
- 95. Cloud Data-Services Architecture AWS Data-Services Flow
- 96. Cloud Data-Services Architecture AWS Data-Services Flow 4 1 2 5 3
- 97. The WRONG Spaghetti Architecture AWS Data-Services Flow =
- 98. With the RIGHT Data-Services Architecture or Stream API Streams Data- Structure Multi-Function 1 2 3
- 99. 99 Because most of you have already been using it as a primary data-store 9
- 100. Redis Enterprise Survey Data Use Cases YES 67% NO 33% Primary Database Move to Redis Enterprise
- 101. 101 You have some responsibility 10
- 102. One Way Ticket to the Cloud…
- 103. One Way Ticket to the Cloud…
- 104. What can you do? ?
- 105. Make the Tech World Better
- 106. Thank you!