Redis High availability and fault tolerance in a multitenant environment
5 likes1,342 views
This document discusses running Redis in a multi-tenant environment. It covers the architecture of using Redis with high availability, including a master-slave setup with load balancing and failover. It also discusses security and isolation techniques when using Redis for multiple customers, such as access control lists and SSL. Finally, it emphasizes the importance of monitoring all aspects of the Redis infrastructure and environment.
![[RedisConf17] Redis Cluster Operability with Kubernetes and OpenShift - Cedri...](https://cdn.slidesharecdn.com/ss_thumbnails/pzx15q7yqbq1uhmnhhdq-signature-510700f4f8879cacf4f5f87f26eb24c4c17070ea94a689c4570aba97b248b1d2-poli-170628134113-thumbnail.jpg?width=560&fit=bounds)
Ad
More Related Content
What's hot (20)
Viewers also liked (18)
Ad
Similar to Redis High availability and fault tolerance in a multitenant environment (20)
Ad
More from Iccha Sethi (8)
Recently uploaded (20)
Redis High availability and fault tolerance in a multitenant environment
- 1. REDIS IN A MULTI-TENANT ENVIRONMENT Iccha Sethi, Software Developer Twitter: @IcchaSethi
- 2. 2 IcchaSethi.__dict__ { “Name”: “Iccha Sethi”, “Occupation”: “Software Developer”, “Team”: “Redis”, “Company”: “Object Rocket, Rackspace”, “Fun Fact”: “loading….” }
- 3. Why does High Availability matter for Redis? 3
- 4. Why does High Availability matter for Redis? 4
- 5. Outline • Architecture • Security and Isolation • Know thy environment 5
- 6. Outline • Architecture • Security and Isolation • Know thy environment 6
- 8. Outline 8 Redis Master Access HA Redis: Dns: read/write port 6379 password
- 9. Outline 9 Redis Master Redis Slave Access HA Redis: DNS: read/write port 6379 password
- 10. Outline 10 VIP LB1 (Master) HAProxy Redis Master Redis Slave Access HA Redis: DNS: read/write port 6379 password
- 11. Outline 11 VIP LB1 (Master) HAProxy LB2 (Backup) HAProxy keepalived Redis Master Redis Slave Access HA Redis: DNS: read/write port 6379 password
- 12. 12 Sentinels
- 13. 13 Highly Available REDIS STASH NOVA Proxy API Sentinels Host 0 Host 1 Host 2 Host 3 Redis Master Redis Slave KeepAliveD HA Proxy KeepAliveD HA Proxy Proxy Agent Proxy Agent Agent Agent Customer :6379 HA Create
- 15. Opportunities • Failover is easy • Easy to introduce changes • Reusable architecture 15
- 16. Opportunities • Failover is easy • Easy to introduce changes • Reusable architecture 16
- 17. 17 Fail Over – Redis Nodes STASH NOVA Proxy API Sentinels Host 0 Host 1 Host 2 Host 3 Redis Master Redis Slave KeepAliveD HA Proxy KeepAliveD HA Proxy Proxy Agent Proxy Agent Agent Agent Customer :6379
- 18. 18 Fail Over – Redis Nodes STASH NOVA Proxy API Sentinels Host 0 Host 1 Host 2 Host 3 Redis Master Redis Slave KeepAliveD HA Proxy KeepAliveD HA Proxy Proxy Agent Proxy Agent Agent Agent Customer :6379
- 19. 19 Fail Over – Redis Nodes STASH NOVA Proxy API Sentinels Host 0 Host 1 Host 2 Host 3 Redis Master Redis Master KeepAliveD HA Proxy KeepAliveD HA Proxy Proxy Agent Proxy Agent Agent Agent Customer :6379
- 20. 20 Fail Over – Redis Nodes STASH NOVA Proxy API Sentinels Host 0 Host 1 Host 2 Host 3 Redis Master Redis Master KeepAliveD HA Proxy KeepAliveD HA Proxy Proxy Agent Proxy Agent Agent Agent Customer :6379
- 21. 21 Fail Over – Redis Nodes STASH NOVA Proxy API Sentinels Host 0 Host 1 Host 2 Host 3 Redis Master Redis Master KeepAliveD HA Proxy KeepAliveD HA Proxy Proxy Agent Proxy Agent Agent Agent Customer :6379
- 22. 22 Fail Over – Redis Nodes STASH NOVA Proxy API Sentinels Host 0 Host 1 Host 2 Host 3 Redis Master Redis Master KeepAliveD HA Proxy KeepAliveD HA Proxy Proxy Agent Proxy Agent Agent Agent Customer :6379
- 23. 23 Fail Over – Redis Nodes STASH NOVA Proxy API Sentinels Host 0 Host 1 Host 2 Host 3 Redis Master Redis Master KeepAliveD HA Proxy KeepAliveD HA Proxy Proxy Agent Proxy Agent Agent Agent Customer :6379
- 24. Opportunities • Failover is easy • Easy to introduce changes • Reusable architecture 24
- 25. 25 Redis Resize – Vertical Scaling STASH Sentinels Redis Master Redis Slave HA Proxy HA Proxy Customer :6379
- 26. 26 Redis Resize – Vertical Scaling STASH Sentinels Redis Master Redis Slave HA Proxy HA Proxy Customer :6379 New Redis Master New Redis Slave
- 27. 27 Redis Resize – Vertical Scaling STASH Sentinels Redis Master Redis Slave HA Proxy HA Proxy Customer :6379 New Redis Master New Redis Slave
- 28. 28 Redis Resize – Vertical Scaling STASH Sentinels HA Proxy HA Proxy Customer :6379 New Redis Master New Redis Slave
- 29. Opportunities • Failover is easy • Easy to introduce changes • Reusable architecture 29
- 32. Challenges • Global sentinels • Redis drivers 32
- 34. 34 Sentinels
- 36. 36 Architecture - Case Study
- 37. Tale of ‘Resize Why you no work?’ 37
- 38. 38 Redis Resize – Vertical Scaling STASH Sentinels Redis Master Redis Slave HA Proxy HA Proxy Customer :6379 New Redis Master New Redis Slave
- 39. Outline • Architecture • Security and Isolation • Know your environment 39
- 41. ACLs 41
- 42. 42 VIP LB1 (Master) HAProxy ACLS LB2 (Backup) HAProxy ACLs keepalived Redis Master Redis Slave Access HA Redis: VIP: read/write port 6379 ACLs
- 43. SSL 43
- 44. 44 VIP LB1 (Master) HAProxy ACLs SSL certs LB2 (Backup) HAProxy ACLs SSL certs keepalived Redis Master Redis Slave Access HA Redis: VIP: read/write port 6379 SSL port 6380 SSL
- 45. Containers 45
- 46. 46 VIP LB1 (Master) HAProxy LB2 (Backup) HAProxy keepalived Redis Master Redis Slave Access HA Redis: VIP: read/write port 6379 Containers
- 47. 47 Security and Isolation – Case Study
- 48. 48 VIP LB1 (Master) HAProxy LB2 (Backup) HAProxy keepalived Redis Master Redis Slave Access HA Redis: VIP: read/write port 6379 Tale of the Customer with 120K connections
- 49. Outline • Architecture • Security and Isolation • Know thy environment 49
- 50. 50
- 51. Monitoring, Monitoring, Monitoring • What do you monitor? • How often is it monitored? • Who does the monitoring? • What action is taken? 51
- 52. Monitoring, Monitoring, Monitoring • What do you monitor? – Front door check – Connection limit – Memory usage – Failover – Failed failover – Redis process down – No valid slaves • How often is it monitored? • Who does the monitoring? • What action is taken? 52
- 53. Monitoring, Monitoring, Monitoring • What do you monitor? • How often is it monitored? – Front door check – Memory usage – Failover – Failed failover – Redis process down – No valid slaves • Who does the monitoring? • What action is taken? 53
- 54. Monitoring, Monitoring, Monitoring • What do you monitor? • How often is it monitored? • Who does the monitoring? • What action is taken? 54
- 55. 55
- 56. 56
- 57. Monitoring, Monitoring, Monitoring • What do you monitoring? • How often is it monitored? • Who does the monitoring? • What action is taken? 57
- 58. 58
- 59. Monitoring, Monitoring, Monitoring • What do you monitoring? • How often is it monitored? • Who does the monitoring? • What action is taken? – Front door check – Memory usage – Failover – Failed failover – Redis process down – No valid slaves 59
- 60. 60 Know Thy Environment - Challenges
- 61. 61
- 62. 62 Know Thy Environment – Case Studies
- 63. Tale of the customer saved by Zendesk 63
- 64. Outline • Architecture • Security and Isolation • Know thy environment 64
- 65. 65
- 66. 66