Secure Redis Cluster At Box: Vova Galchenko, Ravitej Sistla
0 likes362 views
This document presents an overview of how Box uses Redis clusters and caches data. It discusses security requirements for data stores at Box, including authentication and encryption. It then introduces Secure Redis Proxy, a solution developed by Box to add authentication and encryption to Redis clusters while minimizing performance and operational impacts. Secure Redis Proxy classifies Redis commands and determines whether to authenticate, encrypt, or pass commands through without modification based on the type of command. It also supports password rotation for credentials.
![PRESENTED BY
Redis Cluster at Box: Cache Case Study
Cache relationships between objects with the following requirements:
• For each relationship differentiate cache values by relationship facet
• Be able to clear all facets of a single relationship quickly
field value
Key: [“folder”, “folder_id”, “D”, “files”]](https://image.slidesharecdn.com/day1alameda02vovagalchenkoravitejsistla-190508043319/85/Secure-Redis-Cluster-At-Box-Vova-Galchenko-Ravitej-Sistla-10-320.jpg)
![PRESENTED BY
Redis Cluster at Box: Cache Case Study
Cache relationships between objects with the following requirements:
• For each relationship differentiate cache values by relationship facet
• Be able to clear all facets of a single relationship quickly
field value
{“where”: ”none”, “orderby”: [“file_name”, “ASC”]} A, B, C
Key: [“folder”, “folder_id”, “D”, “files”]](https://image.slidesharecdn.com/day1alameda02vovagalchenkoravitejsistla-190508043319/85/Secure-Redis-Cluster-At-Box-Vova-Galchenko-Ravitej-Sistla-11-320.jpg)
![PRESENTED BY
Redis Cluster at Box: Cache Case Study
Cache relationships between objects with the following requirements:
• For each relationship differentiate cache values by relationship facet
• Be able to clear all facets of a single relationship quickly
field value
{“where”: ”none”, “orderby”: [“file_name”, “ASC”]} A, B, C
{“where”: “none”, “orderby”: [“mod_date”, “ASC”]} C, B, A
Key: [“folder”, “folder_id”, “D”, “files”]](https://image.slidesharecdn.com/day1alameda02vovagalchenkoravitejsistla-190508043319/85/Secure-Redis-Cluster-At-Box-Vova-Galchenko-Ravitej-Sistla-12-320.jpg)
![PRESENTED BY
Redis Cluster at Box: Cache Case Study
Cache relationships between objects with the following requirements:
• For each relationship differentiate cache values by relationship facet
• Be able to clear all facets of a single relationship quickly
field value
{“where”: ”none”, “orderby”: [“file_name”, “ASC”]} A, B, C
{“where”: “none”, “orderby”: [“mod_date”, “ASC”]} C, B, A
{“where”: “size > 100”, “orderby”: [“size”, “ASC”]} C, B
Key: [“folder”, “folder_id”, ”D”, “files”]](https://image.slidesharecdn.com/day1alameda02vovagalchenkoravitejsistla-190508043319/85/Secure-Redis-Cluster-At-Box-Vova-Galchenko-Ravitej-Sistla-13-320.jpg)
Ad
More Related Content
What's hot (20)
Similar to Secure Redis Cluster At Box: Vova Galchenko, Ravitej Sistla (20)
Ad
More from Redis Labs (20)
Ad
Recently uploaded (20)
Secure Redis Cluster At Box: Vova Galchenko, Ravitej Sistla
- 1. PRESENTED BY Secure Redis Cluster at Box Vova Galchenko & Ravitej Sistla Box Inc., Database and Cache Infrastructure Team
- 2. PRESENTED BY 1 Redis Cluster at Box What is Box and how we use Redis 2 Security features within Redis Cluster What’s available in Redis Cluster out of the box 3 Introducing Secure Redis Proxy How we secure our Redis Clusters Agenda: 4 Secure Redis Proxy in practice Operational characteristics, open issues and learnings
- 3. PRESENTED BY
- 5. PRESENTED BY Database & Cache Infra Team Provide highly available, consistent, performant and easy- to-use infrastructure for online transaction processing
- 6. PRESENTED BY Redis Cluster at Box CACHE SOURCE OF TRUTH • Transient • Fast • Synchronized with external source of truth • Persistent • Redundant • Capable of handling diverse access patterns
- 7. PRESENTED BY Redis Cluster at Box: Cache Case Study PHP Clients Data Access Service Java Clients Scala Clients Data Access Service
- 8. PRESENTED BY Redis Cluster at Box: Cache Case Study Basic APIs Rich APIs Consistent Performance Spiky Performance Operationally Simple Operationally Complex
- 9. PRESENTED BY Redis Cluster at Box: Cache Case Study Cache relationships between objects: • For each relationship differentiate cache values by relationship facet • Be able to clear all facets of a single relationship quickly D A B C 03/2019 02/2019 01/2019 Folder D’s files ordered by file name: A, B, C Folder D’s files ordered by last modified date: C, B, A Folder D’s files larger than 100KB ordered by size: C, B 10KB 12.5MB 7MB X 01/2019 7MB
- 10. PRESENTED BY Redis Cluster at Box: Cache Case Study Cache relationships between objects with the following requirements: • For each relationship differentiate cache values by relationship facet • Be able to clear all facets of a single relationship quickly field value Key: [“folder”, “folder_id”, “D”, “files”]
- 11. PRESENTED BY Redis Cluster at Box: Cache Case Study Cache relationships between objects with the following requirements: • For each relationship differentiate cache values by relationship facet • Be able to clear all facets of a single relationship quickly field value {“where”: ”none”, “orderby”: [“file_name”, “ASC”]} A, B, C Key: [“folder”, “folder_id”, “D”, “files”]
- 12. PRESENTED BY Redis Cluster at Box: Cache Case Study Cache relationships between objects with the following requirements: • For each relationship differentiate cache values by relationship facet • Be able to clear all facets of a single relationship quickly field value {“where”: ”none”, “orderby”: [“file_name”, “ASC”]} A, B, C {“where”: “none”, “orderby”: [“mod_date”, “ASC”]} C, B, A Key: [“folder”, “folder_id”, “D”, “files”]
- 13. PRESENTED BY Redis Cluster at Box: Cache Case Study Cache relationships between objects with the following requirements: • For each relationship differentiate cache values by relationship facet • Be able to clear all facets of a single relationship quickly field value {“where”: ”none”, “orderby”: [“file_name”, “ASC”]} A, B, C {“where”: “none”, “orderby”: [“mod_date”, “ASC”]} C, B, A {“where”: “size > 100”, “orderby”: [“size”, “ASC”]} C, B Key: [“folder”, “folder_id”, ”D”, “files”]
- 14. PRESENTED BY Redis Cluster at Box: Security & Compliance Requirements • Access to every data store must be authenticated • Interactions with every data store must be encrypted • Credentials must be rotated regularly
- 15. PRESENTED BY Security options within Redis
- 16. PRESENTED BY Redis Philosophy on Security Redis is designed to be accessed by trusted clients inside trusted environments. “ https://redis.io/topics/security
- 17. PRESENTED BY Redis Philosophy on Security Redis is not optimized for maximum security but for maximum performance and simplicity “ https://redis.io/topics/security
- 18. PRESENTED BY Client-side Encryption • Key rotation problematic in use-cases where Redis is used as a system of record • Doesn’t protect against a malicious actor corrupting the dataset • Prevents effective use of ordered data types, i.e. Sorted Sets • Can work well for simple Redis-as-a-cache use-cases Plain TCP
- 19. PRESENTED BY Security options within Redis: Encryption • No native solution exists • Tunneling solutions, such as stunnel and spiped are generally recommended
- 20. PRESENTED BY Security options within Redis: Authentication The requirepass and masterauth configuration directives allow setting a single password to protect access to the cluster • Cluster members must rotate passwords synchronously • Cluster clients must rotate passwords synchronously
- 21. PRESENTED BY Security options within Redis: Authentication The requirepass and masterauth configuration directives allows setting a single password to protect access to the cluster • Cluster members must rotate passwords synchronously • Cluster clients must rotate passwords synchronously
- 22. PRESENTED BY Introducing Secure Redis Proxy
- 23. PRESENTED BY • Add authentication to Redis Cluster with online credential rotation support • Add encryption to Redis Cluster connections with online key rotation support • Minimize operational impact to the rest of the infrastructure − Minimize latency impact (< 2ms per operation) − No downtime when proxy deploys − Minimize changes for existing clients Goals
- 25. PRESENTED BY Architecture Redis clients Php & java Listen 6379 Ext. NIC Loopback addr RPX Listen 10379 Redis Instance Unix Socket iptables Port FWD 10379
- 26. PRESENTED BY iptables Port FWD 10379 RPX Architecture Listen 10379 Redis clients Php & java Listen 6379 Ext. NIC Redis Instance Loopback addr Secure Password Vault Unix Socket
- 28. PRESENTED BY Send Get Request X Failed Auth Authenticated Access using Secure Redis Proxy Clients Secure Redis Proxy Redis Instance
- 29. PRESENTED BY Clients Secure Redis Proxy Redis Instance Authenticated Access using Secure Redis Proxy
- 30. PRESENTED BY Clients Secure Redis Proxy Redis Instance Send Auth Request Authenticated Access using Secure Redis Proxy
- 31. PRESENTED BY Clients Secure Redis Proxy Redis Instance Send Auth Request Auth Success Authenticated Access using Secure Redis Proxy
- 32. PRESENTED BY Clients Secure Redis Proxy Redis Instance Send Auth Request Auth Success GET Command Authenticated Access using Secure Redis Proxy
- 33. PRESENTED BY Clients Secure Redis Proxy Redis Instance Send Auth Request Auth Success GET Command GET Command Authenticated Access using Secure Redis Proxy
- 34. PRESENTED BY Clients Secure Redis Proxy Redis Instance Send Auth Request Auth Success GET Command GET Command GET Response Authenticated Access using Secure Redis Proxy
- 35. PRESENTED BY Clients Secure Redis Proxy Redis Instance Send Auth Request Auth Success GET Command GET Command GET Response GET Response Authenticated Access using Secure Redis Proxy
- 36. PRESENTED BY • Secure Redis Proxy can read and validate multiple passwords provided by Vault • Passwords are periodically rotated and recycled using the following procedure 1. Add new password to Vault 2. Secure Redis Proxy accepts both old and new passwords 3. Clients migrate to use the new password 4. Old password is retired from Vault Password Rotation
- 37. PRESENTED BY • Sudden restart of Secure Proxy might cause a flood of reconnections from clients to Redis leading to thundering herd problem • We deploy two Secure Proxies on each system and swing the traffic between the processes using iptables rules. • To handle long living connections to Secure proxy, we issue a drain connections command to the retiring Secure Proxy process that gradually disconnects the connections Upgrades
- 38. PRESENTED BY
- 39. PRESENTED BY Or did we?????
- 40. PRESENTED BY What about Cluster Communication?
- 41. PRESENTED BY • Cluster bus communications are on a different port and they fly under the radar • No sensitive information is exchanged on this channel • User data is exchanged between cluster nodes via replication commands on the standard Redis port • The design presented so far will block this traffic preventing the cluster from operating effectively • Potential solution • Pass all replication commands originating within the cluster through Secure Redis Proxy without enforcing authentication or encryption What about Cluster Communication?
- 43. PRESENTED BY Commands ping info, command, echo, Harmless Commands Auth Command Auth <password> Other commands Get, Set Hget, Hset, cluster meet/forget etc…. Internal Redis Commands psync, replconf, cluster set-config-epoch
- 44. PRESENTED BY Commands Harmless Commands Auth Command Other commands Get, Set Hget, Hset, cluster meet/forget etc…. Pass to Redis Auth <password> Internal Redis Commands psync, replconf, cluster set-config-epoch
- 45. PRESENTED BY Commands Harmless Commands Auth Command Other commands Get, Set Hget, Hset, cluster meet/forget etc…. Pass to Redis Check Password & Set Connection Authentication State Internal Redis Commands psync, replconf, cluster set-config-epoch
- 46. PRESENTED BY Commands Harmless Commands Auth Command Other commands Pass to Redis Check Password & Set Connection Authentication State Return Forbidden Response Pass to Redis Internal Redis Commands psync, replconf, cluster set-config-epoch Is Authenticated?
- 47. PRESENTED BY Commands Harmless Commands Auth Command Other commands Pass to Redis Check Password & Set Connection Authentication State Return Forbidden Response Pass to Redis Internal Redis Commands Is Intra-cluster connection ? Pass to Redis Is Authenticated ?
- 48. PRESENTED BY
- 49. PRESENTED BY Or did we?????
- 50. PRESENTED BY Redis InstanceRPX Secured Redis Cluster
- 51. PRESENTED BY Malicious Process Meet packet On cluster Bus RPX Redis Instance Secured Redis Cluster Another Redis Instance
- 52. PRESENTED BY Replication RPX Redis Instance Meet packet On cluster Bus Secured Redis Cluster Another Redis Instance
- 53. PRESENTED BY Replication RPX Redis Instance Meet packet On cluster Bus Secured Redis Cluster Another Redis Instance
- 54. PRESENTED BY Replication RPX Redis Instance Malicious Process Meet packet On cluster Bus Secured Redis Cluster
- 55. PRESENTED BY
- 56. PRESENTED BY RPX Redis Instance Secured Redis Cluster
- 60. PRESENTED BY Zookeeper RPX Redis Instance Malicious Process Meet packet On cluster Bus Secured Redis Cluster
- 61. PRESENTED BY RPX Zookeeper Redis Instance Malicious Process Meet packet On cluster Bus Replication Secured Redis Cluster
- 62. PRESENTED BY Zookeeper X Denied RPX Redis Instance Malicious Process Meet packet On cluster Bus Replication Secured Redis Cluster
- 63. PRESENTED BY
- 64. PRESENTED BY Secure Redis Proxy in Practice
- 65. PRESENTED BY • Written in Golang • Currently in production: • 20k new client connections per second per Redis host • 15k concurrent connections per Redis host • 5 GB/s bytes per Redis host at peaks • Memory stats − < 200 MB Heap size − 50 MB/s allocation rate − < 0.5ms per GC pause • Latency Stats − ~0.5ms latency overhead per plain text request • We are currently working on enabling TLS on all proxy connections Operational Characteristics
- 66. PRESENTED BY Client side latency
- 67. Q & A
- 69. Love solving hard problems at scale? Join us at box.com/careers
- 70. PRESENTED BY
- 71. Backup slides
- 72. PRESENTED BY Work in progress for TLS support • Use golang 1.12.1 for better memory performance for Encryption and decryption • Use TLS 1.3 for better handshake performance • If using TLS 1.2 • enable TLS False Start. • configure session resumption (not so helpful in our case) • enable forward secrecy
- 73. PRESENTED BY TLS (Transport Layer Security) Ref: http://blog.fourthbit.com/2014/12/23/traffic-analysis-of-an-ssl-slash-tls-session
- 74. PRESENTED BY TLS (Transport Layer Security) Ref: http://blog.fourthbit.com/2014/12/23/traffic-analysis-of-an-ssl-slash-tls-session
- 75. PRESENTED BY Secure Redis Proxy on TLS Ref: http://blog.fourthbit.com/2014/12/23/traffic-analysis-of-an-ssl-slash-tls-session Yes No New connection from client on RPX Read First 2 bytes Initiate TLS connection Complete HandshakeRead bytes Continue Handshake