Redis in Practice
195 likes19,904 views
The document discusses the use of Redis, a flexible key-value database, in various practical scenarios such as caching, social activity feeds, and friend suggestions. It presents case studies on leveraging Redis for efficient data handling, including global locks, online presence tracking, and friend connection recommendations. The authors, Noah Davis and Luke Melia, also highlight Redis's strengths, features, and its suitability for specific applications in production environments.
![Working with Redis sets 1/3
# SADD key, member
# Adds the specified member to the set stored at key
redis = Redis.new
redis.sadd 'my_set', 'foo' # => true
redis.sadd 'my_set', 'bar' # => true
redis.sadd 'my_set', 'bar' # => false
redis.smembers 'my_set' # => ["foo", "bar"]](https://image.slidesharecdn.com/redis-in-practice-101202225148-phpapp01/85/Redis-in-Practice-26-320.jpg)
![Working with Redis sets 2/3
# SUNION key1 key2 ... keyN
# Returns the members of a set resulting from the union of all
# the sets stored at the specified keys.
# SUNIONSTORE <i>dstkey key1 key2 ... keyN</i></b>
# Works like SUNION but instead of being returned the resulting
# set is stored as dstkey.
redis = Redis.new
redis.sadd 'set_a', 'foo'
redis.sadd 'set_a', 'bar'
redis.sadd 'set_b', 'bar'
redis.sadd 'set_b', 'baz'
redis.sunion 'set_a', 'set_b' # => ["foo", "baz", "bar"]
redis.sunionstore 'set_ab', 'set_a', 'set_b'
redis.smembers 'set_ab' # => ["foo", "baz", "bar"]](https://image.slidesharecdn.com/redis-in-practice-101202225148-phpapp01/85/Redis-in-Practice-27-320.jpg)
![Working with Redis sets 3/3
# SINTER key1 key2 ... keyN
# Returns the members that are present in all
# sets stored at the specified keys.
redis = Redis.new
redis.sadd 'set_a', 'foo'
redis.sadd 'set_a', 'bar'
redis.sadd 'set_b', 'bar'
redis.sadd 'set_b', 'baz'
redis.sinter 'set_a', 'set_b' # => ["bar"]](https://image.slidesharecdn.com/redis-in-practice-101202225148-phpapp01/85/Redis-in-Practice-28-320.jpg)
![via Redis, 1/2
LPUSH “user:11:feed”, “100”
LTRIM “user:11:feed”, 0, 50
LPUSH 'user:11:feed', '100' Key Value
Friend: 11
LTRIM 'user:11:feed', 0, 50
Activity: 100
user:11:feed [100,99,97,96]
LPUSH 'user:22:feed', '100'
Actor 1 Friend: 22
LTRIM 'user:22:feed', 0, 50
user:22:feed [100,99,98]
LPUSH 'user:33:feed', '100'
Teammate: 33
LTRIM 'user:33:feed', 0, 100 user:33:feed [100,99]](https://image.slidesharecdn.com/redis-in-practice-101202225148-phpapp01/85/Redis-in-Practice-40-320.jpg)
![via Redis, 2/2
Key Value
Friend: 11
user:11:feed [100,99,97,96]
Friend: 22
user:22:feed [100,99,98]
Activity: 100
Actor 1
user:33:feed [100,99]
Teammate: 33
Key Value
New York
new_york:feed [100,67]
Boston boston:feed [100,99]](https://image.slidesharecdn.com/redis-in-practice-101202225148-phpapp01/85/Redis-in-Practice-41-320.jpg)
![Friend Suggestions in Redis
zincrby “user:1:suggestions” 1 “donny”
zincrby “user:1:suggestions” 1 “donny”
Sarah
Frank Donny
Me
zincrby “user:1:suggestions” 1 “erika”
Ted
Erika
zincrby “user:1:suggestions” 1 “erika”
[ Donny 2 , Erika 2 ]](https://image.slidesharecdn.com/redis-in-practice-101202225148-phpapp01/85/Redis-in-Practice-50-320.jpg)
![Friend Suggestions in Redis
zincrby “user:1:suggestions” 1 “erika”
Sarah
Frank Donny
Me
Ted
zrevrange “user:1:suggestions” 0 1 Erika
Sue
[ Erika 3 , Donny 2 ]](https://image.slidesharecdn.com/redis-in-practice-101202225148-phpapp01/85/Redis-in-Practice-51-320.jpg)
![As an LRU cache
TTL-support since inception, but with unintuitive behavior
Writing to volatile key replaced it and cleared the TTL
Redis 2.2 changes this behavior and adds key features:
ability to write to, and update expiry of volatile keys
maxmemory [bytes], maxmemory-policy [policy]
policies: volatile-lru, volatile-ttl, volatile-random,
allkeys-lru, allkeys-random](https://image.slidesharecdn.com/redis-in-practice-101202225148-phpapp01/85/Redis-in-Practice-54-320.jpg)
Redis in Practice
- 1. Redis in Practice NoSQL NYC • December 2nd, 2010 Noah Davis & Luke Melia
- 2. About Noah & Luke Weplay.com Rubyists Running Redis in production since mid-2009 @noahd1 and @lukemelia on Twitter
- 4. Tonight’s gameplan Fly-by intro to Redis Redis in Practice View counts Q&A throughout, please. Global locks We love being Presence interrupted. Social activity feeds Friend suggestions Caching
- 5. Salvatore Sanfilippo Original author of Redis. @antirez on Twitter. Lives in Italy. Recently hired by VMWare. Great project leader.
- 6. Describing Redis Remote dictionary server “advanced, fast, persistent key- value database” “Data structures server” “memcached on steroids”
- 7. In Salvatore’s words “I see Redis definitely more as a flexible tool than as a solution specialized to solve a specific problem: his mixed soul of cache, store, and messaging server shows this very well.” - Salvatore Sanfilippo
- 8. Qualities Written in C Few dependencies Fast Single-threaded Lots of clients available Initial release was March 2009
- 9. Features Key-value store where a value is one of: scalar/string, list, set, sorted sets, hash Persistence: in-memory, snapshots, append-only log, VM Replication: master-slave, configureable Pub-Sub Expiry “Transaction”-y In development: Redis Cluster
- 10. What Redis ain’t (...yet) Big data Seamless scaling Ad-hoc query tool The only data store you’ll ever need
- 11. Who’s using Redis? VMWare Grooveshark Github Superfeedr Craigslist Ravelry EngineYard mediaFAIL The Guardian Weplay Forrst More... PostRank
- 12. Our infrastructure App server App server Utility server MySQL master Redis master MySQL slave Redis slave
- 13. Redis in Practice #1 View Counts
- 14. View counts potential for massive amounts of simple writes/reads valuable data, but not mission critical lots of row contention in SQL
- 15. Redis incrementing $ redis-cli redis> incr "medium:301:views" (integer) 1 redis> incr "medium:301:views" (integer) 2 redis> incrby "medium:301:views" 5 (integer) 7 redis> decr "medium:301:views" (integer) 6
- 16. Redis in Practice #2 Distributed Locks
- 17. Subscribe to a Weplay Calendar Subscribe to “ics”
- 18. Subscription challenges Generally static content for long periods of time, but with short sessions of frequent updates Expensive to compute on the fly
- 19. Without Redis/Locking CANCEL EVENT BACKGROUND Pubisher QUEUE Queue: Publish ICS Publish Contention/Redundancy ADD EVENT BACKGROUND Pubisher QUEUE Queue: Publish ICS Publish
- 20. Redis Locking: SetNX redis = Redis.new redis.setnx "locking.key", Time.now + 2.hours => true redis.setnx "locking.key", Time.now + 2.hours => false redis.del "locking.key" => true¨ redis.setnx "locking.key", Time.now + 2.hours => true
- 21. Redis: Obtained Lock ADD EVENT REDIS SETNX "group_34_publish_ics" Returns: 1 BACKGROUND PUBLISHER REDIS QUEUE 5 minutes later: PUBLISH.... Queue Job: Publish.publish_ics("34") DEL "group_34_publish_ics"
- 22. Redis: Locked Out CANCEL EVENT REDIS SETNX "group_34_publish_ics" Returns: 0 ADD EVENT REDIS SETNX "group_34_publish_ics" Returns: 0
- 23. Redis in Practice #3 Presence
- 24. “Who’s online?” Weplay members told us they wanted to be able to see which of their friends were online...
- 25. About sets 0 to N elements Adding a value to a set does not require you Unordered to check if the value No repeated members exists in the set first
- 26. Working with Redis sets 1/3 # SADD key, member # Adds the specified member to the set stored at key redis = Redis.new redis.sadd 'my_set', 'foo' # => true redis.sadd 'my_set', 'bar' # => true redis.sadd 'my_set', 'bar' # => false redis.smembers 'my_set' # => ["foo", "bar"]
- 27. Working with Redis sets 2/3 # SUNION key1 key2 ... keyN # Returns the members of a set resulting from the union of all # the sets stored at the specified keys. # SUNIONSTORE <i>dstkey key1 key2 ... keyN</i></b> # Works like SUNION but instead of being returned the resulting # set is stored as dstkey. redis = Redis.new redis.sadd 'set_a', 'foo' redis.sadd 'set_a', 'bar' redis.sadd 'set_b', 'bar' redis.sadd 'set_b', 'baz' redis.sunion 'set_a', 'set_b' # => ["foo", "baz", "bar"] redis.sunionstore 'set_ab', 'set_a', 'set_b' redis.smembers 'set_ab' # => ["foo", "baz", "bar"]
- 28. Working with Redis sets 3/3 # SINTER key1 key2 ... keyN # Returns the members that are present in all # sets stored at the specified keys. redis = Redis.new redis.sadd 'set_a', 'foo' redis.sadd 'set_a', 'bar' redis.sadd 'set_b', 'bar' redis.sadd 'set_b', 'baz' redis.sinter 'set_a', 'set_b' # => ["bar"]
- 29. Approach 1/2
- 30. Approach 2/2
- 31. Implementation 1/2 # Defining the keys def current_key key(Time.now.strftime("%M")) end def keys_in_last_5_minutes now = Time.now times = (0..5).collect {|n| now - n.minutes } times.collect{ |t| key(t.strftime("%M")) } end def key(minute) "online_users_minute_#{minute}" end
- 32. Implementation 2/2 # Tracking an Active User, and calculating who’s online def track_user_id(id) key = current_key redis.sadd(key, id) end def online_user_ids redis.sunion(*keys_in_last_5_minutes) end def online_friend_ids(interested_user_id) redis.sunionstore("online_users", *keys_in_last_5_minutes) redis.sinter("online_users", "user:#{interested_user_id}:friend_ids") end
- 33. Redis in Practice #4 Social Activity Feeds
- 35. Via SQL, Approach 1/2 Doesn’t scale to more complex social graphs e.g. friends who are SELECT activities.* FROM activities JOIN friendships f1 ON f1.from_id = ‘hidden’ from your feed activities.actor_id JOIN friendships f2 ON f2.to_id = activities.actor_id May still require multiple WHERE f1.to_id = ? OR f2.from_id = ? ORDER BY activities.id DESC queries to grab LIMIT 15 unindexed data
- 36. Via SQL, Approach 2/2 user_id 11 activity_id 96 22 96 11 97 22 97 33 97 11 98 22 98 Friend: 11 11 99 Activity: 100 11 100 Actor 1 Friend: 22 INSERTS 22 100 Teammate: 33 33 100
- 37. Large SQL table Grew quickly Difficult to maintain Difficult to prune
- 38. Redis Lists 1/2 redis> lpush "teams" "yankees" (integer) 1 redis> lpush "teams" "redsox" (integer) 2 redis> llen "teams" (integer) 2 redis> lrange "teams" 0 1 1. "redsox" 2. "yankees" redis> lrange "teams" 0 -1 1. "redsox" 2. "yankees" LTRIM, LLEN, LRANGE
- 39. Redis Lists 2/2 redis> ltrim "teams" 0 0 OK redis> lrange "teams" 0 -1 1. "redsox" LTRIM
- 40. via Redis, 1/2 LPUSH “user:11:feed”, “100” LTRIM “user:11:feed”, 0, 50 LPUSH 'user:11:feed', '100' Key Value Friend: 11 LTRIM 'user:11:feed', 0, 50 Activity: 100 user:11:feed [100,99,97,96] LPUSH 'user:22:feed', '100' Actor 1 Friend: 22 LTRIM 'user:22:feed', 0, 50 user:22:feed [100,99,98] LPUSH 'user:33:feed', '100' Teammate: 33 LTRIM 'user:33:feed', 0, 100 user:33:feed [100,99]
- 41. via Redis, 2/2 Key Value Friend: 11 user:11:feed [100,99,97,96] Friend: 22 user:22:feed [100,99,98] Activity: 100 Actor 1 user:33:feed [100,99] Teammate: 33 Key Value New York new_york:feed [100,67] Boston boston:feed [100,99]
- 43. Redis in Practice #5 Friend Suggestions
- 44. Friend Suggestions suggest new connections based on existing connections expanding the social graph and mirroring real world connections is key
- 45. The Concept Paul Me George John Ringo
- 46. The Yoko Factor Paul Me George John Yoko Ringo
- 47. The Approach 1/2 Sarah 2 Frank Donny Me 2 Ted Erika
- 48. The Approach 2/2 Sarah 2 Frank Donny Me Ted 3 Erika Sue
- 49. ZSETS in Redis “Sorted Sets” Each member of the set has a score Ordered by the score at all times
- 50. Friend Suggestions in Redis zincrby “user:1:suggestions” 1 “donny” zincrby “user:1:suggestions” 1 “donny” Sarah Frank Donny Me zincrby “user:1:suggestions” 1 “erika” Ted Erika zincrby “user:1:suggestions” 1 “erika” [ Donny 2 , Erika 2 ]
- 51. Friend Suggestions in Redis zincrby “user:1:suggestions” 1 “erika” Sarah Frank Donny Me Ted zrevrange “user:1:suggestions” 0 1 Erika Sue [ Erika 3 , Donny 2 ]
- 52. Redis in Practice #6 Caching
- 53. Suitability for caching Excellent match for managed denormalization ex. friendships, teams, teammates Excellent match where you would benefit from persistence and/or replication Historically, not a good match for a “generational cache,” in which you want to optimize memory use by evicting least- recently used (LRU) keys
- 54. As an LRU cache TTL-support since inception, but with unintuitive behavior Writing to volatile key replaced it and cleared the TTL Redis 2.2 changes this behavior and adds key features: ability to write to, and update expiry of volatile keys maxmemory [bytes], maxmemory-policy [policy] policies: volatile-lru, volatile-ttl, volatile-random, allkeys-lru, allkeys-random
- 55. Easy to adapt Namespaced Rack::Session, Rack::Cache, I18n and cache Redis cache stores for Ruby web frameworks implementation is under 1,000 LOC
- 57. Any questions? Thanks! Follow us on Twitter: @noahd1 and @lukemelia Tell your friends in youth sports about Weplay.com
Editor's Notes
- #2: \n
- #3: \n
- #4: \n
- #5: \n
- #6: \n
- #7: \n
- #8: \n
- #9: \n
- #10: \n
- #11: \n
- #12: \n
- #13: \n
- #14: \n
- #15: \n
- #16: \n
- #17: \n
- #18: \n
- #19: \n
- #20: \n
- #21: \n
- #22: \n
- #23: \n
- #24: \n
- #25: \n
- #26: \n
- #27: \n
- #28: \n
- #29: \n
- #30: \n
- #31: \n
- #32: \n
- #33: \n
- #34: \n
- #35: \n
- #36: \n
- #37: \n
- #38: \n
- #39: \n
- #40: \n
- #41: \n
- #42: \n
- #43: \n
- #44: \n
- #45: \n
- #46: \n
- #47: \n
- #48: \n
- #49: \n
- #50: \n
- #51: \n
- #52: \n
- #53: \n
- #54: \n
- #55: http://antirez.com/m/p.php?i=220\nvolatile-lru remove a key among the ones with an expire set, trying to remove keys not recently used.\nvolatile-ttl remove a key among the ones with an expire set, trying to remove keys with short remaining time to live.\nvolatile-random remove a random key among the ones with an expire set.\nallkeys-lru like volatile-lru, but will remove every kind of key, both normal keys or keys with an expire set.\nallkeys-random like volatile-random, but will remove every kind of keys, both normal keys and keys with an expire set.\n\n
- #56: Another 1500 LOC for specs\n
- #57: \n
- #58: \n