![Lists
redis.lpush("events", "first") # => 1
redis.lpush("events", "second") # => 2
redis.lrange("events", 0, -1) #
=> ["second", "first"]
redis.rpush("events", "third") # => 3
redis.lrange("events", 0, -1) #
=> ["second", "first", "third"]
redis.lpop("events") # => "second"
redis.lrange("events", 0, -1) #
=> ["first", "third"]
redis.rpoplpush("events", "fourth") #
=> "third"](https://image.slidesharecdn.com/indexingthousandsofwritespersecondwithredis-110518235308-phpapp01/85/Indexing-thousands-of-writes-per-second-with-redis-61-320.jpg)
![Sets
redis.sadd("user_ids", 1) # => true
redis.scard("user_ids") # => 1
redis.smembers("user_ids") # => ["1"]
redis.sismember(1) # => true
redis.srem("user_ids", 1) # => true](https://image.slidesharecdn.com/indexingthousandsofwritespersecondwithredis-110518235308-phpapp01/85/Indexing-thousands-of-writes-per-second-with-redis-62-320.jpg)
![Sets Continued
# know_paul ["1", "3", "4"]
# know_mike ["3", "5"]
redis.sinter("know_paul", "know_mike") # =>
["3"]
redis.sdiff("know_paul", "know_mike") # =>
["1", "4"]
redis.sdiff("know_mike", "know_paul") # =>
["5"]
redis.sunion("know_paul", "know_mike") # =>
["1", "3", "4", "5"]](https://image.slidesharecdn.com/indexingthousandsofwritespersecondwithredis-110518235308-phpapp01/85/Indexing-thousands-of-writes-per-second-with-redis-63-320.jpg)
![Sorted Sets Continued
redis.zadd("wish_counts", 12, "rubyland")
redis.zrange("wish_counts", 0, -1) # =>
["paul", "rubyland"]
redis.zrange("wish_counts", 0, -1,
:with_scores => true) # =>
["paul", "2", "rubyland", "12"]
redis.zrevrange("wish_counts", 0, -1) # =>
["rubyland", "paul"]](https://image.slidesharecdn.com/indexingthousandsofwritespersecondwithredis-110518235308-phpapp01/85/Indexing-thousands-of-writes-per-second-with-redis-65-320.jpg)
![Sorted Sets Continued
redis.zrevrangebyscore("wish_counts",
"+inf", "-inf") # =>
["rubyland", "paul"]
redis.zrevrangebyscore("wish_counts",
"+inf", "10") # => ["rubyland"]
redis.zrevrangebyscore("wish_counts",
"+inf", "-inf", :limit => [0, 1]) # =>
["rubyland"]](https://image.slidesharecdn.com/indexingthousandsofwritespersecondwithredis-110518235308-phpapp01/85/Indexing-thousands-of-writes-per-second-with-redis-66-320.jpg)
![SORT
redis.sort("bond_ids",
:get => "bonds|*->bid_price") # =>
["96.01", "90.5"]
redis.sort("bond_ids",
:get => "bonds|*->bid_price",
:get => "#") # => ["1", "2"]
redis.sort("bond_ids",
:get => "bonds|*->bid_price",
:get => "#", :limit => [0, 1]) # => ["1"]](https://image.slidesharecdn.com/indexingthousandsofwritespersecondwithredis-110518235308-phpapp01/85/Indexing-thousands-of-writes-per-second-with-redis-96-320.jpg)
![SORT Continued
redis.sort("bond_ids",
:get => "bonds|*->bid_price",
:get => "#", :order => "desc") # =>
["2", "1"]
redis gem returns the
redis.sort("bond_ids", results in reverse order.
wha?!
:get => "bonds|*->ask_price",
:get => "#") # => ["2", "1"]
redis.sort("bond_ids",
:get => "bonds|*->ask_price",
:get => "#",
:store => "bond_ids_sorted_by_ask_price",
:expire => 300) # => 2](https://image.slidesharecdn.com/indexingthousandsofwritespersecondwithredis-110518235308-phpapp01/85/Indexing-thousands-of-writes-per-second-with-redis-97-320.jpg)
![Getting Records
ids = redis_sort_results.map {|id| id.to_i}
bonds = Bond.find(ids) note that prices (high
write volume data) come
bond_ids_to_bond = {} from elsewhere (not the
SQL db)
bonds.each do |bond|
bond_ids_to_bond[bond.id] = bond
end
results = ids.map do |id|
bond_ids_to_bond[id]
end](https://image.slidesharecdn.com/indexingthousandsofwritespersecondwithredis-110518235308-phpapp01/85/Indexing-thousands-of-writes-per-second-with-redis-98-320.jpg)
Ad
More Related Content
What's hot (20)
Viewers also liked (20)
Ad
Similar to Indexing thousands of writes per second with redis (20)
Ad
More from pauldix (7)
Recently uploaded (20)
Indexing thousands of writes per second with redis
- 1. Indexing Thousands of Writes per Second with Redis Paul Dix [email protected] @pauldix http://pauldix.net
- 3. I wrote this book
- 4. Benchmark Solutions* who I work for * weโre hiring, duh! email: [email protected]
- 5. had a spiel about the suit.. Before we get to the talk...
- 7. That bastard stole my thunder!
- 8. You donโt think of suit wearing badass Seรฑor Software Engineer
- 9. I work
- 10. Finance
- 11. the janitors, cleaning staff, and the 18 year old intern get this title too... Vice President
- 12. How could I wear anything but a suit? Finance + VP + Suit = douchebag
- 16. @๏ฌavorjones coauthor of Nokogiri credit: @ebiltwin
- 17. JSON vs. XML
- 18. XML Sucks Hard
- 19. JSON is teh awesome
- 21. 10x slower
- 22. Mike called BS
- 23. A bet!
- 24. and I was like: โsure, for a beerโ
- 25. and Mike was all like: โok, but thatโs lameโ
- 26. โletโs make it interesting. Loser wears my daughterโs fairy wings during your talkโ
- 27. Sure, thatโll be funny and original...
- 28. Dr. Nic in fairy wings
- 29. That bastard stole my thunder!
- 30. Nic may have done it as part of the talk, but he didnโt lose a bet... put wings on in red-faced shame. So who won?
- 32. Nokogiri ~ 6.8x slower
- 33. REXML (ActiveRecord.from_xml) ~ 400x slower
- 35. Lesson: Donโt make bar bets
- 36. However, the bet said nothing about my slides
- 37. Aaron Patterson father of nokogiri 3 slides with @tenderloveโs picture? wtf?!!
- 39. Fairy Godmother?
- 40. Lesson: Learn Photoshop (this shit is embarrassing)
- 41. Anyway, the point of the suit...
- 43. On to the actual talk...
- 44. itโs about...
- 45. Redis
- 46. Sustained write load of ~ 5k per second
- 47. Redis + other data stores = bad assery
- 48. @๏ฌavorjones and maybe about Mike being some kind of virgin mother credit: @ebiltwin
- 49. Lesson: Be speci๏ฌc about the terms of a bet (because at least someone can use Photoshop)
- 51. NoSQL
- 52. Key/Value Store
- 53. Created by Salvatore San๏ฌlippo @antirez
- 56. Basics
- 57. Keys require 'redis' redis = Redis.new redis.set("mike", "grants wishes") # => OK redis.get("mike") # => "grants wishes"
- 58. Counters redis.incr("fairy_references") # => 1 redis.decr("dignity") # => -1 redis.incrby("fairy_references", 23) # => 24 redis.decrby("dignity", 56) # => 57
- 59. Expiration redis.expire("mike", 120) redis.expireat("mike", 1.day.from_now.midnight)
- 60. Hashes redis.hset("paul", "has_wings", true) redis.hget("paul", "has_wings") # => "true" redis.hmset("paul", :location, "Baltimore", :twitter, "@pauldix") redis.hvals("paul") # => { # "has_wings"=>"true", # "location"=>"Baltimore", # "twitter"=>"@pauldix" } redis.hlen("paul") # => 3
- 61. Lists redis.lpush("events", "first") # => 1 redis.lpush("events", "second") # => 2 redis.lrange("events", 0, -1) # => ["second", "first"] redis.rpush("events", "third") # => 3 redis.lrange("events", 0, -1) # => ["second", "first", "third"] redis.lpop("events") # => "second" redis.lrange("events", 0, -1) # => ["first", "third"] redis.rpoplpush("events", "fourth") # => "third"
- 62. Sets redis.sadd("user_ids", 1) # => true redis.scard("user_ids") # => 1 redis.smembers("user_ids") # => ["1"] redis.sismember(1) # => true redis.srem("user_ids", 1) # => true
- 63. Sets Continued # know_paul ["1", "3", "4"] # know_mike ["3", "5"] redis.sinter("know_paul", "know_mike") # => ["3"] redis.sdiff("know_paul", "know_mike") # => ["1", "4"] redis.sdiff("know_mike", "know_paul") # => ["5"] redis.sunion("know_paul", "know_mike") # => ["1", "3", "4", "5"]
- 64. Sorted Sets redis.zadd("wish_counts", 2, "paul") # => true redis.zcard("wish_counts") # => 1 redis.zismember("paul") # => true redis.zrem("wish_counts", "paul") # => true
- 65. Sorted Sets Continued redis.zadd("wish_counts", 12, "rubyland") redis.zrange("wish_counts", 0, -1) # => ["paul", "rubyland"] redis.zrange("wish_counts", 0, -1, :with_scores => true) # => ["paul", "2", "rubyland", "12"] redis.zrevrange("wish_counts", 0, -1) # => ["rubyland", "paul"]
- 66. Sorted Sets Continued redis.zrevrangebyscore("wish_counts", "+inf", "-inf") # => ["rubyland", "paul"] redis.zrevrangebyscore("wish_counts", "+inf", "10") # => ["rubyland"] redis.zrevrangebyscore("wish_counts", "+inf", "-inf", :limit => [0, 1]) # => ["rubyland"]
- 67. Lesson: Keeping examples consistent with a stupid story is hard
- 68. pubsub, transactions, more commadnds. not covered here, leave me alone Thereโs more
- 69. Crazy Fast
- 70. Faster than a greased cheetah
- 71. or a Delorean with 1.21 gigawatts
- 73. No Wishes Granted f-you, f-ball!
- 74. Lesson: Getting someone to pose is easier (also, learn Photoshop)
- 75. Still monolithic not horizontally scalable, oh noes!
- 76. Can shard in client like memcached I know haters, you can do this
- 77. Still not highly available
- 78. Still susceptible to partitions
- 79. However, itโs wicked cool
- 80. Why Index with Redis?
- 81. Donโt you probably donโt need it http://www.๏ฌickr.com/photos/34353483@N00/205467442/
- 82. and youโre all like, โPaul, ...โ But I have to SCALE!
- 83. No you donโt
- 84. Trust me, Iโm wearing a suit
- 85. that means I have authority and... I know shit
- 86. and still you cry: But no, really...
- 87. Sad SQL is Sad thousands of writes per second? No me gusto!
- 88. ok, ๏ฌne.
- 89. My Use Cases
- 91. Updating every 10 seconds
- 93. Average write load of 3k-5k writes per second
- 94. LVC redis.hset("bonds|1", "bid_price", 96.01) redis.hset("bonds|1", "ask_price", 97.53) redis.hset("bonds|2", "bid_price", 90.50) redis.hset("bonds|2", "ask_price", 92.25) redis.sadd("bond_ids", 1) redis.sadd("bond_ids", 2)
- 95. Index on the ๏ฌy
- 96. SORT redis.sort("bond_ids", :get => "bonds|*->bid_price") # => ["96.01", "90.5"] redis.sort("bond_ids", :get => "bonds|*->bid_price", :get => "#") # => ["1", "2"] redis.sort("bond_ids", :get => "bonds|*->bid_price", :get => "#", :limit => [0, 1]) # => ["1"]
- 97. SORT Continued redis.sort("bond_ids", :get => "bonds|*->bid_price", :get => "#", :order => "desc") # => ["2", "1"] redis gem returns the redis.sort("bond_ids", results in reverse order. wha?! :get => "bonds|*->ask_price", :get => "#") # => ["2", "1"] redis.sort("bond_ids", :get => "bonds|*->ask_price", :get => "#", :store => "bond_ids_sorted_by_ask_price", :expire => 300) # => 2
- 98. Getting Records ids = redis_sort_results.map {|id| id.to_i} bonds = Bond.find(ids) note that prices (high write volume data) come bond_ids_to_bond = {} from elsewhere (not the SQL db) bonds.each do |bond| bond_ids_to_bond[bond.id] = bond end results = ids.map do |id| bond_ids_to_bond[id] end
- 99. Getting From Redis redis.hset("bonds|2", "values", data.to_json) raw_json = redis.sort("bond_ids", However, then you have to worry about keeping the t wo data stores in :get => "bonds|*->bid_price", sync. Weโll talk about it later :get => "bonds|*->values") results = raw_json.map do |json| DataObject.new(JSON.parse(json)) end
- 100. Pre-Indexing
- 101. Rolling Index
- 102. Last n Events
- 103. Activity Log
- 104. News Feed
- 105. Use a List O(1) constant time complexity to add O(start + n) for reading N = 500 size = redis.lpush("bond_trades|1", trade_id) # roll the index redis.rpop("bond_trades|1") if size > N # get results redis.lrange("bond_trades|1", 0, 49)
- 106. Indexing Events Since Time T
- 107. Using a List redis.lpush("bond_trades|1|2011-05-19-10", trade_id) redis.lrange("bond_trades|1|2011-05-19-10", 0, -1) results = redis.pipelined do redis.lrange("bond_trades|1|2011-05-19-10", 0, -1) redis.lrange("bond_trades|1|2011-05-19-09", 0, -1) end.flatten
- 108. Rolling the Index # when something trades redis.sadd("bonds_traded|2011-05-19-10", bond_id) # cron task to remove old data traded_ids = redis.smembers( "bonds_traded|2011-05-19-10") keys = traded_ids.map do |id| "bond_trades|#{id}|2011-05-19-10" end keys << "bonds_traded|2011-05-19-10" redis.del(*keys)
- 109. Using a Sorted Set # Time based Rolling Index using sorted set redis.zadd("bond_trades|1", O(log(n)) writes Time.now.to_i, trade_id) O(log(n) + M) reads # last 20 trades redis.zrevrange("bond_trades|1", 0, 20) # trades in the last hour redis.zrevrangebyscore("bond_trades|1", "+inf", 1.hour.ago.to_i)
- 110. Rolling the Index # cron task to roll the index bond_ids = redis.smembers("bond_ids") remove_since_time = 24.hours.ago.to_i redis.pipelined do bond_ids.each do |id| redis.zremrangebyscore( "bond_trades|#{id}", "-inf", remove_since_time) end end
- 111. Or roll on read or write redis.zadd("bond_trades|1", Time.now.to_i, trade_id) redis.zremrangebyscore("bond_trades|1", "-inf", 24.hours.ago)
- 112. Indexing N Values redis.zadd("highest_follower_counts", 2300, 20) redis.zadd("lowest_follower_counts", 2300, 20) # rolling the indexes # keep the lowest size = redis.zcard("lowest_follower_counts") redis.zremrangebyrank("lowest_follower_counts", N, -1) if size > N # keep the highest size = redis.zcard("highest_follower_counts") redis.zremrangebyrank("highest_follower_counts", 0, size - N) if size > N
- 113. rolling requires more roundtrips 2 roundtrips (only with complex pipelining)
- 114. Roll indexes with only one trip
- 115. Tweet to @antirez that you want scripting
- 118. database transactions canโt help you here, youโll have to put them into your application logic No transactions, application logic
- 119. Disaster Recovery
- 121. Web app dies
- 122. Redis server dies
- 123. Could result in index inconsistency
- 125. Write Index Times redis.set("last_bond_trade_indexed", trade.created_at.to_i)
- 126. Restore Each Index time_int = redis.get("last_bond_trade_indexed").to_i index_time = Time.at(time_int) trades = Trade.where( "created_at > :index_time AND created_at <= :now", {:index_time => index_time, :now => Time.now}) trades.each do |trade| list you have to run while not writing new data. trade.index_in_redis Set can be made to run end while writing new data
- 127. Our scale
- 128. Single Process
- 129. Sets donโt work with intersection, union, or diff. SORT wonโt work unless all those keys fall on the same server Easy to Scale (consistent hashing)
- 130. Works like a champ
- 131. Final Thoughts
- 132. Use Only if you have to!
- 133. Index the minimum to keep memory footprint down use rolling indexes, donโt keep more shit in memory than you need. Users wonโt page through 20 pages of results, so donโt store that many
- 134. Plan for disaster and consistency checking
- 135. Finally...
- 136. Look at my circle, bitches!
- 137. Lesson: Never trust a guy in a suit not pull a fast one on you
- 138. Thanks! Paul Dix [email protected] @pauldix http://pauldix.net
Editor's Notes
- #2: \n
- #3: \n
- #4: \n
- #5: \n
- #6: \n
- #7: \n
- #8: \n
- #9: \n
- #10: \n
- #11: \n
- #12: \n
- #13: \n
- #14: \n
- #15: \n
- #16: \n
- #17: \n
- #18: \n
- #19: \n
- #20: \n
- #21: \n
- #22: \n
- #23: \n
- #24: \n
- #25: \n
- #26: \n
- #27: \n
- #28: \n
- #29: \n
- #30: \n
- #31: \n
- #32: \n
- #33: \n
- #34: \n
- #35: \n
- #36: \n
- #37: \n
- #38: \n
- #39: \n
- #40: \n
- #41: \n
- #42: \n
- #43: \n
- #44: \n
- #45: \n
- #46: \n
- #47: \n
- #48: \n
- #49: \n
- #50: \n
- #51: \n
- #52: \n
- #53: \n
- #54: \n
- #55: \n
- #56: \n
- #57: \n
- #58: \n
- #59: \n
- #60: \n
- #61: \n
- #62: \n
- #63: \n
- #64: \n
- #65: \n
- #66: \n
- #67: \n
- #68: \n
- #69: \n
- #70: \n
- #71: \n
- #72: \n
- #73: \n
- #74: \n
- #75: \n
- #76: \n
- #77: \n
- #78: \n
- #79: \n
- #80: \n
- #81: \n
- #82: \n
- #83: \n
- #84: \n
- #85: \n
- #86: \n
- #87: \n
- #88: \n
- #89: \n
- #90: \n
- #91: \n
- #92: \n
- #93: \n
- #94: \n
- #95: \n
- #96: \n
- #97: \n
- #98: \n
- #99: \n
- #100: \n
- #101: \n
- #102: \n
- #103: \n
- #104: \n
- #105: \n
- #106: \n
- #107: \n
- #108: \n
- #109: \n
- #110: \n
- #111: \n
- #112: \n
- #113: \n
- #114: \n
- #115: \n
- #116: \n
- #117: \n
- #118: \n
- #119: \n
- #120: \n
- #121: \n
- #122: \n
- #123: \n
- #124: \n
- #125: \n
- #126: \n
- #127: \n
- #128: \n
- #129: \n
- #130: \n
- #131: \n
- #132: \n
- #133: \n
- #134: \n
- #135: \n
- #136: \n
- #137: \n
- #138: \n
- #139: \n