Fosdem how you can benefit from redis, javier ramirez @ teowaki
4 likes3,680 views
The document discusses Redis and how it is used by various companies. It provides an overview of Redis capabilities and data types. It also shares examples of how companies like Twitter, World of Warcraft, Pinterest, and others use Redis for caching, queues, and more. Specific uses by Teowaki are also highlighted such as intermediate caching, Lua scripting, counters, and caching with Nginx.
Fosdem how you can benefit from redis, javier ramirez @ teowaki
- 1. How you can benefit from using Redis javier ramirez @supercoco9
- 7. I was Lois Lane javier ramirez @supercoco9 https://teowaki.com FOSDEM 2014
- 8. In the beginner's mind there are many possibilities, in the expert's mind there are few. Shunryu Suzuki javier ramirez @supercoco9 https://teowaki.com FOSDEM 2014
- 9. myth: the bottleneck javier ramirez @supercoco9 https://teowaki.com FOSDEM 2014
- 10. redis-benchmark -r 1000000 -n 2000000 -t get,set,lpush,lpop,mset -P 16 -q On my laptop: SET: 513610 requests per second GET: 641436 requests per second LPUSH: 845665 requests per second LPOP: 783392 requests per second MSET (10 keys): 89988 requests per second On a small digital ocean server ($5/month) SET: 227816 requests per second GET: 228258 requests per second LPUSH: 251098 requests per second LPOP: 251572 requests per second MSET (10 keys): 43918 requests per second javier ramirez @supercoco9 https://teowaki.com FOSDEM 2014
- 11. myth: Redis is just like memcached javier ramirez @supercoco9 https://teowaki.com FOSDEM 2014
- 12. myth: Redis is for queues javier ramirez @supercoco9 https://teowaki.com FOSDEM 2014
- 13. open source, BSD licensed, advanced key-value store. It is often referred to as a data structure server since keys can contain strings, hashes, lists, sets and sorted sets. http://redis.io started in 2009 by Salvatore Sanfilippo @antirez 104 contributors at https://github.com/antirez/redis javier ramirez @supercoco9 https://teowaki.com FOSDEM 2014
- 14. The Redis Manifesto 1.A DSL for Abstract Data Types 2.Memory storage is #1 3.Fundamental data structures for a fundamental API 4.Two levels of API 5.Code is like a poem; it's not just something we write to reach some practical result 6.We're against complexity 7.We optimize for joy javier ramirez @supercoco9 https://teowaki.com FOSDEM 2014
- 15. Redis makes you think javier ramirez @supercoco9 https://teowaki.com FOSDEM 2014
- 16. Redis data types Strings Hashes Lists Sets Sorted Sets javier ramirez @supercoco9 https://teowaki.com FOSDEM 2014
- 17. basic commands Setting/getting/deleting/expiring Increment/decrement Pushing/popping Checking membership, size... javier ramirez @supercoco9 https://teowaki.com FOSDEM 2014
- 18. much cooler commands Setting only if exists Blocking pop (Blocking) pop from one list, push to another Get/set string ranges (and bit operations) Set intersections/diffs (and store) Pub/sub javier ramirez @supercoco9 https://teowaki.com FOSDEM 2014
- 19. A chat in 6 lines of code r=Redis.new driver: :hiredis r.subscribe “chatrooms:42” do |on| on.message do |topic, msg| puts “#{topic}: #{msg}” end end r.publish “chatrooms:42”, “Hello Fosdem” javier ramirez @supercoco9 https://teowaki.com FOSDEM 2014
- 20. atomicity single threaded, so no concurrency problems transactions and lua scripts to run multiple operations atomically javier ramirez @supercoco9 https://teowaki.com FOSDEM 2014
- 21. Scripting with lua You can use Lua for scripting Redis when you need to atomically execute a sequence of commands in which the output of a command is used as input for another It reduces the need to use complex lock mechanisms and simplifies dependencies between clients You can even extend the functionality of Redis by using Lua scripts javier ramirez @supercoco9 https://teowaki.com FOSDEM 2014
- 22. Redis keeps everything in memory all the time javier ramirez @supercoco9 https://teowaki.com FOSDEM 2014
- 23. Persistance: RDB Compact binary format Saves snapshots every few minutes Good for backups and synchronizing If Redis crashes, a few minutes worth of data will be lost javier ramirez @supercoco9 https://teowaki.com FOSDEM 2014
- 24. Durability: AOF Log text format Configurable durability Large file, can slow down startup If Redis crashes, typically one second of data could be lost javier ramirez @supercoco9 https://teowaki.com FOSDEM 2014
- 25. replication & scaling out one master, several read-only slaves sharding Twemproxy & redis cluster javier ramirez @supercoco9 https://teowaki.com FOSDEM 2014
- 26. what's being used for javier ramirez @supercoco9 https://teowaki.com FOSDEM 2014
- 27. twitter Every time line (800 tweets per user) is on redis 5000 writes per second avg 300K reads per second javier ramirez @supercoco9 https://teowaki.com FOSDEM 2014
- 28. twitter write API (from browser or client app) follo e per on user id tweet id wer fanout (flockDB) metadata rpushx to Redis user info from gizmoduck (memcached) javier ramirez @supercoco9 tweet info from tweetypie (memcached + mysql) https://teowaki.com your twitter timeline FOSDEM 2014
- 29. World Of Warcraft Blizzard is quite secretive about it Known to have at least 8 servers with 64Gb each to serve user avatars Known to have seen 1.000.000 users concurrently in the Asian version of the game javier ramirez @supercoco9 https://teowaki.com FOSDEM 2014
- 30. pinterest object graph per user Redis SortedSet, with timestamp as the score to store the users followed explicitly store the users followed implicitly store the user’s explicit followers store the user’s implicit followers store boards followed explicitly Redis set to store boards unfollowed explicitly per board Redis Hash to store a board’s explicit followers Redis Set to store a board’s explicit unfollowers javier ramirez @supercoco9 https://teowaki.com FOSDEM 2014
- 31. viacom Object dependency graph. Cache on steroids Redis as a queue for background jobs Activity tracking and view counts buffering before saving to mysql Lua script working on a slave to recalculate ranking and popularity of contents, then send the data to master. The new process takes 1/60th less than the old one in mysql javier ramirez @supercoco9 https://teowaki.com FOSDEM 2014
- 32. stack overflow Three level cache: local cache (no persistence) sessions, and pending view count updates site cache hot question id lists, user acceptance rates... global cache (separate Redis DB) Inboxes, API usage quotas... javier ramirez @supercoco9 https://teowaki.com FOSDEM 2014
- 33. youporn Most data is found in hashes with ordered sets used to know what data to show. zInterStore on: videos:filters:released, Videos:filters:orientation:straight,Videos:filters:categorie s:{category_id}, Videos:ordering:rating Then perform a zRange to get the pages we want and get the list of video_ids back. Then start a pipeline and get all the videos from hashes. javier ramirez @supercoco9 https://teowaki.com FOSDEM 2014
- 34. github Git as a service internally. Redis is used for storing routing info matching user repositories to server names hipchat Redis for caching. Information like which users are in which rooms, presence information, who is online, etc, so it doesn’t matter which XMPP server you connect to. javier ramirez @supercoco9 https://teowaki.com FOSDEM 2014
- 35. when things go wrong javier ramirez @supercoco9 https://teowaki.com FOSDEM 2014
- 36. the instagram case moving from redis to cassandra: 75% savings on servers lesson learnt: know when redis is not appropriate javier ramirez @supercoco9 https://teowaki.com FOSDEM 2014
- 37. the twilio case credit card hell lesson learnt: know what you are doing. Don't change config on the fly javier ramirez @supercoco9 https://teowaki.com FOSDEM 2014
- 38. Numbers can be scary Twitter: 2TB RAM for Redis on its own servers Blizzard: At least 8 servers with 64Gb each Pinteres:t 110 Redis instances Instagram: 1.2 TB of data javier ramirez @supercoco9 https://teowaki.com FOSDEM 2014
- 39. openredis redis as a service on AWS serving over 1000 customers with a single machine javier ramirez @supercoco9 https://teowaki.com FOSDEM 2014
- 40. how teowaki is using redis javier ramirez @supercoco9 https://teowaki.com FOSDEM 2014
- 41. Abusing sidekiq/resque keep track of every activity in the system, even if you don't need them all right now: - every page view - every API request - every time a record is created/updated/deleted benefits: - highly decoupled system - easier to divide into services - you can add behaviour without changing your app javier ramirez @supercoco9 https://teowaki.com FOSDEM 2014
- 42. intermediate cache * As a very fast lightweight storage for analytics data before sending them to our google bigquery based solution * As a cache for attributes frequently looked up in join tables (names, nicknames, guids, delegated or included model names...) javier ramirez @supercoco9 https://teowaki.com FOSDEM 2014
- 43. Some of our uses of Lua Expiring attributes inside a Redis hash Inserting notifications into a list only if there are not pending notifications from the same user for the same scope Paginating a list by an attribute Manipulating JSON directly at the Redis layer javier ramirez @supercoco9 https://teowaki.com FOSDEM 2014
- 44. counters Atomic counters can be safely invoked concurrently from anywhere, so you can implement “like” features, global sequences or usage monitoring systems in highly concurrent applications for free. You can share your counters with any other internal application and still be sure they won't collide. javier ramirez @supercoco9 https://teowaki.com FOSDEM 2014
- 45. Temporary data Redis allows us to self expire keys after a time has passed. You can use this mechanism for a simple cache If you operate on a key with expiration time, you can change its value and still keep the expiration going. By combining a decrementing counter with an expiration time, implementing usage quotas is trivial Also, you can inspect which keys you have in your server efficiently using SCAN commands javier ramirez @supercoco9 https://teowaki.com FOSDEM 2014
- 46. nginx + lua + redis apache + mruby + redis Multiple levels of cache by using Redis on the webserver/ middleware layer http://wiki.nginx.org/HttpRedis javier ramirez @supercoco9 https://teowaki.com FOSDEM 2014
- 47. summarizing * Redis is more powerful than it seems * Very fast, easy to use, simple, good documentation * In-memory data structures, distributed, shared and persisted * Good as data store, intermediate data store, cache or queue * Lots of use cases, both in huge and smaller systems You should probably use it a lot more javier ramirez @supercoco9 https://teowaki.com FOSDEM 2014
- 48. Find related links at https://teowaki.com/teams/javier-community/link-categories/redis Dank u. Merci. If you enjoyed this talk, please sign up for https://teowaki.com Javier Ramírez @supercoco9 FOSDEM 2014