Redis Everywhere - Sunshine PHP
29 likes22,506 views
Redis is an in-memory key-value data store that can be used for caching, sessions, queues, leaderboards, and more. It provides fast performance due to being memory-resident and supporting different data structures like strings, hashes, lists, sets, and sorted sets. Redis is useful for read-heavy and real-time applications but may not be suitable if data does not fit in memory or for relational data needs. The presentation discusses using Redis with PHP and Symfony, data sharding strategies, and war stories from a social game with 7.5M daily users.
![ALSO IN 2.8 (H|Z|S)?SCAN
• Ability
to iterate the data structures
• SCAN
cursor [MATCH pattern] [COUNT count]
• (H|Z|S)SCAN
• Cursor
• Count
• When
key cursor [MATCH pattern] [COUNT count]
is kind of a pointer that we send back and forth
is NOT the number of elements but the complexity
using MATCH some steps in iteration may return nil](https://image.slidesharecdn.com/redis-everywhere-sunshinephp-140208173044-phpapp01/85/Redis-Everywhere-Sunshine-PHP-13-320.jpg)
![LEADERBOARDS
• Super-easy
• Each
with Redis, heavy consuming with other systems
board with a single key, using sorted sets
• ZINCRBY “rankXXX” <increment>
<userId>
• Top
N ZREVRANGE “rankXXX” 0 N [WITHSCORES]
• We
stored several milions of members under 1 key](https://image.slidesharecdn.com/redis-everywhere-sunshinephp-140208173044-phpapp01/85/Redis-Everywhere-Sunshine-PHP-23-320.jpg)
Ad
More Related Content
What's hot (20)
Viewers also liked (10)
Ad
Similar to Redis Everywhere - Sunshine PHP (20)
![NoSQL for great good [hanoi.rb talk]](https://cdn.slidesharecdn.com/ss_thumbnails/kvs-hanoirb-150728163935-lva1-app6891-thumbnail.jpg?width=560&fit=bounds)
Ad
More from Ricard Clau (11)
Recently uploaded (20)
Redis Everywhere - Sunshine PHP
- 2. HELLO WORLD • Ricard Clau, born and grown up in Barcelona • Software engineer at Hailo in London • Symfony2 lover and PHP believer • Open-source • Twitter contributor, sometimes I give talks @ricardclau / Gmail [email protected]
- 3. WHAT IS REDIS? • REmote • Created • Open DIctionary Server in 2009 by Salvatore Sanfilipo (@antirez) source • Advanced in-memory key-value data-structure server • Part of the NoSQL movement • Stop thinking SQL, think back to structures!
- 4. TO ME REDIS IS LIKE...
- 5. REDIS EVERYWHERE? Redis is not a silver bullet... so let´s rephrase that with... The amazing story on how we gradually introduced Redis in a social videogaming company... and it saved us!
- 6. AGENDA • REDIS introduction • Getting started with PHP / Symfony2: sessions, caching • Recipes and comparison with other technologies • Sharding data. And some criticism! • War Stories from a 7.5M DAU project • Final thoughts
- 7. INTRODUCTION TO REDIS And some demystifications
- 8. DATA TYPES • Strings • Lists up to 512 Mb of elements sorted by insertion order (max 2^32 - 1) • Sets unordered collection of elements supporting unions, intersections and differences (max 2^32 - 1) • Hashes key-value pairs (max 2^32 - 1) • Sorted sets automatically ordered by score (max 2^32 - 1)
- 9. ONLY IN-MEMORY? • Your data needs to fit in your memory has configurable persistence: RDB snapshots, AOF persistence logs, combine both or no persistence at all • It • Think like memcached on steroids with persistence • http://redis.io/topics/persistence • “Memory is the new disk, disk is the new tape”
- 10. INTERESTING FEATURES • Master-slave • Pipelining replication to improve performance • Transactions • Publisher • Scripting (kind of with MULTI ... EXEC) / Subscriber with LUA (~stored procedures) • Predictable performance (check commands complexity)
- 11. HARDWARE TIPS is single-threaded, so a 2 core machine is enough • Redis using EC2, maybe m1.large (7.5Gb) is the most suitable, but if it is not enough you might consider any m2 memory optimized type • If • Amazon • CPU • Read ElastiCache finally supports Redis! is rarely the bottleneck with Redis carefully the doc to maximize performance!
- 12. NEW IN REDIS 2.8 • CONFIG REWRITE and IPv6 support • Improvements in scripting and key expiring algorithm • Keyspace notifications via PUB / SUB. (~Triggered events) • Partial resynchronization with slaves • Better •A consistency support (allow writes only when N slaves) fully rewritten Redis Sentinel
- 13. ALSO IN 2.8 (H|Z|S)?SCAN • Ability to iterate the data structures • SCAN cursor [MATCH pattern] [COUNT count] • (H|Z|S)SCAN • Cursor • Count • When key cursor [MATCH pattern] [COUNT count] is kind of a pointer that we send back and forth is NOT the number of elements but the complexity using MATCH some steps in iteration may return nil
- 14. GETTING STARTED WITH PHP It is easier than you may think!
- 15. PHP CLIENTS • Predis (PHP) vs phpredis (PHP extension) • Predis is very mature, actively maintained, feature complete, extendable, composer friendly and supports all Redis versions • Phpredis is faster, but not backwards compatible • Network latency is the biggest performance killer • http://redis.io/clients
- 16. REDIS-CLI AND MONITOR • Redis-cli to connect to a Redis instance • Experiment with http://redis.io/commands • With Monitor we can watch live activity!
- 17. SESSIONS, CACHING • Super-easy to implement with commercial frameworks used: GET <session-id>, SETEX <session-id> <expires> <serialized-data> • Commands • Fast performance and also persistent! • Caching • Be and temp-data storage work exactly equal careful with temp-data storage and memory usage!
- 18. SYMFONY2 INTEGRATION • There must be a bundle for that... • https://github.com/snc/SncRedisBundle of the box: Session management, Monolog handler, Swiftmailer spooling, Doctrine caching • Out • Full integration with Symfony2 profiler
- 20. COOKBOOK And comparisons with other technologies
- 21. REDIS AS A QUEUE • Using • We Lists and LPUSH / RPOP instructions can have different languages accessing data used that to send OpenGraph requests to Facebook (REALLY slow API) with Python daemons • We • ~80k • If messages/minute processed with 1 m1.large EC2 you need some advanced message queuing features check RabbitMQ (and Álvaro Videla´s talk) and others
- 22. REAL-TIME ANALYTICS • We can use hashes to group many stats under one key • Most of the times we have counters: HINCRBY “stats" <key> <increment> • HMGET “stats” <key1> <key2> ... to retrieve values and HMSET “stats” “stat1” <value1> “stat2” <value2> to set them • HGETALL to get the full hash
- 23. LEADERBOARDS • Super-easy • Each with Redis, heavy consuming with other systems board with a single key, using sorted sets • ZINCRBY “rankXXX” <increment> <userId> • Top N ZREVRANGE “rankXXX” 0 N [WITHSCORES] • We stored several milions of members under 1 key
- 24. WHO IS ONLINE? (I) can use SETS to achieve it! • We • One set for every minute, SADD <minute> <userId> Time +1m 5 +2m 2 +3m 1 4 2 1 5 4 1 3 SUNION 1 7 3 1 2 3 7 2
- 25. WHO IS ONLINE? (II) • Online users == everyone active in the last N minutes • SUNION <now> <now-1> <now-2> ... • SUNIONSTORE “online” <now> • Number • Obtain <now-1> <now-2> ... of users: SCARD “online” online users with SMEMBERS “online”
- 26. FRIENDS ONLINE? • If we store user´s friends in a SET with key friends-<user-id> online: SINTERSTORE “friends-<userid>-online” “online” “friends-<userid>” 10 4 2 1 3 7 SINTER ONLINE 5 1 9 3 7 15 FRIENDS-12-ONLINE • Imagine how you would do it without Redis! FRIENDS-12 • Friends
- 27. DISTRIBUTED LOCKING (I) • Problem in heavy-write applications: 2 almost concurrent requests trying to update same records • SELECT FOR UPDATE? Easy to create deadlocks! R1read R1 updates are lost!!!!! R2read timeline R1 R2 R1save R2save
- 28. DISTRIBUTED LOCKING (II) • Locking • SET • And •8 can be solved with Redis with Exclusive SET <lock> 1 NX EX <seconds> keep retrying until a specific timeout m1.xlarge instances to lock 7.5M Daily Active Users Zookeeper, perhaps better for scaling or if we need different locking patterns • Alternatives: Apache
- 29. SHARDING DATA Hopefully, you get there at some point Resharding: a painful experience
- 30. NEEDS TO FIT IN MEMORY • Redis • Proxy Cluster will be available in 3.0 (delayed many times) servers: Twemproxy (also for memcached) • Client-side • Sharding partitioning (supported in many clients) strategies (range vs hash partitioning) • Presharding • Memory can help scaling horizontally is cheaper every day but this is a real need!
- 31. 100000-199999 id % 4 == 0 id % 4 == 1 200000-299999 300000-399999 id % 4 == 2 id % 4 == 3 1-99999 Easy to predict scale Load not properly distributed Load distributed ok Fairly easy to reshard RANGE PARTITIONING May work in some cases, not a silver bullet
- 32. HASH hash(key) % 4 == 0 hash(key) % 4 == 1 CRC16 CRC32 MD5 SHA1 MURMUR3 DIST hash(key) % 4 == 2 hash(key) % 4 == 3 MODULUS KETAMA Distribution also depending on hash algorithm Complicated resharding HASH PARTITIONING Better distribution... but still issues If few keys, it also becomes useless
- 33. Many Redis instances in one initial server When needed, just split it into more servers and you don´t need resharding!!! Be careful configuring memory limits PRESHARDING Maybe overengineered
- 34. CRITICISM IN THE NET Sometimes unfair... sometimes they have a point
- 35. VS OTHER DATABASES • Cassandra or RIAK reshard automatically (at a big cost) • Complicated to fully automate failovers (Sentinel problems) • Severe • Tricky • All issues when slaves get out of sync, getting better to inspect big sets and hashes prior to 2.8 NoSQL have issues, check aphyr.com “Call me maybe”
- 36. ABOUT REDIS CLUSTER • CAP Theorem: Consistency, Availability, Partition • Purists • CAP tolerance blame it to not properly respect the principles theorem is... a theorem and it comes with a cost • The goal is to provide some consistency and some failure resistance without sacrificing other practical qualities • We could debate for hours around that point!
- 37. WAR STORIES https://apps.facebook.com/dragoncity/ (~7.5M DAU Facebook Game)
- 38. PROGRESSIVE USAGE • DragonCity • First had ~2M DAU when we introduced Redis usages: Queues PHP-Python and Temporary storage • Introduced • Used Locking with Redis -> End of deadlocks for temporary contests involving rankings • Session • About tracking, cheat detection and many others 7.5M DAU and similar amount of servers
- 39. REDIS IS SUPER-FAST • Your network gets killed much before Redis is down • Always increase your ulimit values in Redis servers • We had Gigabit network connectivity between machines but it was not enough to handle the data sent and retrieved • Redis • Do was just at 20% of CPU some heavy-load tests before launching!
- 40. SURPRISES WITH MEMORY documentation: “Sets of binary-safe strings”so, we changed values in sets from being just numbers to number + :<char> to add some info to values • Redis • We expected ~20% memory increase but in was 500% • https://github.com/antirez/redis/blob/unstable/src/t_set.c#L55 • If value can be represented as long, it is stored more efficiently • http://alonso-vidales.blogspot.co.uk/2013/06/not-all-redis- values-are-strings.html
- 41. FINAL THOUGHTS When and when not to consider Redis
- 42. REDIS IS PERFECT FOR... • Intensive read-write data applications • Temporary • Data stored data that fits in memory • Problems that fit Redis built-in data types • Predictability in performance needed (all commands complexity documented)
- 43. BUT IS NOT SUITABLE WHEN.. • Big data sets, archive data • Relational • We data (RDBMS are absolutely fine to scale) don´t know how we will access data • Reporting • It applications (no where clauses) gets tricky for distributed applications (replication, clustering) • ALWAYS choose the right tool for the job!
- 44. SPECIAL THANKS • Ronny López (@ronnylt) & Alonso Vidales (@alonsovidales) • All backend and systems engineers at @socialpoint • Of course, to all of you for being here today! our open-source project Redtrine, PR welcome: https://github.com/redtrine/redtrine • Check
- 45. QUESTIONS? • Twitter: @ricardclau • E-mail: [email protected] • Github: https://github.com/ricardclau • Blog about PHP and Symfony2: http://www.ricardclau.com • Please, rate this talk at https://joind.in/talk/view/10531