RedisConf17 - Roblox - How Roblox Keeps Millions of Users Up to Date with Redis Pub/Sub
Download as PPTX, PDF2 likes1,401 views
Roblox is a social gaming platform with millions of users that uses Redis and a custom Redis Pub/Sub backplane to deliver real-time messages at scale. The system handles over 1 million concurrent connections and delivers 20,000 messages per second. It uses Redis for caching, rate limiting, and as a datastore. The system was designed for best effort delivery without guarantees and limits notifications by user, connection, and message size. It has been successfully scaled through capacity planning and automated testing.
Ad
More Related Content
What's hot (20)
Similar to RedisConf17 - Roblox - How Roblox Keeps Millions of Users Up to Date with Redis Pub/Sub (20)
Ad
More from Redis Labs (20)
Ad
Recently uploaded (20)
RedisConf17 - Roblox - How Roblox Keeps Millions of Users Up to Date with Redis Pub/Sub
- 1. Peter Phillips | [email protected] How Roblox keeps millions of users up to date with Redis Pub/Sub
- 2. The world’s largest social platform for play Roblox
- 3. The world’s largest social platform for play Roblox - What
- 6. • 48 million monthly active users • 1.1 million peak concurrent users • 420 million monthly engagement hours • 5.9 billion monthly page views • 201 percent year over year growth in active users Roblox Scale at a Glance
- 8. • General Purpose LRU Cache • Rate-Limiting • Product Data Store • Message Relay How We Use Redis at Roblox
- 10. Redis Labs Enterprise Cluster
- 11. Open Source .NET Redis Client https://github.com/StackExchange/StackExchange.Redis StackExchange.Redis
- 12. A New Chat System
- 13. The Importance of Chat
- 14. • Scalable • Quick to Update • Efficient to Update The Non-Functional Requirements of Chat
- 15. A Real-Time Message Delivery System
- 16. With Use Beyond Chat A Real-Time Message Delivery System
- 17. Introduction to Presentation Edge Servers to Clients
- 18. An open source protocol and implementation of two way server client web communication • signalr.net • github.com/SignalR/SignalR ASP.NET Server Javascript, .NET and other clients SignalR
- 19. Message Delivery from Other Services, Multi-Host Support Choosing a Backplane
- 20. A Custom Redis Pub/Sub Backplane
- 21. Redis Pub/Sub
- 24. A Real-Time Message Delivery System
- 35. Operating At Scale - Design
- 36. One User, One Connection
- 38. Well… okay, one device, one connection One User, One Connection
- 45. Rate Limiting Long Lived Connections One User, Some Connections – But Not Too Many
- 46. No Guaranteed Delivery Best Effort Delivery
- 48. No Guaranteed Delivery Best Effort Delivery
- 49. Introduction to Presentation Limiting the Size of Notifications
- 51. Introduction to Presentation Limiting the Breadth of Notifications
- 53. Assuming the conversation has three participants: User1, User2, User3
- 54. One Way Traffic Down a Two Way Street Websockets: Server to Client Communication Only
- 56. One Way Traffic Down a Two Way Street Websockets: Server to Client Communication Only
- 57. Operating At Scale - Testing
- 58. Easier to test when nobody cares Silent Mode
- 59. Number of subscriptions, message throughput Redis Subscription Density
- 60. How many concurrent connections can a single server maintain Web Server Websocket Connection Density
- 61. When and where to add capacity Scaling Model
- 62. Unit and Integration Tests, Monitoring with Alerts Automated Tests
- 63. Operating At Scale – The Results
- 64. Increased Users and Increased Message Rates New Chat System
- 65. • Launched at around 150K connections • Now at over 1M connections • 5 Redis Pub/Sub servers • 27 Web servers for Websockets • 20K messages delivered per second The Real-Time System
- 66. Operating At Scale – Things to Watch For
- 67. Wait, how many users? One User, One Connection
- 68. Don’t Overstate Things One Connection, No User?
- 69. Come back soon One User, One Connection
- 71. Pushing Redis Pub/Sub Further
- 72. Distributed Cache Invalidation • A local cache on every machine • Every invalidation broadcast to all machines • Reaching a saturation point • Cascading Failures A Problem
- 73. Redis Pub/Sub to the Rescue? • Applied the lessons learnt from the Real-Time project • Tested the scale • Proposed a similar solution • Switch an entity at a time A Potential Solution
- 74. • 24 Redis Pub/Sub servers • Over 1000 application servers • Servers subscribe to entity topics • Publishers publish to one server, subscribers listen to all • 35M messages delivered per second Redis Pub/Sub in Action
- 75. http://grnh.se/g1amyx1 https://corp.roblox.com Peter Phillips | [email protected] How Roblox keeps millions of users up to date with Redis Pub/Sub
Editor's Notes
- #3: Roblox is the world’s largest social platform for play. Every month, over 48 million players imagine, build, and play together within immersive 3D worlds.
- #4: Everything in Roblox is user-generated. Our growing community of 1.7 million creators – most aged between 13 and 20 - produce millions of unique 3D multiplayer experiences using Roblox Studio - our desktop design tool. It’s the reason we’re called “The Imagination Platform.”
- #5: When you make a creation and click publish, it is available to play on Android, iPhones, iPads, PCs, Macs, Xbox One, with support for VR with Oculus Rift and Vive. This makes it easy for our users to play together, regardless of what device or platform they have. And for creators, it’s an incredibly accessible way to create a game, and instantly make it available to millions of players
- #9: Quickly become one of the most important tools Over 400 instances in production use
- #10: Open Source! Easy to spin up Great for ephemeral data
- #11: Additional Resiliency Replication Rebalancing Ease of Creating Isolated Databases – Fits our microservices architecture
- #12: Used throughout the application Wrapped and adapted to our needs. Works very well
- #14: Virality – WOM Encourage communication, building a network of friends, Co-ordinating activity Roblox Chat Design Conversations with friends Visibility into friend presence – who’s online, who’s in game, what game are they playing Driver of the party system Cross Platform
- #15: Fast Scalable Efficient Updating Looked at options such as XMPP, and chat-as-a-service Didn’t find any that fully satisfied our needs Chat has strong integration with the rest of our system Presence Moderation Privacy Rules Apart from specific product requirements
- #18: Pros and cons of different mechanisms of pushing data to clients Definitely Didn’t want to use short polling like the old system Websockets seemed like the ideal candidate – designed for long lived connections with data sent on demand New technology for us Worried about support for older clients
- #19: Connection management Marshalling of RPC calls in both directions Abstraction over the specific communication channel - with support for websockets - as well as server sent events and long polling Refined a C++ client for use across all our client applications
- #20: SignalR has several supported backplane options – including Redis They didn’t match our usage pattern, Required distributing all messages to all server hosts
- #21: We decided to build our own backplane, using Redis Pub/Sub
- #22: Allows a redis client to subscribe to a topic, or a pattern of topics. A redis client can then publish a message to a topic, and any clients subscribed to a matching topic will receive the message
- #26: Created a user based channel system. Every message sent is for a specific user User 1 comes online
- #27: On website, the Page loads, or if on mobile the Roblox app starts RealTime client attemps to connect to a realtime server. The loadbalancer directs the request to an available server
- #28: The server then subscribes to the channel for
- #37: Each user is tied to a specific Redis instance A message for a user needs to be published to only one redis node
- #38: Each user is tied to a specific Redis instance A message for a user needs to be published to only one redis node Because each message only needs to be delivered to one known redis node, we can scale out the number of Redis nodes horizontally as required to distribute the load With one connection per real-time web server, when the websocket connection density gets too great, we can scale that out horizontally to reduce load
- #39: One user, one connection – might not be completely achieveable But can we at least do one DEVICE, one connection?
- #40: One user, one connection – might not be completely achieveable But can we at least do one DEVICE, one connection?
- #41: One user, one connection – might not be completely achieveable But can we at least do one DEVICE, one connection?
- #42: One user, one connection – might not be completely achieveable But can we at least do one DEVICE, one connection?
- #43: One user, one connection – might not be completely achieveable But can we at least do one DEVICE, one connection?
- #44: One user, one connection – might not be completely achieveable But can we at least do one DEVICE, one connection?
- #45: One user, one connection – might not be completely achieveable But can we at least do one DEVICE, one connection?
- #46: Rate Limiting
- #47: - Determining the acceptable loss rate of data of different systems and finding the right balance between guaranteed delivery and cost to implement Not a source of truth Decided it would be a “Best Effort” channel, not guaranteed delivery Delayed messages are not useful – may in fact be detrimental Some missed messages may not matter However, the impact of a single missed notification can be significant For this reason, we use a sequence number system. Sequence number is incremented per user with each message redis publish attempted Built this into the reconnect logic of clients so they can detect when a message has been missed Also has the advantage of knowing we Haven’t missed anything after a reconnect
- #48: If a user is not connected to the Real-Time system when a message is sent, it is dropped The Publish command returns the number of subscribers a message was delivered to, so we can use this as a rough proxy of immediate delivery rates
- #49: We have resilient, guaranteed delivery systems, but didn’t think it was a fit here For instance, the Notification Stream Delayed messages are not useful – may in fact be detrimental Some missed messages may not matter However, the impact of a single missed notification can be significant For this reason, we use a sequence number system. Sequence number is incremented per user with each message redis publish attempted Built this into the reconnect logic of clients so they can detect when a message has been missed Also has the advantage of knowing we Haven’t missed anything after a reconnect Message publishers can always retry if they wish to
- #51: Notice that the content of the message is not passed in the Redis message Real-Time system is general purpose, reduce the traffic volume there. Because all messages are sent through – event if the client doesn’t necesarilly need them, don’t want to over-burden them. == May need to revisit this in the future if the volume is too high, but this helps at the moment = not a source of truth Easier to enforce that people don’t pass huge payloads Also has security benefits. Security and cross origin rules not as tight for websockets As messages are passed around between tabs and local storage we don’t need to worry about any sensitive information being passed around or left behind where it shouldn’t be Take privacy very seriously – knowing there is nothing sensitive in realtime system makes things easier
- #53: This was the example shown, from the perspective of User1. But that’s not necesarilly the only Real-Time message send as a result of that new message Let’s assume the conversation contains three users: User1, User2 and User3
- #54: We actually publish three messages Don’t publish to a group channel, ie. One based on the the conversation id, because we don’t want subrscribers having to subscribe to a large (possibly unbounded) number of topics A user can be in hundreds of conversations, any one of which could receive a message at any time Not to mention other uses Would also mean at the real-time web server level we would need to deliver the message to an unbounded number of clients. This could definitely be useful in the future, be we may want to think about how to do that Or have a background processor iterating over all active users according to the presence system and delivering them a message
- #55: Fit into our micro-services architecture - One user, one connection Websocket Security - Useful patterns for determining ownership of responsibilities between different communication channels
- #57: Fit into our micro-services architecture - One user, one connection Websocket Security Useful patterns for determining ownership of responsibilities between different communication channels We already have a pattern for requesting data client to server – REST apis We don’t want a second parallel method of accessing data – not ready to transition over entirely Would break down our pattern of having microservices with tightly defined areas of responsibility
- #59: Usual Strategy Common Metrics Percentage Rollouts
- #62: Not just requests/sec Concurrent Established Connections Monitor per server
- #63: Unit tests to ensure messages are being triggered when they should be Mocked StackExchange Redis implementation Integration testing on the APIs themselves on our staging and test environments Monitoring of production servers – connection rates, connection counts, message throughput, etc..
- #65: A big success Not unexpected as the old system had deficiencies Exact numbers are rough But a significant multiple of old usage, both in - the proportion of our user base chatting - the number of messages they were sending
- #66: Launched at 150k users – web only, mobile to come Goal of easy scaling to all users – 250-300K. With ability to go easily to double
- #68: So you have a scaling model, based on the number of the users, lets you calculate how many servers you needs But how many users do you really have?
- #69: Don’t bring zombie users back to life – they are now going to be doing things the didn’t before. Don’t’ mistake that for signs of life Don’t get in an infinite refresh loop – watch for and avoid feedback loops
- #70: Balancing the need to allow clients to reconnect quickly after an outage with not flooding our servers with reconnects Reconnect rates Connection Churn Keep up with Keep Alives
- #71: Monitoring request execution time gets tricky
- #72: We think we can push it much further As the redis layer is less utilized at the moment planning to increase its use Maybe you think the 20K msg/sec of realtime is unimpressive – can it be pushed further, much further? I was asked that same question
- #76: That’s how we keep our millions of users up to date, both directly in the UI, and ensuring our servers are serving up correct data, with the help of Redis Pub/Sub Thanks for listening, I hope you got something out of it We also achieve scale with great effort of the engineering team at Roblox. Easy to spin up servers