Redis as a message queue
9 likes8,733 views
Redis is being used as a message queue to asynchronously process image uploads on a website for gaming screenshots. When a user uploads images, the application server adds a message to the Redis queue containing metadata about the upload. A separate process polls the queue and processes each upload by resizing images, creating database entries, and more. This allows upload processing to happen in the background without blocking the user.
![Lets push!
// Create a new redis connection
$r = new Redis();
$r->connect($host, $port);
// Create the message array
$message = [‘uuid’ => $uuid, ‘userId’ => $userId, ‘gameId’ => $gameId];
// Get a new message id by implementing the builtin increment feature of Redis
$id = $r->incr('mmoscreens:screens-queue');
// Push the id into the queue (Redis list)
$r->lpush('mmoscreens:screens-queue', $id);
// Set the $message into Redis
$r->hmset("mmoscreens:screens-queue:$id", $message);
9Wednesday, October 9, 13](https://image.slidesharecdn.com/redisasamessagequeue-131009235751-phpapp02/85/Redis-as-a-message-queue-9-320.jpg)
![Lets pull!
// Create a new redis connection
$r = new Redis()->connect($host, $port);
// pop the message ID from the queue, or block until one is available
$id = $r->brPop('mmoscreens:screens-queue', BLOCK_WAIT_TIME);
// brPop returns [listName, element]
isset($id[1]) && $id = $id[1];
// Get the message from Redis
$message = $r->hgetall("mmoscreens:screens-queue:$id");
$message:
[
‘uuid’ => ‘fef4430c-e4a8-4874-b380-63edd2fa8472’,
‘userId’ => 10611,
‘gameId’ => 8
]
10Wednesday, October 9, 13](https://image.slidesharecdn.com/redisasamessagequeue-131009235751-phpapp02/85/Redis-as-a-message-queue-10-320.jpg)
Ad
More Related Content
What's hot (20)
Similar to Redis as a message queue (20)
Ad
Recently uploaded (20)
Ad
Redis as a message queue
- 1. Redis As a Message Queue 1Wednesday, October 9, 13
- 2. The Plug Essentially, a screenshot site MMORPG (Massively-Multiplayer, Online Role-Playing Game) niche market Site visitors browse games, games have albums Players can upload screenshots 2Wednesday, October 9, 13
- 3. The Architecture (2) 6-core CPU, 16GB RAM, 6-drive 3TB RAID10 (1) 8-core CPU, 16GB RAM, 2-drive 64GB RAID1 SSD, 2-drive 320GB RAID1 (2) layer-4 ipvsadm load balancers in master/backup failover via keepalived (3) PostgreSQL master/slave database (3) nginx+php-fpm (5.5) app servers Replicated file system via GlusterFS for images Memcached instance for php sessions Redis for caching and message queue Amazon Cloudfront for CDN (content delivery network) PhalconPHP framework, Gmagick, Twitter Bootstrap v3.0, REST API 3Wednesday, October 9, 13
- 4. Message Queue? Push messages into a stack Pop (pull) messages off a stack You put (push) data (messages) into the queue (stack) Messages are things you want to process later by pulling them from the queue 4Wednesday, October 9, 13
- 5. Messages? A “message” is just some data Could be an array Could be a string 5Wednesday, October 9, 13
- 6. The Use-Case When a user uploads images, it is bad to block the user while your server processes the photos Image resizing Creating multiple versions Creating database entries 6Wednesday, October 9, 13
- 7. The User Flow User selects images Images are uploaded to app server App server generates a UUID filename and copies the images to the replicated file system App server increments a Redis counter App server creates an array with uuid, user id, game id and pushes this “message” into the Redis list (message queue) 7Wednesday, October 9, 13
- 8. Batch Processing A separate PHP/other process running on any of the app servers (crontab, forking daemon) pops the Redis list to get a message By using GlusterFS as a replicated file system all app servers have all the uploads/images When a message is obtained from Redis queue, it processes the file upload, creates various image sizes Creates a database entry in PostgreSQL 8Wednesday, October 9, 13
- 9. Lets push! // Create a new redis connection $r = new Redis(); $r->connect($host, $port); // Create the message array $message = [‘uuid’ => $uuid, ‘userId’ => $userId, ‘gameId’ => $gameId]; // Get a new message id by implementing the builtin increment feature of Redis $id = $r->incr('mmoscreens:screens-queue'); // Push the id into the queue (Redis list) $r->lpush('mmoscreens:screens-queue', $id); // Set the $message into Redis $r->hmset("mmoscreens:screens-queue:$id", $message); 9Wednesday, October 9, 13
- 10. Lets pull! // Create a new redis connection $r = new Redis()->connect($host, $port); // pop the message ID from the queue, or block until one is available $id = $r->brPop('mmoscreens:screens-queue', BLOCK_WAIT_TIME); // brPop returns [listName, element] isset($id[1]) && $id = $id[1]; // Get the message from Redis $message = $r->hgetall("mmoscreens:screens-queue:$id"); $message: [ ‘uuid’ => ‘fef4430c-e4a8-4874-b380-63edd2fa8472’, ‘userId’ => 10611, ‘gameId’ => 8 ] 10Wednesday, October 9, 13