Redis & ZeroMQ: How to scale your application

Follow this topic:
@rjsmelo
Redis & ZeroMQ: How to
scale your application
RICARDO MELO
Presented at #PHPLX – 11 July 2013

@rjsmelo 2
RICARDO MELO
● CTO @ DRI
● PHP, Mysql, Linux and lots of other
OSS
● ZCE, RHCE, LPI 3, ITIL, etc
● +10 years building (and breaking)
things

@rjsmelo 3
About
● 14 Year old academic spin-off
● Pragmatic OSS Orientation
● PHP, Mysql, SugarCRM, Drupal,
JavaScript, Linux, etc.
● Crafters, Integrators
● Always looking for software developers
– Yes, right now!

1999 - 2013 DRI. Some Rights Reserved. 4
Outline
● Redis
● ZeroMQ
● Use Cases
● Conclusions

Redis
“Redis is an open source, BSD licensed,
advanced key-value store. It is often
referred to as a data structure server
since keys can contain strings, hashes,
lists, sets and sorted sets.”
source: http://redis.io

Predis
● Pure PHP Redis Client
– https://github.com/nrk/predis
● Use composer:
– composer install
{
"name": "rjsmelo/talk-redis-zmq",
"description": "Sample code for Redis & ZeroMQ presentation",
"require": {
"ext-zmq": "*",
"predis/predis": "dev-master"
},
"license": "Apache-2.0",
"authors": [
{
"name": "Ricardo Melo",
"email": "ricardo.melo@dri.pt"
}
]
}

Strings
1 <?php
2 //redis_strings.php
3 require_once __DIR__ . '/../vendor/autoload.php';
4 $redis = new PredisClient('tcp://localhost:6379');
5
6 // Using SET
7 $redis->set("SampleKey", "Throw me any thing here....");
8 echo $redis->get("SampleKey") . "n"; // Throw me any thing here....
9
10 // Remove Key
11 $redis->del("SampleKey");
12
13 // Using APPEND
14 $redis->append("SampleKey", "Hello"); // Hello
15 $redis->append("SampleKey", " World!"); // Hello World!
16 echo $redis->get("SampleKey") . "n";
17
18 // Other commands: incr, decr, incrby, getrange, setrange
19 // http://redis.io/commands/#string

Lists
1 <?php
2 //redis_lists.php
5
6 $redis->del('SampleList');
7
8 // Using {L,R}PUSH
9 $redis->lpush('SampleList', 'a'); // a
10 $redis->lpush('SampleList', 'b'); // b, a
11 $redis->rpush('SampleList', 'c'); // b, a, c
12
13 // Using LLEN
14 echo $redis->llen('SampleList') . "n"; // 3
15
16 // Using LINDEX (note: zero indexed)
17 echo $redis->lindex('SampleList', 1) . "n"; // a
18
19 // Using {L,R}POP
20
21 echo $redis->lpop('SampleList') . "n"; // b
22 echo $redis->rpop('SampleList') . "n"; // c
23
24 // Other commands: lrange, rpoplpush
25 // http://redis.io/commands#list

Sets
1 <?php
2 //redis_sets.php
5
6 $redis->del('SampleSet');
7 $redis->del('OtherSampleSet');
8
9 // Using SADD
10 $redis->sadd('SampleSet', 'Hello'); // Hello
11 $redis->sadd('SampleSet', 'World'); // Hello, World
12 $redis->sadd('SampleSet', 'World'); // Hello, World
13
14 // Using SMEMBERS
15 var_dump($redis->smembers('SampleSet')); // Hello, World
16
17 // Using SINTER
18 $redis->sadd('OtherSampleSet', 'Hello');
19 $redis->sadd('OtherSampleSet', 'All');
20 var_dump($redis->sinter('SampleSet', 'OtherSampleSet')); // Hello
21
22 // Using SUNION
23 var_dump($redis->sunion('SampleSet', 'OtherSampleSet')); // Hello, World, All
24
25 // Other commands: smove, srandmember, srem, scard
26 // http://redis.io/commands#set

Hashes
1 <?php
2 //redis_hashes.php
5
6 $redis->del('SampleHash');
7
8 // Using HMSET
9 $redis->hmset("SampleHash", array(
10 'prop_a' => 'aaa',
11 'prop_b' => 'bbb',
12 ));
13
14 // Using HGETALL
15 var_dump($redis->hgetall("SampleHash")); // prop_a=>aaa, prop_b=>bbb
16
17 // Using HSET
18 $redis->hset('SampleHash', 'prop_b', 'ccc');
19
20 //USING HGET
21 echo $redis->hget("SampleHash", 'prop_b') ."n"; // ccc
22
23 // Other commands: hexists, hdel, hlen, hkeys, hvals
24 // http://redis.io/commands/#hash

Sorted Sets
1 <?php
2 //redis_sorted_sets.php
5
6 $redis->del('SampleSortedSet');
7
8 // Using SADD
9 $redis->zadd('SampleSortedSet', 1, 'Hello'); // Hello(1)
10 $redis->zadd('SampleSortedSet', 2, 'World'); // Hello(1), World(2)
11 $redis->zadd('SampleSortedSet', 3, 'World'); // Hello(1), World(3)
12
13 // Using ZRANGE
14 var_dump($redis->zrange('SampleSortedSet', 0, -1,
array('withscores'=>true))); // Hello(1), World(3)
15
16 // Using ZSCORE
17 echo $redis->zscore('SampleSortedSet', 'World') . "n"; // 3
18
19 // Other commands: zrank, zrevrange, zrangebyscore, zremrangebyscore
20 // http://redis.io/commands#sorted_set

ZeroMQ
“ØMQ is a high-performance asynchronous
messaging library aimed at use in scalable
distributed or concurrent applications. It provides a
message queue, but unlike message-oriented
middleware, a ØMQ system can run without a
dedicated message broker. With bindings for 30+
languages”
source: https://en.wikipedia.org/wiki/%C3%98MQ

PHP Module - zmq
#
# ZMQ instalation - http://www.zeromq.org/intro:get-the-software
#
tar xzvf zeromq-3.2.2.tar.gz
cd zeromq-3.2.2
./configure
make
make install
echo "/usr/local/lib" > /etc/ld.so.conf.d/usr_local_lib.conf
ldconfig
#
# ZMQ PHP Binding Instalation
#
pear channel-discover pear.zero.mq
pecl install pear.zero.mq/zmq-beta
echo "extension=zmq.so" > /etc/php.d/zmq.ini

Socket Types
● Threads in a process (inproc://)
● Processes in a box (ipc://)
● Processes over the network (tcp://)
● Multicast group (pgm://)

Pattern: Request - Reply

Pattern: Request - Reply
1 <?php
2 // zmq_request.php
3 $context = new ZMQContext();
4
5 $requester = new ZMQSocket($context, ZMQ::SOCKET_REQ);
6 $requester->connect("tcp://localhost:5555");
7
8 for ($number = 0 ; $number <= 10 ; $number++) {
9 $mensage = "Hello " . $number . "!";
10 echo "Sending - " . $mensage . "n";
11 $requester->send($mensage);
12 $reply = $requester->recv();
13 echo "Received - " . $reply . "n";
14 }
1 <?php
2 //zmq_reply.php
4
5 $responder = new ZMQSocket($context, ZMQ::SOCKET_REP);
6 $responder->bind("tcp://*:5555");
7
8 while (true) {
9 $request = $responder->recv();
10 echo $request . "n";
11 sleep (1); // some work
12 $responder->send("Reply to: " . $request);
13 }

Pattern: Publish - Subscribe

Pattern: Publish - Subscribe
1 <?php
2 // zmq_publisher.php
4
5 $publisher = $context->getSocket(ZMQ::SOCKET_PUB);
6 $publisher->bind("tcp://*:5556");
7
8 $timezones = array('UTC', 'EST');
9 while (true) {
10 foreach($timezones as $tz) {
11 date_default_timezone_set($tz);
12 $message = date('T:c'); // the message to broadcast
13 $publisher->send($message);
14 }
15 sleep(1);
16 } 1 <?php
2 // zmq_subscriber.php
4
5 $subscriber = $context->getSocket(ZMQ::SOCKET_SUB);
6 $subscriber->connect("tcp://localhost:5556");
7
8 $filter = $_SERVER['argc'] > 1 ? $_SERVER['argv'][1] : "UTC";
9 $subscriber->setSockOpt(ZMQ::SOCKOPT_SUBSCRIBE, $filter);
10
11 while (true) {
12 $message = $subscriber->recv();
13 echo substr($message,4) . "n"; // remove prefix
14 }

Pattern: Pipeline

20
Pattern: Pipeline
1 <?php
2 // zmq_ventilator.php
3 // extracted from: http://zguide.zeromq.org/
4
6
7 // Socket to send messages on
8 $sender = new ZMQSocket($context, ZMQ::SOCKET_PUSH);
9 $sender->bind("tcp://*:5557");
10
11 echo "Press Enter when the workers are ready: ";
12 $fp = fopen('php://stdin', 'r');
13 $line = fgets($fp, 512);
14 fclose($fp);
15 echo "Sending tasks to workersâÀ¦", PHP_EOL;
16
17 // The first message is "0" and signals start of batch
18 $sender->send(0);
19
20 // Send 100 tasks
21 $total_msec = 0; // Total expected cost in msecs
22 for ($task_nbr = 0; $task_nbr < 100; $task_nbr++) {
23 // Random workload from 1 to 100msecs
24 $workload = mt_rand(1, 100);
25 $total_msec += $workload;
26 $sender->send($workload);
27
28 }
29
30 printf ("Total expected cost: %d msecn", $total_msec);
31 sleep (1); // Give 0MQ time to deliver

Pattern: Pipeline
1 <?php
2 // zmq_worker.php
4
6
7 // Socket to receive messages on
8 $receiver = new ZMQSocket($context, ZMQ::SOCKET_PULL);
9 $receiver->connect("tcp://localhost:5557");
10
11 // Socket to send messages to
12 $sender = new ZMQSocket($context, ZMQ::SOCKET_PUSH);
13 $sender->connect("tcp://localhost:5558");
14
15 // Process tasks forever
16 while (true) {
17 $string = $receiver->recv();
18
19 // Simple progress indicator for the viewer
20 echo $string, PHP_EOL;
21
22 // Do the work
23 usleep($string * 1000);
24
25 // Send results to sink
26 $sender->send("");
27 }

22
Pattern: Pipeline
1 <?php
2 // zmq_sink.php
4
5 // Prepare our context and socket
7 $receiver = new ZMQSocket($context, ZMQ::SOCKET_PULL);
8 $receiver->bind("tcp://*:5558");
9
10 // Wait for start of batch
12
13 // Start our clock now
14 $tstart = microtime(true);
15
16 // Process 100 confirmations
17 $total_msec = 0; // Total calculated cost in msecs
18 for ($task_nbr = 0; $task_nbr < 100; $task_nbr++) {
20 if ($task_nbr % 10 == 0) {
21 echo ":";
22 } else {
23 echo ".";
24 }
25 }
26
27 $tend = microtime(true);
28
29 $total_msec = ($tend - $tstart) * 1000;
30 echo PHP_EOL;
31 printf ("Total elapsed time: %d msec", $total_msec);
32 echo PHP_EOL;

Pattern: Pipeline

Use Case: Service Cluster
Reference: http://zguide.zeromq.org/page:all#Service-Oriented-Reliable-Queuing-Majordomo-Pattern

Use Case: “Unix Style” Application
● Lots of simple, “focused” programs
– Or in different servers, different languages, etc
● All components collaborate to get the
job done
– cat file | sort | uniq -c
● Glue everything with ZeroMQ

Use Case: Cache
● We need to cache some values for
speed
● Use SET / GET on Redis
● I could use memcache, but this is a
Redis Talk :-)
● They have similar performance
– http://redis.io/topics/benchmarks

Use Case: Buffer
● Use Redis as a FIFO (or LIFO)
● Hidden Goods
– Decoupling
– Flow control
– rpoplpush

Use Case: Background Tasks
● Your application needs to do some
heavy work
– Send Email
– Image Resizing
– Mega Huge map-reduce query to your pentabyte cluster :-)
● You don't want to keep your user
waiting
● Send things to background

Use Case: Background Tasks
● Use Redis as your Job Queue
– Your application should send job to be run and parameters to the
queue
● Use workers to POP jobs from the
queue and do the heavy work.
● Don't reinvent the well
– Look for php-resqueue

Conclusions
● Both ZeroMQ and Redis are extremely
fast
● ZeroMQ is great to “glue things” as well
as adding flexibility to scale dynamical
● Redis is great as a queue and allows
you to cope with your load.

Follow this topic:
@rjsmelo
QA
Code: https://github.com/rjsmelo/talk-redis-zmq
Feedback: https://joind.in/talk/view/8937

www.dri-global.com
@rjsmelo
ricardo.melo@dri-global.com

Redis & ZeroMQ: How to scale your application

Recommended

More Related Content

What's hot (20)

Viewers also liked (20)

Similar to Redis & ZeroMQ: How to scale your application (20)

More from rjsmelo (7)

Recently uploaded (20)

Redis & ZeroMQ: How to scale your application