Practical Message Queuing Using RabbitMQ (PHPem, 3rd July 2014)
6 likes2,980 views
The document provides a comprehensive guide on using RabbitMQ for practical message queueing, detailing installation, configuration, and various usage scenarios including producer-consumer setups. It explains different types of exchanges such as fanout, direct, and topic exchanges, as well as advanced concepts like clustering and high availability. The guide also presents real-world examples and solutions to common challenges, making it suitable for developers seeking to implement effective background processing solutions.
![fanout/consumer.php
$q = $channel->queue_declare(
'', // Lets RabbitMQ pick a name for queue
false, false, false,
true // Delete this queue
);
$queue_name = $q[0];
$channel->exchange_declare(
'test_exchange', 'fanout', false, false, false);
$channel->queue_bind($queue_name, 'test_exchange');](https://image.slidesharecdn.com/140703-practicalmessagequeueingusingrabbitmqphpemjulymeetup-140704020636-phpapp02/85/Practical-Message-Queuing-Using-RabbitMQ-PHPem-3rd-July-2014-29-320.jpg)
![direct/consumer.php
$q = $channel->queue_declare('', false, false, false, true);
$queue_name = $q[0];
$channel->exchange_declare(
'test_direct', 'direct', false, false, false);
// Bind for each routing key we want (BINDING KEY)
$channel->queue_bind($queue_name, 'test_direct', 'apple');
$channel->queue_bind($queue_name, 'test_direct', 'orange');
$channel->queue_bind($queue_name, 'test_direct', 'banana');](https://image.slidesharecdn.com/140703-practicalmessagequeueingusingrabbitmqphpemjulymeetup-140704020636-phpapp02/85/Practical-Message-Queuing-Using-RabbitMQ-PHPem-3rd-July-2014-35-320.jpg)
![Creating a cluster
node1$ rabbitmqctl cluster_status
Cluster status of node rabbit@node1 ...
[{nodes,[{disc,[rabbit@node1]}]},{running_nodes,[rabbit@node1]}]
...done.
node2$ rabbitmqctl cluster_status
Cluster status of node rabbit@node2 ...
[{nodes,[{disc,[rabbit@node2]}]},{running_nodes,[rabbit@node2]}]
...done.
node3$ rabbitmqctl cluster_status
Cluster status of node rabbit@node3 ...
[{nodes,[{disc,[rabbit@node3]}]},{running_nodes,[rabbit@node3]}]
...done.](https://image.slidesharecdn.com/140703-practicalmessagequeueingusingrabbitmqphpemjulymeetup-140704020636-phpapp02/85/Practical-Message-Queuing-Using-RabbitMQ-PHPem-3rd-July-2014-52-320.jpg)
![node2$ rabbitmqctl join_cluster --ram rabbit@node1
node3$ rabbitmqctl join_cluster rabbit@node2
node3$ rabbitmqctl cluster_status
Cluster status of node rabbit@node3 ...
[{nodes,[{disc,[rabbit@node3,rabbit@node1]},{ram,[rabbit@node2]}]},
{running_nodes,[rabbit@node2,rabbit@node1,rabbit@node3]}]
...done.
Creating a cluster](https://image.slidesharecdn.com/140703-practicalmessagequeueingusingrabbitmqphpemjulymeetup-140704020636-phpapp02/85/Practical-Message-Queuing-Using-RabbitMQ-PHPem-3rd-July-2014-53-320.jpg)
![node1$ rabbitmqctl stop_app
node2$ rabbitmqctl forget_cluster_node rabbit@node1
node1$ rabbitmqctl reset
node1$ rabbitmqctl start_app
node2$ rabbitmqctl cluster_status
Cluster status of node rabbit@node2 ...
[{nodes,[{disc,[rabbit@node3]},{ram,[rabbit@node2]}]},
{running_nodes,[rabbit@node2,rabbit@node3]}]
...done.
Managing Nodes
● Stopping/starting nodes
● Removing nodes:](https://image.slidesharecdn.com/140703-practicalmessagequeueingusingrabbitmqphpemjulymeetup-140704020636-phpapp02/85/Practical-Message-Queuing-Using-RabbitMQ-PHPem-3rd-July-2014-54-320.jpg)
Practical Message Queuing Using RabbitMQ (PHPem, 3rd July 2014)
- 1. Practical Message Queueing Using RabbitMQ James Titcumb PHPem 3rd July 2014
- 2. James Titcumb www.jamestitcumb.com www.protected.co.uk www.phphants.co.uk @asgrim Who is this guy?
- 6. Scaling with Rabbit RabbitMQApplication Background processing
- 7. Scaling with Rabbit RabbitMQApplication Background processing Background processing
- 8. Scaling with Rabbit RabbitMQApplication Background processing Background processing Background processing
- 9. Scaling with Rabbit RabbitMQApplication Background processing Background processing Background processing Background processing
- 10. Scaling with Rabbit RabbitMQApplication Background processing Background processing Background processing Background processing Background processing
- 11. ● Fast logging solution Some real world uses...
- 12. ● Fast logging solution ● Background Processing Some real world uses...
- 13. ● Fast logging solution ● Background Processing ○ Sending emails Some real world uses...
- 14. ● Fast logging solution ● Background Processing ○ Sending emails ○ Sending SMS Some real world uses...
- 15. ● Fast logging solution ● Background Processing ○ Sending emails ○ Sending SMS ○ Analytics, reporting Some real world uses...
- 16. (on precise64, other OSs may vary) Installing RabbitMQ
- 17. ● add apt repo ○ deb http://www.rabbitmq.com/debian/ testing main ● add signing key ○ http://www.rabbitmq.com/rabbitmq-signing-key-public.asc ● apt-get update ● apt-get install rabbitmq-server ● rabbitmq-plugins enable rabbitmq_management ● sudo service rabbitmq-server restart Using Apt
- 20. Objective: Basic Queuing Producer Consumer test_queue 1 2 3 4 5
- 21. composer.json { "require": { "videlalvaro/php-amqplib": "2.*" } } then composer install
- 22. Please wait, connecting... use PhpAmqpLibConnectionAMQPConnection; $connection = new AMQPConnection( 'localhost', 5672, 'guest', 'guest', '/' ); $channel = $connection->channel();
- 23. basic/producer.php use PhpAmqpLibMessageAMQPMessage; $channel->queue_declare( 'test_queue', false, true, false, false); $message = new AMQPMessage('my test message'); $channel->basic_publish($message, '', 'test_queue');
- 24. basic/consumer.php $channel->basic_consume( 'test_queue', // Queue to consume '', // Consumer identifier false, true, // No-ack means messages are "auto acknowledged" false, // Exclusive - no other consumers can use the queue false, function(AMQPMessage $message) { echo $message->body . "n"; } ); while (count($channel->callbacks)) { $channel->wait(); }
- 28. fanout/producer.php use PhpAmqpLibMessageAMQPMessage; $channel->exchange_declare( 'test_exchange', 'fanout', false, false, false); $message = new AMQPMessage('my test message #' . $id); $channel->basic_publish($message, 'test_exchange');
- 29. fanout/consumer.php $q = $channel->queue_declare( '', // Lets RabbitMQ pick a name for queue false, false, false, true // Delete this queue ); $queue_name = $q[0]; $channel->exchange_declare( 'test_exchange', 'fanout', false, false, false); $channel->queue_bind($queue_name, 'test_exchange');
- 31. A word on Temporary Queues ● Only the “current” flow ● Specific use cases test_exchangeProducer Messages go nowhere
- 33. Objective: Direct Exchange test_direct BK = apple BK = banana, apple Consumer Consumer Producer 3 Message Routing Keys 1 = orange 2 = banana 3 = apple 2 3 BK = orange, banana, apple Consumer1 2 3
- 34. direct/producer.php $channel->exchange_declare( 'test_direct', 'fanout', false, false, false); $messageContent = 'my test message, key=' . $routingKey; $message = new AMQPMessage($messageContent); $channel->basic_publish($message, 'test_direct', $routingKey);
- 35. direct/consumer.php $q = $channel->queue_declare('', false, false, false, true); $queue_name = $q[0]; $channel->exchange_declare( 'test_direct', 'direct', false, false, false); // Bind for each routing key we want (BINDING KEY) $channel->queue_bind($queue_name, 'test_direct', 'apple'); $channel->queue_bind($queue_name, 'test_direct', 'orange'); $channel->queue_bind($queue_name, 'test_direct', 'banana');
- 37. Exchanges: Topic
- 38. Objective: Topic Exchange test_topic BK = *.vegetable BK = # Consumer Consumer Producer 1 2 Message Routing Keys 1 = red.vegetable 2 = green.vegetable 3 = green.fruit 4 = red.meat 5 = green.grass.long 1 2 3 4 5 BK = green.# Consumer2 3 5 BK = *.grass.* / *.*.long Consumer5
- 41. Fetch message Logging Sequence ApplicationBrowser Log Server HTTP request JSON via AMQP Error! HTTP response RabbitMQ
- 42. Flexibility! ● Temporary Queues ○ e.g. inspect “debug” messages
- 43. Flexibility! ● Temporary Queues ● Queues to log to DB
- 44. Flexibility! ● Temporary Queues ● Queues to log to DB ● Queues to email “alert/emergency”
- 45. Flexibility! ● Temporary Queues ● Queues to log to DB ● Queues to email “alert/emergency” ● Get creative with routing keys ○ RK = app.api.error … BK = #.api.error ○ RK = app.form.debug … BK = #.debug
- 46. Problem: SPOF
- 48. Clustering RabbitMQ Node 1 RabbitMQ Node 3 RabbitMQ Node 2 RabbitMQ Node 4 RabbitMQ Node 5 RabbitMQ Node 6 Load Balance / Floating IP / Low TTL DNS etc.
- 49. Clustering ● Everything replicated, except queues
- 50. Clustering ● Everything replicated, except queues ● Types: ○ RAM ○ Disk
- 51. Clustering ● Everything replicated, except queues ● Types: ○ RAM ○ Disk ● Configuration: ○ CLI (rabbitmqctl) ○ Configuration files
- 52. Creating a cluster node1$ rabbitmqctl cluster_status Cluster status of node rabbit@node1 ... [{nodes,[{disc,[rabbit@node1]}]},{running_nodes,[rabbit@node1]}] ...done. node2$ rabbitmqctl cluster_status Cluster status of node rabbit@node2 ... [{nodes,[{disc,[rabbit@node2]}]},{running_nodes,[rabbit@node2]}] ...done. node3$ rabbitmqctl cluster_status Cluster status of node rabbit@node3 ... [{nodes,[{disc,[rabbit@node3]}]},{running_nodes,[rabbit@node3]}] ...done.
- 53. node2$ rabbitmqctl join_cluster --ram rabbit@node1 node3$ rabbitmqctl join_cluster rabbit@node2 node3$ rabbitmqctl cluster_status Cluster status of node rabbit@node3 ... [{nodes,[{disc,[rabbit@node3,rabbit@node1]},{ram,[rabbit@node2]}]}, {running_nodes,[rabbit@node2,rabbit@node1,rabbit@node3]}] ...done. Creating a cluster
- 54. node1$ rabbitmqctl stop_app node2$ rabbitmqctl forget_cluster_node rabbit@node1 node1$ rabbitmqctl reset node1$ rabbitmqctl start_app node2$ rabbitmqctl cluster_status Cluster status of node rabbit@node2 ... [{nodes,[{disc,[rabbit@node3]},{ram,[rabbit@node2]}]}, {running_nodes,[rabbit@node2,rabbit@node3]}] ...done. Managing Nodes ● Stopping/starting nodes ● Removing nodes:
- 55. Solution 2: HA
- 56. HA + Queue Mirroring RabbitMQ Node 1 RabbitMQ Node 2 Load Balance / Floating IP / Low TTL DNS etc.
- 58. Questions?