Asynchronous data processing
24 likes6,204 views
Choosing the right way to process data might become a strategic and non trivial decision for many kind of applications. Especially in applications where an high percentage of the time is spent elaborating information behind the scenes. There are different message queuing systems designed to manage and process data asynchronously. Using simple messages, it becomes possible to define many types of queue patterns, from the straightforward solution to the more complex one like routing, publisher/subscriber and topic. The purpose of the talk is to show how to approach the different scenarios in php, adding value to your application.
![S A F E T Y
M E S S A G E A C K S
$callback
=
function($msg){
$msg-‐>body-‐>generatePdf();
$msg-‐>delivery_info['channel']-‐>basic_ack($msg-‐>delivery_info['delivery_tag']);
};
!
$channel-‐>basic_consume('catalog',
'',
false,
false,
false,
false,
$callback);
S W I T C H A C K O N](https://image.slidesharecdn.com/asynchronousdataprocessing-140824123132-phpapp02/85/Asynchronous-data-processing-30-320.jpg)
Ad
More Related Content
What's hot (20)
Similar to Asynchronous data processing (20)
![php[world] 2016 - You Don’t Need Node.js - Async Programming in PHP](https://cdn.slidesharecdn.com/ss_thumbnails/phpworld2016-youdontneednode-161118143948-thumbnail.jpg?width=560&fit=bounds)
Ad
More from Andrea Giuliano (10)
Ad
Recently uploaded (20)
Asynchronous data processing
- 1. A S Y N C H R O N O U S D ATA P R O C E S S I N G A N D R E A G I U L I A N O @bit_shark # C A K E F E S T
- 2. T H E P R O B L E M
- 3. P R O D U C T C ATA L O G G E N E R AT O R • 30k+ records in db representing products • dispose records in pages (with a certain order) • generate a 300+ pages PDF catalog T H E P R O B L E M
- 4. H E AV Y O P E R AT I O N S T O D O H E R E T H E P R O B L E M
- 5. U S E R S D O N ’ T WA N T T O WA I T T H E P R O B L E M
- 6. U S E R S D O N ’ T WA N T T O WA I T T H E P R O B L E M A N D I F T H E Y H AV E T O T H E Y D O N ’ T WA N T T O G E T S T U C K !
- 7. T H E P R O B L E M S Y N C H R O N Y time process A process B Request Response blocked
- 8. T H E N E E D S
- 9. P R E V E N T T I M E O U T T H E N E E D S
- 10. E V E N T U A L LY D E L I V E RY T H E N E E D S “sooner or later your job will be processed.”
- 11. “ E V E R I T H I N G ’ S G O N N A B E A L R I G H T ” N O T I F I C AT I O N T H E N E E D S
- 12. T H E N E E D S A S Y N C H R O N Y time process A process B Request
- 13. I N T E R O P E R A B I L I T Y T H E N E E D S
- 14. A D VA N C E D M E S S A G E Q U E U I N G P R O T O C O L T H E S O L U T I O N
- 15. H O W D O E S I T W O R K A M Q P P R O D U C E R C O N S U M E R Produce Consumes B R O K E R
- 16. H O W D O E S I T W O R K A M Q P P R O D U C E R C O N S U M E R Produce Exchange Consumes B R O K E R
- 17. H O W D O E S I T W O R K A M Q P P R O D U C E R C O N S U M E R Produce Exchange Routes Consumes B R O K E R
- 18. H O W D O E S I T W O R K A M Q P P R O D U C E R C O N S U M E R Produce Exchange Queue Routes Consumes B R O K E R
- 19. T H R O U G H C O M P O S E R I N S TA L L A M Q P L I B R A RY { "require": { "videlalvaro/php-‐amqplib": "@stable", ... } } $ composer.phar install
- 20. D I F F E R E N T S C E N A R I O S
- 21. P R O D U C E R / C O N S U M E R S C E N A R I O
- 22. S C E N A R I O P R O D U C E R / C O N S U M E R P R O D U C E R C O N S U M E R Q U E U E
- 23. S C E N A R I O P R O D U C E R / C O N S U M E R use PhpAmqpLibConnectionAMQPConnection; use PhpAmqpLibMessageAMQPMessage; $connection = new AMQPConnection(HOST, PORT, USER, PASSWORD); $channel = $connection-‐>channel(); S E T T I N G U P C O N N E C T I O N C R E AT E A C H A N N E L
- 24. S C E N A R I O P R O D U C E R / C O N S U M E R P R O D U C E R Q U E U E passive durable exclusive auto-‐delete $channel-‐>queue_declare('catalog', false, false, false, false); foreach ($catalog-‐>getPages() as $page) { $message = new AMQPMessage($page); $channel-‐>basic_publish($message, '', 'catalog'); } $channel-‐>close(); $connection-‐>close();
- 25. S C E N A R I O P R O D U C E R / C O N S U M E R C O N S U M E R Q U E U E $connection = new AMQPConnection(HOST, PORT, USER, PASSWORD); $channel = $connection-‐>channel(); $channel-‐>queue_declare('catalog', false, false, false, false); $callback = function($msg) { $msg-‐>body-‐>generatePdf(); }; $channel-‐>basic_consume('catalog', '', false, true, false, false, $callback); while(count($channel-‐>callbacks)) { $channel-‐>wait(); } $channel-‐>close(); $connection-‐>close();
- 26. S C E N A R I O M U LT I P L E C O N S U M E R S P R O D U C E R C O N S U M E R Q U E U E C O N S U M E R …
- 27. S C E N A R I O M U LT I P L E C O N S U M E R S P R O D U C E R C O N S U M E R Q U E U E C O N S U M E R … parallelize work easy scalability
- 28. C O N S U M E R S C A N D I E
- 29. M E S S A G E A K N O W L E D G E M E N T
- 30. S A F E T Y M E S S A G E A C K S $callback = function($msg){ $msg-‐>body-‐>generatePdf(); $msg-‐>delivery_info['channel']-‐>basic_ack($msg-‐>delivery_info['delivery_tag']); }; ! $channel-‐>basic_consume('catalog', '', false, false, false, false, $callback); S W I T C H A C K O N
- 31. D O N ’ T F O R G E T T O S E N D A C K S
- 32. T H E B R O K E R C A N D I E
- 33. D U R A B I L I T Y
- 34. D U R A B I L I T Y M A R K T H E Q U E U E A N D T H E C H A N N E L $channel-‐>queue_declare('catalog', false, true, false, false); In order to achieve durability $message = new AMQPMessage($data, #the message is now persistent array('delivery_mode' => 2) ); the queue must be declared durable the message must be marked as persistent
- 35. Q O S P O L I T I C S
- 36. Q O S P O L I T I C A B E FA I R C O N S U M E R C O N S U M E R for certain instances of the Round Robin dispatching B R O K E R #1 #3 #5 #2 #4 #6
- 37. Q O S P O L I T I C S B E FA I R $channel-‐>basic_qos(null, 1, null); C O N S U M E R C O N S U M E R the broker sends messages only when it receives acks B R O K E R #1 #3 #3 #2#4
- 38. P U B L I S H / S U B S C R I B E S C E N A R I O
- 39. S C E N A R I O P U B L I S H / S U B S C R I B E chatRoom FA N O U T E X C H A N G E $connection = new AMQPConnection(HOST, PORT, USER, PASSWORD); $channel = $connection-‐>channel(); $channel-‐>exchange_declare('chatRoom', 'fanout', false, false, false); Exchange … Setting up the connection and declare the fanout exchange
- 40. S C E N A R I O P U B L I S H / S U B S C R I B E chatRoom Exchange … $data = getAMessageToSendInTheRoom(); $msg = new AMQPMessage($data); $channel-‐>basic_publish($msg, 'chatRoom'); $channel-‐>close(); $connection-‐>close(); P R O D U C E R Produce Publishing a message to the exchange
- 41. N O W I T ’ S T H E S U B S C R I B E R ’ S T U R N
- 42. S C E N A R I O P U B L I S H / S U B S C R I B E chatRoom Q U E U E B I N D I N G Exchange … //connection setted up ! list($queue_name, ,) = $channel-‐>queue_declare("", false, false, true, false); $channel-‐>queue_bind($queue_name, 'chatRoom'); Bind the Queue on the Exchange amq.gen-A7d bind bind amq.gen-Sb4
- 43. S C E N A R I O P U B L I S H / S U B S C R I B E chatRoom Exchange … amq.gen-A7d $channel-‐>basic_consume($queue_name, '', false, true, false, false, 'readMessage'); $channel-‐>close(); $connection-‐>close(); C O N S U M E R Consumes amq.gen-Sb4
- 44. R O U T I N G S C E N A R I O
- 45. S C E N A R I O R O U T I N G chatRoom Exchange type=direct … amq.gen-A7d amq.gen-Sb4 Consumer1 ConsumerN P R O D U C E R Produce routing_keys = friends routing_keys = friends, colleagues $channel-‐>exchange_declare('chatRoom', 'direct', false, false, false);
- 46. S C E N A R I O R O U T I N G chatRoom Exchange type=direct P R O D U C E R Produce Producing messages //connection setted up ! $channel-‐>exchange_declare('chatRoom', 'direct', false, false, false); $data = getAMessageToSendInTheRoom('friends'); $msg = new AMQPMessage($data); $channel-‐>basic_publish($msg, 'chatRoom', 'friends'); $channel-‐>close(); $connection-‐>close();
- 47. S C E N A R I O R O U T I N G //connection setted up //exchange setted up //$queue_name is a system generated queue name ! $rooms = array('friends', 'colleagues'); foreach($rooms as $room) { $channel-‐>queue_bind($queue_name, 'chatRoom', $room); } $channel-‐>basic_consume($queue_name, '', false, true, false, false, 'readMessage'); $channel-‐>close(); $connection-‐>close(); Bind a Consumer on different routing keys and consuming messages chatRoom amq.gen-Sb4 Consumer routing_keys = friends, colleagues
- 48. T O P I C S C E N A R I O
- 49. G O F U R T H E R T O P I C $channel-‐>exchange_declare('vehicle', 'topic', false, false, false); vehicle Exchange type=topic amq.gen-A7d amq.gen-Sb4 Consumer1 Consumer2 P R O D U C E R Produce routing_keys = *.car.* routing_keys = race.# <type>.<vehicle>.<colour> routing_keys = *.*.red D O T S D E L I M I T E D R O U T I N G K E Y
- 50. A Y E L L O W S P O R T C A R
- 51. T O P I C A Y E L L O W S P O R T C A R vehicle Exchange type=topic amq.gen-A7d amq.gen-Sb4 Consumer1 Consumer2 P R O D U C E R Produce routing_keys = *.car.* routing_keys = race.# <sport>.<car>.<yellow> routing_keys = *.*.red D O T S D E L I M I T E D R O U T I N G K E Y
- 52. T O P I C A Y E L L O W S P O R T C A R vehicle Exchange type=topic amq.gen-A7d amq.gen-Sb4 Consumer1 Consumer2 P R O D U C E R Produce routing_keys = *.car.* routing_keys = race.# <sport>.<car>.<yellow> routing_keys = *.*.red D O T S D E L I M I T E D R O U T I N G K E Y
- 53. A R E D R A C E M O T O R B I K E
- 54. T O P I C A R E D R A C E M O T O R B I K E vehicle Exchange type=topic amq.gen-A7d amq.gen-Sb4 Consumer1 Consumer2 P R O D U C E R Produce routing_keys = *.car.* routing_keys = race.# <race>.<motorbike>.<red> routing_keys = *.*.red D O T S D E L I M I T E D R O U T I N G K E Y
- 55. T O P I C A R E D R A C E M O T O R B I K E vehicle Exchange type=topic amq.gen-A7d amq.gen-Sb4 Consumer1 Consumer2 P R O D U C E R Produce routing_keys = *.car.* routing_keys = race.# <race>.<motorbike>.<red> routing_keys = *.*.red D O T S D E L I M I T E D R O U T I N G K E Y
- 56. A R E D R A C E C A R
- 57. T O P I C A R E D R A C E C A R vehicle Exchange type=topic amq.gen-A7d amq.gen-Sb4 Consumer1 Consumer2 P R O D U C E R Produce routing_keys = *.car.* routing_keys = race.# <race>.<car>.<red> routing_keys = *.*.red D O T S D E L I M I T E D R O U T I N G K E Y
- 58. T O P I C A R E D R A C E C A R vehicle Exchange type=topic amq.gen-A7d amq.gen-Sb4 Consumer1 Consumer2 P R O D U C E R Produce routing_keys = *.car.* routing_keys = race.# <race>.<car>.<red> routing_keys = *.*.red D O T S D E L I M I T E D R O U T I N G K E Y
- 59. A B L U E C I T Y VA N
- 60. T O P I C A B L U E C I T Y VA N vehicle Exchange type=topic amq.gen-A7d amq.gen-Sb4 Consumer1 Consumer2 P R O D U C E R Produce routing_keys = *.car.* routing_keys = race.# <city>.<van>.<blue> routing_keys = *.*.red D O T S D E L I M I T E D R O U T I N G K E Y
- 61. D I S C A R D E D
- 63. R E M E M B E R
- 64. P R O C E S S I N G Y O U R D ATA A S Y N C H R O N O U S LY C A N B E D O N E E A S I LY
- 65. E V E N I N P H P
- 66. C A N M A K E Y O U R ( D E V ) L I F E H A P P I E R
- 67. Y O U R A P P L I C AT I O N S T R O N G E R
- 69. joind.in/11608 Please rate the talk!
- 70. C R E D I T S • https://www.flickr.com/photos/rayofsun/9401226342 • https://www.flickr.com/photos/thereeljames/11376085194 • https://www.flickr.com/photos/ollily/359817111 • https://www.flickr.com/photos/legofenris/4004170937 • https://www.flickr.com/photos/bsabarnowl/10993445723 • https://www.flickr.com/photos/jpott/2984914512 • https://www.flickr.com/photos/kalexanderson/6231391820 • https://www.flickr.com/photos/a2gemma/1448178195 • https://www.flickr.com/photos/dubpics/5619966355 • https://www.flickr.com/photos/umbertofistarol/5747425870 • https://www.flickr.com/photos/streetcarl/6888965017 • https://www.flickr.com/photos/infomastern/12407730413 • https://www.flickr.com/photos/giuseppemilo/11817936944 • https://www.flickr.com/photos/avardwoolaver/7137096221