RabbitMQ with python and ruby RuPy 2009
38 likes5,925 views
The document discusses using RabbitMQ, an open-source message broker based on AMQP, for asynchronous messaging between Python and Ruby applications. It provides an overview of AMQP concepts like producers, consumers, exchanges and queues, and how RabbitMQ implements these using Erlang. Code examples are shown for sending and receiving messages asynchronously in Python and Ruby.
![AMQP
Why is it interesting?
“We use asynchronicity heavily in those
cases where performance is an issue.
This is generally where we send content
from one peer to another. We send off
methods as fast as possible without
waiting for confirmations”
AMQP Specs [0.8 June 2006]](https://image.slidesharecdn.com/rupy-rabbit-091108021036-phpapp02/85/RabbitMQ-with-python-and-ruby-RuPy-2009-5-320.jpg)
RabbitMQ with python and ruby RuPy 2009
- 1. “RabbitMQ python and ruby staring at the looking glass” - an asynchronous tale - • http://www.slideshare.net/hungryblank • http://github.com/hungryblank/rabbitmq_rupy_2009 Paolo Negri @hungryblank
- 3. About me Time GNU/Linux - Dbs Perl “systems” PHP Python Ruby
- 4. AMQP Advanced Message Queuing Protocol • standard for high performance enterprise messaging • totally open • platform agnostic • interoperable
- 5. AMQP Why is it interesting? “We use asynchronicity heavily in those cases where performance is an issue. This is generally where we send content from one peer to another. We send off methods as fast as possible without waiting for confirmations” AMQP Specs [0.8 June 2006]
- 6. Summary “AMQP as a case of asynchronous protocol implementation in python and ruby”
- 7. AMQP - Entities Producer msg Consumer msg Broker http://www.flickr.com/photos/17258892@N05/2588347668/
- 8. AMQP - Entities Producer msg Producer msg Broker Producer msg Consumer msg Consumer msg http://www.flickr.com/photos/17258892@N05/2588347668/
- 9. AMQP Implementations • Producer/Consumer: libs available for python and ruby • Broker no python or ruby implementation
- 10. • RabbitMQ is an implementation of AMQP Broker • It’s opensource • Written in Erlang
- 11. Erlang? • general-purpose concurrent programming language designed by Ericsson • first version written by J. Armstrong in 1986 • distributed • fault tolerant • soft real time • high availability
- 12. Erlang? Erlang's main strength is support for concurrency “Process communication is done via a shared-nothing asynchronous message passing: every process has a “mailbox”, a queue of messages sent by other processes, that are not yet consumed” (Wikipedia)
- 13. + Erlang It’s messages all the way down
- 14. Async Python libs txAMQP • “twisted AMQP” • based on twisted python • https://launchpad.net/txamqp
- 15. Async Python libs amqpev • based on eventlet • experimental, not ready for production • http://bitbucket.org/edarc/amqpev
- 16. Async Python libs pika • based on asyncio • young but mantained by member of RabbitMQ core team • http://github.com/tonyg/pika
- 17. Async Ruby libs amqp • amqp gem • based on EventMachine • mature and widely used • http://github.com/tmm1/amqp/
- 18. Async? • Twisted Python • Asyncio (python stdlib) • eventlet (python) • EventMachine (ruby)
- 19. Reactor Pattern • Non blocking IO and lightweight concurrency • eliminate the complexities of high- performance threaded network programming
- 20. synchronous asynchronous process Time process code code network network operation Free operation use network use network operation operation Callback result result Free Free
- 21. Not only reactor Some of the libs provide more i.e. eventlet provides also coroutines
- 22. Python, pick one pika • asyncio is part of python stdlib • easy syntax • developer member of RabbitMQ team • fits on slides :-P
- 24. Consumer (ruby)
- 25. Behind the scenes Publisher msg Exchange Queue Consumer
- 26. What and where RabbitMQ (Erlang) Producer (ruby/python) TCP/IP Consumer Exchange Queue (ruby/python)
- 28. Producer (python) TCP/IP connection
- 29. Producer (python) channel 1 TCP/IP connection
- 30. Producer (python) channel 1 TCP/IPchannel 2 connection
- 31. Producer (python) channel 1 TCP/IPchannel 2 connection channel n
- 43. Producer (python) Publisher msg Exchange Queue Consumer
- 44. Producer (python) Publisher msg Exchange Queue Consumer key based routing
- 45. Producer (python) binding by key Publisher msg Exchange Queue Consumer key based routing
- 47. Things go wrong If a consumer has a problem we might lose one or more messages http://www.flickr.com/photos/danzen/2288625136/
- 54. Achieved: decoupling Publisher msg Exchange Queue Consumer
- 55. Achieved: easy work load distribution Consumer Publisher msg A Exchange Queue Consumer Consumer
- 56. Achieved: easy work load distribution Consumer Publisher Exchange Queue Consumer msg A Consumer
- 57. Achieved: easy work load distribution Consumer Publisher Exchange Queue msg A Consumer Consumer
- 58. Achieved: easy work load distribution msg A Consumer Publisher Exchange Queue Consumer Consumer
- 60. prefetch? tcp ip buffer http://www.flickr.com/photos/balakov/327431271
- 61. prefetch - ruby
- 62. prefetch - ruby
- 65. More routing models multicasting with fanout exchange
- 72. Fanout Queue1 Cons1 Producer msg A Exchange Queue2 Cons2 Queue3 Cons3
- 73. Fanout Queue1 Cons1 Producer Exchange Queue2 Cons2 msg A Queue3 Cons3
- 74. Fanout msg A Queue1 Cons1 Producer Exchange msg A Queue2 Cons2 msg A Queue3 Cons3
- 79. Exchanges • Direct - based on binding routing key • Fanout - always to all bound queues • Topic - pattern matching on routing key
- 80. Clustering RabbitMQ - node A RabbitMQ - node B TCP/IP TCP/IP RabbitMQ - node C TCP/IP
- 81. Plugins • introduced with v 1.7.0 • Add functionalities to the broker • makes a interesting first erlang project
- 82. Ecosystem • 65 RabbitMQ / AMQP projects on github • + more on bitbucket, and others
- 83. Sync? • easier to add to web apps • suitable for non performance critical messaging • useful to build cli tools
- 84. Sync? • celldee / bunny - Ruby (github) • famoseagle / carrot - Ruby (github) • py-amqplib - Python (http://barryp.org/software/py-amqplib/) • ...
- 85. Distributed systems • ezmobious / nanite - Ruby (github) • ask / carrot - Python (github) • ask / celery - Python (github) • auser / poolparty - Erlang (github)
- 86. Real time web • RabbitHub PubSubHubBub • dansimpson / amqp-js (github)
- 87. Thank you! ?