Distributed messaging with AMQP
2 likes1,819 views
The document discusses distributed messaging using AMQP and RabbitMQ. It provides an overview of AMQP, including the service model with exchanges, queues, bindings and publishers/consumers. Different exchange types like direct, topic and headers are demonstrated. The AMQP protocol layers and wire protocol are also summarized.
![[@NaukriEngineering] Messaging Queues](https://cdn.slidesharecdn.com/ss_thumbnails/queueprocessing-161111064335-thumbnail.jpg?width=560&fit=bounds)
Ad
More Related Content
What's hot (20)
Viewers also liked (11)
Ad
Similar to Distributed messaging with AMQP (20)
Ad
More from Wee Keat Chin (6)
Recently uploaded (20)
Distributed messaging with AMQP
- 1. Distributed Messaging AMQP, RabbitMQ (Quick demo) Tuesday, 10 September 13
- 2. Event Logging Current architecture & problems Tuesday, 10 September 13
- 3. Event Logging Server Client Client Client MongoDB Tuesday, 10 September 13
- 8. Possible Solutions • Server tweaks:TCP conns, ulimit, etc • Multi-master model • Localised queue • ... and quite a few more Tuesday, 10 September 13
- 9. Possible Solutions • Server tweaks:TCP conns, ulimit, etc • Multi-master model •Localised queue • ... and quite a few more Tuesday, 10 September 13
- 10. Localised Queues • Fast (no TCP latency) • Network partition tolerant • Modular & scalable • Reliable & Durable Tuesday, 10 September 13
- 11. Why Localise? Server Client Client Client 80 110 65 MongoDB Q Q Q < 10 Tuesday, 10 September 13
- 14. Why Localise? Server Client Client Client 80 110 65 MongoDB Q Q Q < 10 Tuesday, 10 September 13
- 15. Why Localise? Server Client Client Client 80 110 65 MongoDB Q Q Q < 10 Client Q Tuesday, 10 September 13
- 16. Queue Criteria • Reliable • Durable • Scalable Tuesday, 10 September 13
- 17. AMQP Advanced Message Queueing Protocol Tuesday, 10 September 13
- 18. Why AMQP? • Stable and mature • Designed for SX markets • Widely supported (good interop) • Modular, Fast & Flexible Tuesday, 10 September 13
- 19. Service ModelWire AMQP Overview AMQP Advanced Message Queueing Protocol Tuesday, 10 September 13
- 20. Service Model Wire AMQP Overview AMQP Tuesday, 10 September 13
- 21. ➡ Model/component spec ‣ Exchange ‣ Message Queue ‣ Binding AMQ Model Wire AMQP Overview AMQP Tuesday, 10 September 13
- 22. ➡ Model/component spec ‣ Exchange ‣ Message Queue ‣ Binding AMQ Model Wire AMQP Overview AMQP ➡ Function layer ‣Transactions, Exchange, Queues, etc ➡Transport layer ‣ Protocol format ‣ Data rep/framing ‣ Multiplexing, heart-beating, etc Tuesday, 10 September 13
- 23. AMQ Model Wire AMQP Overview AMQP ➡ Function layer ‣Transactions, Exchange, Queues, etc ➡Transport layer ‣ Protocol format ‣ Data rep/framing ‣ Multiplexing, heart-beating, etc ➡ Model/component spec ‣ Exchange ‣ Message Queue ‣ Binding Tuesday, 10 September 13
- 24. ExchangeQueuesBindingAMQ Model AMQ Model Tuesday, 10 September 13
- 25. QueuesExchange Binding AMQ Model AMQ Model Tuesday, 10 September 13
- 26. Virtual Host QueuesExchange Binding AMQ Model AMQ Model Tuesday, 10 September 13
- 27. Virtual Host QueuesExchange Binding AMQ Model AMQ Model Publisher Consumer Tuesday, 10 September 13
- 28. Virtual Host QueuesExchange Binding AMQ Model Publisher Consumer Tuesday, 10 September 13
- 29. Virtual Host QueuesExchange Binding AMQ Model Router Message Publisher Consumer Tuesday, 10 September 13
- 30. Virtual Host QueuesExchange Binding AMQ Model Router Message Publisher Consumer Tuesday, 10 September 13
- 31. Virtual Host QueuesExchange Binding AMQ Model Router Message Publisher Consumer Tuesday, 10 September 13
- 32. Virtual Host QueuesExchange Binding AMQ Model Router ?Message Publisher Consumer Tuesday, 10 September 13
- 33. Virtual Host QueuesExchange Binding AMQ Model Message Router ? Publisher Consumer Tuesday, 10 September 13
- 34. Virtual Host QueuesExchange Binding AMQ Model Message Router Publisher ➡ Queue: ‣ critical_error_queue ➡ Exchange: ‣ errors_exchange ➡ Filter: ‣ *.error.critical Consumer Tuesday, 10 September 13
- 35. Virtual Host QueuesExchange Binding AMQ Model ➡ Queue: ‣ critical_error_queue ➡ Exchange: ‣ errors_exchange ➡ Filter: ‣ *.error.critical Message Router Publisher Consumer Tuesday, 10 September 13
- 36. QueuesExchange Binding AMQP Service Model Tuesday, 10 September 13
- 37. QueuesExchange Binding AMQP Service Model ‣ routes messages based on criteria ‣ doesn’t store messages ‣ can inspect message content ‣ can be created at runtime consumers Tuesday, 10 September 13
- 38. QueuesExchange Binding AMQP Service Model ‣ store messages ‣ named ‣ bound-able to exchange ‣ criteria ‣ can be created at runtime by consumers ‣ routes messages based on criteria ‣ doesn’t store messages ‣ can inspect message content ‣ can be created at runtime consumers Tuesday, 10 September 13
- 39. QueuesExchange Binding AMQP Service Model ‣ creates a relationship between queues and exchanges ‣ contains criteria and properties ‣ can be created at runtime by consumers ‣ store messages ‣ named ‣ bound-able to exchange ‣ criteria ‣ can be created at runtime by consumers ‣ routes messages based on criteria ‣ doesn’t store messages ‣ can inspect message content ‣ can be created at runtime consumers Tuesday, 10 September 13
- 40. QueuesExchange Binding AMQP Service Model ‣ creates a relationship between queues and exchanges ‣ contains criteria and properties ‣ can be created at runtime by consumers ‣ store messages ‣ named ‣ bound-able to exchange ‣ criteria ‣ can be created at runtime by consumers ‣ routes messages based on criteria ‣ doesn’t store messages ‣ can inspect message content ‣ can be created at runtime consumers Tuesday, 10 September 13
- 41. QueuesExchange Binding AMQP Service Model Publisher Consumer ‣ creates a relationship between queues and exchanges ‣ contains criteria and properties ‣ can be created at runtime by consumers ‣ store messages ‣ named ‣ bound-able to exchange ‣ criteria ‣ can be created at runtime by consumers ‣ routes messages based on criteria ‣ doesn’t store messages ‣ can inspect message content ‣ can be created at runtime consumers Tuesday, 10 September 13
- 42. QueuesExchange Binding AMQP Service Model Publisher Consumer ‣ creates a relationship between queues and exchanges ‣ contains criteria and properties ‣ can be created at runtime by consumers ‣ store messages ‣ named ‣ bound-able to exchange ‣ criteria ‣ can be created at runtime by consumers ‣ routes messages based on criteria ‣ doesn’t store messages ‣ can inspect message content ‣ can be created at runtime consumers ‣ can create exchanges and queues ‣ can create exchanges and queues Tuesday, 10 September 13
- 43. Exchange AMQP Service Model Tuesday, 10 September 13
- 44. FanoutTopicHeadersSystem Exchange AMQP Service Model Types (routing algo) Tuesday, 10 September 13
- 45. Fanout Topic Headers System Exchange AMQP Service Model Types (routing algo) Direct Tuesday, 10 September 13
- 46. Fanout Topic Headers System Exchange AMQP Service Model Types (routing algo) Direct Tuesday, 10 September 13
- 47. Exchange AMQP Service Model Direct Type Tuesday, 10 September 13
- 48. Exchange ‘events’ AMQP Service Model Direct Type Tuesday, 10 September 13
- 49. Exchange ‘events’ AMQP Service Model Direct Type Message Message Message event.user.click event.user.view event.user.share Tuesday, 10 September 13
- 50. Exchange ‘events’ AMQP Service Model Direct Type Message Message Message event.user.click event.user.view event.user.share user_click_queue Tuesday, 10 September 13
- 51. Exchange ‘events’ Binding AMQP Service Model ➡Queue: ‣ user_click_queue Direct Type Message Message Message event.user.click event.user.view event.user.share user_click_queue Tuesday, 10 September 13
- 52. Exchange ‘events’ Binding AMQP Service Model ➡Queue: ‣ user_click_queue ➡Exchange: ‣ events Direct Type Message Message Message event.user.click event.user.view event.user.share user_click_queue Tuesday, 10 September 13
- 53. Exchange ‘events’ Binding AMQP Service Model ➡Queue: ‣ user_click_queue ➡Exchange: ‣ events ➡Routing Key: ‣ event.user.click Direct Type Message Message Message event.user.click event.user.view event.user.share user_click_queue Tuesday, 10 September 13
- 54. Exchange ‘events’ Binding AMQP Service Model ➡Queue: ‣ user_click_queue ➡Exchange: ‣ events ➡Routing Key: ‣ event.user.click Direct Type Message Message Message event.user.click event.user.view event.user.share PHP Script PHP Script PHP Script user_click_queue Tuesday, 10 September 13
- 55. Exchange ‘events’ AMQP Service Model Direct Type Message Message Message event.user.click event.user.view event.user.share PHP Script PHP Script PHP Script user_click_queue ConsumerPublisher/ Producer Tuesday, 10 September 13
- 56. Exchange ‘events’ AMQP Service Model Direct Type Message Message Message event.user.click event.user.view event.user.share user_click_queue user_view_queue user_share_queue Tuesday, 10 September 13
- 57. Exchange ‘events’ AMQP Service Model Direct Type Message Message Message event.user.click event.user.view event.user.share user_click_queue user_view_queue user_share_queue Message Message Message Tuesday, 10 September 13
- 58. Exchange ‘events’ AMQP Service Model What if... Message Message Message event.user.click event.user.view event.user.share user_interaction_queue user_shares_queue Tuesday, 10 September 13
- 59. Exchange ‘events’ AMQP Service Model What if... user_interaction_queue user_shares_queue Message Message Message event.user.click event.user.view event.user.share Tuesday, 10 September 13
- 60. Exchange ‘events’ AMQP Service Model What if... Message Message Message event.user.click event.user.view event.user.share user_interaction_queue user_shares_queue Tuesday, 10 September 13
- 61. Exchange ‘events’ AMQP Service Model Topic Exchange user_interaction_queue Binding Message Message Message event.user.click event.user.view event.user.share Tuesday, 10 September 13
- 62. Exchange ‘events’ AMQP Service Model Topic Exchange user_interaction_queue Binding ➡Queue: ‣ user_interaction_queue ➡Exchange: ‣ events ➡Routing Key: ‣ event.user.* Message Message Message event.user.click event.user.view event.user.share Tuesday, 10 September 13
- 63. Exchange ‘events’ AMQP Service Model Topic Exchange user_interaction_queue Binding ➡Queue: ‣ user_interaction_queue ➡Exchange: ‣ events ➡Routing Key: ‣ event.user.* Message Message Message event.user.click event.user.view event.user.share PHP Script PHP Script PHP Script Tuesday, 10 September 13
- 64. Exchange ‘events’ AMQP Service Model Topic Exchange user_shares_queue Binding Message Message Message event.user.click event.user.view event.user.share Tuesday, 10 September 13
- 65. Exchange ‘events’ AMQP Service Model Topic Exchange user_shares_queue Binding ➡Queue: ‣ user_interaction_queue ➡Exchange: ‣ events ➡Routing Key: ‣ event.user.share Message Message Message event.user.click event.user.view event.user.share Tuesday, 10 September 13
- 66. Exchange ‘events’ AMQP Service Model Topic Exchange user_shares_queue Binding ➡Queue: ‣ user_interaction_queue ➡Exchange: ‣ events ➡Routing Key: ‣ event.user.share Message Message Message event.user.click event.user.view event.user.share PHP Script PHP Script PHP Script Tuesday, 10 September 13
- 67. Message Exchange ‘events’ AMQP Service Model Topic Exchange Message Message Message event.user.click event.user.view event.user.share user_interaction_queue user_shares_queue Message Message event.user.* event.user.share Tuesday, 10 September 13
- 68. Message Exchange ‘events’ AMQP Service Model Topic Exchange Message Message Message event.user.click event.user.view event.user.share user_interaction_queue user_shares_queue Message Message Can be done at runtime event.user.* event.user.share Tuesday, 10 September 13
- 69. Message Exchange ‘events’ AMQP Service Model Topic Exchange Message Message Message event.user.click event.user.view event.user.share user_interaction_queue user_shares_queue Message Message Click vs Share Conversion? event.user.* event.user.share Tuesday, 10 September 13
- 70. Message Exchange ‘events’ AMQP Service Model Topic Exchange Message Message Message event.user.click event.user.view event.user.share user_interaction_queue user_shares_queue Message Message Click vs Share Conversion? event.user.* event.user.share Message click_share_queue Message event.user.share event.user.click Tuesday, 10 September 13
- 71. Message Exchange ‘events’ AMQP Service Model Topic Exchange Message Message Message event.user.click event.user.view event.user.share user_interaction_queue user_shares_queue Message live_view_queue event.user.* event.user.share *.*.view Message Message event.staff.view Message Message Staff vs user views? Tuesday, 10 September 13
- 72. AMQ Model Wire AMQP Overview AMQP ➡ Function layer ‣Transactions, Exchange, Queues, etc ➡Transport layer ‣ Protocol format ‣ Data rep/framing ‣ Multiplexing, heart-beating, etc ➡ Model/component spec ‣ Exchange ‣ Message Queue ‣ Binding Tuesday, 10 September 13
- 73. ➡ Model/component spec ‣ Exchange ‣ Message Queue ‣ Binding AMQ Model Wire AMQP Overview AMQP ➡ Function layer ‣Transactions, Exchange, Queues, etc ➡Transport layer ‣ Protocol format ‣ Data rep/framing ‣ Multiplexing, heart-beating, etc Tuesday, 10 September 13
- 74. Wire Wire Protocol Tuesday, 10 September 13
- 75. Wire Wire Protocol ‣ Datagram specification ‣ Protocol header ‣ Payload format ‣ Data fields specification ‣ Content framing ‣ Heartbeat frames ‣ Channel multiplexing ‣ Visibility guarantee ‣ Channel closure ‣ Content synchronisation ‣ Content ordering Tuesday, 10 September 13
- 76. Wire Protocol Method Payloads Tuesday, 10 September 13
- 77. Wire Protocol Method Payloads Tuesday, 10 September 13
- 78. Virtual Host QueuesExchange Binding Summary ➡ Queue: ‣ critical_error_queue ➡ Exchange: ‣ errors_exchange ➡ Filter: ‣ *.error.critical Message Router Publisher Consumer Tuesday, 10 September 13
- 79. Revisiting event logging A proposed solution Tuesday, 10 September 13
- 80. Event Logging Revisited Client Client Client MongoDB Tuesday, 10 September 13
- 81. Event Logging Revisited Client Client Client MQ MQ MQ MongoDB Tuesday, 10 September 13
- 82. MongoDB Event Logging Revisited MQClient Client Client 80 110 65 MQ MQ MQ Tuesday, 10 September 13
- 83. MongoDB Event Logging Revisited MQClient Client Client 80 110 65 MQ MQ MQ Tuesday, 10 September 13
- 84. MongoDB Event Logging Revisited MQClient Client Client 80 110 65 MQ MQ MQ MQ Tuesday, 10 September 13
- 85. MongoDB Event Logging Revisited MQ Client Client Client 80 110 65 MQ MQ MQ MQ Tuesday, 10 September 13
- 86. Message Exchange ‘events’ AMQP Service Model Topic Exchange Message Message Message event.user.click event.user.view event.user.share user_interaction_queue user_shares_queue Message live_view_queue event.user.* event.user.share *.*.view Message Message event.staff.view Message Message Remember this? Tuesday, 10 September 13
- 87. MongoDB Event Logging Revisited MQClient Client Client 80 110 65 MQ MQ MQ Reporting SQL Notification Tuesday, 10 September 13
- 88. RabbitMQ An AMQP Implementation Tuesday, 10 September 13
- 89. Why RabbitMQ? • Mature (born 2006) • Acquired by SpringSource (now part of VMWare) in 2010 • Used by companies like Nokia, Google, Mozilla, vFabric, SecondLife, OpenStack Tuesday, 10 September 13
- 90. Overview • Written in Erlang/OTP • Supports other protocols too - STOMP, MQTT • Many library supports - Python, PHP, Ruby, .NET • RESTful API is available Tuesday, 10 September 13
- 91. Hands on Demo Tuesday, 10 September 13
- 92. Q&A Tuesday, 10 September 13
- 93. Coming soon... RabbitMQ in Depth (sort of) Tuesday, 10 September 13