Messaging Standards and Systems - AMQP & RabbitMQ
3 likes•1,728 views
The document provides an overview of the Advanced Message Queuing Protocol (AMQP) and RabbitMQ, detailing their features, timeline, components, and common usage patterns. It discusses various versions of AMQP, key terms such as brokers, producers, and consumers, and how RabbitMQ offers multi-protocol support and clustering capabilities. The document also touches on configuration management, performance concerns, and common plugins that enhance RabbitMQ's functionality.
Messaging Standards and Systems - AMQP & RabbitMQ
- 1. Messaging Standards and Systems AMQP & RabbitMQ Gavin M. Roy VP of Architecture AWeber Communications Twitter: @Crad POSSCON April 14, 2015
- 2. About Me VP of Architecture AWeber Communications Blame me for pika, rabbitpy, pamqp, and a handful of RabbitMQ plugins Blog: https://gavinroy.com Github: https://github.com/gmr Book: http://manning.com/roy
- 3. Advanced Message Queueing Protocol ❖ Open Standard ❖ Platform and Vendor Neutral ❖ Multiple Versions
- 4. AMQP Timeline 2003 2005 2006 2008 2011 Initial Specification Working Group formed AMQP 0-8 AMQP 0-9-1 OASIS AMQP 1.0
- 5. AMQP Working Group ❖ JPMorgan Chase ❖ Cisco Systems ❖ IONA Technologies ❖ iMatrix ❖ RedHat ❖ TWIST ❖ Bank of America ❖ Barclays ❖ Credit Suisse ❖ Deutsche Börse Systems ❖ Goldman Sachs ❖ HCL Technologies ❖ Progress Software ❖ IIT Software ❖ INETCO Systems Ltd. ❖ Informatica Corporation ❖ Microsoft Corporation ❖ my-Channels ❖ Novell ❖ Solace Systems ❖ Tervala, Inc. ❖ VMWare ❖ WSO2
- 7. AMQP 0-9-1 ❖ Currently has wider support than AMQP 1.0 ❖ Multiple Broker Implementations: RabbitMQ, Apache Qpid, and SwiftMQ to name a few ❖ Specifies a Model and Protocol
- 8. Native AMQP 0-8/0-9-1 Clients C Clojure Cobol Common Lisp Delphi Erlang Go Groovy Haskell Java JavaScript .NET OCaml Perl PHP Python Ruby Scala
- 9. ❖ Exchange Receives and route messages ❖ Message Queue Stores messages until they can be consumed ❖ Binding Defines the relationship between an Exchange and Queue and provides routing criteria Advanced Message Queueing Model X Exchange Binding Message Queue
- 10. Routing Keys ❖ Provided when publishing a message ❖ Compared against binding keys by exchanges ❖ Ideally provide context to the message: ❖ Connote the type of the message ❖ Categorize the content in the message ❖ Specify the type of consumer that should receive it
- 11. Advanced Message Queueing Protocol ❖ Compact, binary frame format wire protocol ❖ Bi-directional RPC ❖ Commands consist of Classes and Methods: ❖ Example Request: Queue.Declare(name=“foo”) ❖ Response: Queue.DeclareOk(messages=0, consumers=0)
- 12. Common AMQP Terms ❖ Broker A server that implements AMQP ❖ Producer or Publisher A client application that sends messages to a broker ❖ Consumer A client application that reads messages from a queue
- 16. AMQP Messages ❖ Comprised of 3 or more frames: ❖ Method Frame Basic.Publish, Basic.Deliver, etc ❖ Content Header with body size & message properties timestamp, message-id, app-id, etc ❖ n Body Frames with the opaque message payload
- 17. AMQP 0-9-1 Issues & Gotchas ❖ Ambiguous ❖ Authentication ❖ Asynchronous ❖ Connection Negotiation ❖ Exceptions !
- 18. RabbitMQ ❖ Open Source (MPL) ❖ Written in Erlang/OTP ❖ Developed/Maintained by Pivotal ❖ Multi-Protocol AMQP 0-9-1 & 1.0, MQTT, STOMP, XMPP, HTTP, Web-STOMP & More ❖ Roots in AMQP 0-8/0-9-1
- 19. Who Uses It? Agora Games Chef Google AdMob Instagram MeetMe Mercado Libre Mozilla NASA New York Times National Science Foundation Openstack Rapportive Reddit Soundcloud (and many more)
- 20. Why Use RabbitMQ? ❖ Create loosely-coupled applications ❖ Communicate across applications or platforms ❖ Tap into pre-existing message flows for new purposes ❖ Scale-out clustering for growth, throughput, and HA ❖ Federation for WAN latencies and network partitions ❖ Extensible plugin-in architecture
- 21. RabbitMQ Extensions to AMQP Authentication Failure Exchange to Exchange Bindings Delivery Confirmations Basic.Nack Consumer cancellations Consumer priorities Dead Letter Exchanges Alternate Exchanges Connection blocking Message CC & BCC Routing Queue Length Limits Per Queue Message TTL Per Message TTL Queue TTL Message User ID Validation Auto-delete exchanges
- 22. RabbitMQ Clustering ❖ LAN Only ❖ Adds highly-available queues ❖ Is cohesive, publish and consume from any node ❖ Leverages native Erlang clustering and communication ❖ Has multiple strategies for dealing with network partitions ❖ Manually configured via configuration or command line*
- 23. RabbitMQ Federation ❖ Provides loosely coupled inter- node/cluster RabbitMQ communication ❖ Connect multiple RabbitMQ clusters across data centers or the Internet ❖ Can be applied to both exchanges and queues Cluster A X Cluster B X Cluster C X
- 24. RabbitMQ Plugins amqp1_0 Adds AMQP 1.0 support to RabbitMQ auth-backend-amqp AMQP RPC for Authentication auth-backend-ldap Authenticate via an external LDAP server autocluster-consul Automatically create clusters using Consul msg_store_eleveldb_index LevelDB storage for queue messages presence-exchange Publishes messages upon binding changes rabbitmq-top Top like view of RabbitMQ processes in the management UI sharding Scale out RabbitMQ with automatic queue sharding (and many more)
- 25. Message Routing
- 26. Built-In Exchange Types ❖ Direct String matching on the routing key ❖ Fanout No routing key, messages delivered to all bound queues ❖ Topic Pattern matching in the routing key ❖ Headers No routing key, value matching in the headers property
- 27. Topic Exchange Binding Keys Routing Key: namespace.delimited.keys # Receive all messages namespace.# Receive all messages in namespace namespace.delimited.* Receive all namespace.delimited messages namespace.*.keys Receive all namespace messages ending with keys
- 28. Exchange to Exchange Binding X X X Queue Queue Queue Queue Queue Queue Messages published into a topic exchange are routed to two other exchanges and a queue Messages published into the consistent-hashing exchange are distributed amongst three queues Messages published into a direct exchange routed to two queues P
- 29. Alternate Exchanges P X Unroutable Messages X Unroutable messages are sent to an alternate exchange
- 30. Dead Letter Exchanges P X Reject Queue Queue C X Consumer rejects the message without requeue Dead-Letter Exchange routes the message
- 31. Exchange Plugins Consistent Hashing Distribute messages via hashed value of routing key Event Publishes messages on AMQP events such as queue creation Random Distribute messages across all bound queues randomly PostgreSQL LISTEN Subscribes to and publishes PostgreSQL notifications Recent History Sends the last n messages to any newly bound queue Reverse Topic Allows for routing patterns at publish time and not via binding Riak Storage Stores messages published through the exchange into Riak Script Exchange Calls out to external scripts for message routing
- 34. Persisted Messages IO Model Disk Queue Message Queue Queue Messages with delivery-mode 2 sent into RabbitMQ are persisted to disk Pointers to the message are placed in the queue data structures When the message is no longer in a queue, it is removed from disk
- 35. HA Queues & Performance Queue Queue Queue P RabbitMQ Cluster A publisher sends to any node in a RabbitMQ cluster The RabbitMQ servers in the cluster synchronize the state of the message in the queue The message that was published is put in the queue and is stored on each server
- 38. Trending & Monitoring ❖ RabbitMQ Management Plugin provides internal stats ❖ Queue depths, connection counts, throughput, memory usage, etc ❖ Monitor with common tools such as Nagios or Sensu to services such as Boundary and NewRelic ❖ Stream based monitoring with Riemann for anomaly detection
- 39. Thoughts on Configuration ❖ Use configuration management! ❖ Even for exchanges, queues, and bindings if possible ❖ Helpful for disaster recovery ❖ Use RabbitMQ’s Policies when possible ❖ Exchanges and queues are immutable ❖ Deleting and redeclaring for changes can be disruptive
- 43. Listen for Database Notifications X Queue
- 45. RPC C / PC / P P / C C / PResponse Queue X X Worker Queue Front-end web app receives a request Internet App publishes RPC request One of the subscribed worker consumers receives the RPC request After processing the request, the worker publishes the replyApp receives the reply from the response queue App replies to client
- 46. Delayed Messages WARNING: This pattern is a hack and can cause undesirable situations should the wait queue overwhelm the system! P X X Wait Queue Ready Queue C The wait queue is declared with a message TTL and a dead-letter exchangeMessages that will be delivered later are routed to a wait queue When a message times out due to the TTL, it is published to the DLX The internally re-published message is routed to the queue with active consumers
- 47. HA RabbitMQ with Federation Upstream Cluster A Queue A Queue B Upstream Cluster B Queue A Queue B Downstream Cluster A Queue A Queue B Downstream Cluster B Queue A Queue B HA Proxy 1 HA Proxy 2 P P P C A C B
- 48. Questions?