Message Queuing - Gearman, Mosquitto, Kafka and RabbitMQ
1 like66 views
This document summarizes and compares several message queuing systems: Gearman, Mosquitto, Kafka, and RabbitMQ. It discusses their pub/sub models, broker architectures, message/topic storage and delivery methods, actual communication protocols, and Java ecosystems. The key points are that these systems provide asynchronous communication between applications using a broker, each with different architectures, storage and delivery semantics, and Java client libraries.
![[March sn meetup] apache pulsar + apache nifi for cloud data lake](https://cdn.slidesharecdn.com/ss_thumbnails/marchsnmeetupapachepulsarapachenififorclouddatalake-220311125943-thumbnail.jpg?width=560&fit=bounds)
![[@NaukriEngineering] Messaging Queues](https://cdn.slidesharecdn.com/ss_thumbnails/queueprocessing-161111064335-thumbnail.jpg?width=560&fit=bounds)
Ad
More Related Content
Similar to Message Queuing - Gearman, Mosquitto, Kafka and RabbitMQ (20)
More from Marian Marinov (20)
Ad
Recently uploaded (20)
Ad
Message Queuing - Gearman, Mosquitto, Kafka and RabbitMQ
- 2. What are we going to review? ➢ Gearman (the most basic of the bunch) ➢ Mosquitto ➢ Kafka ➢ RabbitMQ
- 3. PUB/SUB model ➢ Gearman - client -> worker ➢ Mosquitto - publish -> subscribe ➢ Kafka - publish -> subscribe ➢ RabbitMQ - producer-> consumer ➢ All of them have a broker in the middle ➢ I intentionally omit ZeroMQ and POSIX Message Queues
- 4. PUB/SUB model ➢ Gearman - client -> worker ➢ Mosquitto - publish -> subscribe ➢ Kafka - publish -> subscribe ➢ RabbitMQ - producer-> consumer ➢ All of them have a broker in the middle ➢ I intentionally omit ZeroMQ and POSIX Message Queues ➢ Also intentionally skipped ActiveMQ and JBossMQ :)
- 5. But what is it? app1 <-> broker | <-> app2 | <-> app2 | <-> app2 <-> app2 main() { user_data = get_user_data(user_id); } where main() will be app1 and get_user_data() will be app2
- 6. But what is it? app1 can be in any language In the different brokers, it is called: ➢ Client (gearman) ➢ Publisher (Kafka/Mosquitto) ➢ Producer (RabbitMQ)
- 7. But what is it? app2 can be in any language, In the different brokers, it is called: ➢ Worker (gearman) ➢ Subscriber (Kafka/Mosquitto) ➢ Consumer (RabbitMQ)
- 8. But what is it? The brokers can implement different schemas of executions: ➢ Round-robin (sync/async) ➢ Weighted ➢ Fanout (sent to all connected workers), only async ➢ Topic (or list of topics)
- 9. Why messaging? ➢ Lighter way to "call" an API
- 10. Why messaging? ➢ Lighter way to "call" an API ➢ Use multiple languages, seamlessly
- 11. Why messaging? ➢ Lighter way to "call" an API ➢ Use multiple languages, seamlessly ➢ Loose the possibility to send values by reference
- 12. Why messaging? ➢ Lighter way to "call" an API ➢ Use multiple languages, seamlessly ➢ Loose the possibility to send values by reference ➢ Self confined interfaces
- 13. Why messaging? ➢ Lighter way to "call" an API ➢ Use multiple languages, seamlessly ➢ Loose the possibility to send values by reference ➢ Self confined interfaces ➢ Parallelism and extreme scalability
- 14. Why messaging? ➢ Lighter way to "call" an API ➢ Use multiple languages, seamlessly ➢ Loose the possibility to send values by reference ➢ Self confined interfaces ➢ Parallelism and extreme scalability ➢ Control over resource usage
- 15. Why messaging? ➢ Lighter way to "call" an API ➢ Use multiple languages, seamlessly ➢ Loose the possibility to send values by reference ➢ Self confined interfaces ➢ Parallelism and extreme scalability ➢ Control over resource usage ➢ Easy access separation
- 16. Broker architecture ➢ Gearman - single server ➢ Mosquitto - single server (HA in Pro) ➢ Kafka - single or distributed ➢ RabbitMQ - distributed but can work on single server * Kafka and RabbitMQ have suggested minimum number of 3 servers
- 17. Message/Topic storage ➢ Gearman ➢ messages are retained until consumed ➢ own protocol (very lightweight) ➢ Mosquitto ➢ message expiration can be configured ➢ topic hierarchy sensors/+/temperature/# ➢ MQTT protocol (lightwight) ➢ Kafka ➢ messages are deleted if not consumed in 7d(default) ➢ own protocol ➢ RabbitMQ ➢ STOMP, MQTT, AMQP, HTTP and WebSockets ➢ message delivery and durability depends :)
- 18. Message/Topic delivery ➢ Gearman ➢ sync or async(round-robin) ➢ Mosquitto ➢ topic hierarchy sensors/+/temperature/# ➢ Kafka ➢ topics ➢ messages are not deleted on consumption ➢ RabbitMQ ➢ direct(sync), fanout(broadcast), topic or match
- 22. Ecosystem - Kafka ➢ Very well integrated into Java and generally the GOTO solution for messaging ➢ Kafka Streams is the library that you have to take a look at
- 23. Ecosystem - Gearman ➢“Java gearman service” is a asynchronous interface written entirely in Java ➢ Gearman Java a pure Java driver exists on Launchpad
- 24. Ecosystem - Mosquitto ➢ Eclipse Paho Java Client
- 25. Ecosystem - RabbitMQ ➢ Spring AMQP ➢ Java Client ➢ JMS Client ➢ Hop ➢ Reactor
- 26. Thank you! If you have any questions find me at the back