MoP(MQTT on Pulsar) - a Powerful Tool for Apache Pulsar in IoT - Pulsar Summit Asia 2021
0 likes406 views
MQTT (Message Queuing Telemetry Transport,) is a message protocol based on the pub/sub model with the advantages of compact message structure, low resource consumption, and high efficiency, which is suitable for IoT applications with low bandwidth and unstable network environments. This session will introduce MQTT on Pulsar, which allows developers users of MQTT transport protocol to use Apache Pulsar. I will share the architecture, principles and future planning of MoP, to help you understand Apache Pulsar's capabilities and practices in the IoT industry.
![Metrics
-Prometheus
-Http endpoint
additionalServlets=mqtt-servlet
additionalServletDirectory=[protocolHandlerDir]
curl http://pulsar-broker-webservice-address:port/mop-stats/
{“cluster":"test","subscriptions":{"subs":["/a/b/c"],"count":1},"clients":
{"total":1,"maximum":1,"active":0,"active_clients":[]},"namespace":"default","messages":
{"received_bytes":57351,"received_count":10,"send_count":20,"send_bytes":60235},"ver
sion":"2.9.0-SNAPSHOT","tenant":"public","uptime":"46 seconds"}
MoP will uniformly output its own metrics to Prometheus.](https://image.slidesharecdn.com/mop-220228031347/85/MoP-MQTT-on-Pulsar-a-Powerful-Tool-for-Apache-Pulsar-in-IoT-Pulsar-Summit-Asia-2021-21-320.jpg)
Ad
More Related Content
What's hot (20)
Similar to MoP(MQTT on Pulsar) - a Powerful Tool for Apache Pulsar in IoT - Pulsar Summit Asia 2021 (20)
Ad
More from StreamNative (20)
Ad
Recently uploaded (19)
MoP(MQTT on Pulsar) - a Powerful Tool for Apache Pulsar in IoT - Pulsar Summit Asia 2021
- 1. MQTT on Pulsar (MoP) Jiwei Guo 2022-01-15
- 2. StreamNative - Apache Pulsar Committer - MoP Maintainer About me https://github.com/Technoboy-
- 3. Agenda - Why MoP? - Protocol handler - Support Features - Implementation details - Future Work
- 4. 1. Why MoP?
- 5. As a MQTT user •Rewrite the client •MQTT API wrapper • MQTT connector
- 6. With MoP •No need to change existed MQTT client code, no matter it’s Java or C++, etc. •No need to write a new connector for Pulsar. •MQTT Producer & Pulsar Consumer or Pulsar producer & MQTT consumer.
- 7. Proxy/gateway? MQTT Producer Proxy/Gateway MQTT requests Pulsar Broker Pulsar requests Bookie MoP is a plugin, which means MoP can access all the internal resources of broker instead sending requests by a client. MoP Bookie BK client Pulsar Broker MQTT Producer
- 9. What’s Protocol Handler? Protocol handler was introduced from Pulsar 2.5.0 as a plugin that is loaded dynamically by broker.
- 10. Definition of the protocol handler
- 11. Load the protocol handler Create an yml f ile under META-INF/services/, building the project to NAR package name: mqtt description: MQTT Protocol Handler handlerClass: io.streamnative.pulsar.handlers.mqtt.MQTTProtocolHandler When a broker starts, it will - Load all NAR packages under protocolHandlerDirectory - Select the package whose name is in messagingProtocols - Initialized the Java class speci f ied in handlerClass - Verify if the name could be accepted by ProtocolHandler#accept.
- 13. Install MoP • Compile -> package -> generate *.nar package. • Add some extra con f igs to broker.conf or standalone.conf. • • Restart Pulsar broker
- 14. MoP Proxy Extend MoP to multiple nodes to achieve horizontally scalable services 1. MQTT clients connect to MoP Proxy. 2. MoP Proxy sends lookup topic to pulsar cluster, searching for topic owner broker URL。 3. Pulsar cluster gives back topic owner broker URL to MoP Proxy。 4. MoP Proxy connects to owner broker，MQTT Clients begin to communicate with owner broker.
- 16. - Support MQTT3, MQTT3.1, MQTT5 - Qos0, Qos1 - Authentication - Authorization - TLS - Topic Filter - Metrics Features
- 17. Authentication mqttAuthenticationEnabled=true mqttAuthenticationMethods=basic authenticationProviders= org.apache.pulsar.broker.authentication.AuthenticationProviderBasic System.setProperty("pulsar.auth.basic.conf", “/xxx/basic”); MQTT mqtt = createMQTTClient(); mqtt.setUserName("xxx"); mqtt.setPassword(“pwd"); mqttAuthenticationEnabled=true mqttAuthenticationMethods=token authenticationProviders= org.apache.pulsar.broker.authentication.AuthenticationProviderToken tokenSecretKey=data:;base64,hXCl3lAdPA= MQTT mqtt = createMQTTClient(); mqtt.setPassword(“hXCl3lAdPA="); - Basic - Token
- 18. Authorization mqttAuthorizationEnabled=true MQTT mqtt = createMQTTClient(); mqtt.setUserName("admin10"); mqtt.setPassword(“123”);
- 20. Topic Filter /a/b/c /a/b/# multi-level wildcard /a/+/c single level wildcard Tips: -Topic filter could not listener for new topics.
- 21. Metrics -Prometheus -Http endpoint additionalServlets=mqtt-servlet additionalServletDirectory=[protocolHandlerDir] curl http://pulsar-broker-webservice-address:port/mop-stats/ {“cluster":"test","subscriptions":{"subs":["/a/b/c"],"count":1},"clients": {"total":1,"maximum":1,"active":0,"active_clients":[]},"namespace":"default","messages": {"received_bytes":57351,"received_count":10,"send_count":20,"send_bytes":60235},"ver sion":"2.9.0-SNAPSHOT","tenant":"public","uptime":"46 seconds"} MoP will uniformly output its own metrics to Prometheus.
- 23. Topic map => persistent://public/default/topicA - tenantA/NamespaceA/topicA => persistent://tenantA/NamespaceA/topicA - /a/b/c => persistent://public/default/%2Fa%2Fb%2Fc - topicA - persistent://public/default/a/b - non-persistent://public/default/a/b
- 24. Connect
- 25. Publish
- 26. Subscribe
- 27. Unsubscribe
- 28. Disconnect
- 29. 5. Future Work
- 30. - Performance Test - Support Qos2 - Management API
- 31. THANK YOU