Making your Life Easier with MongoDB and Kafka (Robert Walters, MongoDB) Kafka Summit 2020
0 likes1,002 views
Kafka Connect makes it possible to easily integrate data sources like MongoDB! In this session we will first explore how MongoDB enables developers to rapidly innovate through the use of the document model. We will then put the document model to life and showcase how to integrate MongoDB and Kafka through the use of the MongoDB Connector with Apache Kafka. Finally, we will explore the different ways of using the connector including the new Confluent Cloud integration.
![Telediagnostics: The Future of Mercedes Benz Services
{
.............
"schema": "3.1.0"
"createdAt": {..},
.............
"vehicleIdentData": {
"chassisNumber": "WDD24708A5432J63",
"countryCode": "4f3490b14e238a5f",
"modelSeries": "f16ad22d42064811",
"modelType": "2196868af1c70d74",
"modelYear": "5e01ac15d73c3e4a",
"steering": "4a3424fe6411461c"
},
"basicData": {
"mileage": {..},
"batteries": [..],
"tanks": [..],
"tiresPressure": [..],
},
"controlUnits": [..],
"affectedFunctions": [..],
"vehicleClusterMessagesData": [..],
"maintenanceData": {...},
..............
}
"controlUnits": [
{
"ecuId": {..}
"name": "a927e49b0549f00f71",
"detailsHardware": {},
"detailsSoftware": {},
"dtc": [
{
"code": "B214F73",
...
"failureText": "81957650ea",
"environmentalData": {...}
},
...
]
},
...
]
More presentations available:
https://www.mongodb.com/world](https://image.slidesharecdn.com/s055robertwaltersmakingyourlifeeasierwithmongodbandkafka-200910040438/85/Making-your-Life-Easier-with-MongoDB-and-Kafka-Robert-Walters-MongoDB-Kafka-Summit-2020-8-320.jpg)
![Kafka Connect
Connect Connect
https://hub.confluent.io
[sources
]
[sinks]
ANY e.g. file systems, data stores, REST endpoints, …
Kafka Connect is an easy way to get data from many heterogeneous platforms into and out of Kafka](https://image.slidesharecdn.com/s055robertwaltersmakingyourlifeeasierwithmongodbandkafka-200910040438/85/Making-your-Life-Easier-with-MongoDB-and-Kafka-Robert-Walters-MongoDB-Kafka-Summit-2020-11-320.jpg)
![#MDBLocal
MongoDB Source: Writing to a Topic
Writes to topic based on database and collection name
Be aware of your message sizes
Optionally specify pipeline to manage change stream output to watch
Optionally set a topic.prefix in the connector configuration
database
collection
{}
[topic.prefix].database-name.collection-name
writes to topic ->
connector
pipeline[]
change
stream](https://image.slidesharecdn.com/s055robertwaltersmakingyourlifeeasierwithmongodbandkafka-200910040438/85/Making-your-Life-Easier-with-MongoDB-and-Kafka-Robert-Walters-MongoDB-Kafka-Summit-2020-14-320.jpg)
Ad
More Related Content
What's hot (20)
Similar to Making your Life Easier with MongoDB and Kafka (Robert Walters, MongoDB) Kafka Summit 2020 (20)
Ad
More from HostedbyConfluent (20)
Ad
Recently uploaded (20)
Making your Life Easier with MongoDB and Kafka (Robert Walters, MongoDB) Kafka Summit 2020
- 1. Making your life easier with MongoDB and Kafka Robert Walters | MongoDB
- 2. Presenter Robert Walters Product Manager Connectors and Things MongoDB https://www.linkedin.com/in/robwaltersprofile/
- 3. Agenda MongoDB Primer Deep Dive: MongoDB Kafka Connector Demo
- 4. MongoDB Adoption 90,000,000+ MongoDB Downloads 1 500,000+ Online Education Students 1m+ MongoDB Atlas Clusters 75,000+ MongoDB User Group Members 1,000+ Technology and Services Partners 17,500+ Customers Across All Industries DB-Engines Rankings Fastest Growing Database over the past decade Worldwide Activations
- 5. How does MongoDB make my life easier?
- 6. Strict Schema vs. Flexible Data Model id title description 12345-WoTLamp-1234 My Lamp A lamp in the room 67890-WoTLamp-1234 My Other Lamp Another lamp in the room id thing_id title readOnly writeOnly 4711-p1 12345-WoTLamp-1234 status false false id thing_id title safe idempodent 4711-a1 12345-WoTLamp-1234 toggle false false id thing_id title readOnly writeOnly 4711-e1 12345-WoTLamp-1234 overheating false false Thing Property Action Event { "id": "123456-WoTLamp-1234", "title": "My Lamp", "description": "A lamp in the room", "properties": { "status": { "type": "string", "readOnly" : false, "writeOnly" : false } }, "actions": { "toggle": { "safe": false, "idempodent": false } }, "events": { "overheating": { "data": { "type": "string", "readOnly" : false, "writeOnly" : false } } } } Relational Example Flexible Document Model
- 7. Telediagnostics: The Future of Mercedes Benz Services Learn more about the project in Madalin Broscaru’s MongoDB.live presentation: Telediagnostics@Mercedes Benz powered by MongoDB https://www.youtube.com/watch?v=MEH9_P_NATk&list=PL4RCxklHWZ9uRUB9qTnZE9cauOsTGE22n&index=50 … the Vehicle Data Conditioning (VDC) where these technical vehicle events are processed … CAC Retail Customer … and the follow-up processes are triggered with real-time recommendations for actions. Vehicles are transmitting regularly status and health data into ... Aggregated Quality Analysis
- 8. Telediagnostics: The Future of Mercedes Benz Services { ............. "schema": "3.1.0" "createdAt": {..}, ............. "vehicleIdentData": { "chassisNumber": "WDD24708A5432J63", "countryCode": "4f3490b14e238a5f", "modelSeries": "f16ad22d42064811", "modelType": "2196868af1c70d74", "modelYear": "5e01ac15d73c3e4a", "steering": "4a3424fe6411461c" }, "basicData": { "mileage": {..}, "batteries": [..], "tanks": [..], "tiresPressure": [..], }, "controlUnits": [..], "affectedFunctions": [..], "vehicleClusterMessagesData": [..], "maintenanceData": {...}, .............. } "controlUnits": [ { "ecuId": {..} "name": "a927e49b0549f00f71", "detailsHardware": {}, "detailsSoftware": {}, "dtc": [ { "code": "B214F73", ... "failureText": "81957650ea", "environmentalData": {...} }, ... ] }, ... ] More presentations available: https://www.mongodb.com/world
- 9. Self-hosted MongoDB Turn-key modern database You install, patch, maintain, scale, etc.. MongoDB as a service plus a whole lot more… MongoDB & MongoDB Atlas
- 11. Kafka Connect Connect Connect https://hub.confluent.io [sources ] [sinks] ANY e.g. file systems, data stores, REST endpoints, … Kafka Connect is an easy way to get data from many heterogeneous platforms into and out of Kafka
- 12. Available on the Confluent Hub: https://www.confluent.io/hub/mongodb/kafka-connect-mongodb MongoDB Connector for Apache Kafka Supported by MongoDB Verified Gold by
- 13. #MDBLocal MongoDB Source MongoDB Database MongoDB Connector topicA topicB topicC Kafka Cluster Subscribes to Change Stream events Writes to Kafka Topic(s) When MongoDB is a Kafka Source... Subscribes to change stream events and writes to Kafka topic(s)
- 14. #MDBLocal MongoDB Source: Writing to a Topic Writes to topic based on database and collection name Be aware of your message sizes Optionally specify pipeline to manage change stream output to watch Optionally set a topic.prefix in the connector configuration database collection {} [topic.prefix].database-name.collection-name writes to topic -> connector pipeline[] change stream
- 15. #MDBLocal MongoDB Sink MongoDB Database MongoDB Connector topicA topicB topicC Kafka Cluster Writes documents to db collection Reads from Kafka Topic(s) When MongoDB is a Kafka Sink... Reads from Kafka topic(s) and writes documents to collection
- 16. #MDBLocal MongoDB Sink: Reading Messages from Topic Reads messages from topic (based on pointer to message in topic) Writes message into MongoDB database collection Moves pointer to next message based on write to database Kafka Topic connector database collection {} 1: pointer to message to read 3: on successful write, moves pointer to next
- 17. MongoDB Connector Configuring the connector as a source and sink Aging data out to cost-effective cloud storage with Atlas Online Archive
- 18. MongoDB Atlas Connector in Confluent Cloud!
- 19. Two Ways to Stream 1919 SELF-MANAGED SOFTWARE Confluent Platform The Enterprise Distribution of Apache Kafka Deploy on any platform, on-prem or cloud VM FULLY-MANAGED SOFTWARE Confluent Cloud Apache Kafka Re-Engineered for the Cloud Available on the leading public clouds Option to purchase via GCP marketplace Both: Subscription products where price scales with usage
- 20. Confluent Cloud MongoDB Atlas Connector
- 21. Preview Limitations Source • MongoDB Atlas only. • Public access (0.0.0.0/0) must be allowed by Atlas. • Output of this connector is String. • 1 task / connector limitation • Pipeline parameter not available Sink ● MongoDB Atlas only. ● The MongoDB Atlas and Kafka cluster must be in the same region. ● Public access (0.0.0.0/0) must be allowed by Atlas. ● 1 task / connector limitation NOTE: These preview limitations only apply to the MongoDB Atlas connector in the Confluent Cloud. The MongoDB Connector for Apache Kafka can be used in a self-hosted environment with MongoDB Atlas without these network restrictions.
- 22. Using MongoDB Atlas with Confluent Cloud
- 23. Want to Know More? https://www.confluent.io/resources/confluent-platform-reference-architecture-mongodb/
- 24. Making your life easier Wrap Up Document Model and MQL The fastest way to innovate MongoDB Connector for Apache Kafka Gold Certified - Available on Confluent Cloud! MongoDB Atlas: More than a database - a data platform
- 25. Thank you That’s all folks Robert Walters | [email protected] | MongoDB