Connected Vehicles and V2X with Apache Kafka
2 likes3,639 views
The document discusses the use of Apache Kafka for real-time data streaming in connected vehicles, highlighting industry trends and technological advancements. It outlines the benefits of streaming analytics, mobility services, and the infrastructure necessary for processing large amounts of connected car data. A live demo is mentioned, showcasing the capabilities of processing data from 100,000 connected cars using various technologies including Kafka, MQTT, and TensorFlow.
Connected Vehicles and V2X with Apache Kafka
- 1. @KaiWaehner - www.kai-waehner.de – V2X and Connected Vehicles with Apache Kafka Real-time Streaming Analytics with 100.000 Connected Cars MQTT, Kafka, Kubernetes and TensorFlow in Action Kai Waehner Field CTO [email protected] @KaiWaehner www.confluent.io www.kai-waehner.de linkedin.com/in/kaiwaehner
- 2. @KaiWaehner - www.kai-waehner.de – V2X and Connected Vehicles with Apache Kafka Agenda 2 1. Industry Trends 2. Data in Motion with Apache Kafka 3. Connected Vehicles and Mobility Services 4. Live Demo – 100.000 Connected Cars
- 3. @KaiWaehner - www.kai-waehner.de – V2X and Connected Vehicles with Apache Kafka Agenda 3 1. Industry Trends 2. Data in Motion with Apache Kafka 3. Connected Vehicles and Mobility Services 4. Live Demo – 100.000 Connected Cars
- 4. @KaiWaehner - www.kai-waehner.de – V2X and Connected Vehicles with Apache Kafka Industry Trends - Connected World 4 People on the Internet
- 5. @KaiWaehner - www.kai-waehner.de – V2X and Connected Vehicles with Apache Kafka “Around 10% of enterprise- generated data is created and processed outside a traditional centralized data center or cloud.” ... By 2025, predicts this figure will reach 75% Industry Trends - Source of Data is Changing
- 6. @KaiWaehner - www.kai-waehner.de – V2X and Connected Vehicles with Apache Kafka Use Cases for Connected Vehicles 6
- 7. @KaiWaehner - www.kai-waehner.de – V2X and Connected Vehicles with Apache Kafka Agenda 7 1. Industry Trends 2. Data in Motion with Apache Kafka 3. Connected Vehicles and Mobility Services 4. Live Demo – 100.000 Connected Cars
- 8. @KaiWaehner - www.kai-waehner.de – V2X and Connected Vehicles with Apache Kafka Apache Kafka is a Platform for Data in Motion MES ERP Sensors Mobile Customer 360 Real-time Alerting System Data warehouse Producers Consumers Streams and storage of real time events Stream processing apps Connectors Connectors Stream processing apps Supplier Alert Forecast Inventory Customer Order 8
- 9. @KaiWaehner - www.kai-waehner.de – V2X and Connected Vehicles with Apache Kafka Kafka Java Client Kafka Cluster Monolith Kafka Decouples Storage and Compute Cloud DWH Snowflake Connector Schema Registry Storage Compute Compute (+ non-Kafka Storage) KSQL App KSQL App KSQL App Compute Compute Compute
- 10. @KaiWaehner - www.kai-waehner.de – V2X and Connected Vehicles with Apache Kafka Global Event Streaming Streaming Replication between Kafka Clusters Bridge to Databases, Data Lakes, Apps, APIs, SaaS Aggregate Small Footprint Edge Deployments with Replication (Aggregation) Simplify Disaster Recovery Operations with Multi-Region Clusters with RPO=0 and RTO=0 Stream Data Globally with Replication and Cluster Linking 10
- 11. @KaiWaehner - www.kai-waehner.de – V2X and Connected Vehicles with Apache Kafka Confluent Tiered Storage for Kafka 11 (Only available in Confluent Platform) Store data forever Hot and cold storage Cheap object store Easy scale up/down No changes in clients
- 12. @KaiWaehner - www.kai-waehner.de – V2X and Connected Vehicles with Apache Kafka Tiered Storage for Apache Kafka KIP-405 – Add Tiered Storage Support to Kafka Confluent is actively working on this with the open source community - Uber is leading this initiative Confluent Tiered Storage is available today in Confluent Platform and used under the hood in Confluent Cloud https://cwiki.apache.org/confluence/display/KAFKA/KIP-405%3A+Kafka+Tiered+Storage
- 13. @KaiWaehner - www.kai-waehner.de – V2X and Connected Vehicles with Apache Kafka Queries for Reprocessing Historical Events Give me all events from time A to time B Real-time Producer Time • New consumer application • Error-handling • Compliance / regulatory processing • Query and analyze existing events • Schema changes in analytics platform • Model training Real-time Consumer Consumer of Historical Data
- 14. @KaiWaehner - www.kai-waehner.de – V2X and Connected Vehicles with Apache Kafka Agenda 14 1. Industry Trends 2. Data in Motion with Apache Kafka 3. Connected Vehicles and Mobility Services 4. Live Demo – 100.000 Connected Cars
- 15. @KaiWaehner - www.kai-waehner.de – V2X and Connected Vehicles with Apache Kafka Connected Car Infrastructure https://www.youtube.com/watch?v=yGLKi3TMJv8 • Real Time Data Analysis • Swarm Intelligence • Collaboration with Partners • Predictive AI • …
- 16. @KaiWaehner - www.kai-waehner.de – V2X and Connected Vehicles with Apache Kafka ‘My Porsche’ A digital service platform for customers, fans, and enthusiasts 16 https://medium.com/porschedev
- 17. @KaiWaehner - www.kai-waehner.de – V2X and Connected Vehicles with Apache Kafka https://www.confluent.io/thank-you/uber-kafka-uber-worlds-realtime-transit-infrastructure/ https://www.confluent.io/thank-you/stream-processing-kafka-uber/ Trillions of messages and multiple petabytes of data per day 17
- 18. @KaiWaehner - www.kai-waehner.de – V2X and Connected Vehicles with Apache Kafka Ride-Hailing More than just Messaging! Data Correlation in Real-Time for map-matching, ETA, cost calculation, and much more… https://eng.lyft.com/a-new-real-time-map-matching-algorithm-at-lyft-da593ab7b006 18
- 19. @KaiWaehner - www.kai-waehner.de – V2X and Connected Vehicles with Apache Kafka Cross-Company Streaming Exchange Streaming Replication and API Management MirrorMaker 2 Confluent Replicator Cluster Linking Tier 2 Supplier OEM Streaming integration between companies API Management (REST et al) is not appropriate for streaming data Infosec and politics are your biggest hurdle Tier 1 Supplier
- 20. @KaiWaehner - www.kai-waehner.de – V2X and Connected Vehicles with Apache Kafka Here Technologies Captures location content such as road networks, buildings, parks and traffic patterns Sells or licenses mapping content, along with map related navigation and location services to other businesses https://developer.here.com/documentation/data-client-library/dev_guide/client/direct-kafka.html
- 21. @KaiWaehner - www.kai-waehner.de – V2X and Connected Vehicles with Apache Kafka CRM Real-Time Location System (RTLS) for Asset Tracking Customer data Advanced Planning and Scheduling (APS) Manager Get report API Customer Customer Customer data Truck schedule Payment data Route details Streams of real time events Customer data Train schedule Payment data Loyalty information Streams of real time events Customer data Train schedule Payment data Loyalty information Streams of real time events Postmodern Asset Tracking Wavelength Public Cloud VPC Carrier #1 5G Wavelength Carrier #2 5G
- 22. @KaiWaehner - www.kai-waehner.de – V2X and Connected Vehicles with Apache Kafka Event Streaming The Foundation for a Smart City Mobility Services Citizen Services Smart Buildings Cybersecurity Smart Grid Weather Data Citizen Data Traffic Data Security Alerts Streams of real time events Personal Data Weather Service Connected Cars Routing Service Traffic Alerts Surveillance 22
- 23. @KaiWaehner - www.kai-waehner.de – V2X and Connected Vehicles with Apache Kafka Open API Platform for Car Data
- 24. @KaiWaehner - www.kai-waehner.de – V2X and Connected Vehicles with Apache Kafka The Smarter City Nervous System Producers Consumers • Client Libraries • Kafka Connectors • Client Libraries • Kafka Connectors X Consumer Groups 24 Streams
- 25. @KaiWaehner - www.kai-waehner.de – V2X and Connected Vehicles with Apache Kafka The Smarter City Nervous System Topic (observation-raw) Metadata and Geo (lat/long) Enrichment Topic (observation-meta) Streams Health Score & Incident Detection Topic (observation-out) Elasticsearch Kafka Connect Elasticsearch Sink Connector 25
- 26. @KaiWaehner - www.kai-waehner.de – V2X and Connected Vehicles with Apache Kafka Agenda 26 1. Industry Trends 2. Data in Motion with Apache Kafka 3. Connected Vehicles and Mobility Services 4. Live Demo – 100.000 Connected Cars
- 27. @KaiWaehner - www.kai-waehner.de – V2X and Connected Vehicles with Apache Kafka Streaming Analytics with MQTT, Kafka and TensorFlow 27 MQTT Proxy MongoDB Storage MongoDB Dashboards Search Analytics Kafka Cluster Kafka Connect Car Sensors Kafka Streams Application All Data Critical Data Ingest Data Potential Detect TensorFlow Train Analytic Model ksqlDB Analytic Model Preprocess Data Consume Data Deploy Analytic Model Tiered Storage Mobile App BI Tool Kafka Ecosystem TensorFlow Other Components
- 28. @KaiWaehner - www.kai-waehner.de – V2X and Connected Vehicles with Apache Kafka Direct streaming ingestion for model training with TensorFlow I/O + Kafka Plugin (no additional data storage like S3 or HDFS required!) Time Model B Model A Producer Distributed Commit Log Streaming Ingestion and Model Training without the need for another Data Lake https://github.com/tensorflow/io 28 Model X (at a later time)
- 29. @KaiWaehner - www.kai-waehner.de – V2X and Connected Vehicles with Apache Kafka “CREATE STREAM AnomalyDetection AS SELECT sensor_id, detectAnomaly(sensor_values) FROM car_engine;“ User Defined Function (UDF) Model Deployment with Apache Kafka, ksqlDB and TensorFlow 29
- 30. @KaiWaehner - www.kai-waehner.de – V2X and Connected Vehicles with Apache Kafka Live Demo: 100,000 Connected Cars Kafka, ksqlDB, MQTT and TensorFlow in Action https://github.com/kaiwaehner/hivemq-mqtt-tensorflow-kafka-realtime-iot-machine-learning-training-inference 30
- 31. @KaiWaehner - www.kai-waehner.de – V2X and Connected Vehicles with Apache Kafka Why Confluent?
- 32. @KaiWaehner - www.kai-waehner.de – V2X and Connected Vehicles with Apache Kafka I N V E S T M E N T & T I M E V A L U E 3 4 5 1 2 Event Streaming Maturity Model Initial Awareness / Pilot (1 Kafka Cluster) Start to Build Pipeline / Deliver 1 New Outcome (1 Kafka Cluster) Mission-Critical Deployment (Stretched, Hybrid, Multi-Region) Build Contextual Event-Driven Apps (Stretched, Hybrid, Multi-Region) Central Nervous System (Global Kafka) Product, Support, Training, Partners, Technical Account Management... 32
- 33. @KaiWaehner - www.kai-waehner.de – V2X and Connected Vehicles with Apache Kafka The Rise of Data in Motion 2010 Apache Kafka created at LinkedIn by Confluent founders 2014 2020 80% Fortune 100 Companies trust and use Apache Kafka 33
- 34. @KaiWaehner - www.kai-waehner.de – V2X and Connected Vehicles with Apache Kafka Data in Motion with Confluent Freedom of Choice Committer-driven Expertise Open Source | Community licensed Fully Managed Cloud Service Self-managed Software Training Partners Enterprise Support Professional Services ARCHITECT OPERATOR DEVELOPER EXECUTIVE Apache Kafka Dynamic Performance & Elasticity Self-Balancing Clusters | Tiered Storage Flexible DevOps Automation Operator | Ansible GUI-driven Mgmt & Monitoring Control Center | Proactive Support Event Streaming Database ksqlDB Rich Pre-built Ecosystem Connectors | Hub | Schema Registry Multi-language Development Non-Java Clients | REST Proxy Admin REST APIs Global Resilience Multi-Region Clusters | Replicator Cluster Linking Data Compatibility Schema Registry | Schema Validation Enterprise-grade Security RBAC | Secrets | Audit Logs TCO / ROI Revenue / Cost / Risk Impact Complete Engagement Model Efficient Operations at Scale Unrestricted Developer Productivity Production-stage Prerequisites Partnership for Business Success
- 35. @KaiWaehner - www.kai-waehner.de – V2X and Connected Vehicles with Apache Kafka Kai Waehner Field CTO [email protected] @KaiWaehner www.kai-waehner.de www.confluent.io linkedin.com/in/kaiwaehner Questions? Feedback? Let’s connect!