Kafka for Real-Time Replication between Edge and Hybrid Cloud
1 like1,588 views
Not all workloads allow cloud computing. Low latency, cybersecurity, and cost-efficiency require a suitable combination of edge computing and cloud integration. This session explores architectures and design patterns for software and hardware considerations to deploy hybrid data streaming with Apache Kafka anywhere. A live demo shows data synchronization from the edge to the public cloud across continents with Kafka on Hivecell and Confluent Cloud.
Ad
More Related Content
What's hot (20)
Similar to Kafka for Real-Time Replication between Edge and Hybrid Cloud (20)
Ad
More from Kai Wähner (20)
Ad
Recently uploaded (20)
![Get & Download Wondershare Filmora Crack Latest [2025]](https://cdn.slidesharecdn.com/ss_thumbnails/revolutionizingresidentialwi-fi-250422112639-60fb726f-250429170801-59e1b240-thumbnail.jpg?width=560&fit=bounds)
Kafka for Real-Time Replication between Edge and Hybrid Cloud
- 1. How to Deploy a Kafka Cluster in Production onto Your Desk or Anywhere! IoT Use Cases, Architectures, and Live Demo for Data in Motion powered by Apache Kafka Kai Waehner Field CTO [email protected] linkedin.com/in/kaiwaehner confluent.io kai-waehner.de @KaiWaehner Dominik Pilat VP of Customer Success & Field CTO [email protected] linkedin.com/in/dpilat hivecell.com
- 2. Data in Motion Everywhere with Apache Kafka in Hybrid Edge and Cloud Infrastructures Agenda • Data in Motion at the Edge and in Hybrid Architectures • Live Demo – Hybrid Data Streaming with Confluent on Hivecell and Confluent Cloud • Designing an Edge Solution • Next steps for Kafka at the Edge
- 3. Data in Motion Everywhere with Apache Kafka in Hybrid Edge and Cloud Infrastructures Agenda • Data in Motion at the Edge and in Hybrid Architectures • Live Demo – Hybrid Data Streaming with Confluent on Hivecell and Confluent Cloud • Designing an Edge Solution • Next steps for Kafka at the Edge
- 4. Data in Motion Everywhere with Apache Kafka in Hybrid Edge and Cloud Infrastructures Real-time Data beats Slow Data. Manufacturing Sensor diagnostics MES/ERP Integration Reporting Edge Computing Condition Monitoring Predictive Maintenance Quality Assurance Logistics Supply Chain Inventory management Track & Trace Context-specific routing Cybersecurity Threat detection Intrusion detection Incident response Fraud detection
- 5. Data in Motion Everywhere with Apache Kafka in Hybrid Edge and Cloud Infrastructures AWS Cloud Outage hit Disney World Visitors… https://www.cnet.com/tech/services-and-software/disney-parks-were-already-facing-heat-from-fans-then-an-aws-outage-came-along/
- 6. Data in Motion Everywhere with Apache Kafka in Hybrid Edge and Cloud Infrastructures Apache Kafka is the Platform for Data in Motion MES ERP Sensors Mobile Customer 360 Real-time Alerting System Data warehouse Producers Consumers Streams and storage of real time events Stream processing apps Connectors Connectors Stream processing apps Supplier Alert Forecast Inventory Customer Order 6
- 7. Data in Motion Everywhere with Apache Kafka in Hybrid Edge and Cloud Infrastructures MQTT Data Historian STREAM PROCESSING CONNECTORS Example Architecture for Data in Motion with Event Streaming ksqlDB KStreams Bi-directional real-time integration between the OT and IT worlds APP Oracle DB ORACLE CDC SOURCE CONNECTOR APP
- 8. Data in Motion Everywhere with Apache Kafka in Hybrid Edge and Cloud Infrastructures Global Event Streaming Streaming Replication between Kafka Clusters Bridge to Databases, Data Lakes, Apps, APIs, SaaS Aggregate Small Footprint Edge Deployments with Replication (Aggregation) Simplify Disaster Recovery Operations with Multi-Region Clusters for RPO=0 and RTO~0 Stream Data Globally with Replication and Cluster Linking 8
- 9. Data in Motion Everywhere with Apache Kafka in Hybrid Edge and Cloud Infrastructures What is the “Edge” for Kafka? • Edge is NOT a data center • Kafka clients AND the Kafka broker(s) • Offline business continuity • Often 100+ locations • Low-footprint and low-touch • Hybrid integration Example: Single broker, 1 GB Ram 100 MB/sec
- 10. Data in Motion Everywhere with Apache Kafka in Hybrid Edge and Cloud Infrastructures Data Processing at the Edge Time P C1 C2 C3 Know-your-customer Loyalty app, predictive behavior, … Estimated time of arrival Connect to the gaming server for kids Play games, earn rewards, communicate with other kids in the train, … Always on (even “offline”) Replayability Cost-efficiency Low latency
- 11. Data in Motion Everywhere with Apache Kafka in Hybrid Edge and Cloud Infrastructures Shipping Industry Marine, Oil Transport, Vessel Fleet, Shipping Line, Drones Real-time Operations, Logistics, Predictive Maintenance, Security Customer Data Crew, Cargo Vessel Data Fuel Consumption, Speed, Planned Maintenance Automatic Identification System (AIS) Unique Identification, Position, Course, Weather, Draft Drone Data Deliveries, Survey/Inspection of Assets such as Oil Rigs, Pipelines, Offshore Turbines Edge Analytics Bidirectional Edge to Cloud Integration Data Ingestion Stream Processing Data Integration Logistics Track&Trace Routing Monitoring Alerting Command&Control Batch Analytics Reporting Machine Learning Backend Systems Oracle, SAP, OSIsoft PI, etc. X = Event Streaming X = Other Technologies Bi-Directional Hybrid Cloud Replication
- 12. Data in Motion Everywhere with Apache Kafka in Hybrid Edge and Cloud Infrastructures Royal Caribbean - Offline Edge for Swimming Retail Stores https://www.confluent.io/kafka-summit-lon19/seamless-guest-experience-with-kafka-streams/
- 13. Data in Motion Everywhere with Apache Kafka in Hybrid Edge and Cloud Infrastructures Devon Energy Oil & Gas Industry Improve drilling and well completion operations Edge stream processing/analytics + closed-loop control ready Vendor agnostic (pumping, wireline, coil, offset wells, drilling operations, producing wells) Replication to the cloud in real-time at scale Cloud agnostic (AWS, GCP, Azure) Source: Energy in Data - Powered by AAPG, SEG & SPE: energyindata.org
- 14. Data in Motion Everywhere with Apache Kafka in Hybrid Edge and Cloud Infrastructures CRM Real-Time Location System (RTLS) for Asset Tracking Customer data Advanced Planning and Scheduling (APS) Manager Get report API Customer Customer Customer data Truck schedule Payment data Route details Streams of real time events Event Streaming in Hybrid Industrial IoT Wavelength Public Cloud VPC Campus 5G Telco Carrier 5G Smart Factory Edge Computing
- 15. Data in Motion Everywhere with Apache Kafka in Hybrid Edge and Cloud Infrastructures Analytics Database Data Lake CRM Kafka as Data Historian Continuous real time data ingestion, processing and monitoring 24/7 at scale MQTT Modbus Firewall Siemens S7 Kafka Connect Integration | Storage | Processing | Access | Cloud | Security Storage Kafka Streams / ksqlDB OPC-UA Stateless + Stateful Kafka Connect Kafka Clients Java, C++. Python, Go, … REST Proxy SCADA DCS ERP MES Syslog File X Y Z A B C Cloud Factory
- 16. Data in Motion Everywhere with Apache Kafka in Hybrid Edge and Cloud Infrastructures Condition Monitoring and Predictive Maintenance Stateless and stateful stream processing for real-time data correlation with Kafka-native tools (Kafka Streams / ksqlDB) 8 9 1 0 1 1 1 2 1 3 1 4 1 5 1 6 3 4 5 6 7 2 1 Time Sensor Events
- 17. Data in Motion Everywhere with Apache Kafka in Hybrid Edge and Cloud Infrastructures Condition Monitoring and Predictive Maintenance Stateless and stateful stream processing for real-time data correlation with Kafka-native tools (Kafka Streams / ksqlDB) 8 9 1 0 1 1 1 2 1 3 1 4 1 5 3 4 5 6 7 2 1 Time Sensor Events 1 6 Condition Monitoring (Temperature Spikes) Stateless Filter Above-Threshold Events Streams builder .stream(”temperature-sensor") .filter((key, sensor-data) -> sensor-data.temperature > 100) .to(”temperature-spikes");
- 18. Data in Motion Everywhere with Apache Kafka in Hybrid Edge and Cloud Infrastructures Condition Monitoring and Predictive Maintenance Stateless and stateful stream processing for real-time data correlation with Kafka-native tools (Kafka Streams / ksqlDB) 8 9 1 0 1 1 1 2 1 3 1 4 1 5 1 6 3 4 5 6 7 2 1 Time Sensor Events Predictive Maintenance (Continuous Anomaly Detection) Stateful Correlation of Events CREATE TABLE anomaly_detection AS SELECT temperature_spike_id, COUNT(*) AS total_spikes, AVG(temperature) AS avg_temperature FROM sensor-data WINDOW TUMBLING (SIZE 1 HOUR) GROUP BY temperature_spike_id EMIT CHANGES;
- 19. Data in Motion Everywhere with Apache Kafka in Hybrid Edge and Cloud Infrastructures Condition Monitoring and Predictive Maintenance Stateless and stateful stream processing for real-time data correlation with Kafka-native tools (Kafka Streams / ksqlDB) 8 9 1 0 1 1 1 2 1 3 1 4 1 5 1 6 3 4 5 6 7 2 1 Time Sensor Events Predictive Maintenance (Continuous Anomaly Detection) Real-time Machine Learning CREATE STREAM anomaly_detection AS SELECT sensor_id, detect_anomaly(sensor_values) FROM machine; TensorFlow model embedded in User Defined Function (UDF)
- 20. Data in Motion Everywhere with Apache Kafka in Hybrid Edge and Cloud Infrastructures Agenda • Data in Motion at the Edge and in Hybrid Architectures • Live Demo – Hybrid Data Streaming with Confluent on Hivecell and Confluent Cloud • Designing an Edge Solution • Next steps for Kafka at the Edge
- 21. Data in Motion Everywhere with Apache Kafka in Hybrid Edge and Cloud Infrastructures Edge Cloud Texas Bavaria Hybrid Data Streaming: Edge and Cloud
- 22. Data in Motion Everywhere with Apache Kafka in Hybrid Edge and Cloud Infrastructures Texas Texas data Streams of real time events Texas data Streams of real time events Bavaria data Bavaria Hybrid Data Streaming: Edge and Cloud
- 23. Data in Motion Everywhere with Apache Kafka in Hybrid Edge and Cloud Infrastructures Agenda • Data in Motion at the Edge and in Hybrid Architectures • Live Demo – Hybrid Data Streaming with Confluent on Hivecell and Confluent Cloud • Designing an Edge Solution • Next steps for Kafka at the Edge
- 24. Data in Motion Everywhere with Apache Kafka in Hybrid Edge and Cloud Infrastructures Conditions of edge environments • No data closet • Or data closet is full, overheating and under powered • Limited or unreliable power supply • Limited number of ports • No IT staff on site • Hundreds or thousands of locations • Limited network bandwidth • Demand for compute power always increasing
- 25. Data in Motion Everywhere with Apache Kafka in Hybrid Edge and Cloud Infrastructures Requirements for an edge solution No special equipment for power, air conditioning or networking No technicians required onsite to install and configure Start with smallest footprint possible to demonstrate ROI Easily add more compute power as workload expands Run distributed software for containers, messaging and machine learning Monitor, manage and upgrade centrally, even when behind a firewall
- 26. Data in Motion Everywhere with Apache Kafka in Hybrid Edge and Cloud Infrastructures Hivecell ✓ No special equipment ✓ No technicians required ✓ Start with smallest footprint ✓ Easily add more compute power ✓ Run distributed software ✓ Monitor, manage and update remotely
- 27. Data in Motion Everywhere with Apache Kafka in Hybrid Edge and Cloud Infrastructures Agenda • Data in Motion at the Edge and in Hybrid Architectures • Live Demo – Hybrid Data Streaming with Confluent on Hivecell and Confluent Cloud • Designing an Edge Solution • Next steps for Kafka at the Edge
- 28. Data in Motion Everywhere with Apache Kafka in Hybrid Edge and Cloud Infrastructures The Rise of Data in Motion for Edge and Hybrid Scenarios 2010 Apache Kafka created at LinkedIn by Confluent founders 2014 2020 80% Fortune 100 Companies trust and use Apache Kafka 29 2017
- 29. Data in Motion Everywhere with Apache Kafka in Hybrid Edge and Cloud Infrastructures I N V E S T M E N T & T I M E V A L U E 3 4 5 1 2 Event Streaming Maturity Model Initial Awareness / Pilot (1 Kafka Cluster) Start to Build Pipeline / Deliver 1 New Outcome (1 Kafka Cluster) Mission-Critical Deployment (Stretched, Hybrid, Multi-Region) Build Contextual Event-Driven Apps (Stretched, Hybrid, Multi-Region) Central Nervous System (Global Kafka) Product, Support, Training, Partners, Technical Account Management... 30
- 30. Data in Motion Everywhere with Apache Kafka in Hybrid Edge and Cloud Infrastructures Car Engine Car Self-driving Car Confluent completes Apache Kafka. Cloud-native. Everywhere.
- 31. Data in Motion Everywhere with Apache Kafka in Hybrid Edge and Cloud Infrastructures Container orchestration at the edge Enterprise streaming at the edge HIVECELL = PAAS AT THE EDGE Hyperconverged infrastructure at the edge HCI
- 32. Data in Motion Everywhere with Apache Kafka in Hybrid Edge and Cloud Infrastructures EDGE AS A SERVICE. EVERYTHING YOU NEED IS INCLUDED. • Kubernetes and Confluent as a service • Framework licenses/support/updates • Hive Control - Provision/monitor system • Operating system (OS) support/updates • Hardware support/replacements • Evergreen Hardware: upgrades every 3 years User just uses Kafka. At the edge.
- 33. Data in Motion Everywhere with Apache Kafka in Hybrid Edge and Cloud Infrastructures 34 Hivecell and Confluent enable Kafka at the Edge. Reliable. Scalable. Simple. Conclusion
- 34. Data in Motion Everywhere with Apache Kafka in Hybrid Edge and Cloud Infrastructures Stay in touch! 35 Confluent Cloud cnfl.io/confluent-cloud 1 hour free edge consulting session with Dominik Pilat https://hivecell.com/contact/
- 35. Kai Waehner Field CTO Confluent [email protected] @KaiWaehner confluent.io kai-waehner.de linkedin.com/in/kaiwaehner Questions? Feedback? Let’s connect! Dominik Pilat VP of Customer Success Field CTO Hivecell [email protected] linkedin.com/in/dpilat hivecell.com