Event Broker (Kafka) in a Modern Data Architecture
1 like1,197 views
Today's modern data architectures and the their implementations contain an Event Broker. What are the benefits of placing an Event Broker in a Modern Data (Analytics) Architecture? What exactly is an Event Broker and what capabilities should it provide? Why is Apache Kafka the most popular realisation of an Event Broker? These and many other questions will be answered in this session. The talk will start with a vendor-neutral definition of the capabilities of an Event Broker. Then the session will highlight the different architecture styles which can be supported using an Event Broker (Kafka), such as Streaming Data Integration, Stream Analytics and Decoupled Event-Driven Applications and how can these be combined into a unified architecture, making the Event Broker the central nervous system of an enterprise architecture. We will end with an overview of the Kafka ecosystem and a placement of the various components onto the Modern Data (Analytics) Architecture.
![Event Broker
Stream
Analytics
Stream Data
Integration
Stream Data
Integration
Vehicle
Environ
mental
Streaming Data Sources
Ware
house
Using Stream Analytics
• Time Windowed State Management
• Stream-to-Table Joins
• Stream-to-Stream Joins
• Event Pattern Detection
• Machine Learning Model Execution
(Inference)
[1]
E-Comm
erce
Stream Data
Integration
Gateway
19.11 – 13:00 – Kafka Livedemo: Umsetzung einer Streaminglösung #slideless](https://image.slidesharecdn.com/event-broker-kafka-in-modern-data-architecture-201118133637/85/Event-Broker-Kafka-in-a-Modern-Data-Architecture-15-320.jpg)
![Event Broker
Stream
Analytics
Streaming
Visualize
Stream Data
Integration
Stream Data
Integration
Vehicle
Environ
mental
Ware
house
Using Streaming Visualization
• ksqlDB pull queries or Kafka Streams
Interactive Queries allow to query state of
stream processor
[2]
E-Comm
erce
Stream Data
Integration
Streaming Data Sources
Stream Data
Integration
Gateway](https://image.slidesharecdn.com/event-broker-kafka-in-modern-data-architecture-201118133637/85/Event-Broker-Kafka-in-a-Modern-Data-Architecture-19-320.jpg)
![Event Broker
Stream
Analytics
Stream Data
IntegrationCDC
Stream Data
Integration
CDC
Streaming
Visualize
Stream Data
Integration
1st Micro
service
2nd Micro
service
Stream Data
Integration
Stream Data
Integration
Stream Data
Integration
NoSQL
RDBMS
Micro-Batch
Visualize
Modern Event-Driven Apps (aka. Microservices)
• 2nd microservice
consumes events
from 1st Bootstrap
from event history
[3]
Stream Data
Integration
Streaming Data Sources
Gateway
Legacy Data Sources
Monolithic
System
Machine
IIoT
Vehicle
Environ
mental
Ware
house
E-Comm
erce](https://image.slidesharecdn.com/event-broker-kafka-in-modern-data-architecture-201118133637/85/Event-Broker-Kafka-in-a-Modern-Data-Architecture-25-320.jpg)
![Event Broker
Stream
Analytics
Stream Data
IntegrationCDC
Stream Data
Integration
CDC
Streaming
Visualize
Stream Data
Integration
1st Micro
service
2nd Micro
service
Stream Data
Integration
Stream Data
Integration
Stream Data
Integration
NoSQL
RDBMS
Micro-Batch
Visualize
Bi-Directional Legacy Systems Integration
[4]AQ
Stream Data
Integration
Streaming Data Sources
Gateway
Legacy Data Sources
Monolithic
System
Machine
IIoT
Vehicle
Environ
mental
Ware
house
E-Comm
erce](https://image.slidesharecdn.com/event-broker-kafka-in-modern-data-architecture-201118133637/85/Event-Broker-Kafka-in-a-Modern-Data-Architecture-26-320.jpg)
![Kai Waehner [Confluent] | Real-Time Streaming Analytics with 100,000 Cars Usi...](https://cdn.slidesharecdn.com/ss_thumbnails/202006digitaltwinkafkainfluxdb-200624103300-thumbnail.jpg?width=560&fit=bounds)
Ad
More Related Content
What's hot (20)
Similar to Event Broker (Kafka) in a Modern Data Architecture (20)
Ad
More from Guido Schmutz (16)
Ad
Recently uploaded (20)
Event Broker (Kafka) in a Modern Data Architecture
- 1. http://[email protected]@gschmutz Event Broker (Kafka) in Modern Data Architecture Guido Schmutz
- 2. Guido Working at Trivadis for more than 23 years Consultant, Trainer, Platform Architect for Java, Oracle, SOA and Big Data / Fast Data Oracle Groundbreaker Ambassador & Oracle ACE Director @gschmutz guidoschmutz.wordpress.com 210th edition
- 4. What exactly is an Event Broker?
- 5. Event Broker Event Broker – as a starting point
- 6. Event Broker Event Broker – an Infrastructure with these capabilities 1. topic semantics (publish/subscribe) – message can be consumed by 0 – n consumers 2. queue semantics – messages can be consumed by exactly one consumer 3. horizontally scalable – throughput increases with more resources 4. auto-scaling – up and down-scaling upon load 5. highly available – no single point of failure 6. Control/handle back-pressure 7. durable – messages may not be lost 8. schema-less – no knowledge on message content and format 9. Efficient support of Stream and Batch Consumers (offline and with large Backlog) 10. (Unlimited) Retention of messages (long term storage) 11. Guaranteed ordering of messages 12. Support re-consumption of events 13. Access control – control over who can produce and consume which events 14. interoperable – support for different clients
- 7. Kafka – the most popular Event Broker
- 8. Kafka – the most popular Event Broker Kafka Cluster Consumer 1 Consumer 2 Broker 1 Broker 2 Broker 3 Zookeeper Ensemble ZK 1 ZK 2ZK 3 Schema Registry Service 1 Management Control Center Kafka Manager KAdmin Producer 1 Producer 2 kafkacat Data Retention: • Never • Time (TTL) or Size-based • Log-Compacted based Producer3Producer3 ConsumerConsumer 3 1. topic semantics 2. queue semantics 3. horizontally scalable 4. auto-scaling 5. highly available 6. back-pressure 7. durable 8. schema-less/opaque 9. Stream and Batch Consumers 10. (Unlimited) Retention 11. Guaranteed ordering 12. re-consumption of events 13. Access Control 14. Interoperable
- 9. • Cloud Services • Cloud Services with Kafka API • Kafka Cloud Services Event Broker - Kafka Alternatives? Cloud Services? • traditional Message Brokers (with a lot of limitations regarding Event Broker capabilities) • Apache Pulsar • Solace • Pravega (Dell Streaming Platform) • Oracle AQ (Kafka API coming) AQ
- 10. Event Broker - core building block of a Modern Data Architecture
- 11. Event Broker Event Broker – as a starting point Vehicle Environ mental Streaming Data Sources Ware house E-Comm erce
- 12. Event Broker Stream Data Integration Stream Data Integration Vehicle Environ mental Streaming Data Sources Ware house Using Stream Data Integration for integrating various data sources E-Comm erce Stream Data Integration
- 13. Event Broker Stream Data Integration Stream Data Integration Vehicle Environ mental Ware house Gateway Using Edge Computing and Stream Data Integration • MQTT as a gateway to Kafka E-Comm erce Stream Data Integration Streaming Data Sources
- 14. Stream Data Integration – Kafka Connect / StreamSets • declarative style, simple data flows • framework is part of Apache Kafka • Many connectors available • Single Message Transforms (SMT) • GUI-based, drag-and drop Data Flow Pipelines • Both stream and batch processing (micro- batching) • custom sources, sinks, processors
- 15. Event Broker Stream Analytics Stream Data Integration Stream Data Integration Vehicle Environ mental Streaming Data Sources Ware house Using Stream Analytics • Time Windowed State Management • Stream-to-Table Joins • Stream-to-Stream Joins • Event Pattern Detection • Machine Learning Model Execution (Inference) [1] E-Comm erce Stream Data Integration Gateway 19.11 – 13:00 – Kafka Livedemo: Umsetzung einer Streaminglösung #slideless
- 16. Stream Analytics - Kafka Streams • Programmatic API, “just” a Java library • fault-tolerant local state • Fixed, Sliding and Session Windowing • Stream-Stream / Stream-Table Joins • At-least-once and exactly-once • Stream Processing with zero coding using SQL-like language • built on top of Kafka Streams • interactive (CLI) and headless (cmd file) trucking_ driver Kafka Broker Java Application Kafka Streams ksqlDB trucking_ driver Kafka Broker ksqlDB Engine Kafka Streams ksqlDB REST Commands ksqlDB CLI push pull
- 17. Event Broker Stream Analytics Stream Data Integration Stream Data Integration Vehicle Environ mental Ware house Using Stream Analytics • Push results back to new topic so other interested parties can use it too! E-Comm erce Stream Data Integration Streaming Data Sources Gateway
- 18. Event Broker Stream Analytics Stream Data Integration Stream Data Integration Vehicle Environ mental Ware house Using Stream Data Integration to callback to Data Source (to Actuator) E-Comm erce Stream Data Integration Streaming Data Sources Gateway
- 19. Event Broker Stream Analytics Streaming Visualize Stream Data Integration Stream Data Integration Vehicle Environ mental Ware house Using Streaming Visualization • ksqlDB pull queries or Kafka Streams Interactive Queries allow to query state of stream processor [2] E-Comm erce Stream Data Integration Streaming Data Sources Stream Data Integration Gateway
- 20. Event Broker Stream Analytics Monolithic System Stream Data IntegrationCDC Streaming Visualize Stream Data Integration Stream Data Integration Stream Data Integration Vehicle Environ mental Ware house (Right-Time) Legacy Systems Integration • Stream-to-Table join E-Comm erce Stream Data Integration Streaming Data Sources Gateway Legacy Data Sources
- 21. Kafka as an Event Broker
- 22. Event Broker Stream Analytics Monolithic System Machine IIoT Stream Data IntegrationCDC Stream Data Integration CDC Streaming Visualize Stream Data Integration Stream Data Integration Stream Data Integration Vehicle Environ mental Ware house (Right-Time) Legacy Systems Integration E-Comm erce Stream Data Integration Streaming Data Sources Gateway Legacy Data Sources
- 23. Event Broker Stream Analytics Stream Data IntegrationCDC Stream Data Integration CDC Streaming Visualize Stream Data Integration Stream Data Integration Stream Data Integration Stream Data Integration NoSQL RDBMS Micro-Batch Visualize Providing “Materialized Views” in RDBMS or NoSQL Datastores Stream Data Integration Streaming Data Sources Gateway • Bootstrap ”Materialized View” from event history Legacy Data Sources Monolithic System Machine IIoT Vehicle Environ mental Ware house E-Comm erce
- 24. Event Broker Stream Analytics Stream Data IntegrationCDC Stream Data Integration CDC Streaming Visualize Stream Data Integration 1st Micro service Stream Data Integration Stream Data Integration Stream Data Integration NoSQL RDBMS Micro-Batch Visualize Modern Event-Driven Apps (aka. Microservices) • Microservice participates as both a consumer and producer of events Stream Data Integration Streaming Data Sources Gateway Legacy Data Sources Monolithic System Machine IIoT Vehicle Environ mental Ware house E-Comm erce
- 25. Event Broker Stream Analytics Stream Data IntegrationCDC Stream Data Integration CDC Streaming Visualize Stream Data Integration 1st Micro service 2nd Micro service Stream Data Integration Stream Data Integration Stream Data Integration NoSQL RDBMS Micro-Batch Visualize Modern Event-Driven Apps (aka. Microservices) • 2nd microservice consumes events from 1st Bootstrap from event history [3] Stream Data Integration Streaming Data Sources Gateway Legacy Data Sources Monolithic System Machine IIoT Vehicle Environ mental Ware house E-Comm erce
- 26. Event Broker Stream Analytics Stream Data IntegrationCDC Stream Data Integration CDC Streaming Visualize Stream Data Integration 1st Micro service 2nd Micro service Stream Data Integration Stream Data Integration Stream Data Integration NoSQL RDBMS Micro-Batch Visualize Bi-Directional Legacy Systems Integration [4]AQ Stream Data Integration Streaming Data Sources Gateway Legacy Data Sources Monolithic System Machine IIoT Vehicle Environ mental Ware house E-Comm erce
- 27. Event Broker Stream Analytics Stream Data IntegrationCDC Stream Data Integration CDC Streaming Visualize Stream Data Integration 1st Micro service 2nd Micro service Stream Data Integration Stream Data Integration Stream Data Integration NoSQL RDBMS Micro-Batch Visualize Hybrid Cloud, Geo-Distributed or Disaster Recovery Scenario AQ Stream Data Integration Streaming Data Sources Event Broker Mirroring Event Broker Gateway Legacy Data Sources Monolithic System Machine IIoT Vehicle Environ mental Ware house E-Comm erce
- 28. Event Broker Stream Analytics Streaming Visualize Stream Data Integration Stream Data Integration Stream Data Integration Batch Analytics Event Broker as “Virtualized” Data Lake for Batch Analytics Stream Data Integration Streaming Data Sources 1st Micro service 2nd Micro service Stream Data Integration NoSQL RDBMS Micro-Batch Visualize Stream Data IntegrationCDC Stream Data Integration CDC AQ Gateway Legacy Data Sources Monolithic System Machine IIoT Vehicle Environ mental Ware house E-Comm erce
- 29. Kafka Storage Local Storage Tiered Storage (Confluent Enterprise) Broker 1 Broker 2 Broker 3 Broker 1 Broker 2 Broker 3 Object Storage hothot & cold cold Data Retention: • Never • Time (TTL) or Size-based • Log-Compacted based 1. topic semantics 2. queue semantics 3. horizontally scalable 4. auto-scaling 5. highly available 6. back-pressure 7. durable 8. schema-less/opaque 9. Stream and Batch Consumers 10. (Unlimited) Retention 11. Guaranteed ordering 12. re-consumption of events 13. Access Control 14. Interoperable
- 30. Event Broker Stream Analytics Stream Data IntegrationCDC Stream Data Integration CDC Streaming Visualize Stream Data Integration 1st Micro service 2nd Micro service Stream Data Integration Stream Data Integration Batch Data Integration Stream Data Integration NoSQL RDBMS Data Lake / DWH Batch Visualize Batch Analytics Micro-Batch Visualize “Materialized” Data Lake for Batch Analytics Stream Data Integration Streaming Data Sources Gateway Legacy Data Sources Monolithic System Machine IIoT Vehicle Environ mental Ware house E-Comm erce
- 31. Event Broker Stream Analytics Stream Data IntegrationCDC Stream Data Integration CDC Streaming Visualize Stream Data Integration 1st Micro service 2nd Micro service Serverless FaaS Stream Data Integration Stream Data Integration Gateway Batch Data Integration Stream Data Integration NoSQL RDBMS Data Lake / DWH Batch Visualize Batch Analytics Micro-Batch Visualize Serverless/Function as a Service (FaaS) Stream Data Integration Streaming Data Sources Legacy Data Sources Monolithic System Machine IIoT Vehicle Environ mental Ware house E-Comm erce
- 32. Event Broker Stream Analytics Stream Data IntegrationCDC Stream Data Integration CDC Streaming Visualize Stream Data Integration 1st Micro service 2nd Micro service Serverless FaaS Stream Data Integration Stream Data Integration Gateway Batch Data Integration Stream Data Integration NoSQL RDBMS Data Lake / DWH Batch Visualize Batch Analytics Micro-Batch Visualize Event Broker becomes the central nervous system for your information! Stream Data Integration Streaming Data Sources Legacy Data Sources Monolithic System Machine IIoT Vehicle Environ mental Ware house E-Comm erce
- 33. Event Broker Stream Analytics Stream Data IntegrationCDC Stream Data Integration CDC Streaming Visualize Stream Data Integration Micro service Micro service Serverless FaaS Stream Data Integration Stream Data Integration Gateway Batch Data Integration Stream Data Integration NoSQL RDBMS Data Lake / DWH Batch Visualize Batch Analytics Micro-Batch Visualize Event Broker becomes the central nervous system for your information! Stream Data Integration Streaming Data Sources Log as a first-class citizen! Turning the database Inside out! Legacy Data Sources Monolithic System Machine IIoT Vehicle Environ mental Ware house E-Comm erce
- 34. Reference 1. Stream Processing Concepts and Frameworks 2. Streaming Visualization 3. Building event-driven (Micro)Services with Apache Kafka 4. Solutions for bi-directional integration between Oracle RDBMS & Apache Kafka
- 35. You are welcome to join us at the Expo area. We're looking forward to meeting you. Link to the Expo area: https://www.vinivia-event- manager.io/e/DOAG/portal/expo/29731 My other talks at DOAG 2020: 18.11 – 10:00 - Big Data, Data Lake, Datenserialisierungsformate 18.11 – 13:00 – Rolle des Event Hubs in einer modernen Daten Architektur 19.11 – 13:00 – Kafka Livedemo: Umsetzung einer Streaminglösung #slideless
- 36. 36