Kafka Connect and Streams (Concepts, Architecture, Features)
8 likes1,876 views
The document discusses the evolution and capabilities of Apache Kafka as a leading streaming platform, highlighting its features such as Kafka Connect for data integration and Kafka Streams for real-time data processing. It outlines the architecture of Kafka, including its components for modifying and processing events, as well as the role of KSQL as a streaming SQL engine. The content emphasizes Kafka's utility in supporting microservices through a unified data architecture.
![33
// Example: reading data from Kafka
KStream<byte[], String> textLines = builder.stream("textlines-topic", Consumed.with(
Serdes.ByteArray(), Serdes.String()));
// Example: transforming data
KStream<byte[], String> upperCasedLines= rawRatings.mapValues(String::toUpperCase));
KStream](https://image.slidesharecdn.com/kafkaconnectandstreamsintroduction-180917082834/85/Kafka-Connect-and-Streams-Concepts-Architecture-Features-33-320.jpg)
Kafka Connect and Streams (Concepts, Architecture, Features)
- 1. Kafka Connect and Kafka Streams The Rise of Apache Kafka as Streaming Platform Kai Waehner Technology Evangelist [email protected] LinkedIn @KaiWaehner www.confluent.io www.kai-waehner.de
- 2. 2https://conferences.oreilly.com/strata/strata-ca/public/schedule/detail/63921 https://qconlondon.com/london2018/presentation/cloud-native-and-scalable-kafka-architecture (2018) (2018) Apache Kafka A Distributed, Scalable Commit Log
- 3. 3 Apache Kafka A Distributed, Scalable Commit Log
- 4. 4 Apache Kafka A Distributed, Scalable Commit Log
- 5. 5 1.0 “Enterprise Ready” A Brief History of Apache Kafka and Confluent 0.11 Exactly-once semantics 0.10 Data processing (Streams API) 0.9 Data integration (Connect API) Intra-cluster replication 0.8 2012 2014 Cluster mirroring0.7 2015 2016 20172013 2018 CP 4.1 KSQL GA
- 6. 6 Apache Kafka The Rise of a Streaming Platform
- 7. 7 Orders Customers Payments Stock Apache Kafka Single Shared Source of Truth for (Micro)Services
- 8. 8 Independent Dev / Test / Prod
- 9. 9 No Matter Where it Runs
- 10. Kafka Connect Declarative Data integration for Apache Kafka
- 11. 11 Apache Kafka as Central Nervous System
- 12. 12 Kafka Connect
- 18. 18 Converters
- 21. 21 Single Message Transforms •Mask sensitive information •Add identifiers •Tag events •Lineage/provenance •Remove unnecessary columns •Route high priority events to faster data stores •Direct events to different Elasticsearch indexes •Cast data types to match destination •Remove unnecessary columns Modify events before storing in Kafka: Modify events going out of Kafka:
- 22. 22 • InsertField – Add a field using either static data or record metadata • ReplaceField – Filter or rename fields • MaskField – Replace field with valid null value for the type (0, empty string, etc) • ValueToKey – Set the key to one of the value’s fields • HoistField – Wrap the entire event as a single field inside a Struct or a Map • ExtractField – Extract a specific field from Struct and Map and include only this field in results • SetSchemaMetadata – modify the schema name or version • TimestampRouter – Modify the topic of a record based on original topic and timestamp. Useful when using a sink that needs to write to different tables or indexes based on timestamps • RegexpRouter – modify the topic of a record based on original topic, replacement string and a regular expression • „Build your own“ – A Transformation is just a Java Class Built-in Transformations
- 23. Kafka Streams Stream Processing natively on top of Apache Kafka without an additional big data cluster
- 24. 24 Kafka Streams - Part of Apache Kafka
- 25. 25 Stream Processing Data at Rest Data in Motion
- 26. 26 Key concepts
- 27. 27 Kafka Streams - Processor Topology 1) Read input from Kafka 2) Operator (directed acyclic graph): • Filter / map / aggregation / joins • Operators can be stateful 3) Write result back to Kafka
- 28. 28 Kafka Streams - Runtime
- 29. 29 Kafka Streams - Distributed State
- 30. 30 Kafka Streams - Scaling
- 31. 31 Kafka Streams - Streams and Tables
- 32. 32 Kafka Streams - Streams and Tables
- 33. 33 // Example: reading data from Kafka KStream<byte[], String> textLines = builder.stream("textlines-topic", Consumed.with( Serdes.ByteArray(), Serdes.String())); // Example: transforming data KStream<byte[], String> upperCasedLines= rawRatings.mapValues(String::toUpperCase)); KStream
- 34. 34 // Example: aggregating data KTable<String, Long> wordCounts = textLines .flatMapValues(textLine -> Arrays.asList(textLine.toLowerCase().split("W+"))) .groupBy((key, word) -> word) .count(); KTable
- 35. 35 Kafka Streams A complete streaming microservices, ready for production at large-scale App configuration Define processing (here: WordCount) Start processing
- 36. 36
- 37. 37 What if you are NOT a Java Coder? Population CodingSophistication Realm of Stream Processing New, Expanded Realm BI Analysts Core Developers Data Engineers Core Developers who don’t like Java Java KSQL
- 39. 39 KSQL – The Streaming SQL Engine for Apache Kafka