Introduction to apache kafka
0 likes622 views
Apache Kafka is a distributed streaming platform and distributed publish-subscribe messaging system. It uses a log abstraction to order events and replicate data across clusters. Kafka allows developers to publish and subscribe to streams of records known as topics. Producers publish data to topics and consumers subscribe to topics to process streams of records. The Kafka ecosystem includes tools like KStreams for stream processing and KSQL for querying streams of data.
![KTables
Read-only & fault-tolerant state store
Can be in memory (on consumer) or global (persistent on the cluster)
RocksDB is a default implementation
Example: keep an up-to-date table of [user id, email]](https://image.slidesharecdn.com/introductiontoapachekafka-191109072412/85/Introduction-to-apache-kafka-28-320.jpg)
Ad
More Related Content
What's hot (20)
Similar to Introduction to apache kafka (20)
Ad
More from Dimitris Kontokostas (14)
Ad
Recently uploaded (20)
Introduction to apache kafka
- 1. Introduction to Apache Kafka Thessaloniki not-only-Java Meetup Dimitris Kontokostas @Diffbot
- 2. About me... Data geek, Software engineer Open source enthusiast, Remote Worker Currently working @ Diffbot
- 3. Summary ● The Log ● Kafka 101 ● KStream ● KSQL ● Connectors ● Schemas Let’s keep this interactive...
- 4. Kafka is A distributed streaming platform A distributed publish-subscribe messaging system/queue A distributed, immutable event logging data store (*)
- 5. The Log What every software engineer should know about real-time data's unifying abstraction A great post by Jay Kreps (creator of Kafka) Parts of the post are used in this presentation
- 6. What is a log - Records events - What happened & when - Append only, left-to-right - Timestamps
- 7. Logs in databases - As internal structures to facilitate ACID - Gradually getting more exposed - As a data subscription mechanism
- 8. Logs in distributed systems Ordering Replication Active-active vs Active-passive - E.g. “+5”, “-2”, “*4” - Vs “0”, “5”, “3”, “12”
- 9. Events & Tables Two sides of the same coin
- 10. events & tables 1. User_add {id:1, email: “[email protected]”} 2. User_change {id:1 email: “[email protected]”} 3. User_add {id:2, email: “[email protected]”} 4. User_change {id:2, email: “[email protected]”} 5. User_add {id:3, email: “[email protected]”} Table User id email 1 ? 2 ? 3 ?
- 11. events & tables 1. User_add {id:1, email: “[email protected]”} 2. User_change {id:1 email: “[email protected]”} 3. User_add {id:2, email: “[email protected]”} 4. User_change {id:2, email: “[email protected]”} 5. User_add {id:3, email: “[email protected]”} Table User id email 1 [email protected] 2 [email protected] 3 [email protected]
- 12. events & tables 1. User_add {id:1, email: “[email protected]”} 2. User_change {id:1 email: “[email protected]”} 3. User_add {id:2, email: “[email protected]”} 4. User_change {id:2, email: “[email protected]”} 5. User_add {id:3, email: “[email protected]”} 6. User_remove{id:1 } Table User id email 1 [email protected] 2 [email protected] 3 [email protected]
- 14. I have a cool idea!!! Report users with for more than 2 email edits per day as suspicious What if we also did X...
- 15. Decoupling from the table Tables are opinionated views of your data The later you can defer your opinion, the better...
- 18. Kafka Basics
- 19. Basic terminology Kafka maintains feeds of messages in categories called topics Kafka Producers are processes that publish messages on a kafka topic Kafka Consumers are processes that subscribe to topics and process the feed of published messages Messages can be in any form, text, json, binary etc. Kafka brokers are the servers that comprise the kafka cluster
- 20. Topics & Logs 1 Topic ≈ 1 Partitioned Log (with configurable retention period) What is the difference with “The Log”? Ordering...
- 21. Kafka Messages Anything can be a message => Strings, JSON, XML, binary formats Messages are preferably small ( < few Kb) => Big messages affect brokers & throughput => Reference big messages as external resources Message IDs define which partition the message is stored => Default behavior / can be manually overwritten Msg ID Body 1 2 3 4 5 ~~~
- 22. Producers & Consumers Message Delivery Semantics => At least once => At most once => Exactly once Multiple (independent) Consumers can read from the same topic => Consumers manage their own offset (stored on Kafka) => Messages remain on Kafka
- 23. Consumer Scaling Every Consumer has a Consumer ID => Kafka keeps track of the offets => rejoining will continue from latest offset Consumers with the same ID belong to the same Consumer Group Consumers from the same group read messages from different partitions => Topic partition size is the hard limit
- 24. KStreams/KSQL
- 25. public class KafkaConsumerExample { ... static void runConsumer() throws InterruptedException { final Consumer<Long, String> consumer = createConsumer(); while (true) { final ConsumerRecords< Long, String> consumerRecords = consumer.poll(); consumerRecords.forEach(record -> { System.out.printf( "Consumer Record:(%d, %s, %d, %d)n" , record.key(), record.value(),record.partition(), record.offset()); }); consumer.commitAsync(); } consumer.close(); }} https://dzone.com/articles/writing-a-kafka-consumer-in-java Print topic messages (Consumer API)
- 26. static void createWordCountStream(final StreamsBuilder builder) { final KStream<String, String> textLines = builder.stream(inputTopic); textLines .forEach(value -> System.out.printLn(value)); } Print topic messages (KStream API)
- 27. KStream API Abstraction on top of classic Kafka API Provide a Java Stream-like API on Kafka => has a source, sink & stream processors => aggregations, windowing, KTables / state Source processor => consumes on or more topics, forwards downstream Stream processors => apply transformations, aggregations, etc Sink Processors => final processors, stores results on a topic or KTable
- 28. KTables Read-only & fault-tolerant state store Can be in memory (on consumer) or global (persistent on the cluster) RocksDB is a default implementation Example: keep an up-to-date table of [user id, email]
- 29. Aggregations & Windowing Example: Users with more than 2 email changes in 24 hours final KTable<Windowed<String>, Long> anomalousUsers = views .groupByKey() .windowedBy(TimeWindows.of(Duration.ofDays(1))) .count() .filter((windowedUserId, count) -> count >= 2); final KStream<String, Long> anomalousUsersForConsole = anomalousUsers .toStream() .filter((windowedUserId, count) -> count != null) .map((windowedUserId, count) -> new KeyValue<>(windowedUserId.toString(), count)); anomalousUsersForConsole.to("AnomalousUsers", Produced.with(stringSerde, longSerde));
- 30. KSQL CREATE TABLE users_view AS SELECT * FROM USERS; CREATE STREAM suspicious_users AS SELECT id, count(*) FROM USERS WINDOW SESSION (1 DAY) GROUP BY id; HAVING count(*) >= 2; Disclaimer: most probably has typos !!!
- 31. Connectors & Schema Registries Big library of source & sink Kafka connectors => S3, most DBs, ES, HDFS, other streaming frameworks => e.g. write this topic to MySQL table X => Kafka Connect API for writing a custom one Schemas always change, not only in RDBMS :) => Message body freedom has a cost => Schema registries come to the… hmm... help… => data validation, de/serialization, backwards compatibility
- 32. Summary Logs are everywhere Logs help us defer data schema decision Kafka is an event logging data store Has a great ecosystem of libraries & integrations
- 33. Thank you for your attention Questions?