Kafka streams - From pub/sub to a complete stream processing platform
- 1. Kafka Streams From pub/sub to a complete stream processing platform Kafka Meetup Utrecht Thursday, 8th June 2017 < paolo @ confluent.io >
- 2. https://www.confluent.io/blog/stream-data-platform-1/ Industry shift from Big Data to Fast Data and Stream Processing
- 3. $ cat < in.txt | grep “apache” | tr a-z A-Z > out.txt Apache Kafka APIs and UNIX analogy
- 4. $ cat < in.txt | grep “apache” | tr a-z A-Z > out.txt Connect APIs Apache Kafka APIs and UNIX analogy
- 5. $ cat < in.txt | grep “apache” | tr a-z A-Z > out.txt Producer/Consumer APIs Apache Kafka APIs and UNIX analogy
- 6. $ cat < in.txt | grep “apache” | tr a-z A-Z > out.txt Streams APIs Apache Kafka APIs and UNIX analogy
- 7. Streams APIs part of Apache Kafka http://kafka.apache.org/documentation/streams http://docs.confluent.io/current/streams
- 8. Build applications, not clusters <dependency> <groupId>org.apache.kafka</groupId> <artifactId>kafka-streams</artifactId> <version>0.10.2.1</version> </dependency>
- 10. How do I run in production?
- 11. How do I run in production? As any other Java applications...
- 12. How do I run in production? Uncool Cool
- 13. How do I run in production? http://docs.confluent.io/current/streams/introduction.html
- 17. Typical high level architecture
- 18. Typical high level architecture Real-time Data Ingestion
- 19. Typical high level architecture Stream Processing Storage Real-time Data Ingestion
- 20. Typical high level architecture Data Publishing / Visualization Stream Processing Storage Real-time Data Ingestion
- 21. How many clusters do you count? NoSQL (Cassandra, HBase, Couchbase, MongoDB, …) or Elasticsearch, Solr, … Storm, Flink, Spark Streaming, Ignite, Akka Streams, Apex, … HDFS, NFS, Ceph, GlusterFS, Lustre, ... Apache Kafka
- 22. Simplicity is the ultimate sophistication Apache Kafka and Kafka Streams APIs Stream Processing Platform Publish & Subscribe to streams of data like a messaging system Store streams of data safely in a distributed replicated cluster Process streams of data efficiently and in real-time Node.js
- 23. Duality of Streams and Tables http://docs.confluent.io/current/streams/concepts.html#duality-of-streams-and-tables
- 24. Duality of Streams and Tables http://docs.confluent.io/current/streams/concepts.html#duality-of-streams-and-tables
- 28. WorldCount (and Java 8) WordCountLambdaExample.java final Properties streamsConfiguration = new Properties(); streamsConfiguration.put(StreamsConfig.APPLICATION_ID_CONFIG, "wordcount-lambda-example"); streamsConfiguration.put(StreamsConfig.BOOTSTRAP_SERVERS_CONFIG, bootstrapServers); ... final Serde<String> stringSerde = Serdes.String(); final Serde<Long> longSerde = Serdes.Long(); final KStreamBuilder builder = new KStreamBuilder(); final KStream<String, String> textLines = builder.stream(stringSerde, stringSerde, "TextLinesTopic"); final Pattern pattern = Pattern.compile("W+", Pattern.UNICODE_CHARACTER_CLASS); final KTable<String, Long> wordCounts = textLines .flatMapValues(value -> Arrays.asList(pattern.split(value.toLowerCase()))) .groupBy((key, word) -> word) .count("Counts"); wordCounts.to(stringSerde, longSerde, "WordsWithCountsTopic"); final KafkaStreams streams = new KafkaStreams(builder, streamsConfiguration); streams.cleanUp(); streams.start(); Runtime.getRuntime().addShutdownHook(new Thread(streams::close));
- 29. Easy to Develop, Easy to Test WordCountLambdaIntegrationTest.java EmbeddedSingleNodeKafkaCluster CLUSTER = new EmbeddedSingleNodeKafkaCluster(); … CLUSTER.createTopic(inputTopic); … Properties producerConfig = new Properties(); producerConfig.put(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG, CLUSTER.bootstrapServers());
- 30. Apache Kafka and Streams APIs benefits • Build applications, not clusters • Native integration with Apacke Kafka • Elastic, fast, distributed, fault-tolerant, secure • Scalable: S, M, L, XL, XXL • Run everywhere: from containers to cloud • Streams (with KStream) and tables (with KTable) • Local state replicated to Kafka for fault-tolerance • Windowing and event time semantics out of the box • Supports late-arriving and out-of-order events
- 31. References • http://kafka.apache.org/ • http://kafka.apache.org/documentation/streams/ • http://docs.confluent.io/ • http://docs.confluent.io/current/streams/ • http://docs.confluent.io/current/streams/javadocs/ • http://blog.confluent.io/ • http://github.com/confluentinc/examples/ • http://github.com/apache/kafka/tree/trunk/streams/
- 32. References
- 33. The easiest way to get you started https://www.confluent.io/download/
- 34. SIMPLICITY WE
- 35. YOUR FEEDBACK!
- 36. Discount code: kafcom17 Use the Apache Kafka community discount code to get $50 off www.kafka-summit.org Kafka Summit San Francisco: August 28 Presented by