What is Apache Kafka®?
0 likes333 views
Viktor Gamov, Confluent, Developer Advocate Apache Kafka is an open source distributed streaming platform that allows you to build applications and process events as they occur. Viktor Gamov (developer Advocate at Confluent) walks through how it works and important underlying concepts. As a real-time, scalable, and durable system, Kafka can be used for fault-tolerant storage as well as for other use cases, such as stream processing, centralized data management, metrics, log aggregation, event sourcing, and more. This talk will explain what a streaming platform such as Apache Kafka is and some of the use cases and design patterns around its use—including several examples of where it is solving real business problems. https://www.meetup.com/Chennai-Kafka/events/269942117/
Ad
More Related Content
What's hot (20)
Similar to What is Apache Kafka®? (20)
Ad
More from confluent (20)
Ad
Recently uploaded (20)
What is Apache Kafka®?
- 1. 1 Your Projector is Not Broken
- 2. What is Apache Kafka Event-Streaming Platform?January, 2020 @gamussa | #confluentvug | @ConfluentINc
- 3. @gamussa | #confluentvug | @ConfluentINc 3
- 4. @gamussa | #confluentvug | @ConfluentINc 4 I build highly scalable Hello World apps
- 5. @gamussa | #confluentvug | @ConfluentINc 5 A company is build on DATA FLOWS but All we have is DATA STORES
- 6. @gamussa | #confluentvug | @ConfluentINc 6 Storage
- 7. @gamussa | #confluentvug | @ConfluentINc 7 ● DB - table ● Hadoop - file ● Kafka - ? Core Abstraction
- 9. @gamussa | #confluentvug | @ConfluentINc 9 LOG
- 10. @gamussa | #confluentvug | @ConfluentINc 10 The log is a simple idea Messages are added at the end of the log Old New
- 11. @gamussa | #confluentvug | @ConfluentINc 11 Messages are added at the end of the log Old New The log is a simple idea
- 12. @gamussa | #confluentvug | @ConfluentINc 12 Pub / Sub
- 13. @gamussa | #confluentvug | @ConfluentINc 13 Time
- 14. @gamussa | #confluentvug | @ConfluentINc 14 C2 C3C1 Time
- 15. @gamussa | #confluentvug | @ConfluentINc 15 Time A B C D hash(key) % numPartitions = N
- 16. @gamussa | #confluentvug | @ConfluentINc 16 Messages will be produced in a round robin fashion Time
- 17. @gamussa | #confluentvug | @ConfluentINc 17 Consumers have a position all of their own Old New Robin is here Scan Viktor is here Scan Ricardo is here Scan
- 18. @gamussa | #confluentvug | @ConfluentINc 18 Old New Robin is here Scan Viktor is here Scan Ricardo is here Scan Consumers have a position all of their own
- 19. @gamussa | #confluentvug | @ConfluentINc 19 Old New Robin is here Scan Viktor is here Scan Ricardo is here Scan Consumers have a position all of their own
- 20. @gamussa | #confluentvug | @ConfluentINc 20 Only Sequential Access Old New Read to offset & scan
- 22. @gamussa | #confluentvug | @ConfluentINc 22 C
- 23. @gamussa | #confluentvug | @ConfluentINc 23 CC C1 CC C2
- 24. @gamussa | #confluentvug | @ConfluentINc 24 C C C C
- 25. @gamussa | #confluentvug | @ConfluentINc 25 0 1 2 3
- 26. @gamussa | #confluentvug | @ConfluentINc 26 0 1 2 3
- 27. @gamussa | #confluentvug | @ConfluentINc 27 0, 3 1 2 3
- 28. @gamussa | #confluentvug | @ConfluentINc 28 Linearly Scalable Architecture Single topic: - Many producers machines - Many consumer machines - Many Broker machines No Bottleneck!! Producers Consumers
- 29. 29 From/to other systems: Kafka Connect and more Tip: Great option to gradually move workloads to Kafka while keeping production running!
- 30. 30 Kafka Connect ● Deployed standalone (development) or as a distributed cluster (production) ● Elastic service that works on bare-metal, VMs, containers, Kubernetes, ... ● The individual ‘Connector’ determines delivery guarantees, e.g., exactly-once VM VM
- 31. 31 Single Message Transforms for real-time ETL Ingress: modify an Event before storing ● Obfuscate sensitive information, e.g. PII ● Add origin of event for lineage tracking ● Remove unnecessary data fields ● … and more Egress: modify an Event on its way out ● Route high-priority events to faster stores ● Direct events to different Elasticsearch indexes ● Cast data types to match destination ● … and more { user: ab123, gender: female, ip: 1.2.3.95 } { user: ab123, ip: 1.2.3.XXX }
- 32. @gamussa | #confluentvug | @ConfluentINc 32 Replicate to get fault tolerance replicate msg msg leader Machine A Machine B
- 33. @gamussa | #confluentvug | @ConfluentINc 33 Partition Leadership and Replication Broker 1 Topic1 partition1 Broker 2 Broker 3 Broker 4 Topic1 partition1 Topic1 partition1 Leader Follower Topic1 partition2 Topic1 partition2 Topic1 partition2 Topic1 partition3 Topic1 partition4 Topic1 partition3 Topic1 partition3 Topic1 partition4 Topic1 partition4
- 34. @gamussa | #confluentvug | @ConfluentINc 34 Replication provides resiliency A replica takes over on machine failure
- 35. @gamussa | #confluentvug | @ConfluentINc 35 Partition Leadership and Replication - node failure Broker 1 Topic1 partition1 Broker 2 Broker 3 Broker 4 Topic1 partition1 Topic1 partition1 Leader Follower Topic1 partition2 Topic1 partition2 Topic1 partition2 Topic1 partition3 Topic1 partition4 Topic1 partition3 Topic1 partition3 Topic1 partition4 Topic1 partition4
- 36. 36 Similar to a traditional messaging system (ActiveMQ, Rabbit etc) but with: (a) Far better scalability (b) Built in fault tolerance / HA (c) Storage The log is a type of durable messaging system
- 37. @gamussa | #confluentvug | @ConfluentINc Stop! Demo time!
- 38. @gamussa | #confluentvug | @ConfluentINc 38 Processing
- 39. @gamussa | #confluentvug | @ConfluentINc 39 Streaming is the toolset for dealing with events as they move!
- 40. @gamussa | #confluentvug | @ConfluentINc 40 authorization_attempts possible_fraud What exactly is Stream Processing?
- 41. @gamussa | #confluentvug | @ConfluentINc 41 CREATE STREAM possible_fraud AS SELECT card_number, count(*) FROM authorization_attempts WINDOW TUMBLING (SIZE 5 MINUTE) GROUP BY card_number HAVING count(*) > 3; authorization_attempts possible_fraud What exactly is Stream Processing?
- 42. @gamussa | #confluentvug | @ConfluentINc 42 CREATE STREAM possible_fraud AS SELECT card_number, count(*) FROM authorization_attempts WINDOW TUMBLING (SIZE 5 MINUTE) GROUP BY card_number HAVING count(*) > 3; authorization_attempts possible_fraud What exactly is Stream Processing?
- 43. @gamussa | #confluentvug | @ConfluentINc 43 CREATE STREAM possible_fraud AS SELECT card_number, count(*) FROM authorization_attempts WINDOW TUMBLING (SIZE 5 MINUTE) GROUP BY card_number HAVING count(*) > 3; authorization_attempts possible_fraud What exactly is Stream Processing?
- 44. @gamussa | #confluentvug | @ConfluentINc 44 CREATE STREAM possible_fraud AS SELECT card_number, count(*) FROM authorization_attempts WINDOW TUMBLING (SIZE 5 MINUTE) GROUP BY card_number HAVING count(*) > 3; authorization_attempts possible_fraud What exactly is Stream Processing?
- 45. @gamussa | #confluentvug | @ConfluentINc 45 CREATE STREAM possible_fraud AS SELECT card_number, count(*) FROM authorization_attempts WINDOW TUMBLING (SIZE 5 MINUTE) GROUP BY card_number HAVING count(*) > 3; authorization_attempts possible_fraud What exactly is Stream Processing?
- 46. @gamussa | #confluentvug | @ConfluentINc 46 CREATE STREAM possible_fraud AS SELECT card_number, count(*) FROM authorization_attempts WINDOW TUMBLING (SIZE 5 MINUTE) GROUP BY card_number HAVING count(*) > 3; authorization_attempts possible_fraud What exactly is Stream Processing?
- 47. @gamussa | #confluentvug | @ConfluentINc 47 Lower the bar to enter the world of streaming User Population CodingSophistication Core developers who use Java/Scala Core developers who don’t use Java/Scala, e.g. .NET, Go Data engineers, architects, DevOps/SRE BI analysts streams
- 48. @gamussa | #confluentvug | @ConfluentINc 48 KSQL #FTW 4 Headless1 UI 2 CLI ksql> 3 REST POST /query
- 49. @gamussa | #confluentvug | @ConfluentINc 49 Interaction with Kafka Kafka (data) KSQL (processing) Application (processing) Java/KStreams, .NET Does not run on Kafka brokers Does not run on Kafka brokers
- 50. @gamussa | #confluentvug | @ConfluentINc 50 Find your local Meetup Group https://cnfl.io/kafka-meetups Join us in Slack http://cnfl.io/slack Grab Stream Processing books https://cnfl.io/book-bundle
- 51. @ @gamussa | #confluentvug | @ConfluentINc Thanks! @gamussa [email protected]