Apache Kafka
Download as PPTX, PDF2 likes2,513 views
Apache Kafka is a distributed streaming platform that allows building event-driven architectures. It provides high throughput and low latency for processing streaming data. Key features include event logging, publish-subscribe messaging, and stream processing capabilities. Some advantages are eventual consistency, scalability, fault tolerance and being more agile to maintain compared to traditional databases. It requires Zookeeper and the Java client API has undergone changes. Performance can be very high, with examples of LinkedIn processing 1.1 trillion messages per day and 2 million writes per second on modest hardware.
Apache Kafka
- 1. Apache Kafka Build a solid application architecture TEBOURBI Maher @mtebourbi
- 2. !
- 5. Web App Search Graph DB ETL …
- 6. Ok Set X = 1 Set X = 1 Set X = 3 Set X = 3 X = 3 X = 1 Ok Ok Ok
- 7. Other drawbacks • Database schema evolution. • Impact of some bugs on data. • System debugging.
- 8. ChangeChange
- 10. F(x) Raw Data Derived Data
- 11. Log to the rescue
- 12. DB 321 4 5 6 7 8 9 10 11 Command log Web App SearchCache
- 13. 321 4 5 6 7 8 9 10 11 Topic Producer Consumer Offset
- 14. 321 4 5 6 7 8 9 10 11 321 4 5 6 7 321 4 5 6 7 8 9 P1 P2 P3 Producer Topic Partitions :
- 16. Pull vs Push
- 17. 321 4 5 6 7 8 9 10 11 Partition Consumer Offset ZooKeeper Topic DB Vs Vs
- 18. 321 4 5 6 7 8 9 10 11 121 3 4 5 4 3 1 6 9 FBC X K D H nil Y W V Key Value Offset Key Value Offset 452 1 6 9 762 9 10 11 HDB Y W V Compaction
- 19. App time window S1 S2 B1 B2 Rt. Services Batch Services
- 20. Advantages • Eventual consistency. • More agile and maintainable. • Scalability. • Fault tolerance.
- 21. Other use cases • Messaging • Website Activity Tracking • Metring • Stream processing
- 22. Some annoyances • Require Zookeeper. • Official java client API changes. • Log compaction not possible with compressed topics. • Partition limited to single machine.
- 23. Performance • Use of OS page cache. • Linux: sendfile() syscall : pagecache -> socket. • 1.1 Trillion Messages Per Day at Linkedin • 2 Million Writes Per Second: https://goo.gl/acmi00 3 Servers : Intel Xeon 2.5 GHz processor with six cores Six 7200 RPM SATA drives 32GB of RAM 1Gb Ethernet Kafka version : 0.8.1