Exactly-once Semantics in Apache Kafka
18 likes7,258 views
Apache Kafka's rise in popularity as a streaming platform has demanded a revisit of its traditional at-least-once message delivery semantics. In this talk, we present the recent additions to Kafka to achieve exactly-once semantics (EoS) including support for idempotence and transactions in the Kafka clients. The main focus will be the specific semantics that Kafka distributed transactions enable and the underlying mechanics which allow them to scale efficiently.
Ad
More Related Content
What's hot (20)
Viewers also liked (14)
Ad
Similar to Exactly-once Semantics in Apache Kafka (20)
Ad
More from confluent (20)
Recently uploaded (20)
![Download Wondershare Filmora Crack [2025] With Latest](https://cdn.slidesharecdn.com/ss_thumbnails/neo4j-howkgsareshapingthefutureofgenerativeaiatawssummitlondonapril2024-240426125209-2d9db05d-250419-250428115407-a04afffa-thumbnail.jpg?width=560&fit=bounds)
Exactly-once Semantics in Apache Kafka
- 1. 1 Introducing Exactly Once Semantics in Apache Kafka™ Apurva Mehta, Software Engineer, Gehrig Kunz, Technical Product Marketing Manager
- 2. 2 Agenda • Why exactly-once? • An overview of messaging semantics • Why are duplicates introduced? • What is exactly-once semantics? • Exactly-once semantics in Kafka: Is it Practical? • Next Steps
- 3. 3 Exactly Once Semantics is a hard problem
- 4. 4 An overview of messaging semantics • At-most once • At-least once • Exactly-once
- 5. 5 Why exactly-once? • Stream processing is becoming the norm; it’s more natural. • Apache Kafka is the most popular streaming platform. • Mission critical applications require stronger guarantees.
- 6. 6 Why exactly-once? • Stream processing is becoming the norm; it’s more natural. • Apache Kafka is the most popular streaming platform. • Mission critical applications require stronger guarantees. In other words: make stream processing easy, simple, and reliable enough for everyone.
- 7. 7 Apache Kafka’s existing semantics At Least Once
- 14. 14 What do we do now??? Kafka’s Existing Semantics
- 15. 15 Kafka’s Existing Semantics: At Least Once
- 16. 16 Kafka’s Existing Semantics: At Least Once
- 17. 17 Kafka’s Existing Semantics: At Least Once
- 18. 18 Why are duplicates introduced? Various failures must be handled correctly: • Broker can fail • Producer-to-Broker RPC can fail • Producer or Consumer client can fail
- 19. 19 TL;DR – What we have today • At least once in order delivery per partition. • Producer retries can introduce duplicates and headaches.
- 20. 20 The age old engineering question Before we make this work, are we sure we should?
- 21. 21 KafkaCash: A Peer to Peer Lending App A peer-to-peer lending platform.
- 22. 22 Help Bob reach $1000, send him $10
- 23. 23 KafkaCash, powered by Kafka
- 26. 26 Lost money! Eek eek!
- 27. 27 How did Kafka add exactly once semantics?
- 28. 28 Exactly-once semantics in Kafka, explained Apache Kafka’s guarantees are stronger in 3 ways: • Idempotent producer: Exactly-once, in-order, delivery per partition. • Transactions: Atomic writes across partitions. • Exactly-once stream processing across read-process- write tasks.
- 29. 29 Part 1/3 : Idempotent Producer Exactly-once, in-order, delivery per partition
- 30. 30 Idempotent Producer Semantics A single --successful!-- producer.send will result in exactly one copy of the message in the log in all circumstances.
- 31. 31 Producer Configs • enable.idempotence = true • max.inflight.requests.per.connection=1 • acks = “all” • retries > 0 (preferably MAX_INT)
- 40. 40 TL;DR: idempotent producer • Works transparently -- only one config change. • Sequence numbers and producer ids are in the log. • Resilient to broker failures, producer retries, etc.
- 41. 41 Part 2/3 : Transactions Atomic writes across multiple partitions.
- 42. 42 Transactions semantics • Atomic writes across multiple partitions. • All messages in a transaction are made visible together, or none are. • Consumers must be configured to skip uncommitted messages.
- 43. 43 Producer config for transactions • transactional.id = ‘some string’ • Typically based on the partition identifier in a partitioned, stateful, app. • Enables transaction recovery across producer sessions.
- 44. 44 The transaction API producer.initTransactions(); try { producer.beginTransaction(); producer.send(record0); producer.send(record1); producer.commitTransaction(); } catch (KafkaException e) { producer.abortTransaction(); }
- 45. 45 Transactions
- 46. 46 1. Initialize the producer producer.initTransactions(); try { producer.beginTransaction(); producer.send(record0); producer.send(record1); producer.commitTransaction(); } catch (KafkaException e) { producer.abortTransaction(); }
- 48. 48 2. Begin transactions and send data producer.initTransactions(); try { producer.beginTransaction(); producer.send(record0); producer.send(record1); producer.commitTransaction(); } catch (KafkaException e) { producer.abortTransaction(); }
- 51. 51 3. Commit transaction producer.initTransactions(); try { producer.beginTransaction(); producer.send(record0); producer.send(record1); producer.commitTransaction(); } catch (KafkaException e) { producer.abortTransaction(); }
- 52. 52 Commit
- 53. 53 Commit
- 54. 54 Commit
- 55. 55 Success!
- 56. 56 Consumer configs • isolation.level: • “read_committed”, or • “read_uncommitted”
- 57. 57 What do you get with isolation levels? • read_committed: consumers read to the point where there are no open transactions. • read_uncommitted: will read everything. • Messages read in offset order.
- 58. 58 TL;DR: Transactions • Atomic, multi-partition, writes. • Use the new producer APIs for transactions. • Consumers can filter out uncommitted or aborted transactional messages.
- 59. 59 Part 3/3 : Stream Processing Stream Processing with Exactly Once Semantics
- 60. 60 Streams config • processing.mode = “exactly_once”
- 61. 61 End-to-end exactly-once semantics • The read-process-write operation is atomic. • Thus streams tasks produce valid answers even when failures happen.
- 63. 63 Exactly Once Semantics in Kafka Is it practical?
- 64. 64 Performance boost for Apache Kafka 0.11! • Up to +20% producer throughput • Up to +50% consumer throughput • Up to -20% disk utilization • Details: https://bit.ly/kafka-eos-perf
- 65. 65 Gains due to more efficient message format
- 66. 66 What about the idempotent producer and transactions? • Transactions: 3-5% overhead for 100ms transactions, 1KB messages. • Longer transactions and better batching result in better performance. • 20% overhead relative to at-most once delivery without ordering guarantees. • Idempotent producer alone has negligible overhead.
- 67. 67 Putting it together • We talked through an idempotent producer • How we added transactions with atomic writes • The impact it has on stream processing
- 68. 68 When is it available? Available to use in Kafka 0.11, June 2017.
- 69. 69 Where we’ve come 2007 High throughput messaging broker 2008 Highly available replicated log 2012 Top Level Apache Project 2016 Streams API Connect API 2017 Exactly Once Semantics
- 70. 70 San Francisco August 28, 2017 Organized by Confluent
- 71. 71 What’s next for you slackpass.io/ confluentcommunity v Try it v v Join the Community Let us know what you think @ConfluentDownload Confluent Open Source
- 72. 72 Thank You!