Apache Kafka’s Transactions in the Wild! Developing an exactly-once KafkaSink in Apache Flink with Fabian Paul | Kafka Summit London 2022
0 likes892 views
The document discusses the development of an exactly-once KafkaSink in Apache Flink and the significance of Apache Kafka in stream processing with delivery guarantees. It highlights Apache Flink's stateful functions and fault tolerance mechanisms, alongside the unified sink framework for managing Kafka transactions. The conclusion emphasizes Kafka's ability to facilitate high-throughput, low-latency applications while addressing caveats specific to its transaction API.
![© 2022 Ververica
Caveats with Apache Kafka’s Transaction API
● Transactional.id is bound to the lifetime of the KafkaProducer
○ Recycling of KafkaProducer is not possible because every
checkpoint changes the transactional.id
○ Aborting transactions initially can be very slow because we
need to create a new KafkaProducer every time
We added a new method setTransactionalId using reflection to reuse
the existing KafkaProducer with a different transactional.id. [1]
[1]
https://github.com/apache/flink/blob/1d347e66eb799646b28100430b0afa65a56d844b/flink-connectors/flink-connector-kafka/src/main/java/org/apache/flink/connector/kafka/sink/FlinkKafkaInternalProduc
er.java#L153](https://image.slidesharecdn.com/bs29fabianpaulververica-220503180459/85/Apache-Kafka-s-Transactions-in-the-Wild-Developing-an-exactly-once-KafkaSink-in-Apache-Flink-with-Fabian-Paul-Kafka-Summit-London-2022-43-320.jpg)
Apache Kafka’s Transactions in the Wild! Developing an exactly-once KafkaSink in Apache Flink with Fabian Paul | Kafka Summit London 2022
- 1. © 2022 Ververica Apache Kafka’s Transactions in the Wild! Developing an exactly-once KafkaSink in Apache Flink Fabian Paul, Ververica - Kafka Summit London 2022
- 2. © 2022 Ververica About Ververica Original creators of Apache Flink® Complete Stream Processing Infrastructure
- 3. © 2022 Ververica ● Apache Kafka is one of the the most widely used tools to support stream processing use cases with Apache Flink ● Different delivery guarantees in processing framework ○ At most once ○ At least once ○ Exactly once Motivation
- 4. © 2022 Ververica ● Apache Kafka is one of the the most widely used tools to support stream processing use cases with Apache Flink ● Different delivery guarantees in processing framework ○ At most once ○ At least once ○ Exactly once ● Demand for streaming applications with stronger guarantees increases constantly i.e. financial data processing Motivation
- 5. © 2022 Ververica Recap Apache Flink
- 6. © 2022 Ververica Apache Flink Apache Flink, Flink®, Apache®, the squirrel logo, and the Apache feather logo are either registered trademarks or trademarks of The Apache Software Foundation.
- 7. © 2022 Ververica Apache Flink Apache Flink, Flink®, Apache®, the squirrel logo, and the Apache feather logo are either registered trademarks or trademarks of The Apache Software Foundation.
- 8. © 2022 Ververica Apache Flink ● Many functions are stateful ○ Streaming data arrives over time ○ Functions need to remember records or temporary results ● Any variable that lives across function invocations is state ○ State must not be lost in case of a failure Fault Tolerance
- 9. © 2022 Ververica Apache Flink ● Many functions are stateful ○ Streaming data arrives over time ○ Functions need to remember records or temporary results ● Any variable that lives across function invocations is state ○ State must not be lost in case of a failure ● Periodically ingest checkpoint barriers into the data stream ● Save task state independently but at the same event time Fault Tolerance
- 10. © 2022 Ververica Apache Flink Fault Tolerance
- 11. © 2022 Ververica Apache Flink Fault Tolerance
- 12. © 2022 Ververica Apache Flink Fault Tolerance
- 13. © 2022 Ververica Apache Flink Fault Tolerance
- 14. © 2022 Ververica Apache Flink Fault Tolerance
- 15. © 2022 Ververica Apache Flink Fault Tolerance
- 16. © 2022 Ververica Apache Flink ● Supports writing data to external system for streaming and batch applications without dedicated implementations ● Offers different mixin interfaces for stateless and stateful sinks ● Utilizes a two phase commit protocol between a Writer and Committer Operator to ensure exactly once guarantees Unified Sink Framework
- 17. © 2022 Ververica Apache Flink Unified Sink Framework void write(InputT element, Context context) throws IOException, InterruptedException; void flush(boolean endOfInput) throws IOException, InterruptedException; Collection<CommT> prepareCommit() throws IOException, InterruptedException; List<WriterStateT> snapshotState(long checkpointId) throws IOException;
- 18. © 2022 Ververica Apache Flink Unified Sink Framework void write(InputT element, Context context) throws IOException, InterruptedException; void flush(boolean endOfInput) throws IOException, InterruptedException; Collection<CommT> prepareCommit() throws IOException, InterruptedException; List<WriterStateT> snapshotState(long checkpointId) throws IOException; Triggered for every incoming element and supposed to write to the external system
- 19. © 2022 Ververica Apache Flink Unified Sink Framework void write(InputT element, Context context) throws IOException, InterruptedException; void flush(boolean endOfInput) throws IOException, InterruptedException; Collection<CommT> prepareCommit() throws IOException, InterruptedException; List<WriterStateT> snapshotState(long checkpointId) throws IOException; Triggered before receiving the checkpoint barrier. After the method completes all records need to be persisted in the external system.
- 20. © 2022 Ververica Apache Flink Unified Sink Framework void write(InputT element, Context context) throws IOException, InterruptedException; void flush(boolean endOfInput) throws IOException, InterruptedException; Collection<CommT> prepareCommit() throws IOException, InterruptedException; List<WriterStateT> snapshotState(long checkpointId) throws IOException; Triggered before the checkpoint barrier. The returned collection is forwarded to the committer operator.
- 21. © 2022 Ververica Apache Flink Unified Sink Framework void write(InputT element, Context context) throws IOException, InterruptedException; void flush(boolean endOfInput) throws IOException, InterruptedException; Collection<CommT> prepareCommit() throws IOException, InterruptedException; List<WriterStateT> snapshotState(long checkpointId) throws IOException; Triggered before the checkpoint barrier. The returned collection (i.e. open transactions) is forwarded to the committer operator.
- 22. © 2022 Ververica Apache Flink Unified Sink Framework void write(InputT element, Context context) throws IOException, InterruptedException; void flush(boolean endOfInput) throws IOException, InterruptedException; Collection<CommT> prepareCommit() throws IOException, InterruptedException; List<WriterStateT> snapshotState(long checkpointId) throws IOException; Triggered before the checkpoint barrier. The returned collection is persisted into the state.
- 23. © 2022 Ververica Apache Flink Unified Sink Framework void commit(Collection<CommitRequest<CommT>> committables) throws IOException, InterruptedException;
- 24. © 2022 Ververica Apache Flink Unified Sink Framework void commit(Collection<CommitRequest<CommT>> committables) throws IOException, InterruptedException; If all operators in the current job successfully checkpoint the commit is triggered. The implementation decides on potential retries.
- 25. © 2022 Ververica Apache Kafka Transactions Recap
- 26. © 2022 Ververica Transactions in Apache Kafka ● Atomic writes to multiple topics/partitions ○ All messages are part of a transaction and all or none are written ● Transactional consumers only read messages that are part of a committed transaction ● Users need to set a dedicated transaction descriptor: transactional.id
- 27. © 2022 Ververica Transactions in Apache Kafka Properties producerProps = new Properties(); producerProps.put("bootstrap.servers", "localhost:9092"); producerProps.put("transactional.id", "prod-1"); KafkaProducer<String, String> producer = new KafkaProducer<>(producerProps); producer.initTransactions(); producer.beginTransaction(); producer.send(new ProducerRecord<>("counts", "value")); producer.commitTransaction(); API
- 28. © 2022 Ververica Apache Flink’s KafkaSink
- 29. © 2022 Ververica Apache Flink’s KafkaSink Committer Writer Kafka - Records Opens a transaction during creation
- 30. © 2022 Ververica Apache Flink’s KafkaSink Committer Writer Kafka - Records Writes records continuously in the open transaction
- 31. © 2022 Ververica Apache Flink’s KafkaSink Committer Writer Kafka - Records - Committables Flush all outstanding messages Forward transaction handle (i.e. transactional.id)
- 32. © 2022 Ververica Apache Flink’s KafkaSink Committer Writer Kafka - Records - Committables Writes received committable into state
- 33. © 2022 Ververica Apache Flink’s KafkaSink Committer Writer Kafka - Records - Committables All tasks have successfully checkpointed Finish all transactions based on the received committables
- 34. © 2022 Ververica How to choose the correct transactional.id that parallel writers do not fence each other??? Apache Flink’s KafkaSink Writer Writer Writer Writer Subtask 0 Subtask 1 Subtask 2 Subtask 3
- 35. © 2022 Ververica How to choose the correct transactional.id that parallel writers do not fence each other??? Apache Flink’s KafkaSink Writer Writer Writer Writer Subtask 0 Subtask 1 Subtask 2 Subtask 3 {transactionalIdPrefix} - {subtaskId} - {checkpointId}
- 36. © 2022 Ververica Apache Flink’s KafkaSink 1. A opens transaction and writes records 2. B opens transaction and writes records 3. A fails 4. B commits the transaction.
- 37. © 2022 Ververica Prior opened transactions for a topic need to be finished before other committed transactions become visible. Apache Flink’s KafkaSink 1. A opens transaction and writes records 2. B opens transaction and writes records 3. A fails 4. B commits the transaction.
- 38. © 2022 Ververica Apache Flink’s KafkaSink How can we make the visibility of new transactions faster than the transaction timeout???
- 39. © 2022 Ververica Apache Flink’s KafkaSink How can we make the visibility of new transactions faster than the transaction timeout??? for (int i = currentSubtaskId; ; i += parallelism) { }
- 40. © 2022 Ververica Apache Flink’s KafkaSink How can we make the visibility of new transactions faster than the transaction timeout??? for (int i = currentSubtaskId; ; i += parallelism) { int abortedTransactions = 0; for (int j = currentCheckpointId; ; j++, abortedTransations++) { abortTransaction({transactionalIdPrefix}-{i}-{j}) } If (abortedTransactions == 0) { return; } }
- 42. © 2022 Ververica Caveats with Apache Kafka’s Transaction API ● Transactional.id is bound to the lifetime of the KafkaProducer ○ Recycling of KafkaProducer is not possible because every checkpoint changes the transactional.id ○ Aborting transactions initially can be very slow because we need to create a new KafkaProducer every time
- 43. © 2022 Ververica Caveats with Apache Kafka’s Transaction API ● Transactional.id is bound to the lifetime of the KafkaProducer ○ Recycling of KafkaProducer is not possible because every checkpoint changes the transactional.id ○ Aborting transactions initially can be very slow because we need to create a new KafkaProducer every time We added a new method setTransactionalId using reflection to reuse the existing KafkaProducer with a different transactional.id. [1] [1] https://github.com/apache/flink/blob/1d347e66eb799646b28100430b0afa65a56d844b/flink-connectors/flink-connector-kafka/src/main/java/org/apache/flink/connector/kafka/sink/FlinkKafkaInternalProduc er.java#L153
- 44. © 2022 Ververica Caveats with Apache Kafka’s Transaction API ● No official way to list all currently open transactions (i.e. internal transaction topic) ○ After a job recovery new records are only visible after the transaction timeout ○ KafkaProducer can overwrite transactions but cannot return whether a transaction already exists
- 45. © 2022 Ververica Caveats with Apache Kafka’s Transaction API ● No official way to list all currently open transactions (i.e. internal transaction topic) ○ After a job recovery new records are only visible after the transaction timeout ○ KafkaProducer can overwrite transactions but cannot return whether a transaction already exists We expose the KafkaProducer epoch to determine whether a transaction is currently open (epoch == 0, means no transaction open)
- 46. © 2022 Ververica Summary ● Apache Kafka offers a good way to build exactly once applications requiring high throughput and low latencies ● Transaction system is not always comparable to transactions known from traditional databases ● KafkaSink is released with Apache Flink 1.14 supporting no, atleast and exactly once guarantees
- 47. © 2022 Ververica Thank You