Message Redelivery: An Unexpected Journey - Pulsar Summit SF 2022
0 likes458 views
This document summarizes the message redelivery process in Apache Pulsar. It discusses how messages are redelivered when producing or consuming messages. When producing, messages are redelivered if the broker does not acknowledge receipt in a timely manner. When consuming, messages are redelivered under three circumstances: if the acknowledgment times out, if messages are negatively acknowledged, or if delivery is delayed. The document provides details on the commands and objects involved in establishing connections, publishing, consuming, acknowledging, and redelivering messages between Pulsar clients and brokers.
Message Redelivery: An Unexpected Journey - Pulsar Summit SF 2022
- 1. Pulsar Summit San Francisco Hotel Nikko August 18 2022 Tech Deep Dive Message Redelivery: An Unexpected Journey David Kjerrumgaard Developer Advocate • StreamNative
- 2. • Apache Pulsar Committer & author of Pulsar In Action • Former Principal Software Engineer on Splunk’s Pulsar-as-a-Service platform • Over 15 years experience in Big Data/Streaming David Kjerrumgaard Developer Advocate StreamNative
- 3. Pulsar Client & Broker
- 4. streamnative.io 10,000 Foot Level ● Pulsar Clients and Brokers interact with one another by exchanging Command Messages back-and-forth. ● Brokers are continually listening for incoming commands on a known port. ● Clients create a connection to the Broker on this port and start issuing commands.
- 5. streamnative.io Command Categories ● These commands can be broken down into different categories based on how they are used.
- 6. streamnative.io Client Commands ● These commands are used by the Pulsar Client & Broker to establish communication, authenticate, and perform topic lookups. ● Commands are used to transition the client through various states, e.g., disconnected to connected.
- 7. Client Creation ● When the build() method is called, under the covers the CONNECT command is sent to the address specified by the serviceUrl property. ● If the client is authenticated, then the Pulsar Broker will respond with a CONNECTED command.
- 8. streamnative.io Client Creation Flow ● The builder dispatches a CONNECT command (including credentials) to establish a connection. ● The Broker authenticates the client and sends a CONNECTED command to indicate success.
- 10. streamnative.io Producer Commands ● These commands are used to move messages from the Producer to the Broker. ● Commands are used to perform message handshaking between the Producer and the Broker, e.g., SEND/SEND_RECEIPT
- 11. Producer Creation ● When the create() method is called, under the covers a PRODUCER command is sent to the Broker. ● If the producer is authorized and has a compatible schema, then the Pulsar Broker will respond with a PRODUCER_SUCCESS command.
- 12. Producer Creation – Configuration ProducerBuilder Method Description enableBatching (boolean enable) Control whether automatic batching of messages is enabled for the producer. batchingMaxBytes (int max) Set the maximum number of bytes permitted in a batch batchingMaxMessages (int max) Set the maximum number of messages permitted in a batch. batchingMaxPublishDelay (long delay, TimeUnit timeUnit) Set the time period within which the messages sent will be batched maxPendingMessages (int max) Set the max size of the queue holding the messages pending to receive an acknowledgment from the broker. sendTimeout(int timeout, TimeUnit unit) If a message is not acknowledged by the server before the sendTimeout expires, an error will be reported.
- 13. streamnative.io Producer Creation Flow ● The PRODUCER command includes additional info such as the Schema, etc. ● The Broker creates a Producer object on the Broker side and sends a PRODUCER_SUCCESS command if successful.
- 14. Broker-Side Producer Object ● An org.apache.pulsar.broker.service.Producer object is created on the broker-side that retains a reference to the corresponding Topic object. ● The Topic object is responsible for writing all the received messages to the Managed Ledger.
- 15. Client-Side Producer Object ● A Producer object is created on the client-side that has a BatchMessageContainer object that is used to retain messages until the batch is considered full. ● Messages published via the send()/sendAsync() methods are stored in the BatchMessageContainer. ● When the batch is full, messages are removed from the BatchMessageContainer and moved to a pendingMessages queue to await acknowledgement.
- 17. Message Publication ● The producing application can call the send() method multiple times. ● Each of these messages are added to the BatchMessageContainer until a batch send is triggered due to size or time.
- 18. Message Publication – cont. ● Once the batch is considered “full”, the raw messages are prepared for transmission to the Broker. ● Each message is annotated with metadata such as the producer's name, sequenceID, and creation time.
- 19. Message Publication – cont. ● The wrapped messages are added to the pendingMessage queue and sent to the Broker via a SEND command.
- 20. Message Publication – Success. ● Once the message is persisted to BookKeeper the Broker notifies the Producer via a SEND_RECEIPT command and the message is removed from the pendingMessage queue.
- 21. Publication Redelivery Scenario #1 ● If the message cannot be persisted, the Broker notifies the Producer via a SEND_ERROR command and the message will be resent from the pendingMessage queue.
- 22. Publication Redelivery Scenario #1 – cont ● If the PRODUCER receives a SEND_ERROR message, then it will resend all the messages in the pendingMessages queue.
- 23. Publication Redelivery Scenario #2 ● If there is an issue during the transmission of the message, then it remains in the pendingMessages queue and will be resent will the Producer reconnects.
- 24. Publication Redelivery Scenario #2 ● After the Producer reconnects, all the messages in the pendingMessages queue are re-sent automatically.
- 26. streamnative.io Consumer Commands ● These commands are used to move messages from the Broker to the Consumer. ● Commands are used to perform message handshaking between the Consumer and the Broker, e.g., MESSAGE/ACK
- 27. Consumer Creation ● When the subscribe() method is called, under the covers a SUBSCRIBE command is sent to the Broker. ● If the Consumer is authorized and has specified a compatible schema, then the Pulsar Broker will respond with a SUCCESS command.
- 28. Consumer Creation – Configuration ConsumerBuilder Method Description acknowledgmentGroupTime (long delay, TimeUnit unit) Group the consumer acknowledgments for the specified time. ackTimeout(long ackTimeout, TimeUnit timeUnit) Set the timeout for unacked messages ackTimeoutTickTime (long tickTime, TimeUnit timeUnit) Define the granularity of the ack-timeout redelivery. ackTimeoutRedeliveryBackoff (Redeli veryBackoff backoff) Allows you to specify a custom redelivery strategy for message that have exceeded the ack timeout negativeAckRedeliveryDelay (long delay, TimeUnit timeUnit) Set the delay to wait before re-delivering messages that have failed to be process. negativeAckRedeliveryBackoff (Redel iveryBackoff backoff) Allows you to specify a custom redelivery strategy for negatively acknowledged messages
- 29. streamnative.io Consumer Creation Flow ● The SUBSCRIBE command includes additional info such as the Schema, etc. ● The Broker creates a Consumer object on the Broker side and sends a SUCCESS command if successful.
- 33. Message Flow ● A pull-based model controlled by the FLOW command.
- 34. Message Flow - Backend ● The messages are chosen by the Subscription and read from BookKeeper.
- 35. Message Flow – cont. ● The messages are added to the pendingAcks HashMap before being sent back via individual MESSAGE commands.
- 36. Message Consumption ● When you consume messages using the receive() / receiveAsync() methods, the messages are taken from the incomingMessages queue and given to the application.
- 37. Automated Message Flow ● The client-side Consumer periodically requests new messages from the broker-side Consumer (pull-based) ● This process is triggered automatically when the client-side Consumer’s incomingMessages queue drops below 50% of its capacity. ● When this condition is detected a FLOW command is dispatched to the broker-side Consumer to request another batch of messages.
- 38. Post-Processing Options ● After the message is processed, it can either be acknowledged, negatively acknowledged, or sent to the retry letter topic to be re-delivered after a specified length of time.
- 40. Message Acknowledgement ● After a message has been successfully consumed, it must be acknowledged to prevent redelivery/reprocessing. ● The behavior of the Acknowledgment process can be controlled by the following settings in the ConsumerBuilder • acknowledgmentGroupTime(long delay, TimeUnit unit) • ackTimeout(long ackTimeout, TimeUnit timeUnit) • ackTimeoutRedeliveryBackoff(RedeliveryBackoff ackBackoff) • ackTimeoutTickTime(long tickTime, TimeUnit timeUnit)
- 41. Message Acknowledgement ● If the acknowledgmentGroupTime property is set, then the acks will be grouped together for the specified interval before being sent together, otherwise an ack is sent immediately.
- 42. Message Acknowledgement - Broker ● On the Broker side, the ack is removed from the pendingAcks list, and the subscription updates the cursor position. An ACK_RESPONSE command is sent back to the client to complete the handshake.
- 43. Redelivery of Messages Due to Ack Timeout
- 44. Waiting ● On the client-side, the UnackedMessageTracker tracks the receipt of the ACK commands that were sent to the Broker to ensure that the subscription cursor is positioned correctly. ● Any ACKs that do not receive a corresponding ACK_RESPONSE command must be resent to complete the handshake. Missing ACK_RESPONSEs indicate the possibility of a network outage, etc. ● How long the client decided to wait for these ACK_RESPONSEs is referred to as the “Ack Timeout”
- 45. The UnackedMessageTracker ● Messages are added to the UnackedMessageTracker as their corresponding ACKs are sent to the Broker.
- 46. The UnackedMessageTracker ● To ensure that we only redeliver messages that have not been ACKed by the Broker, we remove message as their corresponding ACK_RESPONSEs are received from the Broker.
- 47. The UnackedMessageTracker ● There are 3 variations of the UnackedMessageTracker, and the one that is used is determined by the way that the consumer was configured.
- 48. Ack Timeout Configuration ● If the ackTimeout property is set to zero, then ACK tracking is disabled, and we have AT_MOST_ONCE processing semantics. ● Otherwise, we use one of the two timer-based variations that waits a period for the ACK_RESPONSEs before resending the ACKs to the Broker. ● The only difference between the two is how the ack timeout interval is calculated.
- 49. Timer-Based UnackedMessgeTrackers ● Internally, both timer-based UnackedMessageTracker versions use a Timer task to trigger the redelivery of the ACKs. ● The base UnackedMessageTracker operates on a fixed interval equal to either the ackTimeoutTickTime property (if set) or else the ackTimeout property. ● The UnackedMessageRedeliveryTracker operates on a variable interval based upon the value of the ackTimeoutRedeliveryBackoff property.
- 50. Timer-Based UnackedMessgeTrackers ● Here is an example of an UnackedMessageRedeliveryTracker configured to use an Exponential backoff strategy.
- 51. Ack Timeout ● Once the ack timeout interval has elapsed, the consumer sends a REDELIVER_UNACKNOWLEDGED_MESSAGES command to the Broker.
- 52. Ack Timeout ● The REDELIVER_UNACKNOWLEDGED_MESSAGES triggers the Broker-side consumer to resend ALL the messages in PendingAcks.
- 54. The NegativeAcksTracker ● Messages are added to the NegativeAcksTracker when the application calls consumer.negativeAck(msg)
- 55. Negative Acks Configuration ● There is only one variation of the NegativeAcksTracker, and it uses a Timer task to trigger the redelivery of the negatively acked messages. ● The Timer operates on a fixed interval if the negativeAckRedeliveryDelay property is set. Otherwise, it uses the strategy specified by the negativeAckRedeliveryBackoff property.
- 56. Timeout ● Once the redelivery delay has elapsed, the consumer sends a REDELIVER_UNACKNOWLEDGED_MESSAGES command to the Broker.
- 58. Delayed Message Delivery ● If the enableRetry property is set to true, the consumer can request that a message be re-delivered after a specified delay, e.g., consumer.reconsumeLater(msg, 3, TimeUnit.SECONDS) ● Under the covers, this process utilizes the deliverAfter(delayTime, unit) method of the Producer class.
- 59. Delayed Message Delivery ● When you first call the reconsumeLater method on the consumer, a Producer object is initialized using the following code block.
- 60. Delayed Message Delivery ● Next, we check to see how many times we have already tried to redeliver this message by examining the SYSTEM_PROPERTY_RECONSUMETIMES property in the Message’s properties. ● If that value exceeds the maxRedeliverCount setting of the DLQ policy, then the message is routed to the DLQ. ● Otherwise, it is scheduled for delayed deliver via a call to producer.newMessage().deliverAfter(delayTime, unit);
- 61. streamnative.io Key Takeaways ➔ Message redelivery can occur when Producing or Consuming messages in Pulsar ➔ When producing, messages get re-delivered if the Broker doesn’t acknowledge receipt of the message in a timely manner. ➔ When consuming, messages get re-delivered under 3 different circumstances. Ack timeout, negative ack, and delay
- 62. David Kjerrumgaard Thank you! [email protected] @DavidKjerrumga1 Pulsar Summit San Francisco Hotel Nikko August 18 2022