BDW Chicago 2016 - Jayesh Thakrar, Sr. Software Engineer, Conversant - Data Loss and Data Duplication in Kafka
0 likes39 views
We all know that Kafka is a fast, scalable, durable and distributed publish-subscribe messaging system. From large internet companies to startups are using it as part of their streaming data pipeline for its speed, durability and ease of use/administration. Kafka has a strong durability promise, yet, there are certain situations when there can be data loss or data duplication. While data duplication is easy to detect, data loss is not when you are dealing with hundreds of millions of messages. This session will look into those scenarios and possible ways to detect and minimize data loss and data duplication.
Ad
More Related Content
What's hot (20)
Viewers also liked (8)
Ad
Similar to BDW Chicago 2016 - Jayesh Thakrar, Sr. Software Engineer, Conversant - Data Loss and Data Duplication in Kafka (20)
Ad
More from Big Data Week (20)
Recently uploaded (20)
BDW Chicago 2016 - Jayesh Thakrar, Sr. Software Engineer, Conversant - Data Loss and Data Duplication in Kafka
- 1. © 2015, Conversant, Inc. All rights reserved. PRESENTED BY September 22, 2016 Kafka: data loss and duplication Jayesh Thakrar
- 2. © 2015, Conversant, Inc. All rights reserved.2 Topics Kafka Overview Data Loss Data Duplication Audit & Monitoring
- 3. © 2015, Conversant, Inc. All rights reserved.3 Kafka Overview
- 4. © 2015, Conversant, Inc. All rights reserved.4 Kafka As A Log Abstraction Client: Producer Client: Consumer BClient: Consumer A Kafka Server = Kafka Broker Topic: app_events Source: https://engineering.linkedin.com/distributed-systems/log-what-every-software-engineer-should-know-about-real-time-datas-unifying
- 5. © 2015, Conversant, Inc. All rights reserved.5 Topic Partitioning . . . Kafka Broker Client: Producer or Consumer • Log sharded into partitions • Messages ordered within each partition • Message offset = absolute position in partition • Partitions stored on filesystem as ordered sequence of log segments (files) • Producers and clients can explicitly control messages for/from a partition or let API handle it Source: https://engineering.linkedin.com/distributed-systems/log-what-every-software-engineer-should-know-about-real-time-datas-unifying Topic: app_events
- 6. © 2015, Conversant, Inc. All rights reserved.6 Topic Partitioning . . . Scalability (Horizontal) Client: Producer, Consumer Kafka Broker 2 Leader Replica Replica Leader Replica Replica Leader Replica Replica Kafka Broker 0 Kafka Broker 1
- 7. © 2015, Conversant, Inc. All rights reserved.7 Topic Partitioning . . . Scalability (Horizontal) Kafka Broker 2 Leader Replica Replica Leader Replica Replica Leader Replica Replica Kafka Broker 0 Kafka Broker 1 Pull-based inter-broker replication
- 8. © 2015, Conversant, Inc. All rights reserved.8 Other Key Concepts Cluster = collection of brokers (and loosely, zookeeper ensemble) Broker-id = a unique id (integer) assigned to each broker Controller = functionality within each broker responsible for leader assignment and management, with one being the active controller Replica = partition copy, represented by the broker-id Assigned replicas = set of all replicas (broker-ids) for a partition ISR = In-Sync Replicas = subset of assigned replicas (brokers) that are “in-sync/caught-up” with the leader*, including the leader
- 9. © 2015, Conversant, Inc. All rights reserved.9 Data Loss
- 10. © 2015, Conversant, Inc. All rights reserved.10 Data Loss : Inevitable Upto 0.01% data loss: For 700 billion messages per day, that's upto 7 million lost messages
- 11. © 2015, Conversant, Inc. All rights reserved.11 PRODUCER – INNER WORING & DATA LOSS Kafka Producer API API Call-tree kafkaProducer.send() …. accumulator.append() // buffer …. sender.send() // network I/O •Messages accumulate in buffer in batches •Batched by partition, retry at batch level •Expired batches dropped after retries •Error count and other metrics via JMX Data Loss at Producer •Failure to close / flush producer on termination •Dropped batches due to communication or other errors when acks = 0 or retry exhaustion •Data produced faster than delivery, causing BufferExhaustedException (deprecated in 0.10+)
- 12. © 2015, Conversant, Inc. All rights reserved.12 dATA LOSS BY CLUSTER / BROKERS Was it a leader? Detected by Controller via zookeeper Was it in ISR? Other replicas in ISR? Elect another leader Allow unclean election? ISR >= min.insync .replicas? Relax, everything will be fine Partition unavailable !! Other replicas available? Y Y N N Y Y Y Y N Potential data-loss depending upon acks setting at producer Broker Crashes N N N 1 2 4 5 6 3
- 13. © 2015, Conversant, Inc. All rights reserved.13 Config for Data Durability and Consistency Producer config - acks = -1 (or all) - max.block.ms (blocking on buffer full, default = 60000) and retries - request.timeout.ms (default = 30000) – it triggers retries Topic config - min.insync.replicas = 2 (or higher) Broker config - unclean.leader.election.enable = false - timeout.ms (default = 30000) – inter-broker timeout for acks
- 14. © 2015, Conversant, Inc. All rights reserved.14 Config for Availability and Throughput Producer config - acks = 0 (or 1) - buffer.memory, batch.size, linger.ms (default = 100) - request.timeout.ms, max.block.ms (default = 60,000), retries - max.in.flight.requests.per.connection Topic config - min.insync.replicas = 1 (default) Broker config - unclean.leader.election.enable = true
- 15. © 2015, Conversant, Inc. All rights reserved.15 Data Duplication
- 16. © 2015, Conversant, Inc. All rights reserved.16 Data Duplication: How it occurs Client: Producer Client: Consumer BClient: Consumer A Kafka Broker Topic: app_events Producer (API) retries = messages resent after timeout when retries > 1 Consumer consumes messages more than once after restart from unclean shutdown / crash
- 17. © 2015, Conversant, Inc. All rights reserved.17 Data Duplication: How to minimize Client: Producer Client: Consumer BClient: Consumer A Kafka Broker Topic: app_events Supplement Kafka API offset management with: - lookup last processed offset in destination at startup - off-process, low-latency, durable datastore to lookup processed messages by key or topic+partition+offset
- 18. © 2015, Conversant, Inc. All rights reserved.18 Audit & Monitoring
- 19. © 2015, Conversant, Inc. All rights reserved.19 Auditing: detect dropped and duplicates At producer, use call-back in Kafka producer API to: - persist mapping of message-key to Kafka key (Kafka key = topic + partition + offset mapping) - persist message count per time interval/window At consumer check for: - dropped/lost messages by verifying Kafka key to message key - message count per time interval for duplicate data
- 20. © 2015, Conversant, Inc. All rights reserved.20 Monitoring and Operations JMX metrics reporter in producer, consumer and broker https://cwiki.apache.org/confluence/display/KAFKA/Operations (outdated, but still relevant) SignalFX http://www.confluent.io/blog/how-we-monitor-and-run-kafka-at-scale-signalfx/ Netflix http://techblog.netflix.com/2016/04/kafka-inside-keystone-pipeline.html
- 21. © 2015, Conversant, Inc. All rights reserved.21 Questions?
- 22. © 2015, Conversant, Inc. All rights reserved.22 Jayesh Thakrar [email protected]
Editor's Notes
- #3: Kafka is a popular messaging system, being used by startups to some of the largest internet companies. Unlike some other messaging systems, Kafka can loose messages or introduce duplicates. This presentation will look into how it can happen, the tuning knobs to minimize thar and operational tips. The presentation will start with an overview that lays the foundation to understand the data loss and data duplication. Then it will cover the remainder of the key topics of data loss, data duplication and auditing and monitoring. Among other things, auditing and monitoring has pointers on how to collect metrics to quatify any data loss or duplication issues, if any.
- #5: Kafka started out as an log abstraction, in which message producers generate messages and append to an infinte log. The producer is agnostic of how or where the messages are stored or consumed. Messages in turn are retrieved by consumers, that are agnostic of message producers. Messages are stored in the log in the order they are presented to Kafka. A message can be stored as just a value or as a key-value pair. Consumers retrieve messages in the same order, however they can have filters to select message with specific keys or retrieve from a specific starting point in the log. The Kafka consumer API provides for a checkpoint mechanism to store the point of most recently read message.
- #6: If the log is implemented as a single, append-only log file than it becomes a scalability limiting factor. So Kafka has the notion of partitioning a log or topic. By that, the log is logically sharded into partitions. You can determine the sharding rule by giving your own partitioner or can use the built-in hash-based sharding which is applied to the key for sharding. Note that if you don't need or have a key, then use a null value for the key – not an empty string or a constant, otherwise sharding will not work correctly and there will be skew.
- #7: Partitioning allows data or partitions to be shared across multiple servers or Kafka brokers. Also, for durability reasons, partitions may be replicated across servers. All partitions are deemed equal, but one of the replicated partitions will be deemed as the primary or the leader and the others as follower replicas. The producers and consumers only work with the leader replicas i.e. they communicate with the brokers hosting the leader replica.
- #8: For each partition, the follower replica brokers fetch the data from the leader broker and this is all taken care of by the brokers using a pull mechanism. So note that a leader partition or broker serves producer clients, consumer clients and partition replicas.
- #11: Inspite of partition and replication, there is the potential for data loss. And who else then to get that admission of potential data loss than Netflix – who has one of the largest Kafka environment in terms of data flow.