Intelligent, Automatic Restarts for Unhealthy Kafka Consumers on Kubernetes with Chris Shepherd
0 likes401 views
At Cloudflare we are big Kafka adopters and we run Kafka at a massive scale. We deploy our microservices leveraging Kafka on Kubernetes and we have have some interesting experience on how to keep the latter operational to avoid downtime. To do so, we implemented our own Intelligent Smart Health checks for microservices leveraging Kafka. This has allowed our services to be much more self-healing, meaning there is much less manual intervention required. Before we used to get paged when applications got stuck and this also led to different incidents that were also customer impacting. We've implemented this in go, using the Shopify/sarama package but the same concepts can be adopted in different programming languages.
![Our final approach
● Each replica only keeps track of it’s own
partitions offsets
● Sarama library has the functionality to
observe when a rebalancing happens
● We remove keys from the in-memory
offset map so it only includes the
relevant partition values
for {
select {
case message, ok := <-claim.Messages(): // <-- Message received
// Store latest received offset in-memory
offsetMap[message.Partition] = message.Offset
// Handle message
handleMessage(ctx, message)
// Commit message offset
session.MarkMessage(message, "")
case <-session.Context().Done(): // <-- Rebalance happened
// Remove rebalanced partition from in-memory map
delete(offsetMap, claim.Partition())
}
}](https://image.slidesharecdn.com/lt13-20230517-ksl23-chrisshepperd-230602142548-e0a63f05/85/Intelligent-Automatic-Restarts-for-Unhealthy-Kafka-Consumers-on-Kubernetes-with-Chris-Shepherd-16-320.jpg)
![[오픈소스컨설팅] 쿠버네티스와 쿠버네티스 on 오픈스택 비교 및 구축 방법](https://cdn.slidesharecdn.com/ss_thumbnails/osck8svsk8sonopenstackkhoj-210310051504-thumbnail.jpg?width=560&fit=bounds)
![[OpenStack] 공개 소프트웨어 오픈스택 입문 & 파헤치기](https://cdn.slidesharecdn.com/ss_thumbnails/opensource-skimmingopenstackwithkoreausergroup-180824144737-thumbnail.jpg?width=560&fit=bounds)
![[GitOps] Argo CD on GKE (v0.9.2).pdf](https://cdn.slidesharecdn.com/ss_thumbnails/gitopsargocdongkev0-230424043339-b312ca19-thumbnail.jpg?width=560&fit=bounds)
![[GuideDoc] Deploy EKS thru eksctl - v1.22_v0.105.0.pdf](https://cdn.slidesharecdn.com/ss_thumbnails/guidedocdeployeksthrueksctl-v1-220715080545-0fb340c1-thumbnail.jpg?width=560&fit=bounds)
Ad
More Related Content
What's hot (20)
Similar to Intelligent, Automatic Restarts for Unhealthy Kafka Consumers on Kubernetes with Chris Shepherd (20)
Ad
More from HostedbyConfluent (20)
Ad
Recently uploaded (20)
Intelligent, Automatic Restarts for Unhealthy Kafka Consumers on Kubernetes with Chris Shepherd
- 1. Intelligent, Automatic Restarts for Unhealthy Kafka Consumers on Kubernetes Chris Shepherd
- 2. ● Cloudflare has been using Kafka in production since 2014. ● 14 distinct Kafka clusters, across multiple data centers. ● Running roughly 330 nodes. ● Some consumer groups handling 50m messages per second. ● Over 1 trillion messages processed! Kafka at Cloudflare
- 3. Kafka applications on Kubernetes
- 5. ● Liveness probes: Ensures that an application within a container is live and operational. The kubelet uses liveness probes to know when to restart a container. ● Readiness probes: Used to determine when a container is ready to start accepting traffic. A Pod is considered ready when all of its containers are ready. ● Startup probes: Replace liveness checks on slow starting containers to avoid them being killed before they are up & running. Kubernetes health checks
- 7. ● Kafka applications don’t typically accept HTTP traffic, so readiness probes are redundant. For this reason we configure only liveness checks on them. ● These checks are usually configured to start after a brief period (e.g. 10s) and are executed at intervals to check if the application is healthy (e.g. 15s). ● We also usually tolerate a reasonable amount of failed health checks before restarting the pod (e.g. 3 times). Health checks for Kafka applications
- 8. A naive approach to Kafka health checks
- 10. Why it doesn’t always work
- 11. Noisy Oncall
- 12. A first iteration on smart health checks
- 13. ● This uses two values: ● Current (latest) offset - last message sent to the topic ● Committed offset - last message processed by consumer ● Check consumer is moving forwards by ensuring: ● Latest offset is changing - receiving new messages ● Offsets are committed - consumer processing new messages A new approach!
- 15. Why it didn’t work for us
- 16. Our final approach ● Each replica only keeps track of it’s own partitions offsets ● Sarama library has the functionality to observe when a rebalancing happens ● We remove keys from the in-memory offset map so it only includes the relevant partition values for { select { case message, ok := <-claim.Messages(): // <-- Message received // Store latest received offset in-memory offsetMap[message.Partition] = message.Offset // Handle message handleMessage(ctx, message) // Commit message offset session.MarkMessage(message, "") case <-session.Context().Done(): // <-- Rebalance happened // Remove rebalanced partition from in-memory map delete(offsetMap, claim.Partition()) } }
- 17. Another Gotcha!
- 18. ● Helpful debug logging: ● Partition being consumed ● Offsets being committed ● Duration of health checks ● Metrics for visibility on health check performance is vital Visibility is important!
- 19. ● Without proper thought, “dumb” health checks can lead to a false sense of security that a service is running as expected even when it’s not. ● Good health checks can often be the difference between engineers being called out to fix trivial issues and a service which is self-healing ● Besides initial teething problems, we have reduced the number of PagerDuty alerts in our team by 50% since using smart health checks ● It’s important to think about specific behavior of the service and decide what being unhealthy means in each instance, instead of just ensuring that dependent services are connected. Takeaways
- 20. Thank you! chris-shepherd1993 Chris Shepherd Senior Systems Engineer Check out the Cloudflare blog for more information on our smart health checks - Intelligent, automatic restarts for unhealthy Kafka consumers