TL;DR Kafka Metrics | Kafka Summit London

Apr 2, 20240 likes367 views

"Monitoring is a fundamental operation when running Kafka and Kafka applications in production. There are numerous metrics available when using Kafka, however the sheer number is overwhelming, making it challenging to know where to start and how to properly utilise them. This session will introduce you to some of the key metrics that should be monitored and best practices in fine tuning your monitoring. We will delve into which metrics are the indicators for cluster’s availability and performance and are the most helpful when debugging client applications."

Kafka Cluster
3
Controller
Broke
r
Client application
Consumer
Producer
Kafka cluster
Overview

Broke
r
4
Broker metrics
Kafka Cluster
Client application
Consumer
Producer
Controller

5
Broker metrics
Alert metrics
UnderMinIsrPartitionCount
kafka.server:type=ReplicaManager,name=UnderMinIsrPartitionCount

6
Broker metrics
Alert metrics
UnderReplicatedPartitionCount
kafka.server:type=ReplicaManager,name=UnderReplicatedPartitionCount

7
BytesInPerSec | BytesOutPerSec
kafka.server:type=BrokerTopicMetrics,name={BytesInPerSec|BytesOutPerSec}
Cluster Performance
Metrics to monitor
ReplicationBytesInPerSec | ReplicationBytesOutPerSec
kafka.server:type=BrokerTopicMetrics,name={ReplicationBytesInPerSec|ReplicationBytesOutPerSec}

8
RequestHandlerAvgIdlePercent
kafka.server:type=KafkaRequestHandlersPool,name=RequestHandlerAvgIdlePercent
Cluster Performance
Metrics to monitor

9
Unbalanced cluster
Metrics to monitor
PartitionCount | LeaderCount
kafka.server:type=ReplicaManager,name=PartitionCount|LeaderCount

10
RequestsPerSec
kafka.network:type=RequestMetrics,name=RequestsPerSec,request={Produce|FetchConsumer|
FetchFollower}
Slow network
Metrics to monitor

11
NetworkProcessorAvgIdlePercent
kafka.network:type=SocketServer,name=NetworkProcessorAvgIdlePercent
Slow network
Metrics to monitor

12
RequestQueueTimeMs | RequestQueueSize
kafka.network:type=RequestChannel,name=RequestQueueTimeMs|RequestQueueSize
Slow network
Metrics to monitor

Kafka Cluster
13
Controller
Broke
r
Client application
Consumer
Producer
Controller Metrics

14
ActiveControllerCount
kafka.controller:type=KafkaController,name=ActiveControllerCount
Controller metrics
Alert metrics

15
OfflinePartitionCount
kafka.controller:type=KafkaController,name=OfflinePartitionCount
Controller metrics
Alert metrics

16
LeaderElectionRateAndTimeMs
kafka.controller:type=KafkaController,name=LeaderElectionRateAndTimeMs
Controller metrics
Alert metrics

Kafka Cluster
17
Controller
Broke
r
Client application
Consumer
Producer
Client Metrics
External monitoring for clusters

18
connection-count
kafka.[producer|consumer]:type=[producer|consumer]-metrics,client-id=([-.w]+)
Client metrics
Alert metrics

19
incoming|outgoing-byte-rate
kafka.[producer|consumer]:type=[producer|consumer]-metrics,client-id=([-.w]+)
Client metrics
Alert metrics

Client application
Kafka Cluster
Producer
JVM
JVM
20
Broker
Controller
Monitoring tools
How does it all fit together?
Prometheus
jmx_exporter
server
jmx_exporter
server
Alert Manager
PagerDuty
Grafana
Consumer

21
https://github.com/tinaselenge
https://www.linkedin.com/in/gselenge
https://developers.redhat.com/topics/kafka-kubernetes
https://kafka.apache.org/documentation/#monitoring
https://strimzi.io/docs/operators/0.36.1/full/overview#metrics-overview_str
https://cwiki.apache.org/confluence/display/KAFKA/
KIP-714%3A+Client+metrics+and+observability
Thank you

22
Tuesday
2:00 PM - 2:45 PM Breakout Room 4
Getting the Balance Right with Kafka Connect
Kate Stanley
5:30 PM - 6:15 PM Breakout Room 7
Safeguarding Your Kafka Data with Encryption-at-rest
Tom Bentley
Red Hat Sessions
Wednesday
2:00 PM - 2:45 PM Breakout Room 3
Meet the Apache Kafka Committers
Tom Bentley
2:15 PM - 3:00 PM Meetup Hub
Running Kafka on Kube the Native Way with Operators (plus Kafka Connect book signing)
Kate Stanley

You cannot operate what you cannot measure. In this talk, I am going to present the built-in metrics framework of Kafka Streams that supports monitoring Kafka Streams applications. You will learn how to setup monitoring of metrics for your Kafka Streams applications and you will hear about the following recent improvements to the metrics framework that aim to extend and simplify monitoring. KIP-444 aims to simplify and extend the built-in metrics framework. The RocksDB metrics introduced in KIP-471 and KIP-607 allow you to look directly into the built-in persistent state stores of your Kafka Streams applications. Finally, KIP-613 specifies metrics that measure end-to-end latencies in your applications. This talk will help you collect intel about the behavior of your Kafka Streams applications, and will allow you to reason about the deployment. In the end, you will be able to better understand your applications and run them in a more robust manner.

Building Event Driven Services with Apache Kafka and Kafka Streams - Devoxx B...Ben Stopford

Event Driven Services come in many shapes and sizes from tiny functions that dip into an event stream, right through to heavy, stateful services. This talk makes the case for building such services, be they large or small, using a streaming platform. We will walk through a number of patterns for putting these together using a distributed log, the Kafka Streams API and it’s transactional guarantees.

A Practical Deep Dive into Observability of Streaming Applications with Kosta...HostedbyConfluent

This document provides an overview of observability of streaming applications using Kafka. It discusses the three pillars of observability - logging, metrics, and tracing. It describes how to expose Kafka client-side metrics using interceptors, metric reporters, and the Spring Boot framework. It demonstrates calculating consumer lag from broker and client-side metrics. It introduces OpenTelemetry for collecting telemetry data across applications and exporting to various backends. Finally, it wraps up with lessons on monitoring consumer lag trends and selecting the right metrics to ship.

Opa gatekeeperRita Zhang

This document discusses OPA Gatekeeper, which is an admission webhook that helps enforce policies and strengthen governance in Kubernetes clusters. It provides customizable admission controls via configuration instead of code. Gatekeeper uses the Open Policy Agent (OPA) to evaluate policies written in Rego against objects in the Kubernetes API. It started as kube-mgmt and has evolved through several versions. Gatekeeper allows defining policies as templates with parameters and matching rules, and instances of those policies that are enforced as custom resources. It provides capabilities like auditing, CI/CD integration, and replicating cluster state for offline policy checking. The document demonstrates example policies and invites the reader to get involved in the open source project.

Removing performance bottlenecks with Kafka Monitoring and topic configurationKnoldus Inc.

Apache Kafka is a distributed messaging system used to build real-time data pipelines & streaming applications. Since applications rely heavily on efficient data transfer, message passing platforms like Kafka cannot afford a breakdown or poor performance. But how do we ensure that Kafka is running well and successfully streaming messages at low latency? This is where Kafka monitoring steps in. Here’s the agenda of the webinar - > Why Kafka monitoring? > Top 10 Kafka metrics to focus on > How to change Kafka topic configuration at runtime?

Large scale, distributed access management deployment with aruba clear passAruba, a Hewlett Packard Enterprise company

Running or planning on deploying a large ClearPass cluster? See what others are doing in larger environments to improve their deployments This session is designed to help customers that run the largest and most demanding networks learn how to deal with multiple locations, 100k+ endpoints, and strict SLA’s. Come to this session to discuss architecture for distributed deployments and how to better design your install for high performance, high availability needs. This is the one session where we’ll include the most experienced ClearPass team members for what will be a highly interactive session.

Web Scale Reasoning and the LarKC ProjectSaltlux Inc.

The LarKC project aims to build an integrated pluggable platform for large-scale reasoning. It supports parallelization, distribution, and remote execution. The LarKC platform provides a lightweight core that gives standardized interfaces for combining plug-in components, while the real work is done in the plug-ins. There are three types of LarKC users: those building plug-ins, configuring workflows, and using workflows.

Highly Available Kafka Consumers and Kafka Streams on Kubernetes with Adrian ...HostedbyConfluent

"Getting started with a Kafka Consumer or Streams application is relatively straight forward, but having those clients be highly available and resilient in a real-world application where compute is becoming more ephemeral is another matter. Kubernetes is a popular place to deploy these workloads, making these challenges more accessible and prevalent than ever. In this talk I will introduce a simple consumer implementation with a default configuration and discuss the KIPs and features that have been introduced over time to limit how the hostile world of cloud computing can impact your real-time consuming applications. Once the Kafka Consumer configurations are under our belt, we can see how these same concepts are applied and augmented in Kafka Streams, and then cater for new concepts such as maintaining and restoring local state store data. Throughout the talk I will show out-of-the box Kubernetes features and deployment configurations that harmonise with the Kafka clients and their configurations to achieve a highly available consumer or streams deployment. If you have found your real-time streaming applications stopping the world through rebalancing, starting up slower than expected through a routine deployment, taking an age to restore state or over time found them to be less reliable as your platform engineers make your Kubernetes cluster more awesome, you will hopefully find something in this talk you could apply tomorrow."

Autoscaling in kubernetes v1JurajHantk

The document discusses autoscaling in Kubernetes. It describes three levels of scaling: Vertical Pod Autoscaler (VPA), Horizontal Pod Autoscaler (HPA), and Cluster Autoscaler (CA). The HPA scales deployments based on metrics like CPU and memory usage. The VPA can automatically adjust pod resource requests and limits. The CA automatically adjusts the Kubernetes cluster size across availability zones. An example is provided of using these tools to scale a game studio's Trainstation 2 workload based on queue size and database utilization metrics.

Challenges in a Microservices Age: Monitoring, Logging and Tracing on Red Hat...Martin Etmajer

Until recently, applications ran on a handful of carefully curated technologies for years to come, deployments were static and teams siloed. In the microservices age, teams are cross-functional, pick the technologies they consider to work best for them, and are held accountable for what they produce. Application deployments are fine-grained, frequent, scalable and fully automated. What a shift and what a challenge for monitoring such environments! In this presentation, you'll learn how to obtain insights from your OpenShift environment by exploring various open-source monitoring, logging, and tracing technologies, including Elasticsearch, Fluentd, Hawkular, Heapster, Jolokia, Kibana, OpenTracing, Prometheus and ZipKin.

Open Shift.Run2019 マイクロサービスの開発に疲れる前にdaprを使おうkei omizo

Dapr is an open source, portable runtime that makes it easy for developers to build distributed applications across cloud and edge. It provides common services for microservices applications such as service invocation, state management, publish/subscribe, secrets management and more. Dapr runs as a sidecar to manage application logic and provides a standard way for applications to integrate with infrastructure components like databases, queues and object storage.

Kubernetes: Beyond Baby StepsDigitalOcean

Watch this Tech Talk: https://do.co/video_sgupta Designed for developers who have an in-depth understanding of Kubernetes concepts, this talk covers scaling apps with persistent storage and advanced networking. What You’ll Learn - Recent Kubernetes trends - Kubernetes autoscaling - RBAC (Role Based Access control) - Kubernetes resource quotas - Kubernetes extensions - Kubernetes security best practices About the Presenter Saurabh Gupta is a tech enthusiast with more than a decade of experience in the software industry. Currently a Senior Developer Advocate at DigitalOcean, he focuses on open source, DevOps, cloud, containers, and Kubernetes. He is also part of the CNCF Speakers Bureau, and is often found speaking at community meetups and conferences. New to DigitalOcean? Get US $100 in credit when you sign up: https://do.co/deploytoday To learn more about DigitalOcean: https://www.digitalocean.com/ Follow us on Twitter: https://twitter.com/digitalocean Like us on Facebook: https://www.facebook.com/DigitalOcean Follow us on Instagram: https://www.instagram.com/thedigitalocean/ We're hiring: http://do.co/careers

Bringing Kafka Without Zookeeper Into Production with Colin McCabe | Kafka Su...HostedbyConfluent

The document discusses bringing Apache Kafka clusters into production without using ZooKeeper for coordination and metadata storage. It describes how Kafka uses ZooKeeper currently and the problems with this approach. It then introduces KRaft, which replaces ZooKeeper by using Raft consensus to replicate cluster metadata within Kafka. The key aspects of deploying, operating and troubleshooting KRaft-based Kafka clusters are covered, including formatting storage, controller setup, rolling upgrades, and examining the replicated metadata log.

Secure Kafka at scale in true multi-tenant environment ( Vishnu Balusu & Asho...confluent

Application teams in JPMC have started shifting towards building event driven architectures and real time steaming pipelines and Kafka has been at core in this journey. As application teams have started adopting Kafka rapidly, need for a centrally managed Kafka as a service has emerged. We have started delivering Kafka as a service in early 2018 and running in production for more than an year now operating 80+ clusters (and growing) in all environments together. One of the key requirements is to provide truly segregated, secured multi-tenant environment with RBAC model while satisfying financial regulations and controls at the same time. Operating clusters at large scale requires scalable self-service capabilities and cluster management orchestration. In this talk we will present - Our experiences in delivering and operating secured, multi-tenant and resilient Kafka clusters at scale. - Internals of our service framework/control plane which enables self-service capabilities for application teams, cluster build/patch orchestration and capacity management capabilities for TSE/admin teams. - Our approach in enabling automated Cross Datacenter failover for application teams using service framework and confluent replicator.

How to reduce expenses on monitoringRomanKhavronenko

Talk from https://stackconf.eu/talks/how-to-reduce-expenses-on-monitoring-with-victoriametrics/ Given recent economic changes, cost efficiency has become a top priority for many businesses. This is especially important for monitoring because the nature of telemetry data tends to exponential growth. Many monitoring solutions are now switching their focus to optimize costs. The talk will cover open-source instruments from VictoriaMetrics ecosystem for improving monitoring cost-efficiency. Compression optimization. While ZSTD compression and time series specific techniques like delta-encoding are great, there is still room for improvement. I’ll explain what else can be done to reduce disk footprint for long-term storage. Extra compression on data transferring between metrics collectors and TSDB. I’ll explain how VictoriaMetrics collector reduces the traffic volume by 4 times. Pre-computing for telemetry data on collectors, frequently referred to as edge computing. In VictoriaMetrics it is named as streaming aggregation and allows collectors to pre-compute data, reducing its resolution and cardinality before it is pushed to the database. This is especially important for Prometheus-like systems because streaming aggregation is compatible with Prometheus RemoteWrite protocol and can be used with any system which supports it. Cardinality explorer. Interface, which provides useful insights into data stored by the TSDB. It helps to identify the most expensive metrics or labels and see how they have changed in time. Query tracing. This feature provides details about all the stages of query execution, including time spent on index lookups, disk reads, data transfer, computation, and memory expenses. This is similar to SQL EXPLAIN feature, and helps to improve the performance of read queries. Compute efficiency. VictoriaMetrics components for collecting and storing telemetry data consume fewer resources compared to components from Prometheus ecosystem. It may sound like competition or bragging, but this is a real reason why people migrate from Prometheus to VictoriaMetrics – to cut their infrastructure costs by 2-3 times. All the features listed above are open-source and are available for everyone to use. The talk will be mostly concentrated on typical use cases in monitoring and elegant ways to make things more efficient.

Streaming ETL to Elastic with Apache Kafka and KSQLconfluent

Companies are recognizing the importance of a low-latency, scalable, fault-tolerant data backbone, in the form of the Apache Kafka streaming platform. With Kafka, developers can integrate multiple sources and systems, enableing low latency analytics, event-driven architectures and the population of multiple downstream systems. These data pipelines can be built using configuration alone. In this talk we’ll see how easy it is to stream data from sources such as databases and into Kafka using the Kafka Connect API. We’ll use KSQL to filter, aggregate and join it to other data, and then stream this enriched data from Kafka out into targets such as Elasticsearch. All of this can be accomplished without a single line of code!

Flink Forward San Francisco 2018: Andrew Torson - "Extending Flink metrics: R...Flink Forward

This document discusses extending metrics capabilities in Apache Flink for real-time business intelligence (BI) on streaming data pipelines at Walmart Labs. It proposes adding metric calculators and decorators to Flink operators to compute BI metrics as data flows through pipelines. Metrics can be handled in-process for small dimensions or via side outputs for large dimensions, requiring downstream aggregation. The right metrics database and aggregation techniques should be chosen based on dimensionality.

Kubernetes connectivity to Cloud Native Kafka | Evan Shortiss and Hugo Guerre...HostedbyConfluent

If you want to build an ecosystem of streaming data to your Kafka platform, you will need a much easier way for your developer to quickly move what’s on the source to your cluster. Better yet, making the connector serverless so it would NOT waste any resources for being idle, and having a trusted partner manage your Kafka infrastructure for you. In this session, we will show you how easy we have made streaming data with great user experience. Flexible resource management with our new secret weapon in the Apache Camel project -- Kamelet. We’ll also demonstrate how Red Hat OpenShift Streams for Apache Kafka simplifies the provisioning of Kafka deployments in a public cloud, managing the cluster,topics, and configuring secure access to the Kafka cluster for your developers.

AWS Summit Singapore 2019 | Autoscaling Your Kubernetes WorkloadsAWS Summits

This document discusses autoscaling Kubernetes workloads using the Horizontal Pod Autoscaler (HPA). It provides a history of the HPA's development, how to choose an autoscaling metric, and implement autoscaling using a custom metric from Datadog. The document demonstrates setting up a Datadog cluster agent, registering it as an external metrics provider, and creating an HPA that scales a deployment based on a Datadog metric.

stackconf 2023 | How to reduce expenses on monitoring with VictoriaMetrics by...NETWAYS

Given recent economic changes, cost efficiency has become a top priority for many businesses. This is especially important for monitoring because the nature of telemetry data tends to exponential growth. Many monitoring solutions are now switching their focus to optimize costs. The talk will cover open-source instruments from VictoriaMetrics ecosystem for improving monitoring cost-efficiency. Compression optimization. While ZSTD compression and time series specific techniques like delta-encoding are great, there is still room for improvement. I’ll explain what else can be done to reduce disk footprint for long-term storage. Extra compression on data transferring between metrics collectors and TSDB. I’ll explain how VictoriaMetrics collector reduces the traffic volume by 4 times. Pre-computing for telemetry data on collectors, frequently referred to as edge computing. In VictoriaMetrics it is named as streaming aggregation and allows collectors to pre-compute data, reducing its resolution and cardinality before it is pushed to the database. This is especially important for Prometheus-like systems because streaming aggregation is compatible with Prometheus RemoteWrite protocol and can be used with any system which supports it. Cardinality explorer. Interface, which provides useful insights into data stored by the TSDB. It helps to identify the most expensive metrics or labels and see how they have changed in time. Query tracing. This feature provides details about all the stages of query execution, including time spent on index lookups, disk reads, data transfer, computation, and memory expenses. This is similar to SQL EXPLAIN feature, and helps to improve the performance of read queries. Compute efficiency. VictoriaMetrics components for collecting and storing telemetry data consume fewer resources compared to components from Prometheus ecosystem. It may sound like competition or bragging, but this is a real reason why people migrate from Prometheus to VictoriaMetrics – to cut their infrastructure costs by 2-3 times. All the features listed above are open-source and are available for everyone to use. The talk will be mostly concentrated on typical use cases in monitoring and elegant ways to make things more efficient.

Tacker vancouver project updateRobin Gong

Tacker is an OpenStack-based NFV orchestration framework that deploys and operates virtual network functions (VNFs). The presentation summarized Tacker's contributions in the OpenStack Queens release, including multiple flow classifiers per VNF forwarding graph, symmetric VNFFG, VNFFG updating, and a Zabbix plugin for VNF monitoring. It also previewed upcoming features in OpenStack Rocky such as container-based VNFs using Kubernetes, and hybrid VNF deployment across Kubernetes and OpenStack.

Data Transformations on Ops Metrics using Kafka Streams (Srividhya Ramachandr...confluent

This document discusses using Kafka Streams to transform operational metrics data from Priceline applications before loading it into Splunk. It describes how the legacy monitoring system worked, and the motives for moving to Kafka and Kafka Streams. It then explains how data is collected from applications into Kafka topics, and how various transformations like formatting, key application, and aggregation are performed using Kafka Streams before loading to Splunk. It also discusses testing, monitoring, and debugging Kafka Streams applications.

Google Cloud Platform monitoring with ZabbixMax Kuzkin

NDC London 2017 - The Data Dichotomy- Rethinking Data and Services with StreamsBen Stopford

1) The document discusses using streaming platforms like Kafka Streams to build event-driven microservices architectures. It notes the tension between sharing data through databases versus loosely coupling services. 2) Services can share data by publishing domain events to a streaming platform instead of directly calling each other. The streaming platform acts as the system of record storing events indefinitely. 3) Data can be materialized from the event stream into views/tables inside services as needed through streaming queries. This allows querying the system of record without moving data.

Interactive Query in Kafka Streams: The Next Generation with Vasiliki Papavas...HostedbyConfluent

Kafka Streams offers Interactive Queries (IQ) that allow one to interact with the internal stream processing state from outside of the application. This functionality has proven invaluable to users over the years for everything from debugging to serving low latency queries straight from the Streams runtime. However, the actual interfaces for IQ were designed in the very early days of Kafka Streams and have proven cumbersome to use. Adding new custom queries such as reverse scan requires changes to more than 6000 loc spanning 108 files. There is no way currently to customize how a query executes - if, for instance, it should use the caching layer or not. Moreover, there is no way to extend the query result to include extra information such as which store layers or segments participated in the query, execution time, cache hit/miss, etc. Finally, IQ allows users to tradeoff consistency for availability by querying standby stores during rebalances. Eventual consistency, although useful as a concept, makes it difficult to provide applications with a good user experience. In this presentation, we unveil the next generation of Interactive Query (IQv2) that addresses all these shortcomings. We demonstrate the key benefits of the new query API: 1. Customizability: You can use IQv2 to easily plug in your own store implementations and queries. You can use it, for example, to implement complex queries to push down filters and indices into the storage layer itself. 2. Control: It gives you far more control over how each query is executed and exposes all the execution details you need to power high performance, production-grade queries. 3. Consistency: It provides a tunable consistency model, allowing you to choose your tradeoff between consistency, availability, and latency. Attend this talk if you want to learn about how to write high performance, semantically strong applications in a modern, data-in-motion environment.

Transforming Data Streams with Kafka Connect: An Introduction to Single Messa...HostedbyConfluent

"In this talk, attendees will be provided with an introduction to Kafka Connect and the basics of Single Message Transforms (SMTs) and how they can be used to transform data streams in a simple and efficient way. SMTs are a powerful feature of Kafka Connect that allow custom logic to be applied to individual messages as they pass through the data pipeline. The session will explain how SMTs work, the types of transformations they can be used for, and how they can be applied in a modular and composable way. Further, the session will discuss where SMTs fit in with Kafka Connect and when they should be used. Examples will be provided of how SMTs can be used to solve common data integration challenges, such as data enrichment, filtering, and restructuring. Attendees will also learn about the limitations of SMTs and when it might be more appropriate to use other tools or frameworks. Additionally, an overview of the alternatives to SMTs, such as Kafka Streams and KSQL, will be provided. This will help attendees make an informed decision about which approach is best for their specific use case. Whether attendees are developers, data engineers, or data scientists, this talk will provide valuable insights into how Kafka Connect and SMTs can help streamline data processing workflows. Attendees will come away with a better understanding of how these tools work and how they can be used to solve common data integration challenges."

Renaming a Kafka Topic | Kafka Summit LondonHostedbyConfluent

"While Apache Kafka lacks native support for topic renaming, there are scenarios where renaming topics becomes necessary. This presentation will delve into the utilization of MirrorMaker 2.0 as a solution for renaming Kafka topics. It will illustrate how MirrorMaker 2.0 can efficiently facilitate the migration of messages from the old topic to the new one and how Kafka Connect Metrics can be employed to monitor the mirroring progress. The discussion will encompass the complexity of renaming Kafka topics, addressing certain limitations, and exploring potential workarounds when using MirrorMaker 2.0 for this purpose. Despite not being originally designed for topic renaming, MirrorMaker 2.0 has a suitable solution for renaming Kafka topics. Blog Post : https://engineering.hellofresh.com/renaming-a-kafka-topic-d6ff3aaf3f03"

More Related Content

Similar to TL;DR Kafka Metrics | Kafka Summit London (20)

Large scale, distributed access management deployment with aruba clear passAruba, a Hewlett Packard Enterprise company

Web Scale Reasoning and the LarKC ProjectSaltlux Inc.

Highly Available Kafka Consumers and Kafka Streams on Kubernetes with Adrian ...HostedbyConfluent

Autoscaling in kubernetes v1JurajHantk

Challenges in a Microservices Age: Monitoring, Logging and Tracing on Red Hat...Martin Etmajer

Open Shift.Run2019 マイクロサービスの開発に疲れる前にdaprを使おうkei omizo

Kubernetes: Beyond Baby StepsDigitalOcean

Bringing Kafka Without Zookeeper Into Production with Colin McCabe | Kafka Su...HostedbyConfluent

Secure Kafka at scale in true multi-tenant environment ( Vishnu Balusu & Asho...confluent

How to reduce expenses on monitoringRomanKhavronenko

Streaming ETL to Elastic with Apache Kafka and KSQLconfluent

Flink Forward San Francisco 2018: Andrew Torson - "Extending Flink metrics: R...Flink Forward

Kubernetes connectivity to Cloud Native Kafka | Evan Shortiss and Hugo Guerre...HostedbyConfluent

AWS Summit Singapore 2019 | Autoscaling Your Kubernetes WorkloadsAWS Summits

stackconf 2023 | How to reduce expenses on monitoring with VictoriaMetrics by...NETWAYS

Tacker vancouver project updateRobin Gong

Data Transformations on Ops Metrics using Kafka Streams (Srividhya Ramachandr...confluent

Google Cloud Platform monitoring with ZabbixMax Kuzkin

NDC London 2017 - The Data Dichotomy- Rethinking Data and Services with StreamsBen Stopford

Interactive Query in Kafka Streams: The Next Generation with Vasiliki Papavas...HostedbyConfluent

Large scale, distributed access management deployment with aruba clear passAruba, a Hewlett Packard Enterprise company

Web Scale Reasoning and the LarKC ProjectSaltlux Inc.

Highly Available Kafka Consumers and Kafka Streams on Kubernetes with Adrian ...HostedbyConfluent

Autoscaling in kubernetes v1JurajHantk

Challenges in a Microservices Age: Monitoring, Logging and Tracing on Red Hat...Martin Etmajer

Open Shift.Run2019 マイクロサービスの開発に疲れる前にdaprを使おうkei omizo

Kubernetes: Beyond Baby StepsDigitalOcean

Bringing Kafka Without Zookeeper Into Production with Colin McCabe | Kafka Su...HostedbyConfluent

Secure Kafka at scale in true multi-tenant environment ( Vishnu Balusu & Asho...confluent

How to reduce expenses on monitoringRomanKhavronenko

Streaming ETL to Elastic with Apache Kafka and KSQLconfluent

Flink Forward San Francisco 2018: Andrew Torson - "Extending Flink metrics: R...Flink Forward

Kubernetes connectivity to Cloud Native Kafka | Evan Shortiss and Hugo Guerre...HostedbyConfluent

AWS Summit Singapore 2019 | Autoscaling Your Kubernetes WorkloadsAWS Summits

stackconf 2023 | How to reduce expenses on monitoring with VictoriaMetrics by...NETWAYS

Tacker vancouver project updateRobin Gong

Data Transformations on Ops Metrics using Kafka Streams (Srividhya Ramachandr...confluent

Google Cloud Platform monitoring with ZabbixMax Kuzkin

NDC London 2017 - The Data Dichotomy- Rethinking Data and Services with StreamsBen Stopford

Interactive Query in Kafka Streams: The Next Generation with Vasiliki Papavas...HostedbyConfluent

More from HostedbyConfluent (20)

Transforming Data Streams with Kafka Connect: An Introduction to Single Messa...HostedbyConfluent

Renaming a Kafka Topic | Kafka Summit LondonHostedbyConfluent

Evolution of NRT Data Ingestion Pipeline at TrendyolHostedbyConfluent

"Trendyol, Turkey's leading e-commerce company, is committed to positively impacting the lives of millions of customers. Our decision-making processes are entirely driven by data. As a data warehouse team, our primary goal is to provide accurate and up-to-date data, enabling the extraction of valuable business insights. We utilize the benefits provided by Kafka and Kafka Connect to facilitate the transfer of data from the source to our analytical environment. We recently transitioned our Kafka Connect clusters from on-premise VMs to Kubernetes. This shift was driven by our desire to effectively manage rapid growth(marked by a growing number of producers, consumers, and daily messages), ensuring proper monitoring and consistency. Consistency is crucial, especially in instances where we employ Single Message Transforms to manipulate records like filtering based on their keys or converting a JSON Object into a JSON string. Monitoring our cluster's health is key and we achieve this through Grafana dashboards and alerts generated through kube-state-metrics. Additionally, Kafka Connect's JMX metrics, coupled with NewRelic, are employed for comprehensive monitoring. The session will aim to explain our approach to NRT data ingestion, outlining the role of Kafka and Kafka Connect, our transition journey to K8s, and methods employed to monitor the health of our clusters."

Ensuring Kafka Service Resilience: A Dive into Health-Checking TechniquesHostedbyConfluent

"Join our lightning talk to delve into the strategies vital for maintaining a resilient Kafka service. While proactive monitoring is key for issue prevention, failures will still occur. Rapid detection tools will enable you to identify and resolve problems before they impact end-users. This session explores the techniques employed by Kafka cloud providers for this detection, many of which are also applicable if you are managing independent Kafka clusters or applications. The talk focuses on health-checking, a powerful tool that encompasses an application and its monitoring to validate Kafka environment availability. The session navigates through Kafka health-check methods, sharing best practices, identifying common pitfalls, and highlighting the monitoring of critical performance metrics like throughput and latency for early issue detection. Attendees will gain valuable insights into the art of health-checking their Kafka environment, equipping them with the tools to identify and address issues before they escalate into critical problems. We invite all Kafka enthusiasts to join us in this talk to foster a deeper understanding of Kafka health-checking and ensure the continued smooth operation of your Kafka environment."

Exactly-once Stream Processing with Arroyo and KafkaHostedbyConfluent

"Stream processing systems traditionally gave their users the choice between at least once processing and at most once processing: accepting duplicate data or missing data. But ideally we would provide exactly-once processing, where every event in the input data is represented exactly once in the output. Kafka provides a transaction API that enables exactly-once when using Kafka as your source and sink. But this API has turned out to not be well suited for use by high level streaming systems, requiring various work arounds to still provide transactional processing. In this talk, I’ll cover how the transaction API works, and how systems like Arroyo and Flink have used it to build exactly-once support, and how improvements to the transactional API will enable better end-to-end support for consistent stream processing."

Fish Plays Pokemon | Kafka Summit LondonHostedbyConfluent

"In this talk, we will explore the exciting world of IoT and computer vision by presenting a unique project: Fish Plays Pokemon. Using an ESP Eye camera connected to an ESP32 and other IoT devices, to monitor fish's movements in an aquarium. This project showcases the power of IoT and computer vision, demonstrating how even a fish can play a popular video game. We will discuss the challenges we faced during development, including real-time processing, IoT device integration, and Kafka message consumption. By the end of the talk, attendees will have a better understanding of how to combine IoT, computer vision, and the usage of a serverless cloud to create innovative projects. They will also learn how to integrate IoT devices with Kafka to simulate keyboard behavior, opening up endless possibilities for real-time interactions between the physical and digital worlds."

Tiered Storage 101 | Kafla Summit LondonHostedbyConfluent

Building a Self-Service Stream Processing Portal: How And WhyHostedbyConfluent

"Real-time 24/7 monitoring and verification of massive data is challenging – even more so for the world’s second largest manufacturer of memory chips and semiconductors. Tolerance levels are incredibly small, any small defect needs to be identified and dealt with immediately. The goal of semiconductor manufacturing is to improve yield and minimize unnecessary work. However, even with real-time data collection, the data was not easy to manipulate by users and it took many days to enable stream processing requests – limiting its usefulness and value to the business. You’ll hear why SK hynix switched to Confluent and how we developed a self-service stream process portal on top of it. Now users have an easy-to-use service to manipulate the data they want. Results have been impressive, stream processing requests are available the same day – previously taking 5 days! We were also able to drive down costs by 10% as stream processing requests no longer require additional hardware. What you’ll take away from our talk: - What were the pain points in the previous environment - How we transitioned to Confluent without service downtime - Creating a self-service stream processing portal built on top of Connect and ksqlDB - Use case of stream process portal"

From the Trenches: Improving Kafka Connect Source Connector Ingestion from 7 ...HostedbyConfluent

"Discover how default configurations might impact ingestion times, especially when dealing with large files. We'll explore a real-world scenario with a 20,000,000+ line file, assessing metrics and exploring the bottleneck in the default setup. Understand the intricacies of batch size calculations and how to optimize them based on your unique data characteristics. Walk away with actionable insights as we showcase a practical example, turning a 7-hour ingestion process into a mere 30 minutes for over 30,000,000 records in a Kafka topic. Uncover metrics, configurations, and best practices to elevate the performance of your Kafka Connect CSV source connectors. Don't miss this opportunity to optimize your data pipeline and ensure smooth, efficient data flow."

Future with Zero Down-Time: End-to-end Resiliency with Chaos Engineering and ...HostedbyConfluent

"In order to meet the current and ever-increasing demand for near-zero RPO/RTO systems, a focus on resiliency is critical. While Kafka offers built-in resiliency features, a perfect blend of client and cluster resiliency is necessary in order to achieve a highly resilient Kafka client application. At Fidelity Investments, Kafka is used for a variety of event streaming needs such as core brokerage trading platforms, log aggregation, communication platforms, and data migrations. In this lightening talk, we will discuss the governance framework that has enabled producers and consumers to achieve their SLAs during unprecedented failure scenarios. We will highlight how we automated resiliency tests through chaos engineering and tightly integrated observability dashboards for Kafka clients to analyze and optimize client configurations. And finally, we will summarize the chaos test suite and the ""test, test and test"" mantra that are helping Fidelity Investments reach its goal of a future with zero down-time."

Navigating Private Network Connectivity Options for Kafka ClustersHostedbyConfluent

"There are various strategies for securely connecting to Kafka clusters between different networks or over the public internet. Many cloud providers even offer endpoints that privately route traffic between networks and are not exposed to the internet. But, depending on your network setup and how you are running Kafka, these options ... might not be an option! In this session, we’ll discuss how you can use SSH bastions or a self managed PrivateLink endpoint to establish connectivity to your Kafka clusters without exposing brokers directly to the internet. We explain the required network configuration, and show how we at Materialize have contributed to librdkafka to simplify these scenarios and avoid fragile workarounds."

Apache Flink: Building a Company-wide Self-service Streaming Data PlatformHostedbyConfluent

"In my talk, we will examine all the stages of building our self-service Streaming Data Platform based on Apache Flink and Kafka Connect, from the selection of a solution for stateful streaming data processing, right up to the successful design of a robust self-service platform, covering the challenges that we’ve met. I will share our experience in providing non-Java developers with a company-wide self-service solution, which allows them to quickly and easily develop their streaming data pipelines. Additionally, I will highlight specific business use cases that would not have been implemented without our platform.0 characters0 characters"

Explaining How Real-Time GenAI Works in a Noisy PubHostedbyConfluent

"Almost everyone has heard about large language models, and tens of millions of people have tried out OpenAI ChatGPT and Google Bard. However, the intricate architecture and underlying mathematics driving these remarkable systems remain elusive to many. LLM's are fascinating - so let's grab a drink and find out how these systems are built and dive deep into their inner workings. In the length of time it to enjoy a round of drinks, you'll understand the inner workings of these models. We'll take our first sip of word vectors, enjoy the refreshing taste of the transformer, and drain a glass understanding how these models are trained on phenomenally large quantities of data. Large language models for your streaming application - explained with a little maths and a lot of pub stories"

A Window Into Your Kafka Streams Tasks | KSLHostedbyConfluent

Kafka Streams relies on state restoration for maintaining standby tasks as failure recovery mechanism as well as for restoring the state after rebalance scenarios. When you are scaling up or down your application instances, it is necessary to know the current state of the restoration process for each active and standby task in order to prevent a long restoration process as much as possible. During this presentation, you will get an understanding of how KIP-869 provides valuable information about the current active task restoration after a rebalance and KIP-988 opens a window to the continuous process of standby restoration. When you encounter a situation in which you need to choose whether or not to scale up or down your application instances, both KIPs will be an invaluable ally for you.

Mastering Kafka Producer Configs: A Guide to Optimizing PerformanceHostedbyConfluent

"In this talk, we will dive into the world of Kafka producer configs and explore how to understand and optimize them for better performance. We will cover the different types of configs, their impact on performance, and how to tune them to achieve the best results. Whether you're new to Kafka or a seasoned pro, this session will provide valuable insights and practical tips for improving your Kafka producer performance. - Introduction to Kafka producer internal and workflow - Understanding the producer configs like linger.ms, batch.size, buffer.memory and their impact on performance - Learning about producer configs like max.block.ms, delivery.timeout.ms, request.timeout.ms and retries to make producer more resilient. - Discuss configs like enable.idempotence, max.in.flight.requests.per.connection and transaction related configs to achieve delivery guarantees. - Q&A session with attendees to address specific questions and concerns."

Data Contracts Management: Schema Registry and BeyondHostedbyConfluent

"Data contracts are one of the hottest topics in the data management community. A data contract is a formal agreement between a data producer and its consumers, aimed at reducing data downtime and improving data quality. Schemas are an important part of data contracts, but they are not the only relevant element. In this talk, we’ll: 1. see why data contracts are so important but also difficult to implement; 2. identify the characteristics of a well-designed data contract: discuss the anatomy of a data contract, its main elements and, how to formally describe them; 3. show how to manage the lifecycle of a data contract leveraging Confluent Platform's services."

Code-First Approach: Crafting Efficient Flink AppsHostedbyConfluent

"In the realm of stateful stream processing, Apache Flink has emerged as a powerful and versatile platform. However, the conventional SQL-based approach often limits the full potential of Flink applications. We will delve into the benefits of adopting a code-first approach, which provides developers with greater control over application logic, facilitates complex transformations, and enables more efficient handling of state and time. We will also discuss how the code-first approach can lead to more maintainable and testable code, ultimately improving the overall quality of your Flink applications. Whether you're a seasoned Flink developer or just starting your journey, this talk will provide valuable insights into how a code-first approach can revolutionize your stream processing applications."

Debezium vs. the World: An Overview of the CDC EcosystemHostedbyConfluent

"Change Data Capture (CDC) has become a commodity in data engineering, much in part due to the ever-rising success of Debezium [1]. But is that all there is? In this lightning talk, we’ll outline the current state of the CDC ecosystem, and understand why adopting a Debezium alternative is still a hard sell. If you’ve ever wondered what else is out there, but can’t keep up with the sprawling of new tools in the ecosystem; we’ll wrap it up for you! [1] https://debezium.io/"

Beyond Tiered Storage: Serverless Kafka with No Local DisksHostedbyConfluent

"Separation of compute and storage has become the de-facto standard in the data industry for batch processing. The addition of tiered storage to open source Apache Kafka is the first step in bringing true separation of compute and storage to the streaming world. In this talk, we'll discuss in technical detail how to take the concept of tiered storage to its logical extreme by building an Apache Kafka protocol compatible system that has zero local disks. Eliminating all local disks in the system requires not only separating storage from compute, but also separating data from metadata. This is a monumental task that requires reimagining Kafka's architecture from the ground up, but the benefits are worth it. This approach enables a stateless, elastic, and serverless deployment model that minimizes operational overhead and also drives inter-zone networking costs to almost zero."

Automating Speed: A Proven Approach to Preventing Performance Regressions in ...HostedbyConfluent

"Regular performance testing is one of the pillars of Kafka Streams’ reliability and efficiency. Beyond ensuring dependable releases, regular performance testing supports engineers in new feature development with the ability to easily test the performance impact of their features, compare different approaches, etc. In this session, Alex and John share their experience from developing, using, and maintaining a performance testing framework for Kafka Streams that has prevented multiple performance regressions over the last 5 years. They cover guiding principles and architecture, how to ensure statistical significance and stability of results, and how to automate regression detection for actionable notifications. This talk sheds light on how Apache Kafka is able to foster a vibrant open-source community while maintaining a high performance bar across many years and releases. It also empowers performance-minded engineers to avoid common pitfalls and bring high-quality performance testing to their own systems."