Building Retry Architectures in Kafka with Compacted Topics | Matthew Zhou, VilliageMD

Sep 27, 20211 like616 views

In this talk, we'll discuss how VillageMD is able to use Kafka topic compaction for rapidly scaling our reprocessing pipelines to encompass hundreds of feeds. Within healthcare data ecosystems, privacy and data minimalism are key design priorities. Being able to handle data deletion in a reliable, timely manner within event-driven architectures is becoming more and more necessary with key governance frameworks like the GDPR and HIPAA. We'll be giving an overview of the building and governance of dead-letter queues for streaming data processing. We'll discuss: 1. How to architect a data sink for failed records. 2. How topic compaction can reduce duplicate data and enable idempotency. 3. Building a tombstoning system for removing successfully reprocessed records from the queues. 4. Considerations for monitoring a reprocessing system in production -- what metrics, dataops, and SLAs are useful?

Kafka Summit
Implementing Retry Architectures
with Topic Compaction
Matthew Zhou
Senior Data Engineer @ Peloton

Building Maintainable
Data Retry
Architectures
Data processing failures are inevitable --
pipelines should anticipate those failures and
thoughtfully resolve them.
A reprocessing pipeline should seek to catalog common failure
patterns, prevent dropped data, alert the right people at the
right time, and trigger the correct resolution paths.
Each of these points encapsulates a deeper constellation of
data engineering concepts and details -- this talk aims to focus
on idempotent retries and operational lifecycles for data in a
Kafka system.

Why think about privacy
in software architecture?
• Legislation like the GDPR, the CCPA, and other
consumer digital protection bills might
mandate data minimalism and privacy audits
by law.
• Consumers are improving their data literacy
and may want to exercise stronger control and

Within a compacted topic, every payload requires a primary key.
Background threads managed by the Kafka broker compact messages sharing primary
keys to the most recent message. This process is eventually consistent based on a
configurable "dirty ratio". Records with a null payload are considered "tombstone"
records and signal the cleaner threads to remove all messages with that primary key.
Topic
Compaction
in Kafka
Offers finer-grained per-record
retention rather than time-
based retention within a Kafka
topic.

Some
benefits of
topic
compaction
Accommodates streaming per-record
retry needs.
01
Removes the need to track offsets when
reprocessing DLQs.
02
Eliminates duplicate data and
redundant work.
03
Minimizes the footprint of potentially
sensitive data.
04
Allows custom logic-based record
retention rather than time-based
retention.
05

Building a compacted DLQ system
STEP 1
This option is available using either
the Kafka built-in CLIs or the language
SDK being used to interact with the
brokers.
Initialize Kafka Topic with compaction
STEP 2
This consists of:
• Segment block byte size
• Retention time
• Dirty read ratio
Set topic compaction configs
STEP 3
After catching a raised exception
within application logic, define the
attribute set of metadata to inject
into the body payload. Configure a
primary key that will be used as the
compaction key in the DLQ.
Build the data payload and allocate
the primary key
STEP 4
If the Kafka broker cluster is down,
consider a failover pathway that might
hold messages in buffer until the
cluster is restored.
After receiving a successful response
from the broker, emit a tombstone
message to close out the
reprocessing work.
Emit your message and confirm
successful ack
STEP 5
A few useful metrics to persistently
monitor would be:
• queue size
• error volume profiles
• throughput spikiness
• time-alive in the queue
• time-to-resolution for
successfully reprocessed records.
Build monitors around DLQ metadata

Kafka Retry Architectures in Practice
Operational SLAs, Metrics, and Observability!
HOW TO EFFECTIVELY
MONITOR
• Build a consistent process
for registering alarm
thresholds for caught errors.
Post-mortems should
identify relevant metrics for
monitoring gaps.
• Two modes of manual
intervention:
⚬ Error throughput
thresholds
⚬ Error queue size
threshold
HOW TO RESOLVE
ISSUES
• Define your operational
SLAs on data pipelines
and clarify on-call pager
duty rotations.
• Are retries a manual or
automated process?
• What kind of metadata
filters are useful for
narrowing the search
space during
reprocessing?
• Is reprocessing logic
idempotent?
HOW TO GUARANTEE
COVERAGE
• The importance of setting
up health checks for your
infrastructure - who
monitors the monitor?
• Building out an error
catalog that allows fine-
grain error handling in
code.

This document summarizes Heng Zhang's presentation on improving logging ingestion quality at Pinterest. It discusses how Pinterest ingests large volumes of logging data at scale through a pipeline that favors scalability over consistency. This can lead to data corruption and loss issues. The presentation proposes a logging auditing framework to address these problems. It would add CRC checksums, audit headers and events at various stages to detect corrupted messages, track data loss metrics, and process audit events to remove bad data and provide alerts. The framework was tested and rolled out across Pinterest's ingestion pipelines with no downtime, improving data quality.

How Zillow Unlocked Kafka to 50 Teams in 8 months | Shahar Cizer Kobrinsky, Z...HostedbyConfluent

1. Zillow transitioned from using multiple messaging systems and data pipelines to using Kafka as their single streaming platform to unify their data infrastructure. 2. They took a bottom-up approach to gain trust from teams by publishing service level objectives, onboarding non-critical streams quickly, and meeting developers where they were with tools like Terraform. 3. An important lesson was to treat the platform as a product by providing documentation, libraries, and blog posts to make it easy for developers to use.

Enhancing Apache Kafka for Large Scale Real-Time Data Pipeline at Tencent | K...HostedbyConfluent

In this session we share our experience of building a real-time data pipelines at Tencent PCG - one that handles 20 trillion daily messages with 700 clusters and 100Gb/s bursting traffic from a single app. We discuss our roadmap of enhancing Kafka to break its limits in terms of scalability, robustness and cost of operation. We first built a proxy layer that aggregates physical clusters in a way agnostic to the clients. While this architecture solves many operational problems, it requires significant development to stay future-proof. With retrospection with our customer and careful study of the ongoing work from the community, we then designed a region federation solution in the broker layer, which allows us to deploy clusters at a much larger scale than previously possible, while at the same time providing better failure recovery and operability. We discuss how we make this development compatible with KIP-500 and KIP-405, and the two KIP (693, 694) that we submitted for discussion.

Kafka for Real-Time Event Processing in Serverless Environmentsconfluent

(Jeff Sharpe + Alex Srisuwan, Capital One) Kafka Summit SF 2018 Using Kafka as a platform messaging bus is common, but bridging communication between real-time and asynchronous components can become complicated, especially when dealing with serverless environments. This has become increasingly common in modern banking where events need to be processed at near-real-time speed. Serverless environments are well-suited to address these needs, and Kafka remains an excellent solution for providing the reliable, resilient communication layer between serverless components and dedicated stream processing services. In this talk, we will examine some of the strengths and weaknesses of using Kafka for real-time communication, some tips for efficient interactions with Kafka and AWS Lambda, and a number of useful patterns for maximizing the strengths of Kafka and serverless components.

How did we move the mountain? - Migrating 1 trillion+ messages per day across...HostedbyConfluent

Have you ever migrated Kafka clusters from one data center to another being completely transparent to client applications? At PayPal, as part of a massive datacenter migration initiative, Kafka team successfully moved all PayPal Kafka traffic across data centers. This initiative involved migrating 20+ Kafka clusters (1000+ broker and zookeeper nodes), as well as 60+ mirrormaker groups which seamlessly handle Kafka traffic volumes as high as 1 trillion messages per day. Throughout the course of this migration, applications required no modification, encountered 0% service outage, 0% message loss and duplicated messages. The whole migration process was fully transparent to Kafka applications. In this session, you will learn the strategies, techniques and tools the PayPal Kafka team has utilized for managing the migration process. You will also learn the lessons and pitfalls they experienced during this exercise, as well as the secret sauce of making the migration successful.

Capital One Delivers Risk Insights in Real Time with Stream Processingconfluent

Speakers: Ravi Dubey, Senior Manager, Software Engineering, Capital One + Jeff Sharpe, Software Engineer, Capital One Capital One supports interactions with real-time streaming transactional data using Apache Kafka®. Kafka helps deliver information to internal operation teams and bank tellers to assist with assessing risk and protect customers in a myriad of ways. Inside the bank, Kafka allows Capital One to build a real-time system that takes advantage of modern data and cloud technologies without exposing customers to unnecessary data breaches, or violating privacy regulations. These examples demonstrate how a streaming platform enables Capital One to act on their visions faster and in a more scalable way through the Kafka solution, helping establish Capital One as an innovator in the banking space. Join us for this online talk on lessons learned, best practices and technical patterns of Capital One’s deployment of Apache Kafka. -Find out how Kafka delivers on a 5-second service-level agreement (SLA) for inside branch tellers. -Learn how to combine and host data in-memory and prevent personally identifiable information (PII) violations of in-flight transactions. -Understand how Capital One manages Kafka Docker containers using Kubernetes. Watch the recording: https://videos.confluent.io/watch/6e6ukQNnmASwkf9Gkdhh69?.

Bravo Six, Going Realtime. Transitioning Activision Data Pipeline to Streamin...HostedbyConfluent

This document summarizes Activision Data's transition from a batch data pipeline to a real-time streaming data pipeline using Apache Kafka and Kafka Streams. Some key points: - The new pipeline ingests, processes, and stores game telemetry data from over 200k messages per second and over 5PB of data across 9 years of games. - Kafka Streams is used to transform the raw streaming data through multiple microservices with low 10-second end-to-end latency, compared to 6-24 hours previously. - Kafka Connect integrates the streaming data with data stores like AWS S3, Cassandra, and Elasticsearch. - The new pipeline provides real-time and historical access to structured

Scaling Security on 100s of Millions of Mobile Devices Using Apache Kafka® an...confluent

Lookout is a mobile cybersecurity company that ingests telemetry data from hundreds of millions of mobile devices to provide security scanning and apply corporate policies. They were facing scaling issues with their existing data pipeline and storage as the number of devices grew. They decided to use Apache Kafka and Confluent Platform for scalable data ingestion and ScyllaDB as the persistent store. Testing showed the new architecture could handle their target of 1 million devices with low latency and significantly lower costs compared to their previous DynamoDB-based solution. Key learnings included improving Kafka's default partitioner and working through issues during proof of concept testing with ScyllaDB.

Changing landscapes in data integration - Kafka Connect for near real-time da...HostedbyConfluent

1. The document discusses Kafka Connect and its evolution for managing near real-time streaming pipelines. It describes how Kafka Connect can be used for data integration across different systems and challenges around identity when deploying Kafka Connect. 2. It introduces the concept of a managed Kafka Connect which deploys Kafka Connect on the customer's own Kubernetes namespace to avoid security and identity issues. The managed Connect is configured and managed through a centralized control plane. 3. It details how the managed Kafka Connect control plane can be used to provision Kafka Connect clusters, deploy connectors between different systems, override connector configurations, and monitor tasks.

Kafka error handling patterns and best practices | Hemant Desale and Aruna Ka...HostedbyConfluent

Transaction Banking from Goldman Sachs is a high volume, latency sensitive digital banking platform offering. We have chosen an event driven architecture to build highly decoupled and independent microservices in a cloud native manner and are designed to meet the objectives of Security, Availability Latency and Scalability. Kafka was a natural choice – to decouple producers and consumers and to scale easily for high volume processing. However, there are certain aspects that require careful consideration – handling errors and partial failures, managing downtime of consumers, secure communication between brokers and producers / consumers. In this session, we will present the patterns and best practices that helped us build robust event driven applications. We will also present our solution approach that has been reused across multiple application domains. We hope that by sharing our experience, we can establish a reference implementation that application developers can benefit from.

Inside Kafka Streams—Monitoring Comcast’s Outside Plant confluent

(Mike Graham + Dan Carroll, Comcast) Kafka Summit SF 2018 Comcast manages over 2 million miles of fiber and coax, and over 40 million in home devices. This “outside plant” is subject to adverse conditions from severe weather to power grid outages to construction-related disruptions. Maintaining the health of this large and important infrastructure requires a distributed, scalable, reliable and fast information system capable of real-time processing and rapid analysis and response. Using Apache Kafka and the Kafka Streams Processor API, Comcast built an innovative new system for monitoring, problem analysis, metrics reporting and action response for the outside plant. In this talk, you’ll learn how topic partitions, state stores, key mapping, source and sink topics and processors from the Kafka Streams Processor API work together to build a powerful dynamic system. We will dive into the details about the inner workings of the state store—how it is backed by a Kafka “changelog” topic, how it is scaled horizontally by partition and how the instances are rebuilt on startup or on processor failure. We will discuss how these state stores essentially become like materialized views in a SQL database but are updated incrementally as data flows through the system, and how this allows the developers to maintain the data in the optimal structures for performing the processing. The best part is that the data is readily available when needed by the processors. You will see how a REST API using Kafka Streams “interactive queries” can be used to retrieve the data in the state stores. We will explore the deployment and monitoring mechanisms used to deliver this system as a set of independently deployed components.

Launching the Expedia Conversations Platform: From Zero to Production in Four...HostedbyConfluent

When we set out to launch our Conversations Platform at Expedia Group our goals were simple. Enable millions of travelers to have natural language conversations with an automated agent via text, Facebook, or their channel of choice. Let them book trips, make changes or cancellations, and ask questions -- “How long is my layover?” “Does my hotel have a pool?” How much will I get charged if I want to bring my golf clubs?”. Then take all that we know about that customer across all of our brands and apply machine learning models to give customers what they are looking for immediately and automatically, whether it be a straightforward answer or a complex new itinerary. And the final goal: go from zero to production in four months. Such a platform is no place for batch jobs, back-end processing, or offline APIs. To quickly make decisions that incorporate contextual information, the platform needs data in near real-time and it needs it from a wide range of services and systems. Meeting these needs meant architecting the Conversations Platform around a central nervous system based on Confluent Cloud and Apache Kafka. Kafka made it possible to orchestrate data from loosely coupled systems, enrich data as it flows between them so that by the time it reaches its destination it is ready to be acted upon, and surface aggregated data for analytics and reporting. Confluent Cloud made it possible for us to meet our tight launch deadline with limited resources. With event streaming as a managed service, we had no costly new hires to maintain our clusters and no worries about 24x7 reliability. When we built the platform, we did not foresee the worldwide pandemic and the profound effect it would have on the travel industry. Companies were hit with a tidal wave of customer questions, cancellations, and rebookings. Throughout this once-in-a-lifetime event, the Conversations Platform proved up to the challenge, auto-scaling as necessary and taking much of the load off of live agents. In this session, we’ll share how we built and deployed the Conversations Platform in just four months, the lessons we learned along the way, key points to consider for anyone architecting a platform with similar requirements, and how it handled the unprecedented demands placed upon it by the pandemic. We’ll also show a demo of the platform that includes high-level insights obtained from analytics and a visualization of the low-level events that make up a conversation.

Analyzing Petabyte Scale Financial Data with Apache Pinot and Apache Kafka | ...HostedbyConfluent

At Stripe, we operate a general ledger modeled as double-entry bookkeeping for all financial transactions. Warehousing such data is challenging due to its high volume and high cardinality of unique accounts. aFurthermore, it is financially critical to get up-to-date, accurate analytics over all records. Due to the changing nature of real time transactions, it is impossible to pre-compute the analytics as a fixed time series. We have overcome the challenge by creating a real time key-value store inside Pinot that can sustain half million QPS with all the financial transactions. We will talk about the details of our solution and the interesting technical challenges faced.

Kafka Summit SF 2017 - Query the Application, Not a Database: “Interactive Qu...confluent

Interactive Queries in Apache Kafka's Streams API allows users to query the local state of a Kafka Streams application without accessing an external database. It treats the Kafka Streams application as an embedded, lightweight database. The local state is fault-tolerant and can be sharded across tasks to scale horizontally. Users can discover other application instances and their state to perform queries on remote state if needed. Interactive Queries simplifies stateful stream processing by reducing moving parts compared to using an external database.

Mind the App: How to Monitor Your Kafka Streams Applications | Bruno Cadonna,...HostedbyConfluent

You cannot operate what you cannot measure. In this talk, I am going to present the built-in metrics framework of Kafka Streams that supports monitoring Kafka Streams applications. You will learn how to setup monitoring of metrics for your Kafka Streams applications and you will hear about the following recent improvements to the metrics framework that aim to extend and simplify monitoring. KIP-444 aims to simplify and extend the built-in metrics framework. The RocksDB metrics introduced in KIP-471 and KIP-607 allow you to look directly into the built-in persistent state stores of your Kafka Streams applications. Finally, KIP-613 specifies metrics that measure end-to-end latencies in your applications. This talk will help you collect intel about the behavior of your Kafka Streams applications, and will allow you to reason about the deployment. In the end, you will be able to better understand your applications and run them in a more robust manner.

URP? Excuse You! The Three Metrics You Have to Know confluent

(Todd Palino, LinkedIn) Kafka Summit SF 2018 What do you really know about how to monitor a Kafka cluster for problems? Is your most reliable monitoring your users telling you there’s something broken? Are you capturing more metrics than the actual data being produced? Sure, we all know how to monitor disk and network, but when it comes to the state of the brokers, many of us are still unsure of which metrics we should be watching, and what their patterns mean for the state of the cluster. Kafka has hundreds of measurements, from the high-level numbers that are often meaningless to the per-partition metrics that stack up by the thousands as our data grows. We will thoroughly explore three key monitoring concepts in the broker, that will leave you an expert in identifying problems with the least amount of pain: -Under-replicated Partitions: The mother of all metrics -Request Latencies: Why your users complain -Thread pool utilization: How could 80% be a problem? We will also discuss the necessity of availability monitoring and how to use it to get a true picture of what your users see, before they come beating down your door!

Supercharge Your Real-time Event Processing with Neo4j's Streams Kafka Connec...HostedbyConfluent

Do your event streams use connected-data domains such as fraud detection, live logistics routing, or predicting network outages? How can you maintain the analysis and leverage those connections real-time? Graph databases differ from traditional, tabular ones in that they treat connections between data as first class citizens. This means they are optimized for detecting and understanding these relationships – providing insight at speed and at scale. By combining event streams from Kafka along with the power of the Neo4j graph database for interrogating and investigating connections, you make real-time, event-driven intelligent insight a reality. Neo4j Streams integrates Neo4j with Apache Kafka event streams, to serve as a source of data, for instance Change Data Capture or a sink to ingest any kind of Kafka event into your graph. In this session we’ll show you how to get up and running with Neo4j Streams to show you how to sink and source between graphs and streams.

Westpac Bank Tech Talk 1: Dive into Apache Kafkaconfluent

Taming a massive fleet of Python-based Kafka apps at Robinhood | Chandra Kuch...HostedbyConfluent

Robinhood uses Kafka in every line of its business, from stock and crypto trading to clearing and data analytics. One interesting aspect of our architecture is that many of our microservices leveraging Kafka are written in Python. When you combine Python's relatively slow performance coupled, its reliance on process-based parallelism and Robinhood’s scale, the result is a massive fleet of application processes producing to and consuming from our Kafka clusters. This fleet generates an atypical workload on Kafka that warrants a deeper investment in scalability and reliability. This talk discusses our investments in Kafka infrastructure for a large-scale Python-based environment: kafkahood: our librdkafka-based client library wrapper that codifies best practices, sane defaults and deep client-side observability. kafkaproxy: a Rust-based sidecar proxy that reduces connection fan-in from Python gunicorn worker pools to our Kafka clusters. We'll also present challenges we encountered along the way and share our learnings with the audience.

Apache Kafka from 0.7 to 1.0, History and Lesson LearnedGuozhang Wang

The Road Most Traveled: A Kafka Story | Heikki Nousiainen, AivenHostedbyConfluent

When moving to a cloud native architecture Moogsoft knew they needed more scale than Rabbit could provide. Moogsoft moved into Kafka which is known for quick writing and driving heavy event driven workloads on top of niceties such as replayability. Choosing the tool was easy, finding a vendor that ticked all their boxes was not. They needed to ensure scalability, upgradability, builds via existing IAC pipelines, and observability via existing tools. When Moogsoft found Aiven, they were impressed with their offering and ability to scale on demand. During this presentation we will explore how Moogsoft used Aiven for Kafka to manage and scale their data in the cloud.

Building Microservices with Apache Kafkaconfluent

Building distributed systems is challenging. Luckily, Apache Kafka provides a powerful toolkit for putting together big services as a set of scalable, decoupled components. In this talk, I'll describe some of the design tradeoffs when building microservices, and how Kafka's powerful abstractions can help. I'll also talk a little bit about what the community has been up to with Kafka Streams, Kafka Connect, and exactly-once semantics. Presentation by Colin McCabe, Confluent, Big Data Day LA

Kafka Summit SF 2017 - Database Streaming at WePayconfluent

This document discusses WePay's use of Kafka and Debezium for real-time data warehousing. Debezium is used to stream database changes from MySQL to Kafka. The Kafka Connect BigQuery connector then loads data from Kafka into BigQuery. This provides lower latency compared to WePay's previous ETL system. Key benefits include handling schema changes, retries on errors, and view deduplication in BigQuery. Future work includes integrating more of WePay's monolithic database and addressing issues like metrics and compatibility checking as the system scales.

Hadoop made fast - Why Virtual Reality Needed Stream Processing to Surviveconfluent

1. A streaming platform like Kafka can provide the benefits of Hadoop for batch processing but in a faster, real-time way by processing data as it arrives rather than storing all data. 2. Virtual reality applications require stream processing to power features like VR mirroring and capture in real-time. Kafka's stream processing capabilities address challenges like this for VR. 3. The document discusses how AltspaceVR uses Kafka stream processing for applications like VR mirroring and capture, presence tracking, scheduled tasks, and more to power their real-time VR experiences.

Keeping Analytics Data Fresh in a Streaming Architecture | John Neal, QlikHostedbyConfluent

Qlik is an industry leader across its solution stack, both on the Data Integration side of things with Qlik Replicate (real-time CDC) and Qlik Compose (data warehouse and data lake automation), and on the Analytics side with Qlik Sense. These two “sides” of Qlik are coming together more frequently these days as the need for “always fresh” data increases across organizations. When real-time streaming applications are the topic du jour, those companies are looking to Apache Kafka to provide the architectural backbone those applications require. Those same companies turn to Qlik Replicate to put the data from their enterprise database systems into motion at scale, whether that data resides in “legacy” mainframe databases; traditional relational databases such as Oracle, MySQL, or SQL Server; or applications such as SAP and SalesForce. In this session we will look in depth at how Qlik Replicate can be used to continuously stream changes from a source database into Apache Kafka. From there, we will explore how a purpose-built consumer can be used to provide the bridge between Apache Kafka and an analytics application such as Qlik Sense.

Stream Processing with Apache Kafka and .NETconfluent

Presentation from South Bay.NET meetup on 3/30. Speaker: Matt Howlett, Software Engineer at Confluent Apache Kafka is a scalable streaming platform that forms a key part of the infrastructure at many companies including Uber, Netflix, Walmart, Airbnb, Goldman Sachs and LinkedIn. In this talk Matt will give a technical overview of Kafka, discuss some typical use cases (from surge pricing to fraud detection to web analytics) and show you how to use Kafka from within your C#/.NET applications.

Give Your Confluent Platform Superpowers! (Sandeep Togrika, Intel and Bert Ha...HostedbyConfluent

Whether you are a die-hard DC comic enthusiast, mad for Marvel, or completely clueless when it comes to comic books, at the end of the day each of us would love to possess the superpower to transform data in seconds versus minutes or days. But architects and developers are challenged with designing and managing platforms that scale elastically and combine event streams with stored data, to enable more contextually rich data analytics. This made even more complex with data coming from hundreds of sources, and in hundreds of terabytes, or even petabytes, per day. Now, with Apache Kafka and Intel hardware technology advances, organizations can turn massive volumes of disparate data into actionable insights with the ability to filter, enrich, join and process data instream. Let's consider Information Security. IT leaders need to ensure all company data and IP is secured against threats and vulnerabilities. A combination of real-time event streaming with Confluent Platform and Intel Architecture has enabled threat detection efforts that once took hours to be completed in seconds, while simultaneously reducing technical debt and data processing and storage costs. In this session, Confluent and Intel architects will share detailed performance benchmarking results and new joint reference architecture. We’ll detail ways to remove Kafka performance bottlenecks, and improve platform resiliency and ensure high availability using Confluent Control Center and Multi-Region Clusters. And we’ll offer up tips for addressing challenges that you may be facing in your own super heroic efforts to design, deploy, and manage your organization’s data platforms.

How Yelp Leapt to Microservices with More than a Message Queueconfluent

Without seeing what’s wrong with today’s messaging queues, it can be initially confusing to view Apache Kafka as more. By adding additional functionality, true storage, and guarantees it opens opportunities to take full advantage of a publish/subscribe model. Joined by Yelp’s Justin Cunningham we’ll see how their infrastructure has quickly evolved. Powered by Kafka, Yelp has made the leap to microservices and is seeing the benefits of efficiency and performance. Speakers: Justin Cunningham Technical Lead, Software Engineer, Yelp Gehrig Kunz Technical Product Marketing Manager, Confluent

Removing performance bottlenecks with Kafka Monitoring and topic configurationKnoldus Inc.

Apache Kafka is a distributed messaging system used to build real-time data pipelines & streaming applications. Since applications rely heavily on efficient data transfer, message passing platforms like Kafka cannot afford a breakdown or poor performance. But how do we ensure that Kafka is running well and successfully streaming messages at low latency? This is where Kafka monitoring steps in. Here’s the agenda of the webinar - > Why Kafka monitoring? > Top 10 Kafka metrics to focus on > How to change Kafka topic configuration at runtime?

APAC Kafka Summit - Best Of confluent

More Related Content

What's hot (20)