Multi-DC Kafka

Apr 30, 20200 likes292 views

1. The document discusses various options for implementing disaster recovery and high availability with Kafka across multiple data centers, including MirrorMaker, MirrorMaker 2, and stretch clusters. 2. MirrorMaker provides basic asynchronous replication between data centers but has limitations around failover and latency. MirrorMaker 2 and replicators support active-active production in both DCs but with more complexity. 3. Stretch clusters treat the multiple DCs as a single Kafka cluster with synchronous replication and no producer latency, but require more resources and rely on low WAN latency. The best option depends on requirements for developer ease, latency, consistency, and budget.

The talk will start shortly - before then, have a chat with your fellow attendees!
Questions at the end
This is unless the speaker says
otherwise. But generally,
because of internet delays and
lags it can be a bit disruptive
during an Online talk.
Online Meetup Etiquette
Mute during talk
But remember to unmute when
you want to ask a question at
the end (or chat with fellow
attendees before/After)
Use chat to engage
Use Zoom chat and/or create a
thread in the #events channel
on our Community Slack space
so that your discussion can
continue afterwards with fellow
Kafkateers.
Be on Camera & React!
Speakers during these events
are talking to blank screens. And
it can be exhausting! Use Zoom
reactions at the bottom of the
page to give them something to
work off of!
1.
2.
Continue learning and collaborating
WELCOME!
Thank you for joining us in these unique circumstances. We hope you’re safe and well.
3. If you haven’t already joined, get on the our slack workspace
and continue the conversation with other Kafkateers. If
you’re already a member, start a thread for this event in
#events.
A single online source of everything you’ll
need to learn Kafka. Plus it’s totally free and
ungated…
Conﬂuent Developer
developer.conﬂuent.iohttp://bit.ly/ConﬂuentCommunity-Slack
Conﬂuent Community Slack
You are being recorded and this footage may be added to public channels

Multi-DC Kafka
Mitchell Henderson, Staff Technical Account Manager

3
Refresh - What HA does Kafka offer in a single
cluster?
Automatic Client
failover
Automatic client
retries
Prefered partition
leadership election
Durability Guarantees

4
The two categories, they overlap a lot.
Recovery point objective
Recovery time objective
What events am I worried about?
DR
HA
How disconnected are my sites?
What consistency can I handle? At
what price?

What are my options?
7
Disaster recovery or point of presence
MirrorMaker - Replicator
High Availability
MirrorMaker 2 - Replicator
Super High Availability
Stretch Cluster

9
Trade offs
Developer Ease Latency vs Consistency
Cost

10
Mirror Maker
Mirror Maker
Source Destination
01100001 01101110
01101110 01100001

12
Mirror Maker
DNS Changes Offset Translation Custom Client Code

13
Pro
• Simple
• Easy to understand
• Cheap
• Asynchronous
Con
• Single direction
• Difﬁcult failover - All logic
handled by clients
• Undetermined amount of
latency between data
centers.
Recovery Point Objective
Recovery Time Objective ~Minutes

14
Mirror Maker 2 & Replicator
Source Destination
01100001 01101110
01101110 01100001
01101101 01101001
01110100 01100011
01101000
Offsets
Offsets

16
Pro
• Bi-Directional
• Active Active - Can produce
in both data centers
• Asynchronous
Con
• Complex
• Undetermined amount of
latency between data
centers.
• Failover handled by clients
Recovery Point Objective
Recovery Time Objective Minutes

18
Pro
• Single direction
• Active Active Producers on
different topics
• Synchronous - Async
possible with CE
• All the goodness of a single
kafka cluster
Con
• Complex - Maybe?
• Producer latency
• Expensive
• Highly reliant on WAN
Recovery Point Objective
Recovery Time Objective 0

19
Disaster recovery or point of presence - Very easy
MirrorMaker - Replicator
High Availability - middle ground
MirrorMaker 2 - Replicator
Super High Availability - Developer ease - costly
Stretch Cluster

20
Thank You!
@mitch on Conﬂuent Community Slack
http://bit.ly/ConﬂuentCommunity-Slack
mitch@conﬂuent.io

Watch this talk here: https://www.confluent.io/online-talks/apache-kafka-architecture-and-fundamentals-explained-on-demand This session explains Apache Kafka’s internal design and architecture. Companies like LinkedIn are now sending more than 1 trillion messages per day to Apache Kafka. Learn about the underlying design in Kafka that leads to such high throughput. This talk provides a comprehensive overview of Kafka architecture and internal functions, including: -Topics, partitions and segments -The commit log and streams -Brokers and broker replication -Producer basics -Consumers, consumer groups and offsets This session is part 2 of 4 in our Fundamentals for Apache Kafka series.

Spring Boot+Kafka: the New Enterprise PlatformVMware Tanzu

This document discusses how Spring Boot and Kafka can form the basis of a new enterprise application platform focused on continuous delivery, event-driven architectures, and streaming data. It provides examples of companies that have successfully adopted this approach, such as Netflix transitioning to Spring Boot and a banking brand building a new core banking system using Spring Streams and Kafka. The document advocates an "event-first" and microservices-oriented mindset enabled by a streaming data platform and suggests that Spring Boot, Kafka, and related technologies provide a turnkey solution for implementing this new application development approach at large enterprises.

Everything you ever needed to know about Kafka on Kubernetes but were afraid ...HostedbyConfluent

Kubernetes became the de-facto standard for running cloud-native applications. And many users turn to it also to run stateful applications such as Apache Kafka. You can use different tools to deploy Kafka on Kubernetes - write your own YAML files, use Helm Charts, or go for one of the available operators. But there is one thing all of these have in common. You still need very good knowledge of Kubernetes to make sure your Kafka cluster works properly in all situations. This talk will cover different Kubernetes features such as resources, affinity, tolerations, pod disruption budgets, topology spread constraints and more. And it will explain why they are important for Apache Kafka and how to use them. If you are interested in running Kafka on Kubernetes and do not know all of these, this is a talk for you.

Organic Growth and A Good Night Sleep: Effective Kafka Operations at Pinteres...confluent

Vahid Hashemian and Ambud Sharma from Pinterest discuss Kafka operations at their company. Pinterest uses over 50 Kafka clusters with 2,500+ brokers to ingest 20+ GB/s of data and output 50+ GB/s. They faced challenges around performance, costs, and dynamic partitioning. To address these, Pinterest developed automation tools like Orion to manage clusters and topics, upgraded all clusters to Kafka 2.3.1+, and learned lessons around testing upgrade versions and backward compatibility. Going forward, they aim to improve interoperability, scaling, efficiency, and reliability.

Monitoring and Resiliency Testing our Apache Kafka Clusters at Goldman Sachs ...HostedbyConfluent

In our payments platform at Goldman Sachs Transaction Banking, Apache Kafka plays a critical role as the messaging bus in our micro-services architecture. Being a part of the financial service industry we need to ensure high-availability of our platform and quick response time during failures. In this talk we will explore how we monitor and alert on the health of our Kafka clusters using our heartbeat application and clients using DataDog dashboards. We will see how we consolidate JMX metrics such as error-rates, connection-rates, latencies and consumer lag from all producers and consumers using JMX agent sidecar to provide a live view of the health of our entire infrastructure. We will also discuss our culture of game days where we regularly test the resiliency of all the clients in our infrastructure by simulating various failure scenarios to improve the overall availability of our infrastructure.

Building High-Throughput, Low-Latency Pipelines in Kafkaconfluent

William Hill is one of the UK’s largest, most well-established gaming companies with a global presence across 9 countries with over 16,000 employees. In recent years the gaming industry and in particular sports betting, has been revolutionised by technology. Customers now demand a wide range of events and markets to bet on both pre-game and in-play 24/7. This has driven out a business need to process more data, provide more updates and offer more markets and prices in real time. At William Hill, we have invested in a completely new trading platform using Apache Kafka. We process vast quantities of data from a variety of feeds, this data is fed through a variety of odds compilation models, before being piped out to UI apps for use by our trading teams to provide events, markets and pricing data out to various end points across the whole of William Hill. We deal with thousands of sporting events, each with sometimes hundreds of betting markets, each market receiving hundreds of updates. This scales up to vast numbers of messages flowing through our system. We have to process, transform and route that data in real time. Using Apache Kafka, we have built a high throughput, low latency pipeline, based on Cloud hosted Microservices. When we started, we were on a steep learning curve with Kafka, Microservices and associated technologies. This led to fast learnings and fast failings. In this session, we will tell the story of what we built, what went well, what didn’t go so well and what we learnt. This is a story of how a team of developers learnt (and are still learning) how to use Kafka. We hope that you will be able to take away lessons and learnings of how to build a data processing pipeline with Apache Kafka.

Westpac Bank Tech Talk 1: Dive into Apache Kafkaconfluent

Kafka Streams for Java enthusiastsSlim Baltagi

Kafka, Apache Kafka evolved from an enterprise messaging system to a fully distributed streaming data platform (Kafka Core + Kafka Connect + Kafka Streams) for building streaming data pipelines and streaming data applications. This talk, that I gave at the Chicago Java Users Group (CJUG) on June 8th 2017, is mainly focusing on Kafka Streams, a lightweight open source Java library for building stream processing applications on top of Kafka using Kafka topics as input/output. You will learn more about the following: 1. Apache Kafka: a Streaming Data Platform 2. Overview of Kafka Streams: Before Kafka Streams? What is Kafka Streams? Why Kafka Streams? What are Kafka Streams key concepts? Kafka Streams APIs and code examples? 3. Writing, deploying and running your first Kafka Streams application 4. Code and Demo of an end-to-end Kafka-based Streaming Data Application 5. Where to go from here?

Not Your Mother's Kafka - Deep Dive into Confluent Cloud Infrastructure | Gwe...HostedbyConfluent

Confluent Cloud runs a modified version of Apache Kafka - redesigned to be cloud-native and deliver a serverless user experience. In this talk, we will discuss key improvements we've made to Kafka and how they contribute to Confluent Cloud availability, elasticity, and multi-tenancy. You'll learn about innovations that you can use on-prem, and everything you need to make the most of Confluent Cloud.

Asynchronous Transaction Processing With Kafka as a Single Source of Truth - ...HostedbyConfluent

This document discusses using Kafka as a central backbone for building an asynchronous mobile ticketing application. It describes moving from a monolithic backend to a microservices architecture with separate databases for each service. It then outlines using Kafka to facilitate asynchronous communication between services, allowing for easy scaling, distribution, and evolution. Transaction management is achieved through message ordering in Kafka and a SAGA pattern. Leveraging Kafka Streams enables materialized views without a separate database. Overall this approach provides scalability, distribution, ease of evolution, and an improved user experience.

Fundamentals of Apache KafkaChhavi Parasher

Kafka Cluster Federation at Uber (Yupeng Fui & Xiaoman Dong, Uber) Kafka Summ...confluent

Uber has one of the largest Kafka deployment in the industry. To improve the scalability and availability, we developed and deployed a novel federated Kafka cluster setup which hides the cluster details from producers/consumers. Users do not need to know which cluster a topic resides and the clients view a "logical cluster". The federation layer will map the clients to the actual physical clusters, and keep the location of the physical cluster transparent from the user. Cluster federation brings us several benefits to support our business growth and ease our daily operation. In particular, Client control. Inside Uber there are a large of applications and clients on Kafka, and it's challenging to migrate a topic with live consumers between clusters. Coordinations with the users are usually needed to shift their traffic to the migrated cluster. Cluster federation enables much control of the clients from the server side by enabling consumer traffic redirection to another physical cluster without restarting the application. Scalability: With federation, the Kafka service can horizontally scale by adding more clusters when a cluster is full. The topics can freely migrate to a new cluster without notifying the users or restarting the clients. Moreover, no matter how many physical clusters we manage per topic type, from the user perspective, they view only one logical cluster. Availability: With a topic replicated to at least two clusters we can tolerate a single cluster failure by redirecting the clients to the secondary cluster without performing a region-failover. This also provides much freedom and alleviates the risks for us to carry out important maintenance on a critical cluster. Before the maintenance, we mark the cluster as a secondary and migrate off the live traffic and consumers. We will present the details of the architecture and several interesting technical challenges we overcame.

Fundamentals and Architecture of Apache KafkaAngelo Cesaro

Scaling Security on 100s of Millions of Mobile Devices Using Apache Kafka® an...confluent

Lookout is a mobile cybersecurity company that ingests telemetry data from hundreds of millions of mobile devices to provide security scanning and apply corporate policies. They were facing scaling issues with their existing data pipeline and storage as the number of devices grew. They decided to use Apache Kafka and Confluent Platform for scalable data ingestion and ScyllaDB as the persistent store. Testing showed the new architecture could handle their target of 1 million devices with low latency and significantly lower costs compared to their previous DynamoDB-based solution. Key learnings included improving Kafka's default partitioner and working through issues during proof of concept testing with ScyllaDB.

Tradeoffs in Distributed Systems Design: Is Kafka The Best? (Ben Stopford and...HostedbyConfluent

When choosing an event streaming platform, Kafka shouldn’t be the only technology you look at. There are a plethora of others in the messaging space today, including open source and proprietary software as well as a range of cloud services. So how do you know you are choosing the right one? A great way to deepen our understanding of event streaming and Kafka is exploring the trade-offs in distributed system design and learning about the choices made by the Kafka project. We’ll look at how Kafka stacks up against other technologies in the space, including traditional messaging systems like Apache ActiveMQ and RabbitMQ as well as more contemporary ones, such as BookKeeper derivatives like Apache Pulsar or Pravega. This talk focuses on the technical details such as difference in messaging models, how data is stored locally as well as across machines in a cluster, when (not) to add tiers to your system, and more. By the end of the talk, you should have a good high-level understanding of how these systems compare and which you should choose for different types of use cases.

Reducing Microservice Complexity with Kafka and Reactive Streamsjimriecken

My talk from ScalaDays 2016 in New York on May 11, 2016: Transitioning from a monolithic application to a set of microservices can help increase performance and scalability, but it can also drastically increase complexity. Layers of inter-service network calls for add latency and an increasing risk of failure where previously only local function calls existed. In this talk, I'll speak about how to tame this complexity using Apache Kafka and Reactive Streams to: - Extract non-critical processing from the critical path of your application to reduce request latency - Provide back-pressure to handle both slow and fast producers/consumers - Maintain high availability, high performance, and reliable messaging - Evolve message payloads while maintaining backwards and forwards compatibility.

Introducing Confluent labs Parallel Consumer client | Anthony Stubbes, ConfluentHostedbyConfluent

Consuming messages in parallel is what Apache Kafka® is all about, so you may well wonder, why would we want anything else? It turns out that, in practice, there are a number of situations where Kafka’s partition-level parallelism gets in the way of optimal design. This session will go over some of these types of situations that can benefit from parallel message processing within a single application instance (aka slow consumers or competing consumers), and then introduce the new Parallel Consumer labs project from Confluent, which can improve functionality and massively improve performance in such situations. It will cover the - Different ordering modes of the client - Relative performance improvements - Usage with other components like Kafka Streams - An introduction to the internal architecture of the project - How it can achieve all this in a reassignment friendly manner

Apache Kafka - Martin PodvalMartin Podval

Apache Kafka is a distributed messaging system that allows for publishing and subscribing to streams of records, known as topics, in a fault-tolerant and scalable way. It is used for building real-time data pipelines and streaming apps. Producers write data to topics which are committed to disks across partitions and replicated for fault tolerance. Consumers read data from topics in a decoupled manner based on offsets. Kafka can process streaming data in real-time and at large volumes with low latency and high throughput.

KafkaConsumer - Decoupling Consumption and Processing for Better Resource Uti...confluent

When working with KafkaConsumer, we usually employ single thread both for reading and processing of messages. KafkaConsumer is not thread-safe, so using single thread fits in well. Downside of this approach is that you are limited to single thread for processing messages. By decoupling consumption and processing, we can achieve processing parallelization with single consumer and get the most out of multi-core CPU architectures available today. While this can be very useful in certain use-case scenarios, it's not trivial to implement. How do we use multiple threads with KafkaConsumer which is not thread safe? How do we react to consumer group rebalancing? Can we get desired processing and ordering guarantees? In this talk we 'll try to answer these questions and explore challenges we face on our path.

Introduction to apache kafkaDimitris Kontokostas

Apache Kafka is a distributed streaming platform and distributed publish-subscribe messaging system. It uses a log abstraction to order events and replicate data across clusters. Kafka allows developers to publish and subscribe to streams of records known as topics. Producers publish data to topics and consumers subscribe to topics to process streams of records. The Kafka ecosystem includes tools like KStreams for stream processing and KSQL for querying streams of data.

Introducción a Stream Processing utilizando Kafka Streamsconfluent

Delivering: from Kafka to WebSockets | Adam Warski, SoftwareMillHostedbyConfluent

Here's the challenge: we've got a Kafka topic, where services publish messages to be delivered to browser-based clients through web sockets. Sounds simple? It might, but we're faced with an increasing number of messages, as well as a growing count of web socket clients. How do we scale our solution? As our system contains a larger number of servers, failures become more frequent. How to ensure fault tolerance? There’s a couple possible architectures. Each websocket node might consume all messages. Otherwise, we need an intermediary, which redistributes the messages to the proper web socket nodes. Here, we might either use a Kafka topic, or a streaming forwarding service. However, we still need a feedback loop so that the intermediary knows where to distribute messages. We’ll take a look at the strengths and weaknesses of each solution, as well as limitations created by the chosen technologies (Kafka and web sockets).

Building a Web Application with Kafka as your Databaseconfluent

This document provides an overview of a presentation on building a web application using Apache Kafka as the database. The presentation discusses using Kafka for messaging in both request-driven and event-driven architectures. It then covers using Kafka and Kafka Streams as an alternative to a traditional database for the benefits of performance, scaling, and being the system of record. The presentation reviews building a quiz application called Quizzer that uses Kafka Streams to store and access data, including designing topologies for starting quizzes, submitting answers, and retrieving next questions. Key takeaways include using Kafka Streams features like transformations, joins, and state stores to access data within topologies.

URP? Excuse You! The Three Metrics You Have to Know confluent

(Todd Palino, LinkedIn) Kafka Summit SF 2018 What do you really know about how to monitor a Kafka cluster for problems? Is your most reliable monitoring your users telling you there’s something broken? Are you capturing more metrics than the actual data being produced? Sure, we all know how to monitor disk and network, but when it comes to the state of the brokers, many of us are still unsure of which metrics we should be watching, and what their patterns mean for the state of the cluster. Kafka has hundreds of measurements, from the high-level numbers that are often meaningless to the per-partition metrics that stack up by the thousands as our data grows. We will thoroughly explore three key monitoring concepts in the broker, that will leave you an expert in identifying problems with the least amount of pain: -Under-replicated Partitions: The mother of all metrics -Request Latencies: Why your users complain -Thread pool utilization: How could 80% be a problem? We will also discuss the necessity of availability monitoring and how to use it to get a true picture of what your users see, before they come beating down your door!

Learnings from the Field. Lessons from Working with Dozens of Small & Large D...HostedbyConfluent

- Upgrades should be done often to get bug fixes and improvements, following the upgrade guide carefully. Start with a healthy cluster and upgrade components outward from Zookeeper to Kafka brokers to clients. Don't rush the process or have any unresolved partition reassignments. - Collect JMX metrics to monitor the cluster as outages can be prolonged without visibility. The Kafka defaults are suitable for single node deployments but replication factor, threads, and broker configuration should be tuned for larger clusters. - Quotas like replication throttling and bandwidth/request limits per client or topic should be used to protect the cluster and clients. Log files should separate each component and be retained for a few days. Consider multiple clusters by SLA

Kafka Summit SF 2017 - Kafka Connect Best Practices – Advice from the Fieldconfluent

This document discusses best practices for using Apache Kafka Connect. It begins with an overview of Kafka Connect basics like connectors, converters, transforms, and plugins. It then discusses choosing the right connectors for different data sources and sinks, and how to test connectors using the Confluent CLI. The document concludes with recommendations for planning Kafka Connect deployments, such as understanding schemas, deploying connectors across workers, tuning configurations, and minimizing rebalances.

Kafka awsAriel Moskovich

This document discusses building a fault-tolerant Kafka cluster on AWS to handle 2.5 billion requests per day. It covers choosing AWS instance types and broker counts, spreading brokers across availability zones, configuring replication and partitioning, automating fault tolerance, adding metrics and alerts, and testing the cluster's resilience. Key decisions include broker placement, topic partitioning, Zookeeper ensemble sizing, and automation to dynamically reassign partitions and change configurations in response to failures or added capacity.

Can Kafka Handle a Lyft Ride? (Andrey Falko & Can Cecen, Lyft) Kafka Summit 2020HostedbyConfluent

What does a Kafka administrator need to do if they have a user who demands that message delivery be guaranteed, fast, and low cost? In this talk we walk through the architecture we created to deliver for such users. Learn around the alternatives we considered and the pros and cons around what we came up with. In this talk, we’ll be forced to dive into broker restart and failure scenarios and things we need to do to prevent leader elections from slowing down incoming requests. We’ll need to take care of the consumers as well to ensure that they don’t process the same request twice. We also plan to describe our architecture by showing a demo of simulated requests being produced into Kafka clusters and consumers processing them in lieu of us aggressively causing failures on the Kafka clusters. We hope the audience walks away with a deeper understanding of what it takes to build robust Kafka clients and how to tune them to accomplish stringent delivery guarantees.

Distributed Scrum: Dangerous Waters-Be Prepared!TechWell

Even though team collocation is strongly recommended in agile methodologies, a distributed team is often required in the real world today. What is so important about collocating anyway? Can you overcome the challenges of a distributed Scrum team and still remain agile? What are the solutions? Brian Saylor tackles these important questions and more. While Brian realizes that implementing Scrum and agile practices in a distributed team is not easy, he found that it is possible if you understand the inherent problems and work hard-every day-to overcome them. Brian walks you through the reasons collocating is important for agile teams and the extra challenges distributed agile teams face. Then he dives into practical, real-world tools, tips, and techniques that organizations should research and consider before jumping into distributed waters-and don’t forget your life jacket.

Jax Devops 2017 Succeeding in the Cloud – the guidebook of FailSteve Poole

Many have gone before you along this path. Many have failed. A few have succeeded. All have scars. Although the journey is different for everyone there are common aspects to them all. In this talk we will cover our experiences in moving applications into the Cloud. What you must do. What you must not. What matters, what doesn’t. In moving to the cloud there is no try. In this talk: * We’ll cover the core aspects of how the cloud differs from local data centers in terms of application design, runtime characteristics and operational considerations. * We’ll explain through various real life examples where things worked and where they didnt * We end with a summary of the key elements to success and the major pitfalls to avoid.

More Related Content

What's hot (20)

Not Your Mother's Kafka - Deep Dive into Confluent Cloud Infrastructure | Gwe...HostedbyConfluent

Asynchronous Transaction Processing With Kafka as a Single Source of Truth - ...HostedbyConfluent

Fundamentals of Apache KafkaChhavi Parasher

Kafka Cluster Federation at Uber (Yupeng Fui & Xiaoman Dong, Uber) Kafka Summ...confluent

Fundamentals and Architecture of Apache KafkaAngelo Cesaro

Scaling Security on 100s of Millions of Mobile Devices Using Apache Kafka® an...confluent

Tradeoffs in Distributed Systems Design: Is Kafka The Best? (Ben Stopford and...HostedbyConfluent

Reducing Microservice Complexity with Kafka and Reactive Streamsjimriecken

Introducing Confluent labs Parallel Consumer client | Anthony Stubbes, ConfluentHostedbyConfluent

Apache Kafka - Martin PodvalMartin Podval

KafkaConsumer - Decoupling Consumption and Processing for Better Resource Uti...confluent

Introduction to apache kafkaDimitris Kontokostas

Introducción a Stream Processing utilizando Kafka Streamsconfluent

Delivering: from Kafka to WebSockets | Adam Warski, SoftwareMillHostedbyConfluent

Building a Web Application with Kafka as your Databaseconfluent

URP? Excuse You! The Three Metrics You Have to Know confluent

Learnings from the Field. Lessons from Working with Dozens of Small & Large D...HostedbyConfluent

Kafka Summit SF 2017 - Kafka Connect Best Practices – Advice from the Fieldconfluent

Kafka awsAriel Moskovich

Can Kafka Handle a Lyft Ride? (Andrey Falko & Can Cecen, Lyft) Kafka Summit 2020HostedbyConfluent

Not Your Mother's Kafka - Deep Dive into Confluent Cloud Infrastructure | Gwe...HostedbyConfluent

Asynchronous Transaction Processing With Kafka as a Single Source of Truth - ...HostedbyConfluent

Fundamentals of Apache KafkaChhavi Parasher

Kafka Cluster Federation at Uber (Yupeng Fui & Xiaoman Dong, Uber) Kafka Summ...confluent

Fundamentals and Architecture of Apache KafkaAngelo Cesaro

Scaling Security on 100s of Millions of Mobile Devices Using Apache Kafka® an...confluent

Tradeoffs in Distributed Systems Design: Is Kafka The Best? (Ben Stopford and...HostedbyConfluent

Reducing Microservice Complexity with Kafka and Reactive Streamsjimriecken

Introducing Confluent labs Parallel Consumer client | Anthony Stubbes, ConfluentHostedbyConfluent

Apache Kafka - Martin PodvalMartin Podval

KafkaConsumer - Decoupling Consumption and Processing for Better Resource Uti...confluent

Introduction to apache kafkaDimitris Kontokostas

Introducción a Stream Processing utilizando Kafka Streamsconfluent

Delivering: from Kafka to WebSockets | Adam Warski, SoftwareMillHostedbyConfluent

Building a Web Application with Kafka as your Databaseconfluent

URP? Excuse You! The Three Metrics You Have to Know confluent

Learnings from the Field. Lessons from Working with Dozens of Small & Large D...HostedbyConfluent

Kafka Summit SF 2017 - Kafka Connect Best Practices – Advice from the Fieldconfluent

Kafka awsAriel Moskovich

Can Kafka Handle a Lyft Ride? (Andrey Falko & Can Cecen, Lyft) Kafka Summit 2020HostedbyConfluent

Similar to Multi-DC Kafka (20)

Distributed Scrum: Dangerous Waters-Be Prepared!TechWell

Jax Devops 2017 Succeeding in the Cloud – the guidebook of FailSteve Poole

Resilience4j with Spring BootKnoldus Inc.

User Group Meeting PaperVision3DAlmog Koren

The document summarizes a presentation about Papervision3D, an open source 3D engine for Flash. It discusses downloading and setting up Papervision3D, designing interfaces for 3D applications, and provides examples of both good and bad 3D Flash applications. The presentation covers topics such as the basics of Papervision3D, getting started with 3D development in Flash, and designing effective user experiences for 3D Flash games and sites.

Apache Kafka's Common Pitfalls & Intricacies: A Customer Support PerspectiveHostedbyConfluent

"As Apache Kafka gains widespread adoption, an increasing number of people face its pitfalls. Despite completing courses and reading documentation, many encounter hurdles navigating Kafka's subtle complexities. Join us for an enlightening session led by the customer support team of Conduktor, where we engage daily with users grappling with Kafka's subtleties. We've observed recurring themes in user queries: What happens when a consumer group rebalances? What is an advertised listener? Why aren't my records displayed in chronological order when I consume them? How does retention work? For all these questions, the answer is ""It depends"". In this talk, we aim to demystify these uncertainties by presenting nuanced scenarios for each query. That way you will be more confident on how your Kafka infrastructure works behind the scenes, and you'll be equipped to share this knowledge with your colleagues. By being aware of the most common misconceptions, you should be able to both speed up your own learning curve and also help others more effectively."

Asynchronous programming using CompletableFutures in JavaOresztész Margaritisz

This document discusses asynchronous programming using CompletableFuture in Java. It begins by explaining what asynchronous programming is and why it is important to use non-blocking I/O. It then provides examples of how to use CompletableFuture to make asynchronous method calls and combine results. Some pros and cons of using CompletableFuture are discussed, along with design considerations for asynchronous programming.

Introducing Pair ProgrammingSteven Smith

Pair programming involves two programmers working together at one computer. One person acts as the driver who types code while the other navigates and reviews. It has benefits like catching mistakes earlier, improving design quality, transferring knowledge between partners, and creating a stronger sense of team. While there is initially a 15% overhead in time, studies show this is outweighed by fewer defects and a more flexible system in the long run. Effective pair programming requires collaboration, respect, communication and regularly alternating roles.

eduroam diagnostics in NTLR, IdPs and SPsKarri Huhtanen

This document summarizes common issues that cause failed eduroam authentications from the perspective of top level RADIUS servers, service providers, and identity providers. Some frequent problems include incorrectly configured user devices, older devices with outdated supplicant implementations, certificate checking issues, frequent password changes breaking configurations, and identity providers providing invalid configuration instructions. Diagnosing failures requires examining logs from each component to determine where communication breaks down. Standardizing configurations and improving device setup could help address many authentication failures.

Silverlight vs HTML5 - Lessons learned from the real world...Peter Gfader

Quality Assurance for WebRTC ServicesTsahi Levent-levi

Multithreading in Scala Knoldus Inc.

Webiner Presentationo96717393

Flink Forward SF 2017: Dean Wampler - Streaming Deep Learning Scenarios with...Flink Forward

As a low-latency streaming tool, Flink offers the possibility of using machine learning, even "deep learning" (neural networks), with low latency. The growing FlinkML library provides some of the infrastructure support required for this goal, combined with third-party tools. This talk is a progress report on several scenarios we are developing at Lightbend, which combine Flink, Deeplearning4J, Spark, and Kafka to analyze cluster telemetry for anomaly detection, predictive autoscaling, and other scenarios. I'll focus on the pragmatics of training deep learning models in a streaming context, using batch and mini-batch training, combined with low-latency application of those models. I'll discuss the architecture we're using and highlight trade offs of particular tools for certain design problems in the implementation. I'll discuss the drawbacks and workarounds of our design and finish with a look at how future developments in Flink could improve its support for scenarios like ours.

Speed up your Serverless development flowEfi Merdler-Kravitz

Developers arriving at the Serverless scene usually find it hard to use the previous methodologies and tools they are used to in this new arena. Why? Because you can’t run Serverless locally. Up until now, a developer could spin a container or install PostgreSQL or NGNIX on their machine and start playing. The cycle of code → build → test was reasonably fast, but now it changed, building and testing cannot run locally, they have to run in the cloud which causes developers to lose productivity. How do we change it? How do we keep the same productivity levels while enjoying the benefits of Serverless. The following session will cover: Reasons for a slow development cycle in Serverless environments Tips and tricks for speeding up the development flow

HTTP cache @ PUG Rome 03-29-2011Alessandro Nadalin

HTTP caching involves storing copies of resources near clients to serve future requests faster. Caching can happen locally on a client or through shared proxies. Effective caching requires expiration dates, validation of cached responses, and invalidation of cached responses when content changes. Caching allows servers to scale to many users by offloading work to clients and proxies. The HTTP protocol and technologies like ESI were designed to support caching while handling dynamic content.

3 Ways to Deliver an Elastic, Cost-Effective Cloud Architectureconfluent

Introduction to MicronautKnoldus Inc.

Introduction to Resilience4jKnoldus Inc.

ConFoo 2016: Development to DeploymentRachel Andrew

NLJUG speaker academy 2023 - session 1Bert Jan Schrijver

The document provides information and advice for speakers about presenting at conferences. It includes tips for selecting topics, writing proposals and bios, preparing presentations, and delivering talks. The document suggests rehearsing extensively, tailoring talks for the audience, and focusing on storytelling. Hands-on sections allow attendees to practice selecting topics, writing proposals and bios, and delivering short presentations for feedback. The next speaker academy session will be planned for September/October.

Distributed Scrum: Dangerous Waters-Be Prepared!TechWell

Jax Devops 2017 Succeeding in the Cloud – the guidebook of FailSteve Poole

Resilience4j with Spring BootKnoldus Inc.

User Group Meeting PaperVision3DAlmog Koren

Apache Kafka's Common Pitfalls & Intricacies: A Customer Support PerspectiveHostedbyConfluent

Asynchronous programming using CompletableFutures in JavaOresztész Margaritisz

Introducing Pair ProgrammingSteven Smith

eduroam diagnostics in NTLR, IdPs and SPsKarri Huhtanen

Silverlight vs HTML5 - Lessons learned from the real world...Peter Gfader

Quality Assurance for WebRTC ServicesTsahi Levent-levi

Multithreading in Scala Knoldus Inc.

Webiner Presentationo96717393

Flink Forward SF 2017: Dean Wampler - Streaming Deep Learning Scenarios with...Flink Forward

Speed up your Serverless development flowEfi Merdler-Kravitz

HTTP cache @ PUG Rome 03-29-2011Alessandro Nadalin

3 Ways to Deliver an Elastic, Cost-Effective Cloud Architectureconfluent

Introduction to MicronautKnoldus Inc.

Introduction to Resilience4jKnoldus Inc.

ConFoo 2016: Development to DeploymentRachel Andrew

NLJUG speaker academy 2023 - session 1Bert Jan Schrijver

More from confluent (20)

Webinar Think Right - Shift Left - 19-03-2025.pptxconfluent

Migration, backup and restore made easy using Kannikaconfluent

In this presentation, you’ll discover how easily you can migrate data from any Kafka-compatible event hub to Confluent using Kannika’s intuitive self-service interface. We’ll guide you through the process, showing how the same approach can be applied to define specific event data sets and effortlessly spin up secure environments for demos, testing, or other purposes. You’ll also learn how to back up event data in just a few steps by transferring compressed data to the cloud storage location of your choice. In addition, we’ll demonstrate how to restore filtered datasets of topics, ensuring quick recovery and maintaining business continuity when needed.

Five Things You Need to Know About Data Streaming in 2025confluent

Topics that Peter covers: Tapping into the Potential of Data Products: Data drives some of today's most important business use cases. Data products enable instant access to reliable and trustworthy data by eliminating the data mess created by point-to-point connections. The Need to Tap into 'Quick Thinking': The C-level has to reorient itself so it doesn't become the bottleneck to adaptability in a data-driven world. Nine in 10 (90%) business leaders say they must now react in real-time. Learn what you can do to provide executive access to real-time data to enable 'Quick Thinking.' Rise Above Data Hurdles: Discover how to enforce governance at data production. Reestablishing trustworthiness later is almost always harder, so investing in data tools that solve business problems rather than add to them is essential. Paradigm to Shift Left: Shift Left is a new paradigm for processing and governing data at any scale, complexity, and latency. Shift Left moves the processing and governance of data closer to the source, enabling organisations to build their data once, build it right and reuse it anywhere within moments of its creation. The Need for a Strategic View: The positive correlation between data streaming maturity and significant business returns underscores the importance of a long-term, strategic view of data streaming investments. It also highlights the value of advancing beyond initial, siloed use cases to a more integrated approach that leverages data streaming across the enterprise.

Data in Motion Tour Seoul 2024 - Keynoteconfluent

Data in Motion Tour Seoul 2024 - Roadmap Democonfluent

From Stream to Screen: Real-Time Data Streaming to Web Frontends with Conflue...confluent

In this presentation, we’ll demonstrate how Confluent and Lightstreamer come together to tackle the last-mile challenge of extending your Kafka architecture to web and mobile platforms. Learn how to effortlessly build real-time web applications within minutes, subscribing to Kafka topics directly from your web pages, with unmatched low latency and high scalability. Explore how Confluent's leading Kafka platform and Lightstreamer's intelligent proxy work seamlessly to bridge Kafka with the internet frontier, delivering data in real-time.

Confluent per il settore FSI: Accelerare l'Innovazione con il Data Streaming...confluent

Data in Motion Tour 2024 Riyadh, Saudi Arabiaconfluent

Data streaming platforms are becoming increasingly important in today’s fast-paced world. From retail giants who need to monitor inventory levels to ensure stores never run out of items, to new-age, innovative banks who are building out-of-the-box banking solutions for traditional retail banks, data streaming platforms are at the centre, powering these workflows. Data streaming platforms connect all your applications, systems, and teams with a shared view of the most up-to-date, real-time data. From Gen AI, stream governance to stream processing - it’s these cutting edge developments that will be featured during the day.

Build a Real-Time Decision Support Application for Financial Market Traders w...confluent

Strumenti e Strategie di Stream Governance con Confluent Platformconfluent

Compose Gen-AI Apps With Real-Time Data - In Minutes, Not Weeksconfluent

As businesses strive to stay at the forefront of innovation, the ability to quickly develop scalable Generative AI (GenAI) applications is essential. Join us for an exclusive webinar featuring MIA Platform, MongoDB, and Confluent, where you'll learn how to compose GenAI apps with real-time data integration in a fraction of the time. Discover how these three powerful platforms work together to ensure applications remain responsive, relevant, and adaptive to user preferences and contextual changes. Our experts will guide you through leveraging MIA Platform's microservices architecture and low-code development, MongoDB's flexibility, and Confluent's stream processing capabilities. Experience live demonstrations and practical insights that will transform your approach to AI-driven app development, enabling you to accelerate your development process from weeks to mere minutes. Don't miss this opportunity to keep your business at the cutting edge.

Building Real-Time Gen AI Applications with SingleStore and Confluentconfluent

Unlocking value with event-driven architecture by Confluentconfluent

Sfrutta il potere dello streaming di dati in tempo reale e dei microservizi basati su eventi per il futuro di Sky con Confluent e Kafka®. In questo tech talk esploreremo le potenzialità di Confluent e Apache Kafka® per rivoluzionare l'architettura aziendale e sbloccare nuove opportunità di business. Ne approfondiremo i concetti chiave, guidandoti nella creazione di applicazioni scalabili, resilienti e fruibili in tempo reale per lo streaming di dati. Scoprirai come costruire microservizi basati su eventi con Confluent, sfruttando i vantaggi di un'architettura moderna e reattiva. Il talk presenterà inoltre casi d'uso reali di Confluent e Kafka®, dimostrando come queste tecnologie possano ottimizzare i processi aziendali e generare valore concreto.

Il Data Streaming per un’AI real-time di nuova generazioneconfluent

Per costruire applicazioni di AI affidabili, sicure e governate occorre una base dati in tempo reale altrettanto solida. Ancor più quando ci troviamo a gestire ingenti flussi di dati in continuo movimento. Come arrivarci? Affidati a una vera piattaforma di data streaming che ti permetta di scalare e creare rapidamente applicazioni di AI in tempo reale partendo da dati affidabili. Scopri di più! Non perdere il nostro prossimo webinar durante il quale avremo l’occasione di: • Esplorare il paradigma della GenAI e di come questa nuova tecnnologia sta rimodellando il panorama aziendale, rispondendo alla necessità di offrire un contesto e soluzioni in tempo reale che soddisfino le esigenze della tua azienda. • Approfondire le incertezze del panorama dell'AI in evoluzione e l'importanza cruciale del data streaming e dell'elaborazione dati. • Vedere in dettaglio l'architettura in continua evoluzione e il ruolo chiave di Kafka e Confluent nelle applicazioni di AI. • Analizzare i vantaggi di una piattaforma di streaming dei dati come Confluent nel collegare l'eredità legacy e la GenAI, facilitando lo sviluppo e l’utilizzo di AI predittive e generative.

Unleashing the Future: Building a Scalable and Up-to-Date GenAI Chatbot with ...confluent

As businesses strive to remain at the cutting edge of innovation, the demand for scalable and up-to-date conversational AI solutions has become paramount. Generative AI (GenAI) chatbots that seamlessly integrate into our daily lives and adapt to the ever-evolving nuances of human interaction are crucial. Real-time data plays a pivotal role in ensuring the responsiveness and relevance of these chatbots, empowering them to stay abreast of the latest trends, user preferences, and contextual information.

Break data silos with real-time connectivity using Confluent Cloud Connectorsconfluent

Building API data products on top of your real-time data infrastructureconfluent

This talk and live demonstration will examine how Confluent and Gravitee.io integrate to unlock value from streaming data through API products. You will learn how data owners and API providers can document, secure data products on top of Confluent brokers, including schema validation, topic routing and message filtering. You will also see how data and API consumers can discover and subscribe to products in a developer portal, as well as how they can integrate with Confluent topics through protocols like REST, Websockets, Server-sent Events and Webhooks. Whether you want to monetize your real-time data, enable new integrations with partners, or provide self-service access to topics through various protocols, this webinar is for you!

Speed Wins: From Kafka to APIs in Minutesconfluent

Evolving Data Governance for the Real-time Streaming and AI Eraconfluent

Catch the Wave: SAP Event-Driven and Data Streaming for the Intelligence Ente...confluent