Apache Kafka - Martin Podval

Apr 14, 20153 likes3,813 views

Apache Kafka is a distributed messaging system that allows for publishing and subscribing to streams of records, known as topics, in a fault-tolerant and scalable way. It is used for building real-time data pipelines and streaming apps. Producers write data to topics which are committed to disks across partitions and replicated for fault tolerance. Consumers read data from topics in a decoupled manner based on offsets. Kafka can process streaming data in real-time and at large volumes with low latency and high throughput.

What is Apache Kafka?
Messaging System
Distributed
Persistent and Replicable
Very fast - low latency - and scalable
Simple but highly configurable
By Linkedin, open sourced under apache.org

Data Streaming
New kind of data ...
● User or application data (events) streams
● Monitoring - App, System
● App Logging
● High volume

Data Streaming Cont’d
… you want to process
● Using various components
● Into a target form
● Map, reduce, shuffle
● Real time or batch

HP Service Virtualization Use Cases
Process of clients
message streams
Real-time performance
modeling
Logs aggregation

How To Solve It?
Producers and
Consumers
● Distributed
● Decoupled
● Configurable
● Dynamic

Kafka Cluster
Brokers
● = Instances, Nodes
● Topics
● Partitions
● Replicas
ZK
● Coordination

Kafka Topics
Commit Log
● Immutable
● Ordered
● Sequential Offset

Kafka Topics Cont’d
Partitioned
Independently:
● Stored
● Produced
● Consumed
⇒ Scalable
Replicated
● On partition basis
● Different brokers
⇒ Fault Tolerant

What Can I Do?
producer.
write(topic_id, message);
consumer.
read(topic_id, offset);

I Want To Produce
● java/scala client
● address of one or more brokers
● choose a topic where to produce
● highly configurable and tunable:
○ partitioner
○ number of acks (async=0, master=1, replicas=1+?)
○ batching, buffer size, timeouts, retries, ...

I Want To Consume
High Level API
● Groups abstraction
○ To All, To One
○ To Some
● Stream API
● Stores positions to support fault tolerance

I Want To Consume Cont’d
Low Level
● Java/scala client
● Find a leader for a topic
● Calculate an offset
● Fetches messages
○ Re-consume if needed

I Want To Consume Cont’d
Delivery Semantic:
● At most once
● At least once
● Exactly once

Kafka Internals - Disks
Avoid:
● GC
● Random disk
access

Kafka Internals - Disks Cont’d
Disks are fast ...
… when properly used
● sequential access - read ahead, write behind
● rely on operating system
○ avoid heap, materialization and GC
● it’s more like file copy over network
It’s easy … with immutable topics

Kafka Internals - Replication
“In Sync” Replicas
● Replication factor on partition basis
● One leader + 0..n replicas
● Replicas are consumers
○ “In Sync” if they are not “too far” behind a leader
○ Batch sync

Kafka Internals - Replication Cont’d
Tunable Trade-Offs
● Producer’s write method:
○ Not blocked, async
○ Waits for master ACK
○ Waits for all in-sync replicas
● Consumer pulls only committed messages
● Server’s minimum in-sync replicas

Performance
“Incredible”
Scales with:
● clients count, message size
● number of replicas, partitions or topics
Depends on network and disk throughput

Performance Cont’d
Our testing
● 3 nodes, master + 2 replicas
● 500 000 msg/s (100 bytes[])
● 400 mbit/s - 1.2 gbit/s network throughput
● end2end latency 2-3 ms
@see http://bit.ly/1FsIR9a

Easy of Use
● No installation, just run a
java/scala program
● Streams in files & dirs
● Transparent zookeeper
● Ecosystem

Cons
● Beta version
● Dependency on Zookeeper
● The way how it is written in Scala
● No easy way how to remove messages

Watch this talk here: https://www.confluent.io/online-talks/apache-kafka-architecture-and-fundamentals-explained-on-demand This session explains Apache Kafka’s internal design and architecture. Companies like LinkedIn are now sending more than 1 trillion messages per day to Apache Kafka. Learn about the underlying design in Kafka that leads to such high throughput. This talk provides a comprehensive overview of Kafka architecture and internal functions, including: -Topics, partitions and segments -The commit log and streams -Brokers and broker replication -Producer basics -Consumers, consumer groups and offsets This session is part 2 of 4 in our Fundamentals for Apache Kafka series.

Fundamentals of Apache KafkaChhavi Parasher

Kafka 101Clement Demonchy

Kafka Tutorial - Introduction to Apache Kafka (Part 1)Jean-Paul Azar

Why is Kafka so fast? Why is Kafka so popular? Why Kafka? This slide deck is a tutorial for the Kafka streaming platform. This slide deck covers Kafka Architecture with some small examples from the command line. Then we expand on this with a multi-server example to demonstrate failover of brokers as well as consumers. Then it goes through some simple Java client examples for a Kafka Producer and a Kafka Consumer. We have also expanded on the Kafka design section and added references. The tutorial covers Avro and the Schema Registry as well as advance Kafka Producers.

APACHE KAFKA / Kafka Connect / Kafka StreamsKetan Gote

A visual introduction to Apache KafkaPaul Brebner

KONG-APIGateway.pptxAgusto Sipahutar

Apache kafkaNexThoughts Technologies

Apache Kafka is a distributed publish-subscribe messaging system that can handle high volumes of data and enable messages to be passed from one endpoint to another. It uses a distributed commit log that allows messages to be persisted on disk for durability. Kafka is fast, scalable, fault-tolerant, and guarantees zero data loss. It is used by companies like LinkedIn, Twitter, and Netflix to handle high volumes of real-time data and streaming workloads.

kafkaAmikam Snir

Kafka is an open-source distributed commit log service that provides high-throughput messaging functionality. It is designed to handle large volumes of data and different use cases like online and offline processing more efficiently than alternatives like RabbitMQ. Kafka works by partitioning topics into segments spread across clusters of machines, and replicates across these partitions for fault tolerance. It can be used as a central data hub or pipeline for collecting, transforming, and streaming data between systems and applications.

Apache Kafka IntroductionAmita Mirajkar

Kafka presentationMohammed Fazuluddin

Kafka is an open source messaging system that can handle massive streams of data in real-time. It is fast, scalable, durable, and fault-tolerant. Kafka is commonly used for stream processing, website activity tracking, metrics collection, and log aggregation. It supports high throughput, reliable delivery, and horizontal scalability. Some examples of real-time use cases for Kafka include website monitoring, network monitoring, fraud detection, and IoT applications.

Apache kafkaKumar Shivam

This document provides an introduction to Apache Kafka, an open-source distributed event streaming platform. It discusses Kafka's history as a project originally developed by LinkedIn, its use cases like messaging, activity tracking and stream processing. It describes key Kafka concepts like topics, partitions, offsets, replicas, brokers and producers/consumers. It also gives examples of how companies like Netflix, Uber and LinkedIn use Kafka in their applications and provides a comparison to Apache Spark.

Apache KafkaDiego Pacheco

Apache Kafka is a high-throughput distributed messaging system that allows for both streaming and offline log processing. It uses Apache Zookeeper for coordination and supports activity stream processing and real-time pub/sub messaging. Kafka bridges the gaps between pure offline log processing and traditional messaging systems by providing features like batching, transactions, persistence, and support for multiple consumers.

Introduction to Apache KafkaJeff Holoman

The document provides an introduction and overview of Apache Kafka presented by Jeff Holoman. It begins with an agenda and background on the presenter. It then covers basic Kafka concepts like topics, partitions, producers, consumers and consumer groups. It discusses efficiency and delivery guarantees. Finally, it presents some use cases for Kafka and positioning around when it may or may not be a good fit compared to other technologies.

Kafka 101Aparna Pillai

Stream processing using KafkaKnoldus Inc.

Apache kafkaViswanath J

Kafka is a distributed publish-subscribe messaging system that allows both streaming and storage of data feeds. It is designed to be fast, scalable, durable, and fault-tolerant. Kafka maintains feeds of messages called topics that can be published to by producers and subscribed to by consumers. A Kafka cluster typically runs on multiple servers called brokers that store topics which may be partitioned and replicated for fault tolerance. Producers publish messages to topics which are distributed to consumers through consumer groups that balance load.

Apache Kafkaemreakis

Apache Kafka is a distributed streaming platform used for building real-time data pipelines and streaming apps. It provides a unified, scalable, and durable platform for handling real-time data feeds. Kafka works by accepting streams of records from one or more producers and organizing them into topics. It allows both storing and forwarding of these streams to consumers. Producers write data to topics which are replicated across clusters for fault tolerance. Consumers can then read the data from the topics in the order it was produced. Major companies like LinkedIn, Yahoo, Twitter, and Netflix use Kafka for applications like metrics, logging, stream processing and more.

Apache Kafka Fundamentals for Architects, Admins and Developersconfluent

This document summarizes a presentation about Apache Kafka. It introduces Apache Kafka as a modern, distributed platform for data streams made up of distributed, immutable, append-only commit logs. It describes Kafka's scalability similar to a filesystem and guarantees similar to a database, with the ability to rewind and replay data. The document discusses Kafka topics and partitions, partition leadership and replication, and provides resources for further information.

Performance Tuning RocksDB for Kafka Streams' State Stores (Dhruba Borthakur,...confluent

RocksDB is the default state store for Kafka Streams. In this talk, we will discuss how to improve single node performance of the state store by tuning RocksDB and how to efficiently identify issues in the setup. We start with a short description of the RocksDB architecture. We discuss how Kafka Streams restores the state stores from Kafka by leveraging RocksDB features for bulk loading of data. We give examples of hand-tuning the RocksDB state stores based on Kafka Streams metrics and RocksDB’s metrics. At the end, we dive into a few RocksDB command line utilities that allow you to debug your setup and dump data from a state store. We illustrate the usage of the utilities with a few real-life use cases. The key takeaway from the session is the ability to understand the internal details of the default state store in Kafka Streams so that engineers can fine-tune their performance for different varieties of workloads and operate the state stores in a more robust manner.

Kafka basicsJoão Paulo Leonidas Fernandes Dias da Silva

Kafka is an open-source message broker that provides high-throughput and low-latency data processing. It uses a distributed commit log to store messages in categories called topics. Processes that publish messages are producers, while processes that subscribe to topics are consumers. Consumers can belong to consumer groups for parallel processing. Kafka guarantees order and no lost messages. It uses Zookeeper for metadata and coordination.

Delta lake and the delta architectureAdam Doyle

- Delta Lake is an open source project that provides ACID transactions, schema enforcement, and time travel capabilities to data stored in data lakes such as S3 and ADLS. - It allows building a "Lakehouse" architecture where the same data can be used for both batch and streaming analytics. - Key features include ACID transactions, scalable metadata handling, time travel to view past data states, schema enforcement, schema evolution, and change data capture for streaming inserts, updates and deletes.

Apache KafkaSaroj Panyasrivanit

This document provides an overview of Apache Kafka. It begins with defining Kafka as a distributed streaming platform and messaging system. It then lists the agenda which includes what Kafka is, why it is used, common use cases, major companies that use it, how it achieves high performance, and core concepts. Core concepts explained include topics, partitions, brokers, replication, leaders, and producers and consumers. The document also provides examples to illustrate these concepts.

CDC patterns in Apache Kafka®confluent

This document discusses using Apache Kafka as a data hub to capture changes from various data sources using change data capture (CDC). It outlines several common CDC patterns like using modification dates, database triggers, or log files to identify changes. It then discusses using Kafka Connect to integrate various data sources like MongoDB, PostgreSQL and replicate changes. The document provides examples of open source CDC connectors and concludes with suggestions for getting involved in the Apache Kafka community.

Introduction to Apache KafkaShiao-An Yuan

This document provides an introduction to Apache Kafka. It describes Kafka as a distributed messaging system with features like durability, scalability, publish-subscribe capabilities, and ordering. It discusses key Kafka concepts like producers, consumers, topics, partitions and brokers. It also summarizes use cases for Kafka and how to implement producers and consumers in code. Finally, it briefly outlines related tools like Kafka Connect and Kafka Streams that build upon the Kafka platform.

Kafka Streams State Stores Being Persistentconfluent

ksqlDB: A Stream-Relational Database Systemconfluent

Speaker: Matthias J. Sax, Software Engineer, Confluent ksqlDB is a distributed event streaming database system that allows users to express SQL queries over relational tables and event streams. The project was released by Confluent in 2017 and is hosted on Github and developed with an open-source spirit. ksqlDB is built on top of Apache Kafka®, a distributed event streaming platform. In this talk, we discuss ksqlDB’s architecture that is influenced by Apache Kafka and its stream processing library, Kafka Streams. We explain how ksqlDB executes continuous queries while achieving fault tolerance and high vailability. Furthermore, we explore ksqlDB’s streaming SQL dialect and the different types of supported queries. Matthias J. Sax is a software engineer at Confluent working on ksqlDB. He mainly contributes to Kafka Streams, Apache Kafka's stream processing library, which serves as ksqlDB's execution engine. Furthermore, he helps evolve ksqlDB's "streaming SQL" language. In the past, Matthias also contributed to Apache Flink and Apache Storm and he is an Apache committer and PMC member. Matthias holds a Ph.D. from Humboldt University of Berlin, where he studied distributed data stream processing systems. https://db.cs.cmu.edu/events/quarantine-db-talk-2020-confluent-ksqldb-a-stream-relational-database-system/

Introduction to KSQL: Streaming SQL for Apache Kafka®confluent

Join Tom Green, Solution Engineer at Confluent for this Lunch and Learn talk covering KSQL. Confluent KSQL is the streaming SQL engine that enables real-time data processing against Apache Kafka®. It provides an easy-to-use, yet powerful interactive SQL interface for stream processing on Kafka, without the need to write code in a programming language such as Java or Python. KSQL is scalable, elastic, fault-tolerant, and it supports a wide range of streaming operations, including data filtering, transformations, aggregations, joins, windowing, and sessionization. By attending one of these sessions, you will learn: -How to query streams, using SQL, without writing code. -How KSQL provides automated scalability and out-of-the-box high availability for streaming queries -How KSQL can be used to join streams of data from different sources -The differences between Streams and Tables in Apache Kafka

Build real time stream processing applications using Apache KafkaHotstar

This talk was presented at the Hotstar Scale Meetup in Bangalore by Jayesh Sidhwani In this talk, the presenter introduces Apache Kafka and the Apache Kafka Streams library. Starting from the need for building streaming applications to thinking the use-cases as a streaming job - this talk covers all the technicalities. It ends with a short description of how Kafka is deployed and used at Hotstar

Building zero data loss pipelines with apache kafkaAvinash Ramineni

Kafka is playing an increasingly important role in messaging and streaming systems and is becoming the defacto messaging platform in many enterprises. Managing and maintaining Kafka deployments and tuning the data pipelines for high-performance and scalability can become a challenging task. In this session, we will discuss the lessons learned and the best practices for achieving zero data loss pipelines.

More Related Content

What's hot (20)

kafkaAmikam Snir

Apache Kafka IntroductionAmita Mirajkar

Kafka presentationMohammed Fazuluddin

Apache kafkaKumar Shivam

Apache KafkaDiego Pacheco

Introduction to Apache KafkaJeff Holoman

Kafka 101Aparna Pillai

Stream processing using KafkaKnoldus Inc.

Apache kafkaViswanath J

Apache Kafkaemreakis

Apache Kafka Fundamentals for Architects, Admins and Developersconfluent

Performance Tuning RocksDB for Kafka Streams' State Stores (Dhruba Borthakur,...confluent

Kafka basicsJoão Paulo Leonidas Fernandes Dias da Silva

Delta lake and the delta architectureAdam Doyle

Apache KafkaSaroj Panyasrivanit

CDC patterns in Apache Kafka®confluent

Introduction to Apache KafkaShiao-An Yuan

Kafka Streams State Stores Being Persistentconfluent

ksqlDB: A Stream-Relational Database Systemconfluent

Introduction to KSQL: Streaming SQL for Apache Kafka®confluent

kafkaAmikam Snir

Apache Kafka IntroductionAmita Mirajkar

Kafka presentationMohammed Fazuluddin

Apache kafkaKumar Shivam

Apache KafkaDiego Pacheco

Introduction to Apache KafkaJeff Holoman

Kafka 101Aparna Pillai

Stream processing using KafkaKnoldus Inc.

Apache kafkaViswanath J

Apache Kafkaemreakis

Apache Kafka Fundamentals for Architects, Admins and Developersconfluent

Performance Tuning RocksDB for Kafka Streams' State Stores (Dhruba Borthakur,...confluent

Kafka basicsJoão Paulo Leonidas Fernandes Dias da Silva

Delta lake and the delta architectureAdam Doyle

Apache KafkaSaroj Panyasrivanit

CDC patterns in Apache Kafka®confluent

Introduction to Apache KafkaShiao-An Yuan

Kafka Streams State Stores Being Persistentconfluent

ksqlDB: A Stream-Relational Database Systemconfluent

Introduction to KSQL: Streaming SQL for Apache Kafka®confluent

Similar to Apache Kafka - Martin Podval (20)

Build real time stream processing applications using Apache KafkaHotstar

Building zero data loss pipelines with apache kafkaAvinash Ramineni

Netflix Keystone Pipeline at Big Data Bootcamp, Santa Clara, Nov 2015Monal Daxini

Keystone - Processing over Half a Trillion events per day with 8 million events & 17 GB per second peaks, and at-least once processing semantics. We will explore in detail how we employ Kafka, Samza, and Docker at scale to implement a multi-tenant pipeline. We will also look at the evolution to its current state and where the pipeline is headed next in offering a self-service stream processing infrastructure atop the Kafka based pipeline and support Spark Streaming.

Insta clustr seattle kafka meetup presentation bbNitin Kumar

This document summarizes Instaclustr's lessons learned from building a managed Apache Kafka service. It provides an overview of Kafka and how it works, details Instaclustr's offering and development process, and discusses choices around hardware, security configuration, monitoring, backups and restores. Key topics covered include benchmarking storage types, enabling SSL, managing topics and users, and exposing metrics for monitoring brokers and topics.

Stateful stream processing with kafka and samzaGeorge Li

Structured Streaming with Kafkadatamantra

This document provides an overview of structured streaming with Kafka in Spark. It discusses data collection vs ingestion and why they are key. It also covers Kafka architecture and terminology. It describes how Spark integrates with Kafka for streaming data sources. It explains checkpointing in structured streaming and using Kafka as a sink. The document discusses delivery semantics and how Spark supports exactly-once semantics with certain output stores. Finally, it outlines new features in Kafka for exactly-once guarantees and the future of structured streaming.

kafkaAriel Moskovich

This document discusses the evolution of Kafka clusters at AppsFlyer over time. The initial cluster had 4 brokers and handled hundreds of millions of messages with low partitioning and replication. A new cluster was designed with more brokers, replication across availability zones, and higher partitioning to support billions of messages. However, this led to issues like uneven leader distribution and failures. Various solutions were implemented like increasing brokers, splitting topics, and hardware upgrades. Ongoing testing and monitoring helped identify more problems and improvements around replication, partitioning, and automation. Key lessons learned included balancing replication and leaders, supporting dynamic changes, and thorough testing of failure scenarios.

14th Athens Big Data Meetup - Landoop Workshop - Apache Kafka Entering The St...Athens Big Data

Multi-Tenancy Kafka cluster for LINE services with 250 billion daily messagesLINE Corporation

Yuto Kawamura LINE / Z Part Team At LINE we've been operating Apache Kafka to provide the company-wide shared data pipeline for services using it for storing and distributing data. Kafka is underlying many of our services in some way, not only the messaging service but also AD, Blockchain, Pay, Timeline, Cryptocurrency trading and more. Many services feeding many data into our cluster, leading over 250 billion daily messages and 3.5GB incoming bytes in 1 second which is one of the world largest scale. At the same time, it is required to be stable and performant all the time because many important services uses it as a backend. In this talk I will introduce the overview of Kafka usage at LINE and how we're operating it. I'm also going to talk about some engineerings we did for maximizing its performance, solving troubles led particularly by hosting huge data from many services, leveraging advanced techniques like kernel-level dynamic tracing.

Event driven architectures with KinesisMark Harrison

Mark Harrison presented on using Amazon Kinesis for event-driven microservices architectures. He discussed the limitations of traditional monolithic and microservice architectures, and how Kinesis can help address issues like tight coupling, high latency, and lack of event broadcasting. Key concepts around Kinesis included streams, shards, producers, consumers, and constraints like throughput limits. Harrison demonstrated a custom Scala/Akka client for Kinesis that provides asynchronous, non-blocking producers and consumers with features like throttling and checkpointing. Performance tests showed throughput scales linearly with additional shards. In closing, Harrison invited the audience to learn more about open source and job opportunities with Weight Watchers.

An Introduction to Apache KafkaAmir Sedighi

Netflix Open Source Meetup Season 4 Episode 2aspyker

In this episode, we will take a close look at 2 different approaches to high-throughput/low-latency data stores, developed by Netflix. The first, EVCache, is a battle-tested distributed memcached-backed data store, optimized for the cloud. You will also hear about the road ahead for EVCache it evolves into an L1/L2 cache over RAM and SSDs. The second, Dynomite, is a framework to make any non-distributed data-store, distributed. Netflix's first implementation of Dynomite is based on Redis. Come learn about the products' features and hear from Thomson and Reuters, Diego Pacheco from Ilegra and other third party speakers, internal and external to Netflix, on how these products fit in their stack and roadmap.

Tips & Tricks for Apache Kafka®confluent

Kat Grigg, Confluent, Senior Customer Success Architect + Jen Snipes, Confluent, Senior Customer Success Architect This presentation will cover tips and best practices for Apache Kafka. In this talk, we will be covering the basic internals of Kafka and how these components integrate together including brokers, topics, partitions, consumers and producers, replication, and Zookeeper. We will be talking about the major categories of operations you need to be setting up and monitoring including configuration, deployment, maintenance, monitoring and then debugging. https://www.meetup.com/KafkaBayArea/events/270915296/

Netflix Data Pipeline With KafkaAllen (Xiaozhong) Wang

This document summarizes Netflix's use of Kafka in their data pipeline. It discusses how Netflix evolved from using S3 and EMR to introducing Kafka and Kafka producers and consumers to handle 400 billion events per day. It covers challenges of scaling Kafka clusters and tuning Kafka clients and brokers. Finally, it outlines Netflix's roadmap which includes contributing to open source projects like Kafka and testing failure resilience.

Netflix Data Pipeline With KafkaSteven Wu

This document summarizes Netflix's use of Kafka in their data pipeline. It discusses the evolution of Netflix's data pipeline to incorporate Kafka to handle 400 billion events per day. It describes how Netflix uses Kafka clusters with different priorities and configurations. It also outlines some of the challenges of using Kafka at Netflix's scale, such as Zookeeper client issues and cluster scaling, and the solutions Netflix developed to address these challenges.

Twitter’s Apache Kafka Adoption Journey | Ming Liu, TwitterHostedbyConfluent

Until recently, the Messaging team at Twitter had been running an in-house build Pub/Sub system, namely EventBus (built on top of Apache DistributedLog and Apache Bookkeeper, and similar in architecture to Apache Pulsar) to cater to our pubsub needs. In 2018, we made the decision to move to Apache Kafka by migrating existing use cases as well as onboarding new use cases directly onto Apache Kafka. Fast forward to today, Kafka is now an essential piece of Twitter Infrastructure and processes over 200M messages per second. In this talk, we will share the learning and challenges in our journey moving to Apache Kafka.

Building realtime data pipeline with Apache KafkaNagarajan Selvaraj

Apache Kafka is a distributed streaming platform that can be used to build real-time data pipelines. It publishes and subscribes to streams of records in a fault-tolerant and durable way, and helps process streams of records as they occur. Key characteristics of Kafka include high throughput ingestion, fault-tolerant storage, high availability, scalability, and support for concurrent processing and ordering guarantees. It provides functionality similar to a messaging system but with a unique design as a distributed, partitioned, replicated commit log service.

Activity feeds (and more) at mate1Hisham Mardam-Bey

The document provides an overview of Mate1, an online dating company, and their transition to using Kafka and Cassandra for their activity feed and data storage. Some key points: - Mate1 started in 2003 and has grown to around 40 employees, with a 13 person engineering team. They own their own hardware. - They previously used a "fan-out on read" approach where activity data was stored across different databases and querying them to generate feeds was slow. - They transitioned to a "fan-out on write" approach using Kafka to queue activity events and Cassandra to store aggregated user feeds and counters. This improved performance and scalability. - Events are published to Kafka topics and

Uber: Kafka Consumer Proxyconfluent

Haitao Zhang, Uber, Software Engineer + Yang Yang, Uber, Senior Software Engineer Kafka Consumer Proxy is a forwarding proxy that consumes messages from Kafka and dispatches them to a user registered gRPC service endpoint. With Kafka Consumer Proxy, the experience of consuming messages from Apache Kafka for pub-sub use cases is as seamless and user-friendly as receiving (g)RPC requests. In this talk, we will share (1) the motivation for building this service, (2) the high-level architecture, (3) the mechanisms we designed to achieve high availability, scalability, and reliability, and (4) the current adoption status. https://www.meetup.com/KafkaBayArea/events/273834934/

Introduction to apache kafkaSamuel Kerrien

Build real time stream processing applications using Apache KafkaHotstar

Building zero data loss pipelines with apache kafkaAvinash Ramineni

Netflix Keystone Pipeline at Big Data Bootcamp, Santa Clara, Nov 2015Monal Daxini

Insta clustr seattle kafka meetup presentation bbNitin Kumar

Stateful stream processing with kafka and samzaGeorge Li

Structured Streaming with Kafkadatamantra

kafkaAriel Moskovich

14th Athens Big Data Meetup - Landoop Workshop - Apache Kafka Entering The St...Athens Big Data

Multi-Tenancy Kafka cluster for LINE services with 250 billion daily messagesLINE Corporation

Event driven architectures with KinesisMark Harrison

An Introduction to Apache KafkaAmir Sedighi

Netflix Open Source Meetup Season 4 Episode 2aspyker

Tips & Tricks for Apache Kafka®confluent

Netflix Data Pipeline With KafkaAllen (Xiaozhong) Wang

Netflix Data Pipeline With KafkaSteven Wu

Twitter’s Apache Kafka Adoption Journey | Ming Liu, TwitterHostedbyConfluent

Building realtime data pipeline with Apache KafkaNagarajan Selvaraj

Activity feeds (and more) at mate1Hisham Mardam-Bey

Uber: Kafka Consumer Proxyconfluent

Introduction to apache kafkaSamuel Kerrien

Recently uploaded (20)

Expand your AI adoption with AgentExchangeFexle Services Pvt. Ltd.

AgentExchange is Salesforce’s latest innovation, expanding upon the foundation of AppExchange by offering a centralized marketplace for AI-powered digital labor. Designed for Agentblazers, developers, and Salesforce admins, this platform enables the rapid development and deployment of AI agents across industries. Email: [email protected] Phone: +1(630) 349 2411 Website: https://www.fexle.com/blogs/agentexchange-an-ultimate-guide-for-salesforce-consultants-businesses/?utm_source=slideshare&utm_medium=pptNg

Societal challenges of AI: biases, multilinguism and sustainabilityJordi Cabot

LEARN SEO AND INCREASE YOUR KNOWLDGE IN SOFTWARE INDUSTRYNidaFarooq10

How Valletta helped healthcare SaaS to transform QA and compliance to grow wi...Egor Kaleynik

This case study explores how we partnered with a mid-sized U.S. healthcare SaaS provider to help them scale from a successful pilot phase to supporting over 10,000 users—while meeting strict HIPAA compliance requirements. Faced with slow, manual testing cycles, frequent regression bugs, and looming audit risks, their growth was at risk. Their existing QA processes couldn’t keep up with the complexity of real-time biometric data handling, and earlier automation attempts had failed due to unreliable tools and fragmented workflows. We stepped in to deliver a full QA and DevOps transformation. Our team replaced their fragile legacy tests with Testim’s self-healing automation, integrated Postman and OWASP ZAP into Jenkins pipelines for continuous API and security validation, and leveraged AWS Device Farm for real-device, region-specific compliance testing. Custom deployment scripts gave them control over rollouts without relying on heavy CI/CD infrastructure. The result? Test cycle times were reduced from 3 days to just 8 hours, regression bugs dropped by 40%, and they passed their first HIPAA audit without issue—unlocking faster contract signings and enabling them to expand confidently. More than just a technical upgrade, this project embedded compliance into every phase of development, proving that SaaS providers in regulated industries can scale fast and stay secure.

How to Optimize Your AWS Environment for Improved Cloud PerformanceThousandEyes

Pixologic ZBrush Crack Plus Activation Key [Latest 2025] New Versionsaimabibi60507

Copy & Past Link👉👉 https://dr-up-community.info/ Pixologic ZBrush, now developed by Maxon, is a premier digital sculpting and painting software renowned for its ability to create highly detailed 3D models. Utilizing a unique "pixol" technology, ZBrush stores depth, lighting, and material information for each point on the screen, allowing artists to sculpt and paint with remarkable precision .

EASEUS Partition Master Crack + License Codeaneelaramzan63

Maxon CINEMA 4D 2025 Crack FREE Download LINKyounisnoman75

⭕️➡️ FOR DOWNLOAD LINK : http://drfiles.net/ ⬅️⭕️ Maxon Cinema 4D 2025 is the latest version of the Maxon's 3D software, released in September 2024, and it builds upon previous versions with new tools for procedural modeling and animation, as well as enhancements to particle, Pyro, and rigid body simulations. CG Channel also mentions that Cinema 4D 2025.2, released in April 2025, focuses on spline tools and unified simulation enhancements. Key improvements and features of Cinema 4D 2025 include: Procedural Modeling: New tools and workflows for creating models procedurally, including fabric weave and constellation generators. Procedural Animation: Field Driver tag for procedural animation. Simulation Enhancements: Improved particle, Pyro, and rigid body simulations. Spline Tools: Enhanced spline tools for motion graphics and animation, including spline modifiers from Rocket Lasso now included for all subscribers. Unified Simulation & Particles: Refined physics-based effects and improved particle systems. Boolean System: Modernized boolean system for precise 3D modeling. Particle Node Modifier: New particle node modifier for creating particle scenes. Learning Panel: Intuitive learning panel for new users. Redshift Integration: Maxon now includes access to the full power of Redshift rendering for all new subscriptions. In essence, Cinema 4D 2025 is a major update that provides artists with more powerful tools and workflows for creating 3D content, particularly in the fields of motion graphics, VFX, and visualization.

Proactive Vulnerability Detection in Source Code Using Graph Neural Networks:...Ranjan Baisak

As software complexity grows, traditional static analysis tools struggle to detect vulnerabilities with both precision and context—often triggering high false positive rates and developer fatigue. This article explores how Graph Neural Networks (GNNs), when applied to source code representations like Abstract Syntax Trees (ASTs), Control Flow Graphs (CFGs), and Data Flow Graphs (DFGs), can revolutionize vulnerability detection. We break down how GNNs model code semantics more effectively than flat token sequences, and how techniques like attention mechanisms, hybrid graph construction, and feedback loops significantly reduce false positives. With insights from real-world datasets and recent research, this guide shows how to build more reliable, proactive, and interpretable vulnerability detection systems using GNNs.

How can one start with crypto wallet development.pptxlaravinson24

Download Wondershare Filmora Crack [2025] With Latesttahirabibi60507

Copy & Past Link 👉👉 http://drfiles.net/ Wondershare Filmora is a video editing software and app designed for both beginners and experienced users. It's known for its user-friendly interface, drag-and-drop functionality, and a wide range of tools and features for creating and editing videos. Filmora is available on Windows, macOS, iOS (iPhone/iPad), and Android platforms.

Adobe Marketo Engage Champion Deep Dive - SFDC CRM Synch V2 & Usage DashboardsBradBedford3

Join Ajay Sarpal and Miray Vu to learn about key Marketo Engage enhancements. Discover improved in-app Salesforce CRM connector statistics for easy monitoring of sync health and throughput. Explore new Salesforce CRM Synch Dashboards providing up-to-date insights into weekly activity usage, thresholds, and limits with drill-down capabilities. Learn about proactive notifications for both Salesforce CRM sync and product usage overages. Get an update on improved Salesforce CRM synch scale and reliability coming in Q2 2025. Key Takeaways: Improved Salesforce CRM User Experience: Learn how self-service visibility enhances satisfaction. Utilize Salesforce CRM Synch Dashboards: Explore real-time weekly activity data. Monitor Performance Against Limits: See threshold limits for each product level. Get Usage Over-Limit Alerts: Receive notifications for exceeding thresholds. Learn About Improved Salesforce CRM Scale: Understand upcoming cloud-based incremental sync.

Get & Download Wondershare Filmora Crack Latest [2025]saniaaftab72555

Copy & Past Link 👉👉 https://dr-up-community.info/ Wondershare Filmora is a video editing software and app designed for both beginners and experienced users. It's known for its user-friendly interface, drag-and-drop functionality, and a wide range of tools and features for creating and editing videos. Filmora is available on Windows, macOS, iOS (iPhone/iPad), and Android platforms.

Requirements in Engineering AI- Enabled Systems: Open Problems and Safe AI Sy...Lionel Briand

Who Watches the Watchmen (SciFiDevCon 2025)Allon Mureinik

Tests, especially unit tests, are the developers’ superheroes. They allow us to mess around with our code and keep us safe. We often trust them with the safety of our codebase, but how do we know that we should? How do we know that this trust is well-deserved? Enter mutation testing – by intentionally injecting harmful mutations into our code and seeing if they are caught by the tests, we can evaluate the quality of the safety net they provide. By watching the watchmen, we can make sure our tests really protect us, and we aren’t just green-washing our IDEs to a false sense of security. Talk from SciFiDevCon 2025 https://www.scifidevcon.com/courses/2025-scifidevcon/contents/680efa43ae4f5

Avast Premium Security Crack FREE Latest Version 2025mu394968

🌍📱👉COPY LINK & PASTE ON GOOGLE https://dr-kain-geera.info/👈🌍 Avast Premium Security is a paid subscription service that provides comprehensive online security and privacy protection for multiple devices. It includes features like antivirus, firewall, ransomware protection, and website scanning, all designed to safeguard against a wide range of online threats, according to Avast. Key features of Avast Premium Security: Antivirus: Protects against viruses, malware, and other malicious software, according to Avast. Firewall: Controls network traffic and blocks unauthorized access to your devices, as noted by All About Cookies. Ransomware protection: Helps prevent ransomware attacks, which can encrypt your files and hold them hostage. Website scanning: Checks websites for malicious content before you visit them, according to Avast. Email Guardian: Scans your emails for suspicious attachments and phishing attempts. Multi-device protection: Covers up to 10 devices, including Windows, Mac, Android, and iOS, as stated by 2GO Software. Privacy features: Helps protect your personal data and online privacy. In essence, Avast Premium Security provides a robust suite of tools to keep your devices and online activity safe and secure, according to Avast.

Douwan Crack 2025 new verson+ License codeaneelaramzan63

Solidworks Crack 2025 latest new + license codeaneelaramzan63

Not So Common Memory Leaks in Java WebinarTier1 app

This SlideShare presentation is from our May webinar, “Not So Common Memory Leaks & How to Fix Them?”, where we explored lesser-known memory leak patterns in Java applications. Unlike typical leaks, subtle issues such as thread local misuse, inner class references, uncached collections, and misbehaving frameworks often go undetected and gradually degrade performance. This deck provides in-depth insights into identifying these hidden leaks using advanced heap analysis and profiling techniques, along with real-world case studies and practical solutions. Ideal for developers and performance engineers aiming to deepen their understanding of Java memory management and improve application stability.

Exploring Wayland: A Modern Display Server for the FutureICS

Expand your AI adoption with AgentExchangeFexle Services Pvt. Ltd.

Societal challenges of AI: biases, multilinguism and sustainabilityJordi Cabot

LEARN SEO AND INCREASE YOUR KNOWLDGE IN SOFTWARE INDUSTRYNidaFarooq10

How Valletta helped healthcare SaaS to transform QA and compliance to grow wi...Egor Kaleynik

How to Optimize Your AWS Environment for Improved Cloud PerformanceThousandEyes

Pixologic ZBrush Crack Plus Activation Key [Latest 2025] New Versionsaimabibi60507

EASEUS Partition Master Crack + License Codeaneelaramzan63

Maxon CINEMA 4D 2025 Crack FREE Download LINKyounisnoman75

Proactive Vulnerability Detection in Source Code Using Graph Neural Networks:...Ranjan Baisak

How can one start with crypto wallet development.pptxlaravinson24

Download Wondershare Filmora Crack [2025] With Latesttahirabibi60507

Adobe Marketo Engage Champion Deep Dive - SFDC CRM Synch V2 & Usage DashboardsBradBedford3

Get & Download Wondershare Filmora Crack Latest [2025]saniaaftab72555

Requirements in Engineering AI- Enabled Systems: Open Problems and Safe AI Sy...Lionel Briand

Who Watches the Watchmen (SciFiDevCon 2025)Allon Mureinik

Avast Premium Security Crack FREE Latest Version 2025mu394968

Douwan Crack 2025 new verson+ License codeaneelaramzan63

Solidworks Crack 2025 latest new + license codeaneelaramzan63

Not So Common Memory Leaks in Java WebinarTier1 app

Exploring Wayland: A Modern Display Server for the FutureICS

Apache Kafka - Martin Podval

1. Apache Kafka @MartinPodval, hpsv.cz

2. What is Apache Kafka? Messaging System Distributed Persistent and Replicable Very fast - low latency - and scalable Simple but highly configurable By Linkedin, open sourced under apache.org

3. Data Streaming New kind of data ... ● User or application data (events) streams ● Monitoring - App, System ● App Logging ● High volume

4. Data Streaming Cont’d … you want to process ● Using various components ● Into a target form ● Map, reduce, shuffle ● Real time or batch

5. HP Service Virtualization Use Cases Process of clients message streams Real-time performance modeling Logs aggregation

6. How To Solve It? Producers and Consumers ● Distributed ● Decoupled ● Configurable ● Dynamic

7. Kafka Cluster Brokers ● = Instances, Nodes ● Topics ● Partitions ● Replicas ZK ● Coordination

8. Kafka Topics Commit Log ● Immutable ● Ordered ● Sequential Offset

9. Kafka Topics Cont’d Partitioned Independently: ● Stored ● Produced ● Consumed ⇒ Scalable Replicated ● On partition basis ● Different brokers ⇒ Fault Tolerant

10. What Can I Do? producer. write(topic_id, message); consumer. read(topic_id, offset);

11. I Want To Produce ● java/scala client ● address of one or more brokers ● choose a topic where to produce ● highly configurable and tunable: ○ partitioner ○ number of acks (async=0, master=1, replicas=1+?) ○ batching, buffer size, timeouts, retries, ...

12. I Want To Consume High Level API ● Groups abstraction ○ To All, To One ○ To Some ● Stream API ● Stores positions to support fault tolerance

13. I Want To Consume Cont’d Low Level ● Java/scala client ● Find a leader for a topic ● Calculate an offset ● Fetches messages ○ Re-consume if needed

14. I Want To Consume Cont’d Delivery Semantic: ● At most once ● At least once ● Exactly once

15. Kafka Internals - Disks Avoid: ● GC ● Random disk access

16. Kafka Internals - Disks Cont’d Disks are fast ... … when properly used ● sequential access - read ahead, write behind ● rely on operating system ○ avoid heap, materialization and GC ● it’s more like file copy over network It’s easy … with immutable topics

17. Kafka Internals - Replication “In Sync” Replicas ● Replication factor on partition basis ● One leader + 0..n replicas ● Replicas are consumers ○ “In Sync” if they are not “too far” behind a leader ○ Batch sync

18. Kafka Internals - Replication Cont’d Tunable Trade-Offs ● Producer’s write method: ○ Not blocked, async ○ Waits for master ACK ○ Waits for all in-sync replicas ● Consumer pulls only committed messages ● Server’s minimum in-sync replicas

19. Performance “Incredible” Scales with: ● clients count, message size ● number of replicas, partitions or topics Depends on network and disk throughput

20. Performance Cont’d Our testing ● 3 nodes, master + 2 replicas ● 500 000 msg/s (100 bytes[]) ● 400 mbit/s - 1.2 gbit/s network throughput ● end2end latency 2-3 ms @see http://bit.ly/1FsIR9a

21. Easy of Use ● No installation, just run a java/scala program ● Streams in files & dirs ● Transparent zookeeper ● Ecosystem

22. Cons ● Beta version ● Dependency on Zookeeper ● The way how it is written in Scala ● No easy way how to remove messages

23. Questions?

Apache Kafka - Martin Podval

Recommended

More Related Content

What's hot (20)

Similar to Apache Kafka - Martin Podval (20)

Recently uploaded (20)

Apache Kafka - Martin Podval