Apache kafka

Feb 23, 2013Download as PPTX, PDF32 likes24,218 views

This document discusses Apache Kafka, an open-source distributed event streaming platform. It provides an introduction to Kafka's design and capabilities including: 1) Kafka is a distributed publish-subscribe messaging system that can handle high throughput workloads with low latency. 2) It is designed for real-time data pipelines and activity streaming and can be used for transporting logs, metrics collection, and building real-time applications. 3) Kafka supports distributed, scalable, fault-tolerant storage and processing of streaming data across multiple producers and consumers.

Rahul Jain
Software Engineer
www.linkedin.com/in/rahuldausa

 Why Kafka
 Introduction
 Design
 Q&A

*Apache ActiveMQ, JBoss HornetQ, Zero MQ, RabbitMQ are respective brands of Apache Software Foundation,
JBoss Inc, iMatix Corporation and Vmware Inc.

 Transportation of logs
 Activity Stream in Real time.
 Collection of Performance Metrics
◦ CPU/IO/Memory usage
◦ Application Specific
 Time taken to load a web-page.
 Time taken by Multiple Services while building a web-page.
 No of requests.
 No of hits on a particular page/url.

 Scalable: Need to be Highly Scalable. A lot of Data. It can be
billions of message.
 Reliability of messages, What If, I loose a small no. of
messages. Is it fine with me ?.
 Distributed : Multiple Producers, Multiple Consumers
 High-throughput: Does not require to have JMS Standards,
as it may be overkill for some use-cases like transportation of logs.
◦ As per JMS, each message has to be acknowledged back.
◦ Exactly one delivery guarantee requires two-phase commit.

Credit : http://kafka.apache.org/design.html

Handshake
Producer Zookeeper

Consumer

Coordination
Producer
Kafka Broker

.
.
Producer . Consumer
.
.

Producer Kafka Broker

Coordination

Handshake
Store Consumed Offset
Producer Zookeeper and Watch for Cluster Consumer 1
event (groupId1)

Producer
Kafka Broker
(Partition 1)
Consumer 2
. (groupId1)
.
Producer .
.
.

Producer Kafka Broker
(Partition 2) * Consumer 3
(groupId1)

* Consumer 3 would not receive any data, as number
of consumers are more than number of partitions.

 Filesystem Cache
 Zero-copy transfer of messages
 Batching of Messages
 Batch Compression
 Automatic Producer Load balancing.
 Broker does not Push messages to Consumer,
Consumer Polls messages from Broker.
 And Some others.
 Cluster formation of Broker/Consumer using Zookeeper, So on the fly more consumer, broker
can be introduced. The new cluster rebalancing will be taken care by Zookeeper
 Data is persisted in broker and is not removed on consumption (till retention period), so if one
consumer fails while consuming, same message can be re-consume again later from broker.
 Simplified storage mechanism for message, not for each message per consumer.

Producer Performance Consumer Performance

Credit : http://research.microsoft.com/en-us/UM/people/srikanth/netdb11/netdb11papers/netdb11-final12.pdf

Credit: https://cwiki.apache.org/confluence/display/KAFKA/Powered+By

Apache Kafka is a distributed publish-subscribe messaging system that can handle high volumes of data and enable messages to be passed from one endpoint to another. It uses a distributed commit log that allows messages to be persisted on disk for durability. Kafka is fast, scalable, fault-tolerant, and guarantees zero data loss. It is used by companies like LinkedIn, Twitter, and Netflix to handle high volumes of real-time data and streaming workloads.

Kafka presentationMohammed Fazuluddin

Kafka is an open source messaging system that can handle massive streams of data in real-time. It is fast, scalable, durable, and fault-tolerant. Kafka is commonly used for stream processing, website activity tracking, metrics collection, and log aggregation. It supports high throughput, reliable delivery, and horizontal scalability. Some examples of real-time use cases for Kafka include website monitoring, network monitoring, fraud detection, and IoT applications.

Apache Kafka - Messaging System OverviewDmitry Tolpeko

Introduction to Apache KafkaJeff Holoman

The document provides an introduction and overview of Apache Kafka presented by Jeff Holoman. It begins with an agenda and background on the presenter. It then covers basic Kafka concepts like topics, partitions, producers, consumers and consumer groups. It discusses efficiency and delivery guarantees. Finally, it presents some use cases for Kafka and positioning around when it may or may not be a good fit compared to other technologies.

Apache kafkaViswanath J

Kafka is a distributed publish-subscribe messaging system that allows both streaming and storage of data feeds. It is designed to be fast, scalable, durable, and fault-tolerant. Kafka maintains feeds of messages called topics that can be published to by producers and subscribed to by consumers. A Kafka cluster typically runs on multiple servers called brokers that store topics which may be partitioned and replicated for fault tolerance. Producers publish messages to topics which are distributed to consumers through consumer groups that balance load.

Apache Kafkaemreakis

Apache Kafka is a distributed streaming platform used for building real-time data pipelines and streaming apps. It provides a unified, scalable, and durable platform for handling real-time data feeds. Kafka works by accepting streams of records from one or more producers and organizing them into topics. It allows both storing and forwarding of these streams to consumers. Producers write data to topics which are replicated across clusters for fault tolerance. Consumers can then read the data from the topics in the order it was produced. Major companies like LinkedIn, Yahoo, Twitter, and Netflix use Kafka for applications like metrics, logging, stream processing and more.

Apache kafkaKumar Shivam

This document provides an introduction to Apache Kafka, an open-source distributed event streaming platform. It discusses Kafka's history as a project originally developed by LinkedIn, its use cases like messaging, activity tracking and stream processing. It describes key Kafka concepts like topics, partitions, offsets, replicas, brokers and producers/consumers. It also gives examples of how companies like Netflix, Uber and LinkedIn use Kafka in their applications and provides a comparison to Apache Spark.

Apache Kafka IntroductionAmita Mirajkar

Apache kafkaSrikrishna k

Hello, kafka! (an introduction to apache kafka)Timothy Spann

Kafka Tutorial - Introduction to Apache Kafka (Part 1)Jean-Paul Azar

Why is Kafka so fast? Why is Kafka so popular? Why Kafka? This slide deck is a tutorial for the Kafka streaming platform. This slide deck covers Kafka Architecture with some small examples from the command line. Then we expand on this with a multi-server example to demonstrate failover of brokers as well as consumers. Then it goes through some simple Java client examples for a Kafka Producer and a Kafka Consumer. We have also expanded on the Kafka design section and added references. The tutorial covers Avro and the Schema Registry as well as advance Kafka Producers.

Apache kafkaLong Nguyen

Apache Kafka is a distributed publish-subscribe messaging system that allows for high throughput, low latency data ingestion and distribution. It provides reliability through replication, scalability by partitioning topics across brokers, and durability by persisting messages to disk. Common uses of Kafka include metrics collection, log aggregation, and stream processing using frameworks like Spark Streaming. Kafka's architecture includes brokers that store topics which are partitions distributed across a cluster, with ZooKeeper for coordination. Producers write messages to topics and consumers read messages in a subscriber model.

A visual introduction to Apache KafkaPaul Brebner

kafkaAmikam Snir

Kafka is an open-source distributed commit log service that provides high-throughput messaging functionality. It is designed to handle large volumes of data and different use cases like online and offline processing more efficiently than alternatives like RabbitMQ. Kafka works by partitioning topics into segments spread across clusters of machines, and replicates across these partitions for fault tolerance. It can be used as a central data hub or pipeline for collecting, transforming, and streaming data between systems and applications.

Kafka 101Clement Demonchy

Fundamentals of Apache KafkaChhavi Parasher

Introduction to Kafka and ZookeeperRahul Jain

No data loss pipeline with apache kafkaJiangjie Qin

The document discusses how to configure Apache Kafka to prevent data loss and message reordering in a data pipeline. It recommends settings like enabling block on buffer full, using acks=all for synchronous message acknowledgment, limiting in-flight requests, and committing offsets only after messages are processed. It also suggests replicating topics across at least 3 brokers and using a minimum in-sync replica factor of 2. Mirror makers can further ensure no data loss or reordering by consuming from one cluster and producing to another in order while committing offsets. Custom consumer listeners and message handlers allow for mirroring optimizations.

Kafkashrenikp

This document provides an overview of Kafka, a distributed streaming platform. It can publish and subscribe to streams of records, store streams durably across clusters, and process streams as they occur. The Kafka cluster stores streams of records in topics. It has four main APIs: Producer API to publish data, Consumer API to subscribe to topics, Streams API to transform streams, and Connector API to connect Kafka and other systems. Records in Kafka topics are partitioned and ordered with offsets for scalability and fault tolerance. Consumers subscribe to topics in consumer groups to process partitions in parallel.

Apache Kafka Best PracticesDataWorks Summit/Hadoop Summit

Apache Kafka becoming the message bus to transfer huge volumes of data from various sources into Hadoop. It's also enabling many real-time system frameworks and use cases. Managing and building clients around Apache Kafka can be challenging. In this talk, we will go through the best practices in deploying Apache Kafka in production. How to Secure a Kafka Cluster, How to pick topic-partitions and upgrading to newer versions. Migrating to new Kafka Producer and Consumer API. Also talk about the best practices involved in running a producer/consumer. In Kafka 0.9 release, we’ve added SSL wire encryption, SASL/Kerberos for user authentication, and pluggable authorization. Now Kafka allows authentication of users, access control on who can read and write to a Kafka topic. Apache Ranger also uses pluggable authorization mechanism to centralize security for Kafka and other Hadoop ecosystem projects. We will showcase open sourced Kafka REST API and an Admin UI that will help users in creating topics, re-assign partitions, Issuing Kafka ACLs and monitoring Consumer offsets.

APACHE KAFKA / Kafka Connect / Kafka StreamsKetan Gote

Kafka basicsJoão Paulo Leonidas Fernandes Dias da Silva

Kafka is an open-source message broker that provides high-throughput and low-latency data processing. It uses a distributed commit log to store messages in categories called topics. Processes that publish messages are producers, while processes that subscribe to topics are consumers. Consumers can belong to consumer groups for parallel processing. Kafka guarantees order and no lost messages. It uses Zookeeper for metadata and coordination.

Introduction to Kafka StreamsGuozhang Wang

Kafka Streams is a new stream processing library natively integrated with Kafka. It has a very low barrier to entry, easy operationalization, and a natural DSL for writing stream processing applications. As such it is the most convenient yet scalable option to analyze, transform, or otherwise process data that is backed by Kafka. We will provide the audience with an overview of Kafka Streams including its design and API, typical use cases, code examples, and an outlook of its upcoming roadmap. We will also compare Kafka Streams' light-weight library approach with heavier, framework-based tools such as Spark Streaming or Storm, which require you to understand and operate a whole different infrastructure for processing real-time data in Kafka.

Messaging queue - KafkaMayank Bansal

This document discusses messaging queues and compares Kafka and Amazon SQS. It begins by explaining what a messaging queue is and provides examples of software that can be used, including Kafka, SQS, SNS, and RabbitMQ. It then discusses why messaging queues are useful by allowing for asynchronous and failed processing. The document proceeds to provide details on Kafka, including that it is a distributed streaming platform used by companies like LinkedIn, Twitter, and Netflix. It defines Kafka terminology and discusses how producers and consumers work. Finally, it compares features of SQS and Kafka like order of messages, delivery guarantees, retention, security, costs, and throughput.

Introduction to apache kafkaDimitris Kontokostas

Apache Kafka is a distributed streaming platform and distributed publish-subscribe messaging system. It uses a log abstraction to order events and replicate data across clusters. Kafka allows developers to publish and subscribe to streams of records known as topics. Producers publish data to topics and consumers subscribe to topics to process streams of records. The Kafka ecosystem includes tools like KStreams for stream processing and KSQL for querying streams of data.

Apache KafkaSaroj Panyasrivanit

This document provides an overview of Apache Kafka. It begins with defining Kafka as a distributed streaming platform and messaging system. It then lists the agenda which includes what Kafka is, why it is used, common use cases, major companies that use it, how it achieves high performance, and core concepts. Core concepts explained include topics, partitions, brokers, replication, leaders, and producers and consumers. The document also provides examples to illustrate these concepts.

Introduction to Apache KafkaAIMDek Technologies

Apache Kafka is an open-source distributed event streaming platform used for building real-time data pipelines and streaming apps. It was developed by LinkedIn in 2011 to solve problems with data integration and processing. Kafka uses a publish-subscribe messaging model and is designed to be fast, scalable, and durable. It allows both streaming and storage of data and acts as a central data backbone for large organizations.

Introduction to RedisDvir Volk

Kafka RealTime StreamingViyaan Jhiingade

Kafka is a distributed publish-subscribe messaging system that provides high throughput and low latency for processing streaming data. It is used to handle large volumes of data in real-time by partitioning topics across multiple servers or brokers. Kafka maintains ordered and immutable logs of messages that can be consumed by subscribers. It provides features like replication, fault tolerance and scalability. Some key Kafka concepts include producers that publish messages, consumers that subscribe to topics, brokers that handle data streams, topics to categorize related messages, and partitions to distribute data loads across clusters.

Developing Real-Time Data Pipelines with Apache KafkaJoe Stein

Apache Kafka is a distributed streaming platform that allows for building real-time data pipelines and streaming apps. It provides a publish-subscribe messaging system with persistence that allows for building real-time streaming applications. Producers publish data to topics which are divided into partitions. Consumers subscribe to topics and process the streaming data. The system handles scaling and data distribution to allow for high throughput and fault tolerance.

More Related Content

What's hot (20)

Apache kafkaSrikrishna k

Hello, kafka! (an introduction to apache kafka)Timothy Spann

Kafka Tutorial - Introduction to Apache Kafka (Part 1)Jean-Paul Azar

Apache kafkaLong Nguyen

A visual introduction to Apache KafkaPaul Brebner

kafkaAmikam Snir

Kafka 101Clement Demonchy

Fundamentals of Apache KafkaChhavi Parasher

Introduction to Kafka and ZookeeperRahul Jain

No data loss pipeline with apache kafkaJiangjie Qin

Kafkashrenikp

Apache Kafka Best PracticesDataWorks Summit/Hadoop Summit

APACHE KAFKA / Kafka Connect / Kafka StreamsKetan Gote

Kafka basicsJoão Paulo Leonidas Fernandes Dias da Silva

Introduction to Kafka StreamsGuozhang Wang

Messaging queue - KafkaMayank Bansal

Introduction to apache kafkaDimitris Kontokostas

Apache KafkaSaroj Panyasrivanit

Introduction to Apache KafkaAIMDek Technologies

Introduction to RedisDvir Volk

Apache kafkaSrikrishna k

Hello, kafka! (an introduction to apache kafka)Timothy Spann

Kafka Tutorial - Introduction to Apache Kafka (Part 1)Jean-Paul Azar

Apache kafkaLong Nguyen

A visual introduction to Apache KafkaPaul Brebner

kafkaAmikam Snir

Kafka 101Clement Demonchy

Fundamentals of Apache KafkaChhavi Parasher

Introduction to Kafka and ZookeeperRahul Jain

No data loss pipeline with apache kafkaJiangjie Qin

Kafkashrenikp

Apache Kafka Best PracticesDataWorks Summit/Hadoop Summit

APACHE KAFKA / Kafka Connect / Kafka StreamsKetan Gote

Kafka basicsJoão Paulo Leonidas Fernandes Dias da Silva

Introduction to Kafka StreamsGuozhang Wang

Messaging queue - KafkaMayank Bansal

Introduction to apache kafkaDimitris Kontokostas

Apache KafkaSaroj Panyasrivanit

Introduction to Apache KafkaAIMDek Technologies

Introduction to RedisDvir Volk

Similar to Apache kafka (20)

Kafka RealTime StreamingViyaan Jhiingade

Developing Real-Time Data Pipelines with Apache KafkaJoe Stein

Apache Kafka Women Who Code MeetupSnehal Nagmote

Apache Kafka is a distributed streaming platform used at WalmartLabs for various search use cases. It decouples data pipelines and allows real-time data processing. The key concepts include topics to categorize messages, producers that publish messages, brokers that handle distribution, and consumers that process message streams. WalmartLabs leverages features like partitioning for parallelism, replication for fault tolerance, and low-latency streaming.

Apache Kafka - Event Sourcing, Monitoring, Librdkafka, Scaling & PartitioningGuido Schmutz

Apache Kafka is a distributed streaming platform. It provides a high-throughput distributed messaging system with publish-subscribe capabilities. The document discusses Kafka producers and consumers, Kafka clients in different programming languages, and important configuration settings for Kafka brokers and topics. It also demonstrates sending messages to Kafka topics from a Java producer and consuming messages from the console consumer.

Kafka.pptx (uploaded from MyFiles SomnathDeb_PC)somnathdeb0212

Apache KafkaJoe Stein

Apache Kafka is a distributed publish-subscribe messaging system that was originally created by LinkedIn and contributed to the Apache Software Foundation. It is written in Scala and provides a multi-language API to publish and consume streams of records. Kafka is useful for both log aggregation and real-time messaging due to its high performance, scalability, and ability to serve as both a distributed messaging system and log storage system with a single unified architecture. To use Kafka, one runs Zookeeper for coordination, Kafka brokers to form a cluster, and then publishes and consumes messages with a producer API and consumer API.

BDW Chicago 2016 - Jayesh Thakrar, Sr. Software Engineer, Conversant - Data...Big Data Week

We all know that Kafka is a fast, scalable, durable and distributed publish-subscribe messaging system. From large internet companies to startups are using it as part of their streaming data pipeline for its speed, durability and ease of use/administration. Kafka has a strong durability promise, yet, there are certain situations when there can be data loss or data duplication. While data duplication is easy to detect, data loss is not when you are dealing with hundreds of millions of messages. This session will look into those scenarios and possible ways to detect and minimize data loss and data duplication.

Flume-Cassandra Log ProcessorCLOUDIAN KK

Gemini Mobile Technologies ("Gemini") released a Real-Time Log Processing System based on Flume and Cassandra ("Flume-Cassandra Log Processor") as open source. The Flume-Cassandra Log Processor enables massive volumes of production system logs to be collected and processed into graphical reports, in real-time. In addition, logs from multiple data centers can be simultaneously aggregated and analyzed in a single database.

Cluster_Performance_Apache_Kafak_vs_RabbitMQShameera Rathnayaka

The document compares the performance of Apache Kafka and RabbitMQ for streaming data. It finds that without fault tolerance, both brokers have similar latency, but with fault tolerance enabled, Kafka has slightly higher latency than RabbitMQ. Latency increases with message size and is improved after an initial warmup period. Overall, RabbitMQ demonstrated the lowest latency for both configurations. The document also describes how each system is deployed and configured for the performance tests.

KAFKA QuickstartVikram Singh Chandel

Kafka is a distributed publish-subscribe messaging system that allows processes called producers to publish messages to topics, and processes called consumers to subscribe to topics and receive the stream of published messages. It maintains feeds of messages in categories called topics, and messages are distributed across partitions in a fault-tolerant way. Kafka is run as a cluster of servers called brokers that together form a scalable and durable messaging backbone.

Developing Realtime Data Pipelines With Apache KafkaJoe Stein

Developing Realtime Data Pipelines With Apache Kafka. Apache Kafka is publish-subscribe messaging rethought as a distributed commit log. A single Kafka broker can handle hundreds of megabytes of reads and writes per second from thousands of clients. Kafka is designed to allow a single cluster to serve as the central data backbone for a large organization. It can be elastically and transparently expanded without downtime. Data streams are partitioned and spread over a cluster of machines to allow data streams larger than the capability of any single machine and to allow clusters of co-ordinated consumers. Messages are persisted on disk and replicated within the cluster to prevent data loss. Each broker can handle terabytes of messages without performance impact. Kafka has a modern cluster-centric design that offers strong durability and fault-tolerance guarantees.

Introduction and Overview of Apache Kafka, TriHUG July 23, 2013mumrah

Session 23 - Kafka and ZookeeperAnandMHadoop

Fault Tolerance with KafkaEdureka!

This document discusses Apache Kafka, an open-source distributed event streaming platform. It provides an overview of Kafka's architecture, how it achieves fault tolerance through replication, and examples of companies that use Kafka like LinkedIn for powering their newsfeed and recommendations. The document also outlines a hands-on exercise on fault tolerance with Kafka and includes references for further reading.

Consuming, providing and publishing Web ServicesIoannis Baltopoulos

This document discusses writing web service consumers and providers using the Apache Axis framework. It covers generating client-side code using WSDL2Java, writing simple providers using instant deployment, and more advanced custom deployment including generating WSDL files, server-side bindings, and deploying services. The document is a tutorial on the software environment, tools, and steps needed to consume, provide and publish web services with Axis.

Apache Kafka - Scalable Message-Processing and more !Guido Schmutz

Apache Kafka is a distributed publish-subscribe messaging system that allows for scalable message processing. It provides high throughput, fault tolerance, and guarantees delivery. Kafka maintains feeds of messages in topics which can be consumed by applications or services. It is commonly used for processing real-time data streams and event-driven architectures. Confluent provides a platform for Apache Kafka with additional tools for monitoring, management, and integration with other data systems.

Kafka Deep DiveKnoldus Inc.

Kafka overviewShanki Singh Gandhi

Apache Kafka is an open-source stream processing platform written in Scala and Java that provides a unified platform for handling real-time data feeds with high throughput and low latency. It operates as a cluster on one or more servers to store streams of records in categories called topics. Each record consists of a key, value, and timestamp. Producers publish data to topics while consumers subscribe to topics and receive messages. Topics are divided into partitions with replicas for redundancy.

Kafka internalsDavid Groozman

Apache Kafka - Scalable Message Processing and more!Guido Schmutz

Kafka RealTime StreamingViyaan Jhiingade

Developing Real-Time Data Pipelines with Apache KafkaJoe Stein

Apache Kafka Women Who Code MeetupSnehal Nagmote

Apache Kafka - Event Sourcing, Monitoring, Librdkafka, Scaling & PartitioningGuido Schmutz

Kafka.pptx (uploaded from MyFiles SomnathDeb_PC)somnathdeb0212

Apache KafkaJoe Stein

BDW Chicago 2016 - Jayesh Thakrar, Sr. Software Engineer, Conversant - Data...Big Data Week

Flume-Cassandra Log ProcessorCLOUDIAN KK

Cluster_Performance_Apache_Kafak_vs_RabbitMQShameera Rathnayaka

KAFKA QuickstartVikram Singh Chandel

Developing Realtime Data Pipelines With Apache KafkaJoe Stein

Introduction and Overview of Apache Kafka, TriHUG July 23, 2013mumrah

Session 23 - Kafka and ZookeeperAnandMHadoop

Fault Tolerance with KafkaEdureka!

Consuming, providing and publishing Web ServicesIoannis Baltopoulos

Apache Kafka - Scalable Message-Processing and more !Guido Schmutz

Kafka Deep DiveKnoldus Inc.

Kafka overviewShanki Singh Gandhi

Kafka internalsDavid Groozman

Apache Kafka - Scalable Message Processing and more!Guido Schmutz

More from Rahul Jain (14)

Flipkart Strategy Analysis and RecommendationRahul Jain

Flipkart is India's largest e-commerce company. It has a 40% market share in India's online retail industry, which was $64 billion in 2020 and is projected to grow to $200 billion by 2027. Flipkart has made several acquisitions to expand into related businesses like online travel, financial services, and logistics. It aims to increase its market share in key categories like mobile, electronics, fashion, and grocery. To achieve this, Flipkart plans to expand its fulfillment center network to smaller cities, focus on private labels, and increase offerings in high-engagement categories. It also aims to leverage its investments in Myntra, PhonePe and Cleartrip to drive profit

Emerging technologies /frameworks in Big DataRahul Jain

Case study of Rujhaan.com (A social news app )Rahul Jain

Rujhaan.com is a news aggregation app that collects trending news and social media discussions from topics of interest to users. It uses various technologies including a crawler to collect data from social media, Apache Solr for search, MongoDB for storage, Redis for caching, and machine learning techniques like classification and clustering. The presenter discussed the technical architecture and challenges of building Rujhaan.com to provide fast, personalized news content to over 16,000 monthly users while scaling to growing traffic levels.

Building a Large Scale SEO/SEM Application with Apache SolrRahul Jain

Slides from my talk on "Building a Large Scale SEO/SEM Application with Apache Solr" in Lucene/Solr Revolution 2014 where I talk how we handle Indexing/Search of 40 billion records (documents)/month in Apache Solr with 4.6 TB compressed index data. Abstract: We are working on building a SEO/SEM application where an end user search for a "keyword" or a "domain" and gets all the insights about these including Search engine ranking, CPC/CPM, search volume, No. of Ads, competitors details etc. in a couple of seconds. To have this intelligence, we get huge web data from various sources and after intensive processing it is 40 billion records/month in MySQL database with 4.6 TB compressed index data in Apache Solr. Due to large volume, we faced several challenges while improving indexing performance, search latency and scaling the overall system. In this session, I will talk about our several design approaches to import data faster from MySQL, tricks & techniques to improve the indexing performance, Distributed Search, DocValues(life saver), Redis and the overall system architecture.

Real time Analytics with Apache Kafka and Apache SparkRahul Jain

A presentation cum workshop on Real time Analytics with Apache Kafka and Apache Spark. Apache Kafka is a distributed publish-subscribe messaging while other side Spark Streaming brings Spark's language-integrated API to stream processing, allows to write streaming applications very quickly and easily. It supports both Java and Scala. In this workshop we are going to explore Apache Kafka, Zookeeper and Spark with a Web click streaming example using Spark Streaming. A clickstream is the recording of the parts of the screen a computer user clicks on while web browsing.

Introduction to Apache SparkRahul Jain

Introduction to Machine LearningRahul Jain

Introduction to ScalaRahul Jain

What is NoSQL and CAP TheoremRahul Jain

This document discusses NoSQL and the CAP theorem. It begins with an introduction of the presenter and an overview of topics to be covered: What is NoSQL and the CAP theorem. It then defines NoSQL, provides examples of major NoSQL categories (document, graph, key-value, and wide-column stores), and explains why NoSQL is used, including to handle large, dynamic, and distributed data. The document also explains the CAP theorem, which states that a distributed data store can only satisfy two of three properties: consistency, availability, and partition tolerance. It provides examples of how to choose availability over consistency or vice versa. Finally, it concludes that both SQL and NoSQL have valid use cases and a combination

Introduction to Elasticsearch with basics of LuceneRahul Jain

Rahul Jain gives an introduction to Elasticsearch and its basic concepts like term frequency, inverse document frequency, and boosting. He describes Lucene as a fast, scalable search library that uses inverted indexes. Elasticsearch is introduced as an open source search platform built on Lucene that provides distributed indexing, replication, and load balancing. Logstash and Kibana are also briefly described as tools for collecting, parsing, and visualizing logs in Elasticsearch.

Introduction to Apache Lucene/SolrRahul Jain

This document provides an introduction to Apache Lucene and Solr. It begins with an overview of information retrieval and some basic concepts like term frequency-inverse document frequency. It then describes Lucene as a fast, scalable search library and discusses its inverted index and indexing pipeline. Solr is introduced as an enterprise search platform built on Lucene that provides features like faceting, scalability and real-time indexing. The document concludes with examples of how Lucene and Solr are used in applications and websites for search, analytics, auto-suggestion and more.

Introduction to Lucene & Solr and UsecasesRahul Jain

Rahul Jain gave a presentation on Lucene and Solr. He began with an overview of information retrieval and the inverted index. He then discussed Lucene, describing it as an open source information retrieval library for indexing and searching. He discussed Solr, describing it as an enterprise search platform built on Lucene that provides distributed indexing, replication, and load balancing. He provided examples of how Solr is used for search, analytics, auto-suggest, and more by companies like eBay, Netflix, and Twitter.

Hadoop & HDFS for BeginnersRahul Jain

Hibernate tutorial for beginnersRahul Jain

Flipkart Strategy Analysis and RecommendationRahul Jain

Emerging technologies /frameworks in Big DataRahul Jain

Case study of Rujhaan.com (A social news app )Rahul Jain

Building a Large Scale SEO/SEM Application with Apache SolrRahul Jain

Real time Analytics with Apache Kafka and Apache SparkRahul Jain

Introduction to Apache SparkRahul Jain

Introduction to Machine LearningRahul Jain

Introduction to ScalaRahul Jain

What is NoSQL and CAP TheoremRahul Jain

Introduction to Elasticsearch with basics of LuceneRahul Jain

Introduction to Apache Lucene/SolrRahul Jain

Introduction to Lucene & Solr and UsecasesRahul Jain

Hadoop & HDFS for BeginnersRahul Jain

Hibernate tutorial for beginnersRahul Jain

Recently uploaded (20)

Cyber Awareness overview for 2025 month of securityriccardosl1

Massive Power Outage Hits Spain, Portugal, and France: Causes, Impact, and On...Aqusag Technologies

Generative Artificial Intelligence (GenAI) in BusinessDr. Tathagat Varma

DevOpsDays Atlanta 2025 - Building 10x Development Organizations.pptxJustin Reock

Building 10x Organizations with Modern Productivity Metrics 10x developers may be a myth, but 10x organizations are very real, as proven by the influential study performed in the 1980s, ‘The Coding War Games.’ Right now, here in early 2025, we seem to be experiencing YAPP (Yet Another Productivity Philosophy), and that philosophy is converging on developer experience. It seems that with every new method we invent for the delivery of products, whether physical or virtual, we reinvent productivity philosophies to go alongside them. But which of these approaches actually work? DORA? SPACE? DevEx? What should we invest in and create urgency behind today, so that we don’t find ourselves having the same discussion again in a decade?

AI and Data Privacy in 2025: Global TrendsInData Labs

In this infographic, we explore how businesses can implement effective governance frameworks to address AI data privacy. Understanding it is crucial for developing effective strategies that ensure compliance, safeguard customer trust, and leverage AI responsibly. Equip yourself with insights that can drive informed decision-making and position your organization for success in the future of data privacy. This infographic contains: -AI and data privacy: Key findings -Statistics on AI data privacy in the today’s world -Tips on how to overcome data privacy challenges -Benefits of AI data security investments. Keep up-to-date on how AI is reshaping privacy standards and what this entails for both individuals and organizations.

SAP Modernization: Maximizing the Value of Your SAP S/4HANA Migration.pdfPrecisely

Designing Low-Latency Systems with Rust and ScyllaDB: An Architectural Deep DiveScyllaDB

Want to learn practical tips for designing systems that can scale efficiently without compromising speed? Join us for a workshop where we’ll address these challenges head-on and explore how to architect low-latency systems using Rust. During this free interactive workshop oriented for developers, engineers, and architects, we’ll cover how Rust’s unique language features and the Tokio async runtime enable high-performance application development. As you explore key principles of designing low-latency systems with Rust, you will learn how to: - Create and compile a real-world app with Rust - Connect the application to ScyllaDB (NoSQL data store) - Negotiate tradeoffs related to data modeling and querying - Manage and monitor the database for consistently low latencies

Transcript: #StandardsGoals for 2025: Standards & certification roundup - Tec...BookNet Canada

Book industry standards are evolving rapidly. In the first part of this session, we’ll share an overview of key developments from 2024 and the early months of 2025. Then, BookNet’s resident standards expert, Tom Richardson, and CEO, Lauren Stewart, have a forward-looking conversation about what’s next. Link to recording, presentation slides, and accompanying resource: https://bnctechforum.ca/sessions/standardsgoals-for-2025-standards-certification-roundup/ Presented by BookNet Canada on May 6, 2025 with support from the Department of Canadian Heritage.

Andrew Marnell: Transforming Business Strategy Through Data-Driven InsightsAndrew Marnell

AI EngineHost Review: Revolutionary USA Datacenter-Based Hosting with NVIDIA ...SOFTTECHHUB

I started my online journey with several hosting services before stumbling upon Ai EngineHost. At first, the idea of paying one fee and getting lifetime access seemed too good to pass up. The platform is built on reliable US-based servers, ensuring your projects run at high speeds and remain safe. Let me take you step by step through its benefits and features as I explain why this hosting solution is a perfect fit for digital entrepreneurs.

ThousandEyes Partner Innovation Updates for May 2025ThousandEyes

Noah Loul Shares 5 Steps to Implement AI Agents for Maximum Business Efficien...Noah Loul

Artificial intelligence is changing how businesses operate. Companies are using AI agents to automate tasks, reduce time spent on repetitive work, and focus more on high-value activities. Noah Loul, an AI strategist and entrepreneur, has helped dozens of companies streamline their operations using smart automation. He believes AI agents aren't just tools—they're workers that take on repeatable tasks so your human team can focus on what matters. If you want to reduce time waste and increase output, AI agents are the next move.

Mobile App Development Company in Saudi ArabiaSteve Jonas

EmizenTech is a globally recognized software development company, proudly serving businesses since 2013. With over 11+ years of industry experience and a team of 200+ skilled professionals, we have successfully delivered 1200+ projects across various sectors. As a leading Mobile App Development Company In Saudi Arabia we offer end-to-end solutions for iOS, Android, and cross-platform applications. Our apps are known for their user-friendly interfaces, scalability, high performance, and strong security features. We tailor each mobile application to meet the unique needs of different industries, ensuring a seamless user experience. EmizenTech is committed to turning your vision into a powerful digital product that drives growth, innovation, and long-term success in the competitive mobile landscape of Saudi Arabia.

Role of Data Annotation Services in AI-Powered ManufacturingAndrew Leo

Rusty Waters: Elevating Lakehouses Beyond Sparkcarlyakerly1

Spark is a powerhouse for large datasets, but when it comes to smaller data workloads, its overhead can sometimes slow things down. What if you could achieve high performance and efficiency without the need for Spark? At S&P Global Commodity Insights, having a complete view of global energy and commodities markets enables customers to make data-driven decisions with confidence and create long-term, sustainable value. 🌍 Explore delta-rs + CDC and how these open-source innovations power lightweight, high-performance data applications beyond Spark! 🚀

Cybersecurity Identity and Access Solutions using Azure ADVICTOR MAESTRE RAMIREZ

#StandardsGoals for 2025: Standards & certification roundup - Tech Forum 2025BookNet Canada

Book industry standards are evolving rapidly. In the first part of this session, we’ll share an overview of key developments from 2024 and the early months of 2025. Then, BookNet’s resident standards expert, Tom Richardson, and CEO, Lauren Stewart, have a forward-looking conversation about what’s next. Link to recording, transcript, and accompanying resource: https://bnctechforum.ca/sessions/standardsgoals-for-2025-standards-certification-roundup/ Presented by BookNet Canada on May 6, 2025 with support from the Department of Canadian Heritage.

TrustArc Webinar: Consumer Expectations vs Corporate Realities on Data Broker...TrustArc

Most consumers believe they’re making informed decisions about their personal data—adjusting privacy settings, blocking trackers, and opting out where they can. However, our new research reveals that while awareness is high, taking meaningful action is still lacking. On the corporate side, many organizations report strong policies for managing third-party data and consumer consent yet fall short when it comes to consistency, accountability and transparency. This session will explore the research findings from TrustArc’s Privacy Pulse Survey, examining consumer attitudes toward personal data collection and practical suggestions for corporate practices around purchasing third-party data. Attendees will learn: - Consumer awareness around data brokers and what consumers are doing to limit data collection - How businesses assess third-party vendors and their consent management operations - Where business preparedness needs improvement - What these trends mean for the future of privacy governance and public trust This discussion is essential for privacy, risk, and compliance professionals who want to ground their strategies in current data and prepare for what’s next in the privacy landscape.

Quantum Computing Quick Research Guide by Arthur MorganArthur Morgan

This is a Quick Research Guide (QRG). QRGs include the following: - A brief, high-level overview of the QRG topic. - A milestone timeline for the QRG topic. - Links to various free online resource materials to provide a deeper dive into the QRG topic. - Conclusion and a recommendation for at least two books available in the SJPL system on the QRG topic. QRGs planned for the series: - Artificial Intelligence QRG - Quantum Computing QRG - Big Data Analytics QRG - Spacecraft Guidance, Navigation & Control QRG (coming 2026) - UK Home Computing & The Birth of ARM QRG (coming 2027) Any questions or comments? - Please contact Arthur Morgan at [email protected]. 100% human made.

HCL Nomad Web – Best Practices und Verwaltung von Multiuser-Umgebungenpanagenda

Webinar Recording: https://www.panagenda.com/webinars/hcl-nomad-web-best-practices-und-verwaltung-von-multiuser-umgebungen/ HCL Nomad Web wird als die nächste Generation des HCL Notes-Clients gefeiert und bietet zahlreiche Vorteile, wie die Beseitigung des Bedarfs an Paketierung, Verteilung und Installation. Nomad Web-Client-Updates werden “automatisch” im Hintergrund installiert, was den administrativen Aufwand im Vergleich zu traditionellen HCL Notes-Clients erheblich reduziert. Allerdings stellt die Fehlerbehebung in Nomad Web im Vergleich zum Notes-Client einzigartige Herausforderungen dar. Begleiten Sie Christoph und Marc, während sie demonstrieren, wie der Fehlerbehebungsprozess in HCL Nomad Web vereinfacht werden kann, um eine reibungslose und effiziente Benutzererfahrung zu gewährleisten. In diesem Webinar werden wir effektive Strategien zur Diagnose und Lösung häufiger Probleme in HCL Nomad Web untersuchen, einschließlich - Zugriff auf die Konsole - Auffinden und Interpretieren von Protokolldateien - Zugriff auf den Datenordner im Cache des Browsers (unter Verwendung von OPFS) - Verständnis der Unterschiede zwischen Einzel- und Mehrbenutzerszenarien - Nutzung der Client Clocking-Funktion

Cyber Awareness overview for 2025 month of securityriccardosl1

Massive Power Outage Hits Spain, Portugal, and France: Causes, Impact, and On...Aqusag Technologies

Generative Artificial Intelligence (GenAI) in BusinessDr. Tathagat Varma

DevOpsDays Atlanta 2025 - Building 10x Development Organizations.pptxJustin Reock

AI and Data Privacy in 2025: Global TrendsInData Labs

SAP Modernization: Maximizing the Value of Your SAP S/4HANA Migration.pdfPrecisely

Designing Low-Latency Systems with Rust and ScyllaDB: An Architectural Deep DiveScyllaDB

Transcript: #StandardsGoals for 2025: Standards & certification roundup - Tec...BookNet Canada

Andrew Marnell: Transforming Business Strategy Through Data-Driven InsightsAndrew Marnell

AI EngineHost Review: Revolutionary USA Datacenter-Based Hosting with NVIDIA ...SOFTTECHHUB

ThousandEyes Partner Innovation Updates for May 2025ThousandEyes

Noah Loul Shares 5 Steps to Implement AI Agents for Maximum Business Efficien...Noah Loul

Mobile App Development Company in Saudi ArabiaSteve Jonas

Role of Data Annotation Services in AI-Powered ManufacturingAndrew Leo

Rusty Waters: Elevating Lakehouses Beyond Sparkcarlyakerly1

Cybersecurity Identity and Access Solutions using Azure ADVICTOR MAESTRE RAMIREZ

#StandardsGoals for 2025: Standards & certification roundup - Tech Forum 2025BookNet Canada

TrustArc Webinar: Consumer Expectations vs Corporate Realities on Data Broker...TrustArc

Quantum Computing Quick Research Guide by Arthur MorganArthur Morgan

HCL Nomad Web – Best Practices und Verwaltung von Multiuser-Umgebungenpanagenda

Apache kafka

1. Rahul Jain Software Engineer www.linkedin.com/in/rahuldausa

2.  Why Kafka  Introduction  Design  Q&A

3. *Apache ActiveMQ, JBoss HornetQ, Zero MQ, RabbitMQ are respective brands of Apache Software Foundation, JBoss Inc, iMatix Corporation and Vmware Inc.

5.  Transportation of logs  Activity Stream in Real time.  Collection of Performance Metrics ◦ CPU/IO/Memory usage ◦ Application Specific  Time taken to load a web-page.  Time taken by Multiple Services while building a web-page.  No of requests.  No of hits on a particular page/url.

6.  Scalable: Need to be Highly Scalable. A lot of Data. It can be billions of message.  Reliability of messages, What If, I loose a small no. of messages. Is it fine with me ?.  Distributed : Multiple Producers, Multiple Consumers  High-throughput: Does not require to have JMS Standards, as it may be overkill for some use-cases like transportation of logs. ◦ As per JMS, each message has to be acknowledged back. ◦ Exactly one delivery guarantee requires two-phase commit.

7.  An Apache Project, initially developed by LinkedIn's SNA team.  A High-throughput distributed Publish- Subscribe based messaging system.  A Kind of Data Pipeline  Written in Scala.  Does not follow JMS Standards, neither uses JMS APIs.  Supports both queue and topic semantics.

8. Credit : http://kafka.apache.org/design.html

9. Handshake Producer Zookeeper Consumer Coordination Producer Kafka Broker . . Producer . Consumer . . Producer Kafka Broker

10. Coordination Handshake Store Consumed Offset Producer Zookeeper and Watch for Cluster Consumer 1 event (groupId1) Producer Kafka Broker (Partition 1) Consumer 2 . (groupId1) . Producer . . . Producer Kafka Broker (Partition 2) * Consumer 3 (groupId1) * Consumer 3 would not receive any data, as number of consumers are more than number of partitions.

11.  Filesystem Cache  Zero-copy transfer of messages  Batching of Messages  Batch Compression  Automatic Producer Load balancing.  Broker does not Push messages to Consumer, Consumer Polls messages from Broker.  And Some others.  Cluster formation of Broker/Consumer using Zookeeper, So on the fly more consumer, broker can be introduced. The new cluster rebalancing will be taken care by Zookeeper  Data is persisted in broker and is not removed on consumption (till retention period), so if one consumer fails while consuming, same message can be re-consume again later from broker.  Simplified storage mechanism for message, not for each message per consumer.

12. Producer Performance Consumer Performance Credit : http://research.microsoft.com/en-us/UM/people/srikanth/netdb11/netdb11papers/netdb11-final12.pdf

13. Credit: https://cwiki.apache.org/confluence/display/KAFKA/Powered+By

Apache kafka

Recommended

More Related Content

What's hot (20)

Similar to Apache kafka (20)

More from Rahul Jain (14)

Recently uploaded (20)

Apache kafka