Scribd
Upload
17 views16 pages

KAFKA PRESENTATION (1)

Uploaded by

rakeshnagula.eidiko
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
17 views16 pages

KAFKA PRESENTATION (1)

Uploaded by

rakeshnagula.eidiko
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
You are on page 1/ 16

Working With

Apache Kafka

Presented by Arif Jamil Ansari


Content:-
1. Introduction to Kafka
2. A brief Comparison between Kafka and MQ
3. Installing Apache kafka
4. Creating Topics
5. Publishing and Subscribing using Message
Flow
6. Testing Application wi
1. Introduction
Technically speaking, event streaming is the practice of capturing
data in real-time from event sources like databases, sensors, mobile
devices, cloud services, and software applications in the form of
streams of events; storing these event streams durably for later
retrieval; manipulating, processing, and reacting to the event streams
in real-time as well as retrospectively; and routing the event streams
to different destination technologies as needed. Event streaming
thus ensures a continuous flow and interpretation of data so that the
right information is at the right place, at the right time.
Applications of kafka

● To process payments and financial transactions in real-time, such as in stock exchanges,


banks, and insurances.
● To track and monitor cars, trucks, fleets, and shipments in real-time, such as in logistics and
the automotive industry.
● To continuously capture and analyze sensor data from IoT devices or other equipment, such
as in factories and wind parks.
● To collect and immediately react to customer interactions and orders, such as in retail, the
hotel and travel industry, and mobile applications.
● To monitor patients in hospital care and predict changes in condition to ensure timely
treatment in emergencies.
● To connect, store, and make available data produced by different divisions of a company.
● To serve as the foundation for data platforms, event-driven architectures, and
microservices.
An event records the fact that "something happened" in the world or in your business. It is also called
record or message in the documentation. When you read or write data to Kafka, you do this in the
form of events. Conceptually, an event has a key, value, timestamp, and optional metadata headers.
Here's an example event:

● Event key: "Alice"


● Event value: "Made a payment of $200 to Bob"
● Event timestamp: "Jun. 25, 2020 at 2:06 p.m."

Producers are those client applications that publish (write) events to Kafka, and consumers are those
that subscribe to (read and process) these events. In Kafka, producers and consumers are fully
decoupled and agnostic of each other, which is a key design element to achieve the high scalability
that Kafka is known for. For example, producers never need to wait for consumers. Kafka provides
various guarantees such as the ability to process events exactly-once.
Events are organized and durably stored in topics. Very simplified, a topic is similar to a folder in a
filesystem, and the events are the files in that folder. An example topic name could be
"payments". Topics in Kafka are always multi-producer and multi-subscriber: a topic can have
zero, one, or many producers that write events to it, as well as zero, one, or many consumers that
subscribe to these events. Events in a topic can be read as often as needed—unlike traditional
messaging systems, events are not deleted after consumption. Instead, you define for how long
Kafka should retain your events through a per-topic configuration setting, after which old events
will be discarded. Kafka's performance is effectively constant with respect to data size, so storing
data for a long time is perfectly fine.

Topics are partitioned, meaning a topic is spread over a number of "buckets" located on different
Kafka brokers. This distributed placement of your data is very important for scalability because it
allows client applications to both read and write the data from/to many brokers at the same time.
When a new event is published to a topic, it is actually appended to one of the topic's partitions.
Events with the same event key (e.g., a customer or vehicle ID) are written to the same partition,
and Kafka guarantees that any consumer of a given topic-partition will always read that partition's
events in exactly the same order as they were written.
The Google Translate app
can repeat anything you
say in up to NINETY
LANGUAGES from Tip

German and Japanese to Don’t wait till the end of


the presentation to give
Czech and Zulu the bottom line.
Reveal your product or
idea (in this case a
translation app) up front.
2.A brief Comparison between Kafka and MQ
https://hevodata.com/learn/mq-vs-kafka/#:~:text=sort%20of%20communication.-,I
BM%20MQ%20vs%20Kafka%3A%20Use%20Cases,connectors%20and%20provide
s%20stream%20processing.

3.Installing Apache kafka


Steps to Install Apache Kafka
1. Go to Apache Kafka official website or download it here
“https://downloads.apache.org/kafka/3.3.1/kafka_2.13-3.3.1.tgz”
2. After completing downloads extract the folder into C drive.
3. Open the folder and go to this file path
"C:\kafka\kafka_2.13-3.2.3\config\zookeeper.properties" open it with notepad ++ editor and
Look for dataDir Give your C drive kafka folder path here and append it with /zookeeper
4. Again Go to "C:\kafka\kafka_2.13-3.2.3\config\server.properties" this path
and open it and give kafka folder path append it with /kafka-logs Like this
Configuring Kafka

Open cmd and use this command

.\bin\windows\zookeeper-server-start.bat .\config\zookeeper.properties

Open cmd and use this command

.\bin\windows\kafka-server-start.bat .\config\server.properties

Story for illustration purposes only


Creating Topics
To create Topic use this use this Command

.\bin\windows\kafka-topics.bat --create --topic Topic1 --bootstrap-server


localhost:9092

.\bin\windows\kafka-topics.bat --create --topic Topic2 --bootstrap-server


localhost:9092

To describe Topic use this Command

.\bin\windows\kafka-topics.bat --describe --topic Topic1 --bootstrap-server


localhost:9092

.\bin\windows\kafka-topics.bat --describe --topic Topic2 --bootstrap-server


localhost:9092
To Write message use this Command

.\bin\windows\kafka-console-producer.bat --topic Topic1--bootstrap-server


localhost:9092

To read Message use this Command

.\bin\windows\kafka-console-consumer.bat --topic quickstart-start


--from-beginning --bootstrap-server localhost:9092
Publishing and Subscribing using Message Flow
Thank You

Source: theguardian.com

You might also like