Event driven Architecture with Kafka

Program Details
Topic Apache Kafka & Streaming Distributed Messaging Training
Scope Confluence Kafka Enterprise 5.x Kafka
2.x Version
Level Intermediate and Advanced
Languages Java 1.8 64 bit

Lab Setup

Host Machine 64bit Windows/Linux/Mac [Linux is preferred]

Virtual 64bit Linux OS [Mint/Ubuntu OS Based]
Machine
Host Machine 16 GB if developer wants to use Virtual Machine
RAM
RAM 8 GB if developer can use Linux/Mac machine directly without Host Machine

Setup 16 GB RAM
Requirements 40-50 GB GB Free harddrive

OS: Windows 10/Ubuntu Desktop/Mac

Cloud Trainer provides cloud machine for each participant with 160 GB SSD Harddrive, 4
Machine vCPU and 8 GB RAM for participants, cost about 5 Euros per participant. The VM shall
be only for training period. Maximum 15 VMs provided, team need to share
themselves.

For Cloud, the setup shall be done by trainer.

Lap Setup 1. Java 1.8 Java SDK from
https://www.oracle.com/technetwork/java/javase/downloads/jdk8downloads-
2133151.html
2. Intellij Community Edition for Java Development
https://www.jetbrains.com/idea/download/

3. https://www.putty.org/ or https://gitforwindows.org/
4. https://www.confluent.io/download download the Confluent platform free,
need to provide email id.
Event driven Architecture with Kafka

Participants Pre-requites

JDK Good Knowledge on Using JDK, Compiler Setup & Java.

Good Working knowledge on Java /Exposure to Reactive
Programming for Streaming

Maven & POM XML Good Knowledge on Maven & POM XML files handling
Highspeed Internet Access to Highspeed Internet
Editor Good expertise in using IntelliJ/Eclipse Editor
Preferred, but not High level understanding of AMQP
mandatory Understanding Producer and Consumer Design Patterns
Message Queuing
Distributed Systems
Stream vs Batch Processing
SSH Gitbash for windows
SSH client for Linux/Mac/Windows

Day 1
Event driven Architecture with Kafka

Introduction Introduction

Kafka Introduction
Kafka Features
Kafka vs AMQP
Kafka Features
Kafka Real Use Cases

Enterprise Software Digital Software Transformation From Monolithic to

Architecture MicroServices to Serverless Applications

Event-Driven Architecture ( EDA )

- MicroServices
- Serverless
- FaaS
- Streaming
- Event Sourcing
- CQRS

Reactive Manifesto

Kafka Case Studies - MicroServices Log Analysis

- Invoice Processing
- Distributed Transaction Management with SAGA
Pattern
- Data Migration between databases

Kafka Architecture Elements of Kafka [Consumer, Producer, Broker, ZooKeeper,

Cluster]
Kafka Architecture
Kafka Overall Messaging Architecture
Kafka Streaming Architecture
Kafka Consumer and Producer Architecture
Kafka Persistence Architecture
Kafka Clusters
Kafka Consumer Workers Group
Event driven Architecture with Kafka

Kafka Distribution Apache Kafka

Confluence Kafka Open Source
Confluence Kafka Enterprise
Kafka Setup Java JDK
Kafka Setup
[Hands-on] Zookeeper Configuration
Single Broker Configuration
Understanding build-in tools, Kafka server, test publisher, test
consumer
Configuration Kafka Single Node Broker

Start ZooKeeper
Start Kafka Server
Using Example console consumer and publisher

Broker, Consumer, Producer Configuration

Zookeeper connection
Broker ID
Logs
Port

Consumer Configuration
Producer Configuration
Kafka Administrator List all topics
Commands List all brokers Create Topics
Change Topics
[Hands-on] Delete Topics
Topic Description

Topics & Partitions Understanding Topics

Understanding Partitions
Understanding Replications
Event driven Architecture with Kafka

Producer Kafka Producer Introduction

Kafka Producer API
[Hands-on] Writing First Producer
Kafka Producer Class

Build a Kafka Producer

Configure Producers

Kafka Topics, Partitions Configuration

Send messages to Kafka
Send messages Synchronously
Send Message Asynchronously and process the async
acknowledgement
Producing Keyed and Non-Keyed Messages
Serializers, Key/Value Serializer, ProducerRecord
Serialize Using Apache Avro, JSON

Working with Multiple Brokers

Consumer [Hands-on] Kafka Consumer Introduction
Consumers and Consumer Groups
Subscribe for Topics
Consumer Records
Achieving higher performance with Consumer group De-
serialize key and values

Consumer Groups and Partition Rebalance

Creating a Kafka Consumer
The Poll Loop
Configuring Consumers
Commits and Offsets
Rebalance Listeners
Consuming Records with Specific Offsets
Kafka Internal Architecture, Cluster Membership
Storages, Cluster, Partitions The Controller
in Depth Replication workflow, Replication Internals
Request Processing
Physical Storage
Reliability
Partition allocation and number of partitions
Broker Configuration
Using Producers in a Reliable System
Using Consumers in a Reliable System
Validating System Reliability
Performance Tuning in Kafka
Event driven Architecture with Kafka

Day 2

Cluster Architecture and Revising Cluster, Producer, Consumer, Broker, ZooKeeper

Workflow
Data Flow Between all actors in Kafka Systems

Creating Clusters with Multiple Brokers

Broker IDs and Master/Slave Brokers
Fail-over handling in clusters
Understanding Leader Broker
Balancing leadership
Cluster Administration and Configure Cluster
Configuration Replication
[Hands-on] High Availability
Multi-Cluster Architectures
Apache Kafka’s MirrorMaker

Topic Operations

Partition in Depth [Hands-on] Topics and Partitions

Understanding partition numbers
Default partitions
Explicit Partitions
Under the hood partitions
Replications Understanding Replications
Replication Factors and High Availability
Messages in Details Message Models
Message Serialize
[Hands-on] JSON handling
Message IDs

Day 3

Persistence [Hands-on] Persisting Messages on File System

Offline Consumer and Subscription handling
Durability of Messages
Failover and Persistence
Event driven Architecture with Kafka

Kafka Streaming Stream Processing

Stream-Processing Concepts
[Hands-on] Stream-Processing Design Patterns
Kafka Streams by Example
Kafka Streams: Architecture Overview
KStream
KTable
GlobalKTable
KStream to KTable
KTable to KSTream
Stateful/Stateless stream operation
KSQL Kafka and KSQL
KStream and KTable
Creating and running persistent and non-persistent quries
Zoo Keeper Administration Managing Zookeeper Cluster
Zookeeper lead, configurations
[Hands-on]

Day 4

Kafka Monitoring Considerations When Building Data Pipelines

Metric Basics
Kafka Broker Metrics
Client Monitoring
Lag Monitoring
End-to-End Monitoring
Kafka Connect APIs + Event Kafka Connect API overview Kafka
Sourcing + Elastic Search Connect
When to Use Kafka Connect?
[Hands-on] Kafka Connect Properties
Perform File source and sink using Kafka Connect
Kafka Connect vs Consumer/Producer model
Kafka Connect with JDBC [Requires PostgreSQL/MySQL]

Using Kafka with ElasticSearch

Event sourcing using Debizium with PostgreSQL
Kafka Connect architecture and design
Connectors, Configurations, Tasks, Workers
Stand-alone vs distributed mode
Event driven Architecture with Kafka

REST Proxy Managing REST Proxy Configuration

[Hands-on]
Kafka Connect Configuration Managing Kafka Connect
& Administration Managing Sink/Sources
Configuration and settings
[Hands-on]
Confluence Control Centre Introduction to Confluent Control Center
Managing Control Center and Dashboard Kafka
[Hands-on] Monitoring
Schema Registry Managing Schema Registry services
Configuration &
Administration

[Hands-on]
Kafka with Spring Spring-Kafka module
Configuring KafkaTemplate
Produce & Consume Messages with Kafka Template
Spring-cloud-stream with Kafka binder
KStream
KTable
Production Deployment
Configuration, cleaner installation