Table

of Contents
Introduction 1.1
Overview of Kafka 1.2

Kafka Services
AdminManager 2.1
Authorizer 2.2
Cluster 2.3
ConsumerCoordinator 2.4
ConsumerNetworkClient 2.5
DynamicConfigManager 2.6
Fetcher 2.7
GroupCoordinator 2.8
GroupMetadataManager 2.9
Kafka 2.10
KafkaApisAPI Request Handler 2.11
KafkaClient 2.12
NetworkClient 2.12.1
KafkaHealthcheck 2.13
KafkaServerStartableThin Management Layer over KafkaServer 2.14
KafkaServer 2.15
KafkaConfig 2.16
KafkaController 2.17
ControllerEventManager 2.17.1
ControllerEventThread 2.17.2
ControllerEvent 2.17.3
TopicDeletion Controller Event 2.17.3.1
KafkaMetricsReporter 2.18
KafkaRequestHandler 2.19
KafkaRequestHandlerPoolPool of Daemon KafkaRequestHandler Threads 2.20

1
KafkaScheduler 2.21
LogDirFailureHandler 2.22
LogManager 2.23
Metadata 2.24
MetadataCache 2.25
MetadataResponse 2.26
MetadataUpdater 2.27
DefaultMetadataUpdater 2.27.1
OffsetConfig 2.28
Partition 2.29
PartitionStateMachine 2.30
ReplicaManager 2.31
ReplicaStateMachine 2.32
ShutdownableThread 2.33
SocketServer 2.34
TopicDeletionManager 2.35
TransactionCoordinator 2.36
TransactionStateManager 2.37
ZkUtils 2.38

Features
Topic Replication 3.1
Topic Deletion 3.2
Kafka Controller Election 3.3

Architecture
BrokerKafka Server 4.1
Topics 4.2
Messages 4.3
Kafka Clients 4.4
Producers 4.4.1
KafkaProducer 4.4.1.1

2
 Sender 4.4.1.2
Consumers 4.4.2
KafkaConsumerMain Class For Kafka Consumers 4.4.2.1
Deserializer 4.4.2.2
ConsumerConfig 4.4.2.3
Consumer 4.4.2.4
ConsumerInterceptor 4.4.2.5
Clusters 4.5

Metrics
Sensor 5.1
MetricsReporter 5.2
ProducerMetrics 5.3
SenderMetrics 5.4
Kafka Tools 6.1
kafka-configs.sh 6.1.1
kafka-consumer-groups.sh 6.1.2
kafka-topics.sh 6.1.3
Properties 6.2
bootstrap.servers 6.2.1
client.id 6.2.2
enable.auto.commit 6.2.3
group.id 6.2.4
retry.backoff.ms 6.2.5
Logging 6.3
WorkerGroupMember 7.1
ConnectDistributed 7.2

Tips and Tricks

Gradle Tips 8.1
Zookeeper Tips 8.2
Kafka in Scala REPL for Interactive Exploration 8.3

3
Appendix
Further reading or watching 9.1

4
Introduction

Apache Kafka Notebook

Welcome to Apache Kafka Notebook!

Im Jacek Laskowski, an independent consultant who is passionate about Apache Spark,

Apache Kafka, Scala, sbt (with some flavour of Apache Mesos, Hadoop YARN, and
DC/OS). I lead Warsaw Scala Enthusiasts and Warsaw Spark meetups in Warsaw, Poland.

Contact me at [email protected] or @jaceklaskowski to discuss Apache Kafka and Apache

Spark opportunities, e.g. courses, workshops, mentoring or application development
services.

If you like the Apache Kafka notes you should seriously consider participating in my own,
very hands-on Spark Workshops.

This collections of notes (what some may rashly call a "book") serves as the ultimate place
of mine to collect all the nuts and bolts of using Apache Kafka. The notes aim to help me
designing and developing better products with Kafka. It is also a viable proof of my
understanding of Apache Kafka. I do eventually want to reach the highest level of mastery in
Apache Kafka.

Expect text and code snippets from a variety of public sources. Attribution follows.

5
Overview of Kafka

Overview of Kafka
Apache Kafka is an open source project for a distributed publish-subscribe messaging
system rethought as a distributed commit log.

Kafka stores messages in topics that are partitioned and replicated across multiple brokers
in a cluster. Producers send messages to topics from which consumers read.

Language Agnosticproducers and consumers use binary protocol to talk to a Kafka

cluster.

Messages are byte arrays (with String, JSON, and Avro being the most common formats). If
a message has a key, Kafka makes sure that all messages of the same key are in the same
partition.

Consumers may be grouped in a consumer group with multiple consumers. Each consumer
in a consumer group will read messages from a unique subset of partitions in each topic
they subscribe to. Each message is delivered to one consumer in the group, and all
messages with the same key arrive at the same consumer.

DurabilityKafka does not track which messages were read by each consumer. Kafka
keeps all messages for a finite amount of time, and it is consumers' responsibility to track
their location per topic, i.e. offsets.

It is worth to note that Kafka is often compared to the following open source projects:

1. Apache ActiveMQ and RabbitMQ given they are message broker systems, too.

2. Apache Flume for its ingestion capabilities designed to send data to HDFS and Apache
HBase.

6
AdminManager

AdminManager
AdminManager isFIXME

AdminManager is created exclusively when KafkaServer is started.

Figure 1. AdminManager
logIdent is [Admin Manager on Broker [brokerId]].

Table 1. AdminManagers Internal Properties (e.g. Registries and Counters)

Name Description
alterConfigPolicy

createTopicPolicy

topicPurgatory

Enable INFO , DEBUG , TRACE logging levels for kafka.server.AdminManager

logger to see what happens inside.
Add the following line to config/log4j.properties :
Tip
log4j.logger.kafka.server.AdminManager=TRACE, stdout

Refer to Logging.

Creating Topics createTopics Method

createTopics(
timeout: Int,
validateOnly: Boolean,
createInfo: Map[String, CreateTopicsRequest.TopicDetails],
responseCallback: (Map[String, ApiError]) => Unit): Unit

7
AdminManager

createTopics FIXME

createTopics is used exclusively when KafkaApis handles a

Note
CREATE_TOPICS request.

Creating AdminManager Instance

AdminManager takes the following when created:

KafkaConfig

Metrics

MetadataCache

ZkUtils

AdminManager initializes the internal registries and counters.

8
Authorizer

Authorizer
Authorizer isFIXME

configure Method

Caution FIXME

9
Cluster

Cluster
Caution FIXME

topics Method

Caution FIXME

availablePartitionsForTopic Method

Caution FIXME

10
ConsumerCoordinator

ConsumerCoordinator
ConsumerCoordinator is a AbstractCoordinator thatFIXME

ConsumerCoordinator is created exclusively when KafkaConsumer is created.

Figure 1. ConsumerCoordinator and KafkaConsumer

Table 1. ConsumerCoordinators Internal Properties (e.g. Registries and Counters)
Name Description

refreshCommittedOffsetsIfNeeded Method

FIXME

refreshCommittedOffsetsIfNeeded FIXME

Note refreshCommittedOffsetsIfNeeded is used whenFIXME

maybeLeaveGroup Method

FIXME

maybeLeaveGroup FIXME

Note maybeLeaveGroup is used whenFIXME

updatePatternSubscription Method

11
ConsumerCoordinator

FIXME

updatePatternSubscription FIXME

Note updatePatternSubscription is used whenFIXME

needRejoin Method

FIXME

needRejoin FIXME

Note needRejoin is used whenFIXME

timeToNextPoll Method

FIXME

timeToNextPoll FIXME

Note timeToNextPoll is used whenFIXME

poll Method

FIXME

poll FIXME

Note poll is used whenFIXME

commitOffsetsSync Method

FIXME

commitOffsetsSync FIXME

Note commitOffsetsSync is used whenFIXME

12
ConsumerCoordinator

commitOffsetsAsync Method

FIXME

commitOffsetsAsync FIXME

Note commitOffsetsAsync is used whenFIXME

maybeAutoCommitOffsetsNow Method

FIXME

maybeAutoCommitOffsetsNow FIXME

Note maybeAutoCommitOffsetsNow is used whenFIXME

addMetadataListener Method

FIXME

addMetadataListener FIXME

Note addMetadataListener is used whenFIXME

commitOffsetsSync Method

FIXME

commitOffsetsSync FIXME

Note commitOffsetsSync is used whenFIXME

fetchCommittedOffsets Method

FIXME

fetchCommittedOffsets FIXME

13
ConsumerCoordinator

Note fetchCommittedOffsets is used whenFIXME

Creating ConsumerCoordinator Instance

ConsumerCoordinator takes the following when created:

LogContext

ConsumerNetworkClient

Group ID

rebalanceTimeoutMs

sessionTimeoutMs

heartbeatIntervalMs

Collection of PartitionAssignors

Metadata

SubscriptionState

Metrics

Prefix of the metric group

Time

retryBackoffMs

autoCommitEnabled flag

autoCommitIntervalMs

ConsumerInterceptors

excludeInternalTopics flag

leaveGroupOnClose flag

ConsumerCoordinator initializes the internal registries and counters.

In the end, ConsumerCoordinator requests Metadata to update and addMetadataListener.

14
ConsumerNetworkClient

ConsumerNetworkClient
ConsumerNetworkClient is a high-level Kafka consumer thatFIXME

ConsumerNetworkClient is created for KafkaConsumer (and for AdminClient and

WorkerGroupMember ).

Table 1. ConsumerNetworkClients Internal Properties (e.g. Registries and Counters)

Name Description
pendingCompletion

Enable DEBUG logging level for

org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient logger to see what happens
inside.
Add the following line to config/log4j.properties :
Tip
log4j.logger.org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient=DEBUG, stdout

Refer to Logging.

wakeup Method

void wakeup()

wakeup prints out the following DEBUG message to the logs:

Received user wakeup

wakeup turns the internal wakeup flag on and requests KafkaClient to wakeup

Note wakeup is used whenFIXME

ensureFreshMetadata Method

Caution FIXME

send Method

15
ConsumerNetworkClient

Caution FIXME

pendingRequestCount Method

Caution FIXME

leastLoadedNode Method

Caution FIXME

poll Method

void poll(long timeout)

void poll(long timeout, long now, PollCondition pollCondition)
void poll(long timeout, long now, PollCondition pollCondition, boolean disableWakeup)
void poll(RequestFuture<?> future)
boolean poll(RequestFuture<?> future, long timeout)

poll FIXME

poll is used when:

KafkaConsumer polls once

ConsumerNetworkClient does pollNoWakeup, awaitMetadataUpdate,

awaitPendingRequests
Note
AbstractCoordinator does ensureCoordinatorReady and
joinGroupIfNeeded

ConsumerCoordinator commitOffsetsSync and fetchCommittedOffsets

Fetcher is requested for topic metadata or retrieveOffsetsByTimes

awaitMetadataUpdate Method

Caution FIXME

awaitPendingRequests Method

Caution FIXME

16
ConsumerNetworkClient

pollNoWakeup Method

void pollNoWakeup()

pollNoWakeup FIXME

pollNoWakeup is used when:

KafkaConsumer polls

AbstractCoordinator does maybeLeaveGroup

Note
HeartbeatThread runs

ConsumerCoordinator does commitOffsetsAsync

Creating ConsumerNetworkClient Instance

ConsumerNetworkClient takes the following when created:

LogContext

KafkaClient

Metadata

Time

retryBackoffMs

requestTimeoutMs

ConsumerNetworkClient initializes the internal registries and counters.

17
DynamicConfigManager

DynamicConfigManager
DynamicConfigManager isFIXME

DynamicConfigManager is created whenFIXME

startup Method

startup

startup FIXME

Note startup is used exclusively when KafkaServer starts up.

Creating DynamicConfigManager Instance

DynamicConfigManager takes the following when created:

DynamicConfigManager

DynamicConfigManager initializes the internal registries and counters.

18
Fetcher

Fetcher
Fetcher is created exclusively when KafkaConsumer is created.

Figure 1. Fetcher and KafkaConsumer

Table 1. Fetchers Internal Properties (e.g. Registries and Counters) (in alphabetical order)
Name Description

client
ConsumerNetworkClient that is given when Fetcher is
created.

Creating Fetcher Instance

Fetcher takes the following when created:

ConsumerNetworkClient

Minimum number of bytes

Maximum number of bytes

Maximum wait time

Fetch size

How many records to poll

Flag to check CRC or not

Deserializer for keys

Deserializer for values

Metadata

SubscriptionState

Metrics

FetcherMetricsRegistry

Time

Retry backoff in milliseconds

19
Fetcher

IsolationLevel

Fetcher initializes the internal registries and counters.

Fetcher registers itself with SubscriptionState as a listener to receive notifications about

FIXME

sendFetches Method

Caution FIXME

sendMetadataRequest Internal Method

RequestFuture<ClientResponse> sendMetadataRequest(MetadataRequest.Builder request)

Internally, sendMetadataRequest requests ConsumerNetworkClient for the least loaded node.

With the node, sendMetadataRequest requests ConsumerNetworkClient to send the request

to the node.

When no node was found, sendMetadataRequest returns a RequestFuture with

NoAvailableBrokersException .

sendMetadataRequest is used exclusively when Fetcher is requested for topic

Note
metadata.

beginningOffsets Method

Caution FIXME

retrieveOffsetsByTimes Method

Caution FIXME

Getting Topic Metadata getTopicMetadata Method

Map<String, List<PartitionInfo>> getTopicMetadata(MetadataRequest.Builder request, long

timeout)

20
Fetcher

Internally, getTopicMetadata sends the metadata request and requests

ConsumerNetworkClient to poll until it finishes successfully or timeout expires.

After poll finishes, getTopicMetadata takes the cluster information from

MetadataResponse .

When MetadataResponse is successful, getTopicMetadata takes topics (from Cluster ) and

requests Cluster for available partitions for every topic.

In the end, getTopicMetadata creates a collection of topic and partitions pairs.

Caution FIXME Describe the failure path

getTopicMetadata is used when:

Fetcher is requested to find metadata for all topics

Note
KafkaConsumer is requested to find partitions for a topic.

Finding Metadata for All Topics

getAllTopicMetadata Method

Map<String, List<PartitionInfo>> getAllTopicMetadata(long timeout)

getAllTopicMetadata gets topic metadata specifying no topics (which means all topics

available).

getAllTopicMetadata is used exclusively when KafkaConsumer requests

Note
metadata for all topics.

21
GroupCoordinator

GroupCoordinator
GroupCoordinator isFIXME

GroupCoordinator is created whenFIXME

Figure 1. GroupCoordinators Startup

logIdent is [GroupCoordinator [brokerId]].

GroupCoordinator uses isActive flag toFIXME

Creating GroupCoordinator Instance apply Factory

Method

Caution FIXME

Creating GroupCoordinator Instance

GroupCoordinator takes the following when created:

Broker ID

GroupConfig

OffsetConfig

GroupMetadataManager

DelayedOperationPurgatory[DelayedHeartbeat]

DelayedOperationPurgatory[DelayedJoin]

Time

GroupCoordinator initializes the internal registries and counters.

Starting Up (and GroupMetadataManager) startup

Method

22
GroupCoordinator

startup(enableMetadataExpiration: Boolean = true): Unit

startup first prints out the following INFO message to the logs:

INFO [GroupCoordinator [brokerId]]: Starting up. (kafka.coordinator.group.GroupCoordin

ator)

With enableMetadataExpiration input flag enabled, startup requests

GroupMetadataManager to enableMetadataExpiration.

startup turns isActive flag on.

In the end, startup prints out the following INFO message to the logs:

INFO [GroupCoordinator [brokerId]]: Startup complete. (kafka.coordinator.group.GroupCo

ordinator)

Note startup is used exclusively when KafkaServer starts up.

23
GroupMetadataManager

GroupMetadataManager
GroupMetadataManager isFIXME

GroupMetadataManager is created exclusively when GroupCoordinator is created.

Figure 1. Creating GroupMetadataManager

Table 1. GroupMetadataManagers Internal Properties (e.g. Registries and Counters)
Name Description
groupMetadataCache Pool of GroupMetadata by group ID

groupMetadataTopicPartitionCount

scheduler KafkaScheduler

enableMetadataExpiration Method

enableMetadataExpiration(): Unit

enableMetadataExpiration requests KafkaScheduler to start.

enableMetadataExpiration schedules delete-expired-group-metadata task that

cleanupGroupMetadata every offsetsRetentionCheckIntervalMs milliseconds.

enableMetadataExpiration is used exclusively when GroupCoordinator is

Note
started.

24
GroupMetadataManager

Creating GroupMetadataManager Instance

GroupMetadataManager takes the following when created:

Broker ID

ApiVersion

OffsetConfig

ReplicaManager

ZkUtils

Time

GroupMetadataManager initializes the internal registries and counters.

cleanupGroupMetadata Internal Method

cleanupGroupMetadata(): Unit (1)

cleanupGroupMetadata(deletedTopicPartitions: Option[Seq[TopicPartition]]): Unit

1. Calls the other cleanupGroupMetadata with empty deletedTopicPartitions collection

cleanupGroupMetadata takes the current time (using time) and for every GroupMetadata in

cache does the following:

1. FIXME

In the end, cleanupGroupMetadata prints out the following INFO message to the logs:

Removed [offsetsRemoved] expired offsets in [duration] milliseconds

cleanupGroupMetadata is used exclusively when GroupMetadataManager is

Note requested to enableMetadataExpiration (as delete-expired-group-metadata
task).

Getting Number of Partitions for __consumer_offsets Topic

getGroupMetadataTopicPartitionCount Internal
Method

getGroupMetadataTopicPartitionCount: Int

25
GroupMetadataManager

getGroupMetadataTopicPartitionCount requests ZkUtils for getTopicPartitionCount of

__consumer_offsets topic.

If not available, getGroupMetadataTopicPartitionCount requests OffsetConfig for

offsetsTopicNumPartitions.

getGroupMetadataTopicPartitionCount is used exclusively when

Note
GroupMetadataManager is requested for groupMetadataTopicPartitionCount.

26
Kafka

KafkaStandalone Command-Line
Application
kafka.Kafka is a standalone command-line application that starts a Kafka broker.

kafka.Kafka is started using kafka-server-start.sh shell script.

// Using sh -xv to trace kafka-server-start.sh

$ sh -xv ./bin/kafka-server-start.sh config/server.properties
...
exec $base_dir/kafka-run-class.sh $EXTRA_ARGS kafka.Kafka "$@"
+ exec ./bin/kafka-run-class.sh -name kafkaServer -loggc kafka.Kafka config/server.pro
perties
...

getPropsFromArgs Method

Caution FIXME

Starting Kafka Broker on Command Line main Method

main(args: Array[String]): Unit

main merges properties and creates a KafkaServerStartable .

main registers a JVM shutdown hook to shut down KafkaServerStartable .

Note main uses Javas Runtime.addShutdownHook to register the shutdown hook.

In the end, main starts the KafkaServerStartable and waits till it finishes.

main terminates the JVM with status 0 when KafkaServerStartable shuts down properly

and with status 1 in case of any exception.

Note main uses Javas System.exit to terminate a JVM.

Registering INFO Logging Signal Handlers (for TERM, INT

and HUP Signals) registerLoggingSignalHandler
Internal Method

27
Kafka

registerLoggingSignalHandler(): Unit

registerLoggingSignalHandler registers signal handlers for TERM , INT and HUP signals

so that, once received, it prints out the following INFO message to the logs:

Terminating process due to signal [signal]

$ jps -lm | grep -i kafka

79965 kafka.Kafka config/server.properties

// You could use "pkill -TERM -nf kafka" instead

$ kill -TERM 79965

// In the Kafka server's console

INFO Terminating process due to signal SIGTERM (kafka.Kafka$)

registerLoggingSignalHandler is used exclusively when a Kafka broker is

Note
started.

registerLoggingSignalHandler was added to Kafka 1.0.0 in KAFKA-5679; Add

Note
logging for broker termination due to SIGTERM or SIGINT.

28
KafkaApisAPI Request Handler

KafkaApisAPI Request Handler

KafkaApis handles API requests and routes them to their respective handlers.

KafkaApis is created exclusively when KafkaServer is started.

Figure 1. KafkaApis
Table 1. KafkaApiss API Keys and Handlers (in alphabetical order)
Key Handler
CREATE_TOPICS handleCreateTopicsRequest

Routing API Requests handle Method

handle(request: RequestChannel.Request): Unit

handle first prints out the following TRACE message to the logs:

Handling request:[request] from connection [id];securityProtocol:[protocol],principal:

[principal]

handle then finds the handler that is responsible for the apiKey (in the header of the input

request ) and routes the request along.

Note handle is used exclusively when KafkaRequestHandler thread is started.

Handling CREATE_TOPICS Request

handleCreateTopicsRequest Handler

handleCreateTopicsRequest(request: RequestChannel.Request): Unit

29
KafkaApisAPI Request Handler

handleCreateTopicsRequest FIXME

handleCreateTopicsRequest checks whether KafkaController is activeFIXME

handleCreateTopicsRequest authorizes the Create operation for ClusterResource FIXME

In the end, handleCreateTopicsRequest requests AdminManager to create the topics.

Note handleCreateTopicsRequest is used to handle CREATE_TOPICS

authorize Internal Method

Caution FIXME

handleCreatePartitionsRequest Handler

handleCreatePartitionsRequest(request: RequestChannel.Request): Unit

handleCreatePartitionsRequest FIXME

Note handleCreatePartitionsRequest is used whenFIXME

handleDeleteTopicsRequest Handler

handleDeleteTopicsRequest(request: RequestChannel.Request): Unit

handleDeleteTopicsRequest FIXME

Note handleDeleteTopicsRequest is used whenFIXME

handleControlledShutdownRequest Handler

handleControlledShutdownRequest(request: RequestChannel.Request): Unit

handleControlledShutdownRequest FIXME

Note handleControlledShutdownRequest is used whenFIXME

Creating KafkaApis Instance

30
KafkaApisAPI Request Handler

KafkaApis takes the following when created:

RequestChannel

ReplicaManager

AdminManager

GroupCoordinator

TransactionCoordinator

KafkaController

ZkUtils

Broker ID

KafkaConfig

MetadataCache

Metrics

Optional Authorizer

QuotaManagers

BrokerTopicStats

Cluster ID

Time

31
KafkaClient

KafkaClient
KafkaClient is the contract forFIXME

Note NetworkClient is the one and only implementation.

KafkaClient Contract

package org.apache.kafka.clients;

public interface KafkaClient extends Closeable {

void close(String nodeId);
long connectionDelay(Node node, long now);
boolean connectionFailed(Node node);
void disconnect(String nodeId);
boolean hasInFlightRequests();
boolean hasInFlightRequests(String nodeId);
boolean hasReadyNodes();
int inFlightRequestCount();
int inFlightRequestCount(String nodeId);
boolean isReady(Node node, long now);
Node leastLoadedNode(long now);
ClientRequest newClientRequest(String nodeId, AbstractRequest.Builder<?> requestBuil
der,
long createdTimeMs, boolean expectResponse);
ClientRequest newClientRequest(String nodeId, AbstractRequest.Builder<?> requestBuil
der, long createdTimeMs,
boolean expectResponse, RequestCompletionHandler call
back);
boolean ready(Node node, long now);
List<ClientResponse> poll(long timeout, long now);
void send(ClientRequest request, long now);
void wakeup();
}

Table 1. KafkaClient Contract

Method Description

Used when:
wakeup
FIXME

Used when:
poll ConsumerNetworkClient polls

FIXME

32
KafkaClient

33
NetworkClient

NetworkClient
NetworkClient is a KafkaClient thatFIXME

NetworkClient is created whenFIXME

wakeup Method

void wakeup()

Note wakeup is a part of KafkaClient Contract.

wakeup simply requests the internal Selectable to wakeup

Note wakeup is used whenFIXME

poll Method

List<ClientResponse> poll(long timeout, long now)

poll FIXME

Note poll is a part of KafkaClient Contract to..FIXME.

handleCompletedReceives Method

void handleCompletedReceives(List<ClientResponse> responses, long now)

handleCompletedReceives FIXME

Note handleCompletedReceives is used exclusively when NetworkClient polls.

Creating NetworkClient Instance

NetworkClient takes the following when created:

MetadataUpdater

Metadata

34
NetworkClient

Selectable

Client ID

maxInFlightRequestsPerConnection

reconnectBackoffMs

reconnectBackoffMax

socketSendBuffer

socketReceiveBuffer

requestTimeoutMs

Time

discoverBrokerVersions flag

ApiVersions

Sensor

LogContext

NetworkClient initializes the internal registries and counters.

35
KafkaHealthcheck

KafkaHealthcheck
KafkaHealthcheck registers the broker it runs on with Zookeeper (which in turn makes the

broker visible to other brokers that together can form a Kafka cluster).

KafkaHealthcheck is created and started when KafkaServer is requested to start up.

Figure 1. KafkaHealthcheck
Table 1. KafkaHealthchecks Internal Properties (e.g. Registries and Counters)
Name Description
sessionExpireListener SessionExpireListener

Creating KafkaHealthcheck Instance

KafkaHealthcheck takes the following when created:

Broker ID

Advertised endpoints

ZkUtils

Optional rack name

ApiVersion

KafkaHealthcheck initializes the internal registries and counters.

Starting Up startup Method

startup

startup requests ZkUtils to subscribeStateChanges with sessionExpireListener.

36
KafkaHealthcheck

In the end, startup registers the broker with Zookeeper.

Note startup is used exclusively when KafkaServer starts up.

Registering Broker in Zookeeper register Method

register(): Unit

register reads com.sun.management.jmxremote.port System property or defaults to -1 .

For every EndPoint with no host assigned (in advertisedEndpoints), register assigns the
fully-qualified domain name of the local host.

register then finds the first EndPoint with PLAINTEXT security protocol or creates an

empty EndPoint .

Tip Define EndPoint with PLAINTEXT security protocol for older clients to connect.

In the end, register requests ZkUtils to registerBrokerInZk for brokerId, the host and port
of the PLAINTEXT endpoint, the updated endpoints, the JMX port, the optional rack and
protocol version.

Note register makes a broker visible for other brokers to form a Kafka cluster.

register is used when KafkaHealthcheck starts up and handles a new

Note
session.

handleNewSession Method

Caution FIXME

37
KafkaServerStartableThin Management Layer over KafkaServer

KafkaServerStartableThin Management
Layer over KafkaServer
KafkaServerStartable is a thin management layer to manage a single KafkaServer instance.

Figure 1. KafkaServerStartable manages KafkaServer

KafkaServerStartable allows for the KafkaServer instance to be started, shut down and

waited for until shutdown.

Table 1. KafkaServerStartables Internal Properties (e.g. Registries and Counters)

Name Description
KafkaServer instance.
server
Created when KafkaServerStartable is created.

awaitShutdown Method

Caution FIXME

shutdown Method

Caution FIXME

Creating KafkaServerStartable Instance

KafkaServerStartable takes the following when created:

1. KafkaConfig

2. Collection of KafkaMetricsReporters

KafkaServerStartable initializes the internal registries and counters.

Creating KafkaServerStartable From Properties

fromProps Method

38
KafkaServerStartableThin Management Layer over KafkaServer

fromProps(serverProps: Properties): KafkaServerStartable

fromProps creates a KafkaServerStartable with a custom serverProps properties file.

Caution FIXME

fromProps is used when kafka.Kafka runs as a standalone command-line

Note
application

startup Method

startup(): Unit

startup requests the managed KafkaServer to start.

In case of any exceptions, startup exits the JVM with status 1 . You should see the
following FATAL message in the logs if that happens.

FATAL Exiting Kafka.

Note startup uses Javas System.exit to terminate a JVM.

Note startup is used when a Kafka Broker starts (on command line).

39
KafkaServer

KafkaServer
KafkaServer is a Kafka broker.

Figure 1. KafkaServers Startup and Auxiliary Services

KafkaServer registers itself in the JMX system under kafka.server.

Table 1. KafkaServers Internal Properties (e.g. Registries and Counters)

Name Description
adminManager AdminManager

apis KafkaApis

authorizer Optional Authorizer

brokerState BrokerState

_brokerTopicStats BrokerTopicStats

_clusterId Cluster ID

credentialProvider CredentialProvider

40
KafkaServer

dynamicConfigHandlers

dynamicConfigManager DynamicConfigManager

groupCoordinator GroupCoordinator

isStartingUp Flag forFIXME

kafkaController KafkaController

kafkaHealthcheck KafkaHealthcheck

kafkaScheduler KafkaScheduler withFIXME

logContext LogContext

logDirFailureChannel LogDirFailureChannel

logManager LogManager

metadataCache MetadataCache

replicaManager ReplicaManager

Collection of MetricsReporter
reporters
Used whenFIXME

requestHandlerPool KafkaRequestHandlerPool

socketServer SocketServer

transactionCoordinator TransactionCoordinator

quotaManagers QuotaManagers

shutdownLatch Javas java.util.concurrentCountDownLatch

startupComplete Flag forFIXME

zkUtils ZkUtils

Getting Broker ID and Initial Offline Directories

getBrokerIdAndOfflineDirs Internal Method

41
KafkaServer

Caution FIXME

getOrGenerateClusterId Internal Method

Caution FIXME

Connecting to Zookeeper initZk Internal Method

Caution FIXME

checkpointBrokerId Internal Method

Caution FIXME

notifyClusterListeners Method

Caution FIXME

Creating KafkaServer Instance

KafkaServer takes the following when created:

KafkaConfig

Time (defaults to Time.SYSTEM )

Optional thread name prefix

A collection of KafkaMetricsReporters (defaults to no reporters)

Caution FIXME

Note KafkaServer is created when KafkaServerStartable is created.

Starting Auxiliary Services startup Method

startup(): Unit

startup starts a single Kafka server.

Internally, startup first prints out the following INFO message to the logs:

42
KafkaServer

INFO starting (kafka.server.KafkaServer)

startup sets BrokerState as Starting .

startup requests KafkaScheduler to start.

startup connects to Zookeeper (and initializes ZkUtils).

startup getOrGenerateClusterId (that is recorded as cluster id).

You should see the following INFO message in the logs:

INFO Cluster ID = [clusterId] (kafka.server.KafkaServer)

startup gets broker id and initial offline directories.

startup creates the LogContext with [KafkaServer id=[brokerId]] prefix.

startup creates and configures metrics.

1. Requests KafkaConfig for configured instances of metric reporters

2. Adds a JmxReporter (with kafka.server prefix)

3. Creates the MetricConfig

4. Initializes Metrics internal registry

startup registers broker topic metrics (by initializing BrokerTopicStats).

startup initializes QuotaManagers.

startup notifies cluster resource listeners (i.e. KafkaMetricsReporters and the configured

instances of metric reporters).

startup creates the LogDirFailureChannel

startup creates the LogManager and requests it to start up.

startup creates the MetadataCache (for the broker ID).

startup creates the CredentialProvider (per sasl.enabled.mechanisms property).

startup creates the SocketServer (for KafkaConfig, Metrics and CredentialProvider) and

requests it to start up.

startup creates the ReplicaManager and requests it to start up.

43
KafkaServer

startup creates the KafkaController (for KafkaConfig, ZkUtils, Metrics and the optional

threadNamePrefix) and requests it to start up.

startup creates the AdminManager (for KafkaConfig, Metrics, MetadataCache and

ZkUtils).

startup creates the GroupCoordinator (for KafkaConfig, ZkUtils and ReplicaManager) and

requests it to start up.

startup creates the TransactionCoordinator (for KafkaConfig, ReplicaManager, a new

dedicated KafkaScheduler with transaction-log-manager- thread name prefix, ZkUtils,

Metrics and MetadataCache) and requests it to start up.

startup creates a Authorizer (if defined using authorizer.class.name property) and

configures it.

startup creates the KafkaApis (for SocketServer, ReplicaManager, AdminManager,

GroupCoordinator, TransactionCoordinator, KafkaController, ZkUtils, broker ID,

KafkaConfig, MetadataCache, Metrics, Authorizer, QuotaManagers, BrokerTopicStats,
cluster ID).

Note At this point KafkaServer may start processing requests.

startup creates the KafkaRequestHandlerPool (for broker ID, SocketServer, KafkaApis

and num.io.threads).

startup starts the HTTP interface of mx4j (if configured).

startup creates the DynamicConfigManager (for ZkUtils and dynamicConfigHandlers) and

requests it to start up.

startup configures the advertised listeners (if defined).

startup creates the KafkaHealthcheck (for broker ID, the advertised listeners, ZkUtils,

broker.rack and inter.broker.protocol.version Kafka properties) and requests it to start up.

startup checkpoints the broker ID.

startup sets BrokerState as RunningAsBroker , creates the CountDownLatch, enables the

startupComplete flag, disables isStartingUp flag

startup registers AppInfo as an MBean with the MBean server as kafka.server:type=app-

info,id=[brokerId] .

In the end, you should see the following INFO message in the logs:

INFO [Kafka Server [brokerId]], started (kafka.server.KafkaServer)

44
KafkaServer

The INFO message above uses so-called log ident with the value of
Note broker.id property and is always in the format [Kafka Server [brokerId]],
after a Kafka server has fully started.

Note startup is used exclusively when KafkaServerStartable starts up.

45
KafkaConfig

KafkaConfig
KafkaConfig is the configuration of a Kafka server and the services.

Table 1. KafkaConfigs Configuration Values (in alphabetical order)

Value Kafka Property
deleteTopicEnable delete.topic.enable

hostName host.name

listeners listeners (see getListeners)

numNetworkThreads num.network.threads

port port

replicaLagTimeMaxMs

getAdvertisedListeners Internal Method

Caution FIXME

getConfiguredInstances Method

Caution FIXME

Creating Listeners getListeners Internal Method

getListeners: Seq[EndPoint]

getListeners creates the EndPoints if defined using listeners Kafka property or defaults to

PLAINTEXT://[hostName]:[port] (for hostName and port Kafka properties).

getListeners is used when KafkaConfig is created and for

Note
getAdvertisedListeners.

46
KafkaController

KafkaController
KafkaController is a Kafka service responsible for:

topic deletion

FIXME

KafkaController uses listeners as a notification system to monitor znodes in Zookeeper

and react accordingly.

Quoting Kafka Controller Internals:

In a Kafka cluster, one of the brokers serves as the controller, which is responsible for
managing the states of partitions and replicas and for performing administrative tasks
like reassigning partitions.

KafkaController is created and immediately started when KafkaServer starts up.

Figure 1. KafkaController
KafkaController is part of every Kafka broker, but only one KafkaController is active at all

times.

KafkaController emulates a state machine using controller events.

47
KafkaController

Table 1. KafkaControllers Controller Events

Event ControllerState process Handler
BrokerChange BrokerChange

1. Assigns the current controller ID

as the input newControllerId
2. (only when the broker is no
ControllerChange
longer an active controller
ControllerChange Resigns as the active controller
newControllerId : Int
Similar to Reelect event
with the only difference
Note
that it does not trigger
election

ControlledShutdown

ID
ControlledShutdown
controlledShutdownCallback :
Try[Set[TopicAndPartition]]
Unit

1. Assigns the current controller ID

as activeControllerId

Reelect ControllerChange
2. (only when the broker is no
longer an active controller
Resigns as the active controller
3. elect

1. registerSessionExpirationListener
Startup ControllerChange 2. registerControllerChangeListener
3. elect

logIdent is [Controller id=[brokerId]].

48
KafkaController

Table 2. KafkaControllers Internal Properties (e.g. Registries and Counters)

Name Description

The ID of the active KafkaController

activeControllerId
Initialized to -1

controllerContext

ControllerEventManager for
eventManager controllerContext.stats.rateAndTimeMetrics and
updateMetrics listener

isrChangeNotificationListener IsrChangeNotificationListener

kafkaScheduler
KafkaScheduler with a single daemon thread with
prefix kafka-scheduler

partitionStateMachine PartitionStateMachine

replicaStateMachine ReplicaStateMachine

StateChangeLogger with the broker ID and

stateChangeLogger
inControllerContext flag enabled

topicDeletionManager TopicDeletionManager

49
KafkaController

Table 3. KafkaControllers Listeners

Listener Description

BrokerChangeListener for this KafkaController

brokerChangeListener
and eventManager

logDirEventNotificationListener LogDirEventNotificationListener

partitionModificationsListeners PartitionModificationsListener by name

PartitionReassignmentListener for this

partitionReassignmentListener
KafkaController and ControllerEventManager

PreferredReplicaElectionListener for this

preferredReplicaElectionListener
KafkaController and ControllerEventManager

TopicDeletionListener (for this KafkaController

and ControllerEventManager)
Registered in registerTopicDeletionListener when
topicDeletionListener KafkaController does onControllerFailover.

De-registered in deregisterTopicDeletionListener
when KafkaController resigns as the active
controller.

Enable WARN , INFO or DEBUG logging levels for

kafka.controller.KafkaController logger to see what happens inside.

Add the following line to config/log4j.properties :

Tip
log4j.logger.kafka.controller.KafkaController=DEBUG, stdout

Refer to Logging.

initiateReassignReplicasForTopicPartition
Method

initiateReassignReplicasForTopicPartition

initiateReassignReplicasForTopicPartition FIXME

Note initiateReassignReplicasForTopicPartition is used whenFIXME

deregisterPartitionReassignmentIsrChangeListen
50
KafkaController

deregisterPartitionReassignmentIsrChangeListen
ers Method

deregisterPartitionReassignmentIsrChangeListeners

deregisterPartitionReassignmentIsrChangeListeners FIXME

Note deregisterPartitionReassignmentIsrChangeListeners is used whenFIXME

resetControllerContext Method

resetControllerContext

resetControllerContext FIXME

Note resetControllerContext is used whenFIXME

deregisterBrokerChangeListener Method

deregisterBrokerChangeListener

deregisterBrokerChangeListener FIXME

Note deregisterBrokerChangeListener is used whenFIXME

deregisterTopicChangeListener Method

deregisterTopicChangeListener

deregisterTopicChangeListener FIXME

Note deregisterTopicChangeListener is used whenFIXME

Resigning As Active Controller

onControllerResignation Method

onControllerResignation(): Unit

51
KafkaController

onControllerResignation starts by printing out the following DEBUG message to the logs:

Resigning

onControllerResignation unsubscribes from intercepting Zookeeper events for the following

znodes in order:

1. Child changes to /isr_change_notification znode

2. Data changes to /admin/reassign_partitions znode

3. Data changes to /admin/preferred_replica_election znode

4. Child changes to /log_dir_event_notification znode

onControllerResignation requests TopicDeletionManager to reset.

onControllerResignation requests KafkaScheduler to shutdown.

onControllerResignation resets the following internal counters:

offlinePartitionCount

preferredReplicaImbalanceCount

globalTopicCount

globalPartitionCount

onControllerResignation deregisterPartitionReassignmentIsrChangeListeners.

onControllerResignation requests PartitionStateMachine to shutdown.

onControllerResignation deregisterTopicChangeListener.

onControllerResignation deregisterPartitionModificationsListener every listener in

partitionModificationsListeners.

onControllerResignation deregisterTopicDeletionListener.

onControllerResignation requests ReplicaStateMachine to shutdown.

onControllerResignation deregisterBrokerChangeListener.

onControllerResignation resetControllerContext.

In the end, onControllerResignation prints out the following DEBUG message to the logs:

Resigned

52
KafkaController

onControllerResignation is used when:

1. ControllerEventThread processes ControllerChange and Reelect controller

events
Note
2. triggerControllerMove
3. KafkaController shuts down

Unsubscribing from Child Changes to

/isr_change_notification ZNode
deregisterIsrChangeNotificationListener
Internal Method

deregisterIsrChangeNotificationListener(): Unit

deregisterIsrChangeNotificationListener prints out the following DEBUG message to the

logs:

De-registering IsrChangeNotificationListener

deregisterIsrChangeNotificationListener requests ZkUtils to unsubscribe from intercepting

changes to /isr_change_notification znode with IsrChangeNotificationListener.

deregisterIsrChangeNotificationListener is used exclusively when

Note
KafkaController resigns as the active controller.

Unsubscribing from Child Changes to

/log_dir_event_notification ZNode
deregisterLogDirEventNotificationListener
Internal Method

deregisterLogDirEventNotificationListener(): Unit

deregisterLogDirEventNotificationListener prints out the following DEBUG message to the

logs:

De-registering logDirEventNotificationListener

53
KafkaController

deregisterLogDirEventNotificationListener requests ZkUtils to unsubscribe from

intercepting changes to /log_dir_event_notification znode with

LogDirEventNotificationListener.

deregisterLogDirEventNotificationListener is used exclusively when

Note
KafkaController resigns as the active controller.

Unsubscribing from Data Changes to

/admin/preferred_replica_election ZNode
deregisterPreferredReplicaElectionListener
Method

deregisterPreferredReplicaElectionListener(): Unit

deregisterPreferredReplicaElectionListener requests ZkUtils to unsubscribe from

intercepting data changes to /admin/preferred_replica_election znode with

PreferredReplicaElectionListener.

deregisterPreferredReplicaElectionListener is used exclusively when

Note
KafkaController resigns as the active controller.

Unsubscribing from Data Changes to

/admin/reassign_partitions ZNode
deregisterPartitionReassignmentListener
Method

deregisterPartitionReassignmentListener(): Unit

deregisterPartitionReassignmentListener requests ZkUtils to unsubscribe from intercepting

data changes to /admin/reassign_partitions znode with PartitionReassignmentListener.

deregisterPartitionReassignmentListener is used exclusively when

Note
KafkaController resigns as the active controller.

triggerControllerMove Internal Method

triggerControllerMove(): Unit

triggerControllerMove FIXME

54
KafkaController

triggerControllerMove is used when:

1. KafkaController handleIllegalState
Note
2. KafkaController caught an exception while electing or becoming a
controller

handleIllegalState Internal Method

handleIllegalState(e: IllegalStateException): Nothing

handleIllegalState FIXME

handleIllegalState is used when KafkaController catches a

Note IllegalStateException in updateLeaderEpochAndSendRequest,
sendUpdateMetadataRequest and ControlledShutdown event.

sendUpdateMetadataRequest Method

sendUpdateMetadataRequest(): Unit

sendUpdateMetadataRequest FIXME

Note sendUpdateMetadataRequest is used whenFIXME

updateLeaderEpochAndSendRequest Internal Method

updateLeaderEpochAndSendRequest(): Unit

updateLeaderEpochAndSendRequest FIXME

Note updateLeaderEpochAndSendRequest is used whenFIXME

shutdown Method

shutdown(): Unit

shutdown FIXME

Note shutdown is used whenFIXME

55
KafkaController

updateMetrics Internal Method

Caution FIXME

onBrokerStartup Method

onBrokerStartup(newBrokers: Seq[Int]): Unit

onBrokerStartup FIXME

onBrokerStartup is used exclusively when KafkaController processes

Note
BrokerChange controller event.

elect Method

elect(): Unit

elect FIXME

Note elect is used when KafkaController enters Startup and Reelect states.

onControllerFailover Method

Caution FIXME

onControllerFailover is used exclusively when KafkaController is requested

Note
to elect.

isActive Method

isActive: Boolean

isActive says whether the activeControllerId equals the broker ID (from KafkaConfig).

Caution FIXME When could they be different?

Creating KafkaController Instance

KafkaController takes the following when created:

56
KafkaController

KafkaConfig

ZkUtils

Time

Metrics

Optional thread name prefix

KafkaController initializes the internal registries and counters.

Starting ControllerEventManager (and Putting Startup

Event in Event Queue) startup Method

startup(): Unit

startup puts Startup event at the end of the event queue of ControllerEventManager and

requests it to start.

Note startup is used exclusively when KafkaServer is started up.

Registering SessionExpirationListener To Control Session

Recreation registerSessionExpirationListener
Internal Method

registerSessionExpirationListener(): Unit

registerSessionExpirationListener requests ZkUtils to subscribe to state changes with a

SessionExpirationListener (with the KafkaController and ControllerEventManager).

SessionExpirationListener puts Reelect event on the event queue of

Note ControllerEventManager every time the Zookeeper session has expired and a
new session has been created.

registerSessionExpirationListener is used exclusively when Startup event is

Note
processed (after ControllerEventThread is started).

Registering ControllerChangeListener for /controller ZNode

Changes registerControllerChangeListener
Internal Method

57
KafkaController

registerControllerChangeListener(): Unit

registerControllerChangeListener requests ZkUtils to subscribe to data changes for

/controller znode with a ControllerChangeListener (with the KafkaController and

ControllerEventManager).

ControllerChangeListener emits:

1. ControllerChange event with the current controller ID (on the event queue
Note of ControllerEventManager ) every time the data of a znode changes

2. Reelect event when the data associated with a znode has been deleted

registerControllerChangeListener is used exclusively when Startup event is

Note
processed (after ControllerEventThread is started).

registerBrokerChangeListener Internal Method

registerBrokerChangeListener(): Option[Seq[String]]

registerBrokerChangeListener requests ZkUtils to subscribeChildChanges for /brokers/ids

path with BrokerChangeListener.

registerBrokerChangeListener is used exclusively when KafkaController does

Note
onControllerFailover.

Getting Active Controller ID (from JSON under /controller

znode) getControllerID Method

getControllerID(): Int

getControllerID returns the ID of the active Kafka controller that is associated with

/controller znode in JSON format or -1 otherwise.

Internally, getControllerID requests ZkUtils for data associated with /controller znode.

If available, getControllerID parses the data (being the current controller info in JSON
format) to extract brokerid field.

58
KafkaController

$ ./bin/zookeeper-shell.sh 0.0.0.0:2181
Connecting to 0.0.0.0:2181
Welcome to ZooKeeper!
...
get /controller
{"version":1,"brokerid":100,"timestamp":"1506197069724"}
cZxid = 0xf9
ctime = Sat Sep 23 22:04:29 CEST 2017
mZxid = 0xf9
mtime = Sat Sep 23 22:04:29 CEST 2017
pZxid = 0xf9
cversion = 0
dataVersion = 0
aclVersion = 0
ephemeralOwner = 0x15eaa3a4fdd000d
dataLength = 56
numChildren = 0

Otherwise, when no /controller znode is available, getControllerID returns -1 .

getControllerID is used when:

Note 1. Processing Reelect controller event

2. elect

Registering TopicDeletionListener for Child Changes to

/admin/delete_topics ZNode
registerTopicDeletionListener Internal Method

registerTopicDeletionListener(): Option[Seq[String]]

registerTopicDeletionListener requests ZkUtils to subscribeChildChanges to

/admin/delete_topics znode with TopicDeletionListener.

registerTopicDeletionListener is used exclusively when KafkaController

Note
does onControllerFailover.

De-Registering TopicDeletionListener for Child Changes to

/admin/delete_topics ZNode
deregisterTopicDeletionListener Internal
Method

59
KafkaController

deregisterTopicDeletionListener(): Unit

deregisterTopicDeletionListener requests ZkUtils to unsubscribeChildChanges to

/admin/delete_topics znode with TopicDeletionListener.

deregisterTopicDeletionListener is used exclusively when KafkaController

Note
resigns as the active controller.

60
ControllerEventManager

ControllerEventManager
ControllerEventManager isFIXME

ControllerEventManager is created when KafkaController is created.

ControllerEventManager is started when KafkaController is started up.

Figure 1. ControllerEventManager is Created and Started Alongside KafkaController

Table 1. ControllerEventManagers Internal Properties (e.g. Registries and Counters)
Name Description
Javas java.util.concurrent.LinkedBlockingQueue (i.e. an
queue
optionally-bounded blocking queue based on linked
nodes that orders elements in first-in-first-out fashion) of
ControllerEvents.

_state ControllerState with Idle being the initial state

thread
ControllerEventThread with controller-event-thread
thread name

Creating ControllerEventManager Instance

ControllerEventManager takes the following when created:

rateAndTimeMetrics collection of ControllerState and KafkaTimer

eventProcessedListener Procedure of ControllerEvent

ControllerEventManager initializes the internal registries and counters.

61
ControllerEventManager

Inserting Controller Event to Event Queue put Method

put(event: ControllerEvent): Unit

put inserts event at the tail of event queue.

Note put is used whenFIXME

Starting ControllerEventManager (and

ControllerEventThread) start Method

start(): Unit

start requests ControllerEventThread to do the work.

ControllerEventThread is a ShutdownableThread that, once started, triggers

Note
doWork() method.

Note start is used exclusively when KafkaController is started up.

62
ControllerEventThread

ControllerEventThread
ControllerEventThread is a ShutdownableThread that is started when

ControllerEventManager is started

Figure 1. ControllerEventThread is Started Alongside ControllerEventManager

When created, ControllerEventThread takes the name of the thread (which
ControllerEventManager sets as controller-event-thread).

"controller-event-thread" #44 prio=5 os_prio=31 tid=0x00007fac45730800 nid=0xad03 wait

ing on condition [0x0000000178b30000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000007bcb03938> (a java.util.concurrent.locks.Abst
ractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await
(AbstractQueuedSynchronizer.java:2039)
at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
at kafka.controller.ControllerEventManager$ControllerEventThread.doWork(Contro
llerEventManager.scala:48)
at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:64)

Processing ControllerEvents doWork Method

doWork(): Unit

63
ControllerEventThread

doWork takes and removes the head of event queue (waiting if necessary until an element

becomes available).

The very first event in the event queue is Startup that KafkaController puts
Note
when it is started.

doWork sets _state (of ControllerEventManager ) as the state of the event.

doWork finds the KafkaTimer for the state in rateAndTimeMetrics lookup table (of

ControllerEventManager ).

doWork processes the event (i.e. calls ControllerEvent.process method).

In the end, doWork passes the event to eventProcessedListener (of

ControllerEventManager ) and sets _state (of ControllerEventManager ) as Idle .

64
ControllerEvent

ControllerEvent
ControllerEvent is the contract of events in the lifecycle of KafkaController state machine

that, once emitted, triggers state change and the corresponding process action.

package kafka.controller

sealed trait ControllerEvent {

def state: ControllerState
def process(): Unit
}

ControllerEvent is a Scala sealed trait and so all the available events are in a
Note
single compilation unit (i.e. a file).

Table 1. ControllerEvent Contract

Method Description
ControllerState of the ControllerEventManager
state
Used when ControllerEventThread does the work

process
Used when ControllerEventThread does the work to
trigger an action associated with state change.

Table 2. Known ControllerEvents

ControllerEvent ControllerState
TopicDeletion TopicDeletion

65
TopicDeletion Controller Event

TopicDeletion Controller Event

TopicDeletion is a ControllerEvent that is executed on the active KafkaController (and

does nothing otherwise).

Note TopicDeletion uses delete.topic.enable Kafka property.

Note Topics to be deleted are created in /admin/delete_topics path in Zookeeper.

state is TopicDeletion .

process Method

process(): Unit

Note process is a part of ControllerEvent Contract.

Note process is executed on the active controller only (and does nothing otherwise).

process prints out the following DEBUG message to the logs:

Delete topics listener fired for topics [topicsToBeDeleted] to be deleted

process requests ControllerContext for allTopics and finds topics that are supposed to be

deleted, but are not available in the Kafka cluster.

If there are any non-existent topics, process prints out the following WARN message to the
logs and requests ZkUtils to deletePathRecursive /admin/delete_topics/[topicName] znode
for every topic in the list.

Ignoring request to delete non-existing topics [nonExistentTopics]

process branches off per delete.topic.enable Kafka property.

process with delete.topic.enable Enabled

With delete.topic.enable enabled (i.e. true ), process prints out the following INFO
message to the logs:

66
TopicDeletion Controller Event

Starting topic deletion for topics [topicsToBeDeleted]

process requests TopicDeletionManager to markTopicIneligibleForDeletion for topics to be

deleted with partitions in controllerContext.partitionsBeingReassigned list.

process requests TopicDeletionManager to enqueueTopicsForDeletion.

process with delete.topic.enable Disabled

With delete.topic.enable disabled (i.e. false ), process prints out the following INFO
message to the logs (for every topic):

Removing /admin/delete_topics/[topicName] since delete topic is disabled

process requests ZkUtils to deletePath /admin/delete_topics/[topicName] znode (for every

topic).

67
KafkaMetricsReporter

KafkaMetricsReporter
Caution FIXME

68
KafkaRequestHandler

KafkaRequestHandler
KafkaRequestHandler is a thread of execution (i.e. Javas Runnable) that handles Kafka

requests (i.e. ShutdownRequests directly while the others routes to KafkaApis).

KafkaRequestHandler is createdFIXME

logIdent is [Kafka Request Handler [id] on Broker [brokerId]].

Starting Thread run Method

run(): Unit

Caution FIXME

Note run is used when KafkaRequestHandlerPool is created.

Creating KafkaRequestHandler Instance

KafkaRequestHandler takes the following when created:

ID

Broker ID

Aggregate Idle Meter

Total number of handler threads

RequestChannel

KafkaApis

Time

KafkaRequestHandler initializes the internal registries and counters.

69
KafkaRequestHandlerPoolPool of Daemon KafkaRequestHandler Threads

KafkaRequestHandlerPoolPool of Daemon
KafkaRequestHandler Threads
KafkaRequestHandlerPool is a pool of daemon kafka-request-handler threads that are

started immediately when KafkaRequestHandlerPool is created.

Figure 1. KafkaRequestHandlerPool and KafkaRequestHandler Threads

The number of kafka-request-handler threads is controlled by
Note
num.network.threads Kafka property and defaults to 3 .

KafkaRequestHandlerPool is created exclusively when KafkaServer is started.

logIdent is [Kafka Request Handler on Broker [brokerId]].

Table 1. KafkaRequestHandlerPools Internal Properties (e.g. Registries and Counters)

Name Description
aggregateIdleMeter

runnables Collection of KafkaRequestHandler

shutdown Method

Caution FIXME

Creating KafkaRequestHandlerPool Instance

KafkaRequestHandlerPool takes the following when created:

Broker ID

RequestChannel

KafkaApis

70
KafkaRequestHandlerPoolPool of Daemon KafkaRequestHandler Threads

Time

Number of threads (i.e. instances of KafkaRequestHandlers as defined by

numNetworkThreads property)

KafkaRequestHandlerPool initializes the internal registries and counters and starts

numThreads daemon kafka-request-handler threads (as registered in runnables).

71
KafkaScheduler

KafkaScheduler
KafkaScheduler is a Scheduler to schedule tasks in Kafka.

Table 1. KafkaSchedulers Internal Properties (e.g. Registries and Counters)

Name Description

Javas ScheduledThreadPoolExecutor used to schedule

tasks.
executor
Initialized when KafkaScheduler starts up and shut down
when KafkaScheduler shuts down.

Enable INFO or DEBUG logging levels for kafka.utils.KafkaScheduler logger to

see what happens in KafkaScheduler .
Add the following line to config/log4j.properties :
Tip
log4j.logger.kafka.utils.KafkaScheduler=DEBUG, stdout

Refer to Logging.

Starting Up startup Method

startup(): Unit

Note startup is a part of Scheduler contract.

When startup is executed, you should see the following DEBUG message in the logs:

DEBUG Initializing task scheduler. (kafka.utils.KafkaScheduler)

startup initializes executor with threads threads. The name of the threads is in format of

threadNamePrefix followed by schedulerThreadId , e.g. kafka-scheduler-0

Note threads and threadNamePrefix are defined when KafkaScheduler is created.

If KafkaScheduler is already started, startup throws a IllegalStateException with the

message:

This scheduler has already been started!

72
KafkaScheduler

Creating KafkaScheduler Instance

Caution FIXME

shutdown Method

Caution FIXME

ensureRunning Internal Method

Caution FIXME

Scheduling Tasks schedule Method

schedule(name: String, fun: () => Unit, delay: Long, period: Long, unit: TimeUnit): Un
it

Note schedule is a part of Scheduler contract to schedule tasks.

When schedule is executed, you should see the following DEBUG message in the logs:

DEBUG Scheduling task [name] with initial delay [delay] ms and period [period] ms. (ka
fka.utils.KafkaScheduler)

schedule uses Javas java.util.concurrent.TimeUnit to convert delay and

Note
period to milliseconds.

schedule first makes sure that KafkaScheduler is running (which simply means that the

internal executor has been initialized).

schedule creates an execution thread for the input fun .

For positive period , schedule schedules the thread every period after the initial delay .
Otherwise, schedule schedules the thread once.

schedule uses the internal executor to schedule fun using

ScheduledThreadPoolExecutor.scheduleAtFixedRate and
Note
ScheduledThreadPoolExecutor.schedule for periodic and one-off executions,
respectively.

Whenever the thread is executed, and before fun gets triggerred, you should see the
following TRACE message in the logs:

73
KafkaScheduler

Beginning execution of scheduled task '[name]'.

After the execution thread is finished, you should see the following TRACE message in the
logs:

Completed execution of scheduled task '[name]'.

In case of any exceptions, the execution thread catches them and you should see the
following ERROR message in the logs:

Uncaught exception in scheduled task '[name]'

Scheduler Contract

trait Scheduler {
def startup(): Unit
def shutdown(): Unit
def isStarted: Boolean
def schedule(name: String, fun: () => Unit, delay: Long = 0, period: Long = -1, unit
: TimeUnit = TimeUnit.MILLISECONDS)
}

Table 2. Scheduler Contract

Method Description
schedule Schedules a task

74
LogDirFailureHandler

LogDirFailureHandler
LogDirFailureHandler isFIXME

start Method

Caution FIXME

75
LogManager

LogManager
Caution FIXME

startup Method

Caution FIXME

76
Metadata

Metadata
Metadata FIXME

Metadata is created whenFIXME

Table 1. Metadatas Internal Properties (e.g. Registries and Counters)

Name Description

Cluster
cluster Empty when Metadata is created

Updated whenFIXME

The time (in millis) of the last successful update (and

failed update)

lastRefreshMs Used in timeToNextUpdate

Starts 0 when Metadata is created

Reset (to 0 ) in requestUpdateForNewTopics

FlagFIXME
Disabled (i.e. false ) when Metadata is created
needMetadataForAllTopics
Updated when Metadata is requested to set state to
indicate that metadata for all topics in Kafka cluster is
required

Enable DEBUG or TRACE logging levels for org.apache.kafka.clients.Metadata

logger to see what happens inside.
Add the following line to config/tools-log4j.properties :
Tip
log4j.logger.org.apache.kafka.clients.Metadata=TRACE, stdout

Refer to Logging.

timeToNextUpdate Method

synchronized long timeToNextUpdate(long nowMs)

timeToNextUpdate FIXME

77
Metadata

timeToNextUpdate is used when:

ConsumerNetworkClient ensureFreshMetadata
Note
DefaultMetadataUpdater (of NetworkClient ) isUpdateDue and
maybeUpdate

add Method

synchronized void add(String topic)

add FIXME

Note add is used whenFIXME

requestUpdate Method

synchronized int requestUpdate()

requestUpdate FIXME

Note requestUpdate is used whenFIXME

Getting Current Cluster Information fetch Method

synchronized Cluster fetch()

fetch returns current cluster information.

Note fetch is used whenFIXME

Setting Topics to Maintain setTopics Method

Caution FIXME

Updating Cluster Metadata update Method

synchronized void update(Cluster cluster, Set<String> unavailableTopics, long now)

78
Metadata

update FIXME

update is used when:

KafkaConsumer is created
Note
KafkaProducer is created

DefaultMetadataUpdater handles MetadataResponse

Creating Metadata Instance

Metadata takes the following when created:

refreshBackoffMs

metadataExpireMs

allowAutoTopicCreation flag

topicExpiryEnabled flag

ClusterResourceListeners

Metadata initializes the internal registries and counters.

Conditionally Requesting Update For New Topics (for

KafkaConsumer) needMetadataForAllTopics
Method

synchronized void needMetadataForAllTopics(boolean needMetadataForAllTopics)

needMetadataForAllTopics requestUpdateForNewTopics when the input

needMetadataForAllTopics flag is enabled (i.e. true ) and the current

needMetadataForAllTopics is disabled (i.e. false ).

needMetadataForAllTopics sets needMetadataForAllTopics to be the input

needMetadataForAllTopics .

needMetadataForAllTopics is used when KafkaConsumer :

Subscribes to topics matching specified pattern (and

needMetadataForAllTopics flag is then enabled)
Note
Unsubscribes from topics (and needMetadataForAllTopics flag is then
disabled)

79
Metadata

requestUpdateForNewTopics Internal Method

synchronized void requestUpdateForNewTopics()

requestUpdateForNewTopics sets lastRefreshMs to 0 and requests update.

requestUpdateForNewTopics is used when Metadata :

add
Note
needMetadataForAllTopics
setTopics

80
MetadataCache

MetadataCache
MetadataCache isFIXME

81
MetadataResponse

MetadataResponse
cluster Method

Caution FIXME

82
MetadataUpdater

MetadataUpdater
MetadataUpdater Contract

package org.apache.kafka.clients;

interface MetadataUpdater {
List<Node> fetchNodes();
void handleDisconnection(String destination);
void handleCompletedMetadataResponse(RequestHeader requestHeader, long now, Metadata
Response metadataResponse);
boolean isUpdateDue(long now);
long maybeUpdate(long now);
void requestUpdate();
}

Table 1. MetadataUpdater Contract (in alphabetical order)

Method Description

handleCompletedMetadataResponse
Used exclusively when NetworkClient handles
completed receives

83
DefaultMetadataUpdater

DefaultMetadataUpdater
DefaultMetadataUpdater is a MetadataUpdater thatFIXME

Caution FIXME

isUpdateDue Method

Caution FIXME

maybeUpdate Method

Caution FIXME

handleCompletedMetadataResponse Callback Method

handleCompletedMetadataResponse(RequestHeader requestHeader, long now, MetadataRespons

e response)

handleCompletedMetadataResponse FIXME

handleCompletedMetadataResponse is a part of MetadataUpdater Contract

Note
to..FIXME.

84
OffsetConfig

OffsetConfig
OffsetConfig isFIXME

Table 1. OffsetConfigs Properties (in alphabetical order)

Property Default Value
offsetsTopicNumPartitions 50

85
Partition

Partition
A Kafka topic is spread across a Kafka cluster as a virtual group of one or more partitions.

A topic partition can be replicated across a Kafka cluster to one or more Kafka servers.

A topic partition has one partition leader and zero or more topic replicas.

Kafka producers publish messages to topic leaders as do Kafka consumers consume them
from.

Partition isFIXME

Table 1. Partitions Internal Properties (e.g. Registries and Counters)

Name Description
leaderReplicaIdOpt Optional leader replica ID

makeLeader Method

makeLeader(
controllerId: Int,
partitionStateInfo: LeaderAndIsrRequest.PartitionState,
correlationId: Int): Boolean

makeLeader FIXME

Note makeLeader is usedFIXME

makeFollower Method

makeFollower(
controllerId: Int,
partitionStateInfo: LeaderAndIsrRequest.PartitionState,
correlationId: Int): Boolean

makeFollower FIXME

Note makeFollower is usedFIXME

leaderReplicaIfLocal Method

86
Partition

leaderReplicaIfLocal: Option[Replica]

leaderReplicaIfLocal givesFIXME

Note leaderReplicaIfLocal is usedFIXME

maybeShrinkIsr Method

Caution FIXME

Creating Partition Instance

Partition takes the following when created:

Topic name

Partition ID

Time

ReplicaManager

isOffline flag (disabled by default)

Partition initializes the internal registries and counters.

87
PartitionStateMachine

PartitionStateMachine
PartitionStateMachine isFIXME

shutdown Method

FIXME

shutdown FIXME

Note shutdown is used whenFIXME

88
ReplicaManager

ReplicaManager
ReplicaManager is created and started when KafkaServer is requested to start up.

When started, ReplicaManager schedules isr-expiration and isr-change-propagation

recurring tasks.

ReplicaManager is a KafkaMetricsGroup .

Table 1. ReplicaManagers Internal Properties (e.g. Registries and Counters)

Name Description
allPartitions Pool of TopicPartition and Partitions

logDirFailureHandler LogDirFailureHandler

OfflinePartition

getLeaderPartitions Internal Method

getLeaderPartitions: List[Partition]

getLeaderPartitions gives the partitions from allPartitions that are not offline and their

leaderReplicaIfLocal property is defined.

Note getLeaderPartitions is used

maybePropagateIsrChanges Method

Caution FIXME

isr-expiration Task

Caution FIXME

isr-change-propagation Task

Caution FIXME

89
ReplicaManager

Creating ReplicaManager Instance

ReplicaManager takes the following when created:

KafkaConfig

Metrics

Time

ZkUtils

Scheduler

LogManager

isShuttingDown flag

ReplicationQuotaManager

BrokerTopicStats

MetadataCache

LogDirFailureChannel

DelayedOperationPurgatory[DelayedProduce]

DelayedOperationPurgatory[DelayedFetch]

DelayedOperationPurgatory[DelayedDeleteRecords]

Optional thread name prefix

ReplicaManager initializes the internal registries and counters.

Starting ReplicaManager (and Scheduling ISR-Related

Tasks) startup Method

startup(): Unit

startup requests Scheduler to schedule the ISR-related tasks:

1. isr-expiration

2. isr-change-propagation

startup then creates a LogDirFailureHandler and requests it to start.

90
ReplicaManager

startup uses Scheduler that was specified when ReplicaManager was

Note
created.

Note startup is used exclusively when KafkaServer starts up.

maybeShrinkIsr Internal Method

maybeShrinkIsr(): Unit

maybeShrinkIsr prints out the following TRACE message to the logs:

TRACE Evaluating ISR list of partitions to see which replicas can be removed from the
ISR

maybeShrinkIsr requests the partitions (from allPartitions pool that are not offline partitions)

to maybeShrinkIsr (with replicaLagTimeMaxMs setting).

maybeShrinkIsr is used exclusively to schedule isr-expiration recurring task

Note
when ReplicaManager starts up.

91
ReplicaStateMachine

ReplicaStateMachine
ReplicaStateMachine isFIXME

shutdown Method

FIXME

shutdown FIXME

Note shutdown is used whenFIXME

92
ShutdownableThread

ShutdownableThread
ShutdownableThread is the contract for non-daemon threads of execution.

ShutdownableThread contract expects that the objects implement doWork method.

def doWork(): Unit

Table 1. ShutdownableThreads Internal Properties (e.g. Registries and Counters)

Name Description
isRunning Flag that controls how long to execute run method.

shutdownLatch
Javas java.util.concurrentCountDownLatch with the
number of passes being 1

run Method

run(): Unit

Note run is a part of java.lang.Runnable that is executed when the thread is started.

run first prints out the following INFO message to the logs:

Starting

run then executes doWork method until isRunning flag is disabled.

In the end, run decrements the count of shutdownLatch and prints out the following INFO
message to the logs:

Stopped

93
SocketServer

SocketServer
SocketServer is a NIO socket server.

SocketServer is created exclusively when KafkaServer is started.

Figure 1. SocketServers Startup

Table 1. SocketServers Metrics (in kafka.network group)
Name Description
NetworkProcessorAvgIdlePercent

MemoryPoolAvailable

MemoryPoolUsed

94
SocketServer

Table 2. SocketServers Internal Properties (e.g. Registries and Counters) (in alphabetical
order)
Name Description
acceptors Acceptor threads per EndPoint

connectionQuotas ConnectionQuotas

EndPoints (aka listeners) per name (as configured

endpoints
using listeners Kafka property)

maxQueuedRequests

maxConnectionsPerIp

maxConnectionsPerIpOverrides

memoryPool

The number of processors per endpoint (as

numProcessorThreads configured using num.network.threads Kafka
property)

processors Processor threads (initially totalProcessorThreads)

requestChannel

totalProcessorThreads
Total number of processors, i.e.
numProcessorThreads for every endpoint

Creating Processor newProcessor Internal Method

Caution FIXME

Starting Up (and Auxiliary Services) startup Method

startup(): Unit

Internally, startup creates the ConnectionQuotas (with maxConnectionsPerIp and

maxConnectionsPerIpOverrides).

For every endpoint (in endpoints registry) startup does the following:

1. Creates up to numProcessorThreads number of Processors (for ConnectionQuotas and

MemoryPool)

95
SocketServer

2. Creates a Acceptor for the endpoint and processors

3. Records the Acceptor in acceptors internal registry

4. Starts a non-daemon thread for the Acceptor with the name as kafka-socket-acceptor-
[listenerName]-[securityProtocol]-[port] (e.g. kafka-socket-acceptor-

ListenerName(PLAINTEXT)-PLAINTEXT-9092 ) and waits until it has started fully

startup then registers metrics.

In the end, startup prints out the following INFO message to the logs:

INFO [SocketServer brokerId=[brokerID]] Started [number] acceptor threads

Note startup is used exclusively when KafkaServer starts up.

Creating SocketServer Instance

SocketServer takes the following when created:

KafkaConfig

Metrics

Time

CredentialProvider

SocketServer initializes the internal registries and counters.

96
TopicDeletionManager

TopicDeletionManager
TopicDeletionManager isFIXME

TopicDeletionManager is created exclusively when KafkaController is created.

Figure 1. TopicDeletionManager is Created Alongside KafkaController

TopicDeletionManager is controlled by delete.topic.enable Kafka property and does nothing

when it is turned off (i.e. false ).

logIdent is [Topic Deletion Manager [brokerId]].

Table 1. TopicDeletionManagers Internal Properties (e.g. Registries and Counters)

Name Description
partitionsToBeDeleted TopicAndPartitions to be deleted

topicsToBeDeleted The names of the topics to be deleted

topicsIneligibleForDeletion
The names of the topics that must not be deleted (i.e.
are ineligible for deletion)

Enable INFO logging level for kafka.controller.TopicDeletionManager logger to

see what happens inside.
Add the following line to config/log4j.properties :
Tip
log4j.logger.kafka.controller.TopicDeletionManager=INFO, stdout

Refer to Logging.

97
TopicDeletionManager

startReplicaDeletion Internal Method

Caution FIXME

enqueueTopicsForDeletion Method

Caution FIXME

failReplicaDeletion Method

Caution FIXME

Creating TopicDeletionManager Instance

TopicDeletionManager takes the following when created:

KafkaController

ControllerEventManager

TopicDeletionManager initializes the internal registries and counters.

markTopicIneligibleForDeletion Method

markTopicIneligibleForDeletion(topics: Set[String]): Unit

(only with delete.topic.enable Kafka property enabled) markTopicIneligibleForDeletion

computes the intersection between topicsToBeDeleted and the input topics sets and adds
the intersection to topicsIneligibleForDeletion set.

If there are any topics in the intersection, markTopicIneligibleForDeletion prints out the
following INFO message to the logs:

Halted deletion of topics [newTopicsToHaltDeletion]

98
TopicDeletionManager

markTopicIneligibleForDeletion is used when:

KafkaController initiateReassignReplicasForTopicPartition

Note TopicDeletion controller event is processed (for topics to be deleted with

partitions in controllerContext.partitionsBeingReassigned list)
TopicDeletionManager does startReplicaDeletion and failReplicaDeletion

Reseting reset Method

reset(): Unit

(only with delete.topic.enable Kafka property enabled) reset removes all elements from the
following internal registries:

topicsToBeDeleted

partitionsToBeDeleted

topicsIneligibleForDeletion

Note reset does nothing when delete.topic.enable Kafka property is false .

reset is used exclusively when KafkaController resigns as the active

Note
controller.

99
TransactionCoordinator

TransactionCoordinator
TransactionCoordinator isFIXME

TransactionCoordinator is created whenFIXME

startup Method

startup

startup FIXME

Note startup is used exclusively when KafkaServer starts up.

Creating TransactionCoordinator Instance

TransactionCoordinator takes the following when created:

TransactionCoordinator

TransactionCoordinator initializes the internal registries and counters.

100
TransactionStateManager

TransactionStateManager
TransactionStateManager isFIXME

getTransactionTopicPartitionCount Method

getTransactionTopicPartitionCount

getTransactionTopicPartitionCount FIXME

Note getTransactionTopicPartitionCount is used whenFIXME

101
ZkUtils

ZkUtils
ZkUtils isFIXME

ZkUtils is created whenFIXME

Quoting Nodes and ephemeral nodes from the Zookeeper documentation:

Unlike standard file systems, each node in a ZooKeeper namespace can

have data associated with it as well as children. It is like having a file-
system that allows a file to also be a directory. (ZooKeeper was designed
to store coordination data: status information, configuration, location
information, etc., so the data stored at each node is usually small, in the
byte to kilobyte range.) We use the term znode to make it clear that we are
talking about ZooKeeper data nodes.
Note Znodes maintain a stat structure that includes version numbers for data
changes, ACL changes, and timestamps, to allow cache validations and
coordinated updates. Each time a znodes data changes, the version
number increases. For instance, whenever a client retrieves data it also
receives the version of the data.
The data stored at each znode in a namespace is read and written
atomically. Reads get all the data bytes associated with a znode and a
write replaces all the data. Each node has an Access Control List (ACL)
that restricts who can do what.

Table 1. ZkUtilss ZNodes in Zookeeper

ZNode Description
ControllerPath /controller

Table 2. ZkUtilss Internal Properties (e.g. Registries and Counters) (in alphabetical order)
Name Description
persistentZkPaths

zkPath

deletePathRecursive Method

Caution FIXME

deletePath Method

102
ZkUtils

Caution FIXME

Creating ZkUtils Instance apply Factory Method

apply(
zkUrl: String,
sessionTimeout: Int,
connectionTimeout: Int,
isZkSecurityEnabled: Boolean): ZkUtils

apply FIXME

apply is used when:

Note 1. KafkaServer connects to Zookeeper

2. FIXME

Registering Listener for State Changes

subscribeStateChanges Method

subscribeStateChanges(listener: IZkStateListener): Unit

subscribeStateChanges requests ZkClient to subscribeStateChanges with the listener .

subscribeStateChanges is used when:

Note 1. KafkaController is requested to register a SessionExpirationListener

2. FIXME

Registering Listener for Child Changes

subscribeChildChanges Method

subscribeChildChanges(path: String, listener: IZkChildListener): Option[Seq[String]]

subscribeChildChanges FIXME

Note subscribeChildChanges is usedFIXME

103
ZkUtils

De-Registering Listener for Child Changes

unsubscribeChildChanges Method

unsubscribeChildChanges(path: String, childListener: IZkChildListener): Unit

unsubscribeChildChanges requests ZkClient to unsubscribeChildChanges for the input path

and childListener .

Note unsubscribeChildChanges is used whenFIXME

De-Registering Listener for Data Changes

unsubscribeDataChanges Method

unsubscribeDataChanges(path: String, dataListener: IZkDataListener): Unit

unsubscribeDataChanges requests ZkClient to unsubscribeDataChanges for the input path

and dataListener .

Note unsubscribeDataChanges is used whenFIXME

registerBrokerInZk Method

registerBrokerInZk(
id: Int,
host: String,
port: Int,
advertisedEndpoints: Seq[EndPoint],
jmxPort: Int,
rack: Option[String],
apiVersion: ApiVersion): Unit

registerBrokerInZk FIXME

registerBrokerInZk is used exclusively when KafkaHealthcheck is requested

Note
to register.

getTopicPartitionCount Method

getTopicPartitionCount(topic: String): Option[Int]

104
ZkUtils

getTopicPartitionCount FIXME

getTopicPartitionCount is used when:

1. GroupMetadataManager is requested for

Note getGroupMetadataTopicPartitionCount of __consumer_offsets topic

2. TransactionStateManager is requested for
getTransactionTopicPartitionCount of __transaction_state topic

Creating JSON with Broker ID controllerZkData

Method

controllerZkData(brokerId: Int, timestamp: Long): String

controllerZkData creates a JSON with the following fields:

"version":1

"brokerid":[brokerId]

"timestamp":[timestamp]

import kafka.utils._
scala> ZkUtils.controllerZkData(1, System.currentTimeMillis())
res0: String = {"version":1,"brokerid":1,"timestamp":"1506161225262"}

controllerZkData is used exclusively when KafkaController is requested for

Note
elect.

Creating ZkUtils Instance

ZkUtils takes the following when created:

ZkClient

ZkConnection

isSecure flag

ZkUtils initializes the internal registries and counters.

Reading Data Associated with ZNode

readDataMaybeNull Method

105
ZkUtils

readDataMaybeNull(path: String): (Option[String], Stat)

readDataMaybeNull requests ZkClient to readData from path znode.

readDataMaybeNull returns None (for Option[String] ) when path znode is not available.

106
Topic Replication

Topic Replication
Topic Replication is the process to offer fail-over capability for a topic.

Replication factor defines the number of copies of a topic in a Kafka cluster.

Replication factor can be defined at topic level.

./bin/kafka-topics.sh --create \
--topic my-topic \
--replication-factor 1 \ // <-- define replication factor
--partitions 1 \
--zookeeper localhost:2181

Replicas are distributed evenly among Kafka brokers in a cluster.

Leader replica isFIXME

Follower replica isFIXME

Producers always send requests to the broker that is the current leader replica for a topic
partition.

Data from producers is first saved to a commit log before consumers can find out that it is
available. It will only be visible to consumers when the followers acknowledge that they have
got the data and stored in their local logs.

107
Topic Deletion

Topic Deletion
Topic Deletion is a feature of Kafka that allows for deleting topics.

TopicDeletionManager is responsible for topic deletion.

Topic deletion is controlled by delete.topic.enable Kafka property that turns it on when

true .

Start a Kafka broker with broker ID 100 .

$ ./bin/kafka-server-start.sh config/server.properties \
--override delete.topic.enable=true \
--override broker.id=100 \
--override log.dirs=/tmp/kafka-logs-100 \
--override port=9192

Create remove-me topic.

$ ./bin/kafka-topics.sh --zookeeper localhost:2181 \

--create \
--topic remove-me \
--partitions 1 \
--replication-factor 1
Created topic "remove-me".

Use kafka-topics.sh --list to list available topics.

$ ./bin/kafka-topics.sh --zookeeper localhost:2181 --list

__consumer_offsets
remove-me

Use kafka-topics.sh --describe to list details for remove-me topic.

$ ./bin/kafka-topics.sh --zookeeper localhost:2181 --describe --topic remove-me

Topic:remove-me PartitionCount:1 ReplicationFactor:1 Configs:
Topic: remove-me Partition: 0 Leader: 100 Replicas: 100 Isr: 1
00

Note that the broker 100 is the leader for remove-me topic.

Stop the broker 100 and start another with broker ID 200 .

108
Topic Deletion

$ ./bin/kafka-server-start.sh config/server.properties \
--override delete.topic.enable=true \
--override broker.id=200 \
--override log.dirs=/tmp/kafka-logs-200 \
--override port=9292

Use kafka-topics.sh --delete to delete remove-me topic.

$ ./bin/kafka-topics.sh --zookeeper localhost:2181 --delete --topic remove-me

Topic remove-me is marked for deletion.
Note: This will have no impact if delete.topic.enable is not set to true.

List the topics.

$ ./bin/kafka-topics.sh --zookeeper localhost:2181 --list

__consumer_offsets
remove-me - marked for deletion

As you may have noticed, kafka-topics.sh --delete will only delete a topic if the topics
leader broker is available (and can acknowledge the removal). Since the broker 100 is down
and currently unavailable the topic deletion has only been recorded in Zookeeper.

$ ./bin/zkCli.sh -server localhost:2181

[zk: localhost:2181(CONNECTED) 0] ls /admin/delete_topics
[remove-me]

As long as the leader broker 100 is not available, the topic to be deleted remains marked
for deletion.

Start the broker 100 .

$ ./bin/kafka-server-start.sh config/server.properties \
--override delete.topic.enable=true \
--override broker.id=100 \
--override log.dirs=/tmp/kafka-logs-100 \
--override port=9192

With kafka.controller.KafkaController logger at DEBUG level, you should see the following
messages in the logs:

109
Topic Deletion

DEBUG [Controller id=100] Delete topics listener fired for topics remove-me to be dele
ted (kafka.controller.KafkaController)
INFO [Controller id=100] Starting topic deletion for topics remove-me (kafka.controlle
r.KafkaController)
INFO [GroupMetadataManager brokerId=100] Removed 0 expired offsets in 0 milliseconds.
(kafka.coordinator.group.GroupMetadataManager)
DEBUG [Controller id=100] Removing replica 100 from ISR 100 for partition remove-me-0.
(kafka.controller.KafkaController)
INFO [Controller id=100] Retaining last ISR 100 of partition remove-me-0 since unclean
leader election is disabled (kafka.controller.KafkaController)
INFO [Controller id=100] New leader and ISR for partition remove-me-0 is {"leader":-1,
"leader_epoch":1,"isr":[100]} (kafka.controller.KafkaController)
INFO [ReplicaFetcherManager on broker 100] Removed fetcher for partitions remove-me-0
(kafka.server.ReplicaFetcherManager)
INFO [ReplicaFetcherManager on broker 100] Removed fetcher for partitions (kafka.serv
er.ReplicaFetcherManager)
INFO [ReplicaFetcherManager on broker 100] Removed fetcher for partitions remove-me-0
(kafka.server.ReplicaFetcherManager)
INFO Log for partition remove-me-0 is renamed to /tmp/kafka-logs-100/remove-me-0.fe6d0
39ff884498b9d6113fb22a75264-delete and is scheduled for deletion (kafka.log.LogManager
)
DEBUG [Controller id=100] Delete topic callback invoked for org.apache.kafka.common.re
quests.StopReplicaResponse@8c0f4f0 (kafka.controller.KafkaController)
INFO [Controller id=100] New topics: [Set()], deleted topics: [Set()], new partition r
eplica assignment [Map()] (kafka.controller.KafkaController)
DEBUG [Controller id=100] Delete topics listener fired for topics to be deleted (kafk
a.controller.KafkaController)

The topic is now deleted. Use Zookeeper CLI tool to confirm it.

$ ./bin/zkCli.sh -server localhost:2181

[zk: localhost:2181(CONNECTED) 1] ls /admin/delete_topics
[]

110
Kafka Controller Election

Kafka Controller Election

Use the following setup with one Zookeeper server and two Kafka brokers to observe the
Kafka controller election.

Start Zookeeper server.

$ ./bin/zookeeper-server-start.sh config/zookeeper.properties
...
INFO binding to port 0.0.0.0/0.0.0.0:2181 (org.apache.zookeeper.server.NIOServerCnxnFa
ctory)

Add the following line to config/log4j.properties to enable DEBUG logging level for
kafka.controller.KafkaController logger.

log4j.logger.kafka.controller.KafkaController=DEBUG, stdout

Start a Kafka broker.

$ ./bin/kafka-server-start.sh config/server.properties \
--override broker.id=100 \
--override log.dirs=/tmp/kafka-logs-100 \
--override port=9192
...
INFO Registered broker 100 at path /brokers/ids/100 with addresses: EndPoint(192.168.1
.4,9192,ListenerName(PLAINTEXT),PLAINTEXT) (kafka.utils.ZkUtils)
INFO Kafka version : 1.0.0-SNAPSHOT (org.apache.kafka.common.utils.AppInfoParser)
INFO Kafka commitId : 852297efd99af04d (org.apache.kafka.common.utils.AppInfoParser)
INFO [KafkaServer id=100] started (kafka.server.KafkaServer)

Start another Kafka broker.

# Note the different properties

$ ./bin/kafka-server-start.sh config/server.properties \
--override broker.id=200 \
--override log.dirs=/tmp/kafka-logs-200 \
--override port=9292
...
INFO Registered broker 200 at path /brokers/ids/200 with addresses: EndPoint(192.168.1
.4,9292,ListenerName(PLAINTEXT),PLAINTEXT) (kafka.utils.ZkUtils)
INFO Kafka version : 1.0.0-SNAPSHOT (org.apache.kafka.common.utils.AppInfoParser)
INFO Kafka commitId : 852297efd99af04d (org.apache.kafka.common.utils.AppInfoParser)
INFO [KafkaServer id=200] started (kafka.server.KafkaServer)

111
Kafka Controller Election

Connect to Zookeeper using Zookeeper CLI (command-line interface). Use the official
distribution of Apache Zookeeper as described in Zookeeper Tips.

$ ./bin/zkCli.sh -server localhost:2181

Once connected, execute get /controller to get the data associated with /controller
znode where the active Kafka controller stores the controller ID.

[zk: localhost:2181(CONNECTED) 0] get /controller

{"version":1,"brokerid":100,"timestamp":"1506423376977"}
cZxid = 0x191
ctime = Tue Sep 26 12:56:16 CEST 2017
mZxid = 0x191
mtime = Tue Sep 26 12:56:16 CEST 2017
pZxid = 0x191
cversion = 0
dataVersion = 0
aclVersion = 0
ephemeralOwner = 0x15ebdd241840002
dataLength = 56
numChildren = 0

(optional) Clear the consoles of the two Kafka brokers so you have the election logs only.

Delete /controller znode and observe the controller election.

[zk: localhost:2181(CONNECTED) 2] delete /controller

You should see the following in the logs in the consoles of the two Kafka brokers.

DEBUG [Controller id=100] Resigning (kafka.controller.KafkaController)

DEBUG [Controller id=100] De-registering IsrChangeNotificationListener (kafka.controll
er.KafkaController)
DEBUG [Controller id=100] De-registering logDirEventNotificationListener (kafka.contro
ller.KafkaController)
INFO [Controller id=100] Resigned (kafka.controller.KafkaController)
DEBUG [Controller id=100] Broker 200 has been elected as the controller, so stopping t
he election process. (kafka.controller.KafkaController)

and

112
Kafka Controller Election

INFO Creating /controller (is it secure? false) (kafka.utils.ZKCheckedEphemeral)

INFO Result of znode creation is: OK (kafka.utils.ZKCheckedEphemeral)
INFO [Controller id=200] 200 successfully elected as the controller (kafka.controller.
KafkaController)
INFO [Controller id=200] Starting become controller state transition (kafka.controller
.KafkaController)
INFO [Controller id=200] Initialized controller epoch to 39 and zk version 38 (kafka.c
ontroller.KafkaController)
INFO [Controller id=200] Incremented epoch to 40 (kafka.controller.KafkaController)
DEBUG [Controller id=200] Registering IsrChangeNotificationListener (kafka.controller.
KafkaController)
DEBUG [Controller id=200] Registering logDirEventNotificationListener (kafka.controlle
r.KafkaController)
INFO [Controller id=200] Partitions being reassigned: Map() (kafka.controller.KafkaCon
troller)
INFO [Controller id=200] Partitions already reassigned: Set() (kafka.controller.KafkaC
ontroller)
INFO [Controller id=200] Resuming reassignment of partitions: Map() (kafka.controller.
KafkaController)
INFO [Controller id=200] Currently active brokers in the cluster: Set(100, 200) (kafka
.controller.KafkaController)
INFO [Controller id=200] Currently shutting brokers in the cluster: Set() (kafka.contr
oller.KafkaController)
INFO [Controller id=200] Current list of topics in the cluster: Set(my-topic2, NEW, my
-topic, my-topic1) (kafka.controller.KafkaController)
INFO [Controller id=200] List of topics to be deleted: (kafka.controller.KafkaControl
ler)
INFO [Controller id=200] List of topics ineligible for deletion: (kafka.controller.Ka
fkaController)
INFO [Controller id=200] Ready to serve as the new controller with epoch 40 (kafka.con
troller.KafkaController)
INFO [Controller id=200] Partitions undergoing preferred replica election: (kafka.con
troller.KafkaController)
INFO [Controller id=200] Partitions that completed preferred replica election: (kafka
.controller.KafkaController)
INFO [Controller id=200] Skipping preferred replica election for partitions due to top
ic deletion: (kafka.controller.KafkaController)
INFO [Controller id=200] Resuming preferred replica election for partitions: (kafka.c
ontroller.KafkaController)
INFO [Controller id=200] Starting preferred replica leader election for partitions (k
afka.controller.KafkaController)
INFO [Controller id=200] Starting the controller scheduler (kafka.controller.KafkaCont
roller)

113
BrokerKafka Server

BrokerKafka Server
Given the scaladoc of KafkaServer a Kafka server, a Kafka broker and a
Note
Kafka node all refer to the same concept and are hence considered synonyms.

A Kafka broker is a KafkaServer that hosts topics.

Quoting Broker article (from Wikipedia, the free encyclopedia):

A broker is an individual person who arranges transactions between a buyer and a

seller for a commission when the deal is executed.

A brokers prime responsibility is to bring sellers and buyers together and thus a broker
is the third-person facilitator between a buyer and a seller.

A Kafka broker receives messages from producers and stores them on disk keyed by unique
offset.

A Kafka broker allows consumers to fetch messages by topic, partition and offset.

Kafka brokers can create a Kafka cluster by sharing information between each other directly
or indirectly using Zookeeper.

You can start a single Kafka broker using kafka-server-start.sh script.

Starting Kafka Broker

Start Zookeeper.

./bin/zookeeper-server-start.sh config/zookeeper.properties

Only when Zookeeper is up and running you can start a Kafka server (that will connect to
Zookeeper).

./bin/kafka-server-start.sh config/server.properties

Tip Read kafka-server-start.sh script.

kafka-server-start.sh script
kafka-server-start.sh starts a Kafka broker.

114
BrokerKafka Server

$ ./bin/kafka-server-start.sh
USAGE: ./bin/kafka-server-start.sh [-daemon] server.properties [--override property=va
lue]*

Before you run kafka-server-start.sh make sure that Zookeeper is up and

Note
running. Use zookeeper-server-start shell script.

kafka-server-start.sh uses config/log4j.properties for logging configuration that you can

override using KAFKA_LOG4J_OPTS environment variable.

KAFKA_LOG4J_OPTS="-Dlog4j.configuration=file:config/log4j.properties"

kafka-server-start.sh accepts KAFKA_HEAP_OPTS and EXTRA_ARGS environment variables.

Command-line options:

1. -name defaults to kafkaServer when in daemon mode.

2. -loggc enabled when in daemon mode.

3. -daemon enables daemon mode.

4. --override property=value value that should override the value set for property in

server.properties file.

$ ./bin/kafka-server-start.sh config/server.properties --override broker.id=100

...
INFO [KafkaServer id=100] started (kafka.server.KafkaServer)

115
Topics

Topics
Topics are virtual groups of one or many partitions across Kafka brokers in a Kafka cluster.

A single Kafka broker stores messages in a partition in an ordered fashion, i.e. appends
them one message after another and creates a log file.

Producers write messages to the tail of these logs that consumers read at their own pace.

Kafka scales topic consumption by distributing partitions among a consumer group, which is
a set of consumers sharing a common group identifier.

Partitions
Partitions with messagestopics can be partitioned to improve read/write performance
and resiliency. You can lay out a topic (as partitions) across a cluster of machines to allow
data streams larger than the capability of a single machine. Partitions are log files on disk
with sequential write only. Kafka guarantees message ordering in a partition.

The log end offset is the offset of the last message written to a log.

The high watermark offset is the offset of the last message that was successfully copied to
all of the logs replicas.

A consumer can only read up to the high watermark offset to prevent reading
Note
unreplicated messages.

116
Messages

Messages
Messages are the data that brokers store in the partitions of a topic.

Messages are sequentially appended to the end of the partition log file and numbered by
unique offsets. They are persisted on disk (aka disk-based persistence) and replicated within
the cluster to prevent data loss. It has an in-memory page cache to improve data reads.
Messages are in partitions until deleted when TTL occurs or after compaction.

Offsets
Offsets are message positions in a topic.

117
Kafka Clients

Kafka Clients
Producers

Consumers

118
Producers

Producers
Multiple concurrent producers that send (aka push) messages to topics which is appending
the messages to the end of partitions. They can batch messages before they are sent over
the wire to a topic. Producers support message compression. Producers can send
messages in synchronous (with acknowledgement) or asynchronous mode.

import collection.JavaConversions._
import org.apache.kafka.common.serialization._
import org.apache.kafka.clients.producer.KafkaProducer
import org.apache.kafka.clients.producer.ProducerRecord

val cfg = Map(

"bootstrap.servers" -> "localhost:9092",
"key.serializer" -> classOf[IntegerSerializer],
"value.serializer" -> classOf[StringSerializer])
val producer = new KafkaProducer[Int, String](cfg)
val msg = new ProducerRecord(topic = "my-topic", key = 1, value = "hello")

scala> val f = producer.send(msg)

f: java.util.concurrent.Future[org.apache.kafka.clients.producer.RecordMetadata] = org
.apache.kafka.clients.producer.internals.FutureRecordMetadata@2e9e8fe

scala> f.get
res7: org.apache.kafka.clients.producer.RecordMetadata = my-topic-0@1

producer.close

119
KafkaProducer

KafkaProducer
KafkaProducer is one of the main parts of the public API of Apache Kafka to write Kafka

clients that publish records to a Kafka cluster.

KafkaProducer is created whenFIXME

Table 1. KafkaProducers Internal Properties (e.g. Registries and Counters)

Name Description
Sender with the following:
sender
FIXME

Creating KafkaProducer Instance

KafkaProducer takes the following when created:

FIXME

KafkaProducer initializes the internal registries and counters.

120
Sender

Sender
Sender is a thread of execution that handles the sending of produce requests to a Kafka

cluster.

Sender is created exclusively when KafkaProducer is created.

Table 1. Senders Internal Properties (e.g. Registries and Counters)

Name Description
running Flag that controls whether run should finish or not

sendProducerData Internal Method

long sendProducerData(long now)

sendProducerData FIXME

sendProducerData is used exclusively when Sender runs a single iteration of

Note
sending.

Running Single Iteration of Sending run Method

void run(long now)

run FIXME

Note run is used exclusively when Sender is started (as a thread of execution).

run Method (of Javas Runnable )

void run()

Note run is a part of java.lang.Runnable that is executed when the thread is started.

run first prints out the following DEBUG message to the logs:

Starting Kafka producer I/O thread.

121
Sender

run keeps running (with the current time in milliseconds) until running flag is turned off.

run FIXME

Creating Sender Instance

Sender takes the following when created:

LogContext

KafkaClient

Metadata

RecordAccumulator

guaranteeMessageOrder flag

maxRequestSize

acks

The number of retries

SenderMetricsRegistry

Time

requestTimeout

retryBackoffMs

TransactionManager

ApiVersions

Sender initializes the internal registries and counters.

122
Consumers

Kafka Consumers
Multiple concurrent consumers read (aka pull) messages from topics however they want
using offsets. Unlike typical messaging systems, Kafka consumers pull messages from a
topic using offsets.

A Kafka consumer consume records from a Kafka cluster.

Kafka 0.9.0.0 was about introducing a brand new Consumer API aka New
Note
Consumer.

When a consumer is created, it requires bootstrap.servers which is the initial list of brokers
to discover the full set of alive brokers in a cluster from.

A consumer has to subscribe to the topics it wants to read messages from called topic
subscription.

Caution FIXME Building a custom consumption strategy

Using Kafka Consumer API requires kafka-clients dependency in your project.

val kafka = "0.10.2.1"

libraryDependencies += "org.apache.kafka" % "kafka-clients" % kafka

// You should also define the logging binding for slf4j

// Kafka uses slf4j for logging
val logback = "1.2.3"
libraryDependencies += "ch.qos.logback" % "logback-core" % logback
libraryDependencies += "ch.qos.logback" % "logback-classic" % logback

Consumer Contract

public interface Consumer<K, V> extends Closeable {

// FIXME more methods...
ConsumerRecords<K, V> poll(long timeout)
}

Table 1. Consumer Contract

Method Description
poll Used to

123
Consumers

Topic Subscription
Topic Subscription is the process of announcing the topics a consumer wants to read
messages from.

void subscribe(Collection<String> topics)

void subscribe(Collection<String> topics, ConsumerRebalanceListener callback)
void subscribe(Pattern pattern, ConsumerRebalanceListener callback)

subscribe method is not incremental and you always must include the full list
Note
of topics that you want to consume from.

You can change the set of topics a consumer is subscrib to at any time and (given the note
above) any topics previously subscribed to will be replaced by the new list after subscribe .

Automatic and Manual Partition Assignment

Caution FIXME

Consumer Groups
A consumer group is a set of Kafka consumers that share a common link:a set of
consumers sharing a common group identifier#group.id[group identifier].

Partitions in a topic are assigned to exactly one member in a consumer group.

Group Coordination Protocol

Caution FIXME

the new consumer uses a group coordination protocol built into Kafka

For each group, one of the brokers is selected as the group coordinator. The
coordinator is responsible for managing the state of the group. Its main job is to mediate
partition assignment when new members arrive, old members depart, and when topic
metadata changes. The act of reassigning partitions is known as rebalancing the group.

When a group is first initialized, the consumers typically begin reading from either the
earliest or latest offset in each partition. The messages in each partition log are then
read sequentially. As the consumer makes progress, it commits the offsets of messages
it has successfully processed.

124
Consumers

When a partition gets reassigned to another consumer in the group, the initial position is
set to the last committed offset. If a consumer suddenly crashed, then the group
member taking over the partition would begin consumption from the last committed
offset (possibly reprocessing messages that the failed consumer would have processed
already but not committed yet).

Further reading or watching

1. Introducing the Kafka Consumer: Getting Started with the New Apache Kafka 0.9
Consumer Client

125
KafkaConsumerMain Class For Kafka Consumers

KafkaConsumerMain Class For Kafka

Consumers
KafkaConsumer is the main public class that Kafka developers use to write Kafka consumers.

KafkaConsumer is a part of the public API and is created with properties and (key and value)

deserializers as configuration.

bootstrap.servers and group.id properties are mandatory. They usually appear

Note in the code as ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG and
ConsumerConfig.GROUP_ID_CONFIG values, respectively.

KafkaConsumer follows the Consumer contract.

// sandbox/kafka-sandbox
val bootstrapServers = "localhost:9092"
val groupId = "kafka-sandbox"
import org.apache.kafka.clients.consumer.ConsumerConfig
val configs: Map[String, Object] = Map(
// required properties
ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG -> bootstrapServers,
ConsumerConfig.GROUP_ID_CONFIG -> groupId
)
import org.apache.kafka.common.serialization.StringDeserializer
val keyDeserializer = new StringDeserializer
val valueDeserializer = new StringDeserializer

import scala.collection.JavaConverters._
import org.apache.kafka.clients.consumer.KafkaConsumer
val consumer = new KafkaConsumer[String, String](
configs.asJava,
keyDeserializer,
valueDeserializer)

KafkaConsumer registers itself in JMX with kafka.consumer prefix.

Caution FIXME How does the JMX registration happen?

126
KafkaConsumerMain Class For Kafka Consumers

Figure 1. KafkaConsumer
KafkaConsumer does not support multi-threaded access. You should use
only one thread per KafkaConsumer instance.
KafkaConsumer uses light locks protecting itself from multi-threaded access
Important and reports ConcurrentModificationException when it happens.

KafkaConsumer is not safe for multi-threaded access

127
KafkaConsumerMain Class For Kafka Consumers

Table 1. KafkaConsumers Internal Properties (e.g. Registries and Counters)

Name Description

ConsumerNetworkClient
Used mainly (?) to create the Fetcher and
client
ConsumerCoordinator
Used also in poll, pollOnce and wakeup (but I think the
usage should be limited to create Fetcher and
ConsumerCoordinator)

clientId

coordinator ConsumerCoordinator

Fetcher
fetcher Created right when KafkaConsumer is created.
Used whenFIXME

ConsumerInterceptors that holds ConsumerInterceptor

interceptors instances (defined using interceptor.classes setting).
Used whenFIXME

Metadata
metadata Created right when KafkaConsumer is created.
Used whenFIXME

metrics Metrics

retryBackoffMs retry.backoff.ms property or a user-defined value

Corresponds to request.timeout.ms property

requestTimeoutMs KafkaConsumer reports ConfigException when smaller or
equal than session.timeout.ms and fetch.max.wait.ms
properties.

SubscriptionState for auto.offset.reset setting.

subscriptions
Created when KafkaConsumer is created.

128
KafkaConsumerMain Class For Kafka Consumers

Enable DEBUG or TRACE logging levels for

org.apache.kafka.clients.consumer.KafkaConsumer logger to see what happens
inside.

Add the following line to config/tools-log4j.properties :

Tip
log4j.logger.org.apache.kafka.clients.consumer.KafkaConsumer=TRACE, stdout

Refer to Logging.

unsubscribe Method

Caution FIXME

Subscribing to Topics Matching Pattern subscribe

Method

void subscribe(Collection<String> topics)

Note subscribe is a part of Consumer Contract.

subscribe FIXME

Note subscribe is used whenFIXME

Subscribing to Topics subscribe Method

void subscribe(Collection<String> topics) (1)

void subscribe(Collection<String> topics, ConsumerRebalanceListener listener)

1. A short-hand for the other subscribe with NoOpConsumerRebalanceListener as

ConsumerRebalanceListener

subscribe subscribes KafkaConsumer to the given topics.

val topics = Seq("topic1")

println(s"Subscribing to ${topics.mkString(", ")}")

import scala.collection.JavaConverters._
consumer.subscribe(topics.asJava)

129
KafkaConsumerMain Class For Kafka Consumers

Internally, subscribe prints out the following DEBUG message to the logs:

DEBUG Subscribed to topic(s): [comma-separated topics]

subscribe then requests SubscriptionState to subscribe for the topics and listener .

In the end, subscribe requests SubscriptionState for groupSubscription that it then passes
along to Metadata to set the topics to track.

Figure 2. KafkaConsumer subscribes to topics

Poll Specified Milliseconds For ConsumerRecords per

TopicPartitions poll Method

ConsumerRecords<K, V> poll(long timeout)

poll polls for new records until timeout expires.

KafkaConsumer has to be subscribed to some topics or assigned partitions

Note
before calling poll.

The input timeout should be 0 or greater and represents the milliseconds to

Note
poll for records.

val seconds = 10
while (true) {
println(s"Polling for records for $seconds secs")
val records = consumer.poll(seconds * 1000)
// do something with the records here
}

Internally, poll starts by polling once (for timeout milliseconds).

130
KafkaConsumerMain Class For Kafka Consumers

If there are records available, poll checks Fetcher for sendFetches and
ConsumerNetworkClient for pendingRequestCount flag. If either is positive, poll requests
ConsumerNetworkClient to pollNoWakeup.

Caution FIXME Make the above more user-friendly

poll returns the available ConsumerRecords directly when no ConsumerInterceptors are

defined or passes them through ConsumerInterceptors using onConsume.

FIXME Make the above more user-friendly, e.g. when could interceptors
Caution
be empty?

Figure 3. KafkaConsumer polls topics

Note poll is a part of Consumer contract toFIXME

Getting Partitions For Topic partitionsFor Method

Caution FIXME

endOffsets Method

Caution FIXME

offsetsForTimes Method

Caution FIXME

updateFetchPositions Method

Caution FIXME

Polling Once for ConsumerRecords per TopicPartition

pollOnce Internal Method

131
KafkaConsumerMain Class For Kafka Consumers

Caution FIXME

Requesting Metadata for All Topics (From Brokers)

listTopics Method

Map<String, List<PartitionInfo>> listTopics()

Internally, listTopics simply requests Fetcher for metadata for all topics and returns it.

consumer.listTopics().asScala.foreach { case (name, partitions) =>

println(s"topic: $name (partitions: ${partitions.size()})")
}

listTopics uses requestTimeoutMs that corresponds to request.timeout.ms

Note
property.

beginningOffsets Method

Map<TopicPartition, Long> beginningOffsets(Collection<TopicPartition> partitions)

beginningOffsets requests Fetcher for beginningOffsets and returns it.

Creating KafkaConsumer Instance

KafkaConsumer takes the following when created:

Consumer configuration (that is converted internally to ConsumerConfig)

Deserializer for keys

Deserializer for values

KafkaConsumer initializes the internal registries and counters.

KafkaConsumer API offers other constructors that in the end use the public 3-
Note argument constructor that in turn passes the call on to the private internal
constructor.

KafkaConsumer Public Constructor

132
KafkaConsumerMain Class For Kafka Consumers

// Public API
KafkaConsumer(
Map<String, Object> configs,
Deserializer<K> keyDeserializer,
Deserializer<V> valueDeserializer)

When created, KafkaConsumer adds the keyDeserializer and valueDeserializer to configs (as
key.deserializer and value.deserializer properties respectively) and creates a
ConsumerConfig.

KafkaConsumer passes the call on to the internal constructor.

KafkaConsumer Internal Constructor

KafkaConsumer(
ConsumerConfig config,
Deserializer<K> keyDeserializer,
Deserializer<V> valueDeserializer)

When called, the internal KafkaConsumer constructor prints out the following DEBUG
message to the logs:

DEBUG Starting the Kafka consumer

KafkaConsumer sets the internal requestTimeoutMs to request.timeout.ms property.

KafkaConsumer sets the internal clientId to client.id or generates one with prefix consumer-

(starting from 1) if not set.

KafkaConsumer sets the internal Metrics (and JmxReporter with kafka.consumer prefix).

KafkaConsumer sets the internal retryBackoffMs to retry.backoff.ms property.

Caution FIXME Finish me!

KafkaConsumer creates the internal Metadata with the following arguments:

1. retryBackoffMs

2. metadata.max.age.ms

3. allowAutoTopicCreation enabled

4. topicExpiryEnabled disabled

133
KafkaConsumerMain Class For Kafka Consumers

5. ClusterResourceListeners with user-defined list of ConsumerInterceptors in

interceptor.classes property

KafkaConsumer updates metadata with bootstrap.servers.

Caution FIXME Finish me!

KafkaConsumer creates a NetworkClient withFIXME

Caution FIXME Finish me!

KafkaConsumer creates Fetcher with the following properties:

fetch.min.bytes

fetch.max.bytes

fetch.max.wait.ms

max.partition.fetch.bytes

max.poll.records

check.crcs

In the end, KafkaConsumer prints out the following DEBUG message to the logs:

DEBUG Kafka consumer created

Any issues while creating a KafkaConsumer are reported as KafkaException .

org.apache.kafka.common.KafkaException: Failed to construct kafka consumer

wakeup Method

void wakeup()

Note wakeup is a part of Consumer Contract.

wakeup simply requests ConsumerNetworkClient to wakeup.

134
KafkaConsumerMain Class For Kafka Consumers

Figure 4. KafkaConsumers wakeup Method

Quoting wakeup of Javas java.nio.channels.Selector given wakeup simply
passes through the intermediaries and in the end triggers it.

Note Causes the first selection operation that has not yet returned to return
immediately.
Read about Selection in java.nio.channels.Selector's javadoc.

Note wakeup is used whenFIXME

135
Deserializer

Deserializer
Caution FIXME

136
ConsumerConfig

ConsumerConfig
Caution FIXME

Caution FIXME Describe static block when ConsumerConfig is created.

137
Consumer

Consumer
Consumer is the contract for Kafka consumers.

KafkaConsumer is the main public class that Kafka developers use to write
Note
Kafka consumers.

Consumer Contract

138
Consumer

package org.apache.kafka.clients.consumer;

public interface Consumer<K, V> extends Closeable {

void assign(Collection<TopicPartition> partitions);
Set<TopicPartition> assignment();

void subscribe(Collection<String> topics);

void subscribe(Collection<String> topics, ConsumerRebalanceListener callback);
void subscribe(Pattern pattern, ConsumerRebalanceListener callback);
void subscribe(Pattern pattern);
Set<String> subscription();
void unsubscribe();

ConsumerRecords<K, V> poll(long timeout);

void commitSync();
void commitSync(Map<TopicPartition, OffsetAndMetadata> offsets);
void commitAsync();
void commitAsync(OffsetCommitCallback callback);
void commitAsync(Map<TopicPartition, OffsetAndMetadata> offsets, OffsetCommitCallbac
k callback);

void seek(TopicPartition partition, long offset);

void seekToBeginning(Collection<TopicPartition> partitions);
void seekToEnd(Collection<TopicPartition> partitions);

long position(TopicPartition partition);

OffsetAndMetadata committed(TopicPartition partition);
Map<MetricName, ? extends Metric> metrics();
List<PartitionInfo> partitionsFor(String topic);
Map<String, List<PartitionInfo>> listTopics();
Set<TopicPartition> paused();
void pause(Collection<TopicPartition> partitions);
void resume(Collection<TopicPartition> partitions);
Map<TopicPartition, OffsetAndTimestamp> offsetsForTimes(Map<TopicPartition, Long> ti
mestampsToSearch);
Map<TopicPartition, Long> beginningOffsets(Collection<TopicPartition> partitions);
Map<TopicPartition, Long> endOffsets(Collection<TopicPartition> partitions);

void close();
void close(long timeout, TimeUnit unit);

void wakeup();
}

Table 1. Consumer Contract

Method Description
wakeup

139
Consumer

140
ConsumerInterceptor

ConsumerInterceptor
Example

package pl.jaceklaskowski.kafka

import java.util

import org.apache.kafka.clients.consumer.{ConsumerInterceptor, ConsumerRecords, Offset

AndMetadata}
import org.apache.kafka.common.TopicPartition

class KafkaInterceptor extends ConsumerInterceptor[String, String] {

override def onConsume(records: ConsumerRecords[String, String]):
ConsumerRecords[String, String] = {
println(s"KafkaInterceptor.onConsume")
import scala.collection.JavaConverters._
records.asScala.foreach { r =>
println(s"=> $r")
}
records
}

override def close(): Unit = {

println("KafkaInterceptor.close")
}

override def onCommit(offsets: util.Map[TopicPartition, OffsetAndMetadata]): Unit =

{
println("KafkaInterceptor.onCommit")
println(s"$offsets")
}

override def configure(configs: util.Map[String, _]): Unit = {

println(s"KafkaInterceptor.configure($configs)")
}
}

onConsume Method

Caution FIXME

141
Clusters

Clusters
A Kafka cluster is the central data exchange backbone for an organization.

142
Sensor

Sensor
Sensor isFIXME

143
MetricsReporter

MetricsReporter
JmxReporter
JmxReporter is a metrics reporter that is always included in metric.reporters setting with

kafka.consumer metrics prefix.

144
ProducerMetrics

ProducerMetrics
ProducerMetrics isFIXME

// Generate HTML with the metrics

$ ./bin/kafka-run-class.sh org.apache.kafka.clients.producer.internals.ProducerMetrics
> metrics.html

145
SenderMetrics

SenderMetrics
SenderMetrics isFIXME

146
Kafka Tools

Kafka Tools
TopicCommand
kafka.admin.TopicCommand

./bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --par

titions 1 --topic my-topic

./bin/kafka-topics.sh --zookeeper localhost:2181 --describe --topic my-topic

ConsoleProducer
kafka.tools.ConsoleProducer

./bin/kafka-console-producer.sh --broker-list localhost:9092 --topic my-topic

ConsoleConsumer
kafka.tools.ConsoleConsumer

./bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic my-topic

147
kafka-configs.sh

kafka-configs.sh Shell Script

kafka-configs.sh is a shell script thatFIXME

./bin/kafka-configs.sh \
--zookeeper localhost:2181 \
--alter \
--entity-type topics \
--entity-name test \
--add-config retention.ms=5000

148
kafka-consumer-groups.sh

kafka-consumer-groups.sh Shell Script

kafka-consumer-groups.sh is a shell script thatFIXME

149
kafka-topics.sh

kafka-topics.sh Shell Script

kafka-topics.sh is a shell script thatFIXME

150
Properties

Kafka Properties
Table 1. Properties
Default
Name Importance
Value

Reset policywhat to do when the

any more on the server (e.g. becau
earliestautomatically reset
auto.offset.reset latest No latestautomatically reset th

nonethrow an exception to
group
anything else: throw an except

authorizer.class.name

A comma-separated list of
bootstrap.servers (empty) Yes
cluster, e.g. localhost:9092

broker.rack

check.crcs

(random-
client.id
generated)

Enables topic deletion

delete.topic.enable true High
Note Deleting topic through t

enable.auto.commit

fetch.min.bytes

fetch.max.bytes

fetch.max.wait.ms

group.id The name of the consumer group t

heartbeat.interval.ms
The expected time between heartb
management facilities.

host.name (empty) The hostname the default endpoint

151
Properties

inter.broker.protocol.version

Comma-separated list of
interceptor.classes (empty)
props.put(ConsumerConfig

listeners (empty) The addresses the socket server lis

key.deserializer How to deserialize message keys.

max.partition.fetch.bytes How to deserialize message keys.

max.poll.records

metadata.max.age.ms

metric.reporters JmxReporter The list of fully-qualified classes na

metrics.num.samples Number of samples to compute me

metrics.sample.window.ms Time window (in milliseconds) a me

num.io.threads 8 The number of threads that

num.network.threads 3 The number of threads that Socket

port (empty) The port the default endpoint

rebalance.timeout.ms The maximum allowed time for eac

Time to wait before attempting to re

retry.backoff.ms
This avoids repeatedly sending req

request.timeout.ms

sasl.enabled.mechanisms

session.timeout.ms 10000 High The timeout used to detect worker

value.deserializer How to deserialize message values

Caution FIXME Whats worker?

152
Properties

// requires org.apache.kafka:connect-runtime:0.10.0.1 dependency

import org.apache.kafka.connect.runtime.distributed.DistributedConfig
DistributedConfig.SESSION_TIMEOUT_MS_CONFIG

FIXME How to know the current value of a setting on a producers and a

Caution
consumers side?

153
bootstrap.servers

bootstrap.servers Property
bootstrap.servers is a comma-separated list of host and port pairs that are the addresses

of the Kafka brokers in a "bootstrap" Kafka cluster that a Kafka client connects to initially to
bootstrap itself.

A host and port pair uses : as the separator.

localhost:9092
localhost:9092,another.host:9092

bootstrap.servers provides the initial hosts that act as the starting point for a Kafka client to

discover the full set of alive servers in the cluster.

Since these servers are just used for the initial connection to discover the full
cluster membership (which may change dynamically), this list does not have to
Note
contain the full set of servers (you may want more than one, though, in case a
server is down).

Clients (producers or consumers) make use of all servers irrespective of which

Note
servers are specified in bootstrap.servers for bootstrapping.

Use org.apache.kafka.clients.CommonClientConfigs.BOOTSTRAP_SERVERS_CONFIG
Tip
public value to refer to the property.

154
client.id

client.id Property
An optional identifier of a Kafka consumer (in a consumer group) that is passed to a Kafka
broker with every request.

The sole purpose of this is to be able to track the source of requests beyond just ip and port
by allowing a logical application name to be included in Kafka logs and monitoring
aggregates.

155
enable.auto.commit

enable.auto.commit Property
enable.auto.commit FIXME

By default, as the consumer reads messages from Kafka, it will periodically commit its
current offset (defined as the offset of the next message to be read) for the partitions it is
reading from back to Kafka. Often you would like more control over exactly when offsets are
committed. In this case you can set enable.auto.commit to false and call the commit
method on the consumer.

156
group.id

group.id Property
group.id specifies the name of the consumer group a Kafka consumer belongs to.

When the Kafka consumer is constructed and group.id does not exist yet (i.e. there are no
existing consumers that are part of the group), the consumer group will be created
automatically.

If all consumers in a group leave the group, the group is automatically

Note
destroyed.

157
retry.backoff.ms

retry.backoff.ms Property
retry.backoff.ms is the time to wait before attempting to retry a failed request to a given

topic partition.

This avoids repeatedly sending requests in a tight loop under some failure scenarios.

158
Logging

Logging
Kafka Broker
A Kafka broker (started using kafka-server-start.sh ) uses config/log4j.properties for
logging configuration.

config/log4j.properties defaults to INFO logging level with stdout and kafkaAppender

(that spit messages to standard output and logs/server.log , respectively).

log4j.rootLogger=INFO, stdout, kafkaAppender

log4j.appender.stdout=org.apache.log4j.ConsoleAppender
log4j.appender.stdout.layout=org.apache.log4j.PatternLayout
log4j.appender.stdout.layout.ConversionPattern=[%d] %p %m (%c)%n

log4j.appender.kafkaAppender=org.apache.log4j.DailyRollingFileAppender
log4j.appender.kafkaAppender.DatePattern='.'yyyy-MM-dd-HH
log4j.appender.kafkaAppender.File=${kafka.logs.dir}/server.log
log4j.appender.kafkaAppender.layout=org.apache.log4j.PatternLayout
log4j.appender.kafkaAppender.layout.ConversionPattern=[%d] %p %m (%c)%n

If you want to change the logging level of a logger, e.g. kafka.controller.KafkaController ,

to DEBUG add the following line to config/log4j.properties and restart the server.

log4j.logger.kafka.controller.KafkaController=DEBUG, stdout

Kafka Tools
Kafka tools like kafka-console-consumer.sh (that uses KafkaConsumer under the covers)
use config/tools-log4j.properties file.

Note Kafka tools use bin/kafka-run-class.sh to execute their implementations.

Kafka uses Simple Logging Facade for Java (SLF4J) for logging.

159
Logging

Use slf4j-simple library dependency in Scala applications (in build.sbt ) for

basic logging where messages of level INFO and higher are printed to
System.err .

build.sbt
libraryDependencies += "org.slf4j" % "slf4j-simple" % "1.8.0-alpha2"

Tip Replace slf4js simple binding to switch between logging frameworks (e.g. slf4j-
log4j12 for log4j).

build.sbt
val logback = "1.2.3"
libraryDependencies += "ch.qos.logback" % "logback-core" % logback
libraryDependencies += "ch.qos.logback" % "logback-classic" % logback

With logbacks configuration (as described in the above tip) you may see the following message

SLF4J: Class path contains multiple SLF4J bindings.

SLF4J: Found binding in [jar:file:/Users/jacek/.ivy2/cache/org.slf4j/slf4j-log4j12/jars/slf
SLF4J: Found binding in [jar:file:/Users/jacek/.ivy2/cache/ch.qos.logback/logback-classic/j
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]

Note Commenting out logback-classic in build.sbt resolves it.

build.sbt
val logback = "1.2.3"
libraryDependencies += "ch.qos.logback" % "logback-core" % logback
//libraryDependencies += "ch.qos.logback" % "logback-classic" % logback

FIXME: Explain why the commenting out is required?

log4j.properties Logging Configuration File

log4j.properties

log4j.rootLogger=INFO, stdout

log4j.appender.stdout=org.apache.log4j.ConsoleAppender
log4j.appender.stdout.layout=org.apache.log4j.PatternLayout
log4j.appender.stdout.layout.ConversionPattern=[%d] %p %m (%c)%n

log4j.logger.org.apache.kafka.clients.consumer.ConsumerConfig=DEBUG

Save log4j.properties in src/main/resources in your Kafka applications

Tip
project.

160
Logging

KAFKA_LOG4J_OPTS Environment Variable

You can use KAFKA_LOG4J_OPTS environment variable to specify the log4j configuration to
use.

KAFKA_LOG4J_OPTS=-Dlog4j.configuration=file:[your-log4j-configuration-here]

Note Unless defined, kafka-run-class.sh sets it to config/tools-log4j.properties .

161
WorkerGroupMember

WorkerGroupMember
Caution FIXME WorkerCoordinator? DistributedHerder?

162
ConnectDistributed

ConnectDistributed
ConnectDistributed is a command-line utility that runs Kafka Connect in distributed mode.

Caution FIXME Doh, Id rather not enter Kafka Connect yet. Not interested in it yet.

163
Gradle Tips

Gradle Tips
Building Kafka Distribution

gradle -PscalaVersion=2.11.8 clean releaseTarGz install

It takes around 2 minutes (after all the dependencies were downloaded once).

After the command, you can unpack the release as follows:

tar -zxvf core/build/distributions/kafka_2.11-0.10.1.0-SNAPSHOT.tgz

Executing Single Test

gradle -PscalaVersion=2.11.8 :core:test --no-rebuild --tests "*PlaintextProducerSendTe

st"

164
Zookeeper Tips

Zookeeper Tips
The zookeeper shell shipped with Kafka works with no support for command line history
because jline jar is missing (see KAFKA-2385).

A solution is to use the official distribution of Apache Zookeeper (3.4.10 as of this writing)
from Apache ZooKeeper Releases.

Once downloaded, use ./bin/zkCli.sh to connect to Zookeeper that is used for Kafka.

$ ./bin/zkCli.sh -server localhost:2181

...
JLine support is enabled

[zk: localhost:2181(CONNECTED) 0] ls /
[cluster, controller_epoch, controller, brokers, zookeeper, admin, isr_change_notifica
tion, consumers, log_dir_event_notification, latest_producer_id_block, config]

[zk: localhost:2181(CONNECTED) 5] get /controller

{"version":1,"brokerid":200,"timestamp":"1506417983145"}
cZxid = 0x11b
ctime = Tue Sep 26 11:26:23 CEST 2017
mZxid = 0x11b
mtime = Tue Sep 26 11:26:23 CEST 2017
pZxid = 0x11b
cversion = 0
dataVersion = 0
aclVersion = 0
ephemeralOwner = 0x15ebd811a0e0001
dataLength = 56
numChildren = 0

165
Kafka in Scala REPL for Interactive Exploration

Kafka in Scala REPL for Interactive Exploration

Use the following build.sbt and execute sbt followed by console command (while
inside the sbt shell).

The reason for executing console command after sbt has started up is that
Note
command history did not work using the key-up and key-down keys. YMMV

build.sbt

name := "kafka-sandbox"

version := "1.0"

scalaVersion := "2.12.3"

//val kafkaVer = "0.11.0.1"

resolvers += Resolver.mavenLocal
val kafkaVer = "1.0.0-SNAPSHOT"
libraryDependencies += "org.apache.kafka" % "kafka-clients" % kafkaVer
libraryDependencies += "org.apache.kafka" %% "kafka" % kafkaVer

val logback = "1.2.3"

libraryDependencies += "ch.qos.logback" % "logback-core" % logback
libraryDependencies += "ch.qos.logback" % "logback-classic" % logback

sbt with console command

kafka-sandbox sbt
[info] Loading settings from plugins.sbt ...
[info] Loading project definition from /Users/jacek/dev/sandbox/kafka-sandbox/project
[info] Loading settings from build.sbt ...
[info] Set current project to kafka-sandbox (in build file:/Users/jacek/dev/sandbox/ka
fka-sandbox/)
[info] sbt server started at 127.0.0.1:4408
sbt:kafka-sandbox> console
[info] Starting scala interpreter...
Welcome to Scala 2.12.3 (Java HotSpot(TM) 64-Bit Server VM, Java 1.8.0_144).
Type in expressions for evaluation. Or try :help.

scala> import kafka.utils._

import kafka.utils._

scala> ZkUtils.controllerZkData(1, System.currentTimeMillis())

res0: String = {"version":1,"brokerid":1,"timestamp":"1506162122097"}

166
Kafka in Scala REPL for Interactive Exploration

167
Further reading or watching

Further Reading or Watching

Videos
1. Apache Kafka Core Internals: A Deep Dive by Jun Rao, Confluent at Kafka Summit
2017 in New York City

Articles
1. Apache Kafka for Beginners - an excellent article that you should start your Kafka
journey with.

2. Introduction to Apache Kafka for Python Programmers - using Python as the

programming language, but offers enough exposure to the topics of the Kafka
architecture.

168