Hardening Kafka Replication

Hardening Kafka Replication
Jason Gustafson, Engineer@Confluent

● At the heart of Kafka is the log
● Log replication provides high availability
● Kafka has a solid replication protocol
● 99.999% of the time it does the right thing
● This talk is about the remaining 0.001%
Overview

View of a single partition
Key
Value
Offset

Message Appends
Key
Value
Offset

Key
Value
Offset
Message Appends
k0
v0
0

Message Appends
k0 k1 k2
v0 v1 v2
0 1 2
Key
Value
Offset

k0 k1 k2
v0 v1 v2
0 1 2
Key
Value
Offset

k0 k1 k2
v0 v1 v2
0 1 2
Key
Value
Offset
k0 k1 k2
v0 v1 v2

Key
Value
k0 k1 k2
v0 v1 v2
Offset 0

Key
Value
k0 k1 k2
v0 v1 v2

r0 r1 r2
Record

r0 r1 r2
View of a single
partition with 3
replicas

r0 r1 r2 A
B
C
View of a single
partition with 3
replicas

r0 r1 r2 A
B
C
The protocol’s goal is
to replicate the logs
exactly to all replicas

r0 r1 r2
r0 r1 r2
r0 r1 r2
A
B
C
The protocol’s goal is
to replicate the logs
exactly to all replicas

Leader
A
B
C
For each partition, one replica
is elected as the leader

Leader
Follower
Follower
A
B
C
Replicas that are not leaders
are called followers

Leader
Follower
Follower
A
B
C
Leaders accept writes from
producers.

r0 r1 r2Leader
Follower
Follower
A
B
C
Leaders accept writes from
producers.

r0 r1 r2
A
B
C
Leader
Follower
Follower
Followers fetch from the
leader.

r0 r1
r0 r1 r2
A
B
C
Leader
Follower
Follower
leader.

r0 r1
r0 r1 r2
r0
A
B
C
Leader
Follower
Follower
leader.

r0 r1
r0 r1 r2
r0
A
B
C
Leader
Follower
Follower
Leader election is handled by
a separate component known
as the controller

r0 r1
r0 r1 r2
r0
A
B
C
Leader
Follower
Follower
Leader Epoch ISR
B 0 A, B, C
In order to enable election by
the controller, we maintain
state in Zookeeper about the
in-sync replicas (ISR).

r0 r1
r0 r1 r2
r0
A
B
C
Leader
Follower
Follower
When there is a state change
(e.g. a new leader), the
controller sends the updated
state to all the replicas.
Leader Epoch ISR
B 0 A, B, C

r0 r1
r0 r1 r2
r0
A
B
C
Leader
(epoch=0)
Follower
(epoch=0)
Leader Epoch ISR
B 0 A, B, C
Follower
(epoch=0)
When there is a state change
(e.g. a new leader), the
controller sends the updated
state to all the replicas.

r0 r1
r0 r1 r2
r0
A
B
C
Leader
(epoch=0)
Follower
(epoch=0)
Leader Epoch ISR
B 0 A, B, C
Follower
(epoch=0)

r0 r1
r0 r1 r2
r0
A
B
C
Leader
(epoch=0)
Follower
(epoch=0)
Leader Epoch ISR
B 0 A, B, C
Follower
(epoch=0)
High Watermark
The high watermark is
the largest offset known
to be replicated to all
members of the ISR.

r0 r1
r0 r1 r2
r0
A
B
C
Leader
(epoch=0)
Follower
(epoch=0)
Leader Epoch ISR
B 0 A, B, C
Follower
(epoch=0)
The high watermark is
the largest offset known
to be replicated to all
members of the ISR.

r0 r1
r0 r1 r2
r0
A
B
C
Leader
(epoch=0)
Follower
(epoch=0)
Leader Epoch ISR
B 0 A, B, C
Follower
(epoch=0)
Records below the high
watermark are considered
“committed” and are visible
to consumers.
Committed

r0 r1
r0 r1 r2
r0
A
B
C
Leader
(epoch=0)
Follower
(epoch=0)
Leader Epoch ISR
B 0 A, B, C
Follower
(epoch=0)
Records above the high
watermark are considered
uncommitted.
Committed Uncommitted

r0 r1
r0 r1 r2
r0
A
B
C
Leader
(epoch=0)
Follower
(epoch=0)
Leader Epoch ISR
B 0 A, B, C
Follower
(epoch=0)
As records are replicated,
the high watermark moves
forward.

r0 r1
r0 r1 r2
r0 r1
A
B
C
Leader
(epoch=0)
Follower
(epoch=0)
Leader Epoch ISR
B 0 A, B, C
Follower
(epoch=0)
As records are replicated,
the high watermark moves
forward.

r0 r1
r0 r1 r2
r0 r1
A
B
C
Leader
(epoch=0)
Follower
(epoch=0)
Leader Epoch ISR
B 0 A, B, C
Follower
(epoch=0)
If a replica falls behind, it
can be removed from the
ISR by the leader.

r0 r1
r0 r1 r2 r3 r4 r5 r6
r0 r1
A
B
C
Leader
(epoch=0)
Follower
(epoch=0)
Leader Epoch ISR
B 0 A, B, C
Follower
(epoch=0)
ISR by the leader.

r0 r1 r2 r3 r4
r0 r1 r2 r3 r4 r5 r6
r0 r1
A
B
C
Leader
(epoch=0)
Follower
(epoch=0)
Leader Epoch ISR
B 0 A, B, C
Follower
(epoch=0)
ISR by the leader.

r0 r1 r2 r3 r4
r0 r1 r2 r3 r4 r5 r6
r0 r1
A
B
C
Leader
(epoch=0)
Follower
(epoch=0)
Leader Epoch ISR
B 0 A, B
Follower
(epoch=0)
ISR by the leader.

r0 r1 r2 r3 r4
r0 r1 r2 r3 r4 r5 r6
r0 r1
A
B
C
Leader
(epoch=0)
Follower
(epoch=0)
Leader Epoch ISR
B 0 A, B
Follower
(epoch=0)
An out-of-sync replica that
catches up to the high
watermark is added back
to the ISR.

r0 r1 r2 r3 r4
r0 r1 r2 r3 r4 r5 r6
r0 r1 r2 r3 r4 r5
A
B
C
Leader
(epoch=0)
Follower
(epoch=0)
Leader Epoch ISR
B 0 A, B
Follower
(epoch=0)
to the ISR.

r0 r1 r2 r3 r4
r0 r1 r2 r3 r4 r5 r6
r0 r1 r2 r3 r4 r5
A
B
C
Leader
(epoch=0)
Follower
(epoch=0)
Leader Epoch ISR
B 0 A, B, C
Follower
(epoch=0)
to the ISR.

r0 r1 r2 r3 r4
r0 r1 r2 r3 r4 r5 r6
r0 r1 r2 r3 r4 r5
A
B
C
Leader
(epoch=0)
Follower
(epoch=0)
Leader Epoch ISR
B 0 A, B, C
Follower
(epoch=0)

r0 r1 r2 r3 r4
r0 r1 r2 r3 r4 r5 r6
r0 r1 r2 r3 r4 r5
A
B
C
Leader
(epoch=0)
Follower
(epoch=0)
Leader Epoch ISR
B 0 A, B, C
Follower
(epoch=0)
Only replicas in the ISR are
eligible to become leader

r0 r1 r2 r3 r4
r0 r1 r2 r3 r4 r5 r6
r0 r1 r2 r3 r4 r5
A
B
C
Leader
(epoch=0)
Follower
(epoch=0)
Leader Epoch ISR
B 0 A, B, C
Follower
(epoch=0)
When a leader fails, the
controller will take it out of
the ISR and elect a new
leader from the remaining
ISR.

r0 r1 r2 r3 r4
r0 r1 r2 r3 r4 r5 r6
r0 r1 r2 r3 r4 r5
A
B
C
Leader
(epoch=0)
Follower
(epoch=0)
Leader Epoch ISR
A 1 A, C
Follower
(epoch=0)
When a leader fails, the
controller will take it out of
the ISR and elect a new
leader from the remaining
ISR.

r0 r1 r2 r3 r4
r0 r1 r2 r3 r4 r5 r6
r0 r1 r2 r3 r4 r5
A
B
C
Leader
(epoch=0)
Leader
(epoch=1)
Leader Epoch ISR
A 1 A, C
Follower
(epoch=0)
The new leader/ISR state is
propagated to the
remaining replicas

r0 r1 r2 r3 r4 r7 r8
r0 r1 r2 r3 r4 r5 r6
r0 r1 r2 r3 r4 r5
A
B
C
Leader
(epoch=0)
Leader
(epoch=1)
Leader Epoch ISR
A 1 A, C
Follower
(epoch=0)
The leader can begin
accepting writes
immediately.

r0 r1 r2 r3 r4 r7 r8
r0 r1 r2 r3 r4 r5 r6
r0 r1 r2 r3 r4 r5
A
B
C
Leader
(epoch=0)
Leader
(epoch=1)
Leader Epoch ISR
A 1 A, C
Follower
(epoch=1)
Upon becoming a follower,
the replica may have
uncommitted data which
needs to be truncated.

r0 r1 r2 r3 r4 r7 r8
r0 r1 r2 r3 r4 r5 r6
r0 r1 r2 r3 r4
A
B
C
Leader
(epoch=0)
Leader
(epoch=1)
Leader Epoch ISR
A 1 A, C
Follower
(epoch=1)

r0 r1 r2 r3 r4 r7 r8
r0 r1 r2 r3 r4 r5 r6
r0 r1 r2 r3 r4 r7
A
B
C
Leader
(epoch=0)
Leader
(epoch=1)
Leader Epoch ISR
A 1 A, C
Follower
(epoch=1)

A
B
C
Leader
(epoch=0)
Follower
(epoch=0)
Leader Epoch ISR
B 0 A, B, C
Follower
(epoch=0)

A
B
C
Leader
(epoch=0)
Follower
(epoch=0)
Leader Epoch ISR
B 0 A, B, C
Follower
(epoch=0)
High Watermark

A
B
C
Leader
(epoch=0)
Follower
(epoch=0)
Leader Epoch ISR
B 0 A, B, C
Follower
(epoch=0)
High Watermark
Every replica tracks the
high watermark separately

A
B
C
Leader
(epoch=0)
Follower
(epoch=0)
Leader Epoch ISR
B 0 A, B, C
Follower
(epoch=0)

r0 r1 r2
A
B
C
Leader
(epoch=0)
Follower
(epoch=0)
Leader Epoch ISR
B 0 A, B, C
Follower
(epoch=0)

r0 r1
r0 r1 r2
A
B
C
Leader
(epoch=0)
Follower
(epoch=0)
Leader Epoch ISR
B 0 A, B, C
Follower
(epoch=0)

r0 r1
r0 r1 r2
r0
A
B
C
Leader
(epoch=0)
Follower
(epoch=0)
Leader Epoch ISR
B 0 A, B, C
Follower
(epoch=0)

r0 r1
r0 r1 r2
r0
A
B
C
Leader
(epoch=0)
Follower
(epoch=0)
Leader Epoch ISR
B 0 A, B, C
Follower
(epoch=0)
The leader advances its
high watermark based on
the fetch offsets of replicas

r0 r1
r0 r1 r2
r0
A
B
C
Leader
(epoch=0)
Follower
(epoch=0)
Leader Epoch ISR
B 0 A, B, C
Follower
(epoch=0)
The leader piggybacks its
high watermark onto fetch
responses

r0 r1
r0 r1 r2
r0 r1
A
B
C
Leader
(epoch=0)
Follower
(epoch=0)
Leader Epoch ISR
B 0 A, B, C
Follower
(epoch=0)
The leader piggybacks its
high watermark onto fetch
responses

r0 r1
r0 r1 r2
r0 r1
A
B
C
Leader
(epoch=0)
Follower
(epoch=0)
Leader Epoch ISR
B 0 A, B, C
Follower
(epoch=0)
At any point in time, the
follower high watermarks
may be a little behind the
leader’s.

Edge Case 1:
Fast leader elections

r0 r1 r2 r3
r0 r1 r2 r3 r4 r5 r6
r0 r1 r2 r3 r4 r5
A
B
C
Leader
(epoch=0)
Follower
(epoch=0)
Leader Epoch ISR
B 0 A, B, C
Follower
(epoch=0)

r0 r1 r2 r3
r0 r1 r2 r3 r4 r5 r6
r0 r1 r2 r3 r4 r5
A
B
C
Leader
(epoch=0)
Follower
(epoch=0)
Leader Epoch ISR
B 0 A, B, C
Follower
(epoch=0)
Replica B fails.

r0 r1 r2 r3
r0 r1 r2 r3 r4 r5 r6
r0 r1 r2 r3 r4 r5
A
B
C
Leader
(epoch=0)
Follower
(epoch=0)
Leader Epoch ISR
C 1 A, C
Follower
(epoch=0)
Replica B is removed from
the ISR and C is elected as
the new leader.

r0 r1 r2 r3
r0 r1 r2 r3 r4 r5 r6
r0 r1 r2 r3 r4 r5
A
B
C
Leader
(epoch=0)
Follower
(epoch=0)
Leader Epoch ISR
C 1 A, C
Leader
(epoch=1)
Replica B is removed from
the ISR and C is elected as
the new leader.

r0 r1 r2 r3
r0 r1 r2 r3 r4 r5 r6
r0 r1 r2 r3 r4 r5
A
B
C
Leader
(epoch=0)
Follower
(epoch=1)
Leader Epoch ISR
C 1 A, C
Leader
(epoch=1)
Replica A finds the new
leader and truncates its log
to the local high watermark

r0 r1
r0 r1 r2 r3 r4 r5 r6
r0 r1 r2 r3 r4 r5
A
B
C
Leader
(epoch=0)
Follower
(epoch=1)
Leader Epoch ISR
C 1 A, C
Leader
(epoch=1)
Replica A finds the new
leader and truncates its log
to the local high watermark

r0 r1
r0 r1 r2 r3 r4 r5 r6
r0 r1 r2 r3 r4 r5
A
B
C
Leader
(epoch=0)
Follower
(epoch=1)
Leader Epoch ISR
C 1 A, C
Leader
(epoch=1)
Before replica A begins
fetching, the new leader
fails.

r0 r1
r0 r1 r2 r3 r4 r5 r6
r0 r1 r2 r3 r4 r5
A
B
C
Leader
(epoch=0)
Follower
(epoch=1)
Leader Epoch ISR
A 2 A
Leader
(epoch=1)
fails.

r0 r1
r0 r1 r2 r3 r4 r5 r6
r0 r1 r2 r3 r4 r5
A
B
C
Leader
(epoch=0)
Leader
(epoch=2)
Leader Epoch ISR
A 2 A
Leader
(epoch=1)
fails.

r0 r1 r7 r8 r9
r0 r1 r2 r3 r4 r5 r6
r0 r1 r2 r3 r4 r5
A
B
C
Leader
(epoch=0)
Leader
(epoch=2)
Leader Epoch ISR
A 2 A
Leader
(epoch=1)
Leader A then begins
accepting writes.

r0 r1 r7 r8 r9
r0 r1 r2 r3 r4 r5 r6
r0 r1 r2 r3 r4 r5
A
B
C
Leader
(epoch=0)
Leader
(epoch=2)
Leader Epoch ISR
A 2 A
Leader
(epoch=1)
But r2 and r3 had already
been committed to the ISR!

r0 r1 r7 r8 r9
r0 r1 r2 r3 r4 r5 r6
r0 r1 r2 r3 r4 r5
A
B
C
Leader
(epoch=0)
Leader
(epoch=2)
Leader Epoch ISR
A 2 A
Leader
(epoch=1)
Suppose that B eventually
gets restarted.

r0 r1 r7 r8 r9
r0 r1 r2 r3 r4 r5 r6
r0 r1 r2 r3 r4 r5
A
B
C
Follower
(epoch=2)
Leader
(epoch=2)
Leader Epoch ISR
A 2 A
Leader
(epoch=1)
gets restarted.

r0 r1 r7 r8 r9
r0 r1 r2 r3
r0 r1 r2 r3 r4 r5
A
B
C
Follower
(epoch=2)
Leader
(epoch=2)
Leader Epoch ISR
A 2 A
Leader
(epoch=1)
gets restarted.

r0 r1 r7 r8 r9
r0 r1 r2 r3 r9
r0 r1 r2 r3 r4 r5
A
B
C
Follower
(epoch=2)
Leader
(epoch=2)
Leader Epoch ISR
A 2 A
Leader
(epoch=1)
The logs have now
diverged.

r0 r1 r2 r3
r0 r1 r2 r3 r4 r5 r6
r0 r1 r2 r3 r4 r5
A
B
C
Leader
(epoch=0)
Follower
(epoch=1)
Leader Epoch ISR
C 1 A, C
Leader
(epoch=1)
Replica B has failed and
replica A needs to truncate
its log.

r0 r1 r2 r3
r0 r1 r2 r3 r4 r5 r6
r0 r1 r2 r3 r4 r5
A
B
C
Leader
(epoch=0)
Follower
(epoch=1)
Leader Epoch ISR
C 1 A, C
Leader
(epoch=1)
A -> C: What is the end offset
for epoch=0?

r0 r1 r2 r3
r0 r1 r2 r3 r4 r5 r6
r0 r1 r2 r3 r4 r5
A
B
C
Leader
(epoch=0)
Follower
(epoch=1)
Leader Epoch ISR
C 1 A, C
Leader
(epoch=1)
for epoch=0?
C -> A: The end offset is 6
Offset 6

r0 r1 r2 r3
r0 r1 r2 r3 r4 r5 r6
r0 r1 r2 r3 r4 r5
A
B
C
Leader
(epoch=0)
Follower
(epoch=1)
Leader Epoch ISR
C 1 A, C
Leader
(epoch=1)
for epoch=0?
C: Cool, no truncation needed!

r0 r1 r2 r3
r0 r1 r2 r3 r4 r5 r6
r0 r1 r2 r3 r4 r5
A
B
C
Leader
(epoch=0)
Follower
(epoch=1)
Leader Epoch ISR
C 1 A, C
Leader
(epoch=1)

r0 r1 r2 r3
r0 r1 r2 r3 r4 r5 r6
r0 r1 r2 r3 r4 r5
A
B
C
Leader
(epoch=0)
Follower
(epoch=1)
Leader Epoch ISR
A 2 A
Leader
(epoch=1)

r0 r1 r2 r3 r7 r8
r0 r1 r2 r3 r4 r5 r6
r0 r1 r2 r3 r4 r5
A
B
C
Leader
(epoch=0)
Leader
(epoch=2)
Leader Epoch ISR
A 2 A
Leader
(epoch=1)

Edge Case 2:
Fast leader elections redux

r0 r1 r2
r0 r1 r2 r3 r4
r0 r1 r2 r3 r4
A
B
C
Leader
(epoch=0)
Leader
(epoch=1)
Leader Epoch ISR
A 1 A, C
Follower
(epoch=0)
replica A has been elected
as the new leader

r0 r1 r2 r7 r8
r0 r1 r2 r3 r4
r0 r1 r2 r3 r4
A
B
C
Leader
(epoch=0)
Leader
(epoch=1)
Leader Epoch ISR
A 1 A, C
Follower
(epoch=0)
as the new leader

r0 r1 r2 r7 r8
r0 r1 r2 r3 r4
r0 r1 r2 r3 r4
A
B
C
Leader
(epoch=0)
Leader
(epoch=1)
Leader Epoch ISR
A 1 A, C
Follower
(epoch=0)
as the new leader
epoch=0
offset=0
epoch=1
offset=3

r0 r1 r2 r7 r8
r0 r1 r2 r3 r4
r0 r1 r2 r3 r4
A
B
C
Leader
(epoch=0)
Leader
(epoch=1)
Leader Epoch ISR
A 1 A, C
Follower
(epoch=0)
epoch=0
offset=0
epoch=1
offset=3

r0 r1 r2 r7 r8
r0 r1 r2 r3 r4
r0 r1 r2 r3 r4
A
B
C
Leader
(epoch=0)
Leader
(epoch=1)
Leader Epoch ISR
A 1 A, C
Follower
(epoch=0)
Before replica C can
truncate its log, it becomes
the new leader.
epoch=0
offset=0
epoch=1
offset=3

r0 r1 r2 r7 r8
r0 r1 r2 r3 r4
r0 r1 r2 r3 r4
A
B
C
Leader
(epoch=0)
Leader
(epoch=1)
Leader Epoch ISR
C 2 A, C
Follower
(epoch=0)
the new leader.
epoch=0
offset=0
epoch=1
offset=3

r0 r1 r2 r7 r8
r0 r1 r2 r3 r4
r0 r1 r2 r3 r4
A
B
C
Leader
(epoch=0)
Leader
(epoch=1)
Leader Epoch ISR
C 2 A, C
Leader
(epoch=2)
the new leader.
epoch=0
offset=0
epoch=1
offset=3

r0 r1 r2 r7 r8
r0 r1 r2 r3 r4
r0 r1 r2 r3 r4 r9
A
B
C
Leader
(epoch=0)
Leader
(epoch=1)
Leader Epoch ISR
C 2 A, C
Leader
(epoch=2)
the new leader.
epoch=0
offset=0
epoch=1
offset=3
epoch=2
offset=5

r0 r1 r2 r7 r8
r0 r1 r2 r3 r4
r0 r1 r2 r3 r4 r9
A
B
C
Leader
(epoch=0)
Follower
(epoch=2)
Leader Epoch ISR
C 2 A, C
Leader
(epoch=2)
epoch=0
offset=0
epoch=1
offset=3
epoch=2
offset=5
for epoch=1?

r0 r1 r2 r7 r8
r0 r1 r2 r3 r4
r0 r1 r2 r3 r4 r9
A
B
C
Leader
(epoch=0)
Follower
(epoch=2)
Leader Epoch ISR
C 2 A, C
Leader
(epoch=2)
epoch=0
offset=0
epoch=1
offset=3
epoch=2
offset=5
for epoch=1?
C: Cool, no truncation needed!

r0 r1 r2 r7 r8
r0 r1 r2 r3 r4
r0 r1 r2 r3 r4 r9
A
B
C
Leader
(epoch=0)
Follower
(epoch=2)
Leader Epoch ISR
C 2 A, C
Leader
(epoch=2)

r0 r1 r2 r7 r8 r9
r0 r1 r2 r3 r4
r0 r1 r2 r3 r4 r9
A
B
C
Leader
(epoch=0)
Follower
(epoch=2)
Leader Epoch ISR
C 2 A, C
Leader
(epoch=2)

r0 r1 r2 r3
r0 r1 r2 r3 r4 r5 r6
r0 r1 r2
A
B
C
Leader
(epoch=0)
Follower
(epoch=0)
Leader Epoch ISR
A 0 A, B, C
Follower
(epoch=0)

r0 r1 r2 r3
r0 r1 r2 r3 r4 r5 r6
r0 r1 r2
A
B
C
Leader
(epoch=0)
Follower
(epoch=0)
Leader Epoch ISR
A 0 A, B, C
Follower
(epoch=0)
Follower A fails and is
removed from the ISR.

r0 r1 r2 r3
r0 r1 r2 r3 r4 r5 r6
r0 r1 r2
A
B
C
Leader
(epoch=0)
Follower
(epoch=0)
Leader Epoch ISR
A 0 B, C
Follower
(epoch=0)
Follower A fails and is
removed from the ISR.

r0 r1 r2 r3
r0 r1 r2 r3 r4 r5 r6
r0 r1 r2
A
B
C
Leader
(epoch=0)
Follower
(epoch=0)
Leader Epoch ISR
A 0 B, C
Follower
(epoch=0)

r0 r1 r2 r3
r0 r1 r2 r3 r4 r5 r6
r0 r1 r2
A
B
C
Leader
(epoch=0)
Follower
(epoch=0)
Leader Epoch ISR
A 0 B, C
Follower
(epoch=0)
Replica A could not re-register
in order to get the latest
leader/ISR state and continued
fetching from the current
leader.

r0 r1 r2 r3 r4
r0 r1 r2 r3 r4 r5 r6
r0 r1 r2
A
B
C
Leader
(epoch=0)
Follower
(epoch=0)
Leader Epoch ISR
A 0 B, C
Follower
(epoch=0)
Replica A could not re-register
in order to get the latest
leader/ISR state and continued
fetching from the current
leader.

r0 r1 r2 r3 r4
r0 r1 r2 r3 r4 r5 r6
r0 r1 r2
A
B
C
Follower
(epoch=0)
Leader Epoch ISR
A 0 B, C
Follower
(epoch=0)
Leader
(epoch=0)

r0 r1 r2 r3 r4
r0 r1 r2 r3 r4 r5 r6
r0 r1 r2
A
B
C
Follower
(epoch=0)
Leader Epoch ISR
C 1 C
Follower
(epoch=0)
Leader
(epoch=0)

r0 r1 r2 r3 r4
r0 r1 r2 r3 r4 r5 r6
r0 r1 r2
A
B
C
Follower
(epoch=0)
Leader Epoch ISR
C 1 C
Leader
(epoch=1)
Leader
(epoch=0)

r0 r1 r2 r3 r4
r0 r1 r2 r3 r4 r5 r6
r0 r1 r2 r7 r8 r9
A
B
C
Follower
(epoch=0)
Leader Epoch ISR
C 1 C
Leader
(epoch=1)
Leader
(epoch=0)

r0 r1 r2 r3 r4
r0 r1 r2 r3 r4 r5 r6
r0 r1 r2 r7 r8 r9
A
B
C
Follower
(epoch=0)
Leader Epoch ISR
C 1 C
Leader
(epoch=1)
Leader
(epoch=0)
Meanwhile, replica A still
thought B was the leader and
was still trying to make
progress

r0 r1 r2 r3 r4
r0 r1 r2 r3 r4 r5 r6
r0 r1 r2 r7 r8 r9
A
B
C
Follower
(epoch=0)
Leader Epoch ISR
C 1 C
Leader
(epoch=1)
Follower
(epoch=1)

r0 r1 r2 r3 r4
r0 r1 r2
r0 r1 r2 r7 r8 r9
A
B
C
Follower
(epoch=0)
Leader Epoch ISR
C 1 C
Leader
(epoch=1)
Follower
(epoch=1)

r0 r1 r2 r3 r4
r0 r1 r2 r7 r8 r9
r0 r1 r2 r7 r8 r9
A
B
C
Follower
(epoch=0)
Leader Epoch ISR
C 1 C
Leader
(epoch=1)
Follower
(epoch=1)

r0 r1 r2 r3 r4
r0 r1 r2 r7 r8 r9
r0 r1 r2 r7 r8 r9
A
B
C
Follower
(epoch=0)
Leader Epoch ISR
C 1 B, C
Leader
(epoch=1)
Follower
(epoch=1)

r0 r1 r2 r3 r4
r0 r1 r2 r7 r8 r9
r0 r1 r2 r7 r8 r9
A
B
C
Follower
(epoch=0)
Leader Epoch ISR
B 2 B, C
Leader
(epoch=1)
Follower
(epoch=1)
Once back in the ISR, the
controller elected it as leader

r0 r1 r2 r3 r4
r0 r1 r2 r7 r8 r9
r0 r1 r2 r7 r8 r9
A
B
C
Follower
(epoch=0)
Leader Epoch ISR
B 2 B, C
Leader
(epoch=1)
Leader
(epoch=2)
Once back in the ISR, the
controller elected it as leader

r0 r1 r2 r3 r4
r0 r1 r2 r7 r8 r9
r0 r1 r2 r7 r8 r9
A
B
C
Follower
(epoch=0)
Leader Epoch ISR
B 2 B, C
Leader
(epoch=1)
Leader
(epoch=2)
Suddenly, replica A was able to
make progress again!

r0 r1 r2 r3 r4 r9
r0 r1 r2 r7 r8 r9
r0 r1 r2 r7 r8 r9
A
B
C
Follower
(epoch=0)
Leader Epoch ISR
B 2 B, C
Leader
(epoch=1)
Leader
(epoch=2)
Suddenly, replica A was able to
make progress again!

Reflection
● Our mushy brains are not equipped to thinking
about edge cases in distributed systems
● How do we know that our fixes are not just
trading one edge case for another?
● How do we know there are not more edge
cases?

TLA+/TLC
● TLA+ is a specification language
created by Leslie Lamport
● TLC is a model checker
● Think “brute force proof by
mathematical induction”

TLA+/TLCUsing LaTeX syntax makes
model checking just as much
fun as writing research papers!● TLA+ is a specification language
created by Leslie Lamport
● TLC is a model checker
● Think “brute force proof by
mathematical induction”

● Define the state and how to initialize it
● Define the valid state transitions
● Define expected state invariants
● Run model to check invariants
Model
Checklist

r0 r1 r2 r3
r0 r1 r2 r3 r4 r5 r6
r0 r1 r2 r3 r4 r5
A
B
C
Leader
(epoch=0)
Follower
(epoch=0)
Leader Epoch ISR
B 0 A, B, C
Follower
(epoch=0)
1. Records and the log

Log
Representation LogRecords == [
id: Nat,
epoch: Nat
]

Log
Representation LogRecords == [
id: Nat,
epoch: Nat
]
Log == [
endOffset: Nat,
records: [Nat -> LogRecords]
]

r0 r1 r2 r3
r0 r1 r2 r3 r4 r5 r6
r0 r1 r2 r3 r4 r5
A
B
C
Leader
(epoch=0)
Follower
(epoch=0)
Leader Epoch ISR
B 0 A, B, C
Follower
(epoch=0)
2. Replica State

Replica State
Representation CONSTANT Replicas

Replica State
Representation CONSTANT Replicas * {r1, r2, r3}

Replica State
ReplicaState == [
log: Log,
hw: Nat,
leaderEpoch: Nat,
leader: Replicas,
isr: SUBSET Replicas
]

Replica State
ReplicaState == [
log: Log,
hw: Nat,
leaderEpoch: Nat,
leader: Replicas,
]
AllReplicaStates ==
[Replicas -> ReplicaState]

r0 r1 r2 r3
r0 r1 r2 r3 r4 r5 r6
r0 r1 r2 r3 r4 r5
A
B
C
Leader
(epoch=0)
Follower
(epoch=0)
Leader Epoch ISR
B 0 A, B, C
Follower
(epoch=0)
2. Replica State
3. Quorum State

Quorum State
Representation QuorumState == [
leaderEpoch: Nat,
leader: Replicas,
]

r0 r1 r2 r3
r0 r1 r2 r3 r4 r5 r6
r0 r1 r2 r3 r4 r5
A
B
C
Leader
(epoch=0)
Follower
(epoch=0)
Leader Epoch ISR
B 0 A, B, C
Follower
(epoch=0)
2. Replica State
3. Quorum State
4. LeaderAndIsr Propagation

Leader/ISR
Propagation LeaderAndIsrRequests ==
SUBSET QuorumState

Leader/ISR
SUBSET QuorumState
leaderAndIsrRequests: {}
Example: initialization

Leader/ISR
SUBSET QuorumState
leaderAndIsrRequests: {
[leader: A, epoch: 0, isr: {A, B, C}]
}
Example: after first leader election

Leader/ISR
SUBSET QuorumState
leaderAndIsrRequests: {
[leader: A, epoch: 0, isr: {A, B, C}],
[leader: B, epoch: 1, isr: {B, C}]
}
Example: after leader failure and reelection

Next ==
/ ControllerElectLeader
/ ControllerShrinkIsr
/ ReplicaBecomeLeader
/ LeaderExpandIsr
/ LeaderShrinkIsr
/ LeaderWrite
/ LeaderIncHighWatermark
/ ReplicaBecomeFollower
/ FollowerFetch
State
Transitions

Next ==
/ LeaderExpandIsr
/ LeaderShrinkIsr
/ LeaderWrite
/ FollowerFetch
State
Transitions Controller actions

Next ==
/ LeaderExpandIsr
/ LeaderShrinkIsr
/ LeaderWrite
/ FollowerFetch
State
Transitions
Leader actions

Next ==
/ LeaderExpandIsr
/ LeaderShrinkIsr
/ LeaderWrite
/ FollowerFetch
State
Transitions
Follower actions

State
Transitions
Start off with empty logs, a full ISR, and
no leader
Init

State
Transitions
Init
ControllerElectLeader
The first enabled action is leader election.

State
Transitions
Init
Electing the first leader enables several
new state transitions

State
Transitions
Init
ReplicaBecomeLeader

State
Transitions
Init
ReplicaBecomeLeader
ReplicaBecomeFollower

State
Transitions
Init
ReplicaBecomeLeader

State
Transitions
Init
ReplicaBecomeLeader
Every transition enables a different set
of next actions.

State
Transitions
Init
ReplicaBecomeLeader
of next actions.
LeaderWrite ReplicaBecomeFollower
ControllerShrinkIsr

State
Transitions
Init
ReplicaBecomeLeader
of next actions.
LeaderWrite ReplicaBecomeFollower
ControllerShrinkIsr FollowerFetch
LeaderShrinkIsr

State
Transitions
Init
ReplicaBecomeLeader
ReplicaBecomeLeader
LeaderWrite
FollowerFetch

State
Transitions
Init
ControllerElectLeader(epoch=0)
ControllerShrinkIsr
ControllerElectLeader(epoch=1)
ReplicaBecomeLeader(epoch=0)
LeaderWrite(epoch=0)
ReplicaBecomeFollower(epoch=1)
ControllerShrinkIsr

Replication
Invariant StrongIsr == A r1 in Replicas:
/ ~ ReplicaPresumesLeadership(r1)
/ LET hw == replicaState[r1].hw
IN A r2 in quorumState.isr:
HasMatchingLogsUpTo(r1, r2, hw)

Replication
Invariant StrongIsr == A r1 in Replicas:
/ ~ ReplicaPresumesLeadership(r1)
/ LET hw == replicaState[r1].hw
IN A r2 in quorumState.isr:
HasMatchingLogsUpTo(r1, r2, hw)
“If any replica is eligible to return data, then that data
must be replicated to all members of the current ISR”

r0 r1 r2 r3
r0 r1 r2 r3 r4 r5 r6
r0 r1 r2 r3 r4 r5
A
B
C
Leader
(epoch=0)
Follower
(epoch=1)
Leader Epoch ISR
C 1 A, C
Leader
(epoch=1)
Leader A had failed and
replica C was being elected
as the new leader.

r0 r1 r2 r3
r0 r1 r2 r3 r4 r5 r6
r0 r1 r2 r3 r4 r5
A
B
C
Leader
(epoch=0)
Follower
(epoch=1)
Leader Epoch ISR
C 1 A, C
Leader
(epoch=1)
Upon becoming a follower
of C, replica A would
truncate its log to the local
high watermark.

r0 r1
r0 r1 r2 r3 r4 r5 r6
r0 r1 r2 r3 r4 r5
A
B
C
Leader
(epoch=0)
Follower
(epoch=1)
Leader Epoch ISR
C 1 A, C
Leader
(epoch=1)

r0 r1
r0 r1 r2 r3 r4 r5 r6
r0 r1 r2 r3 r4 r5
A
B
C
Leader
(epoch=0)
Follower
(epoch=1)
Leader Epoch ISR
C 1 A, C
Leader
(epoch=1)
This state violates the
StrongIsr property because
leader C is eligible to return
records r2 and r3, though
they are not present on A.

Edge Case 4
(Premature ISR expansion)

r0 r1
r0 r1 r2 r3 r4 r5 r6
r0 r1 r2 r3 r4 r5
A
B
C
Leader
(epoch=0)
Follower
(epoch=0)
Leader Epoch ISR
B 0 B, C
Follower
(epoch=0)
The leader is B and replica
A is trying to catch up to
rejoin the ISR.

r0 r1
r0 r1 r2 r3 r4 r5 r6
r0 r1 r2 r3 r4 r5
A
B
C
Leader
(epoch=0)
Follower
(epoch=0)
Leader Epoch ISR
C 1 B, C
Follower
(epoch=0)
The leader changes to C.

r0 r1
r0 r1 r2 r3 r4 r5 r6
r0 r1 r2 r3 r4 r5
A
B
C
Leader
(epoch=0)
Follower
(epoch=0)
Leader Epoch ISR
C 1 B, C
Leader
(epoch=1)
The leader changes to C.

r0 r1
r0 r1 r2 r3 r4 r5 r6
r0 r1 r2 r3 r4 r5
A
B
C
Leader
(epoch=0)
Follower
(epoch=1)
Leader Epoch ISR
C 1 B, C
Leader
(epoch=1)
Follower A catches up and
rejoins the ISR.

r0 r1 r2
r0 r1 r2 r3 r4 r5 r6
r0 r1 r2 r3 r4 r5
A
B
C
Leader
(epoch=0)
Follower
(epoch=1)
Leader Epoch ISR
C 1 B, C
Leader
(epoch=1)
rejoins the ISR.

r0 r1 r2
r0 r1 r2 r3 r4 r5 r6
r0 r1 r2 r3 r4 r5
A
B
C
Leader
(epoch=0)
Follower
(epoch=1)
Leader Epoch ISR
C 1 A, B, C
Leader
(epoch=1)
rejoins the ISR.

r0 r1 r2
r0 r1 r2 r3 r4 r5 r6
r0 r1 r2 r3 r4 r5
A
B
C
Leader
(epoch=0)
Follower
(epoch=1)
Leader Epoch ISR
C 1 A, B, C
Leader
(epoch=1)
This violates StrongIsr
because replica B may
have returned records r3,
r4, and r5 which A does not
yet have.

r0 r1 r2
r0 r1 r2 r3 r4 r5 r6
r0 r1 r2 r3 r4 r5
A
B
C
Leader
(epoch=0)
Follower
(epoch=1)
Leader Epoch ISR
C 1 B, C
Leader
(epoch=1)
After becoming leader, C
only knows that the true
high watermark is between
its own high watermark and
the end of the log.
True high
watermark

r0 r1 r2
r0 r1 r2 r3 r4 r5 r6
r0 r1 r2 r3 r4 r5
A
B
C
Leader
(epoch=0)
Follower
(epoch=1)
Leader Epoch ISR
C 1 B, C
Leader
(epoch=1)
So we wait until the
follower has reached the
starting offset of this
leader’s own epoch before
allowing it into the ISR.
True high
watermark

r0 r1 r2
r0 r1 r2 r3 r4 r5
r0 r1 r2 r3 r4 r5 r7 r8
A
B
C
Follower
(epoch=1)
Follower
(epoch=1)
Leader Epoch ISR
C 1 B, C
Leader
(epoch=1)
True high
watermark

r0 r1 r2 r3 r4 r5 r7
r0 r1 r2 r3 r4 r5
r0 r1 r2 r3 r4 r5 r7 r8
A
B
C
Follower
(epoch=1)
Follower
(epoch=1)
Leader Epoch ISR
C 1 B, C
Leader
(epoch=1)
True high
watermark

r0 r1 r2 r3 r4 r5 r7
r0 r1 r2 r3 r4 r5
r0 r1 r2 r3 r4 r5 r7 r8
A
B
C
Follower
(epoch=1)
Follower
(epoch=1)
Leader Epoch ISR
C 1 A, B, C
Leader
(epoch=1)
True high
watermark

r0 r1 r2 r3
r0 r1 r2 r5 r6
r0 r1 r2 r5 r6
A
B
C
Follower
(epoch=0)
Leader Epoch ISR
B 2 B, C
Leader
(epoch=1)
Leader
(epoch=2)
Replica A was a zombie
which was still fetching
from B. After a couple
leader elections, replica B
became the leader again.

r0 r1 r2 r3
r0 r1 r2 r5 r6
r0 r1 r2 r5 r6
A
B
C
Follower
(epoch=0)
Leader Epoch ISR
B 2 B, C
Leader
(epoch=1)
Leader
(epoch=2)
A -> B:
Fetch(offset=4, epoch=0)

r0 r1 r2 r3
r0 r1 r2 r5 r6
r0 r1 r2 r5 r6
A
B
C
Follower
(epoch=0)
Leader Epoch ISR
B 2 B, C
Leader
(epoch=1)
Leader
(epoch=2)
A -> B:
Fetch(offset=4, epoch=0)
B -> A:
You are fenced!

KIP-320
Model
Checking
Results
#Replicas Log Size Distinct States Depth Duration
3 3 84,313,696 40 ~2 hours
3 4 133,768,793 20 ~3 hours
4 4 200,534,415 18 ~6 hours

Summary
● Distributed systems are subtle and we are
poorly equipped to reason about edge cases.
● Model checking is a systematic approach to
finding these edge cases and verifying our
fixes address them.
● All of the replication fixes we know of will be
available in Apache Kafka 2.1.0.

Note of
Caution ● The model is not the implementation.
● The implementation will have complexity that
the model cannot capture.

● Kafka TLA+ Specification:
https://github.com/hachikuji/kafka-specification
● TLA+ video tutorial:
https://lamport.azurewebsites.net/video/videos.html
● Kafka Improvement Proposals:
○ KIP-101:
https://cwiki.apache.org/confluence/display/KAFKA/K
IP-101+-+Alter+Replication+Protocol+to+use+Leader+
Epoch+rather+than+High+Watermark+for+Truncation
○ KIP-279:
IP-279%3A+Fix+log+divergence+between+leader+and
+follower+after+fast+leader+fail+over
○ KIP-320:
IP-320%3A+Allow+fetchers+to+detect+and+handle+lo
g+truncation
Resources

r0 r1 r2
r0 r1 r2 r3
r0 r1 r2 r3
A
B
C
Leader
(epoch=0)
Follower
(epoch=1)
Leader Epoch ISR
C 1 B, C
Leader
(epoch=1)
B became a zombie while it
was the leader for epoch 0.

r0 r1 r2
r0 r1 r2 r3
r0 r1 r2 r3 r7 r8
A
B
C
Leader
(epoch=0)
Follower
(epoch=1)
Leader Epoch ISR
C 1 B, C
Leader
(epoch=1)
The new leader will be
accepting writes.

r0 r1 r2
r0 r1 r2 r3 r9 r10
r0 r1 r2 r3 r7 r8
A
B
C
Leader
(epoch=0)
Follower
(epoch=1)
Leader Epoch ISR
C 1 B, C
Leader
(epoch=1)
The old leader may accept
writes as well!

r0 r1 r2
r0 r1 r2 r3 r9 r10
r0 r1 r2 r3 r7 r8
A
B
C
Leader
(epoch=0)
Follower
(epoch=1)
Leader Epoch ISR
C 1 B, C
Leader
(epoch=1)
As long as the leader
cannot advance its high
watermark, there is no
semantic violation.

r0 r1 r2
r0 r1 r2 r3 r9 r10
r0 r1 r2 r3 r7 r8
A
B
C
Leader
(epoch=0)
Follower
(epoch=1)
Leader Epoch ISR Ver
C 1 B, C 1
Leader
(epoch=1)
As long as the leader
cannot advance its high
watermark, there is no
semantic violation.

r0 r1 r2
r0 r1 r2 r3 r9 r10
r0 r1 r2 r3 r7 r8
A
B
C
Leader
(epoch=0)
Follower
(epoch=1)
C 1 B, C 1
Leader
(epoch=1)
The controller sends the
latest version of the leader
and ISR state to replicas in
the LeaderAndIsr request

r0 r1 r2
r0 r1 r2 r3 r9 r10
r0 r1 r2 r3 r7 r8
A
B
C
Leader
(epoch=0,
version=0)
Follower
(epoch=1)
C 1 B, C 1
Leader
(epoch=1,
version=1)
The controller sends the
latest version of the leader
and ISR state to replicas in
the LeaderAndIsr request

r0 r1 r2
r0 r1 r2 r3 r9 r10
r0 r1 r2 r3 r7 r8
A
B
C
Leader
(epoch=0,
version=0)
Follower
(epoch=1)
C 1 B, C 1
Leader
(epoch=1,
version=1)
This allows for CAS
updates, which effectively
fences replicas which have
old state.

Appendix 2:
What goes in a TLA+ Model?

VARIABLES var1, var2, …
Init ==
/ var1 = 1
/ …
Action1 ==
/ var1 leq 10
/ var1’ = var + 1
…
Next ==
/ Action1
/ Action2
/ …
Spec == Init / []Next
Invariant ==
/ var1 geq 1
/ …
TLA+
Overview

Init ==
/ var1 = 1
/ …
Action1 ==
/ var1 leq 10
/ var1’ = var + 1
…
Next ==
/ Action1
/ Action2
/ …
Invariant ==
/ var1 geq 1
/ …
TLA+
Overview
Define the model’s state

Init ==
/ var1 = 1
/ …
Action1 ==
/ var1 leq 10
/ var1’ = var + 1
…
Next ==
/ Action1
/ Action2
/ …
Invariant ==
/ var1 geq 1
/ …
TLA+
Overview
Specify how the state is
initialized

Init ==
/ var1 = 1
/ …
Action1 ==
/ var1 leq 10
/ var1’ = var + 1
…
Next ==
/ Action1
/ Action2
/ …
Invariant ==
/ var1 geq 1
/ …
TLA+
Overview
Specify the valid state
transitions

Init ==
/ var1 = 1
/ …
Action1 ==
/ var1 leq 10
/ var1’ = var + 1
…
Next ==
/ Action1
/ Action2
/ …
Invariant ==
/ var1 geq 1
/ …
TLA+
Overview
Specify the set of valid
state transitions

Init ==
/ var1 = 1
/ …
Action1 ==
/ var1 leq 10
/ var1’ = var + 1
…
Next ==
/ Action1
/ Action2
/ …
Invariant ==
/ var1 geq 1
/ …
TLA+
Overview
The specification is the
conjunction of the initial state
and all the states reachable
by repeatedly applying the
`Next` state transition

Init ==
/ var1 = 1
/ …
Action1 ==
/ var1 leq 10
/ var1’ = var + 1
…
Next ==
/ Action1
/ Action2
/ …
Invariant ==
/ var1 geq 1
/ …
TLA+
Overview
Define the model invariants
that should hold after every
state transition

Appendix 3:
Buggy Replication Optimizations

Hardening Kafka Replication

Recommended

More Related Content

What's hot (20)

More from confluent (20)

Recently uploaded (20)

Hardening Kafka Replication