A New Age of JVM Garbage Collectors (Clojure Conj 2019)

A New Age of JVM
Garbage Collectors
Alexander Yakushev
@unlog1c
2019

Turns out there are already many talks about new JVM GCs
2

3

4

5

6

7
❑ done by smarter people
❑ including authors of those GCs
❑ and who can explain the details much better
So, what’s the point of making another one?

But most talks don’t explain the foundations
8
Without them, you can’t fully appreciate the challenges and
advances in GC technology.
Hence, I decided to do a different talk.

What you should know entering
A New Age of JVM
Garbage Collectors
Alexander Yakushev
@unlog1c
2019

Who am I?
10
❑ I am not a GC writer.
❑ Nor am I a GC expert.
❑ I am a GC power user (= I need a lot from a GC).

Who am I?
11
❑ I am not a GC writer.
❑ Nor am I a GC expert.
❑ I am a GC power user (= I need a lot from a GC).
Why? Because:
❑ I build and run infrastructure for natural language processing.
❑ Grammarly handles humongous amounts of text, searching
for mistakes and enhancements.
❏ For 20 million daily active users.

Typical day of a typical EC2 instance
12
❑ Multiplied by many-many instances.
❑ Everything scales horizontally – except your wallet.
❑ Not only throughput – we do this in real time.
❏ So, GC must both be fast and have adequate latency.

One book to learn it all
13
Jones, Hosking, Moss
“The Garbage Collection Handbook”
Gives you enough knowledge to
follow and understand the recent
developments in the GC space.

Why manage memory?
Manual memory management
Automatic reference counting
Mark-Sweep-Compact garbage collector
Evacuating garbage collector
G1 garbage collector
Shenandoah garbage collector
Conclusions
14

No, let’s take a better CPU
16

CPUs only work with numbers
17
+ - × ÷
35 :RCX
RAX: 21

What if it’s not numbers?
18
String s = "Hello " + "Conj";
6 H e l l o ␣ 4 C o n j ...
Memory
0 7
RAX: 0
7 :RCX

19
RAX: 0
7 :RCX
6 H e l l o ␣ 4 C o n j 10 H e l l o ␣ C o n j ...
Memory
0 7 12
RDX: 12

20
RAX: 0
7 :RCX
6 H e l l o ␣ 4 C o n j 10 H e l l o ␣ C o n j ...
Memory
0 7 12
RDX: 12
Not needed anymore

Why memory management is necessary?
21
❑ Memory is a finite resource.
❏ What is unused has to be freed.
❏ Otherwise you will run out of it.

Why manage memory?
Conclusions
22

23
❑ Language example: C.
❑ OS gives a programmer two functions:
allocate(size) -> pointer
free(pointer) -> void

24
Memory
used unused

25
Memory
// Allocate 2048 bytes.
int* ptr = allocate(2048);
ptr

26
Memory
// Work with it
// .......
ptr

27
Memory
// Work with it
// .......
free(ptr);

Manual memory management is amazing!
❑ No hidden pauses.
❏ Lowest latency.
❑ No redundant code around memory reads and writes.
❏ Maximal throughput.
❑ Doesn’t require any extra memory.

Problems with manual memory management
❑ What happens if you forget to free the memory block?

❏ Memory leak.

❏ Memory leak.
❑ What happens if you access the memory block you have
already freed (or never allocated)?

❏ Memory leak.
❑ What happens if you access the memory block you have
already freed (or never allocated)?
❏ Good luck: SIGSEGV (segmentation fault), program
crashes.
❏ Bad luck: program reads incorrect memory from wrong
address and continues working in corrupted state.

Is allocation fast?
33
Memory
❑ Example behavior of a naive allocator:

Is allocation fast?
34
Memory

Is allocation fast?
35
Memory

Is allocation fast?
36
Memory

Is allocation fast?
37
Memory

Is allocation fast?
38
Memory

Is allocation fast?
39
Memory

Is allocation fast?
40
Memory

Is allocation fast?
41
Memory

Is allocation fast?
42
Memory

Is allocation fast?
43
Memory
?

Is allocation fast?
44
Memory
Allocation failure.

Fragmentation
45
❑ Fragmentation – you have enough total free memory, but
not a single contiguous block to fit allocation.

Fragmentation
46
❑ To fight fragmentation, allocators are usually more complex.
❏ Example: segregated fits allocator.

Fragmentation
47
❑ Complex allocator => slower allocation speed.

Fragmentation
48
❑ Complex allocator => slower allocation speed.
❑ =>
❑ Allocations are not cheap.
❏ Avoided if possible.

Manual memory management conclusions
49
❑ Fast (both in latency and throughput)
❑ Uses only so much memory as really necessary.
But:
❑ Unsafe (with possibility for improvement, e.g. Rust)
❑ Less convenient

Why manage memory?
Conclusions
50

51
❑ Language example: Objective C, Swift.
❑ Each heap object has an number that stores how
many references to that object exist.
"Hello Conj"1
String s = “Hello Conj”;

52
❑ Each heap object has an integer that stores how many
references to object exist.
"Hello Conj"2
String s2 = s;

53
"Hello Conj"1
String s2 = s;
s = null;

54
"Hello Conj"0
String s2 = s;
s = null;
s2 = null;

55
"Hello Conj"0
String s2 = s;
s = null;
s2 = null;

56
❑ Refcount has to be updated on every:
❏ Field reference write
❏ Local variable write
❏ Function entrance/exit
❏ Block entrance/exit

Automatic reference counting is amazing!
❑ It’s automatic!
❑ Cost of memory management spread across the program –
no surprise latencies.
❑ Prompt – doesn’t need much extra memory.
57

ARC: what is not so good
❑ Throughput is lower than with manual mgmt.
❏ Low-level operations have to do extra work.
❏ Refcount has to be atomic.
58

ARC: what is not so good
❑ Throughput is lower than with manual mgmt.
❏ Low-level operations have to do extra work.
❏ Refcount has to be atomic.
❑ Fragmentation possibility is still there.
❏ Needs a complex allocator.
❏ Allocations are still not cheap.
59

Cycle references
ARC can’t free a cycle -> it’s a memory leak.
<Parent>1
<Child>1
60

Automatic reference counting conclusions
61
❑ Great latency
❑ Uses that much memory as needed (except for refcounts)
❑ Safe and convenient
But:
❑ Worse throughput
❑ Allocations are still expensive
❑ Pitfalls (e.g., cycles)

Why manage memory?
Conclusions
62

Mark-Sweep-Compact GC
63
❑ Language example: Java.
❑ Idea behind any tracing GC – find all live objects,
remove all the rest.

Phase 1: Mark
64
❑ The marking starts from the roots.
❑ GC roots in Java:
❏ Static fields
❏ Active threads (local variables)
❏ Classloaders
❏ ...

Phase 1: Tri-color marking
65
Object graph
❑ Black – marked
❑ Gray – wavefront
❑ White – unmarked
❑ Highlighted – roots

66
Object graph

67
Object graph

68
Object graph

69
Object graph

Phase 1: Mark
70
❑ After Mark, all reachable objects in the heap are marked.
❑ Complexity is O(live objects).

Phase 2: Sweep-Compact
71
❑ Walk over each object in the heap.
❏ If it is alive, compute new location for it and store in the
object.
❏ If it’s dead, mark area as free for relocation.
❑ Walk over each live (marked) object in the heap.
❏ Update pointers to to-be relocated objects to new
addresses.
❑ Walk over each live object again.
❏ Move objects to their new destinations.

72
Memory
Dead objects

73
Memory
Step 1: remove dead objects

74
Memory
Step 2: calculate new addresses for live objects

75
Memory
Step 3: update references

76
Memory
Step 4: move live objects to their new positions

Why compaction is beneficial?
77
❑ Allows to use a simpler allocator.
❏ “Pointer bump”
❏ So, allocations are fast.
❑ Improves memory locality.

Mark-Sweep-Compact is amazing!
78
❑ Safe and convenient.
❑ Fast allocations.
❑ Good cache coherence because of compaction.
❑ Complete – doesn’t leave garbage behind (like cycles).
❑ Object access has no overhead.
But...

Pauses
79
❑ While the GC runs, application threads are suspended.
❑ Pauses can be up to seconds or even tens of seconds.
❑ Sweep-Compact is O(heap size).
❏ Larger heap -> longer pauses.

Heap headroom
80
❑ The more dead objects are reclaimed per run, the lower is
time per object.
❑ For tracing GC to be efficient, there must be extra heap
space.
❏ 3-5x of the live data set.
❑ At the same time, larger heap leads to longer pauses.

Why manage memory?
Conclusions
81

Evacuating GC
82
❑ Also known as Scavenging or Semispace Copying GC.
❑ Language example: Java.
❑ Idea: don’t compact objects in-place.
❑ Instead, have two spaces (active and passive) and flip
between them.

Phase 1: Mark
83
❑ Identical Mark as in Mark-Sweep-Compact.

Phase 2: Evacuate
84
Active space
Dead objects
Passive space

Phase 2: Evacuate
85
Active space
Passive space
Step 1: calculate new addresses for live objects

Phase 2: Evacuate
86
Active space
Passive space
Step 2: copy objects and update references

Phase 2: Evacuate
87
Passive space
Active space
Step 3: flip spaces and clear passive space

Phase 2: Evacuate
88
Passive space
Active space
Step 3: flip spaces and clear passive space

Evacuating GC is amazing!
89
❑ All benefits from Mark-Sweep-Compact, plus:
❑ Complexity is O(live objects)
But…
❑ Pauses are still there.
❑ Needs twice as much memory as Mark-Sweep-Compact.

Why manage memory?
Conclusions
90

G1 GC
91
❑ Available in Java since JDK6.
❏ Enabled by default since JDK9.
❑ Generational garbage collector.

Generational collectors
92
❑ Weak generational hypothesis:
❏ “Most objects die young”.
❑ This property can be used when designing a GC.

G1 GC heap
93
Eden
Survivor space
Tenured space

G1 GC heap
94
Eden (evacuating GC to Survivor space)
Survivor space (evacuating GC to Tenured space)
Tenured space (mark-sweep-compact GC)

G1 GC
95
❑ Eden
❏ Evacuating GC is efficient if there are few live objects.
❏ Most objects die young.
❏ = Fast Eden collection.

G1 GC
96
❑ Eden
❑ Survivor space
❏ Acts as an evacuation destination for Eden.
❏ Smaller than Eden because most objects will be dead.

G1 GC
97
❑ Eden
❑ Survivor space
❏ Acts as an evacuation destination for Eden.
❏ Smaller than Eden because most objects will be dead.
❑ Tenured space
❏ Slower to collect.
❏ Collected less frequently than Eden and Survivor space.

G1 GC phases
98
❑ Eden evacuation
❑ Survivor space evacuation
❑ Tenured space marking
❑ Tenured space compaction

G1 GC phases
99
❑ Eden evacuation (Stop-the-World)
❑ Survivor space evacuation (Stop-the-World)

G1 GC phases
100
❑ Eden evacuation
❑ Tenured space marking (concurrent)

G1 GC phases
101
❑ Eden evacuation
❑ Tenured space compaction (Stop-the-World)

Concurrent marking
102
❑ Idea: don’t stop the application during Mark phase.
❑ Problem: racing against the application to walk the object
graph.

Concurrent marking woes
103
Object graph

104
Object graph

105
Object graph

Concurrent marking
106
❑ Idea: don’t stop the application when during Mark phase.
❑ Problem: racing against the application to walk the object
graph.
❑ Solution: track reference changes while Marking.
❏ G1 uses Snapshot-At-The-Beginning (SATB) algorithm
for that.
❏ Requires extra code on each field write and allocation.
❏ Extra code = reduced throughput.

G1 GC is amazing!
107
❑ Evacuating GC for young generation yields quite short pauses.
❑ Survivor space is much smaller than Eden.
❏ Less extra memory than in classic Evacuation alg.
❑ Old generation marking is concurrent.
But:
❑ STW pauses are still there and can last seconds.

Why manage memory?
Conclusions
108

Shenandoah GC
109
❑ Developed by Red Hat.
❑ Available in OpenJDK since 12.
❏ Backports available for JDK8 and JDK11.
❑ Mostly concurrent garbage collector.

Shenandoah GC
110
❑ Two phases:
❏ Concurrent marking
❏ Concurrent evacuation

Concurrent marking
111
❑ Same as in G1
❏ Intercept allocations and field writes during Marking.

Concurrent evacuation
112
❑ Idea: don’t stop the application during evacuation phase.
❑ Problem: racing against the application to copy the object
and update all references to it.

Concurrent evacuation woes
113
Active space
x = 3
Passive space

114
Active space
x = 3
Passive space
copy

115
Active space
x = 3
Passive space
x = 3
copy

116
Active space
x = 4
Passive space
x = 3
write

117
Active space
x = 4
Passive space
x = 3
write
Can’t remove old object anymore,
otherwise we will lose a write.

Concurrent evacuation: to-space invariant
118
❑ Idea: don’t stop the application during evacuation phase.
❑ Problem: racing against the application to copy the object
and update all references to it.
❑ Solution: add a Load Reference Barrier (LRB) code to all
object accesses.
❏ Check if the loaded object is in evacuation set.
❏ Check if the loaded object points to a copy.
❏ If true, update the reference to a copy.
❏ Only then, perform a read or write.

119
Active space
Passive space
x = 3

120
Active space
Passive space
x = 3
copy

121
Active space
Passive space
x = 3
x = 3
copy

122
Active space
Passive space
x = 3
x = 3
LRB

123
Active space
Passive space
x = 3
x = 3

124
Active space
Passive space
x = 4
x = 3
write

125
Active space
Passive space
x = 4
x = 3
copy

126
Passive space
Active space
x = 4

Concurrent evacuation: cost
127
❑ Inflating each object access code is expensive.
❑ Throughput degrades by 0-15%, depending on the app.

Shenandoah GC: pauses
128
❑ 4 very short STW pauses per cycle.
❑ Average pause – ~1ms.
❑ Max pause (when everything goes well) – 50-100ms.

Shenandoah GC: is it worth it?
129

Shenandoah GC is amazing!
130
❑ Used by us in production for 1 year.
❑ With heap sizes of up to 80g.
❑ Pauses from 1ms to 50ms.
❏ Same setup, G1 pauses between 50ms and 5 seconds.
❑ Throughput overhead between 0% (network-bound app) and
20% (CPU-bound app).

Shenandoah GC: failure modes
131
❑ Main problem: when allocation rate exceeds GC rate.
❑ In failure mode, Shenandoah may inject STW pauses into
individual threads or even do full STW GC.
❑ Use jvm-alloc-rate-meter to track your allocation rate.
https://github.com/clojure-goes-fast/jvm-alloc-rate-meter

Why manage memory?
Conclusions
132

Conclusions
133
❑ Memory management is about trading off between:
❏ Safety and convenience
❏ Throughput
❏ Allocation speed
❏ Average latency
❏ Maximum latency
❏ Memory overhead
❏ ...
❑ Understanding how and why such tradeoffs are made is
important when doing high-performance work.

What about Clojure?
134
❑ GC stuff is hard. We get it for free.
❑ Clojure programs often allocate more than Java programs
(because of immutability).
❏ Watch and know your bottlenecks.
❏ jvm-hiccup-meter
https://github.com/clojure-goes-fast/jvm-hiccup-meter
❏ jvm-alloc-rate-meter
https://github.com/clojure-goes-fast/jvm-alloc-rate-meter
❏ clj-async-profiler
https://github.com/clojure-goes-fast/clj-async-profiler

References (and follow-up material)
135
❑ Jones, Hosking, Moss: The Garbage Collection Handbook (2012)
❑ Gil Tene: Understanding Java Garbage Collection https://www.youtube.com/watch?v=Uj1_4shgXpk
❑ Detlefs et al. “Garbage-First Garbage Collection”
http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.63.6386&rep=rep1&type=pdf
❑ G1GC Fundamentals: Lessons from Taming Garbage Collection
https://product.hubspot.com/blog/g1gc-fundamentals-lessons-from-taming-garbage-collection
❑ “Shenandoah GC home page” https://wiki.openjdk.java.net/display/shenandoah/Main
❑ Shenandoah GC: The Great Revolution (slides)
https://shipilev.net/talks/jugbb-Sep2019-shenandoah.pdf
❑ Aleksey Shipilev: What We Know In 2018 https://www.youtube.com/watch?v=qBQtbkmURiQ
❑ Shenandoah GC in production: experience report
http://clojure-goes-fast.com/blog/shenandoah-in-production/
❑ Shenandoah GC in JDK 13, Part 1: Load reference barriers
developers.redhat.com/blog/2019/06/27/shenandoah-gc-in-jdk-13-part-1-load-reference-barriers
❑ Per Linden: ZGC: A Scalable Low-Latency Garbage Collector
https://www.youtube.com/watch?v=kF_r3GE3zOo
❑ Monica Beckwith: In the New Age of Concurrent GCs
https://www.youtube.com/watch?v=8m0RpROUQJE
❑ Jean Philippe Bempel: Understanding Low Latency JVM GCs
https://www.youtube.com/watch?v=MU8NapbG1IQ

Thank you!
twitter.com/unlog1c
clojure-goes-fast.com

A New Age of JVM Garbage Collectors (Clojure Conj 2019)

More Related Content

Similar to A New Age of JVM Garbage Collectors (Clojure Conj 2019) (20)

Recently uploaded (20)

A New Age of JVM Garbage Collectors (Clojure Conj 2019)