What to do if Your Kafka Streams App Gets OOMKilled? with Andrey Serebryanskiy

What to Do if Your
Kafka Streams App Gets OOMKilled?
Andrey Serebryanskiy

{
Andrey Serebryanskiy
Streaming Platform Owner at Raiffeisen Bank
What to Do if Your Kafka
Streams App Gets
OOMKilled?

The problem
3
Kafka Kafka Streams App
kubernetes
RocksDB

What is the problem with
this app?

Launch with resource limits
5
helm/templates/deployment.yaml
...
resources:
limits:
memory: 256Mi
requests:
memory: 128Mi
command:
- java
args:
- -jar
- app.jar
...

How to check if it is OOMKilled?
6
kubectl describe pod your-pod-name -n your-namespace
Name: your-pod-name
...
Containers:
app:
...
Last State: Terminated
Reason: OOMKilled
Exit Code: 137

Example app
8
Simple Kafka Streams topology
Application.java
public static void main(String[] args) {
var builder = new StreamsBuilder();
var stream = builder.stream(INPUT_TOPIC);
builder.addStateStore(Stores.keyValueStoreBuilder(
Stores.persistentKeyValueStore(STATE_STORE_NAME));
var persistedStream = stream.transformValues((readOnlyKey, val) -> {
...
stateStore.put(readOnlyKey, val);
...
}, STATE_STORE_NAME);
persistedStream.foreach((key, val) -> ...);
var topology = builder.build();
var kafkaStreams = new KafkaStreams(topology);
runApp(kafkaStreams);
}

Example app
9
Application.java
...
...
persistedStream.foreach((key, val) -> ...);
}

Example app
10
Application.java
...
...
persistedStream.foreach((key, val) -> logMessageCount());
}

Example app
11
Application.java
...
...
persistedStream.foreach((key, val) -> logMessageCount());
}

Example app
12
Application.java
...
}
Please find full application code here:
https://github.com/a-serebryanskiy/kafka-streams-oom-killed

So you have your app
OOMKilled

Launch with resource limits
14
...
resources:
limits:
memory: 256Mi
requests:
memory: 128Mi
command:
- java
args:
- -jar
- app.jar
...

Let’s add heap limits
15
...
resources:
limits:
memory: 256Mi
requests:
memory: 128Mi
command:
- java
args:
- -XshowSettings:VM
- -XX:MinRAMPercentage=50.0
- -jar
- app.jar
...

Let’s add heap limits
16
...
resources:
limits:
memory: 256Mi
requests:
memory: 128Mi
command:
- java
args:
- -XshowSettings:VM
- -XX:MinRAMPercentage=50.0
- -jar
- app.jar
...
https://www.baeldung.com/java-jvm-parameters-rampercentage
VM settings:
Max. Heap Size (Estimated): 121.81M
Property settings:
java.version = 11.0.12
Turns out 50.0 is already a default value

App memory performance
17
Taken from grafana
container memory limit
container memory usage
jvm memory
100 mb

Kafka Streams app memory
19
JVM Heap + RocksDB
Confluent article about Kafka Streams memory tunning:
https://docs.confluent.io/platform/current/streams/developer-guide/memory-mgmt.html
State Store
Kafka Streams
App
put(key, val)

Kafka Streams app memory
20
JVM Heap + RocksDB
Kafka Streams
App
CachingKeyValueStore
- ThreadCache context.cache()
Indexes
bloom
filters
block
cache
OS page
cache
memtable
State Store - RocksDB
! These items size
depends on number of
unique keys
JVM Heap
Not controlled by JVM
RocksDB memory details: https://github.com/facebook/rocksdb/wiki/Memory-usage-in-RocksDB
put(key, val)
native put(key, val)

How to fix unbounded RocksDB memory usage?
21
Application.java
properties.put(StreamsConfig.ROCKSDB_CONFIG_SETTER_CLASS_CONFIG,
”your.package.BoundedMemoryRocksDBConfig");
your.package.BoundedMemoryRocksDBConfig.java
@Override
public void setConfig(..., Options options, ...) {
BlockBasedTableConfig tableConfig = options.tableFormatConfig();
Cache cache = new LRUCache(computeTotalRocksDbMem(), -1, false);
tableConfig.setBlockCache(cache);
tableConfig.setCacheIndexAndFilterBlocks(true);
options.setWriteBufferManager(writeBufferManager);
...
}

22
Application.java
@Override
options.setWriteBufferManager(writeBufferManager);
...
}

23
Application.java
@Override
}
Pay attention to the number of stores and partitions:
https://docs.confluent.io/platform/current/streams/developer-guide/memory-mgmt.html#rocksdb

How to compute my
RocksDB memory?

Dynamic memory allocation
25
private static long computeTotalRocksDbMem() {
long totalContainerMemoryBytes = getEnv("CONTAINER_MEMORY_LIMIT"));
double osPercentage = getEnv("OS_MEMORY_PERCENTAGE"));
long offHeapSizeMb = getEnv("OFF_HEAP_SIZE_MB"));
long maxHeapSizeMb = getEnv("MAX_HEAP_SIZE_MB"));
long jvmMemoryBytes = (offHeapSizeMb + maxHeapSizeMb) * 1024 * 1024;
return totalContainerMemoryBytes * (1 - osPercentage)) - jvmMemoryBytes;
}

26
}

27
About using container props as env vars:
https://kubernetes.io/docs/tasks/inject-data-application/environment-variable-expose-pod-information/
...
env:
- name: CONTAINER_MEMORY_LIMIT
valueFrom:
resourceFieldRef:
containerName: app
resource: limits.memory
...

28
}

29
env:
- name: OS_MEMORY_PERCENTAGE
value: "0.1"
- name: OFF_HEAP_SIZE_MB
value: "128"
- name: MAX_HEAP_SIZE_MB
value: "{{ .Values.heapSizeMb }}"

30
}

31
env:
- name: OS_MEMORY_PERCENTAGE
value: "0.1"
# computed it based on the jcmd output
value: "128"
value: "{{ .Values.heapSizeMb }}"

If you would like to analyze non-heap JVM mem
32
1. Make sure you use JDK (not JRE) Docker image
2. Add to JVM args -XX:NativeMemoryTracking=summary
3. Execute command in container shell:
command:
- java
args:
- -XX:NativeMemoryTracking=summary
- -jar
- app.jar
bash
kubectl exec pod/your-pod-name -n your-namespace –it -- /bin/bash -c “jcmd 1 VM.native_memory”

33
}

34
env:
value: "128"
value: "{{ .Values.heapSizeMb }}”
args:
- -Xmx{{ .Values.heapSizeMb }}m
- -jar
- app.jar
helm/values.yaml
heapSizeMb: 64

If you would like to profile your app
35
1) helm/templates/deployment.yaml
command:
- java
args:
- -Dcom.sun.management.jmxremote
- -Dcom.sun.management.jmxremote.port=13089
- -Dcom.sun.management.jmxremote.ssl=false
- -Dcom.sun.management.jmxremote.local.only=false
- -Dcom.sun.management.jmxremote.authenticate=false
- -Dcom.sun.management.jmxremote.rmi.port=13089
- -Djava.rmi.server.hostname=localhost
- -jar
- app.jar
ports:
- containerPort: 13089
name: jmx
protocol: TCP
2) kubectl port-forward pod/your-pod-name -n your-namespace 13089:13089

App memory performance (after fix)
36
Taken from grafana

Limitations are not the only way!
37

Links and materials
38
• How JVM analyze memory inside Docker container–
https://merikan.com/2019/04/jvm-in-a-container/#java-10
• How to use jcmd to analyze non-heap memory–
https://www.baeldung.com/native-memory-tracking-in-jvm
• Why use container_memory_working_set_bytes instead of container_memory_usage_bytes
https://blog.freshtracks.io/a-deep-dive-into-kubernetes-metrics-part-3-container-resource-metrics-361c5ee46e6
• How RocksDB store works with Kafka Streams:
https://www.confluent.io/blog/how-to-tune-rocksdb-kafka-streams-state-stores-performance/
• Linux memory controller for cgroups
https://lwn.net/Articles/432224/

/
Thank you!
39
https://t.me/aserebryanskiy
a.serebrianskiy@gmail.com

What to do if Your Kafka Streams App Gets OOMKilled? with Andrey Serebryanskiy

More Related Content

What's hot (20)

Similar to What to do if Your Kafka Streams App Gets OOMKilled? with Andrey Serebryanskiy (7)

More from HostedbyConfluent (20)

Recently uploaded (20)

What to do if Your Kafka Streams App Gets OOMKilled? with Andrey Serebryanskiy