Performance Tuning RocksDB for Kafka Streams' State Stores (Dhruba Borthakur, Rockset, Bruno Cadonna, Confluent) Kafka Summit 2020
0 likes7,168 views
RocksDB is the default state store for Kafka Streams. In this talk, we will discuss how to improve single node performance of the state store by tuning RocksDB and how to efficiently identify issues in the setup. We start with a short description of the RocksDB architecture. We discuss how Kafka Streams restores the state stores from Kafka by leveraging RocksDB features for bulk loading of data. We give examples of hand-tuning the RocksDB state stores based on Kafka Streams metrics and RocksDB’s metrics. At the end, we dive into a few RocksDB command line utilities that allow you to debug your setup and dump data from a state store. We illustrate the usage of the utilities with a few real-life use cases. The key takeaway from the session is the ability to understand the internal details of the default state store in Kafka Streams so that engineers can fine-tune their performance for different varieties of workloads and operate the state stores in a more robust manner.
![Operational Issues
• High disk I/O
• Operating system shows high disk I/O
• Kafka Streams metrics with high values
• memtable-bytes-flushed-[rate | total]
• bytes-[read | written]-compaction-rate
• Kafka Streams metrics with low values
• memtable-hit-ratio
• block-cache-[data | index | filter]-hit-ratio
40](https://image.slidesharecdn.com/s046dhrubaborthakurbrunocadonnaperformancetuningrocksdbforkafkastreamsstatestore1-200910043849/85/Performance-Tuning-RocksDB-for-Kafka-Streams-State-Stores-Dhruba-Borthakur-Rockset-Bruno-Cadonna-Confluent-Kafka-Summit-2020-40-320.jpg)
![Operational Issues
• High disk I/O
• Operating system shows high disk I/O
• Kafka Streams metrics with high values
• memtable-bytes-flushed-[rate | total]
• bytes-[read | written]-compaction-rate
• Kafka Streams metrics with low values
• memtable-hit-ratio
• block-cache-[data | index | filter]-hit-ratio
• Write stalls
• Processing latency of the application increases
• Kafka Streams client gets kicked out of the group
• Kafka Streams metric write-stall-duration-[avg | total] shows high values
41](https://image.slidesharecdn.com/s046dhrubaborthakurbrunocadonnaperformancetuningrocksdbforkafkastreamsstatestore1-200910043849/85/Performance-Tuning-RocksDB-for-Kafka-Streams-State-Stores-Dhruba-Borthakur-Rockset-Bruno-Cadonna-Confluent-Kafka-Summit-2020-41-320.jpg)
![Useful commands- Example output
53
# example output
Table properties for task: 1_9
from [] to []
Process /tmp/kafka-streams/my-app/1_9/rocksdb/my-counts/000006.sst
Sst file format: block-based
Table Properties:
------------------------------
# data blocks: 1
# entries: 2
raw key size: 18
raw average key size: 9.000000
raw value size: 88
raw average value size: 44.000000
data block size: 125
index block size: 35
filter block size: 0
(estimated) table size: 160](https://image.slidesharecdn.com/s046dhrubaborthakurbrunocadonnaperformancetuningrocksdbforkafkastreamsstatestore1-200910043849/85/Performance-Tuning-RocksDB-for-Kafka-Streams-State-Stores-Dhruba-Borthakur-Rockset-Bruno-Cadonna-Confluent-Kafka-Summit-2020-53-320.jpg)
Ad
More Related Content
What's hot (20)
Similar to Performance Tuning RocksDB for Kafka Streams' State Stores (Dhruba Borthakur, Rockset, Bruno Cadonna, Confluent) Kafka Summit 2020 (20)
![How does Riak compare to Cassandra? [Cassandra London User Group July 2011]](https://cdn.slidesharecdn.com/ss_thumbnails/cassandralondonugjuly2011-110720110309-phpapp02-thumbnail.jpg?width=560&fit=bounds)
Ad
More from confluent (20)
Ad
Recently uploaded (20)
Performance Tuning RocksDB for Kafka Streams' State Stores (Dhruba Borthakur, Rockset, Bruno Cadonna, Confluent) Kafka Summit 2020
- 1. Performance Tuning RocksDB for Kafka Streams’ State Store Dhruba Borthakur (Rockset), Bruno Cadonna (Confluent)
- 2. About the Presenters Dhruba Borthakur CTO & Co-founder Rockset rockset.com 2 Bruno Cadonna Contributor to Apache Kafka & Software Engineer at Confluent confluent.io
- 3. Agenda • Kafka Streams and State Stores • Introduction to RocksDB • Compaction Styles in RocksDB • Possible Operational Issues • Tuning RocksDB • RocksDB Command Line Utilities • Takeaways 3
- 4. Kafka Streams and State Stores
- 5. Kafka Streams 5 ● Stateless and stateful processors ● Stateful processors use state stores
- 6. Kafka Streams 6 ● Stateless and stateful processors ● Stateful processors use state stores
- 7. Kafka Streams 7 ● Stateless and stateful processors ● Stateful processors use state stores
- 8. Kafka Streams 8 ● Stateless and stateful processors ● Stateful processors use state stores
- 9. Kafka Streams 9 ● Stateless and stateful processors ● Stateful processors use state stores
- 10. Kafka Streams 10 ● Stateless and stateful processors ● Stateful processors use state stores
- 11. Kafka Streams 11 ● Stateless and stateful processors ● Stateful processors use state stores ● Create one topology per input partition, i.e., task
- 12. State Stores in Kafka Streams 12 • Stateful processor may use one or more state stores • Each partition has its own state store Metrics & De-/Serialization Caching Changelogging Restoration
- 13. State Stores in Kafka Streams 13 • Stateful processor may use one or more state stores • Each partition has its own state store • State stores are layered: Metrics & De-/Serialization Caching Changelogging Restoration
- 14. State Stores in Kafka Streams 14 Metrics & De-/Serialization Caching Changelogging Restoration • Stateful processor may use one or more state stores • Each partition has its own state store • State stores are layered: • collects metrics and de-/serializes records
- 15. State Stores in Kafka Streams 15 01 10 Metrics & De-/Serialization Caching Changelogging Restoration • Stateful processor may use one or more state stores • Each partition has its own state store • State stores are layered: • collects metrics and de-/serializes records
- 16. State Stores in Kafka Streams 16 01 10 Metrics & De-/Serialization Caching Changelogging Restoration • Stateful processor may use one or more state stores • Each partition has its own state store • State stores are layered: • collects metrics and de-/serializes records • caches records
- 17. 01 10 State Stores in Kafka Streams 17 01 10 Metrics & De-/Serialization Caching Changelogging Restoration • Stateful processor may use one or more state stores • Each partition has its own state store • State stores are layered: • collects metrics and de-/serializes records • caches records • writes records to changelog
- 18. 01 10 State Stores in Kafka Streams 18 01 10 01 10 Metrics & De-/Serialization Caching Changelogging Restoration • Stateful processor may use one or more state stores • Each partition has its own state store • State stores are layered: • collects metrics and de-/serializes records • caches records • writes records to changelog
- 19. 01 10 State Stores in Kafka Streams 19 01 10 01 10 Metrics & De-/Serialization Caching Changelogging Restoration • Stateful processor may use one or more state stores • Each partition has its own state store • State stores are layered: • collects metrics and de-/serializes records • caches records • writes records to changelog • writes records to local state store
- 20. 01 10 State Stores in Kafka Streams 20 01 10 01 10 Metrics & De-/Serialization Caching Changelogging Restoration • Stateful processor may use one or more state stores • Each partition has its own state store • State stores are layered: • collects metrics and de-/serializes records • caches records • writes records to changelog • writes records to local state store • State stores are restored from changelog topics • Restoration is byte-based and by-passes wrapping layers
- 21. RocksDB is the Default State Store • Kafka Streams needed a write optimized state store • Kafka Streams 2.6 uses RocksDB 5.18.4 • Kafka Streams provides metrics to monitor RocksDB state stores • RocksDB can be configured by passing a class that implements interface RocksDBConfigSetter to configuration rocksdb.config.setter 21
- 22. Example: Configuring RocksDB in Kafka Streams 22 public static class MyRocksDBConfig implements RocksDBConfigSetter { @Override public void setConfig(final String storeName, final Options options, final Map<String, Object> configs) { // e.g. set compaction style options.setCompactionStyle(CompactionStyle.LEVEL); } @Override public void close(final String storeName, final Options options) {} }
- 24. What is RocksDB? • Key-value persistent store • Embedded C++ & Java library • Server workloads 24
- 25. What is it not? • Not distributed • No failover • Not highly-available. If the machine dies, you lose your data • Focus on performance Kafka Streams makes it fault-tolerant 25
- 26. RocksDB API • Keys and values are byte arrays • Data are stored sorted by key • Update Operations: Put/Delete/Merge • Queries: Get/Iterator 26
- 27. Log Structured Merge Architecture 27 Periodic compaction Read only data in SSD or disk Read write data in RAM Transaction log Scan request from application Write request from application
- 28. RocksDB Write Path 28 Write request Read only MemTable Log Log sst sst sst sst sst sst LS Compaction Flush SwitchSwitch Active MemTable Log
- 29. RocksDB Reads • Data can be in memory or disk • Consult multiple files to find the latest instance of the key • Use bloom filters to reduce IO • Every sst file has a bloom filter • bloom filters are cached in memory • default config: eliminates 99% of reads 29
- 30. RocksDB Read Path 30 Read only MemTable Log Log sst sst sst LS Compaction Flush Active MemTable Log sst sst sst Memory Persistent Storage Read request Get(k) Blooms
- 31. RocksDB Architecture 31 Read only MemTable Log Log sst sst sst LS Compaction Flush Active MemTable Log sst sst Memory Persistent Storage sst Switch Switch Write request Read only BlockCache Read request
- 32. RocksDB Open & Pluggable 32 Pluggable compaction Pluggable sst data format on storage Pluggable Memtable format in RAM Transaction log Blooms Customizable WAL Get or scan request from application Write request from application
- 33. Compaction Styles in RocksDB
- 34. What is Compaction • Multi-threaded • Parallel compactions on different parts of the database • Deletes overwritten keys • Two types of compactions • level compactions • universal compaction 34
- 35. Level compaction • RocksDB default compaction is Level Compaction (for read heavy workloads) • Stores data in multiple levels • More recent data stored in L0 • Older data stored in Lmax • Files in L0 • overlapping keys, sorted by flush time • Files in L1 to Lmax • non overlapping keys, sorted by key • Max space amplification = 10% https://github.com/facebook/rocksdb/wiki/Leveled-Compaction 35
- 36. Universal Compaction • For write heavy workloads • needed if Level style compaction is bottlenecked by disk throughout • Stores all files in L0 • All files are arranged in time order • Decreases write amplification but increases space amplification • Pick up files that are chronologically adjacent to one another • merge them • replace them with a new file in L0 36
- 38. Operational Issues • High memory usage • Application gets slower or even crashes • Operating system shows high memory usage • Kafka Streams metrics for monitoring memory usage of RocksDB (KIP-607, planned for 2.7) show high values 38
- 39. Operational Issues • High memory usage • Application gets slower or even crashes • Operating system shows high memory usage • Kafka Streams metrics for monitoring memory usage of RocksDB (KIP-607, planned for 2.7) show high values • High disk usage • Application crashes with I/O errors • Operating system shows high disk usage 39
- 40. Operational Issues • High disk I/O • Operating system shows high disk I/O • Kafka Streams metrics with high values • memtable-bytes-flushed-[rate | total] • bytes-[read | written]-compaction-rate • Kafka Streams metrics with low values • memtable-hit-ratio • block-cache-[data | index | filter]-hit-ratio 40
- 41. Operational Issues • High disk I/O • Operating system shows high disk I/O • Kafka Streams metrics with high values • memtable-bytes-flushed-[rate | total] • bytes-[read | written]-compaction-rate • Kafka Streams metrics with low values • memtable-hit-ratio • block-cache-[data | index | filter]-hit-ratio • Write stalls • Processing latency of the application increases • Kafka Streams client gets kicked out of the group • Kafka Streams metric write-stall-duration-[avg | total] shows high values 41
- 42. Operational Issues • Too many open files • Application crashes with I/O errors • Kafka Streams metric number-open-files shows high values 42
- 43. Operational Issues • Kafka Streams client gets kicked out of the consumer group during restoration • Before 2.6 Kafka Streams used RocksDB’s bulk loading (Options#prepareForBulkLoad()) feature to restore the state store faster. • Bulk loading basically consists of: • disable automatic compaction and • write all data to level 0 • trigger manual compaction 43
- 44. Operational Issues • Kafka Streams client gets kicked out of the consumer group during restoration • Before 2.6 Kafka Streams used RocksDB’s bulk loading (Options#prepareForBulkLoad()) feature to restore the state store faster. • Bulk loading basically consists of: • disable automatic compaction and • write all data to level 0 • trigger manual compaction • Manual compaction is a blocking call that may take longer than max.poll.interval.ms 44
- 45. Operational Issues • Kafka Streams client gets kicked out of the consumer group during restoration • Before 2.6 Kafka Streams used RocksDB’s bulk loading (Options#prepareForBulkLoad()) feature to restore the state store faster. • Bulk loading basically consists of: • disable automatic compaction and • write all data to level 0 • trigger manual compaction • Manual compaction is a blocking call that may take longer than max.poll.interval.ms • Bulk loading is removed in 2.6 • Currently evaluating alternatives to increase the performance of state store restoration by using other features of RocksDB, e.g., ingesting SST files directly. 45
- 46. Tuning RocksDB
- 47. Debug Kafka Streams OOM • Memory consumption • memtable (for writes) • memtable size, number of memtables • block cache (reads) • configure to share among all the partitions in the kafka store • Kafka Streams keeps index blocks in the block cache • rocksdb-java bugs (https://github.com/facebook/rocksdb/issues/6247) • High disk usage • Use level compaction instead of universal compaction • provision more disk space https://docs.confluent.io/current/streams/developer-guide/memory-mgmt.html 47
- 48. Debug writes stalls • Debug write stalls in RocksDB • Is disk IO utilization at 100%? • add more storage spindles • use universal compaction • Check number of background compaction threads • Kafka Streams uses Max(2, number of available processors) by default • Check memtable configuration • AdvancedColumnFamilyOptions.max_write_buffer_number • ColumnFamilyOptions.write_buffer_size 48
- 49. Debugging file descriptor issues • Too many open files • DBOptions.max_open_files = -1 (default) • opens all sst files at db open time • good for performance but can run out of file descriptors • Increase operating system number of open file descriptors • Set DBOptions.max_open_files = 10000 • will open a max of 10000 files concurrently • Decrease number of files by making each file larger • AdvancedColumnFamilyOptions.target_file_size_base = 128 MB (default is 64 MB) 49
- 50. RocksDB Command Line Utilities
- 51. Build rocksdb command line utilities git clone [email protected]:facebook/rocksdb.git cd rocksdb make ldb sst_dump cp ldb /usr/local/bin cp sst_dump /usr/local/bin 51 Useful RocksDB command line tools: https://github.com/facebook/rocksdb/wiki/Administration-and-Data- Access-Tool Build # change these values accordingly APP_ID=my-app STATE_STORE=my-counts STATE_STORE_DIR=/tmp/kafka-streams TASKS=$(ls $STATE_STORE_DIR/$APP_ID) Change These Values
- 52. Useful commands # View all keys for i in $TASKS; do ldb --db=$STATE_STORE_DIR/$APP_ID/$i/rocksdb/$STATE_STORE scan 2>/dev/null; done # Show table properties for i in $TASKS; do TABLE_PROPERTIES=$(sst_dump -- file=$STATE_STORE_DIR/$APP_ID/$i/rocksdb/$STATE_STORE -- show_properties) echo -e "Table properties for task: $in$TABLE_PROPERTIESnn" done 52
- 53. Useful commands- Example output 53 # example output Table properties for task: 1_9 from [] to [] Process /tmp/kafka-streams/my-app/1_9/rocksdb/my-counts/000006.sst Sst file format: block-based Table Properties: ------------------------------ # data blocks: 1 # entries: 2 raw key size: 18 raw average key size: 9.000000 raw value size: 88 raw average value size: 44.000000 data block size: 125 index block size: 35 filter block size: 0 (estimated) table size: 160
- 54. Takeaways
- 55. Takeaways • RocksDB is the default state store in Kafka Streams • Kafka Streams provides functionality to configure and monitor RocksDB • RocksDB uses a log structured merge (LSM) architecture with different compaction styles • You might run into operational issues, but you can solve them by debugging and tuning RocksDB • RocksDB offers command line utilities for analysing state stores 55
- 56. 56 Thank you! [email protected] [email protected] cnfl.io/meetups cnfl.io/slackcnfl.io/blog Learn how Rockset uses RocksDB https://rockset.com/blog/how-we-use-rocksdb-at-rockset/