How and Why Prometheus' New Storage Engine Pushes the Limits of Time Series Databases
2 likes587 views
The Prometheus monitoring system collects and stores time series data to give valuable insights over hosts, containers, and applications. Its storage engine was designed to be multiple orders of magnitude faster and more space efficient than, say, RRD or SQL storage. However, with the rise of orchestration systems such as Docker Swarm and Kubernetes, and their extensive use of techniques like rolling updates and auto-scaling, environments are becoming increasingly dynamic. This increases the strain on metrics collection systems. To deal with the challenges, a new storage engine has been developed from scratch, bringing a sharp increase in performance and enabling new features. This talk will describe this new storage engine, its architecture, its data structures, and explain why and how it is well suited to gracefully handle high turnover rates of monitoring targets and provide consistent query performance.
![CPU: Cores Used
● Series ---> Series ID cache
Prometheus 2.0:
C = map[string] --> seriesID
id, ok := C[`series{labels}`]
if ok {
ingest(id, t, v)
}](https://image.slidesharecdn.com/tuxpwlyr4wubtyx5adva-signature-4fe5b0bf03df3ecdf72d849c1d60b826ade5c5ab294d0dda9994f2d8df84ffe0-poli-171019124607/85/How-and-Why-Prometheus-New-Storage-Engine-Pushes-the-Limits-of-Time-Series-Databases-65-320.jpg)
![Granular deletes
Series-Ref ---> Deleted Ranges
{
190: [{100, 200}, {300, 600}],
250: [{100, 5000}],
}
When the querier runs it pickups these ranges. If something is deleted, we skip
that range in the query.](https://image.slidesharecdn.com/tuxpwlyr4wubtyx5adva-signature-4fe5b0bf03df3ecdf72d849c1d60b826ade5c5ab294d0dda9994f2d8df84ffe0-poli-171019124607/85/How-and-Why-Prometheus-New-Storage-Engine-Pushes-the-Limits-of-Time-Series-Databases-82-320.jpg)
![[오픈소스컨설팅] Linux Network Troubleshooting](https://cdn.slidesharecdn.com/ss_thumbnails/opensourceconsultingnetworktroubleshootingyjlee20210412-210413014206-thumbnail.jpg?width=560&fit=bounds)
![[231] the simplicity of cluster apps with circuit](https://cdn.slidesharecdn.com/ss_thumbnails/213thesimplicityofclusterappswithcircuit-150915022150-lva1-app6892-thumbnail.jpg?width=560&fit=bounds)
Ad
More Related Content
What's hot (20)
Viewers also liked (20)
Ad
Similar to How and Why Prometheus' New Storage Engine Pushes the Limits of Time Series Databases (20)
Ad
More from Docker, Inc. (20)
Recently uploaded (20)
How and Why Prometheus' New Storage Engine Pushes the Limits of Time Series Databases
- 1. Storage in Prometheus 2.0 Goutham V Student, IIT Hyderabad gouthamve putadent Add picture here
- 2. Introduction What and how much are we storing?
- 3. Time Series (t0, v0), (t1, v1), (t2, v2), (t3, v3), .... time
- 4. Time Series time series A time series
- 5. Time Series requests_total{path="/status", method="GET", instance="10.0.0.1:80"} requests_total{path="/status", method="POST", instance="10.0.0.3:80"} requests_total{path="/", method="GET", instance="10.0.0.2:80"} ...
- 6. Time Series requests_total{path="/status", method="GET", instance="10.0.0.1:80"} requests_total{path="/status", method="POST", instance="10.0.0.3:80"} requests_total{path="/", method="GET", instance="10.0.0.2:80"} Select: requests_total
- 7. Time Series requests_total{path="/status", method="GET", instance="10.0.0.1:80"} requests_total{path="/status", method="POST", instance="10.0.0.3:80"} requests_total{path="/", method="GET", instance="10.0.0.2:80"} Select: requests_total{method="GET"}
- 12. What you expose: requests_total{path="/status", method="GET"} requests_total{path="/status", method="POST"} requests_total{path="/", method="GET"} What prometheus scrapes: requests_total{path="/status", method="GET", instance="10.0.0.1:80"} requests_total{path="/status", method="POST", instance="10.0.0.1:80"} requests_total{path="/", method="GET", instance="10.0.0.1:80"}
- 13. Scale 5 million active time series 30 second scrape interval 1 month of retention 166,000 samples/second 432 billion samples 8 byte timestamp + 8 byte value ⇒ 7 TB on disk 3,000 - 15,000 microservice instances
- 14. Scale 5 million active time series 30 second scrape interval 6 month of retention 166,000 samples/second 432 billion samples 8 byte timestamp + 8 byte value ⇒ 42 TB on disk 3,000 - 15,000 microservice instances
- 15. Compression - Timestamp 1496163646 1496163676 1496163706 1496163735 1496163765
- 16. Compression - Timestamp 1496163646 +30 +30 +29 +30Δ 1496163646 1496163676 1496163706 1496163735 1496163765
- 17. Compression - Timestamp 1496163646 +30 +0 -1 +1ΔΔ 1496163646 +30 +30 +29 +30Δ 1496163646 1496163676 1496163706 1496163735 1496163765
- 19. Compression - Value http://www.vldb.org/pvldb/vol8/p1816-teller.pdf bytes/sample Raw: 16 Compressed: 1.37 7TB => 600GB
- 20. Scale 5 million active time series 30 second scrape interval 1 month of retention 166,000 samples/second 432 billion samples 8 byte timestamp + 8 byte value ⇒ 600GB on disk 3,000 - 15,000 microservice instances
- 22. What you expose: requests_total{path="/status", method="GET"} requests_total{path="/status", method="POST"} requests_total{path="/", method="GET"} What prometheus scrapes: requests_total{path="/status", method="GET", instance="10.0.0.1:80"} requests_total{path="/status", method="POST", instance="10.0.0.1:80"} requests_total{path="/", method="GET", instance="10.0.0.1:80"}
- 23. The new era ● Docker Swarm and Kubernetes ● New IP everytime a service is updated ● Dynamic scaling! ● Which means broken series and new series
- 24. 5 million active time series 150 million total time series 30 second scrape interval 1 month of retention 166,000 samples/second 432 billion samples 8 byte timestamp + 8 byte value ⇒ 600GB on disk Scale 3,000 - 15,000 microservice instances
- 25. Storage in 1.0 Layout and Problems
- 26. Time Series time series 1 file/series Holding compressed chunks
- 27. Time Series time series 1 file/series Holding compressed chunks
- 29. Querying 1. Get series labels 2. Calculate Series ID 3. Add the ID against the label { __name__=”requests_total”, pod=”nginx-34534242-abc723 job=”nginx”, path=”/api/v1/status”, status=”200”, method=”GET”, } Series ID: 3300
- 30. Querying 1. Get series labels 2. Calculate Series ID 3. Add the ID against the label
- 31. Querying { __name__=”requests_total”, pod=”nginx-34534242-abc723 job=”nginx”, path=”/api/v1/status”, status=”200”, method=”GET”, } status=”200”: 1000 500000 99 1 1500 2 1001 5 1502 method=”GET”: 2 999999 4 3 1502 9 6 5 10 ... 1. Get series labels 2. Calculate Series ID 3. Add the ID against the label
- 32. 1.0 Problems ● Too many files ● Huge, unsorted global Index
- 33. Storage in 2.0 Layout and how it fixes the problems
- 35. 2.0: Time Shared Blocks time series
- 36. 2.0 Layout mutable prometheuswrite t0 t1 t2 t3 now query merge
- 37. 2.0 Layout ● index, metadata and tombstones ● A bunch of files holding actual data ● ~100 files per block even in very large setups
- 38. 2.0 Layout ● index, metadata and tombstones ● A bunch of files holding actual data ● ~100 files per block even in very large setups
- 39. 2.0 Layout ● Fixed-size (512MB, fx) per block ● 512MB vs block/series
- 40. 2.0 Index { __name__=”requests_total”, pod=”nginx-34534242-abc723 job=”nginx”, path=”/api/v1/status”, status=”200”, method=”GET”, } ● Assign block-scoped ID to each series ● Maintain sorted lists from label pair to IDs ● Efficient k-way set operations
- 41. 2.0 Index { __name__=”requests_total”, pod=”nginx-34534242-abc723 job=”nginx”, path=”/api/v1/status”, status=”200”, method=”GET”, } ● Assign block-scoped ID to each series ● Maintain sorted lists from label pair to IDs ● Efficient k-way set operations ID: 5
- 42. 2.0 Index { __name__=”requests_total”, pod=”nginx-34534242-abc723 job=”nginx”, path=”/api/v1/status”, status=”200”, method=”GET”, } ● Assign block-scoped ID to each series ● Maintain sorted lists from label pair to IDs ● Efficient k-way set operations status=”200”: 1 2 5 99 1000 1001 1500 1502 500000 method=”GET”: 2 3 4 5 6 9 10 1502 999999 ... ID: 5
- 43. 2.0 Index { __name__=”requests_total”, pod=”nginx-34534242-abc723 job=”nginx”, path=”/api/v1/status”, status=”200”, method=”GET”, } ● Assign block-scoped ID to each series ● Maintain sorted lists from label pair to IDs ● Efficient k-way set operations status=”200”: 1 2 5 99 1000 1001 1500 1502 500000 method=”GET”: 2 3 4 5 6 9 10 1502 999999 ... ID: 5
- 44. 2.0 Index { __name__=”requests_total”, pod=”nginx-34534242-abc723 job=”nginx”, path=”/api/v1/status”, status=”200”, method=”GET”, } ● Assign block-scoped ID to each series ● Maintain sorted lists from label pair to IDs ● Efficient k-way set operations status=”200”: 1 2 5 99 1000 1001 1500 1502 500000 method=”GET”: 2 3 4 5 6 9 10 1502 999999 … Intersect:
- 45. 2.0 Index { __name__=”requests_total”, pod=”nginx-34534242-abc723 job=”nginx”, path=”/api/v1/status”, status=”200”, method=”GET”, } ● Assign block-scoped ID to each series ● Maintain sorted lists from label pair to IDs ● Efficient k-way set operations status=”200”: 1 2 5 99 1000 1001 1500 1502 500000 method=”GET”: 2 3 4 5 6 9 10 1502 999999 … Intersect:
- 46. 2.0 Index { __name__=”requests_total”, pod=”nginx-34534242-abc723 job=”nginx”, path=”/api/v1/status”, status=”200”, method=”GET”, } ● Assign block-scoped ID to each series ● Maintain sorted lists from label pair to IDs ● Efficient k-way set operations status=”200”: 1 2 5 99 1000 1001 1500 1502 500000 method=”GET”: 2 3 4 5 6 9 10 1502 999999 ... Intersect: 2
- 47. 2.0 Index { __name__=”requests_total”, pod=”nginx-34534242-abc723 job=”nginx”, path=”/api/v1/status”, status=”200”, method=”GET”, } ● Assign block-scoped ID to each series ● Maintain sorted lists from label pair to IDs ● Efficient k-way set operations status=”200”: 1 2 5 99 1000 1001 1500 1502 500000 method=”GET”: 2 3 4 5 6 9 10 1502 999999 ... Intersect: 2
- 48. 2.0 Index { __name__=”requests_total”, pod=”nginx-34534242-abc723 job=”nginx”, path=”/api/v1/status”, status=”200”, method=”GET”, } ● Assign block-scoped ID to each series ● Maintain sorted lists from label pair to IDs ● Efficient k-way set operations status=”200”: 1 2 5 99 1000 1001 1500 1502 500000 method=”GET”: 2 3 4 5 6 9 10 1502 999999 ... Intersect: 2
- 49. 2.0 Index { __name__=”requests_total”, pod=”nginx-34534242-abc723 job=”nginx”, path=”/api/v1/status”, status=”200”, method=”GET”, } ● Assign block-scoped ID to each series ● Maintain sorted lists from label pair to IDs ● Efficient k-way set operations status=”200”: 1 2 5 99 1000 1001 1500 1502 500000 method=”GET”: 2 3 4 5 6 9 10 1502 999999 ... Intersect: 2 5
- 50. 2.0 Index { __name__=”requests_total”, pod=”nginx-34534242-abc723 job=”nginx”, path=”/api/v1/status”, status=”200”, method=”GET”, } ● Assign block-scoped ID to each series ● Maintain sorted lists from label pair to IDs ● Efficient k-way set operations status=”200”: 1 2 5 99 1000 1001 1500 1502 500000 method=”GET”: 2 3 4 5 6 9 10 1502 999999 ... Intersect: 2 5
- 51. 2.0 Index { __name__=”requests_total”, pod=”nginx-34534242-abc723 job=”nginx”, path=”/api/v1/status”, status=”200”, method=”GET”, } ● Assign block-scoped ID to each series ● Maintain sorted lists from label pair to IDs ● Efficient k-way set operations status=”200”: 1 2 5 99 1000 1001 1500 1502 500000 method=”GET”: 2 3 4 5 6 9 10 1502 999999 ... Intersect: 2 5
- 52. 2.0 Index { __name__=”requests_total”, pod=”nginx-34534242-abc723 job=”nginx”, path=”/api/v1/status”, status=”200”, method=”GET”, } ● Assign block-scoped ID to each series ● Maintain sorted lists from label pair to IDs ● Efficient k-way set operations status=”200”: 1 2 5 99 1000 1001 1500 1502 500000 method=”GET”: 2 3 4 5 6 9 10 1502 999999 ... Intersect: 2 5 1502
- 53. 2.0 Index { __name__=”requests_total”, pod=”nginx-34534242-abc723 job=”nginx”, path=”/api/v1/status”, status=”200”, method=”GET”, } ● Assign block-scoped ID to each series ● Maintain sorted lists from label pair to IDs ● Efficient k-way set operations status=”200”: 1 2 5 99 1000 1001 1500 1502 500000 method=”GET”: 2 3 4 5 6 9 10 1502 999999 ... Intersect: 2 5 1502
- 54. 2.0 Index { __name__=”requests_total”, pod=”nginx-34534242-abc723 job=”nginx”, path=”/api/v1/status”, status=”200”, method=”GET”, } ● Assign block-scoped ID to each series ● Maintain sorted lists from label pair to IDs ● Efficient k-way set operations status=”200”: 1 2 5 99 1000 1001 1500 1502 500000 method=”GET”: 2 3 4 5 6 9 10 1502 999999 ... Intersect: 2 5 1502
- 56. Problems Fixed ● Just a bunch of files ● Time-sharding reduces index size
- 57. Benchmarks Great! But does it work?
- 58. Benchmarks Kubernetes cluster + dedicated Prometheus nodes 800 microservice instances + Kubernetes components 120,000 samples/second 300,000 active time series Swap out 50% of pods every 10 minutes
- 59. Prometheuses? Prometheus servers? Prometheis Aside: Plural of Prometheus?
- 60. Memory: GB 1.5 Queried 1.5 Unqueried 2.0 Queried 2.0 Unqueried 15GB 5GB
- 61. Memory: GB ● MMAP rather than manual memory management ● Fundamentally different design
- 62. CPU: Cores Used 1.5 Queried 1.5 Unqueried 2.0 Queried 2.0 Unqueried 6 Cores 1 Cores
- 63. CPU: Cores Used ● Assembly accelerated compression. @beorn7 @dgryski
- 64. CPU: Cores Used ● Series ---> Series ID cache Prometheus 1.x: 1. id = hash(series) 2. ingest(id, t, v) ● Hashing millions of times a minute!
- 65. CPU: Cores Used ● Series ---> Series ID cache Prometheus 2.0: C = map[string] --> seriesID id, ok := C[`series{labels}`] if ok { ingest(id, t, v) }
- 66. Disk Writes: MB 1.5 Queried 1.5 Unqueried 2.0 (both) 80MB 20MB
- 67. Disk Writes: MB ● Checkpointing vs WAL ● Checkpoint: Dump all memory to disk every 5 minutes ● WAL: Write the values that come just once
- 68. On Disk Size: GB 1.5 (both) 2.0 (both) 25GB 5GB
- 69. On Disk Size: GB ● An empty file takes 4KB! ● We are having several small files which adds a per-file storage overhead. ● Not a concern for real environments.
- 70. Query Latency: secs 1.5 2.0 25 secs 5 secs
- 71. Moar 2.0! ● Backups ● Granular deletes ● Staleness handling ● …..
- 76. Backups! mutable prometheuswrite t0 t1 t2 t3 now query merge
- 77. Backups! mutable prometheuswrite t0 t1 t2 t3 now query merge Immutable
- 82. Granular deletes Series-Ref ---> Deleted Ranges { 190: [{100, 200}, {300, 600}], 250: [{100, 5000}], } When the querier runs it pickups these ranges. If something is deleted, we skip that range in the query.
- 85. So yeah 2.0! Try it out! ● RC.1 cut TODAY for #DOCKERCON ● Lots of features: * Staleness handling * Backups * Better churn handling (du’h!)
- 86. Try it!!!!