Solving Multi-tenancy and G1GC in Apache HBase
6 likes2,232 views
The document discusses multitenancy in HBase at HubSpot, outlining its benefits and tools such as hbasetracing for monitoring and managing bad actors. It emphasizes the importance of tuning G1 Garbage Collector (G1GC) settings for optimal performance, including parameters like heap size and memory usage, while identifying potential usage limitations. Recommendations for adjusting memory caps and garbage collection practices are provided to improve overall system efficiency.
![Important Metrics for Tuning
● G1GC Eden & Tenured size.
○ GC logs: “[Eden: … Survivors: … Heap: …]”
● HBase memory used by Memstore.
○ RegionServer JMX: “memStoreSize”
● HBase memory used by Block Cache.
○ RegionServer JMX: “blockCacheSize”
● HBase memory used by “static index”.
○ RegionServer JMX: “staticIndexSize”](https://image.slidesharecdn.com/opssession6hubspot-160802225343/85/Solving-Multi-tenancy-and-G1GC-in-Apache-HBase-36-320.jpg)
Solving Multi-tenancy and G1GC in Apache HBase
- 2. Multitenancy
- 3. ● ● ● What does Multitenancy Mean?
- 7. ● ● ● ● Ad-hoc Querying with HBasetracing
- 10. Dealing with Bad Actors
- 11. ● ● ● ● Quotas
- 14. Remember this?
- 15. ● ● ● When Quotas Fall Flat
- 17. ● The Dream
- 18. Handling Failure & Managing Risk
- 20. ● ● Consistency.TIMELINE ● isStale() Read Replica Usage
- 21. ● ● Read Replica Timeout Settings
- 26. Monitoring
- 27. ● ● ● ●
- 28. ● ● next() ● ● ●
- 29. ● ● ● ●
- 30. ● ●
- 31. G1GC: Making it work with HBase
- 32. Why G1GC? ● Designed for large heaps. ○ Divides heap into many smaller G1 regions. ○ G1 regions scanned and collected independently. ● Instead of occasional very long pauses, G1GC has more frequent, shorter pauses. If tuned properly, G1GC can provide performant GC that scales well for large RegionServer heaps.
- 33. The Need for Tuning Out of the box, G1GC hurt our HBase clusters’ performance: ● Too much time spent in GC pauses. ● Occasional very long GC pauses. ● “To-space Exhaustion”, leading to Full GCs, which led to slow RegionServer deaths.
- 36. Important Metrics for Tuning ● G1GC Eden & Tenured size. ○ GC logs: “[Eden: … Survivors: … Heap: …]” ● HBase memory used by Memstore. ○ RegionServer JMX: “memStoreSize” ● HBase memory used by Block Cache. ○ RegionServer JMX: “blockCacheSize” ● HBase memory used by “static index”. ○ RegionServer JMX: “staticIndexSize”
- 37. Necessary Tuning Params ● JVM args: -Xms, -Xmx -XX:G1NewSizePercent -XX:InitiatingHeapOccupancyPercent (aka “IHOP”) ● HBase configs (hbase-site.xml): hfile.block.cache.size hbase.regionserver.global.memstore.size
- 38. Necessary Tuning: Method A. Find max block cache size, memstore size, and static index size from the past month. B. Sum 110% of (A) maxes, add heap waste. C. Set IHOP and heap size such that Initiating Heap Occupancy > (B) by at least 10% heap. D. Ensure IHOP + G1NewSizePercent < 90%. – 90% = 100% - G1ReservePercent (default 10)
- 39. Necessary Tuning: cont. In hbase-site.xml: ● Set hfile.block.cache.size ratio value to 110% max block cache size from the past month. ● Set hbase.regionserver.global.memstore.size ratio value to 110% max Memstore size from the past month.
- 42. Further Tuning & Considerations ● -XX:G1ReservePercent ○ Accommodating for burst-y usage. ● -XX:G1HeapRegionSize ○ Reducing occurrence of humongous objects. ○ Reducing long tail of slow GCs in some cases. ● -XX:G1NewSizePercent ○ Tuning individual pause time vs. % time in GC.
- 43. HBase Usage & Tuning Limits A Full GC isn’t necessarily G1GC’s fault. There’s a level of “bad usage” that’s unreasonable to tune around: ● Unexpected, excessively burst-y traffic. ● Too many/enormous Humongous objects. In either of these cases, the real solution is to fix the client code.
- 44. Usage Note: Caching isn’t Free! Yellow: % time spent in Mixed GC (left axis) | Blue: block cache churn, MB/sec (right axis)
- 45. ...to Summarize: ● Tune heap size, IHOP, & HBase memory caps based on HBase memory usage. ● Tune Eden size based on % time in GC & average Young GC pause times. ● Make adjustments as needed, based on cluster usage. ● Look for suboptimal usage in your HBase clients to further improve HBase GC.
- 46. Links & Reference Blog Post — http://bit.ly/hbasegc G1GC CollectD Plugin — http://bit.ly/collectdgc G1GC Log Visualizer — http://bit.ly/gclogviz