![3
Who I am…
Lars George [EMEA Chief Architect]
• Clouderan since October 2010
• Hadooper since mid 2007
• HBase/Whirr Committer (of Hearts)
• github.com/larsgeorge
©2014 Cloudera, Inc. All rights reserved.](https://image.slidesharecdn.com/hbasesizingguides-hbasemeetup15102014-larsgeorge-141105032329-conversion-gate02/85/HBase-Sizing-Guide-3-320.jpg)
Ad
More Related Content
What's hot (20)
Viewers also liked (20)
Ad
Similar to HBase Sizing Guide (20)
Ad
More from larsgeorge (8)
Recently uploaded (20)
HBase Sizing Guide
- 1. Sizing Your HBase Cluster Lars George | @larsgeorge EMEA Chief Architect @ Cloudera
- 2. 2 Agenda • Introduction • Technical Background/Primer • Best Practices • Summary ©2014 Cloudera, Inc. All rights reserved.
- 3. 3 Who I am… Lars George [EMEA Chief Architect] • Clouderan since October 2010 • Hadooper since mid 2007 • HBase/Whirr Committer (of Hearts) • github.com/larsgeorge ©2014 Cloudera, Inc. All rights reserved.
- 4. 4 Bruce Lee: ”As you think, so shall you become.” ©2014 Cloudera, Inc. All rights reserved.
- 5. 5 Introduction ©2014 Cloudera, Inc. All rights reserved.
- 6. 6 HBase Sizing Is... • Making the most out of the cluster you have by... – Understanding how HBase uses low-level resources – Helping HBase understand your use-case by configuring it appropriately - and/or - – Design the use-case to help HBase along • Being able to gauge how many servers are needed for a given use-case
- 7. 7 Technical Background “To understand your fear is the beginning of really seeing…” — Bruce Lee ©2014 Cloudera, Inc. All rights reserved.
- 8. 8 HBase Dilemma Although HBase can host many applications, they may require completely opposite features Events Entities Time Series Message Store
- 9. 9 Competing Resources • Reads and Writes compete for the same low-level resources – Disk (HDFS) and Network I/O – RPC Handlers and Threads – Memory (Java Heap) • Otherwise they do exercise completely separate code paths
- 10. 10 Memory Sharing • By default every region server is dividing its memory (i.e. given maximum heap) into – 40% for in-memory stores (write ops) – 20% (40%) for block caching (reads ops) – Remaining space (here 40% or 20%) go towards usual Java heap usage • Objects etc. • Region information (HFile metadata) • Share of memory needs to be tweaked
- 11. 11 Writes • The cluster size is often determined by the write performance – Simple schema design implies writing to all (entities) or only one region (events) • Log structured merge trees like – Store mutation in in-memory store and write-ahead log – Flush out aggregated, sorted maps at specified threshold - or - when under pressure – Discard logs with no pending edits – Perform regular compactions of store files
- 12. 12 Writes: Flushes and Compactions Older TIME Newer SIZE (MB) 1000 750 500 250 0
- 13. 13 Flushes • Every mutation call (put, delete etc.) causes a check for a flush • If threshold is met, flush file to disk and schedule a compaction – Try to compact newly flushed files quickly • The compaction returns - if necessary - where a region should be split
- 14. 14 Compaction Storms • Premature flushing because of # of logs or memory pressure – Files will be smaller than the configured flush size • The background compactions are hard at work merging small flush files into the existing, larger store files – Rewrite hundreds of MB over and over
- 15. 15 Dependencies • Flushes happen across all stores/column families, even if just one triggers it • The flush size is compared to the size of all stores combined – Many column families dilute the size – Example: 55MB + 5MB + 4MB
- 16. 16 Write-Ahead Log • Currently only one per region server – Shared across all stores (i.e. column families) – Synchronized on file append calls • Work being done on mitigating this – WAL Compression – Multithreaded WAL with Ring Buffer – Multiple WAL’s per region server ➜ Start more than one region server per node?
- 17. 17 Write-Ahead Log (cont.) • Size set to 95% of default block size – 64MB or 128MB, but check config! • Keep number low to reduce recovery time – Limit set to 32, but can be increased • Increase size of logs - and/or - increase the number of logs before blocking • Compute number based on fill distribution and flush frequencies
- 18. 18 Write-Ahead Log (cont.) • Writes are synchronized across all stores – A large cell in one family can stop all writes of another – In this case the RPC handlers go binary, i.e. either work or all block • Can be bypassed on writes, but means no real durability and no replication – Maybe use coprocessor to restore dependent data sets (preWALRestore)
- 19. 19 Some Numbers • Typical write performance of HDFS is 35-50MB/s Cell Size OPS 0.5MB 70-100 100KB 350-500 10KB 3500-5000 ?? 1KB 35000-50000 ???? This is way to high in practice - Contention!
- 20. 20 Some More Numbers • Under real world conditions the rate is less, more like 15MB/s or less – Thread contention and serialization overhead is cause for massive slow down Cell Size OPS 0.5MB 10 100KB 100 10KB 800 1KB 6000
- 21. 21 Write Performance • There are many factors to the overall write performance of a cluster – Key Distribution ➜ Avoid region hotspot – Handlers ➜ Do not pile up too early – Write-ahead log ➜ Bottleneck #1 – Compactions ➜ Badly tuned can cause ever increasing background noise
- 22. 22 Cheat Sheet • Ensure you have enough or large enough write-ahead logs • Ensure you do not oversubscribe available memstore space • Ensure to set flush size large enough but not too large • Check write-ahead log usage carefully • Enable compression to store more data per node • Tweak compaction algorithm to peg background I/O at some level • Consider putting uneven column families in separate tables • Check metrics carefully for block cache, memstore, and all queues
- 23. 23 Example: Write to All Regions • Java Xmx heap at 10GB • Memstore share at 40% (default) – 10GB Heap x 0.4 = 4GB • Desired flush size at 128MB – 4GB / 128MB = 32 regions max! • For WAL size of 128MB x 0.95% – 4GB / (128MB x 0.95) = ~33 partially uncommitted logs to keep around • Region size at 20GB – 20GB x 32 regions = 640GB raw storage used
- 24. 24 Notes • Compute memstore sizes based on number of written-to regions x flush size • Compute number of logs to keep based on fill and flush rate • Ultimately the capacity is driven by – Java Heap – Region Count and Size – Key Distribution
- 25. 25 Reads • Locate and route request to appropriate region server – Client caches information for faster lookups • Eliminate store files if possible using time ranges or Bloom filter • Try block cache, if block is missing then load from disk
- 26. 26 Seeking with Bloom Filters
- 27. 27 Writes: Where’s the Data at? Older TIME Newer SIZE (MB) 1000 750 500 250 0 Existing Row Mutations Unique Row Inserts
- 28. 28 Block Cache • Use exported metrics to see effectiveness of block cache – Check fill and eviction rate, as well as hit ratios ➜ random reads are not ideal • Tweak up or down as needed, but watch overall heap usage • You absolutely need the block cache – Set to 10% at least for short term benefits
- 29. 29 Testing: Scans HBase scan performance • Use available tools to test • Determine raw and KeyValue read performance – Raw is just bytes, while KeyValue means block parsing • Insert data using YCSB, then compact table – Single region enforced • Two test cases – Small data: 1 column with 1 byte value – Large(r) data: 1 column with 1KB value • About same size for both in total: 15GB ©2014 Cloudera, Inc. All rights reserved.
- 30. 30 Testing: Scans ©2014 Cloudera, Inc. All rights reserved.
- 31. 31 Scan Row Range • Set start and end key to limit scan size
- 32. 32 Best Practices “If you spend too much time thinking about a thing, you'll never get it done.” — Bruce Lee ©2014 Cloudera, Inc. All rights reserved.
- 33. 33 How to Plan Advice on • Number of nodes • Number of disk and total disk capacity • RAM capacity • Region sizes and count • Compaction tuning ©2014 Cloudera, Inc. All rights reserved.
- 34. 34 Advice on Nodes • Use previous example to compute effective storage based on heap size, region count and size – 10GB heap x 0.4 / 128MB x 20GB = 640GB, if all regions are active – Address more storage with read-from-only regions • Typical advice is to use more nodes with fewer, smaller disks (6 x 1TB SATA or 600GB SAS, or SSDs) • CPU is not an issue, I/O is (even with compression) ©2014 Cloudera, Inc. All rights reserved.
- 35. 35 Advice on Nodes • Memory is not an issue, heap sizes small because of Java Garbage Collection limitation – Up to 20GB has been used – Newer versions of Java should help – Use off-heap cache • Current servers typically have 48GB+ memory ©2014 Cloudera, Inc. All rights reserved.
- 36. 36 Advice on Tuning • Trade off throughput against size of single data points – This might cause schema redesign • Trade off read performance against write amplification – Advise users to understand read/write performance and background write amplification Ø This drives the number of nodes needed! ©2014 Cloudera, Inc. All rights reserved.
- 37. 37 Advice on Cluster Sizing • Compute the number of nodes needed based on – Total storage needed – Throughput required for either reads and writes • Assume ≈15MB/s minimum for each read and write – Increasing the KeyValue sizes improves this ©2014 Cloudera, Inc. All rights reserved.
- 38. 38 Example: Twitter Firehose ©2014 Cloudera, Inc. All rights reserved.
- 39. 39 Example: Consume Data ©2014 Cloudera, Inc. All rights reserved.
- 40. 40 HBase Heap Usage • Overall addressable amount of data is driven by heap size – Only read-from regions need space for indexes, filters – Written-to regions also need MemStore space • Java heap space is limited still as garbage collections will cause pauses – Typically up to 20GB heap – Or invest is pause-less GC
- 41. 41 Summary “All fixed set patterns are incapable of adaptability or pliability. The truth is outside of all fixed patterns.” — Bruce Lee ©2014 Cloudera, Inc. All rights reserved.
- 42. 42 WHHAT BRUCE? IT DEPENDS? L ©2014 Cloudera, Inc. All rights reserved.
- 43. 43 Checklist To plan for the size of an HBase cluster you have to: • Know the use-case – Read/write mix – Expected throughput – Retention policy • Optimize the schema and compaction strategy – Devise a schema that allows for only some regions being written to • Take “known” numbers to compute cluster size ©2014 Cloudera, Inc. All rights reserved.