Hive integration: HBase and Rcfile__HadoopSummit2010
15 likes2,347 views
The document outlines the integration of HBase with Hive at Facebook, focusing on reducing latency for dimension data and managing large datasets with RCFile format. It discusses the current testing outcomes, including performance benchmarks, storage handler configurations, and incremental loading speeds. The integration roadmap includes plans for optimization, new features, and future enhancements to support indexing and improve overall performance.
![Questions? [email_address] [email_address]](https://image.slidesharecdn.com/2hiveintegrationhadoopsummit2010-100630134433-phpapp01/85/Hive-integration-HBase-and-Rcfile__HadoopSummit2010-13-320.jpg)
Hive integration: HBase and Rcfile__HadoopSummit2010
- 1. Hive Integration: HBase and RCFile John Sichi and Yongqiang He Facebook
- 2. HBase Integration (John Sichi) RCFile Integration (Yongqiang He) Session Agenda
- 3. HBase: Facebook Warehouse Use Case Reduce latency on dimension data availability HBase (Dimension data) Partitioned RCFiles (Fact data) Periodic Load Continuous Update Hive Queries
- 4. HBase: Storage Handler CREATE TABLE users( userid int, name string, email string, notes string) STORED BY 'org.apache.hadoop.hive.hbase.HBaseStorageHandler' WITH SERDEPROPERTIES ( “ hbase.columns.mapping” = “ small:name,small:email,large:notes”) TBLPROPERTIES ( “ hbase.table.name” = “user_list” ); INSERT, SELECT, JOIN, GROUP BY, UNION etc
- 5. Testing at scale 20-node test cluster Bulk-loaded 6TB of gzip-compressed data from Hive into Hbase in about 30 hours Incremental-loaded from Hive into Hbase at 30GB/hr (with write-ahead logging disabled) Full-table scan queries: currently 5x slower than against native Hive tables (no tuning or optimization yet) HBase: Integration Status
- 6. Retest against HBase trunk with larger (30TB) data Try out new features for accelerating incremental load Bulk load into table with existing data Multiputs Deferred logging Support for “virtual partitions” based on timestamps Support for deletion Push down filters Index join? Optimize scans? HBase: Integration Roadmap
- 7. Why Columnar Storages Better Compression Light weight compression RLE Bit-map Etc CPU, Memory, Storage Columnar Operator Cache conscious (MonetDB) RCFile
- 8. Why RCFile Huge Data Reduce data storage space required Ad-hoc workloads Storage space vs. speed (data performance) Can we get both with no application changes? Reduce storage spaces Accelerate performance for arbitrary applications RCFile
- 9. Pros Work with Column Pruning Only touch needed columns at runtime Lazy decompression Select col1, col2 from tbl_col_10 where col_1 > 30 Will only touch col1 and col2 Col2 is decompressed only when a block contains a col1 value greater than 30 RCFile
- 10. Cons Row Construction Is the main overhead Each column’s data is stored separately, and may be sorted in different order In memory operation for rcfile This could be really painful; a lot of room to improve here RCFile
- 11. Facebook Deployment Default file format in Facebook cluster 20% space savings on average We are transforming old data to the new format RCFile
- 12. Future work Support built in indexing Like bloom filter etc more cache conscious columnar operators Pushing predicate to file reader RCFile
Editor's Notes
- #2: This is the Title slide. Please use the name of the presentation that was used in the abstract submission.
- #3: This is the agenda slide. There is only one of these in the deck.
- #6: This is a topic/content slide. Duplicate as many of these as are needed. Generally, there is one slide per three minutes of talk time.
- #7: This is a topic/content slide. Duplicate as many of these as are needed. Generally, there is one slide per three minutes of talk time.
- #8: This is a topic/content slide. Duplicate as many of these as are needed. Generally, there is one slide per three minutes of talk time.
- #9: This is a topic/content slide. Duplicate as many of these as are needed. Generally, there is one slide per three minutes of talk time.
- #10: This is a topic/content slide. Duplicate as many of these as are needed. Generally, there is one slide per three minutes of talk time.
- #11: This is a topic/content slide. Duplicate as many of these as are needed. Generally, there is one slide per three minutes of talk time.
- #12: This is a topic/content slide. Duplicate as many of these as are needed. Generally, there is one slide per three minutes of talk time.
- #13: This is a topic/content slide. Duplicate as many of these as are needed. Generally, there is one slide per three minutes of talk time.
- #14: This is the final slide; generally for questions at the end of the talk. Please post your contact information here.