Scaling HDFS to Manage Billions of Files with Key-Value Stores
20 likes3,861 views
The document discusses scaling HDFS to manage billions of files. It describes how HDFS usage has grown from millions of files in 2007 to potentially billions of files in the future. To address this, the speakers propose storing HDFS metadata in a key-value store like LevelDB instead of solely in memory. They evaluate this approach and find comparable performance to HDFS for most operations. Future work includes improving operations like compaction and failure recovery in the new architecture.
Ad
More Related Content
What's hot (20)
Viewers also liked (20)
Ad
Similar to Scaling HDFS to Manage Billions of Files with Key-Value Stores (20)
Ad
More from DataWorks Summit (20)
Recently uploaded (20)
Scaling HDFS to Manage Billions of Files with Key-Value Stores
- 1. Scaling HDFS to Manage Billions of Files Haohui Mai, Jing Zhao Hortonworks, Inc.
- 2. About the speakers • Haohui Mai • Active committers and PMC in Hadoop • Ph.D. in Computer Science from UIUC in 2013 • Joined the HDFS team in Hortonworks • 250+ commits in Hadoop
- 3. About the speakers • Jing Zhao • Active committers and PMC in Hadoop • Ph.D. in Computer Science from USC in 2012 • HDFS team member in Hortonworks • 250+ commits in Hadoop
- 4. Past: the scale of data
- 5. Past: the scale of data • In 2007 (PC) • ~500 GB hard drives • thousands of files
- 6. Past: the scale of data • In 2007 (PC) • ~500 GB hard drives • thousands of files • In 2007 (Hadoop) • several hundred nodes • several hundred TBs • millions of files
- 7. Past: the scale of data • In 2007 (Hadoop) • several hundred nodes • several hundred TBs • millions of files
- 8. Past: the scale of data • In 2007 (Hadoop) • several hundred nodes • several hundred TBs • millions of files • In 2015 • 4,000+ nodes (10x) • 150+ PBs (1000x) • 400M+ files (100x)
- 9. Present: a generic storage system
- 10. Present: a generic storage system • SQL-On-Hadoop • Machine learning • Real-time analytics • Data streaming • File archives, NFS…
- 11. Present: a generic storage system • SQL-On-Hadoop • Machine learning • Real-time analytics • Data streaming • File archives, NFS… • From MR-centric filesystem to a generic distributed storage system
- 12. Future: Billions of files in HDFS
- 13. Future: Billions of files in HDFS • HDFS clusters continue to grow
- 14. Future: Billions of files in HDFS • HDFS clusters continue to grow • New use cases emerge • IoT, time series data…
- 15. Future: Billions of files in HDFS • HDFS clusters continue to grow • New use cases emerge • IoT, time series data… • Files are natural abstractions of data • Few big files → many small files in HDFS • Billions of files in a few years
- 16. NameNode limits the scale
- 17. NameNode limits the scale • Master / slave architecture • All metadata in NN, data across multiple DNs • Simple and robust NN DN DN DN
- 18. NameNode limits the scale • Master / slave architecture • All metadata in NN, data across multiple DNs • Simple and robust • Does not scale beyond the size of the NN heap • 400M files ~ 128G heap • GC pauses NN DN DN DN
- 19. Next-gen arch: HDFS on top of KV stores
- 20. Next-gen arch: HDFS on top of KV stores • Namespace (NS) on top of Key- Value (KV) stores • Storing the NS into LevelDB
- 21. Next-gen arch: HDFS on top of KV stores • Namespace (NS) on top of Key- Value (KV) stores • Storing the NS into LevelDB • Working set fits in memory, cold metadata on disks • Match the usage patterns of HDFS Namespace
- 22. Next-gen arch: HDFS on top of KV stores • Namespace (NS) on top of Key- Value (KV) stores • Storing the NS into LevelDB • Working set fits in memory, cold metadata on disks • Match the usage patterns of HDFS • Low adoption cost: fully compatible Namespace
- 23. • Introduction • Namespace on top of KV stores • Evaluation • Future work & conclusions
- 24. Encoding the NS as KV pairs • inode_id → flat binary representation of inode • avoid serialization costs • <pid,foo> → the inode id of the child foo whose parent’s inode id is pid struct inode { uint64_t inode_id; uint64_t parent_id; uint32_t flags; … };
- 26. Encoding directories Key Value 1 root 2 foo 3 bar <1,foo> 2 <2,bar> 3 root foo bar
- 27. Encoding directories Key Value 1 root 2 foo 3 bar <1,foo> 2 <2,bar> 3 root foo bar
- 28. Encoding directories Key Value 1 root 2 foo 3 bar <1,foo> 2 <2,bar> 3 root foo bar
- 29. Resolving paths Key Value 1 root 2 foo 3 bar <1,foo> 2 <2,bar> 3
- 30. Resolving paths Key Value 1 root 2 foo 3 bar <1,foo> 2 <2,bar> 3 / foo / bar
- 31. Resolving paths Key Value 1 root 2 foo 3 bar <1,foo> 2 <2,bar> 3 / foo / barID: 1
- 32. Resolving paths Key Value 1 root 2 foo 3 bar <1,foo> 2 <2,bar> 3 / foo / barID: 1 <1,foo>
- 33. Resolving paths Key Value 1 root 2 foo 3 bar <1,foo> 2 <2,bar> 3 / foo / bar <1,foo> ID: 2
- 34. Resolving paths Key Value 1 root 2 foo 3 bar <1,foo> 2 <2,bar> 3 / foo / bar ID: 2 <2,bar>
- 35. Resolving paths Key Value 1 root 2 foo 3 bar <1,foo> 2 <2,bar> 3 / foo / bar <2,bar> ID: 3
- 36. Listing directories Key Value 1 root 2 foo 3 bar <1,foo> 2 <2,bar> 3
- 37. Listing directories Key Value 1 root 2 foo 3 bar <1,foo> 2 <2,bar> 3 $ ls /
- 38. Listing directories Key Value 1 root 2 foo 3 bar <1,foo> 2 <2,bar> 3 $ ls / ID: 1
- 39. Listing directories Key Value 1 root 2 foo 3 bar <1,foo> 2 <2,bar> 3 $ ls / ID: 1 <1,*>
- 40. Listing directories Key Value 1 root 2 foo 3 bar <1,foo> 2 <2,bar> 3 $ ls / foo ID: 1 <1,*>
- 41. Integrate with existing HDFS features
- 42. Integrate with existing HDFS features • HDFS snapshots • Metadata only operations • Append version ids for each key • Map between snapshot ids and version ids
- 43. Integrate with existing HDFS features • HDFS snapshots • Metadata only operations • Append version ids for each key • Map between snapshot ids and version ids • NameNode High Availability (HA) • Use edit logs instead of the WAL of the KV stores to persist operations • Minimal changes in the current HA mechanisms
- 44. Current status • Phase I — NS on top of KV interfaces (HDFS-8286) • NS on top of an in-memory KV store • Under active development • Phase II — Partial NS in the memory • Working set of the NS in the memory, cold metadata on disks • Scaling the NS beyond the size of heap
- 45. • Introduction • Namespace on top of KV stores • Evaluation • Future work & conclusions
- 46. Evaluation • 4 node clusters, connected with 10GbE networks • 6-core Intel Xeon E5-2630 @ 2.3GHz * 2, 64G DDR3 @ 1333 MHz • WDC 1TB disks @ 7200 RPM, 64MB cache • OpenJDK 1.8.0_25
- 47. Evaluation (cont.) Apache Hadoop 2.7.0 2.7.0 InMem LevelDB NS on top of an in- memory KV map NS on top of LevelDB
- 48. NNThroughput (write) • 300,000 operations • 8 threads • Comparable performance • Simpler implementation on delete() • Syncing edit logs is the bottleneck Throughput(ops/s) 0 125 250 375 500 create mkdirs delete rename 2.7.0 InMem LevelDB
- 49. NNThroughput (read) • Read-only operations Throughput(ops/s) 0 50000 100000 150000 200000 open fileStatus 2.7.0 InMem LevelDB LevelDB-opt
- 50. NNThroughput (read) • Read-only operations • 1/3 throughput of vanilla LevelDB v.s. 2.7.0 • Contentions of the global lock in LevelDB during get() Throughput(ops/s) 0 50000 100000 150000 200000 open fileStatus 2.7.0 InMem LevelDB LevelDB-opt
- 51. NNThroughput (read) • Read-only operations • 1/3 throughput of vanilla LevelDB v.s. 2.7.0 • Contentions of the global lock in LevelDB during get() • A lock-free fast path of get() to recover the performance (LevelDB-opt) Throughput(ops/s) 0 50000 100000 150000 200000 open fileStatus 2.7.0 InMem LevelDB LevelDB-opt
- 52. YCSB: Throughput • YCSB against HBase 1.0.1.1 • Enabled short-circuit reads • 100 threads, 10M records Throughput(ops/s) 0 30000 60000 90000 120000 A B C F D E 2.7.0 InMem LevelDB
- 53. Latency(us) 0 2500 5000 7500 10000 A-Read B-Read C -Read F-RM W F-Read E-Scan 2.7.0 InMem LevelDB YCSB: Latency
- 54. Runtime(s) 0 350 700 1050 1400 TeraG en TeraSort TeraValidate 2.7.0 InMem LevelDB TeraSort • 10G data per node • Comparable performance
- 55. Throughput(op/s) 0 32500 65000 97500 130000 Working set 256M 128M 64M 32M Impact on working set size • Throughput of GetFileStatus() under different sizes of working sets
- 56. • Introduction • Namespace on top of KV stores • Evaluation • Future work & conclusions
- 57. Future work • Implementation and stabilization • Operation concerns • Compaction and fsck • Failure recovery • Cold startup
- 58. Conclusions • HDFS needs to continue to scale
- 59. Conclusions • HDFS needs to continue to scale • Evolve HDFS towards KV-based architecture
- 60. Conclusions • HDFS needs to continue to scale • Evolve HDFS towards KV-based architecture • Scaling beyond the size of the NN heap
- 61. Conclusions • HDFS needs to continue to scale • Evolve HDFS towards KV-based architecture • Scaling beyond the size of the NN heap • Preliminary evaluation looks promising
- 62. Acknowledgement • Xiao Lin, interned with Hortonworks in 2013 • PoC implementation of LevelDB backed namespace • Zhilei Xu, interned with Hortonworks in 2014 • Integration between various HDFS features and LevelDB • Performance tuning
- 63. Thank you!
- 65. Integrating with LevelDB Write operations in HDFS • Updating LevelDB inside the global lock • New LevelDB::Write() w/o blocking calls • Write edit log • logSync()
- 66. Integrating with LevelDB Write operations in HDFS • Updating LevelDB inside the global lock • New LevelDB::Write() w/o blocking calls • Write edit log • logSync() Pruning edit logs • Dump memtable into the disks • Update MANIFEST • Prune edit logs