Alluxio - Scalable Filesystem Metadata Services
0 likes2,278 views
This talk was presented by Alluxio's top contributor and PMC Maintainer Calvin Jia at the Alluxio bay area Meetup. This talk shares our design, implementation and optimization of Alluxio metadata service to address the scalability challenges, focusing on how to apply and combine techniques including tiered metadata storage (based on off-heap KV store RocksDB), fine-grained file system inode tree locking scheme, embedded state-replicate machine (based on RAFT), exploration and performance tuning in the correct RPC frameworks (thrift vs gRPC) and etc.
![Tiered Metadata Storage
• Uses an embedded RocksDB to store inode tree
• Solves the storage heap space problem
• Developed new data structures to optimize for storage in RocksDB
• Internal cache used to mitigate on-disk RocksDB performance
• Solves the serving latency problem
• Performance is comparable to previous on-heap implementation
• [In-Progress] Use tiered recovery to incrementally make the
namespace available on cold start
• Solves the recovery problem](https://image.slidesharecdn.com/alluxio-scalablefilesystemmetadataservices-190621210227/85/Alluxio-Scalable-Filesystem-Metadata-Services-15-320.jpg)
Ad
More Related Content
What's hot (20)
Similar to Alluxio - Scalable Filesystem Metadata Services (20)
![JavaOne2016 - Microservices: Terabytes in Microseconds [CON4516]](https://cdn.slidesharecdn.com/ss_thumbnails/javaone2016microservicesfinal-160922163749-thumbnail.jpg?width=560&fit=bounds)
![JavaOne2016 - Microservices: Terabytes in Microseconds [CON4516]](https://cdn.slidesharecdn.com/ss_thumbnails/javaone2016microservicesfinal-160922165618-thumbnail.jpg?width=560&fit=bounds)
Ad
More from Alluxio, Inc. (20)
Ad
Recently uploaded (20)
![Pixologic ZBrush Crack Plus Activation Key [Latest 2025] New Version](https://cdn.slidesharecdn.com/ss_thumbnails/fashionevolution2-250322112409-f76abaa7-250428124909-b51264ff-250504160528-fc2bb1c5-thumbnail.jpg?width=560&fit=bounds)
Alluxio - Scalable Filesystem Metadata Services
- 1. Scalable Filesystem Metadata Services 06/19 Alluxio Meetup Featuring gRPC, Raft, and RocksDB
- 2. ● Release Manager for Alluxio 2.0.0 ● Contributor since Tachyon 0.4 (2012) ● Founding Engineer @ Alluxio About Me Calvin Jia
- 3. Alluxio Overview • Open source data orchestration • Commonly used for data analytics such as OLAP on Hadoop • Deployed at Huya, Two Sigma, Tencent, and many others • Largest deployments of over 1000 nodes
- 5. Architecture
- 7. Alluxio Master • Responsible for storing and serving metadata in Alluxio • Alluxio Metadata consists of files and blocks • Main data structure is the Filesystem Tree • The namespace for files in Alluxio • Can include mounts of other file system namespaces • The size of the tree can be very large!
- 8. Challenges
- 9. Metadata Storage Challenges • Storing the raw metadata becomes a problem with a large number of files • On average, each file takes 1KB of on-heap storage • 1 billion files would take 1 TB of heap space! • A typical JVM runs with < 64GB of heap space • GC becomes a big problem when using larger heaps
- 10. Metadata Storage Challenges • Durability for the metadata is important • Need to restore state after planned or unplanned restarts or machine loss • The speed at which the system can recover determines the amount of downtime suffered • Restoring a 1TB sized snapshot takes a nontrivial amount of time!
- 11. Metadata Serving Challenges • File operations (ie. getStatus, create) need to be fast • On heap data structures excel in this case • Operations need to be optimized for high concurrency • Generally many readers and few writers
- 12. Metadata Serving Challenges • The metadata service also needs to sustain high load • A cluster of 100 machines can easily house over 5k concurrent clients! • Connection life cycles need to be managed well • Connection handshake is expensive • Holding an idle connection is also detrimental
- 13. Solutions
- 14. RocksDB • https://rocksdb.org/ • RocksDB is an embeddable persistent key-value store for fast storage
- 15. Tiered Metadata Storage • Uses an embedded RocksDB to store inode tree • Solves the storage heap space problem • Developed new data structures to optimize for storage in RocksDB • Internal cache used to mitigate on-disk RocksDB performance • Solves the serving latency problem • Performance is comparable to previous on-heap implementation • [In-Progress] Use tiered recovery to incrementally make the namespace available on cold start • Solves the recovery problem
- 16. Tiered Metadata Storage 16 Alluxio Master Local Disk RocksDB (Embedded) ● Inode Tree ● Block Map ● Worker Block Locations On Heap ● Inode Cache ● Mount Table ● Locks
- 17. RAFT • https://raft.github.io/ • Raft is a consensus algorithm that is designed to be easy to understand. It's equivalent to Paxos in fault- tolerance and performance. • Implemented by https://github.com/atomix/copycat
- 18. Built-in Fault Tolerance • Alluxio Masters are run as a quorum for journal fault tolerance • Metadata can be recovered, solving the durability problem • This was previously done utilizing an external fault tolerance storage • Alluxio Masters leverage the same quorum to elect a leader • Enables hot standbys for rapid recovery in case of single node failure
- 19. gRPC • https://grpc.io/ • gRPC is a modern open source high performance RPC framework that can run in any environment • Works well with Protobuf for serialization
- 20. gRPC Transport Layer • Connection multiplexing to reduce the number of connections from # of application threads to # of applications • Solves the connection life cycle management problem • Threading model enables the master to serve concurrent requests at scale • Solves the high load problem • High metadata throughput needs to be matched with efficient IO • Consolidated Thrift (Metadata) and Netty (IO)
- 21. gRPC Transport Layer 21 Thrift (Metadata) Netty (IO) Alluxio Master Alluxio Worker Alluxio Worker Alluxio Client Alluxio Master Alluxio Worker Alluxio Worker Alluxio Client gRPC (Metadata + IO)
- 22. Questions? Alluxio Website - https://www.alluxio.io Alluxio Community Slack Channel - https://www.alluxio.io/slack Alluxio Office Hours & Webinars - https://www.alluxio.io/events