4K Video Downloader Crack + License Key 2025
- 1. Inside Deepseek 3FS: A Deep Dive into AI-Optimized Distributed Storage Stephen Pu [email protected]
- 2. Agenda ■ Parallel file system landscape for AI ■ 3FS deep dive ● System Architecture ● Software Components ● Read / Write Flows ● FUSE (hf3fs_fuse) & USRBIO ■ Which AI storage stack solution is right for your needs?
- 3. Parallel file system landscape for AI Fire-Flyer File System (3FS) Infinia
- 4. Introducing 3FS DeepSeek 3FS (Fire-Flyer File System) is a high-performance parallel file system designed to address the challenges of AI training and inference workloads. ● RDMA and SSD Flash Utilization ● Decentralized Design ● FUSE Optimization (Async Zero-copy API) ● Strong Consistency (Eventual Consistency)
- 5. System Architecture ● FUSE ● Native C++ API （USRBIO） ETCD / ZooKeeper Metadata Service Storage Service SSD SSD SSD Foundation DB (K/V) Chunk Store RocksDB Chunk Allocator Cache CRAQ Primary Node Metadata Service Storage Service Foundation DB (K/V) Chunk Store RocksDB Chunk Allocator Cache CRAQ Node Client Verbs SSD SSD SSD Verbs RDMA Infiniband gRPC
- 7. Software Components • ETCD / ZooKeeper • Metadata Service • FoundationDB • RocksDB • Rendezvous Hashing • Replication Chain • Storage Service • CRAQ • Chunk Store • Chunk Allocator • Chunk Metadata Cache
- 8. Code Structure Overview GitHub Source Code Main Directory Structure 3FS ├── cmake/ # CMake build-related files ├── docs/ # Design documents and user guides ├── examples/ # Example code ├── scripts/ # Auxiliary scripts (deployment, testing, etc.) ├── src/ # Main source code directory │ ├── client/ # Client implementation │ ├── common/ # Common components (network, storage, protocols, etc.) │ ├── metadata/ # Metadata management service │ ├── storage/ # Storage service │ ├── cluster/ # Cluster manager │ ├── transport/ # Network communication layer (including RDMA support) ├── tests/ # Test cases └── CMakeLists.txt # CMake configuration file
- 9. 3FS/ ├── cmake/ # CMake build-related files ├── docs/ # Design documents and user guides ├── examples/ # Example code ├── scripts/ # Auxiliary scripts (deployment, testing, etc.) ├── src/ # Main source code directory │ ├── client/ # Client implementation │ │ ├── api/ # Client API definitions │ │ ├── cache/ # Data caching mechanisms │ │ ├── transport/ # Client-side network communication │ ├── common/ # Common components (network, storage, protocols, etc.) │ │ ├── net/ # Network abstraction layer │ │ ├── data/ # Data structures for storage and metadata │ │ ├── proto/ # Protocol definitions for inter-component communication │ ├── metadata/ # Metadata management service │ │ ├── server/ # Metadata server implementation │ │ ├── storage/ # Metadata storage backend │ │ ├── consistency/ # CRAQ and consistency management │ ├── storage/ # Storage service │ │ ├── engine/ # Data storage engine │ │ ├── replication/ # Replication and high availability │ │ ├── rdma/ # RDMA-based storage optimizations │ ├── cluster/ # Cluster manager │ │ ├── discovery/ # Node discovery and membership management │ │ ├── load_balance/ # Load balancing mechanisms │ │ ├── failover/ # Failure detection and recovery │ ├── transport/ # Network communication layer (including RDMA support) │ │ ├── rdma/ # RDMA transport layer │ │ ├── tcp/ # TCP transport layer │ │ ├── messaging/ # Message serialization and dispatch ├── tests/ # Test cases │ ├── integration/ # Integration tests │ ├── unit/ # Unit tests └── CMakeLists.txt # CMake configuration file Directory Structure
- 10. Data File Store Each SSD deploys a single Chunk Store by default. And A RocksDB instance, Chunk Allocator, Cache Service (Chunk Metadata) SSD Chunk Store RocksDB Chunk Allocator Cache Service Data File Chunk 1 Chunk 2 Chunk N … • RocksDB Instance: Maintains data block metadata and other system information. • Cache (In-Memory): Stores data block metadata in memory to improve query performance. • Chunk Allocator: Facilitates fast allocation of new data blocks.
- 11. File Write Flow Client request Write() RDMA Network src/client/ Parameters: • path • offset • data content src/common/net gRPC RocksDB Foundation DB Metadata Service src/mds/ • FUSE • API • SDK Copy-on-Write, COW Allocates a new block before modifying data. The old block remains readable until all handles are released. • mds_lookup() • mds_allocate_chunk • mds_commit() Block Engine src/block/ Storage • chunk_alloc() • get_block_metadata() • update_block_metadata() Chunk Allocator src/storage/ • storage_write() • submit_io_request() • commit() • sync_metadata_cache() • send_response() inode Storage Service
- 12. File Read Flow Client request read() RDMA Network Libfabric src/client/ Parameters: • path • offset • size src/common/net gRPC RocksDB Foundation DB Metadata Service src/mds/ • FUSE • API • SDK • mds_lookup() • get_block_location() • chunk_cache_hit() Block Engine src/block/ Storage • chunk_alloc() • get_block_metadata() • update_block_metadata() Chunk Allocator src/storage/ • decode_block_data() • apply_read_offset() • return_read_data() inode Storage Service • net_recv_request() • parse_read_request() • dispatch_read_operation() • get_block_data() • read_from_cache() • read_from_ssd() • rdma_transfer_data()
- 13. Chunk Store – Physical Data Block • Data blocks are ultimately stored in physical blocks 1. Physical Block Size: Ranges from 64KiB to 64MiB, increasing in powers of two, with 11 different size classes. 2. Allocation Strategy: The allocator selects the physical block size closest to the actual block size. • Resource Pool Management 1. Each physical block size corresponds to a resource pool, with 256 physical files per pool. 2. The usage state of physical blocks is tracked using an in-memory bitmap. • Recycling and Allocation 1. When a physical block is reclaimed, its bitmap flag is set to 0, its storage space is preserved, and it is prioritized for future allocations. 2. If available physical blocks are exhausted, the system calls fallocate() to allocate a large contiguous space within a physical file, generating 256 new physical blocks to minimize disk fragmentation.
- 14. FUSE & USRBIO
- 15. FUSE • Based on the libfuse low-level API and requires libfuse version 3.16.1 or higher. • 4 kernel-user context switches and one to two data copies, leading to performance bottlenecks. • POSIX: Not support file lock and xattr • Directory Traversal: readdirplus API • readahead: 16 MB by default • Write Buffer：ʼDIOʼ and ʻBuffered IOʼ • Delayed File Size Update： 30s, close, fsync • Async Close • Deleting Files in Write Mode is delayed: write mode, read mode • Recursive Directory Deletion: rm -rf
- 16. USRBIO • A user-space, asynchronous, zero-copy API. • Requires modifications to the application source code for adaptation, making the adoption threshold higher. • Eliminating context switches and data copies, thereby achieving optimal performance.
- 17. USRBIO
- 18. 3FS Design Tradeoffs Highlights Strengths Costs FUSE and Client Access 3FSʼs custom API USRBIO delivers good performance Low usability as users need to modify each application source code to utilize the 3FS custom API; FUSE performance is very low as 3FS is not designed to optimize for FUSE Read vs Write Optimized for read-heavy scenarios Sacrificed write performance, so users with write needs will not fully appreciate the benefits of HPC File Size Optimizations Optimized for large data files Small file workloads are a second class citizen with lower performance despite small files expedition design
- 19. Positioning of Alluxio and 3FS Alluxio Alluxio is a data abstraction and distributed caching layer between compute and storage layers. Alluxio is NOT a PFS (Parallel File System). Key capabilities that a typical PFS does not provide include: ✔ Deep integration with compute frameworks and cloud storage ecosystems. ✔ Providing high-throughput, low-latency hot data caching using commodity hardware on top of data lakes. ✔ Frequently utilized for supporting multi-cloud, hybrid cloud, and cross-data-center data access. Multi-cloud/hybrid cloud/cross-data-center Low-latency Massive small data files 3FS 3FS is a parallel file system designed to leverage high-end hardware. ✔ 3FS abandons the “general-purpose file system” approach of being comprehensive and instead focuses on large data files and high-throughput scenarios in subset of AI workloads. ✔ For the target workloads, it makes the trades-off in optimization by leveraging high-end hardware like RDMA and NVMe. ✔ At the end of the day, 3FS is a new member of the HPC storage family, competing with existing PFSes such as GPFS and Lustre Large data files High bandwidth High-end hardware Alluxio unifies data in local high-speed storage (including 3FS and other PFS) and data lake via Caching, Data Lifecycle Management, Data Migration Complimentary
- 20. 27 Which AI storage stack is right for you? Low cost + massive scale ✅ Low cost, high reliability due to global distribution ❌ Low performance Low cost + massive scale + low latency On top of object storage, Alluxio: ✅ enables low-latency and high throughput with commodity storage such as S3 ✅ manages data loading transparently ✅ provides hybrid and multi-cloud support Leverage high end hardware with custom solution ✅ High performance from leveraging RDMA ❌ Need to manually copy data into 3FS ❌ High cost of specialized hardware Primary Need Leverage high end hardware with global/remote data lakes ❌✅ Fully leverage your existing high end hardware ✅ Alluxio takes care of global data transfer and removes need to manually copy data into 3FS ✅ Alluxio provides hybrid and multi-cloud support Best Fit Trade Offs S3 like object storage alone S3 + 3FS 3FS +
- 21. Alluxio AI Overview Alluxio Accelerates AI by solving speed, scale, & scarcity challenges through high-performance, distributed caching and unified access to heterogeneous data sources.
- 22. Large-scale distributed caching (petabytes of data; billions of objects) - Eliminates I/O bottlenecks - Increases GPU utilization - Improves performance across AI lifecycle Alluxio Accelerates AI Workloads MODEL TRAINING & FINE TUNING MODEL DISTRIBUTION INFERENCE SERVING AI LIFECYCLE DATA COLLECTION & PREPROCESSING
- 23. Future Stay tuned for Part 2 of this webinar series: ● RDMA Network ● CRAQ ● Cluster / Node Management ● Disaster Recovery Algorithm