Speeding Up Atlas Deep Learning Platform with Alluxio + Fluid
0 likes167 views
Data Orchestration Summit 2020 organized by Alluxio https://www.alluxio.io/data-orchestration-summit-2020/ Speeding Up Atlas Deep Learning Platform with Alluxio + Fluid Yuandong Xie, Platform Researcher (Unisound) About Alluxio: alluxio.io Engage with the open source community on slack: alluxio.io/slack
More Related Content
What's hot (20)
Similar to Speeding Up Atlas Deep Learning Platform with Alluxio + Fluid (20)
More from Alluxio, Inc. (20)
Recently uploaded (20)
![Pixologic ZBrush Crack Plus Activation Key [Latest 2025] New Version](https://cdn.slidesharecdn.com/ss_thumbnails/fashionevolution2-250322112409-f76abaa7-250428124909-b51264ff-250504160528-fc2bb1c5-thumbnail.jpg?width=560&fit=bounds)
Speeding Up Atlas Deep Learning Platform with Alluxio + Fluid
- 1. Speeding Up In Atlas Deep Learning Platform with Alluxio+Fluid Yuandong Xie [email protected] Platform Researcher of Unisound AI labs
- 2. Agenda • Introduction to Unisound & Atlas AI Platform • Challenges & Solutions • Test scenarios & Results • Analysis of speeding up • Contribution to Fluid • Conclusion & Next steps
- 3. Unisound AI Service AI UI AI Chip Chip Cloud Edge Cloud AI Service Provider Application: Medical、Education AI Chip Provider Application: Home Interactive system Provider Application: Car Education Home Medical Car Security Financial Atlas support all application and expand new business
- 4. Atlas AI Platform Infrastructure Computing Controller Manager Application GPU clusterCPU cluster Distributed storage cluster Machine Learning Model Data Preprocessing Feature Extraction Statistical Analysis Resource Management Job Scheduler Model Registry User Management Logging and Monitoring Image Registry Semantic Processing Recommendation System Data Mining Text Analysis Speech Processing Image Processing 100G InfiniBand High Performance Network Deep Learning Model
- 5. Platform Storage - Lustre E • Distributed file storage system provides continuous large space (PB scale) • Multi-Lustre clusters with different networks （ InfiniBand and Ethernet ） • Data verification and replication, ensure data security • Support online expansion and version update Site: https://documen.site/download/03-session-intel-manager-for-lustre_pdf
- 6. Challenges with the current architecture • Storage IO Pressure • Small file(<1Mb) increases pressure of MDT and OST • Single-node multi-task IO preemption • Data duplication
- 7. Impacts of the challenge • IO pressure -> Hardware damage • Small files -> Low QoS • IO preemption -> Resource waste • Duplicate data -> Storage waste
- 8. Current solution for challenges $ lctl set_param ost.OSS.ost_io.nrs_policies="tbf nid” $ lctl set_param ost.OSS.ost_io.nrs_tbf_rule="start other_clients nid={client@o2ib} rate=80" • Limit client IO • Pack small files • Priority scheduling to idle nodes • Check duplicate data Not effective
- 9. New Architecture of Atlas E Computer Job Node Alluxio-fuse Pod Worker Job Worker Pod RAM/SSD/HDD fuse Short circuit Access Control (UGO+RWX) Node Alluxio-fuse Pod Worker Job Worker Pod RAM/SSD/HDD fuse Short circuit Dataset Runtime Master Job Master Pod Access Control (UGO+RWX) /mnt /mnt/$group/$user Dataload Cache(Alluxio+Fluid) Client(atlasctl) Storage(Lustre)
- 10. Alluxio is all we need & Why Poor performance Low QoS Good performance Limited scalability Expensive Good Performance Good scalability Cheap! Atlas Atlas Atlas Customized Storage
- 11. Fluid is all we need & Why Data Type Application Fluid-SchedulerFluid-controller-manager Medium size files Massive small files How to manage different user`s dataset? Natural Language Processing … … Speech Processing Computer Vision Large size files How to speed up different type dataset? How to auto scale alluxio engine? Fluid-Function … How to schedule job with cache locality?
- 12. Scenarios Test • Datasets : • Massive small file (<10Mb) • Medium size file(>100Mb,<1Gb) • Large size file(>100Gb) • Different Scenarios: • Noise Reduction • Image Classification • Optical Character Recognition(OCR) • Comparative Test: • Read directly from Lustre • Read from Cache（Alluxio Engine）
- 13. Experiment #1 --- Massive small files • Test Details : • Noise Reduction Application • 500000 files ,total size 183 Gb • Cache in Memory • Three experiments: • Load raw wav files from lustre • Load clear data from Alluxio(cold), noise data from lustre • Load all data from Alluxio(warm)
- 14. Massive small files --- Speed comparision • Test Results: Speed Up: 10x Massive small files Test Results
- 15. Massive small files --- Bandwidth and GPU Usage comparision • Test Results: 230 Mb/s -> 0 Up: 10% I/O Bandwidth
- 16. Experiment #2 --- Medium size files • Test Details : • Image Classification Application • ImageNet TFRecord (Avg size: 138Mb) • Cache in Memory • Comparative Test: • Run 10 GPUs job (Exclusive) on the same node • Run 7 GPUs job (Preemption) on the same node
- 17. Medium size files • Test Summary Lustre Alluxio(warm) Memory Theoretical peak (Preemption)2500 steps (images/s) per gpu 236.9 601.8 N/A 768.9 (Exclusive)4000 step (images/s) per gpu 247.2 702.6 699.1 765.9 E2E Time 50 min 20 min 15 min • Notes : • Lustre: directly load data from distributed storage system • Alluxio(warm): pre-warm data in Cache • Memory: directly load data from memory • Theoretical peak: load synthetic data Speed Up: 2.5x
- 18. Experiment #3 --- Large size files • Test Details : • Optical Character Recognition(Application) • LMDB size: 125 Gb • Cache in Memory • Three experiments: • Read LMDB from lustre • Read LMDB from Alluxio, but not pre-warm • Read LMDB from Alluxio, pre-warm
- 19. Large size files --- Speed comparision • Test Result: Speed Up: 30x All in Memory Large size files Test Results
- 20. Large size files --- Bandwidth and GPU Usage comparision • Test Result: 1300 Mb/s -> 0 15.6 Mb/s -> 0 Up: 31.5% I/O Bandwidth
- 21. Results Summary • Fluid with Alluxio engine can greatly speed up different type datasets • Fluid with Alluxio engine can increase the GPU usage of the task • Fluid with Alluxio engine can significantly reduce the pressure on the Lustre distributed file system
- 23. Conclusion • Alluxio : High-performance architecture • Fluid customizes and optimizes Alluxio parameters for different datasets of scenarios • Alluxio perfect integration : • Data is always immutable in deep learning life cycle • Deep learning need deterministic job • Fluid perfect integration : • Locality based scheduling is needed for Lustre • Cache share different task is needed
- 24. Next Steps • More Scenarios for Fluid with Alluxio Engine • More Best Practice Sharing • Continuous Content Contributions & Community Activities • ……
- 25. Just for Smart Life Thanks