Atom: A cloud native deep learning platform at Supremind
1 like785 views
Supremind developed a deep learning platform called Atom to manage large-scale training data and models across multiple clouds. Atom uses Alluxio for data orchestration to provide a unified view of distributed data and optimize for data locality. Atom addresses challenges of metadata management, data optimization, and supporting both training and inference workloads. Future work includes improving distributed training, model optimization, and adapting Atom to various hardware platforms.
Ad
More Related Content
What's hot (20)
Similar to Atom: A cloud native deep learning platform at Supremind (20)
Ad
More from Alluxio, Inc. (20)
Ad
Recently uploaded (20)
Atom: A cloud native deep learning platform at Supremind
- 1. Atom：A cloud native deep learning platform at Supremind 李李朝光 Chief Architect, Supremind [email protected]
- 2. Agenda • Who and where we are • Deep training platform (Atom) introduction • Alluxio data echosystem • Atom data management • Atom dataset and workflow management • Future work
- 3. Who and where we are? ATLAB 闪⻢马智能 Supremind Image censor Traffic analysis Crowd gathering Abnormal behavior analysis for picture/video
- 4. Why we need a deep learning platform • Algorithm team & joint laboratory • Easy to use portal • Can not manage data efficiently in cloud • Data virtualization globally • Dataset design • Resource utilization is low • GPU resource pool • Model transition is painful • Tron: +20% performance • Service publishing is painful • Verify/test in 3rd party (aliyun) • Workflow for repeat job
- 5. Agenda • Who and where we are • Deep training platform (Atom) introduction • Alluxio data echosystem • Atom data management • Atom dataset and workflow management • Future work
- 6. Atom - Backbone of supremind production line • 60B+ picture/video • B+ added per day Mass data Knowledge Training Vide structure Knowledge graph Search Research and development team Operation team Deep learning platform Ato m • Atom is developed during d2d deep training • High quality model trained and rectified by Atom platform Production Cooperatio n VAS Video analysis Platform城市⼤大数据 LEGO Big data Platform Crawler Traffic Security Censor Monitor Model detection Face Car plate ReID Crown Behavior Video mgt Camera drive Cloud data Streaming Label system
- 7. Atom labelx labeling sub-system Censo r Car/ Bike Person Trackin g Cloud Private Pre-process Cleaning Label Audit Publish Interface Operating system Certif y Mentor Channel Finance Video Imag e Text QualityTraining Award Ato m Training
- 8. Atom deep training platform architecture Label Dataset Trainin Model Atom deep learning Distributed Distributed Distributed Distributed Schedule Object Bloc GPU Trainin Virtualization, orchestration, docker, kubernetes Hardware, switch Label Dataset Trainin Model Productio Tron/
- 9. Data orchestration in deep learning platform Simple data management: NFS: shared, poor performance Local cluster data management: CEPH(RBD): copying, duplicated Global object data management: Data: alluxio + qiniu object storage Careful resource planning for alluxio Dataset and data flow management: new challenge Intuitive solution Not designed for large scale data Need Meta mgt & Data mobility Later on …
- 10. Agenda • Who and where we are • Deep training platform (Atom) introduction • Alluxio data echosystem • Atom data management • Atom dataset and workflow management • Future work
- 11. Alluxio and Data Ecosystem • Data Locality: • Accelerate big data workloads with transparent tiered local data • Data Accessibility: • Run Spark, Hive, Presto, ML workloads on your data located anywhere • Data Elasticity: • Abstract data silos & storage systems to independently scale data on- demand with compute Apache 2.0 License 4000+ Github Stars 1000+ Github Contributors Originated as Tachyon project at the UC Berkley’s AMP Lab
- 12. Alluxio File System APIs • Spark • Hadoop • POSIX (e.g., Tensorflow and etc) • Java > rdd = sc.textFile(“alluxio://localhost:19998/myInput”) $ hadoop fs -cat alluxio://localhost:19998/myInput $ cat /mnt/alluxio/myInput FileSystem fs = FileSystem.Factory.get(); FileInStream in = fs.openFile(new AlluxioURI("/myInput"));
- 13. Alluxio Reference Architecture Alluxio MasterZookeeper / RAFT Standby Master WAN Alluxio Client Alluxio Client Alluxio Worker RAM / SSD / HDD Alluxio Worker RAM / SSD / HDD … Application Application Under Store 1 Under Store 2
- 14. Agenda • Who and where we are • Deep training platform (Atom) introduction • Alluxio data echosystem • Atom data management • Atom dataset and workflow management • Future work
- 15. Access global object storage for deep training platform FUSE Adapter (on host) FUSE /a1 /a2 alluxio Virtual File System Deep learning training program KODO OSS FUSE /a1 /a2 • Data • Hundreds of TB image • Motivation: • Use cloud data efficiently • Problem: • Storage capacity is limited (CEPH) • Storage sub-sys is not scalable • Deep learning can’t use cloud object • Challenge: • Fuse overhead • Resource allocation • Random read support • First time access Qiniu object storage Aliyun object storage
- 16. Data optimization in deep training platform • Data • Several PB training data • Hundreds of PB inference data • Motivation: • Use data efficiently • Problem: • Resource starvation • Meta data (master) • Video clipping • Distributed training • Optimization: • Shard • Short circuit • Local worker • Cache policy • Data warmup Attr cache Local worker App POD Alluxio cluster 1 Alluxio cluster 2 Block dev pool Short circuit
- 17. Data locality Problems: • Fuse: 30% off • Random r/w • Mmap Solution: • Send worker to GPU • Send data to worker Gain: • Load r/w performance • Random IO perf • Mmap/Direct io Alluxio cluster warmup service Atom Atom scheduler Worker scheduler GPU Atom io agent Alluxio fuse Block dev Alluxio worker GPU Atom io agent Alluxio fuse Block dev Alluxio worker
- 18. Metadata management Challenges: • All memory meta data • Journal in shared folder • Mobility Solutions: • Off-heat • Disk-based • Meta-caching policy • Fuse cache Alluxio-2.0 (preview): • Raft for journal • RocksDB w/ inode cache On heap B+ tree Off heap (mapdb) JNI Share storage (cephfs) Heap cache Block dev Heap cache Block dev Raft Fuse cache inside alluxio ! ! Objects Datasets Training Alluxio cache Training Alluxio cache Datasets Index cluster
- 19. New challenge • Motivation: • Support both training and business • Cost control • Problem: • Data lake • GPU starvation • Star-up mode • Challenges: • Multiple vendors support • Cluster federation (Kubernetes) • Workflow management • Day-to-day training and inference • Complicated data flow managemen • Dataset design Customer Data Cluster federation Qiniu AVA + Alluxio Aliyun AVA + Alluxio Local cluster AVA + Alluxio AWS AVA + Alluxio
- 20. Agenda • Who and where we are • Deep training platform (Atom) introduction • Alluxio data echosystem • Atom data management • Atom dataset and workflow management • Future work
- 21. Alluxio cluster AWS KODO OSSDataset Dataset manager schedulerWarmup srv Data clearn Cloud adpt Training Kubernetes cluster /dir1 ——/dir1-1 ————/dir1-1-1 ——/dir1-2 /dir1 ——/dir1-1 ————/dir1-1-1 ——/dir1-2 Labelx subsystem Service runtime interfaces Service layer Front end portal Atom: adapt to data challenge Pod Pod • Problems • Not production ready • Manual intervention • Not cross cluster • Features • Based on cloud • Access everywhere • Light-weight • Movable • Versioned • Easy to use (file system) • Good performance • Solution • Dataset design • Send data to compute
- 22. Atom dataset management flow Label Ato m DL platform Datasets Evaluate Feedback Training set Testing set Model versionsDeploy model Model Data gathering Online feedback LableX system Lablex pipeline Label personel Data extractor Label Export ReviseDeploy Send dataset to service! Dataset & Alluxio engine
- 23. Atom Workflow management • Why/how work flow management • Multiple clouds support • Data accessible every where (via alluxio) • Need cluster federation • A training job launched in backend • Production efficiency Go: Train on cluster x 1 2 3 4 Pre-defined Workflow Define parameters Resources requirements
- 24. Agenda • Who and where we are • Deep training platform (Atom) introduction • Alluxio data echosystem • Atom data management • Atom dataset and workflow management • Future work
- 25. Future work • Datasets for Distributed Training • Param exchange overwhelm network • Model optimization and chip adaption • Model conversion & compaction (Tron & Shadow) • ARM, AMD, Huawei, etc