Fluid: When Alluxio Meets Kubernetes
1 like188 views
Data Orchestration Summit 2020 organized by Alluxio https://www.alluxio.io/data-orchestration-summit-2020/ Fluid: When Alluxio Meets Kubernetes Yang Che, Staff Engineer at Alibaba Cloud Rong Gu, Professor at Nanjing University About Alluxio: alluxio.io Engage with the open source community on slack: alluxio.io/slack
![[MeetUp][1st] 오리뎅이의_쿠버네티스_네트워킹](https://cdn.slidesharecdn.com/ss_thumbnails/3-191024235922-thumbnail.jpg?width=560&fit=bounds)
![[오픈소스컨설팅]오픈스택에 대하여](https://cdn.slidesharecdn.com/ss_thumbnails/random-151125103039-lva1-app6892-thumbnail.jpg?width=560&fit=bounds)
![[Event] Digital transformation : Enterprise cloud one os one click - PRESENTA...](https://cdn.slidesharecdn.com/ss_thumbnails/eventdigitaltransformation-enterprisecloudoneosoneclick-presentationnutanix-170922084203-thumbnail.jpg?width=560&fit=bounds)
Ad
More Related Content
What's hot (20)
Similar to Fluid: When Alluxio Meets Kubernetes (20)
![[DevDay 2016] OpenStack and approaches for new users - Speaker: Chi Le – Head...](https://cdn.slidesharecdn.com/ss_thumbnails/slideopenstack-devday2016-160414042927-thumbnail.jpg?width=560&fit=bounds)
Ad
More from Alluxio, Inc. (20)
Ad
Recently uploaded (20)
![Download Wondershare Filmora Crack [2025] With Latest](https://cdn.slidesharecdn.com/ss_thumbnails/neo4j-howkgsareshapingthefutureofgenerativeaiatawssummitlondonapril2024-240426125209-2d9db05d-250419-250428115407-a04afffa-thumbnail.jpg?width=560&fit=bounds)
Fluid: When Alluxio Meets Kubernetes
- 1. Rong Gu (Nanjing University) Yang Che (Alibaba Cloud Kubernetes Team) Fluid: When Alluxio meets Kubernetes https://github.com/fluid-cloudnative/fluid
- 2. What IS THIS TALK ABOUT? Data and Application Orchestration In Kubernetes
- 3. an open-source system for automating deployment, scaling, and management of containerized applications. Kubernetes (k8s) is...
- 4. Ecosystem and large Open Source Community Kubernetes Advantages 01 03 04 02 Portability inherit from Docker, better isolation for performance and security Out of box management: Self-healing, Autoscaling, Security, Logging, Monitoring, etc… Heterogeneous Deployment in different cloud
- 5. Computing ECS, EBM, GPU, FPGA, ECI Network VPC, ENI, RDMA, SLB, DNS Storage EBS, NAS, CPFS, OSS Kubernetes Registry Service Marketplace DevOps Micro Service/Service Mesh Enterprise Applications Innovation Gitlab Jenkins GitOps EDAS Istio .net JEE AI Blockchain IoTC++Spring Cloud Public Cloud Edge Computing Private Cloud Serverless Application Gene Encoding Big Data AI Spark Apache Flink TensorFlow PyTorch Operating System (Linux, Windows) Presto MxNet Caffe TF-Serving More and More Data Driven Applications run on Kubernetes
- 6. Cloud SSD But the data accessing in K8s is slow 01 GPUs requires High I/O Throughput and Low Latency 02 Separation of computing and storage causes I/O latency 03 Single-machine cache can hardly meet massive data acceleration
- 7. Alluxio is... A data orchestration platform fundamentally enables separation of storage and compute.
- 8. Motivations towards Alluxio With Kubernetes ● Poor performance ● Poor scalability ● Good performance ● Limited scalability ● Expensive ● Good Performance ● Good scalability ● Low Cost! Lustre Alibaba OSS Alibaba OSS Alibaba OSS
- 9. Kubernetes Scheduler is no aware of cache locality Cache capabilities should be also scalable Cache is important, but the challenges are K8S node K8S node fuse K8S node fuse IMAGENET fuse Distributed Data Cache Po d Po d MXNet MXNet Po d Po d Caffe Caffe OSS/NAS/HDFS CLEVR TensorFlow TensorFlow CIFAR-10 V100 P100 M40 0 1 Resource Pooling causes Cache invalidation • Requires Portability 02 03
- 10. Fluid Provide best Alluxio data orchestration experience in Kubernetes
- 11. Advantages：The Benefit of Alluxio + Fluid Advantages 1 Alluxio automates the data orchestration and management Advantages 2 Fluid orchestrates and auto-scales Alluxio in Kubernetes Fluid is responsible for scheduling data-driven job with data locality knowledge Advantages 3 Advantages 4 Fluid automates the user-facing operations Advantages 5 Fluid makes cache access from Kubernetes transparent
- 12. Architecture
- 13. How to Use Fluid apiVersion: data.fluid.io/v1alpha1 kind: Dataset metadata: name: imagenet spec: mounts: - mountPoint: oss://imagenet/train name: train options: fs.oss.accessKeyId: xxx fs.oss.accessKeySecret: yyy fs.oss.endpoint: oss-cn-huhehaote.aliyuncs.com - mountPoint: pvc://ceph-pvc name: validation 1.Create Dataset 2.provision 3. Create Pod apiVersion: batch/v1 kind: Pod metadata: name: resnet50 spec: containers: - name: train image: resnet50 volumeMounts: - mountPath: /data name: imagenet volumes: - name: imagenet persistentVolumeClaim: claimName: imagenet ImageNet PV/PVC Pod /data train (OSS) Validation(PVC)
- 14. How to check the dataset Data requires to cache Dataset Runtime Status: Cache States: Cache Capacity: 200.00GiB Cached: 0B Cached Percentage: 0% Conditions: Message: The ddc runtime is ready. Reason: DatasetReady Status: True Type: Ready Phase: Bound Runtimes: Category: Accelerate Name: imagenet Namespace: default Type: alluxio Ufs Total: 84.29GiB Status: Cache States: Cache Capacity: 600GiB Cached: 76.08GiB Cached Percentage: 90.3% Conditions: # more conditions Current Fuse Number Scheduled: 4 Current Master Number Scheduled: 1 Current Worker Number Scheduled: 4 Desired Fuse Number Scheduled: 4 Desired Master Number Scheduled: 1 Desired Worker Number Scheduled: 4 Fuse Number Available: 4 Fuse Status: True Type: Ready Phase: Bound Runtimes: Category: Accelerate Name: imagenet Namespace: default Type: alluxio Events: <none> er Ready: 4 Fuse Phase: Ready Master Number Ready: 1 Master Phase: Ready Value File: imagenet-alluxio-values Worker Number Available: 4 Worker Number Ready: 4 Worker Phase: Ready Current Cache Capabilities Current Number of workers • Understand the current cache capabilities • Determine if need to scale out
- 15. Schedule job with the dataset locality client apiVersion: batch/v1 kind: Pod metadata: name: resnet50 spec: containers: - name: train image: resnet50 volumeMounts: - mountPath: /data name: imagenet volumes: - name: cifar10 persistentVolumeClaim: claimName: imagenet Kubernetes Scheduler Fluid Scheduler Fluid Runtime Service Alluxio Alluxio N1 N2 N3 10G Cached 5G Cached Pod 1. Create Pod 2.Find the cacheable Node 3.Query the cache capabilities of then odes 4.Start pod in N1
- 16. Demo 2 Accelerate Remote File Accessing with Fluid Machine Learning with Fluid Accelerate PVC with Fluid Demo 1 Demo 2 Demo 3 Demo
- 17. End-to-End Optimization Workload: ResNet50 && ImageNet
- 18. Github Repo: https://github.com/fluid-cloudnative/fluid Project Homepage: http://pasa-bigdata.nju.edu.cn/fluid/index.html DingDing Talk Group: Learn More about Fluid
- 19. THANKS! Rong Gu, Nanjing University Yang Che, Alibaba Cloud Kubernetes Team