Alluxio Webinar - Maximize GPU Utilization for Model Training

Jun 26, 20230 likes137 views

Alluxio Webinar June 26. 2023 For more Alluxio Events: https://www.alluxio.io/events/ Speaker: Tarik Bennett (Senior Solutions Engineer, Alluxio) Beinan Wang (Tech Lead, Alluxio) When training models on ultra-large datasets, one of the biggest challenges is low GPU utilization. These powerful processors are often underutilized due to inefficient I/O and data access. This mismatch between computation and storage leads to wasted GPU resources, low performance, and high cloud storage costs. The rise of generative AI and GPU scarcity is only making this problem worse. In this webinar, Tarik and Beinan discuss strategies for transforming idle GPUs into optimal powerhouses. They will focus on cost-effective management of ultra-large datasets for AI and analytics.

From Idle to Optimal:
Maximize GPU Utilization
for Model Training
Beinan Wang
Tarik Bennett

Senior Staﬀ Engineer @ Alluxio
Trino Contributor
PrestoDB Committer
Senior Solutions Engineer
@ Alluxio
Dr. Beinan Wang
Tarik Bennett
2

Source: https://www.wsj.com/articles/rush-to-use-generative-ai-pushes-companies-to-get-data-in-order
“Training large language models requires ready
access to vast amounts of data, whose storage,
processing, and protection can be costly.”

High Scalability
Training billions files
ESSENTIAL
High Availability
99.99%
ESSENTIAL
High Performance
Higher GPU utilization
ESSENTIAL
Always Increasing Expectations…
Icons created by kerismaker, HJ Studio - Flaticon

What Does Managing Data Involve?
Data Preprocessing
Improving the quality and reliability
of the data for model training
Model Training
Read training data, vision (image) or
NLP/LLM (text), for DL using GPUs
Model Deployment
Consumption of trained models for
online or oﬀline inference
Feature Engineering
Selecting relevant and informative
features from raw data
PyTorch | Tensorflow | Spark
Spark PyTorch | Tensorflow | Spark
Model
Training Data Result
Model
Compute
Stage
Spark | Trino | Presto
Result
Curated Data
Not discussed today
- Security
- Privacy (PII)
- Data Cleaning
- Data Pipelines
- Data Governance
Curated /
Processed Data

100,000,000,000,000,000,000,000
bytes of data will be stored in the cloud by 2025
6
Source: Cybersecurity Ventures

Issues Managing Ultra-Large Datasets
Non-Functional Storage Requirements
High Performance
- Many options
Cost-Effective
- Commodity storage

10%
of your data is hot data
8
Source: Alluxio

9
Data
Caching
Helps
Boost
Performance
Save Costs
Prevent
Network
Congestion
Oﬀload
Under
Storage

10
Data Caching at Uber Scale
3 Clusters, 1500 Nodes
Source: https://www.uber.com/blog/speed-up-presto-with-alluxio-local-cache/
50%
Input Read
Performance
10%
Data Read Traﬀic
to HDFS

GPUs are
scarce
GPUs are
expensive
Challenges as you try to scale
Low GPU
Utilization

Addressing Low
GPU Utilization with Caching
13

Architecture Overview
Online ML platform
Inference cluster
Models
Models
Training Data
Models
1
2
3
4
5
Offline training platform
Alluxio
Training cluster
Training Data
2
14

AI Training Test with Alluxio
15
Local Folder /dataset
Alluxio
GPU Training
Remote Storage
Kubernetes
Interactive
Notebook
Alluxio
Operator
Visualization
Dashboard

Test Setup
● Alluxio via Kubernetes - Provides caching for training data
● GPU server - AWS EC2/Kubernetes
● Deep learning algorithm (CV) - ResNet (one of the most popular CV algorithms)
● Deep learning framework - PyTorch
● Dataset - ImageNet (subset - ~35k images, each is ~100kB - 200kB)
● Dataset storage - S3 (single region)
● Mounting - FUSE
● Visualization - TensorBoard
● Code execution - Jupyter notebook
16

Training Test Steps
1. Loading the dataset into Alluxio
2. Running the training job
3. Reading the dataset from Alluxio through PyTorch
DataLoader in each epoch
4. Visualizing the GPU utilization and other metrics
17

18
Training Directly from Storage
- > 80% of total time is spent in DataLoader
- Result in Low GPU Utilization Rate (<20%)
Visualization Dashboard Results (Control)

19
Visualization Dashboard Results (Alluxio)
Training with Alluxio
- Reduced DataLoader Rate from 82% to 1% (82X)
- Increase GPU Utilization Rate from 17% to 93% (5X)

Source: https://developer.nvidia.com/blog/accelerating-analytics-and-ai-with-alluxio-and-nvidia-gpus/
“The beneﬁts from GPU acceleration are limited
if data access dominates the execution time. “

Thank You
twitter.com/alluxio slackin.alluxio.io
linkedin.com/alluxio
www.alluxio.io
JOIN THE CONVERSATION
ON SLACK
ALLUXIO.IO/SLACK

Alluxio Product School Webinar Aug. 15, 2023 For more Alluxio Events: https://www.alluxio.io/events/ Speaker: Roland Theron (Senior Solutions Engineer, Alluxio) Organizations are retooling their enterprise data infrastructure in the race for AI/ML. However, growing datasets, extensive data engineering overhead, high GPU costs, and expensive specialized storage can make it difficult to get fast results from model development. The data access layer is the key to accelerating your path to AI/ML. In this webinar, Roland Theron, Senior Solutions Engineer at Alluxio, discusses how the data access layer can help you: - Build AI architecture on your existing data lake without the need for specialized hardware. - Streamline the time-consuming process of managing data copies in data engineering. - Speed up training workloads with high GPU utilization. - Achieve optimal concurrency to deliver models to inference clusters for demanding applications

AI/ML Infra Meetup | Balancing Cost, Performance, and Scale - Running GPU/CPU...Alluxio, Inc.

AI/ML Infra Meetup | Exploring Distributed Caching for Faster GPU Training wi...Alluxio, Inc.

AI/ML Infra Meetup Nov. 7, 2024 Organized by Alluxio For more Alluxio Events: https://www.alluxio.io/events/ Speaker: - Bin Fan (Founding Engineer, VP of Technology @ Alluxio) As large-scale machine learning becomes increasingly GPU-centric, modern high-performance hardware like NVMe storage and RDMA networks (InfiniBand or specialized NICs) are becoming more widespread. To fully leverage these resources, it’s crucial to build a balanced architecture that avoids GPU underutilization. In this talk, we will explore various strategies to address this challenge by effectively utilizing these advanced hardware components. Specifically, we will present experimental results from building a Kubernetes-native distributed caching layer, utilizing NVMe storage and high-speed RDMA networks to optimize data access for PyTorch training.

AI/ML Infra Meetup | Maximizing GPU Efficiency : Optimizing Model Training wi...Alluxio, Inc.

AI/ML Infra Meetup Aug. 29, 2024 Organized by Alluxio For more Alluxio Events: https://www.alluxio.io/events/ Speaker: - Bin Fan (VP of Technology, Founding Engineer @OpenAI) In the rapidly evolving landscape of AI and machine learning, infra teams face critical challenges in managing large-scale data for AI. Performance bottlenecks, cost inefficiencies, and management complexities pose significant challenges for AI platform teams supporting large-scale model training and serving. In this talk, Bin Fan will discuss the challenges of I/O stalls that lead to suboptimal GPU utilization during model training. He will present a reference architecture for running PyTorch jobs with Alluxio in cloud environments, demonstrating how this approach can significantly enhance GPU efficiency. What you will learn: - How to identify GPU utilization and I/O-related performance bottlenecks in model training - Leverage GPU anywhere to maximize resource utilization - Best practices for monitoring and optimizing GPU usage across training and serving pipelines - Strategies for reducing cloud costs and simplifying management of AI infrastructure at scale

AI Infra Day | Accelerate Your Model Training and Serving with Distributed Ca...Alluxio, Inc.

AI Infra Day Oct. 25, 2023 Organized by Alluxio For more Alluxio Events: https://www.alluxio.io/events/ Speaker: - Adit Madan (Director of Product Management, @Alluxio) In this session, Adit Madan, Director of Product Management at Alluxio, presents an overview of using distributed caching to accelerate model training and serving. He explores the requirements of data access patterns in the ML pipeline and offers practical best practices for using distributed caching in the cloud. This session features insights from real-world examples, such as AliPay, Zhihu, and more.

How To Build Efficient ML Pipelines From The Startup Perspective (GTC Silicon...Jaeman An

Data Science und Machine Learning im Kubernetes-Ökosysteminovex GmbH

Wer professionell Datenprodukte bereitstellen möchte, benötigt Plattformen für Data Science, Machine Learning und AI, die Nachvollziehbarkeit, Reproduzierbarkeit, Flexibilität und Skalierbarkeit gewährleisten. Dazu werden skalierbare Storage- und Compute-Kapazitäten genauso benötigt wie eine Workflowsteuerung für ML-Pipelines, ein Analysefrontend oder ein Metadatenmanagement. Wir zeigen, wie auf Basis von Kubernetes eine solche Plattformen aufgebaut werden kann, die sich durch CI/CD-Pipelines flexibel konfigurieren lässt und verschiedene ML-Frameworks und Programmiersprachen unterstützt. Vorkenntnisse Grundlegendes Verständnis von Daten, Data-Science-Prozessen und Container-Plattformen ist hilfreich. Lernziele Die Zuhörer haben einen Überblick über die Anforderungen an Data-Science-Plattformen und wie eine Architektur mit Kubernetes aussehen kann. Event: data2day, 26.09.2018 Speaker: Dr. Stefan Igel, Hans-Peter Zorn Mehr Tech-Vorträge: https://www.inovex.de/de/content-pool/vortraege/ Mehr Tech-Artikel: https://www.inovex.de/blog/

Data Infra Meetup | Accelerate Distributed PyTorch/Ray Workloads in the CloudAlluxio, Inc.

Data Infra Meetup Jan. 25, 2024 Organized by Alluxio For more Alluxio Events: https://www.alluxio.io/events/ Speaker: - Siyuan Sheng (Senior Software Engineer, @Alluxio) - Chunxu Tang (Research Scientist, @Alluxio) In this session, cloud optimization specialists Chunxu and Siyuan break down the challenges and present a fresh architecture designed to optimize I/O across the data pipeline, ensuring GPUs function at peak performance. The integrated solution of PyTorch/Ray + Alluxio + S3 offers a promising way forward, and the speakers delve deep into its practical applications. Attendees will not only gain theoretical insights but will also be treated to hands-on instructions and demonstrations of deploying this cutting-edge architecture in Kubernetes, specifically tailored for Tensorflow/PyTorch/Ray workloads in the public cloud.

Productionizing Machine Learning - Bigdata meetup 5-06-2019Iulian Pintoiu

Distributed TensorFlow on Hops (Papis London, April 2018)Jim Dowling

The document discusses techniques for distributed TensorFlow on Hops. It discusses how distributed deep learning is important for improving models through increased computation and larger training datasets. It describes how Hops provides an integrated platform for machine learning pipelines that supports distributed training, hyperparameter optimization, model serving, and data processing using Spark, TensorFlow and Kubernetes. Hops addresses limitations of other platforms by providing integrated security, high performance distributed storage, and ease of use through fully managed services.

Alluxio Monthly Webinar | Simplify Data Access for AI in Multi-CloudAlluxio, Inc.

Alluxio Monthly Webinar May. 14, 2024 For more Alluxio Events: https://www.alluxio.io/events/ Speaker: - ChanChan Mao (Developer Advocate, Alluxio) - Bin Fan (VP of Technology, Alluxio) Running AI/ML workloads in different clouds present unique challenges. The key to a manageable multi-cloud architecture is the ability to seamlessly access data across environments with high performance and low cost. This webinar is designed for data platform engineers, data infra engineers, data engineers, and ML engineers who work with multiple data sources in hybrid or multi-cloud environments. Chanchan and Bin will guide the audience through using Alluxio to greatly simplify data access and make model training and serving more efficient in these environments. You will learn: - How to access data in multi-region, hybrid, and multi-cloud like accessing a local file system - How to run PyTorch to read datasets and write checkpoints to remote storage with Alluxio as the distributed data access layer - Real-world examples and insights from tech giants like Uber, AliPay and more

GPU and Deep learning best practicesLior Sidi

Despite the increase of deep learning practitioners and researchers, many of them do not use GPUs, this may lead to long training/evaluation cycles and non-practical research. In his talk, Lior shares how to get started with GPUs and some of the best practices that helped him during research and work. The talk is for everyone who works with machine learning (deep learning experience is NOT mandatory!), It covers the very basics of how GPU works, CUDA drivers, IDE configuration, training, inference, and multi-GPU training.

Deep learning for FinTechgeetachauhan

The document discusses deep learning techniques for financial technology (FinTech) applications. It begins with examples of current deep learning uses in FinTech like trading algorithms, fraud detection, and personal finance assistants. It then covers topics like specialized compute hardware for deep learning training and inference, optimization techniques for CPUs and GPUs, and distributed training approaches. Finally, it discusses emerging areas like FPGA and quantum computing and provides resources for practitioners to start with deep learning for FinTech.

BSC LMS DDL Ganesan Narayanasamy

This document discusses three key artificial intelligence capabilities of IBM's Power9 architecture: 1) Large Memory Support enables processing of high-definition images and large models that exceed GPU memory limits. 2) Distributed Deep Learning allows scaling to multiple servers for faster and more accurate training on large datasets. 3) PowerAI Vision provides tools for labeling data, training models for computer vision tasks, and deploying models for production use.

S8277 - Introducing Krylov: AI Platform that Empowers eBay Data Science and E...Henry Saputra

The Krylov Project is the key component in eBay's AI Platform initiative that provides an easy to use, open, and fast AI orchestration engine that is deployed as managed services in eBay cloud. Using Krylov, AI scientists can access eBay's massive datasets; build and train AI models; spin up powerful compute (high-memory or GPU instances) on the Krylov compute cluster; and set up machine learning pipelines, such as using declarative constructs that stitch together pipeline lifecycle.

Big Data Bellevue Meetup | Enhancing Python Data Loading in the Cloud for AI/MLAlluxio, Inc.

Big Data Bellevue Meetup March 21, 2024 For more Alluxio events: https://alluxio.io/events/ Speakers: Bin Fan (VP of Open Source, Alluxio) In this presentation, Bin Fan (VP of Open Source @ Alluxio) will address a critical challenge of optimizing data loading for distributed Python applications within AI/ML workloads in the cloud, focusing on popular frameworks like Ray and Hugging Face. Integration of Alluxio’s distributed caching for Python applications is accomplished using the fsspec interface, thus greatly improving data access speeds. This is particularly useful in machine learning workflows, where repeated data reloading across slow, unstable or congested networks can severely affect GPU efficiency and escalate operational costs. Attendees can look forward to practical, hands-on demonstrations showcasing the tangible benefits of Alluxio’s caching mechanism across various real-world scenarios. These demos will highlight the enhancements in data efficiency and overall performance of data-intensive Python applications. This presentation is tailored for developers and data scientists eager to optimize their AI/ML workloads. Discover strategies to accelerate your data processing tasks, making them not only faster but also more cost-efficient.

Alluxio Webinar | Model Training Across Regions and Clouds – Challenges, Solu...Alluxio, Inc.

Alluxio Webinar October.15, 2024 For more Alluxio Events: https://www.alluxio.io/events/ Speaker: - Tom Luckenbach (Solutions Engineering Manager, Alluxio) AI training workloads running on compute engines like PyTorch, TensorFlow, and Ray require consistent, high-throughput access to training data to maintain high GPU utilization. However, with the decoupling of compute and storage and with today’s hybrid and multi-cloud landscape, AI Platform and Data Infrastructure teams are struggling to cost-effectively deliver the high-performance data access needed for AI workloads at scale. Join Tom Luckenbach, Alluxio Solutions Engineering Manager, to learn how Alluxio enables high-speed, cost-effective data access for AI training workloads in hybrid and multi-cloud architectures, while eliminating the need to manage data copies across regions and clouds. What Tom will share: - AI data access challenges in cross-region, cross-cloud architectures. - The architecture and integration of Alluxio with frameworks like PyTorch, TensorFlow, and Ray using POSIX, REST, or Python APIs across AWS, GCP and Azure. - A live demo of an AI training workload accessing cross-cloud datasets leveraging Alluxio's distributed cache, unified namespace, and policy-driven data management. - MLPerf and FIO benchmark results and cost-savings analysis.

Accelerated Machine Learning with RAPIDS and MLflow, Nvidia/RAPIDSDatabricks

Accelerated Machine Learning with RAPIDS and MLflow, Nvidia/RAPIDS Abstract: We will introduce RAPIDS, a suite of open source libraries for GPU-accelerated data science, and illustrate how it operates seamlessly with MLflow to enable reproducible training, model storage, and deployment. We will walk through a baseline example that incorporates MLflow locally, with a simple SQLite backend, and briefly introduce how the same workflow can be deployed in the context of GPU enabled Kubernetes clusters.

TensorFlow 16: Building a Data Science Platform Seldon

1. The document discusses building a data science platform on DC/OS to operationalize machine learning models. It outlines challenges at each stage of the ML pipeline and how DC/OS addresses them with distributed computing capabilities and services for data storage, processing, model training and deployment. 2. Key stages covered include data preparation, distributed training using frameworks like TensorFlow, model management with storage of trained models, and low-latency model serving for production with TensorFlow Serving. 3. DC/OS provides a full-stack platform to operationalize ML at scale through distributed computing resources, container orchestration, and integration of open source data and ML services.

Innovation with ai at scale on the edge vt sept 2019 v0Ganesan Narayanasamy

This document provides a summary of a presentation on innovating with AI at scale. The presentation discusses: 1. Implementing AI use cases at scale across industries like retail, life sciences, and transportation. 2. Deploying AI models to the edge using tools like TensorFlow and TensorRT for high-performance inference on devices. 3. Best practices and frameworks for distributed deep learning training on large clusters to train models faster.

AI/ML Infra Meetup | Improve Speed and GPU Utilization for Model Training & S...Alluxio, Inc.

AI/ML Infra Meetup May. 23, 2024 Organized by Alluxio For more Alluxio Events: https://www.alluxio.io/events/ Speaker: - Lu Qiu (Data & AI Platform Tech Lead, @Alluxio) - Siyuan Sheng (Senior Software Engineer, @Alluxio) Speed and efficiency are two requirements for the underlying infrastructure for machine learning model development. Data access can bottleneck end-to-end machine learning pipelines as training data volume grows and when large model files are more commonly used for serving. For instance, data loading can constitute nearly 80% of the total model training time, resulting in less than 30% GPU utilization. Also, loading large model files for deployment to production can be slow because of slow network or storage read operations. These challenges are prevalent when using popular frameworks like PyTorch, Ray, or HuggingFace, paired with cloud object storage solutions like S3 or GCS, or downloading models from the HuggingFace model hub. In this presentation, Lu and Siyuan will offer comprehensive insights into improving speed and GPU utilization for model training and serving. You will learn: - The data loading challenges hindering GPU utilization - The reference architecture for running PyTorch and Ray jobs while reading data from S3, with benchmark results of training ResNet50 and BERT - Real-world examples of boosting model performance and GPU utilization through optimized data access

Alluxio Monthly Webinar | Cloud-Native Model Training on Distributed DataAlluxio, Inc.

Alluxio Monthly Webinar Apr. 23, 2024 For more Alluxio Events: https://www.alluxio.io/events/ Speaker: - ChanChan Mao (Developer Advocate, Alluxio) - Shawn Sun (Tech Lead of Cloud Native, Alluxio) Cloud-native model training jobs require fast data access to achieve shorter training cycles. Accessing data can be challenging when your datasets are distributed across different regions and clouds. Additionally, as GPUs remain scarce and expensive resources, it becomes more common to set up remote training clusters from where data resides. This multi-region/cloud scenario introduces the challenges of losing data locality, resulting in operational overhead, latency and expensive cloud costs. In the third webinar of the multi-cloud webinar series, Chanchan and Shawn dive deep into: - The data locality challenges in the multi-region/cloud ML pipeline - Using a cloud-native distributed caching system to overcome these challenges - The architecture and integration of PyTorch/Ray+Alluxio+S3 using POSIX or RESTful APIs - Live demo with ResNet and BERT benchmark results showing performance gains and cost savings analysis

Infrastructure Agnostic Machine Learning Workload DeploymentDatabricks

When it comes to Large Scale data processing and Machine Learning, Apache Spark is no doubt one of the top battle-tested frameworks out there for handling batched or streaming workloads. The ease of use, built-in Machine Learning modules, and multi-language support makes it a very attractive choice for data wonks. However bootstrapping and getting off the ground could be difficult for most teams without leveraging a Spark cluster that is already pre-provisioned and provided as a managed service in the Cloud, while this is a very attractive choice to get going, in the long run, it could be a very expensive option if it’s not well managed. As an alternative to this approach, our team has been exploring and working a lot with running Spark and all our Machine Learning workloads and pipelines as containerized Docker packages on Kubernetes. This provides an infrastructure-agnostic abstraction layer for us, and as a result, it improves our operational efficiency and reduces our overall compute cost. Most importantly, we can easily target our Spark workload deployment to run on any major Cloud or On-prem infrastructure (with Kubernetes as the common denominator) by just modifying a few configurations. In this talk, we will walk you through the process our team follows to make it easy for us to run a production deployment of our Machine Learning workloads and pipelines on Kubernetes which seamlessly allows us to port our implementation from a local Kubernetes set up on the laptop during development to either an On-prem or Cloud Kubernetes environment

Tuning For Deep Learning Inference with Intel® Processor Graphics | SIGGRAPH ...Intel® Software

This document discusses optimizing deep learning inference on Intel processor graphics using the OpenVINOTM toolkit. Some key points include: - Running inference on client devices provides advantages over cloud like privacy, bandwidth savings, and responsiveness. - OpenVINOTM provides tools to optimize models for Intel hardware and achieve 5-10x speedups on Intel GPUs compared to CPU baselines. - A case study demonstrates optimizing a deep image matting model, reducing inference time from 2.35 seconds to 291 milliseconds on Intel GPU using OpenVINOTM. - Emerging technologies like federated learning are discussed which could improve privacy for on-device inference.

OS for AI: Elastic Microservices & the Next Gen of MLNordic APIs

AI has been a hot topic lately, with advances being made constantly in what is possible, there has not been as much discussion of the infrastructure and scaling challenges that come with it. How do you support dozens of different languages and frameworks, and make them interoperate invisibly? How do you scale to run abstract code from thousands of different developers, simultaneously and elastically, while maintaining less than 15ms of overhead? At Algorithmia, we’ve built, deployed, and scaled thousands of algorithms and machine learning models, using every kind of framework (from scikit-learn to tensorflow). We’ve seen many of the challenges faced in this area, and in this talk I’ll share some insights into the problems you’re likely to face, and how to approach solving them. In brief, we’ll examine the need for, and implementations of, a complete “Operating System for AI” – a common interface for different algorithms to be used and combined, and a general architecture for serverless machine learning which is discoverable, versioned, scalable and sharable.

Automated ML Workflow for Distributed Big Data Using Analytics Zoo (CVPR2020 ...Jason Dai

This document summarizes a CVPR 2020 tutorial on the Analytics Zoo platform for automated machine learning workflows for distributed big data using Apache Spark. The tutorial covers an overview of Analytics Zoo and the BigDL distributed deep learning framework. It demonstrates distributed training of deep learning models using TensorFlow and PyTorch on Spark, and features of Analytics Zoo like end-to-end pipelines, ML workflow for automation, and model deployment with cluster serving. Real-world use cases applying Analytics Zoo at companies like SK Telecom, Midea, and MasterCard are also presented.

Azure machine learning serviceRuth Yakubu

Kostiantyn Bokhan, N-iX. CD4ML based on Azure and KubeflowIT Arena

Kostiantyn Bokhan, a technical lead at N-IX, focuses on data science projects. He leads data science projects in several areas: Computer vision, NLP, and signal processing as well as consults clients regarding digital transformations with AI. When free, he conducts research in the deep machine learning area. Kostiantyn has been an associate professor and faculty member of several universities since 2002. His research focuses on machine learning, deep learning, signal, and image processing. He received a PhD degree in network and telecommunications systems with research in digital signal processing in 2013. He has served on the scientific committees and review boards of several conferences. Speech Overview: Applying machine learning to make business applications and services intelligent is more than just training models and serving them. It requires implementing end-to-end and continuously repeatable cycles of training, testing, deploying, monitoring, and operating the models. Continuous delivery for machine learning (CD4ML) is a technique that enables reliable end-to-end cycles of development, deploying, and monitoring machine learning models. There are a lot of tools and frameworks that can be used to implement CD4ML. One of them is Kubeflow. Our experience of using Kubeflow for implementing CD4ML for the manufacturing area based on Azure Kubernetes service will be described in this speech.

How Coupang Leverages Distributed Cache to Accelerate ML Model TrainingAlluxio, Inc.

Alluxio Tech Talk Webinar Apr. 22, 2025 Organized by Alluxio For more Alluxio Events: https://www.alluxio.io/events/ Speaker: - Hyun Jung Baek (Staff Backend Engineer @ Coupang) Description Coupang is a leading e-commerce company in South Korea, with over 50,000 employees and $20+ billion in annual revenue. Coupang's AI platform team builds and manages a large-scale AI platform in AWS for machine learning engineers to train models that enhance and customize product search results and product recommendations for its 100+ million customers. As the search and recommendation models evolve, optimizing the underlying infrastructure for AI/ML workloads is essential for the e-commerce business. Coupang's platform team actively sought to improve their model training pipeline to boost machine learning engineers' productivity, publish models to production faster, and reduce operational costs. Coupang focused on addressing several key areas: - Shortening data preparation and model training time - Improving GPU utilization in training clusters in different regions - Reducing S3 API and egress costs incurred from copying large training datasets across regions - Simplifying the operational complexity of storage system management In this tech talk, Hyun Jung Baek, Staff Backend Engineer at Coupang, will share best practices for leveraging Alluxio to power search and recommendation model training infrastructure. Hyun will discuss: - How Coupang builds a world-class large-scale AI platform for machine learning engineers to deliver better search and recommendation models - How adding distributed caching to their multi-region AI infrastructure improves GPU utilization, accelerates end-to-end training time, and significantly reduces cross-region data transfer costs. - How to simplify platform operations and to easily deploy the same architecture to new GPU clusters.

Alluxio Webinar | Inside Deepseek 3FS: A Deep Dive into AI-Optimized Distribu...Alluxio, Inc.

Alluxio Webinar Apr 1, 2025 For more Alluxio Events: https://www.alluxio.io/events/ Speaker: Stephen Pu (Staff Software Engineer @ Alluxio) Deepseek’s recent announcement of the Fire-flyer File System (3FS) has sparked excitement across the AI infra community, promising a breakthrough in how machine learning models access and process data. In this webinar, an expert in distributed systems and AI infrastructure will take you inside Deepseek 3FS, the purpose-built file system for handling large files and high-bandwidth workloads. We’ll break down how 3FS optimizes data access and speeds up AI workloads as well as the design tradeoffs made to maximize throughput for AI workloads. This webinar you’ll learn about how 3FS works under the hood, including: ✅ The system architecture ✅ Core software components ✅ Read/write flows ✅ Data distribution/placement algorithms ✅ Cluster/node management and disaster recovery Whether you’re an AI researcher, ML engineer, or infrastructure architect, this deep dive will give you the technical insights you need to determine if 3FS is the right solution for you.

More Related Content

Similar to Alluxio Webinar - Maximize GPU Utilization for Model Training (20)

Productionizing Machine Learning - Bigdata meetup 5-06-2019Iulian Pintoiu

Distributed TensorFlow on Hops (Papis London, April 2018)Jim Dowling

Alluxio Monthly Webinar | Simplify Data Access for AI in Multi-CloudAlluxio, Inc.

GPU and Deep learning best practicesLior Sidi

Deep learning for FinTechgeetachauhan

BSC LMS DDL Ganesan Narayanasamy

S8277 - Introducing Krylov: AI Platform that Empowers eBay Data Science and E...Henry Saputra

Big Data Bellevue Meetup | Enhancing Python Data Loading in the Cloud for AI/MLAlluxio, Inc.

Alluxio Webinar | Model Training Across Regions and Clouds – Challenges, Solu...Alluxio, Inc.

Accelerated Machine Learning with RAPIDS and MLflow, Nvidia/RAPIDSDatabricks

TensorFlow 16: Building a Data Science Platform Seldon

Innovation with ai at scale on the edge vt sept 2019 v0Ganesan Narayanasamy

AI/ML Infra Meetup | Improve Speed and GPU Utilization for Model Training & S...Alluxio, Inc.

Alluxio Monthly Webinar | Cloud-Native Model Training on Distributed DataAlluxio, Inc.

Infrastructure Agnostic Machine Learning Workload DeploymentDatabricks

Tuning For Deep Learning Inference with Intel® Processor Graphics | SIGGRAPH ...Intel® Software

OS for AI: Elastic Microservices & the Next Gen of MLNordic APIs

Automated ML Workflow for Distributed Big Data Using Analytics Zoo (CVPR2020 ...Jason Dai

Azure machine learning serviceRuth Yakubu

Kostiantyn Bokhan, N-iX. CD4ML based on Azure and KubeflowIT Arena

Productionizing Machine Learning - Bigdata meetup 5-06-2019Iulian Pintoiu

Distributed TensorFlow on Hops (Papis London, April 2018)Jim Dowling

Alluxio Monthly Webinar | Simplify Data Access for AI in Multi-CloudAlluxio, Inc.

GPU and Deep learning best practicesLior Sidi

Deep learning for FinTechgeetachauhan

BSC LMS DDL Ganesan Narayanasamy

S8277 - Introducing Krylov: AI Platform that Empowers eBay Data Science and E...Henry Saputra

Big Data Bellevue Meetup | Enhancing Python Data Loading in the Cloud for AI/MLAlluxio, Inc.

Alluxio Webinar | Model Training Across Regions and Clouds – Challenges, Solu...Alluxio, Inc.

Accelerated Machine Learning with RAPIDS and MLflow, Nvidia/RAPIDSDatabricks

TensorFlow 16: Building a Data Science Platform Seldon

Innovation with ai at scale on the edge vt sept 2019 v0Ganesan Narayanasamy

AI/ML Infra Meetup | Improve Speed and GPU Utilization for Model Training & S...Alluxio, Inc.

Alluxio Monthly Webinar | Cloud-Native Model Training on Distributed DataAlluxio, Inc.

Infrastructure Agnostic Machine Learning Workload DeploymentDatabricks

Tuning For Deep Learning Inference with Intel® Processor Graphics | SIGGRAPH ...Intel® Software

OS for AI: Elastic Microservices & the Next Gen of MLNordic APIs

Automated ML Workflow for Distributed Big Data Using Analytics Zoo (CVPR2020 ...Jason Dai

Azure machine learning serviceRuth Yakubu

Kostiantyn Bokhan, N-iX. CD4ML based on Azure and KubeflowIT Arena

More from Alluxio, Inc. (20)

How Coupang Leverages Distributed Cache to Accelerate ML Model TrainingAlluxio, Inc.

Alluxio Webinar | Inside Deepseek 3FS: A Deep Dive into AI-Optimized Distribu...Alluxio, Inc.

AI/ML Infra Meetup | Building Production Platform for Large-Scale Recommendat...Alluxio, Inc.

AI/ML Infra Meetup | How Uber Optimizes LLM Training and FinetuneAlluxio, Inc.

AI/ML Infra Meetup | Optimizing ML Data Access with Alluxio: Preprocessing, ...Alluxio, Inc.

AI/ML Infra Meetup Mar. 06, 2025 Organized by Alluxio For more Alluxio Events: https://www.alluxio.io/events/ Speaker: - Bin Fan (VP of Technology @ Alluxio) In this talk, Bin Fan shares his insights on data access challenges in ML applications, with particular emphasis on how Alluxio's distributed caching helps bridge the gap between storage and compute in preprocessing, pretraining and inference.

AI/ML Infra Meetup | Deployment, Discovery and Serving of LLMs at Uber ScaleAlluxio, Inc.

Alluxio Webinar | What’s New in Alluxio AI: 3X Faster Checkpoint File Creatio...Alluxio, Inc.

Alluxio Webinar Feb. 25, 2025 For more Alluxio Events: https://www.alluxio.io/events/ Speaker: Bill Hodak (VP of Marketing and Product Marketing, Alluxio) Tom Luckenbach (Solutions Engineering Manager, Alluxio) Join us to learn about the latest release of Alluxio Enterprise AI. In this webinar, we’ll provide an overviewof the new features and capabilities of Alluxio Enterprise AI, built to accelerate AI workloads and maximize GPU utilization. Key highlights include: - New caching mode accelerates AI checkpoints - Advanced cache eviction policies provide fine-grained control - Python SDK integrations enhance AI framework compatibility - A demo of Alluxio accelerating AI training workloads in AWS

AI/ML Infra Meetup | A Faster and More Cost Efficient LLM Inference StackAlluxio, Inc.

AI/ML Infra Meetup Jan. 23, 2025 Organized by Alluxio For more Alluxio Events: https://www.alluxio.io/events/ Speaker: - Junchen Jiang (Assistant Professor @ University of Chicago) LLM inference can be huge, particularly, with long contexts. In this on-demand video, Junchen Jiang, Assistant Professor at University of Chicago, presents a 10x solution for long contexts inference: an easy-to-deploy stack over multiple vLLM engines with tailored KV-cache backend.

AI/ML Infra Meetup | RAYvolution - The Last Mile: Mastering AI Deployment wit...Alluxio, Inc.

Alluxio Webinar | Accelerate AI: Alluxio 101Alluxio, Inc.

Alluxio Webinar Dec. 3, 2024 For more Alluxio Events: https://www.alluxio.io/events/ Speaker: Bill Hodak (VP of Marketing and Product Marketing, Alluxio) In the rapidly evolving landscape of AI and machine learning, Platform and Data Infrastructure Teams face critical challenges in building and managing large-scale AI platforms. Performance bottlenecks, scalability of the platform, and scarcity of GPUs pose significant challenges in supporting large-scale model training and serving. In this talk, we will introduce how Alluxio helps Platform and Data Infrastructure teams deliver faster, more scalable platforms to ML Engineering teams developing and training AI models. Alluxio’s highly-distributed cache accelerates AI workloads by eliminating data loading bottlenecks and maximizing GPU utilization. Customers report up to 4x faster training performance with high-speed access to petabytes of data spread across billions of files regardless of persistent storage type or proximity to GPU clusters. Alluxio’s architecture lowers data infrastructure costs, increases GPU utilization, and enables workload portability for navigating GPU scarcity challenges.

AI/ML Infra Meetup | The power of Ray in the era of LLM and multi-modality AIAlluxio, Inc.

AI/ML Infra Meetup Nov. 7, 2024 Organized by Alluxio For more Alluxio Events: https://www.alluxio.io/events/ Speaker: - Zhe Zhang (Distinguished Engineer @ NVIDIA) In this talk, Zhe Zhang (NVIDIA, ex-Anyscale) introduced Ray and its applications in the LLM and multi-modal AI era. He shared his perspective on ML infrastructure, noting that it presents more unstructured challenges, and recommended using Ray and Alluxio as solutions for increasingly data-intensive multi-modal AI workloads.

AI/ML Infra Meetup | Big Data and AI, Zoom DevelopersAlluxio, Inc.

AI/ML Infra Meetup Nov. 7, 2024 Organized by Alluxio For more Alluxio Events: https://www.alluxio.io/events/ Speaker: - Sandeep Manchem (ML Platform Engineering Manager @ Zoom) In this talk, Sandeep Manchem (Zoom) discussed big data and AI, covering typical platform architecture and data challenges. We had engaging discussions about ensuring data safety and compliance in Big Data and AI applications.

AI/ML Infra Meetup | TorchTitan, One-stop PyTorch native solution for product...Alluxio, Inc.

AI/ML Infra Meetup Nov. 7, 2024 Organized by Alluxio For more Alluxio Events: https://www.alluxio.io/events/ Speaker: - Tianyu Liu (Research Scientist @ Meta) TorchTitan is a proof-of-concept for Large-scale LLM training using native PyTorch. It is a repo that showcases PyTorch's latest distributed training features in a clean, minimal codebase. In this talk, Tianyu will share TorchTitan’s design and optimizations for the Llama 3.1 family of LLMs, spanning 8 billion to 405 billion parameters, and showcase its performance, composability, and scalability.

AI/ML Infra Meetup | Scaling Experimentation Platform in Digital Marketplaces...Alluxio, Inc.

AI/ML Infra Meetup Aug. 29, 2024 Organized by Alluxio For more Alluxio Events: https://www.alluxio.io/events/ Speaker: - Koundinya Pidaparthi (VP of Analytics @ Poshmark) Scaling experimentation in digital marketplaces is crucial for driving growth and enhancing user experiences. However, varied methodologies and a lack of experiment governance can hinder the impact of experimentation leading to inconsistent decision-making, inefficiencies, and missed opportunities for innovation. At Poshmark, we developed a homegrown experimentation platform, Lightspeed, that allowed us to make reliable and confident reads on product changes, which led to a 10x growth in experiment velocity and positive business outcomes along the way. This session will provide a deep dive into the best practices and lessons learned from successful implementations of large-scale experiments. We will explore the importance of experimentation, overcome scalability challenges, and gain insights into the frameworks and technologies that enable effective testing.

AI/ML Infra Meetup | Scaling Vector Databases for E-Commerce Visual Search: A...Alluxio, Inc.

AI/ML Infra Meetup Aug. 29, 2024 Organized by Alluxio For more Alluxio Events: https://www.alluxio.io/events/ Speaker: - Mahesh Pasupuleti (VP of DS, ML & Data Infra @ Poshmark) In the rapidly evolving world of e-commerce, visual search has become a game-changing technology. Poshmark, a leading fashion resale marketplace, has developed Posh Lens – an advanced visual search engine that revolutionizes how shoppers discover and purchase items. Under the hood of Posh Lens lies Milvus, a vector database enabling efficient product search and recommendation across our vast catalog of over 150 million items. However, with such an extensive and growing dataset, maintaining high-performance search capabilities while scaling AI infrastructure presents significant challenges. In this talk, Mahesh Pasupuleti shares: - The architecture and strategies to scale Milvus effectively within the Posh Lens infrastructure - Key considerations include optimizing vector indexing, managing data partitioning, and ensuring query efficiency amidst large-scale data growth - Distributed computing principles and advanced indexing techniques to handle the complexity of Poshmark's diverse product catalog

Alluxio Webinar | Optimize, Don't Overspend: Data Caching Strategy for AI Wor...Alluxio, Inc.

Alluxio Webinar Sept. 10, 2024 For more Alluxio Events: https://www.alluxio.io/events/ Speaker: - Jingwen Ouyang (Senior Program Manager, Alluxio) As machine learning and deep learning models grow in complexity, AI platform engineers and ML engineers face significant challenges with slow data loading and GPU utilization, often leading to costly investments in high-performance computing (HPC) storage. However, this approach can result in overspending without addressing the core issues of data bottlenecks and infrastructure complexity. A better approach is adding a data caching layer between compute and storage, like Alluxio, which offers a cost-effective alternative through its innovative data caching strategy. In this webinar, Jingwen will explore how Alluxio's caching solutions optimize AI workloads for performance, user experience and cost-effectiveness. What you will learn: - The I/O bottlenecks that slow down data loading in model training - How Alluxio's data caching strategy optimizes I/O performance for training and GPU utilization, and significantly reduces cloud API costs - The architecture and key capabilities of Alluxio - Using Rapid Alluxio Deployer to install Alluxio and run benchmarks in AWS in just 30 minutes

AI/ML Infra Meetup | Preference Tuning and Fine Tuning LLMsAlluxio, Inc.

Alluxio Webinar | What’s new in Alluxio Enterprise AI 3.2: Leverage GPU Anywh...Alluxio, Inc.

Alluxio Webinar July.23, 2024 For more Alluxio Events: https://www.alluxio.io/events/ Speaker: - Shouwei Chen (core maintainer and product manager, Alluxio) In today's AI-driven world, organizations face unprecedented demands for powerful AI infrastructure to fuel their model training and serving workloads. Performance bottlenecks, cost inefficiencies, and management complexities pose significant challenges for AI platform teams supporting large-scale model training and serving. On July 9, 2024, we introduced Alluxio Enterprise AI 3.2, a groundbreaking solution designed to address these critical issues in the ever-evolving AI landscape. In this webinar, Shouwei Chen will introduce exciting new features of Alluxio Enterprise AI 3.2: - Leveraging GPU resources anywhere accessing remote data with the same local performance - Enhanced I/O performance with 97%+ GPU utilization for popular language model training benchmarks - Achieving the same performance as HPC storage on existing data lake without additional HPC storage infrastructure - New Python FileSystem API to seamlessly integrate with Python applications like Ray - Other new features, include advanced cache management, rolling upgrades, and CSI failover

Alluxio Webinar | 10x Faster Trino Queries on Your Data PlatformAlluxio, Inc.

Alluxio Webinar June. 18, 2024 For more Alluxio Events: https://www.alluxio.io/events/ Speaker: - Jianjian Xie (Staff Software Engineer, Alluxio) As Trino users increasingly rely on cloud object storage for retrieving data, speed and cloud cost have become major challenges. The separation of compute and storage creates latency challenges when querying datasets; scanning data between storage and compute tiers becomes I/O bound. On the other hand, cloud API costs related to GET/LIST operations and cross-region data transfer add up quickly. The newly introduced Trino file system cache by Alluxio aims to overcome the above challenges. In this session, Jianjian will dive into Trino data caching strategies, the latest test results, and discuss the multi-level caching architecture. This architecture makes Trino 10x faster for data lakes of any scale, from GB to EB. What you will learn: - Challenges relating to the speed and costs of running Trino in the cloud - The new Trino file system cache feature overview, including the latest development status and test results - A multi-level cache framework for maximized speed, including Trino file system cache and Alluxio distributed cache - Real-world cases, including a large online payment firm and a top ridesharing company - The future roadmap of Trino file system cache and Trino-Alluxio integration

AI/ML Infra Meetup | ML explainability in MichelangeloAlluxio, Inc.

AI/ML Infra Meetup May. 23, 2024 Organized by Alluxio For more Alluxio Events: https://www.alluxio.io/events/ Speaker: - Eric Wang (Software Engineer, @Uber) Uber has numerous deep learning models, most of which are highly complex with many layers and a vast number of features. Understanding how these models work is challenging and demands significant resources to experiment with various training algorithms and feature sets. With ML explainability, the ML team aims to bring transparency to these models, helping to clarify their predictions and behavior. This transparency also assists the operations and legal teams in explaining the reasons behind specific prediction outcomes. In this talk, Eric Wang will discuss the methods Uber used for explaining deep learning models and how we integrated these methods into the Uber AI Michelangelo ecosystem to support offline explaining.