AI/ML Infra Meetup | Perspective on Deep Learning Framework

May 24, 20240 likes219 views

AI/ML Infra Meetup May. 23, 2024 Organized by Alluxio For more Alluxio Events: https://www.alluxio.io/events/ Speaker: - Triston Cao (Senior Deep Learning Software Engineering Manager, @NVIDIA) From Caffe to MXNet, to PyTorch, and more, Xiande Cao, Senior Deep Learning Software Engineer Manager, will share his perspective on the evolution of deep learning frameworks.

Triston Cao, for Alluxio Meetup on May 23, 2024
PERSPECTIVE ON DEEP LEARNING FRAMEWORK

3
COMPUTATION GRAPH AND GRADIENT DECENT
Image credit to Deniz Yuret's Homepage: Alec Radford's animations for optimization
algorithms

4
OPEN-SOURCE FRAMEWORKS
2014 2017 2020
2016 2019
2015 2018 2024
ChatGPT
AlexNet ResNet Transformer

5
WHAT DOES A FRAMEWORK LOOK LIKE
A Hybrid Programming Language Environment

7
OPS, TENSORS, AND PARALLEL EXECUTION
System Level Optimization
https://mxnet.apache.org/versions/1.9.1/api/architecture/note_engine
https://www.oreilly.com/library/view/elegant-scipy/9781491922927/ch01.html

9
CONVOLUTIONS
https://paperswithcode.com/methods/category/convolutional-neural-networks https://cv.gluon.ai/contents.html
https://epynn.net/Convolution.html

10
CUDNN 10TH
ANNIVERSARY
April 2014 – April 2024

11
CUDA
TensorRT NCCL DALI
cuDNN
cuBLAS
…
Deep Learning Frameworks
CPU Libraries

12
SYMBOLIC VS EAGER (IMPERATIVE)
Performance vs Easy of use
Imperative
Static graph
Eager mode JIT
Hybrid

15
DATA LAYOUT MATTERS TOO
Reference: Convolutional Layers User's Guide - NVIDIA Docs

17
NCCL FOR MULTI-NODE TRAINING
https://docs.nvidia.com/deeplearning/nccl/user-guide/docs/usage/operations.html

18
https://www.nvidia.com/en-us/data-center/resources/mlperf-benchmarks/

21
INFERENCE WITH INT8
Ref: Achieving FP32 Accuracy for INT8 Inference Using Quantization Aware Training with NVIDIA TensorRT | NVIDIA Technical Blog

22
COMPILER BASED FRAMEWORK
https://tvm.apache.org/docs/tutorial/relay_quick_start.html
https://www.linkedin.com/pulse/exploring-jax-googles-high-performance-py
thon-library-nagilla-hwauc/
Thunder can optimize Pytorch module with
• torch.compile
• nvFuser
• cuDNN
• Apex
• TransformerEngine
• PyTorch eager
• Custom CUDA kernels through PyCUDA,
Numba, CuPy
• Custom kernels written in OpenAI Triton
https://github.com/Lightning-AI/lightning-thun
der

23
TAKE AWAYS
• Deep learning frameworks are large software projects
• NVIDIA keeps making libraries to server deep learning frameworks for GPU acceleration
• Training and inference have different challenges
• More stabilized by still fast evolving
• Compiler technology getting more integrated into the framework

AI/ML Infra Meetup | Perspective on Deep Learning Framework

This document provides a monthly highlights summary of OpenACC: - OpenACC is a programming model for parallel computing on CPUs and GPUs using compiler directives to add parallelism to existing serial code. - OpenACC is seeing wide adoption across major HPC applications and allows performance portability between CPU and GPU. - The document highlights recent optimizations, events, publications and resources around OpenACC programming.

Harnessing AI for the Benefit of All.Alison B. Lowndes

This document provides an overview of AI and GPU technologies from NVIDIA. It discusses NVIDIA's GPU computing platforms like DGX, Jetson, and AGX which are used for AI training and inference. It also summarizes NVIDIA's tools and frameworks like CUDA, TensorRT, and DeepStream which help accelerate AI workflows. Finally, it promotes NVIDIA's training resources like the Deep Learning Institute to help developers get started with AI.

OpenACC and Open Hackathons Monthly Highlights: July 2022.pptxOpenACC

OpenACC and Open Hackathons Monthly Highlights: September 2022.pptxOpenACC

OpenACC Monthly Highlights- DecemberNVIDIA

OpenACC Monthly Highlights: January 2024OpenACC

[HashiTalk Korea] OCP with Super Tengen Toppahyeongchae lee

The document discusses using Orchestrator, Consul, and ProxySQL for database orchestration. It provides an overview of how WiX Engineering uses Orchestrator to orchestrate ProxySQL with Open Containers Platform (OCP). It also discusses Github's use of Orchestrator for MySQL replication topology management and incident analysis. HashiCorp's Consul is presented as an alternative for service discovery and integration with tools like Orchestrator and ProxySQL. The document demonstrates how ProxySQL can be used with Consul and monitored using PMM. It concludes with a summary of using this "parasite architecture" approach with additional tools like Envoy, Gloo, Vault, and dnsmasq.

OpenACC Monthly Highlights Summer 2019OpenACC

OpenACC and Hackathons Monthly HighlightsOpenACC

OpenACC and Open Hackathons Monthly Highlights: Summer 2024OpenACC

OpenACC Monthly Highlights: May 2020OpenACC

Testbed for Heterogeneous CloudCloudLightning

OpenACC Monthly Highlights: January 2021OpenACC

Containers for sensor web services, applications and research @ Sensor Web Co...Daniel Nüst

Brain in the Cloud: Machine Learning on OpenStack & Kubernetes Done Right - E...Cloud Native Day Tel Aviv

Machine Learning is no doubt the hottest trend in IT nowadays. Deep Neural Network (DNN), a subfield of Machine Learning with mode of operation loosely inspired by the brain, allows us to solve complex problems such as image recognition that has been very difficult to solve using standard programming paradigms. DNN concepts are not new. However, and until recently, applying them in practice could not be realized due to their high computational demands. With the recent development in parallel computing, especially around GPU acceleration and high speed and efficient networking, DNN has become a reality in modern data centers. In this talk we will describe the system requirements to effectively run a machine learning cluster with popular frameworks such as TensorFlow. We will discuss how such a system can be deployed in an OpenStack-based cloud without compromises, enjoying high-performance DNN programming paradigm as well as the benefits of cloud and software-defined data centers.

Running Emerging AI Applications on Big Data Platforms with Ray On Apache SparkDatabricks

With the rapid evolution of AI in recent years, we need to embrace advanced and emerging AI technologies to gain insights and make decisions based on massive amounts of data. Ray (https://github.com/ray-project/ray) is a fast and simple framework open-sourced by UC Berkeley RISELab particularly designed for easily building advanced AI applications in a distributed fashion.

CloudComp 2015 - SDN-Cloud Testbed with Hyper-convergent SmartX BoxesGIST (Gwangju Institute of Science and Technology)

OpenACC Monthly Highlights September 2020OpenACC

OpenACC Monthly Highlights: November 2020OpenACC

How APIs are Transforming Cisco Solutions and Catalyzing an Innovation EcosystemCisco DevNet

This document discusses how APIs are transforming Cisco solutions and catalyzing an innovation ecosystem. It outlines Cisco's DevNet strategy of making the developer the customer and accelerating market opportunities through a vibrant developer ecosystem built on programmable platforms and APIs. It describes how network programmability, APIs, cloudification, new applications and experiences, developer tools, and open source collaboration are driving network innovation and helping developers build solutions.

Dynamic Resource Allocation Algorithm using ContainersIRJET Journal

1) The document proposes a dynamic resource allocation algorithm using containers to optimize resource utilization in server farms. 2) It uses Docker to deploy applications in lightweight containers instead of virtual machines to reduce overhead. A node selection algorithm uses fuzzy logic to determine the most suitable node for container deployment based on resource availability and workload. 3) The proposed approach is tested on a small cluster using Docker, Hadoop and the node selection algorithm to process queries. Results show increased processing speed and better resource utilization compared to traditional virtualization methods.

Model and pilot all cloud layers with OCCIware - Eclipse Day Lyon 2017Marc Dutoo

This document introduces OCCIware, which allows modeling and piloting all cloud layers from IoT to Big Data using the OCCI standard. It provides an overview of OCCIware, demonstrates its use in a smart city use case monitoring energy consumption from IoT sensors to linked open data analytics, and shows a quick demo of Docker Studio and a custom linked data extension. It concludes by discussing next steps for OCCIware and Eclipse.org.

OCCIware presentation at EclipseDay in Lyon, November 2017, by Marc Dutoo, SmileOCCIware

Presentation title: Model and pilot all cloud layers with OCCIware, from IoT to Big Data Abstract: Who uses multi cloud today ? Everybody. Alas, this leads to a lot of "technical glue". Enter OCCIware's Studio and Runtime : manage all layers and domains of the Cloud (XaaS) in a uniform, standard, extensible way - the Cloud consumer platform.presentation. This talk presents how the OCCIware Studio - currently being contributed to the Eclipse Foundation by Inria and Obeo - takes advantage of Eclipse Modeling and SIrius in order to support a metamodel for the generic Open Cloud Computing Interface (OCCI) REST API and build a "studio factory", while providing feedback and lessons learned on various other Eclipse components. It concludes on a live demonstration of using it to model and pilot an IoT (nodeMCU/ESP8266), Linked & Big Data (JSON-LD, Spark), containerized Cloud solution to let electricity consumption be monitored across territories by all actors - individuals, utility providers, up to regional public bodies.

OpenACC Monthly Highlights: July 2020OpenACC

The Big Cloud Native FaaS LebowskiQAware GmbH

Devoxx Poland 2019, Kraków: Talk by Mario-Leander Reimer (@LeanderReimer, Principal Software Architect at QAware) === Please download slides if blurred! === Abstract: Only a few years ago the move towards microservice architecture was the first big disruption in software engineering: instead of running monoliths, systems were now build, composed and run as autonomous services. But this came at the price of added development and infrastructure complexity. Serverless and FaaS seem to be the next disruption, they are the logical evolution trying to address some of the inherent technology complexity we are currently faced when building cloud native apps. FaaS frameworks are currently popping up like mushrooms: Knative, Kubeless, OpenFn, Fission, OpenFaas or Open Whisk are just a few to name. But which one of these is safe to pick and use in your next project? Let's find out. This session will start off by briefly explaining the essence of Serverless application architecture. Leander will then define a criteria catalog for FaaS frameworks and continue by comparing and showcasing the most promising ones.

OpenACC and Open Hackathons Monthly Highlights: October 2024OpenACC

Azure HDInsightAshish Thapliyal

The document discusses HDInsight and provides information on: 1. HDInsight can scale horizontally by adding more nodes to the HDFS cluster. 2. HDInsight clusters on Azure can be used to ingest, transform, and analyze large amounts of data stored in Azure Blob Storage or Azure Data Lake Store. 3. HDInsight supports various query engines like Spark, Hive, and Hadoop for interactive querying and analytics on large datasets.

OpenACC Monthly Highlights: September 2021OpenACC

How Coupang Leverages Distributed Cache to Accelerate ML Model TrainingAlluxio, Inc.

Alluxio Tech Talk Webinar Apr. 22, 2025 Organized by Alluxio For more Alluxio Events: https://www.alluxio.io/events/ Speaker: - Hyun Jung Baek (Staff Backend Engineer @ Coupang) Description Coupang is a leading e-commerce company in South Korea, with over 50,000 employees and $20+ billion in annual revenue. Coupang's AI platform team builds and manages a large-scale AI platform in AWS for machine learning engineers to train models that enhance and customize product search results and product recommendations for its 100+ million customers. As the search and recommendation models evolve, optimizing the underlying infrastructure for AI/ML workloads is essential for the e-commerce business. Coupang's platform team actively sought to improve their model training pipeline to boost machine learning engineers' productivity, publish models to production faster, and reduce operational costs. Coupang focused on addressing several key areas: - Shortening data preparation and model training time - Improving GPU utilization in training clusters in different regions - Reducing S3 API and egress costs incurred from copying large training datasets across regions - Simplifying the operational complexity of storage system management In this tech talk, Hyun Jung Baek, Staff Backend Engineer at Coupang, will share best practices for leveraging Alluxio to power search and recommendation model training infrastructure. Hyun will discuss: - How Coupang builds a world-class large-scale AI platform for machine learning engineers to deliver better search and recommendation models - How adding distributed caching to their multi-region AI infrastructure improves GPU utilization, accelerates end-to-end training time, and significantly reduces cross-region data transfer costs. - How to simplify platform operations and to easily deploy the same architecture to new GPU clusters.

Alluxio Webinar | Inside Deepseek 3FS: A Deep Dive into AI-Optimized Distribu...Alluxio, Inc.

Alluxio Webinar Apr 1, 2025 For more Alluxio Events: https://www.alluxio.io/events/ Speaker: Stephen Pu (Staff Software Engineer @ Alluxio) Deepseek’s recent announcement of the Fire-flyer File System (3FS) has sparked excitement across the AI infra community, promising a breakthrough in how machine learning models access and process data. In this webinar, an expert in distributed systems and AI infrastructure will take you inside Deepseek 3FS, the purpose-built file system for handling large files and high-bandwidth workloads. We’ll break down how 3FS optimizes data access and speeds up AI workloads as well as the design tradeoffs made to maximize throughput for AI workloads. This webinar you’ll learn about how 3FS works under the hood, including: ✅ The system architecture ✅ Core software components ✅ Read/write flows ✅ Data distribution/placement algorithms ✅ Cluster/node management and disaster recovery Whether you’re an AI researcher, ML engineer, or infrastructure architect, this deep dive will give you the technical insights you need to determine if 3FS is the right solution for you.

More Related Content

Similar to AI/ML Infra Meetup | Perspective on Deep Learning Framework (20)

OpenACC and Hackathons Monthly HighlightsOpenACC

OpenACC and Open Hackathons Monthly Highlights: Summer 2024OpenACC

OpenACC Monthly Highlights: May 2020OpenACC

Testbed for Heterogeneous CloudCloudLightning

OpenACC Monthly Highlights: January 2021OpenACC

Containers for sensor web services, applications and research @ Sensor Web Co...Daniel Nüst

Brain in the Cloud: Machine Learning on OpenStack & Kubernetes Done Right - E...Cloud Native Day Tel Aviv

Running Emerging AI Applications on Big Data Platforms with Ray On Apache SparkDatabricks

CloudComp 2015 - SDN-Cloud Testbed with Hyper-convergent SmartX BoxesGIST (Gwangju Institute of Science and Technology)

OpenACC Monthly Highlights September 2020OpenACC

OpenACC Monthly Highlights: November 2020OpenACC

How APIs are Transforming Cisco Solutions and Catalyzing an Innovation EcosystemCisco DevNet

Dynamic Resource Allocation Algorithm using ContainersIRJET Journal

Model and pilot all cloud layers with OCCIware - Eclipse Day Lyon 2017Marc Dutoo

OCCIware presentation at EclipseDay in Lyon, November 2017, by Marc Dutoo, SmileOCCIware

OpenACC Monthly Highlights: July 2020OpenACC

The Big Cloud Native FaaS LebowskiQAware GmbH

OpenACC and Open Hackathons Monthly Highlights: October 2024OpenACC

Azure HDInsightAshish Thapliyal

OpenACC Monthly Highlights: September 2021OpenACC

OpenACC and Hackathons Monthly HighlightsOpenACC

OpenACC and Open Hackathons Monthly Highlights: Summer 2024OpenACC

OpenACC Monthly Highlights: May 2020OpenACC

Testbed for Heterogeneous CloudCloudLightning

OpenACC Monthly Highlights: January 2021OpenACC

Containers for sensor web services, applications and research @ Sensor Web Co...Daniel Nüst

Brain in the Cloud: Machine Learning on OpenStack & Kubernetes Done Right - E...Cloud Native Day Tel Aviv

Running Emerging AI Applications on Big Data Platforms with Ray On Apache SparkDatabricks

CloudComp 2015 - SDN-Cloud Testbed with Hyper-convergent SmartX BoxesGIST (Gwangju Institute of Science and Technology)

OpenACC Monthly Highlights September 2020OpenACC

OpenACC Monthly Highlights: November 2020OpenACC

How APIs are Transforming Cisco Solutions and Catalyzing an Innovation EcosystemCisco DevNet

Dynamic Resource Allocation Algorithm using ContainersIRJET Journal

Model and pilot all cloud layers with OCCIware - Eclipse Day Lyon 2017Marc Dutoo

OCCIware presentation at EclipseDay in Lyon, November 2017, by Marc Dutoo, SmileOCCIware

OpenACC Monthly Highlights: July 2020OpenACC

The Big Cloud Native FaaS LebowskiQAware GmbH

OpenACC and Open Hackathons Monthly Highlights: October 2024OpenACC

Azure HDInsightAshish Thapliyal

OpenACC Monthly Highlights: September 2021OpenACC

More from Alluxio, Inc. (20)

How Coupang Leverages Distributed Cache to Accelerate ML Model TrainingAlluxio, Inc.

Alluxio Webinar | Inside Deepseek 3FS: A Deep Dive into AI-Optimized Distribu...Alluxio, Inc.

AI/ML Infra Meetup | Building Production Platform for Large-Scale Recommendat...Alluxio, Inc.

AI/ML Infra Meetup | How Uber Optimizes LLM Training and FinetuneAlluxio, Inc.

AI/ML Infra Meetup | Optimizing ML Data Access with Alluxio: Preprocessing, ...Alluxio, Inc.

AI/ML Infra Meetup Mar. 06, 2025 Organized by Alluxio For more Alluxio Events: https://www.alluxio.io/events/ Speaker: - Bin Fan (VP of Technology @ Alluxio) In this talk, Bin Fan shares his insights on data access challenges in ML applications, with particular emphasis on how Alluxio's distributed caching helps bridge the gap between storage and compute in preprocessing, pretraining and inference.

AI/ML Infra Meetup | Deployment, Discovery and Serving of LLMs at Uber ScaleAlluxio, Inc.

Alluxio Webinar | What’s New in Alluxio AI: 3X Faster Checkpoint File Creatio...Alluxio, Inc.

Alluxio Webinar Feb. 25, 2025 For more Alluxio Events: https://www.alluxio.io/events/ Speaker: Bill Hodak (VP of Marketing and Product Marketing, Alluxio) Tom Luckenbach (Solutions Engineering Manager, Alluxio) Join us to learn about the latest release of Alluxio Enterprise AI. In this webinar, we’ll provide an overviewof the new features and capabilities of Alluxio Enterprise AI, built to accelerate AI workloads and maximize GPU utilization. Key highlights include: - New caching mode accelerates AI checkpoints - Advanced cache eviction policies provide fine-grained control - Python SDK integrations enhance AI framework compatibility - A demo of Alluxio accelerating AI training workloads in AWS

AI/ML Infra Meetup | A Faster and More Cost Efficient LLM Inference StackAlluxio, Inc.

AI/ML Infra Meetup Jan. 23, 2025 Organized by Alluxio For more Alluxio Events: https://www.alluxio.io/events/ Speaker: - Junchen Jiang (Assistant Professor @ University of Chicago) LLM inference can be huge, particularly, with long contexts. In this on-demand video, Junchen Jiang, Assistant Professor at University of Chicago, presents a 10x solution for long contexts inference: an easy-to-deploy stack over multiple vLLM engines with tailored KV-cache backend.

AI/ML Infra Meetup | Balancing Cost, Performance, and Scale - Running GPU/CPU...Alluxio, Inc.

AI/ML Infra Meetup | RAYvolution - The Last Mile: Mastering AI Deployment wit...Alluxio, Inc.

Alluxio Webinar | Accelerate AI: Alluxio 101Alluxio, Inc.

Alluxio Webinar Dec. 3, 2024 For more Alluxio Events: https://www.alluxio.io/events/ Speaker: Bill Hodak (VP of Marketing and Product Marketing, Alluxio) In the rapidly evolving landscape of AI and machine learning, Platform and Data Infrastructure Teams face critical challenges in building and managing large-scale AI platforms. Performance bottlenecks, scalability of the platform, and scarcity of GPUs pose significant challenges in supporting large-scale model training and serving. In this talk, we will introduce how Alluxio helps Platform and Data Infrastructure teams deliver faster, more scalable platforms to ML Engineering teams developing and training AI models. Alluxio’s highly-distributed cache accelerates AI workloads by eliminating data loading bottlenecks and maximizing GPU utilization. Customers report up to 4x faster training performance with high-speed access to petabytes of data spread across billions of files regardless of persistent storage type or proximity to GPU clusters. Alluxio’s architecture lowers data infrastructure costs, increases GPU utilization, and enables workload portability for navigating GPU scarcity challenges.

AI/ML Infra Meetup | The power of Ray in the era of LLM and multi-modality AIAlluxio, Inc.

AI/ML Infra Meetup Nov. 7, 2024 Organized by Alluxio For more Alluxio Events: https://www.alluxio.io/events/ Speaker: - Zhe Zhang (Distinguished Engineer @ NVIDIA) In this talk, Zhe Zhang (NVIDIA, ex-Anyscale) introduced Ray and its applications in the LLM and multi-modal AI era. He shared his perspective on ML infrastructure, noting that it presents more unstructured challenges, and recommended using Ray and Alluxio as solutions for increasingly data-intensive multi-modal AI workloads.

AI/ML Infra Meetup | Exploring Distributed Caching for Faster GPU Training wi...Alluxio, Inc.

AI/ML Infra Meetup Nov. 7, 2024 Organized by Alluxio For more Alluxio Events: https://www.alluxio.io/events/ Speaker: - Bin Fan (Founding Engineer, VP of Technology @ Alluxio) As large-scale machine learning becomes increasingly GPU-centric, modern high-performance hardware like NVMe storage and RDMA networks (InfiniBand or specialized NICs) are becoming more widespread. To fully leverage these resources, it’s crucial to build a balanced architecture that avoids GPU underutilization. In this talk, we will explore various strategies to address this challenge by effectively utilizing these advanced hardware components. Specifically, we will present experimental results from building a Kubernetes-native distributed caching layer, utilizing NVMe storage and high-speed RDMA networks to optimize data access for PyTorch training.

AI/ML Infra Meetup | Big Data and AI, Zoom DevelopersAlluxio, Inc.

AI/ML Infra Meetup Nov. 7, 2024 Organized by Alluxio For more Alluxio Events: https://www.alluxio.io/events/ Speaker: - Sandeep Manchem (ML Platform Engineering Manager @ Zoom) In this talk, Sandeep Manchem (Zoom) discussed big data and AI, covering typical platform architecture and data challenges. We had engaging discussions about ensuring data safety and compliance in Big Data and AI applications.

AI/ML Infra Meetup | TorchTitan, One-stop PyTorch native solution for product...Alluxio, Inc.

AI/ML Infra Meetup Nov. 7, 2024 Organized by Alluxio For more Alluxio Events: https://www.alluxio.io/events/ Speaker: - Tianyu Liu (Research Scientist @ Meta) TorchTitan is a proof-of-concept for Large-scale LLM training using native PyTorch. It is a repo that showcases PyTorch's latest distributed training features in a clean, minimal codebase. In this talk, Tianyu will share TorchTitan’s design and optimizations for the Llama 3.1 family of LLMs, spanning 8 billion to 405 billion parameters, and showcase its performance, composability, and scalability.

Alluxio Webinar | Model Training Across Regions and Clouds – Challenges, Solu...Alluxio, Inc.

Alluxio Webinar October.15, 2024 For more Alluxio Events: https://www.alluxio.io/events/ Speaker: - Tom Luckenbach (Solutions Engineering Manager, Alluxio) AI training workloads running on compute engines like PyTorch, TensorFlow, and Ray require consistent, high-throughput access to training data to maintain high GPU utilization. However, with the decoupling of compute and storage and with today’s hybrid and multi-cloud landscape, AI Platform and Data Infrastructure teams are struggling to cost-effectively deliver the high-performance data access needed for AI workloads at scale. Join Tom Luckenbach, Alluxio Solutions Engineering Manager, to learn how Alluxio enables high-speed, cost-effective data access for AI training workloads in hybrid and multi-cloud architectures, while eliminating the need to manage data copies across regions and clouds. What Tom will share: - AI data access challenges in cross-region, cross-cloud architectures. - The architecture and integration of Alluxio with frameworks like PyTorch, TensorFlow, and Ray using POSIX, REST, or Python APIs across AWS, GCP and Azure. - A live demo of an AI training workload accessing cross-cloud datasets leveraging Alluxio's distributed cache, unified namespace, and policy-driven data management. - MLPerf and FIO benchmark results and cost-savings analysis.

AI/ML Infra Meetup | Scaling Experimentation Platform in Digital Marketplaces...Alluxio, Inc.

AI/ML Infra Meetup Aug. 29, 2024 Organized by Alluxio For more Alluxio Events: https://www.alluxio.io/events/ Speaker: - Koundinya Pidaparthi (VP of Analytics @ Poshmark) Scaling experimentation in digital marketplaces is crucial for driving growth and enhancing user experiences. However, varied methodologies and a lack of experiment governance can hinder the impact of experimentation leading to inconsistent decision-making, inefficiencies, and missed opportunities for innovation. At Poshmark, we developed a homegrown experimentation platform, Lightspeed, that allowed us to make reliable and confident reads on product changes, which led to a 10x growth in experiment velocity and positive business outcomes along the way. This session will provide a deep dive into the best practices and lessons learned from successful implementations of large-scale experiments. We will explore the importance of experimentation, overcome scalability challenges, and gain insights into the frameworks and technologies that enable effective testing.

AI/ML Infra Meetup | Scaling Vector Databases for E-Commerce Visual Search: A...Alluxio, Inc.

AI/ML Infra Meetup Aug. 29, 2024 Organized by Alluxio For more Alluxio Events: https://www.alluxio.io/events/ Speaker: - Mahesh Pasupuleti (VP of DS, ML & Data Infra @ Poshmark) In the rapidly evolving world of e-commerce, visual search has become a game-changing technology. Poshmark, a leading fashion resale marketplace, has developed Posh Lens – an advanced visual search engine that revolutionizes how shoppers discover and purchase items. Under the hood of Posh Lens lies Milvus, a vector database enabling efficient product search and recommendation across our vast catalog of over 150 million items. However, with such an extensive and growing dataset, maintaining high-performance search capabilities while scaling AI infrastructure presents significant challenges. In this talk, Mahesh Pasupuleti shares: - The architecture and strategies to scale Milvus effectively within the Posh Lens infrastructure - Key considerations include optimizing vector indexing, managing data partitioning, and ensuring query efficiency amidst large-scale data growth - Distributed computing principles and advanced indexing techniques to handle the complexity of Poshmark's diverse product catalog

Alluxio Webinar | Optimize, Don't Overspend: Data Caching Strategy for AI Wor...Alluxio, Inc.

Alluxio Webinar Sept. 10, 2024 For more Alluxio Events: https://www.alluxio.io/events/ Speaker: - Jingwen Ouyang (Senior Program Manager, Alluxio) As machine learning and deep learning models grow in complexity, AI platform engineers and ML engineers face significant challenges with slow data loading and GPU utilization, often leading to costly investments in high-performance computing (HPC) storage. However, this approach can result in overspending without addressing the core issues of data bottlenecks and infrastructure complexity. A better approach is adding a data caching layer between compute and storage, like Alluxio, which offers a cost-effective alternative through its innovative data caching strategy. In this webinar, Jingwen will explore how Alluxio's caching solutions optimize AI workloads for performance, user experience and cost-effectiveness. What you will learn: - The I/O bottlenecks that slow down data loading in model training - How Alluxio's data caching strategy optimizes I/O performance for training and GPU utilization, and significantly reduces cloud API costs - The architecture and key capabilities of Alluxio - Using Rapid Alluxio Deployer to install Alluxio and run benchmarks in AWS in just 30 minutes

AI/ML Infra Meetup | Maximizing GPU Efficiency : Optimizing Model Training wi...Alluxio, Inc.

AI/ML Infra Meetup Aug. 29, 2024 Organized by Alluxio For more Alluxio Events: https://www.alluxio.io/events/ Speaker: - Bin Fan (VP of Technology, Founding Engineer @OpenAI) In the rapidly evolving landscape of AI and machine learning, infra teams face critical challenges in managing large-scale data for AI. Performance bottlenecks, cost inefficiencies, and management complexities pose significant challenges for AI platform teams supporting large-scale model training and serving. In this talk, Bin Fan will discuss the challenges of I/O stalls that lead to suboptimal GPU utilization during model training. He will present a reference architecture for running PyTorch jobs with Alluxio in cloud environments, demonstrating how this approach can significantly enhance GPU efficiency. What you will learn: - How to identify GPU utilization and I/O-related performance bottlenecks in model training - Leverage GPU anywhere to maximize resource utilization - Best practices for monitoring and optimizing GPU usage across training and serving pipelines - Strategies for reducing cloud costs and simplifying management of AI infrastructure at scale