Achieving Massive Concurrency & Sub-second Query Latency on Cloud Warehouses & Data Lakes with Kyligence Cloud

Dec 12, 20200 likes114 views

Data Orchestration Summit 2020 organized by Alluxio https://www.alluxio.io/data-orchestration-summit-2020/ Achieving Massive Concurrency & Sub-second Query Latency on Cloud Warehouses & Data Lakes with Kyligence Cloud George Demarest, Head of Marketing, Kyligence About Alluxio: alluxio.io Engage with the open source community on slack: alluxio.io/slack

DATA ORCHESTRATION SUMMIT
2020
Achieving Massive Concurrency on Cloud Warehouses
and Data Lakes with Kyligence Cloud
George Demarest
Head of Marketing
Kyligence, US

DATA ORCHESTRATION SUMMIT
Modern Data Analytics Stack
Processing LayerData Storage Layer Presentation Layer
Data Services
BI Tools and Dashboards
Natural Language ProcessingPredictive Analytics
Machine Learning Data Science
Data Engineering Data Modeling

DATA ORCHESTRATION SUMMIT
Where We Spend Our Time on ML
Data Engineering Machine Learning Operations
80% 5% 15%

DATA ORCHESTRATION SUMMIT
Engineering the Data Platforms
80%

DATA ORCHESTRATION
SUMMIT
2020
Who Owns Analytics Performance?
Data Engineers? Developers? DataOps?

DATA ORCHESTRATION SUMMIT
What if…
…you could accurately
predict and optimize
response times for 95% of
your queries

DATA ORCHESTRATION SUMMIT
Distributed OLAP – Precomputation
Data Volume
Response
Time
Pre-Computation
Run-Time Computation
How many beers were sold in L.A. in June?

DATA ORCHESTRATION SUMMIT
Apache Kylin
Top Level Project
The only open-source distributed
OLAP platform
AwardWinning
InfoWorld’s Bossies 2015 & 2016
(Best of Open Source Software Awards)
Sub-Second Interactive Query
Large scale, high concurrency,
multi-dimensional, sub-second query
latency
1,000+ Organizations
Adopted by thousands of
organizations globally

DATA ORCHESTRATION SUMMIT
Kyligence Cloud: Extreme Analytics
Supports All Data Platforms
• Data Warehouse
• Data Lake
• Streaming Data
• Cloud Storage
Intelligent Precomputation
• Multi-dimensional cubes and
table Indexes
• AI-assisted query optimization,
data modeling
Supports All BI Tools
• Excel
• Power BI
• Tableau
• MicroStrategy
Massive Concurrency
• 1000s+ concurrent queries
• 10s-100s analysts running
multi-threaded programs
Powered by Apache Kylin

DATA ORCHESTRATION SUMMIT
Kyligence Analytical Cloud
ANSI SQL
MDX
REST APIFinance
Marketing
Sales
Precomputation Layer
Distributed cubes
& table indexes
Modeling Layer
AI-assisted
design
Uniﬁed security &
business context
Semantic Layer
Data as a Service
Machine Learning
Data as a Service
SaaS & Apps
CRM HCM
SCM
Low Latency Queries
Any BI Tool
Any Cloud
Any Data Platform
Data Warehouse
Streaming Data
Data Lake
AI Augmented Engine
Intelligent Precomputation Platform to Accelerate Cloud Analytics

DATA ORCHESTRATION SUMMIT
Kyligence Cloud
ANSI SQL
MDX
REST APIFinance
Marketing
Sales
Precomputation Layer
Distributed cubes
& table indexes
Modeling Layer
AI-assisted
design
Uniﬁed security &
business context
Semantic Layer
Data as a Service
Machine Learning
Data as a Service
SaaS & Apps
CRM HCM
SCM
Low Latency Queries
Any BI Tool
Any Cloud
Any Data Platform
Data Warehouse
Streaming Data
Data Lake
AI Augmented Engine

DATA ORCHESTRATION SUMMIT
Intelligent Query Routing
Route to Precomputation Layer or Pushdown to Native Engine

DATA ORCHESTRATION SUMMIT
Kyligence AI Augmented Engine
Model training
Pre-aggregation
Data processing
Auto-Generate
Cube & Index
AI Augmented
Engine
Self training
Group wisdom
Queries across the org
ingested as input
Benefits
- Continuous model improvement
without human intervention
- Greater user satisfaction
Identify and manage most valuable data automatically

DATA ORCHESTRATION SUMMIT
Kyligence = Kylin + Intelligence
Mission: Develop an AI-Assisted
SQL/OLAP Platform for Interactive
Analytics at Unprecedented Scale
• Founded in 2016 by the creators of Apache Kylin
• CRN Top-10 big data startups in 2018
• Global Presence: San Jose, Shanghai, Beijing,
Seattle, New York
• VCs: Fidelity International, Shunwei Capital,
Broadband Capital, Coatue, Redpoint, Cisco

DATA ORCHESTRATION
SUMMIT
2020
Thank You

In the on-demand economy real-time analytics is both a necessity and a competitive advantage. The next evolution in the on-demand economy is in predictive analytics fueled by live streams of data—in effect knowing what customers want before they do. This session will feature technical examples of real-time pipelines, machine learning, and custom dashboards as well as off-the-shelf dashboards with Tableau.

Spark and the Enterprise by Tony BaerSpark Summit

This document discusses the growing popularity and capabilities of the Apache Spark platform for large-scale data analytics. It notes that Spark has over 40 committers, 1000 contributors, and is being used in 179 projects. The document highlights key features of Spark like its ease of use, performance (10-100x faster than MapReduce), flexibility, and ability to handle both batch and real-time processing. It also provides examples of how Spark can help businesses by enabling more complex analytics like predictive modeling, enabling smarter predictions, and allowing insights from real-time data. The document emphasizes that Spark advocates should focus on illustrating tangible business benefits over technical features when discussing Spark with higher-level business stakeholders.

Tableau Conference 2018: Binging on Data - Enabling Analytics at NetflixBlake Irvine

In this conference session we share how we are using Tableau “out of the box” and also describe how it fits into our overall data environment. In addition, we’ll describe how we expect to use the Data Catalog and Object Model, our explorations of large-scale data stores, and challenges we are working on including governance and data lineage. Video of session can be viewed here: https://youtu.be/Nr24tw3dmZQ

Driving the On-Demand Economy with Spark and Predictive AnalyticsSingleStore

The document discusses how data scientists need real-time analytics capabilities to power the on-demand economy. It introduces MemSQL 5 as a database platform for real-time analytics that can help overcome barriers like slow loading, queries, and ongoing data processing faced with batch processing. MemSQL 5 includes features like Streamliner for building real-time data pipelines and predictive analytics using Spark and MLlib to power applications like predictive scoring and IoT.

JAZOON'13 - Kai Waehner - Hadoop Integrationjazoon13

This document discusses integrating big data beyond Hadoop. It begins with an introduction and overview of Kai Wähner and his background and expertise. The rest of the document focuses on big data integration, with sections on the challenges of integrating big data, choosing Hadoop as a solution, and different approaches to integration including using open source frameworks, suites, and custom connectors. Throughout there are examples of use cases and how companies are leveraging big data and integration to solve business problems. The overall message is that in order to get full value from data, you need to integrate different sources, and that big data integration approaches have advanced to make this more achievable.

In-Memory Computing Webcast. Market Predictions 2017SingleStore

1) In-memory computing is growing rapidly, with the total data market expected to grow from $69 billion in 2015 to $132 billion in 2020. 2) In-memory databases are gaining popularity for applications that require fast response times, like telecommunications and mobile advertising, as memory access is faster than disk access. 3) Modern applications are driving adoption of in-memory solutions as they generate more data from more users and transactions and require faster performance to handle growing traffic. 4) Two examples presented were DellEMC using MemSQL for a real-time customer 360 application and an IoT logistics application called MemEx that processes sensor data from warehouses for predictive analytics.

Building the Ideal Stack for Machine LearningSingleStore

This document discusses building an ideal machine learning stack for real-time applications. It provides an overview of machine learning and real-time use cases. It also demonstrates an example machine learning pipeline for fraud detection using labeled historical data to train a model and score new data in real-time. Finally, it discusses industry examples of using machine learning models for advertising optimization, anomaly detection, and image recognition and provides a demo of predicting images from the MNIST data set.

Integrating Web and Business DataSterling Geo

This document discusses how data can be collected from various sources and transformed into useful information using FME. It outlines a three step process: 1) Connect to data from different formats and sources, 2) Clean the data by keeping useful information and filtering out bad data, and 3) Make the data presentable through visualizations, statistics, and reports. Examples of using FME to visualize trial results in PDFs and create dashboards for infrastructure asset management are provided. QlikMaps, Tableau, and other tools for business intelligence and creating visualizations are also mentioned.

Delivering real time analytics in 1 clickJean-Michel Franco

This presentation is about leveraging Big Data environments including Hadoop, Spark and Storm to: - Easily integrate disparate data sources and streams in real time to capture business events as they occur - Leverage predictive analytics and machine learning across all your data to derive the right insight at the right time - Build decision-centric systems that use this insight to act in real time, so you can capture new opportunities as they occur

Big data at zulilyzulily

InfoTrellis CorporateMichael Harris

This document provides an overview of Infotrellis' services portfolio for helping customers realize the full value of their information assets. It describes Infotrellis' technical and functional practices including master data management, data integration, big data, and analytics solutions. It also introduces Infotrellis' SMART delivery methodology for quickly delivering business value and maintaining collaboration between business and IT stakeholders.

Providing Interactive Analytics on Excel with Billions of RowsTyler Wishnoff

How a Media Data Platform Drives Real-time Insights & Analytics using Apache ...Databricks

Roularta is a leading publishing company in Belgium. As digital news and channels move at a rapid pace and contain massive volumes of data, Roularta decided in 2019 to invest in a Spark-based data platform to drive true real-time website analytics and unlock insights on previously untouched (big) data sources. In this talk we’ll first explain why and how Roularta embarked from a classical data warehouse to a Spark-based Lakehouse using Delta. We’ll outline the series of publishing & marketing use-cases done in the last 12 months and highlight for each use-case the advantages of Spark and how the team further tuned performance to truly deliver insights with high velocity.

Data Science at Netflix - Principles for Speed & Scale [Rev 2019 keynote]Michelle Ufford

Data science powers Netflix. It informs our decisions and challenges our assumptions. It fuels experimentation and innovation at an unprecedented scale. It helps us discover fantastic content and deliver personalized experiences for millions of people around the world. In short, data science has become critical to Netflix, touching nearly every aspect of the business today. In this talk, Michelle Ufford will discuss the philosophies and innovations that have helped Netflix scale its data science practice. She’ll walk through what the process looks like today and how Netflix is tackling some of the biggest challenges facing data scientists. Through it all, she’ll share core principles that have contributed to their success, which you can put to immediate use at your own company.

Ruben Diaz, Vision Banco + Rafael Coss, H2O ai + Luis Armenta, IBM - AI journ...Sri Ambati

This session was recorded in San Francisco on February 5th, 2019 and can be viewed here: https://youtu.be/otq2nQUSV3s We will talk about the AI transformation journey at Vision Banco - Paraguay, from the early initiatives to futures use cases, and how we adopted open source H2O.ai and Driverless AI in our organization. Bio: Ruben Diaz My name is Ruben Diaz, from Asunción, Paraguay. I am married and father of 3 children. I work as Data Scientist at Vision Banco Luis Armenta: Luis holds a BSc in Electrical Engineering from the National University of Mexico and a MSc in Electrical Engineering/Computer Science from the University of Waterloo in Canada. He is also currently completing an Executive MBA at McCombs School of Business at the University of Texas in Austin. Luis has over ~14 years of experience, having started his career as a Research Scientist at Intel Labs before being promoted to 2nd Line Engineering Manager, leading the high-speed interconnect hardware design of Intel’s server portfolio. Luis also has held roles as Product Manager of EM simulators at Ansys, Inc. and as a Systems Engineer of 4K and 8K UHDTVs at Macom.

Building a Just in Time Data Warehouse by Dan Morris and Jason PohlSpark Summit

This document summarizes Viacom's approach to building a just-in-time data warehouse for product analytics. It discusses how Viacom uses Spark and Databricks on Amazon S3 to enable flexible exploratory analysis and A/B testing without needing to pre-define schemas or move data into a data warehouse. This just-in-time approach allows Viacom's product analysts to quickly get insights, test hypotheses, and monitor key metrics through dashboards for their mobile apps across 23 brands in over 170 countries. The summary highlights how Viacom uses this flexible approach to support their goals of growing global audiences through disciplined product development and data-driven decision making.

Unlocking Geospatial Analytics Use Cases with CARTO and DatabricksDatabricks

Many companies need to analyze large datasets that include location information. To be able to derive business insights from these datasets you need a solution that provides geospatial analysis functionalities and can scale to manage large volumes of information. The combination of CARTO and Databricks allows you to solve this kind of large scale geospatial analytics problems. CARTO provides a location intelligence platform to discover and predict key insights through location data. In this session we will see how we can integrate CARTO and Databricks and how we can take advantage of this combination to solve specific problems for industries such as logistics, telecommunications or financial services.

Build a Big Data Warehouse on the Cloud in 30 MinutesCaserta

Streaming Analytics for IoT with Apache SparkImpetus Technologies

Modern IoT operations can drive digital transformation by analyzing the unprecedented amounts of data generated from devices and sensors in real-time. Apache Spark is a widely used stream processing engine for real-time IoT applications. Spark streaming offers a rich set of APIs in the areas of ingestion, cloud integration, multi-source joins, blending streams with static data, time-window aggregations, transformations, data cleansing, and strong support for machine learning and predictive analytics. Join Anand Venugopal, AVP & Business Head, StreamAnalytix and Sameer Bhide, Senior Solutions Architect, StreamAnalytix to learn about the rapid development and operationalization of real-time IoT applications covering an end-to-end flow of ingest, insight, action, and feedback. The webinar will cover the following: Generic IoT application blueprint Case studies on IoT applications built on Apache Spark – connected car and industrial IoT Demonstration of an easy, visual approach to building IoT Spark apps

Introduction to Big Data using AWS ServicesAnjani Phuyal

Building the Ideal Stack for Real-Time AnalyticsSingleStore

This document discusses building an ideal real-time analytics stack. It promotes the MemSQL database platform as being ranked #1 for operational data warehousing and being able to handle real-time analytics at massive scales. The ideal stack presented includes components for message queues, a transformation tier, data persistence with MemSQL, and real-time visualization. Use cases discussed include processing web logs, mobile apps, IoT data, and an energy company's drilling sensor data.

From Business Hindsight to Foresight with Azure Synapse AnalyticsKorcomptenz Inc

From Business Hindsight to Foresight with Azure Synapse Analytics The document discusses how Azure Synapse Analytics can help organizations transition from descriptive analytics of past data to predictive analytics and prescriptive insights. It provides an overview of Azure Synapse's capabilities for data integration, warehousing, and big data analytics. Case studies demonstrate how customers have used Azure Synapse and Power BI to improve operations, customer experiences, and enable predictive maintenance.

TableauChristaJacob1

Machines and the Magic of Fast LearningSingleStore

Human-machine interaction is no longer the exclusive province of science fiction. The advance of the internet and connected devices has inspired data scientists to create machine-learning applications to extract value from these new forms of data. So what's the next frontier? Join MemSQL Engineer Michael Andrews and Sr. Director Mike Boyarski to learn how to use real-time data as a vehicle for operationalizing machine-learning models. Michael and Mike will explore advanced tools, including TensorFlow, Apache Spark, and Apache Kafka, and compelling use cases demonstrating the power of machine learning to effect positive change. You will learn: Top technologies for building the ideal machine-learning stack How to power machine-learning applications with real-time data A use case and demo of machine learning for social good

Microsoft Fabric IntroductionJames Serra

Microsoft Fabric is the next version of Azure Data Factory, Azure Data Explorer, Azure Synapse Analytics, and Power BI. It brings all of these capabilities together into a single unified analytics platform that goes from the data lake to the business user in a SaaS-like environment. Therefore, the vision of Fabric is to be a one-stop shop for all the analytical needs for every enterprise and one platform for everyone from a citizen developer to a data engineer. Fabric will cover the complete spectrum of services including data movement, data lake, data engineering, data integration and data science, observational analytics, and business intelligence. With Fabric, there is no need to stitch together different services from multiple vendors. Instead, the customer enjoys end-to-end, highly integrated, single offering that is easy to understand, onboard, create and operate. This is a hugely important new product from Microsoft and I will simplify your understanding of it via a presentation and demo. Agenda: What is Microsoft Fabric? Workspaces and capacities OneLake Lakehouse Data Warehouse ADF Power BI / DirectLake Resources

Microsoft cloud big data strategyJames Serra

Think of big data as all data, no matter what the volume, velocity, or variety. The simple truth is a traditional on-prem data warehouse will not handle big data. So what is Microsoft’s strategy for building a big data solution? And why is it best to have this solution in the cloud? That is what this presentation will cover. Be prepared to discover all the various Microsoft technologies and products from collecting data, transforming it, storing it, to visualizing it. My goal is to help you not only understand each product but understand how they all fit together, so you can be the hero who builds your companies big data solution.

A Winning Strategy for the Digital EconomyEric Kavanagh

The speed of innovation today creates tremendous opportunities for some, existential threats for others. Companies that win create their own success by leveraging modern data platforms. While architectures vary, the foundation is often in-memory, and the latency is real-time. Register for this Special Edition of The Briefing Room to hear veteran Analyst Dr. Robin Bloor explain how today's data platforms enable the modern enterprise in groundbreaking ways. He'll be briefed by Chris Hallenbeck of SAP who will demonstrate how forward-looking companies are leveraging real-time data platforms to achieve operational excellence, make decisions faster, and find new ways to innovate.

Analytics in a Day Ft. Synapse Virtual WorkshopCCG

More Related Content

What's hot (18)

Building the Ideal Stack for Machine LearningSingleStore

Integrating Web and Business DataSterling Geo

Delivering real time analytics in 1 clickJean-Michel Franco

Big data at zulilyzulily

InfoTrellis CorporateMichael Harris

Providing Interactive Analytics on Excel with Billions of RowsTyler Wishnoff

How a Media Data Platform Drives Real-time Insights & Analytics using Apache ...Databricks

Data Science at Netflix - Principles for Speed & Scale [Rev 2019 keynote]Michelle Ufford

Ruben Diaz, Vision Banco + Rafael Coss, H2O ai + Luis Armenta, IBM - AI journ...Sri Ambati

Building a Just in Time Data Warehouse by Dan Morris and Jason PohlSpark Summit

Unlocking Geospatial Analytics Use Cases with CARTO and DatabricksDatabricks

Build a Big Data Warehouse on the Cloud in 30 MinutesCaserta

Streaming Analytics for IoT with Apache SparkImpetus Technologies

Introduction to Big Data using AWS ServicesAnjani Phuyal

Building the Ideal Stack for Real-Time AnalyticsSingleStore

From Business Hindsight to Foresight with Azure Synapse AnalyticsKorcomptenz Inc

TableauChristaJacob1

Machines and the Magic of Fast LearningSingleStore

Building the Ideal Stack for Machine LearningSingleStore

Integrating Web and Business DataSterling Geo

Delivering real time analytics in 1 clickJean-Michel Franco

Big data at zulilyzulily

InfoTrellis CorporateMichael Harris

Providing Interactive Analytics on Excel with Billions of RowsTyler Wishnoff

How a Media Data Platform Drives Real-time Insights & Analytics using Apache ...Databricks

Data Science at Netflix - Principles for Speed & Scale [Rev 2019 keynote]Michelle Ufford

Ruben Diaz, Vision Banco + Rafael Coss, H2O ai + Luis Armenta, IBM - AI journ...Sri Ambati

Building a Just in Time Data Warehouse by Dan Morris and Jason PohlSpark Summit

Unlocking Geospatial Analytics Use Cases with CARTO and DatabricksDatabricks

Build a Big Data Warehouse on the Cloud in 30 MinutesCaserta

Streaming Analytics for IoT with Apache SparkImpetus Technologies

Introduction to Big Data using AWS ServicesAnjani Phuyal

Building the Ideal Stack for Real-Time AnalyticsSingleStore

From Business Hindsight to Foresight with Azure Synapse AnalyticsKorcomptenz Inc

TableauChristaJacob1

Machines and the Magic of Fast LearningSingleStore

Similar to Achieving Massive Concurrency & Sub-second Query Latency on Cloud Warehouses & Data Lakes with Kyligence Cloud (20)

Microsoft Fabric IntroductionJames Serra

Microsoft cloud big data strategyJames Serra

A Winning Strategy for the Digital EconomyEric Kavanagh

Analytics in a Day Ft. Synapse Virtual WorkshopCCG

Customer Presentation - IBM Cloud Pak for Data Overview (Level 100).PPTXtsigitnist02

This document provides instructions for using a presentation deck on Cloud Pak for Data. It instructs the user to: 1. Delete the first slide before using the deck. 2. Customize the presentation for the intended audience as the deck covers various topics and using all slides may not fit a single meeting. 3. The deck contains 6 embedded video records for a demo that takes 15-25 minutes to present. Guidance on pitching the demo is available. The appendix contains slides on Cloud Pak for Data licensing and IBM's overall strategy.

Accelerate Self-Service Analytics with Virtualization and Visualisation (Thai)Denodo

Watch full webinar here: https://bit.ly/2TF7ATo ติดสปีดการวิเคราะห์ข้อมูลด้วยตัวเองแบบรวดเร็วแม่นยำด้วย Data Virtualization และ Data Visualisation องค์กรชั้นนำกำลังขยับตัวไปสู่การวิเคราะห์ข้อมูลทางธุรกิจด้วยตัวเอง เพราะผู้ใช้งานต้องการการเข้าถึงข้อมูลแบบทันทีเพื่อเห็นภาพแบบองค์รวมและหลากหลายมิติโดยไม่ต้องคำนึงถึง ตำแหน่งและ แหล่งที่มา หรือชนิดของข้อมูลสำหรับการตัดสินใจครั้งสำคัญ Data Virtualization (การทำให้เห็นข้อมูลแบบเสมือนจริง) และ Data Visualisation (การทำให้เห็นข้อมูลเป็นภาพ)ทำงานด้วยกันผ่านชั้นข้อมูลเชิงความหมายที่เป็นรูปแบบสากล Webinar นี้คุณจะได้รู้วิธีว่าเราจะสามารถค้นพบข้อมูลมาวิเคราะห์ด้วยตัวเองอย่างไรและยกระดับประสิทธิภาพการทำรายงานและแดชบอร์ดของคุณได้อย่างไร ใน Webinar นี้ คุณจะได้เรียนรู้เรื่อง อุปสรรคและความท้าทายที่ผู้ใช้งานในองค์กรธุรกิจต้องเผชิญ Data Virtualization ทำให้คุณสามารถวิเคราะห์ข้อมูลได้ด้วยตัวเองได้อย่างไร กรณีศึกษาและบทเรียนจากความสำเร็จของลูกค้า สรุป Tableau Highlight Features

Azure Synapse 101 Webinar PresentationMatthew W. Bowers

Unlock Data-driven Insights in Databricks Using Location IntelligencePrecisely

Today’s data-driven organisations are turning to Databricks for a cloud-based, open, unified platform for data and AI. Yet many companies struggle to unlock the value of the data they have in Databricks. To capitalise on the promise of a competitive edge through increased efficiency and insight, data scientists are turning to location to make sense of massive volumes of business data. Watch this on-demand to hear from The Spatial Distillery Co. and Databricks on how to leverage advanced location intelligence and enrichment solutions in Databricks to: - Simplify the complexity of location data and transform it into valuable insights - Enrich data with thousands of attributes for better, more accurate analytics, AI, and ML models - Leverage the power of Databricks to integrate geospatial data into business processes for real-time answers - Create more meaningful and timely customer interactions by streamlining customer-facing and operational tasks

Overview on Azure Machine LearningJames Serra

Machine learning allows us to build predictive analytics solutions of tomorrow - these solutions allow us to better diagnose and treat patients, correctly recommend interesting books or movies, and even make the self-driving car a reality. Microsoft Azure Machine Learning (Azure ML) is a fully-managed Platform-as-a-Service (PaaS) for building these predictive analytics solutions. It is very easy to build solutions with it, helping to overcome the challenges most businesses have in deploying and using machine learning. In this presentation, we will take a look at how to create ML models with Azure ML Studio and deploy those models to production in minutes.

Ανδρέας Τσαγκάρης, 5th Digital Banking ForumStarttech Ventures

DataOps: Control-M's role in data pipeline orchestrationpzjnjr6rsg

Master the art of Data ScienceInTTrust S.A.

MLOps journey at Swisscom: AI Use Cases, Architecture and Future VisionBATbern

What powers the AI/ML services of Switzerland's leading telecommunication company? In this talk, we will provide an overview of the different AI/ML projects at Swisscom, from Conversational AI and Recommender Systems to Anomaly Detection. Moreover, we will show how we automate, scale, and operationalise these ML pipelines in production, highlighting the MLOps techniques and open source tools that are used. Finally, we will present Swisscom's roadmap towards the cloud with AWS and discuss how we envision a common MLOps solution for the organisation.

Cloud Ready Data: Speeding Your Journey to the CloudDLT Solutions

Project Presentation_Group 2.pptxHarshSingh594306

Microsoft Azure 101 provides an overview of Microsoft's Azure cloud platform. It discusses how Azure offers infrastructure services, platform services and data services to help businesses stay ahead of trends like intelligence, cloud computing and big data. It also provides examples of how Azure can help industries like retail, manufacturing, government, health and financial services. The document summarizes Microsoft's strategy of offering public cloud services alongside hybrid cloud and on-premises options. It highlights some of Azure's core services and capabilities like Azure SQL Database, Azure Machine Learning, Cortana Intelligence Suite, and more.

Modern Thinking área digital MSKM 21/09/2017MSMK - Madrid School of Marketing

Navigating Your Data Landscape With Siddharth Desai and Elena Cuevas | Curren...HostedbyConfluent

Navigating Your Data Landscape With Siddharth Desai and Elena Cuevas | Current 2022 We see data and the way organizations using it to create unique customer experiences is being completely reimagined. Every time a customer clicks, types or swipes data is in constant motion spanning systems, environments and applications. This in turn requires business’ manage complex integrations, synchronizations and processing of data spread across cloud and on-prem environments. To accelerate time to value, data needs to be seamlessly ingested, integrated and/or replicated to a cloud environment to take advantage of its analytical, BI and AI use cases. Google Cloud delivers a simplified approach for all your Data Movement needs through a highly differentiated product portfolio. In this session, learn how organizations can unlock data value using best-in-class, cloud native products on Google Cloud and its partners such as Confluent.

ICP for Data- Enterprise platform for AI, ML and Data ScienceKaran Sachdeva

How does Microsoft solve Big Data?James Serra

So you got a handle on what Big Data is and how you can use it to find business value in your data. Now you need an understanding of the Microsoft products that can be used to create a Big Data solution. Microsoft has many pieces of the puzzle and in this presentation I will show how they fit together. How does Microsoft enhance and add value to Big Data? From collecting data, transforming it, storing it, to visualizing it, I will show you Microsoft’s solutions for every step of the way

Trivadis Azure Data LakeTrivadis

Microsoft Fabric IntroductionJames Serra

Microsoft cloud big data strategyJames Serra

A Winning Strategy for the Digital EconomyEric Kavanagh

Analytics in a Day Ft. Synapse Virtual WorkshopCCG

Customer Presentation - IBM Cloud Pak for Data Overview (Level 100).PPTXtsigitnist02

Accelerate Self-Service Analytics with Virtualization and Visualisation (Thai)Denodo

Azure Synapse 101 Webinar PresentationMatthew W. Bowers

Unlock Data-driven Insights in Databricks Using Location IntelligencePrecisely

Overview on Azure Machine LearningJames Serra

Ανδρέας Τσαγκάρης, 5th Digital Banking ForumStarttech Ventures

DataOps: Control-M's role in data pipeline orchestrationpzjnjr6rsg

Master the art of Data ScienceInTTrust S.A.

MLOps journey at Swisscom: AI Use Cases, Architecture and Future VisionBATbern

Cloud Ready Data: Speeding Your Journey to the CloudDLT Solutions

Project Presentation_Group 2.pptxHarshSingh594306

Modern Thinking área digital MSKM 21/09/2017MSMK - Madrid School of Marketing

Navigating Your Data Landscape With Siddharth Desai and Elena Cuevas | Curren...HostedbyConfluent

ICP for Data- Enterprise platform for AI, ML and Data ScienceKaran Sachdeva

How does Microsoft solve Big Data?James Serra

Trivadis Azure Data LakeTrivadis

More from Alluxio, Inc. (20)

How Coupang Leverages Distributed Cache to Accelerate ML Model TrainingAlluxio, Inc.

Alluxio Tech Talk Webinar Apr. 22, 2025 Organized by Alluxio For more Alluxio Events: https://www.alluxio.io/events/ Speaker: - Hyun Jung Baek (Staff Backend Engineer @ Coupang) Description Coupang is a leading e-commerce company in South Korea, with over 50,000 employees and $20+ billion in annual revenue. Coupang's AI platform team builds and manages a large-scale AI platform in AWS for machine learning engineers to train models that enhance and customize product search results and product recommendations for its 100+ million customers. As the search and recommendation models evolve, optimizing the underlying infrastructure for AI/ML workloads is essential for the e-commerce business. Coupang's platform team actively sought to improve their model training pipeline to boost machine learning engineers' productivity, publish models to production faster, and reduce operational costs. Coupang focused on addressing several key areas: - Shortening data preparation and model training time - Improving GPU utilization in training clusters in different regions - Reducing S3 API and egress costs incurred from copying large training datasets across regions - Simplifying the operational complexity of storage system management In this tech talk, Hyun Jung Baek, Staff Backend Engineer at Coupang, will share best practices for leveraging Alluxio to power search and recommendation model training infrastructure. Hyun will discuss: - How Coupang builds a world-class large-scale AI platform for machine learning engineers to deliver better search and recommendation models - How adding distributed caching to their multi-region AI infrastructure improves GPU utilization, accelerates end-to-end training time, and significantly reduces cross-region data transfer costs. - How to simplify platform operations and to easily deploy the same architecture to new GPU clusters.

Alluxio Webinar | Inside Deepseek 3FS: A Deep Dive into AI-Optimized Distribu...Alluxio, Inc.

Alluxio Webinar Apr 1, 2025 For more Alluxio Events: https://www.alluxio.io/events/ Speaker: Stephen Pu (Staff Software Engineer @ Alluxio) Deepseek’s recent announcement of the Fire-flyer File System (3FS) has sparked excitement across the AI infra community, promising a breakthrough in how machine learning models access and process data. In this webinar, an expert in distributed systems and AI infrastructure will take you inside Deepseek 3FS, the purpose-built file system for handling large files and high-bandwidth workloads. We’ll break down how 3FS optimizes data access and speeds up AI workloads as well as the design tradeoffs made to maximize throughput for AI workloads. This webinar you’ll learn about how 3FS works under the hood, including: ✅ The system architecture ✅ Core software components ✅ Read/write flows ✅ Data distribution/placement algorithms ✅ Cluster/node management and disaster recovery Whether you’re an AI researcher, ML engineer, or infrastructure architect, this deep dive will give you the technical insights you need to determine if 3FS is the right solution for you.

AI/ML Infra Meetup | Building Production Platform for Large-Scale Recommendat...Alluxio, Inc.

AI/ML Infra Meetup | How Uber Optimizes LLM Training and FinetuneAlluxio, Inc.

AI/ML Infra Meetup | Optimizing ML Data Access with Alluxio: Preprocessing, ...Alluxio, Inc.

AI/ML Infra Meetup Mar. 06, 2025 Organized by Alluxio For more Alluxio Events: https://www.alluxio.io/events/ Speaker: - Bin Fan (VP of Technology @ Alluxio) In this talk, Bin Fan shares his insights on data access challenges in ML applications, with particular emphasis on how Alluxio's distributed caching helps bridge the gap between storage and compute in preprocessing, pretraining and inference.

AI/ML Infra Meetup | Deployment, Discovery and Serving of LLMs at Uber ScaleAlluxio, Inc.

Alluxio Webinar | What’s New in Alluxio AI: 3X Faster Checkpoint File Creatio...Alluxio, Inc.

Alluxio Webinar Feb. 25, 2025 For more Alluxio Events: https://www.alluxio.io/events/ Speaker: Bill Hodak (VP of Marketing and Product Marketing, Alluxio) Tom Luckenbach (Solutions Engineering Manager, Alluxio) Join us to learn about the latest release of Alluxio Enterprise AI. In this webinar, we’ll provide an overviewof the new features and capabilities of Alluxio Enterprise AI, built to accelerate AI workloads and maximize GPU utilization. Key highlights include: - New caching mode accelerates AI checkpoints - Advanced cache eviction policies provide fine-grained control - Python SDK integrations enhance AI framework compatibility - A demo of Alluxio accelerating AI training workloads in AWS

AI/ML Infra Meetup | A Faster and More Cost Efficient LLM Inference StackAlluxio, Inc.

AI/ML Infra Meetup Jan. 23, 2025 Organized by Alluxio For more Alluxio Events: https://www.alluxio.io/events/ Speaker: - Junchen Jiang (Assistant Professor @ University of Chicago) LLM inference can be huge, particularly, with long contexts. In this on-demand video, Junchen Jiang, Assistant Professor at University of Chicago, presents a 10x solution for long contexts inference: an easy-to-deploy stack over multiple vLLM engines with tailored KV-cache backend.

AI/ML Infra Meetup | Balancing Cost, Performance, and Scale - Running GPU/CPU...Alluxio, Inc.

AI/ML Infra Meetup | RAYvolution - The Last Mile: Mastering AI Deployment wit...Alluxio, Inc.

Alluxio Webinar | Accelerate AI: Alluxio 101Alluxio, Inc.

Alluxio Webinar Dec. 3, 2024 For more Alluxio Events: https://www.alluxio.io/events/ Speaker: Bill Hodak (VP of Marketing and Product Marketing, Alluxio) In the rapidly evolving landscape of AI and machine learning, Platform and Data Infrastructure Teams face critical challenges in building and managing large-scale AI platforms. Performance bottlenecks, scalability of the platform, and scarcity of GPUs pose significant challenges in supporting large-scale model training and serving. In this talk, we will introduce how Alluxio helps Platform and Data Infrastructure teams deliver faster, more scalable platforms to ML Engineering teams developing and training AI models. Alluxio’s highly-distributed cache accelerates AI workloads by eliminating data loading bottlenecks and maximizing GPU utilization. Customers report up to 4x faster training performance with high-speed access to petabytes of data spread across billions of files regardless of persistent storage type or proximity to GPU clusters. Alluxio’s architecture lowers data infrastructure costs, increases GPU utilization, and enables workload portability for navigating GPU scarcity challenges.

AI/ML Infra Meetup | The power of Ray in the era of LLM and multi-modality AIAlluxio, Inc.

AI/ML Infra Meetup Nov. 7, 2024 Organized by Alluxio For more Alluxio Events: https://www.alluxio.io/events/ Speaker: - Zhe Zhang (Distinguished Engineer @ NVIDIA) In this talk, Zhe Zhang (NVIDIA, ex-Anyscale) introduced Ray and its applications in the LLM and multi-modal AI era. He shared his perspective on ML infrastructure, noting that it presents more unstructured challenges, and recommended using Ray and Alluxio as solutions for increasingly data-intensive multi-modal AI workloads.

AI/ML Infra Meetup | Exploring Distributed Caching for Faster GPU Training wi...Alluxio, Inc.

AI/ML Infra Meetup Nov. 7, 2024 Organized by Alluxio For more Alluxio Events: https://www.alluxio.io/events/ Speaker: - Bin Fan (Founding Engineer, VP of Technology @ Alluxio) As large-scale machine learning becomes increasingly GPU-centric, modern high-performance hardware like NVMe storage and RDMA networks (InfiniBand or specialized NICs) are becoming more widespread. To fully leverage these resources, it’s crucial to build a balanced architecture that avoids GPU underutilization. In this talk, we will explore various strategies to address this challenge by effectively utilizing these advanced hardware components. Specifically, we will present experimental results from building a Kubernetes-native distributed caching layer, utilizing NVMe storage and high-speed RDMA networks to optimize data access for PyTorch training.

AI/ML Infra Meetup | Big Data and AI, Zoom DevelopersAlluxio, Inc.

AI/ML Infra Meetup Nov. 7, 2024 Organized by Alluxio For more Alluxio Events: https://www.alluxio.io/events/ Speaker: - Sandeep Manchem (ML Platform Engineering Manager @ Zoom) In this talk, Sandeep Manchem (Zoom) discussed big data and AI, covering typical platform architecture and data challenges. We had engaging discussions about ensuring data safety and compliance in Big Data and AI applications.

AI/ML Infra Meetup | TorchTitan, One-stop PyTorch native solution for product...Alluxio, Inc.

AI/ML Infra Meetup Nov. 7, 2024 Organized by Alluxio For more Alluxio Events: https://www.alluxio.io/events/ Speaker: - Tianyu Liu (Research Scientist @ Meta) TorchTitan is a proof-of-concept for Large-scale LLM training using native PyTorch. It is a repo that showcases PyTorch's latest distributed training features in a clean, minimal codebase. In this talk, Tianyu will share TorchTitan’s design and optimizations for the Llama 3.1 family of LLMs, spanning 8 billion to 405 billion parameters, and showcase its performance, composability, and scalability.

Alluxio Webinar | Model Training Across Regions and Clouds – Challenges, Solu...Alluxio, Inc.

Alluxio Webinar October.15, 2024 For more Alluxio Events: https://www.alluxio.io/events/ Speaker: - Tom Luckenbach (Solutions Engineering Manager, Alluxio) AI training workloads running on compute engines like PyTorch, TensorFlow, and Ray require consistent, high-throughput access to training data to maintain high GPU utilization. However, with the decoupling of compute and storage and with today’s hybrid and multi-cloud landscape, AI Platform and Data Infrastructure teams are struggling to cost-effectively deliver the high-performance data access needed for AI workloads at scale. Join Tom Luckenbach, Alluxio Solutions Engineering Manager, to learn how Alluxio enables high-speed, cost-effective data access for AI training workloads in hybrid and multi-cloud architectures, while eliminating the need to manage data copies across regions and clouds. What Tom will share: - AI data access challenges in cross-region, cross-cloud architectures. - The architecture and integration of Alluxio with frameworks like PyTorch, TensorFlow, and Ray using POSIX, REST, or Python APIs across AWS, GCP and Azure. - A live demo of an AI training workload accessing cross-cloud datasets leveraging Alluxio's distributed cache, unified namespace, and policy-driven data management. - MLPerf and FIO benchmark results and cost-savings analysis.

AI/ML Infra Meetup | Scaling Experimentation Platform in Digital Marketplaces...Alluxio, Inc.

AI/ML Infra Meetup Aug. 29, 2024 Organized by Alluxio For more Alluxio Events: https://www.alluxio.io/events/ Speaker: - Koundinya Pidaparthi (VP of Analytics @ Poshmark) Scaling experimentation in digital marketplaces is crucial for driving growth and enhancing user experiences. However, varied methodologies and a lack of experiment governance can hinder the impact of experimentation leading to inconsistent decision-making, inefficiencies, and missed opportunities for innovation. At Poshmark, we developed a homegrown experimentation platform, Lightspeed, that allowed us to make reliable and confident reads on product changes, which led to a 10x growth in experiment velocity and positive business outcomes along the way. This session will provide a deep dive into the best practices and lessons learned from successful implementations of large-scale experiments. We will explore the importance of experimentation, overcome scalability challenges, and gain insights into the frameworks and technologies that enable effective testing.

AI/ML Infra Meetup | Scaling Vector Databases for E-Commerce Visual Search: A...Alluxio, Inc.

AI/ML Infra Meetup Aug. 29, 2024 Organized by Alluxio For more Alluxio Events: https://www.alluxio.io/events/ Speaker: - Mahesh Pasupuleti (VP of DS, ML & Data Infra @ Poshmark) In the rapidly evolving world of e-commerce, visual search has become a game-changing technology. Poshmark, a leading fashion resale marketplace, has developed Posh Lens – an advanced visual search engine that revolutionizes how shoppers discover and purchase items. Under the hood of Posh Lens lies Milvus, a vector database enabling efficient product search and recommendation across our vast catalog of over 150 million items. However, with such an extensive and growing dataset, maintaining high-performance search capabilities while scaling AI infrastructure presents significant challenges. In this talk, Mahesh Pasupuleti shares: - The architecture and strategies to scale Milvus effectively within the Posh Lens infrastructure - Key considerations include optimizing vector indexing, managing data partitioning, and ensuring query efficiency amidst large-scale data growth - Distributed computing principles and advanced indexing techniques to handle the complexity of Poshmark's diverse product catalog

Alluxio Webinar | Optimize, Don't Overspend: Data Caching Strategy for AI Wor...Alluxio, Inc.

Alluxio Webinar Sept. 10, 2024 For more Alluxio Events: https://www.alluxio.io/events/ Speaker: - Jingwen Ouyang (Senior Program Manager, Alluxio) As machine learning and deep learning models grow in complexity, AI platform engineers and ML engineers face significant challenges with slow data loading and GPU utilization, often leading to costly investments in high-performance computing (HPC) storage. However, this approach can result in overspending without addressing the core issues of data bottlenecks and infrastructure complexity. A better approach is adding a data caching layer between compute and storage, like Alluxio, which offers a cost-effective alternative through its innovative data caching strategy. In this webinar, Jingwen will explore how Alluxio's caching solutions optimize AI workloads for performance, user experience and cost-effectiveness. What you will learn: - The I/O bottlenecks that slow down data loading in model training - How Alluxio's data caching strategy optimizes I/O performance for training and GPU utilization, and significantly reduces cloud API costs - The architecture and key capabilities of Alluxio - Using Rapid Alluxio Deployer to install Alluxio and run benchmarks in AWS in just 30 minutes

AI/ML Infra Meetup | Maximizing GPU Efficiency : Optimizing Model Training wi...Alluxio, Inc.

AI/ML Infra Meetup Aug. 29, 2024 Organized by Alluxio For more Alluxio Events: https://www.alluxio.io/events/ Speaker: - Bin Fan (VP of Technology, Founding Engineer @OpenAI) In the rapidly evolving landscape of AI and machine learning, infra teams face critical challenges in managing large-scale data for AI. Performance bottlenecks, cost inefficiencies, and management complexities pose significant challenges for AI platform teams supporting large-scale model training and serving. In this talk, Bin Fan will discuss the challenges of I/O stalls that lead to suboptimal GPU utilization during model training. He will present a reference architecture for running PyTorch jobs with Alluxio in cloud environments, demonstrating how this approach can significantly enhance GPU efficiency. What you will learn: - How to identify GPU utilization and I/O-related performance bottlenecks in model training - Leverage GPU anywhere to maximize resource utilization - Best practices for monitoring and optimizing GPU usage across training and serving pipelines - Strategies for reducing cloud costs and simplifying management of AI infrastructure at scale