Gartner Catalyst Conference

12 – 15 August 2019 / San Diego, CA

Don’t Stumble at the Last

Mile: Leveraging MLOps
and DataOps to
Operationalize ML and AI
Sumit Pal

© 2019 Gartner, Inc. and/or its affiliates. All rights reserved. Gartner is a registered trademark of Gartner, Inc. and its affiliates. This publication may not be reproduced or distributed in any form
without Gartner’s prior written permission. It consists of the opinions of Gartner’s research organization, which should not be construed as statements of fact. While the information contained in this
publication has been obtained from sources believed to be reliable, Gartner disclaims all warranties as to the accuracy, completeness or adequacy of such information. Although Gartner research
may address legal and financial issues, Gartner does not provide legal or investment advice and its research should not be construed or used as such. Your access and use of this publication are
governed by Gartner’s Usage Policy. Gartner prides itself on its reputation for independence and objectivity. Its research is produced independently by its research organization without input or
influence from any third party. For further information, see “Guiding Principles on Independence and Objectivity.”
1 © 2019 Gartner, Inc. and/or its affiliates. All rights reserved. Gartner is a registered trademark of Gartner, Inc. and its affiliates.
Don’t Be — Bankrupt in 45 Minutes
Knightmare: A DevOps Cautionary Tale

Knight Capital Group $460M loss in 45 minutes

DevOps, DataOps and MLOps Problem

Systems were NOT setup for the risk they were exposed to.

Processes were inherently prone to error.

Deployment process relied on Humans

2 © 2019 Gartner, Inc. and/or its affiliates. All rights reserved. Gartner is a registered trademark of Gartner, Inc. and its affiliates.
 By 2021, at least 50% of machine
learning projects will not be fully
deployed.

3 © 2019 Gartner, Inc. and/or its affiliates. All rights reserved. Gartner is a registered trademark of Gartner, Inc. and its affiliates.
Gartner Client Question

Why would 50% ML and data science solutions

not be deployed successfully?

ML is not Easy
End result models are built that aren’t being turned into revenue-
generating products and services
• Bootcamps/Courses are great for learning how to build and train models,
• Don’t teach how to take them to the next step.

4 © 2019 Gartner, Inc. and/or its affiliates. All rights reserved. Gartner is a registered trademark of Gartner, Inc. and its affiliates.
Agenda

ML Workflow ML Pipeline Why ML Is Difficult ML Missing Pieces

Problems and
Solutions When Research
Deploying ML Tools Recommendations Recommendations

5 © 2019 Gartner, Inc. and/or its affiliates. All rights reserved. Gartner is a registered trademark of Gartner, Inc. and its affiliates.
ML Workflow

1
Business 2 Data
Understanding Understanding

3 Data
6 Preparation
Deployment

Data 4
Modeling

5
Evaluation

6 © 2019 Gartner, Inc. and/or its affiliates. All rights reserved. Gartner is a registered trademark of Gartner, Inc. and its affiliates.
7 © 2019 Gartner, Inc. and/or its affiliates. All rights reserved. Gartner is a registered trademark of Gartner, Inc. and its affiliates.
Machine Learning Workflow
Build
Problem Data Data
Statement Collection
EDA Engineering DataOps

Train
Model Model Model
Training Evaluation Tuning

MLOps
Deploy
Model Model
Deployment Monitoring

8 © 2019 Gartner, Inc. and/or its affiliates. All rights reserved. Gartner is a registered trademark of Gartner, Inc. and its affiliates.
ML Pipeline
Data Processing
(Feature Engineering)

Processing Engine

Transformation Normalization Cleaning and

Encoding
Execution Deployment
Data Ingestion

ERP
Databases

Preprocessing Sample Training/

Stream Data Selection Testing Set
Processing
Platform

Experimentation Testing Tuning

Mainframe
Model Engineering
Batch Data
Warehouse Machine Algorithms
IoT
Devices Data Storage

Clustering Algorithm Learning Algorithm

Execution

9 © 2019 Gartner, Inc. and/or its affiliates. All rights reserved. Gartner is a registered trademark of Gartner, Inc. and its affiliates.
Why ML Is Difficult

• “Lack of clear abstraction barriers.”

• Debugging is harder.
• “Non-modularity”— if you change anything, you end up changing
everything.
• “Non-stationarity”— the need to account for new data.
• “What is produced is a black box — you can peek in a little bit, we have
some idea of what’s going on, but not a complete idea.”
• Reproducibility is extremely difficult.

10 © 2019 Gartner, Inc. and/or its affiliates. All rights reserved. Gartner is a registered trademark of Gartner, Inc. and its affiliates.
Why ML Is Difficult

• Scaling the model-training and serving process. How can we reliably

and repeatedly take our models from our laptop to production?
• Keeping track of multiple experiments with different hyper-parameters.
• Reproducing the results and retraining models in a predictive manner.
• Keeping track of different models and their model performance
over time (i.e. model drift).
• Dynamically retraining models with new data and rollback models.

11 © 2019 Gartner, Inc. and/or its affiliates. All rights reserved. Gartner is a registered trademark of Gartner, Inc. and its affiliates.
Software Development and ML Development

Software Development ML Development

12 © 2019 Gartner, Inc. and/or its affiliates. All rights reserved. Gartner is a registered trademark of Gartner, Inc. and its affiliates.
ML Missing Pieces
Data Processing
(Feature Engineering)

Processing Engine Code and

Packaging Control

Transformation Normalization Cleaning and

Encoding
Execution Deployment
Data Ingestion

ERP
Databases

Preprocessing Sample Training/

Stream Data Selection Testing Set
Processing
Platform

Experimentation Testing Tuning

Mainframe
Model Engineering
Batch Data
Warehouse Machine Algorithms
IoT
Devices Data Storage

Performance
Clustering Algorithm Learning Algorithm
Execution
Management

Feedback Loop
Model
Data Versioning
13 Management
© 2019 Gartner, Inc. and/or its affiliates. All rights reserved. Gartner is a registered trademark of Gartner, Inc. and its affiliates.
Properties of DS / ML System
Reproducible builds

Ability to run the entire stack locally for development

Local, Continuous Integration/Test (CI/T), Staging, Production environments

identical

Production data (inputs or outputs) is versioned and queryable later on

Trace production data through the system

14 © 2019 Gartner, Inc. and/or its affiliates. All rights reserved. Gartner is a registered trademark of Gartner, Inc. and its affiliates.
Problem 1 — Works on My Machine

15 © 2019 Gartner, Inc. and/or its affiliates. All rights reserved. Gartner is a registered trademark of Gartner, Inc. and its affiliates.
How to Solve — Problem 1

Track Code

Track Environment

Packaging

Consistent Environment and Consistent Packaging

16 © 2019 Gartner, Inc. and/or its affiliates. All rights reserved. Gartner is a registered trademark of Gartner, Inc. and its affiliates.
Solution 1 — Works on My Machine
Run
Data Processing
(Feature Engineering) Tests

Processing Engine GIT CI/CD Dockerize

Docker Registry
Transformation Normalization Cleaning and
Encoding
Execution Deployment
Data Ingestion

ERP
Databases

Preprocessing Sample Training/

Stream Data Selection Testing Set
Processing
Platform

Experimentation Testing Tuning

Mainframe
Model Engineering
Batch Data
Warehouse Machine Algorithms
IoT
Devices Data Storage

Clustering Algorithm Learning Algorithm

Execution

17 © 2019 Gartner, Inc. and/or its affiliates. All rights reserved. Gartner is a registered trademark of Gartner, Inc. and its affiliates.
Problem 2 — What Happens When a Model Is
Deployed and It Doesn’t Work

Who Tracks Models ?

Who Keeps Tracks of All Experimentations You Do ?

Can You Search Your Models ?

Can You Reproduce Your Models ?

18 © 2019 Gartner, Inc. and/or its affiliates. All rights reserved. Gartner is a registered trademark of Gartner, Inc. and its affiliates.
How to Solve — Problem 2

Model Storage — Model Repository — Track, Store, Index,

Searchable Models — Model Versioning

Collaborative Environments for Model Development —

Integrated to the Model Storage

Most recent model— Which Environment Was the Most Model

Trained and Developed

Track Hyperparameters — How do you keep track of hyperparameters

which were experimented with

19 © 2019 Gartner, Inc. and/or its affiliates. All rights reserved. Gartner is a registered trademark of Gartner, Inc. and its affiliates.
Model Variables

Model Has 2 Components

Model Artifacts — Parameters Associated

Model Code + Model Image/Container
With the Model

Dockerize
Model Code Model Container
Image

Hyperparameters

20 © 2019 Gartner, Inc. and/or its affiliates. All rights reserved. Gartner is a registered trademark of Gartner, Inc. and its affiliates.
Solution 2 — Model Tracking Run
Tests

GIT CI/CD Dockerize

Docker Registry

Model Export

21 © 2019 Gartner, Inc. and/or its affiliates. All rights reserved. Gartner is a registered trademark of Gartner, Inc. and its affiliates.
Solution 2 — Model Tracking

22 © 2019 Gartner, Inc. and/or its affiliates. All rights reserved. Gartner is a registered trademark of Gartner, Inc. and its affiliates.
Solution 2 — Model Management - What
Who trained the model

Start and end time of the training job

Full model configuration (features used, hyper-parameter values, etc.)

Reference to training and test data sets

Distribution and relative importance of each feature

Full learned parameters of the model

23 © 2019 Gartner, Inc. and/or its affiliates. All rights reserved. Gartner is a registered trademark of Gartner, Inc. and its affiliates.
Problem 3 — How to Replicate Model Behavior

Have you kept track of the data on which you trained/tested

and validated ?

Was that data versioned ?

Can you Reproduce your Models ?

24 © 2019 Gartner, Inc. and/or its affiliates. All rights reserved. Gartner is a registered trademark of Gartner, Inc. and its affiliates.
How to Solve — Problem 3

Need a Data Versioning System

Version All Your Data Which You Used for Training

Version — Training Data

Version — Validation Data, Version — Testing Data

25 © 2019 Gartner, Inc. and/or its affiliates. All rights reserved. Gartner is a registered trademark of Gartner, Inc. and its affiliates.
Solution 3 — Version Data Run
Tests

GIT CI/CD Dockerize

Docker Registry

Model Export

Data Versioning

Training Validation Test

26 © 2019 Gartner, Inc. and/or its affiliates. All rights reserved. Gartner is a registered trademark of Gartner, Inc. and its affiliates.
Problem 4 — Drift (Data and Model)

Model Issues Post Deployment

What Doesn’t run Doesn’t run

Doesn’t Run
Happened? “good” enough “fast” enough

Data
Root Wrong Code / Wrong Model More
Data Drift Pipeline
Cause? Parameters Environment Drift? Compute
Issue
/
Wrong Data
/
Wrong Schema

27 © 2019 Gartner, Inc. and/or its affiliates. All rights reserved. Gartner is a registered trademark of Gartner, Inc. and its affiliates.
Problem 4 — Drift (Data and Model)

How do you know if your models are keeping up with data drift ?

How do you measure model and data drift ?

Do you know ⎯ .01% error thresholds can result in millions

of $ lost revenue ?

• What aspects of the model are important to watch ?

• What statistical/metrics identifies the quality of output of the models ?
• What is unacceptable output ?
• What is the threshold that defined an unacceptable output ?
• What is the course of action when the threshold is reached ?
• Who is responsible for taking action ?

28 © 2019 Gartner, Inc. and/or its affiliates. All rights reserved. Gartner is a registered trademark of Gartner, Inc. and its affiliates.
How to Solve — Problem 4

Need auditing, performance management system

• Define aspects of the model that are important to watch

• Continuous measurement of model accuracy, and compare with thresholds

Define unacceptable thresholds

Define actions to be taken when thresholds are reached

29 © 2019 Gartner, Inc. and/or its affiliates. All rights reserved. Gartner is a registered trademark of Gartner, Inc. and its affiliates.
Solution 4 — Model Performance Management

Docker Registry
Data and Feature
Engineering Pipeline REST APIs

Feature
Feature Vector
Storage
Real- Time Engineering Model
Data Pipeline

SQL

Model Registry

Alerts/Trigger new
Data Drift Container Continuous
Monitoring Monitoring
model re-build/
Registry Metadata
rollback
Hyperparameters Model Servicing

Model
Metrics Monitoring
30 © 2019 Gartner, Inc. and/or its affiliates. All rights reserved. Gartner is a registered trademark of Gartner, Inc. and its affiliates. Repository
31 © 2019 Gartner, Inc. and/or its affiliates. All rights reserved. Gartner is a registered trademark of Gartner, Inc. and its affiliates.
Orchestrate/Automate Solutions 1 to 4
CI/CT
Data

Code Dockerized Model Data

Build Test Dockerize Sampled Data
Repository Training Code Repository
Model
Training Validation dataset
Bug Fixes
Hyperparameter
Tuning
Code
Change Model
Repository
Dockerized Model
Detect Data Drift Deployment
Generate Model Endpoint

Data Model Log

Results
Inference Results

Continuous
Monitoring/Performance
Management

Detect Model Deviation

32 © 2019 Gartner, Inc. and/or its affiliates. All rights reserved. Gartner is a registered trademark of Gartner, Inc. and its affiliates.
Other Things to Measure

How much calendar time should be deployed to a model from staging

to production?

How much calendar time to add a new feature to the production model?

How long to take to do an end-to-end testing of models?

33 © 2019 Gartner, Inc. and/or its affiliates. All rights reserved. Gartner is a registered trademark of Gartner, Inc. and its affiliates.
New Problems With ML Models
Model Interpretability — Model Governance — GDPR

Model localization

Federated Learning – ML at the edge on devices

Audit Algorithms to ensure Fairness

34 © 2019 Gartner, Inc. and/or its affiliates. All rights reserved. Gartner is a registered trademark of Gartner, Inc. and its affiliates.
Tools
Problem 4 — Model
Problem 2 — Model Problem 3 — Data
Problem 1 — Code Version Serving/Perf
Management Versioning
Management/Ochestration
• Git • ModelDB • Palantir Foundry • Netflix (Meson)
• BitBucket • MLFlow • Databricks Delta Lake • Seldon
• Docker • Scared • DVC • PredictionIO
• Kubernetes • AWS System Manager • Workflow • Tensorflow Serving
Parameter Store • Pachyderm • Vertex.AI
• Google Subpar
Keras Tuner, Training (Airbnb)) • Mleap • Numericcal
• Facebook (XARs)
BigQueue, MLMD, Arbiter, • Quilt • Datatron
• TeamCity, Jenkins, GitLab
Aginity, Algorithmia, Anodot, • Immuta • Hydrosphere.io
• d6tStack (Pandas) Hydrosphere.io, ParallelM • GIT-LFS • Alteryx Promote
• d6tJoin, d6tFlow, d6tPipe Neptune, MLPerf, Iterative.ai • Oracle GraphPipe
Datamo, Google-Lucid,
Comet

SageMaker, Dataiku, Databricks, Determined AI, Uber (Michaelangelo), Airbnb (Bighead), Facebook (FBlearner Flow),
TensorFlow Extended, Polyaxon, dotData, Uber (Manifold), Hadoop (Submarine), Domino (Launchpad), MLAutomator,
DeepThought, Python (d6tflow)

35 © 2019 Gartner, Inc. and/or its affiliates. All rights reserved. Gartner is a registered trademark of Gartner, Inc. and its affiliates.
Practical Recommendations on How to Start …
Think operationalization — don’t just think about algorithms
and frameworks
Version control code/data/models
Establish CI/CD/CM
Canary Releases, automated deployments and testing frameworks
Capture Data Anomalies Early
Automate Data Validation
Data Errors Same Rigor as Code
Continuous Training

36 © 2019 Gartner, Inc. and/or its affiliates. All rights reserved. Gartner is a registered trademark of Gartner, Inc. and its affiliates.
Notebooks Bane or Boon
Notebooks promote - Bad Habits – Dump all files into one Directory
Notebooks - Code and Output gets mixed up
Notebooks - Don’t version control well

37 © 2019 Gartner, Inc. and/or its affiliates. All rights reserved. Gartner is a registered trademark of Gartner, Inc. and its affiliates.
Recommended Gartner Research

 A Guidance Framework for Operationalizing Machine Learning

for AI
Soyeb Barot (G00366587)
 Building a Framework for Managing Effective Machine Learning
Workloads
Sumit Agarwal (G00384678)
 Operationalizing Big Data Workloads
Sumit Pal (G00360371)

For information, please contact your Gartner representative.

38 © 2019 Gartner, Inc. and/or its affiliates. All rights reserved. Gartner is a registered trademark of Gartner, Inc. and its affiliates.