UPDATE THIS PRESENTATION HEADER IN SLIDE MASTER

Implement MLOps
practices on AWS

Iván Castro, Data Scientist Consultant (he/him)

María Cortes, Data Scientist Consultant (she/her)
Professional Services
Amazon Web Services

© 2023, Amazon Web

© 2023,
Services,
Amazon
Inc. or
Webits Services,
affiliates.Inc.
All or
rights
its affiliates.
reserved.All rights reserved.
UPDATE THIS PRESENTATION HEADER IN SLIDE MASTER

Amazon SageMaker
MLOps Workshop
tinyurl.com/mb96dhtp

© 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved. 2
Agenda
UPDATE THIS PRESENTATION HEADER IN SLIDE MASTER

What is MLOps
People, Processes, Technology

MLOps Foundation Roadmap

MLOps Maturity: Initial, Repeatable, Reliable, Scalable

MLOps Delivery Journey

How to shape the team and the engagement

© 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved. 3
ML and Path to Production
UPDATE THIS PRESENTATION HEADER IN SLIDE MASTER

MACHINE LEARNING CODE AND DATA SCIENCE NOTEBOOKS…

ML
Code

© 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved. 4
ML and Path to Production
UPDATE THIS PRESENTATION HEADER IN SLIDE MASTER

…IS ONE SMALL PART OF THE OVERALL DEPLOYMENT PICTURE

Machine Monitoring
Data Verification Resource
Management

Configuration
Data Collection
Serving
ML Analysis Tool Infrastructure
Code

Feature Process
Extraction Management Tools

“Only a small fraction of real-world ML systems is composed of the ML code”

source: Hidden Technical Debt in Machine Learning Systems [D. Sculley, & al.] – 2015
https://papers.nips.cc/paper/5656-hidden-technical-debt-in-machine-learning-systems.pdf
© 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved. 5
Customer Challenges, ML Lifecycle and Personas
UPDATE THIS PRESENTATION HEADER IN SLIDE MASTER

CONSIDERATIONS & CHALLENGES LEAD TO ML AND OPERATIONS (MLOPS)

Processes

• Culture
• Lack of cross-functional teams
• Priorities & needs (personas)
• Organizational structure
• Skillsets
• Unique aspects of ML lifecycle

What is MLOps: The combination of People, Processes, and Technology to productionize ML solutions efficiently
© 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved. 6
 UPDATE THIS PRESENTATION HEADER IN SLIDE MASTER

Why MLOps?
Expected Business Value

© 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved. 7
MLOps Foundation Expected Outcomes
UPDATE THIS PRESENTATION HEADER IN SLIDE MASTER

STANDARDIZE OPERATIONS AND INFRASTRUCTURE FOR YOUR DATA SCIENCE

Business Goal Technical Metric Now Expected Outcomes Business Value

Time to value up to Improve Speed-to-Value

1 Be more efficient in delivery < 3 months
(from research to production) 12 months by 4x
Time to productionize existing ML use Reduce FTE overhead
2 Simplify route-to-live 3-6 months < 2 weeks
cases in average 8x
Focus on innovation
Standardize infrastructure, data,
3 % Template driven development n/a > 85% increasing re-usability
& code
by 85%
Standardize onboarding of new Time to instantiate a new MLOps Accelerate ML adoption
4 40 days < 1 hours
teams and ML use cases infrastructure & ML projects across all business areas
Execute the ML solutions without Your data is safe in your
5 Ensure high security standards n/a No internet
internet access in a private cloud private cloud
Reduce CO2 emissions using managed 75% greener environment
6 Enable Sustainability n/a 75%
services with less cost

Reduce platform, people and operation costs

Customer references building MLOps foundation and business benefits:

• NatWest: https://aws.amazon.com/blogs/machine-learning/part-1-how-natwest-group-built-a-scalable-secure-and-sustainable-mlops-platform/
• BP: https://aws.amazon.com/solutions/case-studies/bp-machine-learning-case-study/

© 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved. 8
UPDATE THIS PRESENTATION HEADER IN SLIDE MASTER

How to Mature on MLOps?

© 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved. 9
MLOps Maturity Model
UPDATE THIS PRESENTATION HEADER IN SLIDE MASTER

Model
Models in Production

g Scalable
peratin
O
Reliable
Templatize and
Repeatable Productionize
Introduce Testing, Multiple ML
Monitoring, and Multi- Solutions
Standardize Code account Deployment
Initial Repositories & ML
Solution Deployment

Establish the
Experimentation
Environment

MLOps Maturity

© 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved. 10
UPDATE THIS PRESENTATION HEADER IN SLIDE MASTER

MLOps Foundation Roadmap

© 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved. 11
MLOPs Initial Phase
UPDATE THIS PRESENTATION HEADER IN SLIDE MASTER

ML EXPERIMENTATION ON AWS USING AMAZON SAGEMAKER STUDIO NOTEBOOKS

© 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved. 12
Amazon SageMaker Studio
UPDATE THIS PRESENTATION HEADER IN SLIDE MASTER

ML EXPERIMENTATION ON AWS USING AMAZON SAGEMAKER STUDIO NOTEBOOKS

© 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved. 13
ML Solution Lifecycle Automation
UPDATE THIS PRESENTATION HEADER IN SLIDE MASTER

CREATING AUTOMATED WORKFLOWS ON AWS – ML PIPELINES

Container
Managed Service
Etc.

Example model training state

Example model training state
machine
machine and common steps
SageMaker Pipelines

© 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved. 14
Amazon SageMaker Processing
UPDATE THIS PRESENTATION HEADER IN SLIDE MASTER

PREPROCESS DATA & OFFLOAD DATA SCIENTISTS ON MANAGING CONTAINERS

Container

Code
Inputs
Outputs

Supported pre-built processors: +

Custom Containers
Framework Processors

© 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved. 15
Amazon SageMaker Pipelines
UPDATE THIS PRESENTATION HEADER IN SLIDE MASTER

AUTOMATED WORKFLOWS TO PRE-PROCESS, TRAIN, EVALUATION, & REGISTER MODELS

Step Types
• Processing
• Training
• Tuning
• CreateModel
• RegisterModel
• Transform
• Condition
• Callback
• Lambda Function
• ClarifyCheck
• QualityCheck
• Amazon EMR
• Fail

© 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved. 16
Amazon SageMaker Model Registry
UPDATE THIS PRESENTATION HEADER IN SLIDE MASTER

Create model groups in Benchmark and observe the model Track the activities, metrics, and
your model registry versions in your model group and settings per model version
promote versions of the model by
changing their status

© 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved. 17
Standardize Repository Structure
UPDATE THIS PRESENTATION HEADER IN SLIDE MASTER

A STANDARDIZE REPO STRUCTURE WILL HELP TO AUTOMATE DEPLOYMENT AND TESTING OF THE ML
WORKFLOWS.
algorithms/
shared_libraries/
<help_functions1>.py
<help_functions2>.py
preprocessing/ # 1 folder per pre-processing job, order is defined in the ml pipeline logic
<preprocessing_job_name> # e.g computer vision: image rotation
test/
input/ # (optional)
output/ # (optional)
test.py
__main__.py
requirements.txt # define library requirements
image_configuration.json # define docker container image configuration in case of custom containers
training/ # (optional) each one is a training job in SageMaker
<training_job_name>/
__main__.py
requirements.txt
postprocessing/ # each one is a processing job in SageMaker
<postprocessing_job_name>/
__main__.py
requirements.txt
inference/ # (optional) for batch inference
<batch_inference_job_name>/ # one job per training job name if we're building multiple models
__main__.py
requirements.txt
mlpipelines/
training/
ml-pipeline-training.py # Define training ML pipelines using SageMaker Pipeline SDK or Step Functions or other orchestrations
input.json # (optional) ML pipeline configuration to enable reusability
inference/
batch/
ml-pipeline-inference.py # Define batch inference ML pipelines using SageMaker Pipeline SDK or Step Functions or other orchestrations
realtime/
endpoint_config.json
ml-pipeline-orcherstrator/ # define which ML pipeline will be deployed in each account
dev.json
preprod.json
prod.json
notebooks/
*.ipynb # the original notebooks as has been created by the data scientists
© 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved. Repository structure example 18
Data Structure and Versioning of ML pipelines
UPDATE THIS PRESENTATION HEADER IN SLIDE MASTER

Inputs & Outputs

STANDARDIZE DATA STORAGE AND VERSIONING BASED ON CI/CD PIPELINE ACTIVITIES

Data structure example

© 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved. 19
MLOPs Initial Phase
UPDATE THIS PRESENTATION HEADER IN SLIDE MASTER

ML EXPERIMENTATION ON AWS USING AMAZON SAGEMAKER STUDIO NOTEBOOKS

© 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved. 20
MLOPs Repeatable Phase
UPDATE THIS PRESENTATION HEADER IN SLIDE MASTER

FROM RESEARCH NOTEBOOKS TO ML PIPELINES & AUTOMATION

© 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved. 21
MLOPs Reliable Phase 1/2
UPDATE THIS PRESENTATION HEADER IN SLIDE MASTER

UNDERSTAND THE NEEDS OF MLOPS & INTRODUCE AUTOMATED TESTING

© 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved. 22
MLOPs Automatic Testing
UPDATE THIS PRESENTATION HEADER IN SLIDE MASTER

UNDERSTAND THE NEEDS OF MLOPS & INTRODUCE AUTOMATED TESTING

Model Build and Deployment Testing Production Testing

Unit test Integration test Shadow test A/B test

Static code, Identify bugs, Data, Modelling, Orchestration, Validate model versions in Validate model versions in
Standardize code Syntax errors 3rd Party tools integration production without returning its production serving both
outputs to users outputs to users

Stress/Load test End-to-end test

Deployments guardrails Data and Model Monitoring
Latency and throughput Smoke test for End-to-End
Securely update models in Continuously monitor for
testing of model functionality
performance production with quick data and model
rollback functionality performance drifts

© 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved. 23
MLOPs Reliable Phase 2/2
UPDATE THIS PRESENTATION HEADER IN SLIDE MASTER

BEST PRACTICE ON MLOPS, ROBUST AND SECURE MODELS

© 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved. 24
How SageMaker Model Deployment Works
UPDATE THIS PRESENTATION HEADER IN SLIDE MASTER

EXAMPLE OF REAL-TIME INFERENCE WITH SAGEMAKER

1 2 3
Prepare your ML model Configure endpoint Real-time HTTP requests

Model artifacts Prepared ML model

Client Application
Container image Deployment mode

Inference Inference
input Compute / GPU
IAM role request result
selection

Model serving stack

SageMaker Endpoint
secure, elastic, managed
Autoscaling policy

Bring your own models, containers, and algorithms; or use ones provided by AWS. Example above illustrates “Real-time Inference”.

© 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved. 25
Amazon SageMaker Model Monitor
UPDATE THIS PRESENTATION HEADER IN SLIDE MASTER

AUTOMATICALLY DETECT DATA AND MODEL QUALITY DRIFTS

Inference Results
Input data

predictions
input features input
(single row) features
Historical data Inference Results
Bucket
Prediction
ut s
n p
i ure target Probability (0-100%)
t
Historical Data fea Training Job
Model
SageMaker
Bucket
Endpoint

Data Drift
Detection
Training
SageMaker features SageMaker
Monitor Baseline Statistics Monitor
(.json) Model Metadata Scheduled Job
Bucket

Model Drift
Detection
Training
SageMaker features + target SageMaker
Monitor Baseline Monitor
Statistics (.json) Scheduled Job

https://github.com/aws/amazon-sagemaker-examples/tree/main/sagemaker_model_monitor
© 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved. 26
MLOPs Reliable Phase 2/2
UPDATE THIS PRESENTATION HEADER IN SLIDE MASTER

BEST PRACTICE ON MLOPS, ROBUST AND SECURE MODELS

© 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved. 27
MLOPs Scalable Phase
UPDATE THIS PRESENTATION HEADER IN SLIDE MASTER

MULTIPLE TEAMS AND ML USE CASES ADOPT MLOPS

© 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved. 28
MLOps Key Personas and Roles
UPDATE THIS PRESENTATION HEADER IN SLIDE MASTER

Platform Team
Secure Cloud/Data/ML Platform
Advance Analytics Team Data Science Team Business
Data Lake Experimentation & MLOps Viz Dashboards, ML Adoption, & ROI

MLOps Engineer/ ML Admin

Standardize CI/CD, user/service role & container
creation, model consumption, testing and deployment
Data Engineer Data Scientist methodology based on business requirements Business Stakeholder
Prepare & Ingest data Create and Maintain the
building ETL pipelines best ML models to solve
Product Owners
for the ML use cases business problems Define business problem, ML use case
business KPIs, interact with other BUs,
and make business decisions
Security
Assess data, user, and service access
creating policies and guardrails

ML Engineer Business Stakeholder

Data Owners Collaborate with DS to
Manage data sharing productionize their code Architects/ SysOps Engineer ML Consumers
and provide access to and models following
Standardize account infrastructure e.g. Consumers of ML results from other
other teams development best
VPC and endpoints, subnets, security BUs, driving business decision making
practices
groups, connectivity, user roles
implementation
© 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved. 30
UPDATE THIS PRESENTATION HEADER IN SLIDE MASTER

Standardize MLOPs Delivery

© 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved. 34
Example Timeline to deliver MLOPs Foundation
UPDATE THIS PRESENTATION HEADER IN SLIDE MASTER

TIMELINE MIGHT VARY DEPENDING ON CUSTOMER SECURITY NEED AND MANUAL PROCESSES

Education & Knowledge Transfer

Sprint 1
Training on AWS
KT KT KT
& ML Use Case
Deep Dive

MLOp Foundation
Sprint 1 Sprint 2 Sprint 3 Sprint 4 Sprint 5 Sprint 6
Establish Code Repositories Multi-account Parameterization Templatization Testing
Experimentation & Standardization Deployment & Promotion to
Env. Pipelines Prod

Initial Repeatable Reliable Scale

Example ML Use Case

Development of your Model Model Monitoring & Alerting
ML Use Cases Model Explainability

Week 1 2 3 4 5 6 7 8 9 10 11 12

© 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved. 35
The Partner Delivery Team
UPDATE THIS PRESENTATION HEADER IN SLIDE MASTER

TEAM MIGHT VARY DEPENDING ON THE COMPLEXITY

DevOps MLOps engineer Cloud infrastructure architect

(on demand)
Who will assist the implementation Who will assist on the Who will work on the foundation of
of CI/CD pipelines and automated productionization of the ML solution the cloud infrastructure
testing suggesting best practices

Lead data scientist

Who will assist on shaping the
operating model based on your needs

© 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved. 36
The Customer Delivery Team
UPDATE THIS PRESENTATION HEADER IN SLIDE MASTER

TEAM MIGHT VARY DEPENDING ON THE COMPLEXITY

Business owners/Product Owner MLOps engineer Enterprise architect - IT

Who has the overall architecture and
Who own the ML Factory, set success Who will be responsible for the integrations and knows where the
criteria, and assess the outcomes productionization of the ML solution data and service are in order to be
able to integrate

Lead data scientist or MLOps engineer

Who will be responsible approving the
productionization operating model
*might be the same person
for data scientists

© 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved. 37
Team sizing example: ML Use Cases
UPDATE THIS PRESENTATION HEADER IN SLIDE MASTER

T-SHIRT SIZES: GROUPING ML USE CASES BASED ON THEIR COMPLEXITY

Personas Phases Tasks S M L XL Technical Outcome

1. Exploratory Data
Analysis (EDA) 1. Code for exploratory data
2. Baseline analysis, modeling, and
Modeling 3. Model Iteration metrics
Data Scientist
4. Finalization 2. Business KPIs
Who will assist the Up to 80 Up to 120 Up to160 Up to 220
implementation of 5. Knowledge 3. Solution Documentation
the ML models on Transfer (KT) person-days person-days person-days person-days
the ML platform

1. Refactor Known Exploratory Innovative Innovative

1. Productionized ML solution
2. Architect Modeling Modeling Modeling Project Modeling Project 2. Refactored/Cleaned code
3. Implement Project Project with Complexity 3. Architecture diagram(s)
Deployment 4. Abstraction
4. Testing code
ML Engineer 5. Test
Who will assist on 6. Knowledge
the Transfer (KT)
productionization of
the ML solution

*Up to 20% flexibility

© 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved. 39
UPDATE THIS PRESENTATION HEADER IN SLIDE MASTER

MLOps Resources

© 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved. 43
Public Resources
UPDATE THIS PRESENTATION HEADER IN SLIDE MASTER

MLOps
https://aws.amazon.com/sagemaker/mlops/

Amazon SageMaker documentation

https://docs.aws.amazon.com/sagemaker/index.html

MLOps Foundation Roadmap for enterprises

https://aws.amazon.com/blogs/machine-learning/mlops-foundation-roadmap-for-enterprises-with-amazon-sagemaker/

Multi-account MLOPs Platform Source Code Examples

https://github.com/aws-samples/sagemaker-custom-project-templates/tree/main/mlops-multi-account-cdk

How NatWest Group built a scalable, secure, and sustainable MLOps platform
https://aws.amazon.com/blogs/machine-learning/part-1-how-natwest-group-built-a-scalable-secure-and-sustainable-mlops-platform/

BP Scales Its Data Science Machine Learning Operations on AWS

https://aws.amazon.com/solutions/case-studies/bp-machine-learning-case-study/

© 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved. 44
UPDATE THIS PRESENTATION HEADER IN SLIDE MASTER

Amazon SageMaker
MLOps Workshop

© 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved. 45
Workshop: Use case
UPDATE THIS PRESENTATION HEADER IN SLIDE MASTER

Bank Telemarketing
• Direct marketing campaigns (phone calls) in a Portuguese banking institution
• Predict if a user will subscribe to a product (bank term deposit) before making a call.

Amazon SageMaker Amazon SageMaker Amazon SageMaker

Studio Processing Jobs Training Jobs

Amazon SageMaker Amazon SageMaker

Pipelines Experiments

© 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved. 46
UPDATE THIS PRESENTATION HEADER IN SLIDE MASTER

Amazon SageMaker
MLOps Workshop
tinyurl.com/mb96dhtp

© 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved. 47
UPDATE THIS PRESENTATION HEADER IN SLIDE MASTER

Thank you!
Iván Castro María Cortés
[email protected] [email protected]

© 2023, Amazon Web

© 2023,
Services,
Amazon
Inc. or
Webits Services,
affiliates.Inc.
All or
rights
its affiliates.
reserved.All rights reserved.