Automate ML: From debugging DL to

detecting model drift in production

Aparna Elangovan
Prototyping Engineer, AI/ML
Amazon Web Services

© 2020, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Amazon SageMaker

A fully managed service that covers the entire machine learning (ML) workflow

Fully
Data labeling One-click
Hyperparameter managed
and fully Built-in, high- training on
tuning and One-click hosting with
managed performance the highest
model deployment automatic
notebook algorithms performing
compilation scaling and
instances infrastructure
Elastic Inference

© 2020, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Amazon SageMaker: re:Invent 2019 announcements

Amazon SageMaker Amazon SageMaker

Amazon SageMaker Model Monitor
Debugger
Studio

Amazon SageMaker Amazon SageMaker Amazon SageMaker

Notebooks Experiments Autopilot

© 2020, Amazon Web Services, Inc. or its affiliates. All rights reserved.
 Amazon SageMaker: Focus in this session

Amazon SageMaker Amazon SageMaker

Amazon SageMaker Studio Model Monitor
Debugger

Amazon SageMaker Amazon SageMaker Amazon SageMaker

Notebooks Experiments Autopilot

© 2020, Amazon Web Services, Inc. or its affiliates. All rights reserved.
© 2020, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Example issues while training ML models
• Vanishing gradients
• Exploding gradients
• Loss not decreasing across steps
• Weight update ratios that are either too small or too large
• Tensor values that are all zeros

All these issues impact the learning process

Debugging them is hard and is even harder when running distributed training

© 2020, Amazon Web Services, Inc. or its affiliates. All rights reserved.
How does Amazon SageMaker Debugger work?
Amazon SageMaker

Action → Stop
Amazon the training
CloudWatch event

Action → Analyze
Training Analysis using Debugger SDK
in progress in progress Amazon
SageMaker notebook

Action → Visualize
Customer’s Amazon S3 bucket tensors using charts
Amazon SageMaker
Studio visualization

• No code change is necessary to emit debug data with built-in algorithms and custom training script
• Analysis occurs in real time as data is emitted, making real-time alerts possible

© 2020, Amazon Web Services, Inc. or its affiliates. All rights reserved.
An example: Vanishing gradients
Input Hidden1 Hidden2 Output Weight update rule
𝑤11,1
𝑤1 𝑤21,1
1,2
Intuition
𝑤31,1 Gradients vanish when they assume a very
small value → almost no weight update
during backpropagation
Why this happens? An example
𝑤31,2
𝑤21,𝑘
Sigmoid
activation
𝑤11,𝑛

Can be small

© 2020, Amazon Web Services, Inc. or its affiliates. All rights reserved.
© 2020, Amazon Web Services, Inc. or its affiliates. All rights reserved.
How does Amazon SageMaker Debugger work?
Amazon SageMaker

Action → Stop
Amazon the training
CloudWatch event

Action → Analyze
Training Analysis using Debugger SDK
in progress in progress Amazon
SageMaker notebook

Action → Visualize
Customer’s Amazon S3 bucket tensors using charts
Amazon SageMaker
Studio visualization

• No code change is necessary to emit debug data with built-in algorithms and custom training script
• Analysis occurs in real time as data is emitted, making real-time alerts possible

© 2020, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Amazon SageMaker Debugger built-in rules
TensorFlow, Apache MXNet, TensorFlow, MXNet,
and PyTorch PyTorch, and XGBoost

DeadRelu Overfit

VanishingGradient ClassImbalance

ExplodingTensor LossNotDecreasing

Many others …

© 2020, Amazon Web Services, Inc. or its affiliates. All rights reserved.
© 2020, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Model drift

© 2020, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Model drift: Example sentiment analysis

TYVM

© 2020, Amazon Web Services, Inc. or its affiliates. All rights reserved.
 How Model Monitor works Amazon SageMaker
endpoint

Training data Amazon SageMaker Model Applications

training job 1. Capture
endpoint data

2. Baseline 3. Scheduled Requests,

processing job monitoring job predictions

Baseline statistics
and constraints Results:
Amazon
Notifications statistics
CloudWatch
and violations
metrics
• Model updates
• Training data updates
• Retraining
Analysis of
results
© 2020, Amazon Web Services, Inc. or its affiliates. All rights reserved.
© 2020, Amazon Web Services, Inc. or its affiliates. All rights reserved.
 How Model Monitor works Amazon SageMaker
endpoint

Training data Amazon SageMaker Model Applications

training job 1. Capture
endpoint data

2. Baseline 3. Scheduled Requests,

processing job monitoring job predictions

Baseline statistics
and constraints Results:
Amazon
Notifications statistics
CloudWatch
and violations
metrics
• Model updates
• Training data updates
• Retraining
Analysis of
results
© 2020, Amazon Web Services, Inc. or its affiliates. All rights reserved.
© 2020, Amazon Web Services, Inc. or its affiliates. All rights reserved.
 How Model Monitor works Amazon SageMaker
endpoint

Training data Amazon SageMaker Model Applications

training job 1. Capture
endpoint data

2. Baseline 3. Scheduled Requests,

processing job monitoring job predictions

Baseline statistics
and constraints Results:
Amazon
Notifications statistics
CloudWatch
and violations
metrics
• Model updates
• Training data updates
• Retraining
Analysis of
results
© 2020, Amazon Web Services, Inc. or its affiliates. All rights reserved.
© 2020, Amazon Web Services, Inc. or its affiliates. All rights reserved.
 How Model Monitor works Amazon SageMaker
endpoint

Training data Amazon SageMaker Model Applications

training job 1. Capture
endpoint data

2. Baseline 3. Scheduled Requests,

processing job monitoring job predictions

Baseline statistics
and constraints Results:
Amazon
Notifications statistics
CloudWatch
and violations
metrics
• Model updates
• Training data updates
• Retraining
Analysis of
results
© 2020, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Summary

Amazon SageMaker Amazon SageMaker

Amazon SageMaker Model Monitor
Debugger
Studio

Amazon SageMaker Amazon SageMaker Amazon SageMaker

Notebooks Experiments Autopilot

© 2020, Amazon Web Services, Inc. or its affiliates. All rights reserved.
AWS machine learning training and certification
Learn like an Amazonian, based on the curriculum we’ve used to train our own developers and data scientists

ML training for Flexibility to learn Validate your

the whole team your way expertise
Explore tailored Learn online with Demonstrate expertise in
learning paths for the 65+ courses or live with building, training, tuning,
following roles: instructor-led training, and deploying ML models
business decision plus hands-on labs and with an industry-recognized
makers, data scientists, opportunities for credential: AWS Certified
developers, and data practical application Machine Learning –
platform engineers Specialty

aws.training/machinelearning
© 2020, Amazon Web Services, Inc. or its affiliates. All rights reserved.
 Thank you for attending AWS Innovate
We hope you found it interesting! Please complete the survey.
Let us know what you thought of today’s event and how we can improve
the event experience for you in the future.

[email protected]
twitter.com/AWSCloud

facebook.com/AmazonWebServices
youtube.com/user/AmazonWebServices

slideshare.net/AmazonWebServices
twitch.tv/aws

© 2020, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Thank you!

© 2020, Amazon Web Services, Inc. or its affiliates. All rights reserved.