AUSOUG - Introducing New AI Ops Innovations in Oracle 19c Autonomous Health Framework

Copyright © 2018, Oracle and/or its affiliates. All rights reserved. |
Introducing New AI Ops
Innovations in Oracle
Autonomous Health Framework
Troy Anthony
Product Manager
Sandesh Rao
VP Autonomous Health and Machine Learning

Safe Harbor Statement
The following is intended to outline our general product direction. It is intended for
information purposes only, and may not be incorporated into any contract. It is not a
commitment to deliver any material, code, or functionality, and should not be relied upon
in making purchasing decisions. The development, release, timing, and pricing of any
features or functionality described for Oracle’s products may change and remains at the
sole discretion of Oracle Corporation.

Oracle 19c
Autonomous
Health
Framework
Powered by Applied Machine Learning
Cluster
Verification
Utility
ORAchk
Cluster
Health
Monitor
Cluster
Health
Advisor
Trace File
Analyzer
Hang
Manager
Memory
Guard
.Quality of
.Service
Management

Agenda
Introducing Applied Machine Learning for Operations
The Autonomous Health Cloud Platform
Applied Machine Learning for Real-time Prevention
Applied Machine Learning for Rapid Recovery
Summary and Q & A
1
2
3
4
5

Oracle’s Vision for Autonomous Database
• Self-Driving
–User defines service levels, database makes them happen
• Self-Securing
–Protection from both external attacks and malicious internal users
• Self-Repairing
–Automated protection from all downtime Autonomous
Database

PaaS Resource Lifecycle
Management
Bare-Metal thru Installation
Upgrade
Patching
Dependency Resolution
Prerequisites Resolution
Required Capabilities
Automatable
Scalable
Online (if possible)
PaaS Application Lifecycle
Management
Installation
Upgrade
Patching
Dependency Resolution
Prerequisites Resolution
Workload Profile Identification
Placement determination
SLA management
Automatable
Provider Interoperable
Cloud Operations Early Warning
Response System
Detect degradations and faults
Pinpoint root cause & component
Push warnings and alerts
Push targeted corrective actions
SLA – based resource management
Real-time Health Dashboard
Continuous and frequent
Autonomous Action Enabled
OSS Integration Enabled
Management Interoperable
Cloud Operations – 3 Strategic Pillars
Resource Lifecycle Management Database Lifecycle Management Database Autonomous Self-Repair

Database Operations Runtime Management
• Solving these challenges requires a holistic approach
– Prevent problems and optimize solutions in real-time
– Recover from failures and identify root cause quickly with minimal intervention
• Human reactions too late and do not scale
• Manual triage and floods of notifications do not scale
• Applied Machine learning techniques effectively respond in real-time and
without huge impact to operations
Prevention and Recovery Pillars

Applied ML in Oracle Cloud Autonomous Health
Oracle Support Services
Applied ML
Real-time Prevention Policies
Preventative
Actions
Manual Auto
Applied ML
Rapid Recovery
Repository
SRs
Corrective
Actions
Manual Auto
Real-time Prevention Rapid Recovery
Inputs
Bugs/SRs
Best
Practices
Metrics Logs Diagnostics
Inputs
Prognostics Alerts Metrics Logs Bugs/SRs
Cloud Admin UI

Why Applied Machine Learning?
• Brings an application’s perspective versus a platform toolkit viewpoint
• Brings data science, algorithms, and domain expertise together
• Packages machine learning into usable, real-world operational algorithms
and models that are applied at runtime
• Produces results and recommendations easily understood and trusted by
non-data scientist/analyst end-users

Applied Machine Learning for Cloud Operations
• Generic ML-extracted Data Clusters
are insufficient for diagnostics
• Operational data correlation does
not determine root cause
• Trusted root cause determination
critical to swift corrective actions
• Algorithms selected and models
built require domain expertise
• Models refined via field feedback
Subject Matter
Expert
ASH
ML
Knowledge
Extraction
Model
Generation
Human
Supervision
Application
Optimized
Models
Feedback
Scrub Data

Real-time Prevention
• Data Ingestion
– Kernel Smoothing and Moving Average
– Interpolation and Imputation
• Prediction and Pattern Recognition
– Multivariate and Auto-Associative Regression
– Clustering, Similarity Operators and Bayes Networks
• Fault and Anomaly Detection
– Sequential Probability Ratio Tests
– Conditional Probability Filters & Hidden Markov
Models
• Prognosis and Diagnosis
– Bayesian Belief Networks and Probabilistic Inference
– Remaining Useful Life Regression and GPM Models
Rapid Recovery
Autonomous Health Platform ML Technologies
• Data Ingestion
– ELK
– Lucene
• Prediction and Pattern Recognition
– TF-IDF and Bag-of-Words modelling
– Sequence Matcher
– K-nearest Neighbour
• Fault and Anomaly Detection
– Decision Trees and Random Forest
– Sequential Pattern Mining
• Prognosis and Diagnosis
– Recurrent neural Network
– Long short-term memory Predictive Analysis

Autonomous Health
Cloud Platform
MachinesSmart Collectors
SRs
Expert Input
Feedback &
Improvement
SRs
Model
Generation
Model
Knowledge
Extraction
Applied Machine Learning
Cloud Ops
Object Store
Admin UI in Control Plane
Oracle Support
Bug DB
SE UI in Support
Tenant (CNS)
Cleansing,
metadata creation
& clustering
5 Model generation
with expert scrubbing
6
Deployed as
part of cloud
image, running
from the start
1 Proactive regular health checking, real-
time fault detection, automatic
incident analysis, diagnostic collection
& masking of sensitive data
2
Use real-time health dashboards for anomaly
detection, root cause analysis & push of
proactive, preventative & corrective actions.
Auto bug search & auto bug & SR creation.
3
Auto SR analysis, diagnosis assistance via
automatic anomaly detection, collaboration
and one click bug creation
4
Message
Broker

• Files automatically extracted, classified,
processed and available
• File classification extendable

Extendable plugins transform & analyze raw data

Extendable plugins detect & highlight important events

Anomaly timeline uses machine
learning to detect unusual events,
which may be problematic

Adaptive Bug Search (ABS)
uses Machine Learning to
Identify possibly related bugs

Where available Recommendations automatically identify
symptoms from files & match them to previous identified
problems…

….then provide
resolution steps

Autonomous DB Operational Health
Managing Operational Performance & Recovery
Customer Session
Database
Infrastructure
Cloud Admin

Autonomous Health - Session Performance
Preventing a session from
blocking performance of others
by holding critical resources

Database Hang Management – Applied Machine Learning
• Actual Internal and External customer
data drives model development
• Purpose-built diagnostic technology
used for knowledge extraction
• Expert Dev team scrubs data
• Hang Heuristic Engine created and
deployed in database background
• HM uses run-time engine to perform
real-time DB hang detection and
resolution
Discovers and Resolves Runtime Database Hangs and Deadlocks
HM Dev Team
ASH
Knowledge
Extraction
Heuristic
Engine
Expert
Supervision
HM
Runtime
Engine
Feedback
HM
HM
Scrub Data

Database Hang Management – Session
• Monitors Session snapshots for progress
• Evaluates potential hangs over time with
based upon Wait Graphs
• Analyzes hang chain of sessions to identify
blocker/victim
• Verifies blocker session is hung for time
period to become victim
• Uses heuristic model to determine victim
resolution
• Terminates session or process to resolve
Autonomously Preserves Database Availability and Performance Session
DIA0
EVALUATE
DETECT
ANALYZE
Hung?
VERIFY
Victim
QoS
Policy
VICTIM

Autonomous Health - Database Performance
Preserving instance performance
when database resources are
constrained

Database Health - Applied Machine Learning
• Actual Internal and External customer
data drives model development
• Applied purpose-built Applied ML for
knowledge extraction
• Expert Dev team scrubs data
• Generates Bayesian Network-based
diagnostic root-cause models
• Uses BN-based run-time models to
perform real-time prognostics
Discovers Potential Cluster & DB Problems
CHA Dev Team
ASH
ML
Knowledge
Extraction
BN
Models
Expert
Supervision
DB+Node
Runtime
Models
Feedback
Scrub Data
CHA
CHA

Database Data Flow Overview
Autonomous Health – Database Performance
OS DataDB Data
Database Prognostics Engine
Alert &
Preventive
Action
• Reads OS and DB Performance data
directly from memory
• Uses Machine Learning models and data
to perform prognostics
• Detects common RAC database problems
• Performs root cause analysis
• Sends alerts and preventative actions to
Cloud Ops per target

Data Sources and Data Points
Time CPU ASM
IOPS
Network
% util
Network_
Packets
Dropped
Log
file
sync
Log file
parallel
write
GC CR
request
GC current
request
GC current
block 2-way
GC current
block busy
Enq: CF
-
conten
tion
…
15:16:00 0.90 4100 13% 0 2 ms 600 us 0 0 300 us 1.5 ms 0
A Data Point contains > 150 signals (statistics and events) from multiple sources
OS, ASM , Network DB ( SH, AWR session, system and PDB statistics )
Statistics are collected at a 1 second internal sampling rate , synchronized,
smoothed and aggregated to a Data Point every 5 seconds

Data Flow Overview

Models Capture the Dynamic Behavior of all Normal Operation
Models Capture all Normal Operating Modes
0
5000
10000
15000
20000
25000
30000
35000
40000
10:00 2:00 6:00
5100
9025
4024
2350
4100
22050
10000
21000
4400
2500
4900
800
IOPS
user commits (/sec)
log file parallel write (usec)
log file sync (usec)
A model captures the normal load phases and their statistics over time , and thus the
characteristics for all load intensities and profiles . During monitoring , any data point similar to one of the vectors
is NORMAL. One could say that the model REMEMBERS the normal operational dynamics over time
In-Memory Reference Matrix
(Part of “Normality” Model)
IOPS #### 2500 4900 800 ####
User Commits #### 10000 21000 4400 ####
Log File Parallel
Write
#### 2350 4100 22050 ####
Log File Sync #### 5100 9025 4024 ####
… … … … … …

CHA Model: Find Similarity with Normal Values
Observed values
(Part of a Data Point)
CHA estimator/predictor (ESEE): “based on my normality model, the value of IOPS should be in the
vicinity of ~ 4900, but it is reported as 10500, this is causing a residual of ~ 5600 in magnitude”,
CHA fault detector: “such high magnitude of residuals should be tracked carefully! I’ll keep an eye
on the incoming sequence of this signal IOPS and if it remains deviant I’ll generate a fault on it”.
In-Memory Reference Matrix
(Part of “Normality” Model)
IOPS #### 2500 4900 800 ####
User Commits #### 10000 21000 4400 ####
Log File Parallel
Write
#### 2350 4100 22050 ####
Log File Sync #### 5100 9025 4024 ####
… … … … … …
10500
20000
4050
10250
…
Residual Values
(Part of a Data Point)
5600
-1000
-50
325
…
Observed -
Predicted =

Inline and Immediate Fault Detection and Diagnostic Inference
Machine Learning,
Pattern Recognition,
& BN Engines
Time CPU ASM
IOPS
Network
% util
Network_
Packets
Dropped
Log
file
sync
Log file
parallel
write
GC CR
request
GC current
request
GC current
block 2-way
GC current
block busy
Enq: CF
-
conten
tion
…
15:16:00 0.90 4100 88% 105 2 ms 600 us 504 ms 513 ms 2 ms 5.9 ms 0
15:16:00
OK OK HIGH
1
HIGH
2
OK OK HIGH
3
HIGH
3
HIGH
4
HIGH
4
OK
Input : Data Point at Time t
Fault Detection and Classification
Diagnostic Inference
15:16:00
Symptoms
1. Network Bandwidth Utilization
2. Network Packet Loss
3. Global Cache Requests Incomplete
4. Global Cache Message Latency
Root Cause
(Target of Corrective Action)
Network Bandwidth Utilization
Diagnostic
Inference
Engine

Cross Node and Cross Instance Diagnostic Inference
Autonomous Health - Cluster Health Advisor
15:16:00
Root Cause
(Target of Corrective
Action)
Network Bandwidth
Utilization
Diagnostic
Inference
Engine
15:16:00
Root Cause
Action)
Network Bandwidth
Utilization
Diagnostic
Inference
Engine
15:16:00
Root Cause
Action)
Network Bandwidth
Utilization
Diagnostic
Inference
Engine
Cross Target
Diagnostic
Inference
Node 1
Node 2
Node 3
Corrective Action Target

Autonomous Health - Infrastructure Performance
Preserving database availability
when an ASM instance is hung or
blocked

Database Hang Management - Infrastructure
• Monitors Session snapshots for progress
• Evaluates potential hangs over time with
based upon Wait Graphs
• Analyzes hang chain of sessions to identify
blocker/victim
• Discovers blocker is located in ASM instance
• Requests ASM terminate session or instance
relying on Flex ASM for recovery
• Detection and resolution is bi-directional
Autonomously Preserves Database Infrastructure Availability
Database
ASM

Autonomous Health - Infrastructure Performance
Preserving node availability
when O/S resources are
congested or overutilized

Node Data Flow Overview
Autonomous Health – Infrastructure Performance
OS Data
Node Prognostics Engine
Alert &
Preventive
Action
• Reads OS Performance data directly from
memory
• Uses Applied Machine Learning models
and data to perform prognostics
• Detects O/S, network and storage
problems
• Performs root cause analysis
• Sends alerts and preventative action to
Cloud Ops per target

Autonomous Health – Alerts, Root Cause & Actions

Autonomous Health - Anomaly Timeline
Remove clutter from log files to
find the most important events to
enable root cause analysis

Anomaly Detection – High Level
Known normal log entry (discard)
Probable anomalous Line (collect)
Log
Collection
File
Type
1
File
Type
2
File
Type
n..
Log File
Anomaly Timeline
Probable
Anomalies

Trace File Analyzer – High Level Anomaly Detection Flow
Log
Cleansing
1 2 3 4 5 6
Entry Feature
Creation
Entry
Clustering
Model
Generation
Expert
Input
Knowledge Base
Creation
Knowledge
Base Indexing
Feedback
Training
Real-time
Log File Processing
Timestamp Correlation & Ranking
8 9
7
Batch
Feedback

Knowledge
Base Indexing
Entry
Clustering
Model
Generation
Entry Feature
Creation
Log
Cleansing
1 2 3 4 5 6
Expert
Input
Knowledge Base
Creation
FeedbackTraining Real-time
Log File
Processing
Timestamp
Correlation &
Ranking
8 97 Batch
Feedback
Log File
Collection
Data
Cleansing &
Reduction
waited for 'ASM file metadata operation', seq_num: 29
2016-10-20 02:12:56.937 : OCRRAW:1: kgfo_kge2slos error
stack at kgfoAl06: ORA-29701: unable to connect to Cluster
Synchronization Service
waited for [STR] seq_num:
[NSTR]
[NSTR] [NSTR] : [NSTR] [NSTR]
unable to connect to Cluster

.. Seen
in
Bugs
Total
Bugs
Seen
Seen
in
Files
Total
Files
Seen
Total
Count
..
.. 13 40 1440 5088 2890 ..
Feature
Extraction
waited for [STR] seq_num: [NSTR]
[NSTR] [NSTR] : [NSTR] [NSTR]
unable to connect to Cluster
Knowledge
Base Indexing
Entry
Clustering
Model
Generation
Log
Cleansing
1 3 4 5 6
Expert
Input
Knowledge Base
Creation
Log File
Processing
Timestamp
Correlation &
Ranking
8 97 Batch
Feedback
Entry Feature
Creation
2

Data
Clustering
Record Merging and feature
aggregation for records belonging
to same log signature
Knowledge
Base Indexing
Entry
Clustering
Model
Generation
Log
Cleansing
1 3 4 5 6
Expert
Input
Knowledge Base
Creation
Log File
Processing
Timestamp
Correlation &
Ranking
8 97 Batch
Feedback
2
Entry Feature
Creation

Model
Generation
Data
Clustering
Expert
Input
Decision Tree
Classifier
First time labelling through
functional rules
Labelled dataset
Result
EvaluationUpdate
Labelling
3
4
5
76
Entry
Clustering
Log
Cleansing
1 3
Training Real-time
Log File
Processing
Timestamp
Correlation &
Ranking
8 9
Batch
Feedback
2
Knowledge
Base Indexing
Model
Generation
4 5 6
Expert
Input
Knowledge Base
Creation
Feedback
7
Entry Feature
Creation

Maintenance Slot Identification

Autonomous Health - Anomaly Analysis
Identify a series of events as
connected and representing the
signature of a problem

Generating Event Signatures
Event Signature 35
Event Signature
3435
Event Signature
494
Event Signature
3948
Event Signature
292
Event Signature
434933
Node Eviction 1
Timeline
Event Signature
3434
Event Signature
3435
Event Signature
4344
Event Signature
3048
Event Signature
202
Event Signature
434983
Node Eviction 2
Timeline
Event Signature 35
Event Signature
3435
Event Signature
3048
Event Signature
3948
Event Signature
292
Event Signature
434933
New Signature
Check for weighted
probabilistic match
Problem Signature Repository

TFA & EXAchk
SRs
Expert Input
Feedback &
Improvement
SRs
Model
Generation
Model
Knowledge
Extraction
Machine Learning Training
Cleansing,
metadata
creation &
clustering
1
Model generation with
expert scrubbing2
Deployed as part of cloud
image, running from the
start, using model to
detect issues
3
Feed Machine Learning Model Back into TFA collector
• Identify Individual/Sequence of Log entries
around which Automatic Collection should be
performed using new additions to
knowledgebase & provide needed collection
context.
• Proactive pre-emptive actions based on
analysis of multiple time series metrics can
be performed.
• Reactive corrective action can be
optionally automated via TFA Service

Autonomous Health - Maintenance Slot Identification
Find the next best window when
maintenance can be performed with
minimal service impact

• Identify Relevant Workload Metrics
– Ex: Average Active Sessions, CPU/Mem/IO Utilization
• Time Series Decomposition
– Trend
– Seasonality
– Residual
• Workload Seasonality Determination Locating Minimas
• Optimum Window Identification and Validation
Model Generation and Training Flow

START_TIME CNT
2018-04-11 15:00:00 290
2018-04-11 16:00:00 31120
2018-04-11 17:00:00 21530
2018-04-11 18:00:00 26240
2018-04-11 19:00:00 40520
2018-04-11 20:00:00 54270
2018-04-11 21:00:00 51460
2018-04-11 22:00:00 44310
2018-04-11 23:00:00 25690
START_TIME
2018-04-11 15:00:00 -0.226098
2018-04-11 16:00:00 -0.069821
2018-04-11 17:00:00 -0.350088
2018-04-11 18:00:00 -0.187483
2018-04-11 19:00:00 -0.513240
2018-04-11 20:00:00 0.019737
2018-04-11 21:00:00 0.059213
2018-04-11 22:00:00 -0.011312
2018-04-11 23:00:00 -0.179156
START_TIME
2018-04-11 15:00:00 5.669881
2018-04-11 16:00:00 10.345606
2018-04-11 17:00:00 9.977203
2018-04-11 18:00:00 10.175040
2018-04-11 19:00:00 10.609551
2018-04-11 20:00:00 10.901727
2018-04-11 21:00:00 10.848560
2018-04-11 22:00:00 10.698966
2018-04-11 23:00:00 10.153857
Current Date : 2018-05-12 15:00:00
Current Position in Seasonality : -0.22609829742533585
Best Maintenance Period in next Cycle : 2018-05-12 19:00:00
Worst Maintenance Period in next Cycle : 2018-05-13 08:00:00
Original observation data1 Apply convolution filter & average2 Calculate seasonality3
Use seasonality to
predict best
maintenance window
4
Seasonality Determination to Window Identification Flow

Validating Performance Against Random or Periodic Window Selection

Autonomous Health - Bug Duplicate Identification
Discovers Duplicate Bugs,
Correlated Issues and Prioritizes
Based Upon Customer Impact

SRDCs (Service Request Diagnostic Collection)
Oracle Grid Infrastructure
& Databases
TFAML
1
TFAML detects a
fault
2Diagnostics
are collected
3
Distributed diagnostics
are consolidated and
packaged
4
Notification of fault is sent
5 Diagnostic collection is
uploaded to Oracle
Storage Service for later
analysis
Object Store

BUG
DB
Adaptive Bug Search – Applied Machine Learning
• Bugs are submitted from over 400 Oracle
products
• Performs ML Logistic Regression on
training set of bugs to generate model
• Displays up to 8 possible duplicates per
bug or SR
• Feedback improves model accuracy
– Direct from developers
– Indirect from bug updates
Discovers Duplicate Bugs and Correlated Issues
ABS Dev Team
Bugs
ML Logistic
Regression
Model
Generation
Expert
Supervision
ABS
Runtime
Model
Dev
Feedback
Bug
Submission Bug and
Duplicates
Together
ABS
Service
Scrub Data

Autonomous Health – Adaptive Bug Search (ABS)
• Issues parsed into different features
– Error stack, Trace data, Problem description, etc.
• Issues represented as a cluster of features
– i.e. All bugs in a bug tree contribute towards the feature set
• Logistic Regression applied to build a model
– Model defines the significance of each feature
• Similarity between issues computed using the model
– Identifies the root of the cluster (aka bug tree)
• Feedback used to improve the model
– Feedback is automatically derived based on how the bug gets closed
High Level Flow

Autonomous Health – Adaptive Bug Search
• ABS is internally offered in BugDB and MOS
GUI for Dev and Support teams
• ABS helps find problems in the same space
• Allows engineers to get the full context of
past known problems
• Provides debugging clues to help diagnose the
reported problem
• Identifies developers who worked in this
space in the past
Discovers Duplicate Bugs and Correlated Issues

“Machine learning (ML) is increasingly being adopted within
data management products for their improved operation and
optimization.
Gartner recommends that data and analytics leaders focused
on data management capabilities make ML-enabled
capabilities a 'must-have' selection criterion for new purchases
of data management products.”
– Donald Feinberg, VP Distinguished Analyst and Merv Adrian,
Research VP, Gartner

“Oracle Autonomous Database is likely to save the average
organization hundreds or, in some cases, thousands of FTE hours per
year for each major database.”
“Oracle is going about the process of automating error handling
within its Autonomous Database product.”
“To scale this service, Oracle developed automated database tuning
and maintenance based on machine learning.”
“Figure 2 shows how Oracle is going about the process of
automating error handling within its Autonomous Database product”
Carl W. Olofson &
David Schubmehl
IDC PERSPECTIVE:
Oracle's Autonomous Database:
AI-Based Automation for
Database Management and
Operations
http://idcdocserv.com/US43571317

Autonomous Health – Part of Oracle AIOps for the Cloud
• Historical Data Management
• Streaming Data Management
• Log Data Ingestion
• Metric Data Ingestion
• Autonomous Collections
• Automated Pattern Discovery and
Prediction
• Anomaly Detection
• Root Cause Determination
• Autonomous Operations

AUSOUG - Introducing New AI Ops Innovations in Oracle 19c Autonomous Health Framework

More Related Content

What's hot (20)

Similar to AUSOUG - Introducing New AI Ops Innovations in Oracle 19c Autonomous Health Framework (20)

More from Sandesh Rao (20)

Recently uploaded (20)

AUSOUG - Introducing New AI Ops Innovations in Oracle 19c Autonomous Health Framework