Analysis of Database Issues using AHF and Machine Learning v2 - AOUG2022

VP AIOps for the Autonomous Database
Sandesh Rao
AOUG 2022
Analysis of Database Issues using AHF
and Machine Learning
@sandeshr
https://www.linkedin.com/in/raosandesh/
https://www.slideshare.net/SandeshRao4

What is AHF
Compliance
Checking
Fault
Detection
Support
Upload
Fault
Notification
Diagnostic
Collection
2 TFA
Automatic issue detection,
diagnostic collection and analysis
along with a single interface for
Database support tools
1 EXAchk
Automatic compliance checking and
warnings when drifting away from best
practices as well as offering pre and
post upgrade advice

Install
OEDA Dec 2019+: AHF is already installed in your base image
Earlier versions either from RU or install from Doc 1070954.1
Installation and staying up to date
Exacloud ECS installer
One release behind latest
Release Updates
One release behind latest
MOS Download
Latest release
Doc 1070954.1
Upgrade Upgrade Upgrade

Automatically upgrade AHF if a new version is available at the software stage location
Auto Upgrade
ahfctl setupgrade -autoupgrade <on/off> -swstage path
ahfctl setupgrade –all
Enter autoupgrade flag <on/off> : on
Enter software stage location : /scratch/ahf_stage
Enter auto upgrade frequency : 30
AHF autoupgrade parameters successfully updated
Successfully synced AHF configuration
Example setting autoupgrade to check for a new version in stage location every 30 days

EXAchk
Automatic proactive warnings
before you’re impacted
Results viewable in the
tool of your choice
Regular emails with
check results
Compliance checks for most
impactful reoccurring problems
No need to send
anything to Oracle
REDUCE
YOUR RISK

Oracle Health Check Collection Manager Dashboard

1. PROS
• Stores Autonomous Health metrics for real-time
and post-mortem analysis
• Cluster Health Monitor (CHM)
• Cluster Health Advisor (CHA)
• DB QoS Management (QoSM)
• Default 72 hours of storage
• Minimized resource footprint
• Built-in Automatic Lifecycle management
• Automatic HA failover support
• No DBA management required
The GIMR – Your Oracle Cluster Diagnostics Repository
7
1. CONS
• Requires minimum 30GB of shared disk
• GI Patching and Upgrade integration requires
significantly longer maintenance window
12.1 12.2+ 18.1+ 19.1+ 19.5+ 21.1+
Optional Mandatory Optional Optional
Copyright © 2021, Oracle and/or its affiliates

Local Option
8 Copyright © 2021, Oracle and/or its affiliates

1. Install the Oracle 21c Grid Infrastructure with Default GIMR Option.
• If using ASM, create a disk group for the GIMR (ex: MGMT)
2. Install an Oracle 21c Database Home in a separate directory as the GI User.
• Install on all nodes as you would an Oracle RAC database.
3. Create the GIMR Database
• OH/bin/mgmtca createGIMRContainer [-storageDiskLocation disk_location]
How To Install a Local 21c GIMR in 3 Steps

10
Generates view of Cluster and Database diagnostic metrics
• Always on - Enabled by default
• Provides Detailed OS Resource Metrics
• Assists Node eviction analysis
• Locally logs all process data
• User can define pinned processes
• Listens to CSS and GIPC events
• Categorizes processes by type
• Supports plug-in collectors (ex. traceroute, netstat, ping, etc.)
• New CSV output for ease of analysis
AHF Component - Cluster Health Monitor (CHM)
GIMR
ologgerd
(master)
osysmond
osysmond
osysmond
osysmond
12c Grid Infrastructure
Management Repository
OS Data OS Data
OS Data
OS Data

11
• Always on - Enabled by default
• Detects node and database performance problems
• Provides early-warning alerts and corrective action
• Supports on-site calibration to improve sensitivity
• Integrated into EMCC Incident Manager and notifications
AHF Component - Cluster Health Advisor (CHA)
OS Data
GIMR
ochad
DB Data
CHM
Node
Health
Prognostics
Engine
Database
Health
Prognostics
Engine
* Requires and Included with RAC or R1N License

Autonomously Preserves Database Availability and Performance
• Monitors Session snapshots for progress
• Evaluates potential hangs over time with based upon
Wait Graphs
• Analyzes hang chain of sessions to identify
blocker/victim
• Verifies blocker session is hung for time period to
become victim
• Uses heuristic model to determine victim resolution
• Terminates session or process to resolve
• Discovers blocker is located in ASM instance
• Requests ASM terminate session or instance relying
on Flex ASM for recovery
Database Hang Management – Session
12
Session
DIA0
EVALUATE
DETECT
ANALYZE
Hung?
VERIFY
Victim
QoS
Policy
VICTIM

13
CONS
• Includes both critical and non-critical events
• Incudes messages not intended for DBAs
• Inconsistently reports severity level
• Can report unintuitive cause and action
• New undocumented messages in every release
PRO
• Destination for Important DB Events
• Single file to monitor by DBAs
• Many tools available to parse
• Supported by TFA for generating alarms
Oracle Database Alert Log

14
Contains only important events requiring customer attention
Includes documented set of messages and attributes
All Messages include these attributes:
• Type
• Urgency
• Scope
• Target User
• Cause and Action
• Additional debug information
The Curated Solution - New 21c Attention Log

15
Oracle Database Attention Log Message Flow
Diagnostic Framework
alert/log.xml log/attention.log
attention.amb
DB
Component
Message Definitions
Attention Log Message
Attention Curated Message

16
SCOPE
1. Session
2. Process
3. PDB-Instance
4. CDB-Instance
5. CDB-Cluster
6. PDB-Persistent
7. CDB-Persistent
URGENCY
1. Immediate
2. Soon
3. Deferable
4. Info
TYPE
1. Error
2. Warning
3. Notification
Attention Log Curation - Message Attributes
TARGET USER
1. App-Dev
2. Sec-Admin
3. Net-Admin
4. Cluster-Admin
5. PDB-Admin
6. CDB-Admin
7. Server-Admin
8. Storage-Admin
9. DataOps-Admin

Discovers Potential Cluster & DB Problems
Actual Internal data drives model
development
Applied purpose-built Applied ML for
knowledge extraction
Expert Dev team scrubs data
Generates Bayesian Network-based
diagnostic root-cause models
Uses BN-based run-time models to
perform real-time prognostics
Database Health - Applied Machine Learning
17
AHF Dev Team
Log
ASH
Metrics
ML
Knowledge
Extraction
BN
Models
Expert
Supervision
DB+Node
Runtime
Models
Feedback
Scrub Data
AHF
AHF

CHA Operational Flow : Anomaly Detection -> Diagnostics -> Prognosis
For each data point …
Cluster Health Advisor
18
Is data valid ? Is behavior
expected ?
Is there a
problem ?
What is
causing the
problem ?
Data Validation
Operating State
Estimation
Fault
Identification
Diagnostic
Decision
Machine Learning,
Pattern Recognition,
& BN Engines
Prognosis
Is a failure
likely ?

AHF and Machine Learning
19
• Machine Learning and Statistical Inference address scale, dynamics
and interdependency in Clusters and Clouds
• An ML Model is an in-memory representation of a normally
behaving application over time, learned from historical operational
data , in the form of a collection of vectors of operational data
• The similarity or distance of a monitored data point to a vector in
the in-memory model is the basis to for a comparison between the
normal data and the actual data

• ORAchk/EXAchk provides a single source for all upgrade checks
• ORAchk , EXAchk checks , Database AutoUpgrade checks , Cluster Verification Utility
(CVU) checks
• AHF proactive findings
• Tail Alert and Attention logs and build the timeline
• Dig deeper – AHF to either collect data , analyze
• Performance Issue
• OS or Network Issue
• Node Evictions ?
Step by Step on how to perform an autopsy of a problem

21
• To check an environment before upgrading run:
• To check an environment after upgrade run:
• To check an environment for best practice violations
• Setup Collection Manager to aggregate the results for a birds eye view
• Autonomous Health Framework (AHF) - Including TFA and ORAchk/EXAchk (Doc ID 2550798.1)
ORAchk/EXAchk provides a single source for all upgrade checks
orachk –preupgrade
orachk –postupgrade
orachk

22
Choosing a Data Set for Calibration – Defining “normal”
Calibrating AHF to your RAC deployment
chactl query calibration –cluster –timeranges ‘start=2021-12-01 07:00:00,end=2021-12-01
13:00:00’
Cluster name : mycluster
Start time : 2021-12-01 07:00:00
End time : 2021-12-01 13:00:00
Total Samples : 11524
Percentage of filtered data : 100%
1) Disk read (ASM) (Mbyte/sec)
MEAN MEDIAN STDDEV MIN MAX
0.11 0.00 2.62 0.00 114.66
<25 <50 <75 <100 >=100
99.87% 0.08% 0.00% 0.02% 0.03%
...

23
...
2) Disk write (ASM) (Mbyte/sec)
0.01 0.00 0.15 0.00 6.77
<50 <100 <150 <200 >=200
100.00% 0.00% 0.00% 0.00% 0.00%
...

24
...
3) Disk throughput (ASM) (IO/sec)
2.20 0.00 31.17 0.00 1100.00
<5000 <10000 <15000 <20000 >=20000
100.00% 0.00% 0.00% 0.00% 0.00%
4) CPU utilization (total) (%)
9.62 9.30 7.95 1.80 77.90
<20 <40 <60 <80 >=80
92.67% 6.17% 1.11% 0.05% 0.00%
...

25
Create and store a new model
Begin using the new model
Confirm the new model is working
Calibrating CHA to your RAC deployment
chactl query calibrate cluster –model daytime –timeranges ‘start=2021-
12-01 07:00:00, end= 2021-12-01 13:00:00’
chactl monitor cluster –model daytime
chactl status –verbose
monitoring nodes svr01, svr02 using model daytime
monitoring database qoltpacdb, instances oltpacdb_1, oltpacdb_2 using model DEFAULT_DB

26
Check for Health Issues and Corrective Actions with CHACTL QUERY DIAGNOSIS
Command line operations
chactl query diagnosis -db oltpacdb -start "2021-12-01 01:42:50" -end "2021-12-01 03:19:15"
2021-12-01 01:47:10.0 Database oltpacdb DB Control File IO Performance (oltpacdb_1) [detected]
2021-12-01 01:47:10.0 Database oltpacdb DB Control File IO Performance (oltpacdb_2) [detected]
2021-12-01 02:59:35.0 Database oltpacdb DB Log File Switch (oltpacdb_1) [detected]
2021-12-01 02:59:45.0 Database oltpacdb DB Log File Switch (oltpacdb_2) [detected]
Problem: DB Control File IO Performance
Description: CHA has detected that reads or writes to the control files are slower than expected.
Cause: The Cluster Health Advisor (CHA) detected that reads or writes to the control files were
slow because of an increase in disk IO.
The slow control file reads and writes may have an impact on checkpoint and Log Writer (LGWR)
performance.
Action: Separate the control files from other database files and move them to faster disks or
Solid
State Devices.
Problem: DB Log File Switch
Description: CHA detected that database sessions are waiting longer than expected
for log switch completions.
Cause: The Cluster Health Advisor (CHA) detected high contention during log switches
because the redo log files were small and the redo logs switched frequently.
Action: Increase the size of the redo logs.

27
Command line operations
HTML diagnostic health output available (-html <file_name>)

Data Sources and Data Points
Autonomous Health – Database Performance
28
Time CPU ASM
IOPS
Network
% util
Network_
Packets
Dropped
Log
file
sync
Log file
parallel
write
GC CR
request
GC current
request
GC current
block 2-way
GC current
block busy
Enq: CF
-
conten
tion
…
15:16:00 0.90 4100 13% 0 2 ms 600 us 0 0 300 us 1.5 ms 0
A Data Point contains > 150 signals (statistics and events) from multiple sources
OS, ASM , Network DB ( SH, AWR session, system and PDB statistics )
Statistics are collected at a 1 second internal sampling rate , synchronized,
smoothed and aggregated to a Data Point every 5 seconds

Data Flow Overview

30
Models Capture the Dynamic Behavior of all Normal Operation
Models Capture all Normal Operating Modes
0
5000
10000
15000
20000
25000
30000
35000
40000
10:00 2:00 6:00
5100
9025
4024
2350
4100
22050
10000
21000
4400
2500
4900
800
IOPS
user commits (/sec)
log file parallel write (usec)
log file sync (usec)
A model captures the normal load phases and their statistics over time , and thus the
characteristics for all load intensities and profiles . During monitoring , any data point similar to one of the vectors
is NORMAL. One could say that the model REMEMBERS the normal operational dynamics over time
In-Memory Reference Matrix
(Part of “Normality” Model)
IOPS #### 2500 4900 800 ####
User Commits #### 10000 21000 4400 ####
Log File Parallel
Write
#### 2350 4100 22050 ####
Log File Sync #### 5100 9025 4024 ####
… … … … … …

Inline and Immediate Fault Detection and Diagnostic Inference
31
Machine Learning,
Pattern Recognition,
& BN Engines
Time CPU ASM
IOPS
Network
% util
Network_
Packets
Dropped
Log
file
sync
Log file
parallel
write
GC CR
request
GC current
request
GC current
block 2-way
GC current
block busy
Enq: CF
-
conten
tion
…
15:16:00 0.90 4100 88% 105 2 ms 600 us 504 ms 513 ms 2 ms 5.9 ms 0
15:16:00
OK OK HIGH
1
HIGH
2
OK OK HIGH
3
HIGH
3
HIGH
4
HIGH
4
OK
Input : Data Point at Time t
Fault Detection and Classification
Diagnostic Inference
15:16:00
Symptoms
1. Network Bandwidth Utilization
2. Network Packet Loss
3. Global Cache Requests Incomplete
4. Global Cache Message Latency
Root Cause
(Target of Corrective Action)
Network Bandwidth Utilization
Diagnostic
Inference
Engine

Cross Node and Cross Instance Diagnostic Inference
Autonomous Health - Cluster Health Advisor
32
15:16:00
Root Cause
(Target of Corrective
Action)
Network Bandwidth
Utilization
Diagnostic
Inference
Engine
15:16:00
Root Cause
Action)
Network Bandwidth
Utilization
Diagnostic
Inference
Engine
15:16:00
Root Cause
Action)
Network Bandwidth
Utilization
Diagnostic
Inference
Engine
Cross Target
Diagnostic
Inference
Node 1
Node 2
Node 3
Corrective Action Target

CHA Diagnoses Focus Areas
33
CHA
Diagnoses
Database
Diagnoses
ControlFile
Diagnoses
UndoSpace
Diagnoses
RedoLog
Diagnoses
I/O
Diagnoses
LogFile
Diagnoses
Recovery
Diagnoses
Archiving
Diagnoses
GlobalCache
Diagnoses
Clusterware
Diagnoses
Interconnect
Diagnoses
ASM
Diagnoses
Host
Diagnoses
Memory
Diagnoses
CPU
Diagnoses

Critical CHA Diagnoses and Their Impacts
34
Instance Eviction
Node Eviction
CHA detects abrupt, significant decrease in
message traffic on the cluster Interconnect
Hang
Instance Eviction
CHA detects slow response times for Global Cache
messages
Hang
CHA detects a capacity issue in the Global Cache
Services
Hang
Instance Eviction
CHA detects Socket Buffer Overflows
Hang
Instance Eviction
Node Eviction
CHA detects network packets are discarded by
private network interface
CHA_PRIV_NW_PATH
CHA_PRIV_IC_LOSS
CHA_GCS_BUSY
CHA_PRIV_NETWORK_MSG
CHA_GC_NIC_CONFIG
Diagnosis ID Description Impact
Hang
Instance Eviction
CHA detects global cache messages on the private
interconnect are lost
CHA_GC_IPC_CONGESTION

CHA Diagnostics Logic: What is in a “Model” ?
CHA Diagnostics Model
35
An increase in
interconnect network
traffic causes cache
fusion packet drops .
Increase the IPC buffers
Network packet
rate higher than
expected
Network traffic
in/sec=HIGH
Network traffic
Out/sec =HIGH
Global Cache
drops packets
Global cache blocks
lost=HIGH
When multiple faults are
detected, they are passed as evidence
to a Probabilistic Bayesian Belief
Network for Cause and Effect Analysis
Prior Probabilities and Dependencies
are determined during development
Based on historical cases
OS DB

36
Statistics from OS, Storage,DB, Network etc.
Grouped into component groups

37
Observed
Predicted

40
Contains only important events requiring customer attention
Includes documented set of messages and attributes
All Messages include these attributes:
• Type
• Urgency
• Scope
• Target User
• Cause and Action
• Additional debug information
The Curated Solution - New 21c Attention Log

41
Example Attention Message Definition – CDB Warning
// TYPE - 1 error, 2 warning, 3 notification
// URGENCY - 1 immediate, 2 soon, 3 deferable, 4 info
// SCOPE - 1 session, 2 process, 3 pdb-instance, 4 cdb-instance, 5 cdb-cluster, 6 pdb-
persistent, 7 cdb-persistent
// TARGETUSER - 1 app-dev, 2 sec-admin, 3 net-admin, 4 cluster-admin, 5 pdb-admin, 6 cdb-
admin, 7 server-admin, 8 storage-admin, 9 dataops-admin
ID::2000
TYPE::2
URGENCY::1
SCOPE::4
TARGETUSER::6
TEXT::Parameter %s specified is high
CAUSE::Memory parameter specified for this instance is high
ACTION::Check alert log or trace file for more information relating to instance configuration,
reconfigure the parameter and restart the instance
STARTVERSION::21.1

42
Example Attention Log Curated Message – CDB Warning
[
IMMEDIATE Parameter SGA_MAX_SIZE specified is high
CAUSE: Memory parameter specified for this instance is high
ACTION: Check alert log or trace file for more information relating to instance
configuration, reconfigure the parameter and restart the instance
CLASS: CDB Instance / CDB ADMINISTRATOR / WARNING / AL-2000
TIME: 2020-05-01T11:09:02.223-07:00
ADDITIONAL INFO: -
WARNING: SGA_MAX_SIZE (6144 MB) is too high - it should be
less than 5634 MB (80 percent of physical memory).
]

43
Example Attention Log Curated Message – CDB Error
[
IMMEDIATE Shutting down ORACLE instance (abort) (OS id: 8394)
CAUSE: A command to shutdown the instance was executed
ACTION: Check alert log for progress and completion of command
CLASS: CDB Instance / CDB ADMINISTRATOR / ERROR / AL-1002
TIME: 2020-05-08T17:09:33.773-07:00
ADDITIONAL INFO: -
Shutdown is initiated by sqlplus@den02tlh (TNS V1-V3).
]

44
Example Attention Log Curated Message – Server Warning
[
SOON Heavy swapping observed on system
CAUSE: Memory usage by one more application is leading to heavy swapping
ACTION: Check alert log for more information, use tools to analyze memory
usage and take action
CLASS: CDB Instance / SERVER ADMINISTRATOR / WARNING / AL-2100
TIME: 2020-05-01T11:09:02.223-07:00
ADDITIONAL INFO: -
WARNING: Heavy swapping observed on system in last 15 mins.
Heavy swapping can lead to timeouts, poor performance, and instance eviction.
]

45
Attention Log Use Cases – Autonomous Health Framework
Autonomous Health
Framework
Trace File Analyzer
…
… CDB DBA
Sys Admin
PDB DBA
Attention Log
Repository

46
Attention Log Use Cases – 3rd Party Monitoring
3rd Party Monitoring,
Parsing and Routing
…
…
Attention Log
Management VCN
Ops Dashboard
Ops Admin
attention.amb Message Definitions
Runbooks

tfactl changes
Output from host : myserver69
------------------------------
[Dec/01/2021 04:54:15.397]: Parameter: fs.aio-nr: Value: 95488 => 97024
[Dec/01/2021 04:54:15.397]: Parameter: fs.inode-nr: Value: 764974 131561 => 740744 131259
[Dec/01/2021 04:54:15.397]: Parameter: kernel.pty.nr: Value: 2 => 1
[Dec/01/2021 04:54:15.397]: Parameter: kernel.random.entropy_avail: Value: 189 => 158
[Dec/01/2021 04:54:15.397]: Parameter: kernel.random.uuid: Value: 36269877-9bc9-40a3-82e0-
1619865096f2 => 7551c5e7-c59f-40fa-b55f-5bd170e8b1ab
[Dec/01/2021 05:46:15.397]: Parameter: fs.inode-nr: Value: 1580316 810036 => 1562320
768555
[Dec/01/2021 05:46:15.397]: Parameter: kernel.random.uuid: Value: 37cc31aa-ee31-459e-8f2a-
0766b34b1b64 => f5176cdc-6390-415d-882e-02c4cff2ae4e
...
Has anything changed recently?

...
------------------------------
[Dec/01/2021 04:54:15.397]: Parameter: fs.inode-nr: Value: 764974 131561 => 740744 131259
[Dec/01/2021 04:54:15.397]: Parameter: kernel.random.entropy_avail: Value: 189 => 158
[Dec/01/2021 04:54:15.397]: Parameter: kernel.random.uuid: Value: 36269877-9bc9-40a3-82e0-
1619865096f2 => 7551c5e7-c59f-40fa-b55f-5bd170e8b1ab
[Dec/01/2021 05:46:15.397]: Parameter: fs.inode-nr: Value: 1580316 810036 => 1562320
768555
[Dec/01/2021 05:46:15.397]: Parameter: kernel.random.uuid: Value: 37cc31aa-ee31-459e-8f2a-
0766b34b1b64 => f5176cdc-6390-415d-882e-02c4cff2ae4e
Has anything changed recently?

tail files
49
tfactl tail alert
------------------------------
/scratch/app/11.2.0.4/grid/log/myserver69/alertmyserver69.log
2021-12-01 23:28:22.532:
[ctssd(5630)]CRS-2409:The clock on host myserver69 is not synchronous with the mean cluster
time. No action has been taken as the Cluster Time Synchronization Service is running in
observer mode.
2021-12-01 23:58:22.964:
[ctssd(5630)]CRS-2409:The clock on host myserver69 is not synchronous with the mean cluster
time. No action has been taken as the Cluster Time Synchronization Service is running in
observer mode.
...

tail files
50
...
/scratch/app/oradb/diag/rdbms/apxcmupg/apxcmupg_2/trace/alert_apxcmupg_2.log
Wed Dec 01 06:00:00 2021 VKRM started with pid=82, OS id=4903
Wed Dec 01 06:00:02 2021 Begin automatic SQL Tuning Advisor run for special tuning task
"SYS_AUTO_SQL_TUNING_TASK"
Wed Dec 01 06:00:37 2021 End automatic SQL Tuning Advisor run for special tuning task
"SYS_AUTO_SQL_TUNING_TASK"
Wed Dec 01 23:00:28 2021 Thread 2 advanced to log sequence 759 (LGWR switch)
Current log# 3 seq# 759 mem# 0: +DATA/apxcmupg/onlinelog/group_3.289.917164707
Current log# 3 seq# 759 mem# 1: +FRA/apxcmupg/onlinelog/group_3.289.917164707
...

Database areas
Errors / Corruption
Performance
Install / patching / upgrade
RAC / Grid Infrastructure
Import / Export
RMAN
Transparent Data Encryption
Storage / partitioning
Undo / auditing
Listener / naming services
Spatial / XDB
Other Server Technology
Enterprise Manager
Data Guard
GoldenGate
Exalogic
Around 100 problem types covered
51
Full list in documentation
tfactl diagcollect –srdc <srdc_type> [-sr <sr_number>]

Manual method
1. Generate ADDM reviewing Document 1680075.1 (multiple steps)
2. Identify “good” and “problem” periods and gather AWR
reviewing Document 1903158.1 (multiple steps)
3. Generate AWR compare report (awrddrpt.sql) using “good” and
“problem” periods
4. Generate ASH report for “good” and “problem” periods reviewing
Document 1903145.1 (multiple steps)
5. Collect OSWatcher data reviewing Document 301137.1 (multiple
steps)
6. Collect Hang Analyze output at Level 4
7. Generate SQL Healthcheck for problem SQL id using Document
1366133.1 (multiple steps)
8. Run support provided sql scripts – Log File sync diagnostic output
using Document 1064487.1 (multiple steps)
9. Check alert.log if there are any errors during the “problem”
period
10. Find any trace files generated during the “problem” period
11. Collate and upload all the above files/outputs to SR
TFA SRDC
1. Run
Manual collection vs TFA SRDC for database performance
52
tfactl diagcollect –srdc dbperf [-sr <sr_number>]

53
tfactl diagcollect –srdc <srdc_type>
• Scans system to identify recent events
• Once the relevant event is chosen, proceeds with diagnostic collection
One command SRDC
tfactl diagcollect -srdc ORA-00600
Enter the time of the ORA-00600 [YYYY-MM-DD HH24:MI:SS,<RETURN>=ALL] :
Enter the Database Name [<RETURN>=ALL] :
1. Dec/01/2021 05:29:58 : [orcl2] ORA-00600: internal error code, arguments: [600], [], [],
[], [], [], [], [], [], [], [], []
2. Dec/01/2021 06:55:08 : [orcl2] ORA-00600: internal error code, arguments: [600], [], [],
[], [], [], [], [], [], [], [], []
Please choose the event : 1-2 [1]
Selected value is : 1 (Dec/01/2021 05:29:58 )

All required files are identified
• Trimmed where applicable
• Package in a zip ready to provide to support
One command SRDC
...
2021/12/01 06:14:24 EST : Getting List of Files to Collect
2021/12/01 06:14:27 EST : Trimming file :
myserver1/rdbms/orcl2/orcl2/trace/orcl2_lmhb_3542.trc with original file size : 163MB
...
2021/12/01 06:14:58 EST : Total time taken : 39s
2021/12/01 06:14:58 EST : Completed collection of zip files.
...
/opt/oracle.ahf/data/repository/srdc_ora600_collection_Wed_Dec_1_06_14_17_EST_2021_node_loca
l/myserver1.tfa_srdc_ora600_Wed_Dec_1_06_14_17_EST_2021.zip

55
tfactl managelogs -show usage
...
.---------------------------------------------------------------------------------.
| Grid Infrastructure Usage |
+---------------------------------------------------------------------+-----------+
| Location | Size |
+---------------------------------------------------------------------+-----------+
| /u01/app/crsusr/diag/afdboot/user_root/host_309243680_94/alert | 28.00 KB |
| /u01/app/crsusr/diag/afdboot/user_root/host_309243680_94/incident | 4.00 KB |
| /u01/app/crsusr/diag/afdboot/user_root/host_309243680_94/trace | 8.00 KB |
...
+---------------------------------------------------------------------+-----------+
| Total | 739.06 MB |
'---------------------------------------------------------------------+-----------’
...
Understand Database log disk space usage
Use -gi to only show grid infrastructure

56
...
.---------------------------------------------------------------.
| Database Homes Usage |
+---------------------------------------------------+-----------+
| Location | Size |
+---------------------------------------------------+-----------+
| /u01/app/crsusr/diag/rdbms/cdb674/CDB674/alert | 1.06 MB |
| /u01/app/crsusr/diag/rdbms/cdb674/CDB674/incident | 4.00 KB |
| /u01/app/crsusr/diag/rdbms/cdb674/CDB674/trace | 146.19 MB |
| /u01/app/crsusr/diag/rdbms/cdb674/CDB674/cdump | 4.00 KB |
| /u01/app/crsusr/diag/rdbms/cdb674/CDB674/hm | 4.00 KB |
+---------------------------------------------------+-----------+
| Total | 147.26 MB |
'---------------------------------------------------+-----------'
Understand Database log disk space usage
Use -database to only show database

57
Understand Database log disk space usage variations
tfactl managelogs -show variation -older 30d
------------------------------
2021-12-01 12:30:42: INFO Checking space variation for 30 days
.---------------------------------------------------------------------------------------------.
| Grid Infrastructure Variation |
+---------------------------------------------------------------------+-----------+-----------+
| Directory | Old Size | New Size |
+---------------------------------------------------------------------+-----------+-----------+
| /u01/app/crsusr/diag/asm/user_root/host_309243680_96/alert | 22.00 KB | 28.00 KB |
+---------------------------------------------------------------------+-----------+-----------+
| /u01/app/crsusr/diag/clients/user_crsusr/host_309243680_96/cdump | 4.00 KB | 4.00 KB |
+---------------------------------------------------------------------+-----------+-----------+
| /u01/app/crsusr/diag/tnslsnr/myserver74/listener/alert | 15.06 MB | 244.10 MB |
+---------------------------------------------------------------------+-----------+-----------+
...

Run a database log purge
58
tfactl managelogs -purge -older 30d
------------------------------
Purging files older than 30 days
Cleaning Grid Infrastructure destinations
Purging diagnostic destination "diag/afdboot/user_root/host_309243680_94" for files - 0 files deleted , 0 bytes freed
Purging diagnostic destination "diag/afdboot/user_crsusr/host_309243680_94" for files - 1 files deleted , 10.16 KB freed
Purging diagnostic destination "diag/asmtool/user_root/host_309243680_96" for files - 1 files deleted , 10.16 KB freed
Purging diagnostic destination "diag/asmtool/user_crsusr/host_309243680_96" for files - 2 files deleted , 29.18 KB freed
Purging diagnostic destination "diag/tnslsnr/myserver74/listener" for files - 2 files deleted , 29.18 KB freed
Purging diagnostic destination "diag/diagtool/user_root/adrci_309243680_96" for files - 2 files deleted , 29.18 KB freed
Purging diagnostic destination "diag/clients/user_crsusr/host_309243680_96" for files - 2 files deleted , 29.18 KB freed
Purging diagnostic destination "diag/asm/+asm/+ASM" for files - 2 files deleted , 29.18 KB freed
Purging diagnostic destination "diag/asm/user_root/host_309243680_96" for files - 2 files deleted , 29.18 KB freed
Purging diagnostic destination "diag/asm/user_crsusr/host_309243680_96" for files - 2 files deleted , 29.18 KB freed
Purging diagnostic destination "diag/crs/myserver74/crs" for files - 2 files deleted , 29.18 KB freed
...

59
Monitor Database performance
tfactl run oratop -database ogg19c

System metric-set detailing the summarizing state of the system.
ex : oclumon dumpnodeview local -system
Querying metrics in AHF
60

Ø Metrics aggregated by Process Groups (DB FG/DB BG/Other/Clusterware)
• ex. OTHER group is consuming ~96% ( across 350 processes) and DB BG ~4%(across 75 processes) of total 29.56% CPU
utilization.
• ex : oclumon dumpnodeview local -procagg
Querying Process Aggregate
61

Processes ordered by CPU, RSS, IO and Open FD’s
ex : oclumon dumpnodeview local -process
Querying Process Metric Set
62

Ø Devices details ordered by service time.
• ex : oclumon dumpnodeview local -device
Ø Network interfaces ordered by net transmission rate .
ex : oclumon dumpnodeview local -nic
Querying Device and NIC Metric Set
63

Ø Individual CPU Core Details (ordered by usage)
• ex : oclumon dumpnodeview local -cpu
Ø File System Details
ex : oclumon dumpnodeview local -filesystem
Querying CPU and File System Metric Set
64

VP AIOps for the Autonomous Database
Sandesh Rao
AOUG 2022
Thank you
@sandeshr
https://www.linkedin.com/in/raosandesh/
https://www.slideshare.net/SandeshRao4

Analysis of Database Issues using AHF and Machine Learning v2 - AOUG2022

More Related Content

What's hot (20)

Similar to Analysis of Database Issues using AHF and Machine Learning v2 - AOUG2022 (20)

More from Sandesh Rao (20)

Recently uploaded (20)

Analysis of Database Issues using AHF and Machine Learning v2 - AOUG2022