15 Troubleshooting Tips and Tricks for database 21c - OGBEMEA KSAOUG

VP AIOps for the Autonomous Database
Sandesh Rao
KSAOUG – Oct 2021
15 Troubleshooting tips and tricks for
Database 21c
@sandeshr
https://www.linkedin.com/in/raosandesh/
https://www.slideshare.net/SandeshRao4

Copyright © 2020, Oracle and/or its affiliates | Confidential: Internal/Restricted/Highly Restricted
2
IT Operations
It’s 2am. Do I page? Who do I page?
ALERT!

4
CONS
• Includes both critical and non-critical events
• Incudes messages not intended for DBAs
• Inconsistently reports severity level
• Can report unintuitive cause and action
• New undocumented messages in every release
PRO
• Destination for Important DB Events
• Single file to monitor by DBAs
• Many tools available to parse
• Supported by TFA for generating alarms
Oracle Database Alert Log
Copyright © 2020, Oracle and/or its affiliates

5
Contains only important events requiring customer attention
Includes documented set of messages and attributes
All Messages include these attributes:
• Type
• Urgency
• Scope
• Target User
• Cause and Action
• Additional debug information
The Curated Solution - New 21c Attention Log

6
Oracle Database Attention Log Message Flow
Diagnostic Framework
alert/log.xml log/attention.log
attention.amb
DB
Component
Message Definitions
Attention Log Message
Attention Curated Message

7
SCOPE
1. Session
2. Process
3. PDB-Instance
4. CDB-Instance
5. CDB-Cluster
6. PDB-Persistent
7. CDB-Persistent
URGENCY
1. Immediate
2. Soon
3. Deferable
4. Info
TYPE
1. Error
2. Warning
3. Notification
Attention Log Curation - Message Attributes
TARGET USER
1. App-Dev
2. Sec-Admin
3. Net-Admin
4. Cluster-Admin
5. PDB-Admin
6. CDB-Admin
7. Server-Admin
8. Storage-Admin
9. DataOps-Admin

8
Example Attention Message Definition – CDB Warning
// TYPE - 1 error, 2 warning, 3 notification
// URGENCY - 1 immediate, 2 soon, 3 deferable, 4 info
// SCOPE - 1 session, 2 process, 3 pdb-instance, 4 cdb-instance, 5 cdb-cluster, 6 pdb-
persistent, 7 cdb-persistent
// TARGETUSER - 1 app-dev, 2 sec-admin, 3 net-admin, 4 cluster-admin, 5 pdb-admin, 6 cdb-
admin, 7 server-admin, 8 storage-admin, 9 dataops-admin
ID::2000
TYPE::2
URGENCY::1
SCOPE::4
TARGETUSER::6
TEXT::Parameter %s specified is high
CAUSE::Memory parameter specified for this instance is high
ACTION::Check alert log or trace file for more information relating to instance configuration,
reconfigure the parameter and restart the instance
STARTVERSION::21.1

9
Example Attention Log Curated Message – CDB Warning
[
IMMEDIATE Parameter SGA_MAX_SIZE specified is high
CAUSE: Memory parameter specified for this instance is high
ACTION: Check alert log or trace file for more information relating to instance
configuration, reconfigure the parameter and restart the instance
CLASS: CDB Instance / CDB ADMINISTRATOR / WARNING / AL-2000
TIME: 2020-05-01T11:09:02.223-07:00
ADDITIONAL INFO: -
WARNING: SGA_MAX_SIZE (6144 MB) is too high - it should be
less than 5634 MB (80 percent of physical memory).
]

10
Example Attention Message Definition – CDB Error
ID::1002
TYPE::1
URGENCY::1
SCOPE::4
TARGETUSER::6
TEXT::Shutting down ORACLE instance (abort) %s
CAUSE::A command to shutdown the instance was executed
ACTION::Check alert log for progress and completion of command
STARTVERSION::21.1

11
Example Attention Log Curated Message – CDB Error
[
IMMEDIATE Shutting down ORACLE instance (abort) (OS id: 8394)
CAUSE: A command to shutdown the instance was executed
ACTION: Check alert log for progress and completion of command
CLASS: CDB Instance / CDB ADMINISTRATOR / ERROR / AL-1002
TIME: 2020-05-08T17:09:33.773-07:00
ADDITIONAL INFO: -
Shutdown is initiated by sqlplus@den02tlh (TNS V1-V3).
]

12
Example Attention Message Definition – Server Warning
ID::2100
TYPE::2
URGENCY::2
SCOPE::4
TARGETUSER::7
TEXT::Heavy swapping observed on system
CAUSE::Memory usage by one more application is leading to heavy swapping
ACTION::Check alert log for more information, use tools to analyze memory usage and take
action
STARTVERSION::21.1

13
Example Attention Log Curated Message – Server Warning
[
SOON Heavy swapping observed on system
CAUSE: Memory usage by one more application is leading to heavy swapping
ACTION: Check alert log for more information, use tools to analyze memory
usage and take action
CLASS: CDB Instance / SERVER ADMINISTRATOR / WARNING / AL-2100
TIME: 2020-05-01T11:09:02.223-07:00
ADDITIONAL INFO: -
WARNING: Heavy swapping observed on system in last 15 mins.
Heavy swapping can lead to timeouts, poor performance, and instance eviction.
]

14
Attention Log Use Cases – Autonomous Health Framework
Autonomous Health
Framework
Trace File Analyzer
…
… CDB DBA
Sys Admin
PDB DBA
Attention Log
Repository

15
Attention Log Use Cases – 3rd Party Monitoring
3rd Party Monitoring,
Parsing and Routing
…
…
Attention Log
Management VCN
Ops Dashboard
Ops Admin
attention.amb Message Definitions
Runbooks

Analyze logs and look for errors
using AHF

Investigate logs and look for errors
tfactl analyze -since 1d
INFO: analyzing all (Alert and Unix System Logs) logs for the last 1440
minutes...
...
Unique error messages for last ~1 day(s)
Occurrences percent server name error
----------- ------- -------------------- -----
1 100.0% myserver1 Errors in file
/u01/oracle/diag/rdbms/orcl2/orcl2/trace/orcl2_ora_12272.trc
(incident=10151):
ORA-00600: internal error code, arguments: [600], [], [], [], [], [], [], [],
[], [], [], []
Incident details in:
/u01/oracle/diag/rdbms/orcl2/orcl2/incident/incdir_10151/orcl2_ora_12272_i101
51.trc
...

tfactl analyze -search "ORA-04031" -last 1d
INFO: analyzing all (Alert and Unix System Logs) logs for the last 1440
minutes...
...
Matching regex: ORA-04031
Case sensitive: false
Match count: 1
[Source: /u01/oracle/diag/rdbms/orcl2/orcl2/trace/alert_orcl2.log, Line: 1941]
Oct 01 12:09:05 2020
Errors in file /u01/oracle/diag/rdbms/orcl2/orcl2/trace/orcl2_ora_6982.trc
(incident=7665):
ORA-04031: unable to allocate bytes of shared memory ("","","","")
Incident details in:
/u01/app/oracle/diag/rdbms/orcl2/orcl2/incident/incdir_7665/orcl2_ora_6982_i76
65.trc
...

Examples
tfactl analyze -since 5h
#Show summary of events from alert logs,
system messages in last 5 hours
tfactl analyze -comp os -since 1d
#Show summary of events from system
messages in last 1 day
tfactl analyze -search "ORA-" -since 2d
#Search string ORA- in alert and system
logs in past 2 days
tfactl analyze -search "/Starting/c" -
since 2d
#Search case sensitive string "Starting"
in past 2 days
tfactl analyze -comp os -for ”Oct/01/2020
11" -search "."
#Show all system log messages at time
Oct/01/2020 11
tfactl analyze -comp osw -since 6h
#Show OSWatcher Top summary in last 6
hours
tfactl analyze -comp oswslabinfo -from
”Oct/01/2020 05:00:01" -to ”Oct/01/2020
06:00:01"
#Show OSWatcher slabinfo summary for
specified time period
tfactl analyze -since 1h -type generic
#Analyze all generic messages in last one
hour

$ ./tfactl analyze -type generic -since 7d
INFO: analyzing all (Alert and Unix System Logs) logs for the last 10080 minutes...
...
Total message count: 54,807, from 01-Oct-2020 02:41:34 PM PST to
08-Oct-2020 02:41:34
Messages matching last ~7 day(s): 3,139, from 02-Oct-2020 02:46:23 PM PST to
08-Oct-2020 02:41:34
last ~7 day(s) generic count: 3,139, from 06-Oct-2020 02:46:23 PM PST to
08-Oct-2020 02:41:34
last ~7 day(s) unique generic count: 94
Message types for last ~7 day(s)
Occurrences percent server name type
----------- ------- -------------------- -----
3,139 100.0% myhost1 generic
...

Unique generic messages for last ~7 day(s)
Occurrences percent server name generic
----------- ------- -------------------- -----
1,504 47.9% myhost1 : [crflogd(13931)]CRS-9520:The storage of Grid
Infrastructure Managem...
487 15.5% myhost1 : [crflogd(13931)]CRS-9520:The storage of Grid
Infrastructure Managem...
336 10.7% myhost1 myhost1 smartd[13812]: Device: /dev/sdv, SMART
Failure: FAILURE...
336 10.7% myhost1 myhost1 smartd[13812]: Device: /dev/sdag, SMART
Failure: FAILURE ...
103 3.3% myhost1 myhost1 last message repeated 9 times
103 3.3% myhost1 myhost1 kernel: oracle: sending ioctl 2285 to a
partition!
...snipping for brevity...

Pattern match search output
tfactl analyze -search "ORA-" -since 7d
...
[Source: /u01/app/oracle/diag/rdbms/ratoda/RATODA1/trace/alert_RATODA1.log, Line:
9494]
Feb 25 22:00:02 2014
Errors in file
/u01/app/oracle/diag/rdbms/ratoda/RATODA1/trace/RATODA1_j003_10948.trc:
ORA-12012: error on auto execute of job "ORACLE_OCM"."MGMT_CONFIG_JOB_2_1"
ORA-29280: invalid directory path
ORA-06512: at "ORACLE_OCM.MGMT_DB_LL_METRICS", line 2436
ORA-06512: at line 1
End automatic SQL Tuning Advisor run for special tuning task
"SYS_AUTO_SQL_TUNING_TASK”
...

OS Watcher top data
tfactl analyze -comp osw -since 6h
...
statistic: t first highest (time) lowest (time) average non zero 3rd last 2nd last last trend
top.cpu.util.id: % 98.0 99.7 @10:35AM 72.8 @03:11PM 97.3 2,059 95.2 96.8 96.0 -2%
top.cpu.util.st: % 0.1 0.1 @09:14AM 0.0 @09:14AM 0.0 889 0.0 0.0 0.0 -100%
top.cpu.util.us: % 0.1 8.8 @11:31AM 0.0 @09:14AM 0.6 1,966 4.3 0.8 3.4 3300%
top.cpu.util.wa: % 1.7 18.7 @03:11PM 0.1 @10:35AM 1.1 2,059 0.3 0.4 0.4 -76%
top.loadavg.last01min: 1.17 3.12 @09:44AM 0.07 @12:45PM 0.93 1,823 0.31 0.26 0.22 -81%
top.loadavg.last05min: 0.94 2.26 @09:44AM 0.27 @12:45PM 0.93 1,823 0.82 0.79 0.77 -18%
top.loadavg.last15min: 0.79 1.60 @09:46AM 0.44 @01:18PM 0.92 1,823 0.96 0.95 0.94 18%
top.mem.buffers: k 808232 808388 @09:41AM 785608 @02:57PM 796511 2,093 785744 785744 785744 -2%
top.mem.free: k 1130332 1291344 @10:02AM 927576 @09:43AM 1188576 2,093 1244020 1265248 1265188 11%
top.swap.used: k 47556 48088 @03:00PM 47556 @09:14AM 47828 2,097 48088 48088 48088 1%
top.tasks.running: 1 4 @12:04PM 1 @09:14AM 1 1,996 1 2 2 100%
top.tasks.total: 514 527 @02:57PM 509 @09:18AM 514 1,996 518 521 520 1%
top.tasks.zombie: 0 5 @11:04AM 0 @09:14AM 0 62 0 0 0 n/a
top.users: 5 6 @03:00PM 5 @09:14AM 5 1,823 6 6 6 20%
...

OS Watcher slabinfo data
tfactl analyze -comp oswslabinfo -from ”Oct/01/2020 05:00:01" -to ”Oct/01/2020 06:00:01"
...
statistic: t first highest (time) lowest (time) average non zero 3rd last 2nd last last trend
slabinfo.acfs_ccb_cache.active_objs: 4 38 @05:52AM 0 @05:01AM 10 294 3 1 8 100%
slabinfo.inet_peer_cache.active_objs: 23 39 @05:59AM 23 @05:00AM 23 351 23 23 39 69%
slabinfo.sigqueue.active_objs: 385 768 @05:28AM 285 @05:27AM 554 351 712 621 577 49%
slabinfo.skbuff_fclone_cache.active_objs: 55 133 @05:51AM 11 @05:20AM 69 351 56 77 70 27%
slabinfo.names_cache.active_objs: 126 180 @05:00AM 110 @05:23AM 146 351 171 166 156 23%
slabinfo.sgpool-8.active_objs: 135 228 @05:31AM 59 @05:11AM 152 351 180 165 157 16%
slabinfo.UDP.active_objs: 568 675 @05:28AM 492 @05:17AM 597 351 630 596 626 10%
slabinfo.size-8192.active_objs: 174 209 @05:36AM 160 @05:14AM 181 351 205 187 188 8%
slabinfo.task_delay_info.active_objs: 1477 1856 @05:28AM 1334 @05:57AM 1574 351 1529 1411 1579 6%
slabinfo.pid.active_objs: 1608 1980 @05:29AM 1452 @05:21AM 1678 351 1564 1487 1689 5%
slabinfo.blkdev_requests.active_objs: 720 880 @05:04AM 651 @05:54AM 745 351 707 736 761 5%
slabinfo.ip_dst_cache.active_objs: 1497 1800 @05:28AM 1279 @05:36AM 1517 351 1594 1466 1560 4%
slabinfo.sock_inode_cache.active_objs: 2168 2329 @05:11AM 2106 @05:56AM 2225 351 2322 2278 2232 2%
...

How to connect to a hung
database for diagnostics

How do you connect to a database when connections are hanging?
• sqlplus preliminary connection will connect to database since no session is created
• You will have limited access to the SGA
• This will help in capturing diagnostic information like a systemstate dump
• Two ways to connect to sqlplus using a preliminary connection:
or
sqlplus -prelim
sqlplus -prelim / as sysdba
SQL> set _prelim on
SQL> connect / as sysdba
Prelim connection established

Always on - Enabled by default
Reliably detects database hangs and deadlocks
Autonomously resolves them
Logs all detections and resolutions
New SQL interface to configure sensitivity (Normal/High) and
trace file sizes
Oracle Hang Manager
Session
DIA0
EVALUATE
DETECT
ANALYZE
Hung?
VERIFY
Victim
Policy

Monitors Session snapshots for progress
Evaluates potential hangs over time with based
upon Wait Graphs
Analyzes hang chain of sessions to identify
blocker/victim
Discovers blocker is located in ASM instance
Requests ASM terminate session or instance
relying on Flex ASM for recovery
Detection and resolution is bi-directional
Database Hang Management - Infrastructure
Database
ASM

Full Resolution Dump Trace File and DB Alert Log Audit Reports
Oracle 21c Hang Manager
Dump file …/diag/rdbms/hm6/hm62/incident/incdir_5753/hm62_dia0_12656_i5753.trc
Oracle Database 12c Enterprise Edition Release 18/19c.0.0.0 - 64bit Beta
With the Partitioning, Real Application Clusters, OLAP, Advanced Analytics
and Real Application Testing options
Build label: RDBMS_MAIN_LINUX.X64_151013
ORACLE_HOME: …/3775268204/oracle
System name: Linux
Node name: slc05kyr
Release: 2.6.39-400.211.1.el6uek.x86_64
Version: #1 SMP Fri Nov 15 13:39:16 PST 2020
Machine: x86_64
VM name: Xen Version: 3.4 (PVM)
Instance name: hm62
Redo thread mounted by this instance: 2
Oracle process number: 19
Unix process pid: 12656, image: oracle@slc05kyr (DIA0)
*** 2020-10-01T16:47:59.541509+17:00
*** SESSION ID:(96.41299) 2020-10-01T16:47:59.541519+17:00
*** CLIENT ID:() 2020-10-01T16:47:59.541529+17:00
*** SERVICE NAME:(SYS$BACKGROUND) 2020-10-01T16:47:59.541538+17:00
*** MODULE NAME:() 2020-10-01T16:47:59.541547+17:00
*** ACTION NAME:() 2020-10-01T16:47:59.541556+17:00
*** CLIENT DRIVER:() 2020-10-01T3T16:47:59.541565+17:00

Full Resolution Dump Trace File and DB Alert Log Audit Reports
2020-10-01T16:47:59.435039+17:00
Errors in file /oracle/log/diag/rdbms/hm6/hm6/trace/hm6_dia0_12433.trc (incident=7353):
ORA-32701: Possible hangs up to hang ID=1 detected
Incident details in: …/diag/rdbms/hm6/hm6/incident/incdir_7353/hm6_dia0_12433_i7353.trc
2020-10-01T16:47:59.506775+17:00
DIA0 requesting termination of session sid:40 with serial # 43179 (ospid:13031) on instance 2
due to a GLOBAL, HIGH confidence hang with ID=1.
Hang Resolution Reason: Automatic hang resolution was performed to free a
significant number of affected sessions.
DIA0: Examine the alert log on instance 2 for session termination status of hang with ID=1.
In the alert log on the instance local to the session (instance 2 in this case),
we see the following:
2020-10-01T16:47:59.538673+17:00
Errors in file …/diag/rdbms/hm6/hm62/trace/hm62_dia0_12656.trc (incident=5753):
ORA-32701: Possible hangs up to hang ID=1 detected
Incident details in: …/diag/rdbms/hm6/hm62/incident/incdir_5753/hm62_dia0_12656_i5753.trc
2020-10-01T16:48:04.222661+17:00
DIA0 terminating blocker (ospid: 13031 sid: 40 ser#: 43179) of hang with ID = 1
requested by master DIA0 process on instance 1
Hang Resolution Reason: Automatic hang resolution was performed to free a
significant number of affected sessions.
by terminating session sid:40 with serial # 43179 (ospid:13031)

Refer MOS Note 1428210.1 for scripts
SQL> SELECT chain_id, num_waiters, in_wait_secs, osid, blocker_osid, substr(wait_event_text,1,30)
FROM v$wait_chains; 2
CHAIN_ID NUM_WAITERS IN_WAIT_SECS OSID BLOCKER_OSID SUBSTR(WAIT_EVENT_TEXT,1,30)
---------- ----------- ------------ ------------------------- ------------------------- -----------------------------
-
1 0 10198 21045 21044 enq: TX - row lock contention
1 1 10214 21044 SQL*Net message from client

Refer MOS Note 1428210.1
SELECT *
FROM (SELECT 'Current Process: '||osid W_PROC, 'SID '||i.instance_name INSTANCE,
'INST #: '||instance INST,'Blocking Process: '||decode(blocker_osid,null,'<none>',blocker_osid)||
' from Instance '||blocker_instance BLOCKER_PROC,'Number of waiters: '||num_waiters waiters,
'Wait Event: ' ||wait_event_text wait_event, 'P1: '||p1 p1, 'P2: '||p2 p2, 'P3: '||p3 p3,
'Seconds in Wait: '||in_wait_secs Seconds, 'Seconds Since Last Wait: '||time_since_last_wait_secs sincelw,
'Wait Chain: '||chain_id ||': '||chain_signature chain_signature,'Blocking Wait Chain:
'||decode(blocker_chain_id,null,
'<none>',blocker_chain_id) blocker_chain
FROM v$wait_chains wc,
v$instance i
WHERE wc.instance = i.instance_number (+)
AND ( num_waiters > 0
OR ( blocker_osid IS NOT NULL
AND in_wait_secs > 10 ) )
ORDER BY chain_id,
num_waiters DESC)
WHERE ROWNUM < 101;

SQL> describe V$DIAG_TRACE_FILE
Name Null? Type
----------------------------------------- -------- ----------------------------
ADR_HOME VARCHAR2(444)
TRACE_FILENAME VARCHAR2(68)
CHANGE_TIME TIMESTAMP(3) WITH TIME ZONE
MODIFY_TIME TIMESTAMP(3) WITH TIME ZONE
CON_ID NUMBER
V$DIAG_TRACE_FILE and V$DIAG_TRACE_FILE_CONTENTS

SQL> describe V$DIAG_TRACE_FILE_CONTENTS
Name Null? Type
----------------------------------------- -------- ----------------------------
RECORD_LEVEL NUMBER
PARENT_LEVEL NUMBER
RECORD_TYPE NUMBER
TIMESTAMP TIMESTAMP(3) WITH TIME ZONE
PAYLOAD VARCHAR2(4000)
SECTION_ID NUMBER
SECTION_NAME VARCHAR2(64)
COMPONENT_NAME VARCHAR2(64)
OPERATION_NAME VARCHAR2(64)
FILE_NAME VARCHAR2(64)
FUNCTION_NAME VARCHAR2(64)
LINE_NUMBER NUMBER
THREAD_ID VARCHAR2(64)
SESSION_ID NUMBER
SERIAL# NUMBER
CON_UID NUMBER
CONTAINER_NAME VARCHAR2(64)
CON_ID NUMBER

SQL> select trace_filename from v$diag_trace_file;
TRACE_FILENAME
--------------------------------------------------------------------
ORCL1_mz00_21108.trc
ORCL1_gcr2_16504.trc
ORCL1_ora_19005.trc

SQL> select payload from v$diag_trace_file_contents where trace_filename ='ORCL1_ora_19005.trc';
PAYLOAD
--------------------------------------------------------------------------------
Trace file /u01/app/oracle/diag/rdbms/orcl_unq/ORCL1/trace/ORCL1_ora_19005.trc
Oracle Database 19c Enterprise Edition Release 19.0.0.0.0 - Production
Version 19.2.0.0.0
Build label: RDBMS_19.2.0.0.0_LINUX.X64_190121
ORACLE_HOME: /u01/app/oracle/product/19c/dbhome_1
System name: Linux
Node name: myserver65
Release: 4.14.35-1844.1.3.el7uek.x86_64
Version: #2 SMP Wed Jan 2 21:18:29 PST 2019
Machine: x86_64
VM name: Xen Version: 4.1 (HVM)
...

...
PAYLOAD
--------------------------------------------------------------------------------
Instance name: ORCL1
Redo thread mounted by this instance: 1
Oracle process number: 12
Unix process pid: 19005, image: oracle@myserver65 (TNS V1-V3)
*** 2020-10-01T01:22:10.770960+00:00
*** SESSION ID:(106.17196) 2020-10-01T01:22:10.771014+00:00
*** CLIENT ID:() 2020-10-01T01:22:10.771027+00:00
*** SERVICE NAME:(SYS$USERS) 2020-10-01T01:22:10.771039+00:00
...

SQL> describe V$DIAG_SESS_SQL_TRACE_RECORDS
Name Null? Type
----------------------------------------- -------- ----------------------------
RECORD_LEVEL NUMBER
PARENT_LEVEL NUMBER
RECORD_TYPE NUMBER
TIMESTAMP TIMESTAMP(3) WITH TIME ZONE
PAYLOAD VARCHAR2(4000)
SECTION_ID NUMBER
SECTION_NAME VARCHAR2(64)
COMPONENT_NAME VARCHAR2(64)
OPERATION_NAME VARCHAR2(64)
FILE_NAME VARCHAR2(64)
FUNCTION_NAME VARCHAR2(64)
LINE_NUMBER NUMBER
THREAD_ID VARCHAR2(64)
SESSION_ID NUMBER
SERIAL# NUMBER
CON_UID NUMBER
CONTAINER_NAME VARCHAR2(64)
CON_ID NUMBER
V$DIAG_SESS_SQL_TRACE_RECORDS

SQL> SELECT sid,serial# FROM v$session WHERE username = 'SYS’;
SID SERIAL#
---------- ----------
33 45888
129 6051
SQL> EXECUTE DBMS_SYSTEM.SET_SQL_TRACE_IN_SESSION(129,6051,TRUE);
PL/SQL procedure successfully completed.
Enable session tracing

SQL> select unique trace_filename from V$DIAG_SESS_SQL_TRACE_RECORDS;
TRACE_FILENAME
--------------------------------------------------------------------
ORCL1_ora_14151.trc
SQL> select payload from V$DIAG_SESS_SQL_TRACE_RECORDS where trace_filename = 'ORCL1_ora_14151.trc';
PAYLOAD
--------------------------------------------------------------------------------
CLOSE #140506358472544:c=19,e=18,dep=0,type=1,tim=7769230586778
=====================
PARSING IN CURSOR #140506358494608 len=97 dep=1 uid=0 oct=3 lid=0 tim=7769230600
163 hv=791757000 ad='7fa0c290' sqlid='87gaftwrm2h68'
select o.owner#,o.name,o.namespace,o.remoteowner,o.linkname,o.subname from obj$
o where o.obj#=:1
END OF STMT
EXEC #140506358494608:c=65,e=65,p=0,cr=0,cu=0,mis=0,r=0,dep=1,og=4,plh=107238262
4,tim=7769230600159
...

...
PAYLOAD
--------------------------------------------------------------------------------
FETCH #140506358494608:c=38,e=37,p=0,cr=2,cu=0,mis=0,r=0,dep=1,og=4,plh=10723826
24,tim=7769230600324
CLOSE #140506358494608:c=5,e=4,dep=1,type=3,tim=7769230600381
EXEC #140506358494608:c=23,e=23,p=0,cr=0,cu=0,mis=0,r=0,dep=1,og=4,plh=107238262
4,tim=7769230600500
FETCH #140506358494608:c=11,e=12,p=0,cr=2,cu=0,mis=0,r=0,dep=1,og=4,plh=10723826
24,tim=7769230600547
...
SQL> EXECUTE DBMS_SYSTEM.SET_SQL_TRACE_IN_SESSION(129,6051,FALSE);
PL/SQL procedure successfully completed.

Keep track of the attribute of
import file pre and post patching

Start tracking using –fileattr start
Automatically discovers Grid Infrastructure and Database directories and files
• Prevent discovery using –excludediscovery
Further configure the list of monitored directories using –includedir
Track attribute changes on important files
tfactl <orachk|exachk> -fileattr start -includedir "/root/myapp/config"
...
List of directories(recursive) for checking file attributes:
/u01/app/oradb/product/11.2.0/dbhome_11203
/root/myapp/config
orachk has taken snapshot of file attributes for above directories at:
/orahome/oradb/orachk/orachk_mysrv21_20201001_041214

Compare current attributes against first snapshot using –fileattr check
When checking, use the same include/exclude arguments you started with
tfactl <orachk|exachk> -fileattr check -includedir "/root/myapp/config”
...
List of directories(recursive) for checking file attributes:
/root/myapp/config
Checking file attribute changes...
"/root/myapp/config/myappconfig.xml" is different:
Baseline : 0644 oracle root
/root/myapp/config/myappconfig.xml
Current : 0644 root root
/root/myapp/config/myappconfig.xml
...

Automatically proceeds to run compliance checks after file attribute checks
• Only run attribute checks by using -fileattronly
File Attribute Changes are shown in HTML report output

Self analysis in MOS using TFA collections

15 Troubleshooting Tips and Tricks for database 21c - OGBEMEA KSAOUG

tfactl diagcollect –srdc <srdc_type>
• Scans system to identify recent events
• Once the relevant event is chosen, proceeds with diagnostic collection
One command SRDC
tfactl diagcollect -srdc ORA-00600
Enter the time of the ORA-00600 [YYYY-MM-DD HH24:MI:SS,<RETURN>=ALL] :
Enter the Database Name [<RETURN>=ALL] :
1. Oct/01/2020 05:29:58 : [orcl2] ORA-00600: internal error code,
arguments: [600], [], [], [], [], [], [], [], [], [], [], []
2. Oct/01/2020 06:55:08 : [orcl2] ORA-00600: internal error code,
arguments: [600], [], [], [], [], [], [], [], [], [], [], []
Please choose the event : 1-2 [1]
Selected value is : 1 (Oct/01/2020 05:29:58 )

All required files are identified
• Trimmed where applicable
• Package in a zip ready to provide to support
One command SRDC
...
2020/10/01 06:14:24 EST : Getting List of Files to Collect
2020/10/01 06:14:27 EST : Trimming file :
myserver1/rdbms/orcl2/orcl2/trace/orcl2_lmhb_3542.trc with original file
size : 163MB
...
2020/10/01 06:14:58 EST : Total time taken : 39s
2020/10/01 06:14:58 EST : Completed collection of zip files.
...
/opt/oracle.ahf/data/repository/srdc_ora600_collection_Tue_Sep_7_06_14_17
_EST_2020_node_local/myserver1.tfa_srdc_ora600_Thu_Oct_1_06_14_17_EST_202
0.zip

Sanitize sensitive information

Sensitive information can be hidden from diagnostics
Machine learning algorithms determine sensitive data like:
• Host names
• IP addresses
• MAC addresses
• Oracle Database names
• Tablespace names
• Service names
• Ports
• Operating system user names
Sanitize or mask sensitive information

Add –sanitize or –mask to any command
• –sanitize replaces a sensitive value with random characters
• myhost123 >>>> JnsF3km9
• –mask replaces a sensitive value with a series of ‘X’
• myhost123 >>>> XXXXXXXX
Sanitize or mask sensitive information

Sanitized hostname
Sanitized hostname
tfactl orachk –preupgrade -sanitize

tfactl orachk -rmap qzh024703246tsa1
TFA using ORAchk : /opt/oracle.ahf/orachk/orachk
___________________________________________________________________________
| Entity Type | Substituted Entity Name | Original Entity Name |
___________________________________________________________________________
| hostname | qzh024703246tsa1 | myserver1 |
___________________________________________________________________________
Reverse map the sanitization

tfactl changes
Output from host : myserver69
------------------------------
[Oct/01/2020 04:54:15.397]: Parameter: fs.aio-nr: Value: 95488 => 97024
[Oct/01/2020 04:54:15.397]: Parameter: fs.inode-nr: Value: 764974 131561 => 740744
131259
[Oct/01/2020 04:54:15.397]: Parameter: kernel.pty.nr: Value: 2 => 1
[Oct/01/2020 04:54:15.397]: Parameter: kernel.random.entropy_avail: Value: 189 =>
158
[Oct/01/2020 04:54:15.397]: Parameter: kernel.random.uuid: Value: 36269877-9bc9-
40a3-82e0-1619865096f2 => 7551c5e7-c59f-40fa-b55f-5bd170e8b1ab
[Oct/01/2020 05:46:15.397]: Parameter: fs.inode-nr: Value: 1580316 810036 =>
1562320 768555
[Oct/01/2020 05:46:15.397]: Parameter: kernel.random.uuid: Value: 37cc31aa-ee31-
459e-8f2a-0766b34b1b64 => f5176cdc-6390-415d-882e-02c4cff2ae4e
...
Has anything changed recently?

...
------------------------------
[Oct/01/2020 04:54:15.397]: Parameter: fs.inode-nr: Value: 764974 131561 => 740744
131259
[Oct/01/2020 04:54:15.397]: Parameter: kernel.random.entropy_avail: Value: 189 =>
158
[Oct/01/2020 04:54:15.397]: Parameter: kernel.random.uuid: Value: 36269877-9bc9-
40a3-82e0-1619865096f2 => 7551c5e7-c59f-40fa-b55f-5bd170e8b1ab
[Oct/01/2020 05:46:15.397]: Parameter: fs.inode-nr: Value: 1580316 810036 =>
1562320 768555
[Oct/01/2020 05:46:15.397]: Parameter: kernel.random.uuid: Value: 37cc31aa-ee31-
459e-8f2a-0766b34b1b64 => f5176cdc-6390-415d-882e-02c4cff2ae4e
Has anything changed recently?

Other Server Technology
Enterprise Manager
Data Guard
GoldenGate
Exalogic
Database areas
Errors / Corruption
Performance
Install / patching / upgrade
RAC / Grid Infrastructure
Import / Export
RMAN
Transparent Data Encryption
Storage / partitioning
Undo / auditing
Listener / naming services
Spatial / XDB
Some problem areas covered in SRDCs
Full list in documentation
Around 100 problem types covered
tfactl diagcollect –srdc <srdc_type>
[-sr <sr_number>]

TFA SRDC
Manual method
Manual collection vs TFA SRDC for database performance
1. Generate ADDM reviewing Document 1680075.1 (multiple steps)
2. Identify “good” and “problem” periods and gather AWR reviewing
Document 1903158.1 (multiple steps)
3. Generate AWR compare report (awrddrpt.sql) using “good” and
“problem” periods
4. Generate ASH report for “good” and “problem” periods reviewing
5. Collect OSWatcher data reviewing Document 301137.1 (multiple steps)
6. Collect Hang Analyze output at Level 4
7. Generate SQL Healthcheck for problem SQL id using Document
1366133.1 (multiple steps)
8. Run support provided sql scripts – Log File sync diagnostic output using
9. Check alert.log if there are any errors during the “problem” period
10. Find any trace files generated during the “problem” period
11. Collate and upload all the above files/outputs to SR
1. Run
tfactl diagcollect –srdc dbperf
[-sr <sr_number>]

All required files are identified
• Trimmed where applicable
• Package in a zip ready to provide to support
One command SRDC
...
2020/10/01 06:14:24 EST : Getting List of Files to Collect
2020/10/01 06:14:27 EST : Trimming file :
myserver1/rdbms/orcl2/orcl2/trace/orcl2_lmhb_3542.trc with original file
size : 163MB
...
2020/10/01 06:14:58 EST : Total time taken : 39s
2020/10/01 06:14:58 EST : Completed collection of zip files.
...
/opt/oracle.ahf/data/repository/srdc_ora600_collection_Thu_Oct_1_06_14_17
_EST_2020_node_local/myserver1.tfa_srdc_ora600_Thu_Oct_1_06_14_17_EST_202
0.zip

TFA can automatically purge database logs
Purging automatically removes logs older than 30 days
• Configurable with
Purging runs every 60 minutes
• Configurable with:
Automatic Database Log Purge
tfactl set manageLogsAutoPurge=ON
tfactl set manageLogsAutoPurgePolicyAge=<n><d|h>
tfactl set manageLogsAutoPurgeInterval=<minutes>

TFA can manage ADR log and trace files
tfactl managelogs <options>
–show usage #Show disk space usage per diagnostic directory for both
GI and database logs
-show variation –older <n><m|h|d> #Show disk space growth for specified period
-purge –older <n><m|h|d> #Remove ADR files older than the time specified
–gi #Restrict command to only files under the GI_BASE
–database [all | dbname] #Restrict command to only files under the database
directory
-dryrun #Use with –purge to estimate how many files will be affected and how
much disk space will be freed by a potential purge command
Manual Database Log Purge

tfactl managelogs -show usage
...
.---------------------------------------------------------------------------------.
| Grid Infrastructure Usage |
+---------------------------------------------------------------------+-----------+
| Location | Size |
+---------------------------------------------------------------------+-----------+
| /u01/app/crsusr/diag/afdboot/user_root/host_309243680_94/alert | 28.00 KB |
| /u01/app/crsusr/diag/afdboot/user_root/host_309243680_94/incident | 4.00 KB |
| /u01/app/crsusr/diag/afdboot/user_root/host_309243680_94/trace | 8.00 KB |
...
+---------------------------------------------------------------------+-----------+
| Total | 739.06 MB |
'---------------------------------------------------------------------+-----------’
...
Understand Database log disk space usage
Use -gi to only show grid infrastructure

...
.---------------------------------------------------------------.
| Database Homes Usage |
+---------------------------------------------------+-----------+
| Location | Size |
+---------------------------------------------------+-----------+
| /u01/app/crsusr/diag/rdbms/cdb674/CDB674/alert | 1.06 MB |
| /u01/app/crsusr/diag/rdbms/cdb674/CDB674/incident | 4.00 KB |
| /u01/app/crsusr/diag/rdbms/cdb674/CDB674/trace | 146.19 MB |
| /u01/app/crsusr/diag/rdbms/cdb674/CDB674/cdump | 4.00 KB |
| /u01/app/crsusr/diag/rdbms/cdb674/CDB674/hm | 4.00 KB |
+---------------------------------------------------+-----------+
| Total | 147.26 MB |
'---------------------------------------------------+-----------'
Understand Database log disk space usage
Use -database to only show database

Understand Database log disk space usage variations
tfactl managelogs -show variation -older 30d
------------------------------
2020-10-01 12:30:42: INFO Checking space variation for 30 days
.---------------------------------------------------------------------------------------------.
| Grid Infrastructure Variation |
+---------------------------------------------------------------------+-----------+-----------+
| Directory | Old Size | New Size |
+---------------------------------------------------------------------+-----------+-----------+
| /u01/app/crsusr/diag/asm/user_root/host_309243680_96/alert | 22.00 KB | 28.00 KB |
+---------------------------------------------------------------------+-----------+-----------+
| /u01/app/crsusr/diag/clients/user_crsusr/host_309243680_96/cdump | 4.00 KB | 4.00 KB |
+---------------------------------------------------------------------+-----------+-----------+
| /u01/app/crsusr/diag/tnslsnr/myserver74/listener/alert | 15.06 MB | 244.10 MB |
+---------------------------------------------------------------------+-----------+-----------+
...

Understand Database log disk space usage variations
...
.---------------------------------------------------------------------------.
| Database Homes Variation |
+---------------------------------------------------+-----------+-----------+
| Directory | Old Size | New Size |
+---------------------------------------------------+-----------+-----------+
| /u01/app/crsusr/diag/rdbms/cdb674/CDB674/hm | 4.00 KB | 4.00 KB |
+---------------------------------------------------+-----------+-----------+
| /u01/app/crsusr/diag/rdbms/cdb674/CDB674/trace | 16.63 MB | 146.19 MB |
+---------------------------------------------------+-----------+-----------+
| /u01/app/crsusr/diag/rdbms/cdb674/CDB674/cdump | 4.00 KB | 4.00 KB |
+---------------------------------------------------+-----------+-----------+
| /u01/app/crsusr/diag/rdbms/cdb674/CDB674/incident | 4.00 KB | 4.00 KB |
+---------------------------------------------------+-----------+-----------+
| /u01/app/crsusr/diag/rdbms/cdb674/CDB674/alert | 1.06 MB | 1.06 MB |
'------------------------------------------------------------+-------------+-------------'

Run a database log purge dryrun
tfactl managelogs -purge -older 30d -dryrun
------------------------------
Estimating files older than 30 days
Estimating purge for diagnostic destination "diag/afdboot/user_root/host_309243680_94" for files ~ 2 files deleted , 22.58 KB freed ]
Estimating purge for diagnostic destination "diag/afdboot/user_crsusr/host_309243680_94" for files ~ 2 files deleted , 11.72 KB freed ]
Estimating purge for diagnostic destination "diag/asmtool/user_root/host_309243680_96" for files ~ 2 files deleted , 21.36 KB freed ]
Estimating purge for diagnostic destination "diag/asmtool/user_crsusr/host_309243680_96" for files ~ 3 files deleted , 23.22 KB freed ]
Estimating purge for diagnostic destination "diag/tnslsnr/myserver74/listener" for files ~ 23 files deleted , 225.33 MB freed ]
Estimating purge for diagnostic destination "diag/diagtool/user_root/adrci_309243680_96" for files ~ 73 files deleted , 517.69 KB freed ]
Estimating purge for diagnostic destination "diag/clients/user_crsusr/host_309243680_96" for files ~ 38 files deleted , 17.15 KB freed ]
Estimating purge for diagnostic destination "diag/asm/+asm/+ASM" for files ~ 0 files deleted , 0 bytes freed ]
Estimating purge for diagnostic destination "diag/asm/user_root/host_309243680_96" for files ~ 1 files deleted , 19.52 KB freed ]
Estimating purge for diagnostic destination "diag/asm/user_crsusr/host_309243680_96" for files ~ 1 files deleted , 20.25 KB freed ]
Estimating purge for diagnostic destination "diag/crs/myserver74/crs" for files ~ 40 files deleted , 219.39 MB freed ]
Estimation for Grid Infrastructure [ Files to delete : ~ 185 files | Space to be freed : ~ 445.36 MB ]
Estimating purge for diagnostic destination "diag/rdbms/cdb674/CDB674" for files ~ 27760 files deleted , 66.57 MB freed ]
Estimation for Database Home [ Files to delete : ~ 27760 files | Space to be freed : ~ 66.57 MB ]

Run a database log purge
tfactl managelogs -purge -older 30d
------------------------------
Purging files older than 30 days
Cleaning Grid Infrastructure destinations
Purging diagnostic destination "diag/afdboot/user_root/host_309243680_94" for files - 0 files deleted , 0 bytes freed
Purging diagnostic destination "diag/afdboot/user_crsusr/host_309243680_94" for files - 1 files deleted , 10.16 KB freed
Purging diagnostic destination "diag/asmtool/user_root/host_309243680_96" for files - 1 files deleted , 10.16 KB freed
Purging diagnostic destination "diag/asmtool/user_crsusr/host_309243680_96" for files - 2 files deleted , 29.18 KB freed
Purging diagnostic destination "diag/tnslsnr/myserver74/listener" for files - 2 files deleted , 29.18 KB freed
Purging diagnostic destination "diag/diagtool/user_root/adrci_309243680_96" for files - 2 files deleted , 29.18 KB freed
Purging diagnostic destination "diag/clients/user_crsusr/host_309243680_96" for files - 2 files deleted , 29.18 KB freed
Purging diagnostic destination "diag/asm/+asm/+ASM" for files - 2 files deleted , 29.18 KB freed
Purging diagnostic destination "diag/asm/user_root/host_309243680_96" for files - 2 files deleted , 29.18 KB freed
Purging diagnostic destination "diag/asm/user_crsusr/host_309243680_96" for files - 2 files deleted , 29.18 KB freed
Purging diagnostic destination "diag/crs/myserver74/crs" for files - 2 files deleted , 29.18 KB freed
...

Run a database log purge
...
Grid Infrastructure [ Files deleted : 18 files | Space Freed : 253.75 KB ]
.-----------------------------------------------------------------------------------------------.
| File System Variation : /u01/app/crsusr/12.2.0/grid2 |
+--------+-----------------------------------+----------+----------+---------+----------+-------+
| State | Name | Size | Used | Free | Capacity | Mount |
+--------+-----------------------------------+----------+----------+---------+----------+-------+
| Before | /dev/mapper/vg_rws1270665-lv_root | 51475068 | 46597152 | 2256476 | 96% | / |
| After | /dev/mapper/vg_rws1270665-lv_root | 51475068 | 46597152 | 2256476 | 96% | / |
'--------+-----------------------------------+----------+----------+---------+----------+-------'

tail files
tfactl tail alert
------------------------------
/scratch/app/11.2.0.4/grid/log/myserver69/alertmyserver69.log
2020-10-01 23:28:22.532:
[ctssd(5630)]CRS-2409:The clock on host myserver69 is not synchronous with
the mean cluster time. No action has been taken as the Cluster Time
Synchronization Service is running in observer mode.
2020-10-01 23:58:22.964:
[ctssd(5630)]CRS-2409:The clock on host myserver69 is not synchronous with
the mean cluster time. No action has been taken as the Cluster Time
Synchronization Service is running in observer mode.
...

tail files
...
/scratch/app/oradb/diag/rdbms/apxcmupg/apxcmupg_2/trace/alert_apxcmupg_2.log
Thu Oct 01 06:00:00 2020 VKRM started with pid=82, OS id=4903
Thu Oct 01 06:00:02 2020 Begin automatic SQL Tuning Advisor run for special
tuning task "SYS_AUTO_SQL_TUNING_TASK"
Thu Oct 01 06:00:37 2020 End automatic SQL Tuning Advisor run for special
tuning task "SYS_AUTO_SQL_TUNING_TASK"
Thu Oct 01 23:00:28 2020 Thread 2 advanced to log sequence 759 (LGWR switch)
Current log# 3 seq# 759 mem# 0:
+DATA/apxcmupg/onlinelog/group_3.289.917164707
Current log# 3 seq# 759 mem# 1:
+FRA/apxcmupg/onlinelog/group_3.289.917164707
...

tail files
...
/scratch/app/oradb/diag/rdbms/ogg11204/ogg112041/trace/alert_ogg112041.log
Clearing Resource Manager plan via parameter
Thu Oct 01 05:59:59 2020
Setting Resource Manager plan DEFAULT_MAINTENANCE_PLAN via parameter
Thu Oct 01 05:59:59 2020
Starting background process VKRM
Thu Oct 01 05:59:59 2020
VKRM started with pid=36, OS id=4901
Thu Oct 01 22:00:31 2020
Thread 1 advanced to log sequence 305 (LGWR switch)
Current log# 1 seq# 305 mem# 0: +DATA/ogg11204/redo01.log
...

tail files
...
/scratch/app/oragrid/diag/asm/+asm/+ASM1/trace/alert_+ASM1.log <==
Thu Oct 01 04:42:22 2020
NOTE: [ocrcheck.bin@myserver69 (TNS V1-V3) 2323] opening OCR file
Thu Oct 01 01:05:39 2020
Thu Oct 01 01:05:41 2020
Thu Oct 01 01:21:12 2020
Thu Oct 01 01:21:12 2020
...

Monitor on-prem database
performance

Near real-time Database monitoring
• Single instance & RAC
• Monitoring current database activities
• Database performance
• Identifying contentions and bottleneck
• Process & SQL Monitoring
• Real time wait events
• Active Data Guard support
• Multitenant Database (CDB) support
oratop (Support Tools Bundle)

Monitor Database performance
tfactl run oratop -database ogg19c

Section 1 DATABASE:
Global database
information
Section 2 INSTANCE:
Database instance Activity
Section 3 EVENT: AWR like
“Top 5 Timed Events“
Section 4 PROCESS | SQL:
Processes or SQL mode
information
Monitor Database performance
more info 1500864.1

Generates view of Cluster and Database diagnostic metrics
• Always on - Enabled by default
• Provides Detailed OS Resource Metrics
• Assists Node eviction analysis
• Locally logs all process data
• User can define pinned processes
• Listens to CSS and GIPC events
• Categorizes processes by type
• Supports plug-in collectors (ex. traceroute, netstat, ping,
etc.)
• New CSV output for ease of analysis
Cluster Health Monitor (CHM)
GIMR
ologgerd
(master)
osysmond
osysmond
osysmond
osysmond
12c Grid Infrastructure
Management Repository
OS Data OS Data
OS Data
OS Data

Cluster Health Monitor (CHM)
Confidential – Oracle Internal/Restricted/Highly Restricted
Oclumon CLI or full integration
with EM Cloud Control

Always on - Enabled by default
Detects node and database performance problems
Provides early-warning alerts and corrective action
Supports on-site calibration to improve sensitivity
Integrated into EMCC Incident Manager and notifications
Standalone Interactive GUI Tool
Cluster Health Advisor (CHA)*
OS Data
GIMR
ochad
DB Data
CHM
Node
Health
Prognostics
Engine
Database
Health
Prognostics
Engine
* Requires and Included with RAC or R1N License

Choosing a Data Set for Calibration – Defining “normal”
Calibrating CHA to your RAC deployment
chactl query calibration –cluster –timeranges ‘start=2020-10-01
07:00:00,end=2020-10-01 13:00:00’
Cluster name : mycluster
Start time : 2020-10-01 07:00:00
End time : 2020-10-01 13:00:00
Total Samples : 11524
Percentage of filtered data : 100%
1) Disk read (ASM) (Mbyte/sec)
MEAN MEDIAN STDDEV MIN MAX
0.11 0.00 2.62 0.00 114.66
<25 <50 <75 <100 >=100
99.87% 0.08% 0.00% 0.02% 0.03%
...

...
2) Disk write (ASM) (Mbyte/sec)
0.01 0.00 0.15 0.00 6.77
<50 <100 <150 <200 >=200
100.00% 0.00% 0.00% 0.00% 0.00%
...

...
3) Disk throughput (ASM) (IO/sec)
2.20 0.00 31.17 0.00 1100.00
<5000 <10000 <15000 <20000 >=20000
100.00% 0.00% 0.00% 0.00% 0.00%
4) CPU utilization (total) (%)
9.62 9.30 7.95 1.80 77.90
<20 <40 <60 <80 >=80
92.67% 6.17% 1.11% 0.05% 0.00%
...

Create and store a new model
Begin using the new model
Confirm the new model is working
chactl query calibrate cluster –model daytime –timeranges
‘start=2020-10-01 07:00:00, end= 2020-10-01 13:00:00’
chactl monitor cluster –model daytime
chactl status –verbose
monitoring nodes svr01, svr02 using model daytime
monitoring database qoltpacdb, instances oltpacdb_1, oltpacdb_2 using
model DEFAULT_DB

Enable CHA monitoring on RAC database with optional model
Enable CHA monitoring on RAC database with optional verbose
Command line operations
chactl monitor database –db oltpacdb [-model model_name]
chactl status –verbose
monitoring nodes svr01, svr02 using model DEFAULT_CLUSTER
monitoring database oltpacdb, instances oltpacdb_1, oltpacdb_2 using
model DEFAULT_DB

Check for Health Issues and Corrective Actions with CHACTL QUERY DIAGNOSIS
chactl query diagnosis -db oltpacdb -start "2020-10-01 01:42:50" -end "2020-10-01 03:19:15"
2020-10-01 01:47:10.0 Database oltpacdb DB Control File IO Performance (oltpacdb_1) [detected]
2020-10-01 02:59:35.0 Database oltpacdb DB Log File Switch (oltpacdb_1) [detected]
Problem: DB Control File IO Performance
Description: CHA has detected that reads or writes to the control files are slower than expected.
Cause: The Cluster Health Advisor (CHA) detected that reads or writes to the control files were
slow because of an increase in disk IO.
The slow control file reads and writes may have an impact on checkpoint and Log Writer (LGWR)
performance.
Action: Separate the control files from other database files and move them to faster disks or Solid
State Devices.
Problem: DB Log File Switch
Description: CHA detected that database sessions are waiting longer than expected
for log switch completions.
Cause: The Cluster Health Advisor (CHA) detected high contention during log switches
because the redo log files were small and the redo logs switched frequently.
Action: Increase the size of the redo logs.

HTML diagnostic health output available (-html <file_name>)

Diagnose cluster health
chactl query diagnosis -db oltpacdb -start ”2020-10-01 01:42:50.0" -end " 2020-10-01 03:19:15.0"
2020-10-01 02:52:15.0 Database oltpacdb DB CPU Utilization (oltpacdb_2) [detected]
2020-10-01 02:52:50.0 Database oltpacdb DB CPU Utilization (oltpacdb_1) [detected]

How to view Patch Details in
Autonomous DB

View Patch Details
View Autonomous Database maintenance event history to see details about past maintenance events
(requires ADMIN user)
OVERVIEW

View Patch Details HOW IT WORKS
SELECT * FROM DBA_CLOUD_PATCH_INFO;
SELECT * FROM DBA_CLOUD_PATCH_INFO WHERE PATCH_VERSION = 'ADBS-21.7.1.2';

Performance monitoring with
Autonomous Database

• Monitor health, capacity, performance of ADB instances
• Uses metrics, alarms, and notifications
• Metrics accessible via OCI console or using APIs
Monitor Performance with ADB Metrics

1. CPU Utilization
2. Memory Utilization
3. Sessions
4. Failed Connections
5. Execute Count
6. Queued Statements
7. Running Statements
8. Failed Logons
9. Current Logons
10. Transaction Count
11. User Calls
12. Parse Count (Total)
Available Service Metrics

Autonomous Database Details page
provides top 6 view of library of service
metrics.
Viewing Top 6 Metrics on ADB
Console

Complete library of
only metrics
available via
the OCI Console
Service Metrics page
or by using the
Monitoring API
Viewing Full Library Database Metrics

Setting Alarms with the
Autonomous Database

Step 1 – Select the required compartment
Setting Up Monitoring Page

Step 2 – Select the metric namespace for autonomous database
Setting Up Monitoring Page

Performance Hub on Autonomous
Database

127 Confidential – Oracle Internal
Time Range selector is displayed on the top of the Performance Hub page
Use the Select Duration field to set the time duration
Default, Last 60 mins is selected
• Specify to view Last 8 hours, Last 24 hours, Last week
• Specify a custom time range
Time Range field shows active sessions in chart form.
• Active sessions chart displays avg number of active sessions broken down by CPU, User I/O,
and Wait.
Time Range field and time slider

128 Confidential – © 2019 Oracle Internal/Restricted/Highly Restricted
A time zone control has been added to
Performance Hub.
This will allow users to view data using an
alternate time zone to UTC.
They will have the choice of viewing data
using either UTC (default), the time zone
of their browser client, or the time zone
of the database.
Timezone Selector

Shows Active Session History (ASH) analytics charts to explore ASH data
Drill down into database performance across multiple dimensions such as Consumer Group, Wait
Class, SQL ID, and User Name
Select an Average Active Sessions dimension and view the top activity for that dimension for the
selected time period.
For information on ASH, see Active Session History (ASH) in Oracle Database Concepts.
Active Session History (ASH) Analytics

SQL only monitored if running for at least five seconds or run in parallel
Displays monitored SQL statement executions by dimensions including Last Active Time, CPU Time, and Database
Time
Displays currently running SQL statements and SQL statements that completed, failed, or were terminated.
Information includes Status, Duration, and SQL ID
• Status column has the following icons:
• A spinning icon indicates that the SQL statement is executing.
• A green check indicates SQL statement completed
• A red cross icon indicates that the SQL statement did not complete
• A clock icon indicates that the SQL statement is queued
SQL Monitoring

View of a slightly busier system…..

Performance Hub has been expanded to include a new tab, "Workload". This can be used to monitor the workload in
the database and helps the user identify performance spikes and bottlenecks.
Workload Metrics on Performance Hub
There are 4 regions:
Each region contains a
group of charts that
indicate the
characteristics of the
workload and the
distribution of the
resources, as shown

Download AWR reports from
Performance Hub

Copyright © 2019 Oracle and/or its affiliates.
1. Open the navigation menu. Under Database, click Autonomous
Transaction Processing or Autonomous Data Warehouse.
2. Choose your Compartment.
3. In the list of Autonomous Databases, click the display name of the
database you want to analyze using Performance Hub reports.
4. Click Performance Hub.
1) Navigate to Performance Hub in Oracle Cloud Infrastructure Console

2) Adjust the Time Range selector

2. Click the Reports drop-down to display a list of available reports.
2. Click AWR to display the Generate AWR Report dialog.
3. You can choose to generate a report either from two
snapshots closest to the current time and date or from a
custom time range of your choice.
• If you choose the default time range, then select
Use the two snapshots closest to date and click Download.
• If you choose to generate a report from a custom time range,
then select Custom and select start and end times for your
range. Click Download.
2. Oracle Database generates a report named AWRReport_date_range.html that downloads to the
default download folder for your browser. View the report after the download completes.
3) Download the AWR Report

SQL to access Tenancy Details in
Cloud

178
When you file a service request for Autonomous Database, you need to provide the tenancy details for your
instance. Tenancy details for the instance are available on the Oracle Cloud Infrastructure console.
However, if you are connected to the database, you can now obtain these details by querying
the CLOUD_IDENTITY column of the V$PDBS view. For example:
...will generate something similar to the following:
How to get the tenancy details for your instance
SELECT cloud_identity FROM v$pdbs;
{"DATABASE_NAME" : "DBxxxxxxxxxxxx",
"REGION" : "us-phoenix-1",
"TENANT_OCID" :
"OCID1.TENANCY.REGION1..ID1",
"DATABASE_OCID" :
"OCID1.AUTONOMOUSDATABASE.OC1.SEA.ID2
",
"COMPARTMENT_OCID" :
"ocid1.tenancy.region1..ID3"}

VP AIOps for the Autonomous Database
Sandesh Rao
KSAOUG – Oct 2021
Thank you
@sandeshr
https://www.linkedin.com/in/raosandesh/
https://www.slideshare.net/SandeshRao4

15 Troubleshooting Tips and Tricks for database 21c - OGBEMEA KSAOUG

Recommended

More Related Content

What's hot (20)

Similar to 15 Troubleshooting Tips and Tricks for database 21c - OGBEMEA KSAOUG (20)

More from Sandesh Rao (20)

Recently uploaded (20)

15 Troubleshooting Tips and Tricks for database 21c - OGBEMEA KSAOUG