Nov.

3rd, 2007

ORACLE 11g Automatic Diagnostic Repository

Fault Management Infrastructure

Inderpal S. Johal,
President

Agenda

11g Fault Management Framework Overview

11g Fault Management Framework Components

ADR Overview

ADR location

ADR Layout

ADRCI command Line utility

Alert Logs in ADR

Trace & Dump files in ADR

OEM Support Workbench

Q&A

Oracle 11g ADR

11g Fault Management Framework

Prevent, detect, diagnose and resolve database problems

Main goal is to Reduce TIME of Resolution

Prevention

 Change Assurance Lower the risk of change

 Automatic Health Check using Oracle Configuration
Manager and is part of Metalink service

Resolution

 Reduce Diagnostic Time

 Reduce Resolution Time
 Proactive Patching

Oracle 11g ADR

Oracle Configuration Manager

Oracle 11g ADR

11g Fault Mgmt. Framework - Prevention

Oracle 11g ADR

11g Fault Mgmt. Framework - Prevention

Oracle 11g ADR

11g Fault Mgmt. Framework - Prevention

Reduce Diagnostic Time

Critical Error

Stored in
Automatic
Diagnostic
Repository

Sent to

ADRCI
OEM

No

DBA

Is it
Known
Bug

Package Incident
Diagnostic Files

Yes

Apply
Patches

DBA
Oracle 11g ADR

11g Fault Mgmt. Framework Components

Automatic Diagnostic Repository (ADR)

ADRCI Command Line Utility

Alert Log

Trace Files, Dumps, and Core Files

Enterprise Manager Support Workbench

Oracle 11g ADR

Automatic Diagnostic Repository - Overview

ADR is the core of 11g Fault Management Framework

Black box for Database error handling

File-based repository for database diagnostic data

 Alert Log
 Core Dumps and Traces and more

Centralized repository for

 CRS
 ASM
 Database Instances
 Other Components like Listener

Can be accessed by EM and new command line utility ADRCI

Oracle 11g ADR

Automatic Diagnostic Repository - Location

11g Deprecated old initialization parameters

[BACKGROUND|USER|CORE]_DUMP_DEST

11g new initialization parameter

DIAGNOSTIC_DEST

Oracle 11g ADR

10

Automatic Diagnostic Repository - Location

If
DIAGNOSTIC_DEST
is set

YES

DIAGNOSTIC_DEST=As Defined

NO

If
ORACLE_BASE
is set

YES

DIAGNOSTIC_DEST=ORACLE_BASE

NO

DIAGNOSTIC_DEST=ORACLE_HOME/log

Oracle 11g ADR

11

Automatic Diagnostic Repository - Location

SQL> show parameter diag
NAME
TYPE
VALUE
------------------------------------ ----------- -----------------------------diagnostic_dest
string
/oracle

SQL> alter system set diagnostic_dest='/oracle/III';

diagnostic_dest='/oracle/III';
alter system set diagnostic_dest='/oracle/III'
diagnostic_dest='/oracle/III'
*
ERROR at line 1:
ORAinvalid
nvalid
ORA-02097: parameter cannot be modified because specified value is i
ORAORA-48108: invalid value given for the diagnostic_dest init.ora parameter
ORA[/oracle/III]
III]
ORA-48140: the specified ADR Base directory does not exist [/oracle/
ORAORA-48187: specified directory does not exist
Linux Error: 2: No such file or directory
Additional information: 1

SQL> !mkdir
!mkdir /oracle/III
SQL> alter system set diagnostic_dest='/oracle/III';
diagnostic_dest='/oracle/III';

Oracle 11g ADR

12

Automatic Diagnostic Repository - Layout

ADR Base

= DIAGNOSTIC_DEST
$ORACLE_BASE
$ORACLE_HOME/log

diag

crs

stage

asm

clients

rdbms

ofm

tnslsnr

+asm

racdb

hostname

+ASM1

racdb1

listener

hm

cdump

alert

trace

incident

lsnrctl

diagtool

= ADR Home

incpkg

Oracle 11g ADR

metadata

13

Automatic Diagnostic Repository - Layout

$ adrci
ADRCI: Release 11.1.0.6.0 - Beta on
Copyright (c) 2007, Oracle. All rights reserved.
ADR base = "/oracle
"/oracle

adrci>

ADR Base
diag

crs

asm

clients

rdbms

ofm

tnslsnr

diagtool

lsnrctl

racdb
racdb1 = ADR Home

stage

hm

cdump

alert

trace

incident

incpkg

metadata

SELECT *
FROM
V$DIAG_INFO

Oracle 11g ADR

14

Automatic Diagnostic Repository - Layout

SQL> select * from v$diag_info order by name;
INST_ID NAME
VALUE
---------- ------------------------- ----------------------------------------------------------2 ADR Base
/oracle
/oracle/diag/rdbms/sunny/sunny2
2
2 ADR Home
/oracle/diag/rdbms/sunny/sunny
2 Active Incident Count
0
2 Active Problem Count
0
/oracle/diag/rdbms/sunny/sunny2/trace/sunny2_ora_18868.trc
2/trace/sunny2_ora_18868.trc
2 Default Trace File
/oracle/diag/rdbms/sunny/sunny
2 Diag Alert
/oracle/diag/rdbms/sunny/sunny2/alert
2 Diag Cdump
/oracle/diag/rdbms/sunny/sunny2/cdump
2 Diag Enabled
TRUE
2 Diag Incident
/oracle/diag/rdbms/sunny/sunny2/incident
2 Diag Trace
/oracle/diag/rdbms/sunny/sunny2/trace
/oracle/diag/rdbms/sunny/sunny2/hm
2/hm
2 Health Monitor
/oracle/diag/rdbms/sunny/sunny
11 rows selected.

Oracle 11g ADR

15

11g Fault Mgmt. Framework Components

Automatic Diagnostic Repository (ADR)

ADRCI Command Line Utility

Alert Log

Trace Files, Dumps, and Core Files

Enterprise Manager Support Workbench

ADRCI Command Line Utility

Oracle 11g ADR

16

ADR ADRCI utility

$ adrci
ADRCI: Release 11.1.0.6.0 - Beta on Wed Oct 31 13:16:53 2007
Copyright (c) 1982, 2007, Oracle. All rights reserved.
ADR base = "/oracle"

adrci> show base

ADR base is "/oracle

adrci> show home

ADR Homes:
diag/rdbms/wdp/wdp
diag/rdbms/wdp/WDP
diag/rdbms/sunny/sunny2
diag/asm/+asm/+ASM2
diag/tnslsnr/prnhs-db01pn/sunny_prnhs-db01pn
diag/tnslsnr/prnhs-db01pn/listener_prnhs-db01pn

Oracle 11g ADR

17

ADR ADRCI utility

adrci> help
HELP [topic]
Available Topics:
CREATE REPORT
EXIT
HOST
PURGE
SET BASE
SET CONTROL
SET EDITOR
SET TERMOUT
SHOW BASE
SHOW HM_RUN
SHOW INCDIR
SHOW PROBLEM
SHOW TRACEFILE

ECHO
HELP
IPS
RUN
SET BROWSER
SET ECHO
SET HOMES | HOME | HOMEPATH
SHOW ALERT
SHOW CONTROL
SHOW HOMES | HOME | HOMEPATH
SHOW INCIDENT
SHOW REPORT
SPOOL

Oracle 11g ADR

18

ADR ADRCI utility

Data in ADR is not intended to be secure No Login Id is

required

Data in ADR can be secure using OS permission on ADR

directories

Oracle 11g ADR

19

11g Fault Mgmt. Framework Components

Automatic Diagnostic Repository (ADR)

ADRCI Command Line Utility

Alert Log

Trace Files, Dumps, and Core Files

Enterprise Manager Support Workbench

ADRCI Command Line Utility

ADR Home/alert/log.xml

 XML Format alert log file

ADR Home/trace/alert_SID.log  Traditional alert Log file

Oracle 11g ADR

20

ADR Alert Logs

= DIAGNOSTIC_DEST
$ORACLE_BASE
$ORACLE_HOME/log

ADR Base
diag

crs

asm

clients

rdbms

ofm

tnslsnr

lsnrctl

diagtool

racdb
racdb1

stage

hm

cdump

alert
log.xml

= ADR Home

trace

incident

incpkg

metadata

alert_racdb2.log

Oracle 11g ADR

21

ADR Alert Logs

alert

log.xml

$ adrci
ADRCI: Release 11.1.0.6.0 - Beta on Wed Oct 31 13:16:53 2007
Copyright (c) 1982, 2007, Oracle. All rights reserved.
ADR base = "/oracle"

adrci> show alert

Choose the alert log from the following homes to view:
1:
2:
3:
4:
5:
6:
Q:

diag/rdbms/wdp/wdp
diag/rdbms/wdp/WDP
diag/rdbms/sunny/sunny2
diag/asm/+asm/+ASM2
diag/tnslsnr/prnhs-db01pn/sunny_prnhs-db01pn
diag/tnslsnr/prnhs-db01pn/listener_prnhs-db01pn
to quit

Please select option:

Oracle 11g ADR

22

ADR Alert Logs

Please select option: 3
20072007-0909-21 12:51:37.107000 -04:00
Starting ORACLE instance (normal)
20072007-0909-21 12:51:38.209000 -04:00
LICENSE_MAX_SESSION = 0
LICENSE_SESSIONS_WARNING = 0
as
s a public interface
Interface type 1 eth0 172.30.106.0 configured from OCR for use a
Shared memory segment for instance monitoring created
Picked latchlatch-free SCN scheme 2
/oracle/apps/11.1.0/db/dbs/arch
/11.1.0/db/dbs/arch
Using LOG_ARCHIVE_DEST_1 parameter default value as /oracle/apps
Autotune of undo retention is turned on.
LICENSE_MAX_USERS = 0
SYS auditing is disabled
Starting up ORACLE RDBMS Version: 11.1.0.6.0.
Using parameter settings in serverserver-side spfile +SUNNYDG/sunny/spfilesunny.ora
+SUNNYDG/sunny/spfilesunny.ora
System parameters with nonnon-default values:
processes
= 150
spfile
= "+SUNNYDG/sunny/spfilesunny.ora
"+SUNNYDG/sunny/spfilesunny.ora"
spfilesunny.ora"
memory_target
= 6496M
control_files
= "+SUNNYDG/sunny/controlfile/current.260.633875941"
db_block_size
= 8192
compatible
= "11.1.0.0.0"
cluster_database
= TRUE
cluster_database_instances=
cluster_database_instances= 2

"/tmp/alert_22473_3086_sunny2_1.ado" 1442L, 60098C

Oracle 11g ADR

23

ADR Alert Logs

adrci>
adrci>
adrci>
adrci>
adrci>
adrci>

show alert
set editor
show alert
show alert
show alert
show alert

alert

log.xml

vi
tail
tail 50
tail f

Display last 10 entries of Alert Log.

Display last 50 entries of Alert Log.
Work like Unix tail f command

adrci>
adrci> show alert -p "message_text
"message_text like '%ORA'%ORA-1507%'"
ADR Home = /home/oracle/app/diag/rdbms/orcl/orcl:
**********************************************************************
Output the results to file: /tmp/alert_627_308637_orcl_1.ado
2007-08-14 11:01:00.885000 -04:00
ORA-1507 signalled during: ALTER DATABASE CLOSE NORMAL...
Will open Vi editor with all ORA error listed in temp file as shown below

adrci> show alert -p "message_text

"message_text like '%ORA%'"

Oracle 11g ADR

24

ADR Alert Logs OEM

Oracle 11g ADR

25

ADR Alert Logs XML format

$ adrci

alert

log.xml

Sample Alert log.xml

<msg time='2007time='2007-0808-13T10:41:06.66813T10:41:06.668-04:00' org_id='oracle'
org_id='oracle' comp_id='
comp_id='rdbms
='rdbms'
rdbms'
client_id=''
client_id='' type='UNKNOWN' level='16'
module='' pid='18527'>
pid='18527'>
<txt>Stopping background process VKTM:
</txt>
</msg
</msg>
msg>

Oracle 11g ADR

26

11g Fault Mgmt. Framework Components

Automatic Diagnostic Repository (ADR)

ADRCI Command Line Utility

Alert Log

Trace Files, Dumps, and Core Files

Enterprise Manager Support Workbench

Oracle 11g ADR

27

ADR Trace and Dump files

= DIAGNOSTIC_DEST
$ORACLE_BASE
$ORACLE_HOME/log

ADR Base
diag

crs

asm

clients

rdbms

ofm

tnslsnr

lsnrctl

diagtool

racdb
racdb1

stage

hm

cdump

alert

Core Dump
Files

= ADR Home

trace

incident

incpkg

metadata

*.trc

Oracle 11g ADR

28

ADR Trace and Dump files

$ adrci
ADRCI: Release 11.1.0.6.0 - Beta on Wed Oct 31 13:16:53 2007
Copyright (c) 1982, 2007, Oracle. All rights reserved.
ADR base = "/oracle"

adrci> show tracefile

diag/rdbms/wdp/wdp/trace/wdp_diag_21872.trc
diag/rdbms/wdp/wdp/trace/wdp_ping_30092.trc

Oracle 11g ADR

29

11g ADR Trace and Dump files

DIAGNOSTIC Data Types
Foreground Process Traces

USER_DUMP_DEST

ADR_HOME/trace

Background Process Traces

BACKGROUND_DUMP_DEST

ADR_HOME/trace

Core Dumps files

CORE_DUMP_DEST

ADR_HOME/cdump

Alert Log Data

BACKGROUND_DUMP_DEST

ADR_HOME/trace & alert

Incident Dumps

USER|BACKGROUND_DUMP_DEST

ADR_HOME/incident/incdir_n

Here ADR_HOME is not a variable but meant for

ADR Base / diag / rdbms / db_name / ORACLE_SID

Oracle 11g ADR

30

11g Fault Mgmt. Framework Components

Automatic Diagnostic Repository (ADR)

ADRCI Command Line Utility

Alert Log

Trace Files, Dumps, and Core Files

Enterprise Manager Support Workbench

Oracle 11g ADR

31

EM Support Workbench

Oracle 11g ADR

32

Problem and Incident

ADR Home

incident

incpkg

hm

alert

trace

cdump

metadata

11g introduced two new terms to critical errors in the database

Problem

Incident

Oracle 11g ADR

33

Problem

It is critical error in the database tracked in ADR

Common Critical errors are

ORA-00600
ORA-07445
ORA-4020
ORA-1578
ORA-353
ORA-4030
ORA-8103

ADR Home

Internal Errors
incident
Operating System Exception
Deadlock on Library object
Data block Corruption
Log Corruption
Out of Process Memory
Object no Longer exists

Each Problem has



Unique Number called Problem Id

Text String called Problem Key like ORA 600 [xxxxx] along
with some argument [if any]

You can look for Problem ID and Problem Key using ADRCI
utility

Oracle 11g ADR

34

Problem
adrci>
adrci> show problem
ADR Home = /home/oracle/app/diag/rdbms/test11g/test11g:
*************************************************************************
*************************************************************************
0 rows fetched
ADR Home = /home/oracle/app/diag/rdbms/orcl/orcl
/home/oracle/app/diag/rdbms/orcl/orcl:
diag/rdbms/orcl/orcl:
*************************************************************************
*************************************************************************
PROBLEM_ID

PROBLEM_KEY

LAST_INCIDENT

LASTINC_TIME
LASTINC_TIME

------------ ------------------- -------------- --------------------------------1

ORA 1578

14775

2007082007
-08
-15 09:08:54.202548 -04:00

1 rows fetched

Oracle 11g ADR

35

Incident

It is single occurrence of the problem.

It is created as soon as problem is detected in the database

Each Incident is assigned a unique number called incident Id in

ADR

Incidents are time stamped and tracked in the ADR along with
Problem Key

If the same problem or critical error occurred several times, each

one will have unique Incident ID
Problem

incident

Oracle 11g ADR

36

Incident
adrci>
adrci> show incident
ADR Home = /home/oracle/app/diag/rdbms/test11g/test11g:
*************************************************************************
*********
****************************************************************
0 rows fetched
ADR Home = /home/oracle/app/diag/rdbms/orcl/orcl
/home/oracle/app/diag/rdbms/orcl/orcl:
diag/rdbms/orcl/orcl:
*************************************************************************
*********
****************************************************************
INCIDENT_ID
PROBLEM_KEY
CREATE_TIME
-------------- ----------------------------- ---------------------------------14773
ORA 1578
20072007-0808-15 09:08:51.749759 -04:00
14772
ORA 1578
20072007-0808-15 09:08:41.329081 -04:00
14771
ORA 1578
20072007-0808-15 09:08:39.554096 -04:00
3 rows fetched

adrci>
adrci> show incdir
ADR Home = /home/oracle/app/diag/rdbms/orcl/orcl
/home/oracle/app/diag/rdbms/orcl/orcl:
diag/rdbms/orcl/orcl:
*************************************************************************
*********
****************************************************************
diag/rdbms/orcl/orcl/incident/incdir_14771/orcl_ora_13914_i14771.trc
.trc
diag/rdbms/orcl/orcl/incident/incdir_14771/orcl_ora_13914_i14771
diag/rdbms/orcl/orcl/incident/incdir_14770/orcl_ora_13914_i14770.trc
.trc
diag/rdbms/orcl/orcl/incident/incdir_14770/orcl_ora_13914_i14770

Oracle 11g ADR

37

Flood Controlled Incident

As Incident is created for each critical error in the

database, 11g controlled the flood of files in ADR.

Flood controlled incident are

 Recorded in alert.log
 Recorded in ADR
 Does not generate incident dump in ADR

An incident is considered as Flood controlled incident

 If same incident occurred 5 times in an hour.Normal

recoding begins after an hour

 If same incident occurred more than 25 times in a day.

Normal recording restart again next day

If same incident occurred more than 50 times in an hour OR

250 times in a day, it will no longer recorded in ADR
Oracle 11g ADR

38

Monitoring Incident using OEM

Oracle 11g ADR

39

Monitoring Incident using OEM

Oracle 11g ADR

40

Monitoring Incident using OEM

Oracle 11g ADR

41

Flood Controlled Incident

Whenever a problem occurs, the database performs the following

Add an entry in the alert.log

Deliver the incident alert to EM Home Page
Gather Diagnostic data like trace files etc. about the incident
Tag an Incident Id to the collected Diagnostic Data

Stores all the Gathered data in

$ADR_HOME/incident/Incident_Id subdirectory.
Here Incident_Id is the unique ID assigned to the Incident

Oracle 11g ADR

42

Flood Controlled Incident

Oracle 11g ADR

43

ADR Retention Policy

No More manual deletion of Trace and Core dumps files in 11g.

The incident metadata retention policy It controls how long the

metadata is kept in ADR. Default setting is one Year

The incident files and dumps retention policy It controls how

long generated dump files are kept in ADR. Default setting is one
Month.

Oracle 11g ADR

44

Incident
adrci> show control
ADR Home = /home/oracle/app/diag/rdbms/orcl/orcl:
************************************************************************
SHORTP_POLICY LONGP_POLICY LAST_MOD_TIME
ADRID
LAST_AUTOPRG_TIME LAST_MANUPRG_TIME ADRDIR_VERSION ADRSCHM_VERSION
ADRSCHMV_SUMMARY
ADRALERT_VERSION CREATE_TIME
1335663986 720->30days 8760->1yr 2007-08-13 10:39:52.325010 -04:00
1
10:39:52.325010 -04:00

2007-08-13

1 rows fetched

adrci> set control (SHORTP_POLICY = 168)

adrci> set control (LONGP_POLICY = 720)
adrci> show control
ADR Home = /home/oracle/app/diag/rdbms/orcl/orcl:
************************************************************************
ADRID
SHORTP_POLICY LONGP_POLICY LAST_MOD_TIME
LAST_AUTOPRG_TIME LAST_MANUPRG_TIME ADRDIR_VERSION ADRSCHM_VERSION
ADRSCHMV_SUMMARY
ADRALERT_VERSION CREATE_TIME
1335663986 168->7days 720->30days 2007-08-13 11:46:04.886562 -04:00
1
2
0
1
2007-08-13
10:39:52.325010 -04:00
1 rows fetched
Oracle 11g ADR

45

Create Incident package for Oracle support

adrci> show incident
ADR Home = /home/oracle/app/diag/rdbms/orcl/orcl:
*************************************************************************
*********
****************************************************************
INCIDENT_ID
PROBLEM_KEY
CREATE_TIME
-------------------- ----------------------------- ---------------------------------2007-0814773
ORA 1578
2007
08-15 09:08:51.749759 -04:00
2007-0814772
ORA 1578
2007
08-15 09:08:41.329081 -04:00

adrci> ips add incident 14772 package 1

Added incident 14769 to package 1

adrci> host "ls -ltr /home/oracle/app/diag/rdbms/orcl/orcl/incident/incdir_14773"

total 2068
-rworcl_ora_13914_i14773.trm
rm
rw-r----- 1 oracle oinstall 52536 Aug 15 09:08 orcl_ora_13914_i14773.t
-rworcl_ora_13914_i14773.trc
rc
rw-r----- 1 oracle oinstall 2053004 Aug 15 09:08 orcl_ora_13914_i14773.t

adrci> ips add file <ADR_HOME>/trace/alert_orcl.log package 1

Added file <ADR_HOME>/trace/alert_orcl.log to package 1

adrci> ips generate package 1 in /tmp

/tmp/ORA1578_20070815125445_COM_1.zip,
p, mode complete
Generated package 1 in file /tmp/ORA1578_20070815125445_COM_1.zi

Oracle 11g ADR

46

ADR Health Monitor Report

$ADR_HOME/hm contains report generated by health monitor

Health Monitor examine following database components

 DB Structure Integrity Check

 Data Block Integrity Check
 Redo Integrity Check
 Logical Block Check
....

 Dictionary Integrity Check etc

You can get the complete list using the below query
Select name from v$hm_check;

Oracle 11g ADR

47

ADR Health Monitor Report

adrci> show hm_run
ADR Home = /home/oracle/app/diag/rdbms/test11g/test11g:
********************************************************************************
********************************************************************************

HM RUN RECORD 1
********************************************************************************************

********************************************************************************************

RUN RECORD 30
********************************************************************************
********************************************************************************

RUN_ID
RUN_NAME
CHECK_NAME
NAME_ID
MODE
START_TIME
RESUME_TIME
END_TIME
MODIFIED_TIME
TIMEOUT
FLAGS
STATUS
SRC_INCIDENT_ID
NUM_INCIDENTS
ERR_NUMBER
REPORT_FILE

621
HM_RUN_621
DB Structure Integrity Check
2
2
20072007-0808-15 13:40:26.219314 -04:00
<NULL>
20072007-0808-15 13:40:26.246310 -04:00
20072007-0808-15 13:40:26.246310 -04:00
0
0
5
0
0
0
<NULL>

30 rows fetched
Oracle 11g ADR

48

ADR Health Monitor Report

adrci> create report hm_run HM_RUN_621
adrci> show report hm_run HM_RUN_621
<?xml version="1.0" encoding="USencoding="US-ASCII"?>
<HM<HM-REPORT REPORT_ID="HM_RUN_661">
<TITLE>HM Report: HM_RUN_661</TITLE>
<RUN_INFO>
<CHECK_NAME>DB Structure Integrity Check</CHECK_NAME>
<RUN_ID>661</RUN_ID>
<RUN_NAME>HM_RUN_661</RUN_NAME>
<RUN_MODE>REACTIVE</RUN_MODE>
<RUN_STATUS>COMPLETED</RUN_STATUS>
<RUN_ERROR_NUM>0</RUN_ERROR_NUM>
<SOURCE_INCIDENT_ID>0</SOURCE_INCIDENT_ID>
<NUM_INCIDENTS_CREATED>0</NUM_INCIDENTS_CREATED>
<RUN_START_TIME>2007<RUN_START_TIME>2007-0808-15 13:43:22.215671 -04:00</RUN_START_TIME>
<RUN_END_TIME>2007<RUN_END_TIME>2007-0808-15 13:43:22.240171 -04:00</RUN_END_TIME>
</RUN_INFO>
<RUN_PARAMETERS/>
<RUN<RUN-FINDINGS/>

adrci> host "ls -ltr /home/oracle/app/diag/rdbms/orcl/orcl/hm/HMREPORT_HM_RUN_621.hm"

-rwrw-r----- 1 oracle oinstall 711 Aug 15 13:50

/home/oracle/app/diag/rdbms/orcl/orcl/hm/HMREPORT_HM_RUN_661.hm
Oracle 11g ADR

49

QUESTIONS
ANSWERS

THANKS
[email protected]

Oracle 11g ADR

51