RAC Troubleshooting and Diagnosability Sangam2016

Copyright © 2016, Oracle and/or its affiliates. All rights reserved. |
Troubleshooting and Diagnosing Oracle
RAC in the Private Cloud
Sandesh Rao, Senior Director , RAC Development

Safe Harbor Statement
The following is intended to outline our general product direction. It is intended for
information purposes only, and may not be incorporated into any contract. It is not a
commitment to deliver any material, code, or functionality, and should not be relied upon
in making purchasing decisions. The development, release, and timing of any features or
functionality described for Oracle’s products remains at the sole discretion of Oracle.
Confidential – Oracle Restricted 2

Agenda
• Architectural Overview
• Troubleshooting Scenarios
• Proactive and Reactive tools
• Q&A

Grid Infrastructure Overview
• Oracle Clusterware is required for 11gR2+ RAC databases
• Oracle Clusterware can manage non RAC database resources using agents
• Oracle Clusterware can manage HA for any Business Critical Application with
agent infrastructure also called XAG
– Oracle publishes Bundled Agents for some non RAC DB resources
• SAP, Golden Gate, Siebel, Apache..
Oracle Grid Infrastructure

• Grid Infrastructure is the name for the combination of :-
– Oracle Cluster Ready Services (CRS)
– Oracle Automatic Storage Management (ASM)
• The Grid Home contains the software for both products
• CRS can also be Standalone for ASM and/or Oracle Restart
• CRS can run by itself or in combination with other vendor clusterware
• Grid Home and RDBMS home must be installed in different locations
– The installer locks the Grid Home path by setting root permissions.

• CRS requires shared Oracle Cluster Registry (OCR) and Voting files
– Must be in ASM or CFS
– OCR backed up every 4 hours automatically GIHOME/cdata
– Kept 4,8,12 hours, 1 day, 1 week
– Restored with ocrconfig
– Voting file backed up into OCR at each change.
– Voting file restored with crsctl

• For network CRS requires
– One/multiple high speed, low latency, redundant private network for inter node
communications
– Think of interconnect as a memory backplane for the cluster
– Should be a separate physical network or managed converged network
– VLANS are supported
– Used for :-
• Clusterware messaging
• RDBMS messaging and block transfer
• ASM messaging
• HANFS for block traffic

• Only one set of Clusterware daemons can run on each node
• The CRS stack is spawned from Oracle HA Services Daemon (ohasd)
• On Unix ohasd runs out of inittab with respawn
• A node can be evicted when deemed unhealthy
– May require reboot but at least CRS stack restart (rebootless restart)
– IPMI integration or diskmon in case of Exadata
• CRS provides Cluster Time Synchronization services
– Always runs but in observer mode if ntpd configured

Grid Infrastructure Processes
11.2+ Agents change everything.
• Multi-threaded Daemons
• Manage multiple resources and types
• Implements entry points for multiple resource types
– Start,stop check,clean,fail
• oraagent, orarootagent, application agent, script agent, cssdagent
• Single process started from init on Unix (ohasd)
• Diagram below shows all core resources

Level 1
Level 2a
Level 2b
Level 3
Level 4a
Level 4b
Level 0

Init Scripts
• /etc/init.d/ohasd ( location O/S dependent )
– RC script with “start” and “stop” actions
– Initiates Oracle Clusterware autostart
– Control file coordinates with CRSCTL
• /etc/init.d/init.ohasd ( location O/S dependent )
– OHASD Framework Script runs from init/upstart
– Control file coordinates with CRSCTL
– Named pipe syncs with OHASD

• Level 1: OHASD Spawns:
– cssdagent - Agent responsible for spawning CSSD
– orarootagent - Agent responsible for managing all root owned ohasd resources
– oraagent - Agent responsible for managing all oracle owned ohasd resources
– cssdmonitor - Monitors CSSD and node health (along with the cssdagent)
• Level 2a: OHASD rootagent spawns:
– CRSD - Primary daemon responsible for managing cluster resources.
– CTSSD - Cluster Time Synchronization Services Daemon
– Diskmon ( Exadata )
– ACFS (ASM Cluster File System) Drivers

• Level 2b: OHASD oraagent spawns:
– MDNSD– Multicast DNS daemon
– GIPCD – Grid IPC Daemon
– GPNPD – Grid Plug and Play Daemon
– EVMD – Event Monitor Daemon
– ASM – ASM instance started here as may be required by CRSD
• Level 3: CRSD spawns:
– orarootagent - Agent responsible for managing all root owned crsd resources.
– oraagent - Agent responsible for managing all nonroot owned crsd resources.
• One is spawned for every user that has CRS resources to manage.

• Level 4: CRSD oraagent spawns:
– ASM Resouce - ASM Instance(s) resource (proxy resource)
– Diskgroup - Used for managing/monitoring ASM diskgroups.
– DB Resource - Used for monitoring and managing the DB and instances
– SCAN Listener - Listener for single client access name, listening on SCAN VIP
– Listener - Node listener listening on the Node VIP
– Services - Used for monitoring and managing services
– ONS - Oracle Notification Service
– eONS - Enhanced Oracle Notification Service ( pre 11.2.0.2 )
– GSD - For 9i backward compatibility
– GNS (optional) - Grid Naming Service - Performs name resolution
Startup Sequence

Troubleshooting Scenarios
Cluster Startup Problem Triage (11.2+)
Startup
Sequence
ps –ef|grep init.ohasd
ps –ef|grep ohasd.bin Running?
YES
NO crsctl config crs
ohasd.log
Obvious?
NO Engage Oracle Support
Engage Sysadmin Team
Cluster Startup
Diagnostic Flow
TFA Collector
ps –ef|grep cssdagent
ps –ef|grep ocssd.bin
ps –ef|grep orarootagent
ps –ef|grep ctssd.bin
ps –ef|grep crsd.bin
ps –ef|grep cssdmonitor
ps –ef|grep oraagent
ps –ef|grep ora.asm
ps –ef|grep gpnpd.bin
ps –ef|grep mdnsd.bin
ps –ef|grep evmd.bin
Crsctl check crs
Crsctl check cluster
Running?
YES
NO
YES
Engage Sysadmin Team
ohasd.log
agent logs
process logs
Obvious?
YES
NO
Engage
Sysadmin Team
Engage
Oracle Support
Sysadmin Team
TFA Collector
ohasd.log
OLR perms
Compare reference system
Obvious?
YESNO
TFA Collector
Engage
Sysadmin Team
Engage
Oracle Support
Sysadmin Team

• Multicast Domain Name Service Daemon (mDNS(d))
– Used by Grid Plug and Play to locate profiles in the cluster, as well as by GNS to perform
name resolution. The mDNS process is a background process on Linux and UNIX and on
Windows.
– Uses multicast for cache updates on service advertisement arrival/departure.
– Advertises/serves on all found node interfaces.
– Log is GI_HOME/log/<node>/mdnsd/mdnsd.log
Cluster Startup Problem Triage

<?xml version="1.0" encoding="UTF-8"?>
<gpnp:GPnP-Profile Version="1.0" xmlns="http://www.grid-pnp.org/2005/11/gpnp-profile" xmlns:gpnp="http://www.grid-
pnp.org/2005/11/gpnp-profile" xmlns:orcl="http://www.oracle.com/gpnp/2005/11/gpnp-profile"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://www.grid-pnp.org/2005/11/gpnp-profile
gpnp-profile.xsd" ProfileSequence="6" ClusterUId="b1eec1fcdd355f2bbf7910ce9cc4a228" ClusterName="staij-cluster"
PALocation="">
<gpnp:Network-Profile><gpnp:HostNetwork id="gen" HostName="*">
<gpnp:Network id="net1" IP=”192.168.1.0" Adapter="eth0" Use="public"/>
<gpnp:Network id="net2" IP=”192.168.2.0" Adapter="eth1“ Use="cluster_interconnect"/>
</gpnp:HostNetworkcss"></gpnp:Network-Profile>
<orcl:CSS-Profile id=" DiscoveryString="+asm" LeaseDuration="400"/>
<orcl:ASM-Profile id="asm" DiscoveryString="" SPFile="+SYSTEM/staij-cluster/asmparameterfile/registry.253.693925293"/>
<ds:Signature xmlns:ds="http://www.w3.org/2000/09/xmldsig#"><ds:SignedInfo><ds:CanonicalizationMethod
Algorithm="http://www.w3.org/2001/10/xml-exc-c14n#"/><ds:SignatureMethod Algorithm="http://www.w3.org/2001/10/xml-
exc-c14n#"> <InclusiveNamespaces xmlns="http://www.w3.org/2001/10/xml-exc-c14n#" PrefixList="gpnp orcl
xsi"/></ds:Transform></ds:Transforms><ds:DigestMethod
Algorithm="http://www.w3.org/2000/09/xmldsig#sha1"/><ds:DigestValue>x1H9LWjyNyMn6BsOykHhMvxnP8U=</ds:DigestValue
></ds:Reference></ds:SignedInfo><ds:SignatureValue>N+20jG4=</ds:SignatureValue></ds:Signature>
</gpnp:GPnP-Profile>

• cssd agent and monitor
– Same functionality in both agent and monitor
– Functionality of several pre-11.2 daemons consolidated in both
• OPROCD – system hang
• OMON – oracle clusterware monitor
• VMON – vendor clusterware monitor
– Run realtime with locked down memory, like CSSD
– Provides enhanced stability and diagnosability
– Logs are
• GI_HOME/log/<node>/agent/oracssdagent_root/oracssdagent_root.log
• GI_HOME/log/<node>/agent/oracssdmonitor_root/oracssdmonitor_root.log
• 12.1 – ORACLE_BASE/diag/node/agent/..

Node Evictions
Eviction
Scenario
Cluster alert
ocssd.log
NHB?
1050693.1
1534949.1
1546004.1
Engage
networking team
YES
NO
DHB?
1549428.1
1466639.1 YES
NO
Obvious?
NO
YES
Obvious?
NO
YES
Fenced?
Resource starvation
YES
NO
NO
YES
Node Eviction
Diagnostic Flow
Resource
Starvation?
NO
Engage
Oracle
Support
Engage
sysadmin
team
Engage storage
team
1531223.1
1328466.1
System log
YES
Engage
appropriate
team
Resolved?
NO
YES
Free memory?
CPU load?
Node Response?
TFA Collector
TFA Collector

Missing Network Heartbeat (1)
• ocssd.log from node 1
• ===> sending network heartbeats other nodes. Normally, this message is output once every 5 messages (seconds)
• 2016-08-13 17:00:20.023: [ CSSD][4096109472]clssnmSendingThread: sending status msg to all nodes
• 2016-08-13 17:00:20.023: [ CSSD][4096109472]clssnmSendingThread: sent 5 status msgs to all nodes
• ===> The network heartbeat is not received from node 2 (drrac2) for 15 consecutive seconds.
• ===> This means that 15 network heartbeats are missing and is the first warning (50% threshold).
• 2016-08-13 17:00:22.818: [ CSSD][4106599328]clssnmPollingThread: node drrac2 (2) at 50% heartbeat fatal, removal in 14.520
seconds
• 2016-08-13 17:00:22.818: [ CSSD][4106599328]clssnmPollingThread: node drrac2 (2) is impending reconfig, flag 132108,
misstime 15480
• ===> continuing to send the network heartbeats and log messages once every 5 messages
• ===> 75% threshold of missing network heartbeat is reached. This is second warning.
seconds

• ===> continuing to send the network heartbeats and log messages once every 5 messages
• ===> continuing to send the network heartbeats, but the message is logged after 4 messages
• ===> Last warning shows that 90% threshold of the missing network heartbeat is reached.
• ===> The eviction will occur in 2.49 seconds.
• 2016-08-13 17:00:34.841: [ CSSD][4106599328]clssnmPollingThread: node drrac2 (2) at 90% heartbeat fatal, removal in
2.490 seconds, seedhbimpd 1
• ===> Eviction of node 2 (drrac2) started
• 2016-08-13 17:00:37.337: [ CSSD][4106599328]clssnmPollingThread: Removal started for node drrac2 (2), flags 0x2040c,
state 3, wt4c 0
• ===> This shows that the node 2 is actively updating the voting disks
• 2016-08-13 17:00:37.340: [ CSSD][4085619616]clssnmCheckSplit: Node 2, drrac2, is alive, DHB (1281744040, 1396854)
more than disk timeout of 27000 after the last NHB (1281744011, 1367154)

• ===> Evicting node 2 (drrac2)
• 2016-08-13 17:00:37.340: [ CSSD][4085619616](:CSSNM00007:)clssnmrEvict: Evicting node 2, drrac2, from the cluster in
incarnation 169934272, node birth incarnation 169934271, death incarnation 169934272, stateflags 0x24000
• ===> Reconfigured the cluster without node 2
• 2016-08-13 17:01:07.705: [ CSSD][4043389856]clssgmCMReconfig: reconfiguration successful, incarnation 169934272 with 1
nodes, local node number 1, master node number 1

• ocssd.log from node 2:
• ===> Logging the message to indicate 5 network heartbeats are sent to other nodes
• ===> First warning of reaching 50% threshold of missing network heartbeats
seconds
• 2016-08-13 17:00:26.213: [ CSSD][4073040800]clssnmPollingThread: node drrac1 (1) is impending reconfig, flag 394254,
misstime 15460
• ===> Second warning of reaching 75% threshold of missing network heartbeats
seconds

• ===> Logging the message to indicate 4 network heartbeats are sent
• ===> Third warning of reaching 90% threshold of missing network heartbeats
• 2016-08-13 17:00:38.236: [ CSSD][4073040800]clssnmPollingThread: node drrac1 (1) at 90% heartbeat fatal, removal in
2.460 seconds, seedhbimpd 1
• ===> Eviction started for node 1 (drrac1)
• 2016-08-13 17:00:40.702: [ CSSD][4073040800]clssnmPollingThread: Removal started for node drrac1 (1), flags 0x6040e,
state 3, wt4c 0
• ===> Node 1 is actively updating the voting disk, so this is a split brain condition
• 2016-08-13 17:00:40.706: [ CSSD][4052061088]clssnmCheckSplit: Node 1, drrac1, is alive, DHB (1281744036, 1243744)
more than disk timeout of 27000 after the last NHB (1281744007, 1214144)
• 2016-08-13 17:00:40.706: [ CSSD][4052061088]clssnmCheckDskInfo: My cohort: 2
• 2016-08-13 17:00:40.707: [ CSSD][4052061088]clssnmCheckDskInfo: Surviving cohort: 1

• ===> Node 2 is aborting itself to resolve the split brain and ensure the cluster integrity
• 2016-08-13 17:00:40.707: [ CSSD][4052061088](:CSSNM00008:)clssnmCheckDskInfo: Aborting local node to avoid splitbrain.
Cohort of 1 nodes with leader 2, drrac2, is smaller than cohort of 1 nodes led by node 1, drrac1, based on map type 2
• 2016-08-13 17:00:40.707: [ CSSD][4052061088]###################################
• 2016-08-13 17:00:40.707: [ CSSD][4052061088]clssscExit: CSSD aborting from thread clssnmRcfgMgrThread
• 2016-08-13 17:00:40.707: [ CSSD][4052061088]###################################

• Observations
1. Both nodes reported missing heartbeats at the same time
2. Both nodes sent heartbeats to other nodes all the time
3. Node 2 aborted itself to resolve split brain
• Conclusion
1. This is likely a network problem, engage network team
2. Check OSWatcher output (netstat and traceroute)
1. Configure private.net file, not configured by default
3. Check CHMOS
4. Check system log

Voting Disk Access Problem (1)
ocssd.log:
===> The first error indicating that it could not read voting disk -- first message to indicate a
problem accessing the voting disk
2016-08-13 18:31:19.787: [ SKGFD][4131736480]ERROR: -9(Error 27072, OS Error (Linux
Error: 5: Input/output error
Additional information: 4
Additional information: -1)
)
2016-08-13 18:31:19.787: [ CSSD][4131736480](:CSSNM00060:)clssnmvReadBlocks: read
failed at offset 529 of /dev/sdb8
2016-08-13 18:31:19.802: [ CSSD][4131736480]clssnmvDiskAvailabilityChange: voting file
/dev/sdb8 now offline

====> The error message that shows a problem accessing the voting disk repeats once every 4 seconds
2016-08-13 18:31:23.782: [ CSSD][150477728]clssnmvDiskOpen: Opening /dev/sdb8
2016-08-13 18:31:23.782: [ SKGFD][150477728]Handle 0xf43fc6c8 from lib :UFS:: for disk :/dev/sdb8:
2016-08-13 18:31:23.782: [ CLSF][150477728]Opened hdl:0xf4365708 for dev:/dev/sdb8:
2016-08-13 18:31:23.787: [ SKGFD][150477728]ERROR: -9(Error 27072, OS Error (Linux Error: 5:
Input/output error
)
2016-08-13 18:31:23.787: [ CSSD][150477728](:CSSNM00060:)clssnmvReadBlocks: read failed at offset 17
of /dev/sdb8

====> The last error that shows a problem accessing the voting disk.
====> Note that the last message is 200 seconds after the first message
====> because the long disktimeout is 200 seconds
2016-08-13 18:34:37.423: [ CSSD][150477728]clssnmvDiskOpen: Opening /dev/sdb8
2016-08-13 18:34:37.423: [ CLSF][150477728]Opened hdl:0xf4336530 for dev:/dev/sdb8:
2016-08-13 18:34:37.429: [ SKGFD][150477728]ERROR: -9(Error 27072, OS Error (Linux Error: 5:
Input/output error
)
2016-08-13 18:34:37.429: [ CSSD][150477728](:CSSNM00060:)clssnmvReadBlocks: read failed at offset 17
of /dev/sdb8

====> This message shows that ocssd.bin tried accessing the voting disk for 200 seconds
2016-08-13 18:34:38.205: [ CSSD][4110736288](:CSSNM00058:)clssnmvDiskCheck: No I/O completions for
200880 ms for voting file /dev/sdb8)
====> ocssd.bin aborts itself with an error message that the majority of voting disks are not available. In
this case, there was only one voting disk, but if three voting disks were available, as long as two
voting disks are accessible, ocssd.bin will not abort.
2016-08-13 18:34:38.206: [ CSSD][4110736288](:CSSNM00018:)clssnmvDiskCheck: Aborting, 0 of 1
configured voting disks available, need 1
2016-08-13 18:34:38.206: [ CSSD][4110736288]###################################
2016-08-13 18:34:38.206: [ CSSD][4110736288]clssscExit: CSSD aborting from thread
clssnmvDiskPingMonitorThread
2016-08-13 18:34:38.206: [ CSSD][4110736288]###################################
• Conclusion
The voting disk was not available, engage storage team

• Time synchronisation issue
• Cluster Time Synchronisation Services daemon
– Provides time management in a cluster for Oracle.
• Observer mode when Vendor time synchronisation s/w is found
–Logs time difference to the CRS alert log
• Active mode when no Vendor time sync s/w is found
Node Eviction Triage

• Cluster Ready Services Daemon
– The CRSD daemon is primarily responsible for maintaining the availability of application
resources, such as database instances. CRSD is responsible for starting and stopping these
resources, relocating them when required to another node in the event of failure, and
maintaining the resource profiles in the OCR (Oracle Cluster Registry). In addition, CRSD is
responsible for overseeing the caching of the OCR for faster access, and also backing up the
OCR.
– Log file is GI_HOME/log/<node>/crsd/crsd.log
• Rotation policy 10-50M
• Retention policy 10 logs
• Dynamic in 12.1 and can be changed

• CRSD oraagent
– CRSD’s oraagent manages
• all database, instance, service and diskgroup resources
• node listeners
• SCAN listeners, and ONS
– If the Grid Infrastructure owner is different from the RDBMS home owner then you would
have 2 oraagents each running as one of the installation owners. The database, and service
resources would be managed by the RDBMS home owner and other resources by the Grid
Infrastructure home owner.
– Log file is
• GI_HOME/log/<node>/agent/crsd/oraagent_<user>/oraagent_<user>.log

• CRSD orarootagent
– CRSD’s rootagent manages
• GNS and it’s VIP
• Node VIP
• SCAN VIP
• network resources.
– Log file is
• GI_HOME/log/<node>/agent/crsd/orarootagent_root/oraagent_root.log

• Agent return codes
– Check entry must return one of the following return codes:
• ONLINE
• UNPLANNED_OFFLINE
– Target=online, may be recovered failed over
• PLANNED_OFFLINE
• UNKNOWN
– Cannot determine, if previously online, partial then monitor
• PARTIAL
– Some of a resources services are available. Instance up but not open.
• FAILED
– Requires clean action

§ Important logs and traces
§ 11.2 – Databases only use ADR
• Grid Infrastructure files in $GI_HOME/log/<node_name>/<component_name>
– $GI_HOME/log/myHost/cssd
– $GI_HOME/log/myHost/alertmyHost.log
§ 12.1 – Grid Infrastructure and Database use ADR
§ Different locations for Grid Infrastructure and Databases
§ Grid Infrastructure
• Alert.log, cssd.log, csrd.log, etc
§ Databases
§ Alert.log, background process traces, foreground process traces
Automatic Diagnostic Repository (ADR)

• What if issues were detected before they
had an impact?
• What if you were notified with a specific
diagnosis and corrective actions?
• What if resource bottlenecks threatening
SLAs were identified early?
• What if bottlenecks could be
automatically relieved just in time?
• What if database hangs and node reboots
could be eliminated?
37
Oracle’s Database and Clusterware Tools
Cluster
Verification
Utility
ORAchkCluster
Health
Monitor
Trace File
Analyzer
Quality of
Service
Management
Hang
Manager
EXAchk
Cluster
Health
Advisor
Memory
Guard
Confidential – Oracle Restricted

–Automated risk identification and proactive notification
before business is impacted
–Health Checks based on most impactful reoccurring
problems across Oracle customer base
–Runs in your environment – no need to send anything to
Oracle
–Scheduled email Health Check reports
–Findings can be integrated into other tools of choice
Oracle EXAchk/Orachk (Proactive)
Engineered
Systems
Non Engineered
Systems
Oracle
EXAchk
Oracle
ORAchk
Common Framework
Lightweight & non intrusive Oracle Stack Health Checks
38

1. Included in base image
and latest OEDA
2. Download latest version
from My Oracle Support
(install < 1 min)
1070954.1
3. Auto update when later
version available
Roll Out & Maintain EXAchk
39

1. Download the
orachk.zip to your local
machine from MOS
Note 1268927.2
2. Transfer to a directory
on the target System
3. Unzip orachk.zip
o As owner of oracle
database or grid home
Installation
40

• Profiles provide logical grouping of
checks which are about similar topics
• Run only checks in a specific profile
• Run everything except checks in a specific
profile
Profiles
./exachk –profile <profile>
./exachk –excludeprofile <profile>
Profile Description
asm ASM Checks
avdf Audit Vault Configuration checks
clusterware Oracle clusterware checks
control_VM Checks only for Control VM(ec1-vm, ovmm, db, pc1, pc2).
No cross node checks
corroborate Exadata checks needs further review by user to determine
pass or fail
dba DBA Checks
ebs Oracle E-Business Suite checks
eci_healthchecks Enterprise Cloud Infrastructure Healthchecks
ecs_healthchecks Enterprise Cloud System Healthchecks
goldengate Oracle GoldenGate checks
hardware Hardware specific checks for Oracle Engineered systems
maa Maximum Availability Architecture Checks
ovn Oracle Virtual Networking
platinum Platinum certification checks
preinstall Pre-installation checks
prepatch Checks to execute before patching
security Security checks
solaris_cluster Solaris Cluster Checks
storage Oracle Storage Server Checks
switch Infiniband switch checks
sysadmin Sysadmin checks
user_defined_checks Run user defined checks from user_defined_checks.xml
41

• Profiles provide logical grouping of
checks which are about similar topics
• Run only checks in a specific profile
• Run everything except checks in a specific
profile
Profiles
./orachk –profile <profile>
./orachk –excludeprofile <profile>
Profile Description
asm ASM Checks
bi_middleware Oracle Business Intelligence checks
clusterware Oracle clusterware checks
dba DBA Checks
ebs Oracle E-Business Suite checks
emagent Cloud control agent checks
emoms Cloud Control management server
em Cloud control checks
goldengate Oracle GoldenGate checks
hardware Hardware specific checks for Oracle Engineered systems
oam Oracle Access Manager checks
oim Oracle Identify Manager checks
oud Oracle Unified Directory server checks
ovn Oracle Virtual Networking
peoplesoft Peoplesoft best practices
preinstall Pre-installation checks
prepatch Checks to execute before patching
security Security checks
siebel Siebel Checks
solaris_cluster Solaris Cluster Checks
storage Oracle Storage Server Checks
switch Infiniband switch checks
sysadmin Sysadmin checks
user_defined_checks Run user defined checks from user_defined_checks.xml
42

Enterprise Manager Integration
•Check results integrated into EM
compliance framework via plugin
•View results in native EM
compliance dashboards
•Related checks grouped into
compliance standards
•View targets checked, violations &
average score
•Drill down into compliance standard
to see individual check results
•View break down by target
43

• Integration is via the Enterprise
Manager ORAchk Healthchecks
plugin with the following Support:
• The following prerequisites must be met
before you can deploy the plug-in:
o Verify that your Engineered Systems hardware
and software are at the supported level as
described in Supported Hardware and Software
Versions
o All Engineered System plug-ins should be
deployed
o InfiniBand switches and storage cells should be
an Enterprise Manager-managed target for the
respective engineered system
o Expect package should be installed on the
hosts
Enterprise Manager Plugin Prerequisites
Hardware Types
Supported
By Plugin
Exadata (physical configuration only) Yes
Exadata (virtual configuration) No
Recovery appliance Yes
Exalogic (physical configuration) Yes
Exalogic (virtualized configuration) Yes
Oracle SuperCluster No
Oracle Private Cloud Machine No
44

JSON Output to Integrate with Kibana, Elastic Search etc
45

Oracle Health Check Collection Manager Dashboard
46

Oracle Stack Coverage
• Engineered Systems
• Oracle Exadata Database Machine
• Oracle SuperCluster
• Oracle Private Cloud Appliance
• Oracle Database Appliance
• Oracle Big Data Appliance
• Oracle Exalogic Elastic Cloud
• Oracle Exalytics In-Memory Machine
• Oracle Zero Data Loss Recovery Appliance
• Oracle ZFS Storage Appliance
• ASR
• Systems
• Oracle Solaris
• Cross stack checks
• Solaris Cluster
• OVN
• Oracle Database
• Standalone Database
• Grid Infrastructure & RAC
• Maximum Availability Architecture (MAA)
Scorecard
• Upgrade Readiness Validation
• Golden Gate
• Enterprise Manager Cloud Control
• Repository
• Agent
• OMS
• Middleware
• Application Continuity
• Oracle Identify and Access Management
Suite (Oracle IAM)
• E-Business Suite
• Oracle Payables
• Oracle Workflow
• Oracle Purchasing
• Oracle Order Management
• Oracle Process Manufacturing
• Oracle Receivables
• Oracle Fixed Assets
• Oracle HCM
• Oracle CRM
• Oracle Project Billing
• Siebel
• Database best practices
• PeopleSoft
• Database best practices
• SAP
• EXAdata best practices
47

Copyright © 2016, Oracle and/or its affiliates. All rights reserved. | 48
Generates Diagnostic Metrics View of Cluster and Databases
Cluster Health Monitor (CHM)
GIMR
ologgerd
(master)
osysmond
osysmond
osysmond
osysmond
12c Grid Infrastructure
Management Repository
• Always on - Enabled by default
• Provides Detailed OS Resource Metrics
• Assists Node eviction analysis
• Locally logs all process data
• User can define pinned processes
• Listens to CSS and GIPC events
• Categorizes processes by type
• Supports plug-in collectors (ex.
traceroute, netstat, ping, etc.)
• New CSV output for ease of analysis
OS Data OS Data
OS Data
OS Data
Confidential – Oracle Internal/Restricted/Highly RestrictedConfidential – Oracle Restricted

Oclumon CLI or Full Integration with EM Cloud Control
Cluster Health Monitor (CHM)
Confidential – Oracle Internal/Restricted/Highly RestrictedConfidential – Oracle Restricted

Why TFA? (Proactive and reactive)
Provides one interface for
all diagnostic needs
Collects data across the
cluster and consolidates it
in one place
Collects all relevant
diagnostic data at the time
of the problem, with only
what is needed to diagnose
the problem
Reduces time required to
obtain diagnostic data,
which saves your business
money
50Oracle Confidential – Internal

• All major Operating Systems are
supported
– Linux (OEL, RedHat, SUSE, Itanium &
zLinux)
– Oracle Solaris (SPARC & x86-64)
– AIX
– HPUX (Itanium & PA-RISC)
– Windows
• All Oracle Database & Grid versions
10.2+ are supported
• You probably already have TFA
installed as it is included with:
– Oracle Grid Infrastructure:
• 11.2.0.4+
• 12.1.0.2+
• 12.2.0.1+
– Oracle Database:
• 12.2.0.1+
• Also available from Doc 1513912.2
51
Supported Platforms and Versions

Monitoring By TFA & Automated Collections
52
Automatically
detect event
Collect & package
relevant
diagnostics
Notify
relevant DBA and
or Sys Admin by
email
Upload collection
to Oracle Support
for further help
Significant
problem occurs
1
2
3
4
TFA
DBA(s) / Sys Admin(s)
& Database(s)

• Trim & collect all important log files updated in
the past 12 hours:
• Collect a problem specific Service Request Data
Collection (SRDC):
53
Collect
tfactl diagcollect
• Collections stored in the repository directory
• Change diagcollect timeframe with –since <n>h|d
• For list of types of srdc collections use tfactl diagcollect -srdc help
tfactl diagcollect -srdc ora600

TFA dbglevel profiles
• Example
– tfactl dbglevel -set node_eviction
– would be used for enhancing diagnostics when node evictions are the being
investigated and would perform the following operation internally
• crsctl set log css "CSSD=4"
• crsctl set log css "CSSDNMC=4"
• crsctl set log css "CLSF=4"
• crsctl set log css "CSSDGMCC=4"
• crsctl set log css "CSSDGMPC=4"
• To revert to the original or default logging levels the following command
– $ tfactl dbglevel -unset node_eviction
• would perform the following operations internally
• crsctl set log css "CSSD=2"
• crsctl set log css "CSSDNMC=2"
• crsctl set log css "CLSF=0"
• crsctl set log css "CSSDGMCC=2"
• crsctl set log css "CSSDGMPC=2" Oracle Confidential – Internal/Restricted/Highly Restricted 54

Log Management
• TFA will be log
management interface for
software stack
– Rotate logs
– Archive logs
– Purge old logs
• Intelligent log management
based on understanding of
what is in logs and what is
still important
Oracle Confidential – Internal/Restricted/Highly Restricted 55
TFA Log
Management All Logs Across
Software Stack
Rotate
Archive
Purge
Action through
prediction or user
input

Set Email Notification Addresses
56
tfactl set notificationAddress=john.doe@oracle.com
Automatically
detect event
Collect &
package relevant
diagnostics
Notify
relevant DBA
and or Sys
Admin by email
Upload
collection to
Oracle Support
for further help
Significant
problem occurs
1
2
3
4
TFA
DBA(s) / Sys Admin(s)
& Database(s)
tfactl set notificationAddress=oracle:another.person@oracle.com
• TFA can send email notification
when significant problems are
detected
• To set notification email for any
problem detected:
• To set notification email for specific
ORACLE_HOMEs include the OS
owner:
Oracle Confidential – Internal

• Analyze all important recent log entries: • Search recent log entries:
57
Analyze
tfactl analyze –since 1d tfactl analyze -search “ora-00600" -since 8h

• TFA includes all key database support tools
• tfactl provides a single interface to them all
Analyze
58
Most of these Support tools are only available in the
My Oracle Support download, they are not included
in the base Grid or Database install
Tool Description Details
ORAchk Oracle Stack Health Checks on non-engineered
systems
1268927.2
EXAchk Oracle Stack Health Checks on Engineered
Systems
1070954.1
oswatcher Collect and archive OS metrics, useful for
instance / node evictions & performance
Issues
301137.1
procwatcher Automate & capture database performance
diagnostics & session level hangs
459694.1
oratop Near real-time database monitoring 1500864.1
sqlt Capture SQL trace date useful for tuning 215187.1
alertsummary Provides summary of events for one or more
database or ASM alert files from all nodes
ls Lists all files TFA knows about for a given file
name pattern across all nodes
Tool Description
pstack Generate process stack for specified processes across all nodes
grep Search alert or trace files with a given database and file name
pattern, for a search string.
summary High level summary of the configuration
vi Open alert or trace files for viewing a given database and file
name pattern in the vi editor
tail Run a tail on an alert or trace files for a given database and file
name pattern
param Show all database and OS parameters that match a specified
pattern
dbglevel Set and unset multiple CRS trace levels with one command
history Show the shell history for the tfactl shell
changes Report any noted changes in the system setup over a given
time period. This includes database a parameters, OS
parameters, patches applied etc

• Use srdc <incident type>:
• To specify sid use –sid <oracle sid>
• To specify database use –db <dbname>
• To specify incident date & time use
–inc_date <YYYY-MM-DD> -inc_time <HH:MM:SS>
• To upload directly to the SR use –sr<SR#>
• For dbperf use these parameters to
specify the good & bad performance
periods to compare:
59
Incident Based Collections with SRDC
tfactl srdc ora4030
Incident Type Description
ora4030 For ORA-04030 errors
dbperf For basic db performance problems
tfactl srdc ora4030 -sid orcl –db RDBMS121
-inc_date 2016-06-15 -inc_time 02:48:23
-sr 3-123456789
Parameter Description
perf_base_sd Start date for a good performance period
perf_base_st Start time for a good performance period
perf_base_ed End date for a good performance period
perf_base_et End time for a good performance period
perf_comp_sd Start date for a bad performance period
perf_comp_st Start time for a bad performance period
perf_comp_ed End date for a bad performance period
perf_comp_et End time for a bad performance period
tfactl srdc dbperf –db RDBMS121
–perf_base_sd 2016-06-15 –perf_base_st 01:30:00
–perf_base_ed 2016-06-15 –perf_base_et 02:00:00
–perf_comp_sd 2016-06-16 –perf_comp_st 09:30:00
–perf_comp_ed 2016-06-16 –perf_comp_et 10:00:00

Cluster Health Advisor (CHA)*
Discovers Potential Cluster & DB Problems - Notifies with Corrective Actions
60
OS Data
GIMR
ochad
• Detects node and database
performance problems
• Provides early-warning alerts and
corrective action
• Supports on-site calibration to improve
sensitivity
• Integrated into EMCC Incident Manager
and notifications
• Standalone Interactive GUI Tool
DB Data
CHM
Node
Health
Prognostics
Engine
Database
Health
Prognostics
Engine
* Requires and Included with RAC or R1N License

Oracle 12c Hang Manager
• Reliably detects database hangs and
deadlocks
• Autonomously resolves them
• Supports QoS Performance Classes, Ranks
and Policies to maintain SLAs
• Logs all detections and resolutions
• New SQL interface to configure sensitivity
(Normal/High) and trace file sizes
Autonomously Preserves Database Availability and Performance Session
DIA0
EVALUATE
DETECT
ANALYZE
Hung?
VERIFY
Victim
QoS
Policy

Full Resolution Dump Trace File and DB Alert Log Audit Reports
Oracle 12c Hang Manager
Dump file …/diag/rdbms/hm6/hm62/incident/incdir_5753/hm62_dia0_12656_i5753.trc
Oracle Database 12c Enterprise Edition Release 12.2.0.0.0 - 64bit Beta
With the Partitioning, Real Application Clusters, OLAP, Advanced Analytics
and Real Application Testing options
Build label: RDBMS_MAIN_LINUX.X64_151013
ORACLE_HOME: …/3775268204/oracle
System name: Linux
Node name: slc05kyr
Release: 2.6.39-400.211.1.el6uek.x86_64
Version: #1 SMP Fri Nov 15 13:39:16 PST 2013
Machine: x86_64
VM name: Xen Version: 3.4 (PVM)
Instance name: hm62
Redo thread mounted by this instance: 2
Oracle process number: 19
Unix process pid: 12656, image: oracle@slc05kyr (DIA0)
*** 2015-10-13T16:47:59.541509+17:00
*** SESSION ID:(96.41299) 2015-10-13T16:47:59.541519+17:00
*** CLIENT ID:() 2015-10-13T16:47:59.541529+17:00
*** SERVICE NAME:(SYS$BACKGROUND) 2015-10-13T16:47:59.541538+17:00
*** MODULE NAME:() 2015-10-13T16:47:59.541547+17:00
*** ACTION NAME:() 2015-10-13T16:47:59.541556+17:00
*** CLIENT DRIVER:() 2015-10-13T16:47:59.541565+17:00
2015-10-13T16:47:59.435039+17:00
Errors in file /oracle/log/diag/rdbms/hm6/hm6/trace/hm6_dia0_12433.trc (incident=7353):
ORA-32701: Possible hangs up to hang ID=1 detected
Incident details in: …/diag/rdbms/hm6/hm6/incident/incdir_7353/hm6_dia0_12433_i7353.trc
2015-10-13T16:47:59.506775+17:00
DIA0 requesting termination of session sid:40 with serial # 43179 (ospid:13031) on instance 2
due to a GLOBAL, HIGH confidence hang with ID=1.
Hang Resolution Reason: Automatic hang resolution was performed to free a
significant number of affected sessions.
DIA0: Examine the alert log on instance 2 for session termination status of hang with ID=1.
In the alert log on the instance local to the session (instance 2 in this case),
we see the following:
2015-10-13T16:47:59.538673+17:00
Errors in file …/diag/rdbms/hm6/hm62/trace/hm62_dia0_12656.trc (incident=5753):
ORA-32701: Possible hangs up to hang ID=1 detected
Incident details in: …/diag/rdbms/hm6/hm62/incident/incdir_5753/hm62_dia0_12656_i5753.trc
2015-10-13T16:48:04.222661+17:00
DIA0 terminating blocker (ospid: 13031 sid: 40 ser#: 43179) of hang with ID = 1
requested by master DIA0 process on instance 1
Hang Resolution Reason: Automatic hang resolution was performed to free a
significant number of affected sessions.
by terminating session sid:40 with serial # 43179 (ospid:13031)

Oracle 12c Domain Services Cluster (DSC)
63
• Hosts Framework as Services
• Reduces local resource footprint
• Centralizes management
• Speeds deployment and patching
• Optional Shared Storage
• Supports multiple versions and
platforms going forward
Deploys with Minimum Footprint and Maximum Manageability
Application
Member
Cluster
Database
Member
Cluster
Database
Member
Cluster
Oracle Domain Services Cluster
Database
Member
Cluster
Application
Member
Cluster
Database
Member
Cluster
ORACLE CLUSTER DOMAIN
Management Repository Service
Trace File Analyzer Receiver
ORAchk Collection Service
Grid Names Service
Storage Services
Rapid Home Provisioning Service

Oracle Domain Services Cluster
Oracle Cluster Domain
IO Service
ACFS
Services
ASM
Service
Database
Member Cluster
Uses ASM
Service
Database
Member Cluster
Uses IO & ASM
Service of DSC
Mgmt
Repository
(GIMR)
Service
Application
Member Cluster
GI only
Database
Member Cluster
Uses local ASM
Shared ASM
Additional
Optional
Services
Rapid Home
Provisioning
(RHP)
Service
Private
Network
SAN
NAS

RAC Troubleshooting and Diagnosability Sangam2016

Recommended

More Related Content

What's hot (20)

Viewers also liked (20)

Similar to RAC Troubleshooting and Diagnosability Sangam2016 (20)

More from Sandesh Rao (20)

RAC Troubleshooting and Diagnosability Sangam2016