Clusterware Stack

Management
and Troubleshooting
by Syed Jaffar Hussain, Kai Yu
In Chapter 1, we mentioned that the Oracle RAC cluster database environment requires
cluster manager software
(“Clusterware”) that is tightly integrated with the operating system (OS) to provide the
cluster management functions
that enable the Oracle database in the cluster environment.
Oracle Clusterware was originally introduced in Oracle 9i on Linux with the original name
Oracle Clusterware
Management Service. Cluster Ready Service (CRS) as a generic cluster manager was
introduced in Oracle 10.1 for all
platforms and was renamed to today’s name, Oracle Clusterware, in Oracle 10.2. Since
Oracle 10g, Oracle Clusterware
has been the required component for Oracle RAC. On Linux and Windows systems, Oracle
Clusterware is the only
clusterware we need to run Oracle RAC, while on Unix, Oracle Clusterware can be
combined with third-party
clusterware such as Sun Cluster and Veritas Cluster Manager.
Oracle Clusterware combines a group of servers into a cluster environment by enabling
communication between
the servers so that they work together as a single logical server. Oracle Clusterware serves
as the foundation of the
Oracle RAC database by managing its resources. These resources include Oracle ASM
instances, database instances,
Oracle databases, virtual IPs (VIPs), the Single Client Access Name (SCAN), SCAN listeners,
Oracle Notification
Service (ONS), and the Oracle Net listener. Oracle Clusterware is responsible for startup
and failover for the resources.
Because Oracle Clusterware plays such a key role in the high availability and scalability of
the RAC database,
the system administrator and the database administrator should pay careful attention to
its configuration and
management.
This chapter describes the architecture and complex technical stack of Oracle Clusterware
and explains how
those components work. The chapter also describes configuration best practices and
explains how to manage and
troubleshoot the clusterware stack. The chapter assumes the latest version of Oracle
Clusterware 12cR1.
The following topics will be covered in this chapter:
Oracle Clusterware 12cR1 and its components
•
Clusterware startup sequence
•
Clusterware management
•
Troubleshooting cluster stack startup failure
•
CRS logs and directory structure
•
RACcheck, diagcollection.sh, and oratop
•
Debugging and tracing CRS components
•
RAC database hang analysis
• CHAPTER 2 ■ CLUSTERWARE STACK MANAGEMENT AND TROUBLESHOOTING

Clusterware 12cR1 and Its Components

Before Oracle 11gR2, Oracle Clusterware was a distinct product installed in a home
directory separate from Oracle
ASM and Oracle RAC database. Like Oracle 11gR2, in a standard 12cR1 cluster, Oracle
Clusterware and Oracle ASM
are combined into a product called Grid Infrastructure and installed together as parts of
the Grid Infrastructure to a
single home directory. In Unix or Linux environments, some part of the Grid
Infrastructure installation is owned by
the root user and the rest is owned by special user grid other than the owner of the
Oracle database software oracle.
The grid user also owns the Oracle ASM instance.
Only one version of Oracle Clusterware can be active at a time in the cluster, no matter
how many different
versions of Oracle Clusterware are installed on the cluster. The clusterware version has to
be the same as the Oracle
Database version or higher. Oracle 12cR1 Clusterware supports all the RAC Database
versions ranging from 10gR1 to
12cR1. ASM is always the same version as Oracle Clusterware and can support Oracle
Database versions ranging from
10gR1 to 12cR1.
Oracle 12cR1 introduced Oracle Flex Cluster and Flex ASM. The architecture of Oracle
Clusterware and Oracle
ASM is different from the standard 12cR1 cluster. We will discuss Oracle Flex Cluster and
Flex ASM in Chapter 5. This
chapter will focus on the standard 12cR1 cluster.

Storage Components of Oracle Clusterware

Oracle Clusterware consists of a storage structure and a set of processes running on each
cluster node. The storage
structure consists of two pieces of shared storage: the Oracle Cluster Registry (OCR) and
voting disk (VD) plus two local
files, the Oracle Local Registry (OLR) and the Grid Plug and Play (GPnP) profile.
OCR is used to store the cluster configuration details. It stores the information about the
resources that Oracle
Clusterware controls. The resources include the Oracle RAC database and instances,
listeners, and virtual IPs (VIPs)
such as SCAN VIPs and local VIPs.
The voting disk (VD) stores the cluster membership information. Oracle Clusterware uses
the VD to determine
which nodes are members of a cluster. Oracle Cluster Synchronization Service daemon
(OCSSD) on each cluster node
updates the VD with the current status of the node every second. The VD is used to
determine which RAC nodes are
still in the cluster should the interconnect heartbeat between the RAC nodes fail.
Both OCR and VD have to be stored in a shared storage that is accessible to all the servers
in the cluster. They can
be stored in raw devices for 10g Clusterware or in block devices in 11gR1 Clusterware.
With 11g R2 and 12cR1 they
should be stored in an ASM disk group or a cluster file system for a freshly installed
configuration. They are allowed
to be kept in raw devices and block devices if the Clusterware was just being upgraded
from 10g or 11gR1 to 11gR2;
however, it is recommended that they should be migrated to an ASM disk group or a
cluster file system soon after
the upgrade. If you want to upgrade your Clusterware and Database stored in raw devices
or block devices to Oracle
Clusterware 12c and Oracle Database 12c, you must move the database and OCR/VDs to
ASM first before you do the
upgrade, as Oracle 12c no longer supports the use of raw device or block storage. To
avoid single-point-of failure, Oracle
recommends that you should have multiple OCRs, and you can have up to five OCRs. Also,
you should have at least three
VDs, always keeping an odd number of the VDs. On Linux, the /etc/oracle/ocr.loc file
records the OCR location:
$ cat /etc/oracle/ocr.loc
ocrconfig_loc=+VOCR
local_only=FALSE
In addition, you can use the following command to find the VD location:
$ ./crsctl query css votedisk
The Oracle ASM disk group is the recommended primary storage option for OCR and VD.
Chapter 5 includes a
detailed discussion of storing OCR and VDs in an ASM disk group.
30CHAPTER 2 ■ CLUSTERWARE STACK MANAGEMENT AND TROUBLESHOOTING
Two files of Oracle Clusterware (OLR) and GPnP profile are stored in the grid home of the
local file system of
each RAC node. OLR is the OCR’s local version, and it stores the metadata for the local
node and is managed by the
Oracle High Availability Services daemon (OHASD). OLR stores less information than OCR,
but OLR can provide this
metadata directly from the local storage without the need to access the OCR stored in an
ASM disk group. One OLR is
configured for each node, and the default location is in $GIHOME/cdata/<hostname>.olr.
The location is also recorded
in /etc/oracle/olr.loc, or you can check it through the ocrcheck command:
$ cat /etc/oracle/olr.loc
olrconfig_loc=/u01/app/12.1.0/grid/cdata/knewracn1.olr
crs_home=/u01/app/12.1.0/grid
$ ocrcheck -local -config
Oracle Local Registry configuration is :
Device/File Name : /u01/app/12.1.0/grid/cdata/knewracn1.olr
The GPnP profile records a lot of important information about the cluster, such as the
network profile and the VD.
The information stored in the GPnP profile is used when adding a node to a cluster.
Figure 2-1 shows an example of the
GPnP profile. This file default is stored in
$GRID_HOME/gpnp/<hostname>/profiles/peer/profile.xml.
Figure 2-1. GPnP profile

Clusterware Software Stack

Beginning with Oracle 11gR2, Oracle redesigned Oracle Clusterware into two software
stacks: the High Availability
Service stack and CRS stack. Each of these stacks consists of several background processes.
The processes of these two
stacks facilitate the Clusterware. Figure 2-2 shows the processes of the two stacks of
Oracle 12cR1 Clusterware.
31CHAPTER 2 ■ CLUSTERWARE STACK MANAGEMENT AND TROUBLESHOOTING
32
High Availability Cluster Service Stack
The High Availability Cluster Service stack is the lower stack of the Oracle Clusterware. It
is based on the Oracle High
Availability Service (OHAS) daemon. The OAHS is responsible for starting all other
clusterware processes. In the next
section, we will discuss the details of the clusterware sequences.
OHAS uses and maintains the information in OLR. The High Availability Cluster Service
stack consists of the
following daemons and services:
GPnP daemon (GPnPD): This daemon accesses and maintains the GPnP profile and
ensures that all the nodes
have the current profile. When OCR is stored in an ASM diskgroup, during the initial
startup of the clusterware, OCR is
not available as the ASM is not available; the GPnP profile contains enough information to
start the Clusterware.
Oracle Grid Naming Service (GNS): This process provides the name resolutions with the
cluster. With 12cR1, GNS
can be used for multiple clusters in contrast to the single-cluster version.
Grid Interprocess Communication (GIPC): This daemon supports Grid Infrastructure
communication by
enabling Redundant Interconnect Usage.
Multicast Domain Name Service (mDNS): This daemon works with GNS to perform name
resolution.
This stack also includes the System Monitor Service daemon (osysmond) and Cluster
Logger Service daemon
(ologgerd).

The CRS Stack

The CRS stack is an upper-level stack of the Oracle Clusterware which requires the
support of the services of the lower
High Availability Cluster Service stack. The CRS stack includes the following daemons and
services:
CRS: This service is primarily responsible for managing high availability operations. The
CRS daemon (CRSD)
manages the cluster resource’s start, stop monitor, and failover operations. CRS maintains
the configuration
information in OCR. If the cluster has an Oracle RAC database, the resources managed by
CRS include the Oracle
database and its instances, listener, ASM instance, VIPs, and so on. This service runs as
the crs.bin process on
Linux/Unix and OracleOHService on Windows.
CSS: This service manages and monitors the node membership in the cluster and updates
the node status information
in VD. This service runs as the ocssd.bin process on Linux/Unix and OracleOHService
(ocssd.exe) on Windows.
CSS Agent: This process monitors, starts, and stops the CSS. This service runs as the
cssdagent process on
Linux/Unix and cssdagent.exe on Windows.
CSS Monitor: This process works with the cssdagent process to provide the I/O fencing to
ensure data integrity
by rebooting the RAC node in case there is an issue with the ocssd.bin process, a CPU
starvation, or an OS locked up.
This service runs as cssdmonitor on Linux/Unix or cssdmonitor.exe on Windows. Both
cssdagent and cssdmonitor
are the new features started in 11gR2 that replace the previous Oracle Process Monitor
daemon (oprocd) in 11gR1.
Cluster Ready Service
Technology Stack
High Availability Service
Technology Stack
CRS
CSS
cssdagent
ASM
CTSS
EVM
ONS
oraagent
orarootagent
ologgerd
appagent
GPNPD
GIPC
mDNS
oraagent
mDNS
scriptagent
osymond
orarootagent
Oracle Clusterware 12c Technology Stack
Figure 2-2. Oracle Clusterware 12cR1 stackCHAPTER 2 ■ CLUSTERWARE STACK
MANAGEMENT AND TROUBLESHOOTING

33
Cluster Time Synchronization Service (CTSS): A new daemon process introduced with
11gR2, which handles the
time synchronization among all the nodes in the cluster. You can use the OS’s Network
Time Protocol (NTP) service to
synchronize the time. Or, if you disable NTP service, CTSS will provide the time
synchronization service. This service
runs as the octssd.bin process on Linux/Unix or octssd.exe on Windows.
Event Management (EVM): This background process publishes events to all the members of
the cluster. On
Linux/Unix, the process name is evmd.bin, and on Windows, it is evmd.exe.
ONS: This is the publish and subscribe service that communicates Fast Application
Notification (FAN) events.
This service is the ons process on Linux/Unix and ons.exe on Windows.
Oracle ASM: Provides the volume manager and shared storage management for Oracle
Clusterware and Oracle
Database.
Clusterware agent processes: Oracle Agent (oraagent) and Oracle Root Agent
(orarootagent). The oraagent
agent is responsible for managing all Oracle-owned ohasd resources. The orarootagent is
the agent responsible for
managing all root-owned ohasd resources.

Clusterware Startup Sequence

Oracle Clusterware is started up automatically when the RAC node starts. This startup
process runs through several
levels. Figure 2-3 shows the multiple-level startup sequences to start the entire Grid
Infrastructure stack plus the
resources that Clusterware manages.
ASM
GNSVIP
Node VIP
Network
sources
ACF Registry

Listener
DB Resource
ASM
instance
SCAN
Listener
Services
GSD
GPNPD
mDNSD
GIPCD
OHASD
oraagent
OHASD
oraclerootagent
Diskgroup
SCANIP
cssdagent
cssdmonitor
CRSD
Diskmon
CTSSD
CRSD
orarootagent
EVMD

ONS
eONS
GNS
CRSD
oraagent
Level 0 Level 1 Level 2 Level 3 Level 4
GPNPD
Process on the High Availability Stack
CTSSD
Process on the Cluster Ready Service Stack
CSSD
Services
Resource managed by Cluster Ready Service

Figure 2-3. Startup sequence of 12cR1 Clusterware processes

INIT
OHASDCHAPTER 2 ■ CLUSTERWARE STACK MANAGEMENT AND TROUBLESHOOTING

34
Level 0: The OS automatically starts Clusterware through the OS’s init process. The init
process spawns only
one init.ohasd, which in turn starts the OHASD process. This is configured in the
/etc/inittab file:
$cat /etc/inittab|grep init.d | grep –v grep
h1:35:respawn:/etc/init.d/init.ohasd run >/dev/null 2>&1 </dev/null
Oracle Linux 6.x and Red Hat Linux 6.x have deprecated inittab. init.ohasd is configured
in startup
in /etc/init/oracle-ohasd.conf:
$ cat /etc/init/oracle-ohasd.conf
......
start on runlevel [35]
stop on runlevel [!35]
respawn
exec /etc/init.d/init.ohasd run >/dev/null 2>&1 </dev/null
This starts up "init.ohasd run", which in turn starts up the ohasd.bin background process:
$ ps -ef | grep ohasd | grep -v grep
root 4056 1 1 Feb19 ? 01:54:34 /u01/app/12.1.0/grid/bin/ohasd.bin reboot
root 22715 1 0 Feb19 ? 00:00:00 /bin/sh /etc/init.d/init.ohasd run
Once OHASD is started on Level 0, OHASD i