Oracle RAC - New Generation

1
Oracle Real Application Clusters
(RAC)
Anil Nair
Sr Principal Product Manager,
Oracle Real Application Clusters (RAC)
@RACMasterPM
http://www.linkedin.com/in/anil-nair-01960b6
http://www.slideshare.net/AnilNair27/
Next Generation Oracle RAC
Copyright © 2020, Oracle and/or its affiliates |

Safe harbor statement
The following is intended to outline our general product direction. It is intended for information
purposes only, and may not be incorporated into any contract. It is not a commitment to deliver any
material, code, or functionality, and should not be relied upon in making purchasing decisions. The
development, release, timing, and pricing of any features or functionality described for Oracle’s
products may change and remains at the sole discretion of Oracle Corporation.

Oracle RAC provides HA and Scalability
Available and Active Active Instances scale Writes, Reads and Hybrid workloads
Private
Network
Failover
Private
Network
0
5000
10000
15000
20000
25000
30000
35000
40000
4 8 32 48 64 80
2035
4010
15520
22416
30016
37040
2 Nodes
3 Nodes
4 Nodes
5 Nodes
# of Cores
Users
*SAP certified SD Benchmark results

Autonomous Database = Oracle RAC on Exadata +..
Autonomous
Database
Automated
Data Center Operations
Oracle Cloud
• Oracle RAC is enabled on Oracle Autonomous Cloud offering
• Oracle RAC meets and exceeds the stringent Autonomous Transaction
Processing Dedicated (ATP-D) and Serverless (ATP-S) requirements
• Successfully providing scalability and availability to the Oracle Database

Oracle RAC scales Autonomous Database workloads
32

Scalable Sequences
Continuous Application
Oracle RAC Sharding
Flex Disk Group
Node Weighting
Gold Image based Install
Application Continuity (AC)
Oracle Flex ASM & Flex Clusters
Fleet Patching and Provisioning (FPP)
Cluster Health Monitor (CHM)
Oracle Quality of Service Management (QoS)
Policy-Based Cluster Management
Oracle RAC One Node & RACcheck
Oracle ASM Cluster File System (ACFS)
Oracle Grid Infrastructure (GI)
UCP and OCI Load Balancing Support for RAC
Cluster Verification Utility (CVU)
Cluster-Managed Services
Oracle Clusterware
Oracle Automatic Storage Management (ASM)
Oracle Real Application Clusters (RAC) Oracle 9i
Oracle 10g
Oracle 11g
Oracle
12c
Oracle 18c
* Documented features list is selective; 20 years include development time
Oracle RAC features power Autonomous DB
Oracle 19c
Co-Location Tag for Client Routing
Dynamic Services Fallback option
RAC Resource Runtime management
20-years of continuous innovation*

New Generation Oracle RAC
Scalable Sequences
Continuous Application Availability
Oracle RAC Sharding
Cluster Health Advisor (CHA)
Rapid Home Provisioning (RHP)
Oracle 11g
Oracle 12g
Oracle
18c
Oracle
19c
5 x
faster
Scalability
* Documented features list is selective

Scalable Sequences
Continuous Application Availability
Oracle RAC Sharding
Cluster Health Advisor (CHA)
Rapid Home Provisioning (RHP)
Oracle 11g
Oracle 12g
Oracle
18c
Oracle
19c
4 x
faster
1.5 x
faster
Availability
* Documented features list is selective

• Major architectural improvements
enable applications to achieve
better availability and scalability
• Ready to meet the needs of next
generation workloads and
applications.
• Ensuring that regardless of failure,
the stack will continue to run
• with minimum or no interruption to
user sessions
• on both On-premises (Generic and
Engineered Systems) and Oracle
Cloud environments
9
Availability Scalability
5x faster6x Improvement

Oracle RAC Development Focus Areas
Better scalability
& availability
*Both Generic and Engineered
systems
Improved Session
management
*connection & faster Failover
Application Server
Efficient life cycle
management for
*Install, Patch, Manage Storage,
Performance

SGA composition is very similar to
Single Instance
• Buffer Cache
• Shared Pool
• Optional In-memory
• Misc. (Java Pool, Result Cache etc.)
Oracle RAC instances includes
additional internal components for
Buffer Cache Management
Simplified Buffer Cache Management
Buffer
Cache
Shared
Pool
In-
Memory
Misc
Total SGA
S
G
A
S
G
A
S
G
A
Buffer
Cache
Shared
Pool
In-
Memory
Misc
Buffer
Cache
Shared
Pool
In-
Memory
Misc
Shared
Pool
In-
Memory
Misc
Buffer
Cache

• Internally each Buffer has a master
which can be same instance or
another instance
• Block master is referred to as Resource
Master
• Resource Master tracks the state of
the buffer (dirty, modified, clone
etc.)
• Single Master can track state of
different Buffers
• “NO” manual setting or
intervention required
Oracle RAC Buffer Cache Optimizations
B BB B
B BB B
B BB B
B BB B
B BB B
B BB B
B BB B
B BB B
B BB B
B BB B
B BB B
B BB B
B BB B
B BB B
B BB B
B BB B
M MM M
M MM M
M MM M
M MM M
M MM M
M MM M
M MM M
M MM M
B Buffer M MasterB M
Total SGA
1 2 3 4

• Oracle RAC processes
• LMS, LMD, LCK, LMON, DIAG
• More fine grained processes
introduced over the years
• Dia0 – Hang Manager
• Gcr0- Global Conflict Resolution slave
• Insv – DG broker communications
• IPC0- Basic messaging
• Lddn – LMD slave
• Ping – Interconnect measurement
• ACMS – CF Writes
• Threaded LMS process starting
with Oracle RAC 18c
Oracle RAC Process optimization
ps -o pid,cls,priority,cmd –e|grep _lms
25111 TS ora_lms0_db1db21 <- 19.4 RDBMS
25113 TS ora_lms1_db1db21 <- 19.4 RDBMS
28682 RR ora_lms0_ustrapt11 <- 12.2 RDBMS
28684 RR ora_lms1_ustrapt11 <- 12.2 RDBMS

• Oracle RAC processes
• LMS, LMD, LCK, LMON, DIAG
• More fine grained processes
introduced over the years
• Dia0 – Hang Manager
• Gcr0- Global Conflict Resolution slave
• Insv – DG broker communications
• IPC0- Basic messaging
• Lddn – LMD slave
• Ping – Interconnect measurement
• ACMS – CF Writes
• Threaded LMS process starting
with Oracle RAC 18c
Oracle RAC Process optimization
$ps -eLo 'pid tid cls cmd ' |grep cdb1|grep '_lm'
6970 6970 TS ora_lmon_rws17c
6972 6972 TS ora_lmd0_rws17c
6974 6977 RR ora_lms0_rws17c
6979 6979 RR ora_lmhb_rws1
6974 6981 TS ora_lms1_rws17c
6974 6983 TS ora_lms2_rws17c
LMS runs in RR LMS slaves run in TS

Timings with different #LMS:
• Total reconfiguration time for an
instance leave & re-join
• 100GB cache
• 2 node RAC
Reconfiguration Performance as of 19c
Buffer Cache Size Reconfiguration
Time
25GB 3.0 sec
50GB 4.9 sec
100GB 8.3 sec
Timings with different cache
sizes:
• Total reconfiguration time for an
instance leave & re-join
• 8 LMS’s
• 2 node RAC
# LMS Reconfiguration
Time
8 LMS’s 8.3 sec
16 LMS’s 5.0 sec
32 LMS’s 3.6 sec

• Sessions can connect to any instance
to retrieve data
• Query from Inst 4 needs a block
which is mastered on Node 1
• It will send message to Block Master
on Node 1
• Node 1 makes a note of this request
and grants the appropriate lock
• Node 3 sends message to Node 2 to
send the block
• 3-Way communication
• Worse case scenario
Fast and Consistent results
Query
Block
Master
Block
Block with
old SCN
1
3
2
1 2 3 4

• Resource master automatically
moved to the instance where
the buffer resides
• Optimizations during startup
ensure that Master is on
instance where most
modifications occur
• Any change during runtime is
managed via DRM (Dynamic
Resource Mastering)
Optimizations in Oracle RAC Buffer Cache management
Total SGA
1 2 3 4

Oracle RAC features for
Scalability Availability

Oracle RAC Always Chooses the “Right Path”
• RAC determines the optimal
path to serve blocks – network
or disk
• SSDs and NVMe storage
technology continue to drive
down latency
• e.g. flash storage may provide
better access times to data than
the private network under high
load
• RAC takes those statistics into
account
Query
BlockBlock
Network
congestion
Benefit from storage improvements

LMS Slaves to mitigate Performance Outliers

Introducing LMS CR Slaves
helps mitigate Performance outliers
• In previous releases, LMS work on incoming consistent read
requests in sequential fashion
• Sessions requesting consistent blocks that require applying lot of
undo may cause LMS to be busy
• Starting with Oracle RAC 12c Release 2, LMS offloads work to ‘CR
slaves’ if the amount of UNDO to be applied exceeds a certain,
dynamic threshold
• Default is 1 slave and additional slaves are spawned as needed

Service-oriented Buffer Cache Access
Optimize for performance by co-locating data with the service to which it belongs
NodeA NodeB
• Service-oriented Buffer Cache Access over time
determines the data (on database object level)
accessed by the service. This information
– Is persisted in the database.
– Is used to improve data access performance
(e.g. do not manage data of a service in an instance
that does not host the service).
– Can be used to pre-warm an instance cache prior to
a service startup (fresh start or relocation).
• Access level statistics are stored in the PDB
dictionary

Hang Manager
• Hang Manager “manages” database
process(es) that are not progressing
• Manage includes detection and resolution
• Considers Cross-Layer Hangs
I.E: Hangs caused by blocked ASM resource
• In addition, deadlocks are now
resolved by Hang Manager
• User defined control via PL/SQL Early
Warning exposed via (V$ view)
• New in 19c is the integration of DRF and
Hang Manager to use DRF statistics
Hang detection and resolution of hangs including cross-layer hangs in the Cluster

Hang Manager interventions reported via ORA-32701
Dump file …/diag/rdbms/hm6/hm62/incident/incdir_5753/hm62_dia0_12656_i5753.trc
Oracle Database 12c Enterprise Edition Release 12.2.0.0.0 - 64bit Beta
With the Partitioning, Real Application Clusters, OLAP, Advanced Analytics
and Real Application Testing options
Build label: RDBMS_MAIN_LINUX.X64_151013
ORACLE_HOME: …/3775268204/oracle
System name: Linux
Node name: slc05kyr
Release: 2.6.39-400.211.1.el6uek.x86_64
Version: #1 SMP Fri Nov 15 13:39:16 PST 2013
Machine: x86_64
VM name: Xen Version: 3.4 (PVM)
Instance name: hm62
Redo thread mounted by this instance: 2
Oracle process number: 19
Unix process pid: 12656, image: oracle@slc05kyr (DIA0)
*** 2015-10-13T16:47:59.541509+17:00
*** SESSION ID:(96.41299) 2015-10-13T16:47:59.541519+17:00
*** CLIENT ID:() 2015-10-13T16:47:59.541529+17:00
*** SERVICE NAME:(SYS$BACKGROUND) 2015-10-13T16:47:59.541538+17:00
*** MODULE NAME:() 2015-10-13T16:47:59.541547+17:00
*** ACTION NAME:() 2015-10-13T16:47:59.541556+17:00
*** CLIENT DRIVER:() 2015-10-13T16:47:59.541565+17:00
2015-10-13T16:47:59.435039+17:00
Errors in file /oracle/log/diag/rdbms/hm6/hm6/trace/hm6_dia0_12433.trc (incident=7353):
ORA-32701: Possible hangs up to hang ID=1 detected
Incident details in: …/diag/rdbms/hm6/hm6/incident/incdir_7353/hm6_dia0_12433_i7353.trc
2015-10-13T16:47:59.506775+17:00
DIA0 requesting termination of session sid:40 with serial # 43179 (ospid:13031) on instance 2
due to a GLOBAL, HIGH confidence hang with ID=1.
Hang Resolution Reason: Automatic hang resolution was performed to free a
significant number of affected sessions.
DIA0: Examine the alert log on instance 2 for session termination status of hang with ID=1.
2015-10-13T16:47:59.538673+17:00
Errors in file …/diag/rdbms/hm6/hm62/trace/hm62_dia0_12656.trc (incident=5753):
ORA-32701: Possible hangs up to hang ID=1 detected
Incident details in: …/diag/rdbms/hm6/hm62/incident/incdir_5753/hm62_dia0_12656_i5753.trc
2015-10-13T16:48:04.222661+17:00
DIA0 terminating blocker (ospid: 13031 sid: 40 ser#: 43179) of hang with ID = 1
requested by master DIA0 process on instance 1
Hang Resolution Reason: Automatic hang resolution was performed to free a
by terminating session sid:40 with serial # 43179 (ospid:13031)
ORA-32701: Possible hangs up to hang ID=1
detected
Incident details in:
…/../hm62_dia0_12656_i5753.trc
DIA0 terminating blocker
(ospid: 13031 sid: 40 ser#: 43179)
requested by master DIA0 process on
instance 1
Hang Resolution Reason: Automatic hang
resolution was performed to free a
Alert log

• Pre-12.2, node eviction follows
a rather “ignorant” pattern
• Example in a 2-node cluster: The node
with the lowest node number survives.
• Customers must not base their
application logic on which node
survives the split brain.
• As this may(!) change in future
releases
Node Eviction Basics
1

• Node Weighting is a new feature that
considers the workload hosted in the
cluster during fencing
• The idea is to let the majority of
work survive,
if everything else is equal
• Example: In a 2-node cluster, the
node hosting the majority of
services (at fencing time) is meant
to survive
Node Weighting in Oracle RAC 12c Release 2
1
More services aka work
Idea: Everything equal, let the majority of work survive

Secondary failure consideration can influence
which node survives. Secondary failure
consideration will be enhanced successively.
30
Smarter Eviction with Node Weighting
A fallback scheme
is applied if considerations do not
lead to an actionable outcome.
Public network card failure.Public network card failure.

CSS_CRITICAL can be set on various levels /
components to mark them as “critical” so that the
cluster will try to preserve them in case of a failure.
Customized Eviction with Node Weighting
CSS_CRITICAL will be honored if no other technical
reason prohibits survival of the node which has at
least one critical component at the time of failure.
?
CriticalCritical More services aka work

Memory Guard – Attempt to prevent Evictions
• Monitors memory allocated and
detects conditions that may result in
swapping
• Issues alert should any server be at
risk
• Stops services transactionally on the
server
• This prevents any new applications
connections
• Re-starts the services once the
excess memory allocation issue is
cleared
Over provisioning of memory is
the leading cause of evictions

Near Zero Downtime Reconfiguration with Recovery Buddy
6x
faster! *
• Tracks blocks changes on buddy
instance
• Quickly identifies blocks requiring
recovery during reconfiguration
• Allows rapid processing of new
transactions
• * ~6 times faster with “Recovery
Buddy” and Optimized (Singleton)
Reconfiguration Time
- 4x compared to 11gR2 and 12cR2c
- 1.5x compared to 11gR2 and 18c
• Fast track non mastered resources
Optimize recovery to dramatically reduce brownouts

• Detect Node/Instance Hang/Death
• Evict the dead/hung Instance/Node
• Elect a Recovery Master (RM)
• One of the surviving instance process
(SMON) will get lock and be elected RM
• RM will then
• read redo of evicted instance
• apply recovery
• signal completion
High Level Reconfiguration Stages
34
Detect
Evict
Elect
Recovery
Read
Redo
Apply
Recovery

• Buddy Instances eliminate the
“Elect Recovery Master” phase
• Redo-read is optimized
via memory-reads
• Apply recovery is optimized as
switching between read and
writes is no longer required
35
Detect
Evict
Elect
Recovery
Read
Redo
Apply
Recovery
Reduced Reconfiguration time with “Recovery Buddy”

1. Buddy Instance mapping is simple ( 1
à 2, 2 à 3, 3 à 4, 4 à 1)
2. Recovery buddies are set
during instance startup
3. RMS0 on each recovery buddy
instance maintains an in-memory
area of redo log change
4. The in-memory area is used during
recovery therefore eliminating the
need to physically read the redo
Buddy Instances – Under the Hood
36
1. Inst1 is recovery buddy for Inst2
2. Inst2 is recovery buddy for 3 and so-on
3. Recovery buddy mapping will change as
new instances join or leave
For e.g If inst3 crashes, a new recovery
buddy will be assigned to Inst 4
Inst 1 Inst 2 Inst 3 Inst 4
Recovery
Buddy 2
Recovery
Buddy 3
Recovery
Buddy 4
MyCluster
Recovery
Buddy 1

During planned shutdown
• Distribute Resource Masters before
shutdown
• Distribution before shutdown does not
require any recovery on the surviving
instances
• Effectively reduces the time spent on
reconfiguration during planned outages
to zero
• During unplanned outages
• Rolling Windows to reduce impact
of reconfiguration
• During runtime
• Smart DRM
Zero Downtime for Planned outages
B BB B
B BB B
B BB B
B BB B
B BB B
B BB B
B BB B
B BB B
B BB B
B BB B
B BB B
B BB B
B BB B
B BB B
B BB B
B BB B
M MM M
M MM M
M MM M
M MM M
M MM M
M MM M
M MM M
M MM M
Global Resource Directory
B Buffer M MasterB M

Oracle RAC HA and Scalability features for
MultiTenant In MemoryData Guard
Even higher HA
and Scalability
on Exadata
&
* selective features depicted

• Until now, Data Guard Standby
redo Apply typically runs on
one instance
• All the instances on the
standby site can receive the
changes from RFS
• But only one instance will be
running the media recovery
process (MRP) process
Data Guard Standby Redo Apply
1

Multi-Instance Redo Apply
Parallel, multi-instance recovery means “the standby DB will keep up”
• Standby recovery - utilizes CPU and I/O across all nodes of RAC standby
• Up to 3500MB+/sec apply rate on an 8 node RAC
Multi-Instance Apply runs on all MOUNTED instances or all OPEN Instances
Exposed in the Broker with the ‘ApplyInstances’ property on standby
Utilize all the RAC Instances on standby to apply Redo
recover managed standby database disconnect using instances 4;

• Data Guard Standby redo Apply can
now run on ALL instances or a
subset of instances
• Media recovery is performed in
parallel as media recovery process
(MRP) process is running on
multiple instances
• MIRA ensures standby can keep up
with changes coming from multiple
Oracle RAC instances on Primary
Multi-Instance Redo Apply (MIRA)
1 2

Oracle RAC provides HA and Scalability to
MultiTenant In MemoryData Guard
Even higher HA
and Scalability
on Exadata
&
* selective features depicted above

Single Domain (domain 0) at the
CDB level for all PDBs
Operations like PDB start, stop,
reconfigurations etc rely on the
single Domain
Accessing a resource from hash
table using single Domain
affected as the number of PDBs
increases (> 250) Pdb-1
Pdb-2
Pdb-3
Pdb-4
Domain 0
Instance
SGA
Resource Hash Table
Oracle Multitenant Domains

With Oracle Database 12c Rel. 2, Each
PDB gets its own PDB specific domain
ID
GES & GCS resources are balanced
across PDB-specific domain ID
Resulting in improved,
consistent performance
• One chatty PDB will not affect the
performance of any another PDB
• Reduced Hash table size as hash table is
only created on the instances where PDB is
running
Oracle Multitenant Scalability Optimization
Domain1
Resource Hash Table
Resource Hash Table
Resource Hash Table
Resource Hash Table
Domain 2
Domain 3
Domain 4

Oracle RAC optimization for Multitenant
Prevents “noisy neighbors” from affecting others with unnecessary chatter
NodeA NodeB
Using Oracle Multitenant, PDBs can be
opened as singletons (in one database
instance only), in a subset of instances or all
in instances at once.
If certain PDBs are only opened on some
instances, Pluggable Database Isolation
• improves performance by
- Reducing DLM operations for
PDBs not open in all instances.
- Optimizing block operations based
on in-memory block separation.
MSG
Messages (MSG)

Oracle RAC PDB Service Isolation
Limits impact of instance failures where PDBs are running
NodeA NodeB
Isolation ensures that instance
failures of instances only
hosting singleton PDBs will not
impact other instances of the
same RAC-based CDB.

Oracle RAC with Database In-Memory
• Auto DOP must be used to
access Data
• With Auto DOP query
coordinator is aware of
IMCU locations
• Query coordinator
automatically starts parallel
server processes on the
correct nodes

Unique Fault Tolerance on Exadata
• Similar to storage mirroring
• Duplicate in-memory
columns on another node
Enabled per table/partition
Application transparent
• Downtime eliminated by
using duplicate after failure
Only Available on Engineered Systems

• Subnet Manager for Instant Failure Detection
• Network (Subnet Manager)
• Disk (Diskmon)
• More details available at
• https://www.slideshare.net/AnilNair27/oracle-rac-
features-on-exadata
Oracle RAC Exadata optimizations
Instant Failure Detection instantly detects node death
2018-02-03 13:33:27.724 : CSSD:866105088:
clssnmeventhndlr: EXADATA NodeDeath detection from
GIPC for node xyzadm1 number 1
2018-02-03 13:33:28.457 : CSSD:971667200:
clssnmvDiskEvict: Begin: Kill block write, file
CSS receives Node death event from GIPC
Eviction detected in less than 1 sec
13:33:27.724
13:33:28.457
00:00:00.733

Cache Fusion Optimizations on Exadata
• Utilize RDMA for
• Propagating BoC (Broadcast on commit)
• Reduce messages (2 instead of 5)
• Reduce CPU usage on LMS
• Sending current read blocks
• Very common access pattern in various workloads
• Sending Commit Cache messaging
• https://www.slideshare.net/AnilNair27/oracle-rac-features-on-exadata (Slide#43)
• Reduces load on LMS from remote node
• Direct read than sending an 8k block
• Sending Undo block
• https://www.slideshare.net/AnilNair27/oracle-rac-features-on-exadata (Slide#44)

Right Growing Index
- Not really a RAC specific issue
• Affects Scalability due to waits on
Index splits
When possible utilize Scalable
Sequences*
Optimized Leaf block splits
• Reduce Messaging
Cache Fusion on Exadata
- Commit Cache, Undo Block RDMA Read,
Smart Fusion Block transfer
Solving Right Growing Index problem
0
10
20
30
40
50
60
0 8 16 24 32 40 48 56 64
Executions/sec(thousands)
# Clients / Instance (4 RAC instances)
11.2.0.4 Baseline
12.1.0.2 Baseline
12.2 with all
optimizations
1 2 3 4 5 6 7
1 2 3 4 5 6 7 8 9 10
5 6 7 8 9 10 11 12
11 12 13 14 158 9 10

Detecting (RGI) Performance Issues
Total Wait Avg % DB Wait
Event Waits Time (sec) Wait time Class
------------------------------ ----------- ---------- --------- ------ --------
gc buffer busy release 556,910 1484.3 2.67ms 32.0 Cluster
enq: TX - index contention 74,060 1006.1 13.58ms 21.7 Concurre
gc buffer busy acquire 853,180 860.7 1.01ms 18.5 Cluster
gc index operation 167,564 721.6 4.31ms 15.5 Cluster
buffer busy waits 235,076 176.2 749.34us 3.8 Concurre
DB CPU 167.5 3.6
latch: cache buffers chains 355,780 97.6 274.20us 2.1 Concurre
latch: ges resource hash list 66,582 46.9 703.73us 1.0 Other
gc current split 6,088 23.4 3.84ms .5 Cluster
gc current block busy 17,548 21.9 1.25ms .5 Cluster
22% for global enqueue handling
50% for global buffer contention

• Improved algorithm reduced the
high rate of leaf node split cancels
under higher concurrency due to
ongoing cleanouts
• Code optimizations improve
concurrency (reduce the cancels)
and improve diagnosability
• Internal benchmark reported a
significant performance
improvement
Improving (RGI) Workload Performance
Baseline Optimized
Overall throughput 5,278 9,381
Leaf node splits /
sec
42 82
Split cancels / sec 3.8 0.7
0
1000
2000
3000
4000
5000
6000
7000
8000
9000
10000
Base Line
Optimized
50%

• Sequences cause Index block
contention during heavy OLTP
especially on tables which use
sequence values as keys
• This is not RAC specific but the
problem gets amplified in RAC
environments
• The effects of this issue is mitigated
via Scalable Sequences
Sequence contention

New option in Create/Alter Sequence
SCALE {Extend |NoExtend | NoScale }
When Scale is specified an internal
algorithm uses the instance id and
other session specific information to
generate an offset
DBAs need to alter existing sequences
DBA/User/ALL_sequences will reflect
the additional attributes
Further Improve RGI workload performance using Scalable Sequences
Scalable Sequences

Maintain a small in-memory cache of committed transactions
Transfer multiple XID’s efficiently in a single batched message
In-memory Transaction Table
0 0.05 0.1 0.15 0.2 0.25 0.3
Base 12.2
Tx Table Enabled
Total Transfers / sec
(normalized by throughput)
Data Block (current)
TX Header (CR)
TX Table Message
14x
Improvement

Undo Block RDMA-read
• In some workloads, more than half of remote reads
are for Undo blocks to satisfy read consistency
• Undo Block RDMA-read uses RDMA to directly and
rapidly access UNDO blocks in remote instances
Commit Cache
• Commit Cache maintains an in-memory table on
each instance which records the commit time of
transactions
• Remote LMS directly reads the commit cache and
sends back commit times for requested
transactions.
• Replaces having to send entire 8K transaction table
block
RAC Optimizations for Exadata
UNDOUNDO
RDMA RDMA

• On Exadata, Oracle does not wait for
the log write notification
• Exadata ensures the log write
completes before changes to block
on another instance commit,
guaranteeing durability
• Wait for Log I/O during transfer of
hot blocks is eliminated
• Up to 40% throughput and 33%
response time improvement in some
heavily contended OLTP workloads
• Storage software ensures correct
ordering of writes
Smart Fusion Block Transfer
1. Issue log write
2. Wait for log
write completion
3. Transfer
block
Exadata Avoids I/O Wait confirmation
Storage

• Exafusion direct-to-wire optimization
eliminates costly context switches
• Works on both RoCE/InfiniBand
deployments
• Exafusion along with IPCLW provides
blazing performance
• Database directly calls InfiniBand
hardware
• Bypassing networking software stack,
interrupts, scheduling
Exafusion Direct-to-Wire OLTP Protocol
8K OLTP Block Transfers/sec
RoCE/InfiniBandExafusion
RoCE/InfiniBand
10GigE
3X Speedup

New background: IPC0
• Registers (pins) IPC buffers for all
processes to the InfiniBand HCA
New memory pool: MGA (Managed
Global Area)
• IPC buffers are allocated in the MGA
(used to be the PGA before Exafusion)
• MGA allows a set of processes to
efficiently share an address space
• MGA is not SGA or PGA, but something
in-between
Exafusion_Enabled is deprecated
Providing even more Scalability
InfiniBand/RoCE
HCA
SGA
Buffer
OracleO/S
Buffer
InfiniBand/RoCE
HCA
SGA
Buffer
Oracle
Global Cache
NIC
Driver ,
IP stack
Context switch
IPCWire
usersys

And even Higher Availability
By lowering brownout time
11204
4 x
1.5 x
12.2 18.1
https://www.slideshare.net/AnilNair27/new-availability-features-in-oracle-rac-12c-release-2-anair-ss

• Sessions connect to the database
directly or via connection pool using
SCAN and service name
• SCAN listener redirects the session to
the database listener
• Sessions can benefit from load
balancing (connection/runtime)
• Connection pool gets notified of CPU
load, # of sessions, node failures and
other such events by Oracle Notification
Services
Session Connection Basics
ODP.Net, OCI, Connection Pool
Single Client Access Network
Failover, Load
and other
Notifications

• Public network and
instance health is
constantly monitored
by Grid Infrastructure
• On failure, GI will
relocate the VIP
• Existing connection
requests will receive
error as GI will send a
GARP
• ONS consumers like
SCAN listeners,
Connection Pools get
notified of failure
Failures Handling
ODP.Net, OCI, Connection Pool
Single Client Access Network
New Connection request
Scan Listener routes to
DB listener on first node
Instance or Public
Network failure
GI relocates VIP to
another instance
Connection attempted
on VIPInstant Error instead of
waiting on TCP timeout
Retry Connection
ONS Event Notification
Gratuitous ARP
Arp Table update

Recommended TNS Connector String
ALIAS =(DESCRIPTION =
(TRANSPORT_CONNECT_TIMEOUT=3)
(ADDRESS_LIST =
(CONNECT_TIMEOUT=90)
(RETRY_COUNT=20) (RETRY_DELAY=3)
(LOAD_BALANCE=on)
( ADDRESS = (PROTOCOL = TCP)(HOST=primary-scan)(PORT=1521)))
(ADDRESS_LIST =
(CONNECT_TIMEOUT=120) (RETRY_COUNT=20) (RETRY_DELAY=3)
(LOAD_BALANCE=on)
( ADDRESS = (PROTOCOL = TCP)(HOST=secondary-scan)(PORT=1521)))
(CONNECT_DATA=(SERVICE_NAME = gold-cloud)))
Standard for All Drivers starting with Oracle Database 12.2
Secs to wait for a TCP connection
*Same as SQLNET.Connect_Timeout
Secs to wait for a successful connection
*Same as SQLNET.Outbound_Connect_Timeout

Test your Failover timings
Case1
Case 2
Case 1
Vanilla tnsnames.ora with default
Connect_Timeout, Retry_count, RETRY_DELAY,
TRANSPORT_CONNECT_TIMEOUT
Simulate network failure
$tnsping OK (62, 860 msec)
~~1 minute wait (TCP_TIMEOUT)
Case 2
Tuned tnsnames.ora with optimized values for
Connect_Timeout = 90, Retry_count = 1,
RETRY_DELAY=3,
TRANSPORT_CONNECT_TIMEOUT=3
$tnsping OK (3070 msec) = ~~3 seconds
(Almost no delay)
Commented

Tag your Sessions to optimally route work
• Optimize your connections by co-
locating similar work pattern to
one Oracle RAC Instance
• Oracle RAC routes connections
with same COLOCATION_TAG to
same instance* (when possible)
• ALIAS =(DESCRIPTION =
(CONNECT_DATA=
(SERVICE_NAME = gold-cloud)))
….
(CONNECT_DATA=
(SERVICE_NAME=Orders)
(COLOCATION_TAG=ORDER)
Orders SalesPayroll

• Connect via a service to a
connection pool (CP)
• CP provides a connection to
an instance to perform
transactions
• Session experiences failure
during ongoing work
• CP provides another session to
surviving instance
• Continues to perform work
• Results returned to the user
While benefitting from Application Continuity to mask Failures
1
2
3
4
5
6
UCP, JDBC, ODP.Net,
OCI Session Pool,
Tuxedo, WebLogic
✓
From both planned and unplanned maintenance
1
2
3
4
5
6

How to enable Application Continuity (AC)?
• Database and client (driver) version >= Oracle 12c
• Enable AC on the Service
• $srvctl modify service –db <mydb> -service myac_svc –failovertype
TRANSACTION -commit_outcome TRUE *
*comprehensive steps included in AC white paper (URL included in notes)
• Use the recommended tnsnames.ora or JDBC URL
• Use Orachk: ($orachk –accheck)
• AWR report: (User calls in = User calls protected)

Session
• Connects via a service an
instance and performs insert,
update and delete operations
• Node/Instance failure in the
middle of ongoing work
• Receives Notification
• Reconnects to surviving
instance and continues where
it was left off
• Results returned to the user
Application Continuity (TAC)
UCP, JDBC, ODP.Net,
OCI Session Pool,
Tuxedo, WebLogic
✓
Can this get Any better?
4
5
1
2
3
4
5
2
3
1

Application Continuity (AC), or TAF
My Application uses.. AC TAC TAF
I don’t know No Yes No
Transactions Yes Yes No
Oracle state (temp lobs,
PL/SQL, temp tables, aq..)
Yes Yes No
No connection pool No Yes Yes
Initial state on the session Yes and custom Yes and custom Yes and custom
Side effects (utl_mail,
dbms_file_transfer..)
Yes and Custom Yes Not replayed No

Install
Patch
Manage
Tune
• Installer Optimizations
• Gold Image
• gridSetup.sh
• Patch
• SwitchGridHome
• Opatch
• Storage Management
• Flex Disk Group
• Tune
• AWR Metrics
Life Cycle Management

1. Download base release
• Unzip into GI HOME
• Unzip patch into separate
directory
2. Apply Patch (RU/RUR)
[-applyPSU]
• gridSetup.sh –applyRU
<path_to_patch>
3. Zip, copy and deploy
customized GI Home multiple
times
Create Custom Gold image
• Opatch lsinventory
• Provides detailed home info
• New Versioning scheme
• 19.7 è Version 19, RU 7
• Possible but not recommended
to have 19.4 GI with 19.7 database

• One command to Install
configure & Upgrade
• Add/Delete Node
• Reduce Intermediate steps
• Lenient mode
• DevOps Friendly
gridSetup
$./gridSetup.sh –dryRunForUpgrade

Reduce Installation steps
Image-based (12.1) Image-based (12.2 and higher)
Download shiphome zip files from OTN Download image zip files from OTN
Unzip grid1/2.zip to some stage location (stage_loc) No stage_loc
Execute <stage_loc>/runInstaller.sh -
Bootstraps files to some temp area (1GB) No bootstrapping
Copies files from <stage_loc> to OH No copy needed
Zip up the OH and store it as image files -
Unzip image files to OH on all nodes of the cluster Unzip image files to OH on one of the nodes of
cluster
Execute clone.pl on all nodes of cluster Not needed
Run config.sh from one of the nodes Run gridSetup.sh from one of the nodes

• Environment changes, incorrect
permissions of binaries can prevent
stack startup
• Oracle 19c Clusterware stack
attempts to auto-diagnose
unsuccessful startup issues
• Provides detailed logging in
case of failures
CRS-41053: checking Oracle Grid Infrastructure for file permission
issues
PRVG-2031 : Owner of file ”…gipcd.bin" did not match the
[Expected="grid(54320)" Found="oracle(54325)"]
Oracle Clusterware start Diagnostics
init
cssdAgent
cssdmonitor
ohasd
oraAgent
cssd
crsd
ctssd
HAIP
ACFS
mdnsd
gipcd
evmd
ASM
*NOT all daemons are shown in illustration above
oraRootAgent

gridSetup features for Upgrade, Patching
• SwitchGridHome
• gridSetup –switchgridhome option for out of place
patching
• Autoupgrade
• Fully integrated with orachk (orachk -preupgrade)
• Updates new home using srvctl –upgrade
/u01/app/grid/19.3
/u01/app/grid/19.7

Compute
Storage

• Provision and Patch 11.2, 12.1, 12.2,
18c, 19c Grid Infrastructure &
Databases
• Performs addNode and deleteNode
operations
• Standardizes Customer software
installs via the use of Gold Images
• Provisions Grid Infrastructure, RAC,
RAC One, Single Instance and
Applications
• Automation to save you countless
hours performing these tiring tasks
Fleet Provisioning and Patching & Upgrade
FPP is new name for RHP
Add/Delete
Node
Install Patch
! ! !

Zero Downtime Migration, Upgrade (ZDM, ZDU)
Migrate
!
Upgrade
!

• Configure during Installation
• Reject non-Oracle I/O
• Stops OS utilities from
overwriting ASM disks
• Protects database files
• Reduce OS resource usage
• Fewer open file descriptors
• Faster node recovery
• Further configuration and
monitoring is conducted by
using the AFDTOOL utility:
• Provision a disk:
$ afdtool -add /dev/dsk1 disk1
• Remove a disk:
$ afdtool -delete disk1
• List the managed disks:
$ afdtool -getdevlist
Manage devices with Filter Driver

• New Disk Group type provides fine
grained mirroring and redundancy
• File Group: a database-specific
construct
• Quota Groups: flexible quota
management at a file group level
• Flexible and modifiable database-
specific properties
• ASM point-in-time database clones
• Define quotas at PDB level
Manage storage with Flex Disk Group
Flex Disk Group
DB1
File 1
File 2
File 3
DB2
File 1
File 2
File 3
File 4
DB3
File 1
File 2
File 3
Quota
DB3
File 1
File 2
File 3

• Monitor memory allocation and usage within the
database
• Optimizing memory usage especially in Large SGAs
• Dynamic Internal Structures
• Smarter DRM
• Optimized Messaging
Oracle RAC Database Reliability Framework (DRF)

Database Reliability Framework Details
Runs in LMHB Process
• Re-startable
• Non Fatal
Relies on Metrics and Actions
Action Related metrics
report high cpu oracle instance
processes
cpu load, cpu threshold, bg
heartbeat, cpu load (global)
report high memory oracle
instance processes
memory load, memory
threshold
kill instance memory hog memory load, memory
threshold
enable rm plan cpu load, bg slowing (2)
switch process to elevated priority cpu load, bg slowing (3)
switch process from elevated
priority
cpu load, bg heartbeat
shrink or grow resource cache library cache pin waits
cap total processes at elevated
priority
cpu number
enable drm cpu load, drm checks
disable drm cpu load, drm checks
increase default number of lms cr blocks congested, current
blocks congested
Metric Scope
o/s memory Node Global
o/s load Node Global
bg heartbeat Global
DRM health check Global
library cache pin waits Local
CFIO waits Global
gc block lost Local
gc block busy Local

AWR Report
Integrated AWR and ADDM

Reconfiguration Diagnosability
**************** BEGIN DLM RCFG HA STATS ****************
Total dlm rcfg time (inc 6): 3.586 secs (394926177, 394929763)
Begin step .........: 0.005 secs (394926177, 394926182)
Freeze step ........: 0.019 secs (394926182, 394926201)
Sync 1 step ........: 0.002 secs (394926264, 394926266)
Sync 2 step ........: 0.024 secs (394926266, 394926290)
Enqueue cleanup step: 0.002 secs (394926290, 394926292)
Sync pcm1 step .....: 0.004 secs (394926293, 394926297)
……
….
Enqueue dubious step: 0.004 secs (394926432, 394926436)
Sync 5 step ........: 0.000 secs (394926436, 394926436)
Enqueue grant step .: 0.001 secs (394926436, 394926437)
Sync 6 step ........: 0.012 secs (394926437, 394926449)
Fixwrt replay step .: 0.885 secs (394928837, 394929722)
Sync 8 step ........: 0.040 secs (394929722, 394929762)
End step ...........: 0.001 secs (394929762, 394929763)
Number of replayed enqueues sent / received .......: 2246 / 893
Number of replayed fusion locks sent / received ...: 124027 / 0
Number of enqueues mastered before / after rcfg ...: 2058 / 1384
**************** END DLM RCFG HA STATS *****************
Detailed timing breakdown
available in LMON trace file

DRM Diagnosability
Dynamic Remastering Statistics DB/Inst: SALES/sales1 Snaps: 393-452
-> Affinity objects - Affinity objects mastered at the begin/end snapshot
-> Read-mostly objects - Read-mostly objects mastered at the begin/end snapshot
per Begin End
Name Total Remaster Op Snap Snap
-------------------------------- ------------ ------------- -------- --------
remaster ops 24 1.00
remastered objects 24 1.00
remaster time (s) 7.4 0.31
freeze time (s) 1.5 0.06
cleanup time (s) 2.4 0.10
replay time (s) 0.3 0.01
fixwrite time (s) 2.4 0.10
sync time (s) 0.1 0.01
affinity objects N/A 3 27
read-mostly objects N/A 0 0
read-mostly objects (persistent) N/A 0 0
Detailed timing breakdown
available in AWR Report

Global AWR – Tune multiple instances with one report
Instance 1 is clearly more busier than rest

Global AWR – Summary along with details

Global AWR – Platform specific Information
Histograms to show outliers

Oracle RAC is the #1 proven Choice for Scalability & Availability

Wide choice of Platforms: On-Prem or Cloud
Autonomous
Database
Exadata Cloud
Service
Oracle RAC on Generic
systems in Oracle Cloud
Oracle
Exadata
Generic
System
AutomaticManual

Oracle RAC - New Generation

Recommended

More Related Content

What's hot (20)

Similar to Oracle RAC - New Generation (20)

More from Anil Nair (7)

Recently uploaded (20)

Oracle RAC - New Generation