Back to the roots: Oracle database I/O management

About me

Jérôme Witt
Senior Consultant

+41 79 961 27 73
jerome.witt[at]dbi-services.com

20.11.2018
Who we are

Experts At Your Service

Over 60 specialists in IT infrastructure
Certified, experienced, passionate

Based In Switzerland
100% self-financed Swiss company
Over CHF 10.5 mio. Turnover

Leading In Infrastructure Services

More than 170 customers in CH, D & F
Over 50 SLAs dbi FlexService contracted

20.11.2018
Agenda

1.Quiz
2.I/O request types
3.Oracle I/O management
4.Troubleshooting
5.Core Message

20.11.2018
Safe harbor statement

We won’t have time to dig into all Oracle I/0 management topics
User and System I/O wait classes details
Tracing events (tkprof), Cost Based Optimizer (system statistics)
Hidden parameters, DB_FILE_MULTIBLOCK_READ_COUNT
Lost Write Protection
PGA tuning

The presentation focus on Linux and all demos based on a simple reverse engineering from
system calls using strace and perl
Output of strace program sent to a FIFO pipe
FIFO pipe read from a perl program

We are speaking about fundamentals!

20.11.2018
Quiz

2
Oracle architecture
Background processes
Performance tuning advanced skills
3

20.11.2018
Quiz
Questions

Cloud based infrastructure? DBaaS ? SaaS ?

Feel free!
Get out of this room!

This stuff is for geeks only!

20.11.2018
Quiz
Questions

Which background process wakes up regularly and writes cold, dirty buffers to disk?
Database Writer (DBWR)
Under which circumstances?
When a server process cannot find a clean reusable buffer within the database buffer cache
LRU touch count algorithm, DBWR WriteQueue
Advance database checkpoint position (piggy backing)

Which background process ensures the durability of a transaction?

Log Writer (LGWR)

Does a SAVEPOINT ensures the consistency and durability of a transaction?

ACID paradigm
No! Only COMMIT (and ROLLBACK) trigger LGWR to persistently
Atomicity: The entire sequence of actions must be either completed or aborted.

20.11.2018
Quiz
Compulsive tuning

Customer complaining about poor database performance because of

V$EVENT_HISTOGRAM shows a large number waiting within the “4ms to 8ms” bucket

Do you see the I/O bottleneck isn’t it ?

Top 5 Timed Events Avg %Total
~~~~~~~~~~~~~~~~~~ wait Call
Event Waits Time (s) (ms) Time
----------------------------------------- ------------ ----------- ------ ------
CPU time 1,976,346 87.3
db file sequential read 86,873,066 81,474 1 3.6
enq: TX - row lock contention 7,476,991 65,838 9 2.9
TCP Socket (KGAS) 1,304,085 50,933 39 2.3
HS message to agent 21,725,410 33,065 2 1.5
-------------------------------------------------------------

20.11.2018
I/O request types

1
I/O software layers
Buffered I/O
Direct I/O
2
Blocking vs Non-blocking I/O
Asynch. I/O 3

20.11.2018
I/O request types
I/O software layers (Linux/Unix)

User-level I/O libraries STDIO, C/C++

User space

Device independent O.S software I/O Scheduler,

buffering, caching,
…

Kernel space Device drivers

multipathd,
bonding …

Interrupts handler [ksoftirqd/?]

Hardware (device controllers, network, aso …)

20.11.2018
I/O request types
Buffered I/O concept (Linux)

Concept
PostgreSQL Oracle MySQL Linux maintains (O.S) blocks in a
User space “cache”
Standard (GNU) C libraries (glibc) Cache shared across all physical
block devices
System calls All kind of blocks new & unused
(open, close, read, write, … )
Unused blocks cleaned regularly
Virtual File System from cache
xfs acfs, … ext4

Kernel space Filesystem buffer cache I/O Impact

Reduce access frequency to
physical disks
Device drivers
Low latency to access (O.S)
blocks (nanoseconds)
Hardware

20.11.2018
I/O request types
Non-buffered I/O (directIO)

Direct I/O is a feature of the file system whereby file reads and writes go directly from the
applications to the storage device

An application invokes direct I/O by opening a file with the O_DIRECT flag.
O_DIRECT (Since Linux 2.4.10)
Try to minimize cache effects of the I/O to and from this file.
In general this will degrade performance, but it is use-
ful in special situations, such as when applications do their own
caching. File I/O is done directly to/from user space
buffers. The I/O is synchronous, that is, at the completion of a
read(2) or write(2), data is guaranteed to have been
transferred. See NOTES below for further discussion.

Source: https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/5/html/global_file_system/s1-manage-direct-io
Linux manpage “man open”

20.11.2018
I/O request types
Blocking vs Non-blocking I/O

Synchronous blocking I/O

The user space application performs a system call that blocks the application until the system call is complete

Application kernel
Application blocked

system call

initiate I/O read

read()

response

data movement

20.11.2018
I/O request types
Blocking vs Non-blocking I/O

Synchronous non-blocking I/O

The user space application performs several system calls until the I/O request has been completed

Application kernel

system call

EAGAIN / WOULDBLOCK initiate I/O read

read()

system call
EAGAIN / WOULDBLOCK
response

system call
data movement

20.11.2018
I/O request types
Asynchronous I/O

Asynchronous blocking I/O

The user space application performs system calls that submit I/O requests and blocks the application until
notified about requests completion (callback)
Application kernel
read()

system call

EAGAIN / WOULDBLOCK initiate I/O read

Application blocked

select()

response
Select – data available

system call
read()

data movement

20.11.2018
I/O request types
Asynchronous I/O

Asynchronous non-blocking I/O

The user space application performs system calls that submit I/O requests and continues other processing
notified about requests completion trough callback or signal
Application kernel
AIO read()

system call

initiate I/O read

other processing

response
data movement
with signal or callback

20.11.2018
Oracle I/O management

1
Initialization Parameters
Filesystem vs Oracle ASM vs Oracle dNFS vs ODM
Quiz: ACID paradigm & database processes
2
System calls
3

20.11.2018
Oracle I/O management
Initialization Parameters

FILESYSTEMIO_OPTIONS = { none | setall | directIO | asynch }

Specifies I/O operations for filesystem files
Depends on the database release and operating system

DISK_ASYNCH_IO = { true | false }

Controls whether I/O to datafiles, control files, and logfiles is asynchronous
Oracle recommends that you leave this parameter set to its default value (TRUE)
if the operating system supports asynch. I/O
> increase I/O throughput or simulate asynchronous I/O using DB_WRITER_PROCESSES or use DBWR_IO_SLAVES

References
Init.ora Parameter "FILESYSTEMIO_OPTIONS" Reference Note (Doc ID 120697.1)
Things To Consider For setting filesystemio_options And disk_asynch_io (Doc ID 1987437.1)
File System's Buffer Cache versus Direct I/O (Doc ID 462072.1)

20.11.2018
Oracle I/O management
Filesystem vs Oracle ASM vs Oracle dNFS vs ODM

Filesystem (xfs, ext4, ACFS, VxFS)

I/O request type depends on parameters FILESYSTEMIO_OPTIONS and DISK_ASYNCH_IO
Depends as well on filesystem mount options
> Things To Consider For Setting filesystemio_options And disk_asynch_io (Doc ID 1987437.1)

Oracle Direct NFS

I/O request type does not depend on O.S support
Bypass FILESYSTEMIO_OPTIONS
Oracle dNFS always issues Asynch. and DirectIO unless misconfigured
> Always set FILESYSTEMIO_OPTIONS accordingly dbi
Initialization parameter 'filesystemio_options' Setting With Direct NFS (dNFS) (Doc ID 1479539.1)

20.11.2018
Oracle I/O management
Filesystem vs Oracle ASM vs Oracle dNFS vs ODM

Oracle Automatic Storage Management (ASM)

I/O request type does not depend on O.S support
Bypass FILESYSTEMIO_OPTIONS
ASM Inherently performs Asynchronous I/O Regardless of filesystemio_options Parameter (Doc ID 751463.1)
DISK_ASYNCH_IO parameter matters

Oracle Disk Manager (ODM)

Defined by Oracle to enhance file management and disk I/O performance
Bypass FILESYSTEMIO_OPTIONS
Native support of Asynch. and DirectIO unless misconfigured dbi
> Always set FILESYSTEMIO_OPTIONS accordingly
Things To Consider For Setting filesystemio_options And disk_asynch_io (Doc ID 1987437.1)

20.11.2018
Oracle I/O management
FILESYSTEMIO_OPTIONS

Controls whether asynchronous and/or direct I/O is attempted for Oracle files available
through a filesystem

Synchronous I/O Asynchronous I/O

Buffered I/O none asynch
Direct I/O directIO setall

Parameter bypassed for RAW devices, Oracle ASM, VxFS with ODM, Oracle dNFS
DB_WRITER_PROCESSES automatically increased implicitly based on CPU_COUNT

Things To Consider For Setting filesystemio_options And disk_asynch_io (Doc ID 1987437.1)

20.11.2018
Oracle I/O management
ACID paradigm & database processes: Quiz

ACID stands for ?

Atomicity: The entire sequence of actions must be either completed or aborted.
The transaction cannot be partially successful
Consistency: The transaction takes the resources from one consistent state to another
Isolation: A transaction's effect is not visible to other transactions until the transaction is committed
Durability: Changes made by the committed transaction are permanent and must survive system failure

Has FILESYSTEMIO_OPTIONS an impact on the durability property?

No! of course not!

Has FILESYSTEMIO_OPTIONS an impact on the database instance performance?

Yes of course, but it depends as well on the technology stack (ASM, dNFS, ODM)

20.11.2018
Oracle I/O management
System calls – big picture

“DB
ASYNC
“Normal
“Directwrite”
I/O
write”
write”
O.S Process Dataaio_write()
files and Redo Unix Kernel
logs ASYNC I/O
flag ACK threads
O_DSYNC
ACK
flag
O_DIRECT flag
Filesystem Cache O_DIRECT

ACK ACK

ACK

Datafiles Redo
Logs

20.11.2018
Oracle I/O management
System calls - filesystems

System call flags depending on FILESYSTEMIO_OPTIONS

User I/O (User process) System I/O (Background processes)
none open("…/swingbench001DB122.dbf", O_RDWR|O_DSYNC) open("…/redog1m1DB122.dbf", O_RDWR|O_DSYNC)
open("…/swingbench001DB122.dbf", O_RDWR) open("…/soe001DB122.dbf", O_RDWR|O_DSYNC)
pread(…) / pwrite(…) pread(…) / pwrite(…)

asynch open("…/swingbench001DB122.dbf", O_RDWR|O_DSYNC) open("…/redog1m1DB122.dbf", O_RDWR|O_DSYNC)

open("…/swingbench001DB122.dbf", O_RDWR) open("…/soe001DB122.dbf", O_RDWR|O_DSYNC)

io_submit(…) / io_getevents(…) io_submit(…) / io_getevents(…)

directio open("…/soe001DB122.dbf", O_RDWR|O_DSYNC|O_DIRECT) open("…/redog1m1DB122.dbf", O_RDWR|O_DSYNC|O_DIRECT)

open("…/soe001DB122.dbf", O_RDWR|O_DSYNC|O_DIRECT)
pread(…) / pwrite(…) pread(…) / pwrite(…)

setall open("…/soe001DB122.dbf", O_RDWR|O_DSYNC|O_DIRECT) open("…/redog1m1DB122.dbf", O_RDWR|O_DSYNC|O_DIRECT)

open("…/soe001DB122.dbf", O_RDWR|O_DSYNC|O_DIRECT)
io_submit(…) / io_getevents(…) io_submit(…) / io_getevents(…)

Oracle ASM uses GNU C Library on Linux

System calls io_submit & io_getevents (except if DISK_ASYNCH_IO = false)
20.11.2018
Oracle I/O management
System calls - miscellaneous

Oracle Disk Manager uses ioctl() system call to manipulate files

FILESYSTEMIO_OPTIONS bypassed
Bypass filesystem buffer cache
Implies Asynch and Direct I/O
ioctl(15, _IOC(_IOC_WRITE, 0x4f, 0x1c, 0x1658), 0x7ffd898babc0) = 0
poll([{fd=15, events=POLLIN|POLLRDNORM}], 1, 0) = 0 (Timeout)$

Oracle dNFS implies the same properties as ODM over an Oracle optimized Remote
Procedure Call network protocol
> NFSv3, NFSv4, NFSv4.1 are protocols supported starting with Oracle 12cR1
connect(32, {…,sin_port=htons(2049),sin_addr=inet_addr(”192.168.101.111")},
sendmsg(32, {msg_name(0)=NULL, msg_iov(1)=[{“1"..., 152}],
poll([{fd=32, events=POLLIN}], 1, 500) = 1 ([{fd=32, revents=POLLIN}])
recvmsg(32, {msg_name(0)=NULL, msg_iov(1)=[{

20.11.2018
Troubleshooting

1
Case studies

20.11.2018
Troubleshooting
Oracle dNFS (directIO + Asynch. I/O)

Customer’s infrastructure
NetApp full flash storage 6x Veritas active/active cluster (single instance
Dedicated storage LAN (FCoE) databases)
RHEL6 x86_64 200 Oracle database instances
NIC all part of bonds Oracle dNFS
> mode active/passive

# top
top - 15:52:18 up 58 days, 19:01, 11 users, load average: 3.66, 3.58, 3.60
Tasks: 2428 total, 3 running, 2425 sleeping, 0 stopped, 0 zombie
%Cpu(s): 1.6 us, 4.2 sy, 0.0 ni, 91.2 id, 0.0 wa, 0.0 hi, 2.9 si, 0.0 st
KiB Mem : 39582355+total, 15929764+free, 21915532+used, 17370568 buff/cache
KiB Swap: 4194300 total, 4194300 free, 0 used. 17346779+avail Mem
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
83 root 20 0 0 0 0 R 70.8 0.0 87:03.47 [ksoftirqd/2]
3254 oracle 20 0 28.364g 36524 29824 S 7.2 0.0 0:59.17 oracleDB12 (LOCAL=NO)
20381 oracle 20 0 20.354g 36744 27888 S 3.6 0.0 2:02.19 oracleDB12 (LOCAL=NO)
....

20.11.2018
Troubleshooting
Oracle dNFS (directIO + Asynch. I/O)

On which device do we face interrupts which consumes almost 1 core ?

# head -n1 /proc/interrupts && cat /proc/interrupts | grep eno52
CPU0 CPU1 CPU2 CPU3 CPU4 …
78: 42 20 2002696841 0 0 IR-PCI-MSI-edge eno52-q0

What is the impact for the database ?

DBA_HIST_SYSTEM_EVENT (User I/O wait class, ie.: db file sequential read)
BEGIN_TIME AVG_MS
-------------------------- ----------
01-SEP-2016 11:00 1.542
01-SEP-2016 12:10 .538
01-SEP-2016 13:00 3.7
01-SEP-2016 13:46 23.302
01-SEP-2016 15:00 28.519

Root cause: Network bandwidth saturation! All servers had the same NIC interface active
within the bond used for FCoE (aka. All traffic went through the same FABRIC port)

20.11.2018
Troubleshooting
I/O Scheduler

Customer’s infrastructure
IBM XIV storage system
FC network
SLES 11 x86_64
EXT3 filesystem
> Huge filesystem buffer cache (> 4 TB)
FILESYSTEMIO_OPTIONS = none
ALL databases protected through Oracle Data Guard physical standby

SLES 11 default I/O Scheduler “cfq”

Completely Fair Queuing
The goal of this I/O scheduler is to provide a fair I/O priority to each process.

20.11.2018
Troubleshooting
I/O Scheduler
# Wait event : log file parallel write

BEGIN_TIME AVG_MS BEGIN_TIME AVG_MS

----------------- ---------- ----------------- ----------
10-FEB-2016 09:00 18.968 11-FEB-2016 09:00 1.48
10-FEB-2016 10:01 17.574 11-FEB-2016 10:00 1.186
10-FEB-2016 11:00 8.238 11-FEB-2016 11:00 1.04
10-FEB-2016 12:00 8.14 11-FEB-2016 12:00 1.07
10-FEB-2016 13:00 7.718 11-FEB-2016 13:00 1.244
10-FEB-2016 14:00 19.881 11-FEB-2016 14:00 1.076
10-FEB-2016 15:00 13.992 11-FEB-2016 15:00 1.352
10-FEB-2016 16:00 10.036 11-FEB-2016 16:00 1.085
10-FEB-2016 17:00 6.268 11-FEB-2016 17:00 .901
10-FEB-2016 18:00 4.443 11-FEB-2016 18:00 1.063
10-FEB-2016 19:00 1.758 11-FEB-2016 19:00 1.885

Very poor I/O throughput I/O scheduler switched to deadline

LGWR response time on COMMIT from cfq
FILESYSTEMIO_OPTIONS = NONE luckily !
> Asynchronous I/O might be a good compromise (buffered I/O)

20.11.2018
Troubleshooting
FILESYSTEMIO_OPTIONS = NONE to Oracle ODM

Proceed step by step

From none to
Asynch. I/O
Increased SGA
From Asynch. I/O
combined with
Direct I/O

New CPUs
2800Mhz /
4 cores
2100Mhz /
16 cores

20.11.2018
Core Message

20.11.2018
Core Message

I/O management should still be seriously taken into account

https://www.ibm.com/developerworks/library/l-async/
What is the Block Size Limit in XFS Filesystem on Oracle Linux 7 (Doc ID 2101949.1)
> Supported and Recommended File Systems on Linux (Doc ID 236826.1)
Operating System (I/O Scheduler)
> Flash, SSD who switched to NOOP ?

Database optimizations Application optimizations

Initialization parameters https://use-the-index-luke.com/de
> 12cR1 LGWR slaves (_use_single_log_writer) Never ending story
Gather system statistics for the CBO
Oracle 12.1.0.2 NF Automatic Big Table Caching
Oracle 11.2.0.1 Database Smart Flash Cache
Direct path reads optimization “_small_table_threshold”
Patch 18498878: SMALL TABLE DOES NOT CACHE PROPERLY
AFTER UPGRADE TO 12.1
20.11.2018
Let’s meet at booth 242