IBM Global Services

Oracle on AIX : Best Practices

Dr. Andreas Gruber

mySAP Technology Services

November 2003 © 2003 IBM Corporation

 IBM Global Services

Agenda

1 Introduction

2 Design of Disk Layout for SAP/Oracle Databases

3 Performance Tuning for the Journaled Filesystem

4 AIX 5L and the Enhanced Journaled Filesystem (JFS2)

5 Multiple SAP Databases on a Single System

2 Oracle on AIX : Best Practices © 2003 IBM Corporation

 IBM Global Services

Agenda

1 Introduction

2 Design of Disk Layout for SAP/Oracle Databases

3 Performance Tuning for the Journaled Filesystem

4 AIX 5L and the Enhanced Journaled Filesystem (JFS2)

5 Multiple SAP Databases on a Single System

3 Oracle on AIX : Best Practices © 2003 IBM Corporation

 IBM Global Services

Introduction

"A best practice is a technique or methodology that, through

experience and research, has proven to reliably lead to a desired
result." http://searchVB.com

§ Knowledge transfer from SAP infrastructure projects based on

• IBM pSeries, Enterprise Storage Server, EMC Symmetrix,
• Storage Area Networks
• Tivoli Storage Manager and Data Protection for SAP

§ Performance tuning as a holistic approach that involves all

components of a system

§ Explanation and advertising of new features implemented by

hardware and software components

4 Oracle on AIX : Best Practices © 2003 IBM Corporation

 IBM Global Services

Introduction > Oracle Performance Tuning

Oracle Performance Tuning

List of Priorities
§ Business Rules
§ Data and Application Design
§ Logical Structure of the Database
§ Database Operations and Access Paths
§ Memory Allocation AIX / Hardware
§ I/O and Physical Structure AIX / Hardware
§ Resource Contention
§ Underlying Platform(s) AIX / Hardware

5 Oracle on AIX : Best Practices © 2003 IBM Corporation

 IBM Global Services

Agenda

1 Introduction

2 Design of Disk Layout for SAP/Oracle Databases

3 Performance Tuning for the Journaled Filesystem

4 AIX 5L and the Enhanced Journaled Filesystem (JFS2)

5 Multiple SAP Databases on a Single System

6 Oracle on AIX : Best Practices © 2003 IBM Corporation

 IBM Global Services

Disk Layout > Old Style

Old Style Disk Layout for SAP/Oracle Database

§ Use of single and separated disks

• SCSI-Attachment
• SSA-Loops

§ Placement of files according to the following criteria

• Type of Files (Redolog, Archives, Datafiles)
• Data / Index Tablespaces
• Master Data / Transaction Data

§ Elimination of arising hot spots via reorganization of tables into

separate tablespaces

§ Problem
• Wasting of disk resources (e.g. Redologs)
• Complex and difficult administration of scattered data files

7 Oracle on AIX : Best Practices © 2003 IBM Corporation

 IBM Global Services

Disk Layout > New Paradigm

New Paradigm for SAP/Oracle Database

§ Intelligent Disksubsystems
• Internal Read/Write Caches (0,5 - 64 GByte)
• Internal RAID-5 or RAID-10 (or similar) implementations
• SCSI or Fibre Channel-Attachment

§ Oracle's S.A.M.E Strategy (Stripe And Mirror Everything)

EMC Symmetrix / DMX IBM Enterprise Storage Server

Striping § Stripe almost everything § Stripe across ESS RAID Arrays

§ Don't subset by individual disks § Physical Partition striping (16/32 MB)
§ Hardware striping for Log Files
§ Use 1 MB stripe size
Mirroring § Mirror for data protection § Use RAID -5 or RAID -10

Placement Policy § Use "Outside Edge" of disks § Use "Middle" for ESS arrays
to minimize seek

8 Oracle on AIX : Best Practices © 2003 IBM Corporation

 IBM Global Services

Disk Layout > Example

Disk Layout Example for SAP/Oracle Database

9 Oracle on AIX : Best Practices © 2003 IBM Corporation

 IBM Global Services

Disk Layout > Golden Rules for ESS (Shark)

Golden Rules for ESS (Shark)

§ Number of Disksubsystems
• Should be based on throughput needs
• Should not be based on capacity needs (Terabytes)

§ Decision Support Systems (e.g. BW)

• 100-200 GByte Data per 2GBit Fibre Channel Port
• Moderate Cache Requirements

§ OLTP Systems
• 200-300 GB Data per 2GBit Fibre Channel Port
• Large Cache Requirements

§ Number of RAID Arrays according to throughput requirements

• 50 MB/s sustained sequential Read/Write Performance

10 Oracle on AIX : Best Practices © 2003 IBM Corporation

 IBM Global Services

Agenda

1 Introduction

2 Design of Disk Layout for SAP/Oracle Databases

3 Performance Tuning for the Journaled Filesystem

4 AIX 5L and the Enhanced Journaled Filesystem (JFS2)

5 Multiple SAP Databases on a Single System

11 Oracle on AIX : Best Practices © 2003 IBM Corporation

 IBM Global Services

Performance Tuning for JFS > JFS vs. Raw Devices

Journaled Filesystem vs. Raw Devices

§ "Raw Devices" correspond to Logical Volumes in AIX

• Disk devices (/dev/rhdisk10) are not allowed to be accessed directly

§ Filesystems are easier to manage for most administrators

§ Journaled Filesystems ensure the integrity of the filesystem

structure and data
• Locking of filesystem structures prevents simultaneous access

§ JFS access to data always uses the AIX Virtual Memory Manager
and the real memory as a buffer for files
• Improves performance when data is accessed multiple times
• Oracle data blocks are buffered twice and consume memory

12 Oracle on AIX : Best Practices © 2003 IBM Corporation

 IBM Global Services

Performance Tuning for JFS > Areas of Tuning

Areas of Tuning

ç Oracle Parameters init.ora

ç Asynchronous I/O

ç Read-Ahead

ç File Caching

ç Database Layout

ç Multipath and SAN

ç Cache and internal access

ç Number and speed of drives

13 Oracle on AIX : Best Practices © 2003 IBM Corporation

 IBM Global Services

Performance Tuning for JFS > File Caching

File Caching

§ Memory is categorised into two different types

• Computational memory: Consists of the pages that belong to
working-storage or executable files.
• File memory: Consists of the remaining pages. These are usually
pages from permanent data files in persistent storage.

§ AIX is using up all memory until only "minfree" pages are left
• The percentage of memory that is used for file cache (numperm) can
be seen with the command
/usr/samples/kernel/vmtune (AIX 4.3 + 5.1)
/usr/bin/vmstat -v (AIX 5.2)
• minperm If numperm falls below this level, the page-replacement
algorithm steals both file and computational pages, regardless of
repage rates.
• maxperm If numperm rises above this level, the page-replacement
algorithm steals only file pages.

14 Oracle on AIX : Best Practices © 2003 IBM Corporation

 IBM Global Services

Performance Tuning for JFS > VMM Tuning

Virtual Memory Manager (VMM) Tuning

§ Tuning of the file cache will only improve performance if the system
was memory constrained before
§ Tuning is meant to avoid duplicate buffering of Oracle data blocks
which leads to a waste of physical memory
§ If the SAP R/3 database server has more than 2 GB memory and is
only used for SAP services
• minperm should be set equivalent to ~ 150 MB
• maxperm should be set equivalent to ~ 300 MB
• In case of 3 GB memory this corresponds to
/usr/samples/kernel/vmtune -p 5 -P 10

§ The number of pages on the freelist

• minfree = 120 * Number of CPUs

15 Oracle on AIX : Best Practices © 2003 IBM Corporation

 IBM Global Services

Performance Tuning for JFS > Tuning Read-Ahead

Tuning Read-Ahead

§ VMM tries to anticipate the future need for pages of a sequential file
by observing the pattern a program uses to access the file.
• minpgahead
The number of pages read ahead when the VMM first detects the
sequential access pattern. If the program continues to access the file
sequentially, the next read ahead will be doubled.
• maxpgahead
The maximum number of pages the VMM will read ahead in a
sequential file.

§ Recommended values for SAP database servers

• minpgahead 2
maxpgahead 8 * #CPUs rounded to the next power of 2
maxfree minfree + maxpgahead
• In case of a 6 CPU System this corresponds to ( < AIX 5.2)
/usr/samples/kernel/vmtune -r 2 -R 64 -f 720 -F 784

16 Oracle on AIX : Best Practices © 2003 IBM Corporation

 IBM Global Services

Performance Tuning for JFS > Read-Ahead for backup

Read-Ahead for backup with TSM and TDP

§ Backup of Oracle databases with TSM and TDP heavily rely on

read-ahead
• Performance Improvements of 100% have been observed

§ Journaled Filesystem
• maxpgahead = 256

§ Enhanced Journal Filesystem (JFS2)

• j2_maxPageReadAhead = 256

§ High values of read-ahead might have negative effects on cache

efficiency of disk subsystems for SAP production
• Switch to these parameters only during backup

17 Oracle on AIX : Best Practices © 2003 IBM Corporation

 IBM Global Services

Performance Tuning for JFS > Asynchronous I/O

Asynchronous I/O
§ Asynchronous I/O is an AIX operating system feature consisting of
• Request Queue for I/O write requests
• Kernel processes (aioservers) that take requests from the queue
§ Programs must use the corresponding application programming
interface in order to use asynchronous I/O
• no wait for the completion of a write request
• no blocking of application execution
§ Parameters for tuning asynchronous I/O
• minservers The minimum number of aioservers that are started for
asynchronous disk I/O. The default value is 1.
• maxservers The maximum number of aioservers that are started for
asynchronous disk I/O. The default value is 10. Since each aioserver
uses memory, this number should not be much larger than the
expected amount of simultaneous asynchronous disk I/O requests.
• maxreqs Maximum number of asynchronous disk I/O requests that
can be stored in the queue. The default value is 4096.

18 Oracle on AIX : Best Practices © 2003 IBM Corporation

 IBM Global Services

Performance Tuning for JFS > AIO Tuning Recommendation

AIO Tuning Recommendations

§ Asynchronous I/O is treated like a device and can be managed via
smit aio
• autoconfig should be switched on, to make it "available" after reboot
§ Recommended starting values are
• minserver 2 or Number of CPUs -1, whatever is larger
• maxserver Two times the number of datafiles
• maxreqs 12288
§ The aioserver processes are started as oracle child processes
• for large SAP systems this number may become very large and the
maximum number of processes per user has to be adopted
chdev -l sys0 -a maxuproc=2000

§ The number of active aioservers can best be monitored with the

performance tool "nmon"
• http://www-106.ibm.com/developerworks/eserver/articles/analyze_aix/

19 Oracle on AIX : Best Practices © 2003 IBM Corporation

 IBM Global Services

Agenda

1 Introduction

2 Design of Disk Layout for SAP/Oracle Databases

3 Performance Tuning for the Journaled Filesystem

4 AIX 5L and the Enhanced Journaled Filesystem (JFS2)

5 Multiple SAP Databases on a Single System

20 Oracle on AIX : Best Practices © 2003 IBM Corporation

 IBM Global Services

AIX 5L and JFS2 > New features

New features with AIX 5L

§ Support for POWER4 processors and LPAR technology
§ Workload Management for CPU, Memory and I/O
§ Alternatively 32-Bit or 64-Bit AIX Kernel
• 64-Bit Kernel recommended for JFS2
§ Enhanced Journal Filesystem (JFS2)
• Max. Filesize 16TB (vs. 64 GB)
• Max. Filesystemsize 16 TB (vs. 1 TB)
• Dynamic number of i-nodes
• B-Tree directory structure
• Higher performance for creation and deletion of files
§ AIX 5.2 implements new commands for performance tuning
• Administration via smit
• Reboot safe

21 Oracle on AIX : Best Practices © 2003 IBM Corporation

 IBM Global Services

AIX 5L and JFS2 > Direct I/O

Direct I/O for JFS2 with AIX 5L

§ Reads under cached I/O - cache miss § Reads with direct I/O

22 Oracle on AIX : Best Practices © 2003 IBM Corporation

 IBM Global Services

AIX 5L and JFS2 > Implications of direct I/O

Implications and restrictions of direct I/O

§ In order to activate direct I/O, the filesystem must be mounted with
the option "mount -o dio"

§ Read-Ahead is disabled for these filesystem

§ The alignment of the read requests must be on 4K boundaries

• In AIX 5.2 ML01 this alignment must be according to "agblksize",
a parameter which is specified during file system creation
• If the alignment criteria is not met, normal cached access takes place
but the cache is discarded immediately
• Oracle Online-Redologs should be located in separate filesystems with
an allocation group blocksize (agblksize) of 512 Byte

§ Applications that see performance benefits when using raw logical

volumes for storage are likely to benefit from the use of Direct I/O
• e.g. Oracle

23 Oracle on AIX : Best Practices © 2003 IBM Corporation

 IBM Global Services

AIX 5L and JFS2 > Locking and serialization

Locking and serialization

§ Filesystem Inodes contain meta information about files or directories

• if a file is changed, also the inode is changed
• if the inode is changed the file might be unchanged
§ Inode locks prevent concurrent modifications of files that lead to
inconsistencies of data
• JFS2 uses a read-shared write-exclusive lock
• Read and write serialization granularity is only at file level
§ Database applications are more sophisticated and have a finer
granularity concerning locking and serializaton of access to data
• Table locking
• Row-level locking

24 Oracle on AIX : Best Practices © 2003 IBM Corporation

 IBM Global Services

AIX 5L and JFS2 > Concurrent I/O

Concurrent I/O

§ AIX 5.2 ML01 provides JFS2 with the concurrent I/O feature
• Multiple threads can simultaneously perform read and write operations
on a shared file
• The option can either be used to open files or a whole filesystem can
be mounted with
mount -o cio

§ If only a subset of files in a filesystem should be opened in

concurrent mode, the subdirectory can be mounted as namefs
• mount –v namefs –o cio /somefs/subsomefs /somefs

§ Direct I/O is implicitly used with concurrent I/O

§ Extension or truncation of files results in a change of the inode lock
from shared to exclusive during the time of modification

25 Oracle on AIX : Best Practices © 2003 IBM Corporation

 IBM Global Services

AIX 5L and JFS2 > Benchmark results

Benchmark results

26 Oracle on AIX : Best Practices © 2003 IBM Corporation

 IBM Global Services

AIX 5L and JFS2 > Tuning Parameters for JFS2

Tuning Parameters for JFS2

§ New tuning commands with AIX 5.2 replace vmtune and schedtune
• vmo, ioo, schedo

§ Values are preserved after reboot, if pre AIX 5.2 compatibility mode
is switched off
• chdev -l sys0 -a pre520tune=disable

§ JFS2 file cache does not use the minperm/maxperm values

• maxclient Hard limit of file pages for JFS2 and NFS and must be set
smaller than or equal to maxperm

§ Read-Ahead parameters must be set separately for JFS2 via ioo

• j2_minPageReadAhead
• j2_maxPageReadAhead

27 Oracle on AIX : Best Practices © 2003 IBM Corporation

 IBM Global Services

Agenda

1 Introduction

2 Design of Disk Layout for SAP/Oracle Databases

3 Performance Tuning for the Journaled Filesystem

4 AIX 5L and the Enhanced Journaled Filesystem (JFS2)

5 Multiple SAP Databases on a Single System

28 Oracle on AIX : Best Practices © 2003 IBM Corporation

 IBM Global Services

Multiple SAP Databases > Motivation

Motivation for consolidation of databases

§ Servers have become more and more powerful

• Provide more performance than can be used by one database
• Less operating system images to maintain
• Number of SAP systems and databases is exploding, budgets are
shrinking

§ Workload Management enables balancing of resources

• Better usage of computing resources, including I/O and network
• Peak usage can be satisfied

§ Dynamic Logical Partitioning enables on-demand resources

• CPUs and memory can be moved around freely
• Storage technology enables flexible disk assignment
• Hot-swapping of adapters

29 Oracle on AIX : Best Practices © 2003 IBM Corporation

 IBM Global Services

Multiple SAP Databases > Guidelines and hints

Guidelines and hints for Oracle

§ Naming and numbering conventions
• Enforce User-ID and Group-ID standards
• Use naming conventions for Networks, Volume Groups and Volumes

§ Oracle Installation Information

• Work with symbolic links for each SID for the installation directories
/oracle/jre, /oracle/oui, /oracle/inventory
(SAP OSS Note 350251)

§ AIX kernel and Oracle kernel extensions

• Oracle 8 is only supported as 32-Bit version on AIX 5L
• Oracle 8 kernel extensions only work with the 32-Bit AIX 5L kernel
• Oracle 8 Patch 2896876 (Metalink note 231901.1) provides extension
for 64-Bit AIX 5L Kernel

§ Oracle listeners
• Separate listeners should be configured for each instance with unique
port numbers

30 Oracle on AIX : Best Practices © 2003 IBM Corporation

 IBM Global Services

Multiple SAP Databases > Guidelines and hints SAP

Guidelines and hints for SAP

§ Transport Directory
• Each system environment should have a separate transport directory, e.g.
/usr/sap/trans<SID>

• The following variables should be set in the SAP system

DIR_TRANS=/usr/sap/trans<SID>
DIR_EPS_ROOT=/usr/sap/trans<SID>/EPS

§ Performance Collector saposcol

• Start the saposcol from an SID independent path such as /etc/sap/saposcol
• Create an entry in the inittab file
mkitab –i strload saposcol:2:once:su – sidadm –c /etc/sap/saposcol –l

31 Oracle on AIX : Best Practices © 2003 IBM Corporation

 IBM Global Services

Multiple SAP Databases > Workload Management

Workload Management

§ Workload Management included in AIX at no additional costs

• Easy setup for SAP and Oracle (30 Minutes)
• Passive mode for testing, performance monitoring and accounting

§ Management of CPU Resources

• Management of memory easier with SAP and Oracle parameters

§ Soft limits and hard limits possible

• No wasting of resources

§ Classification via User-IDs or Tags

• orasid and sidadm are perfectly suited for easy classification
• Separation of classification of Dialog/Background workprocesses through
direct assignment

32 Oracle on AIX : Best Practices © 2003 IBM Corporation

 IBM Global Services

Multiple SAP Databases > Workload Management Example

Workload Management Example

Superclasses for 4 SAP systems
Subclasses SAP, Oracle DB,
I1A,IPC, KPC and IPB
Live Cache for APO system I1A

33 Oracle on AIX : Best Practices © 2003 IBM Corporation

 IBM Global Services

Summary

§ Modern disk subsystems facilitate database layouts through large

caches and internal striping and mirroring

§ Correct configuration of file cache in AIX prevents waste of memory

§ Sufficient asynchronous I/O server processes are key to good

performance

§ JFS2 offers almost raw device performance with concurrent I/O

§ Workload Manager in passive mode ideal for monitoring

34 Oracle on AIX : Best Practices © 2003 IBM Corporation

 IBM Global Services

Questions ... ?

November 2003 © 2003 IBM Corporation

 IBM Global Services

Agenda

1 Introduction

2 Design of Disk Layout for SAP/Oracle Databases

3 Performance Tuning for the Journaled Filesystem

4 AIX 5L and the Enhanced Journaled Filesystem (JFS2)

5 Multiple SAP Databases on a Single System

6 References

36 Oracle on AIX : Best Practices © 2003 IBM Corporation

 IBM Global Services

References

§ IBM Redbooks and Redpapers at www.redbook.ibm.com

• "A holistic approach to a reliable infrastructure for SAP R/3", SG24-5050
• "Consolidating multiple SAP systems on one pSeries", REDP-3626-00

§ SAP Market Place at service.sap.com

• "Database Layout for R/3 installation under Oracle"

§ Oracle Whitepapers at technet.oracle.com

• "Optimal Storage Configuration Made Easy"

§ Enterprise Storage Server white papers for Oracle and SAP

• http://www.storage.ibm.com/disk/ess/whitepaper.htm

§ AIX Whitepapers, e.g. on Concurrent I/O

• http://www-1.ibm.com/servers/eserver/pseries/library/wp_aix_lit.html

37 Oracle on AIX : Best Practices © 2003 IBM Corporation