T EC HN IC AL R E P OR T

Nimble Storage Setup Guide for Single

Instance Oracle 11gR2 on Oracle Linux 6.4

NIMBLE TECHNICAL REPORT: NIMBLE STORAGE FOR ORACLE 1

Document Revision
Table 1.
1

Date Revision Description

1/8/2013 1.0 Initial Draft

1/11/2013 1.1 Revised

2/25/2014 1.2 Revised

7/31/2014 1.3 Revised iSCSI Timeout

11/17/2014 1.4 Updated iSCSI & Multipath

THIS TECHNICAL TIP IS FOR INFORMATIONAL PURPOSES ONLY, AND MAY CONTAIN
TYPOGRAPHICAL ERRORS AND TECHNICAL INACCUURACIES. THE CONTENT IS PROVIDED AS IS,
WITHOUT EXPRESS OR IMPLIED WARRANTIES OF ANY KIND.

Nimble Storage: All rights reserved. Reproduction of this material in any manner whatsoever
without the express written permission of Nimble is strictly prohibited.

NIMBLE TECHNICAL REPORT: NIMBLE STORAGE FOR ORACLE 2

Table of Contents

Introduction ................................................................................................................................................................................. 4

Audience ...................................................................................................................................................................................... 4

Server Setup for Oracle ............................................................................................................................................................. 4

Oracle Linux 6.4 Installation ............................................................................................................ 4

Operating System configuration for Oracle ...................................................................................... 5
Oracle Linux iSCSI Setup ................................................................................................................ 6
Multipath Setup ................................................................................................................................ 6

Nimble Storage Setup ............................................................................................................................................................... 7

Create New Initiator Group .............................................................................................................. 8

Create Volumes and Allow Access .................................................................................................. 9
Protection using Nimble Volume Collection .................................................................................. 14

Oracle 11gR2 Setup ................................................................................................................................................................17

Pre-requisites .................................................................................................................................. 17
Oracle ASM Disks Setup ................................................................................................................ 17
Install Grid Infrastructure ............................................................................................................... 19
Install Oracle Software ................................................................................................................... 19

NIMBLE TECHNICAL REPORT: NIMBLE STORAGE FOR ORACLE 3

Introduction
The purpose of this technical white paper is to describe the basic setup of Oracle Linux 6.4 and Oracle 11g Release 2
with Automatic Storage Management (ASM) when deploying on Nimble Storage.

Oracle performance tuning is beyond the scope of this paper. Please visit www.oracle.com for Oracle Performance
Tuning Guide for more information in tuning your database.

Audience
This guide is intended for Oracle database solution architects, storage engineers, system administrators and IT
managers who analyze, design and maintain a robust database environment on Nimble Storage. It is assumed that the
reader has a working knowledge of network design for iSCSI SAN, and basic Nimble Storage operations. Knowledge of
Oracle Linux operating system, Oracle Clusterware, and Oracle database is also required.

Server Setup for Oracle

Configuration for a single instance Oracle database.

Oracle Linux 6.4 Installation

During the installation of the operating system, we recommend that Logical Volume Manager (LVM) not to be
used for the boot disk. Remove all LVM configurations and use the standard partition method for the boot disk.

• /boot partition
• swap partition
• / partition

[root@oraclelinux1 ~]# df
Filesystem 1K-blocks Used Available Use% Mounted on
/dev/sda3 32998256 22234944 9087076 71% /
tmpfs 6291456 2607808 3683648 42% /dev/shm
/dev/sda1 126931 50862 69516 43% /boot

After the installation of the operating system, the following daemons need to be disabled.
o Disable selinux
o Disable NetworkManager
o Disable iptables
o Disable ip6tables

If not installed already, the following packages are required for Oracle 11gR2 and multipathing,

• compat-db42-4.2.52-15.el6.x86_64.rpm
• compat-db43-4.3.29-15.el6.x86_64.rpm
• compat-db-4.6.21-15.el6.x86_64.rpm
• cvuqdisk-1.0.9-1.rpm
• device-mapper-multipath-0.4.9-64.0.1.el6.x86_64.rpm
• device-mapper-multipath-libs-0.4.9-64.0.1.el6.x86_64.rpm

NIMBLE TECHNICAL REPORT: NIMBLE STORAGE FOR ORACLE 4

• ksh-20100621-19.el6.x86_64.rpm
• libaio-devel-0.3.107-10.el6.x86_64.rpm
• libsysfs-2.1.0-7.el6.x86_64.rpm
• libXp-1.0.0-15.1.el6.x86_64.rpm
• libXp-devel-1.0.0-15.1.el6.x86_64.rpm
• oracleasm-support-2.1.8-1.el6.x86_64.rpm
• sysfsutils-2.1.0-7.el6.x86_64.rpm
• sysstat-9.0.4-20.el6.x86_64.rpm
• unixODBC-2.2.14-12.el6_3.x86_64.rpm
• unixODBC-devel-2.2.14-12.el6_3.x86_64.rpm

Operating System configuration for Oracle

• Add the below settings to the /etc/systctl.conf file. Run the ‘sysctl –p’ command when done.

# For 11g, recommended value for file-max is 6815744

fs.file-max = 6815744
# For 10g, uncomment 'fs.file-max 327679', comment other entries for this parameter and re-
run sysctl -p
# fs.file-max:327679
kernel.msgmni = 2878
kernel.sem = 250 32000 100 142
kernel.shmmni = 4096
net.core.rmem_default = 262144
# For 11g, recommended value for net.core.rmem_max is 4194304
net.core.rmem_max = 4194304
# For 10g, uncomment 'net.core.rmem_max 2097152', comment other entries for this
parameter and re-run sysctl -p
# net.core.rmem_max=2097152
net.core.wmem_default = 262144
# For 11g, recommended value for wmem_max is 1048576
net.core.wmem_max = 1048576
# For 10g, uncomment 'net.core.wmem_max 262144', comment other entries for this
parameter and re-run sysctl -p
# net.core.wmem_max:262144
fs.aio-max-nr = 3145728
# For 11g, recommended value for ip_local_port_range is 9000 65500
net.ipv4.ip_local_port_range = 9000 65500
# For 10g, uncomment 'net.ipv4.ip_local_port_range 1024 65000', comment other entries for
this parameter and re-run sysctl -p
# net.ipv4.ip_local_port_range:1024 65000
# Added min_free_kbytes 50MB to avoid OOM killer on EL4/EL5
vm.min_free_kbytes = 51200

# Nimble Recommended
net.core.wmem_max = 16780000
net.core.rmem_max = 16780000
net.ipv4.tcp_rmem = 10240 87380 16780000
net.ipv4.tcp_wmem = 10240 87380 16780000

• Add the settings below to the /etc/security/limits.conf file.

grid soft nofile 131072

grid hard nofile 131072
grid soft nproc 131072

NIMBLE TECHNICAL REPORT: NIMBLE STORAGE FOR ORACLE 5

grid hard nproc 131072
oracle soft nofile 131072
oracle hard nofile 131072
oracle soft nproc 131072
oracle hard nproc 131072
oracle soft core unlimited
oracle hard core unlimited
oracle soft memlock 50000000
oracle hard memlock 50000000

• Add the code below to the /etc/profile file.

if [ $USER = "oracle" ] || [ $USER = "grid" ] ; then

if [ $SHELL = "/bin/ksh" ]; then
ulimit -p 16384
ulimit -n 65536
else
ulimit -u 16384 -n 65536
fi
fi

Oracle Linux iSCSI Setup

• Leverage the native iSCSI initiator for Oracle. To do this, ensure the software iSCSI initiator shown
below has been installed.

iscsi-initiator-utils-6.2.0.872-41.0.1.el6.x86_64

• Configure iSCSI as follow:

• Edit the /etc/iscsi/iscsid.conf and modify the following parameters:
o node.session.timeo.replacement_timeout = 120
o node.conn[0].timeo.noop_out_interval = 5
o node.conn[0].timeo.noop_out_timeout = 10
o node.session.nr_sessions = 4
o node.session.cmds_max = 2048
o node.session.queue_depth = 1024
• Make sure the iSCSI NICs (eth1, eth2) have been assigned iSCSI network IP addresses and NET
MASK. Should look something similar to this for both iSCSI NICs.

DEVICE="eth1"
BOOTPROTO=static
HWADDR="00:50:56:AE:30:71"
NM_CONTROLLED="no"
ONBOOT="yes"
TYPE="Ethernet"
UUID="4705a3e9-671a-45a0-8477-8b8b98d56c09"
IPADDR=172.18.128.71
NETMASK=255.255.255.0
MTU=9000

Multipath Setup
• Change directory to /etc and using your favorite editor to create a new file called ‘multipath.conf’
with the code below

NIMBLE TECHNICAL REPORT: NIMBLE STORAGE FOR ORACLE 6

 defaults {
user_friendly_names yes
find_multipaths yes
}
devices {
device {
vendor "Nimble"
product "Server"
path_selector "round-robin 0"
path_checker tur
rr_min_io_rq 10
rr_weight priorities
failback immediate
path_grouping_policy group_by_serial
features "1 queue_if_no_path"
}
}

• Start multipathing by running the following:

o [root@oraclelinux1 ~]# chkconfig multipathd on
o [root@oraclelinux1 ~]# modprobe dm_multipath
o [root@oraclelinux1 ~]# service multipathd start

Nimble Storage Setup

Creating iSCSI volumes for the Oracle server on Nimble Storage is simple.
• First, create a new Initiator Group. This Initiator Group contains the Oracle server’s iqn ID.
• Second, create new volumes for the database and allow access to the Initiator Group.

Storage volumes for running Oracle database on an Oracle Linux server

NIMBLE TECHNICAL REPORT: NIMBLE STORAGE FOR ORACLE 7

Create New Initiator Group
To create a new initiator group
• Open the Nimble Management GUI
• Click on “Manage
Manage”
Manage and select “Initiator
Initiator Groups”
Groups

• Click on “New
New Initiator Group”
Group button and assign a name to the Initiator Group. We recommend using
the Oracle server hostname for distinction.

NIMBLE TECHNICAL REPORT: NIMBLE STORAGE FOR ORACLE 8

 • Click on “Add
Add Initiator”
Initiator button. Another window will appear. Enter the Initiator Group Name (oraclelinux1
in the example below) and the Initiator Name (iqn ID) and click “OK
OK”.
OK You can find the iqn ID of the
Oracle server by running the following command in the Oracle Linux server:

[root@oraclelinux1 ~]# cat /etc/iscsi/initiatorname.iscsi

InitiatorName=iqn.1988-12.com.oracle:496c4aead3e1

Create Volumes and Allow Access

• Click on “Manage
Manage”
Manage and select “Volumes
Volumes”
Volumes

NIMBLE TECHNICAL REPORT: NIMBLE STORAGE FOR ORACLE 9

• Click on “New
New Volume”
Volume button. Enter a name for the volume. Enter a short description.
• Select “Limit
Limit access”
access radio button and then select “Limit
Limit access to iSCSI initiator group”
group button and then
click on the drop down arrow to select the initiator group for this server (oraclelinux1 from the earlier
example). Then click “Next
Next”.
Next

• Enter the size for the volume and click “Next

Next”.
Next

NIMBLE TECHNICAL REPORT: NIMBLE STORAGE FOR ORACLE 10

• At the Protection screen, select “None
None”
None for now and click “Finish
Finish”.
Finish We will discuss the Protection option
later in the paper.

Make sure to create enough volumes for the Oracle server. Usually in an Oracle ASM environment, you
create one OCR volume for the Clusterware and create more volumes for the DATA diskgroup and FRA
diskgroup depending on how you want to architect your Oracle database.

Once all volumes have been created on the Nimble array, scan for the new volumes on the Oracle server and
modify the /etc/multipath.conf file in your Oracle Linux server to use the aliases for the Oracle volumes.

Here is how you can setup multipath for the Oracle volumes.

• Scan for disks on the Oracle Linux server. Note that you need to scan for both iSCSI networks. If there is
a single discovery IP address, you can use that IP to run the discovery. Either method, please ensure
that you can see the same volume across two networks.

[root@oraclelinux1 ~]# iscsiadm -m discovery -t st -p <iSCSI IP #1>

[root@oraclelinux1 ~]# iscsiadm -m discovery -t st -p <iSCSI IP #2>
[root@oraclelinux1 ~]# iscsiadm -m node - -login

Note:
Note if you want to login for all volumes that this Linux server has access to, then run the above login
command. If you want to login for a certain volume, then use this command “iscsiadm –m node –T <Target
Volume IQN> - -login”.

• After the scan completes, you can run the following command to find out which disk serial number
belongs to which LUN so you can alias the volumes properly. In the below output, sdb and sdc is one
disk (same serial #) which I used devdata2 as an alias, sdd and sdf is one disk which I used devdata1 as
an alias, and sde and sdg is one disk which I used ocr as an alias.

NIMBLE TECHNICAL REPORT: NIMBLE STORAGE FOR ORACLE 11

 [root@oraclelinux1 ~]# for a in `cat /proc/partitions | awk '{print $4}' | grep sd`; do echo "### $a: `scsi_id
-u -g /dev/$a`" ; done
### sda:
### sda1:
### sda2:
### sda3:
### sdb: 2fed4dc81dcf2582f6c9ce900abdb3466
### sdc: 2fed4dc81dcf2582f6c9ce900abdb3466
### sdd: 21ddbfcd70fbd5dae6c9ce900abdb3466
### sde: 20995db861a86c20e6c9ce900abdb3466
### sdf: 21ddbfcd70fbd5dae6c9ce900abdb3466
### sdg: 20995db861a86c20e6c9ce900abdb3466

• The new /etc/multipath.conf would look something like this. Please refer to Oracle Linux documentation
for specific parameter settings for other storage devices.

defaults {
user_friendly_names yes
find_multipaths yes
}
devices {
device {
vendor "Nimble"
product "Server"
path_selector "round-robin 0"
path_checker tur
rr_min_io_rq 10
rr_weight priorities
failback immediate
features "1 queue_if_no_path"
path_grouping_policy group_by_serial
}
}

blacklist {
wwid 26353900f02796769
devnode "^(ram|raw|loop|fd|md|dm-|sr|scd|st)[0-9]*"
devnode "^hd[a-z]"
}
multipaths {
multipath {
wwid 20995db861a86c20e6c9ce900abdb3466
alias ocr
}
multipath {
wwid 21ddbfcd70fbd5dae6c9ce900abdb3466
alias devdata1
}
multipath {
wwid 2fed4dc81dcf2582f6c9ce900abdb3466
alias devdata2
}
}

• Reload multipathd daemon

NIMBLE TECHNICAL REPORT: NIMBLE STORAGE FOR ORACLE 12

 [root@oraclelinux1 ~]# /etc/init.d/multipathd reload
Reloading multipathd: [ OK ]

• Verify multipath

[root@oraclelinux1 ~]# multipath -ll

devdata2 (2fed4dc81dcf2582f6c9ce900abdb3466) dm-0 Nimble,Server
size=100G features='1 queue_if_no_path' hwhandler='0' wp=rw
`-+- policy='round-robin 0' prio=1 status=active
|- 21:0:0:0 sdb 8:16 active ready running
`- 22:0:0:0 sdc 8:32 active ready running
devdata1 (21ddbfcd70fbd5dae6c9ce900abdb3466) dm-1 Nimble,Server
size=100G features='1 queue_if_no_path' hwhandler='0' wp=rw
`-+- policy='round-robin 0' prio=1 status=active
|- 23:0:0:0 sdd 8:48 active ready running
`- 24:0:0:0 sdf 8:80 active ready running
ocr (20995db861a86c20e6c9ce900abdb3466) dm-2 Nimble,Server
size=5.0G features='1 queue_if_no_path' hwhandler='0' wp=rw
`-+- policy='round-robin 0' prio=1 status=active
|- 25:0:0:0 sde 8:64 active ready running
`- 26:0:0:0 sdg 8:96 active ready running

• Change disk IO scheduler to “noop”

To set at boot time, add the elevator option at the kernel line in the /etc/grub.conf file:

elevator=noop

To manually set for all disks:

root@mktg04 ~]# multipath -ll | grep sd | awk -F":" '{print $4}' | awk '{print $2}' | while read LUN; do echo
noop > /sys/block/${LUN}/queue/scheduler ; done

• Change max_sectors_kb to “1024”

To change max_sectors_kb to 1024 for a single volume:

[root@racnode1 ~]# echo 1024 > /sys/block/sd?/queue/max_sectors_kb

Change all volumes:

multipath -ll | grep sd | awk -F":" '{print $4}' | awk '{print $2}' | while read LUN
do
echo 1024 > /sys/block/${LUN}/queue/max_sectors_kb
done

Note:
Note To make this change persistent after reboot, add the commands in /etc/rc.local file.

• Change vm dirty writeback and expire to “500” and “3000” respectively

NIMBLE TECHNICAL REPORT: NIMBLE STORAGE FOR ORACLE 13

 To change vm dirty writeback and expire:

[root@racnode1 ~]# echo 100 > /proc/sys/vm/dirty_writeback_centisecs

[root@racnode1 ~]# echo 100 > /proc/sys/vm/dirty_expire_centisecs

Note:
Note To make this change persistent after reboot, add the commands in /etc/rc.local file.

• Use “performance” setting for all available CPUs on the host when possible.

To change CPU governor setting to performance:

[root@racnode1 ~]# echo performance > /sys/devices/system/cpu/<cpu #>/cpufreq/scaling_governor

Change all CPUs

[root@racnode1 ~]# for a in $(ls -ld /sys/devices/system/cpu/cpu[0-9]* | awk '{print $NF}') ; do echo
performance > $a/cpufreq/scaling_governor ; done

Note:
Note To make this change persistent after reboot, add the commands in /etc/rc.local file.

Protection using Nimble Volume Collection

Once you have created all volumes for the Oracle server, you can manually add the volumes to a Volume
Collection. Volume Collection allows you to take a snapshot via a schedule or ad hoc using the Nimble
Storage management GUI or CLI. In an Oracle environment where multiples volumes have been created for a
particular database, for consistency, you need to take a snapshot of all volumes concurrently. The Volume
Collection allows you to do just that.

After a snapshot has been taken using the Volume Collection feature, the snapshot(s) will be available for
recovery based on your retention policy setting. The snapshot(s) can be deleted manually if you wish to do
so. Please note that you cannot delete a snapshot if there is a clone volume associated with that snapshot.

This particular feature is very useful in an Oracle environment. It allows you to create a clone of your
production database for Test, Development, or Staging environment quickly and easily.

Below are the steps to create a Volume Collection and assign the Oracle volumes to it.

• In the Nimble management GUI, click on “Manage

Manage”
Manage
“Protection
Protection”
Protection
“Volume
Volume Collection”
Collection

NIMBLE TECHNICAL REPORT: NIMBLE STORAGE FOR ORACLE 14

• Click on “New
New Volume Collection”
Collection button

• A “Create
Create a volume collection”
collection window appears. Enter a name for this volume collection and select
Blank schedule and click “Next
Next”.
Next

NIMBLE TECHNICAL REPORT: NIMBLE STORAGE FOR ORACLE 15

• A “Synchronization
Synchronization”
Synchronization window appears. As of this writing, you can select the “None
None”
None radio button and click
“Next
Next”.
Next

• A “Schedule
Schedule”
Schedule window appears. You must have at least one protection (snapshot) schedule defined.
Enter a name and your desired schedule/retention and click “Next
Next”.
Next

NIMBLE TECHNICAL REPORT: NIMBLE STORAGE FOR ORACLE 16

 • A “Volume
Volume”
Volume window appears. This is where you need to add all of your Oracle volumes to this volume
collection. When done, click “Finish
Finish”.
Finish

Oracle 11gR2 Setup

Now that the operating system and Nimble Storage have been setup, we’ll discuss installing Grid Infrastructure and
Oracle software.

Pre-requisites
• Create a local directory on the boot LUN or a separate Nimble LUN for the Grid Infrastructure and Oracle
software.
• Create group ID of “oinstall”
• Create group ID of “dba”
• Create user ID of “grid”
• Create user ID of “oracle”

Oracle ASM Disks Setup

o Run “/etc/init.d/oracleasm configure” command

[root@oraclelinux1 ~]# /etc/init.d/oracleasm configure

Configuring the Oracle ASM library driver.

This will configure the on-boot properties of the Oracle ASM library
driver. The following questions will determine whether the driver is
loaded on boot and what permissions it will have. The current values
will be shown in brackets ('[]'). Hitting <ENTER> without typing an
answer will keep that current value. Ctrl-C will abort.

Default user to own the driver interface []: grid

NIMBLE TECHNICAL REPORT: NIMBLE STORAGE FOR ORACLE 17

 Default group to own the driver interface []: oinstall
Start Oracle ASM library driver on boot (y/n) []: y
Scan for Oracle ASM disks on boot (y/n) []: y
Writing Oracle ASM library driver configuration: done
Initializing the Oracle ASMLib driver: [ OK ]
Scanning the system for Oracle ASMLib disks: [ OK ]

o Modify the /etc/sysconfig/oracleasm file as follow:

# ORACLEASM_ENABELED: 'true' means to load the driver on boot.

ORACLEASM_ENABLED=true
true

# ORACLEASM_UID: Default user owning the /dev/oracleasm mount point.

ORACLEASM_UID=grid
grid

# ORACLEASM_GID: Default group owning the /dev/oracleasm mount point.

ORACLEASM_GID=oinstall
oinstall

# ORACLEASM_SCANBOOT: 'true' means scan for ASM disks on boot.

ORACLEASM_SCANBOOT=true
true

# ORACLEASM_SCANORDER: Matching patterns to order disk scanning

ORACLEASM_SCANORDER="dm
dm"
dm

# ORACLEASM_SCANEXCLUDE: Matching patterns to exclude disks from scan

ORACLEASM_SCANEXCLUDE="sd
sd"
sd

o Make sure ASM module is loaded. If not, then load it.

[root@oraclelinux1 ~]# lsmod | grep oracleasm

oracleasm 53352 1

o Create ASM disks

[root@oraclelinux1 ~]# /etc/init.d/oracleasm creatdisk OCR /dev/mapper/ocr

[root@oraclelinux1 ~]# /etc/init.d/oracleasm creatdisk DEVDATA1 /dev/mapper/devdata1
[root@oraclelinux1 ~]# /etc/init.d/oracleasm creatdisk DEVDATA2 /dev/mapper/devdata2

o Verify ASM disks

[root@oraclelinux1 ~]# /etc/init.d/oracleasm listdisks

DEVDATA1
DEVDATA2
OCR

NIMBLE TECHNICAL REPORT: NIMBLE STORAGE FOR ORACLE 18

Install Grid Infrastructure
As grid user, you can now install Grid Infrastructure. Follow Oracle documentation on how to install GI at
www.oracle.com. When you get to the part where you have to create ASM disk group for the Clusterware,
change the asm_diskstring to “/dev/oracleasm/disks/*” and select OK. Remember that you need to create
one disk group for the Clusterware and one disk group for the Oracle database. In this case, for the
Clusterware, you can name the disk group as “OCRDG” and select only OCR disk. For the other two disks,
create a new disk group and name it “DATADG”. For both disk groups, make sure you select “EXTERNAL
REDUNDANCY”. Please note that the naming and number of disks specified in this document is an example
only. You can have different names and as many disks as you’d like.

Install Oracle Software

After the installation of GI, change user to oracle and install Oracle software. Again, follow Oracle
documentation on how to install Oracle if you don’t already know how. It is much easier to select the option
to install software only first. After the software installation, run “dbca” to create your database.

NIMBLE TECHNICAL REPORT: NIMBLE STORAGE FOR ORACLE 19

Nimble Storage, Inc.
211 River Oaks Parkway, San Jose, CA 95134
Tel: 877-364-6253) | www.nimblestorage.com | [email protected]
© 2014 Nimble Storage, Inc. Nimble Storage, InfoSight, SmartStack, NimbleConnect, and CASL are trademarks or registered trademarks of
Nimble Storage, Inc. All other trademarks are the property of their respective owners. TR-OG-1114

NIMBLE TECHNICAL REPORT: NIMBLE STORAGE FOR ORACLE 20