Oracle RAC and Docker: The Why and How

Oracle RAC and Docker: The Why and How
Seth Miller
Senior Principal Software Engineer

About Me
• Senior Principal Software Engineer at Veritas
• Oracle DBA since 2005
• Independent Oracle Users Group (IOUG) Board of Directors
• Twin Cities Oracle Users Group (TCOUG) Former Board of Directors
Copyright © 2016 Veritas Technologies LLC2

Download
Copyright © 2015 Symantec Corporation3

Why?

Why Oracle in Containers?
• Resource reduction
–# of VMs
–Memory & disk
• Time reduction
–Provision
–Configure
• Storage Deduplication
–Operating system
–Oracle Binaries

Considerations
• Bleeding edge
• Lack of vendor support
–Not certified
–Off-label
–No published images
• Automation
NOT
FOR
PRODUCTION
…at least not yet

Contradictions
• Shared nothing architecture
• Split brain isolation
• High availability
• Full node control
• Privileged access

Database in container = easy
• Single container
• No privileged access
• No extra networking
• No supporting processes
Single Instance vs. Cluster
https://github.com/oracle/docker-
images/tree/master/OracleDatabase

Cluster in containers = hard
• At least 2 containers
• Privileged access
• Multiple networks
• Needs init/systemd
• Runs processes as root
Single Instance vs. Cluster

What are Containers?
• Containers are not VMs
• User space
– Host kernel
– Linux capabilities — capabilities(7)
• CHOWN, DAC_OVERRIDE, FSETID, FOWNER, MKNOD, NET_RAW, SETGID, SETUID,
SETFCAP, SETPCAP, NET_BIND_SERVICE, SYS_CHROOT, KILL, AUDIT_WRITE
• Networking
– Virtual network adapters
• Storage
– Images
– Containers

Containers vs VMs
11 Copyright © 2016 Veritas Technologies LLC

• User space
– Host kernel
• Networking
• Storage
– Images
– Containers

Container Process Capabilities
ppid pid name command capabilities
10392 5957 root systemd full
5957 815 root init.tfa full
5957 5206 root init.ohasd full
5957 5997 root systemd-journal chown, dac_override, dac_read_search, fowner, setgid, setuid,
sys_ptrace, sys_admin, audit_control, mac_override, syslog,
wake_alarm, block_suspend, audit_read
5957 6042 root systemd-udevd full
5957 6110 root smartd full
5957 6111 81 dbus-daemon audit_write +
5957 6138 root rsyslogd full
5957 6140 root systemd-logind chown, dac_override, dac_read_search, fowner, kill, sys_admin,
sys_tty_config, audit_control, mac_admin, wake_alarm,
block_suspend, audit_read
5957 6159 root gssproxy full
5957 6265 root sshd full
5957 6276 root crond full
5957 6323 root rhnsd full
5957 6392 root ohasd.bin full
...

• User space
– Host kernel
• Networking
• Storage
– Images
– Containers

Virtual Network Adapters
docker0: <BROADCAST,MULTICAST,UP,LOWER_UP> state UP
link/ether 02:42:5d:3b:cc:3e brd ff:ff:ff:ff:ff:ff
inet 172.17.0.1/16 scope global docker0
...
vethd67c304@if32: <BROADCAST,MULTICAST,UP,LOWER_UP> master docker0 state UP
link/ether c6:7a:0e:0d:a3:76 brd ff:ff:ff:ff:ff:ff
...

• User space
– Host kernel
• Networking
• Storage
– Images
– Containers

Storage

How?

https://github.com/Seth-Miller/12c-rac-docker

Oracle Grid Infrastructure Networking
• 1 public network per node
– 1 public static IP per node
– 1 public virtual IP per node
– 3 public SCAN virtual IPs per cluster
– 1 public GNS virtual IP per cluster
• 1 private network per node
– 1 private static IP per node
•2 node cluster = 10 IPs
Static vs DHCP
DNS vs GNS

Platform - CoreOS
• Built for containers
• Built for cloud
• Automated
• Secure
• Lean
• Docker and rkt ready
• Preconfigured

Oracle RAC in Docker
• Host preparation
–Platform
–Storage
• Image preparation
–IPs and DNS
–Virtual Networks
–Storage
–OS
–Clusterware and database binaries
• Containers
–Volumes
–Shared memory
–Networking
–Communication
–Grid infrastructure configuration

Cloud Config
#cloud-config
ssh_authorized_keys:
- ssh-rsa AAAAB3NzaC1yc2EAAAABJQAAA...
coreos:
units:
- name: docker.service
drop-ins:
- name: 10.docker_opts.conf
content: |
[Service]
Environment="DOCKER_OPTS=--storage-driver=devicemapper
--storage-opt=dm.thinpooldev=/dev/mapper/docker-thinpool
--storage-opt=dm.use_deferred_removal=true
--storage-opt=dm.basesize=25G"

Storage
• ext4 + overlayfs (previously Btrfs) by default
• Changed to xfs + direct-lvm devicemapper storage
– Uses LVM2 and thin pools
– Configured on first boot with cloud config and systemd
core@mycoreos ~ $ docker info
…
Storage Driver: devicemapper
Pool Name: docker-thinpool
Pool Blocksize: 524.3 kB
Base Device Size: 26.84 GB
Backing Filesystem: xfs
…

Storage
• Additional storage
– Grid infrastructure and database binaries
• Uses LVM2 and ext4 filesystems
– ASM disk devices
– Configured on first boot with cloud config and systemd
NAME SIZE TYPE MOUNTPOINT
sdc 100G disk
|-data-oracledata 30G lvm /oracledata
sdd 16G disk
sde 16G disk
sdf 16G disk

–Platform
–Storage
–IPs and DNS
–Virtual Networks
–Storage
–OS
• Containers
–Volumes
–Shared memory
–Networking
–Communication

DNS and DHCP
• BIND container for DNS
• DHCPD container for DHCP
• Dynamic DNS updates
• Only bind and dhcpd containers receive static IPs
core@mycoreos ~ $ dig example.com. @10.10.10.10 axfr
example.com. 86400 IN SOA example.com. root.example.com.
example.com. 86400 IN NS localhost.
rac1.example.com. 3600 IN A 10.10.10.134
rac2.example.com. 3600 IN A 10.10.10.135
...

Virtual Networks
• docker network create --subnet=10.10.10.0/24 pub
• docker network create --subnet=11.11.11.0/24 priv

Image Preparation
• Install packages
• Create OS users and groups
• Change OS user passwords
• Modify security limits
• Add udev rules for ASM disks
$ docker exec rac1 ls -l /dev/sd[d-f]
brw-rw----. 1 root oinstall 8, 48 Oct 17 16:49 /dev/sdd
brw-rw----. 1 root oinstall 8, 64 Oct 17 16:49 /dev/sde
brw-rw----. 1 root oinstall 8, 80 Oct 17 16:49 /dev/sdf
$ docker exec rac1 ls -ld /dev/asmdisks/
drwxr-xr-x. 2 root root 100 Oct 17 16:49 /dev/asmdisks/
$ docker exec rac1 ls -l /dev/asmdisks/
total 0
lrwxrwxrwx. 1 root root 6 Oct 17 16:49 asm-clu-121-DATA-disk1 -> ../sdd
lrwxrwxrwx. 1 root root 6 Oct 17 16:49 asm-clu-121-DATA-disk2 -> ../sde
lrwxrwxrwx. 1 root root 6 Oct 17 16:49 asm-clu-121-DATA-disk3 -> ../sdf

Image Preparation
• Install grid infrastructure binaries
• Install database binaries
• Commit container to a new image

–Platform
–Storage
–IPs and DNS
–Virtual Networks
–Storage
–OS
• Containers
–DNS & DHCPD
–Volumes
–Shared memory
–Networking
–Communication

Containers
docker create
--interactive
--tty
--name bind
--hostname bind
--publish 53:53/tcp
--publish 53:53/udp
--volume /srv/docker/bind:/data
--env WEBMIN_ENABLED=false
sethmiller/bind
-4
Create the BIND container
docker network connect
--ip 10.10.10.10 pub
bind
docker start bind

Containers
docker create
--interactive
--tty
--name dhcpd
--hostname dhcpd
--volume /srv/docker/dhcpd:/data
--volume /srv/docker/bind/bind/etc:/keys
--dns 10.10.10.10
networkboot/dhcpd
docker network connect --ip 10.10.10.11 pub dhcpd
docker network connect --ip 11.11.11.11 priv dhcpd
docker start dhcpd
Create the DHCPD container

Containers
docker run
--detach
--privileged
--name rac1
--hostname rac1
--volume /oracledata/stage:/stage
--volume /sys/fs/cgroup:/sys/fs/cgroup:ro
--dns 10.10.10.10
--shm-size 2048m
giinstalled
/usr/lib/systemd/systemd --system --unit=multi-user.target
Create the RAC node container

Docker Networks
$ docker network ls
NETWORK ID NAME DRIVER
4ee6f9eb3bf0 bridge bridge
8bc52205f018 host host
eb82d0dad5c6 none null
60e3892778b4 priv bridge
21319f974244 pub bridge

Containers
ip link add name rac1-pub type veth peer name eth-pub
ip link set rac1-pub master <docker ‘pub’ network>
ip link set rac1-pub up
ip link set eth-pub netns <rac1 namespace PID>
ip netns exec <rac1 namespace PID> ip link set eth-pub up
ip link add name rac1-priv type veth peer name eth-priv
ip link set rac1-priv master <docker ‘priv’ network>
ip link set rac1-priv up
ip link set eth-priv netns <rac1 namespace PID>
ip netns exec <rac1 namespace PID> ip link set eth-priv up
Add the additional virtual
network adapters
Legend
Host level
Container level

Containers
[Service]
ExecStart=dhclient -d -H rac1
-pf /var/run/dhclient-eth-pub.pid eth-pub
ExecStop=dhclient -x eth-pub
[Service]
ExecStart=dhclient -d -H rac1-priv
-pf /var/run/dhclient-eth-priv.pid eth-priv
ExecStop=dhclient -x eth-priv
Create dhcp systemd services

Containers
docker exec rac1 systemctl start dhclient-rac1-eth-pub.service
docker exec rac1 systemctl start dhclient-rac1-eth-priv.service
Start dhcp systemd services

Containers
docker run
--detach
--privileged
--name rac2
--hostname rac2
--volume /oracledata/stage:/stage
--volume /sys/fs/cgroup:/sys/fs/cgroup:ro
--dns 10.10.10.10
--shm-size 2048m
giinstalled
/usr/lib/systemd/systemd --system --unit=multi-user.target
sudo /srv/docker/scripts/networks-rac2.sh
Create the second RAC node
container

Configure Grid Infrastructure
• Establish cluster communication
• Start clusterware processes
• Set up GNS
• Assign IPs
• Set up ASM and create first disk group
• Set up SCAN and local listeners
• Set up Grid Infrastructure Management Repository (GIMR) (optional)

Create Database
docker exec rac1 su - oracle -c '
/u01/app/oracle/product/12.1.0/dbhome_1/bin/dbca -createDatabase -silent
-templateName General_Purpose.dbc
-gdbName orcl
-sysPassword <sys password>
-systemPassword <system password>
-storageType ASM
-diskGroupName DATA
-recoveryGroupName DATA
-characterSet AL32UTF8
-nationalCharacterSet UTF8
-totalMemory 1024
-emConfiguration none
-nodelist rac1,rac2
-createAsContainerDatabase True'

Future Plans
• Fully Automated
• Kubernetes
• Support NFS files for ASM (ganesha NFS)
• Run RAC nodes non-privileged
• Lean out RAC node images
• Install fests (RAC Attack)

Thank you!
Seth Miller
sethmiller.sm@gmail.com
github.com/Seth-Miller/12c-rac-docker

Oracle RAC and Docker: The Why and How

Recommended

More Related Content

What's hot (20)

Similar to Oracle RAC and Docker: The Why and How (20)

Recently uploaded (20)

Oracle RAC and Docker: The Why and How

Editor's Notes