Tuning your Red Hat System for Databases

Sanjay Rao Principal Software Engineer, Red Hat May 06, 2011

Objectives of this session

Share tuning tips

Bare metal

Aspects of tuning Tuning parameters Results of the tuning

Virtual Machines (RHEL)

Tools

Reading Graphs

Green arrow shows direction of best results

What To Know About Tuning

Proactive or Reactive Understand the Trade-offs No silver bullet You get what you pay for

What To Tune

I/O Memory CPU Network This session will cover I/O and Memory extensively

I/O Tuning - Hardware

Know Your Storage

SAS or SATA? Fibre Channel, Ethernet or SSD? Bandwidth limits

Device-mapper multipath Provides multipathing capabilities and LUN persistence Low level I/O tools dd, iozone, dt, etc. I/O representative of the database implementation

Multiple HBAs

How To

I/O Tuning Understanding I/O Elevators

Deadline

Two queues per device, one for read and one for writes I/Os dispatched based on time spent in queue

CFQ

Per process queue Each process queue gets fixed time slice (based on process priority)

Noop

FIFO Simple I/O Merging Lowest CPU Cost

I/O Tuning Configuring I/O Elevators

Boot-time

Grub command line elevator=deadline/cfq/noop echo deadline > /sys/class/block/sda/queue/scheduler

Dynamically, per device

ktune service (RHEL 5 ) tuned (RHEL6 utility)

tuned-adm profile throughput-performance tuned-adm profile enterprise-storage

CFQ vs Deadline 1 thread per device (4 devices)

Comparison CFQ vs Deadline
1 thread per multipath device (4 devices)
450M 4

400M

350M

300M 1 250M 0 200M -1 150M -2 CFQ MP dev Deadline MP dev % Diff (CFQ vs Deadline)

100M

50M

-3

0M

-4

8K-SW

16K-SW

32K-SW

32K-RW

16K-RW

64K-SW

8K-RW

64K-RW

16K-RR

8K-SR

8K-RR

16K-SR

32K-SR

32K-RR

64K-SR

64K-RR

CFQ vs Deadline 4 threads per device (4 devices)

Comparison CFQ vs Deadline
4 threads per multipath device (4 devices)
600M 250

500M

200

400M

150

300M

100

CFQ Deadline % diff (CFQ vs Deadline)

200M

50

100M

0M

-50

8K-SW

8K-RW

16K-SW

32K-SW

32K-RW

64K-SW

64K-RW

16K-RW

16K-RR

32K-RR

64K-RR

8K-SR

8K-RR

16K-SR

32K-SR

64K-SR

I/O Tuning Elevators OLTP - Sybase

Sybase IO scheduler testing - RHEL 5.5
OLTP transactional throughput on a Quad Core 4 Socket 2.5Ghz 96G Physical
180K

165K
160K

164K

162K

140K

120K

100K

80K

60K

40K

20K

DEADLINE

CFQ

NOOP

Impact of I/O Elevator - OLTP Workload

120K

100K

80K
Transactions / min

CFQ Deadline Noop

60K

40K

20K

K 10U 20U

Users

40U

60U

DSS Workload
Comparison CFQ vs Deadline
Oracle DSS Workload (with different thread count)
11:31 70

10:05

58.4 54.01

60

08:38

47.69

50

07:12

Time Elapsed

40 05:46 30 04:19 20 02:53

CFQ Deadline % diff

01:26

10

00:00

16
Parallel degree

32

I/O Tuning - File Systems

Direct I/O

Avoid double caching Predictable performance Reduce CPU overhead Eliminate synchronous I/O stall Critical for I/O intensive applications Database (parameters to configure read ahead) Block devices (getra, setra)

Asynchronous I/O

Configure read ahead

Turn off I/O barriers (RHEL6 and enterprise storage only)

I/O Tuning Effect of Direct I/O, Asynch I/O

OLTP Workload - 4 Socket 2 cores - 16G mem
Mid-level Fibre channel storage
80K

70K

60K

50K

Trans/min

40K

Setall DIO only AIO only No AIO DIO

30K

20K

10K

I/O Tuning Database Layout

Separate files by I/O (data , logs, undo, temp) OLTP data files / logs DSS data files / temp files Use low latency / high bandwidth devices for hot spots

I/O Tuning OLTP Logs - Fusion-IO

OLTP workload - Logs on FC vs Fusion-IO
Single Instance 400K

23.77 22.3 21.01

25

350K

20

300K

250K
Trans/Min

15

200K
10

Logs Fusion-io Logs FC %diff

150K

100K
5

50K

K
10U 40U 80U

I/O Tuning Storage (OLTP database)

OLTP workload - Fibre channel vs Fusion-IO
4 database instances 1400K

1200K

1000K

800K
Trans/Min

600K

400K

200K

4G-FC

Fusion-io

I/O Tuning OLTP Database (vmstats)

4 database instances Fibre channel
r b 62 19 3 20 21 27 7 20 25 18 4 21 1 32 4 17 17 14 35 14 20 13 1 14 23 24 34 12 r b 77 0 77 1 76 3 81 1 82 3 81 3 79 1 79 0 76 2 61 3 77 1 54 4 80 2 swpd free buff cache si so bi bo in cs us sy id wa st 5092 44894312 130704 76267048 0 0 8530 144255 35350 113257 43 4 45 9 0 5092 43670800 131216 77248544 0 0 6146 152650 29368 93373 33 3 53 11 0 5092 42975532 131620 77808736 0 0 2973 147526 20886 66140 20 2 65 13 0 5092 42555764 132012 78158840 0 0 2206 136012 19526 61452 17 2 69 12 0 5092 42002368 132536 78647472 0 0 2466 144191 20255 63366 19 2 67 11 0 5092 41469552 132944 79111672 0 0 2581 144470 21125 66029 22 2 65 11 0 5092 40814696 133368 79699200 0 0 2608 151518 21967 69841 23 2 64 11 0 5092 40046620 133804 80385232 0 0 2638 151933 23044 70294 24 2 64 10 0 5092 39499580 134204 80894864 0 0 2377 152805 23663 72655 25 2 62 10 0 5092 38910024 134596 81436952 0 0 2278 152864 24944 74231 27 2 61 9 0 5092 38313900 135032 81978544 0 0 2091 156207 24257 72968 26 2 62 10 0 5092 37831076 135528 82389120 0 0 1332 155549 19798 58195 20 2 67 11 0 5092 37430772 135936 82749040 0 0 1955 145791 19557 56133 18 2 66 14 0 5092 36864500 136396 83297184 0 0 1546 141385 19957 56894 19 2 67 13 0 swpd free buff cache si so 6604 55179876 358888 66226960 0 0 6604 50630092 359288 70476248 0 0 6604 46031168 360132 74444776 0 0 6604 41510608 360512 78641480 0 0 6604 35358836 361012 84466256 0 0 6604 34991452 361892 84740008 0 0 6604 34939792 362296 84747016 0 0 6604 34879644 362992 84754016 0 0 6604 34844616 363396 84760976 0 0 6604 34808680 363828 84768016 0 0 6604 34781944 364180 84774992 0 0 6604 34724948 364772 84803456 0 0 6604 34701500 365500 84809072 0 0 bi 7325 6873 5818 4970 4011 2126 2323 2275 2275 2209 2172 3031 2216 bo in cs us sy 266895 70185 149686 90 7 306900 70166 149804 88 7 574286 77388 177454 88 8 452939 75322 168464 89 7 441042 74022 162443 88 7 440876 73702 161618 88 7 400324 73091 161592 90 6 412631 73271 160766 89 6 415777 73019 158614 89 6 401522 72367 159100 89 6 401966 73253 159064 90 6 421299 72990 156224 89 6 573246 76404 175922 88 7 id wa st 4 0 0 5 0 0 4 0 0 3 0 0 4 0 0 5 0 0 3 0 0 4 0 0 4 0 0 4 0 0 4 0 0 4 0 0 5 1 0

4 database instances Solid State devices (PCI) - Fusion-io

I/O Tuning DSS - Temp

DSS Workload - Sort-Merge table create - Time Metric - Smaller is better
01:55:12

01:40:48

01:26:24

01:12:00

Elapsed Time

00:57:36

00:43:12

00:28:48

00:14:24

00:00:00

4G-FC

Fusion-IO

I/O Tuning RHEL 6 DSS Workload Sybase IQ 15.2

120K 100K
98K 87K

80K

77K

60K
51K 44K 47K

2 FC Arrays Fusion-io

40K

20K

K
Different measurement metrics

What To Tune

I/O Memory Memory CPU Network

Memory Tuning

Dense Memory Based on Architecture NUMA Huge Pages

Understanding NUMA (Non Uniform Memory Access)

Multi Socket Multi core architecture

NUMA required for scaling RHEL 5 / 6 completely NUMA aware Additional performance gains by enforcing NUMA placement

How to enforce NUMA placement

numactl CPU and memory pinning taskset CPU pinning cgroups (only in RHEL6) libvirt for KVM guests CPU pinning

Memory Tuning Huge Pages

2M pages vs 4K standard linux page Virtual to physical page map is 512 times smaller TLB can map more physical pages, resulting in fewer misses Traditional Huge Pages always pinned Transparent Huge Pages in RHEL6 Most databases support Huge pages How to configure Huge Pages (16G)

echo 8192 > /proc/sys/vm/nr_hugepages vi /etc/sysctl.conf (vm.nr_hugepages=8192)

Memory Tuning Huge Pages Sybase - OLTP

Sybase Huge Pages Testing - RHEL 5.5
OLTP transactional throughput on a Quad Core 4 Socket 2.5Ghz 96G Physical

180K

175K

170K
165K

160K

150K

140K

130K

120K

110K

100K

default

hugepages

OLTP Workload Effect of NUMA and Huge Pages

OLTP workload - Multi Instance
1400K
20

17.82
1200K

18

16

11.7
800K
Trans/Min

12

10

600K

8.23
8

400K

200K
2

0 Non NUMA NUMA non NUMA Huge Pages NUMA Huge Pages

% diff compared to Non NUMA number

1000K

14

NUMA and Huge Pages

Huge page allocation takes place uniformly across NUMA nodes Make sure that database shared segments are sized to fit Workaround Allocate Huge pages / Start DB / De-allocate Huge pages

Physical Memory 128G 4 NUMA nodes Huge Pages 80G 20G in each NUMA node 24G DB Shared Segment using Huge Pages 24G DB Shared Segment using NUMA and Huge Pages Huge Pages 100G 25G in each NUMA node

Tuning Memory Flushing Caches

Drop unused Cache Frees unused memory File cache If the DB uses cache, may notice slowdown

Free pagecache echo 1 > /proc/sys/vm/drop_caches Free slabcache

echo 2 > /proc/sys/vm/drop_caches echo 3 > /proc/sys/vm/drop_caches

Free pagecache and slabcache

CPU Tuning

CPU performance

Clock speed Multiple cores Power savings mode

cpuspeed off performance ondemand powersave

How To

echo "performance" > /sys/devices/system/cpu/cpu0/cpufreq/scaling_governor best of both worlds cron jobs to configure the governor mode ktune (RHEL5) tuned-adm profile server-powersave (RHEL6)

Tuning CPU Impact of Power Settings

RHEL6 - Database - OLTP workload
cpuspeed settings ( Range - 2.27 GHz - 1.06 GHz)
450K 400K 350K 300K 250K 200K 150K 100K 50K K
10U 20U

cpspeed off performance ondemand powersave

Tuning CPU Effect of Power Settings - DSS

DSS workload (I/O intensive)
Time Metric (Lower is better)
10:05

08:38

07:12

05:46

04:19

02:53

01:26

00:00

performance

ondemand

powersave

vmstat output during test: 7 12 7 12 2 0 7 11 1 15 5884 122884416 485900 734376 5884 122885024 485900 734376 5884 122884928 485908 734376 5884 122885056 485912 734372 5884 122885176 485920 734376 0 0 0 0 0 0 184848 39721 9175 37669 0 217766 27468 9904 42807 0 168496 45375 6294 27759 0 178790 40969 9433 38140 0 248283 19807 7710 37788 4 4 4 4 5 1 89 2 87 1 90 1 90 2 86 6 6 5 5 7 0 0 0 0 0

Network Tuning

Network Performance

Separate network for different functions If on same network, use arp_filter to prevent ARP flux

echo 1 > /proc/sys/net/ipv4/conf/all/arp_filter Supports RDMA w/ RHEL6 high performance networking package

10GigE

Infiniband Packet size

Database Performance

Application tuning

Design Reduce locking / waiting Database tools (optimize regularly)

Tuning - Virtualization KVM (RHEL guests)

Virtualization Tuning - Caching

Cache = none

I/O from the guest is not cached on the host I/O from the guest is cached and written through on the host Works well on large systems (lots of memory and CPU) Potential scaling problems with this option with multiple guests (host CPU used to maintain cache) Can lead to swapping on the host

Cache = writethrough

How To Configure I/O - Cache per disk in qemu command line or libvirt

Effect of I/O Cache Settings on Guest Performance

1000K 900K 800K 700K 600K 500K 400K 300K 200K 100K K
0 15 25 35

31.67
30

20 Cache=none Cache=WT %diff

10

5.82
5

1Guest

4Guests

Configurable per device: Virt-Manager - drop-down option under Advanced Options Libvirt xml file - driver name='qemu' type='raw' cache='writethrough' io='native'

AIO Native vs Threaded (default)

1200K

1000K

800K

600K

AIO Native AIO Default (threaded)

400K

200K

K
10U 20U

Configurable per device (only by xml configuration file): Libvirt xml file - driver name='qemu' type='raw' cache='writethrough' io='native'

Virtualization Tuning I/O Elevators - OLTP

Host Running Deadline
Trans / min - Higher is better
300K

250K

200K CFQ Deadline Noop

150K

100K

50K

1Guest

2 Guests

4 Guests

Virtualization Tuning I/O elevators - DSS

Host Running Deadline

Time metric - Lower is better

20:10

17:17

14:24

11:31

CFQ Deadline Noop

08:38

05:46

02:53

00:00

1Guest

2 Guests

4 Guests

Virtualization Tuning Using NUMA

400K
35

350K

28.6

30

300K
25

250K
20

200K
15

150K
10

Guest 4 Guest 3 Guest 2 Guest 1 % improvement

100K
5

50K

0.0
4Guest-24vcpu-56G 4Guest-24vcpu-56G-NUMA

Virtualization Tuning - Network

VirtIO

VirtIO drivers for network Bypass the qemu layer Bypass the host and pass the PCI device to the guest Can be passed only to one guest

vhost_net (low latency close to line speed)

PCI pass through

SR-IOV (Single root I/O Virtualization)

Pass through to the guest Can be shared among multiple guests Limited hardware support

Latency Comparison RHEL 6

Network Latency by Guest Interface Method
Guest Receive (Lower is better)
400 350 300

Latency (usecs)

250 200 150 100 50 0

1 6 16 27 45 64 99 189 256 387 765 4 102 9 153 9 306 6 409 5 4 9 9 6 7 7 614 1228 1638 2457 4914 6553 9830

vhost-net improves latency bringing it close to bare metal

Message Size (Bytes)

host RX virtio RX vhost RX SR-IOV RX

Performance Setting Tool

tuned for RHEL6

Configure system for different performance profiles

laptop-ac-powersave spindown-disk latency-performance laptop-battery-powersave server-powersave throughput-performance desktop-powersave enterprise-storage default

Performance Monitoring Tools

Monitoring tools

top, vmstat, ps, iostat, netstat, sar, perf /proc, sysctl, AltSysRq ethtool, ifconfig oprofile, strace, ltrace, systemtap, perf

Kernel tools

Networking

Profiling

Wrap Up Bare Metal

I/O

Choose the right elevator Eliminated hot spots Direct I/O or Asynchronous I/O Virtualization Caching NUMA Huge Pages Swapping Managing Caches

Memory

RHEL has many tools to help with debugging / tuning

Wrap Up Bare Metal (cont.)

CPU

Check cpuspeed settings Separate networks arp_filter Packet size

Network

Wrap Up Virtualization

VirtIO drivers aio (native) NUMA Cache options (none, writethrough) Network (vhost-net)