Ksplice - Keep your Database systems up to date with no downtime

Oracle Linux
Ksplice
KEEP YOUR DATABASE SYSTEMS
UP TO DATE WITH NO DOWNTIME

Luis & Yves
▪ Luís Marques
▪ Oracle ACE
▪ Lisbon
▪ http://lcmarques.com / @drune / luis.marques@redglue.eu
▪ Yves Colin
▪ Oracle ACE Associate
▪ Paris
▪ http://ycolin.wordpress.com / @ycolin / ycolinora@gmail.com

ksplice - A piece of history (before Oracle)
▪ Created by Jeffrey Arnold, Tim Abbott, Waseem Daher, and Anders Kaseorg at MIT
▪ Released as open source (GPL v2 license) - Latest open source version: 0.9.9
▪ Ksplice (as company) started in 2008 and got 700 customers
▪ Support for RHEL, CentOS, Fedora, Debian, Ubuntu, CloudLinux and Scientific Linux
▪ Announced in LKML as “A system for rebootless kernel security updates” – (23 April
2008) (http://lwn.net/Articles/279378/)

ksplice - A piece of history (after Oracle)
▪ Oracle acquired Ksplice in 2011.
▪
▪ Offers feature only to Oracle Linux Premier Support customers
▪ Support for Oracle Linux (UEK, RH compatible kernels) via ULN – (also supports
RHEL as legacy customers(?))
▪ Support for Ubuntu and Fedora “Desktop” versions as “free community protection”
▪ Strategic move for Oracle: mission critical systems run Linux (eg: Databases,
Exadata, ODA)

ksplice – Why use it?
Allows to install security and reliability patches without reboot and
without downtime
Downtime (and other) dilemmas:
▪ Huge security Linux kernel hole discovered and few Oracle single instance without
DR?
▪ Factory Control Systems, Energy Management System Critical Applications
▪ Rollback an update without another emergency reboot?
▪ SLA or Security?

ksplice – Installation
Kernel Patching – Manual (Recommended)
# wget -N https://www.ksplice.com/uptrack/install-uptrack
# sh install-uptrack ACCESS_KEY
# uptrack-upgrade –y
Kernel Patching - Automatic Mode
# sh install-uptrack ACCESS_KEY –autoinstall
or edit /etc/uptrack/uptrack.conf and change autoinstall = yes

ksplice – Your first live demo with ksplice
Install and view all kernel updates available
# uptrack-show –-available
# uptrack-show
# uptrack-install <ksplice_id> -y
# uptrack-upgrade -y
# uptrack-uname –a

ksplice – Your first live demo with ksplice
You can patch your system without any reboot but remove them
rebootless as well
# uptrack-remove <ksplice_id>
# uptrack-remove –all
# uptrack-uname –a

ksplice and Oracle Database
▪ Ksplice doesn’t patch any Oracle shared libraries or binaries
▪ You want to patch your Database that is running of top of ext4 (?)
▪ Oracle instrumentation doesn’t catch it (?)
▪ Nothing in “Wait event” interface
▪ A few more cpu cycle are recorded during and after the patch, but is negligible.
ext4_file_open needs to be
patched but it is in use by
Oracle and your OS
The thread that can run the
function is suspended for
<1ms – No CPU can execute
it
Execution is resumed
<1ms

Ksplice ,Exadata and AWS
▪ Email from Ksplice support about Exadata:
▪ Check against OVM on EXADATA version 12.1.2.2.1.160119
▪ Tested under Redhat 7.3 EC2 image
▪ Possible on AWS AMI customized XEN kernel (“Supported for legacy customers”)

Kplice – How patch is created? (1/2)
Source Code of Running Kernel
Slice
Source Code of Patch
Ksplice Build Environment:
• Pre Object Code
• Post Object Code Ksplice-id.ko
Ksplice-id-vmlinux-
new.ko
Ksplice-id-vmlinux-
old.ko
Identify modified functions
Object Level Code difference
Patch Created
Slice

Kplice – How patch is created? (2/2)
▪ Ksplice compares code at object code level
▪ In this way Ksplice patch creation could be completely automated(in most of the cases)
▪ Ksplice compiles code(patched/non patched) with special gcc options -ffunction-sections -
fdata-sections

Kplice – How patch is applied?
Connect to the
repository
Download the
Splice (tar.gz)
Files in
/var/cache/uptrack/
Linux/$arch/$kernel
Ksplice-id.ko core module
interface
(userland – kernel land)
Ksplice-id-vmlinux-new.ko Ksplice-id-vmlinux-old.ko
Interaction via
/sys/modules/ksplice-id/ksplice
Inspect thread – patched functions
in execution? (safety check)
Execute stop_machine() to
suspend thread
Patched code loaded in
memory
Patch applied
Thread
execution
resumed

Kplice – How patch is applied (in deep) ?
Compare Running Code in
memory to pre-code(safety check)
Resolve Undefined Symbol in Post
Code/Patched code (where are
they located in running kernel?) pre
matching phase
Find the patched function in
running kernel
Modified Old Function to
insert a JMP instruction
Patch
applied
stop_machine() : disable CPU
interruption and set the
patching process in High
priority scheduling mode
Patched Function
Check kernel stack thread

Ksplice - Alternatives
▪ Kpatch (RedHat) / kGraft (Suse):
▪ Uses ftrace to do the patching
▪ Kpatch uses stop_machine() to load new code (like Ksplice)
▪ Kgraft works on thread level, as soon as the threads reaches
a point it switches to a new function
▪ KernelCare (CloudLinux)
▪ Proprietary software

Kpatch
▪ Kpatch > LivePatch (RedHat)
April 2015
Kernel 4.0
release –
Foundations for
LIVE PATCHING
May 2014
Submitted
Linux Kernel
Mainline
February 2014
Public release
GPLv2
June 2014
RHEL 7 – Preview
Canonical has a service called LivePatch that takes advantage of Kernel 4.x LIVE PATCHING

Ksplice – Web Management tool
▪ Webinterface available at: http://uptrack.ksplice.com or https://status-
ksplice.oracle.com
▪

Ksplice – Web Management tool
▪ Webinterface available at: http://uptrack.ksplice.com or https://status-
ksplice.oracle.com

Ksplice - Uptrack offline client
▪ Remove the need for servers to connect to “public” Ksplice server
▪ Create a local YUM mirror of Ksplice mirror and allow your servers to connect to it
▪ Or, Oracle bundles all available Ksplice patches for each supported kernel version
to one RPM package and install in every server using a pen drive 
▪ A new patch is available, a new RPM is available
▪ You cannot use the Web management tool.
▪ Details for offline client:
https://docs.oracle.com/cd/E37670_01/E37355/html/ol_olcsetup_ksplice.html

Ksplice - Ksplice SNMP Plugin
▪ Use Oracle Enterprise Manager 12c/13c(?) to monitor status of Ksplice on your systems.
▪ Query the status of Ksplice using SNMP command snmpwalk:
[root@phoenix ksplice-46gkktym]# $ snmpwalk -v 1 -c public -O e localhost KSPLICE-UPTRACK-MIB::kspliceStatus
KSPLICE-UPTRACK-MIB::kspliceStatus.0 = STRING: outofdate
▪ [root@phoenix ksplice-46gkktym]# $ snmpwalk -v 1 -c public -O e localhost KSPLICE-UPTRACK-MIB::kspliceAvailTable
▪ KSPLICE-UPTRACK-MIB::kspliceavailName.12 = STRING: [ur5of4nd]
KSPLICE-UPTRACK-MIB::kspliceavailName.13 = STRING: [ue4dtk2k]
KSPLICE-UPTRACK-MIB::kspliceavailName.14 = STRING: [wy52x339]
Details: http://docs.oracle.com/cd/E37670_01/E39380/html/ch01s11s01.html

Watch and learn..to tune wait events 

Ksplice - Keep your Database systems up to date with no downtime

Recommended

More Related Content

What's hot (20)

Similar to Ksplice - Keep your Database systems up to date with no downtime (20)

More from Luis Marques (7)

Recently uploaded (20)

Ksplice - Keep your Database systems up to date with no downtime

Editor's Notes