Fine-grained fault tolerance using device checkpoints

Fine-Grained Fault Tolerance
using Device Checkpoints
Asim Kadav
with Matthew Renzelmann and Michael M. Swift
University of Wisconsin-Madison
Thursday, March 7, 13

The (old) elephant in the room
2
device
drivers
3rd party developers
+
OS
kernel

The (old) elephant in the room
2
device
drivers
3rd party developers
+
OS
kernel
Recipe
for
disaster

Improvement System ValidationValidationValidationImprovement System
Drivers Bus Classes
New functionality Shadow driver migration [OSR09] 1 1 1
RevNIC [Eurosys 10] 1 1 1
Reliability Nooks [SOSP 03] 6 1 2
XFI [ OSDI 06] 2 1 1
CuriOS [OSDI 08] 2 1 2
Type Safety SafeDrive [OSDI 06] 6 2 3
Singularity [Eurosys 06] 1 1 1
Specification Nexus [OSDI 08] 2 1 2
Termite [SOSP 09] 2 1 2
Static analysis tools Windows SDV [Eurosys 06] All All All
Coverity [CACM 10] All All All
Cocinelle [Eurosys 08] All All All
3
Past work mostly looks at detection and isolation

Drivers Bus Classes
XFI [ OSDI 06] 2 1 1
3
Large kernel subsystems and validity of few device types
result in limited adoption of research solutions

Drivers Bus Classes
XFI [ OSDI 06] 2 1 1
3
Limited kernel changes + Applicable to lots of drivers =>
Real Impact

Drivers Bus Classes
XFI [ OSDI 06] 2 1 1
3
Limited kernel changes + Applicable to lots of drivers =>
Real Impact
Goal: Improve recovery with complete solutions
that can be applied to many drivers

State of the art in recovery: Shadow drivers
• Carburizer calls generic recovery
service if check fails
• Low cost transparent recovery
★ Based on shadow drivers
★ Records state of driver at all times
★ Transparently restarts and replays
recorded state on failure
Shadow
Driver
Device
Driver
Device
Taps
Driver-Kernel
Interface
4
Swift [OSDI ’04]

Recovery Performance: Device initialization is slow
5
Cold boot hardware,
flash device memory
Perform EEPROM
checsumming
Set chipset
specific ops
Optional self
test on boot
Allocate driver
structures
Configure device to
working state
Device ready
for requests
Allocate device structures
Module registration
Map BAR and I/O ports
Register device operations
Detect chipset capabilities
Self test?
Self test on boot
Cold boot the device
Verify EEPROM checksum
Set chipset specific ops
Allocate driver structures
Configure device
Device ready
for requests★ Multi-second device probe
★ Identify device
★ Cold boot device
★ Setup device/driver
structures
★ Configuration/Self-test

Recovery Performance: Device initialization is slow
5
Cold boot hardware,
flash device memory
Perform EEPROM
checsumming
Set chipset
specific ops
Optional self
test on boot
Allocate driver
structures
Configure device to
working state
Device ready
for requests
Allocate device structures
Module registration
Map BAR and I/O ports
Register device operations
Detect chipset capabilities
Self test?
Self test on boot
Cold boot the device
Verify EEPROM checksum
Set chipset specific ops
Allocate driver structures
Configure device
Device ready
for requests★ Multi-second device probe
★ Identify device
★ Cold boot device
★ Setup device/driver
structures
★ Configuration/Self-test
★ What does it hurt?
★ Fault tolerance: Driver recovery
★ Virtualization: Live migration,
cloning, consolidation
★ OS functions: Boot, upgrade,
NVM checkpoints

Driver Code
Characteristics
6
★ “Understanding Modern Device Drivers” ASPLOS 2012

0
10
20
30
40
50
Percent-
age of LOC
uwb
net
infiniband
atm
scsi
mtd
md
ide
block
ata
watchdog
video
tty
sound
serial
pnp
platform
parport
misc
message
media
leds
isdn
input
hwmon
hid
gpu
gpio
firewire
edac
crypto
char
cdrom
bluetooth
acpi
init cleanup ioctl config power error proc core intr
0
10
20
30
40
50
Percent-
age of LOC
chardriversblockdriversnetdrivers
Driver Code
Characteristics
6

0
10
20
30
40
50
Percent-
age of LOC
uwb
net
infiniband
atm
scsi
mtd
md
ide
block
ata
watchdog
video
tty
sound
serial
pnp
platform
parport
misc
message
media
leds
isdn
input
hwmon
hid
gpu
gpio
firewire
edac
crypto
char
cdrom
bluetooth
acpi
0
10
20
30
40
50
Percent-
age of LOC
★Initialization/cleanup – 36%
★Core I/O & interrupts – 23%
★Device conﬁguration – 15%
★Power management – 7.4%
★Device ioctl – 6.2%
Driver Code
Characteristics
6

0
10
20
30
40
50
Percent-
age of LOC
uwb
net
infiniband
atm
scsi
mtd
md
ide
block
ata
watchdog
video
tty
sound
serial
pnp
platform
parport
misc
message
media
leds
isdn
input
hwmon
hid
gpu
gpio
firewire
edac
crypto
char
cdrom
bluetooth
acpi
0
10
20
30
40
50
Percent-
age of LOC
★Initialization/cleanup – 36%
★Core I/O & interrupts – 23%
★Device conﬁguration – 15%
★Power management – 7.4%
★Device ioctl – 6.2%
Driver Code
Characteristics
Initialization code dominates driver
LOC and adds to complexity
6

Recovery works by interposing class defined entry points
7
★ Class definition includes:
★ Callbacks registered with the bus,
device and kernel subsystem
network
driver
bus
net device
subsystem
kernel
probe
xmit
config
network
card

Recovery works by interposing class defined entry points
7
How many drivers follow class behavior?
★ Class definition includes:
★ Callbacks registered with the bus,
device and kernel subsystem
network
driver
bus
net device
subsystem
kernel
probe
xmit
config
network
card

Restart/replay doesn’t work with all drivers
★ Non-class behavior stems from:
- Load time parameters, procfs and sysfs interactions, unique ioctls
...
qlcnic_sysfs_write_esw_config
(...)

{

...

switch
(esw_cfg[i].op_mode)
{

case
QLCNIC_PORT_DEFAULTS:

qlcnic_set_eswitch_...(...,&esw_cfg[i]);

...

case
QLCNIC_ADD_VLAN:

qlcnic_set_vlan_config(...,&esw_cfg[i]);

...

case
QLCNIC_DEL_VLAN:

esw_cfg[i].vlan_id
=
0;

qlcnic_set_vlan_config(...,&esw_cfg[i]);

...
Drivers/net/qlcnic/qlcnic_main.c:
Qlogic
driver(network
class)
8★ “Understanding Modern Device Drivers” ASPLOS 2012

Restart/replay doesn’t work with all drivers
★ Non-class behavior stems from:
- Load time parameters, procfs and sysfs interactions, unique ioctls
8
★ Results as measured by our analyses:
★ 36% of drivers use load time parameters
★ 16% of drivers use proc /sysfs support
★ Overall, 44% of drivers do not conform to class
behavior and recovery will not work correctly for
these drivers

Limitations of restart/replay recovery
Shadow
Driver
Device
Driver
Device
Taps
Driver-Kernel
Interface
9
★ Device save/restore limited to
restart/replay
★ Slow: Device initialization is
complex (multiple seconds)
★ Incomplete: Unique device
semantics not captured
★ Hard: Need to be written for
every class of drivers
★ Large changes: Introduces new
large kernel subsystems

Limitations of restart/replay recovery
Shadow
Driver
Device
Driver
Device
Taps
Driver-Kernel
Interface
9
★ Device save/restore limited to
restart/replay
★ Slow: Device initialization is
complex (multiple seconds)
★ Incomplete: Unique device
semantics not captured
★ Hard: Need to be written for
every class of drivers
★ Large changes: Introduces new
large kernel subsystems
Checkpoint/restore of device and driver state
removes the need to reboot device and replay state

Fine-Grained Fault Tolerance (FGFT)
10
Goal: Fault isolation and recovery system based on “pay as
you go” failure model
Fine-Grained Isolation
★ Ability to run select entry points as transactions
Checkpoint based recovery
★ Provides fast and correct recovery semantics
★ Requires incremental changes to drivers and has
low overhead

Outline
11
Introduction
Conclusion
Fine-grained isolation

Outline
12
Introduction
Conclusion

FGFT overview
If
(c==0)
{
.
print
(“Driver

init”);
}
.
.
Driver with
checkpoint support
Static modiﬁcations
13

FGFT overview
If
(c==0)
{
.
print
(“Driver

init”);
}
.
.
Driver with
checkpoint support
13
User supplied
annotations
Source transformation
(adds driver transactions)

FGFT overview
If
(c==0)
{
.
print
(“Driver

init”);
}
.
.
Driver with
checkpoint support
13
If
(c==0)
{
.
print
(“Driver

init”);
}
.
.
If
(c==0)
{
.
print
(“Driver

init”);
}
.
.
User supplied
annotations
Main driver
module
SFI driver
module
SFI = software fault
isolated

FGFT overview
If
(c==0)
{
.
print
(“Driver

init”);
}
.
.
Driver with
checkpoint support
Static modiﬁcations Run-time support
13
If
(c==0)
{
.
print
(“Driver

init”);
}
.
.
If
(c==0)
{
.
print
(“Driver

init”);
}
.
.
User supplied
annotations
Main driver
module
SFI driver
module
isolated

FGFT overview
If
(c==0)
{
.
print
(“Driver

init”);
}
.
.
Driver with
checkpoint support
Communication
and recovery
support
Static modiﬁcations Run-time support
13
If
(c==0)
{
.
print
(“Driver

init”);
}
.
.
If
(c==0)
{
.
print
(“Driver

init”);
}
.
.
1200 LOC
User supplied
annotations
Object tracking
Marshaling/
Demarshaling
Kernel
undo log
Main driver
module
SFI driver
module
isolated

Fault model in FGFT
14
network
driver
network
card
probe
xmit
conﬁg
★ Can be applied to untested code, statically and dynamically
detected suspicious entry points
★ Detect and recover from:
★ Memory errors like NULL pointer accesses
★ Structural errors like malformed structures
★ Processor exceptions like divide by zero, stack corruption

Fault model in FGFT
14
★ Provide fault tolerance to speciﬁc driver entry points
network
driver
network
card
probe
xmit
conﬁg
★ Can be applied to untested code, statically and dynamically
detected suspicious entry points
★ Detect and recover from:
★ Memory errors like NULL pointer accesses
★ Structural errors like malformed structures
★ Processor exceptions like divide by zero, stack corruption

Transactional support through code generation
15
★ Generate code to run driver invocations on a separate
stack with a copy of parameters
★ Reduce copy overhead by copying only referenced fields
in driver and kernel structures to a range table
★ Instrument all memory references in SFI module to
compare accesses against copied fields in range table
Range Table
SFI
network
driver
network
driver
get ringparam
netdev->priv->tx_ring
netdev->priv->rx_ring
Address Access rights
0xffffa000 Read
0xffffa008 Write
0xffffa00a Read

Resource access during isolated execution
16
★ Device registers and I/O memory
★ Grant drivers full access to devices
★ Restore device checkpoint in case of failure
★ Locks: Spinlocks and semaphores
★ Grants read access to locks
★ Maintain kernel log of locks acquired
★ Release locks at the end of entry point/failures
★ Kernel resources like memory
★ All allocations generate range table entry
★ Maintain kernel log of all acquired resources
★ Free resources on failures
0xffffa000 Read
0xffffa008 Write
0xffffa00a Read
Range Tablemalloc
()

Outline
17
Introduction
Conclusion

Checkpointing drivers is hard
★Existing mechanisms limited to capturing memory state
18
network
driver
network
card

18
network
driver
network
card
checkpoint

18
network
driver
network
card
checkpoint
★ Device state is not captured
★ Device conﬁguration space

18
network
driver
network
card
checkpoint
★ Internal device registers and counters

18
network
driver
network
card
checkpoint
★ Memory buffer addresses used for DMA

18
network
driver
network
card
checkpoint
★ Memory buffer addresses used for DMA
★ Unique for every class, bus and vendor

Device checkpoint/restore from PM code
19
Save conﬁg state
Save register state
Disable device
Copy-out s/w state
Suspend device
Restore conﬁg state
Restore register state
Restore s/w state &
reset
Re-attach/Enable
device
Device Ready
Suspend Resume

19
Save conﬁg state
Save register state
Copy-out s/w state
Suspend device
Restore s/w state &
reset
Re-attach/Enable
device
Device Ready
Suspend Resume

19
Save conﬁg state
Save register state
Copy-out s/w state
Restore s/w state &
reset
Re-attach/Enable
device
Device Ready
Suspend Resume

19
Save conﬁg state
Save register state
Copy-out s/w state
Restore s/w state &
reset
Re-attach/Enable
device
Device Ready
ResumeCheckpoint

19
Save conﬁg state
Save register state
Copy-out s/w state
Restore s/w state &
reset
Re-attach/Enable
device
ResumeCheckpoint

19
Save conﬁg state
Save register state
Copy-out s/w state
Restore s/w state &
reset
ResumeCheckpoint

19
Save conﬁg state
Save register state
Copy-out s/w state
Restore s/w state &
reset
RestoreCheckpoint

19
Save conﬁg state
Save register state
Copy-out s/w state
Restore s/w state &
reset
Suspend/resume code provides device
checkpoint functionality
RestoreCheckpoint

Intuition with power management
20

20
★ Intuition: Power management code captures device
speciﬁc state for every driver
★ Our study: Present in 76% of all common classes

20
Linux
driver Device

20
Linux
driver Device
RAM

20
Linux
driver Device
RAM
suspend()

20
Linux
driver Device
RAM
suspend()
resume()

20
★ Refactor power management code for device checkpoints
★ Correct: Developer captures unique device semantics
★ Fast: Avoids probe and latency critical for applications
Linux
driver Device
RAM
suspend()
resume()

Synergy of isolation and recovery
21
★ Goal: Improve driver recovery with minor changes to drivers
★ Solution: Run drivers as transactions using device checkpoints

21
C R
★ Developers export
checkpoint/restore
in drivers
Device state

21
C R
checkpoint/restore
in drivers
Device state Driver state
★ Run drivers invocations as
memory transactions
★ Use source transformation
to copy parameters and
run on separate stack
SFI
network
driver
network
driver

21
C R
checkpoint/restore
in drivers
Device state Driver state
★ Run drivers invocations as
memory transactions
★ Use source transformation
to copy parameters and
run on separate stack
SFI
network
driver
network
driver
Execution model
★ Checkpoint device
★ Execute driver code as
memory transactions
★ On failure, rollback
and restore device
★ Re-use existing device
locks in the driver

SFI
network
driver
Example transactional execution
22
network
driver
probe
xmit
get conﬁg
get ringparam

SFI
network
driver
22
network
driver
probe
xmit
get conﬁg
get ringparam netdev

SFI
network
driver
22
network
driver
probe
xmit
get conﬁg
get ringparam
C
netdev

SFI
network
driver
22
network
driver
probe
xmit
get conﬁg
get ringparam
netdev->priv->rx_ringC
netdev

SFI
network
driver
0xffffa000 Read
0xffffa008 Write
0xffffa00a Read
22
network
driver
probe
xmit
get config
get ringparam
Range Table
C
netdev

SFI
network
driver
0xffffa000 Read
0xffffa008 Write
0xffffa00a Read
22
network
driver
probe
xmit
get config
get ringparam
Range Table
result
Kernel
Log
alloc
C
netdev

SFI
network
driver
Example failed transaction
23
network
driver
probe
xmit
get conﬁg
get ringparam

SFI
network
driver
23
network
driver
probe
xmit
get conﬁg
get ringparam netdev

SFI
network
driver
23
network
driver
probe
xmit
get conﬁg
get ringparam
C
netdev

SFI
network
driver
23
network
driver
probe
xmit
get conﬁg
get ringparam
netdev->priv->rx_ringC
netdev

SFI
network
driver
0xffffa000 Read
0xffffa008 Write
0xffffa00a Read
23
network
driver
probe
xmit
get config
get ringparam
Range Table
C
netdev

SFI
network
driver
0xffffa000 Read
0xffffa008 Write
0xffffa00a Read
23
network
driver
probe
xmit
get config
get ringparam
Range Table
Kernel
Log
alloc
C
netdev

SFI
network
driver
0xffffa000 Read
0xffffa008 Write
0xffffa00a Read
23
network
driver
probe
xmit
get config
get ringparam
Range Table
err
C
netdev

SFI
network
driver
0xffffa000 Read
0xffffa008 Write
0xffffa00a Read
23
network
driver
probe
xmit
get config
get ringparam
Range Table
err
C
R
netdev

SFI
network
driver
0xffffa000 Read
0xffffa008 Write
0xffffa00a Read
23
network
driver
probe
xmit
get config
get ringparam
Range Table
err
C
R
FGFT provides transactional
execution of driver entry points
netdev

Outline
24
Introduction
Evaluation & Conclusions

Outline
25
Introduction
Evaluation & Conclusion

Recovery speedup
Driver Class Bus Restart
recovery
FGFT
recovery
Speedup
8139too net PCI 0.31s 70μs 4400
e1000 net PCI 1.80s 295ms 6
r8169 net PCI 0.12s 40μs 3000
pegasus net USB 0.15s 5ms 30
ens1371 sound PCI 1.03s 115ms 9
psmouse input serio 0.68s 410ms 1.65
26
FGFT provides signiﬁcant
speedup in driver recovery

Static and dynamic fault injection
Driver Injected
Faults
Benign
Faults
Native
Crashes
FGFT
Crashes
8139too 43 0 43 NONE
e1000 47 0 47 NONE
r8169 36 0 36 NONE
pegasus 34 1 33 NONE
ens1371 22 1 21 NONE
psmouse 46 0 46 NONE
TOTAL 258 2 256 NONE
27
FGFT survives multiple static and dynamic faults

Programming eﬀort
Driver LOC Isolation annotationsIsolation annotations Recovery additionsRecovery additions
Driver
annotations
Kernel
annotations
LOC Moved LOC
Added
8139too 1, 904 15 20 26 4
e1000 13, 973 32 32 10
r8169 2, 993 10 17 5
pegasus 1, 541 26 12 22 5
ens1371 2, 110 23 66 16 6
psmouse 2, 448 11 19 19 6
28
FGFT requires limited programmer effort
and needs only 38 lines of new kernel code

Throughput with isolation and recovery
Native
FGFT-‐oﬀ-‐I/O
FGFT-‐I/O-‐1/2
FGFT-‐I/O-‐all
netperf on Intel quad-core machines
29

0
25
50
75
100
Throughput%age(Baseline844Mbps)
e1000 Network Card
Native
FGFT-‐oﬀ-‐I/O
FGFT-‐I/O-‐1/2
FGFT-‐I/O-‐all
29

0
25
50
75
100
100
e1000 Network Card
Native
FGFT-‐oﬀ-‐I/O
FGFT-‐I/O-‐1/2
FGFT-‐I/O-‐all
29
CPU: 2.4%

0
25
50
75
100
100 100
e1000 Network Card
Native
FGFT-‐oﬀ-‐I/O
FGFT-‐I/O-‐1/2
FGFT-‐I/O-‐all
29
CPU: 2.4% 2.4%

0
25
50
75
100
100 100
96
e1000 Network Card
Native
FGFT-‐oﬀ-‐I/O
FGFT-‐I/O-‐1/2
FGFT-‐I/O-‐all
29
CPU: 2.4% 2.4% 2.9%

0
25
50
75
100
100 100
96 93
e1000 Network Card
Native
FGFT-‐oﬀ-‐I/O
FGFT-‐I/O-‐1/2
FGFT-‐I/O-‐all
29
CPU: 2.4% 2.4% 2.9% 3.4%

0
25
50
75
100
100 100
96 93
e1000 Network Card
Native
FGFT-‐oﬀ-‐I/O
FGFT-‐I/O-‐1/2
FGFT-‐I/O-‐all
29
CPU: 2.4% 2.4% 2.9% 3.4%
FGFT can isolate and recover high bandwidth devices
at low overhead without adding kernel subsystems

Summary
30

Summary
30
★ Fine-Grained Fault tolerance based on a pay-
as-you go model
★ Provides fault tolerance at incremental
performance costs and programmer efforts
★ Introduces fast checkpointing for drivers
★ Device checkpoints average ~20micros
★ Reduces recovery time signiﬁcantly
★ Should be explored in other domains apart from
fault tolerance like fast reboot, upgrade etc.

Questions
Asim Kadav
★ http://cs.wisc.edu/~kadav
★ kadav@cs.wisc.edu

Fine-grained fault tolerance using device checkpoints

Recommended

More Related Content

What's hot (20)

Viewers also liked (12)

Similar to Fine-grained fault tolerance using device checkpoints (20)

Recently uploaded (20)

Fine-grained fault tolerance using device checkpoints