Recovery

Recovery

Computer system recovery:

Restore the system to a normal operational state
Process recovery:
Reclaim resources allocated to process,
Undo modification made to databases, and
Restart the process
Or restart process from point of failure and resume
execution
Distributed process recovery (cooperating processes):
Undo effect of interactions of failed process with other
cooperating processes.
Replication (hardware components, processes, data):
Main method for increasing system availability
System:
Set of hardware and software components
Designed to provide a specified service (I.e. meet a set of
requirements)

Recovery (cont.)
System failure:
System does not meet requirements, i.e.does not perform its
services as specified

Error could lead to system failure

Erroneous System State:
State which could lead to a system failure by a sequence of valid
state transitions
Error: the part of the system state which differs from its intended
value

Error is a manifestation of a fault

Fault:
3

Anomalous physical condition, e.g. design errors, manufacturing

problems, damage, external disturbances.

Classification of failures
Process failure:
Behaviour: process causes system state to deviate

from specification (e.g. incorrect computation,

process stop execution)
Errors causing process failure: protection violation,
deadlocks, timeout, wrong user input, etc
Recovery: Abort process or Restart process from
prior state
System failure:
Behaviour: processor fails to execute
Caused by software errors or hardware faults
(CPU/memory/bus// failure)
Recovery: system stopped and restarted in correct
state
Assumption: fail-stop processors, i.e. system stops
execution, internal state is lost
4

 Secondary Storage Failure:

Behaviour: stored data cannot be accessed
Errors causing failure: parity error, head crash,

etc.
Recovery/Design strategies:
Reconstruct content from archive + log of
activities
Design mirrored disk system
Communication Medium Failure:
Behaviour: a site cannot communicate with
another operational site
Errors/Faults: failure of switching nodes or
communication links
Recovery/Design Strategies: reroute, errorresistant communication protocols
5

Backward and Forward Error

Recovery
Failure recovery: restore an erroneous state to an

error-free state
Approaches to failure recovery:
Forward-error recovery:
Remove errors in process/system state (if errors
can be completely assessed)
Continue process/system forward execution
Backward-error recovery:
Restore process/system to previous error-free
state and restart from there

 Comparison: Forward vs. Backward error recovery

Backward-error recovery

(+) Simple to implement

(+) Can be used as general recovery mechanism
(-) Performance penalty
(-) No guarantee that fault does not occur again
(-) Some components cannot be recovered
Forward-error Recovery
(+) Less overhead
(-) Limited use, i.e. only when impact of faults
understood
(-) Cannot be used as general mechanism for
error recovery

Backward-Error Recovery: Basic approach

Principle: restore process/system to a known, error-

free recovery point/ checkpoint.

System model:
CPU
secondary
storage

Main memory

Bring object to MM
to be accessed

Store logs and

recovery points

Write object back

if modified

Approaches:

(1) Operation-based approach

(2) State-based approach
8

stable
storage

Storage that
maintains
information in
the event of
system failure

(1) The Operation-based Approach

Principle:
Record all changes made to state of process (audit

trail or log) such that process can be returned to

a previous state

Example: A transaction based environment where

transactions update a database

It is possible to commit or undo updates on a
per-transaction basis
A commit indicates that the transaction on the
object was successful and changes are
permanent

(1.a) Updating-in-place
Principle: every update (write) operation to an object
creates a log in stable storage that can be used to
undo and redo the operation
Log content: object name, old object state, new
object state
Implementation of a recoverable update operation:
Do operation:update object and write log record
Undo operation: log(old) -> object (undoes the
action performed by a do)
Redo operation: log(new) -> object (redoes the
action performed by a do)
Display operation: display log record (optional)
Problem: a do cannot be recovered if system
crashes after write object but before log record write
(1.b) The write-ahead log protocol
Principle: write log record before updating object
10

(2) State-based Approach

Principle: establish frequent recovery points or

checkpoints saving the entire state of process

Actions:
Checkpointing or taking a checkpoint: saving
process state
Rolling back a process: restoring a process to a
prior state
Note: A process should be rolled back to the most
recent recovery point to minimize the overhead
and delays in the completion of the process

11

 Shadow Pages: Special case of state-based

approach
Only a part of the system state is saved to
minimize recovery
When an object is modified, page containing
object is first copied on stable storage
(shadow page)
If process successfully commits: shadow
page discarded and modified page is made
part of the database
If process fails: shadow page used and the
modified page discarded

12

Recovery in concurrent systems

Issue: if one of a set of cooperating processes fails

and has to be rolled back to a recovery point, all

processes it communicated with since the recovery
point have to be rolled back.
Conclusion: In concurrent and/or distributed systems
all cooperating processes have to establish recovery
points

13

 Orphan messages and the domino effect

X

x1

x2
m

Y
Z

x3

y1
z1

y2
z2

Time

Case 1: failure of X after x3 : no impact on Y or Z

Case 2: failure of Y after sending msg. m
Y rolled back to y2
m orphan massage
X rolled back to x2
Case 3: failure of Z after z2
Y has to roll back to y1
14

X has to roll back to x1

Z has to roll back to z1

Domino Effect

Lost messages
X

x1
m

Failure

y1
Time

Assume that x1 and y1 are the only recovery points

for processes X and Y, respectively

Assume Y fails after receiving message m
Y rolled back to y1, X rolled back to x1
Message m is lost

15

Problem of livelock

Livelock: case where a single failure can cause an infinite

number of rollbacks
X
Y

x1

n1
m1

y1

(a)

X
Y

(a)

(b)

16

x1
y1

n1

Failure
Time

n2
m2

2nd roll back

(b)

Time

Process Y fails before receiving message n1 sent by X

Y rolled back to y1, no record of sending message m1, causing X
to roll back to x1
When Y restarts, sends out m2 and receives n1 (delayed)
When X restarts from x1, sends out n2 and receives m2
Y has to roll back again, since there is no record of n1 being sent
This cause X to be rolled back again, since it has received m2
and there is no record of sending m2 in Y
The above sequence can repeat indefinitely

Consistent set of checkpoints

Checkpointing in distributed systems requires that

all processes (sites) that interact with one another
establish periodic checkpoints

All the sites save their local states: local

checkpoints

All the local checkpoints, one from each site,

collectively form a global checkpoint

The domino effect is caused by orphan messages,

which in turn are caused by rollbacks

1.

Strongly consistent set of checkpoints

17

Establish a set of local checkpoints (one for each

process in the set) such that no information flow
takes place (i.e., no orphan messages) during
the interval spanned by the checkpoints

Consistency of Checkpoint
2. Consistent set of checkpoints

Similar to the consistent global state

Each message that is received in a

checkpoint (state) should also be recorded as
sent in another checkpoint (state)
x1

x2

[
y1

need to deal with

lost messages

y2

Strongly consistent
z1

consistent
z2

Checkpoint/Recovery Algorithm

Synchronous

with global synchronization at checkpointing

Asynchronous

without global synchronization at checkpointing

Preliminary (Assumption)
Goal

Synchronous Checkpointing

To make a consistent global checkpoint

Assumptions

Communication channels are FIFO

No partition of the network
End-to-end protocols cope with message loss
due to rollback recovery and communication
failure
No failure during the execution of the algorithm

Preliminary (Two types of

checkpoint)
Synchronous Checkpointing

tentative checkpoint :

a temporary checkpoint
a candidate for permanent checkpoint

permanent checkpoint :

a local checkpoint at a process

a part of a consistent global checkpoint

Checkpoint Algorithm
Synchronous Checkpointing
Algorithm
1.
2.
3.
4.

5.
6.

an initiating process (a single process that invokes this

algorithm) takes a tentative checkpoint
it requests all the processes to take tentative checkpoints
it waits for receiving from all the processes whether taking a
tentative checkpoint has been succeeded
if it learns all the processes has succeeded, it decides all
tentative checkpoints should be made permanent; otherwise,
should be discarded.
it informs all the processes of the decision
The processes that receive the decision act accordingly

Supplement
Once a process has taken a tentative checkpoint, it shouldnt
send messages until it is informed of initiators decision.

Diagram of Checkpoint
Algorithm
Synchronous Checkpointing
Tentative
checkpoint

Initiator

decide to commit

|
request to
take a
tentative
checkpoint

[
consistent global checkpoint

OK

permanent checkpoint

[
Unnecessary checkpoint

consistent global checkpoint

Optimized Algorithm
Synchronous Checkpointing
Each message is labeled by order of sending
Labeling Scheme

[
x2

x3

: smallest label
y1
y2
: largest label
[
Y
y2
last_label_rcvdX[Y] :
the last message that X received from Y after X has taken its
last permanent or tentative checkpoint. if not exists, is in it.
first_label_sentX[Y] x2
:
the first message that X sent to Y after X took its last
permanent or tentative checkpoint . if not exists, is in it.
ckpt_cohortX :
the set of all processes that may have to take checkpoints
when X decides to take a checkpoint.
Checkpoint request need to be sent to only the processes
included in ckpt_cohort

Optimized Algorithm
Synchronous Checkpointing

ckpt_cohortX : { Y | last_label_rcvdX[Y] > }

Y takes a tentative checkpoint only if
last_label_rcvdX[Y] >= first_label_sentY[X]
>
last_label_rcvdX[Y]
X

[
first_label_sentY[X]

Optimized Algorithm
Synchronous Checkpointing
Algorithm
1.
2.
3.
4.
5.
6.
7.

an initiating process takes a tentative checkpoint

it requests p ckpt_cohort to take tentative
checkpoints ( this message includes
last_label_rcvd[reciever] of sender )
if the processes that receive the request need to take a
checkpoint, they do the same as 1.2.; otherwise, return
OK messages.
they wait for receiving OK from all of p ckpt_cohort
if the initiator learns all the processes have succeeded,
it decides all tentative checkpoints should be made
permanent; otherwise, should be discarded.
it informs p ckpt_cohort of the decision
The processes that receive the decision act accordingly

Correctness
Synchronous Checkpointing

A set of permanent checkpoints taken by this

algorithm is consistent

No process sends messages after taking a

tentative checkpoint until the receipt of the
decision
New checkpoints include no message from the
processes that dont take a checkpoint
The set of tentative checkpoints is fully either
made to permanent checkpoints or discarded.

Recovery Algorithm
Synchronous Recovery
Labeling Scheme
: smallest label
: largest label
last_label_rcvdX[Y] :
the last message that X received from Y after X has taken its
last permanent or tentative checkpoint. If not exists, is in it.
first_label_sentX[Y] :
the first message that X sent to Y after X took its last
permanent or tentative checkpoint . If not exists, is in it.
roll_cohortX :
the set of all processes that may have to roll back to the
latest checkpoint when process X rolls back.
last_label_sentX[Y] :
the last message that X sent to Y before X takes its latest
permanent checkpoint. If not exist, is in it.

Recovery Algorithm
Synchronous Recovery

roll_cohortX = { Y | X can send messages to Y }

Y will restart from the permanent checkpoint
only if
last_label_rcvdY[X] > last_label_sentX[Y]

Recovery Algorithm
Synchronous Recovery
Algorithm
1.
2.
3.
4.
5.
6.

an initiator requests p roll_cohort to prepare to

rollback ( this message includes
last_label_sent[reciever] of sender )
if the processes that receive the request need to
rollback, they do the same as 1.; otherwise, return OK
message.
they wait for receiving OK from all of p ckpt_cohort.
if the initiator learns p roll_cohort have succeeded,
it decides to rollback; otherwise, not to rollback.
it informs p roll_cohort of the decision
the processes that receive the decision act
accordingly

Diagram of Synchronous
Recovery
x1

x2

Failure

X
X
Y

y1
z1

Unnecessary Rollback
31

y2
z2

Drawbacks of Synchronous Approach

Additional messages are exchanged

Synchronization delay
An unnecessary extra load on the system if
failure rarely occurs

Asynchronous Checkpoint
Characteristic

Each process takes checkpoints independently

No guarantee that a set of local checkpoints is
consistent
A recovery algorithm has to search consistent
set of checkpoints
No additional message
No synchronization delay
Lighter load during normal excution

Preliminary (Assumptions)
Asynchronous Checkpoint / Recovery

Goal
To find the latest consistent set of checkpoints

Assumptions

Communication channels are FIFO

Communication channels are reliable
The underlying computation is eventdriven

Preliminary (Two types of

log)
Asynchronous Checkpoint / Recovery

save an event on the memory at receipt of

messages (volatile log)
volatile log periodically flushed to the disk
(stable log) checkpoint

volatile log :
quick access
lost if the corresponding processor fails

stable log :
slow access
not lost even if processors fail

Preliminary (Definition)
Asynchronous Checkpoint / Recovery

Definition
CkPti : the checkpoint (stable log) that i rolled back to
when failure occurs
RCVDij (CkPti / e ) :
the number of messages received by processor i from
processor j, per the information stored in the checkpoint
CkPti or event e.

SENTij(CkPti / e ) :
the number of messages sent by processor i to processor
j, per the information stored in the checkpoint CkPti or
event e

Recovery Algorithm
Asynchronous Checkpoint / Recovery
Algorithm
1.
2.
3.
4.
5.

6.

When one process crashes, it recovers to the latest

checkpoint CkPt.
It broadcasts the message that it had failed. Others
receive this message, and rollback to the latest event.
Each process sends SENT(CkPt) to neighboring
processes
Each process waits for SENT(CkPt) messages from
every neighbor
On receiving SENTji(CkPtj) from j, if i notices RCVDij
(CkPti) > SENTji(CkPtj), it rolls back to the event e
such that RCVDij (e) = SENTji(e),
repeat 3,4,and 5 N times (N is the number of
processes)

Asynchronous Recovery
X:Y
X

Ex0

x1

Ex1

Ex2

Ex3

3 <= 2
2
(Y,2)

Ey0

Ey1

Ey2

y1

Ey3

Ez1

Y:X
1 <= 2

(X,0)
(Y,1)

Ez0

(X,2) (Z,0)

X:Z
0 <= 0
Y:Z
1 <= 1

(Z,1)

Ez2

z1

RCVDij (CkPti) <= SENTji(CkPtj)

Z:X
0 <= 0

Z:Y
1
2 <= 1