Slides for Chapter 15:

Coordination and Agreement

From Coulouris, Dollimore, Kindberg and

Blair
Distributed Systems:
Concepts and Design
Edition 5, © Addison-Wesley 2012
 Overview of Chapter

• Introduction
• Distributed mutual exclusion
• Elections
• Coordination and agreement in group communication (skip)
• Consensus and related problems (skip)

2
 Introduction

Covers two areas:

• Coordinating actions in a distributed system
• Distributed processes agreeing on a result value

• Assumes reliable communication channels for simplicity (failure is

masked by a reliable communication protocol)
• Detecting that a process has failed can be reliable or unreliable
• Use timouts
• Unreliable: replies unsuspected or suspected
• Reliable: replies unsuspected or failed

3
Figure 15.1
A network partition

Cras hed
router

Instructor’s Guide for Coulouris, Dollimore, Kindberg and Blair, Distributed Systems: Concepts and Design Edn. 5
© Pearson Education 2012
 Overview of Chapter

• Introduction
• Distributed mutual exclusion
• Elections
• Coordination and agreement in group communication
• Consensus and related problems

5
 Distributed mutual exclusion

Known as critical section problem:

• Only one process can be in critical section – the process with the
token that allows access to the resource
• Operations include the following:
• enter() – requests access; can be granted or blocked
• resourceAccess() – access resource in critical section
• exit() – leave critical section – other processes may now enter

Conditions:
• ME1 (safety) – at most one process in critical section
• ME2 (liveness) – requests eventually succeed
• ME3 (ordering) – requests follow happened-before relationship

6
 Distributed mutual exclusion

Various algorithms:
• Central server algorithm
• Ring-based algorithm
• Algorithm using multicast and logical clocks
• Voting algorithm
• Others

7
Figure 15.2
Server managing a mutual exclusion token for a set of processes

Server
Queue of
requests
4

2
3. Grant
token

1. Request
token
2. Release p4
p token
1

p2 p
3

Instructor’s Guide for Coulouris, Dollimore, Kindberg and Blair, Distributed Systems: Concepts and Design Edn. 5
© Pearson Education 2012
Figure 15.3
A ring of processes transferring a mutual exclusion token

p
1 p
2

pn

p
3

p
4

Token

Instructor’s Guide for Coulouris, Dollimore, Kindberg and Blair, Distributed Systems: Concepts and Design Edn. 5
© Pearson Education 2012
Figure 15.4
Ricart and Agrawala’s algorithm

On initialization
state := RELEASED;
To enter the section
state := WANTED;
Multicast request to all processes; request processing deferred here
T := request’s timestamp;
Wait until (number of replies received = (N – 1));
state := HELD;

On receipt of a request <Ti, pi> at pj (i ≠ j)

if (state = HELD or (state = WANTED and (T, pj) < (Ti, pi)))
then
queue request from pi without replying;
else
reply immediately to pi;
end if
To exit the critical section
state := RELEASED;
reply to any queued requests;

Instructor’s Guide for Coulouris, Dollimore, Kindberg and Blair, Distributed Systems: Concepts and Design Edn. 5
© Pearson Education 2012
Figure 15.5
Multicast synchronization

41
41 p
3
p Reply
1

Reply 34
Reply
34
41

p 34
2

Instructor’s Guide for Coulouris, Dollimore, Kindberg and Blair, Distributed Systems: Concepts and Design Edn. 5
© Pearson Education 2012
Figure 15.6
Maekawa’s algorithm – part 1

On initialization For pi to exit the critical section

state := RELEASED; state := RELEASED;
voted := FALSE; Multicast release to all processes in Vi;
For pi to enter the critical section On receipt of a release from pi at pj
state := WANTED; if (queue of requests is non-empty)
Multicast request to all processes in Vi; then
Wait until (number of replies received = K); remove head of queue – from pk , say;
state := HELD; send reply to pk ;
voted := TRUE;
On receipt of a request from pi at pj else
if (state = HELD or voted = TRUE) voted := FALSE;
then end if
queue request from pi without replying;
else
send reply to pi;
voted := TRUE;
end if

Instructor’s Guide for Coulouris, Dollimore, Kindberg and Blair, Distributed Systems: Concepts and Design Edn. 5
© Pearson Education 2012
 Overview of Chapter

• Introduction
• Distributed mutual exclusion
• Elections
• Coordination and agreement in group communication
• Consensus and related problems

13
 Elections

Election algorithms:
• Used to choose a particular process for a role – for example, to
choose a central server for distributed algorithms that require a
central server
• Process can call an election; for example, if it detects that central
server has failed

Requirements:
• E1: (safety) only one process is elected – each participant sets
electedi either to P or to undefined if it does not know the elected
process yet (P will be the participant with the largest process id)
• E2: (liveness) all processes participate and either set electedi to the
elected process or crash

14
 Elections

Election algorithms:
• Ring-based algorithm
• Bully algorithm

15
Figure 15.7
A ring-based election in progress

3
17

24

15

28 24

Note: The election was started by process 17.

The highest process identifier encountered so far is 24.
Participant processes are shown in a darker colour
Instructor’s Guide for Coulouris, Dollimore, Kindberg and Blair, Distributed Systems: Concepts and Design Edn. 5
© Pearson Education 2012
 Figure 15.8
The bully algorithm

The election of coordinator p2, election

after the failure of p4 and then p3 election
C
Stage 1
answer
p p p p
1 2 3 4
answer
election

election election C
Stage 2
answer
p1 p p p
2 3 4

timeout
Stage 3
p p p p
1 2 3 4

Eventually.....
coordinator
C
Stage 4
p p p p
1 2 3 4

Instructor’s Guide for Coulouris, Dollimore, Kindberg and Blair, Distributed Systems: Concepts and Design Edn. 5
© Pearson Education 2012
 Overview of Chapter

• Introduction
• Distributed mutual exclusion
• Elections
• Coordination and agreement in group communication (skip)
• Consensus and related problems

18
Figure 15.9
Reliable multicast algorithm

Instructor’s Guide for Coulouris, Dollimore, Kindberg and Blair, Distributed Systems: Concepts and Design Edn. 5
© Pearson Education 2012
Figure 15.10
The hold-back queue for arriving multicast messages

Message
proc es sing

deliv er
Hold-bac k
queue Deliv ery queue

When deliv ery

guarantees are
met
Incoming
mes sages
Instructor’s Guide for Coulouris, Dollimore, Kindberg and Blair, Distributed Systems: Concepts and Design Edn. 5
© Pearson Education 2012
Figure 15.11
Total, FIFO and causal ordering of multicast messages

T1
Notice the consistent T2

ordering of totally ordered

messages T1 and T2,
the FIFO-related messages
F1 and F2 and the causally F1
related messages C1 and C3
– and the otherwise arbitrary F2 F3

delivery ordering of
messages.
Time

C1

C2
C3

P1 P2 P3
Instructor’s Guide for Coulouris, Dollimore, Kindberg and Blair, Distributed Systems: Concepts and Design Edn. 5
© Pearson Education 2012
Figure 15.12
Display from bulletin board program

Bulletin board: os.interesting

Item From Subject

23 A.Hanlon Mach
24 G.Joseph Microkernels
25 A.Hanlon Re: Microkernels
26 T.L’Heureux RPC performance
27 M.Walker Re: Mach
end

Instructor’s Guide for Coulouris, Dollimore, Kindberg and Blair, Distributed Systems: Concepts and Design Edn. 5
© Pearson Education 2012
Figure 15.13
Total ordering using a sequencer

Instructor’s Guide for Coulouris, Dollimore, Kindberg and Blair, Distributed Systems: Concepts and Design Edn. 5
© Pearson Education 2012
Figure 15.14
The ISIS algorithm for total ordering

P2
1 Message
3
22 P4
1

3 Agreed Seq
1
2 P1
3

P3

Instructor’s Guide for Coulouris, Dollimore, Kindberg and Blair, Distributed Systems: Concepts and Design Edn. 5
© Pearson Education 2012
Figure 15.15
Causal ordering using vector timestamps

Instructor’s Guide for Coulouris, Dollimore, Kindberg and Blair, Distributed Systems: Concepts and Design Edn. 5
© Pearson Education 2012
Figure 15.16
Consensus for three processes

d1 :=proceed d2 :=proceed
P1 P2

v1 =proceed v2=proceed
1

Consens us algorithm

v3=abort

P3 (c rashes )

Instructor’s Guide for Coulouris, Dollimore, Kindberg and Blair, Distributed Systems: Concepts and Design Edn. 5
© Pearson Education 2012
Figure 15.17
Consensus in a synchronous system

Instructor’s Guide for Coulouris, Dollimore, Kindberg and Blair, Distributed Systems: Concepts and Design Edn. 5
© Pearson Education 2012
Figure 15.18
Three Byzantine generals

p1 (Commander) p1 (Commander)

1:v 1:v 1:w 1:x

2:1:v 2:1:w
p2 p3 p2 p3
3:1:u 3:1:x

Faulty processes are shown coloured

Instructor’s Guide for Coulouris, Dollimore, Kindberg and Blair, Distributed Systems: Concepts and Design Edn. 5
© Pearson Education 2012
Figure 15.19
Four Byzantine generals

p1 (Commander) p1 (Commander)

1:v 1:v 1:u 1:w

1:v 1:v
2:1:v 2:1:u
p2 3:1:u p3 p2 3:1:w p3
4:1:v 4:1:v 4:1:v 4:1:v
2:1:v 3:1:w 2:1:u 3:1:w

p4 p4
Faulty processes are shown coloured

Instructor’s Guide for Coulouris, Dollimore, Kindberg and Blair, Distributed Systems: Concepts and Design Edn. 5
© Pearson Education 2012