Multicast

 A process group consists of several

processes
A process might belong to multiple groups
 A multicast operation sends a message to
all the members in a group.
 The aim of a multicast operations is, for
each process in a group, the process should
receive a copy of the message sent to the
group.
 Various delivery guarantees might be required.

1
 Ordered multicast

 FIFO ordering: if a process multicasts m

and then multicasts m’, then all processes
in the group that delivers m’ will deliver m
before m’
 Causal ordering: if multicast m happens
before multicast m’, then all the processes
in the group will deliver m before m’
 Total ordering: if a process delivers
message m before m’, then all other
processes will deliver m before m’

2
T1
T2

Time

P1 P2 P3

3
T1
T2

Time

P1 P2 P3

4
T1
T2

Time

P1 P2 P3

5
T1
T2

F1
F2 F3

Time

P1 P2 P3

6
T1
T2

F1
F2 F3

Time

P1 P2 P3

7
T1
T2

F1
F2 F3

Time

P1 P2 P3

8
T1
T2

F1
F2 F3

Time
C1
C2 C3

P1 P2 P3

9
T1
T2

F1
F2 F3

Time
C1
C2 C3

P1 P2 P3

10
T1
T2

F1
F2 F3

Time
C1
C2 C3

P1 P2 P3

11
 Implementing FIFO ordering
 A process assigns a sequence number to the message
that it broadcasts to the group
 Each process maintains some variables to record the
number of messages that are sent to/received from
the other processes
 S is the sequence number assigned to the broadcast message
 Rq is the sequence number of the last message that is
broadcast by process q
 When broadcast a message:
 Assign S as the sequence number of the message
 Increment S
 When receives a message from process q
 If the sequence number of the message is Rq +1, deliver the
message.
 Otherwise, place the message in a waiting queue.
• The message will be delivered after all the preceding messages
are delivered.
12
Implementing total ordering

P2
1 Message

P4
1

1
P1

P3

13
Implementing total ordering

P2
1 Message

2 ed TS P4
r o pos
1 2 P

1
2 P1

P3

14
Implementing total ordering

P2
1 Message
3
2 ed Seq P4
r opos
1 2 P

3 Agreed TS
1
2 P1
3

P3

15
 h1
Holding queue: Delivery queue:

S1 m1

h2
Holding queue: Delivery queue:
m2
S2

16
 h1
Holding queue: Delivery queue:
m1 (1, S1)
S1 m1

h2
Holding queue: Delivery queue:
m2
S2 m2 (1, S2)

17
 h1
S1 ) Holding queue: Delivery queue:
( 1,
m1 m1 (1, S1)
S1 m1

h2
Holding queue: Delivery queue:
m2
S2 )
S2 (1, m2 (1, S2)
m2

18
 h1
Holding queue: Delivery queue:
m1 (1, S1)
S1 m2 (2, S2)
m1 (1, S1)

h2
Holding queue: Delivery queue:

S2 m2 (1, S2)
m1 (2, S1)
m2 (1, S2)

19
 h1
Holding queue: Delivery queue:

S2)
m1 (1, S1)
S1

(2,
m2 (2, S2)

m2
m1 (1, S1)

h2
m
1
(2

Holding queue: Delivery queue:

,S
1)

S2 m2 (1, S2)
m1 (2, S1)
m2 (1, S2)

20
 h1
Holding queue: Delivery queue:
m1 (1, S1)
S1 m2 (2, S2)
m1 (1, S1)
m1 (2, S1)

h2
Holding queue: Delivery queue:

S2 m2 (1, S2)
m1 (2, S1)
m2 (1, S2)
m2 (2, S2)

21
 h1
Holding queue: Delivery queue:
m1 (1, S1)
S1 m1 (2, S1) m2 (2, S2)

h2
Holding queue: Delivery queue:
m2 (2, S2)
S2 m2 (1, S2)
m1 (2, S1)

22
 h1
Holding queue: Delivery queue:
m1 (2, S1)*
S1 m1 (2, S1) m2 (2, S2)

h2
Holding queue: Delivery queue:
m2 (2, S2)
S2 m1 (2, S1)
m2 (2, S2)*

23
 h1
Holding queue: Delivery queue:
m2 (2, S2) m1 (2, S1)
S1 m1 (2, S1)

h2
Holding queue: Delivery queue:
m2 (2, S2)
S2 m1 (2, S1)
m2 (2, S2)*

24
 h1
Holding queue: Delivery queue:
m2 (2, S2)* m1 (2, S1)
S1

h2
Holding queue: Delivery queue:

S2 m1 (2, S1)*
m2 (2, S2)*

25
 h1
Holding queue: Delivery queue:

m1 (2, S1)
S1 m2 (2, S2)

h2
Holding queue: Delivery queue:

S2 m1 (2, S1)
m2 (2, S2)

26
 The sender transmits msg to its destinations.
 Each recipient adds the message to the priority
queue associated with label, tagging it as
undeliverable. It assigns this message a priority
larger than the priority of any message that was
placed in the queue, with the process ID of the
recipient as a suffix. It then informs the sender of
the priority that it assigned to the message.
 The sender collects responses from recipients that
remain operational. It then computes the maximum
value of all the priorities it received, and sends this
value back to the recipients.
 The recipients change the priority of the message to
the value they receive from the sender, tag the
message as deliverable, and re-sort their priority
queues. They then transfer messages from the
priority queue to the delivery queue in order of
increasing priority, until the priority queue becomes
empty or the message with the lowest priority is
undeliverable.
27
 Implementing causal ordering
 Each process maintains a set of variables.
 A variable records the multicast messages received from a
process in the multicast group
 A vector represents the collection of these variables
 When a process multicasts a message, it attaches the
values of these variables to the message
 The values imply the number of multicast messages that
precede the one being multicast
 Attach the vector
 When a process receives a multicast message, the
process only delivers the message after the
preceding multicast messages have been received
 Assumption: the communication between two
processes has the FIFO property

28
 Let Vi[j] represents the message received by process
i from process j
 Initially, on each process Vi[j] =0 (1 ≤ j ≤ N)
 When process i multicasts a message Vi[i] is
incremented
 Assume process i receives a message M from process
j. The following conditions must be satisfied before
M is delivered:
 Vj[j] = Vi[j] + 1
• Ensure FIFO order between i and j
 Vj[k] ≤ Vi[k] (k ≠ j)
• Ensure that multicast messages from other processes that
precedes M are delivered first.

29
V1[1]=0 V2[1]=0 V3[1]=0
V1[2]=0 V2[2]=0 V3[2]=0
V1[3]=0 V2[3]=0 V3[3]=0

P1 P2 P3
30
 V1[1]=1 V2[1]=0 V3[1]=0
V1[2]=0 V2[2]=0 V3[2]=0
V1[3]=0 V2[3]=0 V3[3]=0
M1
V1[1]=1
V1[2]=0 deliver M1
deliver M1 deliver M1
V1[3]=0

P1 P2 P3
31
V1[1]=1 V2[1]=1 V3[1]=1
V1[2]=0 V2[2]=0 V3[2]=0
V1[3]=0 V2[3]=0 V3[3]=0

P1 P2 P3
32
 V1[1]=2 V2[1]=1 V3[1]=1
V1[2]=0 V2[2]=0 V3[2]=0
V1[3]=0 V2[3]=0 V3[3]=1

M2 M3
V1[1]=2 V3[1]=1
V1[2]=0 deliver M2 V3[2]=0
V1[3]=0 deliver M3 V [3]=1
3

P1 P2 P3
33
 V1[1]=2 V2[1]=1 V3[1]=1
V1[2]=0 V2[2]=0 V3[2]=0
V1[3]=0 V2[3]=0 V3[3]=1

M2 M3
deliver M3
V1[1]=2 V3[1]=1
V1[2]=0 V3[2]=0
V1[3]=0 V3[3]=1

deliver M3
deliver M2

P1 P2 P3
34
 V1[1]=2 V2[1]=1 V3[1]=2
V1[2]=0 V2[2]=0 V3[2]=0
V1[3]=1 V2[3]=1 V3[3]=1

M2 M3
V1[1]=2 V3[1]=1
V1[2]=0 V3[2]=0
V1[3]=0 V3[3]=1

P1 P2 P3
35
 V1[1]=2 V2[1]=1 V3[1]=2
V1[2]=0 V2[2]=0 V3[2]=0
V1[3]=1 V2[3]=1 V3[3]=1

M2 M3
V1[1]=2 V3[1]=1
V1[2]=0 V3[2]=0
V1[3]=0 V3[3]=1

deliver M2

P1 P2 P3
36
V1[1]=2 V2[1]=2 V3[1]=2
V1[2]=0 V2[2]=0 V3[2]=0
V1[3]=1 V2[3]=1 V3[3]=1

P1 P2 P3
37
 Reliable multicast
 For some applications, a multicast message is
required to be delivered either to all the
processes or to none at all.
 A command sent to a group of players in an on-line
game
 Reliable multicast means that every process in
the process group receives all multicasts
 Reliability is orthogonal to ordering
 Can implement Reliable-FIFO, or Reliable-Causal, or
Reliable-Total

38
 Since processes might crash during its
execution, ensuring reliable multicast is a
non-trivial problem.
A process might have delivered a multicast
messages to some processes and crashed
before it is able to deliver the message to the
other processes in the multicast group
 We want all the nonfaulty processes
receive the same set of multicast messages
 The faulty processes will not participate in
the computation. So, they are not a
concern.
39
 A naïve algorithm for achieving
reliable multicast
 The sender uses TCP to send multicast
message to each receiver.
 To ensure a message is received by all
processes in a group, each receiver sends a
received multicast message to all the other
processes in the group except the sender
of the multicast message.
A receiver would not re-multicast the same
message more than once.
 What’s the message complexity?

40
 Virtual Synchrony
 Virtual synchrony is a model for managing group
membership and group communications.
 Processes might join or leave a multicast group
(due to failure or planed leave).
 A view of a group refers to the membership of a
multicast group at a given point of time.
 When processes join or leave a group, the view
of the group changes.
 The processes in a group should have the same
group view when a message is multicast.
 Allthe processes agree who should receive the
message.
 When a process join or leave a group, a view
change message should be broadcast to all the
processes.
 The processes should change their view together. 41
 Virtual synchrony guarantees that all view
changes are delivered in the same order at
all non-faulty processes
 If a P1 process receives view changes, say {P1},
{P1, P2, P3}, {P1, P2}, {P1, P2, P4} then non-faulty
process receives the same sequence of view
changes (after it joins the group)
 E.g. P2 receives views {P1, P2, P3}, {P1, P2}, {P1,
P2, P4}

42
 A multicast message M is said to be “delivered in a
view V at process P” if P receives view V and delivers M
before P receives the next view
 Virtual synchrony ensures that
 The set of multicasts delivered in a given view is the same at
all non-faulty processes that were in that view
 The sender of the multicast message also belongs to that view
 If the sender of a message crashes during the multicast, the
message may either be delivered to all nonfaulty processes or
ignored by each of them.
 A multicast with this property is called virtually
synchronous.
 Systems that implemented it have been used in many
mission-critical information systems, e.g. NYSE, Air
Traffic Control System.

43
 examples

Acknowledgement: This example is adapted from Prof. Indranil Gupta of UIUC

44
 examples

Acknowledgement: This example is adapted from Prof. Indranil Gupta of UIUC

45
Acknowledgement: This example is adapted from Prof. Indranil Gupta of UIUC

46
Acknowledgement: This example is adapted from Prof. Indranil Gupta of UIUC

47
Acknowledgement: This example is adapted from Prof. Indranil Gupta of UIUC

48
 Implementing Virtual Synchrony
Assumptions
 A message is multicast using reliable point-to-point
communication (e.g. TCP).
 Although each point-to-point transmission is guaranteed to succeed,
there are no guarantees that all group members receive the message
 The sender might crash during or before transmitting the message
 No process crash during view change

Problem
 Guarantee that all messages sent to group view G are delivered
to all nonfaulty processes in G before the group view changes to
G’ (i.e. G ≠G’)

Solution
 For each message m, each process in G keeps m until it knows
that all members in G have received m
 If m has been received by all members in G, m is called stable.
 Only stable messages can be delivered.

49
 A view change message is sent to all the processes
in group view G when
 A process wants to join or leave the group
 A process detected that failure of another process

 Process 4 noticed
the failure of
process 7 and sends
a view change
message

50
 When a process P receives a view change message, P sends a
copy of any unstable message from the current view to
every process in the current view and marked these
messages as stable
 To indicate that P no longer has any unstable messages and
that it is prepared to install the new view, P multicast a
flush message for installing the new view

 Process 6 sends out

all its unstable
messages followed
by a flush message

51
 After P has received a flush message for the new
view from each other process, P can install the
new view

 Process 6 installs
the new view when
it has received a
flush message
from all other
processes

52
 reviews
 Understand the meaning of FIFO, total and causal
order multicast and use an example to show why
they are important for applications.
 Understand the three algorithms for implementing
the three ordered multicast protocols.
 Can the causal order protocol be used to ensure
FIFO order multicast? Explain why.
 Can the FIFO order multicast protocol be used to
ensure total order multicast? Explain why.
 Design an algorithm that uses the partially
ordered logical timestamp to ensure the casual-
ordering of multicast messages.
 What properties does virtual synchrony have?

53
 reviews

 The implementation of the virtual synchrony

discussed in our lecture assumes that no failure
occurs during the view change. Augment the
scheme so that it can cope with failures during the
view change.
 Assume that multiple processes can contact
different processes in the process group
concurrently for joining the process group.
Augment the scheme discussed in our lecture to
handle the view change correctly.
 Extend the implementation of the virtual
synchrony discussed in our lecture to provide
reliable total-order multicast.
54
 readings

 K. Birman and T. Joseph. Reliable

communications in the presence of failures.
ACM Trans. Comput. Syst., 5(1), 1987
 K. Birman, A. Schiper, and P. Stephenson.
Lightweight causal and atomic group
multicast. ACM Transactions on Computer
Systems, 9(3):272--314, 1991

55