Distributed Systems Fö 5 - 1 Distributed Systems Fö 5 - 2

TIME AND STATE IN DISTRIBUTED Time in Distributed Systems

SYSTEMS

☞ Because each machine in a distributed system has

its own clock there is no notion of global physical time.
1. Time in Distributed Systems • The n crystals on the n computers will run at slightly
different rates, causing the clocks gradually to get
out of synchronization and give different values.

2. Lamport’s Logical Clocks

Problems:

3. Vector Clocks
• Time triggered systems: these are systems in
which certain activities are scheduled to occur at
predefined moments in time. If such activities are to
4. Causal Ordering of Messages be coordinated over a distributed system we need a
coherent notion of time.

Example: time-triggered real-time systems

5. Global States and their Consistency

6. Cuts of a Distributed Computation • Maintaining the consistency of distributed data is

often based on the time when a certain modification
has been performed.

7. Recording of a Global State Example: a make program.

Petru Eles, IDA, LiTH Petru Eles, IDA, LiTH

Distributed Systems Fö 5 - 3 Distributed Systems Fö 5 - 4

Time in Distributed Systems (cont’d) Time in Distributed Systems (cont’d)

Solutions:
The make-program example
☞ Synchronization of physical clocks
•
When the programmer has finished changing some
source files he starts make; make examines the • Computer clocks are synchronized with one
times at which all object and source files were last another to an achievable, known, degree of
modified and decides which source files have to be accuracy ⇒ within the bounds of this accuracy we
(re)compiled. can coordinate activities on different computers
local using each computer’s local clock.
physical
• Physical clock synchronization is needed for
Computer 1 652 653 654 655 656 clock
(compiler) distributed real-time systems.
P.o created
local
physical ☞ Logical clocks
Computer 2 648 649 650 651 652 clock
(editor) • In many applications we are not interested in the
P.c created physical time at which events occur; what is
important is the relative order of events!
The make-program is such an example (slide 3).
Although P.c is modified after P.o has been
generated, because of the clock drift the time
• In such situations we don’t need synchronized
assigned to P.c is smaller.
physical clocks.
Relative ordering is based on a virtual notion of
time - logical time.
P.c will not be recompiled for the new version! • Logical time is implemented using logical clocks.

Petru Eles, IDA, LiTH Petru Eles, IDA, LiTH

 Distributed Systems Fö 5 - 5 Distributed Systems Fö 5 - 6

Lamport’s Logical Clocks Lamport’s Logical Clocks (cont’d)

• If a → b, we say that event a causally affects event b.

☞ The order of events occurring at different processes The two events are causally related.
is critical for many distributed applications.
Example: P.o_created and P.c_created in slide 3.
• There are events which are not related by the
happened-before relation.
☞ Ordering can be based on two simple situations: If both a → e and e → a are false, then a and e are
1. If two events occurred in the same process concurrent events; we write a || e.
then they occurred in the order observed fol-
lowing the respective process;
2. Whenever a message is sent between P1
a b
processes, the event of sending the message

m1
occurred before the event of receiving it.

P2
☞ Ordering by Lamport is based on the happened- c d
before relation (denoted by →):

m2
• a → b, if a and b are events in the same process
and a occurred before b; P3
• a → b, if a is the event of sending a message m in a e f
process, and b is the event of the same message m
being received by another process;
• If a → b and b → c, then a → c (the relation is P1, P2, P3: processes;
transitive). a, b, c, d, e, f: events;

a → b, c → d, e → f, b → c, d → f
a → c, a → d, a → f, b → d, b → f, ...
a || e, c || e, ...

Petru Eles, IDA, LiTH Petru Eles, IDA, LiTH

Distributed Systems Fö 5 - 7 Distributed Systems Fö 5 - 8

Lamport’s Logical Clocks (cont’d) Lamport’s Logical Clocks (cont’d)

☞ Implementation of logical clocks is performed using

☞ Using physical clocks, the happened before relation the following rules for updating the clocks and
can not be captured. It is possible that b → c and at transmitting their values in messages:
the same time Tb > Tc (Tb is the physical time of b).
[R1]: CPi is incremented before each event is issued at
process Pi: CPi := CPi + 1.
☞ Logical clocks can be used in order to capture the
happened-before relation. [R2]: a) When a is the event of sending a message m
from process Pi, then the timestamp tm = CPi(a)
is included in m (CPi(a) is the logical clock
• A logical clock is a monotonically increasing
value obtained after applying rule R1).
software counter.
b) On receiving message m by process Pj, its
• There is a logical clock CPi at each process Pi in
logical clock CPj is updated as follows:
the system.
CPj := max(CPj, tm).
• The value of the logical clock is used to assign
timestamps to events. c) The new value of CPj is used to timestamp the
CPi(a) is the timestamp of event a in process Pi. event of receiving message m by Pj (applying
rule R1).
• There is no relationship between a logical clock and
any physical clock.
• If a and b are events in the same process and a oc-
curred before b, then a → b, and (by R1) C(a) < C(b).
• If a is the event of sending a message m in a
To capture the happened-before relation, logical clocks
process, and b is the event of the same message m
have to be implemented so that
being received by another process, then a → b, and
if a → b, then C(a) < C(b) (by R2) C(a) < C(b).
• If a → b and b → c, then a → c, and (by induction)
C(a) < C(c).

Petru Eles, IDA, LiTH Petru Eles, IDA, LiTH

 Distributed Systems Fö 5 - 9 Distributed Systems Fö 5 - 10

Lamport’s Logical Clocks (cont’d) Problems with Lamport’s Logical Clocks

1 2 ☞ Lamport’s logical clocks impose only a partial order

P1
a b on the set of events; pairs of distinct events generated
by different processes can have identical timestamp.

m1
3 4 • For certain applications a total ordering is needed;
P2 they consider that no two events can occur at the
c d
same time.

m2
• In order to enforce total ordering a global logical
timestamp is introduced:
1 5
P3 - the global logical timestamp of an event a oc-
e f curring at process Pi, with logical timestamp
CPi(a), is a pair (CPi(a), i), where i is an identifi-
er of process Pi;
• For the make-program example we suppose that a - we define
process running a compilation notifies, through a (CPi(a), i) < (CPj(b), j) if and only if
message, the process holding the source file about
CPi(a) < CPj(b), or CPi(a) = CPj(b) and i < j.
the event P.o created ⇒ a logical clock can be used
to correctly timestamp the files.
local (1,1) (2,1)
P1
physical a b
Computer 1 652 653 654 655 656 clock

m1
(compiler) 1 2
P.o created (3,2) (4,2)
P2
local c d
physical
Computer 2 648 650 651 652

m2
649 clock
(editor) 1 3 4 (1,3) (5,3)
P.c created P3
e f

Petru Eles, IDA, LiTH Petru Eles, IDA, LiTH

Distributed Systems Fö 5 - 11 Distributed Systems Fö 5 - 12

Problems with Lamport’s Logical Clocks (cont’d) Problems with Lamport’s Logical Clocks (cont’d)

☞ Lamport’s logical clocks are not powerful enough to

perform a causal ordering of events. send(M1) send(M3)
• if a → b, then C(a) < C(b).
However, the reverse is not always true (if the P1
1 2 3
events occurred in different processes): send(M2)
if C(a) < C(b), then a → b is not necessarily true.
(it is only guaranteed that b → a is not true).
P2
2 3 4
(1,1) (2,1)
P1
a b
P3
m1

1 M2 M1 M3
4 1 3
(3,2) (4,2)
P2
c d
m2

• We would like messages to be processed

according to their causal order.
(1,3) (5,3) We would like to use the associated timestamp for
P3
e f this purpose.
• Process P3 receives messages M1, M2, and M3.
C(e) < C(b), however there is no causal relation send(M1) → send(M2), send(M1) → send(M3),
from event e to event b. send(M3) || send(M2)
• M1 has to be processed before M2 and M3.
• By just looking at the timestamps of the events, we However P3 has not to wait for M3 in order to
cannot say whether two events are causally related process it before M2 (although M3’s logical clock
or not. timestamp is smaller than M2’s).

Petru Eles, IDA, LiTH Petru Eles, IDA, LiTH

 Distributed Systems Fö 5 - 13 Distributed Systems Fö 5 - 14

Vector Clocks Vector Clocks (cont’d)

☞ Vector clocks give the ability to decide whether two

events are causally related or not by simply looking at
their timestamp. ☞ Implementation of vector clocks is performed using
the following rules for updating the clocks and
transmitting their values in messages:

• Each process Pi has a clock CPi v , which is an

v
[R1]: CPi is incremented before each event is issued at
integer vector of length n (n is the number of v
process Pi: CPi v
[i] := CPi[i] + 1.
processes).

• The value of CPi v is used to assign timestamps to [R2]: a) When a is the event of sending a message m
v
events in process Pi. from process Pi, then the timestamp tm = CPi (a)
v
v
CPi(a) is the timestamp of event a in process Pi. is included in m (CPi (a) is the vector clock value
obtained after applying rule R1).
• v
CPi v
[i], the ith entry of CPi ,corresponds to Pi’s own b) On receiving message m by process Pj, its
v is updated as follows:
vector clock CPj
logical time.
∀k ∈ {1,2,..,n}, CPj
v [k] := max(Cv [k], t [k]).
Pj m
v is used to timestamp the
• v
CPi[j], j ≠ i, is Pi’s "best guess" of the logical time at Pj. c) The new value of CPj
v
CPi [j] indicates the (logical) time of occurrence event of receiving message m by Pj (applying
of the last event at Pj which is in a happened- rule R1).
before relation to the current event at Pi.

Petru Eles, IDA, LiTH Petru Eles, IDA, LiTH

Distributed Systems Fö 5 - 15 Distributed Systems Fö 5 - 16

Vector Clocks (cont’d) Causal Ordering of Messages Using

Vector Clocks
(1,0,0) (2,0,0) (3,0,0)
P1
a b c

The problem has been formulated on slide 12:

(0,1,0) (2,2,0) (2,3,1) (2,4,1)
P2
d e f
☞ We would like messages to be processed according
to their causal order.
(0,0,1) (2,4,2)
P3
g h
• If Send(M1) → Send(M2), then every recipient of
both messages M1 and M2 must receive M1 before
For any two vector timestamps u and v, we have: M2.
• u = v if and only if ∀i, u[i] = v[i]
• u ≤ v if and only if ∀i, u[i] ≤ v[i]
• u < v if and only if (u ≤ v ∧ u ≠ v)
• u || v if and only if ¬(u < v) ∧ ¬(v < u)

☞ Two events a and b are causally related if and only if

Cv(a) < Cv(b) or Cv(b) < Cv(a). Otherwise the events
are concurrent.
☞ With vector clocks we get the property which we
missed for Lamport’s logical clocks:
• a → b if and only if Cv(a) < Cv(b).
Thus, by just looking at the timestamps of the
events, we can say whether two events are causally
related or not.

Petru Eles, IDA, LiTH Petru Eles, IDA, LiTH

 Distributed Systems Fö 5 - 17 Distributed Systems Fö 5 - 18

Causal Ordering of Messages Using Causal Ordering of Messages Using

Vector Clocks (cont’d) Vector Clocks (cont’d)

• The events which are of interest here are the

(0,0,0) (0,1,0) (0,1,1) (0,2,1) sending of messages ⇒ vector clocks will be
P1 incremented only for message sending.

☞ Implementation of the protocol is based on the

(0,0,0) (0,1,0) (0,1,1) following rules:
P2
(0,2,1)
[R1]: a) Before broadcasting a message m, a process Pi
increments the vector clock: CvPi[i] := CvPi[i] + 1.
(0,0,0) b) The timestamp tm = CPi v
is included in m.
P3
(0,1,0) (0,1,1) (0,2,1)
[R2]: The receiving side, at process Pj, delays the
delivery of message m coming from Pi until both
the following conditions are satisfied:
☞ A message delivery protocol which preforms causal v [i] = t [i] - 1
1. CPj m
ordering based on vector clocks. v [k] ≥ t [k]
2. ∀k ∈ {1,2,..,n} - {i}, CPj m
Delayed messages are queued at each process in
• Basic Idea: a queue that is sorted by their vector timestamp;
- A message is delivered to a process only if the concurrent messages are ordered by the time of
message immediately preceding it (considering their arrival.
the causal ordering) has been already delivered [R3]: When a message is delivered at process Pj, its
v
to the process. Otherwise, the message is vector clock CPj is updated according to rule R2b
buffered. for vector clock implementation (see slide 14).
• We assume that processes communicate using
broadcast messages. ☞ tm[i] - 1 indicates how many messages originating
(There exist similar protocols for non-broadcast from Pi precede m.
communication too.) Step R2.1 ensures that process Pj has received all
the messages originating from Pi that precede m.
Step R2.2 ensures that Pj has received all those
messages received by Pi before sending m.

Petru Eles, IDA, LiTH Petru Eles, IDA, LiTH

Distributed Systems Fö 5 - 19 Distributed Systems Fö 5 - 20

Global States Global States (cont’d)

Consider a bank system with two accounts A and B at

two different sites; we transfer $50 between A and B.
Ch1: empty
☞ The problem is how to collect and record a consistent
global state in a distributed system. $500 $200
Ch2: empty
A B

Ch1: $50
Why a problem?
$450 $200
• Because there is no global clock (no coherent
notion of time) and no shared memory! Ch2: empty
A B

Ch1: empty
$450 $250
Ch2: empty
A B
C : consistent
A Ch1 B
NC: not consistent
500 empty 200 C
500 50 200 NC
450 50 200 C
450 empty 200 NC
500 50 250 NC
450 50 250 NC
450 empty 250 C
500 empty 250 NC

Petru Eles, IDA, LiTH Petru Eles, IDA, LiTH

 Distributed Systems Fö 5 - 21 Distributed Systems Fö 5 - 22

Formal Definition
Global States (cont’d)

• LSi is the local state of process Pi.

☞ In general, a global state consists of a set of local Beside other information, the local state also includes a
states and a set of states of the communication record of all messages sent and received by the process.
channels. • We consider the global state GS of a system, as the
collection of the local states of its processes:
GS = {LS1, LS2, ..., LSn}.
☞ The state of the communication channel in a • A certain global state can be consistent or not!
consistent global state should be the sequence of
messages sent along the channel before the
sender’s state was recorded, excluding the sequence
of messages received along the channel before the • send(mkij) denotes the event of sending message mkij from
receiver’s state was recorded. Pi to Pj;
rec(mkij) denotes the event of receiving message mkij by Pj.

☞ It is difficult to record channel states to ensure the • send(mkij) ∈ LSi if and only if the sending event occurred
above rule ⇒ global states are very often recorded before the local state was recorded;
without using channel states. rec(mkij) ∈ LSj if and only if the receiving event occurred
This is the case in the definition below. before the local state was recorded.

• transit(LSi,LSj) = {mkij | send(mkij) ∈LSi ∧ rec(mkij) ∉LSj}

inconsistent(LSi,LSj) = {mkij | send(mkij) ∉LSi ∧ rec(mkij) ∈LSj}

Petru Eles, IDA, LiTH Petru Eles, IDA, LiTH

Distributed Systems Fö 5 - 23 Distributed Systems Fö 5 - 24

Formal Definition (cont’d) Formal Definition (cont’d)

☞ A global state GS = {LS1, LS2, ..., LSn} is consistent if

and only if:
∀i, ∀j: 1 ≤ i, j ≤ n :: inconsistent(LSi,LSj) = ∅ LS11 LS12
P1
• In a consistent global state for every received
message a corresponding send event is recorded
in the global state. LS21 LS22 LS23
• In an inconsistent global state, there is at least one P2
message whose receive event is recorded but its
send event is not recorded.
P3
LS31 LS32 LS33
☞ A global state GS = {LS1, LS2, ..., LSn} is transitless if
and only if:
∀i, ∀j: 1 ≤ i, j ≤ n :: transit(LSi,LSj) = ∅
• All messages recorded to be sent are also
{LS11, LS22, LS32} is inconsistent;
recorded to be received.
{LS12, LS23, LS33} is consistent;
{LS11, LS21, LS31} is strongly consistent.
☞ A global state is strongly consistent if it is consistent
and transitless.
• A strongly consistent state corresponds to a
consistent state in which all messages recorded as
sent are also recorded as received.

Note: the global state, as defined here, is seen as a

collection of the local states, without explicitely
capturing the state of the channel.

Petru Eles, IDA, LiTH Petru Eles, IDA, LiTH

 Distributed Systems Fö 5 - 25 Distributed Systems Fö 5 - 26

Formal Definition (cont’d) Cuts of a Distributed Computation

Ch1: empty
$500 $200
☞ A cut is a graphical representation of a global state.
Ch2: empty A consistent cut is a graphical representation of a
A B
consistent global state.
Ch1: $50
$450 $200 • A cut of a distributed computation is a set
Ch2: empty Ct = {c1, c2, ..., cn}, where ci is the cut event at
A B process Pi.
Ch1: empty • A cut event is the event of recording a local state of
the respective process.
$450 $250
Ch2: empty
A B
LS11 LS12
C : consistent P1
A B c1 c6
NC: not consistent
500 200 {A,B}: strongly C
LS21 LS22 LS23
450 200 {A,B}: C P2
(mess1 sent) c2 c 4 c7
500 250 {A,B}: NC
(mess1 received) c3 c5 c8
P3
450 250 {A,B}: strongly C LS31 LS32 LS33
(mess1 sent) (mess1 received)

• After registering of the receive event(s) a consistent

state becomes strongly consistent. It is considered
to be a normal (transient) situation.

Petru Eles, IDA, LiTH Petru Eles, IDA, LiTH

Distributed Systems Fö 5 - 27 Distributed Systems Fö 5 - 28

Cuts of a Distributed Computation (cont’d) Cuts of a Distributed Computation (cont’d)

Theorem
☞ Let ek denote an event at process Pk.
A cut Ct = {c1, c2, ..., cn} is a consistent cut if and only if
A cut Ct = {c1, c2, ..., cn} is a consistent cut if and only if
no two cut events are causally related, that is:
∀Pi,∀Pj, ∃ei,∃ej such that (ei → ej) ∧ (ej → cj) ∧ ¬(ei → ci) ∀ci, ∀cj :: ¬(ci → cj) ∧ ¬(cj → ci)
• A set of concurrent cut events form a consistent cut.
• A cut is consistent if every message that was
received before a cut event was sent before the cut LS11
event at the sender process. e1 LS12
P1
c1 c6
LS11 LS12
e1
P1 LS21 LS22 LS23
c1 c6 P2
c2 e2 c4 c7
LS21 LS22 LS23
P2 c8
c2 e2 c4 c7 c3 c5
P3
LS31 LS32 LS33
c3 c5 c8
P3
LS31 LS32 LS33 {c1,c2,c3}: strongly consistent (no communication line is
crossed)
{c6,c7,c8}: consistent (communication line is crossed but
{c1,c4,c5} is not consistent: (e1→e2) ∧ (e2→c4) ∧ ¬(e1→c1) no causal relation).
{c1,c4,c5}: not consistent; c1 → c4

Petru Eles, IDA, LiTH Petru Eles, IDA, LiTH

 Distributed Systems Fö 5 - 29 Distributed Systems Fö 5 - 30

Global State Recording Global State Recording (cont’d)

(Chandy-Lamport Algorithm)
Some discussion on how to collect a global state:

• The algorithm records a collection of local states

which give a consistent global state of the system. • A process Pi records its local state LSi and later
In addition it records the state of the channels sends a message m to Pj; LSj at Pj has to be
which is consistent with the collected global state. recorded before Pj has received m.

• Such a recorded "view" of the system is called a • The state SChij of the channel Chij consists of all
snapshot. messages that process Pi sent before recording LSi
and which have not been received by Pj when
• We assume that processes are connected through recording LSj.
one directional channels and message delivery is
FIFO. • A snapshot is started at the request of a particular
process Pi, for example, when it suspects a
• We assume that the graph of processes and deadlock because of long delay in accessing a
channels is strongly connected (there exists a path resource; Pi then records its state LSi and, before
between any two processes). sending any other message, it sends a token to
every Pj that Pi communicates with.
• The algorithm is based on the use of a special
message, snapshot token, in order to control the • When Pj receives a token from Pi, and this is the
state collection process. first time it received a token, it must record its state
before it receives the next message from Pi. After
recording its state Pj sends a token to every
process it communicates with, before sending them
any other message.

Petru Eles, IDA, LiTH Petru Eles, IDA, LiTH

Distributed Systems Fö 5 - 31 Distributed Systems Fö 5 - 32

Global State Recording (cont’d) Global State Recording (cont’d)

What about the channel states?

Maybe, you prefer this view:
• Pi sends a token to Pj and this is the first time Pj
received a token ⇒ Pj immediately records its state. token sending
All the messages sent by Pi before sending the
token have been received at Pj ⇒ SChij := ∅. ci
Pi
• Pj receives a token from Pk, but Pj already recorded
its state. M is the set of messages that Pj received
ck
from Pk after Pj recorded its state and before Pj Pk
received the token from Pk ⇒ SChkj := M.

Pk LSk
Pj
cj
state of Chkj
messages sent
after LSk recorded
token from Pk
messages sent
before LSk recorded messages received
after LSj recorded • Don’t forget when you look to the picture: we
messages received assumed that message passing on a channel
state of before LSj recorded connecting two processes is FIFO.
channel Chkj
LSj
Pj

• The algorithm terminates when all processes have

received tokens on all their input channels.
• The process that initiated the snapshot should be
informed; it can collect the global snapshot.

Petru Eles, IDA, LiTH Petru Eles, IDA, LiTH

 Distributed Systems Fö 5 - 33 Distributed Systems Fö 5 - 34

Global State Recording (cont’d) Summary

The algorithm

• In a distributed system there is no exact notion of

global physical time. Physical clocks can be
☞ Rule for sender Pi: synchronized to a certain accuracy.
/* performed by the initiating process and by any
other process at the reception of the first token */ • In many applications not physical time is important
[SR1]: Pi records its state. but only the relative ordering of certain events.
Such an ordering can be achieved using logical
[SR2]: Pi sends a token on each of its outgoing channels. clocks.
• Lamport’s logical clocks are implemented using a
monotonic integer counter at each site. They can
☞ Rule for receiver Pj: be used in order to capture the happened-before
/* executed whenever Pj receives a token from relation.
another process Pi on channel Chij */
[RR1]: if Pj has not yet recorded its state then • The main problem with Lamport’s clocks is that they
are not powerful enough to perform a causal
Record the state of the channel: SChij := ∅. ordering of events.
Follow the "Rule for sender".
else • Vector clocks give the ability to decide whether two
events are causally related or not, by simply looking
Record the state of the channel: SChij := M, at their timestamps.
where M is the set of messages that Pj
received from Pi after Pj recorded its state
and before Pj received the token on Chij.
end if.

Petru Eles, IDA, LiTH Petru Eles, IDA, LiTH

Distributed Systems Fö 5 - 35

Summary (cont’d)

• As there doesn’t exist a global notion of physical

time, it is very difficult to reason about a global state
in a distributed system.
• We can consider a global state as a collection of
local states and, possibly, a set of states of the
communication channels.
• A global state can be consistent or not.
• A cut is a graphical representation of a global state.
Using cuts it is easy to elegantly reason about
consistency of global states.
• It is possible to record local states and states of the
channels, so that together they provide a consistent
view of the system. Such a view is called a
snapshot.

Petru Eles, IDA, LiTH