Chapter 4:

Distributed Systems:
Replication and Consistency

Fall 2013
Jussi Kangasharju
Chapter Outline

n Replication
n Consistency models
n Distribution protocols
n Consistency protocols

Kangasharju: Distributed Systems 2

 Data Replication
user B

user C

user A object

object

Kangasharju: Distributed Systems 3

Reasons for Data Replication

n Dependability requirements
n availability
- at least some server somewhere
- wireless connections => a local cache
n reliability (correctness of data)
- fault tolerance against data corruption
- fault tolerance against faulty operations
n Performance
n response time, throughput
n scalability
- increasing workload
- geographic expansion
n mobile workstations => a local cache
n Price to be paid: consistency maintenance
n performance vs. required level of consistency
(need not care updates immediately visible)

Kangasharju: Distributed Systems 4

 Replication and Scalability

n Requirement: tight consistency

(an operation at any copy => the same result)
n Difficulties
n atomic operations (performance, fault tolerance??)
n timing: when exactly the update is to be performed?
n Solution: consistency requirements vary
n always consistent => generally consistent
(when does it matter? depends on application)
n => improved performance

n Data-centric / client-centric consistency models

Kangasharju: Distributed Systems 5

 Data-Centric Consistency Models (1)

The general organization of a logical data store, physically distributed and

replicated across multiple processes.

Kangasharju: Distributed Systems 6

 Data-Centric Consistency Models (2)

n Contract between processes and the data store:

n processes obey the rules
n the store works correctly

n Normal expectation: a read returns the result of the

last write
n Problem: which write is the last one?
a range of consistency models

Kangasharju: Distributed Systems 7

 Strict Consistency
Any read on a data item x
returns a value corresponding to the result of
the most recent write on x.

Behavior of two processes, operating on the same data item.

a) A strictly consistent store.
b) A store that is not strictly consistent.

A problem: implementation requires absolute global time.

Another problem: a solution may be physically impossible.

Kangasharju: Distributed Systems 8

 Sequential Consistency
The result of any execution is the same as if
the (read and write) operations by all processes on the data store
were executed in some sequential order and
the operations of each individual process appear in this sequence
in the order specified by its program. Note: nothing said about time!

A sequentially consistent data store. A data store that is not sequentially consistent.

Note: a process sees all writes and own reads

Kangasharju: Distributed Systems 9

 Linearizability

The result of any execution is the same as if

the (read and write) operations by all processes on the data store
were executed in some sequential order and
the operations of each individual process appear in this sequence
in the order specified by its program.
In addition,
if TSOP1(x) < TSOP2(y) , then
operation OP1(x) should precede OP2(y) in this sequence.

Linearizability: primarily used to assist formal verification of concurrent

algorithms.

Sequential consistency: widely used, comparable to serializability of

transactions (performance??)

Kangasharju: Distributed Systems 10

 Linearizability and Sequential Consistency (1)

Three concurrently executing processes

Process P1 Process P2 Process P3

x = 1; y = 1; z = 1;
print ( y, z); print (x, z); print (x, y);

Initial values: x = y = z = 0
All statements are assumed to be indivisible.

Execution sequences
- 720 possible execution sequences (several of which violate program order)
- 90 valid execution sequences

Kangasharju: Distributed Systems 11

 Linearizability and Sequential Consistency (2)

x = 1; x = 1; y = 1; y = 1;
print (y, z); y = 1; z = 1; x = 1;
y = 1; print (x,z); print (x, y); z = 1;
print (x, z); print(y, z); print (x, z); print (x, z);
z = 1; z = 1; x = 1; print (y, z);
print (x, y); print (x, y); print (y, z); print (x, y);

Prints: 001011 Prints: 101011 Prints: 010111 Prints: 111111

(a) (b) (c) (d)

Four valid execution sequences for the processes.

The contract:
the process must accept all valid results as proper answers and
work correctly if any of them occurs.

Kangasharju: Distributed Systems 12

Causal Consistency (1)

Necessary condition:

Writes that are potentially causally related must be

seen by all processes in the same order.
Concurrent writes may be seen in a different order on
different machines.

Kangasharju: Distributed Systems 13

 Causal Consistency (2)

This sequence is allowed with a causally-consistent store,

but not with sequentially or strictly consistent store.

Kangasharju: Distributed Systems 14

 Causal Consistency (3)

A violation of a
causally-consistent
store.

A correct
sequence
of events in a
causally-consistent
store.

Kangasharju: Distributed Systems 15

 FIFO Consistency (1)

Necessary Condition:

Writes done by a single process

are seen by all other processes
in the order in which they were issued,
but
writes from different processes
may be seen in a different order by different processes.

Kangasharju: Distributed Systems 16

 FIFO Consistency (2)

A valid sequence of events of FIFO consistency

Guarantee:
writes from a single source must arrive in order
no other guarantees.
Easy to implement!

Kangasharju: Distributed Systems 17

Less Restrictive Consistencies
n Needs
n FIFO too restrictive: sometimes no need to see all writes
n example: updates within a critical section (the variables are
locked => replicas need not be updated -- but the database
does not know it)
n Replicated data and consistency needs
n single user: data-centric consistency needed at all?
- in a distributed (single-user) application: yes!
- but distributed single-user applications exploiting
replicas are not very common
n shared data: mutual exclusion and consistency obligatory
=> combine consistency maintenance with the
implementation of critical regions

Kangasharju: Distributed Systems 18

Consistency of Shared Data (1)

n Assumption: during a critical section the user has access to one

replica only
n Aspects of concern
- consistency maintenance timing, alternatives:
- entry: update the active replica
- exit: propagate modifications to other replicas
- asynchronous: independent synchronization
- control of mutual exclusion:
- automatic, independent
- data of concern:
- all data, selected data

Kangasharju: Distributed Systems 19

Consistency of Shared Data (2)
n Weaker consistency requirements
n Weak consistency
n Release consistency
n Entry consistency

n Implementation method
n control variable
- synchronization / locking
n operation
- synchronize
- lock/unlock and synchronize

Kangasharju: Distributed Systems 20

 Entry Consistency (1)
n Consistency combined with mutual exclusion

n Each shared data item is associated with a

synchronization variable S
n S has a current owner (who has exclusive access to the associated data, which is
guaranteed up-to-date)
n Process P enters a critical section: Acquire(S)
n retrieve the ownership of S
n the associated variables are made consistent
n Propagation of updates at the next Acquire(S) by some other process

Kangasharju: Distributed Systems 21

 Entry Consistency (2)

x
P1: Acq(Lx) W(x)a Acq(Ly) W(y)b Rel(Lx) Rel(Ly)

P2: Acq(Lx) Acq(Lx) x R(x)a R(y)NIL

P3: Acq(Ly) R(y)b

A valid event sequence for entry consistency.

Kangasharju: Distributed Systems 22

 Summary of Consistency Models (1)

Consistency Description
Strict Absolute time ordering of all shared accesses matters.

Linearizability All processes see all shared accesses in the same order.
Accesses are furthermore ordered according to a
(nonunique) global timestamp
Sequential All processes see all shared accesses in the same order.
Accesses are not ordered in time

Causal All processes see causally-related shared accesses in

the same order.

FIFO All processes see writes from each other in the order they
were performed. Writes from different processes may not always
be seen in the same order by other processes.

Consistency models not using synchronization operations.

Kangasharju: Distributed Systems 23

 Summary of Consistency Models (2)

Consistency Description

Weak Shared data can be counted on to be consistent only after a

synchronization is done

Release All shared data are made consistent after the exit out of the
critical section

Entry Shared data associated with a synchronization variable

are made consistent when a critical section is entered.

Models with synchronization operations.

Kangasharju: Distributed Systems 24

 Client-Centric Models
n Environment
n most operations: read
n no simultaneous updates
n a relatively high degree of inconsistency tolerated
(examples: DNS, WWW pages)

n Wanted
n eventual consistency
n consistency seen by one single client

Kangasharju: Distributed Systems 25

Eventual Consistency

Kangasharju: Distributed Systems 26

 Monotonic Reads
If a process reads the value of of a data item x, any successive read operation
on x by that process will always return that same value or a more recent value.
(Example: e-mail )

A monotonic-read consistent
data store

A data store that does not

provide monotonic reads.

WS(xi): write set = sequence of operations on x at node Li

Kangasharju: Distributed Systems 27

 Monotonic Writes
A write operation by a process on a data item x is completed
before any successive write operation on x
by the same process. (Example: software updates)

A monotonic-write
consistent data store.

A data store that does not

provide monotonic-write
consistency.

Kangasharju: Distributed Systems 28

 Read Your Writes
The effect of a write operation by a process on data item x will
always be seen by a successive read operation on x by the same
process. (Example: edit www-page)

A data store that provides

read-your-writes
consistency.

A data store that

does not.

Kangasharju: Distributed Systems 29

 Writes Follow Reads

A writes-follow-reads
consistent data store

A data store that does not

provide writes-follow-reads
consistency

Process P: a write operation (on x) takes place on the same or a more

recent value (of x) that was read. (Example: bulletin board)

Kangasharju: Distributed Systems 30

Distribution Protocols

n Replica placement
n Update propagation
n Epidemic protocols

Kangasharju: Distributed Systems 31

Replica Placement (1)

The logical organization of different kinds of copies of a data

store into three concentric rings.

Kangasharju: Distributed Systems 32

 Replica Placement (2)

mirror

permanent replicas
server-initiated replicas servers
client-initiated replicas clients

Kangasharju: Distributed Systems 33

 Permanent Replicas

n Example: a WWW site

n The initial set of replicas:
constitute a distributed data store
n Organization
n A replicated server
(within one LAN; transparent for the clients)
n Mirrorsites (geographically spread across the Internet;
clients choose an appropriate one)

Kangasharju: Distributed Systems 34

 Server-Initiated Replicas (1)

n Created at the initiative of the data store

(e.g., for temporary needs)
n Need: to enhance performance
n Called as push caches
n Example: www hosting services
n a collection of servers
n provide access to www files belonging to third parties
n replicate files close to demanding clients

Kangasharju: Distributed Systems 35

 Server-Initiated Replicas (2)

n Issues:
n improve response time
n reduce server load; reduce data communication load
bring files to servers placed in the proximity of clients
n Where and when should replicas be created/deleted?
n For example:
n determine two threshold values for each (server, file): rep > del
n #[req(S,F)] > rep => create a new replica
n #[req(S,F)] < del => delete the file (replica)
n otherwise: the replica is allowed to be migrated
n Consistency: responsibility of the data store

Kangasharju: Distributed Systems 36

Client-Initiated Replicas

n Called as client caches

(local storage, temporary need of a copy)
n Managing left entirely to the client
n Placement
n typically: the client machine
n a machine shared by several clients
n Consistency: responsibility of client

n More on replication in the Web in Chapter 6

Kangasharju: Distributed Systems 37

Update Propagation:
State vs. Operations

n Update route: client => copy => {other copies}

n Responsibility: push or pull?
n Issues:
n consistency of copies
n cost: traffic, maintenance of state data
n What information is propagated?
n notification of an update (invalidation protocols)
n transfer of data (useful if high read-to-write ratio)
n propagate the update operation (active replication)

Kangasharju: Distributed Systems 38

 Pull versus Push (1)

n Push
n a server sends updates to other replica servers
n typically used between permanent and server-initiated replicas
n Pull
n client asks for update / validation confirmation
n typically used by client caches
- client to server: {data X, timestamp ti, OK?}
- server to client: OK or {data X, timestamp ti+k}

Kangasharju: Distributed Systems 39

 Pull versus Push Protocols (2)

Issue Push-based Pull-based

State of server List of client replicas and caches None

Messages sent Update (and possibly fetch update later) Poll and update

Response time at
Immediate (or fetch-update time) Fetch-update time
client

A comparison between push-based and pull-based protocols in the case of

multiple client, single server systems.

Kangasharju: Distributed Systems 40

 Pull vs. Push: Environmental Factors
n Read-to-update ratio
- high => push (one transfer many reads)
- low => pull (when needed check)
n Cost-QoS ratio
- factors:
- update rate, number of replicas => maintenance
workload
- need of consistency (guaranteed vs. probably_ok)
- examples
- (popular) web pages
- arriving flights at the airport
n Failure prone data communication
- lost push messages => unsuspected use of stale data
- pull: failure of validation => known risk of usage
- high reqs => combine push (data) and pull

Kangasharju: Distributed Systems 41

 Leases
n Combined push and pull
n A server promise: push updates for a certain time
n A lease expires
=> the client
- polls the server for new updates or
- requests a new lease
n Different types of leases
n age based: {time to last modification}
n renewal-frequency based: long-lasting leases to active users
n state-space overhead: increasing utilization of a server => lower
expiration times for new leases

Kangasharju: Distributed Systems 42

 Propagation Methods

n Data communication
n LAN: push & multicasting, pull & unicasting
n wide-area network: unicasting
n Information propagation: epidemic protocols
n a node with an update: infective
n a node not yet updated: susceptible
n a node not willing to spread the update: removed
n propagation: anti-entropy
- P picks randomly Q
- three information exchange alternatives:
P => Q or P <= Q or PQ
n propagation: gossiping

Kangasharju: Distributed Systems 43

 Gossiping (1)

P starts a gossip round (with a fixed k)

1. P selects randomly {Q1,..,Qk}
2. P sends the update to {Qi}
3. P becomes removed
Qi receives a gossip update
If Qi was susceptible, it starts
a gossip round
else Qi ignores the update

The textbook variant (for an infective P)

P: do until removed
{select a random Qi ; send the update to Qi ;
if Qi was infected then remove P with probability 1/k }

Kangasharju: Distributed Systems 44

 Gossiping (2)
n Coverage: depends on k (fanout)
n a large fanout: good coverage, big overhead
n a small fanout: the gossip (epidemic) dies out too soon
n n: number of nodes, m: parameter (fixed value)
k = log(n)+m =>
P{every node receives} = e ** (- e **(-k))
(esim: k=2 => P=0.87; k=5 => P=0.99)
n Merits
n scalability, decentralized operation
n reliability, robustness, fault tolerance
n no feedback implosion, no need for routing tables

Kangasharju: Distributed Systems 45

 Epidemic Protocols: Removing Data

The problem
1. server P deletes data D => all information on D is destroyed
[server Q has not yet deleted D]
2. communication P Q => P receives D (as new data)
A solution: deletion is a special update (death certificate)
n allows normal update communication
n a new problem: cleaning up of death certificates
n solution: time-to-live for the certificate
- after TTL elapsed: a normal server deletes the certificate
- some special servers maintain the historical certificates
forever (for what purpose?)

Kangasharju: Distributed Systems 46

 Consistency Protocols

n Consistency protocol: implementation of a consistency model

n The most widely applied models
n sequential consistency
n weak consistency with synchronization variables
n atomic transactions
n The main approaches
n primary-based protocols (remote write, local write)
n replicated-write protocols (active replication, quorum based)
n (cache-coherence protocols)

Kangasharju: Distributed Systems 47

 Remote-Write Protocols (1)

Primary-based remote-write protocol with a fixed server to which all read and
write operations are forwarded.

Kangasharju: Distributed Systems 48

 Remote-Write Protocols (2)

Sequential consistency
Read Your Writes
The principle of primary-backup protocol.

Kangasharju: Distributed Systems 49

 Local-Write Protocols (1)

Mobile workstations!
Name service overhead!

Primary-based local-write protocol in which a single copy is migrated

between processes.

Kangasharju: Distributed Systems 50

 Local-Write Protocols (2)

Example: Mobile PC <= primary

server for items to be needed

Primary-backup protocol in which the primary migrates to the process

wanting to perform an update.
Kangasharju: Distributed Systems 51
 Quorum-Based Protocols

n Consistency-guaranteeing update of replicas:

an update is carried out as a transaction
n Problems
n Performance?

n Sensitivity for availability (all or nothing) ?

n Solution:
n a subgroup of available replicas is allowed to update data
n Problem in a partitioned network:
n the groups cannot communicate =>
each group must decide independently whether it is allowed
to carry out operations.
n A quorum is a group which is large enough for the operation.

Kangasharju: Distributed Systems 52

 Quorum-Based Voting (Gifford)

Three voting-case examples:

The constraints:
a) A correct choice of read and write set
b) A choice that may lead to write-write conflicts
1. NR + NW > N
c) A correct choice, known as ROWA (read one, 2. NW > N/2
write all)

Kangasharju: Distributed Systems 53

 Quorum-Based Voting

Read
n Collect a read quorum
n Read from any up-to-date replica (the newest timestamp)

Write
n Collect a write quorum
n If there are insufficient up-to-date replicas, replace non-current
replicas with current replicas (WHY?)
n Update all replicas belonging to the write quorum.

Notice: each replica may have a different number of votes assigned to it.

Kangasharju: Distributed Systems 54

 Quorum Methods Applied
n Possibilities for various levels of reliability
n Guaranteed up-to-date: collect a full quorum
n Limited guarantee: insufficient quora allowed for reads
n Best effort
- read without a quorum
- write without a quorum - if consistency checks available
n Transactions involving replicated data
n Collect a quorum of locks
n Problem: a voting processes meets another ongoing voting
- alternative decisions: abort wait continue without a vote
- problem: a case of distributed decision making
(figure out a solution)

Kangasharju: Distributed Systems 55

Chapter Summary

n Replication
n Consistency models
n Distribution protocols
n Consistency protocols

Kangasharju: Distributed Systems 56