Consistency and

Replication
Distributed Systems
Introduction
• Reasons for replication
• Replication as scaling technique
Reason for Replication: Performance
and Scalability
Main issue: To keep replicas consistent, we generally need to ensure that all
conflicting operations are done in the same order everywhere
Conflicting operations: From the world of transactions
• Read–write conflict: a read operation and a write operation act
concurrently
• Write–write conflict: two concurrent write operations

• Issue: Guaranteeing global ordering on conflicting operations may be a

costly operation, downgrading scalability Solution: weaken
consistency requirements so that hopefully global synchronization can
be avoided
Data-centric Consistency Models
• Continuous consistency
• Consistent ordering of operations
• Eventual consistency
Data-Centric Consistency Model
• Consistency model: A contract between a (distributed) data store and
processes, in which the data store specifies precisely what the
results of read and write operations are in the presence of
concurrency.
• Essential: A data store is a distributed collection of storages:
Process Process Process

Local Copy
Data-Centric Consistency Model:
Continuous Consistency
We can actually talk about a degree of consistency
• replicas may differ in their numerical value
• replicas may differ in their relative staleness
• there may be differences with respect to (number and order) of
performed update operations

• Conit: Consistency unit ⇒ specifies the data unit over

which consistency is to be measured.
Data-Centric Consistency Model:
Continuous Consistency: Conit
Example
Conit (contains the variables g, p,
and d ) Replica A

• Each replica has a vector clock:

([known] time @ A, [known]
time @ B)
• B sends A operation [(5, B) : g ←
g + 45]; A has made this
operation permanent (cannot be
rolled back)
Data-Centric Consistency Model:
Continuous Consistency: Conit
Example
Conit (contains the variables g, p,
and d ) Replica A

⇒ order deviation = 3
• A has three pending operations

max diff is 70 + 412 units ⇒ (2,

• A missed two operations from B;

482)
Data-Centric Consistency Model:
Consistent Ordering Operations
Some Notations: Read and write operations
• Wi(x)a: Process Pi writes value a to data item x
• Ri(x)b: Process Pi reads value b from data item x
• All data items initially have value NIL

Possible behavior
We omit index when possible and draw according to time (x-axis):

P1: W(x)a
P2: R(x)NIL R(x)a
Data-Centric Consistency Model:
Consistent Ordering Operations:
Sequential Consistency
Definition
The result of any execution is the same as if the operations of all processes
were executed in some sequential order, and the operations of each
individual process appear in this sequence in the order specified by its
program.
(a) A sequentially consistent data store. (b) A data store that is not
sequentially consistent
P1: W(x)a P1: W(x)a

P2: W(x)b P2: W(x)b

P3: R(x)b R(x)a P3: R(x)b
R(x)a
P4: R(x)b R(x)a P4:
R(x)a R(x)b
Data-Centric Consistency Model:
Consistent Ordering Operations:
Sequential
Example Consistency
Three concurrent processes (initial values: 0)
Process P1 Process P2 Process P3
x ← 1; y ← 1; z ← 1;
print(y,z); print(x,z); print(x,y);
Data-Centric Consistency Model:
Consistent Ordering Operations:
Example
Three concurrent processes (initial values: 0)
Process P1 Process P2 Process P3
x ← 1; y ← 1; z ← 1;
print(y,z); print(x,z); print(x,y);
Example execution sequences
(signature
Execution1 = [output P1 output Execution2 Execution3 Execution4
P2 output P3 ]
P1: x ← 1; P1: x ← 1; P2 : y ← 1; P2 : y ← 1;
P1: print(y,z); P2 : y ← 1; P3 : z ← 1; P1: x ← 1;
P2 : y ← 1; P2 : print(x,z); P3 : print(x,y); P3 : z ← 1;
P2 : print(x,z); P1: print(y,z); P2 : print(x,z); P2 : print(x,z);
P3 : z ← 1; P3 : z ← 1; P1: x ← 1; P1: print(y,z);
P3 : print(x,y); P3 : print(x,y); P1: print(y,z); P3 : print(x,y);

Prints: 001011 Prints: 101011 Prints: 010111 Prints: 111111

Signature: 0 0 1 0 1 1 Signature: 1 0 1 0 1 1 Signature: 0 1 0 1 1 1 Signature: 1 1 1 1 1 1
Data-Centric Consistency Model:
Consistent Ordering Operations:
How tricky can it get?
Seemingly okay
P1: W(x)a
W(y)a R(x)a
P2: W(y)b W(x)b
R(y)b
Data-Centric Consistency Model:
Consistent Ordering Operations:
How tricky can it get?
Seemingly okay
P1: W(x)a
W(y)a R(x)a
P2: W(y)b W(x)b R(y)b
But not really
Ordering of Operations Result
W1(x )a; W1(y )a; W2 (y )b; W2 (x )b R1(x )b R2 (y )b
W1(x )a; W2 (y )b; W1(y )a; W2 (x )b R1(x )b R2 (y )a
W1(x )a; W2 (y )b; W2 (x )b; W1(y )a R1(x )b R2 (y )a
W2 (y )b; W1(x )a; W1(y )a; W2 (x )b R1(x )b R2 (y )a
W2 (y )b; W1(x )a; W2 (x )b; W1(y )a R1(x )b R2 (y )a
W2 (y )b; W2 (x )b; W1(x )a; W1(y )a R1(x )a R2 (y )a
Data-Centric Consistency Model:
Consistent Ordering Operations:
How tricky can it get?
Linearizability
Each operation should appear to take effect instantaneously at some
moment between its start and completion.
Operations complete within a given time (red fonts)
P1: W(x)a W(y)a R(x)b
P2: W(y)b W(x)b R(y)a
Ordering of Operations Result
With better results W (x )a; W (y )b; W (y )a; W
1 2 1 2R (x )b R (y )a
1 2
(x )b R1(x )b R2 (y )a
W1(x )a; W2 (y )b; W2 (x )b; R1(x )b R2 (y )a
W1(y )a R1(x )b R2 (y )a
W2 (y )b; W1(x )a; W1(y )a; W2
Data-Centric Consistency Model:
Consistent Ordering Operations:
Causal
Definition Consistency
Writes that are potentially causally related must be seen by all
processes in the same order. Concurrent writes may be seen in a
different order by different processes.

P1: W(x)a W(x)c

P2: R(x)a W(x)b
P3: R(x)a R(x)c R(x)b
P4: R(x)a R(x)b R(x)c
Data-Centric Consistency Model:
Consistent Ordering Operations:
Causal Consistency
Definition
Writes that are potentially causally related must be seen by all processes in
the same order. Concurrent writes may be seen in a different order by
different processes.
(a) A violation of a causally-consistent store. (b) A correct sequence of
events in a causally-consistent store
P1: W(x)a P1: W(x)a
P2: R(x)a W(x)b P2: W(x)b
P3: R(x)b R(x)a P3 : R(x)b R(x)a
P4: R(x)a R(x)b P4: R(x)a R(x)b
(a) (b)
Data-Centric Consistency Model:
Consistent Ordering Operations:
Grouping
Entry Operations
consistency: Definition
• Accesses to locks are sequentially consistent.
• No access to a lock is allowed to be performed until all previous
writes have completed everywhere.
• No data access is allowed to be performed until all previous
accesses to
locks have been performed.
Data-Centric Consistency Model:
Consistent Ordering Operations:
Grouping
Entry Operations
consistency: Definition
• Accesses to locks are sequentially consistent.
• No access to a lock is allowed to be performed until all previous
writes have been completed everywhere.
• No data access is allowed to be performed until all previous accesses
to locks have been performed.

• Basic idea: You don't care that reads and writes of a series of
operations are immediately known to other processes. You just want
the effect of the series itself to be known.
Data-Centric Consistency Model:
Consistent Ordering Operations:
Grouping
A Operations
valid event sequence for entry consistency
P1: L(x) W(x)a L(y) W(y)b U(x) U(y)
P2: L(x) R(x)a
R(y) NIL
P3: L(y)
R(y)b

• Observation: Entry consistency implies that we need to lock and

unlock data (implicitly or not).
• Question: What would be a convenient way of making this
consistency more or less transparent to programmers?
Data-Centric Consistency Model:
Eventual Consistency
• write-write conflicts), never occur
• Only read-write conflicts
• In this case, it is acceptable to propagate an update in a lazy fashion
• Can tolerate a relatively high degree of inconsistency
• if no updates take place for a long time, all replicas will gradually
become consistent, that is, have exactly the same data stored -
eventual consistency
Client-centric Consistency Models
• Monotonic reads
• Monotonic writes
• Read your writes
• Writes follow reads
Client-Centric Consistency Model:
Consistency for Mobile Users
Example: Consider a distributed database to which you have access through
your notebook. Assume your notebook acts as a front end to the database.
• At location A you access the database doing reads and updates.
• At location B you continue your work, but unless you access the
same server as the one at location A, you may detect
inconsistencies:
• your updates at A may not have yet been propagated to B
• you may be reading newer entries than the ones available at A
• your updates at B may eventually conflict with those at A

Note: The only thing you really want is that the entries you updated and/or
read at A, are in B the way you left them in A. In that case, the database will
appear to be consistent to you.
Client-Centric Consistency
Architect
Model: BasicClient moves to other location and
(transparently) connects to other replica
ure

The principle of a mobile user accessing different replicas of a

distributed database
Client-Centric Consistency Model:
Monotonic Reads
• Definition: If a process reads the value of a data item x, any successive
read operation on x by that process will always return that same or
a more recent value.
• Notations:
•W1(x2) is the write operation by process P1 that leads to version x2 of x
•W1(xi; xj) indicates P1 produces version xj based on a previous version
xi .
•W1(xi|xj) indicates P1 produces version xj concurrently to version xi.
Client-Centric Consistency Model:
Monotonic Reads
• Definition: If a process reads the value of a data item x, any successive
read operation on x by that process will always return that same or a
more recent value.
• The read operations performed by a single process P at two different
local copies of the same data store. (a) A monotonic-read
consistent data store. (b) A data store that does not provide
monotonic
L1: W1(x1) reads R1(x1) L1: W1(x1) R1(x1)
L2: W2(x1;x2) R1(x2) L2: W2(x1| R1(x2)
x2 )
(a) (b)
Client-Centric Consistency Model:
Monotonic Reads
• Example: Automatically reading your personal calendar updates from
different servers. Monotonic Reads guarantees that the user sees
all updates, no matter from which server the automatic
reading takes place.
• Example: Reading (not modifying) incoming mail while you are on the
move. Each time you connect to a different e-mail server, that
server fetches (at least) all the updates from the server
you previously visited.
Client-Centric Consistency Model:
Monotonic Writes
• Definition: A write operation by a process on a data item x is completed before
any successive write operation on x by the same process.
(a) A monotonic-write consistent data store. (b) A data store that does not
provide
monotonic-write consistency. (c) Again, no consistency as WS(x1|x2) and thus also
WS(x1|x3 ). (d) Consistent as WS(x1; x3) although x1 has apparently overwritten x2.
L1: W1(x1) L1: W1(x1)
L2: W2(x1; x2) W1(x2; x3) L2: W2(x1|x2) W1(x1|x3)
(a) (b)
L1: W1(x1) L1: W1(x1)
L2: W2(x1|x2) W1(x2; x3) L2: W2(x1|x2) W1(x1; x3)
(c) (d)
Client-Centric Consistency Model:
Monotonic Writes
• Example: Updating a program at server S2 , and ensuring that all
components on which compilation and linking depends, are
also placed at S2 .
• Example: Maintaining versions of replicated files in the correct order
everywhere (propagate the previous version to the server where
the newest version is installed).
Client-Centric Consistency Model:
Read Your Writes
• Definition: The effect of a write operation by a process on data item x , will
always be seen by a successive read operation on x by the same
process.
(a) A data store that provides read-your-writes consistency.
(b) A data store that does not.

L1:
L2: W1(xW1)2(x1; x2) R1(x2) L1: W1(xW1) (x |x ) R (x )
2 1 2 1 2
L2: (b)
(a) your Web page and guaranteeing that your Web browser
Example: Updating
shows the newest version instead of its cached copy.
Client-Centric Consistency
Model: Writes follows Reads
• Definition
• A write operation by a process on a data item x following a previous read
operation on x by the same process, is guaranteed to take place on the
same or a more recent value of x that was read.
(a) A writes-follow-reads consistent data store. (b) A data store that does not
provide writes-follow-reads consistency
L1: W1(x1) R2(x1) L1: W1(x1) R2(x1)
L2: W3(x1; W2(x2; x3) L2: W3(x1| W2(x2| x3)
x2) (a) x2) (b)
Example: See reactions to posted articles only if you have the original posting
(a read "pulls in" the corresponding write operation).
Replica Management
• Finding the best server location
• Content replication and placement
• Content distribution
• Managing replicated objects
Replica Management: Finding The
Best Server Location
Essence: Figure out what the best K places are out of N possible
locations.
• Select best location out of N −K for which the average distance to
clients is minimal. Then choose the next best server. (Note: The
first chosen location minimizes the average distance to all clients.)
Computationally expensive.
• Select the K-th largest autonomous system and place a server at the
best-connected host. Computationally expensive.
• Position nodes in a d-dimensional geometric space, where distance
reflects latency. Identify the K regions with the highest density
and place a server in every one. Computationally cheap.
Replica Management: Content
Replication and Placement
The logical organization of different kinds of copies of a data store into
three concentric rings
Replica Management: Content
Replication and Placement
Distinguish different processes: A process is capable of hosting a replica
of an object or data:
• Permanent replicas: Process/machine always having a replica
• Server-initiated replica: Process that can dynamically host a replica on
request of another server in the data store
• Client-initiated replica: Process that can dynamically host a replica on
request of a client (client cache)
Replica Management: Content
Replication and Placement: Server-
Initiated Replicas
• Keep track of access counts per Server without
C2

copy of file F
file, aggregated by
considering server closest to
requesting clients

threshold D ⇒ drop file

• Number of accesses drops below

threshold R ⇒ replicate file

• Number of accesses exceeds
Counting access requests from different clients

and R ⇒ migrate file

• Number of access between D
Replica Management: Content
Distribution
Consider only a client-server combination
• Propagate only notification/invalidation of update (often used for
caches)
• Transfer data from one copy to another (distributed databases):
passive replication
• Propagate the update operation to other copies: active
replication

Note: No single approach is the best, but depends highly on available

bandwidth and read-to-write ratio at replicas.
Replica Management: Content
Distribution: Client/Server System:
Pull
A vs Push
comparison Protocols
between push-based and pull-based protocols in the
case of multiple-client, single-server systems
• Pushing updates: server-initiated approach, in which update is
propagated regardless whether target asked for it.
• Pulling updates: client-initiated approach, in which client requests to
be updated.
Issue Push-based Pull-based
State at server List of client caches None
Update (and possibly fetch Update (and possibly fetch update) Poll and update
update)
Immediate (or fetch-update Immediate (or fetch-update time) Fetch-update time
time)
Replica Management: Content
Distribution: Pull vs Push Protocols
Observation: We can dynamically switch between pulling and pushing using
leases: A contract in which the server promises to push updates to the
client until the lease expires.
Make lease expiration time dependent on system's behavior (adaptive
leases):
• Age-based leases: An object that hasn’t changed for a long time, will not
change in the near future, so provide a long-lasting lease
• Renewal-frequency based leases: The more often a client requests a
specific object, the longer the expiration time for that client (for
that object) will be
• State-based leases: The more loaded a server is, the shorter the
expiration times become
Question: Why are we doing all this?
Thank
You
Questions?
Consistency Protocols
• Continuous consistency
• Sequential consistency: Primary-based protocols
• Sequential consistency: Replicated-write protocols
• Cache-coherence protocols
• Implementing client-centric consistency
Consistency Protocol: Continuous
Consistency Bounding Numerical
Deviation:
Principal operation Numerical Errors
•Every server Si has a log, denoted as Li.
• Consider a data item x and let val(W) denote the numerical change in

∀W : val(W ) > 0
its value after a write operation W. Assume that

• W is initially forwarded to one of the N replicas, denoted as

origin(W).

Sj: TW[i, j] = ∑{val(W)|origin(W) = Sj & W ∈ Li }

•TW[i, j] are the writes executed by server Si that originated from
Consistency Protocol: Continuous
Consistency Bounding Numerical
Deviation:
Actual value v(t) of Numerical Errors
𝑁
x:

𝑣 = 𝑣𝑖𝑛𝑖𝑡 + ෍
𝑡 𝑇𝑊[𝑘, 𝑘]
𝑘=1
Value vi of x at server Si

𝑣𝑖 = 𝑣𝑖𝑛𝑖𝑡 + ෍
�
�

𝑇𝑊[𝑖, 𝑘]
𝑘=1
Consistency Protocol: Continuous
Consistency Bounding Numerical
Deviation:
•Problem: We needNumerical
to ensure that v(t)Errors
−vi < δi for every server Si.
•Approach: Let every server Sk maintain a view TWk[i, j] of what it
believes is the value of TW[i, j]. This information can be
gossiped when an update is propagated.
•Note: 0 ≤ TWk[i, j] ≤ TW[i, j] ≤ TW[j, j]
Consistency Protocol: Continuous
Consistency Bounding Numerical
Deviation:
• Solution: S sends Numerical Errors
operations from its log to S when it sees that TW [i,
k i k
k] is getting too far from TW[k, k], in particular, when TW[k, k] −
TWk[i, k] > δi /(N − 1)

• Push protocol
Consistency Protocol: Continuous
Consistency: Bounding staleness
deviations
•S keep a real-time vector clock RVC
k k

•RVCk[i] = ti means that Sk has seen all writes that have been submitted
to Si up to time ti, ti denotes the time local to Si.
•Server Sk notes that tk − RVCk[i] is about to exceed a specified limit,
pulling in writes that originated from Si with a timestamp later
than RVCk[i].
• Pull protocol
Consistency Protocol: Continuous
Consistency: Bounding ordering
deviations
• replica server tentatively applies updates to the local copy
• actual order in which they are to be applied to the local copy of x still
needs to be determined
• ordering deviation is bounded by specifying the maximal length of the
queue of tentative writes
Consistency Protocol: Sequential
Consistency: Primary-based Protocols:
Primary Backup Protocol: Remote-Write
Client
Primary server

Example primary-backup protocol

Traditionally applied in distributed databases and file systems
that require a high degree of fault tolerance. Replicas are often
placed on same LAN.
Consistency Protocol: Sequential
Consistency: Primary-based Protocols:
Primary Backup Protocol: Local Writes
Client Client

Example primary-backup protocol with local writes

Mobile computing in disconnected mode (ship all relevant files
to user before disconnecting, and update later on).
Consistency Protocol: Sequential
Consistency: Replicated-write
protocols: Quorum-based protocols
Quorum-based protocols: Ensure that each operation is carried out in such a
way that a majority vote is established: distinguish read quorum and write
quorum
Three examples of the voting algorithm. (a) A correct choice of read and
write set. (b) A choice that may lead to write-write conflicts. (c) A correct
choice, known as ROWA (read one, write all)
A B C D A B C D A B C D

E F G H E F G H E F G H

I J K L I J K L I J K L

NR = 3, NW = 10 NR = 7, NW = 6 NR = 1, NW = 12