Distributed Systems

Principles and Paradigms

Maarten van Steen

VU Amsterdam, Dept. Computer Science

Room R4.20, [email protected]

Chapter 04: Communication

Version: November 5, 2009

1 / 36

Contents

Chapter
01: Introduction
02: Architectures
03: Processes
04: Communication
05: Naming
06: Synchronization
07: Consistency & Replication
08: Fault Tolerance
09: Security
10: Distributed Object-Based Systems
11: Distributed File Systems
12: Distributed Web-Based Systems
13: Distributed Coordination-Based Systems

2 / 36 2 / 36

Communication 4.1 Layered Protocols Communication 4.1 Layered Protocols

Layered Protocols

Low-level layers
Transport layer
Application layer
Middleware layer

3 / 36 3 / 36
 Communication 4.1 Layered Protocols Communication 4.1 Layered Protocols

Basic networking model

Application protocol
Application 7
Presentation protocol
Presentation 6
Session protocol
Session 5
Transport protocol
Transport 4
Network protocol
Network 3
Data link protocol
Data link 2
Physical protocol
Physical 1

Network

Drawbacks
Focus on message-passing only
Often unneeded or unwanted functionality
Violates access transparency

4 / 36 4 / 36

Communication 4.1 Layered Protocols Communication 4.1 Layered Protocols

Low-level layers

Recap
Physical layer: contains the specification and implementation of
bits, and their transmission between sender and receiver
Data link layer: prescribes the transmission of a series of bits into
a frame to allow for error and flow control
Network layer: describes how packets in a network of computers
are to be routed.

Observation
For many distributed systems, the lowest-level interface is that of the
network layer.

5 / 36 5 / 36

Communication 4.1 Layered Protocols Communication 4.1 Layered Protocols

Transport Layer

Important
The transport layer provides the actual communication facilities for
most distributed systems.

Standard Internet protocols

TCP: connection-oriented, reliable, stream-oriented
communication
UDP: unreliable (best-effort) datagram communication

Note
IP multicasting is often considered a standard available service (which
may be dangerous to assume).

6 / 36 6 / 36
 Communication 4.1 Layered Protocols Communication 4.1 Layered Protocols

Middleware Layer

Observation
Middleware is invented to provide common services and protocols
that can be used by many different applications

A rich set of communication protocols

(Un)marshaling of data, necessary for integrated systems
Naming protocols, to allow easy sharing of resources
Security protocols for secure communication
Scaling mechanisms, such as for replication and caching

Note
What remains are truly application-specific protocols...
such as?

7 / 36 7 / 36

Communication 4.1 Layered Protocols Communication 4.1 Layered Protocols

Types of communication
Synchronize at Synchronize at Synchronize after
request submission request delivery processing by server
Client

Request

Transmission
interrupt
Storage
facility

Reply

Server Time

Distinguish
Transient versus persistent communication
Asynchrounous versus synchronous communication

8 / 36 8 / 36

Communication 4.1 Layered Protocols Communication 4.1 Layered Protocols

Types of communication
Synchronize at Synchronize at Synchronize after
request submission request delivery processing by server
Client

Request

Transmission
interrupt
Storage
facility

Reply

Server Time

Transient versus persistent

Transient communication: Comm. server discards message when it
cannot be delivered at the next server, or at the receiver.
Persistent communication: A message is stored at a communication
server as long as it takes to deliver it.
9 / 36 9 / 36
 Communication 4.1 Layered Protocols Communication 4.1 Layered Protocols

Types of communication
Synchronize at Synchronize at Synchronize after
request submission request delivery processing by server
Client

Request

Transmission
interrupt
Storage
facility

Reply

Server Time

Places for synchronization

At request submission
At request delivery
After request processing
10 / 36 10 / 36

Communication 4.1 Layered Protocols Communication 4.1 Layered Protocols

Client/Server

Some observations
Client/Server computing is generally based on a model of transient
synchronous communication:
Client and server have to be active at time of commun.
Client issues request and blocks until it receives reply
Server essentially waits only for incoming requests, and
subsequently processes them

Drawbacks synchronous communication

Client cannot do any other work while waiting for reply
Failures have to be handled immediately: the client is waiting
The model may simply not be appropriate (mail, news)

11 / 36 11 / 36

Communication 4.1 Layered Protocols Communication 4.1 Layered Protocols

Messaging

Message-oriented middleware
Aims at high-level persistent asynchronous communication:
Processes send each other messages, which are queued
Sender need not wait for immediate reply, but can do other things
Middleware often ensures fault tolerance

12 / 36 12 / 36
 Communication 4.2 Remote Procedure Call Communication 4.2 Remote Procedure Call

Remote Procedure Call (RPC)

Basic RPC operation

Parameter passing
Variations

13 / 36 13 / 36

Communication 4.2 Remote Procedure Call Communication 4.2 Remote Procedure Call

Basic RPC operation

Observations
Application developers are familiar with simple procedure model
Well-engineered procedures operate in isolation (black box)
There is no fundamental reason not to execute procedures on
separate machine

Wait for result

Conclusion Client

Communication between caller & Call remote

procedure
Return
from call
callee can be hidden by using
procedure-call mechanism. Request Reply
Server
Call local procedure Time
and return results

14 / 36 14 / 36

Communication 4.2 Remote Procedure Call Communication 4.2 Remote Procedure Call

Basic RPC operation

Client machine Server machine

Client process Server process

1. Client call to
procedure Implementation 6. Stub makes
of add local call to "add"
Server stub
k = add(i,j) k = add(i,j)
Client stub
proc: "add" proc: "add"
int: val(i) int: val(i) 5. Stub unpacks
2. Stub builds message
int: val(j) message int: val(j)

proc: "add" 4. Server OS

Client OS int: val(i) Server OS hands message
int: val(j) to server stub

3. Message is sent
across the network

1 Client procedure calls client stub. 6 Server returns result to stub.

2 Stub builds message; calls local OS. 7 Stub builds message; calls OS.
3 OS sends message to remote OS. 8 OS sends message to client’s OS.
4 Remote OS gives message to stub. 9 Client’s OS gives message to stub.
5 Stub unpacks parameters and calls 10 Client stub unpacks result and returns to
server. the client.
15 / 36 15 / 36
 Communication 4.2 Remote Procedure Call Communication 4.2 Remote Procedure Call

RPC: Parameter passing

Parameter marshaling
There’s more than just wrapping parameters into a message:
Client and server machines may have different data
representations (think of byte ordering)
Wrapping a parameter means transforming a value into a
sequence of bytes
Client and server have to agree on the same encoding:
How are basic data values represented (integers, floats, characters)
How are complex data values represented (arrays, unions)
Client and server need to properly interpret messages,
transforming them into machine-dependent representations.

16 / 36 16 / 36

Communication 4.2 Remote Procedure Call Communication 4.2 Remote Procedure Call

RPC: Parameter passing

RPC parameter passing: some assumptions

Copy in/copy out semantics: while procedure is executed, nothing can
be assumed about parameter values.
All data that is to be operated on is passed by parameters. Excludes
passing references to (global) data.

Conclusion
Full access transparency cannot be realized.

Observation
A remote reference mechanism enhances access transparency:
Remote reference offers unified access to remote data
Remote references can be passed as parameter in RPCs

17 / 36 17 / 36

Communication 4.2 Remote Procedure Call Communication 4.2 Remote Procedure Call

Asynchronous RPCs

Essence
Try to get rid of the strict request-reply behavior, but let the client
continue without waiting for an answer from the server.

Client Wait for result Client Wait for acceptance

Call remote Return Call remote Return

procedure from call procedure from call

Request Request Accept request

Reply

Server Call local procedure Time Server Call local procedure Time
and return results
(a) (b)

18 / 36 18 / 36
 Communication 4.2 Remote Procedure Call Communication 4.2 Remote Procedure Call

Deferred synchronous RPCs

Wait for Interrupt client

acceptance
Client

Call remote Return

procedure from call Return
results Acknowledge
Accept
Request request
Server
Call local procedure Time
Call client with
one-way RPC

Variation
Client can also do a (non)blocking poll at the server to see whether
results are available.

19 / 36 19 / 36

Communication 4.2 Remote Procedure Call Communication 4.2 Remote Procedure Call

RPC in practice
Uuidgen

Interface
definition file

IDL compiler

Client code Client stub Header Server stub Server code

#include #include

C compiler C compiler C compiler C compiler

Client Client stub Server stub Server

object file object file object file object file

Runtime Runtime
Linker Linker
library library

Client Server
binary binary
20 / 36 20 / 36

Communication 4.2 Remote Procedure Call Communication 4.2 Remote Procedure Call

Client-to-server binding (DCE)

Issues
(1) Client must locate server machine, and (2) locate the server.

Directory machine

Directory
server
2. Register service
3. Look up server
Server machine
Client machine

5. Do RPC 1. Register endpoint

Server
Client

4. Ask for endpoint DCE

daemon Endpoint
table

21 / 36 21 / 36
 Communication 4.3 Message-Oriented Communication Communication 4.3 Message-Oriented Communication

Message-Oriented Communication

Transient Messaging
Message-Queuing System
Message Brokers
Example: IBM Websphere

22 / 36 22 / 36

Communication 4.3 Message-Oriented Communication Communication 4.3 Message-Oriented Communication

Transient messaging: sockets

Berkeley socket interface

SOCKET Create a new communication endpoint
BIND Attach a local address to a socket
LISTEN Announce willingness to accept N connections
ACCEPT Block until request to establish a connection
CONNECT Attempt to establish a connection
SEND Send data over a connection
RECEIVE Receive data over a connection
CLOSE Release the connection

23 / 36 23 / 36

Communication 4.3 Message-Oriented Communication Communication 4.3 Message-Oriented Communication

Transient messaging: sockets

Server
socket bind listen accept read write close

Synchronization point Communication

socket connect write read close

Client

24 / 36 24 / 36
 Communication 4.3 Message-Oriented Communication Communication 4.3 Message-Oriented Communication

Message-oriented middleware

Essence
Asynchronous persistent communication through support of
middleware-level queues. Queues correspond to buffers at
communication servers.

PUT Append a message to a specified queue

GET Block until the specified queue is nonempty, and re-
move the first message
POLL Check a specified queue for messages, and remove
the first. Never block
NOTIFY Install a handler to be called when a message is put
into the specified queue

25 / 36 25 / 36

Communication 4.3 Message-Oriented Communication Communication 4.3 Message-Oriented Communication

Message broker

Observation
Message queuing systems assume a common messaging protocol: all
applications agree on message format (i.e., structure and data
representation)

Message broker
Centralized component that takes care of application heterogeneity in
an MQ system:
Transforms incoming messages to target format
Very often acts as an application gateway
May provide subject-based routing capabilities ⇒ Enterprise
Application Integration

26 / 36 26 / 36

Communication 4.3 Message-Oriented Communication Communication 4.3 Message-Oriented Communication

Message broker

Repository with
conversion rules
Source client Message broker and programs Destination client

Broker
program

Queuing
layer
OS OS OS

Network

27 / 36 27 / 36
 Communication 4.3 Message-Oriented Communication Communication 4.3 Message-Oriented Communication

IBM’s WebSphere MQ

Basic concepts
Application-specific messages are put into, and removed from
queues
Queues reside under the regime of a queue manager
Processes can put messages only in local queues, or through an
RPC mechanism

28 / 36 28 / 36

Communication 4.3 Message-Oriented Communication Communication 4.3 Message-Oriented Communication

IBM’s WebSphere MQ

Message transfer
Messages are transferred between queues
Message transfer between queues at different processes, requires
a channel
At each endpoint of channel is a message channel agent
Message channel agents are responsible for:
Setting up channels using lower-level network communication
facilities (e.g., TCP/IP)
(Un)wrapping messages from/in transport-level packets
Sending/receiving packets

29 / 36 29 / 36

Communication 4.3 Message-Oriented Communication Communication 4.3 Message-Oriented Communication

IBM’s WebSphere MQ
Client's receive
Routing table Send queue queue Receiving client
Sending client

Queue Queue
Program manager manager Program

MQ Interface

Server Server
Stub MCA MCA MCA MCA Stub
stub stub

RPC Local network

(synchronous) Enterprise network
To other remote
Message passing queue managers
(asynchronous)

Channels are inherently unidirectional

Automatically start MCAs when messages arrive
Any network of queue managers can be created
Routes are set up manually (system administration)
30 / 36 30 / 36
 Communication 4.3 Message-Oriented Communication Communication 4.3 Message-Oriented Communication

IBM’s WebSphere MQ
Routing
By using logical names, in combination with name resolution to local queues,
it is possible to put a message in a remote queue

Alias table Routing table

LA1 QMC QMB SQ1 Alias table Routing table
LA2 QMD QMC SQ1 LA1 QMA QMA SQ1
QMD SQ2 LA2 QMD QMC SQ1
QMD SQ1
SQ2
SQ1
QMA SQ1
QMB

Routing table SQ1 QMC Routing table

QMA SQ1
QMA SQ1
QMC SQ2 SQ2 QMB SQ1
QMB SQ1
QMD SQ1
Alias table
LA1 QMA SQ1
LA2 QMC
QMD

31 / 36 31 / 36

Communication 4.4 Stream-Oriented Communication Communication 4.4 Stream-Oriented Communication

Stream-oriented communication

Support for continuous media

Streams in distributed systems
Stream management

32 / 36 32 / 36

Communication 4.4 Stream-Oriented Communication Communication 4.4 Stream-Oriented Communication

Continuous media

Observation
All communication facilities discussed so far are essentially based on a
discrete, that is time-independent exchange of information

Continuous media
Characterized by the fact that values are time dependent:
Audio
Video
Animations
Sensor data (temperature, pressure, etc.)

33 / 36 33 / 36
 Communication 4.4 Stream-Oriented Communication Communication 4.4 Stream-Oriented Communication

Continuous media

Transmission modes
Different timing guarantees with respect to data transfer:
Asynchronous: no restrictions with respect to when data is to be
delivered
Synchronous: define a maximum end-to-end delay for individual
data packets
Isochronous: define a maximum and minimum end-to-end delay
(jitter is bounded)

34 / 36 34 / 36

Communication 4.4 Stream-Oriented Communication Communication 4.4 Stream-Oriented Communication

Stream

Definition
A (continuous) data stream is a connection-oriented communication
facility that supports isochronous data transmission.

Some common stream characteristics

Streams are unidirectional
There is generally a single source, and one or more sinks
Often, either the sink and/or source is a wrapper around hardware
(e.g., camera, CD device, TV monitor)
Simple stream: a single flow of data, e.g., audio or video
Complex stream: multiple data flows, e.g., stereo audio or
combination audio/video

35 / 36 35 / 36

Communication 4.4 Stream-Oriented Communication Communication 4.4 Stream-Oriented Communication

Streams and QoS

Essence
Streams are all about timely delivery of data. How do you specify this
Quality of Service (QoS)? Basics:
The required bit rate at which data should be transported.
The maximum delay until a session has been set up (i.e., when an
application can start sending data).
The maximum end-to-end delay (i.e., how long it will take until a
data unit makes it to a recipient).
The maximum delay variance, or jitter.
The maximum round-trip delay.

36 / 36 36 / 36
 Communication 4.4 Stream-Oriented Communication Communication 4.4 Stream-Oriented Communication

Enforcing QoS

Observation
There are various network-level tools, such as differentiated services
by which certain packets can be prioritized.

Also
Use buffers to reduce jitter:
Packet departs source 1 2 3 4 5 6 7 8

Packet arrives at buffer 1 2 3 4 5 6 7 8

Time in buffer
Packet removed from buffer 1 2 3 4 5 6 7 8
Gap in playback
0 5 10 15 20
Time (sec)

37 / 36 37 / 36

Communication 4.4 Stream-Oriented Communication Communication 4.4 Stream-Oriented Communication

Enforcing QoS

Problem
How to reduce the effects of packet loss (when multiple samples are in
a single packet)?

38 / 36 38 / 36

Communication 4.4 Stream-Oriented Communication Communication 4.4 Stream-Oriented Communication

Enforcing QoS

Lost packet
Sent 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16

Delivered 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16

Gap of lost frames

(a)

Lost packet
Sent 1 5 9 13 2 6 10 14 3 7 11 15 4 8 12 16

Delivered 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16

Lost frames
(b)

39 / 36 39 / 36
 Communication 4.4 Stream-Oriented Communication Communication 4.4 Stream-Oriented Communication

Stream synchronization

Problem
Given a complex stream, how do you keep the different substreams in
synch?

Example
Think of playing out two channels, that together form stereo sound.
Difference should be less than 20–30 µsec!

40 / 36 40 / 36

Communication 4.4 Stream-Oriented Communication Communication 4.4 Stream-Oriented Communication

Stream synchronization

Receiver's machine

Application
Procedure that reads
two audio data units for
each video data unit

Incoming stream
OS

Network

Alternative
Multiplex all substreams into a single stream, and demultiplex at the
receiver. Synchronization is handled at multiplexing/demultiplexing
point (MPEG).

41 / 36 41 / 36

Communication 4.5 Multicast Communication Communication 4.5 Multicast Communication

Multicast communication

Application-level multicasting
Gossip-based data dissemination

42 / 36 42 / 36
 Communication 4.5 Multicast Communication Communication 4.5 Multicast Communication

Application-level multicasting

Essence
Organize nodes of a distributed system into an overlay network and use that
network to disseminate data.

Chord-based tree building

1 Initiator generates a multicast identifier mid.
2 Lookup succ(mid), the node responsible for mid.
3 Request is routed to succ(mid), which will become the root.
4 If P wants to join, it sends a join request to the root.
5 When request arrives at Q:
Q has not seen a join request before ⇒ it becomes forwarder; P
becomes child of Q. Join request continues to be forwarded.
Q knows about tree ⇒ P becomes child of Q. No need to forward
join request anymore.

43 / 36 43 / 36

Communication 4.5 Multicast Communication Communication 4.5 Multicast Communication

ALM: Some costs

End host
Router
A 1 1 C
30 20
Ra Re Rc

5
7
40 Rd
1 Rb 1
Internet D
B
Overlay network

Link stress: How often does an ALM message cross the same
physical link? Example: message from A to D needs to cross
hRa, Rbi twice.
Stretch: Ratio in delay between ALM-level path and network-level
path. Example: messages B to C follow path of length 71 at ALM,
but 47 at network level ⇒ stretch = 71/47.
44 / 36 44 / 36

Communication 4.5 Multicast Communication Communication 4.5 Multicast Communication

Epidemic Algorithms

General background
Update models
Removing objects

45 / 36 45 / 36
 Communication 4.5 Multicast Communication Communication 4.5 Multicast Communication

Principles

Basic idea
Assume there are no write–write conflicts:
Update operations are performed at a single server
A replica passes updated state to only a few neighbors
Update propagation is lazy, i.e., not immediate
Eventually, each update should reach every replica

Two forms of epidemics

Anti-entropy: Each replica regularly chooses another replica at random,
and exchanges state differences, leading to identical states at both
afterwards
Gossiping: A replica which has just been updated (i.e., has been
contaminated), tells a number of other replicas about its update
(contaminating them as well).

46 / 36 46 / 36

Communication 4.5 Multicast Communication Communication 4.5 Multicast Communication

Anti-entropy

Principle operations
A node P selects another node Q from the system at random.
Push: P only sends its updates to Q
Pull: P only retrieves updates from Q
Push-Pull: P and Q exchange mutual updates (after which they
hold the same information).

Observation
For push-pull it takes O(log(N)) rounds to disseminate updates to all
N nodes (round = when every node as taken the initiative to start an
exchange).

47 / 36 47 / 36

Communication 4.5 Multicast Communication Communication 4.5 Multicast Communication

Gossiping

Basic model
A server S having an update to report, contacts other servers. If a
server is contacted to which the update has already propagated, S
stops contacting other servers with probability 1/k.

Observation
If s is the fraction of ignorant servers (i.e., which are unaware of the
update), it can be shown that with many servers

s = e−(k+1)(1−s)

48 / 36 48 / 36
 Communication 4.5 Multicast Communication Communication 4.5 Multicast Communication

Gossiping

k
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 Consider 10,000 nodes
k s Ns
-2.5 1 0.203188 2032
-5.0 2 0.059520 595
3 0.019827 198
-7.5 4 0.006977 70
-10.0 5 0.002516 25
ln(s) 6 0.000918 9
-12.5
7 0.000336 3
-15.0

Note
If we really have to ensure that all servers are eventually updated,
gossiping alone is not enough

49 / 36 49 / 36

Communication 4.5 Multicast Communication Communication 4.5 Multicast Communication

Deleting values

Fundamental problem
We cannot remove an old value from a server and expect the removal
to propagate. Instead, mere removal will be undone in due time using
epidemic algorithms

Solution
Removal has to be registered as a special update by inserting a death
certificate

50 / 36 50 / 36

Communication 4.5 Multicast Communication Communication 4.5 Multicast Communication

Deleting values

Next problem
When to remove a death certificate (it is not allowed to stay for ever):
Run a global algorithm to detect whether the removal is known
everywhere, and then collect the death certificates (looks like garbage
collection)
Assume death certificates propagate in finite time, and associate a
maximum lifetime for a certificate (can be done at risk of not reaching all
servers)

Note
It is necessary that a removal actually reaches all servers.

Question
What’s the scalability problem here?

51 / 36 51 / 36
 Communication 4.5 Multicast Communication Communication 4.5 Multicast Communication

Example applications

Typical apps
Data dissemination: Perhaps the most important one. Note that
there are many variants of dissemination.
Aggregation: Let every node i maintain a variable xi . When two
nodes gossip, they each reset their variable to

xi , xj ← (xi + xj )/2

Result: in the end each node will have computed the average
x̄ = ∑i xi /N.

Question
What happens if initially xi = 1 and xj = 0, j 6= i?

52 / 36 52 / 36