CSE3009 - Parallel and Distributed Computing

Course Type: LTP Credits: 4

Prepared by
Dr Komarasamy G
Senior Associate Professor
School of Computing Science and Engineering
VIT Bhopal University
 Unit-3
• Introduction to Distributed Systems – Definition,
Issues, Goals, Types of distributed systems,
Distributed System Models, Hardware concepts,
Software Concept, Design Issues.
• Communication – Layered Protocols, Remote
Procedure Call, Remote Object Invocation,
Message Oriented Communication, Stream
Oriented Communication – Case Study (RPC and
Java RMI).
• Parallel Random Access Machine (PRAM) model,
PRAM architecture.
Unit-3 / Dr Komarasamy G 2
 Parallel Random Access Machine (PRAM) model
• Parallel Random Access Machine (PRAM) is a model considered
for most of the parallel algorithms.
• It helps to write a precursor parallel algorithm without any
architecture constraints and also allows parallel-algorithm
designers to treat processing power as unlimited.
• It ignores the complexity of inter-process communication.
• PRAM algorithms are mostly theoretical but can be used as a
basis for developing an efficient parallel algorithm for practical
machines and can also motivate building specialized machines.

Unit-3 / Dr Komarasamy G 3
 Parallel Random Access Machine (PRAM) model
• PRAM Architecture Model
The following are the modules of which a PRAM consists of:
• It consists of a control unit, global memory, and an unbounded
set of similar processors, each with its own private memory.
• An active processor reads from global memory, performs
required computation, and then writes to global memory.

Unit-3 / Dr Komarasamy G 4
 Parallel Random Access Machine (PRAM) model
PRAM Architecture Model

If there are N processors in a PRAM, then N number of independent

operations can be performed in a particular unit of time.

Unit-3 / Dr Komarasamy G 5
 Parallel Random Access Machine (PRAM) model
Models of PRAM
• While accessing the shared memory, there can be conflicts
while performing the read and write operation (i.e.), a
processor can access a memory block that is already being
accessed by another processor.
• Therefore, there are various constraints on a PRAM model
which handles the read or write conflicts. They are:
• EREW: also called Exclusive Read Exclusive Write is a constraint
that doesn’t allow two processors to read or write from the
same memory location at the same instance.
• CREW: also called Concurrent Read Exclusive Write is a
constraint that allows all the processors to read from the same
memory location but are not allowed to write into the same
memory location at the same time.
Unit-3 / Dr Komarasamy G 6
 Parallel Random Access Machine (PRAM) model
• ERCW: also called Exclusive Read Concurrent Write is a
constraint that allows all the processors to write to the same
memory location but are now allowed to read the same
memory location at the same time.
• CRCW: also called Concurrent Read Concurrent Write is a
constraint that allows all the processors to read from and write
to the same memory location parallelly.

Unit-3 / Dr Komarasamy G 7
 Parallel Random Access Machine (PRAM) model
• Example: Suppose we wish to add an array consisting of N
numbers.
• We generally iterate through the array and use N steps to find
the sum of the array.
• So, if the size of the array is N and for each step, let’s assume
the time taken to be 1 second.
• Therefore, it takes N seconds to complete the iteration. The
same operation can be performed more efficiently using a
CRCW model of a PRAM.
• Let there be N/2 parallel processors for an array of size N, then
the time taken for the execution is 4 which is less than N = 6
seconds.

Unit-3 / Dr Komarasamy G 8
 Parallel Random Access Machine (PRAM) model
• Example: Suppose we wish to add an array consisting of N
numbers.

Unit-3 / Dr Komarasamy G 9
Parallel Random Access Machine (PRAM) model

Unit-3 / Dr Komarasamy G 10
 Parallel Random Access Machine (PRAM) model

Example:
https://chatgpt.com/share/67d7a350-83e4-8002-b971-bc91a53aa176

Unit-3 / Dr Komarasamy G 11
 Parallel Random Access Machine (PRAM) model
• There are many methods to implement the PRAM model, but
the most prominent ones are −
1. Shared memory model
2. Message passing model
3. Data parallel model

Unit-3 / Dr Komarasamy G 12
 Parallel Random Access Machine (PRAM) model
• Shared Memory Model
• Shared memory emphasizes on control parallelism than
on data parallelism. In the shared memory model, multiple
processes execute on different processors independently, but
they share a common memory space. Due to any processor
activity, if there is any change in any memory location, it is
visible to the rest of the processors.
• As multiple processors access the same memory location, it
may happen that at any particular point of time, more than one
processor is accessing the same memory location.
• Suppose one is reading that location and the other is writing on
that location. It may create confusion. To avoid this, some
control mechanism, like lock / semaphore, is implemented to
ensure mutual exclusion.
Unit-3 / Dr Komarasamy G 13
 Parallel Random Access Machine (PRAM) model
• Shared Memory Model

Unit-3 / Dr Komarasamy G 14
 Parallel Random Access Machine (PRAM) model
• Shared memory programming has been implemented in the
following −
• Thread libraries − The thread library allows multiple threads of
control that run concurrently in the same memory location.
• Thread library provides an interface that supports
multithreading through a library of subroutine. It contains
subroutines for
– Creating and destroying threads
– Scheduling execution of thread
– passing data and message between threads
– saving and restoring thread contexts

Unit-3 / Dr Komarasamy G 15
 Parallel Random Access Machine (PRAM) model
• Distributed Shared Memory (DSM) Systems − DSM systems
create an abstraction of shared memory on loosely coupled
architecture in order to implement shared memory
programming without hardware support.
• They implement standard libraries and use the advanced
user-level memory management features present in modern
operating systems. Examples include Tread Marks System,
Munin, IVY, Shasta, Brazos, and Cashmere.
• Program Annotation Packages − This is implemented on the
architectures having uniform memory access characteristics.
• The most notable example of program annotation packages is
OpenMP.
• OpenMP implements functional parallelism. It mainly focuses
on parallelization of loops.
Unit-3 / Dr Komarasamy G 16
 Parallel Random Access Machine (PRAM) model
Merits of Shared Memory Programming
• Global address space gives a user-friendly programming
approach to memory.
• Due to the closeness of memory to CPU, data sharing among
processes is fast and uniform.
• There is no need to specify distinctly the communication of data
among processes.
• Process-communication overhead is negligible.
• It is very easy to learn.
Demerits of Shared Memory Programming
• It is not portable.
• Managing data locality is very difficult.

Unit-3 / Dr Komarasamy G 17
 Parallel Random Access Machine (PRAM) model
• Message Passing Model
• Message passing is the most commonly used parallel
programming approach in distributed memory systems.
• Here, the programmer has to determine the parallelism. In this
model, all the processors have their own local memory unit
and they exchange data through a communication network.

• Processors use message-passing libraries for communication

among themselves.
Unit-3 / Dr Komarasamy G 18
 Parallel Random Access Machine (PRAM) model
• Processors use message-passing libraries for communication
among themselves. Along with the data being sent, the
message contains the following components −
– The address of the processor from which the message is being
sent;
– Starting address of the memory location of the data in the
sending processor;
– Data type of the sending data;
– Data size of the sending data;
– The address of the processor to which the message is being sent;
– Starting address of the memory location for the data in the
receiving processor.

Unit-3 / Dr Komarasamy G 19
 Parallel Random Access Machine (PRAM) model
• Processors can communicate with each other by any of the
following methods −
1. Point-to-Point Communication
2. Collective Communication
3. Message Passing Interface
• Point-to-Point Communication
• Point-to-point communication is the simplest form of message
passing. Here, a message can be sent from the sending processor
to a receiving processor by any of the following transfer modes −
• Synchronous mode − The next message is sent only after the
receiving a confirmation that its previous message has been
delivered, to maintain the sequence of the message.
• Asynchronous mode − To send the next message, receipt of the
confirmation of the delivery of the previous message is not
required.
Unit-3 / Dr Komarasamy G 20
 Parallel Random Access Machine (PRAM) model
• Collective Communication
• Collective communication involves more than two processors
for message passing. Following modes allow collective
communications −
• Barrier − Barrier mode is possible if all the processors included
in the communications run a particular bock (known as barrier
block) for message passing.
• Broadcast − Broadcasting is of two types −
– One-to-all − Here, one processor with a single operation
sends same message to all other processors.
– All-to-all − Here, all processors send message to all other
processors.

Unit-3 / Dr Komarasamy G 21
 Parallel Random Access Machine (PRAM) model
• Messages broadcasted may be of three types −
• Personalized − Unique messages are sent to all other
destination processors.
• Non-personalized − All the destination processors receive the
same message.
• Reduction − In reduction broadcasting, one processor of the
group collects all the messages from all other processors in the
group and combine them to a single message which all other
processors in the group can access.

Unit-3 / Dr Komarasamy G 22
 Parallel Random Access Machine (PRAM) model
Merits of Message Passing
• Provides low-level control of parallelism;
• It is portable;
• Less error prone;
• Less overhead in parallel synchronization and data distribution.
Demerits of Message Passing
• As compared to parallel shared-memory code, message-passing
code generally needs more software overhead.

Unit-3 / Dr Komarasamy G 23
 Parallel Random Access Machine (PRAM) model
• Message Passing Libraries
• There are many message-passing libraries. Here, we will discuss
two of the most-used message-passing libraries −
– Message Passing Interface (MPI)
– Parallel Virtual Machine (PVM)
• It is a universal standard to provide communication among all
the concurrent processes in a distributed memory system. Most
of the commonly used parallel computing platforms provide at
least one implementation of message passing interface.
• It has been implemented as the collection of predefined
functions called library and can be called from languages such
as C, C++, Fortran, etc.
• MPIs are both fast and portable as compared to the other
message passing libraries.
Unit-3 / Dr Komarasamy G 24
 Parallel Random Access Machine (PRAM) model
• Merits of Message Passing Interface
• Runs only on shared memory architectures or distributed
memory architectures;
• Each processors has its own local variables;
• As compared to large shared memory computers, distributed
memory computers are less expensive.
• Demerits of Message Passing Interface
• More programming changes are required for parallel algorithm;
• Sometimes difficult to debug; and
• Does not perform well in the communication network between
the nodes.

Unit-3 / Dr Komarasamy G 25
 Parallel Random Access Machine (PRAM) model
• Parallel Virtual Machine (PVM)
• PVM is a portable message passing system, designed to connect
separate heterogeneous host machines to form a single virtual
machine. It is a single manageable parallel computing resource.
• Large computational problems like superconductivity studies,
molecular dynamics simulations, and matrix algorithms can be
solved more cost effectively by using the memory and the
aggregate power of many computers.
• It manages all message routing, data conversion, task
scheduling in the network of incompatible computer
architectures.

Unit-3 / Dr Komarasamy G 26
 Parallel Random Access Machine (PRAM) model
Features of PVM
• Very easy to install and configure;
• Multiple users can use PVM at the same time;
• One user can execute multiple applications;
• It’s a small package;
• Supports C, C++, Fortran;
• For a given run of a PVM program, users can select the group of
machines;
• It is a message-passing model,
• Process-based computation;
• Supports heterogeneous architecture.

Unit-3 / Dr Komarasamy G 27
 Parallel Random Access Machine (PRAM) model
• Data Parallel Programming
• The major focus of data parallel programming model is on
performing operations on a data set simultaneously. The data set
is organized into some structure like an array, hypercube, etc.
Processors perform operations collectively on the same data
structure. Each task is performed on a different partition of the
same data structure.
• It is restrictive, as not all the algorithms can be specified in terms
of data parallelism. This is the reason why data parallelism is not
universal.
• Data parallel languages help to specify the data decomposition
and mapping to the processors. It also includes data distribution
statements that allow the programmer to have control on data –
for example, which data will go on which processor – to reduce
the amount of communication within the processors.
Unit-3 / Dr Komarasamy G 28
 MPI – Lab programs
• 5. MPI – Basics of MPI
• 6. MPI – Communication between MPI process
• 7. MPI – Collective operation with "synchronization"
• 8. MPI – Collective operation with "data movement"
• 9. MPI – Collective operation with "collective computation"
• 10. MPI – Non-blocking operation

How to run MPI program:

• https://youtu.be/T_BVqSya1Is?si=SostwsSfiDQBth-X

Unit-3 / Dr Komarasamy G 29