slides1-1

Chapter 1

Parallel Computers

Slides for Parallel Programming Techniques & Applications Using Networked Workstations & Parallel Computers 2nd ed., by B. Wilkinson & M. Allen, 2004 Pearson Education Inc. All rights reserved.

slides1-2

Demand for Computational Speed

Continual demand for greater computational speed from a computer
system than is currently possible

Areas requiring great computational speed include numerical

modeling and simulation of scientific and engineering problems.

Computations must be completed within a reasonable time period.

Slides for Parallel Programming Techniques & Applications Using Networked Workstations & Parallel Computers 2nd ed., by B. Wilkinson & M. Allen, 2004 Pearson Education Inc. All rights reserved.

slides1-3

Grand Challenge Problems

A grand challenge problem is one that cannot be solved in a
reasonable amount of time with todays computers.

Obviously, an execution time of 10 years is always unreasonable.

Examples

Modeling large DNA structures

Global weather forecasting

Modeling motion of astronomical bodies.

Slides for Parallel Programming Techniques & Applications Using Networked Workstations & Parallel Computers 2nd ed., by B. Wilkinson & M. Allen, 2004 Pearson Education Inc. All rights reserved.

slides1-4

Weather Forecasting
Atmosphere modeled by dividing it into 3-dimensional cells.
Calculations of each cell repeated many times to model passage of
time.

Slides for Parallel Programming Techniques & Applications Using Networked Workstations & Parallel Computers 2nd ed., by B. Wilkinson & M. Allen, 2004 Pearson Education Inc. All rights reserved.

slides1-5

Global Weather Forecasting Example

Whole global atmosphere divided into cells of size 1 mile 1 mile
1 mile to a height of 10 miles (10 cells high) - about 5 108 cells.
Suppose each calculation requires 200 floating point operations. In
one time step, 1011 floating point operations necessary.
To forecast the weather over 7 days using 1-minute intervals, a
computer operating at 1Gflops (109 floating point operations/s)
would take 106 seconds or over 10 days.
To perform the calculation in 5 minutes would require a computer
operating at 3.4 Tflops (3.4 1012 floating point operations/sec).
Slides for Parallel Programming Techniques & Applications Using Networked Workstations & Parallel Computers 2nd ed., by B. Wilkinson & M. Allen, 2004 Pearson Education Inc. All rights reserved.

slides1-6

Modeling Motion of Astronomical Bodies

Each body attracted to each other body by gravitational forces.
Movement of each body predicted by calculating total force on each
body. With N bodies, N 1 forces to calculate for each body, or
approx. N2 calculations. (N log2 N for an efficient approx. algorithm.)
After determining new positions of bodies, calculations repeated.

A galaxy might have, say, 1011 stars. Even if each calculation could
be done in 1 s (an extremely optimistic figure), it would take 109
years for one iteration using the N2 algorithm and almost a year for
one iteration using an efficient N log2 N approximate algorithm.
Slides for Parallel Programming Techniques & Applications Using Networked Workstations & Parallel Computers 2nd ed., by B. Wilkinson & M. Allen, 2004 Pearson Education Inc. All rights reserved.

slides1-7

Astrophysical N-body simulation by Scott Linssen (undergraduate

University of North Carolina at Charlotte [UNCC] student).
Slides for Parallel Programming Techniques & Applications Using Networked Workstations & Parallel Computers 2nd ed., by B. Wilkinson & M. Allen, 2004 Pearson Education Inc. All rights reserved.

slides1-8

Parallel Computing
Using more than one computer, or a computer with more than one
processor, to solve a problem.

Motives
Usually faster computation - very simple idea - that n computers
operating simultaneously can achieve the result n times faster - it
will not be n times faster for various reasons.

Other motives include: fault tolerance, larger amount of memory

available, ...
Slides for Parallel Programming Techniques & Applications Using Networked Workstations & Parallel Computers 2nd ed., by B. Wilkinson & M. Allen, 2004 Pearson Education Inc. All rights reserved.

slides1-9

Background
Parallel computers - computers with more than one processor - and
their programming - parallel programming - has been around for
more than

40 years.

Slides for Parallel Programming Techniques & Applications Using Networked Workstations & Parallel Computers 2nd ed., by B. Wilkinson & M. Allen, 2004 Pearson Education Inc. All rights reserved.

slides1-10

Gill writes in 1958:

... There is therefore nothing new in the idea of parallel
programming, but its application to computers. The author cannot
believe that there will be any insuperable difficulty in extending it to
computers. It is not to be expected that the necessary programming
techniques will be worked out overnight. Much experimenting
remains to be done. After all, the techniques that are commonly
used in programming today were only won at the cost of
considerable toil several years ago. In fact the advent of parallel
programming may do something to revive the pioneering spirit in
programming which seems at the present to be degenerating into a
rather dull and routine occupation ...
Gill, S. (1958), Parallel Programming, The Computer Journal, vol. 1, April, pp. 2-10.
Slides for Parallel Programming Techniques & Applications Using Networked Workstations & Parallel Computers 2nd ed., by B. Wilkinson & M. Allen, 2004 Pearson Education Inc. All rights reserved.

slides1-11

Notation
p = number of processors or processes
n = number of dtata items (used later)

Slides for Parallel Programming Techniques & Applications Using Networked Workstations & Parallel Computers 2nd ed., by B. Wilkinson & M. Allen, 2004 Pearson Education Inc. All rights reserved.

slides1-12

Speedup Factor
S(p) =

ts
Execution time using one processor (best sequential algorithm)
=
Execution time using a multiprocessor with p processors
tp

where ts is execution time on a single processor and tp is execution

time on a multiprocessor.
S(p) gives increase in speed by using multiprocessor.

Notice use best sequential algorithm with single processor system.

Underlying algorithm for parallel implementation might be (and is

usually) different.
Slides for Parallel Programming Techniques & Applications Using Networked Workstations & Parallel Computers 2nd ed., by B. Wilkinson & M. Allen, 2004 Pearson Education Inc. All rights reserved.

slides1-13

Speedup factor can also be cast in terms of computational steps:

S(p) =

Number of computational steps using one processor

Number of parallel computational steps with p processors

Can also extend time complexity to parallel computations - see later.

Slides for Parallel Programming Techniques & Applications Using Networked Workstations & Parallel Computers 2nd ed., by B. Wilkinson & M. Allen, 2004 Pearson Education Inc. All rights reserved.

slides1-14

Maximum Speedup
Maximum speedup is usually p with p processors (linear speedup).
Possible to get superlinear speedup (greater than p) but usually a
specific reason such as:

Extra memory in multiprocessor system

Nondeterministic algorithm

Slides for Parallel Programming Techniques & Applications Using Networked Workstations & Parallel Computers 2nd ed., by B. Wilkinson & M. Allen, 2004 Pearson Education Inc. All rights reserved.

slides1-15

Maximum Speedup - Amdahls law

ts
fts

(1 - f)ts

Serial section

Parallelizable sections

(a) One processor

(b) Multiple
processors

p processors

tp

(1 - f)ts /p

Slides for Parallel Programming Techniques & Applications Using Networked Workstations & Parallel Computers 2nd ed., by B. Wilkinson & M. Allen, 2004 Pearson Education Inc. All rights reserved.

slides1-16

Speedup factor is given by:

S(p) =

ts
p
=
fts + (1 f )ts /p
1 + (p 1)f

This equation is known as Amdahls law

Slides for Parallel Programming Techniques & Applications Using Networked Workstations & Parallel Computers 2nd ed., by B. Wilkinson & M. Allen, 2004 Pearson Education Inc. All rights reserved.

slides1-17

Speedup against number of processors

20

f = 0%

16
12
f = 5%
8
4

f = 10%
f = 20%

4
8
12 16 20
Number of processors, p

Even with infinite number of processors, maximum speedup limited

to 1/f.
Example: With only 5% of computation being serial, maximum
speedup is 20, irrespective of number of processors.
Slides for Parallel Programming Techniques & Applications Using Networked Workstations & Parallel Computers 2nd ed., by B. Wilkinson & M. Allen, 2004 Pearson Education Inc. All rights reserved.

slides1-18

Superlinear Speedup example - Searching

(a) Searching each sub-space sequentially
Start

Time
ts

ts/p
Sub-space
search

t
xts/p

Solution found

x indeterminate

Slides for Parallel Programming Techniques & Applications Using Networked Workstations & Parallel Computers 2nd ed., by B. Wilkinson & M. Allen, 2004 Pearson Education Inc. All rights reserved.

slides1-19

(b) Searching each sub-space in parallel

Solution found
Slides for Parallel Programming Techniques & Applications Using Networked Workstations & Parallel Computers 2nd ed., by B. Wilkinson & M. Allen, 2004 Pearson Education Inc. All rights reserved.

slides1-20

Speed-up is then given by

x t s + t
----

p
S ( p ) = -----------------------------t

Slides for Parallel Programming Techniques & Applications Using Networked Workstations & Parallel Computers 2nd ed., by B. Wilkinson & M. Allen, 2004 Pearson Education Inc. All rights reserved.

slides1-21

Worst case for sequential search when solution found in last subspace search. Then parallel version offers greatest benefit, i.e.

p 1 t + t
- s
----------p
S ( p ) = ---------------------------------------- as t tends to zero
t
Least advantage for parallel version when solution found in first subspace search of the sequential search, i.e.

t
S( p) =
----- = 1
t
Actual speed-up depends upon which subspace holds solution but
could be extremely large.

Slides for Parallel Programming Techniques & Applications Using Networked Workstations & Parallel Computers 2nd ed., by B. Wilkinson & M. Allen, 2004 Pearson Education Inc. All rights reserved.

slides1-22

Types of Parallel Computers

Two principal types:

Shared memory multiprocessor

Distributed memory multicomputer

Slides for Parallel Programming Techniques & Applications Using Networked Workstations & Parallel Computers 2nd ed., by B. Wilkinson & M. Allen, 2004 Pearson Education Inc. All rights reserved.

slides1-23

Shared Memory Multiprocessor

Slides for Parallel Programming Techniques & Applications Using Networked Workstations & Parallel Computers 2nd ed., by B. Wilkinson & M. Allen, 2004 Pearson Education Inc. All rights reserved.

slides1-24

Conventional Computer
Consists of a processor executing a program stored in a (main)
memory:

Main memory
Instructions (to processor)
Data (to or from processor)
Processor

Each main memory location located by its address. Addresses start

at 0 and extend to 2b 1 when there are b bits (binary digits) in
address.
Slides for Parallel Programming Techniques & Applications Using Networked Workstations & Parallel Computers 2nd ed., by B. Wilkinson & M. Allen, 2004 Pearson Education Inc. All rights reserved.

slides1-25

Natural way to extend single processor model - have multiple

processors connected to multiple memory modules, such that each
processor can access any memory module - so-called shared
memory configuration:

One
address
space

Memory modules

Interconnection
network

Processors
Slides for Parallel Programming Techniques & Applications Using Networked Workstations & Parallel Computers 2nd ed., by B. Wilkinson & M. Allen, 2004 Pearson Education Inc. All rights reserved.

slides1-26

Simplistic view of a small shared memory

multiprocessor
Processors

Shared memory

Bus

Examples:

Dual Pentiums

Quad Pentiums

Slides for Parallel Programming Techniques & Applications Using Networked Workstations & Parallel Computers 2nd ed., by B. Wilkinson & M. Allen, 2004 Pearson Education Inc. All rights reserved.

slides1-27

Quad Pentium Shared Memory Multiprocessor

Processor

Processor

Processor

Processor

L1 cache

L1 cache

L1 cache

L1 cache

L2 Cache

L2 Cache

L2 Cache

L2 Cache

Bus interface

Bus interface

Bus interface

Bus interface

Processor/
memory
bus
I/O interface

Memory Controller

I/O bus

Shared memory

Memory

Slides for Parallel Programming Techniques & Applications Using Networked Workstations & Parallel Computers 2nd ed., by B. Wilkinson & M. Allen, 2004 Pearson Education Inc. All rights reserved.

slides1-28

Programming Shared Memory Multiprocessors

Threads - programmer decomposes the program into individual
parallel sequences, (threads), each being able to access
variables declared outside threads.
Example Pthreads
A sequential programming language with preprocessor compiler
directives to declare shared variables and specify parallelism.
Example OpenMP - industry standard - needs OpenMP compiler
A sequential programming language with added syntax to
declare shared variables and specify parallelism.
Example UPC (Unified Parallel C) - needs a UPC compiler.
A parallel programming language with syntax to express
parallelism, in which the compiler creates the appropriate
executable code for each processor (not now common)
A sequential programming language and ask a parallelizing
compiler to convert it into parallel executable code. - also not
now common
Slides for Parallel Programming Techniques & Applications Using Networked Workstations & Parallel Computers 2nd ed., by B. Wilkinson & M. Allen, 2004 Pearson Education Inc. All rights reserved.

slides1-29

Message-Passing Multicomputer
Complete computers connected through an interconnection
network:

Interconnection
network
Messages
Processor

Local
memory

Computers
Slides for Parallel Programming Techniques & Applications Using Networked Workstations & Parallel Computers 2nd ed., by B. Wilkinson & M. Allen, 2004 Pearson Education Inc. All rights reserved.

slides1-30

Interconnection Networks
With direct links between computers

Exhausive connections

2-dimensional and 3-dimensional meshs

Hypercube

Using Switches:

Crossbar

Trees

Multistage interconnection networks

Slides for Parallel Programming Techniques & Applications Using Networked Workstations & Parallel Computers 2nd ed., by B. Wilkinson & M. Allen, 2004 Pearson Education Inc. All rights reserved.

slides1-31

Two-dimensional array (mesh)

Links

Computer/
processor

Also three-dimensional - used in some large high performance

systems.
Slides for Parallel Programming Techniques & Applications Using Networked Workstations & Parallel Computers 2nd ed., by B. Wilkinson & M. Allen, 2004 Pearson Education Inc. All rights reserved.

slides1-32

Three-dimensional hypercube

110
100

111

101

010
000

011
001

Slides for Parallel Programming Techniques & Applications Using Networked Workstations & Parallel Computers 2nd ed., by B. Wilkinson & M. Allen, 2004 Pearson Education Inc. All rights reserved.

slides1-33

Four-dimensional hypercube

0110
0100

0111
0101

0010
0000

1100

0011
0001

1110

1111
1101

1010

1000

1011
1001

Hypercubes popular in 1980s - not now

Slides for Parallel Programming Techniques & Applications Using Networked Workstations & Parallel Computers 2nd ed., by B. Wilkinson & M. Allen, 2004 Pearson Education Inc. All rights reserved.

slides1-34

Crossbar switch

Memories

Processors

Switches

Slides for Parallel Programming Techniques & Applications Using Networked Workstations & Parallel Computers 2nd ed., by B. Wilkinson & M. Allen, 2004 Pearson Education Inc. All rights reserved.

slides1-35

Tree

Root
Links

Switch
element

Processors

Slides for Parallel Programming Techniques & Applications Using Networked Workstations & Parallel Computers 2nd ed., by B. Wilkinson & M. Allen, 2004 Pearson Education Inc. All rights reserved.

slides1-36

Multistage Interconnection Network

Example: Omega network
2 2 switch elements
(straight-through or
crossover connections)
000
001
010
011
Inputs

000
001
010
011
Outputs

100
101

100
101

110
111

110
111

Slides for Parallel Programming Techniques & Applications Using Networked Workstations & Parallel Computers 2nd ed., by B. Wilkinson & M. Allen, 2004 Pearson Education Inc. All rights reserved.

slides1-37

Distributed Shared Memory

Making the main memory of a group of interconnected computers
look as though it is a single memory with a single address space.
Then can use shared memory programming techniques.

Interconnection
network
Messages
Processor

Shared
memory
Computers
Slides for Parallel Programming Techniques & Applications Using Networked Workstations & Parallel Computers 2nd ed., by B. Wilkinson & M. Allen, 2004 Pearson Education Inc. All rights reserved.

slides1-38

Flynns Classifications
Flynn (1966) created a classification for computers based upon
instruction streams and data streams:

Single instruction stream-single data stream (SISD) computer

In a single processor computer, a single stream of instructions is
generated from the program. The instructions operate upon a single
stream of data items. Flynn called this single processor computer a
single instruction stream-single data stream (SISD) computer.

Slides for Parallel Programming Techniques & Applications Using Networked Workstations & Parallel Computers 2nd ed., by B. Wilkinson & M. Allen, 2004 Pearson Education Inc. All rights reserved.

slides1-39

Multiple Instruction Stream-Multiple Data Stream (MIMD)

Computer
General-purpose multiprocessor system - each processor has a
separate program and one instruction stream is generated from
each program for each processor. Each instruction operates upon
different data.

Both the shared memory and the message-passing multiprocessors

so far described are in the MIMD classification.

Slides for Parallel Programming Techniques & Applications Using Networked Workstations & Parallel Computers 2nd ed., by B. Wilkinson & M. Allen, 2004 Pearson Education Inc. All rights reserved.

slides1-40

Single Instruction Stream-Multiple Data Stream (SIMD)

Computer
A specially designed computer in which a single instruction stream
is from a single program, but multiple data streams exist. The
instructions from the program are broadcast to more than one
processor. Each processor executes the same instruction in
synchronism, but using different data.

Developed because there are a number of important applications

that mostly operate upon arrays of data.

Slides for Parallel Programming Techniques & Applications Using Networked Workstations & Parallel Computers 2nd ed., by B. Wilkinson & M. Allen, 2004 Pearson Education Inc. All rights reserved.

slides1-41

Multiple Program Multiple Data (MPMD)

Structure
Within the MIMD classification, which we are concerned with, each
processor will have its own program to execute:

Program
Instructions

Program
Instructions

Processor

Processor

Data

Data

Slides for Parallel Programming Techniques & Applications Using Networked Workstations & Parallel Computers 2nd ed., by B. Wilkinson & M. Allen, 2004 Pearson Education Inc. All rights reserved.

slides1-42

Single Program Multiple Data (SPMD) Structure

Single source program is written and each processor will execute its
personal copy of this program, although independently and not in
synchronism.

The source program can be constructed so that parts of the

program are executed by certain computers and not others
depending upon the identity of the computer.

Slides for Parallel Programming Techniques & Applications Using Networked Workstations & Parallel Computers 2nd ed., by B. Wilkinson & M. Allen, 2004 Pearson Education Inc. All rights reserved.

slides1-43

Networked Computers as a Multicomputer

Platform
A network of computers became a very attractive alternative to
expensive supercomputers and parallel computer systems for highperformance computing in early 1990s.

Several early projects.

Notable:

Berkeley NOW (network of workstations) project.

NASA Beowulf project. (Will look at this one later)

Term now used - cluster computing.

Slides for Parallel Programming Techniques & Applications Using Networked Workstations & Parallel Computers 2nd ed., by B. Wilkinson & M. Allen, 2004 Pearson Education Inc. All rights reserved.

slides1-44

Key advantages:

Very high performance workstations and PCs readily

available at low cost.

The latest processors can easily be incorporated into

the system as they become available.

Existing software can be used or modified.

Slides for Parallel Programming Techniques & Applications Using Networked Workstations & Parallel Computers 2nd ed., by B. Wilkinson & M. Allen, 2004 Pearson Education Inc. All rights reserved.

slides1-45

Message Passing Parallel Programming

Software Tools for Clusters
Parallel Virtual Machine (PVM) - developed in late 1980s. Became
very popular.

Message-Passing Interface (MPI) - standard defined in 1990s.

Both provide a set of user-level libraries for message passing. Use

with regular programming languages (C, C++, ...).

Slides for Parallel Programming Techniques & Applications Using Networked Workstations & Parallel Computers 2nd ed., by B. Wilkinson & M. Allen, 2004 Pearson Education Inc. All rights reserved.

slides1-46

Beowulf Clusters*
A group of interconnected commodity computers achieving high
performance with low cost.

Typically using commodity interconnects - high speed Ethernet, and

Linux OS.

* Beowulf comes from name given by NASA Goddard Space Flight

Center cluster project.

Slides for Parallel Programming Techniques & Applications Using Networked Workstations & Parallel Computers 2nd ed., by B. Wilkinson & M. Allen, 2004 Pearson Education Inc. All rights reserved.

slides1-47

Cluster Interconnects

Originally fast Ethernet on low cost clusters

Gigabit Ethernet - easy upgrade path
More Specialized/Higher Performance

Myrinet - 2.4 Gbits/sec - disadvantage: single vendor

cLan
SCI (Scalable Coherent Interface)
QNet
Infiniband - may be important as infininbnand
interfaces may be intergrated on next generation PCs

Slides for Parallel Programming Techniques & Applications Using Networked Workstations & Parallel Computers 2nd ed., by B. Wilkinson & M. Allen, 2004 Pearson Education Inc. All rights reserved.

slides1-48

Dedicated cluster with a master node

Dedicated Cluster

User

Compute nodes
Master node
Up link
Switch

2nd Ethernet
interface

External network

Slides for Parallel Programming Techniques & Applications Using Networked Workstations & Parallel Computers 2nd ed., by B. Wilkinson & M. Allen, 2004 Pearson Education Inc. All rights reserved.