Introduction to High Performance Scientific Computing

Autumn, 2016

Lecture 15

Imperial College Prasun Ray

London 28 November 2016
 Parallel computing paradigms

Distributed memory
•  Each (4-core) chip has its own memory

•  The chips are connected by network ‘cables’

•  MPI coordinates communication between two or more CPUs

Imperial College
London
 Parallel computing paradigms

Related approaches:
•  Hybrid programming: mix of shared-memory (OpenMP) and
distributed-memory (MPI) programming

•  GPU’s: Shared memory programming (CUDA or OpenCL)

•  Coprocessors and co-array programming

Imperial College
London
 MPI intro
•  MPI: Message Passing Interface

•  Standard for exchanging data between processors

•  Supports Fortran, c, C++

•  Can also be used with Python

Imperial College
London
 OpenMP schematic
Program starts with
single master thread

Then, launch parallel Start program

region with multiple
threads.
master thread
Each thread has
access to all FORK
variables introduced
previously Parallel region (4 threads)

Can end parallel JOIN

region if/when
desired and launch
Serial region (1 thread)
parallel regions again
in future as needed

Imperial College
London
 MPI schematic
Program starts with
all processes
running
Start program
MPI controls
communication
between processes
Parallel region (4 processes)

Imperial College
London
 MPI intro
•  Basic idea: calls to MPI subroutines control data exchange
between processors

•  Example:

call MPI_BCAST(n, 1, MPI_INTEGER,0,MPI_COMM_WORLD,ierr)

This will send the integer n which has size 1 from processor 0 to all
of the other processors.

Imperial College
London
 MPI broadcast

P0 data P0 data

P1 P1 data

P2 P2 data

P3 P3 data

Imperial College
London
 MPI intro
•  Basic idea: calls to MPI subroutines control data exchange
between processors

•  Example:

call MPI_BCAST(n, 1, MPI_INTEGER,0,MPI_COMM_WORLD,ierr)

This will send the integer n which has size 1 from processor 0 to all
of the other processors.

Generally, need to specify:

•  source and/or destination of message
•  size of data contained in message
•  type of data contained in message (integer, double precision, …)
•  the data itself (or its location)

Imperial College
London
 Fortran code structure
! Basic Fortran 90 code structure!
!
!1. Header!
program template!
!
!2. Variable declarations (e.g. integers, real numbers,...)!
!
!3. basic code: input, loops, if-statements, subroutine calls!
print *, 'template code'!
!
!
!4. End program!
end program template!
!
! To compile this code:!
! $ gfortran -o f90template.exe f90template.f90!
! To run the resulting executable: $ ./f90template.exe

Imperial College
London
 MPI intro
! Basic MPI + Fortran 90 code structure! See mpif90template.f90
!
!1. Header!
program template!
use mpi!
!
!2a. Variable declarations (e.g. integers, real numbers,...)!
integer :: myid, numprocs, ierr!
!
!2b. Initialize MPI!
call MPI_INIT(ierr)!
call MPI_COMM_RANK(MPI_COMM_WORLD, myid, ierr)!
call MPI_COMM_SIZE(MPI_COMM_WORLD, numprocs, ierr)!
!
!3. basic code: input, loops, if-statements, subroutine calls!
print *, 'this is proc # ',myid, 'of ', numprocs!
!
!
!4. End program!
call MPI_FINALIZE(ierr)!
end program template!
!
! To compile this code:!
! $ mpif90 -o mpitemplate.exe mpif90template.f90!
! To run the resulting executable with 4 processes:$ mpiexec -n 4 mpitemplate.exe
Imperial College
London
 MPI intro
! Basic MPI + Fortran 90 code structure! See mpif90template.f90
!
!1. Header!
program template!
use mpi!
!
!2a. Variable declarations (e.g. integers, real numbers,...)!
integer :: myid, numprocs, ierr!
!
!2b. Initialize MPI!
call MPI_INIT(ierr)!
call MPI_COMM_RANK(MPI_COMM_WORLD, myid, ierr)!
call MPI_COMM_SIZE(MPI_COMM_WORLD, numprocs, ierr)!
!
!3. basic code: input, loops, if-statements, subroutine calls!
print *, 'this is proc # ',myid, 'of ', numprocs!
!
!
!4. End program!
call MPI_FINALIZE(ierr)!
end program template!
!
! To compile this code:!
! $ mpif90 -o mpitemplate.exe mpif90template.f90!
! To run the resulting executable with 4 processes:$ mpiexec -n 4 mpitemplate.exe
Imperial College
London
 MPI intro
! Basic MPI + Fortran 90 code structure! See mpif90template.f90
!
!1. Header!
program template!
use mpi!
!
!2a. Variable declarations (e.g. integers, real numbers,...)!
integer :: myid, numprocs, ierr!
!
!2b. Initialize MPI!
call MPI_INIT(ierr)!
call MPI_COMM_RANK(MPI_COMM_WORLD, myid, ierr)!
call MPI_COMM_SIZE(MPI_COMM_WORLD, numprocs, ierr)!
!
!3. basic code: input, loops, if-statements, subroutine calls!
print *, 'this is proc # ',myid, 'of ', numprocs!
!
!
!4. End program!
call MPI_FINALIZE(ierr)!
end program template!
!
! To compile this code:!
! $ mpif90 -o mpitemplate.exe mpif90template.f90!
! To run the resulting executable with 4 processes:$ mpiexec -n 4 mpitemplate.exe
Imperial College
London
 MPI intro
•  Compile + run:

$ mpif90 -o mpif90template.exe mpif90template.f90!

!
$ mpiexec -n 4 mpif90template.exe!
this is proc # 0 of 4!
this is proc # 3 of 4!
this is proc # 1 of 4!
this is proc # 2 of 4!

Note: The number of processes specified with mpiexec can be

larger than the number of cores on your machine, but then tasks are
run sequentially.

Imperial College
London
 MPI+Fortran example: computing an integral

•  Estimate integral
with midpoint rule,

Imperial College
London
 MPI+Fortran quadrature
Two most important tasks:

1.  Decide how many intervals per processor

2.  Each processor will compute its own partial sum, sum_proc,
how do we compute sum(sum_proc)?

Imperial College
London
 MPI+Fortran quadrature
Two most important tasks:

1.  Decide how many intervals per processor

2.  Each processor will compute its own partial sum, sum_proc,
how do we compute sum(sum_proc)?

•  N = number of intervals

•  numprocs = number of processors

•  Need to compute Nper_proc: intervals per processor

Imperial College
London
 MPI+Fortran quadrature
•  N = number of intervals

•  numprocs = number of processors

•  Need to compute Nper_proc: intervals per processor

§  Basic idea: if N = 8 * numprocs, Nper_proc = 8

§  But, if N <= numprocs, N/numprocs = 0

Nper_proc = (N + numprocs – 1)/numprocs

Imperial College
London
 MPI+Fortran quadrature
Two most important tasks:

1.  Decide how many intervals per processor

2.  Each processor will compute its own partial sum, sum_proc,
how do we compute sum(sum_proc)?

Use MPI_REDUCE

Imperial College
London
 MPI reduce

P0 data1 Reduction P0 result

P1 data2 P1

P2 data3 P2

P3 data4 P3

Imperial College
London
 MPI+Fortran quadrature
Two most important tasks:

1.  Decide how many intervals per processor

2.  Each processor will compute its own partial sum, sum_proc,
how do we compute sum(sum_proc)?

•  Use MPI_REDUCE

•  Reduction options: MPI_MAX, MPI_MIN, MPI_SUM, MPI_PROD

Imperial College
London
 MPI+Fortran quadrature
Two most important tasks:

1.  Decide how many intervals per processor

2.  Each processor will compute its own partial sum, sum_proc,
how do we compute sum(sum_proc)?

•  Use MPI_REDUCE

•  Reduction options: MPI_MAX, MPI_MIN, MPI_SUM, MPI_PROD

•  For quadrature, we need MPI_SUM

Imperial College
London
 MPI+Fortran quadrature

For quadrature, we need MPI_SUM:

call MPI_REDUCE(data, result, 1, MPI_DOUBLE_PRECISION,

0,MPI_COMM_WORLD,ierr)

This will:

1.  Collect the double precision variable data which has size 1 from
each processor.

2.  Compute the sum (because we have chosen and store

the value in result on processor 0.

Note: Only processor 0 will have the final sum. With

MPI_ALLREDUCE, the result will be on every processor.

Imperial College
London
 MPI+Fortran quadrature
midpoint_p.f90: distribute data

!set number of intervals per processor!

Nper_proc = (N + numprocs - 1)/numprocs!
!
!starting and ending points for processor!
istart = myid * Nper_proc + 1!
iend = (myid+1) * Nper_proc!
if (iend>N) iend = N!
!

Imperial College
London
 MPI+Fortran quadrature
midpoint_p.f90: 1. distribute data, 2. compute sum_proc

!set number of intervals per processor!

Nper_proc = (N + numprocs - 1)/numprocs!
!
!starting and ending points for processor!
istart = myid * Nper_proc + 1!
iend = (myid+1) * Nper_proc!
if (iend>N) iend = N!
!
!loop over intervals computing each interval's contribution to
integral!
do i1 = istart,iend!
xm = dx*(i1-0.5) !midpoint of interval i1!
call integrand(xm,f)!
sum_i = dx*f!
sum_proc = sum_proc + sum_i !add contribution from interval
to total integral!
end do!

Imperial College
London
 MPI+Fortran quadrature
midpoint_p.f90: 1. distribute data, 2. compute sum_proc, 3. reduction

!set number of intervals per processor!

Nper_proc = (N + numprocs - 1)/numprocs!
!
!starting and ending points for processor!
istart = myid * Nper_proc + 1!
iend = (myid+1) * Nper_proc!
if (iend>N) iend = N!
!
!loop over intervals computing each interval's contribution to integral!
do i1 = istart,iend!
xm = dx*(i1-0.5) !midpoint of interval i1!
call integrand(xm,f)!
sum_i = dx*f!
sum_proc = sum_proc + sum_i !add contribution from interval to
total integral!
end do!
!collect double precision variable, sum, with size 1 on process 0 using
the MPI_SUM option!
call MPI_REDUCE(sum_proc,sum,1,MPI_DOUBLE_PRECISION,MPI_SUM,
0,MPI_COMM_WORLD,ierr)
Imperial College
London
 MPI+Fortran quadrature
Compile and run:

$ mpif90 -o midpoint_p.exe midpoint_p.f90!

!
$ mpiexec -n 2 midpoint_p.exe !
number of intervals = 1000!
number of procs = 2!
Nper_proc= 500!
The partial sum on proc # 0 is: 1.8545905426699112 !
The partial sum on proc # 1 is: 1.2870021942532193 !
N= 1000!
sum= 3.1415927369231307 !
error= 8.3333337563828991E-008!
!

Imperial College
London
 Other collective operations
•  Scatter and gather

Imperial College
London
 MPI scatter

P0 [f1,f2,f3,f4] P0 f1

P1 P1 f2

P2 P2 f3

P3 P3 f4

Imperial College
London
 MPI gather

P0 [f1,f2,f3,f4] P0 f1

P1 P1 f2

P2 P2 f3

P3 P3 f4

Imperial College
London
 k 1
Ti+1 2Tik + Tik 11
Other Other collective
collective
2
= operations
Si
operations
x
2
x 1 k 1
•  •  Scatter
Scatter and k and gather
Ti gather
= Si + Ti+1 + Tik 11
2 2
dT •  Gather
•  iGather allTparticles
all particles 2Ton
i+ Ti 1
processor
i+1 on processor
=•  S
•  Compute i (t)interaction
Compute + interaction
forcesforces
2 for,particles
i =on1,that
for particles 2,
on..., Nprocessor
that
processor
dt x
2 XN
d xi
2
= f (|xi xj |), i = 1, 2, ..., N
dt j=1

•  Avoid
•  Avoid forproblems
for big big problems (why?)
(why?)

Imperial College
Imperial College
London London
 MPI collective data movement

Imperial College
London
From Using MPI