MATLAB Parallel Computing

John Burkardt (ARC/ICAM) & Gene Cliff (AOE/ICAM)

Virginia Tech
..........
Introduction to Parallel MATLAB at Virginia Tech
http://people.sc.fsu.edu/∼burkardt/presentations/parallel matlab 2010.
..........
ARC: Advanced Research Computing
AOE: Department of Aerospace and Ocean Engineering
ICAM: Interdisciplinary Center for Applied Mathematics

08 February 2010

Burkardt/Cliff MATLAB Parallel Computing

MATLAB Parallel Computing

Introduction
Local Parallel Computing
The MD Example
PRIME NUMBER Example
Remote Computing
KNAPSACK Example
SPMD Parallelism
fmincon Example
Codistributed Arrays
A 2D Heat Equation
Conclusion

Burkardt/Cliff MATLAB Parallel Computing

INTRO: MATLAB Adds Parallelism

The MathWorks has recognized that parallel computing is

necessary for scientific computation.
The underlying MATLAB core and algorithms are being extended
to work with parallelism.
An explicit set of commands has been added to allow the user to
request parallel execution or to control distributed memory.
New protocols and servers allow multiple copies of MATLAB to
carry out the user’s requests, to transfer data and to communicate.
MATLAB’s parallelism can be enjoyed by novices and exploited by
experts.

Burkardt/Cliff MATLAB Parallel Computing

INTRO: Local Parallelism

MATLAB has developed a Parallel Computing Toolbox which is

required for all parallel applications.
The Toolbox allows a user to run a job in parallel on a desktop
machine, using up to 8 ”workers” (additional copies of MATLAB)
to assist the main copy.
If the desktop machine has multiple processors, the workers will
activate them, and the computation should run more quickly.
This use of MATLAB is very similar to the shared memory parallel
computing enabled by OpenMP; however, MATLAB requires much
less guidance from the user.

Burkardt/Cliff MATLAB Parallel Computing

INTRO: Local and Remote MATLAB Workers

Burkardt/Cliff MATLAB Parallel Computing

INTRO: Remote Parallelism

MATLAB has developed a Distributed Computing Server or DCS.

Assuming the user’s code runs properly under the local parallel
model, then it will also run under DCS with no further changes.
With the DCS, the user can start a job on the desktop that gets
assistance from workers on a remote cluster.

Burkardt/Cliff MATLAB Parallel Computing

INTRO: Local and Remote MATLAB Workers

Burkardt/Cliff MATLAB Parallel Computing

INTRO: SPMD for Distributed Data

If a cluster is available, the shared memory model makes less sense

than a distributed memory model.
In such a computation, very large arrays can be defined and
manipulated. Each computer does not have a copy of the same
array, but instead a distinct portion of the array. In this way, the
user has access to a memory space equal to the sum of the
memories of all the participating computers.
MATLAB provides the spmd command (“Single Program, Multiple
Data”) to allow a user to declare such distributed arrays, and
provides a range of operators that are appropriate for carrying out
computations on such arrays.

Burkardt/Cliff MATLAB Parallel Computing

INTRO: BATCH for Remote Jobs

MATLAB also includes a batch command that allows you to write

a script to run a job (parallel or not, remote or local) as a separate
process.
This means you can use your laptop or desktop copy of MATLAB
to set up and submit a script for running a remote job. You can
exit the local copy of MATLAB, turn off your laptop or do other
work, and later check on the remote job status and retrieve your
results.
Many computer clusters that have parallel MATLAB installed
require users to submit their jobs only in batch mode. Currently,
Virginia Tech permits interactive access to the cluster as well, but
may soon also go to batch-only access.

Burkardt/Cliff MATLAB Parallel Computing

INTRO: PMODE: Interactive Parallel Mode

A typical parallel MATLAB user working interactively still sees the

familiar MATLAB command window, which we may think of as
being associated with the “master” copy of MATLAB.
However, MATLAB also allows a user to open a parallel command
window. This is known as pmode.
Commands given in pmode are executed simultaneously on all the
workers. Within pmode, the user has access to distributed arrays,
parallel functions, and message-passing functions that are not
visible or accessible in the normal command window.

Burkardt/Cliff MATLAB Parallel Computing

INTRO: ITHACA

Virginia Tech has installed the ITHACA cluster of 84 nodes. Each

node is a separate computer with 2 quadcore processors.
This means each node can run 8 MATLAB workers.
At Virginia Tech, 8 nodes with 8 cores are dedicated to the parallel
MATLAB cluster, so theoretically you can run a job with 64
workers.
You should not routinely ask for all 64 workers. Currently, one
node is down, so there are only 56. Moreover, if one job ties up all
the workers, no one else can run. So we encourage the use of 24 or
32 workers at a time instead.

Burkardt/Cliff MATLAB Parallel Computing

MATLAB Parallel Computing

Introduction
Local Parallel Computing
The MD Example
PRIME NUMBER Example
Remote Computing
KNAPSACK Example
SPMD Parallelism
fmincon Example
Codistributed Arrays
A 2D Heat Equation
Conclusion

Burkardt/Cliff MATLAB Parallel Computing

LOCAL: Local Parallel Computing

If your desktop or laptop computer is fairly recent, it may have

more than one processor; the processors may have multiple cores.
Executing MATLAB in the regular way only engages one core.
(although some MATLAB linear algebra routines already use
multithreading to involve more cores.).
The Parallel Computing Toolbox runs up to 8 cooperating copies
of MATLAB, using the extra cores on your machine.
You’ll need:
the right version of MATLAB;
the Parallel Computing Toolbox;
a MATLAB M-file that uses new parallel keywords.

Burkardt/Cliff MATLAB Parallel Computing

LOCAL: What Do You Need?

1 Your machine must have multiple cores:

On a PC: Go to Start, choose Settings, then Control Panel,
then System.
On a Mac: From the Apple Menu, choose About this Mac,
then More Info....
2 Your MATLAB must be version 2008a or later:
To check MATLAB’s version, go to the HELP menu, and
choose About Matlab.
3 Your MATLAB needs the Parallel Computing Toolbox:
To list all your toolboxes, type the MATLAB command ver.

Burkardt/Cliff MATLAB Parallel Computing

LOCAL: Running A Program

Suppose you have a MATLAB M-file modified to compute in

parallel (we’ll explain that later!).
To do local parallel programming, start MATLAB the regular way.
This copy of MATLAB will be called the client copy; the extra
copies created later are known as workers or sometimes as labs.
Running in parallel requires three steps:
1 request a number of (local) workers;
2 issue the normal command to run the program. The client
MATLAB will call on the workers for help as needed;
3 release the workers.

Burkardt/Cliff MATLAB Parallel Computing

LOCAL: Running A Program Interactively

To run an M file called, say, md parallel.m in parallel on your

machine, type:

matlabpool open local 4

md_parallel
matlabpool close

The word local is choosing the local configuration, that is, the
cores assigned to be workers will be on the local machine.
The value ”4” is the number of workers you are asking for. It can
be up to 8 on a local machine. It does not have to match the
number of cores you have.

Burkardt/Cliff MATLAB Parallel Computing

LOCAL: Running A Program

If all is well, the program runs the same as before... but faster.
Output will still appear in the command window in the same way,
and the data will all be available to you.
What has happened is simply that some of the computations were
carried out by other cores in a way that was hidden from you.
The program may seem like it ran faster, but it’s important to
measure the time exactly.

Burkardt/Cliff MATLAB Parallel Computing

LOCAL: Timing A Program

To time a program, you can use tic and toc:

matlabpool open local 4

tic
md_parallel
toc

matlabpool close

tic starts the clock, toc stops the clock and prints the time.

Burkardt/Cliff MATLAB Parallel Computing

LOCAL: Timing A Program

To measure the speedup of a program, you can try different

numbers of workers:

for labs = 0 : 4
if ( 0 < labs ) matlabpool ( ’open’, ’local’, labs )
tic
md_parallel
toc
if ( 0 < labs ) matlabpool ( ’close’ )
end

Because labs is a variable, we must use the “function” form of

matlabpool() with parentheses and quoted strings.

Burkardt/Cliff MATLAB Parallel Computing

LOCAL: Parallel Starts at 2 Labs

To run a parallel job with 0 labs means to run it sequentially.

To run a parallel job with 1 lab means the client sends all the data
to the single lab and waits while the 1 lab does the job.
Only when we get to 2 labs do we have any hope of a speedup.
Since it takes some time to set up the parallel execution and
transfer data, we still won’t see a speedup if the job is too small.
Since the machines in this classroom only have 2 processors, this
means our demonstrations won’t get much speedup today!
(For local parallel computing, it is possible to run 1 client and 2
workers on 2 cores. For remote computing, we would need 3
cores, with one dedicated to the client.)

Burkardt/Cliff MATLAB Parallel Computing

MATLAB Parallel Computing

Introduction
Local Parallel Computing
The MD Example
PRIME NUMBER Example
Remote Computing
KNAPSACK Example
SPMD Parallelism
fmincon Example
Codistributed Arrays
A 2D Heat Equation
Conclusion

Burkardt/Cliff MATLAB Parallel Computing

MD: The Molecular Dynamics Example

The MD program runs a simple molecular dynamics simulation.

The problem size N counts the number of molecules being
simulated.
The program takes a long time to run, and it would be very useful
to speed it up.
There are many for loops in the program, but it is a mistake to try
to parallelize everything!
MATLAB has a profile command that can report where the CPU
time was spent - which is where we should try to parallelize.

Burkardt/Cliff MATLAB Parallel Computing

MD: Run MATLAB’s Profiler

>> profile on
>> md
>> profile viewer

Step Potential Kinetic (P+K-E0)/E0

Energy Energy Energy Error

1 498108.113974 0.000000 0.000000e+00

2 498108.113974 0.000009 1.794265e-11
... ... ... ...
9 498108.111972 0.002011 1.794078e-11
10 498108.111400 0.002583 1.793996e-11

CPU time = 415.740000 seconds.

Wall time = 378.828021 seconds.
Burkardt/Cliff MATLAB Parallel Computing
 This is a static copy of a profile report

MD: Profile
Home Results
Profile Summary
Generated 27-Apr-2009 15:37:30 using cpu time.

Function Name Calls Total Time Self Time* Total Time Plot
(dark band = self time)

md 1 415.847 s 0.096 s

compute 11 415.459 s 410.703 s

repmat 11000 4.755 s 4.755 s

timestamp 2 0.267 s 0.108 s

datestr 2 0.130 s 0.040 s

timefun/private/formatdate 2 0.084 s 0.084 s

update 10 0.019 s 0.019 s

datevec 2 0.017 s 0.017 s

now 2 0.013 s 0.001 s

datenum 4 0.012 s 0.012 s

datestr>getdateform 2 0.005 s 0.005 s

initialize 1 0.005 s 0.005 s

etime 2 0.002 s 0.002 s

Self time is the time spent in a function excluding the time spent in its child functions. Self time also includes overhead res
the process of profiling.
Burkardt/Cliff MATLAB Parallel Computing
MD: The COMPUTE Function

f u n c t i o n [ f , pot , k i n ] = compute ( np , nd , pos , v e l , mass )

f = z e r o s ( nd , np ) ;
pot = 0 . 0 ;
pi2 = pi / 2.0;

f o r i = 1 : np
R i = p o s − re p ma t ( p o s ( : , i ) , 1 , np ) ; % a r r a y of v e c t o r s to ’ i ’
D = s q r t ( sum ( R i . ˆ 2 ) ) ; % array of distances
Ri = Ri ( : , ( D > 0.0 ) ) ;
D = D( D > 0 . 0 ) ; % save o n l y pos v a l u e s
D2 = D . ∗ ( D <= p i 2 ) + p i 2 ∗ ( D > pi2 ) ; % truncate the p o t e n t i a l .
p o t = p o t + 0 . 5 ∗ sum ( s i n ( D2 ).ˆ2 ); % accumulate pot . energy
f ( : , i ) = R i ∗ ( s i n ( 2∗D2 ) . / D ); % f o r c e on p a r t i c l e ’ i ’
end

k i n = 0 . 5 ∗ mass ∗ sum ( d i a g ( v e l ’ ∗ v e l ) ) ; % k i n e t i c e n e r g y

return
end

Burkardt/Cliff MATLAB Parallel Computing

MD: Speedup

In this compute function, the important quantity is the force vector

f. For each particle i, the force is computed by determining the
distance to all other particles, squaring, truncating, and taking the
sine.
The important thing to notice is that the computation for each
particle can be done independently. That means we could compute
each value on a separate worker, at the same time.
The MATLAB command parfor can replace for in this situation. It
will distribute the iterations of the loop across the available
workers.

Burkardt/Cliff MATLAB Parallel Computing

MD: Speedup
By inserting a PARFOR in COMPUTE, here is our speedup:

Burkardt/Cliff MATLAB Parallel Computing

MD: Speedup

This simple example demonstrates a case in which parallel

execution of a MATLAB program gives a huge improvement in
performance.
There is some overhead in starting up the parallel process, and in
transferring data to and from the workers each time a parfor loop
is encountered. So we should not simply try to replace every for
loop with parfor.
That’s why, in this example, we first searched for the function that
was using most of the execution time.
The parfor command is the simplest way to make a parallel
program, but we will see some alternatives as well.

Burkardt/Cliff MATLAB Parallel Computing

MATLAB Parallel Computing

Introduction
Local Parallel Computing
The MD Example
PRIME NUMBER Example
Remote Computing
KNAPSACK Example
SPMD Parallelism
fmincon Example
Codistributed Arrays
A 2D Heat Equation
Conclusion

Burkardt/Cliff MATLAB Parallel Computing

PRIME: The Prime Number Example

For our next example, we want a simple computation involving a

loop which we can set up to run for a long time.
We’ll choose a program that determines how many prime numbers
there are between 1 and N.
If we want the program to run longer, we increase the variable N.
Doubling N makes the run time increase by a factor of 4.

Burkardt/Cliff MATLAB Parallel Computing

PRIME: Program Text

f u n c t i o n t o t a l = prime number ( n )

%% PRIME NUMBER r e t u r n s t h e number o f p r i m e s b e t w e e n 1 and N .

total = 0;

for i = 2 : n

prime = 1;

for j = 2 : sqrt ( i )
i f ( mod ( i , j ) == 0 )
prime = 0;
break
end
end

t o t a l = t o t a l + prime ;

end

return
end

Burkardt/Cliff MATLAB Parallel Computing

PRIME: We can run this in parallel

We can parallelize the loop whose index is i, replacing for by

parfor. The computations for different values of i are independent.
There is one variable that is not independent of the loops, namely
total. This is simply computing a running sum (a reduction
variable), and we only care about the final result. MATLAB is
smart enough to be able to handle this summation in parallel.
To make the program parallel, we replace for by parfor. That’s all!

Burkardt/Cliff MATLAB Parallel Computing

PRIME: Execution Commands

m a t l a b p o o l ( ’ open ’ , ’ l o c a l ’ , 4 )

n = 50;

w h i l e ( n <= 500000 )
primes = prime number parallel ( n ) ;
f p r i n t f ( 1 , ’ %8d %8d\n ’ , n , p r i m e s ) ;
n = n ∗ 10;
end

matlabpool ( ’ c l o s e ’ )

Burkardt/Cliff MATLAB Parallel Computing

PRIME: Timing

PRIME_NUMBER_PARALLEL_RUN
Run PRIME_NUMBER_PARALLEL with 0, 1, 2, and 4 labs.

N 1+0 1+1 1+2 1+4

50 0.067 0.179 0.176 0.278

500 0.008 0.023 0.027 0.032
5000 0.100 0.142 0.097 0.061
50000 7.694 9.811 5.351 2.719
500000 609.764 826.534 432.233 222.284

Burkardt/Cliff MATLAB Parallel Computing

PRIME: Timing Comments

There are many thoughts that come to mind from these results!
Why does 500 take less time than 50? (It doesn’t, really).
How can ”1+1” take longer than ”1+0”?
(It does, but it’s probably not as bad as it looks!)
This data suggests two conclusions:
Parallelism doesn’t pay until your problem is big enough;
AND
Parallelism doesn’t pay until you have a decent number of workers.

Burkardt/Cliff MATLAB Parallel Computing

PRIME: Using the BATCH Command

Instead of running a program from the MATLAB command line,

we can use the batch command, and have it execute “elsewhere”.
Elsewhere might simply be on other workers; later we will see that
we could also run the job on a remote cluster, such as ITHACA.
We have to run a script, not a function (we can’t give it input!).
So we might run our prime number parallel function with n fixed
at 500,000.
The matlabpool command now needs 1 extra worker to be the
client. On our desktop PC’s, we only have 2 cores, so we won’t
gain anything in speed.

Burkardt/Cliff MATLAB Parallel Computing

PRIME: A parallel script version

n = 500000;

%f u n c t i o n t o t a l = p r i m e n u m b e r ( n )

%% PRIME NUMBER r e t u r n s t h e number o f p r i m e s b e t w e e n 1 and N .

total = 0;

parfor i = 2 : n

prime = 1;

parfor j = 2 : sqrt ( i )
i f ( mod ( i , j ) == 0 )
prime = 0;
break
end
end

t o t a l = t o t a l + prime ;

end

%end

Burkardt/Cliff MATLAB Parallel Computing

PRIME: Using the BATCH Command

job = batch ( ’prime_number_script’, ...

’configuration’, ’local’, ... <-- Run it locally.
’matlabpool’, 2 ) <-- Two workers.

wait ( job ); <-- One way to find out when job is done.

load ( job ); <-- Load the output variables from

the job into the MATLAB workspace.

total <-- We can examine the value of TOTAL.

destroy ( job ); <-- Clean up

Burkardt/Cliff MATLAB Parallel Computing

PRIME: Using the BATCH Command

Using the wait command is easy, but it locks up your MATLAB

session.
Using batch, you can submit multiple jobs:

job1 = batch ( ... )

job2 = batch ( ... )

Using get, you can check on any job’s status:

get ( job1, ’state’ )

Using load, you can examine just a single output variable from a
finished job if you list its name:

total = load ( job2, ’total’ )

Burkardt/Cliff MATLAB Parallel Computing

PRIME: Remote Usage

The BATCH command can run your job elsewhere!

job = batch ( ’prime_number_script’, ...

’configuration’, ’ithaca_2009b’, ... <-- Run remotely
’matlabpool’, 32 ) <-- Use 32 workers.

get ( job, ’State’ ); <-- ’finished’ if job is done,

(and doesn’t lock your session like wait() does).

load ( job ); <-- Loads all output from the job.

total <-- We can examine the value of TOTAL.

destroy ( job ); <-- Clean up

Burkardt/Cliff MATLAB Parallel Computing

MATLAB Parallel Computing

Introduction
Local Parallel Computing
The MD Example
PRIME NUMBER Example
Remote Computing
KNAPSACK Example
SPMD Parallelism
fmincon Example
Codistributed Arrays
A 2D Heat Equation
Conclusion

Burkardt/Cliff MATLAB Parallel Computing

REMOTE: Enabling Remote Computing

MATLAB can run your programs on a remote machine.

From your desktop, you can submit jobs and review results.
Setting this up takes some work:
1 Update your MATLAB to R2009(a/b) with PCT;
2 Copy some functions into your toolbox/local directory;
3 Copy and customize a new configuration file;
4 Create a MATLAB work directory on your PC;
5 Get an account on Ithaca;
6 Create a MATLAB work directory on Ithaca;
7 Enable passwordfree logins.
The file matlab remote submission.pdf discusses these steps.

Burkardt/Cliff MATLAB Parallel Computing

REMOTE: Communication

It’s important to have some idea of how this communication works.

Your desktop batch command prompts MATLAB to copy the
script file (and any file dependencies) and send them up to Ithaca’s
MATLAB work directory.
On Ithaca, a command is sent to the queueing system, requesting
access to the appropriate number of nodes, to run your script with
Ithaca’s copy of MATLAB.
Output from the script is copied into Ithaca’s MATLAB work
directory and then copied back to your PC work directory.
You don’t actually have to be running MATLAB on your PC while
the job is running on Ithaca.

Burkardt/Cliff MATLAB Parallel Computing

PRIME: The BATCH Command

job_id = batch (
’script_to_run’, ...
’configuration’, ’local’ or ’ithaca_2009b’, ...
’FileDependencies’, ’file’ or {’file1’,’file2’}, ...
’PathDependencies’, ’path’ or {’path1’,’path2’}, ...
’matlabpool’, number of workers (can be zero!) )

Note that you do not include the file extension when naming the
script to run, or the files in the FileDependencies.
See page 13-2 of the PCT User’s Guide for more information.
This slide is NOT in your handouts.

Burkardt/Cliff MATLAB Parallel Computing

REMOTE: The Job State

When you submit a job to run remotely, a file is created in the

local MATLAB work directory, containing a string that is the job’s
current state. From your local MATLAB, the command

get ( job, ’State’ )

will print out the current value (by simply printing this file’s
contents).
Thus, instead of using the wait(job) command, you can simply
check the job’s state from time to time to see if it is ’finished’, and
otherwise go on talking to MATLAB.

Burkardt/Cliff MATLAB Parallel Computing

REMOTE: The Job State

Typical values of the job state variable include:

’pending’: not yet submitted to the queue
’queued’: submitted to the queue
’running’: running
’finished’: ran successfully
’failed’: did not run or failed during run
’destroyed’: you discarded this information

Burkardt/Cliff MATLAB Parallel Computing

REMOTE: The Configuration File

The configuration file tells MATLAB the names of work directories,

the number of workers on the remote system, and so on.
If you want to set up access to Ithaca, you will get a partially filled
out configuration file. You complete it by filling in names of
directories for job data on:
your PC: C:\matlab jobdata
or your Mac: /Users/burkardt/matlab jobdata
Ithaca: /home/burkardt/matlab jobdata
These names are arbitrary, but the named directories must exist or
be created before MATLAB can use them.

Burkardt/Cliff MATLAB Parallel Computing

REMOTE: Example Configuration File

Burkardt/Cliff MATLAB Parallel Computing

MATLAB Parallel Computing

Introduction
Local Parallel Computing
The MD Example
PRIME NUMBER Example
Remote Computing
KNAPSACK Example
SPMD Parallelism
fmincon Example
Codistributed Arrays
A 2D Heat Equation
Conclusion

Burkardt/Cliff MATLAB Parallel Computing

KNAPSACK: Distributed Computing

In the examples of parallel programming that we have seen, the

parallelism really does mean ”parallel”, that is, at the same time.
In a parfor computation, the master process starts with all the data
and the program, the workers always start and end together, and
the results from each worker are collected by the master program.
The term distributed computing describes a “looser” computation
which may be broken into completely independent parts which
don’t communicate, don’t run at the same time, and don’t run in
any particular order.
We will look at an example, called the knapsack problem, which
demonstrates this method of computing.

Burkardt/Cliff MATLAB Parallel Computing

KNAPSACK: Problem Definition

Suppose we have a knapsack with a limited capacity, and a

number of objects of varying weights. We want to find a subset of
the objects which exactly meets the capacity of the knapsack.
(This is sometimes called the greedy burglar’s problem!)
Symbolically, we are given a target value t, and n weights w. We
seek k indices s, a subset of the weights, so that
k
X
t= w (s(i))
i=1

We don’t know if a given problem has 0, 1, or many solutions.

Burkardt/Cliff MATLAB Parallel Computing

KNAPSACK: Encoding
A solution of the problem is a subset of w. A subset of a set of n
elements can be represented by a binary string of length n.
Therefore every binary string from 0 to 2n − 1 is also a code for a
subset that is a possible solution.
For weights w={15,11,10,8,3}, target t=24, we have:
Code Binary Code Subset Weight
0 00000 {} 0
1 00001 {1} 3
2 00010 {2} 8
3 00011 {2,1} 11
4 00100 {3} 10
5 00101 {3,1} 13
6 00110 {3,2} 18
... ... ... ...
31 11111 {5,4,3,2,1} 47
Burkardt/Cliff MATLAB Parallel Computing
KNAPSACK: Algorithm

Although more sophisticated methods are available, a simple

search scheme can be used. We simply examine code over the
range 0 to 2n − 1, compute the corresponding subset, add up the
selected weights, and compare to t.
For instance, the code of 22 = binary 10110 = subset {5,3,2} and
hence a weight of 15+10+8=33, which is too high.
Notice that the process of checking one possibility is completely
independent of checking any other.
One program could check them all, or we could subdivide the
range, and check the subranges in any order and at any time.

Burkardt/Cliff MATLAB Parallel Computing

KNAPSACK: MATLAB Program

function [ code, subset ] = knapsack ( w, t )

n = length ( w );
for code = 0 : 2^n-1
% Convert CODE into vector of indices in W.
subset = find ( bitget ( code, 1:n ) );
% Did we match the target sum?
if ( sum ( w(subset) ) == t )
return
end
end

return
end

Burkardt/Cliff MATLAB Parallel Computing

KNAPSACK: Distributed Version

Suppose we break the problem into distinct subranges to check.

MATLAB’s distributed computing option calls the original problem
the job. Checking a subrange is one task of the job. Each task
calls the same MATLAB function with different arguments.
MATLAB lets us “submit” the job; a task is assigned to a worker
that is available. These tasks can run locally or remotely,
simultaneously or sequentially or at substantially different times.
Because each tasks runs when it can, and they don’t communicate,
overhead and scheduling delays are avoided.
The job completes when all tasks are run.

Burkardt/Cliff MATLAB Parallel Computing

KNAPSACK: Distributed MATLAB Program

function [ code, subset ] = knapdist ( w, t, range )

n = length ( w );
for code = range(1) : range(2)
% Convert CODE into vector of indices in W.
subset = find ( bitget ( code, 1:n ) );
% Did we match the target sum?
if ( sum ( w(subset) ) == t )
return
end
end

return
end

Burkardt/Cliff MATLAB Parallel Computing

KNAPSACK: Distributed MATLAB Program

The program can work on the whole problem or a given subrange,

depending on the values in range.
The MATLAB function bitget returns a vector of 0’s and 1’s for
positions 1 to n in code.
The function find returns the locations of the 1’s, which is how we
get our list of weights to try for this subset.

Burkardt/Cliff MATLAB Parallel Computing

KNAPSACK: Define the Job and its Tasks

job = createJob ( ’configuration’, ’local’, ...

’FileDependencies’, ...
{ ’knapdist.m’, ’knapweights.mat’ } );

i2 = -1;
for task = 1 : 4
i1 = i2 + 1;
i2 = floor ( ( 2^n - 1 ) * task / 4 );
createTask ( job, @knapdist, 2, { w, t, [ i1, i2 ] } );
end

Burkardt/Cliff MATLAB Parallel Computing

KNAPSACK: Define the Job and its Tasks

The createJob command is like the batch command, except it

doesn’t say what we’re going to do, or request a specific number of
workers.

job_id = createJob (
’configuration’, ’local’ or ’ithaca_2009b’, ...
’FileDependencies’, ’file’ or {’file1’,’file2’}, ...
’PathDependencies’, ’path’ or {’path1’,’path2’} )

See page 13-52 of the PCT User’s Guide.

This slide is NOT in your handouts.

Burkardt/Cliff MATLAB Parallel Computing

KNAPSACK: Define the Job and its Tasks

The createTask command defines the tasks that make up the job.
In particular, it names the MATLAB function that will be called,
the number of output arguments it has, and the values of the input
arguments.

task_id = createTask (
job_id, ... <-- ID of the job
@function, ... <-- MATLAB function to be called
numarg, ... <-- Number of output arguments
{ arg1,arg2,...} ) <-- Input arguments

See page 13-59 of the PCT User’s Guide.

This slide is NOT in your handouts.

Burkardt/Cliff MATLAB Parallel Computing

KNAPSACK: Submit the Job

With the following commands, we submit the job, and then pause
our interactive MATLAB session until the job is finished.
We then retrieve the output arguments from each task, in a cell
array we call results.

submit ( job ); <-- Sends the job

wait ( job ); <-- Waits for completion.
results = getAllOutputArguments ( job );
destroy ( job ); <-- Clean up

Burkardt/Cliff MATLAB Parallel Computing

KNAPSACK: BATCH versus CREATEJOB

The batch command can be thought of as a simplified version of

createJob + createTask + submit.
It assumes you only have 1 task and that the task is defined by a
script with no input arguments, and it’s ready to submit.
Otherwise, both commands are doing the same thing, finding out
what you want to and where it should be executed, assigning it a
logical identifier, and sending the work to the right machine.
For the distributed KNAPSACK job, of course, we needed the
extra flexibility of the createJob command.

Burkardt/Cliff MATLAB Parallel Computing

KNAPSACK: Examine the results

Because your job involved multiple tasks, the output must be

returned to you in a cell array. To see output result 2 from task 3,
you refer to results{3,2}.

for task = 1 : 4
if ( isempty ( results{task,1} ) )
fprintf ( 1, ’Task %d found no solutions.\n’, task );
else
disp ( ’Weights:’ );
disp ( results{task,1} );
disp ( ’Weight Values:’ );
disp ( results{task,2} );
end
end

Burkardt/Cliff MATLAB Parallel Computing

KNAPSACK: Running Jobs Remotely

If you have set up your machine so that the local copy of MATLAB
can talk to remote copies, then this same distributed job can be
run on a remote machine, such as the Virginia Tech ithaca cluster.
All you have to do is change the configuration argument when you
define the job:

job = createJob ( ’configuration’, ’ithaca_2009b’, ...

’FileDependencies’, ...
{ ’knapdist.m’, ’knapweights.mat’ } );

Burkardt/Cliff MATLAB Parallel Computing

MATLAB Parallel Computing

Introduction
Local Parallel Computing
The MD Example
PRIME NUMBER Example
Remote Computing
KNAPSACK Example
SPMD Parallelism
fmincon Example
Codistributed Arrays
A 2D Heat Equation
Conclusion

Burkardt/Cliff MATLAB Parallel Computing

SPMD: Single Program, Multiple Data

The parfor command is easy to use, but it only lets us do

parallelism in terms of loops. The only choice we make is whether
a loop is to run in parallel. We can’t determine how the loop
iterations are divided up, we can’t be sure which lab runs which
iteration, we can’t examine the work of any individual lab.
Distributed programming allows us to run different programs, or
the same program with different inputs, but they can’t talk to each
other, that is, communicate or share results.
The SPMD command is like working with a very simplified version
of MPI. There is a client process and workers, but now the workers
are given identifiers. Each worker decides what to do based on its
ID. Each worker can communicate with the client. Any two
workers can communicate through the client.

Burkardt/Cliff MATLAB Parallel Computing

SPMD: The SPMD Command

Let’s assume we’ve issued a matlabpool command, and have a

client (that is, the “main” copy of MATLAB) and a number of
workers or labs.
The first thing to notice about a program using SPMD is that
certain blocks of code are delimited:

fprintf ( 1, ’ Set up the integration limits:\n’ );

spmd
a = ( labindex - 1 ) / numlabs;
b = labindex / numlabs;
end

Burkardt/Cliff MATLAB Parallel Computing

SPMD: Each worker has an ID

The spmd delimiter marks a section of code which is to be carried

out by each lab, and not by the client.
The fact that the MATLAB program can be marked up into
instructions for the client and instructions for the workers explains
the single program part of SPMD.
But how do multiple workers do different things if they see the
same instructions? Luckily, each worker is assigned a unique
identifier, the value of the variable labindex.
The worker also gets the value of numlabs, the total number of
workers. This information is enough to ensure that each worker
can be assigned different tasks. This explains the multiple data
part of SPMD!

Burkardt/Cliff MATLAB Parallel Computing

SPMD: Use the ID to assign work

Now let’s go back to our program fragment. But first we must

explain that we are trying to approximate an integral over the
interval [0,1]. Using SPMD, we are going to have each lab pick a
portion of that interval to work on, and we’ll sum the result at the
end. Now let’s look more closely at the statements:

fprintf ( 1, ’ Set up the integration limits:\n’ );

spmd
a = ( labindex - 1 ) / numlabs;
b = labindex / numlabs;
end

Burkardt/Cliff MATLAB Parallel Computing

SPMD: One Name Must Reference Several Values

Each worker will compute different values of a and b. These values

are stored locally on that worker’s memory.
The client can access the values of these variables, but it must
specify the particular lab from whom it wants to check the value,
using “curly brackets”: a{i}.
The variables stored on the workers are called composite variables;
they are somewhat similar to MATLAB’s cell arrays.
It’s important to respect the rules for composite variable names. In
particular, if a is used on the workers, then the name a is also
“reserved” on the client program (although there it’s an indexed
variable). The client should not try to use the name a for other
variables!

Burkardt/Cliff MATLAB Parallel Computing

SPMD: Dealing with Composite Variables
So we could print all the values of a and b in two ways:

spmd
a = ( labindex - 1 ) / numlabs;
b = labindex / numlabs;
fprintf ( 1, ’ A = %f, B = %f\n’, a, b );
end

or

spmd
a = ( labindex - 1 ) / numlabs;
b = labindex / numlabs;
end
for i = 1 : 4 <-- "numlabs" wouldn’t work here!
fprintf ( 1, ’ A = %f, B = %f\n’, a{i}, b{i} );
end
Burkardt/Cliff MATLAB Parallel Computing
SPMD: The Solution in 4 Parts
Assuming we’ve defined our limits of integration, we now want
to carry out the trapezoid rule for integration:
spmd
x = linspace ( a, b, n );
fx = f ( x );
quad_part = ( fx(1) + 2 * sum(fx(2:n-1)) + fx(n) )
/2 /(n-1);
fprintf ( 1, ’ Partial approx %f\n’, quad_part );
end
with result:
2 Partial approx 0.874676
4 Partial approx 0.567588
1 Partial approx 0.979915
3 Partial approx 0.719414
Burkardt/Cliff MATLAB Parallel Computing
SPMD: Combining Partial Results

We really want one answer, the sum of all these approximations.

One way to do this is to gather the answers back on the client:

quad = sum ( quad_part{1:4} );

fprintf ( 1, ’ Approximation %f\n’, quad );

with result:

Approximation 3.14159265

Burkardt/Cliff MATLAB Parallel Computing

SPMD: Source Code for QUAD SPMD
f u n c t i o n v a l u e = quad spmd ( n )

f p r i n t f ( 1 , ’ Compute l i m i t s \n ’ ) ;
spmd
a = ( l a b i n d e x − 1 ) / n umlab s ;
b = labindex / n umlab s ;
f p r i n t f ( 1 , ’ Lab %d w o r k s on [% f ,% f ] . \ n ’ , l a b i n d e x , a , b ) ;
end

f p r i n t f ( 1 , ’ Each l a b e s t i m a t e s p a r t o f t h e i n t e g r a l . \ n ’ ) ;
spmd
i f ( n == 1 )
quad part = ( b − a ) ∗ f ( ( a + b ) / 2 ) ;
else
x = linspace ( a , b , n );
fx = f ( x );
q u a d p a r t = ( b − a ) ∗ ( f x ( 1 ) + 2 ∗ sum ( f x ( 2 : n−1) ) + f x ( n ) ) ...
/ 2.0 / ( n − 1 ) ;
end
f p r i n t f ( 1 , ’ Approx %f \n ’ , q u a d p a r t ) ;
end

f p r i n t f ( 1 , ’ Use GPLUS t o sum t h e p a r t s . \ n ’ ) ;

spmd
quad = g p l u s ( q u a d p a r t ) ;
i f ( l a b i n d e x == 1 )
fprintf ( 1, ’ A p p r o x i m a t i o n = %f \n ’ , quad )
end
end

return
end
Burkardt/Cliff MATLAB Parallel Computing
SPMD: Combining Values Directly

MATLAB also provides commands to combine values directly on

the labs. The command we need is called gplus(); it computes the
sum across all the labs of the given variable, and returns the value
of that sum to each lab:

spmd
x = linspace ( a, b, n );
fx = f ( x );
quad_part = ( fx(1) + 2 * sum(fx(2:n-1)) + fx(n) )
/2 /(n-1);
quad = gplus(quad_part);
if ( labindex == 1 )
fprintf ( 1, ’ Approximation %f\n’, quad );
end
end

Burkardt/Cliff MATLAB Parallel Computing

SPMD: Reduction Operators
gplus() is implemented by the gop() command, which carries
out an operation across all the labs.
gplus(a) is really shorthand for gop ( @plus, a ), where plus is
the name of MATLAB’s function that actually adds numbers.
Other reduction operations include:

gop(@max,a), maximum of a;
gop(@min,a), minimum of a;
gop(@and.a), AND of a;
gop(@or.a), OR of a;
gop(@xor.a), XOR of a;
gop(@bitand.a), bitwise AND of a;
gop(@bitor.a), bitwise OR of a;
gop(@bitxor.a), bitwise XOR of a.

Burkardt/Cliff MATLAB Parallel Computing

SPMD: MPI-Style Messages

SPMD supports some commands that allow the programmer to do

message passing, in the MPI style:

labSend, send data directly to another lab;

labReceive, receive data directly from another lab;
labSendReceive, interchange data with another lab.

For details on how these commands work, start with the MATLAB
HELP facility!
For more information, refer to the documentation for the Parallel
Computing Toolbox.

Burkardt/Cliff MATLAB Parallel Computing

MATLAB Parallel Computing

Introduction
Local Parallel Computing
The MD Example
PRIME NUMBER Example
Remote Computing
KNAPSACK Example
SPMD Parallelism
fmincon Example
Codistributed Arrays
A 2D Heat Equation
Conclusion

Burkardt/Cliff MATLAB Parallel Computing

fmincon and UseParallel

In most cases, making use of parallelism requires some re-coding,

perhaps even serious restructuring of your approach. Beginning
with Version 4.0 (R2008a) of the Optimization Toolbox we can
easily take advantage of parallelism in constructing finite-difference
estimates of the gradient of the cost functional and the Jacobian
of any nonlinear constraint functions.
Using the optimset command we simply set the flag
UseParallel to (the string) always.
In the run opt example we seek an optimal steering history for a
boat moving in a spatially varying current. The control history is
approximated as piecewise constant on a given time-grid. The
optimization parameter is the vector of the values of the steering
angle on the intervals. The cost functional and constraints depend
on the final position of the boat in the plane.

Burkardt/Cliff MATLAB Parallel Computing

fmincon and UseParallel

The main work in evaluating these functions is the (numerical)

integration of the dynamics with a prescribed steering history.
The dynamics are given by

ẋ(t) = −κy (t) + cos(θ(t))

ẏ (t) = sin(θ(t))

with initial condition x(0) = y (0) = 0.

The problem is to maximize x(tf ) with the constraint y (tf ) > yf
(tf , yf , and κ are given).

Burkardt/Cliff MATLAB Parallel Computing

The RUN OPT Example
f u n c t i o n z s t a r = r u n o p t ( f name , n )
% Function to run a f i n i t e d i m e n s i o n a l o p t i m i z a t i o n problem
% b a s e d on a d i s c r e t i z a t i o n o f a Mayer p r o b l e m i n o p t i m a l c o n t r o l .

% f n a m e p o i n t s t o a u s e r−s u p p l i e d f u n c t i o n w i t h a s i n g l e i n p u t argument
% n i s a d i s c r e t i z a t i o n p a r a m e t e r . The f i n i t e −d i m e n s i o n a l p r o b l e m a r i s e s
% by t r e a t i n g t h e ( s c a l a r ) c o n t r o l a s p i e c e w i s e c o n s t a n t
% The f u n c t i o n r e f e r e n c e d by f n a m e must d e f i n e t h e e l e m e n t s o f
% t h e u n d e r l y i n g o p t i m a l c o n t r o l p r o b l e m . See ’ z e r m e l o ’ a s an e x a m p l e .

%% Problem d a t a

PAR = f e v a l ( s t r 2 f u n c ( f n a m e ) , n ) ;

% some l i n e s o m i t t e d

%% A l g o r i t h m s e t up
OPT = o p t i m s e t ( o p t i m s e t ( ’ f m i n c o n ’ ) , . . .
’ LargeScale ’ , ’ off ’ , . . .
’ A l g o r i t h m ’ , ’ a c t i v e −s e t ’ , ...
’ Display ’ , ’ i t e r ’ , . . .
’ U s e P a r a l l e l ’ , ’ Always ’ ) ;
h c o s t = @( z ) g e n e r a l c o s t ( z , PAR ) ;
h c n s t = @( z ) g e n e r a l c o n s t r a i n t ( z , PAR ) ;

%% Run t h e a l g o r i t h m
[ z star , f star , exit ] = . . .
f m i n c o n ( h c o s t , z0 , [ ] , [] , [] , [ ] , LB , UB, h c n s t , OPT ) ;
i f e x i t >=0 && i s f i e l d (PAR , ’ p l o t ’ )
f e v a l (PAR . p l o t , z s t a r , PAR)
end

Burkardt/Cliff MATLAB Parallel Computing

The RUN OPT Example: source material

A folder with the software and example output is in the

parallel matlab folder on your desktop. The folder looks like:

Burkardt/Cliff MATLAB Parallel Computing

Cell Arrays

cell arrays are rectangular arrays, whose content can be any

Matlab variable, including a cell array

>> A = eye(2); B = ones(2); C = rand(3,4); D = ’a string’;

>> G = { A B ; C D};
>> G

G = [2x2 double] [2x2 double]

[3x4 double] ’a string’

>> isa(G, ’cell’)

ans = 1

Burkardt/Cliff MATLAB Parallel Computing

Cell Arrays: Two ways of indexing

A cell array may be indexed in two ways:

1 G(1) - the result of cell indexing is a cell array
2 G{1} - the result of content indexing is the contents of the
cell(s)

>> F1 = G(1, 1:2)

F1 = [2x2 double] [2x2 double]

>> isa(F1, ’cell’)

ans = 1

Burkardt/Cliff MATLAB Parallel Computing

Cell Arrays: Two ways of indexing
G{1} - the result of content indexing is the cell’s contents

>> F2 = G{1, 2}
F2= 1 1
1 1
>> whos
Name Size Bytes Class Attributes

A 2x2 32 double
B 2x2 32 double
C 3x4 96 double
D 1x8 16 char
F1 1x2 184 cell
F2 2x2 32 double
G 2x2 416 cell

Burkardt/Cliff MATLAB Parallel Computing

SPMD mode: composite variables

SPMD mode creates a composite object on the client

composite objects are indexed in the same ways as cell arrays

>> spmd
V = eye(2) + (labindex -1);
end
>> V{1}
ans = 1 0
0 1
>> V{2}
ans = 2 1
1 2
>> whos
Name Size Bytes Class Attributes
V 1x2 373 Composite

Burkardt/Cliff MATLAB Parallel Computing

MATLAB Parallel Computing

Introduction
Local Parallel Computing
The MD Example
PRIME NUMBER Example
Remote Computing
KNAPSACK Example
SPMD Parallelism
fmincon Example
Codistributed Arrays
A 2D Heat Equation
Conclusion

Burkardt/Cliff MATLAB Parallel Computing

Codistributed Arrays

Codistributed arrays allow the user to build (m × n) matrices so

that, for example, each ’lab’ stores/operates on a contiguous block
of columns. More general (rectangular) constructs are possible but
are not covered here.
We shall demonstrate these ideas in pmode

>> pmode start 4

Starting pmode using the parallel configuration ’local’.
Waiting for parallel job to start...
Connected to a pmode session with 4 labs.

Many of the builtin Matlab matrix constructors can be assigned

the class ’codistributed’. For example:

>> M = speye(1000, codistributor());

Burkardt/Cliff MATLAB Parallel Computing

Codistributed arrays (cont’d)

’codistributor’ is the constructor and specifies which dimension is

used to distribute the array. With no argument, we take the
default, which is ’1d’ or one-dimensional. By default, two
dimensional arrays are distributed by columns.
codistributor(M) returns information about the distributed
structure of the array M.
If the number of columns is an integer multiple of the number of
’labs’, then the (default) distribution of columns among the labs is
obvious. Else we invoke codistributor (or other Matlab
supplied procedure).
getLocalPart(M) returns the part of the codistributed array on
this lab.

Burkardt/Cliff MATLAB Parallel Computing

Codistributed arrays (cont’d)

%%%% run these in Matlab

>> pmode start 4
>> M = speye(1000, codistributor() )
>> codistributor(M)

>> M = ones(1000, 1, codistributor() )

>> codistributor(M)
%%%%

Burkardt/Cliff MATLAB Parallel Computing

Codistributed arrays (cont’d)

One can construct local arrays on the labs and assemble them into
a codistributed array:

%%%% run these in Matlab

>> M = rand(100, 25) + labindex;
>> Mc = codistributed(M);
>> max(max(abs(M - getLocalPart(Mc))))
>> Mc(12,13)
%%%%

Of course, in applications the construction on each lab will involve

user-defined code. We will now demonstrate this idea for an
unsteady heat equation in two space dimensions.

Burkardt/Cliff MATLAB Parallel Computing

MATLAB Parallel Computing

Introduction
Local Parallel Computing
The MD Example
PRIME NUMBER Example
Remote Computing
KNAPSACK Example
SPMD Parallelism
fmincon Example
Codistributed Arrays
A 2D Heat Equation
Conclusion

Burkardt/Cliff MATLAB Parallel Computing

2DHEAT: State Equations

An example: 2D unsteady heat equation:

   
∂T ∂ ∂T ∂ ∂T
σCp = kx + ky + F (x, y , t)
∂t ∂x ∂x ∂y ∂y
(x, y ) ∈ {(x, y ) | 0 ≤ x ≤ L, 0 ≤ y ≤ w } ⊂ IR2 , t > 0 ,

where:
F (x, y , t) is a specified source term,
σ > 0 is the areal density of the material,
Cp > 0 is the thermal capacitance of the material, and
kx > 0 (ky > 0) is the conductivity in the x direction (the
y -direction).

Burkardt/Cliff MATLAB Parallel Computing

2DHEAT: Boundary Conditions

Boundary conditions for our problem are:

∂T (x, 0) ∂T (x, w )
= =0,
∂y ∂y

∂T (L, y )
kx = f (y ) ,
∂x
∂T (0, y )
kx = α(y ) (T (0, y ) − β(y )) .
∂x

Burkardt/Cliff MATLAB Parallel Computing

2DHEAT: Time-Discretized State Equations

We use backward Euler in time and finite-elements in space to

arrive at
Z  
n+1 n ∆t
T −T − F (x, y , tn+1 ) Ψ dω
Ω σ Cp
Z Z 
∆t n+1

n+1


+ k∇T · ∇Ψ dω + k∇T · n̂ Ψ dσ = 0 ,
σ Cp Ω ∂Ω

4
where T n (x, y ) = T (n ∆t, x, y ), and Ψ ∈ H 1 (Ω) is a test function.

Burkardt/Cliff MATLAB Parallel Computing

2DHEAT: Time-Discretized Boundary Conditions

Imposing the specified boundary conditions, the boundary term

evaluates to
Z Z w
n+1


k∇T · n̂ Ψ dσ = f (y )Ψ(L, y ) dy
∂Ω 0
Z 0
α(y ) T n+1 (0, y ) − β(y ) Ψ(0, y ) dy .
 
−
w

Details are described in the 2D heat ex.pdf file in the distribution

material.

Burkardt/Cliff MATLAB Parallel Computing

2DHEAT: Finite Element Grid
We use quadratic functions on triangular elements
Impose a regular nx × ny = ((2` + 1) × (2m + 1)) grid.
Using the odd-labeled points we generate ` m rectangles; diagonals
divide these into 2 ` m triangles.
Here’s the case nx = ny = 5 (8 elements, 25 grid points):

Burkardt/Cliff MATLAB Parallel Computing

2DHEAT: Finite Element Equations
PN
Seek an approximate solution: TNn (x, y ) = n
=1 z Φ (x, y ) .

X Z
Φ (x, y ) Φı (x, y ) dω
 Ω
Z Z 0 
∆t
+ (k∇Φ · ∇Φı ) dω + α(y ) Φ (0, y ) Φı (0, y ) dy zn+1
σ Cp Ω w
X Z  
∆t
Z 
− Φ (x, y ) Φı (x, y ) dω zn − F (x, y , tn+1 )Φı dω
 Ω σ Cp Ω
Z w Z 0 
∆t
− f (y )Φı (L, y ) dy + α(y )β(y )Φı (0, y ) dy = 0
σ Cp 0 w

We can rewrite this in matrix terminology:

(M1 + M2 ) z n+1 − M1 z n + F(tn+1 ) + b = 0 .

Burkardt/Cliff MATLAB Parallel Computing

2DHEAT: Modifying a Serial Code

So our computation requires repeatedly forming and solving

systems of the form:

(M1 + M2 ) z n+1 − M1 z n + F(tn+1 ) + b = 0 .

We began with a serial code for building M1 , M2 , F and b.

Here, we briefly note the changes to build codistributed
versions of these.

Burkardt/Cliff MATLAB Parallel Computing

2DHEAT: ASSEMB CO Source Code (begin)

f u n c t i o n [ M1, M2, F , b , x , e c o n n ] = a s s e m b c o ( param )

% The FEM e q u a t i o n f o r t h e temp . d i s t a t t i m e t {n+1} s a t i s f i e s
% ( M 1 + M 2 ) z ˆ{n+1} − M 1 z ˆn + F + b = 0

%% I n i t i a l i z a t i o n & g e o m e t r y
%−−−−l i n e s omitted
%% S e t up c o d i s t r i b u t e d s t r u c t u r e

% column p o i n t e r s and s u c h f o r c o d i s t r i b u t e d a r r a y s
Vc = c o d c o l o n ( 1 , n e q u a t i o n s ) ;
l P = l o c a l P a r t ( Vc ) ; l P 1 = l P ( 1 ) ; l P e n d = l P ( end ) ;
dPM = d i s t r i b u t i o n P a r t i t i o n ( c o d i s t r i b u t o r ( Vc ) ) ;
c o l s h f t = [ 0 cumsum (dPM ( 1 : end − 1 ) ) ] ;

% local sparse arrays

M1 lab = s p a r s e ( n e q u a t i o n s , dPM( l a b i n d e x ) ) ; M2 lab = M1 lab ;
b l a b = s p a r s e (dPM( l a b i n d e x ) , 1 ) ; F l a b = b l a b ;

%% B u i l d t h e f i n i t e e l e m e n t m a t r i c e s − B e g i n l o o p o v e r e l e m e n t s
f o r n e l =1: n e l e m e n t s
nodes local = e conn ( n e l , : ) ;% which nodes a r e i n t h i s element
% s u b s e t o f n o d e s / c o l u m n s on t h i s l a b
l a b n o d e s l o c a l = my extract ( n o d e s l o c a l , lP 1 , lP end ) ;
i f ˜ isempty ( l a b n o d e s l o c a l ) % c o n t i n u e the c a l c u l a t i o n f o r t h i s elmnt
%−−− c a l c u l a t e l o c a l a r r a y s − l i n e s o m i t t e d

Burkardt/Cliff MATLAB Parallel Computing

2DHEAT: ASSEMB CO Source Code (end)
%% A s s e m b l e c o n t r i b u t i o n s i n t o t h e g l o b a l s y s t e m m a t r i c e s ( on t h i s l a b )
%−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−
%
for n t = 1: nel dof % l o c a l DOF − t e s t f c n
t glb = nodes local ( n t ); % g l o b a l DOF − t e s t f c n
f o r n u = 1 : s i z e ( l a b n o d e s l o c a l , 1)
n l o c j = l a b n o d e s l o c a l ( n u , 1 ) ; % l o c a l DOF i n c u r r e n t n e l
n g l b j = l a b n o d e s l o c a l ( n u , 2) . . .
−c o l s h f t ( l a b i n d e x ) ; % g l o b a l DOF
M1 lab ( t g l b , n g l b j ) = M1 lab ( t g l b , n g l b j ) . . .
+ M1 loc ( n t , n l o c j ) ;
M2 lab ( t g l b , n g l b j ) = M2 lab ( t g l b , n g l b j ) . . .
+ param . d t ∗ M 2 l o c ( n t , n l o c j ) ;
end

%
if t g l b >= l P 1 && t g l b <= l P e n d % i s node on t h i s l a b ?
t loc = t glb − col shft ( labindex );
b l a b ( t l o c , 1 ) = b l a b ( t l o c , 1 ) − param . d t ∗ b l o c ( n t , 1 ) ;
F l a b ( t l o c , 1 ) = F l a b ( t l o c , 1 ) − param . d t ∗ F l o c ( n t , 1 ) ;
end
end % f o r n t
end % i f n o t empty
end % n e l

%
% Assemble the l a b c o n t r i b u t i o n s i n a c o d i s t r i b u t e d format
M1 = c o d i s t r i b u t e d ( M1 lab , c o d i s t r i b u t o r ( ’ 1d ’ , 2 ) ) ;
M2 = c o d i s t r i b u t e d ( M2 lab , c o d i s t r i b u t o r ( ’ 1d ’ , 2 ) ) ;
b = c o d i s t r i b u t e d ( b l a b , c o d i s t r i b u t o r ( ’ 1d ’ , 1 ) ) ;
F = c o d i s t r i b u t e d ( F l a b , c o d i s t r i b u t o r ( ’ 1d ’ , 1 ) ) ;

Burkardt/Cliff MATLAB Parallel Computing

2DHEAT: 5 × 5 grid on 4 labs
There are 8 triangular elements, and 25 nodes.
The nodes are color-coded for the four labs.

Note that lab 1 (green) requires evaluation on 4 of 8 elements,

while lab 2 (blue) requires 7 of 8.
Clearly, our naive nodal assignment to labs leaves the
computational load badly balanced.
Burkardt/Cliff MATLAB Parallel Computing
2DHEAT: 5 × 5 grid on 4 labs

%%%% run these in Matlab

>> pmode start 4
>> Vc = codcolon(1, 25)
>> dPM = distributionPartition(codistributor(Vc))
>> col_shft = [ 0 cumsum(dPM(1:end-1))]
>> whos
%%%%

Burkardt/Cliff MATLAB Parallel Computing

2DHEAT: RUN ASSEMB CO Source Code

% S c r i p t t o a s s e m b l e m a t r i c e s f o r a 2D d i f f u s i o n p r o b l e m

%% s e t p a t h
a d d p a t h ’ . / s u b s s o u r c e / oned ’ ; a d d p a t h ’ . / s u b s s o u r c e / twod ’

%% s e t p a r a m e t e r v a l u e s and a s s e m b l e a r r a y s
param = p d a t a ( ) ;
[ M1, M2, F , b , x , e c o n n ] = a s s e m b c o ( param ) ;

%% c l e a n −up p a t h
rmpath ’ . / s u b s s o u r c e / oned ’ ; rmpath ’ . / s u b s s o u r c e / twod ’

%% S t e a d y s t a t e s o l u t i o n s
z tmp = − f u l l (M2)\ f u l l ( F+b ) ; % T e m p e r a t u r e d i s t r i b u t i o n
z s s = g a t h e r ( z tmp , 1 ) ;

%% P l o t and s a v e a s u r f a c e p l o t
i f l a b i n d e x == 1
xx = x ( 1 : param . n o d e s x , 1 ) ;
yy = x ( 1 : param . n o d e s x : param . n o d e s x ∗param . n o d e s y , 2 ) ;
figure
s u r f ( xx , yy , r e s h a p e ( z s s , param . n o d e s x , param . n o d e s y ) ’ ) ;
x l a b e l ( ’\bf x ’ ) ; y l a b e l ( ’\bf y ’ ) ; z l a b e l ( ’\bf T ’ )
t axis = axis ;
p r i n t −dpng f i g s s . png
close a l l
end

Burkardt/Cliff MATLAB Parallel Computing

2DHEAT: RUN ASSEMB CO Source Material

A folder with the software, example output and descriptive

material is in the parallel matlab folder on your desktop. The
folder should look like:

Burkardt/Cliff MATLAB Parallel Computing

MATLAB Parallel Computing

Introduction
Local Parallel Computing
The MD Example
PRIME NUMBER Example
Remote Computing
KNAPSACK Example
SPMD Parallelism
fmincon Example
Codistributed Arrays
A 2D Heat Equation
Conclusion
Burkardt/Cliff MATLAB Parallel Computing
Conclusion: Desktop Experiments

Virginia Tech has a limited number of concurrent MATLAB

licenses, which include the Parallel Computing Toolbox.
This is one way you can test parallel MATLAB on your desktop
machine.
If you don’t have a multicore machine, you won’t see any speedup,
but you may still be able to run some “parallel” programs.

Burkardt/Cliff MATLAB Parallel Computing

Conclusion: Cluster Experiments

If you want to work with parallel MATLAB on Ithaca, there should

soon be a way to apply for accounts online at the ARC website:

http://www.arc.vt.edu/index.php

Until then, you can get a “friendly user” account by sending mail
to John Burkardt [email protected].

Burkardt/Cliff MATLAB Parallel Computing

Conclusion: PC-to-Cluster Submission

If you want to use parallel MATLAB regularly, you may want to set
up a way to submit jobs from your PC to Ithaca, without logging
in directly.
This requires defining a configuration file on your PC, adding some
scripts to your MATLAB directory, and setting up a secure
connection between your PC and Ithaca. The steps for doing this
are described in the document:

http://people.sc.fsu.edu/~burkardt/pdf/...
matlab_remote_submission.pdf

We will be available to help you with this process.

Burkardt/Cliff MATLAB Parallel Computing

Conclusion: Documentation

On the MathWorks directory, there is a directory for the Parallel

Computing Toolbox:

http://www.mathworks.com/products/parallel-computing/

Look under ”Documentation”. This is a reference manual, not a

guide. It will tell you the details about how to use a particular
command, but not so much which commands you should use.
Under ”Demos and Webinars” you may find help about how to
design the right kind of parallel program for your problem.

Burkardt/Cliff MATLAB Parallel Computing