AMATH 301 – Autumn 2020

Homework 3
Due: 11:59pm, October 23, 2020.
KEY

Writeup problems
1. In this problem we will compare the speed and accuracy for multiple solves of a
special 2000 × 2000 matrix using Gaussian Elimination, LU, and the matrix inverse.
It is often important to consider the trade off between speed and accuracy when
deciding which method to use.
You need to do all of the following parts, but you only need to include
the table in part (f ) and the discussion in parts (g)-(i) in your writeup.

(a) We are going to start by making a special matrix. In matlab, use

N = 2000;
A = -1*diag(ones(N,1)) + 4*diag(ones(N-1,1),1) + 4*diag(ones(N-1,1),-1);

In python use
import numpy as np
N = 2000;
offDiag = np.ones(N-1)
A = -1*np.diag(np.ones(N)) + 4*np.diag(offDiag,1) + 4*np.diag(offDiag,-1)

(b) Use a for loop to solve Ax = b using backslash 100 times for 100 differ-
ent b vectors. Each time through the loop, create a new b vector using the
rand (matlab) or np.random.rand (python) commands. Use tic and toc
(matlab) or time.time() (python) to time how long it takes to do all 100
solves. The for loop should be outside of the loop.
(c) Do the same as in part (b) using LU decomposition. For LU decomposition,
you should calculate the matrices L, U, and P just once outside of your for
loop but inside of your timer.
(d) Do the same as in part (b) using the matrix inverse, inv(A) (matlab) or
np.linalg.inv(A) (python). You should calculate the inverse of your matrix,
A−1 , just once outside of your for loop but inside of your timer.

1
(e) The residual vector, r = Ax−b, is a measure of how accurate the solver was in
finding the true solution. If we solved it exactly, we would have r = 0. Usually
we talk about the residual as the norm of the residual vector, ||r||. Recall that
you take a norm in matlab using norm and in python using np.linalg.norm.
Alter your code so that you are computing the residual each time you solve
Ax = b for each vector b in the loop and for each method. Add up the total
residual for each method (this should be 3 numbers: the sum of the residuals
for the inverse method, the sum of the residuals for the LU method, and
the sum of the residuals for Gaussian Elimination). This should be just like
Homework 2 where you were asked to compute the sum of a number added to
itself several times.
(f) Record your results in a table: Solution: matlab:
Time for 100 vectors ||r||
Backslash 29.6212 4.3573e-12
LU 2.0380 3.0411e-12
inv 1.2978 1.6519e-10
python:
Time for 100 vectors ||r||
Backslash 27.24 6.17e-11
LU 1.01 5.07e-11
inv 1.36 2.44e-08
where ||r|| is the sum of the residual for each method.
(g) From the table, which of the methods is the fastest for this task (solving
Ax = b for multiple different vectors b)?
Solution: For matlab the answer is inverse, for python the answer is LU (by
just a hair).
(h) From the table, which of the methods has the most residual error?
Solution: For both methods, inverse has the highest residual error. This is
what we would expect, inverse is a bad way to solve Ax=b.
(i) Consider a task where you absolutely cannot get the wrong answer. The
wrong answer means life or death. But, you also want to do it as fast as you
can. Which of the three methods would you use? Why?
Solution: In either case I am choosing LU. I can’t choose inverse because it
has the most residual error. Error means people die. So, I’m left between a
direct solve (backslash or scipy.linalg.solve) or LU. LU is faster, so I’ll go
with LU.

Code for problem 2

MATLAB

2
%% Problem 1
clear; clc; close all;

N = 2000;
A = -1*diag(ones(N,1)) + 4*diag(ones(N-1,1),1) + 4*diag(ones(N-1,1),-1);

% Setup vectors for residual error to augment later

backslashresid = 0;
LUresid = 0;
invresid = 0;

numIters = 100;

% First do the backslash time

tic
for k=1:numIters
b = rand(N,1);
x = A\b;
backslashresid = backslashresid + norm(A*x-b);
end
backslashtime = toc;

% Then do the LU time

tic
[L,U,P] = lu(A); % Compute this outside the loop to save time
for k=1:numIters
b = rand(N,1);
x = L\(P*b);
x = U\x;
LUresid = LUresid + norm(A*x-b);
end
LUtime = toc;

% Finally do the inverse time

tic
Ainv = inv(A); % Compute this outside the loop to save time
for k=1:numIters
b = rand(N,1);
x = Ainv*b;
invresid = invresid + norm(A*x-b);
end
invtime = toc;

3
python

import numpy as np
import matplotlib.pyplot as plt
import time
import scipy
import scipy.linalg
import timeit

## Problem 1

N = 2000
offDiag = np.ones(N-1)
A = -1*np.diag(np.ones([N,1])) + 4*np.diag(offDiag,1) + 4*np.diag(offDiag,-1)

# Setup vectors for residual error to augment later

backslashresid = 0
LUresid = 0
invresid = 0

numIters = 100

# First do the backslash time

backtime = time.time()
for k in range(numIters):
b = np.random.rand(N,1)
x = scipy.linalg.solve(A,b)
backslashresid = backslashresid + np.linalg.norm(A@x-b)

backslashtime = time.time() - backtime

# Then do the LU time

LUtime = time.time()
# Compute outside the loop to save time. Use the faster version
(lu, piv) = scipy.linalg.lu_factor(A)
for k in range(numIters):
b = np.random.rand(N,1)
x = scipy.linalg.lu_solve( (lu,piv), b)
LUresid = LUresid + np.linalg.norm(A@x-b)

LUtime = time.time() - LUtime

# Finally do the inverse time

invTime = time.time()
Ainv = scipy.linalg.inv(A) # Compute this outside the loop to save time

4
 for k in range(numIters):
b = np.random.rand(N,1)
x = Ainv@b
invresid = invresid + np.linalg.norm(A@x-b)

invTime = time.time() - invTime

print([[backslashtime, backslashresid], [LUtime, LUresid], [invTime, invresid]])

# Print it all together in an easy to read table

2. In this problem we want to see how the time for solving Ax = b changes as the
size of the matrix changes. We will compare the times for solving matrices using
Gaussian elimination (backslash or scipy.linalg.solve), LU factorization, and
inverse.
You need to do all of the following parts, but you only need to include
your final plot and the discussion of the plot (part (j)) in your writeup.
(a) Create matrices A = rand(n) in matlab or A = np.random.rand(n,n) in
python, and vectors b = rand(n,1) in matlab or b = np.random.rand(n,1)
in python, for
n = 700, 1200, 2400, 6000, and 9000.
(b) Using the appropriate timing commands, save the time it takes to solve Ax = b
using backslash (matlab) or scipy.linalg.solve for each n. In other words,
create a vector of length 5 with the time for each solve.
(c) Using the appropriate timing commands, save the time it takes to solve Ax = b
using LU decomposition for each n.
(d) Using the appropriate timing commands, save the time it takes to solve Ax = b
using A−1 (inv(A)) for each n.
(e) On a log-log plot, plot the time it takes to solve Ax = b using the backslash
operator (matlab) or scipy.linalg.solve versus n. This plot should be
made with blue circles without lines between them.
(f) Using a log-log plot on the same axes as above, plot the time it takes to solve
Ax = b using LU decomposition versus n. This plot should be in red diamonds
without lines between them and on the same axes as the blue circles.
(g) Using a log-log plot on the same axes as above, plot the time it takes to solve
Ax = b using A−1 (inv(A)) versus n. This plot should be in green squares
without lines between them and on the same axes as the other plots.
(h) Add trendlines for O(n2 ) and O(n3 ) as I did in the lecture. These should be
different colors from the other plots and should be thick enough to easily see.
(i) Label your axes and include a legend!
(j) Include the plot in your writeup and comment on how the speed of each of
the three methods compare with each other. Your answer should include a
comparison between each of the three methods.

5
Solution: Consider the following plots.

101

100

Time
10 1

10 2
103 104
Size of matrix, n

The figure found using The figure found using python

matlab

I’ve added in lines to indicate O(n2 ) and O(n3 ) (in cyan and magenta respectively).
We see that the time to solve Ax = b using all three methods appears to be growing
at about the same rate: somewhere between O(n2 ) and O(n3 ). The inverse is
consistently slower than the other two methods. Not surprisingly, LU decomposition
and backslash take about the same amount of time.

Code for problem 2

MATLAB

%% Problem 2
close all; clear; clc;

rng(1); % This sets up the seed so everytime

% I run my code I use the same randon numbers

% Create vectors that we will fill in later

backslashtime = [];
LUtime = [];
invtime = [];

nVector = [700, 1200, 2400, 6000, 9000];

index = 1;
for n = nVector
A = rand(n); % Create the random matrix
b = rand(n,1); % Create the random vector

% First get the time it takes to do backslash

tic
x = A\b;
backslashtime(index) = toc;

6
 % Then find the time it takes to do LU
tic
[L,U, P] = lu(A); % You can do this or L U P, either way.
x = L\(P*b);
x = U\x;
LUtime(index) = toc;

% Finally, find the time it takes to do the inverse

tic
x = inv(A)*b;
invtime(index) = toc;
index = index+1
end

loglog(nVector, backslashtime, ’bo’)

hold on
loglog(nVector, LUtime, ’rd’)
loglog(nVector, invtime, ’gs’)

x = linspace(min(nVector),max(nVector),1000);
nSquared = 4.5*10^(-8)*x.^2; % I scaled these so that they would be
% near the data
nCubed = 0.8*10^(-10)*x.^3; % Scaled so they would be near the data

loglog(x, nSquared, ’c’, ’linewidth’,2)

loglog(x, nCubed, ’m’,’linewidth’, 2)
xlabel(’Size of matrix, n’)
ylabel(’Time’)

print(’Prob2’,’-dpng’) % Save the figure

python

## Problem 2

np.random.seed(1) # This sets up the seed so everytime

# I run my code I use the same random numbers

nVector = np.array([700, 1200, 2400, 6000, 9000])

# Create vectors that we will fill in later
backslashtime = np.zeros(nVector.shape)
LUtime = np.zeros(nVector.shape)
invtime = np.zeros(nVector.shape)
index = 0

7
for n in nVector:
A = np.random.rand(n,n) # Create the np.random.random matrix
b = np.random.rand(n,1) # Create the np.random.random vector

# First get the time it takes to do backslash

start = time.time()
#time.sleep(1) # I need to do this because it looks like a lot of my times ar
# which is a problem I will be aware of when grading. This ad
# so it is fair.
x = scipy.linalg.solve(A, b)
end = time.time()
backslashtime[index] = end - start

# Then find the time it takes to do LU

start = time.time()
(lu, piv) = scipy.linalg.lu_factor(A)
x = scipy.linalg.lu_solve( (lu,piv), b)
end = time.time()
LUtime[index] = end - start

# Finally, find the time it takes to do the scipy.linalg.inverse

start = time.time()
x = scipy.linalg.inv(A)@b
end = time.time()

invtime[index] = end - start

index = index+1 #Update the index

fig, ax = plt.subplots()
ax.loglog(nVector, backslashtime, ’bo’)
ax.loglog(nVector, LUtime, ’rd’)
ax.loglog(nVector, invtime, ’gs’)

x = np.linspace(min(nVector),max(nVector),1000)
nSquared = 4.5*10**(-8)*x**2 # I scaled these so that they would be
# near the data
nCubed = 0.8*10**(-10)*x**3 # Scaled so they would be near the data

ax.loglog(x, nSquared, ’c’, linewidth = 2)

ax.loglog(x, nCubed, ’m’, linewidth = 2)
ax.set_xlabel(’Size of matrix, n’)
ax.set_ylabel(’Time’)

fig.savefig(’Prob2_python.pdf’) # Save the figure

#plt.show()