Labs for

Foundations of Applied
Mathematics
Volume 1
Mathematical Analysis

Jeffrey Humpherys & Tyler J. Jarvis, managing editors

 List of Contributors

B. Barker T. Christensen
Brigham Young University Brigham Young University
E. Evans M. Cook
Brigham Young University Brigham Young University
R. Evans M. Cutler
Brigham Young University Brigham Young University
J. Grout R. Dorff
Drake University Brigham Young University
J. Humpherys B. Ehlert
Brigham Young University Brigham Young University
T. Jarvis M. Fabiano
Brigham Young University Brigham Young University
J. Whitehead K. Finlinson
Brigham Young University Brigham Young University
J. Adams J. Fisher
Brigham Young University Brigham Young University
K. Baldwin R. Flores
Brigham Young University Brigham Young University
J. Bejarano R. Fowers
Brigham Young University Brigham Young University
J. Bennett A. Frandsen
Brigham Young University Brigham Young University
A. Berry R. Fuhriman
Brigham Young University Brigham Young University
Z. Boyd T. Gledhill
Brigham Young University Brigham Young University
M. Brown S. Giddens
Brigham Young University Brigham Young University
A. Carr C. Gigena
Brigham Young University Brigham Young University
C. Carter M. Graham
Brigham Young University Brigham Young University
S. Carter F. Glines
Brigham Young University Brigham Young University

i
ii List of Contributors

C. Glover E. Manner
Brigham Young University Brigham Young University
M. Goodwin M. Matsushita
Brigham Young University Brigham Young University
R. Grout R. McMurray
Brigham Young University Brigham Young University
D. Grundvig S. McQuarrie
Brigham Young University Brigham Young University
S. Halverson E. Mercer
Brigham Young University Brigham Young University
E. Hannesson D. Miller
Brigham Young University Brigham Young University
K. Harmer J. Morrise
Brigham Young University Brigham Young University
J. Henderson M. Morrise
Brigham Young University Brigham Young University
J. Hendricks A. Morrow
Brigham Young University Brigham Young University
A. Henriksen R. Murray
Brigham Young University Brigham Young University
I. Henriksen J. Nelson
Brigham Young University Brigham Young University
B. Hepner C. Noorda
Brigham Young University Brigham Young University
C. Hettinger A. Oldroyd
Brigham Young University Brigham Young University
S. Horst A. Oveson
Brigham Young University Brigham Young University
R. Howell E. Parkinson
Brigham Young University Brigham Young University
E. Ibarra-Campos M. Probst
Brigham Young University Brigham Young University
K. Jacobson M. Proudfoot
Brigham Young University Brigham Young University
R. Jenkins D. Reber
Brigham Young University Brigham Young University
J. Larsen H. Ringer
Brigham Young University Brigham Young University
J. Leete C. Robertson
Brigham Young University Brigham Young University
Q. Leishman M. Russell
Brigham Young University Brigham Young University
J. Lytle R. Sandberg
Brigham Young University Brigham Young University
List of Contributors iii

C. Sawyer T. Thompson
Brigham Young University Brigham Young University
N. Sill B. Trendler
Brigham Young University Brigham Young University
D. Smith
M. Victors
Brigham Young University
Brigham Young University
J. Smith
Brigham Young University E. Walker
P. Smith Brigham Young University
Brigham Young University J. Webb
M. Stauffer Brigham Young University
Brigham Young University R. Webb
E. Steadman Brigham Young University
Brigham Young University
J. West
J. Stewart
Brigham Young University
Brigham Young University
S. Suggs R. Wonnacott
Brigham Young University Brigham Young University
A. Tate A. Zaitzeff
Brigham Young University Brigham Young University
iv List of Contributors
 Preface

This lab manual is designed to accompany the textbook Foundations of Applied Mathematics
Volume 1: Mathematical Analysis by Humpherys, Jarvis and Evans. The labs focus mainly on
important numerical linear algebra algorithms, with applications to images, networks, and data
science. The reader should be familiar with Python [VD10] and its NumPy [Oli06, ADH+ 01, Oli07]
and Matplotlib [Hun07] packages before attempting these labs. See the Python Essentials manual
for introductions to these topics.

©This work is licensed under the Creative Commons Attribution 3.0 United States License.
You may copy, distribute, and display this copyrighted work only if you give credit to Dr. J. Humpherys.
All derivative works must include an attribution to Dr. J. Humpherys as the owner of this work as
well as the web address to
https://github.com/Foundations-of-Applied-Mathematics/Labs
as the original source of this work.
To view a copy of the Creative Commons Attribution 3.0 License, visit
http://creativecommons.org/licenses/by/3.0/us/
or send a letter to Creative Commons, 171 Second Street, Suite 300, San Francisco, California, 94105,
USA.

v
vi Preface
 Contents

Preface v

I Labs 1

1 Introduction to GitHub 3

2 Linear Transformations 7

3 Linear Systems 19

4 The QR Decomposition 33

5 Least Squares and Computing Eigenvalues 45

6 Image Segmentation 59

7 The SVD and Image Compression 71

8 Facial Recognition 81

9 Differentiation 89

10 Newton’s Method 99

11 Conditioning and Stability 109

12 Monte Carlo Integration 121

13 Visualizing Complex-valued Functions 127

14 The PageRank Algorithm 135

15 Iterative Solvers 147

16 The Drazin Inverse 159

17 The Arnoldi Iteration 167

vii
viii Contents

18 GMRES 173

II Appendices 179

A NumPy Visual Guide 181

B Matplotlib Customization 185

Bibliography 201
Part I

Labs

1
1
Introduction to GitHub

Lab Objective: Git is a version control system that helps you manage changes to your code over
time. It allows you to keep track of different versions of your code, collaborate with others, and revert
changes if necessary. In ACME, Git will allow the Lab Assistants to see and grade your code. In
this mini-lab you will learn how to successfully save your code to a GitHub repository.

Before you begin this lab, you should have already gone through the Getting Started tutorials.
Specifically:

• The course materials should be downloaded and stored in an accessible place on your computer

• VSCode (or another code editor) should be installed and set up on your computer

• You should have created a GitHub account with repositories for Volume 1 and Volume 2

• Python should be installed on your computer

If you have missed any of these steps, stop here and refer back to the Getting Started pdf and
the accompanying tutorial videos.
Each week there will be an assigned lab in both Volume 1 and Volume 2 that will supplement
the material presented in class. These labs will involve editing, running, and saving code frequently.
To assist in this process, GitHub will be used to save your code and allow the instructors to grade
each week. The following example problem will help outline this process.

Problem 1. In the lab folder titled GitHubIntro, you will find the file github_intro.py.
Open github_intro.py with VSCode (or your favorite code editor). In the function labeled
prob1(), remove the line that says raise NotImplementedError() and replace it with return
"Student Name" putting your actual name in the quotation marks. Make sure to not change
the indentation, otherwise your code will not run correctly. Be sure to save the file after you’ve
finished editing.
Now, run the file using python. This can be done either in your code editor, or via the
terminal (instructions are given later in this tutorial). If everything worked correctly, you should
see your name appear in the console.

We will now demonstrate how to run your file in python and upload your changed file to GitHub.

3
4 Lab 1. Introduction to GitHub

To start, open a terminal window. The following coding examples will teach you how to navigate
to the GitHubIntro/ directory via the terminal. Instead of clicking icons to open folders, you will
need to use typed commands within your terminal to move from folder to folder. To see your current
location, type the command pwd which stands for "print working directory."
~$ pwd
/home/username

Use the command ls to list the contents of your current directory.

~$ ls
Desktop Downloads Public Videos
Documents Pictures

Use the command cd to change directory. For example, if I wanted to navigate into the
Documents folder, I would use the following command:
~$ cd Documents # Change to the Documents folder
~$ pwd # Check that the current directory changed
/home/username/Documents

If you wish to go back a directory, use cd followed by a space and two periods “..”
~$ cd ..
~$ pwd
/home/username

Using these commands, navigate to the GitHubIntro directory. Once you’re there, use pwd to
check that you’re in the right folder. It should look like the following example.
~$ pwd
/home/username/.../GitHubIntro

Now that you are in the correct directory, you can run the following command to see that your
code compiles correctly:
~$ python github_intro.py
Student Name

Now, load your changed file to GitHub using the following commands:

# Pull changes from your online repository to your local machine

~$ git pull origin master

# Tell git to track changes made to github_intro.py

~$ git add github_intro.py

# Commit saves the changes you made to your file, acting like a time stamp
~$ git commit -m "Finished GitHub lab"
 5

# Send your commits to the repository

~$ git push origin master

Now, open a browser and log in to your GitHub account. Open your Volume 1 repository and
open your github_intro.py file. If everything worked correctly, you should be able to see the edits
you made to your file. Additionally, you can click on Commits in the menu on the left and see your
most recent commit statements. Congratulations, you have completed your GitHub introduction!
In the future, you will submit your labs in a similar way. The process is to first navigate to the
correct directory within your terminal, then input the following commands:
~$ git pull origin master
~$ git add <changed files>
~$ git commit -m "<descriptive message>"
~$ git push origin master

Note: following this ordering will help you to avoid merge conflicts with GitHub!
6 Lab 1. Introduction to GitHub
 Linear

2 Transformations

Lab Objective: Linear transformations are the most basic and essential operators in vector space
theory. In this lab we visually explore how linear transformations alter points in the Cartesian plane.
We also empirically explore the computational cost of applying linear transformations via matrix
multiplication.

Linear Transformations
A linear transformation is a mapping between vector spaces that preserves addition and scalar
multiplication. More precisely, let V and W be vector spaces over a common field F. A map
L : V → W is a linear transformation from V into W if
L(ax1 + bx2 ) = aLx1 + bLx2
for all vectors x1 , x2 ∈ V and scalars a, b ∈ F.
Every linear transformation L from an m-dimensional vector space into an n-dimensional vector
space can be represented by an m × n matrix A, called the matrix representation of L. To apply L
to a vector x, left multiply by its matrix representation. This results in a new vector x′ , where each
component is some linear combination of the elements of x. For linear transformations from R2 to
R2 , this process has the form
      ′ 
a b x ax + by x
Ax = = = = x′ .
c d y cx + dy y′
Linear transformations can be interpreted geometrically. To demonstrate this, consider the
array of points H that collectively form a picture of a horse, stored in the file horse.npy. The
coordinate pairs xi are organized by column, so the array has two rows: one for x-coordinates, and
one for y-coordinates. Matrix multiplication on the left transforms each coordinate pair, resulting in
another matrix H ′ whose columns are the transformed coordinate pairs:
   
 
x1 x2 x3 . . .
AH = A = A  x1 x2 x3 . . .  =  Ax1 Ax2 Ax3 . . . 
y1 y2 y3 . . .
 
x′1 x′2 x′3
 
...
=  x′1 x′2 x′3 ...  = = H ′.
y1′ y2′ y3′ ...

7
8 Lab 2. Linear Transformations

To begin, use np.load() to extract the array from the npy file, then plot the unaltered points
as individual pixels. See Figure 2.1 for the result.

>>> import numpy as np

>>> from matplotlib import pyplot as plt

# Load the array from the .npy file.

>>> data = np.load("horse.npy")

# Plot the x row against the y row with black pixels.

>>> plt.plot(data[0], data[1], 'k,')

# Set the window limits to [-1, 1] by [-1, 1] and make the window square.
>>> plt.axis([-1,1,-1,1])
>>> plt.gca().set_aspect("equal")
>>> plt.show()

Types of Linear Transformations

Linear transformations from R2 into R2 can be classified in a few ways.
• Stretch: Stretches or compresses the vector along each axis. The matrix representation is
diagonal:
 
a 0
.
0 b
If a = b, the transformation is called a dilation. The stretch in Figure 2.1 uses a = 1
2 and b = 6
5
to compress the x-axis and stretch the y-axis.
• Shear: Slants the vector by a scalar factor horizontally or vertically (or both simultaneously).
The matrix representation is
 
1 a
.
b 1
Pure horizontal shears (b = 0) skew the x-coordinate of the vector while pure vertical shears
(a = 0) skew the y-coordinate. Figure 2.1 has a horizontal shear with a = 21 , b = 0.
• Reflection: Reflects the vector about a line that passes through the origin. The reflection
T
about the line spanned by the vector [a, b] has the matrix representation
 2
a − b2

1 2ab
.
a2 + b2 2ab b2 − a2
The reflection in Figure 2.1 reflects the image about the y-axis (a = 0, b = 1).
• Rotation: Rotates the vector around the origin. A counterclockwise rotation of θ radians has
the following matrix representation:
 
cos θ − sin θ
sin θ cos θ
A negative value of θ performs a clockwise rotation. Choosing θ = π
2 produces the rotation in
Figure 2.1.
 9

Original Stretch Shear

Reflection Rotation Composition

Figure 2.1: The points stored in horse.npy under various linear transformations.

Problem 1. Write a function for each type of linear transformation. Each function should
accept an array to transform and the scalars that define the transformation (a and b for stretch,
shear, and reflection, and θ for rotation). Construct the matrix representation, left multiply it
with the input array, and return a transformation of the data.
To test these functions, write a function to plot the original points in horse.npy together
with the transformed points in subplots for a side-by-side comparison. Compare your results
to Figure 2.1.

Compositions of Linear Transformations

Let V , W , and Z be finite-dimensional vector spaces. If L : V → W and K : W → Z are linear
transformations with matrix representations A and B, respectively, then the composition function
KL : V → Z is also a linear transformation, and its matrix representation is the matrix product BA.
For example, if S is a matrix representing a shear and R is a matrix representing a rotation,
then RS represents a shear followed by a rotation. In fact, any linear transformation L : R2 → R2
is a composition of the four transformations discussed above. Figure 2.1 displays the composition of
all four previous transformations, applied in order (stretch, shear, reflection, then rotation).
10 Lab 2. Linear Transformations

Affine Transformations
All linear transformations map the origin to itself. An affine transformation is a mapping between
vector spaces that preserves the relationships between points and lines, but that may not preserve
the origin. Every affine transformation T can be represented by a matrix A and a vector b. To apply
T to a vector x, calculate Ax + b. If b = 0 then the transformation is linear, and if A = I but b ̸= 0
then it is called a translation.
T
For example, if T is the translation with b = 34 , 21 , then applying T to an image will shift it


right by 43 and up by 12 . This translation is illustrated below.

Original Translation

Affine transformations include all compositions of stretches, shears, rotations, reflections, and
translations. For example, if S represents a shear and R a rotation, and if b is a vector, then RSx+b
shears, then rotates, then translates x.

Modeling Motion with Affine Transformations

Affine transformations can be used to model particle motion, such as a planet rotating around the
sun. Let the sun be the origin, the planet’s location at time t be given by the vector p(t), and suppose
the planet has angular velocity ω (a measure of how fast the planet goes around the sun). To find
the planet’s position at time t given the planet’s initial position p(0), rotate the vector p(0) around
the origin by tω radians. Thus if R(θ) is the matrix representation of the linear transformation that
rotates a vector around the origin by θ radians, then p(t) = R(tω)p(0).

p(t)

tω radians

Origin p(0)
 11

Composing the rotation with a translation shifts the center of rotation away from the origin,
yielding more complicated motion.

Problem 2. The moon orbits the earth while the earth orbits the sun. Assuming circular
orbits, we can compute the trajectories of both the earth and the moon using only linear and
affine transformations.
Assume an orientation where both the earth and moon travel counterclockwise, with the
sun at the origin. Let pe (t) and pm (t) be the positions of the earth and the moon at time t,
respectively, and let ωe and ωm be each celestial body’s angular velocity. For a particular time
t, we calculate pe (t) and pm (t) with the following steps.

1. Compute pe (t) by rotating the initial vector pe (0) counterclockwise about the origin by
tωe radians.

2. Calculate the position of the moon relative to the earth at time t by rotating the vector
pm (0) − pe (0) counterclockwise about the origin by tωm radians.

3. To compute pm (t), translate the vector resulting from the previous step by pe (t).

Write a function that accepts a final time T , initial positions xe and xm , and the angular
momenta ωe and ωm . Assuming initial positions pe (0) = (xe , 0) and pm (0) = (xm , 0), plot
pe (t) and pm (t) over the time interval t ∈ [0, T ].
Setting T = 3π2 , xe = 10, xm = 11, ωe = 1, and ωm = 13, your plot should resemble
the following figure (fix the aspect ratio with ax.set_aspect("equal")). Note that a more
celestially accurate figure would use xe = 400, xm = 401 (the interested reader should see
http://www.math.nus.edu.sg/aslaksen/teaching/convex.html).

10

5
Earth
10 Moon
10 5 0 5 10
12 Lab 2. Linear Transformations

Timing Matrix Operations

Linear transformations are easy to perform via matrix multiplication. However, performing matrix
multiplication with very large matrices can strain a machine’s time and memory constraints. For
the remainder of this lab we take an empirical approach in exploring how much time and memory
different matrix operations require.

Timing Code
Recall that the time module’s time() function measures the number of seconds since the Epoch.
To measure how long it takes for code to run, record the time just before and just after the code in
question, then subtract the first measurement from the second to get the number of seconds that have
passed. Additionally, in IPython, the quick command %timeit uses the timeit module to quickly
time a single line of code.

In [1]: import time

In [2]: def for_loop():

...: """Go through ten million iterations of nothing."""
...: for _ in range(int(1e7)):
...: pass

In [3]: def time_for_loop():

...: """Time for_loop() with time.time()."""
...: start = time.time() # Clock the starting time.
...: for_loop()
...: return time.time() - start # Return the elapsed time.

In [4]: time_for_loop()
0.24458789825439453

In [5]: %timeit for_loop()

248 ms +- 5.35 ms per loop (mean +- std. dev. of 7 runs, 1 loop each)

Timing an Algorithm
Most algorithms have at least one input that dictates the size of the problem to be solved. For
example, the following functions take in a single integer n and produce a random vector of length n
as a list or a random n × n matrix as a list of lists.

from random import random

def random_vector(n): # Equivalent to np.random.random(n).tolist()
"""Generate a random vector of length n as a list."""
return [random() for i in range(n)]

def random_matrix(n): # Equivalent to np.random.random((n,n)).tolist()

"""Generate a random nxn matrix as a list of lists."""
return [[random() for j in range(n)] for i in range(n)]
 13

Executing random_vector(n) calls random() n times, so doubling n should about double the
amount of time random_vector(n) takes to execute. By contrast, executing random_matrix(n) calls
random() n2 times (n times per row with n rows). Therefore doubling n will likely more than double
the amount of time random_matrix(n) takes to execute, especially if n is large.
To visualize this phenomenon, we time random_matrix() for n = 21 , 22 , . . . , 212 and plot n
against the execution time. The result is displayed below on the left.

>>> domain = 2**np.arange(1,13)

>>> times = []
>>> for n in domain:
... start = time.time()
... random_matrix(n)
... times.append(time.time() - start)
...
>>> plt.plot(domain, times, 'g.-', linewidth=2, markersize=15)
>>> plt.xlabel("n", fontsize=14)
>>> plt.ylabel("Seconds", fontsize=14)
>>> plt.show()

2.0 2.0

1.5 1.5
Seconds

Seconds

1.0 1.0
0.5 0.5
0.0 0.0
0 1000 2000 3000 4000 0 1000 2000 3000 4000
n n

The figure on the left shows that the execution time for random_matrix(n) increases quadrat-
ically in n. In fact, the blue dotted line in the figure on the right is the parabola y = an2 , which
fits nicely over the timed observations. Here a is a small constant, but it is much less significant
than the exponent on the n. To represent this algorithm’s growth, we ignore a altogether and write
random_matrix(n) ∼ n2 .

Note
An algorithm like random_matrix(n) whose execution time increases quadratically with n is
called O(n2 ), notated by random_matrix(n) ∈ O(n2 ). Big-oh notation is common for indicating
both the temporal complexity of an algorithm (how the execution time grows with n) and the
spatial complexity (how the memory usage grows with n).
14 Lab 2. Linear Transformations

Problem 3. Let A be an m × n matrix with entries aij , x be an n × 1 vector with entries xk ,

and B be an n × p matrix with entries bij . The matrix-vector product Ax = y is a new m × 1
vector and the matrix-matrix product AB = C is a new m × p matrix. The entries yi of y and
cij of C are determined by the following formulas:
n
X n
X
yi = aik xk cij = aik bkj
k=1 k=1

These formulas are implemented below without using NumPy arrays or operations.

def matrix_vector_product(A, x): # Equivalent to np.dot(A,x).tolist()

"""Compute the matrix-vector product Ax as a list."""
m, n = len(A), len(x)
return [sum([A[i][k] * x[k] for k in range(n)]) for i in range(m)]

def matrix_matrix_product(A, B): # Equivalent to np.dot(A,B).tolist()

"""Compute the matrix-matrix product AB as a list of lists."""
m, n, p = len(A), len(B), len(B[0])
return [[sum([A[i][k] * B[k][j] for k in range(n)])
for j in range(p) ]
for i in range(m) ]

Time each of these functions with increasingly large inputs. Generate the inputs A, x,
and B with random_matrix() and random_vector() (so each input will be n × n or n × 1).
Only time the multiplication functions, not the generating functions.
Report your findings in a single figure with two subplots: one with matrix-vector times,
and one with matrix-matrix times. Choose a domain for n so that your figure accurately
describes the growth, but avoid values of n that lead to execution times of more than 1 minute.
Your figure should resemble the following plots.

Matrix-Vector Multiplication Matrix-Matrix Multiplication

0.007 2.5
0.006
2.0
0.005
Seconds

Seconds

0.004 1.5
0.003 1.0
0.002
0.5
0.001
0.000 0.0
0 50 100 150 200 250 0 50 100 150 200 250
n n
 15

Logarithmic Plots
Though the two plots from Problem 3 look similar, the scales on the y-axes show that the actual
execution times differ greatly. To be compared correctly, the results need to be viewed differently.
A logarithmic plot uses a logarithmic scale—with values that increase exponentially, such as
101 , 102 , 103 , . . .—on one or both of its axes. The three kinds of log plots are listed below.

• log-lin: the x-axis uses a logarithmic scale but the y-axis uses a linear scale.
Use plt.semilogx() instead of plt.plot().

• lin-log: the x-axis is uses a linear scale but the y-axis uses a log scale.
Use plt.semilogy() instead of plt.plot().

• log-log: both the x and y-axis use a logarithmic scale.

Use plt.loglog() instead of plt.plot().

Since the domain n = 21 , 22 , . . . is a logarithmic scale and the execution times increase
quadratically, we visualize the results of the previous problem with a log-log plot. The default base
for the logarithmic scales on logarithmic plots in Matplotlib is 10. To change the base to 2 on each
axis, specify the keyword arguments base=2.
Suppose the domain of n values are stored in domain and the corresponding execution times
for matrix_vector_product() and matrix_matrix_product() are stored in vector_times and
matrix_times, respectively. Then the following code produces Figure 2.5.

>>> ax1 = plt.subplot(121) # Plot both curves on a regular lin-lin plot.

>>> ax1.plot(domain, vector_times, 'b.-', lw=2, ms=15, label="Matrix-Vector")
>>> ax1.plot(domain, matrix_times, 'g.-', lw=2, ms=15, label="Matrix-Matrix")
>>> ax1.legend(loc="upper left")

>>> ax2 = plot.subplot(122) # Plot both curves on a base 2 log-log plot.

>>> ax2.loglog(domain, vector_times, 'b.-', base=2, lw=2)
>>> ax2.loglog(domain, matrix_times, 'g.-', base=2, lw=2)

>>> plt.show()

2.5 22
Matrix-Vector
Matrix-Matrix 21
2.0
24
1.5 27
1.0 2 10
0.5 2 13
2 16
0.0
0 50 100 150 200 250 21 22 23 24 25 26 27 28

Figure 2.5
16 Lab 2. Linear Transformations

In the log-log plot, the slope of the matrix_matrix_product() line is about 3 and the slope of
the matrix_vector_product() line is about 2. This reflects the fact that matrix-matrix multipli-
cation (which uses 3 loops) is O(n3 ), while matrix-vector multiplication (which only has 2 loops) is
only O(n2 ).

Problem 4. NumPy is built specifically for fast numerical computations. Repeat the experi-
ment of Problem 3, timing the following operations:

• matrix-vector multiplication with matrix_vector_product().

• matrix-matrix multiplication with matrix_matrix_product().

• matrix-vector multiplication with np.dot() or @.

• matrix-matrix multiplication with np.dot() or @.

Create a single figure with two subplots: one with all four sets of execution times on a
regular linear scale, and one with all four sets of execution times on a log-log scale. Compare
your results to Figure 2.5.

Note
Problem 4 shows that matrix operations are significantly faster in NumPy than in
plain Python. Matrix-matrix multiplication grows cubically regardless of the implementation;
however, with lists the times grows at a rate of an3 while with NumPy the times grow at a rate
of bn3 , where a is much larger than b. NumPy is more efficient for several reasons:

1. Iterating through loops is very expensive. Many of NumPy’s operations are implemented
in C, which are much faster than Python loops.

2. Arrays are designed specifically for matrix operations, while Python lists are general
purpose.

3. NumPy carefully takes advantage of computer hardware, efficiently using different levels
of computer memory.

However, in Problem 4, the execution times for matrix multiplication with NumPy seem
to increase somewhat inconsistently. This is because the fastest layer of computer memory can
only handle so much information before the computer has to begin using a larger, slower layer
of memory.
 17

Additional Material
Image Transformation as a Class
Consider organizing the functions from Problem 1 into a class. The constructor might accept an
array or the name of a file containing an array. This structure would makes it easy to do several
linear or affine transformations in sequence.

>>> horse = ImageTransformer("horse.npy")

>>> horse.stretch(.5, 1.2)
>>> horse.shear(.5, 0)
>>> horse.relect(0, 1)
>>> horse.rotate(np.pi/2.)
>>> horse.translate(.75, .5)
>>> horse.display()

Animating Parametrizations
The plot in Problem 2 fails to fully convey the system’s evolution over time because time itself is not
part of the plot. The following function creates an animation for the earth and moon trajectories.

from matplotlib.animation import FuncAnimation

def solar_system_animation(earth, moon):

"""Animate the moon orbiting the earth and the earth orbiting the sun.
Parameters:
earth ((2,N) ndarray): The earth's postion with x-coordinates on the
first row and y coordinates on the second row.
moon ((2,N) ndarray): The moon's postion with x-coordinates on the
first row and y coordinates on the second row.
"""
fig, ax = plt.subplots(1,1) # Make a figure explicitly.
plt.axis([-15,15,-15,15]) # Set the window limits.
ax.set_aspect("equal") # Make the window square.
earth_dot, = ax.plot([],[], 'C0o', ms=10) # Blue dot for the earth.
earth_path, = ax.plot([],[], 'C0-') # Blue line for the earth.
moon_dot, = ax.plot([],[], 'C2o', ms=5) # Green dot for the moon.
moon_path, = ax.plot([],[], 'C2-') # Green line for the moon.
ax.plot([0],[0],'y*', ms=20) # Yellow star for the sun.

def animate(index):
earth_dot.set_data(earth[0,index], earth[1,index])
earth_path.set_data(earth[0,:index], earth[1,:index])
moon_dot.set_data(moon[0,index], moon[1,index])
moon_path.set_data(moon[0,:index], moon[1,:index])
return earth_dot, earth_path, moon_dot, moon_path,
a = FuncAnimation(fig, animate, frames=earth.shape[1], interval=25)
plt.show()
18 Lab 2. Linear Transformations
 Linear Systems

3
Lab Objective: The fundamental problem of linear algebra is solving the linear system Ax = b,
given that a solution exists. There are many approaches to solving this problem, each with different
pros and cons. In this lab we implement the LU decomposition and use it to solve square linear
systems. We also introduce SciPy, together with its libraries for linear algebra and working with
sparse matrices.

Gaussian Elimination
The standard approach for solving the linear system Ax = b on paper is reducing the augmented
matrix [A | b] to row-echelon form (REF) via Gaussian elimination, then using back substitution.
The matrix is in REF when the leading non-zero term in each row is the diagonal term, so the matrix
is upper triangular.
At each step of Gaussian elimination, there are three possible operations: swapping two rows,
multiplying one row by a scalar value, or adding a scalar multiple of one row to another. Many
systems, like the one displayed below, can be reduced to REF using only the third type of operation.
First, use multiples of the first row to get zeros below the diagonal in the first column, then use a
multiple of the second row to get zeros below the diagonal in the second column.
       
1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1
 1 4 2 3  −→  0 3 1 2  −→  0 3 1 2  −→  0 3 1 2 
4 7 8 9 4 7 8 9 0 3 4 5 0 0 3 3

Each of these operations is equivalent to left-multiplying by a type III elementary matrix, the
identity with a single non-zero non-diagonal term. If row operation k corresponds to matrix Ek , the
following equation is E3 E2 E1 A = U .
      
1 0 0 1 0 0 1 0 0 1 1 1 1 1 1 1 1
 0 1 0   0 1 0   −1 1 0   1 4 2 3  =  0 3 1 2 
0 −1 1 −4 0 1 0 0 1 4 7 8 9 0 0 3 3

However, matrix multiplication is an inefficient way to implement row reduction. Instead,

modify the matrix in place (without making a copy), changing only those entries that are affected
by each row operation.

19
20 Lab 3. Linear Systems

>>> import numpy as np

>>> A = np.array([[1, 1, 1, 1],

... [1, 4, 2, 3],
... [4, 7, 8, 9]], dtype=np.float)

# Reduce the 0th column to zeros below the diagonal.

>>> A[1,0:] -= (A[1,0] / A[0,0]) * A[0]
>>> A[2,0:] -= (A[2,0] / A[0,0]) * A[0]

# Reduce the 1st column to zeros below the diagonal.

>>> A[2,1:] -= (A[2,1] / A[1,1]) * A[1,1:]
>>> print(A)
[[ 1. 1. 1. 1.]
[ 0. 3. 1. 2.]
[ 0. 0. 3. 3.]]

Note that the final row operation modifies only part of the third row to avoid spending the
computation time of adding 0 to 0.
If a 0 appears on the main diagonal during any part of row reduction, the approach given above
tries to divide by 0. Swapping the current row with one below it that does not have a 0 in the same
column solves this problem. This is equivalent to left-multiplying by a type II elementary matrix,
also called a permutation matrix.

Achtung!
Gaussian elimination is not always numerically stable. In other words, it is susceptible to
rounding error that may result in an incorrect final matrix. Suppose that, due to roundoff
error, the matrix A has a very small entry on the diagonal.
 −15 
10 1
A=
−1 0

Though 10−15 is essentially zero, instead of swapping the first and second rows to put A in
REF, a computer might multiply the first row by 1015 and add it to the second row to eliminate
the −1. The resulting matrix is far from what it would be if the 10−15 were actually 0.
 −15   −15 
10 1 10 1
−→
−1 0 0 1015

Round-off error can propagate through many steps in a calculation. The NumPy routines
that employ row reduction use several tricks to minimize the impact of round-off error, but
these tricks cannot fix every matrix.
 21

Problem 1. Write a function that reduces an arbitrary square matrix A to REF. You may
assume that A is invertible and that a 0 will never appear on the main diagonal (so only use
type III row reductions, not type II). Avoid operating on entries that you know will be 0 before
and after a row operation. Use at most two nested loops.
Test your function with small test cases that you can check by hand. Consider using
np.random.randint() to generate a few manageable tests cases.

The LU Decomposition
The LU decomposition of a square matrix A is a factorization A = LU where U is the upper
triangular REF of A and L is the lower triangular product of the type III elementary matrices
whose inverses reduce A to U . The LU decomposition of A exists when A can be reduced to REF
using only type III elementary matrices (without any row swaps). However, the rows of A can always
be permuted in a way such that the decomposition exists. If P is a permutation matrix encoding the
appropriate row swaps, then the decomposition P A = LU always exists.
Suppose A has an LU decomposition (not requiring row swaps). Then A can be reduced
to REF with k row operations, corresponding to left-multiplying the type III elementary matrices
E1 , . . . , Ek . Because there were no row swaps, each Ei is lower triangular, so each inverse Ei−1 is also
lower triangular. Furthermore, since the product of lower triangular matrices is lower triangular, L
is lower triangular:

Ek . . . E2 E1 A = U −→ A = (Ek . . . E2 E1 )−1 U
= E1−1 E2−1 . . . Ek−1 U
= LU.

Thus, L can be computed by right-multiplying the identity by the matrices used to reduce U .
However, in this special situation, each right-multiplication only changes one entry of L, matrix mul-
tiplication can be avoided altogether. The entire process, only slightly different than row reduction,
is summarized below.

Algorithm 1
1: procedure LU Decomposition(A)
2: m, n ← shape(A) ▷ Store the dimensions of A.
3: U ← copy(A) ▷ Make a copy of A with np.copy().
4: L ← Im ▷ The m × m identity matrix.
5: for j = 0 . . . n − 1 do
6: for i = j + 1 . . . m − 1 do
7: Li,j ← Ui,j /Uj,j
8: Ui,j: ← Ui,j: − Li,j Uj,j:
9: return L, U

Problem 2. Write a function that finds the LU decomposition of a square matrix. You may
assume that the decomposition exists and requires no row swaps.
22 Lab 3. Linear Systems

Forward and Backward Substitution

If P A = LU and Ax = b, then LU x = P Ax = P b. This system can be solved by first solving
Ly = P b, then U x = y. Since L and U are both triangular, these systems can be solved with
backward and forward substitution. We can thus compute the LU factorization of A once, then use
substitution to efficiently solve Ax = b for various values of b.
Since the diagonal entries of L are all 1, the triangular system Ly = b has the form

0 ··· 0
    
1 0 y1 b1
 l21 1
 0 ··· 0    y 2   b2 
   
 l31 l32 1 · · · 0   y3   b3 
= .
 . .. .. .. ..   ..   .. 
 
 .. . . . .   .   . 
ln1 ln2 ln3 · · · 1 yn bn

Matrix multiplication yields the equations

y1 = b1 , y1 = b1 ,
l21 y1 + y2 = b2 , y2 = b2 − l21 y1 ,
.. ..
. .
k−1
X k−1
X
lkj yj + yk = bk , y k = bk − lkj yj . (3.1)
j=1 j=1

The triangular system U x = y yields similar equations, but in reverse order:

u11 u12 u13 · · · u1n

    
x1 y1
 0 u22 u23 · · · u2n   x2   y2 
    
 0 0 u33 · · · u3n    x3  =  y3  ,
   
 . . . .. . . .

 .. .. .. . ..   ..   .. 
    

0 0 0 · · · unn xn yn

1
unn xn = yn , xn = yn ,
unn
1
un−1,n−1 xn−1 + un−1,n xn = yn−1 , xn−1 = (yn−1 − un−1,n xn ) ,
un−1,n−1
.. ..
. .
 
n n
X 1  X
ukj xj = yk , xk = yk − ukj xj  . (3.2)
ukk
j=k j=k+1

Problem 3. Write a function that, given A and b, solves the square linear system Ax = b.
Use the function from Problem 2 to compute L and U , then use (3.1) and (3.2) to solve for y,
then x. You may again assume that no row swaps are required (P = I in this case).
 23

Unit Test
Write a unit test for Problem 3, your solve function. It can be found in the test_linear_systems
.py file and the unit test is named test_solve.
There are example unit tests for Problems 1 and 2 to help you structure your unit test.

SciPy
SciPy [JOP+ ] is a powerful scientific computing library built upon NumPy. It includes high-level
tools for linear algebra, statistics, signal processing, integration, optimization, machine learning, and
more.
SciPy is typically imported with the convention import scipy as sp. However, SciPy is set
up in a way that requires its submodules to be imported individually.1

>>> import scipy as sp

>>> hasattr(sp, "stats") # The stats module isn't loaded yet.
False

>>> from scipy import stats # Import stats explicitly. Access it

>>> hasattr(sp, "stats") # with 'stats' or 'sp.stats'.
True

Linear Algebra
NumPy and SciPy both have a linear algebra module, each called linalg, but SciPy’s module is the
larger of the two. Some of SciPy’s common linalg functions are listed below.

Function Returns
det() The determinant of a square matrix.
eig() The eigenvalues and eigenvectors of a square matrix.
inv() The inverse of an invertible matrix.
norm() The norm of a vector or matrix norm of a matrix.
solve() The solution to Ax = b (the system need not be square).

This library also includes routines for computing matrix decompositions.

>>> from scipy import linalg as la

# Make a random matrix and a random vector.

>>> A = np.random.random((1000,1000))
>>> b = np.random.random(1000)

# Compute the LU decomposition of A, including pivots.

>>> L, P = la.lu_factor(A)
1 SciPy modules like linalg are really packages, which are not initialized when SciPy is imported alone.
24 Lab 3. Linear Systems

# Use the LU decomposition to solve Ax = b.

>>> x = la.lu_solve((L,P), b)

# Check that the solution is legitimate.

>>> np.allclose(A @ x, b)
True

As with NumPy, SciPy’s routines are all highly optimized. However, some algorithms are, by
nature, faster than others.

Problem 4. Write a function that times different scipy.linalg functions for solving square
linear systems.
For various values of n, generate a random n × n matrix A and a random n-vector b using
np.random.random(). Time how long it takes to solve the system Ax = b with each of the
following approaches:

1. Invert A with la.inv() and left-multiply the inverse to b.

2. Use la.solve().

3. Use la.lu_factor() and la.lu_solve() to solve the system with the LU decomposition.

4. Use la.lu_factor() and la.lu_solve(), but only time la.lu_solve() (not the time
it takes to do the factorization with la.lu_factor()).

Plot the system size n versus the execution times. Use log scales if needed.

Achtung!
Problem 4 demonstrates that computing a matrix inverse is computationally expensive. In fact,
numerically inverting matrices is so costly that there is hardly ever a good reason to do it. Use
a specific solver like la.lu_solve() whenever possible instead of using la.inv().

Sparse Matrices

Large linear systems can have tens of thousands of entries. Storing the corresponding matrices in
memory can be difficult: a 105 × 105 system requires around 40 GB to store in a NumPy array (4
bytes per entry × 1010 entries). This is well beyond the amount of RAM in a normal laptop.
In applications where systems of this size arise, it is often the case that the system is sparse,
meaning that most of the entries of the matrix are 0. SciPy’s sparse module provides tools for
efficiently constructing and manipulating 1- and 2-D sparse matrices. A sparse matrix only stores
the nonzero values and the positions of these values. For sufficiently sparse matrices, storing the
matrix as a sparse matrix may only take megabytes, rather than gigabytes.
 25

For example, diagonal matrices are sparse. Storing an n × n diagonal matrix in the naïve way
means storing n2 values in memory. It is more efficient to instead store the diagonal entries in a
1-D array of n values. In addition to using less storage space, this allows for much faster matrix
operations: the standard algorithm to multiply a matrix by a diagonal matrix involves n3 steps, but
most of these are multiplying by or adding 0. A smarter algorithm can accomplish the same task
much faster.
SciPy has seven sparse matrix types. Each type is optimized either for storing sparse matrices
whose nonzero entries follow certain patterns, or for performing certain computations.

Name Description Advantages

bsr_matrix Block Sparse Row Specialized structure.
coo_matrix Coordinate Format Conversion among sparse formats.
csc_matrix Compressed Sparse Column Column-based operations and slicing.
csr_matrix Compressed Sparse Row Row-based operations and slicing.
dia_matrix Diagonal Storage Specialized structure.
dok_matrix Dictionary of Keys Element access, incremental construction.
lil_matrix Row-based Linked List Incremental construction.

Creating Sparse Matrices

A regular, non-sparse matrix is called full or dense. Full matrices can be converted to each of the
sparse matrix formats listed above. However, it is more memory efficient to never create the full
matrix in the first place. There are three main approaches for creating sparse matrices from scratch.

• Coordinate Format: When all of the nonzero values and their positions are known, create
the entire sparse matrix at once as a coo_matrix. All nonzero values are stored as a coordinate
and a value. This format also converts quickly to other sparse matrix types.

>>> from scipy import sparse

# Define the rows, columns, and values separately.

>>> rows = np.array([0, 1, 0])
>>> cols = np.array([0, 1, 1])
>>> vals = np.array([3, 5, 2])
>>> A = sparse.coo_matrix((vals, (rows,cols)), shape=(3,3))
>>> print(A)
(0, 0) 3
(1, 1) 5
(0, 1) 2

# The toarray() method casts the sparse matrix as a NumPy array.

>>> print(A.toarray()) # Note that this method forfeits
[[3 2 0] # all sparsity-related optimizations.
[0 5 0]
[0 0 0]]

• DOK and LIL Formats: If the matrix values and their locations are not known beforehand,
construct the matrix incrementally with a dok_matrix or a lil_matrix. Indicate the size of
the matrix, then change individual values with regular slicing syntax.
26 Lab 3. Linear Systems

>>> B = sparse.lil_matrix((2,6))
>>> B[0,2] = 4
>>> B[1,3:] = 9

>>> print(B.toarray())
[[ 0. 0. 4. 0. 0. 0.]
[ 0. 0. 0. 9. 9. 9.]]

• DIA Format: Use a dia_matrix to store matrices that have nonzero entries on only certain
diagonals. The function sparse.diags() is one convenient way to create a dia_matrix from
scratch. Additionally, every sparse matrix has a setdiags() method for modifying specified
diagonals.

# Use sparse.diags() to create a matrix with diagonal entries.

>>> diagonals = [[1,2],[3,4,5],[6]] # List the diagonal entries.
>>> offsets = [-1,0,3] # Specify the diagonal they go on.
>>> print(sparse.diags(diagonals, offsets, shape=(3,4)).toarray())
[[ 3. 0. 0. 6.]
[ 1. 4. 0. 0.]
[ 0. 2. 5. 0.]]

# If all of the diagonals have the same entry, specify the entry alone.
>>> A = sparse.diags([1,3,6], offsets, shape=(3,4))
>>> print(A.toarray())
[[ 3. 0. 0. 6.]
[ 1. 3. 0. 0.]
[ 0. 1. 3. 0.]]

# Modify a diagonal with the setdiag() method.

>>> A.setdiag([4,4,4], 0)
>>> print(A.toarray())
[[ 4. 0. 0. 6.]
[ 1. 4. 0. 0.]
[ 0. 1. 4. 0.]]

• BSR Format: Many sparse matrices can be formulated as block matrices, and a block matrix
can be stored efficiently as a bsr_matrix. Use sparse.bmat() or sparse.block_diag() to
create a block matrix quickly.

# Use sparse.bmat() to create a block matrix. Use 'None' for zero blocks.
>>> A = sparse.coo_matrix(np.ones((2,2)))
>>> B = sparse.coo_matrix(np.full((2,2), 2.))
>>> print(sparse.bmat([[ A , None, A ],
[None, B , None]], format='bsr').toarray())
[[ 1. 1. 0. 0. 1. 1.]
[ 1. 1. 0. 0. 1. 1.]
[ 0. 0. 2. 2. 0. 0.]
 27

[ 0. 0. 2. 2. 0. 0.]]

# Use sparse.block_diag() to construct a block diagonal matrix.

>>> print(sparse.block_diag((A,B)).toarray())
[[ 1. 1. 0. 0.]
[ 1. 1. 0. 0.]
[ 0. 0. 2. 2.]
[ 0. 0. 2. 2.]]

Note
If a sparse matrix is too large to fit in memory as an array, it can still be visualized with
Matplotlib’s plt.spy(), which colors in the locations of the non-zero entries of the matrix.

>>> from matplotlib import pyplot as plt

# Construct and show a matrix with 50 2x3 diagonal blocks.

>>> B = sparse.coo_matrix([[1,3,5],[7,9,11]])
>>> A = sparse.block_diag([B]*50)
>>> plt.spy(A, markersize=1)
>>> plt.show()

0 20 40 60 80 100 120 140

0

20

40

60

80
28 Lab 3. Linear Systems

Problem 5. Let I be the n × n identity matrix, and define

−4
   
B I 1
 I B I   1 −4 1 
.. .. .. ..
   
. . . .
   
A= I , B= 1 ,
.. .. .. ..
   
. . . .
   
 I   1 
I B 1 −4

where A is n2 × n2 and each block B is n × n. The large matrix A is used in finite difference
2 2
methods for solving Laplace’s equation in two dimensions, ∂∂xu2 + ∂∂yu2 = 0.
Write a function that accepts an integer n and constructs and returns A as a sparse matrix.
Use plt.spy() to check that your matrix has nonzero values in the correct places.

Sparse Matrix Operations

Once a sparse matrix has been constructed, it should be converted to a csr_matrix or a csc_matrix
with the matrix’s tocsr() or tocsc() method. The CSR and CSC formats are optimized for row or
column operations, respectively. To choose the correct format to use, determine what direction the
matrix will be traversed.
For example, in the matrix-matrix multiplication AB, A is traversed row-wise, but B is tra-
versed column-wise. Thus A should be converted to a csr_matrix and B should be converted to a
csc_matrix.

# Initialize a sparse matrix incrementally as a lil_matrix.

>>> A = sparse.lil_matrix((10000,10000))
>>> for k in range(10000):
... A[np.random.randint(0,9999), np.random.randint(0,9999)] = k
...
>>> A
<10000x10000 sparse matrix of type '<type 'numpy.float64'>'
with 9999 stored elements in LInked List format>

# Convert A to CSR and CSC formats to compute the matrix product AA.
>>> Acsr = A.tocsr()
>>> Acsc = A.tocsc()
>>> Acsr.dot(Acsc)
<10000x10000 sparse matrix of type '<type 'numpy.float64'>'
with 10142 stored elements in Compressed Sparse Row format>

Beware that row-based operations on a csc_matrix are very slow, and similarly, column-based
operations on a csr_matrix are very slow.

Achtung!
 29

Many familiar NumPy operations have analogous routines in the sparse module. These meth-
ods take advantage of the sparse structure of the matrices and are, therefore, usually significantly
faster. However, SciPy’s sparse matrices behave a little differently than NumPy arrays.

Operation numpy scipy.sparse

Component-wise Addition A + B A + B
Scalar Multiplication 2 * A 2 * A
Component-wise Multiplication A * B A.multiply(B)
Matrix Multiplication A.dot(B), A @ B A * B, A.dot(B), A @ B

Note in particular the difference between A * B for NumPy arrays and SciPy sparse
matrices. Do not use np.dot() to try to multiply sparse matrices, as it may treat the inputs
incorrectly. The syntax A.dot(B) is safest in most cases.

SciPy’s sparse module has its own linear algebra library, scipy.sparse.linalg, designed for
operating on sparse matrices. Like other SciPy modules, it must be imported explicitly.

>>> from scipy.sparse import linalg as spla

Problem 6. Write a function that times regular and sparse linear system solvers.
For various values of n, generate the n2 × n2 matrix A described in Problem 5 and a
random vector b with n2 entries. Time how long it takes to solve the system Ax = b with each
of the following approaches:

1. Convert A to CSR format and use scipy.sparse.linalg.spsolve() (spla.spsolve()).

2. Convert A to a NumPy array and use scipy.linalg.solve() (la.solve()).

In each experiment, only time how long it takes to solve the system (not how long it takes to
convert A to the appropriate format).
Plot the system size n2 versus the execution times. As always, use log scales where
appropriate and use a legend to label each line.

Achtung!
Even though there are fast algorithms for solving certain sparse linear system, it is still very
computationally difficult to invert sparse matrices. In fact, the inverse of a sparse matrix is
usually not sparse. There is rarely a good reason to invert a matrix, sparse or dense.

See http://docs.scipy.org/doc/scipy/reference/sparse.html for additional details on

SciPy’s sparse module.
30 Lab 3. Linear Systems

Additional Material
Improvements on the LU Decomposition
Vectorization

Algorithm 1 uses two loops to compute the LU decomposition. With a little vectorization, the process
can be reduced to a single loop.

Algorithm 2
1: procedure Fast LU Decomposition(A)
2: m, n ← shape(A)
3: U ← copy(A)
4: L ← Im
5: for k = 0 . . . n − 1 do
6: Lk+1:,k ← Uk+1:,k /Uk,k
T
7: Uk+1:,k: ← Uk+1:,k: − Lk+1:,k Uk,k:
8: return L, U

Note that step 7 is an outer product, not the regular dot product (xyT instead of the usual
x y). Use np.outer() instead of np.dot() or @ to get the desired result.
T

Pivoting

Gaussian elimination iterates through the rows of a matrix, using the diagonal entry xk,k of the
matrix at the kth iteration to zero out all of the entries in the column below xk,k (xi,k for i ≥ k).
This diagonal entry is called the pivot. Unfortunately, Gaussian elimination, and hence the LU
decomposition, can be very numerically unstable if at any step the pivot is a very small number.
Most professional row reduction algorithms avoid this problem via partial pivoting.
The idea is to choose the largest number (in magnitude) possible to be the pivot by swapping
the pivot row2 with another row before operating on the matrix. For example, the second and fourth
rows of the following matrix are exchanged so that the pivot is −6 instead of 2.
     
× × × × × × × × × × × ×
 0
 2 × ×   −→  0 −6 × ×  −→  0 −6 × × 
   
 0 4 × ×   0 4 × ×   0 0 × × 
0 −6 × × 0 2 × × 0 0 × ×

A row swap is equivalent to left-multiplying by a type II elementary matrix, also called a

permutation matrix.
    
1 0 0 0 × × × × × × × ×
 0 0 0 1  0
  2 × ×   0 −6
= × × 

 0 0 1 0  0 4 × ×   0 4 × × 
0 1 0 0 0 −6 × × 0 2 × ×

For the LU decomposition, if the permutation matrix at step k is Pk , then P = Pk . . . P2 P1

yields P A = LU . The complete algorithm is given below.
2 Complete pivoting involves row and column swaps, but doing both operations is usually considered overkill.
 31

Algorithm 3
1: procedure LU Decomposition with Partial Pivoting(A)
2: m, n ← shape(A)
3: U ← copy(A)
4: L ← Im
5: P ← [0, 1, . . . , n − 1] ▷ See tip 2 below.
6: for k = 0 . . . n − 1 do
7: Select i ≥ k that maximizes |Ui,k |
8: Uk,k: ↔ Ui,k: ▷ Swap the two rows.
9: Lk,:k ↔ Li,:k ▷ Swap the two rows.
10: Pk ↔ Pi ▷ Swap the two entries.
11: Lk+1:,k ← Uk+1:,k /Uk,k
T
12: Uk+1:,k: ← Uk+1:,k: − Lk+1:,k Uk,k:
13: return L, U, P

The following tips may be helpful for implementing this algorithm:

1. Since NumPy arrays are mutable, use np.copy() to reassign the rows of an array simultane-
ously.
2. Instead of storing P as an n × n array, fancy indexing allows us to encode row swaps in a 1-D
array of length n. Initialize P as the array [0, 1, . . . , n]. After performing a row swap on A,
perform the same operations on P . Then the matrix product P A will be the same as A[P].

>>> A = np.zeros(3) + np.vstack(np.arange(3))

>>> P = np.arange(3)
>>> print(A)
[[ 0. 0. 0.]
[ 1. 1. 1.]
[ 2. 2. 2.]]

# Swap rows 1 and 2.

>>> A[1], A[2] = np.copy(A[2]), np.copy(A[1])
>>> P[1], P[2] = P[2], P[1]
>>> print(A) # A with the new row arrangement.
[[ 0. 0. 0.]
[ 2. 2. 2.]
[ 1. 1. 1.]]
>>> print(P) # The permutation of the rows.
[0 2 1]
>>> print(A[P]) # A with the original row arrangement.
[[ 0. 0. 0.]
[ 1. 1. 1.]
[ 2. 2. 2.]]

There are potential cases where even partial pivoting does not eliminate catastrophic numerical
errors in Gaussian elimination, but the odds of having such an amazingly poor matrix are essentially
zero. The numerical analyst J.H. Wilkinson captured the likelihood of encountering such a matrix
in a natural application when he said, “Anyone that unlucky has already been run over by a bus!”
32 Lab 3. Linear Systems

In Place

The LU decomposition can be performed in place (overwriting the original matrix A) by storing U
on and above the main diagonal of the array and storing L below it. The main diagonal of L does
not need to be stored since all of its entries are 1. This format saves an entire array of memory, and
is how scipy.linalg.lu_factor() returns the factorization.

More Applications of the LU Decomposition

The LU decomposition can also be used to compute inverses and determinants with relative efficiency.

• Inverse: (P A)−1 = (LU )−1 =⇒ A−1 P −1 = U −1 L−1 =⇒ LU A−1 = P . Solve LU ai = pi

with forward and backward substitution (as in Problem 3) for every column pi of P . Then
 

A−1 =  a1 a2 ··· an  ,

the matrix where ak is the kth column.

• Determinant: det(A) = det(P −1 LU ) = det(L) det(U )
det(P ) . The determinant of a triangular matrix
is the product of its diagonal entries. Since every diagonal entry of L is 1, det(L) = 1. Also, P
is just a row permutation of the identity matrix (which has determinant 1), and a single row
swap negates the determinant. Then if S is the number of row swaps, the determinant is
n
Y
det(A) = (−1)S uii .
i=1

The Cholesky Decomposition

A square matrix A is called positive definite if zT Az > 0 for all nonzero vectors z. In addition, A is
called Hermitian if A = AH = AT . If A is Hermitian positive definite, it has a Cholesky Decomposition
A = U H U where U is upper triangular with real, positive entries on the diagonal. This is the matrix
equivalent to taking the square root of a positive real number.
The Cholesky decomposition takes advantage of the conjugate symmetry of A to simultaneously
reduce the columns and rows of A to zeros (except for the diagonal). It thus requires only half of the
calculations and memory of the LU decomposition. Furthermore, the algorithm is numerically stable,
which means, roughly speaking, that round-off errors do not propagate throughout the computation.

Algorithm 4
1: procedure Cholesky Decomposition(A)
2: n, n ← shape(A)
3: U ← np.triu(A) ▷ Get the upper-triangular part of A.
4: for i = 0 . . . n − 1 do
5: for j = i + 1 . . . n − 1 do
6: Uj,j: ← Uj,j: − Ui,j: Uij /Uii
√
7: Ui,i: ← Ui,i: / Uii
8: return U

As with the LU decomposition, SciPy’s linalg module has optimized routines,

la.cho_factor() and la.cho_solve(), for using the Cholesky decomposition.
4
The QR
Decomposition

Lab Objective: The QR decomposition is a fundamentally important matrix factorization. It

is straightforward to implement, is numerically stable, and provides the basis of several important
algorithms. In this lab we explore several ways to produce the QR decomposition and implement a
few immediate applications.

The QR decomposition of a matrix A is a factorization A = QR, where Q is has orthonormal

columns and R is upper triangular. Every m × n matrix A of rank n ≤ m has a QR decomposition,
with two main forms.

• Reduced QR: Q is m × n, R is n × n, and the columns {qj }nj=1 of Q form an orthonormal

basis for the column space of A.

• Full QR: Q is m × m and R is m × n. In this case, the columns {qj }m j=1 of Q form an
orthonormal basis for all of Fm , and the last m − n rows of R only contain zeros. If m = n,
this is the same as the reduced factorization.

We distinguish between these two forms by writing Q

b and R
b for the reduced decomposition and Q
and R for the full decomposition.

b (m × n)
Q b (n × n)
R
r11 · · · r1n
   
.. ..
. .
   
   
   
   rnn 
 = A (m × n)
 q1 ··· qn qn+1 ··· qm  
   0 ··· 0 
.. ..
   
   
   . . 
0 ··· 0
Q (m × m) R (m × n)

QR via Gram-Schmidt
The classical Gram-Schmidt algorithm takes a linearly independent set of vectors and constructs an
orthonormal set of vectors with the same span. Applying Gram-Schmidt to the columns of A, which
are linearly independent since A has rank n, results in the columns of Q.

33
34 Lab 4. The QR Decomposition

Let {xj }nj=1 be the columns of A. Define

x1 xk − pk−1
q1 = , qk = , k = 2, . . . , n,
∥x1 ∥ ∥xk − pk−1 ∥

k−1
X
p0 = 0, pk−1 = ⟨qj , xk ⟩qj , k = 2, . . . , n.
j=1

Each pk−1 is the projection of xk onto the span of {qj }k−1

j=1 , so qk = xk − pk−1 is the residual
′

vector of the projection. Thus qk is orthogonal to each of the vectors in {qj }k−1
′
j=1 . Therefore,
normalizing each q′k produces an orthonormal set {qj }nj=1 .
To construct the reduced QR decomposition, let Q b be the matrix with columns {qj }n , and
j=1
let R
b be the upper triangular matrix with entries

rkk = ∥xk − pk−1 ∥, rjk = ⟨qj , xk ⟩ = qT

j xk , j < k.

This clever choice of entries for R

b reverses the Gram-Schmidt process and ensures that Q
bRb = A.

Modified Gram-Schmidt
If the columns of A are close to being linearly dependent, the classical Gram-Schmidt algorithm
often produces a set of vectors {qj }nj=1 that are not even close to orthonormal due to rounding
errors. The modified Gram-Schmidt algorithm is a slight variant of the classical algorithm which
more consistently produces a set of vectors that are “very close” to orthonormal.
Let q1 be the normalization of x1 as before. Instead of making just x2 orthogonal to q1 , make
each of the vectors {xj }nj=2 orthogonal to q1 :
xk = xk − ⟨q1 , xk ⟩q1 , k = 2, . . . , n.
Next, define q2 = ∥x2 ∥ .
x2
Proceed by making each of {xj }nj=3 orthogonal to q2 :

xk = xk − ⟨q2 , xk ⟩q2 , k = 3, . . . , n.
Since each of these new vectors is a linear combination of vectors orthogonal to q1 , they are orthogonal
to q1 as well. Continuing this process results in the desired orthonormal set {qj }nj=1 . The entire
modified Gram-Schmidt algorithm is described below.

Algorithm 1
1: procedure Modified Gram-Schmidt(A)
2: m, n ← shape(A) ▷ Store the dimensions of A.
3: Q ← copy(A) ▷ Make a copy of A with np.copy().
4: R ← zeros(n, n) ▷ An n × n array of all zeros.
5: for i = 0 . . . n − 1 do
6: Ri,i ← ∥Q:,i ∥
7: Q:,i ← Q:,i /Ri,i ▷ Normalize the ith column of Q.
8: for j = i + 1 . . . n − 1 do
9: Ri,j ← QT :,j Q:,i
10: Q:,j ← Q:,j − Ri,j Q:,i ▷ Orthogonalize the jth column of Q.
11: return Q, R
 35

Problem 1. Write a function that accepts an m × n matrix A of rank n. Use Algorithm 1 to

compute the reduced QR decomposition of A.
Consider the following tips for implementing the algorithm.

• Use scipy.linalg.norm() to compute the norm of the vector in step 6.

• Note that steps 7 and 10 employ scalar multiplication or division, while step 9 uses vector
multiplication.

To test your function, generate test cases with NumPy’s np.random module. Verify that
R is upper triangular, Q is orthonormal, and QR = A. You may also want to compare your
results to SciPy’s QR factorization routine, scipy.linalg.qr().

>>> import numpy as np

>>> from scipy import linalg as la

# Generate a random matrix and get its reduced QR decomposition via SciPy.
>>> A = np.random.random((6,4))
>>> Q,R = la.qr(A, mode="economic") # Use mode="economic" for reduced QR.
>>> print(A.shape, Q.shape, R.shape)
(6,4) (6,4) (4,4)

# Verify that R is upper triangular, Q is orthonormal, and QR = A.

>>> np.allclose(np.triu(R), R)
True
>>> np.allclose(Q.T @ Q, np.identity(4))
True
>>> np.allclose(Q @ R, A)
True

Consequences of the QR Decomposition

The special structures of Q and R immediately provide some simple applications.

Determinants
Let A be n × n. Then Q and R are both n × n as well.1 Since Q is orthonormal and R is upper-
triangular,
Yn
det(Q) = ±1 and det(R) = ri,i .
i=1

Then since det(AB) = det(A) det(B),

n
Y
|det(A)| = |det(QR)| = |det(Q) det(R)| = |det(Q)| |det(R)| = ri,i . (4.1)
i=1

1 An n × n orthonormal matrix is sometimes called unitary in other texts.

36 Lab 4. The QR Decomposition

Problem 2. Write a function that accepts an invertible matrix A. Use the QR decomposition
of A and (4.1) to calculate |det(A)|. You may use your QR decomposition algorithm from
Problem 1 or SciPy’s QR routine. Can you implement this function in a single line?
(Hint: np.diag() and np.prod() may be useful.)
Check your answer against la.det(), which calculates the determinant.

Linear Systems
The LU decomposition is usually the matrix factorization of choice to solve the linear system Ax = b
because the triangular structures of L and U facilitate forward and backward substitution. However,
the QR decomposition avoids the potential numerical issues that come with Gaussian elimination.
Since Q is orthonormal, Q−1 = QT . Therefore, solving Ax = b is equivalent to solving the
system Rx = QT b. Since R is upper-triangular, Rx = QT b can be solved quickly with back
substitution.2

Problem 3. Write a function that accepts an invertible n × n matrix A and a vector b of

length n. Use the QR decomposition to solve Ax = b in the following steps:

1. Compute Q and R.

2. Calculate y = QT b.

3. Use back substitution to solve Rx = y for x.

QR via Householder
The Gram-Schmidt algorithm orthonormalizes A using a series of transformations that are stored
in an upper triangular matrix. Another way to compute the QR decomposition is to take the
opposite approach: triangularize A through a series of orthonormal transformations. Orthonormal
transformations are numerically stable, meaning that they are less susceptible to rounding errors. In
fact, this approach is usually faster and more accurate than Gram-Schmidt methods.
The idea is for the kth orthonormal transformation Qk to map the kth column of A to the span
of {ej }kj=1 , where the ej are the standard basis vectors in Rm . In addition, to preserve the work of
the previous transformations, Qk should not modify any entries of A that are above or to the left of
the kth diagonal term of A. For a 4 × 3 matrix A, the process can be visualized as follows.
       
∗ ∗ ∗ ∗ ∗ ∗ ∗ ∗ ∗ ∗ ∗ ∗
 ∗ ∗ ∗  = Q3 Q2  0 ∗ ∗  = Q3  0 ∗ ∗  =  0 ∗ ∗ 
      
Q3 Q2 Q1  ∗ ∗ ∗   0 ∗ ∗   0 0 ∗   0 0 ∗ 
∗ ∗ ∗ 0 ∗ ∗ 0 0 ∗ 0 0 0

Thus Q3 Q2 Q1 A = R, so that A = QT 1 Q2 Q3 R since each Qk is orthonormal. Furthermore, the

T T

product of square orthonormal matrices is orthonormal, so setting Q = QT 1 Q2 Q3 yields the full QR

T T

decomposition.
How to correctly construct each Qk isn’t immediately obvious. The ingenious solution lies in
one of the basic types of linear transformations: reflections.
2 See the Linear Systems lab for details on back substitution.
 37

Householder Transformations
The orthogonal complement of a nonzero vector v ∈ Rn is the set of all vectors x ∈ Rn that are
orthogonal to v, denoted v⊥ = {x ∈ Rn | ⟨x, v⟩ = 0}. A Householder transformation is a linear
transformation that reflects a vector x across the orthogonal complement v⊥ for some specified v.
The matrix representation of the Householder transformation corresponding to v is given by
T
Hv = I − 2 vvvT v . Since HvT Hv = I, Householder transformations are orthonormal.

Hv x

v⊥ v

Figure 4.1: The vector v defines the orthogonal complement v⊥ , which in this case is a line. Applying
the Householder transformation Hv to x reflects x across v⊥ .

Householder Triangularization
The Householder algorithm uses Householder transformations for the orthonormal transformations
in the QR decomposition process described on the previous page. The goal in choosing Qk is to send
xk , the kth column of A, to the span of {ej }kj=1 . In other words, if Qk xk = yk , the last m − k entries
of yk should be 0, i.e.,

z1 y1
   
..   .. 
.   . 


   
 zk   yk 
Qk xk = Qk 
 zk+1
=
  0  = yk .

 .   . 
   
 ..   .. 
zm 0

To begin, decompose xk into xk = x′k + x′′k , where x′k and x′′k are of the form

T T
x′k = [z1 ··· zk−1 0 ··· 0] , x′′k = [0 ··· 0 zk ··· zm ] .

Because x′k represents elements of A that lie above the diagonal, only x′′k needs to be altered by the
reflection.
The two vectors x′′k ± ∥x′′k ∥ek both yield Householder transformations that send x′′k to the
span of ek (see Figure 4.2). Between the two, the one that reflects x′′k further is more numerically
stable. This reflection corresponds to

vk = x′′k + sign(zk )∥x′′k ∥ek ,

where zk is the first nonzero component of x′′k (the kth component of xk ).

38 Lab 4. The QR Decomposition

v1 x
v2
Hv2 x
Hv1 x

Figure 4.2: There are two reflections that map x into the span of e1 , defined by the vectors v1 and
v2 . In this illustration, Hv2 is the more stable transformation since it reflects x further than Hv1 .

v vT
After choosing vk , set uk = ∥vvkk ∥ . Then Hvk = I − 2 ∥vkk ∥k2 = I − 2uk uT
k , and hence Qk is given
by the block matrix
   
Ik−1 0 Ik−1 0
Qk = = .
0 Hvk 0 Im−k+1 − 2uk uT k

Here Ip denotes the p × p identity matrix, and thus each Qk is m × m.

It is apparent from its form that Qk does not affect the first k − 1 rows and columns of any
matrix that it acts on. Then by starting with R = A and Q = I, at each step of the algorithm we
need only multiply the entries in the lower right (m − k + 1) × (m − k + 1) submatrices of R and Q
by I − 2uk uTk . This completes the Householder algorithm, detailed below.

Algorithm 2
1: procedure Householder(A)
2: m, n ← shape(A)
3: R ← copy(A)
4: Q ← Im ▷ The m × m identity matrix.
5: for k = 0 . . . n − 1 do
6: u ← copy(Rk:,k )
7: u0 ← u0 + sign(u0 )∥u∥ ▷ u0 is the first entry of u.
8: u ← u/∥u∥ ▷ Normalize u.
▷ Apply the reflection to R.


9: Rk:,k: ← Rk:,k: − 2u uT Rk:,k:
▷ Apply the reflection to Q.


10: Qk:,: ← Qk:,: − 2u uT Qk:,:
11: return QT , R

Problem 4. Write a function that accepts as input a m×n matrix A of rank n. Use Algorithm
2 to compute the full QR decomposition of A.
Consider the following implementation details.

• NumPy’s np.sign() is an easy way to implement the sign() operation in step 7. However,
np.sign(0) returns 0, which will cause a problem in the rare case that u0 = 0 (which is
possible if the top left entry of A is 0 to begin with). The following code defines a function
that returns the sign of a single number, counting 0 as positive.
 39

sign = lambda x: 1 if x >= 0 else -1

• In steps 9 and 10, the multiplication of u and (uT X) is an outer product (xyT instead of
the usual xT y). Use np.outer() instead of np.dot() to handle this correctly.

Use NumPy and SciPy to generate test cases and validate your function.

>>> A = np.random.random((5, 3))

>>> Q,R = la.qr(A) # Get the full QR decomposition.
>>> print(A.shape, Q.shape, R.shape)
(5,3) (5,5) (5,3)
>>> np.allclose(Q @ R, A)
True

Upper Hessenberg Form

An upper Hessenberg matrix is a square matrix that is nearly upper triangular, with zeros below
the first subdiagonal. Every n × n matrix A can be written A = QHQT where Q is orthonormal
and H, called the Hessenberg form of A, is an upper Hessenberg matrix. Putting a matrix in upper
Hessenberg form is an important first step to computing its eigenvalues numerically.
This algorithm also uses Householder transformations. To find orthogonal Q and upper Hes-
senberg H such that A = QHQT , it suffices to find such matrices that satisfy QT AQ = H. Thus,
the strategy is to multiply A on the left and right by a series of orthonormal matrices until it is in
Hessenberg form.
Using the same Qk as in the kth step of the Householder algorithm introduces n − k zeros in
the kth column of A, but multiplying Qk A on the right by QT k destroys all of those zeros. Instead,
choose a Q1 that fixes e1 and reflects the first column of A into the span of e1 and e2 . The product
Q1 A then leaves the first row of A alone, and the product (Q1 A)QT 1 leaves the first column of (Q1 A)
alone.
∗ ∗ ∗ ∗ ∗ ∗ ∗ ∗ ∗ ∗ ∗ ∗ ∗ ∗ ∗
     
 ∗ ∗ ∗ ∗ ∗   ∗ ∗ ∗ ∗ ∗   ∗ ∗ ∗ ∗ ∗ 
     
 ∗ ∗ ∗ ∗ ∗  Q1  0 ∗ ∗ ∗ ∗  QT 1
 0 ∗ ∗ ∗ ∗ 
  − →   −→  
 ∗ ∗ ∗ ∗ ∗   0 ∗ ∗ ∗ ∗   0 ∗ ∗ ∗ ∗ 
∗ ∗ ∗ ∗ ∗ 0 ∗ ∗ ∗ ∗ 0 ∗ ∗ ∗ ∗
A Q1 A (Q1 A)QT 1

Continuing the process results in the upper Hessenberg form of A.

∗ ∗ ∗ ∗ ∗
 
 ∗ ∗ ∗ ∗ ∗ 
Q3 Q2 Q1 AQT T T
 
Q Q
1 2 3 = 
 0 ∗ ∗ ∗ ∗ 

 0 0 ∗ ∗ ∗ 
0 0 0 ∗ ∗

This implies that A = QT

1 Q2 Q3 HQ3 Q2 Q1 , so setting Q = Q1 Q2 Q3 results in the desired
T T T T T

factorization A = QHQ . T
40 Lab 4. The QR Decomposition

Constructing the Reflections

Constructing the Qk uses the same approach as in the Householder algorithm, but shifted down one
element. Let xk = yk′ + yk′′ where yk′ and yk′′ are of the form
T T
yk′ = [z1 ··· zk 0 ··· 0] , yk′′ = [0 ··· 0 zk+1 ··· zm ] .

Because yk′ represents elements of A that lie above the first subdiagonal, only yk′′ needs to be altered.
This suggests using the reflection
   
Ik 0 Ik 0
Qk = = , where
0 Hvk 0 Im−k − 2uk uT k
vk
vk = yk′′ + sign(zk )∥yk′′ ∥ek , uk = .
∥vk ∥

The complete algorithm is given below. Note how similar it is to Algorithm 2.

Algorithm 3
1: procedure Hessenberg(A)
2: m, n ← shape(A)
3: H ← copy(A)
4: Q ← Im
5: for k = 0 . . . n − 3 do
6: u ← copy(Hk+1:,k )
7: u0 ← u0 + sign(u0 )∥u∥
8: u ← u/∥u∥
9: Hk+1:,k: ← Hk+1:,k: − 2u(uT Hk+1:,k: ) ▷ Apply Qk to H.
10: H:,k+1: ← H:,k+1: − 2(H:,k+1: u)uT ▷ Apply QT
k to H.
11: Qk+1:,: ← Qk+1:,: − 2u(uT Qk+1:,: ) ▷ Apply Qk to Q.
12: return H, QT

Problem 5. Write a function that accepts a nonsingular n × n matrix A. Use Algorithm 3 to

compute the upper Hessenberg H and orthogonal Q satisfying A = QHQT .
Compare your results to scipy.linalg.hessenberg().

# Generate a random matrix and get its upper Hessenberg form via SciPy.
>>> A = np.random.random((8,8))
>>> H, Q = la.hessenberg(A, calc_q=True)

# Verify that H has all zeros below the first subdiagonal and QHQ^T = A.
>>> np.allclose(np.triu(H, -1), H)
True
>>> np.allclose(Q @ H @ Q.T, A)
True
 41

Additional Material
Complex QR Decomposition
The QR decomposition also exists for matrices with complex entries. The standard inner product in
Rm is ⟨x, y⟩ = xT y, but the (more general) standard inner product in Cm is ⟨x, y⟩ = xH y. The H
stands for the Hermitian conjugate, the conjugate of the transpose. Making a few small adjustments
in the implementations of Algorithms 1 and 2 accounts for using the complex inner product.

1. Replace any transpose operations with the conjugate of the transpose.

>>> A = np.reshape(np.arange(4) + 1j*np.arange(4), (2,2))

>>> print(A)
[[ 0.+0.j 1.+1.j]
[ 2.+2.j 3.+3.j]]

>>> print(A.T) # Regular transpose.

[[ 0.+0.j 2.+2.j]
[ 1.+1.j 3.+3.j]]

>>> print(A.conj().T) # Hermitian conjugate.

[[ 0.-0.j 2.-2.j]
[ 1.-1.j 3.-3.j]]

2. Conjugate the first entry of vector or matrix multiplication before multiplying with np.dot().

>>> x = np.arange(2) + 1j*np.arange(2)

>>> print(x)
[ 0.+0.j 1.+1.j]

>>> np.dot(x, x) # Standard real inner product.

2j

>>> np.dot(x.conj(), y) # Standard complex inner product.

(2 + 0j)

3. In the complex plane, there are infinitely many reflections that map a vector x into the span
of ek , not just the two displayed in Figure 4.2. Using sign(zk ) to choose one is still a valid
method, but it requires updating the sign() function so that it can handle complex numbers.

sign = lambda x: x/np.abs(x) if x!=0 else 1

QR with Pivoting
The LU decomposition can be improved by employing Gaussian elimination with partial pivoting,
where the rows of A are strategically permuted at each iteration. The QR factorization can be
similarly improved by permuting the columns of A at each iteration. The result is the factorization
AP = QR, where P is a permutation matrix that encodes the column swaps. To compute the pivoted
QR decomposition with scipy.linalg.qr(), set the keyword pivoting to True.
42 Lab 4. The QR Decomposition

# Get the decomposition AP = QR for a random matrix A.

>>> A = np.random.random((8,10))
>>> Q,R,P = la.qr(A, pivoting=True)

# P is returned as a 1-D array that encodes column ordering,

# so A can be reconstructed with fancy indexing.
>>> np.allclose(Q @ R, A[:,P])
True

QR via Givens
The Householder algorithm uses reflections to triangularize A. However, A can also be made upper
triangular using rotations. To illustrate the idea, recall that the matrix for a counterclockwise rotation
of θ radians is given by
 
cos θ − sin θ
Rθ = .
sin θ cos θ

T
This transformation is orthonormal. Given x = [a, b] , if θ is the angle between x and e1 , then
R−θ maps x to the span of e1 .

θ
a
T
Figure 4.3: Rotating clockwise by θ sends the vector [a, b] to the span of e1 .

In terms of a and b, cos θ = √ a

a2 +b2
and sin θ = √ b
a2 +b2
. Therefore,
 
√ a √ b  √
a2 +b2 a2 +b2
     
cos θ sin θ a a a2 + b2
R−θ x = = = .
 
− sin θ cos θ b b 0

− √a2b+b2 √ a
a2 +b2

The matrix Rθ above is an example of a 2 × 2 Givens rotation matrix. In general, the Givens
matrix G(i, j, θ) represents the orthonormal transformation that rotates the 2-dimensional span of
ei and ej by θ radians. The matrix representation of this transformation is a generalization of Rθ .
 
I 0 0 0 0

 0 c 0 −s 0 

G(i, j, θ) = 
 0 0 I 0 0 

 0 s 0 c 0 
0 0 0 0 I

Here I represents the identity matrix, c = cos θ, and s = sin θ. The c’s appear on the ith and
jth diagonal entries.
 43

Givens Triangularization

As demonstrated, θ can be chosen such that G(i, j, θ) rotates a vector so that its jth-component is
0. Such a transformation will only affect the ith and jth entries of any vector it acts on (and thus
the ith and jth rows of any matrix it acts on).

To compute the QR decomposition of A, iterate through the subdiagonal entries of A in the

order depicted by Figure 4.4. Zero out the ijth entry with a rotation in the plane spanned by ei−1
and ei , represented by the Givens matrix G(i − 1, i, θ).

1 2 3 4 5
Figure 4.4: The order in which to zero out subdiagonal entries in the Givens triangularization
algorithm. The heavy black line is the main diagonal of the matrix. Entries should be zeroed out
from bottom to top in each column, beginning with the leftmost column.

On a 2 × 3 matrix, the process can be visualized as follows.

       
∗ ∗ ∗ ∗ ∗ ∗ ∗ ∗
 ∗ ∗  G(2, 3, θ1 )  ∗ ∗  G(1, 2, θ2 )  0 ∗  G(2, 3, θ3 )  0 ∗ 
−−−−−−−→ −−−−−−−→ −−−−−−−→
∗ ∗ 0 ∗ 0 ∗ 0 0

At each stage, the boxed entries are those modified by the previous transformation. The final
transformation G(2, 3, θ3 ) operates on the bottom two rows, but since the first two entries are zero,
they are unaffected.

Assuming that at the ijth stage of the algorithm aij is nonzero, Algorithm 4 computes the
Givens triangularization of a matrix. Notice that the algorithm does not actually form the entire
matrices G(i, j, θ); instead, it modifies only those entries of the matrix that are affected by the
transformation.
44 Lab 4. The QR Decomposition

Algorithm 4
1: procedure Givens Triangularization(A)
2: m, n ← shape(A)
3: R ← copy(A)
4: Q ← Im
5: for j = 0 . . . n − 1 do
6: for i = m − 1 . . . j + 1 do
7: a, b ← Ri−1,j , Ri,j √
8: G ← [[a, b], [−b, a]]/ a2 + b2
9: Ri−1:i+1,j: ← GRi−1:i+1,j:
10: Qi−1:i+1,: ← GQi−1:i+1,:
11: return QT , R

QR of a Hessenberg Matrix via Givens

The Givens algorithm is particularly efficient for computing the QR decomposition of a matrix that is
already in upper Hessenberg form, since only the first subdiagonal needs to be zeroed out. Algorithm
5 details this process.

Algorithm 5
1: procedure Givens Triangularization of Hessenberg(H)
2: m, n ← shape(H)
3: R ← copy(H)
4: Q ← Im
5: for j = 0 . . . min{n − 1, m − 1} do
6: i=j+1
7: a, b ← Ri−1,j , Ri,j √
8: G ← [[a, b], [−b, a]]/ a2 + b2
9: Ri−1:i+1,j: ← GRi−1:i+1,j:
10: Qi−1:i+1,:i+1 ← GQi−1:i+1,:i+1
11: return QT , R

Note
When A is symmetric, its upper Hessenberg form is a tridiagonal matrix, meaning its only
nonzero entries are on the main diagonal, the first subdiagonal, and the first superdiagonal.
This is because the Qk ’s zero out everything below the first subdiagonal of A and the QT
k ’s zero
out everything to the right of the first superdiagonal. Tridiagonal matrices make computations
fast, so computing the Hessenberg form of a symmetric matrix is very useful.
5
Least Squares and
Computing
Eigenvalues

Lab Objective: Because of its numerical stability and convenient structure, the QR decomposition
is the basis of many important and practical algorithms. In this lab we introduce linear least squares
problems, tools in Python for computing least squares solutions, and two fundamental algorithms for
computing eigenvalues. The QR decomposition makes solving several of these problems quick and
numerically stable.

Least Squares
A linear system Ax = b is overdetermined if it has more equations than unknowns. In this situation,
there is no true solution, and x can only be approximated.
The least squares solution of Ax = b, denoted x b, is the “closest” vector to a solution, meaning
it minimizes the quantity ∥Ab x − b∥2 . In other words, x
b is the vector such that Ab x is the projection
of b onto the range of A, and can be calculated by solving the normal equations,1

AT Ab
x = AT b.

If A is full rank (which it usually is in applications) its QR decomposition provides an efficient

way to solve the normal equations. Let A = Q bR b be the reduced QR decomposition of A, so Q b is
m × n with orthonormal columns and R b is n × n, invertible, and upper triangular. Since QT
b Qb = I,
and since RbT is invertible, the normal equations can be reduced as follows (we omit the hats on Q b
and R
b for clarity).

AT Ab
x = AT b
(QR)T QRb
x = (QR)T b
RT QT QRb
x = RT QT b
RT Rb
x = RT QT b
x = QT b
Rb (5.1)

Thus xb is the least squares solution to Ax = b if and only if Rb b T b. Since R

bx = Q b is upper
triangular, this equation can be solved quickly with back substitution.
1 See Volume 1 for a formal derivation of the normal equations.

45
46 Lab 5. Least Squares and Computing Eigenvalues

Problem 1. Write a function that accepts an m × n matrix A of rank n and a vector b of

length m. Use the reduced QR decomposition of A and (5.1) to solve the normal equations
corresponding to Ax = b.
You may use either SciPy’s reduced QR routine (la.qr() with mode="economic") or one
of your own reduced QR routines. In addition, you may use la.solve_triangular(), SciPy’s
optimized routine for solving triangular systems.

Fitting a Line
The least squares solution can be used to find the best fit curve of a chosen type to a set of points.
Consider the problem of finding the line y = ax + b that best fits a set of m points {(xk , yk )}mk=1 .
Ideally, we seek a and b such that yk = axk + b for all k. These equations can be simultaneously
represented by the linear system
   
x1 1 y1
 x2 1      y2 

 x3 1  a
 
 y3 
Ax =  =  = b. (5.2)
 . ..  b  . 

 .. .   .
. 
xm 1 ym

Note that A has full column rank as long as not all of the xk values are the same.
Because this system has two unknowns, it is guaranteed to have a solution if it has two or fewer
equations. However, if there are more than two data points, the system is overdetermined if any set
of three points is not collinear. We therefore seek a least squares solution, which in this case means
finding the slope ba and y-intercept bb such that the line y = b
ax + bb best fits the data.
Figure 5.1 is a typical example of this idea where b a ≈ 21 and bb ≈ −3.

Data Points
2 Least Squares Fit
1
0
1
2
3
0 2 4 6 8 10
Figure 5.1: A linear least squares fit.
 47

Problem 2. The file housing.npy contains the purchase-only housing price index, a measure
of how housing prices are changing, for the United States from 2000 to 2016.a Each row in the
array is a separate measurement; the columns are the year and the price index, in that order.
To avoid large numerical computations, the year measurements start at 0 instead of 2000.
Find the least squares line that relates the year to the housing price index (i.e., let year
be the x-axis and index the y-axis).

1. Construct the matrix A and the vector b described by (5.2).

(Hint: np.vstack(), np.column_stack(), and/or np.ones() may be helpful.)

2. Use your function from Problem 1 to find the least squares solution.

3. Plot the data points as a scatter plot.

4. Plot the least squares line with the scatter plot.

a See http://www.fhfa.gov/DataTools/Downloads/Pages/House-Price-Index.aspx.

Note
The least squares problem of fitting a line to a set of points is often called linear regression,
and the resulting line is called the linear regression line. SciPy’s specialized tool for linear
regression is scipy.stats.linregress(). This function takes in an array of x-coordinates and
a corresponding array of y-coordinates, and returns the slope and intercept of the regression
line, along with a few other statistical measurements.
For example, the following code produces Figure 5.1.

>>> import numpy as np

>>> from scipy.stats import linregress

# Generate some random data close to the line y = .5x - 3.

>>> x = np.linspace(0, 10, 20)
>>> y = .5*x - 3 + np.random.randn(20)

# Use linregress() to calculate m and b, as well as the correlation

# coefficient, p-value, and standard error. See the documentation for
# details on each of these extra return values.
>>> a, b, rvalue, pvalue, stderr = linregress(x, y)

>>> plt.plot(x, y, 'k*', label="Data Points")

>>> plt.plot(x, a*x + b, label="Least Squares Fit")
>>> plt.legend(loc="upper left")
>>> plt.show()
48 Lab 5. Least Squares and Computing Eigenvalues

Fitting a Polynomial
Least squares can also be used to fit a set of data to the best fit polynomial of a specified degree. Let
k=1 be the set of m data points in question. The general form for a polynomial of degree
{(xk , yk )}m
n is
n
X
pn (x) = cn xn + cn−1 xn−1 + · · · + c2 x2 + c1 x + c0 = ci xi .
i=0

Note that the polynomial is uniquely determined by its n + 1 coefficients {ci }ni=0 . Ideally, then, we
seek the set of coefficients {ci }ni=0 such that

yk = cn xnk + cn−1 xn−1

k + · · · + c2 x2k + c1 xk + c0

for all values of k. These m linear equations yield the linear system

cn
 
xn1 xn−1 ··· x21
   
1 x1 1 y1
xn2 xn−1 x22   cn−1
 
 2 ··· x2 1   ..
  y2 
xn−1  .
   
Ax = 
 xn3 3 ··· x23 x3 1 =
  y3  = b.

(5.3)
.. .. .. .. .. ..

   c2   
 . . . . .
 c1
 
 . 
xnm xn−1
m ··· x2m xm 1 ym
c0

If m > n + 1 this system is overdetermined, requiring a least squares solution.

Working with Polynomials in NumPy

The m × (n + 1) matrix A of (5.3) is called a Vandermonde matrix.2 NumPy’s np.vander() is a

convenient tool for quickly constructing a Vandermonde matrix, given the values {xk }m
k=1 and the
number of desired columns.

>>> print(np.vander([2, 3, 5], 2))

[[2 1] # [[2**1, 2**0]
[3 1] # [3**1, 3**0]
[5 1]] # [5**1, 5**0]]

>>> print(np.vander([2, 3, 5, 4], 3))

[[ 4 2 1] # [[2**2, 2**1, 2**0]
[ 9 3 1] # [3**2, 3**1, 3**0]
[25 5 1] # [5**2, 5**1, 5**0]
[16 4 1]] # [4**2, 4**1, 4**0]

NumPy also has powerful tools for working efficiently with polynomials. The class np.poly1d
represents a 1-dimensional polynomial. Instances of this class are callable like a function.3 The
constructor accepts the polynomial’s coefficients, from largest degree to smallest.
Table 5.1 lists some attributes and methods of the np.poly1d class.
2 Vandermonde matrices have many special properties and are useful for many applications, including polynomial

interpolation and discrete Fourier analysis.

3 Class instances can be made callable by implementing the __call__() magic method.
 49

Attribute Description
coeffs The n + 1 coefficients, from greatest degree to least.
order The polynomial degree (n).
roots The n roots of the polynomial.

Method Returns
deriv() The coefficients of the polynomial after being differentiated.
integ() The coefficients of the polynomial after being integrated (with c0 = 0).

Table 5.1: Attributes and methods of the np.poly1d class.

# Create a callable object for the polynomial f(x) = (x-1)(x-2) = x^2 - 3x + 2.

>>> f = np.poly1d([1, -3, 2])
>>> print(f)
2
1 x - 3 x + 2

# Evaluate f(x) for several values of x in a single function call.

>>> f([1, 2, 3, 4])
array([0, 0, 2, 6])

Problem 3. The data in housing.npy is nonlinear, and might be better fit by a polynomial
than a line.
Write a function that uses (5.3) to calculate the polynomials of degree 3, 6, 9, and 12 that
best fit the data. Plot the original data points and each least squares polynomial together in
individual subplots.
(Hint: define a separate, refined domain with np.linspace() and use this domain to smoothly
plot the polynomials.)
Instead of using Problem 1 to solve the normal equations, you may use SciPy’s least
squares routine, scipy.linalg.lstsq().

>>> from scipy import linalg as la

# Define A and b appropriately.

# Solve the normal equations using SciPy's least squares routine.

# The least squares solution is the first of four return values.
>>> x = la.lstsq(A, b)[0]

Compare your results to np.polyfit(). This function receives an array of x values, an

array of y values, and an integer for the polynomial degree, and returns the coefficients of the
best fit polynomial of that degree.
50 Lab 5. Least Squares and Computing Eigenvalues

Achtung!
Having more parameters in a least squares model is not always better. For a set of m points, the
best fit polynomial of degree m − 1 interpolates the data set, meaning that p(xk ) = yk exactly
for each k. In this case there are enough unknowns that the system is no longer overdetermined.
However, such polynomials are highly subject to numerical errors and are unlikely to accurately
represent true patterns in the data.
Choosing to have too many unknowns in a fitting problem is (fittingly) called overfitting,
and is an important issue to avoid in any statistical model.

Fitting a Circle
Suppose the set of m points {(xk , yk )}m
k=1 are arranged in a nearly circular pattern. The general
equation of a circle with radius r and center (c1 , c2 ) is

(x − c1 )2 + (y − c2 )2 = r2 . (5.4)

The circle is uniquely determined by r, c1 , and c2 , so these are the parameters that should be
solved for in a least squares formulation of the problem. However, (5.4) is not linear in any of these
variables.

(x − c1 )2 + (y − c2 )2 = r2
x2 − 2c1 x + c21 + y 2 − 2c2 y + c22 = r2
x2 + y 2 = 2c1 x + 2c2 y + r2 − c21 − c22 (5.5)

The quadratic terms x2 and y 2 are acceptable because the points {(xk , yk )}mk=1 are given.
To eliminate the nonlinear terms in the unknown parameters r, c1 , and c2 , define a new variable
c3 = r2 − c21 − c22 . Then for each point (xk , yk ), (5.5) becomes

2c1 xk + 2c2 yk + c3 = x2k + yk2 .

These m equations are linear in c1 , c2 , and c3 , and can be written as the linear system

x21 + y12
   
2x1 2y1 1  
 2x2 2y2 1  c1  x22 + y22 
.. .. ..  c2  =  .. . (5.6)
   
. . . .

c3
   
2xm 2ym 1 x2m + ym
2

After solving for the least squares solution, r can be recovered with the relation r = c21 + c22 + c3 .
p

Finally, plotting a circle is best done with polar coordinates. Using the same variables as before, the
circle can be represented in polar coordinates by setting

x = r cos(θ) + c1 , y = r sin(θ) + c2 , θ ∈ [0, 2π]. (5.7)

To plot the circle, solve the least squares system for c1 , c2 , and r, define an array for θ, then use
(5.7) to calculate the coordinates of the points the circle.
 51

# Load some data and construct the matrix A and the vector b.
>>> xk, yk = np.load("circle.npy").T
>>> A = np.column_stack((2*xk, 2*yk, np.ones_like(xk)))
>>> b = xk**2 + yk**2

# Calculate the least squares solution and solve for the radius.
>>> c1, c2, c3 = la.lstsq(A, b)[0]
>>> r = np.sqrt(c1**2 + c2**2 + c3)

# Plot the circle using polar coordinates.

>>> theta = np.linspace(0, 2*np.pi, 200)
>>> x = r*np.cos(theta) + c1
>>> y = r*np.sin(theta) + c2
>>> plt.plot(x, y) # Plot the circle.
>>> plt.plot(xk, yk, 'k*') # Plot the data points.
>>> plt.axis("equal")

2
4 2 0 2 4 6 8 10

Problem 4. The general equation for an ellipse is

ax2 + bx + cxy + dy + ey 2 = 1.

Write a function that calculates the parameters for the ellipse that best fits the data in the
file ellipse.npy. Plot the original data points and the ellipse together, using the following
function to plot the ellipse.

def plot_ellipse(a, b, c, d, e):

"""Plot an ellipse of the form ax^2 + bx + cxy + dy + ey^2 = 1."""
theta = np.linspace(0, 2*np.pi, 200)
cos_t, sin_t = np.cos(theta), np.sin(theta)
52 Lab 5. Least Squares and Computing Eigenvalues

A = a*(cos_t**2) + c*cos_t*sin_t + e*(sin_t**2)

B = b*cos_t + d*sin_t
r = (-B + np.sqrt(B**2 + 4*A)) / (2*A)
plt.plot(r*cos_t, r*sin_t, lw=2)
plt.gca().set_aspect("equal", "datalim")

Computing Eigenvalues
The eigenvalues of an n×n matrix A are the roots of its characteristic polynomial det(A−λI). Thus,
finding the eigenvalues of A amounts to computing the roots of a polynomial of degree n. However,
for n ≥ 5, it is provably impossible to find an algebraic closed-form solution to this problem.4 In
addition, numerically computing the roots of a polynomial is a famously ill-conditioned problem,
meaning that small changes in the coefficients of the polynomial (brought about by small changes
in the entries of A) may yield wildly different results. Instead, eigenvalues must be computed with
iterative methods.

The Power Method

The dominant eigenvalue of the n × n matrix A is the unique eigenvalue of greatest magnitude, if
such an eigenvalue exists. The power method iteratively computes the dominant eigenvalue of A and
its corresponding eigenvector.
Begin by choosing a vector x0 such that ∥x0 ∥2 = 1, and define

Axk
xk+1 = .
∥Axk ∥2

If A has a dominant eigenvalue λ, and if the projection of x0 onto the subspace spanned by the
eigenvectors corresponding to λ is nonzero, then the sequence of vectors (xk )∞
k=0 converges to an
eigenvector x of A corresponding to λ.
Since x is an eigenvector of A, Ax = λx. Left multiplying by xT on each side results in
T
x Ax = λxT x, and hence λ = xxTAx
T
x
. This ratio is called the Rayleigh quotient. However, since each
xk is normalized, xT x = ∥x∥22 = 1, so λ = xT Ax.
The entire algorithm is summarized below.

Algorithm 1
1: procedure PowerMethod(A)
2: m, n ← shape(A) ▷ A is square so m = n.
3: x0 ← random(n) ▷ A random vector of length n
4: x0 ← x0 /∥x0 ∥2 ▷ Normalize x0
5: for k = 0, 1, . . . , N − 1 do
6: xk+1 ← Axk
7: xk+1 ← xk+1 /∥xk+1 ∥2
8: return xT
N AxN , xN

4 This result, called Abel’s impossibility theorem, was first proven by Niels Heinrik Abel in 1824.
 53

The power method is limited by a few assumptions. First, not all square matrices A have
a dominant eigenvalue. However, the Perron-Frobenius theorem guarantees that if all entries of
A are positive, then A has a dominant eigenvalue. Second, there is no way to choose an x0 that is
guaranteed to have a nonzero projection onto the span of the eigenvectors corresponding to λ, though
a random x0 will almost surely satisfy this condition. Even with these assumptions, a rigorous proof
that the power method converges is most convenient with tools from spectral calculus, and as such
will not be pursued here.

Problem 5. Write a function that accepts an n×n matrix A, a maximum number of iterations
N , and a stopping tolerance tol. Use Algorithm 1 to compute the dominant eigenvalue of A
and a corresponding eigenvector. Continue the loop in step 5 until either ∥xk+1 − xk ∥2 is less
than the tolerance tol, or until iterating the maximum number of times N .
Test your function on square matrices with all positive entries, verifying that Ax = λx.
Use SciPy’s eigenvalue solver, scipy.linalg.eig(), to compute all of the eigenvalues and
corresponding eigenvectors of A and check that λ is the dominant eigenvalue of A. There is
also a file called test_lstsq_eigs.py that has prewritten unit tests for this problem that you
can use to check your code.

# Construct a random matrix with positive entries.

>>> A = np.random.random((10,10))

# Compute the eigenvalues and eigenvectors of A via SciPy.

>>> eigs, vecs = la.eig(A)

# Get the dominant eigenvalue and eigenvector of A.

# The eigenvector of the kth eigenvalue is the kth column of 'vecs'.
>>> loc = np.argmax(eigs)
>>> lamb, x = eigs[loc], vecs[:,loc]

# Verify that Ax = lambda x.

>>> np.allclose(A @ x, lamb * x)
True

The QR Algorithm
An obvious shortcoming of the power method is that it only computes one eigenvalue and eigenvector.
The QR algorithm, on the other hand, attempts to find all eigenvalues of A.
Let A0 = A, and for arbitrary k let Qk Rk = Ak be the QR decomposition of Ak . Since A is
square, so are Qk and Rk , so they can be recombined in reverse order:
Ak+1 = Rk Qk .
This recursive definition establishes an important relation between the Ak :
Q−1 −1 −1
k Ak Qk = Qk (Qk Rk )Qk = (Qk Qk )(Rk Qk ) = Ak+1 .

Thus, Ak is orthonormally similar to Ak+1 , and similar matrices have the same eigenvalues. The
series of matrices (Ak )∞
k=0 converges to the block matrix
54 Lab 5. Least Squares and Computing Eigenvalues

∗ ∗ ··· ∗
 

S1 ∗ ··· ∗
 s1
.. ..  0 s2,1 s2,2 ··· ∗ 
. .
 
0 S2
 
For example, S =  s2,3 s2,4 ··· ∗
 
S= . .
 
.. .. .. .. ..
. . . ∗
 
.
 
.
   
0 · · · 0 Sm sm

Each Si is either a 1×1 or 2×2 matrix.5 In the example above on the right, since the first subdiagonal
entry is zero, S1 is the 1 × 1 matrix with a single entry, s1 . But as s2,3 is not zero, S2 is 2 × 2.

Since S is block upper triangular, its eigenvalues are the eigenvalues of its diagonal Si blocks.
Then because A is similar to each Ak , those eigenvalues of S are the eigenvalues of A.

When A has real entries but complex eigenvalues, 2 × 2 Si blocks appear in S. Finding eigen-
values of a 2 × 2 matrix is equivalent to finding the roots of a 2nd degree polynomial,

a−λ b
det(Si − λI) = = (a − λ)(d − λ) − bc = λ2 − (a + d)λ + (ad − bc), (5.8)
c d−λ

which has a closed form solution via the quadratic equation. This also demonstrates that complex
eigenvalues come in conjugate pairs.

Hessenberg Preconditioning

A matrix in upper Hessenberg form is one that has all entries below the first subdiagonal equal to
zero. This is similar to an upper triangular matrix, except that the entries directly below the diagonal
are also allowed to be nonzero. The QR algorithm works more accurately and efficiently on matrices
that are in upper Hessenberg form, as upper Hessenberg matrices are already close to triangular.
Furthermore, if H = QR is the QR decomposition of upper Hessenberg H then RQ is also upper
Hessenberg, so the almost-triangular form is preserved at each iteration. Putting a matrix in upper
Hessenberg form before applying the QR algorithm is called Hessenberg preconditioning.

With preconditioning in mind, the entire QR algorithm is as follows.

5 If all of the S are 1 × 1 matrices, then the upper triangular S is called the Schur form of A. If some of the S are
i i
2 × 2 matrices, then S is called the real Schur form of A.
 55

Algorithm 2
1: procedure QR_Algorithm(A, N )
2: m, n ← shape(A)
3: S ← hessenberg(A) ▷ Put A in upper Hessenberg form.
4: for k = 0, 1, . . . , N − 1 do
5: Q, R ← S ▷ Get the QR decomposition of Ak .
6: S ← RQ ▷ Recombine Rk and Qk into Ak+1 .
7: eigs ← [] ▷ Initialize an empty list of eigenvalues.
8: i←0
9: while i < n do
10: if Si is 1 × 1 then
11: Append the only entry si of Si to eigs
12: else if Si is 2 × 2 then
13: Calculate the eigenvalues of Si
14: Append the eigenvalues of Si to eigs
15: i←i+1
16: i←i+1 ▷ Move to the next Si .
17: return eigs

Problem 6. Write a function that accepts an n × n matrix A, a number of iterations N , and

a tolerance tol. Use Algorithm 2 to implement the QR algorithm with Hessenberg precondi-
tioning, returning the eigenvalues of A.
Consider the following implementation details.

• Use scipy.linalg.hessenberg() or your own Hessenburg algorithm to reduce A to

upper Hessenberg form in step 3.

• The loop in step 4 should run for N total iterations.

• Use scipy.linalg.qr() or one of your own QR factorization routines to compute the

QR decomposition of S in step 5. Note that since S is in upper Hessenberg form, Givens
rotations are the most efficient way to produce Q and R.

• Assume that Si is 1 × 1 in step 10 if one of two following criteria hold:

– Si is the last diagonal entry of S.

– The absolute value of element below the ith main diagonal entry of S (the lower left
element of the 2 × 2 block) is less than tol.

• If Si is 2 × 2, use the quadratic formula and (5.8) to compute its eigenvalues. Use the
function cmath.sqrt() to correctly compute the square root of a negative number.

Test your function on small random symmetric matrices, comparing your results to SciPy’s
scipy.linalg.eig(). While the QR algorithm works on arbitrary matrices, it has better con-
vergence properties for symmetric matrices, which makes them better for testing. To construct
a random symmetric matrix, note that A + AT is always symmetric.
56 Lab 5. Least Squares and Computing Eigenvalues

Unit Test
There is a file called test_lstsq_eigs.py that contains some prewritten unit tests for Problem
5. There is a place for you to add your own unit tests for Problem 6 called test_qr_algorithm.
You are required to include at least one unit test which will be graded.

Note
Algorithm 2 is theoretically sound, but can still be greatly improved. Most modern computer
packages instead use the implicit QR algorithm, an improved version of the QR algorithm, to
compute eigenvalues.
For large matrices, there are other iterative methods besides the power method and the
QR algorithm for efficiently computing eigenvalues. They include the Arnoldi iteration, the
Jacobi method, the Rayleigh quotient method, and others.
 57

Additional Material
Variations on the Linear Least Squares Problem
If W is an n × n is symmetric positive-definite matrix, then the function ∥ · ∥W 2 : Rn → R given by
√
∥x∥W 2 = ∥W x∥2 = xT W T W x

defines a norm and is called a weighted 2-norm. Given the overdetermined system Ax = b, the
problem of choosing x
b to minimize ∥Ab x − b∥W 2 is called a weighted least squares (WLS) problem.
This problem has a slightly different set of normal equations,

AT W T W Ab
x = AT W T W b.

However, letting C = W A and z = W b, this equation reduces to the usual normal equations,

C T Cb
x = C T z,

so a WLS problem can be solved in the same way as an ordinary least squares (OLS) problem.
Weighted least squares is useful when some points in a data set are more important than others.
Typically W is chosen to be a diagonal matrix, and each positive diagonal entry Wi,i indicate how
much weight should be given to the ith data point. For example, Figure 5.2a shows OLS and WLS
fits of an exponential curve y = aekx to data that gets more sparse as x increases, where the matrix
W is chosen to give more weight to the data with larger x values.
Alternatively, the least squares problem can be formulated with other common vector norms,
but such problems cannot be solved via the normal equations. For example, minimizing ∥Ax − b∥1 or
∥Ax−b∥∞ is usually done by solving an equivalent linear program, a type of constrained optimization
problem. These norms may be better suited to a particular application than the regular 2-norm.
Figure 5.2b illustrates how different norms give slightly different results in the context of Problem 4.

800 OLS 5
WLS || ||2 fit || || fit
data 4 || ||1 fit data
600
3
400 2
1
200
0
0 1
0.0 0.2 0.4 0.6 0.8 1.0 2
1e9 4 2 0 2 4

(a) Ordinary and weighted least squares fits for (b) Best fits for elliptical data with respect to
exponential data. different vector norms.

Figure 5.2: Variations on the ordinary least squares problem.

The Inverse Power Method

The major drawback of the power method is that it only computes a single eigenvector-eigenvalue
pair, and it is always the eigenvalue of largest magnitude. The inverse power method, sometimes
simply called the inverse iteration, is a way of computing an eigenvalue that is closest in magnitude
to an initial guess. They key observation is that if λ is an eigenvalue of A, then 1/(λ − µ) is an
eigenvalue of (A − µI)−1 , so applying the power method to (A − µI)−1 yields the eigenvalue of A
that is closest in magnitude to µ.
58 Lab 5. Least Squares and Computing Eigenvalues

The inverse power method is more expensive than the regular power method because at each
iteration, instead of a matrix-vector multiplication (step 6 of Algorithm 1), a system of the form
(A − µI)x = b must be solved. To speed this step up, start by taking the LU or QR factorization of
A − µI before the loop, then use the factorization and back substitution to solve the system quickly
within the loop. For instance, if QR = A − µI, then since Q−1 = QT ,

b = (A − µI)x = QRx ⇔ Rx = QT b,

which is a triangular system. This version of the algorithm is described below.

Algorithm 3
1: procedure InversePowerMethod(A, µ)
2: m, n ← shape(A)
3: x0 ← random(n)
4: x0 ← x0 /∥x0 ∥
5: Q, R ← A − µI ▷ Factor A − µI with la.qr().
6: for k = 0, 1, 2, . . . , N − 1 do
7: Solve Rxk+1 = QT xk ▷ Use la.solve_triangular().
8: xk+1 ← xk+1 /∥xk+1 ∥
9: return xT
N AxN , xN

It is worth noting that the QR algorithm can be improved with a similar technique: instead of
computing the QR factorization of Ak , factor the shifted matrix Ak − µk I, where µk is a guess for
an eigenvalue of A, and unshift the recombined factorization accordingly. That is, compute

Qk Rk = Ak − µk I,
Ak+1 = Rk Qk + µk I.

This technique yields the single-shift QR algorithm. Another variant, the practical QR algorithm, uses
intelligent shifts and recursively operates on smaller blocks of Ak+1 where possible. See [QSS10, TB97]
for further discussion.
 Image Segmentation

6
Lab Objective: Graph theory has a variety of applications. A graph (or network) can be represented
in many ways on a computer. In this lab we study a common matrix representation for graphs and
show how certain properties of the matrix representation correspond to inherent properties of the
original graph. We also introduce tools for working with images in Python, and conclude with an
application of using graphs and linear algebra to segment images.

Graphs as Matrices
A graph is a mathematical structure that represents relationships between objects. Graphs are
defined by G = (V, E), where V is a set of vertices (or nodes) and E is a set of edges, each of which
connects one node to another. A graph can be classified in several ways.

• The edges of an undirected graph are bidirectional: if an edge goes from node A to node B,
then that same edge also goes from B to A. For example, the graphs G1 and G2 in Figure 6.1
are both undirected. In a directed graph, edges only go one way, usually indicated by an arrow
pointing from one node to another. In this lab, we focus on undirected graphs.
• The edges of a weighted graph have a weight assigned to them, such as G2 . A weighted graph
could represent a collection of cities with roads connecting them: each vertex would represent
a city, and the edges would represent roads between the cities. The length of each road could
be the weight of the corresponding edge. An unweighted graph like G1 does not have weights
assigned to its edges, but any unweighted graph can be thought of as a weighted graph by
assigning a weight of 1 to every edge.

4 5 1 3 5

2
6 1 3 1 1

3 2 2 4 6
.5
(a) G1 , an unweighted undirected graph. (b) G2 , a weighted undirected graph.

Figure 6.1

59
60 Lab 6. Image Segmentation

Adjacency, Degree, and Laplacian Matrices

For computation and analysis, graphs are commonly represented by a few special matrices. For these
definitions, let G be a graph with N nodes and let wij be the weight of the edge connecting node i
to node j (if such an edge exists).

1. The adjacency matrix of G is the N × N matrix A with entries

(
wij if an edge connects node i and node j
aij =
0 otherwise.

The adjacency matrices A1 of G1 and A2 of G2 are

   
0 1 0 0 1 1 0 3 0 0 0 0
 1 0 1 0 1 0   3 0 0 0 0 0 
 
 
 0 1 0 1 0 0   0 0 0 1 0 0 
A1 = 
 0
, A2 =  .
 0 1 0 1 1    0
 0 1 0 2 .5 

 1 1 0 1 0 0   0 0 0 2 0 1 
1 0 0 1 0 0 0 0 0 .5 1 0
Notice that these adjacency matrices are symmetric. This is always the case for undirected
graphs since the edges are bidirectional.
2. The degree matrix of G is the N × N diagonal matrix D whose ith diagonal entry is
N
X
dii = wij . (6.1)
j=1

The degree matrices D1 of G1 and D2 of G2 are

   
3 0 0 0 0 0 3 0 0 0 0 0
 0 3 0 0 0 0   0 3 0 0 0 0 
   
 0 0 2 0 0 0   0 0 1 0 0 0 
D1 =  0 0
, D 2 =  .
 0 3 0 0    0 0 0
 3.5 0 0 

 0 0 0 0 3 0   0 0 0 0 3 0 
0 0 0 0 0 2 0 0 0 0 0 1.5
The ith diagonal entry of D is called the degree of node i, the sum of the weights of the edges
leaving node i.
3. The Laplacian matrix of G is the N × N matrix L defined as

L = D − A, (6.2)

where D is the degree matrix of G and A is the adjacency matrix of G. For G1 and G2 , the
Laplacian matrices L1 and L2 are
3 −1 −1 −1 3 −3
   
0 0 0 0 0 0
 −1 3 −1 0 −1 0   −3 3 0 0 0 0 
   
 0 −1 2 −1 0 0   0 0 1 −1 0 0 
L1 =  , L2 =  .
 0
 0 −1 3 −1 −1    0 0 −1 3.5 −2 −.5 

 −1 −1 0 −1 3 0   0 0 0 −2 3 −1 
−1 0 0 −1 0 2 0 0 0 −.5 −1 1.5
 61

Problem 1. Write a function that accepts the adjacency matrix A of a graph G. Use (6.1)
and (6.2) to compute the Laplacian matrix L of G.
(Hint: The diagonal entries of D can be computed in one line by summing A over an axis.)
Test your function on the graphs G1 and G2 from Figure 6.1 and validate your results
with scipy.sparse.csgraph.laplacian().

Connectivity
A connected graph is a graph where every vertex is connected to every other vertex by at least one
path. For example, G1 is connected, whereas G2 is not because there is no path from node 1 (or
node 2) to node 3 (or nodes 4, 5, or 6). The naïve brute-force algorithm for determining if a graph
is connected is to check that there is a path from each edge to every other edge. While this may
work for very small graphs, most interesting graphs have thousands of vertices, and for such graphs
this approach is prohibitively expensive. Luckily, an interesting result from algebraic graph theory
relates the connectivity of a graph to its Laplacian matrix.
If L is the Laplacian matrix of a graph, then the definition of D and the construction L = D −A
guarantees that the rows (and columns) of L must each sum to 0. Therefore L cannot have full rank,
so λ = 0 must be an eigenvalue of L. Furthermore, if L represents a graph that is not connected,
more than one of the eigenvalues of L must be zero. To see this, let J ⊂ {1, 2, . . . , N } such that the
vertices {vj }j∈J form a connected component of the graph, meaning that there is a path between
each pair of vertices in the set. Next, let x be the vector with entries
(
1, k ∈ J
xk =
0, k ̸∈ J.
Then x is an eigenvector of L corresponding to the eigenvalue λ = 0.
For example, the example graph G2 has two connected components.

1. J1 = {1, 2} so that x1 = [1, 1, 0, 0, 0, 0]T . Then

3 −3
    
0 0 0 0 1 0
 −3 3 0 0 0 0   1   0 
    
 0 0 1 −1 0 0 
 0
   0 
L2 x 1 =  =  = 0.
 0
 0 −1 3.5 −2 −.5 
 0
  
  0 

 0 0 0 −2 3 −1  0
   0 
0 0 0 −.5 −1 1.5 0 0

2. J2 = {3, 4, 5, 6} and hence x2 = [0, 0, 1, 1, 1, 1]T . Then

3 −3
    
0 0 0 0 0 0
 −3 3 0 0 0 0  0   0 
    
 0 0 1 −1 0 0  1   0 
L2 x2 =   =  = 0.
 0
 0 −1 3.5 −2 −.5 
 1  
  0 

 0 0 0 −2 3 −1  1   0 
0 0 0 −.5 −1 1.5 1 0

In fact, it can be shown that the number of zero eigenvalues of the Laplacian exactly equals
the number of connected components. This makes calculating how many connected components are
in a graph only as hard as calculating the eigenvalues of its Laplacian.
62 Lab 6. Image Segmentation

A Laplacian matrix L is always a positive semi-definite matrix when all weights in the graph
are positive, meaning that its eigenvalues are each nonnegative. The second smallest eigenvalue of
L is called the algebraic connectivity of the graph. It is clearly 0 for non-connected graphs, but
for a connected graph, the algebraic connectivity provides useful information about its sparsity or
“connectedness.” A higher algebraic connectivity indicates that the graph is more strongly connected.

Problem 2. Write a function that accepts the adjacency matrix A of a graph G and a small
tolerance value tol. Compute the number of connected components in G and its algebraic
connectivity. Consider all eigenvalues that are less than the given tol to be zero.
Use scipy.linalg.eig() or scipy.linalg.eigvals() to compute the eigenvalues of
the Laplacian matrix. These functions return complex eigenvalues (with negligible imaginary
parts); use np.real() to extract the real parts.

Unit Test
Write unit tests for Problem 2 in test_image_segmentation.py. There are example unit tests
for Problem 1 to help check the Laplacian.

Images as Matrices
Computer images are stored as arrays of integers that indicate pixel values. Most m × n grayscale
(black and white) images are stored in Python as a m × n NumPy arrays, while most m × n color
images are stored as 3-dimensional m × n × 3 arrays. Color image arrays can be thought of as a stack
of three m × n arrays, one each for red, green, and blue values. The datatype for an image array is
np.uint8, unsigned 8-bit integers that range from 0 to 255. A 0 indicates a black pixel while a 255
indicates a white pixel.
Use imageio.imread() to read an image from a file and imageio.imwrite() to save an image.
Matplotlib’s plt.imshow() displays an image array, but it displays arrays of floats between 0 and 1
more cleanly than arrays of 8-bit integers. Therefore it is customary to scale the array by dividing
each entry by 255 before processing or showing the image. In this case, a 0 still indicates a black
pixel, but now a 1 indicates pure white.

>>> from imageio import imread

>>> from matplotlib import pyplot as plt

>>> image = imread("dream.png") # Read a (very) small image.

>>> print(image.shape) # Since the array is 3-dimensional,
(48, 48, 3) # this is a color image.

# The image is read in as integers from 0 to 255.

>>> print(image.min(), image.max(), image.dtype)
0 254 uint8

# Scale the image to floats between 0 and 1 for Matplotlib.

>>> scaled = image / 255.
 63

>>> print(scaled.min(), scaled.max(), scaled.dtype)

0.0 0.996078431373 float64

# Display the scaled image.

>>> plt.imshow(scaled)
>>> plt.axis("off")

A color image can be converted to grayscale by averaging the RGB values of each pixel, resulting
in a 2-D array called the brightness of the image. To properly display a grayscale image, specify the
keyword argument cmap="gray" in plt.imshow().

# Average the RGB values of a colored image to obtain a grayscale image.

>>> brightness = scaled.mean(axis=2) # Average over the last axis.
>>> print(brightness.shape) # Note that the array is now 2-D.
(48, 48)

# Display the image in gray.

>>> plt.imshow(brightness, cmap="gray")
>>> plt.axis("off")

Finally, it is often important in applications to flatten an image matrix into a large 1-D array.
Use np.ravel() to convert a m × n array into a 1-D array with mn entries.

>>> import numpy as np

>>> A = np.random.randint(0, 10, (3,4))
>>> print(A)
[[4 4 7 7]
[8 1 2 0]
[7 0 0 9]]

# Unravel the 2-D array (by rows) into a 1-D array.

>>> np.ravel(A)
array([4, 4, 7, 7, 8, 1, 2, 0, 7, 0, 0, 9])

# Unravel a grayscale image into a 1-D array and check its size.
>>> M,N = brightness.shape
>>> flat_brightness = np.ravel(brightness)
>>> M*N == flat_brightness.size
True
>>> print(flat_brightness.shape)
(2304,)

Problem 3. Define a class called ImageSegmenter.

64 Lab 6. Image Segmentation

1. Write the constructor so that it accepts the name of an image file. Read the image, scale
it so that it contains floats between 0 and 1, then store it as an attribute. If the image is
in color, compute its brightness matrix by averaging the RGB values at each pixel (if it is
a grayscale image, the image array itself is the brightness matrix). Flatten the brightness
matrix into a 1-D array and store it as an attribute.

2. Write a method called show_original() that displays the original image. If the original
image is grayscale, remember to use cmap="gray" as part of plt.imshow().

Achtung!
Matplotlib’s plt.imread() also reads image files. However, this function automatically scales
PNG image entries to floats between 0 and 1, but it still reads non-PNG image entries as 8-bit
integers. To avoid this inconsistent behavior, always use imageio.imread() to read images
and divide by 255 when scaling is desired.

Graph-based Image Segmentation

Image segmentation is the process of finding natural boundaries in an image and partitioning the
image along those boundaries (see Figure 6.2). Though humans can easily pick out portions of an
image that “belong together,” it takes quite a bit of work to teach a computer to recognize boundaries
and sections in an image. However, segmenting an image often makes it easier to analyze, so image
segmentation is ongoing area of research in computer vision and image processing.

Figure 6.2: The image dream.png and its segments.

There are many ways to approach image segmentation. The following algorithm, developed by
Jianbo Shi and Jitendra Malik in 2000 [SM00], converts the image to a graph and “cuts” it into two
connected components.

Constructing the Image Graph

Let G be a graph whose vertices are the mn pixels of an m × n image (either grayscale or color).
Each vertex i has a brightness B(i), the grayscale or average RGB value of the pixel, as well as a
coordinate location X(i), the indices of the pixel in the original image array.
 65

Define wij , the weight of the edge between pixels i and j, by

 
exp − |B(i)−B(j)| − ∥X(i)−X(j)∥
(
2 2 if ∥X(i) − X(j)∥ < r
wij = σB σX
(6.3)
0 otherwise,

where r, σB2
and σX 2
are constants for tuning the algorithm. In this context, ∥ · ∥ is the standard
euclidean norm, meaning that ∥X(i) − X(j)∥ is the physical distance between vertices i and j,
measured in pixels.
With this definition for wij , pixels that are farther apart than the radius r are not connected at
all in G. Pixels within r of each other are more strongly connected if they are similar in brightness
and close together (the value in the exponential is negative but close to zero). On the other hand,
highly contrasting pixels where |B(i) − B(j)| is large have weaker connections (the value in the
exponential is highly negative).

0 4 8 12
0 0
1
2
3
4 4
0 1 2 3 5 1 4 5 6 9
6
4 5 6 7
7
8 9 10 11 8 8
9
12 13 14 15
10
11
image
12 12
13
14
15

flattened image adjacency matrix A

Figure 6.3: The grid on the left represents a 4 × 4 (m × n) image with 16 pixels. On the right is the
corresponding 16 × 16 (mn × mn) adjacency matrix with all nonzero entries shaded. For example, in
row 5, entries 1, 4, 5, 6, and 9 are nonzero because those pixels are within radius r = 1.2 of pixel 5.

Since there are mn total pixels, the adjacency matrix A of G with entries wij is mn × mn. With
a relatively small radius r, A is relatively sparse, and should therefore be constructed and stored as
a sparse matrix. The degree matrix D is diagonal, so it can be stored as a regular 1-dimensional
NumPy array. The procedure for constructing these matrices can be summarized in just a few steps.
1. Initialize A as a sparse mn × mn matrix and D as a vector with mn entries.
2. For each vertex i (i = 0, 1, . . . , mn − 1),
(a) Find the set of all vertices Ji such that ∥X(i) − X(j)∥ < r for each j ∈ Ji . For example,
in Figure 6.3 i = 5 and Ji = {1, 4, 5, 6, 9}.
66 Lab 6. Image Segmentation

(b) Calculate the weights wij for each j ∈ Ji according to (6.3) and store them in A.
(c) Set the ith element of D to be the sum of the weights, di = j∈Ji wij .
P

The most difficult part to implement efficiently is step 2a, computing the neighborhood Ji of
the current pixel i. However, the computation only requires knowing the current index i, the radius
r, and the height and width m and n of the original image. The following function takes advantage
of this fact and returns (as NumPy arrays) both Ji and the distances ∥X(i) − X(j)∥ for each j ∈ Ji .

def get_neighbors(index, radius, height, width):

"""Calculate the flattened indices of the pixels that are within the given
distance of a central pixel, and their distances from the central pixel.

Parameters:
index (int): The index of a central pixel in a flattened image array
with original shape (radius, height).
radius (float): Radius of the neighborhood around the central pixel.
height (int): The height of the original image in pixels.
width (int): The width of the original image in pixels.

Returns:
(1-D ndarray): the indices of the pixels that are within the specified
radius of the central pixel, with respect to the flattened image.
(1-D ndarray): the euclidean distances from the neighborhood pixels to
the central pixel.
"""
# Calculate the original 2-D coordinates of the central pixel.
row, col = index // width, index % width

# Get a grid of possible candidates that are close to the central pixel.
r = int(radius)
x = np.arange(max(col - r, 0), min(col + r + 1, width))
y = np.arange(max(row - r, 0), min(row + r + 1, height))
X, Y = np.meshgrid(x, y)

# Determine which candidates are within the given radius of the pixel.
R = np.sqrt(((X - col)**2 + (Y - row)**2))
mask = R < radius
return (X[mask] + Y[mask]*width).astype(np.int), R[mask]

To see how this works, consider Figure 6.3 where the original image is 4 × 4 and the goal is to
compute the neighborhood of the pixel i = 5.

# Compute the neighbors and corresponding distances from the figure.

>>> neighbors_1, distances_1 = get_neighbors(5, 1.2, 4, 4)
>>> print(neighbors_1, distances_1, sep='\n')
[1 4 5 6 9]
[ 1. 1. 0. 1. 1.]

# Increasing the radius from 1.2 to 1.5 results in more neighbors.

 67

>>> neighbors_2, distances_2 = get_neighbors(5, 1.5, 4, 4)

>>> print(neighbors_2, distances_2, sep='\n')
[ 0 1 2 4 5 6 8 9 10]
[ 1.41421356 1. 1.41421356 1. 0. 1.
1.41421356 1. 1.41421356]

Problem 4. Write a method for the ImageSegmenter class that accepts floats r defaulting to
2
5, σB defaulting to .02, and σX2
defaulting to 3. Compute the adjacency matrix A and the
degree matrix D according to the weights specified in (6.3).
Initialize A as a scipy.sparse.lil_matrix, which is optimized for incremental construc-
tion. Fill in the nonzero elements of A one row at a time. Use get_neighbors() at each step
to help compute the weights.
(Hint: Try to compute and store an entire row of weights at a time. What does the command
A[5, np.array([1, 4, 5, 6, 9])] = weights do?)
Finally, convert A to a scipy.sparse.csc_matrix, which is faster for computations.
Then return A and D.
Use blue_heart.png to test A and D, saved as HeartMatrixA.npz and HeartMatrixD.npy
datafiles.

Segmenting the Graph

With an image represented as a graph G, the goal is to now split G into two distinct connected
components by removing edges from the existing graph. This is called cutting G, and the set of edges
that are removed is called the cut. The cut with the least weight will best segment the image.
Let D be the degree matrix and L be the Laplacian matrix of G. Shi and Malik [SM00] proved
that the eigenvector corresponding to the second smallest1 eigenvalue of D−1/2 LD−1/2 can be used
to minimize the cut: the indices of its positive entries are the indices of the pixels in the flattened
image which belong to one segment, and the indices of its negative entries are the indices of the pixels
which belong to the other segment.p In this context D
−1/2
refers to element-wise exponentiation, so
the (i, j)th entry of D −1/2
is 1/ dij .
Because A is mn × mn, the desired eigenvector has mn entries. Reshaping the eigenvector to
be m × n allows it to align with the original image. Use the reshaped eigenvector to create a boolean
mask that indexes one of the segments. That is, construct a m × n array where the entries belonging
to one segment are True and the other entries are False.

>>> x = np.arange(-5,5).reshape((5,2)).T
>>> print(x)
[[-5 -3 -1 1 3]
[-4 -2 0 2 4]]

# Construct a boolean mask of x describing which entries of x are positive.

>>> mask = x > 0
>>> print(mask)
1 Both D and L are symmetric matrices, so all eigenvalues of D −1/2 LD −1/2 are real, and therefore “the second

smallest one” is well-defined.

68 Lab 6. Image Segmentation

[[False False False True True]

[False False False True True]]

# Use the mask to zero out all of the nonpositive entries of x.

>>> x * mask
array([[0, 0, 0, 1, 3],
[0, 0, 0, 2, 4]])

Problem 5. Write a method for the ImageSegmenter class that accepts an adjacency matrix
A as a scipy.sparse.csc_matrix and a degree matrix D as a 1-D NumPy array. Construct
an m × n boolean mask describing the segments of the image.

1. Compute the Laplacian L with scipy.sparse.csgraph.laplacian() or by converting

D to a sparse diagonal matrix and computing L = D − A (do not use your function from
Problem 1 unless it works correctly and efficiently for sparse matrices).

2. Construct D−1/2 as a sparse diagonal matrix using D and scipy.sparse.diags(), then

compute D−1/2 LD−1/2 . Use @ or the dot() method of the sparse matrix for the matrix
multiplication, not np.dot().

3. Use scipy.sparse.linalg.eigsh() to compute the eigenvector corresponding to the

second-smallest eigenvalue of D−1/2 LD−1/2 . Set the keyword arguments which="SM"
and k=2 to compute only the two smallest eigenvalues and their eigenvectors.

4. Reshape the eigenvector as a m × n matrix and use this matrix to construct the desired
boolean mask. Return the mask.

Multiplying the boolean mask component-wise by the original image array produces the positive
segment, a copy of the original image where the entries that aren’t in the segment are set to 0.
Computing the negative segment requires inverting the boolean mask, then multiplying the inverted
mask with the original image array. Finally, if the original image is a m × n × 3 color image, the
mask must be stacked into a m × n × 3 array to facilitate entry-wise multiplication.

>>> mask = np.arange(-5,5).reshape((5,2)).T > 0

>>> print(mask)
[[False False False True True]
[False False False True True]]

# The mask can be negated with the tilde operator ~.

>>> print(~mask)
[[ True True True False False]
[ True True True False False]]

# Stack a mask into a 3-D array with np.dstack().

>>> print(mask.shape, np.dstack((mask, mask, mask)).shape)
(2, 5) (2, 5, 3)
 69

Problem 6. Write a method for the ImageSegmenter class that accepts floats r, σB 2
, and σX
2
,
with the same defaults as in Problem 4. Call your methods from Problems 4 and 5 to obtain the
segmentation mask. Plot the original image, the positive segment, and the negative segment
side-by-side in subplots. Your method should work for grayscale or color images.
Use dream.png as a test file and compare your results to Figure 6.2.
70 Lab 6. Image Segmentation
7
The SVD and Image
Compression

Lab Objective: The Singular Value Decomposition (SVD) is an incredibly useful matrix factor-
ization that is widely used in both theoretical and applied mathematics. The SVD is structured in
a way that makes it easy to construct low-rank approximations of matrices, and it is therefore the
basis of several data compression algorithms. In this lab we learn to compute the SVD and use it to
implement a simple image compression routine.

The SVD of a matrix A is a factorization A = U ΣV H where U and V have orthonormal columns

and Σ is diagonal. The diagonal entries of Σ are called the singular values of A and are the square
roots of the eigenvalues of AH A. Since AH A is always positive semidefinite, its eigenvalues are all
real and nonnegative, so the singular values are also real and nonnegative. The singular values σi
are usually sorted in decreasing order so that Σ = diag(σ1 , σ2 , . . . , σn ) with σ1 ≥ σ2 ≥ . . . ≥ σn ≥ 0.
The columns ui of U , the columns vi of V , and the singular values of A satisfy Avi = σi ui .
Every m × n matrix A of rank r has an SVD with exactly r nonzero singular values. Like the
QR decomposition, the SVD has two main forms.
• Full SVD: Denoted A = U ΣV H . U is m × m, V is n × n, and Σ is m × n. The first r columns
of U span R(A), and the remaining n − r columns span N (AH ). Likewise, the first r columns
of V span R(AH ), and the last m − r columns span N (A).
• Compact (Reduced) SVD: Denoted A = U1 Σ1 V1H . U1 is m × r (the first r columns of U ),
V1 is n × r (the first r columns of V ), and Σ1 is r × r (the first r × r block of Σ). This smaller
version of the SVD has all of the information needed to construct A and nothing more. The
zero singular values and the correpsonding columns of U and V are neglected.

U1 (m × r) Σ1 (r × r) V1H (r × n)

σ1 v1H
     
.. ..
. .
     
     
     
   σr   vrH 
 u1 ··· ur ur+1 ··· um    
H

   0   vr+1 
.. ..
     
.
     
     . 
0 vnH

U (m × m) Σ (m × n) V H (n × n)

71
72 Lab 7. The SVD and Image Compression

Finally, the SVD yields an outer product expansion of A in terms of the singular values and the
columns of U and V ,
Xr
A= σi ui viH . (7.1)
i=1
Note that only terms from the compact SVD are needed for this expansion.

Computing the Compact SVD

It is difficult to compute the SVD from scratch because it is an eigenvalue-based decomposition.
However, given an eigenvalue solver such as scipy.linalg.eig(), the algorithm becomes much
simpler. First, obtain the eigenvalues and eigenvectors of AH A, and use these to compute Σ. Since
AH A is normal, it has an orthonormal eigenbasis, so set the columns of V to be the eigenvectors of
AH A. Then, since Avi = σi ui , construct U by setting its columns to be ui = σ1i Avi .
The key is to sort the singular values and the corresponding eigenvectors in the same manner.
In addition, it is computationally inefficient to keep track of the entire matrix Σ since it is a matrix
of mostly zeros, so we need only store the singular values as a vector σ. The entire procedure for
computing the compact SVD is given below.

Algorithm 1
1: procedure compact_SVD(A)

√ eig(A A) ▷ Calculate the eigenvalues and eigenvectors of AH A.

H
2: λ, V ←
3: σ← λ ▷ Calculate the singular values of A.
4: σ ← sort(σ) ▷ Sort the singular values from greatest to least.
5: V ← sort(V ) ▷ Sort the eigenvectors the same way as in the previous step.
6: r ← count(σ ̸= 0) ▷ Count the number of nonzero singular values (the rank of A).
7: σ 1 ← σ :r ▷ Keep only the positive singular values.
8: V1 ← V:,:r ▷ Keep only the corresponding eigenvectors.
9: U1 ← AV1 /σ 1 ▷ Construct U with array broadcasting.
H
10: return U1 , σ 1 , V1

Problem 1. Write a function that accepts a matrix A and a small error tolerance tol. Use
Algorithm 1 to compute the compact SVD of A. In step 6, compute r by counting the number
of singular values that are greater than tol.
Consider the following tips for implementing the algorithm.

• The Hermitian AH can be computed with A.conj().T.

• In step 4, the way that σ is sorted needs to be stored so that the columns of V can be
sorted the same way. Consider using np.argsort() and fancy indexing to do this, but
remember that by default it sorts from least to greatest (not greatest to least).

• Step 9 can be done by looping over the columns of V , but it can be done more easily and
efficiently with array broadcasting.

Test your function by calculating the compact SVD for random matrices. Verify that U
and V are orthonormal, that U ΣV H = A, and that the number of nonzero singular values is
the rank of A. You may also want to compre your results to SciPy’s SVD algorithm.
 73

>>> import numpy as np

>>> from scipy import linalg as la

# Generate a random matrix and get its compact SVD via SciPy.
>>> A = np.random.random((10,5))
>>> U,s,Vh = la.svd(A, full_matrices=False)
>>> print(U.shape, s.shape, Vh.shape)
(10, 5) (5,) (5, 5)

# Verify that U is orthonormal, U Sigma Vh = A, and the rank is correct.

>>> np.allclose(U.T @ U, np.identity(5))
True
>>> np.allclose(U @ np.diag(s) @ Vh, A)
True
>>> np.linalg.matrix_rank(A) == len(s)
True

Visualizing the SVD

An m × n matrix A defines a linear transformation that sends points from Rn to Rm . The SVD
decomposes a matrix into two rotations and a scaling, so that any linear transformation can be easily
described geometrically. Specifically, V H represents a rotation, Σ a rescaling along the principal axes,
and U another rotation.

Problem 2. Write a function that accepts a 2 × 2 matrix A. Generate a 2 × 200 matrix S

representing a set of 200 points on the unit circle, with x-coordinates on the top row and y-
coordinates on the bottom row (recall the equation for the unit circle in polar coordinates:
x = cos(θ), y = sin(θ), θ ∈ [0, 2π]). Also define the matrix
 
  1 0 0
E = e1 0 e2 = ,
0 0 1

so that plotting the first row of S against the second row of S displays the unit circle, and
plotting the first row of E against its second row displays the standard basis vectors in R2 .
Compute the full SVD A = U ΣV H using scipy.linalg.svd(). Plot four subplots to
demonstrate each step of the transformation, plotting S and E, V H S and V H E, ΣV H S and
ΣV H E, then U ΣV H S and U ΣV H E.
For the matrix  
3 1
A= ,
1 3
your function should produce Figure 7.1.
(Hint: Use plt.axis("equal") to fix the aspect ratio so that the circles don’t appear elliptical.)
74 Lab 7. The SVD and Image Compression

1.0 1.0

0.5 0.5

0.0 0.0

0.5 0.5

1.0 1.0
1.0 0.5 0.0 0.5 1.0 1.0 0.5 0.0 0.5 1.0
(a) S (b) V H S

4 4

2 2

0 0

2 2

4 4
4 2 0 2 4 4 2 0 2 4
(c) ΣV H S (d) U ΣV H S

Figure 7.1: Each step in transforming the unit circle and two unit vectors using the matrix A.

Using the SVD for Data Compression

Low-Rank Matrix Approximations

If A is a m × n matrix of rank r < min{m, n}, then the compact SVD offers a way to store A with
less memory. Instead of storing all mn values of A, storing the matrices U1 , Σ1 and V1 only requires
saving a total of mr + r + nr values. For example, if A is 100 × 200 and has rank 20, then A has
20, 000 values, but its compact SVD only has total 6, 020 entries, a significant decrease.
The truncated SVD is an approximation to the compact SVD that allows even greater efficiency
at the cost of a little accuracy. Instead of keeping all of the nonzero singular values, the truncated
SVD only keeps the first s < r singular values, plus the corresponding columns of U and V . In this
case, (7.1) becomes
s
X
As = σi ui viH .
i=1

More precisely, the truncated SVD of A is As = U bΣb Vb H , where U

b is m × s, Vb is n × s, and Σ
b
is s × s. The resulting matrix As has rank s and is only an approximation to A, since r − s nonzero
singular values are neglected.
 75

b (m × s)
U b (s × s)
Σ Vb H (s × n)
σ1 v1H
     
.. ..
. .
     
     
     
   σs   vsH 
 u1 ··· us us+1 ··· ur    
H

   σs+1   vs+1 
.. ..
     
.
     
     . 
σr vrH
U1 (m × r) Σ1 (r × r) V1H (r × n)

The beauty of the SVD is that it makes it easy to select the information that is most important.
Larger singular values correspond to columns of U and V that contain more information, so dropping
the smallest singular values retains as much information as possible. In fact, given a matrix A, its
rank-s truncated SVD approximation As is the best rank s approximation of A with respect to both
the induced 2-norm and the Frobenius norm. This result is called the Schmidt, Mirsky, Eckhart-Young
theorem, a very significant concept that appears in signal processing, statistics, machine learning,
semantic indexing (search engines), and control theory.

Problem 3. Write a function that accepts a matrix A and a positive integer s.

1. Use your function from Problem 1 or scipy.linalg.svd() to compute the compact SVD
of A, then form the truncated SVD by stripping off the appropriate columns and entries
from U1 , Σ1 , and V1 . Return the best rank s approximation As of A (with respect to the
induced 2-norm and Frobenius norm).

2. Also return the number of entries required to store the truncated form U b Vb H (where Σ
bΣ b
is stored as a one-dimensional array, not the full diagonal matrix). The number of entries
stored in NumPy array can be accessed by its size attribute.

>>> A = np.random.random((20, 20))

>>> A.size
400

3. If s is greater than the number of nonzero singular values of A (meaning s > rank(A)),
raise a ValueError.

Use np.linalg.matrix_rank() to verify the rank of your approximation.

Unit Test
Write a unit test for Problem 3 to check your low rank SVD approximation. The unit test
can be found in the file test_svd_image_compression.py, and you will edit the problem
test_svd_approx.
There is an example unit test for Problem 1 to help you make your unit test.
76 Lab 7. The SVD and Image Compression

Error of Low-Rank Approximations

Another result of the Schmidt, Mirsky, Eckhart-Young theorem is that the exact 2-norm error of the
best rank-s approximation As for the matrix A is the (s + 1)th singular value of A:

∥A − As ∥2 = σs+1 . (7.2)

This offers a way to approximate A within a desired error tolerance ε: choose s such that σs+1 is the
largest singular value that is less than ε, then compute As . This As throws away as much information
as possible without violating the property ∥A − As ∥2 < ε.

Problem 4. Write a function that accepts a matrix A and an error tolerance ε.

1. Compute the compact SVD of A, then use (7.2) to compute the lowest rank approximation
As of A with 2-norm error less than ε. Avoid calculating the SVD more than once.
(Hint: np.argmax(), np.where(), and/or fancy indexing may be useful.)

2. As in the previous problem, also return the number of entries needed to store the resulting
approximation As via the truncated SVD.

3. If ε is less than or equal to the smallest singular value of A, raise a ValueError; in this
case, A cannot be approximated within the tolerance by a matrix of lesser rank.

This function should be close to identical to the function from Problem 3, but with the extra
step of identifying the appropriate s. Construct test cases to validate that ∥A − As ∥2 < ε.

Image Compression

Images are stored on a computer as matrices of pixel values. Sending an image over the internet or
a text message can be expensive, but computing and sending a low-rank SVD approximation of the
image can considerably reduce the amount of data sent while retaining a high level of image detail.
Successive levels of detail can be sent after the inital low-rank approximation by sending additional
singular values and the corresponding columns of V and U.

Examining the singular values of an image gives us an idea of how low-rank the approximation
can be. Figure 7.2 shows the image in hubble_gray.jpg and a log plot of its singular values. The
plot in 7.2b is typical for a photograph—the singular values start out large but drop off rapidly.
In this rank 1041 image, 913 of the singular values are 100 or more times smaller than the largest
singular value. By discarding these relatively small singular values, we can retain all but the finest
image details, while storing only a rank 128 image. This is a huge reduction in data size.
 77

105
104

Singular values
103
102
101
100
0 200 400 600 800 1000
Matrix rank
(a) NGC 3603 (Hubble Space Telescope). (b) Singular values on a log scale.

Figure 7.2

Figure 7.3 shows several low-rank approximations of the image in Figure 7.2a. Even at a low
rank the image is recognizable. By rank 120, the approximation differs very little from the original.

(a) Rank 2 (b) Rank 20 (c) Rank 120

Figure 7.3

Grayscale images are stored on a computer as 2-dimensional arrays, while color images are
stored as 3-dimensional arrays—one layer each for red, green, and blue arrays. To read and display
images, use imageio.imread() and plt.imshow(). Images are read in as integer arrays with entries
between 0 and 255 (dtype=np.uint8), but plt.imshow() works better if the image is an array of
floats in the interval [0, 1]. Scale the image properly by dividing the array by 255.

>>> from imageio import imread

>>> from matplotlib import pyplot as plt

# Send the RGB values to the interval (0,1).

>>> image_gray = imread("hubble_gray.jpg") / 255.
>>> image_gray.shape # Grayscale images are 2-d arrays.
(1158, 1041)
>>> image_color = imread("hubble.jpg") / 255.
78 Lab 7. The SVD and Image Compression

>>> image_color.shape # Color images are 3-d arrays.

(1158, 1041, 3)

# The final axis has 3 layers for red, green, and blue values.
>>> red_layer = image_color[:,:,0]
>>> red_layer.shape
(1158, 1041)

# Display a gray image.

>>> plt.imshow(red_layer, cmap="gray")
>>> plt.axis("off") # Turn off axis ticks and labels.
>>> plt.show()

# Display a color image.

>>> plt.imshow(image_color) # cmap=None by default.
>>> plt.axis("off")
>>> plt.show()

Problem 5. Write a function that accepts the name of an image file and an integer s. Use
your function from Problem 3, to compute the best rank-s approximation of the image. Plot
the original image and the approximation in separate subplots. In the figure title, report the
difference in number of entries required to store the original image and the approximation (use
plt.suptitle()).
Your function should be able to handle both grayscale and color images. Read the image
in and check its dimensions to see if it is color or not. Grayscale images can be approximated
directly since they are represented by 2-dimensional arrays. For color images, let R, G, and B
be the matrices for the red, green, and blue layers of the image, respectively. Calculate the low-
rank approximations Rs , Gs , and Bs separately, then put them together in a new 3-dimensional
array of the same shape as the original image.
(Hint: np.dstack() may be useful for putting the color layers back together.)
Finally, it is possible for the low-rank approximations to have values slightly outside the
valid range of RGB values. Set any values outside of the interval [0, 1] to the closer of the two
boundary values.
(Hint: fancy indexing and/or np.clip() may be useful here.)
To check, compressing hubble_gray.jpg with a rank 20 approximation should appear
similar to Figure 7.3b and save 1, 161, 478 matrix entries.
 79

Additional Material
More on Computing the SVD
For an m × n matrix A of rank r < min{m, n}, the compact SVD of A neglects last m − r columns
of U and the last n − r columns of V . The remaining columns of each matrix can be calculated by
using Gram-Schmidt orthonormalization. If m < r < n or n < r < m, only one of U1 and V1 will
need to be filled in to construct the full U or V . Computing these extra columns is one way to obtain
a basis for N (AH ) or N (A).
Algorithm 1 begins with the assumption that we have a way to compute the eigenvalues and
eigenvectors of AH A. Computing eigenvalues is a notoriously difficult problem, and computing the
SVD from scratch without an eigenvalue solver is much more difficult than the routine described by
Algorithm 1. The procedure involves two phases:
1. Factor A into A = Ua BVaH where B is bidiagonal (only nonzero on the diagonal and the first
superdiagonal) and Ua and Va are orthonormal. This is usually done via Golub-Kahan Bidi-
agonalization, which uses Householder reflections, or Lawson-Hanson-Chan bidiagonalization,
which relies on the QR decomposition.
2. Factor B into B = Ub ΣVbH by the QR algorithm or a divide-and-conquer algorithm. Then the
SVD of A is given by A = (Ua Ub )Σ(Va Vb )H .
For more details, see Lecture 31 of [TB97] or Section 5.4 of Applied Numerical Linear Algebra by
James W. Demmel.

Animating Images with Matplotlib

Matplotlib can be used to animate images that change over time. For instance, we can show how
the low-rank approximations of an image change as the rank s increases, showing how the image is
recovered as more ranks are added. Try using the following code to create such an animation.

from matplotlib import pyplot as plt

from matplotlib.animation import FuncAnimation

def animate_images(images):
"""Animate a sequence of images. The input is a list where each
entry is an array that will be one frame of the animation.
"""
fig = plt.figure()
plt.axis("off")
im = plt.imshow(images[0], animated=True)

def update(index):
plt.title("Rank {} Approximation".format(index))
im.set_array(images[index])
return im, # Note the comma!

a = FuncAnimation(fig, update, frames=len(images), blit=True)

plt.show()

See https://matplotlib.org/examples/animation/dynamic_image.html for another example.

80 Lab 7. The SVD and Image Compression
8
Facial Recognition

Lab Objective: Facial recognition algorithms attempt to match a person’s portrait to a database
of many portraits. Facial recognition is becoming increasingly important in security, law enforcement,
artificial intelligence, and other areas. Though humans can easily match pictures to people, computers
are beginning to surpass humans at facial recognition. In this lab, we implement a basic facial
recognition system that relies on eigenvectors and the SVD to efficiently determine the difference
between faces.

Preparing an Image Database

The faces94 face image dataset1 contains several photographs of 153 people, organized into folders
by person. To perform facial recognition on this dataset, select one image per person and convert
these images into a database. For this particular facial recognition algorithm, the entire database
can be stored in just a few NumPy arrays.
Digital images are stored on computers as arrays of pixels. Therefore, an m × n image can be
stored in memory as an m × n matrix or, equivalently, as an mn-vector by concatenating the rows
of the matrix. Then a collection of k images can be stored as a single mn × k matrix F , where each
column of F represents a single image. That is, if
 

F =  f1 f2 ··· fk  ,

then each fi is a mn-vector representing a single image.

The following function obtains one image for each person in the faces94 dataset and converts
the collection of images into an mn × k matrix F described above.

import os
import numpy as np
from imageio import imread

def get_faces(path="./faces94"):
# Traverse the directory and get one image per subdirectory.
1 See http://cswww.essex.ac.uk/mv/allfaces/faces94.html.

81
82 Lab 8. Facial Recognition

faces = []
for (dirpath, dirnames, filenames) in os.walk(path):
for fname in filenames:
if fname[-3:]=="jpg": # Only get jpg images.
# Load the image, convert it to grayscale,
# and flatten it into a vector.
faces.append(np.ravel(imread(dirpath+"/"+fname, as_gray=True)))
break
# Put all the face vectors column-wise into a matrix.
return np.transpose(faces)

Problem 1. Write a function that accepts an image as a flattened mn-vector, along with its
original dimensions m and n. Use np.reshape() to convert the flattened image into its original
m × n shape and display the result with plt.imshow().
(Hint: use cmap="gray" in plt.imshow() to display images in grayscale.)
Unzip the faces94.zip archive and use get_faces() to construct F . Each faces94
image is 200 × 180, and there are 153 people in the dataset, so F should be 36000 × 153. Use
your function to display one of the images stored in F .

The Eigenfaces Method

With the image database F , we could construct a simple facial recognition system with the following
strategy. Let g be an mn-vector representing an unknown face that is not part of the database F .
Then the fi that minimizes ∥g−fi ∥2 is the matching face. Unfortunately, computing ∥g−fi ∥2 for each
i is very computationally expensive, especially if the images are high-resolution and/or the database
contains a large number of images. The eigenfaces method is a way to reduce the computational cost
of finding the closest matching face by focusing on only the most important features of each face.
Because the method ignores less significant facial features, it is also usually more accurate than the
naïve method.
The first step of the algorithm is to shift the images by the mean face. Shifting a set of data by
the mean exaggerates the distinguishing features of each entry. In the context of facial recognition,
shifting by the mean accentuates the unique features of each face. For the images vectors stored in
F , the mean face µ is the defined to be the element-wise average of the fi :
k
1X
µ= fi .
k i=1

Hence, the ith mean-shifted face vector f̄i is given by

f̄i = fi − µ.

Next, define F̄ as the mn × k matrix whose columns are given by the mean-shifted face vectors,
 

F̄ =  f̄1 f̄2 ··· f̄k  .

 83

(a) The mean face. (b) An original face. (c) A mean-shifted face.

Figure 8.1

Problem 2. Write a class called FacialRec whose constructor accepts a path to a directory
of images. In the constructor, use get_faces() to construct F , then compute the mean face
µ and the shifted faces F̄ . Store each array as an attribute.
(Hint: Both µ and F̄ can be computed in a single line of code by using NumPy functions and/or
array broadcasting.)
Use your function from Problem 1 to visualize the mean face, and compare it to Figure
8.1a. Also display an original face and its corresponding mean-shifted face. Compare your
results with Figures 8.1b and 8.1c.

To increase computational efficiency and minimize storage, the face vectors can be represented
with fewer values by projecting F̄ onto a lower-dimensional subspace. Let s be a natural number
such that s < r, where r is the rank of F̄ . By projecting F̄ onto an s-dimensional subspace, each
face can be stored with only s values.
Specifically, let U ΣV H be the compact SVD of F̄ with rank r, which can also be represented by
r
X
F̄ = σi ui viH .
i=1

The first r columns of U form a basis for the range of F̄ . Recall that the Schmidt, Mirsky, Eckart-
Young Theorem states that the matrix
s
X
F̄s = σi ui viH
i=1

is the best rank-s approximation of F̄ for each s < r. This means that ∥F̄ − F¯s ∥ is minimized against
all other ∥F̄ − B∥ where B has rank s. As a consequence of this theorem, the first s columns of U
form a basis that provides the “best” s-dimensional subspace for approximating F̄ .
The s basis vectors u1 , . . . , us are are commonly called the eigenfaces because they are eigen-
vectors of F̄ F̄ T and because they resemble face images. Each original face image can be efficiently
represented in terms of these eigenfaces. See Figure 8.2 for visualizations of the some of the eigenfaces
for the facesd94 data set.
84 Lab 8. Facial Recognition

Figure 8.2: The first, 50th, and 100th eigenfaces.

In general, the lower eigenfaces provide a more general information of a face and higher-ordered
eigenfaces provide the details necessary to distinguish particular faces [MMH04]. These eigenfaces
will be used to construct the face images in the dataset. The more eigenfaces used, the more detailed
the resulting image will be.
Next, let Us be the matrix with the first s eigenfaces as columns. Since the eigenfaces {ui }si=1
form an orthonormal set, Us is an orthonormal matrix (independent of s) and hence UsT Us = I.
The matrix Ps = Us UsT projects vectors in Rmn to the subspace spanned by the orthonormal basis
{ui }si=1 , and the change of basis matrix UsT puts the projection in terms of the basis of eigenfaces.
Thus the projection b fi of f̄i in terms of the basis of eigenfaces is given by
fi = UsT Ps f̄i = UsT Us UsT f̄i = UsT f̄i .
b (8.1)
Note carefully that though the shifted image f¯i has mn entries, the projection b
fi has only s entries
since Us is mn × s. Likewise, the matrix Fb that has the projections b
fi as columns is s × k, and
Fb = UsT F. (8.2)

Problem 3. In the constructor of FacialRec, calculate the compact SVD of F̄ and save the
matrix U as an attribute. Compare the computed eigenfaces (the columns of U ) to Figure 8.2.
Also write a method that accepts a vector of length mn or an mn × ℓ matrix, as well as
an integer s < mn. Construct Us by taking the first s columns of U , then use (8.1) or (8.2) to
calculate the projection of the input vector or matrix onto the span of the first s eigenfaces.
(Hint: this method should be implemented with a single line of code.)

Reducing the mean-shifted face image f̄i to the lower-dimensional projection b fi drastically re-
duces the computational cost of the facial recognition algorithm, but this efficiency gain comes at
a price. A projection image only approximates the corresponding original image, but as long as s
isn’t too small, the approximation is usually good enough for the algorithm to work well. Before
completing the facial recognition system, we reconstruct some of these projections to visualize the
amount of information lost.
From (8.1), since UsT projects f̄i and performs a change of basis to get b
fi , its transpose Us puts
fi back into the original basis with as little error as possible. That is,
b

fi ≈ f̄i = fi − µ,
Usb
 85

so that we have the approximation

fi + µ ≈ fi .
fi = Usb
e (8.3)
This e
fi is called the reconstruction of fi .

(a) A reconstruction with s = 5. (b) A reconstruction with s = 19. (c) A reconstruction with s = 75.

Figure 8.3: An image rebuilt with various numbers of eigenfaces. The image is already recognizable
when it is reconstructed with only 19 eigenfaces, less than an eighth of the 153 eigenfaces correspond-
ing to nonzero eigenvalues or F̄ F̄ T . Note the similarities between this method and regular image
compression via the truncated SVD.

Problem 4. Instantiate a FacialRec object that draws from the faces94 dataset. Select one
of the shifted images f̄i . For at least 4 values of s, use your method from Problem 3 to compute
the corresponding s-projection b fi , then use (8.3) to compute the reconstruction e
fi . Display the
various reconstructions and the original image. Compare your results to Figure 8.3

Matching Faces
Let g be a vector representing an unknown face that is not part of the database. We determine which
image in the database is most like g by comparing g b to each of the b
fi . First, shift g by the mean to
obtain ḡ, then project ḡ using a given number of eigenfaces:

b = UsT ḡ = UsT (g − µ)
g (8.4)

Next, determine which b fi is closest to g

b. Since the columns of Us are an orthonormal basis,
the computation in this basis yields the same result as computing in the standard Euclidean basis
would. Then setting
j = argmin ∥bfi − g
b∥2 , (8.5)
i

we have that the jth face image fj is the best match for g. Again, since b
fi and g
bi only have s entries,
the computation in (8.5) is much cheaper than comparing the raw fi to g.
86 Lab 8. Facial Recognition

Problem 5. Write a method for the FacialRec class that accepts an image vector g and an
integer s. Use your method from Problem 3 to compute Fb and gb for the given s, then use (8.5)
to determine the best matching face in the database. Return the index of the matching face.
(Hint: scipy.linalg.norm() and np.argmin() may be useful.)

Note
This facial recognition system works by solving a nearest neighbor search, since the goal is to
find the fi that is “nearest” to the input image g. Nearest neighbor searches can be performed
more efficiently with the use of a k-d tree, a binary search tree for storing vectors. The system
could also be called a k-neighbors classifier with k = 1.

Problem 6. Write a method for the FacialRec class that accepts an flat image vector g, an
integer s, and the original dimensions of g. Use your method from Problem 5 to find the index
j of the best matching face, then display the original face g alongside the best match fj .
The following generator yields random faces from faces94 that can be used as test cases.

def sample_faces(num_faces, path="./faces94"):

# Get the list of possible images.
files = []
for (dirpath, dirnames, filenames) in os.walk(path):
for fname in filenames:
if fname[-3:]=="jpg": # Only get jpg images.
files.append(dirpath+"/"+fname)

# Get a subset of the image names and yield the images one at a time.
test_files = np.random.choice(files, num_faces, replace=False)
for fname in test_files:
yield np.ravel(imread(fname, as_gray=True))

The yield keyword is like a return statement, but the next time the generator is called, it will
resume immediately after the last yield statement.a
Use sample_faces() to get at least 5 random faces from faces94, and match each random
face to the database with s = 38. Iterate through the random faces with the following syntax.

for test_image in sample_faces(5):

# 'test_image' is a now flattened face vector.

a See the Python Essentials lab on Profiling for more on generators.

Although there are other approaches to facial recognition that utilize more complex techniques,
the method of eigenfaces remains a wonderfully simple and effective solution.
 87

Additional Material
Improvements on the Facial Recognition System with Eigenfaces
The FacialRec class does its job well, but it could be improved in several ways. Here are a few ideas.

• The most computationally intensive part of the algorithm is computing Fb. Instead of recom-
puting Fb every time the method from Problem 5 is called, store Fb and s as attributes the first
time the method is called. In subsequent calls, only recompute Fb if the user specifies a different
value for s.

• Load a scipy.spatial.KDTree object with Fb and use its query() method to compute (8.5).
Building a kd-tree is expensive, so be sure to only build a new tree when necessary (i.e., the
user specifies a new value for s).

• Include an error tolerance ε in the method for Problem 5. If ∥fj − g∥ > ε, print a message or
raise an exception to indicate that there is no suitable match for g in the database. In this
case, add g to the database for future reference.

• Generalize the system by turning it into a k-neighbors classifier. In the constructor, add several
faces per person to the database (this requires modifying get_faces()). Assign each individual
a unique ID so that the system knows which faces correspond to the same person. Modify the
method from Problem 5 so that it also accepts an integer k, then use scipy.spatial.KDTree
to find the k nearest images to g. Choose the ID that belongs to the most nearest neighbors,
then return an index that corresponds to an individual with that ID.
In other words, choose the k faces fi that give the smallest values of ∥fi − g
b∥2 . These faces then
get to vote on which person g belongs to.

• Improve the user interface of the class by modifying the method from Problem 6 so that it
accepts a file name to read from instead of an array. A few lines of code from get_faces() or
sample_faces() might be helpful for this.

Other Methods for Facial Recognition

The method of facial recognition presented here is more formally called principal component analysis
(PCA) using eigenfaces. Several other machine learning and optimization techniques, such as linear
discriminant analysis (LDA), elastic matching, dynamic link matching, and hidden Markov models
(HMMs) have also been applied to the facial recognition problem. Other techniques focus on getting
better information about the faces in the first place, the most prevalent being 3-dimensional recog-
nition and thermal imaging. See https://en.wikipedia.org/wiki/Facial_recognition_system
for a good survey of different approaches to the facial recognition problem.
88 Lab 8. Facial Recognition
9
Differentiation

Lab Objective: Derivatives are central in many applications. Depending on the application and
on the available information, the derivative may be calculated symbolically, numerically, or with
differentiation software. In this lab we explore these three ways to take a derivative, discuss what
settings they are each appropriate for, and demonstrate their strengths and weaknesses.

Symbolic Differentiation
The derivative of a known mathematical function can be calculated symbolically with SymPy. This
method is the most precise way to take a derivative, but it is computationally expensive and requires
knowing the closed form formula of the function. Use sy.diff() to take a symbolic derivative.

>>> import sympy as sy

>>> x = sy.symbols('x')
>>> sy.diff(x**3 + x, x) # Differentiate x^3 + x with respect to x.
3*x**2 + 1

Problem 1. Write a function that defines f (x) = (sin(x) + 1)sin(cos(x)) and takes its symbolic
derivative with respect to x using SymPy. Lambdify the resulting function so that it can accept
NumPy arrays and return the resulting function handle.
Hint: You can test your function by plotting f and its derivative f ′ over the domain [−π, π]. It
may be helpful to move the bottom spine to 0 so you can see where the derivative crosses the
x-axis. Note: Do NOT include this in the final code for this problem.

>>> from matplotlib import pyplot as plt

>>> ax = plt.gca()
>>> ax.spines["bottom"].set_position("zero")

89
90 Lab 9. Differentiation

Numerical Differentiation
One definition for the derivative of a function f : R → R at a point x0 is

f (x0 + h) − f (x0 )
f ′ (x0 ) = lim .
h→0 h

Since this definition relies on h approaching 0, choosing a small, fixed value for h approximates f ′ (x0 ):

f (x0 + h) − f (x0 )
f ′ (x0 ) ≈ . (9.1)
h
This approximation is called the first order forward difference quotient. Using the points x0 and
x0 − h in place of x0 + h and x0 , respectively, results in the first order backward difference quotient,

f (x0 ) − f (x0 − h)
f ′ (x0 ) ≈ . (9.2)
h

Forward difference quotients use values of f at x0 and points greater than x0 , while backward
difference quotients use the values of f at x0 and points less than x0 . A centered difference quotient
uses points on either side of x0 , and typically results in a better approximation than the one-sided
quotients. Combining (9.1) and (9.2) yields the second order centered difference quotient,

1 ′ 1 f (x0 + h) − f (x0 ) f (x0 ) − f (x0 − h) f (x0 + h) − f (x0 − h)

f ′ (x0 ) = f (x0 ) + f ′ (x0 ) ≈ + = .
2 2 2h 2h 2h

x̄ x̄ + h x̂ − h x̂ x̂ + h x̃ − h x̃

* * * * * * * * * * *

f ′ (x̄) f ′ (x̂) f ′ (x̃)

Forward Difference Centered Difference Backward Difference

Figure 9.1

Note
The finite difference quotients in this section all approximate the first derivative of a function.
The terms first order and second order refers to how quickly the approximation converges on
the actual value of f ′ (x0 ) as h approaches 0, not to how many derivatives are being taken.
There are finite difference quotients for approximating higher order derivatives, such as
f ′′ or f ′′′ . For example, the centered difference quotient

f (x0 − h) − 2f (x0 ) + f (x0 + h)

f ′′ (x0 ) ≈
h2
approximates the second derivative. This particular quotient is important for finite difference
methods that approximate numerical solutions to some partial differential equations.
 91

While we do not derive them here, there are other finite difference quotients that use more
points to approximate the derivative, some of which are listed below. Using more points generally
results in better convergence properties.

Type Order Formula

f (x0 +h)−f (x0 )

1 h
Forward

−3f (x0 )+4f (x0 +h)−f (x0 +2h)

2 2h

f (x0 )−f (x0 −h)

1 h
Backward

3f (x0 )−4f (x0 −h)+f (x0 −2h)

2 2h

f (x0 +h)−f (x0 −h)

2 2h
Centered

f (x0 −2h)−8f (x0 −h)+8f (x0 +h)−f (x0 +2h)

4 12h

Table 9.1: Common finite difference quotients for approximating f ′ (x0 ).

Problem 2. Write a function for each of the finite difference quotients listed in Table 9.1. Each
function should accept a function handle f , an array of points x, and a float h; each should
return an array of the difference quotients evaluated at each point in x.
To test your functions, approximate the derivative of f (x) = (sin(x) + 1)sin(cos(x)) at each
point of a domain over [−π, π]. Plot the results and compare them to the results of Problem 1.

Convergence of Finite Difference Quotients

Finite difference quotients are typically derived using Taylor’s formula. This method also shows how
the accuracy of the approximation increases as h → 0:
f (x0 + h) − f (x0 ) R2 (h)
f (x0 + h) = f (x0 ) + f ′ (x0 )h + R2 (h) =⇒ − f ′ (x0 ) = , (9.3)
h h
R1
where R2 (h) = h2 0
(1 − t)f ′′ (x0 + th) dt. Thus the absolute error of the first order forward difference
quotient is
Z 1 Z 1
R2 (h)
= |h| (1 − t)f ′′ (x0 + th) dt ≤ |h| |1 − t||f ′′ (x0 + th)| dt.
h 0 0
92 Lab 9. Differentiation

If f ′′ is continuous, then for any δ > 0, setting M = supx∈(x0 −δ,x0 +δ) f ′′ (x) guarantees that
Z 1
R2 (h)
≤ |h| M dt = M |h| ∈ O(h).
h 0

whenever |h| < δ. That is, the error decreases at the same rate as h. If h gets twice as small, the error
does as well. This is what is meant by a first order approximation. In a second order approximation,
the absolute error is O(h2 ), meaning that if h gets twice as small, the error gets four times smaller.

Note
The notation O(f (n)) is commonly used to describe the temporal or spatial complexity of an
algorithm. In that context, a O(n2 ) algorithm is much worse than a O(n) algorithm. However,
when referring to error, a O(h2 ) algorithm is better than a O(h) algorithm because it means
that the accuracy improves faster as h decreases.

Problem 3. Write a function that accepts a point x0 at which to compute the derivative of
f (x) = (sin(x) + 1)sin(cos(x)) . Use your function from Problem 1 to compute the exact value of
f ′ (x0 ). Then use each your functions from Problem 2 to get an approximate derivative f˜′ (x0 )
for h = 10−8 , 10−7 , . . . , 10−1 , 1. Track the absolute error |f ′ (x0 ) − f˜′ (x0 )| for each trial, then
plot the absolute error against h on a log-log scale (use plt.loglog()).
Instead of using np.linspace() to create an array of h values, use np.logspace(). This
function generates logarithmically spaced values between two powers of 10.

>>> import numpy as np

>>> np.logspace(-3, 0, 4) # Get 4 values from 1e-3 to 1e0.
array([ 0.001, 0.01 , 0.1 , 1. ])

For x0 = 1, your plot should resemble the following figure.

100 Order 1 Forward

Order 2 Forward
10 2 Order 1 Backward
Order 2 Backward
10 4 Order 2 Centered
Absolute Error

Order 4 Centered
10 6
10 8
10 10
10 12
10 8 10 7 10 6 10 5 10 4 10 3 10 2 10 1 100
h
 93

Achtung!
Mathematically, choosing smaller h values results in tighter approximations of f ′ (x0 ). However,
Problem 3 shows that when h gets too small, the error stops decreasing. This numerical error
is due to the denominator in each finite difference quotient becoming very small. The optimal
value of h is usually one that is small, but not too small.

Problem 4. The radar stations A and B, separated by the distance a = 500 m, track a plane
C by recording the angles α and β at one-second intervals. Your goal, back at air traffic control,
is to determine the speed of the plane.a

Let the position of the plane at time t be givenp by (x(t), y(t)). The speed at time t is the
magnitude of the velocity vector, ∥ dt d
(x(t), y(t))∥ = x′ (t)2 + y ′ (t)2 . The closed forms of the
functions x(t) and y(t) are unknown (and may not exist at all), but we can still use numerical
methods to estimate x′ (t) and y ′ (t). For example, at t = 3, the second order centered difference
quotient for x′ (t) is
x(3 + h) − x(3 − h) 1
x′ (3) ≈ = (x(4) − x(2)).
2h 2
In this case h = 1 since data comes in from the radar stations at 1 second intervals.
Successive readings for α and β at integer times t = 7, 8, . . . , 14 are stored in the file
plane.npy. Each row in the array represents a different reading; the columns are the observation
time t, the angle α (in degrees), and the angle β (also in degrees), in that order. The Cartesian
coordinates of the plane can be calculated from the angles α and β as follows.

tan(β) tan(β) tan(α)

x(α, β) = a y(α, β) = a (9.4)
tan(β) − tan(α) tan(β) − tan(α)
94 Lab 9. Differentiation

Load the data, convert α and β to radians, then compute the coordinates x(t) and y(t) at
each given t using 9.4. Approximate x′ (t) and y ′ (t) using a first order forward difference
quotient for t = 7, a first order backward difference quotient for t = 14, and a second order
centered difference quotient for t = 8, 9, . . . , 13 (see Figure 9.1). Return the values of the speed
x′ (t)2 + y ′ (t)2 at each t.
p

(Hint: np.deg2rad() will be helpful.)

a This problem is adapted from an exercise in [Kiu13].

Numerical Differentiation in Higher Dimensions

Finite difference quotients can also be used to approximate derivatives in higher dimensions. The
Jacobian matrix of a function f : Rn → Rm at a point x0 ∈ Rn is the m × n matrix J whose entries
are given by
∂fi
Jij = (x0 ).
∂xj

For example, the Jacobian for a function f : R3 → R2 is defined by

 
∂f1 ∂f1 ∂f1  
∂x1 ∂x2 ∂x3
  x1
h i f1 (x)
J= ∂f ∂f ∂f
= , where f (x) = , x =  x2  .
 
∂x1 ∂x2 ∂x3 f2 (x)
∂f2 ∂f2 ∂f2
∂x1 ∂x2 ∂x3
x3

The difference quotients in this case resemble directional derivatives. The first order forward
difference quotient for approximating a partial derivative is

∂f f (x0 + hej ) − f (x0 )

(x0 ) ≈ ,
∂xj h

where ej is the jth standard basis vector. The second order centered difference approximation is

∂f f (x0 + hej ) − f (x0 − hej )

(x0 ) ≈ . (9.5)
∂xj 2h

Problem 5. Write a function that accepts a function f : Rn → Rm , a point x0 ∈ Rn , and a

float h. Approximate the Jacobian matrix of f at x using the second order centered difference
quotient in (9.5).
(Hint: the standard basis vector ej is the jth column of the n × n identity matrix I.)
To test your function, define a simple function like f (x, y) = [x2 , x3 − y]T where the
Jacobian is easy to find analytically, then check the results of your function against SymPy or
your own scratch work.
 95

Differentiation Software
Many machine learning algorithms and structures, especially neural networks, rely on the gradient of a
cost or objective function. To facilitate their research, several organizations have recently developed
Python packages for numerical differentiation. For example, the Harvard Intelligent Probabilistic
Systems Group (HIPS) started developing autograd in 2014 (https://github.com/HIPS/autograd)
and Google created JAX (https://github.com/google/jax) as a successor to autograd. Popular
deep learning libraries also contain automatic differentiation libraries. These tools use an algorithm
known as automatic differentiation that is incredibly robust: they can differentiate functions with
NumPy routines, if statements, while loops, and even recursion.
We conclude with a brief introduction to JAX. It can be installed as follows on Mac and Linux:

pip install "jax[cpu]"

Installation directly via pip is not currently supported on Windows, however. Some unofficial builds
for Windows are available at https://github.com/cloudhan/jax-windows-builder. JAX also has
additional installation options that allow it to do computations on a GPU using the CUDA library.
See https://github.com/google/jax#installation for additional options for these cases.
JAX’s grad() accepts a scalar-valued function and returns its gradient as a function that
accepts the same parameters as the original. To support most of the NumPy features, JAX comes
with its own thinly-wrapped version of Numpy, jax.numpy. Import this version of NumPy as jnp to
avoid confusion.

>>> from jax import numpy as jnp # Use JAX's version of NumPy.
>>> from jax import grad

>>> g = lambda x: jnp.exp(jnp.sin(jnp.cos(x)))

>>> dg = grad(g) # dg() is a callable function.
>>> dg(1.) # Use floats as input, not ints.
DeviceArray(-1.2069776, dtype=float32, weak_type=True)

Functions that grad() produces do not support array broadcasting, meaning they do not accept
arrays as input. The easiest way to create a function is to use jnp.vectorize() on the derivative.

>>> pts = jnp.array([1, 2, 3], dtype=float)

>>> dg = jnp.vectorize(grad(g)) # Calculate g'(x) with array support.
>>> dg(pts) # Evaluate g'(x) at each of the points.
DeviceArray([-1.2069776 , -0.5551414 , -0.03356146], dtype=float32)

SymPy would have no trouble differentiating g(x) in these examples. However, JAX can also
differentiate Python functions that look nothing like traditional mathematical functions. For exam-
ple, the following code computes the Taylor series of ex with a loop.

>>> from sympy import factorial

>>> def taylor_exp(x, tol=.0001):

... """Compute the Taylor series of e^x with terms greater than tol."""
... result, i, term = 0, 0, 1
... while jnp.abs(term) > tol:
96 Lab 9. Differentiation

... term = x**i / int(factorial(i))

... result, i = result + term, i + 1
... return result
...
>>> d_exp = grad(taylor_exp)
>>> d_exp(2., .1), d_exp(2., .0001)
(DeviceArray(7.266667, dtype=float32, weak_type=True),
DeviceArray(7.3889947, dtype=float32, weak_type=True))

Problem 6. The Chebyshev Polynomials satisfy the recursive relation

T0 (x) = 1, T1 (x) = x, Tn (x) = 2xTn−1 (x) − Tn−2 (x).

Write a function that accepts an array x and an integer n and recursively computes Tn (x). Use
JAX and your first function to create a function for Tn′ (x). Use this last function to plot each
Tn′ (x) over the domain [−1, 1] for n = 0, 1, 2, 3, 4.
(Hint: Use jnp.ones_like(x) to handle the case when n = 0.)

Problem 7. Let f (x) = (sin(x) + 1)sin(cos(x)) as in Problems 1 and 3. Write a function that
accepts an integer N and performs the following experiment N times.

1. Choose a random value x0 .

2. Use your function from Problem 1 to calculate the “exact” value of f ′ (x0 ). Time how long
the entire process takes, including calling your function (each iteration).

3. Time how long it takes to get an approximation f˜′ (x0 ) of f ′ (x0 ) using the fourth-order
centered difference quotient from Problem 3. Record the absolute error |f ′ (x0 ) − f˜′ (x0 )|
of the approximation.

4. Time how long it takes to get an approximation f¯′ (x0 ) of f ′ (x0 ) using JAX (calling grad()
every time). Record the absolute error |f ′ (x0 ) − f¯′ (x0 )| of the approximation.

Plot the computation times versus the absolute errors on a log-log plot with different
colors for SymPy, the difference quotient, and JAX. For SymPy, assume an absolute error of
1e-18 (since only positive values can be shown on a log plot).
For N = 200, your plot should resemble the following figure. Note that SymPy has the
least error but longer computation time, and that the difference quotient takes the least amount
of time but has the most error. JAX, on the other hand, does not appear to be as well-suited
to this particular problem. However, for more complicated functions and functions of multiple
variables, it tends to be a “happy medium” between the two, with faster runtime than SymPy.
 97

SymPy
10 4 Difference Quotients
JAX
10 7

Absolute Error 10 10

10 13

10 16

10 3 10 2
Computation Time (seconds)
Figure 9.2: Solution with N = 200.
98 Lab 9. Differentiation

Additional Material
More JAX
For scalar-valued functions with multiple inputs, the parameter argnums specifies the variable that
the derivative is computed with respect to. Providing a list for argnums gives several outputs.

>>> f = lambda x,y: 3*x*y + 2*y - x

# Take the derivative of f with respect to the first variable, x.

>>> dfdx = grad(f, argnums=0) # Should be dfdx(x,y) = 3y - 1,
>>> dfdx(5., 1.) # so dfdx(5,1) = 3 - 1 = 2.
DeviceArray(2., dtype=float32, weak_type=True)

# Take the gradient with respect to the second variable, y.

>>> dfdy = grad(f, argnums=1) # Should be dfdy(x,y) = 3x + 2,
>>> dfdy(5., 1.) # so dfdy(5,1) = 15 + 2 = 17.
DeviceArray(17., dtype=float32, weak_type=True)

# Get the full gradient.

>>> grad_f = grad(f, argnums=[0,1])
>>> jnp.array(grad_f(5., 1.))
DeviceArray([ 2., 17.], dtype=float32)

Finally, JAX’s jacobian() can differentiate vector-valued functions.

>>> from jax import jacobian

>>> f = lambda x: jnp.array([x[0]**2, x[0]+x[1]])

>>> f_jac = jacobian(f)
>>> f_jac(jnp.array([1., 1.]))
DeviceArray([[2., 0.],
[1., 1.]], dtype=float32)
10
Newton’s Method

Lab Objective: Newton’s method, the classical method for finding the zeros of a function, is
one of the most important algorithms of all time. In this lab we implement Newton’s method in
arbitrary dimensions and use it to solve a few interesting problems. We also explore in some detail
the convergence (or lack of convergence) of the method under various circumstances.

Iterative Methods
An iterative method is an algorithm that must be applied repeatedly to obtain a result. The general
idea behind any iterative method is to make an initial guess at the solution to a problem, apply a
few easy computations to better approximate the solution, use that approximation as the new initial
guess, and repeat until done. More precisely, let F be some function used to approximate the solution
to a problem. Starting with an initial guess of x0 , compute

xk+1 = F (xk ) (10.1)

for successive values of k to generate a sequence (xk )∞

k=0 that hopefully converges to the true solution.
If the terms of the sequence are vectors, they are denoted by xk .
In the best case, the iteration converges to the true solution x, written limk→∞ xk = x or
xk → x. In the worst case, the iteration continues forever without approaching the solution. In
practice, iterative methods require carefully chosen stopping criteria to guarantee that the algorithm
terminates at some point. The general approach is to continue iterating until the difference between
two consecutive approximations is sufficiently small, and to iterate no more than a specific number
of times. That is, choose a very small ε > 0 and an integer N ∈ N, and update the approximation
using (10.1) until either
|xk − xk−1 | < ε or k > N. (10.2)

The choices for ε and N are significant: a “large” ε (such as 10−6 ) produces a less accurate
result than a “small” ε (such 10−16 ), but demands less computations; a small N (10) also potentially
lowers accuracy, but detects and halts nonconvergent iterations sooner than a large N (10,000). In
code, ε and N are often named tol and maxiter, respectively (or similar).
While there are many ways to structure the code for an iterative method, probably the cleanest
way is to combine a for loop with a break statement. As a very simple example, let F (x) = x2 . This
method converges to x = 0 independent of starting point.

99
100 Lab 10. Newton’s Method

>>> F = lambda x: x / 2
>>> x0, tol, maxiter = 10, 1e-9, 8
>>> for k in range(maxiter): # Iterate at most N times.
... print(x0, end=' ')
... x1 = F(x0) # Compute the next iteration.
... if abs(x1 - x0) < tol: # Check for convergence.
... break # Upon convergence, stop iterating.
... x0 = x1 # Otherwise, continue iterating.
...
10 5.0 2.5 1.25 0.625 0.3125 0.15625 0.078125

In this example, the algorithm terminates after N = 8 iterations (the maximum number of
allowed iterations) because the tolerance condition |xk − xk−1 | < 10−9 is not met fast enough. If N
had been larger (say 40), the iteration would have quit early due to the tolerance condition.

Newton’s Method in One Dimension

Newton’s method is an iterative method for finding the zeros of a function. That is, if f : R → R,
the method attempts to find a x̄ such that f (x̄) = 0. Beginning with an initial guess x0 , calculate
successive approximations for x̄ with the recursive sequence

f (xk )
xk+1 = xk − . (10.3)
f ′ (xk )

The sequence converges to the zero x̄ of f if three conditions hold:

1. f and f ′ exist and are continuous,

2. f ′ (x̄) ̸= 0, and

3. x0 is “sufficiently close” to x̄.

In applications, the first two conditions usually hold. If x̄ and x0 are not “sufficiently close,” Newton’s
method may converge very slowly, or it may not converge at all. However, when all three conditions
hold, Newton’s method converges quadratically, meaning that the maximum error is squared at every
iteration. This is very quick convergence, making Newton’s method as powerful as it is simple.

Problem 1. Write a function that accepts a function f , an initial guess x0 , the derivative f ′ ,
a stopping tolerance defaulting to 10−5 , and a maximum number of iterations defaulting to 15.
Use Newton’s method as described in (10.3) to compute a zero x̄ of f . Terminate the algorithm
when |xk − xk−1 | is less than the stopping tolerance or after iterating the maximum number
of allowed times. Return the last computed approximation to x̄, a boolean value indicating
whether or not the algorithm converged, and the number of iterations completed.
Test your function against functions like f (x) = ex − 2 (see Figure 10.1) or f (x) = x4 − 3.
Check that the computed zero x̄ satisfies f (x̄) ≈ 0. Also consider comparing your function to
scipy.optimize.newton(), which accepts similar arguments.
 101

f(x) = ex 2 x0

x1
x3 x2
0.0 0.5 1.0 1.5 2.0

Figure 10.1: Newton’s method approximates the zero of a function (blue) by choosing as the next
approximation the x-intercept of the tangent line (red) that goes through the point (xk , f (xk )). In
this example, f (x) = ex − 2, which has a zero at x̄ = log(2). Setting x0 = 2 and using (10.3) to
e2 −2
iterate, we have x1 = x0 − ff′(x 0)
(x0 ) = 2 − e2 ≈ 1.2707. Similarly, x2 ≈ 0.8320, x3 ≈ .7024, and
x4 ≈ 0.6932. After only a few iterations, the zero log(2) ≈ 0.6931 is already computed to several
digits of accuracy.

Note
Newton’s method can be used to find zeros of functions that are hard to solve for analytically.
For example, the function f (x) = sin(x)
x − x is not continuous on any interval containing 0, but
it can be made continuous by defining f (0) = 1. Newton’s method can then be used to compute
the zeros of this function.

Problem 2. Suppose that an amount of P1 dollars is put into an account at the beginning of
years 1, 2, ..., N1 and that the account accumulates interest at a fractional rate r (so r = .05
corresponds to 5% interest). In addition, at the beginning of years N1 + 1, N1 + 2, ..., N1 + N2 ,
an amount of P2 dollars is withdrawn from the account and that the account balance is exactly
zero after the withdrawal at year N1 + N2 . Then the variables satisfy

P1 [(1 + r)N1 − 1] = P2 [1 − (1 + r)−N2 ].

Write a function that, given N1 , N2 , P1 , and P2 , uses Newton’s method to determine r.

For the initial guess, use r0 = 0.1.
(Hint: Construct f (r) such that when f (r) = 0, the equation is satisfied. Also compute f ′ (r).)
To test your function, if N1 = 30, N2 = 20, P1 = 2000, and P2 = 8000, then r ≈ 0.03878.
(From Atkinson, page 118).
102 Lab 10. Newton’s Method

There is a file called test_newtons_method.py that contains a unit test for this problem
that you can use to check your code.

Backtracking

Newton’s method may not converge for a variety of reasons. One potential problem occurs when the
step from xk to xk+1 is so large that the zero is stepped over completely. Backtracking is a strategy
that combats the problem of overstepping by moving only a fraction of the full step from xk to xk+1 .
This suggests a slight modification to (10.3),

f (xk )
xk+1 = xk − α , α ∈ (0, 1]. (10.4)
f ′ (xk )

Note that setting α = 1 results in the exact same method defined in (10.3), but for α ∈ (0, 1), only
a fraction of the step is taken at each iteration.

Problem 3. Modify your function from Problem 1 so that it accepts a parameter α that
defaults to 1. Incorporate (10.4) to allow for backtracking.
To test your modified function, consider f (x) = x1/3 . The command x**(1/3.) fails
when x is negative, so the function and its derivative can be defined with NumPy as follows.

import numpy as np
f = lambda x: np.sign(x) * np.power(np.abs(x), 1./3)
Df = lambda x: np.power(np.abs(x), -2./3) / 3.

With x0 = .01 and α = 1, the iteration should not converge. However, setting α = .4, the
iteration should converge to a zero that is close to 0.

The backtracking constant α is significant, as it can result in faster convergence or convergence

to a different zero (see Figure 10.2). However, it is not immediately obvious how to choose an optimal
value for α.
 103

0.50
0.25 x0 x
0.00
0.25 no backtracking
0.50 backtracking
0.75
1.00 x

1 0 1 2 3 4 5
Figure 10.2: Starting at the same initial value but using different backtracking constants can result
in convergence to two different solutions. The blue line converges to x̃ = (0, −1) with α = 1 in 5
iterations of Newton’s method while the orange line converges to x̂ = (3.75, .25) with α = 0.4 in 15
iterations. Note that the points in this example are 2-dimensional, which is discussed in the next
section.

Problem 4. Write a function that accepts the same arguments as your function from Problem
3 except for α. Use Newton’s method to find a zero of f using various values of α in the interval
(0, 1]. Plot the values of α against the number of iterations performed by Newton’s method.
Return a value for α that results in the lowest number of iterations.
A good test case for this problem is the function f (x) = x1/3 discussed in Problem 3. In
this case, your plot should show that the optimal value for α is actually closer to .3 than to .4.

Newton’s Method in Higher Dimensions

Newton’s method can be generalized to work on functions with a multivariate domain and range.
Let f : Rn → Rn be given by f (x) = [f1 (x) f2 (x) . . . fk (x)]T , with fi : Rn → R for each i. The
derivative Df : Rn → Rn×n is the n × n Jacobian matrix of f .
 ∂f1 ∂f1 
∂x1 · · · ∂x k

Df =  ... ..
.
.. 
. 

∂fn ∂fn
∂x1 · · · ∂xk

In this setting, Newton’s method seeks a vector x̄ such that f (x̄) = 0, the vector of n zeros.
With backtracking incorporated, (10.4) becomes
−1
xk+1 = xk − αDf (xk ) f (xk ). (10.5)
Note that if n = 1, (10.5) is exactly (10.4) because in that case, Df (x)−1 = 1/f ′ (x).
This vector version of Newton’s method terminates when the maximum number of iterations is
reached or the difference between successive approximations is less than a predetermined tolerance ε
with respect to a vector norm, that is, ||xk − xk−1 || < ε.
104 Lab 10. Newton’s Method

Problem 5. Modify your function from Problems 1 and 3 so that it can compute a zero of a
function f : Rn → Rn for any n ∈ N. Take the following tips into consideration.

• If n > 1, f should be a function that accepts a 1-D NumPy array with n entries and
returns another NumPy array with n entries. Similarly, Df should be a function that
accepts a 1-D array with n entries and returns a n × n array. In other words, f and Df
are callable functions, but f (x) is a vector and Df (x) is a matrix.

• np.isscalar() may be useful for determining whether or not n > 1.

• Instead of computing Df (xk )−1 directly at each step, solve the system Df (xk )yk = f (xk )
and set xk+1 = xk − αyk . In other words, use la.solve() instead of la.inv().

• The stopping criterion now requires using a norm function instead of abs().

After your modifications, carefully verify that your function still works in the case that
n = 1, and that your functions from Problems 2 and 4 also still work correctly. In addition,
your function from Problem 4 should also work for any n ∈ N.

Unit Test
The file test_newtons_method.py contains unit tests for Problem 2. There is a place to add
your own unit tests to test your function for Problem 5, which will be graded.

Problem 6. Bioremediation involves the use of bacteria to consume toxic wastes. At a steady
state, the bacterial density x and the nutrient concentration y satisfy the system of nonlinear
equations

γxy − x(1 + y) = 0
−xy + (δ − y)(1 + y) = 0,

where γ and δ are parameters that depend on various physical features of the system.a
For this problem, assume the typical values γ = 5 and δ = 1, for which the system has
solutions at (x, y) = (0, 1), (0, −1), and (3.75, .25). Write a function that finds an initial point
x0 = (x0 , y0 ) such that Newton’s method converges to either (0, 1) or (0, −1) with α = 1, and
to (3.75, .25) with α = 0.55. As soon as a valid x0 is found, return it (stop searching).
(Hint: search within the rectangle [− 14 , 0] × [0, 14 ].)
a This problem is adapted from exercise 5.19 of [Hea02] and the notes of Homer Walker).
 105

Basins of Attraction
When a function f has many zeros, the zero that Newton’s method converges to depends on the
initial guess x0 . For example, the function f (x) = x2 − 1 has zeros at −1 and 1. If x0 < 0, then
Newton’s method converges to −1; if x0 > 0 then it converges to 1 (see Figure 10.3a). The regions
(−∞, 0) and (0, ∞) are called the basins of attraction of f . Starting in one basin of attraction leads
to finding one zero, while starting in another basin yields a different zero.
When f is a polynomial of degree greater than 2, the basins of attraction are much more
interesting. For example, the basis of attraction for f (x) = x3 − x are shown in Figure 10.3b. The
basin for the zero at the origin is connected, but the other two basins are disconnected and share a
kind of symmetry.

2.0 1.5 1.0 0.5 0.0 0.5 1.0 1.5 2.0 2.0 1.5 1.0 0.5 0.0 0.5 1.0 1.5 2.0

(a) Basins of attraction for f (x) = x2 − 1. (b) Basins of attraction for f (x) = x3 − x.

Figure 10.3: Basins of attraction with α = 1. Since choosing a different value for α can change which
zero Newton’s method converges to, the basins of attraction may change for other values of α.

It can be shown that Newton’s method converges in any Banach space with only slightly stronger
hypotheses than those discussed previously. In particular, Newton’s method can be performed over
the complex plane C to find imaginary zeros of functions. Plotting the basins of attraction over C
yields some interesting results.
√
The zeros of f (x) = x3 −1 are 1, and − 21 ± 23 i. To plot the basins of attraction for f (x) = x3 −1
on the square complex domain X = {a+bi | a ∈ [− 23 , 32 ], b ∈ [− 32 , 32 ]}, create an initial grid of complex
points in this domain using np.meshgrid().

>>> x_real = np.linspace(-1.5, 1.5, 500) # Real parts.

>>> x_imag = np.linspace(-1.5, 1.5, 500) # Imaginary parts.
>>> X_real, X_imag = np.meshgrid(x_real, x_imag)
>>> X_0 = X_real + 1j*X_imag # Combine real and imaginary parts.

The grid X0 is a 500 × 500 array of complex values to use as initial points for Newton’s method.
Array broadcasting makes it easy to compute an iteration of Newton’s method at every grid point.

>>> f = lambda x: x**3 - 1

>>> Df = lambda x: 3*x**2
>>> X_1 = X_0 - f(X_0)/Df(X_0)
106 Lab 10. Newton’s Method

After enough iterations, the (i, j)th element of the grid Xk corresponds to the zero of f that
results from using the (i, j)th element of X√ 0 as the initial
√
point. For example, with f (x) = x3 − 1,
each entry of Xk should be close to 1, − 12 + 23 i, or − 21 − 23 i. Each entry of Xk can then be assigned
a value indicating which zero it corresponds to. Some results of this process are displayed below.

(a) Basins of attraction for f (x) = x3 − 1. (b) Basins of attraction for f (x) = x3 − x.

Figure 10.4

Note
Notice that in some portions of Figure 10.4a, whenever red and blue try to come together, a
patch of green appears in between. This behavior repeats on an infinitely small scale, producing
a fractal. Because it arises from Newton’s method, this kind of fractal is called a Newton fractal.
Newton fractals show that the long-term behavior of Newton’s method is extremely
sensitive to the initial guess x0 . Changing x0 by a small amount can change the output of
Newton’s method in a seemingly random way. This phenomenon is called chaos in mathematics.

Problem 7. Write a function that accepts a function f : C → C, its derivative f ′ : C → C,

an array zeros of the zeros of f , bounds [rmin , rmax , imin , imax ] for the domain of the plot, an
integer res that determines the resolution of the plot, and number of iterations iters to run
the iteration. Compute and plot the basins of attraction of f in the complex plane over the
specified domain in the following steps.

1. Construct a res×res grid X0 over the domain {a + bi | a ∈ [rmin , rmax ], b ∈ [imin , imax ]}.

2. Run Newton’s method (without backtracking) on X0 iters times, obtaining the res×res
array xk . To avoid the additional computation of checking for convergence at each step,
do not use your function from Problem 5.
 107

3. Xk cannot be directly visualized directly because its values are complex. Solve this issue
by creating another res×res array Y . To compute the (i, j)th entry Yi,j , determine
which zero of f is closest to the (i, j)th entry of Xk . Set Yi,j to the index of this zero in
the array zeros. If there are R distinct zeros, each Yi,j should be one of 0, 1, . . . , R − 1.
(Hint: np.argmin() may be useful.)

4. Use plt.pcolormesh() to visualize the basins. Recall that this function accepts three
array arguments: the x-coordinates (in this case, the real components of the initial grid),
the y-coordinates (the imaginary components of the grid), and an array indicating color
values (Y ). Set cmap="brg" to get the same color scheme as in Figure 10.4.

Test your function using f (x) = x3 − 1 and f (x) = x3 − x. The resulting plots should
resemble Figures 10.4a and 10.4b, respectively (perhaps with the colors permuted).
108 Lab 10. Newton’s Method
11
Conditioning and
Stability

Lab Objective: The condition number of a function measures how sensitive that function is to
changes in the input. On the other hand, the stability of an algorithm measures how accurately that
algorithm computes the value of a function from exact input. Both of these concepts are important
for answering the crucial question, “is my computer telling the truth?” In this lab we examine the
conditioning of common linear algebra problems, including computing polynomial roots and matrix
eigenvalues. We also present an example to demonstrate how two different algorithms for the same
problem may not have the same level of stability.

Note: There may be some variation in the solutions to problems in this lab between the different
updates of NumPy, SciPy, and SymPy. Consider updating these packages if you are currently using
older versions.

Conditioning
The absolute condition number of a function f : Rm → Rn at a point x ∈ Rm is defined by

∥f (x + h) − f (x)∥
κ̂(x) = lim+ sup . (11.1)
δ→0 ∥h∥<δ ∥h∥

In other words, the absolute condition number of f is the limit of the change in output over
the change of input. Similarly, the relative condition number of f is the limit of the relative change
in output over the relative change in input,
  
∥f (x + h) − f (x)∥ ∥h∥ ∥x∥
κ(x) = lim sup = κ̂(x). (11.2)
δ→0+ ∥h∥<δ ∥f (x)∥ ∥x∥ ∥f (x)∥

A function with a large condition number is called ill-conditioned. Small changes to the input
of an ill-conditioned function may produce large changes in output. It is important to know if a
function is ill-conditioned because floating point representation almost always introduces some input
error, and therefore the outputs of ill-conditioned functions cannot be trusted.

109
110 Lab 11. Conditioning and Stability

The condition number of a matrix A, κ(A) = ∥A∥∥A−1 ∥, is an upper bound on the condition
number for many of the common problems associated with the matrix, such as solving the system
Ax = b. If A is square but not invertible, then κ(A) = ∞ by convention. To compute κ(A), we often
use the matrix 2-norm, which is the largest singular value σmax of A. Recall that if σ is a singular
value of A, σ1 is a singular value of A−1 . Thus, we have that
σmax
κ(A) = , (11.3)
σmin
which is also a valid equation for non-square matrices.

Achtung!
Ill-conditioned matrices can wreak havoc in even simple applications. For example, the matrix
 
1 1
A=
1 1.0000000001

is extremely ill-conditioned, with κ(A) ≈ 4 × 1010 . Solving the systems Ax = b1 and Ax = b2

can result in wildly different answers, even when b1 and b2 are extremely close.

>>> import numpy as np

>>> from scipy import linalg as la

>>> A = np.array([[1, 1], [1, 1+1e-10]])

>>> np.linalg.cond(A)
39999991794.058899

# Set up and solve a simple system of equations.

>>> b1 = np.array([2, 2])
>>> x1 = la.solve(A, b1)
>>> print(x1)
[ 2. 0.]

# Solve a system with a very slightly different vector b.

>>> b2 = np.array([2, 2+1e-5])
>>> la.norm(b1 - b2)
>>> x2 = la.solve(A, b2)
>>> print(x2)
[-99997.99172662 99999.99172662] # This solution is hugely different!

If you find yourself working with matrices that have large condition numbers, check your
math carefully or try to reformulate the problem entirely.

Note
 111

An orthonormal matrix U has orthonormal columns and satisfies U T U = I and ∥U ∥2 = 1. If

U is square, then U −1 = U T and U T is also orthonormal. Therefore κ(U ) = ∥U ∥2 ∥U −1 ∥2 = 1.
Even if U is not square, all of its singular values are equal to 1, and again κ(U ) = σmax /σmin = 1.
The condition number of a matrix cannot be less than 1 since σmax ≥ σmin by definition.
Thus orthonormal matrices are, in a sense, the best kind of matrices for computations. This is
one of the main reasons why numerical algorithms based on the QR decomposition or the SVD
are so important.

Problem 1. Write a function that accepts a matrix A and computes its condition number
using (11.3). Use scipy.linalg.svd(), or scipy.linalg.svdvals() to compute the singular
values of A. Avoid computing A−1 . If the smallest singular value is 0, return ∞ (np.inf).
Validate your function by comparing it to np.linalg.cond(). Check that orthonormal
matrices have a condition number of 1 (use scipy.linalg.qr() to generate an orthonormal
matrix) and that singular matrices have a condition number of ∞ according to your function.

Unit Test
The file test_conditioning_stability.py contains unit tests to test your fuction from Prob-
lem 1 with orthonormal matrices. There is a place to add your own unit tests to test your
function with other kinds of matrices, which will be graded.

The Wilkinson Polynomial

Let f : Cn+1 → Cn be the function that maps a collection of n + 1 coefficients (cn , cn−1 , . . . , c0 ) to
the n roots of the polynomial cn xn + cn−1 xn−1 + . . . + c2 x2 + c1 x + c0 . Finding polynomial roots is
an extremely ill-conditioned problem in general, so the condition number of f is likely very large. To
see this, consider the Wilkinson polynomial, made famous by James H. Wilkinson in 1963:

20
Y
w(x) = (x − r) = x20 − 210x19 + 20615x18 − 1256850x17 + · · · .
r=1

Let w̃(x) be w(x) where the coefficient on x19 is very slightly perturbed from −210 to −209.9999999.
The following code computes and compares the roots of w̃(x) and w(x) using NumPy and SymPy.

Figure 11.1a plots w(x) and w̃(x) together, and Figure 11.1b and compares their roots in the
complex plane.
112 Lab 11. Conditioning and Stability

4 1e13
Original Original
3 Perturbed 2 Perturbed
2
1

Imaginary Axis
1
0 0
w(x)

1 1
2
3 2
4
0.0 2.5 5.0 7.5 10.0 12.5 15.0 17.5 20.0 2.5 5.0 7.5 10.0 12.5 15.0 17.5 20.0
x Real Axis
(a) The original and perturbed Wilkinson polynomi- (b) Roots of the original and perturbed Wilkinson
als. They match for about half of the domain, then polynomials. About half of the perturbed roots are
differ drastically. complex.

Figure 11.1

Figure 11.1 clearly indicates that a very small change in just a single coefficient drastically
changes the nature of the polynomial and its roots. To quantify the difference, estimate the condition
numbers (this example uses the ∞ norm to compute κ̂ and κ).

>>> import sympy as sy

>>> from matplotlib import pyplot as plt

# The roots of w are 1, 2, ..., 20.

>>> w_roots = np.arange(1, 21)

# Get the exact Wilkinson polynomial coefficients using SymPy.

>>> x, i = sy.symbols('x i')
>>> w = sy.poly_from_expr(sy.product(x-i, (i, 1, 20)))[0]
>>> w_coeffs = np.array(w.all_coeffs())
>>> print(w_coeffs[:6])
[1 -210 20615 -1256850 53327946 -1672280820]

# Perturb one of the coefficients very slightly.

>>> h = np.zeros(21)
>>> h[1]=1e-7
>>> new_coeffs = w_coeffs + h
>>> print(new_coeffs[:6])
[1 -209.999999900000 20615 -1256850 53327946 -1672280820]

# Use NumPy to compute the roots of the perturbed polynomial.

>>> new_roots = np.roots(np.poly1d(new_coeffs))

# Sort the roots to ensure that they are in the same order.
>>> w_roots = np.sort(w_roots)
>>> new_roots = np.sort(new_roots)

# Estimate the absolute condition number in the infinity norm.

 113

>>> k = la.norm(new_roots - w_roots, np.inf) / la.norm(h, np.inf)

>>> print(k)
27841936.8061

# Estimate the relative condition number in the infinity norm.

>>> k * la.norm(w_coeffs, np.inf) / la.norm(w_roots, np.inf)
1.92161703373792e+25 # This is huge!!

There are some caveats to this example.

1. Computing the quotients in (11.1) and (11.2) for a fixed perturbation h only approximates the
condition number. The true condition number is the limit of such quotients. We hope that
when ∥h∥ is small, a random quotient is at least the same order of magnitude as the limit, but
there is no way to be sure.

2. This example assumes that NumPy’s root-finding algorithm, np.roots(), is stable, so that the
difference between w_roots and new_roots is due to the difference in coefficients, and not to
problems with np.roots(). We will return to this issue in the next section.

Even with these caveats, it is apparent that root finding is a difficult problem to solve correctly.
Always check your math carefully when dealing with polynomial roots.

Problem 2. Write a function that carries out the following experiment 100 times.

1. Randomly perturb the true coefficients of the Wilkinson polynomial by replacing each
coefficient ci with ci ∗ ri , where ri is drawn from a normal distribution centered at 1 with
standard deviation 1e-10 (use np.random.normal()).

2. Plot the perturbed roots as small points in the complex plane. That is, plot the real part
of the coefficients on the x-axis and the imaginary part on the y-axis. Plot on the same
figure in each experiment.
(Hint: use a pixel marker, marker=',', to avoid overcrowding the figure.)

3. Compute the absolute and relative condition numbers with the ∞ norm.

Plot the roots of the unperturbed Wilkinson polynomial with the perturbed roots. Your final
plot should resemble Figure 11.2. Finally, return the average computed absolute and relative
condition numbers.
114 Lab 11. Conditioning and Stability

6
Perturbed
4 Original
2
Imaginary Axis 0
2
4
6
0 5 10 15 20
Real Axis
Figure 11.2: This figure replicates Figure 12.1 on p. 93 of [TB97].

Calculating Eigenvalues
Let f : Mn (C) → Cn be the function that maps an n × n matrix with complex entries to its n
eigenvalues. This problem is well-conditioned for symmetric matrices, but it can be extremely ill-
conditioned for non-symmetric matrices. Let A be an n × n matrix and let λ be the vector of the n
eigenvalues of A. If à = A + H is a pertubation of A and λ̃ are its eigenvalues, then the condition
numbers of f can be estimated by

∥λ − λ̃∥ ∥A∥
κ̂(A) = , κ(A) = κ̂(A). (11.4)
∥H∥ ∥λ∥

Problem 3. Write a function that accepts a matrix A and estimates the condition number of
the eigenvalue problem using (11.4). For the perturbation H, construct a matrix with complex
entries where the real and imaginary parts are drawn from normal distributions centered at 0
with standard deviation σ = 10−10 .

reals = np.random.normal(0, 1e-10, A.shape)

imags = np.random.normal(0, 1e-10, A.shape)
H = reals + 1j*imags

Use scipy.linalg.eig() or scipy.linalg.eigvals() to compute the eigenvalues of A and

A + H, and use the 2-norm for both the vector and matrix norms. Return the absolute and
relative condition numbers.
 115

The order scipy.linalg.eig() and scipy.linalg.eigvals() return eigenvalues is com-

pletely arbitrary, which means that the eigenvalues of A and A+H may be returned in different
orders. Without correcting for this, the computed value of ∥λ − λ̃∥ can be very large even if the
two matrices have very similar eigenvalues. Additionally, np.sort() does not help, as many
matrices have sets of eigenvalues for which the sort order of this function is itself ill-conditioned.
So, before comparing the two lists of eigenvalues, use the following function to reorder both to
be as close as possible to each other:

def reorder_eigvals(orig_eigvals, pert_eigvals):

"""Reorder the perturbed eigenvalues to be as close to the original ←-
eigenvalues as possible.
Parameters:
orig_eigvals ((n,) ndarray) - The eigenvalues of the unperturbed ←-
matrix A
pert_eigvals ((n,) ndarray) - The eigenvalues of the perturbed ←-
matrix A+H
Returns:
((n,) ndarray) - the reordered eigenvalues of the perturbed matrix
"""
n = len(pert_eigvals)
sort_order = np.zeros(n).astype(int)
dists = np.abs(orig_eigvals - pert_eigvals.reshape(-1,1))
for _ in range(n):
index = np.unravel_index(np.argmin(dists), dists.shape)
sort_order[index[0]] = index[1]
dists[index[0],:] = np.inf
dists[:,index[1]] = np.inf
return pert_eigvals[sort_order]

Problem 4. Write a function that accepts bounds [xmin , xmax , ymin , ymax ] and an integer res.
Use your function from Problem 3 to compute the relative condition number of the eigenvalue
problem for the 2 × 2 matrix  
1 x
y 1
at every point of an evenly spaced res×res grid over the domain [xmin , xmax ] × [ymin , ymax ].
Plot these estimated relative condition numbers using plt.pcolormesh() and the colormap
cmap='gray_r' (you can use plt.colorbar() to create the colorbar). With res=200, your
plot should look similar to the following figure.
116 Lab 11. Conditioning and Stability

Problem 4 shows that the conditioning of the eigenvalue problem depends heavily on the matrix,
and that it is difficult to know a priori how bad the problem will be. Luckily, most real-world problems
requiring eigenvalues are symmetric. In their book on Numerical Linear Algebra, L. Trefethen and
D. Bau III summed up the issue of conditioning and eigenvalues when they stated, “if the answer is
highly sensitive to perturbations, you have probably asked the wrong question.”

Stability
The stability of an algorithm is measured by the error in its output. Let f : Rm → Rn be a problem
to be solved, as in the previous section, and let f˜ be an actual algorithm for solving the problem.
The forward error of f at x is ||f (x) − f˜(x)||, and the relative forward error of f at x is

||f (x) − f˜(x)||

.
||f (x)||

An algorithm is called stable if its relative forward error is small.1

As an example, consider again NumPy’s root-finding algorithm that we used to investigate the
Wilkinson polynomial. The exact roots of w(x) are clearly 1, 2, . . . , 20. Had we not known this, we
could have tried computing the roots from the coefficients using np.roots() (without perturbing
the coefficients at all).

# w_coeffs holds the coefficients and w_roots holds the true roots.
>>> computed_roots = np.sort(np.roots(np.poly1d(w_coeffs)))
>>> print(computed_roots[:6]) # The computed roots are close to integers.
[ 1. 2. 3. 3.99999999 5.00000076 5.99998749]

# Compute the forward error.

>>> forward_error = la.norm(w_roots - computed_roots)
>>> print(forward_error)
0.020612653126379665

1 See the Additional Material section for alternative (and more rigorous) definitions of algorithmic stability.
 117

# Compute the relative forward error.

>>> forward_error / la.norm(w_roots)
0.00038476268486104599 # The error is nice and small.

This analysis suggests that np.roots() is a stable algorithm, so large condition numbers of
Problem 2 really are due to the poor conditioning of the problem, not the way in which the problem
was solved.

Note
Conditioning is a property of a problem to be solved, such as finding the roots of a polynomial
or calculating eigenvalues. Stability is a property of an algorithm to solve a problem, such
as np.roots() or scipy.linalg.eig(). If a problem is ill-conditioned, any algorithm used to
solve that problem may result in suspicious solutions, even if that algorithm is stable.

Least Squares
The ordinary least squares (OLS) problem is to find the x that minimizes ∥Ax − b∥2 for fixed A and
b. It can be shown that an equivalent problem is finding the solution of AH Ax = AH b, called the
normal equations. A common application of least squares is polynomial approximation. Given a set
k=1 , the goal is to find the set of coefficients {ci }i=0 such that
of m data points {(xk , yk )}m n

yk ≈ cn xnk + cn−1 xn−1

k + · · · + c2 x2k + c1 xk + c0
for all k, with the smallest possible error. These m linear equations yield the linear system
cn
 
x1 xn−1
 n 2
· · ·
  
1 x 1 x 1 1 y1
 xn2 x2 n−1 cn−1
· · · x22 x2 1 
 
  .   y2 
   
 x3 xn−1 · · · x23 x3 1   ..   y3 
 n
Ax =  3 =  = b. (11.5)
 . .. .. .. ..     . 

 . . . . . . 
c2   .  .
 c1 
n n−1 2
xm xm · · · xm xm 1 ym
c0

Problem 5. Write a function that accepts an integer n. Solve for the coefficients of the poly-
nomial of degree n that best fits the data found in stability_data.npy. Use two approaches
to get the least squares solution:

1. Use la.inv() to solve the normal equations: x = (AT A)−1 AT b. Although this approach
seems intuitive, it is actually highly unstable and can return an answer with a very large
forward error.

2. Use la.qr() with mode='economic' and la.solve_triangular() to solve the system

Rx = QT b, which is equivalent to solving the normal equations. This algorithm has the
advantage of being stable.

Load the data and set up the system (11.5) with the following code.
118 Lab 11. Conditioning and Stability

xk, yk = np.load("stability_data.npy").T
A = np.vander(xk, n+1)

Plot the resulting polynomials together with the raw data points. Return the forward
error ∥Ax − b∥2 of both approximations.
(Hint: The function np.polyval() will be helpful for plotting the resulting polynomials.)
Test your function using various values of n, taking special note of what happens for values
of n near 14.

Catastrophic Cancellation
When a computer takes the difference of two very similar numbers, the result is often stored with
a small number of significant digits and the tiniest bit of information is lost. However, these small
errors can propagate into large errors later down the line. This phenomenon is called catastrophic
cancellation, and is a common cause for numerical instability.
Catastrophic cancellation is a potential problem whenever floats or large integers that are very
close to one another are subtracted. This problem can be avoided by either rewriting the program
to not use subtraction, or by increasing the number of significant digits that the computer tracks.
√ √
For example, consider the simple problem of computing a − b. The computation can be
done directly with subtraction, or by performing the equivalent division
√
√ √ √ √ √a + b a−b
a − b = ( a − b) √ √ =√ √ .
a+ b a+ b

>>> from math import sqrt # np.sqrt() fails for very large numbers.

>>> a = 10**20 + 1
>>> b = 10**20
>>> sqrt(a) - sqrt(b) # Do the subtraction directly.
0.0 # a != b, so information has been lost.

>>> (a - b) / (sqrt(a) + sqrt(b)) # Use the alternative formulation.

5e-11 # Much better!

In this
√ example, a and b are distinct enough√that the computer can still tell that a − b = 1, but
√ √
a and b are so close to each other that a − b is computed as 0.

R1
Problem 6. Let I(n) = 0
xn ex−1 dx. It can be shown that for a positive integer n,

n!
I(n) = (−1)n (!n − ), (11.6)
e
Pn (−1)k
where !n = n! k=0 k! is the subfactorial of n. Write a function to do the following.
 119

1. Use SymPy’s sy.integrate() to evaluate the integral form of I(n) for n = 5, 10, . . . , 50.
Convert the symbolic results of each integration to a float. Since this is done symbolically,
these values can be accepted as the true values of I(n). For this problem, use sy.exp()
in the integrand.
(Hint: be careful that the values of n in the integrand are of type int.)

2. Use (11.6) to compute I(n) for the same values of n. Use sy.subfactorial() to compute
!n and sy.factorial() to compute n!. The function used for e in this equation changes
the returned error value. For this problem, use np.e instead of sy.exp().
(Hint: be careful to only pass Python integers to these functions.)

3. Plot the relative forward error of the results computed in step 2 at each of the given
values of n. When computing the relative forward error use absolute values instead of
la.norm(). Use a log scale on the y-axis.

The examples presented in this lab are just a few of the ways that a mathematical problem can
turn into a computational train wreck. Always use stable algorithms when possible, and remember
to check if problems are well conditioned or not.
120 Lab 11. Conditioning and Stability

Additional Material
Other Notions of Stability
The definition of stability can be made more rigorous in the following way. Let f be a problem to
solve and f˜ an algorithm to solve it. If for every x in the domain there exists a x̃ such that

∥x̃ − x∥ ∥f˜(x) − f (x̃)∥

and
∥x∥ ∥f (x̃)∥

are small (close to εmachine ≈ 10−16 ), then f˜ is called stable. In other words, “A stable algorithm
gives nearly the right answer to nearly the right question” (Trefethen, Bao, 104). Note carefully that
the quantity on the right is slightly different from the plain forward error introduced earlier.
Stability is desirable, but plain stability isn’t the best possible condition. For example, if for
every input x there exists a x̃ such that ∥x̃−x∥/∥x∥ is small and f˜(x) = f (x̃) exactly, then f˜ is called
backward stable. Thus “A backward stable algorithm gives exactly the right answer to nearly the right
question” (Trefethen, Bao, 104). Backward stable algorithms are generally more trustworthy than
stable algorithms, but they are also less common.

Stabilty of Linear System Solvers

The algorithms presented so far in this manual have different levels of stability. The LU decomposition
(with pivoting) is usually very good, but there are some pathological examples of matrices that
can cause it to break down. Even so, scipy.linalg.solve() uses the LU decomposition. The
QR decomposition (also with pivoting) is generally considered to be a better option than the LU
decomposition and is more stable. However, solving a linear system using the SVD is even more
stable than using the QR decomposition. For this reason, scipy.linalg.lstsq() uses the SVD.
12
Monte Carlo
Integration

Lab Objective: Many important integrals cannot be evaluated symbolically because the integrand
has no antiderivative. Traditional numerical integration techniques like Newton-Cotes formulas and
Gaussian quadrature usually work well for one-dimensional integrals, but rapidly become inefficient
in higher dimensions. Monte Carlo integration is an integration strategy that has relatively slow
convergence, but that does extremely well in high-dimensional settings compared to other techniques.
In this lab we implement Monte Carlo integration and apply it to a classic problem in statistics.

Volume Estimation
Since the area of a circle of radius r is A = πr2 , one way to numerically estimate π is to compute
the area of the unit circle. Empirically, we can estimate the area by randomly choosing points in a
domain that encompasses the unit circle. The percentage of points that land within the unit circle
approximates the percentage of the area of the domain that the unit circle occupies. Multiplying this
percentage by the total area of the sample domain gives an estimate for the area of the circle.
Since the unit circle has radius r = 1, consider the square domain Ω = [−1, 1] × [−1, 1]. The
following code samples 2000 uniformly distributed random points in Ω, determines what percentage
of those points are within the unit circle, then multiplies that percentage by 4 (the area of Ω) to get
an estimate for π.

>>> import numpy as np

>>> from scipy import linalg as la

# Get 2000 random points in the 2-D domain [-1,1]x[-1,1].

>>> points = np.random.uniform(-1, 1, (2,2000))

# Determine how many points are within the circle.

>>> lengths = la.norm(points, axis=0)
>>> num_within = np.count_nonzero(lengths < 1)

# Estimate the circle's area.

>>> 4 * (num_within / 2000)
3.198

121
122 Lab 12. Monte Carlo Integration

The estimate π ≈ 3.198 isn’t perfect, but it only differs from the true value of π by about
0.0564. On average, increasing the number of sample points decreases the estimate error.

500 points; 3.088 2000 points; 3.198 8000 points; 3.167

1.00 1.00 1.00
0.75 0.75 0.75
0.50 0.50 0.50
0.25 0.25 0.25
0.00 0.00 0.00
0.25 0.25 0.25
0.50 0.50 0.50
0.75 0.75 0.75
1.00 1.00 1.00
1.0 0.5 0.0 0.5 1.0 1.0 0.5 0.0 0.5 1.0 1.0 0.5 0.0 0.5 1.0

Figure 12.1: Estimating the area of the unit circle using random points.

Problem 1. The n-dimensional open unit ball is the set Un = {x ∈ Rn | ∥x∥2 < 1}. Write a
function that accepts an integer n and a keyword argument N defaulting to 104 . Estimate the
volume of Un by drawing N points over the n-dimensional domain [−1, 1]×[−1, 1]×· · ·×[−1, 1].
(Hint: the volume of [−1, 1] × [−1, 1] × · · · × [−1, 1] is 2n .)
When n = 2, this is the same experiment outlined above so your function should return
an approximation of π. The volume of the U3 is 43 π ≈ 4.18879, and the volume of U4 is
π2
2 ≈ 4.9348. Try increasing the number of sample points N to see if your estimates improve.

Integral Estimation
The strategy for estimating π can be formulated as an integral problem. Define f : R2 → R by
(
1 if ∥x∥2 < 1 (x is within the unit circle)
f (x) =
0 otherwise,

and let Ω = [−1, 1] × [−1, 1] as before. Then

Z 1 Z 1 Z
f (x, y) dx dy = f (x) dV = π.
−1 −1 Ω

To estimate the integral we chose N random points {xi }N i=1 in Ω. Since f indicates whether or not a
point lies within the unit circle, the total number of random points that lie in the circle is the sum of
the f (xi ). Then the average of these values, multiplied by the volume V (Ω), is the desired estimate:
Z N
1 X
f (x) dV ≈ V (Ω) f (xi ). (12.1)
Ω N i=1

This remarkably simple equation can be used to estimate the integral of any integrable function
f : Rn → R over any domain Ω ⊂ Rn and is called the general formula for Monte Carlo integration.
 123

PN
The intuition behind (12.1) is that N1 i=1 f (xi ) approximates the average value of f on Ω,
and multiplying the approximate average value by the volume of Ω yields the approximate integral
of f over Ω. This is a little easier to see in one dimension: for a single-variable function f : R → R,
the Average Value Theorem states that the average value of f over an interval [a, b] is given by
Z b
1
favg = f (x) dx.
b−a a
PN
Then using the approximation favg ≈ 1
N i=1 f (xi ), the previous equation becomes
Z b N
1 X
f (x) dx = (b − a)favg ≈ V (Ω) f (xi ), (12.2)
a N i=1

which is (12.1) in one dimension. In this setting Ω = [a, b] and hence V (Ω) = b − a.

Problem 2. Write a function that accepts a function f : R → R, bounds of integration a and

b, and an integer N defaulting to 104 . Use np.random.uniform() to sample N points over the
interval [a, b], then use (12.2) to estimate the integral
Z b
f (x) dx.
a

Test your function on the following integrals, or on other integrals that you can check by hand.
Z 2 Z 2π Z 10
2 1
x dx = 24 sin(x) dx = 0 dx = log(10) ≈ 2.30259
−4 −2π 1 x
Z 5 √
sin(10x) cos(10x) + x sin(3x) dx ≈ 4.502
1

Achtung!
Be careful not to use Monte Carlo integration to estimate integrals that do not converge. For
example, since 1/x approaches ∞ as x approaches 0 from the right, the integral
Z 1
1
dx
0 x

does not converge. Even so, attempts at Monte Carlo integration still return a finite value. Use
various numbers of sample points to see whether or not the integral estimate is converging.

>>> for N in [5000, 7500, 10000]:

... print(np.mean(1. / np.random.uniform(0, 1, N)), end='\t')
...
11.8451683722 25.5814419888 7.64364735049 # No convergence.
124 Lab 12. Monte Carlo Integration

Integration in Higher Dimensions

The implementation of (12.1) for a function f : Rn → R with n > 1 introduces a few tricky details,
but the overall procedure is the same for the case when n = 1. We consider only the case where
Ω ⊂ Rn is an n-dimensional box [a1 , b1 ] × [a2 , b2 ] × · · · × [an , bn ].

1. If n = 1 then Ω is a line, so V (Ω) = b1 − a1 . If n = 2 then Ω is a rectangle, and hence

V (Ω) = (b1 − a1 )(b2 − a2 ), the product of the side lengths. The volume of a higher-dimensional
box Ω is also the product of the side lengths,
n
Y
V (Ω) = (bi − ai ) (12.3)
i=1

2. It is easy to sample uniformly over an interval [a, b] with np.random.uniform(), or even over
the n-dimensional cube [a, b] × [a, b] × · · · × [a, b] (such as in Problem 1). However, if ai ̸= aj
or bi ̸= bj for any i ̸= j, the samples need to be constructed in a slightly different way.
The interval [0, 1] can be transformed to the interval [a, b] by scaling it so that it is the same
length as [a, b], then shifting it to the appropriate location.
scale by b−a shift by a
[0, 1] −−−−−−−−→ [0, b − a] −−−−−−→ [a, b]

This suggests a strategy for sampling over [a1 , b1 ] × [a2 , b2 ] × · · · × [an , bn ]: sample uniformly
from the n-dimensional box [0, 1]×[0, 1]×· · ·×[0, 1], multiply the ith component of each sample
by bi − ai , then add ai to that component.
scale shift
[0, 1] × · · · × [0, 1] −−−→ [0, b1 − a1 ] × · · · × [0, bn − an ] −−−→ [a1 , b1 ] × · · · × [an , bn ] (12.4)

Problem 3. Write a function that accepts a function f : Rn → R, a list of lower bounds

[a1 , a2 , . . . , an ], a list of upper bounds [b1 , b2 , . . . , bn ], and an integer N defaulting to 104 . Use
(12.1), (12.3), and (12.4) with N sample points to estimate the integral
Z
f (x) dV,
Ω

where Ω = [a1 , b1 ] × [a2 , b2 ] × · · · × [an , bn ]. Return your answer as a float.

(Hint: use a list comprehension to calculate all of the f (xi ) quickly.)

Test your function on the following integrals below, as well as the single dimensional exam-
ples from Problem 2.
(Hint: make sure bounds are inputted as lists, even in the single dimensional case).
Z 1 Z 1 Z 1 Z 3
2 2 2
x + y dx dy = 3x − 4y + y 2 dx dy = 54
0 0 3 −2 1

Z 4 Z 3 Z 2 Z 1
x + y − wz 2 dx dy dz dw = 0
−4 −3 −2 −1
 125

Note carefully how the order of integration defines the domain; in the last example, the x-y-z-w
domain is [−1, 1] × [−2, 2] × [−3, 3] × [−4, 4], so the lower and upper bounds passed to your
function should be [−1, −2, −3, −4] and [1, 2, 3, 4], respectively.

Convergence
Monte Carlo integration has some obvious pros and cons. On the one hand, it is difficult√to get highly
precise estimates. In fact, the error of the Monte Carlo method is proportional to 1/ N , where N
is the number of points used in the estimation. This means that dividing the error by 10 requires
using 100 times more sample points.
On the other hand, the convergence rate is independent of the number of dimensions of the
problem. That is, the error converges at the same rate whether integrating a 2-dimensional function or
a 20-dimensional function. This gives Monte Carlo integration a huge advantage over other methods,
and makes it especially useful for estimating integrals in high dimensions where other methods become
computationally infeasible.

Problem 4. The probability density function of the joint distribution of n independent normal
random variables, each with mean 0 and variance 1, is the function f : Rn → R defined by
1 xT x
f (x) = n/2
e− 2 .
(2π)

Though this is a critical distribution in statistics, f does not have a symbolic antiderivative.
Integrate f several times to study the convergence properties of Monte Carlo integration.

1. Let n = 4 and Ω = [− 32 , 43 ] × [0, 1] × [0, 21 ] × [0, 1] ⊂ R4 . Define f and Ω so that you can
integrate f over Ω using your function from Problem 3.

2. Use scipy.stats.mvn.mvnun() to get the “exact” value of F = Ω f (x) dV . As an

R

example, the following code computes the integral over [−1, 1] × [−1, 3] × [−2, 1] ⊂ R3 .

>>> from scipy import stats

# Define the bounds of integration.

>>> mins = np.array([-1, -1, -2])
>>> maxs = np.array([ 1, 3, 1])

# The distribution has mean 0 and covariance I (the nxn identity).

>>> means, cov = np.zeros(3), np.eye(3)

# Compute the integral with SciPy.

>>> stats.mvn.mvnun(mins, maxs, means, cov)[0]
0.4694277116055261
126 Lab 12. Monte Carlo Integration

3. Use np.logspace() to get 20 integer values of N that are roughly logarithmically spaced
from 101 to 105 . For each value of N , use your function from Problem 3 to compute an
estimate F̃ (N ) of the integral with N samples. Compute the relative error |F −|F
F̃ (N )|
| for
each value of N .

4. Plot
√ the relative error against the sample size N on a log-log scale. Also plot the line
1/ N for comparison. Your results should be similar to Figure 12.2.

Relative Error
10 1
1/ N
10 2

10 3

10 4

10 5
101 102 103 104 105
√
Figure 12.2: Monte Carlo integration converges at the same rate as 1/ N where N is the number of
samples used in the estimate. However, the convergence is independent of dimension, which is why
this strategy is so commonly used for high-dimensional integration.
13
Visualizing
Complex-valued
Functions

Lab Objective: Functions that map from the complex plane into the complex plane are difficult
to fully visualize because the domain and range are both 2-dimensional. However, such functions
can be visualized at the expense of partial information. In this lab we present methods for analyzing
complex-valued functions visually, including locating their zeros and poles in the complex plane. We
recommend completing the exercises in a Jupyter Notebook.

Representations of Complex Numbers

A complex number z = x + iy can be written in polar coordinates as z = reiθ where

• r = |z| = x2 + y 2 is the magnitude of z, and

p

• θ = arg(z) = arctan(y/x) is the argument of z, the angle in radians between z and 0.

Conversely, Euler’s formula is the relation reiθ = r cos(θ) + ir sin(θ). Then setting reiθ = x + iy and
equating real and imaginary parts yields the equations x = r cos(θ) and y = r sin(θ).

z
iy
r

θ
x

Figure 13.1: The complex number z can be represented in Cartesian coordinates as z = x + iy and
in polar coordinates as z = reiθ , when θ is in radians.

NumPy makes it easy to work with complex numbers and convert between coordinate systems.
The function np.angle() returns the argument θ of a complex number (between −π and π) and
np.abs() (or np.absolute()) returns the magnitude r. These functions also operate element-wise
on NumPy arrays.

127
128 Lab 13. Visualizing Complex-valued Functions

>>> import numpy as np

>>> z = 2 - 2*1j # 1j is the imaginary unit i = sqrt(-1).

>>> r, theta = np.abs(z), np.angle(z)
>>> print(r, theta) # The angle is between -pi and pi.
2.82842712475 -0.785398163397

# Check that z = r * e^(i*theta)

>>> np.isclose(z, r*np.exp(1j*theta))
True

# These function also work on entire arrays.

>>> np.abs(np.arange(5) + 2j*np.arange(5))
array([ 0. , 2.23606798, 4.47213595, 6.70820393, 8.94427191])

Complex Functions
A function f : C → C is called a complex-valued function. Visualizing f is difficult because C has 2
real dimensions, so the graph of f should be 4-dimensional. However, since it is possible to visualize
3-dimensional objects, f can be visualized by ignoring one dimension. There are two main strategies
for doing this: assign a color to each point z ∈ C corresponding to either the argument θ of f (z), or
to the magnitude r of f (z). The graph that uses the argument is called a complex color wheel graph.
Figure 13.2 displays the identity function f (z) = z using these two methods.

Figure 13.2: The identity function f : C → C defined by f (z) = z. On the left, the color at each
point z represents the angle θ = arg(f (z)). As θ goes from −π to π, the colors cycle smoothly
counterclockwise from white to blue to red and back to white (this colormap is called "twilight").
On the right, the color represents the magnitude r = |f (z)|. The further a point is from the origin,
the greater its magnitude (the colormap is the default, "viridis").
 129

The plots in Figure 13.2 use Cartesian coordinates in the domain and polar coordinates in
the codomain. The procedure for plotting in this way is fairly simple. Begin by creating a grid of
complex numbers: create the real and imaginary parts separately, then use np.meshgrid() to turn
them into a single array of complex numbers. Pass this array to the function f , compute the angle
and argument of the resulting array, and plot them using plt.pcolormesh(). The following code
sets up the complex domain grid.

>>> x = np.linspace(-1, 1, 400) # Real domain.

>>> y = np.linspace(-1, 1, 400) # Imaginary domain.
>>> X, Y = np.meshgrid(x, y) # Make grid matrices.
>>> Z = X + 1j*Y # Combine the grids into a complex array.

Visualizing the argument and the magnitude separately provides different perspectives of the
function f . The angle plot is generally more useful for visualizing function behavior, though the
magnitude plot often makes it easy to spot important points such as zeros and poles.

√
Figure 13.3: Plots of f (z) = z 2 + 1 on {x + iy | x, y ∈ [−3, 3]}. Notice how a discontinuity is clearly
visible in the angle plot on the left, but disappears from the magnitude plot on the right.

Problem 1. Write a function that accepts a function f : C → C, bounds [rmin , rmax , imin , imax ]
for the domain, an integer res that determines the resolution of the plot, and a string to set
the figure title. Plot arg(f (z)) and |f (z)| on an equally-spaced res×res grid over the domain
{x + iy | x ∈ [rmin , rmax ], y ∈ [imin , imax ]} in separate subplots.

1. For arg(f (z)), set the plt.pcolormesh() keyword arguments vmin and vmax to −π and
π, respectively. This forces the color spectrum to work well with np.angle(). Use the
colormap "twilight", which starts and ends white, so that the color is the same for −π
and π.
130 Lab 13. Visualizing Complex-valued Functions

2. For |f (z)|, set norm=matplotlib.colors.LogNorm() in plt.pcolormesh() so that the

color scale is logarithmic. Use a sequential colormap like "viridis" or "magma".

3. Set the aspect ratio to "equal" in each plot. Give each subplot a title, and set the overall
figure title with the given input string.
√
Use your function to visualize f (z) = z on {x + iy | x, y ∈ [−1, 1]} and f (z) = z 2 + 1 on
{x + iy | x, y ∈ [−3, 3]}. Compare the resulting plots to Figures 13.2 and 13.3, respectively.

Interpreting Complex Plots

Plots of a complex function can be used to quickly identify important points in the function’s domain.

Zeros
A complex number z0 is called a zero of the complex-valued function f if f (z0 ) = 0. The mutliplicity
or order of z0 is the largest integer n such that f can be written as f (z) = (z − z0 )n g(z) where
g(z0 ) ̸= 0. In other words, f has a zero of order n at z0 if the Taylor series of f centered at z0 can
be written as
X∞
f (z) = ak (z − z0 )k , an ̸= 0.
k=n

Angle and magnitude plots make it easy to locate a function’s zeros and to determine their
multiplicities.

Problem 2. Use your function from Problem 1 to plot the following functions on the domain
{x + iy | x, y ∈ [−1, 1]}.

• f (z) = z n for n = 2, 3, 4.

• f (z) = z 3 − iz 4 − 3z 6 . Compare the resulting plots to Figure 13.4.

Use a Markdown cell to write a sentence or two about how the zeros of a function and their
multiplicity appear in angle and magnitude plots.

Problem 2 shows that in an angle plot of f (z) = z n , the colors cycle n times counterclockwise
around 0. This is explained by looking at z n in polar coordinates,

z n = (reiθ )n = rn ei(nθ) .

Multiplying θ by a number greater than 1 compresses the graph along the “θ-axis” by a factor of n.
In other words, the output angle repeats itself n times in one cycle of θ. This is similar to taking a
scalar-valued function f : R → R and replacing f (x) with f (nx).
Problem 2 also shows that the plot of f (z) = z 3 − iz 4 − 3z 6 looks very similar to the plot of
f (z) = z 3 near the origin. This is because when z is close to the origin, z 4 and z 6 are much smaller
in magnitude than z 3 , and so the behavior of z 3 dominates the function. In terms of the Taylor series
centered at z0 = 0, the quantity |z − z0 |n+k is much smaller than |z − z0 |n for z close to z0 , and so
the function behaves similar to an (z − z0 )n .
 131

Figure 13.4: The angle plot of f (z) = z 3 − iz 4 − 3z 6 on {x + iy | x, y ∈ [−1, 1]}. The angle plot
shows that f (z) has a zero of order 3 at the origin and 3 distinct zeros of order 1 scattered around
the origin. The magnitude plot makes it easier to pinpoint the location of the zeros.

Poles
A complex number z0 is called a pole of the complex-valued function f if f can be written as
f (z) = g(z)/(z − z0 ) where g(z0 ) ̸= 0. From this definition it is easy to see that limz→z0 |f (z)| = ∞,
but knowing that limz→z1 |f (z)| = ∞ is not enough information to conclude that z1 is a pole of f .
The order of z0 is the largest integer n such that f can be written as f (z) = g(z)/(z − z0 )n
with g(z0 ) ̸= 0. In other words, f has a pole of order n at z0 if its Laurent series on a punctured
neighborhood of z0 can be written as
∞
X
f (z) = ak (z − z0 )k , a−n ̸= 0.
k=−n

Problem 3. Plot the following functions on domains that show all of its zeros and/or poles.

• f (z) = z −n for n = 1, 2, 3.

• f (z) = z 2 + iz −1 + z −3 .

Use a Markdown cell to write a sentence or two about how the poles of a function appear in
angle and magnitude plots. How can you tell the multiplicity of the poles from the plot?

Problem 3 shows that in angle plot of z −1 , the colors cycle n times clockwise around 0, as
opposed to the counter-clockwise rotations seen around roots. Again, this can be explained by
looking at the polar representation,

z −n = (reiθ )−n = r−n ei(−nθ) .

132 Lab 13. Visualizing Complex-valued Functions

The minus sign on the θ reverses the direction of the colors, and the n makes them cycle n times.
From Problem 3 it is also clear that f (z) = z 2 + iz −1 + z −3 behaves similarly to z −3 for z near
the pole at z0 = 0. Since |z − z0 |−n+k is much smaller than |z − z0 |−n when |z − z0 | is small, near
z0 the function behaves like a−n (z − z0 )−n . This is why the order of a pole can be estimated by
counting the number of times the colors circle a point in the clockwise direction.

Counting Zeros and Poles

The Fundamental Theorem of Algebra states that a polynomial f with highest degree n has exactly
n zeros, counting multiplicity. For example, f (z) = z 2 + 1 has two zeros, and f (z) = (z − i)3 has
three zeros, all at z0 = i (that is, z0 = i is a zero with multiplicity 3).
The number of poles of function can also be apparent if it can be written as a quotient of
polynomials. For example, f (z) = z/(z+i)(z−i)2 has one zeros and three poles, counting multiplicity.

Problem 4. Plot the following functions and count the number and order of their zeros and
poles. Adjust the bounds of each plot until you have found all zeros and poles.

• f (z) = −4z 5 + 2z 4 − 2z 3 − 4z 2 + 4z − 4

• f (z) = z 7 + 6z 6 − 131z 5 − 419z 4 + 4906z 3 − 131z 2 − 420z + 4900

16z 4 +32z 3 +32z 2 +16z+4
• f (z) = 16z 4 −16z 3 +5z 2

It is usually fairly easy to see how many zeros or poles a polynomial or quotient of polynomials
has. However, it can be much more difficult to know how many zeros or poles a different function
may or may not have without visualizing it.

Problem 5. Plot the following functions on the domain {x + iy | x, y ∈ [−8, 8]}. Explain
carefully in a Markdown cell what each graph reveals about the function and why the function
behaves that way.

• f (z) = ez

• f (z) = tan(z)

(Hint: use the polar coordinate representation to mathematically examine the magnitude and
angle of each function.)

Essential Poles
A complex-valued function f has an essential pole at z0 if its Laurent series in a punctured neigh-
borhood of z0 requires infinitely many terms with negative exponents. For example,
∞
X 1 1 1 1 1 1
e1/z = n
=1+ + 2
+ + ··· .
n!z z 2z 6 z3
k=0

An essential pole can be thought of as a pole of order ∞. Therefore, in an angle plot the colors cycle
infinitely many times around an essential pole.
 133

Figure 13.5: Angle plot of f (z) = e1/z on the domain {x + iy | x, y ∈ [−1, 1]}. The colors circle
clockwise around the origin because it is a pole, not a zero. Because the pole is essential, the colors
repeat infinitely many times.

Achtung!
Often, color plots like the ones presented in this lab can be deceptive because of a bad choice
of domain. Be careful to validate your observations mathematically.

Problem 6. For each of the following functions, plot the function on {x + iy | x, y ∈ [−1, 1]}
and describe what this view of the plot seems to imply about the function. Then plot the
function on a domain that allows you to see the true nature of the roots and poles and describe
how it is different from what the original plot implied. Use Markdown cells to write your
answers.

• f (z) = 100z 2 + z

• f (z) = sin 100z

1
.

(Hint: zoom way in.)

134 Lab 13. Visualizing Complex-valued Functions
14
The PageRank
Algorithm

Lab Objective: Many real-world systems—the internet, transportation grids, social media, and
so on—can be represented as graphs (networks). The PageRank algorithm is one way of ranking the
nodes in a graph by importance. Though it is a relatively simple algorithm, the idea gave birth to the
Google search engine in 1998 and has shaped much of the information age since then. In this lab we
implement the PageRank algorithm with a few different approaches, then use it to rank the nodes of
a few different networks.

The PageRank Model

The internet is a collection of webpages, each of which may have a hyperlink to any other page. One
possible model for a set of n webpages is a directed graph, where each node represents a page and
node j points to node i if page j links to page i. The corresponding adjacency matrix A satisfies
Aij = 1 if node j links to node i and Aij = 0 otherwise.

b c
a b c d
 
a 0 0 0 0
A= b 
 1 0 1 0 

c  1 0 0 1 
d 1 0 1 0
a d

Figure 14.1: A directed unweighted graph with four nodes, together with its adjacency matrix. Note
that the column for node b is all zeros, indicating that b is a sink —a node that doesn’t point to any
other node.

If n users start on random pages in the network and click on a link every 5 minutes, which page
in the network will have the most views after an hour? Which will have the fewest? The goal of the
PageRank algorithm is to solve this problem in general, therefore determining how “important” each
webpage is.

135
136 Lab 14. The PageRank Algorithm

Before diving into the mathematics, there is a potential problem with the model. What happens
if a webpage doesn’t have any outgoing links, like node b in Figure 14.1? Eventually, all of the users
will end up on page b and be stuck there forever. To obtain a more realistic model, modify each sink
in the graph by adding edges from the sink to every node in the graph. This means users on a page
with no links can start over by selecting a random webpage.

b c
a b c d
 
a 0 1 0 0
e= b 
A  1 1 1 0 

c  1 1 0 1 
d 1 1 1 0
a d

Figure 14.2: Here the graph in Figure 14.1 has been modified to guarantee that node b is no longer
a sink (the added links are blue). We denote the modified adjacency matrix by A.
e

Now let pk (t) be the likelihood that a particular internet user is surfing webpage k at time t.
Suppose at time t + 1, the user clicks on a link to page i. Then pi (t + 1) can be computed by counting
the number of links pointing to page i, weighted by the total number of outgoing links for each node.
As an example, consider the graph in Figure 14.2. To get to page a at time t + 1, the user had
to be on page b at time t. Since there are four outgoing links from page b, assuming links are chosen
with equal likelihood,

1
pa (t + 1) = pb (t).
4
Similarly, to get to page b at time t + 1, the user had to have been on page a, b, or c at time t. Since
a has 3 outgoing edges, b has 4 outgoing edges, and c has 2 outgoing edges,

1 1 1
pb (t + 1) = pa (t) + pb (t) + pc (t).
3 4 2
The previous equations can be written in a way that hints at a more general linear form:

1
pa (t + 1) = 0pa (t) + pb (t) + 0pc (t) + 0pd (t),
4
1 1 1
pb (t + 1) = pa (t) + pb (t) + pc (t) + 0pd (t).
3 4 2
The coefficients of the terms on the right hand side are precisely the entries of the ith row of the
modified adjacency matrix A,e divided by the jth column sum. In general, pi (t + 1) satisfies

n
eij P pj (t) .
X
pi (t + 1) = A n
(14.1)
j=1 k=1 A
ekj

Pn e
Note that the column sum k=1 A kj in the denominator can never be zero since, after the fix in
Figure 14.2, none of the nodes in the graph are sinks.
 137

Accounting for Boredom

The model in (14.1) assumes that the user can only click on links from their current page. It is more
realistic to assume that the user sometimes gets bored and randomly picks a new starting page. Let
0 ≤ ε ≤ 1, called the damping factor, be the probability that a user stays interested at step t. Then
the probability that the user gets bored at any time (and then chooses a new random page) is 1 − ε,
and (14.1) becomes
n
!
X p j (t) 1−ε
pi (t + 1) = ε Aeij P
n
+ . (14.2)
j=1 k=1 A
ekj n
| {z } | {z }
User stayed interested and User got bored and
clicked a link on the current page chose a random page

Note that (14.2) can be rewritten as the matrix equation

1−ε
p(t + 1) = εAp(t)
b + 1, (14.3)
n
where p(t) = [p1 (t), p2 (t), . . . , pn (t)]T , 1 is a vector of n ones, and A
b is the n × n matrix with entries

bij = P Aij
e
A . (14.4)
k=1 Akj
e

In other words, A
b is A
e normalized so that the columns each sum to 1. For the graph in Figure 14.2,
the matrix A
b is given by

a b c d
 
a 0 1/4 0 0
b= b 
A  1/3 1/4 1/2 0 
. (14.5)
c  1/3 1/4 0 1 
d 1/3 1/4 1/2 0

Problem 1. Write a class for representing directed graphs via their adjacency matrices. The
constructor should accept an n × n adjacency matrix A and a list of node labels (such as
[a, b, c, d]) defaulting to None. Modify A as in Figure 14.2 so that there are no sinks in
the corresponding graph, then calculate the A b from (14.4). Save A b and the list of labels as
attributes. Use [0, 1, . . . , n − 1] as the labels if none are provided. Finally, raise a ValueError
if the number of labels is not equal to the number of nodes in the graph.
(Hint: use array broadcasting to compute A b efficiently.)
For the graph in Figure 14.1, check that your A b matches (14.5).

Computing the Rankings

In the model (14.2), define the rank of node i as the limit

pi = lim pi (t).
t→∞

There are several ways to solve for p = limt→∞ p(t).

138 Lab 14. The PageRank Algorithm

Linear System

If p exists, then taking the limit as t → ∞ to both sides of (14.3) gives the following.
 
1−ε
lim p(t + 1) = lim εAp(t)
b + 1
t→∞ t→∞ n
1 − ε
p = εAp
b + 1
n
b p = 1 − ε1
 
I − εA (14.6)
n
This linear system is easy to solve as long as the number of nodes in the graph isn’t too large.

Eigenvalue Problem

Let E be an n × n matrix of ones. Then Ep(t) = 1 since i=1 pi (t) = 1. Substituting into (14.3),
P

 
1−ε 1−ε
p(t + 1) = εAp(t) +
b Ep(t) = εA + b E p(t) = Bp(t), (14.7)
n n

where B = εA
b+ 1−ε
n E. Now taking the limit at t → ∞ of both sides of (14.7),

Bp = p.

That is, p is an eigenvector of B corresponding to the eigenvalue λ = 1. In fact, since the columns
of B sum to 1, and because the entries of B are strictly positive (because the entries of E are all
positive), Perron’s theorem guarantees that λ = 1 is the unique eigenvalue of B of largest magnitude,
and that the corresponding eigenvector p is unique up to scaling. Furthermore, p can be scaled so
that each of its entires are positive, meaning p/∥p∥1 is the desired PageRank vector.

Note
A Markov chain is a weighted directed graph where each node represents a state of a discrete
system. The weight of the edge from node j to node i is the probability of transitioning from
state j to state i, and the adjacency matrix of a Markov chain is called a transition matrix.
Since B from (14.7) contains nonnegative entries and its columns all sum to 1, it can be
viewed as the transition matrix of a Markov chain. In that context, the limit vector p is called
the steady state of the Markov chain.

Iterative Method

Solving (14.6) or (14.7) is feasible for small networks, but they are not efficient strategies for very
large systems. The remaining option is to use an iterative technique. Starting with an initial guess
p(0), use (14.3) to compute p(1), p(2), . . . until ∥p(t) − p(t − 1)∥ is sufficiently small. From (14.7),
we can see that this is just the power method1 for finding the eigenvector corresponding to the
dominant eigenvalue of B.

1 See the Least Squares and Computing Eigenvalues lab for details on the power method.
 139

Problem 2. Add the following methods to your class from Problem 1. Each should accept a
damping factor ε (defaulting to 0.85), compute the PageRank vector p, and return a dictionary
mapping label i to its PageRank value pi .

1. linsolve(): solve for p in (14.6).

2. eigensolve(): solve for p using (14.7). Normalize the resulting eigenvector so its entries
sum to 1. Your answer should be a real number, so use .real to constrain the sum to
the reals (it might try to have an imaginary part equal to 0).

3. itersolve(): in addition to ε, accept an integer maxiter and a float tol. Iterate on

(14.3) until ∥p(t) − p(t − 1)∥1 < tol or t > maxiter. Use p(0) = [ n1 , n1 , . . . , n1 ]T as the
initial vector (any positive vector that sums to 1 will do, but this assumes equal starting
probabilities).

Check that each method yields the same results. For the graph in Figure 14.1 with ε = 0.85,
you should get the following dictionary mapping labels to PageRank values.

{'a': 0.095758635, 'b': 0.274158285, 'c': 0.355924792, 'd': 0.274158285}

Unit Test
The file test_pagerank.py contains a prewritten unit test to test your linsolve() method
for Problem 2. There is a place for you to add unit tests to test your other two methods for
Problem 2, eigensolve() and itersolve(), which will be graded.

Problem 3. Write a function that accepts a dictionary mapping labels to PageRank values,
like the outputs in Problem 2. Return a list of labels sorted from highest to lowest rank.
(Hint: if d is a dictionary, use list(d.keys()) and list(d.values()) to get the list of keys
and values in the dictionary, respectively.)
For the graph in Figure 14.1 with ε = 0.85, this is the list [c, b, d, a] (or [c, d, b, a], since
b and d have the same PageRank value).

Problem 4. The file web_stanford.txt contains information on Stanford University web-

pagesa and the hyperlinks between them, gathered in 2002.b Each line of the file is formatted
as a/b/c/d/e/f..., meaning the webpage with ID a has hyperlinks to webpages with IDs b,
c, d, and so on.
140 Lab 14. The PageRank Algorithm

Write a function that accepts a damping factor ε defaulting to 0.85. Read the data and
get a list of the n unique page IDs in the file (the labels). Construct the n × n adjacency matrix
of the graph where node j points to node i if webpage j has a hyperlink to webpage i. Use your
class from Problem 1 and its itersolve() method from Problem 2 to compute the PageRank
values of the webpages, then rank them with your function from Problem 3. In the case where
two webpages have the same rank, resolve ties by first listing the webpage whose ID comes first
alphabetically. Note that even though the IDs are numbers, we can sort them alphabetically
because they are defined as strings. (Hint: Sorting the list of unique webpage IDs by string
before ranking them will place the site IDs in the desired order; there is no need to convert the
IDs to integers.) Return the ranked list of webpage IDs.
(Hint: After constructing the list of webpage IDs, make a dictionary that maps a webpage ID
to its index in the list. For Figure 14.1, this would be {'a': 0, 'b': 1, 'c': 2, 'd': 3}.
The values are the row/column indices in the adjacency matrix for each label.)
With ε = 0.85, the top three ranked webpage IDs are 98595, 32791, and 28392.
a http://www.stanford.edu/
b See http://snap.stanford.edu/data/web-Stanford.html for the original (larger) dataset.

PageRank on Weighted Graphs

Nothing in the formulation of the PageRank model (14.3) requires that the edges of the graph are
unweighted. If Aij is the weight of the edge from node j to node i (weight 0 meaning there is no edge
from j to i), then the columns of A b still sum to 1. Thus B = εA
b + 1−ε E is still positive definite, so
n
we can expect a unique PageRank vector p to exist.
Adding weights to the edges can improve the fidelity of the model and produce a slightly more
realistic PageRank ordering. On a given webpage, for example, if hyperlinks to page a are clicked on
more frequently hyperlinks to page b, the edge from node a should be given more weight than the
edge to node b.

a b c d
 
a 0 0 0 0
A= b 
 2 0 1 0 

1 c  1 0 0 2 
1 d 1 0 2 0
1
b c
a b c d
1 a

0 1/4 0 0

1 2 2 2 b  1/2 1/4 1/3 0 
A=
b
1
 
c  1/4 1/4 0 1 
1 d 1/4 1/4 2/3 0
a d

Figure 14.3: A directed weighted graph with four nodes, together with its adjacency matrix and the
corresponding PageRank transition matrix. Edges that are added to fix sinks have weight 1, so the
computation of Ae and Ab are exactly the same as in Figure 14.2 and (14.4), respectively.
 141

Problem 5. The files ncaa2010.csv, ncaa2011.csv, . . ., ncaa2017.csv each contain data for
men’s college basketball for a given school year.a Each line (except the very first line, which is
a header) represents a different basketball game, formatted winning_team,losing_team.
Write a function that accepts a filename and a damping factor ε defaulting to 0.85. Read
the specified file (skipping the first line) and get a list of the n unique teams in the file. Construct
the n × n adjacency matrix of the graph where node j points to node i with weight w if team
j was defeated by team i in w games. That is, edges point from losers to winners. For
instance, the graph in Figure 14.3 would indicate that team c lost to team b once and to team
d twice, team b was undefeated, and team a never won a game. Use your class from Problem
1 and its itersolve() method from Problem 2 to compute the PageRank values of the teams,
then rank them with your function from Problem 3. Return the ranked list of team names.
Using ncaa2010.csv with ε = 0.85, the top three ranked teams (of the 607 total teams)
should be UConn, Kentucky, and Louisville, in that order. That season, UConn won the
championship, Kentucky was a semifinalist, and Louisville lost in the first tournament round
(a surprising upset).
a ncaa2010.csv has data for the 2010–2011 season, ncaa2011.csv for the 2011–2012 season, and so on.

Note
In Problem 5, the damping factor ε acts as an “upset” factor: a larger ε puts more emphasis on
win history; a smaller ε allows more randomness in the system, giving underdog teams a higher
probability of defeating a team with a better record.
It is also worth noting that the sink-fixing procedure is still reasonable for this model
because it gives every other team equal likelihood of beating an undefeated team. That is, the
additional edges don’t provide an extra advantage to any one team.

PageRank with NetworkX

NetworkX, usually imported as nx, is a third-party package for working with networks. It represents
graphs internally with dictionaries, thus taking full advantage of the sparsity in a graph. The base
class for directed graphs is called nx.DiGraph. Nodes and edges are usually added or removed
incrementally with the following methods.

Method Description
add_node() Add a single node.
add_nodes_from() Add a list of nodes.
add_edge() Add an edge between two nodes, adding the nodes if needed.
add_edges_from() Add multiple edges (and corresponding nodes as needed).
remove_edge() Remove a single edge (no nodes are removed).
remove_edges_from() Remove multiple edges (no nodes are removed).
remove_node() Remove a single node and all adjacent edges.
remove_nodes_from() Remove multiple nodes and all adjacent edges.

Table 14.1: Methods of the nx.DiGraph class for inserting or removing nodes and edges.
142 Lab 14. The PageRank Algorithm

For example, the weighted graph in Figure 14.3 can be constructed with the following code.

>>> import networkx as nx

# Initialize an empty directed graph.

>>> DG = nx.DiGraph()

# Add the directed edges (nodes are added automatically).

>>> DG.add_edge('a', 'b', weight=2) # a --> b (adds nodes a and b)
>>> DG.add_edge('a', 'c', weight=1) # a --> c (adds node c)
>>> DG.add_edge('a', 'd', weight=1) # a --> d (adds node d)
>>> DG.add_edge('c', 'b', weight=1) # c --> b
>>> DG.add_edge('c', 'd', weight=2) # c --> d
>>> DG.add_edge('d', 'c', weight=2) # d --> c

Once constructed, an nx.Digrah object can be queried for information about the nodes and
edges. It also supports dictionary-like indexing to access node and edge attributes, such as the weight
of an edge.

Method Description
has_node(A) Return True if A is a node in the graph.
has_edge(A,B) Return True if there is an edge from A to B.
edges() Iterate through the edges.
nodes() Iterate through the nodes.
number_of_nodes() Return the number of nodes.
number_of_edges() Return the number of edges.

Table 14.2: Methods of the nx.DiGraph class for accessing nodes and edges.

# Check the nodes and edges.

>>> DG.has_node('a')
True
>>> DG.has_edge('b', 'a')
False
>>> list(DG.nodes())
['a', 'b', 'c', 'd']
>>> list(DG.edges())
[('a', 'b'), ('a', 'c'), ('a', 'd'), ('c', 'b'), ('c', 'd'), ('d', 'c')]

# Change the weight of the edge (a, b) to 3.

>>> DG['a']['b']["weight"] += 1
>>> DG['a']['b']["weight"]
3

NetworkX efficiently implements several graph algorithms. The function nx.pagerank() com-
putes the PageRank values of each node iteratively with sparse matrix operations. This function
returns a dictionary mapping nodes to PageRank values, like the methods in Problem 2.
 143

# Calculate the PageRank values of the graph.

>>> nx.pagerank(DG, alpha=0.85) # alpha is the damping factor (epsilon).
{'a': 0.08767781186947843,
'b': 0.23613138394239835,
'c': 0.3661321209576019,
'd': 0.31005868323052127}

Achtung!
NetworkX also has a class, nx.Graph, for undirected graphs. The edges in an undirected graph
are bidirectional, so the corresponding adjacency matrix is symmetric.
The PageRank algorithm is not very useful for undirected graphs. In fact, the PageRank
value for node is close to its degree—the number of edges it connects to—divided by the total
number of edges. In Problem 5, that would mean the team who simply played the most games
would be ranked the highest. Always use nx.DiGraph, not nx.Graph, for PageRank and other
algorithms that rely on directed edges.

Problem 6. The file top250movies.txt contains data from the 250 top-rated movies accord-
ing to IMDb.a Each line in the file lists a movie title and its cast as title/actor1/actor2/...,
with the actors listed mostly in billing order (stars first), though some casts are listed alpha-
betically or in order of appearance.
Create a nx.DiGraph object with a node for each actor in the file. The weight from actor
a to actor b should be the number of times that actor a and b were in a movie together but actor
b was listed first. That is, edges point to higher-billed actors (see Figure 14.4). Compute
the PageRank values of the actors and use your function from Problem 3 to rank them. Return
the list of ranked actors.
(Hint: Consider using itertools.combinations() while constructing the graph. Also, use
encoding="utf-8" as an argument to open() to read the file, since several actors and actresses
have nonstandard characters in their names such as ø and æ.)
With ε = 0.7, the top three actors should be Leonardo DiCaprio, Robert De Niro, and
Tom Hanks, in that order.
a https://www.imdb.com/search/title?groups=top_250&sort=user_rating
144 Lab 14. The PageRank Algorithm

Anne
Hathaway

Scarlett 1 Hugh
Johansson Jackman

1 2

1 1 1 1

Christian Michael
Bale 4 Caine

Figure 14.4: A portion of the graph from Problem 6. Michael Caine was in four movies with Christian
Bale where Christian Bale was listed first in the cast.
 145

Additional Material
Sparsity
On very large networks, the PageRank algorithm becomes computationally difficult because of the
size of the adjacency matrix A. Fortunately, most adjacency matrices are highly sparse, meaning
the number of edges is much lower than the number of entries in the matrix. Consider adding
functionality to your class from Problem 1 so that it stores A
b as a sparse matrix and performs sparse
linear algebra operations in the methods from Problem 2 (use scipy.sparse.linalg).

PageRank as a Predictive Model

The data files in Problem 5 include tournament games for their respective seasons, so the resulting
rankings naturally align with the outcome of the championship. However, it is also useful to use
PageRank as a predictive model: given data for all regular season games, can the outcomes of the
tournament games be predicted? Over 40 million Americans fill out 60–100 million March Madness
brackets each year and bet over $9 billion on the tournament, so being able to predict the outcomes
of the games is a big deal. See http://games.espn.com/tournament-challenge-bracket for more
details.
Given regular season data, PageRank can be used to predict tournament results as in Problem
5. There are some pitfalls though; for example, how should ε be chosen? Using ε = .5 with
ncaa2010.csv minus tournament data (all but the last 63 games in the file) puts UConn—the actual
winner that year—in seventh place, while ε = .9 puts UConn in fourth. Both values for ε also rank
BYU as number one, but BYU lost in the Sweet Sixteen that year. In practice, Google uses .85 as
the damping factor, but there is no rigorous reasoning behind that particular choice.

Other Centrality Measures

In network theory, the centrality of a node refers to its importance. Since there are lots of ways to
measure importance, there are several different centrality measures.

• Degree centrality uses the degree of a node, meaning the number of edges adjacent to it (inde-
pendent of edge direction), for ranking. An academic paper that has been cited many times
has a high degree and is considered more important than a paper that has only been cited once.
• Eigenvector centrality is an extension of degree centrality. Instead of each neighbor contributing
equally to the centrality, nodes that are important are given a higher weight. Thus a node
connected to lots of unimportant nodes can have the same measure as a node connected to a
few, important nodes. Eigenvector centrality is measured by the eigenvector associated with
the largest eigenvalue of the adjacency matrix of the network.
• Katz centrality is a modification to eigenvalue centrality for directed networks. Outgoing nodes
contribute centrality to their neighbors, so an important node makes its neighbors more im-
portant.
• PageRank adapts Katz centrality by averaging out the centrality that a node can pass to its
neighbors. For example, if Google—a website that should have high centrality—points to a
million websites, then it shouldn’t pass on that high centrality to all of million of its neighbors,
so each neighbor gets one millionth of Google’s centrality.

For more information on these centralities, as well as other ways to measure node importance,
see [New10].
146 Lab 14. The PageRank Algorithm
15
Iterative Solvers

Lab Objective: Many real-world problems of the form Ax = b have tens of thousands of
parameters. Solving such systems with Gaussian elimination or matrix factorizations could require
trillions of floating point operations (FLOPs), which is of course infeasible. Solutions of large systems
must therefore be approximated iteratively. In this lab we implement three popular iterative methods
for solving large systems: Jacobi, Gauss-Seidel, and Successive Over-Relaxation.

Iterative methods are often useful to solve large systems of equations. In this lab, let x(k) denote
the kth iteration of the iterative method for solving the problem Ax = b for x. Furthermore, let xi
(k)
be the ith component of x so that xi is the ith component of x in the kth iteration. Like other
iterative methods, there are two stopping parameters: a very small ε > 0 and an integer N ∈ N.
Iterations continue until either
∥x(k−1) − x(k) ∥ < ε or k > N. (15.1)

The Jacobi Method

The Jacobi Method is a simple but powerful method used for solving certain kinds of large linear
systems. The main idea is simple: solve for each variable in terms of the others, then use the previous
values to update each approximation. As a (very small) example, consider the 3 × 3
2x1 − x3 = 3,
−x1 + 3x2 + 2x3 = 3,
+ x2 + 3x3 = −1.
Solving the first equation for x1 , the second for x2 , and the third for x3 yields
1
x1 = 2 (3 + x3 ),
1
x2 = 3 (3 + x1 − 2x3 ),
1
x3 = 3 (−1 − x2 ).

(0) (0) (0)

Now begin with an initial guess x(0) = [x1 , x2 , x3 ]T = [0, 0, 0]T . To compute the first
approximation x(1) , use the entries of x(0) as the variables on the right side of the previous equations:
(1) 1 (0) 1 3
x1 = 2 (3 + x3 ) = 2 (3
+ 0) = 2,
(1) 1 (0) (0) 1
x2 = 3 (3 + x1 − 2x3 ) = 3 (3 + 0 − 0) = 1,
(1) 1 (0) 1 1
x3 = 3 (−1 − x2 ) = 3 (−1 − 0) = − 3.

147
148 Lab 15. Iterative Solvers

Thus x(1) = [ 32 , 1, − 13 ]T . Computing x(2) is similar:

(2) 1 (1) 1
x1 = 2 (3 + x3 ) = 2 (3− 13 ) = 4
3,
(2) 1 (1) (1) 1 3 2 31
x2 = 3 (3 + x1 − 2x3 ) = 3 (3 + 2 + 3 ) = 18 ,
(2) 1 (1) 1 2
x3 = 3 (−1 − x2 ) = 3 (−1 − 1) = − 3.

The process is repeated until at least one of the two stopping criteria in (15.1) is met. For this
particular problem, convergence to 8 decimal places (ε = 10−8 ) is reached in 29 iterations.

(k) (k) (k)

x1 x2 x3
(0)
x 0 0 0
x(1) 1.5 1 −0.33333333
x(2) 1.33333333 1.72222222 −0.66666667
x(3) 1.16666667 1.88888889 −0.90740741
x(4) 1.04629630 1.99382716 −0.96296296
.. .. .. ..
. . . .
x(28) 0.99999999 2.00000001 −0.99999999
x(29) 1 2 −1

Matrix Representation
The iterative steps performed above can be expressed in matrix form. First, decompose A into its
diagonal entries, its entries below the diagonal, and its entries above the diagonal, as A = D + L + U .
 

a11 0 . . . 0
 
0 0 ... 0
 0 a12 . . . a1n
..
 0 a22 . . . 0   a21 0 ... 0   0 0 ... .
 

 .. .. . ..   .. . . .. 
   
. . . . . .
 
 . . . .   . . . .   ..

.. .. a

n−1,n

0 0 . . . ann an1 . . . an,n−1 0 0 0 ... 0
D L U

With this decomposition, x can be expressed in the following way.

Ax = b
(D + L + U )x = b
Dx = −(L + U )x + b
x = D−1 (−(L + U )x + b)

Now using x(k) as the variables on the right side of the equation to produce x(k+1) on the left,
and noting that L + U = A − D, we have the following.

x(k+1) = D−1 (−(A − D)x(k) + b)

= D−1 (Dx(k) − Ax(k) + b)
= x(k) + D−1 (b − Ax(k) ) (15.2)

There is a potential problem with (15.2): calculating a matrix inverse is the cardinal sin of
numerical linear algebra, yet the equation contains D−1 . However, since D is a diagonal matrix,
D−1 is also diagonal, and is easy to compute.
 149

1
0 ... 0
 
a11
 0 1
a22 ... 0 
D−1 =  . .. .. ..
 
 .. .

. . 
1
0 0 ... ann

Because of this, the Jacobi method requires that A have nonzero diagonal entries.
The diagonal D can be represented by the 1-dimensional array d of the diagonal entries. Then
the matrix multiplication Dx is equivalent to the component-wise vector multiplication d ∗ x = x ∗ d.
Likewise, the matrix multiplication D−1 x is equivalent to the component-wise “vector division” x/d.

Problem 1. Write a function that accepts a matrix A, a vector b, a convergence tolerance tol
defaulting to 10−8 , and a maximum number of iterations maxiter defaulting to 100. Implement
the Jacobi method using (15.2), returning the approximate solution to the equation Ax = b.
Run the iteration until ∥x(k−1) − x(k) ∥∞ < tol, and only iterate at most maxiter times.
Avoid using la.inv() to calculate D−1 , but use la.norm() to calculate the vector ∞-norm.
Your function should be robust enough to accept systems of any size. To test your
function, generate a random b with np.random.random() and use the following function to
generate an n × n matrix A for which the Jacobi method is guaranteed to converge. Run the
iteration, then check that Ax(k) and b are close using np.allclose(). There is a file called
test_iterative_solvers.py that contains prewritten unit tests to test your function for this
problem.

def diag_dom(n, num_entries=None, as_sparse=False):

"""Generate a strictly diagonally dominant (n, n) matrix.
Parameters:
n (int): The dimension of the system.
num_entries (int): The number of nonzero values.
Defaults to n^(3/2)-n.
as_sparse: If True, an equivalent sparse CSR matrix is returned.
Returns:
A ((n,n) ndarray): A (n, n) strictly diagonally dominant matrix.
"""
if num_entries is None:
num_entries = int(n**1.5) - n
A = sparse.dok_matrix((n,n))
rows = np.random.choice(n, size=num_entries)
cols = np.random.choice(n, size=num_entries)
data = np.random.randint(-4, 4, size=num_entries)
for i in range(num_entries):
A[rows[i], cols[i]] = data[i]
B = A.tocsr() # convert to row format for the next step
for i in range(n):
A[i,i] = abs(B[i]).sum() + 1
return A.tocsr() if as_sparse else A.toarray()
150 Lab 15. Iterative Solvers

Also test your function on random n × n matrices. If the iteration is non-convergent, the
successive approximations will have increasingly large entries.

Convergence
Most iterative methods only converge under certain conditions. For the Jacobi method, convergence
mostly depends on the nature of the matrix A. If the entries aij of A satisfy the property
X
|aii | > |aij | for all i = 1, 2, . . . , n,
j̸=i

then A is called strictly diagonally dominant (diag_dom() in Problem 1 generates a strictly diagonally
dominant n × n matrix). If this is the case,1 then the Jacobi method always converges, regardless of
the initial guess x0 . This is a very different convergence result than many other iterative methods
such as Newton’s method where convergence is highly sensitive to the initial guess.
There are a few ways to determine whether or not an iterative method is converging. For
example, since the approximation x(k) should satisfy Ax(k) ≈ b, the normed difference ∥Ax(k) − b∥∞
should be small. This value is called the absolute error of the approximation. If the iterative method
converges, the absolute error should decrease to ε.

Problem 2. Modify your Jacobi method function in the following ways.

1. Add a keyword argument called plot, defaulting to False.

2. Keep track of the absolute error ∥Ax(k) − b∥∞ of the approximation at each iteration.

3. If plot is True, produce a lin-log plot (use plt.semilogy()) of the error against iteration
count. Remember to still return the approximate solution x.

If the iteration converges, your plot should resemble the following figure.

1 Although this seems like a strong requirement, most real-world linear systems can be represented by strictly

diagonally dominant matrices.

 151

Convergence of Jacobi Method

Absolute Error of Approximation

10 1

10 3

10 5

10 7

0 5 10 15 20 25 30
Iteration

The Gauss-Seidel Method

The Gauss-Seidel method is essentially a slight modification of the Jacobi method. The main differ-
ence is that in Gauss-Seidel, new information is used immediately. As an example, consider again
the system from the previous section,

2x1 − x3 = 3,
−x1 + 3x2 + 2x3 = 3,
+ x2 + 3x3 = −1.

As with the Jacobi method, solve for x1 in the first equation, x2 in the second equation, and
x3 in the third equation:
1
x1 = 2 (3 + x3 ),
1
x2 = 3 (3 + x1 − 2x3 ),
1
x3 = 3 (−1 − x2 ).

(1)
Using x(0) to compute x1 in the first equation as before,

(1) 1 (0) 1 3
x1 = (3 + x3 ) = (3 + 0) = .
2 2 2
(1) (1)
Now, however, use the updated value of x1 in the calculation of x2 :

(1) 1 (1) (0) 1 3 3

x2 = (3 + x1 − 2x3 ) = (3 + − 0) = .
3 3 2 2
(1) (1) (1)
Likewise, use the updated values of x1 and x2 to calculate x3 :

(1) 1 (1) 1 3 5
x3 = (−1 − x2 ) = (−1 − ) = − .
3 3 2 6
This process of using calculated information immediately is called forward substitution, and causes
the algorithm to (generally) converge much faster.
152 Lab 15. Iterative Solvers

(k) (k) (k)

x1 x2 x3
x(0) 0 0 0
x(1) 1.5 1.5 −0.83333333
x(2) 1.08333333 1.91666667 −0.97222222
x(3) 1.01388889 1.98611111 −0.99537037
x(4) 1.00231481 1.99768519 −0.99922840
.. .. .. ..
. . . .
x(11) 1.00000001 1.99999999 −1
x(12) 1 2 −1

Notice that Gauss-Seidel converges in less than half as many iterations as Jacobi does for this system.

Implementation
Because Gauss-Seidel updates only one element of the solution vector at a time, the iteration cannot
be summarized by a single matrix equation. Instead, the process is most generally described by the
equation  
(k+1) 1 X (k)
X (k)
xi = bi − aij xj − aij xj  . (15.3)
aii j<i j>i

Let ai be the ith row of A. The two sums closely resemble the regular vector product of ai
(k)
and x(k) without the ith term aii xi . This suggests the simplification

(k+1) 1  (k)

xi = bi − aTix
(k)
+ aii xi
aii
(k) 1  
= xi + bi − a T
i x (k)
. (15.4)
aii
One sweep through all the entries of x completes one iteration.

Problem 3. Write a function that accepts a matrix A, a vector b, a convergence tolerance

tol defaulting to 10−8 , a maximum number of iterations maxiter defaulting to 100, and a
keyword argument plot that defaults to False. Implement the Gauss-Seidel method using
(15.4), returning the approximate solution to the equation Ax = b.
Use the same stopping criterion as in Problem 1. Also keep track of the absolute errors
of the iteration, as in Problem 2. If plot is True, plot the error against iteration count. Use
diag_dom() to generate test cases.
 153

Achtung!
Since the Gauss-Seidel algorithm operates on the approximation vector in place (modifying
it one entry at a time), the previous approximation x(k−1) must be stored at the beginning
of the kth iteration in order to calculate ∥x(k−1) − x(k) ∥∞ . Additionally, since NumPy
arrays are mutable, the past iteration must be stored as a copy.

>>> x0 = np.random.random(5) # Generate a random vector.

>>> x1 = x0 # Attempt to make a copy.
>>> x1[3] = 1000 # Modify the "copy" in place.
>>> np.allclose(x0, x1) # But x0 was also changed!
True

# Instead, make a copy of x0 when creating x1.

>>> x0 = np.copy(x1) # Make a copy.
>>> x1[3] = -1000
>>> np.allclose(x0, x1)
False

Convergence
Whether or not the Gauss-Seidel method converges depends on the nature of A. If all of the eigenval-
ues of A are positive, A is called positive definite. If A is positive definite or if it is strictly diagonally
dominant, then the Gauss-Seidel method converges regardless of the initial guess x(0) .

Solving Sparse Systems Iteratively

Iterative solvers are best suited for solving very large sparse systems. However, using the Gauss-Seidel
method on sparse matrices requires translating code from NumPy to scipy.sparse. The algorithm
is the same, but there are some functions that are named differently between these two packages.2

Problem 4. Write a new function that accepts a sparse matrix A, a vector b, a convergence
tolerance tol, and a maximum number of iterations maxiter (plotting the convergence is not
required for this problem). Implement the Gauss-Seidel method using (15.4), returning the
approximate solution to the equation Ax = b. Use the usual default stopping criterion.
The Gauss-Seidel method requires extracting the rows Ai from the matrix A and com-
puting ATi x. There are many ways to do this that cause some fairly serious runtime issues, so
we provide the code for this specific portion of the algorithm.

# Get the indices of where the i-th row of A starts and ends if the
# nonzero entries of A were flattened.
rowstart = A.indptr[i]

2 See the lab on Linear Systems for a review of scipy.sparse matrices and syntax.
154 Lab 15. Iterative Solvers

rowend = A.indptr[i+1]

# Multiply only the nonzero elements of the i-th row of A with the
# corresponding elements of x.
Aix = A.data[rowstart:rowend] @ x[A.indices[rowstart:rowend]]

To test your function, remeber to call diag_dom() using as_sparse=True

>>> A = diag_dom(1000, as_sparse=True)

>>> b = np.random.random(1000)

Unit Test
There is a file called test_iterative_solvers.py that contains prewritten unit tests for Prob-
lem 1. There is a place for you to add your own unit tests to test your function for Problem 4,
which will be graded.

Successive Over-Relaxation
There are many systems that meet the requirements for convergence with the Gauss-Seidel method,
but for which convergence is still relatively slow. A slightly altered version of the Gauss-Seidel
method, called Successive Over-Relaxation (SOR), can result in faster convergence. This is achieved
by introducing a relaxation factor ω ≥ 1 and modifying (15.3) as
 
(k+1) (k) ω  X (k)
X (k) 
xi = (1 − ω)xi + bi − aij xj − aij xj .
aii j<i j>i

Simplifying the equation, we have

(k+1) (k) ω  
xi = xi + bi − a T
i x(k)
. (15.5)
aii

Note that when ω = 1, SOR reduces to Gauss-Seidel. The relaxation factor ω weights the new
iteration between the current best approximation and the next approximation in a way that can
sometimes dramatically improve convergence.

Problem 5. Write a function that accepts a sparse matrix A, a vector b, a relaxation factor
ω, a convergence tolerance tol, and a maximum number of iterations maxiter. Implement
SOR using (15.5), compute the approximate solution to the equation Ax = b. Use the usual
stopping criterion. Return the approximate solution x as well as a boolean indicating whether
the function converged and the number of iterations computed.
(Hint: this requires changing only one line of code from the sparse Gauss-Seidel function.)
 155

Figure 15.1: On the left, an example of a 6 × 6 grid (n = 4) where the red dots are hot boundary
zones and the blue dots are cold boundary zones. On the right, the green dots are the neighbors of
the interior black dot that are used to approximate the heat at the black dot.

A Finite Difference Method

Laplace’s equation is an important partial differential equation that arises often in both pure and
applied mathematics. In two dimensions, the equation has the following form.

∂2u ∂2u
+ 2 =0 (15.6)
∂x2 ∂y

Laplace’s equation can be used to model heat flow. Consider a square metal plate where the
top and bottom borders are fixed at 0◦ Celsius and the left and right sides are fixed at 100◦ Celsius.
Given these boundary conditions, we want to describe how heat diffuses through the rest of the plate.
The solution to Laplace’s equation describes the plate when it is in a steady state, meaning that the
heat at a given part of the plate no longer changes with time.
It is possible to solve (15.6) analytically. However, the problem can also be solved numerically
using a finite difference method. To begin, we impose a discrete, square grid on the plate with uniform
spacing. Denote the points on the grid by (xi , yj ) and the value of u at these points (the heat) as
u(xi , yj ) = Ui,j . Using the centered difference quotient for second derivatives to approximate the
partial derivatives,

∂2u ∂2u
0= + 2
∂x2 ∂y
Ui+1,j − 2Ui,j + Ui−1,j Ui,j+1 − 2Ui,j + Ui,j−1
≈ +
h2 h2
1
= 2 (−4Ui,j + Ui+1,j + Ui−1,j + Ui,j+1 + Ui,j−1 ) , (15.7)
h
where h = xi+1 − xi = yj+1 − yj is the distance between the grid points in either direction. This
problem can be formulated as a linear system. Suppose the grid has exactly (n + 2) × (n + 2) entries.
Then the interior of the grid (where u(x, y) is unknown) is n × n, and can be flattened into an n2 × 1
vector u. The entire first row goes first, then the second row, proceeding to the nth row.
 T
u = U1,1 U1,2 ··· U1,n U2,1 U2,2 ··· U2,n ··· Un,n

From (15.7), for an interior point Ui,j , we have

−4Ui,j + Ui+1,j + Ui−1,j + Ui,j+1 + Ui,j−1 = 0. (15.8)

156 Lab 15. Iterative Solvers

If any of the neighbors to Ui,j is a boundary point on the grid, its value is already determined by the
boundary conditions. For example, the neighbor U3,0 of the gridpoint for U3,1 is fixed at U3,0 = 100.
In this case, (15.8) becomes

−4U3,1 + U2,1 + U3,2 + U4,1 = −100.

The constants on the right side of (15.8) become the n2 × 1 vector b. All nonzero entries of b
correspond to interior points that touch the left or right boundaries.
As an example, writing (15.8) for the 16 interior points of the grid in Figure 15.1 results in the
following 16 × 16 system Au = b. Note the block structure (empty blocks are all zeros).
−4 −100
    
1 0 0 1 0 0 0 U1,1

 1 −4 1 0 0 1 0 0 
 U1,2  
  0 


 0 1 −4 1 0 0 1 0 
 U1,3  
  0 


 0 0 1 −4 0 0 0 1 
 U1,4  
  −100 


 1 0 0 0 −4 1 0 0 1 0 0 0 
 U2,1  
  −100 


 0 1 0 0 1 −4 1 0 0 1 0 0 
 U2,2  
  0 


 0 0 1 0 0 1 −4 1 0 0 1 0 
 U2,3  
  0 

0 0 0 1 0 0 1 −4 0 0 0 1 U2,4 −100
    
=
    
1 0 0 0 −4 1 0 0 1 0 0 0 U3,1 −100
  
    
0 1 0 0 1 −4 1 0 0 1 0 0 U3,2 0
    
    
−4
    
 0 0 1 0 0 1 1 0 0 1 0  U3,3   0 
    

 0 0 0 1 0 0 1 −4 0 0 0 1 
 U3,4  
  −100 


 1 0 0 0 −4 1 0 0 
 U4,1  
  −100 


 0 1 0 0 1 −4 1 0 
 U4,2  
  0 

 0 0 1 0 0 1 −4 1  U4,3   0 
0 0 0 1 0 0 1 −4 U4,4 −100

More concisely, for any positive integer n, the matrix A can be written as

−4
   
B I 1
 I B I   1 −4 1 
.. .. .. ..
   
. . where B =  . .  is n × n.
   
A= I , 1
.. .. .. ..
   
. . . .
   
 I   1 
I B 1 −4

Problem 6. Write a function that accepts an integer n, a relaxation factor ω, a convergence

tolerance tol that defaults to 10−8 , a maximum number of iterations maxiter that defaults
to 100, and a bool plot that defaults to False. Generate and solve the corresponding system
Au = b using Problem 5. Also return a boolean indicating whether the function converged and
the number of iterations computed.
(Hint: see Problem 5 of the Linear Systems lab for the construction of A. Also, np.tile()
may be useful for constructing b.)
If plot=True, visualize the solution u with a heatmap using plt.pcolormesh() (the
colormap "coolwarm" is a good choice in this case). This shows the distribution of heat over
the hot plate after it has reached its steady state. Note that the u must be reshaped as an
n × n array to properly visualize the result.
 157

Problem 7. To demonstrate how convergence is affected by the value of the relaxation factor
ω in SOR, run your function from Problem 6 with ω = 1, 1.05, 1.1, . . . , 1.9, 1.95 and n = 20.
Plot the number of computed iterations as a function of ω. Return the value of ω that results
in the least number of iterations.
Note that the matrix A from Problem 6 is not strictly diagonally dominant. However,
A is positive definite, so the algorithm will converge. Unfortunately, convergence for these
kinds of systems usually requires more iterations than for strictly diagonally dominant systems.
Therefore, set tol=1e-2 and maxiter=1000.
Recall that ω = 1 corresponds to the Gauss-Seidel method. Choosing a more optimal
relaxation factor saves a large number of iterations. This could translate to saving days or
weeks of computation time while solving extremely large linear systems on a supercomputer.
158 Lab 15. Iterative Solvers
16
The Drazin Inverse

Lab Objective: The Drazin inverse of a matrix is a pseudoinverse which preserves certain spectral
properties of the matrix. In this lab we compute the Drazin inverse using the Schur decomposition,
then use it to compute the effective resistance of a graph and perform link prediction.

Definition of the Drazin Inverse

The index of an n × n matrix A is the smallest nonnegative integer k for which N (Ak ) = N (Ak+1 ).
The Drazin inverse AD of A is the unique n × n matrix satisfying the following properties.

• AAD = AD A

• Ak+1 AD = Ak

• AD AAD = AD

Note that if A is invertible, in which case k = 0, then AD = A−1 . On the other hand, if A is nilpotent,
meaning Aj = 0 for some nonnegative integer j, then AD is the zero matrix.

Problem 1. Write a function that accepts an n × n matrix A, the index k of A, and an n × n

matrix AD . Use the criteria described above to determine whether or not AD is the Drazin
inverse of A. Return True if AD satisfies all three conditions; otherwise, return False.
Use the following matrices as test cases for your function.
   
1 3 0 0 1 −3 9 81
 0 1 3 0  D
 0 1 −3 −18 
A=  0 0 1 3 , A =  0 0
  , k = 1
1 3 
0 0 0 0 0 0 0 0
   
1 1 3 0 0 0
B= 5 2 6 , BD = 0 0 0 , k=3
−2 −1 −3 0 0 0
(Hint: np.allclose() and np.linalg.matrix_power() may be useful).

159
160 Lab 16. The Drazin Inverse

Computing the Drazin Inverse

The Drazin inverse is often defined theoretically in terms of the eigenprojections of a matrix. However,
eigenprojections are often costly or unstable to calculate, so we resort to a different method to
calculate the Drazin inverse.
Every n × n matrix A can be written in the form
 
M 0
A=S −1
S, (16.1)
0 N

where S is a change of basis matrix, M is nonsingular, and N is nilpotent. Then the Drazin inverse
can be calculated as
M −1 0
 
D
A =S −1
S. (16.2)
0 0
To put A into the form in (16.1), we can use the Schur decomposition of A, given by

A = QT Q−1 , (16.3)

where Q is orthonormal and T is upper triangular. Since T is similar to A, the eigenvalues of A are
listed along the diagonal of T . If A is singular, at least one diagonal entry of T must be 0.
In general, Schur decompositions are not unique; the eigenvalues along the diagonal of T can
be reordered. To find M , N , and S, we compute the Schur decomposition of A twice, ordering the
eigenvalues differently in each decomposition.
First, we sort so that the nonzero eigenvalues are listed first along the diagonal of T . Then, if
k is the number of nonzero eigenvalues, the upper left k × k block of T forms the nonsingular matrix
M , and the first k columns of Q form the first k columns of the change of basis matrix S.
Computing the decomposition a second time, we reorder so that the 0 eigenvalues are listed
first along the diagonal of T . Then the upper left (n − k) × (n − k) block forms the nilpotent matrix
N , and the first n − k columns of Q form the last n − k columns of S. This completes a change of
basis matrix that will put A into the desired block diagonal form. Lastly, we use (16.2) to compute
AD .
SciPy’s la.schur() is a routine for computing the Schur decomposition of a matrix, but it
does not automatically sort it by eigenvalue. However, sorting can be accomplished by specifying the
sort keyword argument. Given an eigenvalue, the sorting function should return a boolean indicating
whether to sort that eigenvalue to the top left of the diagonal of T .

>>> from scipy import linalg as la

# The standard Schur decomposition.

>>> A = np.array([[0,0,2],[-3,2,6],[0,0,1]])
>>> T,Z = la.schur(A)
>>> T # The eigenvalues (2, 0, and 1) are not sorted.
array([[ 2., -3., 6.],
[ 0., 0., 2.],
[ 0., 0., 1.]])

# Specify a sorting function to get the desired result.

>>> f = lambda x: abs(x) > 0
>>> T1,Z1,k = la.schur(A, sort=f)
>>> T1
 161

array([[ 2. , 0. , 6.70820393],
[ 0. , 1. , 2. ],
[ 0. , 0. , 0. ]])
>>> k # k is the number of columns satisfying the sort,
2 # which is the number of nonzero eigenvalues.

The procedure for finding the Drazin inverse using the Schur decomposition is given in Algo-
rithm 1. Due to possible floating point arithmetic errors, consider all eigenvalues smaller than a
certain tolerance to be 0.

Algorithm 1
1: procedure Drazin(A, tol)
2: (n, n) ← shape(A)
3: T1 , Q1 , k1 ← schur(A, |x| > tol) ▷ Sort the Schur decomposition with 0 eigenvalues last.
4: T2 , Q2 , k2 ← schur(A, |x| ≤ tol) ▷ Sort the Schur decomposition with 0 eigenvalues first.
5: U ← [Q1:,:k1 | Q2:,:n−k1 ] ▷ Create change of basis matrix.
6: U −1 ← inverse(U)
7: V ← U −1 AU ▷ Find block diagonal matrix in (16.1)
8: Z ← 0n×n
9: if k1 ̸= 0 then
10: M −1 ← inverse(V:k1 ,:k1 )
11: Z:k1 ,:k1 ← M −1
12: return U ZU −1

Problem 2. Write a function that accepts an n × n matrix A and a tolerance for rounding
eigenvalues to zero. Use Algorithm 1 to compute the Drazin inverse AD . Use your function
from Problem 1 to verify your implementation.

Achtung!
Because the algorithm for the Drazin inverse requires calculation of the inverse of a matrix, it
is unstable when that matrix has a high condition number. If the algorithm does not find the
correct Drazin inverse, check the condition number of V from Algorithm 1

Note
The Drazin inverse is called a pseudoinverse because AD = A−1 for invertible A, and for
noninvertible A, AD always exists and acts similarly to an inverse. There are other matrix
pseudoinverses that preserve different qualities of A, including the Moore-Penrose pseudoinverse
A† , which can be thought of as the least squares approximation to A−1 .
162 Lab 16. The Drazin Inverse

Applications of the Drazin Inverse

Effective Resistance
The effective resistance between two nodes in a undirected graph is a measure of how connected
those nodes are. The concept originates from the study of circuits to measure the resistance between
two points on the circuit. A resistor is a device in a circuit which limits or regulates the flow of
electricity. Two points that have more resistors between them have more resistance, while those with
fewer resistors between them have less resistance. The entire circuit can be represented by a graph
where the nodes are the points of interest and the number of edges connecting two nodes indicates
the number of resistors between the corresponding points. See Figure 16.1 for an example.

a c e

b d f
Figure 16.1: A graph with a resistor on each edge.

In electromagnetism, there are rules for manually calculating the effective resistance between
two nodes for relatively simple graphs. However, this is infeasible for large or complicated graphs.
Instead, we can use the Drazin inverse to calculate effective resistance for any graph.
First, create the adjacency matrix 1 of the graph, the matrix where the (ij)th entry is the
number of connections from node i to node j. Next, calculate the Laplacian L of the adjacency
matrix. Then if Rij is the effective resistance from node i to node j,
(
e j )D if i ̸= j
(L
Rij = ii
(16.4)
0 if i = j,

where L
e j is the Laplacian with the jth row of the Laplacian replaced by the jth row of the identity
matrix, and (Le j )D is its Drazin inverse.

Problem 3. Write a function that accepts the n × n adjacency matrix of an undirected graph.
Use (16.4) to compute the effective resistance from each node to every other node. Return an
n × n matrix where the (ij)th entry is the effective resistance from node i to node j. Keep the
following in mind:

• The resulting matrix should be symmetric.

• The effective resistance from a node to itself is 0.

1 See Problem 1 of Image Segmentation for a refresher on adjacency matrices and the Laplacian.
 163

• Consider creating the matrix column by column instead of entry by entry. Every time
you compute the Drazin inverse, the whole diagonal of the matrix can be used.

Test your function using the graphs and values from Figure 16.2.

Rac = 2, Rad = 3 Rab = 1

a b c d a b

Rab = 23 , Rac = 2
3 a b Rab = 1
3
a b

1 1
a b Rab = 2 a b Rab = 4

Figure 16.2: The effective resistance between two points for several simple graphs. Nodes that are
farther apart have a larger effective resistance, while nodes that are nearer or better connected have
a smaller effective resistance.

Link Prediction
Link prediction is the problem of predicting the likelihood of a future association between two uncon-
nected nodes in a graph. Link prediction has application in many fields, but the canonical example
is friend suggestions on Facebook. The Facebook network can be represented by a large graph where
each user is a node, and two nodes have an edge connecting them if they are “friends.” Facebook
aims to predict who you would like to become friends with in the future, based on who you are
friends with now, as well as discover which friends you may have in real life that you have not yet
connected with online. To do this, Facebook must have some way to measure how closely two users
are connected.
We will compute link prediction using effective resistance as a metric. Effective resistance
measures how closely two nodes are connected, and nodes that are closely connected at present are
more likely to be connected in the future. Given an undirected graph, the next link should connect
the two unconnected nodes with the least effective resistance between them.

Problem 4. Write a class called LinkPredictor for performing link prediction. Implement
the __init__() method so that it accepts the name of a csv file containing information about a
social network. Each row of the file should contain the names of two nodes which are connected
by an (undirected) edge.
164 Lab 16. The Drazin Inverse

Store each of the names of the nodes of the graph as an ordered list. Next, create the
adjacency matrix for the network where the ith row and column of the matrix correspond to the
ith member of the list of node names. Finally, use your function from Problem 3 to compute
the effective resistance matrix. Save the list of names, the adjacency matrix, and the effective
resistance matrix as attributes.

Problem 5. Implement the following methods in the LinkPredictor class:

1. predict_link(): Accept a parameter node which is either None or a string representing a

node in the network. If node is None, return a tuple with the names of the nodes between
which the next link should occur. However, if node is a string, return the name of the
node which should be connected to node next out of all other nodes in the network. If
node is not in the network, raise a ValueError. Take the following into consideration:

(a) You want to find the two nodes which have the smallest effective resistance between
them which are not yet connected. Use information from the adjacency matrix to
zero out all entries of the effective resistance matrix that represent connected nodes.
The “*" operator multiplies arrays component-wise, which may be helpful.
(b) Find the next link by finding the minimum value of the array that is nonzero. Your
array may be the whole matrix or just a column if you are only considering links for
a certain node. This can be accomplished by passing np.min() a masked version of
your matrix to exclude entries that are 0.
(c) NumPy’s np.where() is useful for finding the minimum value in an array:

>>> A = np.random.randint(-9,9,(3,3))
>>> A
array([[ 6, -8, -9],
[-2, 1, -1],
[ 4, 0, -3]])

# Find the minimum value in the array.

>>> minval = np.min(A)
>>> minval
-9

# Find the location of the minimum value.

>>> loc = np.where(A==minval)
>>> loc
(array([0], dtype=int64), array([2], dtype=int64))

2. add_link(): Take as input two names of nodes, and add a link between them. If either
name is not in the network, raise a ValueError. Add the link by updating the adjacency
matrix and the effective resistance matrix.
 165

Figure 16.3 visualizes the data in social_network.csv. Use this graph to verify that
your class is suggesting plausible new links. You should observe the following:

• In the entire network, Emily and Oliver are most likely to become friends next.

• Melanie is predicted to become friends with Carol next.

• Alan is expected to become friends with Sonia, then with Piers, and then with Abigail.

Figure 16.3: The social network contained in social_network.csv. Adapted from data by Wayne.
W Zachary (see https://en.wikipedia.org/wiki/Zachary%27s_karate_club).

1. Piers 10. Alan 19. Max 28. Thomas

2. Abigail 11. Trevor 20. Eric
29. Christopher
3. Oliver 12. Jake 21. Theresa
30. Charles
4. Stephanie 13. Mary 22. Paul
5. Carol 14. Anna 23. Alexander 31. Madeleine
6. Melanie 15. Ruth 24. Colin
32. Tracey
7. Stephen 16. Evan 25. Jake
33. Sonia
8. Sally 17. Connor 26. Jane
9. Penelope 18. John 27. Brandon 34. Emily
166 Lab 16. The Drazin Inverse
17
The Arnoldi Iteration

Lab Objective: The Arnoldi Iteration is an efficient method for finding the eigenvalues of extremely
large matrices. Instead of using standard methods, the iteration uses Krylov subspaces to approximate
how a linear operator acts on vectors. With this approach, the Arnoldi Iteration facilitates the
computation of eigenvalues for enormous matrices without needing to physically create the matrix
in memory. We will explore this subject by implementing the Arnoldi iteration algorithm, using our
implementation for eigenvalue computation, and then graphically representing the accuracy of our
approximated eigenvalues.

Krylov Subspaces
One of the biggest difficulties in numerical linear algebra is the amount of memory needed to store
a large matrix and the amount of time needed to read its entries. Methods using Krylov subspaces
avoid this difficulty by studying how a matrix acts on vectors, making it unnecessary in many cases
to create the matrix itself.
The Arnoldi Iteration is an algorithm for finding an orthonormal basis of a Krylov subspace.
One of its strengths is that it can run on any linear operator without knowing the operator’s under-
lying matrix representation. The outputs of the Arnoldi algorithm can then be used to approximate
the eigenvalues of the matrix of the linear operator.
The order-n Krylov subspace of A generated by x is

Kn (A, x) = span{x, Ax, A2 x, . . . , An−1 x}.

If the vectors {x, Ax, A2 x, . . . , An−1 x} are linearly independent, then they form a basis for Kn (A, x).
However, An x frequently converges to a dominant eigenvector of A as n gets large, which fills the
basis with many almost parallel vectors. This yields a basis prone to ill-conditioned computations
and numerical instability.

The Arnoldi Iteration Algorithm

The Arnoldi iteration focuses on efficiently creating an orthonormal basis for Kn (A, x) by integrating
the creation of {x, Ax, A2 x, . . . , An−1 x} with the modified Gram-Schmidt algorithm. This process
yields an orthonormal basis for Kn (A, x) that can be used for further computations.

167
168 Lab 17. The Arnoldi Iteration

The algorithm begins by initializing a matrix H which will be an upper Hessenberg matrix and
a matrix Q which will be filled with the basis vectors of our Krylov subspace. It also requires an
initial vector b ̸= 0 which is normalized to get q1 = b/ ∥b∥. This represents the basis for the initial
Krylov subspace, K1 (A, b).
For the kth iteration, compute the next basis vector qk+1 by using the modified Gram-Schmidt
process to make Aqk orthonormal to qk . This entails making each column of Q orthogonal to qk
before proceeding to the next iteration. The vectors {qi }ki=1 are then a basis for Kk (A, b). If ∥qk+1 ∥
is below a certain tolerance, stop and return H and Q. Otherwise, normalize the new basis vector
new qk+1 and continue to the next iteration.

Algorithm 1 The Arnoldi iteration. This algorithm accepts a square matrix A and a starting vector
b. It iterates k times or until the norm of the next vector in the iteration is less than tol. The
algorithm returns an upper Hessenberg H and an orthonormal Q such that H = QH AQ.
1: procedure Arnoldi(b, A, k, tol)
2: Q ← empty(size(b), k + 1) ▷ Some initialization steps
3: H ← zeros(k + 1, k)
4: Q:,0 ← b/ ∥b∥2
5: for j = 0 . . . k − 1 do ▷ Perform the actual iteration.
6: Q:,j+1 ← A(Q:,j )
7: for i = 0 . . . j do ▷ Modified Gram-Schmidt.
8: Hi,j ← QH Q
:,i :,j+1
9: Q:,j+1 ← Q:,j+1 − Hi,j Q:,i
10: Hj+1,j ← ∥Q:,j+1 ∥2 ▷ Set subdiagonal element of H.
11: if |Hj+1,j | < tol then ▷ Stop if ∥Q:,j+1 ∥2 is small enough.
12: return H:j+1,:j+1 , Q:,:j+1
13: Q:,j+1 ← Q:,j+1 /Hj+1,j ▷ Normalize qj+1 .
14: return H:−1,: , Q ▷ Return Hk and Q.

Achtung!
If the starting vector x is an eigenvector of A with corresponding eigenvalue λ, then by definition
Kk (A, x) = span{x, λx, λ2 x, . . . , λk x}, which is equal to the span of x. So, when x is normalized
with q1 = x/∥x∥, q2 = Aq1 = λq1 .
The vector q2 is supposed to be the next vector in the orthonormal basis for Kk (A, x),
but it is not linearly independent of q1 . In fact, q1 already spans Kk (A, x). Hence, the Gram-
Schmidt process fails and results in a ZeroDivisionError or an extremely early termination
of the algorithm. A similar phenomenon may occur if the starting vector x is contained in a
proper invariant subspace of A.

Arnoldi Iteration on Linear Operators

A major strength of the Arnoldi iteration is that it can run on a linear operator, even without knowing
the matrix representation of the operator. If L is some linear function, then we can modify the
pseudocode above by replacing AQ:,j with Amul (Q:,j ). This makes it possible to find the eigenvalues
of an arbitrary linear transformation.
 169

Problem 1. Write a function that accepts a starting vector b for the Arnoldi Iteration, a
function handle L that describes a linear operator, the number of times n to perform the
iteration, and a tolerance tol that defaults to 10−8 . Use Algorithm 1 to implement the Arnoldi
Iteration with these parameters. Return the upper Hessenberg matrix H and the orthonormal
matrix Q from the iteration.
Consider the following implementation details.

1. Since H and Q will eventually hold complex numbers, initialize them as complex arrays
(e.g., A = np.empty((3,3), dtype=np.complex128)).

2. This function can be tested on a matrix A by passing in A.dot for a linear operator.

3. Remember to use complex inner products. Here is an example of how to evaluate AH A:

b = A.conj() @ B

Test your function by comparing the resulting H with QH AQ.

Finding Eigenvalues Using the Arnoldi Iteration

Let A be an n × n matrix. Let Qk be the matrix whose columns q1 , . . . , qk are the orthonormal basis
for Km (A, x) generated by the Arnoldi algorithm, and let Hk be the k × k upper Hessenburg matrix
defined at the kth stage of the algorithm. Then these matrices satisfy

Hk = QH
k AQk . (17.1)

If k < n, then Hk is a low-rank approximation to A and the eigenvalues of Hk may be used as

approximations for the eigenvalues of A. The eigenvalues of Hk are called Ritz Values, and we will
later show that they converge quickly to the largest eigenvalues of A.

Problem 2. Write a function that accepts a function handle L that describes a linear operator,
the dimension of the space dim that the linear operator works on, the number of times k to
perform the Arnoldi Iteration, and the number of Ritz values n to return. Use the previous
implementation of the Arnoldi Iteration and an eigenvalue function such as scipy.linalg.
eigs() to compute the largest Ritz values of the given operator. Return the n largest Ritz
values.

One application of the Arnoldi iteration is to find the eigenvalues of linear operators that are
20
too large to store in memory. For example, if an operator acts on a vector x ∈ C2 , then its matrix
representation contains 240 complex values. Storing such a matrix would require 64 terabytes of
memory!
An example of such an operator is the Fast Fourier Transform, cited by SIAM as one of the
top algorithms of the century [Cip00]. The Fast Fourier Transform is used very commonly in signal
processing.
170 Lab 17. The Arnoldi Iteration

Problem 3. The four largest eigenvalues of the Fast Fourier Transform are known to be
√ √ √ √
{− n, n, −i n, i n} where n is the dimension of the space on which the transform acts.
Use your function from Problem 2 to approximate the eigenvalues of the Fast Fourier
Transform. Set k = 10 and dim = 220 . For the argument L, use the scipy.fftpack.fft().

The Arnoldi iteration for finding eigenvalues is implemented in a Fortran library called ARPACK.
Scipy interfaces with the Arnoldi iteration in this library via the function scipy.sparse.linalg.
eigs(). This function has many more options than the implementation we wrote in Problem 2. In
this example, the keyword argument k=5 specifies that we want five Ritz values. Note that even
though this function comes from the sparse library in Scipy, we can still call it on regular Numpy
arrays.

>>> from scipy.sparse import linalg as spla

>>> B = np.random.random((100,100))
>>> spla.eigs(B, k=5, return_eigenvectors=false)
array([ -1.15577072-2.59438308j, -2.63675878-1.09571889j,
-2.63675878+1.09571889j, -3.00915592+0.j , 50.14472893+0.j ])

Convergence
As more iterations of the Arnoldi method are performed, our approximations are of higher rank.
Consequently, the Ritz values become more accurate approximations to the eigenvalues of the linear
operator.
This technique converges quickly to eigenvalues whose magnitude is distinctly larger than the
rest. For example, matrices with random entries tend to have one eigenvalue of distinctly greatest
magnitude. Convergence of the Ritz values for such a matrix is plotted in Figure 17.1a.
However, Ritz values converge more slowly for matrices with random eigenvalues. Figure 17.1b
plots convergence of the Ritz values for a matrix with eigenvalues uniformly distributed in [0, 1).

Problem 4. Write a function that accepts a linear operator A, the number of Ritz values
to plot n, and the the number of times to perform the Arnoldi iteration iters. Use these
parameters to create a plot of the absolute error between the largest Ritz values of A and the
largest eigenvalues of A.

1. Find n eigenvalues of A of largest magnitude. Store these in order.

2. Create an empty array to store the relative errors for every k = 0, 1, . . . , iters.

(a) Use your Ritz function to find the n largest Ritz values of the operator. Note that
for small k, the matrix Hk may not have this many eigenvalues. Due to this, the
graphs of some eigenvalues have to begin after a few iterations.
(b) Store the absolute error between the eigenvalues of A and the Ritz values of H. Make
sure that the errors are stored in the correct order.

3. Iteratively plot the errors for each eigenvalue with the range of the iterations.
 171

101 102
10 2 10 1

10 5 10 4
10 8
10 7
10 11
10 10
10 14
10 17 10 13

0 25 50 75 100 125 150 175 200 0 50 100 150 200 250 300
(a) (b)

Figure 17.1: These plots show the relative error of the ritz values as approximations to the eigenvalues
of a matrix. The figure on the left plots the largest 15 Ritz values for a 500 × 500 matrix with random
entries and demonstrates that the largest eigenvalue (the blue line) converges after 20 iterations.
The figure at right plots the largest 15 Ritz values for a 500 × 500 matrix with uniformly distributed
eigenvalues in [0, 1) and demonstrates that all the eigenvalues take from 150 to 250 iterations to
converge.

Hints: If x̃ is an an approximation to x, then the absolute error in the approximation is ∥x− x̃∥.
Sort your eigenvalues from greatest to least. An example of how to do this is included:

# Evaluate the eigenvalues

eigvalues = la.eig(A)[0]
# Sort them from greatest to least (use np.abs to account for complex ←-
parts)
eigvalues = eigvalues[np.sort(np.abs(eigvalues))[::-1]]

In addition, remember that certain eigenvalues of H will not appear until we are computing
enough iterations in the Arnoldi algorithm. As a result, we will have to begin the graphs of
several eigenvalues after we are computing sufficient iterations of the algorithm.
Run your function on these examples. The plots should be fairly similar to Figures 17.1b
and 17.1a.

>>> A = np.random.rand(300, 300)

>>> plot_ritz(a, 10, 175)

>>> # A matrix with uniformly distributed eigenvalues

>>> d = np.diag(np.random.rand(300))
>>> B = A @ d @ la.inv(A)
>>> plot_ritz(B, 10, 175)
172 Lab 17. The Arnoldi Iteration

Additional Material
The Lanczos Iteration
The Lanczos iteration is a version of the Arnoldi iteration that is optimized to operate on symmetric
matrices. If A is symmetric, then (17.1) shows that Hk is symmetric and hence tridiagonal. This
leads to two simplifications of the Arnoldi algorithm.
First, we have 0 = Hk,n = ⟨qk , Aqn ⟩ for k ≤ n − 2; i.e., Aqn is orthogonal to q1 , . . . , qn−2 .
Thus, if the goal is only to compute Hk (say to find the Ritz values), then we only need to store the
two most recently computed columns of Q. Second, the data of Hk can also be stored in two vectors,
one containing the main diagonal and one containing the first subdiagonal of Hk (by symmetry, the
first superdiagonal equals the first subdiagonal of Hk ).

Algorithm 2 The Lanczos Iteration. This algorithm operates on a vector b of length n and an
n × n symmetric matrix A. It iterates k times or until the norm of the next vector in the iteration
is less than tol. It returns two vectors x and y that respectively contain the main diagonal and first
subdiagonal of the current Hessenberg approximation.
1: procedure Lanczos(b, A, k, tol)
2: q0 ← zeros(size(b)) ▷ Some initialization
3: q1 ← b/ ∥b∥2
4: x ← empty(k)
5: y ← empty(k)
6: for i = 0 . . . k − 1 do ▷ Perform the iteration.
7: z ← Aq1 ▷ z is a temporary vector to store qi+1 .
8: x[i] ← qT 1z ▷ q1 is used to store the previous qi .
9: z ← z − x[i]q1 + y[i − 1]q0 ▷ q0 is used to store qi−1 .
10: y[i] = ∥z∥2 ▷ Initialize y[i].
11: if y[i] < tol then ▷ Stop if ∥qi+1 ∥2 is too small.
12: return x[: i + 1], y[: i]
13: z = z/y[i]
14: q0 , q1 = q1 , z ▷ Store new qi+1 and qi on top of q1 and q0 .
15: return x, y[: −1]

As it is described in Algorithm 2, the Lanczos iteration is not stable. Roundoff error may cause
the qi to be far from orthogonal. In fact, it is possible for the qi to be so adulterated by roundoff
error that they are no longer linearly independent.
There are modified versions of the Lanczos iteration that are numerically stable. One of these,
the Implicitly Restarted Lanczos Method, is found in SciPy as scipy.sparse.linalg.eigsh().
 GMRES

18
Lab Objective: The Generalized Minimal Residuals (GMRES) algorithm is an iterative Krylov
subspace method for efficiently solving large linear systems. In this lab we implement the basic GM-
RES algorithm, then make an improvement by using restarts. We then discuss the convergence of the
algorithm and its relationship with the eigenvalues of a linear system. Finally, we introduce SciPy’s
version of GMRES.

The GMRES Algorithm

GMRES is an iterative method that uses Krylov subspaces to reduce a high-dimensional problem to
a sequence of smaller dimensional problems. Let A be an invertible m × m matrix and let b be a
vector of length m. Let Kn (A, b) be the order-n Krylov subspace generated by A and b. Instead
of solving the system Ax = b directly, GMRES uses least squares to find xn ∈ Kn that minimizes
the residual rn = ∥b − Axn ∥2 . The algorithm terminates when this residual is smaller than some
predetermined value. In many situations, this happens when n is much smaller than m.

The GMRES algorithm uses the Arnoldi iteration for numerical stability. The Arnoldi iteration
produces Hn , an (n + 1) × n upper Hessenberg matrix, and Qn , a matrix whose columns make up an
orthonormal basis of Kn (A, b), such that AQn = Qn+1 Hn . The GMRES algorithm finds the vector
xn which minimizes the norm ∥b − Axn ∥2 , where xn = Qn yn + x0 for some yn ∈ Rn . Since the
columns of Qn are orthonormal, the residual can be equivalently computed as

∥b − Axn ∥2 = ∥Qn+1 (βe1 − Hn yn )∥2 = ∥Hn yn − βe1 ∥2 . (18.1)

Here e1 is the vector [1, 0, . . . , 0]T of length n + 1 and β = ∥b − Ax0 ∥2 , where x0 is an initial
guess of the solution. Thus, to minimize ∥b − Axn ∥2 , the right side of (18.1) can be minimized, and
xn can be computed as xn = Qn yn + x0 .

173
174 Lab 18. GMRES

Algorithm 1 The GMRES algorithm. This algorithm operates on a vector b and a linear operator
A. It iterates k times or until the residual is less than tol, returning an approximate solution to
Ax = b and the error in this approximation.
1: procedure GMRES(A, b, x0 , k, tol)
2: Q ← empty(size(b), k + 1) ▷ Initialization.
3: H ← zeros(k + 1, k)
4: r0 ← b − A(x0 )
5: Q:,0 = r0 / ∥r0 ∥2
6: for j = 0 . . . k − 1 do ▷ Perform the Arnoldi iteration.
7: Q:,j+1 ← A(Q:,j )
8: for i = 0 . . . j do
9: Hi,j ← QT :,i Q:,j+1
10: Q:,j+1 ← Q:,j+1 − Hi,j Q:,i
11: Hj+1,j ← ∥Q:,j+1 ∥2
12: if |Hj+1,j | > tol then ▷ Avoid dividing by zero.
13: Q:,j+1 ← Q:,j+1 /Hj+1,j
14: y ← least squares solution to ∥H:j+2,:j+1 x − βe1 ∥2 ▷ β and e1 as in (18.1).
15: res ← ∥H:j+2,:j+1 y − βe1 ∥2
16: if res < tol then
17: return Q:,:j+1 y + x0 , res
18: return Q:,:j+1 y + x0 , res

Problem 1. Write a function that accepts a matrix A, a vector b, and an initial guess x0 , a
maximum number of iterations k defaulting to 100, and a stopping tolerance tol that defaults
to 10−8 . Use Algorithm 1 to approximate the solution to Ax = b using the GMRES algorithm.
Return the approximate solution and the residual at the approximate solution.
You may assume that A and b only have real entries. Use scipy.linalg.lstsq() to
solve the least squares problem. Be sure to read the documentation so that you understand
what the function returns.
Compare your function to the following code.

>>> A = np.array([[1,0,0],[0,2,0],[0,0,3]])
>>> b = np.array([1, 4, 6])
>>> x0 = np.zeros(b.size)
>>> gmres(A, b, x0, k=100, tol=1e-8)
(array([ 1., 2., 2.]), 7.174555448775421e-16)

Convergence of GMRES
One of the most important characteristics of GMRES is that it will always arrive at an exact solution
(if one exists). At the n-th iteration, GMRES computes the best approximate solution to Ax = b for
xn ∈ Kn . If A is full rank, then Km = Fm , so the mth iteration will always return an exact answer.
Sometimes, the exact solution x ∈ Kn for some n < m, in this case xn is an exact solution. In either
case, the algorithm is convergent after n steps if the nth residual is sufficiently small.
 175

The rate of convergence of GMRES depends on the eigenvalues of A.

Problem 2. Add a keyword argument plot defaulting to False to your function from Problem
1. If plot=True, keep track of the residuals at each step of the algorithm. At the end of the
iteration, before returning the approximate solution and its residual error, create a figure with
two subplots.

1. Make a scatter plot of the eigenvalues of A on the complex plane.

2. Plot the residuals versus the iteration counts using a log scale on the y-axis
(use ax.semilogy()).

Problem 3. Use your function from Problem 2 to investigate how the convergence of GMRES
relates to the eigenvalues of a matrix as follows. Define an m × m matrix

An = nI + P,

where I is the identity matrix and P is an m × m matrix with entries taken from a random
√
normal distribution with mean 0 and standard deviation 1/(2 m). Call your function from
Problem 2 on An for n = −4, −2, 0, 2, 4. Use m = 200, let b be an array of all ones, and let
x0 = 0.
Use np.random.normal() to create the matrix P . When analyzing your results, pay
special attention to the clustering of the eigenvalues in relation to the origin. Compare your
results with n = 2, m = 200 to Figure 18.1.
Ideas for this problem were taken from Example 35.1 on p. 271 of [TB97].

0.4 10 1

10 2
0.2 10 3

0.0 10 4

10 5
0.2 10 6

0.4 10 7

10 8

1.6 1.8 2.0 2.2 2.4 0 2 4 6 8 10 12

Figure 18.1: On the left, the eigenvalues of the matrix A2 defined in Problem 3. On the right,
the rapid convergence of the GMRES algorithm on A2 with starting vector b = (1, 1, . . . , 1).

GMRES with Restarts

The first few iterations of GMRES have low spatial and temporal complexity. However, as k increases,
the kth iteration of GMRES becomes more expensive temporally and spatially. In fact, computing
the kth iteration of GMRES for very large k can be prohibitively complex.
176 Lab 18. GMRES

This issue is addressed by using GMRES(k), or GMRES with restarts. When k becomes large,
this algorithm restarts GMRES with an improved initial guess. The new initial guess is taken to
be the vector that was found upon termination of the last GMRES iteration run. The algorithm
GMRES(k) will always have manageable spatial and temporal complexity, but it is less reliable than
GMRES. If the true solution x to Ax = b is nearly orthogonal to the Krylov subspaces Kn (A, b) for
n ≤ k, then GMRES(k) could converge very slowly or not at all.

Problem 4. Write a function that implements GMRES with restarts as follows.

1. Perform the GMRES algorithm for a maximum of k iterations.

2. If the desired tolerance was reached, terminate the algorithm. If not, repeat step 1 using
xk from the previous GMRES algorithm as a new initial guess x0 .

3. Repeat step 2 until the desired tolerance has been obtained or until a given maximum
number of restarts has been reached.

Your function should accept all of the same inputs as the function you wrote in Problem 1 with
the exception of k, which will now denote the number of iterations before restart (defaults to 5),
and an additional parameter restarts which denotes the maximum number of restarts before
termination (defaults to 50).

GMRES in SciPy
The GMRES algorithm is implemented in SciPy as the function scipy.sparse.linalg.gmres().
Here we use this function to solve Ax = b where A is a random 300 × 300 matrix and b is a random
vector.

>>> import numpy as np

>>> from scipy import sparse
>>> from scipy.sparse import linalg as spla

>>> A = np.random.rand(300, 300)

>>> b = np.random(300)
>>> x, info = spla.gmres(A, b)
>>> print(info)
3000

The function outputs two objects: the approximate solution x and an integer info which gives
information about the convergence of the algorithm. If info=0 then convergence occured; if info
is positive then it equals the number of iterations performed. In the previous case, the function
performed 3000 iterations of GMRES before returning the approximate solution x. The following
code verifies how close the computed value was to the exact solution.

>>> la.norm((A @ x) - b)
4.744196381683801

A better approximation can be obtained using GMRES with restarts.

 177

# Restart after 1000 iterations.

>>> x, info = spla.gmres(A, b, restart=1000)
>>> info
0
>>> la.norm((A @ x) - b)
1.0280404494143551e-12

This time, the returned approximation x is about as close to a true solution as can be expected.

Problem 5. Plot the runtimes of your implementations of GMRES from Problems 1 and 4
and scipy.sparse.linalg.gmres() use the default tolerance and restart=1000 with different
matrices. Use the m×m matrix P with m = 25, 50, . . . 200 and with entries taken from a random
√
normal distribution with mean 0 and standard deviation 1/(2 m). Use a vector of ones for b
and a vector of zeros for x0 . Use a single figure for all plots, plot the runtime on the y-axis and
m on the x-axis.
178 Lab 18. GMRES
Part II

Appendices

179
A
NumPy Visual Guide

Lab Objective: NumPy operations can be difficult to visualize, but the concepts are straightforward.
This appendix provides visual demonstrations of how NumPy arrays are used with slicing syntax,
stacking, broadcasting, and axis-specific operations. Though these visualizations are for 1- or 2-
dimensional arrays, the concepts can be extended to n-dimensional arrays.

Data Access
The entries of a 2-D array are the rows of the matrix (as 1-D arrays). To access a single entry, enter
the row index, a comma, and the column index. Remember that indexing begins with 0.

   
× × × × × × × × × ×
 × × × × ×   × × × × × 
A[0] = 
 ×
 A[2,1] =  
× × × ×   × × × × × 
× × × × × × × × × ×

Slicing
A lone colon extracts an entire row or column from a 2-D array. The syntax [a:b] can be read as
“the ath entry up to (but not including) the bth entry.” Similarly, [a:] means “the ath entry to the
end” and [:b] means “everything up to (but not including) the bth entry.”

   
× × × × × × × × × ×
 × × × × ×   × × × × × 
A[1] = A[1,:] = 
 × ×
 A[:,2] =  
× × ×   × × × × × 
× × × × × × × × × ×

   
× × × × × × × × × ×
 × × × × ×   × × × × × 
A[1:,:2] = 
 ×
 A[1:-1,1:-1] =  
× × × ×   × × × × × 
× × × × × × × × × ×

181
182 Appendix A. NumPy Visual Guide

Stacking
np.hstack() stacks sequence of arrays horizontally and np.vstack() stacks a sequence of arrays
vertically.

   
× × × ∗ ∗ ∗
A= × × ×  B= ∗ ∗ ∗ 
× × × ∗ ∗ ∗

 
× × × ∗ ∗ ∗ × × ×
np.hstack((A,B,A)) = ×
 × × ∗ ∗ ∗ × × × 
× × × ∗ ∗ ∗ × × ×

 
× × ×

 × × × 


 × × × 


 ∗ ∗ ∗ 

np.vstack((A,B,A)) =  ∗ ∗ ∗
 

∗ ∗ ∗
 
 
× × ×
 
 
 
 × × × 
× × ×

Because 1-D arrays are flat, np.hstack() concatenates 1-D arrays and np.vstack() stacks them
vertically. To make several 1-D arrays into the columns of a 2-D array, use np.column_stack().

x= y=
   
× × × × ∗ ∗ ∗ ∗

np.hstack((x,y,x)) =
 
× × × × ∗ ∗ ∗ ∗ × × × ×

 
  × ∗ ×
× × × ×  × ∗ × 
np.vstack((x,y,x)) =  ∗ ∗ ∗ ∗  np.column_stack((x,y,x)) = 
 ×

∗ × 
× × × ×
× ∗ ×

The functions np.concatenate() and np.stack() are more general versions of np.hstack() and
np.vstack(), and np.row_stack() is an alias for np.vstack().

Broadcasting
NumPy automatically aligns arrays for component-wise operations whenever possible. See http:
//docs.scipy.org/doc/numpy/user/basics.broadcasting.html for more in-depth examples and
broadcasting rules.
 183

 
1 2 3
A= 1 x=
 
2 3  10 20 30
1 2 3

 
1 2 3
1 2 3  
11 22 33
A + x= 1 2 3 =  11 22 33 
 +  11 22 33
10 20 30

     
1 2 3 10 11 12 13
A + x.reshape((1,-1)) =  1 2 3  +  20  =  21 22 23 
1 2 3 30 31 32 33

Operations along an Axis

Most array methods have an axis argument that allows an operation to be done along a given axis.
To compute the sum of each column, use axis=0; to compute the sum of each row, use axis=1.

 
1 2 3 4
 1 2 3 4 
A=
 1

2 3 4 
1 2 3 4

 
1 2 3 4
 1 2 3 4 
A.sum(axis=0) = 
 
= 4 8 12 16
 1 2 3 4 
1 2 3 4

 
1 2 3 4
 1 2 3 4 
A.sum(axis=1) = 
 
 = 10 10 10 10
 1 2 3 4 
1 2 3 4
184 Appendix A. NumPy Visual Guide
B
Matplotlib Syntax and
Customization Guide

Lab Objective: The documentation for Matplotlib can be a little difficult to maneuver and basic
information is sometimes difficult to find. This appendix condenses and demonstrates some of the
more applicable and useful information on plot customizations. It is not intended to be read all at
once, but rather to be used as a reference when needed. For an interative introduction to Matplotlib,
see the Introduction to Matplotlib lab in Python Essentials. For more details on any specific function,
refer to the Matplotlib documentation at https: // matplotlib. org/ .

Matplotlib Interface
Matplotlib plots are made in a Figure object that contains one or more Axes, which themselves
contain the graphical plotting data. Matplotlib provides two ways to create plots:

1. Call plotting functions directly from the module, such as plt.plot(). This will create the plot
on whichever Axes is currently active.

2. Call plotting functions from an Axes object, such as ax.plot(). This is particularly useful for
complicated plots and for animations.

Table B.1 contains a summary of functions that are used for managing Figure and Axes objects.

Function Description
add_subplot() Add a single subplot to the current figure
axes() Add an axes to the current figure
clf() Clear the current figure
figure() Create a new figure or grab an existing figure
gca() Get the current axes
gcf() Get the current figure
subplot() Add a single subplot to the current figure
subplots() Create a figure and add several subplots to it

Table B.1: Basic functions for managing plots.

185
186 Appendix B. Matplotlib Customization

Axes objects are usually managed through the functions plt.subplot() and plt.subplots().
The function subplot() is used as plt.subplot(nrows, ncols, plot_number). Note that if the
inputs for plt.subplot() are all integers, the commas between the entries can be omitted. For
example, plt.subplot(3,2,2) can be shortened to plt.subplot(322).
The function subplots() is used as plt.subplots(nrows, ncols), and returns a Figure
object and an array of Axes. This array has the shape (nrows, ncols), and can be accessed as any
other array. Figure B.1 demonstrates the layout and indexing of subplots.

1 2 3
4 5 6
Figure B.1: The layout of subplots with plt.subplot(2,3,i) (2 rows, 3 columns), where i is the
index pictured above. The outer border is the figure that the axes belong to.

The following example demonstrates three equivalent ways of producing a figure with two
subplots, arranged next to each other in one row:

>>> x = np.linspace(-5, 5, 100)

# 1. Use plt.subplot() to switch the current axes.

>>> plt.subplot(121)
>>> plt.plot(x, 2*x)
>>> plt.subplot(122)
>>> plt.plot(x, x**2)

# 2. Use plt.subplot() to explicitly grab the two subplot axes.

>>> ax1 = plt.subplot(121)
>>> ax1.plot(x, 2*x)
>>> ax2 = plt.subplot(122)
>>> ax2.plot(x, x**2)

# 3. Use plt.subplots() to get the figure and all subplots simultaneously.

>>> fig, axes = plt.subplots(1, 2)
>>> axes[0].plot(x, 2*x)
>>> axes[1].plot(x, x**2)
 187

Achtung!
Be careful not to mix up the following similarly-named functions:

1. plt.axes() creates a new place to draw on the figure, while plt.axis() or ax.axis()
sets properties of the x- and y-axis in the current axes, such as the x and y limits.

2. plt.subplot() (singular) returns a single subplot belonging to the current figure, while
plt.subplots() (plural) creates a new figure and adds a collection of subplots to it.

Plot Customization
Styles
Matplotlib has a number of built-in styles that can be used to set the default appearance of plots.
These can be used via the function plt.style.use(); for instance, plt.style.use("seaborn")
will have Matplotlib use the "seaborn" style for all plots created afterwards. A list of built-in
styles can be found at https://matplotlib.org/stable/gallery/style_sheets/style_sheets_
reference.html.
The style can also be changed only temporarily using plt.style.context() along with a with
block:

with plt.style.context('dark_background'):
# Any plots created here use the new style
plt.subplot(1,2,1)
plt.plot(x, y)
# ...
# Plots created here are unaffected
plt.subplot(1,2,2)
plt.plot(x, y)

Plot layout
Axis properties

Table B.2 gives an overview of some of the functions that may be used to configure the axes of a
plot.
The functions xlim(), ylim(), and axis() are used to set one or both of the x and y ranges
of the plot. xlim() and ylim() each accept two arguments, the lower and upper bounds, or a single
list of those two numbers. axis() accepts a single list consisting, in order, of xmin, xmax, ymin,
ymax. Passing None instead of one of the numbers to any of these functions will make it not change
the corresponding value from what it was. Each of these functions can also be called without any
arguments, in which case it will return the current bounds. Note that axis() can also be called
directly on an Axes object, while xlim() and ylim() cannot.
axis() also can be called with a string as its argument, which has several options. The most
common is axis('equal'), which makes the scale of the x- and y-scales equal (i.e. makes circles
circular).
188 Appendix B. Matplotlib Customization

Function Description
axis() set the x- and y-limits of the plot
grid() add gridlines
xlim() set the limits of the x-axis
ylim() set the limits of the y-axis
xticks() set the location of the tick marks on the x-axis
yticks() set the location of the tick marks on the y-axis
xscale() set the scale type to use on the x-axis
yscale() set the scale type to use on the y-axis
ax.spines[side].set_position() set the location of the given spine
ax.spines[side].set_color() set the color of the given spine
ax.spines[side].set_visible() set whether a spine is visible

Table B.2: Some functions for changing axis properties. ax is an Axes object.

To use a logarithmic scale on an axis, the functions xscale("log") and yscale("log") can
be used.
The functions xticks() and yticks() accept a list of tick positions, which the ticks on the
corresponding axis are set to. Generally, this works the best when used with np.linspace(). This
function also optionally accepts a second argument of a list of labels for the ticks. If called with no
arguments, the function returns a list of the current tick positions and labels instead.
The spines of a Matplotlib plot are the black border lines around the plot, with the left and
bottom ones also being used as the axis lines. To access the spines of a plot, call ax.spines[side],
where ax is an Axes object and side is 'top', 'bottom', 'left', or 'right'. Then, functions can
be called on the Spine object to configure it.
The function spine.set_position() has several ways to specify the position. The two simplest
are with the arguments 'center' and 'zero', which place the spine in the center of the subplot or
at an x- or y-coordinate of zero, respectively. The others are a passed as a tuple (position_type,
amount):

• 'data': place the spine at an x- or y-coordinate equal to amount.

• 'axes': place the spine at the specified Axes coordinate, where 0 corresponds to the bottom
or left of the subplot, and 1 corresponds to the top or right edge of the subplot.

• 'outward': places the spine amount pixels outward from the edge of the plot area. A negative
value can be used to move it inwards instead.

spine.set_color() accepts any of the color formats Matplotlib supports. Alternately, using
set_color('none') will make the spine not be visible. spine.set_visible() can also be used for
this purpose.
The following example adjusts the ticks and spine positions to improve the readability of a plot
of sin(x). The result is shown in Figure B.2.

>>> x = np.linspace(0,2*np.pi,150)
>>> plt.plot(x, np.sin(x))
>>> plt.title(r"$y=\sin(x)$")

#Set the ticks to multiples of pi/2, make nice labels

>>> ticks = np.pi / 2 * np.array([0,1,2,3,4])
 189

>>> tick_labels = ["$0$", r"$\frac{\pi}{2}$", r"$\pi$", r"$\frac{3\pi}{2}$",

... r"$2\pi$"]
>>> plt.xticks(ticks, tick_labels)

#Move the bottom spine to zero, remove the top and right ones
>>> ax = plt.gca()
>>> ax.spines['bottom'].set_position('zero')
>>> ax.spines['right'].set_color('none')
>>> ax.spines['top'].set_color('none')

>>> plt.show()

y = sin(x)
1.0

0.5

0.0
0 2
3
2
2
0.5

1.0
Figure B.2: Plot of y = sin(x) with axes modified for clarity

Plot Layout

The position and spacing of all subplots within a figure can be modified using the function plt
.subplots_adjust(). This function accepts up to six keyword arguments that change different
aspects of the spacing. left, right, top, and bottom are used to adjust the rectangle around all of
the subplots. In the coordinates used, 0 corresponds to the bottom or left edge of the figure, and 1
corresponds to the top or right edge of the figure. hspace and wspace set the vertical and horizontal
spacing, respectively, between subplots. The units for these are in fractions of the average height
and width of all subplots in the figure. If more fine control is desired, the position of individual Axes
objects can also be changed using ax.get_position() and ax.set_position().
The size of the figure can be configured using the figsize argument when creating a figure:

>>> plt.figure(figsize=(12,8))

Note that many environments will scale the figure to fill the available space. Even so, changing the
figure size can still be used to change the aspect ratio as well as the relative size of plot elements.
The following example uses subplots_adjust() to create space for a legend outside of the
plotting space. The result is shown in Figure B.3.
190 Appendix B. Matplotlib Customization

#Generate data
>>> x1 = np.random.normal(-1, 1.0, size=60)
>>> y1 = np.random.normal(-1, 1.5, size=60)
>>> x2 = np.random.normal(2.0, 1.0, size=60)
>>> y2 = np.random.normal(-1.5, 1.5, size=60)
>>> x3 = np.random.normal(0.5, 1.5, size=60)
>>> y3 = np.random.normal(2.5, 1.5, size=60)

#Make the figure wider

>>> fig = plt.figure(figsize=(5,3))

#Plot the data

>>> plt.plot(x1, y1, 'r.', label="Dataset 1")
>>> plt.plot(x2, y2, 'g.', label="Dataset 2")
>>> plt.plot(x3, y3, 'b.', label="Dataset 3")

#Create a legend to the left of the plot

>>> lspace = 0.35
>>> plt.subplots_adjust(left=lspace)
#Put the legend at the left edge of the figure
>>> plt.legend(loc=(-lspace/(1-lspace),0.6))
>>> plt.show()

Dataset 1 4
Dataset 2
Dataset 3 2
0
2
4
2 0 2 4
Figure B.3: Example of repositioning axes.
 191

Colors
The color that a plotting function uses is specified by either the c or color keyword arguments; for
most functions, these can be used interchangeably. There are many ways to specific colors. The most
simple is to use one of the basic colors, listed in Table B.3. Colors can also be specified using an
RGB tuple such as (0.0, 0.4, 1.0), a hex string such as "0000FF", or a CSS color name like "
DarkOliveGreen" or "FireBrick". A full list of named colors that Matplotlib supports can be found
at https://matplotlib.org/stable/gallery/color/named_colors.html. If no color is specified
for a plot, Matplotlib automatically assigns it one from the default color cycle.

Code Color
Code Color
'b' blue
'y' yellow
'g' green
'k' black
'r' red
'w' white
'c' cyan
'C0' - 'C9' Default colors
'm' magenta

Table B.3: Basic colors available in Matplotlib

Plotting functions also accept an alpha keyword argument, which can be used to set the
transparency. A value of 1.0 corresponds to fully opaque, and 0.0 corresponds to fully transparent.
The following example demonstrates different ways of specifying colors:

#Using a basic color

>>> plt.plot(x, y, 'r')
#Using a hexadecimal string
>>> plt.plot(x, y, color='FF0080')
#Using an RGB tuple
>>> plt.plot(x, y, color=(1, 0.5, 0))
#Using a named color
>>> plt.plot(x, y, color='navy')

Colormaps

Certain plotting functions, such as heatmaps and contour plots, accept a colormap rather than a
single color. A full list of colormaps available in Matplotlib can be found at https://matplotlib.
org/stable/gallery/color/colormap_reference.html. Some of the more commonly used ones
are "viridis", "magma", and "coolwarm". A colorbar can be added by calling plt.colorbar()
after creating the plot.
Sometimes, using a logarithmic scale for the coloring is more informative. To do this, pass a
matplotlib.colors.LogNorm object as the norm keyword argument:

# Create a heatmap with logarithmic color scaling

>>> from matplotlib.colors import LogNorm
>>> plt.pcolormesh(X, Y, Z, cmap='viridis', norm=LogNorm())
192 Appendix B. Matplotlib Customization

Function Description Usage

annotate() adds a commentary at a given point on the plot annotate(’text’,(x,y))
arrow() draws an arrow from a given point on the plot arrow(x,y,dx,dy)
colorbar() Create a colorbar colorbar()
legend() Place a legend in the plot legend(loc=’best’)
text() Add text at a given position on the plot text(x,y,’text’)
title() Add a title to the plot title(’text’)
suptitle() Add a title to the figure suptitle(’text’)
xlabel() Add a label to the x-axis xlabel(’text’)
ylabel() Add a label to the y-axis ylabel(’text’)

Table B.4: Text and annotation functions in Matplotlib

Text and Annotations

Matplotlib has several ways to add text and other annotations to a plot, some of which are listed in
Table B.4. The color and size of the text in most of these functions can be adjusted with the color
and fontsize keyword arguments.
Matplotlib also supports formatting text with LATEX, a system for creating technical docu-
ments.1 To do so, use an r before the string quotation mark and surround the text with dollar
signs. This is particularly useful when the text contains a mathematical expression. For example,
the following line of code will make the title of the plot be 12 sin(x2 ):

>>> plt.title(r"$\frac{1}{2}\sin(x^2)$")

The function legend() can be used to add a legend to a plot. Its optional loc keyword
argument specifies where to place the legend within the subplot. It defaults to 'best', which will
cause Matplotlib to place it in whichever location overlaps with the fewest drawn objects. The other
locations this function accepts are 'upper right', 'upper left', 'lower left', 'lower right',
'center left', 'center right', 'lower center', 'upper center', and 'center'. Alternately,
a tuple of (x,y) can be passed as this argument, and the bottom-left corner of the legend will be
placed at that location. The point (0,0) corresponds to the bottom-left of the current subplot, and
(1,1) corresponds to the top-right. This can be used to place the legend outside of the subplot,
although care should be taken that it does not go outside the figure, which may require manually
repositioning the subplots.
The labels the legend uses for each curve or scatterplot are specified with the label keyword
argument when plotting the object. Note that legend() can also be called with non-keyword argu-
ments to set the labels, although it is less confusing to set them when plotting.
The following example demonstrates creating a legend:

>>> x = np.linspace(0,2*np.pi,250)

# Plot sin(x), cos(x), and -sin(x)

# The label argument will be used as its label in the legend.
>>> plt.plot(x, np.sin(x), 'r', label=r'$\sin(x)$')
>>> plt.plot(x, np.cos(x), 'g', label=r'$\cos(x)$')
>>> plt.plot(x, -np.sin(x), 'b', label=r'$-\sin(x)$')
1 See http://www.latex-project.org/ for more information.
 193

# Create the legend

>>> plt.legend()

Line and marker styles

Matplotlib supports a large number of line and marker styles for line and scatter plots, which are
listed in Table B.5.

character description character description

- solid line style 3 tri_left marker
-- dashed line style 4 tri_right marker
-. dash-dot line style s square marker
: dotted line style p pentagon marker
. point marker * star marker
, pixel marker h hexagon1 marker
o circle marker H hexagon2 marker
v triangle_down marker + plus marker
ˆ triangle_up marker x x marker
< triangle_left marker D diamond marker
> triangle_right marker d thin_diamond marker
1 tri_down marker | vline marker
2 tri_up marker _ hline marker

Table B.5: Available line and marker styles in Maplotlib.

The function plot() has several ways to specify this argument; the simplest is to pass it as the
third positional argument. The marker and linestyle keyword arguments can also be used. The
size of these can be modified using markersize and linewidth. Note that by specifying a marker
style but no line style, plot() can be used to make a scatter plot. It is also possible to use both a
marker style and a line style. To set the marker using scatter(), use the marker keyword argument,
with s being used to change the size.
The following code demonstrates specifying marker and line styles. The results are shown in
Figure B.4.

#Use dashed lines:

>>> plt.plot(x, y, '--')
#Use only dots:
>>> plt.plot(x, y, '.')
#Use dots with a normal line:
>>> plt.plot(x, y, '.-')
#scatter() uses the marker keyword:
>>> plt.scatter(x, y, marker='+')

#With plot(), the color to use can also be specified in the same string.
#Order usually doesn't matter.
#Use red dots:
>>> plt.plot(x, y, '.r')
194 Appendix B. Matplotlib Customization

#Equivalent:
>>> plt.plot(x, y, 'r.')

#To change the size:

>>> plt.plot(x, y, 'v-', linewidth=1, markersize=15)
>>> plt.scatter(x, y, marker='+', s=12)

plt.plot(x, y, '--') plt.plot(x, y, '.') plt.plot(x, y, '.-') plt.scatter(x, y, marker='+')

plt.plot(x, y, 'v-', plt.scatter(x, y,

plt.plot(x, y, '.r') plt.plot(x, y, 'r.') linewidth=1, markersize=15) marker='+', s=12)

Figure B.4: Examples of setting line and marker styles.

Plot Types

Matplotlib has functions for creaing many different types of plots, many of which are listed in Table
B.6. This section gives details on using certain groups of these functions.
 195

Function Description Usage

bar makes a bar graph bar(x,height)
barh makes a horizontal bar graph barh(y,width)
boxplots makes one or more boxplots boxplots(data)
contour makes a contour plot contour(X,Y,Z)
contourf makes a filled contour plot contourf(X,Y,Z)
imshow shows an image imshow(image)
fill plots lines with shading under the curve fill(x,y)
fill_between plots lines with shading between two given y values fill_between(x,y1, y2=0)
hexbin creates a hexbin plot hexbin(x,y)
hist plots a histogram from data hist(data)
pcolormesh makes a heatmap pcolormesh(X,Y,Z)
pie makes a pie chart pie(x)
plot plots lines and data on standard axes plot(x,y)
plot_surface plot a surface in 3-D space plot_surface(X,Y,Z)
polar plots lines and data on polar axes polar(theta,r)
loglog plots lines and data on logarithmic x and y axes loglog(x,y)
scatter plots data in a scatterplot scatter(x,y)
semilogx plots lines and data with a log scaled x axis semilogx(x,y)
semilogy plots lines and data with a log scaled y axis semilogy(x,y)
specgram makes a spectogram from data specgram(x)
spy plots the sparsity pattern of a 2D array spy(Z)
triplot plots triangulation between given points triplot(x,y)

Table B.6: Some basic plotting functions in Matplotlib.

Line plots
Line plots, the most basic type of plot, are created with the plot() function. It accepts two lists of
x- and y-values to plot, and optionally a third argument of a string of any combination of the color,
line style, and marker style. Note that this method only works with the single-character color codes;
to use other colors, use the color argument. By specifying only a marker style, this function can
also be used to create scatterplots.
There are a number of functions that do essentially the same thing as plot() but also change
the axis scaling, including loglog(), semilogx(), semilogy(), and polar. Each of these functions
is used in the same manner as plot(), and has identical syntax.

Bar Plots
Bar plots are a way to graph categorical data in an effective way. They are made using the bar()
function. The most important arguments are the first two that provide the data, x and height. The
first argument is a list of values for each bar, either categorical or numerical; the second argument is
a list of numerical values corresponding to the height of each bar. There are other parameters that
may be included as well. The width argument adjusts the bar widths; this can be done by choosing
a single value for all of the bars, or an array to give each bar a unique width. Further, the argument
bottom allows one to specify where each bar begins on the y-axis. Lastly, the align argument can
be set to ’center’ or ’edge’ to align as desired on the x-axis. As with all plots, you can use the color
keyword to specify any color of your choice. If you desire to make a horizontal bar graph, the syntax
follows similarly using the function barh(), but with argument names y, width, height and align.
196 Appendix B. Matplotlib Customization

Box Plots
A box plot is a way to visualize some simple statistics of a dataset. It plots the minimum, maximum,
and median along with the first and third quartiles of the data. This is done by using boxplot()
with an array of data as the argument. Matplotlib allows you to enter either a one dimensional
array for a single box plot, or a 2-dimensional array where it will plot a box plot for each column of
the data in the array. Box plots default to having a vertical orientation but can be easily laid out
horizontally by setting vert=False.

Scatter and hexbin plots

Scatterplots can be created using either plot() or scatter(). Generally, it is simpler to use plot(),
although there are some cases where scatter() is better. In particular, scatter() allows changing
the color and size of individual points within a single call to the function. This is done by passing a
list of colors or sizes to the c or s arguments, respectively.
Hexbin plots are an alternative to scatterplots that show the concentration of data in regions
rather than the individual points. They can be created with the function hexbin(). Like plot()
and scatter(), this function accepts two lists of x- and y-coordinates.

Heatmaps and contour plots

Heatmaps and contour plots are used to visualize 3-D surfaces and complex-valued functions on a
flat space. Heatmaps are created using the pcolormesh() function. Contour plots are created using
contour() or contourf(), with the latter creating a filled contour plot.
Each of these functions accepts the x-, y-, and z-coordinates as a mesh grid, or 2-D array. To
create these, use the function np.meshgrid():

>>> x = np.linspace(0,1,100)
>>> y = np.linspace(0,1,80)
>>> X, Y = np.meshgrid(x, y)

The z-coordinate can then be computed using the x and y mesh grids.
Note that each of these functions can accept a colormap, using the cmap parameter. These
plots are sometimes more informative with a logarithmic color scale, which can be used by passing a
matplotlib.colors.LogNorm object in the norm parameter of these functions.
With pcolormesh(), it is also necessary to pass shading='auto' or shading='nearest' to
avoid a deprecation error.
The following example demonstrates creating heatmaps and contour plots, using a graph of
z = (x2 + y) sin(y). The results is shown in Figure B.5

>>> from matplotlib.colors import LogNorm

>>> x = np.linspace(-3,3,100)
>>> y = np.linspace(-3,3,100)
>>> X, Y = np.meshgrid(x, y)
>>> Z = (X**2+Y)*np.sin(Y)

#Heatmap
>>> plt.subplot(1,3,1)
 197

>>> plt.pcolormesh(X, Y, Z, cmap='viridis', shading='nearest')

>>> plt.title("Heatmap")

#Contour
>>> plt.subplot(1,3,2)
>>> plt.contour(X, Y, Z, cmap='magma')
>>> plt.title("Contour plot")

#Filled contour
>>> plt.subplot(1,3,3)
>>> plt.contourf(X, Y, Z, cmap='coolwarm')
>>> plt.title("Filled contour plot")
>>> plt.colorbar()

>>> plt.show()

Figure B.5: Example of heatmaps and contour plots.

Showing images
The function imshow() is used for showing an image in a plot, and can be used on either grayscale
or color images. This function accepts a 2-D n × m array for a grayscale image, or a 3-D n × m × 3
array for a color image. If using a grayscale image, you also need to specify cmap='gray', or it will
be colored incorrectly.
It is best to also use axis('equal') alongside imshow(), or the image will most likely be
stretched. This function also works best if the images values are in the range [0, 1]. Some ways to
load images will format their values as integers from 0 to 255, in which case the values in the image
array should be scaled before using imshow().

3-D Plotting
Matplotlib can be used to plot curves and surfaces in 3-D space. In order to use 3-D plotting, you
need to run the following line:
198 Appendix B. Matplotlib Customization

>>> from mpl_toolkits.plot3d import Axes3D

The argument projection='3d' also must be specified when creating the subplot for the 3-D object:

>>> plt.subplot(1,1,1, projection='3d')

Curves can be plotted in 3-D space using plot(), by passing in three lists of x-, y-, and z-
coordinates. Surfaces can be plotted using ax.plot_surface(). This function can be used similar
to creating contour plots and heatmaps, by obtaining meshes of x- and y- coordinates from np.
meshgrid() and using those to produce the z-axis. More generally, any three 2-D arrays of meshes
corresponding to x-, y-, and z-coordinates can be used. Note that it is necessary to call this function
from an Axes object.
The following example demonstrates creating 3-D plots. The results are shown in Figure B.6.

#Create a plot of a parametric curve

ax = plt.subplot(1,3,1, projection='3d')
t = np.linspace(0, 4*np.pi, 160)
x = np.cos(t)
y = np.sin(t)
z = t / np.pi
plt.plot(x, y, z, color='b')
plt.title("Helix curve")

#Create a surface plot from np.meshgrid

ax = plt.subplot(1,3,2, projection='3d')
x = np.linspace(-1,1,80)
y = np.linspace(-1,1,80)
X, Y = np.meshgrid(x, y)
Z = X**2 - Y**2
ax.plot_surface(X, Y, Z, color='g')
plt.title(r"Hyperboloid")

#Create a surface plot less directly

ax = plt.subplot(1,3,3, projection='3d')
theta = np.linspace(-np.pi,np.pi,80)
rho = np.linspace(-np.pi/2,np.pi/2,40)
Theta, Rho = np.meshgrid(theta, rho)
X = np.cos(Theta) * np.cos(Rho)
Y = np.sin(Theta) * np.cos(Rho)
Z = np.sin(Rho)
ax.plot_surface(X, Y, Z, color='r')
plt.title(r"Sphere")

plt.show()
 199

Helix curve Hyperboloid Sphere

4 1.0
3 0.5 0.5
2 0.0 0.0
1 0.5 0.5
0 1.0
1 1 1
1 0 1 0 1 0
0 1 0 1 0 1
1 1 1
Figure B.6: Examples of 3-D plotting.

Additional Resources
rcParams
The default plotting parameters of Matplotlib can be set individually and with more fine control than
styles by using rcParams. rcParams is a dictionary that can be accessed as either plt.rcParams or
matplotlib.rcParams.
For instance, the resolution of plots can be changed via the "figure.dpi" parameter:

>>> plt.rcParams["figure.dpi"] = 600

A list of parameters that can set via rcParams can be found at https://matplotlib.org/
stable/api/matplotlib_configuration_api.html#matplotlib.RcParams.

Animations
Matplotlib has capabilities for creating animated plots. The Animations lab in Volume 4 has detailed
instructions on how to do so.

Matplotlib gallery and tutorials

The Matplotlib documentation has a number of tutorials, found at https://matplotlib.org/
stable/tutorials/index.html. It also has a large gallery of examples, found at https://matplotlib.
org/stable/gallery/index.html. Both of these are excellent sources of additional information
about ways to use and customize Matplotlib.
200 Appendix B. Matplotlib Customization
 Bibliography

[ADH+ 01] David Ascher, Paul F Dubois, Konrad Hinsen, Jim Hugunin, Travis Oliphant, et al.
Numerical python, 2001.

[Cip00] Barry A. Cipra. The best of the 20th century: editors name top 10 algorithms. Siam
news, 33(4), 16 May 2000.

[Hea02] Michael T Heath. Scientific computing. McGraw-Hill New York, 2002.

[Hun07] J. D. Hunter. Matplotlib: A 2d graphics environment. Computing In Science & Engi-

neering, 9(3):90–95, 2007.

[JOP+ ] Eric Jones, Travis Oliphant, Pearu Peterson, et al. SciPy: Open source scientific tools
for Python, 2001–. [Online; accessed 11/05/18].

[Kiu13] Jaan Kiusalaas. Numerical methods in engineering with Python 3. Cambridge university
press, 2013.

[MMH04] Neil Muller, Lourenco Magaia, and B. M. Herbst. Singular value decomposition, eigen-
faces, and 3D reconstructions. SIAM Rev., 46(3):518–545, 2004.

[New10] Mark Newman. Networks: an introduction. Oxford university press, 2010.

[Oli06] Travis E Oliphant. A guide to NumPy, volume 1. Trelgol Publishing USA, 2006.

[Oli07] Travis E Oliphant. Python for scientific computing. Computing in Science & Engineering,
9(3), 2007.

[QSS10] Alfio Quarteroni, Riccardo Sacco, and Fausto Saleri. Numerical mathematics, volume 37.
Springer Science & Business Media, 2010.

[SM00] Jianbo Shi and Jitendra Malik. Normalized cuts and image segmentation. IEEE Trans-
actions on Pattern Analysis and Machine Intelligence, 22(8):888–905, 2000.

[TB97] Lloyd N. Trefethen and David Bau, III. Numerical linear algebra. Society for Industrial
and Applied Mathematics (SIAM), Philadelphia, PA, 1997.

[VD10] Guido VanRossum and Fred L Drake. The python language reference. Python software
foundation Amsterdam, Netherlands, 2010.

201