we can solve by back substitution:

b4 (b3 − a34 x4 ) (b2 − a23 x3 − a24 x4 )

x4 = , x3 = , x2 = ,
a44 a33 a22
(b1 − a12 x2 − a13 x3 − a14 x4 )
x1 = .
a11
we can solve by forward elimination:
b1 (b2 − a21 x1 ) (b3 − a31 x1 − a32 x2 )
x1 = , x2 = , x3 = ,
a11 a22 a33
(b4 − a41 x1 − a42 x2 − a43 x3 )
x4 = .
a44
Since e⊤
i A = row i of A, the row operations can be written as

e⊤
i A
(1)
= e⊤ ⊤
i A − ℓi1 e1 A, 1 ≤ i ≤ 4, li1 = ai1 /a11

provided we let ℓ11 = 0, so that

 
0
ℓ21 
ℓi1 = e⊤
i ℓ
(1)
where ℓ(1) ℓ31  .
= 

ℓ41

In this way,
e⊤
i A
(1)
= e⊤ (1) ⊤ ⊤
i (A − ℓ e1 A) = ei M1 A

where the Gauss transformation (or elementary matrix) is defined by

M1 = I − ℓ(1) e⊤
1.

Therefore, the first step of the elimination is equivalent to left-multiplication

by the unit lower triangular matrix
 
1 · · ·
−ℓ21 1 · ·
M1 =  −ℓ31 ·
,
1 ·
−ℓ41 · · 1
that is,

1
  
× × × ×
· + + +
A(1) = M1 A = 
·
.
+ + +
· + + +
Here, we have introduced the convention that × indicates an entry unchanged
from the previous step, whereas + indicates an entry that has possibly changed.
Entries that must be zero are indicated by a dot (·). It might happen that
some of the × and + are also zero.
Let

U = A(3) = M3 A(2) = M3 M2 A(1) = M3 M2 M1 A,

and conclude that A = LU where

L = (M3 M2 M1 )−1 = M−1 −1 −1

1 M 2 M3 .
 
1 · · ·
ℓ21 1 · · 
L=ℓ31 ℓ32 1 ·  .


ℓ41 ℓ42 ℓ43 1

The simple version of Gaussian elimination, described above, will fail if any of
(1) (2) (n−2)
pivots a11 , a22 , a33 , . . ., an−1,n−1 is zero because we divide by these numbers
when computing the multipliers. As well as avoiding exactly zero pivots, we
should avoid very small pivots. Thus, L and U contain entries that are much
larger in magnitude than the entries of A. We will see later (Topic 7) that
extreme growth in the entries of L or UQ can lead to large roundoff errors
in the computed solution x.
In general, performing appropriate row swaps during the elimination process
produces a factorisation PA = LU where P is a permutation matrix and the
multipliers satisfy |ℓij | ≤ 1 for 1 ≤ j < i ≤ n.

M̂1 = I − ℓ̂1 e⊤
1

produces  
× × × ×
· + + +
A(1) = M̂1 P1 A = 
·
.
+ + +
· + + +

2
In the second step, suppose that
(1) (1)
|a42 | = max |ai2 |,
2≤i≤4

and swap rows 2 and 4 using

 
1 · · ·
· · · 1
P2 = 
·
.
· 1 ·
· 1 · ·

U = M3 P3 M̂2 P2 M̂1 P1 A = M3 M2 P3 P2 M̂1 P1 A = M3 M2 M1 P3 P2 P1 A,

giving

PA = LU where P = P3 P2 P1 and L = (M3 M2 M1 )−1 .

If we solve the unit lower triangular system Ly = Pb followed by the upper
triangular system Ux = y, then Ax = P−1 LUx = P−1 Ly = P−1 Pb = b.

procedure SWAP(j, p, A) for j = 1 : n − 1 do

pj = j SWAP(j, p, A)
for i = j + 1 : n do for i = j + 1 : n do
if |aij | > |apj ,j | then if ajj ̸= 0 then
pj ← i aij ← aij /ajj
end if for k = j + 1 : n do
end for aik ← aik − aij ajk
if pj ̸= j then end for
for k = 1 : n do end if
ajk ↔ apj ,k end for
end for end for
end if
end procedure

A matrix Q ∈ Rn×n is orthogonal if Q⊤ = Q−1 . An important geometric

property of orthogonal matrices is that they preserve lengths and angles, as
we see from the next theorem. If Q ∈ Rn×n is orthogonal, then (Qx) · (Qy) =
x · y for x, y ∈ Rn . ∥Qx∥ = ∥x∥.

3
Proof.
For any matrix A ∈ Rn×n ,
n n
! n n
!
X X X X
(Ax) · y = aij xj yi = xj aij yi = x · (A⊤ y).
i=1 j=1 j=1 i=1

A Householder transformation is a matrix H ∈ Rn×n of the form

2
H = I − τ vv⊤ for v ̸= 0 ∈ Rn and τ = .
∥v∥2
Equivalently,
v
H = I − 2v̂v̂⊤ where v̂ = .
∥v∥
First, since (vv⊤ )⊤ = (v⊤ )⊤ v⊤ = vv⊤ , we have

H⊤ = (I − τ vv⊤ )⊤ = I⊤ − τ (vv⊤ )⊤ = I − τ vv⊤ = H,

showing that H is symmetric. Second, since v̂⊤ v̂ = ∥v̂∥2 = 1,

H2 = (I − 2v̂v̂⊤ )(I − 2v̂v̂⊤ ) = I − 4v̂v̂⊤ + 4v̂v̂⊤ v̂v̂⊤ = I,

showing that H = H−1 .

Let x, y ∈ Rn and λ ∈ R. If ∥x∥ = ∥y∥, x ̸= y and λ ̸= 0, then the choice

v = λ(x − y)

defines a Householder transformation

H = I − 2v̂v̂⊤

with the property that

y = Hx and x = Hy.

Let a ∈ Rm and 1 ≤ j ≤ m − 1, and assume aj:m ̸= 0. Put

(
1, if aj ≥ 0,
σ=
−1, if aj < 0,

4
and define w ∈ Rm by

ai ,
 if 1 ≤ i ≤ j − 1,
wi = −σ∥aj:m ∥, if i = j,

0, if j + 1 ≤ i ≤ m,

so that
j−1
X m
X
2 2 2
∥w∥ = |ai | + ∥aj:m ∥ = |ai |2 = ∥a∥2 .
i=1 i=1

Since ∥a∥ = ∥w∥ and a ̸= w, for any λ ̸= 0 we have

2
Ha = w if H = I − τ vv⊤ , v = λ(a − w), τ= .
∥v∥2
We will use the choice
1
λ=
σ(|aj | + ∥aj:m ∥)
so that vj = λ(aj − wj ) = λ(σ|aj | + σ∥aj:m ∥) = λσ(|aj | + ∥aj:m ∥) = 1.
If j = 3 and m = 5, then
   
0 a1
 0   a2 
   
v= 1  and Ha = w = −σ∥a3:5 ∥ .
   
λa4   0 
λa5 0
   
× × × 0
 · + + 1
(2) (1)
   
A = H2 A =   · · +  with v2 = v32  .
  
 · · + v42 
· · + v52
Put
R = A(3) = H3 H2 H1 A
and, recalling that H−1 ⊤
j = Hj = Hj ,

Q = (H3 H2 H1 )−1 = H−1 −1 −1

1 H 2 H3 = H1 H2 H3 ,

5
If Q = [Q1 Q2 ] with Q1 ∈ Rm×n and Q2 ∈ Rm×(m−n) , and
 
R1
R= with R1 ∈ Rn×n and 0 ∈ R(m−n)×n .
0
Since QR = Q1 R1 + Q2 0 = Q1 R1 , we obtain the reduced QR-factorisation,
A = Q1 R1 .
A vector x ∈ Rn is a least-squares solution of (3) iff

∥Ax − b∥ = minn ∥Ay − b∥.

y∈R

Assume A ∈ Rm×n with rank(A) = n ≤ m. Then, in the reduced QR-

factorisation A = Q1 R1 , the factor R1 is non-singular, and the unique least-
squares solution of (3) is the solution of the n×n upper triangular system R1 x =
Q⊤1 b.
Proof.
Since Q is non-singular and A = QR we have n = rank(A) = rank(R) =
rank(R1 ), implying that R1 is non-singular. Furthermore,
Ay − b = QRy − QQ⊤ b = Q(Ry − Q⊤ b)
so, because Q is orthogonal,
∥Ay − b∥ = ∥Ry − Q⊤ b∥.
Moreover,
   ⊤   ⊤

R 1 y Q b R 1 y − Q b
Ry − Q⊤ b = − 1
= 1
,
0 Q⊤
2 b −Q ⊤
2 b
implying that
∥Ax − b∥2 = ∥R1 y − Q⊤ 2 ⊤ 2
1 b∥ + ∥Q2 b∥ ,

which is minimised precisely when y = x where R1 x = Q⊤

1 b. And ∥Ay−b∥ =
⊤
∥Q2 b∥
Definition
An n × n Givens transformation is a matrix of the form
X
⊤ ⊤ ⊤ ⊤
G = cep ep + seq ep − sep eq + ceq eq + ei e⊤
i
1≤i≤n
i∈{p,q}
/

6
where 1 ≤ p < q ≤ n and c2 + s2 = 1.
Given a ∈ Rn , we can choose G so that b = Ga satisfies
q
bp = a2p + a2q , bq = 0, bi = ai for i ∈ / {p, q}.
ap −aq
c= q and s = q .
a2p + a2q a2p + a2q
Definition
We say that A = R⊤ R is a Cholesky factorisation if R is upper triangular.
Consider the 3 × 3 case
   
a11 a12 a13 r11 r12 r13
A = a21
 a22 a23  and R =  · r22 r23  .
a31 a32 a33 · · r33

We find that
2

r11 r11 r12 r11 r13
R⊤ R = r11 r12 2
r12 2
+ r22 r12 r13 + r22 r23  .
2 2 2
r11 r13 r13 r12 + r23 r22 r13 + r23 + r33

In the general n × n case, A = R⊤ R iff

i−1
X
aij = rii rij + rki rkj for 1 ≤ i < j ≤ n,
k=1

and
j−1
X
2
ajj = (rjj ) + (rkj )2 for 1 ≤ j ≤ n.
k=1
Thus, we require
i−1
!
1 X
rij = aij − rki rkj for 1 ≤ i < j ≤ n,
rii
k=1

and v
u
u j−1
X
rjj = tajj − (rkj )2 for 1 ≤ j ≤ n.
k=1

7
Let A ∈ Rm×n .
1. The upper bandwidth of aij = 0 is the smallest integer U = U(A) such
that
aij = 0 whenever j − i > U.
2. The lower bandwidth of aij = 0 is the smallest integer L = L(A) such that
aij = 0 whenever i − j > L.
Theorem
Let A ∈ Rm×n and B ∈ Rn×p . U(AB) ≤ U(A) + U(B). L(AB) ≤ L(A) +
L(B).
Theorem
Assume that A ∈ Rn×n has the LU-factorisation A = LU (with no pivoting).
U(U) ≤ U(A). L(L) ≤ L(A).
We can use the LAPACK band storage scheme to store A in an (L+1+U)×n
array AB by setting
AB[U + 1 + i − j, j] = aij
for max(1, j − U) ≤ i ≤ min(m, j + L).
Again with n = 5, L = 2 and U = 1, the band LU-factorisation can be
re-written so that it overwrites A with
 
∗ u12 u23 u34 u45
u11 u22 u33 u44 u55 
 
 ℓ21 ℓ32 ℓ43 ℓ54 ∗
ℓ31 ℓ42 ℓ54 ∗ ∗
Theorem
Suppose that PA = LU is the LU-factorisation of A with pivoting.
U(U) ≤ L(A) + U(A). L(L) ≤ L(A).
Theorem
Let A = QR as above. U(R) ≤ L(A) + U(A). L(Q) ≤ L(A).
Example
Consider once again the matrix
 
0 −2 1 0 0 −3 0 1
 0 1 2 0 −5 1 1 0
 
A=  1 0 0 −7 0 0 0 3 .

−2 0 1 8 0 3 −1 0
0 5 −6 0 1 0 0 0

8
Storing in CSC format, we have
colptr = [1, 3, 6, 10, 12, 14, 17, 19, 21],
rowval = [3, 4, 1, 2, 5, 1, 2, 4, 5, 3, 4, 2, 5, 1, 2, 4, 2, 4, 1, 3],
nzval= [1, -2, -2, 1, 5, 1, 2, 1, -6, -7, 8, -5, 1, -3, 1, 3, 1, -1, 1, 3].

Storing in CSR format (with row-major ordering) as follows.

rowptr = [1, 5, 10, 13, 18, 21],
colval = [2, 3, 6, 8, 2, 3, 5, 6, 7, 1, 4, 8, 1, 3, 4, 6, 7, 2, 3, 5],
nzval = [-2, 1, -3, 1, 2, -5, 1, 1, 1, -7, 3, -2, 1, 8, 3, -1, 5, -6, 1].