Solution to Systems of

Linear Equations II
Outline

= 𝑛)
• Direct Methods (𝑚

• Inverse Method
• Cramer’s Rule
• L-U Decomposition
Solution to Systems of Linear Equations
• Direct Methods are based on algebraic elimination that obtains the
solution in fixed number of operations.
• Usually for small number of equations (≤ 100)
• Most of the coefficients 𝑎 𝑖𝑗 are nonzero
• The system of equations is not diagonally dominant
• The system of equations is ill-conditioned.
• Iterative Methods are based on guessing the solution and then using
a systematic method to improve the estimated solution repeatedly.
• Usually used for large number of equations (> 100)
Solution to Systems of Linear Equations
• Diagonally dominant matrix is a matrix where the magnitude of the
diagonal elements is greater than or equal to the other elements of
the same row.
• An ill-conditioned matrix is a matrix where a small change in the
constant coefficients results in a large change in the solution.

𝑥+𝑦=2 𝑥+𝑦=
• Example:

ቊ ቊ
𝑥+2 1.001𝑦 𝑥 + 1.001𝑦 =
and
• Solution: = 𝑥,
2 𝑦= 2.001
𝑥, 𝑦=
(2,0) (1,1)

https://emtiyaz.github.io/pcml15/illconditioned.pdf
Solution to Systems of Linear Equations
• The following direct methods are only applicable when 𝑚
=𝑛
(number of equations is the same as number of unknowns):
• Inverse Method
• Cramer’s Rule

• The following direct methods are applicable when 𝑚 ≥ 𝑛:

• L-U Decomposition

• Gaussian Elimination Method

• Gauss-Jordan Reduction Method
Inverse Method

𝐴𝑋 = 𝐵
• Given a system of equations in matrix form:

• We multiply the inverse of 𝐴 (or simply 𝐴−1) to the left of both

𝐴−1𝐴𝑋 = 𝐴−1𝐵
sides of the equation:

• Note that it must be specified that it is multiplied to the left since matrix
multiplication is associative but NOT COMMUTATIVE.
Inverse Method
• Simplifying: (𝐼 is the identity matrix of same size as matrix 𝐴)
𝐴−1𝐴 = 𝐼
• Recall: an identity matrix of size n is a square matrix with size n whose
elements are zeros except for the elements of the main diagonal which are
equal to one.

𝐼𝑋 = 𝐴−1𝐵
• Substituting:

• Recall: An identity matrix 𝐼 multiplied to any matrix 𝑀 will result to the

𝑀 provided that the matrices can be multiplied.

matrix

𝑿 = 𝑨−𝟏𝑩
• Simplifying:
Inverse Method
𝑿 = 𝑨−𝟏𝑩
• The Inverse Method makes use of the properties of
matrix multiplication and inverse to solve the system of linear
equation.
• FAQ:
• Q: What if the matrix of coefficients is not invertible (determinant is equal to
zero)?
• A: The system of linear equations has no unique solution.
• Q: How to solve the system using the inverse method?
• A: Just multiply the inverse of the matrix of coefficients to the vector of constants
(coefficients on the left, constants on the right)
Inverse Method
−𝑥 + 2𝑦 − 𝑧 =
3/31 4/31 11/3 9
9
• Solve: 3𝑥 − 7𝑦 − 2𝑧 = −20 𝑋 = 𝐴−1𝐵 7/31 −1/3 1 −2
= −20/3 1 5/31 0
2𝑥 + 2𝑦 + 𝑧 =
1 −6/3 −1/3 2
2 𝑥 3 4 11
−1 2 9 9 + 1 −20 + 1 ( )
• 𝐴𝑋
−1 = 𝐵 𝑦 −2 3 31 (2) 31
3 −7 = 0 1 5
•
1
−2 =
• det 𝑧 2 𝑋 9 + −20 +
37 − ( )(2)
𝐴2 2 −3 −4
−31 = 31
1 6 1
1 𝐴 −11
• 𝑎𝑑𝑗 −2 9 + 31 −20 + (− )
3 3 3
= −7 1 −5 0 − (2)
1 1 1
20 64/311 11/3
3/31 𝒙 −𝟏
• 𝐴−1 7/31 −1/3 1 𝑿 =𝒚 𝟑
−20/3 1 5/31 = −
= =
• 𝑋 1 −6/3 −1/3 𝒛 𝟐
1 1
𝐴−1𝐵
Cramer’s Rule
𝑨
𝒙�
|𝒊
=
• where 𝐴𝑖 is the matrix you get when 𝑨|replacing the 𝑖 𝑡ℎ column
�

of matrix 𝐴 (coefficients) with the elements of vector 𝐵

(constants)
• How to solve systems using Cramer’s Rule:
1. Replace first column of matrix A with vector B
2. Solve the determinants of new matrix and original matrix A
3. Divide the determinants, this gives the value of the first unknown
4. Repeat for the rest of the columns of matrix A
Cramer’s Rule
−𝑥 + 2𝑦 − 𝑧 = 9 2 −1
−20 −7
9 −2
𝑥1 = 𝑥 = 2 −31 2 1 =
•Solve: 3𝑥 − 7𝑦 − 2𝑧 = −𝟏
−1 9
−20 −1
−1 2 2𝑥 −1+ � + 𝑧 9=
2𝑦 3 −20
• 3 −72 −2 � = −2 2
𝑥2 = 𝑦 = −2 −31 2 1

2 2 � 0 =𝟑
• 𝐴𝑋
det = = 𝐵 −1 2 9
1 2 3 −7
•
𝐴 𝐴 −31 �
�
−20
• 𝑥𝑖 𝑥3 = 𝑧 = 2
−3
2 2
|𝐴|
= = −𝟐 1
𝑖 �
L-U Decomposition
• L-U Decomposition (L-U Factorization) – L-U in the name means
Lower-Upper. This is because we factor the matrix of coefficients A

𝐴 = 𝐿𝑈
into two: An upper triangular matrix and a lower triangular matrix.
𝑎11 𝑎12 𝑎1 0 𝑢11 𝑢12
𝑙1
𝑎21 𝑎22 3 = 𝑙1 0
𝑎31 𝑎32 𝑎23 𝑙 0 𝑢13
2
1 22
𝑎 33 0 𝑢22
𝑙3 𝑙32 𝑢23
• Three methods:
1 0 0
• Crout’s Method
𝑙33
𝑢33
• Doolittle’s Method
• Cholesky’s Method
L-U Decomposition
• L-U Decomposition (L-U Factorization)

𝐴 = 𝐿𝑈
• Crout’s Method

𝑎1 𝑎1 𝑎13 0 1 𝑢12
𝑙1
1 2 𝑎23 = 𝑙1 0 𝑢13
𝑎21 𝑎22 𝑎33 2
𝑙22 0 0 1 𝑢23
1
𝑎31 𝑎32 0 0 1
𝑙3 𝑙32
1
𝑙33
L-U Decomposition
• L-U Decomposition (L-U Factorization)

• 𝑓𝑜𝑟 𝑖 = 1: 𝑛
• Crout’s Method
𝑙 1,1 , 𝑢 1,2 , 𝑢 1,3 , … , 𝑢(1,
• Compute the elements in the order
𝑛)
• 𝑢 𝑖, 𝑖 = 1; 𝑙 2,1 , 𝑙 2,2 , 𝑢 2,3 , … , 𝑢 2, 𝑛
• 𝑓𝑜𝑟 𝑗 = …
1: 𝑛 𝑙 𝑛, 1 , 𝑙 𝑛, 2 , 𝑙 𝑛, 3 , … ,
𝑙(𝑛, 𝑛)
• 𝑖𝑓 𝑖 ≥ 𝑗
𝑗−1

−෍ 𝑙 ∗
𝑙 𝑖, 𝑗= 𝑎 𝑖, 𝑖, 𝑘=1
𝑘 𝑢(𝑘, 𝑗)
𝑗 • 𝑒𝑙𝑠𝑒𝑖𝑓 𝑖
<
𝑎 𝑗 1 1
𝑖−

𝑢 𝑖, 𝑖, 𝑗 − ෍ 𝑙 ∗
𝑗 𝑙 𝑖, 𝑖, 𝑘 𝑢(𝑘, 𝑗)
=
𝑙 𝑖 𝑘=1
𝑖, 𝑖
L-U Decomposition
• L-U Decomposition (L-U Factorization)

𝐴 = 𝐿𝑈
• Doolittle’s Method

𝑎1 𝑎12 𝑎1 1 0 𝑢11 𝑢12

1 𝑎22 3 = 0
𝑎21 𝑎32 𝑎23 𝑙21 1 𝑢13
𝑎31 𝑎33 0 0 𝑢22
𝑢23
𝑙31 𝑙32 1
0 0
𝑢33
L-U Decomposition
• L-U Decomposition (L-U Factorization)

• 𝑓𝑜𝑟 𝑖 = 1: 𝑛
• Doolittle’s Method
𝑢 1,1 , 𝑢 1,2 , … , 𝑢 1, 𝑛 −
• Compute the elements in the order
1 , 𝑢(1, 𝑛)
• 𝑙 𝑖, 𝑖 = 1; 𝑙 2,1 , 𝑢 2,2 , … , 𝑢 2, 𝑛 − 1 , 𝑢
• 𝑓𝑜𝑟 𝑗 = 2, 𝑛
1: 𝑛 …
𝑙 𝑛, 1 , 𝑢 𝑛, 2 , … , 𝑙 𝑛, 𝑛 −
• 𝑖𝑓 𝑖 ≤ 𝑗
𝑖−1
1 , 𝑢 𝑛, 𝑛
−෍ 𝑙 ∗
𝑢 𝑖, 𝑗= 𝑎 𝑖, 𝑘=1
𝑘 𝑢(𝑘, 𝑗)
𝑖, 𝑗 • 𝑒𝑙𝑠𝑒𝑖𝑓 𝑖 > 𝑗
𝑗−1
𝑎 1
𝑙 𝑖, 𝑖, 𝑗 − ෍ 𝑙 ∗
𝑗 𝑢 𝑗, 𝑖, 𝑘 𝑢(𝑘, 𝑗)
=
𝑢 𝑗, 𝑗 𝑘=1
𝑗
L-U Decomposition
• L-U Decomposition (L-U Factorization)

𝐴 = 𝐿𝐿𝑇
• Cholesky’s Method – useful for symmetric matrices

𝑎11 𝑎12 𝑎13 0

𝑙 𝑙11 𝑙21
𝑎21 𝑎22 𝑎23 = 1 0
𝑙121
𝑎31 𝑎32 𝑎33 𝑙22 0 𝑙31
𝑙31 or 0 𝑙22
𝑙32 𝑙32
𝐴=
𝑎11 𝑎12 𝑎13 𝑢11𝑈 𝑇0𝑈
𝑙33 0
𝑢11
0
𝑢12
𝑎21 𝑎22 𝑎23 = 𝑢12 0 𝑙33
𝑎31 𝑎32 𝑎33 𝑢13 𝑢22 0 𝑢13
0 𝑢22
𝑢23 𝑢23
L-U Decomposition
• L-U Decomposition (L-U Factorization)

• 𝑓𝑜𝑟 𝑖 = 1: 𝑛
• Cholesky’s Method
𝑢 1,1 , 𝑢 1,2 , … , 𝑢 1, 𝑛 −
• Compute the elements in the order
• 𝑓𝑜𝑟 𝑗 = 1: 𝑛 1 , 𝑢(1, 𝑛)
• 𝑖𝑓 𝑖 == 𝑗 𝑙 2,1 , 𝑢 2,2 , … , 𝑢 2, 𝑛 − 1 , 𝑢
𝑖−1 2, 𝑛
𝑢 𝑖, 𝑖 = 𝑠𝑞𝑟𝑡 𝑎 𝑖, − ෍ 𝑢 2 …
𝑙 𝑛, 1 , 𝑢 𝑛, 2 , … , 𝑙 𝑛, 𝑛 −
𝑖 𝑘, 𝑖
1 , 𝑢 𝑛, 𝑛
𝑘=1
• 𝑒𝑙𝑠𝑒𝑖𝑓
𝑎 𝑖, 𝑗 𝑖 < 𝑗
𝑖−1

𝑢 𝑖, 𝑗 1 − ෍ 𝑢 ∗
𝑢 𝑖, 𝑖 𝑢 𝑖,
= 𝑘, 𝑖 𝑢(𝑘, 𝑗)
𝑖
• 𝑒𝑙𝑠𝑒𝑖𝑓 𝑖 > 𝑗 𝑘=1
𝑢 𝑖, 𝑗

=0
L-U Decomposition
𝐴𝑋 = 𝐵
𝐴 = 𝐿𝑈
𝐿𝑈𝑋 = 𝐵
𝐿 𝑈𝑋
=𝐵

Let Y = 𝑈𝑋
Recall: Matrix multiplication is ASSOCIATIVE but not commutative

𝐿𝑌 = 𝐵
By using L-U Decomposition, we can break down the process of solving the
system of equations into two “simpler” steps. (Simpler in terms of computer
1. Solve for 𝑌 in 𝐿𝑌 = 𝐵 using forward substitution
calculations, but not manually)
2. Solve for 𝑋 in 𝑈𝑋 = 𝑌 using back substitution
L-U Decomposition
−𝑥 + 2𝑦 − 𝑧 =
9 𝑎11 =
1 + 0 0 + 0 0 = 𝑙11 =
• Solve: 3𝑥 − 7𝑦 − 2𝑧 = −20 𝑎12 = 𝑙12𝑢 −112 + 0 1 +0 0
2𝑥 + 2𝑦 + 𝑧 = 𝑙11 = 𝑙 𝑢 + 0 =+𝑙 0𝑢1 = 2 =
𝑎
2
13 11 13 𝑙11𝑢13 = −1
11 12

𝑥 𝑎 =
−1 2 9 𝑎 21 = 𝑙21𝑢
𝑢22 112++ 𝑙22 10 + 0 0 = 𝑙21 = 3
• 𝐴𝑋
−1 = 𝐵 𝑦 −2
23
𝑙22 = 𝑙21𝑢12 +
𝑙21 0
3 −7 = 0 𝑎
=23−7= 𝑙21𝑢13 + 𝑙22𝑢23 + 𝑙
•

• 𝐿𝑌−2= 𝐵 𝑧 2 0 1
22

2
• 𝑈𝑋 = 𝑌
2 = 𝑙21𝑢13 + 𝑙22𝑢 023+=𝑙33−20 = 𝑙31 =
1 𝑎
𝑎31 = 𝑙 +
32 = 𝑙3131𝑢12 +2𝑙321 + 𝑙33 = 𝑙31𝑢12 +
𝑙32
𝑙32 0
• Crout’s Method:
−1 2 0 1 𝑢12 𝑢13 =
𝑎332 = 𝑙31𝑢13 + 𝑙32𝑢23 + 1
𝑙11
• −1 = 0 0 1 𝑢23 𝑙33
3 −7 𝑙21 𝑙22 0 0 0 1 = 𝑙31𝑢13 + 𝑙32𝑢23 + 𝑙33
−2
𝑙31 𝑙32 = 1
2 2
1 𝑙33
L-U Decomposition
𝑙11 = −1 𝑙11 = −1
−𝑥 + 2𝑦 − 𝑧 =
9
• Solve: 3𝑥 − 7𝑦 − 2𝑧 = −20 𝑙11𝑢12 = 2 𝑢12 = −2
2𝑥 + 2𝑦 + 𝑧 = 𝑙11𝑢13 = −1 𝑢13 = 1
−1 2
2
9 𝑙21 = 3 𝑙21 = 3
𝐴𝑋
• −1 = 𝐵 � = −2 𝑙21𝑢12 + 𝑙22 = −7 𝑙22 = −1
�

� 𝑙21𝑢13 + 𝑙22𝑢23 = −2 𝑢23 = 5

•
3 −7 0
• 𝐿𝑌−2= 𝐵 2 𝑙31 = 2 𝑙31 = 2
𝑙32 = 6
�
𝑙31𝑢12 + 𝑙32 = 2
2 2
�

• 𝑈𝑋1 = 𝑌
𝑙33 =
𝑙31𝑢13 + 𝑙32𝑢23 + 𝑙33 =
�

0 0 −31
• Crout’s Method
−1 2 −1 𝑙11 1 𝑢12 −1 0 0
3 −7 −2 = 𝑙21 𝑙22 0 0
𝑢13 1 𝐿 = 3 −1 0
1
2
•
2 2 1 𝑙31 𝑙32 𝑙33 0
−1 2 −1 −1 0 0 1 𝑢−2 1
6
23

• 3 −7 −2 = 3 −1 0
0 0 1 5 1 −2
−31
2 2 1 2 6 −31 0 1 0 1
𝑈 1
= 0 1
5
0 0
L-U Decomposition
−𝑥 + 2𝑦 − 𝑧 =
9 • 𝐿𝑌
• Solve: 3𝑥 − 7𝑦 − 2𝑧 = −20 =−1 𝐵 0 0 𝑦 9
2𝑥 + 2𝑦 + 𝑧 =
• 3 −1 0 1 = −2
𝑦
2
−1 2 9 2 6 0
𝐴𝑋
• −1 = 𝐵 � = −2 𝑦 − 2
�
2
�
•
3 −7 0
−2= 𝐵 2 2 = 9
1−31 𝑦
• 𝐿𝑌
2 2 � • 𝑦 −
�
3
• 𝑈𝑋 = 𝑌
1
• 𝑈𝑋
3 = 7
𝑌1 −2 −1 𝑥
�

−
• Crout’s Method
−1 0 0
• 𝐿 = 3 −1 • 0 1 25 1 = 9
0 0 0 1 𝑥 −
2 𝑥1 7
• 𝑈 2
= 6 𝑥 −
= −1 2
•
3
−31 𝑥2
1
−2
1 3
Seatwork
−𝑥 + 2𝑦 − 𝑧 = 9
• Perform the lower-upper decomposition method on the same given using Doolittle’s Method:

•Solve: 3𝑥 − 7𝑦 − 2𝑧 = −20
2𝑥 + 2𝑦 + 𝑧 = 2
• 𝐴𝑋 =𝐵
−1 2 −1 9
• 3 −7 −2 �= −2
�

2 2 � 0
• 𝐿𝑌 1 2
=𝐵 �
�

• 𝑈𝑋
−1 =2 −1 1 0
�
𝑢11 𝑢12
•𝑌 3 −7 −2 = 0
2 2 𝑙21 1 𝑢13
1 0 0 𝑢22
𝑢23
𝑙31 𝑙32 1
0 0
Any questions?