Chapter 4

N U MER IC AL APPR O XIM AT IONS F O R SYST EM S O F AL G EBR AIC

EQ U AT ION S, A ⋅ x= b

1
 Slides are adapted from:

http://nm.mathforcollege.com/NumericalM
ethodsTextbookUnabridged/

Creative Commons — Attribution-NonCommercial-ShareAlike

4.0 International — CC BY-NC-SA 4.0

3
R E V IEW O F M ATR I CES A N D BA S I C L I N EA R A LGE B R A

4
source: Slide 3 5
 quantities
q1 q2 q3

𝑡𝑦𝑝𝑒1 1 2 3
sold types 𝑡𝑦𝑝𝑒2 2 5 1
𝑡𝑦𝑝𝑒3 1 2 2

source: Slide 3 6
source: Slide 3 7
source: Slide 3 8
source: Slide 3 9
source: Slide 3 10
source: Slide 3 11
source: Slide 3 12
source: Slide 3 13
source: Slide 3 14
source: Slide 3 15
source: Slide 3 16
source: Slide 3 17
source: Slide 3 18
source: Slide 3 19
source: Slide 3 20
source: Slide 3 21
source: Slide 3 22
source: Slide 3 23
source: Slide 3 24
source: Slide 3 25
source: Slide 3 26
source: Slide 3 27
source: Slide 3 28
source: Slide 3 29
source: Slide 3 30
source: Slide 3 31
source: Slide 3 32
source: Slide 3 33
source: Slide 3 34
source: Slide 3 35
source: Slide 3 36
source: Slide 3 37
source: Slide 3 38
source: Slide 3 39
source: Slide 3 40
source: Slide 3 41
source: Slide 3 42
source: Slide 3 43
source: Slide 3 44
source: Slide 3 45
source: Slide 3 46
source: Slide 3 47
Example

Let 𝑓𝑙𝑐ℎ𝑜𝑝 (𝑥) be the IEEE machine arithmetic model of 𝑥. Assume that the upper bound of the absolute machine
representation error of 𝑥 due to chopping is
𝑥 − 𝑓𝑙𝑐ℎ𝑜𝑝 𝑥 ≤ 𝜖𝑚𝑎𝑐ℎ
By using this computer system, the matrix summation
𝑲(𝑵×𝑵) = 𝑳(𝑵×𝑵) + 𝑼(𝑵×𝑵)

𝑲(𝑵×𝑵) = 𝑳෠ (𝑵×𝑵) + 𝑼
෡ (𝑵×𝑵) + 𝝐(𝑵×𝑵)

given below will be calculated. What will be the upper bound of the 𝐿2 −norm of the round-off error matrix
created by calculating 𝑲(𝑵×𝑵) ? 𝐿2 −norm of matrix A is calculated as

48
Example

Consider a matrix system given below

1+𝑘 1 1
𝑲(𝑵×𝑵) = 0 1−𝑘 1
0 0 𝑘

if the matrix is known to be invertible, what should be the value of 𝑘?

49
SYSTEM OF EQUATIONS

50
source: Slide 3 51
 Series 1
300 279

250
200 177

velocity
150 106
100
50
0
5 8 12
time

source: Slide 3 52
source: Slide 3 53
source: Slide 3 54
source: Slide 3 55
source: Slide 3 56
source: Slide 3 57
source: Slide 3 58
source: Slide 3 59
source: Slide 3 60
 This is not a computationally efficient way of computing the
determinant, and its complexity is proportional to n!
source: Slide 3 61
GAUSSIAN ELIMINATION

62
source: Slide 3 63
 1. Forward elimination

2. Back substitution

source: Slide 3 64
source: Slide 3 65
source: Slide 3 66
source: Slide 3 67
 11

22

33

44

55

source: Slide 3 68
source: Slide 3 69
source: Slide 3 70
 0
0 0
0 0 0
… … …
0 0 0

sum_

Source: Jaan Kiusalaas Numerical Methods in Engineering with Python3, Cambridge Unv. Press. 71
Source: Jaan Kiusalaas Numerical Methods in Engineering with Python3, Cambridge Unv. Press. 72
 300 279

250
200 177

velocity
150
106
100
50
0
5 8 12
time

73
 300 279

250
200 177

velocity
150
106
100
50
0
5 8 12
time

74
Forward Elimination

Back Substitutions

75
source: Slide 3 76
source: Slide 3 77
Forward Elimination

Multiplication of diagonal elements

source: Slide 3 78
source: Slide 3 79
 2𝑘 ⋯ 3
2−𝑘 0 The distance between 2𝑘 and 2−𝑘 may result with
⋯ ⋯ ⋯ the loss of precision inside the forward phase.
−1 ⋯ 1

source: Slide 3 80
Example

• Find the upper triangular matrix of the equation system below at the end of the Gaussian forward
elimination steps.

1𝑥1 + 15𝑥2 = 1
4𝑥1 − 20𝑥2 = 2

• Find the upper triangular matrix of the equation system below at the end of the Gaussian forward
elimination steps. Here 𝑞, 𝑝 ∈ ℜ+ are constants and 𝑥, 𝑦 ∈ ℜ are unknown variables.

𝑞𝑥 − 𝑝𝑦 = 1
𝑝𝑥 + 𝑞𝑦 = 2
What is the determinant of this system?

81
Examples

• Determinant of a 𝐵𝑛𝑛 upper triangular matrix can be found by the product of diagonal terms. Then,
det 𝐵 = ς𝑛𝑖=1 𝑏𝑖𝑖 . What are these diagonal elements for the matrix below.

1𝑥1 + 15𝑥2 = 1
0𝑥1 − 20𝑥2 = 2

• Determinant of a 𝐵𝑛𝑛 upper triangular matrix can be found by the product of diagonal terms. Then,
det 𝐵 = ς𝑛𝑖=1 𝑏𝑖𝑖 . Determine these diagonal elements for the matrix that represents the linear
system below.
1𝑥1 + 15𝑥2 = 1
20𝑥1 = 2

82
LU DECOMPOSITION

83
source: Slide 3 84
source: Slide 3 85
source: Slide 3 86
source: Slide 3 87
source: Slide 3 88
source: Slide 3 89
source: Slide 3 90
 Solve equations with LU

source: Slide 3 91
source: Slide 3 92
source: Slide 3 93
Time Complexity of An Algorithm

Time Complexity

The amount of time taken by an algorithm to run, as a function of the length of the input. If the
operation uses a matrix, then the size of the matrix should be considered.

For gauss elimination this is to be:

Then if all algebraic operations take the same CPU time, then the total execution time for GE will be a
8 4𝑛
function of polynomial 𝑛3 + 12𝑛2 + . Here 𝑛 is the size of the LHS matrix.
3 3

94
Example

Assume that the Gaussian Elimination process time complexity is a function of

2𝑛3
𝑇𝐺𝐸 =𝑇
3

where 𝑛 is the size of the matrix. If each algebraic operation takes 𝑝 seconds inside CPU, then how
much longer it takes to solve 𝑛 equations in 𝑛 unknowns if 𝑛 is tripled?

95
Example

GE algorithm have different time complexities for FE and BS. Assume the GE complexity below is
holding

if the Back Substitution phase took 𝑞 seconds in a CPU, then estimate the time required for Forward
Elimination phase. Show your result analytically.

96
Example
Assume that GE algorithm have a time complexity close to

2𝑛3
𝑇𝐺𝐸 =𝑇
3

then what will be the total time required for solving 𝑘 number of different linear systems by GE. This 𝑘
systems is supposed to have a dimension increment from 𝑛1 = 𝑛0 to 𝑛100 .

𝑛𝑖 = 𝑛𝑖 + 1

(The sum of integer cubes is S = [n2 (n + 1)2]/4 from k=1 to n )

97
ITERATIVE ALGORITHMS TO SOLVE LINEAR SYSTEM OF EQUATIONS

98
Fixed Point Iterations for Nonlinear Systems

For determining the position of nonlinear root estimate we can apply a short method called fixed point
iterations. The method is so simple that is commonly used represent the dynamics of iterative maps.

Assume we try to solve for 𝑓 𝑥 = 0. By arranging the original function to obey this form, one can get a
fixed form of iterative function,

𝑥=𝑔 𝑥

where 𝑓 𝑥 = 𝑔 𝑥 − 𝑥

As an example think about 𝑓 𝑥 = sin(𝑥). To solve 𝑓 𝑥 = 0, we will form a different version such that,

𝑓 𝑥 = sin 𝑥 − 𝑥 + 𝑥

which does not change the functional form.

99
Fixed Point Iterations for Nonlinear Systems

Since 𝑓 𝑥 = sin 𝑥 − 𝑥 + 𝑥, then

𝑥 = sin 𝑥 + 𝑥

For the iterative map, we can construct

𝑥𝑖+1 = 𝑔 𝑥𝑖

then now there is a link between the new prediction and the old value of root estimate.

𝑥𝑖+1 = sin 𝑥𝑖 + 𝑥𝑖

Now we can start with an initial 𝑥0 and find the next 𝑥𝑖 .

100
 Fixed Point Iterations for Nonlinear Systems

Kaynak: https://www.geogebra.org/m/s9ayD4NS 101

 Fixed Point Iterations for Nonlinear Systems

Kaynak: https://www.geogebra.org/m/s9ayD4NS 102

 Fixed Point Iterations for Nonlinear Systems

Kaynak: https://www.geogebra.org/m/s9ayD4NS 103

Fixed Point Iterations for Nonlinear Systems
For example, 𝑓 𝑥 = exp −𝑥 − 𝑥, then find the solution of 𝑓 𝑥 =0 by fixed point iter.

The iterative scheme will be:

𝑥𝑖+1 = exp −𝑥𝑖
Convergence
Assume the final root will be 𝑥𝑟 = 𝑔 𝑥𝑟 , since this will be the converged value at the end

𝑥𝑖+1 = 𝑔 𝑥𝑖
𝑥𝑟 = 𝑔 𝑥𝑟

if these two terms are summed up and by using derivative-mean value theorem

𝑔′ 𝜉 = (𝑔 𝑥𝑟 − 𝑔 𝑥𝑖 )/(𝑥𝑟 − 𝑥𝑖 ) → 𝑔 𝑥𝑟 − 𝑔 𝑥𝑖 = 𝑔′ 𝜉 𝑥𝑟 − 𝑥𝑖

𝑥𝑟 − 𝑥𝑖+1 = 𝑔′ 𝜉 𝑥𝑟 − 𝑥𝑖
𝐸𝑖+1 𝐸𝑖

If 𝑔′ 𝜉 < 1 errors decrease in each iteration, else errors cause divergence. 104
Fixed Point Iterations for Linear Matrix Systems
Similar to fixed point iterations for nonlinear equations, iterations can be applied for high dimensional
linear systems. For example the Jacobi Method is a form of Fixed-Point iterations.

For a 𝑁 × 𝑁 matrix 𝑨, let 𝑫 be the main diagonal of 𝑨, 𝑼 be the upper triangular part and 𝑳 be the
lower triangular part of 𝑨 where their diagonal entires are zero (𝑑𝑖𝑎𝑔 𝐿 = 0, 𝑑𝑖𝑎𝑔 𝑈 = 0).

For the linear system:

𝑨⋅𝒙=𝒃
Then,
(𝑫 + 𝑼 + 𝑳) ⋅ 𝒙 = 𝒃

and
𝑫 ⋅ 𝒙 = 𝒃 − (𝑼 + 𝑳) ⋅ 𝒙

𝒙 = 𝑫−𝟏 (𝒃 − 𝑼 + 𝑳 ⋅ 𝒙)

the last line is a fixed point representation similar to univariate case.

105
Fixed Point Iterations for Linear Matrix Systems

Jacobi Method for the linear system:

𝑨⋅𝒙=𝒃

𝒙 = 𝑫−𝟏 (𝒃 − 𝑼 + 𝑳 ⋅ 𝒙)

the iterative scheme is

𝒙𝑖+1 = 𝑫−𝟏 (𝒃 − 𝑼 + 𝑳 ⋅ 𝒙𝑖 )

for an initial vector 𝒙𝟎 .

106
Fixed Point Iterations for Linear Matrix Systems
Convergence of Iterative Methods: Lets consider Jacobi iterations,

𝒙𝑖+1 = 𝑫−𝟏 ⋅ 𝑼 + 𝑳 ⋅ 𝒙𝑖 + 𝑫−𝟏 𝒃

If the true solution is 𝒙, then at the end of an expected convergence 𝑖 → ∞

𝒙 = 𝑫−𝟏 ⋅ 𝑼 + 𝑳 ⋅ 𝒙 + 𝑫−𝟏 𝒃

is satisfied. If you take the difference of previous two equations side by side, it yields

𝒙 − 𝒙𝑖+1 = 𝑫−𝟏 ⋅ 𝑼 + 𝑳 ⋅ (𝒙 − 𝒙𝑖 )

𝑬𝒊+𝟏 = 𝑫−𝟏 ⋅ 𝑼 + 𝑳 ⋅ 𝑬𝒊

107
Fixed Point Iterations for Linear Matrix Systems
Convergence of Jacobi iterations, define 𝑲 = 𝑫−𝟏 ⋅ 𝑼 + 𝑳

𝑬𝒊+𝟏 = 𝑲 ⋅ 𝑬𝒊

it is seen that for iteration 1 and starting from 𝑖 = 0

𝑬1 = 𝑲 ⋅ 𝑬0
𝑬2 = 𝑲 ⋅ 𝑬1
…
𝑬𝑖 = 𝑲 ⋅ 𝑬𝒊−𝟏

if you replace error terms starting from 𝑬𝟏 side by side,

𝑬𝑖 = 𝑲𝑖 𝑬𝟎

Then convergence of error 𝑬𝑖 → 𝟎 can only be achieved for 𝑲𝑖 → 𝟎 which requires diagonal dominance.

108
Jacobi Iterations
• Jacobi iterations,

109
Fixed Point Iterations for Linear Matrix Systems

Theorem for Iterative Convergence

The iteration 𝑥𝑖+1 = 𝐵𝑥𝑖 + 𝑐 converges if and only if 𝜌 (𝐵) < 1. Here 𝜌 (𝐵) is called the spectral
radius of 𝐵. 𝜌 (𝐵) is defined by 𝜌 𝐵 = max |𝜆𝑘 | , where 𝜆𝑘 are the eigenvalues of 𝐵.
1≤𝑘≤𝑛

Example
An iterative algorithm below is given for 𝑥 ∈ ℜ.

𝑥 1
𝒄 − 𝒖𝑖 = + 𝒖𝑖 − 𝒖𝑖+1
0 𝑥

determine the region of 𝑥 values that guarantees the convergence of the iterative scheme.

110
Fixed Point Iterations for Linear Matrix Systems

Example
An iterative algorithm of type 𝑥𝑖+1 = 𝐵𝑥𝑖 + 𝑐 is given below.

2 −1
𝒖𝑖+1 = 𝒒 + 𝒖𝑖 − 𝒖𝑖
1 1

𝑇
Calculate the spectral radius for the convergence of iterative scheme where 𝒒 = 1,1 is a vector.

111
GAUSS-SEIDEL ITERATIONS

112
source: Slide 3 113
Gauss Seidel Iterations
• Gauss Seidel iterations,

114
source: Slide 3 115
source: Slide 3 116
Diagonally dominant matrices

(strictly) diagonally dominant if

117
source: Slide 3 118
>

>

>

119
source: Slide 3 120
source: Slide 3 121
Question
Solve the system with GS-iterations (2 iterations)

Solution:

122
source: Slide 3 123
ERROR MAGNIFICATION AND CONDITION NUMBER

124
 Condition of a System of Equations

Norm and condition number of a matrix, ||.|| is a norm

definition

https://courses.physics.illinois.edu/cs357/sp2020/notes/ref-10-condition.html
125
 Condition of a System of Equations

Norm and condition number of a matrix, ||.|| is a norm definition

𝑨𝒙 = 𝒃

How a change/error 𝝐𝑏 in 𝒃 will effect the resulting solution vector 𝒙 ?

𝑨 (𝒙 + 𝝐𝑥 ) = 𝒃 + 𝝐𝑏

Then, shortly

𝑨 𝝐𝑥 = 𝝐𝑏 → 𝝐𝑥 = 𝑨−1 𝝐𝑏

The error in 𝒃 is magnified by the inverse of 𝑨

https://courses.physics.illinois.edu/cs357/sp2020/notes/ref-10-condition.html
126
 Condition of a System of Equations

The error in 𝒃 is magnified by the inverse of 𝑨

𝑨 𝝐𝑥 = 𝝐𝑏 → 𝝐𝑥 = 𝑨−1 𝝐𝑏

What will be the relative error level of the norm of 𝒙 ?

The relative error of the norm of 𝒙 is defined by,

𝝐𝒙
𝝐𝒓𝒆𝒍,𝒙 =
𝒙

The relative error of the norm of 𝒃 is defined by,

𝝐𝒃
𝝐𝒓𝒆𝒍,𝒃 =
𝒃

https://courses.physics.illinois.edu/cs357/sp2020/notes/ref-10-condition.html
127
 Condition of a System of Equations
𝝐𝑥
𝝐𝒓𝒆𝒍,𝒙 𝒙 𝝐𝒙 𝒃
= 𝝐 =
𝝐𝒓𝒆𝒍,𝒃 𝒃 𝒙 𝝐𝒃
𝒃

𝝐𝑥
𝒙 𝑨−1 𝝐𝑏 𝑨𝒙 𝑨−𝟏 𝝐𝑏 𝑨 𝒙
𝝐𝒃 = 𝒙 𝝐𝒃
≤
𝒙 𝝐𝒃
𝒃

𝝐𝑥
𝒙 −𝟏
𝝐𝒃 = 𝑨 𝑨 = cond 𝑨
𝒃

https://courses.physics.illinois.edu/cs357/sp2020/notes/ref-10-condition.html
128
 Condition of a System of Equations
Condition number can also be given by the ratio of max and min valued eigenvalues of a matrix.

𝝐𝑥
𝒙 −𝟏
𝝐𝒃 ≤ 𝑨 𝑨 = cond 𝑨
𝒃

𝜆𝑚𝑎𝑥
cond 𝑨 =
𝜆𝑚𝑖𝑛

Condition number is always >1. High condition numbers >>1 are evaluated as ill-conditioned situations.

https://courses.physics.illinois.edu/cs357/sp2020/notes/ref-10-condition.html
129
Question
Calculate condition number of 𝑯 by using the eigenvalues.

1 4
𝑯=
0 2

Calculate the condition number of 𝑯 by using inverse matrix calculation,

130