Chapter 2: Solving Linear Equations

Vu Van Thieu, Dinh Viet Sang, Ban Ha Bang

SoICT
Hanoi University of Science and Technology

1
Introduction
1 What is a system of linear equations?
2 3-dimensional example
3 Permutation matrices and triangular matrices
4 LU analysis
5 Role of the pivot element
6 Impact of rounding error
7 Bad determinism matrix and matrix conditions
Matrix norm
Number of matrix conditions
Evaluate error when the number of conditions of the matrix is known
8 Solving a system of linear equations using matrix analysis
9 Summary
2
System of Linear Equations

Definition
a11 x1 + a12 x2 + · · · + a1n xn = b1
a21 x1 + a22 x2 + · · · + a2n xn = b2
···
am1 x1 + am2 x2 + · · · + amn xn = bm
Denote
A = (aij ) where i = 1, · · · , m and j = 1, · · · , n is the coefficient matrix A.
b = (b1 , b2 , · · · , bm )T is the right-hand side vector.
x = (x1 , x2 , · · · , xn )T is the variable vector.
We can rewrite the system of linear equations in matrix form

Ax = b
3
System of Linear Equations

Example 1 :
Consider a system of linear equations with
!
3 2
Coefficient Matrix A =
1 −1
!
−first
The right-hand side vector is b =
first
!
0.2
then the system has a unique solution x =
−0.8

4
System of Linear Equations

Example 2 :
Consider a system of linear equations with
!
1 0 3
Coefficient Matrix A =
0 1 −5
!
1
The right-hand side vector is b =
2
 
1 − 3t
then the system has infinitely many solutions x = 2 + 5t  for every t ∈ R.
 

5
System of Linear Equations

Example 3 :
Consider a system of linear equations with
 
1 0
Coefficient Matrix A = 0 1
 

3 4
 
1
The right-hand side vector is b = 2
 

3
then the system has no solution.

6
System of Linear Equations

For a system of possible linear equations

m = n : square system (number of equations equals the unknowns, often with a unique
solution)
m < n : missing system (the number of equations is less than the number of unknowns,
the system is usually infinitely many solutions)
m > n : residual system (number of pequations more than unknowns, the system usually
has no solution)

7
Solve a system of linear equations

System of square equations

Ax = b
where A ∈ Rn×n and x and b are vectors ∈ Rn

Solve the system of square equations

If the matrix A is not singular, the only solution to the equation is

x = A−1 b

Matlab
» x=inv(A)*b

8
Solve a system of linear equations

Example 4: :
Solve the system of equations A = (7) and b = (21) or equation

7x = 21

Method 1: Solve the division directly x = 21/7 = 3

Method 2: Invert 7−1 and multiply by 21 will result in

x = 7−1 × 21 = 0.142857 × 21 = 2.99997

Obviously, method 1 is better than method 2, in addition, method 2 has a larger

computational load when determining the inverse 7−1 .

9
Solve a system of linear equations

Comment
The use of the inverse matrix gives a less precise solution.
When solving a system of linear equations, we often find the solution directly and only use the
inverse matrix A−1 in a few situations.
Some methods:
LU Factorization (LU Factorization)
Cholesky Analysis (Cholesky Factorization)
QR Decomposition

10
Solve a system of linear equations

Matrix division operator in Matlab

If A is any matrix and B is a matrix with the same number of rows as A, then the solution of
the system of equations
AX = B
then we use left division X = A\B.
And the solution of the system of equations

XA = B

then we use right division X = B/A.

11
Solve a system of linear equations

Example 5: Split left

» A=[3 2;1 -1];b=[-1;1];
» x = A\b;
x = 0.2000
-0.8000

Example 6: Right split

» AA=A’;bb=b’;
» xx = bb/AA;
xx = 0.2000 -0.8000

12
Solve a system of linear equations

Basic quantities when solving systems of square equations

Determinant is an important numerical feature of square matrices that allows the
determination of the number of solutions of HPT (zero or infinitely many solutions, or
unique solutions).
Trace is the sum of the main diagonal elements.
Rank is the maximum number of linearly independent rows or columns of the matrix.

Matlab
» D=det(A)
» T=trace(A)
» R=rank(A)

13
Solve a system of linear equations

Kronecker-Capelli theorem
The system of linear equations Ax = b has a solution if and only if

rank(A) = rank(Ab)

Example 7 : same rank

» A=[1 2 3; 4 5 6; 8 10 12];
» b=[5;6;12];
» rA=rank(A);
» rAb=rank([Ab])
rA = 2
rAb = 2

14
Solve a system of square linear equations

Instances when solving Ax = b

The system of equations has a unique solution if det(A) ̸= 0.
When det(A) = 0 the system of equations can have infinitely many solutions or no
solutions (We can apply the Kronecker-Capelli theorem to determine whether it has no
solution or infinitely many solutions).
When det(A) ̸= 0, there exists an inverse matrix of A and A which is called a
non-degenerate matrix.
When det(A) = 0 then the inverse matrix A−1 does not exist and A is called degenerate
matrix.

15
Solve a system of square linear equations

Example 8 :
» A1=[-1 1; -2 2];b1=[1 ; 0];
% system has no solution
» x1 = A1\b1, D1 = det(A1)
Warning : Matrix is singular to working precision.
x1 = Inf
Inf
D1=0

16
Solve a system of square linear equations

Example 9 :
» A2=[-1 1; -2 2];b2=[1 ; 2];
% system has infinitely many solutions
» x2 = A2\b2
Warning : Matrix is singular to working precision.
x2 = -1
0

17
Solve a system of square linear equations

18
1 What is a system of linear equations?
2 3-dimensional example
3 Permutation matrices and triangular matrices
4 LU analysis
5 Role of the pivot element
6 Impact of rounding error
7 Bad determinism matrix and matrix conditions
Matrix norm
Number of matrix conditions
Evaluate error when the number of conditions of the matrix is known
8 Solving a system of linear equations using matrix analysis
9 Summary

19
Solve a system of square linear equations

3-dimensional example
Given the following system of 3rd degree equations:
    
10 −7 0 x1 7
−3 2 6 x2  = 4
    

5 −1 5 x3 6

We rewrite it as a system of linear equations

10x1 − 7x2 =7
−3x1 + 2x2 + 6x3 = 4
5x1 − x2 + 5x3 = 6

20
Solve a system of square linear equations

3-dimensional example (continued)

System of linear equations

10x1 − 7x2 =7 (1)

−3x1 + 2x2 + 6x3 = 4 (2)
5x1 − x2 + 5x3 = 6 (3)

we proceed to solve
Remove x1 ⇒ (2) − (1) × (−0.3) and (3) − (1) × 0.5
The factor of 10 of the x1 in (1) is called pivot element, the coefficients -0.3 and 0.5 are
called factor.

21
Solve a system of square linear equations

3-dimensional example (continued)

System of linear equations after eliminating x1

10x1 − 7x2 =7 (4)

−0.1x2 + 6x3 = 6.1 (5)
2.5x2 + 5x3 = 2.5 (6)

we continue to solve
Remove x2 ⇒ because the pivot element of x2 in (5) is -0.1 has a small absolute value,
we proceed to swap the two equations (5) and (6) and then proceed to eliminate x2 .
This is called rotation

22
Solve a system of square linear equations

3-dimensional example (continued)

System of linear equations after performing the rotation

10x1 − 7x2 =7 (7)

2.5x2 + 5x3 = 2.5 (8)
−0.1x2 + 6x3 = 6.1 (9)

we continue to solve
Remove x2 ⇒ (9) − (8) × (−0.04)

23
Solve a system of square linear equations

3-dimensional example (continued)

System of linear equations after performing the elimination x2

10x1 − 7x2 =7 (10)

2.5x2 + 5x3 = 2.5 (11)
6.2x3 = 6.2 (12)

we continue to solve
From equation (12) ⇒ x3 = 1.
replace x3 in (11) then 2.5x2 + 5 × (1) = 2.5 ⇒ x2 = −1.
replace x2 in (10) then 10x1 − 7 × (−1) = 7 ⇒ x1 = 0.

24
Solve a system of square linear equations
Matrixes L,U,P
The entire solution just presented can be encapsulated in the following matrices
     
1 0 0 10 −7 0 1 0 0
L =  0.5 1 0 , U =  0 2.5 5  , P = 0 0 1
     

−0.3 −0.04 1 0 0 6.2 0 1 0

With
L is the matrix containing the factors
U is the final coefficient matrix
P is the permutation matrix describing the rotation
then we have

LU = PA
25
Solve a system of square linear equations

26
1 What is a system of linear equations?
2 3-dimensional example
3 Permutation matrices and triangular matrices
4 LU analysis
5 Role of the pivot element
6 Impact of rounding error
7 Bad determinism matrix and matrix conditions
Matrix norm
Number of matrix conditions
Evaluate error when the number of conditions of the matrix is known
8 Solving a system of linear equations using matrix analysis
9 Summary

27
Solve a system of square linear equations

Permutation matrix
The permutation matrix is obtained from the unit matrix I by permuting its rows.
With permutation matrix : P −1 = P T
Multiplication PX to permute rows of matrix X
Multiplication XP to permute matrix columns X

Example 10
 
0 1 0 0
 
0 0 0 1
P=
0

 0 1 0

1 0 0 0

28
Solve a system of square linear equations

Example 10 (continued)
 
0 1 0 0
 
0 0 0 1
P=
0

 0 1 0

1 0 0 0
Short form in Matlab
p = [2 4 3 1]

29
Solve a system of square linear equations
Matrix triangle
The matrix X ∈ Rn×n is upper triangular matrix if xij = 0 ∈ X ∀i > j means of the form
 
x11 x12 · · · x1n
 0 x22 · · · x2n 
 
X = .
 .. . . .. 
 .. . . . 


0 0 · · · xnn

When this matrix has major diagonal elements xii = 1 ∀i = 1, · · · , n is called unit upper
triangular matrix.
The determinant of an upper triangular matrix is non-zero if and only if all the elements
lie on the main diagonal are different from zero.
⇒ Similar definition we have lower triangle matrix and unit lower triangle matrix.
30
Solve a system of square linear equations

Matrix triangle
The system of equations with the matrix of triangular coefficients can be solved easily. Start
solving the last row equation to find the last unknown; then alternately substitute the above
equations to find the remaining unknowns.

Example 11 :
Solve a system of triangle equations on Ux = b
»x = zeros(n,1);
for k = n:-1:1
x(k) = b(k)/U(k,k);
i=(1:k-1)’;
b(i) = b(i) - x(k) * U(i,k);
end

31
Solve a system of square linear equations

Example 12 :
Solving a system of linear equations

3x1 + 4x2 + 5x3 = 7

2x2 − 3x3 = 8
5x3 = 11

% Matlab Programs
» U=[3,4,5;0.2,-3;0,0,5]; b = [7;8;11];n=3;x=zeros(n,1);
» for k=n:-1:1
» x(k) = b(k)/U(k,k);
» i=(1:k-1)’;
» b(i) = b(i) - x(k) * U(i,k);
» end;
32
1 What is a system of linear equations?
2 3-dimensional example
3 Permutation matrices and triangular matrices
4 LU analysis
5 Role of the pivot element
6 Impact of rounding error
7 Bad determinism matrix and matrix conditions
Matrix norm
Number of matrix conditions
Evaluate error when the number of conditions of the matrix is known
8 Solving a system of linear equations using matrix analysis
9 Summary

33
Solve a system of square linear equations

LU Analyze
The common algorithm used to solve a system of square linear equations has two stages
Forward elimination is the transformation of the square matrix into the upper triangular
form used to eliminate each unknown, with compatible factors and pivot elements
combined with rotation.
- consists of n − 1 steps
- at step k = 1, · · · , n − 1 multiply the equation k by the factor and then subtract the
remaining equations to de-anonymize the number xk .
- If the coefficients of xk are small then we should swap the equations.
Backward subtitution solves the last row equation to find the last solution, and then
reverses the upper rows to find the remaining unknowns. (see example 12)

34
Solve a system of square linear equations

LU analysis (continued)
Let Pk be the permutation matrices at steps k = 1, · · · , n − 1
Let Mk be the unit lower triangular matrices obtained by inserting the "addition" factors
used in the k step below the diagonal position of the k column of the unit matrix .
Let U be the final upper triangular matrix obtained at the end of the forward elimination
phase.
The elimination process is rewritten in matrix form as follows

U = Mn−1 Pn−1 · · · M1 P1 A

35
Solve a system of square linear equations

LU analysis (continued)
The equation can be rewritten as:

L1 L2 · · · Ln−1 U = Pn−1 · · · P1 A

where Lk is obtained from Mk by permutation and change sign of the factors below the
diagonal. So if we put

L = L1 L2 · · · Ln−1
P = Pn−1 · · · P2 P1

then we get the final formula

LU = PA

36
Solve a system of square linear equations
Example 12 :
 
10 −7 0
Going back to the first 3-dimensional example, we have A = −3 2 6 then the
 

5 −1 5
matrices determined in the forward elimination process are
   
1 0 0 1 0 0
P1 = 0 1 0 , M1 =  0.3 1 0
   

0 0 1 −0.5 0 1
   
1 0 0 1 0 0
P2 = 0 0 1 , M2 = 0 1 0
   

0 1 0 0 0.04 1
37
Solve a system of square linear equations

Example 12 (continued):
The matrices L1 ,L2 are respectively
   
1 0 0 1 0 0
L1 =  0.5 1 0 , L2 = 0 1 0
   

−0.3 0 1 0 −0.04 1

Note :
When calculating the elimination phase, we will calculate directly on the rows of the matrix,
not perform the matrix multiplication as above.

Analyze LU
The relation LU = PA is called LU analysis or triangular decomposition of the matrix A.
38
Solve a system of square linear equations
LU analysis
With the system of equations
Ax = b
where the matrix A is non-degenerate and PA = LU is the LU analysis of A, the system of
equations can be solved in two steps.
Forward elimination Solve the system

Ly = Pb

to find y , since L is a lower unit matrix, y can be found with a forward elimination (from
top to bottom).
Backward substitution Solve the system

Ux = y
39
1 What is a system of linear equations?
2 3-dimensional example
3 Permutation matrices and triangular matrices
4 LU analysis
5 Role of the pivot element
6 Impact of rounding error
7 Bad determinism matrix and matrix conditions
Matrix norm
Number of matrix conditions
Evaluate error when the number of conditions of the matrix is known
8 Solving a system of linear equations using matrix analysis
9 Summary

40
Role of the pivot element

pivot element
The elements that lie on the main diagonal of the matrix U.
The k-th pivot element is the coefficient of the solution xk in the k-th equation at the k
step of the elimination phase.
Both the forward elimination and backward substitution steps need to be divided by the
pivot element, so they cannot be zero.

Intuition :
The system of equations solves badly if the pivot element is close to zero.

41
Role of the pivot element

Example 13 :
Slight change in the second row of the above examples
    
10 −7 0 x1 7
−3 2.099 6 x2  = 3, 901
    

5 −1 5 x3 6
Thus, suppose all calculations are performed on a hypothetical computer equipped with
arithmetic operations with 5-digit floating point real numbers.
Coefficient x2 in row two changed from 2,000 to 2,099
Also the corresponding right side changed from 4,000 to 3,901
the goal is to keep the solution (0, −1, 1)T of the system of equations.

42
Role of the pivot element

Example 13 (continued) :
The first step of the elimination phase
    
10 −7 0 x1 7
 0 −0.001 6 x2  = 6.001
    

0 2.5 5 x3 2.5

continue to perform elimination even though the pivot element −0.001 is small compared to
other coefficients of the matrix without performing the rotation. So we
Multiply the second row equation by 2.5 × 103 and then add the third row.
On the right hand side of this equation, multiplying 6, 001 by 2.5 × 103 results in
1, 50025 × 104 rounded to 1, 5002 × 104

43
Role of the pivot element
Example 13 (continued) :
    
10 −7 0 x1 7
 0 −0.001 6  x2  =  6.001
    

0 0 1, 5005 × 10 4 x3 1, 5004 × 104

continue ...
The result on the right hand side of the second equation is rounded 1, 5002 × 104 is
added to the 2.5 which is the right hand side of the third equation and is rounded again.
So equation three becomes 1, 5005 × 104 x3 = 1, 5004 × 104 solving we have

1.5004 × 104
x3 = = 0.99993
1.5005 × 104
Obviously, with the exact value of the unknown x3 = 1, the value solved by this equation is
not worrisome.
44
Role of the pivot element
Example 13 (continued) :
    
10 −7 0 x1 7
 0 −0.001 6  x2  =  6.001
    

0 0 1, 5005 × 10 4 x3 1, 5004 × 104

continue ...
for unknown x2
−0.001x2 + 6 × (0.99993) = 6.001
1.5×10−3
Candlestick x2 = −1.0×10−3
= −1.5
Finally substitute the first equation to find the solution x1

10x1 − 7 × (−1.5) = 7

deduce x1 = −3.5
45
Role of the pivot element

Example 13 (continued) :
Thus, when not performing rotation, select the pivot element
    
10 −7 0 x1 7
−3 −0.001 6 x2  = 6.001
    

5 2.5 5 x3 2.5

instead of the solution (0, −1, 1)T we get the solution (−0.35, −1.5, 0.99993)T .

Why is this problem?

The error is because we choose the pivot element is too small. So we should choose the
pivot element with the largest absolute value at each k-th elimination step.

46
1 What is a system of linear equations?
2 3-dimensional example
3 Permutation matrices and triangular matrices
4 LU analysis
5 Role of the pivot element
6 Impact of rounding error
7 Bad determinism matrix and matrix conditions
Matrix norm
Number of matrix conditions
Evaluate error when the number of conditions of the matrix is known
8 Solving a system of linear equations using matrix analysis
9 Summary

47
Impact of rounding error

How to measure difference

Normally, when we get a solution x ∗ that is different from the correct solution x, we often use
two ways of measuring the difference:
Error : e = x − x ∗
Deviation : r = b − Ax ∗
Theoretically, if A is not degenerate, then these two quantities are equal to zero, but when
calculating in the computer these two quantities are not in sync.

48
Impact of rounding error

Example 14 :
Consider the system of equations

0.780x1 + 0.563x2 = 0.217

0.913x1 + 0.659x2 = 0.254

Gaussian elimination like the previous example, apply the rule of choosing the largest pivot
element, but all calculations are accurate to 3 digits after the decimal point.

49
Impact of rounding error

Example 14 (continued):
continue ....
Perform the rotation, so that 0.913 becomes the pivot element.

0.913x1 + 0.659x2 = 0.254

0.780x1 + 0.563x2 = 0.217

Calculate the coefficient 0.780/0.913 = 0.854

Multiply the factor 0.854 by the first equation and then subtract the second equation. We
have:

0.913x1 + 0.659x2 = 0.254

0.001x2 = 0.001

50
Impact of rounding error

Example 14 (continued):

0.913x1 + 0.659x2 = 0.254

0.001x2 = 0.001

continue ....
Hide x2 = 0.001/0.001 =1,000 (exactly)
Substitute for the above pt, x1 = (0.254 − 0.659x2 )/0.913 = −0.443
Finally we get the solution x ∗ = (−0.443, 1, 000)T

51
Impact of rounding error

Example 14 (continued):
Measuring the difference, it is clear that the true solution of the system x = (1, −1)T
Error : e = x − x ∗ = (1, 433, −2)T
Deviation :
!
0.217 − (0.780(−0.443) + 0.563(1, 000))
r = b − Ax ∗ =
0.254 − (0.913(−0.443) + 0.659(1, 000))
!
−0.000460
=
−0.000541

Obviously, while the deviation is acceptable when we round to three decimal places, the error
is even larger than the solution.

52
Impact of rounding error

Questions for rounding errors

Why is the Deviation so small?
Why is the error so large?
The determinant of the system 0.780 × 0.659 − 0.913 × 0.563 = 10−6 is not near the
cause of this phenomenon?

53
Impact of rounding error

Example 14 (continued):
Replacing the assumption of rounding with 3 decimals to rounding with 6 decimals after the
decimal point, we get a system of equations after Gauss elimination
! ! !
0.913000 0.659000 x1 0.254000
=
0.000000 0.000001 x2 −0.0000001
Notice, the value of the right hand side of the second equation has changed. In fact the
approximate solution is also the exact solution of the system

−0.00001
x1 = = −1.000000
0.00001
0.254 − 0.659x2
x2 = = 1.000000
0.913
54
Impact of rounding error

Explain why the deviation is small

The small deviation of the two equations due to the near degenerate matrix
det(A) = 10−6 leads to the two equations being almost linearly dependent.
So the solution pair (x1 , x2 ) satisfying the first equation also satisfies the second equation
⇒ If we know for sure that the determinant is zero, we don’t need to worry about the second
equation because every solution of the first system of equations satisfies the second
system of equations.

Important conclusion :
When we perform Gaussian elimination with the maximum pivot element on the column make
sure the deviation r = b − Ax ∗ is small.

55
Impact of rounding error

56
 Algorithm installation on Matlab

1 function [L , U ,p ]= lutx ( A )
2 [n , n ]= size ( A );
3 p =(1: n ) ’;
4 for k =1: n -1
5 [r , m ]= max ( abs ( A ( k :n , k )));
6 m = m +k -1;
7 if ( A (m , k )~=0)
8 if ( m ~= k )
9 A ([ k m ] ,:)= A ([ m k ] ,:);
0 p ([ k m ])= p ([ m k ]);
1 end
2 i = k +1: n ;
3 A (i , k )= A (i , k )/ A (k , k );
4 j = k +1: n ;
5 A (i , j )= A (i , j ) - A (i , k )* A (k , j );
6 end
7 end
8 L = tril (A , -1)+ eye (n , n );
9 U = triu ( A );
0 end
56
Exercise

Write a function bslashtx that implements MatLab’s (simplified) left division to solve a system
of linear equations.
function x=bslashtx(A,b)
n=size(A,1);
%Su use lutx(A);
...
% Downside
...
%The source
...

57
1 What is a system of linear equations?
2 3-dimensional example
3 Permutation matrices and triangular matrices
4 LU analysis
5 Role of the pivot element
6 Impact of rounding error
7 Bad determinism matrix and matrix conditions
Matrix norm
Number of matrix conditions
Evaluate error when the number of conditions of the matrix is known
8 Solving a system of linear equations using matrix analysis
9 Summary

58
Bad determinism matrix and matrix conditions

Coefficients are rarely known exactly

because of
For systems of equations that appear in the application, the coefficients that are normally
attributed to the empirical value should be associated with the observation error.
Many other systems of equations have coefficients calculated by formulas and therefore
they are known exactly to rounding errors when calculated according to the given formula.
Even for systems of equations that are stored exactly in the computer, errors are
inevitable (Review the representations of integers, signed integers, and floating-point real
numbers in the textbook). h General Informatics)
So the question is:
If there is an error in the representation of coefficients of a system of linear equations, how
does that affect the solution? Or when solving Ax = b how can the sensitivity of x be
measured when there is a change in A, b ?

59
Bad determinism and matrix conditions

There are a few comments after

If A is degenerate then for some b x either has no solution or infinitely many solutions. In
the case where A has a small determinant, a small change in A and b can lead to a large
change in the solution.
Think about the size of the pivot elements and the concept of near degeneracy. Because if
the arithmetic operations are performed exactly, all the pivot elements are non-zero if and
only if the matrix is non-degenerate. From that, the following statement is drawn: ’If the
pivot elements are small, the matrix is near degenerate’ the opposite is not true, or in
other words, there is a near degenerate matrix where the pivot elements are not small.

60
Bad determinism and matrix conditions

Vector norms
Definition: The function v : Rn 7→ R is said to be a vector norm over Rn if and only if
1 v (x) ≥ 0 ∀x ∈ Rn and v (x) = 0 if and only if x = 0
2 v (αx) = |α|v (x) ∀x ∈ Rn , ∀α ∈ R
3 v (x + y ) ≤ v (x) + v (y ) ∀x, y ∈ Rn this is the triangle inequality.
Normally v (x) is denoted by ||x||

61
Bad determinism and matrix conditions

Vector norm (continued)

Some commonly used standards
qP
n 2
||x||2 = i=1 xi is (l2 ) or Euclidean
||x||1 = ni=1 |xi | is (l1 )
P

||x||∞ = max1≤i≤n |xi | is (l∞ )

||x||p = ( ni=1 |xi |p )1/p is (lp )
P

Matlab
norm(x,p) for lp
and for p = 2 the function is simpler than norm(x)

62
Bad determinism and matrix conditions

Matrix norm
Definition: Function ||.|| : Rn×n 7→ R is said to be matrix norm if

||Ax||
||A|| = max ||Ax|| = max
||x||=1,x∈Rn ||x||̸=0,x∈Rn ||x||

where ||Ax|| is the norm of the vector Ax. Of course, we have the inequality ||Ax|| ≤ ||A|| ||x||

63
Bad determinism and matrix conditions

Matrix norm (continued)

The properties of the matrix norm
1 ||A|| ≥ 0;||A|| = 0 if and only if A = 0.
2 ||αA|| = ||α||||A||, α ∈ R
3 ||A + B|| ≤ ||A|| + ||B||
4 ||AB|| ≤ ||A|| × ||B||
5 ||Ax|| ≤ ||A||||x||

64
Bad determinism and matrix conditions

Matrix norm (continued)

The vector norms generate the corresponding matrix norms
Euclidean norm : ||A||2 = max||x||2 =1 ||Ax||2
Pn
max "total rows" : ||A||∞ = max||x||∞ =1 ||Ax||∞ = max1≤i leqn |aij |
Pn j=1
max "total column" : ||A||1 = max||x||1 =1 ||Ax||1 = max1≤j≤n i=1 |aij |

Pn 1/2
Frobenius Standard : ||A||F = Tr (AT A)1/2 = a 2
i,j=1 ij
In Matlab
norm(A,p) where p = 1, 2, inf

65
Bad determinism and matrix conditions

Number of matrix conditions

Definition : The condition number cond(A), usually denoted by κp (A), of a square matrix A
computed for a given matrix p standard is a number

cond(A) = ||A|| · ||A−1 ||

where, we convention cond(A) = ∞ when A is degenerate. Because of,

||Ax||
maxx̸=0 ||x||
−1
||A|| · ||A || = ||Ax||
minx̸=0 ||x||

so the number of conditions that measure the ratio of the maximum expansion to the
maximum contraction that the matrix can act on for a vector is non-zero.

66
Bad determinism and matrix conditions

Number of conditions of the matrix (continued)

The number of conditions indicates how close the matrix is to degeneracy: the larger the
condition number, the closer near degenerate (the corresponding system of equations is
poorly defined), whereas the closer the matrix to 1 the number of conditions. far from near
degenerate.

Note :
The matrix determinant is not a good characterization for approximation. Although when
det(A) = 0, the matrix is degenerate, but the magnitude or small of the determinant is not
contains information about whether the matrix is near degenerate or not.
For example, for the matrix det(αIn ) = αn may be a very small number when |α| < 1 but the
matrix αIn has good conditions with cond(αIn ) = 1. Where In is the n dimensional unit
matrix.

67
Bad determinism and matrix conditions

Some conditional numerical properties of matrices

1 For all matrices A : cond(A) ≥ 1
2 For all unit matrices I : cond(I) = 1
3 For any permutation matrix P : cond(P) = 1
4 For all matrices A and non-zero reals α : cond(αA) = cond(A)
max{|di |}
5 For any diagonal matrix D = diag (di ) : cond(D) = min{|di |}
6 The number of conditions is important in evaluating the accuracy of the solution of a
system of linear equations.

68
Bad determinism and matrix conditions

Matlab with number of conditions

cond(A,p) to calculate κp (A) with p = 1, 2, inf .
cond(A) or cond(A,2) calculates κ2 (A). Use svd(A). Should be used with small matrix.
cond(A,1) calculates κ1 (A). Use the function inv(A). Requires less computation time than
cond(A,2).
cond(A,inf) calculates κ∞ (A). Use the function inv(A). Requires less computation time than
cond(A,1).
condest(A) to evaluate κ1 (A). Using the function lu(A) and the Higham-Tisseur algorithm.
Recommended for large matrices.
rcond(A) to evaluate 1/κ1 (A)

69
Bad determinism and matrix conditions

Evaluate the error when knowing the number of conditions of the matrix
Let x be the exact solution of Ax = b, and x ∗ be the solution of the system Ax ∗ = b + ∆b
(note we only consider b to be additive noise. ). Put ∆x = x ∗ − x, we have
b + ∆b = Ax ∗ = A(x + ∆x) = Ax + A∆x since Ax = b substitute in, then ∆x = A−1 ∆b.

b = Ax ⇒ ||b|| ≤ ||A||||x|| (13)

−1 −1
∆x = A ∆b ⇒ ||∆x|| ≤ ||A ||||∆b|| (14)

Multiply the two inequalities (13) (14) and use the definition cond(A) = ||A||||A−1 || we have
evaluation
||∆x|| ||∆b||
≤ cond(A)
||x|| ||b||

70
Bad determinism and matrix conditions

Evaluate error when knowing the number of conditions of the matrix (continued)
continue...
||∆x|| ||∆b||
≤ cond(A)
||x|| ||b||
So the number of conditions allows us to determine the relative error variation in the solution
||∆x|| ||∆b||
||x|| given the relative change in the right-hand side. ||b||
When cond(A) is large or the system is near degenerate, the relative transformation of
the right side will ’force’ the corresponding error change in the solution.
Conversely, when cond(A) approaches 1 or the system is well-conditioned, the equivalent
transformation of the right hand side and the solution are the same.

71
Bad determinism and matrix conditions

Well-conditioned system Ill-conditioned system

60 10

40 5

20 0

0 −5

−20 −10
−0.5 0 0.5 1 1.5 2 2.5 3 3.5 −200 −100 0 100 200 300 400

72
Bad determinism and matrix conditions

Evaluate the error when knowing the number of conditions of the matrix (Conclusion)
If the input data is represented approximately to computer accuracy, then the relative error
estimate of the calculated solution is given by the formula:

||x ∗ − x||
≈ cond(A)ϵM
||x||

The calculated solution will lose an interval log10 (cond(A)) in decimal places in the relative
error of the data precision.

Conclusion
The system of linear equations Ax = b is ill-conditioned if cond(A) is large, then a small
change in the data can lead to a large change in the solution.

73
Bad determinism and matrix conditions

Example 15 :
Consider the system of equations

0.789x1 + 0.563x2 = 0.127

0.913x1 + 0.659x2 = 0.254

Results when using Matlab

» A=[0.789 0.563;0.913 0.659];
» fprintf(’cond(A)=%d ; det(A)=%d ’,cond(A),det(A))
» cond(A) = 2.193219e+006 ; det(A)=1.000000e-006

74
Bad determinism and matrix conditions

Example 16 :
Consider the system of equations
! ! !
4.1 2.8 x1 4.1
=
9.7 6.6 x2 9.7

This is an ill-conditioned system because cond(A, 1) = 2494.4 and the exact solution of the
system is x = (1, 0)T . If we substitute the right hand side b + ∆b = (4.11, 9.70)T then the
solution of the system will be x ∗ = (0.34, 0.97)T .
In Matlab we have
» A = [4.1 2.8; 9.7 6.6]; b = [4.1 ; 9.7]; b1=[4.11 ; 9.7];
» x = (A \ b)’, x1 = (A \ b1)’
x=10
x1 = 0.3400 0.9700
75
Bad determinism and matrix conditions

76
1 What is a system of linear equations?
2 3-dimensional example
3 Permutation matrices and triangular matrices
4 LU analysis
5 Role of the pivot element
6 Impact of rounding error
7 Bad determinism matrix and matrix conditions
Matrix norm
Number of matrix conditions
Evaluate error when the number of conditions of the matrix is known
8 Solving a system of linear equations using matrix analysis
9 Summary

77
Solve a system of linear equations using matrix analysis

Electrical network analysis

Solving a system of linear equations has important applications in power network analysis.
Example for the following electrical network

R1 i1 R3 i2 R5 i3

V1 R2 R4 V2

78
Solving a system of linear equations by matrix analysis (continued)

Electrical network analysis (continued)

According to Kirchoff’s law, the voltage across the loops must be zero. We have the following
system of linear equations

−V1 + R1 i1 + R2 (i1 − i2 ) = 0
R2 (i2 − i1 ) + R3 i2 + R4 (i2 − i3 ) = 0
R4 (i3 − i2 ) + R5 i3 + V2 = 0

converted into     
R1 + R2 −R2 0 i1 V1
 −R2 R2 + R3 + R4 −R4  i2  =  0 
    

0 −R4 R4 + R5 i3 −V2

79
System of Linear Equations

Summary
Features of a system of linear equations (linear algebra)
Recalling permutation matrices and triangular matrices
LU analysis is used to solve system of linear equations
The role of cylinders in LU analysis can cause errors in the results
Rounding error effect
Determines the condition of a matrix in a linear equation that causes the resulting error
An example of applying a system of linear equations

80
Solve a system of linear equations using matrix analysis

More homework
We can use many methods besides LU analysis to solve the system of equations
Cholesky Analysis
▶ The concept of a positive semi-deterministic matrix
▶ If A is a positive definite matrix then there exists a positive lower triangular matrix L such
that A = LLT
▶ Eliminate forward Ly = b, backward substitution LT x = y
Decay QR
▶ Orthogonal matrix concept
▶ Decay QR : if A ∈ Rm×n with m ≥ n and has rank n then there exists an orthogonal matrix
Q ∈ Rm×n and a triangular matrix on R ∈ Rn×n with positive elements on the diagonal such
that A = QR
▶ Solve least squares problem
min{||Ax − b||2 |x ∈ R}
so the solution x is the stopping point when minimizing the above problem.
81