Systems of Linear Equations

Systems of Linear Equations

• Vectors
• Matrices
• Special matrices
• Vector – vector scalar product
• Matrix multiplied by vector
• Linear system of equations
• Matrix multiplication
Systems of Linear Equations

– Is there a solution?
– If there is a solution, is there only one?
– How can we express the solution with
the inverse of the coefficient matrix?
– Is it usually a good idea to get the
solution via the inverse?
• The determinant is a number that can be calculated
knowing the elements of the (cofficient) matrix.
• The matrix inverse doesn’t exist if the determinant
is zero.
• A condition for the inverse to exist is that the
determinant is not zero.
• Determinant zero; coefficient matrix is singular;
solution does not exist; system is contradictory OR
undetermined; matrix rank is less than number of
equations (or just division by zero, overflow etc.):
all mean the same…
 Solution of Systems of
Linear Equations

Ax = b

• solution
X = A-1b (A-1 is the inverse matrix)
or, Ix = A-1b (I is identity matrix)
Types of matrix problems
Example 1 - Unique solution.
Consider the system 4
Equation 1
2x + 3y = 8 2 Equation 2

5x - 6y = 30 0
0 2 4 6 8 10

Geometrically this -2

corresponds to -4

two lines -6

crossing in one
point.
Example 2 - No unique solution.

Consider the system

4
2x + 3y = 8 Equation 1
Equation 2
2

4x + 6y = 16
0

Geometrically this 0 2 4 6 8 10

-2
corresponds to two
coincident lines. -4

Infinite number of -6

solutions.
 Example 3 - No solution.

Consider the system

6

2x + 3y = 8 4
Equation 1
Equation 2

4x + 6y = 30 2

Geometrically this 0
0 2 4 6 8 10

corresponds to -2

two parallel lines -4

never crossing. -6
Example 4 – Ill-conditioned matrix

Consider the system

4
2x + 3y = 8 Equation 1

2 Equation 2
4x + 7y = 15
Geometrically this 0
0 2 4 6 8 10
corresponds to two -2

almost parallel -4
lines barely
crossing. Sensitive -6

to round-off error.
Existence and Uniqueness
Statement with Rank
• The rank of a matrix tells us how many
independent rows (or how many independent
columns) the coefficient matrix has -- it may be
less than the total number of equations.
• If rank is less than the number of equations,
two cases can happen:
– Consistent (if rank of coeff matrix = rank of
augmented matrix) Infinite number of
solutions
– Non-Consistent (if rank of coeff matrix < rank
of augmented matrix) No solution
Definition: Augmented matrix
a11 . . . b1

. . . A . .

. . . . .

. . . ann bn
 Example 1
Consider the system 2 3 8
2x + 3y = 8
4x + 6y = 16 4 6 16

Thus the determinant is 2*6 - 3*4 = 0.

And indeed the second equation is 2 times the first.
The solution can be written as x = 4 - 3/2 y for any y.
Geometrically this corresponds to two coincident
lines.
In terms of rank: the coefficient matrix has rank 1,
the augmented matrix has rank 1
 Example 2
2 3 8
Consider the system
2x + 3y = 8
4 6 17
4x + 6y = 17

Thus the determinant is 2*6 - 3*4 = 0.

There is no solution.

Geometrically this corresponds to two

parallel lines never crossing.
In terms of rank: the coefficient matrix has
rank 1, the augmented matrix has rank 2
Direct Methods
• Gauss-Jordan elimination
• Gaussian elimination
• LU decomposition / factorization
• Thomas algorithm

Iterative Methods
• Gauss-Seidel
 Gaussian-Jordan
Step 1. Forward elimination:
• Choose a pivot element on the main diagonal (start at
top left corner)
• Eliminate (set to zero) all elements above and below the
main diagonal
• Move to the next column
• Repeat until matrix is an Identity matrix
 Gauss-Jordan Elimination
• Reduce an augmented matrix to the
identity to solve a system of linear
equations.
• Example:
 2  1 1   x1  4
  
Solve 1 1 1   x 2   3  
3  1  1  x 3  1 
Gauss-Jordan Elimination
Augmented matrix :
 2  1 1 4
1 1 1 3 
 
3  1  1 1 
Divide row 1 by 2 :
1  1 / 2 1 / 2 2
1 1 1 3 

3  1  1 1
Gauss-Jordan Elimination
Set row 2  row 2  row 1
and row 3  row 3  3row 1 :

1  1 / 2 1 / 2 2
0 3 / 2 1 / 2 1 
 
0 1 / 2  5 / 2  5
Gauss-Jordan Elimination
Divide row 2 by 3 / 2 :

1  1 / 2 1 / 2 2 
0 1 1 / 3 2 / 3
 
0 1 / 2  5 / 2  5 
Gauss-Jordan Elimination
Set row 1  row 1  1 / 2 row 2
Set row 3  row 3  1 / 2row 2 :

1 0 2 / 3 7/3 
0 1 1 / 3 2 / 3 
 
0 0  8 / 3  16 / 3
Gauss-Jordan Elimination
Divide row 3 by  8 / 3 :

1 0 2 / 3 7 / 3
0 1 1 / 3 2 / 3
 
0 0 1 2 
Gauss-Jordan Elimination
Set row 2  row 2  1 / 3 row 3
Set row 1  row 1  2 / 3 row 3 :
1 0 0 1 
0 1 0 0 
 
0 0 1 2
1 
Solution : 
x  0 
2
 Summary
• Gauss-Jordan process
– loop over rows (i)
– scale row such that diagonal is 1.0
– subtract a multiple of row i from every other
row in order to fill the rest of the column
with zeroes.
– move to next row
– Final solution appears at the end of the the
augmented matrix.
Sub GaussJordan(A() As Double) 'Augmented matrix
Dim PivElt As Double, TarElt As Double
Dim n As Integer 'number of equations
Dim PivRow As Integer, TarRow As Integer 'pivot row; target row
Dim i As Integer, j As Integer
n = UBound(A, 1)
For PivRow = 1 To n 'process every row
PivElt = A(PivRow, PivRow) 'choose pivot element
If PivElt = 0 Then
MsgBox ("Zero pivot element encountered")
Stop
End If
For j = 1 To n + 1
A(PivRow, j) = A(PivRow, j) / PivElt 'divide whole row
Next
For TarRow = 1 To n 'now replace all other rows
If Not (TarRow = PivRow) Then
TarElt = A(TarRow, PivRow)
For j = 1 To n + 1
A(TarRow, j) = A(TarRow, j) - A(PivRow, j) * TarElt
Next j
End If
Next TarRow
Next PivRow
End Sub
 Gaussian Elimination
Pivoting:
• If the main diagonal pivot element is small or
zero, interchange rows (from below) to
maximize the pivot element

The determinant:
• After forward elimination, the determinant is the
product of all the main diagonal elements of the upper
triangular matrix. det(A) = u11 u22 u33 unn
● ● ●●● ●
Gaus. Elim. – forward elimination

• Eliminate elements in column 1

• Example:

 2  1 1   x1  4
  
Solve 1 1 1   x 2   3  
3  1  1  x 3  1 
Gaus. Elim. – forward elimination

• Eliminate a21 by calculating a ratio:

R = a21/a11
• Set row 2 = row 2 – row 1

2  1 1   x1  4
    
Result : 0 3 / 2 1 / 2  x2   1  
3  1  1   x 3  1 
Gaus. Elim. – forward elimination

• Continue until all elements below

main diagonal = 0
• Matrix is now upper triangular

2  1 1   x1   4 
    
Result : 0 3 / 2 1 / 2   x2    1  
0 0  8 / 3  x 3   16 / 3
Gaus. Elim. – backward substitution
Solve the upper triangular matrix by solving the last equation first :
 8 / 3 x3  16 / 3
x3  2
then solve next to last equation :
3 / 2 x2  1  1 / 2 x3  1  1 / 2  2 
x2  0
and continue working "backwards :
2 x1  4  x2  x3  4  (0)  (2)
x1  1
1 
x  0
2
 Gaussian elimination
Step 1. Forward elimination:
• Choose a pivot element on the main diagonal (start at
top left corner)
• Eliminate (set to zero) all elements below the main
diagonal
• Move to the next column
• Repeat until matrix is upper triangular
Step 2. Backward substitution:
• Solve the last row explicitly for the last unknown
• Solve explicitly for the next row up
• Repeat until all the rows are solved
 LU Factorization
• LU Factorization is a form of
Gaussian Elimination
• The A matrix (in the problem Ax=b) is
factored into the product of two
matrices L and U (A=LU)
• L is a lower triangular matrix and U is
an upper triangular matrix
 LU Factorization
 1 0 0  U11 U12 U13   a 11 a 12 a 13 
L 1 0  0 U U   a a a 
 21  22 23   21 22 23 

L 31 L 32 1  0 0 U 33  a 31 a 32 a 33 

• How do we find the entries in L and U?
– there are nine unknowns (3 L’s and 6
U’s)
– there are nine equations (each setting
a row of L times a column of U equal
to one of the A values)
 LU Factorization
 1 0 0  U11 U12 U13   a 11 a 12 a 13 
L 1  
0  0 U 22  
U 23   a 21 a 22 a 23 
 21
 L 31 L 32 1  0 0 U 33  a 31 a 32 a 33 

1 U11  a 11 so U11  a 11
a 21 a 21
L 21  U11  a 21 so L 21  
U11 a 11
a 31 a 31
L 31  U11  a 31 so L 31  
U11 a 11
etc.
 Pivoting
• What would happen if a11 were zero?
– We would get a divide by zero error when
building L.

• Algorithms for LU factorization check for

this and swap the order of the rows in A
(pivoting) to avoid this problem.
 LU Factorization
• Once L and U have been determined we have:
LUx = b

• How do we solve for x?

– Set Ux=L-1b = y
– Therefore Ly = b
– Solve Ly=b for y
– Solve Ux = y for x
 Forward Substitution
• We solve Ly=b using a process
called forward substitution.
 1 0 0  y1   b1 
L 1 0  y 2   b 2 
 21
L 31 L 32 1  y3   b 3 

1 y1  b1
 y1  b1
L 21  y1  1 y 2  b 2
 y 2  b 2  L 21  y1
L 31  y1  L 32  y 2  1 y 3  b 3
 y 3  b 3  L 31  y1  L 32  y 2
 Back Substitution
• We solve Ux=y using a process
called back substitution.
 U11 U12 U13   x1   y1 
 0 U 22    
U 23   x 2    y 2  

 0 0 U 33   x 3   y 3 

U 33  x 3  y 3
y3
 x3 
U 33
 Back Substitution
U 22  x 2  U 23  x 3  y 2
y 2  U 23  x 3
 x2 
U 22
U11  x1  U12  x 2  U13  x 3  y1
y1  U12  x 2  U13  x 3
 x1 
U11
LU Decomposition and
Backsubstitution

Basic:

Call LUDecompose(A, Indx)

Call LUSubstitute(A, Indx, b, x)

Sub LUDecompose(A() As Double, Indx() As Integer)
' input a(n,n)
' output a(n,n) and indx(n)
Const Tiny As Double = 1E-16
Dim n As Integer
n = UBound(A, 1)
Dim i As Integer, j As Integer, k As Integer, m As
Integer
Dim dum As Double
For k = 1 To n - 1
m = k
For i = k + 1 To n
If (Abs(A(i, k)) > Abs(A(m, k))) Then m = i
Next i
Indx(k) = m
dum = A(m, k)
If m > k Then
A(m, k) = A(k, k)
A(k, k) = dum
End If
If Abs(dum) > Tiny Then
dum = 1 / dum
Else
MsgBox ("Singular coefficient matrix")
Stop
End If
Next k
Sub LUSubstitute(A() As Double, Indx() As Integer, _
b() As Double, x() As Double)
' input A(n,n) containing the LU decomposition
' input Indx(n) index vector
' input b(n) right hand side
' output x(n) solution
Dim n As Integer
n = UBound(A, 1)
Dim i As Integer, j As Integer, k As Integer
Dim dum As Double
For i = 1 To n
……

……
End sub
 Direct vs Iterative
• Gauss elimination, Gauss-Jordan, LU-
decomposition are DIRECT methods
• We can predict the number of
multiplications and additions if the number
of equations (n) is known
• There are other direct methods for special
matrices. One of them is important for us: It
is the Thomas algorithm for Tridiagonal
equations.
 Tri-diagonal matrix
• Occurs often in reservoir simulation

• Thomas algorithm is efficient

• Uses LU factorization
Tri-diagonal matrix
X1 d1
. .
. .
. .
. .
. .
. .
. .
. .
. .
. .
. .
. .
. .

xn dn
Thomas algorithm – save a,b,c
vectors, not aii elements
X1 d1
. .
. .
. .
. .
. .
.
ai .
.
.
bi
ci .
.
.
.
. .
. .
. .
. .

xn dn
 Thomas algorithm
Sub Thomas (a() As Double, b() As Double, c() As Double, d() As Double, x() As
Double)
'Tridiagonal system of equations
Dim n As Integer, i As Integer
n = UBound(b)
ReDim w(n) As Double, g(n) As Double
w(1) = b(1)
g(1) = d(1) / w(1)
For i = 2 To n
w(i) = b(i) - a(i) * c(i - 1) / w(i - 1)
g(i) = (d(i) - a(i) * g(i - 1)) / w(i)
Next i
x(n) = g(n)
For i = n - 1 To 1 Step -1
x(i) = g(i) - c(i) * x(i + 1) / w(i)
Next i
End Sub
 Direct vs Iterative
• Iterative methods are generalization of
the direct iteration we learned for one
variable x=g(x) equations.
• Iterative methods are used for very
large but sparse systems (reservoir
simulation).
• You can not predict the number of
operations for an indirect method and
you need a starting guess and a
convergence criterion.
• The method we are going to learn is the
Gauss-Seidel method.
 Iterative methods
• Gauss Seidel
 Gauss-Seidel
• Suppose we are trying to solve a
system of three equations:
3x 1  2 x 2  x 3  8
x1  4 x 2  2 x 3  11
x1  x 2  2 x 3  1
 Gauss-Seidel
• These equations can be rewritten as:

8  2x 2  x 3
x1 
3
11  x1  2x 3
x2 
4
1  x1  x 2
x3 
2
 Gauss-Seidel Iteration
• Gauss-Seidel starts from an initial
guess of the x’s and updates that
guess in an attempt to converge to a
solution: 8  2x k  x k
x1k 1  2 3
3
k 1
11  x  2 x k
x k2 1  1 3

4
k 1 k 1
1  x  x
x 3k 1  1 2
2
Diagonally Dominant Matrices
• Diagonally dominant matrices:
The absolute value of the diagonal element is
greater than the sum of the absolute values
of all the other elements in a given row.
– Identity matrix is diagonally dominant
– Diagonally Dominant?

YES NO
 4 1 2   4 1 2 
 2 5  2  2 3  2
   
 2 3 10   2 3 10 
 Diagonal Dominance

• Gauss-Seidel will only work if/only if

the matrix is diagonally dominant.