Numerical Technique (MA-201)

Lecture Notes-10-13

by
Prof. Santwana Mukhopadhyay

Department of Mathematical Sciences

IIT(BHU), Varanasi

Prof. S. Mukhopadhyay (Department of Mathematical

Numerical
SciencesTechnique
IIT(BHU),(MA-201)
Varanasi) 1 / 34
 Contents

1 System of Equations

2 Direct Methods

Prof. S. Mukhopadhyay (Department of Mathematical

Numerical
SciencesTechnique
IIT(BHU),(MA-201)
Varanasi) 2 / 34
 System of equations

Definition:
Consider a system of m linear equations in n (unknown) variables:

a11 x1 + a12 x2 + .... + a1n xn = b1

a21 x1 + a22 x2 + .... + a2n xn = b2
..................................................
..................................................
am1 x1 + am2 x2 + .... + amn xn = bm (1.1)

where aij and bj are real or complex numbers.

Prof. S. Mukhopadhyay (Department of Mathematical

Numerical
SciencesTechnique
IIT(BHU),(MA-201)
Varanasi) 3 / 34
 System of equations contd...

Write
     
a11 a12 ..... a1n b1 x1
 a21 a22 ..... a2n   b2   x2 
A= , b =  , x = 
     
... ... ... ... ... ...

     
am1 am2 ... amn bm xn
Then, A is called the coefficient matrix of the system (1.1). We also
write A = (aij ).
In matrix form, the system (1.1) is written as
Ax = b (1.2)
If b = 0, then the system (1.2) would be called a homogeneous
system. So,
Ax = 0 (1.3)
is a homogeneous system of linear equation.
Then, x = 0 is a solution of the homogeneous system (1.3), to be
called the trivial solution.
Prof. S. Mukhopadhyay (Department of Mathematical
Numerical
SciencesTechnique
IIT(BHU),(MA-201)
Varanasi) 4 / 34
 System of equations contd...

Corresponding to system (1.1), define the augmented matrix

 
a11 a12 ..... a1n b1
 a21 a22 ..... a2n b2 
[A|b] =  (1.4)
 
... ... ... ... ...

 
am1 am2 ... amn bm

In deed, the system (1.1) and the augmented matrix (1.4) has the
same information/data. The row operations performed on system
(1.1), can be performed on the augmented matrix (1.4), in stead.

Prof. S. Mukhopadhyay (Department of Mathematical

Numerical
SciencesTechnique
IIT(BHU),(MA-201)
Varanasi) 5 / 34
 System of equations contd...

There are three possibilities for the solution of system (1.1):

1 The system (1.1), may not have any solution.

2 The system (1.1), may have infinitely many solution.
3 The system (1.1), may have a unique solution. For this possibility, we
need at least n equations.

Prof. S. Mukhopadhyay (Department of Mathematical

Numerical
SciencesTechnique
IIT(BHU),(MA-201)
Varanasi) 6 / 34
 System of equations contd...

We focuses on the case when m = n: the number of equations is same as

number of unknown x1 , ..., xn .
When n = m, then the coefficient matrix A of (1.1) is a square matrix
of size n × n.
Recall, a square matrix A is invertible ⇐⇒ |A| =
6 0.
If |A| =
6 0, then the unique solution of system (1.2) Ax = b is

x = A−1 b (1.5)

Prof. S. Mukhopadhyay (Department of Mathematical

Numerical
SciencesTechnique
IIT(BHU),(MA-201)
Varanasi) 7 / 34
 System of equations contd...

Eigenvalues and Eigenvectors:

Suppose A is a square matrix of size n × n.
A scalar λ ∈ C is said to be an Eigenvalue of A , if |A − I| = 0.
The following are equivalent:
λ ∈ C is an Eigenvalue of A
|A−λI| = 0
The system (A − I)x = 0 has nontrivial solutions.
There are non-zero vectors x such that Ax = x .
Accordingly, a vector x 6= 0 is said to be an eigenvector, for an eigenvalue λ of A,
if Ax = λx .

Prof. S. Mukhopadhyay (Department of Mathematical

Numerical
SciencesTechnique
IIT(BHU),(MA-201)
Varanasi) 8 / 34
 System of equations contd...

Spectral radius: The spectral radius of a square matrix A is the

largest absolute value of its eigenvalues. It is denoted by ρ(A), i.e.,

ρ(A) = max (|λ1 | , |λ2 | , ..., |λn |)

where λ1 , λ2 , ..., λn are the eigenvalues of A.

Diagonally dominant matrix: A matrix A is said to be diagonally

dominant if it satisfies the inequality
n
|aij | < |aii | , i = 1, 2, ..., n
X

j=1,j6=i

Prof. S. Mukhopadhyay (Department of Mathematical

Numerical
SciencesTechnique
IIT(BHU),(MA-201)
Varanasi) 9 / 34
 System of equations contd...

Types of matrices:
Let A = (aij ) be a n × n real matrix. Then
A is nonsingular if |A| =
6 0.
A is singular if |A| = 0.
A is symmetric if A = AT .
A is skew symmetric if A = −AT .
A is null if aij = 0 for all i and j.
A is diagonal if aij = 0 for all i 6= j.

Prof. S. Mukhopadhyay (Department of Mathematical

Numerical
SciencesTechnique
IIT(BHU),(MA-201)
Varanasi) 10 / 34
 System of equations contd...

Types of matrices contd...

A is identity (unit) if aij = 0, i 6= j, aii = 1 for all i.
A is lower triangular if aij = 0, j > i.
A is upper triangular if aij = 0, i > j.
A is lower triangular if aij = 0, j > i.
A is orthogonal if A−1 = AT .

Prof. S. Mukhopadhyay (Department of Mathematical

Numerical
SciencesTechnique
IIT(BHU),(MA-201)
Varanasi) 11 / 34
 System of equations contd...

Types of matrices contd...

Let A = (aij ) be a n × n complex matrix. Then
 T
A is Hermitian if A = A where A is the complex conjugate of . It
is denoted by A∗ or AH .
 T
A is unitary if A−1 = A .
A is normal if AA∗ = A∗ A.

Prof. S. Mukhopadhyay (Department of Mathematical

Numerical
SciencesTechnique
IIT(BHU),(MA-201)
Varanasi) 12 / 34
 System of equations contd...

Norm of a vector:
Let x = (x1 , x2 , ..., xn ) be a vector. Then
Absolute norm (l1 norm)
n
kx k1 =
X
|xi |
i=1

Euclidean norm !1/2

n
2
kx k2 =
X
|xi |
i=1

Maximum norm (l∞ norm)

kx k∞ = max |xi |
1≤i≤n

Prof. S. Mukhopadhyay (Department of Mathematical

Numerical
SciencesTechnique
IIT(BHU),(MA-201)
Varanasi) 13 / 34
 System of equations contd...

Norm of a matrix:
Let A be n × n matrix. Then
Frobenius or Euclidean norm
n
!1/2
X
F (A) = |aij | 2

i,j=1

Maximum norm X
kAk∞ = max |aik |
i
k

Hilbert norm or spectral norm √

kAk2 = λ
where λ = ρ(A∗ A). If A is Hermitian or real and symmetric, then
λ = ρ(A2 ) = [ρ(A)]2 and ||A||2 = ρ(A).

Prof. S. Mukhopadhyay (Department of Mathematical

Numerical
SciencesTechnique
IIT(BHU),(MA-201)
Varanasi) 14 / 34
 System of equations contd...

Condition number:
The condition number of a matrix A is defined and denoted by

cond(A) = K (A) = kAkkA−1 k

where ||.|| is any suitable norm.

Note: No eigenvalue of a matrix A exceeds the norm of a matrix, i.e.

ρ(A) ≤ kAk

Prof. S. Mukhopadhyay (Department of Mathematical

Numerical
SciencesTechnique
IIT(BHU),(MA-201)
Varanasi) 15 / 34
 Elementary row operations
Row operations, Basically elementary row operations, are at the heart of
most numerical methods.
Consider any one of the equations in a set of linear equations such as

ai,1 x1 + ai,2 x2 + . . . + ai,j xj + . . . + ai,n xn = b

This equation may be written as Ax=b

If we multiply or divide the coefficients and the right hand term by a
non-zero number the equation remains unchanged. Multiplication or
division of a row with a non-zero number as above is an elementary row
operation.
• The solution of a system of linear equations is independent of the order
in which the equations are arranged. We may therefore interchange any
two rows without affecting the solution, giving thus another elementary
row operation.
• We may add or subtract two rows after multiplying each by a chosen
number. This constitutes another elementary row operation.
Prof. S. Mukhopadhyay (Department of Mathematical
Numerical
SciencesTechnique
IIT(BHU),(MA-201)
Varanasi) 16 / 34
 Direct methods

Direct Methods:
Gauss elimination method
Gauss-Jordan method
Gauss elimination method:
Let us introduce the Gaussian elimination method for n = 3. The method for a general
n × n system is similar.
a11 x1 + a12 x2 + a13 x3 = b1 (2.1)
a21 x1 + a22 x2 + a23 x3 = b2 (2.2)
a31 x1 + a32 x2 + a33 x3 = b3 (2.3)
The solution of the above system of equations can be obtained by Gauss elimination
methd in the following steps:

Prof. S. Mukhopadhyay (Department of Mathematical

Numerical
SciencesTechnique
IIT(BHU),(MA-201)
Varanasi) 17 / 34
 Gauss elimination method contd...

Step 1:

Assume that a11 6= 0 (otherwise interchange the row for which the coefficient of x1 is
non-zero). Let us eliminate x1 from (2.2) and (2.3). For this define
a21 a31
m21 =
, m31 =
a11 a11
Multiply (2.1) with m21 and subtract with (2.2), and multiply (2.1) with m31 and
subtract with (2.3) to give

a11 x1 + a12 x2 + a13 x3 = b1 (2.4)

(2) (2) (2)
a22 x2 + a23 x3 = b2 (2.5)
(2) (2) (2)
a32 x2 + a33 x3 = b3 (2.6)
(2)
The coefficients aij are defined by
(2)
aij = aij − mi1 a1j , i, j = 2, 3
(2)
bi = bi − mi1 b1 , i = 2, 3

Prof. S. Mukhopadhyay (Department of Mathematical

Numerical
SciencesTechnique
IIT(BHU),(MA-201)
Varanasi) 18 / 34
 Gauss elimination method contd...

Step 2:
(2)
Assume that a22 6= 0 and eliminate x2 from (2.6). Define
(2)
a32
m32 = (2)
a22
Subtract m32 times (2.5) from (2.6) to get

a11 x1 + a12 x2 + a13 x3 = b1 (2.7)

(2) (2) (2)
a22 x2 + a23 x3 = b2 (2.8)
(3) (3)
a33 x3 = b3 (2.9)
The new coefficients are defined by
(3) (2) (2) (3) (2) (2)
a33 = a33 − m32 a23 , b3 = b3 − m32 b2

Prof. S. Mukhopadhyay (Department of Mathematical

Numerical
SciencesTechnique
IIT(BHU),(MA-201)
Varanasi) 19 / 34
 Gauss elimination method contd...

Step 3:

Using back substitution to solve successively for x3 , x2 and x1 , we get

(3)
b3
x3 = (3)
a33
(2) (2)
b2 − a23 x3
x2 = (2)
a22
b1 − a12 x2 − a13 x3
x1 =
a11
The algorithm for n = 3 is easily extended to a general n × n non-singular linear system.
Note: Gaussian elimination method is a direct method which solves the linear system
exactly. However, sometime, this method fails to give the correct solution as illustrated
in the following example.

Prof. S. Mukhopadhyay (Department of Mathematical

Numerical
SciencesTechnique
IIT(BHU),(MA-201)
Varanasi) 20 / 34
 Gauss elimination method contd...

Example:
Solve the given system of equations

6x1 + 2x2 + 2x3 = − 2

2 1
2x1 + x2 + x3 =1
3 3
x1 + 2x2 − x3 =0

using Gaussian elimination method with digits upto four decimal places.

Prof. S. Mukhopadhyay (Department of Mathematical

Numerical
SciencesTechnique
IIT(BHU),(MA-201)
Varanasi) 21 / 34
 Gauss elimination method contd...

Solution:

The given system can be written as

6.000x1 + 2.000x2 + 2.000x3 = − 2.000

2.000x1 + 0.6667x2 + 0.3333x3 =1.000
1.000x1 + 2.000x2 − 1.000x3 =0.000

The augmented matrix for the system Ax = b is

6 2 2
 
−2
 2 0.6667 0.3333 1 
1 2 −1 0

Prof. S. Mukhopadhyay (Department of Mathematical

Numerical
SciencesTechnique
IIT(BHU),(MA-201)
Varanasi) 22 / 34
 Continue

‘Set a11 to be 1. I.e., divide first roq by 6:

1 1/3 1/3
 
−1/3
∼  2 0.6667 0.3333 1.000 
1 2 −1.0 0.0

Set a21 =0=a31 to get the augmented matrix as

1 1/3 1/3
 
−1/3
∼ 0 0.0001 −0.3333 1.667 
0 1.667 −1.333 0.3333

Prof. S. Mukhopadhyay (Department of Mathematical

Numerical
SciencesTechnique
IIT(BHU),(MA-201)
Varanasi) 23 / 34
 Gauss elimination method contd...

Solution contd...

Set a22 =1 and then a32 = 0, so that we get the augmented matrix as
1 1/3 1/3
 
−1/3
∼ 0 1 −3333 16670 
0 0 5554.7777 −27788

Prof. S. Mukhopadhyay (Department of Mathematical

Numerical
SciencesTechnique
IIT(BHU),(MA-201)
Varanasi) 24 / 34
 Continue

Hence, finally we get the system of equation transformed to the form

1.000x1 + 0.3333x2 + 0.3333x3 = − 0.3333

x2 −3333x3 =16670
5555x3 =−27790

Therefore, by using the back substitution, sequentially we get

x3 = −5.003, x2 = −4.999, x1 = 3.0004.
Does it satisfy all equations? Also find that the actual solution is x1 = 2.6,
x2 = −3.8 and x3 = −5.
Verify from whiteboard calculations and understand why is this difference
? That’s why pivoting is needed to get more accurate result in such cases.

Prof. S. Mukhopadhyay (Department of Mathematical

Numerical
SciencesTechnique
IIT(BHU),(MA-201)
Varanasi) 25 / 34
 Gauss elimination method contd...
Solution contd...

Alternative:
The difficulty with this elimination process as given above, is that in (??),
the element in row 2 and column 2 should be nonzero, but rounding error
prevented it and makes the relative error very large. To avoid this,
interchange row 2 and 3 and then continue the elimination. The final
system is (with m32 = 0.00005999)
6.000x1 + 2.000x2 + 2.000x3 = − 2.000
1.667x2 − 1.333x3 =0.3334
−0.3332x3 =1.667
with back substitution, we obtain the approximate solution as x1 = 2.602,
x2 = −3.801 and x3 = −5.003, which is close to exact solution.
Note: Pivoting process helped to avoid the problem encountered earlier
and to get more accurate result.
Prof. S. Mukhopadhyay (Department of Mathematical
Numerical
SciencesTechnique
IIT(BHU),(MA-201)
Varanasi) 26 / 34
 Gauss elimination method contd...

A serious problem with the Gauss elimination process is division by

the diagonal term while converting the augmented matrix into upper
triangular form.
If the diagonal element is zero or a vanishingly very small then the
elements of the rows below this diagonal become very large in
magnitude and difficult to handle.
To overcome this problem, we convert the system such that the
element which has large magnitude in that column comes at the
pivotal position i.e., the diagonal position.

Prof. S. Mukhopadhyay (Department of Mathematical

Numerical
SciencesTechnique
IIT(BHU),(MA-201)
Varanasi) 27 / 34
 Gauss elimination method contd...

Partial Pivoting:
To fix up the diagonal element at ith column, find the entry in the
remaining part from ith position of that column with the largest
absolute value. This entry is called the pivot.
Perform a row interchange, if necessary, so that the pivot is in the ith
row.

Divide the ith row by the pivot.

Use elementary row operations to set the remaining entries in the ith
column to zero.

Prof. S. Mukhopadhyay (Department of Mathematical

Numerical
SciencesTechnique
IIT(BHU),(MA-201)
Varanasi) 28 / 34
 Gauss elimination method contd...

Complete Pivoting:
The term partial in partial pivoting refers to the fact that in each pivot
search only entries in the left column of the matrix or submatrix are
considered. This search can be extended to include every entry in the
coefficient matrix or submatrix; the resulting technique is called Gaussian
elimination with complete pivoting.
Unfortunately, neither complete pivoting nor partial pivoting solves all
problems of rounding error.

Note: Complete pivoting requires a lot of overhead and , therefore, it is

not generally used.

Prof. S. Mukhopadhyay (Department of Mathematical

Numerical
SciencesTechnique
IIT(BHU),(MA-201)
Varanasi) 29 / 34
 Gauss elimination method contd...

Example:
Use Gaussian elimination with partial pivoting to solve the following system

0.143x1 + 0.357x2 + 2.01x3 = − 5.173

−1.31x1 + 0.911 + 1.99x3 = − 5.458
11.2x1 − 4.30x2 − 0.605x3 =4.415

After each intermediate calculation, round the result to three significant

digits.

Prof. S. Mukhopadhyay (Department of Mathematical

Numerical
SciencesTechnique
IIT(BHU),(MA-201)
Varanasi) 30 / 34
 Gauss elimination method contd...
Solution:

The augmented matrix for the system Ax = b is

0.143 0.357 2.01
 
−5.17
 −1.31 0.911 1.99 −5.46 
[11.2] −4.30 −0.605 4.42
In the left column 11.2 is the pivot because it is the entry that has the largest
absolute value. Therefore we interchange the first and third rows and apply
elementary row operations as follows (interchange the first and third rows)
[11.2] −4.30 −0.605 4.42
 
 −1.31 0.911 1.99 −5.46 
0.143 0.357 2.01 5.17
Dividing the first row by 11.2 produces a new first row
1.00 −0.384 0.395
 
−0.054
 −1.31 0.911 1.99 −5.46 
0.143 0.357 2.01 −5.17
Prof. S. Mukhopadhyay (Department of Mathematical
Numerical
SciencesTechnique
IIT(BHU),(MA-201)
Varanasi) 31 / 34
 Gauss elimination method contd...

Solution contd...

Adding 1.31 times the first row to the second row produces a new second row
1.00 −0.384 −0.0540 0.395
!
0.00 0.408 1.92 −4.94
0.143 0.357 2.01 −5.17
Adding – 0.143 times the first row to the third row produces a new third row
1.00 −0.384 −0.0540 0.395
!
0.00 0.408 1.99 −4.94
0.00 0.412 2.02 −5.23
This completes the first step. We consider the submatrix formed by deleting the first
row and first column. In this matrix the pivot is 0.412, which means that the second and
third rows should be interchanged (interchange the second and third rows)
1.00 −0.384 −0.054 0.395
!
0.00 [0.412] 2.02 −5.23
0.00 0.408 1.92 −4.94

Prof. S. Mukhopadhyay (Department of Mathematical

Numerical
SciencesTechnique
IIT(BHU),(MA-201)
Varanasi) 32 / 34
 Gauss elimination method contd...

Solution contd...

Dividing the second row by 0.412 produces a new second row

1.00 −0.384 −0.054 0.395

!
0.00 1.00 4.90 −12.7
0.00 0.408 1.92 −4.94

Adding –0.408 times the second row to the third row produces a new third row

1.00 −0.384 −0.054 0.395

!
0.00 1.00 4.90 −12.7
0.00 0.00 −0.080 0.240

This completes the second step. By using back substitution, the solution can be
obtained as
x3 = −3.00, x2 = 2.00, x1 = 1.00
which agrees with the exact solution of x1 = 1, x2 = 2 and x3 = −3 when rounded to
three significant digits.

Prof. S. Mukhopadhyay (Department of Mathematical

Numerical
SciencesTechnique
IIT(BHU),(MA-201)
Varanasi) 33 / 34
 Task

Task:
Solve the following system of equations by the Gauss elimination method with partial
pivoting:
(a)
4x1 + x2 + x3 = 4

x1 + 4x2 –2x3 = 4

3x1 + 2x2 –4x3 = 6

(b)
x1 + x2 –x3 = 2
2x1 + 3x2 + 5x3 = –3

3x1 + 2x2 –3x3 = 6

********

Prof. S. Mukhopadhyay (Department of Mathematical

Numerical
SciencesTechnique
IIT(BHU),(MA-201)
Varanasi) 34 / 34