Engineering College Chemical Engineering Dept.

Chapter -4-
Indirect Methods for Solving Linear Systems
4.1 Introduction
In this chapter we describe some popular iterative techniques which require an initial
approximation to the solution. These methods will not be expected to return the exact
solution even if all the calculations could be performed using exact arithmetic. In many
instances, however, they are more effective than the direct methods, since they can require
far less computational effort and round-off error is reduced. This is particularly true when
the matrix is sparse—that is, when it has a high percentage of zero entries.
A linear system of n equations in n unknowns 𝑥1 , 𝑥2 , … , 𝑥𝑛 is a set of equations
𝐸1 , 𝐸2 , … , 𝐸𝑛 of the form
𝐸1 : 𝑎11 𝑥1 + ⋯ + 𝑎1𝑛 𝑥𝑛 = 𝑏1
𝐸2 : 𝑎21 𝑥1 + ⋯ + 𝑎2𝑛 𝑥𝑛 = 𝑏2
(1)
… … … … … … … … … … … … … … ..
𝐸𝑛 : 𝑎𝑛1 𝑥1 + ⋯ + 𝑎𝑛𝑛 𝑥𝑛 = 𝑏𝑛
where the coefficients 𝑎𝑗𝑘 , & 𝑏𝑗 re given numbers. The system is called homogeneous
4.2 Iterative Methods for Solving Linear Systems
4.2.1 The Jacobi Method
These are classic methods that date to the late eighteenth century, but they find current
application in problems where the matrix is large and has mostly zero entries in predictable
locations. Applications of this type are common, for example, in the study of large
integrated circuits and in the numerical solution of boundary value problems and partial-
differential equations.
An iterative technique for solving the n×n linear system (1)
𝐴𝑥 = 𝑏
starts with an initial approximation x(0) to the solution x and generates a sequence of vectors
{𝑥 (𝑘) }∞
𝑘=1 that converges to x. These iterative techniques involve a process that converts
the system Ax = b into an equivalent system of the form

32
 Engineering College Chemical Engineering Dept.

𝑛
(𝑘)
𝑎𝑖𝑗 𝑥𝑗 (𝑘−1) 𝑏𝑖
𝑥𝑖 = ∑ (− )+ 𝑒𝑞(2)
𝑎𝑖𝑖 𝑎𝑖𝑖
𝑗=1
𝑗≠𝑖

or
𝑥 (𝑘) = 𝑇 𝑥 (𝑘−1) + 𝑐
for each k = 1,2, 3,....
The method is written in the form 𝑥 (𝑘) = 𝑇 𝑥 (𝑘−1) + 𝑐 by splitting A into its diagonal and
off-diagonal parts. To see this, let D be the diagonal matrix whose diagonal entries are those
of A, -L be the strictly lower-triangular part of A, and -U be the strictly upper triangular
part of A. With this notation,
𝑎11 𝑎12 … 𝑎1𝑛
𝑎21 𝑎22 … 𝑎2𝑛
𝐴=[… … … … ] is split into
𝑎𝑛1 𝑎𝑛2 … 𝑎𝑛𝑛
𝑎11 0 … 0 0 0 … 0 0 −𝑎12 … −𝑎1𝑛
0 𝑎22 … 0 −𝑎 0 … 0 0 0 … −𝑎2𝑛
𝐴=[ ] − [ 21 ]−[ ]
… … … … … … … … … … … …
0 0 … 𝑎𝑛𝑛 −𝑎𝑛1 −𝑎𝑛2 … 0 0 0 … 0
𝐴=𝐷−𝐿−𝑈
𝐴𝑥 = 𝑏 𝑜𝑟 (𝐷 − 𝐿 − 𝑈)𝑥 = 𝑏
is then transformed into
𝐷𝑥 = (𝐿 + 𝑈)𝑥 + 𝑏
𝑥 = 𝐷 −1 (𝐿 + 𝑈)𝑥 + 𝐷−1 𝑏
This results in the matrix form of the Jacobi iterative technique:
𝑥 (𝑘) = 𝑇𝑗 𝑥 (𝑘−1) + 𝑐𝑗
𝑇𝑗 = 𝐷−1 (𝐿 + 𝑈)𝑥 & 𝑐𝑗 = 𝐷−1 𝑏
If aii = 0 for some i and the system is nonsingular, a reordering of the equations is
performed so that no aii = 0. To speed convergence, the equations should be arranged so
that aii is as large as possible.

33
 Engineering College Chemical Engineering Dept.

Example 1: Solve the system

𝐸1 : 10 𝑥1 − 𝑥2 + 2 𝑥3 = 6
𝐸2 : − 𝑥1 + 11 𝑥2 − 𝑥3 + 3 𝑥4 = 25
𝐸3 : 2𝑥1 − 𝑥2 + 10 𝑥3 − 𝑥4 = −11
𝐸4 : 3𝑥2 − 𝑥3 + 8 𝑥4 = 15

Sol: to converts the system Ax = b to 𝑥 (𝑘) = 𝑇 𝑥 (𝑘−1) + 𝑐

1 2 6
𝑥1 (𝑘) = 𝑥2 (𝑘−1) − 𝑥3 (𝑘−1) +
10 10 10
1 1 3 25
𝑥2 (𝑘) = 𝑥1 (𝑘−1) + 𝑥3 (𝑘−1) − 𝑥4 (𝑘−1) +
11 11 11 11
2 1 1 11
𝑥3 (𝑘) =− 𝑥1 (𝑘−1) + 𝑥2 (𝑘−1) + 𝑥4 (𝑘−1) −
10 10 10 10
3 1 15
𝑥4 (𝑘) = − 𝑥2 (𝑘−1) + 𝑥3 (𝑘−1) +
8 8 8

0 1/10 −2/10 0 6/10

1/11 0 1/11 −3/11 25/11
𝑇=[ ] &𝑐 =[ ]
−2/10 1/10 0 1/10 −11/10
0 −3/8 1/8 0 15/8
For an initial approximation, suppose 𝑥 (0) = [0 0 0 0]. Then x(1) is given by

1 2 6
𝑥1 (1) = 𝑥2 (0) − 𝑥3 (0) + = 0.6
10 10 10
1 1 3 25
𝑥2 (1) = 𝑥1 (0) + 𝑥3 (0) − 𝑥4 (0) + = 2.2727
11 11 11 11
2 1 1 11
𝑥3 (1) =− 𝑥1 (0) + 𝑥2 (0) + 𝑥4 (0) − = −1.1
10 10 10 10

34
 Engineering College Chemical Engineering Dept.

3 (0) 1 (0) 15
𝑥4 (1) = − 𝑥 + 𝑥3 + = 1.875
8 2 8 8
Additional iterates, are generated in a similar manner and are presented in Table below
The decision to stop after 10 iterations was based on the criterion
‖𝑥 (10) − 𝑥 (9) ‖∞ = 0.000444 < 10−3

k 0 1 2 3 4 5 6 7 8 9 10
x1 0 0.6 1.047273 0.932636 1.015199 0.988991 1.003199 0.998128 1.000625 0.999674 1.000119
x2 0 2.272727 1.715909 2.053306 1.953696 2.011415 1.992241 2.002307 1.99867 2.000448 1.999768
x3 0 -1.1 -0.80523 -1.04934 -0.96811 -1.01029 -0.99452 -1.00197 -0.99904 -1.00037 -0.99983
x4 0 1.875 0.885227 1.130881 0.973843 1.021351 0.994434 1.003594 0.998888 1.000619 0.999786

The exact solution 𝑥 = [1 2 −1 1]

4.2.2 The Gauss-Seidel Method

A likely improvement on the Jacobi method can be seen by reconsidering eq (2). In this
equation all the components of 𝑥 (𝑘−1) are used to compute each of the 𝑥𝑖 (𝑘) . Since the
components 𝑥1 (𝑘) , 𝑥2 (𝑘) , … . 𝑥𝑖−1 (𝑘) have already been computed and are probably better
approximations to the actual solutions 𝑥1 , 𝑥2 , … . , 𝑥𝑖−1 than are
(𝑘−1) (𝑘−1) (𝑘−1) (𝑘)
𝑥1 , 𝑥2 , … . 𝑥𝑖−1 . We can compute 𝑥𝑖 using these most recently calculated
values. That is, we can use
𝑖−1 𝑛
𝑎𝑖𝑗 𝑥𝑗 (𝑘) 𝑎𝑖𝑗 𝑥𝑗 (𝑘−1) 𝑏𝑖
𝑥𝑖 (𝑘) = − ∑( )− ∑ ( )+ 𝑒𝑞(2)
𝑎𝑖𝑖 𝑎𝑖𝑖 𝑎𝑖𝑖
𝑗=1 𝑗=𝑖+1

Example 2: Solve the system using

𝐸1 : 10 𝑥1 − 𝑥2 + 2 𝑥3 = 6
𝐸2 : − 𝑥1 + 11 𝑥2 − 𝑥3 + 3 𝑥4 = 25
𝐸3 : 2𝑥1 − 𝑥2 + 10 𝑥3 − 𝑥4 = −11
𝐸4 : 3𝑥2 − 𝑥3 + 8 𝑥4 = 15

35
 Engineering College Chemical Engineering Dept.

Sol: to converts the system Ax = b to 𝑥 (𝑘) = 𝑇 𝑥 (𝑘−1) + 𝑐

For an initial approximation, suppose 𝑥 (0) = [0 0 0 0].
1 2 6
𝑥1 (𝑘) = 𝑥2 (𝑘−1) − 𝑥3 (𝑘−1) +
10 10 10
1 1 3 25
𝑥2 (𝑘) = 𝑥1 (𝑘) + 𝑥3 (𝑘−1) − 𝑥4 (𝑘−1) +
11 11 11 11
2 1 1 11
𝑥3 (𝑘) =− 𝑥1 (𝑘) + 𝑥2 (𝑘) + 𝑥4 (𝑘−1) −
10 10 10 10
3 (𝑘) 1 (𝑘) 15
𝑥4 (𝑘) = − 𝑥 + 𝑥3 +
8 2 8 8

Then x(1) is given by

1 2 6
𝑥1 (1) = ∗0− ∗0+ = 0.6
10 10 10
1 1 3 25
𝑥2 (1) = ∗ 0.6 + ∗0− ∗0+ = 2.327273
11 11 11 11
2 1 1 11
𝑥3 (1) = − ∗ 0.6 + ∗ 2.327273 + ∗0− = −0.98727
10 10 10 10
3 1 15
𝑥4 (1) = − ∗ 2.327273 + ∗ −0.98727 + = 0.878864
8 8 8

x(2) is given by
1 2 6
𝑥1 (2) = ∗ 2.327273 − ∗ −0.98727 + = 1.03018
10 10 10
1 1 3 25
𝑥2 (2) = ∗ 1.03018 + ∗ −0.98727 − ∗ 0.878864 + = 2.0369
11 11 11 11
2 1 1 11
𝑥3 (2) = − ∗ 1.03018 + ∗ 2.0369 + ∗ 0.8788 − = −1.01446
10 10 10 10
3 1 15
𝑥4 (2) = − ∗ 2.0369 + ∗ −1.01446 + = 0.98434
8 8 8

36
 Engineering College Chemical Engineering Dept.

k 0 1 2 3 4 5 6 7 8
x1 0 0.6 1.030182 1.006585 1.000861 1.000091 1.000008 1.000001 1
x2 0 2.327273 2.036938 2.003555 2.000298 2.000021 2.000001 2 2
x3 0 -0.98727 -1.01446 -1.00253 -1.00031 -1.00003 -1 -1 -1
x4 0 0.878864 0.984341 0.998351 0.99985 0.999988 0.999999 1 1

𝑥 (5) is accepted as a reasonable approximation to the solution. Note that Jacobi’s

method in Example 1 required twice as many iterations for the same accuracy.

4.2.3The SOR Method

The SOR method is similar to the Jacobi and Gauss-Seidel methods, but it uses a scaling
factor to more rapidly reduce the approximation error.
The SOR technique is one of a class of relaxation methods that compute approximations
𝑥 (𝑘) by the formula
𝑖−1 𝑛
𝜔
𝑥𝑖 (𝑘) = (1 − 𝜔)𝑥𝑖 (𝑘−1) + [𝑏𝑖 − ∑ 𝑎𝑖𝑗 𝑥𝑗 (𝑘) − ∑ 𝑎𝑖𝑗 𝑥𝑗 (𝑘−1) ]
𝑎𝑖𝑖
𝑗=1 𝑗=𝑖+1

where ω is the scaling factor. When ω = 1, we have the Gauss-Seidel method.

under-relaxation methods:
When 0 < ω <1, the procedures can be used to obtain convergence of some systems that
are not convergent by the Gauss-Seidel method.
over-relaxation methods:
When 0 < ω <1, the procedures are used to accelerate the convergence for systems that are
convergent by the Gauss-Seidel technique. These methods are abbreviated SOR for
Successive Over Relaxation and are used for solving the linear systems that occur in the
numerical solution of certain partial-differential equations.

37
 Engineering College Chemical Engineering Dept.

Example 3: The linear system Ax = b given by

𝐸1 : 4𝑥1 + 3 𝑥2 = 24
𝐸2 : 3𝑥1 + 4 𝑥2 − 𝑥3 = 30
𝐸3 : − 𝑥2 + 4𝑥3 = −24
By 1- Gauss-Seidel method
2- SOR method with ω = 1.25
Sol . let 𝑥 (0) = [1 1 1 ]𝑡
1- Gauss-Seidel method
3
𝑥1 (1) = − 𝑥2 (0) + 6 = 5.25
4
3 1 30
𝑥2 (1) = − 𝑥1 (1) + 𝑥3 (0) + = 3.8125
4 4 4
1
𝑥3 (1) = 𝑥2 (1) − 6 = −5.046875
4

38
 Engineering College Chemical Engineering Dept.

2- SOR method

1.25
𝑥1 (1) = (1 − 1.25)𝑥1 (0) + [24 − 3 𝑥2 (0) ]
4
1.25
𝑥2 (1) = (1 − 1.25)𝑥2 (0) + [30 − 3 𝑥1 (1) + 𝑥3 (0) ]
4
1.25
𝑥3 (1) = (1 − 1.25)𝑥3 (0) + [−24 + 𝑥2 (1) ]
4
𝑡ℎ𝑒 𝑓𝑖𝑛𝑎𝑙 𝑓𝑜𝑟𝑚 𝑤𝑖𝑙𝑙 𝑏𝑒
(0)
𝑥1 (1) = −0.25𝑥1 (0) − 0.9375 𝑥2 + 7.5
(1) (0)
𝑥2 (1) = −0.25𝑥2 (0) − 0.9375 𝑥1 + 0.3125 𝑥3 + 9.375
(1)
𝑥3 (1) = −0.25𝑥3 (0) + 0. 3125 𝑥2 − 7.5

Gauss-Seidel
k 0 1 2 3 4 5 6 7
x1 1 5.25 3.140625 3.087891 3.054932 3.034332 3.021458 3.013411
x2 1 3.8125 3.882813 3.926758 3.954224 3.97139 3.982119 3.988824
x3 1 -5.04688 -5.0293 -5.01831 -5.01144 -5.00715 -5.00447 -5.00279

SOR with ω = 1.25

k 0 1 2 3 4 5 6 7
x1 1 6.3125 2.622314 3.133303 2.957051 3.003721 2.996328 3.00005
x2 1 3.519531 3.958527 4.010265 4.007484 4.002925 4.000926 4.000259
x3 1 -6.65015 -4.60042 -5.09669 -4.97349 -5.00571 -4.99828 -5.00035

The first seven iterates for each method are listed in Tables above. To be accurate to
seven decimal places, the Gauss-Seidel method required 34 iterations, as opposed to
only 14 iterations for the SOR method with ω = 1.25.

value of ω
The obvious question to ask is how the appropriate value of ω is chosen. Although no
complete answer to this question is known for the general n×n linear system, the
following result can be used in certain situations.

39
 Engineering College Chemical Engineering Dept.

Recall 𝑇𝑗 = 𝐷−1 (𝐿 + 𝑈)
Example 4: The linear system Ax = b given by
𝐸1 : 4𝑥1 + 3 𝑥2 = 24
𝐸2 : 3𝑥1 + 4 𝑥2 − 𝑥3 = 30
𝐸3 : − 𝑥2 + 4𝑥3 = −24
For this system specify the value of ω
Sol.
4 3 0
𝐴 = [3 4 −1] = 𝐷 − 𝐿 − 𝑈
0 −1 4
4 0 0 0 0 0 0 −3 0
𝐷 = [0 4 0] , 𝐿 = [−3 0 0] & 𝑈 = [0 0 1]
0 0 4 0 1 0 0 0 0
1/4 0 0 0 −3 0 0 −0.75 0
−1 (𝐿 0 1/4 0
𝑇𝑗 = 𝐷 + 𝑈) = [ ] [−3 0 1] = [−0.75 0 0.25]
0 0 1/4 0 1 0 0 0.25 0
−𝜆 −0.75 0
𝑇𝑗 − 𝜆𝐼 = [−0.75 𝜆 0.25]
0 0.25 𝜆
𝑑𝑒𝑡(𝑇𝑗 − 𝜆𝐼) = −𝜆 (𝜆2 − 0.625) = 0 → 𝜌(𝑇𝑗 ) = 𝜆 = √0.625 < 1

40
 Engineering College Chemical Engineering Dept.

2
𝜔= ≈ 1.24
1 + √1 − [𝜌(𝑇𝑗 )]2

HW: For these matrices, that are both tridiagonal and positive definite, use the SOR
method with the optimal choice of ω.

41