3.

8 ITERATIVE TECHNIQUES FOR LINEAR SYSTEMS: BASIC CONCEPTS AND

METHODS
Having just devoted several sections to the development of direct techniques for lin
ear systems-techniques that produce an answer in a fixed number of operations -it
is natural to ask why we would want or even need to develop iterative techniques.
For systems of small dimension, there is no need. Direct techniques will perform
very efficiently. However, linear systems arising from practical applications will fre
quently be quite large. The coefficient matrices associated with these systems also
tend to be sparse, meaning that only a small percentage of the entries are nonzero.
We will encounter systems of this type in Chapter 9 when we treat the solution of
elliptic partial differerntial equations.
For systens with large, sparse coefficient matrices, direct techniques are often
less efficient than iterative techniques. Even though multiple iterations need to
be performed to achieve convergence, an iterative solution will typically require
fewer total operations than a direct solution. It will often happen that the nonzero
elements in the coefficient matrix will exhibit a well-defined pattern. In these cases,
an iterative solution will not require the storage of the coeficient matrix at all
only the structure of the equations will be needed. As an added bonus, iterative
techniques are generally insensitive to roundoff error.
Basic Concepts
Basic iterative techniques for the solution of linear systerns of equations are analo
gous to the fixed-point techniques which were discussed in Chapter 2. The original
linear system Ax = b, which can be interpreted as the rootfinding problem
find the n-vector x so that Ax- b=0,
is first converted to the fixed point problerm
fnd the n-vector x so that x=Tx+ c,
for some matrix T and vector c. Next, starting from some initial approximation
to the solution of the fixed point problem, x(, a sequence of vectors (x} is
computed according to the rule
x(k+1) = Tx(*) + c. (1)
Within this context, the matrix T is called the iteration matriz. The functional
iteration is terminated when some appropriate measure of the difference between
224 Chapter 3 Systems of Equations

successive vectors in the sequence, x(*) and xk+1), falls below a user specified
tolerance.
The analysis of the functional iteration scheme given by (1) boils down to
four important questions. First, what conditions guarantee a unique solution to the
ixed point problem? Second, under what conditions will the sequence generated
by (1) converge to this unique fixed point? Third, when the sequence generated
by (1) converges, how quickly does it converge? Fourth, what conditions must
the matrix T and the vector c satisfy in order for the fxed point problem to be
consistent with the original rootinding problem (i.e., for the two problems to have
the same solution)?
The following theorem from general matrix theory plays a major role in es
tablishing answers to these questions.
Theorem. Let A be an n x n matrix. Then the following statements are
equivalent:
1. p(A) <l, where p(A) denotes the spectral radius of A;
2. A* ’0 as k ’ oo; and
3. A*x’0as k ’ o for any vector x.
A proof of this result can be found in Isaacson and Keller (1.
Let's start with the question of the uniqueness of the solution of the fixed
point problem. Manipulating the fixed-point equation, we find
x=Tx+cx- Tx = c
(l-Tx =c.
Erom this last equation, it follows that the fixed point problem has a unique solution
if andonly ifthe matrix I-T is nonsingular. Asufficient condition forl-T to be
nonsingular is p(T) <1 (see Exercise 10 from Section 3.3). Hence, the fixed point
problem is guaranteed to have a unique solution whenever p(T) <1.
To establish convergence, let x* denote the solution to the fixed point problem,
and define the iteration error vector by ef) = x(*) -x*. Subtracting the equation
x* =Tx* +c from equation (1) yields the error evolution equation
elk+1) = Telk),

Working backward through this equation, we find

elk+1) = Te(k)
- T(Tek-1) =Telk-)
=T°(Tek-2)) =T°elk-2)
=Tk+e(0),
 Section 3.8 Iterative Techniques, Basic Concepts 225

Ideally, elk+)) should approach 0 as k ’ o for any choice of initial vector x(°),
that is, for any initial error vector e(0), The theorern stated above indicates that
this will happen if and only if p(T) <1. Hence, the iteration scheme defined by
equation (1) will converge for any choice of the initial vector x(0) if and only if
P(T) < 1.
From the error evolution equation, we find

for any vector norm || || and associated natural matrix norm. Provided ||T|| <1,
it can be shown that

(see Exercise 14). These two inequalities imply that the sequence (x*)} converges
linearly with an asymptotic error constant that is less than or equal to ||T||. Carry
ing out a more precise analysis, it can be shown that the asymptotic error constant
is equal to p(T). The proof is based on the fact that p(T) is the greatest lower
bound for all natural matrix norms of T. See Ortega 2 for details. Thus, the
smaller the spectral radius of the iteration matrix, the faster the convergence of the
corresponding iterative scheme.
The final preliminary issue to discuss is that of consistency. In order for the
iteration defined by (1) to be of any practical use, the solution of the fixed point
problem, x* = (|-T)-'c, must be identical to the solution of the original linear
system, x* = A-lb. Hence, when constructing the fixed point problem from the
linear system, we must be certain that T and c satisfy the relation
(I-T)'c= A'b.
Splitting Methods
A broad class of consistent iterative methods, known as splitting metbods, can be
constructed by introducing the notion of a splitting.
Definition. Let A be a given n xn matrix. If M and N are n xn matrices with
nonsingular and A = M- N, then the pair (M, N) is called a SPLITTING
of the matrix A.

So let's suppose that (M, N) forms a splitting of the matrix A. Then

Ax = b is equivalent to (M - N)x = b.
Clearing the parentheses and transposing the term involving the matrix to the
right-hand side of the equation yields
Mx = Nx +b.

Finally, premultiplying by M-l produces

x= M-Nx+ M-b.
226 Chapter 3 Systems of Equations

Hence the splitting A = M- N determines the fixed point problem x = Tx+c

and associated iteration scheme x{k+1)=Tx*) +c, where
T= M-N and c= M-'b.

To establish that splitting methods are always consistent, first note that with
T= M-N

I-T=I- M-'N

= M-(M- N)
= M-'A.

Therefore, (I - T)-1 = A-'M. Finally, withc= M-'b,

(U-T) 'c=A-'M:M-b
= A-'b,

as required. Descriptions of the three most commonly used splitting methods are
presented below. The convergence properties for these three methods will then be
discussed at the end of the section.
The Jacobi Method, Gauss-Seidel Method, and SOR Method
To identify the splittings associated with the Jacobi method, the Gauss- Seidel
method, and the SOR method, first express the coefficient matrix A in the form
A=D-L-U.

Here, D is the diagonal part of A, -L is the strictly lower triangular part of A, and
-U is the strictly upper triangular part. It is important to keep in mind that the
matrices L and U used here are in no way related to the LU decomposition of the
coefficient matrix. As an example, suppose
5 1 2

A= -31 92 -74
Then

5 0 0 0 -1 -2
D=0 9 and U=|0 0 -4
0 0 -7 -1 -2 0 00 0

The Jacobi method is based on the splitting M = D and N= L+ U. In

order for M to be nonsingular, it must be the case that, for each i, d,, = ai,i #0.
If this relationship does not hold for even a single value of i, then the equations
in the system must be reordered before the Jacobi method can be applied. With
 Section 3.8 Iterative Techniques, Basic Concepts 227

the specific choice of splitting indicated above, the iteration scheme for the Jacobi
method is defined by
(k+1) T;acx + Cjac1 (2)
where

Tjac = D(L+U) and Cjac = D-'b

Taking into account the structure of the iteration matrix, Tjacs and the vector cjac1
the individual components of equation (2) can be written as

1
(3)
j=1 j=i+l

Hence, the Jacobi method is equivalent to solving the i-th equation in tbe system
for the unknown TË.
Since, in general, the value of z will be needed to compute z ) for each
j=i+1,i+2,...,n, the value of z cannot be overwritten bythe newly computed
value of z*+), This implies that when implementing the Jacobi method, two
storage arrays will have to be maintained, one for the old approximation vector
x(*) and one for the new approximation vector xlk+1), It also follows that the
components of xk+1) can be computed in any order and that, on a parallel or
vector machine, all components of xk+1) can be computed simultaneously. For this
reason, the Jacobi method is often called the method of Simultaneous Relaxation.

EXAMPLE 3.22 The Jacobi Method in Action

Consider the system of equations

5z1 + + 23 10
-3t1 + 9t2 + 4T3 -14
+ 212 - 73 -33.

The Jacobi method, when applied to tbis system, will produce the sequence of
approximations {x4)} according to the rules
228 Chapter 3 Systems of Equations

If we start with x(0) =0 0 0|,then the components of x) are

10--2 =2
14
9
33

The following table summarizes the 14 iterations of the Jacobi method that were
needed for l|x(k+1)- x()| to fall below 5 x 10-4, Other stopping criteria can be
inposed, but this is the most comnon. The exact solution to this problem is
x= [1 -3 4j.
Hence, |x- x14)| = 2.43 x 10-4,
x(*)
[0.000000 0.000000 0.000000 1
1 2.000000 -1.555556 4.714286
T
2 0.425397 -2.984127 4.555556
3 0.774603 -3.438448 3.922449
4 1.118710 -3.040665 3.842530
5 1.071121 -2.890443 4.005340
6 0.975953 -2.978666 4.041462 T
7 0.979148 -3.026443 4.002660
1.004225 -3.008133 3.989466
9 1.005840 -2.993910 3.998280
10 0.999470 -2.997289 4.002574
11 0.998428 -3.001321 4.000699
12 0.999985 -3.000835 3.999398 1
13 1.000408 -2.999738 3.999759
14 1.000044 -2.999757 4.000133|

An obvious improvement that can be made to the Jacobi method is to use the
value of (k+1) as s0on as it has been calculated in the computation of all subsequent
entries in the vector x(*+), rather than waiting until the next iteration. After all,
is supposed to be a better approximation to z, than . This modiication
amounts to changing equation (3) to
i-1
|b-gak+1). -

(4)
i, j=l
 Section 3.8 Iterative Techniques, Basic Concepts 229

the only difference between the equations is that the superscript on z in the first
summation is now k+1. The iteration scheme corresponding to equation (4) is
known as the Gauss-Seidel method. Note that the Gauss-Seidel method is not
vectorizable. The entries in x*+1) must be computed in succession. Hence, the
Gauss-Seidel method is also known as Successive Relaxation.
Working back ward from equation (4), we find that the splitting upon which
the Gauss-Seidel method is based is

M=D-L and N=æ.

Thus, the iteration matrix for the Gauss-Seidel method is given by

Tg, = (D - L)-'u,
and the vector c is given by
Cga = (D - L)-'b.
The necessary and sufficient condition for the matrix M to be nonsingular is the
same as above: for each i, we must have di, ai,i #0.
EXAMPLE 3.23 The Gauss-Seidel Method Action

Reconsider the system of equations

+ + 2c3 10
-3c1 + 4T3 = -14
2 703 = -33.

The Gauss-Seidel method, when applied to this system, will produce the sequence
of approximations {x} according to the rules

at*10- -2
a ) 4+3=t)-af)

If we start with x(0) = 0 0 o1,then the components of x)are

299
63
230 Chapter 3 Systems of Equations

The following table summarizes the 10 iterations of the Gauss-Seidel method that
were needed for l|x(*+1) x*)|o to fall below 5 x 10-. Recall that the exact
solution to this problem is

x= [1 -3 4 ].
Hence, x x(0= 7.8 x 10-5, Note that convergence is obtained with the
Gauss-Seidel method in roughly 30% fewer iterations than the Jacobi method. Fur
ther, the error in the final Gauss-Seidel approximation is roughly one-third the error
in the final Jacobi approximation.
x*)
[0.000000 0.000000 0.000000 1
1 2.000000 -0.888889 4.746032
2 0.279365 -3.571781 3.733686
3 | 1.220882 -2.808011 4.086409
4 0.927039 -3.062724 3.971656
5 1.023883 -2.979442 4.009286
6 0.992174 -3.006736 3.996958
7 1.002564 -2.997793 4.000997
8 0.999160 -3.000723 3.999673
1.000275 -2.999763 4.000107
10 0.999910 -3.000078 3.999965 |

The final iterative technique that we will discuss in this section is the SOR
method. An explanation for the name of the method will be provided shortly. This
technique attempts to improve upon the convergence of the Gauss-Seidel met.hod
by computing z** as a weighted average of z and the value produced by
the Gauss-Seidel method, as given in equation (4). Let the weighting parameter,
also known as a relaxation parameter, be denoted by w. Then the analogue of
equations (3) and (4) for the SOR method is
i-!

«*)=(1-w)a+-a) j=l
(5)

Note that when w= 1, the SOR method reduces to the Gauss-Seidel method. Typi
cally, there exists a range of w values for which the SOR method will converge faster
than the Gauss-Seidel method. The splitting associated with the SOR method is

M= -D-L and
w-(-)ou.
 Section 3.8 Iterative Techniques, Basic Concepts 231

Therefore,
Thor =

and

Cgor =
-(o-)"'
EXAMPLE 3.24 The SOR Method in Action
One more time we will consider the system of equations
5z1 + + 223 10
-3T1 4T3 -14
232 73 -33.

With w =0.9, the iteration equations for the SOR method become
0.9
*)=0.1z+ 5
0.9
9 -4 +3=+1) - az]
0.9
zf+1) =0.1zs) 7 -3- t*)-r*)).
If we start with x(0) =|0 001, then the components of x)are
0.9

a =0.1z0) 0.9
9 -14+3ef -a=-0.36
T-33 - -2z4= 4.253143.
The table shown below summarizes the 6 iterations of the SOR method that were
needed for |x(k+1) - x(*)| to fall below 5 x 10-4, Note that |x - x(6)|o =
6.0 x 10-5
x(*)
0.000000 0.000000 0.000000 1
1.800000 -0.860000 4.253143
0.603669 -3.006157 3.972774
3 0.971276 -2.998342 3.994011
T
0.998985 -2.997743 3.999851
5 0.999546 -2.999851 3.999965
6 0.999940 -2.999989 3.999992
 232 Chapter 3 Systems of Equations

Gaaa-6ede

0.6
laato panats,

The selection of the parameter w is crucial to the performance of the SOR

method. The figure above displays the number of iterations
required for the SOR
method to converge to within a tolerance of 5 x 10- as a function of w. The hori
zontal lines indicate the number of iterations required by the
(bottom line) and the Jacobi method (top line). We see thatGauss-Seidel method
there is a range of
values, roughly 0.65<ws1.0, for which the performance of the SOR method is as
good as or better than the perfornance of the
the SOR method outperforms the Jacobi method Gauss-Seidel method. As expected,
over a broader range of w values.

For the example problem we have been examining in this section, the SOR
method perfornel better than the Gauss-Seidel method primarily for w<1. How
ever, for nmany of the practical problems to which the SOR method is applied (such
as the systems of equations associated with the solution of clliptic partial differen
tial equations which we will encounter in Chapter 9), performance is better than
the Gauss-Seidel metbod for w>1. When w is selected greater than 1, the
iterative
method is referred to as an overrelaxation scheme. Since the Gauss-Seidel method
is successive relaxation, this new technique is Successive overrelaxation, or the SOR
method for short.

Specific Convergence Properties for the Jacobi, Gauss-Seidel, and SOR

Methods
We know that the general iterative method x() = Txl4)+cconverges if and only if
p(T) < 1. Are there conditions that, when imposed upon the coefficient matrix A,
willguarantee that the Jacobi, Gauss-Seidel, and SOR methods converge? The
answer to this question is yes; uníortunately, there is no general theory, just a
collection of special cases. For example, it is known that strict diagonal doninance
 Section 3.8 Iterative Techniques, Basic Concepts 233

of A is sufficient to guarantee that both the Jacobi method and the Gauss-Seidel
method will converge for any choice of the initial vector x(0). The proof for the
Jacobi met hod is considered in Exercise 15, while the proof for the Gauss-Seidel
method can be found in Ortega (2].
The following results regarding the Gauss-Seidel method are also useful in
practice.
Theorem.
1. If A is real and symmetric with all positive diagonal elements, then the
Gauss-Seidel method converges if and only if A is positive definite.
2. If A is positive definite, then the Gauss-Seidel method will converge for
any choice of the initial vector x0),
A proof of the first part of this theorem can be found in Isaacson and Keller (1).
Consult Ortega (2 or Ralston and Rabinowitz (3] for a proof of the second part.
For the SOR method, the most important convergence results are as follows:
Theorem.
1. If A has all nonzero diagonal elements, then p(Tsor) > lw-1|. Therefore,
the SOR method can converge only if 0<w<2.
2. If Ais positive definite and 0 <w< 2, then the SOR method will
converge for any choice of the initial vector x(0),

See Ortega (2), Young (4), or Golub and Ortega [5] for details.
What about the speed of convergence? For the sample problem treated in
this section, it was found that the Gauss-Seidel method converged fewer itera
tions than the Jacobi method. Will this relative performance bold in general? The
answer to this question is no. There are, in fact, coefficient matrices for which
the Jacobi method will converge, but the Gauss-Seidel method will notsee Exer
cise 13. Furthermore, there is no general theory to indicate which method, Jacobi or
Gauss-Seidel, will perform best on an arbitrary problem, just a collection of special
cases. For example (see Ralston and Rabinowitz (3] or Young (4] for a proof).
Theorem. Suppose A is an n x n matrix. If ai. >0for each i and a.i S0
whenever i#j, then one and only one of the following statements holds:
1. 0< p(Ts) = p(Tjc) < 1;
2. 1<p(Tjac) < p(Tgs);
3. p(Tjac) = p(T,.) = 0;
4. p(T;ac) = p(T,.) = 1.
Thus, under the hypotheses of this theorem, when one method converges, so will
the other method, with the Gauss-Seidel method converging faster. On the other
hand, when one method diverges, so will the other, with the Gauss-Seidel method
diverging faster.
 234 Chapter 3 Systems of Equations

The inal issue we will address is that of the choice of the relaxation parameter
for the SOR method. In practice, one of the most important special cases is the
following
Theorem. If Ais positive definite and tridiagonal, then p(Tgs) = p(T;ac)]" <
1, and the optimal choice of the relaxation parameter, w, for the SOR method
2

1+ 1-(p(T,c)?
With this choice of w, p(Tor) =w-1.
Ortega (2 provides a proof of this resuit. There are more general conditions under
which the formula for the optimal value of w given in the above theorem holds, but
these require more advanced matrix theory concepts than we have developed here.
For details consult Stoer and Bulirsch [6), Young (4], or Varga [7]. The optimal
choice for w for an arbitrary linear system remains an open question. Methods
for computing the optimal value of w during the iterative process are discussed by
Hageman and Young [8].
References
1. E. Isaacson and B. Keller, Analysis of Numerical Methods, John Wiley and
Sons, New York, 1966.
2. J. M. Ortega, Numerical Analysis: A Second Course, Academic Press, New
York, 1972.
3. A. Ralston and P. Rabinowitz, A First Course Numerical Analysis, 2nd
edition, McGraw-Hill, New York, 1978.
4. D. M. Young, Iterative Solution of Large Linear Systems, Academic Press, New
York, 1971.
5. G. H.Golub and J. M. Ortega, Seientific Computing and Diferentiai Equations:
An Introduction to Numerical Methods, Academic Press, Boston, 1992.
6. J. Stoer and R. Bulirsch, Introduction to Numerical Analysis, Springer-Verlag,
New York, 1980.
7. R. S.Varga, Matriz Iterative Analysis, Prentice Hall, Englewood Cliffs, 1962.
8. L. A. Hageman and D. M. Young, Appied Iterative Methods, Academic Press,
New York, 1981.

EXERCISES

In Exercises 1-4,
(a) Compute Tjac and T, for the given matrix.
(b) Determine the spectral radius of each iteration matrix from part (a).
(c) Will the Jacobi method converge for any choice of initial vector x0)? Will the
Gauss-Seide! method converge for any choice of initial vector x? Expiain.
 Section 3.8 Iterative Techniques, Basic Concepts 235

1.3
1

[ 4 -l -2 1
3. -1 3 0
0 -1 3
3 2 -21
4. -2 -2 1
5 -5
5. For each of the following coefcient matrices and right-hand side vectors, write
out the components of the Jacobi method iteration equation. Then, starting
with the initial vector x(0).=0, perform two iterations of the Jacobi method.
2 -1 0 1 -1

(a)-10 2 3 2 6
3 -1 1
(b) 2 -6 3
-9 7 -20
4 2-1
(c) 2 4 1
-1 1
4 -1 0 0
2 4 -1 0
(d) 0 -2 4 -1 -3
0 0 -2 4 1
6. Repeat Exercise 5 for the Gauss-Seidel method.
In Exercises 7-10, use both the Jacobi method and the Gauss-Seidel method to solve
the indicated linear system of equations. Take x0) = 0, and terminate iteration when
||x*+1) - x falls below 5 x 10-6. Record the number of iterations required to
achieve convergence.
7.
4T1 + Z4
+ 8z2 + 213 + 3z4 23
212 523 9
+ 2I3 + 4z4 = 4

8.
4z1
4
+
432 I3
+ 4z3 = 10
9.
721 312 4

-3Z1 + + -6
333 3
10z4 -

4I5 7
-4z4 + 65 = 2
236 Chapter 3 Systems of Equations

10.
4z1 2r4
+ 472 0
+ 4z3 216 1
4T4 -2
+
D2 Z4 + 4zs |+ Z6 1
I3 I 4Z6 2

11. The linear systems in Exercises 8 and 9 have positive definite, tridiagonal co
efficient matrices. Determine the optimal value of the relaxation paraneter for
the SOR method for each system. Using the corresponding optimal value of w,
solve the systems from Exercises 8 and 9. Take x(0) = 0, and terminate iteration
when |x+) - o falls below 5 x 10-6,
12. For each of the linear systems in Exercises 7-10, generate a plot of the number
of iterations required by the SOR method to achieve convergence as a function
of the relaxation parameter w. Take x0 = 0, and terminate iteration when
|(k+1) x falls below 5 x10-6, Over roughly what range of wvalues
does the SOR method outperform the Gauss-Seidel method? the Jacobi method?
13. Let
2 4 4
A-310 103 35
(a) Write out the iteration matrix Tjac corresponding to the matrix A, and
determine p(T;ac). Wil the Jacobi method converge for any choice of the
initial vector x(0
(b) Write out the iteration matrix Tgs corresponding to the matrix A, and
determine p(Ts). Will the Gass-Seidel method converge for any choice of
the initial vector x(0?
14. Consider the iteration scheme xlk+)) = Tx) +c, and suppose that |T| < 1
for some natural matrix norm. Show that for any x0 e R"

| x - s )- 1.
(Hint: Review the proof of the fixed point iteration convergence theorem in
Section 2.3.)
15. iteration
Let Abe amatrix
strictlyassociated
diagonallywith
dominant matrix and let Tjac be the Jacobi method
A. Show that p(Tjac) < 1. (Hint: Show that
|Tjaclloo<l and use the fact that the spectral radius of a matrix is smaller than
any natural imatrix norm of that matrix.)
16. Suppose that p(T) <1. Show that

(U-T)=)T.
ke0
 Section 3.9 Conjugate Gradient Method 237

17. The variables of interest in an absorption column are the steady-state composi
tion of solute in the liquid, z, on each plate and the steady-state composition of
the solute in the gas, Vi, on each plate. Suppose we have a six plate absorption
column, where the inlet compositions, zo = 0.05 kg solute/kg liquid and y7 =0.3
kg solute/kg inert gas, are known, as are the liquid and gas fiow rates, L = 40.8
kg/min and G = 66.7 kg/min. Further, we will assume that the linear equilib
rium relationship y, = 0.72z; holds. Performing a material balance around an
arbitrary plate, we find that the z; satisfy the system
-88.824 48.024 0 0
40.8 -88.824 48.024 0 0 T2
40.8 -88.824 48.024 0 T3
0 40.8 -88.824 48.024
0 0 40.8 -88.824 48.024
0 0 0 40.8 -88.824
-2.04

0
0
-20.01

Determine the z; using

(a) the Jacobi method;
(b) the Gaus-Seidel method; and
(c) the SOR method with w ranging from 1.l through 1.9 in increments of 0.1.

3.9 ITERATIVE TECHNIQUES FOR LINEAR SYSTEMS: CONJUGATE GRADIENT

METHOD
Not all iterative methods for the solution of linear systems are based on the notion
of a splitting and the conversion to a fixed point problern. For example, there is a
completely separate class of techniques based on the equivalence between the solu
tion of a linear system and the minimization of an associated quadratic functional.
The conjugate gradient method, which is a popular choice for the solution of large
sparse systems, belongs to this latter class of techniques. For an over view of the
many other methods which belong to this class of minimization methods, consult
Ueberhuber 1].
Minimizing a Quadratic Functional and Solving a Linear System
Suppose that A is a symmetric and positive definite n x n matrix. This assumption
willbe made throughout the section. Consider f: R" ’Rdefined by

f*) -;Ax - bTx +c.