Krylov Subspace Methods for Solving Large Unsymmetric Linear Systems

Author(s): Y. Saad
Source: Mathematics of Computation , Jul., 1981, Vol. 37, No. 155 (Jul., 1981), pp. 105-
126
Published by: American Mathematical Society

Stable URL: http://www.jstor.com/stable/2007504

REFERENCES
Linked references are available on JSTOR for this article:
http://www.jstor.com/stable/2007504?seq=1&cid=pdf-
reference#references_tab_contents
You may need to log in to JSTOR to access the linked references.

JSTOR is a not-for-profit service that helps scholars, researchers, and students discover, use, and build upon a wide
range of content in a trusted digital archive. We use information technology and tools to increase productivity and
facilitate new forms of scholarship. For more information about JSTOR, please contact [email protected].

Your use of the JSTOR archive indicates your acceptance of the Terms & Conditions of Use, available at
https://about.jstor.org/terms

American Mathematical Society is collaborating with JSTOR to digitize, preserve and extend
access to Mathematics of Computation

This content downloaded from

103.232.241.147 on Mon, 27 Jan 2025 16:22:05 UTC
All use subject to https://about.jstor.org/terms
MATHEMATICS OF COMPUTATION
VOLUME 37, NUMBER 155
JULY 1981

Krylov Subspace Methods for Solving

Large Unsymmetric Linear Systems

By Y. Saad*

Abstact. Some algorithms based upon a projection process onto the Krylov subspace
Km = Span(ro,Aro, . . .,AmA- rO) are developed, generalizing the method of conjugate
gradients to unsymmetric systems. These methods are extensions of Arnoldi's algorithm for
solving eigenvalue problems. The convergence is analyzed in terms of the distance of the
solution to the subspace Km and some error bounds are established showing, in particular, a
similarity with the conjugate gradient method (for symmetric matrices) when the eigenvalues
are real. Several numerical experiments are described and discussed.

1. Introduction. Few efficient iterative methods have been developed for treating
large nonsymmetric linear systems. Some methods amount to solving the normal
equations A HAx = AHb associated with the system Ax = b or with some other
system derived by a preconditioning technique.
This, unfortunately, is sensitive to the conditioning of A HA which is in general
much worse than that of A. Techniques using Chebyshev iteration (12] do not
suffer from this drawback but require the computation of some eigenvalues of A.
A powerful method for solving symmetric linear systems is provided by the
conjugate gradient algorithm. This method achieves a projection process onto the
Krylov subspace Km = Span(ro, Aro, . .. , A m- 1ro), where ro is the initial residual
vector. Although the process should theoretically produce the exact solution in at
most N steps, it is well known that a satisfactory accuracy is often achieved for
values of m for less than N [15]. Concus and Golub [5] have proposed a generaliza-
tion of the conjugate gradient method which is based upon the splitting of A into
its symmetric and skew-symmetric parts.
The purpose of the present paper is to generalize the conjugate gradient method
regarded as a projection process onto the Krylov subspace K,,. We shall say of a
method realizing such a process that it belongs to the class of Krylov subspace
methods. It will be seen that these methods can be efficient for solving large
nonsymmetric systems.
The next section describes the Krylov subspace methods from a theoretical point
of view. In Section 3 some algorithms are proposed. They are essentially the
extensions of the Arnoldi-like methods for solving large eigenvalue problems
described in [18]. Section 4 deals with the convergence of the Krylov subspace
methods. Finally, some numerical experiments are described in Section 5.

Received October 10, 1980.

1980 Mathematics Subject ClassificationL Primary 65F05.
* On leave from the Laboratoire d'Informatique et de Mathematiques Appliquees de Grenoble;
Visiting Research Associate, University of Illinois, Department of Computer Science, Urbana, Illinois.
Sunnorted in nart by the U. S. Denartment of Ener,v- Grant EY-76-S-02-2383.
C 1981 American Mathematical Society
0025-5718/81/0000-0108/$06.50
105

This content downloaded from

103.232.241.147 on Mon, 27 Jan 2025 16:22:05 UTC
All use subject to https://about.jstor.org/terms
 106 Y. SAAD

2. The Krylov Subspace Methods-Theoretical Aspects.

2.1. General Projection Process-Notations. Consider the linear system
(2.1) Ax-b =0,

where A is a (complex or real) N x N matrix, and let Vm = [vl, . .. , v,,,] be a

system of m lineariy independent vectors in Cv. The projection process onto the
subspace Km = Span(v1, . v. , v,,) seeks an approximation x(m) to the solution of
(2.1) by requiring that

(2.2) ~~~x(m) E- Kms

Ax(m) - b I vj, j = 1, 2, ... ,m
Writing x(m) = Vm y(m), it is immediate that y(m) must satisfy the m x m linear
system

(2.3) V[A Vm - y(m) - V:'b = 0,

where V,' denotes the transpose of the conjugate of Vm: V[ = VTm.
the orthogonal projector onto the subspace Km. Then another formulation of (2.2)
is the following

(2.4) f (m) C Kin,

7Tm(Ax(m) - b) = 0.
It will be assumed for simplicity that b E Km. We shall deno
of s7mA to Km, so that x(m) is the solution in K,,, of the equ
(2.5) Amx - b = 0.

(Note that b E Km, so that rTm

The problem (2.1) is therefore replaced by the m-dimensional problem (2.5). In
order to study the convergence properties of this process, one may express the error
in terms of the distance between the exact solution x* and the subspace Km, that is
in terms of jj(I - -m)x* j; see [8].
Note here that when A is Hermitian definite positive, the convergence is more
easily studied by using the fact that the approximate solution x(m) minimizes the
error function E(x) = (x - x*)HA(x - x*) over all elements x in Km. Unfor-
tunately, this property does not extend to the nonsymmetric case, so it becomes
necessary to make a different approach. Suppose that the exact solution x* is close
to Km, in that 7TmX* is close to x*. Then it is possible to show that x(m) is close to
7TXm (hence to x*) by showing that the residual of 7TmX* for the problem (2.5) is
small. More, precisely,

PROPOSITION 2.1. Let Ym = 117TmA(I - 7t)I. Then the residual of umx* for problem
(2.5) satisfies
(2.6) lb - Am7TmX*|ll < ymll(I - 7m)X*11-

Proof.
b - AmTmX* = b - 7TmA7Tmx* = b - 7ImA[X* - (I -m)x*]
= 7TmA(I - 7Tm)x*-

Observing that (I - '7Tm) is a projector, we can write

ilb - Am7TmX*ll = 117TmA(I - 7m)(I - 7Tm)x*II < Ym|I(I- 7Tm)X*ll,
which completes the proof. OJ

This content downloaded from

103.232.241.147 on Mon, 27 Jan 2025 16:22:05 UTC
All use subject to https://about.jstor.org/terms
 LARGE UNSYMMETRIC LINEAR SYSTEMS 107

As a consequence, we can state the next corollary which gives a bound for
||X* - x(m) .

COROLLARY 2. 1. Let Ym be defined as above and let Km be the norm of the inverse of
Am. Then the error x* - x(m) satisfies

(2.7) 11X* - X(M)II 1 + IV 11( - 7Tm)X*IL

Proof. By Proposition (2.1) and the fact that x(m) - 7Tmx* = A7,(b
we get

(2.8) ||7TM(X* - x(m ))| < YmKmII(I - 7TM)X*II

(remark that 7Tmx(m) = X(m)). Writing

(2.9) X* - X(M) = (I - 7TM)X* + Tm(X* - X(m))

and observing that the two vectors on the right-hand side of (2.9) are orthogonal,
we obtain

IIx* -x(m)112 - ||(I - 7Tm)X*I2 + Ihm(X* X(m))112,

which, in view of (2.8), gives the desired result (2.7). El
The above results show that the error llx(m) - x*jj will be of the same order as
jj(I -7m)x*ll, provided that the approximate problem (2.4) is not badly condi-
tioned.
2.2. Krylov Subspace Methods. Let xo be an initial guess at the solution x* of
(2.1), and let ro be the initial residual rO = b - Axo. If the unknown x is decom-
posed as x = xo + z, then clearly the new unknown z must satisfy
(2.10) Az - ro = 0.
By a Krylov subspace method we shall refer to any method that obtains an
approximation z(m) to problem (2.10) by applying a projection process to the
system (2. 10) onto the Krylov subspace Km = Span[ro, Aro, . . ., A m - 'ro].
We shall assume, throughout, that the vectors ro, Aro, ... , A mn-i,ro are linearly
independent, which means that

(2.11) dim(Km) = m.
If Vm _ [v, ... ., vm] is any basis of Km then
can be expressed as z(m) = Vm y(), where y(m) is the solution of the m x m system

(2.12) V[A Vm * y(m) - V:ro = 0,

and the approximate x(m) of problem (2.1) is related to z(m) by x(m) = x0 + z(
If z* = A -ro denotes the exact solution of the system (2.10), then we no

(2.13) X* - X(m) = Z* - Z(M)

which means that x(m) and z(m) admit the same error vector for (2.1) and (2.10),
respectively.

3. Practical Methods. Some algorithms based upon the Krylov subspace methods
described above will now be presented. We first propose an adaptation of Arnoldi's
method [1], [18] to the solution of systems of linear equations. The algorithm
constructs an orthonormal basis Vm = [v,, ... ., vm] of Km such that V'TA Vm ha
Hessenberg form. An iterative version of this method is also given so as to avoid

This content downloaded from

103.232.241.147 on Mon, 27 Jan 2025 16:22:05 UTC
All use subject to https://about.jstor.org/terms
 108 Y. SAAD

the storage of too large arrays in memory. Then another class of algorithms is
derived from the incomplete orthogonalization method described in [18].
3.1. The Method of Arnoldi. Arnoldi's algorithm builds an orthonormal basis
vl, ... , v,,, of Km = Span[ro, Aro, .. 'Am- A ro] by the recurrence
k

(3.1) hk+l,kVk+I = Avk - E hikv

i=l1

starting with v1 = ro/ lIlroll and choosing hik i = 1

Vk+ 1Iv1, . .. , vk and IIvk+ I = 1. In exact arithm
follows.
Algorithm 3.1.

1. Compute rO = b - Axo and takevI := ro/lIroII.

2. For k := I until m do

(3.2) w := Avk - hikvi with hik = (Avk, vi),

i=l

(3.3) hk+ 1,k I= || wIl, Vk+ 1 = W/hk+ l,k

See [18] for some remarks on the practical realization of this algorithm. It is easily
seen that [vI, V2, ... I Vm] is an orthonormal basis of Km and that the matrix
V:'A Vm is the Hessenberg matrix Hm whose nonzero elements are the h. defined by
(3.2) and (3.3). As a consequence, the vector V,:rO in (2.7) is equal to ,B. VHvj =
fBeI, where ,B = IIroll. Thus, the system (2.7) becomes
(3.4) Hm .y(m) =,Bel

and the approximate solution x(m),

Z(m) where

(3.5) Z(m) = f8VmH-lel.

The following estimate for the residual norm

stopping criterion

(3.6) lb - Ax(m)Il = hm+i1mIem:yv(m)I.

Equality (3.6) follows immediately from the relation

AVm = VmHm + hm+l,mVm+le:

which can be derived from the algorithm and from equality (2.8).
An interesting practical method would be to generate the vectors vk and the
matrix Hk, k = 1, 2, ... , m, . . ., to compute periodically the estimate
hm+ ImIe:y(m)I of the norm of the residual and to stop as soon as this i
enough. As was suggested in [15] for the symmetric case, there are various ways of
updating Ie4Hy(m)l without even actually computing the vector y(m). Let us give a few
indications about the problem of computing the estimation je,y(m)j, since it will
appear in several parts along the paper. Parlett [15] suggests utilizing a recurrence
relation proposed by Paige and Saunders [14], which is based upon the LQ
factorization of Hm.

This content downloaded from

103.232.241.147 on Mon, 27 Jan 2025 16:22:05 UTC
All use subject to https://about.jstor.org/terms
 LARGE UNSYMMETRIC LINEAR SYSTEMS 109

Another interesting possibility is to perform the more economical factorization

provided by the Gaussian elimination with partial pivoting on the matrix Hj. Th
factorization of Hj can be easily performed by using the information at the
previous step. Supposing that no pivoting has been necessary for steps 1 through
j -1, and writing the LU factorization of Hj, Hj = LU, it can be easily seen that
pj = 1+ I e, yI e is simply

pj = hj+ 'i (p t 11 u

where the 1j, i = 1, . . ., j-1, are the successive pivots. More generally, it can b
shown that when no pivoting has been necessary at steps i, i E I, where I c
{1, 2, ... ., j- 1), then pj becomes

PI = hk+. Ij(,ll)/ u1.

This means that pj can be updated at each step at a negligible cost. Finally, a
is decided that the estimate of the residual norm is small enough, the final
factorization of Hm will be used to fully solve the system (3.4). The Gaussian
elimination with partial pivoting gives satisfactory results in general, but one might
as well use a more stable decomposition, as the LQ decomposition in [14], [15],
although at a high cost.
As m increases, the process of computing the vi becomes, unfortunately, intoler-
ably expensive and core memory demanding. To remedy this, one can use the
algorithm in an iterative way, as is described next.
3.2. Iterative Arnoldi Method. Due to core memory capacities, the number m of
steps in Algorithm 3.1 is inevitably limited. After having computed the approxi-
mate solution x(m) with the maximum number of steps allowed, one may find that
the accuracy is still unsatisfactory. This naturally raises the question of how to
improve the accuracy of x(m). The simplest idea is to restart the algorithm with x0
replaced by the approximation x(m) obtained. The idea is similar to that of the m
step steepest descent in the symmetric case; see [6]. One can restart as many times
as necessary to ensure satisfactory accuracy. We now give a more detailed descrip-
tion of this iterative version. Let us start with an initial guess x0 and form
ro= b - Axo. Then construct Hm and Vm by algorithm (3.1) and compute the
approximate solution x(m) = x0 + z(m). The estimation (3.6) can- be used to de-
termine whether the process must be stopped or restarted. Suppose a restart is
necessary. Then take x, = x0 + z(m) and compute r = b - Ax,. (Remark that r1 is
also equal to the residual ro - Az(m).) Construct again Vm and Hm starting with
= r1/ llr1l in Algorithm 3.1. Then an approximate solution z4m) to the equation
Az = rI is obtained yielding the new approximation x2 = xI + Z2m) to the solution
x*, and so forth.
At the sth iteration, the approximate solution x, is equal to xo + z(m)
+ * + *z4). Thus, the algorithm can be formulated as follows. (The subscript
(m) is dropped for simplifications.)

This content downloaded from

103.232.241.147 on Mon, 27 Jan 2025 16:22:05 UTC
All use subject to https://about.jstor.org/terms
 110 Y. SAAD

Algorithm 3.2.
1. Start. Choose m and x0; ro b - Axo.
2. Fors:=O,1,..., do
*Compute vI, v2, .. v, m and Hm by Algorithm 3.
rS/( , := llrsll)
*Solve the system Hm y = y eI
*Zs+ 1 Vm Y
*Xs+1= Xs + Zs+1
rs+l := rs- Azs+1
* If hm+ 1,m Ie,I < c, stop else continue.
3.3. Incomplete Orthogonalization Methods.
3.3.1. The construction of the vectors v,, . .. , vm by Algorithm 3.1 amounts to
orthogonalizing the vectors Avk against all previous vectors v1, . .. , Vk. This is
costly and some numerical observations suggest to orthogonalize Avk against the
precedingp + 1 vectors rather than all; see [18].

The system produced is such that (vi, vj) = 8, for i, j satisfying li - jj < p.
Algorithm 3.3.

1. Choose p and m such that p < m; compute ro := b - Axo and vI := ro/ IIroII.
2. Forj:= 1,2,...,mdo
io max(l, -p + 1)
w := vj - j.ohii vi with
(3.7) huj: (Avj, vi),
(3.8) vj+ I w/(h+1j := IIwll).
Under the assumption (2.11), this algorithm will not stop before the mth step and
will produce a system of vectors v,, . .. , v, locally orthogonal and a (banded)
Hessenberg matrix of the form

Hm= 0]

whose nonzero elements are computed from (3.7) and (3.8). The generalized
Lanczos approximation z(m) must satisfy the equations

(3.9) V,AV,,my(m) - V:rO = 0, z(m) - V,,y(m).

In the present case, however, the matrix V,HA Vm does not have any particular
structure as before, so we need to transform (3.9) into a simpler problem.
Let us set Hm = ( VVm)-l V,HA Vm. Note that this is just the matricial representa-
tion of the linear operator Am = 7TmAi,, (see Subsection 2.1) in the basis
{VI, v2, .. V,m}. It was shown in [18] that Hm differs from Hm only in its last
column. More precisely,

THEOREM 3. 1. Let Sm = hm + I,m(VVm)-1 Vmm[Vm + I Then

H
(3.10) Hm = Hm + smem;.

This content downloaded from

103.232.241.147 on Mon, 27 Jan 2025 16:22:05 UTC
All use subject to https://about.jstor.org/terms
 LARGE UNSYMMETRIC LINEAR SYSTEMS 111

Proof. From Algorithm 3.3 we get the basic equation

AVm = VmHm + hm+i,mVm+le:

which yields (3.10) on multiplying by (V:'Vm)-'V:. f1

Multiplying (3.9) by (Vp:Vm)-l gives the equivalent equation

Hny )- ( V[Vm)' Vm ro = 0.
Observing that (V:,Vm)-'Vp[ro = 13e, where f = lroll, we obtain the system
(3.11) H)(m)- f3el = 0.

If we set 9(m) = 3H^'lei and9(m) = 8iH,-

formula [7], these two vectors are related by

(3.12) Ym = Ym-aHrn'Sm,

where a = e:.Y(m)/(i + e:'H,mlsm).

On the practical side, the only difficulty lies in the computation of the corrective
column Sm. Note that Sm = hm+ I,Vm+vm,+I and that Sm is the solution of the least
squares problem (see [19])

(3.13) min| Vms - hm+b,mvm+lll,

for which many efficient algorithms are available; see [3], [13]. It should be added
that only a moderate accuracy is needed in practice, so the bidiagonalization
algorithm BIDIAG described in [13] is suitable for solving (3.13) with moderate
accuracy. We can now give an algorithm based upon all the above observations.
Algorithm 3.4. Incomplete Orthogonalization with Correction.
Start. Choose two integers p and m with p < m. Compute rO := b - Axo, B
Ilrlrl; VI := ro/lB .
Iterate. Comment compute Hm and v, ... . vm.
Forj = 1, 2, . . ., m do
io max(l,j - p)
w:=Avj-i .(h. := (Av, vi)) xv,
vj+I := wI(hj+l,j := liwll)
Correct:
1. Compute least squares solution Sm of (3.13).
2. Compute

Ym := f3J,'len
x := IIlsM

a := e,ym/ (1 + e'x)

Ym =Ym - ax

3. Form the approximate solution

x(M) = x0 + Vm *m.
We shall now give some additional practical details.
1. If necessary, the vectors vI, v2, . . ., Vm may be stored in auxiliary memory,
one by one as soon as they are computed. Only the p vectors vj, vj . . ., +1
must be kept in main memory for more efficiency.

This content downloaded from

103.232.241.147 on Mon, 27 Jan 2025 16:22:05 UTC
All use subject to https://about.jstor.org/terms
 112 Y. SAAD

2. The storage of Hm now requires only the storage of (p + 1) x m elements

instead of the previous M2.
3. For the choice of the integer p, we should first point out that p is limited by
the available core memory. In theory, the larger p, the better. If p is large, the
system (v, . . . , v,,,) will, in practice, be close to orthogonality, and the solution
the least squares problem (3.13) in step correct becomes easier [at the limit
p = m, then the solution is just hm+ I'm Vn'vm+ I = 01. But in that case the computa-
tions in the step iterate are more expensive. If p is too small, on the other hand, it is
very likely that the problem (3.13) will become difficult to solve (if not impossible
numerically) as the vectors (v1, . .. , v,,) will become nearly linearly dependent.
Note that this depends also upon m. When m = p, the system is orthonormal, and
as m increases it is observed that the system departs from orthogonality, in a slow
manner at the beginning. All these observations suggest thatp must first be chosen
according to the main memory capacity and some arbitrary limitation p < Pma.
Afterwards, a maximum number of steps mnmax should be fixed. Then a test must be
included at the end of the step iterate in order to shift to the correction step as soon
as the system {v1, v2 .. . , V 1} is suspected to be too far from orthonormal, as for
example

if I(vj+ 1, v1)I > -1 goto correct,

where -q is a certain tolerance. The heuristic criterion given above is not the best.
4. When the matrix A is symmetric, then, by takingp = 2, we obtain a version of
the conjugate gradient method which is known to be equivalent to the Lanczos
algorithm; see [14]. In that case the vectors v1, .v. , Vm are theoretically orthogonal
Suppose now that A is nearly symmetric and take p = 2 again. By a continuity
argument, it is clear that the system (vl, . .. , vm) will be nearly orthonormal,
making the choice p = 2 optimal in a certain sense. This suggests that, when it is
known that A is close to a symmetric matrix, p could be taken small (or even
p = 2). However, it is not easy to give a rigorous meaning to the notion of nearly
symmetric, and it is even more difficult to monitor automatically the choice of the
parameterp.
3.3.2. In the following we develop another algorithm which is, in particular, more
appropriate for the cases of almost symmetric matrices. As pointed out above, the
correction step can be expensive and one may ask whether an acceptable accuracy
could be achieved by ignoring the corrective step and replacing the approximate
solution x(m) = x0 + V,,;9m by

(3.14) x(M) = xo + Vjm.

Tlhe answer is yes, provided that Vm+ I is n
A H
writing Hm = Hm- smemS, we can derive the fol

hm+i,mep:[jm .i
(3.15) Ym = 9m + 1 A ISMI,H
It is remarkable that, by (3.6), the term hm+ me
lro -Az(m)II, except for the sign, and hence it
{ v1, .. , vm+ 1) is nearly orthonormal, then VVM m
in general. This shows that, in general, the second term on the right-hand side of

This content downloaded from

103.232.241.147 on Mon, 27 Jan 2025 16:22:05 UTC
All use subject to https://about.jstor.org/terms
 LARGE UNSYMMETRIC LINEAR SYSTEMS 113

(3.15) can be neglected (in comparison with 9m) as long as Vm+i remains nearly
orthonormal. This fact is confirmed by the experiments, and it is observed that the
residual norms behave in the same manner as the residual norms obtained for the
incomplete orthogonalization method applied to the eigenvalue problem; see [18,
Section 4.2].

The residual norms IIro - Aim decrease rapidly until a certain step and then
start oscillating and decreasing more slowly. This suggests restarting immediately
after a residual norm is larger than the previous one. Here, again, the formula (3.6)
remains very useful for estimating the residual norm. This leads to the following
algorithm.
Algorithm 3.5. Incomplete Orthogonalization Without Correction.

Start. x := xo; r := b - Axo; ,B := IIII; v1 := r/fl;

Iterate. Forj = 1, 2, . . ., mma do
1. Compute

hj+ l jj+ I := Avj - hyi,

i -io

where io and the h.'s are as in Algorithm 3.4.

2. Update the factoriza'tion of Hj and the estimate pj of the residual norm
(see Subsection 3.1).
3. Test for convergence performed every q steps only (e.g., every q = 5 steps).
a. If pj < e goto restart.
b. If pj > Pj-q goto restart; otherwise take m j and continue.
Restart:

z(m) :- fVmHHel

i - _AE(m)
r := r- Ai(m)

If j3 < e stop else goto iterate.

The numerical experiments (Section 5) will reveal that this last algorithm is to be
preferred to the iterative Arnoldi algorithm and to the incomplete orthogonaliza-
tion method with correction. Surprisingly, it is often the case that no restart is
necessary, even for matrices that are not nearly symmetric.
We shall conclude this section by a remark concerning the application of
preconditioning techniques to the algorithms described above. Suppose that we can
find a matrix M, for which linear systems are easily solvable and such that M-IA is
closer to the identity than A. In this case it is advantageous, in general, to replace
the system Ax = b by the new system M-'Ax = M-lb before applying one of the
previous methods. There are two reasons for this. The first is that the rate of
convergence of the second system will, in general, be higher than that of the first
because the spectrum will be included in a disk with center one and with small
radius, and the next section will show that in that case the smaller the radius, the
higher the rate of convergence. The second is that M-IA, which is close to the

This content downloaded from

103.232.241.147 on Mon, 27 Jan 2025 16:22:05 UTC
All use subject to https://about.jstor.org/terms
 114 Y. SAAD

identity matrix, is clearly close to a symmetric matrix (the Identity), so that the
application of incomplete orthogonalization without correction is most effective; cf.
Subsection (5.5).

4. Rates of Convergence for the Krylov Subspace Methods.

4.1. Introduction. We shall now consider the problem of the convergence of the
approximate x(m) toward the exact solution x*. We first point out that the
convergence is achieved in at most N steps where N is the dimension of A. (This is
immediate from the fact that KN is the whole subspace CN and from the definition
2.2.) Therefore, the problem is not to show the convergence but rather to establish
theoretical error bounds showing that one can obtain a satisfactory accuracy for
values of m much less than the dimension N, which is supposed to be very large.
Another way of stating the problem is to suppose that A is an operator on a Hilbert
space (N = om) such that the convergence, the rate of convergence..., of the
infinite sequence x(m) can be discussed. We shall not, however, adopt this extension
in the present paper.
In view of relation (2.13), it is equivalent to study either the convergence of x(M)
to x* or the convergence of Z(m) to z*. In addition, Corollary 2.1 shows that the
convergence can be studied in terms of 11(I - irm)z* 1, where 7Tm is the orthogonal
projection onto the Krylov subspace Km = Span[ro, Aro, . . ., A m-ro]. Let us de-
note by Pk the space of polynomials of degree not exceeding k. Then, a useful
expression for the distance 11(I - -m)Z*II can be derived by remarking that Km is
nothing but the subspace of CN constituted by all the elements q(A)ro, where q
belongs to Pm_l1

PROPOSITION 4.1. The distance 11(I - 7m)z*ll between z* and the Krylov subspace
Km satisfies

(4.1) 11(I - 7Tm)Z*Il- min IIp(A)z*11.

p(O) -

Proof. The following equalities are easy to show

-m)z*II = minm Iz* - zIl = 1Z* - q(A)roll

= mi JJz* - q(A)Az*j = mi 11(1 - Aq(A))z*jj

= min J[p(A)z*11.
pE- Pm
p(O)= 1

In order to obtain an upper bound for (4.1), we shall assume that A admits N
eigenvectors 4,. 02, ... I ON of norm one, associated with the eigenvalues
XI, XN. Then the solution z* can be expressed as
N

Z*= I
i-i

and we can formulate the next theorem.

This content downloaded from

103.232.241.147 on Mon, 27 Jan 2025 16:22:05 UTC
All use subject to https://about.jstor.org/terms
 LARGE UNSYMMETRIC LINEAR SYSTEMS 115

THEOREM 4.1. Set a = a IaA, where the at, are the components of the solution z*
in the eigenbasis of A.
Then

(4.2) 11(1 - 7Tm)Z*I| < a mm max lP(y)I-

p(O) -1

Proof. Letp E P,m withp(O) = 1. Then

N N

l/p(A)z*lj = Fp(
N N N

l |aip(i\)O

Therefore, for any polynomial of degree not exceeding m such that p(O) = 1, we
have

(4.3) ||p(A)z*|| a max lp(Xj)l

Hence,

min lp(A)z*Il < a min max (Xj)

p EPE1 p PE Pm.j -I.
p(O)= I p(O) -

which, by equality (4.1), completes the proof. Q

We point out here that from classical results it can be shown that the polynomial
realizing the minimum in (4.2) exists and is unique provided that m < N; see [11].
We should also add that there is unfortunately no upper bound for a.
We shall set, throughout,

(4.4) E^=mn j max p(,,

p(O) - I

so that inequality (4.2) simplifies to

(4.5) (- 7m)Z*| < ae(m)

and the result (2.7) becomes

iux* - x(M)I = llz* - z(m)lI < a 1 + K E(m)

We, therefore, need to show that the sequence E(m) decreases rapidly to zero. Note
that e(N) = 0 which shows again that the process will give the exact solution in at
most N steps. The rate of convergence of the sequence e(m) to zero provides a
bound for the actual rate of convergence. Estimating E(M) is, unfortunately, a
difficult problem in general. The number E(m) is the degree of best approximation of
the zero function by polynomial of degree m satisfying the constraintp(O) = 1, over
the set X,, A2, .. ., XN; see [11].
4.2. An Exact Expression for E(m). The following theorem gives an expression for
e(m) in terms of m + 1 eigenvalues of A.

This content downloaded from

103.232.241.147 on Mon, 27 Jan 2025 16:22:05 UTC
All use subject to https://about.jstor.org/terms
 116 Y. SAAD

THEOREM 4.2. Let m < N - 1. Then there exist m + 1 eigenvalues which, without

ambiguity, can be labelled A1, A2, I ., +I such that

m+l m+l A --
(4.6) e(m) E II | -

We omit the proof of this equality. An analogous result will be proved in a

forthcoming paper dealing with the convergence of Arnoldi-like methods for
computing eigenelements.

The result does not specify which are the eigenvalues A1, ... ., XM+,I but it stil
gives an interesting indication. If the origin is well separated from the spectrum,
then e(m) is likely to be very small. Indeed, if A1 is, for example, the eigenvalue the
closest to zero, among those eigenvalues involved in the theorem, then, in general,
we shall have IAkI > IAX - AkI, k = 1, . . . , N, as seen in Figure 1. Therefore,
m+1 IAk I

k-2 AXk - Il
and it is seen from (4.6) that e(m) will be small. There are particular distributions of
the eigenvalues where e(m) is known exactly (for m = N - 1). But, in general, the
result (4.6) is not useful for giving an estimation of the rate of convergence. Upper
bounds for e(m) must be established for that purpose.

Im(X)

___\__ Re (A)

? k'

FIGuRE 1
4.3. Bounds for e(m). In the real case one usually obtains bounds for e(m) by
majorizing the discrete norm maxj ,NIP(?)I by the continuous norm max,.,iIp(X)I,
where I is an interval (or the union of two intervals) containing the eigenvalues X.
and not zero.
In the complex case, however, one encounters the difficulty of choosing an
adequate continuum containing all the eigenvalues and not zero. An infinity of
choices are possible, but, except for some particular shapes such as circles,
ellipses..., there is no simple expression for the minimax quantity

minpEPm ;P(O) I maxz e D I P(Z)I

We first deal with the simplest case where all the eigenvalues of A are real and
positive. The next case to consider is, naturally, the case where the eigenvalues are
almost real. The general case will be considered in Subsections 4.3.3 and 4.3.4.

This content downloaded from

103.232.241.147 on Mon, 27 Jan 2025 16:22:05 UTC
All use subject to https://about.jstor.org/terms
 LARGE UNSYMMETRIC LINEAR SYSTEMS 117

4.3.1. Case of a Purely Real Spectrum.

THEOREM 4.3. Suppose that all the eigenvalues of A are real and positive and let

X;\. and Xm. be the smallest and the largest of them. Then
(4.7) I(1 - TM)Z*II < a/Tm(y)
where a is as before, y = (Ama + .mjn)/(.mac- Xi), and wh
Chebyshev polynomial of degree m of the first kind.

This result is an immediate application of a well-known bound for (4.4) when the
At are real [2]. It is also possible to establish some results when the eigenvalues are
known to lie in two or more intervals; see [21, [101.
Inequality (4.7) shows that the generalized Lanczos method converges at least as

rapidly as [Tm(y)f-- (y + Ay2 - 1 ) such that the rate of convergence

bounded by y + v-1 .
Finally, note that similar results can easily be obtained if all the eigenvalues are
purely imaginary or if they lie on a straight line of C, containing the origin.
4.3.2. Almost Purely Real Spectra. In the following we shall assume that the
spectrum lies inside a certain ellipse which has center c on the real line and foci
c + e, c - e where e is the eccentricity. Furthermore, we shall assume that the
origin is not inside that ellipse (see Figure 2). Let us denote by E the closed domain
bounded by the ellipse defined above. Consider the variable transform z'=
(c - z)/e; then E(M) satisfies the inequality

(4.8) pe m zemaEx
p(c/e)= I

where the domain E' is bounded by the ellipse centered at origin with eccentricity
one and major semiaxis a/e. It was shown by Clayton [4] that the above minimax
is realized for the polynomial Tm(z')/ Tm(c/e).

Im(z)

e DI

a |rV_

Re (z)

FIGURE 2

This content downloaded from

103.232.241.147 on Mon, 27 Jan 2025 16:22:05 UTC
All use subject to https://about.jstor.org/terms
 118 Y. SAAD

THEOREM 4.4. Assume that the eigenvalues of A lie within an ellipse with center c
on the real axis, foci c + e, c - e, and with major semiaxis a. Suppose that the origin
is not inside this ellipse. Then

(4.9) E(m) <Tm (a/e)

I Tm(c/e)l
In view of (4.10), this inequality is a simple corollary of Clayton's result. Since
the proof is tedious, we shall give a direct proof of (4.9) and bypass Clayton's
result.
Proof. Considering the particular polynomial Tm(z')/ Tm(c/e), we get from (4.8)

(4.10) (m) < max Tm(c/e)

By the maximum principle, the maximum on the right-hand side is realized for z'
belonging to the boundary aE' of the ellipse E' centered at the origin and having
major semiaxis a/e and eccentricity one. Thus, (4.2) becomes

(4.11) E(m) < maix ITm(Z')I.

ITm(c/e)I z'E8-E'
Consider now the transform u: w *->z' = (w + l/w). It is known [11], [17] that
when w belongs to the circle Cp, centered at the origin and having radius p, z' will
belong to the ellipse aEp having eccentricity one and major semiaxis (p + p-1)/2.
We may take p = a/e + (a/e)2-1 such that aEp is just ME'. Tm(z) can be
defined by Tm(z) = ch(m u), where u and z are related by ch(u) = z. Setting
eu = w, we see that another definition for Tm(z) is Tm(z) = (wm + w"')/2, where
w and z are related by (w + w')/2 = z. Hence,

nmax I Tm(Z')| = max - Iwm + w n = max - meimO + pineiinG.

zle W ~ w EECp 2 G E=[0,27TJ]2'

It is easily seen that the above maximum is just

1(mm)1[(~a (a)2 )in( (a) )m]

2(pM + p m) = _ ( + )- + ( a -\-
a
mteJ

which completes the proof. L

The upper bound Tm(a/e)/ Tm(Ic/al) for ei(m) is asymptotically equivalent to
I
ale +V (ale)2-
Iclcel +\0ce2
so that an up er bound for the asymptotic rate of convergence is given by

(4.12) Idc + -
a + a2 e2

This content downloaded from

103.232.241.147 on Mon, 27 Jan 2025 16:22:05 UTC
All use subject to https://about.jstor.org/terms
 LARGE UNSYMMETRIC LINEAR SYSTEMS 119

When the eigenvalues are all real, then the ellipse degenerates to the interval
[A1, AN], and we shall have e = a = (AN - A1)/2, c = (A1 + AN)/2 such that T will
become y + y2 - 1 with y = (AN + Al)/(AN - A1). This means that the result
(2.17) coincides with that of Corollary 2.1 when the spectrum lies on the real line.
Consider now the family of all ellipses having center c and major semiaxis a, and
let the eccentricity decrease from a to zero. Then the ellipse will pass from the
interval (c - a, c + a) to the circle with center c and radius a. It is easily seen
that the bound (4.12) for the rate of convergence will decrease from

Tmax = Ic/al + V(c/a)2 -_1 to Tmin = Ic/al. Therefore, we may asser

convergence is likely to be better if the eigenvalues are close to the real line and
that, when the spectrum has a circular shape, the convergence is likely to be slower.
Note that the comparison is made for the same relative separation Ic/al from the
origin. The above comments are confirmed by a numerical example in Subsection
5.1.
Before considering the more general case where the ellipse containing the
spectrum does not stretch along the real axis, let us point out that inequality (4.9)
cannot be improved, as Clayton's result shows. By this we mean that if one replaces
the discrete set {1, . .. , XN) by the set of all points contained in an ellipse of the
form described in Figure 2, one cannot find a better inequality than (4.9).
4.2.3. Spectrum Contained in an Ellipse. If the spectrum lies inside an ellipse with
center c and foci c + e, c - e, where now both c and e are complex, it is easily
seen that the proof of Theorem 4.4 is still valid. Therefore, we can establish that

(4.13) E(m) < I Tm(a/e)I

IlTm(c/e)l
where c, e are the center and the "eccentricity" and are complex, while a, the
(complex) major semiaxis, is such that c + a and c - a are the coordinates of the
two points of the ellipse situated on the major semiaxis. Note that a/e is real while
c/e is not. The interpretation of (4.13) will, therefore, not be easy in general. It can
be shown, however, that the right-hand side of (4.13) converges to zero as m -o cc;
see [12]. The next subsection gives a result which is weaker, in general, but easier to
interpret.
4.3.4. Spectrum Contained in a Circle. In this subsection we shall assume that the
spectrum lies in a certain domain bounded by a circle having center c and radius a.
Furthermore, let us assume that the origin lies outside the circle (cf. Figure 3). Then
we have

THEOREM 4.4. Suppose that there exists a disk D(c, a), with center c and radius a,
that contains all the eigenvalues of A and not the origin. Then

(4.14) E(M) a

Proof. Consider the particular polynomial p(z) = [(c - z)/c]m. p has degree m
and satisfiesp(O) = 1. Hence, by (2.13),

E(m) <j=I,N
max >()j< c
|-Aic
| < E| a

This content downloaded from

103.232.241.147 on Mon, 27 Jan 2025 16:22:05 UTC
All use subject to https://about.jstor.org/terms
 120 Y. SAAD

Im(z)

i~~~~~~ a

Re (z)

FIGuRE 3
The coefficient Ia/cl in (2.21) is smaller than one, and one can even choose an
"optimal" circle for which la/cl is the least. The optimal center j should minimize
max N,, I(c - Xj)/cI over all complex c, c :# 0, and the optimal radius J is
simply max - .NIc- 1 The inequality (2.21) is the best bound possible for e(m)
that can be obtained by replacing the discrete set (X1, . .. , XN} by the disk D(c, a)
in the formula (2.13). This is due to the next theorem, proved by Zarantonello in
[22].

THEOREM 2.3. The polynomial ((c - z)/c)m is the polynomial of degree m having
least uniform norm over the disk D(c, a) when a < Ici. Furthermore
a M
min max = -|
p E Pm z E D(c,a) C
p(O)=l

5. Numerical Experiments. The experiments described in Subsections 5.1 to 5.4

have been performed on the Prime 650 computer of the Department of Computer
Science at the University of Illinois at Urbana-Champaign. The computations have
been made in double precision, using a 48-digit mantissa.
5.1. The purpose of this first experiment is to illustrate the comments of
Subsection 4.3.2 on the convergence properties in the case of complex eigenvalues.
Let us consider the block-diagonal matrix A, whose diagonal blocks are 2 x 2 and
have the form

Dk [ k[ke k]
-k d] k = 1,29...,.n.

The dk and ek are chosen in such a way that the eigenvalues Xk = dk ? iek of A lie
on the ellipse having center c = 1 and major semiaxis a = 0.8. The eccentricity e
varies from e = 0 to e = 0.8. The real parts dk of the eigenvalues are uniformly
distributed on the interval [c - a, c + a]. In other words

dk = 0.2+ k-i

ek = (a2-e2)2/2[ -(k2)]", k = I, 2, . ..., n,

where c = 1; a = 0.8; 0 < e S 0.8. The number of blocks is n = 40,so that A has
dimension N = 80.

This content downloaded from

103.232.241.147 on Mon, 27 Jan 2025 16:22:05 UTC
All use subject to https://about.jstor.org/terms
 LARGE UNSYMMETRIC LINEAR SYSTEMS 121

We compare, for different values of e, the estimated logarithmic rates of

convergence Pest = Log(T), where T is given by (4.12), with the "actual" logarithmic
rates - (l/m)Log(llx* - x(m)II), where x* and x(M) are the exact and the ap-
proximate solutions, respectively. The method used was Arnoldi's algorithm de-
scribed in Subsection 3.1. The right-hand side b of the system Ax = b was the
vector b = Af where f = (1, 1, . . ., 1)T so the solution is equal to f. The starting
vector xo was set to zero. The next table gives the results obtained when m = 30 for
various values of e.

TABLE 1

e 11 x* - x(m)II Pact Pest

0.00 2.68 x I0-V 0.199 0.223
0.10 2.38 x 10-3 0.201 0.224
0.20 2.11 x l0-3 0.205 0.228
0.30 1.69 x 1i-3 0.212 0.237
0.40 1.18 x O-3 0.225 0.250
0.50 6.71 x 10-4 0.243 0.270
0.60 2.62 x 104 0.275 0.303
0.70 4.22 x 1i-' 0.335 0.367
0.75 6.40 x 10- 0.398 0.432
0.79 1.62 x 10-7 0.521 0.555
0.80 1.55 x 1010 0.753 0.693

Note that in passing from e = 0.79 to e = 0.80 the spectrum of the matrix A
becomes purely real and consists in 40 double eigenvalues, which explains the jump
in the actual rate of convergence.
The values Pact and Pest of Table 1 are plotted in Figure 4.

R
A/
T
E
s

F .

C /I
0
NNJ /

R . 4 -
E
N

0. 1 .
0. .4 .
.2 .8

E C C E N T R I C I T Y
FIGURE 4

This content downloaded from

103.232.241.147 on Mon, 27 Jan 2025 16:22:05 UTC
All use subject to https://about.jstor.org/terms
 122 Y. SAAD

5.2. We shall compare, in the following experiment, the method of conjugate

gradients applied to the problem A HAx = A Hb with the iterative Arnoldi algo-
rithm. Consider the block-tridiagonal matrices

A = -I * * I] with a

anda = -1 + 8; b =- 1-.
L

00
F

A -4
L

0. 100. 200,
50. 150.

N U M B E R 0 F S T E P S

FIGuRE 5

Conjugate gradients for A TAx = A Tb (upper curve) and

iterative Arnoldi method.
m = 10 middle curve, m = 20 lower curve

These matrices come from a discretization of partial differential equations

involving a nonselfadjoint operator; see [12], [18]. When 8 is small, the matrix A is
almost symmetric. The conjugate gradient algorithm was run for the following
case: 8 = 0.01, B has dimension 10, and A has dimension 200. The right-hand side
b was set to Af, wheref = [1, . . ., IfT, and the initial vector was chosen randomly.
We have compared the results with those obtained with the iterative Arnoldi
method using 10 steps per iteration (m = 10) and 20 steps per iteration. The initia
vector, as well as the right-hand side, is the same as above. Figure 5 shows, in a
logarithmic scale, the evolution of the error norms obtained for the same total
number of steps. Notice that although the total number of steps required to achieve
convergence is smaller with Arnoldi's method, the total amount of work required in
this example is in favor of the conjugate gradient method because the cost of
computing Av is not high. The method of Arnoldi will be appropriate whenever the

This content downloaded from

103.232.241.147 on Mon, 27 Jan 2025 16:22:05 UTC
All use subject to https://about.jstor.org/terms
 LARGE UNSYMMETRIC LINEAR SYSTEMS 123

cost of computing Av dominates all the other costs in each step, but this will not
always be the case. Figure 5 also shows that, when the matrix by vector multiplica-
tion is costly, it may be advantageous to choose m as large as possible.
5.3. In the previous example, the matrix treated is nearly symmetric and so the
use of the incomplete orthogonalization method without correction is more suit-
able. Takingp = 2, and starting with the same initial vector as in the experiment of
5.2, yielded a rapidly decreasing sequence of residual norm estimates. No restart
was necessary, and convergence occurred after 90 steps with a residual norm equal
to 4.6 x 10-1". Clearly, the amount of work required here is far less than that
required by either of the methods compared in 5.2.
5.4. We shall now compare the incomplete orthogonalization methods with and
without corrective step on the 100 x 100 block-tridiagonal matrix A of Subsection
5.2, obtained by taking 8 = 0.2. In a first test an iterative method based upon the
incomplete orthogonalization algorithm with correction (Algorithm 3.4) was tried.
As soon as the estimate f8hm+ I,mIe:NymI of the residual norm stops decreasing or
when the number of steps reaches the maximum number of steps allowed, mm, =
40, the algorithm is halted, a corrective step is taken, and the algorithm is either
stopped (if the residual norm is small enough) or restarted. For the present
example, the algorithm halted first at m = 20 and gave a residual norm of 1.8.
After the correction step, the residual norm dropped down to 6.2 x 10-3. In the
second iteration the algorithm halted at m = mmax = 40 and gave the residual
norms 9.6 x 10-5 before the correction and 1.14 x 106 after.
It is important to mention that, here, the corrective steps necessitate the use of
the bidiagonalization algorithm to compute the corrective column Sm, which is
usually very expensive.
The results obtained with the incomplete orthogonalization method without
correction are by far superior from the point of view of the run times. Algorithm
3.5 was first tested with p = 2. At the first iteration the residual norms decreased
from 7.6 to 1.8 at the 15th step and then a restart was made. At the second iteration
the residual norms kept decreasing rapidly to 2.1 x 106 at the 60th step. The test
with p = 4 yielded a steadily decreasing sequence of residual norm estimates and
therefore no restart has been necessary. The final residual norm obtained at
m = 60 was 7.88 x 10-7.
5.5. Finally, we shall describe an experiment on a more difficult example
considered in [19]. The runs reported below have been made on a CDC CYBER
175 computer using a word of 60 bits and a mantissa of 48 bits (single precision).
The problem Ax = b treated has dimension N = 1000 and the nonzero part of A
consists of 7 diagonals

(The nonzero elements of the first row and first column of A are Al,, A12,
A1,10, A1,,1, A21, Aj,0, A1O,O.) The problem originated from the simulation of

This content downloaded from

103.232.241.147 on Mon, 27 Jan 2025 16:22:05 UTC
All use subject to https://about.jstor.org/terms
 124 Y. SAAD

a reservoir and is known to be badly conditioned. It has been solved in (18] by

using Chebyshev iteration combined with a preconditioning technique. The matrix
A was first decomposed as A = LU + F where M = LU is an approximate LU
decomposition of A provided by one step of the SIP algorithm described in [211.
Then Richardson iteration was run for the problem M-Ax = M-'b, yielding the
sequence of approximate solutions

(5.1) x(k+l) = x(k) + tkM-lr(k),

where r(k) is the residual b - Ax(k) and tk is an acceleration parameter. The

acceleration parameters were first chosen a priori and, as the iteration proceeded,
they were periodically adjusted in such a way that the iteration (5.1) matches the
(optimal) Chebyshev iteration [12] for the problem M-1A = M-lb. After 60 steps,
the residual norm has decreased by a factor of (see [19]):

11 r(60) 1/ 1r(0) 1 2.025 x 10-5.

The initial vector xo was generated randomly. Note that an important part of the
calculations lies in the computation of a few eigenvalues of A, as these are needed
for determining the optimal parameters tk.

G
0.

R
E -1.
S
I
0
U
A -2.
L

N
0
R -3-
Mk \

-4.

-8.

0. 40.
20. 80.

ITERATIONS

FIGURE 6

Convergence of Algorithm 3.5 on exanWle of Subsection 5.5.

Upper curve p = 2, lower p = 4

Two runs have been made with Algorithm 3.5, the first with p = 2 and the
second with p = 4. The same preconditioning matrix M = LU as above has been
used. Figure 6 shows the evolution of the residual norms IIM-Ax(k) - M-'bI and
confirms the remarks ending Section 3. In either case, no restart was necessary

This content downloaded from

103.232.241.147 on Mon, 27 Jan 2025 16:22:05 UTC
All use subject to https://about.jstor.org/terms
 LARGE UNSYMMETRIC LINEAR SYSTEMS 125

and, at the 60th step, the actual residual norms

of

llr (60)11/ 1 1 r?) 11 4.44 x 10-7 forp = 2

and

I I r'60) rlo/ I I 1.62 x 10-7 forp = 4.

Clearly, here the choice p = 2 is more suitable than p = 4. Note that, with p = 2,
each step of Algorithm 3.5 requires about 21 N operations, while each step of the
first method requires an everage of 16.7 N operations per step [19]. Considering
that it takes 40 steps for the second method to get the residual norm reduced by a
factor of I Ir40)II/Ilr)II 3.3 x I0-5, it is easily seen that the total number of
operations is about 16% less with Algorithm 3.5. Thus, the total numbers of
operations are comparable. The first method requires, however, 5 N more memory
locations than the second. (These are used to estimate the eigenvalues of M -'A.)
Let us mention that on another example, similar to the present one, the Chebyshev
iteration failed to converge, while the I.O.M. gave the solution without any
problem with p = 2.

Acknowledgments. The author is indebted to Professor P. Saylor for providing

the example treated in Subsection 5.5, and to the referee for his valuable remarks
and his careful corrections.

Department of Computer Science

University of Illinois at Urbana-Champaign
Urbana, Illinois 61801

1. W. E. ARNOLDI, "The principle of miniized iterations in the solution of the matrix eigenvalue
problem," Quart. Appl. Math., v. 9, 1951, pp. 17-29.
2. 0. AXELSSON, Solution of Linear Systems of Equations: Iterative Methods, Lecture Notes in Math.,
vol. 572 (V. A. Barker, Ed.), Springer-Verlag, Berlin and New York, 1977, pp. 1-5 1.
3. A. Bj6RK & T. ELFVING, "Accelerated projection methods for computing pseudo inverse solutions
of systems of linear equations," BIT, v. 19, 1979, pp. 145-163.
4. A. CLAYTON, Further Results on Polynomials Having Least Maximum Modules Over an Ellipse in
the Complex Plane, UKAEA Report AEEW-7348, 1963.
5. P. CONCUS & G. H. GOLUB, A Generalized Conjugate Gradient Methodfor Non-Symmetric Systems
of Linear Equations, Report STAN-CS-75-535, Computer Science Dept., Stanford University, 1976.
6. D. K. FADDEEV & V. N. FADDEEVA, Computational Methods of Linear Algebra, Freeman, San
Francisco, Calif., 1963.
7. A. S. HOUSEHOLDER, The Theory of Matrices in Numerical Analysis, Blaisdell, New York, 1964.
8. M. A. KRASNOSELSKHI ET AL., Approximate Solutions of Operator Equations, Wolters-Noordhoff,
Groningen, 1972.
9. C. C. LANczos, "Solution of systems of linear equations by minimized iterations," J. Res. Nat.
Bur. Standards, v. 49, 1952, pp. 33-53.
10. V. I. LEBEDEV, "Iterative methods for solution of operator equations with their spectrum on
several intervals," Z. Vycisl. Mat. i Mat. Fiz., v. 9, 1969, pp. 1247-1252.
11. G. G. LoRENrz, Approximation of Functions, Holt, Rinehart & Winston, New York, 1966.
12. T. A. MANTEUFFEL, An Iterative Method for Solving Nonsymmetric Linear Systems With Dynamic
Estimation of Parameters, Report UIUCDCS-R-75-758, Dept. of Computer Science, Univ. of Illinois at
Urbana-Champaign; Ph.D. thesis, 1975.
13. C. C. PAIGE, "Bidiagonalization of matrices and solution of linear equations," SIAM J. Numer.
Anal., v. 11, 1974, pp. 197-209.
14. C. C. PAIGE & M. A. SAUNDERS, "Solution of sparse indefinite systems of linear equations," SIAM
J. Numer. Anal., v. 12, 1975, pp. 617-629.

This content downloaded from

103.232.241.147 on Mon, 27 Jan 2025 16:22:05 UTC
All use subject to https://about.jstor.org/terms
 126 Y. SAAD

15. B. N. PARLETr, "A new look at the Lanczos algorithm for solving symmetric systems of linear
equations," Linear Algebra Appl., v. 29, 1980, pp. 323-346.
16. J. K.- REID, "On the method of conjugate gradients for the solution of large sparse systems of
linear equations," in Large Sparse Sets of Linear Equations (J. K. Reid, Ed.), Academic Press, New
York, 1971.
17. T. J. RIVIIN, The Chebyshev Polynomials, Wiley, New York, 1976.
18. Y. SAAD, "Variations on Armoldi's method for computing eigenelements of large unsymmetric
matrices," Linear Algebra Appl., v. 34, 1980, pp. 269-295.
19. P. E. SAYLOR, Richardson's Iteration With Dynamic Parameters and the SIP Approximate
Factorization for the Solution of the Pressure Equation, Society of Petroleum Engineers of AIME Fifth
Symposium on Reservoir Simulation, Denver, Colorado, 1979, SPE 7688.
20. G. W. STEWART, Introduction to Matrix Computation, Academic Press, New York, 1973.
21. H. L. STONE, "Iterative solution of implicit approximations of multidimensional partial differential
equations," SIAM J. Numer. Anal., v. 5, 1968, pp. 530-558.
22. R. S. VARGA "A comparison of the successive overrelaxation method and semi-iterative methods
using Chebyshev polynomials," J. Soc. Indust. Appl. Math., v. 5, 1957, pp. 39-46.
23. H. E. WRIGLEY, "Accelerating the Jacobi method for solving simultaneous equations by
Chebyshev extrapolation when the eigenvalues of the iteration matrtix are complex," Comput. J., v. 6,
1963, pp. 169-176.

This content downloaded from

103.232.241.147 on Mon, 27 Jan 2025 16:22:05 UTC
All use subject to https://about.jstor.org/terms