Ph.D.

Seminar Report-1
Advanced Numerical Analysis (MA6305)

A UNIFIED PROOF FOR THE CONVERGENCE OF JACOBI AND

GAUSS-SEIDEL METHODS

Submitted by

Silpi Saha
Roll Number: 524MA1007

Under the Supervision of

Prof. Sangita Jha

DEPARTMENT OF MATHEMATICS
NATIONAL INSTITUTE OF TECHNOLOGY ROURKELA
Rourkela- 769008

1
Contents
1 Introduction 3

2 Definitions 3
2.1 Definition-1 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3
2.2 Definition-2 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3
2.3 Definition-3 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4

3 Observations 4
3.1 Observation-1 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4
3.2 Observation-2 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4

4 Iterative Methods 4

5 Main Theorem 5
5.1 Theorem-1 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5
5.2 Lemma-1 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6

2
 Abstract
We present a new unified proof for the convergence of both the Jacobi and the Gauss-Seidel methods for solving systems of linear
equations under the criterion of either (a) strict diagonal dominance of the matrix, or (b) diagonal dominance and irreducibility of the
matrix. These results are well known. The proof for criterion (a) makes use of Gersgorin’s theorem, while the proof for criterion (b) uses
Taussky’s theorem that extends Gersgorin’s work. Hence the topic is interesting for teaching purposes.

1 Introduction
Here, We present a unified proof for the convergence of both the Jacobi and the Gauss-Seidel iterative methods for solving systems of
linear equations under the criterion of either (a) strict diagonal dominance of the matrix, or (b) diagonal dominance and irreducibility
of the matrix. These results are well known [2]. For each criterion, the proof is unified in that it uses similar arguments for both the
methods. The proof is simple because low level details are contained in an easy Lemma and a couple of Observations, while the rest of
the analysis is based on elementary properties of matrices and their eigenvalues.
We recall three fundamental results about the location of the eigenvalues, known as Gersgorin type theorems [1]. Let A ∈ Cn×n be
a complex matrix: the Gersgorin (row) disks of A are given, for i = 1, . . . , n, by
( n
)
X
Ki = z ∈ C |z − aii | ≤ |aij | . (1)
j=1
j̸=i

The first Gersgorin theorem says that the eigenvalues of A are contained within the union, K, of the disks, where
n
[
K= Ki . (2)
i=1

The second Gersgorin theorem says further that any union of k disks of A not intersecting the remaining (n − k) disks, must contain
exactly k of the eigenvalues (counting multiplicities).
The third Gersgorin type theorem was established by Taussky [1] and says that if A is irreducible and has an eigenvalue λ on the
boundary of K, then λ is on the boundary of each of the n Gersgorin disks of A. Notice that since A and AT have exactly the same
eigenvalues, analogous results hold when column sums of A, instead of row sums, are used to define the radii of the corresponding
(column) disks.

2 Definitions
2.1 Definition-1
A complex matrix A ∈ Cn×n is reducible iff there exist a permutation matrix Π (i.e., Π is obtained from the identity, I, by a permutation
of the rows of I) and an integer k ∈ {1, . . . , n − 1} such that
 
A11 A12
ΠAΠT = , (3)
0 A22

where A11 is k × k and A22 is (n − k) × (n − k). If A is not reducible, than A is said to be irreducible (see [1] for further discussion of
irreducibility).
Note that premultiplication of A by Π, produces a matrix ΠA that consists of the rows of A in the same order that the rows of I
appear in Π. On the other hand, when the columns of I are permuted in that order, then the matrix ΠT is produced. Thus, AΠT is a
matrix consisting of the columns of A in that same order.

2.2 Definition-2
A ∈ Cn×n is said to be strictly diagonally dominant (by rows) iff, for each i = 1, . . . , n,
n
X
|aii | > |aij |; (4)
j=1
j̸=i

similarly, A is said to be strictly diagonally dominant (by columns) iff, for each j = 1, . . . , n,
n
X
|ajj | > |aij |. (5)
i=1
i̸=j

3
2.3 Definition-3
$A∈ Cn×n is said to be diagonally dominant (by rows) iff, for each i = 1, . . . , n,
n
X n
X
|aii | ≥ |aij | and ∃ s ∈ {1, . . . , n} such that |ass | > |asj |; (6)
j=1 j=1
j̸=i j̸=s

similarly, A is said to be diagonally dominant (by columns) iff, for each j = 1, . . . , n,

n
X n
X
|ajj | ≥ |aij | and ∃ t ∈ {1, . . . , n} such that |att | > |ait |. (7)
i=1 i=1
i̸=j i̸=t

3 Observations
Here are a couple of observations which are useful in proving the convergence theorem:

3.1 Observation-1
If the matrix A, is strictly diagonally dominant, then A is nonsingular.

Proof. Consider the Gersgorin disks of A as defined in equation (1). For a strictly diagonally dominant matrix, for each i = 1, . . . , n,
we have:
n
X
|aii | > |aij |.
j=1
j̸=i

This implies that the distance from aii to the origin is greater than the radius of the i-th Gersgorin disk. Therefore, the origin cannot lie
in any of the Gersgorin disks of A.
By the first Gersgorin theorem, all eigenvalues of A must lie within the union of these disks. Since the origin is not contained in any
of the disks, zero cannot be an eigenvalue of A. Hence, A is nonsingular.

3.2 Observation-2
If the matrix A, is irreducible and diagonally dominant, then A is nonsingular.
Proof. Suppose, for contradiction, that A is singular. Then zero is an eigenvalue of A. By the first Gersgorin theorem, this eigenvalue
must lie within at least one of the Gersgorin disks.
From the diagonal dominance condition (Definition 3), we know that for each i = 1, . . . , n:
n
X
|aii | ≥ |aij |,
j=1
j̸=i

P for at least one s ∈ {1, . . . , n}. This means that zero cannot be in the interior of any disk Ki because that would
with strict inequality
require |aii | < j̸=i |aij |, which contradicts the diagonal dominance condition.
Therefore, if zero is an eigenvalue, it must lie on the boundary of the union of the Gersgorin disks. By Taussky’s theorem (the third
Gersgorin type theorem), since A is irreducible, zero must then lie on the boundary of each of the n PGersgorin disks.
However, for the disk Ks corresponding to the row s wherePstrict inequality holds (|ass | > j̸=s |asj |), zero cannot be on its
boundary because the boundary condition would require |ass | = j̸=s |asj |, which contradicts the strict inequality. This contradiction
establishes that A must be nonsingular.

4 Iterative Methods
The Jacobi and Gauss-Seidel iterative methods to solve the system

Ax = b (8)

where A ∈ Cn×n , x, b ∈ Cn , are formulated in terms of a decomposition of the matrix A:

A = D − B − C, (9)

4
where the components of D, B, and C are defined by
  
aij , if i = j, −aij , if i > j, −aij , if i < j,
dij = bij = cij = (10)
0, if i ̸= j; 0, if i ≤ j; 0, if i ≥ j.

Both iterative methods begin with an arbitrary initial vector x0 ∈ Cn and then produce a sequence of vectors, xk , for k = 1, 2, 3, . . .,
by solving either:

Jacobi: Dxk+1 = (B + C)xk + b, or (11)

k+1 k
Gauss-Seidel: (D − B)x = Cx + b, (12)

for k = 0, 1, 2, . . .. It is clear that, for the Jacobi method to be applicable, D must be invertible. Similarly, the invertibility of (D − B)
is required in order to apply the Gauss-Seidel method. The iteration matrices of the methods are then given, respectively, by

Jacobi: J = D−1 (B + C), (13)

−1
Gauss-Seidel: G = (D − B) C. (14)

The iterative steps of the methods are then defined, respectively, by

Jacobi: xk+1 = Jxk + D−1 b, or (15)

k+1 k −1
Gauss-Seidel: x = Gx + (D − B) b, (16)

for k = 0, 1, 2, . . .. Let ρ[Q] denote the spectral radius of the matrix Q, for any Q ∈ Cn×n . The Jacobi method is convergent iff
ρ[J] < 1, while the Gauss-Seidel method is convergent iff ρ[G] < 1.

5 Main Theorem
5.1 Theorem-1
Let A ∈ Cn×n be decomposed as in (9), and let J and G be defined as in (13) and (14), respectively. Under either the criterion (a): A is
strictly diagonally dominant; or the criterion (b): A is diagonally dominant and irreducible, then
(i) both D and (D − B) are invertible; and
(ii) ρ[J] < 1, and ρ[G] < 1.

P (a): Since A is strictly diagonally dominant, we have aii ̸= 0 for all i = 1, . . . , n (because if aii = 0, then the
Proof of (i). For criterion
strict inequality 0 > j̸=i |aij | would be impossible as the right side is non-negative). Therefore, D is invertible.
For (D − B), note that its diagonal entries are the same as those of A (since D contains the diagonal of A and B has zeros on
and above the diagonal). Thus, (D − B) is also strictly diagonally dominant (by the same argument as for A), and hence invertible by
Observation 1.
For criterion (b): By Observation 2, A is nonsingular and hence does not have a zero row. If either D or (D − B) were singular, then
there would exist some k such that dkk = 0 (for D) or (D − B)kk = 0 (for (D − B)).
But (D − B)kk = akk (since B has zeros on the diagonal). Thus, in either case, we would have akk = 0. By diagonal dominance
(Definition 3), this would imply:
Xn
0 = |akk | ≥ |akj | ≥ 0,
j=1
j̸=k

meaning row k of A would be entirely zero, making A singular. This contradicts Observation 2. Therefore, both D and (D − B) must
be invertible.
Proof of (ii). In order to show that both spectral radii are less than 1, we define the matrices AJ (λ) and AG (λ), where λ ∈ C, by

AJ (λ) = λD − B − C; (17)
AG (λ) = λ(D − B) − C; (18)

and establish the following Lemma:

5
5.2 Lemma-1
For each λ ∈ C, with |λ| ≥ 1, if A satisfies any of the following properties:
(a) A is strictly diagonally dominant (by rows or by columns);
(b) A is diagonally dominant (by rows, or by columns);
(c) A is irreducible;
then both AJ (λ) and AG (λ) satisfy the same properties.
Proof. (a) Let A be strictly diagonally dominant by rows (the proof for the other case is almost the same). By hypothesis, for each
i = 1, . . . , n,
n
X
|aii | > |aij |. (19)
j=1
j̸=i

If |λ| ≥ 1 then, for each i = 1, . . . , n,

|λaii | = |λ||aii | (20)
X
> |λ| |aij | (21)
j=1
j̸=i
i−1
X n
X
= |λ| |aij | + |λ| |aij | (22)
j=1 j=i+1
j̸=i
i−1
X n
X
≥ |λ| |aij | + |aij | (23)
j=1 j=i+1
j̸=i
i−1
X n
X
= |λaij | + |aij | (hence the thesis for AG (λ)), (24)
j=1 j=i+1
j̸=i
i−1
X n
X
≥ |aij | + |aij | (hence the thesis for AJ (λ)). (25)
j=1 j=i+1
j̸=i

(b) Very similar to (a). The only difference is that the hypothesis ensures that strict inequality holds in the disequation marked with
(∗) for at least one i ∈ {1, . . . , n}. Weak inequality is guaranteed for all the other cases.
(c) Since the three matrices A, AJ (λ), and AG (λ), for λ ̸= 0, have zero components in exactly the same locations, it follows that if
a permutation matrix Π reduces one of these matrices, then Π also reduces the other two matrices.
We now resume the proof of (ii), Theorem 1. The eigenvalues λ of J are all and only the solutions of the equation
det(λI − J) = 0, (26)
but, from properties of matrices and determinants we have
det(λI − J) = det(λI − D−1 (B + C)) (27)
−1 −1
= det(λD D−D (B + C)) (28)
−1
= det(D (λD − B − C)) (29)
−1
= det(D ) det(λD − B − C) (30)
−1
= det(D ) det(AJ (λ)). (31)
So, for (21) to hold we must have det(AJ (λ)) = 0, as we have already shown that D is nonsingular. But, since AJ (λ) is nonsingular
for |λ| ≥ 1 (by the Lemma and the Observations), it follows that all of the eigenvalues of J must satisfy |λ| < 1. Hence, ρ[J] < 1.
Similarly, the eigenvalues λ of G are all and only the solutions of the equation
det(λI − G) = det(λI − (D − B)−1 C) (32)
−1 −1
= det(λ(D − B) (D − B) − (D − B) C) (33)
−1
= det((D − B) (λ(D − B) − C)) (34)
−1
= det((D − B) ) det(λ(D − B) − C) (35)
−1
= det((D − B) ) det(AG (λ)) (36)
= 0. (37)

6
 Reasoning in the exactly the same way as above we conclude that ρ[G] < 1.

References
[1] P. Lancaster and M. Tismenetsky, The Theory of Matrices, Second Edition with Applications, Academic Press, Orlando, 1985.
[2] J. Stoer and R. Bulirsch, Introduction to Numerical Analysis, Springer-Verlag, New York, 1980.