f04 – Simultaneous Linear Equations Introduction – f04

NAG C Library Chapter Introduction

f04 – Simultaneous Linear Equations

Contents
1 Scope of the Chapter . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2

2 Background to the Problems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2

2.1 Unique Solution of Ax ¼ b . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2

2.2 The Least-squares Solution of Ax ’ b, m > n, rankðAÞ ¼ n . . . . . . . . . . . . 3
2.3 Rank-deficient Cases . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4
2.4 The Rank of a Matrix . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4
2.5 Generalized Linear Least Squares Problems . . . . . . . . . . . . . . . . . . . . . . . . 5
2.6 Calculating the Inverse of a Matrix . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5

3 Recommendations on Choice and Use of Available Functions . . . . . . . . 5

3.1 Black Box and General Purpose Functions . . . . . . . . . . . . . . . . . . . . . . . . 5

3.2 Systems of Linear Equations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6
3.3 Linear Least-squares Problems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6
3.4 Sparse Matrix Functions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6

4 Decision Tree . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6

5 Index . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7

6 Functions Withdrawn or Scheduled for Withdrawal . . . . . . . . . . . . . . . 7

7 References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7

[NP3645/7] f04.1
Introduction – f04 NAG C Library Manual

1 Scope of the Chapter

This chapter is concerned with the solution of the matrix equation AX ¼ B, where B may be a single
vector or a matrix of multiple right-hand sides. The matrix A may be real, complex, symmetric,
Hermitian, positive-definite, positive-definite Toeplitz or banded. It may also be rectangular, in which case
a least-squares solution is obtained.
For a general introduction to sparse systems of equations, see the f11 Chapter Introduction, which provides
functions for sparse systems.

2 Background to the Problems

A set of linear equations may be written in the form
Ax ¼ b
where the known matrix A, with real or complex coefficients, is of size m by n, (m rows and n columns),
the known right-hand vector b has m components (m rows and one column), and the required solution
vector x has n components (n rows and one column). There may also be p vectors bi , i ¼ 1; 2; . . . ; p on
the right-hand side and the equations may then be written as
AX ¼ B;
the required matrix X having as its p columns the solutions of Axi ¼ bi , i ¼ 1; 2; . . . ; p. Some functions
deal with the latter case, but for clarity only the case p ¼ 1 is discussed here.
The most common problem, the determination of the unique solution of Ax ¼ b, occurs when m ¼ n and
A is not singular, that is rankðAÞ ¼ n. This is discussed in Section 2.1 below. The next most common
problem, discussed in Section 2.2 below, is the determination of the least-squares solution of Ax ’ b
required when m > n and rankðAÞ ¼ n, i.e., the determination of the vector x which minimizes the norm
of the residual vector r ¼ b
Ax. All other cases are rank deficient, and they are treated in Section 2.3.
Most of the functions of the chapter are based on those published in the book edited by Wilkinson and
Reinsch (1971). We are very grateful to the late Dr J H Wilkinson FRS for his help and interest during the
implementation of this chapter of the Library.

2.1 Unique Solution of Ax ¼ b

Most functions in this chapter solve this particular problem. The computation starts with the triangular
decomposition A ¼ P LU, where L and U are respectively lower and upper triangular matrices and P is a
permutation matrix, chosen so as to ensure that the decomposition is numerically stable. The solution is
then obtained by solving in succession the simpler equations
Ly ¼ PTb
Ux ¼ y
the first by forward-substitution and the second by back-substitution.
If A is real symmetric and positive-definite, U ¼ LT , while if A is complex Hermitian and positive-
definite, U ¼ LH ; in both these cases P is the identity matrix (i.e., no permutations are necessary). In all
other cases either U or L has unit diagonal elements.
Due to rounding errors the computed ‘solution’ x0 , say, is only an approximation to the true solution x.
This approximation will sometimes be satisfactory, agreeing with x to several figures, but if the problem is
ill-conditioned then x and x0 may have few or even no figures in common, and at this stage there is no
means of estimating the ‘accuracy’ of x0 .
It must be emphasised that the ‘true’ solution x may not be meaningful, that is correct to all figures quoted,
if the elements of A and b are known with certainty only to say p figures, where p is smaller than the
word-length of the computer.
One approach to assessing the accuracy of the solution is to compute or estimate the condition number of
A, defined as

ðAÞ ¼ kAkkA
1 k:

f04.2 [NP3645/7]
f04 – Simultaneous Linear Equations Introduction – f04

Roughly speaking, errors or uncertainties in A or b may be amplified in the solution by a factor
ðAÞ.
Thus, for example, if the data in A and b are only accurate to 5 digits and
ðAÞ  103 , then the solution
cannot be guaranteed to have more than 2 correct digits. If
ðAÞ  105 , the solution may have no
meaningful digits.
To be more precise, suppose that
Ax ¼ b and ðA þ AÞðx þ xÞ ¼ b þ b:
Here A and b represent perturbations to the matrices A and b which cause a perturbation x in the
solution. We can define measures of the relative sizes of the perturbations in A, b and x as
kAk kbk kxk
A ¼ ; b ¼ and x ¼ respectively:
kAk kbk kxk
Then

ðAÞ
x  ð þ b Þ
1

ðAÞA A
provided that
ðAÞA < 1. Often
ðAÞA  1 and then the bound effectively simplifies to
x 
ðAÞðA þ b Þ:
Hence, if we know
ðAÞ, A and b , we can compute a bound on the relative errors in the solution. Note
that A , b and x are defined in terms of the norms of A, b and x. If A, b or x contains elements of
widely differing magnitude, then A , b and x will be dominated by the errors in the larger elements, and
x will give no information about the relative accuracy of smaller elements of x.
Another way to obtain useful information about the accuracy of a solution is to solve two sets of equations,
one with the given coefficients, which are assumed to be known with certainty to p figures, and one with
the coefficients rounded to (p
1) figures, and to count the number of figures to which the two solutions
agree. In ill-conditioned problems this can be surprisingly small and even zero.
Chapter f07 contains functions for estimating condition numbers and for returning error bounds.

2.2 The Least-squares Solution of Ax ’ b, m > n, rankðAÞ ¼ n

^ which minimizes the sum of the squares of the residuals,
The least-squares solution is the vector x
xÞT ðb
A^
S ¼ ðb
A^ xk22 :
xÞ ¼ kb
A^
The solution is obtained in two steps.
(i) Householder Transformations are used to reduce A to ‘simpler form’ via the equation QA ¼ R,
where R has the appearance
!
R^
0

^ a non-singular upper triangular n by n matrix and 0 a zero matrix of shape ðm
nÞ by n.

with R
Similar operations convert b to Qb ¼ c, where


c
c¼ 1
c2
with c1 having n rows and c2 having (m
n) rows.
(ii) The required least-squares solution is obtained by back-substitution in the equation
^x
R ^ ¼ c1 :
^ 0 is only an approximation to the required x
Again due to rounding errors the computed x ^.

[NP3645/7] f04.3
Introduction – f04 NAG C Library Manual

2.3 Rank-deficient Cases

If, in the least-squares problem just discussed, rankðAÞ < n, then a least-squares solution exists but it is
not unique. In this situation it is usual to ask for the least-squares solution ‘of minimal length’, i.e., the
vector x which minimizes kxk2 , among all those x for which kb
Axk2 is a minimum.
This can be computed from the Singular Value Decomposition (SVD) of A, in which A is factorized as
A ¼ QDP T
where Q is an m by n matrix with orthonormal columns, P is an n by n orthogonal matrix and D is an n
by n diagonal matrix. The diagonal elements of D are called the ‘singular values’ of A; they are non-
negative and can be arranged in decreasing order of magnitude:
d1  d2      dn  0:
The columns of Q and P are called respectively the left and right singular vectors of A. If the singular
values drþ1 ; . . . ; dn are zero or negligible, but dr is not negligible, then the rank of A is taken to be r (see
also Section 2.4) and the minimal length least-squares solution of Ax ’ b is given by
^ ¼ Dy QT b
x
where Dy is the diagonal matrix with diagonal elements d
1
1
1
1 ; d2 ; . . . ; dr ; 0; . . . ; 0.

The SVD may also be used to find solutions to the homogeneous system of equations Ax ¼ 0, where A is
m by n. Such solutions exist if and only if rankðAÞ < n, and are given by
X
n
x¼ i pi
i¼rþ1

where the i are arbitrary numbers and the pi are the columns of P which correspond to negligible
elements of D.
^ þ x, where x
The general solution to the rank-deficient least-squares problem is given by x ^ is the minimal
length least-squares solution and x is any solution of the homogeneous system of equations Ax ¼ 0.

2.4 The Rank of a Matrix

In theory the rank is r if n
r elements of the diagonal matrix D of the singular value decomposition are
exactly zero. In practice, due to rounding and/or experimental errors, some of these elements have very
small values which usually can and should be treated as zero.
For example, the following 5 by 8 matrix has rank 3 in exact arithmetic:
0 1
22 14
1
3 9 9 2 4
B 10 7 13
2 8 1
6 5C
B C
B 2 10
1 13 1
7 6 0C
B C:
@ 3 0
11
2
2 5 5
2 A
7 8 3 4 4
1 1 2
On a computer with 7 decimal digits of precision the computed singular values were
3:5  101 ; 2:0  101 ; 2:0  101 ; 1:3  10
6 ; 5:5  10
7
and the rank would be correctly taken to be 3.
It is not, however, always certain that small computed singular values are really zero. With the 7 by 7
Hilbert matrix, for example, where aij ¼ 1=ði þ j
1Þ, the singular values are
1:7; 2:7  10
1 ; 2:1  10
2 ; 1:0  10
3 ; 2:9  10
5 ; 4:9  10
7 ; 3:5  10
9 :
Here there is no clear cut-off between small (i.e., negligible) singular values and larger ones. In fact, in
exact arithmetic, the matrix is known to have full rank and none of its singular values is zero. On a
computer with 7 decimal digits of precision, the matrix is effectively singular, but should its rank be taken
to be 6, or 5, or 4?

f04.4 [NP3645/7]
f04 – Simultaneous Linear Equations Introduction – f04

It is therefore impossible to give an infallible rule, but generally the rank can be taken to be the number of
singular values which are neither zero nor very small compared with other singular values. For example, if
there is a sharp decrease in singular values from numbers of order unity to numbers of order 10
7, then the
latter will almost certainly be zero in a machine in which 7 significant decimal figures is the maximum
accuracy. Similarly for a least-squares problem in which the data is known to about four significant figures
and the largest singular value is of order unity then a singular value of order 10
4 or less should almost
certainly be regarded as zero.
It should be emphasised that rank determination and least-squares solutions can be sensitive to the scaling
of the matrix. If at all possible the units of measurement should be chosen so that the elements of the
matrix have data errors of approximately equal magnitude.

2.5 Generalized Linear Least Squares Problems

The simple type of linear least-squares problem described in Section 2.2 can be generalized in various
ways.
1. Linear least-squares problems with equality constraints:
find x to minimize S ¼ kc
Axk22 subject to Bx ¼ d;
where A is m by n and B is p by n, with p  n  m þ p. The equations Bx ¼ d may be regarded as
a set of equality constraints on the problem of minimizing S. Alternatively the problem may be
regarded as solving an overdetermined system of equations



A c
x¼ ;
B d
where some of the equations (those involving B) are to be solved exactly, and the others (those
involving A) are to be solved in a least-squares sense. The
problem has a unique solution on the
A
assumptions that B has full row rank p and the matrix has full column rank n. (For linear
B
least-squares problems with inequality constraints, refer to Chapter e04.)
2. General Gauss–Markov linear model problems:
minimize kyk2 subject to d ¼ Ax þ By;
where A is m by n and B is m by p, with n  m  n þ p. When B ¼ I, the problem reduces to an
ordinary linear least-squares problem. When B is square and nonsingular, it is equivalent to a
weighted linear least-squares problem:
find x to minimize kB
1 ðd
AxÞk2 :
The problem has a unique solution on the assumptions that A has full column rank n, and the matrix
ðA; BÞ has full row rank m.

2.6 Calculating the Inverse of a Matrix

The functions in this chapter can also be used to calculate the inverse of a square matrix A by solving the
equation
AX ¼ I
where I is the identity matrix. However, solving the equations AX ¼ B by calculation of the inverse of
the coefficient matrix A, i.e., by X ¼ A
1 B, is definitely not recommended.
Similar remarks apply to the calculation of the pseudo inverse of a singular or rectangular matrix.

3 Recommendations on Choice and Use of Available Functions

3.1 Black Box and General Purpose Functions
Most of the functions in this chapter are categorised as Black Box functions or general purpose functions.

[NP3645/7] f04.5
Introduction – f04 NAG C Library Manual

Black Box functions solve the equations Axi ¼ bi , for i ¼ 1; 2; . . . ; p, in a single call with the matrix A
and the right-hand sides bi being supplied as data. These are the simplest functions to use and are suitable
when all the right-hand sides are known in advance and do not occupy too much storage.
General purpose functions, in general, require a previous call to a function in Chapter f01 or Chapter f03 to
factorize the matrix A. This factorization can then be used repeatedly to solve the equations for one or
more right-hand sides which may be generated in the course of the computation. The Black Box functions
simply call a factorization function and then a general purpose function to solve the equations.

3.2 Systems of Linear Equations

Most of the functions in this chapter solve linear equations Ax ¼ b when A is n by n and a unique
solution is expected (case 2.1). If this turns out to be untrue the functions go to a failure exit. The matrix
A may be ‘general’ real or complex, or may have special structure or properties, e.g., it may be banded,
tridiagonal, almost block-diagonal, sparse, symmetric, Hermitian, positive-definite (or various combinations
of these). For some of the combinations see Chapter f07. nag_real_cholesky_skyline_solve (f04mcc)
(which needs to be preceded by a call to nag_real_cholesky_skyline (f01mcc)) can be used for the solution
of variable band-width (skyline) positive-definite systems.
It must be emphasised that it is a waste of computer time and space to use an inappropriate function, for
example one for the complex case when the equations are real. It is also unsatisfactory to use the special
functions for a positive-definite matrix if this property is not known in advance.
Other functions for solving linear equation systems, computing inverse matrices, and estimating condition
numbers can be found in Chapter f07, which contains LAPACK software.

3.3 Linear Least-squares Problems

Functions for solving linear least-squares problems using the QR factorization or the SVD can be found in
Chapters f01, f02 and f08. When m  n and a unique solution is expected, the QR factorization can be
used, otherwise the QR factorization with pivoting, or the SVD should be used. For m  n, the SVD is
not significantly more expensive than the QR factorization. See Chapter f08 for further discussion.
Problems with linear equality constraints can be solved by functions in Chapter f08 provided that the
problems are of full rank. Problems with linear inequality constraints can be solved by nag_opt_lin_lsq
(e04ncc) in Chapter e04.
General Gauss–Markov linear model problems, as formulated in Section 2.5, can be solved by functions in
Chapter f08.

3.4 Sparse Matrix Functions

For the solution of sparse linear equations see Chapter f11.

4 Decision Tree
If at any stage the answer to a question is ‘Don’t know’ this should be read as ‘No’.
The name of the function (if any) that should be used to factorize the matrix A is given in brackets after
the name of the function for solving the equations.
Tree 1: Black Box functions for unique solution of Ax = b

Is A a real matrix? yes f04arc

no
f04adc

f04.6 [NP3645/7]
f04 – Simultaneous Linear Equations Introduction – f04

Tree 2: General Purpose functions for unique solution of Ax = b

Is A a symmetric
Is A a variable band
Is A a real matrix? yes positive-definite yes matrix? yes f04mcc (f01mcc)
matrix?
no
f04agc (f03aec)
no
f04ajc (f03afc)
no

Is A a Hermitian
positive-definite yes f04awc (f01bnc)
matrix?
no
f04akc (f03ahc)

5 Index
Black Box functions, Ax ¼ b
Complex matrix, ............................................................ nag_complex_lin_eqn_mult_rhs (f04adc)
Real matrix,
Single right-hand side, ...................................................................... nag_real_lin_eqn (f04arc)
General Purpose functions, Ax ¼ b
Complex Hermitian positive-definite matrix, ............ nag_hermitian_lin_eqn_mult_rhs (f04awc)
Complex matrix, .......................................................... nag_complex_lu_solve_mult_rhs (f04akc)
Real band symmetric positive-definite matrix,
variable bandwidth, ............................................ nag_real_cholesky_skyline_solve (f04mcc)
Real matrix, ........................................................................ nag_real_lu_solve_mult_rhs (f04ajc)
Real symmetric positive-definite matrix, ............. nag_real_cholesky_solve_mult_rhs (f04agc)

6 Functions Withdrawn or Scheduled for Withdrawal

None.

7 References
Golub G H and Van Loan C F (1996) Matrix Computations (3rd Edition) Johns Hopkins University Press,
Baltimore
Lawson C L and Hanson R J (1974) Solving Least-squares Problems Prentice–Hall
Wilkinson J H and Reinsch C (1971) Handbook for Automatic Computation II, Linear Algebra Springer–
Verlag

[NP3645/7] f04.7 (last)