ot104_HighamFM-B:Gockenbach 2/8/2008 2:47 PM Page 1

Functions
of Matrices
ot104_HighamFM-B:Gockenbach 2/8/2008 2:47 PM Page 2
ot104_HighamFM-B:Gockenbach 2/8/2008 2:47 PM Page 3

Functions
of Matrices
Theory and Computation

Nicholas J. Higham
University of Manchester
Manchester, United Kingdom

Society for Industrial and Applied Mathematics • Philadelphia

Copyright © 2008 by the Society for Industrial and Applied Mathematics.

10 9 8 7 6 5 4 3 2 1

All rights reserved. Printed in the United States of America. No part of this book may
be reproduced, stored, or transmitted in any manner without the written permission of the
publisher. For information, write to the Society for Industrial and Applied Mathematics,
3600 Market Street, 6th Floor, Philadelphia, PA 19104-2688 USA.

Trademarked names may be used in this book without the inclusion of a trademark symbol.
These names are used in an editorial context only; no infringement of trademark is intended.

Maple is a registered trademark of Waterloo Maple, Inc.

Mathematica is a registered trademark of Wolfram Research, Inc.

MATLAB is a registered trademark of The MathWorks, Inc. For MATLAB product

information, please contact The MathWorks, Inc., 3 Apple Hill Drive, Natick, MA
01760-2098 USA, 508-647-7000, Fax: 508-647-7101, [email protected],
www.mathworks.com.

Library of Congress Cataloging-in-Publication Data

Higham, Nicholas J., 1961-
Functions of matrices : theory and computation / Nicholas J. Higham.
p. cm.
Includes bibliographical references and index.
ISBN 978-0-89871-646-7 1. Matrices. 2. Functions. 3. Factorization (Mathematics)
I. Title.

QA188.H53 2008
512.9'434--dc22
2007061811

is a registered trademark.
ot104_HighamFM-B:Gockenbach 2/8/2008 2:47 PM Page 5

To Françoise

ot104_HighamFM-B:Gockenbach 2/8/2008 2:47 PM Page 6
Contents

List of Figures xiii

List of Tables xv

Preface xvii

1 Theory of Matrix Functions 1

1.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1
1.2 Definitions of f (A) . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2
1.2.1 Jordan Canonical Form . . . . . . . . . . . . . . . . . . . . . . 2
1.2.2 Polynomial Interpolation . . . . . . . . . . . . . . . . . . . . . 4
1.2.3 Cauchy Integral Theorem . . . . . . . . . . . . . . . . . . . . . 7
1.2.4 Equivalence of Definitions . . . . . . . . . . . . . . . . . . . . 8
1.2.5 Example: Function of Identity Plus Rank-1 Matrix . . . . . . 8
1.2.6 Example: Function of Discrete Fourier Transform Matrix . . . 10
1.3 Properties . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10
1.4 Nonprimary Matrix Functions . . . . . . . . . . . . . . . . . . . . . . 14
1.5 Existence of (Real) Matrix Square Roots and Logarithms . . . . . . . 16
1.6 Classification of Matrix Square Roots and Logarithms . . . . . . . . . 17
1.7 Principal Square Root and Logarithm . . . . . . . . . . . . . . . . . . 20
1.8 f (AB) and f (BA) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21
1.9 Miscellany . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23
1.10 A Brief History of Matrix Functions . . . . . . . . . . . . . . . . . . . 26
1.11 Notes and References . . . . . . . . . . . . . . . . . . . . . . . . . . . 27
Problems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29

2 Applications 35
2.1 Differential Equations . . . . . . . . . . . . . . . . . . . . . . . . . . . 35
2.1.1 Exponential Integrators . . . . . . . . . . . . . . . . . . . . . . 36
2.2 Nuclear Magnetic Resonance . . . . . . . . . . . . . . . . . . . . . . . 37
2.3 Markov Models . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 37
2.4 Control Theory . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 39
2.5 The Nonsymmetric Eigenvalue Problem . . . . . . . . . . . . . . . . . 41
2.6 Orthogonalization and the Orthogonal Procrustes Problem . . . . . . 42
2.7 Theoretical Particle Physics . . . . . . . . . . . . . . . . . . . . . . . 43
2.8 Other Matrix Functions . . . . . . . . . . . . . . . . . . . . . . . . . . 44
2.9 Nonlinear Matrix Equations . . . . . . . . . . . . . . . . . . . . . . . 44
2.10 Geometric Mean . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 46
2.11 Pseudospectra . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 47
2.12 Algebras . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 47

vii
viii Contents

2.13 Sensitivity Analysis . . . . . . . . . . . . . . . . . . . . . . . . . . . . 48

2.14 Other Applications . . . . . . . . . . . . . . . . . . . . . . . . . . . . 48
2.14.1 Boundary Value Problems . . . . . . . . . . . . . . . . . . . . 48
2.14.2 Semidefinite Programming . . . . . . . . . . . . . . . . . . . . 48
2.14.3 Matrix Sector Function . . . . . . . . . . . . . . . . . . . . . . 48
2.14.4 Matrix Disk Function . . . . . . . . . . . . . . . . . . . . . . . 49
2.14.5 The Average Eye in Optics . . . . . . . . . . . . . . . . . . . . 50
2.14.6 Computer Graphics . . . . . . . . . . . . . . . . . . . . . . . . 50
2.14.7 Bregman Divergences . . . . . . . . . . . . . . . . . . . . . . . 50
2.14.8 Structured Matrix Interpolation . . . . . . . . . . . . . . . . . 50
2.14.9 The Lambert W Function and Delay Differential Equations . 51
2.15 Notes and References . . . . . . . . . . . . . . . . . . . . . . . . . . . 51
Problems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 52

3 Conditioning 55
3.1 Condition Numbers . . . . . . . . . . . . . . . . . . . . . . . . . . . . 55
3.2 Properties of the Fréchet Derivative . . . . . . . . . . . . . . . . . . . 57
3.3 Bounding the Condition Number . . . . . . . . . . . . . . . . . . . . 63
3.4 Computing or Estimating the Condition Number . . . . . . . . . . . 64
3.5 Notes and References . . . . . . . . . . . . . . . . . . . . . . . . . . . 69
Problems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 70

4 Techniques for General Functions 71

4.1 Matrix Powers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 71
4.2 Polynomial Evaluation . . . . . . . . . . . . . . . . . . . . . . . . . . 72
4.3 Taylor Series . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 76
4.4 Rational Approximation . . . . . . . . . . . . . . . . . . . . . . . . . 78
4.4.1 Best L∞ Approximation . . . . . . . . . . . . . . . . . . . . . 79
4.4.2 Padé Approximation . . . . . . . . . . . . . . . . . . . . . . . 79
4.4.3 Evaluating Rational Functions . . . . . . . . . . . . . . . . . . 80
4.5 Diagonalization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 81
4.6 Schur Decomposition and Triangular Matrices . . . . . . . . . . . . . 84
4.7 Block Diagonalization . . . . . . . . . . . . . . . . . . . . . . . . . . . 89
4.8 Interpolating Polynomial and Characteristic Polynomial . . . . . . . 89
4.9 Matrix Iterations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 91
4.9.1 Order of Convergence . . . . . . . . . . . . . . . . . . . . . . . 91
4.9.2 Termination Criteria . . . . . . . . . . . . . . . . . . . . . . . 92
4.9.3 Convergence . . . . . . . . . . . . . . . . . . . . . . . . . . . . 93
4.9.4 Numerical Stability . . . . . . . . . . . . . . . . . . . . . . . . 95
4.10 Preprocessing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 99
4.11 Bounds for kf (A)k . . . . . . . . . . . . . . . . . . . . . . . . . . . . 102
4.12 Notes and References . . . . . . . . . . . . . . . . . . . . . . . . . . . 104
Problems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 105

5 Matrix Sign Function 107

5.1 Sensitivity and Conditioning . . . . . . . . . . . . . . . . . . . . . . . 109
5.2 Schur Method . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 112
5.3 Newton’s Method . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 113
5.4 The Padé Family of Iterations . . . . . . . . . . . . . . . . . . . . . . 115
5.5 Scaling the Newton Iteration . . . . . . . . . . . . . . . . . . . . . . . 119
Contents ix

5.6 Terminating the Iterations . . . . . . . . . . . . . . . . . . . . . . . . 121

5.7 Numerical Stability of Sign Iterations . . . . . . . . . . . . . . . . . . 123
5.8 Numerical Experiments and Algorithm . . . . . . . . . . . . . . . . . 125
5.9 Best L∞ Approximation . . . . . . . . . . . . . . . . . . . . . . . . . 128
5.10 Notes and References . . . . . . . . . . . . . . . . . . . . . . . . . . . 129
Problems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 131

6 Matrix Square Root 133

6.1 Sensitivity and Conditioning . . . . . . . . . . . . . . . . . . . . . . . 133
6.2 Schur Method . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 135
6.3 Newton’s Method and Its Variants . . . . . . . . . . . . . . . . . . . . 139
6.4 Stability and Limiting Accuracy . . . . . . . . . . . . . . . . . . . . . 144
6.4.1 Newton Iteration . . . . . . . . . . . . . . . . . . . . . . . . . 144
6.4.2 DB Iterations . . . . . . . . . . . . . . . . . . . . . . . . . . . 145
6.4.3 CR Iteration . . . . . . . . . . . . . . . . . . . . . . . . . . . . 146
6.4.4 IN Iteration . . . . . . . . . . . . . . . . . . . . . . . . . . . . 146
6.4.5 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 147
6.5 Scaling the Newton Iteration . . . . . . . . . . . . . . . . . . . . . . . 147
6.6 Numerical Experiments . . . . . . . . . . . . . . . . . . . . . . . . . . 148
6.7 Iterations via the Matrix Sign Function . . . . . . . . . . . . . . . . . 152
6.8 Special Matrices . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 154
6.8.1 Binomial Iteration . . . . . . . . . . . . . . . . . . . . . . . . . 154
6.8.2 Modified Newton Iterations . . . . . . . . . . . . . . . . . . . 157
6.8.3 M-Matrices and H-Matrices . . . . . . . . . . . . . . . . . . . 159
6.8.4 Hermitian Positive Definite Matrices . . . . . . . . . . . . . . 161
6.9 Computing Small-Normed Square Roots . . . . . . . . . . . . . . . . 162
6.10 Comparison of Methods . . . . . . . . . . . . . . . . . . . . . . . . . . 164
6.11 Involutory Matrices . . . . . . . . . . . . . . . . . . . . . . . . . . . . 165
6.12 Notes and References . . . . . . . . . . . . . . . . . . . . . . . . . . . 166
Problems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 168

7 Matrix pth Root 173

7.1 Theory . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 173
7.2 Schur Method . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 175
7.3 Newton’s Method . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 177
7.4 Inverse Newton Method . . . . . . . . . . . . . . . . . . . . . . . . . . 181
7.5 Schur–Newton Algorithm . . . . . . . . . . . . . . . . . . . . . . . . . 184
7.6 Matrix Sign Method . . . . . . . . . . . . . . . . . . . . . . . . . . . . 186
7.7 Notes and References . . . . . . . . . . . . . . . . . . . . . . . . . . . 187
Problems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 189

8 The Polar Decomposition 193

8.1 Approximation Properties . . . . . . . . . . . . . . . . . . . . . . . . 197
8.2 Sensitivity and Conditioning . . . . . . . . . . . . . . . . . . . . . . . 199
8.3 Newton’s Method . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 202
8.4 Obtaining Iterations via the Matrix Sign Function . . . . . . . . . . . 202
8.5 The Padé Family of Methods . . . . . . . . . . . . . . . . . . . . . . . 203
8.6 Scaling the Newton Iteration . . . . . . . . . . . . . . . . . . . . . . . 205
8.7 Terminating the Iterations . . . . . . . . . . . . . . . . . . . . . . . . 207
8.8 Numerical Stability and Choice of H . . . . . . . . . . . . . . . . . . 209
x Contents

8.9 Algorithm . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 210

8.10 Notes and References . . . . . . . . . . . . . . . . . . . . . . . . . . . 213
Problems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 216

9 Schur–Parlett Algorithm 221

9.1 Evaluating Functions of the Atomic Blocks . . . . . . . . . . . . . . . 221
9.2 Evaluating the Upper Triangular Part of f (T ) . . . . . . . . . . . . . 225
9.3 Reordering and Blocking the Schur Form . . . . . . . . . . . . . . . . 226
9.4 Schur–Parlett Algorithm for f (A) . . . . . . . . . . . . . . . . . . . . 228
9.5 Preprocessing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 230
9.6 Notes and References . . . . . . . . . . . . . . . . . . . . . . . . . . . 231
Problems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 231

10 Matrix Exponential 233

10.1 Basic Properties . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 233
10.2 Conditioning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 238
10.3 Scaling and Squaring Method . . . . . . . . . . . . . . . . . . . . . . 241
10.4 Schur Algorithms . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 250
10.4.1 Newton Divided Difference Interpolation . . . . . . . . . . . . 250
10.4.2 Schur–Fréchet Algorithm . . . . . . . . . . . . . . . . . . . . . 251
10.4.3 Schur–Parlett Algorithm . . . . . . . . . . . . . . . . . . . . . 251
10.5 Numerical Experiment . . . . . . . . . . . . . . . . . . . . . . . . . . 252
10.6 Evaluating the Fréchet Derivative and Its Norm . . . . . . . . . . . . 253
10.6.1 Quadrature . . . . . . . . . . . . . . . . . . . . . . . . . . . . 254
10.6.2 The Kronecker Formulae . . . . . . . . . . . . . . . . . . . . . 256
10.6.3 Computing and Estimating the Norm . . . . . . . . . . . . . . 258
10.7 Miscellany . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 259
10.7.1 Hermitian Matrices and Best L∞ Approximation . . . . . . . 259
10.7.2 Essentially Nonnegative Matrices . . . . . . . . . . . . . . . . 260
10.7.3 Preprocessing . . . . . . . . . . . . . . . . . . . . . . . . . . . 261
10.7.4 The ψ Functions . . . . . . . . . . . . . . . . . . . . . . . . . . 261
10.8 Notes and References . . . . . . . . . . . . . . . . . . . . . . . . . . . 262
Problems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 265

11 Matrix Logarithm 269

11.1 Basic Properties . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 269
11.2 Conditioning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 272
11.3 Series Expansions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 273
11.4 Padé Approximation . . . . . . . . . . . . . . . . . . . . . . . . . . . 274
11.5 Inverse Scaling and Squaring Method . . . . . . . . . . . . . . . . . . 275
11.5.1 Schur Decomposition: Triangular Matrices . . . . . . . . . . . 276
11.5.2 Full Matrices . . . . . . . . . . . . . . . . . . . . . . . . . . . 278
11.6 Schur Algorithms . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 279
11.6.1 Schur–Fréchet Algorithm . . . . . . . . . . . . . . . . . . . . . 279
11.6.2 Schur–Parlett Algorithm . . . . . . . . . . . . . . . . . . . . . 279
11.7 Numerical Experiment . . . . . . . . . . . . . . . . . . . . . . . . . . 280
11.8 Evaluating the Fréchet Derivative . . . . . . . . . . . . . . . . . . . . 281
11.9 Notes and References . . . . . . . . . . . . . . . . . . . . . . . . . . . 283
Problems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 284
Contents xi

12 Matrix Cosine and Sine 287

12.1 Basic Properties . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 287
12.2 Conditioning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 289
12.3 Padé Approximation of Cosine . . . . . . . . . . . . . . . . . . . . . . 290
12.4 Double Angle Algorithm for Cosine . . . . . . . . . . . . . . . . . . . 290
12.5 Numerical Experiment . . . . . . . . . . . . . . . . . . . . . . . . . . 295
12.6 Double Angle Algorithm for Sine and Cosine . . . . . . . . . . . . . . 296
12.6.1 Preprocessing . . . . . . . . . . . . . . . . . . . . . . . . . . . 299
12.7 Notes and References . . . . . . . . . . . . . . . . . . . . . . . . . . . 299
Problems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 300

13 Function of Matrix Times Vector: f (A)b 301

13.1 Representation via Polynomial Interpolation . . . . . . . . . . . . . . 301
13.2 Krylov Subspace Methods . . . . . . . . . . . . . . . . . . . . . . . . 302
13.2.1 The Arnoldi Process . . . . . . . . . . . . . . . . . . . . . . . 302
13.2.2 Arnoldi Approximation of f (A)b . . . . . . . . . . . . . . . . . 304
13.2.3 Lanczos Biorthogonalization . . . . . . . . . . . . . . . . . . . 306
13.3 Quadrature . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 306
13.3.1 On the Real Line . . . . . . . . . . . . . . . . . . . . . . . . . 306
13.3.2 Contour Integration . . . . . . . . . . . . . . . . . . . . . . . . 307
13.4 Differential Equations . . . . . . . . . . . . . . . . . . . . . . . . . . . 308
13.5 Other Methods . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 309
13.6 Notes and References . . . . . . . . . . . . . . . . . . . . . . . . . . . 309
Problems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 310

14 Miscellany 313
14.1 Structured Matrices . . . . . . . . . . . . . . . . . . . . . . . . . . . . 313
14.1.1 Algebras and Groups . . . . . . . . . . . . . . . . . . . . . . . 313
14.1.2 Monotone Functions . . . . . . . . . . . . . . . . . . . . . . . 315
14.1.3 Other Structures . . . . . . . . . . . . . . . . . . . . . . . . . 315
14.1.4 Data Sparse Representations . . . . . . . . . . . . . . . . . . . 316
14.1.5 Computing Structured f (A) for Structured A . . . . . . . . . 316
14.2 Exponential Decay of Functions of Banded Matrices . . . . . . . . . . 317
14.3 Approximating Entries of Matrix Functions . . . . . . . . . . . . . . . 318

A Notation 319

B Background: Definitions and Useful Facts 321

B.1 Basic Notation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 321
B.2 Eigenvalues and Jordan Canonical Form . . . . . . . . . . . . . . . . 321
B.3 Invariant Subspaces . . . . . . . . . . . . . . . . . . . . . . . . . . . . 323
B.4 Special Classes of Matrices . . . . . . . . . . . . . . . . . . . . . . . . 323
B.5 Matrix Factorizations and Decompositions . . . . . . . . . . . . . . . 324
B.6 Pseudoinverse and Orthogonality . . . . . . . . . . . . . . . . . . . . 325
B.6.1 Pseudoinverse . . . . . . . . . . . . . . . . . . . . . . . . . . . 325
B.6.2 Projector and Orthogonal Projector . . . . . . . . . . . . . . . 326
B.6.3 Partial Isometry . . . . . . . . . . . . . . . . . . . . . . . . . . 326
B.7 Norms . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 326
B.8 Matrix Sequences and Series . . . . . . . . . . . . . . . . . . . . . . . 328
B.9 Perturbation Expansions for Matrix Inverse . . . . . . . . . . . . . . 328
xii Contents

B.10 Sherman–Morrison–Woodbury Formula . . . . . . . . . . . . . . . . . 329

B.11 Nonnegative Matrices . . . . . . . . . . . . . . . . . . . . . . . . . . . 329
B.12 Positive (Semi)definite Ordering . . . . . . . . . . . . . . . . . . . . . 330
B.13 Kronecker Product and Sum . . . . . . . . . . . . . . . . . . . . . . . 331
B.14 Sylvester Equation . . . . . . . . . . . . . . . . . . . . . . . . . . . . 331
B.15 Floating Point Arithmetic . . . . . . . . . . . . . . . . . . . . . . . . 331
B.16 Divided Differences . . . . . . . . . . . . . . . . . . . . . . . . . . . . 332
Problems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 334

C Operation Counts 335

D Matrix Function Toolbox 339

E Solutions to Problems 343

Bibliography 379

Index 415
List of Figures

2.1 The scalar sector function sectp (z) for p = 2: 5. . . . . . . . . . . . . . 49

3.1 Relative errors in the Frobenius norm for the finite difference approx-
imation (3.22) to the Fréchet derivative. . . . . . . . . . . . . . . . . 68

4.1 2-norms of first 99 terms in Taylor series of eA . . . . . . . . . . . . . 75

4.2 Relative errors for inversion of A = 3In , n = 25: 60, via the charac-
teristic polynomial. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 90

5.1 The function gr (x) = tanh(r arctanh(x)) for r = 2, 4, 8, 16. . . . . . . 118

5.2 Best L∞ approximation r(x) to sign(x) from R3,4 on [−2, −1] ∪ [1, 2]. 129

6.1 The cardioid (6.45), shaded, together with the unit circle . . . . . . . 157

7.1 Convergence of the Newton iteration (7.6) for a pth root of unity . . 179
7.2 Regions of a ∈ C for which the inverse Newton iteration (7.15) con-
verges to a−1/p . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 183

8.1 Bounds on number of iterations for Newton iteration with optimal

scaling for 1 ≤ κ2 (A) ≤ 1016 . . . . . . . . . . . . . . . . . . . . . . . . 207

9.1 Normwise relative errors for funm mod and condrel (exp, A)u. . . . . . 230

10.1 2-norms of first 20 powers of A in (10.39). . . . . . . . . . . . . . . . 249

10.2 2-norm of exp(At) for A in (10.39). . . . . . . . . . . . . . . . . . . . 249
10.3 Normwise relative errors for MATLAB’s funm, expm, expmdemo1, and
funm mod. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 253
10.4 Same data as in Figure 10.3 presented as a performance profile. . . . 254

11.1 Illustration of condition (b) of Theorem 11.4. . . . . . . . . . . . . . 272

11.2 Normwise relative errors for MATLAB’s logm, logm old, logm iss schur,
and logm iss. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 281
11.3 Same data as in Figure 11.2 presented as a performance profile. . . . 282

12.1 Normwise relative errors for Algorithm 12.6, MATLAB’s funm, and
Algorithm 12.7. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 296
12.2 Same data as in Figure 12.1 presented as a performance profile. . . . 297
12.3 Normwise relative errors for Algorithm 12.6, Algorithm 12.7, Algo-
rithm 12.8, funm, and sine obtained as shifted cosine from Algo-
rithm 12.6. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 299

xiii
List of Tables

4.1 Number of matrix multiplications required by the Paterson–Stockmeyer

method and Algorithms 4.2 and 4.3 to evaluate a degree m matrix
polynomial. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 74
4.2 Number of matrix multiplications required by the Paterson–Stockmeyer
method to evaluate both pmm (A) and qmm (A). . . . . . . . . . . . . . 80
4.3 Errors keA − Fbk/keA k for Fb from funm simple for the matrix A =
gallery(’triw’,8). . . . . . . . . . . . . . . . . . . . . . . . . . . . 89
4.4 Square root and sign iterations applied to Wilson matrix in single
precision arithmetic. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 96

5.1 Iteration functions fℓm from the Padé family (5.27). . . . . . . . . . . 116
5.2 Number of iterations for scaled Newton iteration. . . . . . . . . . . . 125
5.3 Newton iteration with spectral scaling for Jordan block J(2) ∈ R16×16 . 126
5.4 Newton iteration with determinantal scaling for random A ∈ R16×16
with κ2 (A) = 1010 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 126
5.5 Newton iteration with determinantal scaling for random A ∈ R16×16
with real eigenvalues parametrized by d. . . . . . . . . . . . . . . . . 127

6.1 Cost per iteration of matrix square root iterations. . . . . . . . . . . 143

6.2 Summary of stability and limiting accuracy of square root iterations. 147
6.3 Results for rank-1 perturbation of I. . . . . . . . . . . . . . . . . . . 150
6.4 Results for Moler matrix. . . . . . . . . . . . . . . . . . . . . . . . . . 150
6.5 Results for nonnormal matrix. . . . . . . . . . . . . . . . . . . . . . . 151
6.6 Results for Chebyshev–Vandermonde matrix. . . . . . . . . . . . . . . 151

8.1 Results for nearly orthogonal matrix, n = 16. . . . . . . . . . . . . . . 212

8.2 Results for binomial matrix, n = 16. . . . . . . . . . . . . . . . . . . . 212
8.3 Results for Frank matrix, n = 16. . . . . . . . . . . . . . . . . . . . . 212

10.1 Some formulae for eA . . . . . . . . . . . . . . . . . . . . . . . . . . . . 234

10.2 Maximal values θm of k2−s Ak such that the backward error bound
(10.31) does not exceed u = 2−53 , values of νm = min{ |x| : qm (x) =
0}, and upper bound ξm for kqm (A)−1 k. . . . . . . . . . . . . . . . . 244
10.3 Number of matrix multiplications, πm , required to evaluate pm (A)
and qm (A), and measure of overall cost Cm in (10.35). . . . . . . . . 245
Pm
10.4 Coefficients b(0 : m) in numerator pm (x) = i=0 bi xi of Padé approx-
imant rm (x) to ex , normalized so that b(m) = 1. . . . . . . . . . . . . 246
10.5 Zeros αj of numerator p8 and βj of denominator q8 of [8/8] Padé
approximant r8 to τ (x) = tanh(x)/x, shown to 5 significant digits. . . 258

xv
xvi List of Tables

11.1 Maximal values θm of kXk such that the bound (11.19) ensures krm (X)−
log(I +X)k does not exceed u = 2−53 , along with upper bound (11.20)
for κ(qm (X)) and upper bound (11.21) for φm , both with kXk = θm . 277

12.1 Number of matrix multiplications π2m required to evaluate p2m (A)

and q2m (A). . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 290
12.2 Maximum value θ2m of θ such that the absolute error bound (12.24)
does not exceed u = 2−53 . . . . . . . . . . . . . . . . . . . . . . . . . 292
12.3 Upper bound for κ(q2m (A)) when θ ≤ θ2m , based on (12.26) and (12.27),
where the θ2m are given in Table 12.2. . . . . . . . . . . . . . . . . . 293
12.4 Upper bounds for ke p2m k∞ and ke q2m k∞ for θ ≤ θ2m . . . . . . . . . . . 293
12.5 Logic for choice of scaling and Padé approximant degree d ≡ 2m. . . 294
12.6 Maximum value βm of kAk such that the absolute error bound (12.28)
does not exceed u = 2−53 . . . . . . . . . . . . . . . . . . . . . . . . . 297
12.7 Number of matrix multiplications π e2m to evaluate p2m (A), q2m (A),
pe2m+1 (A), and qe2m+1 (A). . . . . . . . . . . . . . . . . . . . . . . . . . 298

14.1 Structured matrices associated with some scalar products. . . . . . . 314

B.1 Constants αpq such that kAkp ≤ αpq kAkq , A ∈ Cm×n . . . . . . . . . . 327

C.1 Cost of some matrix computations. . . . . . . . . . . . . . . . . . . . 336

C.2 Cost of some matrix factorizations and decompositions. . . . . . . . . 337

D.1 Contents of Matrix Function Toolbox and corresponding parts of this

book. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 340
D.2 Matrix-function-related M-files in MATLAB and corresponding algo-
rithms in this book. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 341
Preface

Functions of matrices have been studied for as long as matrix algebra itself. Indeed,
in his seminal A Memoir on the Theory of Matrices (1858), Cayley investigated the
square root of a matrix, and it was not long before definitions of f (A) for general f
were proposed by Sylvester and others. From their origin in pure mathematics, ma-
trix functions have broadened into a subject of study in applied mathematics, with
widespread applications in science and engineering. Research on matrix functions in-
volves matrix theory, numerical analysis, approximation theory, and the development
of algorithms and software, so it employs a wide range of theory and methods and
promotes an appreciation of all these important topics.
My first foray into f (A) was as a graduate student when I became interested in
the matrix square root. I have worked on matrix functions on and off ever since.
Although there is a large literature on the subject, including chapters in several
books (notably Gantmacher [203, ], Horn and Johnson [296, ], Lancaster
and Tismenetsky [371, ], and Golub and Van Loan [224, ]), there has not
previously been a book devoted to matrix functions. I started to write this book in
2003. In the intervening period interest in matrix functions has grown significantly,
with new applications appearing and the literature expanding at a fast rate, so the
appearance of this book is timely.
This book is a research monograph that aims to give a reasonably complete treat-
ment of the theory of matrix functions and numerical methods for computing them,
as well as an overview of applications. The theory of matrix functions is beautiful and
nontrivial. I have strived for an elegant presentation with illuminating examples, em-
phasizing results of practical interest. I focus on three equivalent definitions of f (A),
based on the Jordan canonical form, polynomial interpolation, and the Cauchy inte-
gral formula, and use all three to develop the theory. A thorough treatment is given
of problem sensitivity, based on the Fréchet derivative. The applications described
include both the well known and the more speculative or recent, and differential
equations and algebraic Riccati equations underlie many of them.
The bulk of the book is concerned with numerical methods and the associated
issues of accuracy, stability, and computational cost. Both general purpose methods
and methods for specific functions are covered. Little mention is made of methods
that are numerically unstable or have exorbitant operation counts of order n4 or
higher; many methods proposed in the literature are ruled out for at least one of
these reasons.
The focus is on theory and methods for general matrices, but a brief introduction
to functions of structured matrices is given in Section 14.1. The problem of computing
a function of a matrix times a vector, f (A)b, is of growing importance, though as yet
numerical methods are relatively undeveloped; Chapter 13 is devoted to this topic.
One of the pleasures of writing this book has been to explore the many connec-
tions between matrix functions and other subjects, particularly matrix analysis and
numerical analysis in general. These connections range from the expected, such as

xvii
xviii Preface

divided differences, the Kronecker product, and unitarily invariant norms, to the un-
expected, which include the Mandelbrot set, the geometric mean, partial isometries,
and the role of the Fréchet derivative beyond measuring problem sensitivity.
I have endeavoured to make this book more than just a monograph about matrix
functions, and so it includes many useful or interesting facts, results, tricks, and
techniques that have a (sometimes indirect) f (A) connection. In particular, the book
contains a substantial amount of matrix theory, as well as many historical references,
some of which appear not to have previously been known to researchers in the area.
I hope that the book will be found useful as a source of statements and applications
of results in matrix analysis and numerical linear algebra, as well as a reference on
matrix functions.
Four main themes pervade the book.
Role of the sign function. The matrix sign function has fundamental theoretical
and algorithmic connections with the matrix square root, the polar decomposition,
and, to a lesser extent, matrix pth roots. For example, a large class of iterations for
the matrix square root can be obtained from corresponding iterations for the matrix
sign function, and Newton’s method for the matrix square root is mathematically
equivalent to Newton’s method for the matrix sign function.
Stability. The stability of iterations for matrix functions can be effectively defined
and analyzed in terms of power boundedness of the Fréchet derivative of the iteration
function at the solution. Unlike some earlier, more ad hoc analyses, no assumptions
are required on the underlying matrix. General results (Theorems 4.18 and 4.19)
simplify the analysis for idempotent functions such as the matrix sign function and
the unitary polar factor.
Schur decomposition and Parlett recurrence. The use of a Schur decomposition
followed by reordering and application of the block form of the Parlett recurrence
yields a powerful general algorithm, with f -dependence restricted to the evaluation
of f on the diagonal blocks of the Schur form.
Padé approximation. For transcendental functions the use of Padé approximants,
in conjunction with an appropriate scaling technique that brings the matrix argument
close to the origin, yields an effective class of algorithms whose computational building
blocks are typically just matrix multiplication and the solution of multiple right-hand
side linear systems. Part of the success of this approach rests on the several ways
in which rational functions can be evaluated at a matrix argument, which gives the
scope to find a good compromise between speed and stability.
In addition to surveying, unifying, and sometimes improving existing results and
algorithms, this book contains new results. Some of particular note are as follows.
• Theorem 1.35, which relates f (αIm + AB) to f (αIn + BA) for A ∈ Cm×n and
B ∈ Cn×m and is an analogue for general matrix functions of the Sherman–
Morrison–Woodbury formula for the matrix inverse.
• Theorem 4.15, which shows that convergence of a scalar iteration implies con-
vergence of the corresponding matrix iteration when applied to a Jordan block,
under suitable assumptions. This result is useful when the matrix iteration
can be block diagonalized using the Jordan canonical form of the underlying
matrix, A. Nevertheless, we show in the context of Newton’s method for the
matrix square root that analysis via the Jordan canonical form of A does not
always give the strongest possible convergence result. In this case a stronger
result, Theorem 6.9, is obtained essentially by reducing the convergence analysis
to the consideration of the behaviour of the powers of a certain matrix.
Preface xix

• Theorems 5.13 and 8.19 on the stability of essentially all iterations for the ma-
trix sign function and the unitary polar factor, and the general results in The-
orems 4.18 and 4.19 on which these are based.
• Theorems 6.14–6.16 on the convergence of the binomial, Pulay, and Visser iter-
ations for the matrix square root.
• An improved Schur–Parlett algorithm for the matrix logarithm, given in Sec-
tion 11.6, which makes use of improved implementations of the inverse scaling
and squaring method in Section 11.5.

The Audience
The book’s main audience is specialists in numerical analysis and applied linear al-
gebra, but it will be of use to anyone who wishes to know something of the theory
of matrix functions and state of the art methods for computing them. Much of the
book can be understood with only a basic grounding in numerical analysis and linear
algebra.

Using the Book

The book can be used as the basis for a course on functions of matrices at the graduate
level. It is also a suitable reference for an advanced course on applied or numerical
linear algebra, which might cover particular topics such as definitions and properties
of f (A), or the matrix exponential and logarithm. It can be used by instructors at all
levels as a supplementary text from which to draw examples, historical perspective,
statements of results, and exercises. The book, and the subject itself, are particularly
well suited to self-study.
To a large extent the chapters can be read independently. However, it is ad-
visable first to become familiar with Sections 1.1–1.3, the first section of Chapter 3
(Conditioning), and most of Chapter 4 (Techniques for General Functions).
The Notes and References are an integral part of each chapter. In addition to
containing references, historical information, and further details, they include material
not covered elsewhere in the chapter and should always be consulted, in conjunction
with the index, to obtain the complete picture.
This book has been designed to be as easy to use as possible and is relatively
self-contained. Notation is summarized in Appendix A, while Appendix B (Back-
ground: Definitions and Useful Facts) reviews basic terminology and needed results
from matrix analysis and numerical analysis. When in doubt about the meaning of
a term the reader should consult the comprehensive index. Appendix C provides a
handy summary of operation counts for the most important matrix computation ker-
nels. Each bibliography entry shows on which pages the item is cited, which makes
browsing through the bibliography another route into the book’s content.
The exercises, labelled “problems”, are an important part of the book, and many
of them are new. Solutions, or occasionally a reference to where a solution can be
found, are given for almost every problem in Appendix E. Research problems given
at the end of some sets of problems highlight outstanding open questions.
A Web page for the book can be found at
http://www.siam.org/books/ot104
It includes
xx Preface

• The Matrix Function Toolbox for MATLAB, described in Appendix D. This

toolbox contains basic implementations of the main algorithms in the book.

• Updates relating to material in the book.

• A BibTEX database functions-of-matrices.bib containing all the references

in the bibliography.

Acknowledgments
A number of people have influenced my thinking about matrix functions. Discussions
with Ralph Byers in 1984, when he was working on the matrix sign function and I was
investigating the polar decomposition, first made me aware of connections between
these two important tools. The work on the matrix exponential of Cleve Moler and
Charlie Van Loan has been a frequent source of inspiration. Beresford Parlett’s ideas
on the exploitation of the Schur form and the adroit use of divided differences have
been a guiding light. Charles Kenney and Alan Laub’s many contributions to the
matrix function arena have been important in my own research and are reported on
many pages of this book. Finally, Nick Trefethen has shown me the importance of the
Cauchy integral formula and has offered valuable comments on drafts at all stages.
I am grateful to several other people for providing valuable help, suggestions, or
advice during the writing of the book:

Rafik Alam, Awad Al-Mohy, Zhaojun Bai, Timo Betcke, Rajendra Bhatia,
Tony Crilly, Philip Davies, Oliver Ernst, Andreas Frommer, Chun-Hua Guo,
Gareth Hargreaves, Des Higham, Roger Horn, Bruno Iannazzo, Ilse Ipsen,
Peter Lancaster, Jörg Liesen, Lijing Lin, Steve Mackey, Roy Mathias,
Volker Mehrmann, Thomas Schmelzer, Gil Strang, Françoise Tisseur, and
Andre Weideman.

Working with the SIAM staff on the publication of this book has been a pleasure. I
thank, in particular, Elizabeth Greenspan (acquisitions), Sara Murphy (acquisitions),
Lois Sellers (design), and Kelly Thomas (copy editing).
Research leading to this book has been supported by the Engineering and Physical
Sciences Research Council, The Royal Society, and the Wolfson Foundation.

Manchester Nicholas J. Higham

December 2007
Chapter 1
Theory of Matrix Functions

In this first chapter we give a concise treatment of the theory of matrix functions,
concentrating on those aspects that are most useful in the development of algorithms.
Most of the results in this chapter are for general functions. Results specific to
particular functions can be found in later chapters devoted to those functions.

1.1. Introduction
The term “function of a matrix” can have several different meanings. In this book we
are interested in a definition that takes a scalar function f and a matrix A ∈ Cn×n
and specifies f (A) to be a matrix of the same dimensions as A; it does so in a way
that provides a useful generalization of the function of a scalar variable f (z), z ∈ C.
Other interpretations of f (A) that are not our focus here are as follows:

• Elementwise operations on matrices, for example sin A = (sin aij ). These oper-
ations are available in some programming languages. For example, Fortran 95
supports “elemental operations” [423, ], and most of MATLAB’s elemen-
tary and special functions are applied in an elementwise fashion when given
matrix arguments. However, elementwise operations do not integrate well with
matrix algebra, as is clear from the fact that the elementwise square of A is not
equal to the matrix product of A with itself. (Nevertheless, the elementwise
product of two matrices, known as the Hadamard product or Schur product, is
a useful concept [294, ], [296, , Chap. 5].)

• Functions producing a scalar result, such as the trace, the determinant, the
spectral radius, the condition number κ(A) = kAk kA−1 k, and one particular
generalization to matrix arguments of the hypergeometric function [359, ].

• Functions mapping Cn×n to Cm×m that do not stem from a scalar function.
Examples include matrix polynomials with matrix coefficients, the matrix trans-
pose, the adjugate (or adjoint) matrix, compound matrices comprising minors
of a given matrix, and factors from matrix factorizations. However, as a special
case, the polar factors of a matrix are treated in Chapter 8.

• Functions mapping C to Cn×n , such as the transfer function f (t) = B(tI −

A)−1 C, for B ∈ Cn×m , A ∈ Cm×m , and C ∈ Cm×n .

Before giving formal definitions, we offer some motivating remarks. When f (t)
is a polynomial or rational function with scalar coefficients and a scalar argument,
t, it is natural to define f (A) by substituting A for t, replacing division by matrix

1
2 Theory of Matrix Functions

inversion (provided that the matrices to be inverted are nonsingular), and replacing
1 by the identity matrix. Then, for example,

1 + t2
f (t) = ⇒ f (A) = (I − A)−1 (I + A2 ) if 1 ∈
/ Λ(A).
1−t
Here, Λ(A) denotes the set of eigenvalues of A (the spectrum of A). Note that rational
functions of a matrix commute, so it does not matter whether we write (I − A)−1 (I +
A2 ) or (I + A2 )(I − A)−1 . If f has a convergent power series representation, such as

t2 t3 t4
log(1 + t) = t − + − + ···, |t| < 1,
2 3 4
we can again simply substitute A for t to define

A2 A3 A4
log(I + A) = A − + − + ···, ρ(A) < 1. (1.1)
2 3 4
Here, ρ denotes the spectral radius and the condition ρ(A) < 1 ensures convergence of
the matrix series (see Theorem 4.7). In this ad hoc fashion, a wide variety of matrix
functions can be defined. However, this approach has several drawbacks:

• In order to build up a general mathematical theory, we need a way of defining

f (A) that is applicable to arbitrary functions f .

• A particular formula may apply only for a restricted set of A, as in (1.1). If we

define f (A) from such a formula (rather than obtain the formula by applying
suitable principles to a more general definition) we need to check that it is
consistent with other definitions of the same function.

• For a multivalued function (multifunction), such as the logarithm or square

root, it is desirable to classify all possible f (A) that can be obtained by using
different branches of the function and to identify any distinguished values.

For these reasons we now consider general definitions of functions of a matrix.

1.2. Definitions of f (A)

There are many equivalent ways of defining f (A). We focus on three that are of
particular interest. These definitions yield primary matrix functions; nonprimary
matrix functions are discussed in Section 1.4.

1.2.1. Jordan Canonical Form

It is a standard result that any matrix A ∈ Cn×n can be expressed in the Jordan
canonical form

Z −1 AZ = J = diag(J1 , J2 , . . . , Jp ), (1.2a)
 
λk 1
 .. 
 λk . 
Jk = Jk (λk ) =  .  ∈ Cmk ×mk , (1.2b)
 .. 1 
λk
1.2 Definitions of f (A) 3

where Z is nonsingular and m1 + m2 + · · · + mp = n. The Jordan matrix J is unique

up to the ordering of the blocks Ji , but the transforming matrix Z is not unique.
Denote by λ1 , . . . , λs the distinct eigenvalues of A and let ni be the order of the
largest Jordan block in which λi appears, which is called the index of λi .
We need the following terminology.

Definition 1.1. 1 The function f is said to be defined on the spectrum of A if the

values
f (j) (λi ), j = 0: ni − 1, i = 1: s
exist. These are called the values of the function f on the spectrum of A.

In most cases of practical interest f is given by a formula, such as f (t) = et .

However, the following definition of f (A) requires only the values of fPon the spectrum
s
of A; it does not require any other information about f . Indeed any i=1 ni arbitrary
numbers can be chosen and assigned as the values of f on the spectrum of A. It is
only when we need to make statements about global properties such as continuity
that we will need to assume more about f .

Definition 1.2 (matrix function via Jordan canonical form). Let f be defined on
the spectrum of A ∈ Cn×n and let A have the Jordan canonical form (1.2). Then

f (A) := Zf (J)Z −1 = Z diag(f (Jk ))Z −1 , (1.3)

where  
′ f (mk −1) )(λk )
 f (λk ) f (λk ) ...
(mk − 1)! 
 
 .. .. 
f (Jk ) := 
 f (λk ) . . .
 (1.4)
 .. 
 . ′
f (λk ) 
f (λk )
 1/2 1

A simple example illustrates the definition. For the Jordan block J = 0 1/2
and f (x) = x3 , (1.4) gives
   
f (1/2) f ′ (1/2) 1/8 3/4
f (J) = = ,
0 f (1/2) 0 1/8

which is easily verified to be J 3 .

To provide some insight into this definition we make several comments. First,
the definition yields an f (A) that can be shown to be independent of the particular
Jordan canonical form that is used; see Problem 1.1.
Second, note that if A is diagonalizable then the Jordan canonical form reduces
to an eigendecomposition A = ZDZ −1 , with D = diag(λi ) and the columns of Z
eigenvectors of A, and Definition 1.2 yields f (A) = Zf (D)Z −1 = Z diag(f (λi ))Z −1 .
Therefore for diagonalizable matrices f (A) has the same eigenvectors as A and its
eigenvalues are obtained by applying f to those of A.
1 This is the terminology used by Gantmacher [203, , Chap. 5] and Lancaster and Tismenetsky

[371, , Chap. 9]. Note that the values depend not just on the eigenvalues but also on the maximal
Jordan block sizes ni .
4 Theory of Matrix Functions

Finally, we explain how (1.4) can be obtained from Taylor series considerations.
In (1.2b) write Jk = λk I +Nk ∈ Cmk ×mk , where Nk is zero except for a superdiagonal
of 1s. Note that for mk = 3 we have
   
0 1 0 0 0 1
Nk =  0 0 1  , Nk2 =  0 0 0  , Nk3 = 0.
0 0 0 0 0 0
In general, powering Nk causes the superdiagonal of 1s to move a diagonal at a time
towards the top right-hand corner, until at the mk th power it disappears: Ekmk = 0;
so Nk is nilpotent. Assume that f has a convergent Taylor series expansion
f (j) (λk )(t − λk )j
f (t) = f (λk ) + f ′ (λk )(t − λk ) + · · · + + ···.
j!
On substituting Jk ∈ Cmk ×mk for t we obtain the finite series
f (mk −1) (λk )Nkmk −1
f (Jk ) = f (λk )I + f ′ (λk )Nk + · · · + , (1.5)
(mk − 1)!
since all powers of Nk from the mk th onwards are zero. This expression is easily seen
to agree with (1.4). An alternative derivation of (1.5) that does not rest on a Taylor
series is given in the next section.
Definition 1.2 requires the function f to take well-defined values on the spectrum
of A—including values associated
√ with derivatives, where appropriate. Thus in the
case of functions such as t and log t it is implicit that a single branch has been
chosen in (1.4). Moreover, if an eigenvalue occurs in more than one Jordan block
then the same choice of branch must be made in each block. If the latter requirement
is violated then a nonprimary matrix function is obtained, as discussed in Section 1.4.

1.2.2. Polynomial Interpolation

The second definition is less obvious than the first, yet it has an elegant derivation
and readily yields some useful properties. We first introduce some background on
polynomials at matrix arguments.
The minimal polynomial of A ∈ Cn×n is defined to be the unique monic poly-
nomial ψ of lowest degree such that ψ(A) = 0. The existence of ψ is easily proved;
see Problem 1.5. A key property is that the minimal polynomial divides any other
polynomial p for which p(A) = 0. Indeed, by polynomial long division any such p can
be written p = ψq + r, where the degree of the remainder r is less than that of ψ.
But 0 = p(A) = ψ(A)q(A) + r(A) = r(A), and this contradicts the minimality of the
degree of ψ unless r = 0. Hence r = 0 and ψ divides p.
By considering the Jordan canonical form it is not hard to see that
s
Y
ψ(t) = (t − λi )ni , (1.6)
i=1

where, as in the previous section, λ1 , . . . , λs are the distinct eigenvalues of A and ni is

the dimension of the largest Jordan block in which λi appears. It follows immediately
that ψ is zero on the spectrum of A (in the sense of Definition 1.1).
For any A ∈ Cn×n and any polynomial p(t), it is obvious that p(A) is defined
(by substituting A for t) and that p is defined on the spectrum of A. Our interest in
polynomials stems from the fact that the values of p on the spectrum of A determine
p(A).
1.2 Definitions of f (A) 5

Theorem 1.3. For polynomials p and q and A ∈ Cn×n , p(A) = q(A) if and only if
p and q take the same values on the spectrum of A.
Proof. Suppose that two polynomials p and q satisfy p(A) = q(A). Then d = p−q
is zero at A so is divisible by the minimal polynomial ψ. In other words, d takes only
the value zero on the spectrum of A, that is, p and q take the same values on the
spectrum of A.
Conversely, suppose p and q take the same values on the spectrum of A. Then
d = p − q is zero on the spectrum of A and so must be divisible by the minimum
polynomial ψ, in view of (1.6). Hence d = ψr for some polynomial r, and since
d(A) = ψ(A)r(A) = 0, it follows that p(A) = q(A).
Thus it is a property of polynomials that the matrix p(A) is completely determined
by the values of p on the spectrum of A. It is natural to generalize this property to
arbitrary functions and define f (A) in such a way that f (A) is completely determined
by the values of f on the spectrum of A.

Definition 1.4 (matrix function via Hermite interpolation). Let f be defined on

the spectrum of A ∈ Cn×n and let ψ be the minimal polynomial of A. Then
f (A) := p(A), where p is the polynomial of degree less than
s
X
ni = deg ψ
i=1

that satisfies the interpolation conditions

p(j) (λi ) = f (j) (λi ), j = 0: ni − 1, i = 1: s. (1.7)

There is a unique such p and it is known as the Hermite interpolating polynomial.

√
An example is useful for clarification. Consider f (t) = t and
 
2 2
A= .
1 3
The eigenvalues are 1 and 4, so s = 2 and n1 = n2 = 1. We take f (t) as the principal
branch t1/2 of the square root function and find that the required interpolant satisfying
p(1) = f (1) = 1 and p(4) = f (4) = 2 is
t−4 t−1 1
p(t) = f (1) + f (4) = (t + 2).
1−4 4−1 3
Hence  
1 1 4 2
f (A) = p(A) = (A + 2I) = .
3 3 1 5
It is easily checked that f (A)2 = A. Note that the formula A1/2 = (A + 2I)/3 holds
more generally for any diagonalizable n × n matrix A having eigenvalues 1 and/or 4
(and hence having a minimal polynomial that divides ψ(t) = (t−1)(t−4))—including
the identity matrix. We are not restricted to using the same branch of the square root
function at each eigenvalue. For example, with f (1) = 1 and f (4) = −2 we obtain
p(t) = 2 − t and  
0 −2
f (A) = .
−1 −1
6 Theory of Matrix Functions

We make several remarks on this definition.

Remark 1.5. If q is a polynomial that satisfies the interpolation conditions (1.7)

and some additional interpolation conditions (at the same or different λi ) then q and
the polynomial p of Definition 1.4 take the same values on the spectrum of A. Hence
by Theorem 1.3, q(A) = p(A) = f (A). Sometimes, in constructing a polynomial q
for which q(A) = f (A), it is convenient to impose more interpolation conditions than
necessary—typically if the eigenvalues of A are known but the Jordan form is not
(see the next remark, and Theorem 3.7, for example). Doing so yields a polynomial
of higher degree than necessary but does not affect the ability of the polynomial to
produce f (A).

Remark 1.6. The Hermite interpolating polynomial p is given explicitly by the

Lagrange–Hermite formula
 ! 
Xs nXi −1 Y
1 (j)
p(t) =  φi (λi )(t − λi )j (t − λj )nj  , (1.8)
i=1 j=0
j!
j6=i

Q
where φi (t) = f (t)/ j6=i (t − λj )nj . For a matrix with distinct eigenvalues (ni ≡ 1,
s = n) this formula reduces to the familiar Lagrange form
n
X Yn  
t − λj
p(t) = f (λi )ℓi (t), ℓi (t) = . (1.9)
i=1 j=1
λi − λj
j6=i

An elegant alternative to (1.8) is the Newton divided difference form

p(t) = f [x1 ] + f [x1 , x2 ](t − x1 ) + f [x1 , x2 , x3 ](t − x1 )(t − x2 ) + · · ·

+ f [x1 , x2 , . . . , xm ](t − x1 )(t − x2 ) . . . (t − xm−1 ), (1.10)

where m = deg ψ and the set {xi }m i=1 comprises the distinct eigenvalues λ1 , . . . , λs
with λi having multiplicity ni . Here the f [. . .] denote divided differences, which are
defined in Section B.16. Another polynomial q for which f (A) = q(A) is given by
(1.10) with m = n and {xi }ni=1 the set of all n eigenvalues of A:

q(t) = f [λ1 ] + f [λ1 , λ2 ](t − λ1 ) + f [λ1 , λ2 , λ3 ](t − λ1 )(t − λ2 ) + · · ·

+ f [λ1 , λ2 , . . . , λn ](t − λ1 )(t − λ2 ) . . . (t − λn−1 ). (1.11)

This polynomial is independent of the Jordan structure of A and is in general of

higher degree than p. However, the properties of divided differences ensure that q
and p take the same values on the spectrum of A, so q(A) = p(A) = f (A).

Remark 1.7. This definition explicitly makes f (A) a polynomial in A. It is impor-

tant to note, however, that the polynomial p depends on A, through the values of f on
the spectrum of A, so it is not the case that f (A) ≡ q(A) for some fixed polynomial
q independent of A.

Remark 1.8. If f is given by a power series, Definition 1.4 says that f (A) is never-
theless expressible as a polynomial in A of degree at most n−1. Another way to arrive
at this conclusion is as follows. The Cayley–Hamilton theorem says that any matrix
1.2 Definitions of f (A) 7

satisfies its own characteristic equation: q(A) = 0,2 where q(t) = det(tI − A) is the
characteristic polynomial. This theorem follows immediately from the fact that the
minimal polynomial ψ divides q (see Problem 1.18 for another proof). Hence the nth
power of A, and inductively all higher powers, are expressible as a linear combination
of I, A, . . . , An−1 . Thus any power series in A can be reduced to a polynomial in A
of degree at most n − 1. This polynomial is rarely of an elegant form or of practical
interest; exceptions are given in (1.16) and Problem 10.13.

Remark 1.9. It is natural to ask whether f (A) is real whenever A is real. By

considering real, diagonal A, it is clear that for this condition to hold it is necessary
that the scalar function f is real on the subset of the real line on which it is defined.
Since the nonreal eigenvalues of a real matrix occur in complex conjugate pairs λ, λ
it is reasonable also to assume that f (λ), f (λ) form a complex conjugate pair, and
likewise for higher derivatives. The interpolation conditions (1.7) can be written in
the form of a dual (confluent) Vandermonde system of equations whose solution is a
vector comprising the coefficients of r. Considering, for a moment, a 2 × 2 real matrix
with eigenvalues λ, λ (λ 6= λ) this system is, under the assumption on f above,
      
1 λ r0 f (λ) f (λ)
= = .
1 λ r1 f (λ) f (λ)
 1 1
Premultiplying by the matrix −i i /2 yields the system
    
1 Re λ r0 Re f (λ)
=
0 Im λ r1 Im f (λ)

with a real coefficient matrix and right-hand side. We conclude that r has real
coefficients and hence f (A) = p(A) is real when A is real. This argument extends to
real n × n matrices under the stated condition on f . As a particular example, we can
conclude that if A is real and nonsingular with no eigenvalues on the negative real
axis then A has a real square root and a real logarithm. For a full characterization
of the existence of real square roots and logarithms see Theorem 1.23. Equivalent
conditions to f (A) being real for real A when f is analytic are given in Theorem 1.18.

Remark 1.10. We can derive directly from Definition 1.4 the formula (1.4) for a
function of the Jordan block Jk in (1.2). It suffices to note that the interpolation
conditions are p(j) (λk ) = f (j) (λk ), j = 0: mk − 1, so that the required Hermite
interpolating polynomial is

f ′′ (λk )(t − λk )2 f (mk −1) (λk )(t − λk )mk −1

p(t) = f (λk ) + f ′ (λk )(t − λk ) + + ··· + ,
2! (mk − 1)!

and then to evaluate p(Jk ), making use of the properties of the powers of Nk noted
in the previous section (cf. (1.5)).

1.2.3. Cauchy Integral Theorem

Perhaps the most concise and elegant definition of a function of a matrix is a gener-
alization of the Cauchy integral theorem.
2 It is incorrect to try to prove the Cayley–Hamilton theorem by “q(A) = det(AI − A) = 0”.
8 Theory of Matrix Functions

Definition 1.11 (matrix function via Cauchy integral). For A ∈ Cn×n ,

Z
1
f (A) := f (z)(zI − A)−1 dz, (1.12)
2πi Γ

where f is analytic on and inside a closed contour Γ that encloses Λ(A).

The integrand contains the resolvent, (zI − A)−1 , which is defined on Γ since Γ
is disjoint from the spectrum of A.
This definition leads to short proofs of certain theoretical results and has the
advantage that it can be generalized to operators.

1.2.4. Equivalence of Definitions

Our three definitions are equivalent, modulo the requirement in the Cauchy integral
definition that f be analytic in a region of the complex plane containing the spectrum.

Theorem 1.12. Definition 1.2 (Jordan canonical form) and Definition 1.4 (Hermite
interpolation) are equivalent. If f is analytic then Definition 1.11 (Cauchy integral )
is equivalent to Definitions 1.2 and 1.4.

Proof. Definition 1.4 says that f (A) = p(A) for a Hermite interpolating poly-
nomial p satisfying (1.7). If A has the Jordan form (1.2) then f (A) = p(A) =
p(ZJZ −1 ) = Zp(J)Z −1 = Z diag(p(Jk ))Z −1 , just from elementary properties of ma-
trix polynomials. But since p(Jk ) is completely determined by the values of p on the
spectrum of Jk , and these values are a subset of the values of p on the spectrum of A,
it follows from Remark 1.5 and Remark 1.10 that p(Jk ) is precisely (1.4). Hence the
matrix f (A) obtained from Definition 1.4 agrees with that given by Definition 1.2.
For the equivalence of Definition 1.11 with the other two definitions, see Horn and
Johnson [296, , Thm. 6.2.28].
We will mainly use (for theoretical purposes) Definitions 1.2 and 1.4. The polyno-
mial interpolation definition, Definition 1.4, is well suited to proving basic properties
of matrix functions, such as those in Section 1.3, while the Jordan canonical form
definition, Definition 1.2, excels for solving matrix equations such as X 2 = A and
eX = A. For many purposes, such as the derivation of the formulae in the next
section, either of the definitions can be used.
In the rest of the book we will refer simply to “the definition of a matrix function”.

1.2.5. Example: Function of Identity Plus Rank-1 Matrix

To illustrate the theory, and the consistency of the different ways of defining f (A), it is
instructive to consider the cases where A is a rank-1 matrix and a rank-1 perturbation
of the identity matrix.
Consider, first, a rank-1 matrix A = uv ∗ . The interpolation definition provides
the easiest way to obtain f (A). We first need to determine the Jordan structure of
A. If v ∗ u 6= 0 then A has an eigenpair (v ∗ u, u) and 0 is a semisimple eigenvalue of
multiplicity n − 1. The interpolation conditions (1.7) are therefore simply

p(v ∗ u) = f (v ∗ u), p(0) = f (0),

1.2 Definitions of f (A) 9

and so
t − v∗ u t−0
p(t) = f (0) + ∗ f (v ∗ u).
0 − v∗ u v u−0
Hence
f (0) uv ∗
f (A) = p(A) = − ∗ uv ∗ + f (0)I + f (v ∗ u) ∗
 v u∗  v u
f (v u) − f (0)
= f (0)I + uv ∗ (1.13)
v∗ u − 0
= f (0)I + f [v ∗ u, 0] uv ∗ .

We have manipulated the expression into this form involving a divided difference
because it is suggestive of what happens when v ∗ u = 0. Indeed f [0, 0] = f ′ (0) and so
when v ∗ u = 0 we may expect that f (A) = f (0)I + f ′ (0)uv ∗ . To confirm this formula,
note that v ∗ u = 0 implies that the spectrum of A consists entirely of 0 and that
A2 = (v ∗ u)uv ∗ = 0. Hence, assuming A 6= 0, A must have one 2 × 2 Jordan block
corresponding to the eigenvalue 0, with the other n − 2 zero eigenvalues occurring in
1 × 1 Jordan blocks. The interpolation conditions (1.7) are therefore

p(0) = f (0), p′ (0) = f ′ (0),

and so p(t) = f (0) + tf ′ (0). Therefore p(A) = f (0)I + f ′ (0)uv ∗ , as anticipated. To

summarize, the formula

f (uv ∗ ) = f (0)I + f [v ∗ u, 0] uv ∗ (1.14)

is valid for all u and v. We could have obtained this formula directly by using the
divided difference form (1.10) of the Hermite interpolating polynomial r, but the
derivation above gives more insight.
We now show how the formula is obtained from Definition 1.2 when v ∗ u 6= 0 (for
the case v ∗ u = 0 see Problem 1.15). The Jordan canonical form can be written as
 ∗ ∗ 
v /(v u)
A = [ u X ] diag(v ∗ u, 0, . . . , 0) ,
Y

where X and Y are chosen so that AX = 0, [ u X ] is nonsingular, and

 ∗ ∗ 
v /(v u)
[u X ] = I. (1.15)
Y

Hence
 
v ∗ /(v ∗ u) uv ∗
f (A) = [ u X ] diag(f (v ∗ u), f (0), . . . , f (0)) = f (v ∗ u) ∗ + f (0)XY.
Y v u

But XY = I − uv ∗ /(v ∗ u), from (1.15), and hence (1.13) is recovered.

If f has a power series expansion then (1.14) can also be derived by direct substi-
tution into the power series, using Ak = (v ∗ u)k−1 uv ∗ .
The Cauchy integral definition (1.12) can also be used to derive (1.14) when f is
analytic, by using the Sherman–Morrison formula (B.11).
Even in the rank-1 case issues of nonexistence
√ are present. For f the square root,
(1.14) provides the two square roots uv ∗ / v ∗ u for v ∗ u 6= 0. But if v ∗ u = 0 the
10 Theory of Matrix Functions

formula breaks down because f ′ (0) is undefined. In this  case

 A has no square roots—
essentially because the Jordan form of A has a block 00 10 , which has no square roots.
Also note that if u and v are real, f (uv ∗ ) will be real only if f [v ∗ u, 0] is real.
Analysis very similar to that above provides a formula for a function of the identity
plus a rank-1 matrix that generalizes (1.14) (see Problem 1.16):

f (αI + uv ∗ ) = f (α)I + f [α + v ∗ u, α]uv ∗ . (1.16)

For a more general result involving a perturbation of arbitrary rank see Theorem 1.35.

1.2.6. Example: Function of Discrete Fourier Transform Matrix

Another interesting example is provided by the discrete Fourier transform (DFT)
matrix
1  n
Fn = 1/2 exp(−2πi(r − 1)(s − 1)/n) ∈ Cn×n . (1.17)
n r,s=1

Fn is a very special matrix: it is complex symmetric and unitary (and is a Vander-

monde matrix based on the roots of unity). Let us see how to evaluate f (Fn ).
The DFT has the special property that Fn4 = I, from which it follows that the
minimal polynomial of Fn is ψ(t) = t4 − 1 for n ≥ 4. The interpolating polynomial
in (1.7) therefore has degree 3 for all n ≥ 4 and can be expressed in Lagrange form
(1.9) as

1
p(t) = f (1) (t + 1) (t − i) (t + i) − f (−1) (t − 1) (t − i) (t + i)
4 
+ if (i) (t − 1) (t + 1) (t + i) − if (−i) (t − 1) (t + 1) (t − i) . (1.18)

Thus f (A) = p(A), and in fact this formula holds even for n = 1: 3, since incorpo-
rating extra interpolation conditions does not affect the ability of the interpolating
polynomial to yield f (A) (see Remark 1.5). This expression can be quickly evaluated
in O(n2 log n) operations because multiplication of a vector by Fn can be carried out
in O(n log n) operations using the fast Fourier transform (FFT).
Because Fn is unitary and hence normal, Fn is unitarily diagonalizable: Fn =
QDQ∗ for some unitary Q and diagonal D. (Indeed, any matrix with minimal polyno-
mial ψ(t) has distinct eigenvalues and so is diagonalizable.) Thus f (Fn ) = Qf (D)Q∗ .
However, this formula requires knowledge of Q and D and so is much more compli-
cated to use than (1.18).

1.3. Properties
The sign of a good definition is that it leads to the properties one expects or hopes
for, as well as some useful properties that are less obvious. We collect some general
properties that follow from the definition of f (A).

Theorem 1.13. Let A ∈ Cn×n and let f be defined on the spectrum of A. Then
(a) f (A) commutes with A;
(b) f (AT ) = f (A)T ;
(c) f (XAX −1 ) = Xf (A)X −1 ;
(d) the eigenvalues of f (A) are f (λi ), where the λi are the eigenvalues of A;
1.3 Properties 11

(e) if X commutes with A then X commutes with f (A);

(f) if A = (Aij ) is block triangular then F = f (A) is block triangular with the
same block structure as A, and Fii = f (Aii );
(g) if A = diag(A11 , A22 , . . . , Amm ) is block diagonal then


f (A) = diag f (A11 ), f (A22 ), . . . , f (Amm ) ;

(h) f (Im ⊗ A) = Im ⊗ f (A), where ⊗ is the Kronecker product;

(i) f (A ⊗ Im ) = f (A) ⊗ Im .

Proof. Definition 1.4 implies that f (A) is a polynomial in A, p(A) say. Then
f (A)A = p(A)A = Ap(A) = Af (A), which proves the first property. For (b) we have
f (A)T = p(A)T = p(AT ) = f (AT ), where the last equality follows from the fact that
the values of f on the spectrum of A are the same as the values of f on the spectrum of
AT . (c) and (d) follow immediately from Definition 1.2. (e) follows from (c) when X is
nonsingular; more generally it is obtained from Xf (A) = Xp(A) = p(A)X = f (A)X.
For (f), f (A) = p(A) is clearly block triangular and its ith diagonal block is p(Aii ).
Since p interpolates f on the spectrum of A it interpolates f on the spectrum of each
Aii , and hence p(Aii ) = f (Aii ). (g) is a special case of (f). (h) is a special case of
(g), since Im ⊗ A = diag(A, A, . . . , A). Finally, we have A ⊗ B = Π(B ⊗ A)Π T for a
permutation matrix Π, and so

f (A⊗Im ) = f (Π(Im ⊗A)Π T ) = Πf (Im ⊗A)Π T = Π(Im ⊗f (A))Π T = f (A)⊗Im .

Theorem 1.14 (equality of two matrix functions). With the notation of Section 1.2,
f (A) = g(A) if and only if

f (j) (λi ) = g (j) (λi ), j = 0: ni − 1, i = 1: s.

Equivalently, f (A) = 0 if and only if

f (j) (λi ) = 0, j = 0: ni − 1, i = 1: s.

Proof. This result is immediate from Definition 1.2 or Definition 1.4.

The next three results show how different functions interact in combination. It
is worth emphasizing why these results are nontrivial. It is not immediate from
any of the definitions of f (A) how to evaluate at A a composite function, such as
f (t) = e−t sin(t) or g(t) = t − (t1/2 )2 . Replacing “t” by “A” in these expressions
needs to be justified, as does the deduction g(A) = 0 from g(t) = 0. However, in
any polynomial (which may be an expression made up from other polynomials) the
“t → A” substitution is valid, and the proofs for general functions therefore work
by reducing to the polynomial case. The first result concerns a sum or product of
functions.

Theorem 1.15 (sum and product of functions). Let f and g be functions defined on
the spectrum of A ∈ Cn×n .
(a) If h(t) = f (t) + g(t) then h(A) = f (A) + g(A).
(b) If h(t) = f (t)g(t) then h(A) = f (A)g(A).
12 Theory of Matrix Functions

Proof. Part (a) is immediate from any of the definitions of h(A). For part
(b), let p and q interpolate f and g on the spectrum of A, so that p(A) = f (A)
and q(A) = g(A). By differentiating and using the product rule we find that the
functions h(t) and r(t) = p(t)q(t) have the same values on the spectrum of A. Hence
h(A) = r(A) = p(A)q(A) = f (A)g(A).
The next result generalizes the previous one and says that scalar functional re-
lationships of a polynomial nature are preserved by matrix functions. For example
sin2 (A) + cos2 (A) = I, (A1/p )p = A, and eiA = cos(A) + i sin(A). Of course, gener-
alizations of scalar identities that involve two or more noncommuting matrices may
fail; for example, eA+B , eA eB , and eB eA are in general all different (see Section 10.1).

Theorem 1.16 (polynomial functional identities). Let Q(u1 , . . . , ut ) be a polynomial

in u1 , . . . , ut and let f1 , . . . , ft be functions defined on the spectrum of A ∈ Cn×n .
If f (λ) = Q(f1 (λ), . . . , ft (λ)) takes zero values on the spectrum of A then f (A) =
Q(f1 (A), . . . , ft (A)) = 0.
Proof. Let the polynomials p1 , . . . , pt interpolate f1 , . . . , ft on the spectrum of A.
Then pi (A) = fi (A), i = 1: t. Let p(λ) = Q(p1 (λ), . . . , pt (λ)), and note that p(λ) is a
polynomial in λ. Since pi and fi take the same values on the spectrum of A, so do f
and p. But f takes zero values on the spectrum of A, by assumption, and hence so
does p. Therefore, by Theorem 1.14, f (A) = p(A) = 0.
The next result concerns a composite function in which neither of the constituents
need be a polynomial.

Theorem 1.17 (composite function). Let A ∈ Cn×n and let the distinct eigenvalues
of A be λ1 , . . . , λs with indices n1 , . . . , ns . Let h be defined on the spectrum of A (so
that the values h(j) (λi ), j = 0: ni − 1, i = 1: s exist) and let the values g (j) (h(λi )),
j = 0: ni − 1, i = 1: s exist. Then f (t) = g(h(t)) is defined on the spectrum of A and
f (A) = g(h(A)).

Proof. Let µk = h(λk ), k = 1: s. Since

f (λk ) = g(µk ), (1.19a)

′ ′ ′
f (λk ) = g (µk )h (λk ), (1.19b)
..
.
(nk −1)
f (λk ) = g (nk −1) (µk )h′ (λk )nk −1 + · · · + g ′ (µk )h(nk −1) (λk ), (1.19c)

and all the derivatives on the right-hand side exist, f is defined on the spectrum of A.
Let p(t) be any polynomial satisfying the interpolation conditions

p(j) (µi ) = g (j) (µi ), j = 0: ni − 1, i = 1: s. (1.20)

From Definition 1.2 it is clear that the indices of the eigenvalues µ1 , . . . , µs of h(A)
are at most n1 , . . . , ns , so the values on the right-hand side of (1.20) contain the
values of g on the spectrum of B = h(A); thus g(B) is defined and p(B) = g(B). It
now follows by (1.19) and (1.20) that the values of f (t) and p(h(t)) coincide on the
spectrum of A. Hence by applying Theorem 1.16 to Q(f (t), h(t)) = f (t) − p(h(t)) we
conclude that
f (A) = p(h(A)) = p(B) = g(B) = g(h(A)),
1.3 Properties 13

as required.
The assumptions in Theorem 1.17 on g for f (A) to exist are stronger than neces-
sary in certain cases where a Jordan block of Asplits
 under evaluation of h. Consider,
for example, g(t) = t1/3 , h(t) = t2 , and A = 00 10 . The required derivative g ′ (0) in
Theorem 1.17 does not exist, but f (A) = (A2 )1/3 = 0 nevertheless does exist. (A
full description of the Jordan canonical form of f (A) in terms of that of A is given in
Theorem 1.36.)
Theorem 1.17 implies that exp(log A) = A, provided that log is defined on the
spectrum of A. However, log(exp(A)) = A does not hold unless the spectrum of
A satisfies suitable restrictions, since the scalar relation log(et ) = t is likewise not
generally true in view of et = et+2kπi for any integer k; see Problem 1.39.
Although f (AT ) = f (A)T always holds (Theorem 1.13 (b)), the property f (A∗ ) =
f (A)∗ does not. The next result says essentially that for an analytic function f
defined on a suitable domain that includes a subset S of the real line, f (A∗ ) = f (A)∗
holds precisely when f maps S back into the real line. This latter condition also
characterizes when A real implies f (A) real (cf. the sufficient conditions given in
Remark 1.9).

Theorem 1.18 (Higham, Mackey, Mackey, and Tisseur). Let f be analytic on an open
subset Ω ⊆ C such that each connected component of Ω is closed under conjugation.
Consider the corresponding matrix function f on its natural domain in Cn×n , the set
D = { A ∈ Cn×n : Λ(A) ⊆ Ω }. Then the following are equivalent:
(a) f (A∗ ) = f (A)∗ for all A ∈ D.
(b) f (A) = f (A) for all A ∈ D.
(c) f (Rn×n ∩ D) ⊆ Rn×n .
(d) f (R ∩ Ω) ⊆ R.

Proof. The first two properties are obviously equivalent, in view of Theorem 1.13 (b).
Our strategy is therefore to show that (b) ⇒ (c) ⇒ (d) ⇒ (b).
(b) ⇒ (c): If A ∈ Rn×n ∩ D then

f (A) = f (A) (since A ∈ Rn×n )

= f (A) (given),

so f (A) ∈ Rn×n , as required.

(c) ⇒ (d): If λ ∈ R ∩ Ω then λI ∈ Rn×n ∩ D. So f (λI) ∈ Rn×n by (c), and hence,
since f (λI) = f (λ)I, f (λ) ∈ R.
The argument that (d) ⇒ (b) is more technical and involves complex analysis; see
Higham, Mackey, Mackey, and Tisseur [283, , Thm. 3.2].
Our next result shows that although the definition of f (A) utilizes only the values
of f on the spectrum of A (the values assumed by f elsewhere in C being arbitrary),
f (A) is a continuous function of A under suitable assumptions on f and the domain.

Theorem 1.19 (continuity). Let D be an open subset of R or C and let f be n − 1

times continuously differentiable on D. Then f (A) is a continuous matrix function
on the set of matrices A ∈ Cn×n with spectrum in D.

Proof. See Horn and Johnson [296, , Thm. 6.2.27 (1)], and Mathias [412,
, Lem. 1.1] for the conditions as stated here.
14 Theory of Matrix Functions

For continuity of f (A) on the set of normal matrices just the continuity of f is
sufficient [296, , Thm. 6.2.37].
Our final result shows that under mild conditions to check the veracity of a matrix
identity it suffices to check it for diagonalizable matrices.

Theorem 1.20. Let f satisfy the conditions of Theorem 1.19. Then f (A) = 0 for all
A ∈ Cn×n with spectrum in D if and only if f (A) = 0 for all diagonalizable A ∈ Cn×n
with spectrum in D.
Proof. See Horn and Johnson [296, , Thm. 6.2.27 (2)].
For an example of the use of Theorem 1.20 see the proof of Theorem 11.1. The-
orem 1.13 (f) says that block triangular structure is preserved by matrix functions.
An explicit formula can be given for an important instance of the block 2 × 2 case.

Theorem 1.21. Let f satisfy the conditions of Theorem 1.19 with D containing the
spectrum of n−1 1
 
n−1 B c
A= ∈ Cn×n .
1 0 λ
Then  
f (B) g(B)c
f (A) = , (1.21)
0 f (λ)
/ Λ(B) then g(B) = (B − λI)−1 (f (B) −
where g(z) = f [z, λ]. In particular, if λ ∈
f (λ)I).
Proof. We need only to demonstrate the formula for the (1,2) block F12 of f (A).
Equating (1,2) blocks in f (A)A = Af (A) (Theorem 1.13 (a)) yields BF12 + cf (λ) =
f (B)c + F12 λ, or (B − λI)F12 = (f (B) − f (λ)I)c. If λ ∈
/ Λ(B) the result is proved.
Otherwise, the result follows by a continuity argument: replace λ by λ(ǫ) = λ + ǫ, so
that λ(ǫ) ∈
/ Λ(B) for sufficiently small ǫ, let ǫ → 0, and use the continuity of divided
differences and of f (A).
For an expression for a function of a general block 2 × 2 block triangular matrix
see Theorem 4.12.

1.4. Nonprimary Matrix Functions

One of the main uses of matrix functions is for solving nonlinear matrix equations,
g(X) = A. Two particular cases are especially important. We will call any solution
of X 2 = A a square root of A and any solution of eX = A a logarithm of A. We
naturally turn to the square root and logarithm functions to solve the latter two
equations. But for certain matrices A some of the solutions of g(X) = A are not
obtainable as a primary matrix function of A, that is, they cannot be produced by
our (three equivalent) definitions of f (A) (with f = g −1 or otherwise). These X are
examples of nonprimary matrix functions. Informally, a nonprimary matrix function
is a “matrix equation solving function” that cannot be expressed as a primary matrix
function; we will not try to make this notion precise.
Suppose we wish to find square roots of
 
1 0
A= ,
0 1
1.4 Nonprimary Matrix Functions 15

√
that is, solve X 2 = A. Taking f (t) = t, the interpolation
√ conditions in Defini-
tions 1.4 are (with s = 1, n1 = 1) simply p(1) = 1. The interpolating polynomial
is therefore either p(t) = 1 or p(t) = −1, corresponding to the two square roots of
1, giving I and −I as square roots of A. Both of these square roots are, trivially,
polynomials in A. Turning to Definition 1.2, the matrix A is already in Jordan form
with two 1 × 1 Jordan blocks, and the definition provides the same two square roots.
However, if we ignore the prescription at the end of Section 1.2.1 about the choice of
branches then we can obtain two more square roots,
   
−1 0 1 0
, ,
0 1 0 −1
in which the two eigenvalues 1 have been sent to different square roots. Moreover,
since A = ZIZ −1 is a Jordan canonical form for any nonsingular Z, Definition 1.2
yields the square roots
   
−1 0 1 0
Z Z −1 , Z Z −1 , (1.22)
0 1 0 −1
and these formulae provide an infinity of square roots, because only for diagonal Z
are the matrices in (1.22) independent of Z. Indeed, one infinite family of square
roots of A comprises the Householder reflections
 
cos θ sin θ
H(θ) = , θ ∈ [0, 2π].
sin θ − cos θ
Definitions 1.2, 1.4, and 1.11 yield primary matrix functions. In most applications
it is primary matrix functions that are of interest, and virtually all the existing the-
ory and available methods are for such functions. Nonprimary matrix functions are
obtained from Definition 1.2 when two equal eigenvalues in different Jordan blocks
are mapped to different values of f ; in other words, different branches of f are taken
for different Jordan blocks with the same eigenvalue. The function obtained thereby
depends on the matrix Z in (1.3). This possibility arises precisely when the function is
multivalued and the matrix is derogatory, that is, the matrix has multiple eigenvalues
and an eigenvalue appears in more than one Jordan block.
Unlike primary matrix functions, nonprimary ones are not expressible as polyno-
mials in the matrix. However, a nonprimary function obtained from Definition 1.2,
using the prescription in the previous paragraph, nevertheless commutes with the ma-
trix. Such a function has the form X = Z diag(fk (Jk ))Z −1 , where A = Z diag(Jk )Z −1
is a Jordan canonical form and where the notation fk denotes that the branch of f
taken depends on k. Then XA = AX, because fk (Jk ) is a primary matrix function
and so commutes with Jk .
But note that not all nonprimary matrix functions are obtainable
  from the Jordan
canonical form prescription above. For example, A = 00 00 has the square root
 
X = 00 10 , and X is a Jordan block larger than the 1 × 1 Jordan blocks of A. This
example also illustrates that a nonprimary function can have the same spectrum as a
primary function, and so in general a nonprimary function cannot be identified from
its spectrum alone.
Nonprimary functions can be needed when, for a matrix A depending on a param-
eter t, a smooth curve of functions f (A(t)) needs to be computed and eigenvalues of
A(t) coalesce. Suppose we wish to compute square roots of
 
cos θ sin θ
G(θ) =
− sin θ cos θ
16 Theory of Matrix Functions

as θ varies from 0 to 2π. Since multiplication of a vector by G(θ) represents a rotation

through θ radians clockwise, G(θ/2) is the natural square root. However, for θ = π,
   
−1 0 0 1
G(π) = , G(π/2) = .
0 −1 −1 0

The only primary square roots of G(π) are ±iI, which are nonreal. While it is
nonprimary, G(π/2) is the square root we need in order to produce a smooth curve
of square roots.
An example of an application where nonprimary logarithms arise is the embed-
dability problems for Markov chains (see Section 2.3).
A primary matrix function with a nonprimary flavour is the matrix sign function
(see Chapter 5), which for a matrix A ∈ Cn×n is a (generally) nonprimary square
root of I that depends on A.
Unless otherwise stated, f (A) denotes a primary matrix function throughout this
book.

1.5. Existence of (Real) Matrix Square Roots and Logarithms

If A is nonsingular, or singular with a semisimple zero eigenvalue, then the square
root function is defined on the spectrum of A and so primary square roots exist. If A
is singular with a defective zero eigenvalue then while it has no primary square roots
it may have nonprimary ones. The existence of a square root of either type can be
neatly characterized in terms of null spaces of powers of A.

Theorem 1.22 (existence of matrix square root). A ∈ Cn×n has a square root if and
only if in the “ascent sequence” of integers d1 , d2 , . . . defined by

di = dim(null(Ai )) − dim(null(Ai−1 ))

no two terms are the same odd integer.

Proof. See Cross and Lancaster [122, ] or Horn and Johnson [296, , Cor.
6.4.13].
To illustrate, consider a Jordan block J ∈ Cm×m with eigenvalue zero. We have
dim(null(J 0 )) = 0, dim(null(J)) = 1, dim(null(J 2 )) = 2, . . . , dim(null(J m )) = m,
and so the ascent sequence comprises m 1s. Hence Jk does not have a square root
unless m = 1. However, the matrix
 
0 1 0
0 0 0 (1.23)
0 0 0

has ascent sequence 2, 1, 0, . . . and so does have a square root—for example, the matrix
 
0 0 1
0 0 0 (1.24)
0 1 0

(which is the 3 × 3 Jordan block with eigenvalue 0 with rows and columns 2 and 3
interchanged).
1.6 Classification of Matrix Square Roots and Logarithms 17

Another important existence question is “If A is real does there exist a real f (A),
either primary or nonprimary?” For most common functions the answer is clearly yes,
by considering a power series representation. For the square root and logarithm the
answer is not obvious; the next result completes the partial answer to this question
given in Remark 1.9 and Theorem 1.18.

Theorem 1.23 (existence of real square root and real logarithm).

(a) A ∈ Rn×n has a real square root if and only if it satisfies the condition of
Theorem 1.22 and A has an even number of Jordan blocks of each size for every
negative eigenvalue.
(b) The nonsingular matrix A ∈ Rn×n has a real logarithm if and only if A has
an even number of Jordan blocks of each size for every negative eigenvalue.
(c) If A ∈ Rn×n has any negative eigenvalues then no primary square root or
logarithm is real.

Proof. For the last part consider the real Schur decomposition, QT AQ = R (see
Section B.5), where Q ∈ Rn×n is orthogonal and R ∈ Rn×n is upper quasi-triangular.
Clearly, f (A) is real if and only if QT f (A)Q = f (R) is real, and a primary matrix
function f (R) is block upper triangular with diagonal blocks f (Rii ). If A has a
negative real eigenvalue then some Rii is 1 × 1 and negative, making f (Rii ) nonreal
for f the square root and logarithm.
The result of (b) is due to Culver [126, ], and the proof for (a) is similar; see
also Horn and Johnson [296, , Thms. 6.4.14, 6.4.15] and Nunemacher [451, ].

Theorem 1.23 implies that −In has a real, nonprimary square root and logarithm
for every even n. For some insight into part (a), note that if A has two Jordan blocks
J of the same size then its Jordan matrix has a principal submatrix of the form
 J 0   0 I 2
0J = J 0 .

1.6. Classification of Matrix Square Roots and Logarithms

The theory presented above provides a means for identifying some of the solutions
to nonlinear matrix equations such as X 2 = A, eX = A, and cos(X) = A, since in
each case X can be expressed as a function of A. However, more work is needed to
classify all the solutions. In particular, the possibility remains that there are solutions
X that have a spectrum of the form required for a primary matrix function but that
are not primary matrix functions according to our definition. This possibility can be
ruled out when the inverse of the function of interest has a nonzero derivative on the
spectrum of X.
We will concentrate on the matrix square root. Entirely analogous arguments
apply to the logarithm, which we briefly discuss, and the matrix pth root, which is
treated in Section 7.1. For f the square root function and λk 6= 0 we write
(j ) (j )
Lk k ≡ Lk k (λk ) = f (Jk (λk )),

where f (Jk (λk )) is given in (1.4) and where jk = 1 or 2 denotes the branch of f ; thus
(1) (2)
Lk = −Lk . Our first result characterizes all square roots.
18 Theory of Matrix Functions

Theorem 1.24 (Gantmacher). Let A ∈ Cn×n be nonsingular with the Jordan canon-
ical form (1.2). Then all solutions to X 2 = A are given by
(j ) (j )
X = ZU diag(L1 1 , L2 2 , . . . , L(jp)
p )U
−1 −1
Z , (1.25)

where U is an arbitrary nonsingular matrix that commutes with J.

Proof. Let X be any square root of A. Since A is nonsingular so is X, and hence

the derivative of the function x2 is nonzero at the eigenvalues of X. By Theorem 1.36,
given that A has the Jordan canonical form J = diag(J1 (λ1 ), J2 (λ2 ), . . . , Jp (λp )), X
must have the Jordan canonical form

JX = diag(J1 (µ1 ), J2 (µ2 ), . . . , Jp (µp )), (1.26)

where µ2k = λk , k = 1: p.
Now consider the matrix
(j ) (j )
L = diag(L1 1 , L2 2 , . . . , L(jp)
p ), (1.27)

(j )
where we choose the jk so that Lk k has eigenvalue µk for each k. Since L is a
square root of J, by the same argument as above L must have the Jordan canonical
form JX . Hence X = W LW −1 for some nonsingular W . From X 2 = A we have
W JW −1 = W L2 W −1 = ZJZ −1 , which can be rewritten as (Z −1 W )J = J(Z −1 W ).
Hence U = Z −1 W is an arbitrary matrix that commutes with J, which completes the
proof.
The structure of the matrix U in Theorem 1.24 is described in the next result.

Theorem 1.25 (commuting matrices). Let A ∈ Cn×n have the Jordan canonical
form (1.2). All solutions of AX = XA are given by X = ZW Z −1 , where W = (Wij )
with Wij ∈ Cmi ×mj (partitioned conformably with J in (1.2)) satisfies

0, λi 6= λj ,
Wij =
Tij , λi = λj ,

where Tij is an arbitrary upper trapezoidal Toeplitz matrix which, for mi < mj , has
the form Tij = [0, Uij ], where Uij is square.

Proof. See Lancaster and Tismenetsky [371, , Thm. 12.4.1].

Next we refine Theorem 1.24 to classify the square roots into primary and non-
primary square roots.

Theorem 1.26 (classification of square roots). Let the nonsingular matrix A ∈ Cn×n
have the Jordan canonical form (1.2) with p Jordan blocks, and let s ≤ p be the number
of distinct eigenvalues of A. Then A has precisely 2s square roots that are primary
functions of A, given by
(j ) (j )
Xj = Z diag(L1 1 , L2 2 , . . . , L(jp)
p )Z
−1
, j = 1: 2s ,

corresponding to all possible choices of j1 , . . . , jp , jk = 1 or 2, subject to the constraint

that ji = jk whenever λi = λk .
1.6 Classification of Matrix Square Roots and Logarithms 19

If s < p, A has nonprimary square roots. They form parametrized families

(j ) (j )
Xj (U ) = ZU diag(L1 1 , L2 2 , . . . , L(jp)
p )U
−1 −1
Z , j = 2s + 1: 2p ,

where jk = 1 or 2, U is an arbitrary nonsingular matrix that commutes with J, and

for each j there exist i and k, depending on j, such that λi = λk while ji 6= jk .

Proof. The proof consists of showing that for the square roots (1.25) for which
ji = jk whenever λi = λk ,
(j ) (j ) (j ) (j )
U diag(L1 1 , L2 2 , . . . , L(jp)
p )U
−1
= diag(L1 1 , L2 2 , . . . , L(jp)
p ),

that is, U commutes with the block diagonal matrix in the middle. This commutativ-
ity follows from the explicit form for U provided by Theorem 1.25 and the fact that
upper triangular Toeplitz matrices commute.
Theorem 1.26 shows that the square roots of a nonsingular matrix fall into two
classes. The first class comprises finitely many primary square roots, which are “iso-
lated”, being characterized by the fact that the sum of any two of their eigenvalues
is nonzero. The second class, which may be empty, comprises a finite number of pa-
rametrized families of matrices, each family containing infinitely many square roots
sharing the same spectrum.
Theorem 1.26 has two specific implications of note. First, if λk 6= 0 then the two
upper triangular square roots of Jk (λk ) given by (1.4) with f the square root function
are the only square roots of Jk (λk ). Second, if A is nonsingular and nonderogatory,
that is, none of the s distinct eigenvalues appears in more than one Jordan block,
then A has precisely 2s square roots, each of which is a primary function of A.
There is no analogue of Theorems 1.24 and 1.26 for singular A. Indeed the Jordan
block structure of a square root (when one exists) can be very different from that
of A. The search for square roots X of a singular matrix is aided by Theorem 1.36
below, which helps identify the possible Jordan forms of X; see Problem 1.29.
Analogous results, with analogous proofs, hold for the matrix logarithm.

Theorem 1.27 (Gantmacher). Let A ∈ Cn×n be nonsingular with the Jordan canon-
ical form (1.2). Then all solutions to eX = A are given by
(j ) (j )
X = ZU diag(L1 1 , L2 2 , . . . , L(jp)
p )U
−1 −1
Z ,

where
(j )
Lk k = log(Jk (λk )) + 2jk πiImk ; (1.28)
log(Jk (λk )) denotes (1.4) with the f the principal branch of the logarithm, defined by
Im(log(z)) ∈ (−π, π]; jk is an arbitrary integer; and U is an arbitrary nonsingular
matrix that commutes with J.

Theorem 1.28 (classification of logarithms). Let the nonsingular matrix A ∈ Cn×n

have the Jordan canonical form (1.2) with p Jordan blocks, and let s ≤ p be the number
of distinct eigenvalues of A. Then eX = A has a countable infinity of solutions that
are primary functions of A, given by
(j ) (j )
Xj = Z diag(L1 1 , L2 2 , . . . , L(jp)
p )Z
−1
,
20 Theory of Matrix Functions

(j )
where L1 1 is defined in (1.28), corresponding to all possible choices of the integers
j1 , . . . , jp , subject to the constraint that ji = jk whenever λi = λk .
If s < p then eX = A has nonprimary solutions. They form parametrized families

(j ) (j )
Xj (U ) = ZU diag(L1 1 , L2 2 , . . . , L(jp)
p )U
−1 −1
Z ,

where jk is an arbitrary integer, U is an arbitrary nonsingular matrix that commutes

with J, and for each j there exist i and k, depending on j, such that λi = λk while
ji 6= jk .

1.7. Principal Square Root and Logarithm

Among the square roots and logarithms of a matrix, the principal square root and
principal logarithm are distinguished by their usefulness in theory and in applications.
We denote by R− the closed negative real axis.

Theorem 1.29 (principal square root). Let A ∈ Cn×n have no eigenvalues on R− .

There is a unique square root X of A all of whose eigenvalues lie in the open right
half-plane, and it is a primary matrix function of A. We refer to X as the principal
square root of A and write X = A1/2 . If A is real then A1/2 is real.

Proof. Note first that a nonprimary square root of A, if one exists, must have
eigenvalues µi and µj with µi = −µj , and hence the eigenvalues cannot all lie in the
open right half-plane. Therefore only a primary square root can have spectrum in the
open right half-plane. Since A has no eigenvalues on R− , it is clear from Theorem 1.26
that there is precisely one primary square root of A whose eigenvalues all lie in the
open right half-plane. Hence the existence and uniqueness of A1/2 is established.
That A1/2 is real when A is real follows from Theorem 1.18 or Remark 1.9.

See Problem 1.27 for an extension of Theorem 1.29 that allows A to be singular.

Corollary 1.30. A Hermitian positive definite matrix A ∈ Cn×n has a unique Her-
mitian positive definite square root.

Proof. By Theorem 1.29 the only possible Hermitian positive definite square
root is A1/2 . That A1/2 is Hermitian positive definite follows from the expression
A1/2 = QD1/2 Q∗ , where A = QDQ∗ is a spectral decomposition (Q unitary, D
diagonal), with D having positive diagonal entries.

For a proof of the corollary from first principles see Problem 1.41.

Theorem 1.31 (principal logarithm). Let A ∈ Cn×n have no eigenvalues on R− .

There is a unique logarithm X of A all of whose eigenvalues lie in the strip { z :
−π < Im(z) < π }. We refer to X as the principal logarithm of A and write
X = log(A). If A is real then its principal logarithm is real.

Proof. The proof is entirely analogous to that of Theorem 1.29.

1.8 f (AB) and f (BA) 21

1.8. f (AB) and f (BA)

Although the matrices AB and BA are generally different, their Jordan structures
are closely related. We show in this section that for arbitrary functions f , f (AB) and
f (BA) also enjoy a close relationship—one that can be exploited both in theory and
computationally. Underlying all these relations is the fact that for any polynomial p,
and any A and B for which the products AB and BA are defined,

Ap(BA) = p(AB)A. (1.29)

This equality is trivial for monomials and follows immediately for general polynomials.
First we recap a result connecting the Jordan structures of AB and BA. We
denote by zi (X) the nonincreasing sequence of the sizes z1 , z2 , . . . , of the Jordan
blocks corresponding to the zero eigenvalues of the square matrix X.

Theorem 1.32 (Flanders). Let A ∈ Cm×n and B ∈ Cn×m . The nonzero eigenvalues
of AB are the same as those of BA and have the same Jordan structure. For the zero
eigenvalues (if any), |zi (AB) − zi (BA)| ≤ 1 for all i, where the shorter sequence
is appended with zeros as necessary, and any such set of inequalities is attained for
some A and B. If m 6= n then the larger (in dimension) of AB and BA has a zero
eigenvalue of geometric multiplicity at least |m − n|.

Proof. See Problem 1.43.

Theorem 1.33. Let A ∈ Cn×n and B ∈ Cm×m and let f be defined on the spectrum
of both A and B. Then there is a single polynomial p such that f (A) = p(A) and
f (B) = p(B).

Proof. Let p be the Hermite interpolating polynomial satisfying the union of

the interpolation conditions (1.7) for A with those for B. Let r be the Hermite
interpolating polynomial to f on the spectrum of A. Then p and r take the same
values on the spectrum of A, so f (A) := r(A) = p(A). By the same argument with A
and B interchanged, f (B) = p(B), as required.

Corollary 1.34. Let A ∈ Cm×n and B ∈ Cn×m and let f be defined on the spectra
of both AB and BA. Then
Af (BA) = f (AB)A. (1.30)

Proof. By Theorem 1.33 there is a single polynomial p such that f (AB) = p(AB)
and f (BA) = p(BA). Hence, using (1.29),

Af (BA) = Ap(BA) = p(AB)A = f (AB)A.

When A and B are square and A, say, is nonsingular, another proof of Corol-
lary 1.34 is as follows: AB = A(BA)A−1 so f (AB) = Af (BA)A−1 , or f (AB)A =
Af (BA).
As a special case of the corollary, when AB (and hence also BA) has no eigenvalues
on R− (which implies that A and B are square, in view of Theorem 1.32),

A(BA)1/2 = (AB)1/2 A.
22 Theory of Matrix Functions

In fact, this equality holds also when AB has a semisimple zero eigenvalue and the
definition of A1/2 is extended as in Problem 1.27.
Corollary 1.34 is useful for converting f (AB) into f (BA) within an expression, and
vice versa; see, for example, (2.26), the proof of Theorem 6.11, and (8.5). However,
when m > n, (1.30) cannot be directly solved to give an expression for f (AB) in
terms of f (BA), because (1.30) is an underdetermined system for f (AB). The next
result gives such an expression, and in more generality.

Theorem 1.35. Let A ∈ Cm×n and B ∈ Cn×m , with m ≥ n, and assume that BA
is nonsingular. Let f be defined on the spectrum of αIm + AB, and if m = n let f be
defined at α. Then


f (αIm + AB) = f (α)Im + A(BA)−1 f (αIn + BA) − f (α)In B. (1.31)
Proof. Note first that by Theorem 1.32, the given assumption on f implies that
f is defined on the spectrum of αIn + BA and at α.
Let g(t) = f [α + t, α] = t−1 (f (α + t) − f (α)), so that f (α + t) = f (α) + tg(t).
Then, using Corollary 1.34,
f (αIm + AB) = f (α)Im + ABg(AB)
= f (α)Im + Ag(BA)B


= f (α)Im + A(BA)−1 f (αIn + BA) − f (α)In B,
as required.
This result is of particular interest when m > n, for it converts the f (αIm + AB)
problem—a function evaluation of an m × m matrix—into the problem of evaluating
f and the inverse on n × n matrices. Some special cases of the result are as follows.
(a) With n = 1, we recover (1.16) (albeit with the restriction v ∗ u 6= 0).
(b) With f the inverse function and α = 1, (1.31) yields, after a little manipulation,
the formula (I + AB)−1 = I − A(I + BA)−1 B, which is often found in textbook
exercises. This formula in turn yields the Sherman–Morrison–Woodbury formula
(B.12) on writing A + U V ∗ = A(I + A−1 U · V ∗ ). Conversely, when f is analytic
we can obtain (1.31) by applying the Sherman–Morrison–Woodbury formula to the
Cauchy integral formula (1.12). However, Theorem 1.35 does not require analyticity.
As an application of Theorem 1.35, we now derive a formula for f (αIn +uv ∗ +xy ∗ ),
where u, v, x, y ∈ Cn , thereby extending (1.16) to the rank-2 case. Write
 ∗
v
uv ∗ + xy ∗ = [ u x ] ∗ ≡ AB.
y
Then  
v∗ u v∗ x
C := BA = ∈ C2×2 .
y∗ u y∗ x
Hence
 
∗ ∗ −1

v∗
f (αIn + uv + xy ) = f (α)In + [ u x]C f (αI2 + C) − f (α)I2 . (1.32)
y∗

The evaluation of both C −1 and f (αI2 + C) can be done explicitly (see Problem 1.9
for the latter), so (1.32) gives a computable formula that can, for example, be used
for testing algorithms for the computation of matrix functions.
1.9 Miscellany 23

1.9. Miscellany
In this section we give a selection of miscellaneous results that either are needed
elsewhere in the book or are of independent interest.
The first result gives a complete description of the Jordan canonical form of f (A)
in terms of that of A. In particular, it shows that under the action of f a Jordan
block J(λ) splits into at least two smaller Jordan blocks if f ′ (λ) = 0.

Theorem 1.36 (Jordan structure of f (A)). Let A ∈ Cn×n with eigenvalues λk , and
let f be defined on the spectrum of A.
(a) If f ′ (λk ) 6= 0 then for every Jordan block J(λk ) in A there is a Jordan block
of the same size in f (A) associated with f (λk ).
(b) Let f ′ (λk ) = f ′′ (λk ) = · · · = f (ℓ−1) (λk ) = 0 but f (ℓ) (λk ) 6= 0, where ℓ ≥ 2,
and consider a Jordan block J(λk ) of size r in A.
(i ) If ℓ ≥ r, J(λk ) splits into r 1 × 1 Jordan blocks associated with f (λk ) in
f (A).
(ii ) If ℓ ≤ r − 1, J(λk ) splits into the following Jordan blocks associated with
f (λk ) in f (A):
• ℓ − q Jordan blocks of size p,
• q Jordan blocks of size p + 1,
where r = ℓp + q with 0 ≤ q ≤ ℓ − 1, p > 0.

Proof. We prove just the first part. From Definition 1.2 it is clear that f either
preserves the size of a Jordan block Jk (λk ) ∈ Cmk ×mk of A—that is, f (Jk (λk )) has
Jordan form Jk (f (λk )) ∈ Cmk ×mk —or splits Jk (λk ) into two or more smaller blocks,
each with eigenvalue f (λk ). When f ′ (λk ) 6= 0, (1.4) shows that f (Jk (λk ))−f (λk )I has
rank mk − 1, which implies that f does not split the block Jk (λk ). When f ′ (λk ) = 0,
it is clear from (1.4) that f (Jk (λk )) − f (λk )I has rank at most mk − 2, which implies
that f (Jk (λk )) has at least two Jordan blocks. For proofs of the precise splitting
details, see Horn and Johnson [296, , Thm. 6.2.25] or Lancaster and Tismenetsky
[371, , Thm. 9.4.7].
To illustrate the result, consider the matrix
 
0 1 0 0
0 0 1 0
A= ,
0 0 0 1
0 0 0 0

which is in Jordan form with one Jordan block of size 4. Let

 
0 0 0 1
0 0 0 0
f (A) = A3 =  .
0 0 0 0
0 0 0 0

Clearly f (A) has Jordan form comprising two 1 × 1 blocks and one 2 × 2 block. We
have f ′ (0) = f ′′ (0) = 0 and f ′′′ (0) 6= 0. Applying Theorem 1.36 (b) with ℓ = 3,
r = 4, p = 1, q = 1, the theorem correctly predicts ℓ − q = 2 Jordan blocks of size
24 Theory of Matrix Functions

1 and q = 1 Jordan block of size 2. For an example of a Jordan block splitting with
f (X) = X 2 , see the matrices (1.23) and (1.24).
Theorem 1.36 is useful when trying to solve nonlinear matrix equations, because
once the Jordan form of f (A) is known it narrows down the possible Jordan forms of
A; see, e.g., Problems 1.30 and 1.51.
We noted in Section 1.4 that a nonprimary function of a derogatory A may com-
mute with A but is not a polynomial in A. The next result shows that all matrices
that commute with A are polynomials in A precisely when A is nonderogatory—that
is, when no eigenvalue appears in more than one Jordan block in the Jordan canonical
form of A.

Theorem 1.37. Every matrix that commutes with A ∈ Cn×n is a polynomial in A if

and only if A is nonderogatory.
Proof. This result is a consequence of Theorem 1.25. See Lancaster and Tis-
menetsky [371, , Prop. 12.4.1] for the details.
While commuting with A is not sufficient to be a polynomial in A, commuting
with every matrix that commutes with A is sufficient.

Theorem 1.38. B ∈ Cn×n commutes with every matrix that commutes with A ∈
Cn×n if and only if B is a polynomial in A.
Proof. See Horn and Johnson [296, , Thm. 4.4.19].
The following result is useful for finding solutions of a nonlinear matrix equation
of the form f (X) = A.

Theorem 1.39. Consider the equation f (X) = A ∈ Cn×n .

(a) If A is upper triangular and nonderogatory then any solution X is upper tri-
angular.
(b) If A is a single Jordan block J(λ) then any solution X is upper triangular
with constant diagonal elements xii ≡ ξ, where f (ξ) = λ.
(c) If the equation with A = θI has a solution X that is not a multiple of I then
there are infinitely many solutions to the equation.
Proof.
(a) The nonderogatory matrix A = f (X) commutes with X so, by Theorem 1.37, X
is a polynomial in A, which means that X is upper triangular.
(b) This follows from the proof of (a) on noting that a polynomial in J(λ) has con-
stant diagonal.
(c) Since f (X) = θI, for any nonsingular Z we have θI = Z −1 f (X)Z = f (Z −1 XZ),
so Z −1 XZ is a solution. The result now follows from the fact that any matrix other
than a scalar multiple of the identity shares its Jordan canonical form with infinitely
many other matrices.
The next result shows that a family of pairwise commuting matrices can be simul-
taneously unitarily triangularized.

Theorem 1.40. If A1 , A2 , . . . , Ak ∈ Cn×n satisfy Ai Aj = Aj Ai for all i and j then

there exists a unitary U ∈ Cn×n such that U ∗ Ai U is upper triangular for all i.
1.9 Miscellany 25

Proof. See Horn and Johnson [295, , Thm. 2.3.3].

We denote by λi (A) the ith eigenvalue of A in some given ordering.

Corollary 1.41. Suppose A, B ∈ Cn×n and AB = BA. Then for some ordering of
the eigenvalues of A, B, and AB we have λi (AopB) = λi (A)opλi (B), where op = +,
−, or ∗.
Proof. By Theorem 1.40 there exists a unitary U such that U ∗ AU = TA and
U BU = TB are both upper triangular. Thus U ∗ (A op B)U = TA op TB is upper
∗

triangular with diagonal elements (TA )ii op (TB )ii , as required.

This corollary will be used in Section 11.1. Note that for any A and B we have
trace(A+B) = trace(A)+trace(B) and det(AB) = det(A) det(B), but the conclusion
of the corollary for commuting A and B is much stronger.
Related to Theorem 1.40 and Corollary 1.41 are the following characterizations of
A and B for which “λi (p(A, B)) = p(λi (A), λi (B))”.

Theorem 1.42 (McCoy). For A, B ∈ Cn×n the following conditions are equiva-
lent.
(a) There is an ordering of the eigenvalues such that λi (p(A, B)) = p(λi (A), λi (B))
for all polynomials of two variables p(x, y).
(b) There exists a unitary U ∈ Cn×n such that U ∗ AU and U ∗ BU are upper
triangular.
(c) p(A, B)(AB − BA) is nilpotent for all polynomials p(x, y) of two variables.

Theorem 1.43. A ∈ Cn×n is unitary if and only if A = eiH for some Hermitian H.
In this representation H can be taken to be Hermitian positive definite.
Proof. The Schur decomposition of A has the form A = QDQ∗ with Q unitary
and D = diag(exp(iθj )) = exp(iΘ), where Θ = diag(θj ) ∈ Rn×n . Hence A =
Q exp(iΘ)Q∗ = exp(iQΘQ∗ ) = exp(iH), where H = H ∗ . Without loss of generality
we can take θj > 0, whence H is positive definite.

Theorem 1.44. A ∈ Cn×n has the form A = eS with S real and skew-symmetric if
and only if A is real orthogonal with det(A) = 1.
Proof. “⇒”: IfP S is real and skew-symmetric then A is real, ATA = e−S eS = I,
and det(eS ) = exp( λi (S)) = exp(0) = 1, since the eigenvalues of S are either zero
or occur in pure imaginary complex conjugate pairs.
“⇐”: If A is real orthogonal then it has the real Schur decomposition A =QDQT
aj bj
with Q orthogonal and D = diag(Dii ), where each Dii is 1, −1, or of the form −b j aj

with a2j + b2j = 1. Since det(A) = 1, there is an even number of −1s, and so we can
 aj bj 
include the −1 blocks among the −b j aj
blocks. It is easy to show that
     
aj bj cos θj sin θj 0 θj
≡ = exp =: exp(Θj ). (1.33)
−bj aj − sin θj cos θj −θj 0

We now construct a skew-symmetric K such that D = eK : K has the same block

structure as D, kii = 0 if dii = 1, and the other blocks have the form Θj in (1.33).
T
Hence A = QeK QT = eQKQ = eS , where S is real and skew-symmetric.
26 Theory of Matrix Functions

Theorem 1.45. For A ∈ Cn×n , det(eA ) = exp(trace(A)).

Proof. We have
n
Y n
Y

det(eA ) = λi (eA ) = eλi (A) = eλ1 (A)+···+λn (A) = exp trace(A) .
i=1 i=1

Note that another way of expressing Theorem 1.45 is that for any logarithm of a
nonsingular X, det(X) = exp(trace(log(X))).

1.10. A Brief History of Matrix Functions

Sylvester (1814–1897) [465, ] coined the term “matrix” in 1850 [553, ]. Cay-
ley (1821–1895) [121, ], in his A Memoir on the Theory of Matrices [99, ],
was the first to investigate the algebraic properties of matrices regarded as objects
of study in their own right (in contrast with earlier work on bilinear and quadratic
forms). Matrix theory was subsequently developed by Cayley, Sylvester, Frobenius,
Kronecker, Weierstrass, and others; for details, see [253, ], [254, ], [255, ],
[463, ].
The study of functions of matrices began in Cayley’s 1858 memoir, which treated
the square roots of 2 × 2 and 3 × 3 matrices, and he later revisited these cases in [100,
]. Laguerre [367, ], and later Peano [467, ], defined the exponential of
a matrix via its power series. The interpolating polynomial definition of f (A) was
stated by Sylvester [557, ] for n × n A with distinct eigenvalues λi , in the form
n
X Y A − λj I
f (A) = f (λi ) .
i=1
λi − λj
j6=i

Buchheim gave a derivation of the formula [84, ] and then generalized it to mul-
tiple eigenvalues using Hermite interpolation [85, ].
Weyr [614, ] defined f (A) using a power series for f and showed that the
series converges if the eigenvalues of A lie within the radius of convergence of the
series. Hensel [258, ] obtained necessary and sufficient conditions for convergence
when one or more eigenvalues lies on the circle of convergence (see Theorem 4.7).
Metzler [424, ] defined the transcendental functions eA , log(A), sin(A), and
arcsin(A), all via power series.
The Cauchy integral representation was anticipated by Frobenius [195, ], who
states that if f is analytic then f (A) is the sum of the residues of (zI − A)−1 f (z) at
the eigenvalues of A. Poincaré [473, ] uses the Cauchy integral representation,
and this way of defining f (A) was proposed in a letter from Cartan to Giorgi, circa
1928 [216, ].
The Jordan canonical form definition is due to Giorgi [216, ]; Cipolla [109,
] extended it to produce nonprimary matrix functions.
Probably the first book (actually a booklet) to be written on matrix functions is
that of Schwerdtfeger [513, ]. With the same notation as in Definitions 1.2 and
1.4 he defines
Xs nXi −1
f (j) (λi )
f (A) = Ai (A − λi I)j ,
i=1 j=0
j!

where the Ai are the Frobenius covariants: Ai = Z diag(gi (Jk ))Z −1 , where gi (Jk ) = I
if λi is an eigenvalue of Jk and gi (Jk ) = 0 otherwise, where A = Z diag(Jk )Z −1 is the
1.11 Notes and References 27

Jordan canonical form. This is just a rewritten form of the expression for f (A) given
by Definition 1.2 or by the Lagrange–Hermite formula (1.8). It can be restated as
s nX
X i −1

f (A) = f (j) (λi )Zij ,

i=1 j=0

where the Zij depend on A but not on f . For more details on these formulae see Horn
and Johnson [296, , pp. 401–404, 438] and Lancaster and Tismenetsky [371, ,
Sec. 9.5].
The equivalence of all the above definitions of f (A) (modulo their different levels
of generality) was first shown by Rinehart [493, ] (see the quote at the end of the
chapter).
One of the earliest uses of matrices in practical applications was by Frazer, Duncan,
and Collar of the Aerodynamics Department of the National Physical Laboratory
(NPL), England, who were developing matrix methods for analyzing flutter (unwanted
vibrations) in aircraft. Their book Elementary Matrices and Some Applications to
Dynamics and Differential Equations [193, ] emphasizes the important role of
the matrix exponential in solving differential equations and was “the first to employ
matrices as an engineering tool” [71, ], and indeed “the first book to treat matrices
as a branch of applied mathematics” [112, ].
Early books with substantial material on matrix functions are Turnbull and Aitken
[579, , Sec. 6.6–6.8]; Wedderburn [611, , Chap. 8], which has a useful bibliog-
raphy arranged by year, covering 1853–1933; MacDuffee [399, , Chap. IX], which
gives a concise summary of early work with meticulous attribution of results; Ferrar
[184, , Chap. 5]; and Hamburger and Grimshaw [245, ]. Papers with useful
historical summaries include Afriat [5, ] and Heuvers and Moak [259, ].
Interest in computing matrix functions grew rather slowly following the advent of
the digital computer. As the histogram on page 379 indicates, the literature expanded
rapidly starting in the 1970s, and interest in the theory and computation of matrix
functions shows no signs of abating, spurred by the growing number of applications.
A landmark paper is Moler and Van Loan’s “Nineteen Dubious Ways to Compute
the Exponential of a Matrix” [437, ], [438, ], which masterfully organizes
and assesses the many different ways of approaching the eA problem. In particular,
it explains why many of the methods that have been (and continue to be) published
are unsuitable for finite precision computation.
The “problem solving environments” MATLAB, Maple, and Mathematica have
been invaluable for practitioners using matrix functions and numerical analysts de-
veloping algorithms for computing them. The original 1978 version of MATLAB
included the capability to evaluate the exponential, the logarithm, and several other
matrix functions. The availability of matrix functions in MATLAB and it competitors
has undoubtedly encouraged the use of succinct, matrix function-based solutions to
problems in science and engineering.

1.11. Notes and References

The theory of functions of a matrix is treated in a number of books, of which sev-
eral are of particular note. The most encyclopedic treatment is given by Horn and
Johnson [296, , Chap. 6], who devote a chapter of 179 pages to the subject.
A more concise but very elegant exposition emphasizing the interpolation definition
28 Theory of Matrix Functions

is given by Lancaster and Tismenetsky [371, , Chap. 9]. A classic reference is
Gantmacher [203, , Chap. 5]. Golub and Van Loan [224, , Chap. 11] briefly
treat the theory before turning to computational matters. Linear algebra and matrix
analysis textbooks with a significant component on f (A) include Cullen [125, 1972],
Pullman [481, ], and Meyer [426, ].
For more details on the Jordan canonical form see Horn and Johnson [295, ,
Chap. 3] and Lancaster and Tismenetsky [371, , Chap. 6].
Almost every textbook on numerical analysis contains a treatment of polynomial
interpolation for distinct nodes, including the Lagrange form (1.9) and the Newton
divided difference form (1.10). Textbook treatments of Hermite interpolation are
usually restricted to once-repeated nodes; for the general case see, for example, Horn
and Johnson [296, , Sec. 6.1.14] and Stoer and Bulirsch [542, , Sec. 2.1.5].
For the theory of functions of operators (sometimes called the holomorphic func-
tional calculus), see Davies [133, ], Dunford and Schwartz [172, ], [171, ],
and Kato [337, ].
Functions of the DFT matrix, and in particular fractional powers, are considered
by Dickinson and Steiglitz [151, ], who obtain a formula equivalent to (1.18).
Much has been written about fractional transforms, mainly in the engineering litera-
ture; for the fractional discrete cosine transform, for example, see Cariolaro, Erseghe,
and Kraniauskas [96, ].
Theorems 1.15–1.17 can be found in Lancaster and Tismenetsky [371, , Sec. 9.7].
Theorem 1.18 is from Higham, Mackey, Mackey, and Tisseur [283, , Thm. 3.2].
The sufficient condition of Remark 1.9 and the equivalence (c) ≡ (d) in Theorem 1.18
can be found in Richter [491, ].
Different characterizations of the reality of f (A) for real A can be found in Evard
and Uhlig [179, , Sec. 4] and Horn and Piepmeyer [298, ].
The terminology “primary matrix function” has been popularized through its use
by Horn and Johnson [296, , Chap. 6], but the term was used much earlier by
Rinehart [495, ] and Cullen [125, 1972].
A number of early papers investigate square roots and pth roots of (singular)
matrices, including Taber [561, ], Metzler [424, ], Frobenius [195, ],
Kreis [363, ], Baker [40, ], and Richter [491, ], and Wedderburn’s book
also treats the topic [611, , Secs. 8.04–8.06].
Theorem 1.24 is a special case of a result of Gantmacher for pth roots [203, ,
Sec. 8.6]. Theorem 1.26 is from Higham [268, ]. Theorem 1.27 is from [203, ,
Sec. 8.8].
Theorem 1.32 is proved by Flanders [188, ]. Alternative proofs are given by
Thompson [566, ] and Horn and Merino [297, , Sec. 6]; see also Johnson and
Schreiner [321, ].
We derived Theorem 1.35 as a generalization of (1.16) while writing this book;
our original proof is given in Problem 1.45. Harris [249, , Lem. 2] gives the result
for α = 0 and f a holomorphic function, with the same method of proof that we have
given. The special case of Theorem 1.35 with f the exponential and α = 0 is given
by Celledoni and Iserles [102, ].
Formulae for a rational function of a general matrix plus a rank-1 perturbation,
r(C + uv ∗ ), are derived by Bernstein and Van Loan [61, ]. These are more
complicated and less explicit than (1.31), though not directly comparable with it since
C need not be a multiple of the identity. The formulae involve the coefficients of r and
so cannot be conveniently applied to an arbitrary function f by using “f (A) = p(A)
Problems 29

for some polynomial p.”

Theorem 1.42 is due to McCoy [415, ]. See also Drazin, Dungey, and Gru-
enberg [164, ] for a more elementary proof and the discussions of Taussky [564,
], [565, ]. A complete treatment of simultaneous triangularization is given
in the book by Radjavi and Rosenthal [483, ].

Problems
The only way to learn mathematics is to do mathematics.
— PAUL R. HALMOS, A Hilbert Space Problem Book (1982)

1.1. Show that the value of f (A) given by Definition 1.2 is independent of the par-
ticular Jordan canonical form that is used.
1.2. Let Jk be the Jordan block (1.2b). Show that
 
f (mk −1) (−λk )
 f (−λk ) −f ′ (−λk ) ... (−1)mk −1
 (mk − 1)!  
 .. .. 
f (−Jk ) =  f (−λk ) . . . (1.34)
 
 .. 
 . ′
−f (−λk ) 
f (−λk )

1.3. (Cullen [125, , Thm. 8.9]) Define f (A) by the Jordan canonical form defi-
nition. Use Theorem 1.38 and the property f (XAX −1 ) = Xf (A)X −1 to show that
f (A) is a polynomial in A.
1.4. (a) Let A ∈ Cn×n have an eigenvalue λ and corresponding eigenvector x. Show
that (f (λ), x) is a corresponding eigenpair for f (A).
(b) Suppose A has constant row sums α, that is, Ae = αe, where e = [1, 1, . . . , 1]T .
Show that f (A) has row sums f (α). Deduce the corresponding result for column sums.
1.5. Show that the minimal polynomial ψ of A ∈ Cn×n exists, is unique, and has
degree at most n.
1.6. (Turnbull and Aitken [579, , p. 75]) Show that if A ∈ Cn×n has minimal
polynomial ψ(A) = A2 − A − I then (I − 31 A)−1 = 35 (A + 2I).
1.7. (Pullman [481, , p. 56]) The matrix
 
−2 2 −2 4
 −1 2 −1 1 
A= 
0 0 1 0
−2 1 −1 4

has minimal polynomial ψ(t) = (t − 1)2 (t − 2). Find cos(πA).

1.8. Find the characteristic polynomial and the minimal polynomial of the nonzero
rank-1 matrix uv ∗ ∈ Cn×n .
1.9. Use (1.11) to give an explicit formula for f (A) for A ∈ C2×2 requiring knowledge
only of the eigenvalues of A.
30 Theory of Matrix Functions

1.10. Let J = eeT ∈ Rn×n denote the matrix of 1s. Show using Definition 1.4 that

f (αI + βJ) = f (α)I + n−1 (f (α + nβ) − f (α))J.

1.11. What are the interpolation conditions (1.7) for the polynomial p such that
p(A) = A?
1.12. Let A ∈ Cn×n have only two distinct eigenvalues, λ1 and λ2 , both semisimple.
Obtain an explicit formula for f (A).
1.13. Show using each of the three definitions (1.2), (1.4), and (1.11) of f (A) that
AB = BA implies f (A)B = Bf (A).
1.14. For a given A ∈ Cn×n and a given function f explain how to reliably compute
in floating point arithmetic a polynomial p such that f (A) = p(A).
1.15. Show how to obtain the formula (1.14) from Definition 1.2 when v ∗ u = 0 with
uv ∗ 6= 0.
1.16. Prove the formula (1.16) for f (αI + uv ∗ ). Use this formula to derive the
Sherman–Morrison formula (B.11).
 
1.17. Use (1.16) to obtain an explicit formula for f (A) for A = λIn−1 c
0 λ ∈ C
n×n
.
Check your result against Theorem 1.21.
1.18. (Schwerdtfeger [513, ]) Let p be a polynomial and A ∈ Cn×n . Show that
p(A) = 0 if and only if p(t)(tI − A)−1 is a polynomial in t. Deduce the Cayley–
Hamilton theorem.
1.19. Cayley actually discovered a more general version of the Cayley–Hamilton the-
orem, which appears in a letter to Sylvester but not in any of his published work
[120, ], [121, , p. 470], [464, , Letter 44]. Prove his general version: if
A, B ∈ Cn×n , AB = BA, and f (x, y) = det(xA − yB) then f (B, A) = 0. Is the
commutativity condition necessary?
1.20. Let f satisfy f (−z) = ±f (z). Show that f (−A) = ±f (A) whenever the pri-
mary matrix functions f (A) and f (−A) are defined. (Hint: Problem 1.2 can be
used.)
1.21. Let P ∈ Cn×n be idempotent (P 2 = P ). Show that f (aI + bP ) = f (a)I +
(f (a + b) − f (a))P .
1.22. Is f (A) = A∗ possible for a suitable choice of f ? Consider, for example,
f (λ) = λ.
1.23. Verify the Cauchy integral formula (1.12) in the case f (λ) = λj and A = Jn (0),
the Jordan block with zero eigenvalue.
1.24. Show from first principles that for λk 6= 0 a Jordan block Jk (λk ) has exactly
two upper triangular square roots. (There are in fact only two square roots of any
form, as shown by Theorem 1.26.)
1.25. (Davies [131, 2007]) Let A ∈ Cn×n (n > 1) be nilpotent of index n (that is,
An = 0 but An−1 6= 0). Show that A has no square root but that A + cAn−1 is a
square root of A2 for any c ∈ C. Describe all such A.
Problems 31

1.26. Suppose that X ∈ Cn×n commutes with A ∈ Cn×n , and let A have the Jordan
canonical form Z −1 AZ = diag(J1 , J2 , . . . , Jp ) = J. Is Z −1 XZ block diagonal with
blocking conformable with that of J?
1.27. (Extension of Theorem 1.29.) Let A ∈ Cn×n have no eigenvalues on R− except
possibly for a semisimple zero eigenvalue. Show that there is a unique square root X
of A that is a primary matrix function of A and whose nonzero eigenvalues lie in the
open right half-plane. Show that if A is real then X is real.
1.28. Investigate the square roots of the upper triangular matrix
 
0 1 1
A= 1 1.
0

1.29. Find all the square roots of the matrix

 
0 1 0
A = 0 0 0
0 0 0

(which is the matrix in (1.23)). Hint: use Theorem 1.36.

1.30. Show that if A ∈ Cn×n has a defective zero eigenvalue (i.e., a zero eigenvalue
appearing in a Jordan block of size greater than 1) then A does not have a square
root that is a polynomial in A.
1.31. The symmetric positive definite matrix A with aij = min(i, j) has a square
root X with 
0, i + j ≤ n,
xij =
1, i + j > n.
For example,
 2  
0 0 0 1 1 1 1 1
0 0 1 1 1 2 2 2
  = .
0 1 1 1 1 2 3 3
1 1 1 1 1 2 3 4
Is X a primary square root of A? Explain how X fits in with the theory of matrix
square roots.
1.32. Show that any square root or logarithm X of A ∈ Cn×n (primary or non-
primary) commutes with A. Show further that if A is nonderogatory then X is a
polynomial in A.
1.33. Find a logarithm of the upper triangular Toeplitz matrix
 1 1 1 
1 2! 3! . . . (n−1)!
 .. .. 
 1 2! 1 . . 
 
 .. .. 
A= .
 1 . . 
 .. 
 . 1 
2!
1
Hence find all the logarithms of A.
32 Theory of Matrix Functions

1
1.34. Let A ∈ Cn×n have no eigenvalues on R− . Show that A1/2 = e 2 log A , where
the logarithm is the principal logarithm.
1.35. Let A, B ∈ Cn×n and AB = BA. Is it true that (AB)1/2 = A1/2 B 1/2 when the
square roots are defined?
1.36. (Hille [290, ]) Show that if eA = eB and no two elements of Λ(A) differ by
a nonzero integer multiple of 2πi then AB = BA. Given an example to show that
this conclusion need not be true without the assumption on Λ(A).
1.37. Show that if eA = eB and no eigenvalue of A differs from an eigenvalue of B
by a nonzero integer multiple of 2πi then A = B.
1.38. Let A ∈ Cn×n be √nonsingular. Show that if f is an even function (f (z) = f (−z)
for all z ∈ C) then f ( A) is the same for all choices of square root (primary or
nonprimary). Show that if f is an odd function (f (−z) = −f (z) for all z ∈ C) then
√ ±1 √
A f ( A) is the same for all choices of square root.
1.39. Show that for A ∈ Cn×n , log(eA ) = A if and only if | Im(λi )| < π for every
eigenvalue λi of A, where log denotes the principal logarithm. (Since ρ(A) ≤ kAk for
any consistent norm, kAk < π is sufficient for the equality to hold.)
1.40. Let A, B ∈ Cn×n and let f and g be functions such that g(f (A)) = A and
g(f (B)) = B. Assume also that B and f (A) are nonsingular. Show that f (A)f (B) =
f (B)f (A) implies AB = BA. For example, if the spectra of A and B lie in the open
right half-plane we can take f (x) = x2 and g(x) = x1/2 , or if ρ(A) < π and ρ(B) < π
we can take f (x) = ex and g(x) = log x (see Problem 1.39).
1.41. Give a proof from first principles (without using the theory of matrix functions
developed in this chapter) that a Hermitian positive definite matrix A ∈ Cn×n has a
unique Hermitian positive definite square root.
1.42. Let A ∈ Cn×n have no eigenvalues on R− . Given that A has a square root X
with eigenvalues in the open right half-plane and that X is a polynomial in A, show
from first principles, and without using any matrix decompositions, that X is the
unique square root with eigenvalues in the open right half-plane.
1.43. Prove the first and last parts of Theorem 1.32. (For the rest, see the sources
cited in the Notes and References.)
1.44. Give another proof of Corollary 1.34 for m 6= n by using the identity
     
AB 0 Im A I A 0 0
= m (1.35)
B 0 0 In 0 In B BA

(which is (1.36) below with α = 0). What additional hypotheses are required for this
proof?
1.45. Give another proof of Theorem 1.35 based on the identity
     
AB + αIm 0 Im A Im A αIm 0
= . (1.36)
B αIn 0 In 0 In B BA + αIn

What additional hypotheses are required for this proof?

1.46. Show that Corollary 1.34 can be obtained from Theorem 1.35.
Problems 33

1.47. Can (1.31) be generalized to f (D+AB) with D ∈ Cm×m diagonal by “replacing

αI by D”?
1.48. (Klosinski, Alexanderson, and Larson [355, ]) If A and B are n×n matrices
does ABAB = 0 imply BABA = 0?
1.49. Let A ∈ Cm×n and B ∈ Cn×m . Show that det(Im + AB) = det(In + BA).

1.50.
1 1996
(Borwein,
 Bailey, and Girgensohn [77, , p. 216]) Does the equation sin A =
0 1 have a solution? (This was Putnam Problem 1996-B4.)
1.51. Show that the equation
 
1 a a ... a
 1 a ... a

 .. 

cosh(A) =  1 ... .  ∈ Cn×n
 .. 
 . a
1

has no solutions for a 6= 0 and n > 1.

1.52. An interesting application of the theory of matrix functions is to the Drazin
inverse of A ∈ Cn×n , which can be defined as the unique matrix AD satisfying
AD AAD = AD , AAD = AD A, Ak+1 AD = Ak , where k is the index of A (see Sec-
tion B.2). If A ∈ Cn×n has index k then it can be written
 
B 0
A=P P −1 , (1.37)
0 N

where B is nonsingular and N is nilpotent of index k (and hence has dimension at

least k), and then  −1 
B 0
AD = P P −1 .
0 0

(a) For what function f is AD = f (A)?

(b) Show that if p is a polynomial such that for B in (1.37), B −1 = p(B), then
AD = Ak p(A)k+1 .
(c) Determine (uv ∗ )D , for nonzero u, v ∈ Cn .

1.53. How might the definition of f (A) be extended to rectangular matrices?

34 Theory of Matrix Functions

After developing some properties of “linear transformations” in earlier papers,

Cayley finally wrote “A Memoir on the Theory of Matrices” in 1858
in which a matrix is considered as a single mathematical quantity.
This paper gives Cayley considerable claim to the honor of
introducing the modern concept of matrix,
although the name is due to Sylvester (1850).
— CYRUS COLTON MACDUFFEE, Vectors and Matrices (1943)

It will be convenient to introduce here a notion . . .

namely that of the latent roots of a matrix. . .
There results the important theorem that the latent roots of any function of a matrix
are respectively the same functions of the latent roots of the matrix itself.
— J. J. SYLVESTER, On the Equation to the
Secular Inequalities in the Planetary Theory (1883)

There have been proposed in the literature since 1880

eight distinct definitions of a matric function,
by Weyr, Sylvester and Buchheim,
Giorgi, Cartan, Fantappiè, Cipolla,
Schwerdtfeger and Richter . . .
All of the definitions except those of Weyr and Cipolla
are essentially equivalent.
— R. F. RINEHART, The Equivalence of Definitions of a Matric Function (1955)

I have not thought it necessary to undertake the

labour of a formal proof of the theorem in the
general case of a matrix of any degree.3
— ARTHUR CAYLEY, A Memoir on the Theory of Matrices (1858)

On reaching the chapter on functions of matrices I found that,

starting from a few ‘well-known’ facts,
the theory unfolded itself naturally and easily,
but that only patches of it here and there appeared to have been published before.
— W. L. FERRAR, Finite Matrices (1951)

If one had to identify the two most important topics

in a mathematics degree programme,
they would have to be calculus and matrix theory.
Noncommutative multiplication underlies the whole of quantum theory
and is at the core of some of the most exciting current research
in both mathematics and physics.
— E. BRIAN DAVIES, Science in the Looking Glass (2003)
3 Re the Cayley–Hamilton theorem.
Chapter 2
Applications

Functions of matrices play an important role in many applications. We describe some

examples in this chapter.
It is important to bear in mind that while the appearance of f (A) in a formula
may be natural and useful from the point of view of theory, it does not always mean
that it is necessary or desirable to compute f (A) in this context, as is well known for
f (A) = A−1 . Rearranging or reformulating an expression may remove or modify the
f (A) term, and it is always worth exploring these possibilities. Here are two examples
in which computation of a matrix square root can be avoided.
• For Hermitian positive definite A, the quantity kA1/2 xk2 can be computed as
(x∗ Ax)1/2 .
• The generalized eigenvalue problem Ax = λBx, with A Hermitian and B Her-
mitian positive definite, can be rewritten as B −1/2 AB −1/2 (B 1/2 x) = λ(B 1/2 x),
which is a standard Hermitian eigenvalue problem Cy = λy. This reduction
is useful for theoretical purposes [461, , Sec. 15.10], but for practical com-
putation the reduction is usually accomplished with a Cholesky factorization
B = R∗ R, for which the reduced problem is R−∗ AR−1 (Rx) = λ(Rx) [137,
], [538, , Sec. 3.4]. The Cholesky factorization is generally much less
expensive to compute than the square root.

2.1. Differential Equations

Differential equations provide a rich source of f (A) problems, because of the funda-
mental role that the exponential function plays in linear differential equations. The
classic scalar problem
dy
= ay, y(0) = c
dt
has solution y(t) = eat c, while the analogous vector problem
dy
= Ay, y(0) = c, y ∈ Cn , A ∈ Cn×n , (2.1)
dt
has solution y(t) = eAt c. More generally, with suitable assumptions on the smoothness
of f , the solution to the inhomogeneous system
dy
= Ay + f (t, y), y(0) = c, y ∈ Cn , A ∈ Cn×n (2.2)
dt
satisfies Z t
y(t) = eAt c + eA(t−s) f (s, y) ds, (2.3)
0

35
36 Applications

which is an explicit formula for y in the case that f is independent of y. These formulae
do not necessarily provide the best way to compute the solutions numerically; the large
literature on the numerical solution of ordinary differential equations (ODEs) provides
alternative techniques. However, the matrix exponential is explicitly used in certain
methods—in particular the exponential integrators described in the next subsection.
In the special case f (t, y) = b, (2.3) is
 t
y(t) = eAt c + eAt −A−1 e−As b 0 , (2.4)

which can be reworked into the form

y(t) = t ψ1 (tA)(b + Ac) + c, (2.5)

where
ez − 1 z z2
ψ1 (z) = =1+ + + ···. (2.6)
z 2! 3!
The expression (2.5) has the advantage over (2.4) that it is valid when A is singular.
Some matrix differential equations have solutions expressible in terms of the matrix
exponential. For example, the solution of

dY
= AY + Y B, Y (0) = C
dt
is easily verified to be
Y (t) = eAt CeBt .

Trigonometric matrix functions, as well as matrix roots, arise in the solution of

second order differential equations. For example, the problem

d2 y
+ Ay = 0, y(0) = y0 , y ′ (0) = y0′ (2.7)
dt2
has solution √ √
−1 √
y(t) = cos( A t)y0 + A sin( A t)y0′ , (2.8)
√
where A denotes any square root of A √ (see Problems 2.2 and 4.1). The solution
exists for all A. When A is singular (and A possibly does not exist) this formula is
√ √
−1 √
interpreted by expanding cos( A t) and A sin( A t) as power series in A.

2.1.1. Exponential Integrators

Many semidiscretized partial differential equations (PDEs) naturally take the form
(2.2) with A representing a spatially discretized linear operator and f (t, y) containing
the nonlinear terms. Exponential integrators are a broad class of methods for (2.2)
that treat the linear term exactly and integrate the remaining part of the solution
(the integral in (2.3)) numerically, using an explicit scheme. They are based on the
premise that most of the difficulty (e.g., stiffness) lies with the matrix A and not with
the nonlinear term f .
Exponential integrators, which date back to the 1960s, have the property that
they use the exponential (or related) function of the Jacobian of the differential equa-
tion or an approximation to it. They have attracted renewed interest since the late
2.2 Nuclear Magnetic Resonance 37

1990s due principally to advances in numerical linear algebra that have made efficient
implementation of the methods possible.
A simple example of an exponential integrator is the exponential time differencing
(ETD) Euler method
yn+1 = ehA yn + hψ1 (hA)f (tn , yn ), (2.9)
where yn ≈ y(tn ), tn = nh, and h is a stepsize. The function ψ1 , defined in (2.6),
is one of a family of functions {ψk } that plays an important role in these methods;
for more on the ψk see Section 10.7.4. The method (2.9) requires the computation
of the exponential and ψ1 (or, at least, their actions on a vector) at each step of the
integration.
For an overview of exponential integrators see Minchev and Wright [431, ],
and see LeVeque [380, ] for a concise textbook treatment. A few key papers are
Cox and Matthews [118, ], Hochbruck, Lubich, and Selhofer [292, ], Kassam
and Trefethen [336, ], and Schmelzer and Trefethen [503, ], and a MATLAB
toolbox is described by Berland, Skaflestad, and Wright [59, ].

2.2. Nuclear Magnetic Resonance

Two-dimensional nuclear magnetic resonance (NMR) spectroscopy is a tool for de-
termining the structure and dynamics of molecules in solution. The basic theory for
the nuclear Overhauser effect experiment specifies that a matrix of intensities M (t)
is related to a symmetric, diagonally dominant matrix R, known as the relaxation
matrix, by the Solomon equations
dM
= −RM, M (0) = I.
dt
Hence M (t) = e−Rt . This relation is used in both directions: in simulations and
testing to compute M (t) given R, and in the inverse problem to determine R from
observed intensities. The latter problem can be solved with a matrix logarithm eval-
uation, but in practice not all the mij are known and estimation methods, typically
based on least squares approximations and requiring matrix exponential evaluations,
are used.

2.3. Markov Models

The matrix exponential and logarithm play an important role in Markov models,
which are used in a variety of different subjects. Consider a time-homogeneous
continuous-time Markov process in which individuals move among n states. The
transition probability matrix P (t) ∈ Rn×n has (i, j) entry equal to the probability
that an individual starting in state i at time 0 will be in state j at time t. The row
sums of P are all 1, so P is a stochastic matrix. Associated with the process is the
transition intensity matrix Q ∈ Rn×n , which is related to P by
P (t) = eQt .
The elements of Q satisfy
n
X
qij ≥ 0, i 6= j, qij = 0, i = 1: n. (2.10)
j=1
38 Applications

For any such Q, eQt is nonnegative for all t ≥ 0 (see Theorem 10.29) and has unit
row sums, so is stochastic.
Now consider a discrete-time Markov process with transition probability matrix
P in which the transition probabilities are independent of time. We can ask whether
P = eQ for some intensity matrix Q, that is, whether the process can be regarded
as a discrete manifestation of an underlying time-homogeneous Markov process. If
such a Q exists it is called a generator and P is said to be embeddable. Necessary
and sufficient conditions for the existence of a generator for general n are not known.
Researchers in sociology [525, ], statistics [331, ], and finance [315, ]
have all investigated this embeddability problem. A few interesting features are as
follows:

• If P has distinct, real positive eigenvalues then the only real logarithm, and
hence the only candidate generator, is the principal logarithm.

• P may have one or more real negative eigenvalues, so that the principal loga-
rithm is undefined, yet a generator may still exist. For example, consider the
matrix [525, , Ex. 10]
 
1 + 2x 1 − x 1 − x √
1
P = 1 − x 1 + 2x 1 − x  , x = −e−2 3π ≈ −1.9 × 10−5 .
3
1 − x 1 − x 1 + 2x

P is diagonalizable, with eigenvalues 1, x, and x. Every primary logarithm is

complex, since it cannot have complex conjugate eigenvalues. Yet the nonpri-
mary logarithm  
√ −2/3 1/2 1/6
Q = 2 3π  1/6 −2/3 1/2 
1/2 1/6 −2/3
is a generator.

• More than one generator may exist.

Suppose a given transition matrix P ≡ P (1) has a generator Q = log(P ). Then Q

can be used to construct P (t) at other times, through P (t) = exp(Qt). For example,
if P is the transition matrix for the time period of one year then the transition
matrix for a month is P (1/12) = elog(P )/12 . However, it is more direct and efficient to
compute P (1/12) as P 1/12 , thus avoiding the computation of a generator. Indeed, the
standard inverse scaling and squaring method for the principal logarithm of a matrix
requires the computation of a matrix root, as explained in Section 11.5. Similarly,
the transition matrix for a week can be computed directly as P 1/52 .
This use of matrix roots is suggested by Waugh and Abel [610, ], mentioned
by Israel, Rosenthal, and Wei [315, ], and investigated in detail by Kreinin and
Sidelnikova [362, ]. The latter authors, who are motivated by credit risk models,
address the problems that the principal root and principal logarithm of P may have
the wrong sign patterns; for example, the root may have negative elements, in which
case it is not a transition matrix. They show how to optimally adjust these matrices
to achieve the required properties—a process they term regularization. Their pre-
ferred method for obtaining transition matrices for short times is to regularize the
appropriate matrix root. Questions of existence and uniqueness of stochastic roots of
stochastic matrices arise (see Problem 7.17).
2.4 Control Theory 39

2.4. Control Theory

In control theory a linear dynamical system can be expressed as a continuous-time
system

dx
= F x(t) + Gu(t), F ∈ Cn×n , G ∈ Cn×m ,
dt
y = Hx(t) + Ju(t), H ∈ Cp×n , J ∈ Cp×m ,

or as the corresponding discrete-time state-space system

xk+1 = Axk + Buk , A ∈ Cn×n , B ∈ Cn×m ,

yk = Hxk + Juk .

Here, x is the state vector and u and y are the input and output vectors, respectively.
The connection between the two forms is given by
Z τ 
A = eF τ , B= eF t dt G,
0

where τ is the sampling period. Therefore the matrix exponential and logarithm are
needed for these conversions. In the MATLAB Control System Toolbox [413] the
functions c2d and d2c carry out the conversions, making use of MATLAB’s expm and
logm functions.
We turn now to algebraic equations arising in control theory and the role played
by the matrix sign function. The matrix sign function was originally introduced by
Roberts [496] in 1971 as a tool for solving the Lyapunov equation and the algebraic
Riccati equation. It is most often defined in terms of the Jordan canonical form
A = ZJZ −1 of A ∈ Cn×n . If we arrange that
 
J1 0
J= ,
0 J2

where the eigenvalues of J1 ∈ Cp×p lie in the open left half-plane and those of J2 ∈
Cq×q lie in the open right half-plane, then
 
−Ip 0
sign(A) = Z Z −1 . (2.11)
0 Iq

The sign function is undefined if A has an eigenvalue on the imaginary axis. Note
that sign(A) is a primary matrix function corresponding to the scalar sign function

1, Re z > 0,
sign(z) = z ∈ C,
−1, Re z < 0,

which maps z to the nearest square root of unity. For more on the sign function,
including alternative definitions and iterations for computing it, see Chapter 5.
The utility of the sign function is easily seen from Roberts’ observation that the
Sylvester equation

AX + XB = C, A ∈ Cm×m , B ∈ Cn×n , C ∈ Cm×n ,

40 Applications

is equivalent to the equation

     −1
A −C Im X A 0 Im X
= .
0 −B 0 In 0 −B 0 In

If sign(A) = I and sign(B) = I then

       
A −C I X Im 0 Im −X Im −2X
sign = m = , (2.12)
0 −B 0 In 0 −In 0 In 0 −In

so the solution X can be read from the (1, 2) block of the sign of the block upper
triangular matrix A0 −B
−C
. The conditions that sign(A) and sign(B) are identity
matrices are certainly satisfied for the Lyapunov equation, in which B = A∗ , in the
common case where A is positive stable, that is, Re λi (A) > 0 for all i.
Consider now the algebraic Riccati equation

XF X − A∗ X − XA − G = 0, (2.13)

where all matrices are n × n and F and G are Hermitian. The desired solution is
Hermitian and stabilizing, in the sense that the spectrum of A − F X lies in the open
left half-plane. Such a solution exists and is unique under suitable conditions that we
will not describe; see [349, ] and [370, , Chap. 22] for details. The equation
can be written in the equivalent form
     −1
A∗ G X −In −(A − F X) −F X −In
W = = .
F −A In 0 0 (A − F X)∗ In 0
(2.14)
By assumption, A − F X has eigenvalues with negative real part. Hence we can apply
the sign function to (2.14) to obtain
   −1
X −In In Z X −In
sign(W ) =
In 0 0 −In In 0

for some Z. Writing sign(W ) − I2n = [M1 M2 ], where M1 , M2 ∈ C2n×n , the latter
equation becomes
    
X −In X −In 0 Z
[ M1 M2 ] = , (2.15)
In 0 In 0 0 −2In

which gives
M1 X = −M2 ,
which is a system of 2n equations in n2 unknowns. The system is consistent, by
2

construction, and by rewriting (2.15) as

    
0 In 0 Z 0 In
[ M1 M2 ] =
−In X 0 −2In −In X

we see that [−In X]M1 = 2In , which implies that M1 has full rank. To summarize:
the Riccati equation (2.13) can be solved by computing the sign of a 2n × 2n matrix
and then solving an overdetermined but consistent system; the latter can be done
using a QR factorization.
2.5 The Nonsymmetric Eigenvalue Problem 41

2.5. The Nonsymmetric Eigenvalue Problem

The matrix sign function, defined in the previous section, can be used to count how
many eigenvalues of a matrix lie in particular regions of the complex plane and to
obtain the corresponding invariant subspaces. From (2.11) we have trace(sign(A)) =
q − p, and since p + q = n this gives the formulae
1
1

p = n − trace(sign(A) , q = n + trace(sign(A)
2 2
for the number of eigenvalues lying in the open left half-plane and open right half-
plane, respectively. Moreover, writing Z in (2.11) as Z = [Z1 Z2 ] with Z1 ∈ Cn×p
and Z2 ∈ Cn×q , then I + sign(A) = 2[Z1 Z2 ] diag(0, Iq )Z −1 = 2Z2 Z −1 (p + 1: n, :),
so the columns of (I + sign(A))/2 span the invariant subspace corresponding to the
eigenvalues of A in the open right half-plane (indeed, (I+sign(A))/2 is a projector onto
this subspace—see Theorem 5.1). In fact, as the next theorem shows, the eigenvalue
problem for A can be deflated into two smaller problems for the eigenvalues in the
two half-planes.

Theorem 2.1 (spectrum splitting via sign function). Let A ∈ Rn×n have no pure
imaginary eigenvalues and define W = (sign(A) + I)/2. Let
q n−q
 
q R11 R12
QT W Π =
n−q 0 0
be a rank-revealing QR factorization, where Π is a permutation matrix and q =
rank(W ). Then
q n−q
 
T q A11 A12
Q AQ = ,
n−q 0 A22
where the eigenvalues of A11 lie in the open right half-plane and those of A22 lie in
the open left half-plane.
Proof. See Problem 2.3.
The number of eigenvalues in more complicated regions of the complex plane can
be counted by suitable sequences of matrix sign evaluations. For example, assuming
that A has no eigenvalues lying on the edges of the relevant regions:
• The number of eigenvalues of A lying
in the vertical strip Re z ∈ (ξ1 , ξ2 ) is
1
2 trace sign(A − ξ1 I) − sign(A − ξ2 I) .

• Let ξw , ξe , ξn , ξs be complex numbers at the
corners of a rectangle oriented at

π/4 to the axes. With N (ξ) = sign (A − ξI)2 , the number of eigenvalues of A
lying inside the rectangle is
1

trace N (ξw ) + N (ξe ) − N (ξn ) − N (ξs ) .
4
For details, and analogues of the deflation in Theorem 2.1, see Howland [302, ]
and Bai and Demmel [27, ]. By combining these techniques it is possible to set
up a divide and conquer algorithm for a partial or complete set of eigenvalues [27,
], [29, ].
The use of the matrix sign function as described in this section is particularly
attractive in the context of high-performance computing.
42 Applications

2.6. Orthogonalization and the Orthogonal Procrustes Problem

In many applications a matrix that should be orthogonal turns out not to be because
of errors of measurement, truncation, or rounding. The question then arises how best
to orthogonalize the matrix. One approach is to apply Gram–Schmidt orthogonal-
ization or, equivalently, to compute the orthogonal factor in a QR factorization. An
alternative, “optimal orthogonalization”, is to replace A by the nearest orthogonal
matrix. More generally, let A ∈ Cm×n (m ≥ n) and define the distance to the nearest
matrix with orthonormal columns by

min{ kA − Qk : Q∗ Q = I }.

For the 2- and Frobenius norms an optimal Q is U in the polar decomposition A =

U H, where U ∗ U = I and H is Hermitian positive semidefinite; see Theorem 8.4.
The factor H can be written as H = (A∗ A)1/2 and so if A is of full rank then
U = A(A∗ A)−1/2 . An important advantage of the unitary polar factor U over the
QR factor Q is that it is basis-independent: if we transform A → W1 AW2 with the Wi
unitary then U transforms likewise to W1 U W2 (since W1 AW2 = W1 U W2 · W2∗ HW2 is
a polar decomposition), but Q does not change in such a predictable way. The polar
factors are not functions of A according to our definition, because of the appearance
of A∗ in these relations. Nevertheless, there are strong connections with the matrix
square root and the matrix sign function and so we devote Chapter 8 to the polar
decomposition.
Optimal orthogonalization is used in a number of applications. In aerospace com-
putations the direction cosine matrix (DCM) D(t) ∈ R3×3 describes the rotation of
a coordinate system relative to a reference coordinate system. It satisfies the matrix
ODE
dD
= SD, S = −S T , D(0) orthogonal.
dt
This system is typically solved by ODE methods that do not preserve orthogonal-
ity, yet D(t) = exp(St)D(0) is orthogonal (see Theorem 1.44). Approximate DCMs
therefore need to be reorthogonalized periodically; see Mao [405, ] and the ref-
erences therein. Optimal orthogonalization is also used in the numerical solution of
more general matrix ODEs with orthogonal solutions:

Y ′ (t) = F (t, Y (t)), Y (0)T Y (0) = I, (2.16)

where Y (t) ∈ Rm×n , m ≥ n, and Y (t)T Y (t) = I for all t > 0; see Hairer, Lubich, and
Wanner [239, , Sec. 4.4], D. J. Higham [262, ], and Sofroniou and Spaletta
[534, ].
In quantum chemistry orthogonalization using the unitary polar factor is called
Löwdin orthogonalization; see Bhatia and Mukherjea [67, ], Goldstein and Levy
[221, ], and Jansik et al. [318, ]. An application of the polar decomposi-
tion to determining the orientation of “parallel spherical wrist” robots is described
by Vertechy and Parenti-Castelli [601, ]. Moakher [434, ] shows that a cer-
tain geometric mean of a set of rotation matrices can be expressed in terms of the
orthogonal polar factor of their arithmetic mean.
The polar decomposition is also used in computer graphics as a convenient way
of decomposing a 3 × 3 or 4 × 4 linear transformation into simpler component parts
(see Shoemake and Duff [520, ]) and in continuum mechanics for representing
2.7 Theoretical Particle Physics 43

the deformation gradient as the product of a rotation tensor and a stretch tensor (see
Bouby, Fortuné, Pietraszkiewicz, and Vallée [78, ]).
The orthogonal Procrustes problem is to solve

min kA − BQkF : Q ∈ Cn×n , Q∗ Q = I , (2.17)

where A, B ∈ Cm×n ; thus a unitary matrix is required that most nearly transforms a
rectangular matrix B into a matrix A of the same dimensions in a least squares sense.
A solution is given by the unitary polar factor of B ∗A; see Theorem 8.6. The orthogo-
nal Procrustes problem is a well-known and important problem in factor analysis and
in multidimensional scaling in statistics; see the books by Gower and Dijksterhuis
[226, ] and Cox and Cox [119, ]. In these applications the matrices A and
B represent sets of experimental data, or multivariate samples, and it is necessary to
determine whether the sets are equivalent up to rotation. An important variation of
(2.17) requires Q to be a “pure rotation”, that is, det(Q) = 1; one application area
is shape analysis, as discussed by Dryden and Mardia [168, ]. Many other varia-
tions of the orthogonal Procrustes problem exist, including those involving two-sided
transformations, permutation transformations, and symmetric transformations, but
the solutions have weaker connections with the polar decomposition and with matrix
functions.
The polar decomposition and the orthogonal Procrustes problem both arise in
numerical methods for computing analytic singular value decompositions, as explained
by Mehrmann and Rath [419, ].

2.7. Theoretical Particle Physics

Lattice quantum chromodynamics (QCD) is a research area of physics that has in
recent years made extensive use of matrix functions. QCD is a physical theory that
describes the strong interactions between quarks as the constituents of matter. Lattice
QCD formulates the theory on a four dimensional space-time lattice, and its numerical
simulations currently occupy large amounts of high-performance computer time.
An important recent development in lattice QCD is the overlap-Dirac operator
of Neuberger [446, ], the study of which requires the solution of n-dimensional
linear systems of the form

G − sign(H) x = b. (2.18)
Here, G = diag(±1), H is sparse, complex, and Hermitian, and n is extremely large:
of order perhaps 106 . For Hermitian A, the matrix sign function can be written as
sign(A) = Q diag(sign(λi ))Q∗ , where A = Q diag(λi )Q∗ is a spectral decomposition
with Q unitary. The system (2.18) is currently solved by Krylov subspace techniques,
which require matrix–vector products with the coefficient matrix. Hence a key step
in the solution of (2.18) is the computation of sign(H)c for given vectors c. In view
of the huge dimension of H, this product must be computed without forming the
dense matrix sign(H). A variety of methods have been proposed for this computa-
tion, including Krylov techniques and methods based on rational matrix sign function
approximations. For details, see van den Eshof, Frommer, Lippert, Schilling, and Van
der Vorst [585, ], the articles in Frommer, Lippert, Medeke, and Schilling [196,
], and the summary in Frommer and Simoncini [198, ]. A collection of rele-
vant MATLAB functions is given by Boriçi [75].
In Monte Carlo QCD simulations, det(A) must be estimated for a large, sparse,
symmetric positive definite matrix A known as the fermion matrix. One approach
44 Applications

makes use of the identity, for symmetric positive definite A,

log(det(A)) = trace(log(A)),
where log denotes the principal logarithm (see Theorem 1.31); for such A this is
equivalent to Theorem 1.45. This identity converts the problem into one of estimating
the diagonal elements of log(A). See, for example, Bai, Fahey, and Golub [32, ]
and Thron, Dong, Liu, and Ying [567, ].
Another area where determinants must be evaluated is computational quantum
field theory [378, ]. Motivated by this application, Ipsen and Lee [310, ]
exploit the relation det(A) = exp(trace(log(A))) (Theorem 1.45) to derive numerical
approximations to the determinant.
Quantum Monte Carlo (QMC) simulations with the Hubbard model of particle
interactions rely on a number of fundamental linear algebra operations. In particular,
solving linear systems M x = b is the key computational kernel in QMC and forming M
requires the computation of matrix exponentials. For details see Bai, Chen, Scalettar,
and Yamazaki [26, ].

2.8. Other Matrix Functions

One matrix function generates the need for another. In testing the quality of a
computed approximation fb(A) ≈ f (A) we may want to form f −1 (fb(A)) − A or to
test how closely an identity involving f is satisfied. Thus eA may generate the need
for log(A) and cos(A) for sin(A). Moreover, methods for different functions are often
interrelated. The inverse scaling and squaring algorithm for the matrix logarithm
requires matrix square roots (see Section 11.5), while one method for computing the
square root of a symmetric positive definite matrix employs the polar decomposition
of a related matrix (see Section 6.8.4).

2.9. Nonlinear Matrix Equations

One reason for developing a theory of matrix functions is to aid the solution of non-
linear matrix equations. Ideally, closed form solutions can be found in terms of an
appropriate matrix function.
The algebraic Riccati equation (2.13) and its special cases form an important class
of nonlinear matrix equations. We examine the special case
XAX = B, A, B ∈ Cn×n . (2.19)
This equation has the solution
X = B(AB)−1/2 , (2.20)
provided that AB has no eigenvalues on R− , as is easily verified by substitution. Here,
any square root of AB can be taken. The formula (2.20) can be rewritten in alternative
forms. Corollary 1.34, implies that X = (BA)−1/2 B. Other equivalent forms are
A−1 (AB)1/2 and (BA)1/2 A−1 . We can also derive a more symmetric expression for
X as follows:

−1/2
X = B B −1/2 · B 1/2 AB 1/2 · B 1/2

−1/2 1/2
= BB −1/2 B 1/2 AB 1/2 B
1/2 1/2


1/2 −1/2 1/2
=B B AB B , (2.21)
2.9 Nonlinear Matrix Equations 45

where again any square root can be taken, and again this expression is easily verified
to satisfy (2.19). Another expression for X is as the (1,2) block of sign( A0 B0 ), which
follows from the sign-based solution of (2.13) and also from Theorem 5.2. As this
example illustrates, there may be several ways to express the solutions to a matrix
equation in terms of matrix functions.
If A and B are Hermitian positive definite then there is a unique Hermitian positive
definite solution to (2.19), given by any of the expressions above, where the Hermitian
positive definite square root is always taken. The uniqueness follows from writing
(2.19) as Y 2 = C, where Y = A1/2 XA1/2 and C = A1/2 BA1/2 and using the fact that
a Hermitian positive definite matrix has a unique Hermitian positive definite square
root (Corollary 1.30). (Note that this approach leads directly to (2.21).) In this case
formulae that are more computationally efficient than those above are available; see
Problem 2.7 and Algorithm 6.22.
An equation that generalizes the scalar quadratic in a different way to the algebraic
Riccati equation is the quadratic matrix equation

AX 2 + BX + C = 0, A, B, C ∈ Cn×n . (2.22)

Unfortunately, there is no closed-form expression for X in general, and the theory

of such equations is nontrivial; see Higham and Kim [280, ], and the references
therein. A special case in which the usual quadratic formula generalizes is when
A = I, B commutes with C, and B 2 − 4C has a square root. Then we can complete
the square in the usual way to obtain
1 1
X = − B + (B 2 − 4C)1/2 ,
2 2
where the square root can be any primary square root. One way of solving (2.22) is
by Newton’s method [281, ]. If a sufficiently good starting matrix is not available
then continuation can be used. Assume that a solution X(t) to the quadratic √ matrix
equation AX(t)2 + tBX(t) + C = 0 exists for t ∈ [0, 1]. Then X(0) = −A−1 C
and we can solve for X(tk+1 ) by using Newton’s method with X(tk ) as the starting
matrix, for 0 = t0 < t1 < t2 < · · · < tm = 1. The aim is to take the number of
steps m as small as possible such that each application of Newton’s method converges
quickly.
We mention two applications in which the quadratic (2.22) arises. Quasi-birth-
death processes are two-dimensional Markov chains with a block tridiagonal transition
probability matrix. They are widely used as stochastic models in telecommunications,
computer performance, and inventory control. Analysis using the matrix-geometric
method leads to three quadratic matrix equations whose elementwise minimal non-
negative solutions can be used to characterize most of the features of the Markov
chain. Excellent references are the books by Bini, Latouche, and Meini [70, ] and
Latouche and Ramaswami [373, ].
A second application is the solution of the quadratic eigenvalue problem

Q(λ)x = (λ2 A + λB + C)x = 0, A, B, C ∈ Cn×n , (2.23)

which arises in the analysis of damped structural systems and vibration problems
[369, ], [570, ]. The standard approach is to reduce (2.23) to a generalized
eigenproblem (GEP) Gx = λHx of twice the dimension, 2n. This “linearized” prob-
lem can be further converted to a standard eigenvalue problem of dimension 2n under
suitable nonsingularity conditions on the coefficients A, B, and C. However, if we
46 Applications

can find a solution X of the associated quadratic matrix equation (2.22) then we can
write
λ2 A + λB + C = −(B + AX + λA)(X − λI), (2.24)
and so the eigenvalues of (2.23) are those of X together with those of the GEP
(B + AX)x = −λAx, both of which are n × n problems. Bridges and Morris [82,
] employ this approach in the solution of differential eigenproblems.
For a less obvious example of where a matrix function arises in a nonlinear matrix
equation, consider the problem of finding an orthogonal Q ∈ Rn×n such that

Q − QT = S, S = −S T ∈ Rn×n given, (2.25)

which arises in the analysis of the dynamics of a rigid body [94, ] and in the
solution of algebraic Riccati equations [309, ]. The equation can be rewritten
Q−Q−1 = S, which implies both Q2 −I = QS and Q2 −I = SQ. Hence Q2 − 21 (QS +
p
SQ) − I = 0, or (Q − 21 S)2 = I + S 2 /4. Thus Q is of the form Q = 12 S + I + S 2 /4
for some square root. Using the theory of matrix square roots the equation (2.25) can
then be fully analyzed.
There is little in the way of numerical methods for solving general nonlinear matrix
equations f (X) = A, other than Newton’s method. By using the Jordan canonical
form it is usually possible to determine and classify all solutions (as we did in Sec-
tion 1.6 for the square root and logarithm), but this approach is usually not feasible
computationally; see Horn and Johnson [296, , Cor. 6.2.12, Sec. 6.4] for details.
Finally, we note that nonlinear matrix equations provide useful test problems for
optimization and nonlinear least squares solvers, especially when a reference solution
can be computed by matrix function techniques. Some matrix square root problems
are included in the test collection maintained by Fraley on Netlib [191] and in the
CUTEr collection [225, ].

2.10. Geometric Mean

The geometric mean of positive scalars can be generalized to Hermitian positive defi-
nite matrices in various ways, which to a greater or lesser extent possess the properties
one would like of a mean. Let A, B ∈ Cn×n be Hermitian positive definite. The ge-
ometric mean A # B is defined as the unique Hermitian positive definite solution to
XA−1 X = B, or (cf. (2.19)–(2.21))

X = B 1/2 (B −1/2 AB −1/2 )1/2 B 1/2 = B(B −1 A)1/2 = (AB −1 )1/2 B, (2.26)

where the last equality can be seen using Corollary 1.34. The geometric mean has
the properties (see Problem 2.5)

A # A = A, (2.27a)
−1 −1 −1
(A # B) = A # B , (2.27b)
A # B = B # A, (2.27c)
1
A # B ≤ (A + B), (2.27d)
2
√
all of which generalize properties of the scalar geometric mean a # b = ab. Here,
X ≥ 0 denotes that the Hermitian matrix X is positive semidefinite; see Section B.12.
2.11 Pseudospectra 47

The geometric mean also satisfies the extremal property

   
∗ A X
A # B = max X = X : ≥0 . (2.28)
X B

The geometric mean yields the solution to more general equations than XA−1 X =
B. For example, if A and B are Hermitian positive definite then the unique Hermitian
positive definite solution to XA−1 X ± X − B = 0 is X = 21 (∓A + A # (A + 4B)) [391,
, Thm. 3.1].
Another definition of geometric mean of Hermitian positive definite matrices A
and B is
E(A, B) = exp( 12 (log(A) + log(B))), (2.29)
where log is the principal logarithm. This is called the log-Euclidean mean by Arsigny,
Fillard, Pennec, and Ayache [20, ], who investigate its properties.

2.11. Pseudospectra
Pseudospectra are not so much an application of matrix functions as objects with
intimate connections to them. The ǫ-pseudospectrum of A ∈ Cn×n is defined, for a
given ǫ > 0 and a subordinate matrix norm, to be the set

Λǫ (A) = { z : z is an eigenvalue of A + E for some E with kEk < ǫ }. (2.30)

It can also be represented, in terms of the resolvent (zI − A)−1 , as

Λǫ (A) = { z : k(zI − A)−1 k > ǫ−1 }.

The resolvent therefore provides a link between pseudospectra and matrix functions,
through Definition 1.11.
Pseudospectra provide a means for judging the sensitivity of the eigenvalues of a
matrix to perturbations in the matrix elements. For example, the 0.01-pseudospectrum
indicates the uncertainty in the eigenvalues if the elements are known to only two dec-
imal places. More generally, pseudospectra have much to say about the behaviour of
a matrix. They provide a way of describing the effects of nonnormality on processes
such as matrix powering and exponentiation.
Matrix functions and pseudospectra have several features in common: they are
applicable in a wide variety of situations, they are “uninteresting” for normal matrices,
and they are nontrivial to compute.
For more on pseudospectra we can do no better than refer the reader to the
ultimate reference on the subject: Trefethen and Embree [573, ].

2.12. Algebras
While this book is concerned with matrices over the real or complex fields, some of the
results and algorithms are applicable to more general algebras, and thereby provide
tools for working with these algebras. For example, the GluCat library [217] is a
generic library of C++ templates that implements universal Clifford algebras over
the real and complex fields. It includes algorithms for the exponential, logarithm,
square root, and trigonometric functions, all based on algorithms for matrices.
48 Applications

2.13. Sensitivity Analysis

Ideally a numerical algorithm returns not only an approximate solution but also an
estimate or bound for the error in that solution. Producing an a priori error bound
for an algorithm can be very difficult, as it involves analysis of truncation errors and
rounding errors and their propagation. A separate question, usually easier to answer,
is how sensitive is the solution of the problem to perturbations in the data. Knowledge
of problem sensitivity can be crucial in applications since it gives insight into whether
the problem has been well formulated, allows prediction of the effects of inaccuracy
in the data, and indicates the best accuracy that any algorithm can be expected
to provide in floating point arithmetic. Sensitivity is determined by the derivative
of the function that maps the input data to the solution. For matrix functions the
appropriate derivative is the Fréchet derivative, and its norm determines a condition
number for the problem, as explained in the next chapter. Thus every f (A) problem
gives rise to the related problem of characterizing, computing, and estimating the
Fréchet derivative of f and its norm.

2.14. Other Applications

Finally, we describe some more speculative or less well established applications.

2.14.1. Boundary Value Problems

Schmitt [504, ] proposes a symmetric difference scheme for linear, stiff, or sin-
gularly perturbed constant coefficient boundary value problems of first order based
on the stability function f (z) = z + (1 + z 2 )1/2 . The function f agrees with ez up
to terms in z of second order, and the idea is that f may bring more favourable
stability properties than a polynomial or rational stability function when the eigen-
values of the Jacobian matrix vary greatly in magnitude and location. Implementing
the method requires computing on each step of the integration a matrix square root
(ω 2 I + h2k A2 )1/2 , where ω is a parameter, hk a stepsize, and A is the Jacobian of the
system. Schmitt uses the (unscaled) Denman–Beavers iteration (6.15) to compute the
square roots.

2.14.2. Semidefinite Programming

Semidefinite programming problems are a class of constrained optimization problems
in which the variable is a symmetric positive semidefinite matrix, X ∈ Rn×n . Various
algorithms are available for the solution of such problems. An algorithm suggested by
Kanzow and Nagel [333, ] requires the computation of square roots of symmetric
positive semidefinite matrices. The authors observe that the condition XS = 0,
with X and S symmetric positive semidefinite, which arises in the conditions for
X to be an optimal solution, can be expressed as φ(X, S) = 0, where φ(X, S) =
X + S − (X 2 + S 2 )1/2 . Their algorithm involves line searches that involve repeated
evaluation of φ(X, S) at arguments that may differ only slightly.

2.14.3. Matrix Sector Function

The matrix sector function, introduced by Shieh, Tsay, and Wang [519, ], is a
generalization of the matrix sign function. For a given p, the matrix sector function
2.14 Other Applications 49

Figure 2.1. The dots are the pth roots of unity and the lines the sector boundaries, illustrated
for p = 2: 5. The scalar sector function sectp (z) maps z ∈ C to the nearest pth root of unity.
p = 2 gives the sign function.

can be defined via the Jordan canonical form, in an analogous way as for the sign
function in Section 2.4, but now mapping each eigenvalue to the nearest pth root
of unity. For p = 2, the matrix sign function is obtained. More precisely, for A ∈
Cn×n having no eigenvalues with argument (2k + 1)π/p, k = 0: p − 1, the matrix
p-sector function can be defined by sectp (A) = A(Ap )−1/p (where the principal pth
root is taken; see Theorem 7.2). Figure 2.1 illustrates the scalar sector function. The
sector function has attracted interest in the control theory literature because it can
be used to determine the number of eigenvalues in a specific sector and to obtain the
corresponding invariant subspace [358, ]. However, a good numerical method for
computing the matrix sector function is currently lacking.

2.14.4. Matrix Disk Function

Let A ∈ Cn×n have Jordan canonical form A = ZJZ −1 with J = diag(J1 , J2 ), where
the eigenvalues of J1 are inside the unit disk and the eigenvalues of J2 are outside
the unit disk. The matrix disk function is defined by disk(A) = Z diag(I, 0)Z −1 ;
if A has an eigenvalue on the unit circle then disk(A) is undefined. An alternative
representation is
1


disk(A) = I − sign (A − I)−1 (A + I) .
2
The matrix disk function was introduced in the same paper by Roberts [496] that
introduced the matrix sign function. It can be used to obtain invariant subspaces in
an analogous way as for the matrix sign function. For more details, see Benner and
Byers [53, ], Benner, Byers, Mehrmann, and Xu [54, ], Sun and Quintana-
Ortı́ [551, ], and the references therein.
50 Applications

2.14.5. The Average Eye in Optics

In optics there is interest in defining an average of the optical characteristics of a set of

eyes, or more generally a set of optical systems. The first-order
  character of an optical
system is characterized by a transference matrix T = S0 1δ ∈ R5×5 , where S ∈ R4×4
 
is symplectic, that is, S T JS = J, where J = −I02 I02 . A straightforward average
P m
m−1 i=1 Ti of m transference matrices Ti is not in general a transference Pm matrix.
Harris [250, ] proposes as a suitable average the matrix exp(m−1 i=1 log(Ti )),
where the log is the principal logarithm (cf. the geometric mean (2.29)); that this
average is a transference matrix follows from the facts that the principal logarithm of
a symplectic matrix with no eigenvalues on R− is Hamiltonian and the exponential
of a Hamiltonian matrix is symplectic [152, ], [251, ]

2.14.6. Computer Graphics

In computer graphics a transformation can be represented as a matrix, usually of di-

mension 3 or 4, and multiplication by the matrix effects the transformation. Alexa [9,
] defines a “commutative addition” operation A ⊕ B = exp(log(A) + log(B)), us-
ing which he constructs linear combinations of transformations. If A and B commute
this is just the usual matrix product A ⊕ B = AB = BA. One use of the operation
is to interpolate between two transformations A and B via tA ⊕ (1 − t)B, t ∈ [0, 1].
Note that this is the same mathematical idea as in Section 2.14.5. The interpolation
problem with A and B rotation matrices is discussed by Gallier and Xu [199, ].

2.14.7. Bregman Divergences

Matrix nearness problems ask for the distance from a given matrix to the nearest
matrix with a certain property, and for that nearest matrix. The nearest unitary
matrix problem mentioned in Section 2.6 is of this type. The use of a Bregman
divergence in place of a matrix norm is proposed by Dhillon and Tropp [150, ].
The Bregman divergence of X ∈ Cn×n from Y ∈ Cn×n is defined by D(X, Y ) =
ψ(X) − ψ(Y ) − h∇ψ(Y ), X − Y i, where ψ : Cn×n → R+ is strictly convex and h·i is an
inner product. A particular instance applying to Hermitian positive definite

matrices
is the von Neumann divergence D(X, Y ) = trace X(log(X) − log(Y ) − X + Y ), the
use of which leads to the need to evaluate expressions such as exp(log(Y ) + W ). Thus
Bregman divergences provide another application of matrix functions.

2.14.8. Structured Matrix Interpolation

Numerical solution of a matrix differential equation dY /dt = f (t, Y ), Y (0) given,

leads to a sequence of pairs (ti , Yi ), where Yi ≈ Y (ti ). In order to approximate Y at
non-mesh points t it is necessary to interpolate the data, which can be done by stan-
dard procedures. However, if Y (t) has structure the interpolated matrices may not
possess that structure. Dieci, Morini, Papini, and Pasquali [155, ] propose inter-
polating (ti , f (Yi )), to obtain the polynomial P (t) and then taking g(t) = f −1 (P (t))
as the interpolation function. The idea is to choose f so that the required structure
is enforced. In [155, ] the use of f (Y ) = log(Y ) is advocated for orthogonal,
symplectic, and Hermitian positive definite matrices.
2.15 Notes and References 51

2.14.9. The Lambert W Function and Delay Differential Equations

The solutions of the equation ses = a are given by s = Wk (a), where Wk (z) is the kth
branch of the Lambert W function. This function is a “log-like” function and arises in
many different situations, ranging from the enumeration of trees in combinatorics to
the explicit solution of various differential equations. The matrix Lambert W function
can be defined via (1.2) or (1.4). One application of the matrix function is to delay
differential equations. Consider the model problem

y ′ (t) = Ay(t − 1), t ≥ 0, A ∈ Cn×n ,

where
y(t) = g(t), −1 ≤ t ≤ 0,
for a given function g. If we look for solutions y(t) = exp(tS)c for some con-
stant c ∈ Cn then we are led to the matrix equation S exp(S) = A, and hence
to S =PWk (A). The general solution to the problem can then be expressed as
∞
y(t) = k=−∞ eWk (A)t ck , where the vectors ck are determined by g.
For more details of the scalar Lambert W function see Corless, Gonnet, Hare,
Jeffrey, and Knuth [115, ]. The matrix Lambert W function is analyzed by
Corless, Ding, Higham, and Jeffrey [114, ], who show that as a primary matrix
function it does not yield all solutions of S exp(S) = A, which is analogous to the
fact that the primary matrix logarithm does not provide all solutions of eX = A
(Theorem 1.28). For the application to delay differential equation see Jarlebring and
Damm [320, ] and the references therein, and Heffernan and Corless [256, ].
A good reference on the numerical solution of delay differential equations is Bellen
and Zennaro [50, ].

2.15. Notes and References

Good references for matrix differential equations are Bellman [51, , Chap. 10],
Gantmacher [203, , Sec. 5.5], and Godunov [219, ]. The expression (2.5)
involving ψ1 can be found in Schwerdtfeger [513, , p. 44].
Our description of NMR is based on Havel, Najfeld, and Yang [252, ], [445,
]; see also Levitt [383, ]. Our description of the need to compute roots
of transition matrices is adapted from Guo and Higham [233, ]; see also Prob-
lem 7.15.
For more on continuous-time and discrete-time linear dynamical systems see, for
example, Chen [106, , Chap. 4] or Franklin, Powell, and Workman [192, ].
Further details on solution of Riccati equations via the matrix sign function can
be found in Byers [88, ], Gardiner [204, ], Lancaster and Rodman [370, ,
Chap. 22], and Laub [374, ]. See also Benner and Quintana-Ortı́ [55, ] for the
generalized Lyapunov equation A∗ XE + E ∗ XA − G = 0 and Benner, Quintana-Ortı́,
and Quintana-Ortı́ [56, ] for the Sylvester equation. With Newton’s method for
computing it, the matrix sign function is a versatile way of solving these equations; it
can be adapted to compute low rank solutions efficiently and to exploit the hierarchical
H-matrix storage format. See Baur and Benner [47, ] and Grasedyck, Hackbusch,
and Khoromskij [228, ].
Theorem 2.1 is based on a result of Lin and Zmijewski [392, ], incorporating
refinements by Bai and Demmel [27, ].
52 Applications

The idea of using the matrix sign function to compute eigensystems in divide and
conquer fashion was first investigated by Denman and Beavers in the early 1970s;
see [146, ] and the references therein. However, in these early papers nonunitary
transformations are used, in contrast to the approach in Section 2.5.
Early references for (2.19) and formulae of the form (2.20) are Frobenius [195,
], Baker [40, ], and Turnbull and Aitken [579, , p. 152].
For more on geometric means of positive definite matrices see Bhatia [65, ,
Chap. 4], Lawson and Lim [377, ], Ando, Li, and Mathias [15, ], and Moakher
[435, ]. Proofs of (2.28) can be found in Bhatia [65, , Thm. 4.1.3] and Ando
[13, , Thm. 2], [14, , Thm. 2.8]. The geometric mean (2.26) appears to have
been first introduced by Pusz and Woronowicz [482, ].
The factorization (2.24) and other properties of matrix polynomials are treated
by Davis [140, ], Dennis, Traub, and Weber [147, ], Gohberg, Lancaster,
and Rodman [220, ], Lancaster [369, ], and Lancaster and Tismenetsky [371,
].

Problems
Though mathematics is much easier to watch than do,
it is a most unrewarding spectator sport.
— CHARLES G. CULLEN, Matrices and Linear Transformations (1972)

2.1. Derive the formula (2.3) for the solution of problem (2.2).
2.2. Reconcile the fact that the initial value problem (2.7) has a unique solution with
√ √
−1 √
the observation that cos( A t)y0 + A sin( A t)y0′ is a solution for any square
√
root A of A.
2.3. Prove Theorem 2.1.
2.4. Show that if the Hermitian positive definite matrices A and B commute then
the geometric means # in (2.26) and E in (2.29) are given by A # B = A1/2 B 1/2 =
E(A, B).
2.5. Prove the relations (2.27) satisfied by the geometric mean #.
2.6. (Bhatia [65, , p. 111]) Show that for Hermitian positive definite A, B ∈ C2×2 ,
√
αβ
A#B = p (α−1 A + β −1 B),
det(α A + β −1 B)
−1

with α2 = det(A), β 2 = det(B).

2.7. Consider the Riccati equation XAX = B, where A and B are Hermitian positive
definite. Show that the Hermitian positive definite solution X can be computed as
R−1 (RBR∗ )1/2 R−∗ , where A = R∗R is a Cholesky factorization.
Problems 53

Whenever there is too much talk of applications,

one can rest assured that the theory has very few of them.
— GIAN-CARLO ROTA, Indiscrete Thoughts (1997)

The 1930’s . . . saw the diversification of aeroelastic studies. . .

and the first practical use of the hitherto
recondite branch of pure mathematics known as “matrices”.
— A. R. COLLAR, The First Fifty Years of Aeroelasticity (1978)

The main computational challenge in the

implementation of any exponential integrator is the need for
fast and computationally stable evaluations
of the exponential and the related ψ -functions.
— BORISLAV V. MINCHEV and WILL M. WRIGHT, A Review of Exponential Integrators
for First Order Semi-linear Problems (2005)

Prior to about 1978, solving an algebraic Riccati equation (ARE)

was something that was most definitely to be avoided,
and with good reason.
Existing techniques were often quite unreliable.
— ALAN J. LAUB, Invariant Subspace Methods for the
Numerical Solution of Riccati Equations (1991)

This work is intended for graphics

where standard matrix packages only offer elementary matrix operations.
For this reason, implementations are provided using only
matrix inversion, multiplication, and addition.
— MARC ALEXA, Linear Combination of Transformations (2002)

Since W is such a simple function,

we would expect by Pareto’s principle
(eighty percent of your work is accomplished with twenty percent of your tools)
that W would have many applications.
In fact this is the case, although the presence of W often goes unrecognized.
— ROBERT M. CORLESS, GASTON H. GONNET, D. E. G. HARE,
DAVID J. JEFFREY, and DONALD E. KNUTH,
On the Lambert W Function (1996)
Chapter 3
Conditioning

In practice most data are inexact or uncertain. Computations with exact data are
subject to rounding errors, and the rounding errors in an algorithm can often be
interpreted as being equivalent to perturbations in the data, through the process
of backward error analysis. Therefore whether the data are exact or inexact, it is
important to understand the sensitivity of matrix functions to perturbations in the
data. Sensitivity is measured by condition numbers. This chapter is concerned with
defining appropriate condition numbers and showing how to evaluate or estimate
them efficiently. The condition numbers can be expressed in terms of the norm of
the Fréchet derivative, so we investigate in some detail the properties of the Fréchet
derivative.

3.1. Condition Numbers

It is useful to begin by recalling how condition numbers are defined for scalar functions,
f (x). The standard definition of (relative) condition number is
 
 f (x + ∆x) − f (x) 
condrel (f, x) := lim sup  ,

ǫ→0 |∆x|≤ǫ|x| ǫf (x)

which measures by how much, at most, small changes in the data can be magnified in
the function value, when both changes are measured in a relative sense. Assuming for
simplicity that f is twice continuously differentiable, f (x + ∆x) − f (x) = f ′ (x)∆x +
o(∆x), which can be rewritten as
 ′ 
f (x + ∆x) − f (x) xf (x) ∆x
= + o(∆x).
f (x) f (x) x

(Recall that h = o(ǫ) means that khk/ǫ → 0 as ǫ → 0.) It is then immediate that
 ′ 
 xf (x) 
condrel (f, x) =  . (3.1)
f (x) 

The definition of condition number extends readily to matrix functions f : Cn×n →

n×n
C . We define the relative condition number by4

kf (X + E) − f (X)k
condrel (f, X) := lim sup , (3.2)
ǫ→0 kEk≤ǫkXk ǫkf (X)k
4 The definition is applicable to arbitrary functions f . Most of the results of Section 3.2 onwards

assume f is a primary matrix function as defined in Chapter 1. We will use the condition number
(3.2) for a more general f in Section 8.2.

55
56 Conditioning

where the norm is any matrix norm. This definition implies that
kf (X + E) − f (X)k kEk
≤ condrel (f, X) + o(kEk), (3.3)
kf (X)k kXk
and so provides an approximate perturbation bound for small perturbations E.
Some care is needed in interpreting (3.3) for functions not defined throughout
Cn×n . The definition (3.2) is clearly valid as long as f is defined in a neighbourhood
of X. The bound (3.3) is therefore valid for X + E in that neighbourhood. An
example is given in Section 5.1 that shows how blindly invoking (3.3) can lead to a
patently incorrect bound.
A corresponding absolute condition number, in which the change in the data and
the function are measured in an absolute sense, is defined by
kf (X + E) − f (X)k
condabs (f, X) := lim sup . (3.4)
ǫ→0 kEk≤ǫ ǫ

Note that
kXk
condrel (f, X) = condabs (f, X) , (3.5)
kf (X)k
so the two condition numbers differ by just a constant factor. Usually, it is the relative
condition number that is of interest, but it is more convenient to state results for the
absolute condition number.
To obtain explicit expressions analogous to (3.1) we need an appropriate notion
of derivative for matrix functions. The Fréchet derivative of a matrix function f :
Cn×n → Cn×n at a point X ∈ Cn×n is a linear mapping
L
Cn×n −→ Cn×n
E 7−→ L(X, E)

such that for all E ∈ Cn×n

f (X + E) − f (X) − L(X, E) = o(kEk). (3.6)

The Fréchet derivative may not exist, but if it does it is unique (see Problem 3.3).
The notation L(X, E) can be read as “the Fréchet derivative of f at X in the direction
E ”, or “the Fréchet derivative of f at X applied to the matrix E”. If we need to show
the dependence on f we will write Lf (X, E). When we want to refer to the mapping
at X and not its value in a particular direction we will write L(X). In the case n = 1
we have, trivially, L(x, e) = f ′ (x)e, and more generally if X and E commute then
L(X, E) = f ′ (X)E = Ef ′ (X) (see Problem 3.8).
The absolute and relative condition numbers can be expressed in terms of the
norm of L(X), which is defined by
kL(X, Z)k
kL(X)k := max . (3.7)
Z6=0 kZk

Theorem 3.1 (Rice). The absolute and relative condition numbers are given by

condabs (f, X) = kL(X)k, (3.8)

kL(X)k kXk
condrel (f, X) = . (3.9)
kf (X)k
3.2 Properties of the Fréchet Derivative 57

Proof. In view of (3.5), it suffices to prove (3.8). From (3.4) and (3.6), and using
the linearity of L, we have
kf (X + E) − f (X)k
condabs (f, X) = lim sup
ǫ→0 kEk≤ǫ ǫ

L(X, E) + o(kEk)
= lim sup
ǫ→0 kEk≤ǫ ǫ

= lim sup L(X, E/ǫ) + o(kEk)/ǫ
ǫ→0 kEk≤ǫ

= sup kL(X, Z)k.

kZk≤1

Finally, the sup can be replaced by a max, since we are working on a finite dimensional
vector space, and the maximum is attained with kZk = 1.
To illustrate, for f (X) = X 2 we have f (X + E) − f (X) = XE + EX + E 2 , so

Lx2 (X, E) = XE + EX (3.10)

and kLx2 (X)k ≤ 2kXk.

It is usually not straightforward to obtain an explicit formula or representation for
the Fréchet derivative, not least because matrix multiplication is not commutative,
but in later chapters we will see how this can be done for certain functions.

3.2. Properties of the Fréchet Derivative

The condition number of f is essentially the norm of the Fréchet derivative (see
Theorem 3.1). In this section we describe some properties of the Fréchet derivative
that will be useful in bounding or estimating the condition number.
Related to the Fréchet derivative is the directional or Gâteaux derivative
f (X + tE) − f (X) d 
G(X, E) = lim =  f (X + tE). (3.11)
t→0 t dt t=0
If the Fréchet derivative exists at X then it is equal to the Gâteaux derivative (see
Problem 3.4). But the converse is not true: the existence of directional derivatives in
all directions is a weaker notion of differentiability than the existence of the Fréchet
derivative. However, if the Gâteaux derivative exists, is a linear function of E, and
is continuous in X, then it is also the Fréchet derivative. (See the works cited in the
Notes and References.) Depending on the context, (3.6) or (3.11) may be the more
useful expression to work with.
We begin with four rules that show how to obtain the Fréchet derivatives of a
sum or product and of composite and inverse functions. In the following four results
f, g, h : Cn×n → Cn×n .

Theorem 3.2 (sum rule). If g and h are Fréchet differentiable at A then so is f =

αg + βh and Lf (A, E) = αLg (A, E) + βLh (A, E).
Proof. The proof is immediate from the definition of the Fréchet derivative.

Theorem 3.3 (product rule). If g and h are Fréchet differentiable at A then so is

f = gh and Lf (A, E) = Lg (A, E)h(A) + g(A)Lh (A, E).
58 Conditioning

Proof. We have

f (A + E) = g(A + E)h(A + E)



= g(A) + Lg (A, E) + o(kEk) h(A) + Lh (A, E) + o(kEk)
= f (A) + Lg (A, E)h(A) + g(A)Lh (A, E) + o(kEk).

Theorem 3.4 (chain rule). Let h and g be Fréchet differentiable at A and h(A),
respectively, and let f = g ◦ h (i.e., f (A) = g(h(A))). Then f is
Fréchet differentiable
at A and Lf = Lg ◦ Lh , that is, Lf (A, E) = Lg h(A), Lh (A, E) .

Proof.

f (A + E) − f (A) = g(h(A + E)) − g(h(A))

= g h(A) + Lh (A, E) + o(kEk) − g h(A)




= g h(A) + Lg h(A), Lh (A, E) + o(kEk) + o(kEk) − g h(A)


= Lg h(A), Lh (A, E) + o(kEk).

The following theorem says that the Fréchet derivative of the inverse function is
the inverse of the Fréchet derivative of the function.

Theorem 3.5 (derivative of inverse function). Let f and f −1 both exist and be con-
tinuous in an open neighbourhood of X and f (X), respectively, and assume Lf ex-
ists and is nonsingular at X. Then Lf −1 exists at Y = f (X) and Lf −1 (Y, E) =


L−1 −1
f (X, E), or equivalently, Lf X, Lf −1 (Y, E) = E. Hence kLf −1 (Y )k = kLf (X)k.

Proof. For the existence see Dieudonné [159, , Thm. 8.2.3]. The formulae
can be obtained by applying the chain
rule to the relation f (f −1 (Y )) = Y , which
gives the equality Lf X, Lf −1 (Y, E) = E.
2 −1 1/2
To illustrate Theorem 3.5

we take f (x) = x and f (x) =2 x . The theorem says
that Lx2 X, Lx1/2 (X , E) = E, i.e., using (3.10), XLx1/2 (X , E)+Lx1/2 (X 2 , E)X =
2

E. In other words, L = Lx1/2 (A, E) is the solution of the Sylvester equation A1/2 L +
LA1/2 = E.
The following theorem will also be very useful. For the rest of this section D
denotes an open subset of R or C.

Theorem 3.6 (Mathias). Let f be 2n − 1 times continuously differentiable on D.

Let A(t) ∈ Cn×n be differentiable at t = 0 and assume that the spectrum of A(t) is
contained in D for all t in some neighbourhood of 0. Then, with A = A(0),
 #
  " d 
′
A A (0) f (A) dt  f (A(t))
f = t=0 . (3.12)
0 A 0 f (A)

Proof. Let ǫ 6= 0 and define

 
I ǫ−1 I
U= .
0 I
3.2 Properties of the Fréchet Derivative 59

Then
 A(ǫ)−A(0)    A(ǫ)−A(0)  
A(0) −1 A(0)
f ǫ = Uf U ǫ U U −1
0 A(ǫ) 0A(ǫ)
 
A(0) 0
= Uf U −1
0 A(ǫ)
 
f (A(0)) 0
=U U −1
0 f (A(ǫ))
 (A(0)) 
f (A) f (A(ǫ))−fǫ
= .
0 f (A(ǫ))

Since f is 2n − 1 times continuously differentiable the 2n × 2n matrix on the left-hand

side is defined for small enough ǫ, and since f is a continuous matrix function on the
2n × 2n matrices with spectrum in D by Theorem 1.19, the limit as ǫ → 0 exists and
is  
A(0) A′ (0)
f .
0 A(0)
The result now follows by taking the limit of the matrix on the right.
The assumption that f is 2n − 1 times continuously differentiable is needed in
Theorem 3.6 to cater for the “worst case” Jordan structure of the argument of f in
(3.12); see Problem 3.5.
If A is Hermitian, or more generally normal, then the differentiability condition
on f in all the results of this section can be relaxed to f being just once continuously
differentiable, as a result of the fact that all Jordan blocks of A are 1 × 1.
We actually need only a special case of (3.12). Letting A(t) = A + tE, we have
from (3.12), 
  " d 
#
A E f (A) dt  f (A + tE)
f = t=0 . (3.13)
0 A 0 f (A)
The next result is useful because it implies that the Gâteaux derivative of a gen-
eral function f at a particular point agrees with the Gâteaux derivative of a certain
polynomial at that point, and polynomials are easier to work with.

Theorem 3.7 (Horn and Johnson). Let f be 2n − 1 times continuously differentiable

on D. Let A(t) ∈ Cn×n be differentiable at t = 0 and assume that the spectrum of
A(t) is contained in D for all t in some neighbourhood of 0. Then, with A = A(0),

d  d 
 f (A(t)) =  pA⊕A (A(t)), (3.14)
dt t=0 dt t=0
where pA⊕A interpolates f and its derivatives at the zeros of the characteristic poly-
nomial of A ⊕ A ≡ diag(A, A), that is,
(j)
pA⊕A (λi ) = f (j) (λi ), j = 0: 2ri − 1, i = 1: s, (3.15)

where λ1 , . . . , λs are the distinct eigenvalues of A, with algebraic multiplicities r1 , . . . , rs ,

respectively.
60 Conditioning

Proof. Define  
A A′ (0)
B= .
0 A
Theorem 3.6 shows that
d  d 
 f (A(t)) = [f (B)]12 = [p (B)]12 =  p(A(t)),
dt t=0 dt t=0
where the second equality holds for any polynomial p that takes the same values as f
on the spectrum of B. By its definition (3.15), pA⊕A is such as polynomial. (Note that
pA⊕A may satisfy more interpolation conditions than are required in order to take
the same values as f on the spectrum of B. The polynomial pA⊕A is essentially an
“overestimate” that has the required properties and can be defined without knowledge
of the Jordan structure of B; see Remark 1.5.)
With the aid of the previous two results we can now identify sufficient conditions
on f for the Fréchet derivative to exist and be continuous.

Theorem 3.8 (existence and continuity of Fréchet derivative). Let f be 2n−1 times
continuously differentiable on D. For X ∈ Cn×n with spectrum in D the Fréchet
derivative L(X, E) exists and is continuous in the variables X and E.

Proof. Since f has 2n − 1 continuous derivatives, we know from Theorem 1.19

that f is a continuous matrix function on the set of 2n×2n matrices with spectrum in
D. Also, the map from a 2n×2n matrix to its (1: n, n+1: 2n) submatrix is continuous.
Therefore since (3.13) shows the Gâteaux derivative G(X, E) to be the composition
of the two maps just described, it is continuous in X and E. Moreover, G(X, E) is
a linear function of E. This follows from (3.13) when f is a polynomial, and then in
general from Theorem 3.7 with A(t) = X +tE, which shows that f and the polynomial
pX⊕X have the same Gâteaux derivative at X. But the linearity in E and continuity
in X of G imply that G is the Fréchet derivative, L.
Under the conditions of Theorem 3.8 we can rewrite (3.13) as
   
X E f (X) L(X, E)
f = . (3.16)
0 X 0 f (X)

The significance of this formula is that it converts the problem of evaluating the
Fréchet derivative in a particular direction to that of computing a single matrix
function—albeit for a matrix of twice the dimension. This is useful both in theory
and in practice.
We now find the eigenvalues of the Fréchet derivative. An eigenpair (λ, V ) of L(X)
comprises a scalar λ, the eigenvalue, and a nonzero matrix V ∈ Cn×n , the eigenvector,
such that L(X, V ) = λV .
Since L is a linear operator

vec(L(X, E)) = K(X) vec(E) (3.17)

2 2
for some K(X) ∈ Cn ×n that is independent of E. We refer to K(X) as the Kronecker
form of the Fréchet derivative. If (λ, V ) is an eigenpair of L(X) then K(X)v = λv,
where v = vec(V ), so (λ, v) is an eigenpair of K(X) in the usual matrix sense. The
following lemma identifies eigenpairs of L(X).
3.2 Properties of the Fréchet Derivative 61

Recall that the divided difference f [λ, µ] is defined by


 f (λ) − f (µ) , λ 6= µ,
f [λ, µ] = λ−µ
 ′
f (λ), λ = µ.

Theorem 3.9 (eigenvalues of Fréchet derivative). Let f be 2n−1 times continuously

differentiable on D and let X ∈ Cn×n have spectrum in D. The eigenvalues of the
Fréchet derivative L of f at X are

f [λi , λj ], i, j = 1: n,

where the λi are the eigenvalues of X. If ui and vj are nonzero vectors such that
Xui = λi ui and vjT X = λj vjT , then ui vjT is an eigenvector of L(X) corresponding to
f [λi , λj ].
Pm
Proof. Suppose, first, that f is a polynomial: f (t) = k=0 ak tk . Then (see the
more general Problem 3.6)
m
X k
X
L(X, E) = ak X j−1 EX k−j .
k=1 j=1

Hence, using (B.16),

m
X k
X
K(X) = ak (X k−j )T ⊗ X j−1 .
k=1 j=1

From (B.17) the eigenvalues of K(X) are, for p, q = 1: n,

 m
 X λkp − λkq

 ak , λp 6= λq ,
m
X Xk 
 λp − λq
k=1
ak λk−j
p λj−1
q = m

 X
k=1 j=1 
 ak kλk−1 , λp = λq ,
 p
k=1

 f (λp ) − f (λq ) , λ 6= λ ,
p q
= λp − λq
 ′
f (λp ), λp = λq ,
= f [λp , λq ].

It is easy to show that vj ⊗ui is an eigenvector of K(X) corresponding to the eigenvalue

f [λi , λj ] and that this corresponds to an eigenvector ui vjT of L(X).
Now consider a general function f . Theorem 3.7 implies that the Fréchet derivative
L of f at X is the same as that of the polynomial pX⊕X . We can therefore use
the first part to deduce that the eigenvalues of L(X) are the numbers pX⊕X [λi , λj ],
i, j = 1: n. But by definition, pX⊕X has the same values as f on the spectrum of X,
so pX⊕X [λi , λj ] = f [λi , λj ] for all i and j. The form of the eigenvectors follows from
the first part.
Problem 3.10 shows how to identify eigenpairs of L(X) without employing the
Kronecker form. However, with that approach it is difficult to show that all the
62 Conditioning

eigenvalues of L(X) have been accounted for. Note that Theorem 3.9 does not nec-
essarily identify all the eigenvectors of L(X); see Problem 3.11.
Theorem 3.9 enables us to deduce when the Fréchet derivative is nonsingular.

Corollary 3.10. Let f be 2n− 1 times continuously differentiable on D. The Fréchet

derivative L of f at a matrix X ∈ Cn×n with eigenvalues λi ∈ D is nonsingular when
f ′ (λi ) 6= 0 for all i and f (λi ) = f (λj ) ⇒ λi = λj .

The next result shows that the Fréchet derivative in any direction at a diagonal
matrix is formed simply by Hadamard multiplication by the matrix of divided dif-
ferences of the eigenvalues. Here, ◦ denotes the Hadamard (or Schur) product of
A, B ∈ Cn×n : A ◦ B = (aij bij ).

Theorem 3.11 (Daleckiı̆ and Kreı̆n). Let f be 2n − 1 times continuously differen-

tiable on D. Let D = diag(λi ) ∈ Cn×n be diagonal and λi ∈ D for all i. Then

L(D, E) = (f [λi , λj ]eij ) = (f [λi , λj ]) ◦ E. (3.18)

Proof. From (3.16) we have

   
D E f (D) L(D, E)
f = ,
0 D 0 f (D)

and using the fact that f (A) commutes with A (Theorem 1.13 (a)) we obtain
     
f (D) L(D, E) D E D E f (D) L(D, E)
= .
0 f (D) 0 D 0 D 0 f (D)

Equating the (1,2) blocks of this equation gives

f (D)E − Ef (D) = DL(D, E) − L(D, E)D,

or, since D = diag(λi ),

(f (λi ) − f (λj ))eij = (λi − λj )(L(D, E))ij , i, j = 1: n.

If the λi are distinct then the latter equation immediately gives (3.18). If the λi
are not distinct then consider D + diag(1, 2, . . . , n)ǫ, which has distinct eigenvalues
for ǫ sufficiently small and positive, and for which (3.18) therefore holds. Letting
ǫ → 0, since L(D, E) is continuous in D and E by Theorem 3.8, (3.18) holds for D
by continuity.

Corollary 3.12. Let f be 2n − 1 times continuously differentiable on D and let X ∈

Cn×n have spectrum in D and be diagonalizable: X = ZDZ −1 , D = diag(λi ). Then


L(X, E) = ZL(D, Z −1 EZ)Z −1 = Z f [λi , λj ] ◦ Z −1 EZ Z −1 .

The final result is an analogue for the Fréchet derivative of the fact that the 2-norm
of a diagonal matrix equals its spectral radius.

Corollary 3.13. Under the conditions of Theorem 3.11, kL(D)kF = maxi,j |f [λi , λj ]|.
3.3 Bounding the Condition Number 63

Proof. Using (3.7) and Theorem 3.11 we have

kL(D, Z)kF
kL(D)kF = max
Z6=0 kZkF
k(f [λi , λj ]) ◦ ZkF
= max
Z6=0 kZkF
k diag(f [λi , λj ]) vec(Z)k2
= max
vec(Z)6=0 kvec(Z)k2
= max |f [λi , λj ]|.
i,j

3.3. Bounding the Condition Number

Our aim in this section is to develop bounds for the condition number. We assume
throughout that f is 2n − 1 times continuously differentiable on an open subset of R
or C, which by Theorem 3.8 implies that the Fréchet derivative exists.
Let λ and E be an eigenvalue and corresponding eigenvector of L(X). Then
L(X, E) = λE and hence, from (3.7),

kL(X)k ≥ |λ|. (3.19)

Note that this is essentially the standard result (B.8) that no eigenvalue of a matrix
can exceed any norm of the matrix.

Theorem 3.14. For any norm,

condabs (f, X) ≥ max |f [λ, µ]|.

λ,µ∈Λ(X)

Proof. From (3.8), condabs (f, X) = kL(X)k. The bound of the theorem is
obtained by maximizing over all the eigenvalues, using (3.19) and Theorem 3.9.
By specializing to the Frobenius norm we can obtain an upper bound for the con-
dition number. Here we need the matrix condition number with respect to inversion,
κ(Z) = kZkkZ −1 k for Z ∈ Cn×n .

Theorem 3.15. Let X ∈ Cn×n be diagonalizable: X = ZDZ −1 , D = diag(λi ).

Then, for the Frobenius norm,

condabs (f, X) ≤ κ2 (Z)2 max |f [λ, µ]|.

λ,µ∈Λ(X)

e −1 , where E
Proof. By Corollary 3.12 we have L(X, E) = ZL(D, E)Z e = Z −1 EZ.
Hence, using (B.7),
e F ≤ κ2 (Z)kL(D)kF kEk
kL(X, E)kF ≤ κ2 (Z)kL(D, E)k e F ≤ κ2 (Z)2 kL(D)kF kEkF .

Now D is diagonal, so by Corollary 3.13 kL(D)kF = maxλ,µ∈Λ(D) |f [λ, µ]|.

Corollary 3.16. Let X ∈ Cn×n be normal. Then, for the Frobenius norm,

condabs (f, X) = max |f [λ, µ]|.

λ,µ∈Λ(X)
64 Conditioning

Proof. A normal matrix is diagonalizable by a unitary similarity. Hence we can

take κ2 (Z) = 1 in Theorem 3.15, and Theorem 3.14 then shows that the upper and
lower bounds are equalities.
The theorems show that for diagonalizable X, cond(f, X) is governed principally
by two factors: the maximum first order divided difference on the eigenvalues and
(possibly, since it appears only in an upper bound) the nonnormality of X, as mea-
sured by the minimum of κ2 (Z) over all diagonalizing transformations Z. The divided
difference terms may be innocuous, such as for f (x) = x2 , for which √ f [λ,√µ] = λ + µ,
or potentially large, such as for f (x) = x1/2 , for which f [λ, µ] = ( λ + µ)−1 . The
nonnormality term can of course be arbitrarily large.

3.4. Computing or Estimating the Condition Number

The essential problem in computing or estimating the absolute or relative condition
number of f at X is to compute or estimate kL(X)k. For the Frobenius norm the link
(3.17) between the operator L(X) and the n2 × n2 Kronecker matrix K(X) yields

kL(X, E)kF k vec(L(X, E))k2

kL(X)kF = max = max
E6=0 kEkF E6=0 k vec(E)k2
kK(X) vec(E)k2
= max
E6=0 k vec(E)k2
1/2
1/2
= kK(X)k2 = kK(X)∗ K(X)k2 = λmax K(X)∗ K(X) . (3.20)

Given the ability to compute L(X, E) we can therefore compute the condition number
exactly in the Frobenius norm by explicitly forming K(X).

Algorithm 3.17 (exact condition number). Given X ∈ Cn×n and a function f and
its Fréchet derivative this algorithm computes condrel (f, X) in the Frobenius norm.

1 for j = 1: n
2 for i = 1: n
3 Compute Y = L(X, ei eTj ).
4 K(: , (j − 1)n + i) = vec(Y )
5 end
6 end
7 condrel (f, X) = kKk2 kXkF /kf (X)kF

Cost: O(n5 ) flops, assuming f (X) and L(X, E) cost O(n3 ) flops.
For large n, Algorithm 3.17 is prohibitively expensive and so the condition number
must be estimated rather than computed exactly. In practice, what is needed is an
estimate that is of the correct order of magnitude—more than one correct significant
digit is not needed.
No analogous equalities to (3.20) hold for the 1-norm, but we can bound the ratio
of the 1-norms of L(X) and K(X).

Lemma 3.18. For X ∈ Cn×n and any function f ,

kL(X)k1
≤ kK(X)k1 ≤ nkL(X)k1 .
n
3.4 Computing or Estimating the Condition Number 65

Proof. For E ∈ Cn×n we have kEk1 ≤ k vec(E)k1 ≤ nkEk1 (with equality on the
left for E = eeT1 and on the right for E = eeT ). Hence, using (3.17),

1 kL(X, E)k1 kK(X) vec(E)k1 kL(X, E)k1

≤ ≤n .
n kEk1 k vec(E)k1 kEk1

Maximizing over all E gives the result.

One of the uses of the condition number is to estimate the error in a computed
result produced by a backward stable method. Since rounding error bounds invariably
contain pessimistic constants that are quadratic or cubic in n, the agreement between
kL(X)k1 and kK(X)k1 to within a factor n is sufficient for it to be reasonable to use
the latter quantity to estimate the former.
In considering how to estimate kK(X)k we treat first the Frobenius norm. In
view of (3.20) the power method can be applied. We first state the power method for
estimating the 2-norm of a general matrix.

Algorithm 3.19 (power method). Given A ∈ Cn×n this algorithm uses the power
method applied to A∗A to produce an estimate γ ≤ kAk2 .

1 Choose a nonzero starting vector z0 ∈ Cn

2 for k = 0: ∞
3 wk+1 = Azk
4 zk+1 = A∗ wk+1
5 γk+1 = kzk+1 k2 /kwk+1 k2
6 if converged, γ = γk+1 , quit, end
7 end

In practice we would normalize wk and zk to unit 2-norm after computing them,

to avoid subsequent overflow and underflow. We have omitted the normalizations to
avoid cluttering the algorithm.
To analyze convergence we can exploit the fact that we are applying the power
method to the Hermitian positive semidefinite matrix A∗A. Let A have singular values
σ1 = · · · = σp > σp+1 ≥ · · · ≥ σn (1 ≤ p ≤ n). It is straightforward to show that
γk → kAk2 linearly as k → ∞ provided that z0 has a nonzero component in the space
spanned by the right singular vectors corresponding to σ1 , . . . , σp .
Note that given wk+1 = Azk and zk+1 = A∗ wk+1 = A∗Azk we have three lower
bounds for kAk2 , given in
"  1/2 #
kwk+1 k2 kzk+1 k2 kzk+1 k2
kAk2 ≥ max , , . (3.21)
kzk k2 kwk+1 k2 kzk k2

Now
kwk+1 k22 = wk+1
∗
wk+1 = zk∗ A∗ wk+1 = zk∗ zk+1 ,
so kwk+1 k22 ≤ kzk k2 kzk+1 k2 , which implies

kwk+1 k2 kzk+1 k2 kzk+1 k2 kzk+1 k22

≤ , ≤ .
kzk k2 kwk+1 k2 kzk k2 kwk+1 k22

Hence γ in Algorithm 3.19 is the best estimate obtainable from the lower bounds in
(3.21).
66 Conditioning

To estimate kL(X)kF we simply apply Algorithm 3.19 to A = K(X). The re-

sulting algorithm can be written entirely in terms of L(X) and L⋆ (X), the adjoint
of L(X) defined with respect to the inner product hX, Y i = trace(Y ∗ X). When
X ∈ Rn×n and f : Rn×n → Rn×n , the adjoint is given by L⋆ (X) = L(X T ). In the
complex case, L⋆ ∗
f (X) = Lf (X ), where f (z) := f (z), so that if f has a power series
representation then f is obtained by conjugating the coefficients.

Algorithm 3.20 (power method on Fréchet derivative). Given X ∈ Cn×n and the
Fréchet derivative L of a function f , this algorithm uses the power method to produce
an estimate γ ≤ kL(X)kF .

1 Choose a nonzero starting matrix Z0 ∈ Cn×n

2 for k = 0: ∞
3 Wk+1 = L(X, Zk )
4 Zk+1 = L⋆ (X, Wk+1 )
5 γk+1 = kZk+1 kF /kWk+1 kF
6 if converged, γ = γk+1 , quit, end
7 end

A random Z0 is a reasonable choice in Algorithm 3.20. A possible expression for

“converged” in the convergence tests of Algorithms 3.19 and 3.20 is

k > it max or |γk+1 − γk | ≤ tol γk+1 ,

where it max is the maximum allowed number of iterations and tol is a relative conver-
gence tolerance. Since an estimate of just the correct order of magnitude is required,
tol = 10−1 or 10−2 may be suitable. However, since linear convergence can be arbi-
trarily slow it is difficult to construct a truly reliable convergence test.
An alternative to the power method for estimating the largest eigenvalue of a
Hermitian matrix is the Lanczos algorithm. Mathias [408, ] gives a Lanczos-
based analogue of Algorithm 3.20. He shows that the Lanczos approach generates
estimates at least as good as those from Algorithm 3.20 at similar cost, but notes
that for obtaining order of magnitude estimates the power method is about as good
as Lanczos.
Turning to the 1-norm, we need the following algorithm.

Algorithm 3.21 (LAPACK matrix norm estimator). Given A ∈ Cn×n this algo-
rithm computes γ and v = Aw such that γ ≤ kAk1 with kvk1 /kwk1 = γ (w is
not returned). For z ∈ C, sign(z) = z/|z| if z 6= 0 and sign(0) = 1.5

1 v = A(n−1 e)
2 if n = 1, quit with γ = |v1 |, end
3 γ = kvk1
4 ξ = sign(v)
5 x = A∗ ξ
6 k=2
7 repeat
8 j = min{ i: |xi | = kxk∞ }
9 v = Aej
5 This definition of sign is different from that used in Chapter 5.
3.4 Computing or Estimating the Condition Number 67

10 γ=γ
11 γ = kvk1
12 if (A is real and sign(v) = ±ξ) or γ ≤ γ, goto line 17, end
13 ξ = sign(v)
14 x = A∗ ξ
15 k =k+1
16 until (kxk∞ = xj or k
> 5)
i−1
17 xi = (−1)i+1 1 + n−1 , i = 1: n
18 x = Ax
19 if 2kxk1 /(3n) > γ then
20 v=x
21 γ = 2kxk1 /(3n)
22 end

Algorithm 3.21 is the basis of all the condition number estimation in LAPACK
and is used by MATLAB’s rcond function. MATLAB’s normest1 implements a block
generalization of Algorithm 3.21 due to Higham and Tisseur [288, ] that iterates
with an n × t matrix where t ≥ 1; for t = 1, Algorithm 3.21 (without lines 17–22) is
recovered.
Key properties of Algorithm 3.21 are that it typically requires 4 or 5 matrix–
vector products, it frequently produces an exact estimate (γ = kAk1 ), it can produce
an arbitrarily poor estimate on specially constructed “counterexamples”, but it almost
invariably produces an estimate correct to within a factor 3. Thus the algorithm is a
very reliable means of estimating kAk1 .
We can apply Algorithm 3.21 with A = K(X) and thereby estimate kL(X)k1 .

Algorithm 3.22 (LAPACK matrix norm estimator on Fréchet derivative). Given a

matrix X ∈ Cn×n this algorithm uses the LAPACK norm estimator to produce an
estimate γ of kL(X)k1 , given the ability to computeL(X, E) and L⋆ (X, E) for
 any
E. More precisely, γ ≤ kK(X)k1 , where kK(X)k1 ∈ n−1 kL(X)k1 , nkL(X)k1 .

1 Apply Algorithm 3.21 to the matrix A := K(X),

noting that Ay ≡ vec(L(X, E)) and A∗ y ≡ vec(L⋆ (X, E)), where vec(E) = y.

Advantages of Algorithm 3.22 over Algorithm 3.20 are a “built-in” starting matrix
and convergence test and a more predictable number of iterations.
Algorithms 3.20 and 3.22 both require two Fréchet derivative evaluations per iter-
ation. One possibility is to approximate these derivatives by finite differences, using,
from (3.11),
f (X + tE) − f (X)
L(X, E) ≈ =: ∆f (X, t, E) (3.22)
t
for a small value of t. The choice of t is a delicate matter—more so than for scalar
finite difference approximations because the effect of rounding errors on the evaluation
of f (X + tE) is more difficult to predict. A rough guide to the choice of t can be
developed by balancing the truncation and rounding errors. For a sufficiently smooth
f , (3.6) implies f (X + tE) − f (X) − L(X, tE) = O(t2 kEk2 ). Hence the truncation
error ∆f (X, t, E) − L(X, E) can be estimated by tkEk2 . For the evaluation of f (X)
we have at best f l(f (X)) = f (X) + E, where kEk ≤ ukf (X)k. Hence

kf l(∆f (X, t, E)) − ∆f (X, t, E)k ≤ u(kf (X + tE)k + kf (X)k)/t ≈ 2ukf (X)k/t.
68 Conditioning

−4
10

−5
Relative error 10

−6
10

−7
10

−8
10
−11 −10 −9 −8 −7 −6
10 10 10 10 10 10
t

Figure 3.1. Relative errors in the Frobenius norm for the finite difference approximation
(3.22) with f (A) = eX , X and E random matrices from the normal (0, 1) distribution, and
250 different t. The dotted line is u1/2 and the circle on the x-axis denotes topt in (3.23).

The error due to rounding is therefore estimated by ukf (X)k/t. The natural choice
of t is that which minimizes the maximum of the error estimates, that is, t for which
tkEk2 = ukf (X)k/t, or
 1/2
ukf (X)k
topt = . (3.23)
kEk2

The minimum is u1/2 kf (X)k1/2 kEk. In practice, topt is usually fairly close to mini-
mizing the overall error. Figure 3.1 shows a typical example.
For methods for computing f (X) that employ a Schur decomposition, X = QT Q∗ ,
an efficient way to obtain the function evaluation f (X + G) in (3.22), where G ≡ tE,
has been suggested by Mathias [411, ]. The idea is to write f (X + G) = Qf (T +
e ∗ , where G
G)Q e = Q∗ GQ, and then reduce G e to upper triangular form by nearly
unitary transformations, exploiting the fact that G e is small. Then the underlying
method is used to evaluate f at the triangular matrix. See [411, ] for details.
Finally, we briefly mention a probabilistic approach to condition estimation. To
first order in t, (3.22) gives k∆f (X, t, E)kF ≤ kL(X)kF kEkF . Kenney and Laub [346,
] propose choosing E with elements from the normal (0,1) distribution, scaling
so that kEkF = 1, and then approximating kL(X)kF by φn k∆f (X, t, E)kF , where φn
is a constant such that the expected value of k∆f (X, t, E)kF is kL(X)kF . They give
an explicit formula for φn and show how to evaluate the probability that the estimate
is within a certain factor of kL(X)kF , assuming that the O(t) error in (3.22) can be
ignored. Several independent E can be used in order to get a better estimate. For
example, with two E the estimate is within a factor 5 of kL(X)kF with probability
at least 0.9691 (for all n). The main weaknesses of this approach are that the theory
applies only to real matrices, it is unclear how small t must be for the theory to be
valid, and the method is expensive if many significant digits are required with high
3.5 Notes and References 69

probability. Probabilistic estimates for the power method and Lanczos method are
also available; see Dixon [160, ], Kuczyński and Woźniakowski [365, ], and
Van Dorsselaer, Hochstenbach, and Van der Vorst [590, ].

3.5. Notes and References

For more details on the Fréchet derivative, and on connections between Fréchet and
Gâteaux derivatives, see, for example, Aubin and Ekeland [23, , Sec. 1.4], Atkin-
son and Han [22, , Sec. 5.3], Bhatia [64, , Sec. X.4], or Ortega and Rheinboldt
[453, , Sec. 3.1].
There is a literature on matrix differential calculus aimed at application areas
and not focusing on matrix functions as defined in this book. See, for example, the
book by Magnus and Neudecker [402, ] concerned particularly with statistics and
psychometrics.
Our definitions (3.2) and (3.4) of condition number, and Theorem 3.1, are special
cases of definitions and results of Rice [487, ].
Theorem 3.6 is from Mathias [412, ], where it is stated in a form that requires
f to be only 2m − 1 times continuously differentiable, where m is the size of the
largest Jordan block of A(t), for all t in some neighbourhood of 0. The identity (3.13)
is proved by Najfeld and Havel [445, , Thm. 4.11] under the assumption that f
is analytic. Theorem 3.7 is from Horn and Johnson [296, , Thm. 6.6.14], with
conditions modified as in [412, ].
Theorem 3.9 appears to be new in the form stated. A weaker version that assumes
f has a power series expansion and does not show that all eigenvalues of L(X) are
accounted for is given by Kenney and Laub [340, ].
Theorem 3.11 is due to Daleckiı̆ and Kreı̆n [129, ], [130, ]. Presentations
of this and more general results can be found in Bhatia [64, , Sec. V.3] and Horn
and Johnson [296, , Sec. 6.6].
Theorem 3.14 and Corollary 3.16 are obtained by Kenney and Laub [340, ]
in the case where f has a convergent power series representation. Their proofs work
with K(X), which in this case has an explicit representation in terms of Kronecker
products; see Problem 3.6. That Corollary 3.16 holds without this restriction on f is
noted by Mathias [411, ].
Convergence analysis for the power method for computing an eigenpair of a general
matrix B can be found in Golub and Van Loan [224, , Sec. 7.3], Stewart [538, ,
Sec. 2.1], Watkins [607, , Sec. 5.3], and Wilkinson [616, , Sec. 9.3]. In these
analyses the assumption of a dominant eigenvalue is needed to guarantee convergence;
for Algorithm 3.19, with B ≡ A∗A, no such assumption is needed because eigenvalues
of B of maximal modulus are necessarily equal.
The power method in Algorithm 3.20 is suggested by Kenney and Laub [340,
].
Algorithm 3.21 was developed by Higham [270, ]. The algorithm is based
on a p-norm power method of Boyd [79, ], also investigated by Tao [563, ]
and derived independently for the 1-norm by Hager [237, ]. For more details see
Higham [270, ], [271, ], and [276, , Chap. 15].
70 Conditioning

Problems
3.1. Evaluate the Fréchet derivatives L(X, E) of F (X) = I, F (X) = X, and F (X) =
cos(X), assuming in the last case that XE = EX.
3.2. Show that if X = QT Q∗ is a Schur decomposition then L(X, E) = QL(T, Q∗ EQ)Q∗ .
3.3. Show that the Fréchet derivative is unique.
3.4. Prove (3.11), namely that the Fréchet derivative is a directional derivative.
 
3.5. Let A = A11 A12
0 A22 , where A11 ∈ C
n1 ×n1
and A22 ∈ Cn2 ×n2 with n = n1 + n2 .
What is the maximum size of a Jordan block of A?
P∞
3.6. Let the power series f (x) = i=0 ai xi have radius of convergence r. Show that
for X, E ∈ Cn×n with kXk < r, the Fréchet derivative
∞
X i
X
L(X, E) = ai X j−1 EX i−j , (3.24)
i=1 j=1

and hence that K(X) in (3.17) is given by

∞
X i
X
K(X) = ai (X T )i−j ⊗ X j−1 .
i=1 j=1

3.7. Show that if f has a power series expansion with real coefficients then L(X ∗ , E) =
L(X, E ∗ )∗ .
3.8. Show that if X and E commute then L(X, E) = f ′ (X)E = Ef ′ (X), where f ′
denotes the derivative of the scalar function f .
3.9. (Stickel [541, ], Rinehart [494, ]) Suppose that f is analytic on and
inside a closed contour Γ that encloses Λ(X). Show that the Fréchet derivative of f
is given by Z
1
L(X, E) = f (z)(zI − X)−1 E (zI − X)−1 dz.
2πi Γ
Deduce that if XE = EX then L(X, E) = f ′ (X)E = Ef ′ (X), where f ′ denotes the
derivative of the scalar function f .
3.10. Consider any two eigenvalues λ and µ of X ∈ Cn×n , with corresponding right
and left eigenvectors u and v, so that Xu = λu and v T X = µv T . Show directly
(without using Kronecker products or reducing to the case that f is a polynomial)
that uv T is an eigenvector of L(X) with corresponding eigenvalue f [λ, µ].
3.11. (Research problem) Determine the Jordan form of the Fréchet derivative
L(X) of f in terms of that
 of X. To see that this question is nontrivial, note that for
f (X) = X 2 and X = 00 10 , L(X) has one Jordan block of size 3 and one of size 1,
both for the eigenvalue 0.
Chapter 4
Techniques for General Functions

Many different techniques are available for computing or approximating matrix func-
tions, some of them very general and others specialized to particular functions. In this
chapter we survey a variety of techniques applicable to general functions f . We be-
gin with the basic tasks of evaluating polynomial and rational functions, and address
the validity of matrix Taylor series and the truncation error when a finite number of
terms are summed. Then we turn to methods based on similarity transformations,
concentrating principally on the use of the Schur decomposition and evaluation of a
function of a triangular matrix. Matrix iterations are an important tool for certain
functions, such as matrix roots. We discuss termination criteria, show how to define
stability in terms of a Fréchet derivative, and explain how a convergence result for
a scalar iteration can be translated into a convergence result for the corresponding
matrix iteration. Finally, we discuss preprocessing, which may be beneficial before
applying a particular algorithm, and present several bounds on kf (A)k. Many of the
methods for specific f described later in the book make use of one or more of the
techniques treated in this chapter.
We describe the cost of an algorithm in one of two ways. If, as is often the case,
the algorithm is expressed at the matrix level, then we count the number of matrix
operations. Thus

• M denotes a matrix multiplication, AB,

• I denotes a matrix inversion, A−1 ,

• D denotes a “matrix division”, that is, the solution of a multiple right-hand

side linear system AX = B,

where the matrices A and B are n × n. Other operation counts are given in terms
of flops, where a flop denotes any of the four elementary operations on scalars +, −,
∗, /. The costs in flops of various matrix operations are summarized in Appendix C,
where some comments on the relevance and interpretation of these different measures
are given.
It is worth clarifying the terms “method” and “algorithm”. For us, a method is
usually a general technique that, when the details are worked out, can lead to more
than one algorithm. An algorithm is a specific automatic procedure that we name
“Algorithm” and specify using pseudocode.

4.1. Matrix Powers

A basic requirement in some methods for matrix functions is to compute a power of
a matrix. A sequence of successive powers A2 , A3 , . . . , should be computed in the

71
72 Techniques for General Functions

obvious way: by repeated multiplication by A. But if a single power, Am , is needed,

it is not necessary to compute all the lower powers first. Instead, repeated squaring
can be used, as described in the following algorithm. The initial while loop simply
ensures that multiplication of a power of A with the identity matrix is avoided.

Algorithm 4.1 (binary powering). This algorithm evaluates X = Am for A ∈ Cn×n .

Pt i
1 Let m = i=0 βi 2 be the binary representation of m, with βt 6= 0.
2 P =A
3 i=0
4 while βi = 0
5 P = P2
6 i=i+1
7 end
8 X =P
9 for j = i + 1: t
10 P = P2
11 if βj = 1
12 X = XP
13 end
14 end


Cost: ⌊log2 m⌋ + µ − 1 M ≤ 2⌊log2 m⌋M , where µ ≤ ⌈log2 m⌉ is the number of 1s in
the binary representation of m.
k
A special case of binary powering is repeated squaring to form A2 , which is used
in the scaling and squaring method for the matrix exponential; see Section 10.3.

4.2. Polynomial Evaluation

Many methods for computing matrix functions require the evaluation of a matrix
polynomial
Xm
pm (X) = bk X k , X ∈ Cn×n . (4.1)
k=0
The economics of the evaluation are rather different than for scalar polynomials.
Whereas Horner’s method (nested multiplication) is almost always used in the scalar
case, for matrix polynomials there are four competing methods.
First, pm can be evaluated by Horner’s method.

Algorithm 4.2 (Horner’s method). This algorithm evaluates the polynomial (4.1)
by Horner’s method.
1 Sm−1 = bm X + bm−1 I
2 for k = m − 2: −1: 0
3 Sk = XSk+1 + bk I
4 end
5 pm = S 0
Cost: (m − 1)M .
Horner’s method is not suitable when m is not known at the start of the evaluation,
as is often the case when a truncated power series is to be summed. In this case pm
can be evaluated by explicitly forming each power of X.
4.2 Polynomial Evaluation 73

Algorithm 4.3 (evaluate polynomial via explicit powers). This algorithm evaluates
the polynomial (4.1) by explicitly forming matrix powers.

1 P =X
2 S = b0 I + b1 X
3 for k = 2: m
4 P = PX
5 S = S + bk P
6 end
7 Pm = S

Cost: (m − 1)M .
Note that while Algorithms 4.2 and 4.3 have the same cost for matrices, when X
is a scalar Algorithm 4.3 is twice as expensive as Algorithm 4.2.
Another method factorizes the polynomial pm (x) = bm (x − ξ1 ) . . . (x − ξm ) and
then evaluates this factorized form at the matrix X.

Algorithm 4.4 (evaluate polynomial in factored form). This algorithm evaluates the
polynomial (4.1) given the roots ξ1 , . . . , ξm of pm .

1 S = X − ξm I
2 for k = m − 1: −1: 1
3 S = S(X − ξk I)
4 end
5 pm = S

Cost: (m − 1)M .
One drawback to Algorithm 4.4 is that some of the roots ξj may be complex and so
complex arithmetic can be required even when the polynomial and X are real. In such
situations the algorithm can be adapted in an obvious way to employ a factorization
of pm into real linear and quadratic factors.
The fourth and least obvious method is that of Paterson and Stockmeyer [466,
], [224, , Sec. 11.2.4], in which pm is written as
r
X
pm (X) = Bk · (X s )k , r = ⌊m/s⌋, (4.2)
k=0

where s is an integer parameter and


bsk+s−1 X s−1 + · · · + bsk+1 X + bsk I, k = 0: r − 1,
Bk =
bm X m−sr + · · · + bsr+1 X + bsr I, k = r.

The powers X 2 , . . . , X s are computed; then (4.2) is evaluated by Horner’s method,

with each Bk formed when needed. The two extreme cases are s = 1, which reduces
to Algorithm 4.2, and s = m, which reduces to Algorithm 4.3. As an example, for
m = 6 and s = 3 we have

p6 (X) = b6 I (X 3 )2 + (b5 X 2 + b4 X + b3 I )X 3 + (b2 X 2 + b1 X + b0 I ),

|{z} | {z } | {z }
B2 B1 B0
74 Techniques for General Functions

Table 4.1. Number of matrix multiplications required by the Paterson–Stockmeyer method

and Algorithms 4.2 and 4.3 to evaluate a degree m matrix polynomial.

m 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16
PS method 1 2 2 3 3 4 4 4 5 5 5 6 6 6 6
Algs 4.2/4.3 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15

which can be evaluated in 3M , compared with the 5M required for Horner’s method.
Note that (4.2) is a polynomial with matrix coefficients, and the cost of evaluation
given the Bk and X s is rM . The total cost of evaluating pm is

n
1 if s divides m,
s + r − 1 − f (s, m) M, f (s, m) = (4.3)
0 otherwise.
√ √
This√quantity is approximately minimized by s = m, so we take for s either ⌊ m⌋
or ⌈ m ⌉; it can be shown that both choices yield the same operation count. As an
2
extreme case, this method evaluates Aq as (Aq )q , which clearly requires much less
work than the previous two methods for large q, though for a single high power of A
binary powering (Algorithm 4.1) is preferred.
Table 4.1 shows the cost of the Paterson–Stockmeyer method for m = 2: 16; for
each m ≥ 4 it requires strictly fewer multiplications than Algorithms 4.2 and 4.3. For
each m in the table it can be shown that both choices of s minimize (4.3) [247, ].
Unfortunately, the Paterson–Stockmeyer method requires (s + 2)n2 elements of
storage. This can be reduced to 4n2 by computing pm a column at a time, as shown
by Van Loan [596, ], though the p cost of evaluating pm then increases to (2s +
r − 3 − f (s, m))M . The value s = m/2 approximately minimizes the cost of Van
Loan’s variant, and it then costs about 40% more than the original method.
It is important to understand the effect of rounding errors on these four polynomial
evaluation methods. The next theorem provides error bounds for three of the methods.
For matrices, absolute values and inequalities are defined componentwise. We write
γ
en = cnu/(1−cnu), where u is the unit roundoff and c is a small integer constant whose
precise value is unimportant. For details of our model of floating point arithmetic see
Section B.15.

Theorem 4.5. The computed polynomial pbm obtained by applying Algorithm 4.2,
Algorithm 4.3, or the Paterson–Stockmeyer method to pm in (4.1) satisfies

|pm − pbm | ≤ γ
emn pem (|X|),
Pm
where pem (X) = k=0 |bk |X k . Hence kpm − pbm k1,∞ ≤ γ
emn pem (kXk1,∞ ).

The bound of the theorem is pessimistic in the sense that inequalities such as
|X j | ≤ |X|j are used in the derivation. But the message of the theorem is clear: if
there is significant cancellation in forming pm then the error in the computed result
can potentially be large. This is true even in the case of scalar x. Indeed, Stegun and
Abramowitz [535, ] presented the now classic example of evaluating e−5.5 from a
truncated Taylor series and showed how cancellation causes a severe loss of accuracy
in floating point arithmetic; they also noted that computing e5.5 and reciprocating
avoids the numerical problems.
4.2 Polynomial Evaluation 75

10
10

0
10

−10
10
−16
10
0 10 20 30 40 50 60 70 80 90 100

Figure 4.1. 2-norms of first 99 terms in Taylor series of eA , for A in (4.4) with α = 25.

For a matrix example we take

 
0 α
A= , (4.4)
−α 0

for which  
A cos α sin α
e = .
− sin α cos α

We took α = 25 and summed the first 99 terms of the Taylor series for eA using
Algorithm 4.3; at this point adding further terms makes no difference to the computed
sum. The computed sum X b has error keA − Xk b 2 = 1.5 × 10−7 , which represents a
loss of 9 significant digits in all components of X. This loss of significance can be
understood with the aid of Figure 4.1, which shows that the terms in the series grow
rapidly in norm, reaching a maximum of order 1010 . Since all the elements of eA are
of order 1, there is clearly massive cancellation in the summation, and based on the
size of the maximum term a loss of about 10 significant digits would be expected.
Turning to Theorem 4.5, the 2-norm of u pem (|X|) is 8 × 10−6 , so the upper bound of
the theorem is reasonably sharp in this example if we set the constant γ emn to u.
Unlike in the scalar example of Stegun and Abramowitz, computing e−A and then
T
inverting does not help in this example, since e−A = eA in this case and so the
Taylor series is merely transposed.
Error analysis for Algorithm 4.4 is essentially the same as error analysis of a matrix
product (see [276, , Secs. 3.7, 18.2]).

Theorem 4.6. The computed polynomial pbm obtained by applying Algorithm 4.4 to
pm in (4.1) satisfies

|pm − pbm | ≤ γ
emn |bm ||X − ξ1 I| . . . |X − ξm I|. (4.5)

Note that this theorem assumes the ξj are known exactly. The ξk can be ill
conditioned functions of the coefficients bk , so in practice the errors in computing the
ξj could have a significant effect. The main thing to note about the bound (4.5) is
that it depends on the ordering of the ξk , since the matrices |X −ξk I| do not commute
with each other in general. An ordering that has been suggested is the Leja ordering
[485, ]; see the discussion in [444, ]. We will not consider Algorithm 4.4
further because the special nature of the polynomials in matrix function applications
tends to make the other algorithms preferable.
76 Techniques for General Functions

4.3. Taylor Series

A basic tool for approximating matrix functions is the Taylor series. We begin with
a theorem that guarantees the validity of a matrix Taylor series if the eigenvalues of
the “increment” lie within the radius of convergence of the associated scalar Taylor
series.

Theorem 4.7 (convergence of matrix Taylor series). Suppose f has a Taylor series
expansion
∞
X  
f (k) (α)
f (z) = ak (z − α)k ak = (4.6)
k!
k=0

with radius of convergence r. If A ∈ Cn×n then f (A) is defined and is given by

∞
X
f (A) = ak (A − αI)k (4.7)
k=0

if and only if each of the distinct eigenvalues λ1 , . . . , λs of A satisfies one of the

conditions
(a) |λi − α| < r,
(b) |λi − α| = r and the series for f (ni −1) (λ) (where ni is the index of λi ) is
convergent at the point λ = λi , i = 1: s.

Proof. It is easy to see from Definition 1.2 that it suffices to prove the theorem
n×n
Pm A = J(λ)k= λI + N ∈ C
for a Jordan block, , where N is strictly upper triangular.
Let fm (z) = k=0 ak (z − α) . We have
m
X
k
fm (J(λ)) = ak (λ − α)I + N
k=0
m
X Xk  
k
= ak (λ − α)k−i N i
i=0
i
k=0
Xm Xm  
i k
= N ak (λ − α)k−i
i=0
i
k=i
m
X m
Ni X
= ak k(k − 1) . . . (k − i + 1)(λ − α)k−i
i=0
i!
k=i
m min(m,n−1)
X N i (i) X N i (i)
= f (λ) = f (λ).
i=0
i! m i=0
i! m

(i)
Evidently, limm→∞ fm (J(λ)) exists if and only if limm→∞ fm (λ) exists for i = 1: n−1,
which is essentially the statement of case (b), because if the series for f differentiated
term by term ni − 1 times converges at λ then so does the series differentiated j times
for j = 0: ni − 1. Case (a) follows from the standard result in complex analysis that
a power series differentiated term by term converges within the radius of convergence
of the original series.
4.3 Taylor Series 77

The four most important matrix Taylor series are

A2 A3
exp(A) = I + A + + + ···,
2! 3!
2 4 6
A A A
cos(A) = I − + − + ···,
2! 4! 6!
A3 A5 A7
sin(A) = A − + − + ···,
3! 5! 7!
A2 A3 A4
log(I + A) = A − + − + ···, ρ(A) < 1,
2 3 4
the first three series having infinite radii of convergence. These series can be used to
approximate the respective functions, by summing a suitable finite number of terms.
Two types of error need to be addressed: truncation errors, and rounding errors in
the floating point evaluation. Truncation errors are bounded in the following result.

Theorem 4.8 (Taylor series truncation error bound). Suppose f has the Taylor se-
ries expansion (4.6) with radius of convergence r. If A ∈ Cn×n with ρ(A − αI) < r
then for any matrix norm
s−1
X
1
f (A) − k
ak (A − αI) ≤ max (A − αI)s f (s) (αI + t(A − αI)) . (4.8)
s! 0≤t≤1
k=0

Proof. See Mathias [409, , Cor. 2]. Note that this bound does not con-
tain a factor depending on n, unlike the 2-norm version of the bound in [224, ,
Thm. 11.2.4]. Moreover, the norm need not be consistent.
In order to apply this theorem we need to bound the term max0≤t≤1 kZ s f (s) (αI +
tZ)k∞ . For certain f this is straightforward. We illustrate using the cosine function.
With α = 0, s = 2k + 2, and
2k
X (−1)i
T2k (A) = A2i ,
i=0
(2i)!

the bound of Theorem 4.8 is, for the ∞-norm,

1
k cos(A) − T2k (A)k∞ ≤ max kA2k+2 cos(2k+2) (tA)k∞
(2k + 2)! 0≤t≤1
1
≤ kA2k+2 k∞ max k cos(2k+2) (tA)k∞ .
(2k + 2)! 0≤t≤1

Now

max k cos(2k+2) (tA)k∞ = max k cos(tA)k∞

0≤t≤1 0≤t≤1

kAk2∞ kAk4∞
≤1+ + + · · · = cosh(kAk∞ ),
2! 4!
and so the error in the truncated Taylor series approximation to the matrix cosine
satisfies the bound
kA2k+2 k∞
k cos(A) − T2k (A)k∞ ≤ cosh(kAk∞ ). (4.9)
(2k + 2)!
78 Techniques for General Functions

We also need to bound the error in evaluating T2k (A) in floating point arithmetic.
From Theorem 4.5 we have that if T2k (A) is evaluated by any of the methods of
Section 4.2 then the computed Tb2k satisfies

kT2k − Tb2k k∞ ≤ γ
ekn cosh(kAk∞ ).

Hence  
k cos(A) − Tb2k k∞ kA2k+2 k∞ cosh(kAk∞ )
≤ +γ
ekn . (4.10)
k cos(A)k∞ (2k + 2)! k cos(A)k∞
We can draw two conclusions. First, for maximum accuracy we should choose k so
that
kA2k+2 k∞
≈γekn .
(2k + 2)!
Second, no matter how small the truncation error, the total relative error can poten-
tially be as large as γ
ekn cosh(kAk∞ )/k cos(A)k∞ , and this quantity can be guaranteed
to be of order γekn only if kAk∞ <∼ 1. The essential problem is that if kAk∞ ≫ 1 then
there can be severe cancellation in summing the series.
If kAk∞ ≤ 1 then, using k cos (A)k∞ ≥ 1 − (cosh(kAk∞ ) − 1) = 2 − cosh(kAk∞ ),
we have
0.45 ≤ 2 − cosh(1) ≤ k cos (A)k∞ ≤ cosh(1) ≤ 1.55, (4.11)
which gives
cosh(kAk∞ )
≤ 3.4.
k cos(A)k∞
We conclude that a relative error k cos(A) − Tb2k k∞ /k cos(A)k∞ of order γ
ekn is guar-
anteed if kAk∞ ≤ 1 and k is sufficiently large. In fact, since 18! ≈ 6 × 1015 , k = 8
suffices in IEEE standard double precision arithmetic, for which the unit roundoff
u ≈ 1.1 × 10−16 .

4.4. Rational Approximation

Rational functions
pkm (x)
rkm (x) = ,
qkm (x)
where pkm and qkm are polynomials in x of degree at most k and m, respectively, are
powerful tools for approximation. An advantage over polynomials is that they are
better able than polynomials to mimic the behaviour of general nonlinear functions.
When a scalar approximation f (x) ≈ rkm (x) is translated into a matrix ap-
proximation f (A) ≈ rkm (A), the scalar approximation is essentially being applied
on the spectrum of A. However, a good approximation on the spectrum does not
imply a good approximation overall. Suppose, for simplicity, that A is diagonaliz-
able with A = XDX −1 , D = diag(λi ). Defining e(x) = f (x) − rkm (x), we have
e(A) = Xe(D)X −1 and so
ke(A)k ≤ κ(X)ke(D)k.
If A is normal, in which case we can take X unitary and hence κ2 (X) = 1, this bound
is very satisfactory. But in general κ(X) can be large and ke(A)k can be large even
though ke(D)k is small. Therefore, to produce useful error bounds either rational
approximations need to be applied to a restricted class of A or approximation errors
must be analyzed at the matrix level.
4.4 Rational Approximation 79

The main classes of rational approximation used in computing matrix functions

are those described in the following two subsections.
Let Rk,m denote the space of rational functions with numerator and denominator
of degrees at most k and m, respectively.

4.4.1. Best L∞ Approximation

The rational function r is a best L∞ (or minimax, or Chebyshev) approximation to f
on [a, b] from Rk,m if

kr(x) − f (x)k∞ = min ks(x) − f (x)k∞ ,

s∈Rk,m

where kgk∞ = maxx∈[a,b] |g(x)|. Best L∞ rational approximations can be constructed

using the Remez algorithm, a standard algorithm in approximation theory [477, ],
[609, ]. These approximations are usually employed for Hermitian matrices only,
so that error bounds for the scalar problem translate directly into error bounds at the
matrix level.
In this book we will discuss best L∞ approximations for the matrix sign function
(Section 5.9) and the matrix exponential (Section 10.7.1).

4.4.2. Padé Approximation

For a given scalar function f (x) the rational function rkm (x) = pkm (x)/qkm (x) is a
[k/m] Padé approximant of f if rk,m ∈ Rk,m , qkm (0) = 1, and

f (x) − rkm (x) = O(xk+m+1 ). (4.12)

If a [k/m] Padé approximant exists then it is unique; see Problem 4.2. It is usually
required that pkm and qkm have no common zeros, so that pkm and qkm are unique.
For a given f , k, and m, a [k/m] Padé approximant might not exist, though for certain
f existence has been proved for all k and m.
The condition (4.12) shows that rkm reproduces the first k + m + 1 terms of the
Taylor series of f about the origin, and of course if m = 0 then rkm is precisely a
truncated Taylor series.
Continued fraction representations
a1 x
f (x) = b0 +
a2 x
b1 +
a3 x
b2 +
b3 + · · ·
are intimately connected with Padé approximation and provide a convenient way of
obtaining them. Specifically, if b1 = b2 = · · · = 1 and the ai are all nonzero then the
convergents
a1 x
rm (x) ≡ rmm (x) = b0 + . (4.13)
a2 x
b1 +
a3 x
b2 +
a2m−1 x
b3 + · · · +
a2m x
b2m−1 +
b2m
80 Techniques for General Functions

Table 4.2. Number of matrix multiplications required by the Paterson–Stockmeyer method to

evaluate √
both pmm (A) and qmm
√ (A). The second row indicates whether this cost is achieved
for s = ⌊ 2m⌋ (F ) or s = ⌈ 2m ⌉ (C ), or both.

m 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16
1 2 3 4 4 5 5 6 6 7 7 8 8 8 9
FC C F F F C FC C C F F F F F C

are the [0/0], [1/0], [1/1], [2/1], [2/2], . . . Padé approximants of f [38, , Sec 4.D],
[39, , Thm. 4.2.1].
Padé approximants are of particular interest in matrix function approximation for
three main reasons:

• they can potentially produce approximations of a given accuracy with lower

computational cost than a polynomial approximation such as a truncated Taylor
series;
• while for good accuracy they require x to be near the origin, which can for some
important functions be arranged by a suitable scaling, they do not require x to
be real;
• the theory of Padé approximation is very well developed, and Padé approxi-
mants of some important functions are known explicitly, sometimes in several
representations.

Details of Padé approximants to specific functions will be given in later chapters.

Padé approximants can be computed symbolically with Maple, Mathematica, or
MATLAB with the Extended Symbolic Math Toolbox.

4.4.3. Evaluating Rational Functions

An important question is how to evaluate a given rational function at a matrix ar-
gument. Several possibilities exist, corresponding to different representations of the
rational.
Suppose, first, that rkm is specified by its numerator and denominator polynomials
pkm and qkm . The obvious approach is to evaluate pkm and qkm by any of the methods
discussed in Section 4.2 and then to compute rkm (A) = qkm (A)−1 pkm (A) by solving a
multiple right-hand side linear system. In some cases it is possible to exploit similarity
between the coefficients of pkm and qkm in order to reduce the work.
When the Paterson–Stockmeyer method is used to evaluate pmm (A) and qmm (A),
some savings can be made. Referring to the description in Section 4.2, the powers
X 2 , . . . , X s can be computed once and used in both the p
mm and the qmm evaluations.
The cost of evaluating rmm is then s + 2r − 1 − 2f (s, m) M + D, where f is defined in
√
(4.3), and this quantity √ is approximately
√ minimized by s = 2m. We therefore take
for s whichever of ⌊ 2m⌋ and ⌈ 2m ⌉ yields the smaller operation count. Table 4.2
shows the cost of the evaluation for m = 2: 16.
Consider now the continued fraction representation (4.13). This expansion can be
evaluated at the matrix X in two ways: top down or bottom up. Top-down evaluation
(which converts the continued fraction to rational form) is effected by the following
recurrence, which dates back to Wallis (1655).
4.5 Diagonalization 81

Algorithm 4.9 (continued fraction, top-down). This algorithm evaluates the con-
tinued fraction (4.13) in top-down fashion at the matrix X ∈ Cn×n .

1 P−1 = I, Q−1 = 0, P0 = b0 I, Q0 = I
2 for j = 1: 2m
3 Pj = bj Pj−1 + aj XPj−2
4 Qj = bj Qj−1 + aj XQj−2
5 end
6 rm = P2m Q−1 2m

Cost: 2(2m−2)M +D. (Note that for j = 1, 2 the multiplications XPj−2 and XQj−2
are trivial, since Pj−2 and Qj−2 are multiples of I.)
Using bottom-up evaluation, rm (X) is evaluated as follows.

Algorithm 4.10 (continued fraction, bottom-up). This algorithm evaluates the con-
tinued fraction (4.13) in bottom-up fashion at the matrix X ∈ Cn×n .

1 Y2m = (a2m /b2m )X

2 for j = 2m − 1: −1: 1
3 Solve (bj I + Yj+1 )Yj = aj X for Yj .
4 end
5 rm = b0 I + Y1

Cost: (2m − 1)D.

The top-down evaluation is computationally expensive, but it is well suited to
situations in which the aj and bj are independent of m and the whole sequence
r1 (X), r2 (X), . . . , needs to be evaluated; in this case the bottom-up evaluation has
to start afresh each time.
Another representation of rm in (4.13) is in partial fraction form:
m (m)
X αj x
rm (x) = (m)
. (4.14)
j=1 1+ βj x

(m)
The coefficients βj are minus the reciprocals of the roots of the denominator poly-
nomial qmm (x) and so may be complex; in this case an alternative partial fraction
with quadratic denominators could be considered. The cost of evaluating (4.14) at
the matrix X is just mD, but on a parallel computer the m terms in (4.14) can be
evaluated in parallel.
Of course, the numerical stability of these different methods of evaluation needs
to be considered along with the computational cost. Since the stability depends
very much on the function f , we delay further consideration until later sections on
particular f .

4.5. Diagonalization
A wide class of methods for evaluating matrix functions is based on exploiting the
relation f (ZBZ −1 ) = Zf (B)Z −1 (Theorem 1.13 (c)). The idea is to factor A =
ZBZ −1 , with B of a form that allows easy computation of f (B). Then f (A) =
Zf (B)Z −1 is readily obtained.
82 Techniques for General Functions

The most obvious choice of similarity is a diagonalizing one, assuming A is indeed

diagonalizable: A = Z diag(λi )Z −1 , where the λi are the eigenvalues of A and the
columns of Z are eigenvectors. Then f (A) = Z diag(f (λi ))Z −1 . Listing 4.1 lists a
MATLAB function funm ev implementing this formula. Here is an example of how
the function is used to compute the matrix square root:
>> A = [3 -1; 1 1]; X = funm_ev(A,@sqrt)
X =
1.7678e+000 -3.5355e-001
3.5355e-001 1.0607e+000

>> norm(A-X^2)
ans =
9.9519e-009
Given that A has norm of order 1 and the unit roundoff u ≈ 10−16 , the residual
kA − X 2 k2 ≈ 10−8 of the computed X is disappointing—especially considering that
MATLAB’s sqrtm function achieves a residual of order u:
>> Y = sqrtm(A); norm(A-Y^2)
ans =
6.4855e-016
The explanation lies with the ill conditioning of the matrix Z:
>> [Z,D] = eig(A); cond(Z)
ans =
9.4906e+007
That κ2 (Z)u is roughly the size of the residual is no coincidence. Suppose the only
error in the process is an error E in evaluating f (B). Then we obtain

fe(A) = Z(f (B) + E)Z −1 = f (A) + ZEZ −1

and
kfe(A) − f (A)k ≤ kZk kEk kZ −1 k = κ(Z) kEk.
When B is diagonal and Gaussian elimination with partial pivoting is used in the
evaluation, we should interpret κ(Z) as

min{ κ(ZD) : D diagonal, nonsingular }, (4.15)

which for any p-norm is approximately achieved (and exactly achieved when p = 1)
when ZD has columns of unit p-norm; see [276, , Thm. 7.5, Sec. 9.8]. The con-
clusion is that we must expect errors proportional to κ(Z) in our computed function.
Since the conditioning of f (A) is not necessarily related to κ(Z), this diagonalization
method may be numerically unstable.
Diagonalization was used to compute certain matrix functions in the original For-
tran version of MATLAB (“Classic MATLAB”, 1978–1984), which was designed for
teaching purposes:
< M A T L A B >
Version of 01/10/84
4.5 Diagonalization 83

Listing 4.1: MATLAB function funm ev.

function F = funm_ev(A,fun)
%FUNM_EV Evaluate general matrix function via eigensystem.
% F = FUNM_EV(A,FUN) evaluates the function FUN at the
% square matrix A using the eigensystem of A.
% This function is intended for diagonalizable matrices only
% and can be numerically unstable.

[V,D] = eig(A);
F = V * diag(feval(fun,diag(D))) / V;

HELP is available

<>
help fun

FUN For matrix arguments X , the functions SIN, COS, ATAN,

SQRT, LOG, EXP and X**p are computed using eigenvalues D
and eigenvectors V . If <V,D> = EIG(X) then f(X) =
V*f(D)/V . This method may give inaccurate results if V
is badly conditioned. Some idea of the accuracy can be
obtained by comparing X**1 with X .
For vector arguments, the function is applied to each
component.

If A is not diagonalizable we can in theory evaluate f from Definition 1.2 based on

the Jordan canonical form. However, the Jordan canonical form cannot be reliably
computed in floating point arithmetic; even if it could, the similarity transformation
that produces it can again be very ill conditioned.
The diagonalization approach can be recommended only when the diagonalizing
transformation is guaranteed to be well conditioned. An important class of matri-
ces for which this guarantee holds is the class of normal matrices: A ∈ Cn×n for
which A∗A = AA∗ . This class includes orthogonal matrices, symmetric matrices,
skew-symmetric matrices, and their complex analogues the unitary, Hermitian, and
skew-Hermitian matrices. In fact, the normal matrices are precisely those that are
diagonalizable by a unitary matrix, and unitary matrices are perfectly conditioned in
the 2-norm.
One minor problem with the diagonalization approach is that if A is real with some
complex eigenvalues and f (A) is real, then rounding errors may cause the computed
f (A) to have a tiny nonzero imaginary part. This problem can be overcome by
discarding the computed imaginary part, but of course numerical instability may
potentially produce a large spurious imaginary part. (For symmetric matrices the
eigensystem is real and these considerations do not apply.) One way to overcome this
problem of nonreal computed f (A) for real A is to employ a block diagonalization
A = ZDZ −1 in which D, Z ∈ Rn×n and D has 2 × 2 diagonal blocks corresponding
to complex conjugate pairs of eigenvalues. This approach falls into the class of block
diagonalization methods considered in Section 4.7. Another approach is described in
84 Techniques for General Functions

the next section.

In summary, for a normal matrix, computing f (A) via a (unitary) diagonalization
is the method of choice if the diagonalization can be computed. In particular, this
method is recommended if A is symmetric or Hermitian.

4.6. Schur Decomposition and Triangular Matrices

We saw in the last section the importance of using well conditioned similarity transfor-
mations in order to maintain numerical stability. Taking this view to the extreme, we
can restrict to unitary transformations. In general, the closest one can go to diagonal
form via unitary similarities is the Schur triangular form. The Schur decomposition
factors A ∈ Cn×n as
Q∗ AQ = T,
where Q ∈ Cn×n is unitary and T ∈ Cn×n is upper triangular. The eigenvalues of A
appear on the diagonal of T .
The Schur decomposition can be computed with perfect backward stability by the
QR algorithm, and hence it is a standard tool in numerical linear algebra. For us,
the significance of the decomposition is that, since f (A) = Qf (T )Q∗ , it reduces the
f (A) problem to that of computing f (T ). We therefore now turn our attention to
functions of triangular matrices.
It is easy to see from any of the definitions that f (T ) is upper triangular with
diagonal elements f (tii ). In fact, explicit formulae are available for all the elements
of f (T ).

Theorem 4.11 (function of triangular matrix). Let T ∈ Cn×n be upper triangular

and suppose that f is defined on the spectrum of T . Then F = f (T ) is upper triangular
with fii = f (tii ) and
X
fij = ts0 ,s1 ts1 ,s2 . . . tsk−1 ,sk f [λs0 , . . . , λsk ],
(s0 ,...,sk )∈Sij

where λi = tii , Sij is the set of all strictly increasing sequences of integers that start
at i and end at j, and f [λs0 , . . . , λsk ] is the kth order divided difference of f at
λs0 , . . . , λsk .

Proof. See Davis [139, ], Descloux [148, ], or Van Loan [592, ].

It is worth examining the case n = 2 in Theorem 4.11. For λ1 6= λ2 we have

 
  f (λ2 ) − f (λ1 )
λ1 t12 f (λ1 ) t12
f = λ2 − λ1 . (4.16)
0 λ2
0 f (λ2 )

In the case λ1 = λ2 = λ we have, using (B.27),

   
λ t12 f (λ) t12 f ′ (λ)
f = . (4.17)
0 λ 0 f (λ)

(This formula is a special case of (3.16).)

4.6 Schur Decomposition and Triangular Matrices 85

We can also use Theorem 4.11 to check formula (1.4) for a function of a Jordan
block, T = Jk (λk ). Since the only nonzero off-diagonal elements of T are 1s on the
superdiagonal, we have, using (B.27) again,

f (j−i) (λk )
fij = ti,i+1 . . . tj−1,j f [λk , λk , . . . , λk ] = , i < j.
| {z } (j − i)!
j − i + 1 times

An expression for a function of an arbitrary block 2 × 2 block upper triangular

matrix is given in the following result, which generalizes (4.16) and (4.17).

Theorem 4.12 (Kenney and Laub). Let f be 2n−1 times continuously differentiable
and let      
A11 A12 A11 0 0 A12
A= , D= , N= .
0 A22 0 A22 0 0
Then f (A) = f (D) + L(D, N ) (i.e., the o(·) term in (3.6) is zero).

Proof. If f is a polynomial of degree m then the result is trivial if m ≤ 1 and

otherwise follows from the fact that f (D + N ) − f (D) − L(D, N ) comprises sums of
products of the form X1 X2 . . . Xp , p ≥ 2, where each Xi is either D or N and at least
two of the Xi are N . Each of these products is zero. For general f , Theorem 3.7
implies that the Fréchet derivative L(D, N ) of f is the same as that of the polynomial
pD⊕D . The result therefore follows from the first part.
An interesting property revealed by the theorem is that the (1,2) block of f (A)
depends only linearly on the (1,2) block of A.
To compute f (T ) via Theorem 4.11 would cost O(2n ) flops, which is prohibitively
expensive. A method that avoids the combinatorial explosion in the formulae of the
theorem was derived by Parlett [460, ]. He notes that f (T ) commutes with T (see
Theorem 1.13 (a)) and that, since the diagonal of F = f (T ) is known, the equation
T F = F T can be solved for the off-diagonal elements of F . Indeed, equating (i, j)
elements in this equation for i < j yields
j
X j
X
tik fkj = fik tkj
k=i k=i

or
j−1
X
fij (tii − tjj ) = tij (fii − fjj ) + (fik tkj − tik fkj ), (4.18)
k=i+1

which gives, if tii 6= tjj ,

j−1
X
fii − fjj fik tkj − tik fkj
fij = tij + , i < j.
tii − tjj tii − tjj
k=i+1

The right-hand side depends only on the elements to the left of fij and below it.
Hence this recurrence enables F to be computed either a superdiagonal at a time,
starting with the diagonal, or a column at a time, from the second column to the last,
moving up each column.
86 Techniques for General Functions

Algorithm 4.13 (Parlett recurrence). Given an upper triangular T ∈ Cn×n with

distinct diagonal elements and a function f defined on the spectrum of T , this algo-
rithm computes F = f (T ) using Parlett’s recurrence.

1 fii = f (tii ), i = 1: n
2 for j = 2: n
3 for i = j − 1: −1: 1
 X
j−1 
fii − fjj 
4 fij = tij + fik tkj − tik fkj (tii − tjj )
tii − tjj
k=i+1
5 end
6 end

Cost: 2n3/3 flops.

Parlett’s recurrence has a major drawback: it breaks down when tii = tjj for some
i 6= j, that is, when T has repeated eigenvalues. In this situation (4.18) provides no
information about fij .
A way around this difficulty is to employ a block form of the recurrence, also
described by Parlett [459, ]. Let T = (Tij ) be block upper triangular with square
diagonal blocks, possibly of different sizes. We will assume that T is also triangular,
though this is not necessary to derive the recurrence. Then F = (Fij ) has the same
block structure and equating (i, j) blocks in T F = F T leads to
j−1
X
Tii Fij − Fij Tjj = Fii Tij − Tij Fjj + (Fik Tkj − Tik Fkj ), i < j. (4.19)
k=i+1

This recurrence can be used to compute F a block superdiagonal at a time or a block

column at a time, provided we can evaluate the diagonal blocks Fii = f (Tii ) and solve
the Sylvester equations (4.19) for the Fij . The Sylvester equation (4.19) is nonsingular
if and only if Tii and Tjj have no eigenvalue in common (see Section B.14). Therefore
in order to use this block recurrence we need first to reorder the matrix T so that no
two diagonal blocks have an eigenvalue in common; here, reordering means applying a
unitary similarity transformation to permute the diagonal elements whilst preserving
triangularity. Reordering can be achieved by standard techniques and we return to
this topic in Section 9.3.

Algorithm 4.14 (block Parlett recurrence). Given a triangular matrix T = (Tij ) ∈

Cn×n partitioned in block form with no two diagonal blocks having an eigenvalue in
common, and a function f defined on the spectrum of T , this algorithm computes
F = f (T ) using the block form of Parlett’s recurrence.

1 Fii = f (Tii ), i = 1: n
2 for j = 2: n
3 for i = j − 1: −1: 1
4 Solve for Fij the Sylvester equation
Pj−1
Tii Fij − Fij Tjj = Fii Tij − Tij Fjj + k=i+1 (Fik Tkj − Tik Fkj )
5 end
6 end

Cost: Dependent on the block sizes and f , in general.

4.6 Schur Decomposition and Triangular Matrices 87

In Algorithm 4.14, computing Fii = f (Tii ) for a block of dimension greater than
1 is a nontrivial problem that we pursue in Section 9.1.
The block recurrence can be used in conjunction with the real Schur decomposition
of A ∈ Rn×n ,
QT AQ = T,
where Q ∈ Rn×n is orthogonal and T ∈ Rn×n is quasi upper triangular, that is,
block upper triangular with 1 × 1 or 2 × 2 diagonal blocks, with any 2 × 2 diagonal
blocks having complex conjugate eigenvalues. When f (A) is real, this enables it to
be computed entirely in real arithmetic.
We turn now to numerical considerations. In Listing 4.2 we give a function
funm simple that employs Algorithm 4.13 in conjunction with an initial Schur re-
duction to triangular form. The function is in principle applicable to any matrix with
distinct eigenvalues. It is very similar to the function funm in versions 6.5 (R14) and
earlier of MATLAB; version 7 of MATLAB introduced a new funm that implements
Algorithm 9.6 described in Section 9.4, which itself employs Algorithm 4.14.
Function funm simple often works well. For example the script M-file
format rat, A = gallery(’parter’,4), format short
evals = eig(A)’
X = real(funm_simple(A,@sqrt))
res = norm(A-X^2)
produces the output
A =
2 -2 -2/3 -2/5
2/3 2 -2 -2/3
2/5 2/3 2 -2
2/7 2/5 2/3 2
evals =
1.5859 - 2.0978i 1.5859 + 2.0978i 2.4141 - 0.7681i
2.4141 + 0.7681i
X =
1.4891 -0.6217 -0.3210 -0.2683
0.2531 1.5355 -0.5984 -0.3210
0.1252 0.2678 1.5355 -0.6217
0.0747 0.1252 0.2531 1.4891
res =
1.6214e-014
A major weakness of funm simple is demonstrated by the following experiment. Let
A be the 8 × 8 triangular matrix with aii ≡ 1 and aij ≡ −1 for j > i, which is MAT-
LAB’s gallery(’triw’,8). With f the exponential, Table 4.3 shows the normwise
relative errors for A and two small perturbations of A, one full and one triangular.
The condition number of f (A) (see Chapter 3) is about 2 in each case, so we would
expect to be able to compute f (A) accurately. For A itself, funm simple yields an
error of order 1, which is expected since it is unable to compute any of the super-
diagonal elements of f (A). For A plus the random full perturbation (which, being
full, undergoes the Schur reduction) the eigenvalues are distinct and at distance at
least 10−2 apart. But nevertheless, funm simple loses 6 significant digits of accuracy.
For the third matrix, in which the perturbation is triangular and the eigenvalues are
88 Techniques for General Functions

Listing 4.2: MATLAB function funm simple.

function F = funm_simple(A,fun)
%FUNM_SIMPLE Simplified Schur-Parlett method for function of a matrix.
% F = FUNM_SIMPLE(A,FUN) evaluates the function FUN at the
% square matrix A by the Schur-Parlett method using the scalar
% Parlett recurrence (and hence without blocking or reordering).
% This function is intended for matrices with distinct eigenvalues
% only and can be numerically unstable.
% FUNM should in general be used in preference.

n = length(A);

[Q,T] = schur(A,’complex’); % Complex Schur form.

F = diag(feval(fun,diag(T))); % Diagonal of F.

% Compute off-diagonal of F by scalar Parlett recurrence.

for j=2:n
for i = j-1:-1:1
s = T(i,j)*(F(i,i)-F(j,j));
if j-i >= 2
k = i+1:j-1;
s = s + F(i,k)*T(k,j) - T(i,k)*F(k,j);
end
d = T(i,i) - T(j,j);
if d ~= 0
F(i,j) = s/d;
end
end
end

F = Q*F*Q’;

about 10−9 apart, funm simple is spectacularly inaccurate: it produces a computed

answer with elements increasing away from the diagonal to a maximum of 1044 in the
top right corner. On the other hand, MATLAB’s funm, for which the errors are shown
in the second column, produces answers accurate to almost full machine precision.
The conclusion of this experiment is that it is not just repeated eigenvalues that
are bad for the scalar Parlett recurrence. Close eigenvalues can lead to a severe loss
of accuracy, and even when the eigenvalues are far apart the recurrence can produce
more inaccurate answers than expected. Indeed it is a well-known phenomenon in
numerical analysis that near confluence is much more dangerous and difficult to deal
with than exact confluence [326, ]. The block recurrence in Algorithm 4.14 is
therefore not a panacea: while the recurrence will always run to completion under
the distinct spectra condition, when two diagonal blocks have close spectra we can
expect difficulties.
We return to the Schur–Parlett approach in Chapter 9, where we develop a prac-
tical algorithm that performs well in finite precision arithmetic with only minimal
assumptions on the matrix and function.
4.7 Block Diagonalization 89

Table 4.3. Errors keA − Fbk/keA k for Fb from funm simple for A = gallery(’triw’,8).

funm simple funm

A 1.0e0 5.0e-16
A + rand(8)*1e-8 4.6e-10 3.4e-15
A + triu(rand(8))*1e-8 4.5e44 4.1e-16

4.7. Block Diagonalization

An alternative to diagonalization is block diagonalization, in which we compute A =
XDX −1 , where D is block diagonal. As we saw in Section 4.5 it is important for
numerical stability reasons that X be well conditioned. We assume that a block
diagonalization is computed in the usual way by first computing the Schur form and
then eliminating off-diagonal blocks by solving Sylvester equations [48, ], [224,
, Sec. 7.6.3], [375, ]. In order to guarantee a well conditioned X a bound
must be imposed on the condition of the individual transformations; this bound will
be a parameter in the algorithm.
Since f (A) = Xf (D)X −1 , the problem reduces to computing f (D) and hence to
computing f (Dii ) for each diagonal block Dii . The Dii are triangular but no particu-
lar eigenvalue distribution is guaranteed, because of the limitations on the condition
of the transformations; therefore f (Dii ) is still a nontrivial calculation. When A is
upper triangular, it is even possible that the block diagonalization procedure leaves A
unchanged. We therefore prefer the Schur–Parlett method to be described in Chap-
ter 9.
The Schur–Parlett method and the block diagonalization method are closely re-
lated. Both employ a Schur decomposition, both solve Sylvester equations, and both
must compute f (Tii ) for triangular blocks Tii . Parlett and Ng [462, , Sec. 5] show
that the two methods are mathematically equivalent, differing only in the order in
which two commuting Sylvester operators are applied. See Problem 4.3.

4.8. Interpolating Polynomial and Characteristic Polynomial

The definition f (A) = p(A), where A is the Hermite interpolating polynomial p in
(1.7) or a polynomial such as that in (1.11) that satisfies additional interpolation
conditions, suggests a numerical method. However, the method is impractical in
general—for two reasons. First, it requires O(n) matrix multiplications and hence
O(n4 ) flops to produce f (A), even given the polynomial p in monomial or divided
difference form, whereas most methods require only O(n3 ) flops. Second, the numer-
ical stability of this approach is highly dubious, and no error analysis is available to
justify its usage. For one specialized context in which a viable numerical method can
be built from this approach, see Section 10.4.1.
Frequently in the literature methods have been proposed for computing f (A) that
first compute the characteristic polynomial of A and then make use of this polynomial
in one way or another. We have eschewed all such methods in this book because the
characteristic polynomial cannot be reliably computed in floating point arithmetic.
Hence we do not believe that any method based on the characteristic polynomial is
a viable practical method, except possibly for very special A. We give an example to
illustrate the numerical dangers. Consider computation of the inverse of a nonsingular
90 Techniques for General Functions

Csanky
0
10

−5
10

−10
10
25 30 35 40 45 50 55 60

poly (MATLAB)
0
10

−5
10

−10
10
25 30 35 40 45 50 55 60

Figure 4.2. Relative errors for inversion of A = 3In , n = 25: 60, via the characteristic
polynomial.

A ∈ Cn×n with characteristic polynomial det(A − xI) = xn + c1 xn−1 + · · · + cn . By

the Cayley–Hamilton theorem,

 n−1
X 
−1 1 n−1 n−i−1
A =− A + ci A .
cn i=1

The coefficients ci can be obtained by computing the eigenvaluesQλi of A and then

n
using recurrences obtained from the factorization det(xI − A) = i=1 (x − λi ); this
method is implemented in the MATLAB function poly. Alternatively, the ci can
be obtained by computing the traces sk = trace(Ak ), k = 1: n and then solving the
Newton identities [301, , p. 37], [311, , p. 324]

   
  c1 −s1
1
   
 s1 2   c2   −s2 
   ..   .. 
 s2 s1 3  .   . 
  .  =  . . (4.20)
 s3 s2 s1 4  .   . 
 .. .. .. .. ..  .   . 
 . . . . .   ..   .. 
 .   . 
sn−1 ... s3 s2 s1 n cn −sn

The method based on the latter system was proposed for parallel computation by
Csanky [124, ] (it takes O(log2 n) time on O(n4 ) processors) but in fact goes
back at least to Bingham [68, ]. Figure 4.2 plots the ∞-norm relative errors
when A = 3In is inverted by these two approaches. All accuracy is lost by the time
n = 55, despite A being perfectly conditioned.
4.9 Matrix Iterations 91

4.9. Matrix Iterations

A number of matrix functions f (A) are amenable to computation by iteration:

Xk+1 = g(Xk ). (4.21)

For the iterations used in practice, X0 is not arbitrary but is a fixed function of
A—usually X0 = I or X0 = A. The iteration function g may or may not depend
on A.
Considerations of computational cost usually dictate that g is a polynomial or
rational function. Rational g require the solution of linear systems with multiple right-
hand sides, or even explicit matrix inversion. On modern computers with hierarchical
memories, matrix multiplication is usually much faster than solving a matrix equation
or inverting a matrix, so iterations that are multiplication-rich, which means having a
polynomial g, are preferred. The drawback is that such iterations usually have weaker
convergence properties than rational iterations.
A standard means of deriving matrix iterations is to apply Newton’s method to
an algebraic equation satisfied by f (A) and then to choose X0 so that the iteration
formula is simplified. In subsequent chapters we will study iterations that can be
derived in this way for computing the matrix sign function, the matrix square root,
matrix pth roots, and the polar decomposition.
It is important to keep in mind that results and intuition from scalar nonlinear
iterations do not necessarily generalize to the matrix case. For example, standard
convergence conditions expressed in terms of derivatives of g at a fixed point in the
scalar case do not directly translate into analogous conditions on the Frechét and
higher order derivatives in the matrix case.

4.9.1. Order of Convergence

One of the key properties of a matrix iteration (4.21) is its order (or rate) of conver-
gence. If Xk is a sequence of matrices converging to X∗ we say the convergence is of
order p if p is the largest number such that

kX∗ − Xk+1 k ≤ ckX∗ − Xk kp (4.22)

for all sufficiently large k, for some positive constant c. The iteration (4.21) is said to
have order p if the convergent sequences it generates have order p. Linear convergence
corresponds to p = 1 and quadratic convergence to p = 2. The convergence is called
superlinear if limk→∞ kX∗ − Xk+1 k/kX∗ − Xk k = 0.
The convergence of a sequence breaks into two parts: the initial phase in which the
error is reduced safely below 1, and the asymptotic phase in which (4.22) guarantees
convergence to zero. The order of convergence tells us about the rate of convergence
in the asymptotic phase, but it has nothing to say about how many iterations are
taken up by the first phase. This is one reason why a higher order of convergence
is not necessarily better. Other factors to consider when comparing iterations with
different orders of convergence are as follows.

• The overall efficiency of an iteration depends on the computational cost of eval-

uating Xk+1 as well as the convergence behaviour. Two successive steps of a
quadratically convergent iteration (such as Newton’s method) define a quar-
tically convergent iteration, so any other quartic iteration needs to have cost
92 Techniques for General Functions

per iteration no larger than twice that of the quadratic one if it is to be worth
considering.
• In practice we often need to scale an iteration, that is, introduce scaling param-
eters to reduce the length of the initial convergence phase. The higher the order
the less opportunity there is to scale (relative to the amount of computation).
• Numerical stability considerations may rule out certain iterations from consid-
eration for practical use.

4.9.2. Termination Criteria

An important issue is how to terminate a matrix iteration. The first question that
must be addressed is the aim of the test: to stop when Xk has relative error below a
tolerance or to stop when some suitable residual (such as Xk2 − A for the square root
function) is below the tolerance. The relevant aim will in general be function and
problem dependent.
Often, a stopping test is based on the relative difference
kXk+1 − Xk k
δk+1 =
kXk+1 k
between two successive iterates, which is regarded as an approximation to the relative
error kXk+1 − X∗ k/kX∗ k in Xk+1 , where X∗ = limk→∞ Xk . However, the relative
difference is actually approximating the relative error in Xk , kXk −X∗ k/kX∗ k. Indeed

Xk+1 − X∗ = (Xk+1 − Xk ) + (Xk − X∗ ),

and for sufficiently fast convergence kXk+1 − X∗ k ≪ kXk − X∗ k and hence the two
terms on the right-hand side are of roughly equal norm (and their largest elements
are of opposite signs). To obtain more insight, consider a quadratically convergent
method, for which
kXk+1 − X∗ k ≤ c kXk − X∗ k2 (4.23)
close to convergence, where c is a constant. Now

kXk − X∗ k ≤ kXk − Xk+1 k + kXk+1 − X∗ k

≤ kXk − Xk+1 k + ckXk − X∗ k2 .

“Solving” for kXk − X∗ k and substituting in (4.23) gives

 2
kXk+1 − Xk k
kXk+1 − X∗ k ≤ c
1 − ckXk − X∗ k
≤ 2c kXk+1 − Xk k2 , (4.24)

for small enough kXk − X∗ k, so the error in Xk+1 is bounded in terms of the square
of kXk+1 − Xk k. The conclusion is that a stopping test that accepts Xk+1 when δk+1
is of the order of the desired relative error may terminate one iteration too late.
The analysis above suggests an alternative stopping test for a quadratically con-
vergent iteration. From (4.24) we expect kXk+1 − X∗ k/kXk+1 k ≤ η if
 1/2
ηkXk+1 k
kXk+1 − Xk k ≤ . (4.25)
2c
4.9 Matrix Iterations 93

The value of c in (4.23) is usually known. The test (4.25) can be expected to terminate
one iteration earlier than the test δk+1 ≤ η, but it can potentially terminate too soon
if rounding errors vitiate (4.23) or if the test is satisfied before the iteration enters
the regime where (4.23) is valid.
A stopping test based on the distance between successive iterates is dangerous be-
cause in floating point arithmetic there is typically no guarantee that δk (or indeed the
relative error) will reach a given tolerance. One solution is to stop when a preassigned
number of iterations have passed. An alternative, suitable for any quadratically or
higher order convergent iteration, is to terminate once the relative change has not de-
creased by a factor at least 2 since the previous iteration. A reasonable convergence
test is therefore to terminate the iteration at Xk+1 when
δk+1 ≤ η or δk+1 ≥ δk /2, (4.26)
where η is the desired relative error. This test can of course be combined with others,
such as (4.25).

4.9.3. Convergence
Convergence analysis for the matrix iterations in this book will be done in two general
ways. The first is to work entirely at the matrix level. This is our preference, when
it is possible, because it tends to give the most insight into the behaviour of the
iteration. The second approach breaks into three parts:
• show, if possible, that the iteration converges for an arbitrary matrix if and only
if it converges when applied to the Jordan blocks of the matrix (why this is not
always possible is explained in the comments after Theorem 6.9);
• prove convergence of the scalar iteration when applied to the eigenvalues (which
implies convergence of the diagonal of the Jordan blocks);
• prove convergence of the off-diagonals of the Jordan blocks.
The last step can be done in some generality, as our next theorem shows.
Consider, for example, the iteration
1
Xk+1 = (Xk + Xk−1 A), X0 = A ∈ Cn×n , (4.27)
2
which is an analogue for matrices of Heron’s method for the square root of a scalar.
As is well known, for n = 1 the iteration converges to the principal square root of
a if a does not lie on R− . If A has the Jordan canonical form A = ZJZ −1 then
it is easy to see that the iterates from (4.27) are given by Xk = ZJk Z −1 , where
Jk+1 = 21 (Jk + Jk−1 J), J0 = J. Hence Jk has the same block diagonal structure as
J, and convergence reduces to the case where J is a single Jordan block. The next
result allows us to deduce convergence of the Xk when A has no eigenvalues on R− .
This result is quite general and will be of use for other iterations in later chapters.

Theorem 4.15. Let g(x, t) be a rational function of both of its arguments. Let the
scalar sequence generated by xk+1 = g(xk , λ), x0 = φ0 (λ) converge to x∗ = f (λ),
∂g
where φ0 is a rational function and λ ∈ C, and assume that | ∂x (x∗ , λ)| < 1 (i.e.,
x∗ is an attracting fixed point of the iteration). Then the matrix sequence generated
by Xk+1 = g(Xk , J(λ)), X0 = φ0 (J(λ)), where J(λ) ∈ Cm×m is a Jordan block,
converges to a matrix X∗ with (X∗ )ii ≡ f (λ).
94 Techniques for General Functions

Proof. Let φk (t) = g(φk−1 (t), t), k ≥ 1, so that φk is a rational function of t and
xk = φk (λ). Upper triangular Toeplitz structure is preserved under inversion and
multiplication, and hence under any rational function. Hence
 
a1 (k) a2 (k) . . . am (k)
 . .. 
 a1 (k) . . . 
Xk = g(Xk−1 , J(λ)) = φk (J(λ)) =  
.

 ..
. a2 (k) 
a1 (k)

Clearly, the diagonal elements of Xk satisfy a1 (k) = φk (λ) = xk and hence tend to
x∗ . It remains to show that the elements in the strictly upper triangular part of Xk
converge.
From (1.4) we know that

1 dj−1 

aj (k) = φ k (t) .
(j − 1)! dtj−1 λ

For j = 2, using the chain rule,

dφk  d

a2 (k) = (t) = g φk−1 (t), t 
dt λ dt λ
 
∂g
′ ∂g

= φk−1 (t), t φk−1 (t) + φk−1 (t), t 
∂x ∂t 
λ
∂g ∂g
= (xk−1 , λ)φ′k−1 (λ) + (xk−1 , λ)
∂x ∂t
∂g ∂g
= (xk−1 , λ)a2 (k − 1) + (xk−1 , λ).
∂x ∂t
∂g
Since | ∂x (x∗ , λ)| < 1, by Problem 4.7 it follows that

∂g ∂g
a2 (k) → (x∗ , λ)/(1 − (x∗ , λ)) as k → ∞. (4.28)
∂t ∂x
As an induction hypothesis suppose that ai (k) has a limit as k → ∞ for i = 2: j −1.
Then by the chain rule

1 dj−1  1 dj−1


aj (k) = φ k (t) = g φ k−1 (t), t 
(j − 1)! dtj−1 λ (j − 1)! dtj−1 λ
 j−2   
1 d ∂g
′ ∂g

= φk−1 (t), t φk−1 (t) + φk−1 (t), t 
(j − 1)! dtj−2 ∂x ∂t λ
 j−1
 
1 ∂g
d (k)

= φk−1 (t), t j−1 φk−1 (t) + τj φk−1 (t), t 
(j − 1)! ∂x dt λ
∂g 1 (k)
= (xk−1 , λ)aj (k − 1) + τ (xk−1 , λ), (4.29)
∂x (j − 1)! j
(k)
where τj (xk−1 , λ) is a sum of terms comprising products of one or more elements
ai (k − 1), i = 1: j − 1, and derivatives of g evaluated at (xk−1 , λ), the number of
terms in the sum depending on j but not k. By the inductive hypothesis, and since
4.9 Matrix Iterations 95

(k)
xk−1 → x∗ as k → ∞, τj (xk−1 , λ) has a limit as k → ∞. Hence by Problem 4.7,
aj (k) has a limit as k → ∞, as required.
The notation “g(xk , λ)” in Theorem 4.15 may seem unnecessarily complicated. It
is needed in order for the theorem to allow either or both of the possibilities that
(a) the iteration function g depends on A,
(b) the starting matrix X0 depends on A.
For the Newton iteration (4.27), both g and X0 depend on A, while for the sign
iteration in (4.30) below only X0 depends on A.
Notice that Theorem 4.15 does not specify the off-diagonal elements of the limit
matrix X∗ . These can usually be deduced from the equation X∗ = g(X∗ , A) together
with knowledge of the eigenvalues of X∗ . For example, for the Newton square root
iteration we know that X∗ = 12 (X∗ + X∗−1 A), or X∗2 = A, and, from knowledge of
the scalar iteration, that X∗ has spectrum in the open right half-plane provided that
A has no eigenvalues on R− . It follows that X∗ is the principal square root (see
Theorem 1.29). A variant of Theorem 4.15 exists that assumes that f is analytic and
guarantees convergence to X∗ = f (J(λ)); see Iannazzo [307, ].
In the special case where the iteration function does not depend on the parameter
λ, the limit matrix must be diagonal, even though the starting matrix is a Jordan
block.

Corollary 4.16 (Iannazzo). Under the conditions of Theorem 4.15, if the iteration
function g does not depend on λ then the limit X∗ of the matrix iteration is diagonal.

Proof. In the notation of the proof of Theorem 4.15 we need to show that aj (k) →
0 as k → ∞ for j = 2: m. By assumption, ∂g/∂t ≡ 0 and so a2 (k) → 0 as k → ∞
by (4.28). An inductive proof using (4.29) then shows that aj (k) → 0 as k → ∞ for
j = 3: m.

4.9.4. Numerical Stability

If X0 is a function of A then the iterates from (4.21) are all functions of A and
hence commute with A, and indeed commutativity properties are frequently used
when deriving an iteration and in proving convergence. In finite precision arithmetic,
rounding errors cause a loss of commutativity that can manifest itself as numerical
instability. Looked at another way, iterations for matrix functions are often not self-
correcting, so arbitrary errors can potentially propagate unstably. We will therefore
pay careful attention to stability when deriving and analyzing matrix iterations.
The stability properties of superficially similar matrix iterations can be very dif-
ferent, as we now illustrate. Consider the Newton square root iteration (4.27). As
noted in the previous section, for any A with no eigenvalues on R− , Xk converges
quadratically to A1/2 (see also Theorem 6.9). Table 4.4 shows the relative errors and
(1, 1) elements of the iterates when the iteration is carried out in IEEE single precision
arithmetic with A the Wilson matrix
 
10 7 8 7
 7 5 6 5 
A= ,
8 6 10 9
7 5 9 10
96 Techniques for General Functions

Table 4.4. Square root iteration (4.27) and sign iteration (4.30) applied to Wilson matrix in
single precision arithmetic. (A1/2 )11 = 2.389 to four significant figures, while sign(A) = I.

Square root Sign

kA1/2 −Xk k2
(Xk )11 kI − Yk k2 (Yk )11
kA1/2 k2
1 1.84e0 5.50e0 4.83e1 1.75e1
2 5.97e-1 3.36e0 2.36e1 8.90e0
3 1.12e-1 2.57e0 1.13e1 4.67e0
4 5.61e-3 2.40e0 5.21e0 2.61e0
5 4.57e-3 2.40e0 2.19e0 1.63e0
6 1.22e-1 2.21e0 7.50e-1 1.20e0
7 3.26e0 7.20e0 1.61e-1 1.04e0
8 8.74e1 −1.26e2 1.11e-2 1.00e0
9 2.33e3 3.41e3 6.12e-5 1.00e0
10 1.91e4 2.79e4 9.78e-10 1.00e0
11 1.97e4 −2.87e4 0 1.00e0
12 8.45e3 −1.23e4 0 1.00e0
13 1.08e3 −1.58e3 0 1.00e0
14 8.37e3 1.22e4 0 1.00e0

which is symmetric positive definite and moderately ill conditioned, with κ2 (A) ≈
2984. For such a benign matrix, we might expect no difficulty in computing A1/2 .
The computed Xk behave as they would in exact arithmetic up to around iteration
4, but thereafter the iterates rapidly diverge. It turns out that iteration (4.27) is
unstable unless A has very closely clustered eigenvalues. The instability is related
to the fact that (4.27) fails to converge for some matrices X0 in a neighbourhood of
A1/2 ; see Section 6.4 for the relevant analysis.
Let us now modify the square root iteration by replacing A in the iteration formula
by I:
1
Yk+1 = (Yk + Yk−1 ), Y0 = A. (4.30)
2
For any A for which sign(A) is defined, Yk converges quadratically to sign(A). (Note,
for consistency with (4.27), that sign(A) is one of the square roots of I.) Iteration
(4.30) is stable for all A (see Theorem 5.13), and Table 4.4 confirms the stability for
the Wilson matrix.
In order to understand fully the behaviour of a matrix iteration in finite precision
arithmetic we would like to

• bound or estimate the minimum relative error kX∗ − X bk k/kX∗ k over the com-
puted iterates Xbk , and determine whether this error is consistent with the con-
ditioning of the problem,

• determine how the iteration propagates errors introduced by roundoff and, in

particular, whether the error growth is bounded or unbounded.

The first task is very difficult for most iterations, though the conditioning can
be determined, as shown in the previous chapter. To make progress on this task we
examine what happens when X0 is very close to X∗ , which leads to the notion of
limiting accuracy. The second task, which is related to the first, is also difficult when
4.9 Matrix Iterations 97

considered over the whole iteration, but near to convergence the propagation of errors
is more amenable to analysis.
We will use the following definition of stability of an iteration. We write Li (X) to
denote the ith power of the Fréchet derivative

L at X, defined as i-fold composition;
thus L3 (X, E) ≡ L X, L(X, L(X, E)) .

Definition 4.17 (stability). Consider an iteration Xk+1 = g(Xk ) with a fixed point
X. Assume that g is Fréchet differentiable at X. The iteration is stable in a neigh-
borhood of X if the Fréchet derivative Lg (X) has bounded powers, that is, there exists
a constant c such that kLig (X)k ≤ c for all i > 0.

Note that stability concerns behaviour close to convergence and so is an asymptotic

property. We will reserve the term “numerical stability” for global properties of the
iteration.
To see the relevance of the definition, let X0 = X + E0 , with arbitrary E0 of
sufficiently small norm, and let Ek := Xk − X. Then, by the definition of Fréchet
derivative,

Xk = g(Xk−1 ) = g(X + Ek−1 ) = g(X) + Lg (X, Ek−1 ) + o(kEk−1 k).

So, since g(X) = X,

Ek = Lg (X, Ek−1 ) + o(kEk−1 k). (4.31)
This relation can be recurred to give
k−1
X

Ek = Lkg (X, E0 ) + Lig X, o(kEk−1−i k) ,
i=0

so for a stable iteration,

k−1
X
kEk k ≤ ckE0 k + c o(kEk−1−i k) ≤ ckE0 k + kc · o(kE0 k).
i=0

The definition therefore ensures that in a stable iteration sufficiently small errors
introduced near a fixed point have a bounded effect, to first order, on succeeding
iterates.
To test for stability we need to find the Fréchet derivative of the iteration function
g and then determine the behaviour of its powers, possibly by working with the Kro-
necker matrix form of the Fréchet derivative. Useful here is the standard result that
a linear operator on Cn×n (or its equivalent n2 × n2 matrix) is power bounded if its
spectral radius is less than 1 (see Problem 4.6) and not power bounded if its spectral
radius exceeds 1. The latter property can be seen from the fact that kLkg (X)k ≥ |λ|k
for any eigenvalue λ of Lg (X) (cf. (3.19)). We will sometimes find the pleasing situa-
tion that Lg (X) is idempotent, that is, L2g (X, E) = Lg (X, Lg (X, E)) = Lg (X, E), in
which case power boundedness is immediate.
Note that any superlinearly convergent scalar iteration xk+1 = g(xk ) has zero
derivative g ′ at a fixed point, so for such scalar iterations convergence implies stability.
For matrix iterations, however, the Fréchet derivative is not generally zero at a fixed
point. For example, for the sign iteration (4.30) it is easy to see that at a fixed point Y
we have Lg (Y, E) = 12 (E − Y EY ) (see Theorem 5.13). Although Y 2 = I, Y E 6= EY
98 Techniques for General Functions

in general and so Lg (Y, E) 6= 0. This emphasizes that stability is a more subtle and
interesting issue for matrices than in the scalar case.
An important special case is when the underlying function f has the property that
f (f (A)) = f (A) for all A for which f is defined, that is, f is idempotent. For such
an f we can show that Lf (X) is idempotent at X = f (X).

Theorem 4.18. Let f be an idempotent function that is Fréchet differentiable at

X = f (X). Then Lf (X) is idempotent.
Proof. Let h(t) = f (f (t)). By the chain rule (Theorem 3.4),

Lh (X, E) = Lf (f (X), Lf (X, E)) = Lf (X, Lf (X, E)).

But because f is idempotent, h(A) ≡ f (A) and so Lh (X, E) = Lf (X, E). Hence
Lf (X, Lf (X, E)) = Lf (X, E), which shows that Lf (X) is idempotent.
Stability is determined by the Fréchet derivative of the iteration function, not that
of f . However, these two derivatives are one and the same if we add to the condition
of Theorem 4.18 the conditions that in the iteration Xk+1 = g(Xk ) the function g is
independent of the starting matrix X0 and that the iteration is superlinearly conver-
gent when started sufficiently close to a fixed point. These conditions are satisfied for
the matrix sign function and the unitary polar factor, and the corresponding itera-
tions of interest, but not for matrix roots (since (A1/p )1/p 6= A1/p and the iteration
function g invariably depends on X0 ).

Theorem 4.19. Let f be an idempotent function that is Fréchet differentiable at X ≡

f (X) with Fréchet derivative Lf (X). Let Xk+1 = g(Xk ) be superlinearly convergent
to f (X0 ) for all X0 sufficiently close to X and assume that g is independent of X0 .
Then the Fréchet derivative of g at X is Lg (X) = Lf (X).
Proof. For sufficiently small E, f (X + E) is defined and

f (X + E) = f (X) + Lf (X, E) + o(kEk) = X + Lf (X, E) + o(kEk).

Hence f (X + E) − (X + E) = Lf (X, E) − E + o(kEk) = O(kEk), since Lf (X, E) is

linear in E. Since the iteration is (locally) superlinearly convergent,

kf (X + E) − g(X + E)k = o(kf (X + E) − (X + E)k) = o(O(kE)k) = o(kEk).

Hence

g(X + E) − g(X) = g(X + E) − X

= f (X + E) − X + o(kEk))
= Lf (X, E) + o(kEk),

which shows that Lg (X, E) = Lf (X, E).

The significance of Theorems 4.18 and 4.19 is that when they are applicable they
tell us that all superlinearly convergent iterations are stable, without the need to
compute Fréchet derivatives and test their power boundedness.
For the iterations described earlier in this section:

• f (A) = A1/2 is not idempotent and the iteration function in (4.27) depends on
X0 = A, so the theorems are not applicable.
4.10 Preprocessing 99

• f (A) = sign(A) is idempotent and Fréchet differentiable and the iteration (4.30)
is quadratically convergent with iteration function independent of Y0 = A, so
the theorems apply and the iteration is therefore stable. See Section 5.7 for
details.

The Fréchet derivative also allows us to estimate the limiting accuracy of an iter-
ation.

Definition 4.20 (limiting accuracy). For an iteration Xk+1 = g(Xk ) with a fixed
point X the (relative) limiting accuracy is ukLg (X)k.
To interpret the definition, consider one iteration applied to the rounded exact
solution, X0 = X + E0 , where kE0 k <
∼ ukXk. From (4.31) we have
kE1 k < <
∼ kLg (X, E0 )k ≤ kLg (X)k kE0 k ∼ ukLg (X)k kXk,
and so the limiting accuracy is a bound for the relative error kX − X1 k/kXk. We can
therefore think of the limiting accuracy as the smallest error we can reasonably expect
to achieve in floating point arithmetic once—and indeed if —the iteration enters an
O(u) neighbourhood of a fixed point. Limiting accuracy is once again an asymptotic
property.
While stability corresponds to the boundedness of the powers of Lg (X), which
depends only on the eigenvalues of Lg (X), the limiting accuracy depends on the
norm of Lg (X), and so two stable iterations can have quite different limiting accuracy.
However, neither of these two notions necessarily gives us a reliable estimate of the
accuracy of a computed solution or the size of its residual. Clearly, an unstable
iteration may never achieve its limiting accuracy, because instability may prevent it
reaching the region of uncertainty around the solution whose size limiting accuracy
measures.
Finally, it is important to note that the Fréchet derivative analysis treats the
propagation of errors by the exact iteration. In practice, rounding errors are incurred
during the evaluation of the iteration formula, and these represent another source of
error that is dependent on how the formula is evaluated. For example, as noted in
Section 4.4.3, rational functions can be evaluated in several ways and these potentially
have quite different numerical stability properties.
Analysis based on the Fréchet derivative at the solution will prove to be informa-
tive for many of the iterations considered in this book, but because of the limitations
explained above this analysis cannot always provide a complete picture of the numer-
ical stability of a matrix iteration.

4.10. Preprocessing
In an attempt to improve the accuracy of an f (A) algorithm we can preprocess the
data. Two available techniques are argument reduction (or translation) and balancing.
Both aim to reduce the norm of the matrix, which is important when Taylor or Padé
approximants—most accurate near the origin—are to be applied.
Argument reduction varies slightly depending on the function. For the matrix
exponential, eA−µI = e−µ eA , so any multiple of I can be subtracted from A. For
trigonometric functions such as the cosine,

cos(A − πjI) = (−1)j cos(A), j ∈ Z, (4.32)

100 Techniques for General Functions

so we can subtract integer multiples of π. Norm-minimizing real shifts are known

for three particular norms. (The optimal 1-norm shift is obtained from the optimal
∞-norm shift using kAk1 = kA∗ k∞ .)

Theorem 4.21 (norm-minimizing shifts). Let A ∈ Cn×n .

(a) minµ∈R kA − µIkF is attained for µ = n−1 trace(A).
(b) If aii ∈ R for all i then

1

min kA − µIk∞ = max(aii + ri ) + max(−aii + ri ) , (4.33)
µ∈R 2 i i
P
where ri = j6=i |aij |, and the optimal µ is

1

µ= max(aii + ri ) − max(−aii + ri ) .
2 i i

Proof. (a) We have kA−µIkF = k vec(A)−vec(I)µk2 , so the problem is essentially

a 1-variable linear least squares problem. The normal equations are vec(I)∗ vec(I)µ =
vec(I)∗ vec(A), or nµ = trace(A), as required.
(b) We have
X
kAk∞ = max |aij | = max max(aii + ri , −aii + ri )
i i
j


= max max(aii + ri ), max(−aii + ri ) .
i i

The inner max terms in the last expression are the rightmost point and the negative
of the leftmost point in the union of the Gershgorin discs for A. It is clear that the
optimal shift µ must make these extremal points equidistant from the origin; hence it
must satisfy maxi (aii − µ + ri ) = maxi (−aii + µ + ri ), that is, µ = 21 (maxi (aii + ri ) −
maxi (−aii + ri )). The formula (4.33) follows on substitution of the optimal µ.
In the computation of scalar elementary functions it is well known that special
techniques must be used to avoid severe loss of accuracy in argument reduction for
large arguments [441, , Chap. 8]. The standard techniques are not directly appli-
cable to the matrix case, so we must recognize that argument reduction is potentially
a significant source of error.
Balancing is a heuristic that attempts to equalize the norms of the ith row and ith
column, for each i, by a diagonal similarity transformation. It is known that balancing
in the 2-norm is equivalent to minimizing kD−1 ADkF over all nonsingular diagonal
D [454, ]. Nowadays, “balancing” is synonymous with the balancing algorithms
in LAPACK [12] and MATLAB [414], which compute B = D−1 AD, where D is a
permuted diagonal matrix with diagonal elements powers of the machine base chosen
so that the 1-norms of the ith row and ith column of B are of similar magnitude for
all i. Balancing is an O(n2 ) calculation that can be performed without roundoff. It
is not guaranteed to reduce the norm, so it is prudent to replace A by the balanced
B only if kBk < kAk.
Balancing can be combined with argument reduction. Since trace(A) = trace(D−1 AD)
and the balancing transformation is independent of the diagonal elements of the ma-
trix, argument reduction in the Frobenius norm yields the same shift before balancing
as after. Therefore it makes no difference in which order these two operations are done.
4.10 Preprocessing 101

In some special cases we can say more about argument reduction and balancing.
n×n
Pn that an intensity matrix is a matrix Q ∈ R
Recall from Section 2.3 such that
qij ≥ 0 for i 6= j and j=1 qij = 0, i = 1: n.

Corollary 4.22 (Melloy and Bennett). Let Q ∈ Rn×n be an intensity matrix. Then
min kQ + µIk∞ = max |qii | = kQk∞ /2.
µ∈R i

The minimum is attained for µ = maxi |qii |, for which Q + µI is nonnegative with
row sums all equal to µ.
Proof. In the notation of Theorem 4.21 (b), we have qii + ri = 0, so −qii + ri =
2|qii |, and the result then follows from the theorem.
Corollary 4.22 shows that for an intensity matrix the optimal shift reduces the
∞-norm by a factor 2. We now show that the shifted matrix is already of minimal
∞-norm under diagonal similarities. We need the following more general result.

Theorem 4.23 (Ström). For A ∈ Cn×n ,

inf{ kD−1 ADk∞ : D = diag(di ) is nonsingular } = ρ(|A|). (4.34)
If |A|x = ρ(|A|)x with x > 0 then D = diag(x) achieves the lower bound.
Proof. For any A ∈ Cn×n and nonsingular diagonal D we have
kD−1 ADk∞ = k |D−1 AD| k∞ = kD−1 |A|Dk∞ ≥ ρ(D−1 |A|D) = ρ(|A|). (4.35)
If x satisfies the stated conditions then with D = diag(x) we have
|D−1 AD|e = D−1 |A|De = D−1 |A|x = ρ(|A|)D−1 x = ρ(|A|)e.
Taking the ∞-norm gives kD−1 ADk∞ = ρ(|A|), as required.
If A is irreducible then from the Perron–Frobenius theory (Theorem B.6) we know
that a positive eigenvector x of |A| exists corresponding to an eigenvalue ρ(|A|), and
so the lower bound (4.35) is attained. If A is reducible then for any ǫ > 0 we can
choose B so that kBk∞ < ǫ and |A| + |B| is irreducible. Then
inf kD−1 ADk∞ = inf kD−1 |A|Dk∞ ≤ inf kD−1 (|A| + |B|)Dk∞
D D D
= ρ(|A| + |B|)
≤ ρ(|A|) + ρ(|B|) ≤ ρ(|A|) + kBk∞
< ρ(|A|) + ǫ,
where we have used (B.8) and (B.13), and (4.34) follows.

Corollary 4.24. Let A ∈ Rn×n be a scalar multiple of a stochastic matrix. Then

diagonal similarities cannot reduce the ∞-norm of A.
Proof. The result follows from Theorem 4.23 because Ae = µe for some µ with
|µ| = kAk∞ = ρ(A) = ρ(|A|), so kAk∞ is already minimal.
Corollary 4.24 shows that any nonnegative matrix with equal row sums has min-
imal ∞-norm under diagonal similarities. In particular, this is true of the optimally
shifted intensity matrix. To summarize: to preprocess an intensity matrix it suffices
to shift it as in Corollary 4.22.
102 Techniques for General Functions

4.11. Bounds for kf (A)k

For both theoretical and practical purposes it is useful to be able to bound the norm
of f (A). We give a variety of different bounds in this section, beginning with one
based on the Jordan canonical form.

Theorem 4.25. Let A ∈ Cn×n have the Jordan canonical form (1.2), with distinct
eigenvalues λ1 , . . . , λs . Then
 (j) 
 f (λi ) 
kf (A)kp ≤ nmax κp (Z) max   , p = 1: ∞, (4.36)
i=1:s
j=0:n −1
j! 
i

where nmax = maxi=1:s ni .

Proof. The proof is straightforward from Definition 1.2 and on using kBkp ≤
1/p 1−1/p
kBk1 kBk∞ [276, , Sec. 6.3].
If A = Z diag(λi )Z −1 is diagonalizable then (4.36) simplifies to
kf (A)kp ≤ κp (Z)f (ρ(A)). (4.37)
In both (4.36) and (4.37), κp (Z) should be interpreted as the minimum over all Jordan
canonical forms. If A is normal then Z can be taken to be unitary and these bounds
are then an equality for p = 2: kf (A)k2 = f (ρ(A)). Thus only the nonnormal case is
interesting.
A pseudospectral analogue of (4.37) is as follows. Recall that the ǫ-pseudospectrum
is defined by (2.30).

Theorem 4.26 (Trefethen and Embree). Let A ∈ Cn×n and ǫ > 0. Let f be analytic
on and inside a closed contour Γǫ that encloses Λǫ (A). Then
Lǫ
kf (A)k2 ≤ max |f (z)|, (4.38)
2πǫ z∈Γǫ
where Lǫ is the arc length of Γǫ . In particular,
ρǫ
kf (A)k2 ≤ max |f (ρǫ eiθ )|, (4.39)
ǫ θ∈[0,2π]
where ρǫ = ρǫ (A) = max{ |z| : z ∈ Λǫ (A) } is the pseudospectral radius.
Proof. To prove (4.38) let Γ := Γǫ in the Cauchy integral (1.12), take norms,
and use the fact that k(zI − A)−1 k ≤ ǫ−1 on and outside Λǫ (A). The bound (4.39) is
obtained from (4.38) by taking Γǫ to be a circle with centre 0 and radius ρǫ .
The next result does not involve a similarity taking A to Jordan form but it needs
knowledge of the derivatives of f on a larger region than just the spectrum. Denote
by conv(S) the convex hull of the set S.

Theorem 4.27 (Young). Let A ∈ Cn×n and let f be analytic in a convex open set
containing Λ(A). Then
 !2 1/2
X  n  kAkiF supz∈conv(Λ(A)) |f (i) (z)|
n−1
kf (A)kF ≤   . (4.40)
i=0
i+1 i!
4.11 Bounds for kf (A)k 103

A bound can also be given in terms of the Schur decomposition.

Theorem 4.28 (Golub and Van Loan). Let Q∗ AQ = T = diag(λi ) + N be a Schur

decomposition of A ∈ Cn×n , where N is strictly upper triangular. If f (z) is analytic
on a closed convex set Ω containing Λ(A) then
n−1
X |N |i ωi

kf (A)kF ≤ ωi ≤ max k(I − |N |)−1 kF , (4.41)
i! i=0
0≤i≤n−1 i!
F

where ωi = supz∈Ω |f (i) (z)|.

(r)
Proof. Let Sij denote the set of all strictly increasing sequences of r + 1 integers
that start at i and end at j, and note that with Sij defined as in Theorem 4.11,
j−i
[ (r)
Sij = Sij .
r=1

From Theorem 4.11, F = f (T ) is given by, for j ≥ i,

j−i
X X
fij = ts0 ,s1 ts1 ,s2 . . . tsr−1 ,sr f [λs0 , . . . , λsr ].
r=1 (s ,...,s )∈S (r)
0 r ij

Using (B.28) we have

j−i
X X
ωr
|fij | ≤ |ts0 ,s1 ||ts1 ,s2 | . . . |tsr−1 ,sr |.
r=1
r! (r)
(s0 ,...,sr )∈Sij

(r)
Now it can be shown that |N |r = (nij ) satisfies
(
0, j < i + r,
(r)
nij = P (r) |ts0 ,s1 ||ts1 ,s2 | . . . |tsr−1 ,sr |, j ≥ i + r.
(s0 ,...,sr )∈S ij

Hence for j > i,

j−i
X n−1
X
ωr ωr
|fij | ≤ (|N |r )ij = (|N |r )ij .
r=1
r! r=1
r!
Extending the summation to r = 0 correctly bounds |fii | ≤ ω0 . Hence |f (T )| ≤
Pn−1 r
r=0 ωr |N | /r!. This yields the first bound in (4.41) on using the unitary invariance
of the Frobenius norm. The second inequality follows from I + |N | + · · · + |N |n−1 =
(I − |N |)−1 .
A weaker variant of Theorem 4.28 is given by Descloux [148, ]: under the
same conditions,
n−1
X ωi
kf (A)k2 ≤ kN kiF . (4.42)
i=0
i!
The quantity ∆(A) := kN kF is the departure from normality,
Pn introduced by
Henrici [257, ]. It can be expressed as ∆(A)2 = kAk2F − i=1 |λi (A)|2 and is zero
when A is normal. Another bound involving the departure from normality is one of
Gil that is in a similar vein to the bound of Theorem 4.27.
104 Techniques for General Functions

Theorem 4.29 (Gil). Let A ∈ Cn×n and let f be analytic in a convex open set
containing Λ(A). Then
n−1
X
kf (A)k2 ≤ sup |f (i) (z)| ∆(A)i (i!)−3/2 . (4.43)
i=0 z∈conv(Λ(A))

4.12. Notes and References

For more on the theory of Padé approximation see Baker [38, ], Baker and Graves-
Morris [39, ], Gragg [227, ], or Sidi [521, , Chap. 17]. Some authors use
an alternative definition of the [k/m] Padé approximant of f in which f (x)−rkm (x) is
required not necessarily to be of order xk+m+1 but rather to be of the highest possible
order; with this definition the approximant always exists but the order of the error is
not immediately apparent.
Theorem 4.5 (a less general version of which is stated in [278, ]) can be proved
using techniques from [276, ].
Theorem 4.7 is due to Hensel [258, ]. Weyr [614, ] had previously proved
part (a) of the theorem.
Calvetti, Gallopoulos, and Reichel [90, ] give a detailed discussion and exten-
sive bibliography on matrix partial fraction expansions and they investigate “incom-
plete” expansions in an aim to obtain improved numerical stability properties.
Theorem 4.12 is given by Kenney and Laub [348, ].
Parlett’s recurrence is derived independently by Filipponi [186, ].
In his thesis, Kågström [323, ] handles confluency in Parlett’s recurrence by a
different means than by using the block form. He uses the scalar recurrence, and if tii
is equal or very close to tjj he invokes the formula for fij in Theorem 4.11, replacing
the divided difference by a derivative.
Theorem 4.15 generalizes, and is modeled on, a result of Kenney and Laub [343,
, Lem. 5.1] that applies to rational iterations for the matrix sign function. The
theorem can also be proved using complex analysis, as shown by Iannazzo [307, ],
who pointed out Corollary 4.16. An early use of the Jordan form in convergence
analysis is in Laasonen’s analysis of the Newton square root iteration [366, ].
The framework given here for stability analysis, based on Fréchet derivatives,
was initiated by Cheng, Higham, Kenney, and Laub [108, ] and developed by
Higham, Mackey, Mackey, and Tisseur [283, ]. Our treatment here expands
significantly on that in [283, ]. Theorem 4.19 is new. In a different style of
analysis used by Higham [267, ] and employed in subsequent papers by various
authors, perturbation expansions are developed for the iterations, from which scalar
error amplification factors are identified under the assumption that the matrix A is
diagonalizable. The approach used here leads to shorter and more insightful analysis,
especially when the Fréchet derivative is idempotent at the solution.
A more general treatment of matrix argument reduction for periodic functions is
given by Ng [448, ], in which different eigenvalues can undergo different shifts.
His algorithm for shifting makes use of a Schur decomposition, and he applies it to
the matrix exponential, for which the period is 2πi.
Ward’s implementation of the scaling and squaring algorithm for computing the
matrix exponential [606, ] uses both argument reduction and balancing.
Part (b) of Theorem 4.21 is stated without proof by Arioli, Codenotti, and Fassino
[17, ] for an unspecified class of A.
Problems 105

Corollary 4.22 is due to Melloy and Bennett [422, ]. Further analysis of
balancing can be found in Ström [545, ] and Fenner and Loizou [183, ].
Theorem 4.23 is from Ström [545, ]. Corollary 4.24 generalizes a result of Melloy
and Bennett [422, ] stated for a shifted intensity matrix.
Theorem 4.27 is from Young [620, ]. Theorem 4.26 is from Trefethen and Em-
bree [573, ]. Theorem 4.28 is from Golub and Van Loan [224, , Thm. 11.2.2]
and it appeared first in Van Loan [592, , Thm. 4]. Theorem 4.29 is from Gil [214,
]. For some other approaches to bounding kf (A)k see Crouzeix [123, ] and
Greenbaum [230, ].

Problems
Since the purpose of mathematics is to solve problems,
it is impossible to judge one’s progress without
breaking a lance on a few problems from stage to stage.
— RICHARD BELLMAN, Introduction to Matrix Analysis (1970)

 
4.1. LetPA ∈ Cm×n , B ∈ Cn×m , and C = B0 A0 ∈ C(m+n)×(m+n) . Show that if
∞
f (z) = i=0 ai z i then (within the radius of convergence)
" P∞ i
P∞ i
#
i=0 a2i (AB) A i=0 a2i+1 (BA)
f (C) = P∞ P∞ .
i i
B i=0 a2i+1 (AB) i=0 a2i (BA)

Deduce the form of eC . Hence obtain the solution (2.8) to (2.7).

4.2. Show that if a [k/m] Padé approximant to f (x) exists then it is unique.
4.3. Let  
T11 T12
T = ∈ Cn×n
0 T22
be block upper triangular with Λ(T11 )∩Λ(T22 ) = ∅. Let f be defined on the spectrum
of T and let  
f (T11 ) F12
F = f (T ) = .
0 f (T22 )
The block Parlett recurrence yields F12 as the solution of the Sylvester equation

T11 F12 − F12 T22 = f (T11 )T12 − T12 f (T22 ). (4.44)

But F12 can also be obtained by block diagonalization. Show that if

T11 X − XT22 = T12 (4.45)

then    −1   
T11 0 I −X T11 T12 I −X
= ,
0 T22 0 I 0 T22 0 I
and deduce that
F12 = f (T11 )X − Xf (T22 ). (4.46)
Show that the formulae (4.44) and (4.46) are equivalent.
106 Techniques for General Functions

4.4. (Parlett [459, ]) Let S, T ∈ Cn×n be upper triangular and let X = f (T −1 S),
Y = f (ST −1 ). Show that

SX − Y S = 0, T X − Y T = 0, (4.47)

and hence show how to compute X and Y together, by a finite recurrence, without
explicitly forming T −1 S. When does your recurrence break down?
4.5. Consider the conditions of Theorem 4.15 under the weaker assumption that
∂g
| ∂x (x∗ , λ)| = 1. Construct examples with n = 2 to show that the corresponding
matrix iteration may or may not converge.
4.6. Show that A ∈ Cn×n is power bounded (that is, for any norm there exists a
constant c such that kAk k ≤ c for all k ≥ 0) if ρ(A) < 1. Give a necessary and
sufficient condition for A to be power bounded.
4.7. (Elsner [176, ]) Consider the recurrence yk+1 = ck yk + dk , where ck → c
and dk → d as k → ∞, with |c| < 1. Show that limk→∞ yk = d/(1 − c).
4.8. (Kahan [326, ]) Show that A ∈ Cn×n is nilpotent of index k if and only if
trace(Ai ) = 0, i = 1: k.
4.9. (Research problem) Develop bounds for kf (A)−r(A)k for nonnormal A and r
a best L∞ approximation or Padé approximant, for any suitable norm. Some bounds
for particular f and Padé approximants can be found in later chapters.
4.10. (Research problem) Develop new bounds on kf (A)k to add to those in
Section 4.11.
Chapter 5
Matrix Sign Function

The scalar sign function is defined for z ∈ C lying off the imaginary axis by

1, Re z > 0,
sign(z) =
−1, Re z < 0.

The matrix sign function can be obtained from any of the definitions in Chapter 1.
Note that in the case of the Jordan canonical form and interpolating polynomial
definitions, the derivatives sign(k) (z) are zero for k ≥ 1. Throughout this chapter,
A ∈ Cn×n is assumed to have no eigenvalues on the imaginary axis, so that sign(A)
is defined. Note that this assumption implies that A is nonsingular.
As we noted in Section 2.4, if A = ZJZ −1 is a Jordan canonical form arranged
so that J = diag(J1 , J2 ), where the eigenvalues of J1 ∈ Cp×p lie in the open left
half-plane and those of J2 ∈ Cq×q lie in the open right half-plane, then
 
−Ip 0
sign(A) = Z Z −1 . (5.1)
0 Iq

Two other representations have some advantages. First is the particularly concise
formula (see (5.5))
sign(A) = A(A2 )−1/2 , (5.2)
which generalizes the scalar formula sign(z) = z/(z 2 )1/2 . Recall that B 1/2 denotes the
principal square root of B (see Section 1.7). Note that A having no pure imaginary
eigenvalues is equivalent to A2 having no eigenvalues on R− . Next, sign(A) has the
integral representation (see Problem 5.3)
Z ∞
2
sign(A) = A (t2 I + A2 )−1 dt. (5.3)
π 0

Some properties of sign(A) are collected in the following theorem.

Theorem 5.1 (properties of the sign function). Let A ∈ Cn×n have no pure imagi-
nary eigenvalues and let S = sign(A). Then
(a) S 2 = I (S is involutory);
(b) S is diagonalizable with eigenvalues ±1;
(c) SA = AS;
(d) if A is real then S is real;
(e) (I + S)/2 and (I − S)/2 are projectors onto the invariant subspaces associated
with the eigenvalues in the right half-plane and left half-plane, respectively.

107
108 Matrix Sign Function

Proof. The properties follow from (5.1)–(5.3). Of course, properties (c) and (d)
hold more generally for matrix functions, as we know from Chapter 1 (see Theo-
rem 1.13 (a) and Theorem 1.18).
Although sign(A) is a square root of the identity matrix, it is not equal to I or
−I unless the spectrum of A lies entirely in the open right half-plane or open left
half-plane, respectively. Hence, in general, sign(A) is a nonprimary square root of I.
Moreover, although sign(A) has eigenvalues ±1, its norm can be arbitrarily large.
The early appearance of this chapter in the book is due to the fact that the sign
function plays a fundamental role in iterative methods for matrix roots and the polar
decomposition. The definition (5.2) might suggest that the sign function is a “special
case” of the square root. The following theorem, which provides an explicit formula
for the sign of a block 2 × 2 matrix with zero diagonal blocks, shows that, if anything,
the converse is true: the square root can be obtained from the sign function (see
(5.4)). The theorem will prove useful in the next three chapters.

Theorem 5.2 (Higham, Mackey, Mackey, and Tisseur). Let A, B ∈ Cn×n and sup-
pose that AB (and hence also BA) has no eigenvalues on R− . Then
   
0 A 0 C
sign = ,
B 0 C −1 0

where C = A(BA)−1/2 .
 
Proof. The matrix P = B0 A0 cannot have any eigenvalues on the imaginary
 
axis, because if it did then P 2 = AB 0 −
0 BA would have an eigenvalue on R . Hence
sign(P ) is defined and
  −1/2
2 −1/2 0 A AB 0
sign(P ) = P (P ) =
B 0 0 BA
  −1/2

0 A (AB) 0
=
B 0 0 (BA)−1/2
   
0 A(BA)−1/2 0 C
= =: .
B(AB)−1/2 0 D 0

Since the square of the matrix sign of any matrix is the identity,
 2  
2 0 C CD 0
I = (sign(P )) = = ,
D 0 0 DC

so D = C −1 . Alternatively, Corollary 1.34 may be used to see more directly that

CD = A(BA)−1/2 B(AB)−1/2 is equal to I.
A special case of the theorem, first noted by Higham [274, ], is
   
0 A 0 A1/2
sign = . (5.4)
I 0 A−1/2 0

In addition to the association with matrix roots and the polar decomposition
(Chapter 8), the importance of the sign function stems from its applications to Ric-
cati equations (Section 2.4), the eigenvalue problem (Section 2.5), and lattice QCD
(Section 2.7).
5.1 Sensitivity and Conditioning 109

In this chapter we first give perturbation theory for the matrix sign function and
identify appropriate condition numbers. An expensive, but stable, Schur method for
sign(A) is described. Then Newton’s method and a rich Padé family of iterations,
having many interesting properties, are described and analyzed. How to scale and how
to terminate the iterations are discussed. Then numerical stability is considered, with
the very satisfactory conclusion that all sign iterations of practical interest are sta-
ble. Numerical experiments illustrating these various features are presented. Finally,
best L∞ rational approximation via Zolotarev’s formulae, of interest for Hermitian
matrices, is described.
As we will see in Chapter 8, the matrix sign function has many connections with
the polar decomposition, particularly regarding iterations for computing it. Some
of the results and ideas in Chapter 8 are applicable, with suitable modification, to
the sign function, but are not discussed here to avoid repetition. See, for example,
Problem 8.26.

5.1. Sensitivity and Conditioning

Associated with the matrix sign function is the matrix sign decomposition

A = SN, S = sign(A), N = (A2 )1/2 . (5.5)

To establish the decomposition note that N = S −1 A = SA. Since S commutes

with A, N 2 = A2 , and since the spectrum of SA lies in the open right half-plane,
N = (A2 )1/2 .
The matrix sign factor N is useful in characterizing the Fréchet derivative of the
matrix sign function.
Let S + ∆S = sign(A + ∆A), where the sign function is assumed to be defined in
a ball of radius k∆Ak about A. The definition (3.6) of Fréchet derivative says that

∆S − L(A, ∆A) = o(k∆Ak), (5.6)

where L(A, ∆A) is the Fréchet derivative of the matrix sign function at A in the
direction ∆A. Now from (A + ∆A)(S + ∆S) = (S + ∆S)(A + ∆A) we have

A∆S − ∆SA = S∆A − ∆AS + ∆S∆A − ∆A∆S = S∆A − ∆AS + o(k∆Ak), (5.7)

since ∆S = O(k∆Ak). Moreover, (S + ∆S)2 = I gives

S∆S + ∆SS = −∆S 2 = o(k∆Ak).

Premultiplying (5.7) by S and using the latter equation gives

N ∆S + ∆S N = ∆A − S∆AS + o(k∆Ak). (5.8)

Theorem 5.3 (Kenney and Laub). The Fréchet derivative L = Lsign (A, ∆A) of the
matrix sign function satisfies

N L + LN = ∆A − S∆AS, (5.9)

where A = SN is the matrix sign decomposition.

110 Matrix Sign Function

Proof. Since the eigenvalues of N lie in the open right half-plane, the Sylvester
equation (5.9) has a unique solution L which is a linear function of ∆A and, in view
of (5.8), differs from ∆S = sign(A + ∆A) − S by o(k∆Ak). Hence (5.6) implies that
L = L(A, ∆A).
By applying the vec operator and using the relation (B.16) we can rewrite (5.9)
as
P vec(L) = (In2 − S T ⊗ S) vec(∆A),
where
P = I ⊗ N + N T ⊗ I.
Hence
max kL(A, ∆A)kF = max kP −1 (In2 − S T ⊗ S) vec(∆A)k2
k∆AkF =1 k∆AkF =1

= kP −1 (In2 − S T ⊗ S)k2 .
The (relative) condition number of sign(A) in the Frobenius norm is therefore
kAkF
κsign (A) := condrel (sign, A) = kP −1 (In2 − S T ⊗ S)k2 . (5.10)
kSkF
If S = I, which means that all the eigenvalues of A are in the open right half-plane,
then cond(S) = 0, which corresponds to the fact that the eigenvalues remain in this
half-plane under sufficiently small perturbations of A.
To gain some insight into the condition number, suppose that A is diagonalizable:
A = ZDZ −1 , where D = diag(λi ). Then S = ZDS Z −1 and N = ZDN Z −1 , where
DS = diag(σi ) and DN = diag(σi λi ), with σi = sign(λi ). Hence
kAkF
κsign (A) = k(Z −T ⊗ Z) · (I ⊗ DN + DN ⊗ I)−1 (In2 − DST ⊗ DS ) · (Z T ⊗ Z −1 )k2 .
kSkF
The diagonal matrix in the middle has elements (1 − σi σj )/(σi λi + σj λj ), which are
either zero or of the form 2/|λi − λj |. Hence
 
2 1 kAkF
κsign (A) ≤ 2κ2 (Z) max : Re λi Re λj < 0 . (5.11)
|λi − λj | kSkF
Equality holds in this bound for normal A, for which Z can be taken to unitary.
The gist of (5.11) is that the condition of S is bounded in terms of the minimum
distance between eigenvalues across the imaginary axis and the square of the condition
of the eigenvectors. Note that (5.11) is precisely the bound obtained by applying
Theorem 3.15 to the matrix sign function.
One of the main uses of κsign is to indicate the sensitivity of sign(A) to perturba-
tions in A, through the perturbation bound (3.3), which we rewrite here for the sign
function as
k sign(A + E) − sign(A)kF kEkF
≤ κsign (A) + o(kEkF ). (5.12)
k sign(A)kF kAkF
This bound is valid as long as sign(A + tE) is defined for all t ∈ [0, 1]. It is instructive
to see what can go wrong when this condition is not satisfied. Consider the example,
from [347, ],
A = diag(1, −ǫ2 ), E = diag(0, 2ǫ2 ), 0 < ǫ ≪ 1.
5.1 Sensitivity and Conditioning 111

We have sign(A) = diag(1, −1) and√sign(A + E) = I. Because A is normal, (5.11)

gives κsign (A) = (2/(1 + ǫ2 ))kAkF / 2. Hence the bound (5.12) takes the form

2 2 √
√ ≤ √ 2ǫ2 + o(ǫ2 ) = 2 2ǫ2 + o(ǫ2 ).
2 2
2(1 + ǫ )
This bound is clearly incorrect. The reason is that the perturbation E causes eigenval-
ues to cross the imaginary axis; therefore sign(A + tE) does not exist for all t ∈ [0, 1].
Referring back to the analysis at the start of this section, we note that (5.7) is valid
for k∆AkF < kEkF /3, but does not hold for ∆A = E, since then ∆S 6= O(k∆Ak).
Another useful characterization of the Fréchet derivative is as the limit of a matrix
iteration; see Theorem 5.7.
Consider now how to estimate κsign (A). We need to compute a norm of B =
P −1 (In2 − S T ⊗ S). For the 2-norm we can use Algorithm 3.20 (the power method).
Alternatively, Algorithm 3.22 can be used to estimate the 1-norm. In both cases we
need to compute L(A, E), which if done via (5.9) requires solving a Sylvester equation
involving N ; this can be done via a matrix sign evaluation (see Section 2.4), since N
is positive stable. We can compute L⋆ (X, E) in a similar fashion, solving a Sylvester
equation of the same form. Alternatively, L(A, E) can be computed using iteration
(5.23) or estimated by finite differences. All these methods require O(n3 ) operations.
It is also of interest to understand the conditioning of the sign function for A ≈
sign(A), which is termed the asymptotic conditioning. The next result provides useful
bounds.

Theorem 5.4 (Kenney and Laub). Let A ∈ Cn×n have no pure imaginary eigenval-
ues and let S = sign(A). If k(A − S)Sk2 < 1, then

kSk22 − 1 κsign (A) kSk22 + 1

≤ ≤ . (5.13)
2(1 + k(A − S)Sk2 ) kAkF /kSkF 2(1 − k(A − S)Sk2 )
In particular,
kSk22 − 1 kSk22 + 1
≤ κsign (S) ≤ . (5.14)
2 2
Proof. We need to bound kLsign (A)kF = κsign (A)kSkF /kAkF . Let ∆S =
Lsign (A, ∆A). Then by (5.9),

N ∆S + ∆SN = ∆A − S∆AS,

where N = SA = AS. Defining G = AS − S 2 = N − I, we have

2∆S = ∆A − S∆AS − G∆S − ∆SG. (5.15)

Taking norms, using (B.7), leads to

(kSk22 + 1)k∆AkF
k∆SkF ≤ ,
2(1 − kGk2 )
which gives the upper bound.
Now let σ = kSk2 and Sv = σu, u∗ S = σv ∗ , where u and v are (unit-norm) left
and right singular vectors, respectively. Putting ∆A = vu∗ in (5.15) gives

2∆S = vu∗ − Svu∗ S − G∆S − ∆S G = vu∗ − σ 2 uv ∗ − G∆S − ∆S G.

112 Matrix Sign Function

Hence
(kSk22 − 1)k∆AkF = (σ 2 − 1)k∆AkF ≤ 2k∆SkF (1 + kGk2 ),
which implies the lower bound.
Setting A = S in (5.13) gives (5.14).
Theorem 5.4 has something to say about the attainable accuracy of a computed
sign function. In computing S = sign(A) we surely cannot do better than if we
computed sign(f l(S)). But Theorem 5.4 says that relative errors in S can be magnified
when we take the sign by as much as kSk2 /2, so we cannot expect a relative error in
our computed sign smaller than kSk2 u/2, whatever the method used.

5.2. Schur Method

The first method that we consider for computing sign(A) is expensive but has excellent
numerical stability. Because the method utilizes a Schur decomposition it is not
suitable for the applications in Sections 2.4 and 2.5, since those problems can be
solved directly by the use of a Schur decomposition, without explicitly forming the
sign function.
Let A ∈ Cn×n have the Schur decomposition A = QT Q∗ , where Q is unitary
and T is upper triangular. Then sign(A) = Q sign(T )Q∗ (see Theorem 1.13 (c)).
The problem therefore reduces to computing U = sign(T ), and clearly U is upper
triangular with uii = sign(tii ) = ±1 for all i. We will determine uij from the equation
U 2 = I when possible (namely, when uii + ujj 6= 0), and from T U = U T otherwise
(in which case tii 6= tjj ), employing the Parlett recurrence (Algorithm 4.13) in this
second case.

Algorithm 5.5 (Schur method). Given A ∈ Cn×n having no pure imaginary eigen-
values, this algorithm computes S = sign(A) via a Schur decomposition.

1 Compute a (complex) Schur decomposition A = QT Q∗ .

2 uii = sign(tii ), i = 1: n
3 for j = 2: n
4 for i = j −1: −1: 1
Pj−1


 k=i+1 uik ukj
− , uii + ujj 6= 0,
uii + ujj
5 uij = Pj−1

 uii − ujj (uik tkj − tik ukj )

 tij + k=i+1 , uii + ujj = 0.
tii − tjj tii − tjj
6 end
7 end
8 S = QU Q∗

Cost: 25n3 flops for the Schur decomposition plus between n3/3 and 2n3/3 flops for
U and 3n3 flops to form S: about 28 23 n3 flops in total.
It is worth noting that the sign of an upper triangular matrix T will usually
have some zero elements in the upper triangle. Indeed, suppose for some j > i that
tii , ti+1,i+1 , . . . , tjj all have the same sign, and let Tij = T (i: j, i: j). Then, since
all the eigenvalues of Tij have the same sign, the corresponding block S(i: j, i: j) of
S = sign(T ) is ±I. This fact could be exploited by reordering the Schur form so that
the diagonal of T is grouped according to sign. Then sign(T ) would have the form
5.3 Newton’s Method 113

 ±IW

0 ∓I , where W is computed by the Parlett recurrence. The cost of the reordering
may or may not be less than the cost of (redundantly) computing zeros from the first
expression for uij in Algorithm 5.5.

5.3. Newton’s Method

The most widely used and best known method for computing the sign function is the
Newton iteration, due to Roberts:

Newton iteration (matrix sign function):

1
Xk+1 = (Xk + Xk−1 ), X0 = A. (5.16)
2

The connection of this iteration with the sign function is not immediately obvious,
but in fact the iteration can be derived by applying Newton’s method to the equation
X 2 = I (see Problem 5.8), and of course sign(A) is one solution of this equation
(Theorem 5.1 (a)). The following theorem describes the convergence of the iteration.

Theorem 5.6 (convergence of the Newton sign iteration). Let A ∈ Cn×n have no
pure imaginary eigenvalues. Then the Newton iterates Xk in (5.16) converge quadrat-
ically to S = sign(A), with
1
kXk+1 − Sk ≤ kX −1 k kXk − Sk2 (5.17)
2 k
for any consistent norm. Moreover, for k ≥ 1,
k k
Xk = (I − G20 )−1 (I + G20 )S, where G0 = (A − S)(A + S)−1 . (5.18)

Proof. For λ = reiθ we have λ + λ−1 = (r + r−1 ) cos θ + i(r − r−1 ) sin θ, and hence
eigenvalues of Xk remain in their open half-plane under the mapping (5.16). Hence
Xk is defined and nonsingular for all k. Moreover, sign(Xk ) = sign(X0 ) = S, and so
Xk + S = Xk + sign(Xk ) is also nonsingular.
Clearly the Xk are (rational) functions of A and hence, like A, commute with S.
Then
1

Xk+1 ± S = Xk + Xk−1 ± 2S
2
1 −1 2

= Xk Xk ± 2Xk S + I
2
1
= Xk−1 (Xk ± S)2 , (5.19)
2
and hence
2
(Xk+1 − S)(Xk+1 + S)−1 = (Xk − S)(Xk + S)−1 .
k+1
Defining Gk = (Xk − S)(Xk + S)−1 , we have Gk+1 = G2k = · · · = G02 . Now
G0 = (A − S)(A + S)−1 has eigenvalues (λ − sign(λ))/(λ + sign(λ)), where λ ∈ Λ(A),
k
all of which lie inside the unit circle since λ is not pure imaginary. Since Gk = G20
and ρ(G0 ) < 1, by a standard result (B.9) Gk → 0 as k → ∞. Hence

Xk = (I − Gk )−1 (I + Gk )S → S as k → ∞. (5.20)
114 Matrix Sign Function

The norm inequality (5.17), which displays the quadratic convergence, is obtained by
taking norms in (5.19) with the minus sign.

Theorem 5.6 reveals quadratic convergence of the Newton iteration, but also dis-
plays in (5.18) precisely how convergence occurs: through the powers of the matrix
G0 converging to zero. Since for any matrix norm,

  2k
k k |λ − sign(λ)|
kG20 k ≥ ρ(G20 ) = max . , (5.21)
λ∈Λ(A) |λ + sign(λ)|

It is clear that convergence will be slow if either ρ(A) ≫ 1 or A has an eigenvalue

close to the imaginary axis. We return to the speed of convergence in Section 5.5.
For the behaviour of the iteration when it does not converge, see Problem 5.11.
The Newton iteration provides one of the rare circumstances in numerical analysis
where the explicit computation of a matrix inverse is required. One way to try to
remove the inverse from the formula is to approximate it by one step of Newton’s
method for the matrix inverse, which has the form Yk+1 = Yk (2I −BYk ) for computing
B −1 ; this is known as the Newton–Schulz iteration [512, ] (see Problem 7.8).
Replacing Xk−1 by Xk (2I − Xk2 ) in (5.16) (having taken Yk = B = Xk ) gives

Newton–Schulz iteration:
1
Xk+1 = Xk (3I − Xk2 ), X0 = A. (5.22)
2

This iteration is multiplication-rich and retains the quadratic convergence of Newton’s

method. However, it is only locally convergent, with convergence guaranteed for
kI − A2 k < 1; see Theorem 5.8.
The Newton iteration also provides a way of computing the Fréchet derivative of
the sign function.

Theorem 5.7 (Kenney and Laub). Let A ∈ Cn×n have no pure imaginary eigenval-
ues. With Xk defined by the Newton iteration (5.16), let

1
Yk+1 = (Yk − Xk−1 Yk Xk−1 ), Y0 = E. (5.23)
2

Then limk→∞ Yk = Lsign (A, E).

 
Proof. Denote by Bk the Newton sign iterates (5.16) for the matrix B = A0 E A ,
which clearlyhas no pure imaginary eigenvalues. It is easy to show by induction that
Bk = X0k X Yk
k
. By Theorem 5.6 and (3.16) we have
 
sign(A) Lsign (A, E)
Bk → sign(B) = .
0 sign(A)

The result follows on equating the (1,2) blocks.

5.4 The Padé Family of Iterations 115

5.4. The Padé Family of Iterations

The Newton iteration is by no means the only rational matrix iteration for computing
the matrix sign function. A variety of other iterations have been derived, with various
aims, including to avoid matrix inversion in favour of matrix multiplication, to achieve
a higher order of convergence, and to be better suited to parallel computation. Ad hoc
manipulations can be used to derive new iterations, as we now indicate for the scalar
case. By setting yk = x−1 −1
k in the Newton formula xk+1 = (xk + xk )/2, we obtain
the “inverse Newton” variant
2yk
yk+1 = , y0 = a, (5.24)
yk2 + 1

which has quadratic convergence to sign(a). Combining two Newton steps yields
yk+2 = (yk4 + 6yk2 + 1)/(4yk (yk2 + 1)), and we can thereby define the quartically con-
vergent iteration
y 4 + 6yk2 + 1
yk+1 = k , y0 = a.
4yk (yk2 + 1)
While a lot can be done using arguments such as these, a more systematic development
is preferable. We describe an elegant Padé approximation approach, due to Kenney
and Laub [343, ], that yields a whole table of methods containing essentially all
those of current interest.
For non–pure imaginary z ∈ C we can write
z z z
sign(z) = = = , (5.25)
(z 2 )1/2 (1 − (1 − z 2 ))1/2 (1 − ξ)1/2

where ξ = 1 − z 2 . Hence the task of approximating sign(z) leads to that of approxi-

mating
h(ξ) = (1 − ξ)−1/2 , (5.26)
where we may wish to think of ξ as having magnitude less than 1. Now h is a
particular case of a hypergeometric function and hence much is known about [ℓ/m]
Padé approximants rℓm (ξ) = pℓm (ξ)/qℓm (ξ) to h, including explicit formulae for pℓm
and qℓm . (See Section 4.4.2 for the definition of Padé approximants.) Kenney and
Laub’s idea is to set up the family of iterations

pℓm (1 − x2k )
xk+1 = fℓm (xk ) := xk , x0 = a. (5.27)
qℓm (1 − x2k )

Table 5.1 shows the first nine iteration functions fℓm from this family. Note that f11
gives Halley’s method (see Problem 5.12), while f10 gives the Newton–Schulz iteration
(5.22). The matrix versions of the iterations are defined in the obvious way:

Padé iteration:

Xk+1 = Xk pℓm (I − Xk2 ) qℓm (I − Xk2 )−1 , X0 = A. (5.28)

Two key questions are “what can be said about the convergence of (5.28)?” and “how
should the iteration be evaluated?”
The convergence question is answered by the following theorem.
116 Matrix Sign Function

Table 5.1. Iteration functions fℓm from the Padé family (5.27).

m=0 m=1 m=2

2x 8x
ℓ=0 x
1 + x2 3 + 6x2 − x4
x x(3 + x2 ) 4x(1 + x2 )
ℓ=1 (3 − x2 )
2 1 + 3x2 1 + 6x2 + x4
x x (15 + 10x2 − x4 ) x(5 + 10x2 + x4 )
ℓ=2 (15 − 10x2 + 3x4 )
8 4 1 + 5x2 1 + 10x2 + 5x4

Theorem 5.8 (convergence of Padé iterations). Let A ∈ Cn×n have no pure imagi-
nary eigenvalues. Consider the iteration (5.28) with ℓ + m > 0 and any subordinate
matrix norm.
(a) For ℓ ≥ m − 1, if kI − A2 k < 1 then Xk → sign(A) as k → ∞ and kI − Xk2 k <
k
kI − A2 k(ℓ+m+1) .
(b) For ℓ = m − 1 and ℓ = m,
 (ℓ+m+1)k
(S − Xk )(S + Xk )−1 = (S − A)(S + A)−1

and hence Xk → sign(A) as k → ∞.

Proof. See Kenney and Laub [343, ].
Theorem 5.8 shows that the iterations with ℓ = m − 1 and ℓ = m are globally
convergent, while those with ℓ ≥ m + 1 have local convergence, the convergence rate
being ℓ + m + 1 in every case.
We now concentrate on the cases ℓ = m − 1 and ℓ = m which we call the principal
Padé iterations. For these ℓ and m we define

gr (x) ≡ gℓ+m+1 (x) = fℓm (x). (5.29)

The gr are the iteration functions from the Padé table taken in a zig-zag fashion from
the main diagonal and first superdiagonal:

2x x(3 + x2 )
g1 (x) = x, g2 (x) = , g3 (x) = ,
1 + x2 1 + 3x2
4x(1 + x2 ) x(5 + 10x2 + x4 ) x(6 + 20x2 + 6x4 )
g4 (x) = , g5 (x) = , g 6 (x) = .
1 + 6x2 + x4 1 + 10x2 + 5x4 1 + 15x2 + 15x4 + x6
We know from Theorem 5.8 that the iteration Xk+1 = gr (Xk ) converges to sign(X0 )
with order r whenever sign(X0 ) is defined. These iterations share some interesting
properties that are collected in the next theorem.

Theorem 5.9 (properties of principal Padé iterations). The principal Padé iteration
function gr defined in (5.29) has the following properties.
(1 + x)r − (1 − x)r
(a) gr (x) = . In other words, gr (x) = pr (x)/qr (x), where
(1 + x)r + (1 − x)r
pr (x) and qr (x) are, respectively, the odd and even parts of (1 + x)r .
5.4 The Padé Family of Iterations 117

(b) gr (x) = tanh(r arctanh(x)).

(c) gr (gs (x)) = grs (x) (the semigroup property).
(d) gr has the partial fraction expansion

2⌈ r−2 ⌉
2 X′ x
gr (x) =     , (5.30)
r i=0 sin2 (2i+1)π
+ cos2 (2i+1)π x2
2r 2r

where the prime on the summation symbol denotes that the last term in the sum is
halved when r is odd.
Proof.
(a) See Kenney and Laub [343, 1991, Thm. 3.2].
(b) Recalling that tanh(x) = (ex − e−x )/(ex + e−x ), it is easy to check that
 
1 1+x
arctanh(x) = log .
2 1−x
Hence  r/2
1+x
r arctanh(x) = log .
1−x
Taking the tanh of both sides gives
 r/2  r/2
1+x 1−x
−
1−x 1+x (1 + x)r − (1 − x)r
tanh(r arctanh(x)) =  r/2  r/2 = (1 + x)r + (1 − x)r = gr (x).
1+x 1−x
+
1−x 1+x

(c) Using (b) we have

gr (gs (x)) = tanh(r arctanh(tanh(s arctanh(x)))) = tanh(rs arctanh(x))

= grs (x).

(d) The partial fraction expansion is obtained from a partial fraction expansion for
the hyperbolic tangent; see Kenney and Laub [345, , Thm. 3].

Some comments on the theorem are in order. The equality in (a) is a scalar equiv-
alent of (b) in Theorem 5.8, and it provides an easy way to generate the gr . Property
(c) says that one rth order principal Padé iteration followed by one sth order iteration
is equivalent to one rsth order iteration. Whether or not it is worth using higher order
iterations therefore depends on the efficiency with which the different iterations can
be evaluated. The properties in (b) and (c) are analogous to properties of the Cheby-
shev polynomials. Figure 5.1 confirms, for real x, that gr (x) = tanh(r arctanh(x))
approximates sign(x) increasingly well near the origin as r increases.
Some more insight into the convergence, or nonconvergence, of the iteration xk+1 =
gr (xk ) from (5.29) can be obtained by using Theorem 5.8 (b) to write, in polar form,
 r  r
ρk+1 eiθk+1 := (s − xk+1 )(s + xk+1 )−1 := (s − xk )(s + xk )−1 = ρk eiθk ,
118 Matrix Sign Function

0.5

−0.5

r= 2
r= 4
−1 r= 8
r = 16

−3 −2 −1 0 1 2 3

Figure 5.1. The function gr (x) = tanh(r arctanh(x)) for r = 2, 4, 8, 16.

where s = sign(x0 ). Hence

ρk+1 = ρrk , θk+1 = rθk .

These relations illustrate the convergence of xk to s for x0 off the imaginary axis,

since ρ0 < 1. But they also reveal a chaotic aspect to the convergence through θk ,
which, in view of the periodicity of eiθk , can be written

θk+1 = rθk mod 2π. (5.31)

This recurrence can be described as a linear congruential random number generator

[211, , Sec. 1.2], [357, , Sec. 3.2], though with a real, rather than integer,
modulus. If x0 is pure imaginary then the iteration does not converge: ρk ≡ 1, xk
remains pure imaginary for all k, and (s − xk )(s + xk )−1 wanders chaotically around
the circle of radius 1 centred at the origin; see also Problem 5.11.
We turn now to evaluation of the matrix iteration Xj+1 = gr (Xj ). As discussed
in Section 4.4.2, several approaches are possible, based on different representations of
the rational iteration function gk . Evaluating gk (Xj ) as the ratio of two polynomials
may require more flops than via the partial fraction expansion (5.30). For example,
evaluating g3 from the formula x(3 + x2 )/(1 + 3x2 ) at an n × n matrix requires 6 32 n3
flops, whereas (5.30) can be written as
 
1 8x
g3 (x) = x+ (5.32)
3 1 + 3x2

and evaluated in 4 23 n3 flops. An attractive feature of the partial fraction expansion

(5.30) is that it comprises ⌈ r−2
2 ⌉ independent matrix inversions (or multiple right-
hand side linear systems), which can be carried out in parallel.
5.5 Scaling the Newton Iteration 119

5.5. Scaling the Newton Iteration

For scalar a, the Newton iteration (5.16) is
1
xk+1 = (x + x−1
k ), x0 = a, (5.33)
2 k
which converges to sign(a) = ±1 if a is not pure imaginary. This is precisely Newton’s
method for the square root of 1 and convergence is at a quadratic rate, as described by
(5.17). Once the error is sufficiently small (in practice, less than, say, 0.5), successive
errors decrease rapidly, each being approximately the square of the previous one (see
(5.19)). However, initially convergence can be slow: if |xk | ≫ 1 then xk+1 ≈ xk /2
and the iteration is an expensive way to divide by 2! From (5.18) and (5.21) we also
see that slow convergence will result when a is close to the imaginary axis. Therefore
a way is needed of speeding up the initial phase of convergence in these unfavourable
cases. For matrices, the same comments apply to the eigenvalues, because the New-
ton iteration (5.16) is effectively performing the scalar iteration (5.33) independently
on each eigenvalue. However, the behaviour of the matrix iteration is not entirely
determined by the eigenvalues: nonnormality of A can delay, though not prevent,
convergence, as the following finite termination result shows.

Theorem 5.10 (Kenney and Laub). For the Newton iteration (5.16), if Xk has eigen-
values ±1 for some k then Xk+p = sign(A) for 2p ≥ m, where m is the size of the
largest Jordan block of Xk (which is no larger than the size of the largest Jordan block
of A).
Proof. Let Xk have the Jordan form Xk = ZJk Z −1 , where Jk = D + Nk , with
D = diag(±1) = sign(Jk ) and Nk strictly upper triangular. Nk has index of nilpotence
m, that is, Nkm = 0 but all lower powers are nonzero. We can restrict our attention
to the convergence of the sequence beginning with Jk to diag(±1), and so we can set
Z = I. The next iterate, Xk+1 = D + Nk+1 , satisfies, in view of (5.19),
1 −1 2
Nk+1 =X Nk .
2 k
Since Nk has index of nilpotence m, Nk+1 must have index of nilpotence ⌈m/2⌉. Ap-
plying this argument repeatedly shows that for 2p ≥ m, Nk+p has index of nilpotence
1 and hence is zero, as required. That m is no larger than the order of the largest
Jordan block of A follows from Theorem 1.36.
An effective way to enhance the initial speed of convergence is to scale the iterates:
prior to each iteration, Xk is replaced by µk Xk , giving the scaled Newton iteration

Scaled Newton iteration:

1

Xk+1 = µk Xk + µ−1 −1
k Xk , X0 = A. (5.34)
2
As long as µk is real and positive, the sign of the iterates is preserved. Three main
scalings have been proposed:
determinantal scaling: µk = | det(Xk )|−1/n , (5.35)
q
spectral scaling: µk = ρ(Xk−1 )/ρ(Xk ), (5.36)
q
norm scaling: µk = kXk−1 k/kXk k. (5.37)
120 Matrix Sign Function

For determinantal scaling, | det(µk Xk )| = 1, so that the geometric mean of the eigen-
values of µk Xk has magnitude 1. This scaling has the property that µk minimizes
d(µk Xk ), where
Xn
d(X) = (log |λi |)2
i=1

and the are λi the eigenvalues of X. Hence determinantal scaling tends to bring the
eigenvalues closer to the unit circle; see Problem 5.13.
When evaluating the determinantal scaling factor (5.35) some care is needed to
avoid unnecessary overflow and underflow, especially when n is large. The quan-
tity µk should be within the range of the floating point arithmetic, since its re-
ciprocal has magnitude the geometric mean of the eigenvalues of Xk and hence
lies between the moduli of the smallest and largest eigenvalues. But det(Xk ) can
underflow or overflow. Assuming that an LU factorization P Xk = Lk Uk is com-
−1/n
puted, where Uk hasPndiagonal elements uii , we can rewrite µk = |u11 . . . unn | as
µk = exp((−1/n) i=1 log |uii |). The latter expression avoids underflow and over-
flow; however, cancellation in the summation can produce an inaccurate computed
µk , so it may be desirable to use one of the summation methods from Higham [276,
, Chap. 4].
For spectral scaling, if λn , . . . , λ1 are the eigenvalues of Xk ordered by increasing
magnitude, then µk = |λ1 λn |−1/2 and so µk Xk has eigenvalues of smallest and largest
magnitude |µk λn | = |λn /λ1 |1/2 and |µk λ1 | = |λ1 /λn |1/2 . If λ1 and λn are real, then
in the Cayley metric

|x − 1|/|x + 1|, Re x > 0,
C(x, sign(x)) :=
|x + 1|/|x − 1|, Re x < 0,

µk λn is the same distance from sign(λn ) as µk λ1 is from sign(λ1 ), so in this case

spectral scaling equalizes the extremal eigenvalue errors in the Cayley metric. The
norm scaling (5.37) can be regarded as approximating the spectral scaling.
What can be said about the effectiveness of these scaling strategies? In general,
all of them work well, but there are some specific advantages and disadvantages.
Spectral scaling is essentially optimal when all the eigenvalues of A are real; indeed
it yields finite termination, as the following result shows.

Theorem 5.11 (Barraud). Let the nonsingular matrix A ∈ Cn×n have all real eigen-
values and let S = sign(A). Then, for the Newton iteration (5.34) with spectral scaling,
Xd+p−1 = sign(A), where d is the number of distinct eigenvalues of A and 2p ≥ m,
where m is the size of the largest Jordan block of A.
Proof. We will need to use the following easily verified properties of the iteration
function f (x) = 21 (x + 1/x):

f (x) = f (1/x), (5.38a)

0 ≤ x2 ≤ x1 ≤ 1 or 1 ≤ x1 ≤ x2 ⇒ 1 ≤ f (x1 ) ≤ f (x2 ). (5.38b)

Let the eigenvalues of X0 = A, which we know to be real, be ordered |λn | ≤ · · · ≤ |λ1 |.

Then, from (5.36), µ0 = |λn λ1 |−1/2 , and the eigenvalues of µ0 X0 have moduli lying
between |µ0 λn | = |λn /λ1 |1/2 and |µ0 λ1 | = |λ1 /λn |1/2 . These values are reciprocals,
and hence by (5.38a), and since the eigenvalues are real, λn and λ1 are mapped
to values with the same modulus. By (5.38b) these values are the eigenvalues of
5.6 Terminating the Iterations 121

(1) (1)
X1 of largest modulus. Hence X1 has eigenvalues λi satisfying |λn | ≤ · · · ≤
(1) (1)
|λ2 | = |λ1 |. Each subsequent iteration increases by at least 1 the number of
eigenvalues with maximal modulus until, after d − 1 iterations, Xd−1 has eigenvalues
of constant modulus. Then µd−1 Xd−1 has converged eigenvalues ±1 (as does Xd ).
By Theorem 5.10, at most a further p iterations after Xd−1 are needed to dispose of
the Jordan blocks (and during these iterations µk ≡ 1, since the eigenvalues are fixed
at ±1).

For 1 × 1 matrices spectral scaling and determinantal scaling are equivalent, and
both give convergence in at most two iterations (see Problem 5.14). For 2×2 matrices
spectral scaling and determinantal scaling are again equivalent, and Theorem 5.11
tells us that we have convergence in at most two iterations if the eigenvalues are
real. However, slightly more is true: both scalings give convergence in at most two
iterations for any real 2 × 2 matrix (see Problem 5.14).
Determinantal scaling can be ineffective when there is a small group of outlying
eigenvalues and the rest are nearly converged. Suppose that A has an eigenvalue
10q (q ≥ 1) with the rest all ±1. Then determinantal scaling gives µk = 10−q/n ,
whereas spectral scaling gives µk = 10−q/2 ; the former quantity is close to 1 and hence
the determinantally scaled iteration will behave like the unscaled iteration. Spectral
scaling can be ineffective when the eigenvalues of A cluster close to the imaginary
axis (see the numerical examples in Section 5.8).
All three scaling schemes are inexpensive to implement. The determinant det(Xk )
can be computed at negligible cost from the LU factorization that will be used to
compute Xk−1 . The spectral scaling parameter can be cheaply estimated by applying
the power method to Xk and its inverse, again exploiting the LU factorization in the
latter case. Note, however, that for a real spectrum spectral scaling increases the
number of eigenvalues with maximal modulus on each iteration, which makes reliable
implementation of the power method more difficult. The norm scaling is trivial to
compute for the Frobenius norm, and for the 2-norm can be estimated using the power
method (Algorithm 3.19).
The motivation for scaling is to reduce the length of the initial phase during
which the error is reduced below 1. Should we continue to scale throughout the whole
iteration? All three scaling parameters (5.35)–(5.37) converge to 1 as Xk → S, so
scaling does not destroy the quadratic convergence. Nor does it bring any benefit, so
it is sensible to set µk ≡ 1 once the error is sufficiently less than 1.

5.6. Terminating the Iterations

Crucial to the success of any sign iteration is an inexpensive and effective way to decide
when to terminate it. We begin with a lemma that provides some bounds that help
guide the choice of stopping criterion in both relative error-based and residual-based
tests.

Lemma 5.12 (Kenney, Laub, Pandey, and Papadopoulos). Let A ∈ Cn×n have no
pure imaginary eigenvalues, let S = sign(A), and let k · k be any subordinate matrix
norm. If kS(A − S)k = ǫ < 1 then
   
1−ǫ 1+ǫ
kA − A−1 k ≤ kA − Sk ≤ kA − A−1 k (5.39)
2+ǫ 2−ǫ
122 Matrix Sign Function

and
kA2 − Ik kA − Sk
≤ ≤ kA2 − Ik. (5.40)
kSk(kAk + kSk) kSk
The lower bound in (5.40) always holds.
Proof. Let E = A − S. Since S 2 = I, we have A = S + E = (I + ES)S. It is
then straightforward to show that

E(2I + ES) = (A − A−1 )(I + ES),

using the fact that A and S, and hence also E and S, commute. The upper bound
in (5.39) is obtained by postmultiplying by (2I + ES)−1 and taking norms, while
postmultiplying by (I + ES)−1 and taking norms gives the lower bound.
The lower bound in (5.40) is obtained by taking norms in A2 −I = (A−S)(A+S).
For the upper bound, we write the last equation as A − S = (A2 − I)(A + S)−1 and
need to bound k(A + S)−1 k. Since A + S = 2S(I + 21 S(A − S)), we have
1
1 −1
−1 kS −1 k
k(A + S)−1 k = kS I + 12 S(A − S) k≤ 2 ≤ kSk.
2 1 − 12 ǫ

Note that since the iterations of interest satisfy sign(Xk ) = sign(A), the bounds
of Lemma 5.12 are applicable with A replaced by an iterate Xk .
We now describe some possible convergence criteria, using η to denote a con-
vergence tolerance proportional to both the unit roundoff (or a larger value if full
accuracy is not required) and a constant depending on the matrix dimension, n. A
norm will denote any easily computable norm such as the 1-, ∞-, or Frobenius norms.
We begin with the Newton iteration, describing a variety of existing criteria followed
by a new one.
A natural stopping criterion, of negligible cost, is

kXk+1 − Xk k
δk+1 := ≤ η. (5.41)
kXk+1 k

As discussed in Section 4.9, this criterion is really bounding the error in Xk , rather
than Xk+1 , so it may stop one iteration too late. This drawback can be seen very
clearly from (5.39): since Xk+1 −Xk = 21 (Xk−1 −Xk ), (5.39) shows that kXk+1 −Xk k ≈
kS − Xk k is an increasingly good approximation as the iteration converges.
The test (5.41) could potentially never be satisfied in floating point arithmetic.
The best bound for the error in the computed Z bk = f l(X −1 ), which we assume to
k
be obtained by Gaussian elimination with partial pivoting, is of the form [276, ,
Sec. 14.3]
kZbk − X −1 k
k
≤ cn uκ(Xk ), (5.42)
kXk−1 k
where cn is a constant. Therefore for the computed sequence Xk , kXk+1 − Xk k ≈
1 b
2 kZk − Xk k might be expected to be proportional to κ(Xk )kXk ku, suggesting the
test δk+1 ≤ κ(Xk )η. Close to convergence, Xk+1 ≈ Xk ≈ S = S −1 and so κ(Xk ) ≈
kXk+1 k2 . A test δk+1 ≤ kXk+1 k2 η is also suggested by the asymptotic conditioning
of the sign function, discussed at the end of Section 5.1. On the other hand, a test of
the form δk+1 ≤ kXk+1 kη is suggested by Byers, He, and Mehrmann [89, ], based
5.7 Numerical Stability of Sign Iterations 123

on a perturbation bound for the sign function. To summarize, there are arguments
for using the stopping criterion

δk+1 ≤ kXk+1 kp η (5.43)

for each of p = 0, 1, and 2.

A different approach is based on the bound (5.17): kXk+1 − Sk ≤ 21 kXk−1 k kXk −
2
Sk . Since kXk+1 − Xk k ≈ kS − Xk k close to convergence, as noted above,
1
kXk+1 − Sk < −1 2
∼ 2 kXk k kXk+1 − Xk k .

Hence we can expect kXk+1 − Sk/kXk+1 k <

∼ η if
 1/2
kXk+1 k
kXk+1 − Xk k ≤ 2η . (5.44)
kXk−1 k

This is essentially the same test as (4.25), bearing in mind that in the latter bound
c = kS −1 k/2 ≈ kXk−1 k/2. This bound should overcome the problem of (5.41) of
stopping one iteration too late, but unlike (5.43) with p = 1, 2 it takes no explicit
account of rounding error effects. A test of this form has been suggested by Benner
and Quintana-Ortı́ [55, ]. The experiments in Section 5.8 give further insight.
For general sign iterations, intuitively appealing stopping criteria can be devised
based on the fact that trace(sign(A)) is an integer, but these are of little practical
use; see Problem 5.16.
The upper bound in (5.40) shows that kA − Xk k/kXk k ≤ kXk2 − Ik and hence
suggests stopping when
kXk2 − Ik ≤ η. (5.45)
This test is suitable for iterations that already form Xk2 , such as the Schulz iteration
(5.22). Note, however, that the error in forming f l(Xk2 − I) is bounded at best by
cn ukXk k2 ≈ cn ukSk2 , so when kSk is large it may not be possible to satisfy (5.45),
and a more suitable test is then
kXk2 − Ik
≤ η.
kXk k2

5.7. Numerical Stability of Sign Iterations

The question of the stability of sign iterations, where stability is defined in Defini-
tion 4.17, has a particularly nice answer for all the iterations of interest.

Theorem 5.13 (stability of sign iterations). Let S = sign(A), where A ∈ Cn×n has
no pure imaginary eigenvalues. Let Xk+1 = g(Xk ) be superlinearly convergent to
sign(X0 ) for all X0 sufficiently close to S and assume that g is independent of X0 .
Then the iteration is stable, and the Fréchet derivative of g at S is idempotent and is
given by Lg (S, E) = L(S, E) = 21 (E − SES), where L(S) is the Fréchet derivative of
the matrix sign function at S.
Proof. Since the sign function is idempotent, stability, the idempotence of Lg ,
and the equality of Lg (S) and L(S), follow from Theorems 4.18 and 4.19. The formula
for L(S, E) is obtained by taking N = I in Theorem 5.3.
124 Matrix Sign Function

Theorem 5.13 says that the Fréchet derivative at S is the same for any superlinearly
convergent sign iteration and that this Fréchet derivative is idempotent. Unbounded
propagation of errors near the solution is therefore not possible for any such iteration.
The constancy of the Fréchet derivative is not shared by iterations for all the functions
in this book, as we will see in the next chapter.
Turning to limiting accuracy (see Definition 4.20), Theorem 5.13 yields kLg (S, E)k ≤
1
2 (1 + kSk2 )kEk, so an estimate for the limiting accuracy of any superlinearly conver-
gent sign iteration is kSk2 u. Hence if, for example, κ(S) = kSk2 ≤ u−1/2 , then we
can hope to compute the sign function to half precision.
If S commutes with E then Lg (S, E) = 0, which shows that such errors E are
eliminated by the iteration to first order. To compare with what convergence con-
siderations say about E, note first that in all the sign iterations considered here
the matrix whose sign is being computed appears only as the starting matrix and
not within the iteration. Hence if we start the iteration at S + E then the iter-
ation converges to sign(S + E), for sufficiently small kEk (so that the sign exists
and any convergence conditions are satisfied). Given that S has the form (5.1),
any E commuting with S has the form Z diag(F11 , F22 )Z −1 , so that sign(S + E) =
Z sign(diag(−Ip + F11 , Iq + F22 ))Z −1 . Hence there is an ǫ such that for all kEk ≤ ǫ,
sign(S + E) = S. Therefore, the Fréchet derivative analysis is consistent with the
convergence analysis.
Of course, to obtain a complete picture, we also need to understand the effect
of rounding errors on the iteration prior to convergence. This effect is surprisingly
difficult to analyze, even though the iterative methods are built purely from matrix
multiplication and inversion. The underlying behaviour is, however, easy to describe.
Suppose, as discussed above, that we have an iteration for sign(A) that does not
contain A, except as the starting matrix. Errors on the (k − 1)st iteration can be
accounted for by perturbing Xk to Xk + Ek . If there are no further errors then
(regarding Xk + Ek as a new starting matrix) sign(Xk + Ek ) will be computed. The
error thus depends on the conditioning of Xk and the size of Ek . Since errors will
in general occur on each iteration, the overall error will be a complicated function of
κsign (Xk ) and Ek for all k.
We now restrict our attention to the Newton iteration (5.16). First, we note that
the iteration can be numerically unstable: the relative error is not always bounded by
a modest multiple of the condition number κsign (A), as is easily shown by example (see
the next section). Nevertheless, it generally performs better than might be expected,
given that it inverts possibly ill conditioned matrices. We are not aware of any
published rounding error analysis for the computation of sign(A) via the Newton
iteration.
Error analyses aimed at the application of the matrix sign function to invariant
subspace computation (Section 2.5) are given by Bai and Demmel [29, ] and By-
ers, He, and Mehrmann [89, ]. These analyses show that the matrix sign function
may be more ill conditioned than the problem of evaluating the invariant subspaces
corresponding to eigenvalues in the left half-plane and right half-plane. Neverthe-
less, they show that when Newton’s method is used to evaluate the sign function the
computed invariant subspaces are usually about as good as those computed by the
QR algorithm. In other words, the potential instability rarely manifests itself. The
analyses are complicated and we refer the reader to the two papers for details.
In cases where the matrix sign function approach to computing an invariant sub-
space suffers from instability, iterative refinement can be used to improve the com-
5.8 Numerical Experiments and Algorithm 125

Table 5.2. Number of iterations for scaled Newton iteration. The unnamed matrices are
(quasi)-upper triangular with normal (0, 1) distributed elements in the upper triangle.

Scaling
Matrix none determinantal spectral norm
Lotkin 25 9 8 9
Grcar 11 9 9 15
ˆ 1 (j/n)1000
˜
A(j: j + 1, j: j + 1) = −(j/n)1000 1
24 16 19 19
ajj = 1 + 1000i(j − 1)/(n − 1) 24 16 22 22
a11 = 1000, ajj ≡ 1, j ≥ 2 14 12 6 10
a11 = 1 + 1000i, ajj ≡ 1, j ≥ 2 24 22 8 19

puted subspace [29, ]. Iterative refinement can also be used when the sign function
is used to solve algebraic Riccati equations (as described in Section 2.4) [88, ].
Finally, we note that all existing numerical stability analysis is for the unscaled
Newton iteration. Our experience is that scaling tends to improve stability, not worsen
it.

5.8. Numerical Experiments and Algorithm

We present some numerical experiments to illustrate the theory of the previous three
sections and to give further insight into the choice of iteration, acceleration scheme,
and stopping criterion. In all the tests, scaling was used as long as the relative change
δk = kXk − Xk−1 k∞ /kXk k∞ exceeded 10−2 ; thereafter µk ≡ 1 and, where relevant,
µk is not shown in the tables.
First, we consider the effects of scaling. For a variety of matrices we ran the
Newton iteration (5.34) with no scaling and with the scalings (5.35)–(5.37), with the
2-norm used for norm scaling. We recorded how many iterations are required to
produce an error kS − Xk k∞ /kSk∞ ≤ 5 × 10−14 . The matrices are as follows:

1. The 8×8 Lotkin matrix, MATLAB’s gallery(’lotkin’,8): badly conditioned

with many negative eigenvalues of small magnitude.

2. The 25 × 25 Grcar matrix, gallery(’grcar’,25): a Toeplitz matrix with sen-

sitive eigenvalues.

3. 25 × 25 (quasi-) upper triangular matrices with elements in the upper triangle

(outside the diagonal blocks) from the normal (0,1) distribution.

Table 5.2 reports the results. The Lotkin matrix is a typical example of how scaling
can greatly reduce the number of iterations. The Grcar example shows how norm
scaling can perform poorly (indeed being worse than no scaling). The third matrix
(real) and fourth matrix (complex) have eigenvalues on a line with real part 1 and
imaginary parts between 0 and 1000. Here, spectral scaling and norm scaling are
both poor. The fifth and sixth matrices, again real and complex, respectively, have
eigenvalues all equal to 1 except for one large outlier, and they are bad cases for
determinantal scaling.
Table 5.3 illustrates the convergence results in Theorems 5.10 and 5.11 by showing
the behaviour of the Newton iteration with spectral scaling for J(2) ∈ R16×16 , which
126 Matrix Sign Function

Table 5.3. Newton iteration with spectral scaling for Jordan block J(2) ∈ R16×16 .

kS − Xk k∞ kXk2 − Ik∞
k δk µk (5.41) (5.44)
kSk∞ kXk k2∞
1 2.5e-1 1.8e+0 3.6e-1 5.0e-1
2 2.5e-2 2.2e-1 4.8e-2 1.0e0
3 3.0e-4 2.5e-2 6.0e-4 1.0e0
4 0 3.0e-4 0 1.0e0
√ √
5 0 0 0

Table 5.4. Newton iteration with determinantal scaling for random A ∈ R16×16 with κ2 (A) =
1010 ; κsign (A) = 3 × 108 , kSkF = 16.

kS − Xk k∞ kXk2 − Ik∞
k δk µk (5.41) (5.44)
kSk∞ kXk k2∞
1 4.3e3 1.0e0 1.1e-1 1.0e5
2 1.5e1 2.8e2 1.3e-1 6.8e-3
3 1.9e0 6.3e0 5.9e-2 1.4e-1
4 2.1e-1 1.7e0 2.1e-2 6.1e-1
5 6.4e-2 2.3e-1 4.3e-3 9.5e-1
6 2.0e-3 6.2e-2 1.6e-4 9.8e-1
7 4.1e-6 2.0e-3 3.3e-7 1.0e0
8 2.1e-9 4.1e-6 8.9e-13
√
9 2.1e-9 1.1e-11 3.2e-17
√ √
10 2.1e-9 1.5e-15 3.5e-17

is a Jordan block with eigenvalue 2. Here and below the last two columns of the table
indicate with a tick iterations on which the convergence conditions (5.41) and (5.44)
are satisfied for the ∞-norm, with η = n1/2 u. In Theorem 5.11, d = 1 and p = 4,
and indeed Xd+p−1 = X4 = sign(J(2)). At the start of the first iteration, µ0 X0 has
eigenvalues 1, and the remaining four iterations remove the nonnormal part; it is easy
to see that determinantal scaling gives exactly the same results.
Table 5.4 reports 12 iterations for a random A ∈ R16×16 with κ2 (A) = 1010
generated in MATLAB by gallery(’randsvd’,16,1e10,3). Determinantal scaling
was used. Note that the relative residual decreases significantly after the error has
stagnated. The limiting accuracy of kSk22 u is clearly not relevant here, as the iterates
do not approach S sufficiently closely.
Both these examples confirm that the relative change δk+1 is a good estimate
of the relative error in Xk (compare the numbers in the third column with those
immediately to the northwest) until roundoff starts to dominate, but thereafter the
relative error and relative change can behave quite differently.
Finally, Table 5.5 gives examples with large kSk. The matrix is of the form
A = QT QT , where Q is a random orthogonal matrix and T ∈ R16×16 is generated as
an upper triangular matrix with normal (0,1) distributed elements and tii is replaced
by d|tii | for i = 1: 8 and by −d|tii | for i = 9: 16. As d is decreased the eigenvalues
of A approach the origin (and hence the imaginary axis). Determinantal scaling was
used and we terminated the iteration when the relative error stopped decreasing sig-
5.8 Numerical Experiments and Algorithm 127

Table 5.5. Newton iteration with determinantal scaling for random A ∈ R16×16 with real
eigenvalues parametrized by d.

kS − Xk k∞
d no. iterations mink kAk2 κ2 (A) kSk2 κsign (A)
kSk∞
1 6 2.7e-13 6.7 4.1e3 1.3e2 4.7e3
3/4 6 4.1e-10 6.5 5.7e5 5.4e3 6.5e5
1/2 6 2.6e-6 6.2 2.6e8 3.9e5 6.5e7
1/3 3 7.8e-1 6.4 2.6e15 7.5e11 3.9e7

nificantly. This example shows that the Newton iteration can behave in a numerically
unstable way: the relative error can greatly exceed κsign (A)u. Note that the limiting
accuracy kSk22 u provides a good estimate of the relative error for the first three values
of d.
Our experience indicates that (5.44) is the most reliable termination criterion,
though on badly behaved matrices such as those in Table 5.5 no one test can be relied
upon to terminate at the “right moment”, if at all.
Based on this and other evidence we suggest the following algorithm based on the
scaled Newton iteration (5.34).

Algorithm 5.14 (Newton algorithm for matrix sign function). Given a nonsingular
A ∈ Cn×n with no pure imaginary eigenvalues this algorithm computes X = sign(A)
using the scaled Newton iteration. Two tolerances are used: a tolerance tol cgce
for testing convergence and a tolerance tol scale for deciding when to switch to the
unscaled iteration.

1 X0 = A; scale = true
2 for k = 1: ∞
3 Yk = Xk−1
4 if scale
5 Set µk to one of the scale factors (5.35)–(5.37).
6 else
7 µk = 1
8 end
9 Xk+1 = 12 (µk Xk + µ−1k Yk )
10 δk+1 = kXk+1 − Xk kF /kXk+1 kF
11 if scale = true and δk+1 ≤ tol scale, scale = false, end
12 if kXk+1 − Xk kF ≤ (tol cgcekXk+1 k/kYk k)1/2 or
(δk+1 > δk /2 and scale = false)
13 goto line 16
14 end
15 end
16 X = Xk+1

Cost: 2kn3 flops, where k iterations are used.

The algorithm uses the unscaled Newton iteration once the relative change in the
iterates is less than tol scale. A value of tol scale safely less than 1 is intended and
the motivation is to avoid the (nonoptimal) scaling parameters interfering with the
128 Matrix Sign Function

quadratic convergence once the convergence has set in. The convergence test is (5.44)
combined with the requirement to stop if, in the final convergence phase, δk has not
decreased by at least a factor 2 during the previous iteration (which is a sign that
roundoff errors are starting to dominate).
We have left the choice of scale factor at line 5 open, as the best choice will depend
on the class of problems considered.

5.9. Best L∞ Approximation

Most applications of the matrix sign function involve nonnormal matrices of small
to medium size. An exception is the application in lattice quantum chromodynamics
(QCD) described in Section 2.7, where the action on a vector of the sign of a large,
sparse, Hermitian matrix is required. For Hermitian A, approximating sign(A) re-
duces to approximating sign(x) at the eigenvalues of A, which is a scalar problem
on the real axis. The full range of scalar approximation techniques and results can
therefore be brought into play. In particular, we can use best L∞ rational approxima-
tions. For the sign function and the interval [−δmax , −δmin ] ∪ [δmin , δmax ] an explicit
formula for the best L∞ approximation is known. It follows from a corresponding
result for the inverse square root. The result is phrased in terms of elliptic functions.
The Jacobi elliptic function sn(w; κ) = x is defined implicitly by the elliptic integral
Z x
1
w= p dt
0 (1 − t )(1 − κ2 t2 )
2

and the complete elliptic integral (for the modulus κ) is defined by

Z 1
1
K= p dt.
0 (1 − t2 )(1 − κ2 t2 )

Theorem 5.15 (Zolotarev, 1877).

(a) The best L∞ approximation re from Rm−1,m to x−1/2 on the interval [1,
(δmax /δmin )2 ] is
Qm−1
j=1 (x + c2j )
re(x) = D Qm ,
j=1 (x + c2j−1 )

where
sn2 (jK/(2m); κ)
cj = ,
1 − sn2 (jK/(2m); κ)
κ = (1 − (δmin /δmax )2 )1/2 , and K is the complete elliptic integral for the modulus κ.
The constant D is determined by the condition
√ √
max 2
(1 − x re(x)) = − min 2
(1 − x re(x)),
x∈[1,(δmin /δmax ) )] x∈[1,(δmin /δmax ) )]

and the extrema occur at xj = dn−2 (jK/(2m)), j = 0: 2m, where dn2 (w; κ) = 1 −
κ2 sn2 (w; κ).
(b) The best L∞ approximation r from R2m−1,2m to sign(x) on the interval
r((x/δmin )2 ), where re is defined in
[−δmax , −δmin ] ∪ [δmin , δmax ] is r(x) = (x/δmin )e
(a).
5.10 Notes and References 129

sign(x)
−1
r(x)
−2 −1.5 −1 −0.5 0 0.5 1 1.5 2

−0.9998 1.0000

−1.0000 0.9998
−2 −1.5 −1 1 1.5 2

Figure 5.2. Best L∞ approximation r(x) to sign(x) from R3,4 on [−2, −1] ∪ [1, 2]. The lower
two plots show r(x) in particular regions of the overall plot above.

Figure 5.2 plots the best L∞ approximation to sign(x) from R3,4 on [−2, −1] ∪
[1, 2], and displays the characteristic equioscillation property of the error, which has
maximum magnitude about 10−4 . In the QCD application δmin and δmax are chosen
so that the spectrum of the matrix is enclosed and r is used in partial fraction form.

5.10. Notes and References

The matrix sign function was introduced by Roberts [496] in 1971 as a tool for model
reduction and for solving Lyapunov and algebraic Riccati equations. He defined the
sign function as a Cauchy integral and obtained the integral (5.3). Roberts also
proposed the Newton iteration (5.16) for computing sign(A) and proposed scaling the
iteration, though his scale parameters are not as effective as the ones described here.
Interest in the sign function grew steadily in the 1970s and 1980s, initially among
engineers and later among numerical analysts. Kenney and Laub give a thorough
survey of the matrix sign function and its history in [347, ].
The attractions of the concise representation sign(A) = A(A2 )−1/2 in (5.2) were
pointed out by Higham [273, ], though the formula can be found in earlier work
of Tsai, Shieh, and Yates [576, 1988].
Theorem 5.2 is from Higham, Mackey, Mackey, and Tisseur [283, ].
Theorems 5.3 and 5.7 are due to Kenney and Laub [342, ]. The expression
(5.10) and upper bound (5.11) for the matrix sign function condition number are from
Higham [273, ]. Theorem 5.4 is a refined version of a result of Kenney and Laub
[342, ]. Another source of perturbation results for the matrix sign function is Sun
[548, ].
The Schur method, Algorithm 5.5, is implemented in function signm of the Matrix
Computation Toolbox [264] (on which signm in the Matrix Function Toolbox is based)
but appears here in print for the first time.
For more on the recursions related to (5.26), and related references, see Chapter 8.
130 Matrix Sign Function

It is natural to ask how sharp the sufficient condition for convergence kI −A2 k < 1
in Theorem 5.8 (a) is for ℓ > m and what can be said about convergence for ℓ < m−1.
These questions are answered experimentally by Kenney and Laub [343, ], who
give plots showing the boundaries of the regions of convergence of the scalar iterations
in C.
The principal Padé iterations for the sign function were first derived by Howland
[302, ], though for even k his iteration functions are the inverses of those given
here. Iannazzo [307, ] points out that these iterations can be obtained from the
general König family (which goes back to Schröder [509, ], [510, ]) applied to
the equation x2 −1 = 0. Parts (b)–(d) of Theorem 5.9 are from Kenney and Laub [345,
]. Pandey, Kenney, and Laub originally obtained the partial fraction expansion
(5.30), for even k only, by applying Gaussian quadrature to an integral expression for
h(ξ) in (5.26) [457, ]. The analysis leading to (5.31) is from Kenney and Laub
[345, ].
Theorem 5.10 is due to Kenney and Laub [344, ], and the triangular matrices
in Table 5.2 are taken from the same paper.
Theorem 5.11 is due to Barraud [44, , Sec. 4], but, perhaps because his paper
is written in French, his result went unnoticed until it was presented by Kenney and
Laub [344, , Thm. 3.4].
Lemma 5.12 collects results from Kenney, Laub, and Papadopoulos [350, ]
and Pandey, Kenney, and Laub [457, ].
The spectral scaling (5.36) and norm scaling (5.37) were first suggested by Barraud
[44, ], while determinantal scaling (5.35) is due to Byers [88, ].
Kenney and Laub [344, ] derive a “semioptimal” scaling for the Newton iter-
ation that requires estimates of the dominant eigenvalue (not just its modulus, i.e.,
the spectral radius) of Xk and of Xk−1 . Numerical experiments show this scaling to
be generally at least as good as the other scalings we have described. Semioptimal
scaling does not seem to have become popular, probably because it is more delicate
to implement than the other scalings and the other scalings typically perform about
as well in practice.
Theorem 5.13 on the stability of sign iterations is new. Indeed we are not aware
of any previous analysis of the stability of sign iterations.
Our presentation of Zolotarev’s Theorem 5.15 is based on that in van den Eshof,
Frommer, Lippert, Schilling, and Van der Vorst [585, ] and van den Eshof [586,
]. In the numerical analysis literature this result seems to have been first pointed
out by Kenney and Laub [347, , Sec. III]. Theorem 5.15 can also be found in
Achieser [1, , Sec. E.27], Kennedy [338, ], [339, ], and Petrushev and
Popov [470, , Sec. 4.3].
A “generalized Newton sign iteration” proposed by Gardiner and Laub [205, ]
has the form
1
Xk+1 = (Xk + BXk−1 B), X0 = A.
2
If B is nonsingular this is essentially the standard Newton iteration applied to B −1 A
and it converges to B sign(B −1 A). For singular B, convergence may or may not
occur and can be at a linear rate; see Bai, Demmel, and Gu [31, ] and Sun and
Quintana-Ortı́ [550, ]. This iteration is useful for computing invariant subspaces
of matrix pencils A − λB (generalizing the approach in Section 2.5) and for solving
generalized algebraic Riccati equations.
Problems 131

Problems
5.1. Show that sign(A) = A for any involutory matrix.
5.2. How are sign(A) and sign(A−1 ) related?
5.3. Derive the integral formula (5.3) from (5.2) by using the Cauchy integral formula
(1.12).
5.4. Show that sign(A) = (2/π) limt→∞ tan−1 (tA).
5.5. Can  
−1 1 1/2
A= 0 1 −1 
0 0 1
be the sign of some matrix?
5.6. Show that the geometric mean A # B of two Hermitian positive definite matrices
A and B satisfies
   
0 A#B 0 B
= sign .
(A # B)−1 0 A−1 0

5.7. (Kenney and Laub [342, ]) Verify that for A ∈ R2×2 the matrix sign
decomposition (5.5) is given as follows. If det(A) > 0 and trace(A) 6= 0 then
S = sign(trace(A))I and N = sign(trace(A))A; if det(A) < 0 then



S = µ A − det(A)A−1 , N = µ A2 − det(A)I ,

where
−1/2
µ = − det(A − det(A)A−1 ) ;
otherwise S is undefined.
5.8. Show that the Newton iteration (5.16) for the matrix sign function can be derived
by applying Newton’s method to the equation X 2 = I.
5.9. By expanding the expression sign(S + E) = (S + E)((S + E)2 )−1/2 from (5.2),
show directly that the Fréchet derivative of the matrix sign function at S = sign(S)
is given by L(S, E) = 12 (E − SES).
5.10. Consider the scalar Newton sign iteration xk+1 = 21 (xk + x−1
k ). Show that if
x0 = coth θ0 then xk = coth 2k θ0 . Deduce a convergence result.
5.11. (Schroeder [511, ]) Investigate the behaviour of the Newton iteration (5.16)
for scalar, pure imaginary x0 . Hint: let x0 = ir0 ≡ −i cot(πθ0 ) and work in θ
coordinates.
5.12. Halley’s iteration for solving f (x) = 0 is [201, ]

fk /fk′
xk+1 = xk − ,
1 − 21 fk fk′′ /(fk′ )2

where fk , fk′ , and fk′′ denote the values of f and its first two derivatives at xk . Show
that applying Halley’s iteration to f (x) = x2 − 1 yields the iteration function f1,1 in
Table 5.1.
132 Matrix Sign Function

5.13. (Byers [88, ]) Show that determinantal scaling µ = | det(X)|−1/n mini-
mizes d(µX), where
Xn
d(X) = (log |λi |)2
i=1

and the λi are the eigenvalues of X. Show also that d(X) = 0 if and only if the
spectrum of X lies on the unit circle and that d(X) is an increasing function of
|1 − |λi || for each eigenvalue λi .
5.14. Consider the Newton iteration (5.34), with determinantal scaling (5.35) and
spectral scaling (5.36). Show that with both scalings the iteration converges in at
most two iterations (a) for scalars and (b) for any real 2 × 2 matrix.
5.15. (Higham, Mackey, Mackey, and Tisseur [283, ]) Suppose that sign(A) = I
and A2 = I + E, where kEk < 1, for some consistent norm. Show that
kEk
kA − Ik ≤ p < kEk.
1 + 1 − kEk

How does this bound compare with the upper bound in (5.40)?
5.16. Discuss the pros and cons of terminating an iteration Xk+1 = g(Xk ) for the
matrix sign function with one of the tests

| trace(Xk2 ) − n| ≤ η, (5.46)
| trace(Xk ) − round(trace(Xk ))| ≤ η, (5.47)

where round(x) denotes the nearest integer to x.

5.17. (Byers [88, ]) The matrix
 ∗ 
A G
W = , F = F ∗, G = G∗ ,
F −A

arising in (2.14) in connection with the Riccati equation is Hamiltonian,

 that is, it
satisfies the condition that JW is Hermitian, where J = −I0n I0n . Show that the
Newton iteration for sign(W ) can be written in such a way that only Hermitian
matrices need to be inverted. The significance of this fact is that standard algorithms
or software for Hermitian matrices can then be used, which halves the storage and
computational costs compared with treating W as a general matrix.

The sign function of a square matrix can be defined in terms of a contour integral
or as the result of an iterated map Zr+1 = 12 (Zr + Zr−1 ).
Application of this function enables a matrix to be decomposed into
two components whose spectra lie on opposite sides of the imaginary axis.
— J. D. ROBERTS, Linear Model Reduction and Solution of the
Algebraic Riccati Equation by Use of the Sign Function (1980)

The matrix sign function method is an elegant and,

when combined with defect correction,
effective numerical method for the algebraic Riccati equation.
— VOLKER MEHRMANN, The Autonomous Linear Quadratic Control Problem:
Theory and Numerical Solution (1991)
Chapter 6
Matrix Square Root

The matrix square root is one of the most commonly occurring matrix functions,
arising most frequently in the context of symmetric positive definite matrices. The
key roles that the square root plays in, for example, the matrix sign function (Chap-
ter 5), the definite generalized eigenvalue problem (page 35), the polar decomposition
(Section 2.6 and Chapter 8), and the geometric mean (Section 2.10), make it a useful
theoretical and computational tool. The rich variety of methods for computing the
matrix square root, with their widely differing numerical stability properties, are an
interesting subject of study in their own right.
We will almost exclusively be concerned with the principal square root, A1/2 .
Recall from Theorem 1.29 that for A ∈ Cn×n with no eigenvalues on R− , A1/2 is the
unique square
√ root X of A whose spectrum lies in the open right half-plane. We will
denote by A an arbitrary, possibly nonprincipal square root.
We note the integral representation
Z ∞
2
A1/2 = A (t2 I + A)−1 dt, (6.1)
π 0

which is a special case of (7.1) in the next chapter. The integral can be deduced from
that for the matrix sign function (see Problem 6.1).
This chapter begins with analysis of the conditioning of the matrix square root
and the sensitivity of the relative residual. Then a Schur method, and a version work-
ing entirely in real arithmetic, are described. Newton’s method and several variants
follow, with a stability analysis revealing that the variants do not suffer the instability
that vitiates the Newton iteration. After a discussion of scaling, numerical experi-
ments are given to provide insight into the analysis. A class of coupled iterations
obtained via iterations for the matrix sign function are derived and their stability
proved. Linearly convergent iterations for matrices that are “almost diagonal”, as
well as for M-matrices, are analyzed, and a preferred iteration for Hermitian posi-
tive definite matrices is given. The issue of choosing from among the many square
roots of a given matrix is addressed by considering how to compute a small-normed
square root. A brief comparison of the competing methods is given. Finally, appli-
cations of involutory matrices, and some particular involutory matrices with explicit
representations, are described.

6.1. Sensitivity and Conditioning

To analyze the sensitivity of the matrix square root we need to assume that we
keep fixed the choice of branches of the square root function in the neighbourhoods
of the eigenvalues of the matrix of interest, since otherwise the square root is not

133
134 Matrix Square Root

even continuous. We will concentrate on primary square roots, since a nonprimary

square root belongs to a parametrized family of square roots (see Theorem 1.26) and
so must be regarded as infinitely √ sensitive. By Theorem 3.5, the Fréchet derivative
L = L(A, E) of X = f (A) = A is the solution of XL + LX = E, which is the
inverse of the Fréchet derivative of the function X 2 . Using (B.16), we can rewrite this
equation as (I ⊗ X + X T ⊗ I) vec(L) = vec(E), from which it can be deduced, using
(3.7), that kLkF = k(I ⊗ X + X T ⊗ I)−1 k2 . Hence, by Theorem 3.1, the Frobenius
norm (relative) condition number of the matrix square root at A is

k(I ⊗ X + X T ⊗ I)−1 k2 kAkF

κsqrt (X) = , (6.2)
kXkF

where the argument of κsqrt denotes the particular square root under consideration.
It follows that
1 kAkF
κsqrt (X) ≥ , (6.3)
mini,j=1:n |µi + µj | kXkF
√
where the µj are the eigenvalues of X = A (and this inequality can also be obtained
from Theorem 3.14). This inequality is interesting because it reveals two distinct
situations in which κsqrt must be large. The first situation is when A (and hence X)
has an eigenvalue of small modulus. The second situation is when the square root is
the principal square root and a real A has a pair of complex conjugate eigenvalues
1/2
close to the negative real axis: λ = rei(π−ǫ) (0 < ǫ ≪ 1) and λ. Then |λ1/2 + λ | =
r1/2 |ei(π−ǫ)/2 + e−i(π−ǫ)/2 | = r1/2 |e−iǫ/2 − eiǫ/2 | = r1/2 O(ǫ). In this latter case A is
close to a matrix for which the principal square root is not defined.
If A is normal and X is normal (as is any primary square root of a normal A)
then, either directly from (6.2) or from Corollary 3.16, we have equality in (6.3).
The formula for κsqrt allows us to identify the best conditioned square root of a
Hermitian positive definite matrix. As usual, κ(X) = kXk kX −1 k.

Lemma 6.1. If A ∈ Cn×n is Hermitian positive definite and X is any primary square
root of A then
1/2
kA−1 k2 kAkF
κsqrt (A1/2 ) = ≤ κsqrt (X).
2 kA1/2 kF
Moreover,
1 1
3/2
κF (A1/2 ) ≤ κsqrt (A1/2 ) ≤ κF (A1/2 ).
2n 2

Proof. Let A have eigenvalues 0 < λn ≤ λn−1 ≤ · · · ≤ λ1 . For X = A1/2 ,

1/2
A and X are normal so (6.3) is an equality and mini,j=1:n |µi + µj | = 2λn =
1/2
2/kA−1 k2 , which gives the expression for κsqrt (A1/2 ). Any other primary square
p
root X has eigenvalues µj with moduli λj , so the upper bound of κsqrt (X) follows
√ Pn
from mini,j=1:n |µi + µj | ≤ 2 λn together with the fact that kXk2F = i=1 λi is the
same for all primary (and hence Hermitian) square roots of A. The upper and lower
bounds for κsqrt (A1/2 ) follow using standard norm inequalities [276, 2002, Chap. 6].

Staying with positive definite matrices for the moment, the next result gives an
elegant bound for the difference between the principal square roots of two matrices.
6.2 Schur Method 135

Theorem 6.2. If A, B ∈ Cn×n are Hermitian positive definite then for any unitarily
invariant norm
1
kA1/2 − B 1/2 k ≤ kA − Bk,
λmin (A)1/2 + λmin (B)1/2

where λmin denotes the smallest eigenvalue.

Proof. This is a special case of a result of van Hemmen and Ando [591, ,
Prop. 3.2]; see also Bhatia [62, ].
e = X + E be an approximation to a square root X of A ∈ Cn×n , where
Let X
e 2 = A + XE + EX + E 2 , which leads to the relative residual
kEk ≤ ǫkXk. Then X
bound
e 2k
kA − X
≤ (2ǫ + ǫ2 )α(X),
kAk
where
kXk2 kXk2
α(X) = = ≥ 1. (6.4)
kAk kX 2 k
The quantity α(X) can be regarded as a condition number for the relative residual
of X; if it is large then a small perturbation of X (such as f l(X)—the rounded
square root) can have a relative residual much larger than the size of the relative
perturbation. An important conclusion is that we cannot expect a numerical method
to do better than provide a computed square root X b with relative residual of order
b
α(X)u, where u is the unit roundoff. Where there is a choice of square root, one of
minimal norm is therefore to be preferred. It is easy to show that

κ(X)
≤ α(X) ≤ κ(X).
κ(A)

Thus a large value of α(X) implies that X is ill conditioned, and if A is well condi-
tioned then α(X) ≈ κ(X). If X is normal then α(X) = 1 in the 2-norm.

6.2. Schur Method

Let A ∈ Cn×n be nonsingular and let f (A) denote any primary square root of A.
Given the Schur decomposition A = QT Q∗ , where Q is unitary and T is upper
triangular, f (A) = Qf (T )Q∗ , so computing square roots of general A reduces to
computing square roots U = f (T ) of upper triangular matrices, which are themselves
triangular. The (i, i) and (i, j) (j > i) elements of the equation U 2 = T can be written

u2ii = tii ,
j−1
X
(uii + ujj )uij = tij − uik ukj . (6.5)
k=i+1

We can compute the diagonal of U and then solve for the uij either a superdiagonal at
a time or a column at a time. The process cannot break down, because 0 = uii +ujj =
f (tii ) + f (tjj ) is not possible, since the tii are nonzero and f , being a primary matrix
function, maps equal tii to the same square root. We obtain the following algorithm.
136 Matrix Square Root

Algorithm√6.3 (Schur method). Given a nonsingular

√ A ∈ Cn×n this algorithm com-
putes X = A via a Schur decomposition, where · denotes any primary square root.
∗
1 Compute
√ a (complex) Schur decomposition A = QT Q .
2 uii = tii , i = 1: n
3 for j = 2: n
4 for i = j − 1: −1: 1
Pj−1
tij − k=i+1 uik ukj
5 uij =
uii + ujj
6 end
7 end
8 X = QU Q∗

Cost: 25n3 flops for the Schur decomposition plus n3/3 for U and 3n3 to form X:
28 31 n3 flops in total.
Algorithm 6.3 generates all the primary square roots of A as different choices of
√ 1/2 √
sign in uii = tii ≡ ±tii are used, subject to the restriction that tii = tjj ⇒ tii =
√
tjj .
If A is singular with a semisimple zero eigenvalue of multiplicity k then Algo-
rithm 6.3 can be adapted by ordering the Schur decomposition so that the zero eigen-
values are in the trailing k diagonal elements of T . Then T must have the structure

n−k k
 
n−k T11 T12
T = , (6.6)
k 0 0

with T11 nonsingular. Indeed if some element of the trailing k × k block of T were
nonzero then the rank of T would exceed n − k, but 0 being a semisimple eigenvalue
of multiplicity k implies rank(T ) = n − k. It is clear that any primary square root has
the same block structure as T and that Algorithm 6.3 computes such a square root
provided that we set uij = 0 when i > n − k and j > n − k, which are the cases where
the algorithm would otherwise incur division by zero. The behaviour of the algorithm
for singular A without reordering to the form (6.6) is examined in Problem 6.5.
If A is real but has some nonreal eigenvalues then Algorithm 6.3 uses complex
arithmetic. This is undesirable, because complex arithmetic is more expensive than
real arithmetic and also because rounding errors may cause a computed result to be
produced with nonzero imaginary part. By working with a real Schur decomposition
complex arithmetic can be avoided.
Let A ∈ Rn×n have the real Schur decomposition A = QRQT , where Q is orthog-
onal and R is upper quasi-triangular with 1 × 1 and 2 × 2 diagonal blocks. Then
f (A) = Qf (R)QT , where U = f (R) is upper quasi-triangular with the same block
structure as R. The equation U 2 = R can be written

Uii2 = Rii ,
j−1
X
Uii Uij + Uij Ujj = Rij − Uik Ukj . (6.7)
k=i+1

Once the diagonal blocks Uii have been computed (6.7) provides a way to compute
the remaining blocks Uij a block superdiagonal or a block column at a time. The
condition for the Sylvester equation (6.7) to have a unique solution Uij is that Uii
6.2 Schur Method 137

and −Ujj have no eigenvalue in common (see (B.20)), and this is guaranteed for any
primary square root when A is nonsingular. When neither Uii nor Ujj is a scalar,
(6.7) can solved by writing it in the form

T Pj−1

(I ⊗ Uii + Ujj ⊗ I) vec(Uij ) = vec Rij − k=i+1 Uik Ukj ,

which is a linear system Ax = b of order 2 or 4 that can by solved by Gaussian

elimination with partial pivoting.
√
We now consider the computation of Rii for 2 × 2 blocks Rii , which necessarily
have distinct complex conjugate eigenvalues.

Lemma 6.4. Let A ∈ R2×2 have distinct complex conjugate eigenvalues. Then A
has four square roots, all primary functions of A. Two of them are real, with complex
conjugate eigenvalues, and two are pure imaginary, with eigenvalues that are not
complex conjugates.

Proof. Since A has distinct eigenvalues θ ± iµ, Theorem 1.26 shows that A has
four square roots, all of them functions of A. To find them let
 
−1 1 0
Z AZ = diag(λ, λ) = θI + iµK, K= .
0 −1

Then
A = θI + µW,

where W = iZKZ −1 , and since θ, µ ∈ R it follows that W ∈ R2×2 .

If (α + iβ)2 = θ + iµ, then the four square roots of A are given by X = ZDZ −1 ,
where D = ± diag(α+iβ, ±(α−iβ)), that is, D = ±(αI +iβK) or D = ±(αK +iβI) =
±i(βI − iαK). Thus
X = ±(αI + βW ),

that is, two real square roots with eigenvalues ±(α + iβ, α − iβ); or

X = ±i(βI − αW ),

that is, two pure imaginary square roots with eigenvalues ±(α + iβ, −α + iβ).
1/2
The proof of the lemma gives a way to construct Rii . Writing
 
r r12
Rii = 11 ,
r21 r22

the eigenvalues of Rii are θ ± iµ, where

1 1
1/2
θ= (r11 + r22 ), µ= −(r11 − r22 )2 − 4r21 r12 . (6.8)
2 2

We now require α and β such that (α + iβ)2 = θ + iµ. A stable way to compute them
is as follows.
138 Matrix Square Root

Algorithm 6.5. This algorithm computes the square root α+iβ of θ+iµ with α ≥ 0.

1 if θ = 0 and µ = 0, α = 0, β = 0, quit, end

1/2
2 t = |θ| + (θ2 + µ2 )1/2 /2
3 if θ ≥ 0
4 α = t, β = µ/(2α)
5 else
6 β = t, α = µ/(2β)
7 end

Finally, the real square roots of Rii are obtained from

1

Uii = ± αI + (Rii − θI)
2α
 1 1 
α+ (r11 − r22 ) r12
= ± 4α 2α . (6.9)
1 1
r21 α− (r11 − r22 )
2α 4α
Before giving an algorithm, we pause to summarize our theoretical findings.

Theorem 6.6. Let A ∈ Rn×n be nonsingular. If A has a real negative eigenvalue then
A has no real square roots that are primary functions of A. If A has no real negative
eigenvalues, then there are precisely 2r+c real primary square roots of A, where r is
the number of distinct real eigenvalues and c the number of distinct complex conjugate
eigenvalue pairs.
Proof. Let A have the real Schur decomposition A = QRQT . Since f (A) =
Qf (R)QT , f (A) is real if and only if f (R) is real. If A has a real negative eigenvalue,
Ri = (rii ) say, then f (Ri ) is necessarily nonreal; this gives the first part of the
theorem.
If A has no real negative eigenvalues, consider the 2s primary square roots of A
described in Theorem 1.26. We have s = r + 2c. Lemma 6.4 shows that each 2 × 2
block Rii has two real primary square roots. Hence, of the 2s = 2r+2c primary square
roots of A, precisely 2r+c of them are real.

Algorithm 6.7 (real Schur method).

√ Given A ∈ Rn×n with no eigenvalues
√ on R
−

this algorithm computes X = A via a real Schur decomposition, where · denotes

any real primary square root.

1 Compute a real √ Schur decomposition A = QRQT , where R is block m × m.

2 Compute Uii = Rii , i = 1: m, using (6.9) whenever Rii is 2 × 2.
3 for j = 2: m
4 for i = j − 1: −1: 1
Pj−1
5 Solve Uii Uij + Uij Ujj = Rij − k=i+1 Uik Ukj for Uij .
6 end
7 end
8 X = QU QT

Cost: 28 31 n3 flops.
Two comments are necessary. First, the principal square root is computed if the
principal square root is taken at line 2, which for 2×2 blocks means taking the positive
6.3 Newton’s Method and Its Variants 139

sign in (6.9). Second, as for Algorithm 6.3, it is necessary that whenever Rii and Rjj
have the same eigenvalues, we take the same square root.
Now we consider the numerical stability of Algorithms 6.3 and 6.7. A straight-
forward rounding error analysis shows that the computed square root U b of T in
Algorithm 6.3 satisfies (see Problem 6.6)

b 2 = T + ∆T,
U |∆T | ≤ γ b |2 ,
en |U

where the inequality is to be interpreted elementwise. Computation of the Schur de-

composition (by the QR algorithm) is a backward stable process [224, , Sec. 7.5.6],
and standard error analysis leads to the overall result

b 2 = A + ∆A,
X k∆AkF ≤ γ b 2F ,
en3 kXk

which can be expressed as

b 2 kF
kA − X
≤γ b
en3 αF (X). (6.10)
kAkF

where α is defined in (6.4). We conclude that Algorithm 6.3 has essentially optimal
stability. The same conclusion holds for Algorithm 6.7, which can be shown to satisfy
the same bound (6.10).

6.3. Newton’s Method and Its Variants

Newton’s method for solving X 2 = A can be derived as follows. Let Y be an
approximate solution and set Y + E = X, where E is to be determined. Then
A = (Y + E)2 = Y 2 + Y E + EY + E 2 . Dropping the second order term in E leads
to Newton’s method:

X0 given,

Solve Xk Ek + Ek Xk = A − Xk2
k = 0, 1, 2, . . . . (6.11)
Xk+1 = Xk + Ek

At each iteration a Sylvester equation must be solved for Ek . The standard way
of solving Sylvester equations is via Schur decomposition of the coefficient matrices,
which in this case are both Xk . But the Schur method of the previous section can
compute a square root with just one Schur decomposition, so Newton’s method is
unduly expensive in the form (6.11). The following lemma enables us to reduce
the cost. Note that Ek in (6.11) is well-defined, that is, the Sylvester equation is
nonsingular, if and only if Xk and −Xk have no eigenvalue in common (see (B.20)).

Lemma 6.8. Suppose that in the Newton iteration (6.11) X0 commutes with A and
all the iterates are well-defined. Then, for all k, Xk commutes with A and Xk+1 =
1 −1
2 (Xk + Xk A).

−1
Proof. The proof is by induction. Let Y0 = X0 and Yk = 12 (Xk−1 + Xk−1 A),
k ≥ 1. For the inductive hypothesis we take Xk A = AXk and Yk = Xk , which is
trivially true for k = 0. The matrix Fk = 21 (Xk−1 A − Xk ) is easily seen to satisfy
Xk Fk + Fk Xk = A − Xk2 , so Fk = Ek and Xk+1 = Xk + Fk = 21 (Xk + Xk−1 A) = Yk+1 ,
140 Matrix Square Root

which clearly commutes with A since Xk does. Hence the result follows by induction.

The lemma shows that if X0 is chosen to commute with A then all the Xk and Ek
in (6.11) commute with A, permitting great simplification of the iteration.
The most common choice is X0 = A (or X0 = I, which generates the same X1 ),
giving the Newton iteration

Newton iteration (matrix square root):

1
Xk+1 = (Xk + Xk−1 A), X0 = A. (6.12)
2

If A is nonsingular, standard convergence theory for Newton’s method allows us to

deduce quadratic convergence of (6.11) to a primary square root for X0 sufficiently
close to the square root, since the Fréchet derivative of F (X) = X 2 − A is nonsingular
at a primary square root. The next result shows unconditional quadratic convergence
of (6.12) to the principal square root. Moreover, it shows that (6.12) is equivalent to
the Newton sign iteration (5.16) applied to A1/2 .

Theorem 6.9 (convergence of Newton square root iteration). Let A ∈ Cn×n have
no eigenvalues on R− . The Newton square root iterates Xk from (6.12) with any
X0 that commutes with A are related to the Newton sign iterates
1
Sk+1 = (Sk + Sk−1 ), S0 = A−1/2 X0
2
by Xk ≡ A1/2 Sk . Hence, provided that A−1/2 X0 has no pure imaginary eigenval-
ues, the Xk are defined and Xk converges quadratically to A1/2 sign(A−1/2 X0 ). In
particular, if the spectrum of A−1/2 X0 lies in the right half-plane then Xk converges
quadratically to A1/2 and, for any consistent norm,
1
kXk+1 − A1/2 k ≤ kX −1 k kXk − A1/2 k2 . (6.13)
2 k

Proof. Note first that any matrix that commutes with A commutes with A±1/2 ,
by Theorem 1.13 (e). We have X0 = A1/2 S0 and so S0 commutes with A. Assume
that Xk = A1/2 Sk and Sk commutes with A. Then Sk commutes with A1/2 and
1 1/2 1
Xk+1 = (A Sk + Sk−1 A−1/2 A) = A1/2 · (Sk + Sk−1 ) = A1/2 Sk+1 ,
2 2
and Sk+1 clearly commutes with A. Hence Xk ≡ A1/2 Sk by induction. Then, using
Theorem 5.6, limκ→∞ Xk = A1/2 limκ→∞ Sk = A1/2 sign(S0 ) = A1/2 sign(A−1/2 X0 ),
and the quadratic convergence of Xk follows from that of Sk .
For the last part, if S0 = A−1/2 X0 has spectrum in the right half-plane then
sign(S0 ) = I and hence Xk → A1/2 . Using the commutativity of the iterates with A,
it is easy to show that
1 −1
Xk+1 ± A1/2 = X (Xk ± A1/2 )2 , (6.14)
2 k
which, with the minus sign, gives (6.13).
6.3 Newton’s Method and Its Variants 141

We make several remarks on this result.

First, an implication of Theorem 6.9 of theoretical interest, which can also be
deduced from the connection with the full Newton method, is that (6.12) converges
to A1/2 for any X0 that commutes with A and is sufficiently close to A1/2 .
Second, it is worth noting that the sequence {Xk } from (6.12) may be well-defined
when that for the full Newton method (6.11) is not; see Problem 6.7. No analogue of
the condition in Theorem 6.9 guaranteeing that the Xk are well-defined is available
for (6.11).
Third, this analysis is more powerful than the Jordan-based analysis in Sec-
tion 4.9.3, which also showed convergence of (6.12) to the principal square root. If X0
is known only to commute with A then the Xk do not necessarily share the Jordan
block structure of A (see Problem 1.26), and so the analysis cannot reduce to that
for a single Jordan block.
Fourth, we noted in Section 5.3 that the Newton iteration for sign(A) requires
many iterations when A has an eigenvalue close to the imaginary axis. Theorem 6.9
therefore implies that the Newton iteration (6.12) for A1/2 will require many iterations
when A has an eigenvalue close to the negative real axis, as can be seen from, using
(5.18),

k k
Xk = A1/2 (I − G20 )−1 (I + G20 ), where k ≥ 1 and G0 = (A1/2 − I)(A1/2 + I)−1 .

Fifth, when A is positive definite the convergence of (6.12) is monotonic from

above in the positive semidefinite ordering; see Problem 6.9 (b).
Finally, it is interesting to consider how (6.12) behaves when X0 does not com-
mute with A, even though commutativity was assumed in the derivation. Lack of
commutativity can cause quadratic convergence, and even convergence itself, to be
lost; see Problem 6.8.
A coupled version of (6.12) can be obtained by defining Yk = A−1 Xk . Then
Xk+1 = 21 (Xk + Yk−1 ) and Yk+1 = A−1 Xk+1 = 21 (Yk + Xk−1 ), on using the fact that
Xk commutes with A. This is the iteration of Denman and Beavers [146, ]:

DB iteration:
1

Xk+1 = Xk + Yk−1 , X0 = A,
2
(6.15)
1

Yk+1 = Yk + Xk−1 , Y0 = I.
2

Under the conditions of Theorem 6.9,

lim Xk = A1/2 , lim Yk = A−1/2 . (6.16)

k→∞ k→∞

Defining Mk = Xk Yk , we have Mk+1 = (2I + Xk Yk + Yk−1 Xk−1 )/4 = (2I + Mk +

Mk−1 )/4. This gives the product form of the DB iteration, identified by Cheng,
Higham, Kenney, and Laub [108, ], in which we iterate with Mk and either
142 Matrix Square Root

Xk or Yk :

Product form DB iteration:

 
1 Mk + Mk−1
Mk+1 = I+ , M0 = A,
2 2
1 (6.17)
Xk+1 = Xk (I + Mk−1 ), X0 = A,
2
1
Yk+1 = Yk (I + Mk−1 ), Y0 = I.
2

Clearly, limk→∞ Mk = I (and (6.16) continues to hold). The product form DB

iteration has the advantage in efficiency over the DB iteration that it has traded one
of the matrix inversions for a matrix multiplication. Another attraction of (6.17) is
that a convergence test can be based on the error kMk − Ik, which is available free
of charge.
Yet another variant of (6.12) can be derived by noting that
1 −1
Ek+1 = (X A − Xk+1 ) (6.18)
2 k+1
1 −1 2
= Xk+1 (A − Xk+1 )
2
1 −1  1 
= Xk+1 A − (Xk + Xk−1 A)2
2 4
1 −1  2A − Xk2 − Xk−2 A2 
= Xk+1
2 4
1 −1 (Xk − Xk−1 A)2
= − Xk+1
2 4
1 −1 2 1 −1
= − Xk+1 Ek = − Ek Xk+1 Ek .
2 2
Setting Yk = 2Ek and Zk = 4Xk+1 we obtain the iteration

CR iteration:

Yk+1 = −Yk Zk−1 Yk , Y0 = I − A,

(6.19)
Zk+1 = Zk + 2Yk+1 , Z0 = 2(I + A).

From the derivation we have Yk → 0 and Zk → 4A1/2 . This iteration is derived in a

different way by Meini [421, ]: she shows that 14 A−1/2 is the constant coefficient
in the inverse of the Laurent matrix polynomial R(z) = (I −A)z −1 +2(I +A)+(I −A)z
and applies cyclic reduction to an associated bi-infinite block tridiagonal matrix in
order to compute the inverse of this coefficient.
A minor variation of (6.19) is worth noting. If we set Xk = Zk /4 and Ek = Yk+1 /2
then (6.19) becomes

IN iteration:

Xk+1 = Xk + Ek , X0 = A,
1 −1 1 (6.20)
Ek+1 = − Ek Xk+1 Ek , E0 = (I − A).
2 2
6.3 Newton’s Method and Its Variants 143

Table 6.1. Cost per iteration of matrix square root iterations.

Iteration Operations Flops

Newton, (6.12) D 8n3/3
DB, (6.15) 2I 4n3
Product DB, (6.17) M +I 4n3
CR, (6.19) M +D 14n3/3
IN, (6.20) M +D 14n3/3

Here, Xk → A1/2 and Ek → 0. This incremental form of the Newton iteration,

suggested by Iannazzo [305, ], is of interest because it updates Xk by a correction
term that is, ultimately, small and accurately computable (in an absolute sense)—
this formulation therefore adheres to one of the maxims of designing stable algorithms
[276, , Sec. 1.18]. (The same comments apply to (6.19), in which the incremental
form is slightly less clear.)
The computational cost of the Newton iteration and its variants is compared in
Table 6.1, where M , I, and D denote a matrix multiplication, matrix inversion, and
solution of a multiple right-hand side linear system, respectively. Clearly the Newton
iteration (6.12) is the least expensive iteration.
Suppose now that A is singular with semisimple zero eigenvalues, so that a unique
primary square root whose nonzero eigenvalues lie in the open right half-plane exists,
by Problem 1.27. We will denote this square root by A1/2 . One difference between
(6.19) and (6.20) and the other iterations is that the former do not invert A on the
first step and so can potentially be used for singular A. However, the Newton iteration
(6.12) is also applicable provided that we apply the first iteration formally and start
with X1 = 12 (I + A). The next result describes the convergence behaviour of (6.12)
(and hence also of (6.19) and (6.20)).

Theorem 6.10. Let the singular matrix A ∈ Cn×n have semisimple zero eigenvalues
and nonzero eigenvalues lying off R− . The iterates Xk from the Newton iteration
(6.12) started with X1 = 12 (I + A) are nonsingular and converge linearly to A1/2 , with

kXk − A1/2 k = O(2−k ). (6.21)

Proof. We can write the Jordan canonical form of A as A = Z diag(J1 , 0)Z −1 ,
where J1 contains the Jordan blocks corresponding to the nonzero eigenvalues and
hence is nonsingular. Then X1 = Z diag((J1 + I)/2, I/2)Z −1 . It is easy to see that
(k) (k)
the Xk have the form Xk = Z diag(J1 , 2−k I)Z −1 for all k ≥ 1, where the J1 are
1/2 1/2
the Newton iterates for J1 . Hence, using Theorem 6.9, Xk → Z diag(J1 , 0)Z −1 =
1/2
A and
(k) 1/2
kXk − A1/2 k ≤ κ(Z)k diag(J1 − J1 , 2−k I)k = O(2−k ).
Despite the convergence result of Theorem 6.10, it is not recommended to straight-
forwardly apply any of the iterations to a singular matrix; numerical instability is
likely because the iterations all invert a matrix that is converging to a singular limit.
It is better to iterate until the “nonsingular part” of the iterates has converged to the
desired accuracy, at Xk say, and then compute the “correction step”
1
Xk+1 = Xk−1 A = Xk − 2 · (Xk − Xk−1 A). (6.22)
2
144 Matrix Square Root

This is a double-length Newton step, whose benefits for problems with a singular
Fréchet derivative at the solution have been noted in the more general context of
algebraic Riccati equations by Guo and Lancaster [234, ]. From the proof of
(k) −1 1/2
Theorem 6.10 we see that Xk−1 A = Z diag(J1 J1 , 0)Z −1 ≈ Z diag(J1 , 0)Z −1 =
(k) 1/2
A1/2 , since J1 ≈ J1 by assumption.
While the iterations described in this section are mathematically equivalent, they
are not equivalent in finite precision arithmetic. We will see in the next section that
they have quite different stability properties.

6.4. Stability and Limiting Accuracy

Standard convergence theory for Newton’s method for a nonlinear system guarantees
quadratic convergence to a solution provided that the Jacobian is nonsingular at the
solution and the starting point is close enough to the solution. Under these conditions
(sufficiently small) rounding errors in one step will be damped out in the next, close
to convergence, because they are seen by Newton’s method as perturbing an iterate to
another point in the region of convergence. Iteration (6.11)—full Newton’s method for
the matrix square root—is therefore not unduly affected by rounding errors. The same
cannot necessarily be said for iterations (6.12), (6.15), (6.17), (6.19), and (6.20): these
iterations are equivalent to (6.11) with X0 = A only in exact arithmetic. Indeed we
have already seen the instability of (6.12) in Section 4.9.4. In the presence of roundoff,
the commutativity conditions in Lemma 6.8 and elsewhere that we used in deriving
these iterations no longer hold and rounding errors can potentially be magnified by
the iterations. Looked at another way, the Newton iteration (6.12) converges to
A1/2 along a path of matrices all of which are functions of A and hence commute
with A. Rounding errors tend to perturb iterates off the path; Theorem 6.9 is then
not applicable, and the local quadratic convergence of Newton’s method cannot be
invoked, because the equivalence of (6.12) to full Newton’s is vitiated by roundoff.
In the following subsections we investigate the stability and limiting accuracy of
the square root iterations using the framework of Section 4.9.4. We assume throughout
that A has no eigenvalues on R− .

6.4.1. Newton Iteration

We consider first the Newton iteration (6.12), for which the iteration function is
g(X) = 21 (X + X −1 A). It is easy to show that the Fréchet derivative is given by
Lg (X, E) = 12 (E − X −1 EX −1 A). The relevant fixed point of g is X = A1/2 , for
which
1
Lg (A1/2 , E) = (E − A−1/2 EA1/2 ). (6.23)
2
The eigenvalues of Lg (A1/2 ), which are most easily obtained by applying (B.17) to
T
the Kronecker matrix form 21 (I − A1/2 ⊗ A−1/2 ), are

1 1/2 −1/2
(1 − λi λj ), i, j = 1: n,
2
where the λi are the eigenvalues of A. Hence to guarantee stability we need
1  
1/2 −1/2 
ψN := max 1 − λi λj  < 1. (6.24)
i,j 2
6.4 Stability and Limiting Accuracy 145

This is a severe restriction on the matrix A. For example, if A is Hermitian positive

definite then the condition is equivalent to κ2 (A) < 9, so that A must be extremely
well conditioned.
The validity of the conclusion of this stability analysis is easily demonstrated.
Suppose that A = ZΛZ −1 , where Λ = diag(λi ), and let X0 = ZD0 Z −1 , where
D0 = diag(di ). Consider a rank-1 perturbation E0 = ǫuj vi∗ ≡ ǫ(Zej )(eTi Z −1 ) to X0 ,
where i 6= j. We have (X0 + E0 )−1 = X0−1 − X0−1 E0 X0−1 by the Sherman–Morrison
formula (B.11). The induced perturbation in X1 is
1
E1 = (E0 − X0−1 E0 X0−1 A)
2
1
= (E0 − ZD0−1 Z −1 E0 ZD0−1 Z −1 A)
2
1
= (E0 − ǫZD0−1 ej eTi D0−1 ΛZ −1 )
2
1 1 λi −1 
= E0 − ǫ Zej eTi Z
2 dj di
1 λi 
= 1− E0 .
2 di dj
1/2 1/2 −1/2

If we set X0 = A1/2 , so that di = λi , then E1 = 12 1 − λi λj E0 , and after k
iterations we have
h1 ik
1/2 −1/2

Xk + Ek = A1/2 + 1 − λi λj E0 .
2
This analysis shows that the Newton iteration (6.12) can diverge when started ar-
bitrarily close to the desired square root if (6.24) is not satisfied. Of course this
perturbation takes the iteration off the path of matrices that commute with A, so
that Theorem 6.9 is not applicable.
Turning to the limiting accuracy, from (6.23) we have
1
kLg (A1/2 , E)k ≤ (1 + κ(A1/2 ))kEk,
2
giving an estimate for the relative limiting accuracy of 12 (1 + κ(A1/2 ))u.

6.4.2. DB Iterations
For the DB iteration (6.15) the iteration function is
 
1 X + Y −1
G(X, Y ) = .
2 Y + X −1

The Fréchet derivative of G at (X, Y ) in the direction (E, F ) is

    
E 1 E − Y −1 F Y −1
Lg (X, Y ; E, F ) ≡ Lg X, Y ; = .
F 2 F − X −1 EX −1

Any point of the form (X, Y ) = (B, B −1 ) is a fixed point of G, and

 
1 E − BF B
Lg (B, B −1 ; E, F ) = . (6.25)
2 F − B −1 EB −1
146 Matrix Square Root

A straightforward computation shows that Lg (B, B −1 ) is idempotent. Hence the DB

iteration is stable at the fixed point (A1/2 , A−1/2 ).
For the product form (6.17) of the DB iteration, the iteration function is
 
1 I + 12 (M + M −1 )
G(M, X) = .
2 X(I + M −1 )
We have  
1 −1
1 2 (E − M EM −1 )
Lg (M, X; E, F ) = .
2 F (I + M −1 ) − XM −1 EM −1
At a fixed point (I, X),
    
0 0 0 E
Lg (I, X; E, F ) = = . (6.26)
F − 12 XE − 12 X I F

It is easy to see that Lg (I, X) is idempotent, and hence the product form of the DB
iteration is stable at (M, X) = (I, A1/2 ) and at (M, Y ) = (I, A−1/2 ).
To determine the limiting accuracy, with B = A1/2 , kEk ≤ ukA1/2 k, and kF
k ≤
ukA−1/2 k, the (1,1) block of (6.25) is bounded by 21 kA1/2 k + kA1/2 k2 kA−1/2 k u =
1
2 kA
1/2
k(1 + κ(A1/2 ))u and the (2,1) block by 21 kA−1/2 k(1 + κ(A1/2 ))u. The relative
limiting accuracy estimate is therefore 12 (1 + κ(A1/2 ))u, as for the Newton iteration.
For the product DB iteration, by considering (6.26) with kEk ≤ u and kF k ≤
kA1/2 ku we obtain a relative limiting accuracy estimate of (3/2)u, which is indepen-
dent of A.

6.4.3. CR Iteration
For the CR iteration (6.19) the iteration function is
 
−Y Z −1 Y
G(Y, Z) = .
Z − 2Y Z −1 Y

At any fixed point (0, Z),  

0
Lg (0, Z; E, F ) = ,
F
so again Lg (0, Z) is idempotent and stability of the iteration follows.
The relative limiting accuracy is trivially of order u.

6.4.4. IN Iteration
The iteration function for the IN iteration (6.20) is
 
X +H
G(X, H) = .
− 12 H(X + H)−1 H

It is easy to see that at the fixed point (A1/2 , 0),

    
E+F I I E
Lg (A1/2 , 0; E, F ) = = .
0 0 0 F

Therefore Lg (A1/2 , 0) is idempotent and the iteration is stable. The relative limiting
accuracy is again trivially of order u.
6.5 Scaling the Newton Iteration 147

Table 6.2. Summary of stability and limiting accuracy of square root iterations.

Iteration Stable? ˛ ˛ Limiting accuracy

˛ 1/2 −1/2 ˛
Newton, (6.12) Only if maxi,j 12 ˛1−λi λj ˛ < 1, κ(A1/2 )u
where λi ∈ Λ(A)
DB, (6.15) Yes κ(A1/2 )u
Product DB, (6.17) Yes u
CR, (6.19) Yes u
IN, (6.20) Yes u

6.4.5. Summary
Table 6.2 summarizes what our analysis says about the stability and limiting accu-
racy of the iterations. The Newton iteration (6.12) is unstable at A1/2 unless the
eigenvalues λi of A are very closely clustered, in the sense that (λi /λj )1/2 lies in a
ball of radius 2 about z = 1 in the complex plane, for all i and j. The four rewritten
versions of Newton’s iteration, however, are all stable at A1/2 . The price to be paid
for stability is a coupled iteration costing more than the original Newton iteration
(see Table 6.1). The limiting accuracy of the DB iteration is essentially κ(A1/2 )u, but
the other three stable iterations have limiting accuracy of order u.

6.5. Scaling the Newton Iteration

The best way to obtain scaling parameters for the Newton iteration and its variants
is to exploit connections with the matrix sign function. Theorem 6.9 implies that
scaling parameters for the Newton sign iterates Sk can be translated into ones for
the Newton square root iterates Xk using Xk = A1/2 Sk : thus we scale Xk ← µk Xk
at the start of the iteration, using (for example) the formulae (5.35)–(5.37) with Xk
replaced by A−1/2 Xk . The resulting parameters depend on the unknown A1/2 , but
for the determinantal scaling (5.35) we can use det(A1/2 ) = det(A)1/2 to obtain

Newton iteration (scaled):

 
1  det(Xk ) −1/n
Xk+1 = (µk Xk + µ−1 −1
µk =   . (6.27)
k Xk A), X0 = A,
2 det(A)1/2 

For the DB iteration, the µk in (6.27) can be expressed in terms of just Xk and
Yk , by using the relation Yk = A−1 Xk . The determinantally scaled DB iteration is

DB iteration (scaled):
 
 det(Xk ) −1/n  −1/(2n)
µk =  1/2

 or µk = det(Xk ) det(Yk ) ,
det(A)
1
(6.28)
Xk+1 = µk Xk + µ−1
k Yk
−1
, X0 = A,
2
1

Yk+1 = µk Yk + µ−1
k Xk
−1
, Y0 = I.
2
148 Matrix Square Root

For the product form of the DB iteration determinantal scaling gives, using Mk =
Xk Yk ,

Product form DB iteration (scaled):

 −1/(2n)
µk = det(Mk ) ,
 −1 
1 µk Mk + µ−2
2
k Mk
Mk+1 = I+ , M0 = A,
2 2
1 (6.29)
Xk+1 = µk Xk (I + µ−2 −1
k Mk ), X0 = A,
2
1
Yk+1 = µk Yk (I + µ−2 −1
k Mk ), Y0 = I.
2

Incorporating scaling into the IN iteration requires a little algebra. In terms of

Xk , the Newton increment Ek in (6.20) is given by Ek = 12 (Xk−1 A−Xk ). After scaling
Xk ← µk Xk we have

ek = 1 (µ−1 X −1 A − µk Xk )
E
2 k k
1 −1 1 −1 1
= µ−1
k · (Xk A − Xk ) + µk Xk − µk Xk
2 2 2
1 1
= µ−1
k (Ek + Xk ) − µk Xk .
2 2
The scaled iteration is therefore

IN iteration (scaled):

X0 = A, E0 = 12 (I − A),
 
 det(Xk ) −1/n

µk =   ,
det(A)1/2 
ek = µ−1 (Ek + 1 Xk ) − 1 µk Xk ,
E (6.30)
k 2 2
ek ,
Xk+1 = µk Xk + E
−1
Ek+1 = − 12 Ek Xk+1 Ek .

Although the formulae (6.30) appear very different from, and less elegant than, (6.27),
the two iterations generate exactly the same sequence of iterates Xk .
A suitable stopping test for all these iterations is from (4.25) and (6.13),
 1/2
kXk+1 k
kXk+1 − Xk k ≤ η , (6.31)
kXk−1 k

where η is a relative error tolerance.

6.6. Numerical Experiments

We present some numerical experiments to illustrate and compare the behaviour of
the methods described in this chapter. For several matrices we give the relative
6.6 Numerical Experiments 149

error kX − Xkb ∞ /kXk∞ and relative residual kA − X b 2 k∞ /kAk∞ of the computed X. b

For the iterative methods X b is taken to be the first iterate after which the relative
error does not decrease significantly. If the iteration is converging slowly, this is the
last iterate before the error starts to increase. The iterative methods are used both
without scaling and with determinantal scaling. In the latter case, scaling is used only
while the relative ∞-norm change in successive iterates exceeds 10−2 , because after
this point the quadratic convergence (if indeed it is seen) leads to rapid convergence.
The matrices are as follows.
1. A = I + uv T , where u = [1 22 . . . n2 ]T , v = [0 1 22 . . . (n − 1)2 ]T . (The principal
square root is known exactly from (1.16).)
2. The 16 × 16 Moler matrix, MATLAB’s gallery(’moler’,16), which is sym-
metric positive definite. It has 15 eigenvalues of order 1 and one small eigenvalue
of order 10−9 .
3. A moderately nonnormal 8 × 8 matrix formed in MATLAB as

n = 8;
Q = gallery(’orthog’,n);
A = Q*rschur(n,2e2)*Q’;

The function rschur(n,mu), from the Matrix Computation Toolbox [264], gen-
erates an upper quasi-triangular matrix with eigenvalues αj + iβj , αj = −j 2 /10,
βj = −j, j = 1: n/2 and (2j, 2j + 1) elements mu.
4. A 16×16 Chebyshev–Vandermonde matrix, gallery(’chebvand’,16) in MAT-
LAB, which has 8 complex eigenvalues with modulus of order 1 and 8 real,
positive eigenvalues between 3.6 and 10−11 .
5. A 9 × 9 singular nonsymmetric matrix resulting from discretizing the Neumann
problem with the usual five point operator on a regular mesh. The matrix has
real nonnegative eigenvalues and a one-dimensional null space with null vector
the vector of 1s.
The first matrix, for which the results are given in Table 6.3, represents an easy
problem. The Newton iteration is unstable, as expected since ψN > 1 (see (6.24)):
the error reaches a minimum of about u1/2 and then grows unboundedly. The other
methods all produce tiny errors and residuals. Scaling produces a useful reduction in
the number of iterations.
For the Moler matrix, we can see from Table 6.4 that the Schur method performs
well: the residual is consistent with (6.10), and the error is bounded by κsqrt (X)u, as
it should be given the size of the residual (which is also the backward error). Newton’s
method is unstable, as expected given the eigenvalue distribution. The convergence
of the sequence of errors (not shown) for the DB, product DB, and IN iterations
is more linear than quadratic, and a large number of iterations are required before
convergence is reached. The DB and product DB iterations have error significantly
exceeding κsqrt (X)u, and are beaten for accuracy and stability by the IN iteration,
which matches the Schur method. For this matrix, scaling has little effect.
For the nonnormal matrix the tables are turned and it is the IN iteration that
performs badly, giving an error larger than those from the other iterative methods
by a factor about 104 ; see Table 6.5. The Newton iteration performs well, which
150 Matrix Square Root

Table 6.3. Results for rank-1 perturbation of I. α∞ (X) = 1.4, κsqrt (X) = 40, κ2 (A1/2 ) =
1.7 × 102 , ψN = 39.

Iterations Relative error Relative residual

Schur 1.5e-15 6.5e-16
Real Schur 1.4e-15 6.7e-16
Newton, unscaled 10 8.1e-7 8.2e-7
Newton, scaled 7 7.8e-8 1.5e-7
DB, unscaled 11 1.7e-15 3.4e-15
DB, scaled 8 1.5e-15 2.9e-15
Product DB, unscaled 11 7.3e-15 3.8e-15
Product DB, scaled 8 1.2e-14 8.3e-15
IN, unscaled 11 1.5e-14 3.0e-14
IN, scaled 8 1.2e-14 2.4e-14

Table 6.4. Results for Moler matrix. α∞ (X) = 1.1, κsqrt (X) = 8.3 × 104 , κ2 (A1/2 ) =
3.6 × 105 , ψN = 105 .

Iterations Relative error Relative residual

Schur 5.3e-13 3.0e-15
Real Schur 5.3e-13 3.0e-15
Newton, unscaled 8 5.3e-4 1.0e-5
Newton, scaled 7 1.9e-2 3.8e-2
DB, unscaled 17 6.6e-10 5.1e-10
DB, scaled 13 1.1e-9 6.4e-10
Product DB, unscaled 17 1.2e-10 4.8e-11
Product DB, scaled 13 1.5e-10 5.5e-11
IN, unscaled 18 1.2e-13 1.4e-15
IN, scaled 14 8.5e-14 1.5e-15

is consistent with ψN being only slightly larger than 1. The residual of the Schur
methods is consistent with (6.10).
The Chebyshev–Vandermonde matrix shows poor performance of the DB and
product DB iterations, which are beaten by the IN iteration. The Newton iteration
is wildly unstable.
These experiments confirm the predictions of the stability analysis. However, the
limiting accuracy is not a reliable predictor of the relative errors. Indeed, although
the product DB iteration has better limiting accuracy than the DB iteration, this is
not seen in practice. The main conclusion is that no one of the DB, product DB, and
IN iterations is always best. Based on these limited experiments one might choose the
IN iteration when α(X) = O(1) and one of the DB or product DB iterations when
α(X) is large, but this reasoning has no theoretical backing (see Problem 6.24).
Finally, for the singular Neumann matrix we applied the IN iteration (6.20) with-
out scaling until the relative residual was smaller than 10−4 and then computed the
double step (6.22). Six iterations of (6.20) were required, after which the relative
residual was of order 10−5 and the relative error of order 10−3 . After applying (6.22)
these measures both dropped to 10−14 , showing that the double step can be remark-
ably effective.
6.6 Numerical Experiments 151

Table 6.5. Results for nonnormal matrix. α∞ (X) = 1.4 × 108 , κsqrt (X) = 5.8 × 107 ,
κ2 (A1/2 ) = 6.2 × 1010 , ψN = 1.3.

Iterations Relative error Relative residual

Schur 5.1e-10 3.6e-8
Real Schur 5.1e-10 4.3e-8
Newton, unscaled 5 9.2e-7 2.0e-3
Newton, scaled 5 8.5e-8 1.6e-5
DB, unscaled 7 2.9e-8 1.2e-5
DB, scaled 5 2.1e-7 1.6e-4
Product DB, unscaled 6 8.5e-8 8.7e-2
Product DB, scaled 5 9.8e-7 4.8e-2
IN, unscaled 5 2.8e-4 9.9e-1
IN, scaled 4 3.2e-4 1.1e0

Table 6.6. Results for Chebyshev–Vandermonde matrix. α∞ (X) = 2.8, κsqrt (X) = 5.2 × 106 ,
κ2 (A1/2 ) = 1.3 × 107 , ψN = 3.3 × 105 .

Iterations Relative error Relative residual

Schur 1.0e-10 2.4e-15
Real Schur 1.9e-10 2.3e-15
Newton, unscaled 4 2.4e-2 3.4e-3
Newton, scaled 2 6.9e-1 5.9e-1
DB, unscaled 18 1.0e-5 1.4e-5
DB, scaled 11 9.1e-8 1.2e-7
Product DB, unscaled 17 1.7e-5 2.9e-5
Product DB, scaled 11 9.8e-7 7.8e-7
IN, unscaled 22 2.9e-11 5.5e-16
IN, scaled 11 4.8e-12 2.5e-14
152 Matrix Square Root

6.7. Iterations via the Matrix Sign Function

The Newton variants described above by no means represent all possible iterations for
computing the matrix square root. A wide class of iterations can be obtained from
iterations
0 A for the matrix sign function, by applying a sign iteration to the matrix
I 0 and using (5.4), which we repeat here:
   
0 A 0 A1/2
sign = . (6.32)
I 0 A−1/2 0
The next result makes this idea precise. It assumes a specific form g(X) = X h(X 2 )
for the sign iteration function, in order to provide a convenient formula for the square
root iteration. This form holds for all the Padé iterations, in view of (5.27), and is
very natural in view of (5.2) and (5.3).

Theorem 6.11 (Higham, Mackey, Mackey, and Tisseur). Let A ∈ Cn×n have no eigen-
values on R− . Consider any iteration 2
 0 A  of the form Xk+1 = g(Xk ) ≡ Xk h(Xk ) that
converges to sign(X0 ) for X0 = I 0 with order of convergence m. Then in the
coupled iteration
Yk+1 = Yk h(Zk Yk ), Y0 = A,
(6.33)
Zk+1 = h(Zk Yk )Zk , Z0 = I,
Yk → A1/2 and Zk → A−1/2 as k → ∞, both with order of convergence m, Yk
commutes with Zk , and Yk = AZk for all k.
Proof. Observe that
     
0 Yk 0 Yk Yk Zk 0
g = h
Zk 0 Zk 0 0 Zk Yk
  
0 Yk h(Yk Zk ) 0
=
Zk 0 0 h(Zk Yk )
 
0 Yk h(Zk Yk )
=
Zk h(Yk Zk ) 0
   
0 Yk h(Zk Yk ) 0 Yk+1
= = ,
h(Zk Yk )Zk 0 Zk+1 0
where the penultimate equality follows from Corollary 1.34. The initial conditions
Y0 = A and Z0 = I together with (6.32) now imply that Yk and Zk converge to A1/2
and A−1/2 , respectively. It is easy to see that Yk and Zk are polynomials in A for all
k, and hence Yk commutes with Zk . Then Yk = AZk follows by induction. The order
of convergence of the coupled iteration (6.33) is clearly the same as that of the sign
iteration from which it arises.
Theorem 6.11 provides an alternative derivation of the DB iteration (6.15). Take
g(X) = 12 (X +X −1 ) = X · 12 (I +X −2 ) ≡ Xh(X 2 ). Then Yk+1 = Yk · 21 (I +(Zk Yk )−1 ) =
1 −1 −1 1 −1 1 −1
2 Yk (I + Yk Zk ) = 2 (Yk + Zk ), and, likewise, Zk+1 = 2 (Zk + Yk ).
New iterations are obtained by applying the theorem to the Padé family (5.28):

Padé iteration:

Yk+1 = Yk pℓm (1 − Zk Yk ) qℓm (1 − Zk Yk )−1 , Y0 = A,

(6.34)
−1
Zk+1 = pℓm (1 − Zk Yk ) qℓm (1 − Zk Yk ) Zk , Z0 = I.
6.7 Iterations via the Matrix Sign Function 153

For 0 ≤ ℓ, m ≤ 2, pℓm (1 − x2 ) and qℓm (1 − x2 ) can be read off from Table 5.1. For
ℓ = m − 1 and ℓ = m, pℓm /qℓm has the partial fraction form given in Theorem 5.9
(d). From Theorem 5.8 and Theorem 6.11 we deduce that Yk → A1/2 and Zk →
A−1/2 with order ℓ + m + 1 unconditionally if ℓ = m − 1 or ℓ = m, or provided
k diag(I − A, I − A)k < 1 if ℓ ≥ m + 1.
For ℓ = 1, m = 0, (6.34) gives a Newton–Schulz iteration:

Newton–Schulz iteration:

Yk+1 = 21 Yk (3I − Zk Yk ), Y0 = A,
1
(6.35)
Zk+1 = 2 (3I − Zk Yk )Zk , Z0 = I.

A sufficient condition for convergence of this inverse-free iteration is kI − Akp < 1,

for p = 1, 2, or ∞. For ℓ = 0, m = 1, (6.34) is

Yk+1 = 2Yk (Zk Yk + I)−1 , Y0 = A,

(6.36)
Zk+1 = 2(Zk Yk + I)−1 Zk , Z0 = I.

This is closely related to the DB iteration (6.15) in that Yk from (6.36) equals the
inverse of Xk from (6.15) with X0 = A−1 , and similarly for the Zk . The DB iteration
is generally preferable to (6.36) because it requires less work per iteration.
What can be said about the stability of the coupled iteration (6.33)? Since Yk
and Zk commute, the iteration formulae can be rewritten in several ways, and the
particular choice of formula turns out to be crucial to the stability. For example,
(6.36) is stable, but with the rearrangement Zk+1 = 2Zk (Zk Yk + I)−1 , stability is
lost. The next theorem shows that all instances of (6.33) are stable and hence that
the particular formulae (6.33) are the right choice from the point of view of stability.

Theorem 6.12 (Higham, Mackey, Mackey, and Tisseur). Consider any iteration of
the form (6.33) and its associated mapping
 
Y h(ZY )
G(Y, Z) = ,
h(ZY )Z

where Xk+1 = g(Xk ) ≡ Xk h(Xk2 ) is any superlinearly convergent iteration for sign(X0 ).
Then any matrix pair of the form P = (B, B −1 ) is a fixed point for G, and the Fréchet
derivative of G at P is given by
 
1 E − BF B
Lg (P ; E, F ) = . (6.37)
2 F − B −1 EB −1

The derivative map Lg (P ) is idempotent (Lg (P )◦Lg (P ) = Lg (P )) and hence iteration

(6.33) is stable.
Proof. We know from Theorem 5.13 that Lg (S, G) = 21 (G − SGS) if S = sign(S).
Now sign(I) = I implies I = g(I) = h(I). Hence (B, B −1 ) is a fixed point for G. In
view of the relations in the proof of Theorem 6.11, we have Lg (P ; E, F ) = Lg (S, H),
where    
0 B 0 E
S= , H = .
B −1 0 F 0
Inserting these expressions into the formula for Lg (S, H) gives the result.
154 Matrix Square Root

This result proves stability of all the Padé iterations. Note that (6.37) is identical
to the expression (6.25) for the Fréchet derivative of the DB iteration, as might be
expected since the inverse of the DB iteration is a member of the Padé family, as
noted above.
An underlying theme in this section is that the use of commutativity is best
avoided if stable iterations are to be obtained. Note that the proof of Theorem 6.11
does not draw on commutativity. Simply rearranging the “Z formula” in (6.33) to
Zk+1 = Zk h(Zk Yk ), which is valid mathematically, changes the Fréchet derivative,
and the new iteration is generally unstable (as mentioned above for ℓ = 0, m = 1)
[283, , Lem. 5.4]. What it is safe to do is to rearrange using Corollary 1.34. Thus
changing the Z formula to Zk+1 = Zk h(Yk Zk ) does not affect the stability of (6.33).

6.8. Special Matrices

We now consider methods for computing square roots of some special classes of ma-
trices of practical importance, including matrices close to the identity or with “large
diagonal”, M -matrices, H-matrices, and Hermitian positive definite matrices. We
introduce some linearly convergent iterations, all of which are matrix multiplication-
based and hence easy to implement. In some applications a low accuracy approxima-
tion to a square root is sufficient and in these cases it may be enough to carry out a
few iterations of one of the procedures below.

6.8.1. Binomial Iteration

The methods of this section are all related to the binomial expansion

X∞ 1 X∞
(I − C)1/2 = 2 (−C)j ≡ I − αj C j , αj > 0, (6.38)
j=0
j j=1

which is valid when ρ(C) < 1 (see Theorem 4.7).

When ρ(A − I) < 1 we can approximate A1/2 by evaluating a partial sum of
(6.38) with A ≡ I − C. Convergence will be slow unless kCk ≪ 1, but convergence
acceleration techniques such as Aitken extrapolation (Aitken’s ∆2 -method) [542, ,
Sec. 5.10] can be used in an effort to obtain a better approximation from a fixed
number of terms of the series.
When ρ(A − I) exceeds 1 we can write

A = s(I − C), (6.39)

and try to choose s so that ρ(C) < 1. When A has real, positive eigenvalues, 0 <
λn ≤ λn−1 ≤ · · · ≤ λ1 , the s that minimizes ρ(C), and the corresponding minimal
ρ(C), are (see Problem 6.17)

λ1 − λn
s = (λ1 + λn )/2, ρ(C) = < 1. (6.40)
λ1 + λn

An alternative choice of s, valid for all A, is s = trace(A∗ A)/ trace(A∗ ), which mini-
mizes kCkF ; see Problem 6.18. This choice may or may not achieve ρ(C) < 1.
Assume now that ρ(C) < 1. An iterative method can be derived by defining
(I − C)1/2 =: I − P and squaring the equation to obtain P 2 − 2P = −C. A natural
6.8 Special Matrices 155

iteration for computing P is

Binomial iteration:
1
Pk+1 = (C + Pk2 ), P0 = 0. (6.41)
2

Our choice of name for this iteration comes from the observation that I−Pk reproduces
the binomial expansion (6.38) up to and including terms of order C k . So (6.41) can
be thought of as a convenient way of generating the binomial expansion.
Before analyzing the convergence of the binomial iteration we note some properties
of P . From (6.38) we have
X∞
P = αj C j , (6.42)
j=1

where thePαj are positive. The eigenvalues P∞ of P and C are therefore related by
∞
λi (P ) = j=1 αj λi (C)j . Hence ρ(P ) ≤ j=1 αj ρ(C)j = 1 − (1 − ρ(C))1/2 . Since
1 − (1 − x)1/2 ≤ x for 0 ≤ x ≤ 1, we conclude that ρ(P ) ≤ ρ(C) < 1. Similarly, we
find that kP k ≤ kCk for any consistent matrix norm for which kCk < 1. Finally, it
is clear that of all square roots I − Q of I − C, the principal square root is the only
one for which ρ(Q) < 1, since ρ(Q) < 1 implies that the spectrum of I − Q lies in the
open right half-plane.
We first analyze convergence in a special case. For A ∈ Rn×n , let A ≥ 0 denote
that aij ≥ 0 for all i and j. If C ≥ 0 then the binomial iteration enjoys mononotic
convergence.

Theorem 6.13. Let C ∈ Rn×n satisfy C ≥ 0 and ρ(C) < 1 and write (I − C)1/2 =
I − P . Then in the binomial iteration (6.41), Pk →P with

0 ≤ Pk ≤ Pk+1 ≤ P, k ≥ 0; (6.43)

that is, the Pk converge monotonically to P from below.

Proof. From (6.41) and (6.42) we have 0 = P0 ≤ P , Pk ≥ 0 for all k, and
P − Pk+1 = 21 (P 2 − Pk2 ), so by induction 0 ≤ Pk ≤ P for all k. Moreover, C/2 = P1 ≥
P0 = 0, and Pk+1 − Pk = 21 (Pk2 − Pk−1 2
), so by induction Pk ≤ Pk+1 for all k. The
inequalities (6.43) are therefore established. Since the sequence {Pk } is nondecreasing
and bounded above, it has a limit, P∗ . This limit satisfies P∗ = 12 (C + P∗2 ), and so
(I − P∗ )2 = I − C. But ρ(Pk ) ≤ ρ(P ) < 1 for all k by (B.13), so ρ(P∗ ) < 1. Thus
I − P∗ is the principal square root, that is, P∗ = P .
It is now natural to try to prove convergence for general C with ρ(C) < 1. Consider
the scalar iteration
1
pk+1 = (c + p2k ), p0 = 0,
2
with (1 − c)1/2 = 1 − p, |p| ≤ |c| < 1. Defining
1
qk+1 = (|c| + qk2 ), q0 = 0,
2

p that |pk | ≤ |qk | for all p

it follows by induction k. By Theorem 6.13, qk is monotonically
increasing to 1 − 1 − |c|. Thus |pk | ≤ 1 − 1 − |c|. Now pk+1 − p = 21 (p2k − p2 ) =
156 Matrix Square Root

1
2 (pk + p)(pk − p) and so

1
|pk+1 − p| ≤ (|pk | + |p|)|pk − p|
2
1
≤ (1 − (1 − |c|)1/2 + |c|)|pk − p| =: θ|pk − p|, (6.44)
2
where θ = 21 (1 − (1 − |c|)1/2 + |c|) ≤ |c| < 1. Hence pk → p with a monotonically
decreasing error. This argument can be built into a proof that in the matrix iteration
(6.41) Pk converges to P if ρ(C) < 1. But this restriction on C is stronger than
necessary! The next result describes the actual region of convergence of the binomial
iteration, which is closely connected with the Mandelbrot set.

Theorem 6.14 (convergence of binomial iteration). Let the eigenvalues of C ∈ Cn×n

lie in the cardioid
D = { 2z − z 2 : z ∈ C, |z| < 1 }. (6.45)
Then (I − C)1/2 =: I − P exists and in the binomial iteration (6.41), Pk → P linearly.

Proof. C clearly has no eigenvalues on the interval [1, ∞), so I − C has no

eigenvalues on R− and therefore (I − C)1/2 exists.
Since Pk is a polynomial in C, the Jordan canonical form of C can be used to
block-diagonalize (6.41). Hence we first analyze the behaviour of the scalar iteration
pk+1 = 21 (c + p2k ) =: f (pk ), p0 = 0, where (1 − c)1/2 = 1 − p and c is an eigenvalue of
C, so that c ∈ D. This iteration has two fixed points: 1 ± (1 − c)1/2 . If c ∈ D then
1 − c = 1 − 2z + z 2 = (1 − z)2 , where |z| < 1, and hence (1 − c)1/2 = 1 − z, so that
|f ′ (1 − (1 − c)1/2 )| = |z| < 1. Therefore the fixed point 1 − (1 − c)1/2 is attracting,
while 1 + (1 − c)1/2 = 2 − z is repelling.
Change variables to xk = pk /2. Then
c
xk+1 = x2k + =: Qc/4 (xk ), x0 = 0.
4
The quadratic map z → Qa (z) = z 2 + a in the complex plane is much studied. The
(k)
set of a ∈ C for which Qa (0) is bounded is the famous Mandelbrot set. Existing
analysis shows that if a ∈ { p/2 − p2 /4 : |p| < 1 }, which is the interior of the main
(k)
cardioid of the Mandelbrot set, then Qa (0) converges to the attracting fixed point
[97, , Thm. 1.3, p. 126], [149, , Thm. 4.6]. We conclude that if c ∈ D then
pk converges to 1 − (1 − c)1/2 .
Our analysis of the scalar iteration combines with Theorem 4.15 to show that Pk
converges to a matrix P whose eigenvalues are those of I − (I − C)1/2 and such that
(I − P )2 = I − C. The only P satisfying these conditions is P = I − (I − C)1/2 .
Convergence is linear since this is a fixed point iteration and the derivative of the
iteration function is nonzero at the eigenvalues of P .
Figure 6.1 plots the cardioid (6.45), which is much bigger than the unit disk,
reaching −3 on the negative real axis and containing points with imaginary part as
large as ±2.6i. Although the binomial series expansion (6.38) does not converge for C
with spectral radius greater than 1, the iteration nevertheless continues to converge
when the eigenvalues of C lie outside the unit disk but within the cardioid. The
monotonic decrease of the error in the eigenvalues throughout the iteration, shown in
(6.44), holds only within the unit disk, however.
6.8 Special Matrices 157

−1

−2

−4 −3 −2 −1 0 1 2

Figure 6.1. The cardioid (6.45), shaded, together with the unit circle. The binomial iteration
converges for matrices C whose eigenvalues lie inside the cardioid.

If ρ(C) < 1, there is a consistent norm such that kCk < 1 (see Section B.7).
Hence when (6.41) is applicable to I − C, so is the Newton–Schulz iteration (6.35),
which also requires only matrix multiplication but which has quadratic convergence.
If medium to high accuracy is required, the Newton–Schulz iteration will be more
efficient than the binomial iteration, despite requiring three matrix multiplications
per iteration versus one for (6.41).

6.8.2. Modified Newton Iterations

The Newton iteration (6.11) can be written as Xk+1 = Xk + Ek , where L(Xk , Ek ) =
−F (Xk ) and L(X, E) is the Fréchet derivative of F (X) = X 2 −A at X in the direction
E. The iterations of Section 6.3 reduce the cost of Newton’s method by taking an
X0 that commutes with A. Another way to reduce the cost of the method is to use
approximations to the Fréchet derivative that ease solution of the Sylvester equation
L(Xk , Ek ) = Xk Ek + Ek Xk = −F (Xk ). In particular, we can freeze the Fréchet
derivative at X0 to obtain the modified Newton iteration

Solve X0 Ek + Ek X0 = A − Xk2
k = 0, 1, 2, . . . , (6.46)
Xk+1 = Xk + Ek
and taking X0 diagonal makes these equations easy to solve. Our first method has
precisely this form, but we will derive it in a different way.
Let A ∈ Cn×n have a principal square root
A1/2 = D1/2 + B,
where D is a known diagonal matrix with real, positive diagonal elements. The natural
choice of D is D = diag(A), assuming A has positive diagonal elements. By squaring
both sides of this equation we obtain the quadratic equation
D1/2 B + BD1/2 = A − D − B 2 . (6.47)
158 Matrix Square Root

We can attempt to compute B by functional iteration:

Pulay iteration:

D1/2 Bk+1 + Bk+1 D1/2 = A − D − Bk2 , B0 = 0. (6.48)

1/2 1/2
Equation (6.48) is easily solved for Bk+1 : (Bk+1 )ij = (A − D − Bk2 )ij /(dii + djj ),
i, j = 1: n. The Pulay iteration is easily shown to be a rewritten version of the
modified Newton iteration (6.46) with X0 = D1/2 ; see Problem 6.16. The next result
gives a sufficient condition for convergence to the principal square root.

Theorem 6.15 (convergence of Pulay iteration). Let A ∈ Cn×n have no eigenvalues

on R− , let D be diagonal with positive diagonal entries, and let B = A1/2 − D1/2 . If
kBk 2
θ= 1/2
< (6.49)
mini di 3

for some consistent matrix norm then in the iteration (6.48), Bk → A1/2 − D1/2
linearly.
Proof. Let Ek = B − Bk and subtract (6.48) from (6.47) to obtain

D1/2 Ek+1 + Ek+1 D1/2 = Bk2 − B 2 = Ek2 − BEk − Ek B.

It follows that kEk k ≤ ek , where ek+1 = e2k /(2σ) + (β/σ)ek , where e0 = β = kBk and
1/2
σ = mini di . The change of variable fk = ek /β yields, since θ = β/σ,
θ 2
fk+1 = f + θfk , f0 = 1. (6.50)
2 k
Suppose θ < 2/3. If |fk | ≤ 1 then
 
θ 
|fk+1 | =  fk + θ |fk | < |fk |.
2

Since f0 = 1 it follows by induction that the |fk | form a monotonically decreasing

sequence, which, being bounded below by 0, has a limit. The iteration (6.50) has two
fixed points, 0 and 2/θ − 2 > 1, so the limit must be 0. Hence Bk → B, as required,
and the convergence is clearly linear.
Very roughly, the condition (6.49) says that D1/2 is a reasonable approximation
to A1/2 , in the sense that |D1/2 | ≥ 32 kBkI.
The Pulay iteration can be simplified by taking D to be a multiple of the identity
matrix. Equivalently, we can set X0 = I/(2α) in (6.46). This gives the iteration

Visser iteration:

Xk+1 = Xk + α(A − Xk2 ), X0 = (2α)−1 I. (6.51)

Theorem 6.16 (convergence of Visser iteration). Let A ∈ Cn×n and α > 0. If the
eigenvalues of I − 4α2 A lie in the cardioid (6.45) then A1/2 exists and the iterates Xk
from (6.51) converge linearly to A1/2 .
6.8 Special Matrices 159

Proof. The eigenvalue condition clearly implies that A has no eigenvalues on R−

e = θ−2 A. Then
and so A1/2 exists. Let θ = 1/(2α), Xk = θYk , and A

1 e
Yk+1 = Yk + (A − Yk2 ), Y0 = I.
2
e ≡ I − C and Yk = I − Pk .
This is equivalent to the binomial iteration (6.41) with A
Hence Theorem 6.14 shows that we have convergence of Yk to A e1/2 if the eigenvalues
e
of C = I − A lie in the cardioid (6.45). In other words, Xk → A1/2 (since α > 0) if
the eigenvalues of I − 4α2 A lie in the cardioid.

If the eigenvalues of A are real and positive then the condition of the theorem is
0 < α < ρ(A)−1/2 . Convergence under these assumptions, but with equality allowed
in the upper bound for α, was proved by Elsner [176, ].
The advantage of the Pulay iteration over the binomial iteration and the Visser
iteration is that it can be applied to a matrix whose diagonal is far from constant,
provided the matrix is sufficiently diagonally dominant. For example, consider the
16 × 16 symmetric positive definite matrix A with aii = i2 and aij = 0.1, i 6= j. For
the Pulay iteration (6.48) with D = diag(A), we have θ = 0.191 in (6.49) and just 9
iterations are required to compute A1/2 with a relative residual less than nu in IEEE
double precision arithmetic. For the binomial iteration (6.41), writing A = s(I − C)
with s given by (6.40), ρ(C) = 0.992 and 313 iterations are required with the same
convergence tolerance. Similarly, the Visser iteration (6.51) requires 245 iterations
with α = 0.058, which was determined by hand to approximately minimize the number
of iterations. (The Newton–Schulz iteration (6.35) does not converge for this matrix.)
The stability of (6.51) under the assumptions of the theorem is easily demonstrated
under the assumption that A has positive real eigenvalues. The Fréchet derivative
of the iteration function g(X) = X + α(A − X 2 ) is Lg (X, E) = E − α(XE + EX).
1/2 1/2
The eigenvalues of Lg (A1/2 ) are µij = 1 − α(λi + λj ), i, j = 1: n, where λi
denotes an eigenvalue of A. The maximum µmax = maxi,j |µij | is obtained for i = j,
1/2
so µmax = maxi |1 − 2αλi |. Since 1 − 4α2 λi lies in the cardioid (6.45), we have
1/2
1 − 4α2 λi > −3, i.e., αλi < 1, so µmax < 1. Hence Lg (A1/2 ) is power bounded and
the iteration is stable.

6.8.3. M-Matrices and H-Matrices

M-matrices arise in a variety of scientific settings, including in finite difference meth-
ods for PDEs, input-output analysis in economics, and Markov chains in stochastic
processes [60, ].
The M-matrices are a subset of the real, square matrices with nonpositive off-
diagonal elements.

Definition 6.17. A ∈ Rn×n is a nonsingular M-matrix if

A = sI − B, where B ≥ 0 and s > ρ(B). (6.52)

Since ρ(B) ≥ maxi bii (see Section B.11), (6.52) implies that any nonsingular M-
matrix has positive diagonal entries. The representation (6.52) is not unique. We can
take s = maxi aii (see Problem 6.20), which is useful for computational purposes.
160 Matrix Square Root

An archetypal M-matrix is the tridiagonal Toeplitz matrix illustrated for n = 3

by
 
2 −1 0
A =  −1 2 −1  . (6.53)
0 −1 2
It is minus the second difference matrix that arises when discretizing a second deriva-
tive by a central difference, and is symmetric positive definite.
The definition of nonsingular M-matrix ensures that the eigenvalues all have pos-
itive real part and hence that the principal square root exists (see Theorem 1.29).
Since this square root has spectrum in the right half-plane it is a candidate for being
an M-matrix itself. In fact, the principal square root is an M-matrix.

Theorem 6.18 (Alefeld and Schneider). For any nonsingular M-matrix the princi-
pal square root exists, is an M-matrix, and is the only square root that is an M-matrix.

Proof. The first part was shown above. The analysis below of the binomial
iteration (6.41) provides a constructive proof of the second part. For the third part
note that any nonsingular M-matrix has spectrum in the right half-plane and that
there is only one square root with this property by Theorem 1.29.

For any nonsingular M-matrix A we can write

A = s(I − C), C = s−1 B ≥ 0, ρ(C) < 1, (6.54)

and so a splitting of the form used in Section 6.8.1 automatically exists. It follows
that we can use the binomial iteration (6.41) to compute (I − C)1/2 =: I − P . Since
C ≥ 0, the monotonic convergence shown in Theorem 6.13 is in effect. Moreover,
since we showed in Section 6.8.1 that P ≥ 0 and ρ(P ) ≤ ρ(C) < 1, it follows that
I − P , and hence A1/2 , is an M-matrix (which completes the proof of Theorem 6.18).
As mentioned in Section 6.8.1, the Newton–Schulz iteration (6.35) can be used to
compute (I−C)1/2 and will generally be preferable. However, (6.41) has the advantage
that it is structure-preserving: at whatever point the iteration is terminated we have
an approximation s1/2 (I − Pk ) ≈ A1/2 that is an M-matrix; by contrast, the Newton–
Schulz iterates Yk are generally not M-matrices.
The Newton iterations of Section 6.3 are applicable to M-matrices and are structure-
preserving: if A is a nonsingular M-matrix and X0 = A then so are all the iterates Xk
(or, in the case of the CR iteration (6.19), Zk ). This nonobvious property is shown
by Meini [421, ] for (6.19) via the cyclic reduction derivation of the iteration; a
direct proof for the Newton iteration (6.12) does not seem easy.
A numerical experiment illustrates well the differing convergence properties of
the binomial and Newton–Schulz iterations. For the 8 × 8 instance of (6.53), with
s = maxi aii = 2, we find ρ(C) = 0.94. The binomial iteration (6.41) requires
114 iterations to produce a relative residual less than nu in IEEE double precision
arithmetic, whereas the Newton–Schulz iteration requires just 9 iterations. When
the residual tolerance is relaxed to 10−3 , the numbers of iterations are 15 and 6,
respectively. If we increase the diagonal of the matrix from 2 to 4 then ρ(C) = 0.47
and the numbers of iterations are, respectively, 26 and 6 (tolerance nu) and 4 and
3 (tolerance 10−3 ), so iteration (6.41) is more competitive for the more diagonally
dominant matrix.
6.8 Special Matrices 161

A class of matrices that includes the M-matrices is defined with the aid of the
comparison matrix associated with A:

|aii |, i = j,
M (A) = (mij ), mij = (6.55)
−|aij |, i 6= j.

Definition 6.19. A ∈ Cn×n is a nonsingular H-matrix if M (A) is a nonsingular

M-matrix.
An analogue of Theorem 6.18 holds for the subclass of the H-matrices with (real)
positive diagonal entries.

Theorem 6.20 (Lin and Liu). For any nonsingular H-matrix A ∈ Cn×n with pos-
itive diagonal entries the principal square root A1/2 exists and is the unique square
root that is an H-matrix with positive diagonal entries.
Proof. By Problem 6.20 we can write M (A) = sI − B, where B ≥ 0, s > ρ(B),
and s = maxi |aii | = maxi aii . This means that A = sI − C, e where |C|
e = B and
e e
hence ρ(C) ≤ ρ(|C|) = ρ(B) < s. It follows that the eigenvalues of A lie in the
open right half-plane. Hence the principal square root A1/2 exists. We now need
to show that A1/2 is an H-matrix with positive diagonal entries. This can be done
by applying the binomial expansion (6.38) to A = s(I − C), where C = C/s e and
1/2 1/2
ρ(C) < 1. We have A = s (I − P ) where, from (6.42), ρ(P ) ≤ ρ(|P |) ≤ ρ(|C|) =
e < 1. Since maxi pii ≤ ρ(P ) < 1, A1/2 has positive diagonal elements.
s−1 ρ(|C|)
Moreover, M (A1/2 ) = s1/2 (I − |P |) and ρ(|P |) < 1, so A1/2 is an H-matrix.
Finally, A1/2 is the only square root that is an H-matrix with positive diagonal
entries, since it is the only square root with spectrum in the open right half-plane.

For computing the principal square root of an H-matrix with positive diagonal
elements we have the same options as for an M-matrix: the binomial iteration (6.41),
the Newton iterations of Section 6.3, and the Newton–Schulz iteration (6.35). The bi-
nomial iteration no longer converges monotonically. For the behaviour of the Newton
iterations, see Problem 6.25.

6.8.4. Hermitian Positive Definite Matrices

When implementing the Newton iteration variants of Section 6.3 for Hermitian pos-
itive definite A, advantage can be taken of the fact that the iterates are all Her-
mitian positive definite. However, our preferred iteration is based on the fact that
the Hermitian positive definite square root H of A is the Hermitian polar factor of
the Cholesky factor of A: if A = R∗ R (Cholesky factorization: R upper triangular)
and R = U H (polar decomposition: U unitary, H Hermitian positive definite) then
A = HU ∗ U H = H 2 .

Algorithm 6.21. Given a Hermitian positive definite matrix A ∈ Cn×n this algo-
rithm computes H = A1/2 .
1 A = R∗ R (Cholesky factorization).
2 Compute the Hermitian polar factor H of R by applying Algorithm 8.20
to R (exploiting the triangularity of R).
Cost: Up to about 15 23 n3 flops.
162 Matrix Square Root

Algorithm 8.20, involved in step 2, is a Newton method for computing the Her-
mitian polar factor; see Section 8.9. Algorithm 6.21 has the advantage that it it-
erates on R, whose 2-norm condition number is the square root of that of A, and
that it takes advantage of the excellent stability and the ready acceleration possi-
bilities of the polar iteration. The algorithm can be extended to deal with positive
semi definite matrices by using a Cholesky factorization with pivoting [286, ]. The
algorithm is also easily adapted to compute the inverse square root, A−1/2 , at the
same cost: after computing the unitary polar factor U by Algorithm 8.20, compute
A−1/2 = R−1 U (= H −1 ).
An alternative approach is to use the binomial iteration (6.41). If A ∈ Cn×n is
Hermitian positive definite then it has real, positive eigenvalues λn ≤ · · · ≤ λ1 , and
so using (6.39) and (6.40) we can write A = s(I − C), where s = (λ1 + λn )/2 and
ρ(C) = kCk2 = (κ2 (A) − 1)(κ2 (A) + 1) < 1, and this choice of s minimizes kCk2 .
The convergence of (6.41) is monotonic nondecreasing in the positive semidefinite
ordering, where for Hermitian matrices A and B, A ≤ B denotes that B −A is positive
semidefinite (see Section B.12); this can be shown by diagonalizing the iteration
and applying Theorem 6.13 to the resulting scalar iterations. However, unless A
is extremely well conditioned (κ2 (A) < 3, say) the (linear) convergence of (6.41) will
be very slow.
We now return to the Riccati equation XAX = B discussed in Section 2.9, which
generalizes the matrix square root problem. When A and B are Hermitian positive
definite we can use a generalization of Algorithm 6.21 to compute the Hermitian
positive definite solution. For a derivation of this algorithm see Problem 6.21. The
algorithm is more efficient than direct use of the formulae in Section 2.9 or Problem 2.7
and the same comments about conditioning apply as for Algorithm 6.21.

Algorithm 6.22. Given Hermitian positive definite matrices A, B ∈ Cn×n this al-
gorithm computes the Hermitian positive definite solution of XAX = B.

1 A = R∗ R, B = S ∗ S (Cholesky factorizations).
2 Compute the unitary polar factor U of SR∗ using Algorithm 8.20.
3 X = R−1 U ∗ S (by solving the triangular system RX = U S).

Cost: Up to about 19 31 n3 flops.

6.9. Computing Small-Normed Square Roots

We saw in Section 6.1 that the best a priori bound for the relative residual kA −
Xb 2 k/kAk of the computed square root X b of A ∈ Cn×n produced by any method in
b
floating point arithmetic is of the order α(X)u, where α(X) = kXk2 /kAk. Assuming
that we are willing to compute any square root of A, one with minimal kXk might
therefore be preferred, on the grounds that it should give the smallest relative residual.
We therefore consider the question of determining min{ kXk : X 2 = A }. We
begin with an instructive example. Consider
 
ǫ 1 0
A = 0 ǫ 0, 0 < ǫ ≪ 1, (6.56)
0 0 ǫ
6.9 Computing Small-Normed Square Roots 163

which is in Jordan form with a 2 × 2 block and a 1 × 1 block. The two primary square
roots of A are  1/2 1 −1/2 
ǫ 2ǫ 0
X = ± 0 ǫ1/2 0 ,
0 0 ǫ1/2
and kXk is therefore of order ǫ−1/2 . But A also has two families of nonprimary square
roots:  1/2 1 −1/2 
ǫ 2ǫ 0
X = ±W  0 ǫ1/2 0  W −1 ,
0 0 −ǫ1/2
where  
a b c
W = 0 a 0,
0 d e
with the parameters a, b, c, d, e arbitrary subject to W being nonsingular (see Theo-
rems 1.26 and 1.25). Setting a = e = 1, b = 0, c = −1/(2ǫ1/2 ), and d = −c gives the
particular square roots
 1/2 
ǫ 0 1
X = ± 0 ǫ1/2 0 . (6.57)
0 1 −ǫ1/2
These two nontriangular square roots have kXk = O(1), so they are much smaller
normed than the primary square roots. This example shows that the square root of
minimal norm may be a nonprimary square root and that the primary square roots
can have norm much larger than the minimum over all square roots. Of course, if A is
1/2
normal then the minimal possible value kXk2 = kAk2 is attained at every primary
(and hence normal) square root. It is only for nonnormal matrices that there is any
question over the size of kXk.
How to characterize or compute a square root of minimal norm is a difficult open
question. The plausible conjecture that for a matrix with distinct, real, positive
eigenvalues the principal square root will be the one of minimal norm is false; see
Problem 6.22.
For the computation, Björck and Hammarling [73, ] consider applying general
purpose constrained optimization methods. Another approach is to augment the
Schur method with a heuristic designed to steer it towards a minimal norm square
root. Write the equations (6.5) as
1/2
ujj = ±tjj ,
Pj−1
tij − k=i+1 uik ukj (6.58)
uij = , i = j − 1: −1: 1.
uii + ujj
Denote the values uij resulting from the two possible choices of ujj by u+ −
ij and uij .
Suppose u11 , . . . , uj−1,j−1 have been chosen and the first j −1 columns of U have been
computed. The idea is to take whichever sign for ujj leads to the smaller 1-norm for
the jth column of U , a strategy that is analogous to one used in matrix condition
number estimation [276, , Chap. 15].

an upper triangular T ∈ Cn×n

Algorithm 6.23 (small-normed square root). Given √
this algorithm computes a primary square root U = T and attempts to minimize
kU k1 .
164 Matrix Square Root

1/2
1 u11 = t11
2 for j = 2: n
3 Compute from (6.58) u+ −
ij , uij (i = j − 1: −1: 1),
P j P j
s+j =
+ −
i=1 |uij |, and sj =
−
i=1 |uij |.
4 if sj ≤ sj then uij = uij , i = 1: j, else uij = u−
+ − +
ij , i = 1: j.
5 end

Cost: 23 n3 flops.
The cost of Algorithm 6.23 is twice that of a direct computation of T 1/2 with
an a priori choice of the uii . This is a minor overhead in view of the overall cost of
Algorithm 6.3.
For T with distinct eigenvalues, every square root of T is a candidate for compu-
tation by Algorithm 6.23. If T has some repeated eigenvalues then the algorithm will
avoid nonprimary square roots, because these yield a zero divisor in (6.58) for some
i and j.
It is easy to see that when T has real, positive diagonal elements Algorithm 6.23
will assign positive diagonal to U . This choice is usually, but not always, optimal, as
can be confirmed by numerical example.
A weakness of Algorithm 6.23 is that it locally minimizes column norms without
judging the overall matrix norm. Thus the size of the (1, n) element of the square
root, for example, is considered only at the last stage. The algorithm can therefore
produce a square root of norm far from the minimum (as it does on the matrix in
Problem 6.22, for example). Generally, though, it performs quite well.

6.10. Comparison of Methods

We offer a few observations concerning the choice of method for computing a matrix
square root. For general nonsymmetric matrices, the (real) Schur method is the
method of choice if it can be afforded, both because of its optimal stability and
because of its ability to compute any desired primary square root (including a well
conditioned one, as described in the previous section).
The iterative methods may or may not be able to beat the 28 13 n3 flops cost of the
Schur method, depending on how many iterations are required, which in turn depends
on the desired accuracy and the properties of the matrix. Since their flops are mostly
expended in matrix multiplication and matrix inversion, these methods should run
faster, relative to the Schur method, than the flop counts suggest.
For symmetric matrices the Schur method simplifies into computation of a spec-
∗
√ decomposition
tral √ A = QDQ (D = diag(λi ), Q unitary) followed by evaluation of
A = Q diag( λi )Q∗ , which costs 10n3 flops. For Hermitian positive definite ma-
trices that are not very ill conditioned, Algorithm 6.21 is an attractive alternative.
Another alternative in this case is an approximation algorithm of Lu [396, ].
This algorithm unitarily reduces A to tridiagonal form, √ A = QT Q∗ , then approx-
1/2
imates T using a Padé approximant of the function 1 + x expressed in linear
partial fraction form. The cost of the method is 10n3/3 + 5n2 m/2, where m is the
required degree of the Padé approximant (which is computed during the algorithm).
The value of m depends on the required accuracy and the condition number of the
matrix, and m ≪ n is possible, as shown by two well conditioned test problems
in [396, ].
6.11 Involutory Matrices 165

6.11. Involutory Matrices

The identity matrix has infinitely many square roots: the involutory matrices. The
study of involutory matrices has been motivated by an application to cryptography.
In a so-called homogeneous linear substitution cipher, encipherment is done by rep-
resenting groups of characters in a message by numeric vectors and multiplying the
vectors by a matrix (in modular arithmetic). If the matrix is involutory then it can
be used for both encipherment and decipherment, that is, the cipher is self-reciprocal.
The practical use of this idea was proposed by Hill [289, ], who received a US
patent for a machine implementing it in 1932. Bauer [46, , p. 85] notes that “the
importance of Hill’s invention stems from the fact that since then the value of math-
ematical methods in cryptology has been unchallenged.” This method of encryption
is nowadays mainly of pedagogical interest (see, e.g., Moler [436, , Sec. 1.5]).
Potts [476, ] gives some constructions of symmetric involutory matrices. Levine
and Nahikian [381, ] show that A ∈ Cn×n is involutory if and only if it can be
written A = I − QP , where Q ∈ Cn×p , P ∈ Cp×n , and P Q = 2I.
Some particular involutory matrices with explicit representations are worth noting.
These are all nonprimary square roots of I, because the only primary square roots
are ±I. With H = (1/(i + j − 1)) denoting the Hilbert matrix and
   
i n+i−1 n−1
D = diag (−1) i ,
i i−1
both DHand HD are involutory, as noted by Householder and Carpenter [300, ].
An involutory matrix with integer entries mentioned by several authors (e.g., Bellman


[51, , Ex. 20, p. 28]) is the upper triangular matrix U with uij = (−1)i−1 j−1
i−1 ,
which is obtained from the Cholesky factor of the Pascal matrix6 by changing the
signs of alternate rows. For example, for n = 4,
 
1 1 1 1
 0 −1 −2 −3 
U =  0
. (6.59)
0 1 3 
0 0 0 −1
This matrix arises in the stepsize changing mechanism in an ODE code based on
backward differentiation formulae; see Shampine and Reichelt [516, ]. The prin-
cipal submatrices U (i: j, i: j) are also involutory [21, ]. The Cholesky factor of
the Pascal matrix is also an ingredient in the involutory matrix
A = 2(1−n)/2 LDU,
where
   
i−1 n−i+1
lij = , D = diag(1, −2, . . . , (−2)n−1 ), uij = ,
j−1 n−j+1
with L (unit) lower triangular and U is upper triangular. For n = 4,
 
1 3 3 1
 1 1 −1 −1 
A = 2−3/2  1 −1 −1
.
1 
1 −3 3 −1
6 The
`i+j−2´
Pascal matrix has (i, j) entry j−1
; see [276, , Sec. 28.4] for properties of this matrix.
166 Matrix Square Root

This matrix is discussed by Boyd, Micchelli, Strang, and Zhou [80, ], who describe
various applications of matrices from a more general class of “binomial matrices”.
The latter three matrices are available in MATLAB as

gallery(’invol’,n) % D*H
pascal(n,1)’
gallery(’binomial’,n) % Without the 2^((1-n)/2) scale factor.

6.12. Notes and References

The study of matrix square roots began with Cayley [99, ], in one of the first
papers on matrix theory; he gave formulae for square roots of 2 × 2 and 3 × 3 matrices
(see Problem 6.2). He later gave more details of both cases, motivated by a question
of Tait of “finding the square root of a strain” [100, ].
In the engineering literature A1/2 is sometimes used to denote a “square root
factor” of a Hermitian positive definite matrix A: a matrix X, usually understood to
be the Cholesky factor, such that A = X ∗X. See, for example, Kailath, Sayed, and
Hassibi [330, , Sec. 12.1].
The formulae and inequalities of Section 6.1 for κsqrt (X) are from Higham [268,
]. The residual condition number α(X) was first defined and investigated by
Björck and Hammarling [73, ].
Algorithm 6.3 is due to Björck and Hammarling [73, ] (see the quote at the
end of this section). The extension to real arithmetic, Algorithm 6.7, is from Higham
[268, ].
For a derivation of Algorithm 6.5 see Friedland [194, ] or Higham [276, ,
Prob. 1.4]. For a more low level algorithm that scales to avoid overflow and underflow
and is specialized to base 2 arithmetic see Kahan [327, , p, 201].
The scalar version of the Newton iteration (6.12) is known as Heron’s method, since
it is found in the work of the Greek mathematician Heron of Alexandria (1st century
AD). One step of the iteration was used by the Old Babylonians to approximate
square roots (c. 2000–1600 BC) [190, ].
The quadratic convergence of the Newton iteration (6.12) to A1/2 is shown by
Laasonen [366, ] under the assumption that A has real, positive eigenvalues.
Higham [267, ] weakens the assumption to that in Theorem 6.9: that A has no
eigenvalues on R− . Theorem 6.9 is more general than existing results in the literature,
which typically require X0 = A or X0 a positive multiple of I. The link exploited
in the theorem between Newton square root iterates and Newton sign iterates was
pointed out (for X0 = A) by Cheng, Higham, Kenney, and Laub [108, , Thm. 4.1].
A thorough analysis of the behaviour of the full Newton method for (general)
nonlinear systems in floating point arithmetic is given by Tisseur [569, ], who
determines the limiting accuracy and limiting residual and the factors that influence
them. See also Higham [276, , Chap. 25].
The equivalence of the CR iteration (6.19) with Newton’s method is noted by
Meini [421, ] and investigated further by Iannazzo [305, ]. Our derivation of
the scaled IN iteration (6.30) is from [305, ].
The observation that the Newton iteration (6.12) can be used for singular matrices
with semisimple zero eigenvalues is new. The linear rate of convergence identified in
Theorem 6.10 was obtained for the (equivalent) CR iteration by Meini [421, ].
6.12 Notes and References 167

The instability of Newton’s iteration (6.12) was first noted by Laasonen [366, ].
He stated without proof that, for matrices with real, positive eigenvalues, Newton’s
method “if carried out indefinitely, is not stable whenever the ratio of the largest to
the smallest eigenvalue of A exceeds the value 9”. Analysis justifying this claim was
given by Higham [267, ], who obtained the condition (6.24) and also proved the
stability of the DB iteration. Some brief comments on the instability of (6.12) are
given by Liebl [390, ], who computes the eigenvalues of the Fréchet derivative of
the iteration function (Exercise E.10.1-18 of [453, ] summarizes Liebl’s paper).
The stability of the product form of the DB iteration and of the CR iteration is
proved by Cheng, Higham, Kenney, and Laub [108, ] and Iannazzo [305, ],
respectively.
The Padé iterations (6.34) with ℓ = m − 1 are derived and investigated by Higham
[274, ], who proves the stability and instability of particular variants and shows
how to scale the iterations. Theorems 6.11 and 6.12 are from Higham, Mackey,
Mackey, and Tisseur [283, , Thms. 4.5, 5.3]. Our proof of Theorem 6.12 is
much shorter than the one in [283, ], which does not exploit the stability of the
underlying sign iterations.
Using the binomial expansion (6.38) to compute matrix square roots is suggested
by Waugh and Abel [610, ] in connection with the application discussed in Sec-
tion 2.3 involving roots of transition matrices. The binomial expansion is also inves-
tigated by Duke [170, ].
The binomial iteration (6.41) is discussed for C ≥ 0 by Alefeld and Schneider [8,
] and Butler, Johnson, and Wolkowicz [86, ], where it is used for theoretical
purposes rather than suggested as a computational tool. Theorem 6.13 is essentially
found in these papers. Albrecht [7, ] suggests the use of (6.41) for Hermitian
positive definite matrices. Theorem 6.14 is new.
The iteration (6.48) was suggested for symmetric positive definite matrices by
Pulay [480, ] (see also Bellman [51, , Ex. 1, p. 334]), but no convergence
result is given there. Theorem 6.15 is new. The approximation B1 obtained after one
iteration of (6.48) for symmetric positive definite A and D = diag(A) has been used
in a stochastic differential equations application by Sharpa and Allen [517, ].
The iteration (6.51) can be traced back to Visser [602, ], who used it to show
that a bounded positive definite self-adjoint operator on Hilbert space has a bounded
positive definite self-adjoint square root without applying the spectral theorem. This
use of the iteration can also be found in functional analysis texts, such as those
of Debnath and Mikusiński [144, ], Riesz and Sz.-Nagy [492, , Sec. 104],
Schechter [502, , Sec. 8.2], and Halmos [244, , Prob. 121]. For a generalization
of Visser’s technique to pth roots see Problem 7.7. Theorem 6.16 is new. Elsner [176,
] obtains the stability result derived at the end of Section 6.8.2. Iteration (6.51)
and its stability are also studied for symmetric positive definite matrices by Liebl
[390, ] and Babuška, Práger, and Vitásek [25, , Sec. 2.4.5].
Theorem 6.18 is an interpretation of results of Alefeld and Schneider [8, ],
who also treat square roots of singular M-matrices. Theorem 6.20 is proved for real
H-matrices by Lin and Liu [393, ].
Algorithm 6.21 for Hermitian positive definite matrices is due to Higham [266,
]. Algorithm 6.22 is suggested by Iannazzo [307, ].
That the matrix (6.56) has a small-normed square root (6.57) is pointed out by
Björck and Hammarling [73, ]. Algorithm 6.23 is from Higham [268, ].
For more on linear substitution ciphers see Bauer [46, ].
168 Matrix Square Root

Early contributions on involutory matrices are those of Cayley [99, ], [100,
] and Sylvester [556, ]. The number of involutory matrices over a finite field
is determined by Hodges [293, ].
Finally, an application in which the inverse matrix square root appears is the
computation of tight windows of Gabor frames [319, ].

Problems
6.1. Prove the integral formula (6.1) by using the integral (5.3) for the matrix sign
function together with (5.4).
6.2. (Cayley [99, ], [100, ], Levinger [382, ]) Let A ∈ C2×2 . If A has
distinct complex conjugate eigenvalues then the square roots of A can be obtained
explicitly from the formulae following Lemma 6.4. More generally, the following
approach can be considered. By the Cayley–Hamilton theorem any 2 × 2 matrix X
satisfies X 2 − trace(X)X + det(X)I = 0. Let X 2 = A. Then
p
A − trace(X)X + det(A) I = 0. (6.60)

Taking the trace gives

p
trace(A) − trace(X)2 + 2 det(A) = 0. (6.61)

We can solve (6.61) for trace(X) and then solve (6.60) for X to obtain
p
A + det(A) I
X=q p .
trace(A) + 2 det(A)

Discuss the effectiveness of this procedure.

6.3. Construct a parametrized matrix A(ǫ) such that kA(ǫ)1/2 k2 /kA(ǫ)k → ∞ as
ǫ → 0.
6.4. Show that for any primary square root X of A ∈ Cn×n , range(X) = range(A).
Show that this equality does not necessarily hold for a nonprimary square root.
6.5. Explain the behaviour of the Schur algorithm (Algorithm 6.3) when A is singular.
6.6. Show that in floating point arithmetic Algorithms 6.3 and 6.7 produce a com-
puted square root Ub of, respectively, a triangular matrix T ∈ Cn×n and a quasi-
b 2 = T + ∆T with |∆T | ≤ γ
triangular matrix T ∈ Rn×n , satisfying U b |2 for Algo-
en |U
rithm 6.3 and k∆T kF ≤ γ b k for Algorithm 6.7.
en kU 2
F

6.7. Let A = diag(a, b) ∈ C2×2 with a, b ∈

/ R− . The Newton iteration (6.12) converges
1/2
to A by Theorem 6.9. Show that if a + b = −2, the full Newton iteration (6.11)
with X0 = A breaks down on the second iteration.
6.8. Investigate the behaviour of the Newton iteration (6.12) for
   
1 0 1 θ
A= , X 0 = , θ 6= 0.
0 µ2 0 µ

Note that this X0 does not commute with A.

Problems 169

6.9. (Elsner [176, ]; the ideas in this problem have been generalized to algebraic
Riccati equations: see Mehrmann [418, , Thm. 11.3] and Lancaster and Rodman
[370, , Thms. 9.1.1, 9.2.1].)
(a) Let > and ≥ denote the orderings on Hermitian matrices defined in Section B.12.
Show that if A > 0 and X0 > 0 then the iterates from the full Newton iteration (6.11)
satisfy
0 < A1/2 ≤ · · · ≤ Xk+1 ≤ Xk ≤ · · · ≤ X1 , A ≤ Xk2 , k ≥ 1,
and deduce that Xk converges monotonically to A1/2 in the positive semidefinite
ordering. Hint: first show that if C > 0 and H ≥ 0 (H > 0) then the solution of the
Sylvester equation XC + CX = H satisfies X ≥ 0 (X > 0).
(b) Note that X0 is an arbitrary positive definite matrix: it need not commute with
A. What can be said about the convergence of the simplified Newton iteration (6.12)?
(c) Suppose A is not Hermitian positive definite but that there is a nonsingular Z
such that Z −1 AZ and Z −1 X0 Z are Hermitian positive definite. How can the result
of (a) be applied?
6.10. Explain the behaviour of the Newton iteration (6.12) when A has an eigenvalue
on R− .
6.11. (Iannazzo [305, ]) Give another derivation of the DB iteration (6.15) as
follows. “Symmetrize” (6.12) by writing it as
1

Xk+1 = Xk + A1/2 Xk−1 A1/2 , X0 = A.
2
Heuristically, this symmetrization should improve the numerical stability. Remove
the unwanted square roots by introducing a new variable Yk = A−1/2 Xk A−1/2 .
6.12. In the DB iteration (6.15) it is tempting to argue that when computing Yk+1 the

latest X iterate, Xk+1 , should be used instead of Xk , giving Xk+1 = 12 Xk + Yk−1
−1


and Yk+1 = 12 Yk + Xk+1 . (This is the same reasoning that produces the Gauss–
Seidel iteration from the Jacobi iteration.) What effect does this change have on the
convergence of the iteration?
6.13. From (6.25) we obtain the relation for the DB iteration function G:
    2 !
1 E − A1/2 F A1/2 E
1/2
G(A + E, A −1/2
+ F) = −1/2 −1/2
+O
F .
2 F −A EA

Explain how the quadratic convergence of the DB iteration can be seen from this
relation. In other words, reconcile the stability analysis with the convergence analysis.
6.14. In the coupled Padé iteration (6.34) we know that Yk = AZk for all k, so we
can rewrite the Yk recurrence in the uncoupled form
Yk+1 = Yk pℓm (1 − A−1 Yk2 ) qℓm (1 − A−1 Yk2 )−1 , Y0 = A. (6.62)
What are the pros and cons of this rearrangement? Note that for ℓ = 0, m = 1, and
A ← A−1 , (6.62) is the iteration
Yk+1 = 2Yk (I + Yk AYk )−1 (6.63)
for computing A−1/2 , which appears occasionally in the literature [518, ].
170 Matrix Square Root

6.15. Discuss whether the requirement ρ(C) < 1 in Theorem 6.13 can be relaxed by
making use of Theorem 6.14.
6.16. Show that Pulay’s iteration (6.48) is equivalent to the modified Newton itera-
tion (6.46) with X0 = D1/2 , and specifically that Xk = D1/2 + Bk .
6.17. Show that for A ∈ Cn×n with real, positive eigenvalues 0 < λn ≤ λn−1 ≤
· · · ≤ λ1 , mins∈R { ρ(C) : A = s(I − C) } is attained at s = (λ1 + λn )/2, and ρ(C) =
(λ1 − λn )/(λ1 + λn ).
6.18. Show that for A ∈ Cn×n , mins∈C { kCkF : A = s(I − C) } is attained at
s = trace(A∗ A)/ trace(A∗ ). (Cf. Theorem 4.21 (a).)
6.19. Show that Theorem 6.16 can be extended to A having a semisimple zero eigen-
value.
6.20. Show that if A is an M-matrix then in the representation (6.52) we can take
s = maxi aii .
6.21. Derive Algorithm 6.22 for solving XAX = B.
6.22. Show that for the matrix
 
1 −1 1/ǫ2
T =  0 (1 + ǫ)2 1 , 0 < ǫ ≪ 1,
0 0 (1 + 2ǫ)2

the principal square root is not the square root of minimal norm.
6.23. (Meyer [426, , Ex. 3.6.2]) Show that for all A ∈ Cm×n and B ∈ Cn×m the
matrix  
I − BA B
2A − ABA AB − I
is involutory.
6.24. (Research problem) Which of the DB, product DB, and IN iterations is to
be preferred in which circumstances?
6.25. (Research problem) When the Newton iteration (6.12) is applied to a non-
singular H-matrix A with positive diagonal entries are all the iterates Xk nonsingular
H-matrices with positive diagonal entries?
6.26. (Research problem) Develop algorithms for computing directly the Cholesky
factors of (a) the principal square root of a Hermitian positive definite matrix and
(b) the Hermitian positive definite solution X of XAX = B, where A and B are
Hermitian positive definite.
Problems 171

The unexpectedly rapid convergence of Newton’s method

led us to discover the simple method of the next section.
— ÅKE BJÖRCK and SVEN HAMMARLING, A Schur Method
for the Square Root of a Matrix (1983)

Although no proof of convergence will be given,

the procedure converged rapidly in all cases examined by us.
— PÉTER PULAY, An Iterative Method for the Determination of
the Square Root of a Positive Definite Matrix (1966)

Some simplifications in the handling of a digraphic system based on a

linear transformation would be obtained if we
designed the system so that the
enciphering matrix is its own inverse.
— ABRAHAM SINKOV, Elementary Cryptanalysis: A Mathematical Approach (1966)

Blackwell: You know the algorithm for calculating the square root? . . .
It occurred to me that maybe this algorithm would work for positive definite matrices.
You take some positive definite X , add it to SX −1 and divide by two.
The question is: Does this converge to the square root of X . . . .
In a particular example, the error at first was tremendous,
then dropped down to about .003.
Then it jumped up a bit to .02, then jumped up quite a bit to .9,
and then it exploded. Very unexpected.. . .
It turns out that the algorithm works provided that the matrix you start with
commutes with the matrix whose square root you want.
You see, it’s sort of natural because you
have to make a choice between SX −1 and X −1 S ,
but of course if they commute it doesn’t make any difference.
Of course I started out with the identity matrix and it should commute with anything.
So what happened?
MP: You must have been having some kind of roundoff.
Blackwell: Exactly! If the computer had calculated exactly it would have converged.
The problem is that the matrix the computer used didn’t quite commute.
— DAVID BLACKWELL7 , in Mathematical People: Profiles and Interviews (1985)
7 From [6, ].
Chapter 7
Matrix pth Root

X is a pth root of A ∈ Cn×n if X p = A. The pth root of a matrix, for p > 2, arises
less frequently than the square root, but nevertheless is of interest both in theory and
in practice. One application is in the computation of the matrix logarithm through
the relation log A = p log A1/p (see Section 11.5), where p is chosen so that A1/p
can be well approximated by a polynomial or rational function. Here, A1/p denotes
the principal pth root, defined in the next section. Roots of transition matrices are
required in some finance applications (see Section 2.3). Related to the pth root is the
matrix sector function sectp (A) = (Ap )−1/p A discussed in Section 2.14.3.
The matrix pth root is an interesting object of study because algorithms and
results for the case p = 2 do not always generalize easily, or in the manner that they
might be expected to.
In this chapter we first give results on existence and classification of pth roots.
Then we generalize the Schur method described in Chapter 6 for the square root.
Newton iterations for the principal pth root and its inverse are explored, leading to a
hybrid Schur–Newton algorithm. Finally, we explain how the pth root can be obtained
via the matrix sign function.
Throughout the chapter p is an integer. Matrix roots Aα with α a real number
can be defined (see Problem 7.2) but the methods of this chapter are applicable only
when α is the reciprocal of an integer.

7.1. Theory
We summarize some theoretical results about matrix pth roots, all of which generalize
results for p = 2 already stated.

Theorem 7.1 (classification of pth roots). Let the nonsingular matrix A ∈ Cn×n
have the Jordan canonical form Z −1 AZ = J = diag(J1 , J2 , . . . , Jm ), with Jk =
(j )
Jk (λk ), and let s ≤ m be the number of distinct eigenvalues of A. Let Lk k =
(j )
Lk k (λk ), k = 1: m, denote the p pth roots given by (1.4), where jk ∈ {1, 2, . . . , p}
denotes the branch of the pth root function. Then A has precisely ps pth roots that
are primary functions of A, given by
(j ) (j )
Xj = Z diag(L1 1 , L2 2 , . . . , L(jm)
m )Z
−1
, j = 1: ps ,

corresponding to all possible choices of j1 , . . . , jm , subject to the constraint that ji = jk

whenever λi = λk .
If s < m, A has nonprimary pth roots. They form parametrized families
(j ) (j )
Xj (U ) = ZU diag(L1 1 , L2 2 , . . . , L(jm)
m )U
−1 −1
Z , j = ps + 1: pm ,

173
174 Matrix pth Root

where jk ∈ {1, 2, . . . , p}, U is an arbitrary nonsingular matrix that commutes with J,

and for each j there exist i and k, depending on j, such that λi = λk while ji 6= jk .
Proof. The proof is similar to that of Theorem 1.26; see Smith [530, ].

Theorem 7.2 (principal pth root). Let A ∈ Cn×n have no eigenvalues on R− . There
is a unique pth root X of A all of whose eigenvalues lie in the segment { z : −π/p <
arg(z) < π/p }, and it is a primary matrix function of A. We refer to X as the
principal pth root of A and write X = A1/p . If A is real then A1/p is real.
Proof. The proof is entirely analogous to that of Theorem 1.29.

Theorem 7.3 (existence of pth root). A ∈ Cn×n has a pth root if and only if the
“ascent sequence” of integers d1 , d2 , . . . defined by

di = dim(null(Ai )) − dim(null(Ai−1 ))

has the property that for every integer ν ≥ 0 no more than one element of the sequence
lies strictly between pν and p(ν + 1).
Proof. See Psarrakos [479, ].
We note the integral representation
Z ∞
p sin(π/p)
A1/p = A (tp I + A)−1 dt, (7.1)
π 0

generalizing (6.1), which holds for any real p > 1. For an arbitrary real power, see
Problem 7.2.
Our final result shows that a pth root can be obtained from the invariant subspace
of a block companion matrix associated with the matrix polynomial λp I − A.

Theorem 7.4 (Benner, Byers, Mehrmann, and Xu). Let A ∈ Cn×n . If the columns
of U = [U1∗ , . . . , Up∗ ]∗ ∈ Cpn×n span an invariant subspace of
 
0 I
 0 I 
 .. .. 
 
C= . .  ∈ Cpn×pn , (7.2)
 . 
 .. I 
A 0
that is, CU = U Y for some nonsingular Y ∈ Cn×n , and U1 is nonsingular, then
X = U2 U1−1 is a pth root of A.
Proof. The equation CU = U Y yields

U2 = U1 Y, ..., Up = Up−1 Y, AU1 = Up Y.

Hence

A = Up Y U1−1 = Up−1 Y 2 U1−1 = · · · = U2 Y p−1 U1−1 = U1 Y p U1−1

= U1 U1−1 U2 )p U1−1 = U2 U1−1 )p .
7.2 Schur Method 175

7.2. Schur Method

Let A ∈ Rn×n have the real Schur decomposition A = QRQT , where Q is orthogonal
and R is upper quasi-triangular and block m × m. The (real) pth root problem now
reduces to computing an upper quasi-triangular U such that U p = R. For p = 2 we
were able in Section 6.2 to directly obtain a recurrence for the Uij . For p > 2 this
approach can be generalized by setting up p − 1 coupled recurrences.
We begin by considering the cube root case: U 3 = R. Let V = U 2 . In order to
compute U we will need to compute V as well. The equation U V = R gives
j−1
X
Rij = Uii Vij + Uij Vjj + Uik Vkj
k=i+1
(1)
=: Uii Vij + Uij Vjj + Bij ,

and likewise V = U 2 gives

j−1
X
Vij = Uii Uij + Uij Ujj + Uik Ukj
k=i+1
(0)
=: Uii Uij + Uij Ujj + Bij .

Substituting the latter equation for Vij into the equation for Rij gives
(0)
(1)
Rij = Uii Uii Uij + Uij Ujj + Bij + Uij Vjj + Bij ,

which may be rearranged, using Vii = Uii2 , as

(0) (1)
Vii Uij + Uii Uij Ujj + Uij Vjj = Rij − Uii Bij − Bij .

This is a generalized Sylvester equation for Uij . The right-hand side depends only on
blocks of U and V lying to the left of and below the (i, j) blocks. Hence, just as in
the square root case, we can solve for Uij and Vij a block column at a time or a block
superdiagonal at a time.

Algorithm 7.5 (Schur method

√ A ∈ Rn×n this
for cube root). Given a nonsingular √
3
algorithm computes X = A via a real Schur decomposition, where 3 · denotes any
real primary cube root.

1 Compute a real Schur decomposition A = QRQT , where R is block m × m.

2 for j = 1: m
p
3 Ujj = 3 Rjj
2
4 Vjj = Ujj
5 for i = j − 1: −1: 1
Pj−1
6 B0 = k=i+1 Uik Ukj
Pj−1
7 B1 = k=i+1 Uik Vkj
8 Solve Vii Uij + Uii Uij Ujj + Uij Vjj = Rij − Uii B0 − B1 for Uij .
9 Vij = Uii Uij + Uij Ujj + B0
10 end
11 end
12 X = QU QT
176 Matrix pth Root

Cost: 25n3 flops for the Schur decomposition plus 2n3/3 for U and 3n3 to form X:
(0)
28 23 n3 flops in total. (The cost of forming U is dominated by that of computing Bij
(1)
and Bij , which is essentially that of forming U 2 and U V , respectively.)
This approach can be generalized to obtain an algorithm for pth roots. Let U p =
R. The idea is to define V (k) = U k+1 , k = 0: p−2 and obtain recurrences for the blocks
of the V (k) using the relations U V (k) = V (k+1) , k = 0: p − 3 and U V (p−2) = R. We
simply state the algorithm; for the derivation, which involves some tedious algebra,
see Smith [530, ].

Algorithm 7.6 (Schur method

√ for pth root). Given a nonsingular A√ ∈ Rn×n this
p
algorithm computes X = A via a real Schur decomposition, where p · denotes any
real primary pth root.

1 Compute a real Schur decomposition A = QRQT , where R is block m × m.

2 for j = 1: m√
p
3 Ujj = Rjj
(k) k+1
4 Vjj = Ujj , k = −1: p − 2
5 for i = j − 1: −1: 1
6 for k = 0: p − 2
Pj−1 (k)
7 Bk = ℓ=i+1 Uiℓ Vℓj
8 end
Pp−1 (p−2−k) (k−1) Pp−2 (p−3−k)
9 Solve k=0 Vii Uij Vjj = Rij − k=0 Vii Bk for Uij .
10 for k = 0: p − 2
(k) Pk (k−ℓ−1) (ℓ−1) Pk−1 (k−2−ℓ)
11 Vij = ℓ=0 Vii Uij Vjj + ℓ=0 Vii Bℓ
12 end
13 end
14 end
15 X = QU QT

Cost: 25n3 flops for the Schur decomposition plus (p − 1)n3/3 for U (which is essen-
tially the cost of computing the Bk ) and 3n3 to form X: (28 + (p − 1)/3)n3 flops in
total.
√ Two details in Algorithms 7.5 and 7.6 remain to be described: how to compute
p
Rjj and how to √ solve the generalized Sylvester equations.
p
A formula for Rjj , for Rjj ∈ R2×2 , can be obtained by adapting the approach
used for p = 2 in Section 6.2. Since Rjj has distinct eigenvalues we can write (as in
the proof of Lemma 6.4)
 
1 0
Z −1 Rjj Z = diag(λ, λ) = θI + iµK, K= .
0 −1

Hence Rjj = θI + µW , where W = iZKZ −1 , and since θ, µ ∈ R it follows that

W ∈ R2×2 . Let (α + iβ)p = θ + iµ. Then a pth root of Rjj is given by
β
Ujj = αI + βW = αI + (Rjj − θI).
µ
If, as is most likely, the principal pth root is required, α and β can be obtained from
the polar form θ + iµ = reiφ as α + iβ = r1/p eiφ/p . Recall that θ and µ are given by
(6.8).
7.3 Newton’s Method 177

The generalized Sylvester equation at line 9 of Algorithm 7.6 can be solved by

applying the vec operator to obtain the linear system
p−1
X   p−2
X 
(k−1) T (p−2−k) (p−3−k)
Vjj ⊗ Vii · vec(Uij ) = vec Rij − Vii Bk ,
k=0 k=0

which has dimension 1, 2, or 4 and can be solved by Gaussian elimination with partial
pivoting. To check the nonsingularity of the coefficient matrix we note that, by (B.17),
this matrix has eigenvalues
 p p
p−1
X  λr − µs , λ 6= µ ,
k p−1−k r s
λr µs = λr − µs
 p−1
k=0 pλr , λr = µs ,
where λr ∈ Λ(Ujj ) and µs ∈ Λ(Uii ), with λr and µs nonzero by the assumption that
A is nonsingular. Hence nonsingularity holds when λpr 6= µps for all r and s, which
requires that λr 6= µs e2πik/p , k = 1: p − 1. This condition is certainly satisfied if
any eigenvalue appearing in two different blocks Rii and Rjj is mapped to the same
1/p 1/p
pth root in Rii and Rjj . Hence the algorithm will succeed when computing any
primary pth root.
Algorithm 7.6 requires storage for the 2p − 2 intermediate upper quasi-triangular
matrices V (k) and B (k) . For large p, this storage can be a significant cost. However,
if p is composite the pth root can be obtained by successively computing the roots
given by the prime factors of p, which saves both storage and computation.
If A is nonsingular and has a negative real eigenvalue then the principal pth root
is not defined. For √ odd p, Algorithm 7.6 can nevertheless compute a real, primary
pth root by taking p λ = −|λ|1/p when λ < 0. For even p, there is no real, primary
pth root in this situation.
The stability properties of Algorithm 7.6 are entirely analogous to those for the
square root case. The computed X b satisfies

b p kF
kA − X
≤γ b
en3 αF (X), (7.3)
kAkF
where
kXkp kXkp
α(X) = = ≥ 1.
kAk kX p k
The quantity α(X) can be regarded as a condition number for the relative residual of
X, based on the same reasoning as used in Section 6.1.
Algorithms 7.5 and 7.6 are readily specialized to use the complex Schur decompo-
sition, by setting all blocks to be 1 × 1.

7.3. Newton’s Method

Given the important role Newton’s method plays in iterations for the matrix sign
decomposition and matrix square root, it is natural to apply it to the matrix pth
root. Newton’s method for the system X p − A = 0 defines an increment Ek to an
iterate Xk by
Xp
Xkp−i Ek Xki−1 = A − Xkp . (7.4)
i=1
178 Matrix pth Root

No O(n3 ) flops methods are known for solving this generalized Sylvester equation
for p > 2 and so it is necessary to simplify the equation by using commutativity
properties, just as in Lemma 6.8. When X0 commutes with A so do all the Xk , and
Newton’s method can be written (see Problem 7.5)

Newton iteration (matrix pth root):

1 
Xk+1 = (p − 1)Xk + Xk1−p A , X0 A = AX0 . (7.5)
p

The convergence properties of the iteration, and their dependence on X0 , are much
more complicated than in the square root case.
The most obvious choice, X0 = A, is unsatisfactory because no simple conditions
are available that guarantee convergence to the principal pth root for general A. To
illustrate, we consider the scalar case. Note that if we define yk = a−1/p xk then (7.5)
reduces to
1 
yk+1 = (p − 1)yk + yk1−p , y0 = a−1/p x0 , (7.6)
p
which is the Newton iteration for a pth root of unity with a starting value depending
on a and x0 . Figure 7.1 plots for p = 2: 5, a = 1, and y0 ranging over a 400 × 400
grid with Re y0 , Im y0 ∈ [−2.5, 2.5], the root to which yk from (7.6) converges, with
each root denoted by a different grayscale from white (the principal root) to black.
Convergence is declared if after 50 iterations the iterate is within relative distance
10−13 of a root; the relatively small number of points for which convergence was not
observed are plotted white. For p = 2, the figure confirms what we already know from
Chapter 6: convergence is obtained to whichever square root of unity lies nearest y0 .
But for p ≥ 2 the regions of convergence have a much more complicated structure,
involving sectors with petal-like boundaries. This behaviour is well known and is
illustrative of the theory of Julia sets of rational maps [468, ], [511, ]. In
spite of these complications, the figures are suggestive of convergence to the principal
root of unity when y0 lies in the region

{ z : |z| ≥ 1, −π/(2p) < arg(z) < π/(2p) },

which is marked by the solid line in the figure for p ≥ 3. In fact, this is a region of
convergence, as the next theorem, translated back to (7.5) with varying a and x0 = 1
shows. We denote by R+ the open positive real axis.

Theorem 7.7 (convergence of scalar Newton iteration). For all p > 1, the iteration
1 
xk+1 = (p − 1)xk + x1−p
k a , x0 = 1,
p
converges quadratically to a1/p if a belongs to the set

S = { z ∈ C : Re z > 0 and |z| ≤ 1 } ∪ R+ . (7.7)

Proof. The rather complicated proof is given by Iannazzo [306, ].

Corollary 7.8 (convergence of matrix Newton iteration). Let A ∈ Cn×n have no eigen-
values on R− . For all p > 1, the iteration (7.5) with X0 = I converges quadratically
to A1/p if each eigenvalue λi of A belongs to the set S in (7.7).
7.3 Newton’s Method 179

p=2 p=3
2.5 2.5
2 2
1.5 1.5
1 1
0.5 0.5
0 0
−0.5 −0.5
−1 −1
−1.5 −1.5
−2 −2
−2.5 −2.5
−2 −1 0 1 2 −2 −1 0 1 2
p=4 p=5
2.5 2.5
2 2
1.5 1.5
1 1
0.5 0.5
0 0
−0.5 −0.5
−1 −1
−1.5 −1.5
−2 −2
−2.5 −2.5
−2 −1 0 1 2 −2 −1 0 1 2

Figure 7.1. Convergence of the Newton iteration (7.6) for a pth root of unity. Each point
y0 in the region is shaded according to the root to which the iteration converges, with white
denoting the principal root. Equivalently, the plots are showing convergence of (7.5) in the
scalar case with a = 1 and the shaded points representing x0 .

Proof. Theorems 7.7 and 4.15 together imply that Yk has a limit Y∗ whose
eigenvalues are the principal pth roots of the eigenvalues of A. The limit clearly
satisfies Y∗p = A, and so is a pth root of A. By Theorem 7.2, the only pth root of A
having the same spectrum as Y∗ is the principal pth root, and so Y∗ = A1/p .

The restrictive condition on the eigenvalues of A in Corollary 7.8 can be overcome

quite simply with some preprocessing and postprocessing and use of the identity

2
A1/p = (A1/2 )1/p .

Algorithm 7.9 (Matrix pth root via Newton iteration). Given A ∈ Cn×n having no
eigenvalues on R− this algorithm computes X = A1/p using the Newton iteration.

1 B = A1/2
2 C = B/θ, where θ is any upper bound for ρ(B) (e.g., θ = kBk)
3 Use the
 Newton iteration (7.5) with X0 = I to compute
C 2/p , p even,
X=
2
C 1/p , p odd.
4 X ← kBk2/p X
180 Matrix pth Root

To see why the algorithm works, note that the eigenvalues of B, and hence also
C, lie in the open right half-plane, and ρ(C) ≤ 1. Therefore C satisfies the conditions
of the corollary.
Unfortunately, instability vitiates the convergence properties of iteration (7.5) in
practical computation—indeed we have already seen in Chapters 4 and 6 that this is
the case for p = 2. Analysis similar to that in Section 6.4.1 (see Problem 7.12) shows
that the eigenvalues of Lg (A1/p ), where g(X) = p−1 [(p − 1)X + X 1−p A], are of the
form
p−1 
X r/p !
1 λi
(p − 1) − , i, j = 1: n, (7.8)
p r=1
λj
where the λi are the eigenvalues of A. To guarantee stability we need these eigenvalues
to be less than 1 in modulus, which is a very severe restriction on A.
Inspired by Problem 6.11, we might try to obviate the instability by rewriting the
iteration in the more symmetric form (using the commutativity of Xk with A1/p )
1

Xk+1 = (p − 1)Xk + A1/p Xk−1 A1/p Xk−1 . . . A1/p Xk−1 A1/p
p
1

= (p − 1)Xk + (A1/p Xk−1 )p−1 A1/p . (7.9)
p
This iteration is stable. Denoting the iteration function by g, it is straightforward to
show that Lg (A1/p , E) = 0, and stability is immediate. For practical computation,
however, we need another way of rewriting the Newton iteration that does not involve
A1/p . Taking a cue from the product form DB iteration (6.17) for the square root,
we define Mk = Xk−p A. Then it is trivial to obtain, using the mutual commutativity
of A, Xk , and Mk ,

Coupled Newton iteration (matrix pth root):

 
(p − 1)I + Mk
Xk+1 = Xk , X0 = I,
p
 −p (7.10)
(p − 1)I + Mk
Mk+1 = Mk , M0 = A.
p

When Xk → A1/p , Mk → I. The cost of (7.10) is (θ log2 p + 14/3)n3 flops per

iteration, where θ ∈ [1, 2], assuming the pth power in (7.10) is formed by binary
powering (Algorithm 4.1). This will be less than the O(p n3 ) cost of the Schur method
if p is large and not too many iterations are required.
To test for stability we write the iteration function as
   
(p − 1)I + M
 X p  
G(X, M ) =  
 (p − 1)I + M −p
.

M
p
It is easy to show that
 X  
I p E
Lg (X, I; E, F ) = . (7.11)
0 0 F

Hence Lg (A1/p , I) is idempotent and the iteration is stable.

7.4 Inverse Newton Method 181

7.4. Inverse Newton Method

Another Newton iteration, this time for the inverse pth root, is obtained by applying
Newton’s method to the system X −p − A = 0:

Inverse Newton iteration (inverse matrix pth root):

1 
Xk+1 = (p + 1)Xk − Xkp+1 A , X0 A = AX0 . (7.12)
p

(Of course, (7.12) is (7.5) with p ← −p.) For p = 1 this is the well-known Newton–
Schulz iteration for the matrix inverse [512, ] and the residuals Rk = I − Xkp A
satisfy Rk+1 = Rk2 (see Problem 7.8). The next result shows that a p-term recursion
for the residuals holds in general.

Theorem 7.10 (Bini, Higham, and Meini). The residuals Rk = I −Xkp A from (7.12)
satisfy
p+1
X
Rk+1 = ai Rki , (7.13)
i=2
Pp+1
where the ai are all positive and i=2 ai = 1. Hence if 0 < kR0 k < 1 for some
consistent matrix norm then kRk k decreases monotonically to 0 as k → ∞, with
kRk+1 k < kRk k2 .

Proof. We have Xk+1 = p−1 Xk (pI + Rk ). Since Xk commutes with A, and hence
with Rk , we obtain

1
Rk+1 = I − (I − Rk )(pI + Rk )p (7.14)
pp
1h p i
X p
=I− p p I+ bi Rki − Rkp+1
p i=1
" p #
1 X p+1
i
=− p bi R k − R k ,
p i=1

where
   
p p−i p
bi = p − pp−i+1
i i−1
    
p−i p p
=p − p
i i−1
 
p−i p! p! p
=p −
i!(p − i)! (i − 1)!(p − i + 1)!
 
p−i p! 1 p
=p − .
(i − 1)!(p − i)! i (p − i + 1)

It is easy to see that b1 = 0 and bi < 0 for i ≥ 2. Hence (7.13) holds, with ai > 0 for
Pp+1
all i. By setting Rk ≡ I in (7.13) and (7.14) it is easy to see that i=2 ai = 1.
182 Matrix pth Root

If 0 < kRk k < 1, then taking norms in (7.13) yields

p+1
X p+1
X
kRk+1 k ≤ |ai | kRk ki < kRk k2 |ai | = kRk k2 < kRk k.
i=2 i=2

Since 0 < kR0 k < 1, by induction the kRk k form a monotonically decreasing sequence
that converges to zero.
If X0 does not commute with A then little can be said about the behaviour of the
residuals for p ≥ 2; see Problems 7.8 and 7.9.
Theorem 7.10 does not immediately imply the convergence of Xk (except in the
scalar case). We can conclude that the sequence of pth powers, {Xkp }, is bounded, but
the boundedness of {Xk } itself does not follow when n > 1. If the sequence {Xk } is
indeed bounded then by writing (7.12) as Xk+1 − Xk = p1 Xk (I − Xkp A) = p1 Xk Rk we
see that {Xk } is a Cauchy sequence and thus converges (quadratically) to a matrix
X∗ . This limit satisfies I − X∗p A = 0 and so is some inverse pth root of A, but not
necessarily the inverse principal pth root.
The next result proves convergence when all the eigenvalues of A lie on the real
interval (0, p + 1) and X0 = I.

Theorem 7.11 (Bini, Higham, and Meini). Suppose that all the eigenvalues of A are
real and positive. Then iteration (7.12) with X0 = I converges to A−1/p if ρ(A) <
p + 1. If ρ(A) = p + 1 the iteration does not converge to the inverse of any pth root
of A.

Proof. By the same reasoning as in the proof of Corollary 7.8 it suffices to analyze
the convergence of the iteration on the eigenvalues of A. We therefore consider the
scalar iteration
1 
xk+1 = (1 + p)xk − xp+1 k a , x0 = 1, (7.15)
p
with a > 0. Let yk = a1/p xk . Then
1 
yk+1 = (1 + p)yk − ykp+1 =: f (yk ), y0 = a1/p , (7.16)
p

and we need to prove that yk → 1 if y0 = a1/p < (p + 1)1/p . We consider two cases.
If yk ∈ (0, 1) then  
1 − ykp
yk+1 = yk 1 + > yk .
p
Moreover, since
p+1
f (0) = 0, f (1) = 1, f ′ (y) = p (1 − y p ) > 0 for y ∈ [0, 1),

it follows that f (y) < 1 for y ∈ [0, 1). Hence yk < yk+1 < 1 and so the yk form
a monotonically increasing sequence tending to a limit in (0, 1]. But the only fixed
points of the iteration are the roots of unity, so the limit must be 1. Now suppose
y0 ∈ (1, (p + 1)1/p ). We have f (1) = 1 and f ((p + 1)1/p ) = 0, and f ′ (y) < 0 for y > 1.
It follows that f maps (1, (p + 1)1/p ) into (0, 1) and so after one iteration y1 ∈ (0, 1)
and the first case applies. The last part of the result follows from f ((p + 1)1/p ) = 0
and the fact that 0 is a fixed point of the iteration.
7.4 Inverse Newton Method 183

p=1 p=2 p=3

2 2 2

1 1 1

0 0 0

−1 −1 −1

−2 −2 −2

0 1 2 0 1 2 3 0 1 2 3 4

p=4 p=8 p = 16

2 2 2

1 1 1

0 0 0

−1 −1 −1

−2 −2 −2

0 1 2 3 4 5 0 1 2 3 4 5 6 7 8 9 0 4 8 12 16

Figure 7.2. Regions of a ∈ C for which the inverse Newton iteration (7.15) converges to
a−1/p . The dark shaded region is E(1, p) in (7.17). The union of that region with the lighter
shaded points is the experimentally determined region of convergence. The solid line marks
the disk of radius 1, centre 1. Note the differing x-axis limits.

The following result builds on the previous one to establish convergence when the
spectrum of A lies within a much larger wedge-shaped convex region in the complex
plane, the region depending on a parameter c. We denote by conv the convex hull.

Theorem 7.12 (Guo and Higham). Let A ∈ Cn×n have no eigenvalues on R− . For
all p ≥ 1, the iterates Xk from (7.12) with X0 = 1c I and c ∈ R+ converge quadratically
to A−1/p if all the eigenvalues of A are in the set

E(c, p) = conv { z : |z − cp | ≤ cp }, (p + 1)cp \ { 0, (p + 1)cp }. (7.17)

The actual convergence region, determined experimentally, is shown together with

E(c, p) in Figure 7.2 for c = 1 and several values of p.
If A happens to have just one eigenvalue and we know that eigenvalue then finite
convergence is obtained for an appropriate choice of c.

Lemma 7.13 (Guo and Higham). Suppose that A ∈ Cn×n has a positive eigenvalue
λ of multiplicity n and that the largest Jordan block is of size q. Then for the iteration
(7.12) with X0 = λ−1/p I we have Xm = A−1/p for the first m such that 2m ≥ q.

Proof. Let A have the Jordan form A = ZJZ −1 . Then R0 = I − X0p A =

m
Z(I − λ1 J)Z −1 . Thus R0q = 0. By Theorem 7.10, Rm = (R0 )2 h(R0 ), where h(R0 )
184 Matrix pth Root

is a polynomial in R0 . Thus Rm = 0 if 2m ≥ q. That convergence is to A1/p follows

from Theorem 7.12.

We can now build a practical algorithm, beginning by considering stability.

A stability analysis of iteration (7.12) leads to a stability condition similar to, but
even more restrictive than, (7.8). However, by introducing the matrix Mk = Xkp A
the iteration can be rewritten in a coupled form analogous to (7.10), in which we now
take X0 as in Theorem 7.12:

Coupled inverse Newton iteration (inverse matrix pth root):

 
(p + 1)I − Mk 1
Xk+1 = Xk , X0 = I,
p c
 p (7.18)
(p + 1)I − Mk 1
Mk+1 = Mk , M0 = p A.
p c

It is easy to show that this variant is stable (see Problem 7.14).

We begin by taking the square root twice by any iterative method. This prepro-
cessing step brings the spectrum into the sector arg z ∈ (−π/4, π/4). The nearest
point to the origin
√ that is both within this sector and on the boundary of E(c, p) is
at a distance cp √2. Hence the inverse Newton iteration can be applied to B = A1/4
with c ≥ (ρ(B)/ 2)1/p . We obtain an algorithm in the same spirit as Algorithm 7.9.

Algorithm 7.14 (Matrix inverse pth root via inverse Newton iteration). Given A ∈
Cn×n having no eigenvalues on R− this algorithm computes X = A−1/p using the
inverse Newton iteration.

1 B = A1/4 (computed as two successive square√roots)

2 Use the Newton iteration (7.18) with c = (θ/ 2)1/p , where θ is any
upperbound for ρ(B), to compute
 B −1/s , p = 4s ,
X = (B −1/p )4 , p = 4s + 1 or p = 4s + 3,
 −1/(2s+1) 2
(B ) , p = 4s + 2.

7.5. Schur–Newton Algorithm

The flop counts of Algorithms 7.9 and 7.14 are generally greater than that for the
Schur method unless p is large, although the fact that the Newton iterations are
rich in matrix multiplication and matrix inversion compensates somewhat on modern
computers. However it is possible to combine the Schur and Newton methods to
advantage. An initial Schur decomposition reveals the eigenvalues, which can be used
to make a good choice of the parameter c in the inverse Newton iteration, and the
subsequent computations are with triangular matrices. The algorithm can be arranged
so that it computes either A1/p or A−1/p , by using either (7.18) or the variant of it
7.5 Schur–Newton Algorithm 185

that is obtained by setting Yk = Xk−1 :

Coupled inverse Newton iteration (matrix pth root):

 −1
(p + 1)I − Mk
Yk+1 = Yk , Y0 = cI,
p
 p (7.19)
(p + 1)I − Mk 1
Mk+1 = Mk , M0 = p A.
p c

Here, Yk → A1/p under the same conditions (those in Theorem 7.12) that Xk →
A−1/p .
We state the algorithm for real matrices, but an analogous algorithm is obtained
for complex matrices by using the complex Schur decomposition.

Algorithm 7.15 (Schur–Newton algorithm for (inverse) matrix pth root).

Given A ∈ Rn×n with no eigenvalues on R− this algorithm computes X = A1/p or
X = A−1/p , where p = 2k0 q with k0 ≥ 0 and q odd, using a Schur decomposition and
a Newton iteration.
1 Compute a real Schur decomposition A = QRQT .
2 if q = 1
3 k1 = k0
4 else
k1
5 Choose k1 ≥ k0 such that |λ1 /λn |1/2 ≤ 2,
where the eigenvalues of A are ordered |λn | ≤ · · · ≤ |λ1 |.
6 end
7 If the λi are not all real and q 6= 1, increase k1 as necessary so that
1/2k1

arg λi ∈ (−π/8, π/8) for all i.
k1
8 Compute B = R1/2 by k1 invocations of Algorithm 6.7.
9 if q = 1, goto line 22, end
k1 k1
10 Let µ1 = |λ1 |1/2 , µn = |λn |1/2 .
11 if the λi are all real
12 if µ1 6= µn
 1/q
α1/q µ1 − µn µ1
13 c= 1/q
, where α = .
(α − 1)(q + 1) µn
14 else
1/q
15 c = µn
16 end
17 else
 1/q
µ1 + µn
18 c=
2
19 end 
X = B −1/q by (7.18), if A−1/p required,
20 Compute
X = B 1/q by (7.19), if A1/p required.
2k1 −k0
21 X←X (repeated squaring).
22 X ← QXQT
The key idea in the algorithm is to preprocess A so that the Newton iteration
converges in a small number of iterations. The algorithm begins by taking as many
186 Matrix pth Root

square roots as it can; these square roots are inexpensive since the matrix is triangular.
When the eigenvalues are all real the algorithm uses a choice of c with a certain
optimality property. For nonreal eigenvalues, the requirement on the arguments in
line 7 ensures that the corresponding choice of c in line 18 leads to fast convergence.
For more details see Guo and Higham [233, ], where numerical experiments show
Algorithm 7.15 to perform in a numerically stable manner.
The cost of the algorithm is about
   
2 1 k2 k2
28 + (k1 + k2 ) − + k0 + log2 p n3 flops,
3 3 2 2

where we assume that k2 iterations of (7.18) or (7.19) are needed. When k0 = 0,

k1 = 3, and k2 = 4, for example, the flop count becomes (32 23 + 2 log2 p)n3 , while the
count is always (28 + p−1 3
3 )n for the Schur method. Algorithm 7.15 is slightly more
expensive than the Schur method if p is small or highly composite (assuming that the
Schur method is applied over the prime factors of p), but it is much less expensive
than the Schur method if p is large and has a small number of prime factors.

7.6. Matrix Sign Method

Theorem 7.4 shows how to recover a pth root of A from an n-dimensional invariant
subspace of the block companion matrix
 
0 I
 0 I 
 .. .. 
 
C= . .  ∈ Cpn×pn . (7.20)
 . 
 .. I 
A 0

The relation (5.4) shows that for p = 2 the sign of C directly reveals A1/p , so we
might hope that A1/p is recoverable from sign(C) for all p. This is true when p is
even and not a multiple of 4.

Theorem 7.16 (Bini, Higham, and Meini). If p = 2q where q is odd, then the first
block column of the matrix sign(C) is given by
 
γ0 I
 γ1 X 
1 γ X 2


V =  2 ,
p .. 
 . 
p−1
γp−1 X
Pp−1 kj
where X = A1/p , γk = j=0 ωp θj , ωp = e
2πi/p
, and θj = −1 for j = ⌊q/2⌋ +
1: ⌊q/2⌋ + q, θj = 1 otherwise. If A is Hermitian positive definite then this result
holds also for odd p.

This result allows us to compute the principal pth root of A from the (2, 1) block
of sign(C). For p = 2 we have θ0 = 1, θ1 = −1, γ0 = 0, γ1 = 2, and Theorem 7.16
reproduces the first block column of (5.4).
7.7 Notes and References 187

Algorithm 7.17 (Matrix pth root via matrix sign function). Given A ∈ Cn×n hav-
ing no eigenvalues on R− this algorithm computes X = A1/p via the matrix sign
function.

1 If p is odd
2 p ← 2p, A ← A2
3 else
4 if p is a multiple of 4
5 while p/2 is even, A ← A1/2 , p = p/2, end
6 end
7 end
8 S = sign(C), where C is given in (7.20).
p P⌊q/2⌋
9 X = 2σ S(n + 1: 2n, 1: n) where σ = 1 + 2 j=1 cos(2πj/p) and q = p/2.

Assuming the matrix sign function is computed using one of the Newton or Padé
methods of Chapter 5, the cost of Algorithm 7.17 is O(n3 p3 ) flops, since C is np × np.
The algorithm therefore appears to be rather expensive in computation and storage.
However, both computation and storage can be saved by exploiting the fact that the
sign iterates belong to the class of A-circulant matrices, which have the form
 
W0 W1 ... Wp−1
 .. .. 
 AWp−1 W0 . . 
 .
 .. . . 
 . . . . . W1 
AW1 . . . AWp−1 W0

In particular, an A-circulant matrix can be inverted in O(n3 p log p + n2 p log2 p) flops.

See Bini, Higham, and Meini [69, , Sec. 7] for details.

7.7. Notes and References

In several papers Sylvester stated that the number of pth roots of an n × n matrix
is pn , sometimes mentioning exceptional cases (see [555, ], [557, ], and the
quote at the end of the chapter). His count of pn is correct if the matrix has distinct
eigenvalues (see Theorem 7.1).
The representation (7.1) can be deduced from a standard identity in complex
analysis, as noted by Bhatia [64, , Ex. V.1.10] and Mehta [420, , Sec. 5.5.5].
It can also be found in a slightly different form and for Aα , for 0 < Re α < 1, in
the functional analysis literature: see Balakrishnan [41, ] and Haase [236, ,
Prop. 3.1.12].
Theorem 7.4 is from Benner, Byers, Mehrmann, and Xu [54, ].
Björck and Hammarling [73, , Sec. 5] show how to compute a cube root using
the (complex) Schur decomposition. Section 7.2 is based on M. I. Smith [530, ],
who derives Algorithm 7.6 and proves (7.3).
The Newton iteration (7.5) is used to compute pth roots of scalars in the GNU
multiple precision arithmetic library [218].
The difficulty of analyzing the convergence of the Newton iteration (7.5) in the
scalar case for p ≥ 3 was noted by Cayley [101, ]; see the quote at the end of this
chapter.
188 Matrix pth Root

Prior to Iannazzo’s analysis the only available convergence result for the Newton
iteration was for symmetric positive definite matrices [299, ].
The condition for stability of the Newton iteration (7.5) that the eigenvalues in
(7.8) are of modulus less than 1 is obtained by M. I. Smith [530, ].
Algorithm 7.9 is due to Iannazzo [306, ], who also obtains, and proves stable,
iterations (7.9), (7.10), and (7.18).
Section 7.4 is based on Bini, Higham, and Meini [69, ] and Guo and Higham
[233, ], from where the results and algorithms therein are taken.
For p = 2, the inverse Newton iteration (7.12) is well known in the scalar case as
an iteration for the inverse square root. It is employed in floating point hardware to
compute the square root a1/2 via a−1/2 × a, since the whole computation can be done
using only multiplications [117, ], [334, ]. The inverse Newton iteration is
also used to compute a1/p in arbitrarily high precision in the MPFUN and ARPREC
packages [36, ], [37, ].
The inverse Newton iteration with p = 2 is studied for symmetric positive definite
matrices by Philippe, who proves that the residuals Rk = I − Xk2 A satisfy Rk+1 =
3 2 1 3
4 Rk + 4 Rk [471, , Prop. 2.5]. For arbitrary p, R. A. Smith [531, , Thm. 5]
proves quadratic convergence of (7.12) to some pth root when ρ(R0 ) < 1, as does Lakić
[368, ] under the assumption that A is diagonalizable, but neither determines to
which root convergence is obtained.
The coupled iteration (7.18) is a special case of a family of iterations of Lakić [368,
], who proves stability of the whole family.
Section 7.5 is based on Guo and Higham [233, ]. Section 7.6 is based on Bini,
Higham, and Meini [69, ].
Some other methods for computing the pth root are described by Bini, Higham,
and Meini [69, ], all subject to the same restriction on p as in Theorem 7.16. One
comprises numerical evaluation at the Fourier points of a transformed version of the
integral (7.1), in which the integration is around the unit circle. Another computes a
Wiener–Hopf factorization of the matrix Laurent polynomial F (z) = z −p/2 ((1+z)p A−
(1 − z)p I), from which A1/p is readily obtained. Another method expresses A1/p in
terms of the central coefficients H0 , . . . , Hp/2−1 of the matrix Laurent series H(z) =
P+∞
H0 + i=1 (z i + z −i )Hi , where H(z)F (z) = I. Further work is needed to understand
the behaviour of these methods in finite precision arithmetic; see Problem 7.18.
The principal pth root of a nonsingular M-matrix (see Section 6.8.3) is an M-
matrix, but for p > 3 it need not be the only pth root that is an M-matrix; see
Fiedler and Schneider [185, ].
Sylvester [554, ], [555, ] was interested in finding pth roots of the 2 × 2
identity because of a connection with a problem solved earlier by Babbage in his
“calculus of functions”. The problem is to find a Möbius transformation φ(x) =
(ax + b)/(cx + d) such that φ(i) (x) = x for a given i. This is easily seen to be
equivalent to finding a matrix A = ac db for which Ai = I2 (see, e.g., Mumford,
Series, and Wright [442, ]).
A few results are available on integer pth roots of the identity matrix. Turnbull
[577, ], [578,

, p. 332] observes that the antitriangular matrix An with aij =
(−1)n−j n−j
i−1 for i + j ≤ n + 1 is a cube root of In . For example, A34 = I4 , where
 
−1 1 −1 1
 −3 2 −1 0
A4 =  .
−3 1 0 0
−1 0 0 0
Problems 189

This matrix is a rotated variant of the Cholesky factor of the Pascal matrix (cf. the
involutory matrix in (6.59)). Up to signs, An is the MATLAB matrix pascal(n,2).
Vaidyanathaswamy [582, ], [583, ] derives conditions on p for the existence
of integer pth roots of In ; in particular, he shows that for a given n there are only
finitely many possible p. Problem 7.6 gives an integer (n + 1)st root of In .
Finally, we note that Janovská and Opfer [317, , Sec. 8] show how computing
a pth root of a quaternion can be reduced to computing a pth root of a matrix,
though the special structure of the matrix needs to be exploited to be competitive
with methods working in the algebra of quaternions.

Problems
7.1. Does an m × m Jordan block with eigenvalue zero have a pth root for any p?
7.2. How should Aα be defined for real α ∈ [0, ∞)?
7.3. Show that if X p = In then the eigenvalues of X are pth roots of unity and X is
diagonalizable.
7.4. Show that the Frobenius norm (relative) condition number for the matrix pth
root X of A ∈ Cn×n is given by
P −1
p p−j T
j=1 (X ) ⊗ X j−1 kAkF
2
cond(X) = (7.21)
kXkF
and that this condition number is finite if and only if X is a primary pth root of A.
7.5. Suppose that in the Newton iteration for a matrix pth root defined by (7.4) and
Xk+1 = Xk + Ek , X0 commutes with A and all the iterates are well-defined. Show
that, for all k, Xk commutes with A and Xk+1 = p1 ((p − 1)Xk + Xk1−p A).
7.6. (Bambaii and Chowla [42, ]) Let
 
−1 −1 ... ... −1
 1 0 ... ... 0 

 .. .. 

An =  0 1 . .  ∈ Rn×n .
 .
 .. .. .. .. 
. . . 
0 ... ... 1 0

Prove, or verify computationally, that An+1

n = In .
7.7. (Rice [488, ]) This problem develops an elementary proof of the existence
of the Hermitian positive definite pth root of a Hermitian positive definite matrix
A ∈ Cn×n . Let ≥ denote the Hermitian positive semidefinite ordering defined in
Section B.12. First, show that if 0 ≤ Z ≤ Y ≤ I and Y Z = ZY then (Y p − Z p )/p ≤
Y − Z. Then show that we can assume A ≤ I without loss of generality. Next,
consider the iteration
1
Xk+1 = Xk + (A − Xkp ), X0 = 0.
p
Show that 0 ≤ Xk ≤ Xk+1 ≤ I for all k, and deduce that Xk converges to a pth root
of A.
190 Matrix pth Root

7.8. Show that for the Newton–Schulz iteration for the inverse of A ∈ Cn×n ,

Xk+1 = 2Xk − Xk AXk , (7.22)

the residuals Rk = I − Xk A and Rk = I − AXk both satisfy Rk+1 = Rk2 for any X0 .
(Note that (7.22) is a more symmetric variant of (7.12) with p = 1. For (7.12) this
residual recurrence is assured only for X0 that commute with A.)
7.9. Investigate the behaviour of the inverse Newton iteration (7.12) for p = 2 with
   
1 0 1 θ
A= , X 0 = , θ 6= 0.
0 µ−2 0 µ

(Cf. Problem 6.8.)

7.10. For the inverse Newton iteration (7.12) how does the asymptotic error constant
for the residual vary with p? (Hint: evaluate a2 in Theorem 7.10.)
7.11. Given an approximation X ≈ A−1/p , how can an approximation Y ≈ A1/p be
computed using only matrix multiplication?
1/p
7.12. (Smith [530,
 ]) Show that
 the eigenvalues of the Fréchet derivative Lg (A )
−1 1−p
of g(X) = p (p − 1)X + X A (the Newton iteration function in (7.5)) are given
by (7.8). Hint: use the expansion
p−1
X
(A + E)1−p = A1−p − Ar−p EA−r + O(kEk2 ).
r=1

7.13. Assume that the Newton iteration (7.5) converges to a pth root B of A. By
making use of the factorization
p−2
X
(p − 1)xp − p xp−1 b + bp = (x − b)2 (i + 1)xi bp−2−i ,
i=0

obtain a constant c so that kXk+1 − Bk ≤ ckXk − Bk2 for all sufficiently large k.
7.14. Prove the stability of the iteration (7.18).
7.15. (Guo and Higham [233, ]) Let A ∈ Rn×n be a stochastic matrix. Show
that if A is strictly diagonally dominant (by rows—see (B.2)) then the inverse Newton
iteration (7.12) with X0 = I converges to A−1/p . The need for computing roots of
stochastic matrices arises in Markov model applications; see Section 2.3. Transition
matrices arising in the credit risk literature are typically strictly diagonally dominant
[315, ], and such matrices are known to have at most one generator [127, ].
7.16. (Guo and Higham [233, ]) Let X e = X + E, with kEk ≤ ǫkXk, be an
approximate pth root of A ∈ C n×n e p k that is sharp to
. Obtain a bound for kA − X
first order in kEk and hence explain why

kA − Xe pk
e =
ρA (X) Pp−1

e
kXk e p−1−i T ⊗ X
e i
i=0 X

is a more appropriate definition of relative residual (e.g., for testing purposes) than
kA − Xe p k/kX
e p k.
Problems 191

7.17. (Research problem) Determine conditions under which a stochastic matrix

has a pth root that is also stochastic. How can it be computed? When is such a
pth root unique? For some results on square roots of stochastic matrices and doubly
stochastic matrices (those for which the matrix and its transpose are both stochastic)
see Achilles and Sinkhorn [2, ], Iwanik and Shiflett [316, ], Marcus and Minc
[406, ], and Minc [429, , Sec. 5.4].
7.18. (Research problem) Investigate, and where necessary improve, the behaviour
in finite precision arithmetic of Algorithm 7.17 and the other methods in Bini, Higham,
and Meini [69, ].
7.19. (Research problem) Extend the analysis in Section 6.1 from square roots to
pth roots.

It is nevertheless possible to form the powers

(positive or negative, integral or fractional) of a matrix,
and thence to arrive at the notion of a rational and integral function,
or generally of any algebraical function, of a matrix.
— ARTHUR CAYLEY, A Memoir on the Theory of Matrices (1858)

The solution is easy and elegant in the case of a quadric equation,

but the next succeeding case of the cubic equation
appears to present considerable difficulty.
— ARTHUR CAYLEY, The Newton–Fourier Imaginary Problem (1879)

In general, for a matrix of the order ω , the number of mth roots is mω

and each of them is perfectly determinate.
But when the matrix is a unit-matrix or a zero-matrix . . .
there are distinct genera and species of such roots,
and every species contains its own appropriate number of arbitrary constants.
— J. J. SYLVESTER, Note on the Theory of Simultaneous
Linear Differential or Difference Equations with Constant Coefficients (1881)
Chapter 8
The Polar Decomposition

The polar decomposition is the generalization to matrices of the familiar polar repre-
sentation z = reiθ of a complex number. It is intimately related to the singular value
decomposition (SVD), as our proof of the decomposition reveals.

Theorem 8.1 (Polar decomposition). Let A ∈ Cm×n with m ≥ n. There exists

a matrix U ∈ Cm×n with orthonormal columns and a unique Hermitian positive
semidefinite matrix H ∈ Cn×n such that A = U H. The matrix H is given by
(A∗A)1/2 . All possible U are given by
 
Ir 0
U =P Q∗ , (8.1)
0 W
  ∗
where A = P Σ0r 0m−r,n−r
0
Q is an SVD, r = rank(A), and W ∈ C(m−r)×(n−r) is
arbitrary subject to having orthonormal columns. If rank(A) = n then H is positive
definite and U is uniquely determined.
Proof. Note first that one particular polar decomposition can be written down
immediately in terms of the SVD:
   
Ir 0 ∗ Σr 0
A=P Q ·Q Q∗ ≡ U H. (8.2)
0 Im−r,n−r 0 0n−r

We have A∗A = HU ∗ U H = H 2 and since H is Hermitian positive semidefinite it

follows that H = (A∗A)1/2 , which uniquely defines H. If r = n then H is clearly
nonsingular and hence U = AH −1 is unique.
To determine all possible U , write A = U H as
   
Σr 0 ∗ Σr 0
P Q = UQ Q∗ .
0 0 0 0
 
0 −1
Premultiplying by P ∗ and postmultiplying by Q Σ0r In−r gives
   
Ir 0 I 0
= P UQ r
∗
.
0 0 0 0
 V
It follows that P ∗ U Q has the form I0r W for some V and W . The requirement that
U has orthonormal columns forces V to be zero and W to have orthonormal columns.

With a slight abuse of nomenclature (since a unitary matrix must be square), we

will refer to U as the unitary polar factor.

193
194 The Polar Decomposition

When m ≤ n we can apply Theorem 8.1 to A∗ to obtain another version of the

polar decomposition in which the Hermitian factor appears first: A = H eUe , with
e
H ∈C m×m e
,U ∈C m×n
. In fact, by constructing this factorization in an analogous
e = (AA∗ )1/2 and that we can take U
way to (8.2) it is clear that H e = U . (Equivalently,
∗
the unitary polar factor of A can be taken as the conjugate transpose of that of A.)
When A is square and nonsingular U and U e are unique and necessarily equal; this
equality can be expressed as A(A∗A)−1/2 = (AA∗ )−1/2 A, which is a particular case of
Corollary 1.34. In fact, these two polar decomposition variants are identical (modulo
any nonuniqueness of U ) precisely when A is normal (A∗A = AA∗ ), as the next result
shows.

Theorem 8.2. Let A ∈ Cn×n . In the polar decomposition A = U H the factors U

and H commute if and only if A is normal.
Proof. If U H = HU then, since U is unitary, A∗A = HU ∗ U H = H 2 =
HU U ∗ H = U HHU ∗ = AA∗ , so A is normal.
If A∗A = AA∗ then H 2 = U H 2 U ∗ . Taking the principal square root of both sides
gives H = U HU ∗ , or HU = U H, as required.
Although it is rarely discussed in the literature, the polar decomposition A = U H
exists also for m < n, with U now having orthonormal rows. However, neither U nor
H is unique when m < n; see Problem 8.7.
The nonuniqueness of the polar decomposition for rank-deficient matrices with
m ≥ n, and for all matrices with m < n, can be eradicated by relaxing the orthonor-
mality requirement on U while at the same time insisting on equality of the ranges
of U ∗ and H. The next result provides an elegant way to obtain a unique “canon-
ical polar decomposition” for all m and n. Although this decomposition is largely
of theoretical interest, we will see in the next section that its U factor possesses a
best approximation property, as does the unitary polar factor. A matrix U ∈ Cm×n
is a partial isometry if kU xk2 = kxk2 for all x ∈ range(U ∗ ). A more concrete char-
acterization of a partial isometry is as a matrix whose singular values are all 0 or
1. For more details on partial isometries see Section B.6.3. We denote by H + the
pseudoinverse of H. As the proof of the following theorem shows, the pseudoinverse
is a very useful tool for analyzing the canonical polar decomposition.

Theorem 8.3 (canonical polar decomposition). Let A ∈ Cm×n . There is a unique

decomposition A = U H with U ∈ Cm×n a partial isometry, H ∈ Cn×n Hermitian
positive semidefinite, and range(U ∗ ) = range(H). The last condition is equivalent 8
to U ∗ U = HH + . Moreover, A+ = H + U + . The factors  U and H  ∗are given by
H = (A∗A)1/2 and U = AH + . Furthermore, if A = P Σ0r 0m−r,n−r 0
Q is an SVD
then    
Ir 0 ∗ Σr 0
U =P Q , H=Q Q∗ . (8.3)
0 0m−r,n−r 0 0n−r
Proof. Note that U ∗ = U + for a partial isometry (see Lemma B.2). Hence
the condition U ∗ U = HH + is equivalent to U + U = HH + , which is equivalent to
range(U ∗ ) = range(H), since BB + and B + B are the orthogonal projectors onto
range(B) and range(B ∗ ), respectively (see Problem B.10).
The formula (8.3) shows that a decomposition of the claimed form exists. However,
for completeness, we prove existence without using the SVD. Note first that by
8 In fact, U ∗ U = HH + implies that U is a partial isometry.
The Polar Decomposition 195

Problem B.2, H + is Hermitian and commutes with H. Hence, by the first and second
Moore–Penrose conditions (B.3) (i) and (B.3) (ii), any product comprising p factors
H and q factors H + , in any order, equals H if p = q + 1, H + if q = p + 1, and HH +
if p = q. Define H = (A∗ A)1/2 and U = AH + . We first need to show that U is a
partial isometry (U ∗ = U + ) and U ∗ U = HH + . To prove the latter equality, we have

U ∗ U = H + A∗ AH + = H + H 2 H + = HH + .

It remains to show that U ∗ = U + , which involves checking the Moore–Penrose con-

ditions. We have

U U ∗ U = AH + · H + A∗ · AH + = A(H + )2 H 2 H + = AH + = U,
U ∗ U U ∗ = H + A∗ · AH + · H + A∗ = H + H 2 (H + )2 A∗ = H + A∗ = U ∗ ,

and U U ∗ and U ∗ U are trivially Hermitian, as required.

Now we need to verify that A = U H. By Problem 6.4 we have range(H) =
range((A∗A)1/2 ) = range(A∗A) = range(A∗ ), using (B.4) for the last equality. Hence
HH + = A+ A, by Problem B.10. Therefore U H = AH + · H = AHH + = AA+ A = A.
We now show that the decomposition is unique. If A = U H and U ∗ U = HH +
then A∗A = HU ∗ U H = HHH + H = H 2 , so H = (A∗A)1/2 is uniquely determined.
Then AH + = U HH + = U U ∗ U = U U + U = U , which determines U uniquely.
Finally, that A+ = H + U + =: X is shown by verifying the Moore–Penrose con-
ditions, of which we just illustrate the first: AXA = U H · H + U + · U H = U HH + ·
HH + · H = U H = A.
We make several comments on this theorem.

• Other ways to express the condition range(U ∗ ) = range(H) are as null(U ) =

null(H) and as range(U ) = range(A) (or equivalently U U + = AA+ ); see Prob-
lem 8.8.

• Let m ≥ n. Then the factor U in Theorem 8.3 is the matrix U in (8.1) with
W = 0; U in (8.3) has orthonormal columns precisely when A has full rank; and
if A has full rank the polar decomposition and canonical polar decomposition
are the same.

• The decompositions with m = 1 and n = 1 are

n = 1 : a = kak−1
2 a · kak2 ,
m = 1 : a∗ = kak−1 ∗ −1 ∗
2 a · kak2 aa . (8.4)

• If A = U H and A∗ = V G are the decompositions in Theorem 8.3 then V = U ∗ ,

that is, A and A∗ have U factors that are conjugate transposes of each other.
This can be seen from
 +  +  +
∗
V = A∗ (AA∗ )1/2 = (A∗A)1/2 A∗ = A (A∗A)1/2 = U ∗, (8.5)

where the second equality follows from Corollary 1.34 with f the function

D
f (x) = x1/2 . Here, D denotes the Drazin inverse (see Problem 1.52), which
is identical to the pseudoinverse for Hermitian matrices, to which it is being
applied here.
196 The Polar Decomposition

In most of the rest of this chapter we will restrict to the standard polar decompo-
sition with m ≥ n.
Neither polar factor is a function of A (though this can be arranged by modifying
the definition of matrix function, as explained in Problem 1.53; see also Problem 8.9).
The reason for including the polar decomposition in this book is that it has many
intimate relations with the matrix sign function and the matrix square root, and so
it is naturally treated alongside them.
A fundamental connection between the matrix sign function and the polar decom-
position is obtained from Theorem 5.2 with B = A∗ when A ∈ Cn×n is nonsingular:
   
0 A 0 U
sign = . (8.6)
A∗ 0 U∗ 0

This relation can be used to obtain an integral formula for U from the integral formula
(5.3) for the sign function:
Z ∞
2
U= A (t2 I + A∗A)−1 dt. (8.7)
π 0

This expression in fact holds for any A ∈ Cm×n with rank(A) = n.

The Hermitian polar factor H = (A∗A)1/2 is also known as the matrix absolute
value and written |A| (which is not to be confused with the determinant of A, always
written det(A) in this book). The matrix absolute value has been much studied in
the linear algebra and functional analysis literatures.
The outline of this chapter is as follows. We start with some important approx-
imation properties of the polar factors U and H. Some useful perturbation results
are presented and condition numbers identified. Iterative methods of computation
are treated, beginning with Newton’s method and the Newton–Schulz method. We
explain how any iteration for the matrix sign function leads to a corresponding it-
eration for the unitary polar factor and use this connection to derive a Padé family
of iterations. Scaling to accelerate convergence, stopping tests for the iterations, and
numerical stability are examined. The chapter finishes with an algorithm based on
the Newton iteration along with some numerical illustrations.
Two general points need to be made concerning computation. First, if compu-
tational cost is not a concern then there is no better way to compute the polar
decomposition than via the SVD, as in the proof of Theorem 8.1. However, this is an
expensive approach, and indeed the converse strategy of computing the SVD via the
polar decomposition has been suggested for parallel machines [284, ]. Second, we
can restrict our attention to square, nonsingular matrices if we are willing to carry
out a preliminary orthogonal reduction of a rectangular matrix. Let A ∈ Cm×n with
m ≥ n have the QR factorization A = QR, Q ∈ Cm×n , R ∈ Cn×n . If R = U H is a
polar decomposition then QU · H is a polar decomposition of A. If R is singular then
the reduction can be continued to produce a
 
R 0
A=P Q∗ , (8.8)
0 0

where P ∈ Cm×m and Q ∈ Cn×n are unitary, and R ∈ Cr×r is nonsingular and upper
triangular [224, , Sec. 5.4.2]. The polar decomposition of R can now be computed
and the polar decomposition of A pieced back together.
8.1 Approximation Properties 197

8.1. Approximation Properties

Given that the polar decomposition is just a “compressed SVD”, as the proof of
Theorem 8.1 shows, it is natural to ask why the polar decomposition merits separate
study. One reason is that both polar factors have best approximation properties.

Theorem 8.4 (Fan and Hoffman). Let A ∈ Cm×n (m ≥ n) have the polar decom-
position A = U H. Then kA − U k = min{ kA − Qk : Q∗ Q = In } for any unitarily
invariant norm. The minimizer is unique for the Frobenius norm if A has full rank.

Proof. If Q∗ Q = In then Q has singular values all equal to 1. Hence, by Theo-

rem B.3, if A has the SVD A = P ΣV ∗ then kA − Qk ≥ kΣ − Im,n k for any unitarily
invariant norm. Equality is achieved for Q = U because, since H = V Σ1 V ∗ , where
Σ = [Σ1 0]T ,

kA−U k = kU (H−In )k = kH−In k = kV Σ1 V ∗ −In k = kΣ1 −In k = kΣ−Im,n k, (8.9)

where we have used Problems B.7 and B.6.

To prove the uniqueness result for the Frobenius norm we prove the result again
from scratch, thereby illustrating a proof technique applicable to related problems.
Let Q ∈ Cm×n be any matrix satisfying Q∗ Q = In and set E = Q − U . Then
In = Q∗ Q = E ∗ U + U ∗ E + E ∗ E + In , so that

E ∗ U + U ∗ E + E ∗ E = 0. (8.10)

Hence

(A − Q)∗ (A − Q) = (A − U )∗ (A − U ) − (A − U )∗ E − E ∗ (A − U ) + E ∗ E
= (A − U )∗ (A − U ) − A∗ E − E ∗ A. (8.11)

Taking the trace in (8.11) gives kA − Qk2F = kA − U k2F − trace(A∗ E + E ∗ A), and

trace(A∗ E + E ∗ A) = trace(HU ∗ E + E ∗ U H) = trace(H(U ∗ E + E ∗ U ))

= − trace(HE ∗ E) = − trace(EHE ∗ ) ≤ 0,

where we have used (8.10). Hence kA − Qk2F ≥ kA − U k2F , with equality only if
trace(EHE ∗ ) = 0, and if A has full rank then H is nonsingular and this last condition
implies E = 0, giving the uniqueness condition.
For m < n we can apply Theorem 8.4 to A∗ to deduce that if A∗ = U bH
b is a
b ∗ ∗
polar decomposition then kA − U k = min{ kA − Qk : Q Q = Im } for any unitarily
invariant norm. The next result, concerning best approximation by partial isometries,
holds for arbitrary m and n and reduces to Theorem 8.4 when rank(A) = n.

Theorem 8.5 (Laszkiewicz and Ziȩtak). Let A ∈ Cm×n have the canonical polar de-
composition A = U H. Then U solves

min { kA − Qk : Q ∈ Cm×n is a partial isometry with range(Q) = range(A) }

for any unitarily invariant norm.

198 The Polar Decomposition

Proof. Let Q ∈ Cm×n be any partial isometry

 ∗ satisfying range(Q) = range(A),
and let A have the SVD A = P Σ0r 0m−r,n−r
0
V , where r = rank(A). Since rank(Q) =
rank(A), Theorem B.3 implies that
 
Σr − Ir 0
kA − Qk ≥

0 0m−r,n−r

for any unitarily invariant norm. Equality is achieved for Q = U (which satisfies the
range condition by Problem 8.8) because
   
Σr 0 Ir 0
kA − U k = P ∗
V −P V ∗
0 0m−r,n−r 0 0m−r,n−r
 
Σr − Ir 0
=
.
0 0m−r,n−r

Theorem 8.4 says that the polar factor U is the nearest matrix to A with orthonor-
mal columns. Hence the polar decomposition provides an optimal way of orthogonal-
izing a matrix. Applications are discussed in Section 2.6. For square matrices U is
the nearest unitary matrix to A. A related approximation problem is that of finding
the nearest unitary matrix with determinant 1—the nearest “rotation matrix”. This
problem is a special case of one of the Procrustes problems discussed in Section 2.6,
for which we now derive solutions.

Theorem 8.6 (orthogonal Procrustes problems). Let A, B ∈ Cm×n .

(a) Any solution to the orthogonal Procrustes problem

min kA − BW kF : W ∈ Cn×n , W ∗ W = I (8.12)

is a unitary polar factor of B ∗A. There is a unique solution if B ∗A is nonsingular.

(b) For real A and B the rotation variant

min kA − BW kF : W ∈ Rn×n , W ∗ W = I, det(W ) = 1 (8.13)

has solution W = P V Q∗ , where V = diag(1, . . . , 1, det(P Q∗ )) and B ∗A = P ΣQ∗ is

an SVD. The solution is unique if (i) det(P Q∗ ) = 1 and σn−1 6= 0 or (ii) det(P Q∗ ) =
−1 and σn−1 > σn , where Σ = diag(σi ).
Proof. Since

kA − BW k2F = kAk2F − trace(A∗ BW + W ∗ B ∗A) + kBk2F , (8.14)

the problem reduces to maximizing trace(A∗ BW + W ∗ B ∗A) = 2 Re trace(W ∗ B ∗A).

The result follows from Problem 8.13.
Before explaining the approximation properties of the Hermitian polar factor we
note the solution to the problem of finding the nearest Hermitian matrix. For A ∈
Cn×n , we introduce the notation
1 1
AH = (A + A∗ ), AK = (A − A∗ ), A ≡ AH + AK .
2 2
AH is called the Hermitian part of A and AK the skew-Hermitian part.
8.2 Sensitivity and Conditioning 199

Theorem 8.7 (Fan and Hoffman). For A ∈ Cn×n and any unitarily invariant norm,
kA−AH k = min{ kA−Xk : X = X ∗ }. The solution is unique for the Frobenius norm.

Proof. For any Hermitian Y ,

1
kA − AH k = kAK k = k(A − Y ) + (Y ∗ − A∗ )k
2
1 1
≤ kA − Y k + k(Y − A)∗ k
2 2
= kA − Y k,

using the fact that kAk = kA∗ k for any unitarily invariant norm. The uniqueness for
the Frobenius norm is a consequence of the strict convexity of the norm.

While AH is the nearest Hermitian matrix to A, the nearest positive semidefinite

matrix involves the Hermitian polar factor of AH .

Theorem 8.8. Let A ∈ Cn×n and let AH have the polar decomposition AH = U H.
Then X = (AH + H)/2 is the unique solution to

min{ kA − XkF : X is Hermitian positive semidefinite }.

Proof. Let X be any Hermitian positive semidefinite matrix. From the fact that
kS + Kk2F = kSk2F + kKk2F if S = S ∗ and K = −K ∗ , we have

kA − Xk2F = kAH − Xk2F + kAK k2F ,

so the problem reduces to that of approximating AH . Let AH = QΛQ∗ be a spectral

decomposition (Q∗ Q = I, Λ = diag(λi )) and let Y = Q∗ XQ. Then
X X X X
kAH − Xk2F = kΛ − Y k2F = 2
yij + |λi − yii |2 ≥ |λi − yii |2 ≥ λ2i ,
i6=j i λi <0 λi <0

since yii ≥ 0 because Y is positive semidefinite. This lower bound is uniquely at-
tained when Y = diag(di ) ≡ diag(max(λi , 0)), for which X = Q diag(di )Q∗ . The
representation X = (AH + H)/2 follows, since H = Q diag(|λi |)Q∗ .

If A is Hermitian then Theorem 8.8 says that the nearest Hermitian positive
semidefinite matrix is obtained by replacing every eigenvalue λi of A by max(λi , 0)
and leaving the eigenvectors unchanged. If, instead, λi is replaced by |λi | then H
itself is obtained, and kA − HkF is at most twice the Frobenius norm distance to
the nearest Hermitian positive semidefinite matrix. These ideas have been used in
Newton’s method in optimization to modify indefinite Hessian matrices to make them
definite [215, , Sec. 4.4.2], [449, , Sec. 6.3],

8.2. Sensitivity and Conditioning

We now consider the sensitivity of the polar factors to perturbations in the matrix.
Here, and in the rest of this chapter, we denote the ith largest singular value of A by
σi = σi (A).
200 The Polar Decomposition

Theorem 8.9. Let A ∈ Cm×n (m ≥ n) have the polar decomposition A = U H and

let A + ∆A have the polar decomposition A + ∆A = (U + ∆U )(H + ∆H). Then
√
k∆HkF ≤ 2 k∆AkF .

If A has full rank and k∆Ak2 < σn then U and U + ∆U are uniquely defined and

k∆U kF ≤ θ k∆AkF + O(k∆Ak2F ),

where the value of θ is given in the table

A, ∆A ∈ Rm×n A, ∆A ∈ Cm×n
m=n 2/(σn + σn−1 ) σn−1
m>n σn−1 σn−1

and in all cases the bound is attainable to first order in k∆AkF .

Proof. For the bound on k∆HkF , see Problem 8.15. We outline the proofs for
m = n, which are from Kenney and Laub [342, , Thms. 2.2, 2.3]. To first order
we have

∆A = ∆U H + U ∆H, (8.15)
∗ ∗
0 = ∆U U + U ∆U. (8.16)

Taking the conjugate transpose of (8.15) and premultiplying and postmultiplying by

U gives
U ∆A∗ U = U H∆U ∗ U + U ∆H = −U HU ∗ ∆U + U ∆H,
where we have used (8.16). Subtracting this equation from (8.15) gives

U HU ∗ ∆U + ∆U H = ∆A − U ∆A∗ U.

Inserting the expressions for U and H in terms of the SVD A = P ΣQ∗ we have

P ΣP ∗ ∆U + ∆U QΣQ∗ = ∆A − P Q∗ ∆A∗ P Q∗ ,

or
ΣW + W Σ = E − E ∗ , W = P ∗ ∆U Q, E = P ∗ ∆AQ.
Hence
eij − eji
wij = ,
σi + σj
so that
|eij |2 + |eji |2
|wij |2 ≤ .
2σn2
Thus k∆U kF = kW kF ≤ kEkF /σn = k∆AkF /σn . This bound is attained for ∆A =
P (ien eTn )Q∗ .
For real A and ∆A the same argument gives ΣW + W Σ = E − E T , with real W
and E given by W = P T ∆U Q and E = P T ∆AQ. We have
eij − eji
wij = ,
σi + σj
8.2 Sensitivity and Conditioning 201

and wii ≡ 0. For i 6= j,

2(|eij |2 + |eji |2 )
|wij |2 ≤ .
(σn + σn−1 )2

Hence k∆U kF = kW kF ≤ 2kEkF /(σn + σn−1 ) = 2k∆AkF /(σn + σn−1 ). The bound
is attained when ∆A = P (en−1 eTn − en eTn−1 )QT .
For the bounds for m > n, see Chaitin-Chatelin and Gratton [103, ].

We can define condition numbers of the polar factors using (3.2) and (3.4), where
f maps A to U or √ H and we take the Frobenius norm. We can conclude from the
theorem that κ√ H ≤ 2, where κH is the absolute condition
√ number for H. The actual
value of κH is 2(1 + κ2 (A)2 )1/2 /(1 + κ2 (A)) ∈ [1, 2], as shown by Chaitin-Chatelin
and Gratton [103, ]. For U , κU = θkAkF /kU kF is a relative condition number.
Interestingly, for square A, κU depends very much on whether the data are real or
complex, and in the complex case it is essentially the condition number with respect
to inversion, κ(A). For rectangular A, this dichotomy between real and complex data
does not hold; see Problem 8.17. To summarize, H is always perfectly conditioned,
but the condition of U depends on the smallest one or two singular values of A.
A number of variations on the bounds for ∆U are available. Particularly elegant
are the following nonasymptotic bounds. Problem 8.16 asks for a proof of the first
bound for n = 1.

Theorem 8.10. Let A, A e ∈ Cn×n be nonsingular, with smallest singular values σn

and σ e of A
en , respectively. The unitary polar factors U of A and U e satisfy

ek ≤ 2 e
kU − U kA − Ak
σn + σ
en

for any unitarily invariant norm.

Proof. See R.-C. Li [386, , Thm. 1].

Note that the bound of the theorem is attained whenever A and Ae are unitary.
e
The bound remains valid in the Frobenius norm for A, A ∈ C m×n
of rank n [389,
, Thm. 2.4].

Theorem 8.11. Let A, A e ∈ Rn×n be nonsingular, with ith largest singular values σi
and σ e F < σn + σ
ei , respectively. If kA − Ak en then the unitary polar factors U of A
e e
and U of A satisfy

e kF ≤ 4 e F.
kU − U kA − Ak
σn−1 + σn + σ
en−1 + σ
en

Proof. See W. Li and Sun [389, , Thm. 3.4].

A perturbation result for the orthogonal Procrustes problem (8.13) with the con-
straint det(W ) = 1 is given by Söderkvist [533, ].
202 The Polar Decomposition

8.3. Newton’s Method

Newton’s method for the unitary polar factor of a square matrix A can be derived by
applying Newton’s method to the equation X ∗ X = I (see Problem 8.18).

Newton iteration (polar decomposition):

1
Xk+1 = (Xk + Xk−∗ ), X0 = A. (8.17)
2

Theorem 8.12. Let A ∈ Cn×n be nonsingular. Then the Newton iterates Xk in

(8.17) converge quadratically to the unitary polar factor U of A, with
1
kXk+1 − U k ≤ kX −1 k kXk − U k2 . (8.18)
2 k
Proof. Let A = P ΣQ∗ be an SVD, so that U = P Q∗ . It is easy to show by
induction that Xk = P Dk Q∗ , where Dk+1 = 12 (Dk + Dk−1 ), D0 = Σ. This latter
iteration is the Newton iteration (5.16) for sign(Σ) = I, so Dk → I quadratically by
Theorem 5.6. Finally, (8.18) is obtained from Xk+1 − U = 21 (Xk − U )Xk−1 (Xk − U ).

A variant of (8.17) is

Newton iteration variant:

Yk+1 = 2Yk (I + Yk∗ Yk )−1 , Y0 = A, (8.19)

for which Yk ≡ Xk−∗ , k ≥ 1 (see Problem 8.19) and limk→∞ Yk = limk→∞ Xk−∗ =
U −∗ = U . Note that this iteration is applicable to rectangular A.
As for the Newton sign iteration in Section 5.3, one step of the Schulz iteration
can be used to remove the matrix inverse from (8.17), giving

Newton–Schulz iteration:
1
Xk+1 = Xk (3I − Xk∗ Xk ), X0 = A. (8.20)
2
√
This iteration is quadratically convergent to U if 0 < σi < 3 for every singular value
σi of A (see Problem 8.20). Again, the iteration is applicable to rectangular A.
Iterations (8.19) and (8.20) are both members of the Padé family (8.22).

8.4. Obtaining Iterations via the Matrix Sign Function

An alternative way to derive the three iterations of the previous section is by using the
relation (8.6) in conjunction with appropriate iterations for the matrix sign function.
Specifically, the
 idea  is to apply an iteration for the matrix sign
 function
 to the block
2×2 matrix A0∗ A0 , check that the iterates have the structure X0∗ X0k , and then read
k
off an iteration for the unitary polar factor from the (1,2) blocks. This derivation is
valid only for square A, although the resulting iterations may work for rectangular
A. An alternative connection that is valid for rectangular A is explained in the next
result.
8.5 The Padé Family of Methods 203

Theorem 8.13 (Higham, Mackey, Mackey, and Tisseur). Let A ∈ Cm×n be of rank
n and have the polar decomposition A = U H. Let g be any matrix function of the
form g(X) = X h(X 2 ) such that the iteration Xk+1 = g(Xk ) converges to sign(X0 )
for X0 = H with order of convergence m. Assume that g has the property that
g(X)∗ = g(X ∗ ). Then the iteration
Yk+1 = Yk h(Yk∗ Yk ), Y0 = A (8.21)
converges to U with order of convergence m.
Proof. Let Xk+1 = g(Xk ) with X0 = H, so that limk→∞ Xk = sign(H) = I. We
claim that Xk∗ = Xk and Yk = U Xk for all k. These equalities are trivially true for
k = 0. Assuming that they are true for k, we have
∗
Xk+1 = g(Xk )∗ = g(Xk∗ ) = g(Xk ) = Xk+1
and
Yk+1 = U Xk h(Xk∗ U ∗ U Xk ) = U Xk h(Xk2 ) = U Xk+1 .
The claim follows by induction. Hence limk→∞ Yk = U limk→∞ Xk = U . The order
of convergence is readily seen to be m.
This result is analogous to Theorem 6.11 (recall that the assumed form of g is
justified at the start of Section 6.7). Another way to prove the result is to use the
SVD to show that the convergence of (8.21) is equivalent to the convergence of the
scalar sign iteration xk+1 = g(xk ) with starting values the singular values of A.
To illustrate the theorem, we write the Newton sign iteration (5.16) as Xk+1 =
1 −1 1 2 −1
2 (X k + Xk ) = Xk · 2 (I + (Xk ) ). Then, for square matrices, the theorem yields
Yk+1 = Yk · 2 (I + (Yk Yk ) ) = 21 Yk (I + Yk−1 Yk−∗ ) = 12 (Yk + Yk−∗ ). More straightfor-
1 ∗ −1

wardly, starting with the Newton–Schulz iteration (5.22), Xk+1 = 12 Xk (3I − Xk2 ), we
immediately obtain (8.20).

8.5. The Padé Family of Methods

Theorem 8.13 enables us to translate the Padé family of iterations (5.28) for the
matrix sign function into a family of iterations for the unitary polar factor9 :

Padé iteration:

Xk+1 = Xk pℓm (I − Xk∗ Xk ) qℓm (I − Xk∗ Xk )−1 , X0 = A ∈ Cs×n . (8.22)

Recall that pℓm (ξ)/qℓm (ξ) is the [ℓ/m] Padé approximant to h(ξ) = (1 − ξ)−1/2 and
that some of the iteration functions fℓm (x) = xpℓm (1 − x2 )/qℓm (1 − x2 ) are given in
Table 5.1. (Of course, the derivation of the Padé iterations could be done directly for
the polar decomposition by using the analogue of (5.25): eiθ = z(1 − (1 − |z|2 ))−1/2 ).
In particular, ℓ = 0, m = 1 gives (8.19), while ℓ = m = 1 gives the Halley iteration
1 
−1 
Xk+1 = Xk (3I + Xk∗ Xk )(I + 3Xk∗ Xk )−1 = Xk I + 8 I + 3Xk∗ Xk , X0 = A,
3
in which the second expression is the more efficient to evaluate (cf. (5.32)). The
convergence of the Padé iterations is described in the next result.
9 In this section only we denote the row dimension of A by s in order to avoid confusion with the

Padé denominator degree.

204 The Polar Decomposition

Corollary 8.14. Let A ∈ Cs×n be of rank n and have the polar decomposition A =
U H. Consider the iteration (8.22) with ℓ+m > 0 and any subordinate matrix norm.
(a) For ℓ ≥ m − 1, if kI − A∗Ak < 1 then Xk → U as k → ∞ and kI − Xk∗ Xk k <
k
kI − A∗Ak(ℓ+m+1) .
(b) For ℓ = m − 1 and ℓ = m,
 (ℓ+m+1)k
(I − Hk )(I + Hk )−1 = (I − H)(I + H)−1 ,

where Xk = U Hk is a polar decomposition, and hence Xk → U as k → ∞ with rate

of convergence ℓ + m + 1.
Proof. Substituting Xk = U Hk in (8.22) yields Hk+1 = Hk pℓm (I − Hk2 ) qℓm (I −
Hk2 )−1 ,
H0 = H. The result therefore follows by applying Theorem 5.8 to H.
The partial fraction expansion (5.30) can be used to rewrite (8.22). For example,
the iterations with ℓ = m−1 have the partial fraction form (after a little manipulation
of (5.30))
X 1 m
1
Xk+1 = Xk (Xk∗ Xk + αi2 I)−1 , X0 = A ∈ Cs×n , (8.23)
m i=1
ξi

where  
1 (2i − 1)π 1
ξi = 1 + cos , αi2 = − 1, i = 1: m.
2 2m ξi
For s = 1 this formula is just (8.19). The following lemma proves a somewhat unex-
pected property of these iterations: after the first iteration all iterates have residual
less than 1.

Lemma 8.15. Let A ∈ Cs×n be of rank n and have the polar decomposition A = U H.
The iterates from (8.22) with ℓ = m − 1 satisfy kXk∗ Xk − Ik2 < 1 for k ≥ 1.
Proof. By using the SVDs of A and Xk the inequality can be reduced to the
corresponding inequality for the singular values of Xk , which satisfy the scalar it-
eration xk+1 = xk pr (1 − x2k )/qr (1 − x2k ) =: gr (xk ). Applying Theorem 5.9(a) with
r = ℓ + m + 1 = 2m it suffices to note that (a) 0 < gr (x) for x > 0 and (b) gr (x) < 1
for all x since r is even.
The lemma is useful because for the iterations with ℓ > m we need kI − A∗Ak < 1
to ensure convergence (Corollary √ 8.14). An example is the Newton–Schulz iteration
(8.20) (for which 0 < kAk2 < 3 is in fact sufficient for convergence, as noted in
Section 8.3). The lemma opens the possibility of carrying out at least one step of
one of the iterations with ℓ = m − 1 and then switching to (8.20) or some other
multiplication-rich iteration, safe in the knowledge that this second iteration will
converge.
These convergence results for the Padé iterations can be generalized to rank-
deficient A. As we have already observed, it suffices to consider the convergence of
the scalar iteration on the singular values of A. Zero singular values are fixed points
of the iteration and for the nonzero singular values the convergence to 1 is determined
by the results above. So for ℓ = m − 1 and ℓ = m we are guaranteed convergence
to the factor U in the canonical polar decomposition, while for ℓ > m convergence
to U is assured if |1 − σi2 | < 1 for all nonzero singular values σi of A. However,
8.6 Scaling the Newton Iteration 205

these results are mainly of theoretical interest because in practice rounding errors
will usually perturb the zero singular values, which will then converge to 1; the limit
matrix will then be of full rank and therefore a factor U in the polar decomposition.

8.6. Scaling the Newton Iteration

Scaling iterations for the polar decomposition in order to speed up the convergence
is desirable for the same reasons as for the matrix sign function (see Section 5.5). An
analogous theory holds, but the results are stronger—essentially because the iterations
can be reduced to uncoupled scalar iterations via unitary transformations. Consider
the Newton iteration (8.17) and its scaled form, for nonsingular A ∈ Cn×n and µk
real and positive,

Scaled Newton iteration:

1

Xk+1 = µk Xk + µ−1 −∗
k Xk , X0 = A. (8.24)
2

We consider three scalings:

−1/2
optimal scaling: µopt
k = σ1 (Xk )σn (Xk ) , (8.25)
 −1 −1 1/4
kXk k1 kXk k∞
1, ∞-norm scaling: µ1,∞
k = , (8.26)
kXk k1 kXk k∞
 1/2
kXk−1 kF
Frobenius norm scaling: µFk = . (8.27)
kXk kF

It is easy to show that the iterates from (8.24) satisfy

 
 µk σi (Xk ) − 1 
2
kXk+1 − U k2 ≤ θ(µk ) kXk+1 + U k2 , θ(µk ) = max  .
i=1:n µk σi (Xk ) + 1 

The quantity (8.25) minimizes θ(µk ) over all µk , and so is in this sense optimal. To
analyze optimal scaling it suffices to analyze the convergence of the singular values
(k)
of Xk to 1 (see the proof of Theorem 8.12). Write σi = σi (Xk ). Observe that this
scaling makes the smallest and largest singular values of Xk reciprocals of each other:
s s
(k) (k)
σ1 σn
σ1 (µk Xk ) = (k)
, σ n (µk X k ) = (k)
. (8.28)
σn σ1
These reciprocal values then map to the same value by (5.38a), which is the largest
singular value of X1 by (5.38b). Hence the singular values of Xk+1 satisfy
s s 
(k) (k)
(k+1) (k+1) 1 σ σ n
1 ≤ σn(k+1) ≤ · · · ≤ σ2 = σ1 =  1
(k)
+ (k)
. (8.29)
2 σn σ 1

It follows that
 
1 1
κ2 (Xk+1 ) ≤ κ2 (Xk )1/2 + ≤ κ2 (Xk )1/2 .
2 κ2 (Xk )1/2
206 The Polar Decomposition

(k)
By comparison, µk = 1 and if we assume for simplicity that σn = 1 then κ2 (Xk+1 ) =
1 (k) (k) 1
2 (σ1 + 1/σ1 ) ≈ 2 κ2 (Xk ), which shows a much less rapid reduction in large values
of κ2 (Xk ) for the unscaled iteration. A particularly interesting feature of optimal
scaling is that it ensures finite termination of the iteration.

Theorem 8.16 (Kenney and Laub). For the scaled Newton iteration (8.24) with op-
timal scaling (8.25), Xd = U , where d is the number of distinct singular values of A.
Proof. The argument just before the theorem shows that the multiplicity of the
largest singular value of Xk increases by one on every iteration. Hence at the end of
the dth iteration Xd−1 has all its singular values equal. The next scaling maps all the
singular values to 1 (i.e., µd−1 Xd−1 = U ), and so Xd = U .
In general we will not know that d in Theorem 8.16 is small, so the theorem is
of little practical use. However, it is possible to predict the number of iterations
accurately and cheaply via just scalar computations. By (8.29), the extremal singular
(1) (1)
values of X1 satisfy 1 ≤ σn ≤ σ1 . The largest singular value of µ1 X1 is then
(1)

(1) 1/2

(1) 1/2
σ1 /σn ≤ σ1 , and so from the properties in (5.38) of the map f (x) =
1
2 (x + 1/x) it follows that the singular values of X2 are
s s   
(1) (1) q q 
(2) (2) 1  σ 1 σ n
 1  (1) 1  (1)
1 ≤ σn ≤ · · · ≤ σ1 = (1)
+ (1)
≤ σ 1 + q = f σ 1 .
2 σn σ1 2 (1)
σ1

Therefore, with g (i) denoting the i-fold composition of g,

  1 √ √
(0)
kU − Xk k2 ≤ g (k) σ1 /σn(0) − 1, g(x) = ( x + 1/ x). (8.30)
2
Hence, given only knowledge of the extremal singular values of A we can bound the
number of iterations required by the Newton iteration with optimal scaling. For the
unscaled iteration a similar procedure can be employed to obtain the error exactly;
see Problem 8.22.
Figure 8.1 plots a bound on the number of iterations for optimal scaling for a
range of κ2 (A) = σ1 /σn . The bound is the smallest value of k for which the bound in
(8.30) is at most u = 2−53 . The figure shows, for example, that if κ2 (A) ≤ 1013 then
seven iterations suffice.
The optimal scaling (8.25) requires the extremal singular values of Xk . While
these can be approximated by the power method on A and A−1 , directly computable
scale parameters are perhaps more attractive. The 1, ∞-norm p scaling (8.26) √
is within
a constant factor of µopt
k : in view of the inequalities kAk2 ≤ kAk1 kAk∞ ≤ nkAk2 ,
we have
1
1/4
µopt
k ≤ µ1,∞
k ≤ n1/4 µopt
k .
n
The Frobenius norm scaling (8.27) has the property that it minimizes kXk+1 kF over
all µk ; see Problem 8.23.
Scaling can also be applied to the Padé iterations (8.22). Indeed because of the
relations in Theorem 5.9, for the principal Padé iterations the same scalings Xk ←
µk Xk with µk given by (8.25), (8.26), or (8.27) are applicable. Three caveats must
be borne in mind, however. First, higher order iterations permit less frequent scaling
than the Newton iteration (relative to the total number of iterations), so the benefits
8.7 Terminating the Iterations 207

7
Iterations
5

1
0 2 4 6 8 10 12 14 16
log (κ (A))
10 2

Figure 8.1. Bounds on number of iterations for Newton iteration with optimal scaling for
1 ≤ κ2 (A) ≤ 1016 .

of scaling are reduced. Second, the scaling parameters require the computation of
Xk−1 , which does not appear in the Padé iterations, so this is an added cost. Third,
practical experience shows that scaling degrades the numerical stability of the Padé
iterations [285, ], so there is a tradeoff between speed and stability. For more on
numerical stability, see Section 8.8.

8.7. Terminating the Iterations

We begin with two results that are analogues of Lemma 5.12 for the matrix sign
function.

Lemma 8.17. Let A ∈ Cm×n (m ≥ n) have the polar decomposition A = U H. Then

kA∗A − Ik kA∗A − Ik
≤ kA − U k ≤ ,
1 + σ1 (A) 1 + σn (A)
for any unitarily invariant norm.
Proof. It is straightforward to show that A∗A − I = (A − U )∗ (A + U ). Taking
norms and using (B.7) gives the lower bound. Since A + U = U (H + I) we have, from
the previous relation,

(A − U )∗ U = (A∗A − I)(H + I)−1 .

Hence
kA∗A − Ik
kA − U k = k(A − U )∗ U k ≤ kA∗A − Ik k(H + I)−1 k2 ≤ ,
1 + σn (A)
since the eigenvalues of H are the singular values of A.
This result shows that the two measures of orthonormality kA∗A − Ik and kA − U k
are essentially equivalent, in that they have the same order of magnitude if kAk < 1/2
(say). Hence the residual kXk∗ Xk − Ik of an iterate is essentially the same as the error
kU − Xk k. This result is useful for the Padé iterations, which form Xk∗ Xk .
The next result bounds the distance kA − U k in terms of the Newton correction
1
2 (A − A−∗ ).
208 The Polar Decomposition

Lemma 8.18. Let A ∈ Cn×n be nonsingular and have the polar decomposition A =
U H. If kA − U k2 = ǫ < 1 then for any unitarily invariant norm
   
1−ǫ 1+ǫ
kA − A−∗ k ≤ kA − U k ≤ kA − A−∗ k.
2+ǫ 2−ǫ

Proof. Let E = A − U . It is straightforward to show that

E = (A − A−∗ )(I + E ∗ U )(2I + E ∗ U )−1 .

Since kE ∗ U k2 = kEk2 = ǫ < 1, taking norms and using (B.7) yields

 
−∗ 1+ǫ
kEk ≤ kA − A k .
2−ǫ

The lower bound for E is obtained by taking norms in

A − A−∗ = E(2I + E ∗ U )(I + E ∗ U )−1 .

Note that if all the singular values of A are at least 1 (as is the case for the
scaled or unscaled Newton iterates Xk for k ≥ 1) then kA − U k ≤ kA − A−∗ k for
any unitarily invariant norm, with no restriction on kA − U k. This follows from the
inequality σi − 1 ≤ σi − σi−1 for each singular value of A and the characterization of
kAk as a symmetric gauge function of the singular values.
Let us consider how to build a termination criterion for the (unscaled) Newton
iteration. We can use similar reasoning as we used to derive the test (5.44) for
the matrix sign function. Lemma 8.18 suggests that once convergence has set in,
kXk − U k ≈ 12 kXk − Xk−∗ k. However, having computed Xk−∗ we might as well
compute the next iterate, Xk+1 , so it is the error in Xk+1 that we wish to bound in
terms of kXk − Xk−∗ k. In view of (8.18) we expect that

1 1 1
kXk+1 − U kF < −1 −∗ 2 −1 2
∼ 2 kXk kF 4 kXk − Xk kF = 2 kXk kF kXk+1 − Xk kF .

This suggests that we accept Xk+1 when

 1/2
2ηkXk+1 kF
kXk+1 − Xk kF ≤ ,
kXk−1 kF

where η is the desired relative error tolerance. Since Xk is converging to a unitary

matrix, close to convergence this test is effectively

kXk+1 − Xk kF ≤ (2η)1/2 . (8.31)

Modulo the constant 2, this is precisely the test (4.25) when Xk ≈ U , since c = n1/2 /2
therein. This test with an extra factor n1/2 on the right-hand side is also recommended
by Kielbasiński and Ziȩtak [352, , App. B]. Such a test should avoid the weakness
of a test of the form δk ≤ η that it tends to terminate the iteration too late (see the
discussion in Section 4.9.2). In particular, (8.31) allows termination after the first
iteration, which will be required in orthogonalization applications where the starting
matrix can be very close to being orthogonal (as in [601, ], for example).
8.8 Numerical Stability and Choice of H 209

8.8. Numerical Stability and Choice of H

What can be said about the computed polar factors obtained using any of the methods
described in this chapter? Let A ∈ Cm×n with m ≥ n. The best that can be expected
b and H
in floating point arithmetic is that the computed matrices U b satisfy

b = V + ∆U,
U V ∗ V = I, k∆U k ≤ ǫkV k, (8.32a)
b = K + ∆H,
H b ∗ = H,
H b k∆Hk ≤ ǫkKk, (8.32b)
V K = A + ∆A, k∆Ak ≤ ǫkAk, (8.32c)

where K is Hermitian positive semidefinite (and necessarily positive definite for the
2-norm, by (8.32c), if A has full rank and κ2 (A) < 1/ǫ) and ǫ is a small multiple of
the unit roundoff u. These conditions say that U b and Hb are relatively close to the
true polar factors of a matrix near to A. If the polar decomposition is computed via
the SVD then these ideal conditions hold with ǫ a low degree polynomial in m and
n, which can be shown using the backward error analysis for the SVD [224, ,
Sec. 5.5.8].
We turn now to iterations for computing the unitary polar factor. The next result
describes their stability.

Theorem 8.19 (stability of iterations for the unitary polar factor). Let the nonsin-
gular matrix A ∈ Cn×n have the polar decomposition A = U H. Let Xk+1 = g(Xk )
be superlinearly convergent to the unitary polar factor of X0 for all X0 sufficiently
close to U and assume that g is independent of X0 . Then the iteration is stable, and
the Fréchet derivative of g at U is idempotent and is given by Lg (U, E) = L(U, E) =
1 ∗ ∗ −1/2
2 (E − U E U ), where L(U ) is the Fréchet derivative of the map A → A(A A)
at U .

Proof. Since the map is idempotent, stability, the idempotence of Lg , and the
equality of Lg (U ) and L(U ), follow from Theorems 4.18 and 4.19. To find L(U, E) it
suffices to find Lg (U, E) for the Newton iteration, g(X) = 21 (X + X −∗ ). From

1
∗

g(U + E) = U + E + U −1 − U −1 EU −1 + O(kEk2 )
2
1

= U + E + U − U E ∗ U + O(kEk2 )
2

we have L(U, E) = Lg (U, E) = 21 (E − U E ∗ U ).

Theorem 8.19 shows that, as for the matrix sign iterations, all the iterations in
this book for the unitary polar factor are stable. The polar iterations generally have
better behaviour than the sign iterations because the limit matrix is unitary (and
hence of unit 2-norm) rather than just idempotent (and hence of unrestricted norm).
Since these iterations compute only U , a key question is how to obtain H. Given
a computed U b it is natural to take H
b = U b ∗ A. This matrix will in general not be
Hermitian, so we will replace it by the nearest Hermitian matrix (see Theorem 8.7):

b∗ ∗ b∗
b = (U A) + U A .
H (8.33)
2
210 The Polar Decomposition

(Ideally, we would take the nearest Hermitian positive semidefinite matrix, but this
would be tantamount to computing the polar decomposition of H, b in view of Theo-
b satisfies (8.32a), then U
rem 8.8). If U b ∗U
b = I + O(ǫ) and

b F = 1 k(U
b Hk
kA − U b ∗ A)∗ − U
b ∗ AkF + O(ǫ).
2
Hence the quantity
b ∗ A)∗ − U
k(U b ∗ AkF
b) =
β(U (8.34)
2kAkF
is a good approximation to the relative residual kA − U b Hk
b F /kAkF . Importantly,
b
β(U ) can be computed without an extra matrix multiplication.
We now summarize how to measure a posteriori the quality of a computed polar
decomposition A ≈ U bH b obtained via an iterative method for U b and (8.33). The
quantities kUb U
∗ b − IkF and kA − U b Hk
b F /kAkF should be of order the convergence
b
tolerance and H (which is guaranteed to be Hermitian) should be positive semidefinite.
The latter condition can be tested by computing the smallest eigenvalue or attempting
a Cholesky decomposition (with pivoting). If kU b ∗U
b − IkF is small then the relative
residual can be safely approximated by β(U ). b
The ultimate question regarding any iteration for the unitary polar factor is “what
can be guaranteed a priori about the computed U b ?” In other words, is there an a priori
forward or backward error bound that takes account of all the rounding errors in the
iteration as well as the truncation error due to terminating an iterative process? Such
results are rare for any matrix iteration, but Kielbasiński and Ziȩtak [352, ] have
done a detailed analysis for the Newton iteration with the 1, ∞-norm scaling (8.26).
Under the assumptions that matrix inverses are computed in a mixed backward–
forward stable way and that µ1,∞ k is never too much smaller than µopt
k they show that
the computed factors are backward stable. The assumption on the computed inverses
is not always satisfied when the inverse is computed by Gaussian elimination with
partial pivoting [276, , Sec. 14.1], but it appears to be necessary in order to push
through the already very complicated analysis. Experiments and further analysis are
given by Kielbasiński, Zieliński, and Ziȩtak [351, ].

8.9. Algorithm
We give an algorithm based on the Newton iteration (8.24) with the 1, ∞-norm scaling
(8.26).

Algorithm 8.20 (Newton algorithm for polar decomposition). Given A ∈ Cm×n of

rank n this algorithm computes the polar decomposition A = U H using the Newton
iteration. Two tolerances are used: a tolerance tol cgce for testing convergence and
a tolerance tol scale for deciding when to switch to the unscaled iteration.

1 if m > n
2 compute a QR factorization A = QR (R ∈ Cn×n )
3 A := R
4 end
5 X0 = A; scale = true
6 for k = 1: ∞
8.9 Algorithm 211

7 Yk = Xk−1 % If m > n, exploit triangularity of X0 .

8 if scale
1/4
9 µk = kYk k1 kYk k∞ /(kXk k1 kXk k∞ )
10 else
11 µk = 1
12 end
13 Xk+1 = 21 (µk Xk + µ−1 ∗
k Yk )
14 δk+1 = kXk+1 − Xk kF /kXk+1 kF
15 if scale = true and δk+1 ≤ tol scale, scale = false, end
16 if kXk+1 − Xk kF ≤ (tol cgce)1/2 or (δk+1 > δk /2 and scale = false)
17 goto line 20
18 end
19 end
20 U = Xk+1
21 H1 = U ∗ A
22 H = 21 (H1 + H1∗ )
23 if m > n, U = QU , end

Cost: For m = n: 2(k + 1)n3 flops, where k iterations are used. For m > n: 6mn2 +
(2k − 3 13 )n3 flops (assuming Q is kept in factored form).
The strategy for switching to the unscaled iteration, and the design of the conver-
gence test, are exactly as for Algorithm 5.14.
Possible refinements to the algorithm include:
(a) Doing an initial complete orthogonal decomposition (8.8) instead of a QR factor-
ization when the matrix is not known to be of full rank.
(b) Switching to the matrix multiplication-rich Newton–Schulz iteration once kI −
Xk∗ Xk k ≤ θ, for some θ < 1. The computation of Xk∗ Xk in this test can be avoided
by applying a matrix norm estimator to I − Xk∗ Xk [276, , Chap. 15] (see Higham
and Schreiber [286, ]).
We now give some numerical examples to illustrate the theory, obtained with
a MATLAB implementation of Algorithm 8.20. We took tol cgce = n1/2 u and
tol scale = 10−2 . We used three real test matrices:

1. a nearly orthogonal matrix, orth(gallery(’moler’,16))+ones(16)*1e-3,

2. a scalar multiple of an involutory matrix, gallery(’binomial’,16), and

3. the Frank matrix, gallery(’frank’,16).

We report in Tables 8.1–8.3 various statistics for each iteration, including δk+1 =
kXk+1 − Xk kF /kXk+1 kF and β(U b ) in (8.34). The line across each table follows the
iteration number for which the algorithm detected convergence. In order to compare
the stopping criterion in the algorithm with the criterion δk+1 ≤ tol cgce, we contin-
ued the iteration until the latter test was satisfied, and those iterations appear after
the line.
Several points are worth noting. First, the convergence test is reliable and for these
matrices saves one iteration over the simple criterion δk+1 ≤ tol cgce. Second, the
number of iterations increases with κ2 (A), as expected from the theory, but even for
the most ill conditioned matrix only seven iterations are required. Third, the Frank
matrix is well conditioned with respect to U but very ill conditioned with respect
212 The Polar Decomposition

Table 8.1. Results for nearly orthogonal matrix, n = 16. κU = 1.0, κ2 (A) = 1.0, β(Ub ) = 3.3×
10−16 b ∗b
, kU U − IkF = 1.4 × 10 −15 b b
, kA − U HkF /kAkF = 5.1 × 10 −16 b
, λmin (H) = 9.9 × 10−1 .

kXk − U kF
k kXk∗ Xk − IkF δk+1 kXk kF µk
kU kF
1 1.3e-5 1.1e-4 2.9e-3 4.0e+0
2 3.3e-10 2.6e-9 1.3e-5 4.0e+0
3 8.1e-16 1.5e-15 3.3e-10 4.0e+0
4 8.2e-16 1.4e-15 2.7e-16 4.0e+0

b) =
Table 8.2. Results for binomial matrix, n = 16. κU = 1.7 × 103 , κ2 (A) = 4.7 × 103 , β(U
b ∗U
3.5 × 10−16 , kU b − IkF = 1.4 × 10−15 , kA − U
b Hk
b F /kAkF = 4.2 × 10−16 , λmin (H)
b = 2.6.

kXk − U kF
k kXk∗ Xk − IkF δk+1 kXk kF µk
kU kF
1 1.7e+1 2.4e+3 2.6e+2 7.0e+1 5.5e-3
2 1.4e+0 2.2e+1 7.0e+0 9.2e+0 1.8e-1
3 1.3e-1 1.1e+0 1.2e+0 4.5e+0 5.9e-1
4 2.6e-3 2.1e-2 1.3e-1 4.0e+0 9.2e-1
5 1.4e-6 1.1e-5 2.6e-3 4.0e+0
6 1.3e-12 1.0e-11 1.4e-6 4.0e+0
7 3.3e-14 1.2e-15 1.3e-12 4.0e+0
8 3.3e-14 1.4e-15 2.3e-16 4.0e+0

b ) = 2.5×
Table 8.3. Results for Frank matrix, n = 16. κU = 5.2×101 , κ2 (A) = 2.3×1014 , β(U
b ∗U
10−16 , kU b − IkF = 1.1 × 10−15 , kA − U
b Hk
b F /kAkF = 3.7 × 10−16 , λmin (H)
b = 3.5 × 10−13 .

kXk − U kF
k kXk∗ Xk − IkF δk+1 kXk kF µk
kU kF
1 2.9e+6 8.4e+13 1.0e+0 1.1e+7 1.8e+5
2 1.9e+0 4.5e+1 1.1e+6 1.0e+1 1.1e-6
3 2.7e-1 2.5e+0 1.4e+0 4.9e+0 4.2e-1
4 9.5e-3 7.7e-2 2.6e-1 4.0e+0 8.3e-1
5 3.9e-5 3.1e-4 9.5e-3 4.0e+0
6 2.5e-9 2.0e-8 3.9e-5 4.0e+0
7 3.3e-15 9.6e-16 2.5e-9 4.0e+0
8 3.3e-15 1.1e-15 1.7e-16 4.0e+0
8.10 Notes and References 213

to inversion, since it has just one small singular value. Since the Newton iteration
inverts A on the first step, we might expect instability. Yet the algorithm performs in
a backward stable fashion, as it does for all three matrices. Finally, if optimal scaling
is used then the numbers of iterations are unchanged for the first and last matrices
and one less for the second, emphasizing the effectiveness of the 1, ∞-norm scaling.
Experience from these and many other experiments (see, e.g., [625, ]) suggests
that

• Algorithm 8.20 with tol cgce = nu requires at most about 8 iterations in IEEE
double precision arithmetic for matrices A not too close to being rank-deficient
(say, κ2 (A) ≤ 1014 ) and at most 1 or 2 more iterations for matrices numerically
rank-deficient;

• as long as A is not numerically rank-deficient, the algorithm produces computed

b and H
U b with kU b ∗U
b − IkF and kA − U b Hk
b F /kAkF of order tol cgce and H b
positive definite.

Thus the algorithm is remarkably stable, quick to converge, and robust—much more
so than the Newton sign iteration with any scaling (cf. Table 5.2, for example). And
its flop count is less than that for computation of the polar factors via the SVD (see
Problem 8.24).

8.10. Notes and References

The polar decomposition was introduced by Autonne in 1902 [24]; hence it is much
older than the matrix sign function and almost as old as the SVD (which was derived
by Beltrami in 1873). For a detailed history of the polar decomposition see Horn and
Johnson [296, , Sec. 3.0].
Some applications of the polar decomposition can be found in Section 2.6. An-
other application is to the construction of block Householder transformations (block
reflectors), which have the form H = Im − 2Z(Z ∗ Z)+ Z ∗ , where Z ∈ Cm×n (m ≥ n).
The desired task is to construct H so that for a given E ∈ Cm×n , HE = [F ∗ 0]∗ ,
where F ∈ Cn×n . The stable algorithms given by Schreiber and Parlett [508, ]
need to compute the unitary polar factor of one or more blocks of a matrix whose
columns form an orthonormal basis for range(E).
The terminology “canonical polar decomposition” is ours; unfortunately, there is
no standard name for this decomposition, which, confusingly, is often simply called
“the polar decomposition” in the literature. The term “generalized polar decompo-
sition” has been used by some authors, but this term is best reserved for decompo-
sitions defined in terms of an underlying scalar product, with U belonging to the
corresponding matrix automorphism group, or for appropriate decompositions in a
Lie group setting; see the references cited at the end of Section 14.1.1. Some authors
use the same name “polar decomposition” for the decomposition in Theorem 8.1 and
that in Theorem 8.3 without any restriction on the ranges of U and H, and regard
them as one decomposition. We feel this is inappropriate. What we call the polar
decomposition is defined only for m ≥ n and is distinguished by providing the nearest
matrix with orthonormal columns. The canonical polar decomposition is defined for
any m and n and is nonunique without the range condition, but with this condition
the factors U and H are both rank-deficient if A is. We are dealing with two different
decompositions.
214 The Polar Decomposition

Theorem 8.3 goes back at least to von Neumann [603, , Satz 7], but it has
been rediscovered by several authors. The result is most easily found in the literature
on generalized inverses—for example in Penrose’s classic paper [469, ] and in
Ben-Israel and Greville [52, , Thm. 7, p. 220]. For square A, the result appears
in an exercise of Halmos [243, , p. 171].
A “refined” polar decomposition A = U P D ∈ Cm×n (m ≥ n) is investigated
by Eirola [175, ]. Here, U has orthonormal columns, P is Hermitian positive
semidefinite with unit diagonal (and so is a correlation matrix), and D is diagonal with
nonnegative diagonal elements. Eirola shows that the decomposition always exists
(the proof is nontrivial) and considers uniqueness, computation, and an application.
An analytic polar decomposition A(t) = U (t)H(t) can be defined for a real matrix
A(t) whose elements are analytic functions of a real variable t. The factors are required
to be analytic, U (t) orthogonal, and H(t) symmetric but not definite. Existence of
the analytic polar decomposition is discussed by Mehrmann and Rath [419, ,
Sec. 3.1].
The integral formula (8.7) is due to Higham [273, ].
The case m = n in Theorem 8.4 is proved by Fan and Hoffman [181, ]; for
m > n the result is stated without proof by Rao [484, ] and a proof is given by
Laszkiewicz and Ziȩtak [372, ].
Theorem 8.5 is equivalent to a result of Laszkiewicz and Ziȩtak [372, ], in
which the constraint on Q is rank(Q) = rank(A), and was obtained for the Frobenius
norm by Sun and Chen [549, ]. For a related result with no constraints on the
partial isometry and that generalizes earlier results of Maher [404, ] and Wu [618,
] see Problem 8.12.
Theorem 8.7 is from Fan and Hoffman [181, ]. Theorem 8.8 is from Higham [269,
]. The best approximation property of (AH + H)/2 identified in Theorem 8.8
remains true in any unitarily invariant norm if A is normal, as shown by Bhatia and
Kittaneh [66, ]. For general matrices and the 2-norm, the theory of best Her-
mitian positive semidefinite approximation is quite different and does not involve the
polar decomposition; see Halmos [242, ] and Higham [269, ].
The orthogonal Procrustes problem (8.12) was first solved by Green [229, ]
(for full rank A and B) and Schönemann [507, ] (with no restrictions on A and
B). The rotation variant (8.13) was posed by Wahba [605, ], who explains that
it arises in the determination of the attitude of a satellite. Unlike the problem of
finding the nearest matrix with orthonormal columns, the solution to the orthogonal
Procrustes problem is not the same for all unitarily invariant norms, as shown by
Mathias [410, ].
Given that the QR factorization A = QR can be more cheaply computed than the
polar decomposition A = U H, for the purposes of orthogonalization it is natural to
ask whether Q can be used in place of U ; in other words, is kA − Qk close to kA − U k?
Two bounds provide some insight, both holding for A ∈ Cm×n of rank n under the
assumption that R has positive √ diagonal elements. Chandrasekaran and Ipsen [105,
] show that kA − QkF ≤ 5 n kA − U k2 , assuming that A has columns of unit
2-norm. Sun [547, ] proves that if kA∗ A − Ik2 < 1 then
1 + kAk2
kA − QkF ≤ √ kA − U kF .
2(1 − kA∗A − Ik2 )
The latter bound is the sharper for kA∗ A − Ik2 < 1/2.
The use of the complete orthogonal decomposition as a “preprocessor” before
applying an iterative method was suggested by Higham and Schreiber [286, ].
8.10 Notes and References 215

Theorem 8.9 has an interesting history. A version of the theorem was first de-
veloped by Higham [266, ], with a larger constant in the bound for ∆U and the
sharpness of the bounds not established. Barrlund [45, ] and Kenney and Laub
[342, , Thm. 2.3] were the first to recognize the differing sensitivity of U for real
and complex data. Unbeknown to the numerical analysts, in the functional analysis
literature the bound for ∆H had already been obtained via the analysis of the Lips-
chitz continuity of the
√ absolute value map. Araki and Yamagami√ [16, ] showed
that k|A| − |B|kF ≤ 2kA − BkF and that the constant 2 is as small as possible;
see Problem 8.15 for an elegant proof using ideas of Kittaneh. For some history of
this topic and further bounds on the norm of |A| − |B| see Bhatia [63, , Sec. 5],
[64, , Sec. X.2].
R.-C. Li [387, ] gives a different style of perturbation result for U in which the
perturbation is expressed in multiplicative form (A → XAY ); it shows that the change
in U depends only on how close X and Y are to the identity and not on the condition
of A. In subsequent work Li [388, ] obtains a perturbation bound for H for the
case where A is graded, that is, A = BD where B is well conditioned and the scaling
matrix D (usually diagonal) can be very ill conditioned; the bounds show that the
smaller elements of H can be much less sensitive to perturbations in A than the bound
in Theorem 8.9 suggests. Li also investigates how to compute H accurately, suggesting
the use of the SVD computed by the one-sided Jacobi algorithm; interestingly, it
proves important to compute H from U ∗ A and not directly from the SVD factors.
Some authors have carried out perturbation analysis for the canonical polar de-
composition under the assumption that rank(A) = rank(A + ∆A); see, for example,
R.-C. Li [385, ] and W. Li and Sun [389, ].
The polar decomposition cannot in general be computed in a finite number of
arithmetic operations and radicals, as shown by George and Ikramov [212, ],
[213, ]. For a companion matrix the polar decomposition is finitely computable
and Van Den Driessche and Wimmer [584, ] provide explicit formulae; formulae
for the block companion matrix case are given by Kalogeropoulos and Psarrakos [332,
].
The Newton–Schulz iteration (8.20) is used in Mathematica’s NDSolve function
within the projected integration method that solves matrix ODEs with orthogonal
solutions; see Sofroniou and Spaletta [534, ].
Theorem 8.13 is essentially a special case of a result of Higham, Mackey, Mackey,
and Tisseur [283, , Thm. 4.6] that applies to the generalized polar decomposition
referred to above.
Early papers on iterations for computing the unitary polar factor are those by
Björck and Bowie [72, ], Kovarik [361, ], and Leipnik [379, ]. Each
of these papers (none of which cites the others) develops families of polynomial
iterations—essentially the [ℓ/0] Padé iterations.
The Newton iteration (8.17) was popularized as a general tool for computing the
polar decomposition by Higham [266, ]. It had earlier been used in the aerospace
application mentioned in Section 2.6 to orthogonalize the 3×3 direction cosine matrix;
see Bar-Itzhack, Meyer, and Fuhrmann [43, ].
Gander [202, ] (see also Laszkiewicz and Ziȩtak [372, ]) obtains conditions
on h for iterations of the form (8.21) to have a particular order of convergence and
also considers applying such iterations to rank deficient matrices.
The particular Padé iteration (8.23) in partial fraction form was derived from
the corresponding sign iteration by Higham [273, ]. It was developed into a
216 The Polar Decomposition

practical algorithm for parallel computers by Higham and Papadimitriou [285, ],
who obtain order of magnitude speedups over computing the polar factors via the
SVD on one particular virtual shared memory MIMD computer.
Lemma 8.15 is due to Higham [273, ].
The optimal scaling (8.25) and its approximations (8.26) and (8.27) were suggested
by Higham [265, ], [266, ]. All three scalings are analyzed in detail by Kenney
and Laub [344, ]. The Frobenius norm scaling is also analyzed by Dubrulle [169,
]; see Problem 8.23.
Theorem 8.16 is due to Kenney and Laub [344, ], who also derive (8.30).
Lemmas 8.17 and 8.18 are from Higham [273, ]. Theorem 8.19 is new.
All the globally convergent iterations described in this chapter involve matrix
inverses or the solution of multiple right-hand side linear systems. Problem 8.26
describes how the Newton iteration variant (8.19) can be implemented in an inversion-
free form. The idea behind this implementation is due to Zha and Zhang [622, ],
who consider Hermitian matrices and apply the idea to subspace iteration and to
iterations for the matrix sign function.
A drawback of the algorithms described in this chapter is that they cannot take
advantage of a known polar decomposition of a matrix close to A. Thus knowing a
polar decomposition A e=U eHe is of no help when computing the polar decomposition
of A, however small the norm or rank of A − A. e Indeed the iterations all require A
as the starting matrix. The same comment applies to the iterations for the matrix
sign function and matrix roots. The trace maximization algorithm outlined in Prob-
lem 8.25 can take advantage of a “nearby” polar decomposition but, unfortunately,
the basic algorithm is not sufficiently efficient to be of practical use. An SVD updat-
ing technique of Davies and Smith [138, ] could potentially be used to update
the polar decomposition. See Problem 8.27.

Problems
8.1. Find all polar decompositions of the Jordan block Jm (0) ∈ Cm×m .
8.2. (Uhlig [581, ]) Verify that for nonsingular A ∈ R2×2 the polar factors are



U = γ A + | det(A)|A−T , H = γ ATA + | det(A)|I ,

where 
−1/2
γ = det A + | det(A)| A−T  .

8.3. What are the polar and canonical polar decompositions of 0 ∈ Cm×n ?
8.4. (Higham, Mackey, Mackey, and Tisseur [283, , Thm. 4.7]) If Q ∈ Cn×n is
unitary and −1 ∈/ Λ(Q) what is the polar decomposition of I + Q? How might we
compute Q1/2 iteratively using (8.17)?
8.5. (Moakher [434, ]) Show that if Q1 , Q2 ∈ Cn×n are unitary and −1 ∈ /
Λ(Q∗1 Q2 ) then then the unitary polar factor of Q1 + Q2 is Q1 (Q∗1 Q2 )1/2 .
8.6. Let A ∈ Cm×n . Show that if H = (A∗A)1/2 then null(H) = null(A).
8.7. For A ∈ Cm×n with m < n investigate the existence and uniqueness of polar
decompositions A = U H with U ∈ Cm×n having orthonormal rows and H ∈ Cn×n
Hermitian positive semidefinite.
Problems 217

8.8. Show that the condition range(U ∗ ) = range(H) in Theorem 8.3 is equivalent to
range(U ) = range(A).
8.9. Let A ∈ Cn×n be normal and nonsingular. Show that the polar factors U and
H may be expressed as functions of A.
8.10. Give a proof of Theorem 8.4 from first principles for the 2-norm.
m×n

 A,∗B ∈ C
8.11. Let . Show that min kA − BW kF : W ∈ Cn×n , W ∗ W = I
and min kB A − W kF : W ∈ Cn×n , W ∗ W = I are attained at the same matrix
W . Thus the orthogonal Procrustes problem problem reduces to the nearest unitary
matrix problem.
8.12. (Laszkiewicz and Ziȩtak [372, ]) Show that for A ∈ Cm×n and any unitarily
invariant norm
min { kA − Qk : Q ∈ Cm×n is a partial isometry } = max max(σi , |1 − σi |),
i=1:min(m,n)

where the σi are the singular values of A, and determine a maximizer.

8.13. (a) Show that for A ∈ Cn×n , Re trace(W ∗ A) is maximized over all unitary
W ∈ Cn×n if and only if W is a unitary polar factor of A (or equivalently, W ∗ A
is Hermitian positive semidefinite). Deduce that the maximizer is unique if A is
nonsingular.
(b) Show that for A ∈ Rn×n , Re trace(W ∗ A) is maximized over all orthogonal W ∈
Rn×n with det(W ) = 1 by W = P diag(1, . . . , 1, det(P Q∗ ))Q∗ , where A = P ΣQ∗ is
an SVD. Show that the maximizer is unique if (i) det(P Q∗ ) = 1 and σn−1 6= 0 or (ii)
det(P Q∗ ) = −1 and σn−1 > σn , where Σ = diag(σi ).
8.14. (Beattie and Smith [49, ]) Solve the following weighted variation of the
nearest matrix with orthonormal columns problem in Theorem 8.4:
min{ kM 1/2 (A − Q)kF : Q∗ M Q = In },
where A ∈ Cm×n (m ≥ n) and M ∈ Cn×n is Hermitian positive definite. This
problem arises in structural identification.
8.15. (Bhatia [63, ], Bhatia [64, , Sec. VII.5], Kittaneh [353, ], Kittaneh
[354, ]) Show that for all A, B ∈ Cm×n
√
k |A| − |B| kF ≤ 2kA − BkF (8.35)
by following these steps.

(i) Let f be any function satisfying

|f (z) − f (w)| ≤ c|z − w| for all z, w ∈ C.
Show that for all normal A ∈ Cn×n and all X ∈ Cn×n ,
kf (A)X − Xf (A)kF ≤ ckAX − XAkF .
   
(ii) By applying the previous part to A ← A0 B0 and X ← 00 X0 (“”) deduce that
kf (A)X − Xf (B)kF ≤ ckAX − XBkF
for all normal A, B ∈ Cn×n and all X ∈ Cn×n .
218 The Polar Decomposition

(iii) Show that if A and B are both normal then

k|A| − |B|kF ≤ kA − BkF .

 0 A
  0 B

(iv) By applying the previous part to A ← A∗ 0 and B ← B∗ 0 deduce that for
all A, B ∈ Cm×n

k|A| − |B|k2F + k|A∗ | − |B ∗ |k2F ≤ 2kA − Bk2F .

8.16. (Scalar version of Theorem 8.10. This bound is not readily found in texts on
complex analysis.) Show if z1 = r1 eiθ1 and z2 = r2 eiθ2 are complex numbers in polar
form then
|z1 − z2 |
|eiθ1 − eiθ2 | ≤ 2 .
r1 + r2

8.17. (Li [386, ]) Let

   
1 0 1 0
A = 0 ǫ, e = 0
A ǫ,
0 0 0 δ

where ǫ ≫ δ > 0. Show that the difference kU − U e kF between the respective polar
factors is of order δ/ǫ, showing that the sensitivity of U depends on 1/σn and not on
1/(σn + σn−1 ) as it would if A were square.
8.18. Show that the Newton iteration (8.17) can be derived by applying Newton’s
method to the equation X ∗ X = I.
8.19. Prove that for nonsingular A ∈ Cn×n the iterates Xk from (8.17) and Yk from
(8.19) are related by Yk = Xk−∗ for k ≥ 1.
8.20. Let A ∈ Cm×n of rank n have the polar decomposition A = U H. Show that√in
the Newton–Schulz iteration (8.20), Xk → U quadratically as k → ∞ if kAk2 < 3,
and that
1
kXk+1 − U k2 ≤ kXk + 2U k2 kXk − U k22 . (8.36)
2
Show also that Rk = I − Xk∗ Xk satisfies Rk+1 = 43 Rk2 + 41 Rk3 .
8.21. The Newton iteration (8.17) and the Newton–Schulz iteration (8.20) are both
quadratically convergent. The convergence of the Newton iteration is described by
(8.18) and that of Newton–Schulz by (8.36). What do the error constants in the error
relations for these two iterations imply about their relative speeds of convergence?
(1)
8.22. Show for the unscaled Newton iteration (8.17) that kU −Xk k2 = f (k−1) (σ1 )−
(k)
1, where f (x) = 21 (x + 1/x) and σi = σi (Xk ). [Cf. (8.30) for the optimally scaled
iteration.]
8.23. (Dubrulle [169, ]) (a) Show that the Frobenius norm scaling (8.27) mini-
mizes kXk+1 kF over all µk .
(b) Show that the 1, ∞-norm scaling (8.26) minimizes a bound on kXk+1 k1 kXk+1 k∞ .
8.24. Compare the flop count of Algorithm 8.20 with that for computation of the
polar factors via the SVD.
Problems 219

8.25. We know from Problem 8.13 that the polar factors U and H of A ∈ Rn×n satisfy
max{ trace(W ∗ A) : W ∗ W = I } = trace(U ∗ A) = trace(H). Thus premultiplying A
by U ∗ both symmetrizes it and maximizes the trace. This suggests developing an
algorithm for computing the polar decomposition that iteratively maximizes the trace
by premultiplying A by a succession of suitably chosen orthogonal matrices.
 
(a) Show that if aij 6= aji then a Givens rotation G = −cos
sin
θ sin θ
θ cos θ exists such that

T aii aij

G aji ajj is symmetric and has maximal trace over all θ.
(b) Show that if A is symmetric but indefinite then a Householder transformation
G = I − 2vv T /(v T v) can be chosen so that trace(GA) > trace(A).
(c) Develop an algorithm for computing the polar decomposition based on a combi-
nation of the transformations in (a) and (b).

8.26. (a) Let A ∈ Cm×n and let

n n
   
n I n Q1
= R
m A m Q2

be a QR factorization. The polar decomposition of this matrix is

n
 
n I
= U (I + A∗ A)1/2 .
m A

Since the columns of Q and U span the same space,

   
Q1 I
= UM = (I + A∗ A)−1/2 · M
Q2 A

for some unitary M . By examining the blocks of this equation, obtain an expression
for Q2 Q∗1 and hence show that the Newton iteration variant (8.19), which we rewrite
as
Xk+1 = 2Xk (I + Xk∗ Xk )−1 , X0 = A, (8.37)
can be written as

n n
   (k) 
n I n Q1
= (k) Rk (QR factorization), (8.38a)
m Xk m Q2
(k) (k) ∗
Xk+1 = 2Q2 Q1 . (8.38b)

This is an inverse-free implementation of the iteration.

(b) Evaluate and compare the operation counts for (8.37) and (8.38).
(c) How can iteration (8.38) be scaled?
8.27. (Research problem) Develop ways to efficiently update the polar decompo-
sition after a small-normed or low rank perturbation.
Chapter 9
Schur–Parlett Algorithm

We have seen in earlier chapters that reliable algorithms for computing the matrix
sign function and matrix roots can be constructed by using the Schur decomposition
to reduce the problem to the triangular case and then exploiting particular properties
of the functions. We now develop a general purpose algorithm for computing f (A)
via the Schur decomposition. We assume that the reader has read the preliminary
discussion in Section 4.6.
Our algorithm for computing f (A) consists of several stages. The Schur decom-
position A = QT Q∗ is computed; T is reordered and blocked to produce another
triangular matrix Te with the property that distinct diagonal blocks have “sufficiently
distinct” eigenvalues and the eigenvalues within each diagonal block are “close”; the
diagonal blocks f (Teii ) are computed; the rest of f (Te) is obtained using the block form
of the Parlett recurrence; and finally the unitary similarity transformations from the
Schur decomposition and the reordering are reapplied. We consider first, in Sec-
tion 9.1, the evaluation of f on the atomic triangular blocks Teii , for which we use
a Taylor series expansion. “Atomic” refers to the fact that these blocks cannot be
further reduced. This approach is mainly intended for functions whose Taylor series
have an infinite radius of convergence, such as the trigonometric and hyperbolic func-
tions, but for some other functions this step can be adapted or replaced by another
technique, as we will see in later chapters. In Section 9.2 we analyze the use of the
block form of Parlett’s recurrence. Based on the conflicting requirements of these
two stages we describe a Schur reordering and blocking strategy in Section 9.3. The
overall algorithm is summarized in Section 9.4, where its performance on some test
matrices is illustrated. The relevance of several preprocessing techniques is discussed
in Section 9.5.

9.1. Evaluating Functions of the Atomic Blocks

In this section T ∈ Cn×n represents an atomic block of the reordered Schur form.
It is therefore an upper triangular matrix whose eigenvalues are “close”. Given an
arbitrary function f , we need a method for evaluating f (T ) efficiently and accurately.
A natural approach is to expand f in a Taylor series about the mean of the eigenvalues
of T . Write
T = σI + M, σ = trace(T )/n, (9.1)

which defines M as T shifted by the mean of its eigenvalues. If f has a Taylor series
representation
∞
X f (k) (σ) k
f (σ + z) = z (9.2)
k!
k=0

221
222 Schur–Parlett Algorithm

for z in an open disk containing Λ(T − σI), then, by Theorem 4.7,

∞
X f (k) (σ)
f (T ) = M k. (9.3)
k!
k=0

If T has just one eigenvalue, so that tii ≡ σ, then M is strictly upper triangular
and hence is nilpotent with M n = 0; the series (9.3) is then finite. More generally, if
the eigenvalues of T are sufficiently close, then the powers of M can be expected to
decay quickly after the (n − 1)st, and so a suitable truncation of (9.3) should yield
good accuracy. This notion is made precise in the following lemma, in which M is
represented by M = D + N , with D diagonal and N strictly upper triangular and
hence nilpotent with M n = 0.

Lemma 9.1 (Davies and Higham). Let D ∈ Cn×n be diagonal with |D| ≤ δI and let
N ∈ Cn×n be strictly upper triangular. Then
min(k,n−1)  
X k k−i
k
|(D + N ) | ≤ δ |N |i
i=0
i

and the same inequality holds with the absolute values replaced by any matrix norm
subordinate to an absolute vector norm.
Proof. The bound follows from
|(D + N )k | ≤ (|D| + |N |)k ≤ (δI + |N |)k ,
followed by a binomial expansion of the last term. Since |N |n−1 = 0 we can drop
the terms involving |N |i for i ≥ n − 1. The corresponding bound for matrix norms
is obtained by taking norms in the binomial expansion of (D + N )k and using (B.6).

If δ < 1 and δ ≪ kN k in Lemma 9.1, then, for k ≥ n − 1,

k(D + N )k k = O(δ k+1−n kN kn−1 ),
and hence the powers of D + N decay rapidly after the (n − 1)st, irrespective of N .
This analysis shows that as long as the scalar multipliers f (k) (σ)/k! in (9.3) are
not too large we should be able to truncate the series (9.3) soon after the (n − 1)st
term (and possibly much earlier if M is small).
We need a reliable criterion for deciding when to truncate the Taylor series. When
summing a series whose terms decrease monotonically it is safe to stop as soon as a
term is smaller than the desired error. Unfortunately, our matrix Taylor series can
exhibit very nonmonotonic convergence. Indeed, when n = 2, M = T − σI always
has the form  
ǫ α
M= , (9.4)
0 −ǫ
and its powers are
   
2k ǫ2k 0 2k+1 ǫ2k+1 αǫ2k
M = , M = .
0 ǫ2k 0 −ǫ2k+1

For |ǫ| < 1, kM k k → 0 as k → ∞, but kM 2k+1 k ≫ kM 2k k for α ≫ 1. The

next theorem shows that this phenomenon of the “disappearing nonnormal part” is
connected with the fact that f can map distinct λi into the same value.
9.1 Evaluating Functions of the Atomic Blocks 223

Theorem 9.2 (Davies and Higham). Let D ∈ Cn×n be diagonal with distinct eigen-
values λ1 , . . . , λp (1 ≤ p ≤ n) of multiplicity k1 , . . . , kp , respectively, and let the values
f (j) (λi ), j = 0: ki − 1, i = 1: p, be defined. Then f (D + N ) = f (D) for all strictly
triangular N ∈ Cn×n if and only if f (D) = f (λ1 )I and

f (j) (λi ) = 0, j = 1: ki − 1, i = 1: p. (9.5)

Note that (9.5) is vacuous when ki = 1.

Proof. See Problem 9.1.

Applying Theorem 9.2 to the function f (x) = xk we obtain the following corollary.

Corollary 9.3. Let D ∈ Cn×n be a nonzero diagonal matrix and let k ≥ 2. Then
(D + N )k = Dk for all strictly triangular matrices N ∈ Cn×n if and only if

D = β diag(e2k1 πi/k , e2k2 πi/k , . . . , e2kn πi/k ),

where β 6= 0, ki ∈ {0, 1, . . . , k − 1} and the ki are distinct (and hence k ≥ n).

Proof. By Theorem 9.2, all the diagonal elements of D must be kth roots of
the same number, β k say. The condition (9.5) implies that any repeated diagonal
element dii must satisfy f ′ (dii ) = kdk−1
ii = 0, which implies dii = 0 and hence D = 0;
therefore D has distinct diagonal elements.
As a check, we note that the diagonal of M in (9.4) is of the form in the corollary
for even powers k. The corollary shows that this phenomenon of very nonmonotonic
convergence of the Taylor series can occur when the eigenvalues are a constant multiple
of kth roots of unity. As is well known, the computed approximations to multiple
eigenvalues occurring in a single Jordan block tend to have this distribution. We
will see in the experiment of Section 9.4 that this eigenvalue distribution also causes
problems in finding a good blocking.
We now develop a strict bound for the truncation error of the Taylor series, which
we will use to decide when to terminate the series. We apply Theorem 4.8 with
A := σI + M , α := σ, M from (9.1), and the Frobenius norm, and so we need to be
able to bound max0≤t≤1 kM s f (s) (σI + tM )kF . We will bound it by the product of
the norms, noting that the term M s is needed anyway if we form the next term of
the series. To bound max0≤t≤1 kf (s) (σI + tM )kF we can use Theorem 4.28 to show
that
ωs+r
max kf (s) (σI + tM )kF ≤ max k(I − |N |)−1 kF , (9.6)
0≤t≤1 0≤r≤n−1 r!

where ωs+r = supz∈Ω |f (s+r) (z)| and N is the strictly upper triangular part of M .
By using (9.6) in (4.8) we can therefore bound the truncation error. Approximating
the Frobenius norm by the ∞-norm, the term k(I − |N |)−1 k∞ can be evaluated
in just O(n2 ) flops, since I − |N | is an M -matrix: we solve the triangular system
(I − |N |)y = e, where e = [1, 1, . . . , 1]T , and then kyk∞ = k(I − |N |)−1 k∞ [276, ,
Sec. 8.3].
We now state our algorithm for evaluating a function of an atomic block via the
Taylor series.

Algorithm 9.4 (evaluate function of atomic block). Given a triangular matrix T ∈

Cn×n whose eigenvalues λ1 , . . . , λn are “close,” a function f having the Taylor series
224 Schur–Parlett Algorithm

Pn
(9.2) for z in an open disk containing λi − σ, i = 1: n, where σ = n−1 i=1 λi , and
the ability to evaluate derivatives of f , this algorithm computes F = f (T ) using a
truncated Taylor series.

Pn
1 σ = n−1 i=1 λi , M = T − σI, tol = u
2 µ = kyk∞ , where y solves (I − |N |)y = e and N is the strictly
upper triangular part of T .
3 F0 = f (σ)In
4 P =M
5 for s = 1: ∞
6 Fs = Fs−1 + f (s) (σ)P
7 P = P M/(s + 1)
8 if kFs − Fs−1 kF ≤ tolkFs kF
% Successive terms are close so check the truncation error bound.
9 Estimate or bound ∆ = max0≤r≤n−1 ωs+r /r!, where
ωs+r = supz∈Ω |f (s+r) (z)|, with Ω a closed convex set containing Λ(T ).
10 if µ∆kP kF ≤ tolkFs kF , quit, end
11 end
12 end

Unless we are able to exploit particular properties of f , we can in practice take

ωs+r = max{ |f (s+r) (λi )| : λi ∈ Λ(T ) }.
Algorithm 9.4 costs O(n4 ) flops, since even if T has constant diagonal, so that M
is nilpotent with M n = 0, the algorithm may need to form the first n − 1 powers of
M . Although we usually insist on O(n3 ) flops algorithms in numerical linear algebra,
this higher order operation count is mitigated by two factors. First, n here is the size
of a block, and in most cases the blocks will be of much smaller dimension than the
original matrix. Second, M is an upper triangular matrix, so forming all the powers
M 2 , . . . , M n−1 costs n4 /3 flops—a factor 6 less than the flop count for multiplying
full matrices.
Since in our overall f (A) algorithm we are not able to impose a fixed bound on
the spread maxi,j |tii − tjj | of the diagonal of T , Algorithm 9.4 is suitable in its stated
form only for functions that have a Taylor series with an infinite radius of convergence,
such as exp, cos, sin, cosh, and sinh.
We now turn to the effects of rounding errors on Algorithm 9.4. Ignoring trunca-
tion errors, standard error analysis shows that the best possible forward error bound
is of the form
∞
X |f (k) (σ)|
|F − Fb| ≤ γ
en |M |k .
k!
k=0

If there is heavy cancellation in the sum (9.3) then a large relative error kF −Fbk/kF k is
possible. This danger is well known, particularly in the case of the matrix exponential
(see Chapter 10). A mitigating factor here is that our matrix T is chosen to have
eigenvalues that are clustered, which tends to limit the amount of cancellation in the
sum. However, for sufficiently far from normal T , damaging cancellation can take
place. For general functions there is little we can do to improve the accuracy; for
particular f we can of course apply alternative methods, as illustrated in Chapters 10
and 11.
9.2 Evaluating the Upper Triangular Part of f (T ) 225

9.2. Evaluating the Upper Triangular Part of f (T )

Let T denote the triangular Schur factor of A, which we assume to have been reordered
and blocked so that Tii and Tjj have no eigenvalue in common for i 6= j. To evaluate
the upper triangular part of F = f (T ) we use the block form (4.19) of Parlett’s
recurrence, which we rewrite here as
j−1
X
Tii Fij − Fij Tjj = Fii Tij − Tij Fjj + (Fik Tkj − Tik Fkj ). (9.7)
k=i+1

This Sylvester equation is nonsingular and it is easy to see that Fij can be computed
a column at a time from first to last, with each column obtained as the solution of a
triangular system. Of particular concern is the propagation of errors in the recurrence.
These errors are of two sources: errors in the evaluation of the diagonal blocks Fii ,
and rounding errors in the formation and solution of (9.7). To gain insight into both
types of error we consider the residual of the computed solution Fb:

T Fb − FbT =: R, (9.8)

where Rij is the residual from the solution of the Sylvester equation (9.7). Although
it is possible to obtain precise bounds on R, these are not important to our argument.
Writing Fb = F + ∆F , on subtracting T F − F T = 0 from (9.8) we obtain

T ∆F − ∆F T = R.

As for the original equation T F − F T = 0, this equation uniquely determines the

off-diagonal blocks ∆F in terms of the diagonal blocks. Equating (i, j) blocks yields
j−1
X
Tii ∆Fij − ∆Fij Tjj = Rij + ∆Fii Tij − Tij ∆Fjj + (∆Fik Tkj − Tik ∆Fkj )
k=i+1
=: Bij , (9.9)

and these equations can be solved to determine ∆Fij a block superdiagonal at a time.
It is straightforward to show that

k∆Fij kF ≤ sep(Tii , Tjj )−1 kBij kF , (9.10)

where sep is the separation of Tii and Tjj :

kTii X − XTjj kF
sep(Tii , Tjj ) = min .
X6=0 kXkF
It follows that rounding errors introduced during the stage at which Fij is com-
puted (i.e., represented by Rij ) can lead to an error ∆Fij of norm proportional to
sep(Tii , Tjj )−1 kRij k. Moreover, earlier errors (represented by the ∆Fij terms on the
right-hand side of (9.9)) can be magnified by a factor sep(Tii , Tjj )−1 . It is also clear
from (9.9) that even if sep(Tii , Tjj )−1 is not large, serious growth of errors in the
recurrence (9.9) is possible if some off-diagonal blocks Tij are large.
To minimize the bounds (9.10) for all i and j we need the blocks Tii to be as
well separated as possible in the sense of sep. However, trying to maximize the
separations between the diagonal blocks Tii tends to produce larger blocks with less
226 Schur–Parlett Algorithm

tightly clustered eigenvalues, which increases the difficulty of evaluating f (Tii ), so any
strategy for reordering the Schur form is necessarily a compromise.
Computing sep(Tii , Tjj ) exactly when both blocks are m × m costs O(m4 ) flops,
while condition estimation techniques allow an estimate to be computed at the cost of
solving a few Sylvester equations, that is, in O(m3 ) flops [87, ], [270, ], [324,
]. It is unclear how to develop a reordering and blocking strategy for produc-
ing “large seps” at reasonable cost; in particular, it is unclear how to define “large.”
Indeed the maximal separations are likely to be connected with the conditioning of
f (T ), but little or nothing is known about any such connections. More generally, how
to characterize matrices for which the condition number of f is large is not well under-
stood, even for the matrix exponential (see Section 10.2). Recalling the equivalence
mentioned in Section 4.7 between block diagonalization and the use of the Parlett
recurrence, a result of Gu [232, ] provides further indication of the difficulty of
maximizing the seps: he shows that, given a constant τ , finding a similarity transfor-
mation with condition number bounded by τ that block diagonalizes (with at least
two diagonal blocks) a triangular matrix is NP-hard.
In the next section we will adopt a reordering and blocking strategy that bounds
the right-hand side of the approximation
1
sep(Tii , Tjj )−1 ≈
min{ |λ − µ| : λ ∈ Λ(Tii ), µ ∈ Λ(Tjj ) }
by the reciprocal of a given tolerance. The right-hand side is a lower bound for the
left that can be arbitrarily weak, but it is a reasonable approximation for matrices
not too far from being normal.
It is natural to look for ways of improving the accuracy of the computed Fb from the
Parlett recurrence. One candidate is fixed precision iterative refinement of the systems
(9.7). However, these systems are essentially triangular, and standard error analysis
shows that the backward error is already small componentwise [276, , Thm. 8.5];
fixed precision iterative refinement therefore cannot help. The only possibility is to
use extended precision when solving the systems.

9.3. Reordering and Blocking the Schur Form

We wish to reorder the upper triangular Schur factor T into a partitioned upper
triangular matrix Te = U ∗ T U = (Teij ), where U is unitary and two conditions hold:
1. separation between blocks:
min{ |λ − µ| : λ ∈ Λ(Teii ), µ ∈ Λ(Tejj ), i 6= j } > δ, (9.11)

2. separation within blocks: for every block Teii with dimension bigger than 1, for
every λ ∈ Λ(Teii ) there is a µ ∈ Λ(Teii ) with µ 6= λ such that |λ − µ| ≤ δ.
Here, δ > 0 is a blocking parameter. The second property implies that for Teii ∈ Rm×m
(m > 1)
max{ |λ − µ| : λ, µ ∈ Λ(Teii ), λ 6= µ } ≤ (m − 1)δ,
and this bound is attained when, for example, Λ(Teii ) = {δ, 2δ, . . . , mδ}.
The following algorithm is the first step in obtaining such an ordering. It can be
interpreted as finding the connected components of the graph on the eigenvalues of
T in which there is an edge between two nodes if the corresponding eigenvalues are a
distance at most δ apart.
9.3 Reordering and Blocking the Schur Form 227

Algorithm 9.5 (block pattern). Given a triangular matrix T ∈ Cn×n with eigenval-
ues λi ≡ tii and a blocking parameter δ > 0, this algorithm produces a block pattern,
defined by an integer vector q, for the block Parlett recurrence: the eigenvalue λi is
assigned to the set Sqi , and it satisfies the conditions that min{|λi − λj |: λi ∈ Sp , λj ∈
Sq , p 6= q} > δ and, for each set Si with more than one element, every element of Si
is within distance at most δ from some other element in the set. For each set Sq , all
the eigenvalues in Sq are intended to appear together in an upper triangular block Teii
of Te = U ∗ T U .

1 p=1
2 Initialize the Sp to empty sets.
3 for i = 1: n
4 if λi ∈
/ Sq for all 1 ≤ q < p
5 Assign λi to Sp .
6 p=p+1
7 end
8 for j = i + 1: n
9 Denote by Sqi the set that contains λi .
10 if λj ∈ / Sqi
11 if |λi − λj | ≤ δ
12 if λj ∈
/ Sk for all 1 ≤ k < p
13 Assign λj to Sqi .
14 else
15 Move the elements of Smax(qi ,qj ) to Smin(qi ,qj ) .
16 Reduce by 1 the indices of sets Sq for q > max(qi , qj ).
17 p=p−1
18 end
19 end
20 end
21 end
22 end

Algorithm 9.5 provides a mapping from each eigenvalue λi of T to an integer qi

such that the set Sqi contains λi . Our remaining problem is equivalent to finding a
method for swapping adjacent elements in q to obtain a confluent permutation q ′ .
A confluent permutation of n integers, q1 , . . . , qn , is a permutation such that any
repeated integers qi are next to each other. For example, there are 3! confluent per-
mutations of (1, 2, 1, 3, 2, 1), which include (1, 1, 1, 3, 2, 2) and (3, 2, 2, 1, 1, 1). Ideally
we would like a confluent permutation that requires a minimal number of swaps to
transform q to q ′ . Ng [448, ] notes that finding such a permutation is an NP-
complete problem. He proves that the minimum number of swaps required to obtain
2
a given confluent permutation is bounded above by n2 (1 − k1 ), where k is the number
of distinct qi , and that this bound is attainable [448, , Thm. A.1]. In practice,
since the QR algorithm tends to order the eigenvalues by absolute value in the Schur
form, complicated strategies for determining a confluent permutation are not needed.
The following method works well in practice: find the average index of the integers
in q and then order the integers in q ′ by ascending average index. If we take our
example (1, 2, 1, 3, 2, 1) and let gk denote the average index of the integer k, we see
that g1 = (1 + 3 + 6)/3 = 3 31 , g2 = (2 + 5)/2 = 3 12 , and g3 = 4. Therefore we try
to obtain the confluent permutation q ′ = (1, 1, 1, 2, 2, 3) by a sequence of swaps of
228 Schur–Parlett Algorithm

adjacent elements:

q = (1, 2, 1, 3, 2, 1) → (1, 1, 2, 3, 2, 1) → (1, 1, 2, 3, 1, 2)

→ (1, 1, 2, 1, 3, 2) → (1, 1, 1, 2, 3, 2) → (1, 1, 1, 2, 2, 3) = q ′ . (9.12)

Swapping two adjacent diagonal elements of T requires 20n flops, plus another 20n
flops to update the Schur vectors, so the cost of the swapping is 40n times the num-
ber of swaps. The total cost is usually small compared with the overall cost of the
algorithm.

9.4. Schur–Parlett Algorithm for f (A)

The complete algorithm for computing f (A) is as follows.

Algorithm 9.6 (Schur–Parlett algorithm). Given A ∈ Cn×n , a function f analytic

on a closed convex set Ω whose interior contains the eigenvalues of A, and the ability
to evaluate derivatives of f , this algorithm computes F = f (A).

1 Compute the Schur decomposition A = QT Q∗ (Q unitary, T upper triangular).

2 If T is diagonal, F = f (T ), goto line 12, end
3 Using Algorithm 9.5 with blocking parameter δ = 0.1,
assign each eigenvalue λi to a set Sqi .
4 Choose a confluent permutation q ′ of q ordered by average index.
5 Reorder T according to q ′ and update Q.
% Now A = QT Q∗ is our reordered Schur decomposition, with block m × m T .
6 for i = 1: m
7 Use Algorithm 9.4 to evaluate Fii = f (Tii ).
8 for j = i − 1: −1: 1
9 Solve the Sylvester equation (9.7) for Fij .
10 end
11 end
12 F = QF Q∗

The cost of Algorithm 9.6 depends greatly on the eigenvalue distribution of A, and is
roughly between 28n3 flops and n4 /3 flops. Note that Q, and hence F , can be kept
in factored form, with a significant computational saving. This is appropriate if F
needs just to be applied to a few vectors, for example.
We have set the blocking parameter δ = 0.1, which our experiments indicate is as
good a default choice as any. The optimal choice of δ in terms of cost or accuracy is
problem-dependent.
Algorithm 9.6 has a property noted as being desirable by Parlett and Ng [462,
]: it acts simply on simple cases. Specifically, if A is normal, so that the Schur
decomposition is A = QDQ∗ with D diagonal, the algorithm simply evaluates f (A) =
Qf (D)Q∗ . At another extreme, if A has just one eigenvalue of multiplicity n, then
the algorithm works with a single block, T11 ≡ T , and evaluates f (T11 ) via its Taylor
series expanded about the eigenvalue.
P Another attraction of Algorithm 9.6 is that it allows a function of the form f (A) =
i fi (A) (e.g., f (A) = sin A + cos A) to be computed with less work than is required
to compute each fi (A) separately, since the Schur decomposition and its reordering
need only be computed once.
9.4 Schur–Parlett Algorithm for f (A) 229

Reordering the Schur form is a nontrivial subject, the state of the art of which
is described by Bai and Demmel [28, ]. The algorithm described therein uses
unitary similarities and effects a sequence of swaps of adjacent blocks as in (9.12).
The algorithm has guaranteed backward stability and, for swapping only 1 × 1 blocks
as we are here, always succeeds. LAPACK routine xTREXC implements this algorithm
and is called by the higher level Schur reordering routine xTRSEN [12, ]; the
MATLAB function ordschur calls xTRSEN.
Algorithm 9.6 was developed by Davies and Higham [135, ] and is imple-
mented by the MATLAB function funm.
For real matrices, it might seem that by using the real Schur decomposition in the
first step of Algorithm 9.6 it would be possible to work entirely in real arithmetic.
However, the algorithm’s strategy of placing eigenvalues that are not close in different
blocks requires splitting complex conjugate pairs of eigenvalues having large imaginary
parts, forcing complex arithmetic, so the algorithm does not in general lend itself to
exploitation of the real Schur form. However, if A is real and normal then the real
Schur decomposition is block diagonal, no reordering is necessary, and Algorithm 9.6
can be reduced to computation of the Schur form and evaluation of f on the diagonal
blocks. For the 2 × 2 diagonal blocks Problem 9.2 provides appropriate formulae.
We focus for a moment on some negative aspects of the algorithm revealed by
numerical experiments in [135, ]. The algorithm can be unstable, in the sense
that the normwise relative error can greatly exceed condrel (f, A)u. Changing the
blocking parameter δ (say from 0.1 to 0.2) may produce a different blocking that
cures the instability. However, instability can be present for all choices of δ. Moreover,
instability can be present for all nontrivial blockings (i.e., any blocking with more than
one block)—some of which it might not be possible to generate by an appropriate
choice of δ in the algorithm. The latter point indicates a fundamental weakness of
the Parlett recurrence.
To illustrate the typical behaviour of Algorithm 9.6 we present a numerical exper-
iment with f the exponential function. We took 71 test matrices, which include some
from MATLAB (in particular, from the gallery function), some from the Matrix
Computation Toolbox [264], and test matrices from the eA literature; most matrices
are 10 × 10, with a few having smaller dimension. We evaluated the normwise relative
errors of the computed matrices from a modified version funm mod of MATLAB 7.6’s
funm (when invoked as funm(A,@exp) the modified version uses Algorithm 9.4 for
the diagonal blocks), where the “exact” eA is obtained at 100 digit precision using
MATLAB’s Symbolic Math Toolbox.
Figure 9.1 displays the relative errors together with a solid line representing
condrel (exp, A)u, where condrel (exp, A) is computed using Algorithm 3.17 with Al-
gorithm 10.27, and the results are sorted by decreasing condition number; the norm
is the Frobenius norm. For funm to perform in a forward stable manner its error
should lie not too far above this line on the graph; note that we must accept some
dependence of the error on n.
The errors are mostly very satisfactory, but with two exceptions. The first ex-
ception is the MATLAB matrix gallery(’chebspec’,10). This matrix is similar
to a Jordan block of size 10 with eigenvalue 0 (and hence is nilpotent), modulo the
rounding errors in its construction. The computed eigenvalues lie roughly on a circle
with centre 0 and radius 0.2; this is the most difficult distribution for Algorithm 9.6
to handle. With the default δ = 0.1, funm mod chooses a blocking with eight 1 × 1
blocks and one 2 × 2 block. This leads to an error ≈ 10−7 , which greatly exceeds
230 Schur–Parlett Algorithm

−2
10

−4
10

−6
10

−8
10

−10
10

−12
10

−14
10

−16
10

−18
10
0 10 20 30 40 50 60 70

Figure 9.1. Normwise relative errors for funm mod (∗) and condrel (exp, A)u (solid line).

condrel (exp, A)u ≈ 10−13 . However, increasing δ to 0.2 produces just one 10 × 10
block, which after using 20 terms of the Taylor series leads to an error ≈ 10−13 . The
other exceptional matrix is gallery(’forsythe’,10), which is a Jordan block of
size 10 with eigenvalue 0 except for a (10,1) entry of u1/2 . The computed eigenvalue
distribution and the behaviour for δ = 0.1 and δ = 0.2 are similar to that for the
chebspec matrix.
The main properties of Algorithm 9.6 can be summarized as follows.
1. The algorithm requires O(n3 ) flops unless close or repeated eigenvalues force a
large block Tii to be chosen, in which case the operation count can be up to
n4 /3 flops.
2. The algorithm needs to evaluate derivatives of the function when there are
blocks of dimension greater than 1.
3. In practice the algorithm usually performs in a forward stable manner. However,
the error can be greater than the condition of the problem warrants; if this
behaviour is detected then a reasonable strategy is to recompute the blocking
with a larger δ.
Points 1 and 2 are the prices to be paid for catering for general functions and
nonnormal matrices with possibly repeated eigenvalues.

9.5. Preprocessing
In an attempt to improve the accuracy of Algorithm 9.6 we might try to preprocess
the data before applying a particular stage of the algorithm, using one or more of the
techniques discussed in Section 4.10.
Translation has no effect on our algorithm. Algorithm 9.4 for evaluating the Taylor
series already translates the diagonal blocks, and further translations before applying
9.6 Notes and References 231

the Parlett recurrence are easily seen to have no effect, because (9.7) is invariant
under translations T → T − αI and F → F − βI.
A diagonal similarity transformation could be applied at any stage of the algo-
rithm and then undone later. For example, such a transformation could be used in
conjunction with Parlett’s recurrence in order to make U := D−1 T D less nonnormal
than T and to increase the separations between diagonal blocks. In fact, by choosing
D of the form D = diag(θn−1 , . . . , 1) we can make U arbitrarily close to diagonal form.
Unfortunately, no practical benefit is gained: Parlett’s recurrence involves solving tri-
angular systems and the substitution algorithm is invariant under diagonal scalings
(at least, as long as they involve only powers of the machine base). Similar comments
apply to the evaluation of the Taylor series in Algorithm 9.4.
It may be beneficial to apply balancing at the outset, prior to computing the Schur
decomposition, particularly when we are dealing with badly scaled matrices.

9.6. Notes and References

This chapter is based on Davies and Higham [135, ].
The use of the Taylor series expansion (9.3) was suggested by Stewart [438, ,
Method 18] for the matrix exponential and investigated for general f by Kågström
[323, ].
For more on the separation of two matrices see Golub and Van Loan [224, ,
Sec. 7.2.4], Stewart [536, ], and Varah [599, ].
Parlett’s recurrence was used by Kågström in his thesis [323, ]. There are three
main differences between Kågström’s approach and that of this chapter. First, he uses
an initial block diagonalization, carried out with the method of Kågström and Ruhe
[325, ], whereas we compute a Schur decomposition and reorder the triangular
form. Second, Kågström uses the scalar rather than the block form of the Parlett
recurrence and when tii and tjj are sufficiently close he uses the explicit formula for
fij in Theorem 4.11 with derivatives substituted for the divided differences. Finally,
we use a combination of Taylor series and the Parlett recurrence, whereas Kågström
investigates the separate use of these two tools upon his block diagonal form.

Problems
9.1. Prove Theorem 9.2.
9.2. Let A ∈ R2×2 have distinct complex conjugate eigenvalues λ = θ + iµ and λ and
let f (λ) = α + iβ. Show that f (A) = αI + βµ−1 (A − θI). If A is normal how does
this formula simplify?
Chapter 10
Matrix Exponential

The matrix exponential is by far the most studied matrix function. The interest in
it stems from its key role in the solution of differential equations, as explained in
Chapter 2. Depending on the application, the problem may be to compute eA for a
given A, to compute eAt for a fixed A and many t, or to apply eA or eAt to a vector
(cf. (2.3)); the precise task affects the choice of method.
Many methods have been proposed for computing eA , typically based on one of
the formulae summarized in Table 10.1. Most of them are of little practical interest
when numerical stability, computational cost, and range of applicability are taken
into account. In this chapter we make no attempt to survey the range of existing
methods but instead restrict to a few of proven practical value.
A broad assortment of methods is skilfully classified and analyzed in the classic
“Nineteen dubious ways” paper of Moler and Van Loan [437, ], reprinted with an
update in [438, ]. The conclusion of the paper is that there are three candidates
for best method. One of these—employing methods for the numerical solution of
ODEs—is outside the scope of this book, and in fact the converse approach of using
the matrix exponential in the solution of differential equations has received increasing
attention in recent years (see Section 2.1.1). The others—the scaling and squaring
method and the use of the Schur form—are treated here in some detail. The scaling
and squaring method has become the most widely used, not least because it is the
method implemented in MATLAB.
This chapter begins with a summary of some basic properties, followed by results
on the Fréchet derivative and the conditioning of the eA problem. The scaling and
squaring method based on underlying Padé approximation is then described in some
detail. Three approaches based on the Schur decomposition are outlined. The be-
haviour of the scaling and squaring method and two versions of the Schur method of
the previous chapter is illustrated on a variety of test problems. Several approaches
for approximating the Fréchet derivative and estimating its norm are explained. A
final section treats miscellaneous topics: best rational L∞ approximation, essentially
nonnegative matrices, preprocessing, and the ψ functions that we first saw in Sec-
tion 2.1.

10.1. Basic Properties

The matrix exponential is defined for A ∈ Cn×n by
A2 A3
eA = I + A + + + ···. (10.1)
2! 3!
We know from Theorem 4.7 that the series has infinite radius of convergence. Hence
d At
we can differentiate term by term to obtain dt e = AeAt = eAt A.

233
234 Matrix Exponential

Table 10.1. Some formulae for eA .

Power series Limit Scaling and squaring

A2 A3 s s s
I +A+ + + ··· lim (I + A/s) (eA/2 )2
2! 3! s→∞

Cauchy integral Jordan form Interpolation

Z n
X i−1
Y
1
ez (zI − A)−1 dz Z diag(eJk )Z −1 f [λ1 , . . . , λi ] (A − λj I)
2πi Γ i=1 j=1

Differential system Schur form Padé approximation

′ T ∗
Y (t) = AY (t), Y (0) = I Q diag(e )Q pkm (A)qkm (A)−1

Another representation is

eA = lim (I + A/s)s . (10.2)

s→∞

This formula is the limit of the first order Taylor expansion of A/s raised to the power
s ∈ Z. More generally, we can take the limit as r → ∞ or s → ∞ of r terms of the
Taylor expansion of A/s raised to the power s, thereby generalizing both (10.1) and
(10.2). The next result shows that this general formula yields eA and it also provides
an error bound for finite r and s.

Theorem 10.1 (Suzuki). For A ∈ Cn×n , let

"
r  i #s
X 1 A
Tr,s = . (10.3)
i=0
i! s

Then for any consistent matrix norm

kAkr+1 kAk
keA − Tr,s k ≤ e (10.4)
sr (r + 1)!

and limr→∞ Tr,s (A) = lims→∞ Tr,s (A) = eA .

Pr
Proof. Let T = i=0 i!1 (A/s)i and B = eA/s . Then, since B and T commute,

eA − Tr,s = B s − T s = (B − T )(B s−1 + B s−2 T + · · · + T s−1 ).

Hence
keA − Tr,s k ≤ kB − T k s max kBki kT ks−i−1 .
i=0:s−1
Pr 1 i
Now kT k ≤ i=0 i! (kAk/s) ≤ ekAk/s , and kBk satisfies the same bound, so
s−1
keA − Tr,s k ≤ s keA/s − T k e s kAk .
10.1 Basic Properties 235

But, by Theorem 4.8,

 r+1
1 kAk
keA/s − T k ≤ ekAk/s .
(r + 1)! s

This yields (10.4) and the limits are immediate.

Although eA+B 6= eA eB in general, the equality holds when A and B commute,
as the following well-known result shows.

Theorem 10.2. For A, B ∈ Cn×n , e(A+B)t = eAt eBt for all t if and only if AB =
BA.

Proof. If AB = BA then all the terms in the power series expansions of e(A+B)t
and eAt eBt commute and so these matrices are equal for the same reasons as in the
scalar case. If e(A+B)t = eAt eBt for all t then equating coefficients of t2 in the power
series expansions of both sides yields (AB + BA)/2 = AB or AB = BA.
The commutativity
 0  of A  and B is not necessary for eA+B = eA eB to hold, as the
example A = 00 2πi , B = 00 2πi
1
shows (eA+B = eA = eB = I). But if A and B have
algebraic entries then their commutativity is necessary for eA+B = eA eB = eB eA to
hold. An algebraic number is defined by the property that it is a root of a polynomial
with rational (or equivalently, integer) coefficients.

Theorem 10.3 (Wermuth). Let A ∈ Cn×n and B ∈ Cn×n have algebraic elements
and let n ≥ 2. Then eA eB = eB eA if and only if AB = BA.

Proof. The “if” part is trivial. For the “only if”, note that Lindemann’s theorem
on the transcendence of π implies that no two eigenvalues of A differ by a nonzero
integer multiple of 2πi, since the eigenvalues of A, being the roots of a polynomial
with algebraic coefficients, are themselves algebraic. Hence A is a primary logarithm
of eA , since the nonprimary logarithms (if any) are characterized by two copies of
a repeated eigenvalue being mapped to different logarithms, which must differ by a
nonzero integer multiple of 2πi (see Theorem 1.28). Thus A is a polynomial in eA ,
and likewise B is a polynomial in eB . Since eA and eB commute, A and B commute.

The Lie bracket, or commutator , is defined by [A, B] = AB −BA. It can be shown

[296, , Prob. 6.5.35] that if A and B commute with [A, B] then
1 1 1
eA eB = eA+B+ 2 [A,B] = eA+B e 2 [A,B] = e 2 [A,B] eA+B .

This relation can be seen as the first two terms in the following general result con-
necting etA etB and et(A+B) .

Theorem 10.4 (Baker–Campbell–Hausdorff formula). For A, B ∈ Cn×n we have

 t2 t3 
etA etB = exp t(A + B) + [A, B] + ([A, [A, B]] − [B, [A, B]]) + O(t4 ) , (10.5)
2 12
where the O(t4 ) term is expressible as an infinite series of iterated commutators of A
and B.
236 Matrix Exponential

Note that the third order term in (10.5) can be written in other ways, such as
(t3 /12)([[A, B], B]+[[B, A], A]). A kind of dual to (10.5) is an infinite product formula
of Zassenhaus.

Theorem 10.5 (Zassenhaus formula). For A, B ∈ Cn×n we have

2 3
et(A+B) = etA etB eC2 t eC3 t . . . , (10.6a)
C2 = −[A, B]/2, C3 = [B, [A, B]]/3 + [A, [A, B]]/6, .... (10.6b)

P∞Yet another variant of relations between etA etB and et(A+B) is et(A+B) = etA etB +
i
i=2 Ei t , for which Richmond [489, ] gives recurrences for the Ei . In a different
vein, the Strang splitting breaks etB into its square root factors and thereby provides
a second order accurate approximation: et(A+B) = etB/2 etA etB/2 + O(t3 ).
Another way to relate the exponential of a sum to a related product of exponentials
is via the limit in the next result.

Theorem 10.6 (Suzuki). For A1 , . . . , Ap ∈ Cn×n and any consistent matrix norm,
 p 2  p 
A1 +···+Ap A1 /m


Ap /m m 2 X m+2 X
ke − e ...e k≤ kAj k exp kAj k . (10.7)
m j=1 m j=1

Hence
m
eA1 +···+Ap = lim eA1 /m . . . eAp /m . (10.8)
m→∞

Proof. Let G = e(A1 +···+Ap )/m and H = eA1 /m . . . eAp /m . Then we need to bound
kG − H m k. Using Lemma B.4 and Theorem 10.10 we have
m

kGm − H m k ≤ kG − Hk kGkm−1 + kGkm−2 kHk + · · · + kHkm−1
m−1 Pp
kAj k
≤ m kG − Hk e m j=1 .

Now
 P  p 
−1 2 p
kAj k 2 X
kG − Hk ≤ kHk kGH − Ik ≤ kHk e m j=1 − 1+ kAj k
m j=1
 p 2
1 2 X 2
Pp
≤ kHk kAj k e m j=1 kAj k ,
2 m j=1
2
Pp
where for the second inequality we used kGH −1 k ≤ kGkkH −1 k ≤ e m j=1 kAj k and
the fact that the first two terms of the expansion of GH −1 − I are zero and for the
third inequality we used a Taylor series with remainder (Theorem 4.8). Hence
Pp
Xp 2 P
m−1 Pp
m m 1
kAj k 2 2 p
j=1 kAj k e m j=1 kAj k ,
kG − H k ≤ m e m j=1 kA j k e m
m2 j=1

as required.

Corollary 10.7 (Trotter product formula). For A, B ∈ Cn×n ,

m
eA+B = lim eA/m eB/m . (10.9)
m→∞
10.1 Basic Properties 237

The following result provides a better error bound for p = 2 than (10.7) in the
sense that the bound is small when the commutator is small.

Theorem 10.8. For A, B ∈ Cn×n ,

m k[A, B]k2 kAk2 +kBk2
keA+B − eA/m eB/m k2 ≤ e .
2m
Proof. See Moler and Van Loan [438, , App. 2].
The commutativity requirement in Theorem 10.2 can be removed if the usual
summation and multiplication operators are replaced by their Kronecker counterparts
(see Section B.13 for the definitions and basic properties of the Kronecker product ⊗
and Kronecker sum ⊕).

Theorem 10.9. Let A ∈ Cn×n and B ∈ Cm×m . Then eA⊗I = eA ⊗I, eI⊗B = I ⊗eB ,
and eA⊕B = eA ⊗ eB .
Proof. The first two relations are easily obtained from the power series (10.1) and
also follow from Theorem 1.13 (h), (i). The proof of the second relation is analogous.
Since A⊗I and I ⊗B commute, eA⊕B = eA⊗I+I⊗B = eA⊗I eI⊗B = (eA ⊗I)(I ⊗eB ) =
eA ⊗ eB .
Bounds for the norm of the matrix exponential are of great interest. We state just
three of the many bounds available in the literature. We need the spectral abscissa,

α(A) = max{ Re λi : λi ∈ Λ(A) }, (10.10)

and the logarithmic norm [546, ]

µ(A) = λmax ( 12 (A + A∗ )). (10.11)

Theorem 10.10. For A ∈ Cn×n and any subordinate matrix norm,

e−kAk ≤ keA k ≤ ekAk .

The upper bound is attained in the ∞-norm if A is a nonnegative multiple of a stochas-

tic matrix.
Proof. The upper bound is immediate on taking norms in the power series for
eA . Since I = eA e−A we have 1 ≤ keA k ke−A k ≤ keA k ekAk , which gives the lower
bound. For the last part, write A = cP , where c ≥ 0 and P is stochastic. Then
Af = cP f = cf , where f = [1, 1, . . . , 1]T , so c = kAk∞ . Hence, since eA ≥ 0,
keA k∞ = keA f k∞ = k(I + A + A2 /2! + · · ·)f k∞ = k(1 + c + c2 /2! + · · ·)f k∞ = ec =
ekAk∞ .

Theorem 10.11. For A ∈ Cn×n and any unitarily invariant norm,

∗
keA k ≤ ke(A+A )/2
k.

Proof. See Bhatia [64, , Thm. IX.3.1].

For the 2-norm, Theorem 10.11 says that keA k2 ≤ eµ(A) , a bound due to Dahlquist
[128, ].
238 Matrix Exponential

Theorem 10.12. Let A ∈ Cn×n have the Schur decomposition A = Q(D + N )Q∗ ,
where D is diagonal and N is strictly upper triangular. Then
n−1
X kN kk2
eα(A) ≤ keA k2 ≤ eα(A) . (10.12)
k!
k=0

Proof. See Van Loan [594, ].

A Frobenius norm analogue to (10.12) can be obtained from Theorem 4.28.
The 2-norm bounds in Theorems 10.11 and 10.12 are equalities when A is normal.

10.2. Conditioning
In this section we investigate the sensitivity of the matrix exponential to perturba-
tions.
The matrix exponential satisfies the identity (see Problem 10.1)
Z t
(A+E)t At
e =e + eA(t−s) Ee(A+E)s ds. (10.13)
0
(A+E)s
Using this expression to substitute for e inside the integral yields
Z t
e(A+E)t = eAt + eA(t−s) EeAs ds + O(kEk2 ). (10.14)
0

Hence, from the definition (3.6), the Fréchet derivative of the exponential at A in the
direction E is given by

Fréchet derivative of matrix exponential:

Z 1
L(A, E) = eA(1−s) EeAs ds. (10.15)
0

This formula simplifies to L(A, E) = EeA = eA E when A commutes with E. Another

representation of the Fréchet derivative is, from the Taylor series for eA+E − eA ,
AE + EA A2 E + AEA + EA2
L(A, E) = E + + + ···,
2! 3!
but the integral formula is generally easier to work with.
The following result provides three explicit expressions for vec(L(A, E)). Define
the sinch function by 
sinh(x)/x, x 6= 0,
sinch(x) = (10.16)
1, x = 0.
Note that sinch can be written in terms of the more common sinc function—defined
by sinc(x) = sin(x)/x if x 6= 0 or 1 otherwise—as sinch(x) = sinc(ix).

Theorem 10.13 (Kronecker representation of Fréchet derivative). For A ∈ Cn×n ,

2 2
vec(L(A, E)) = K(A) vec(E), where K(A) ∈ Cn ×n has the representations



 (I ⊗ eA ) ψ1 AT ⊕ (−A) , (10.17a)



AT /2 A/2 1 T
K(A) = (e ⊗ e ) sinch 2 [A ⊕ (−A)] , (10.17b)

 1 (eAT ⊕ eA ) τ 1 [AT ⊕ (−A)]
,

 (10.17c)
2 2
10.2 Conditioning 239

where ψ1 (x) = (ex − 1)/x and τ (x) = tanh(x)/x. The third expression is valid if
1 T
2 kA ⊕ (−A)k < π/2 for some consistent matrix norm.

Proof. Applying the vec operator to (10.15) gives

Z 1
T

vec(L(A, E)) = eA s ⊗ eA(1−s) vec(E) ds
0
Z 1
T

= (I ⊗ eA ) eA s
⊗ e−As ds · vec(E)
0
Z 1
T
= (I ⊗ eA ) e[A ⊕(−A)]s
ds · vec(E) (10.18)
0

by Theorem 10.9. Now

Z 1
x x2 ex − 1
exs ds = 1 + + + ··· = = ψ1 (x) (10.19)
0 2! 3! x
 x/2 x/2

 e (e − e−x/2 ) sinh(x/2)
 = ex/2 ,
x x/2
= (10.20)

 ex/2 x/2 ex/2 − e−x/2 ex + 1 tanh(x/2)
 (e + e−x/2 ) x/2 −x/2
= .
x e +e 2 x/2
The first formula in (10.17) follows from (10.18) and (10.19). The second formula
follows from (10.20) and by using Theorem 10.9:
1 T

K(A) = (I ⊗ eA ) e 2 [A ⊕(−A)]
sinch 1
2 [A
T
⊕ (−A)]
T

= (I ⊗ eA )(eA /2
⊗ e−A/2 ) sinch 1
2 [A
T
⊕ (−A)]
T

= (eA /2
⊗ eA/2 ) sinch 1
2 [A
T
⊕ (−A)] .

Similarly, for the third formula,

T

K(A) = (I ⊗ eA ) 21 (eA ⊕(−A)
+ I) τ 1
2 [A
T
⊕ (−A)]
T

= (I ⊗ eA ) 12 (eA ⊗ e−A + I) τ ⊕ (−A)] 1
2 [A
T

T

= 12 (eA ⊗ I + I ⊗ eA ) τ 12 [AT ⊕ (−A)]
T

= 12 (eA ⊕ eA ) τ 21 [AT ⊕ (−A)] .

The norm restriction is due to tanh(x) having poles with |x| = π/2.
Each of the three expressions (10.17) is of potential interest for computational
purposes, as we will see in Section 10.6.
Note that the matrix AT ⊕ (−A) has eigenvalues λi (A) − λj (A), i, j = 1: n (see
Section B.13), so it is singular with at least n zero eigenvalues. This does not affect
the formulae (10.17) since φ, τ , sinch, and their derivatives are defined from their
power series expansions at the origin.
The relative condition number of the exponential is (see Theorem 3.1)
kL(A)k kAk
κexp (A) = .
keA k
From Theorem 10.13 we obtain explicit expressions for kL(A)kF .
240 Matrix Exponential

Corollary 10.14 (norm of Fréchet derivative). For A ∈ Cn×n ,



 (I ⊗ eA ) ψ1 AT ⊕ (−A) ,

 T 2


kL(A)kF = (eA /2 ⊗ eA/2 ) sinch 21 [AT ⊕ (−A)] 2 , (10.21)




1 (eAT /2 ⊕ eA/2 ) τ 1 [AT ⊕ (−A)] .
2 2 2

Proof. The formulae follow immediately from the theorem on using the fact that
k vec(B)k2 = kBkF .
The next lemma gives upper and lower bounds for κexp (A).

Lemma 10.15. For A ∈ Cn×n we have, for any subordinate matrix norm,
ekAk kAk
kAk ≤ κexp (A) ≤ . (10.22)
keA k
Proof. From (10.15) we have
Z 1 Z 1
kAk(1−s) kAks
kL(A, E)k ≤ kEk e e ds = kEk ekAk ds = kEkekAk ,
0 0
R1
so that kL(A)k ≤ ekAk . Also, kL(A)k ≥ kL(A, I)k = k 0
eA dsk = keA k, and the
result follows.
An important class of matrices that has minimal condition number is the normal
matrices.

Theorem 10.16 (Van Loan). If A ∈ Cn×n is normal then in the 2-norm, κexp (A) =
kAk2 .
Proof. We need a slight variation of the upper bound in (10.22). First, note that
for a normal matrix B, keB k2 = eα(B) . From (10.15) we have
Z 1
kL(A, E)k2 ≤ kEk2 eα(A)(1−s) eα(A)s ds
0
Z 1
= kEk2 eα(A) ds = eα(A) kEk2 = keA k2 kEk2 .
0

Hence kL(A)k2 ≤ keA k2 , which implies the result in view of the lower bound in
(10.22).
A second class of matrices with perfect conditioning comprises nonnegative scalar
multiples of stochastic matrices.

Theorem 10.17 (Melloy and Bennett). Let A ∈ Rn×n be a nonnegative scalar mul-
tiple of a stochastic matrix. Then in the ∞-norm, κexp (A) = kAk∞ .

Proof. From Theorem 10.10 we have keA k∞ = ekAk∞ . Hence the result follows
from (10.22) with the ∞-norm.
For a third class of perfectly conditioned matrices, see Theorem 10.30 below.
Unfortunately, no useful characterization of matrices for which κexp is large is
known; see Problem 10.15.
The next result is useful because it shows that argument reduction A ← A − µI
reduces the condition number if it reduces the norm. Since eA−µI = e−µ eA , argument
reduction is trivial to incorporate in any algorithm.
10.3 Scaling and Squaring Method 241

Theorem 10.18 (Parks). If µ is a scalar such that kA − µIk < kAk then κexp (A −
µI) < κexp (A).

Proof. For any E,

Z 1

kL(A − µI, E)k =
e(A−µI)(1−s)
Ee(A−µI)s
ds

0
Z 1
−µI
= e eA(1−s) EeAs ds
=e
−Re µ
kL(A, E)k.
0

Hence kL(A − µI)k = e−Re µ kL(A)k. Thus if kA − µIk < kAk then

kL(A − µI)k kA − µIk kL(A − µI)k kA − µIk

κexp (A − µI) = A−µI
=
ke k keA ke−Re µ
kL(A)k kAk
< = κexp (A).
keA k

10.3. Scaling and Squaring Method

The scaling and squaring method exploits the relation eA = (eA/σ )σ , for A ∈ Cn×n
and σ ∈ C, together with the fact that eA can be well approximated by a Taylor or
Padé approximant near the origin, that is, for small kAk. The idea is to choose σ
an integral power of 2, σ = 2s say, so that A/σ has norm of order 1; approximate
s
eA/2 ≈ r(A/2s ), where r is a Taylor or Padé approximant to the exponential; and
s
then take eA ≈ r(A/2s )2 , where the approximation is formed by s repeated squarings.
Padé approximants are preferred to Taylor series approximations in this context
because they provide a given accuracy with lower computational cost [438, ,
Sec. 3]. Recall from Section 4.4.2 that the [k/m] Padé approximant rkm (x) =
pkm (x)/qkm (x) to the exponential is defined by the properties that p and q are poly-
nomials of degrees at most k and m, respectively, and that ex − rkm (x) = O(xk+m+1 ).
These Padé approximants are known explicitly for all k and m:

Xk Xm
(k + m − j)! k! xj (k + m − j)! m! (−x)j
pkm (x) = , qkm (x) = . (10.23)
j=0
(k + m)! (k − j)! j! j=0
(k + m)! (m − j)! j!

Note that pkm (x) = qmk (−x), which reflects the property 1/ex = e−x of the expo-
nential function. Later we will exploit the fact that pmm (x) and qmm (x) approx-
imate ex/2 and e−x/2 , respectively, though they do so much less accurately than
rmm = pmm /qmm approximates ex . That rkm satisfies the definition of Padé approx-
imant is demonstrated by the error expression

k! m!
ex − rkm (x) = (−1)m xk+m+1 + O(xk+m+2 ). (10.24)
(k + m)!(k + m + 1)!

We also have the exact error expression, for A ∈ Cn×n [438, , App. A], [501,
],
Z 1
(−1)m k+m+1
eA − rkm (A) = A qkm (A)−1 etA (1 − t)k tm dt. (10.25)
(k + m)! 0
242 Matrix Exponential

Diagonal approximants (k = m) are preferred, since rkm with k 6= m is less accu-

rate than rjj , where j = max(k, m), but rjj can be evaluated at a matrix argument
at the same cost. Moreover, the diagonal approximants have the property that if the
eigenvalues of A lie in the open left half-plane then the eigenvalues of rmm (A) have
modulus less than 1, which is important in applications to differential equations [600,
, Chap. 8]. We will write the diagonal approximants as rm (x) = pm (x)/qm (x).
Our aim is to choose s, in the initial scaling A ← A/2s , and the degree m of
the Padé approximant, so that the exponential is computed with backward error
bounded by the unit roundoff and with minimal cost. In bounding the backward
error we assume exact arithmetic and examine solely the effects of the approximation
errors in the Padé approximant.
The choice of s will be based on kAk, where the norm can be any consistent matrix
norm. Our initial aim is therefore to bound the backward error in terms of k2−s Ak
and then to determine, for each degree m, the maximum k2−s Ak for which rm can
be guaranteed to deliver the desired backward error. Let

e−A rm (A) = I + G = eH , (10.26)

where we assume that kGk < 1, so that H = log(I + G) is guaranteed to exist. (Here,
log denotes the principal logarithm.) It is easy to show that kHk ≤ − log(1 − kGk)
(see Problem 10.8). Now G is clearly a function of A hence so is H, and therefore H
commutes with A. It follows that

rm (A) = eA eH = eA+H .

Now we replace A by A/2s , where s is a nonnegative integer, and raise both sides of
this equation to the power 2s , to obtain
s
rm (A/2s )2 = eA+E ,

where E = 2s H satisfies
kEk ≤ −2s log(1 − kGk)
and G satisfies (10.26) with A replaced by 2−s A. We summarize our findings in the
following theorem.

Theorem 10.19. Let the diagonal Padé approximant rm satisfy

−s
e−2 A
rm (2−s A) = I + G, (10.27)

where kGk < 1 and the norm is any consistent matrix norm. Then
s
rm (2−s A)2 = eA+E ,

where E commutes with A and

kEk − log(1 − kGk)
≤ . (10.28)
kAk k2−s Ak

Theorem 10.19 is a backward error result: it interprets the truncation errors in

the Padé approximant as equivalent to a perturbation in the original matrix A. (Note
that the result holds for any rational approximation rm , as we have not yet used
specific properties of a Padé approximant.) The advantage of this backward error
10.3 Scaling and Squaring Method 243

viewpoint is that it takes into account the effect of the squaring phase on the error
in the Padé approximant and, compared with a forward error bound, avoids the need
to consider the conditioning of the problem.
Our task now is to bound the norm of G in (10.27) in terms of k2−s Ak. One way
to proceed is to assume an upper bound on kAk and use the error formula (10.25) to
obtain an explicit bound on G, or at least one that is easy to evaluate. This approach,
which is used by Moler and Van Loan [438, ] and is illustrated in Problem 10.10,
is mathematically elegant but does not yield the best possible bounds and hence does
not lead to the best algorithm. We will use a bound on kGk that makes no a priori
assumption on kAk and is as sharp as possible. The tradeoff is that the bound is hard
to evaluate, but this is a minor inconvenience because the evaluation need only be
done during the design of the algorithm.
Define the function
ρ(x) = e−x rm (x) − 1.
In view of the Padé approximation property (10.24), ρ has a power series expansion
∞
X
ρ(x) = ci xi , (10.29)
i=2m+1

and this series will converge absolutely for |x| < min{ |t| : qm (t) = 0 } =: νm . Hence
∞
X
kGk = kρ(2−s A)k ≤ |ci |θi =: f (θ), (10.30)
i=2m+1

where θ := k2−s Ak < νm . It is clear that if A is a general matrix and only kAk is
known then (10.30) provides the smallest possible bound on kGk. The corresponding
bound of Moler and Van Loan [438, , Lem. 4] is easily seen to be less sharp, and
a refined analysis of Dieci and Papini [157, , Sec. 2], which bounds a different
error, is also weaker when adapted to bound kGk.
Combining (10.30) with (10.28) we have

kEk − log(1 − f (θ))

≤ . (10.31)
kAk θ

Evaluation of f (θ) in (10.30) would be easy if the coefficients ci were one-signed, for
then we would have f (θ) = |ρ(θ)|. Experimentally, the ci are one-signed for some, but
not all, m. Using MATLAB’s Symbolic Math Toolbox we evaluated f (θ), and hence
the bound (10.31), in 250 decimal digit arithmetic, summing the first 150 terms of
the series, where the ci in (10.29) are obtained symbolically. For m = 1: 21 we used a
zero-finder to determine the largest value of θ, denoted by θm , such that the backward
error bound (10.31) does not exceed u = 2−53 ≈ 1.1 × 10−16 . The results are shown
to two significant figures in Table 10.2.
The second row of the table shows the values of νm , and we see that θm < νm in
each case, confirming that the bound (10.30) is valid. The inequalities θm < νm also
confirm the important fact that qm (A) is nonsingular for kAk ≤ θm (which is in any
case implicitly enforced by our analysis).
Next we need to determine the cost of evaluating rm (A). Because of the relation
qm (x) = pm (−x) between the numerator and denominator polynomials, an efficient
scheme can be based on explicitly computing the even powers of A, forming pm and
244 Matrix Exponential

Table 10.2. Maximal values θm of k2−s Ak such that the backward error bound (10.31) does not
exceed u = 2−53 , values of νm = min{ |x| : qm (x) = 0}, and upper bound ξm for kqm (A)−1 k.

m 1 2 3 4 5 6 7 8 9 10
θm 3.7e-8 5.3e-4 1.5e-2 8.5e-2 2.5e-1 5.4e-1 9.5e-1 1.5e0 2.1e0 2.8e0
νm 2.0e0 3.5e0 4.6e0 6.0e0 7.3e0 8.7e0 9.9e0 1.1e1 1.3e1 1.4e1
ξm 1.0e0 1.0e0 1.0e0 1.0e0 1.1e0 1.3e0 1.6e0 2.1e0 3.0e0 4.3e0

m 11 12 13 14 15 16 17 18 19 20 21
θm 3.6e0 4.5e0 5.4e0 6.3e0 7.3e0 8.4e0 9.4e0 1.1e1 1.2e1 1.3e1 1.4e1
νm 1.5e1 1.7e1 1.8e1 1.9e1 2.1e1 2.2e1 2.3e1 2.5e1 2.6e1 2.7e1 2.8e1
ξm 6.6e0 1.0e1 1.7e1 3.0e1 5.3e1 9.8e1 1.9e2 3.8e2 8.3e2 2.0e3 6.2e3

Pm i
qm , and then solving the matrix equation qm rm = pm . If pm (x) = i=0 bi x , we
have, for the even degree case,

p2m (A) = b2m A2m + · · · + b2 A2 + b0 I + A(b2m−1 A2m−2 + · · · + b3 A2 + b1 I)

=: U + V, (10.32)

which can be evaluated with m + 1 matrix multiplications by forming A2 , A4 , . . . ,

A2m . Then
q2m (A) = U − V
is available at no extra cost. For odd degrees,

p2m+1 (A) = A(b2m+1 A2m + · · · + b3 A2 + b1 I) + b2m A2m + · · · + b2 A2 + b0 I

=: U + V, (10.33)

so p2m+1 and q2m+1 = −U + V can be evaluated at exactly the same cost as p2m and
q2m . However, for m ≥ 12 this scheme can be improved upon. For example, we can
write

p12 (A) = A6 (b12 A6 + b10 A4 + b8 A2 + b6 I) + b4 A4 + b2 A2 + b0 I

 
+ A A6 (b11 A4 + b9 A2 + b7 I) + b5 A4 + b3 A2 + b1 I
=: U + V, (10.34)

and q12 (A) = U − V . Thus p12 and q12 can be evaluated in just six matrix mul-
tiplications (for A2 , A4 , A6 , and three additional multiplications). For m = 13 an
analogous formula holds, with the outer multiplication by A transferred to the U
term. Similar formulae hold for m ≥ 14. Table 10.3 summarizes the number of
matrix multiplications required to evaluate pm and qm , which we denote by πm , for
m = 1: 21.
The information in Tables 10.2 and 10.3 enables us to determine the optimal
algorithm when kAk ≥ θ21 . From Table 10.3, we see that the choice is between
m = 1, 2, 3, 5, 7, 9, 13, 17 and 21 (there is no reason to use m = 6, for example, since
the cost of evaluating the more accurate q7 is the same as the cost of evaluating
q6 ). Increasing from one of these values of m to the next requires an extra matrix
multiplication to evaluate rm , but this is offset by the larger allowed θm = k2−s Ak if
θm jumps by more than a factor 2, since decreasing s by 1 saves one multiplication in
10.3 Scaling and Squaring Method 245

Table 10.3. Number of matrix multiplications, πm , required to evaluate pm (A) and qm (A),
and measure of overall cost Cm in (10.35).

m 1 2 3 4 5 6 7 8 9 10
πm 0 1 2 3 3 4 4 5 5 6
Cm 25 12 8.1 6.6 5.0 4.9 4.1 4.4 3.9 4.5

m 11 12 13 14 15 16 17 18 19 20 21
πm 6 6 6 7 7 7 7 8 8 8 8
Cm 4.2 3.8 3.6 4.3 4.1 3.9 3.8 4.6 4.5 4.3 4.2

the final squaring stage. Table 10.2 therefore shows that m = 13 is the best choice.
Another way to arrive at this conclusion is to observe that the cost of the algorithm in
matrix multiplications is, since s = ⌈log2 kAk/θm ⌉ if kAk ≥ θm and s = 0 otherwise,

πm + s = πm + max (⌈log2 kAk − log2 θm ⌉ , 0) .

(We ignore the required matrix equation solution, which is common to all m.) We
wish to determine which m minimizes this quantity. For kAk ≥ θm we can remove the
max and ignore the kAk term, which is essentially a constant shift, so we minimize

Cm = πm − log2 θm . (10.35)

The Cm values are shown in the second line of Table 10.3. Again, m = 13 is optimal.
We repeated the computations with u = 2−24 ≈ 6.0 × 10−8 , which is the unit roundoff
in IEEE single precision arithmetic, and u = 2−105 ≈ 2.5 × 10−32 , which corresponds
to quadruple precision arithmetic; the optimal m are now m = 7 (θ7 = 3.9) and
m = 17 (θ17 = 3.3), respectively.
Now we consider the effects of rounding errors on the evaluation of rm (A). We
immediately rule out m = 1 and m = 2 because r1 and r2 can suffer from loss of
significance in floating point arithmetic. For example, r1 requires kAk to be of order
10−8 after scaling, and then the expression r1 (A) = (I + A/2)(I − A/2)−1 loses about
half the significant digits in A in double precision arithmetic; yet if the original A has
norm of order at least 1 then all the significant digits of some of the elements of A
should contribute to the result. Applying Theorem 4.5 to pm (A), where kAk1 ≤ θm ,
and noting that pm has all positive coefficients, we deduce that

kpm (A) − pbm (A)k1 ≤ γ

emn pm (kAk1 )
≈γemn ekAk1 /2
emn keA/2 k1 ekAk1
≤γ
emn kpm (A)k1 ekAk1 ≤ γ
≈γ emn kpm (A)k1 eθm .

Hence the relative error is bounded approximately by γemn eθm , which is a satisfactory
bound given the values of θm in Table 10.2. Replacing A by −A in the latter bound
we obtain
kqm (A) − qbm (A)k1 <
∼γemn kqm (A)k1 eθm .
In summary, the errors in the evaluation of pm and qm are nicely bounded.
To obtain rm we solve a multiple right-hand side linear system with qm (A) as
coefficient matrix, so to be sure that this system is solved accurately we need to check
246 Matrix Exponential

Pm
Table 10.4. Coefficients b(0: m) in numerator pm (x) = i=0 bi xi of Padé approximant rm (x)
to ex , normalized so that b(m) = 1.

m b(0: m)
3 [120, 60, 12, 1]
5 [30240, 15120, 3360, 420, 30, 1]
7 [17297280, 8648640, 1995840, 277200, 25200, 1512, 56, 1]
9 [17643225600, 8821612800, 2075673600, 302702400, 30270240,
2162160, 110880, 3960, 90, 1]
13 [64764752532480000, 32382376266240000, 7771770303897600,
1187353796428800, 129060195264000, 10559470521600,
670442572800, 33522128640, 1323241920,
40840800, 960960, 16380, 182, 1]

that qm (A) is well conditioned. It is possible to obtain a priori bounds for kqm (A)−1 k
under assumptions such as (for any subordinate matrix norm) kAk ≤ 1/2 [438, ,
Lem. 2], kAk ≤ 1 [606, , Thm. 1], or qm (−kAk) < 2 [157, , Lem. 2.1] (see
Problem 10.9), but these assumptions are not satisfied for all the m and kAk of interest
to us. Therefore we take a similar approach to the way we derived the constants θm .
With kAk ≤ θm and by writing


qm (A) = e−A/2 I + eA/2 qm (A) − I) =: e−A/2 (I + F ),

we have, if kF k < 1,

eθm /2
kqm (A)−1 k ≤ keA/2 k k(I + F )−1 k ≤ .
1 − kF k
P∞ P∞
We can expand ex/2 qm (x) − 1 = i=2 di xi , from which kF k ≤ i=2 |di |θm
i
follows.
Our overall bound is
eθm /2
kqm (A)−1 k ≤ P∞ i
.
1 − i=2 |di |θm
By determining the di symbolically and summing the first 150 terms of the sum in
250 decimal digit arithmetic, we obtained the bounds in the last row of Table 10.2,
which confirm that qm is very well conditioned for m up to about 13 when kAk ≤ θm .
The overall algorithm is as follows. It first checks whether kAk ≤ θm for m ∈
{3, 5, 7, 9, 13} and, if so, evaluates rm for the smallest such m. Otherwise it uses the
scaling and squaring method with m = 13.

Algorithm 10.20 (scaling and squaring algorithm). This algorithm evaluates the ma-
trix exponential X = eA of A ∈ Cn×n using the scaling and squaring method. It uses
the constants θm given in Table 10.2 and the Padé coefficients in Table 10.4. The
algorithm is intended for IEEE double precision arithmetic.

1 for m = [3 5 7 9]
2 if kAk1 ≤ θm
% Form rm (A) = [m/m] Padé approximant to A.
3 Evaluate U and V using (10.33) and solve (−U + V )X = U + V .
4 quit
10.3 Scaling and Squaring Method 247

5 end
6 end
7 A ← A/2s with s ≥ 0 a minimal integer such that kA/2s k1 ≤ θ13
(i.e., s = ⌈log2 (kAk1 /θ13 )⌉).
8 % Form [13/13] Padé approximant to eA .
9 A2 = A2 , A4 = A22 , A6 = A2 A4 
10 U = A A6 (b13 A6 + b11 A4 + b9 A2 ) + b7 A6 + b5 A4 + b3 A2 + b1 I
11 V = A6 (b12 A6 + b10 A4 + b8 A2 ) + b6 A6 + b4 A4 + b2 A2 + b0 I
12 Solve (−U + V )r13 = U + V for r13 .
s
13 X = r13 2 by repeated squaring.


Cost: πm +⌈log2 (kAk1 /θm )⌉ M +D, where m is the degree of Padé approximant
used and πm is tabulated in Table 10.3. (M and D are defined at the start of
Chapter 4.)
2k 2k+1
It is readily checked that the sequences θ13 b2k and θ13 b2k+1 are approximately
monotonically decreasing with k, and hence the ordering given in Algorithm 10.20 for
evaluating U and V takes the terms in approximately increasing order of norm. This
ordering is certainly preferable when A has nonnegative elements, and since there
cannot be much cancellation in the sums it cannot be a bad ordering [276, ,
Chap. 4].
The part of the algorithm most sensitive to rounding errors is the final scaling
phase. The following general result, in which we are thinking of B as the Padé
approximant, shows why.

Theorem 10.21. For B ∈ Rn×n let X b = f l(B 2k ) be computed by repeated squaring.

Then, for the 1-, ∞-, and Frobenius norms,
k
b ≤ (2k − 1)nukBk2 · kB 2 k kB 4 k . . . kB 2k−1 k + O(u2 ).
kB 2 − Xk (10.36)

Proof. See Problem 10.11.

k
If B 2 were computed by repeated multiplication, the upper bound in (10.36)
k
would be 2k nukBk2 + O(u2 ) [276, , Lem. 3.6]. This bound is much weaker
than (10.36), so repeated squaring can be expected to be more accurate as well as
more efficient than repeated multiplication. To see that (10.36) can nevertheless be
unsatisfactory, we rewrite it as the relative error bound
k
kB 2 − Xk b
2 k ≤ µ(2k − 1)nu + O(u2 ), (10.37)
kB k

where
k−1
kBk2 kB 2 k kB 4 k . . . kB 2 k
µ= ≥ 1. (10.38)
kB 2k k
The ratio µ can be arbitrarily large, because cancellation can cause an intermediate
j k
power B 2 (j < k) to be much larger than the final power B 2 . In other words, the
powers can display the hump phenomenon, illustrated in Figure 10.1 for the matrix
 
−0.97 25
A= . (10.39)
0 −0.3
248 Matrix Exponential

We see that while the powers ultimately decay to zero (since ρ(A) = 0.97 < 1),
initially they increase in norm, producing a hump in the plot. The hump can be
arbitrarily high relative to the starting point kAk. Moreover, there may not be a
single hump, and indeed scalloping behaviour is observed for some matrices. Analysis
of the hump for 2 × 2 matrices, and various bounds on matrix powers, can be found
in [276, , Chap. 18] and the references therein.
Another way of viewing this discussion is through the curve keAt k, shown for the
matrix (10.39) in Figure 10.2. This curve, too, can be hump-shaped. Recall that we
are using the relation eA = (eA/σ )σ , and if t = 1/σ falls under a hump but t = 1 is
beyond it, then keA k ≪ keA/σ kσ . The connection between powers and exponentials
is that if we set B = eA/r then the values kB j k are the points on the curve keAt k for
t = 1/r, 2/r, . . ..
The bound (10.37) also contains a factor 2k − 1. If kAk > θ13 then 2k ≈ kAk and
so the overall relative error bound contains a term of the form µkAknu. However,
since κexp (A) ≥ kAk (see Lemma 10.15), a factor kAk is not troubling.
Our conclusion is that the overall effect of rounding errors in the final squaring
stage may be large relative to the computed exponential X, b and so Xb may have large
relative error. This may or may not indicate instability of the algorithm, depending
on the conditioning of the eA problem at the matrix A. Since little is known about
the size of the condition number κexp for nonnormal A, no clear general conclusions
can be drawn about the stability of the algorithm (see Problem 10.16).
In the special case where A is normal the scaling and squaring method is guaran-
teed to be forward stable. For normal matrices there is no hump because kAk k2 =
kAkk2 , so µ = 1 in (10.38), and 2k ≈ kAk2 = κexp (A) by Theorem 10.16. Therefore the
squaring phase is innocuous and the error in the computed exponential is consistent
with the conditioning of the problem. Another case in which the scaling and squaring
method is forward stable is when aij ≥ 0 for i 6= j, as shown by Arioli, Codenotti, and
Fassino [17, ]. The reason is that the exponential of such a matrix is nonnegative
(see Section 10.7.2) and multiplying nonnegative matrices is a stable procedure since
there can be no cancellation.
Finally, we note that the scaling and squaring
 method
 has a weakness when applied
to block triangular matrices. Suppose A = A11 A12
0 A22 . Then
 Z 1 
e
A11
eA11 (1−s) A12 eA22 s ds 
eA = 0 (10.40)
0 eA22

(see Problem 10.12). The linear dependence of the (1,2) block of eA on A12 sug-
gests that the accuracy of the corresponding block of a Padé approximant should
not unduly be affected by kA12 k and hence that in the scaling and squaring method
only the norms of A11 and A22 should influence the amount of scaling (specified by
s in Algorithm 10.20). But since s depends on the norm of A as a whole, when
kA12 k ≫ max(kA11 k, kA22 k) the diagonal blocks are overscaled with regard to the
computation of eA11 and eA22 , and this may have a harmful effect on the accuracy
of the computed exponential (cf. the discussion on page 245 about rejecting degrees
m = 1, 2). In fact, the block triangular case merits special treatment. If the spectra of
A11 and A22 are well separated then it is best to compute eA11 and eA22 individually
and obtain F12 from the block Parlett recurrence by solving a Sylvester equation of
the form (9.7). In general, analysis of Dieci and Papini [157, ] suggests that if
the scaling and squaring method is used with s determined so that 2−s kA11 k and
10.3 Scaling and Squaring Method 249

34

32

30

28
||Ak||2

26

24

22

20
0 5 10 15 20
k

Figure 10.1. 2-norms of first 20 powers of A in (10.39).

16

14

12

10
||eAt||2
8

0
0 2 4 6 8 10
t

Figure 10.2. 2-norm of exp(At) for A in (10.39).

250 Matrix Exponential

2−s kA22 k are appropriately bounded, without consideration of kA12 k, then an accu-
rate approximation to eA will still be obtained.

10.4. Schur Algorithms

Another general purpose way to compute eA is to employ a Schur decomposition
A = QT Q∗ , thereby reducing the problem to that of computing the exponential of a
triangular matrix. In this section we describe the three main contenders for how to
compute eT .

10.4.1. Newton Divided Difference Interpolation

Parlett and Ng [462, ] develop an algorithm based on the Schur decomposition
and the Parlett recurrence that is tailored to the exponential in both its blocking (a
two-level scheme) and how the diagonal blocks are exponentiated. We will concentrate
on the latter aspect.
To exponentiate an atomic block Tii ∈ Cm×m , Parlett and Ng use the Newton
divided difference form of the Hermite interpolating polynomial
m
X i−1
Y
p(t) = ci (t − λj ), (10.41)
i=1 j=1

where ci = f [λ1 , λ2 , . . . , λi ] with λi ≡ tii and f (t) = et . The cost of evaluating p(Tii )
is O(m4 ) flops.
For general functions f (or a set of given function values) divided differences are
computed using the standard recurrence (B.24). However, the recurrence can produce
inaccurate results in floating point arithmetic [276, , Sec. 5.3]. This can be seen
from the first order divided difference f [λk , λk+1 ] = (f (λk+1 ) − f (λk ))/(λk+1 − λk )
(λk 6= λk+1 ), in which for λk close to λk+1 the subtraction in the numerator will
suffer cancellation and the resulting error will be magnified by a small denominator.
Obtaining accurate divided differences is important because the matrix product terms
in (10.41) can vary greatly in norm. For a particular f that is given in functional
form rather than simply as function values f (λi ), we would hope to be able to obtain
the divided differences more accurately by exploiting properties of f . The next result
offers one way to do this because it shows that evaluating f at a certain bidiagonal
matrix yields the divided differences in the first row.

Theorem 10.22 (Opitz). The divided difference f [λ1 , λ2 , . . . , λm ] is the (1, m) ele-
ment of f (Z), where
 
λ1 1
 λ2 1 
 
 .. .. 
Z= . . .
 . 
 .. 1 
λm
For m = 2 the theorem is just (4.16), while for λ1 = λ2 = · · · = λm it reproduces
the Jordan block formula (1.4), in view of (B.27).
McCurdy, Ng, and Parlett [416, ] investigate in detail the accurate evaluation
of the divided differences of the exponential. They derive a hybrid algorithm that
10.4 Schur Algorithms 251

uses the standard recurrence when it is safe to do so (i.e., when the denominator is
not too small) and otherwise uses the Taylor series of the exponential in conjunction
with Theorem 10.22 in a sophisticated way that employs scaling and squaring. They
also discuss the use of matrix argument reduction, which has some benefits for their
algorithm since it can reduce the size of the imaginary parts of the eigenvalues. Note
that if Jk (λ) is a Jordan block then exp(Jk (λ)) = exp(Jk (λ − 2πij)) for any integer
j (since in (1.4) the values of et and its derivatives are unchanged by the shift λ →
λ − 2πij). Hence for an arbitrary A, each eigenvalue can be shifted by an integer
multiple of 2πi so that its imaginary part is of modulus at most π without changing
eA . If A is in triangular form then a technique developed by Ng [448, ] based on
the Parlett recurrence can be used to carry out matrix argument reduction. Li [384,
] shows how when A is real (10.41) can be evaluated in mainly real arithmetic,
with complex arithmetic confined to the computation of the divided differences.
Unfortunately, no precise statement of an overall algorithm for eA is contained in
the above references and thorough numerical tests are lacking therein.

10.4.2. Schur–Fréchet Algorithm

Kenney and Laub [348,  ]  develop an algorithm based on Theorem 4.12, which
shows that if T = T11 T12
0 T22 then the (1,2) block of eT
is the (1,2) block of the
 0 T12 

Fréchet derivative L diag(T11 , T22 ), 0 0 . They evaluate the Fréchet derivative
using Algorithm 10.27 described in Section 10.6.2 below. For an overall algorithm,
eT11 and eT22 are recursively blocked and evaluated in the same way. The rather
intricate details can be found in [348, ].

10.4.3. Schur–Parlett Algorithm

A simple modification of Algorithm 9.6 (Schur–Parlett) to call Algorithm 10.20 (scal-
ing and squaring) on the diagonal blocks specializes it to the matrix exponential.
 
However, it is worth evaluating the exponential of any 2 × 2 diagonal blocks λ01 tλ122
from an explicit formula. The usual formula (4.16) for the (1, 2) element in the non-
confluent case suffers from cancellation when λ1 ≈ λ2 . But we can rewrite this formula
as

eλ2 − eλ1 e(λ2 −λ1 )/2 − e(λ1 −λ2 )/2

t12 = t12 e(λ1 +λ2 )/2
λ2 − λ1 λ2 − λ1


sinh (λ2 − λ1 )/2
= t12 e(λ1 +λ2 )/2 .
(λ2 − λ1 )/2

This leads to the formula, valid for all λ1 and λ2 ,

  

λ1 t12 eλ1 t12 e(λ1 +λ2 )/2 sinch (λ1 − λ2 )/2
exp = . (10.42)
0 λ2 0 eλ2

This will provide an accurate result as long as an accurate implementation of sinch is

available.

Algorithm 10.23 (Schur–Parlett algorithm for matrix exponential). Given A ∈ Cn×n

this algorithm computes F = eA via a Schur decomposition.
252 Matrix Exponential

1 Execute Algorithm 9.6 with line 7 replaced by

“7 Evaluate Fii = eTii directly if Tii is 1 × 1, by (10.42) if Tii is 2 × 2,
or else by Algorithm 10.20 (scaling and squaring algorithm).”

The evaluation of the exponential of the atomic diagonal blocks by Algorithm 10.20
brings the greater efficiency of Padé approximation compared with Taylor approxi-
mation to the exponential and also exploits scaling and squaring (though the latter
could of course be used in conjunction with a Taylor series).
Compared with Algorithm 10.20 applied to the whole (triangular) matrix, the
likelihood of overscaling is reduced because the algorithm is being applied only to the
(usually small-dimensioned) diagonal blocks (and not to the 1 × 1 or 2 × 2 diagonal
blocks).
Algorithm 10.23 is our preferred alternative to Algorithm 10.20. It has the ad-
vantage over the methods of the previous two subsections of simplicity and greater
efficiency, and its potential instabilities are more clear.

10.5. Numerical Experiment

We describe an experiment that compares the accuracy of four methods for computing
eA . The matrices (mostly 10 × 10) and most of the details are exactly the same as in
the experiment of Section 9.4. The methods are as follows.

1. MATLAB’s expm, which implements Algorithm 10.20.

2. The modified version funm mod of MATLAB’s funm tested in Section 9.4, which
exponentiates the diagonal blocks in the Schur form using Taylor series.

3. MATLAB 7.6’s funm, which implements Algorithm 10.23 when it is invoked as

funm(A,@exp).

4. MATLAB’s expmdemo1: a function that implements the scaling and squaring

method with m = 6 and k2−s Ak∞ ≤ 0.5 as the scaling criterion. This is an
M-file version of the expm function that was used in MATLAB 7 (R14SP3) and
earlier versions.

Figure 10.3 shows the normwise relative errors kXb −eA kF /keA kF of the computed
b
X. Figure 10.4 presents the same data in the form of a performance profile: for a
given α on the x-axis, the y coordinate of the corresponding point on the curve is the
probability that the method in question has an error within a factor α of the smallest
error over all the methods on the given test set. Both plots are needed to understand
the results: the performance profile reveals the typical performance, while Figure 10.3
highlights the extreme cases.10 For more on performance profiles see Dolan and Moré
[161, ] and Higham and Higham [263, , Sec. 22.4].
Several observations can be made.

• expm is the most accurate and reliable code overall.

• funm and funm mod perform very similarly.

10 To be more precise, a performance profile shows the existence and extent of unusually large errors

for a method if the x-axis is extended far enough to the right, but we have limited to x ∈ [1, 15] for
readability.
10.6 Evaluating the Fréchet Derivative and Its Norm 253

0
10
funm_mod
funm
expm
expmdemo1
−5
10

−10
10

−15
10

−20
10
0 10 20 30 40 50 60 70

Figure 10.3. Normwise relative errors for MATLAB’s funm, expm, expmdemo1, and funm mod;
the solid line is κexp (A)u.

• expmdemo1 is the least reliable of the four codes. This is mainly due to its
suboptimal choice of m and s; expm usually takes a larger s and hence requires
fewer squarings.

10.6. Evaluating the Fréchet Derivative and Its Norm

In some applications it is desired to compute not only eA but also the Fréchet deriva-
tive, in order to obtain sensitivity information or to apply an optimization algorithm
requiring derivatives. Two problems are of interest: approximating L(A, E) for a
given E and computing or estimating kL(A)k. We will begin by discussing several
different approaches to the former problem and return to the norm problem at the
end of the section.
The relation (3.16) gives
   A 
A E e L(A, E)
exp = . (10.43)
0 A 0 eA

Hence the Fréchet derivative L(A, E) can be obtained by applying any existing method
for the exponential to the above block upper triangular 2n×2n matrix and reading off
the (1,2) block. Of course it may be possible to take advantage of the block triangular
form in the method. This approach has the major advantage of simplicity but is likely
to be too expensive if n is large.
It is interesting to note a duality between the matrix exponential and its Fréchet
derivative. Either one can be used to compute the other: compare (10.43) with
Section 10.4.2.
Some other techniques for evaluating the Frechét derivative exploit scaling and
squaring. Applying the chain rule (Theorem 3.4) to the identity eA = (eA/2 )2 gives,
254 Matrix Exponential

0.9

0.8

0.7

0.6
p
0.5

0.4

0.3

0.2 expm
funm_mod
0.1 funm
expmdemo1
0
2 4 6 8 10 12 14
α

Figure 10.4. Same data as in Figure 10.3 presented as a performance profile.

on recalling that Lx2 (A, E) = AE + EA (see (3.10)),

Lexp (A, E) = Lx2 eA/2 , Lexp (A/2, E/2)
= eA/2 Lexp (A/2, E/2) + Lexp (A/2, E/2)eA/2 .

This relation yields the following recurrence for computing L0 = Lexp (A, E):

Ls = Lexp (2−s A, 2−s E), (10.44a)

2−i A 2−i A
Li−1 = e Li + Li e , i = s: −1: 1. (10.44b)

The advantage of the recurrence is that it reduces the problem of approximating the
Fréchet derivative for A to that of approximating the Fréchet derivative for 2−s A, and
s can be chosen to make k2−s Ak small enough that the latter problem can be solved
−i −(i+1)
2
accurately and efficiently. In the recurrence the relation e2 A = e2 A
can be
exploited to save computational effort, although repeated squaring could worsen the
effects of rounding error. Another attraction of the recurrence is that it is possible to
intertwine the computation of L(A, E) with that of eA by the scaling and squaring
method and thereby compute both matrices with less effort than is required to com-
−s
pute each separately; however, the requirements on s for the approximation of e2 A
and L2−s A may differ, as we will see below. We now describe two approaches based
on the above recurrence.

10.6.1. Quadrature
One way to approximate L(A, E) is to apply quadrature to the integral representation
(10.15). For example, we can apply the m-times repeated trapezium rule
Z 1
1
f (t) dt ≈ ( 21 f0 + f1 + f2 + · · · + fm−1 + 21 fm ), fi := f (i/m)
0 m
10.6 Evaluating the Fréchet Derivative and Its Norm 255

to obtain
1 1 A 
m−1
X 1
L(A, E) ≈ e E+ eA(1−i/m) EeAi/m + EeA .
m 2 i=1
2

This formula is expensive to evaluate, because it requires eA/m , e2A/m , . . ., e(m−1)A/m .

However, when m = 2s a much more efficient evaluation is possible, for any integration
rule.
R1 Pp
Theorem 10.24. Consider a quadrature formula 0 f (t) dt ≈ i=1 wi f (ti ) = R1 (f )
and denote the result
Pof applying its m-times repeated form to (10.15) by Rm (A, E),
p
so that R1 (A, E) = i=1 wi eA(1−ti ) EeAti . If

Qs = R1 (2−s A, 2−s E), (10.45a)

2−i A 2−i A
Qi−1 = e Qi + Qi e , i = s: −1: 1, (10.45b)

then Q0 = R2s (A, E).

Proof. We have
p
1X
R1 (A/2, E/2) = wi eA(1−ti )/2 EeAti /2 .
2 i=1

Now
p
1X
eA/2 R1 (A/2, E/2) + R1 (A/2, E/2)eA/2 = wi eA(1−ti /2) EeAti /2
2 i=1


+ eA(1−ti )/2 EeA(1+ti )/2
= R2 (A, E), (10.46)
1
Pp

since R2 (f ) = 2 i=1 wi f (ti /2)+f (1/2+ti /2) . Repeated use of the relation (10.46)
yields the result.
Note that the approximation Q0 can be derived by using Lexp ≈ R1 in (10.44). The
extra information in Theorem 10.24 is that the resulting approximation is precisely
R 2s .
Two key questions are how large the error R2s (A, E) − L(A, E) is and which
quadrature rule to choose.

The next result helps in these regards. We define µe(A) =
max 0, 21 (µ(A) + µ(−A) , where µ is the logarithmic norm (10.11), and note that
both µ e(A) and µ(A) are bounded above by kAk2 .

Theorem 10.25 (error bounds). Assume that c := kAk22 eµe(A)/m /(6m2 ) < 1. Then
for the repeated trapezium rule RT,m , the repeated Simpson rule RS,m , and any uni-
tarily invariant norm,

2c
kL(A, E) − RT,m (A, E)k ≤ kL(A)k kEk, (10.47)
1−c
eµe(A)/m
kL(A, E) − RS,m (A, E)k ≤ km−1 Ak42 kL(A)k kEk. (10.48)
180(1 − c)
256 Matrix Exponential

Proof. See Mathias [408, , Cor. 2.7].

If, for example, kAk2 /m ≤ 1/2 then c < 0.07 and kL(A, E) − RT,m (A, E)k ≤
0.15 kL(A)k kEk.
In order that s not be unduly large, the quadrature approach is restricted to
providing low accuracy approximations to L(A, E).
Note that (10.47) and (10.48) are not relative errors bounds, because kL(A)k
rather than kL(A, E)k appears on the right-hand side, and kL(A, E)k ≪ kL(A)k kEk
is possible. However, by maximizing these bounds over all E we obtain kL(A) −
RT,m (A)k ≤ θkL(A)k, where θ = 2c/(1 − c), and similarly for RS,m . Also note that
(10.47) readily yields

(1 − θ)kL(A)k ≤ kRT,m (A)k ≤ (1 + θ)kL(A)k,

and similarly for (10.48).

In order to guarantee that c < 1, given only kAk2 , we need to assume that
kAk2 /m < 1.28. If we intertwine the computation of L(A) with Algorithm 10.20
then since θ13 ≈ 5.37 exceeds 1.28 by more than a factor 4, we may be forced to scale
by an extra factor 8 in the scaling and squaring method, resulting in 3 extra squarings
and possible loss of accuracy. Hence evaluation of L(A, E) by quadrature does not
mesh particularly well with the scaling and squaring method.
Mathias [408, ] shows that if we scale so that kAk2 /2s ∈ (1/4, 1/2] (for which
c = 0.069 and 2c/(1 − c) = 0.15) then Simpson’s rule RS,2s −1 is more accurate than
the trapezium rule TS,2s both in theory (from Theorem 10.25) and in practice, while
having the same cost.

Algorithm 10.26 (low accuracy approximation to Fréchet derivative of eA ). Given

A ∈ Cn×n this algorithm approximates the Fréchet derivative L = L(A, E) of the
matrix exponential via the repeated Simpson rule, aiming to provide a result with
norm of the correct order of magnitude.

1 B = A/2s with s ≥ 0 a minimal integer such that kA/2s k1 ≤ 1/2

2 X = eB
3 Xe = eB/2
4 Qs = 2−s (XE + 4XE e Xe + EX)/6
5 for i = s: −1: 1
−i
6 if i < s, X = e2 A , end
7 Qi−1 = XQi + Qi X
8 end
9 L = Q0

Cost: (4 + 2s)M and s + 1 matrix exponentials (or 1 exponential and (4 + 3s)M if

repeated squaring is used at line 6).

10.6.2. The Kronecker Formulae

Another way to approximate L(A, E) is by using the Kronecker formulae (10.17).
Consider the formula
1 AT

vec(L(A, E)) = (e ⊕ eA ) τ 1
2 [A
T
⊕ (−A)] vec(E), (10.49)
2
10.6 Evaluating the Fréchet Derivative and Its Norm 257

where τ (x) = tanh(x)/x and k 12 [AT ⊕ (−A)]k < π/2 is assumed. Using (B.16), we
have
1

L(A, E) = (Y eA + eA Y ), vec(Y ) = τ 21 [AT ⊕ (−A)] vec(E).
2
Note that 12 [AT ⊕(−A)] vec(E) = 21 (AT ⊗I −I ⊗A) vec(E) = 12 vec(EA−AE). Hence
if r(x) is a rational approximation to τ (x) in which both

numerator and denominator
are factored into linear factors then r 21 [AT ⊕ (−A)] vec(E) can be evaluated at the
cost of solving a sequence of Sylvester equations containing “Sylvester products” on
the right-hand side. To be specific, let
m
Y
τ (x) ≈ rm (x) = (x/βi − 1)−1 (x/αi − 1) (10.50)
i=1

be a rational approximation. Then we can approximate L(A, E) using the following

“Sylvester cascade”:
G0 = E,
       
A A A A
I+ Gi + Gi I− = I+ Gi−1 + Gi−1 I − , i = 1: m,
βi βi αi αi
1

L(A, E) ≈ Gm eA + eA Gm .
2
A natural choice for rm is a Padé approximant. Padé approximants to τ can be
obtained by truncating the continued fraction expansion
1
τ (x) = tanh(x)/x = 1 + 2 . (10.51)
x /(1 · 3)
1+
x2 /(3 · 5)
1+
x2 /((2k − 1) · (2k + 1))
1 + ··· +
1 + ···
Kenney and Laub [348, ] show that for kAk < π/2 we have kτ (A) − rm (A)k ≤
g(kAk), where g(x) = τ (ix) − rm (ix) and the norm is any consistent matrix norm.
Defining C = 12 [AT ⊕ (−A)] and noting that kCkp ≤ kAkp for 1 ≤ p ≤ ∞, it follows
that if kAkp ≤ 1 then kτ (C) − r8 (C)kp ≤ g(1) = 1.44 × 10−16 , and the condition
kCkp ≤ π/2 for (10.49) to be valid is then satisfied. The condition kAkp ≤ 1 can be
arranged by scaling A ← 2−s A and using the recurrence (10.44) to undo the effects
of the scaling. The [8/8] Padé approximant is given by
p8 (x) x8 + 990x6 + 135135x4 + 4729725x2 + 34459425
r8 (x) = =
q8 (x) 45x8 + 13860x6 + 945945x4 + 16216200x2 + 34459425
and the zeros of p8 and q8 are given in Table 10.5. In choosing how far to scale A we
also need to consider the conditioning of the Sylvester cascade. This boils down to
ensuring that the upper bound in
m
Y m
Y m
Y
1 + kC/αi kp 1 + kA/αi kp
k(C/βi − I)−1 kp kC/αi − Ikp ≤ ≤
i=1 i=1
1 − kC/βi kp i=1
1 − kA/βi kp

is not too large. For m = 8 and the 1-norm this bound is at most 55.2 if kAkp ≤ 1,
which is quite acceptable.
The ideas above are collected in the following algorithm.
258 Matrix Exponential

Table 10.5. Zeros αj of numerator p8 and βj of denominator q8 of [8/8] Padé approximant

r8 to τ (x) = tanh(x)/x, shown to 5 significant digits.

αj βj
±3.1416e0i ±1.5708e0i
±6.2900e0i ±4.7125e0i
±1.0281e1i ±7.9752e0i
±2.8894e1i ±1.4823e1i

Algorithm 10.27 (Fréchet derivative of matrix exponential). Given A ∈ Cn×n this

algorithm evaluates the Fréchet derivative L = L(A, E) of the matrix exponential via
(10.49), using scaling and squaring and the [8/8] Padé approximant to τ = tanh(x)/x.
It uses the constants in Table 10.5. The algorithm is intended for IEEE double
precision arithmetic.

1 B = A/2s with s ≥ 0 a minimal integer such that kA/2s k1 ≤ 1.

2 G0 = 2−s E
3 for i = 1: 8
4 Solve for Gi the Sylvester equation (I + B/βi )Gi + Gi (I − B/βi ) =
(I + B/αi )Gi−1 + Gi−1 (I − B/αi ).
5 end
6 X = eB
7 Ls = (G8 X + XG8 )/2
8 for i = s: −1: 1
−i
9 if i < s, X = e2 A , end
10 Li−1 = XLi + Li X
11 end
12 L = L0

Cost: (18 + 2s)M and s matrix exponentials (or 1 exponential and (17 + 3s)M if
repeated squaring is used at line 9), and the solution of 8 Sylvester equations.
In practice we would use an initial Schur decomposition of A to reduce the cost,
making use of Problem 3.2; this is omitted from the algorithm statement to avoid
clutter.
Similar algorithms could be developed for the other two formulae in (10.17).

10.6.3. Computing and Estimating the Norm

The Fréchet derivative norm kL(A)kF is given by the explicit formulae in Corol-
lary 10.14, which involve the 2-norms of n2 × n2 matrices. If n is small it is practical
to compute the matrix directly from its formula, for example by Algorithm 9.6, pro-
vided a means for evaluating the relevant scalar functions and their derivatives is
available; for ψ1 see Section 10.7.4. However, direct evaluation in this way is prone
to overflow. For example, if A
has a negative eigenvalue λi with |λi | ≫ |λj | ≥ 1 for
some j 6= i, then ψ1 AT
⊕ (−A) has an eigenvalue approximately ψ1 (−λi ) = ψ1 (|λi |)
and so kψ1 AT ⊕ (−A) k will be large and could overflow, even though eA may be of
modest norm.
An alternative approach, which avoids unnecessary overflow, is to apply Algo-
rithm 3.17, using Algorithm 10.26 or 10.27 to evaluate L(X, E) for the n2 specified
10.7 Miscellany 259

choice of E. For large n, this is prohibitively expensive and the condition num-
ber must be estimated rather than exactly computed. One possibility is to use
the power method, Algorithm 3.20, in conjunction with Algorithm 10.26 or Algo-
rithm 10.27. A good starting matrix is Z0 = XX ∗ + X ∗ X, where X = eA , the
reason being that X = L(A, I) and the adjoint step of the power method gives
R1 ∗
L⋆ (A, X) = L(A∗ , X) = 0 eA (1−s) eA eA s ds ≈ (XX ∗ + X ∗ X)/2, using a trapezium
∗

rule approximation. Hence Z0 approximates at relatively small cost the result of a

power method iteration.
The following is our preferred approach.

Algorithm 10.28 (condition number estimate for matrix exponential). Given A ∈

Cn×n this algorithm produces an estimate γ of the 1-norm condition number κexp (A)
of the matrix exponential; more precisely, γ is a lower bound for a quantity in the
interval [n−1 κexp (A), nκexp (A)].

1 Invoke Algorithm 3.22 to produce an estimate γ of kL(A)k1 ,

using Algorithm 10.27 to evaluate L(A, E) and L⋆ (A, E) = L(A∗ , E).
2 γ ← γkAk1 /keA k1

Cost: About 4–5 invocations of Algorithm 10.27.

For efficiency, A would in practice be reduced to Schur form at the start of Algo-
rithm 3.17 and Algorithm 10.28.
We ran Algorithm 10.28 on the same set of matrices used in Section 10.5. We
also ran a modified version of Algorithm 10.28 that estimates L(A, E) by the finite
difference (3.22) with increment (3.23). We compared the estimates with a 1-norm
analogue of Algorithm 3.17 in which each norm on line 7 is the 1-norm and L(A, E)
is evaluated by Algorithm 10.27. The three estimates were mostly of the same order
of magnitude, the only exceptions being for extremely ill conditioned problems where
the estimate via finite differences was occasionally several orders of magnitude too
large (a relatively harmless overestimation).

10.7. Miscellany
In this section we discuss some miscellaneous methods, techniques, and special classes
of matrices, as well as a variation of the eA problem.

10.7.1. Hermitian Matrices and Best L∞ Approximation

If A is Hermitian (or, more generally, normal) then approximating eA is equivalent
to approximating the exponential function on the spectrum of A. Best rational L∞
approximations rm from Rm,m to ex on (−∞, 0] (or, equivalently, to e−x on [0, ∞))
have been determined for m ≤ 14 by Cody, Meinardus, and Varga [110, ] and
to high accuracy by Carpenter, Ruttan, and Varga [98, ] (see Section 4.4 for
the definition of Rm,m ). Define the L∞ error Em = supx∈(−∞,0] |rm (x) − ex |. It is
1/m
known that limm→∞ Em = 1/9.289025 . . . [98, ] (see Trefethen, Weideman, and
Schmelzer [574, ] or Finch [187, , Sec. 4.5] for the interesting history of this
result and references), which implies that Em ≈ 10−m .
The approximations rm are valid on the whole negative real axis, whereas Padé
approximations are accurate only near the origin. In applications from differential
260 Matrix Exponential

equations in which eAh is required, with h a timestep, these best L∞ approximations

allow the use of much larger timesteps than Padé approximants [200, ].
For evaluating rm (A) the partial fraction form is preferred. The poles have been
observed in practice to be distinct and to come in complex conjugate pairs; see [200,
], where values of the poles are tabulated for degrees 10 and 14.
Lu [395, ] suggests shifting by the smallest eigenvalue µ and evaluating eA ≈
µ
e rm (A − µI), using unitary reduction to tridiagonal form to reduce the evaluation of
rm to the tridiagonal case and then employing the partial fraction expansion of rm .

10.7.2. Essentially Nonnegative Matrices

A matrix A ∈ Rn×n is essentially nonnegative if aij ≥ 0 for i 6= j. This class of
matrices merits special consideration because of its occurrence in important applica-
tions such as Markov chains (see Section 2.3) and the numerical solution of PDEs
[600, , Sec. 8.2]. It is worth noting that the set Zn×n arising in the theory of
nonnegative matrices comprises the negatives of the essentially nonnegative matrices
[60, , Chap. 6].
Interestingly, the essentially nonnegative matrices are precisely those for which
eAt is nonnegative for all t ≥ 0.

Theorem 10.29. Let A ∈ Rn×n . Then eAt ≥ 0 for all t ≥ 0 if and only if A is
essentially nonnegative.
Proof. Suppose eAt ≥ 0 for all t ≥ 0. Taking t ≥ 0 sufficiently small in the
expansion eAt = I + At + O(t2 ) shows that the off-diagonal elements of A must be
nonnegative.
Conversely, suppose A is essentially nonnegative. We can write A = D + N , where
D = diag(A) and N ≥ 0, which can be rewritten as
A = θI + (D − θI + N ) =: θI + B, θ = min aii . (10.52)
i

Then B ≥ 0 and eAt = eθt eBt ≥ 0 for t ≥ 0, since the exponential of a matrix with
nonnegative elements is clearly nonnegative.
The formula eA = eθ eB with θ, B as in (10.52) is used in the uniformization
method in Markov chain analysis [523, ], [524, , Sec. 2], [540, , Chap. 8],
in which eB v (v ≥ 0) is approximated using a truncated Taylor series expansion of
eB . The Taylor series for eB contains only nonnegative terms, whereas that for eA
contains terms of both sign and so can suffer from cancellation. Note that in the case
of A being an intensity matrix the shift θ is optimal in the sense of Corollary 4.22.
Recall from Section 2.3 that an intensity matrix is an essentially nonnegative
matrix with zero row sums, and that the exponential of such a matrix is stochastic
and so has unit ∞-norm. An intensity matrix is perfectly conditioned for the matrix
exponential.

Theorem 10.30. If A ∈ Rn×n is an intensity matrix then in the ∞-norm, κexp (A) =
kAk∞ .
Proof. For s ∈ [0, 1], sA is also an intensity matrix. Hence on taking ∞-norms
in (10.15) we obtain
Z 1
kL(A, E)k∞ ≤ keA(1−s) k∞ kEk∞ keAs k∞ ds
0
10.7 Miscellany 261

Z 1
= kEk∞ ds = kEk∞ = keA k∞ kEk∞ .
0

Thus κexp (A) ≤ kAk∞ , and this is an equality in view of (10.22).

10.7.3. Preprocessing
Balancing, and argument reduction via a single shiftA ← A − µI, can be used as
preprocessing steps in any of the algorithms described in this chapter. Theorem 10.18
in conjunction with Theorem 4.21 (a) suggests that µ = n−1 trace(A) is an excellent
choice of shift. However, this shift is not always appropriate. If A has one or more
eigenvalues with large negative real part then µ can be large and negative and A − µI
can have an eigenvalue with large positive real part, causing eA−µI to overflow.
Preprocessing is potentially particularly helpful to Algorithm 10.20 (the scaling
and squaring method), since it can reduce the norm and hence directly reduce the cost
of the method. Our own numerical experience suggests that preprocessing typically
has little effect on the cost or accuracy of Algorithm 10.20, and since it could even
worsen the accuracy we do not recommend its automatic use.

10.7.4. The ψ Functions

P∞
The functions ψk (z) = j=0 z j /(j + k)! are closely related to the exponential:

ez − 1 ez − 1 − z ez − 1 − z − 12 z 2
ψ0 (z) = ez , ψ1 (z) = , ψ2 (z) = , ψ3 (z) = ,....
z z2 z3
As explained in Section 2.1, these functions play a fundamental role in exponential
integrators for ordinary differential equations, and we saw earlier in this chapter that
ψ1 arises in one of the representations (10.17) for the Fréchet derivative of the matrix
exponential. An integral representation is
Z 1
1
ψk (z) = e(1−t)z tk−1 dt (10.53)
(k − 1)! 0

and the ψk satisfy the recurrence relation

1
ψk (z) = z ψk+1 (z) + . (10.54)
k!
From the last two equations it follows that ψk+1 is the divided difference

ψk+1 (z) = ψk [z, 0].

Even for scalar arguments it is a nontrivial task to evaluate the ψk accurately. For
example, ψ1 (z) suffers badly from cancellation if evaluated as (ez − 1)/z for |z| ≪ 1,
though this formula can be reworked in a nonobvious way to avoid the cancellation
[276, , Sec. 1.14.1] (or the function expm1(x) := ex − 1, available as expm1 in
MATLAB, can be invoked [562, ]).
Padé approximants to the ψ functions are known explicitly. The following theorem
includes (10.23) and (10.24) (with k = m therein) as special cases.
262 Matrix Exponential

Theorem 10.31. The diagonal [m/m] Padé approximant rm (z) = pm (z)/qm (z) to
ψk (z) is given by
 
Xm i
X j
m!  (2m + k − j)!(−1)  zi,
pm (z) =
(2m + k)! i=0 j=0 j!(m − j)!(k + i − j)!
m
X (2m + k − i)!
m!
qm (z) = (−z)i ,
(2m + k)! i=0 i!(m − i)!

and
(−1)m m!(m + k)!
ψk (z) − rm (z) = z 2m+1 + O(z 2m+2 ).
(2m + k)!(2m + k + 1)!
Proof. See Skaflestad and Wright [527, ]. Magnus and Wynn [401, ] had
earlier obtained the [k/m] Padé approximants to ψ0 .
With the aid of these Padé approximants, evaluation of ψj , j = 0: k, can be done
via an analogue of the scaling and squaring method for the exponential. The squaring
phase makes use (with α = β = 1) of the identity [527, ], for α, β ∈ R,
 Xk 

1 k αj β k−j
ψk (α + β)z = β ψ 0 (αz)ψ k (βz) + ψ j (αz) . (10.55)
(α + β)k j=1
(k − j)!

A basic Padé-based scaling and squaring algorithm is proposed by Hochbruck, Lubich,

and Selhofer [292, ] for ψ1 (A), while Koikari [360, ] and Skaflestad and Wright
[527, ] develop more sophisticated versions, analogous to Algorithm 10.20, for
evaluating the sequence ψ0 (A), . . . , ψk (A).
As a special case of (10.40) (or directly, by using Bf (B) = f (B)B), we have
   A 
A I e ψ1 (A)
B= ⇒ eB = , (10.56)
0 0 0 I

so the evaluation of ψ1 can be reduced to that of the exponential. The practical

use of (10.56) for Hermitian A is explored by Lu [397, ]. This relation can be
extended to provide a sequence of higher order ψ functions from a single exponential
evaluation; see Sidje [522, , Thm. 1] and Minchev [430, ].

10.8. Notes and References

An early exposition of the matrix exponential and its use in solving differential equa-
tions is given by Frazer, Duncan, and Collar [193, ].
Moler and Van Loan’s classic paper [437, ], [438, ] is highly cited, with
over 500 citations in the ISI Citation Index. Golub and Van Loan [224, , Sec. 11.3]
give an overview of the scaling and squaring method and of perturbation theory for the
matrix exponential. Van Loan [592, ] is a good source for fundamental theoretical
results concerning the matrix exponential.
Theorem 10.1 is due to Suzuki [552, ] and Theorem 10.3 to Wermuth [613,
].
For more details on the Baker–Campbell–Hausdorff formula (10.5) see Hall [241,
, Chap. 3] or Varadarajan [598, , Thm. 2.15.4], and for its history see Weiss
10.8 Notes and References 263

and Maradudin [612, ]. The domain of convergence of the formula is discussed by
Blanes and Casas [74, ]. A simple way to compute the coefficients in the formula,
involving a matrix log(eF eG ) where F and G are certain strictly upper bidiagonal
matrices, is given by Reinsch [486, ].
For more on the Zassenhaus formula (10.6) see Magnus [403, ] or Wilcox [615,
], and see Scholz and Weyrauch [506, ] for a simple way to compute the
coefficients Ci .
The Strang splitting is due to Strang [544, ]. For a survey of splitting methods
see McLachlan and Quispel [417, ].
Theorem 10.6 is from Suzuki [552, ]. The formula (10.8) is widely used in
physics, where it is known as the Suzuki–Trotter formula. For an example of the
use of the formula see Bai, Chen, Scalettar, and Yamazaki [26, ]. The Trotter
product formula in Corollary 10.7 is from Trotter [575, ].
Other bounds on keA k can be found in Van Loan [594, ] and Kågström [322,
]. For various bounds on supt≥0 ketA k see Trefethen and Embree [573, ].
The representation (10.15) for the Fréchet derivative appears in Karplus and
Schwinger [335, ], and Najfeld and Havel [445, ] state that this is the earliest
appearance.
Theorem 10.13 is a mixture of relations from Kenney and Laub [348, ] and
Najfeld and Havel [445, ]. It is possible to “unvec” the relations (10.17), but the
resulting expressions (see [580, ] or [241, 2003, Thm. 3.5] for (10.17a)) are less
useful for computational purposes.
Theorem 10.16 is from Van Loan [594, ]. Theorem 10.17 is due to Melloy
and Bennett [422, ], whose paper seems to have hitherto been overlooked. The-
orem 10.18 is from Parks [458, ].
For more on the theory and computation of the Fréchet derivative (and higher
derivatives) of the exponential see Najfeld and Havel [445, ].
The formulae (10.23) and (10.24) are due to Padé, and proofs are given by Gautschi
[207, , Thm. 5.5.1]. A proof of the “ρ(rmm (A)) < 1” property can be found in
[207, , p. 308].
It is known that the poles of the [k/m] Padé approximant rkm (x) = pkm (x)/qkm (x),
that is, the zeros of qkm , are distinct, though they are complex; see Zakian [621, ,
Lem. 4.1]. Thus rkm can be expressed in linear partial fraction form, as is done by
Gallopoulos and Saad [200, ], for example.
An early reference to the scaling and squaring method is Lawson [376, ], and
Ward [606, ] attributes the method to Lawson.
The evaluation schemes (10.32) and (10.33) were suggested by Van Loan [593,
].
Algorithm 10.20 and the supporting analysis are due to Higham [278, ]. Sec-
tion 10.3 is based on [278].
The analysis of Moler and Van Loan [438, , Sec. 3, App. A] proceeds as in
Problem 10.10 and makes the assumption that k2−s Ak ≤ 1/2. The backward error
bound is used to determine a suitable Padé degree m. Based on this analysis, in
MATLAB 7 (R14SP3) and earlier the expm function used an implementation of the
scaling and squaring method with m = 6 and k2−s Ak∞ ≤ 1/2 as the scaling criterion
(and Sidje [522, ] uses the same parameters in his function padm). Version 7.2
(R2006a) of MATLAB introduced a new expm that implements Algorithm 10.20. This
new version of expm is more efficient than the old, as the experiment of Section 10.5
indicates, and typically more accurate, as a side effect of the reduced scaling and
264 Matrix Exponential

hence fewer squarings. Mathematica’s MatrixExp function has used Algorithm 10.20
for matrices of machine numbers since Version 5.1.
Ward [606, ] uses m = 8 and k2−s Ak1 ≤ 1 in his implementation of the scaling
and squaring method and preprocesses with argument reduction and balancing, as
discussed in Section 10.7.3. He gives a rounding error analysis that leads to an
a posteriori forward error bound. This analysis is similar to that given here, but for
the squaring phase it uses a recurrence based on computed quantities in place of the
a prior bound of Theorem 10.21.
Najfeld and Havel [445, ] suggest a variation of the scaling and squaring
method that uses Padé approximations to the function

τ (x) = x coth(x) = x(e2x + 1)(e2x − 1)−1

x2
=1+ , (10.57)
x2
3+
x2
5+
7 + ···

from which the exponential can be recovered via e2x = (τ (x) + x)/(τ (x) − x). The
continued fraction expansion provides an easy way of generating the Padé approxi-
mants to τ . Higham [278, ] shows that the algorithm suggested by Najfeld and
Havel is essentially a variation of the standard scaling and squaring method with di-
rect Padé approximation, but with weaker guarantees concerning its behaviour both
in exact arithmetic (because a backward error result is lacking) and in floating point
arithmetic (because a possibly ill conditioned matrix must be inverted).
The overscaling phenomenon discussed at the end of Section 10.3 was first pointed
out by Kenney and Laub [348, ] and Dieci and Papini [157, ].
Equation (10.40) shows that the (1,2) block of the exponential of a block triangular
matrix can be expressed as an integral. Van Loan [595, ] uses this observation in
the reverse direction to show how certain integrals involving the matrix exponential
can be obtained by evaluating the exponential of an appropriate block triangular
matrix.
A conditioning analysis of the exponential taking account of the structure of block
2 × 2 triangular matrices is given by Dieci and Papini [158, ].
Theorem 10.22 is due to Opitz [452, ] and was rediscovered by McCurdy: see
[416, ]. The result is also proved and discussed by de Boor [143, ].
The use of quadrature for evaluating L(A, E) was investigated by Kenney and
Laub [340, ], who concentrate on the repeated trapezium rule, for which they
obtain the recurrence (10.45) and a weaker error bound than (10.47).
Section 10.6.2 is based on Kenney and Laub [348, ], wherein the focus is not on
computing the Fréchet derivative of the exponential per se, but on using the Fréchet
derivative within the Schur–Fréchet algorithm (see Section 10.4.2). The continued
fraction expansion (10.51) can be found in Baker [38, , p. 65]. The starting
matrix Z0 = XX ∗ + X ∗ X for the power method mentioned in Section 10.6.3 was
suggested by Kenney and Laub [340, ].
Theorem 10.29 can be found in Bellman [51, , p. 176] and in Varga [600, ,
Sec. 8.2]. Theorem 10.30 is new.
While MATLAB, Maple, and Mathematica all have functions to compute the
matrix exponential, the only freely available software we are aware of is Sidje’s Expokit
package of MATLAB and Fortran codes [522, ].
Problems 265

Problems
10.1. Prove the identity (10.13).
10.2. Give a proof of AB = BA ⇒ e(A+B)t = eAt eBt for all t (the “if” part of
Theorem 10.2) that does not use the power series for the exponential.
10.3. Show that for any A, B ∈ Cn×n ,

keA − eB k ≤ kA − Bk emax(kAk,kBk) .

10.4. For A, B ∈ Cn×n show that det(eA+B ) = det(eA eB ) (even though eA+B 6=
eA eB in general).
10.5. Show that for A ∈ Cn×n and any subordinate matrix norm, κexp (A) ≤ e2kAk kAk.
10.6. Let A ∈ Cn×n be normal. Theorem 10.16 says that the absolute condition
number condabs (exp, X) = keA k2 , but Corollary 3.16 says that condabs (exp, A) =
maxλ,µ∈Λ(A) |f [λ, µ]|, where f = exp. Reconcile these two formulae.
10.7. Determine when the Fréchet derivative L(A) of the exponential is nonsingular.
10.8. Show that if kGk < 1 then

kGk
k log(I + G)k ≤ − log(1 − kGk) ≤ (10.58)
1 − kGk

while if kGk < 2/3 then

 
kGk
kGk 1 − ≤ k log(I + G)k,
2(1 − kGk)

where the norm is any consistent matrix norm.

10.9. (Dieci and Papini [157, ], Moler and Van Loan [438, , Lem. 2]) Show
that if the denominator qm (A) of the [m/m] Padé approximant to eA satisfies qm (−kAk) <
2 then it is nonsingular and
1
kqm (A)−1 k ≤ .
2 − qm (−kAk)

The norm here is any subordinate matrix norm. Show further that if kAk < log 4 =
1.39 then kqm (A)−1 k ≤ 1/(2 − ekAk/2 ), and hence that kqm (A)−1 k ≤ 3 for kAk ≤ 1.
10.10. Let rm = pm /qm be the [m/m] Padé approximant to the exponential. Show
that G = e−A rm (A) − I satisfies

qm (kAk)  kAk 
kGk ≤ e − rm (kAk) (10.59)
2 − qm (−kAk)

for any subordinate matrix norm, assuming that kGk < 1 and qm (−kAk) < 2.
10.11. Prove Theorem 10.21, using the basic result that for the 1-, ∞-, and Frobenius
norms [276, , Sec. 3.5], kAB − f l(AB)k ≤ γn kAk kBk.
10.12. Derive the formula (10.40) for the exponential of a block 2×2 block triangular
matrix.
266 Matrix Exponential

10.13. Prove Rodrigues’ formula [407, , p. 291] for the exponential of a skew-
symmetric matrix A ∈ R3×3 :
sin θ 1 − cos θ 2
eA = I + A+ A , (10.60)
θ θ2
p
where θ = kAk2F /2. (For generalizations of this formula to higher dimensions see
Gallier and Xu [199, ].)
10.14. Given that eA is the matrix
 
cos t 0 0 0 0 0 0 − sin t
 
 0 cos t 0 0 0 0 − sin t 0 
 
 
 0 0 cos t 0 0 − sin t 0 0 
 
 
 0 0 0 cos t − sin t 0 0 0 
 
 ,
 0 0 0 sin t cos t 0 0 0 
 
 
 0 0 sin t 0 0 cos t 0 0 
 
 
 0 sin t 0 0 0 0 cos t 0 
 
sin t 0 0 0 0 0 0 cos t

what is A?
10.15. (Research problem) Obtain conditions that are necessary or sufficient—
and ideally both—for κexp (A) to be large.
10.16. (Research problem) Prove the stability or otherwise of the scaling and
squaring algorithm, by relating rounding errors in the square root phase to the con-
ditioning of the eA problem.
 Problems 267

The problem of expressing eAt eBt in the form eCt ,

where A and B do not commute,
has important ramifications not only in the theory of Lie groups and algebras,
but also in modern quantum field theory.
— RICHARD BELLMAN, Introduction to Matrix Analysis (1970)

Calculating the exponential of a matrix can be quite complicated,

requiring generalized eigenvectors or
Jordan Canonical Form if you want explicit formulas.
Although, compared to nonlinear problems, this is still “simple,”
if you try to put this into practice,
you will come to understand the expletive that
“hell is undiagonalizable matrices.”
— JOHN H. HUBBARD and BEVERLY H. WEST, Differential Equations:
A Dynamical Systems Approach. Higher Dimensional Systems (1995)

The [error of the ] best rational approximation to e−x on [0, +∞) exhibits
geometric convergence to zero.
It is this geometric convergence which has fascinated so many researchers.
— A. J. CARPENTER, A. RUTTAN, and R. S. VARGA, Extended Numerical Computations
on the “ 1/9” Conjecture in Rational Approximation Theory (1984)

In this survey we try to describe all the methods that appear to be practical,
classify them into five broad categories,
and assess their relative effectiveness.
— CLEVE B. MOLER and CHARLES F. VAN LOAN,
Nineteen Dubious Ways to Compute the Exponential of a Matrix (1978)

[The ] availability of expm(A) in early versions of MATLAB

quite possibly contributed to
the system’s technical and commercial success.
...
Scaling and squaring has emerged as the
least dubious of the original nineteen ways,
but we still do not understand the method completely.
— CLEVE B. MOLER and CHARLES F. VAN LOAN,
Nineteen Dubious Ways to Compute the Exponential of a Matrix,
Twenty-Five Years Later (2003)
Chapter 11
Matrix Logarithm

A logarithm of A ∈ Cn×n is any matrix X such that eX = A. As we saw in Theo-

rem 1.27, any nonsingular A has infinitely many logarithms. In this chapter A ∈ Cn×n
is assumed to have no eigenvalues on R− and “log” always denotes the principal log-
arithm, which we recall from Theorem 1.31 is the unique logarithm whose spectrum
lies in the strip { z : −π < Im(z) < π }.
The importance of the matrix logarithm can be ascribed to it being the inverse
function of the matrix exponential and this intimate relationship leads to close con-
nections between the theory and computational methods for the two functions.
This chapter is organized as follows. We begin by developing some basic properties
of the logarithm, including conditions under which the product formula log(BC) =
log(B) + log(C) holds. Then we consider the Fréchet derivative and conditioning.
The Mercator and Gregory series expansions are derived and various properties of
the diagonal Padé approximants to the logarithm are explained. Two versions of
the inverse scaling and squaring method are developed in some detail, one using the
Schur form and the other working with full matrices. A Schur–Parlett algorithm
employing inverse scaling and squaring on the diagonal blocks together with a special
formula for 2 × 2 blocks is then derived. A numerical experiment comparing four
different methods is then presented. Finally, an algorithm for evaluating the Fréchet
derivative is described.

11.1. Basic Properties

We begin with an integral expression for the logarithm.

Theorem 11.1 (Richter). For A ∈ Cn×n with no eigenvalues on R− ,

Z 1  −1
log(A) = (A − I) t(A − I) + I dt. (11.1)
0

Proof. It suffices to prove the result for diagonalizable A, by Theorem 1.20, and
R1  −1
hence it suffices to show that log x = 0 (x − 1) t(x − 1) + 1 dt for x ∈ C lying off
R− ; this latter equality is immediate.
Now we turn to useful identities satisfied by the logarithm. Because of the multi-
valued nature of the logarithm it is not generally the case that log(eA ) = A, though
a sufficient condition for the equality is derived in Problem 1.39. To understand the
scalar case of this equality we use the unwinding number of z ∈ C defined by

z − log(ez )
U(z) = . (11.2)
2πi

269
270 Matrix Logarithm

It is easy to show that  

Im z − π
U(z) = ∈ Z.
2π
The equation
log(ez ) = z − 2πiU(z) (11.3)
completely describes the relationship between log(ez ) and z, in particular showing
that log(ez ) = z for Im z ∈ (−π, π].
The next result gives an important identity.

Theorem 11.2. For A ∈ Cn×n with no eigenvalues on R− and α ∈ [−1, 1] we have

log(Aα ) = α log(A). In particular, log(A−1 ) = − log(A) and log(A1/2 ) = 21 log(A).

Proof. Consider first the scalar case. By definition, aα = eα log a . Hence, using
(11.3), log aα = log(eα log a ) = α log a − 2πiU(α log a). But U(α log a) = 0 for a ∈
/ R−
and α ∈ [−1, 1]. Thus the identity holds for scalar A and follows for general A by
Theorem 1.20. (For the last part, cf. Problem 1.34.)
We now turn to the question of when log(BC) = log(B) + log(C). First, consider
the scalar case. Even here, the equality may fail. Let b = c = e(π−ǫ)i for ǫ small and
positive. Then

log bc = −2ǫi 6= (π − ǫ)i + (π − ǫ)i = log b + log c. (11.4)

Understanding of this behaviour is aided by using the unwinding number. We will

denote by arg z ∈ (−π, π] the principal value of the argument of z ∈ C. For z1 , z2 ∈ C
we have z1 z2 = elog z1 elog z2 = elog z1 +log z2 and hence, by (11.3),


log(z1 z2 ) = log elog z1 +log z2 = log z1 + log z2 − 2πiU(log z1 + log z2 ). (11.5)

Note that since Im log z ∈ (−π, π], we have U(log z1 + log z2 ) ∈ {−1, 0, 1}. Now
U(log z1 + log z2 ) = 0 precisely when | arg z1 + arg z2 | < π, so this is the condition for
log(z1 z2 ) = log z1 + log z2 to hold. As a check, the inequality in (11.4) is confirmed
by the fact that U(log b + log c) = U(2π − 2ǫ) = 1.
The next theorem generalizes this result from scalars to matrices. Recall from
Corollary 1.41 that if X and Y commute then for each eigenvalue λj of X there is an
eigenvalue µj of Y such that λj + µj is an eigenvalue of X + Y . We will call µj the
eigenvalue corresponding to λj .

Theorem 11.3. Suppose B, C ∈ Cn×n both have no eigenvalues on R− and that

BC = CB. If for every eigenvalue λj of B and the corresponding eigenvalue µj of C,

| arg λj + arg µj | < π, (11.6)

then log(BC) = log(B) + log(C).

Proof. Note first that log(B), log(C), and log(BC) are all defined—by (11.6) in
the latter case. The matrices log(B) and log(C) commute, since B and C do. By
Theorem 10.2 we therefore have

elog(B)+log(C) = elog(B) elog(C) = BC.

11.1 Basic Properties 271

Thus log(B) + log(C) is some logarithm of BC. The imaginary part of the jth
eigenvalue of log(B) + log(C) is

Im(log λj + log µj ) = arg λj + arg µj ∈ (−π, π)

by (11.6), so log(B) + log(C) is the principal logarithm of BC.

The following variant of Theorem 11.3 may be easier to apply in some situations.
We need to define the open half-plane associated with z = ρeiθ , which is the set of
complex numbers w = ζeiφ such that −π/2 < φ − θ < π/2. Figure 11.1 gives a
pictorial representation of condition (b) of the result.

Theorem 11.4 (Cheng, Higham, Kenney, and Laub). Let B, C ∈ Cn×n . Suppose
that A = BC has no eigenvalues on R− and
(a) BC = CB,
(b) every eigenvalue of B lies in the open half-plane of the corresponding eigen-
value of A1/2 (or, equivalently, the same condition holds for C).
Then log(A) = log(B) + log(C).
Proof. As noted above, since B and C commute there is a correspondence between
the eigenvalues a, b, and c (in some ordering) of A, B, and C: a = bc. Express these
eigenvalues in polar form as

a = αeiθ , b = βeiφ , c = γeiψ , −π < θ, φ, ψ ≤ π.

Since A has no eigenvalues on R− ,

−π < θ < π. (11.7)

The eigenvalues of B lie in the open half-planes of the corresponding eigenvalues of

A1/2 , that is,
π θ π
− <φ− < . (11.8)
2 2 2
From (11.7) and (11.8) it follows that −π < φ < π. Now a = bc implies θ = φ+ψ+2kπ
for some integer k. Subtracting one half (11.7) from (11.8) gives −π < φ − θ < π,
that is, −π < ψ + 2kπ < π, and since −π < ψ ≤ π we must have k = 0, so that
−π < ψ < π and θ = φ + ψ. Thus, by (11.7), φ + ψ ∈ (−π, π) and so the result holds
by Theorem 11.3.
In the terminology of Theorem 11.4, the reason for the behaviour in (11.4) is that
b and c are equal to a nonprincipal square root of a = bc and hence are not in the
half-plane of a1/2 .
A special case of Theorems 11.2–11.4 is the relation log(A) = 2 log(A1/2 ), which
is the basis of the inverse scaling and squaring method described in Section 11.5.
Finally, it is worth noting that the difference between log(XY ) and log(X)+log(Y )
is described for X and Y sufficiently close to I by setting t = 1, A = log(X), and
B = log(Y ) in the Baker–Campbell–Hausdorff formula (10.5) and taking the log of
both sides, giving

log(XY ) = log(X) + log(Y ) + [log(X), log(Y )]/2 + higher order terms.

The “error”, however, is expressed in terms of log(X) and log(Y ) rather than X
and Y .
272 Matrix Logarithm

λ
B λ1/2
A

Im λ

Re λ

Figure 11.1. Illustration of condition (b) of Theorem 11.4, which requires every eigenvalue of
1/2
B (λB ) to lie in the open half-plane (shaded ) of the corresponding eigenvalue of A1/2 (λA ).

11.2. Conditioning
From Theorem 3.5 we know that the Fréchet derivative of the logarithm is the
in-
verse of the Fréchet derivative of the exponential, so that Lexp log(A), Llog (A, E) =
R1
E. From (10.15) we have Lexp (A, E) = 0 eA(1−s) EeAs ds, and hence L(A, E) =
R1
Llog (A, E) satisfies E = 0 elog(A)(1−s) L(A, E)elog(A)s ds, that is (using Theorem 11.2),
Z 1
E= A1−s L(A, E)As ds. (11.9)
0

A different representation of the Fréchet derivative can be obtained from Theo-

rem 11.1 (see Problem 11.2):
Z 1

−1
−1
L(A, E) = t(A − I) + I E t(A − I) + I dt. (11.10)
0
R1
−2 
On setting E = I we obtain L(A, I) = 0 t(A − I) + I dt = −(A − I)−1 (t(A −
1
I) + I)−1 0 = A−1 . Hence kL(A)k ≥ kA−1 k and thus
κ(A)
κlog (A) ≥ .
k log(A)k
(Note that taking norms in (11.9) gives the weaker bound kL(A)k ≥ kAk−1 .) It is
reasonable to guess that equality will be achieved in this lower bound for any normal
matrix. However, this is not the case. From the general lower bound in Theorem 3.14
we have
!
1 | log λ − log µ|
kL(A)k ≥ max |f [λ, µ]| = max max , max . (11.11)
λ,µ∈Λ(A) λ∈Λ(A) |λ| λ,µ∈Λ(A) |λ − µ|
λ6=µ
11.3 Series Expansions 273

For a normal matrix and the Frobenius norm this inequality is an equality (Corol-
lary 3.16). But even for normal A, the right-hand side generally exceeds maxλ∈Λ(A) |λ|−1 .
For Hermitian positive definite matrices we do, however, have the equality kL(A)kF =
maxλ∈Λ(A) |λ|−1 = kA−1 k2 .
The lower bound (11.11) is large in two situations: if A has an eigenvalue of small
modulus or if A has a pair of complex eigenvalues lying close to, but on opposite sides
of, the negative real axis; in both cases A is close to a matrix for which the principal
logarithm is not defined.

11.3. Series Expansions

One way to obtain the matrix logarithm is via series expansions, of which there are
several variants. We begin with scalars. By integrating (1 + t)−1 = 1 − t + t2 − t3 + · · ·
between 0 and x we obtain Mercator’s series (1668),

x2 x3 x4
log(1 + x) = x − + − + ···, (11.12)
2 3 4
which converges for |x| < 1 (and in fact at every x on the unit circle except x = −1).
Replacing x by −x we have

x2 x3 x4
log(1 − x) = −x − − − − ···
2 3 4
and subtracting from the first equation gives
 
1+x x3 x5
log =2 x+ + + ··· . (11.13)
1−x 3 5

This is Gregory’s series (1668). Writing

1−x 1−y
y := ⇐⇒ x=
1+x 1+y

we have  2k+1
∞
X 1 1−y
log y = −2 . (11.14)
2k + 1 1+y
k=0

This series converges when |1 − y|/|1 + y| < 1, that is, Re y > 0.

For matrices the corresponding series are

A2 A3 A4
log(I + A) = A − + − + ···, ρ(A) < 1 (11.15)
2 3 4
and
∞
X 1
2k+1
log(A) = −2 (I − A)(I + A)−1 , min Re λi (A) > 0. (11.16)
2k + 1 i
k=0

The Gregory series (11.16) has a much larger region of convergence than (11.15), and
in particular converges for any Hermitian positive definite A. However, convergence
can be slow if A is not close to I.
274 Matrix Logarithm

11.4. Padé Approximation

The diagonal Padé approximants rm to log(1 + x) have two useful explicit represen-
tations. First is the continued fraction expansion
c1 x
rm (x) = , (11.17a)
c2 x
1+
c3 x
1+
c2m−2 x
1 + ··· +
c2m−1 x
1+
1 + c2m x
j j
c1 = 1, c2j = , c2j+1 = , j = 1, 2, . . . , (11.17b)
2(2j − 1) 2(2j + 1)

which is the truncation of an infinite continued fraction for log(1 + x). Second is the
partial fraction form
m (m)
X αj x
rm (x) = (m)
, (11.18)
j=1 1 + βj x
(m) (m)
where the αj are the weights and the βj the nodes of the m-point rule on [0, 1].
Codes for computing these weights and nodes, which the theory of Gaussian quadra-
ture guarantees are real, are given in [142, , App. 2], [206, ], [208, ,
Sec. 3.1], [478, , Sec. 4.6].
An interesting connection is that rm (A − I) is produced by applying the m-point
Gauss–Legendre quadrature rule to (11.1) [154, , Thm. 4.3].
The logarithm satisfies log(1 + x) = − log(1/(1 + x)). The diagonal Padé approx-
imant satisfies the corresponding identity.

Lemma 11.5 (Kenney and Laub). The diagonal Padé approximant rm to log(1 + x)
satisfies rm (x) = −rm (−x/(1 + x)).
Proof. The result is a special case of a more general invariance result applying
to an arbitrary function under an origin-preserving linear fractional transformation
of its argument [38, , Thm. 9.1], [39, , Thm. 1.5.2].
The next result states the important fact that the error in matrix Padé approx-
imation is bounded by the error in scalar Padé approximation at the norm of the
matrix.

Theorem 11.6 (Kenney and Laub). For kXk < 1 and any subordinate matrix norm,
 
krm (X) − log(I + X)k ≤ rm (−kXk) − log(1 − kXk). (11.19)

The bound (11.19) is sharp in that it is exact for nonpositive scalar X and hence
for nonpositive diagonal X in any p-norm.
Several possibilities exist for evaluating rm (X) = qm (X)−1 pm (X):
• Horner’s method applied to pm and qm (Algorithm 4.2), followed by solution of
qm rm = pm ;
• the Paterson–Stockmeyer method applied to pm and qm (see Section 4.4.3),
followed by solution of qm rm = pm ;
11.5 Inverse Scaling and Squaring Method 275

• the continued fraction (11.17) evaluated top-down (Algorithm 4.9) or bottom-up

(Algorithm 4.10);

• the partial fraction expansion (11.18).

The choice between these methods is a compromise between computational cost and
numerical stability. For the first two approaches it is crucial to establish that the
denominator polynomial is not too ill conditioned. The next result provides a bound.

Lemma 11.7 (Kenney and Laub). For kXk < 1 and any subordinate matrix norm,
the denominator qm of the diagonal Padé approximant rm satisfies

qm (kXk)
κ(qm (X)) ≤ . (11.20)
qm (−kXk)

In the detailed analysis of Higham [275, ] the partial fraction method emerges
as the best overall method for evaluating rm . Its cost is m solutions of multiple
right-hand side linear systems. Its numerical stability is governed by the condition of
the linear systems that are solved, and for a stable linear system solver the normwise
relative error will be bounded approximately by d(m, n)uφm , where
(m)
(m)
1 + |βj |kXk
φm = max κ I + βj X ≤ max (m)
. (11.21)
j j 1− |βj |kXk

(m)
Since βj ∈ (0, 1), φm is guaranteed to be of order 1 provided that kXk is not too
close to 1. An advantage for parallel computation is that the m terms in (11.18) can
be evaluated in parallel.
So far, we have concentrated on Padé approximants to log(1 + x). Motivated by

the Gregory series (11.13) we might also consider approximating log (1 + x)/(1 − x) .
It turns out that these two sets of Padé approximants are closely related.

Lemma 11.8.

The Padé approximants rm (x) to log(1 + x) and sm (x) to log (1 +
x)/(1 − x) are related by rm (x) = sm (x/(x + 2)).

Proof. The proof is very similar to that of Lemma 11.5.

Analogously to Theorem 11.6, the error in sm satisfies the bound



sm (X) − log (I + X)(I − X)−1 ≤ sm (kXk) − log (1 + kXk)/(1 − kXk ,

provided that kXk < 1. This inequality is obtained in the same way as (11.19) from
a more general result in [341, , Cor. 4] holding for a class of hypergeometric
functions.

11.5. Inverse Scaling and Squaring Method

One of the best methods for computing the matrix logarithm is a generalization of
the method that Briggs used in the 17th century to compute tables of logarithms of
numbers. Briggs knew that for positive real numbers a and b, log(ab) = log a + log b
and hence that log a = 2 log a1/2 . Using this equation repeatedly he obtained log a =
k
2k log a1/2 . Briggs also knew that log(1+x) ≈ x is a good approximation for small x.
276 Matrix Logarithm

k
He used this approximation with a1/2 = 1+x and a sufficiently large k, and multiplied
by the scale factor needed to obtain logarithms to base 10. His approximation was
k
therefore log10 a ≈ 2k · log10 e · (a1/2 − 1).
For a matrix A with no eigenvalues on R− , the analogue of Briggs’ idea is to use
the identity, from Theorem 11.2,
k

log(A) = 2k log A1/2 . (11.22)
k
The integer k is to be chosen so that log(A1/2 ) is easy to approximate. We will use a
k
Padé approximant of log(1 + x), so we require A1/2 sufficiently close to the identity,
k
which holds for large enough k since limk→∞ A1/2 = I. This method is called the
inverse scaling and squaring method by analogy with the scaling and squaring method
for the matrix exponential.
At this point it is worth pointing out a danger in implementing Briggs’ origi-
nal method in floating point arithmetic. Suppose we are working in IEEE double
precision arithmetic and wish to obtain the logarithm to full accuracy of about 16
k
significant decimal digits. Then we will need |a1/2 − 1| ≈ 10−8 (as can be seen from
k
Theorem 4.8). The subtraction a1/2 − 1 must therefore suffer massive cancellation,
k
with about half of the significant digits in a1/2 being lost. This was not a problem
for Briggs, who calculated to 30 decimal digits in order to produce a table of 14-digit
logarithms. And the problem can be avoided by using higher order approximations to
k
log a1/2 . However, in the matrix context a value of k sufficiently large for the matrix
as a whole may be too large for some particular submatrices and could lead to dam-
aging subtractive cancellation. This is an analogue of the overscaling phenomenon
discussed at the end of Section 10.3 in the context of the scaling and squaring method
for the exponential.
The design of an algorithm depends on what kinds of operation we are prepared
to carry out on the matrix. We consider first the use of a Schur decomposition, which
reduces the problem to that of computing the logarithm of a triangular matrix, and
then consider working entirely with full matrices.

11.5.1. Schur Decomposition: Triangular Matrices

If an initial Schur decomposition is carried out then the problem becomes that of
computing the logarithm of a triangular matrix T ∈ Cn×n . Our task is to decide how
many square roots to take and what degree of Padé approximant to use, with the
aim of finding a compromise between minimal cost and maximal accuracy. Two key
facts are that a square root of a triangular matrix can be computed in n3/3 flops (by
Algorithm 6.3) and that rm (T ) can be evaluated by the partial fraction expansion
(11.18) in mn3/3 flops (see Table C.1). Hence an extra square root is worth taking
if it reduces the degree m of the required Padé approximant by more than 1. The
required m will be determined from (11.19).
Table 11.1 reports, for a range of m from 1 to 64 and to three significant figures,
the largest kXk, denoted by θm , such that the bound (11.19) guarantees krm (X) −
log(I + X)k ≤ u = 2−53 ≈ 1.1 × 10−16 . By Problem 10.8, for m < ∼ 12 the absolute
error bound (11.19) with kXk ≤ θm is also essentially a relative error bound. The
table also reports the bound in (11.20) for κ(qm (X)) and the bound in (11.21) for
(m)

φm = maxj κ I + βj X , both for kXk = θm . Note first that the bounds for
κ(qm (X)) grow rapidly for m ≥ 10, which is a concern if Horner’s method is used
11.5 Inverse Scaling and Squaring Method 277

Table 11.1. Maximal values θm of kXk such that the bound (11.19) ensures krm (X) − log(I +
X)k does not exceed u = 2−53 , along with upper bound (11.20) for κ(qm (X)) and upper bound
(11.21) for φm , both with kXk = θm .

m 1 2 3 4 5 6 7 8 9
θm 1.10e-5 1.82e-3 1.62e-2 5.39e-2 1.14e-1 1.87e-1 2.64e-1 3.40e-1 4.11e-1
κ(qm ) 1.00e0 1.00e0 1.05e0 1.24e0 1.77e0 3.09e0 6.53e0 1.63e1 4.62e1
φm 1.00e0 1.00e0 1.03e0 1.11e0 1.24e0 1.44e0 1.69e0 2.00e0 2.36e0

m 10 11 12 13 14 15 16 32 64
θm 4.75e-1 5.31e-1 5.81e-1 6.24e-1 6.62e-1 6.95e-1 7.24e-1 9.17e-1 9.78e-1
κ(qm ) 1.47e2 5.07e2 1.88e3 7.39e3 3.05e4 1.31e5 5.80e5 >1e16 >1e16
φm 2.76e0 3.21e0 3.71e0 4.25e0 4.84e0 5.47e0 6.14e0 2.27e1 8.85e1

to evaluate rm . However, we will use the partial fraction expansion, which the φm
values show will deliver a fully accurate evaluation for all the m of interest.
In interpreting the data in the table we need to know the effect of taking a square
root of T on the required
m. This can be
determined irrespective of the triangularity
1/2k+1 1/2k+1 1/2k k
of T . Since I − A I+A = I−A and A1/2 → I as k → ∞ it
follows that for large k,
k+1 1 k
kI − A1/2 k≈ kI − A1/2 k, (11.23)
2
k
so that once A1/2 has norm approximately 1 a further square root should approxi-
mately halve the distance to the identity matrix.11
Since θm /2 < θm−2 for m > 7, to minimize the cost we should keep taking
square roots until kXk ≤ θ7 , since each such square root costs only half the resultant
saving (at least two multiple right-hand side solves) in the cost of evaluating the Padé
approximant. The question is whether this choice needs to be moderated by stability
considerations. We have already noted from the φm values in Table 11.1 that the
evaluation of rm for such X is as accurate as can be hoped. The only danger in
scaling X to have norm at most θ7 , rather than some larger value, is that the greater
number of square roots may increase the error in the matrix whose Padé approximant
is computed. However, the later square roots are of matrices close to I, which are
well conditioned in view of (6.2), so these extra square roots should be innocuous.
We are now ready to state an algorithm. Rather than tailor the algorithm to the
particular precision, we express it in such a way that the same logic applies whatever
the precision, with changes needed only to the integer constants and the θi .

Algorithm 11.9 (inverse scaling and squaring algorithm with Schur decomposition).
Given A ∈ Cn×n with no eigenvalues on R− this algorithm computes X = log(A) via
a Schur decomposition and the inverse scaling and squaring method. It uses the con-
stants θi given in Table 11.1. The algorithm is intended for IEEE double precision
arithmetic.

1 Compute a (complex) Schur decomposition A = QT Q∗ .

11 Briggs cleverly exploited (11.23) by computing differences that enabled him to reduce the number

of square root evaluations needed for his logarithm tables.

278 Matrix Logarithm

2 k = 0, p = 0
3 while true
4 τ = kT − Ik1
5 if τ ≤ θ7
6 p=p+1
7 j1 = min{ i : τ ≤ θi , i = 3: 7 }
8 j2 = min{ i : τ /2 ≤ θi , i = 3: 7 }
9 if j1 − j2 ≤ 1 or p = 2, m = j1 , goto line 14, end
10 end
11 T ← T 1/2 using Algorithm 6.3.
12 k =k+1
13 end
14 Evaluate U = rm (T − I) using the partial fraction expansion (11.18).
15 X = 2k QU Q∗

Cost: 25n3 flops for the Schur 3

3 decomposition plus (k + m)n /3 flops for U and
3 k
3n to form X: about 30 + 3 n flops in total.
Assuming that (11.23) is a reasonable approximation and that k > 0, Algo-
rithm 11.9 will take m = 5, 6, or 7 (see Problem 11.5). In the unlikely event that
(11.23) is not a good approximation, the algorithm still uses a degree m appropriate
to the desired accuracy but may take one more square root than necessary (the test
“p = 2” on line 9 limits the number of unnecessary square roots).
At the start
 of this section we mentioned the danger of overscaling. Writing
T = T11 T12
0 T22 in Algorithm 11.9, a large kT12 k may cause more square roots to be
taken than are necessary for accurate computation of the logarithms of the diagonal
blocks. This phenomenon is mitigated in the Schur–Parlett algorithm of Section 11.6.2
below, which applies inverse scaling and squaring to diagonal blocks rather than the
whole of T .

11.5.2. Full Matrices

Suppose that we are not willing to Schur factorize A. Square roots must therefore be
computed using one of the Newton methods from Section 6.3. The economics of the
method are now quite different, because the square roots are relatively expensive to
compute and the cost depends on the number of iterations required.
For any of the Newton iterations for the square root, the number of iterations
required will vary from typically up to 10 on the first few iterations to 4 or 5 for the
last few. From Table 6.1 we see that the cost of one iteration is 4n3 flops for the DB
and product DB iterations. Evaluating the partial fraction expansion (11.18) costs
8mn3/3 flops. So a later square root costs about 16n3 flops, while decreasing m by
1 decreases the cost of evaluating rm by 8n3/3 flops. Therefore once we are in the
regime where Padé approximation can be applied it is likely to be worth taking a
further square root only if it allows m to be decreased by at least 7. From Table 11.1
and further θm values not shown here we find that such a decrease is achieved when
kXk ≥ θ16 (assuming that (11.23) is a good approximation). Our algorithm predicts
whether another square root is worthwhile by assuming that the next Newton iteration
would require the same number of iterations as the last one and thereby uses the values
of θm and kXk to compare the total cost with and without a further square root.
11.6 Schur Algorithms 279

Algorithm 11.10 (inverse scaling and squaring algorithm). Given A ∈ Cn×n with
no eigenvalues on R− this algorithm computes X = log(A) by the inverse scaling
and squaring method. It uses the constants θi given in Table 11.1. The algorithm is
intended for IEEE double precision arithmetic.

1 k = 0, it0 = 5, p = 0
2 while true
3 τ = kA − Ik1
4 if τ < θ16
5 p=p+1
6 j1 = min{ i : τ ≤ θi , i = 3: 16 }
7 j2 = min{ i : τ /2 ≤ θi , i = 3: 16 }
8 if 2(j1 − j2 )/3 ≤ itk or p = 2, m = j1 , goto line 13, end
9 end
10 A ← A1/2 using the scaled product DB iteration (6.29); let itk+1 be
the number of iterations required.
11 k =k+1
12 end
13 Evaluate Y = rm (A − I) using the partial fraction expansion (11.18).
14 X = 2k Y
Pk

Cost: i=1 iti 4n3 + 8mn3/3 flops.

11.6. Schur Algorithms

We now describe two other algorithms that make use of a Schur decomposition A =
QT Q∗ .

11.6.1. Schur–Fréchet Algorithm

Kenney and Laub [348, ] develop an algorithm for computing log(A) that com-
putes the off-diagonal blocks of log(T ) via Fréchet derivatives, using Algorithm 11.12
below. The algorithm is entirely analogous to the Schur–Fréchet algorithm for the
exponential described in Section 10.4.2. For the details see [348, ].

11.6.2. Schur–Parlett Algorithm

Algorithm 9.6, the general Schur–Parlett algorithm, is not applicable to the matrix
logarithm because the Taylor series of the logarithm has a finite (and small) radius
of convergence. However, we can adapt it to the logarithm by providing a means to
compute the logarithms of the diagonal blocks.
The only
 difficulty

in computing the logarithm of a 2 × 2 upper triangular matrix,
F = log t11 t12
0 t22 , is in obtaining an accurate (1,2) element. The formula f12 =
t12 (log λ2 − log λ1 )/(λ2 − λ1 ) from (4.16) can suffer damaging subtractive cancellation
when λ1 ≈ λ2 . To avoid it we write
 
λ2
log λ2 − log λ1 = log + 2πiU(log λ2 − log λ1 )
λ1
 
1+z
= log + 2πiU(log λ2 − log λ1 ),
1−z
280 Matrix Logarithm

where U is the unwinding number (11.2) and z = (λ2 − λ1 )/(λ2 + λ1 ). Using the
hyperbolic arc tangent, defined by
 
1 1+z
atanh(z) := log , (11.24)
2 1−z

we have
2 atanh(z) + 2πiU(log λ2 − log λ1 )
f12 = t12 . (11.25)
λ2 − λ1
It is important to note that atanh can be defined in a different way, as recommended
in [81, ] and [327, ]:
1
atanh(z) := (log(1 + z) − log(1 − z)). (11.26)
2
With this definition a slightly more complicated formula than (11.25) is needed:
 
2 atanh(z) + 2πi U(log λ2 − log λ1 ) + U(log(1 + z) − log(1 − z))
f12 = t12 . (11.27)
λ2 − λ1
MATLAB’s atanh function is defined by (11.24). Of course, the use of (11.25) or
(11.27) presupposes the availability of an accurate atanh implementation. Our overall
formula is as follows, where we avoid the more expensive atanh formula when λ1 and
λ2 are sufficiently far apart:

 t12

 , λ1 = λ 2 ,

 λ1
f12 = log λ2 − log λ1 (11.28)

 t12 , |λ1 | < |λ2 |/2 or |λ2 | < |λ1 |/2 ,

 λ 2 − λ 1

(11.25) or (11.27), otherwise.

Algorithm 11.11 (Schur–Parlett algorithm for matrix logarithm). Given A ∈ Cn×n

this algorithm computes F = log(A) via a Schur decomposition.

1 Execute Algorithm 9.6 with line 7 replaced by

“7 Evaluate Fii = log(Tii ) directly if Tii is 1 × 1, using (11.28) if Tii is 2 × 2,
or else by Algorithm 11.9 (inverse scaling and squaring method).”

Algorithm 11.11 is our preferred alternative to Algorithm 11.9.

11.7. Numerical Experiment

We describe an experiment analogous to that for the matrix exponential in Sec-
tion 10.5. The matrices are the exponentials of most of those used in the latter
experiment, supplemented with matrices from the log(A) literature and some 2 × 2
triangular matrices; all are size 10 × 10 or less.
The methods are
1. MATLAB 7.5 (R2007b)’s logm, denoted logm old, which implements a more
basic version of Algorithm 11.11 that does not treat 2 × 2 diagonal blocks spe-
cially and uses a version of Algorithm 11.9 that employs a Padé approximant
k
of fixed degree m = 8 that is used once kI − T 1/2 k ≤ 0.25;
11.8 Evaluating the Fréchet Derivative 281

0
logm_old
10 logm
logm_iss_schur
logm_iss

−5
10

−10
10

−15
10

0 10 20 30 40 50 60

Figure 11.2. Normwise relative errors for MATLAB’s logm, logm old, logm iss schur, and
logm iss; the solid line is κlog (A)u.

2. the logm function in MATLAB 7.6 (R2008a) onwards, which implements Algo-
rithm 11.11;
3. logm iss schur, which implements Algorithm 11.9;
4. logm iss, which implements Algorithm 11.10. The convergence test for the
scaled product DB iteration is based on kMk − Ik.

Figure 11.2 shows the normwise relative errors kXb − log(A)kF /k log(A)kF , where
b
X is the computed logarithm and log(A) is computed at 100 digit precision, along with
κlog (A)u, where κlog = condrel (log, A) is computed with the aid of Algorithm 11.12.
Figure 11.3 presents the same data in the form of a performance profile.
Several observations can be made.

• logm is clearly superior to logm old. Further investigation shows that both
the special treatment of 2 × 2 diagonal blocks and the use of Algorithm 11.9
contribute to the improvement in accuracy.
• logm iss schur is even more markedly superior to logm iss, showing the ben-
efits for accuracy of applying the inverse scaling and squaring method to trian-
gular matrices.

11.8. Evaluating the Fréchet Derivative

The relationship between the Fréchet derivatives of the exponential and the logarithm
can be summarized in the equation
Z 1
L= eA(1−s) M eAs ds,
0
282 Matrix Logarithm

0.9

0.8

0.7

0.6
p
0.5

0.4

0.3
logm
logm_iss_schur
0.2
logm_old
logm_iss
0.1
1 2 3 4 5 6 7 8 9 10
α

Figure 11.3. Same data as in Figure 11.2 presented as a performance profile.

where L is the Fréchet derivative of the exponential at A in the direction M , and M

is the Fréchet derivative of the logarithm at eA in the direction L (see (10.15) and
(11.9)). The formula (10.49) therefore translates into

T

−1 T
vec(Llog (A, E)) = 2τ 1
2 [X ⊕ (−X)] (A ⊕ A)−1 vec(E), X = log(A), (11.29)

where τ (x) = tanh(x)/x and kXk ≤ 1 is assumed for some consistent matrix norm.
That the inverses in (11.29) exist is verified in Problem 11.8.
In order to use this formula we first need to ensure that X = log(A) has norm
s
at most 1. This is achieved by taking repeated square roots: if kI − A1/2 k ≤ 1 −
s
1/e then k log(A1/2 )k ≤ 1 (see Problem 11.4), and so we can apply the formula to
s
A1/2 . To take account of the square root phase we note that applying the chain rule

(Theorem 3.4) to log(A) = 2 log(A1/2 ) gives Llog (A, E) = 2Llog A1/2 , Lx1/2 (A, E) .
s
Hence, by recurring this relation, we have Llog (A, E) = Llog (A1/2 , Es ), where

E0 = 2s E, (11.30a)
i
1/2 1/2i
A Ei + E i A = Ei−1 , i = 1: s. (11.30b)

We obtain an algorithm for evaluating Llog (A, E) by using this Sylvester recurrence
and reversing the steps in the derivation of Algorithm 10.27.

Algorithm 11.12 (Fréchet derivative of matrix logarithm). Given A ∈ Cn×n this

algorithm evaluates the Fréchet derivative L = L(A, E) of the matrix logarithm
via (11.29), using scaling and squaring and the [8/8] Padé approximant to τ (x) =
tanh(x)/x. The parameters {αi , βi }8i=1 are given in Table 10.5. The algorithm is
intended for IEEE double precision arithmetic.
s s
1 B = A1/2 with s ≥ 0 a minimal integer such that kI − A1/2 k1 ≤ 1 − 1/e.
11.9 Notes and References 283

2 E0 = 2s E
3 for i = 1: s
i i
4 Solve for Ei the Sylvester equation A1/2 Ei + Ei A1/2 = Ei−1 .
5 end
6 Solve for G9 the Sylvester equation BG9 + G9 B = Es .
7 X = log(B)
8 for i = 8: −1: 1
9 Solve for Gi the Sylvester equation (I + X/αi )Gi + Gi (I − X/αi ) =
(I + X/βi )Gi+1 + Gi+1 (I − X/βi ).
10 end
11 L = 2G0
Cost: 16M , s matrix square roots, 1 matrix logarithm, and the solution of s + 9
Sylvester equations.
In practice, it is advisable to combine the square root computations in line 1 with
the loop in lines 3–5 in order to avoid having to store the intermediate square roots.
As with Algorithm 10.27 for the exponential, a Schur reduction of A to triangular
form should be exploited in conjunction with Algorithm 11.12 to reduce the cost.
Algorithm 11.12 can combined with Algorithm 11.9, with the number of square
roots determined by Algorithm 11.9, which has the more stringent requirement on
kT − Ik.
It is also possible to estimate the Fréchet derivative by quadrature. For one
approach, see Dieci, Morini, and Papini [154, ].
An obvious analogue of Algorithm 10.28 can be formulated for computing and
estimating the condition number of the matrix logarithm.

11.9. Notes and References

The integral representation in Theorem 11.1 was obtained by Richter [490, ] for
A having distinct eigenvalues and by Wouk [617, ] without this restriction.
The unwinding number was introduced by Corless and Jeffrey [116, ] and is
further investigated by Bradford, Corless, Davenport, Jeffrey, and Watt [81, ].
Theorem 11.4 is from Cheng, Higham, Kenney, and Laub [108, ].
Von Neumann [604, , pp. 11–12] derives the sufficient conditions kB − Ik < 1,
kC − Ik < 1, and kBC − Ik < 1 for log(BC) = log(B) + log(C) when B and C
commute. These are much stronger conditions than those in Theorems 11.3 and
11.4 and stem from the fact that von Neumann defined the matrix logarithm via the
Mercator series (11.15).
The formula (11.10) for the Fréchet derivative is due to Dieci, Morini, and Papini
[154, ].
Good references for history of the Mercator series (11.12), the Gregory series
(11.13), and related series are Goldstine [222, ] and Hairer and Wanner [238,
].
The continued fraction expansion (11.17) can be found in Baker and Graves-Morris
[39, , p. 174].
Lemma 11.5, Theorem 11.6, and Lemma 11.7 are from Kenney and Laub [341,
].
Henry Briggs (1556–1630) was a contemporary of John Napier (1550–1617), the
inventor of logarithms. More details of Briggs’ calculations can be found in Goldstine
[222, ] and Phillips [472, ].
284 Matrix Logarithm

The inverse scaling and squaring method was proposed by Kenney and Laub [340,
] for use with a Schur decomposition. It was further developed by Cheng, Higham,
Kenney, and Laub [108, ] in a transformation-free form that takes as a parameter
the desired accuracy. The two key ideas therein are prediction of the most efficient
Padé degree, based on the bound (11.19), and the use of an incomplete form of the
product DB iteration, with analysis to show how early the iteration can be terminated
without compromising the overall accuracy; see Problem 11.7. The implementation in
s
Kenney and Laub [340, ] requires kXk = kI − A1/2 k ≤ 0.25 and uses fixed Padé
degree 8, while Dieci, Morini, and Papini [154, ] require kXk ≤ 0.35 and take
Padé degree 9. Cardoso and Silva Leite [95, ] propose a version of the inverse

scaling and squaring method based on Padé approximants to log (1 + x)/(1 − x) .
It explicitly forms a matrix Cayley transform and requires a matrix squaring at each
square root step, so its numerical stability is questionable and it appears to offer no
general cost advantages over the use of log(1 + x) (cf. Lemma 11.8).
The inverse scaling and squaring method is used by Arsigny, Commowick, Pennec,
and Ayache [19, ] to compute the logarithm of a diffeomorphism, which they
then use to compute the log-Euclidean mean (2.29) of three-dimensional geometrical
transformations.
The formula (11.25) can be found in Kahan and Fateman [329, ].
Section 11.8 is based on Kenney and Laub [348, ].
In connection with overscaling in the inverse scaling and squaring method, Dieci
and Papini [156, ] consider a block 2 × 2 upper triangular T = (Tij ) ∈ Cn×n and
obtain a bound for krm (T − I) − log(T )k of the form kT12 k multiplied by a function of
maxi=1,2 kTii − Ik (assumed less than 1) but having no direct dependence on kT − Ik.
This bound potentially allows a reduction in the number of square roots required
by the inverse scaling and squaring method compared with the use of (11.19), since
maxi=1,2 kTii − Ik may be less than 1 when kT − Ik greatly exceeds 1, due to a large
T12 . Related results are obtained by Cardoso and Silva Leite [95, ] for the Padé
approximants sm defined in Lemma 11.8. A difficulty in applying these bounds is
that it is not clear how best to choose the blocking. A more natural approach is
Algorithm 11.11, which automatically blocks the Schur form and applies the inverse
scaling and squaring method to the (usually small) diagonal blocks.
For Hermitian positive definite matrices, Lu [394, ] proposes an algorithm en-
tirely analogous to those of his described in Sections 6.10 and 10.7.1 for the square root
and the exponential, based on the linear partial fraction form of Padé approximants
to log(1 + x).

Problems
11.1. Show that for any λ not lying on R− the (principal) logarithm log(J(λ)) of a
Jordan block J(λ) ∈ Cm×m can be defined via the series (11.15). Thus explain how
(11.15) can be used to define log(A) for any A ∈ Cn×n having no eigenvalues on R− ,
despite the finite radius of convergence of the series.

11.2. Derive the formula (11.10) for the Fréchet derivative of the logarithm, using
the definition (3.6) and the integral (11.1).

11.3. Show that kI − Ak < 1 for some consistent matrix norm is sufficient for the
convergence of the Gregory series (11.16).
Problems 285

11.4. Let k·k be a consistent norm. Show that if kI −M k ≤ 1−1/e then k log(M )k ≤
1.
11.5. Show that if Algorithm 11.9 takes at least one square root and the approxi-
mation (11.23) is exact once kT − Ik1 ≈ θ7 then the algorithm will take m = 5, 6,
or 7.
11.6. Show that for A ∈ Cn×n with no eigenvalues on R− , log(A) = log(A/kAk) +
log(kAkI). This relation could be used prior to calling Algorithm 11.9 or Algo-
rithm 11.10 on the matrix A/kAk whose norm is 1. Is this worthwhile?
11.7. (Cheng, Higham, Kenney, and Laub [108, ]) Show that the iterates from
the product DB iteration (6.17) satisfy log(A) = 2 log(Xk ) − log(Mk ). Suppose we
terminate the iteration after k iterations and set X (1) = Xk , and now apply the
product DB iteration to X (1) , again for a finite number of iterations. Show that
continuing this process leads after s steps to

log(A) = 2s log(X (s) ) − log(M (1) ) − 2 log(M (2) ) − · · · − 2s−1 log(M (s) ), (11.31)

where X (i) and M (i) are the final iterates from the product DB iteration applied to
X (i−1) . The (unknown) log(M (i) ) terms on the right-hand side can be bounded using
Problem 11.4.
11.8. Let A ∈ Cn×n have no eigenvalues on R− . Let X = log(A) and assume kXk ≤ 1
for some consistent matrix norm.
(a) Show that AT ⊕ A is nonsingular.
Q∞
(b) Using the expansion τ (x)

= tanh(x)/x = k=1 (π 2 + x2 /k 2 )/(π 2 + 4x2 /(2k − 1)2 )
show that τ 21 [X T ⊕ (−X)] is nonsingular.
11.9. Let R be the upper triangular Cholesky factor of the n × n Pascal matrix
(see Section 6.11). Show that log(R) is zero except on the first superdiagonal, which
comprises 1, 2, . . . , n − 1. Thus for n = 4,
   
1 1 1 1 1 0 1 0 0 0
 0 1 2 3 4   0 0 2 0 0
   
log  0 0 1 3 6  =  0 0 0 3 0.
   
0 0 0 1 4 0 0 0 0 4
0 0 0 0 1 0 0 0 0 0

Explain why all logarithms of R are upper triangular and why this is the only real
logarithm of R.
b be an approximate logarithm of A. Derive a formula for the relative
11.10. Let X
b
backward error of X.
11.11. (Kenney) Let A ∈ Cn×n be nonsingular. Consider the iterations

Xk+1 = Xk − I + e−Xk A, (11.32)

1

Xk+1 = Xk + e−Xk A − A−1 eXk . (11.33)
2
Show that if X0 A = AX0 then (11.32) is Newton’s method for eX − A = 0 and
(11.33) is Newton’s method for (eX − A)(eX + A)−1 = 0, and that AXk = Xk A and
Xk Xk+1 = Xk+1 Xk for all k in both cases. Show that if X0 is a function of A then
286 Matrix Logarithm

(11.32) has local quadratic convergence to a primary logarithm of A while (11.33)

has local cubic convergence to a primary logarithm of A. Which of the two iterations
should be preferred in practice? Iteration (11.32) is used in multiple precision libraries
to compute the logarithm of a scalar to high precision [36, ], [528, ].

In 1614 John Napier (1550–1617), of Merchiston Castle, near (now in) Edinburgh,
published his book Mirifici Logarithmorum Canonis Descriptio,
in which he gives a table of logarithms and an account of how he computed them.
If anyone is entitled to use the word “logarithm” it is Napier, since he coined the word,
deriving it from the two Greek words
λóγoς (meaning “reckoning,” in this context) and
ὰριθµóς (meaning “number”).
— GEORGE M. PHILLIPS, Two Millennia of Mathematics:
From Archimedes to Gauss (2000)

Another common felony is to exploit either of the transformations

ln(uv) ⇋ ln u + ln v even when both u and v could be negative.
For example,
ln((−1)(−1)) → ln 1 → 0, whereas
ln(−1) + ln(−1) → πi + πi → 2πi.
— DAVID R. STOUTEMYER, Crimes and Misdemeanors
in the Computer Algebra Trade (1991)

Briggs must be viewed as one of the

great figures in numerical analysis.
— HERMAN H. GOLDSTINE, A History of Numerical Analysis
from the 16th through the 19th Century (1977)

For values of x close to zero,

’LNP1(x)’ returns a more accurate result than does ’LN(x+1)’.
Using LNP1 allows both the argument and the result to be near zero. . .
The calculator can express numbers within 10−449 of zero,
but within only 10−11 of 1.
— HP 48 Programmer’s Reference Manual (1990)

Pre-programmed functions let you perform

log, trig and exponential calculations
10 times faster than with a slide rule.
— The Hewlett-Packard HP-35 Scientific Pocket Calculator (1973)

A real Floquet factorization exists with T -periodic LFY (t, 0)

if and only if Φ ≡ Φ(T, 0) has a real logarithm.
— PIERRE MONTAGNIER, CHRISTOPHER C. PAIGE, and RAYMOND J. SPITERI,
Real Floquet Factors of Linear Time-Periodic Systems (2003)
Chapter 12
Matrix Cosine and Sine

We now turn our attention to the two most important trigonometric functions: the
cosine and the sine. We saw in Section 2.1 that the matrix sine and cosine arise in the
solution of the second order differential system (2.7). More general problems of this
type, with a forcing term f (t) on the right-hand side, arise from semidiscretization of
the wave equation and from mechanical systems without damping, and their solutions
can be expressed in terms of integrals involving the sine and cosine [514, ].
This chapter begins with a review of addition formulae and normwise upper and
lower bounds. Expressions for the Fréchet derivatives are then derived. A detailed
derivation is given of an algorithm for cos(A) that employs variable degree Padé ap-
proximation in conjunction with the double angle formula for the cosine. A numerical
illustration is given of the performance of that algorithm and two alternatives. Then
an algorithm is developed for computing cos(A) and sin(A) simultaneously that inter-
twines the cosine and sine double angle recurrences. Finally, preprocessing is briefly
discussed.

12.1. Basic Properties

The matrix cosine and matrix sine can be defined for all A ∈ Cn×n by
A2 A4 A6
cos(A) = I − + − + ···,
2! 4! 6!
A3 A5 A7
sin(A) = A − + − + ···.
3! 5! 7!
A fundamental relationship is the matrix analogue of Euler’s formula,

eiA = cos(A) + i sin(A), (12.1)

which yields
eiA + e−iA eiA − e−iA
cos(A) = , sin(A) = (12.2)
2 2i
and
cos2 (A) + sin2 (A) = I. (12.3)
iA iA
For real A, we can write cos(A) = Re e , sin(A) = Im e .
A number of basic properties are analogous to those for the matrix exponential.

Theorem 12.1 (addition formulae). For A, B ∈ Cn×n , the addition formulae

cos((A + B)t) = cos(At) cos(Bt) − sin(At) sin(Bt), (12.4)

sin((A + B)t) = sin(At) cos(Bt) + cos(At) sin(Bt) (12.5)

287
288 Matrix Cosine and Sine

hold for all t if and only if AB = BA.

Proof. The result follows from Theorem 10.2 on using (12.1).
As a special case of the theorem we have the double angle formulae
cos(2A) = 2 cos(A)2 − I, (12.6)
sin(2A) = 2 sin(A) cos(A). (12.7)
The addition formulae hold for all A and B in the Kronecker sense.

Theorem 12.2 (Kronecker addition formulae). Let A ∈ Cn×n and B ∈ Cm×m . Then
f (A ⊗ I) = f (A) ⊗ I and f (I ⊗ B) = I ⊗ f (B) for f = cos, sin, and
cos(A ⊕ B) = cos(A) ⊗ cos(B) − sin(A) ⊗ sin(B), (12.8)
sin(A ⊕ B) = sin(A) ⊗ cos(B) + cos(A) ⊗ sin(B). (12.9)
Proof. The result follows from Theorem 10.9 on using (12.2).
The next result provides some easily evaluated upper and lower bounds on the
norm of the sine and cosine.

Theorem 12.3 (norm bounds). For A ∈ Cn×n and any subordinate matrix norm we
have
2 − cosh(kAk) ≤ 2 − cosh(kA2 k1/2 )
≤ k cos(A)k ≤ cosh(kA2 k1/2 ) ≤ cosh(kAk), (12.10)
kAk3
kAk −
≤ k sin(A)k ≤ sinh(kAk). (12.11)
6 1 − kAk2 /20
P∞ P∞
Proof. We have the bound k cos(A)k ≤ k=0 kA2k k/(2k)! ≤ k=0 kA2 kk /(2k)! =
cosh(kA2 k1/2 ). Similarly,
∞
X ∞
X
k cos(A)k ≥ 1 − kA2k k/(2k)! ≥ 1 − kA2 kk /(2k)!
k=1 k=1
2 1/2
= 1 − (cosh(kA k ) − 1) = 2 − cosh(kA2 k1/2 ).
This gives (12.10), since kA2 k1/2 ≤ kAk and hence cosh(kA2 k1/2 ) ≤ cosh(kAk). The
upper bound for k sin(A)k is straightforward. For the lower bound we have
X∞  
kA2k+1 k kA3 k kA2 k kA4 k
k sin(A)k ≥ kAk − ≥ kAk − 1+ + + ···
(2k + 1)! 6 4·5 4·5·6·7
k=1
  2
!
kA3 k kA2 k kA2 k kAk3
≥ kAk − 1+ + + · · · ≥ kAk −
.
6 4·5 4·5 6 1 − kAk2 /20

Since kA2 k ≤ kAk2 can be an arbitrarily weak inequality for nonnormal A (con-
sider involutory matrices, for example), the inner bounds in (12.10) can be smaller
than the outer ones by an arbitrary factor. Numerical methods are likely to form
A2 when evaluating cos(A), since cos is an even function, enabling the tighter bound
to be evaluated at no extra cost (and in some cases, such as (2.8), A2 is the given
matrix).
12.2 Conditioning 289

12.2. Conditioning
By using (10.15) and (12.2) we can obtain expressions for the Fréchet derivatives of
the cosine and sine functions:
Z 1
 
Lcos (A, E) = − cos(A(1 − s))E sin(As) + sin(A(1 − s))E cos(As) ds, (12.12)
0
Z 1 
Lsin (A, E) = cos(A(1 − s))E cos(As) − sin(A(1 − s))E sin(As) ds. (12.13)
0

If A and E commute then these formulae reduce to Lcos (A, E) = −E sin(A) =

− sin(A)E and Lsin (A, E) = E cos(A) = cos(A)E. Setting E = I we obtain (re-
calling (3.7)) the relations kLcos (A)k ≥ k sin(A)k and kLsin (A)k ≥ k cos(A)k, and
hence the relative condition numbers satisfy

k sin(A)k kAk k cos(A)k kAk

κcos (A) ≥ , κsin (A) ≥ . (12.14)
k cos(A)k k sin(A)k

Just as for the exponential and the logarithm we can obtain explicit expressions
for the Fréchet derivatives with the aid of vec and the Kronecker product.

Theorem 12.4 (Kronecker representation of Fréchet derivatives). For A, E ∈ Cn×n

we have vec(Lcos (A, E)) = Kcos (A) vec(E) and vec(Lsin (A, E)) = Ksin (A) vec(E),
where
 
Kcos (A) = − sinc(AT ⊕ (−A)) sin(I ⊗ A) + f (AT ⊕ (−A)) cos(I ⊗ A) , (12.15)
Ksin (A) = sinc(AT ⊕ (−A)) cos(I ⊗ A) − f (AT ⊕ (−A)) sin(I ⊗ A), (12.16)

where f (x) = (1 − cos x)/x.

Proof. From (12.12) and (12.9) we have

Z 1
− vec(Lcos (A, E)) = sin(AT s) ⊗ cos(A(1 − s))
0

+ cos(AT s) ⊗ sin(A(1 − s)) ds · vec(E)
Z 1

= sin AT s ⊕ A(1 − s) ds · vec(E)
0
Z 1

= sin [AT ⊕ (−A)]s + I ⊗ A ds · vec(E).
0

Since AT ⊕ (−A) = AT ⊗ I − I ⊗ A and I ⊗ A commute we can use the addition

formula (12.5) to obtain
Z 1


Kcos (A) = − sin [AT ⊕ (−A)]s cos(I ⊗ A) + cos [AT ⊕ (−A)]s sin(I ⊗ A) ds.
0
R1
The formula (12.15) now follows on using 0 sin(as) ds = (1 − cos(a))/a = f (a) and
R1
0
cos(as) ds = sin(a)/a ≡ sinc(a). The proof of (12.16) is analogous.
290 Matrix Cosine and Sine

Table 12.1. Number of matrix multiplications π2m required to evaluate p2m (A) and q2m (A).

2m 2 4 6 8 10 12 14 16 18 20 22 24 26 28 30
π2m 1 2 3 4 5 5 6 6 7 7 8 8 9 9 9

12.3. Padé Approximation of Cosine

It is not known whether [k/m] Padé approximants rkm of cos(x) exist for all m,
though formulae of Magnus and Wynn [401, ] give the coefficients of the diagonal
approximants rm as ratios of determinants of matrices whose entries involve binomial
coefficients. Since cos is an even function we need consider only even degrees 2m.
Padé approximants are readily computed symbolically, for example using MATLAB’s
Extended Symbolic Math Toolbox. The first two nontrivial Padé approximants are

5 2 115 2 313 4
1− x 1− x + x
12 252 15120
r2 (x) = , r4 (x) = .
1 11 2 13 4
1 + x2 1+ x + x
12 252 15120

Thereafter the numerators and denominators of the rational coefficients grow rapidly
in size; for example,

260735 2 4375409 4 7696415 80737373

1− x + x − x6 + x8
545628 141863280 13108167072 23594700729600
r8 (x) = .
12079 2 34709 109247 11321
1+ x + x4 + x6 + x8
545628 141863280 65540835360 1814976979200

There is no convenient continued fraction or partial

Pm fraction form for rP
m.
m
To evaluate rm we will explicitly evaluate p2m = i=0 a2i x2i and q2m = i=0 b2i x2i
and then solve the multiple right-hand side system q2m r2m = p2m . The most efficient
evaluation scheme is to treat p2m and q2m as degree m polynomials in A2 and apply
the Paterson–Stockmeyer method, as described in Section 4.4.3. Of equal cost for
8 ≤ 2m ≤ 28 are the schemes illustrated for 2m = 12 by

A2 = A2 , A4 = A22 , A6 = A2 A4 , (12.17a)
p12 = a0 I + a2 A2 + a4 A4 + a6 A6 + A6 (a8 A2 + a10 A4 + a12 A6 ), (12.17b)
q12 = b0 I + b2 A2 + b4 A4 + b6 A6 + A6 (b8 A2 + b10 A4 + b12 A6 ). (12.17c)

Table 12.1 summarizes the cost of evaluating p2m and q2m for 2m = 2: 2: 30.

12.4. Double Angle Algorithm for Cosine

When kAk < ∼ 1, cos(A) is readily approximated using a Taylor or Padé approximation,
as we have already seen in Section 4.3 in the case of the Taylor series. For large kAk we
can reduce the norm of the matrix whose cosine is approximated by using an analogue
of the scaling and squaring method for the matrix exponential. Let us define

Ci = cos(2i−s A).
12.4 Double Angle Algorithm for Cosine 291

The integer s is chosen so that 2−s A has norm small enough that we can obtain a good
approximation of C0 = cos(2−s A) at reasonable cost. By applying the cosine double
angle formula (12.6) we can compute Cs = cos(A) from C0 using the recurrence

Ci+1 = 2Ci2 − I, i = 0: s − 1. (12.18)

To develop an algorithm we need to decide how to choose s and how to approximate

C0 , taking into account computational cost and the effects of truncation and rounding
errors.
We begin by examining the propagation of errors in the double angle recurrence.
The following result describes the overall effect of the error in approximating C0 and
the rounding errors in the evaluation of the recurrence.

Theorem 12.5 (Higham and Smith). Let C bi = Ci + Ei , where C0 = cos(2−s A), the
b b 2
Ci satisfy (12.18), and Ci+1 = f l(2Ci − I). For the 1-, ∞−, and Frobenius norms,
assuming that kEi k ≤ 0.05kCi k for all i, we have

kEi k ≤ (4.1)i kE0 k kC0 k kC1 k . . . kCi−1 k

i−1
X
+γ
en+1 4.1i−j−1 (2.21αn kCj k2 + 1) kCj+1 k . . . kCi−1 k, (12.19)
j=0

where αn = n for the 2-norm and αn = 1 otherwise.

Proof. We have
bi+1 = f l(2C
C bi2 − I) = 2C
bi2 − I + Ri , (12.20)

where
kRi k ≤ γ bi k2 + 1).
en+1 (2αn kC (12.21)
We can rewrite (12.20) as

Ei+1 = 2(Ei2 + Ei Ci + Ci Ei ) + Ri .

Taking norms, we obtain

kEi+1 k ≤ 2kEi k(kEi k + 2kCi k) + kRi k. (12.22)

The assumption on kEi k then gives

kEi+1 k ≤ 4.1kEi k kCi k + kRi k. (12.23)

This recurrence is easily solved to give

i
X
kEi+1 k ≤ 4.1i+1 kE0 k kC0 k kC1 k . . . kCi k + 4.1i−j kRj k kCj+1 k . . . kCi k
j=0
i+1
≤ 4.1 kE0 k kC0 k kC1 k . . . kCi k
i
X
+γ
en+1 4.1i−j (2.21αn kCj k2 + 1) kCj+1 k . . . kCi k.
j=0
292 Matrix Cosine and Sine

Table 12.2. Maximum value θ2m of θ such that the absolute error bound (12.24) does not
exceed u = 2−53 .
2m 2 4 6 8 10 12 14 16 18 20 22 24 26 28 30
θ2m 6.1e-3 0.11 0.43 0.98 1.7 2.6 3.6 4.7 5.9 7.1 8.3 9.6 10.9 12.2 13.6

Consider the special case where A is normal with real eigenvalues, λi . Then
kCi k2 = max1≤j≤n | cos(2i−s λj )| ≤ 1 for all i and (12.19) yields

kEs k2 ≤ 4.1s (kE0 k2 + ne

γn+1 ).

This bound reflects the fact that, because the double angle recurrence multiplies the
square of the previous iterate by 2 at each stage, the errors could grow by a factor 4
at each stage, though this worst-case growth is clearly extremely unlikely.
It is natural to require the approximation C b0 to satisfy a relative error bound
of the form kC0 − C b0 k/kC0 k ≤ u. The lower bound in (12.10) guarantees that the
denominator is nonzero if kAk or kA2 k1/2 is less than cosh−1 (2) = 1.317 . . .. Consider
the term kE0 k in the bound (12.19). With the relative error bound and the norm
restriction just described, we have kE0 k ≤ ukC0 k ≤ 2u, which is essentially the same
as if we imposed an absolute error bound kE0 k ≤ u. An absolute bound has the
advantage of putting no restrictions on kAk or kA2 k1/2 and thus potentially allowing
a smaller s, which means fewer double angle recurrence steps. The norms of the
matrices C0 , . . . , Cs−1 are different in the two cases, and we can expect an upper
bound for kC0 k to be larger in the absolute case. But if the absolute criterion permits
fewer double angle steps then, as is clear from (12.19), significant gains in accuracy
could accrue. In summary, the error analysis provides support for the use of an
absolute error criterion if kC0 k is not too large. Algorithms based on a relative error
bound were developed by Higham and Smith [287, ] and Hargreaves and Higham
[248, ], but experiments in the latter paper show that the absolute bound leads
to a more efficient and more accurate algorithm. We now develop an algorithm based
on an absolute error bound.
As for the exponential and the logarithm, Padé approximants to the cosine provide
greater efficiency than Taylor series for the same quality of approximation. The
truncation error for r2m has the form
∞
X
cos(A) − r2m (A) = c2i A2i .
i=2m+1

Hence
∞
X
k cos(A) − r2m (A)k ≤ |c2i |θ2i , (12.24)
i=2m+1

where
θ = θ(A) = kA2 k1/2 .

Define θ2m to be the largest value of θ such that the bound in (12.24) does not exceed
u. Using the same mixture of symbolic and high precision calculation as was employed
in Section 10.3 for the exponential, we found the values of θ2m listed in Table 12.2.
12.4 Double Angle Algorithm for Cosine 293

Table 12.3. Upper bound for κ(q2m (A)) when θ ≤ θ2m , based on (12.26) and (12.27), where
the θ2m are given in Table 12.2. The bound does not exist for 2m ≥ 26.

2m 2 4 6 8 10 12 14 16 18 20 22 24
Bound 1.0 1.0 1.0 1.0 1.1 1.2 1.4 1.8 2.4 3.5 7.0 9.0e1

Table 12.4. Upper bounds for ke

p2m k∞ and ke
q2m k∞ for θ ≤ θ2m .

2m 2 4 6 8 10 12 14 16 18 20 22 24
ke
p2m k∞ 1.0 1.0 1.1 1.5 2.7 6.2 1.6e1 4.3e1 1.2e2 3.7e2 1.2e3 3.7e3
ke
q2m k∞ 1.0 1.0 1.0 1.0 1.1 1.1 1.2 1.3 1.4 1.6 1.7 2.0

We now need to consider the effects of rounding errors on the evaluation of r2m .
Consider, first, the evaluation of p2m and q2m , and assume initially that A2 is evalu-
ated exactly. Let g2m (A2 ) denote either of the even polynomials p2m (A) and q2m (A).
It follows from Theorem 4.5 that for the 1- and ∞-norms,

kg2m (A2 ) − f l(g2m (A2 ))k ≤ γ

emn ge2m (kA2 k), (12.25)

where ge2m denotes g2m with its coefficients replaced by their absolute values. When we
take into account the error in forming A2 we find that the bound (12.25) is multiplied
by a term that is approximately µ(A) = k|A|2 k/kA2 k ≥ 1. The quantity µ can
be arbitrarily large. However, µ∞ (A) ≤ (kAk∞ /θ∞ (A))2 , so a large µ implies that
basing the algorithm on θ(A) rather than kAk produces a smaller s, which means that
potentially increased rounding errors in the evaluation of p2m and q2m are balanced
by potentially decreased error propagation in the double angle phase.
Since we obtain r2m by solving a linear system with coefficient matrix q2m (A), we
require q2m (A) to
Pbe well conditioned to be sure that the system is solved accurately.
m
From q2m (A) = i=0 b2i A2i , we have
m
X
kq2m (A)k ≤ |b2i |θ2i . (12.26)
i=0

Using the inequality k(I + E)−1 k ≤ (1 − kEk)−1 for kEk < 1 gives
1 1
kq2m (A)−1 k ≤ Pm 2i
≤ Pm . (12.27)
|b0 | − k i=1 b2i A k |b0 | − i=1 |b2i |θ2i

Table 12.3 tabulates the bound for κ(q2m (A)) obtained from (12.26) and (12.27), the
latter being finite only for 2m ≤ 24. It shows that q2m is well conditioned for all m
in this range.
Now we consider the choice of m. In view of Table 12.3, we will restrict to 2m ≤ 24.
Table 12.4, which concerns the error bound (12.25) for the evaluation of p2m and q2m ,
suggests further restricting 2m ≤ 20, say. From Table 12.1 it is then clear that we
need consider only 2m = 2, 4, 6, 8, 12, 16, 20. Dividing A (and hence θ) by 2 results in
one extra matrix multiplication in the double angle phase, whereas for θ ≤ θ2m the
cost of evaluating the Padé approximant increases by one matrix multiplication with
each increase in m in our list of considered values. Since the numbers θ12 , θ16 , and θ20
294 Matrix Cosine and Sine

Table 12.5. Logic for choice of scaling and Padé approximant degree d ≡ 2m. Assuming A
has already been scaled, if necessary, so that θ ≤ θ20 = 7.1, further scaling should be done to
bring θ within the range for the indicated value of d.
Range of θ d
[0, θ16 ] = [0, 4.7] smallest d ∈ {2, 4, 6, 8, 12, 16} such that θ ≤ θd
(θ16 , 2θ12 ] = (4.7, 5.2] 12 (scale by 1/2)
(2θ12 , θ20 ] = (5.2, 7.1] 20 (no scaling)

differ successively by less than a factor 2, the value of d ≡ 2m that gives the minimal
work depends on θ. For example, if θ = 7 then d = 20 is best, because nothing would
be gained by a further scaling by 1/2, but if θ = 5 then scaling by 1/2 enables us to
use d = 12, and the whole computation then requires one less matrix multiplication
than if we immediately applied d = 20. Table 12.5 summarizes the relevant logic.
The tactic, then, is to scale so that θ ≤ θ20 and to scale further only if a reduction in
work is achieved.
With this scaling strategy we have, by (12.10), kC0 k ≤ cosh(θ20 ) ≤ 606. Since
this bound is not too much larger than 1, the argument given at the beginning of this
section provides justification for the following algorithm.

Algorithm 12.6 (double angle algorithm). Given A ∈ Cn×n this algorithm approx-
imates C = cos(A). It uses the constants θ2m given in Table 12.2. The algorithm is
intended for IEEE double precision arithmetic. At lines 5 and 12 the Padé approxi-
mants are to be evaluated via a scheme of form (12.17), making use of the matrix B.

1 B = A2
1/2
2 θ = kBk∞
3 for d = [2 4 6 8 12 16]
4 if θ ≤ θd
5 C = rd (A)
6 quit
7 end
8 end
9 s = ⌈log2 (θ/θ20 )⌉ % Find minimal integer s such that 2−s θ ≤ θ20 .
10 Determine optimal d from Table 12.5 (with θ ← 2−s θ); increase s as necessary.
11 B ← 4−s B
12 C = rd (2−s A)
13 for i = 1: s
14 C ← 2C 2 − I
15 end

1/2

Cost: πd + ⌈log2 (kA2 k∞ /θd )⌉ M + D, where πd and θd are tabulated in Ta-
bles 12.1 and 12.2, respectively.
In the next section we will compare Algorithm 12.6 with the following very simple
competitor based on (12.2).

Algorithm 12.7 (cosine and sine via exponential). Given A ∈ Cn×n this algorithm
computes C = cos(A) and S = sin(A) via the matrix exponential.
12.5 Numerical Experiment 295

1 X = eiA
2 If A is real
3 C = Re X
4 S = Im X
5 else
6 C = (X + X −1 )/2
7 S = (X − X −1 )/(2i)
8 end

An obvious situation in which Algorithm 12.7 is likely to produce an inaccurate

computed C is when kCk ≪ kSk ≈ keiA k, or conversely with the roles of C and S
interchanged. However, (12.14) shows that in this situation cos(A) is ill conditioned,
so an accurate result cannot be expected.

12.5. Numerical Experiment

We describe an experiment that compares the accuracy of three methods for comput-
ing cos(A).

1. Algorithm 12.6.

2. MATLAB’s funm, called as funm(A,@cos). This is Algorithm 9.6: the Schur–

Parlett algorithm with the cosines of the diagonal blocks of the triangular matrix
evaluated by Taylor series.

3. Algorithm 12.7, with the exponential evaluated by MATLAB’s expm (which

implements Algorithm 10.20).

We used a set of 55 test matrices including matrices from MATLAB, from the
Matrix Computation Toolbox [264], and from [287, ]. The matrices are mostly
10 × 10 and their norms range from order 1 to 107 , though more than half have ∞-
norm 10 or less. We evaluated the relative error kC b − CkF /kCkF , where Cb is the
computed approximation to C and C = cos(A) is computed at 100-digit precision.
Figure 12.1 shows the results. The solid line is the unit roundoff multiplied by an
estimate of κcos obtained using Algorithm 3.20 with finite differences. Figure 12.2
gives a performance profile representing the same data.
Some observations on the results:

• The three algorithms are all behaving in a numerically stable way apart from
occasional exceptions for each algorithm.

• The average cost of Algorithm 12.6 in this experiment is about 9 matrix mul-
tiplications, or 18n3 flops, which compares favourably with the cost of at least
28n3 flops of funm.

• It is reasonable to conclude that on this test set Algorithm 12.7 is the most
accurate solver overall, followed by Algorithm 12.6 and then funm.

• Algorithm 12.7 is expensive because it requires a matrix exponential evaluation

in complex arithmetic: complex arithmetic costs significantly more than real
arithmetic (see Section B.15) and a complex matrix requires twice the storage
of a real matrix. The good performance of the algorithm is dependent on that of
296 Matrix Cosine and Sine

5
10 Alg. 12.6
funm
Alg. 12.7

0
10

−5
10

−10
10

−15
10

0 10 20 30 40 50

Figure 12.1. Normwise relative errors for Algorithm 12.6, MATLAB’s funm, and Algo-
rithm 12.7; the solid line is an estimate of κcos (A)u.

expm: when MATLAB’s expmdemo1 (described in Section 10.5) is used in place

of expm we find that Algorithm 12.7 jumps from having the best performance
profile to the worst.

12.6. Double Angle Algorithm for Sine and Cosine

There is no natural analogue of Algorithm 12.6 for the sine, because the correspond-
ing double angle recurrence sin(2A) = 2 sin(A) cos(A) requires cosines. However,
computing the sine reduces to computing the cosine through sin(A) = cos(A − π2 I).
Suppose now that we wish to compute both sin(A) and cos(A). Algorithm 12.7
provides one way to do so, but requires complex arithmetic even when A is real. We
will develop an analogue of Algorithm 12.6 that scales A by a power of 2, computes
Padé approximants to both the sine and cosine of the scaled matrix, and then applies
the double angle formulae cos(2A) = 2 cos2 (A) − I and sin(2A) = 2 sin(A) cos(A).
Computational savings are possible in the evaluation of the Padé approximants and
in the double angle recurrences by reusing the cos terms.
Denote the [m/m] Padé approximant to the sine function by rem (x) = pem (x)/e
qm (x).
Then pem is odd and qem even, their degrees being, respectively, m and m − 1 if m is
odd, otherwise m − 1 and m [401, ], so the error in rem has the form
∞
X
sin(A) − rem (A) = c2i+1 A2i+1 .
i=m

Since this expansion contains only odd powers of A we bound the series in terms of
kAk instead of θ(A):
∞
X
k sin(A) − rem (A)k ≤ |c2i+1 |β 2i , β = kAk. (12.28)
i=m
12.6 Double Angle Algorithm for Sine and Cosine 297

0.9

0.8

0.7

0.6
p
0.5

0.4

0.3

0.2
Alg. 12.7
0.1 Alg. 12.6
funm
0
5 10 15 20 25 30 35 40 45 50
α

Figure 12.2. Same data as in Figure 12.1 presented as a performance profile.

Table 12.6. Maximum value βm of kAk such that the absolute error bound (12.28) does not
exceed u = 2−53 .
m 2 3 4 5 6 7 8 9 10 11 12
βm 1.4e-3 1.8e-2 6.4e-2 1.7e-1 0.32 0.56 0.81 1.2 1.5 2.0 2.3
m 13 14 15 16 17 18 19 20 21 22 23 24
βm 2.9 3.3 3.9 4.4 5.0 5.5 6.2 6.7 7.4 7.9 8.7 9.2

Define βm to be the largest value of β such that the bound (12.28) does not exceed
u. Using the same technique as for the cosine, we computed the values shown in
Table 12.6. These values of βm can be compared with the values of θ2m in Table 12.2.
On comparing Table 12.6 with Table 12.2 we see that for 4 ≤ 2m ≤ 22 we have
β2m < θ2m < β2m+1 . We could therefore scale so that k2−s Ak ≤ β2m and then use
the [2m/2m] Padé approximants to the sine and cosine, or scale so that k2−s Ak ≤ θ2m
and use the [2m/2m] Padé approximant to the cosine and the [2m + 1/2m + 1]
Padé approximant to the sine. Since we can write an odd polynomial in A (e pm (A))
as A times an even polynomial of degree one less, it turns out to be as cheap to
evaluate re2m+1 and r2m as to evaluate re2m and r2m . Therefore we will scale so that
k2−s Ak ≤ θ2m and then evaluate r2m for the cosine and re2m+1 for the sine. Evaluating
p2m , q2m , pe2m+1 , and qe2m+1 reduces to evaluating four even polynomials of degree
2m. This can be done by forming the powers A2 , A4 , . . . , A2m , at a total cost of
m + 1 multiplications. However, for 2m ≥ 20 it is more efficient to use the schemes of
the form (12.17). We summarize the cost of evaluating p2m , q2m , pe2m+1 , and qe2m+1
for m = 2: 2: 24 in Table 12.7.
Now we consider the choice of degree, d ≡ 2m. Bounds analogous to those in
Table 12.3 show that qej+1 is well conditioned for d ≤ 24, and bounds for pej+1 and qej+1
analogous to those in Table 12.4 suggest restricting to d ≤ 20 (the same restriction
298 Matrix Cosine and Sine

Table 12.7. Number of matrix multiplications π

e2m to evaluate p2m (A), q2m (A), pe2m+1 (A),
and qe2m+1 (A).
2m 2 4 6 8 10 12 14 16 18 20 22 24
π
e2m 2 3 4 5 6 7 8 9 10 10 11 11

that was made in Section 12.4 for the Padé approximants for the cosine). It is then
clear from Table 12.7 that we need only consider d = 2, 4, 6, 8, 10, 12, 14, 16, 20. Noting
that dividing A by 2 results in two extra multiplications in the double-angle phase
and that increasing from one value of d to the next in our list of considered values
increases the cost of evaluating the Padé approximants by one multiplication, we can
determine the most efficient choice of d by a similar argument to that in the previous
section. The result is that we should scale so that θ ≤ θ20 , and scale further according
to exactly the same strategy as in Table 12.5 except that in the first line of the table
“10” and “14” are added to the set of possible d values.
The algorithm can be summarized as follows.

Algorithm 12.8 (double angle algorithm for sine and cosine). Given A ∈ Cn×n this
algorithm approximates C = cos(A) and S = sin(A). It uses the constants θ2m given
in Table 12.2. The algorithm is intended for IEEE double precision arithmetic.

1 for d = [2 4 6 8 10 12 14 16]
2 if kAk∞ ≤ θd
3 C = rd (A), S = red+1 (A)
4 quit
5 end
6 end
7 s = ⌈log2 (kAk∞ /θ20 )⌉ % Find minimal integer s such that 2−s kAk∞ ≤ θ20 .
8 Determine optimal d from modified Table 12.5 (with θ = 2−s kAk∞ )
and increase s as necessary.
9 C = rd (2−s A), S = red+1 (2−s A)
10 for i = 1: s
11 S ← 2CS, C ← 2C 2 − I
12 end


Cost: π ed + ⌈log2 (kAk∞ /θd )⌉ M + 2D, where θd and π ed are tabulated in Ta-
bles 12.2 and 12.7, respectively.
How much work does Algorithm 12.8 save compared with separate computation of
cos(A) and sin(A) = cos(A − π2 I) by Algorithm 12.6? The answer is roughly 2πd − π ed
matrix multiplies, which rises from 1 when d = 4 to 4 when d = 20; the overall saving
is therefore up to about 29%.
We tested Algorithm 12.8 on the same set of test matrices as in Section 12.5.
Figure 12.3 compares the relative errors for the computed sine and cosine with the
corresponding errors from funm, invoked as funm(A,@sin) and funm(A,@cos); with
Algorithm 12.7; and with sin(A) computed as the shifted cosine sin(A) = cos(A − π2 I)
using Algorithm 12.6. Note that the cost of the two funm computations can be reduced
by using the same Schur decomposition for sin(A) as for cos(A). Algorithm 12.8
provides similar or better accuracy to funm and the shifted cosine on this test set. Its
cost varies from 9 matrix multiplies and solves to 55, with an average of 17.
12.7 Notes and References 299

Alg 12.6
−5
10 Alg. 12.8 (cos)
funm (cos)

−10
10

−15
10

0 10 20 30 40 50

0 Alg. 12.6 (shifted)

10 Alg 12.7 (sin)
Alg. 12.8 (sin)
funm (sin)

−10
10

0 10 20 30 40 50

Figure 12.3. Normwise relative errors for Algorithm 12.6, Algorithm 12.7, Algorithm 12.8,
funm, and sine obtained as shifted cosine from Algorithm 12.6. The solid line is an estimate
of κcos (A)u (top) and κsin (A)u (bottom).

12.6.1. Preprocessing
Balancing and shifting can be used to reduce the norm prior to applying any of the
algorithms in this chapter. We can exploit the periodicity relation (4.32) to shift A
by πjI. Theorem 4.21 motivates taking j as whichever of the two integers nearest to
trace(A)/(nπ) gives the smaller value of kA − πjIkF , or as 0 if neither choice reduces
the Frobenius norm of A. Thus preprocessing consists of computing A e = D−1 (A −
πqI)D, where D represents balancing, and recovering cos(A) = (−1)q D cos(A)D e −1
q e
and sin(A) = (−1) D sin(A)D . −1

12.7. Notes and References

The bounds (12.10) are from Hargreaves and Higham [248, ]. The Fréchet deriva-
tive formulae in Section 12.2 appear to be new.
Serbin and Blalock [515, ] suggest computing cos(A) by a Taylor or Padé
approximation in conjunction with the double angle recurrence, but do not propose a
specific algorithm. Higham and Smith [287, ] develop an algorithm that chooses s
so that k2−s Ak∞ ≤ 1, approximates cos(A) ≈ r8 (2−s A), and then uses the double an-
gle recurrence. Ad hoc analysis in [287, ] shows that the [8/8] Padé approximant
provides full normwise relative accuracy in IEEE double precision. Hargreaves and
Higham [248, ] make several improvements. They phrase truncation error bounds
in terms of kA2 k1/2 instead of kAk; choose the degree of the Padé approximant adap-
tively to minimize the computational cost subject to achieving a desired truncation
error; and use an absolute, rather than relative, error criterion in the choice of Padé
approximant. Theorem 12.5 is from Higham and Smith [287, ] and Algorithms
12.6 and 12.8 from Hargreaves and Higham [248, ].
300 Matrix Cosine and Sine

The matrix cosine and sine are used in a matrix multiplication-dominated algo-
rithm proposed by Yau and Lu [619, ] for solving the symmetric eigenvalue prob-
lem Ax = λx. This algorithm requires cos(A) and sin(A), which are approximated
using Chebyshev expansions in conjunction with the cosine double angle recurrence.
For more details see [619, ] and Tisseur [568, ].

Problems
12.1. Show that if A ∈ Cn×n is involutory (A2 = I) then cos(kπA) = (−1)k I for all
integers k.
 
12.2. (Pólya and Szegö [475, , Problem 210]) Does the equation sin(A) = 10 11
have a solution?
12.3. Derive (12.1), (12.3), (12.4), and (12.5) using the results of Section 1.3.
12.4. Assuming that X0 is nonsingular, show that X1 obtained from one step of the
Newton sign iteration (5.16) can be written X1 = cos(i−1 log(X0 )). (Cf. line 6 of
Algorithm 12.7.)
12.5. Suppose we know cos(A). Can we determine sin(A) from (12.3), i.e., from the
relation cos2 (A) + sin2 (A) = I? Consider separately the cases where A is a general
matrix and A is triangular.

The reader is expected not to be worried by expressions such as

the sine of a matrix A.
— SIMON L. ALTMANN, Rotations, Quaternions, and Double Groups (1986)

Rational approximations for the cosine function

play an important role in the linear stability analysis of
numerical methods for initial-value problems of the special form
y ′′ = f (x, y), y(x0 ) = y0 , y ′ (x0 ) = z0 .
— JOHN P. COLEMAN, Rational Approximations for the Cosine Function;
P-Acceptability and Order (1992)
Chapter 13
Function of Matrix Times Vector: f (A)b

In some applications it is not f (A) that is required but rather the action of f (A)
on a vector: f (A)b. Indeed, if A is sparse then f (A) may be much denser than A
and for large dimensions it may be too expensive to compute or store f (A), while
computing and storing f (A)b may be feasible. In this chapter we consider various
facets of the f (A)b problem. Our treatment is relatively concise, for two reasons.
First, some of the techniques used for f (A)b are closely related to those for f (A) that
have been treated earlier in the book. Second, the f (A)b problem overlaps with the
theory and practice of Krylov subspace methods and of the numerical evaluation of
contour integrals. A thorough treatment of these latter topics is outside the scope
of this book. Moreover, the application of these techniques to the f (A)b problem is
a relatively new and active topic of research and the state of the art is advancing
rapidly.
The development of a method for f (A)b must be guided by the allowable opera-
tions with A. A minimal assumption is that matrix–vector products can be formed.
Another possible assumption is that A is a large sparse matrix for which linear sys-
tems (αA + βI)x = c can be solved efficiently by sparse direct methods but the
computation of the Hessenberg and Schur forms are impractical. Alternatively, we
may assume that any necessary factorization can be computed.
We begin this chapter by obtaining a b-specific representation of f (A)b in terms of
Hermite interpolation. Then we consider computational approaches based on Krylov
subspace methods, quadrature, and differential equations.
We emphasize that the methods described in this chapter are not intended for
f (A) = A−1 , for which the special form of f can usually be exploited to derive more
effective variants of these methods.

13.1. Representation via Polynomial Interpolation

Definition 1.4 specifies f (A) = p(A), where p is a Hermite interpolating polynomial

determined by the values of f on the spectrum of A. The degree of p is the degree
of the minimal polynomial ψA of A and it may be greater than necessary in order to
produce f (A)b. To see why, consider the minimal polynomial of A with respect to b,
which is the unique monic polynomial ψA,b of lowest degree such that ψA,b (A)b = 0.
The results developed in Section 1.2.2 generalize from ψA to ψA,b . Specifically, ψA,b
divides any polynomial p for which p(A)b = 0; ψA,b has the form

s
Y
ψA,b (t) = (t − λi )ℓi , 0 ≤ ℓi ≤ ni ,
i=1

301
302 Function of Matrix Times Vector: f (A)b

where ℓi depends on b as well as A and λ1 , . . . , λs are the distinct eigenvalues of A,

with ni the dimension of the largest Jordan block in which λi appears; and we have
the following generalization of Theorem 1.3.

Theorem 13.1. For polynomials p and q and A ∈ Cn×n , p(A)b = q(A)b if and only
if p(j) (λi ) = q (j) (λi ), j = 0: ℓi − 1, i = 1: s.

With this background we can show that the required polynomial is of degree only
deg ψA,b ≤ deg ψA .

Theorem 13.2. Let f be defined on the spectrum of A ∈ Cn×n and let ψA,b be the
minimal polynomial of A with respect to b. Then f (A)b
P= q(A)b, where q is the unique
s
Hermite interpolating polynomial of degree less than i=1 ℓi = deg ψA,b that satisfies
the interpolation conditions

q (j) (λi ) = f (j) (λi ), j = 0: ℓi − 1, i = 1: s. (13.1)

Proof. Let p be the polynomial given by Definition 1.4 such that f (A) = p(A) and
consider q defined by (13.1). By (1.7), (13.1), and Theorem 13.1, q(A)b = p(A)b =
f (A)b.
Of course, the polynomial q in Theorem 13.2 depends on both A and b, and
q(A)c = f (A)c is not guaranteed for c 6= b.
The kth Krylov subspace of A ∈ Cn×n and a nonzero vector b ∈ Cn is defined by

Kk (A, b) = span{b, Ab, . . . , Ak−1 b},

and it can also be written as

Kk (A, b) = span{ q(A)b : q is a polynomial of degree ≤ k − 1 }.

Theorem 13.2 says that f (A)b ∈ Kd (A, b), where d = deg ψA,b . Thus the size of a
Krylov subspace necessary to capture f (A)b depends on both A and b. Note also that
deg ψA,b can be characterized as the smallest k such that Kk (A, b) = Kk+1 (A, b).
We consider Krylov subspace methods in the next section.

13.2. Krylov Subspace Methods

13.2.1. The Arnoldi Process
The Arnoldi process for A ∈ Cn×n attempts to compute the Hessenberg reduction
Q∗ AQ = H, where Q ∈ Cn×n is unitary and H ∈ Cn×n is upper Hessenberg. Writing
Q = [q1 , . . . , qn ] and equating kth columns in AQ = QH gives
k+1
X
Aqk = hik qi , k = 1: n − 1. (13.2)
i=1

This may be rewritten as

k
X
hk+1,k qk+1 = Aqk − hik qi =: rk , (13.3)
i=1
13.2 Krylov Subspace Methods 303

where, using the fact that Q is unitary,

hik = qi∗ Aqk , i = 1: k. (13.4)

Provided that rk 6= 0, qk+1 = rk /hk+1,k with hk+1,k = krk k2 .

From (13.2) it follows by induction that

span{q1 , . . . , qk } = span{q1 , Aq1 , . . . , Ak−1 q1 },

that is, the Arnoldi vectors {qi }ki=1 form an orthonormal basis for the Krylov subspace
Kk (A, q1 ). The Arnoldi process produces the factorization

AQk = Qk Hk + hk+1,k qk+1 eTk , (13.5)

where Qk = [q1 , . . . , qk ] and Hk = (hij ) is k × k upper Hessenberg. Columns 1 to

k − 1 of this factorization are just (13.2), while column k is (13.3). Note that (13.4)
can be written
Q∗k AQk = Hk , (13.6)
which says that Hk is the orthogonal projection of A onto span{q1 , . . . , qk } = Kk (A, q1 ).
We noted above that Km (A, q1 ) = Km+1 (A, q1 ) for m = deg ψA,q1 , so after gener-
ating the orthonormal basis q1 , . . . , qm for Km (A, q1 ) the Arnoldi process must produce
rm = 0, i.e., the process terminates.
The Arnoldi process can be summarized as follows.

Algorithm 13.3 (Arnoldi process). Given A ∈ Cn×n and q1 ∈ Cn of unit 2-norm,

this algorithm uses the Arnoldi process to compute the factorization AQ = QH,
where Q ∈ Cn×m (m = deg ψA,q1 ≤ n) has orthonormal columns and H ∈ Cm×m is
upper Hessenberg.

1 for k = 1: n
2 z = Aqk
3 for i = 1: k
4 hik = qi∗ z
5 z = z − hik qi % modified Gram–Schmidt
6 end
7 hk+1,k = kzk2
8 if hk+1,k = 0, m = k, quit, end
9 qk+1 = z/hk+1,k
10 end

A few comments on the Arnoldi process are in order.

Implicit A. The matrix A does not need to be known or stored explicitly, since it
appears in the Arnoldi process only in the matrix–vector products Aqk . Therefore A
can be given in any implicit form that permits efficient evaluation of the action of A
on a vector.
Orthogonality. Algorithm 13.3 uses the modified Gram–Schmidt process to or-
thogonalize Aqk against q1 , . . . , qk . Classical Gram–Schmidt can be used instead, in
which case line 2 is replaced by the assignments rk = Aqk , z = rk and in line 5
onwards z is replaced by rk . Loss of orthogonality of the Arnoldi vectors can occur in
finite precision arithmetic for both Gram–Schmidt processes (the more so for classical
Gram–Schmidt), and this can be cured by reorthogonalization. Loss of orthogonality
304 Function of Matrix Times Vector: f (A)b

can alternatively be avoided completely by using an implementation of the Arnoldi

process based on Householder transformations.
Hermitian A. When A is Hermitian most of the inner products in the inner loop
of the Arnoldi process are zero and so need not be computed. Indeed for Hermitian
A the Arnoldi process reduces to the Lanczos process, which involves a three-term
recurrence. The Lanczos process yields real, symmetric tridiagonal Hk .

13.2.2. Arnoldi Approximation of f (A)b

To approximate f (A)b with the Arnoldi process we take q1 = b/kbk2 and, for some
appropriate k,

fk := kbk2 Qk f (Hk )e1 , (13.7)

= Qk f (Hk )Q∗k b.

We are effectively evaluating f on the smaller Krylov subspace Kk (A, q1 ) and then
expanding the result back onto the original space Cn .
When is (13.7) an exact approximation? Clearly, if k = n is reached then Qk ∈
Cn×n is unitary and f (A)b = f (Qn Hn Q∗n )b = Qn f (Hn )Q∗n b = Qn f (Hn )kbk2 e1 = fk .
More precisely, upon termination of the Arnoldi process on step m = deg ψA,b , we
have fm = f (A)b (see Problem 13.1).
Some more insight into the approximation (13.7) is provided by the next two re-
sults. The first says that the approximation is exact if f is a polynomial of sufficiently
low degree.

Lemma 13.4 (Saad). Let A ∈ Cn×n and Qk , Hk be the result of k steps of the
Arnoldi process on A. Then for any polynomial pj of degree j ≤ k − 1 we have

pj (A)q1 = Qk pj (Hk )e1 .

Proof. It suffices to prove the result for pj (t) = tj , which we do by induction.

The result is trivially true for j = 0. Assume that it is true for j with j ≤ k − 2.
Then

Aj+1 q1 = A · Aj q1 = AQk Hkj e1

= (Qk Hk + hk+1,k qk+1 eTk )Hkj e1
= Qk Hkj+1 e1 + hk+1,k qk+1 eTk Hkj e1
= Qk Hkj+1 e1 ,

since eTk Hkj e1 = 0 for j ≤ k − 2. Thus the result is true for j + 1, as required.
The next result identifies the approximation (13.7) as being obtained from a poly-
nomial that interpolates f on the spectrum of Hk rather than the spectrum of A.

Theorem 13.5 (Saad). Let Qk , Hk be the result of k steps of the Arnoldi process on
A ∈ Cn×n and b ∈ Cn . Then

kbk2 Qk f (Hk )e1 = pek−1 (A)b,

where pek−1 is the unique polynomial of degree at most k − 1 that interpolates f on the
spectrum of Hk .
13.2 Krylov Subspace Methods 305

Proof. Note first that the upper Hessenberg matrix Hk has nonzero subdiagonal
elements hj+1,j , which implies that it is nonderogatory (see Problem 13.3). Conse-
quently, the minimal polynomial of H is the characteristic polynomial and so has
degree k. By Definition 1.4, f (Hk ) = pek−1 (Hk ) for a polynomial pek−1 of degree at
most k − 1. Hence

kbk2 Qk f (Hk )e1 = kbk2 Qk pek−1 (Hk )e1 = kbk2 pek−1 (A)q1 = pek−1 (A)b,

using Lemma 13.4 for the middle equality.

We briefly discuss some of the important issues associated with the approximation
(13.7).
Restarted Arnoldi. As k increases so does the cost of storing the Arnoldi vectors
q1 , . . . , qk and computing Hk . It is standard practice in the context of linear systems
(f (x) = 1/x) and the eigenvalue problem to restart the Arnoldi process after a fixed
number of steps with a judiciously chosen vector that incorporates information gleaned
from the computations up to that step. For general f restarting is more difficult
because of the lack of a natural residual. Eiermann and Ernst [174, ] show how
restarting can be done for general f by a technique that amounts to changing at the
restart both the starting vector and the function. This method requires the evaluation
of f at a block Hessenberg matrix of size km, where m is the restart length. This
expense is avoided in a different implementation by Afanasjew, Eiermann, Ernst, and
Güttel [3, ], which employs a rational approximation to f and requires only the
solution of k Hessenberg linear systems of size m. Further analysis of restarting is
given by Afanasjew, Eiermann, Ernst, and Güttel [4, ].
Existence of f (Hk ). For k < deg ψA,b it is possible that f (Hk ) is not defined, even
though f (A) is. From (13.6) it follows that the field of values of Hk is contained in
the field of values of A. Therefore a sufficient condition for f (Hk ) to be defined is
that f and its first n derivatives are defined at all points within the field of values
of A.
Computing f (Hk ). The Arnoldi approximation (13.7) requires the computation
of f (Hk )e1 , which is the first column of f (Hk ). Since k is of moderate size we can
compute the whole matrix f (Hk ) by any of the methods described in earlier chapters
of this book, taking advantage of the Hessenberg structure if possible. Whether or not
it is possible or advantageous to apply a method specific to this f (Hk )b problem—
such as those described in the following sections—depends on f , k, and the desired
accuracy.
Convergence, error bounds, and stopping test. The quality of the approximations
(13.7) depends on how well pek−1 (A)b approximates f (A)b, where pek−1 is defined
in Theorem 13.5. It is well known that the eigenvalues in the outermost part of
the spectrum tend to approximated first in the Arnoldi process, so we might expect
the components of the approximations (13.7) in the eigenvectors corresponding to
the outermost part of the spectrum to be the most accurate. We state a result
that describes convergence for the exponential function and Hermitian matrices with
spectrum in the left half-plane.

Theorem 13.6. Let A be a Hermitian negative semidefinite matrix with eigenvalues

on [−4ρ, 0], where ρ > 0, and set β = kbk2 . Then the error in the approximation
306 Function of Matrix Times Vector: f (A)b

(13.7) of eA b is bounded by
 2 √
 10β e−k /(5ρ) , 2 ρ ≤ k ≤ 2ρ,
ke b − βQk e e1 k2 ≤ 10β −ρ  eρ k
A Hk
 e , k ≥ 2ρ.
ρ k
Proof. See Hochbruck and Lubich [291, ].
The bound of the theorem shows that rapid convergence is obtained for k ≥
1/2
kAk2 . Similar convergence bounds for f (x) = ex are given by Hochbruck and
Lubich in [291, ] for general A under the assumption that the field of values of A
lies in a disk or a sector. However, convergence for general A and general f is poorly
understood (see Problem 13.5).
We are not aware of a criterion for general f and general A for determining when
the approximation (13.7) is good enough and hence when to terminate the Arnoldi
process. For the exponential, an error estimate

keA b − Qk eHk Q∗k bk2 ≈ kbk2 hk+1,k |eTk eHk e1 |

is suggested by Saad [499, , Sec. 5.2].

For Hermitian A and a given rational approximation r to f expressed in partial
fraction form, Frommer and Simoncini [197, ] develop error bounds suitable for
terminating the conjugate gradient method applied to the resulting shifted linear
systems.

13.2.3. Lanczos Biorthogonalization

An alternative to the Arnoldi process for general A is Lanczos biorthogonalization.
It constructs biorthogonal bases for Km (A, v) and Km (A∗ , w) using two three-term
recurrences involving matrix–vector products with A and A∗ , producing two matrices
Vk , Wk ∈ Cn×k such that Wk∗ Vk = Ik and Wk∗ AVk =: Tk is tridiagonal. An advantage
over Arnoldi is the requirement to store fewer vectors. A major disadvantage is that
the process can break down. Analogues of (13.5) and the approximation (13.7) apply
to Lanczos biorthogonalization, and similar error bounds apply, though the results
are weaker because the matrices Vk and Wk do not have orthonormal columns.

13.3. Quadrature
13.3.1. On the Real Line
Ra
Suppose we have a representation f (A) = 0 g(A, t) dt for some rational function g
and a ∈ R. Then the use of a quadrature formula
Z a m
X
g(t) dt ≈ ck g(tk ) (13.8)
0 k=1

is natural, yielding
m
X
f (A)b ≈ g(A, tk )b. (13.9)
k=1

We have seen that the matrix sign function, matrix pth roots, the unitary polar factor,
and the matrix logarithm all have the required integral representations with a rational
13.3 Quadrature 307

function g (see (5.3), (7.1), (8.7), and (11.1)) and so are amenable to this approach.
Because g is rational, (13.9) reduces to matrix–vector products and the solution of
linear systems. If a Hessenberg reduction A = QHQ∗ can be computed then (13.9)
transforms to
Xm
f (A)b ≈ Q g(H, tk )d, d = Q∗ b. (13.10)
k=1

Since a Hessenberg system can be solved in O(n2 ) flops, as opposed to O(n3 ) flops for
a dense system, (13.10) will be more efficient than (13.9) for sufficiently large m or
sufficiently many different vectors b that need to be treated with the same A; indeed,
ultimately the most efficient approach will be to employ a Schur decomposition instead
of a Hessenberg decomposition.
The quadrature formula (13.8) might be a Gauss rule, a repeated rule, such as
the repeated trapezium or repeated Simpson rule, or the result of applying adaptive
quadrature with one of these rules. Gaussian quadrature has close connections with
Padé approximation, and we noted in Section 11.4 that for the logarithm the use
of Gauss–Legendre quadrature with the integral (11.1) leads to the diagonal Padé
approximants to the logarithm, via (11.18). Focusing on log(I + X) for the moment,
Theorem 11.6 shows that an error bound for the Padé approximant, and hence for
Gaussian quadrature, is available for kXk < 1. For kXk > 1 the inverse scaling and
squaring approach is not attractive in the context of log(A)b, since it requires matrix
square roots. However, adaptive quadrature can be used and automatically chooses
the location and number of the quadrature points in order to achieve the desired
accuracy. For more details, including numerical experiments, see Davies and Higham
[136, ].

13.3.2. Contour Integration

For general f we can represent y = f (A)b using the Cauchy integral formula (1.12):
Z
1
y= f (z)(zI − A)−1 b dz, (13.11)
2πi Γ

where f is analytic on and inside a closed contour Γ that encloses the spectrum of A.
Suppose we take for the contour Γ a circle with centre α and radius β, { z : z − α =
βeiθ , 0 ≤ θ ≤ 2π }, and then approximate the integral using the repeated trapezium
rule. Using dz = iβeiθ dθ = idθ(z(θ) − α) and writing the integrand in (13.11) as g(z),
we obtain Z Z 2π
g(z)dz = i (z(θ) − α)g(z(θ)) dθ. (13.12)
Γ 0

The integral in (13.12) is a periodic function of θ with period 2π. Applying the
m-point repeated trapezium rule to (13.12) gives
Z m−1
2πi X
g(z) dz ≈ (zk − α)g(zk ),
Γ m
k=0

where zk − α = βe2πki/m , that is, z0 , . . . , zm are equally spaced points on the contour
Γ (note that since Γ is a circle we have z0 = zm ). When A is real and we take α real
it suffices to use just the zk in the upper half-plane and then take the real part of the
result. The attraction of the trapezium rule is that it is exponentially accurate when
308 Function of Matrix Times Vector: f (A)b

applied to an analytic integrand on a periodic domain [142, , Sec. 4.6.5], [571,
].
Although this is a natural approach, it is in general very inefficient unless A is
well conditioned, as shown by Davies and Higham [136, ] and illustrated below.
The problem is that Γ is contained in a very narrow annulus of analyticity. However,
by using a suitable change of variables in the complex plane, carefully constructed
based on knowledge of the extreme points of the spectrum and any branch cuts or
singularities of f , quadrature applied to the Cauchy integral can be very effective, as
shown by Hale, Higham, and Trefethen [240, ].
For illustration consider the computation of A1/2 for A the 5 × 5 Pascal matrix,
which is symmetric positive definite with spectrum on the interval [0.01, 92.3]. First,
we choose for Γ the circle with centre (λmin + λmax )/2 and radius λmax /2, which is a
compromise between enclosing the spectrum and avoiding the negative real axis. We
find that about 32,000 and 262,000 points are needed to provide 2 and 13 decimal
digits of accuracy, respectively. As an alternative we instead conformally map the
region of analyticity of f and the resolvent to an annulus, with the interval containing
the spectrum mapping to the inner boundary circle and the negative real axis mapping
to the outer boundary circle. Then we apply the trapezium rule over a circle in the
annulus. With the mapping constructed as described in [240, ], we need just 5
and 35 points to provide 2 and 13 decimal digits of accuracy, respectively—a massive
improvement, due to the enlarged annulus of analyticity. By further exploiting the
properties of the square root function these numbers of points can be reduced even
more, with just 20 points yielding 14 digits. Theorems are available to explain this
remarkably fast (geometric) convergence [240, ].
When a quadrature rule is applied to a (transformed) Cauchy integral it produces
an answer that can be interpreted as the exact integral of a rational function whose
poles are the nodes and whose residues are the weights. It is therefore natural to ask
what rational approximations the trapezium rule is producing in the above example.
It turns out the last of the approaches mentioned above reproduces what is essentially
the best rational approximation to x−1/2 in Theorem 5.15 identified by Zolotarev.
However, conformal mapping and the trapezium rule can be applied to arbitrary
functions and to matrices with spectra that are not all real, for which associated best
rational approximations are usually not known.
Numerical evaluation of the ψ functions (see Section 10.7.4) via contour integrals,
in the context of exponential integrators, is discussed by Kassam and Trefethen [336,
] and Schmelzer and Trefethen [503, ]. For the case of the exponential and
matrices with negative real eigenvalues, see Trefethen, Weideman, and Schmelzer [574,
].

13.4. Differential Equations

Another approach to computing y = f (A)b is to express y as the solution of an
initial value differential equation problem and apply an ODE solver. As an example,
consider f (A) = Aα for real α.

Theorem 13.7 (Davies and Higham). For A ∈ Cn×n with no eigenvalues on R−

and α ∈ R, the initial value ODE problem.

dy  −1
= α(A − I) t(A − I) + I y, y(0) = b, 0 ≤ t ≤ 1, (13.13)
dt
13.5 Other Methods 309

 α
has the unique solution y(t) = t(A − I) + I b, and hence y(1) = Aα b.

Proof. The existence of a unique solution follows from the fact that the ODE
satisfies a Lipschitz condition with Lipschitz constant sup0≤t≤1 k(A − I) t(A − I) +
−1
I k < ∞. It is easy to check that y(t) is this solution.
Thus y(1) = Aα b can be obtained by applying an ODE solver to (13.13). The
initial value problem can potentially be stiff, depending on α, the matrix A, and the
requested accuracy, so some care is needed in choosing a solver. Again, a Hessenberg
or Schur reduction of A can be used to reduce the cost of evaluating the differential
equation.

13.5. Other Methods

In this section we briefly outline some other approaches to the estimation or compu-
tation of f (A)b.
Another class of methods forms approximations f (A)b ≈ pk (A)b where pk is a
polynomial obtained by truncating an expansion of f in terms of some suitable infi-
nite sequence of polynomials satisfying a short (usually three-term) recurrence. This
class of approximations is well developed in the case of linear systems. The use of
Chebyshev series for Hermitian matrices was suggested by Druskin and Knizhnerman
[167, ]. For non-Hermitian matrices the Faber series are more appropriate, and
their use has been proposed by Moret and Novati [440, ]. Novati [450, ]
chooses pk as an interpolant to f at the Fejér points. These methods all need knowl-
edge of the extremal eigenvalues of A.

13.6. Notes and References

The original reference for the Arnoldi process is Arnoldi [18, ].
Our treatment of Krylov subspace methods is necessarily very brief. We give here
a few pointers to the literature, without trying to cite all the relevant papers.
Excellent general references on Krylov subspace methods include Bai, Demmel,
Dongarra, Ruhe, and Van der Vorst [30, ], Golub and Van Loan [224, ,
Chap. 9], Saad [500, , Chaps. 6, 7], Watkins [608, , Chap. 9].
The earliest uses in the numerical analysis literature of the approximation (13.7)
appear to be by Knizhnerman [356, ] for general functions and by Gallopoulos
and Saad [200, ], [498, ] for the exponential.
Lemma 13.4 is from Saad [498, , Lem. 3.1]. Theorem 13.5 is a trivial gener-
alization of Saad [498, , Thm. 3.3], which applies to f (x) = ex .
Error analysis of the modified Gram–Schmidt process is summarized in Higham
[276, , Sec. 19.8]. For classical Gram–Schmidt see Smoktunowicz, Barlow, and
Langou [532, ]. For the Householder-based implementation of the Arnoldi process
see Saad [500, , Sec. 6.3.2]. Reorthogonalization is described in Stewart [538,
, Sec. 5.1] and in [30, ].
For more on Lanczos biorthogonalization see Saad [500, , Chap. 7] or Watkins
[608, , Sec. 9.6].
An early treatment of the f (A)b problem is that of Van Der Vorst [587, ]; see
also [588, ] and [589, , Chap. 11].
For a summary of a sign(A)b problem arising in lattice QCD see Section 2.7.
310 Function of Matrix Times Vector: f (A)b

The effects of finite precision arithmetic on approximation of f (A)b for Hermitian

A by the Lanczos approach is analyzed by Druskin, Greenbaum, and Knizhnerman
[166, ]. A survey of matrix functions with particular emphasis on Krylov methods
is given by Frommer and Simoncini [198, ].
Theorem 13.7 is from Davies and Higham [136, ]. In the case α = 1/2 and A
symmetric positive definite, the theorem and its use with an ODE initial value solver
are given by Allen, Baglama and Boyd [10, ], who describe an application in the
numerical solution of stochastic differential equations. Another differential equations
application in which A±1/2 b is needed for symmetric positive definite A is described
by Hughes, Levit, and Winget [304, ].

Problems
13.1. Show that after m = deg ψA,b steps of the Arnoldi process with q1 = b/kbk2 we
have fm = kbk2 Qf (Hm )e1 ≡ f (A)b.
13.2. Show that dim(Km (A, b)) = min(m, deg ψA,b ).
13.3. Show that if H ∈ Cn×n is upper Hessenberg with hj+1,j 6= 0 for all j then H
is nonderogatory, that is, no eigenvalue appears in more than one Jordan block.
13.4. (T. Lippert, communicated by A. Frommer) The following recursive algorithm
computes x = Xk b, where Xk is the kth iterate from the Newton–Schulz iteration
(5.22) for the matrix sign function, using only matrix–vector products.

1 function x = N S(A, b, k)
2 if k = 1
3 x = 12 A(3b − A(Ab))
4 else
5 x = N S(A, b, k − 1)
6 x = N S(A, x, k − 1)
7 x = 3b − x
8 x = 12 N S(A, x, k − 1)
9 end

Explain why the algorithm works and evaluate its cost, comparing it with the cost of
computing Xk and then forming Xk b.
13.5. (Research problem) Obtain error bounds for the Arnoldi approximation
(13.7) for general A and general f .
13.6. (Research problem) For f (x) = ex the Arnoldi approximation (13.7) re-
quires the computation of eH , where H is upper Hessenberg. Can Algorithm 10.20
(scaling and squaring) be usefully adapted to take advantage of the Hessenberg struc-
ture?
Problems 311

The basic idea of the Krylov subspace techniques considered in this paper is to
approximately project the exponential of the large matrix onto a small Krylov subspace.
The only matrix exponential operation performed
is therefore with a much smaller matrix.
— Y. SAAD, Analysis of Some Krylov Subspace Approximations to
the Matrix Exponential Operator (1992)

We can consider each Krylov subspace method as

theoretically made from two main parts,
a Krylov subspace process, and a subproblem solution.
— C. C. PAIGE, Krylov Subspace Processes,
Krylov Subspace Methods, and Iteration Polynomials (1994)

The Gaussian rules were out of favor in the days of

paper-and-pencil scientific computation,
as the numbers involved were helter-skelter irrational decimals,
impossible to remember and difficult to enter on a keyboard without error.
— PHILIP J. DAVIS and AVIEZRI S. FRAENKEL, Remembering Philip Rabinowitz (2007)
Chapter 14
Miscellany

This final chapter treats a few miscellaneous topics that do not fit elsewhere in the
book.

14.1. Structured Matrices

In many applications the matrices that arise are structured and when structure is
present it is natural to try to exploit it. The benefits to be gained include faster and
more accurate algorithms and reduced storage, as well as more physically meaningful
solutions in the presence of rounding and truncation errors. The interaction of struc-
ture with matrix functions is a deep and fascinating subject. A great deal is known,
but many open questions remain. A thorough treatment of structure is outside the
scope of this book, and indeed no single book could adequately cover all its aspects.
In this section we mention just a few examples of structured f (A) problems and give
pointers to the literature. The following list is not meant to be complete, but is
intended to give a flavor of this very active area of research.

14.1.1. Algebras and Groups

If A ∈ Cn×n is unitary then A−1 = A∗ . Assuming that A has no eigenvalues on R− ,
we can take the (principal) square root in this relation to obtain

∗
A−1/2 = A−1 )1/2 = A∗ )1/2 = A1/2 ,

which shows that A1/2 is also unitary. Therefore the square root function preserves
the property of being unitary. However, underlying this result is a much more general
one. Suppose that M −1 A∗ M = A−1 for some nonsingular M ∈ Cn×n . Then we have

1/2
1/2
∗
A−1/2 = M −1 A∗ M = M −1 A∗ M = M −1 A1/2 M, (14.1)

which shows that A1/2 satisfies a relation of the same form. Thus the same proof
has shown preservation of structure for a much wider class of matrices. This line
of thinking suggests that there are gains to be made by carrying out analysis in a
suitably general setting. A setting that has proved very fruitful is a scalar product
space and its associated structures.
Let K = R or C and consider a scalar product on Kn , that is, a bilinear or
sesquilinear form h·, ·iM defined by any nonsingular matrix M : for x, y ∈ Kn ,
 T
x M y for real or complex bilinear forms,
hx, yiM =
x∗ M y for sesquilinear forms.

313
314 Miscellany

Table"14.1. Structured
# matrices associated with some scalar products.
. 1 ,
» – » –
0 In Ip 0
R= .. J=
−In 0
, Σp,q =
0 −Iq
with p + q = n.
1

Space M Automorphism Group Jordan Algebra Lie Algebra

G = {G : G⋆ = G−1 } J = {S : S ⋆ = S} L = {K : K ⋆ = −K}

Bilinear forms
Rn I Real orthogonals Symmetrics Skew-symmetrics
Cn I Complex orthogonals Complex symmetrics Cplx skew-symmetrics
Rn Σp,q Pseudo-orthogonals Pseudosymmetrics Pseudoskew-symmetrics
Cn Σp,q Cplx pseudo-orthogonals Cplx pseudo-symm. Cplx pseudo-skew-symm.
Rn R Real perplectics Persymmetrics Perskew-symmetrics
R2n J Real symplectics Skew-Hamiltonians Hamiltonians
C2n J Complex symplectics Cplx J-skew-symm. Complex J-symmetrics

Sesquilinear forms
Cn I Unitaries Hermitian Skew-Hermitian
Cn Σp,q Pseudo-unitaries Pseudo-Hermitian Pseudoskew-Hermitian
C2n J Conjugate symplectics J-skew-Hermitian J-Hermitian

The adjoint of A with respect to the scalar product h·, ·iM , denoted by A⋆ , is uniquely
defined by the property hAx, yiM = hx, A⋆ yiM for all x, y ∈ Kn . It can be shown that
the adjoint is given explicitly by

M −1 AT M for bilinear forms,
A⋆ = (14.2)
M −1 A∗ M for sesquilinear forms.

Associated with h·, ·iM is an automorphism group G, a Lie algebra L, and a Jordan
algebra J, which are the subsets of Kn×n defined by

G := {G : hGx, GyiM = hx, yiM ∀x, y ∈ Kn } = G : G⋆ = G−1 , (14.3)
n
 ⋆
L := {L : hLx, yiM = −hx, LyiM ∀x, y ∈ K } = L : L = −L , (14.4)
n
 ⋆
J := {S : hSx, yiM = hx, SyiM ∀x, y ∈ K } = S : S = S . (14.5)

G is a multiplicative group, while L and J are linear subspaces. Table 14.1 shows a
sample of well-known structured matrices in G, L, or J associated with some scalar
products.
In this language, (14.1) and its analog with “∗” replaced by “T ”, shows that A ∈ G
implies A1/2 ∈ G, that is, the square root function preserves matrix automorphism
groups. One can ask which other functions preserve G. A thorough investigation of
this question is given by Higham, Mackey, Mackey, and Tisseur [283, ].
Mappings between G, L, and J provide examples of interplay between these struc-
⋆
tures. For example, if A ∈ L then X = eA ∈ G, because X ⋆ = (eA )⋆ = eA = e−A =
−1
X . This generalizes the familiar property that the exponential of a skew-Hermitian
matrix is unitary. A more general setting in which similar questions can be consid-
ered is a Lie group and its associated Lie algebra (not necessarily arising from a scalar
14.1 Structured Matrices 315

product), and the observation just made is a special case of the well-known one that
the exponential map takes the Lie algebra into the corresponding Lie group. This
mapping is important in the numerical solution of ODEs on Lie groups by geometric
integration methods. For details, see Hairer, Lubich, and Wanner [239, ], Iserles,
Munthe-Kaas, Nørsett, and Zanna [313, ], and Iserles and Zanna [314, ].
Two particularly important classes in Table 14.1 are the Hamiltonian matrices
and the symplectic matrices. Results on logarithms of such matrices can be found in
Dieci [152, ] and [153, ]. The result that every real skew-Hamiltonian matrix
has a real Hamiltonian square root is proved by Faßbender, Mackey, Mackey, and Xu
[182, ], while Ikramov proves the corresponding result for complex matrices [308,
].
Some of the techniques developed in these contexts can be applied even more
generally. For example, A ∈ Cn×n is centrosymmetric if JAJ = A (where J is
defined in Table 14.1) and by the same argument as above, JA1/2 J = A1/2 , so the
square root function preserves centrosymmetry.
Perturbation analysis of matrix functions can be done in such way as to restrict
perturbations to those that maintain the structure of the matrices being perturbed,
yielding structured condition numbers. Such analysis has been carried out by Davies
for the Jordan and Lie algebras J and L [134, ].
The polar decomposition A = U H interacts with the above structures in an inter-
esting way. First, various results can be proved about how U and H inherit structure
from A. Second, the polar decomposition can be generalized to a decomposition
A = U H in which U ∈ G and H ∈ J, subject to suitable conditions on A. For
details and further references see Higham [277, ], Higham, Mackey, Mackey, and
Tisseur [282, ], [283, ], and Mackey, Mackey, and Tisseur [400, ] (see, in
particular, the discussion in Section 6), and for similar considerations in a Lie group
setting see Iserles and Zanna [314, ] and Munthe-Kaas, Quispel, and Zanna [443,
].

14.1.2. Monotone Functions

A function f : Cn×n → Cn×n is nonnegative if f (A) ≥ 0 whenever A ≥ 0 and mono-
tone if f (A) ≥ f (B) whenever A ≥ B, where the ordering is the positive semidefinite
ordering on Hermitian matrices (see Section B.12). Much is known about functions
that are monotone or nonnegative, and this area overlaps substantially with the topic
of matrix inequalities. A sample result is that the function f (A) = Ar is monotone
for r ∈ [0, 1]. We cite just the books by Donoghue [163, ] and Bhatia [64, ],
[65, ], which contain many further references to the literature.

14.1.3. Other Structures

A Stieltjes matrix is a symmetric positive definite matrix A ∈ Rn×n such that aij ≤ 0
for i 6= j. Micchelli and Willoughby [427, ] characterize functions f that preserve
the class of Stieltjes matrices.
A ∈ Cn×n is coninvolutory if AA = I. Horn and Johnson [296, , Cor. 6.4.22]
shows that every coninvolutory matrix has a coninvolutory square root.
Toeplitz matrix structure is not in general preserved by matrix functions. But
a primary matrix function of a triangular Toeplitz matrix is again triangular and
Toeplitz, which follows from the facts that f (A) is a polynomial in A and that the
sum and product of two triangular Toeplitz matrices are triangular and Toeplitz.
316 Miscellany

More generally, a primary matrix function of a circulant matrix is a circulant. This

follows from the fact that circulant matrices C ∈ Cn×n are precisely the matrices
diagonalized by the discrete Fourier transform (DFT) matrix (1.17):

Fn CFn−1 = D = diag(di ).

Hence f (C) = Fn−1 f (D)Fn is a circulant.

14.1.4. Data Sparse Representations

Some other types of structure—very different to those considered in the previous

sections—can be put under the heading of “data sparse”, which describes a matrix
that is not necessarily sparse in the usual sense but can be represented by a relatively
small number of parameters. One such class of matrices is the semiseparable matrices,
which can be defined as the matrices whose upper and lower triangles agree with the
corresponding triangles of two rank r matrices, where r is the semiseparability rank
[597, ]. An example of such a matrix is the inverse of a tridiagonal matrix having
no zeros on the subdiagonal or superdiagonal, which is semiseparable with r = 1 (see,
e.g., Meurant [425, ]). There are other ways in which low rank blocks can render
a matrix data sparse, and it may be sufficient that low rank approximations can be
found. The hierarchical, H-matrix format is a recursive representation that employs
low rank approximations, and matrices arising in certain applications are well suited
to this representation; for an overview see Börm, Grasedyck, and Hackbusch [76,
].

14.1.5. Computing Structured f (A) for Structured A

Assuming we know that A and f (A) are both structured (possibly with different, but
related structures), what methods are available that exploit the structure? This is
a very general question about which much is known. We give a small selection of
references.
For algorithms for computing the square root or logarithm of orthogonal and
unitary matrices, or more generally matrices in a group G, see Cardoso, Kenney, and
Silva Leite [93, ], Cheng, Higham, Kenney, and Laub [107, ], and Higham,
Mackey, Mackey, and Tisseur [283, ].
If f preserves a certain structure, do the Padé approximants to f also preserve the
structure? It is well known that diagonal Padé approximants rm to the exponential
have the property that r(A) is unitary if A is skew-Hermitian. Dieci [152, ,
Thm. 2.2] shows a kind of converse: that diagonal Padé approximants to the logarithm
yield skew-Hermitian matrices when evaluated at unitary matrices.
Computing functions of matrices stored in a data sparse format in a way that
exploits the structure is relatively little explored to date, other than of course for the
matrix inverse. For some work on functions of H-matrices see Baur and Benner [47,
], Gavrilyuk, Hackbusch, and Khoromskij [209, ], and Grasedyck, Hackbusch,
and Khoromskij [228, ].
14.2 Exponential Decay of Functions of Banded Matrices 317

14.2. Exponential Decay of Functions of Banded Matrices

Let A be the 6 × 6 symmetric tridiagonal matrix

 
4 1 0 0 0 0
1 4 1 0 0 0
 
0 1 4 1 0 0
A= .
0 0 1 4 1 0
 
0 0 0 1 4 1
0 0 0 0 1 4

The magnitudes of three functions of A are as follows, where the elements are shown
to one significant figure:
 
9e+1 8e+1 3e+1 1e+1 3e+0 5e-1
 8e+1 1e+2 9e+1 4e+1 1e+1 3e+0
 
 3e+1 9e+1 1e+2 9e+1 4e+1 1e+1
e =
A
 1e+1
,
 4e+1 9e+1 1e+2 9e+1 3e+1
 3e+0 1e+1 4e+1 9e+1 1e+2 8e+1
5e-1 3e+0 1e+1 3e+1 8e+1 9e+1
 
1e+0 3e-1 3e-2 6e-3 1e-3 2e-4
 3e-1
 1e+0 3e-1 4e-2 6e-3 1e-3 

 3e-2 3e-1 1e+0 3e-1 4e-2 6e-3 
,
log(A) = 
 6e-3
 4e-2 3e-1 1e+0 3e-1 3e-2 

 1e-3 6e-3 4e-2 3e-1 1e+0 3e-1 
2e-4 1e-3 6e-3 3e-2 3e-1 1e+0
 
2e+0 3e-1 2e-2 2e-3 4e-4 7e-5
 3e-1
 2e+0 3e-1 2e-2 2e-3 4e-4 

 2e-2 3e-1 2e+0 3e-1 2e-2 2e-3 
.
A 1/2
=
 2e-3
 2e-2 3e-1 2e+0 3e-1 2e-2 

 4e-4 2e-3 2e-2 3e-1 2e+0 3e-1 
7e-5 4e-4 2e-3 2e-2 3e-1 2e+0

In all three cases there is decay of the elements away from the main diagonal, the
rate of decay depending on the function. This is a general phenomenon for symmetric
band matrices and is not limited to matrices that are diagonally dominant.

Theorem 14.1. Let f be analytic in an ellipse containing Λ(A) and let A ∈ Cn×n be
Hermitian and of bandwidth m (aij = 0 for |i − j| > m). Then f (A) = (fij ) satisfies
|fij | ≤ Cρ|i−j| , where C is a constant and ρ = q 1/m , where q ∈ (0, 1) depends only
on f .

Proof. See Benzi and Golub [57, ].

The theorem shows that the elements of f (A) are bounded in an exponentially
decaying manner away from the diagonal, with the bound decreasing as the bandwidth
decreases. Note that this does not necessarily mean that “decay to zero” is observed
318 Miscellany

in practice, as the following example illustrates:

   
0 0.5 0 0 0 0 2e+0 1e+1 5e+1 6e+2 8e+2 6e+2
 0.5 1 3 0 0 0   1e+1 3e+2 1e+3 2e+4 3e+4 2e+4
   
 0 3 1 1 0 0   5e+1 1e+3 6e+3 9e+4 1e+5 9e+4
A= , e = 
A
 6e+2

 0 0 1 1 10 0   2e+4 9e+4 1e+6 2e+6 1e+6

   8e+2 3e+4 1e+5 2e+6 2e+6 2e+6
0 0 0 10 1 10
0 0 0 0 10 2 6e+2 2e+4 9e+4 1e+6 2e+6 1e+6

Further decay results applying to general A and with a certain “graph theoretic
distance” replacing |i − j| in the exponent are provided by Benzi and Razouk [58,
].
Bounds of the same form as in Theorem 14.1 specialized to the exponential and
for general A are obtained by Iserles [312, ].

14.3. Approximating Entries of Matrix Functions

A variation on the f (A) problem is to approximate not f (A) or f (A)b but rather
individual entries of f (A), or perhaps a function of those entries such as a bilinear
form g(u, v) = uT f (A)v. For u = ei and v = ej , g(u, v) = f (A)ij .
An approach to this problem developed by Golub and his collaborators for Her-
mitian matrices expresses g(u, v) as a Riemann–Stieltjes integral with respect to a
suitable measure and approximates the integral by Gauss-type quadrature rules. The
Lanczos process provides an efficient means to construct these rules using only matrix–
vector products with A. See Bai, Fahey, and Golub [32, ], Bai, Fahey, Golub,
Menon, and Richter [33, ], Bai and Golub [34, ], [35, ], and Golub and
Meurant [223, ].
Extensions to non-Hermitian matrices using the Arnoldi process have been devel-
oped by Calvetti, Kim, and Reichel [91, ] and Guo and Renaut [235, ].
Appendix A
Notation

Page numbers denote the pages on which a definition can be found.

A≥B componentwise inequality, p. 329; also positive (semi)definite or-

dering (Löwner (partial) ordering), p. 330
1/2
A
√ the principal square root, p. 20
A (arbitrary) square root, p. 133
A1/p principal pth root, p. 174
A+ pseudoinverse, p. 325
⌈x⌉ ceiling function, p. 321
C the complex numbers
condabs absolute condition number, p. 56
condrel relative condition number, p. 55
conv convex hull, p. 102
det(A) determinant
e (vector) [1, 1, . . . , 1]T
{λi (A)} the set of eigenvalues of A
f [. . .] divided difference, p. 332
# geometric mean, p. 46
◦ Hadamard (Schur) product, p. 62; also composition of functions
⌊x⌋ floor function, p. 321
κ(A) matrix condition number, p. 328
⊗ Kronecker product, p. 331
⊕ Kronecker sum, p. 331
Lf Fréchet derivative of f , p. 56
Λ(A) spectrum (set of eigenvalues), p. 322
Λǫ (A) ǫ-pseudospectrum, p. 47
log(A) principal logarithm, p. 20
ni the index of an eigenvalue λi , pp. 3, 322
null(A) null space of matrix, p. 321
R the real numbers
R− closed negative real axis
range(A) range of matrix, p. 321
ρ(A) spectral radius, p. 322
sign(A) matrix sign function, pp. 39, 107
trace(A) trace of matrix, p. 321
vec vec operator, p. 331
Z the integers

319
Appendix B
Background: Definitions and Useful Facts

This appendix collects together a variety of basic definitions, properties, and results
from matrix analysis and numerical analysis that are needed in the main text. Terms
being defined are in boldface. Where specific references are not given, general sources
of further information are Horn and Johnson [295, ] and Lancaster and Tismenet-
sky [371, ].

B.1. Basic Notation

R denotes the real numbers and R− the closed negative real axis. C denotes the
complex numbers. Z denotes the integers.
The real and imaginary parts of a complex number z are denoted by Re z and
Im z, respectively.
The identity matrix is Im,n = (δij ) ∈ Rm×n and In ≡ In,n . The ith column of
In is denoted by ei . Similarly, the m × n zero matrix is 0m,n .
For A = (aij ) ∈ Cm×n , AT denotes the transpose (aji ) and A∗ denotes the
T
conjugate transpose A = (aji ). Pn
The trace of A ∈ Cn×n is trace(A) = i=1 aii . Two key properties are that
trace(A) is the sum of the eigenvalues of A and that trace(AB) = trace(BA) for all
A ∈ Cm×n and B ∈ Cn×m .
The determinant of A ∈ Cn×n is denoted by det(A). Key properties are
det(AB) = det(A) det(B), det(αA) = αn det(A) (α ∈ C), and that det(A) is the
product of the eigenvalues of A.
The floor and ceiling functions are defined by ⌊x⌋ = max{ n ∈ Z : n ≤ x } and
⌈x⌉ = min{ n ∈ Z : x ≤ n }, respectively.
The empty set is denoted by ∅.
The range of A ∈ Cm×n is range(A) = { Ax : x ∈ Cn } and the null space of A
is null(A) = { x ∈ Cn : Ax = 0 }.
The dimension dim(V ) of a vector space V is the maximum number of linearly
independent vectors in V .
The notation 1: n denotes the sequence 1, 2, 3, . . . , n. A(i: j, r: s) denotes the sub-
matrix of A comprising the intersection of rows i to j and columns r to s.
The notation X = O(kEk) denotes that kXk ≤ ckEk for some constant c for all
sufficiently small kEk, while X = o(kEk) means that kXk/kEk → 0 as E → 0.

B.2. Eigenvalues and Jordan Canonical Form

Let A ∈ Cn×n . The scalar λ ∈ C is an eigenvalue with corresponding eigenvector
x 6= 0 if Ax = λx. The eigenvalues are the zeros of the characteristic polynomial

321
322 Background: Definitions and Useful Facts

q(t) = det(tI − A), which has degree n. The minimal polynomial of A is the unique
monic polynomial ψ of lowest degree such that ψ(A) = 0.
The set of eigenvalues of A, called the spectrum, is denoted by Λ(A) = {λ1 , . . . , λn }.
We sometimes denote by λi (A) the ith eigenvalue of A in some (usually arbitrary)
ordering. The eigenspace of A corresponding to an eigenvalue λ is the vector space
{ x ∈ Cn : Ax = λx }.
The algebraic multiplicity of λ is its multiplicity as a zero of the characteristic
polynomial q(t). The geometric multiplicity of λ is dim(null(A − λI)), which is
the dimension of the eigenspace of A corresponding to λ, or equivalently the number
of linearly independent eigenvectors associated with λ.
Any matrix A ∈ Cn×n can be expressed in the Jordan canonical form

Z −1 AZ = J = diag(J1 , J2 , . . . , Jp ), (B.1a)
 
λk 1
 .. 
 λk . 
Jk = Jk (λk ) =  ..  ∈ Cmk ×mk , (B.1b)
 . 1 
λk
where Z is nonsingular and m1 + m2 + · · · + mp = n. The λk are the eigenvalues of
A. The matrices Jk are called Jordan blocks. The Jordan matrix J is unique up to
the ordering of the blocks Jk , but the transforming matrix Z is not unique.
In terms of the Jordan canonical form (B.1), the algebraic multiplicity of λ is the
sum of the dimensions of the Jordan blocks in which λ appears, while the geometric
multiplicity is the number of Jordan blocks in which λ appears.
An eigenvalue is called semisimple if its algebraic and geometric multiplicities
are the same or, equivalently, if it occurs only in 1 × 1 Jordan blocks. An eigenvalue is
defective if is not semisimple, that is, if it appears in a Jordan block of size greater
than 1, or, equivalently, if its algebraic multiplicity exceeds its geometric multiplicity.
A matrix is defective if it has a defective eigenvalue, or, equivalently, if it does not
have a complete set of linearly independent eigenvectors.
A matrix is derogatory if in the Jordan canonical form an eigenvalue appears
in more than one Jordan block. Equivalently, a matrix is derogatory if its minimal
polynomial (see page 4) has degree less than that of the characteristic polynomial.
The properties of being defective and derogatory are independent. A matrix that is
not derogatory is called nonderogatory.
A matrix is diagonalizable if in the Jordan canonical form the Jordan matrix J
is diagonal, that is, mk ≡ 1 and p = n in (B.1).
The index of an eigenvalue is the dimension of the largest Jordan block in which
it appears. The index of a matrix is the index of its zero eigenvalue, which can be
characterized as the smallest nonnegative integer k such that rank(Ak ) = rank(Ak+1 ).
The spectral radius ρ(A) = max{ |λ| : det(A − λI) = 0 }.
The field of values of A ∈ Cn×n is the set of all Rayleigh quotients:
 ∗ 
z Az n
F (A) = : 0 6= z ∈ C .
z∗z

The set F (A) is convex, and when A is normal it is the convex hull of the eigenvalues.
For a Hermitian matrix F (A) is a segment of the real axis and for a skew-Hermitian
matrix it is a segment of the imaginary axis.
The following eigenvalue localization theorem is often useful.
B.3 Invariant Subspaces 323

Theorem B.1 (Gershgorin, 1931). The eigenvalues of A ∈ Cn×n lie in the union of
the n disks in the complex plane
 n
X 
Di = z ∈ C : |z − aii | ≤ |aij | , i = 1: n.
j=1
j6=i

B.3. Invariant Subspaces

A subspace X of Cn is an invariant subspace for A if AX ⊆ X , that is, x ∈ X
implies Ax ∈ X .
Let the columns of X ∈ Cn×p form a basis for X . Then X is an invariant subspace
for A if and only if AX = XB for some B ∈ Cp×p . When the latter equation holds,
the spectrum of B is contained within that of A. More specifically (see Problem B.1),
if (λ, x) is an eigenpair of A with x ∈ X then (λ, X + x) is an eigenpair of B (where
X + is the pseudoinverse of X, defined in Section B.6.1), while if (λ, z) is an eigenpair
of B then (λ, Xz) is an eigenpair of A. References: Stewart [538, , Sec. 4.1] and
Watkins [607, , Sec. 6.1], [608, , Sec. 2.1].

B.4. Special Classes of Matrices

T ∈ Cn×n is upper (lower) triangular if tij = 0 for i > j (i < j). A triangular
matrix T is strictly triangular if tii = 0 for all i. T is upper trapezoidal if tij = 0
for i > j. T = (Tij ) is upper quasi-triangular if it is block upper triangular with
the diagonal blocks Tii either 1 × 1 or 2 × 2.  
A ∈ C2n×2n is Hamiltonian if JA is Hermitian, where J = −I0n I0n , and sym-
plectic if AT JA = J.
Let A ∈ Cn×n for the rest of this section.
A is diagonalizable if there exists a nonsingular X such that X −1 AX = D is
diagonal.
A is symmetric if AT = A. A is Hermitian if A∗ = A. A Hermitian matrix A
is positive definite if x∗ Ax > 0 for all nonzero x ∈ Cn and positive semidefinite
if x∗ Ax ≥ 0 for all nonzero x ∈ Cn . Equivalent conditions for a Hermitian A to be
positive (semi)definite are that λi (A) > 0 for all i (λi (A) ≥ 0 for all i).
The product of two Hermitian positive definite matrices has positive eigenvalues
(proof: AB = A1/2 · A1/2 BA1/2 · A−1/2 , so AB is similar to a Hermitian positive
definite matrix).
A ∈ Rn×n is orthogonal if AT A = I, that is, AT = A−1 . A ∈ Cn×n is unitary
if A∗ A = I, that is, A∗ = A−1 .
A permutation matrix is a matrix obtained by reordering the columns of the
identity matrix. A permutation matrix is orthogonal.
A is normal if AA∗ = A∗ A, or, equivalently, if A is diagonalizable by a unitary
matrix. There are many other equivalent conditions for a matrix to be normal [231,
], [295, , Sec. 2.5].
A is nilpotent if Ak = 0 for some k ≥ 0. Such a matrix has only zero eigenvalues
and its index of nilpotency is the index of the zero eigenvalue, which is the smallest
k such that Ak = 0.
A is involutory if A2 = I and idempotent if A2 = A.
324 Background: Definitions and Useful Facts

We write block diagonal matrices

 
D1
diag(D1 , . . . , Dm ) = diag(Di ) =  .. ,
.
Dm

which illustrates the convention that blank entries in a matrix denote zeros.
A Toeplitz matrix is one for which the entries along each diagonal are constant:
aij = ri−j for some parameters rk . A circulant matrix is a special type of Toeplitz
matrix in which each row is a cyclic permutation one element to the right of the row
above. Examples:
   
a b c a b c
T = d a b, C = c a b.
e d a b c a

A ∈ Cn×n is strictly diagonally dominant (by rows) if

X
|aij | < |aii |, i = 1: n. (B.2)
j6=i

A Vandermonde matrix has the form

 
1 1 ... 1
 α1 α2 ... αn 
V = V (α1 , α2 , . . . , αn ) = 
 ... .. ..  ∈ Cn×n .
. ... . 
α1n−1 α2n−1 ... αnn−1

A linear system V T x = b is called a dual Vandermonde system. A confluent Vander-

monde matrix Vc is obtained when some successive αi repeat and each column cor-
responding to a repeated αi is obtained by “differentiating” the previous one. For
example,
 
1 0 0 1
α 1 0 α2 
Vc (α1 , α1 , α1 , α2 ) =  12 .
α1 2α1 2 α22
3 2 3
α1 3α1 6α1 α2
For more details see [276, , Chap. 22].

B.5. Matrix Factorizations and Decompositions

An LU factorization of A ∈ Cn×n is a factorization A = LU , where L is unit lower
triangular (lower triangular with 1s on the diagonal) and U is upper triangular. In
practice, unless A has special properties that guarantee the existence and numerical
stability of the factorization, partial pivoting is used, which produces the factorization
P A = LU , where P is a permutation matrix.
Every Hermitian positive definite A ∈ Cn×n has a unique Cholesky factoriza-
tion A = R∗R, where R ∈ Cn×n is upper triangular with positive diagonal elements.
Any A ∈ Cm×n with m ≥ n has a QR factorization A = QR, where Q ∈ Cm×m
is unitary and R is upper trapezoidal, that is, R = [R1T 0]T , where R1 ∈ Cn×n is
upper triangular.
B.6 Pseudoinverse and Orthogonality 325

Any A ∈ Cn×n has a Schur decomposition Q∗ AQ = T , where Q is unitary and

T upper triangular. The eigenvalues of A appear on the diagonal of T . For each k,
the leading k columns of Q span an invariant subspace of A.
Any A ∈ Rn×n has a real Schur decomposition QT AQ = R, where Q is real
orthogonal and R is real upper quasi-triangular with any 2 × 2 diagonal blocks having
complex conjugate eigenvalues. If A is normal then the 2 × 2 blocks Rii have the form
[295, , Thm. 2.5.8]
 
a b
Rii := , b 6= 0.
−b a

Any Hermitian A ∈ Cn×n has a spectral decomposition A = QΛQ∗ , where Q

is unitary and Λ = diag(λi ), with λi the eigenvalues of A, which are real.
Any A ∈ Cm×n has a singular value decomposition (SVD)

A = U ΣV ∗ , Σ = diag(σ1 , σ2 , . . . , σp ) ∈ Rm×n , p = min(m, n),

where U ∈ Cm×m and V ∈ Cn×n are unitary and σ1 ≥ σ2 ≥ · · · ≥ σp ≥ 0. The σi

are the singular values of A, and they are the nonnegative square roots of the p
largest eigenvalues of A∗A. The columns of U and V are the left and right singular
vectors of A, respectively. The rank of A is equal to the number of nonzero singular
values. If A is real, U and V can be taken to be real. The essential SVD information is
contained in the compact SVD A = U ΣV ∗ , where U ∈ Cm×r , Σ = diag(σ1 , . . . , σr ),
V ∈ Cn×r , and r = rank(A). The extent to which the SVD is unique is described in
Problem B.11.
For more details on the above factorizations and decompositions see Demmel [145,
], Higham [276, ], Stewart [538, ], Trefethen and Bau [572, ], and
Watkins [607, ].

B.6. Pseudoinverse and Orthogonality

B.6.1. Pseudoinverse
The pseudoinverse X ∈ Cn×m of A ∈ Cm×n is the unique matrix satisfying the four
Moore–Penrose conditions

(i) AXA = A, (ii) XAX = X,

(B.3)
(iii) AX = (AX)∗ , (iv) XA = (XA)∗ .

The pseudoinverse is denoted by A+ . If A = U ΣV ∗ is an SVD then A+ = V Σ + U ∗ ,

where Σ + = diag(σ1−1 , . . . , σr−1 , 0, . . . , 0) ∈ Rn×m and r = rank(A). When A has
rank n, A+ = (A∗ A)−1 A∗ .
In general it is not the case that (AB)+ = B + A+ for A ∈ Cm×n , B ∈ Cn×p . A
sufficient condition for this equality to hold is that rank(A) = rank(B) = n; see, e.g.,
Campbell and Meyer [92, , Sec. 1.4].
For any A ∈ Cm×n we have (see Problem B.3)

range(A∗A) = range(A∗ ). (B.4)

For more on the pseudoinverse see any textbook on numerical linear algebra or
Ben-Israel and Greville [52, ].
326 Background: Definitions and Useful Facts

B.6.2. Projector and Orthogonal Projector

Let S be a subspace of Cm and let PS ∈ Cm×m .

• PS is the projector onto S if range(PS ) = S and PS2 = PS . The projector is

not unique.

• PS is the orthogonal projector onto S if range(PS ) = S, PS2 = PS , and

PS∗ = PS . The orthogonal projector is unique (see Problem B.9). In terms of
the pseudoinverse, Prange(A) = AA+ and Prange(A∗ ) = A+ A (see Problem B.10).

An excellent treatment of projectors can be found in Meyer [426, , p. 386].

B.6.3. Partial Isometry

U ∈ Cm×n is a partial isometry (or subunitary matrix) if kU xk2 = kxk2 for all
x ∈ range(U ∗ ); in words: U is norm-preserving on the orthogonal complement of its
null space. Several equivalent conditions for a matrix to be a partial isometry are
collected in the following result.

Lemma B.2. For U ∈ Cm×n each of the following conditions is equivalent to U being
a partial isometry:
(a) U + = U ∗ ,
(b) U U ∗ U = U ,
(c) the singular values of U are all 0 or 1.

Proof. The equivalences are straightforward and can be obtained, for example,
using Problem B.4.
For more about partial isometries see Erdelyi [177, ] and Campbell and Meyer
[92, , Chap. 4].

B.7. Norms
A matrix norm on Cm×n is a function k · k : Cm×n → R satisfying the following
conditions:

1. kAk ≥ 0 with equality if and only if A = 0.

2. kαAk = |α| kAk for all α ∈ C, A ∈ Cm×n .

3. kA + Bk ≤ kAk + kBk for all A, B ∈ Cm×n (the triangle inequality).

Given a vector norm on Cn , the corresponding subordinate matrix norm is

defined by
kAxk
kAk = max .
x6=0 kxk

The most important case is where the vector norm is the p-norm, defined by kxkp =
Pn

p 1/p
i=1 |xi | (1 ≤ p < ∞) and kxk∞ = maxi=1:n |xi |, which gives the matrix p-
norm kAkp = maxx6=0 kAxkp /kxkp . The three most useful subordinate matrix norms
B.7 Norms 327

Table B.1. Constants αpq such that kAkp ≤ αpq kAkq , A ∈ Cm×n .
q
1 2 ∞ F
√ √
1 1 m m m
√ √
2 n 1 m 1
√ √
p ∞ n n 1 n
√ p √
F n rank(A) m 1

are
Pm
kAk1 = max1≤j≤n i=1 |aij |, “max column sum”,

1/2
kAk2 = ρ(A∗A) = σmax (A), spectral norm,
Pn ∗
kAk∞ = max1≤i≤m j=1 |aij | = kA k1 , “max row sum”,
These norms are all within a constant factor (depending only on the dimensions of
A) of each other, as summarized in Table B.1.
The Frobenius norm for A ∈ Cm×n is
X m X n 1/2
2

1/2
kAkF = |aij | = trace(A∗A) . (B.5)
i=1 j=1
√
For any subordinate matrix norm kIn k = 1, whereaskIn kF = n. When results in
this book assume the norm to be subordinate it is usually because they use kIk = 1.
If kABk ≤ kAk kBk for all A ∈ Cm×n and B ∈ Cn×p then the norm is called
consistent. (Note that there can be three different norms in this relation, so we
should say the norms are consistent.) All subordinate matrix norms are consistent,
as is the Frobenius norm.
A vector norm on Cn is absolute if k |x| k = kxk for all x ∈ Cn . A matrix norm
subordinate to an absolute vector norm satisfies [295, , Thm. 5.6.37]

k diag(di )k = max |di |. (B.6)

i

A norm on Cm×n for which kU AV k = kAk for all unitary U ∈ Cm×m and V ∈
n×n
C and all A ∈ Cm×n is called a unitarily invariant norm. Such a norm is a
function only of the singular values of A, and hence it satisfies kAk = kA∗ k. To be
more precise, for a unitarily invariant norm kAk is a symmetric gauge function of
the singular values. A symmetric gauge function is an absolute, permutation-invariant
norm on Rn , that is, a vector norm g such that g(|x|) = g(x), where |x| = (|xi |), and
g(x) = g(P x) for any permutation matrix P . A vector norm is absolute if and only
if it is monotone, where a monotone norm is one for which |x| ≤ |y| ⇒ kxk ≤ kyk.
For more details, see Horn and Johnson [295, , Sec. 7.4] or Stewart and Sun [539,
].
For any unitarily invariant norm [296, , Cor. 3.5.10],

kABCk ≤ kAk2 kBk kCk2 , A ∈ Cr×m , B ∈ Cm×n , C ∈ Cn×s (B.7)

(in fact, any two of the norms on the right-hand side can be 2-norms).
The following result is used in Chapter 8.
328 Background: Definitions and Useful Facts

Theorem B.3 (Mirsky). Let A, B ∈ Cm×n have SVDs with diagonal matrices ΣA , ΣB ∈
Rm×n , where the diagonal elements are arranged in nonincreasing order. Then kA −
Bk ≥ kΣA − ΣB k for every unitarily invariant norm.
Proof. See Mirsky [432, , Thm. 5] or Horn and Johnson [295, , Thm.
7.4.51].
The condition number (with respect to inversion) of A ∈ Cn×n is κ(A) =
kAk kA−1 k.
For any consistent norm and A ∈ Cn×n
ρ(A) ≤ kAk (B.8)
(see Problem B.5).
For any A ∈ Cn×n and ǫ > 0 there is a consistent matrix norm (depending on A)
such that kAk ≤ ρ(A) + ǫ [295, , Lem. 5.6.10]. In particular, if ρ(A) < 1 there is
a consistent matrix norm such that kAk < 1. This fact provides an easy way to prove
the result that
ρ(A) < 1 ⇒ lim Ak = 0. (B.9)
k→∞

Lemma B.4. For A, B ∈ Cn×n and any consistent matrix norm we have


kAm − B m k ≤ kA − Bk kAkm−1 + kAkm−2 kBk + · · · + kBkm−1 .
Pm−1
Proof. The bound follows from the identity Am −B m = i=0 Ai (A−B)B m−1−i ,
which can be proved by induction. The inductive step uses Am − B m = (Am−1 −
B m−1 )B + Am−1 (A − B).

B.8. Matrix Sequences and Series

(k)
Let Ak = (aij ) ∈ Cm×n . The sequence of matrices {Ak }∞
k=1 is said to converge to A =
(k)
limk→∞ Ak if aij → aij as k → ∞ for all i and j. Equivalently, limk→∞ kA−Ak k = 0
for some matrix norm.
P∞ Pk
The notation k=1 Ak is an abbreviation for limk→∞ i=1 Ai . A sufficient con-
dition for convergence of a matrix series is given in the following result [433, ,
Thm. 11.2.1].
P∞ P∞
Lemma B.5. The series k=1 Ak converges if the series k=1 kAk k converges for
some matrix norm.

B.9. Perturbation Expansions for Matrix Inverse

Let k · k be any subordinate matrix norm. If kEk < 1 then I + E is nonsingular and
k(I + E)−1 k ≤ 1/(1 − kEk). If A is nonsingular and kA−1 Ek < 1 then A + E is
nonsingular and
(A + E)−1 = A−1 − A−1 EA−1 + O(kEk2 ). (B.10)
A useful identity related to (B.10) is
A−1 − B −1 = A−1 (B − A)B −1 .
B.10 Sherman–Morrison–Woodbury Formula 329

B.10. Sherman–Morrison–Woodbury Formula

If A ∈ Cn×n is nonsingular and v ∗ A−1 u 6= −1 then A + uv ∗ is nonsingular and
A−1 uv ∗ A−1
(A + uv ∗ )−1 = A−1 − . (B.11)
1 + v ∗ A−1 u
This is known as the Sherman–Morrison formula.
More generally, If U, V ∈ Cn×p and I + V ∗ A−1 U is nonsingular then A + U V ∗ is
nonsingular and
(A + U V ∗ )−1 = A−1 − A−1 U (I + V ∗ A−1 U )−1 V ∗ A−1 , (B.12)
which is the Sherman–Morrison–Woodbury formula.
For historical background on these formulae see Zhang [624, , Chaps 1 and
6].

B.11. Nonnegative Matrices

For A ∈ Rm×n , |A| denotes the matrix with (i, j) element |aij |. For A ∈ Rm×n , A ≥ 0
(A > 0) denotes that aij ≥ 0 (aij > 0) for all i and j. We say A is nonnegative if
A ≥ 0 and positive if A > 0.
For A, B ∈ Rn×n ,
|A| ≤ B ⇒ ρ(A) ≤ ρ(|A|) ≤ ρ(B) (B.13)
[295, , Thm. 8.1.18]. A corollary of this result is that ρ(A) ≥ maxi aii [295, ,
Cor. 8.1.20].
A ∈ Rn×n is a stochastic matrix if A ≥ 0 and the row sums are all equal to 1
(i.e., Ae = e, where e = [1, 1, . . . , 1]T ).
A ∈ Cn×n with n ≥ 2 is reducible if there is a permutation matrix P such that
 
T A11 A12
P AP = ,
0 A22
where A11 and A22 are square, nonempty submatrices. A is irreducible if it is not
reducible.
The following two results summarize some of the key spectral properties of non-
negative matrices proved by Perron and Frobenius.

Theorem B.6 (Perron–Frobenius). If A ∈ Rn×n is nonnegative and irreducible then

(a) ρ(A) > 0,
(b) ρ(A) is an eigenvalue of A,
(c) there is an x > 0 such that Ax = ρ(A)x,
(d) ρ(A) is an eigenvalue of algebraic multiplicity 1.

Theorem B.7 (Perron–Frobenius). If A ∈ Rn×n is nonnegative then

(a) ρ(A) is an eigenvalue of A,
(b) there is a nonnegative eigenvector of A corresponding to ρ(A).
For more details of Perron–Frobenius theory see Berman and Plemmons [60, ,
Chap. 2], Horn and Johnson [295, , Chap. 8], or Lancaster and Tismenetsky [371,
, Chap. 15].
330 Background: Definitions and Useful Facts

B.12. Positive (Semi)definite Ordering

For Hermitian matrices X and Y , X ≥ Y denotes that X − Y is positive semidefinite
while X > Y denotes that X − Y is positive definite. (Note that this usage of
inequality signs differs from that in the previous section. Which usage is in effect
will usually be clear from the context.) X ≤ Y (X < Y ) is equivalent to Y ≥ X
(Y > X). The ordering ≥ on positive semidefinite matrices is sometimes called the
Löwner (partial) ordering. Some of the properties of inequalities on real numbers
carry over to Hermitian matrices. For this book we need the results below.

Lemma B.8. Let A, B, C ∈ Cn×n be Hermitian. Then

(a)
A ≥ 0, B ≥ 0, AB = BA ⇒ AB ≥ 0; (B.14)

(b)
A ≥ 0, B ≤ C, AB = BA, AC = CA ⇒ AB ≤ AC; (B.15)

(c) if {Xk } is a sequence of Hermitian matrices satisfying B ≤ Xk+1 ≤ Xk for

all k then Xk → X∗ ≥ B as k → ∞.

Proof. (a) AB is Hermitian, since (AB)∗ = B ∗ A∗ = BA = AB. Let X and Y be

the Hermitian positive semidefinite square roots of A and B. X and Y are functions
of A and B, so X = p(A), Y = q(B) for some polynomials p and q, and therefore X
and Y commute. Then AB = X 2 Y 2 = (XY )2 , which implies that the eigenvalues of
AB are nonnegative, as required.
(b) The inequality can be written A(C − B) ≥ 0, which follows from (a).
(c) The following proof is due to Roy Mathias. For arbitrary x ∈ Cn consider
the scalar sequence αk (x) = x∗ Xk x. Since x∗ Bx ≤ αk+1 (x) ≤ αk (x), we have
αk (x) → α∗ (x) ≥ x∗ Bx. Suppose, first, that the Xk are all real. Consider the
polarization identity

1

Re x∗ Ay = (x + y)∗ A(x + y) − (x − y)∗ A(x − y)
4
and set A = Xk . For any given x, y ∈ Rn the right-hand side has a limit, α∗ (x, y),
as k → ∞, from the result just shown. Hence x∗ Xk y → α∗ (x, y). Now put x = ei ,
y = ej , to deduce that (Xk )ij → α∗ (ei , ej ) =: (X∗ )ij . Clearly, X∗ ≥ B. If the Xk are
not all real then an analogous proof holds using the complex polarization identity

1
x∗ Ay = (x + y)∗ A(x + y) − (x − y)∗ A(x − y)
4  
− i (x + iy)∗ A(x + iy) − (x − iy)∗ A(x − iy) .

Theorem B.9. If A ≥ B ≥ 0 then Ap ≥ B p for p ∈ [0, 1].

Proof. See, e.g., Bhatia [64, , Thm. V.1.9], [65, , Thm. 4.2.1] or Zhan
[623, , Thm. 1.1].
B.13 Kronecker Product and Sum 331

B.13. Kronecker Product and Sum

The Kronecker product of A ∈ Cm×n and B ∈ Cp×q is A ⊗ B = (aij B) ∈ Cmp×nq .
A key property of the Kronecker product is (A ⊗ B)(C ⊗ D) = AC ⊗ BD.
The vec operator stacks the columns of a matrix into one long vector: if A =
[a1 , a2 , . . . , am ] then vec(A) = [aT1 aT2 . . . aTm ]T .
The vec-permutation matrix Π is the permutation matrix defined by the prop-
erty that vec(AT ) = Π vec(A). It satisfies (A ⊗ B)Π = Π(B ⊗ A).
We have
vec(AXB) = (B T ⊗ A) vec(X). (B.16)
If A ∈ Cn×n has eigenvalues λr and B ∈ Cm×m has eigenvalues µs then [371,
, Thm. 12.2.1]
X
k  k
X
i j
Λ cij A ⊗ B = cij λir µjs , r = 1: n, s = 1: m. (B.17)
i,j=0 i,j=0

The Kronecker sum of A ∈ Cm×m and B ∈ Cn×n is defined by A ⊕ B =

A ⊗ In + Im ⊗ B. By (B.17), the eigenvalues of A ⊕ B are λij = λi (A) + λj (B),
i = 1: m, j = 1: n.
For any p-norm we have kA ⊗ Bkp = kAkp kBkp and kA ⊕ Bkp ≤ kAkp + kBkp .
For more details of Kronecker products and sums, see Horn and Johnson [296,
, Chap. 4] or Lancaster and Tismenetsky [371, , Chap. 12]. Note that some
authors define A ⊕ B = In ⊗ A + B ⊗ Im , which is B ⊕ A in our notation; the elegant
statement of Theorem 10.9 is a reason for preferring our definition.

B.14. Sylvester Equation

The linear matrix equation
AX − XB = C, (B.18)
where A ∈ Rm×m , B ∈ Rn×n , and C ∈ Rm×n are given and X ∈ Rm×n is to be
determined, is called the Sylvester equation. By applying the vec operator, it can
be rewritten in the form

(In ⊗ A − B T ⊗ Im ) vec(X) = vec(C). (B.19)

The mn eigenvalues of the coefficient matrix are given by

λij (In ⊗ A − B T ⊗ Im ) = λi (A) − λj (B), i = 1: m, j = 1: n, (B.20)

and hence the Sylvester equation is nonsingular precisely when A and B have no
eigenvalue in common.

B.15. Floating Point Arithmetic

All rounding error analysis in the book is based on the standard model of floating
point arithmetic,

f l(x op y) = (x op y)(1 + δ), |δ| ≤ u, op = +, −, ∗, /, (B.21)

332 Background: Definitions and Useful Facts

where f l denotes the computed value of an expression and u is the unit roundoff .
In addition to (B.21), we can also use the variant
x op y
f l(x op y) = , |δ| ≤ u. (B.22)
1+δ
In IEEE double precision arithmetic, u = 2−53 ≈ 1.11 × 10−16 , while for IEEE
single precision arithmetic, u = 2−24 ≈ 5.96 × 10−8 .
This model applies to real x and y. For complex data the model must be adjusted
by increasing the bound for |δ| slightly [276, , Lem. 3.5].
Error analysis results are often stated in terms of the constants
nu cnu
γn = , γ
en = ,
1 − nu 1 − cnu
where c is a small integer constant whose precise value is unimportant. For all the
algorithms in this book we can state error bounds that are valid for both real and
complex arithmetic by using the γe notation.
The following lemma [276, , Lem. 3.1] is fundamental for carrying out round-
ing error analysis.

Lemma B.10. If |δi | ≤ u and ρi = ±1 for i = 1: n, and nu < 1, then

n
Y
(1 + δi )ρi = 1 + θn ,
i=1

where
nu
|θn | ≤ = γn .
1 − nu
Complex arithmetic costs significantly more than real arithmetic: in particular,
for scalars, a complex addition requires 3 real additions and a complex multiplication
requires 4 real multiplications and 2 real additions, though the latter can be reduced
at the cost of some stability [272, ]. The ratio of execution time of a numer-
ical algorithm implemented in real versus complex arithmetic depends on both the
algorithm and the computer, but a ratio of 4 is reasonably typical.
For the computed product C b = f l(AB) of A, B ∈ Cn×n we have kC − Ck b p ≤
en kAkp kBkp , p = 1, ∞, F .
γ
For more on floating point arithmetic and rounding error analysis see Higham
[276, ].

B.16. Divided Differences

Let x0 , x1 , . . . , xn ∈ C be ordered so that equal points are contiguous, that is,

xi = xj (i < j) ⇒ xi = xi+1 = · · · = xj . (B.23)

Divided differences of a function f at the points xk are defined recursively by

f [xk ] = f (xk ),

 f (xk+1 ) − f (xk ) ,

xk 6= xk+1 ,
f [xk , xk+1 ] = xk+1 − xk

 ′
f (xk+1 ), xk = xk+1 ,
B.16 Divided Differences 333


 f [x1 , x2 , . . . , xk+1 ] − f [x0 , x1 , . . . , xk ]

 , x0 6= xk+1 ,
xk+1 − x0
f [x0 , x1 , . . . , xk+1 ] = f (k+1) (x (B.24)

 k+1 )
 , x0 = xk+1 .
(k + 1)!

One of their main uses is in constructing the Newton divided difference form of the in-
terpolating polynomial, details of which can be found in numerical analysis textbooks,
such as Neumaier [447, , Sec. 3.1] and Stoer and Bulirsch [542, , Sec. 2.1.5].
Another important representation of a divided difference is as a multiple integral.
Assume f is k times continuously differentiable. It is easy to check that f [x0 , x1 ] =
R1 ′
0
f (x0 + (x1 − x0 )t) dt. This formula generalizes to the Genocchi–Hermite formula
Z 1 Z t1 Z t2 Z tk−1  k
X 
f [x0 , x1 , . . . , xk ] = ... f (k) x0 + tj (xj − xj−1 dtk . . . dt2 dt1 ,
0 0 0 0 j=1
(B.25)
which expresses f [x0 , x1 , . . . , xk ] as an average of f (k) over the simplex with vertices
x0 , x1 , . . . , xk . Note that this formula, like (B.29) below, does not require the xi to
be ordered as in (B.23), so it gives a meaning to divided differences for an arbitrary
ordering.
It follows from (B.25) that if f is k-times continuously differentiable then the di-
vided difference f [x0 , x1 , . . . , xk ] is a continuous function of its arguments. Moreover,
for real points xi ,

f (k) (ξ)
f [x0 , x1 , . . . , xk ] = for some ξ ∈ [mini xi , maxi xi ]. (B.26)
k!
Hence for confluent arguments we recover the confluent case in (B.24):

f (k) (x)
f [x, x, . . . , x] = . (B.27)
| {z } k!
k + 1 times

No result of the form (B.26) holds for complex xi , even for k = 1 [178, ]. It nev-
ertheless follows from (B.25) that if Ω is a closed convex set containing x0 , x1 , . . . , xk ∈
C then  
maxz∈Ω f (k) (z)
|f [x0 , x1 , . . . , xk ]| ≤ . (B.28)
k!
Yet another definition of divided differences can be given in terms of contour inte-
gration. Let f be analytic inside and on a closed contour Γ that encloses x0 , x1 , . . . , xk .
Then Z
1 f (z)
f [x0 , x1 , . . . , xk ] = dz. (B.29)
2πi Γ (z − x0 )(z − x1 ) . . . (z − xk )
The properties (B.24)–(B.27) are readily deduced from this definition.
For more details of divided difference see, for example, Conte and de Boor [113,
, Thm. 2.5], Isaacson and Keller [311, , Sec. 6.1], and Neumaier [447, ,
Thm. 3.1.8]. The formulae (B.25) and (B.29) are less commonly treated, hence we
give a list of references in which they can be found: de Boor [143, ], Gel’fond [210,
, Sec. 1.4.3], Horn and Johnson [296, , Sec. 6.1], Isaacson and Keller [311,
, Sec. 6.1], Kahan [328, ], Kahan and Fateman [329, ], Milne-Thompson
[428, , Chap. 1], and Ostrowski [455, ].
334 Background: Definitions and Useful Facts

Problems
B.1. Prove the eigenpair relations between A and B stated in Section B.3.
B.2. Use the Moore–Penrose conditions (B.3) to show that if A is Hermitian then so
is its pseudoinverse. Deduce that if A is Hermitian then A commutes with A+ .
B.3. Show that for A ∈ Cm×n , range(A∗A) = range(A∗ ).
 Ir 0
 ∗
B.4. Show that every partial isometry A ∈ Cm×n has the form A = P 0 0m−r,n−r Q ,
for some unitary P ∈ Cm×m and Q ∈ Cn×n , where r = rank(A).
B.5. Show that for any A ∈ Cn×n and any consistent matrix norm, ρ(A) ≤ kAk,
where ρ is the spectral radius.
 
B.6. Let A ∈ Cm×n . Show that A 0
= kAk for any unitarily invariant norm.

B.7. Let A ∈ Cn×n and U ∈ Cm×n , where U ∗ U = In . Show that kAk = kU Ak

for any unitarily invariant norm. (This result is immediate from the definition of
unitarily invariant norm if m = n.)
B.8. Show that for A ∈ Cm×n and B ∈ Cn×p , kABkF ≤ kAk2 kBkF and kABkF ≤
kAkF kBk2 (special cases of (B.7)).
B.9. Show that the orthogonal projector onto a subspace S is unique.
B.10. Show that AA+ and A+ A are the orthogonal projectors onto range(A) and
range(A∗ ), respectively.
B.11. Let A ∈ Cn×n have the SVD A = P ΣQ∗ with n × n factors, and let r =
rank(A). Show that if A = PeΣ Q e ∗ is another SVD then Pe = P diag(D, W1 ) and
e ∗ = diag(D∗ , W2 )Q∗ , where D ∈ Cr×r is unitary and block diagonal with dij = 0 if
Q
σi 6= σj , and W1 , W2 ∈ C(n−r)×(n−r) are arbitrary unitary matrices.
Appendix C
Operation Counts

In this appendix we summarize the cost of some standard matrix computations. The
unit of measure is the flop, which denotes any of the four elementary scalar operations
+, −, ∗, /. The operation counts assume the use of substitution for solving triangular
systems and LU factorization, and appropriate symmetric variants thereof in the
symmetric case, for solving linear systems. When complex arithmetic is involved, the
flops should be interpreted as counting operations on complex numbers.
For details of the relevant algorithms see, for example, Golub and Van Loan [224,
], Higham [276, ], Trefethen and Bau [572, ], or Watkins [607, ].
It is important to stress that the run time of an algorithm in a particular com-
puting environment may or may not be well predicted by the flop count. Different
matrix operations may run at different speeds, depending on the machine architec-
ture. In particular, matrix multiplication exploits hierarchical memory architectures
better than matrix inversion, so it is generally desirable to make algorithms matrix
multiplication-rich. See Dongarra, Duff, Sorensen and Van der Vorst [162, ] and
Golub and Van Loan [224, ] for more details.

335
336 Operation Counts

Table C.1. Cost of some matrix computations for real, n × n matrices. Here A and B are
general, nonsymmetric; T is triangular; H and M are symmetric. X denotes a matrix to be
determined; Y denotes a symmetric matrix to be determined. x, b are n-vectors.

Operation Number of flops

AB 2n3
A−1 2n3
H −1 n3
T −1 n3/3
AT n3
HM  3 n3/3
2n /3 if T1 T2 is full
T1 T2
n3/3 if T1 T2 is triangular
LU 2n3/3
Tx = b  n2
n3 for general B
TX = B
n3/3 if B has same triangular structure as T
TY = B n3
AX = B 8n3/3 a
HX = B 7n3/3 b
HY = B n3 b
AX + XB = C 60n3 c
T1 X + XT2 = C 2n3
a Assuming an LU factorization of A is computed.
b Assuming an LDLT or Cholesky factorization of H is computed.
c Assuming Schur decompositions of A and B are computed.
Operation Counts 337

Table C.2. Cost of some matrix factorizations and decompositions. A is n × n, except for
QR and SVD, where it is m × n (m ≥ n).

Factorization/decomposition Number of flops

LU factorization with partial pivoting (P A = LU ) 2n3/3
LU factorization with partial pivoting of upper n2
Hessenberg matrix (P A = LU )
Cholesky factorization (A = R∗ R) n3/3
2
Householder QR factorization (A = QR) 2n (m − n/3) for R;
4(m2 n − mn2 + n3/3) for m × m Q;
2n2 (m − n/3) for m × n Q;
2np(2m − n) for QB with m × p B
and Q held in factored form.
SVDa (A = P ΣQ∗ ) 14mn2 + 8n3 (P (:, 1: n), Σ, and Q)b
6mn2 + 20n3 (P (:, 1: n), Σ, and Q)c
Hessenberg decomposition (A = QHQ∗ ) 14n3/3 (Q and H), 10n3/3 (H only)
Schur decompositiona (A = QT Q∗ ) 25n3 (Q and T ), 10n3 (T only)
For Hermitian A:
Tridiagonal reduction (A = QT Q∗ ) 8n3/3 (Q and T ), 4n3/3 (T only)
Spectral decomposition (A = QDQ∗ ) 9n3 (Q and D), 4n3/3 (D only)
a Thesecosts are estimates taken from Golub and Van Loan [224, ].
b Golub–Reinsch SVD.
c Golub–Reinsch SVD with preliminary QR factorization.
Appendix D
Matrix Function Toolbox

The Matrix Function Toolbox contains MATLAB implementations of many of the

algorithms described in this book. The toolbox is intended to facilitate understand-
ing of the algorithms through MATLAB experiments, to be useful for research in the
subject, and to provide a basis for the development of more sophisticated implemen-
tations. The codes are “plain vanilla” versions; they contain the core algorithmic
aspects with a minimum of inessential code. In particular, the following features
should be noted.

• The codes have little error checking of input arguments.

• The codes do not print intermediate results or the progress of an iteration.

• For the iterative algorithms a convergence tolerance is hard-coded (in function

mft tolerance). For greater flexibility this tolerance could be made an input
argument.

• The codes are designed for simplicity and readability rather than maximum
efficiency.
• Algorithmic options such as preprocessing are omitted.

• The codes are intended for double precision matrices. Those algorithms in which
the parameters can be adapted to the precision have not been written to take
advantage of single precision inputs.

The contents of the toolbox are listed in Table D.1, along with a reference to the
corresponding algorithms, theorems, or sections in the book. We have not provided
codes for algorithms that are already provided as part of MATLAB. Such matrix-
function M-files are listed in Table D.2, along with a reference to the corresponding
algorithm in this book.
The Matrix Function Toolbox is available from

http://www.ma.man.ac.uk/~higham/mftoolbox

To test that the toolbox is working correctly, run the function mft test one or
more times.

339
340 Matrix Function Toolbox

Table D.1. Contents of Matrix Function Toolbox and corresponding parts of this book.

arnoldi Arnoldi iteration Algorithm 13.3

ascent seq Ascent sequence for square (singular) ma- Theorem 1.22
trix.
cosm Matrix cosine by double angle algorithm. Algorithm 12.6
cosm pade Evaluate Padé approximation to the matrix Section 12.3
cosine.
cosmsinm Matrix cosine and sine by double angle al- Algorithm 12.8
gorithm.
cosmsinm pade Evaluate Padé approximations to matrix co- Section 12.6
sine and sine.
expm cond Relative condition number of matrix expo- Algorithms 3.17
nential. and 10.27
expm frechet pade Fréchet derivative of matrix exponential via Algorithm 10.27
Padé approximation.
expm frechet quad Fréchet derivative of matrix exponential via Algorithm 10.26
quadrature.
fab arnoldi f (A)b approximated by Arnoldi method. Section 13.2.2
funm condest1 Estimate of 1-norm condition number of Algorithm 3.22
matrix function.
funm condest fro Estimate of Frobenius norm condition num- Algorithm 3.20
ber of matrix function.
funm ev Evaluate general matrix function via eigen- Section 4.5
system.
funm simple Simplified Schur–Parlett method for func- Section 4.6
tion of a matrix.
logm cond Relative condition number of matrix loga- Section 11.8
rithm.
logm frechet pade Fréchet derivative of matrix logarithm via Algorithm 11.12
Padé approximation.
logm iss Matrix logarithm by inverse scaling and Algorithm 11.10
squaring method.
logm pade pf Evaluate Padé approximation to matrix log- Section 11.4
arithm by partial fraction form.
mft test Test the Matrix Function Toolbox.
mft tolerance Convergence tolerance for matrix iterations.
polar newton Polar decomposition by scaled Newton iter- Algorithm 8.20
ation.
polar svd Canonical polar decomposition via singular Theorem 8.1
value decomposition.
polyvalm ps Evaluate polynomial at matrix argument by Section 4.2
Paterson–Stockmeyer algorithm.
power binary Power of matrix by binary powering (re- Algorithm 4.1
peated squaring).
quasitriang struct Block structure of upper quasitriangular
matrix.
riccati xaxb Solve Riccati equation XAX = B in posi- Algorithm 6.22
tive definite matrices.
rootpm newton Coupled Newton iteration for matrix pth (7.19)
root.
rootpm real P th root of real matrix via real Schur form. Algorithm 7.6
rootpm schur newton Matrix pth root by Schur–Newton method. Algorithm 7.15
rootpm sign Matrix pth root via matrix sign function. Algorithm 7.17
Matrix Function Toolbox 341

Table D.1. (continued )

signm Matrix sign decomposition. Algorithm 5.5

signm newton Matrix sign function by Newton iteration. Algorithm 5.14
sqrtm db Matrix square root by Denman–Beavers it- (6.15), (6.28)
eration.
sqrtm dbp Matrix square root by product form of (6.17), (6.29)
Denman–Beavers iteration.
sqrtm newton Matrix square root by Newton iteration (6.12), (6.27)
(unstable).
sqrtm newton full Matrix square root by full Newton method. (6.11)
sqrtm pd Square root of positive definite matrix via Algorithm 6.21
polar decomposition.
sqrtm pulay Matrix square root by Pulay iteration. (6.48)
sqrtm real Square root of real matrix by real Schur Algorithm 6.7
method.
sqrtm triang min norm Estimated minimum norm square root of Algorithm 6.23
triangular matrix.
sylvsol Solve Sylvester equation.

Table D.2. Matrix-function-related M-files in MATLAB and corresponding algorithms in this

book.

expm Matrix exponential. Algorithm 10.20

funm Evaluate general matrix function. Algorithm 9.6
logm Matrix logarithm. Algorithm 11.11
polyvalm Evaluate polynomial with matrix Algorithm 4.2
argument.
sqrtm Matrix square root. Algorithm 6.3
Appendix E
Solutions to Problems

Remember that very little is gained by

reading the solution to a problem
before seriously attempting to solve it.
— WILLIAM BRIGGS, Ants, Bikes, and Clocks:
Problem Solving for Undergraduates (2005)

If repeated efforts have been unsuccessful,

the reader can afterwards analyze the solution . . . with more incisive attention,
bring out the actual principle which is the salient point,
assimilate it, and commit it to his memory.
— GEORGE PÓLYA AND GABOR SZEGÖ,
Problems and Theorems in Analysis I (1998)

1.1. Consider two Jordan canonical forms

A = ZJZ −1 = W JW −1 (E.1)

(by incorporating a permutation matrix in W we can assume without loss of generality that
J is the same matrix in both cases). The definition gives f1 (A) = Zf (J)Z −1 , f2 (A) =
W f (J)W −1 and we need to show that f1 (A) = f2 (A), that is, W −1 Zf (J)Z −1 W = f (J), or
X −1 f (J)X = f (J) where X = Z −1 W . Now by (E.1) we have X −1 JX = J, which implies
f (J) = f (X −1 JX) = X −1 f (J)X, the last equality following from Definition 1.2. Hence
f1 (A) = f2 (A), as required.
Further insight can be gained by noting that the general form of X is given by Theo-
rem 1.25 and that this form commutes with f (J) given by (1.4).

1.2. We have −Jk = DJk (−λk )D, where D = diag(1, −1, 1, . . . , (−1)mk −1 ). Hence f (−Jk ) =
Df (Jk (−λk ))D, from which the result follows. Alternatively, we can write −Jk = −λk I +Nk ,
where Nk is zero except for a superdiagonal of −1s, and expand f (−λk I + Nk ) in a Taylor
series (cf. (1.5)).

1.3. By Theorem 1.38, it suffices to show that AX = XA implies f (A)X = Xf (A). If X

is nonsingular then Xf (A)X −1 = f (XAX −1 ) = f (AXX −1 ) = f (A), so Xf (A) = f (A)X.
If X is singular we can choose ǫ > 0 such that X + ǫI is nonsingular. Clearly, X + ǫI
commutes with A, and so by the previous part (X + ǫI)f (A) = f (A)(X + ǫI), which implies
Xf (A) = f (A)X, as required.

1.4. We will use properties from Theorem 1.13 without comment.

(a) The Jordan canonical form of A can be ordered so that A = ZJZ −1 with j11 = λ
and Z(:, 1) = x. Now f (A) = Zf (J)Z −1 and so f (A)Z = Zf (J). The first column of
this equation gives f (A)x = f (λ)x, as required. For an alternative proof, let p interpolate
to the values of f on the spectrum of A. Then p(λ) = f (λ). Since Ak x = λk x for all k,
f (A)x = p(A)x = p(λ)x = f (λ)x, as required.

343
344 Solutions to Problems

(b) Setting (λ, x) ≡ (α, e) in (a) gives the row sum result. If A has column sums α then
AT e = αe and applying the row sum result to AT gives f (α)e = f (AT )e = f (A)T e. So f (A)
has column sums f (α).

1.5. There is clearly a lower bound on the degrees of polynomials p such that p(A) = 0, and
by the Cayley–Hamilton theorem the characteristic polynomial is a candidate for having
minimal degree, so the minimum is at most n. Hence ψ exists, as we can always normalize
to obtain a monic polynomial. Let p1 and p2 be two monic polynomials of lowest degree.
Then p3 = p2 − p1 is a polynomial of degree less than p1 and p2 and p3 (A) = 0; hence p3 = 0,
i.e., p1 = p2 . So ψ is unique.

1.6. Let B = I − 31 A. From ψ(A) = 0 we obtain 9B 2 − 15B + 5I = 0. Premultiplying by

B −1 and rearranging gives B −1 = (3A + 6I)/5, as required.

1.7. It is easiest to use the polynomial interpolation definition (1.4), which says that cos(πA) =
p(A), where p(1) = cos π = −1, p′ (1) = −π sin π = 0, p(2) = cos 2π = 1. Writing
p(t) = a + bt + ct2 we have
2 32 3 2 3
1 1 1 a −1
40 1 254 b 5 = 4 05,
1 2 4 c 1
which can be solved to give p(t) = 1 − 4t + 2t2 . Hence
2 3
−3 0 0 4
6 0 −1 0 07
2 6
cos(πA) = p(A) = I − 4A + 2A = 4 7.
0 0 −1 05
−2 0 0 3

Evaluating cos(πA) from its power series would be much more complicated.

1.8. From the spectral properties of uv ∗ identified in Section 1.2.5 we deduce that the char-
acteristic polynomial is p(t) = tn−1 (t−v ∗ u) and the minimal polynomial is ψ(t) = t(t−v ∗ u).
As a check, we have ψ(uv ∗ ) = uv ∗ (uv ∗ − (v ∗ u)I) = v ∗ u(uv ∗ − uv ∗ ) = 0.

1.9. Let the eigenvalues of A be λ1 and λ2 . We have the explicit formula

f (A) = f (λ1 )I + f [λ1 , λ2 ](A − λ1 I). (E.2)

To understand the formula it is useful to examine the Jordan

p form of A.
ˆ ˜
The matrix A = ac db has eigenvalues λ = (a + d ± (a − d)2 + 4bc)/2. Hence A has
distinct eigenvalues unless (a − d)2 + 4bc = 0. Suppose A has a double eigenvalue, λ. If A
is diagonalizable then A = X diag(λ, λ)X −1 = λI, so A must be diagonal. This means that
we can characterize the Jordan structure in terms of the elements of A as follows:

1 if (a − d)2 + 4bc 6= 0
2 A has distinct eigenvalues
3 else if b = c = 0
4 A has a double eigenvalue in two 1 × 1 Jordan blocks and A = λI (λ = a = d)
5 else
6 A has a Jordan block of size 2
7 end

In the second of the three cases, the derivative f ′ (λ) appears to enter in the formula (E.2),
even though the theory says it shouldn’t. The appearance is illusory, because A − λ1 I = 0.
Solutions to Problems 345

1.10. The formula is trivial for β = 0, so assume β 6= 0. The matrix A = αI + βJ is

symmetric, with n − 1 eigenvalues α and one eigenvalue α + nβ. The interpolation conditions
(1.7) are p(α) = f (α), p(α + nβ) = f (α + nβ). Hence

t − (α + nβ) t−α
p(t) = f (α) + f (α + nβ).
−nβ nβ

Thus
βJ − nβI βJ
f (A) = f (α) + f (α + nβ) = f (α)I + n−1 (f (α + nβ) − f (α))J.
−nβ nβ

Of course, the formula is a special case of (1.16).

1.11. With p(t) = t we have, in the notation of (1.7), p(λi ) = λi , i = 1: s; p′ (λi ) = 1 if

ni ≥ 1, i = 1: s; p(j) (λi ) = 0 for 2 ≤ j ≤ ni − 1, i = 1: s.

1.12. The interpolating polynomial in Definition 1.4 is

t − λ2 t − λ1
p(t) = f (λ1 ) + f (λ2 ) .
λ1 − λ2 λ2 − λ1
Hence
f (λ1 ) f (λ2 )
f (A) = p(A) = (A − λ2 I) + (A − λ1 I).
λ1 − λ2 λ2 − λ1

1.13. For the Jordan canonical form definition (1.2), use Theorem 1.25 to write down the
form of a general matrix B that commutes with A. The verification of f (A)B = Bf (A) then
reduces to the fact that upper triangular Toeplitz matrices commute.
For the interpolation definition (1.4) the result is immediate: f (A) is a polynomial in A
and so commutes with B since A does.
For the Cauchy integral definition (1.11) we have, using (zI − A)B = B(zI − A),
Z
1
f (A)B = f (z)(zI − A)−1 B dz
2πi Γ
Z
1
= B f (z)(zI − A)−1 dz = Bf (A).
2πi Γ

1.14. The Jordan structure of A cannot be reliably computed in floating point arithmetic,
but the eigenvalues can (in the sense of obtaining backward stable results). Therefore we
can find the Hermite interpolating polynomial p satisfying p(j) (λi ) = f (j) (λi ), j = 0: ji − 1,
i = 1: s, where λ1 , . . . , λs are the distinct eigenvalues and λi has algebraic multiplicity ji .
This polynomial p satisfies more interpolation conditions than in (1.7), but nevertheless
p(A) = f (A) (see Remark 1.5). Perhaps the most elegant way to obtain p is in the divided
difference form (1.10), which needs no attempt to identify repeated eigenvalues.

1.15. The Jordan canonical form can be written as

2 32 ∗ 3
0 1 0 u /(u∗ u)
∗
A = uv = [ u v/(v v) X ] 4 0 0
∗
054 v∗ 5,
0 0 0 Y

where X and Y are chosen so that AX = 0, [ u v/(v ∗ v) X ] is nonsingular, and

2 ∗ 3
u /(u∗ u)
[ u v/(v v) X ] 4
∗
v ∗ 5 = I. (E.3)
Y
346 Solutions to Problems

Hence
2 32 ∗ 3
f (0) f ′ (0) 0 u /(u∗ u)
f (A) = [ u v/(v ∗ v) X ]4 0 f (0) 0 54 v∗ 5
0 0 f (0) Y
uu∗ vv ∗
= f (0) ∗
+ f ′ (0)uv ∗ + f (0) ∗ + f (0)XY.
u u v v
But XY = I − uu∗ /(u∗ u) − vv ∗ /(v ∗ v) from (E.3), and hence f (A) = f (0)I + f ′ (0)uv ∗ .

1.16. The formula can be obtained from the polynomial interpolation definition by noting
that:
(a) For v ∗ u 6= 0, A = αI + uv ∗ has a semisimple eigenvalue α of multiplicity n − 1 and
an eigenvalue α + v ∗ u.
(b) For v ∗ u = 0, A has n eigenvalues α; n − 2 of these are in 1 × 1 Jordan blocks and
there is one 2 × 2 Jordan block.
` ´−1
The Sherman–Morrison formula is obtained by writing (A+uv ∗ )−1 = A(I +A−1 u·v ∗ ) =
(I + A−1 u · v ∗ )−1 A−1 and applying (1.16) with f (x) = x−1 .
ˆc˜
1.17. Write A = λIn + eTn . Then, by (1.16),
0
» – » –
c T f (λ)In−1 f ′ (λ)c
f (A) = f (λ)In + f [λ, λ] en = .
0 0 f (λ)

This of course agrees with the result of applying Theorem 1.21.

1.18. It is easy to see that for scalars x and y, p(x)−p(y) = q(x, y)(x−y) for some polynomial
q. We can substitute tI for x and A for y to obtain

p(t)I − p(A) = q(tI, A)(tI − A). (E.4)

If p(A) = 0 then we have p(t)(tI − A)−1 = q(tI, A), so that p(t)(tI − A)−1 is a polynomial
in t. Conversely, if p(t)(tI − A)−1 is a polynomial in t then from (E.4) it follows that
p(A)(tI − A)−1 = p(t)(tI − A)−1 − q(tI, A) is a polynomial. Since p(A) is a constant this
implies that p(A) = 0.
To obtain the Cayley–Hamilton theorem set p(t) = det(tI − A). From the formula
B −1 = adj(B)/ det(B), where the adjugate adj is the transpose of the matrix of cofactors, it
follows that p(t)(tI − A)−1 = adj(tI − A) is a polynomial in t, so p(A) = 0 by the first part.

1.19. By Theorem 1.40 there is a unitary U such that U ∗ AU = R and U ∗ BU

Q= S with R and
S upper triangular. Thus f (x, y) = det(U (xR − yS)U ∗ ) = det(xR − yS) = ni=1 (xrii − ysii ).
Hence
n
Y n
Y n
Y
f (B, A) = (Brii − Asii ) = (U (Srii − Rsii )U ∗ ) = U · (Srii − Rsii ) · U ∗ .
i=1 i=1 i=1
Qn ∗
Now f (B, A) is of the form U · i=1 Ti · U , where Ti is Qupper triangular with zero (i, i)
element, and it follows from [295, , Lem. 2.4.1] that ni=1 Ti = 0. If A and B do not
commute we can modify the proof to use the generalized Schur decomposition Q∗ AZ = T ,
Q∗ BZ = S, with Q and Z unitary and T and S upper triangular, but the proof then fails
at the second equality in the displayed equation.

1.20. Let Z −1 AZ = J = diag(Jk ) be a Jordan canonical form. Then Z −1 (−A)Z = −J

and so Z −1 f (−A)Z = f (−J) = diag(f (−Jk )). Now f (−z) = ±f (z) implies f (j) (−z) =
±(−1)j f (j) (z), and from (1.34) we see that f (−Jk ) = ±f (Jk ). Hence diag(f (−Jk )) =
diag(±f (Jk )) = ±f (J), and the result follows.
Solutions to Problems 347

If f is analytic, a shorter proof is obtained from the Cauchy integral definition. Let Γ be
a closed contour enclosing Λ(A) and Γ1 its reflection in the imaginary axis, which encloses
Λ(−A). From (1.12),
Z
1
f (−A) = f (z)(zI + A)−1 dz
2πi Γ1
Z
1
= f (−w)(wI − A)−1 dw (w = −z)
2πi Γ
Z
1
=± f (w)(wI − A)−1 dw = ±f (A).
2πi Γ

1.21. Assume P 6= 0 and P 6= I, since otherwise the result

ˆ ˜is immediate. Then the minimal
polynomial is t2 − t = 0, which implies that Z −1 P Z = I0 00 for some nonsingular Z. Hence
ˆ ˜ −1
aI + bP = Z (a+b)I
0 aI
0
Z , so
» – „ » –«
f (a + b)I 0 [f (a + b) − f (a)]I 0
f (aI + bP ) = Z Z −1 = Z f (a)I + Z −1
0 f (a)I 0 0
= f (a)I + (f (a + b) − f (a))P.

1.22. No. f (A) is a polynomial in A and so cannot equal A∗ in general (consider triangular
A). However, if A is normal then A = QDQ∗ for some unitary Q and D = diag(λi ), and for
f (λ) = λ we have f (A) = Qf (D)Q∗ = QDQ∗ = A∗ . As a check, for unitary matrices we
have f (A) = A∗ = A−1 and for skew-Hermitian matrices f (A) = A∗ = −A, so f is clearly a
matrix function in both cases.

1.23. We have 2 z −1 z −2 ... z −n 3

6 z −1 ... z −n+1 7
(zI − A)−1 = 6
4 .. .. 7 .
. . 5
z −1
Hence
2 z j−1 z j−2 ... z j−n 3
Z Z z j−1 ... z j−n+1 7
1 1 6
f (A) = z j (zI − A)−1 dz = 6 .. .. 7 dz
2πi 2πi 4 . . 5
Γ Γ
z j−1
j+1
2 3
0 ... 1 ... 0
6 .. .. 7
6 0 . .7
6 7
6 .. 7
= 6 . 17 = Aj ,
6 7
6 .. .. 7
4 . .5
0

using the Cauchy residue theorem.

1.24. The formula (1.4) provides two upper triangular square roots, X1 and X2 , correspond-
ing to the two different branches of f at λk . We have to show that these are the only upper
triangular square roots. Let X be an upper triangular square root of Jk . Then, equating
(i, i) and (i, i + 1) elements in X 2 = Jk gives x2ii = λk and (xii + xi+1,i+1 )xi,i+1 = 1,
i = 1: mk − 1. The second equation implies xii + xi+1,i+1 6= 0, so from the first, x11 = x22 =
1/2
· · · = xmk ,mk = ±λk . Since xii + xjj 6= 0 for all i and j, X is uniquely determined by its
diagonal elements (see Algorithm 4.13 or Algorithm 6.3); these are the same as those of X1
or X2 , so X = X1 or X = X2 .
348 Solutions to Problems

1.25. An = 0 implies that all the eigenvalues of A are zero, and An−1 6= 0 then implies that
the Jordan form of A comprises just a single n × n Jordan block. Hence A has no square root
by Theorem 1.22. Or, in a more elementary fashion, if A = X 2 then X 2n = An = 0, which
implies X n = 0 since X is n × n. But then An−1 = X 2(n−1) = X n X n−2 = 0 since n ≥ 2,
which is a contradiction. For the last part, we have (A+cAn−1 )2 = A2 +2cAn +c2 A2n−2 = A2
for any c.

1.26. Yes, if A is nonderogatory, but in general no: Theorem 1.25 gives the general form,
from which it is clear that Z −1 XZ is necessarily block diagonal if and only if A is nonderoga-
tory.

1.27. We can write the Jordan canonical form of A as A = Z diag(J1 , 0)Z −1 , where J1
contains the Jordan blocks corresponding to the nonzero eigenvalues. With f denoting the
square root function, any primary square root of A has the form

f (A) = Zf (diag(J1 , 0))Z −1 = Z diag(f (J1 ), f (0))Z −1 = Z diag(f (J1 ), 0)Z −1 .

1/2
By Theorem 1.29, f (J1 ) = J1 is the unique square root of J1 all of whose eigenvalues lie in
1/2
the open right half-plane, and it is a primary matrix function of A. Hence Z diag(J1 , 0)Z −1
is the unique square root of the required form (being a primary square root it is independent
of the choice of Z). That this square root is real when A is real can be seen with the aid of
the real Schur decomposition or from Theorem 1.18 or Remark 1.9.

1.28. By direct calculation we find that all upper triangular square roots of A are of the
form 2 3
0 1 θ
X(θ) = ± 4 1 15,
0
where θ ∈ C is arbitrary. Now A is involutory (A = A2 ) so any polynomial in A has the
form p(A) = αI + βA, which has equal (1, 2) and (1, 3) elements. It follows that ±X(1) are
the only primary square roots of A.
Since dim(null(A)) = 2, 0 is a semisimple eigenvalue of A and hence A is diagonalizable.
Indeed 2 3
1 −1 0
V −1 AV = diag(1, 0, 0), V = 4 1 −1 −1 5 .
0 1 1
A family of nonprimary square roots is obtained as
2 3
ˆ 0 θ ˜ −1 −θ 1+θ 1
Y = V diag(1, 0 0 )V = 4 −θ 1+θ 15.
θ −θ 0

Note that for θ 6= 0, Y has a different Jordan structure than A—a phenomenon that for
matrix square roots can happen only when A is singular.

1.29. Note that A = diag(J2 (0), 0). Let X 2 = A and consider the possible Jordan block
structures of X. Applying Theorem 1.36 with f (x) = x2 to any Jordan block of X we find
that ℓ = 2 and case b(ii) must pertain, with r = 3 and p = q = 1. Hence X = ZJ3 (0)Z −1 for
some nonsingular Z. To determine (as far as possible) Z we write X 2 = A as ZJ3 (0)2 = AZ
and examine the resulting equations, which force Z to have the form
2 3
a b c
Z = 40 0 a5,
0 d e
Solutions to Problems 349

where det(Z) = −a2 d must be nonzero. Evaluating X = ZJ3 (0)Z −1 we find that some of
the remaining parameters in Z are redundant and the general form of X is
2 3
0 y x
X = 40 0 05, x 6= 0.
0 1/x 0

1.30. Let X be a square root of A and let X have Jordan matrix

diag(J1 (0), . . . , Jk (0), 0, . . . , 0, Jr+1 (λr+1 ), . . . , Jp (λp )),

where Ji (0) is of size at least 2 for i = 1: k and λi 6= 0 for i ≥ r + 1. By Theorem 1.36, Ji (0)
splits into smaller Jordan blocks when squared for i = 1: k, since f ′ (0) = 0 for f (x) = x2 .
Therefore A = X 2 has more Jordan blocks than X. But any polynomial in A has no more
Jordan blocks than A. Therefore X cannot be a polynomial in A.

1.31. The form of X is rather surprising. Since any primary square root of A is a polynomial
in A, a first reaction might be to think that X is a nonprimary square root. However, X and
A are both symmetric and structural considerations do not rule out X being a polynomial
in A. In fact, A has distinct eigenvalues (known to be 0.25 sec(iπ/(2n + 1))2 , i = 1: n [189,
]), so all its square roots are primary. X is clearly not A1/2 , since X has zero elements
on the diagonal and so is not positive definite. In fact, X has ⌈n/2⌉ positive eigenvalues
and ⌊n/2⌋ negative eigenvalues (which follows from the inertia properties of a 2 × 2 block
symmetric matrix—see, for example, Higham and Cheng [279, , Thm. 2.1]). X is an
indefinite square root that “just happens” to have a very special structure.

1.32. For any square root X we have XA = X · X 2 = X 2 · X = AX. Likewise, a logarithm

X of A is a solution of eX = A, so XA = XeX = eX X = AX. If A is nonderogatory then
by Theorem 1.37, X is a polynomial in A.

1.33. A = eB where B is the Jordan block Jn (0), as is easily seen from (1.4). B +2kπi is also
a logarithm for any k ∈ Z, and these are all the logarithms, as can be seen from Theorem 1.28
on noting that A has just one block in its Jordan canonical form since rank(A − I) = n − 1.

1.34. Let X = log A and Y = eX/2 . Then Y 2 = eX/2 eX/2 = eX = A, using Theorem 10.2.
So Y is some square root of A. The eigenvalues of Y are of the form eλi /2 , where λi is an
eigenvalue of X and has Im λi ∈ (−π, π), and so −π/2 < arg λi /2 < π/2. Thus the spectrum
of Y lies in the open right half-plane, which means that Y = A1/2 .

1.35. A1/2 is a polynomial in A and B 1/2 is a polynomial in B, so A1/2 commutes with B 1/2 .
Therefore (A1/2 B 1/2 )2 = A1/2 B 1/2 A1/2 B 1/2 = A1/2 A1/2 B 1/2 B 1/2 = AB. Thus A1/2 B 1/2
is some square root of AB. By Corollary 1.41 the eigenvalues of A1/2 B 1/2 are of the form
λi (A1/2 )λi (B 1/2 ) and so lie in the open right half-plane if the eigenvalues of A and B lie in
the open right half-plane. The latter condition is needed to ensure the desired equality, as
is clear by taking A = B.

1.36. Let C = eA . The condition on Λ(A) is equivalent to A being a primary logarithm of

C, since the nonprimary logarithms are characterized by two copies of a repeated eigenvalue
being mapped to different logarithms, which must differ by a nonzero integer multiple of
2πi. So A = log(C) = p(C) = p(eB ) = pe(B) for some polynomials p and pe, and A therefore
commutes with B.
An example showing the necessity of the condition on Λ(A) is
2 3 2 3
0 2π − 1 1 0 2π 1
A = 4 −2π 0 05, B = 4 −2π 0 0 5 ,
−2π 0 0 0 0 0
350 Solutions to Problems

for which eA = eB = I, both matrices have spectrum {0, 2πi, −2πi}, and AB 6= BA. Such
examples are easily constructed using the real Jordan form analogue of Theorem 1.27.
Schmoeger [505, ] investigates what can be concluded from eA = eB when the
condition on Λ(A) is not satisfied but A is normal.
1.37. Let λ1 , . . . , λs be the distinct eigenvalues of C = eA and denote by qi the algebraic
multiplicity of λi . The qk copies of λk are mapped to log λk + 2πrj i in A and log λk + 2πsj i
in B, for some integers rj and sj , j = 1: qk . The given condition implies that rj = sj ≡ tk
for all j, so that all copies of λk are mapped to the same logarithm. This is true for all k,
so A and B are primary logarithms with the same spectrum, which means that they are the
same matrix.
1.38. Suppose, first, that f is even. From Theorem 1.26 we know that for any square root
X of A, f (X) has the form
(j )
f (X) = ZU diag(f (Lk k ))U −1 Z −1 ,
(j )
where A = ZJZ −1 is a Jordan canonical form with J = diag(Jk ), Lk k is a square root of
(1) (2)
Jk , jk = 1, 2, and U commutes with J. But Lk = −Lk implies
(1) (2)
f (Lk ) = f (Lk ) (E.5)
(j) (j)
by Problem 1.20. Hence U diag(f (Lk ))U −1 = diag(f (Lk ))(cf. the proof of Theorem 1.26).
(jk )
Thus f (X) = Z diag(f (Lk ))Z −1 , which is the same for all choices of the jk by (E.5).
(j ) (j )
The proof for f odd is very similar, reducing to the observation that (Lk k )±1 f (Lk k )
is the same for jk = 1 as for jk = 2.
1.39. If A = log(eA ) then max{ | Im(λi )| : λi ∈ Λ(A) } < π is immediate from the definition
of the principal logarithm. Suppose the latter eigenvalue condition holds and let X = eA .
X has no eigenvalues on R− and so log(X) is defined. A is clearly some logarithm of X and
its eigenvalues satisfy Im λ ∈ (−π, π). By Theorem 1.28, every logarithm other than log(X)
has at least one eigenvalue with | Im λ| ≥ π. Therefore A must be the principal logarithm,
log(X).
1.40. By applying g to the equation f (A)f (B)f (A)−1 = f (B) and using Theorem 1.13 (c)
we obtain f (A)Bf (A)−1 = B, which can be rewritten as B −1 f (A)B = f (A). Applying g to
this equation gives B −1 AB = A, or AB = BA, as required.
1.41. Let A have the spectral decomposition A = QΛQ∗ , where, without loss of generality,
we can suppose that the eigenvalues are ordered so that Λ = diag(λ1 I1 , . . . , λm Im ), with
λ1 , . . . , λm distinct. Suppose X is a Hermitian positive definite square root of A, so that
A = X 2 . Then Λ = Q∗ AQ = Q∗ X 2 Q = (Q∗ XQ)2 =: Y 2 , where Y is Hermitian positive
definite. Now Y clearly commutes with Λ = Y 2 , so Y = diag(Yk ), where the blocking is
conformable with that of Λ. It remains to determine the Yk , which satisfy Yk2 = λk I. Now
1/2
the eigenvalues of Yk must be ±λk , and since Yk is positive definite, the eigenvalues must all
1/2 1/2 1/2
be λk . But the only Hermitian matrix with all its eigenvalues equal to λk is Yk = λk Ik .
1/2
Hence X = QY Q∗ = Q diag(λk Ik )Q∗ = QΛ1/2 Q∗ is the unique Hermitian positive definite
square root of A.
1.42. Let Y be another square root of A with eigenvalues in the open right half-plane. Since
Y A = Y 3 = AY , Y commutes with A, and hence with any polynomial in A; in particular Y
commutes with X. Therefore
(X + Y )(X − Y ) = X 2 − Y 2 − XY + Y X = X 2 − Y 2 = A − A = 0.
Since X and Y commute, the eigenvalues of X + Y are of the form λi (X) + λi (Y ), by
Corollary 1.41, and these are all nonzero since the spectra of X and Y are in the open right
half-plane. Hence X + Y is nonsingular and thus X − Y = 0, as required.
Solutions to Problems 351

1.43. Using (1.29), we have

A(αI − BA)k = (αI − AB)k A.

Let α 6= 0, x 6= 0, and (αI − BA)k x = 0. Then, on premultiplying the latter equation by

A, we obtain (αI − AB)k Ax = 0. Now Ax 6= 0, since otherwise αx = 0, which would be a
contradiction. Hence 0 6= x ∈ null((αI − BA)k ) implies 0 6= Ax ∈ null((αI − AB)k ). This
relation obviously holds with A and B interchanged. Hence null((αI − BA)k ) is isomorphic
to null((αI − AB)k ) for all k when α 6= 0. Taking α to be the nonzero eigenvalues gives the
first part of the result.
For the last part, suppose first that m > n. Since Bx = 0 implies ABx = 0, AB has at
least m − n linearly independent null vectors, namely those of B, and hence AB has a zero
eigenvalue with geometric multiplicity at least m − n. For m < n the analogous argument
applies.
ˆ ˜
1.44. Since the matrix Im A
0 In
is nonsingular, the two outer matrices in (1.35) are similar,
and so by applying f to this identity and using Theorem 1.13 (c) and (f) we obtain
» –» – „» –« » – » – „» –«
f (AB) 0 Im A AB 0 Im A Im A 0 0
=f = f
X f (0) 0 In B 0 0 In 0 In B BA
» –» –
Im A f (0) 0
=
0 In Y f (BA)

for some X and Y . Equating (1,2) blocks gives f (AB)A = Af (BA).

This proof obviously requires that f (0) be defined, which need not be the case in Corol-
ˆ AB 01.34
lary ˜ when m = n. The proof requires, moreover, that f be defined on the spectrum of
B 0
. Now from
» – » –» – » –» – » –
AB 0 A 0 B 0 B 0 A 0 BA 0
= =: CD, DC = =
B 0 I 0 0 0 0 0 I 0 0 0
ˆ ˜
together with Theorem 1.32, it follows that for f to be defined on the spectrum of AB 0
B 0
for all A and B we need that f (k) (0) be defined, where k is one more than the dimension
of the largest Jordan block corresponding to zero in BA. Consider, for example, the scalar
case A = 0, B = 1. The same requirement applies to AB. To summarize, this proof requires
the existence of one extra derivative at 0, and so yields a slightly weaker result.

1.45. In a way similar to the proof of Problem 1.44, we find

» –» – » –» –
f (AB + αIm ) 0 Im A Im A f (α)Im 0
= (E.6)
X1 f (α)In 0 In 0 In X2 f (BA + αIn )

for some X1 and X2 . Equating (1,1) blocks gives f (AB + αIm ) = f (α)Im + AX2 . To
determine X2 we use Theorem 1.13 (a) to obtain
» –» – » –
αIm 0 f (α)Im 0 f (α)Im 0
=
B BA + αIn X2 f (BA + αIn ) X2 f (BA + αIn )
» –
αIm 0
× .
B BA + αIn

Equating (2, 1) blocks gives f (α)B + (BA + αIn )X2 = αX2 + f (BA + αIn )B, or X2 =
(BA)−1 (f (BA + αIn ) − f (α)In )B. The result follows. As in Problem 1.44 this proof requires
an extra assumption: in this case the existence of an extra derivative of f at α.
352 Solutions to Problems

1.46. Setting α = 0 in (1.31) gives

` ´
f (AB) = f (0)Im + A(BA)−1 f (BA) − f (0)In B.
Postmultiplying by A gives
` ´
f (AB)A = f (0)A + A(BA)−1 f (BA) − f (0)In BA
= f (0)A + A(BA)−1 f (BA)BA − f (0)A
= Af (BA),
using Theorem 1.13 (a). Note that this proof requires the existence of f (0), which is not
needed for Corollary 1.34.
1.47. The formula (1.31) does not generalize in this way because h(X) = f (D + X) is not
a function of X according to our definition—more precisely, it does not correspond to a
scalar “stem function” evaluated at X, because of the presence of D. In the “αI → D”
generalization, the right-hand side of (1.31) would contain f (Dn + BA) where Dn is an n × n
diagonal matrix obtained from D, yet there is no reasonable way to define Dn .
1.48. Yes for n ≤ 2; no for n > 2. AB must have a Jordan form with eigenvalues all
zero, these eigenvalues appearing in 1 × 1 or 2 × 2 blocks. BA has the same eigenvalues as
AB, so the question is whether BA has Jordan blocks of dimension only 1 or 2. This can
be answered using Theorem 1.32. But working instead from first principles, note that the
dimensions of the Jordan blocks cannot exceed 3, because BABABA = B(ABAB)A = 0.
There are obviously no counterexamples with n = 2, but for n = 3 we find in MATLAB
A =
0 0 1
0 0 0
0 1 0
B =
0 0 1
1 0 0
0 0 0
>> [A*B A*B*A*B]
ans =
0 0 0 0 0 0
0 0 0 0 0 0
1 0 0 0 0 0
>> [B*A B*A*B*A]
ans =
0 1 0 0 0 1
0 0 1 0 0 0
0 0 0 0 0 0
1.49. Theorem 1.32 shows that AB and BA have the same nonzero eigenvalues, so Im + AB
and In + BA have the same nonunit eigenvalues. The result follows, since the determinant
is the product of the eigenvalues. Alternatively, take determinants in (1.36).
1.50. The answer is no. The problem can be answered from first principles, using just the
power series definition of sin A. Assume the equation has a solution A. The eigenvalues
of sin A are 1 and 1; hence the eigenvalues of A must be of the form d1 = (2p + 21 )π,
d2 = (2q + 21 )π for some integers p and q. If p 6= q then A has distinct eigenvalues and
so is diagonalizable: A = XDX −1 for some X, with D = diag(di ). But then sin A =
X sin(D)X −1 = XIX −1 = I, which is a contradiction. Hence p = q, and by ˆthe ˜same
argument the equation cannot be satisfied if A is diagonalizable. Therefore A = X θ0 θ1 X −1
for some X, with θ = (2p + 12 )π. Then, either from the power series or from (4.17), sin A =
ˆ ˜ ˆ ˜
X 10 cos1θ X −1 = X 10 01 X −1 = I, which is again a contradiction.
Solutions to Problems 353

1.51. Denote the given matrix by B, and note that B has just one n × n Jordan block in
its Jordan form, since rank(B − I) = n − 1. Let f (x) = cosh(x). The eigenvalues of A must
be f −1 (1) = 0. We have f ′ (0) = sinh(0) = 0, f ′′ (0) = cosh(0) = 1. Theorem 1.36(b) (with
ℓ = 2) says that f (A) must have more than one Jordan block. This contradicts the Jordan
structure of B and hence the equation has no solution.

1.52. (a) f (z) = 1/z for z 6= 0 and f (j) (0) = 0 for all j. Hence f is discontinuous at
zero. Nevertheless, f (A) is defined.
(b) This formula is readily verified by direct computation using (1.37). Of course, Def-
inition 1.4 directly yields AD as a polynomial in A, and this polynomial may be of much
smaller degree than xk p(x)k+1 . We note also that if B −k−1 = q(B) then AD = Ak q(A),
which provides an alternative formula.
(c) As explained in Section 1.2.5, the index of uv ∗ is 1 if v ∗ u 6= 0 and 2 otherwise. If
the index is 1 then (uv ∗ )D = 0. Otherwise, (uv ∗ )D = uv ∗ /(v ∗ u)2 , as can be obtained from
(1.14), for example.
For more on the Drazin inverse, see Campbell and Meyer [92, ].

1.53. If A ∈ Cm×n has the (compact) SVD A = P ΣQ∗ with P ∈ Cm×r , Σ = diag(σi ) ∈
Rr×r , Q ∈ Cr×n , where r = rank(A), we can define f (A) = P f (Σ)Q∗ , where f (Σ) =
diag(f (σi )). An alternative representation is f (A) = U f (H), where A = U H is a polar
decomposition (see Chapter 8) and f (H) is the usual function of a matrix. This definition,
which is given and investigated by Ben-Israel and Greville [52, , Sec. 6.7], does not
reduce to the usual definition when m = n. The definition does not appear to lead to any
useful new insights or have any significant applications.

2.1. We have
d ` −At ´
e y(t) = −Ae−At y + e−At y ′ = e−At (y ′ − Ay) = e−At f (t, y).
dt
−As
Rs
Hence e y(s) − y(0) = 0 e−At f (t, y) dt. Multiplying through by eAs and interchanging
the roles of s and t gives the result.

2.2. The function

√ g(x) = cos(xt) + x−1 sin(xt) √
is an even function of x. Hence by Prob-
lem 1.38, g( A) is the same for√all square roots A of A. Even more directly, expanding g
as a power√ series shows that g( A) is a power series in A and hence is independent of the
choice of A.

2.3. Let A have the Jordan canonical form

» –
J1 0
A = ZJZ −1 = Z Z −1 ,
0 J2

where the eigenvalues of J1 ∈ Cp×p lie in the left half-plane and those of J2 lie in the
right half-plane. Then sign(A) = Z diag(−Ip , In−p )Z −1 and, as we noted in Section 2.5,
range(W ) is the invariant subspace corresponding to the eigenvalues in the right half-plane.
Write Q = [ Q1 Q2 ], where Q1 ∈ Cn×q . Then
» –
R11 R12
W Π = [ Q 1 Q2 ] = Q1 [ R11 R12 ] .
0 0
Hence Q1 is an orthogonal basis for range(W ), which means that range(Q1 ), too, is an
invariant subspace corresponding to the eigenvalues in the right half-plane, and hence q =
n − p. Thus AQ1 = Q1 Y , for some Y whose eigenvalues lie in the right half-plane. Hence
» T – » – » –
Q1 AQ1 QT1 AQ2 Y QT1 AQ2 Y QT1 AQ2
QT AQ = = = ,
QT2 AQ1 QT2 AQ2 (QT2 Q1 )Y QT2 AQ2 0 QT2 AQ2

since QT2 Q1 = 0. The result follows.

354 Solutions to Problems

2.4. Since A and B commute, A1/2 and B 1/2 commute. Hence X = A1/2 B 1/2 satisfies
XA−1 X = A1/2 B 1/2 A−1 A1/2 B 1/2 = B 1/2 A1/2 A−1 A1/2 B 1/2 = B. Moreover, X is clearly
Hermitian, and it has positive eigenvalues because it is the product of positive definite
matrices.
For the log-Euclidean geometric mean E we have, since log(A) and log(B) commute,
1 1 1 1 1/2
) log(B 1/2 )
E(A, B) = e 2 log(A)+ 2 log(B) = e 2 log(A) e 2 log(B) = elog(A e = A1/2 B 1/2 ,

where we have used Theorems 10.2 and 11.2 (or Problem 1.34).

2.5. A # A = A follows from any of the formulae. Next, using (2.26),

` ´−1
(A # B)−1 = B(B −1 A)1/2 = (B −1 A)−1/2 B −1 = (A−1 B)1/2 B −1 = A−1 # B −1 .

Inverting both sides of XA−1 X = B gives X −1 AX −1 = B −1 and thence XB −1 X = A,

which shows that A # B = B # A.
Finally, using the first formula in (2.26), the given inequality is

1
B 1/2 (B −1/2 AB −1/2 )1/2 B 1/2 ≤ (A + B).
2

Premultiplying and postmultiplying by B −1/2 gives the equivalent inequality C 1/2 ≤ 21 (C +

I), where C = (B −1/2 AB −1/2 )1/2 is Hermitian positive definite, and this may be rewritten
(C 1/2 − I)2 ≥ 0, which is trivially true.

2.6. Since A#B = ( A

α
#B
β
)(αβ)1/2 we can assume without loss of generality that α = β = 1.
From A # B = B(B −1 A)1/2 and Problem 6.2 we have
p
det(B −1 A) B
A+
A#B = q p .
trace(B −1 A) + 2 det(B −1 A)

Let Λ(B −1 A) = {λ1 , λ2 }. Since det(B −1

pA) = det(A)/ det(B) = α2 /β 2 = 1, we have
−1 −1 −1
λ1 = 1/λ2 ≡ λ. Hence trace(B A) + 2 det(B A) = λ + λ + 2. But det(A + B) =
det(B) det(B −1 A + I) = (λ + 1)(λ−1 + 1) = λ + λ−1 + 2, which yields the result.

2.7. We have XR∗RX = B and hence (RXR∗ )2 = RBR∗ , so RXR∗ = (RBR∗ )1/2 . Hence
X = R−1 (RBR∗ )1/2 R−∗ , which is clearly Hermitian positive definite. Any of the methods
from Chapter 6 can be used to evaluate (RBR∗ )1/2 . This given formula is more efficient to
evaluate than the formulae given in Section 2.9, but Algorithm 6.22 is even better.

3.1. By determining the linear part of f (X + E) − f (X), we find that L(X, E) = 0, E, and
− sin(X)E, respectively, using (12.4) in the latter case. If the second of these expressions
seems counterintuitive, note that L(X, E) = E says that L(X) is the identity operator.

3.2. We have
f (X + tE) − f (X)
L(X, E) = lim
t→0 t
!
e − f (T )
f (T + tE)
= Q lim Q∗
t→0 t
e ∗,
= QL(T, E)Q

e = Q∗ EQ, as required.
where E
Solutions to Problems 355

3.3. If L and M are both Fréchet derivatives of f then

f (X + E) − f (X) = L(X, E) + o(kEk) = M (X, E) + o(kEk),

and so L(X, E) − M (X, E) = o(kEk). Let E = tE e with Ee fixed. Then, by the linearity
e e = o(tkEk),
of the Fréchet derivative, L(X, E) − M (X, E) = t(L(X, E) − M (X, E)) e that is,
e e −1 e e
L(X, E) − M (X, E) = t o(tkEk), Taking the limit as t → 0 gives L(X, E) = M (X, E). e
e
Since E is arbitrary, L(X) = M (X).

3.4. The Fréchet derivative is defined by

f (X + E) − f (X) − L(X, E) = o(kEk).

Setting E = tF , for fixed F and varying t, we have, since L(X) is a linear operator,

f (X + tF ) − f (X) − tL(X, F ) = f (X + tF ) − f (X) − L(X, tF ) = o(ktF k),

which implies
“ f (X + tF ) − f (X) ”
lim − L(X, F ) = 0,
t→0 t
showing that L(X, F ) is the derivative in the direction F .

3.5. The maximum Jordan block size is n. The maximum is attained when A11 and A22 are
Jordan blocks corresponding to the same eigenvalue λ and A12 = en1 eTn2 . For example, with
n1 = 2, n2 = 3,
2 3
λ 1 0 0 0
6 0 λ 1 0 0 7
6 7
A=6 6 λ 1 0 7 7
4 λ 1 5
λ
and rank(A − λI) = 4, so A has a single 5 × 5 Jordan block corresponding to λ. For a more
precise result of this form see Mathias [412, , Lem. 3.1].

3.6. It suffices to check that

∞
X i
X
f (X + E) = f (X) + ai X j−1 EX i−j + O(kEk2 ),
i=1 j=1

since L(X, E) is the linear term in this expansion. The matrix power series has the same
radius of convergence as the given scalar series (see Theorem 4.7), so if kXk < r we can
scale E → θE so that kX + θEk < r and the expansion is valid. But L(X, θE) = θL(X, E),
so the scaled expansion yields L(X, E). K(X) is obtained by using (B.16).

3.7. Straightforward from (3.24).

Pm i
3.8. If f is aP polynomial, p(t) = i=0 ai t , then as a special case of (3.24) we see that
m i−1 ′ ′
L(X, E) = E i=1 iai X = Ef (X) = f (X)E. For general f , Theorem 3.7 implies that
the Fréchet derivative L(X, E) of f is the same as that of the polynomial pX⊕X and so the
result follows from the polynomial case.

3.9. From the Cauchy integral definition (1.11) we have, for small enough kEk,
Z
1
f (X + E) = f (z)(zI − X − E)−1 dz.
2πi Γ
356 Solutions to Problems

By expanding the inverse we find that

Z
1 ˆ ˜
f (X + E) − f (X) = f (z) (zI − X − E)−1 − (zI − X)−1 dz
2πi Γ
Z
1
= f (z)(zI − X)−1 E(zI − X)−1 dz + O(kEk2 ).
2πi Γ
The expression for L(X, E) follows. If X commutes with E then all the terms in the integral
commute and by using Cauchy’s integral formula for derivatives we find that this expression
is equal to f ′ (X)E and Ef ′ (X).
3.10. Let X have the Jordan canonical form X = ZJZ −1 , where J = diag(Jk ). Without
loss of generality we can assume that u = Zei and v T = eTjZ −1 for some i and j. Then
Jei = λei and eTj J = µeTj , which means that the (i, i) entry of J marks the start of a Jordan
block and the (j, j) entry marks the end of a Jordan block. There are now two cases.
(a) i = j. In this case λ = µ and J has a 1 × 1 Jordan block λ in the (i, i) position. With
E = uv T ,
f (X + tE) = f (Z(J + tei eTi )Z −1 ) = Zf (J + tei eTi )Z −1 .
Now f (J + tei eTi ) agrees with f (J) except in the (i, i) position, where it equals f (λ + t) =
f (λ) + tf ′ (λ) + O(t2 ). Hence
f (X + tE) = Zf (J)Z −1 + [tf ′ (λ) + O(t2 )]Zei eTiZ −1
= f (X) + [tf ′ (λ) + O(t2 )]uv T
= f (X) + f ′ (λ)tE + O(t2 ),
which shows that L(X, E) = f ′ (λ)E.
(b) i 6= j. With E = uv T , we have f (X + tE) = Zf (J + tei eTj )Z −1 . It is easy to see
from Theorem 4.11 that f (J + tei eTj ) agrees with f (J) except in the (i, j) entry, which equals
f [λ, µ]. Hence
f (X + tE) = f (X) + f [λ, µ] tZei eTjZ −1 = f (X) + f [λ, µ] tE,
which means that L(X, E) = f [λ, µ]E.
4.1. The formula for f (C) is obtained by forming the powers of C, its validity following from
Theorem 4.7. We can express eC concisely as
" √ √ √ #
cosh AB A( BA)−1 sinh BA
eC = √ √ √ , (E.7)
B( AB)−1 sinh AB cosh BA
√
where X denotes any square root of X.
By introducing z = y ′ , we can rewrite (2.7) as
» –′ » –» – » – » ′–
z 0 −A z z(0) y0
= , = .
y I 0 y y(0) y0
Hence » – „» –« » ′ –
z 0 −A y0
= exp ,
y I 0 y0
and (2.8) follows on using (E.7) and cosh(iA) = cos(A), sinh(iA) = i sin(A).
4.2. Let pkm /qkm and pekm /e qkm be two [k/m] Padé approximants to f (x). Then from the
definition (4.12) it follows that
pkm qekm − pekm qkm pkm pekm
= − = O(xk+m+1 ). (E.8)
qkm qekm qkm qekm
But qkm qekm = O(1), since qkm (0) = qekm (0) = 1, and pkm qekm − pekm qkm is a polynomial
of degree at most k + m. Hence (E.8) can hold only if pkm qekm − pekm qkm = 0, that is,
pkm /qkm = pekm /e
qkm .
Solutions to Problems 357

4.3. Obtaining (4.46) is straightforward. To show the equivalence of (4.44) and (4.46) we
just have to show that F12 given by (4.46) satisfies (4.44), since we know that (4.44) has a
unique solution. Using f (Tii )Tii = Tii f (Tii ), we have

T11 F12 − F12 T22 = T11 f (T11 )X − T11 Xf (T22 ) − f (T11 )XT22 + Xf (T22 )T22
= f (T11 )(T11 X − XT22 ) − (T11 X − XT22 )f (T22 )
= f (T11 )T12 − T12 f (T22 )

by (4.45), so that F12 satisfies (4.44).

4.4. We have Y = f (T T −1 ST −1 ) = T f (T −1 S)T −1 = T XT −1 . By Theorem 1.13 (a),

T −1 SX = XT −1 S, i.e., SX = T XT −1 S = Y S. Similarly, ST −1 Y = Y ST −1 , i.e., T −1 Y T =
S −1 Y S = X, or Y T = T X.
Now xii = yii = f (sii /tii ), i = 1: n. The off-diagonal elements are computed using an
analogue of the Parlett recurrence obtained from (4.47):
» –» – » P –
sii −sjj xij sij (yii − xjj ) + j−1 (yik skj − sik xkj )
= Pk=i+1 .
tii −tjj yij tij (fii − fjj ) + j−1
k=i+1 (fik tkj − tik fkj )

This system can be solved provided the coefficient matrix is nonsingular, that is, sii tjj −
sjj tii 6= 0.
ˆ ˜ ˆ k˜
4.5. Let J(λ) = 10 11 , λ = 1, φ0 (λ) = λ. For g(x) = x2 we have x∗ ≡ xk = 1 yet Xk = 10 21
diverges. On the other hand, for g(x) = (x2 +1)/2 we have x∗ ≡ xk = 1 yet Xk ≡ J(λ) = X∗ .

4.6. Let δ −1 A = XJX −1 = X(δ −1 D +M )X −1 be the Jordan canonical form of δ −1 A, where

δ > 0, with D = diag(λi ) containing the eigenvalues of A and M the off-diagonal part of the
Jordan form. Then A = X(D + δM )X −1 , so Ak = X(D + δM )k X −1 and hence

kAk kp ≤ κp (X)(ρ(A) + δ)k (E.9)

for any p-norm. If ρ(A) < 1 we can choose δ > 0 such that ρ(A) + δ < 1 and so kAk kp is
bounded for all k. By the equivalence of matrix norms (see, e.g., [537, 1998, Thm. 4.6]) the
result holds for any matrix norm. The bound (E.9) is a special case of a result of Ostrowski
[455, , Thm. 20.1]. Notice that this argument actually shows that Ak → 0 if ρ(A) < 1.
More precisely, A is power bounded if and only if ρ(A) ≤ 1 and for any eigenvalue λ such
that |λ| = 1, λ is semisimple (i.e., λ appears only in 1 × 1 Jordan blocks). However, for our
purposes of obtaining sufficient conditions for stability, the sufficient condition ρ(A) < 1 is
all we need.

4.7. Any limit y∗ must satisfy y∗ = cy∗ + d, so that y∗ = d/(1 − c), and

yk+1 − y∗ = ck (yk − y∗ ) + y∗ (ck − c) + dk − d,

| {z }
→0

so it suffices to take d = 0 and show that yk → 0. There exist k and q ∈ [0, 1) such that
|ci | ≤ q < 1 for i ≥ k; let D = maxi≥k |di |. For E ≥ D/(1 − q), if |yi | ≤ E and i ≥ k then
|yi+1 | ≤ qE + D ≤ E. Hence with M = max{E, maxi≤k |yi |} we have |yi | ≤ M for all i.
Given ǫ > 0, there exists n(ǫ) such that for all i ≥ n(ǫ), |ci | ≤ q < 1 and |di | ≤ ǫ. Then,
for i ≥ n(ǫ), |yi+1 | ≤ q|yi | + ǫ, and so |yn(ǫ)+j | ≤ q j M + ǫ/(1 − q) ≤ 2ǫ/(1 − q) for large
enough j. It follows that yi → 0, as required.

4.8. This result is immediate from the Newton identities (4.20).

5.1. From (5.2) we have sign(A) = A(A2 )−1/2 = A · I −1/2 = A.

358 Solutions to Problems

5.2. sign(A) = sign(A−1 ). The easiest way to see this is from the Newton iteration (5.16),
because both X0 = A and X0 = A−1 lead to X1 = 12 (A + A−1 ) and hence the same sequence
{Xk }k≥1 .

5.3. Applying the Cauchy integral formula with f (z) = z −1/2 and using a Hankel contour
that goes from −∞ − 0i to 0 then around the origin and back to −∞ + 0i, with t = iz 1/2 ,
so that dt = 21 iz −1/2 dz, we have
Z ∞ Z ∞
1 1
A(A2 )−1/2 = 2 · A z −1/2 (zI − A2 )−1 dz = A (t2 I + A2 )−1 dt.
2πi 0 π 0
R
5.4. Using the matrix analogue of the formula (x2 + a2 )−1 dx = a−1 arctan(x/a), from
(5.3) we have
2 ˛∞ 2
sign(A) = arctan(tA−1 )˛0 = lim arctan(tA−1 ).
π t→∞ π

The result follows, since sign(A) = sign(A−1 ).

5.5. No: A2 differs from I in the (1,3) and (2,3) entries. A quick way to arrive at the
answer without computing A2 is to note that if A is the sign of some matrix then since
a22 = a33 = 1 we must have A(2: 3, 2: 3) = I (see the discussion following Algorithm 5.5),
which is a contradiction.

5.6. The result follows from Theorem 5.2, since C = B(A−1 B)−1/2 = B(B −1 A)1/2 .

5.8. Let Y be an approximate solution to X 2 = I and write Y + E = S, where S = sign(A).

Then I = S 2 = (Y + E)2 = Y 2 + Y E + EY + E 2 . Dropping the second order term
in E gives Newton’s method, which defines E as the solution to the Sylvester equation
Y E + EY = I − Y 2 . If Y is nonsingular then a solution to this equation (the unique solution
if λi (Y ) + λj (Y ) 6= 0 for all i and j) is E = 21 (Y −1 − Y ), whence Y + E = 21 (Y + Y −1 ),
which leads to the iteration (5.16) if we take X0 = A.

5.9. Let F = I − (S + E)2 = −(SE + ES + E 2 ). Then, for sufficiently small E,

sign(S + E) = (S + E)(I − F )−1/2

= (S + E)(I + 12 F + O(kF k2 ))
1
= S + E − (E + SES) + O(kEk2 )
2
1
= S + (E − SES) + O(kEk2 ),
2
and the result follows.

5.10. Let xk = coth θk . Then

1 cosh2 θk + sinh2 θk
coth θk+1 = (coth θk + tanh θk ) =
2 2 cosh θk sinh θk
cosh 2θk
= = coth 2θk .
sinh 2θk

Hence θk+1 = 2θk , and so θk = 2k θ0 follows by induction. Now coth x = (ex +e−x )/(ex −e−x ),
so coth 2k θ0 → 1 or −1 as k → ∞ according as Re θ0 > 0 or Re θ0 < 0, or equivalently,
Re x0 > 0 or Re x0 < 0.

5.11. Let x0 = ir0 with r0 real. It is easy to show by induction that xk = irk , where rk is
real and „ «
1 1
rk+1 = rk − .
2 rk
Solutions to Problems 359

The xk cannot converge because they are pure imaginary and the only possible limits are
±1. Setting rk = − cot(πθk ), we have
„ «
1 1
− cot(πθk+1 ) = rk+1 = − cot(πθk ) +
2 cot(πθk )
1 − cos(πθk )2 + sin(πθk )2
=
2 cos(πθk ) sin(πθk )
1 cos(2πθk )
=− 1 = − cot(2πθk ).
2 2 sin(2πθk )
So θk+1 = 2θk , or equivalently, given the periodicity of cot,
θk+1 = 2θk mod 1. (E.10)
The behaviour of the rk is completely described by this simple iteration. If θ0 has a periodic
binary expansion then periodic orbits are produced. If θ0 has a terminating binary expansion
then eventually θk = 0, that is, rk = ∞. Irrational θ0 lead to sequences rk in which the same
value never occurs twice. The mapping (E.10) is known as the Bernoulli shift [165, 1992,
Ex. 3.8].
5.13. For µ > 0 we have
n
X
g(µ) := d(µX) = (log µ + log |λi |)2 ,
i=1

and hence
n
2X
g ′ (µ) = (log µ + log |λi |).
µ i=1
Solving g ′ (µ) = 0 gives log(|λ1 | . . . |λn |) = − log µn , or µ = (|λ1 | . . . |λn |)−1/n = | det(X)|−1/n .
The last part is trivial.
5.14. (a) If a is real, x1 = sign(a). Otherwise, a = reiθ and γ0 x0 = eiθ lies on the unit
circle, x1 = cos θ is real, and hence x2 = sign(a).
(b) In view of Theorem 5.11, it suffices to consider the case where A ∈ R2×2 has a
complex conjugate pair of eigenvalues, λ = re±iθ , θ ∈ (0, π). Then µ0 X0 has eigenvalues
e±iθ and X1 has equal eigenvalues cos θ. The next iterate, X2 , has eigenvalues ±1 and the
iteration has converged, since the Jordan form of A is necessarily diagonal.
5.15. We make use of P∞ the observation that P∞ if |x| < 1 then (1√+ x)1/2 has a convergent
Maclaurin series 1 + k=1 ak x such that k=1 |ak ||x|k = 1 − 1 − x. Since sign(A) = I
k
P
we have A = (A2 )1/2 and hence A = (I + E)1/2 = I + ∞ k
k=1 ak E , since kEk < 1. Then
‚∞ ‚
‚X ‚ X ∞
p
‚ ‚ kEk
kA − Ik = ‚ ak E k ‚ ≤ |ak |kEkk = 1 − 1 − kEk = p < kEk.
‚ ‚ 1 + 1 − kEk
k=1 k=1

The upper bound in (5.40), when specialized to sign(A) = I, is kA − Ik ≤ kEk, which is up

to a factor 2 weaker. The conditions under which the two bounds hold, kI − Ak < 1 and
kI − A2 k < 1, are not easily compared, but the latter can hold when the former is far from
holding.
5.16. The test (5.46), based on the fact that Xk2 → I, is a weaker version of (5.45). While
trace(Xk2 ) can be computed on O(n2 ) flops, kXk2 − Ik can be estimated in O(n2 ) flops by
Algorithm 3.21, so there is no real advantage to (5.46) for iterations that do not form Xk2 .
The test (5.47), which exploits the fact that trace(sign(S)) is an integer, is at first sight
attractive, as it is of negligible cost and may appear to be immune to the effects of roundoff.
The major drawback of the test is that since it is based on the (sum of the) eigenvalues
of Xk , it ignores nonnormality in Xk , and so may stop too early (see Theorem 5.10). In
addition, the test could be satisfied by chance, well before convergence.
360 Solutions to Problems

5.17. Consider the sign iteration (5.16) with X0 = W . It is easy to check that the Xk are
all Hamiltonian. Write the iteration as
1
Xk+1 = (Xk + (JXk )−1 J), X0 = W,
2
or
1
Yk+1 = (Yk + JYk−1 J), Y0 = JW,
2
where the matrix Yk = JXk is Hermitian and is just Xk with its blocks rearranged and their
signs changed.
ˆ ˜
6.1. Straightforward on evaluating the (1,2) block of (5.3) with A ← I0 A0 .
ˆa b˜
6.2. Let A = c d
and δ = det(A)1/2 = (ad − bc)1/2 . Then trace(X) = ±(a + d ± 2δ)1/2 .
Hence » –
a±δ b
X = ±(a + d ± 2δ)−1/2 .
c d±δ
If a + d ± 2δ 6= 0 then A has distinct eigenvalues and all four square roots are given by this
formula. Otherwise, A has repeated eigenvalues and the formula breaks down for at least
one of the choices of sign in the term ±2δ. In this case there may be no square roots; there
may be just two (when the Jordan form of A has one 2 × 2 Jordan block), which the formula
gives; or there may be infinitely many square roots (in this case A = aI) and the formula
gives just ±a1/2 I.
ˆ ˜
6.3. Let A(ǫ) = 0ǫ 1ǫ . Then
2 3
1
1/2 ǫ1/2
A(ǫ) =4 2ǫ 1/2 5.
0 ǫ1/2

So kA(ǫ)1/2 k2 /kA(ǫ)k ≈ ǫ−1 /4 → ∞ as ǫ → 0.

6.4. Let X be a primary square root of A. If z ∈ range(A) then, for some y, z = Ay =

X 2 y = X(Xy) ∈ range(X), so range(A) ⊆ range(X). The converse inclusion (and indeed
the desired equality) is trivial if A, and hence X, is nonsingular, so we can assume A is
singular. If z ∈ range(X) then, for some y, z = Xy = p(A)y, for some polynomial p, by
Definition 1.4. Now since 0 is an eigenvalue of A, one of the interpolation conditions (1.7)
is p(0) = f (0) = 0, so p has zero constant term and thus has the form p(t) = tq(t) for a
polynomial q. Hence z = Aq(A)y ∈ range(A), giving range(X) ⊆ range(A), as required.
If A is singular and the square
ˆ root
˜ is nonprimary we can have range(A) ⊂ range(X), as
is the case for A = 0 and X = 00 10 , for example.

6.5. If there is just one zero eigenvalue then it is easy to see that Algorithm 6.3 runs to
completion and computes a primary square root. Otherwise, uii = ujj = 0 for some i 6= j
and the algorithm breaks down with division by zero at the stage where it is solving (6.5).
There are now two possibilities. First, (6.5) may be inconsistent. Second, (6.5) may be
automatically satisfied because the right-hand side is zero; if so, what value to assign to
uij in order to obtain a primary square root, or indeed any square root, may depend on
information from the later steps of the algorithm.
For example, consider
2 3 2 32 3
0 1 1 0 1 x 0 1 x
T = 4 0 1 1 5 = 4 0 1 1 5 4 0 1 1 5 = U 2.
0 0 0 0 0 0 0 0 0
T has a semisimple zero eigenvalue. Algorithm 6.3 computes the first superdiagonal of U
and then for i = 1, j = 3, (6.5) has the form 0 · x = 1 − 1 = 0. We can assign x any value,
Solutions to Problems 361

but only x = 1 produces a primary square root: if x 6= 1 then U has rank 2 and hence its
zero eigenvalue appears in a Jordan block of size 2.
For the matrix
20 1 0 1 03
6 1 0 1 07
6 7
T =6 0 1 07,
4 5
1 0
0
which has semisimple zero eigenvalue of multiplicity 3, any upper triangular square root has
the form
20 1 a 1 − a b3
2
1
6 1 0 2
07
6 7
U =6 0 1 c7,
4 5
1 0
0
where a, b, and c are arbitrary subject to ac = 0. But the constraint on a and c is not
discovered until the last step of Algorithm 6.3, and for U to be a primary square root we
need rank(U ) = 2 and hence b = c = 0 and a = 0.
The conclusion is that for singular matrices it is best to employ the reordered Schur
form, as described in Section 6.2.

6.6. Consider Algorithm 6.3 and suppose, first, that T ∈ Rn×n . The diagonal elements
satisfy
√
bii = tii (1 + δi ),
u |δi | ≤ u.
For the off-diagonal elements, using the analysis of inner products in [276, , Sec. 3.1] we
find that, whatever ordering is used in the summation,

j−1
X
(b
uii + u
bjj )b
uij (1 + θ3 ) = tij − u
bik u
bkj (1 + θn−2 ),
k=i+1

where |θk | ≤ γk . Hence U b 2 = T + ∆T , |∆T | ≤ γn−2 |U b |2 . The same analysis holds for
complex data but the constants must be increased slightly. This can be accounted for by
replacing γn−2 by γen−2 , or γ
en for simplicity. For Algorithm 6.7 the errors in forming Uii in
(6.9) mean that only a normwise bound can be obtained.

6.7. We find X1 = 21 (A + I) = 21 diag(a + 1, b + 1) = 12 diag(a + 1, −(a + 1)). Hence for k = 1,

the linear system in (6.11) is singular (in view of (B.18)–(B.20)) and the method breaks
down. However, the system is consistent and (6.12) generates one particular solution—the
diagonal one.

6.8. Since X0 does not commute with A we cannot invoke Theorem 6.9. Applying the
iteration we find that » –
1 ξθ 1−µ
X1 = , ξ= ,
0 µ 2
and hence » –
1 ξk θ
Xk = .
0 µ
Thus we have linear convergence to A1/2 if |ξ| < 1 (except when ξ = 0, i.e., µ = 1, which
gives convergence in one step) and divergence if |ξ| ≥ 1. For real µ, these two situations
correspond to −1 < µ < 3 and µ < −1 or µ ≥ 3, respectively. Hence quadratic convergence,
and even convergence itself, can be lost when X0 does not commute with A.
362 Solutions to Problems

6.9. (a) The result in the hint is proved by using the spectral decomposition C = QDQ∗
e
(Q unitary, D = diag(di ) > 0) to rewrite the equation as XD+D Xe = H,
e where Xe = Q∗ XQ,
e ∗ e e −1
H = Q HQ. Then X = H ◦D1 , where D1 = ((di +dj ) ) and ◦ is the Hadamard (entrywise)
product. The matrix D1 is a Cauchy matrix with positive parameters and hence is positive
definite (as follows from its upper triangular LU factor having positive diagonal [276, ,
Sec. 28.1]). The Hadamard product of a Hermitian positive definite matrix with a Hermitian
positive (semi)definite matrix is Hermitian positive (semi)definite [296, , Thm. 5.2.1],
e and hence X are Hermitian positive (semi)definite.
so X
From (6.11) follow the three relations

Xk (Xk+1 − Xk ) + (Xk+1 − Xk )Xk = A − Xk2 , (E.11)

Xk Xk+1 + Xk+1 Xk = A + Xk2 , (E.12)
2
Xk+1 − A = (Xk+1 − Xk )2 . (E.13)

From X0 > 0, (E.12), and the hint, it follows that X1 > 0, and (E.13) gives X12 ≥ A. Assume
Xk2 ≥ A and Xk > 0. Then
(i) Xk+1 ≤ Xk by (E.11) and the hint,
(ii) Xk+1 > 0 by (E.12) and the hint,
2
(iii) Xk+1 ≥ A by (E.13).
Hence all but one of the required inequalities follow by induction. The remaining inequality,
Xk ≥ A1/2 , follows from (iii) on invoking Theorem B.9.
The sequence Xk is nonincreasing in the positive semidefinite ordering and bounded below
by A1/2 , so it converges to a limit, X∗ > 0, by Lemma B.8 (c). From (E.13) it follows that
X∗2 = A. But A1/2 is the only positive definite square root, so X∗ = A1/2 .
(b) If X0 commutes with A then by Lemma 6.8 the full and simplified Newton iterations
generate exactly the same sequence, so monotonic convergence holds for (6.12). However,
for arbitrary X0 > 0 the simplified iteration (6.12) does not, in general, converge.
(c) Since X 2 = A if and only if (Z −1 XZ)2 = Z −1 AZ, part (a) can be applied to
Ae = Z −1 AZ and X e0 = Z −1 X0 Z, for which Xek = Z −1 Xk Z. Monotonic convergence holds
e
for the Xk > 0.

6.10. If the iteration is to converge then it must converge on the spectrum of A, that is, the
iterations „ «
1 λ
xk+1 = xk + , x0 = λ
2 xk
must converge for each eigenvalue λ of A. If λ ∈ R− then xk ∈ R for all k and so xk cannot
converge to either of the square roots of λ, both of which are pure imaginary. Hence the
Newton iteration does not converge. In view of the relation with the Newton sign iteration
given in Theorem 6.9 (which is valid for scalar a even when a ∈ R− ), the behaviour of the
iteration is described by Problem 5.11.

6.11. The symmetrization step is legal because Xk A = AXk ⇒ Xk A1/2 = A1/2 Xk . The
variables Xk and Yk are easily seen to satisfy (6.15).

6.12. The modified iteration is only linearly convergent, as is easily verified numerically. The
reasoning used to derive the modified iteration is dubious for an iteration that is already
quadratically convergent.

6.13. The iterates satisfy Yk = A−1 Xk , so to analyze the errors in the (exact) iteration we
must set F = A−1 E, which gives
» – ‚» –‚2 !
1 E − A−1/2 EA1/2 ‚ E ‚
G(A 1/2
+ E, A −1/2
+ F) = +O ‚ ‚
‚ F ‚ .
2 A−1 E − A−1/2 EA−1/2
Solutions to Problems 363

Now all iterates Xk are functions of A, as is A1/2 , so E, which represents Xk − A1/2 , is a

function of A and so commutes with A and A1/2 . Thus the first order term is zero and
‚» –‚2 !
‚ E ‚
G(A 1/2
+ E, A −1/2
+ F) = O ‚ ‚
‚ F ‚ ,

which implies the quadratic convergence of the DB iteration (near the solution—this analysis
does not prove global convergence).

6.14. The uncoupled recurrence is numerically unstable [274, ], so this “simplification”
is not recommended. The iteration (6.63) has exactly the same stability properties as the
Newton iteration (6.12).

6.15. C ≥ 0 implies that ρ(C) is an eigenvalue of C (see Theorem B.7), and since the cardioid
extends only to 1 on the positive real axis, if Λ(C) lies in the cardioid then ρ(C) < 1, so the
spectrum of C must lie inside the unit circle. So the requirement on ρ(C) cannot be relaxed.

6.16. We need show that Xk defined by (6.46) with X0 = D1/2 is related to Bk from (6.48)
by Xk = D1/2 + Bk . This is clearly true for k = 0. Suppose it is true for k. Then the
modified Newton iteration (6.46) is

D1/2 Ek + Ek D1/2 = A − Xk2 = A − D − Bk2 − D1/2 Bk − Bk D1/2 ,

that is,
D1/2 (Ek + Bk ) + (Ek + Bk )D1/2 = A − D − Bk2 .
Comparing with (6.48), we see that Bk+1 = Ek + Bk = Xk+1 − Xk + Bk = Xk+1 − D1/2 , as
required.

6.17. The eigenvalues µi of C are µi = 1 − λi /s. At a minimum we must have 1 − λ1 /s =

mini µi = − maxi µi = −(1 − λn /s), which yields the claimed values of s and ρ(C).

6.18. We have C = I −A/s ≡ I −µA, where µ = 1/s. Now kI −µAkF = k vec(I)−vec(A)µk2 ,

so the problem is essentially a 1-variable linear least squares problem. The normal equations
are vec(A)∗ vec(A)µ = vec(A)∗ vec(I), or trace(A∗ A)µ = trace(A∗ ), as required.

6.19. For A with a semisimple zero eigenvalue linear convergence to A1/2 holds if the nonzero
eigenvalues of A satisfy the conditions of Theorem 6.16, with A1/2 now denoting the matrix
in Problem 1.27.
By using the Jordan form and Theorem 4.15 the convergence problem as a whole is
reduced to showing the linear convergence to λ1/2 of xk+1 = xk + α(λ − x2k ), x0 = (2α)−1 ,
for every eigenvalue λ of A. Only the convergence for a semisimple λ = 0 is in question. We
just have to show that the iteration xk+1 = xk − αx2k , x0 = (2α)−1 converges to 0 when 0 <
α ≤ ρ(A)−1/2 ; in fact, irrespective of the value of α > 0 we have 0 < xk+1 < xk < · · · < x0
and so xk converges to the unique fixed point 0.

6.20. We are given that A = sI − B with B ≥ 0 and s > ρ(B). Since diag(B) = diag(s −
aii ) ≥ 0, s ≥ maxi aii is necessary. Let α = maxi aii . Then A = αI − C, where C =
B + (α − s)I. Now C ≥ 0, since cij = bij ≥ 0 for i 6= j and cii = α − aii ≥ 0. Since B ≥ 0
and C ≥ 0, ρ(B) is an eigenvalue of B and ρ(C) an eigenvalue of C, by Theorem B.7. Hence
ρ(C) = ρ(B) + α − s < α. Hence we can take s = α in (6.52).

6.21. One way to derive the algorithm isˆvia the ˜ matrix sign function. We noted in Sec-
tion 2.9 that X is the (1,2) block of sign( A0 B0 ). Given Cholesky factorizations A = R∗ R
and B = S ∗ S we have
» – » – » −1 –» –» –
0 B 0 S∗S R 0 0 RS ∗ R 0
= = .
A 0 R∗ R 0 0 S −1 SR∗ 0 0 S
364 Solutions to Problems

Hence „» –« » –» –» –
0 B R−1 0 0 U R 0
sign = ,
A 0 0 S −1 U −1 0 0 S
where, in view of (8.6), RS ∗ = U H is a polar decomposition. From the (1,2) block of this
equation we obtain X = R−1 U S. That X is Hermitian positive definite can be seen from
X = R−1 U · S = S ∗ H −1 · S. In fact, we have obtained a variant of Algorithm 6.22. For any
nonsingular A, the unitary polar factor of A is the conjugate transpose of the unitary polar
factor of A∗ . Hence it is equivalent to find the unitary polar factor V = U ∗ of SR∗ and then
set X = R−1 V ∗ S ≡ R−1 U S.

6.22. The principal square root is

2 3
2(1 + ǫ)
1 − (2 + ǫ)−1
6 ǫ2 (2 + ǫ) (2 + 3 ǫ) 7
6 7,
4 0 1+ǫ (2 + 3 ǫ)−1 5
0 0 1 + 2ǫ

which has norm O(1/ǫ2 ). However, the non-principal square root

2 3
1 ǫ−1 0
4 0 −(1 + ǫ) ǫ −1 5
0 0 1 + 2ǫ

has norm only O(1/ǫ).

7.1. We use Theorem 7.3. As noted in Section 1.5, the ascent sequence is d1 = 1, d2 = 1,
. . . , dm = 1, 0 = dm+1 = dm+2 = · · ·. Hence for ν = 0 and m > 1 there is more than one
element of the sequence in the interval (pν, p(ν + 1)) = (0, p). Thus for m > 1 there is no
pth root.

7.2. Assuming A has no eigenvalues on R− , we can define Aα = exp(α log A), where the log
is the principal logarithm. For α = 1/p, with p an integer, this definition yields A1/p , as
defined in Theorem 7.2. To see this, note that, by commutativity, (Aα )p = exp(α log A)p =
exp(pα log A) = exp(log A) = A, so that Aα is some pth root of A. To determine which root
1 log λ
it is we need to find its spectrum. The eigenvalues of Aα are of the form e p , where λ is
1 log λ
an eigenvalue of A. Now log λ = x + iy with y ∈ (−π, π), and so e p = ex/p eiy/p lies in
the segment { z : −π/p < arg(z) < π/p }. The spectrum of Aα is therefore precisely that of
A1/p , so these two matrices are one and the same.
For α ∈ (0, 1) we can also define Aα by (7.1) with p = 1/α.

7.3. Let X have Jordan canonical form X = ZJZ −1 . Then In = X p = ZJ p Z −1 , that is,
J p = In . This implies that the eigenvalues λ (the diagonal elements of J) satisfy λp = 1,
i.e., they are pth roots of unity. But then if the Jordan form is nontrivial we see from (1.4)
that J p has nonzero elements in the upper triangle. This contradicts J p = In , so J must be
diagonal.
p −1 1/p
7.4. From Theorem 3.5Pwith f (x) = x and f (x) = x we have Lxp (X, Lx1/p (X p , E)) =
E. Now Lxp (X, E) = pj=1 X j−1 EX p−j (cf. (3.24)) and so with X p = A, L = Lx1/p (A, E)
P
satisfies pj=1 X j−1 LX p−j = E, or
p
X
(X p−j )T ⊗ X j−1 · vec(L) = vec(E). (E.14)
j=1

The formula for cond(X) follows on solving for vec(L)

Ppand taking 2-norms. The eigenvalues
p−j j−1
of the coefficient matrix in (E.14) are, by (B.20), j=1 µr µs = (µpr − µps )/(µr − µs )
Solutions to Problems 365

for µr 6= µs , where the µi are the eigenvalues of X. It follows that the coefficient matrix is
nonsingular, and the condition number finite, precisely when every copy of an eigenvalue of
A is mapped to the same pth root, which is equivalent to saying X that is a primary pth
root, by Theorem 7.1.

7.5. The proof is analogous to that of Lemma 6.8.

1/p
7.7. Since A1/p = kAk2 C 1/p where C = A/kAk2 ≤ I, we can assume A ≤ I without loss
of generality. Given 0 ≤ Z ≤ Y ≤ I and Y Z = ZY we have, using (B.15),
Y p −Z p = (Y −Z)(Y p−1 +Y p−2 Z +· · ·+Z p−1 ) ≤ (Y −Z)(I +I +· · ·+I) = p(Y −Z). (E.15)
Suppose I ≥ Xk ≥ Xk−1 . Then
1 1
Xk+1 − Xk = Xk + (A − Xkp ) − Xk−1 − (A − Xk−1
p
)
p p
1
= Xk − Xk−1 − (Xkp − Xk−1 p
)
p
≥0
by (E.15). Moreover, X0 ≤ I and if Xk ≤ I then, using (E.15) again,
1 1
Xk+1 = Xk + (A − Xkp ) ≤ Xk + (I − Xkp ) ≤ Xk + I − Xk = I.
p p
Since 0 = X0 < X1 = A/p ≤ I, it follows that 0 ≤ Xk ≤ Xk+1 ≤ I for all k, by induction.
The sequence Xk , being monotonically nondecreasing and bounded above, has a limit, X > 0
(see Lemma B.8 (c)). But X must satisfy X = X + p1 (A − X p ), and so X p = A, as required.

7.8. For Rk = I − Xk A we have Rk+1 = I − (2I − Xk A)Xk A = I − 2Xk A + (Xk A)2 =

(I − Xk A)2 = Rk2 , and likewise for Rk = I − AXk .

7.9. Since X0 does not commute with A we cannot invoke Theorem 7.10 or Theorem 7.12.
Applying the iteration we find that
» –
1 ξθ 2µ2 − µ − 1
X1 = , ξ= ,
0 µ 2µ2
and hence » –
1 ξk θ
Xk = .
0 µ
Thus we have linear convergence to A−1/2 if |ξ| < 1 (except when ξ = 0, which gives
convergence in one step) and divergence if |ξ| ≥ 1. For real µ, these two situations correspond
to µ > −1 and µ ≤ −1, respectively. This example shows that (quadratic) convergence can
be lost when X0 does not commute with A.

7.10. We find from the proof of Theorem 7.10 that a2 = (1/2)(1 + 1/p), which decreases
from 3/4 for p = 2 to 1/2 as p → ∞. So once the error is sufficiently small we can expect
slightly faster convergence for larger values of p.

7.11. The formula A1/p = AX p−1 can be used.

7.13. Since X0 , and hence all the iterates and B, commute with A, we can use the given
factorization with x ← Xk and b ← B to obtain
Xk1−p
Xk+1 − B = ((p − 1)Xkp − p Xkp−1 B + A)
p
p−2
Xk1−p X
= (Xk − B)2 (i + 1)Xki B p−2−i .
p i=0
366 Solutions to Problems

We conclude that
p−2
kXk1−p k X
kXk+1 − Bk ≤ kXk − Bk2 (i + 1)kXk ki kBkp−2−i .
p i=0

Hence we can take c = (p − 1)kB 1−p k kBkp−2 ≤ (p − 1)κ(B)p−1 /kBk.

7.14. The relation (7.11) holds for this iteration, too, if in (7.11) X/p is replaced by −X/p.

7.15. Gershgorin’s theorem (Theorem B.1) shows that every eigenvalue lies in one of the
disks |z − aii | ≤ 1 − aii , and by diagonal dominance we have aii > 0.5, so the spectrum lies
in E(1, p) in (7.17). Hence the iteration with X0 = I converges to A−1/p by Theorem 7.12.

7.16. We have
p−1
!
X ` ´T
e p) = −
vec(A − X X p−1−i ⊗ Xi vec(E) + O(kEk2 ).
i=0

Hence, taking the 2-norm,

‚X ‚
‚ p−1` p−1−i ´T ‚
e p kF ≤ kEkF ‚
kA − X X ⊗ X i‚ 2
‚ ‚ + O(kEkF )
i=0 2

e is a correctly rounded pth root

is a sharp bound, to first order in E. In particular, if X
e
then we would expect ρA (X) to be of order u. The alternative quantity kA − X e p k/kX
e pk
e p Pp−1 i p−1−i 2
is smaller and results from writing X = A + i=0 X EX + O(kEk ) and bounding
kX i EX p−1−i k ≤ ǫkXkp , which is not a sharp bound.

8.1. Direct computation shows that H = (A∗A)1/2 = diag(0, 1, . . . , 1). Then A = U H im-
plies that U (:, 2: m) = I(:, 1: m − 1). The first column of U is determined by the requirement
U ∗ U = I and must be of the form θem , where |θ| = 1. If we require a real decomposition
then θ = ±1.

8.3. In both cases H = 0. For the polar decomposition U can be any matrix with orthonor-
mal columns. For the canonical polar decomposition, U = AH + = 0.

8.4. Note that V = Q1/2 is unitary with spectrum in the open right half-plane, so V + V ∗ =
V + V −1 is Hermitian and positive definite. Thus the polar decomposition is I + Q = Q1/2 ·
(Q1/2 +Q−1/2 ) ≡ U H. Hence Q1/2 can be computed by applying (8.17) with A = I +Q. This
result is a special case of a more general result applying to matrices Q in the automorphism
group of a scalar product [283, , Thm. 4.7].

8.5. Let Q1 + Q2 = U H. It is straightforward to verify that H = (Q∗1 Q2 )1/2 + (Q∗1 Q2 )−1/2

satisfies (Q1 + Q2 )∗ (Q1 + Q2 ) = 2I + Q∗1 Q2 + Q∗2 Q1 = H 2 . The given eigenvalue condi-
tion ensures that H is positive definite. Hence U = (Q1 + Q2 )H −1 = (Q1 + Q2 ) · (Q1 +
Q2 )−1 Q1 (Q∗2 Q2 )1/2 . This of course generalizes Problem 8.4.

8.6. The result is immediate from kHxk22 = x∗ H 2 x = x∗ A∗Ax = kAxk22 .

8.7. Let A have theˆ SVD˜ A = P [Σ 0]Q∗ , where Σ ∈ Rm×m is possibly singular. Then
A = P [Im 0]Q∗ · Q Σ0 G0 Q∗ ≡ U H is a polar decomposition for any Hermitian positive
semidefinite G ∈ C(n−m)×(n−m) . The decomposition always exists, but H is never unique.
The nonuniqueness is clear in the extreme case m = 1, with A = a∗ , U = u∗ . Then
e −1
a∗ H
a∗ = u∗ H =: e −1 k2 H
· ka∗ H e
e
ka H −1 k2
∗

e (e.g., H
is a polar decomposition for any Hermitian positive definite H e = I). Another polar
decomposition is (8.4): a∗ = kak−1
2 a ∗
· kak −1
2 aa ∗
.
Solutions to Problems 367

8.8. We are given that A = U H with U ∗ = U + and H = H ∗ . The given condition

range(U ∗ ) = range(H) is equivalent to U + U = HH + , by Theorem 8.3. We know from
the theorem that A+ = H + U + . Thus AA+ = U HH + U + = U U + U U + = U U + , so that
range(A) = range(U ).
Conversely, we are given range(U ) = range(A), i.e., U U + = AA+ , and need to show that
range(U ∗ ) = range(H), i.e., (since U ∗ = U + ) U + U = HH + . Now, recalling that H and H +
commute, since U = AH + we have

U + U = U ∗ U = H + A∗AH + = H + H 2 H + = HH + HH + = HH + ,

as required.

8.9. Using the spectral decomposition of A, we have A = QΛQ∗ = Q Sign(Λ)Q∗ · Q|Λ|Q∗ ≡

U H. Thus U = f (A) with f (z) = Sign(z) = z/|z| (not the same sign function as in
Chapter 5) and H = g(A) with g(z) = |z|.

8.10. We use the same notation as in the proof given for the Frobenius norm. Note that
A∗ E + E ∗ A = HU ∗ E + E ∗ U H = H(U ∗ Q − In ) + (Q∗ U − In )H = HU ∗ Q + Q∗ U H − 2H. Let
t be an eigenvalue of H that maximizes (t − 1)2 and let w with kwk2 = 1 be a corresponding
eigenvector. Then, from (8.9),
kA − U k22 = (t − 1)2 . (E.16)
Since kBk22 = maxkzk2 =1 z ∗ B ∗ Bz, (8.11) implies

kA − Qk22 ≥ w∗ (H − In )2 w − w∗ (HU ∗ Q + Q∗ U H − 2H)w

= (t − 1)2 − (tw∗ U ∗ Qw + tw∗ Q∗ U w − 2t)
= (t − 1)2 − 2t(Re w∗ U ∗ Qw − 1)
≥ (t − 1)2 (E.17)

on using Re w∗ U ∗ Qw ≤ |w∗ U ∗ Qw| ≤ kU wk2 kQwk2 = 1. The result follows from (E.16) and
(E.17).

8.11. The result follows from (8.14) and

kB ∗A − W k2F = trace(A∗ BB ∗ A + I) − trace(A∗ BW + W ∗ B ∗A).

Note that while the minimizers are the same, the minimal values of the two objective func-
tions are different.
ˆ Σr 0
˜
8.12. Let Q ∈ Cm×n be any partial isometry and let A have the SVD A = P 0 0m−r,n−r
Q∗ ,
where r = rank(A). Then, by Theorem B.3,

kA − Qk ≥ k diag(σ1 − µ1 , . . . , σr − µr , −µr+1 , . . . , −µmin(m,n) k,

where the µi are the singular values of Q. Since µi ∈ {0, 1} for all i (see Lemma B.2) we
have
kA − Qk ≥ k diag(f (σ1 ), . . . , f (σr ), 0, . . . , 0)k,
where f (x) = min{x, |1−x|}. The lower bound is attained for Q = P diag(µi )V ∗ with µi = 1
if σi ≥ 1/2, µi = 0 if σi ≤ 1/2, i = 1: r, and µi = 0 for i > r.

8.13. Substituting the SVD A = P ΣQ∗ we have to maximize

f (W ) = Re trace(W ∗ A) = Re trace(W ∗ P ΣQ∗ ) = Re trace(Q∗ W ∗ P Σ)

=: Re trace(V Σ),

where V = Q∗ W ∗ P is unitary with det(V ) = det(P Q∗ ) det(W ∗ ). Let r = rank(A).

368 Solutions to Problems

Pr
(a) Since V is unitary, |vii | ≤ 1, and so f (W ) ≤ i=1 σr . Equality is attained iff
V = diag(Ir , Z), for some unitary Z, and since W = P V ∗ Q∗ , (8.1) shows that this condition
holds iff W is a unitary polar factor of A. If A is nonsingular then the unitary polar factor
is unique and so W is the unique maximizer.
(b) If det(P Q∗ ) = 1 then (a) shows that all solutions are given by unitary polar factors
(8.1) of A with det(W ) = det(Z) = 1. If σn−1 6= 0 then Z either is empty (if r = n) or else
has just one degree of freedom (vnn = ±1), which is used up by the condition det(W ) = 1,
so the solution is unique.
If det(P Q∗ ) = −1 then it is easy to see that det(W ) = 1 implies f (W ) = Re trace(V Σ) ≤
σ1 + · · · + σn−1 − σn , with equality when V = diag(1, . . . , 1, −1), and this V is clearly unique
if σn−1 > σn . (For a detailed proof, see Hanson and Norris [246, ].) It remains to check
that the expression for the maximizer is independent of the choice of SVD (i.e., of P and
Q); this is easily seen using Problem B.11.

8.14. Define Z = M 1/2 Q. Then the problem is

min{ kM 1/2 A − ZkF : Z ∗ Z = In },

which is just the problem in Theorem 8.4. So the solution is Q = M −1/2 U , where U is the
unitary polar factor of M 1/2 A. The solution can also be expressed as Q = R−1 U , where U
is the unitary polar factor of RA and and M = R∗R is the Cholesky factorization.

8.15. By using the spectral decomposition A = Q diag(λi )Q∗ we can reduce (i) to the case
where A = diag(λi ). Then
X X
kf (A)X − Xf (A)k2F = |(f (λi ) − f (λj ))xij |2 ≤ c2 |λi − λj |2 |xij |2 = c2 kAX − XAk2F .
i,j i,j

Parts (ii) and (iv) are straightforward. Part (iii) is obtained by taking X = I and f (z) = |z|
in part (ii).

8.16. We have
r1 − r2 ei(θ2 −θ1 ) = e−iθ1 (z1 − z2 ). (E.18)
Swapping the roles of z1 and z2 and taking the complex conjugate gives

r2 − r1 ei(θ2 −θ1 ) = eiθ2 (z2 − z1 ). (E.19)

Adding these two equations yields

r1 (1 − ei(θ2 −θ1 ) ) + r2 (1 − ei(θ2 −θ1 ) ) = e−iθ1 (z1 − z2 ) + eiθ2 (z2 − z1 ).

Hence
e−iθ1 (z1 − z2 ) + eiθ2 (z2 − z1 )
1 − ei(θ2 −θ1 ) = ,
r1 + r2
which yields the result on multiplying through by eiθ1 and taking absolute values. This proof
is a specialization of Li’s proof of the matrix version of the bound (Theorem 8.10).

8.17. We have 2 3 2 3
1 0 1 √0
U = 40 15, e = 40
U ǫ/ √ǫ2 + δ 2 5 ,
0 0 0 δ/ ǫ2 + δ 2
e is of order δ/ǫ, as required.
so the (3,2) element of U − U
Solutions to Problems 369

8.18. Let Y be an approximate solution to X ∗ X = I and write Y + E = U , where U is the

unitary polar factor of A. Then I = U ∗ U = (Y + E)∗ (Y + E) = Y ∗ Y + Y ∗ E + E ∗ Y + E ∗ E.
Dropping the second order term in E gives Newton’s method, which defines E as the solution
to the Sylvester equation Y ∗ E + E ∗ Y = I − Y ∗ Y . This equation is of the form Z + Z ∗ =
I − Y ∗ Y and so determines the Hermitian part of Z = Y ∗ E as 21 (I − Y ∗ Y ). Setting the
skew-Hermitian part to zero gives E = 12 (Y −∗ − Y ). Then Y + E = 21 (Y + Y −∗ ), which
leads to the iteration (8.17) if we take Y0 = A and recur this argument.

8.19. We have
1 A−∗ ∗
X1 = (A + A−∗ ) = (A A + I) = Y1−∗ .
2 2
If Yk = Xk−∗ then

1 1 1
Xk+1 = (Xk + Xk−∗ ) = (Yk−∗ + Yk ) = Yk−∗ (I + Yk∗ Yk ) = Yk+1
−∗
.
2 2 2

The result follows by induction.

8.20. We will show that for the scalar Newton–Schulz iteration xk+1 = g(xk ) = 12 xk (3 − x2k )
√
with x0 ∈ (0, 3), xk → 1 quadratically as√k → ∞. This yields the quadratic convergence
of the matrix iteration for 0 < σi (A) < 3 by using the SVD of A to diagonalize the
iteration. Now g agrees with f in (7.16) for p = 2, and the argument in the proof of
Theorem 7.11 shows that xk → 1 as k → ∞, with quadratic convergence described by
xk+1 − 1 = − 21 (xk − 1)2 (xk + 2). The latter equation leads to (8.36). The residual recurrence
can be shown directly or deduced from Theorem 7.10 with p = 2.

8.21. The error bound (8.18) shows a multiplier in the quadratic convergence condition
of kXk−1 k2 /2, which converges to 1/2 as Xk → U . For the Newton–Schulz iteration the
multiplier is kXk + 2U k2 /2, which converges to 3/2 in the limit. We conclude that the
Newton iteration has an asymptotic error constant three times smaller than that for the
Newton–Schulz iteration, and so can be expected to converge a little more quickly in general.

(1) (1)
8.22. In view of (5.38), X1 has singular values 1 ≤ σn ≤ · · · ≤ σ1 . For k ≥ 1, Xk therefore
(k) (k) (1)
has singular values 1 ≤ σn ≤ · · · ≤ σ1 = f (k−1) (σ1 ). The result follows.

8.23. (a) From (8.24), and using the trace characterization (B.5) of the Frobenius norm, we
have
1` 2 −1 2 ´
kXk+1 k2F = µk kXk k2F + 2 Re trace(Xk Xk−∗ ) + µ−2
k kXk kF .
4
Differentiating with respect to µk shows that
` the minimum is obtained
´ at µF
k.
1 −1 −∗
(b) Write X ≡ Xk . Using kXk+1 k ≤ 2 µk kXk + µk kXk k for the 1- and ∞-norms,
and kAk1 = kA∗ k∞ , we obtain

1` 2 ´
kXk+1 k1 kXk+1 k∞ ≤ µk kXk1 kXk∞ +kXk1 kX −1 k1 +kXk∞ kX −1 k∞ +µ−2
k kX
−1
k1 kX −1 k∞ .
4

Differentiating reveals the minimum at µk = µ1,∞

k .

8.24. Using Table C.2 we see that for m > n, the SVD approach requires about min(14mn2 +
8n3 , 6mn2 + 20n3 ) + 2mn2 + n3 flops, while Algorithm 8.20 requires 6mn2 + (2k − 3 13 )n3 ≤
6mn2 + 17n3 flops. The SVD approach is clearly the more expensive. For m = n the
operation counts are 25n3 flops for the SVD versus at most 22n3 flops for Algorithm 8.20.
Note the comments in Appendix C concerning the relevance of flop counts.
370 Solutions to Problems

8.25. (a) For the first part, writing c = cos θ, s = sin θ, we find that
» – » –
aii aij caii − saji caij − sajj
A′ = GT = .
aji ajj saii + caji saij + cajj
The trace of A′ is f (θ) = c(aii + ajj ) + s(aij − aji ). Since f ′ (θ) = −s(aii + ajj ) + c(aij − aji )
we see that f ′ (θ) = 0 is precisely the condition for symmetry. It is easily checked that for a
suitable choice of the signs of c and s, trace(A′ ) > trace(A) and hence the trace is maximized
(rather than minimized).
(b) trace(GA) = trace(A) − trace(2vv T/(v T v) · A) = trace(A) − 2v TAv/(v T v). This
quantity is maximized when the Rayleigh quotient v TAv/(v T v) is minimized, which occurs
when v is an eigenvector corresponding to λmin (A), which is negative by assumption. Hence
maxv trace(GA) = trace(A) + 2|λmin (A)| > trace(A).
(c) Details can be found in Smith [529, , Chap. 3]. This idea was originally suggested
by Faddeev and Faddeeva [180, ]; Kublanovskaya [364, ] had earlier investigated
a symmetrization process based on just Givens rotations. The algorithm is only linearly
convergent (with slow linear convergence) and a proof of global convergence appears difficult.
Unfortunately, the idea does not live up to its promise.
8.26. (a) The blocks of Q satisfy
Q1 = (I + A∗ A)−1/2 M, Q2 = A(I + A∗ A)−1/2 M,
giving
Q2 Q∗1 = A(I + A∗ A)−1 .
Using this formula with A ≡ Xk gives (8.38).
(b) The flop counts per iteration are mn2 + 7n3/3 for (8.37) and 6mn2 + 8n3/3 for (8.38).
If advantage is taken of the leading identity block in (8.38) the flop count can be reduced,
but not enough to approach the operation count of (8.37).
(c) As discussed at the end of Section 8.6, we can scale (8.37) by setting Xk ← µk Xk ,
with one of the scalings (8.25)–(8.27). The problem is how to compute µk , since Xk−1 is
not available (and does not exist if m 6= n). Concentrating on (8.25), from (8.38a) we have
Rk∗ Rk = I + Xk∗ Xk , and since Rk is triangular we can apply the power and inverse power
methods to estimate the extremal singular values of Rk and hence those of Xk . Unfortu-
nately, we obtain µk only after Xk+1 has been (partly) computed, so we can only use µk to
scale the next iterate: Xk+1 ← µk Xk+1 .
9.1. (⇐) For any strictly triangular N let D + N = Z diag(J1 , . . . , Jq )Z −1 (q ≥ p) be a
Jordan canonical form of D + N with Jordan blocks
2λ 1 3
i
6 λi 1 7
6 .. .. 7
6 . . 7 m ×m
Ji = 6 7 ∈ C i i,
6 .. 7
4 . 5
1
λi
where, necessarily, mi does not exceed the kj corresponding to λi . Then
f (D + N ) = Z diag(f (J1 ), . . . , f (Jq ))Z −1 ,
where, from (1.4),
2 f (mi −1) (λi )
3
f (λi ) f ′ (λi ) ... ... (mi −1)!
6 7
6 .. 7
6 f (λi ) f ′ (λi ) ... . 7
6 7
f (Ji ) = 6 .. .. .. 7. (E.20)
6 . . . 7
6 7
6 .. 7
4 . f ′ (λi ) 5
f (λi )
Solutions to Problems 371

Since the derivatives of f are zero on any repeated eigenvalue and f (λi ) = f (λ1 ) for all i,
f (D + N ) = Zf (D)Z −1 = Zf (λ1 )IZ −1 = f (λ1 )I = f (D).
(⇒) Let F = f (D +N ), and note that by assumption F = f (D) and hence F is diagonal.
The equation F (D + N ) = (D + N )F reduces to F N = N F , and equating (i, j) elements
for j > i gives (fii − fjj )nij = 0. Since this equation holds for all strictly triangular N , it
follows that fii = fjj for all i and j and hence that F = f (λ1 )I.
If at least one of the λi is repeated, then we can find a permutation matrix P and a strictly
upper bidiagonal matrix B such that P DP T + B = P (D + P T BP )P T is nonderogatory
and is in Jordan canonical form, and N = P T BP is strictly upper triangular. We have
Λ(D) = Λ(D + N ) and so the requirement f (D + N ) = f (D) implies that f (P DP T + B) =
P f (D)P T = f (λ1 )I, and hence, in view of (E.20), (9.5) holds.

9.2. The proof is entirely analogous to that of Lemma 6.4. For

ˆ athe˜ last part, a 2 × 2 real
b
normal matrix with distinct eigenvalues a±ib has the form A = −b a
and if f (a+ib) = c+id
ˆ c d˜
then f (A) = −d c .

10.1. X(t) = e(A+E)t satisfies the differential equation X ′ (t) = AX(t) + EX(t), X(0) = I.
Hence from the matrix analogue of (2.3), we have
Z t
X(t) = eAt + eA(t−s) EX(s) ds,
0

which is (10.13) on substituting for X.

10.2. Let AB = BA and f (t) = e(A+B)t − eAt eBt . Then f (0) = 0 and, because etA is a
polynomial in tA (as is any function of tA) and hence commutes with B,

f ′ (t) = (A + B)et(A+B) − AetA etB − etA BeBt

= (A + B)et(A+B) − (A + B)etA etB
= (A + B)(et(A+B) − etA etB )
= (A + B)f (t).

The unique solution to this initial value problem is f (t) ≡ 0, as required.

10.3. From (10.13) with t = 1 and B ≡ A + E, we have

Z 1
eB = eA + eA(1−s) (B − A)eBs ds.
0

Hence
Z 1
keA − eB k ≤ kA − Bk ekAk(1−s) ekBks ds
0
Z 1
≤ kA − Bk emax(kAk,kBk) ds = kA − Bk emax(kAk,kBk) .
0

Qn Q
10.4. det(e A+B
) = i=1 λi (e A+B
) = n i=1 e
λi (A+B)
= etrace(A+B) = etrace(A)+trace(B) =
trace(A) trace(B) A B A B
e e = det(e ) det(e ) = det(e e ).

10.5. The bound is immediate from Lemma 10.15 and Theorem 10.10.

10.6. From (B.28) we have |f [λ, µ]| ≤ maxz∈Ω |f ′ (z)| = maxz∈Ω |ez | = max(Re λ, Re µ) ≤
α(A), where α is defined in (10.10) and Ω is the line joining λ and µ, and there is equality
for λ = µ. Hence maxλ,µ∈Λ(A) |f [λ, µ]| = α(A) = keA k2 . Thus the two different formulae
are in fact equivalent.
372 Solutions to Problems

10.7. By Corollary 3.10, since eλi 6= 0 and eλi = eλj if and only if λi − λj = 2mπi for some
integer m, L(A) is nonsingular precisely when no two eigenvalues of A differ by 2mπi for
some nonzero integer m.
P∞ j+1 j
10.8. Since log(I + G) = j=1 (−1) G /j (see (1.1)),

X∞ X∞
kGkj kGk
k log(I + G)k ≤ = − log(1 − kGk) ≤ kGk kGkj = .
j=1
j j=0
1 − kGk

Similarly,

kG2 k kG3 k
k log(I + G)k ≥ kGk − − − ···
2 3
kG2 k kGk2
≥ kGk − (1 + kGk + kGk2 + · · ·) = kGk − .
2 2(1 − kGk)

10.9. From (10.23),

Xm
(2m − j)! m! (−A)j
qm (A) = I + =: I + F.
j=1
(2m)! (m − j)! j!

Now
Xm
(2m − j)! m! kAkj
kF k ≤ = qm (−kAk) − 1,
j=1
(2m)! (m − j)! j!

and so kqm (A)−1 k = k(I + F )−1 k ≤ 1/(1 − kF k) ≤ 1/(2 − qm (−kAk)).

Since, for 1 ≤ j ≤ m,
(2m − j)! m! “ m ”j
≤ = 2−j ,
(2m)! (m − j)! 2m
we have
Xm
1
kF k ≤ (kAk/2)j = ekAk/2 − 1.
j=1
j!

Hence kqm (A)−1 k ≤ 1/(1 − kF k) ≤ 1/(2 − ekAk/2 ) provided kAk < log 4. The last part is
straightforward to check.

10.10. We can rewrite (10.25) as

Z 1
(−1)m+1 k+m+1
G = e−A rkm (A) − I = A qkm (A)−1 e(t−1)A (1 − t)k tm dt.
(k + m)! 0

Taking norms gives

Z 1
kAkk+m+1
kGk ≤ kqkm (A)−1 k e(1−t)kAk (1 − t)k tm dt
(k + m)! 0
Z 1
kAkk+m+1
= kqkm (A)−1 k etkAk tk (1 − t)m dt.
(k + m)! 0

Using (10.25) with A replaced by kAk, this bound can be rewritten as

˛ ˛
kGk ≤ qmk (kAk) kqkm (A)−1 k ˛ekAk − rmk (kAk)˛.

Specializing to k = m and using Problem 10.9 to rewrite the bound entirely in terms of kAk
gives (10.59).
Solutions to Problems 373

10.11. The proof is by induction. We have

kA2 − f l(A2 )k ≤ γn kAk2 = nukAk2 + O(u2 ),
bk = f l(A2k ) =:
so the result is true for k = 1. Assume the result is true for k − 1 and write X
k
A2 + Ek . We have
bk = f l(X
X bk−1
2 bk−1
)=X 2
+ Fk , bk−1 k2 .
kFk k ≤ γn kX
Hence
` k−1 ´2
bk = A2
X + Ek−1 + Fk
k k−1 k−1
= A2 + A2 Ek−1 + Ek−1 A2 2
+ Ek−1 + Fk .
Hence
k−1
kEk k ≤ 2kA2 k kEk−1 k + kEk−1 k2 + kFk k
k−1 k−2
≤ 2(2k−1 − 1)nukA2 k kAk2 kA2 k kA4 k . . . kA2 bk−1 k2 + O(u2 )
k + γn kX
k−1 k−1
= 2(2k−1 − 1)nukAk2 kA2 k kA4 k . . . kA2 k + γn kA2 k2 + O(u2 )
k−1
≤ (2k − 1)nukAk2 kA2 k kA4 k . . . kA2 k + O(u2 ).
10.12. The form of the diagonal blocks is immediate. From Theorem 4.12 and (10.15) we
see that the (1,2) block of eA is the (1,2) block of
Z 1 » A11 (1−s) –» – » A11 s –
e 0 0 A12 e 0
ds,
0 0 eA22 (1−s) 0 0 0 eA22 s
which gives the result.
10.13. The formula can be obtained from the polynomial interpolation definition of eA , but
the following approach is more elegant. Since A is skew-symmetric we know that Q∗ AQ =
diag(0, iθ, −iθ) =: iθK for some θ ∈ R. Then Q∗ eA Q = diag(1, eiθ , e−iθ ) = diag(1, c + is, c −
is) = M + cL + isK, where c = cos θ, s = sin θ, M = diag(1, 0, 0), and L = diag(0, 1, 1).
Hence eA = Q(M +cL+isK)Q∗ = M +cQLQ∗ +(s/θ)A. But A2 = Q(iθK)2 Q∗ = −θ2 QLQ∗ ,
so cQLQ∗ = −cA2 /θ2 and QM Q∗ = Q(I − L)Q∗ = I − QLQ∗ = I + A2 /θ2 . It remains to
note that kAk2F = 2θ2 .
Note that the Cayley–Hamilton theorem shows that eA ∈ R3×3 must be a cubic poly-
nomial in A; the simplification in (10.60) results from the fact that A3 = −θ2 A, a relation
specific to n = 3.
ˆ ˜
10.14. A = t R0 −R 0
, where R is the 4 × 4 “reverse identity matrix” with rij = 1 for i + j = 5
and rij = 0 otherwise.
11.1. We have
2 3 2 3
λ 1 1 λ−1
6 .. 7 6 .. 7
6 λ . 7 6 1 . 7
J(λ) = 6 .. 7 = λ6 .. 7 = λ(I + N ).
4 . 1 5 4 . λ−1 5
λ 1
Hence log(J(λ)) = (log λ)I + log(I + N ), since the conditions of Theorems 11.3 and 11.4
are
Pm−1trivially satisfied for B = λI, C = I + N . Since N is nilpotent, log(I + N ) =
i+1 i
i=1 (−1) N /i. Hence
2 1 (−1)m
3
log λ λ
. . . (m−1)λ m−1
6 .. 7
6 .. 7
6 log λ . . 7
log(J(λ)) = 6 7,
6 . .. 7
4 1 5
λ
log λ
374 Solutions to Problems

which of course agrees with (1.4). Via the Jordan canonical form, log(A) can then be defined
for A with no eigenvalues on R− .

11.2. The proof is straightforward, by using the integral and (B.10) to obtain the first order
part of the expansion of log(A + E) − log(A).

11.3. We have maxi |1−λi (A)| = ρ(I −A) ≤ kI −Ak < 1, which implies that every eigenvalue
of A lies in the open right half-plane, as required. The convergence also follows from

kI − Ak
k(I − A)(I + A)−1 k = kI − Ak k(2I + A − I)−1 k ≤ < 1.
2 − kI − Ak

11.4. Let X = M −I. From (10.58), k log(M )k ≤ − log(1−kXk). Thus if kI −M k ≤ 1−1/e

then k log(M )k ≤ | log(1 − (1 − 1/e)| = 1.

11.5. The property follows from consideration of Table 11.1 and, in particular, these num-
bers:
θ3 θ4 θ5 θ7 /2 θ6 2θ5 θ7 2θ6 θ8
0.0162 0.0539 0.114 0.132 0.187 0.228 0.264 0.374 0.340

11.6. The equality follows from Theorem 11.4. This preprocessing is rather pointless: we
need A ≈ I, but making kAk = 1 in general makes no useful progress towards this goal.

11.7. The individual eigenvalues of Xk follow the scalar iteration xk+1 = (xk + ax−1k )/2,
x0 = a, where a is an eigenvalue of A. With the transformation yk = a−1/2 xk this becomes
yk+1 = (yk + yk−1 )/2, y0 = a1/2 , and it follows that −π/2 < arg yk < π/2 for all k, or
equivalently that xk is in the open half plane of a1/2 for all k.
From the derivation of the product DB iteration we know that Mk = Xk A−1 Xk and
that Xk and Mk are functions of A. By commutativity and Theorem 1.40 we know that
for each eigenvalue λj of A there are corresponding eigenvalues µj of Mk and ξj of Xk ,
with µj = ξj2 /λj . By Theorem 1.20 it suffices to show that log µj = 2 log ξj − log λj for
1/2 1/2
all j. Let z = ξj /λj . Then arg z = arg ξj − arg λj + 2πk for some k ∈ Z, and since
1/2
| arg z| < π/2 (as we just saw in the previous paragraph) and | arg λj | < π/2, we have k = 0.
1/2 1/2
Hence | arg ξj − arg λj | < π, which implies, by Theorem 11.3, log z = log ξj − log λj =
log ξj − 12 log λj . Thus log µj = 2 log ξj − log λj , as required. Equation (11.31) follows by
induction.

11.8. (a) Since X = log(A) we have A = eX , so λ ∈ Λ(A) implies λ = eµ for some µ.

The eigenvalues of B = AT ⊕ A = AT ⊗ I + I ⊗ A are of the form λj + λk = eµj + eµk . If B is
singular then eµj + eµk = 0 for some j and k, that is, eµj −µk = −1, or µj − µk = (2ℓ − 1)πi
for some ℓ ∈ Z. But then kXk ≥ ρ(X) ≥ maxj,k (|µj |, |µk |) > 1, which contradicts kXk ≤ 1.
Hence B is nonsingular.
(b) The zeros of τ are x = ±ikπ and satisfy |x| ≥ π. Since ρ(X) ≤ kXk ≤ 1, no
eigenvalue of X is a zero of τ and so τ (X) is nonsingular.

11.9. For the first part see Edelman and Strang [173, ]. Since rank(R − I) = n − 1, R is
similar to a single Jordan block J(0). Hence by Theorem 1.28, in which p = 1, all logarithms
are primary matrix functions and have the form log(R) + 2jπiI for j ∈ Z, which is always
upper triangular and is real only for j = 0.

11.10. The relative backward error of Xb is the smallest value of kEk/kAk such that X
b =
b
X
log(A + E). There is only one E satisfying the latter equation, namely E = e − A, so the
b
relative backward error is keX − Ak/kAk.
Solutions to Problems 375

11.11. Consider first (11.32). Newton’s method for f (X) = 0 solves for E in the first order
perturbation expansion of f (X + E) = 0 and sets X ← X + E. For f (X) = eX − A we have,
if XE = EX, f (X + E) = eX eE − A = eX (I + E) − A to first order, or E = e−X A − I,
which gives X + E = X − I + e−X A. But XA = AX clearly implies XE = EX, so under
the assumption XA = AX, E is the Newton update (which we are implicitly assuming is
unique). Since X + E clearly commutes with A, this argument can be repeated and we have
derived (11.32). The commutativity relations involving Xk are easily proved by induction.
To analyze the asymptotic convergence, let A = eX with X a primary logarithm of
A, and write Xk+1 − X = Xk − X − I + e−Xk A = Xk − X − I + eX−Xk , since X and
Xk are functions of A and hence commute. Theorem 4.8 gives eX−Xk = I + (X − Xk ) +
1
2
max0≤t≤1 k(X − Xk )2 et(X−Xk ) k, which gives kX − Xk+1 k ≤ 12 kX − Xk k2 ekX−Xk k for any
subordinate matrix norm.
The analysis for (11.33) is very similar, if a little more tedious. The convergence bound
is kX − Xk+1 k ≤ 61 kX − Xk k3 ekX−Xk k .
The extra cost of (11.33) over (11.32) is one matrix inversion per iteration, which is
negligible compared with the cost of the matrix exponential. Therefore, all other things
being equal, the cubically convergent iteration (11.33) should be preferred.
P (−1)i `P∞ (−1)i ´
12.1. With B = kπA, B 2i = (kπ)2i I, so cos(B) = ∞ i=0 (2i)! B
2i
= i=0 (2i)! (kπ)
2i
I=
cos(kπ)I = (−1)k I.
ˆ ˜
12.2. From Theorem 1.36 the only possible Jordan form of A is J2 (λ) = 10 11 with sin λ = 1.
But then cos(λ) = 0 and so by Theorem 1.36 the Jordan blockˆ splits ˜ into two 1 × 1 blocks
in sin(J2 (λ)). Hence sin(A) cannot be equal to (or similar to) 10 11 .
12.3. All four identities follow from Theorems 1.16 and 1.20.
12.4. Let B = i−1 log(X0 ). Then eiB = X0 and so we have X1 = 21 (X0 + X0−1 ) = 12 (eiB +
e−iB ) = cos(B), as required.
p
12.5. sin(A) = I − cos2 (A) for some square root, but without knowing the eigenvalues of
A we cannot determine which is the required square root. But if A is triangular then the
eigenvalues of A are known and we can proceed. Consider, first, the scalar case and note
that
cos(x + iy) = cos(x) cos(iy) − sin(x) sin(iy) = cos(x) cosh(y) − i sin(x) sinh(y),
sin(x + iy) = sin(x) cos(iy) + cos(x) sin(iy) = sin(x) cosh(y) + i cos(x) sinh(y).
Since cosh(y) ≥ 0, from xpwe can determine the sign of Re sin(x+iy) and hence which square
root to take in sin(z) = 1 −p cos2 (z). Returning to the matrix case, we can determine each
diagonal element of sin(A) = I − cos2 (A) by the procedure just outlined. If the eigenvalues
of A are distinct then we can use the Parlett recurrence to solve for the off-diagonal part of
sin(A). However, if A has a repeated eigenvalue then the recurrence breaks down. Indeed,
the equation cos2 (A)+sin2 (A) = I may not contain enough information to determine sin(A).
For example, from
» – » – » –
0 x 1 0 0 x
A= , cos(A) = , sin(A) = ,
0 0 0 1 0 0
it is clear that sin(A) cannot be recovered from I − cos2 (A) = 0 without using knowledge
of A.
13.1. By (13.5), AQm = Qm Hm , where Qm ˆ∈ Cn×m ˜ , Hm ∈ C
m×m
. Let U be such that
[Qm U ] is unitary. Then [Qm U ]∗ A [Qm U ] = H0m 00 . Hence
„» –« » ∗ –
Hm 0 Qm
f (A)b = [Qm U ]f ∗ b
0 U AU U∗
» –
f (Hm ) 0
= [Qm U ] kbk2 e1 = kbk2 Qm f (Hm )e1 = fm .
0 f (U ∗ AU )
376 Solutions to Problems

13.2. Let d = deg ψA,b . If m < d then dim(K

Pm m (A,i b)) = m. Indeed dim(Km (A, b)) ≤ m
trivially, and if dim(Km (A, b)) < m then i=0 ci A b = 0 for some ci not all zero, which
contradicts the definition of ψA,b . If m ≥ d then dim(Km (A, b)) = d because Am b and
higher powers can be expressed as linear combinations of b, Ab, . . . , Ad−1 b using the equation
ψA,b (A)b = 0.

13.3. For any µ ∈ C, H − µI is upper Hessenberg with nonzero subdiagonal elements and so
rank(H − µI) ≥ n − 1. For µ an eigenvalue we therefore have rank(H − µI) = n − 1, which
implies that µ appears in only one Jordan block.

13.4. The correctness of the algorithm is easily proved by induction. For k = 1 the correct-
2
ness is immediate. For general k it is easy to see that x = 21 Xk−1 (3I − Xk−1 )b.
Since the function makes 3 recursive calls, with 3 matrix–vector multiplications at the
lowest level (k = 1), the cost is 3k matrix–vector multiplications. This is to be compared
with 2k matrix multiplications and 1 matrix–vector multiplication if we compute Xk and
then Xk b. The recursive computation requires fewer flops if 3k n2 < 3
∼ 2kn . If k = 8, this
>
requires n ∼ 410. The temporary storage required by the recursive algorithm is just k vectors
(one per level of recursion).
Note that this algorithm does not lend itself to dynamic determination of k, unlike for
the usual Newton–Schulz iteration: if we know Xk−1 b then computing Xk b costs 2/3 as much
as computing Xk b from scratch.

B.1. Since the columns of X form a basis they are independent, and so X + X = I. Let
(λ, x) be an eigenpair of A with x ∈ X . Then x = Xz for a unique z 6= 0, and z = X + x.
Hence
λx = Ax = AXz = XBz.
Multiplying on the left by X + gives λz = λX + x = Bz, so (λ, z) is an eigenpair for B.
Let (λ, z) be an eigenpair for B. Then AXz = XBz = λXz, and Xz 6= 0 since the
columns of X are independent, so (λ, Xz) is an eigenpair for A.

B.2. Let A be Hermitian and write X = A+ . Taking the conjugate transposes of (B.3) (i)
and (B.3) (ii) yields AX ∗ A = A and X ∗ AX ∗ = X ∗ . Using (B.3) (iii) we have (X ∗ A)∗ =
AX = (AX)∗ = X ∗ A, and (B.3) (iv) yields (AX ∗ )∗ = XA = (XA)∗ = AX ∗ . So X ∗ satisfies
the same four Moore–Penrose conditions as X, which means that X = X ∗ by the uniqueness
of the pseudoinverse. Finally, by (B.3) (iii) we have AX = (AX)∗ = XA, so A and X
commute.

B.3. If x ∈ range(A∗A) then, for some y, x = A∗Ay = A∗ (Ay) ∈ range(A∗ ), so range(A∗A) ⊆

range(A∗ ). We indicate three ways to obtain the reverse inclusion. (a) If x ∈ range(A∗ ) then,
for some z and with X = A+ , x = A∗ z = A∗ X ∗ A∗ z = A∗ (AX)∗ z = A∗ (AX)z = A∗A(Xz) ∈
range(A∗ A), so range(A∗ ) ⊆ range(A∗A). (b) A∗Az = 0 implies kAzk22 = z ∗ A∗Az = 0, so
that null(A∗A) ⊆ null(A), which implies that rank(A∗A) ≥ rank(A) = rank(A∗ ) and hence
range(A∗ ) ⊆ range(A∗A). (c) Since range(A) ∪ null(A∗ ) = Cm , any z ∈ Cm can be written
z = Ax + w, w ∈ null(A∗ ). Hence A∗ z = A∗Ax, which implies range(A∗ ) ⊆ range(A∗A).
ˆ Σr 0
˜
B.4. Let A have the SVD A = P 0 0m−r,n−r
Q∗ and partition P = [P1 P2 ] and Q =
[Q1 Q2 ], where P1 ∈ Cm×r and Q1 ∈ C n×r
satisfies Q∗1 Q1 = Ir . Then A∗ = Q1 Σr P1∗ , so
∗ ∗
range(A ) = range(Q1 ). So x ∈ range(A ) iff x = Q1 y for some y. For such an x we have
‚» –» – ‚
‚ Σr 0 I ‚
kAxk2 = kAQ1 yk2 = ‚ y ‚ = kΣr yk2 .
‚ 0 0m−r,n−r 0 ‚2

Since kxk2 = kyk2 , A is a partial isometry iff kΣr yk2 = kyk2 for all y, or equivalently,
Σr = I r .
Solutions to Problems 377

B.5. Let λ be an eigenvalue of A and x the corresponding eigenvector, and form the matrix
X = [x, x, . . . , x] ∈ Cn×n . Then AX = λX, so |λ|kXk = kAXk ≤ kAk kXk, showing that
|λ| ≤ kAk. For a subordinate norm it suffices to take norms in the equation Ax = λx.
ˆA˜
B.6. Let B = 0
. Since B ∗B = A∗A, B and A have the same singular values, so kBk = kAk.

B.7. For any unitarily invariant norm, kBk depends only on the singular values of B. Now
(U A)∗ (U A) = A∗ U ∗ U A = A∗A, so U A and A have the same singular values. Hence kAk =
kU Ak. Alternatively, choose V ∈ Cm×(m−n) so that [U, V ] is unitary and use Problem B.6
to deduce that ‚» –‚ ‚ » –‚
‚ A ‚ ‚ A ‚
‚
kAk = ‚ ‚ ‚
= [U V ] ‚ = kU Ak.
0 ‚ ‚ 0 ‚

B.8. If A = P DQ∗ is an SVD then

kABkF = kP DQ∗ BkF = kD(Q∗ B)kF

` ´
≤ k diag max σi (Q∗ B)kF = (max σi )kQ∗ BkF = kAk2 kBkF .
i i

B.9. Let P1 and P2 be orthogonal projectors onto S. Since range(P1 ) = range(P2 ), P2 =

P1 X for some X. Then P1 P2 = P12 X = P1 X = P2 . Likewise, P2 P1 = P1 . Hence, for any z,

k(P1 − P2 )zk22 = z ∗ (P1 − P2 )(P1 − P2 )z

= z ∗ (P12 + P22 − P1 P2 − P2 P1 )z
= z ∗ (P1 + P2 − P2 − P1 )z = 0.

Therefore P1 − P2 = 0.

B.10. AA+ is Hermitian by (B.3) (iii), and (AA+ )2 = AA+ AA+ = AA+ by (B.3) (i). It
remains to show that range(AA+ ) = range(A).
Let x ∈ range(A), so that x = Ay for some y. Then, by (B.3) (i), x = AA+ Ay = AA+ x,
so x ∈ range(AA+ ). Conversely, x ∈ range(AA+ ) implies x = AA+ y for some y and then
x = A(A+ y) ∈ range(A). Thus AA+ is the orthogonal projector onto range(A).
By the first part, the orthogonal projector onto range(A∗ ) is A∗ (A∗ )+ = A∗ (A+ )∗ =
(A A)∗ = A+ A.
+

B.11. We have AA∗ = P Σ 2 P ∗ = PeΣ 2 Pe∗ , so (Pe∗ P )Σ 2 = Σ 2 (Pe∗ P ). By equating elements

on both sides (or invoking Theorem 1.25) we find that Pe∗ P = D1 , where D1 is unitary and
block diagonal with (D1 )ij = 0 if σi 6= σj . Similarly, A∗A = QΣ 2 Q∗ = QΣ e 2Q e ∗ implies
Qe Q = D2 , where D2 is unitary and block diagonal with (D2 )ij = 0 if σi 6= σj . Then
∗

P ΣQ∗ = A = PeΣ Q e ∗ = P D1∗ ΣD2 Q∗ , which implies Σ = D1∗ ΣD2 = ΣD1∗ D2 , since D1
commutes with Σ and hence with (Σ 2 )1/2 . Write Σ = diag(Σ1 , 0n−r ), where Σ1 ∈ Rr×r
2

is nonsingular. Then it is easy to see that D1∗ D2 = diag(Ir , W ), where W is an arbitrary

unitary matrix. The result follows.
Bibliography

You will find it a very good practice always to verify your references, sir.
— MARTIN JOSEPH ROUTH (1878)

The distribution of the year of publication of the references in this bibliography

is shown in the following graph.

30

25
Number of references

20

15

10

0
1850 1870 1890 1910 1930 1950 1970 1990 2008

379
380 Bibliography

[1] N. I. Achieser. Theory of Approximation. Frederick Ungar Publishing Co., New York,
1956. x+307 pp. (Cited on p. 130.)
[2] Eva Achilles and Richard Sinkhorn. Doubly stochastic matrices whose squares are
idempotent. Linear and Multilinear Algebra, 38:343–349, 1995. (Cited on p. 191.)
[3] Martin Afanasjew, Michael Eiermann, Oliver G. Ernst, and Stefan Güttel. Implemen-
tation of a restarted Krylov subspace method for the evaluation of matrix functions.
Manuscript, July 2007. 27 pp. (Cited on p. 305.)
[4] Martin Afanasjew, Michael Eiermann, Oliver G. Ernst, and Stefan Güttel. On the
steepest descent method for matrix functions. Manuscript, 2007. 19 pp. (Cited on
p. 305.)
[5] S. N. Afriat. Analytic functions of finite dimensional linear transformations. Proc.
Cambridge Philos. Soc., 55(1):51–61, 1959. (Cited on p. 27.)
[6] Donald J. Albers and G. L. Alexanderson, editors. Mathematical People: Profiles and
Interviews. Birkhäuser, Boston, MA, USA, 1985. xvi+372 pp. ISBN 0-8176-3191-7.
(Cited on p. 171.)
[7] J. Albrecht. Bemerkungen zu Iterationsverfahren zur Berechnung von A1/2 und A−1 .
Z. Angew. Math. Mech., 57:T262–T263, 1977. (Cited on p. 167.)
[8] G. Alefeld and N. Schneider. On square roots of M -matrices. Linear Algebra Appl.,
42:119–132, 1982. (Cited on p. 167.)
[9] Marc Alexa. Linear combination of transformations. ACM Trans. Graphics, 21(3):
380–387, 2002. (Cited on pp. 50, 53.)
[10] E. J. Allen, J. Baglama, and S. K. Boyd. Numerical approximation of the product of
the square root of a matrix with a vector. Linear Algebra Appl., 310:167–181, 2000.
(Cited on p. 310.)
[11] Simon L. Altmann. Rotations, Quaternions, and Double Groups. Oxford University
Press, 1986. xiv+317 pp. ISBN 0-19-855372-2. (Cited on p. 300.)
[12] E. Anderson, Z. Bai, C. H. Bischof, S. Blackford, J. W. Demmel, J. J. Dongarra, J. J.
Du Croz, A. Greenbaum, S. J. Hammarling, A. McKenney, and D. C. Sorensen. LA-
PACK Users’ Guide. Third edition, Society for Industrial and Applied Mathematics,
Philadelphia, PA, USA, 1999. xxvi+407 pp. ISBN 0-89871-447-8. (Cited on pp. 100,
229.)
[13] T. Ando. Concavity of certain maps on positive definite matrices and applications to
Hadamard products. Linear Algebra Appl., 26:203–241, 1979. (Cited on p. 52.)
[14] T. Ando. Operator-Theoretic Methods for Matrix Inequalities. Hokusei Gakuen Uni-
versity, March 1998. 77 pp. (Cited on p. 52.)
[15] T. Ando, Chi-Kwong Li, and Roy Mathias. Geometric means. Linear Algebra Appl.,
385:305–334, 2004. (Cited on p. 52.)
[16] Huzihiro Araki and Shigeru Yamagami. An inequality for Hilbert–Schmidt norm.
Commun. Math. Phys., 81:89–96, 1981. (Cited on p. 215.)
[17] M. Arioli, B. Codenotti, and C. Fassino. The Padé method for computing the matrix
exponential. Linear Algebra Appl., 240:111–130, 1996. (Cited on pp. 104, 248.)
[18] W. E. Arnoldi. The principle of minimized iterations in the solution of the matrix
eigenvalue problem. Quart. Appl. Math., 9:17–29, 1951. (Cited on p. 309.)
[19] Vincent Arsigny, Oliver Commowick, Xavier Pennec, and Nicholas Ayache. A log–
Euclidean framework for statistics on diffeomorphisms. In Medical Image Computing
and Computer-Assisted Intervention—MICCAI 2006, Rasmus Larsen, Mads Nielsen,
and Jon Sporring, editors, number 4190 in Lecture Notes in Computer Science, Spring-
er-Verlag, Berlin, 2006, pages 924–931. (Cited on p. 284.)
Bibliography 381

[20] Vincent Arsigny, Pierre Fillard, Xavier Pennec, and Nicholas Ayache. Geometric means
in a novel vector space structure on symmetric positive-definite matrices. SIAM J.
Matrix Anal. Appl., 29(1):328–347, 2007. (Cited on p. 47.)
[21] Ashkan Ashrafi and Peter M. Gibson. An involutory Pascal matrix. Linear Algebra
Appl., 387:277–286, 2004. (Cited on p. 165.)
[22] Kendall E. Atkinson and Weimin Han. Theoretical Numerical Analysis: A Functional
Analysis Framework. Second edition, Springer-Verlag, New York, 2005. xviii+576 pp.
ISBN 0-387-25887-6. (Cited on p. 69.)
[23] Jean-Pierre Aubin and Ivar Ekeland. Applied Nonlinear Analysis. Wiley, New York,
1984. xi+518 pp. ISBN 0-471-05998-6. (Cited on p. 69.)
[24] L. Autonne. Sur les groupes linéaires, réels et orthogonaux. Bulletin de la Société
Mathématique de France, 30:121–134, 1902. (Cited on p. 213.)
[25] Ivo Babuška, Milan Práger, and Emil Vitásek. Numerical Processes in Differential
Equations. Wiley, London, 1966. (Cited on p. 167.)
[26] Zhaojun Bai, Wenbin Chen, Richard Scalettar, and Ichitaro Yamazaki. Lecture notes
on advances of numerical methods for Hubbard quantum Monte Carlo simulation.
Manuscript, 2007. (Cited on pp. 44, 263.)
[27] Zhaojun Bai and James W. Demmel. Design of a parallel nonsymmetric eigenroutine
toolbox, Part I. In Proceedings of the Sixth SIAM Conference on Parallel Processing
for Scientific Computing, Volume I, Richard F. Sincovec, David E. Keyes, Michael R.
Leuze, Linda R. Petzold, and Daniel A. Reed, editors, Society for Industrial and Ap-
plied Mathematics, Philadelphia, PA, USA, 1993, pages 391–398. Also available as
Research Report 92-09, Department of Mathematics, University of Kentucky, Lexing-
ton, KY, USA, December 1992, 30 pp. (Cited on pp. 41, 51.)
[28] Zhaojun Bai and James W. Demmel. On swapping diagonal blocks in real Schur form.
Linear Algebra Appl., 186:73–95, 1993. (Cited on p. 229.)
[29] Zhaojun Bai and James W. Demmel. Using the matrix sign function to compute
invariant subspaces. SIAM J. Matrix Anal. Appl., 19(1):205–225, 1998. (Cited on
pp. 41, 124, 125.)
[30] Zhaojun Bai, James W. Demmel, Jack J. Dongarra, Axel Ruhe, and Henk A. Van der
Vorst, editors. Templates for the Solution of Algebraic Eigenvalue Problems: A Prac-
tical Guide. Society for Industrial and Applied Mathematics, Philadelphia, PA, USA,
2000. xxix+410 pp. ISBN 0-89871-471-0. (Cited on p. 309.)
[31] Zhaojun Bai, James W. Demmel, and Ming Gu. Inverse free parallel spectral divide
and conquer algorithms for nonsymmetric eigenproblems. Numer. Math., 76:279–308,
1997. (Cited on p. 130.)
[32] Zhaojun Bai, Mark Fahey, and Gene H. Golub. Some large-scale matrix computation
problems. J. Comput. Appl. Math., 74:71–89, 1996. (Cited on pp. 44, 318.)
[33] Zhaojun Bai, Mark Fahey, Gene H. Golub, M. Menon, and E. Richter. Computing
partial eigenvalue sum in electronic structure calculations. Technical Report SCCM-
98-03, SCCM, Stanford University, January 1998. 19 pp. (Cited on p. 318.)
[34] Zhaojun Bai and Gene H. Golub. Bounds for the trace of the inverse and the deter-
minant of symmetric positive definite matrices. Ann. Numer. Math., 4:29–38, 1997.
(Cited on p. 318.)
[35] Zhaojun Bai and Gene H. Golub. Some unusual eigenvalue problems. In Vector and
Parallel Processing—VECPAR’98, J. Palma, J. Dongarra, and V. Hernández, editors,
volume 1573 of Lecture Notes in Computer Science, Springer-Verlag, Berlin, 1999,
pages 4–19. (Cited on p. 318.)
382 Bibliography

[36] David H. Bailey. Algorithm 719: Multiprecision translation and execution of FOR-
TRAN programs. ACM Trans. Math. Software, 19(3):288–319, 1993. (Cited on
pp. 188, 286.)
[37] David H. Bailey, Yozo Hida, Xiaoye S. Li, and Brandon Thompson. ARPREC: An
arbitrary precision computation package. Technical Report LBNL-53651, Lawrence
Berkeley National Laboratory, Berkeley, California, March 2002. 8 pp. (Cited on
p. 188.)
[38] George A. Baker, Jr. Essentials of Padé Approximants. Academic Press, New York,
1975. xi+306 pp. (Cited on pp. 80, 104, 264, 274.)
[39] George A. Baker, Jr. and Peter Graves-Morris. Padé Approximants, volume 59 of En-
cyclopedia of Mathematics and Its Applications. Second edition, Cambridge University
Press, 1996. xiv+746 pp. (Cited on pp. 80, 104, 274, 283.)
[40] H. F. Baker. The reciprocation of one quadric into another. Proc. Cambridge Philos.
Soc., 23:22–27, 1925. (Cited on pp. 28, 52.)
[41] A. V. Balakrishnan. Fractional powers of closed operators and the semigroups gener-
ated by them. Pacific J. Math., 10(2):419–437, 1960. (Cited on p. 187.)
[42] R. P. Bambah and S. Chowla. On integer cube roots of the unit matrix. Science and
Culture, 12:105, 1946. (Cited on p. 189.)
[43] Itzhack Y. Bar-Itzhack, J. Meyer, and P. A. Fuhrmann. Strapdown matrix orthogonal-
ization: The dual iterative algorithm. IEEE Trans. Aerospace and Electronic Systems,
AES-12(1):32–37, 1976. (Cited on p. 215.)
[44] A. Y. Barraud. Investigations autour de la fonction signe d’une matrice application a
l’équation de Riccati. R.A.I.R.O. Automatique/Systems Analysis and Control, 13(4):
335–368, 1979. (Cited on p. 130.)
[45] Anders Barrlund. Perturbation bounds on the polar decomposition. BIT, 30:101–113,
1990. (Cited on p. 215.)
[46] Friedrich L. Bauer. Decrypted Secrets: Methods and Maxims of Cryptology. Third
edition, Springer-Verlag, Berlin, 2002. xii+474 pp. ISBN 3-540-42674-4. (Cited on
pp. 165, 167.)
[47] U. Baur and Peter Benner. Factorized solution of Lyapunov equations based on hier-
archical matrix arithmetic. Computing, 78:211–234, 2006. (Cited on pp. 51, 316.)
[48] Connice A. Bavely and G. W. Stewart. An algorithm for computing reducing subspaces
by block diagonalization. SIAM J. Numer. Anal., 16(2):359–367, 1979. (Cited on
p. 89.)
[49] C. A. Beattie and S. W. Smith. Optimal matrix approximants in structural iden-
tification. J. Optimization Theory and Applications, 74(1):23–56, 1992. (Cited on
p. 217.)
[50] Alfredo Bellen and Marino Zennaro. Numerical Methods for Delay Differential Equa-
tions. Oxford University Press, 2003. xiv+395 pp. ISBN 0-19-850654-6. (Cited on
p. 51.)
[51] Richard Bellman. Introduction to Matrix Analysis. Second edition, McGraw-Hill,
New York, 1970. xxiii+403 pp. Reprinted by Society for Industrial and Applied
Mathematics, Philadelphia, PA, USA, 1997. ISBN 0-89871-399-4. (Cited on pp. 51,
105, 165, 167, 264, 267.)
[52] Adi Ben-Israel and Thomas N. E. Greville. Generalized Inverses: Theory and Appli-
cations. Second edition, Springer-Verlag, New York, 2003. xv+420 pp. ISBN 0-387-
00293-6. (Cited on pp. 214, 325, 353.)
Bibliography 383

[53] Peter Benner and Ralph Byers. Disk functions and their relationship to the matrix
sign function. In Proceedings of the European Control Conference ECC97, Paper 936,
BELWARE Information Technology, Waterloo, Belgium, 1997. CD ROM. (Cited on
p. 49.)
[54] Peter Benner, Ralph Byers, Volker Mehrmann, and Hongguo Xu. A unified deflating
subspace approach for classes of polynomial and rational matrix equations. Preprint
SFB393/00-05, Zentrum für Technomathematik, Universität Bremen, Bremen, Ger-
many, January 2000. 29 pp. (Cited on pp. 49, 187.)
[55] Peter Benner and Enrique S. Quintana-Ortı́. Solving stable generalized Lyapunov
equations with the matrix sign function. Numer. Algorithms, 20(1):75–100, 1999.
(Cited on pp. 51, 123.)
[56] Peter Benner, Enrique S. Quintana-Ortı́, and Gregorio Quintana-Ortı́. Solving stable
Sylvester equations via rational iterative schemes. J. Sci. Comput., 28:51–83, 2006.
(Cited on p. 51.)
[57] Michele Benzi and Gene H. Golub. Bounds for the entries of matrix functions with
applications to preconditioning. BIT, 39(3):417–438, 1999. (Cited on p. 317.)
[58] Michele Benzi and Nader Razouk. Decay bounds and O(n) algorithms for approxi-
mating functions of sparse matrices. Electron. Trans. Numer. Anal., 28:16–39, 2007.
(Cited on p. 318.)
[59] Håvard Berland, Bård Skaflestad, and Will Wright. EXPINT—A MATLAB package
for exponential integrators. ACM Trans. Math. Software, 33(1):Article 4, 2007. (Cited
on p. 37.)
[60] Abraham Berman and Robert J. Plemmons. Nonnegative Matrices in the Mathematical
Sciences. Society for Industrial and Applied Mathematics, Philadelphia, PA, USA,
1994. xx+340 pp. Corrected republication, with supplement, of work first published
in 1979 by Academic Press. ISBN 0-89871-321-8. (Cited on pp. 159, 260, 329.)
[61] David S. Bernstein and Charles F. Van Loan. Rational matrix functions and rank-1
updates. SIAM J. Matrix Anal. Appl., 22(1):145–154, 2000. (Cited on p. 28.)
[62] Rajendra Bhatia. Some inequalities for norm ideals. Commun. Math. Phys., 111:
33–39, 1987. (Cited on p. 135.)
[63] Rajendra Bhatia. Matrix factorizations and their perturbations. Linear Algebra Appl.,
197/198:245–276, 1994. (Cited on pp. 215, 217.)
[64] Rajendra Bhatia. Matrix Analysis. Springer-Verlag, New York, 1997. xi+347 pp.
ISBN 0-387-94846-5. (Cited on pp. 69, 187, 215, 217, 237, 315, 330.)
[65] Rajendra Bhatia. Positive Definite Matrices. Princeton University Press, Princeton,
NJ, USA, 2007. ix+254 pp. ISBN 0-691-12918-5. (Cited on pp. 52, 315, 330.)
[66] Rajendra Bhatia and Fuad Kittaneh. Approximation by positive operators. Linear
Algebra Appl., 161:1–9, 1992. (Cited on p. 214.)
[67] Rajendra Bhatia and Kalyan K. Mukherjea. On weighted Löwdin orthogonalization.
Int. J. Quantum Chemistry, 29:1775–1778, 1986. (Cited on p. 42.)
[68] M. D. Bingham. A new method for obtaining the inverse matrix. J. Amer. Statist.
Assoc., 36(216):530–534, 1941. (Cited on p. 90.)
[69] Dario A. Bini, Nicholas J. Higham, and Beatrice Meini. Algorithms for the matrix pth
root. Numer. Algorithms, 39(4):349–378, 2005. (Cited on pp. 187, 188, 191.)
[70] Dario A. Bini, Guy Latouche, and Beatrice Meini. Numerical Methods for Structured
Markov Chains. Oxford University Press, 2005. xi+327 pp. ISBN 0-19-852768-3.
(Cited on p. 45.)
384 Bibliography

[71] R. E. D. Bishop. Arthur Roderick Collar. 22 February 1908–12 February 1986. Bi-
ographical Memoirs of Fellows of the Royal Society, 33:164–185, 1987. (Cited on
p. 27.)
[72] Åke Björck and C. Bowie. An iterative algorithm for computing the best estimate of
an orthogonal matrix. SIAM J. Numer. Anal., 8(2):358–364, 1971. (Cited on p. 215.)
[73] Åke Björck and Sven Hammarling. A Schur method for the square root of a matrix.
Linear Algebra Appl., 52/53:127–140, 1983. (Cited on pp. 163, 166, 167, 171, 187.)
[74] Sergio Blanes and Fernando Casas. On the convergence and optimization of the Baker–
Campbell–Hausdorff formula. Linear Algebra Appl., 378:135–158, 2004. (Cited on
p. 263.)
[75] Artan Boriçi. QCDLAB project. http://phys.fshn.edu.al/qcdlab.html. (Cited on
p. 43.)
[76] Steffen Börm, Lars Grasedyck, and Wolfgang Hackbusch. Hierarchical matrices. Lec-
ture Note No. 21, Max-Planck-Institute for Mathematics in the Sciences, Leipzig, Ger-
many, 2003. 71 pp. Revised June 2006. (Cited on p. 316.)
[77] Jonathan M. Borwein, David Bailey, and Roland Girgensohn. Experimentation in
Mathematics: Computational Paths to Discovery. A K Peters, Natick, Massachusetts,
2004. x+357 pp. ISBN 1-56881-136-5. (Cited on p. 33.)
[78] C. Bouby, D. Fortuné, W. Pietraszkiewicz, and C. Vallée. Direct determination of
the rotation in the polar decomposition of the deformation gradient by maximizing a
Rayleigh quotient. Z. Angew. Math. Mech., 85(3):155–162, 2005. (Cited on p. 43.)
[79] David W. Boyd. The power method for ℓp norms. Linear Algebra Appl., 9:95–101,
1974. (Cited on p. 69.)
[80] Geoff Boyd, Charles A. Micchelli, Gilbert Strang, and Ding-Xuan Zhou. Binomial
matrices. Adv. in Comput. Math., 14:379–391, 2001. (Cited on p. 166.)
[81] Russell J. Bradford, Robert M. Corless, James H. Davenport, David J. Jeffrey, and
Stephen M. Watt. Reasoning about the elementary functions of complex analysis. An-
nals of Mathematics and Artificial Intelligence, 36:303–318, 2002. (Cited on pp. 280,
283.)
[82] T. J. Bridges and P. J. Morris. Differential eigenvalue problems in which the parameter
appears nonlinearly. J. Comput. Phys., 55:437–460, 1984. (Cited on p. 46.)
[83] William Briggs. Ants, Bikes, and Clocks: Problem Solving for Undergraduates. Society
for Industrial and Applied Mathematics, Philadelphia, PA, USA, 2005. 168 pp. ISBN
0-89871-574-1. (Cited on p. 343.)
[84] A. Buchheim. On the theory of matrices. Proc. London Math. Soc., 16:63–82, 1884.
(Cited on p. 26.)
[85] A. Buchheim. An extension of a theorem of Professor Sylvester’s relating to matrices.
Phil. Mag., 22(135):173–174, 1886. Fifth series. (Cited on p. 26.)
[86] G. J. Butler, Charles R. Johnson, and H. Wolkowicz. Nonnegative solutions of a
quadratic matrix equation arising from comparison theorems in ordinary differential
equations. SIAM J. Alg. Discrete Methods, 6(1):47–53, 1985. (Cited on p. 167.)
[87] Ralph Byers. A LINPACK-style condition estimator for the equation AX − XB T = C.
IEEE Trans. Automat. Control, AC-29(10):926–928, 1984. (Cited on p. 226.)
[88] Ralph Byers. Solving the algebraic Riccati equation with the matrix sign function.
Linear Algebra Appl., 85:267–279, 1987. (Cited on pp. 51, 125, 130, 132.)
[89] Ralph Byers, Chunyang He, and Volker Mehrmann. The matrix sign function method
and the computation of invariant subspaces. SIAM J. Matrix Anal. Appl., 18(3):615–
632, 1997. (Cited on pp. 122, 124.)
Bibliography 385

[90] D. Calvetti, E. Gallopoulos, and L. Reichel. Incomplete partial fractions for parallel
evaluation of rational matrix functions. J. Comput. Appl. Math., 59:349–380, 1995.
(Cited on p. 104.)
[91] Daniela Calvetti, Sun-Mi Kim, and Lothar Reichel. Quadrature rules based on the
Arnoldi process. SIAM J. Matrix Anal. Appl., 26(3):765–781, 2005. (Cited on p. 318.)
[92] S. L. Campbell and C. D. Meyer, Jr. Generalized Inverses of Linear Transformations.
Pitman, London, 1979. xi+272 pp. Reprinted by Dover, New York, 1991. ISBN
0-486-66693-X. (Cited on pp. 325, 326, 353.)
[93] João R. Cardoso, Charles S. Kenney, and F. Silva Leite. Computing the square root
and logarithm of a real P -orthogonal matrix. Appl. Numer. Math., 46:173–196, 2003.
(Cited on p. 316.)
[94] João R. Cardoso and F. Silva Leite. The Moser–Veselov equation. Linear Algebra
Appl., 360:237–248, 2003. (Cited on p. 46.)
[95] João R. Cardoso and F. Silva Leite. Padé and Gregory error estimates for the logarithm
of block triangular matrices. Appl. Numer. Math., 56:253–267, 2006. (Cited on p. 284.)
[96] Gianfranco Cariolaro, Tomaso Erseghe, and Peter Kraniauskas. The fractional discrete
cosine transform. IEEE Trans. Signal Processing, 50(4):902–911, 2002. (Cited on
p. 28.)
[97] Lennart Carleson and Theodore W. Gamelin. Complex Dynamics. Springer-Verlag,
New York, 1993. ix+175 pp. ISBN 0-387-97942-5. (Cited on p. 156.)
[98] A. J. Carpenter, A. Ruttan, and R. S. Varga. Extended numerical computations on
the “1/9” conjecture in rational approximation theory. In Rational Approximation and
Interpolation, P. R. Graves-Morris, E. B. Saff, and R. S. Varga, editors, volume 1105 of
Lecture Notes in Mathematics, Springer-Verlag, Berlin, 1984, pages 383–411. (Cited
on pp. 259, 267.)
[99] Arthur Cayley. A memoir on the theory of matrices. Philos. Trans. Roy. Soc. London,
148:17–37, 1858. (Cited on pp. 26, 34, 166, 168, 191.)
[100] Arthur Cayley. On the extraction of the square root of a matrix of the third order.
Proc. Roy. Soc. Edinburgh, 7:675–682, 1872. (Cited on pp. 26, 166, 168.)
[101] Arthur Cayley. The Newton–Fourier imaginary problem. Amer. J. Math., 2(1):97,
1879. (Cited on pp. 187, 191.)
[102] Elena Celledoni and Arieh Iserles. Approximating the exponential from a Lie algebra
to a Lie group. Math. Comp., 69(232):1457–1480, 2000. (Cited on p. 28.)
[103] F. Chaitin-Chatelin and S. Gratton. On the condition numbers associated with the
polar factorization of a matrix. Numer. Linear Algebra Appl., 7:337–354, 2000. (Cited
on p. 201.)
[104] Raymond H. Chan, Chen Greif, and Dianne P. O’Leary, editors. Milestones in Matrix
Computation: The Selected Works of Gene H. Golub, with Commentaries. Oxford
University Press, 2007. xi+565 pp. ISBN 978-0-19-920681-0. (Cited on p. 391.)
[105] Shivkumar Chandrasekaran and Ilse C. F. Ipsen. Backward errors for eigenvalue and
singular value decompositions. Numer. Math., 68:215–223, 1994. (Cited on p. 214.)
[106] Chi-Tsong Chen. Linear System Theory and Design. Third edition, Oxford University
Press, 1999. xiii+334 pp. ISBN 0-19-511777-8. (Cited on p. 51.)
[107] Sheung Hun Cheng, Nicholas J. Higham, Charles S. Kenney, and Alan J. Laub. Return
to the middle ages: A half-angle iteration for the logarithm of a unitary matrix. In
Proceedings of the Fourteenth International Symposium of Mathematical Theory of
Networks and Systems, Perpignan, France, 2000. CD ROM. (Cited on p. 316.)
386 Bibliography

[108] Sheung Hun Cheng, Nicholas J. Higham, Charles S. Kenney, and Alan J. Laub. Ap-
proximating the logarithm of a matrix to specified accuracy. SIAM J. Matrix Anal.
Appl., 22(4):1112–1125, 2001. (Cited on pp. 104, 141, 166, 167, 283, 284, 285.)
[109] M. Cipolla. Sulle matrice espressione analitiche di un’altra. Rendiconti Circolo Matem-
atico de Palermo, 56:144–154, 1932. (Cited on p. 26.)
[110] W. J. Cody, G. Meinardus, and R. S. Varga. Chebyshev rational approximations to e−x
in [0, +∞) and applications to heat-conduction problems. J. Approximation Theory,
2:50–65, 1969. (Cited on p. 259.)
[111] John P. Coleman. Rational approximations for the cosine function; P-acceptability
and order. Numer. Algorithms, 3:143–158, 1992. (Cited on p. 300.)
[112] A. R. Collar. The first fifty years of aeroelasticity. Aerospace (Royal Aeronautical
Society Journal), 5:12–20, 1978. (Cited on pp. 27, 53.)
[113] Samuel D. Conte and Carl de Boor. Elementary Numerical Analysis: An Algorithmic
Approach. Third edition, McGraw-Hill, Tokyo, 1980. xii+432 pp. ISBN 0-07-066228-2.
(Cited on p. 333.)
[114] Robert M. Corless, Hui Ding, Nicholas J. Higham, and David J. Jeffrey. The solution of
S exp(S) = A is not always the Lambert W function of A. In ISSAC ’ 07: Proceedings
of the 2007 International Symposium on Symbolic and Algebraic Computation, New
York, 2007, pages 116–121. ACM Press. (Cited on p. 51.)
[115] Robert M. Corless, Gaston H. Gonnet, D. E. G. Hare, David J. Jeffrey, and Donald E.
Knuth. On the Lambert W function. Adv. in Comput. Math., 5(4):329–359, 1996.
(Cited on pp. 51, 53.)
[116] Robert M. Corless and David J. Jeffrey. The unwinding number. ACM SIGSAM
Bulletin, 30(2):28–35, 1996. (Cited on p. 283.)
[117] Marius Cornea-Hasegan and Bob Norin. IA-64 floating-point operations and the IEEE
standard for binary floating-point arithmetic. Intel Technology Journal, 3, 1999. http:
//developer.intel.com/technology/itj/. (Cited on p. 188.)
[118] S. M. Cox and P. C. Matthews. Exponential time differencing for stiff systems. J.
Comput. Phys., 176:430–455, 2002. (Cited on p. 37.)
[119] Trevor F. Cox and Michael A. A. Cox. Multidimensional Scaling. Chapman and Hall,
London, 1994. xi+213 pp. ISBN 0-412-49120-6. (Cited on p. 43.)
[120] Tony Crilly. Cayley’s anticipation of a generalised Cayley–Hamilton theorem. Historia
Mathematica, 5:211–219, 1978. (Cited on p. 30.)
[121] Tony Crilly. Arthur Cayley: Mathematician Laureate of the Victorian Age. Johns
Hopkins University Press, Baltimore, MD, USA, 2006. xxi+610 pp. ISBN 0-8018-
8011-4. (Cited on pp. 26, 30.)
[122] G. W. Cross and P. Lancaster. Square roots of complex matrices. Linear and Multi-
linear Algebra, 1:289–293, 1974. (Cited on p. 16.)
[123] Michel Crouzeix. Bounds for analytical functions of matrices. Integral Equations and
Operator Theory, 48:461–477, 2004. (Cited on p. 105.)
[124] L. Csanky. Fast parallel matrix inversion algorithms. SIAM J. Comput., 5(4):618–623,
1976. (Cited on p. 90.)
[125] Charles G. Cullen. Matrices and Linear Transformations. Second edition, Addison-
Wesley, Reading, MA, USA, 1972. xii+318 pp. Reprinted by Dover, New York, 1990.
ISBN 0-486-66328-0. (Cited on pp. 28, 29, 52.)
[126] Walter J. Culver. On the existence and uniqueness of the real logarithm of a matrix.
Proc. Amer. Math. Soc., 17:1146–1151, 1966. (Cited on p. 17.)
Bibliography 387

[127] James R. Cuthbert. On uniqueness of the logarithm for Markov semi-groups. J. London
Math. Soc., 4:623–630, 1972. (Cited on p. 190.)
[128] Germund Dahlquist. Stability and Error Bounds in the Numerical Integration of Or-
dinary Differential Equations. PhD thesis, Royal Inst. of Technology, Stockholm, Swe-
den, 1958. Reprinted in Trans. Royal Inst. of Technology, No. 130, Stockholm, Sweden,
1959. (Cited on p. 237.)
[129] Ju. L. Daleckiı̆. Differentiation of non-Hermitian matrix functions depending on a
parameter. Amer. Math. Soc. Transl., Series 2, 47:73–87, 1965. (Cited on p. 69.)
[130] Ju. L. Daleckiı̆ and S. G. Kreı̆n. Integration and differentiation of functions of Her-
mitian operators and applications to the theory of perturbations. Amer. Math. Soc.
Transl., Series 2, 47:1–30, 1965. (Cited on p. 69.)
[131] E. B. Davies. Approximate diagonalization. SIAM J. Matrix Anal. Appl., 29(4):1051–
1064, 2007. (Cited on p. 30.)
[132] E. Brian Davies. Science in the Looking Glass: What Do Scientists Really Know?
Oxford University Press, 2003. x+295 pp. ISBN 0-19-852543-5. (Cited on p. 34.)
[133] E. Brian Davies. Linear Operators and their Spectra. Cambridge University Press,
Cambridge, UK, 2007. xii+451 pp. ISBN 978-0-521-86629-3. (Cited on p. 28.)
[134] Philip I. Davies. Structured conditioning of matrix functions. Electron. J. Linear
Algebra, 11:132–161, 2004. (Cited on p. 315.)
[135] Philip I. Davies and Nicholas J. Higham. A Schur–Parlett algorithm for computing
matrix functions. SIAM J. Matrix Anal. Appl., 25(2):464–485, 2003. (Cited on pp. 229,
231.)
[136] Philip I. Davies and Nicholas J. Higham. Computing f (A)b for matrix functions f .
In QCD and Numerical Analysis III, Artan Boriçi, Andreas Frommer, Báalint Joó,
Anthony Kennedy, and Brian Pendleton, editors, volume 47 of Lecture Notes in Com-
putational Science and Engineering, Springer-Verlag, Berlin, 2005, pages 15–24. (Cited
on pp. 307, 308, 310.)
[137] Philip I. Davies, Nicholas J. Higham, and Françoise Tisseur. Analysis of the Cholesky
method with iterative refinement for solving the symmetric definite generalized eigen-
problem. SIAM J. Matrix Anal. Appl., 23(2):472–493, 2001. (Cited on p. 35.)
[138] Philip I. Davies and Matthew I. Smith. Updating the singular value decomposition.
J. Comput. Appl. Math., 170:145–167, 2004. (Cited on p. 216.)
[139] Chandler Davis. Explicit functional calculus. Linear Algebra Appl., 6:193–199, 1973.
(Cited on p. 84.)
[140] George J. Davis. Numerical solution of a quadratic matrix equation. SIAM J. Sci.
Statist. Comput., 2(2):164–175, 1981. (Cited on p. 52.)
[141] Philip J. Davis and Aviezri S. Fraenkel. Remembering Philip Rabinowitz. Notices
Amer. Math. Soc., 54(11):1502–1506, 2007. (Cited on p. 311.)
[142] Philip J. Davis and Philip Rabinowitz. Methods of Numerical Integration. Second
edition, Academic Press, London, 1984. xiv+612 pp. ISBN 0-12-206360-0. (Cited on
pp. 274, 308.)
[143] Carl de Boor. Divided differences. Surveys in Approximation Theory, 1:46–69, 2005.
(Cited on pp. 264, 333.)
[144] Lokenath Debnath and Piotr Mikusiński. Introduction to Hilbert Spaces with Appli-
cations. Second edition, Academic Press, San Diego, CA, USA, 1999. xviii+551 pp.
ISBN 0-12-208436-5. (Cited on p. 167.)
[145] James W. Demmel. Applied Numerical Linear Algebra. Society for Industrial and
Applied Mathematics, Philadelphia, PA, USA, 1997. xi+419 pp. ISBN 0-89871-389-7.
(Cited on p. 325.)
388 Bibliography

[146] Eugene D. Denman and Alex N. Beavers, Jr. The matrix sign function and computa-
tions in systems. Appl. Math. Comput., 2:63–94, 1976. (Cited on pp. 52, 141.)
[147] John E. Dennis, Jr., J. F. Traub, and R. P. Weber. The algebraic theory of matrix
polynomials. SIAM J. Numer. Anal., 13(6):831–845, 1976. (Cited on p. 52.)
[148] Jean Descloux. Bounds for the spectral norm of functions of matrices. Numer. Math.,
15:185–190, 1963. (Cited on pp. 84, 103.)
[149] Robert L. Devaney. An Introduction to Chaotic Dynamical Systems. Second edition,
Addison-Wesley, Reading, MA, USA, 1989. xvi+336 pp. ISBN 0-201-13046-7. (Cited
on p. 156.)
[150] Inderjit S. Dhillon and Joel A. Tropp. Matrix nearness problems with Bregman diver-
gences. SIAM J. Matrix Anal. Appl., 29(4):1120–1146, 2007. (Cited on p. 50.)
[151] Bradley W. Dickinson and Kenneth Steiglitz. Eigenvectors and functions of the discrete
Fourier transform. IEEE Trans. Acoust., Speech, Signal Processing, ASSP-30(1):25–31,
1982. (Cited on p. 28.)
[152] Luca Dieci. Considerations on computing real logarithms of matrices, Hamiltonian
logarithms, and skew-symmetric logarithms. Linear Algebra Appl., 244:35–54, 1996.
(Cited on pp. 50, 315, 316.)
[153] Luca Dieci. Real Hamiltonian logarithm of a symplectic matrix. Linear Algebra Appl.,
281:227–246, 1998. (Cited on p. 315.)
[154] Luca Dieci, Benedetta Morini, and Alessandra Papini. Computational techniques for
real logarithms of matrices. SIAM J. Matrix Anal. Appl., 17(3):570–593, 1996. (Cited
on pp. 274, 283, 284.)
[155] Luca Dieci, Benedetta Morini, Alessandra Papini, and Aldo Pasquali. On real loga-
rithms of nearby matrices and structured matrix interpolation. Appl. Numer. Math.,
29:145–165, 1999. (Cited on p. 50.)
[156] Luca Dieci and Alessandra Papini. Conditioning and Padé approximation of the log-
arithm of a matrix. SIAM J. Matrix Anal. Appl., 21(3):913–930, 2000. (Cited on
p. 284.)
[157] Luca Dieci and Alessandra Papini. Padé approximation for the exponential of a block
triangular matrix. Linear Algebra Appl., 308:183–202, 2000. (Cited on pp. 243, 246,
248, 264, 265.)
[158] Luca Dieci and Alessandra Papini. Conditioning of the exponential of a block triangular
matrix. Numer. Algorithms, 28:137–150, 2001. (Cited on p. 264.)
[159] J. Dieudonné. Foundations of Modern Analysis. Academic Press, New York, 1960.
xiv+361 pp. (Cited on p. 58.)
[160] John D. Dixon. Estimating extremal eigenvalues and condition numbers of matrices.
SIAM J. Numer. Anal., 20(4):812–814, 1983. (Cited on p. 69.)
[161] Elizabeth D. Dolan and Jorge J. Moré. Benchmarking optimization software with
performance profiles. Math. Programming, 91:201–213, 2002. (Cited on p. 252.)
[162] Jack J. Dongarra, Iain S. Duff, Danny C. Sorensen, and Henk A. Van der Vorst.
Numerical Linear Algebra for High-Performance Computers. Society for Industrial
and Applied Mathematics, Philadelphia, PA, USA, 1998. xviii+342 pp. ISBN 0-
89871-428-1. (Cited on p. 335.)
[163] William F. Donoghue, Jr. Monotone Matrix Functions and Analytic Continuation.
Springer-Verlag, Berlin, 1974. 182 pp. ISBN 3-540-06543-1. (Cited on p. 315.)
[164] M. P. Drazin, J. W. Dungey, and K. W. Gruenberg. Some theorems on commutative
matrices. J. London Math. Soc., 26(2):221–228, 1951. (Cited on p. 29.)
Bibliography 389

[165] P. G. Drazin. Nonlinear Systems. Cambridge University Press, Cambridge, UK, 1992.
xiii+317 pp. ISBN 0-521-40668-4. (Cited on p. 359.)
[166] V. Druskin, A. Greenbaum, and L. Knizhnerman. Using nonorthogonal Lanczos vectors
in the computation of matrix functions. SIAM J. Sci. Comput., 19(1):38–54, 1998.
(Cited on p. 310.)
[167] Vladimir L. Druskin and Leonid A. Knizhnerman. Two polynomial methods of calcu-
lating functions of symmetric matrices. U.S.S.R. Comput. Maths. Math. Phys., 29(6):
112–121, 1989. (Cited on p. 309.)
[168] Ian L. Dryden and Kanti V. Mardia. Statistical Shape Analysis. Wiley, New York,
1998. xvii+347 pp. ISBN 0-471-95816-6. (Cited on p. 43.)
[169] Augustin A. Dubrulle. An optimum iteration for the matrix polar decomposition.
Electron. Trans. Numer. Anal., 8:21–25, 1999. (Cited on pp. 216, 218.)
[170] B. J. Duke. Certification of Algorithm 298 [F1]: Determination of the square root of
a positive definite matrix. Comm. ACM, 12(6):325–326, 1969. (Cited on p. 167.)
[171] Nelson Dunford and Jacob T. Schwartz. Linear Operators. Part I: General Theory.
Wiley, New York, 1988. xiv+858 pp. Wiley Classics Library edition. ISBN 0-471-
60848-3. (Cited on p. 28.)
[172] Nelson Dunford and Jacob T. Schwartz. Linear Operators. Part III: Spectral Operators.
Wiley, New York, 1971. xix+1925–2592 pp. (Cited on p. 28.)
[173] Alan Edelman and Gilbert Strang. Pascal matrices. Amer. Math. Monthly, 111(3):
189–197, 2004. (Cited on p. 374.)
[174] Michael Eiermann and Oliver G. Ernst. A restarted Krylov subspace method for the
evaluation of matrix functions. SIAM J. Numer. Anal., 44(6):2481–2504, 2006. (Cited
on p. 305.)
[175] Timo Eirola. A refined polar decomposition: A = U P D. SIAM J. Matrix Anal. Appl.,
22(3):824–836, 2000. (Cited on p. 214.)
[176] Ludwig Elsner. Iterative Verfahren zur Lösung der Matrizengleichung X 2 − A = 0.
Buletinul Institutului Politehnic din Iasi, xvi(xx):15–24, 1970. (Cited on pp. 106, 159,
167, 169.)
[177] Ivan Erdelyi. On partial isometries in finite-dimensional Euclidean spaces. SIAM J.
Appl. Math., 14(3):453–467, 1966. (Cited on p. 326.)
[178] J.-Cl. Evard and F. Jafari. A complex Rolle’s theorem. Amer. Math. Monthly, 99(9):
858–861, 1992. (Cited on p. 333.)
[179] Jean-Claude Evard and Frank Uhlig. On the matrix equation f (X) = A. Linear
Algebra Appl., 162–164:447–519, 1992. (Cited on p. 28.)
[180] D. K. Faddeev and V. N. Faddeeva. Numerische Methoden der Linearen Algebra. Veb
Deutscher Verlag der Wissenschaften, Berlin, 1964. (Cited on p. 370.)
[181] Ky Fan and A. J. Hoffman. Some metric inequalities in the space of matrices. Proc.
Amer. Math. Soc., 6:111–116, 1955. (Cited on p. 214.)
[182] Heike Faßbender, D. Steven Mackey, Niloufer Mackey, and Hongguo Xu. Hamiltonian
square roots of skew-Hamiltonian matrices. Linear Algebra Appl., 287:125–159, 1999.
(Cited on p. 315.)
[183] T. I. Fenner and G. Loizou. Optimally scalable matrices. Philos. Trans. Roy. Soc.
London Ser. A, 287(1345):307–349, 1977. (Cited on p. 105.)
[184] W. L. Ferrar. Finite Matrices. Oxford University Press, 1951. vii+182 pp. (Cited on
pp. 27, 34.)
[185] Miroslav Fiedler and Hans Schneider. Analytic functions of M -matrices and general-
izations. Linear and Multilinear Algebra, 13:185–201, 1983. (Cited on p. 188.)
390 Bibliography

[186] P. Filipponi. An algorithm for computing functions of triangular matrices. Computing,

26:67–71, 1981. (Cited on p. 104.)
[187] Steven R. Finch. Mathematical Constants. Cambridge University Press, Cambridge,
UK, 2003. xix+602 pp. ISBN 0-521-81805-2. (Cited on p. 259.)
[188] Harley Flanders. Elementary divisors of AB and BA. Proc. Amer. Math. Soc., 2(6):
871–874, 1951. (Cited on p. 28.)
[189] J. Fortiana and C. M. Cuadras. A family of matrices, the discretized Brownian bridge,
and distance-based regression. Linear Algebra Appl., 264:173–188, 1997. (Cited on
p. 349.)
[190] David Fowler and Eleanor Robson. Square root approximations in old Babylonian
mathematics: YBC 7289 in context. Historia Mathematica, 25:366–378, 1998. (Cited
on p. 166.)
[191] Chris Fraley. Test problems for unconstrained optimization and nonlinear least squares.
http://www.netlib.org/uncon. (Cited on p. 46.)
[192] Gene F. Franklin, J. David Powell, and Michael L. Workman. Digital Control of
Dynamic Systems. Third edition, Addison-Wesley, Reading, MA, USA, 1998. xxiii+742
pp. ISBN 0-201-33153-5. (Cited on p. 51.)
[193] R. A. Frazer, W. J. Duncan, and A. R. Collar. Elementary Matrices and Some Ap-
plications to Dynamics and Differential Equations. Cambridge University Press, 1938.
xviii+416 pp. 1963 printing. (Cited on pp. 27, 262.)
[194] Paul Friedland. Algorithm 312: Absolute value and square root of a complex number.
Comm. ACM, 10(10):665, 1967. (Cited on p. 166.)
[195] G. Frobenius. Über die cogredienten Transformationen der bilinearen Formen.
Sitzungsber K. Preuss. Akad. Wiss. Berlin, 16:7–16, 1896. (Cited on pp. 26, 28,
52.)
[196] Andreas Frommer, Thomas Lippert, Björn Medeke, and Klaus Schilling, editors. Nu-
merical Challenges in Lattice Quantum Chromodynamics, volume 15 of Lecture Notes
in Computational Science and Engineering. Springer-Verlag, Berlin, 2000. viii+184
pp. ISBN 3-540-67732-1. (Cited on p. 43.)
[197] Andreas Frommer and Valeria Simoncini. Stopping criteria for rational matrix func-
tions of Hermitian and symmetric matrices. Manuscript, 2007. 24 pp. (Cited on
p. 306.)
[198] Andreas Frommer and Valeria Simoncini. Matrix functions. In Model Order Reduction:
Theory, Research Aspects and Applications, W. Schilders and H. A. Van der Vorst,
editors, Springer-Verlag, Berlin, 2008. To appear. (Cited on pp. 43, 310.)
[199] Jean Gallier and Dianna Xu. Computing exponentials of skew-symmetric matrices and
logarithms of orthogonal matrices. International Journal of Robotics and Automation,
17(4):1–11, 2002. (Cited on pp. 50, 266.)
[200] E. Gallopoulos and Y. Saad. Efficient solution of parabolic equations by Krylov ap-
proximation methods. SIAM J. Sci. Statist. Comput., 13(5):1236–1264, 1992. (Cited
on pp. 260, 263, 309.)
[201] Walter Gander. On Halley’s iteration method. Amer. Math. Monthly, 92(2):131–134,
1985. (Cited on p. 131.)
[202] Walter Gander. Algorithms for the polar decomposition. SIAM J. Sci. Statist. Com-
put., 11(6):1102–1115, 1990. (Cited on p. 215.)
[203] F. R. Gantmacher. The Theory of Matrices, volume one. Chelsea, New York, 1959.
x+374 pp. ISBN 0-8284-0131-4. (Cited on pp. xvii, 3, 28, 51.)
[204] Judith D. Gardiner. A stabilized matrix sign function algorithm for solving algebraic
Riccati equations. SIAM J. Sci. Comput., 18(5):1393–1411, 1997. (Cited on p. 51.)
Bibliography 391

[205] Judith D. Gardiner and Alan J. Laub. A generalization of the matrix-sign-function

solution for algebraic Riccati equations. Internat. J. Control, 44(3):823–832, 1986.
(Cited on p. 130.)
[206] Walter Gautschi. Algorithm 726: ORTHPOL—A package of routines for generating
orthogonal polynomials and Gauss-type quadrature rules. ACM Trans. Math. Software,
20(1):21–62, 1994. (Cited on p. 274.)
[207] Walter Gautschi. Numerical Analysis: An Introduction. Birkhäuser, Boston, MA,
USA, 1997. xiii+506 pp. ISBN 0-8176-3895-4. (Cited on p. 263.)
[208] Walter Gautschi. Orthogonal Polynomials: Computation and Approximation. Oxford
University Press, 2004. viii+301 pp. ISBN 0-19-850672-4. (Cited on p. 274.)
[209] Ivan P. Gavrilyuk, Wolfgang Hackbusch, and Boris N. Khoromskij. H-matrix approxi-
mation for the operator exponential with applications. Numer. Math., 92:83–111, 2002.
(Cited on p. 316.)
[210] A. O. Gel’fond. Calculus of Finite Differences. Hindustan Publishing Corporation,
Delhi, 1971. vi+451 pp. Authorized English translation of the third Russian edition.
(Cited on p. 333.)
[211] James E. Gentle. Random Number Generation and Monte Carlo Methods. Second
edition, Springer-Verlag, New York, 2003. xv+381 pp. ISBN 0-387-00178-6. (Cited
on p. 118.)
[212] Alan George and Kh. Ikramov. Is the polar decomposition finitely computable? SIAM
J. Matrix Anal. Appl., 17(2):348–354, 1996. (Cited on p. 215.)
[213] Alan George and Kh. Ikramov. Addendum: Is the polar decomposition finitely com-
putable? SIAM J. Matrix Anal. Appl., 18(1):264, 1997. (Cited on p. 215.)
[214] Michael Gil. Estimate for the norm of matrix-valued functions. Linear and Multilinear
Algebra, 35:65–73, 1993. (Cited on p. 105.)
[215] Philip E. Gill, Walter Murray, and Margaret H. Wright. Practical Optimization. Aca-
demic Press, London, 1981. xvi+401 pp. ISBN 0-12-283952-8. (Cited on p. 199.)
[216] G. Giorgi. Nuove osservazioni sulle funzioni delle matrici. Atti Accad. Lincei Rend., 6
(8):3–8, 1928. (Cited on p. 26.)
[217] Glucat: Generic library of universal Clifford algebra templates. http://glucat.
sourceforge.net/. (Cited on p. 47.)
[218] GNU MP: Multiple precision arithmetic library. http://www.swox.com/gmp/. (Cited
on p. 187.)
[219] S. K. Godunov. Ordinary Differential Equations with Constant Coefficient, volume
169 of Translations of Mathematical Monographs. American Mathematical Society,
Providence, RI, USA, 1997. ix+282 pp. ISBN 0-8218-0656-4. (Cited on p. 51.)
[220] I. Gohberg, Peter Lancaster, and Leiba Rodman. Matrix Polynomials. Academic Press,
New York, 1982. xiv+409 pp. ISBN 0-12-287160-X. (Cited on p. 52.)
[221] Jerome A. Goldstein and Mel Levy. Linear algebra and quantum chemistry. Amer.
Math. Monthly, 98(8):710–718, 1991. (Cited on p. 42.)
[222] Herman H. Goldstine. A History of Numerical Analysis from the 16th through the 19th
Century. Springer-Verlag, New York, 1977. xiv+348 pp. ISBN 0-387-90277-5. (Cited
on pp. 283, 286.)
[223] Gene H. Golub and Gerard Meurant. Matrices, moments and quadrature. In Numer-
ical Analysis 1993, Proceedings of the 15th Dundee Conference, D. F. Griffiths and
G. A. Watson, editors, volume 303 of Pitman Research Notes in Mathematics, Addi-
son Wesley Longman, Harlow, Essex, UK, 1994, pages 105–156. Reprinted in [104].
(Cited on p. 318.)
392 Bibliography

[224] Gene H. Golub and Charles F. Van Loan. Matrix Computations. Third edition, Johns
Hopkins University Press, Baltimore, MD, USA, 1996. xxvii+694 pp. ISBN 0-8018-
5413-X (hardback), 0-8018-5414-8 (paperback). (Cited on pp. xvii, 28, 69, 73, 77,
89, 105, 139, 196, 209, 231, 262, 309, 335, 337.)
[225] Nicholas I. M. Gould, Dominique Orban, and Philippe L. Toint. CUTEr and SifDec:
A constrained and unconstrained testing environment, revisited. ACM Trans. Math.
Software, 29(4):373–394, 2003. (Cited on p. 46.)
[226] John C. Gower and Garmt B. Dijksterhuis. Procrustes Problems. Oxford University
Press, 2004. xiv+233 pp. ISBN 0-19-851058-6. (Cited on p. 43.)
[227] W. B. Gragg. The Padé table and its relation to certain algorithms of numerical
analysis. SIAM Rev., 14(1):1–62, 1972. (Cited on p. 104.)
[228] L. Grasedyck, W. Hackbusch, and B. N. Khoromskij. Solution of large scale algebraic
matrix Riccati equations by use of hierarchical matrices. Computing, 70:121–165, 2003.
(Cited on pp. 51, 316.)
[229] Bert F. Green. The orthogonal approximation of an oblique structure in factor analysis.
Psychometrika, 17(4):429–440, 1952. (Cited on p. 214.)
[230] Anne Greenbaum. Some theoretical results derived from polynomial numerical hulls
of Jordan blocks. Electron. Trans. Numer. Anal., 18:81–90, 2004. (Cited on p. 105.)
[231] R. Grone, C. R. Johnson, E. M. Sá, and H. Wolkowicz. Normal matrices. Linear
Algebra Appl., 87:213–225, 1987. (Cited on p. 323.)
[232] Ming Gu. Finding well-conditioned similarities to block-diagonalize nonsymmetric
matrices is NP-hard. Journal of Complexity, 11(3):377–391, 1995. (Cited on p. 226.)
[233] Chun-Hua Guo and Nicholas J. Higham. A Schur–Newton method for the matrix pth
root and its inverse. SIAM J. Matrix Anal. Appl., 28(3):788–804, 2006. (Cited on
pp. 51, 186, 188, 190.)
[234] Chun-Hua Guo and Peter Lancaster. Analysis and modification of Newton’s method
for algebraic Riccati equations. Math. Comp., 67(223):1089–1105, 1998. (Cited on
p. 144.)
[235] Hongbin Guo and Rosemary A. Renaut. Estimation of uT f (A)v for large-scale un-
symmetric matrices. Numer. Linear Algebra Appl., 11:75–89, 2004. (Cited on p. 318.)
[236] Markus Haase. The Functional Calculus for Sectorial Operators. Number 169 in
Operator Theory: Advances and Applications. Birkhäuser, Basel, Switzerland, 2006.
xiv+392 pp. ISBN 3-7643-7697-X. (Cited on p. 187.)
[237] William W. Hager. Condition estimates. SIAM J. Sci. Statist. Comput., 5(2):311–316,
1984. (Cited on p. 69.)
[238] E. Hairer and G. Wanner. Analysis by Its History. Springer-Verlag, New York, 1996.
x+374 pp. ISBN 0-387-94551-2. (Cited on p. 283.)
[239] Ernst Hairer, Christian Lubich, and Gerhard Wanner. Geometric Numerical Integra-
tion: Structure-Preserving Algorithms for Ordinary Differential Equations. Springer-
Verlag, Berlin, 2002. xiii+515 pp. ISBN 3-540-43003-2. (Cited on pp. 42, 315.)
[240] Nicholas Hale, Nicholas J. Higham, and Lloyd N. Trefethen. Computing Aα , log(A)
and related matrix functions by contour integrals. MIMS EPrint 2007.103, Manchester
Institute for Mathematical Sciences, The University of Manchester, UK, August 2007.
19 pp. (Cited on p. 308.)
[241] Brian C. Hall. Lie Groups, Lie Algebras, and Representations. Springer-Verlag, New
York, 2003. xiv+351 pp. ISBN 0-387-40122-9. (Cited on pp. 262, 263.)
[242] Paul R. Halmos. Positive approximants of operators. Indiana Univ. Math. J., 21(10):
951–960, 1972. (Cited on p. 214.)
Bibliography 393

[243] Paul R. Halmos. Finite-Dimensional Vector Spaces. Springer-Verlag, New York, 1974.
viii+200 pp. Reprint of the Second edition published by Van Nostrand, Princeton, NJ,
1958. ISBN 0-387-90093-4. (Cited on p. 214.)
[244] Paul R. Halmos. A Hilbert Space Problem Book. Second edition, Springer-Verlag,
Berlin, 1982. xvii+369 pp. ISBN 0-387-90685-1. (Cited on pp. 29, 167.)
[245] H. L. Hamburger and M. E. Grimshaw. Linear Transformations in n-Dimensional
Vector Space: An Introduction to the Theory of Hilbert Space. Cambridge University
Press, 1951. x+195 pp. (Cited on p. 27.)
[246] Richard J. Hanson and Michael J. Norris. Analysis of measurements based on the
singular value decomposition. SIAM J. Sci. Statist. Comput., 2(3):363–373, 1981.
(Cited on p. 368.)
[247] Gareth Hargreaves. Topics in Matrix Computations: Stability and Efficiency of Algo-
rithms. PhD thesis, University of Manchester, Manchester, England, 2005. 204 pp.
(Cited on p. 74.)
[248] Gareth I. Hargreaves and Nicholas J. Higham. Efficient algorithms for the matrix
cosine and sine. Numer. Algorithms, 40(4):383–400, 2005. (Cited on pp. 292, 299.)
[249] Lawrence A. Harris. Computation of functions of certain operator matrices. Linear
Algebra Appl., 194:31–34, 1993. (Cited on p. 28.)
[250] W. F. Harris. The average eye. Opthal. Physiol. Opt., 24:580–585, 2005. (Cited on
p. 50.)
[251] W. F. Harris and J. R. Cardoso. The exponential-mean-log-transference as a possible
representation of the optical character of an average eye. Opthal. Physiol. Opt., 26(4):
380–383, 2006. (Cited on p. 50.)
[252] Timothy F. Havel, Igor Najfeld, and Ju-xing Yang. Matrix decompositions of two-
dimensional nuclear magnetic resonance spectra. Proc. Nat. Acad. Sci. USA, 91:7962–
7966, 1994. (Cited on p. 51.)
[253] Thomas Hawkins. The theory of matrices in the 19th century. In Proceedings of the
International Congress of Mathematicians, Vancouver, volume 2, 1974, pages 561–570.
(Cited on p. 26.)
[254] Thomas Hawkins. Another look at Cayley and the theory of matrices. Arch. Internat.
Histoire Sci., 27(100):82–112, 1977. (Cited on p. 26.)
[255] Thomas Hawkins. Weierstrass and the theory of matrices. Archive for History of Exact
Sciences, 12(2):119–163, 1977. (Cited on p. 26.)
[256] Jane M. Heffernan and Robert M. Corless. Solving some delay differential equations
with computer algebra. Mathematical Scientist, 31(1):21–34, 2006. (Cited on p. 51.)
[257] Peter Henrici. Bounds for iterates, inverses, spectral variation and fields of values of
non-normal matrices. Numer. Math., 4:24–40, 1962. (Cited on p. 103.)
[258] Kurt Hensel. Über Potenzreihen von Matrizen. J. Reine Angew. Math., 155(2):107–
110, 1926. (Cited on pp. 26, 104.)
[259] Konrad J. Heuvers and Daniel Moak. Matrix solutions of the functional equation of
the gamma function. Aequationes Mathematicae, 33:1–17, 1987. (Cited on p. 27.)
[260] The Hewlett-Packard HP-35 Scientific Pocket Calculator. Hewlett-Packard, Cupertino,
CA, USA, 1973. 4 pp. Advertising brochure. 5952-6000 Rev. 7/73. (Cited on p. 286.)
[261] HP 48 Programmer’s Reference Manual. Hewlett-Packard, Corvallis Division, Corval-
lis, OR, USA, July 1990. 504 pp. Mfg. No. 00048-90053. (Cited on p. 286.)
[262] Desmond J. Higham. Time-stepping and preserving orthonormality. BIT, 37(1):24–36,
1997. (Cited on p. 42.)
394 Bibliography

[263] Desmond J. Higham and Nicholas J. Higham. MATLAB Guide. Second edition, Society
for Industrial and Applied Mathematics, Philadelphia, PA, USA, 2005. xxiii+382 pp.
ISBN 0-89871-578-4. (Cited on p. 252.)
[264] Nicholas J. Higham. The Matrix Computation Toolbox. http://www.ma.man.ac.uk/
~higham/mctoolbox. (Cited on pp. 129, 149, 229, 295.)
[265] Nicholas J. Higham. Nearness Problems in Numerical Linear Algebra. PhD thesis,
University of Manchester, Manchester, England, July 1985. 173 pp. (Cited on p. 216.)
[266] Nicholas J. Higham. Computing the polar decomposition—with applications. SIAM
J. Sci. Statist. Comput., 7(4):1160–1174, 1986. (Cited on pp. 167, 215, 216.)
[267] Nicholas J. Higham. Newton’s method for the matrix square root. Math. Comp., 46
(174):537–549, 1986. (Cited on pp. 104, 166, 167.)
[268] Nicholas J. Higham. Computing real square roots of a real matrix. Linear Algebra
Appl., 88/89:405–430, 1987. (Cited on pp. 28, 166, 167.)
[269] Nicholas J. Higham. Computing a nearest symmetric positive semidefinite matrix.
Linear Algebra Appl., 103:103–118, 1988. (Cited on p. 214.)
[270] Nicholas J. Higham. FORTRAN codes for estimating the one-norm of a real or complex
matrix, with applications to condition estimation (Algorithm 674). ACM Trans. Math.
Software, 14(4):381–396, 1988. (Cited on pp. 69, 226.)
[271] Nicholas J. Higham. Experience with a matrix norm estimator. SIAM J. Sci. Statist.
Comput., 11(4):804–809, 1990. (Cited on p. 69.)
[272] Nicholas J. Higham. Stability of a method for multiplying complex matrices with three
real matrix multiplications. SIAM J. Matrix Anal. Appl., 13(3):681–687, 1992. (Cited
on p. 332.)
[273] Nicholas J. Higham. The matrix sign decomposition and its relation to the polar
decomposition. Linear Algebra Appl., 212/213:3–20, 1994. (Cited on pp. 129, 214,
215, 216.)
[274] Nicholas J. Higham. Stable iterations for the matrix square root. Numer. Algorithms,
15(2):227–242, 1997. (Cited on pp. 108, 167, 363.)
[275] Nicholas J. Higham. Evaluating Padé approximants of the matrix logarithm. SIAM
J. Matrix Anal. Appl., 22(4):1126–1135, 2001. (Cited on p. 275.)
[276] Nicholas J. Higham. Accuracy and Stability of Numerical Algorithms. Second edi-
tion, Society for Industrial and Applied Mathematics, Philadelphia, PA, USA, 2002.
xxx+680 pp. ISBN 0-89871-521-0. (Cited on pp. 69, 75, 82, 102, 104, 120, 122,
134, 143, 163, 165, 166, 210, 211, 223, 226, 247, 248, 250, 261, 265, 309, 324,
325, 332, 335, 361, 362.)
[277] Nicholas J. Higham. J-orthogonal matrices: Properties and generation. SIAM Rev.,
45(3):504–519, 2003. (Cited on p. 315.)
[278] Nicholas J. Higham. The scaling and squaring method for the matrix exponential
revisited. SIAM J. Matrix Anal. Appl., 26(4):1179–1193, 2005. (Cited on pp. 104,
263, 264.)
[279] Nicholas J. Higham and Sheung Hun Cheng. Modifying the inertia of matrices arising
in optimization. Linear Algebra Appl., 275–276:261–279, 1998. (Cited on p. 349.)
[280] Nicholas J. Higham and Hyun-Min Kim. Numerical analysis of a quadratic matrix
equation. IMA J. Numer. Anal., 20(4):499–519, 2000. (Cited on p. 45.)
[281] Nicholas J. Higham and Hyun-Min Kim. Solving a quadratic matrix equation by
Newton’s method with exact line searches. SIAM J. Matrix Anal. Appl., 23(2):303–
316, 2001. (Cited on p. 45.)
Bibliography 395

[282] Nicholas J. Higham, D. Steven Mackey, Niloufer Mackey, and Françoise Tisseur. Com-
puting the polar decomposition and the matrix sign decomposition in matrix groups.
SIAM J. Matrix Anal. Appl., 25(4):1178–1192, 2004. (Cited on p. 315.)
[283] Nicholas J. Higham, D. Steven Mackey, Niloufer Mackey, and Françoise Tisseur. Func-
tions preserving matrix groups and iterations for the matrix square root. SIAM J.
Matrix Anal. Appl., 26(3):849–877, 2005. (Cited on pp. 13, 28, 104, 129, 132, 154,
167, 215, 216, 314, 315, 316, 366.)
[284] Nicholas J. Higham and Pythagoras Papadimitriou. A new parallel algorithm for com-
puting the singular value decomposition. In Proceedings of the Fifth SIAM Conference
on Applied Linear Algebra, John G. Lewis, editor, Society for Industrial and Applied
Mathematics, Philadelphia, PA, USA, 1994, pages 80–84. (Cited on p. 196.)
[285] Nicholas J. Higham and Pythagoras Papadimitriou. A parallel algorithm for computing
the polar decomposition. Parallel Comput., 20(8):1161–1173, 1994. (Cited on pp. 207,
216.)
[286] Nicholas J. Higham and Robert S. Schreiber. Fast polar decomposition of an arbitrary
matrix. SIAM J. Sci. Statist. Comput., 11(4):648–655, 1990. (Cited on pp. 162, 211,
214.)
[287] Nicholas J. Higham and Matthew I. Smith. Computing the matrix cosine. Numer.
Algorithms, 34:13–26, 2003. (Cited on pp. 292, 295, 299.)
[288] Nicholas J. Higham and Françoise Tisseur. A block algorithm for matrix 1-norm
estimation, with an application to 1-norm pseudospectra. SIAM J. Matrix Anal. Appl.,
21(4):1185–1201, 2000. (Cited on p. 67.)
[289] Lester S. Hill. Cryptography in an algebraic alphabet. Amer. Math. Monthly, 36:
306–312, 1929. (Cited on p. 165.)
[290] Einar Hille. On roots and logarithms of elements of a complex Banach algebra. Math.
Annalen, 136(1):46–57, 1958. (Cited on p. 32.)
[291] Marlis Hochbruck and Christian Lubich. On Krylov subspace approximations to the
matrix exponential operator. SIAM J. Numer. Anal., 34(5):1911–1925, 1997. (Cited
on p. 306.)
[292] Marlis Hochbruck, Christian Lubich, and Hubert Selhofer. Exponential integrators for
large systems of differential equations. SIAM J. Sci. Comput., 19(5):1552–1574, 1998.
(Cited on pp. 37, 262.)
[293] John H. Hodges. The matrix equation X 2 − I = 0 over a finite field. Amer. Math.
Monthly, 65(7):518–520, 1958. (Cited on p. 168.)
[294] Roger A. Horn. The Hadamard product. In Matrix Theory and Applications, Charles R.
Johnson, editor, volume 40 of Proceedings of Symposia in Applied Mathematics, Amer-
ican Mathematical Society, Providence, RI, USA, 1990, pages 87–169. (Cited on
p. 1.)
[295] Roger A. Horn and Charles R. Johnson. Matrix Analysis. Cambridge University Press,
1985. xiii+561 pp. ISBN 0-521-30586-1. (Cited on pp. 25, 28, 321, 323, 325, 327,
328, 329, 346.)
[296] Roger A. Horn and Charles R. Johnson. Topics in Matrix Analysis. Cambridge Uni-
versity Press, 1991. viii+607 pp. ISBN 0-521-30587-X. (Cited on pp. xvii, 1, 8, 13,
14, 16, 17, 23, 24, 27, 28, 46, 69, 213, 235, 315, 327, 331, 333, 362.)
[297] Roger A. Horn and Dennis I. Merino. Contragredient equivalence: A canonical form
and some applications. Linear Algebra Appl., 214:43–92, 1995. (Cited on p. 28.)
[298] Roger A. Horn and Gregory G. Piepmeyer. Two applications of the theory of primary
matrix functions. Linear Algebra Appl., 361:99–106, 2003. (Cited on p. 28.)
396 Bibliography

[299] W. D. Hoskins and D. J. Walton. A faster, more stable method for computing the pth
roots of positive definite matrices. Linear Algebra Appl., 26:139–163, 1979. (Cited on
p. 188.)
[300] A. S. Householder and John A. Carpenter. The singular values of involutory and of
idempotent matrices. Numer. Math., 5:234–237, 1963. (Cited on p. 165.)
[301] Alston S. Householder. The Numerical Treatment of a Single Nonlinear Equation.
McGraw-Hill, New York, 1970. viii+216 pp. (Cited on p. 90.)
[302] James Lucien Howland. The sign matrix and the separation of matrix eigenvalues.
Linear Algebra Appl., 49:221–232, 1983. (Cited on pp. 41, 130.)
[303] John H. Hubbard and Beverly H. West. Differential Equations: A Dynamical Systems
Approach. Higher Dimensional Systems. Springer-Verlag, New York, 1995. xiv+601
pp. ISBN 0-387-94377-3. (Cited on p. 267.)
[304] Thomas J. R. Hughes, Itzhak Levit, and James Winget. Element-by-element implicit
algorithms for heat conduction. J. Eng. Mech., 109(2):576–585, 1983. (Cited on
p. 310.)
[305] Bruno Iannazzo. A note on computing the matrix square root. Calcolo, 40:273–283,
2003. (Cited on pp. 143, 166, 167, 169.)
[306] Bruno Iannazzo. On the Newton method for the matrix P th root. SIAM J. Matrix
Anal. Appl., 28(2):503–523, 2006. (Cited on pp. 178, 188.)
[307] Bruno Iannazzo. Numerical Solution of Certain Nonlinear Matrix Equations. PhD
thesis, Università degli studi di Pisa, Pisa, Italy, 2007. 180 pp. (Cited on pp. 95, 104,
130, 167.)
[308] Khakim D. Ikramov. Hamiltonian square roots of skew-Hamiltonian matrices revisited.
Linear Algebra Appl., 325(1-3):101–107, 2001. (Cited on p. 315.)
[309] F. Incertis. A skew-symmetric formulation of the algebraic Riccati equation problem.
IEEE Trans. Automat. Control, AC-29(5):467–470, 1984. (Cited on p. 46.)
[310] Ilse C. F. Ipsen and Dean J. Lee. Determinant approximations. Manuscript, 2003. 13
pp. (Cited on p. 44.)
[311] Eugene Isaacson and Herbert Bishop Keller. Analysis of Numerical Methods. Wiley,
New York, 1966. xv+541 pp. Reprinted by Dover, New York, 1994. ISBN 0-486-68029-
0. (Cited on pp. 90, 333.)
[312] Arieh Iserles. How large is the exponential of a banded matrix? J. New Zealand Maths
Soc., 29:177–192, 2000. (Cited on p. 318.)
[313] Arieh Iserles, Hans Z. Munthe-Kaas, Syvert P. Nørsett, and Antonella Zanna. Lie-
group methods. Acta Numerica, 9:215–365, 2000. (Cited on p. 315.)
[314] Arieh Iserles and Antonella Zanna. Efficient computation of the matrix exponential
by generalized polar decompositions. SIAM J. Numer. Anal., 42(5):2218–2256, 2005.
(Cited on p. 315.)
[315] Robert B. Israel, Jeffrey S. Rosenthal, and Jason Z. Wei. Finding generators for
Markov chains via empirical transition matrices, with applications to credit ratings.
Mathematical Finance, 11(2):245–265, 2001. (Cited on pp. 38, 190.)
[316] Anzelm Iwanik and Ray Shiflett. The root problem for stochastic and doubly stochastic
operators. J. Math. Anal. and Appl., 113(1):93–112, 1986. (Cited on p. 191.)
[317] Drahoslava Janovská and Gerhard Opfer. Computing quaternionic roots by Newton’s
method. Electron. Trans. Numer. Anal., 26:82–102, 2007. (Cited on p. 189.)
[318] Branislav Jansik, Stinne Høst, Poul Jørgensen, Jeppe Olsen, and Trygve Helgaker.
Linear-scaling symmetric square-root decomposition of the overlap matrix. J. Chem.
Phys., 126:12404, 2007. (Cited on p. 42.)
Bibliography 397

[319] A. J. E. M. Janssen and Thomas Strohmer. Characterization and computation of

canonical tight windows for Gabor frames. The Journal of Fourier Analysis and Ap-
plications, 8(1):1–28, 2002. (Cited on p. 168.)
[320] Elias Jarlebring and Tobias Damm. The Lambert W function and the spectrum of some
multidimensional time-delay systems. Automatica, 43(12):2124–2128, 2007. (Cited on
p. 51.)
[321] Charles R. Johnson and Eric Schreiner. The relationship between AB and BA. Amer.
Math. Monthly, 103(7):578–582, 1996. (Cited on p. 28.)
[322] Bo Kågström. Bounds and perturbation bounds for the matrix exponential. BIT, 17:
39–57, 1977. (Cited on p. 263.)
[323] Bo Kågström. Numerical computation of matrix functions. Report UMINF-58.77,
Department of Information Processing, University of Umeå, Sweden, July 1977. 109
pp. (Cited on pp. 104, 231.)
[324] Bo Kågström and Peter Poromaa. Distributed and shared memory block algorithms
for the triangular Sylvester equation with sep−1 estimators. SIAM J. Matrix Anal.
Appl., 13(1):90–101, 1992. (Cited on p. 226.)
[325] Bo Kågström and Axel Ruhe. An algorithm for numerical computation of the Jordan
normal form of a complex matrix. ACM Trans. Math. Software, 6(3):398–419, 1980.
(Cited on p. 231.)
[326] W. Kahan. Conserving confluence curbs ill-condition. Technical Report 6, Computer
Science Department, University of California, Berkeley, August 1972. 54 pp. (Cited
on pp. 88, 106.)
[327] W. Kahan. Branch cuts for complex elementary functions or much ado about nothing’s
sign bit. In The State of the Art in Numerical Analysis, A. Iserles and M. J. D. Powell,
editors, Oxford University Press, 1987, pages 165–211. (Cited on pp. 166, 280.)
[328] W. Kahan. Derivatives in the complex z-plane. Course notes available from http://
www.cs.berkeley.edu/~wkahan/Math185/, September 2006. 16 pp. (Cited on p. 333.)
[329] W. Kahan and Richard J. Fateman. Symbolic computation of divided differences.
ACM SIGSAM Bulletin, 33(2):7–28, 1999. (Cited on pp. 284, 333.)
[330] Thomas Kailath, Ali H. Sayed, and Babak Hassibi. Linear Estimation. Prentice-Hall,
Upper Saddle River, NJ, USA, 2000. xxvi+854 pp. ISBN 0-13-022464-2. (Cited on
p. 166.)
[331] J. D. Kalbfleisch and J. F. Lawless. The analysis of panel data under a Markov
assumption. J. Amer. Statist. Assoc., 80(392):863–871, 1985. (Cited on p. 38.)
[332] G. Kalogeropoulos and Panayiotis Psarrakos. The polar decomposition of block com-
panion matrices. Computers Math. Applic., 50(3):529–537, 2005. (Cited on p. 215.)
[333] Christian Kanzow and Christian Nagel. Semidefinite programs: New search directions,
smoothing-type methods, and numerical results. SIAM J. Optim., 13(1):1–23, 2002.
(Cited on p. 48.)
[334] Alan H. Karp and Peter Markstein. High-precision division and square root. ACM
Trans. Math. Software, 23(4):561–589, 1997. (Cited on p. 188.)
[335] Robert Karplus and Julian Schwinger. A note on saturation in microwave spectroscopy.
Phys. Rev., 73(9):1020–1026, 1948. (Cited on p. 263.)
[336] Aly-Khan Kassam and Lloyd N. Trefethen. Fourth-order time-stepping for stiff PDEs.
SIAM J. Sci. Comput., 26(4):1214–1233, 2005. (Cited on pp. 37, 308.)
[337] Tosio Kato. Perturbation Theory for Linear Operators. Second edition, Springer-Ver-
lag, Berlin, 1976. xxi+619 pp. ISBN 3-540-97588-5. (Cited on p. 28.)
398 Bibliography

[338] A. D. Kennedy. Approximation theory for matrices. Nuclear Physics B (Proc. Suppl.),
128:107–116, 2004. (Cited on p. 130.)
[339] A. D. Kennedy. Fast evaluation of Zolotarev coefficients. In QCD and Numerical
Analysis III, Artan Boriçi, Andreas Frommer, Báalint Joó, Anthony Kennedy, and
Brian Pendleton, editors, volume 47 of Lecture Notes in Computational Science and
Engineering, Springer-Verlag, Berlin, 2005, pages 169–189. (Cited on p. 130.)
[340] Charles S. Kenney and Alan J. Laub. Condition estimates for matrix functions. SIAM
J. Matrix Anal. Appl., 10(2):191–209, 1989. (Cited on pp. 69, 264, 284.)
[341] Charles S. Kenney and Alan J. Laub. Padé error estimates for the logarithm of a
matrix. Internat. J. Control, 50(3):707–730, 1989. (Cited on pp. 275, 283.)
[342] Charles S. Kenney and Alan J. Laub. Polar decomposition and matrix sign function
condition estimates. SIAM J. Sci. Statist. Comput., 12(3):488–504, 1991. (Cited on
pp. 129, 131, 200, 215.)
[343] Charles S. Kenney and Alan J. Laub. Rational iterative methods for the matrix sign
function. SIAM J. Matrix Anal. Appl., 12(2):273–291, 1991. (Cited on pp. 104, 115,
116, 117, 130.)
[344] Charles S. Kenney and Alan J. Laub. On scaling Newton’s method for polar decom-
position and the matrix sign function. SIAM J. Matrix Anal. Appl., 13(3):688–706,
1992. (Cited on pp. 130, 216.)
[345] Charles S. Kenney and Alan J. Laub. A hyperbolic tangent identity and the geometry
of Padé sign function iterations. Numer. Algorithms, 7:111–128, 1994. (Cited on
pp. 117, 130.)
[346] Charles S. Kenney and Alan J. Laub. Small-sample statistical condition estimates for
general matrix functions. SIAM J. Sci. Comput., 15(1):36–61, 1994. (Cited on p. 68.)
[347] Charles S. Kenney and Alan J. Laub. The matrix sign function. IEEE Trans. Automat.
Control, 40(8):1330–1348, 1995. (Cited on pp. 110, 129, 130.)
[348] Charles S. Kenney and Alan J. Laub. A Schur–Fréchet algorithm for computing the
logarithm and exponential of a matrix. SIAM J. Matrix Anal. Appl., 19(3):640–663,
1998. (Cited on pp. 104, 251, 257, 263, 264, 279, 284.)
[349] Charles S. Kenney, Alan J. Laub, and Edmond A. Jonckheere. Positive and negative
solutions of dual Riccati equations by matrix sign function iteration. Systems and
Control Letters, 13:109–116, 1989. (Cited on p. 40.)
[350] Charles S. Kenney, Alan J. Laub, and P. M. Papadopoulos. A Newton-squaring al-
gorithm for computing the negative invariant subspace of a matrix. IEEE Trans.
Automat. Control, 38(8):1284–1289, 1993. (Cited on p. 130.)
[351] Andrzej Kielbasiński, Pawel Zieliński, and Krystyna Ziȩtak. Numerical experiments
with Higham’s scaled method for polar decomposition. Technical report, Institute
of Mathematics and Computer Science, Wroclaw University of Technology, Wroclaw,
Poland, May 2006. 38 pp. (Cited on p. 210.)
[352] Andrzej Kielbasiński and Krystyna Ziȩtak. Numerical behaviour of Higham’s scaled
method for polar decomposition. Numer. Algorithms, 32:105–140, 2003. (Cited on
pp. 208, 210.)
[353] Fuad Kittaneh. On Lipschitz functions of normal operators. Proc. Amer. Math. Soc.,
94(3):416–418, 1985. (Cited on p. 217.)
[354] Fuad Kittaneh. Inequalities for the Schatten p-norm. III. Commun. Math. Phys., 104:
307–310, 1986. (Cited on p. 217.)
[355] Leonard F. Klosinski, Gerald L. Alexanderson, and Loren C. Larson. The fifty-first
William Lowell Putnam mathematical competition. Amer. Math. Monthly, 98(8):719–
727, 1991. (Cited on p. 33.)
Bibliography 399

[356] Leonid A. Knizhnerman. Calculation of functions of unsymmetric matrices using

Arnoldi’s method. U.S.S.R. Comput. Maths. Math. Phys., 31(1):1–9, 1991. (Cited on
p. 309.)
[357] Donald E. Knuth. The Art of Computer Programming, Volume 2, Seminumerical
Algorithms. Third edition, Addison-Wesley, Reading, MA, USA, 1998. xiii+762 pp.
ISBN 0-201-89684-2. (Cited on p. 118.)
[358] Çetin Kaya Koç and Bertan Bakkaloğlu. Halley’s method for the matrix sector func-
tion. IEEE Trans. Automat. Control, 40(5):944–949, 1995. (Cited on p. 49.)
[359] Plamen Koev and Alan Edelman. The efficient evaluation of the hypergeometric func-
tion of a matrix argument. Math. Comp., 75(254):833–846, 2006. (Cited on p. 1.)
[360] S. Koikari. An error analysis of the modified scaling and squaring method. Computers
Math. Applic., 53:1293–1305, 2007. (Cited on p. 262.)
[361] Zdislav Kovarik. Some iterative methods for improving orthonormality. SIAM J.
Numer. Anal., 7(3):386–389, 1970. (Cited on p. 215.)
[362] Alexander Kreinin and Marina Sidelnikova. Regularization algorithms for transition
matrices. Algo Research Quarterly, 4(1/2):23–40, 2001. (Cited on p. 38.)
[363] H. Kreis. Auflösung der Gleichung X n = A. Vierteljschr. Naturforsch. Ges. Zürich,
53:366–376, 1908. (Cited on p. 28.)
[364] V. N. Kublanovskaya. On certain iteration processes for the symmetrisation of a
matrix. U.S.S.R. Computational Math. and Math. Phys., 2(5):859–868, 1963. (Cited
on p. 370.)
[365] J. Kuczyński and H. Woźniakowski. Estimating the largest eigenvalue by the power
and Lanczos algorithms with a random start. SIAM J. Matrix Anal. Appl., 13(4):
1094–1122, 1992. (Cited on p. 69.)
[366] Pentti Laasonen. On the iterative solution of the matrix equation AX 2 − I = 0.
M.T.A.C., 12:109–116, 1958. (Cited on pp. 104, 166, 167.)
[367] Edmond Nicolas Laguerre. Le calcul des systèmes linéaires, extrait d’une lettre adressé
à M. Hermite. In Oeuvres de Laguerre, Ch. Hermite, H. Poincaré, and E. Rouché,
editors, volume 1, Gauthier–Villars, Paris, 1898, pages 221–267. The article is dated
1867 and is “Extrait du Journal de l’École Polytechnique, LXIIe Cahier”. (Cited on
p. 26.)
[368] S. Lakić. On the computation of the matrix k-th root. Z. Angew. Math. Mech., 78(3):
167–172, 1998. (Cited on p. 188.)
[369] Peter Lancaster. Lambda-Matrices and Vibrating Systems. Pergamon Press, Oxford,
1966. xiii+196 pp. Reprinted by Dover, New York, 2002. ISBN 0-486-42546-0. (Cited
on pp. 45, 52.)
[370] Peter Lancaster and Leiba Rodman. Algebraic Riccati Equations. Oxford University
Press, 1995. xvii+480 pp. ISBN 0-19-853795-6. (Cited on pp. 40, 51, 169.)
[371] Peter Lancaster and Miron Tismenetsky. The Theory of Matrices. Second edition,
Academic Press, London, 1985. xv+570 pp. ISBN 0-12-435560-9. (Cited on pp. xvii,
3, 18, 23, 24, 27, 28, 52, 321, 329, 331.)
[372] B. Laszkiewicz and Krystyna Ziȩtak. Approximation of matrices and a family of Gan-
der methods for polar decomposition. BIT, 46(2):345–366, 2006. (Cited on pp. 214,
215, 217, 409.)
[373] Guy Latouche and V. Ramaswami. Introduction to Matrix Analytic Methods in
Stochastic Modeling. Society for Industrial and Applied Mathematics, Philadelphia,
PA, USA, 1999. xiv+334 pp. ISBN 0-89871-425-7. (Cited on p. 45.)
400 Bibliography

[374] Alan J. Laub. Invariant subspace methods for the numerical solution of Riccati equa-
tions. In The Riccati Equation, Sergio Bittanti, Alan J. Laub, and Jan C. Willems,
editors, Springer-Verlag, Berlin, 1991, pages 163–196. (Cited on pp. 51, 53.)
[375] P.-F. Lavallée, A. Malyshev, and M. Sadkane. Spectral portrait of matrices by block
diagonalization. In Numerical Analysis and Its Applications, Lubin Vulkov, Jerzy
Waśniewski, and Plamen Yalamov, editors, volume 1196 of Lecture Notes in Computer
Science, Springer-Verlag, Berlin, 1997, pages 266–273. (Cited on p. 89.)
[376] J. Douglas Lawson. Generalized Runge-Kutta processes for stable systems with large
Lipschitz constants. SIAM J. Numer. Anal., 4(3):372–380, 1967. (Cited on p. 263.)
[377] Jimmie D. Lawson and Yongdo Lim. The geometric mean, matrices, metrics, and
more. Amer. Math. Monthly, 108(9):797–812, 2001. (Cited on p. 52.)
[378] Dean J. Lee and Ilse C. F. Ipsen. Zone determinant expansions for nuclear lattice
simulations. Physical Review C, 68:064003, 2003. (Cited on p. 44.)
[379] R. B. Leipnik. Rapidly convergent recursive solution of quadratic operator equations.
Numer. Math., 17:1–16, 1971. (Cited on p. 215.)
[380] Randall J. LeVeque. Finite Difference Methods for Ordinary and Partial Differential
Equations: Steady-State and Time-Dependent Problems. Society for Industrial and
Applied Mathematics, Philadelphia, PA, USA, 2007. xv+341 pp. ISBN 978-0-898716-
29-0. (Cited on p. 37.)
[381] Jack Levine and H. M. Nahikian. On the construction of involutory matrices. Amer.
Math. Monthly, 69(4):267–272, 1962. (Cited on p. 165.)
[382] Bernard W. Levinger. The square root of a 2 × 2 matrix. Math. Mag., 53(4):222–224,
1980. (Cited on p. 168.)
[383] Malcolm H. Levitt. Spin Dynamics: Basics of Nuclear Magnetic Resonance. Wiley,
Chichester, UK, 2001. xxiv+686 pp. ISBN 0-471-48921-2. (Cited on p. 51.)
[384] Jing Li. An Algorithm for Computing the Matrix Exponential. PhD thesis, Mathematics
Department, University of California, Berkeley, CA, USA, 1988. 77 pp. (Cited on
p. 251.)
[385] Ren-Cang Li. A perturbation bound for the generalized polar decomposition. BIT,
33:304–308, 1993. (Cited on p. 215.)
[386] Ren-Cang Li. New perturbation bounds for the unitary polar factor. SIAM J. Matrix
Anal. Appl., 16(1):327–332, 1995. (Cited on pp. 201, 218.)
[387] Ren-Cang Li. Relative perturbation bounds for the unitary polar factor. BIT, 37(1):
67–75, 1997. (Cited on p. 215.)
[388] Ren-Cang Li. Relative perturbation bounds for positive polar factors of graded matri-
ces. SIAM J. Matrix Anal. Appl., 27(2):424–433, 2005. (Cited on p. 215.)
[389] Wen Li and Weiwei Sun. Perturbation bounds of unitary and subunitary polar factors.
SIAM J. Matrix Anal. Appl., 23(4):1183–1193, 2002. (Cited on pp. 201, 215.)
[390] Pietr Liebl. Einige Bemerkungen zur numerischen Stabilität von Matrizeniterationen.
Aplikace Matematiky, 10(3):249–254, 1965. (Cited on p. 167.)
[391] Yongdo Lim. The matrix golden mean and its applications to Riccati matrix equations.
SIAM J. Matrix Anal. Appl., 29(1):54–66, 2007. (Cited on p. 47.)
[392] Chih-Chang Lin and Earl Zmijewski. A parallel algorithm for computing the eigenval-
ues of an unsymmetric matrix on an SIMD mesh of processors. Report TRCS 91-15,
Department of Computer Science, University of California, Santa Barbara, July 1991.
(Cited on p. 51.)
[393] Lu Lin and Zhong-Yun Liu. On the square root of an H-matrix with positive diagonal
elements. Annals of Operations Research, 103:339–350, 2001. (Cited on p. 167.)
Bibliography 401

[394] Ya Yan Lu. Computing the logarithm of a symmetric positive definite matrix. Appl.
Numer. Math., 26:483–496, 1998. (Cited on p. 284.)
[395] Ya Yan Lu. Exponentials of symmetric matrices through tridiagonal reductions. Linear
Algebra Appl., 279:317–324, 1998. (Cited on p. 260.)
[396] Ya Yan Lu. A Padé approximation method for square roots of symmetric positive
definite matrices. SIAM J. Matrix Anal. Appl., 19(3):833–845, 1998. (Cited on
p. 164.)
[397] Ya Yan Lu. Computing a matrix function for exponential integrators. J. Comput.
Appl. Math., 161:203–216, 2003. (Cited on p. 262.)
[398] Cyrus Colton MacDuffee. Vectors and Matrices. Number 7 in The Carus Mathematical
Monographs. Mathematical Association of America, 1943. xi+203 pp. (Cited on p. 34.)
[399] Cyrus Colton MacDuffee. The Theory of Matrices. Chelsea, New York, 1946. v+110
pp. Corrected reprint of first edition (J. Springer, Berlin, 1933). Also available as
Dover edition, 2004. (Cited on p. 27.)
[400] D. Steven Mackey, Niloufer Mackey, and Françoise Tisseur. Structured factorizations
in scalar product spaces. SIAM J. Matrix Anal. Appl., 27:821–850, 2006. (Cited on
p. 315.)
[401] Arne Magnus and Jan Wynn. On the Padé table of cos z. Proc. Amer. Math. Soc., 47
(2):361–367, 1975. (Cited on pp. 262, 290, 296.)
[402] Jan R. Magnus and Heinz Neudecker. Matrix Differential Calculus with Applications in
Statistics and Econometrics. Revised edition, Wiley, Chichester, UK, 1999. xviii+395
pp. ISBN 0-471-98633-X. (Cited on p. 69.)
[403] Wilhelm Magnus. On the exponential solution of differential equations for a linear
operator. Comm. Pure Appl. Math., 7:649–673, 1954. (Cited on p. 263.)
[404] P. J. Maher. Partially isometric approximation of positive operators. Illinois Journal
of Mathematics, 33(2):227–243, 1989. (Cited on p. 214.)
[405] Jianqin Mao. Optimal orthonormalization of the strapdown matrix by using singular
value decomposition. Computers Math. Applic., 12A(3):353–362, 1986. (Cited on
p. 42.)
[406] Marvin Marcus and Henryk Minc. Some results on doubly stochastic matrices. Proc.
Amer. Math. Soc., 13(4):571–579, 1962. (Cited on p. 191.)
[407] Jerrold E. Marsden and Tudor S. Ratiu. Introduction to Mechanics and Symmetry.
Second edition, Springer-Verlag, New York, 1999. xviii+582 pp. ISBN 0-387-98643-X.
(Cited on p. 266.)
[408] Roy Mathias. Evaluating the Frechet derivative of the matrix exponential. Numer.
Math., 63:213–226, 1992. (Cited on pp. 66, 256.)
[409] Roy Mathias. Approximation of matrix-valued functions. SIAM J. Matrix Anal. Appl.,
14(4):1061–1063, 1993. (Cited on p. 77.)
[410] Roy Mathias. Perturbation bounds for the polar decomposition. SIAM J. Matrix Anal.
Appl., 14(2):588–597, 1993. (Cited on p. 214.)
[411] Roy Mathias. Condition estimation for matrix functions via the Schur decomposition.
SIAM J. Matrix Anal. Appl., 16(2):565–578, 1995. (Cited on pp. 68, 69.)
[412] Roy Mathias. A chain rule for matrix functions and applications. SIAM J. Matrix
Anal. Appl., 17(3):610–620, 1996. (Cited on pp. 13, 69, 355.)
[413] Control Systems Toolbox Documentation. The MathWorks, Inc., Natick, MA, USA.
Online version. (Cited on p. 39.)
[414] MATLAB. The MathWorks, Inc., Natick, MA, USA. http://www.mathworks.com.
(Cited on p. 100.)
402 Bibliography

[415] Neal H. McCoy. On the characteristic roots of matric polynomials. Bull. Amer. Math.
Soc., 42:592–600, 1936. (Cited on p. 29.)
[416] A. McCurdy, K. C. Ng, and B. N. Parlett. Accurate computation of divided differences
of the exponential function. Math. Comp., 43(168):501–528, 1984. (Cited on pp. 250,
264.)
[417] Robert I. McLachlan and G. Reinout W. Quispel. Splitting methods. Acta Numerica,
11:341–434, 2002. (Cited on p. 263.)
[418] Volker Mehrmann. The Autonomous Linear Quadratic Control Problem: Theory and
Numerical Solution, volume 163 of Lecture Notes in Control and Information Sciences.
Springer-Verlag, Berlin, 1991. 177 pp. ISBN 0-340-54170-5. (Cited on pp. 132, 169.)
[419] Volker Mehrmann and Werner Rath. Numerical methods for the computation of an-
alytic singular value decompositions. Electron. Trans. Numer. Anal., 1:72–88, 1993.
(Cited on pp. 43, 214.)
[420] Madan Lal Mehta. Matrix Theory: Selected Topics and Useful Results. Second edi-
tion, Hindustan Publishing Company, Delhi, 1989. xvii+376 pp. ISBN 81-7075-012-1.
(Cited on p. 187.)
[421] Beatrice Meini. The matrix square root from a new functional perspective: Theoretical
results and computational issues. SIAM J. Matrix Anal. Appl., 26(2):362–376, 2004.
(Cited on pp. 142, 160, 166.)
[422] Brian J. Melloy and G. Kemble Bennett. Computing the exponential of an intensity
matrix. J. Comput. Appl. Math., 46:405–413, 1993. (Cited on pp. 105, 263.)
[423] Michael Metcalf and John K. Reid. Fortran 90/95 Explained. Second edition, Oxford
University Press, 1999. xv+341 pp. ISBN 0-19-850558-2. (Cited on p. 1.)
[424] W. H. Metzler. On the roots of matrices. Amer. J. Math., 14(4):326–377, 1892. (Cited
on pp. 26, 28.)
[425] Gérard Meurant. A review on the inverse of symmetric tridiagonal and block tridiag-
onal matrices. SIAM J. Matrix Anal. Appl., 13(3):707–728, 1992. (Cited on p. 316.)
[426] Carl D. Meyer. Matrix Analysis and Applied Linear Algebra. Society for Industrial and
Applied Mathematics, Philadelphia, PA, USA, 2000. xii+718 pp. ISBN 0-89871-454-0.
(Cited on pp. 28, 170, 326.)
[427] Charles A. Micchelli and R. A. Willoughby. On functions which preserve the class of
Stieltjes matrices. Linear Algebra Appl., 23:141–156, 1979. (Cited on p. 315.)
[428] L. M. Milne-Thompson. The Calculus of Finite Differences. Macmillan, London, 1933.
xxiii+558 pp. (Cited on p. 333.)
[429] Henryk Minc. Nonnegative Matrices. Wiley, New York, 1988. xiii+206 pp. ISBN
0-471-83966-3. (Cited on p. 191.)
[430] Borislav V. Minchev. Computing analytic matrix functions for a class of exponential
integrators. Reports in Informatics 278, Department of Informatics, University of
Bergen, Norway, June 2004. 11 pp. (Cited on p. 262.)
[431] Borislav V. Minchev and Will M. Wright. A review of exponential integrators for first
order semi-linear problems. Preprint 2/2005, Norwegian University of Science and
Technology, Trondheim, Norway, 2005. 44 pp. (Cited on pp. 37, 53.)
[432] L. Mirsky. Symmetric gauge functions and unitarily invariant norms. Quart. J. Math.,
11:50–59, 1960. (Cited on p. 328.)
[433] L. Mirsky. An Introduction to Linear Algebra. Oxford University Press, 1961. viii+440
pp. Reprinted by Dover, New York, 1990. ISBN 0-486-66434-1. (Cited on p. 328.)
[434] Maher Moakher. Means and averaging in the group of rotations. SIAM J. Matrix
Anal. Appl., 24(1):1–16, 2002. (Cited on pp. 42, 216.)
Bibliography 403

[435] Maher Moakher. A differential geometric approach to the geometric mean of symmetric
positive-definite matrices. SIAM J. Matrix Anal. Appl., 26(3):735–747, 2005. (Cited
on p. 52.)
[436] Cleve B. Moler. Numerical Computing with MATLAB. Society for Industrial and
Applied Mathematics, Philadelphia, PA, USA, 2004. xi+336 pp. Also available elec-
tronically from www.mathworks.com. ISBN 0-89871-560-1. (Cited on p. 165.)
[437] Cleve B. Moler and Charles F. Van Loan. Nineteen dubious ways to compute the
exponential of a matrix. SIAM Rev., 20(4):801–836, 1978. (Cited on pp. 27, 233,
262, 267.)
[438] Cleve B. Moler and Charles F. Van Loan. Nineteen dubious ways to compute the
exponential of a matrix, twenty-five years later. SIAM Rev., 45(1):3–49, 2003. (Cited
on pp. 27, 231, 233, 237, 241, 243, 246, 262, 263, 265, 267.)
[439] Pierre Montagnier, Christopher C. Paige, and Raymond J. Spiteri. Real Floquet factors
of linear time-periodic systems. Systems and Control Letters, 50:251–262, 2003. (Cited
on p. 286.)
[440] I. Moret and P. Novati. The computation of functions of matrices by truncated Faber
series. Numer. Funct. Anal. Optim., 22(5&6):697–719, 2001. (Cited on p. 309.)
[441] Jean-Michel Muller. Elementary Functions: Algorithms and Implementation.
Birkhäuser, Boston, MA, USA, 1997. xv+204 pp. ISBN 0-8176-3990-X. (Cited
on p. 100.)
[442] David Mumford, Caroline Series, and David Wright. Indra’s Pearls: The Vision of
Felix Klein. Cambridge University Press, 2002. xix+396 pp. ISBN 0-521-35253-3.
(Cited on p. 188.)
[443] H. Z. Munthe-Kaas, G. R. W. Quispel, and A. Zanna. Generalized polar decompo-
sitions on Lie groups with involutive automorphisms. Found. Comput. Math., 1(3):
297–324, 2001. (Cited on p. 315.)
[444] Noël M. Nachtigal, Satish C. Reddy, and Lloyd N. Trefethen. How fast are nonsym-
metric matrix iterations? SIAM J. Matrix Anal. Appl., 13(3):778–795, 1992. (Cited
on p. 75.)
[445] Igor Najfeld and Timothy F. Havel. Derivatives of the matrix exponential and their
computation. Advances in Applied Mathematics, 16:321–375, 1995. (Cited on pp. 51,
69, 263, 264.)
[446] Herbert Neuberger. Exactly massless quarks on the lattice. Phys. Lett. B, 417(1-2):
141–144, 1998. (Cited on p. 43.)
[447] Arnold Neumaier. Introduction to Numerical Analysis. Cambridge University Press,
2001. viii+356 pp. ISBN 0-521-33610-4. (Cited on p. 333.)
[448] Kwok Choi Ng. Contributions to the computation of the matrix exponential. Technical
Report PAM-212, Center for Pure and Applied Mathematics, University of California,
Berkeley, February 1984. 72 pp. PhD thesis. (Cited on pp. 104, 227, 251.)
[449] Jorge Nocedal and Stephen J. Wright. Numerical Optimization. Springer-Verlag, New
York, 1999. xx+636 pp. ISBN 0-387-98793-2. (Cited on p. 199.)
[450] Paolo Novati. A polynomial method based on Fejér points for the computation of
functions of unsymmetric matrices. Appl. Numer. Math., 44:201–224, 2003. (Cited
on p. 309.)
[451] Jeffrey Nunemacher. Which matrices have real logarithms? Math. Mag., 62(2):132–
135, 1989. (Cited on p. 17.)
[452] G. Opitz. Steigungsmatrizen. Z. Angew. Math. Mech., 44:T52–T54, 1964. (Cited on
p. 264.)
404 Bibliography

[453] James M. Ortega and Werner C. Rheinboldt. Iterative Solution of Nonlinear Equations
in Several Variables. Society for Industrial and Applied Mathematics, Philadelphia,
PA, USA, 2000. xxvi+572 pp. Republication of work first published by Academic
Press in 1970. ISBN 0-89871-461-3. (Cited on pp. 69, 167.)
[454] E. E. Osborne. On pre-conditioning of matrices. J. Assoc. Comput. Mach., 7:338–345,
1960. (Cited on p. 100.)
[455] A. M. Ostrowski. Solution of Equations in Euclidean and Banach Spaces. Academic
Press, New York, 1973. xx+412 pp. Third edition of Solution of Equations and Systems
of Equations. ISBN 0-12530260-6. (Cited on pp. 333, 357.)
[456] C. C. Paige. Krylov subspace processes, Krylov subspace methods, and iteration poly-
nomials. In Proceedings of the Cornelius Lanczos International Centenary Conference,
J. David Brown, Moody T. Chu, Donald C. Ellison, and Robert J. Plemmons, editors,
Society for Industrial and Applied Mathematics, Philadelphia, PA, USA, 1994, pages
83–92. (Cited on p. 311.)
[457] Pradeep Pandey, Charles S. Kenney, and Alan J. Laub. A parallel algorithm for the
matrix sign function. Int. J. High Speed Computing, 2(2):181–191, 1990. (Cited on
p. 130.)
[458] Michael James Parks. A Study of Algorithms to Compute the Matrix Exponential. PhD
thesis, Mathematics Department, University of California, Berkeley, CA, USA, 1994.
53 pp. (Cited on p. 263.)
[459] Beresford N. Parlett. Computation of functions of triangular matrices. Memorandum
ERL-M481, Electronics Research Laboratory, College of Engineering, University of
California, Berkeley, November 1974. 18 pp. (Cited on pp. 86, 106.)
[460] Beresford N. Parlett. A recurrence among the elements of functions of triangular
matrices. Linear Algebra Appl., 14:117–121, 1976. (Cited on p. 85.)
[461] Beresford N. Parlett. The Symmetric Eigenvalue Problem. Society for Industrial
and Applied Mathematics, Philadelphia, PA, USA, 1998. xxiv+398 pp. Unabridged,
amended version of book first published by Prentice-Hall in 1980. ISBN 0-89871-402-8.
(Cited on p. 35.)
[462] Beresford N. Parlett and Kwok Choi Ng. Development of an accurate algorithm for
exp(Bt). Technical Report PAM-294, Center for Pure and Applied Mathematics,
University of California, Berkeley, August 1985. 23 pp. Fortran program listings are
given in an appendix with the same report number printed separately. (Cited on
pp. 89, 228, 250.)
[463] Karen Hunger Parshall. Joseph H. M. Wedderburn and the structure theory of algebras.
Archive for History of Exact Sciences, 32(3-4):223–349, 1985. (Cited on p. 26.)
[464] Karen Hunger Parshall. James Joseph Sylvester. Life and Work in Letters. Oxford
University Press, 1998. xv+321 pp. ISBN 0-19-850391-1. (Cited on p. 30.)
[465] Karen Hunger Parshall. James Joseph Sylvester. Jewish Mathematician in a Victorian
World. Johns Hopkins University Press, Baltimore, MD, USA, 2006. xiii+461 pp. ISBN
0-8018-8291-5. (Cited on p. 26.)
[466] Michael S. Paterson and Larry J. Stockmeyer. On the number of nonscalar multiplica-
tions necessary to evaluate polynomials. SIAM J. Comput., 2(1):60–66, 1973. (Cited
on p. 73.)
[467] G. Peano. Intégration par séries des équations différentielles linéaires. Math. Annalen,
32:450–456, 1888. (Cited on p. 26.)
[468] Heinz-Otto Peitgen, Hartmut Jürgens, and Dietmar Saupe. Fractals for the Classroom.
Part Two: Complex Systems and Mandelbrot Set. Springer-Verlag, New York, 1992.
xii+500 pp. ISBN 0-387-97722-8. (Cited on p. 178.)
Bibliography 405

[469] R. Penrose. A generalized inverse for matrices. Proc. Cambridge Philos. Soc., 51(3):
406–413, 1955. (Cited on p. 214.)
[470] P. P. Petrushev and V. A. Popov. Rational Approximation of Real Functions. Cam-
bridge University Press, Cambridge, UK, 1987. ISBN 0-521-33107-2. (Cited on p. 130.)
[471] Bernard Philippe. An algorithm to improve nearly orthonormal sets of vectors on
a vector processor. SIAM J. Alg. Discrete Methods, 8(3):396–403, 1987. (Cited on
p. 188.)
[472] George M. Phillips. Two Millennia of Mathematics: From Archimedes to Gauss.
Springer-Verlag, New York, 2000. xii+223 pp. ISBN 0-387-95022-2. (Cited on pp. 283,
286.)
[473] H. Poincaré. Sur les groupes continus. Trans. Cambridge Phil. Soc., 18:220–255, 1899.
(Cited on p. 26.)
[474] George Pólya and Gabor Szegö. Problems and Theorems in Analysis I. Series. Integral
Calculus. Theory of Functions. Springer-Verlag, New York, 1998. xix+389 pp. Reprint
of the 1978 edition. ISBN 3-540-63640-4. (Cited on p. 343.)
[475] George Pólya and Gabor Szegö. Problems and Theorems in Analysis II. Theory of
Functions. Zeros. Polynomials. Determinants. Number Theory. Geometry. Springer-
Verlag, New York, 1998. xi+392 pp. Reprint of the 1976 edition. ISBN 3-540-63686-2.
(Cited on p. 300.)
[476] Renfrey B. Potts. Symmetric square roots of the finite identity matrix. Utilitas Math-
ematica, 9:73–86, 1976. (Cited on p. 165.)
[477] M. J. D. Powell. Approximation Theory and Methods. Cambridge University Press,
Cambridge, UK, 1981. x+339 pp. ISBN 0-521-29514-9. (Cited on p. 79.)
[478] William H. Press, Saul A. Teukolsky, William T. Vetterling, and Brian P. Flannery.
Numerical Recipes: The Art of Scientific Computing. Third edition, Cambridge Uni-
versity Press, 2007. xxi+1235 pp. ISBN 978-0-521-88068-8. (Cited on p. 274.)
[479] Panayiotis J. Psarrakos. On the mth roots of a complex matrix. Electron. J. Linear
Algebra, 9:32–41, 2002. (Cited on p. 174.)
[480] Péter Pulay. An iterative method for the determination of the square root of a positive
definite matrix. Z. Angew. Math. Mech., 46:151, 1966. (Cited on pp. 167, 171.)
[481] Norman J. Pullman. Matrix Theory and its Applications: Selected Topics. Marcel
Dekker, New York, 1976. vi+240 pp. ISBN 0-8247-6420-X. (Cited on pp. 28, 29.)
[482] W. Pusz and S. L. Woronowicz. Functional calculus for sesquilinear forms and the
purification map. Reports on Mathematical Physics, 8(2):159–170, 1975. (Cited on
p. 52.)
[483] Heydar Radjavi and Peter Rosenthal. Simultaneous Triangularization. Springer-Ver-
lag, New York, 2000. xii+318 pp. ISBN 0-387-98466-6. (Cited on p. 29.)
[484] C. R. Rao. Matrix approximations and reduction of dimensionality in multivariate sta-
tistical analysis. In Multivariate Analysis—V, P. R. Krishnaiah, editor, North Holland,
Amsterdam, 1980, pages 3–22. (Cited on p. 214.)
[485] Lothar Reichel. The application of Leja points to Richardson iteration and polynomial
preconditioning. Linear Algebra Appl., 154/156:389–414, 1991. (Cited on p. 75.)
[486] Matthias W. Reinsch. A simple expression for the terms in the Baker–Campbell–
Hausdorff series. J. Math. Phys., 41(4):2434–2442, 2000. (Cited on p. 263.)
[487] John R. Rice. A theory of condition. SIAM J. Numer. Anal., 3(2):287–310, 1966.
(Cited on p. 69.)
[488] Norman M. Rice. On nth roots of positive operators. Amer. Math. Monthly, 89(5):
313–314, 1982. (Cited on p. 189.)
406 Bibliography

[489] A. N. Richmond. Expansions for the exponential of a sum of matrices. In Applications

of Matrix Theory, M. J. C. Gover and S. Barnett, editors, Oxford University Press,
1989, pages 283–289. (Cited on p. 236.)
[490] Hans Richter. Zum Logarithmus einer Matrix. Archiv der Mathematik, 2(5):360–363,
1949. (Cited on p. 283.)
[491] Hans Richter. Über Matrixfunktionen. Math. Ann., 22(1):16–34, 1950. (Cited on
p. 28.)
[492] Frigyes Riesz and Béla Sz.-Nagy. Functional Analysis. Second edition, Blackie & Son,
London and Glasgow, 1956. xii+468 pp. (Cited on p. 167.)
[493] R. F. Rinehart. The equivalence of definitions of a matric function. Amer. Math.
Monthly, 62:395–414, 1955. (Cited on pp. 27, 34.)
[494] R. F. Rinehart. The derivative of a matric function. Proc. Amer. Math. Soc., 7:2–5,
1956. (Cited on p. 70.)
[495] R. F. Rinehart. Elements of a theory of intrinsic functions on algebras. Duke Math.
J., 27:1–19, 1960. (Cited on p. 28.)
[496] J. D. Roberts. Linear model reduction and solution of the algebraic Riccati equation
by use of the sign function. Internat. J. Control, 32(4):677–687, 1980. First issued as
report CUED/B-Control/TR13, Department of Engineering, University of Cambridge,
1971. (Cited on pp. 39, 49, 129, 132.)
[497] Gian-Carlo Rota. Indiscrete Thoughts. Birkhäuser, Boston, 1997. xxii+280 pp. Edited
by Fabrizio Palombi. ISBN 0-8176-3866-0. (Cited on p. 53.)
[498] Y. Saad. Analysis of some Krylov subspace approximations to the matrix exponential
operator. SIAM J. Numer. Anal., 29(1):209–228, 1992. (Cited on pp. 309, 311.)
[499] Youcef Saad. Numerical Methods for Large Eigenvalue Problems. Manchester Univer-
sity Press, Manchester, and Halsted Press, New York, 1992. 346 pp. ISBN 0-7190-
3386-1. (Cited on p. 306.)
[500] Yousef Saad. Iterative Methods for Sparse Linear Systems. Second edition, Society
for Industrial and Applied Mathematics, Philadelphia, PA, USA, 2003. xviii+528 pp.
ISBN 0-89871-534-2. (Cited on p. 309.)
[501] E. B. Saff. On the degree of best rational approximation to the exponential function.
J. Approximation Theory, 9:97–101, 1973. (Cited on p. 241.)
[502] Martin Schechter. Principles of Functional Analysis. Academic Press, New York, 1971.
xix+383 pp. ISBN 0-12-622750-0. (Cited on p. 167.)
[503] Thomas Schmelzer and Lloyd N. Trefethen. Evaluating matrix functions for exponen-
tial integrators via Carathéodory-Fejér approximation and contour integrals. Report
Number 06/20, Numerical Analysis Group, Oxford University Computing Laboratory,
Oxford, UK, 2006. 19 pp. (Cited on pp. 37, 308.)
[504] Bernhard A. Schmitt. An algebraic approximation for the matrix exponential in singu-
larly perturbed boundary value problems. SIAM J. Numer. Anal., 27(1):51–66, 1990.
(Cited on p. 48.)
[505] Christoph Schmoeger. On the operator equation eA = eB . Linear Algebra Appl., 359:
169–179, 2003. (Cited on p. 350.)
[506] Daniel Scholz and Michael Weyrauch. A note on the Zassenhaus product formula. J.
Math. Phys., 47:033505.1–033505.7, 2006. (Cited on p. 263.)
[507] Peter H. Schönemann. A generalized solution of the orthogonal Procrustes problem.
Psychometrika, 31(1):1–10, 1966. (Cited on p. 214.)
[508] Robert S. Schreiber and Beresford N. Parlett. Block reflectors: Theory and computa-
tion. SIAM J. Numer. Anal., 25(1):189–205, 1988. (Cited on p. 213.)
Bibliography 407

[509] Ernst Schröder. Ueber unendliche viele Algorithmen zur Auflösung der Gleichungen.
Math. Annalen, 2:317–365, 1870. (Cited on pp. 130, 407.)
[510] Ernst Schröder. On infinitely many algorithms for solving equations. Technical Report
TR-92-121, Department of Computer Science, University of Maryland, College Park,
MD, USA, November 1992. 57 pp. Translation of [509] by G. W. Stewart. (Cited on
p. 130.)
[511] Manfred Schroeder. Fractals, Chaos, Power Laws: Minutes from an Infinite Paradise.
W. H. Freeman, New York, 1991. xviii+429 pp. ISBN 0-7167-2136-8. (Cited on
pp. 131, 178.)
[512] Günther Schulz. Iterative Berechnung der reziproken Matrix. Z. Angew. Math. Mech.,
13:57–59, 1933. (Cited on pp. 114, 181.)
[513] Hans Schwerdtfeger. Les Fonctions de Matrices. I. Les Fonctions Univalentes. Number
649 in Actualités Scientifiques et Industrielles. Hermann, Paris, France, 1938. 58 pp.
(Cited on pp. 26, 30, 51.)
[514] Steven M. Serbin. Rational approximations of trigonometric matrices with application
to second-order systems of differential equations. Appl. Math. Comput., 5(1):75–92,
1979. (Cited on p. 287.)
[515] Steven M. Serbin and Sybil A. Blalock. An algorithm for computing the matrix cosine.
SIAM J. Sci. Statist. Comput., 1(2):198–204, 1980. (Cited on p. 299.)
[516] Lawrence F. Shampine and Mark W. Reichelt. The MATLAB ODE suite. SIAM J.
Sci. Comput., 18(1):1–22, 1997. (Cited on p. 165.)
[517] Wyatt D. Sharpa and Edward J. Allen. Stochastic neutron transport equations for
rod and plane geometries. Annals of Nuclear Energy, 27(2):99–116, 2000. (Cited on
p. 167.)
[518] N. Sherif. On the computation of a matrix inverse square root. Computing, 46:295–305,
1991. (Cited on p. 169.)
[519] L. S. Shieh, Y. T. Tsay, and C. T. Wang. Matrix sector functions and their applications
to system theory. IEE Proc., 131(5):171–181, 1984. (Cited on p. 48.)
[520] Ken Shoemake and Tom Duff. Matrix animation and polar decomposition. In Pro-
ceedings of the Conference on Graphics Interface ’92, Morgan Kaufmann Publishers
Inc., San Francisco, CA, USA, 1992, pages 258–264. (Cited on p. 42.)
[521] Avram Sidi. Practical Extrapolation Methods; Theory and Applications. Cambridge
University Press, Cambridge, UK, 2003. xxii+519 pp. ISBN 0-521-66159-5. (Cited
on p. 104.)
[522] Roger B. Sidje. Expokit: A software package for computing matrix exponentials. ACM
Trans. Math. Software, 24(1):130–156, 1998. (Cited on pp. 262, 263, 264.)
[523] Roger B. Sidje, Kevin Burrage, and Shev MacNamara. Inexact uniformization method
for computing transient distributions of Markov chains. SIAM J. Sci. Comput., 29(6):
2562–2580, 2007. (Cited on p. 260.)
[524] Roger B. Sidje and William J. Stewart. A numerical study of large sparse matrix
exponentials arising in Markov chains. Computational Statistics & Data Analysis, 29:
345–368, 1999. (Cited on p. 260.)
[525] Burton Singer and Seymour Spilerman. The representation of social processes by
Markov models. Amer. J. Sociology, 82(1):1–54, 1976. (Cited on p. 38.)
[526] Abraham Sinkov. Elementary Cryptanalysis: A Mathematical Approach. Mathematical
Association of America, Washington, D.C., 1966. ix+222 pp. ISBN 0-88385-622-0.
(Cited on p. 171.)
408 Bibliography

[527] Bård Skaflestad and Will M. Wright. The scaling and modified squaring method for
matrix functions related to the exponential. Preprint, Norwegian University of Science
and Technology, Trondheim, Norway, 2006. 17 pp. (Cited on p. 262.)
[528] David M. Smith. Algorithm 693: A FORTRAN package for floating-point multiple-
precision arithmetic. ACM Trans. Math. Software, 17(2):273–283, 1991. (Cited on
p. 286.)
[529] Matthew I. Smith. Numerical Computation of Matrix Functions. PhD thesis, Uni-
versity of Manchester, Manchester, England, September 2002. 157 pp. (Cited on
p. 370.)
[530] Matthew I. Smith. A Schur algorithm for computing matrix pth roots. SIAM J. Matrix
Anal. Appl., 24(4):971–989, 2003. (Cited on pp. 174, 176, 187, 188, 190.)
[531] R. A. Smith. Infinite product expansions for matrix n-th roots. J. Austral. Math. Soc.,
8:242–249, 1968. (Cited on p. 188.)
[532] Alicja Smoktunowicz, Jesse L. Barlow, and Julien Langou. A note on the error analysis
of classical Gram–Schmidt. Numer. Math., 105:299–313, 2006. (Cited on p. 309.)
[533] Inge Söderkvist. Perturbation analysis of the orthogonal Procrustes problem. BIT, 33:
687–694, 1993. (Cited on p. 201.)
[534] Mark Sofroniou and Giulia Spaletta. Solving orthogonal matrix differential systems
in Mathematica. In Computational Science—ICCS 2002 Proceedings, Part III, Peter
M. A. Sloot, C. J. Kenneth Tan, Jack J. Dongarra, and Alfons G. Hoekstra, editors,
volume 2002 of Lecture Notes in Computer Science, Springer-Verlag, Berlin, 2002,
pages 496–505. (Cited on pp. 42, 215.)
[535] Irene A. Stegun and Milton Abramowitz. Pitfalls in computation. J. Soc. Indust. Appl.
Math., 4(4):207–219, 1956. (Cited on p. 74.)
[536] G. W. Stewart. Error and perturbation bounds for subspaces associated with certain
eigenvalue problems. SIAM Rev., 15(4):727–764, 1973. (Cited on p. 231.)
[537] G. W. Stewart. Matrix Algorithms. Volume I: Basic Decompositions. Society for
Industrial and Applied Mathematics, Philadelphia, PA, USA, 1998. xx+458 pp. ISBN
0-89871-414-1. (Cited on p. 357.)
[538] G. W. Stewart. Matrix Algorithms. Volume II: Eigensystems. Society for Industrial
and Applied Mathematics, Philadelphia, PA, USA, 2001. xix+469 pp. ISBN 0-89871-
503-2. (Cited on pp. 35, 69, 309, 323, 325.)
[539] G. W. Stewart and Ji-guang Sun. Matrix Perturbation Theory. Academic Press,
London, 1990. xv+365 pp. ISBN 0-12-670230-6. (Cited on p. 327.)
[540] William J. Stewart. Introduction to the Numerical Solution of Markov Chains. Prince-
ton University Press, Princeton, NJ, USA, 1994. xix+539 pp. ISBN 0-691-03699-3.
(Cited on p. 260.)
[541] Eberhard Stickel. On the Fréchet derivative of matrix functions. Linear Algebra Appl.,
91:83–88, 1987. (Cited on p. 70.)
[542] Josef Stoer and R. Bulirsch. Introduction to Numerical Analysis. Third edition,
Springer-Verlag, New York, 2002. xv+744 pp. ISBN 0-387-95452-X. (Cited on
pp. 28, 154, 333.)
[543] David R. Stoutemyer. Crimes and misdemeanors in the computer algebra trade. No-
tices Amer. Math. Soc., 38(7):778–785, 1991. (Cited on p. 286.)
[544] Gilbert Strang. On the construction and comparison of difference schemes. SIAM J.
Numer. Anal., 5(3):506–517, 1968. (Cited on p. 263.)
[545] Torsten Ström. Minimization of norms and logarithmic norms by diagonal similarities.
Computing, 10:1–7, 1972. (Cited on p. 105.)
Bibliography 409

[546] Torsten Ström. On logarithmic norms. SIAM J. Numer. Anal., 12(5):741–753, 1975.
(Cited on p. 237.)
[547] Ji-guang Sun. A note on backward perturbations for the Hermitian eigenvalue problem.
BIT, 35:385–393, 1995. (Cited on p. 214.)
[548] Ji-guang Sun. Perturbation analysis of the matrix sign function. Linear Algebra Appl.,
250:177–206, 1997. (Cited on p. 129.)
[549] Ji-guang Sun and C.-H. Chen. Generalized polar decomposition. Math. Numer. Sinica,
11:262–273, 1989. In Chinese. Cited in [372]. (Cited on p. 214.)
[550] Xiaobai Sun and Enrique S. Quintana-Ortı́. The generalized Newton iteration for the
matrix sign function. SIAM J. Sci. Comput., 24(2):669–683, 2002. (Cited on p. 130.)
[551] Xiaobai Sun and Enrique S. Quintana-Ortı́. Spectral division methods for block gen-
eralized Schur decompositions. Math. Comp., 73(248):1827–1847, 2004. (Cited on
p. 49.)
[552] Masuo Suzuki. Generalized Trotter’s formula and systematic approximants of ex-
ponential operators and inner derivations with applications to many-body problems.
Commun. Math. Phys., 51(2):183–190, 1976. (Cited on pp. 262, 263.)
[553] J. J. Sylvester. Additions to the articles, “On a New Class of Theorems,” and “On
Pascal’s Theorem”. Philosophical Magazine, 37:363–370, 1850. Reprinted in [558,
pp. 1451–151]. (Cited on p. 26.)
[554] J. J. Sylvester. Note on the theory of simultaneous linear differential or difference
equations with constant coefficients. Amer. J. Math., 4(1):321–326, 1881. Reprinted
in [559, pp. 551-556]. (Cited on pp. 188, 191.)
[555] J. J. Sylvester. Sur les puissances et les racines de substitutions linéaires. Comptes
Rendus de l’Académie des Sciences, 94:55–59, 1882. Reprinted in [559, pp. 562–564].
(Cited on pp. 187, 188.)
[556] J. J. Sylvester. Sur les racines des matrices unitaires. Comptes Rendus de l’Académie
des Sciences, 94:396–399, 1882. Reprinted in [559, pp. 565–567]. (Cited on p. 168.)
[557] J. J. Sylvester. On the equation to the secular inequalities in the planetary theory.
Philosophical Magazine, 16:267–269, 1883. Reprinted in [560, pp. 110–111]. (Cited on
pp. 26, 34, 187.)
[558] The Collected Mathematical Papers of James Joseph Sylvester, volume 1 (1837–1853).
Cambridge University Press, 1904. xii+650 pp. (Cited on p. 409.)
[559] The Collected Mathematical Papers of James Joseph Sylvester, volume III (1870–1883).
Chelsea, New York, 1973. xv+697 pp. ISBN 0-8284-0253-1. (Cited on p. 409.)
[560] The Collected Mathematical Papers of James Joseph Sylvester, volume IV (1882–1897).
Chelsea, New York, 1973. xxxvii+756 pp. ISBN 0-8284-0253-1. (Cited on p. 409.)
[561] Henry Taber. On the theory of matrices. Amer. J. Math., 12(4):337–396, 1890. (Cited
on p. 28.)
[562] Ping Tak Peter Tang. Table-driven implementation of the expm1 function in IEEE
floating-point arithmetic. ACM Trans. Math. Software, 18(2):211–222, 1992. (Cited
on p. 261.)
[563] Pham Dinh Tao. Convergence of a subgradient method for computing the bound norm
of matrices. Linear Algebra Appl., 62:163–182, 1984. In French. (Cited on p. 69.)
[564] Olga Taussky. Commutativity in finite matrices. Amer. Math. Monthly, 64(4):229–235,
1957. (Cited on p. 29.)
[565] Olga Taussky. How I became a torchbearer for matrix theory. Amer. Math. Monthly,
95(9):801–812, 1988. (Cited on p. 29.)
410 Bibliography

[566] R. C. Thompson. On the matrices AB and BA. Linear Algebra Appl., 1:43–58, 1968.
(Cited on p. 28.)
[567] C. Thron, S. J. Dong, K. F. Liu, and H. P. Ying. Padé-Z2 estimator of determinants.
Physical Review D, 57(3):1642–1653, 1997. (Cited on p. 44.)
[568] Françoise Tisseur. Parallel implementation of the Yau and Lu method for eigenvalue
computation. International Journal of Supercomputer Applications and High Perfor-
mance Computing, 11(3):197–204, 1997. (Cited on p. 300.)
[569] Françoise Tisseur. Newton’s method in floating point arithmetic and iterative re-
finement of generalized eigenvalue problems. SIAM J. Matrix Anal. Appl., 22(4):
1038–1057, 2001. (Cited on p. 166.)
[570] Françoise Tisseur and Karl Meerbergen. The quadratic eigenvalue problem. SIAM
Rev., 43(2):235–286, 2001. (Cited on p. 45.)
[571] L. N. Trefethen and J. A. C. Weideman. The fast trapezoid rule in scientific computing.
Paper in preparation, 2007. (Cited on p. 308.)
[572] Lloyd N. Trefethen and David Bau III. Numerical Linear Algebra. Society for Industrial
and Applied Mathematics, Philadelphia, PA, USA, 1997. xii+361 pp. ISBN 0-89871-
361-7. (Cited on pp. 325, 335.)
[573] Lloyd N. Trefethen and Mark Embree. Spectra and Pseudospectra: The Behavior of
Nonnormal Matrices and Operators. Princeton University Press, Princeton, NJ, USA,
2005. xvii+606 pp. ISBN 0-691-11946-5. (Cited on pp. 47, 105, 263.)
[574] Lloyd N. Trefethen, J. A. C. Weideman, and Thomas Schmelzer. Talbot quadratures
and rational approximations. BIT, 46(3):653–670, 2006. (Cited on pp. 259, 308.)
[575] H. F. Trotter. On the product of semi-groups of operators. Proc. Amer. Math. Soc.,
10(4):545–551, 1959. (Cited on p. 263.)
[576] J. S. H. Tsai, L. S. Shieh, and R. E. Yates. Fast and stable algorithms for computing
the principal nth root of a complex matrix and the matrix sector function. Comput.
Math. Applic., 15(11):903–913, 1988. (Cited on p. 129.)
[577] H. W. Turnbull. The matrix square and cube roots of unity. J. London Math. Soc., 2
(8):242–244, 1927. (Cited on p. 188.)
[578] H. W. Turnbull. The Theory of Determinants, Matrices, and Invariants. Blackie,
London and Glasgow, 1929. xvi+338 pp. (Cited on p. 188.)
[579] H. W. Turnbull and A. C. Aitken. An Introduction to the Theory of Canonical Matrices.
Blackie, London and Glasgow, 1932. xiii+200 pp. Reprinted with appendix, 1952.
(Cited on pp. 27, 29, 52.)
[580] G. M. Tuynman. The derivation of the exponential map of matrices. Amer. Math.
Monthly, 102(9):818–820, 1995. (Cited on p. 263.)
[581] Frank Uhlig. Explicit polar decomposition and a near-characteristic polynomial: The
2 × 2 case. Linear Algebra Appl., 38:239–249, 1981. (Cited on p. 216.)
[582] R. Vaidyanathaswamy. Integer-roots of the unit matrix. J. London Math. Soc., 3(12):
121–124, 1928. (Cited on p. 189.)
[583] R. Vaidyanathaswamy. On the possible periods of integer matrices. J. London Math.
Soc., 3(12):268–272, 1928. (Cited on p. 189.)
[584] P. Van Den Driessche and H. K. Wimmer. Explicit polar decomposition of companion
matrices. Electron. J. Linear Algebra, 1:64–69, 1996. (Cited on p. 215.)
[585] J. van den Eshof, A. Frommer, Th. Lippert, K. Schilling, and H. A. Van der Vorst.
Numerical methods for the QCD overlap operator. I. Sign-function and error bounds.
Computer Physics Communications, 146:203–224, 2002. (Cited on pp. 43, 130.)
Bibliography 411

[586] Jasper van den Eshof. Nested Iteration methods for Nonlinear Matrix Problems. PhD
thesis, Utrecht University, Utrecht, Netherlands, September 2003. vii+183 pp. (Cited
on p. 130.)
[587] Henk A. Van der Vorst. An iterative solution method for solving f (A)x = b, using
Krylov subspace information obtained for the symmetric positive definite matrix A. J.
Comput. Appl. Math., 18:249–263, 1987. (Cited on p. 309.)
[588] Henk A. Van der Vorst. Solution of f (A)x = b with projection methods for the matrix
A. In Numerical Challenges in Lattice Quantum Chromodynamics, Andreas Frommer,
Thomas Lippert, Björn Medeke, and Klaus Schilling, editors, volume 15 of Lecture
Notes in Computational Science and Engineering, Springer-Verlag, Berlin, 2000, pages
18–28. (Cited on p. 309.)
[589] Henk A. Van der Vorst. Iterative Krylov Methods for Large Linear Systems. Cambridge
University Press, 2003. xiii+221 pp. ISBN 0-521-81828-1. (Cited on p. 309.)
[590] Jos L. M. van Dorsselaer, Michiel E. Hochstenbach, and Henk A. Van der Vorst.
Computing probabilistic bounds for extreme eigenvalues of symmetric matrices with
the Lanczos method. SIAM J. Matrix Anal. Appl., 22(3):837–852, 2000. (Cited on
p. 69.)
[591] J. L. van Hemmen and T. Ando. An inequality for trace ideals. Commun. Math. Phys.,
76:143–148, 1980. (Cited on p. 135.)
[592] Charles F. Van Loan. A study of the matrix exponential. Numerical Analysis Report
No. 10, University of Manchester, Manchester, UK, August 1975. Reissued as MIMS
EPrint 2006.397, Manchester Institute for Mathematical Sciences, The University of
Manchester, UK, November 2006. (Cited on pp. 84, 105, 262.)
[593] Charles F. Van Loan. On the limitation and application of Padé approximation to the
matrix exponential. In Padé and Rational Approximation: Theory and Applications,
E. B. Saff and R. S. Varga, editors, Academic Press, New York, 1977, pages 439–448.
(Cited on p. 263.)
[594] Charles F. Van Loan. The sensitivity of the matrix exponential. SIAM J. Numer.
Anal., 14(6):971–981, 1977. (Cited on pp. 238, 263.)
[595] Charles F. Van Loan. Computing integrals involving the matrix exponential. IEEE
Trans. Automat. Control, AC-23(3):395–404, 1978. (Cited on p. 264.)
[596] Charles F. Van Loan. A note on the evaluation of matrix polynomials. IEEE Trans.
Automat. Control, AC-24(2):320–321, 1979. (Cited on p. 74.)
[597] R. Vandebril, M. Van Barel, G. H. Golub, and N. Mastronardi. A bibliography on
semiseparable matrices. Calcolo, 42:249–70, 2005. (Cited on p. 316.)
[598] V. S. Varadarajan. Lie Groups, Lie Algebras, and Their Representations. Prentice-
Hall, Englewood Cliffs, NJ, USA, 1974. xiii+430 pp. ISBN 0-13-535732-2. (Cited on
p. 262.)
[599] J. M. Varah. On the separation of two matrices. SIAM J. Numer. Anal., 16(2):216–222,
1979. (Cited on p. 231.)
[600] Richard S. Varga. Matrix Iterative Analysis. Second edition, Springer-Verlag, Berlin,
2000. x+358 pp. ISBN 3-540-66321-5. (Cited on pp. 242, 260, 264.)
[601] R. Vertechy and V. Parenti-Castelli. Real-time direct position analysis of parallel
spherical wrists by using extra sensors. Journal of Mechanical Design, 128:288–294,
2006. (Cited on pp. 42, 208.)
[602] Cornelis Visser. Note on linear operators. Proc. Kon. Akad. Wet. Amsterdam, 40(3):
270–272, 1937. (Cited on p. 167.)
[603] John von Neumann. Über Adjungierte Funktionaloperatoren. Ann. of Math. (2), 33
(2):294–310, 1923. (Cited on p. 214.)
412 Bibliography

[604] John von Neumann. Über die analytischen Eigenschaften von Gruppen linearer Trans-
formationen und ihrer Darstellungen. Math. Zeit., 30:3–42, 1929. (Cited on p. 283.)
[605] Grace Wahba. Problem 65-1, a least squares estimate of satellite attitude. SIAM Rev.,
7(3):409, 1965. Solutions in 8(3):384–386, 1966. (Cited on p. 214.)
[606] Robert C. Ward. Numerical computation of the matrix exponential with accuracy
estimate. SIAM J. Numer. Anal., 14(4):600–610, 1977. (Cited on pp. 104, 246, 263,
264.)
[607] David S. Watkins. Fundamentals of Matrix Computations. Second edition, Wiley, New
York, 2002. xiii+618 pp. ISBN 0-471-21394-2. (Cited on pp. 69, 323, 325, 335.)
[608] David S. Watkins. The Matrix Eigenvalue Problem: GR and Krylov Subspace Methods.
Society for Industrial and Applied Mathematics, Philadelphia, PA, USA, 2007. x+442
pp. ISBN 978-0-898716-41-2. (Cited on pp. 309, 323.)
[609] G. A. Watson. Approximation Theory and Numerical Methods. Wiley, Chichester, UK,
1980. x+229 pp. ISBN 0-471-27706-1. (Cited on p. 79.)
[610] Frederick V. Waugh and Martin E. Abel. On fractional powers of a matrix. J. Amer.
Statist. Assoc., 62:1018–1021, 1967. (Cited on pp. 38, 167.)
[611] J. H. M. Wedderburn. Lectures on Matrices, volume 17 of American Mathematical
Society Colloquium Publications. American Mathematical Society, Providence, RI,
USA, 1934. vii+205 pp. (Cited on pp. 27, 28.)
[612] G. H. Weiss and A. A. Maradudin. The Baker-Hausdorff formula and a problem in
crystal physics. J. Math. Phys., 3(4):771–777, 1962. (Cited on p. 263.)
[613] Edgar M. E. Wermuth. Two remarks on matrix exponentials. Linear Algebra Appl.,
117:127–132, 1989. (Cited on p. 262.)
[614] Edouard Weyr. Note sur la théorie de quantités complexes formées avec n unités
principales. Bull. Sci. Math. II, 11:205–215, 1887. (Cited on pp. 26, 104.)
[615] R. M. Wilcox. Exponential operators and parameter differentiation in quantum
physics. J. Math. Phys., 8(4):962–982, 1967. (Cited on p. 263.)
[616] J. H. Wilkinson. The Algebraic Eigenvalue Problem. Oxford University Press, 1965.
xviii+662 pp. ISBN 0-19-853403-5 (hardback), 0-19-853418-3 (paperback). (Cited on
p. 69.)
[617] Arthur Wouk. Integral representation of the logarithm of matrices and operators. J.
Math. Anal. and Appl., 11:131–138, 1965. (Cited on p. 283.)
[618] Pei Yuan Wu. Approximation by partial isometries. Proc. Edinburgh Math. Soc., 29:
255–261, 1986. (Cited on p. 214.)
[619] Shing-Tung Yau and Ya Yan Lu. Reducing the symmetric matrix eigenvalue problem
to matrix multiplications. SIAM J. Sci. Comput., 14(1):121–136, 1993. (Cited on
p. 300.)
[620] N. J. Young. A bound for norms of functions of matrices. Linear Algebra Appl., 37:
181–186, 1981. (Cited on p. 105.)
[621] V. Zakian. Properties of IM N and JM N approximants and applications to numerical
inversion of Laplace transforms and initial value problems. J. Inst. Maths. Applics.,
50:191–222, 1975. (Cited on p. 263.)
[622] Hongyuan Zha and Zhenyue Zhang. Fast parallelizable methods for the Hermitian
eigenvalue problem. Technical Report CSE-96-041, Department of Computer Science
and Engineering, Pennsylvania State University, University Park, PA, May 1996. 19
pp. (Cited on p. 216.)
[623] Xingzhi Zhan. Matrix Inequalities, volume 1790 of Lecture Notes in Mathematics.
Springer-Verlag, Berlin, 2002. ISBN 3-540-43798-3. (Cited on p. 330.)
Bibliography 413

[624] Fuzhen Zhang, editor. The Schur Complement and Its Applications. Springer-Verlag,
New York, 2005. xvi+295 pp. ISBN 0-387-24271-6. (Cited on p. 329.)
[625] Pawel Zieliński and Krystyna Ziȩtak. The polar decomposition—properties, applica-
tions and algorithms. Applied Mathematics, Ann. Pol. Math. Soc., 38:23–49, 1995.
(Cited on p. 213.)
Index

Sherlock Holmes sat moodily at one side of the fireplace

cross-indexing his records of crime.
— ARTHUR CONAN DOYLE, The Five Orange Pips (1891)

A suffix “t” after a page number denotes a table, “n” a footnote, and “q” a quotation.
Mathematical symbols and Greek letters are indexed as if they were spelled out. The solution
to a problem is not usually indexed if the problem itself is indexed under the same term.

A-circulant matrix, 187 binomial expansion for matrix square root,

AB and BA, eigenvalues of, 21 154
absolute norm, 327 binomial iteration for matrix square root,
absolute value, of matrix, 196, 215 154–157
adjoint, 66, 314 block diagonal matrix, 324
Aitken extrapolation, 154 block diagonalization
algebra finding well conditioned is NP-hard,
Jordan, 314 226
Lie, 314 for evaluating matrix function, 89
algebraic Riccati equation, 40 block Householder transformation, 213
XAX = B, 44, 162, 170 block Parlett recurrence, 86–88, 225–226
analytic polar decomposition, 214 rounding errors in, 225
argument reduction, 99–101, 104, 170, 251 block reflector, 213
Arnoldi approximation of f (A)b, 304–306 bounds for norm of matrix function, 102–
Arnoldi process, 302–304, 318 104
restarted, 305 Bregman divergence, 50
ARPREC, 188 Briggs’ making of logarithm tables, 275–
ascent sequence, 16, 174 276, 286 q
automorphism group, 314
canonical polar decomposition, 194–196,
average eye, 50
213, 215
Cauchy integral formula, 8, 9, 22, 30, 102,
Baker–Campbell–Hausdorff formula, 235, 307, 345, 347, 355, 358
271 history of, 26
balancing, 100–101 Cayley–Hamilton theorem, 6, 30, 34 n, 90,
banded matrix, exponential decay of func- 373
tion of, 317–318 ceiling function, 321
Berberian’s trick, 217 centrosymmetric matrix, 315
best L∞ rational approximation, 79 characteristic polynomial, 7, 29, 321
to exponential, 259–260 instability of methods based on, 89–
to sign function, 128–129 90
to square root, 128–129 Chebyshev rational approximation, see best
bilinear form, 313 L∞ rational approximation
computing, 318 Chebyshev series, 309
binary powering, 72 Cholesky factorization, 52, 161, 162, 324

415
416 Index

circulant matrix, 316, 324 defective matrix, 322

commutator, 235 (function) defined on the spectrum of ma-
commuting matrices trix, 3
formula for, 18 delay differential equations, 51
polynomial representation, 24 Denman–Beavers iteration, 141, 278
simultaneous triangularization, 24 stability, 145–146
comparison matrix, 161 derogatory matrix, 322
complete orthogonal decomposition, 196, determinant, 321
211, 214 determinantal scaling for Newton sign it-
complex arithmetic vs. real arithmetic, 332 eration, 119–121, 132
condition number diagonal dominance, 190, 324
absolute, 56 diagonalizable, 322
bounds, 63–64 diagonalizable matrix, 323
estimating, 64–69 diagonalization, for evaluating matrix func-
probabilistic estimates, 68–69 tion, 81–84
relative, 55, 56 differential equations, 35–37
structured, 315 with orthogonal solutions, 42
confluent permutation, 227 dimension of a vector space, 321
confluent Vandermonde matrix, 324 direction cosine matrix, 42
coninvolutory matrix, 315 directional derivative, 57
conjugate transpose, 321 discrete Fourier transform (DFT) matrix,
consistent norm, 327 10, 316
continued fraction, 79 disk function, 49
evaluation bottom-up, 81 divide and conquer algorithm, 41
evaluation top-down, 81 divided difference, 6, 9, 61, 84, 250–251,
(for) logarithm, 274 332–333
(for) tanh(x)/x, 257 contour integral, 333
(for) x coth(x), 264 from function of bidiagonal matrix,
convergence, order of, 91–92 250
cosine of a matrix, 287–301 multiple integral, 333
addition formula, 287 recursion, 332
double angle algorithm, 290–295 Drazin inverse, 33, 195
(for) cosine and sine, 296–298
error analysis, 291–293 eigenspace, 322
Fréchet derivative, 289 eigenvalue, 321–323
(in) solution of differential equations, algebraic multiplicity, 322
36 counting via sign function, 41
Kronecker addition formula, 288 defective, 322
matrix exponential formula, 287 geometric multiplicity, 322
computation, 294–295 index, 3, 322
norm, bounds for, 288 semisimple, 322
Padé approximants, 290 elliptic function, 128
power series, 287 embeddability problem, 38
preprocessing, 299 ǫ-pseudospectrum, 47, 102
sensitivity, 289 essentially nonnegative matrix, 260–261
truncation error bound for Taylor se- Euler’s formula, 287
ries, 77–78 exponential decay of functions of banded
CR iteration for matrix square root, 142 matrices, 317–318
stability, 146 exponential integrators, 36–37
cryptography, 165 exponential of a matrix, 233–268
Csanky’s method, 90 Baker–Campbell–Hausdorff formula,
235
data sparse representations, 316 of block 2 × 2 triangular matrix, 248
defective eigenvalue, 322 condition number
Index 417

bounds, 240 Faber series, 309

computation, 258–259 fast Fourier transform (FFT), 10
estimation, 258–259 Fejér points, 309
formula, 239 field of values, 322
(of) intensity matrix, 260 finite difference approximation to Fréchet
(of) normal matrix, 240 derivative, 67
(of) stochastic matrix, 240 floating point arithmetic, 331–332
under shifting, 241 floor function, 321
(in) differential equations, solution of, flop, 335
35–36 count, see operation count
(in) dynamical system conversion, 39 Fréchet derivative, see also cosine of a ma-
Fréchet derivative, 238–239 trix, Fréchet derivative; expo-
computation via Padé approxima- nential of a matrix, Fréchet deriva-
tion, 256–258 tive; square root of a matrix,
integral formula, 238 Fréchet derivative; polar decom-
Kronecker formulae, 238–239, 256– position, {Fréchet derivative of
257 iteration function; unitary fac-
norm computation, 258–259 tor, Fréchet derivative}; sign func-
norm estimation, 258–259 tion, Fréchet derivative
quadrature approximation, 254–256 chain rule, 58
scaling and squaring recurrence, 253– continuity of, 60
254 definition, 56
Taylor series formula, 238 eigensystem of, 61
hump phenomenon, 247 existence of, 60
Lie algebra, of matrix in, 314 finite difference approximation, 67
limit representation, 234 from function of block 2 × 2 block
norm, bounds for, 237–238 triangular matrix, 59, 60
of Kronecker sum, 237 of idempotent function, 98–99
of sum of matrices, 235–237 of inverse function, 58
Padé approximants, 241–242 Kronecker form of, 60, 64
evaluating, 243–246 nonsingularity, characterization of, 62
power series representation, 233 norm estimation, 64–69
preprocessing, 261 (and) numerical stability, 97–99
Rodrigues’ formula, 266 product rule, 57
scaling and squaring method, 241– sum rule, 57
250 Frobenius norm, 327
backward error result, 242 function
overscaling, 248–250 of block Toeplitz block 2 × 2 block
stability, 248 triangular matrix, 58
Schur algorithms, 250–252 of block 2×2 block triangular matrix,
Schur–Fréchet algorithm, 251 14, 85
Schur–Parlett algorithm, 251–252 of discrete Fourier transform matrix,
sensitivity, 238–241 10
skew-Hermitian/symmetric matrix, of, of identity plus low rank matrix, 22
25 of identity plus rank-1 matrix, 8–10
Suzuki–Trotter formula, 236 of 2 × 2 matrix, 29
Trotter product formula, 236 of 2 × 2 triangular matrix, 84
truncation error bound for series ap- function of matrix, see matrix function
proximation, 234 function of matrix times vector, 301–312
of 2 × 2 triangular matrix, 251 differential equation methods, 308–
uniformization method, 260 310
Zassenhaus formula, 236 Krylov subspace methods, 302–306
exponential time differencing (ETD) Eu- quadrature methods, 306–308
ler method, 37 function of operator, 28
418 Index

γn (error constant), 332 via sign function, 41

γ
en (error constant), 332 inverse scaling and squaring method
Gâteaux derivative, 57 for logarithm, 275–279
gauge function, symmetric, 327 involutory matrix, 165–166, 170, 300, 323
Gauss–Legendre quadrature, 274, 307 irreducible matrix, 329
generalized eigenvalue problem, 35 iterative refinement, 226
generalized Sylvester equation, 177
generator (of Markov process), 38 Jordan algebra, 314
geometric mean, 46–47, 131 Jordan block, 2, 322
Gershgorin’s theorem, 323, 366 splitting under f (A), 23–24
GluCat library, 47 Jordan canonical form, 2, 322
Gram–Schmidt orthogonalization, 42, 303 of f (A), 23
Gregory series, 273 of rank-1 matrix, 8–9
convergence, 284
Kronecker product, 331
H-matrix, 161 Kronecker sum, 331
H-matrix storage format, 51, 316 Krylov subspace, 302
Hadamard product, 62 Krylov subspace method, 43, 302–306
Halley’s method, 115, 131
Hamiltonian matrix, 50, 132, 315, 323 Lambert W function, 51
Hankel contour, 358 Lanczos algorithm, 66, 304
Hermite interpolating polynomial, see in- Lanczos biorthogonalization, 306
terpolating polynomial Lanczos process, 318
Hermitian matrix, 323 LAPACK, 100
Hermitian positive (semi)definite matrix, LAPACK matrix norm estimator, 66
323 on Fréchet derivative, 67
Heron’s method, 93, 166 latent root, 34 q
Hessenberg reduction, 302, 307 lattice quantum chromodynamics, 43
Hilbert matrix, 165 Leja ordering, 75
Hill cipher, 165 Lie algebra, 314, 315
holomorphic functional calculus, 28 Lie bracket, 235
Horner’s method, 72 Lie group, 315
hump phenomenon, 247 limiting accuracy (of iteration), 99
hypergeometric function, 115 linear convergence, 91
log-Euclidean mean, 47, 284
idempotent function, 98 logarithm of a matrix, 269–287
idempotent matrix, 30, 323 Briggs’ table making, 275–276, 286 q
identity matrix, 321 classification of all logarithms, 19
IEEE standard arithmetic, 332 condition number
IN iteration for matrix square root, 142 bounds, 272–273
stability, 146 (in) dynamical system conversion, 39
index existence of real logarithm, 17
of eigenvalue, 3, 322 (of) exponential, 32
of matrix, 322 Fréchet derivative, 272, 282–283
intensity matrix, 37, 101, 260 Gregory series, 273
interpolating polynomial convergence, 284
Hermite interpolation conditions, 5, integral formula, 269
302 inverse scaling and squaring method,
(for) Jordan block, 7 275–279
Lagrange–Hermite form, 6 full matrices, 278–279
Newton divided difference form, 6, overscaling, 276, 278
250 Schur method, 276–278
interpolation, for structured matrices, 50 (of) matrix product, 270–271
invariant subspace, 323 Mercator’s series, 2, 273
Index 419

Newton iteration, 285 discrete Fourier transform (DFT), 10

Padé approximants, 274–275 Drazin inverse, 33, 195
error bound, 274 essentially nonnegative, 260–261
partial fraction form, 274 H-matrix, 161
principal, 20 Hamiltonian, 50, 132, 315, 323
Schur algorithms, 279–280 Hermitian, 323
Schur–Fréchet algorithm, 279 Hermitian positive (semi)definite , 323
Schur–Parlett algorithm, 279–280 Hilbert, 165
sensitivity, 272–273 idempotent, 30, 323
times vector, by quadrature, 307 identity, 321
(of) 2 × 2 upper triangular matrix, index, 322
279–280 intensity, 37, 101, 260
(of) unitary matrix, computing, 316 inverse, perturbation expansions for,
Löwdin orthogonalization, 42 328
Löwner (partial) ordering, 330 involutory, 165–166, 170, 300, 323
LU factorization, 324 irreducible, 329
Lyapunov equation, 40 M-matrix, 159, 188
nilpotent, 4, 222, 224, 323
M-matrix, 159, 188 nonderogatory, 24, 322
Mandelbrot set, 156 nonnegative, 329
Markov process, 37–38, 45 normal, 323
MATLAB, 100 orthogonal, 323
Control System Toolbox, 39 partial isometry, 194–195, 197–198,
expm, 39, 267, 295, 296, 341 217, 326, 334
expmdemo1, 296 Pascal, 165, 189
Extended Symbolic Math Toolbox, 290 permutation, 323
fun in “Classic MATLAB”, 82 pseudoinverse (Moore–Penrose), 194–
funm, 87–89, 295, 298, 341 195, 325–326
gallery, 166 quasi-triangular, 323
logm, 39, 280, 341 reducible, 329
Matrix Computation Toolbox, 129, semiseparable, 316
149, 229, 295 Stieltjes, 315
Matrix Function Toolbox, 129, 339– stochastic, 37, 101, 329
341 root of, 38, 190, 191
normest1, 67 strictly triangular, 323
pascal, 166, 189 symmetric, 323
poly, 90 symplectic, 50, 315, 323
rcond, 67 Toeplitz, 315, 324
sqrtm, 341 transition probability, 37
Symbolic Math Toolbox, 243 trapezoidal, 323
matrix triangular, 323
A-circulant, 187 unitary, 323
absolute value, 196, 215 Vandermonde, 324
block diagonal, 324 vec-permutation, 331
centrosymmetric, 315 matrix cosine, see cosine of a matrix
circulant, 316, 324 matrix exponential, see exponential of a
commuting, formula for, 18 matrix
comparison, 161 matrix function, see also function
coninvolutory, 315 alternative meanings, 1
defective, 322 composite, 12
derogatory, 322 conditions to be real for real argu-
diagonalizable, 323 ment, 7, 13
diagonally dominant, 190, 324 continuity, 13
direction cosine, 42 definition
420 Index

ad hoc, 1–2 matrix sign function, see sign function

Cauchy integral, 8 matrix sine, see sine of a matrix
Jordan canonical form, 3 matrix square root, see square root of a
polynomial interpolation, 5 matrix
eigenvalues of, 10, 34 q Mercator’s series, 2, 273
evaluation minimal polynomial, 4, 7, 29, 301, 322
by (block) Parlett recurrence, 85– minimal polynomial with respect to vec-
88 tor, 301–302
by block diagonalization, 89 minimax rational approximation, see best
by characteristic polynomial, 89– L∞ rational approximation
90 Möbius transformation, 188
by diagonalization, 81–84 monotone function, 315
by interpolating polynomial, 89– monotone norm, 327
90 Monte Carlo simulations, 43–44
by Schur decomposition, 84–88 Moore–Penrose conditions, 325
evaluation by block Parlett recurrence, Moore–Penrose pseudoinverse, see pseu-
225–230 doinverse (Moore–Penrose)
evaluation by real Schur decomposi- MPFUN, 188
tion, 87
even, 30, 32 nearest Hermitian matrix, 198–199, 209
history of, 26–27 nearest Hermitian positive semidefinite ma-
nonprimary, 4, 14–16, 163 trix, 199
norm, bounds for, 102–104 nearest orthogonal matrix, see polar de-
odd, 30, 32 composition, unitary factor, best
polynomial identity, preservation of, approximation property
12 nearest unitary matrix, see polar decom-
primary, 2, 28 position, unitary factor, best ap-
product, 11 proximation property
properties, 10–14 Newton divided difference interpolation,
(for) rectangular A, 33 6, 250–251
(of) triangular matrix, formula for, Newton identities, 90
84 Newton iteration, see pth root of a ma-
Matrix Function Toolbox (MATLAB), 339– trix, Newton iteration; polar de-
341 composition, Newton iteration;
matrix iteration, 91–95 sign function, Newton iteration;
convergence analysis, 93–95 square root of a matrix, Newton
numerical stability, 95–99 iteration
definition, 97 Newton–Schulz iteration
order of convergence, 91 for matrix inverse, 114, 181, 190
termination criteria, 92–93 for matrix sign function, 114
termination criteria for polar itera- for matrix sign function times vector,
tions, 207–208 310
termination criteria for sign iterations, for matrix square root, 153, 157, 160
121–123, 132 for polar decomposition, 202, 211, 218
matrix logarithm, see logarithm of a ma- nilpotent matrix, 4, 222, 224, 323
trix nonderogatory matrix, 24, 322
matrix multiplication, binary powering, 72 nonlinear matrix equations, 44–46
matrix norm, see norm algebraic Riccati equation, 40
matrix pth root, see pth root of a matrix quadratic matrix equation, 45
matrix root, see pth root of a matrix solving, 24
matrix sector function, see sector function XAX = B, 44, 162, 170
matrix sequence, convergence, 328 nonnegative function, 315
matrix series, convergence, 328 nonnegative matrix, 329
matrix sign decomposition, 109 nonprimary matrix function, 4, 14–16, 163
Index 421

norm, 326–328 partial isometry, 194–195, 197–198, 217,

absolute, 327 326, 334
consistent, 327 Pascal matrix, 165, 189
estimator, 211 Paterson–Stockmeyer method for polyno-
LAPACK, 66 mial evaluation, 73–74
Frobenius, 327 performance profile, 252
matrix norm equivalence constants, permutation matrix, 323
327 t Perron–Frobenius theorem, 329
monotone, 327 polar decomposition, 42–43, 161, 162, 193–
p-norm, 326 219
subordinate, 326 analytic polar decomposition, 214
unitarily invariant, 327–328, 334 canonical polar decomposition, 194–
normal matrix, 323 196, 213, 215
notation, 319 (of block) companion matrix, 215
nuclear magnetic resonance, 37 Fréchet derivative of iteration func-
null space, 321 tion, 209
numerical stability Hermitian factor
of matrix iteration, 95–99 condition number, 201
definition, 97 perturbation bound, 200, 217
integral formula for unitary factor,
open half-plane associated with complex 196
number, 271 iterations via sign function, 202–203
operation counts, 335 measuring quality of computed fac-
notation, 71 tors, 210
order of convergence, 91–92 Newton algorithm, 210–213
ordering, Löwner (partial), 330 Newton iteration, 202
ordering, positive (semi)definite, 330 convergence, 202
ordinary differential equations, see differ- convergence with optimal scaling,
ential equations 206
orthogonal matrix, 323 derivation, 218
orthogonal Procrustes problem, 43, 198, inversion-free form, 216, 219
214 rounding error analysis, 210
rotation form, 43, 198 scaling, 205–207
perturbation bound, 201 Newton–Schulz iteration, 202, 211
orthogonal projector, 326, 334 convergence, 218
orthogonalization, 42 (of) normal matrix (commuting fac-
overlap-Dirac operator, 43 tors), 194
numerical stability of iterations, 209–
Padé approximation, 79–80 210
continued fraction connection, 79 of nearby matrix, 216
definition, 79 Padé iteration, 203–205
structure, preservation of, 316 scaling, 206
to tanh(x)/x, 257 perturbation analysis, 199–201
to cosine, 290 graded matrices, 215
to exponential, 241–242 multiplicative, 215
to logarithm, 274–275 sensitivity, 199–201
to sine, 290, 296 sign function, connection with, 196
Parlett recurrence, 85–88 (for) structured matrices, 315
block, 86–88, 225–226 termination criteria for iterations, 207–
relation with block diagonalization, 208
89 trace maximization characterization,
rounding errors in, 225 217, 219
partial fraction form of rational function, (of) 2 × 2 matrix, 216
81 unitary factor
422 Index

basis independence, 42 Newton iteration, 177–180

best approximation property, 42, convergence, 178–179
197–198, 217 coupled, 180
condition number, 201 stability, 180
Fréchet derivative, 209 principal pth root, 174
integral formula for, 196 Schur method, 175–177
perturbation bound, 200, 201 Schur–Newton algorithm, 184–186
versus QR factorization, 214 sign function method, 186–187
polarization identity, 330 (of) stochastic matrix, 190, 191
polynomial evaluation of stochastic matrix, 38
explicit powers, 73 (of) 2 × 2 matrix with complex con-
factored form, 73 jugate eigenvalues, 176
Horner’s method, 72 Pulay iteration for matrix square root, 158–
(in) Padé approximant to exponen- 159
tial, 244
Paterson–Stockmeyer method, 73–74 QR factorization, 324
rounding error bound, 74, 75 QR factorization versus polar decomposi-
positive (semi)definite ordering, 330 tion, 214
power boundedness, 97, 106 quadratic convergence, 91
power method, 65 quadratic eigenvalue problem, 45
on Fréchet derivative, 66 quadratic matrix equation, 45
preprocessing, 99–101 quasi-triangular matrix, 323
for matrix cosine and sine, 299 quaternion, 189
for matrix exponential, 261
for Schur–Parlett algorithm, 230–231 range, 321
primary matrix function, 2, 28 rank, 325
principal logarithm, 20 rational function
principal square root, 20 continued fraction form, 79
Procrustes problem, see orthogonal Pro- evaluation, 80–81
crustes problem partial fraction form, 81
product form Denman–Beavers iteration, Rayleigh quotient, 322
142, 278–279, 285 real Schur decomposition, 325
projector, 326 for evaluating matrix function, 87
pseudoinverse (Moore–Penrose), 194–195, reducible matrix, 329
325–326 repeated matrix squaring, 72
pseudospectra, 47, 102 error bound for, 247
ψ functions, 36, 261–262 research problems, 70, 106, 170, 191, 219,
evaluation by quadrature, 308 266, 310
integral representation, 261 resolvent, 8, 47
Padé approximants, 262 Riccati equation, algebraic, see algebraic
scaling and squaring method, 262 Riccati equation
pth root of a matrix, 173–191 Rodrigues’ formula, 266
classification of all pth roots, 173 root of matrix, see pth root of a matrix
condition number, 189
existence, 174 scalar product, 313
(of) identity matrix, 188 scaling and squaring method
integral formula, 174 for ψ functions, 262
invariant subspace characterization, for matrix exponential, 241–250
174 backward error result, 242
inverse Newton iteration, 181–184 overscaling, 248–250
convergence, 182–184 stability, 248
coupled, 184, 185 Schulz iteration, see Newton–Schulz iter-
residual recurrence, 181 ation
(of) M-matrix, 188 Schur decomposition, 325
Index 423

for evaluating matrix function, 84–88 numerical stability of iterations, 123–

Schur product, 62 125
Schur–Fréchet algorithm Padé iteration, 115–118
for matrix exponential, 251 convergence, 116
for matrix logarithm, 279 polar decomposition, connection with,
Schur–Parlett algorithm, 221–231 196
blocking, 226–230 principal Padé iteration, 116
blocking parameter, 226, 228 properties, 116–117
(for) matrix exponential, 251–252 properties, 107
(for) matrix logarithm, 279–280 pth root, connection with, 186–187
preprocessing, 230–231 Schur method, 112–113
reordering, 226–230 sensitivity, 109–112
sector function, 48 square root formula, 107
semidefinite programming, 48 square root, connection with, 108
semiseparable matrix, 316 termination criteria for iterations, 121–
semisimple eigenvalue, 322 123
sensitivity analysis, 48 (of) triangular matrix, 112
separation (sep) of two matrices, 225–226 (of) 2 × 2 matrix, 131
sequence of matrices, convergence, 328 Simpson’s rule, 255
sesquilinear form, 313 simultaneous triangularization of commut-
Sherman–Morrison formula, 30, 329 ing matrices, 24
Sherman–Morrison–Woodbury formula, 22, sinc function, 238
329 sinch function, 238
sign function, 39–41, 43, 107–132 sine of a matrix, 287–301
attainable accuracy, 112 addition formula, 287
best L∞ rational approximation, 128– double angle algorithm
129 (for) sine and cosine, 296–298
(of) block 2 × 2 matrix, 108 Fréchet derivative, 289
condition number, 110 (in) solution of differential equations,
conditioning, asymptotic, 111 36
eigenvalue counting, 41 Kronecker addition formula, 288
Fréchet derivative, 109 matrix exponential formula, 287
at sign(A), 123, 131 computation, 294–295
iteration for, 114 norm, bounds for, 288
of iteration function, 123 Padé approximants, 290, 296
integral formula, 107 power series, 287
invariant subspace relations, 41 preprocessing, 299
Jordan canonical form formula, 39, sensitivity, 289
107 singular value, 325
Newton algorithm, 127–128 singular value decomposition (SVD), 325
Newton iteration, 113–114 uniqueness, 334
convergence, 113 singular vector, 325
convergence of off-diagonal of Jor- Solomon equations, 37
dan form, 119 spectral decomposition, 325
convergence with spectral scaling, spectral radius, 322, 328
120 spectral scaling for Newton sign iteration,
generalized, 130 120–121, 132
numerical stability, 124–125 spectrum, 322
pure imaginary eigenvalue, behaviour square root of a matrix, 133–171
for, 117–118, 131 automorphism group, of matrix in,
scaling, 119–121, 130, 132 314
termination criteria, 122–123 best L∞ rational approximation, 128–
Newton–Schulz iteration, 114, 115 129
for sign function times vector, 310 binomial expansion, 154
424 Index

binomial iteration, 154–157 stability, 153

classification of all square roots, 18– principal, 20
19 (of) product, 32
classification of real primary square Pulay iteration, 158–159
roots, 138 real Schur method, 136–139
comparison of methods, 164 Schur method, 135–139
(of) complex number, 138 behaviour for singular A, 136
condition number, 134 numerical stability, 139
condition number for relative resid- sensitivity, 133–135
ual, 135 sign function, connection with, 108
CR iteration, 142 (of) skew-Hamiltonian matrix, 315
stability, 146 small-normed, computing, 162–164
Denman–Beavers iteration, 141, 278 times vector, 308–310
product form, 142, 278–279, 285 (of) triangular matrix, 135–136
scaled, 147 (of) 2 × 2 matrix, 168
stability, 145–146 (of) 2 × 2 matrix with complex con-
existence, 16 jugate eigenvalues, 137
existence of real primary square root, (of) unitary matrix, 313
138 computing, 316
existence of real square root, 17 Visser iteration, 158–159
expressed via exponential and loga- use in operator theory, 167
rithm, 32 Stieltjes matrix, 315
Fréchet derivative, 134 stochastic matrix, 37, 101, 329
(of) H-matrix, 161 root of, 38, 190, 191
(of) Hermitian positive definite ma- stopping test, see matrix iteration, termi-
trix, 20, 32 nation criteria
computing, 161–162 Strang splitting, 236
(of) identity (involutory matrix), 165– strictly triangular matrix, 323
166 subordinate matrix norm, 326
IN iteration, 142 superlinear convergence, 91
scaled, 148 Suzuki–Trotter formula, 236
stability, 146 SVD, see singular value decomposition
integral formula, 133 Sylvester cascade, 257
iterations via sign function, 152–154 Sylvester equation, 39, 86, 139, 225, 331
(of) Jordan block, 30 generalized, 177
(of) M-matrix, 159–161 solution via sign function, 39–40
modified Newton iteration, 157–159 symmetric gauge function, 327
Newton iteration, 93, 139–148 symmetric matrix, 323
convergence, 140–141 symplectic matrix, 50, 315, 323
convergence for singular A, 143–
144 Taylor series, 76–78
instability of, 95–96, 144–145, 166 (for) atomic triangular matrix, 221–
negative eigenvalue, behaviour for, 224
169 (for) Jordan block, 4
relation to Newton sign iteration, nonmonotonic convergence, 222–223
140–141 radius of convergence, 76
scaling, 147 rounding errors in evaluation, 74–75,
stability, 144–145 224
(full) Newton method, 139 truncation criterion, 222–224
monotonic convergence, 169 truncation error bound, 77–78
Newton–Schulz iteration, 153, 157, 160 Toeplitz matrix, 315, 324
numerical stability of iterations, 144– trace, 321
147 transition probability matrix, 37
Padé iteration, 152 transpose, 321
Index 425

trapezium rule, 254, 255, 307–308

trapezoidal matrix, 323
triangular matrix, 323
Trotter product formula, 236

uniformization method, 260

unit roundoff, 332
unitarily invariant norm, 327–328, 334
unitary matrix, 323
unwinding number, 269–270, 280

values of function on spectrum of matrix,

3
Vandermonde matrix, 324
vec operator, 331
vec-permutation matrix, 331
Visser iteration for matrix square root,
158–159
use in operator theory, 167
von Neumann divergence, 50

Wiener–Hopf factorization, 188

Zolotarev’s theorem, 128