EE103 (Fall 2011-12)

6. Cholesky factorization
triangular matrices forward and backward substitution the Cholesky factorization solving Ax = b with A positive denite inverse of a positive denite matrix permutation matrices sparse Cholesky factorization

6-1

Triangular matrix
a square matrix A is lower triangular if aij = 0 for j > i a11 a21 . . 0 a22 . . 0 0 0 0 ... 0 0 an1,n1 0 an,n1 ann

A= an1,1 an1,2 an1 an2

A is upper triangular if aij = 0 for j < i (AT is lower triangular)

a triangular matrix is unit upper/lower triangular if aii = 1 for all i

Cholesky factorization

6-2

Forward substitution
solve Ax = b when A is lower triangular with nonzero diagonal elements b1 x1 a11 0 0 a21 a22 0 x2 b2 . = . . . . . . . . . . . . bn xn an1 an2 ann algorithm: x1 := b1/a11 x2 := (b2 a21x1)/a22 x3 := (b3 a31x1 a32x2)/a33 . .

xn := (bn an1x1 an2x2 an,n1xn1)/ann cost: 1 + 3 + 5 + + (2n 1) = n2 ops

Cholesky factorization 6-3

Back substitution
solve Ax = b when A is upper triangular with nonzero diagonal elements x1 a11 a1,n1 a 1n b1 . . ... . . . . . . . = . 0 an1,n1 an1,n xn1 bn1 xn 0 0 ann bn algorithm: xn := bn/ann xn2 := (bn2 an2,n1xn1 an2,nxn)/an2,n2 . . x1 := (b1 a12x2 a13x3 a1nxn)/a11 cost: n2 ops
Cholesky factorization 6-4

xn1 := (bn1 an1,nxn)/an1,n1

Inverse of a triangular matrix

triangular matrix A with nonzero diagonal elements is nonsingular Ax = b is solvable via forward/back substitution; hence A has full range therefore A has a zero nullspace, is invertible, etc. (see p.4-8) inverse can be computed by solving AX = I column by column A X1 X2 Xn = e1 e2 en

inverse of lower triangular matrix is lower triangular inverse of upper triangular matrix is upper triangular
Cholesky factorization 6-5

Cholesky factorization
every positive denite matrix A can be factored as A = LLT where L is lower triangular with positive diagonal elements cost: (1/3)n3 ops if A is of order n

L is called the Cholesky factor of A can be interpreted as square root of a positive dene matrix

Cholesky factorization

6-6

Cholesky factorization algorithm

partition matrices in A = LLT as a11 AT 21 A21 A22 = = algorithm 1. determine l11 and L21: l11 = 2. compute L22 from
T A22 L21LT 21 = L22 L22

l11 0 L21 L22

l11 LT 21 T 0 L22

2 l11 l11LT 21 T l11L21 L21L21 + L22LT 22

a11,

L21 =

1 l11

A21

this is a Cholesky factorization of order n 1

Cholesky factorization 6-7

proof that the algorithm works for positive denite A of order n step 1: if A is positive denite then a11 > 0 step 2: if A is positive denite, then A22 L21LT 21 = A22 is positive denite (see page 4-23) hence the algorithm works for n = m if it works for n = m 1 it obviously works for n = 1; therefore it works for all n 1 A21AT 21 a11

Cholesky factorization

6-8

Example
l11 l21 l31 l11 0 0 25 15 5 15 18 0 = l21 l22 0 0 l22 l32 0 0 l33 l31 l32 l33 5 0 11 rst column of L 5 3 1 5 0 0 25 15 5 15 18 0 = 3 l22 0 0 l22 l32 0 0 l33 1 l32 l33 5 0 11 second column of L 18 0 0 11 3 1 9 3 3 10
Cholesky factorization

3 1 =

l22 0 l32 l33 3 1 0 l33

l22 l32 0 l33

3 0 1 l33

6-9

2 third column of L: 10 1 = l33 , i.e., l33 = 3

conclusion: 5 3 1 5 0 0 25 15 5 15 18 1 0 = 3 3 0 0 3 0 0 3 1 1 3 5 0 11

Cholesky factorization

6-10

Solving equations with positive denite A

Ax = b (A positive denite of order n)

algorithm factor A as A = LLT solve LLT x = b forward substitution Lz = b back substitution LT x = z cost: (1/3)n3 ops factorization: (1/3)n3 forward and backward substitution: 2n2
Cholesky factorization 6-11

Inverse of a positive denite matrix

suppose A is positive denite with Cholesky factorization A = LLT L is invertible (its diagonal elements are nonzero) X = LT L1 is a right inverse of A: AX = LLT LT L1 = LL1 = I X = LT L1 is a left inverse of A: XA = LT L1LLT = LT LT = I

hence, A is invertible and A 1 = L T L 1

Cholesky factorization 6-12

Summary
if A is positive denite of order n

A can be factored as LLT the cost of the factorization is (1/3)n3 ops Ax = b can be solved in (1/3)n3 ops A is invertible with inverse: A1 = LT L1

Cholesky factorization

6-13

Sparse positive denite matrices

a matrix is sparse if most of its elements are zero a matrix is dense if it is not sparse Cholesky factorization of dense matrices (1/3)n3 ops on a current PC: a few seconds or less, for n up to a few 1000 Cholesky factorization of sparse matrices if A is very sparse, then L is often (but not always) sparse

if L is sparse, the cost of the factorization is much less than (1/3)n3 very large sets of equations (n 106) are solved by exploiting sparsity
Cholesky factorization 6-14

exact cost depends on n, #nonzero elements, sparsity pattern

Eect of ordering
sparse equation (a is an (n 1)-vector with a < 1) 1 aT a I factorization 1 aT a I = 1 0 a L22 1 aT 0 LT 22 where I aaT = L22LT 22 u v = b c

factorization with 100% ll-in

Cholesky factorization 6-15

reordered equation I aT factorization I aT a 1 = I aT 0 1 aT a I a 0 1 aT a a 1 v u = c b

factorization with zero ll-in

Cholesky factorization 6-16

Permutation matrices
a permutation matrix is the identity matrix with its rows reordered, e.g., 0 1 0 0 1 0 0 0 1 1 0 0 , 1 0 0 0 0 1 the vector Ax is a permutation of x x2 x1 0 1 0 0 0 1 x2 = x3 x1 x3 1 0 0 AT x is the inverse permutation applied to x x3 x1 0 0 1 1 0 0 x2 = x1 x2 x3 0 1 0 AT A = AAT = I , so A is invertible and A1 = AT
Cholesky factorization 6-17

Solving Ax = b when A is a permutation matrix

the solution of Ax = b is x = AT b

example

1. 5 x1 0 1 0 0 0 1 x2 = 10.0 2. 1 x3 1 0 0 solution is x = (2.1, 1.5, 10.0) cost: zero ops

Cholesky factorization

6-18

Sparse Cholesky factorization

if A is sparse and positive denite, it is usually factored as A = P LLT P T P a permutation matrix; L lower triangular with positive diagonal elements interpretation: we permute the rows and columns of A and factor P T AP = LLT

choice of P greatly aects the sparsity L many heuristic methods (that we dont cover) exist for selecting good permutation matrices P

Cholesky factorization

6-19

Example
sparsity pattern of A Cholesky factor of A

pattern of P T AP

Cholesky factor of P T AP

Cholesky factorization

6-20

Solving sparse positive denite equations

solve Ax = b via factorization A = P LLT P T algorithm 1. b := P T b 2. solve Lz = b by forward substitution 3. solve LT y = z by back substitution 4. x := P y

interpretation: we solve

(P T AP ) y = b

using the Cholesky factorization of P T AP

Cholesky factorization

6-21