8 Finite Differences: Partial Differential Equations

The world is defined by structure in space and time, and it is forever changing in complex
ways that can’t be solved exactly. Therefore the numerical solution of partial differential
equations leads to some of the most important, and computationally intensive, tasks in
all of numerical analysis (such as forecasting the weather). This chapter introduces finite
difference techniques; the next two will look at other ways to discretize partial differential
equations (finite elements and cellular automata). Just as we used a Taylor expansion to
derive a numerical approximation for ordinary differential equations, the same procedure
can be applied to partial differential equations. Because the discretization must be done
in space as well as time, there are many more possible strategies for finding good (and
bad) approximations.
We will start with two degrees of freedom, say one spatial variable x and a time t.
Given a function u(x, t), its spatial derivatives are found from the Taylor expansion
 
∂u  (∆x)2 ∂ 2 u 
u(x + ∆x, t) = u(x, t) + ∆x + + O[(∆x)3 ] . (8.1)
∂x x,t 2! ∂x2 x,t

The first partial derivative can be approximated by the forward difference


u(x + ∆x, t) − u(x, t) ∂u 
= + O[∆x] . (8.2)
∆x ∂x x,t

If we replace ∆x with −∆x, this becomes the equally reasonable backwards difference
approximation

u(x, t) − u(x − ∆x, t) ∂u 
= + O[∆x] . (8.3)
∆x ∂x x,t

Their order can be raised by taking the difference between two time steps, which subtracts
out the quadratic term:

u(x + ∆x, t) − u(x − ∆x, t) ∂u 
= + O[(∆x)2 ] . (8.4)
2∆x ∂x x,t

Although this might appear always to be preferable, we will see that it can have surprising
undesirable stability properties.
The straightforward finite difference approximation to the second partial derivative is
 DRAFT 8.1 Hyperbolic Equations: Waves 93

n + 1 n + 1 n + 1
u j-1 u j u j+ 1

n n n
u j-1 u j u j+ 1

Figure 8.1. A computational cluster.

also correct to first order:

 
1 u(x + ∆x, t) − u(x, t) u(x, t) − u(x − ∆x, t)
− =
∆x ∆x ∆x

u(x + ∆x, t) − 2u(x, t) + u(x − ∆x, t) ∂ 2 u 
= + O[(∆x)2 ] . (8.5)
(∆x)2 ∂x2 x,t
Numerical methods for partial differential equations are usually classified by the char-
acteristics for the equation that they apply to (Chapter 4), which measure how information
from the boundary conditions influences the solution. Characteristics can even be used
as the basis for numerical solvers [Ames, 1992], but here we will simply use them as con-
venient labels for the most common cases: a wave equation (hyperbolic characteristics),
diffusive processes (parabolic), and boundary value problems (elliptic). More complex
systems can have some or all of these elements.

8.1 H Y P E R B O L I C E Q U A T I O N S : W A V E S

To see how the stability of the solution depends on the finite difference scheme, let’s
start with a simple first-order hyperbolic PDE for a conserved quantity in one dimension
∂u ∂u
= −v . (8.6)
∂t ∂x
Substitution readily shows that this is solved by any function of the form
u = f (x − vt) . (8.7)
Writing u(j∆x, n∆t) = unj to make the notation clearer, a simple discretization is
first-order in time and second-order in space:
 n 
un+1
j − unj uj+1 − unj−1
= −v
∆t 2∆x
v∆t n
un+1
j = unj − (u − unj−1 ) (8.8)
2∆x j+1
(using a first-order spatial approximation would make it asymmetrical). It can be con-
venient to represent such approximations by drawing the cluster of values used in the
update rule (Figure 8.1). Given an initial distribution unj , it is straightforward to iterate
this rule forward in time.
To analyze the stability of a finite difference scheme, the von Neumann stability
94 Finite Differences: Partial Differential Equations DRAFT

analysis locally linearizes the equations (if they are not linear) and then separates the
temporal and spatial dependence (Section 4.3) to look at the growth of the linear modes
unj = A(k)n eijk∆x . (8.9)
This assumed form has an oscillatory dependence on space, which can be used to syn-
thesize any initial condition, and an exponential dependence on time to test the stability.
It’s customary to define k with respect to the spatial separation ∆x, because that will
parameterize the solution, but use an integer n for the time step since that will be used
only for checking the stability.
Plugging in this ansatz gives a solution to the finite difference equation for A(k). If
|A(k)| > 1 for some k, then these modes will diverge and the scheme will be unstable
(remember that the exact solution (8.7) does not diverge). For equation (8.8) this gives
v∆t  n i(j+1)k∆x 
An+1 eijk∆x = An eijk∆x − A e − An ei(j−1)k∆x
2∆x
v∆t  ik∆x 
A= 1− e − e−ik∆x
2∆x
v∆t
= 1−i sin k∆x . (8.10)
∆x
The absolute magnitude of this is always greater than 1, and so this scheme is always
unstable. Any initial condition will diverge!
This disturbing behavior in such a sensible approximation is easily corrected with the
Lax method, which averages the neighbors for the time derivative:
1 v∆t n
un+1
j = (unj+1 + unj−1 ) − (u − unj−1 ) . (8.11)
2 2∆x j+1
Repeating the stability analysis shows that the amplitude of a solution is
v∆t
A = cos k∆x − i sin k∆x . (8.12)
∆x
Requiring that the magnitude be less than 1,
 
2 2 v∆t 2 2
|A| = cos k∆x + sin k∆x ≤ 1
∆x
|v|∆t
⇒ ≤1 . (8.13)
∆x
This is the Courant–Friedrichs–Levy stability criterion, and it will recur for a number
of other schemes. It says that the velocity at which information propogates within the
numerical algorithm (∆x/∆t) must be faster than the velocity of the solution v. For space
and time steps that satisfy this condition, the Lax method will be stable. Otherwise, there
is a “numerical boom” as the real solution tries to out-run the rate at which the numerical
solution can advance. The lateral averaging for the time derivative in the Lax method
helps the numerical information propagate, compared to the unstable approximation that
we started with (equation 8.8). The origin of this stability becomes clearer if the Lax
method is rewritten by subtracting unj from both sides:
 n 
un+1
j − unj uj+1 − unj−1 1
= −v + (un − 2unj + unj−1 ) . (8.14)
∆t 2∆x 2∆t j+1
 DRAFT 8.2 Parabolic Equations: Diffusion 95

This is just our original equation (8.8), with an extra fictitious diffusion term added that
depends on the discretization:

∂u ∂u (∆x)2 ∂ 2 u
= −v + . (8.15)
∂t ∂x 2∆t ∂x2
This is an example of an artificial numerical dissipation, which can occur (and even
be added intentionally) in stable schemes. In this case it is good, because it serves to
damp out the spurious high-frequency modes (k ∼ 1) while preserving the desired long
wavelength solutions. In other cases it might be a problem if the goal is to look at the
long-term behavior of a nondissipative system.
The Lax method cures the stability problem and is accurate to second order in space,
but it is only first-order in time. This means that v∆t will need to be much smaller than
∆x to have the same accuracy in time and space (even though a much larger time step
will be stable). A natural improvement is to go to second order in time:

v∆t n

un+1
j = ujn−1 − uj+1 − unj−1 . (8.16)
∆x
The stability analysis for this equation now leads to a quadratic polynomial for the
amplitude, giving two solutions
s
 2
v∆t v∆t
A = −i sin(k∆x) ± 1 − sin(k∆x) . (8.17)
∆x ∆x

If |v|∆t/∆x ≤ 1 then the radical will be real, and |A|2 = 1 independent of k. The Courant
condition applies again, but now there is no dependence of the amplitude on the spatial
wavelength k and so there is no artificial damping (unlike the Lax method). This is called
the leapfrog method because it separates the space into two interpenetrating lattices that
do not influence each other (un+1
j does not depend on unj ). Numerical round-off errors
can lead to a divergence of the sublattices over long times, requiring the addition of an
artificial coupling term.
Problem 8.1 considers the finite difference approximaton to the wave equation.

8.2 P A R A B O L I C E Q U A T I O N S : D I F F U S I O N

We will next look for finite difference approximations for the 1D diffusion equation
 
∂u ∂ ∂u
= D , (8.18)
∂t ∂x ∂x

and will assume that the diffusion coefficient is constant

∂u ∂2u
=D 2 . (8.19)
∂t ∂x
The methods to be described will have natural generalizations when D is not constant.
96 Finite Differences: Partial Differential Equations DRAFT

The straightforward discretization is

 n 
un+1
j − unj uj+1 − 2unj + unj−1
=D
∆t (∆x)2

D∆t  n 
un+1
j = unj + 2
uj+1 − 2unj + unj−1 . (8.20)
(∆x)
Solving the stability analysis,
D∆t h i
A = 1+ eik∆x − 2 + e−ik∆x
(∆x)2
| {z }
2 cos k∆x − 2
| {z }
 
2 k∆x
2 2 cos −1 −2
2
4D∆t 2 k∆x
= 1− sin
(∆x)2 2
4D∆t 2D∆t
|A| ≤ 1 ⇒ ≤2 ⇒ ≤1 . (8.21)
(∆x)2 (∆x)2
The method is stable for small step sizes, but since for a diffusive process the time t
to expand a distance L is roughly t ∼ L2 /D (Problem 4.2), the number of time steps
required to model this will be ∼ L2 /(∆x)2 (i.e., a very large number).
The stability can be improved by evaluating the space derivative forwards in time:
" #
un+1
j − unj un+1
j+1 − 2uj
n+1
+ un+1
j−1
=D
∆t (∆x)2

− D∆t2 un+1
 
un+1
j j+1 − 2uj
n+1
+ un+1
j−1 = uj
n
. (8.22)
(∆x)
The stability analysis for this is
D∆t h ik∆x −ik∆x
i
A− Ae − 2A + Ae =1
(∆x)2
 
4D∆t 2 k∆x
A 1+ sin =1
(∆x)2 2
1
A= ≤1. (8.23)
4D∆t 2 k∆x
1+ sin
(∆x)2 2
This scheme is stable for all step sizes, but might appear to be useless: how can we
implement it since we don’t know the forward values used in the space derivative? These
future values are implicitly determined by the past values, and the trick is to recognize
that the full set of equations can be inverted. The stability follows because peeking into
the future in this way helps move information through the solution more quickly.
The boundary conditions are typically given as either fixed (u1 and uN are specified)
or periodic (u1 = uN +1 , so that the system does not have edges). If we assume fixed
boundary conditions and define α = D∆t/(∆x)2 , then equation (8.22) can be written as
 DRAFT 8.2 Parabolic Equations: Diffusion 97

a matrix problem
    
1 0 0 ··· 0 un+1 un1
1
 −α 1 + 2α −α 0 · · · 0 
  un+1
2
 
  un2 


 . .. . .. . ..

 ..   .. 
  .   . 
 .   
..   n+1
 =  uni
 .  u   
 . 0 −α 1 + 2α −α 0 .  i  



. . .
 ..   .. 

 .. .. .. 
 .   . 
  un+1   n 

 0 ··· 0 −α 1 + 2α −α  N −1
  uN −1 
0 ··· 0 0 1 un+1
N unN

This is a tridiagonal matrix (all the elements are zero, except for the diagonal and the
adjacent elements), and it can easily be solved to find un+1 in terms of un without doing
all of the work needed to invert an arbitrary matrix.
The system of equations corresponding to an arbitrary tridiagonal matrix is
b1 u1 + c1 u2 = d1
ai ui−1 + bi ui + ci ui+1 = di (2 ≤ i ≤ N − 1) .
aN uN −1 + bN uN = dN (8.24)
For us, ai = ci = −α, bi = 1 + 2α, and the di = un+1 i . These can be solved in two
passes. In a system of equations, multiplying one equation by a constant and adding it
to another one does not change the solution. If we multiply the first row by −a2 /b1 and
add it to the second row this will eliminate the a2 term. If we then divide the second row
by the new b2 term, and repeat these steps (Gauss elimination) down the matrix, we will
get a new matrix with zeros below the diagonal and ones on the diagonal (this is called
an upper-diagonal matrix). Then, a reverse pass back up the matrix that multiplies the
new N th row by the new values for −cN −1 /bN and adds it to the previous row, and
so forth, converts the matrix to a diagonal one and the solution can be read off. Using
primes for the values after the forward pass, a bit of algebra shows that
c1 d1
c′1 = d′1 =
b1 b1

ci+1 di+1 − ai+1 d′i

c′i+1 = d′i+1 = . (8.25)
bi+1 − ai+1 c′i bi+1 − ai+1 c′i
Then, the reverse pass gives
uN = d′N

ui = d′i − c′i ui+1 . (8.26)

This is an O(N ) steps algorithm, and so there is little performance penalty for using an
implicit discretization instead of an explicit one.
The accuracy can be improved to second order in time by averaging the spatial deriva-
tive at the beginning and the end of the interval:
un+1
j − unj D
= [(un+1 − 2un+1 + un+1 n n n
j−1 ) + (uj+1 − 2uj + uj−1 )] . (8.27)
∆t 2(∆x)2 j+1 j
98 Finite Differences: Partial Differential Equations DRAFT

u n + 1
n + 1 j,k + 1
u n + 1 /2
j- 1 ,k
u j,k

n + 1 /2
n + 1 u j,k
u j,k - 1 u n
j,k + 1
n n + 1 /2
u j,k u j+ 1 ,k
n
u j,k - 1

Figure 8.2. Computational clusters for alternating implicit steps

for the x (solid) and y (dashed) coordinates.

This is called the Crank–Nicholson method, and the stability analysis shows that it is
stable for any time step:
1 − (2D∆t/(∆x)2 ) sin2 (kx/2)
A= . (8.28)
1 + (2D∆t/(∆x)2 ) sin2 (kx/2)
To solve a diffusion problem in higher dimensions we could make it implicit in all of
the dimensions. This works, but results in a banded matrix that is no longer tridiagonal.
Although it can be inverted (with much more effort), a simple trick recovers tridiagonal
matrices. This is the Alternating-Direction Implicit method (ADI), which is an ex-
ample of the general principle of Operator Splitting. In 2D, instead of advancing both
coordinates in one step, the x coordinates are advanced in a first implicit (tridiagonal)
step of ∆t/2, and then in a second implicit step of ∆t/2 the new y coordinates are found.
For the implicit method this is
n+1/2 D∆t n+1/2 n+1/2 n+1/2
uj,k = unj,k + (u − 2uj,k + uj−1,k +
2(∆x)2 j+1,k
unj,k+1 − 2unj,k + unj,k−1 )
n+1/2 D∆t n+1/2 n+1/2 n+1/2
un+1
j,k = uj,k + (u − 2uj,k + uj−1,k +
2(∆x)2 j+1,k
un+1 n+1 n+1
j,k−1 − 2uj,k + uj,k+1 ) (8.29)
(shown in Figure 8.2).

8.3 E L L I P T I C E Q U A T I O N S : B O U N D A R Y V A L U E S

The remaining class of partial differential equations to be discussed are of the form of
the elliptic boundary value problem
∇2 u = ρ . (8.30)
This is Poisson’s equation; if the source term ρ = 0 then it becomes Laplace’s equation.
The boundary condition can be specified by giving the value of u on the boundary
(Dirichlet conditions), the value of the normal derivative of u on the boundary (Neumann
conditions), or a mixture of these. Poisson’s equation and Laplace’s equation are among
 DRAFT 8.3 Elliptic Equations: Boundary Values 99

the most commonly solved numerical equations because they apply to so many different
areas. It’s instructive to look at some important examples to see how they arise:

• Heat Flow
The heat flux F~ in a material is proportional to the gradient of the temperature T by
the thermal conductivity K:
~ = −K∇T
F . (8.31)

The change of heat Q in a volume is related to the temperature change by the specific
heat C and the density ρ by
Z
dQ ∂T
= Cρ dV , (8.32)
dt V ∂t
and it is also equal to the surface integral of the heat flux
Z
dQ
= − F~ · dA ~ . (8.33)
dt S

Equating these and using Gauss’ theorem,

Z Z
∂T
Cρ dV = − F~ · dA
~
∂t
Z
= − ∇ · F~ dV
Z
= K∇2 T dV
1 ∂T
⇒ ∇2 T = , (8.34)
κ ∂t
where κ = K/Cρ is the thermal diffusivity. This is a diffusion equation, and for a
steady-state problem the time derivative of T will vanish, leaving Laplace’s equation
for the temperature distribution.
• Fluid Flow
The continuity equation for a fluid of density ρ and velocity ~v is
∂ρ
+ ∇ · ρ~v = 0 . (8.35)
∂t
If the density is constant this reduces to

∇ · ~v = 0 . (8.36)

A second condition on the velocity field is that if it starts out irrotational (∇ ×~v = 0) it
will remain irrotational (for example, this will be the case if at −∞ the flow is uniform;
see any fluids text such as [Batchelor, 1967] for a derivation). If the curl of a vector
field vanishes, it can be written as the gradient of a potential:

∇ × ~v = 0 ⇒ ~v = −∇ϕ . (8.37)
100 Finite Differences: Partial Differential Equations DRAFT

Combining this with the continuity equation gives Laplace’s equation again
∇ · (∇ϕ) = ∇2 ϕ = 0 . (8.38)
• Electric Fields
~ is determined in terms of the charge density ρ, the
In MKS units, the electric field E
~
magnetic field B, and the polarizability ǫ by
~
~ =ρ
∇ · ǫE ∇×E ~ = − ∂B . (8.39)
∂t
~ is zero, and so the electric field
For a steady-state problem, the time derivative of B
is the gradient of a potential ϕ
~ =0⇒E
∇×E ~ = −∇ϕ . (8.40)
Combining this with the charge equation and assuming that ǫ does not depend on
position gives Poisson’s equation
ρ
∇2 ϕ = − . (8.41)
ǫ
A problem closely related to Laplace’s equation is Helmholtz’s Equation, which we
found by separating out the time dependence in a wave equation:
1 ∂2u
∇2 u =
c2 ∂t2
ω 2 iωt
u = Aeiωt ⇒ eiωt ∇2 A = − Ae
c2
ω2
∇2 A + A=0 . (8.42)
c2
|{z}
k2
The obvious finite difference approximation for Poisson’s equation is (in 1D)
uj+1 − 2uj + uj−1
= ρj . (8.43)
(∆x)2
Unlike the initial value problems we have been studying, this has no time dependence.
It can be written as a matrix problem which can be solved exactly:
A · ~u = (∆x)2 ρ~ ⇒ ~u = (∆x)2 A−1 · ρ~ , (8.44)
where
   
−2 1 0 ··· 0 1 u1
 1 −2 1 0 · · · 0 
  
 u2 


 0 . .. . .. . ..

  .. 
   . 
 .  
..   
 ..
A= and ~u =  ui  . (8.45)

0 1 −2 1 0 .   

. . .
  .. 

 .. .. .. 0 
  . 
   
 0 ··· 0 1 −2 1   uN −1 
1 0 ··· 0 1 −2 uN
 DRAFT 8.3 Elliptic Equations: Boundary Values 101

This is for periodic boundary conditions; with fixed boundary conditions the upper-right
and lower-left corner elements of A would be zero. For Helmholtz’s equation the matrix
problem requires finding the eigenvalues and eigenvectors of A to determine the modes.
With fixed boundary conditions A is tridiagonal, which we saw in Section 8.2 is easy
to solve. For periodic boundary conditions it is almost tridiagonal, but with the extra
corner elements added. Fortunately, if a matrix is related to another one by adding the
outer product of two vectors

A → (A + ~u ⊗ ~v ) (~u ⊗ ~v )i,j ≡ ui vj (8.46)

then there is a simple relationship between their inverses. In our case, ~u = ~v =

(1, 0, . . . , 0, 1), and −1 is subtracted from the upper-left and lower-right elements of
A before the inversion. The inverses are related by the Sherman–Morrison formula,
derived by doing a power series expansion of the inverse and then using the associativity
of the inner and outer products

(A + ~u ⊗ ~v )−1 = (A · (1 + A−1 · ~u ⊗ ~v ))−1

= (1 + A−1 · ~u ⊗ ~v )−1 · A−1
= [1 − (A−1 · ~u ⊗ ~v ) +
(A−1 · ~u ⊗ ~v ) · (A−1 · ~u ⊗ ~v ) − · · ·] · A−1
= [1 − (A−1 · ~u ⊗ ~v ) +
A−1 · ~u ⊗ (~v · A−1 · ~u) ⊗~v − · · ·] · A−1
| {z }
β
= [1 − (A−1 · ~u ⊗ ~v )(1 − β + β 2 − · · ·)] · A−1
(A−1 · ~u) ⊗ (~v · A−1 )
= A−1 − . (8.47)
1+β

In 2D, the finite difference approximation for Poisson’s equation is

uj+1,k + uj−1,k + uj,k+1 + uj,k−1 − 4uj,k

= ρj,k . (8.48)
(∆x)2

This can be also be solved by a matrix inversion:

   
u1,1 ρ1,1

 u2,1 


 ρ2,1 

 ..   .. 

 . 


 . 

 u   ρ 
 N,1   N,1 
   
A· u  = ∆x2  ρ1,2 , (8.49)
 1,2

  . 
 ..   . 
 .   . 
   
 uN,2   ρN,2 
   
 ..   .. 
 .   . 
uN,N ρN,N
102 Finite Differences: Partial Differential Equations DRAFT

where
 
.. .. .. .. ..
. . . . .
 
 0
 1 0 ··· 0 1 −4 1 0 ··· 0 1 0 ··· 

A =  ··· 0 1 0 ··· 0
 1 −4 1 0 ··· 0 1 0 ···  

 ··· 0 1 0 ··· 0 1 −4 1 0 ··· 0 1 0  
.. .. .. .. ..
. . . . .
(8.50)
The generalization to higher dimensions is straightforward, making a column vector from
a multidimensional array by sequentially reading down the axes.
Matrix (8.50) is banded-diagonal, having 1s offset by N elements on either side of the
diagonal, and in higher dimensions there will be more bands. Although these are sparse
matrices which can be solved directly (see [Press et al., 2007] for techniques), the effort
required in two or more dimensions can quickly become prohibitive.
One alternative that is applicable for constant-coefficient linear problems is Fourier
Transform methods. In 2D, the discrete Fourier transform of the field is
M−1
XN −1
X
ûm,n = uj,k e2πimj/M e2πink/N , (8.51)
j=0 k=0

and the inverse transform is

M−1 N −1
1 X X
uj,k = ûm,n e−2πijm/M e−2πikn/N . (8.52)
N 2 m=0 n=0
Plugging the transforms of u and ρ into equation (8.48), and recognizing that the trans-
form of a function can vanish everywhere only if the function itself is equal to zero,
gives
 
ûm,n e2πim/M + e−2πim/N + e2πin/M + e−2πin/N − 4

= ρ̂m,n (∆x)2 . (8.53)

Rearranging terms and simplifying the complex exponentials,
ρ̂m,n (∆x)2
ûm,n = . (8.54)
2πm 2πn
2 cos + 2 cos −4
M N
Therefore, the forward transform of the source term can be calculated, this can be used
to find the û, and then the inverse transform can be taken to find u. This solution
imposes periodic boundary conditions; some other boundary conditions can be imposed
by choosing the form of the expansion (for example, using only sines if the solution
vanishes on the boundary).
The Fourier transform method is so simple only for linear constant-coefficient prob-
lems with boundary conditions along the coordinate axes; some kind of iterative algorithm
is needed for arbitrary geometries. An important class of techniques is found by remem-
bering that Poisson’s equation is the steady-state solution of a diffusion equation
∂u
= ∇2 u − ρ (8.55)
∂t
 DRAFT 8.3 Elliptic Equations: Boundary Values 103

whether or not it originally arose from diffusion. This means that all of the techniques
for solving diffusion problems can be applied here, with the asymptotic answer giving
the solution to Poisson’s equation.
The simplest is Jacobi’s method, which just takes forward time differences
∆t
un+1 n
j,k = uj,k + (un + unj−1,k + unj,k+1 + unj,k−1 − 4unj,k ) − ∆tρj,k . (8.56)
(∆x)2 j+1,k
Here the time step does not have any physical significance; we just want the largest
possible step that converges to the solution. In 2D the Courant condition is ∆t/(∆x)2 ≤
1/4, leading to
1 n (∆x)2
un+1
j,k = (u + u n
+ u n
+ u n
) − ρj,k . (8.57)
4 j+1,k j−1,k j,k+1 j,k−1
4
This has a very natural interpretation: starting from a random guess, at each time step
each lattice site is set to the average of its neighbors and then a source term is added. This
process is repeated until the solution stops changing, a technique called relaxation. A
related algorithm, the Gauss–Seidel method, uses updated values as soon as they become
available:
1 n (∆x)2
un+1 n+1 n n+1
j,k = (uj+1,k + uj−1,k + uj,k+1 + uj,k−1 ) − ρj,k (8.58)
4 4
(assuming that the updating proceeds down rows).
These both work, but the convergence is too slow for them to be useful. This can be
seen by rewriting them in terms of the matrix problem
A · ~u = ρ
~ (8.59)
and then separating A into lower-triangular, diagonal, and upper-triangular parts
(L + D + U) · ~u = ρ
~ . (8.60)
If ~un is the solution after the nth iteration, the Jacobi method moves the lower- and
upper-triangular parts to the right hand side to update ~u:
D · ~un+1 = −(L + U) · ~un + ρ
~ (8.61)
The convergence rate will be determined by the eigenvalues of the iteration matrix −D−1 ·
(L + U). The magnitude of all of the eigenvalues must be less than 1 for stability, and
the largest eigenvalue determines the overall convergence rate (the largest eigenvalue is
called the spectral radius rs ). For a large N × N square lattice problem, the spectral
radius is asymptotically equal to [Ames, 1992]
π2
rJacobi ≃ 1 − . (8.62)
2N 2
Therefore, reducing the error by a factor of 10 requires − ln 10/ ln rs ≃ N 2 steps. In the
Gauss-Seidel method, the lower-triangular part is moved over to the left side:
(L + D) · ~un+1 = −U · ~un + ρ~ . (8.63)
For the square 2D lattice, this has a spectral radius of
π2
rGauss−Seidel ≃ 1 − , (8.64)
N2
104 Finite Differences: Partial Differential Equations DRAFT

and so the number of steps needed to reduce the error by a factor of 10 is half that required
by the Jacobi method. For a 100 × 100 lattice, both of these methods require ∼ 104 steps
for an improvement of a factor of 10 in the answer, which is usually prohibitive. The
Gauss-Seidel method is preferable to the Jacobi method because it converges faster and
does not require auxiliary storage, but something better than both is needed.
The Gauss-Seidel method can be rewritten in a suggestive form as follows:
(L + D) · ~un+1 = −U · ~un + ρ~
~un+1 = (L + D)−1 [−U · ~un + ρ
~]
= ~un − (L + D)−1 · [U · ~un − ρ~] − ~un
= ~un − (L + D)−1 · [(L + D + U) · ~un − ρ
~]
= ~un − (L + D)−1 · [A · ~un − ρ
~]
−1 ~
= ~un − (L + D) · En , (8.65)
where E~ n is the error at the nth time step. In each update, the error gets multiplied
by (L + D)−1 and subtracted from the state. The idea of Successive Over-Relaxation
(SOR) is to extrapolate this correction and subtract a larger change
~n
~un+1 = ~un − α(L + D)−1 · E . (8.66)
It can be shown that this converges for 0 < α < 2 [Ames, 1992]. When α = 1 this is just
the Gauss–Seidel method, α < 1 is underrelaxation (which slows the convergence), and
1 < α < 2 is overrelaxation. The convergence rate depends on the value of α; choosing
a value that is too large is as bad as choosing one that is too small because the solution
will overshoot the final value. The optimal relaxation rate is
2
α= q , (8.67)
1+ 1 − ρ2Jacobi

which leads to an asymptotic spectral radius of

2π
rSOR ≃ 1 − . (8.68)
N
This reduces the number of steps needed to reduce the error by a factor of 10 to O(N ),
which is now proportional to the grid size rather than the square of the grid size. Written
out in components, for the 2D case SOR is
α n α(∆x)2
un+1 n
j,k = (1 − α)uj,k + (uj+1,k + un+1
j−1,k + u n
j,k+1 + u n+1
j,k−1 ) − ρj,k (8.69)
4 4
(remember that this is not an implicit method, because the forward time steps on the
right-hand side come from preceeding spatial updates).
SOR is very easy to program, but does require determining the relaxation parameter
α (although this can be estimated empirically, since if α is too large the solution will
oscillate). An alternative is to use ADI, which permits larger time steps at the expense
of algorithm complexity. For large problems that require repeated fast solution both
techniques have been superseded by multigrid methods [Press et al., 2007], which find
the final solution on N grid points in O(N ) steps. These methods are based on iteratively
coarse-graining the problem to produce a simpler one that can be solved quickly, and
 DRAFT 8.4 Problems 105

then interpolating to find the approximate solution at higher resolution. This is analogous
to the Richardson extrapolation methods for ODEs, but because of the extra dimensions
they are more complicated to implement.
Finally, for a problem such as weather forecasting, the simulation must contend with
not just numerical errors but also uncertainty in the initial conditions and the model
selection. Although these can’t be eliminated, they can be reduced by combining predic-
tions from ensembles of models [Krishnamurti et al., 1999]. This theme will return in
Part Three.

8.4 S E L E C T E D R E F E R E N C E S
[Ames, 1992] Ames, William F. (1992). Numerical Methods for Partial Differential
Equations. 3rd edn. Boston, MA: Academic Press.
Ames is a classic reference for numerical methods for PDEs, with a bit more
emphasis on mathematical rigor than on practical advice.

8.5 P R O B L E M S

(8.1) Consider the 1D wave equation

∂2u 2
2∂ u
= v . (8.70)
∂t2 ∂x2
(a) Write down the straightforward finite-difference approximation.
(b) What order approximation is this in time and in space?
(c) Use the von Neumann stability criterion to find the mode amplitudes.
(d) Use this to find a condition on the velocity, time step, and space step for stability
(hint: consider the product of the two amplitude solutions).
(e) Do different modes decay at different rates for the stable case?
(f) Numerically solve the wave equation for the evolution from an initial condition
with u = 0 except for one nonzero node, and verify the stability criterion.
(g) If the equation is replaced by
∂2u 2
2∂ u ∂ ∂2u
= v + γ , (8.71)
∂t2 ∂x2 ∂t ∂x2
assume that
u(x, t) = Aei(kx−ωt) (8.72)
and find a relationship between k and ω, and simplify it for small γ. Comment
on the relationship to the preceeding question.
(h) Repeat the numerical solution of the wave equation with the same initial con-
ditions, but include the damping term.
(8.2) Write a program to solve a 1D diffusion problem on a lattice of 500 sites, with an
initial condition of zero at all the sites, except the central site which starts at the
value 1.0. Take D = ∆x = 1, and use fixed boundary conditions set equal to zero.
106 Finite Differences: Partial Differential Equations DRAFT

(a) Use the explicit finite difference scheme, and look at the behavior for ∆t = 1,
0.5, and 0.1. What step size is required by the Courant condition?
(b) Now repeat this using implicit finite differences and compare the stability.
(8.3) Use ADI to solve a 2D diffusion problem on a lattice, starting with randomly seeded
values.
(8.4) Use SOR to solve Laplace’s equation in 2D, with boundary conditions uj,1 = u1,k =
0, uN,k = −1, uj,N = 1, and explore how the convergence rate depends on α, and
how the best choice for α depends on the lattice size.