Numerical Differentiation

Introduction
•  Some numerical algorithms require
evaluation of the derivative of a function.

•  When the function is difficult to

differentiate analytically or when the
function is given empirically as in table of
values, approximate calculation of the
derivative is necessary.
 Approximating the derivative
The definition of the derivative is, the limit as
h→0 of
f (x + h ) − f (x )
f ' (x ) ≅ (1)
h
practically suggests the program to pick a
small value of h.
Example. Compute the derivative of f(x)=ex
at x=1.
 Limit of a difference quotient
Derivative of f(x)=ex at x=1 for different values of h.
 Approximating the derivative
•  When h is large, the difference quotient is
the slope of the secant line joining two
points that are not close enough to each
other.
•  When h is too small, the difference
quotient can exhibit the problem of loss of
significance due to the subtraction of
quantities that are nearly equal.
 Approximating the derivative
•  To determine the best approximation
before the terms start to move away from
the limit is to compute Dk until

DN +1 − DN ≥ DN − DN −1
•  In the example,
0.0007 = D6 − D5 > D5 − D4 = 0.00012
 Sources of error
•  The truncation error εt comes from higher
terms in the Taylor series expansion:
f (x + h ) − f (x ) 1
f '= − hf "− !
h 2
•  Round-off error εr must be expected since
real numbers don’t have exact
representation in binary.
 Central-difference formulas
•  Assume that f ∈C3[a,b] and that x-h, x, x+h
∈[a,b]. Then
f (x + h ) − f (x − h )
f ' (x ) ≈ + εt ( 2)
2h
where, there exists a number c=c(x) ∈[a,b]
such that
2 (3 )
h f (c )
εt = − = O(h )
2

6
 Centered-formula of order O(h4)
Assume that f ∈C5[a,b] and that x-2h, x-h, x x
+h, x+2h ∈[a,b]. Then
− f ( x + 2h ) + 8 f ( x + h ) − 8 f ( x − h ) + f ( x − 2h )
f ' (x ) ≈ + εt
12h
(3)
where 4 (5 )
h f (c )
εt = −
30
=O h 4
( )
 Example
Let f(x)=cos(x). Calculate approximations for
f’(0.8) using Eqs. (2) and (3) with step
sizes h=0.1, 0.01, 0.001, 0.0001 and
compare with true value f’(0.8) = -sin(0.8).

Carry out nine decimal places in

calculations.
 Example
Eq. (2)
f ( x + h ) − (x − h )
f ' (x ) ≈
2h
Eq. (3)
− f ( x + 2h ) + 8 f (x + h ) − 8 f ( x − h ) + f ( x − 2h )
f ' (x ) ≈
12h
 Differentiating an
Interpolating Polynomial
•  The Lagrange quadratic polynomial P2(x)
that passes through x=0.7,0.8, and 0.9.
P2 (x ) = 1.046875165 − 0.159260044x − 0.348063157x 2

•  Differentiating

P2 ' (0.8) = −0.716161095

 Differentiating an
Interpolating Polynomial
•  The Lagrange quadric polynomial P4(x)
that passes through x=0.6,0.7,0.8, 0.9 and
1.0.
P4 (x ) = 0.998452927 + 0.00963839 x − 0.523291341x 2
+ 0.026521229 x 3 + 0.028981100 x 4

•  Differentiating
P4 ' (0.8) = −0.717353703
 Differentiation using Limits
To approximate f ’(x) numerically, generate
a sequence

f ' (x ) ≈ Dk =
( −k
) ( −k
f x + 10 h − f x − 10 h )
( )
−k
2 10 h

for k=0,..,n until |Dn+1 – Dn| ≥ |Dn-1 – Dn| or |

Dn – Dn-1|≤ tolerance.
 Richardson’s Extrapolation
Let
f k = f (xk ) = f (x0 + kh )
and D0(h) and D0(2h) denote
approximations to f ’(x0) that are obtained
using Eq. (2) with step sizes h and 2h.
2
f ' (x0 ) ≈ D0 (h) + Ch
f ' (x0 ) ≈ D0 (2h) + 4Ch 2 (4)
 Richardson’s Extrapolation
Manipulating these two equations, we obtain
4 D0 (h ) − D0 (2h ) − f 2 + 8 f1 − 8 f −1 + f −2
f ' (x0 ) ≈ =
3 12h
which is the central difference formula Eq.
(3).

This method of obtaining a formula for f ’(x0)

of higher order from a formula of lower
order is called extrapolation.
 Richardson’s Extrapolation
Suppose that two approximations of order
O(h2k) for f ’(x0) are Dk-1(h) and Dk-1(2h)
and that they satisfy
2k 2 k +2
f ' (x0 ) ≈ Dk −1 (h ) + c1h + c2 h +!
f ' (x0 ) ≈ Dk −1 (2h ) + 4k c1h 2 k + 4k +1 c2 h 2 k +2 + !
Then an improved approximation is
4k Dk −1 (h) − Dk −1 (2h)
(
f ' (x0 ) = Dk (h ) + O h 2 k +2 ) =
4k − 1
+ O h(2 k +2
)
 Example
Let
f(x)=cos(x).
Use Eq. (4) with h=0.01 to show how the
linear combination (4D0(h) -D0(2h))/3 can
be used to obtain the approximation to
f’(0.8).
Carry nine decimal places in all
calculations.
 Example
Using Eq. (4), we obtain
f (0.81) − f (0.79 )
D (h ) ≈
0 = −0.717344150
0.02
f (0.82 ) − f (0.78 )
D0 (2h ) ≈ = −0.717308275
0.04
The linear combination becomes
4 D0 (h ) − D0 (2h )
f ' (0.8) ≈ = −0.717356108
3
 Higher Derivatives
•  Taylor series can be used to obtain
central-difference formulas for higher
derivatives.

•  The popular choices are those of order

O(h2) and O(h4).
 Higher Derivatives
To derive f ”(x) of order O(h2), we write
h 2 f " (x ) h3 f (3) (x )
f (x + h ) = f (x ) + h ⋅ f ' (x ) + + +!
2 6
h 2 f " (x ) h3 f (3) (x )
f (x − h ) = f (x ) − h ⋅ f ' (x ) + − +!
2 6
adding
2h 2 f " (x ) 2h 4 f (4 ) (x )
f (x + h ) + f (x − h ) = 2 f (x ) + + +!
2 24
 Higher Derivatives
Solving for f ”(x) yields
f (x + h ) − 2 f (x ) + f (x − h )
f " (x ) = 2
h
2h 2 f (4 ) (x ) 2h 4 f (6 ) (x ) 2h 2k −2 f (2k ) (x )
− − −!− −!
4! 6! (2k )!
If truncated at the fourth derivative
2 (4 )
f1 − 2 f 0 + f −1 h f (c )
f " (x ) = 2
−
h 12
Central difference formulas
of Order O(h2)
f1 − f −1
f ' (x0 ) ≈
2h
f1 − 2 f 0 + f −1
f " (x0 ) ≈ 2
h
(3 ) f 2 − f1 + 2 f −1 − f −2
f (x0 ) ≈ 3
2h
(4 ) f 2 − 4 f1 + 6 f 0 − 4 f −1 + f −2
f (x0 ) ≈ 4
h
 Central difference formulas
of Order O(h4)
− f 2 + 8 f1 − 8 f −1 + f −2
f ' (x0 ) ≈
12 h
− f 2 + 16 f1 − 30 f 0 + 16 f −1 − f −2
f " (x0 ) ≈
12 h 2
(3 ) − f 3 + 8 f 2 − 13 f1 + 13 f −1 − 8 f −2 + f −3
f (x0 ) ≈ 3
8h
(4 ) − f 3 + 12 f 2 − 39 f1 + 56 f 0 − 39 f −1 + 12 f −2 − f −3
f (x0 ) ≈
6h 4
 Example
Let f(x)=cos(x). Use the formula
f1 − 2 f 0 + f −1
f " (x ) = 2
h
with h=0.1, 0.01, and 0.001 to find
approximations to f ”(0.8).
Compare with the true value of f ”(0.8) =
-cos(0.8).
 Example
•  The calculation for h=0.01 is
f (0.81) − 2 f (0.80 ) + f (0.79 )
f " (0.8) ≈ = −0.696690000
0.0001

with an error of -0.000016709

•  The other calculations are summarized:
 Exercise 1
Let f(x)=sin(x), where x is measured in
radians.
(a) Calculate approximations to f ’(0.8)
using Eq.(3) with h=0.1, 0.01, and
compare with f ’( 0.8)=cos(0.8).
(b) Use extrapolation formula to compute
the approximation to f ’( 0.8)
 Exercise 2
Use numerical differentiation formula

− f 2 + 16 f1 − 30 f 0 + 16 f −1 − f −2
f " (x0 ) ≈
12h 2

and h=0.1 to approximate f ”(1) for the

functions
(a) f(x) = x2 (b) f(x) = x4