Numerical Differentiation and integration

Calculus is the mathematics of change. Because engineers must continuously deal with terms and processes
that change, calculus is an essential tool of our profession. Standing at the heart of calculus are the related
mathematical concepts of differentiation and integration.
The mathematical definition of the derivative begins with a difference approximation:
y f xi  x   f xi 

x x
if ∆x is allowed to approach zero, the difference becomes a derivative:
dy f xi  x   f xi 
 lim .
dx x 0 x
The inverse process to differentiation in calculus is integration. Mathematically integration is represented by

I   f x dx
b

which stands for the integral of the function f(x) with respect to the independent variable x evaluated
between the limits x = a to x = b. The integration is the total value or the summation of f(x)dx over the range
x=a to x=b.
1. The Taylor Series:
Taylor’s theorem and its associated formula, the Taylor series, is of a great value in the study of numerical
methods. In essence the Taylor series provides a means to predict a function value at one point in terms of
the function value and its derivative at another point. In particular, the theorem states that any smooth
function can be approximated as a polynomial.
𝑓 ′′ (𝑥𝑖 ) 𝑓 ′′′ (𝑥𝑖 ) 𝑓 (∞ ) (𝑥𝑖 )
𝑓(𝑥𝑖+1 ) = 𝑓(𝑥𝑖 ) + 𝑓 ′ (𝑥𝑖 )(𝑥𝑖+1 − 𝑥𝑖 ) + 2
(𝑥𝑖+1 − 𝑥𝑖 ) + 3
(𝑥𝑖+1 − 𝑥𝑖 ) + ⋯ + (𝑥𝑖+1 − 𝑥𝑖 )∞
2! 3! ∞!
A useful way to gain insight into the Taylor series is to build it term by term. For example, the first term in
the series is 𝑓(𝑥𝑖+1 ) ≅ 𝑓(𝑥𝑖 ). This relationship, called zero-order approximation, indicates that the value
of 𝑓 at the new point is the same as its value at the old point. This result makes intuitive sense because if 𝑥𝑖
and 𝑥𝑖+1 are close to each other, or the new value is similar to the old value (constant).
The first-order approximation is developed by adding another term to yield:
𝑓(𝑥𝑖+1 ) ≅ 𝑓(𝑥𝑖 ) + 𝑓 ′ (𝑥𝑖 )(𝑥𝑖+1 − 𝑥𝑖 )
The additional term is consist of a slope f’(xi) multiplied by the distance between xi and xi+1. Thus, the
expression is now in the form of a straight line.
In a similar manner, additional terms can be included to develop the complete Taylor series:
𝑓 ′′ (𝑥𝑖 ) 𝑓 ′′′ (𝑥𝑖 ) 𝑓 (𝑛) (𝑥𝑖 )
𝑓(𝑥𝑖+1 ) = 𝑓(𝑥𝑖 ) + 𝑓 ′ (𝑥𝑖 )(𝑥𝑖+1 − 𝑥𝑖 ) + (𝑥𝑖+1 − 𝑥𝑖 )2 + (𝑥𝑖+1 − 𝑥𝑖 )3 + ⋯ + (𝑥𝑖+1 − 𝑥𝑖 )𝑛
2! 3! 𝑛!
+ 𝑅𝑛
The remainder (Rn) term is included to account for all terms n+1 to infinity: Rn  O(h n 1 ) which means that

the truncation error is of the order of ℎ𝑛+1. That is the error is proportional to the step size h raised to the
(𝑛 + 1)𝑡ℎ power. For example, if the error is O(h) halving the step size will halve the error. On other hand,

1
if the error is O(h2), halving the step size will quarter the error. To simplify the Taylor series, let us define
the step size h = xi+1 –xi :
𝑓 ′′ (𝑥𝑖 ) 2 𝑓 ′′′ (𝑥𝑖 ) 3 𝑓 (𝑛) (𝑥𝑖 ) 𝑛
𝑓(𝑥𝑖+1 ) = 𝑓(𝑥𝑖 ) + 𝑓 ′ (𝑥𝑖 )ℎ + ℎ + ℎ +⋯+ ℎ + 𝑅𝑛 and Rn  O(h n 1 )
2! 3! 𝑛!
EX: use zero to forth order Taylor series expansions to approximate the function:
f(x) = -0.1 x4-0.15x3-0.5x2-0.25x+1.2
from xi=0 with h = 1 . That is to predict the function's value at xi+1 = 1.
Solution:
Because we know the function f(0) =1.2 and f(1) = 0.2. Thus the true value try to predict is 0.2.The Taylor
series approximation with n=0 is f(xi+1)≈ 1.2. The zero order approximation is a constant the resulting
error Et=0.2 -1.2 =-1.
f(x)
For n=1 the first derivative must be determined and evaluated at x=0 f(xi)
Zero order
′ (𝑥) 3 2
𝑓 = 0.4𝑥 − 0.45𝑥 − 𝑥 − 0.25 First ord
1.0 er
𝑓′(0) = −0.4(0.0)3 – 0.45(0.0)2 – 1.0(0.0) − 0.25 = −0.25

se
co
nd
Therefore the first approximation is:

or
de
0.5

r
𝑓(𝑥𝑖+1 ) = 𝑓(1) ≈ 1.2 − 0.25 ℎ f(xi+1)

which can be used to compute f(1)= 0.95 Et = 0.2 -0.95 = -0.75.

xi=0 xi+1=1
0 h x
For n = 2 the second derivative is evaluated at x = 0 :
𝑓 ′′ (𝑥) = 1.2𝑥 2 − 0.9𝑥 − 1 ⇒ 𝑓"(0) = 1.2(0)2 – 0.9(0)– 1.0 = −1.0
Therefore, f(xi+1) ≈ 1.2 – 0.25h – 0.5 h2 and substituting h = 1, f(1) = 0.45. Et= 0.2 – 0.45 = -0.25.
In the same way the third and fourth derivative results in exactly the same equation we started with :
f(x) = 1.2 - 0.25 h – 0.5 h2 – 0.15 h3 – 0.1 h4
Where Rn  O(h n 1 ) =0. Because the fifth derivative of a forth order polynomial is zero. Consequently, the
Taylor series expansion to the fourth derivative yields an exact estimate at xi+1=1:
f(1) = 1.2 – 0.25(1) – 0.5(1)2 – 0.15 (1)3 -0.1 (1)4 = 0.2
2. Numerical Differentiation
It can represent the first derivative from the Taylor series as
f xi 1   f xi  f
f ' ( xi )   Oxi 1  xi  or f ' xi   i  Oh  .
xi 1  xi h
Where ∆fi is referred to as the first difference and h is called the step size. It is termed forward difference
because it utilizes data at 𝑖 and 𝑖 + 1 to estimate the derivative. The term ∆f / h is referred to as a first finite
divided difference.
Backward Difference Approximation of the first derivative: the Taylor series can be expand backward to
calculate a pervious value on the basis of a present value as in:
f " xi  2
f xi 1   f ( xi )  f ' xi h  h  .
2!
Truncating this equation after first derivative and rearranging yields:

2
 f xi   f xi 1  f 1
f ' ( xi )  
h h
where the error is O(h) and f i is referred to as the first backward difference.
Centered Difference Approximation of the First Derivative: a third way to approximate first derivative is to
subtract backward Taylor series expansion:
f " xi  2
{ f xi 1   f ( xi )  f ' xi h  h   }
2!
from the forward Taylor series expansion:
f " xi  2
{ f xi 1   f ( xi )  f ' xi h  h  }
2!
to yield:
f 3   x i  3
f xi 1   f xi 1   2 f ' xi h  2 h 
3!
which can be solved for
f xi 1   f xi 1  f 3 xi  2
f ' xi    h  
2h 6
f xi 1   f xi 1 
f ' xi  
2h
 
 O h2

EX: use forward and backward difference approximation of O(h) and a centered difference approximation
of O(h2) to estimate the first derivative of :
f(x) = -0.1 x4-0.15x3-0.5x2-0.25x+1.2
at x = 0.5 using step size h = 0.5. Repeat the computation using h = 0.25. Note that the derivative can be
calculated directly as:
f'(x) = -0.4 x3-0.45x2-1x-0.25
and can be used to compute the true value as f'(0.5) = -0.9125.
Solution: for h = 0.5 the function can be employed to determine
xi-1 = 0 f(xi-1) = 1.2
xi = 0.5 f(xi) = 0.925
xi+1 = 1.0 f(xi+1) = 0.2
These values can be used to compute the forward divided difference
0.2  0.925
f ' 0.5   1.45  t  58.9%
0.5
0.925  1.2
The backward divided difference f ' 0.5   0.55  t  39.7%
0.5
0.2  1.2
And the centered divided difference f ' 0.5   1  t  9.6%
1
for h = 0.25 the function can be employed to determine
xi-1 = 0.25 f(xi-1) = 1.10351563

3
 xi = 0.5 f(xi) = 0.925
xi+1 = 0.75 f(xi+1) = 0.63632813
These values can be used to compute the forward divided difference:
0.63632813  0.925
f ' 0.5   1.155  t  26.5%
0.25
0.925  1.10351563
The backward divided difference f ' 0.5   0.714  t  21.7%
0.25
0.63632813  1.10351563
And the centered divided difference f ' 0.5   0.934  t  2.4%
0.5
For both step sizes, the centered difference approximation is more accurate than forward or backward
differences. Also, as predicted by the Taylor series analysis halving the step size approximately halves the
error of the backward and forward differences and quarters the error of the centered difference.
Finite Difference Approximations of Higher Derivatives: besides first derivatives the Taylor series
expansion can be used to derive numerical estimates of higher derivatives. To do this, we write a forward
Taylor series expansion for f(xi+2) in terms of f(xi):
f "xi 
f xi 2   f ( xi )  f ' xi 2h  2h2   .
2!
f " xi  2
Multiplying the forward Taylor series expansion{ f xi 1   f ( xi )  f ' xi h  h   }by 2 and
2!
subtracted from the previous equation to give:
f xi  2   2 f xi 1    f xi   f " ( xi )h 2  
which can be solved
f xi  2   2 f xi 1   f xi 
f " ( xi )   Oh 
h2
This relationship is called the second forward finite divided difference. Similar manipulations can be
employed to derive a backward version
f xi   2 f xi 1   f xi  2 
f " ( xi )   Oh 
h2
And centered version
f xi 1   2 f xi   f xi 1 
f " ( xi )  2
 
 O h2
h
notice also that the centered version can be Alternatively expressed as:
f xi 1   f xi  f ( xi )  f xi 1) 

f " ( xi )  h h  O h2  
h
Thus, just as the second derivative is a derivative of a derivative, the second divided difference
approximation is a difference of two first divided differences.

4
2.1 High-Accuracy Differentiation Formulas:
High accuracy divided difference formula can be generated by including additional terms from the Taylor
series expansion. For example, the forward Taylor series expansion can be written as:
f " xi  2
f xi 1   f ( xi )  f ' xi h  h  
2
which can be solved for
f xi 1   f xi  f " ( xi )
f ' ( xi ) 
h

2
 
h  O h2

f xi  2   2 f xi 1   f xi 

Where f " ( xi )   Oh  .
h2
Substituted this equation to the previous one yield:
f xi 1   f xi  f xi  2   2 f xi 1   f xi   f xi  2   4 f xi 1   3 f xi 
f ' ( xi ) 
h
 2
h  O h2    2h
 
 O h2
2h
Similar improved version can be developed for the backward and centered formulas as well as for
approximation of higher derivative.
EX: estimate the first derivative of f(x) = -0.1 x4-0.15x3-0.5x2-0.25x+1.2 at x = 0.5 using step size h = 0.25.
Solution: the data needed for the example are
xi-2 = 0 f(xi-2) = 1.2
xi-1 = 0.25 f(xi-1) = 1.10351563
xi = 0.5 f(xi) = 0.925
xi+1 = 0.75 f(xi+1) = 0.63632813
xi+2 = 1.0 f(xi+2) = 0.2
The forward difference of accuracy O(h2) is computed as
 0.2  40.6363281  3(0.925)
f ' (0.5)   0.859375  t  5.82%
2(0.25)
The backward difference of accuracy O(h2) is computed as
 30.925  41.1035156  1.2
f ' (0.5)   0.878125  t  3.77%
2(0.25)
The centered difference of accuracy O(h4) is computed as
 0.2  80.6363281  8(1.035156)  1.2
f ' (0.5)   0.9125  t  0%
12(0.25)
As expected, the errors for the forward and backward differences are more accurate than the results from the
previous example. However, the centered difference yields a perfect result.
3. Numerical Integration:
The Newton-Cotes formulas are the most common numerical integration schemes. They are based on the
strategy of replacing a complicated function of tabulated data with an approximating function that is easy to
integrate:

5
 I   f x dx   f n x dx
b b f (x)
a a

where fn(x) = a polynomial of the form:

fn(x) = a0 + a1 x + ……..+ an-1xn-1 + an xn
where n is the order of the polynomial. For example in the
figure (a), a first order polynomial ( a straight line) is used as
a b x a b x
an approximation. In (b), a parabola is employed for the same (a) (b)

Straight Line Single Parabola

purpose.
3.1 The Trapezoidal Rule:
The trapezoidal rule is the first of the Newton-Cotes closed integration formulas. It corresponds to the case
where the polynomial is first order:

I   f x dx   f1 x dx .
b b

a a

A straight line can be represented as:

f b   f a 
f 1 x   f a   x  a  .
ba
The area under this straight line is an estimate of the f(x) between the limits a and b:

 f b   f a 
x  a dx
b
I    f a  
a 
ba 
. the result of the integration is:
f a   f b 
I  b  a 
2
which is called the trapezoidal rule.
Geometrically the trapezoidal rule is equivalent to approximating the area of the trapezoid under the straight
line connecting f(a) and f(b) as shown in the figure. The formula for computing the area of a trapezoidal is
the height times the average of the base. Therefore the integral estimate can be represented as
I ≈ width * average height ≈ (b-a)* average height.
f (x)
3.1.1 Error of the Trapezoidal Rule
When we employ the integral under a straight line segment to F(a)
approximate the integral under a curve, we obviously can incur an
error as shown in previous figure. F(a)

EX: estimate the integral for the

f(x) = 400x5 – 900 x4+675 x3 - 200x2 + 25x+0.2
from a = 0 to b = 0.8. a b x
(a)
Solution: the function values f(0) = 0.2 f(0.8) = 0.232
Graphical depiction of the trapezoidal rule
0.2  0.232
I  0.8  0.1728
2
which represents an error of :

6
Et = 1.640533 – 0.1728 = 1.467733  t  89.5%
F(x2) F(x3)
f (x) F(x1) F(x4)
3.1.2 The Multiple-Application Trapezoidal Rule: F(xn-2)
F(x0) F(xn-3)
F(x5) F(xn-1)
One way to improve the accuracy of the trapezoidal
F(xn)
rule is to divide the integration interval from a to b
into a number of a segments and apply the method to
each segment. The areas of individual segments can
then be added to yield the integral for the entire
interval. The resulting equations are called multiple- x0 x1 x2 x3 x4 x5 xn-3 xn-2 xn-1 xn
x
application integration formulas. ba
h
x0 = a n xn = b
The previous figure shows the general format and nomenclature we will use to characterize multiple
application integrals. There are n+1 equally spaced base points(x0, x1, x2,……..xn). consequently, there are n
ba
segments of equal width: h  the total integral can be represented as:
n

I   f x dx   f x dx     f x dx .

x1 x2 xn

x0 x1 xn 1

Substituting the trapezoidal rule for each integral yields:

f x0   f x1  f x1   f x2  f xn1   f xn  h  n 1

I h h  h   f x0   2 f ( xi )  f xn  .
2 2 2 2 i 1 
EX: use the two segment trapezoidal rule to estimate the integral of :
f(x) = 400x5 – 900 x4+675 x3 - 200x2 +25x+0.2
from a = 0 to b = 0.8. The correct answer for the integral is 1.640533.
Solution: n=2 (h = 0.4 ) f(0) = 0.2 f(0.4) = 2.456 f(0.8) = 0.232
0.2  22.456  0.232
I  0.8  1.0688
4
Et = 1.640533 – 1.0688 = 0.57173  t  34.9%

3.1.3 Computer Algorithms for the Trapezoidal Rule:

Function Trapm(h,n,f)
sum = f0
DO I = 1,n-1
sum = sum + 2*fi
END DO f (x) f (x)
sum = sum + fn
Trapm = h* sum /2
END Trapm
3.2 Simpson's Rule:
Aside from applying the trapezoidal rule with
finer segmentation, another way to obtain a
more accurate estimate of an integral is to
x x
extra point midway between f(a) and f(b), the (a) (b)

three points can be connected with parabola. If Simpson's 1/3 rule Simpson's 3/8 rule

7
there are two points equally spaced between f(a) and f(b), the four points can be connected with third order
polynomial. The formulas that result from taking the integrals under these polynomials are called Simpson's
rules.
3.2.1 Simpson's 1/3 rule:
Simpson's 1/3 rule results when a second order interpolating polynomial is substituted into equation:

I   f ( x)dx   f 2 x dx
b b

a a

If a and b are designated as x0 and x2 and f2(x) is represented by a second-order Lagrange polynomial, the
integral becomes:
x2   x  x  x  x 
I   1 2
f  x0  
x  x0 x  x2  f x   x  x0 x  x1  f x  dx
x0  x  x  x  x 
 0 1 0 2 x1  x0 x1  x2  1 x2  x0 x2  x1  2 
After integration and algebraic manipulation, the following formula results:

I
h
 f x0   4 f x1   f x2  .
3
The label "1/3" stems from the fact that h is divided by 3 in previous equation.
f x 0   4 f x1   f x 2 
I  b-a 
 6 
Width  
Average height

EX: : use the Simpson's 1/3 rule to estimate the integral of

f(x) = 400x5 – 900 x4+675 x3 - 200x2 +25x+0.2
from a = 0 to b = 0.8. The correct answer for the integral is 1.640533.
Solution: f(0) = 0.2 f(0.4) = 2.456 f(0.8) = 0.232 Therefore,
0.2  42.456  0.232
I  0.8
6
which represent an exact error Et = 1.640533 – 1.367467 = 0.27367
3.2.2 The Multiple-Application Simpson's 1/3 Rule:
Just as with the trapezoidal rule, Simpson's rule can be improved by dividing the integration interval into a
ba
number of segments of equal width h  . The total integral can be represented as:
n

I   f x dx   f x dx     f x dx .

x2 x4 xn

x0 x2 xn  2

Substituting Simpson's 1/3 rule for the individual integral yields:

f x0   4 f x1   f x 2  f x 2   4 f x3   f x 4  f x n  2   4 f x n 1   f x n 
I  2h  2h    2h .
6 6 6
Or combining terms:
n 1 n2
f x 0   4  f xi   2  f x   f x 
j n

I  b  a 
i 1, 3, 5 j  2, 4, 6
 3n
Width 
Average height

8
EX: use The Multiple-Application Simpson's 1/3 Rule with n=4 to estimate the integral of:
f(x) = 400x5 – 900 x4+675 x3 - 200x2 +25x+0.2
from a = 0 to b = 0.8. The correct answer for the integral is 1.640533.
Solution: n=4 (h = 0.2 ) f(0) = 0.2 f(0.2) = 1.288 f(0.4) = 2.456 f(0.6) = 3.464 f(0.8) = 0.232
0.2  41.288  3.464  22.456  0.232
I  0.8  1.623467
12
Et = 1.640533 – 1.623467 = 0.017067  t  1.04%
F(x)

The pervious example illustrates that the multiple- application version of

Simpson's 1/3 rule yields very accurate results. For this reason, it is
considered superior to the trapezoidal rule for most applications. However,
as mentioned previously it is limited to cases where the values are
equispaced. Further, it is limited to situations where there are an even
number of segments and an odd number of points. As discussed in next
section, an odd-segment-even point formula known as Simpson’s 3/8 rule
x
is used in conjunction with 1/3 rule to permit evaluation of both even and
odd numbers of segments. 1/3 rule 3/8 rule

3.2.3 Simpson’s 3/8 Rule

In similar manner to the derivation of the trapezoidal and Simpson’s 1/3 rule, a third order Lagrange
polynomial can be fit to four points and integrated:

I   f ( x)dx   f 3 x dx
b b

a a

to yield

I
3h
 f x0   3 f x1   3 f x2   f ( x3 )
8
the 3/8 rule can also be expressed in the form:
f x 0   3 f x1   3 f x 2   f ( x3 )
I  b-a 
 8 
Width   
Average height