CISE-301: Numerical Methods

Topic 1:
Introduction to Numerical Methods and Taylor Series
Lectures 1-4:

CISE301_Topic1 1
 Lecture 1
Introduction to Numerical Methods

 What are NUMERICAL METHODS?

 Why do we need them?
 Topics covered in CISE301.

Reading Assignment: Pages 3-10 of textbook

CISE301_Topic1 2
Numerical Methods
Numerical Methods:
Algorithms that are used to obtain numerical
solutions of a mathematical problem.
Why do we need them?
1. No analytical solution exists,
2. An analytical solution is difficult to obtain
or not practical.

CISE301_Topic1 3
What do we need?
Basic Needs in the Numerical Methods:
 Practical:
Can be computed in a reasonable amount of time.
 Accurate:
 Good approximate to the true value,
 Information about the approximation error
(Bounds, error order,… ).

CISE301_Topic1 4
Outlines of the Course
 Taylor Theorem  Solution of linear
 Number Equations
Representation  Least Squares curve
 Solution of nonlinear fitting
Equations  Solution of ordinary
 Interpolation differential equations
 Numerical  Solution of Partial
Differentiation differential equations
 Numerical Integration

CISE301_Topic1 5
Solution of Nonlinear Equations
 Some simple equations can be solved analytically:
x2  4x  3  0
4 4 2  4(1)(3)
Analytic solution roots 
2(1)
x  1 and x  3

 Many other equations have no analytical solution:

x 9  2 x 2  5  0

x  No analytic solution
xe 

CISE301_Topic1 6
Methods for Solving Nonlinear
Equations

o Bisection Method

o Newton-Raphson Method

o Secant Method

CISE301_Topic1 7
Solution of Systems of Linear
Equations
x1  x2  3
x1  2 x2  5
We can solve it as :
x1  3  x2 , 3  x2  2 x 2  5
 x2  2, x1  3  2  1
What to do if we have
1000 equations in 1000 unknowns.
CISE301_Topic1 8
Cramer’s Rule is Not Practical
Cramer' s Rule can be used to solve the system :
3 1 1 3
5 2 1 5
x1   1, x2  2
1 1 1 1
1 2 1 2

But Cramer' s Rule is not practical for large problems.

To solve N equations with N unknowns, we need (N  1)(N  1)N!
multiplications.
To solve a 30 by 30 system, 2.3 1035 multiplications are needed.
A super computer needs more than 1020 years to compute this.
CISE301_Topic1 9
Methods for Solving Systems of Linear
Equations
o Naive Gaussian Elimination

o Gaussian Elimination with Scaled

Partial Pivoting

o Algorithm for Tri-diagonal Equations

CISE301_Topic1 10
Curve Fitting
 Given a set of data:

x 0 1 2
y 0.5 10.3 21.3

 Select a curve that best fits the data. One

choice is to find the curve so that the sum
of the square of the error is minimized.

CISE301_Topic1 11
Interpolation
 Given a set of data:

xi 0 1 2
yi 0.5 10.3 15.3

 Find a polynomial P(x) whose graph

passes through all tabulated points.

yi  P( xi ) if xi is in the table
CISE301_Topic1 12
Methods for Curve Fitting
o Least Squares
o Linear Regression
o Nonlinear Least Squares Problems

o Interpolation
o Newton Polynomial Interpolation
o Lagrange Interpolation

CISE301_Topic1 13
Integration
 Some functions can be integrated
analytically:

3 3
1 2 9 1
1 xdx  2 x 1  2  2  4
But many functions have no analyticalsolutions :
a

e
 x2
dx  ?
0

CISE301_Topic1 14
Methods for Numerical Integration
o Upper and Lower Sums

o Trapezoid Method

o Romberg Method

o Gauss Quadrature

CISE301_Topic1 15
Solution of Ordinary Differential Equations
A solution to the differential equation :
x(t )  3x (t )  3x(t )  0
x (0)  1; x(0)  0
is a function x(t) that satisfies the equations.

* Analytical solutions are available for

special cases only.
CISE301_Topic1 16
Solution of Partial Differential Equations
Partial Differential Equations are more
difficult to solve than ordinary differential
equations:

2 2
 u  u
2
 2
20
x t
u (0, t )  u (1, t )  0, u ( x,0)  sin(x)

CISE301_Topic1 17
Summary
Topics Covered in the Course
 Numerical Methods:
 Solution of Nonlinear Equations
Algorithms that are
used to obtain  Solution of Linear Equations
numerical solution of a  Curve Fitting
mathematical problem.
 Least Squares
 We need them when
No analytical solution
 Interpolation
exists or it is difficult  Numerical Integration
to obtain it.  Numerical Differentiation
 Solution of Ordinary Differential
Equations
 Solution of Partial Differential
Equations

CISE301_Topic1 18
 Lecture 2
Number Representation and
Accuracy
 Number Representation
 Normalized Floating Point Representation
 Significant Digits
 Accuracy and Precision
 Rounding and Chopping

Reading Assignment: Chapter 3

CISE301_Topic1 19
Representing Real Numbers
 You are familiar with the decimal system:

312.45  3  10 2  1 101  2 100  4 101  5 10 2

 Decimal System: Base = 10 , Digits (0,1,…,9)

 Standard Representations:

 3 1 2 . 4 5
sign integral fraction
part part
CISE301_Topic1 20
Normalized Floating Point Representation
 Normalized Floating Point Representation:

 d . f1 f 2 f 3 f 4  10 n
sign mantissa exponent

d  0,  n : signed exponent
 Scientific Notation: Exactly one non-zero digit appears
before decimal point.
 Advantage: Efficient in representing very small or very
large numbers.
CISE301_Topic1 21
Binary System
 Binary System: Base = 2, Digits {0,1}

 1. f1 f 2 f 3 f 4  2  n
sign mantissa signed exponent

(1.101)2  (1  1  2 1  0  2  2  1  2 3 )10  (1.625)10

CISE301_Topic1 22
Fact
 Numbers that have a finite expansion in one numbering
system may have an infinite expansion in another
numbering system:

(1.1)10  (1.0001100110 01100 ...) 2

 You can never represent 1.1 exactly in binary system.

CISE301_Topic1 23
IEEE 754 Floating-Point Standard
 Single Precision (32-bit representation)
 1-bit Sign + 8-bit Exponent + 23-bit Fraction

S Exponent8 Fraction23

 Double Precision (64-bit representation)

 1-bit Sign + 11-bit Exponent + 52-bit Fraction

S Exponent11 Fraction52
(continued)

CISE301_Topic1 24
Significant Digits

 Significant digits are those digits that can be

used with confidence.
 Single-Precision: 7 Significant Digits

1.175494… × 10-38 to 3.402823… × 1038

 Double-Precision: 15 Significant Digits

2.2250738… × 10-308 to 1.7976931… × 10308

CISE301_Topic1 25
Remarks
 Numbers that can be exactly represented are called
machine numbers.

 Difference between machine numbers is not uniform

 Sum of machine numbers is not necessarily a machine

number

CISE301_Topic1 26
Calculator Example
 Suppose you want to compute:
3.578 * 2.139
using a calculator with two-digit fractions

3.57 * 2.13 = 7.60

True answer: 7.653342

CISE301_Topic1 27
Significant Digits - Example
48.9

CISE301_Topic1 28
Accuracy and Precision

 Accuracy is related to the closeness to the true

value.

 Precision is related to the closeness to other

estimated values.

CISE301_Topic1 29
CISE301_Topic1 30
Rounding and Chopping

 Rounding: Replace the number by the nearest

machine number.

 Chopping: Throw all extra digits.

CISE301_Topic1 31
Rounding and Chopping

CISE301_Topic1 32
Error Definitions – True Error
Can be computed if the true value is known:

Absolute True Error

Et  true value  approximation
Absolute Percent Relative Error
true value  approximation
t  *100
true value

CISE301_Topic1 33
Error Definitions – Estimated Error
When the true value is not known:

Estimated Absolute Error

Ea  current estimate  previous estimate
Estimated Absolute Percent Relative Error
current estimate  previous estimate
a  *100
current estimate

CISE301_Topic1 34
Notation
We say that the estimate is correct to n
decimal digits if:
n
Error  10

We say that the estimate is correct to n

decimal digits rounded if:
1 n
Error   10
2

CISE301_Topic1 35
Summary
 Number Representation
Numbers that have a finite expansion in one numbering system
may have an infinite expansion in another numbering system.

 Normalized Floating Point Representation

 Efficient in representing very small or very large numbers,
 Difference between machine numbers is not uniform,
 Representation error depends on the number of bits used in
the mantissa.

CISE301_Topic1 36
 Lectures 3-4
Taylor Theorem
 Motivation
 Taylor Theorem
 Examples

Reading assignment: Chapter 4

CISE301_Topic1 37
Motivation
 We can easily compute expressions like:
3 10 2
2( x  4)
But, How do you compute 4.1, sin(0.6) ?

Can we use the definition

b
a
to compute sin(0.6)?
0.6
Is this a practical way?

CISE301_Topic1 38
Remark

 In this course, all angles are assumed to

be in radian unless you are told otherwise.

CISE301_Topic1 39
Taylor Series
The Taylor series expansion of f ( x ) about a :
' f ( 2) (a ) 2 f ( 3) ( a )
f (a )  f (a ) ( x  a )  ( x  a)  ( x  a ) 3  ...
2! 3!
or

1 (k )
Taylor Series   k!
f (a ) ( x  a )k
k 0
If the series converge, we can write :
∞
1 (k )
f ( x)  ∑ k!
f (a ) ( x  a )k
k 0

CISE301_Topic1 40
Maclaurin Series
 Maclaurin series is a special case of Taylor
series with the center of expansion a = 0.
The Maclaurin series expansion of f ( x ) :
(2) ( 3)
' f ( 0 ) 2 f ( 0) 3
f ( 0 )  f ( 0) x  x  x  ...
2! 3!
If the series converge, we can write :
∞
1 (k )
f ( x)  ∑ f ( 0) x k
k 0
k!
CISE301_Topic1 41
Maclaurin Series – Example 1
Obtain Maclaurin series expansion of f ( x )  e x
f ( x)  e x f ( 0)  1
f ' ( x)  e x f ' ( 0)  1
f ( 2) ( x )  e x f ( 2 ) ( 0)  1
f (k ) ( x)  e x f ( k ) (0)  1 for k  1
∞ ∞
1 (k ) xk x2 x3
ex  ∑ f ( 0) x  ∑
k
 1 x    ...
k 0
k! k 0
k! 2! 3!
The series converges for x  ∞.
CISE301_Topic1 42
Taylor Series
3

Example 1
2.5
exp(x)
1+x+0.5x 2
2

1+x

1.5

1
1

0.5

0
-1 -0.8 -0.6 -0.4 -0.2 0 0.2 0.4 0.6 0.8 1

CISE301_Topic1 43
Maclaurin Series – Example 2
Obtain Maclaurin series expansion of f ( x )  sin( x ) :
f ( x )  sin( x ) f ( 0)  0
f ' ( x )  cos( x ) f ' (0)  1
f ( 2 ) ( x )   sin( x ) f ( 2 ) ( 0)  0
f ( 3) ( x )   cos( x ) f ( 3) (0)  1
∞
f ( k ) ( 0) k x3 x5 x7
sin( x )  ∑ x  x     ....
k 0
k! 3! 5! 7!
The series converges for x  ∞.
CISE301_Topic1 44
 4

3
x

1 x-x 3/3!+x 5/5!

0 sin(x)

-1
x-x 3/3!

-2

-3

-4
-4 -3 -2 -1 0 1 2 3 4

CISE301_Topic1 45
Maclaurin Series – Example 3
Obtain Maclaurin series expansion of : f ( x)  cos( x)
f ( x )  cos( x ) f ( 0)  1
f ' ( x )   sin( x ) f ' ( 0)  0
f ( 2 ) ( x )   cos( x ) f ( 2 ) ( 0 )  1
f ( 3) ( x )  sin( x ) f ( 3) (0)  0
∞
f ( k ) ( 0) x 2
x 4
x 6
cos( x )  ∑ ( x ) k  1     ....
k 0
k! 2! 4! 6!
The series converges for x  ∞.
CISE301_Topic1 46
Maclaurin Series – Example 4
1
Obtain Maclaurin series expansion of f(x) 
1 x
1
f ( x)  f ( 0)  1
1 x
1
f ' ( x)  f ' ( 0)  1
1  x  2
(2) 2
f ( x)  f ( 2 ) ( 0)  2
1  x  3
6
f ( 3) ( x )  f ( 3) (0)  6
1  x  4
1
Maclaurin Series Expansion of :  1  x  x 2  x 3  ...
1 x
Series converges for | x |  1
CISE301_Topic1 47
Example 4 - Remarks
 Can we apply the series for x≥1??

 How many terms are needed to get a good

approximation???

These questions will be answered using

Taylor’s Theorem.

CISE301_Topic1 48
Taylor Series – Example 5
1
Obtain Taylor series expansion of f(x)  at a  1
x
1
f ( x)  f (1)  1
x
1
f ' ( x)  2 f ' (1)  1
x
2
f ( 2) ( x )  3 f ( 2 ) (1)  2
x
6
f ( 3) ( x )  4 f ( 3) (1)  6
x
Taylor Series Expansion ( a  1) : 1  ( x  1)  ( x  1) 2  ( x  1) 3  ...
CISE301_Topic1 49
Taylor Series – Example 6
Obtain Taylor series expansion of f(x)  ln( x ) at ( a  1)

1 ( 2) 1 ( 3) 2
f ( x )  ln( x ) , f ' ( x )  , f ( x )  2 , f ( x )  3
x x x
f (1)  0, f ' (1)  1, f ( 2) (1)  1 f ( 3) (1)  2

1 2 1
Taylor Series Expansion : ( x  1)  ( x  1)  ( x  1) 3  ...
2 3

CISE301_Topic1 50
Convergence of Taylor Series

 The Taylor series converges fast (few terms

are needed) when x is near the point of
expansion. If |x-a| is large then more terms
are needed to get a good approximation.

CISE301_Topic1 51
Taylor’s Theorem
If a function f ( x ) possesses derivatives of orders 1, 2, ..., ( n  1)
on an interval containing a and x then the value of f ( x ) is given by :

(n+1) terms Truncated

Taylor Series
n
f ( k ) (a )
f ( x)  ∑ k!
( x  a)k  Rn
k 0
Remainder
where :
f ( n 1) ( )
Rn  ( x  a ) n 1 and  is between a and x.
( n  1)!
CISE301_Topic1 52
Taylor’s Theorem
We can apply Taylor' s theorem for :
1
f(x)  with the point of expansion a  0 if | x |  1.
1 x

If x  1, then the function and its

derivatives are not defined.
 Taylor Theorem is not applicable.

CISE301_Topic1 53
Error Term
To get an idea about the approximation error,
we can derive an upper bound on :
( n 1)
f( )
Rn  ( x  a ) n 1
( n  1)!
for all values of  between a and x.

CISE301_Topic1 54
Error Term - Example
x
How large is the error if we replaced f ( x )  e by
the first 4 terms ( n  3) of its Taylor series expansion
at a  0 when x  0.2 ?
f (n) ( x)  e x f ( n ) ( ) ≤ e 0.2 for n ≥ 1
f ( n 1) ( )
Rn  ( x  a ) n 1
( n  1)!
e 0.2 n 1
Rn   0.2   R3  8.14268E  05
( n  1)!
CISE301_Topic1 55
Alternative form of Taylor’s Theorem
Let f ( x ) have derivatives of orders 1, 2, ..., ( n  1)
on an interval containing x and x  h then :
n (k )
f ( x) k
f ( x  h)   k!
h  Rn ( h  step size)
k 0

f ( n 1) ( ) n 1
Rn  h where  is between x and x  h
( n  1)!

CISE301_Topic1 56
Taylor’s Theorem – Alternative
forms
n ( n 1)
f ( k ) (a ) f ( )
f ( x)   k
( x  a)  ( x  a ) n 1
k 0 k! ( n  1)!
where  is between a and x.

a  x, x  x  h

n
f ( k ) ( x ) k f ( n 1) ( ) n 1
f ( x  h)   h  h
k 0 k! ( n  1)!
where  is between x and x  h.
CISE301_Topic1 57
Mean Value Theorem
If f ( x ) is a continuous function on a closed interval [a, b]
and its derivative is defined on the open interval ( a, b)
then there exists ξ  ( a , b)
f(b)  f(a)
f ' (ξ ) 
ba
Proof : Use Taylor' s Theorem for n  0, x  a , x  h  b
f(b)  f(a)  f ' (ξ ) (b  a )

CISE301_Topic1 58
Alternating Series Theorem
Consider the alternating series :
S  a1  a2  a3  a4  
 a  a  a  a  The series converges
 1 2 3 4

If  and then  and
 lim a  0  S  S n  an 1
 n n 

S n : Partial sum (sum of the first n terms)

an 1 : First omitted term
CISE301_Topic1 59
Alternating Series – Example
1 1 1
sin(1) can be computed using : sin(1)  1     
3! 5! 7!
This is a convergent alternating series since :
a1  a2  a3  a4   and lim an  0
n 

Then :
 1 1
sin(1)  1   
 3!  5!
 1 1 1
sin(1)  1    
 3! 5!  7!
CISE301_Topic1 60
Example 7
Obtain the Taylor series expansion
of f ( x )  e 2 x 1 at a  0.5 (the center of expansion)
How large can the error be when ( n  1) terms are used
to approximate e 2 x 1 with x  1 ?

CISE301_Topic1 61
Example 7 – Taylor Series
Obtain Taylor series expansion of f ( x )  e 2 x 1 , a  0.5
f ( x)  e 2 x 1 f (0.5)  e 2
f ' ( x)  2e 2 x 1 f ' (0.5)  2e 2
f ( 2) ( x)  4e 2 x 1 f ( 2) (0.5)  4e 2
f ( k ) ( x)  2 k e 2 x 1 f ( k ) (0.5)  2 k e 2
∞
f ( k ) (0.5)
e 2 x 1
 ∑ k!
( x  0.5) k
k 0

2 2 ( x  0.5) 2
2 k 2 ( x  0.5)
k
 e  2e ( x  0.5)  4e  ...  2 e  ...
2! k!
CISE301_Topic1 62
Example 7 – Error Term
f ( k ) ( x)  2 k e 2 x 1
f ( n 1) ( )
Error  ( x  0.5) n 1
(n  1)!
n 1 2 1 (1  0.5) n 1
Error  2 e
(n  1)!
n 1
( 0. 5)
Error  2 n 1 max e 2 1
(n  1)!  [ 0.5,1]
e3
Error 
(n  1)!
CISE301_Topic1 63