Advanced Algorithms Analysis

and Design

By

Dr. Nazir Ahmad Zafar

Dr Nazir A. Zafar Advanced Algorithms Analysis and Design

 Lecture No. 10

Time Complexity of Algorithms

(Asymptotic Notations)

Dr Nazir A. Zafar Advanced Algorithms Analysis and Design

 Today Covered
• Major Factors in Algorithms Design
• Complexity Analysis
• Growth of Functions
• Asymptotic Notations
• Usefulness of Notations
• Reflexivity, Symmetry, Transitivity Relations over
, , O,  and o
• Relation between ,  and O
• Various Examples Explaining each concept

Dr Nazir A. Zafar Advanced Algorithms Analysis and Design

 What is Complexity?
• The level in difficulty in solving mathematically
posed problems as measured by
– The time
(time complexity)
– number of steps or arithmetic operations
(computational complexity)
– memory space required
– (space complexity)

Dr Nazir A. Zafar Advanced Algorithms Analysis and Design

 Major Factors in Algorithms Design
1. Correctness
An algorithm is said to be correct if
• For every input, it halts with correct output.
• An incorrect algorithm might not halt at all OR
• It might halt with an answer other than desired one.
• Correct algorithm solves a computational problem
2. Algorithm Efficiency
Measuring efficiency of an algorithm,
• do its analysis i.e. growth rate.
• Compare efficiencies of different algorithms for the
same problem.
Dr Nazir A. Zafar Advanced Algorithms Analysis and Design
 Algorithms Growth Rate
Algorithm Growth Rates
• It measures algorithm efficiency
What means by efficient?
 If running time is bounded by polynomial in the input
Notations for Asymptotic performance
• How running time increases with input size
• O, Omega, Theta, etc. for asymptotic running time
• These notations defined in terms of functions whose
domains are natural numbers
• convenient for worst case running time
• Algorithms, asymptotically efficient best choice
Dr Nazir A. Zafar Advanced Algorithms Analysis and Design
 Complexity Analysis
• Algorithm analysis means predicting resources such as
– computational time
– memory
– computer hardware etc
• Worst case analysis
– Provides an upper bound on running time
– An absolute guarantee
• Average case analysis
– Provides the expected running time
– Very useful, but treat with care: what is “average”?
• Random (equally likely) inputs
• Real-life inputs

Dr Nazir A. Zafar Advanced Algorithms Analysis and Design

 Worst-case Analysis
Let us suppose that
• Dn = set of inputs of size n for the problem
• I = an element of Dn.
• t(I) = number of basic operations performed on I
• Define a function W by
W(n) = max{t(I) | I  Dn}
called the worst-case complexity of the algorithm
• W(n) is the maximum number of basic operations
performed by the algorithm on any input of size n.
• Please note that the input, I, for which an algorithm
behaves worst depends on the particular algorithm.
Dr Nazir A. Zafar Advanced Algorithms Analysis and Design
 Average Complexity
• Let Pr(I) be the probability that input I occurs.
• Then the average behavior of the algorithm is defined as
A(n) = Pr(I) t(I), summation over all I  Dn
• We determine t(I) by analyzing the algorithm, but Pr(I)
cannot be computed analytically.
• Average cost =
A(n) = Pr(succ)Asucc(n) + Pr(fail)Afail(n)
• An element I in Dn may be thought as a set or
equivalence class that affect the behavior of the algorithm
Worst Analysis computing average cost
• Take all possible inputs, compute their cost, take average

Dr Nazir A. Zafar Advanced Algorithms Analysis and Design

 Asymptotic Notations Properties
• Categorize algorithms based on asymptotic growth
rate e.g. linear, quadratic, polynomial, exponential
• Ignore small constant and small inputs
• Estimate upper bound and lower bound on growth
rate of time complexity function
• Describe running time of algorithm as n grows to .
• Describes behavior of function within the limit.
Limitations
• not always useful for analysis on fixed-size inputs.
• All results are for sufficiently large inputs.

Dr Nazir A. Zafar Advanced Algorithms Analysis and Design

 Asymptotic Notations
Asymptotic Notations , O, , o, 
 We use  to mean “order exactly”,
 O to mean “order at most”,
  to mean “order at least”,
 o to mean “tight upper bound”,
  to mean “tight lower bound”,

Define a set of functions: which is in practice used to

compare two function sizes.

Dr Nazir A. Zafar Advanced Algorithms Analysis and Design

 Big-Oh Notation (O)
If f, g: N  R+, then we can define Big-Oh as
For a given function g  n   0, denoted by  g  n   the set of functions,
 g  n     f  n  : there exist positive constants c and no such that
0  f  n   cg  n  , for all n n o 
f  n    g  n   means function g  n  is an asymptotically
upper bound for f  n .

We may write f(n) = O(g(n)) OR f(n)  O(g(n))

Intuitively:
Set of all functions whose rate of growth is the same as or lower
than that of g(n).
Dr Nazir A. Zafar Advanced Algorithms Analysis and Design
 Big-Oh Notation

f(n)  O(g(n))

 c > 0,  n0  0 and n  n0, 0  f(n)  c.g(n)

g(n) is an asymptotic upper bound for f(n).

Dr Nazir A. Zafar Advanced Algorithms Analysis and Design

 Examples
Examples
Example 1: Prove that 2n2  O(n3)
Proof:
Assume that f(n) = 2n2 , and g(n) = n3
f(n)  O(g(n)) ?
Now we have to find the existence of c and n0

f(n) ≤ c.g(n)  2n2 ≤ c.n3  2 ≤ c.n

if we take, c = 1 and n0= 2 OR
c = 2 and n0= 1 then
2n2 ≤ c.n3
Hence f(n)  O(g(n)), c = 1 and n0= 2
Dr Nazir A. Zafar Advanced Algorithms Analysis and Design
 Examples
Examples
Example 2: Prove that n2  O(n2)
Proof:
Assume that f(n) = n2 , and g(n) = n2
Now we have to show that f(n)  O(g(n))

Since
f(n) ≤ c.g(n)  n2 ≤ c.n2  1 ≤ c, take, c = 1, n0= 1

Then
n2 ≤ c.n2 for c = 1 and n  1
Hence, 2n2  O(n2), where c = 1 and n0= 1
Dr Nazir A. Zafar Advanced Algorithms Analysis and Design
 Examples
Examples
Example 3: Prove that 1000.n2 + 1000.n  O(n2)
Proof:
Assume that f(n) = 1000.n2 + 1000.n, and g(n) = n2
We have to find existence of c and n0 such that
0 ≤ f(n) ≤ c.g(n)  n  n0
1000.n2 + 1000.n ≤ c.n2 = 1001.n2, for c = 1001
1000.n2 + 1000.n ≤ 1001.n2
 1000.n ≤ n2  n2  1000.n  n2 - 1000.n  0
 n (n-1000)  0, this true for n  1000
f(n) ≤ c.g(n)  n  n0 and c = 1001

Hence f(n)  O(g(n)) for c = 1001 and n0 = 1000

Dr Nazir A. Zafar Advanced Algorithms Analysis and Design
 Examples
Examples

Example 4: Prove that n3  O(n2)

Proof:
On contrary we assume that there exist some
positive constants c and n0 such that
0 ≤ n3 ≤ c.n2  n  n0
0 ≤ n3 ≤ c.n2  n ≤ c
Since c is any fixed number and n is any arbitrary
constant, therefore n ≤ c is not possible in general.
Hence our supposition is wrong and n3 ≤ c.n2,
 n  n0 is not true for any combination of c and n0.
And hence, n3  O(n2)
Dr Nazir A. Zafar Advanced Algorithms Analysis and Design
 SomeExamples
More Examples
1. n2 + n3 = O(n4)
2. n2 / log(n) = O(n . log n)
3. 5n + log(n) = O(n)
4. nlog n = O(n100)
5. 3n = O(2n . n100)
6. n! = O(3n)
7. n +1 = O(n)
8. 2n+1 = O(2n)
9. (n+1)! = O(n!)
10. 1 + c + c2 +…+ cn = O(cn) for c > 1
11. 1 + c + c2 +…+ cn = O(1) for c < 1

Dr Nazir A. Zafar Advanced Algorithms Analysis and Design

 Big-Omega Notation ()
If f, g: N  R+, then we can define Big-Omega as
For a given function g  n  denote by  g  n   the set of functions,
 g  n     f  n  : there exist positive constants c and no such that
0  cg  n   f  n  for all n  n o 
f  n    g  n   , means that function g  n  is an asymptotically
lower bound for f  n .

We may write f(n) = (g(n)) OR f(n)  (g(n))

Intuitively:
Set of all functions whose rate of growth is the same as or higher
than that of g(n).

Dr Nazir A. Zafar Advanced Algorithms Analysis and Design

 Big-Omega Notation

f(n)  (g(n))

 c > 0,  n0  0 , n  n0, f(n)  c.g(n)

g(n) is an asymptotically lower bound for f(n).
Dr Nazir A. Zafar Advanced Algorithms Analysis and Design
 Examples
Examples

Example 1: Prove that 5.n2  (n)

Proof:
Assume that f(n) = 5.n2 , and g(n) = n
f(n)  (g(n)) ?
We have to find the existence of c and n0 s.t.
c.g(n) ≤ f(n)  n  n0
c.n ≤ 5.n2  c ≤ 5.n
if we take, c = 5 and n0= 1 then
c.n ≤ 5.n2  n  n0
And hence f(n)  (g(n)), for c = 5 and n0= 1

Dr Nazir A. Zafar Advanced Algorithms Analysis and Design

 Examples
Examples

Example 2: Prove that 5.n + 10  (n)

Proof:
Assume that f(n) = 5.n + 10, and g(n) = n
f(n)  (g(n)) ?
We have to find the existence of c and n0 s.t.
c.g(n) ≤ f(n)  n  n0
c.n ≤ 5.n + 10  c.n ≤ 5.n + 10.n  c ≤ 15.n
if we take, c = 15 and n0= 1 then
c.n ≤ 5.n + 10  n  n0
And hence f(n)  (g(n)), for c = 15 and n0= 1
Dr Nazir A. Zafar Advanced Algorithms Analysis and Design
 Examples
Examples
Example 3: Prove that 100.n + 5  (n2)
Proof:
Let f(n) = 100.n + 5, and g(n) = n2
Assume that f(n)  (g(n)) ?
Now if f(n)  (g(n)) then there exist c and n0 s.t.
c.g(n) ≤ f(n)  n  n0 
c.n2 ≤ 100.n + 5 
c.n ≤ 100 + 5/n 
n ≤ 100/c, for a very large n, which is not possible

And hence f(n)  (g(n))

Dr Nazir A. Zafar Advanced Algorithms Analysis and Design
 Theta Notation ()

If f, g: N  R+, then we can define Big-Theta as

For a given function g  n  denoted by  g  n   the set of functions,

 g  n     f  n  : there exist positive constants c1 , c2 and no such that
0  c1 g  n   f  n   c2 g  n  for all n n o 
f  n    g  n   means function f  n  is equal to g  n  to within a constant
factor, and g  n  is an asymptotically tight bound for f  n .

We may write f(n) = (g(n)) OR f(n)  (g(n))

Intuitively: Set of all functions that have same rate of growth as g(n).

Dr Nazir A. Zafar Advanced Algorithms Analysis and Design

 Theta Notation

f(n)  (g(n))

 c1> 0, c2> 0,  n0  0,  n  n0, c2.g(n)  f(n)  c1.g(n)

We say that g(n) is an asymptotically tight bound for f(n).
Dr Nazir A. Zafar Advanced Algorithms Analysis and Design
 Theta Notation
Example 1: Prove that ½.n2 – ½.n = (n2)
Proof
Assume that f(n) = ½.n2 – ½.n, and g(n) = n2
f(n)  (g(n))?
We have to find the existence of c1, c2 and n0 s.t.
c1.g(n) ≤ f(n) ≤ c2.g(n)  n  n0
Since, ½ n2 - ½ n ≤ ½ n2 n ≥ 0 if c2= ½ and
½ n2 - ½ n ≥ ½ n2 - ½ n . ½ n ( n ≥ 2 ) = ¼ n2, c1= ¼
Hence ½ n2 - ½ n ≤ ½ n2 ≤ ½ n2 - ½ n
c1.g(n) ≤ f(n) ≤ c2.g(n) n ≥ 2, c1= ¼, c2 = ½
Hence f(n)  (g(n))  ½.n2 – ½.n = (n2)
Dr Nazir A. Zafar Advanced Algorithms Analysis and Design
 Theta Notation
Example 2: Prove that a.n2 + b.n + c = (n2) where a, b, c
are constants and a > 0
Proof
If we take c1 = ¼.a, c2 = 7/4. a and

n0  2. max(( b / a ), ( c / a ))
Then it can be easily verified that
0 ≤ c1.g(n) ≤ f(n) ≤ c2.g(n),n ≥ n0, c1= ¼.a, c2 = 7/4.a

Hence f(n)  (g(n))  a.n2 + b.n + c = (n2)

Hence any polynomial of degree 2 is of order (n2)

Dr Nazir A. Zafar Advanced Algorithms Analysis and Design

 Theta Notation
Example 1: Prove that 2.n2 + 3.n + 6  (n3)
Proof: Let f(n) = 2.n2 + 3.n + 6, and g(n) = n3
we have to show that f(n)  (g(n))
On contrary assume that f(n)  (g(n)) i.e.
there exist some positive constants c1, c2 and n0 such
that: c1.g(n) ≤ f(n) ≤ c2.g(n)
c1.g(n) ≤ f(n) ≤ c2.g(n)  c1.n3 ≤ 2.n2 + 3.n + 6 ≤ c2. n3 
c1.n ≤ 2 + 3/n + 6/n2 ≤ c2. n 
c1.n ≤ 2 ≤ c2. n, for large n 
n ≤ 2/c1 ≤ c2/c1.n which is not possible
Hence f(n)  (g(n))  2.n2 + 3.n + 6  (n3)
Dr Nazir A. Zafar Advanced Algorithms Analysis and Design
 Little-Oh Notation
o-notation is used to denote a upper bound that is not
asymptotically tight.
For a given function g  n   0, denoted by o g  n   the set of functions,
 f  n  : for any positive constants c, there exists a constant no
o g  n    
such that 0  f  n   cg  n  for all n n o 

f(n) becomes insignificant relative to g(n) as n

approaches infinity
f  n
  lim
e.g., 2n  o n but 2n  o n .. n  g  n 
2 2 2  0  
g(n) is an upper bound for f(n), not asymptotically tight
Dr Nazir A. Zafar Advanced Algorithms Analysis and Design
 Examples
Examples
Example 1: Prove that 2n2  o(n3)
Proof:
Assume that f(n) = 2n2 , and g(n) = n3
f(n)  o(g(n)) ?

Now we have to find the existence n0 for any c

f(n) < c.g(n) this is true
 2n2 < c.n3  2 < c.n
This is true for any c, because for any arbitrary c
we can choose n0 such that the above inequality
holds.
Hence f(n)  o(g(n))
Dr Nazir A. Zafar Advanced Algorithms Analysis and Design
 Examples
Examples

Example 2: Prove that n2  o(n2)

Proof:
Assume that f(n) = n2 , and g(n) = n2
Now we have to show that f(n)  o(g(n))

Since
f(n) < c.g(n)  n2 < c.n2  1 ≤ c,

In our definition of small o, it was required to prove

for any c but here there is a constraint over c .
Hence, n2  o(n2), where c = 1 and n0= 1

Dr Nazir A. Zafar Advanced Algorithms Analysis and Design

 Examples
Examples

Example 3: Prove that 1000.n2 + 1000.n  o(n2)

Proof:
Assume that f(n) = 1000.n2 + 1000.n, and g(n) = n2
we have to show that f(n)  o(g(n)) i.e.
We assume that for any c there exist n0 such that
0 ≤ f(n) < c.g(n)  n  n0
1000.n2 + 1000.n < c.n2
If we take c = 2001, then,1000.n2 + 1000.n < 2001.n2
 1000.n < 1001.n2 which is not true
Hence f(n)  o(g(n)) for c = 2001
Dr Nazir A. Zafar Advanced Algorithms Analysis and Design
 Little-Omega Notation

Little- notation is used to denote a lower bound

that is not asymptotically tight.
For a given function g  n  , denote by   g  n   the set of all functions.
  g  n     f  n  : for any positive constants c, there exists a constant no such that
0  cg  n   f  n  for all n n o 

f(n) becomes arbitrarily large relative to g(n) as n

f  n
approaches infinity
lim 
n  g  n 

 
2 2
n n
e.g.,    n  but   n ..
2

2 2
Dr Nazir A. Zafar Advanced Algorithms Analysis and Design
 Examples
Examples

Example 1: Prove that 5.n2  (n)

Proof:
Assume that f(n) = 5.n2 , and g(n) = n
f(n)  (g(n)) ?
We have to prove that for any c there exists n0 s.t.,
c.g(n) < f(n)  n  n0
c.n < 5.n2  c < 5.n
This is true for any c, because for any arbitrary c
e.g. c = 1000000, we can choose n0 = 1000000/5
= 200000 and the above inequality does hold.
And hence f(n)  (g(n)),

Dr Nazir A. Zafar Advanced Algorithms Analysis and Design

 Examples
Examples

Example 2: Prove that 5.n + 10  (n)

Proof:
Assume that f(n) = 5.n + 10, and g(n) = n
f(n)  (g(n)) ?

We have to find the existence n0 for any c, s.t.

c.g(n) < f(n)  n  n0
c.n < 5.n + 10, if we take c = 16 then
16.n < 5.n + 10  11.n < 10 is not true for any
positive integer.
Hence f(n)  (g(n))

Dr Nazir A. Zafar Advanced Algorithms Analysis and Design

 Examples
Examples

Example 3: Prove that 100.n  (n2)

Proof:
Let f(n) = 100.n, and g(n) = n2
Assume that f(n)  (g(n))
Now if f(n)  (g(n)) then there n0 for any c s.t.
c.g(n) < f(n)  n  n0 this is true
 c.n2 < 100.n  c.n < 100
If we take c = 100, n < 1, not possible
Hence f(n)  (g(n)) i.e. 100.n  (n2)

Dr Nazir A. Zafar Advanced Algorithms Analysis and Design

 Usefulness of Notations
• It is not always possible to determine behaviour of
an algorithm using Θ-notation.
• For example, given a problem with n inputs, we may
have an algorithm to solve it in a.n2 time when n is
even and c.n time when n is odd. OR
• We may prove that an algorithm never uses more
than e.n2 time and never less than f.n time.
• In either case we can neither claim (n) nor (n2) to
be the order of the time usage of the algorithm.
• Big O and  notation will allow us to give at least
partial information

Dr Nazir A. Zafar Advanced Algorithms Analysis and Design