Analysis

of
Algorithm
s
COURSE CODE: CSC 402
SECOND YEAR SEMESTER IV
COMPUTER ENGINEERING
Marks Distribution

Total: 150 Marks

CSC 402: Analysis of Algorithms

• End Semester Exam: 80 Marks

• Internal Assessment ( I & II): 20 Marks

CSL 401: Analysis of Algorithms Lab

• Termwork: 25 Marks
• Practical: 25 Marks
 Syllabus

Module No. Description Hours

Introduction
• Performance analysis, space and time complexity, Growth of function, Asymptotic
1 (Big- Oh, Omega, Theta) notation, Mathematical background for algorithm Analysis, 8
Analysis of selection sort, insertion sort.
• Recurrences: The substitution method, Recursion tree method, Master Method

Divide and Conquer Approach

2 • General method, Merge sort, Quick sort, Finding minimum and maximum algorithms 6
and their Analysis, Analysis of Binary search.

Greedy Method Approach

3
• General Method, Single source shortest path: Dijkstra Algorithm, Fractional Knapsack 6
problem, Job sequencing with deadlines, Minimum cost spanning trees: Kruskal and
Prim’s algorithms
Syllabus

Module No. Description Hours

Dynamic Programming Approach

• General Method, Multistage graphs, Single source shortest path: Bellman
4 Ford Algorithm, All pair shortest path: Floyd Warshall Algorithm, 9
Assembly-line scheduling Problem0/1 knapsack Problem, Travelling
Salesperson problem, Longest common subsequence

Backtracking and Branch and bound

5 • General Method, Backtracking: N-queen problem, Sum of subsets, 6
Graph coloring

String Matching Algorithms

6 • The Naïve string-matching algorithm, The Rabin Karp algorithm, The 4
Knuth-Morris-Pratt algorithm
Books

Textbook Reference Books

T. H. Cormen, C.E. Leiserson,R.L. Sanjoy Dasgupta, Christos
Rivest, and C. Stein, “Introduction to Papadimitriou, Umesh Vazirani,
algorithms”, 2nd Edition, PHI Publication “Algorithms”, Tata McGrawHill Edition.
2005. S. K. Basu, “Design Methods and
Ellis Horowitz, Sartaj Sahni, S. Analysis of Algorithm”, PHI
Rajsekaran. “Fundamentals of
computer algorithms” University Press.
Analysis of Algorithms Lab

 Term work should consist of 10 experiments.

 Journal must include at least 2 assignments on content of theory and practical of
“Analysis of Algorithms”.
 Total 25 Marks (Experiments: 15-marks, Attendance Theory & Practical: 05-marks,
Assignments: 05-marks)
 Experiment (15 Marks)
 Execution (10 Marks)
 Viva (3 Marks)
 Timely Submission (2 Marks)
 Course
Objective
 Analyze the running time and space
complexity of algorithms.
 Analyze the complexity of divide and
conquer strategy.
 Analyze the complexity of greedy
strategy.
 Analyze the complexity of dynamic
programming strategy.
 Apply backtracking, branch and
techniques to deal with some hard
problems.
 Apply string matching techniques to
deal with some hard problems.
 Module 1 : Introduction

• Performance analysis • Mathematical background for

• Space and Time complexity algorithm Analysis
• Growth of function • Analysis of Selection Sort and
Insertion Sort.
• Asymptotic notation
• Big- Oh notation
• Recurrences
• Omega notation • Substitution Method
• Theta) notation • Recursion Tree Method
• Master Method
Learning Objective

To identify the different To define the

types of the algorithms measuring units of the
algorithm
Algorithm

 An algorithm is
 A sequence of finite instructions
 Which converts a given input
 Into the desired Output
 To solve a well formulated problem

Input Algorith
m Output
Analysis

Consider you want to travel from Location A to B.

Bus Train Car

If you travel by Bus, If you travel by Train, If you travel by Car,

then it will take 2 Hours then it will take 1 Hours then it will take 1.25
and Rs. 40 but you will and Rs. 10 but you will Hours and Rs. 200 with
find the moderate rush. find the maximum rush. maximum comfort.

Which to select?
If you can spend If you don’t want to
You can spend any
some more money spend too much
amount of money for
and want some money but ready to
the comfort
comfort also. lose the comfort
  Sort the following 10 numbers
1, 3, 8, 2, 7, 6, 4, 9, 0, 5
Different ways,
Selection Sort
Insertion Sort
Example Quick Sort
Merge Sort
Radix Sort
Shell Sort
…..& so, on
Types of Algorithm

Divide &
Iterative Recursive Greedy
Conquer
Algorithm Algorithm Method
Algorithm

Dynamic
Back Branch & String
Programmin
Tracking Bound Algorithm
g
Iterative Algorithm

 An iterative algorithm executes steps in iterations. It aims to find

successive approximation in sequence to reach a solution. They are
most used in linear programs where large numbers of variables are
involved.
 Example
 Selection Sort
 Insertion Sort
Recursive Algorithm

 Recursive algorithm is a method of simplification

that divides the problem into sub-problems of the
same nature. The result of one recursion is the
input for the next recursion. The repletion is in the
self-similar fashion. The algorithm calls itself with
smaller input values and obtains the results by
simply performing the operations on these smaller
values.
 Example
 Fibonacci Series
 Factorial
Divide & Conquer Algorithm

 A divide-and-conquer algorithm recursively breaks down a problem into

two or more sub-problems of the same or related type, until these
become simple enough to be solved directly. The solutions to the sub-
problems are then combined to give a solution to the original problem.
 Example
 Quick Sort
 Merge Sort
Greedy Algorithm

 Among all the algorithmic approaches, the simplest and straightforward

approach is the Greedy method. In this approach, the decision is taken based
on current available information without worrying about the effect of the
current decision in future.
 Greedy algorithms build a solution part by part, choosing the next part in such
a way, that it gives an immediate benefit. This approach never reconsiders the
choices taken previously. This approach is mainly used to solve optimization
problems. Greedy method is easy to implement and quite efficient in most of
the cases. Hence, we can say that Greedy algorithm is an algorithmic
paradigm based on heuristic that follows local optimal choice at each step
with the hope of finding global optimal solution.
 In many problems, it does not produce an optimal solution though it gives an
approximate (near optimal) solution in a reasonable time.
Optimization Problem

 An optimization problem is the problem of finding the best solution from all feasible
solutions.
 Optimization problems can be divided into two categories, depending on whether the
variables are continuous or discrete:
 An optimization problem with discrete variables is known as a discrete optimization, in which
an object such as an integer, permutation or graph must be found from a countable set.
 A problem with continuous variables is known as a continuous optimization, in which an
optimal value from a continuous function must be found. They can include constrained
problems and multimodal problems.
Maxima Minima

 The point at which a function takes the

minimum value is called as global minima.
 Those several points which appear to be
minima but is not the point where the
function takes the minimum value is called
as local minima.
 The point at which a function takes the
maximum value is called as global
maxima.
 Those several points which appear to be
maxima but is not the point where the
function takes the maximum value is
called as local maxima.
Greedy Method

Sr. No Weight Profit

Capacity = 20
1 10 20

2 5 30
Profit = 66
3 6 18
Example

B
60
10

20 30
A C E

30 10
D
Dynamic Programming

 Dynamic programming is basically, recursion plus using

common sense. What it means is that recursion allows you to
express the value of a function in terms of other values of that
function. Where the common sense tells you that if you
implement your function in a way that the recursive calls are
done in advance, and stored for easy access, it will make your
program faster.
Backtracking

 Backtracking is a general algorithm for finding all

(or some) solutions to some computational
problems, notably constraint satisfaction
problems, that incrementally builds candidates to
the solutions, and abandons a candidate
("backtracks") as soon as it determines that the
candidate cannot possibly be completed to a valid
solution.
 Branch & Bound

 Branch and bound algorithms are used to find the optimal solution
for combinatory, discrete, and general mathematical optimization
problems. In general, given an NP-Hard problem, a branch and
bound algorithm explores the entire search space of possible
solutions and provides an optimal solution.
 String Algorithm

 In computer science, string-searching algorithms, sometimes

called string-matching algorithms, are an important class of string
algorithms that try to find a place where one or several strings
(also called patterns) are found within a larger string or text.
Space & Time Complexity

 The space complexity of an algorithm or a computer

program is the amount of memory space required to
solve an instance of the computational problem as a
function of characteristics of the input. It is the
memory required by an algorithm to execute a
program and produce output.
 The time complexity is the computational complexity
that describes the amount of time it takes to run an
algorithm. Time complexity is commonly estimated
by counting the number of elementary operations
performed by the algorithm, supposing that each
elementary operation takes a fixed amount of time to
perform.
Growth of Function

 The relationship between input size and the performance of the

algorithm is called order of growth.
 Few Efficiency Classes:
Efficiency Class Order of growth
rate
Constant 1
Logarithmic
Linear n
Quadratic
Cubic
Exponential
Factorial n!
Asymptotic Notation

Asymptotic notations are

used to represent the There are three notations
complexities of algorithms that are commonly used.
for asymptotic analysis.

Big – Oh Notation: The Upper Bound

Big – Omega Notation (Ω): The Lower
Bound
Big – Theta Notation (Θ): The Tight Bound

These notations are

mathematical tools to
represent the complexities.
Big Oh (O) Notation

Let f(n) and g(n) be two non-negative

functions indicating the time complexity
of two algorithms. We can say that g(n)
is upper bound of f(n) if there exist some
positive constant c and such that to the
right of value of f(n) always lies on or
below the value of c*g(n). It is denoted
as,
Big Omega (Ω) Notation

Let f(n) and g(n) be two non-negative

functions indicating the time complexity
of two algorithms. We can say that g(n)
is lower bound of f(n) if there exist some
positive constant c and such that to the
right of value of f(n) always lies on or
above the value of c*g(n). It is denoted
as,
Big Theta (Θ) Notation

Let f(n) and g(n) be two non-negative

functions indicating the time complexity
of two algorithms. We can say that g(n)
is tight bound of f(n) if there exist some
positive constant and such that to the
right of value of f(n) always lies between
value of inclusively. It is denoted as,
 Asymptotic Notation

Let f(n) and g(n) be two non-negative Let f(n) and g(n) be two non-negative Let f(n) and g(n) be two non-negative
functions indicating the time functions indicating the time functions indicating the time
complexity of two algorithms. We can complexity of two algorithms. We can complexity of two algorithms. We can
say that g(n) is upper bound of f(n) if say that g(n) is lower bound of f(n) if say that g(n) is tight bound of f(n) if

and 𝑛_0 such that to the right of 𝑛_0 and 𝑛_0 such that to the right of 𝑛_0
there exist some positive constant c there exist some positive constant c there exist some positive constant
and such that to the right of value of
f(n) always lies on or below the value value of f(n) always lies on or above value of f(n) always lies between

𝑓(𝑛)=Ω(𝑔(𝑛)) 𝑓(𝑛)=Θ(𝑔(𝑛))
of c*g(n). the value of c*g(n). It is denoted as, value of inclusively. It is denoted as,
It is denoted as,
Mathematic Background

∑ 1=1+1+1+1+ …+1=𝑛=O (𝑛)

𝑖 =1

𝑛
𝑛 ( 𝑛+1 ) 𝑛2 +𝑛
∑ 𝑖=1+2+ 3+ 4+ …+𝑛= ∑ 𝑛=
2
=
2
=O (𝑛2
)
𝑖 =1

𝑛
𝑛𝑘
∑ 𝑖 𝑘
=1+2𝑘
+ 3𝑘
+ 4𝑘
+ …+𝑛𝑘
=
𝑘+1
=O(𝑛𝑘+ 1
)
𝑖 =1

𝑛
𝑘𝑛+1
∑ 𝑘 =𝑘+𝑘 +𝑘 +𝑘 +…+ 𝑘 = 𝑘− 1 =O(𝑘𝑛 )
𝑖 2 3 4 𝑛

𝑖 =1
3 cases of Analysis

Best Case Worst Case Average Case

It is the case in which the It is the case in which the It is the case in which the
time complexity is minimum. time complexity is maximum. time complexity lies between
This is the most favorable This is the most unwanted the best & worst. This is the
condition or scenario. condition or scenario. case of maximum interest.
Selection Sort

Step Frequency
1 n
2 n-1
3 (n-1)*n
4 (n-1)*(n-1)
5 (n-1)*(n-1)* p(T)
6 (n-1)*(n-1)
7 (n-1)*(n-1)
8 (n-1)*(n-1)
Total
 Insertion Sort

¿
Algorithm Cost Frequency
for j  2 to length [A] c1 n
do Key  A[j] c2 n-1
I  j-1 C3 n-1
while i > 0 and A[i]
> Key
c4

do A[i+1]  A[i]
c5

i  i -1
c6

A[i +1]  Key c7 n-1

 Insertion Sort

Best Case:
In case of best case the while loop will get executed (n-1) times whereas statements within the while will not be
executed. Hence, we can say that

Hence above equation can be re-written as follows,

 Insertion Sort

Worst Case and Average Case:

In case of average case and worst case the while loop will get executed (n-1) times and also statements
within the while will be executed. Hence, we can say that

Hence above equation can be re-written as follows,

Selection Sort
Insertion Sort
Recurrence Relation

 An equation or inequality that describes a function in terms of its value

on smaller inputs.

 Recurrences arise when an algorithm contains recursive calls to itself

 What is the actual running time of the algorithm?
 Need to solve the recurrence–
 Find an explicit formula of the expression
 Bound the recurrence by an expression that involves n
Example Recurrences

 Recursive algorithm that loops through the input to eliminate one item
T(n) = T(n-1) + n
 Recursive algorithm that halves the input in one step
T(n) = T(n/2) + c
 Recursive algorithm that halves the input but must examine every item in
the input
T(n) = T(n/2) + n O(n)
 Recursive algorithm that splits the input into 2 halves and does a constant
amount of other work
T(n) = 2T(n/2) + 1 O(n)
Example Recurrences

 Binary Search

 Merge Sort

 Fibonacci Series
Methods for Solving Recurrences

ITERATION SUBSTITUTIO RECURSION MASTER

METHOD N METHOD TREE METHOD
METHOD
Function Growth Rate
Substitution Method

 The Substitution Method Consists of two main steps:

 Guess the Solution.
 Use the mathematical induction to find the boundary condition and shows
that the guess is correct.
 Question: Now if we put n =1 then we will get,
T(1) = 1 and T(1) = c log (1) = 0.
Solve the equation by Substitution
Method. If we put n=2 then we will get, T(2)
= T(1)+1 = 1+1 =2 and T(2) = c log
Substitution We must show that it is
(2) = c. so above condition will hold
true if value of c ≥ 2.
Method - asymptotically bound by O (log n). Hence, we can say our assumption is
true for .
Example Answer:
For T (n) = O (log n)
We must show that for some
constant c,
T (n) ≤ c log n
Put this in given Recurrence
Equation.

Hence our assumption is true if c ≥

1.
Substitution Method - Example

and guess solution is O(n log n)

Recursion Tree Method

1. Recursion Tree Method is a pictorial representation of a recursive method which is in the

form of a tree where at each level nodes are expanded.
2. In general, we consider the second term in recurrence as root.
3. It is useful when the divide & Conquer algorithm is used.
4. It is sometimes difficult to produce a good guess. In Recursion tree, each root and child
represents the cost of a single subproblem.
5. We sum the costs within each of the levels of the tree to obtain a set of pre-level costs
and then sum all pre-level costs to determine the total cost of all levels of the recursion.
6. A Recursion Tree is best used to generate a good guess, which can be verified by the
Substitution Method.
 Recursion Tree Method - Example

𝐶𝑜𝑛𝑠𝑖𝑑𝑒𝑟 𝑎𝑛𝑑 𝑟𝑒𝑐𝑢𝑟𝑟𝑒𝑛𝑐𝑒 𝑟𝑒𝑙𝑎𝑡𝑖𝑜𝑛 , 𝑇 ( 𝑛 ) =2 𝑇 ( )

𝑛
2
+𝑛

The Recursion tree for the above recurrence is

 Recursion Tree Method - Example
2
𝑛
( ) ( ) ( )
2 2 2
𝑛 𝑛 𝑛
4 4 4

( )( ) ( )
2 2 2
𝑛 𝑛 𝑛
2
4 16 16

11 1
 Recursion Tree Method - Example
2
𝑛
( ) ( ) ( )
2 2 2
𝑛 𝑛 𝑛
4 4 4

( )( ) ( )
2 2 2
𝑛 𝑛 𝑛
2
4 16 16

11 1
 Recursion Tree Method - Example
2
𝑛
( ) ( ) ( )
2 2 2
𝑛 𝑛 𝑛
4 4 4

( )( ) ( )
2 2 2
𝑛 𝑛 𝑛
2
4 16 16

11 1
Recursion Tree Method - Example

𝐶𝑜𝑛𝑠𝑖𝑑𝑒𝑟 𝑎𝑛𝑑 𝑟𝑒𝑐𝑢𝑟𝑟𝑒𝑛𝑐𝑒 𝑟𝑒𝑙𝑎𝑡𝑖𝑜𝑛 , 𝑇 ( 𝑛 ) = 4 𝑇 ( )

𝑛
2
+𝑛
Recursion Tree Method - Example

𝐶𝑜𝑛𝑠𝑖𝑑𝑒𝑟 𝑎𝑛𝑑 𝑟𝑒𝑐𝑢𝑟𝑟𝑒𝑛𝑐𝑒𝑟𝑒𝑙𝑎𝑡𝑖𝑜𝑛 , 𝑇 ( 𝑛 ) =𝑇 () ( )

𝑛
3
+𝑇
2𝑛
3
+𝑛
Recursive Tree Example

𝐶𝑜𝑛𝑠𝑖𝑑𝑒𝑟 𝑎𝑛𝑑 𝑟𝑒𝑐𝑢𝑟𝑟𝑒𝑛𝑐𝑒 𝑟𝑒𝑙𝑎𝑡𝑖𝑜𝑛 , 𝑇 ( 𝑛 ) =𝑇 ( )

𝑛
4
+𝑛
Master Method Theorem

 The Master Method provides a method to solve the recurrence equation of the form

where, a ≥ 1 and b > 1 are constant and f(n) is asymptotically positive function. The
master method requires memorization of the three cases, but the solution of many
recurrence can be determined quite easily.
 The above recurrence describes the running time of complexity of an algorithm that
divides a problem of size “n” into “a” subproblems each of size where a & b are
constants. The subproblems are recursively solved in The cost of dividing the problem
& combining the result of the subproblem is described by the function f(n).
 can be interpreted as either the floor or celling value.
Master Method Theorem Statement

Let a ≥ 1 and b > 1 be constant, and f(n) be a non-negative function & let T(n) be defined
on non-negative integers by recurrence

Where we interpret as either , then T(n) can be bounded asymptotically as follows,

1. If
2. If
 Example

Question :
Solve the below recurrence equation using Master Method.

Answer:
Comparing above recurrence equation with we get,
a = 4, b = 2 & f(n) = n.

Applying case 1 of master method theorem we can say,

 Example

Question :
Solve the below recurrence equation using Master Method.

Answer:
Comparing above recurrence equation with we get,
a = 2, b = 2 & f(n) = n.

Applying case 2 of master method theorem we can say,

 Example

Question :
Solve the below recurrence equation using Master Method.

Answer:
Comparing above recurrence equation with we get,
a = 3, b = 4 & f(n) = .

For Sufficiently large value of n,

Applying case 3 of master method theorem we can say,

Selection Sort
Insertion Sort
Insertion Sort v/s Bubble Sort
Proof of Master Method Theorem

Click Here