DESIGN AND ANALYSIS

OF
ALGORITHMS

Dr. T. Anusha
Algorithms

What is the subject about?

• Theoretical study of design and analysis of computer algorithms

• Analysis
• Predict the cost of an algorithm in terms of resources and performance
• Design
• Design algorithms which minimize the cost
What is an Algorithm?

• It is a well-defined computational procedure that takes a value, or set of values, as input and
produces a value, or set of values, as output in a unit amount of time. An algorithm is thus a
sequence of computational steps that transform the input into the output.
Goals of an Algorithm

Goals of an Algorithm

• Always correct
• Always terminates
• Performance

Correctness of an Algorithm

• An algorithm for a computational problem is correct if, for every problem instance
provided as input, it halts, finishes its computing in unit time and outputs the
correct solution.
 FOUR DESIRED PROPERTIES OF ALGORITHMS

Definiteness
• Each instruction must be clear and unambiguous

Finiteness
• The algorithm must terminate

Correctness
• It must provide correct output when presented with legal input

Efficiency
• Each instruction must be very basic so that it can be easily carried out.
Sor ng problem

Input: sequence 〈a1, a2, …, an〉 of numbers.

Output: permutation 〈a'1, a'2, …, a'n〉 such that a'1 ≤ a'2 ≤ … ≤ a'n .

Example:
Input: 8 2 4 9 3 6
Such an input is called as the instance of a problem.
Output: 2 3 4 6 8 9
ti
Sor ng problem

The largest element 7 is bubbled to the top

ti
Sor ng problem
ti
Inser on Sort

An efficient algorithm for sorting a small number of elements

The index “i” indicates the current card being inserted into the hand.
ti
Inser on sort
ti
Verifying the Correctness of Algorithm

• Algorithm Validation
• Process of checking the correctness of algorithms.
• Performed by providing all valid inputs to the algorithm
and checking its outputs with expected results.
• Gives a guarantee that the given algorithm always gives
correct answers.
• The ability of an algorithm to give correct outputs for valid
inputs is called algorithm correctness.
• Program correctness checks whether the algorithm
conforms to the given specifications.
Prove the Correctness of Algorithm
Correctness of Algorithm
Correctness of Algorithm

A loop invariant is a condition that is true at

the beginning and end of every iteration of a
loop in a computer program
 Correctness of Algorithm

A good loop invariant should satisfy three properties:

• Initialization
• The loop invariant must be true before the first execution of the loop.
• Maintenance
• If the invariant is true before an iteration of the loop, it should be true also after
the iteration.

• Termination
• When the loop is terminated the invariant should tell us something useful,
something that helps us understand the algorithm.
 Correctness of Inser on Sort

• At the start of each iteration of the for loop, the subarray A[1: i-1] consists of the elements
originally in A[1: i-1], but in sorted order. ==> Loop Invariant
ti
 Correctness of Inser on Sort

• Initialization
• The first iteration happens when i is 2. A[1: i-1] has only one element, the first one.
• The array having one element is sorted.
• Maintenance
• The iteration moves A[i-1], A[i-2] ,… A[0] by one position to the right, so that A[i] is
placed in correct position.

• Thus A[1: i-1] keeps the original elements but in sorted order.
• Termination
• The loop terminates when i = n+1; then A[1: i] is in sorted order.
Hence the algorithm is correct.
ti
Time Complexity Analysis of Inser on Sort

• Runtime of a program depends on the programming language, computer on which it is

run, dependent libraries and parallel background tasks running during the test.

• This makes it difficult to compare the performances of different algorithms.

• Instead analyse the algorithm in terms of number of instructions that are executed
• The runtime of an algorithm with n number of input is denoted as T(n)
• Let ck be the cost of executing a statement
• Then, if a statement runs for m times, then the cost is m * ck
ti
Time Complexity Analysis of Inser on Sort

ti
Best case for Inser on sort

Best case: The input is already sorted. Here ti is 1.

Best case:
Time complexity is a linear function
of n (input size)
Best case:
T(n)= a * n - b
Where a = c1 + c2 + c4 + c5 + c8 and
b = c2 + c4 + c5 + c8
ti
 Worst case for Inser on sort

Worst case:
Worst case: Input is in descending order. Here ti is i.
T(n)= x * n2 + y * n - z

Worst case:
Time complexity is a quadratic
function of n (input size)
ti
Average case for Inser on sort

• The worst case running time gives the upper bound on the run time.
• Worst case occurs often.
• Looking for a non-existent data in a database
• The average case occurs as often as the worst case.
• If n random numbers are chosen, then half the elements in A[1: i-1] are greater than A[i]
• So, ti is i/2. This again leads to n2 as the lead term just like the worst case.
ti
Worst, Best and Average case
Worst, Best and Average case
Algorithm Analysis

Algorithm
Analysis

Operations Count Asymptotic Recurrence Amortized

Steps Count
Analysis Relations Analysis
Step Count Method

Steps Frequency Total

1 (1) (1)
1 (n+1) (n+1)
1 (n) (n)
1 1 1
2n+3
Step Count Method

getMax(arr, n):
index := 0
LinearSearch(A, n, key)
max := arr[0]
I := 0
while i < n, do for i in range 1 to n - 1, do
if A[i] = key, then if arr[i] > max, then
break
max := arr[i]
end if
done index := i
return i end if
done
return index
Opera on Count Method

Steps Frequency Total

C1 (1) C1
C2 (n+1) C2 * n+ C2

C3 (n) C3 * n
C4 1 C4
Total=c1+c2*(n+1)+c3*n+c4
ti
 is large,= so that means <roughly proportional to when is large= and
Rate of growth
means / Order
<roughly of growth
proportional to of an Algorithm
when is large= We’ll use -notation
informally in this chapter and deûne it precisely in Chapter 3.
We usually consider one algorithm to be more efûcient than another if its worst-
• case
The rate of growth is important for analysing an algorithm for large input size
running time has a lower order of growth. Due to constant factors and lower-
• order
So, theterms, an algorithm
leading whose running time has a higher order of growth might
terms are considered.
take less time for small inputs than an algorithm whose
Worst caserunning time
insertion has
sort: a lower or-
• der
Drop the constant co-effiecients
of growth. But on large enough inputs, an algorithm whose worst-case running
time is , for example, takes less time T(n)=
in x
the * n 2+y*n-z
worst case than an algorithm
• Drop the lower order terms
whose worst-case running time is . Regardless of the constants hidden by
• the -notation, there is always some. number, say , such that for all input sizes
The insertion sort’s leading term is n 2

, the algorithm beats the algorithm in the worst case.

• Greek letter (theta) “Θ” notation is used to denote the order of growth.
• Exercises
Insertion sort has a order of growth of Θ(n 2 ).

2.2-1
Express the function in terms of -notation.
Rate of growth / Order of growth of an Algorithm
Asympto c Analysis

•Goal: to simplify analysis of running time by getting rid of”details”, which may be affected
by specific implementation and hardware
•like“rounding”: 1,000,001=1,000,000
•3n2=n2

•Capturing the essence: how the running time of an algorithm increases with the size of the
input in the limit.
•Asymptotically more efficient algorithms are best for all but small inputs

•50 n log n is O(n log n)

•7n -3 is O(n)
•8n2log n + 5n2+ n is O(n2log n)
ti
Asympto c Analysis

•The “big-Oh” O-Notation

•asymptotic upper bound
•f(n) is O(g(n)), if there exists constants c and n0,
s.t. f(n) <=c g(n) for all n >= n0
•f(n) and g(n) are functions over non-negative
integers
ti
Asympto c Analysis

•The “big-Omega” Ὡ- Notation

•asymptotic lower bound
•f(n) is Ὡ (g(n)) if there exists constants c and n0,
s.t. c g(n) <=f(n) for n >=n0
•Used to describe best-case running times or lower
bounds for algorithmic problems
•E.g., lower-bound for searching in an unsorted
array is Ὡ (n).
ti
Asympto c Analysis

•The “big-Theta” Θ - Notation

•asymptotically tight bound
•Θf(n) is Θ(g(n)) if there exists constants c1, c2, and n0,
s.t. c1 g(n) <=f(n) <=c2 g(n) for n >= n0
•f(n) is Θ(g(n)) if and only if f(n) is O(g(n)) and f(n) is
Θ(g(n))
•O(f(n)) is often misused instead of Θ(f(n))
ti
Asympto c Analysis

•Two more asymptotic notations

•"Little-Oh" notation f(n) is o(g(n))non-tight analogue of Big-Oh
•For every c>0, there should exist n0 ,s.t. f(n) <=c g(n)for n >=n0
•Used for comparisons of running times. If f(n) is o(g(n)), it is said that
g(n) dominates f(n).
•"Little-omega" notation f(n) is w(g(n)) non-tight analogue of Big-Omega
ti
 Constant vs Variable Coefficient
In the generic linear recurrence equation

a t + a t + … + a t = f(n),
0 n 1 n−1 k n−k

the terms ai can be constants or variables.

Based on this fact, one can classify linear recurrence equations into two types:
linear recurrence equations with constant coefficients and those with variable
coefficients.

Consider the following linear recurrence equation:

t=n×t
n n−2

This recurrence equation is dependent on the variable n and does not have
constant coefficients.
Analysis of Framework

Consider he sequence 1, 4, 7, 10, …,

What is the Recurrence Relation?

And what is the Base Condition?

 Analysis of Framework
The sequence 1, 4, 7, 10, …, one can write the recurrence equation as follows:
T(n) = T(n − 1) + 3
T(0) = 1

It can be observed that T(0) = 1 is a base condition.

From this equation, T(1) can be generated as

T(0) + 3 = 4 and
T(2) as T(1) + 4 = 7.

Similarly, all terms of the sequence can be generated.

The preceding equation can be denoted as follows:

tn = tn−1 + 3
t0 = 0
 Example

What is the recurrence equation for the sequence 1000, 2000, 4000,
8000, …?
 Example

Recurrence equation for the sequence 1000, 2000, 4000, 8000, …?

Solution

t0 = 1000

t1 = 2000 = 2 × 1000 = 2 × t0

t2 = 4000 = 2 × 2000 = 2 × t1 = 22 t0

∴, one would guess

tn = 2 × t(n-1)

It can be observed that tn = 2 × t(n-1) is the required recurrence equation of this problem.
 Example

Find the recurrence equation and the initial condition of the following
sequence:

7, 21/4 , 63/16, 189/64 , …

 Example

Find the recurrence equation and the initial condition of the following sequence:
7, 21/4 , 63/16, 189/64 , …

Solution
Let the initial condition be t0 = 7. Let us observe the patterns. Let us calculate the
ratios of the successive elements as follows:

t1 /t0 = 21/4 ÷ 7 = 21 ÷ 28 = 3/4;

t2 /t1 = 63/16 ÷21 4 = 63 /16 x4 /31 = 3/4;
t3 /t2 = 189 /64 ÷ 63/ 16 = 189 /64 × 16 /63 = 3/4
Therefore, one can predict that tn /tn–1 = 3/4
Therefore, tn = tn−1 × 3/4
Thus, one can conclude that the recurrence equation is tn = tn-1 × 3/4.
Techniques for Solving

Guess and verify

Substitution Method

Recurrence tree method

 Guess and Verify Methods
Known by various names such as guess and verify, guess and test and
solution by mathematical induction or method of
substitution(Cormen)

It comprises of two important aspects

Guess: Guess the solution by substituting the different values of n to

generate the sequence from the recurrence equation.
• The closed form can be obtained by verifying the sequences
carefully
• The technique is thus to guess the closed form that may be the
solution of the given recurrence equation.
Verify: The second part of this method is to verify the solution
guessed in the first phase.
• Verification is required because the solution is just a guess.
• Justification is carried out using the techniques of mathematical
induction.
• Based on verification, solutions are established for given
 Example
Solve the recurrence equation tn = tn−1 + 2 t0 = 1 using the guess-and-verify
method.

Solution As said earlier, first make a guess of the solution and then verify it.

Guess: For making a guess, use different values of n in the recurrence equation
as follows:
t0 = 1
t1 = t1−1 + 2 = t0 + 2 = 3
t2 = t2−1 + 2 = t1 + 2 = 5
t3 = t3−1 + 2 = t2 + 2 = 7
:
:
The sequence obtained (1, 3, 5, 7, …) indicates that every term differs from the
previous one by 2. This is an odd-number series. Therefore, one can guess that
the solution for the recurrence equation would be 2n + 1. As this is a non-recursive
formula in terms of n, this can be a solution. To confirm this, one should verify the
guess.
Examples of Input Size
Example
Another Example

T(n) = 2T(n − 1) + 1
Base case: T(0) = 0
 Another Example

T(0)=0, T(1)=1, T(2)=3, T(3)=7, T(4)=15, T(5)=31, T(6)=63

 Substitution method
Involves 2 Steps
• Plug: Substitute Repeatedly
• Chug: Simplify the expressions
 Backward Substitution

The process is continued till the closed form is obtained, which can be confirmed by the observation of the common pattern
that emerges from the process of repeated substitution
Forward Substitution
Recurrence Tree Method
Recurrence Tree Method
Designing Algorithms
• The method used in insertion sort - Increment Method
• Divide and Conquer method is used to design an algorithm with less worst-case run time.
• Divide
• Break the original problem into several sub problems
• Sub problems are similar to original problem
• But they are smaller in size
• Conquer
• Solve the sub problems recursively
• Combine
• Combine the solutions of sub problems to form the solution of the original problem.
• If the base problem is small, it is solved directly, else it is solved recursively.
Designing Algorithms

Merge Sort
Starts with the original array A[1:n]
In each step, it sorts a sub-array, recursing to smaller and smaller sub arrays.
MERGE-SORT A[1 . . n]

1.If n = 1, done.
2.Recursively sort A[ 1 . . ⎡n/2⎤ ] and A[ ⎡n/2⎤+1 . . n ] .
3.“Merge” the 2 sorted lists.

Key subroutine: MERGE

The recursion ends when the sub array can’t be spliced further, i.e, have only one element.
 p q r
Merge Sort 1
12 3 7
2 3 4 5
9 14 6 11 2
6 7 8

divide 1 11
p q r p q r
1 2 3 4 5 6 7 8
12 3 7 9 14 6 11 2

divide 2 6 12 16
p,q r p,q r p,q r p,q r
1 2 3 4 5 6 7 8
12 3 7 9 14 6 11 2

divide 3 4 7 8 13 14 17 18
p,r p,r p,r p,r p,r p,r p,r p,r
1 2 3 4 5 6 7 8
12 3 7 9 14 6 11 2

merge 5 9 15 19
p,q r p,q r p,q r p,q r
1 2 3 4 5 6 7 8
3 12 7 9 6 14 2 11

merge 10 20
p q r p q r
1 2 3 4 5 6 7 8
3 7 9 12 2 6 11 14

merge 21
p q r
1 2 3 4 5 6 7 8
2 3 6 7 9 11 12 14
Merge Rou ne

Merging two sorted sub arrays [1, 4, 7, 11, 12] and [2, 3, 9, 10, 15]
Table 1

L R L R L R L R L R L R L R L R L R

12 15 12 15 12 15 12 15 12 15 12 15 12 15 12 15 12 15

11 10 11 10 11 10 11 10 11 10 11 10 11 10 11

7 9 7 9 7 9 7 9 7 9 9

4 3 4 3 4 3 4

1 2 2

1 2 3 4 7 9 10 11 12 15
ti
 M ERGE
Merge Sort 1
2
// length of
// length of
3 let and be new arrays
4 for to // copy into
5
6 for to // copy into
7
2.3 Designing algorithms 39 8 // indexes the smallest remaining element in
9 // indexes the smallest remaining element in
10 // indexes the location in to ûll
M ERGE -S ORT 11 // As long as each of the arrays and contains an unmerged element,
1 if // zero or one element? // copy the smallest unmerged element back into .
2 return 12 while and
3 // midpoint of 13 if
4 M ERGE -S ORT // recursively sort 14
5 M ERGE -S ORT // recursively sort 15
6 // Merge and into . 16 else
7 M ERGE 17
18
19 // Having gone through one of and entirely, copy the
rays to form sorted subarrays of length , merges those to form sorted subarrays
// remainder of the other to the end of .
of length , and merges those to form the ûnal sorted subarray of length . If
20 while
is not an exact power of , then some divide steps create subarrays whose lengths
21
differ by . (For example, when dividing a subarray of length , one subarray has
length and the other has length .) Regardless of the lengths of the two subarrays 22
being merged, the time to merge a total of items is . 23
24 while
25
2.3.2 Analyzing divide-and-conquer algorithms
26
When an algorithm contains a recursive call, you can often describe its running 27
Time Complexity of Merge Sort