Analysis and Design of Algorithms (ADA)

GTU # 3150703

Unit-2:
Analysis of Algorithm

Dr. Gopi Sanghani

Computer Engineering Department
Darshan Institute of Engineering & Technology, Rajkot
[email protected]
9825621471
 Outline
Looping
 Analysis of Algorithm
 The efficient algorithm
 Average, Best and Worst case analysis
 Asymptotic Notations
 Analyzing control statement
 Loop invariant and the correctness of the algorithm
 Sorting Algorithms and analysis: Bubble sort, Selection sort,
Insertion sort, Shell sort and Heap sort
 Sorting in linear time : Bucket sort, Radix sort and Counting sort
 Amortized analysis
Analysis of Algorithm
Introduction

What is Analysis of an Algorithm?

 Analyzing an algorithm means calculating/predicting the resources that the
algorithm requires.
 Analysis provides theoretical estimation for the required resources of an
algorithm to solve a specific computational problem.
 Two most important resources are computing time (time complexity) and
storage space (space complexity).
Why Analysis is required?
 By analyzing some of the candidate algorithms for a problem, the most efficient
one can be easily identified.

Dr. Gopi Sanghani #3150703 (ADA)  Unit 2 – Analysis of Algorithm 4

Efficiency of Algorithm
 The efficiency of an algorithm is a measure of the amount of resources consumed in solving a
problem of size 𝑛.
 An algorithm must be analyzed to determine its resource usage.
 Two major computational resources are execution time and memory space.
 Memory Space requirement can not be compared directly, so the important resource is
computational time required by an algorithm.
 To measure the efficiency of an algorithm requires to measure its execution time using any of
the following approaches:
1. Empirical Approach: To run it and measure how much processor time is needed.
2. Theoretical Approach: Mathematically computing how much time is needed as a function of input size.

Dr. Gopi Sanghani #3150703 (ADA)  Unit 2 – Analysis of Algorithm 5

How Analysis is Done?

Empirical (posteriori) approach Theoretical (priori) approach

 Programming different competing   Determining mathematically the

techniques & running them on resources needed by each
various inputs using computer. algorithm.
 Implementation of different  Uses the algorithm instead of an
techniques may be difficult. implementation.
 The same hardware and software  The speed of an algorithm can be
environments must be used for determined independent of the
comparing two algorithms. hardware/software environment.
 Results may not be indicative of the  Characterizes running time as a
running time on other inputs not function of the input size considers
included in the experiment. all possible values.

Dr. Gopi Sanghani #3150703 (ADA)  Unit 2 – Analysis of Algorithm 6

Time Complexity
 Time complexity of an algorithm quantifies the amount of time taken by an algorithm to run as
a function of the length of the input.
 Running time of an algorithm depends upon,
1. Input Size
2. Nature of Input
 Generally time grows with the size of input, for example, sorting 100 numbers will take less
time than sorting of 10,000 numbers.
 So, running time of an algorithm is usually measured as a function of input size.
 Instead of measuring actual time required in executing each statement in the code, we consider
how many times each statement is executed.
 So, in theoretical computation of time complexity, running time is measured in terms of number
of steps/primitive operations performed.

Dr. Gopi Sanghani #3150703 (ADA)  Unit 2 – Analysis of Algorithm 7

Problem & Instance
  Instance: An Instance of a problem consists of the input needed to compute the solution to the
problem.
 Example:
Problem: to multiply two positive integers
Instance:
 Instance size: Any integer (generally 𝒏) that in some way measures the number of components
in an instance.
 Examples:
1. Sorting problem: Instance size is number of elements to be sorted.
2. Graph problem: Instance size is number of nodes or edges or both

Dr. Gopi Sanghani #3150703 (ADA)  Unit 2 – Analysis of Algorithm 8

Linear Search
 Suppose, you are given a jar containing some business cards.
 You are asked to determine whether the name “Bill Gates" is in the jar. 
 To do this, you decide to simply go through all the cards one by one.
 How long this takes?
 Can be determined by how many cards are in the jar, i.e., Size of Input.
 Linear search is a method for finding a particular value from the given list.
 The algorithm checks each element, one at a time and in sequence, until the desired element is
found.
 Linear search is the simplest search algorithm.
 It is a special case of brute-force search.

Dr. Gopi Sanghani #3150703 (ADA)  Unit 2 – Analysis of Algorithm 9

Linear Search – Example
  Search for in given array

Comparing value of ith index with the given element one by one, until we get the required
element or end of the array
Step 1: i=1 Step 3: i=3

i i
 𝟐 ≠𝟏  𝟑 ≠𝟏
Step 2: i=2 Step 4: i=4

𝟏
 

i i
 𝟗 ≠ 𝟏
Element found at ith index, i=4
Dr. Gopi Sanghani #3150703 (ADA)  Unit 2 – Analysis of Algorithm 10
Linear Search - Algorithm
# Input : Array A, element x
# Output : First index of element x in A or -1 if not found

Algorithm: Linear_Search
for i = 1 to last index of A
if A[i] equals element x
return i
return -1

Dr. Gopi Sanghani #3150703 (ADA)  Unit 2 – Analysis of Algorithm 11

Linear Search - Analysis
 The required element in the given array can be found at,

Case 1: element 2 which is at Case 2: element 3 anywhere Case 3: element 7 at last

the first position so minimum after the first position so, an position or element does not
comparison is required average number of found at all, maximum
comparison is required comparison is required

Best Case Average Case Worst Case

Worst Case

 Search  𝟑
for  𝟕

Best Case
Average Case

Dr. Gopi Sanghani #3150703 (ADA)  Unit 2 – Analysis of Algorithm 13

Analysis of Algorithm

Best Case Average Case Worst Case

Resource usage is minimum Resource usage is average Resource usage is maximum

Algorithm’s behavior under Algorithm’s behavior under Algorithm’s behavior under

optimal condition random condition the worst condition
Minimum number of steps Average number of steps or Maximum number of steps
or operations operations or operations
Lower bound on running Average bound on running Upper bound on running
time time time
Generally do not occur in Average and worst-case performances are the most used in
real algorithm analysis.

Dr. Gopi Sanghani #3150703 (ADA)  Unit 2 – Analysis of Algorithm 14

Book Finder Example  Suppose, you are writing a program to find a
book from the shelf.
 For any required book, it will start checking
books one by one from the bottom.
 If you wanted Harry Potter 3, it would only
take 3 actions (or tries) because it’s the third
book in the sequence.
 If Harry Potter 7 — it’s the last book so it
would have to check all 7 books.
 What if there are total 10 books? How about
10,00,000 books? It would take 1 million tries.

Dr. Gopi Sanghani #3150703 (ADA)  Unit 2 – Analysis of Algorithm 15

Number Sorting - Example
 Suppose you are sorting numbers in Ascending / Increasing order.
 The initial arrangement of given numbers can be in any of the following three orders.

Case 1: Numbers are already Case 2: Numbers are Case 3: Numbers are initially
in required order, i.e., randomly arranged initially. arranged in Descending order
Ascending order Some numbers will change so, all numbers will change
No change is required their position their position

Best Case Average Case Worst Case

Dr. Gopi Sanghani #3150703 (ADA)  Unit 2 – Analysis of Algorithm 17

Best, Average, & Worst Case

Problem Best Case Average Case Worst Case

Linear Search Element at the first Element in any of the Element at last position or
position middle positions not present
Book Finder The first book Any book in-between The last book

Sorting Already sorted Randomly arranged Sorted in reverse order

Dr. Gopi Sanghani #3150703 (ADA)  Unit 2 – Analysis of Algorithm 18

Asymptotic Notations
Introduction
 The theoretical (priori) approach of analyzing an algorithm to measure the efficiency does not
depend on the implementation of the algorithm.
 In this approach, the running time of an algorithm is describes as Asymptotic Notations.
 Computing the running time of algorithm’s operations in mathematical units of computation
and defining the mathematical formula of its run-time performance is referred to as Asymptotic
Analysis.
 An algorithm may not have the same performance for different types of inputs. With the
increase in the input size, the performance will change.
 Asymptotic analysis accomplishes the study of change in performance of the algorithm with
the change in the order of the input size.
 Using Asymptotic analysis, we can very well define the best case, average case, and worst case
scenario of an algorithm.

Dr. Gopi Sanghani #3150703 (ADA)  Unit 2 – Analysis of Algorithm 20

Asymptotic Notations
  Asymptotic notations are mathematical notations used to represent the time complexity of
algorithms for Asymptotic analysis.
 Following are the commonly used asymptotic notations to calculate the running time
complexity of an algorithm.
1. Notation
2. Notation
3. Notation
 This is also known as an algorithm’s growth rate.
 Asymptotic Notations are used,
1. To characterize the complexity of an algorithm.
2. To compare the performance of two or more algorithms solving the same problem.

Dr. Gopi Sanghani #3150703 (ADA)  Unit 2 – Analysis of Algorithm 21

 1. -Notation (Big notation) (Upper Bound)
  The notation is the formal way to express the upper bound of an algorithm's running time.
 It measures the worst case time complexity or the longest amount of time an algorithm can
possibly take to complete.
 For a given function , we denote by the set of functions,

 
= { : there exist positive constants and such that for all }

Dr. Gopi Sanghani #3150703 (ADA)  Unit 2 – Analysis of Algorithm 22

   
Big() Notation  is an asymptotically upper bound for .

 implies:
𝒄  . 𝒈(𝒏)
 An upper bound of an algorithm defines the
 𝒇
maximum time required, we can always solve
(𝒏)
the problem in at most time.

 Time taken by a known algorithm to solve a

problem with worse case input gives the
upper bound.
 𝒏
 𝒏𝟎
𝒇 (𝒏)=𝑶(𝒈(𝒏))
 

Dr. Gopi Sanghani #3150703 (ADA)  Unit 2 – Analysis of Algorithm 23

 2. -Notation (Omega notation) (Lower Bound)
  Big Omega notation () is used to define the lower bound of any algorithm or we can say the best
case of any algorithm.
 This always indicates the minimum time required for any algorithm for all input values,
therefore the best case of any algorithm.
 When a time complexity for any algorithm is represented in the form of big-Ω, it means that the
algorithm will take at least this much time to complete it's execution. It can definitely take more
time than this too.
 For a given function 𝑔(𝑛), we denote by 𝑔(𝑛)) the set of functions,

 
there exist positive constants and such that for all

Dr. Gopi Sanghani #3150703 (ADA)  Unit 2 – Analysis of Algorithm 24

   
Big() Notation  is an asymptotically lower bound for

 implies:
 𝒇 (𝒏)
 A lower bound of an algorithm defines the
𝒄  . 𝒈(𝒏) minimum time required, it is not possible to
have any other algorithm (for the same
problem) whose time complexity is less than
for random input.

 𝒏
 𝒏𝟎
𝒇 (𝒏)=𝜴(𝒈(𝒏))
 

Dr. Gopi Sanghani #3150703 (ADA)  Unit 2 – Analysis of Algorithm 25

 3. -Notation (Theta notation) (Same order)
  The notation is the formal way to enclose both the lower bound and the upper bound of an
algorithm's running time.
 Since it represents the upper and the lower bound of the running time of an algorithm, it is used
for analyzing the average case complexity of an algorithm.
 The time complexity represented by the Big- notation is the range within which the actual
running time of the algorithm will be.
 So, it defines the exact Asymptotic behavior of an algorithm.
 For a given function 𝑔(𝑛), we denote by θ(𝑔(𝑛)) the set of functions,

 
there exist positive constants

Dr. Gopi Sanghani #3150703 (ADA)  Unit 2 – Analysis of Algorithm 26

   
-Notation  is a set, we can write to indicate that is a
𝒄  𝟐 . 𝒈(𝒏) member of

 is an asymptotically tight bound for

 𝒇 (𝒏)
 implies:
𝒄  𝟏 . 𝒈(𝒏)

 𝒏
 𝒏𝟎
𝒇 (𝒏)=𝜽 (𝒈 (𝒏))
 

Dr. Gopi Sanghani #3150703 (ADA)  Unit 2 – Analysis of Algorithm 27

Asymptotic Notations
1. O-Notation (Big O notation) (Upper Bound)

  = { : there exist positive constants and such thatfor all} 𝐟  (𝐧)=𝐎(𝐠(𝐧))

2. Ω-Notation (Omega notation) (Lower Bound)

  = { : there exist positive constants and such that for all } 𝐟  ( 𝐧 )=Ω(𝐠(𝐧))

3. θ-Notation (Theta notation) (Same order)

  = { : there exist positive constants , andsuch that for all } 𝐟  (𝐧)=𝛉 (𝐠 (𝐧))

Dr. Gopi Sanghani #3150703 (ADA)  Unit 2 – Analysis of Algorithm 28

Asymptotic Notations – Examples
  Example 1:   Example 2:
and and

Algo. 1 running Algo. 2 running Algo. 1 running Algo. 2 running

time time time time

𝑓 ( 𝑛 ) ≥ 𝑔 ( 𝑛 ) ⟹ 𝑓 ( 𝑛 ) =Ω(𝑔 (𝑛))
  𝑓 ( 𝑛 ) ≤ 𝑔 ( 𝑛 ) ⟹ 𝑓 ( 𝑛 ) =O(𝑔 (𝑛))
 

1 1 1 1 1 1
2 4 2 2 2 4
3 9 3 3 3 9
4 16 4 4 4 16
5 25 5 5 5 25

Dr. Gopi Sanghani #3150703 (ADA)  Unit 2 – Analysis of Algorithm 29

Asymptotic Notations – Examples
  Example 3: and

𝑓 ( 𝑛 ) ≤ 𝑔 ( 𝑛 ) ⟹ 𝑓 ( 𝑛 ) =O(𝑔 (𝑛))
 

   
1 1 2
2 4 4
3 9 8
4 16 16  
Here for ,
5 25 32
6 36 64
7 49 128

Dr. Gopi Sanghani #3150703 (ADA)  Unit 2 – Analysis of Algorithm 30

   Example 4:
Asymptotic Notations – Examples is in the order of , or
is order , or

𝒇 (𝒏)=𝑶(𝒈(𝒏))
 
g (n)=n +1
2
Value of function 

f(n)=30n+8
 In general, any function is faster-growing than
any function.

  Base value
Increasing n 

Dr. Gopi Sanghani #3150703 (ADA)  Unit 2 – Analysis of Algorithm 31

Common Orders of Magnitude

4 2 8 16 64 16 24
4 2 8 16 64 16 24
16 4 64 256 4096 65536 2.09 x 1013
16 4 64 256 4096 65536 2.09 x 1013
64 6 384 4096 262144 1.84 × 1019 1.26 x 1029
64 6 384 4096 262144 1.84 × 1019 1.26 x 1029
256 8 2048 65536 16777216 1.15 × 1077
256 8 2048 65536 167772169 1.15 × 10308
77
1024 10 10240 1048576 1.07 × 10 1.79 × 10
1024 10 10240 1048576 1.07 × 10109 1.79 ×1233
10308
4096 12 49152 16777216 6.87 × 10 10
4096 12 49152 16777216 6.87 × 1010 101233

Dr. Gopi Sanghani #3150703 (ADA)  Unit 2 – Analysis of Algorithm 32

Asymptotic Notations in Equations
  Consider an example of buying elephants and goldfish:
Cost = cost_of_elephants + cost_of_goldfish
Negligible
Cost ≈ cost_of_elephants (approximation)

 Maximum Rule: Let, the max rule says that:

𝑂( 𝑓(𝑛)+𝑔(𝑛))=𝑂(max⁡(𝑓(𝑛),𝑔(𝑛)))

1. - is

 The low order terms in a function are relatively insignificant for large 𝒏

Dr. Gopi Sanghani #3150703 (ADA)  Unit 2 – Analysis of Algorithm 34

Exercises
1.  
 Express the function in terms of θ notation.
2. Express in terms of O notation.
3. Express in terms of O notation.
4. Prove or disprove (i) Is 2n+1 = O(2n) (ii) Is 22n = O(2n)
5. Check the correctness for the following equality,
6. Find θ notation for the following function

7. Find O notation for the following function

8. Find Ω notation for the following function

Dr. Gopi Sanghani #3150703 (ADA)  Unit 2 – Analysis of Algorithm 35

Analyzing Control Statements
For Loop
# Input : int A[n], array of n integers
# Output : Sum of all numbers in array A

Algorithm: int Sum(int A[], int n)

{
int s=0; n+1
for (int i=0; i<n; i++)
1 s = s + A[i];
return s;
} n

Total time taken = n+1+n+2 = 2n+3

Time Complexity f(n) = 2n+3
Dr. Gopi Sanghani #3150703 (ADA)  Unit 2 – Analysis of Algorithm 37
Running Time of Algorithm
  The time complexity of the algorithm is :
 Estimated running time for different values of 𝑛 :

 𝒏=𝟏𝟎   steps
 𝒏=𝟏𝟎𝟎   steps
 𝒏=𝟏𝟎𝟎𝟎   steps

𝒏=𝟏𝟎𝟎𝟎𝟎
    steps

 As 𝑛 grows, the number of steps grow in linear proportion to 𝑛 for the given algorithm Sum.
 The dominating term in the function of time complexity is : As gets large, the becomes
insignificant.
 The time is linear in proportion to .

Dr. Gopi Sanghani #3150703 (ADA)  Unit 2 – Analysis of Algorithm 38

Analyzing Control Statements
Example 1: Example 3:
   𝐜    𝒄 𝟏 ( 𝒏 +𝟏 )
 Statement is executed once only 𝒄
  𝟐 𝒏 ( 𝒏+𝟏 )  
 So, The execution time is some
𝒄  𝟑 ∗ 𝒏 ∗𝒏
constant
 Analysis
Example 2:
    𝐜
 ∗ ( 𝒏+𝟏 )
𝟏

    𝐜
∗ (𝟐
𝒏)
 Total time is denoted as,

Dr. Gopi Sanghani #3150703 (ADA)  Unit 2 – Analysis of Algorithm 39

Analyzing Control Statements
Example 4: Example 6:
   
𝟐
  (𝒏 )
𝜽

 𝜽 ( 𝒏 )

 
𝒕 ( 𝒏 ) =𝜽 ( 𝒏𝟑 )
 𝜽 ( 𝟏 )
printf(“sum is now %d”,)
Example 5:
   

  𝟔
𝒕 ( 𝒏 ) =𝜽 ( 𝒏 )
Dr. Gopi Sanghani #3150703 (ADA)  Unit 2 – Analysis of Algorithm 40
Sorting Algorithms
Bubble Sort, Selection Sort, Insertion Sort
Introduction
 Sorting is any process of arranging items systematically or arranging items in a sequence
ordered by some criterion.
 Applications of Sorting
1. Phone Bill: the calls made are date wise sorted.
2. Bank statement or Credit card Bill: transactions made are date wise sorted.
3. Filling forms online: “select country” drop down box will have the name of countries sorted in Alphabetical
order.
4. Online shopping: the items can be sorted price wise, date wise or relevance wise.
5. Files or folders on your desktop are sorted date wise.

Dr. Gopi Sanghani #3150703 (ADA)  Unit 2 – Analysis of Algorithm 42

Bubble Sort – Example

Sort the following array in Ascending order

45 34 56 23 12
Pass 1 :

34
45 swap 34 34 34
34
45 45 45 45
56 56 56
23 23

swap
23 23 23
56 56
12

swap
12 12 12 12
56

Dr. Gopi Sanghani #3150703 (ADA)  Unit 2 – Analysis of Algorithm 43

Bubble Sort – Example
Pass 2 : Pass 3 : Pass 4 :

34 34 34 23
34 23 12
23

swap
swap
45 45
23 23 23
34 34
12 12
23

swap
swap
23 23
45 45
12 12 12
34 34

swap
12 12 12
45 45 45 45
56 56 56 56 56 56

Dr. Gopi Sanghani #3150703 (ADA)  Unit 2 – Analysis of Algorithm 44

Bubble Sort - Algorithm
# Input: Array A
# Output: Sorted array A

Algorithm: Bubble_Sort(A)
for i ← 1 to n-1 do 𝛉  ( 𝐧 )
for j ← 1 to n-i do
if A[j] > A[j+1] then
temp ← A[j]
swap(A[j], A[j+1])   ( 𝐧𝟐 )
𝛉
A[j] ← A[j+1]
A[j+1] ← temp

Dr. Gopi Sanghani #3150703 (ADA)  Unit 2 – Analysis of Algorithm 45

Bubble Sort
  It is a simple sorting algorithm that works by comparing each pair of adjacent items and
swapping them if they are in the wrong order.
 The pass through the list is repeated until no swaps are needed, which indicates that the list is
sorted.
 As it only uses comparisons to operate on elements, it is a comparison sort.
 Although the algorithm is simple, it is too slow for practical use.
 The time complexity of bubble sort is

Dr. Gopi Sanghani #3150703 (ADA)  Unit 2 – Analysis of Algorithm 46

Bubble Sort Algorithm – Best Case Analysis
# Input: Array A Pass 1 : i = 1
# Output: Sorted array A
12 j = 1
Algorithm: Bubble_Sort(A)
23 j = 2
int flag=1; 34 j = 3
for i ← 1 to n-1 do Condition never 45 j = 4
becomes true
for j ← 1 to n-i do 59
if A[j] > A[j+1] then   Best case time
flag = 0; complexity =
swap(A[j],A[j+1])
if(flag == 1)
cout<<"already sorted"<<endl
break;
Dr. Gopi Sanghani #3150703 (ADA)  Unit 2 – Analysis of Algorithm 47
Selection Sort – Example 1
Sort the following elements in Ascending order
5 1 12 -5 16 2 12 14

Step 1 :
Unsorted Array
5 1 12 -5 16 2 12 14
1 2 3 4 5 6 7 8

Step 2 :
 Minj denotes the current index and Minx is the value stored
Unsorted Array (elements 2 to 8) at current index.
 So, Minj = 1, Minx = 5
-5
5 1 12 -5
5 16 2 12 14  Assume that currently Minx is the smallest value.
 Now find the smallest value from the remaining entire
1 2 3 4 5 6 7 8
Unsorted array.
Swap Index = 4, value = -5
Dr. Gopi Sanghani #3150703 (ADA)  Unit 2 – Analysis of Algorithm 48
Selection Sort – Example 1

Step 3 :
Unsorted Array (elements 3 to 8)  Now Minj = 2, Minx = 1
 Find min value from remaining
-5 1 12 5 16 2 12 14 unsorted array
1 2 3 4 5 6 7 8
Index = 2, value = 1

No Swapping as min value is already at right place

Step 4 :
Unsorted Array  Minj = 3, Minx = 12
(elements 4 to 8)  Find min value from remaining
unsorted array
-5 1 12
2 5 16 12
2 12 14 Index = 6, value = 2
1 2 3 4 5 6 7 8

Swap

Dr. Gopi Sanghani #3150703 (ADA)  Unit 2 – Analysis of Algorithm 49

Selection Sort – Example 1

Step 5 : Unsorted Array

 Now Minj = 4, Minx = 5
(elements 5 to 8)
 Find min value from remaining
unsorted array
-5 1 2 5 16 12 12 14
1 2 3 4 5 6 7 8 Index = 4, value = 5

No Swapping as min value is already at right place

Step 6 :
 Minj = 5, Minx = 16
 Find min value from remaining
Unsorted Array unsorted array
(elements 6 to 8)
Index = 6, value = 12
-5 1 2 5 12
16 16
12 12 14
1 2 3 4 5 6 7 8

Swap
Dr. Gopi Sanghani #3150703 (ADA)  Unit 2 – Analysis of Algorithm 50
Selection Sort – Example 1

Step 7 :
Unsorted Array  Now Minj = 6, Minx = 16
(elements 7 to 8)  Find min value from remaining
unsorted array
-5 1 2 5 12 12
16 16
12 14
1 2 3 4 5 6 7 8 Index = 7, value = 12

Swap

Step 8 :
Unsorted Array  Minj = 7, Minx = 16
(element 8)  Find min value from remaining
unsorted array
-5 1 2 5 12 12 14
16 16
14
Index = 8, value = 14
1 2 3 4 5 6 7 8

Swap The entire array is sorted now.

Dr. Gopi Sanghani #3150703 (ADA)  Unit 2 – Analysis of Algorithm 51
Selection Sort
  Selection sort divides the array or list into two parts,
1. The sorted part at the left end
2. and the unsorted part at the right end.
 Initially, the sorted part is empty and the unsorted part is the entire list.
 The smallest element is selected from the unsorted array and swapped with the leftmost
element, and that element becomes a part of the sorted array.
 Then it finds the second smallest element and exchanges it with the element in the second
leftmost position.
 This process continues until the entire array is sorted.
 The time complexity of selection sort is

Dr. Gopi Sanghani #3150703 (ADA)  Unit 2 – Analysis of Algorithm 52

Selection Sort - Algorithm
# Input: Array A
# Output: Sorted array A

Algorithm: Selection_Sort(A)
for i ← 1 to n-1 do 𝛉  ( 𝐧 )
minj ← i;
minx ← A[i];
for j ← i + 1 to n do
if A[j] < minx then
  ( 𝐧𝟐 )
𝛉
minj ← j;
minx ← A[j];
A[minj] ← A[i];
A[i] ← minx;

Dr. Gopi Sanghani #3150703 (ADA)  Unit 2 – Analysis of Algorithm 53

Selection Sort – Example 2
Algorithm: Selection_Sort(A)
Pass 1 :
for i ← 1 to n-1 do
minj ← i; minx ← A[i]; i = 1
for j ← i + 1 to n do minj ← 12
if A[j] < minx then
34
minx ← 45 No Change
minj ← j ; minx ← A[j];
A[minj] ← A[i]; j = 2 3
A[i] ← minx; A[j] = 34
56
Sort in Ascending order

45 34 56 23 12
1 2 3 4 5

Dr. Gopi Sanghani #3150703 (ADA)  Unit 2 – Analysis of Algorithm 54

Selection Sort – Example 2
Algorithm: Selection_Sort(A)
Pass 1 :
for i ← 1 to n-1 do
minj ← i; minx ← A[i]; i = 1
for j ← i + 1 to n do 4
minj ← 2
5
if A[j] < minx then
minx ← 34
23
12
minj ← j ; minx ← A[j];
A[minj] ← A[i]; j = 2 3 4 5
A[i] ← minx; A[j] = 12
23
Sort in Ascending order Unsorted Array

45 34 56 23 12 12
45 34 56 23 45
12
1 2 3 4 5 1 2 3 4 5

Swap
12
45 23
34 34
56 34
45
56
23 56
45
Dr. Gopi Sanghani #3150703 (ADA)  Unit 2 – Analysis of Algorithm 55
Insertion Sort – Example
Sort the following elements in Ascending order
5 1 12 -5 16 2 12 14

Step 1 :
Unsorted Array
5 1 12 -5 16 2 12 14
1 2 3 4 5 6 7 8
Step 2 :

  𝒋 𝒊=𝟐
  , 𝒙=𝟏  𝒋=𝒊 – 𝟏 𝒂𝒏𝒅 𝒋>𝟎
51 1 12 -5 16 2 12 14  while do
1 2 3 4 5 6 7 8

Shift down

Dr. Gopi Sanghani #3150703 (ADA)  Unit 2 – Analysis of Algorithm 56

Insertion Sort – Example

Step 3 :
  𝒋 𝒊=𝟑
  , 𝒙=𝟏𝟐  𝒋=𝒊 – 𝟏 𝒂𝒏𝒅 𝒋>𝟎
1 5 12 -5 16 2 12 14  while do
1 2 3 4 5 6 7 8

No Shift will take place

Step 4 :
𝒊=𝟒
  , 𝒙=−𝟓  𝒋=𝒊 – 𝟏 𝒂𝒏𝒅 𝒋>𝟎
 while do
  𝒋   𝒋
-5
1 5 12 -5 16 2 12 14
1 2 3 4 5 6 7 8
Shift down
Shift down
Shift down

Dr. Gopi Sanghani #3150703 (ADA)  Unit 2 – Analysis of Algorithm 57

Insertion Sort – Example

Step 5 :
  𝒋 𝒊=𝟓
  , 𝒙=𝟏𝟔  𝒋=𝒊 – 𝟏 𝒂𝒏𝒅 𝒋>𝟎
-5 1 5 12 16 2 12 14  while do
1 2 3 4 5 6 7 8

No Shift will take place

Step 6 :
𝒊=𝟔
  , 𝒙=𝟐  𝒋=𝒊 – 𝟏 𝒂𝒏𝒅 𝒋>𝟎
  𝒋   𝒋  while do

-5 1 52 12 16 2 12 14
1 2 3 4 5 6 7 8

Shift down Shift down

Shift down

Dr. Gopi Sanghani #3150703 (ADA)  Unit 2 – Analysis of Algorithm 58

Insertion Sort – Example

Step 7 :
  𝒋 𝒊=𝟕
  , 𝒙=𝟏𝟐  𝒋=𝒊 – 𝟏 𝒂𝒏𝒅 𝒋>𝟎
-5 1 2 5 12 12 14
12 16  while do
1 2 3 4 5 6 7 8

Shift down

Step 8 :
𝒊=𝟖
  , 𝒙=𝟏𝟒  𝒋=𝒊 – 𝟏 𝒂𝒏𝒅 𝒋>𝟎
  𝒋  while do

-5 1 2 5 12 12 14
16 14
1 2 3 4 5 6 7 8

Shift down The entire array is sorted now.

Dr. Gopi Sanghani #3150703 (ADA)  Unit 2 – Analysis of Algorithm 59
Insertion Sort - Algorithm
# Input: Array T
# Output: Sorted array T

Algorithm: Insertion_Sort(T[1,…,n])
for i ← 2 to n do
x ← T[i];
𝛉  ( 𝐧 )
j ← i – 1;
while x < T[j] and j > 0 do
T[j+1] ← T[j];   𝟐
𝛉 (𝐧 )
j ← j – 1;
T[j+1] ← x;

Dr. Gopi Sanghani #3150703 (ADA)  Unit 2 – Analysis of Algorithm 60

Insertion Sort Algorithm – Best Case Analysis
# Input: Array T Pass 1 :
# Output: Sorted array T
12
23 i=2 x=23 T[j]=12
Algorithm: Insertion_Sort(T[1,…,n])
34 i=3 x=34 T[j]=23
for i ← 2 to n do
𝛉  ( 𝐧 ) 45 i=4 x=45 T[j]=34
x ← T[i]; 59 i=5 x=59 T[j]=45
j ← i – 1;
while x < T[j] and j > 0 do  The best case time complexity
T[j+1] ← T[j]; of Insertion sort is
j ← j – 1; The average and worst case
T[j+1] ← x; time complexity of Insertion
sort is

Dr. Gopi Sanghani #3150703 (ADA)  Unit 2 – Analysis of Algorithm 61

Heap & Heap Sort Algorithm
Introduction
 A heap data structure is a binary tree with the following two properties.
1. It is a complete binary tree: Each level of the tree is completely filled, except possibly the bottom level. At
this level it is filled from left to right.
2. It satisfies the heap order property: the data item stored in each node is greater than or equal to the
data item stored in its children node.

a a 9 9

b c b c 6 7 6 7

d e f d e f 2 4 8 2 4 1

Binary Tree but not a Heap Complete Binary Tree - Heap Not a Heap Heap

Dr. Gopi Sanghani #3150703 (ADA)  Unit 2 – Analysis of Algorithm 63

Array Representation of Heap
 Heap can be implemented using an Array.
 An array 𝐴 that represents a heap is an object with two attributes:
1. 𝑙𝑒𝑛𝑔𝑡ℎ[𝐴], which is the number of elements in the array, and
2. ℎ𝑒𝑎𝑝−𝑠𝑖𝑧𝑒[𝐴], the number of elements in the heap stored within array 𝐴

16

14 10

Array representation of heap

8 7 9 3

2 4 1 Heap 16 14 10 8 7 9 3 2 4 1

Dr. Gopi Sanghani #3150703 (ADA)  Unit 2 – Analysis of Algorithm 64

Array Representation of Heap
 In the array 𝐴, that represents a heap
1. length[𝐴] = heap-size[𝐴]
2. For any node 𝒊 the parent node is 𝒊/𝟐
3. For any node 𝒋, its left child is 𝟐𝒋 and right child is 𝟐𝒋+𝟏

 For node , parent node is

𝟏 
16
 Fornode ,
𝟐  𝟑 
14 10 Left child node is node
Right child is node
𝟒  𝟓  𝟔  𝟕 
8 7 9 3

𝟖  𝟗  𝟏𝟎
 
2 4 1 Heap 16 14 10 8 7 9 3 2 4 1

Dr. Gopi Sanghani #3150703 (ADA)  Unit 2 – Analysis of Algorithm 65

Types of Heap
1. Max-Heap − Where the value of the root node is 9
greater than or equal to either of its children.

6 7

2 4 1

1 2. Min-Heap − Where the value of the root node is less than

or equal to either of its children.
2 4

6 7 9
Dr. Gopi Sanghani #3150703 (ADA)  Unit 2 – Analysis of Algorithm 66
Introduction to Heap Sort
1. Build the complete binary tree using given elements.
2. Create Max-heap to sort in ascending order.
3. Once the heap is created, swap the last node with the root node and delete the last node from
the heap.
4. Repeat step 2 and 3 until the heap is empty.

Dr. Gopi Sanghani #3150703 (ADA)  Unit 2 – Analysis of Algorithm 67

Heap Sort – Example 1
Sort the following elements in Ascending order
4 10 3 5 1

Step 1 : Create Complete Binary Tree

4

4 10 3 5 1
10 3

Now, a binary tree is 5 1

created and we have to
convert it into a Heap.

Dr. Gopi Sanghani #3150703 (ADA)  Unit 2 – Analysis of Algorithm 68

Heap Sort – Example 1
Sort the following elements in Ascending order
4 10 3 5 1

Step 2 : Create Max Heap

4
10 10 is greater than 4
So, swap 10 & 4
4
10 10
4 3 5 1
4
10 3
Swap

In a Max Heap, parent node 5 1

is always greater than or
equal to the child nodes.

Dr. Gopi Sanghani #3150703 (ADA)  Unit 2 – Analysis of Algorithm 69

Heap Sort – Example 1
Sort the following elements in Ascending order
4 10 3 5 1

Step 2 : Create Max Heap

10 5 is greater than 4
So, swap 5 & 4
10 54 3 45 1
5
4 3
Swap

In a Max Heap, parent node 4

5 1
is always greater than or
equal to the child nodes. Max Heap is created

Dr. Gopi Sanghani #3150703 (ADA)  Unit 2 – Analysis of Algorithm 70

Heap Sort – Example 1
Sort the following elements in Ascending order
4 10 3 5 1

Step 3 : Apply Heap Sort

1
10

10
1 5 3 4 1
10
5 3
Swap

1. Swap the first and the 4 10

1
last nodes and
2. Delete the last node.

Dr. Gopi Sanghani #3150703 (ADA)  Unit 2 – Analysis of Algorithm 71

Heap Sort – Example 1
Sort the following elements in Ascending order
4 10 3 5 1

Step 3 : Apply Heap Sort

5
1 Max Heap Property is
violated so, create a
15 451 3 41 10 Max Heap again.
1
4
5 3
Swap

1
4

Dr. Gopi Sanghani #3150703 (ADA)  Unit 2 – Analysis of Algorithm 72

Heap Sort – Example 1
Sort the following elements in Ascending order
4 10 3 5 1

Step 3 : Apply Heap Sort

1
5 Max Heap is created

51 4 3 15 10
4 3
Swap

1. Swap the first and the 5

1
last nodes and
2. Delete the last node.

Dr. Gopi Sanghani #3150703 (ADA)  Unit 2 – Analysis of Algorithm 73

Heap Sort – Example 1
Sort the following elements in Ascending order
4 10 3 5 1

Step 3 : Apply Heap Sort

34
1 Create Max Heap
again
134 41 43 5 10 Max Heap is created
1
4 34
Swap

1. Swap the first and the

last nodes and
2. Delete the last node.

Dr. Gopi Sanghani #3150703 (ADA)  Unit 2 – Analysis of Algorithm 74

Heap Sort – Example 1
Sort the following elements in Ascending order
4 10 3 5 1

Step 3 : Apply Heap Sort

1
3 Already a Max Heap

31 13 4 5 10
3
1
Swap

1. Swap the first and the

last nodes and
2. Delete the last node.

Dr. Gopi Sanghani #3150703 (ADA)  Unit 2 – Analysis of Algorithm 75

Heap Sort – Example 1
Sort the following elements in Ascending order
4 10 3 5 1

Step 3 : Apply Heap Sort

1 Already a Max Heap

1 3 4 5 10

Remove the last element

from heap and the
sorting is over.

Dr. Gopi Sanghani #3150703 (ADA)  Unit 2 – Analysis of Algorithm 76

Heap Sort – Example 2
 Sort given element in ascending order using heap sort. 19, 7, 16, 1, 14, 17

19 7
14 16
17 1 14
7 17
16

Step 1: Create binary tree Step 2: Create Max-heap

19 19

7 16 14
7 16
17

1 14 17 1 14
7 16
17

Dr. Gopi Sanghani #3150703 (ADA)  Unit 2 – Analysis of Algorithm 77

Heap Sort – Example 2
Step 3 Step 4

19
16 14 17 1 7 16
19 17
16 14 17
16 1 7 19

19
16 16
17 Create Max-heap
Swap &
remove
14 17 the last 14 17
16
element

1 7 16
19 1 7

Dr. Gopi Sanghani #3150703 (ADA)  Unit 2 – Analysis of Algorithm 78

Heap Sort – Example 2
Step 5 Step 6

17
7 14 16 1 7
17 19 7
16 14 16
7 1 17 19

17
7 16
7 Create Max-heap
Swap &
remove
14 16 the last 14 7
16
element

1 17
7 1

Dr. Gopi Sanghani #3150703 (ADA)  Unit 2 – Analysis of Algorithm 79

Heap Sort – Example 2
Step 7 Step 8

16
1 14 7 1
16 17 19 1
14 14
1 7 16 17 19

16
1 1
14 Create Max-heap
Swap &
remove
14 7 the last 14
1 7
element

16
1

Dr. Gopi Sanghani #3150703 (ADA)  Unit 2 – Analysis of Algorithm 80

Heap Sort – Example 2
Step 9 Step 10

14
7 1 14
7 16 17 19 17 17 14 16 17 19

14
7 Swap & 71 Already a Max-heap
remove the Swap & remove the last
last element element
1 7
14 1
7

Step 11
1 7 14 16 17 19

Remove the
1 last element
The entire array is sorted now.

Dr. Gopi Sanghani #3150703 (ADA)  Unit 2 – Analysis of Algorithm 81

Exercises
 Sort the following elements using Heap Sort Method.
1. 34, 18, 65, 32, 51, 21
2. 20, 50, 30, 75, 90, 65, 25, 10, 40

 Sort the following elements in Descending order using Hear Sort Algorithm.
1. 65, 77, 5, 23, 32, 45, 99, 83, 69, 81

Dr. Gopi Sanghani #3150703 (ADA)  Unit 2 – Analysis of Algorithm 82

Heap Sort – Algorithm
#  Input: Array A

# Output: Sorted array A

Algorithm: Heap_Sort(A[1,…,n])
BUILD-MAX-HEAP(A)
for i length[A] downto 2
do exchange A[1] A[i]
heap-size[A] heap-size[A] – 1
MAX-HEAPIFY(A, 1, n)

Dr. Gopi Sanghani #3150703 (ADA)  Unit 2 – Analysis of Algorithm 83

Heap Sort – Algorithm
Algorithm: BUILD-MAX-HEAP(A) 4 1 3 2 9 7
heap-size[A] ← length[A] 1
4
for i ← ⌊length[A]/2⌋ downto 1
2 3
do MAX-HEAPIFY(A, i) 1 3
4 5 6
heap-size[A] = 6 2 9 7

4 1 7 2 9 3 4 9 7 2 1 3 9 4 7 2 1 3
i=3 1 i=2 1 i=1 1
4 4 49
2 3 2 3 2 3
1 37 91 7 94 7
4 5 6 4 5 6 4 5 6
2 9 73 2 91 3 2 1 3

Dr. Gopi Sanghani #3150703 (ADA)  Unit 2 – Analysis of Algorithm 84

Heap Sort – Algorithm
#  Input: Array A

# Output: Sorted array A 39 4 7 2 1 93

1
Algorithm: Heap_Sort(A[1,…,n]) 9
BUILD-MAX-HEAP(A) 2 3
for i length[A] downto 2 4 7
4 5 6
do exchange A[1] A[i]
heap-size[A] heap-size[A] – 1 2 1 3
MAX-HEAPIFY(A, 1, n)

Dr. Gopi Sanghani #3150703 (ADA)  Unit 2 – Analysis of Algorithm 85

Heap Sort – Algorithm
  
Algorithm: Max-heapify(A, i, n)
l LEFT(i)  l 2 1 3 4 7 2 1 9
r RIGHT(i)  r 3
1
if l ≤ n and A[l] > A[i] Yes
3
2 3
then largest l  largest 2 7
4
else largest i 4 5
if r ≤ n and A[r] > A[largest] Yes 2 1
then largest r  largest 3

if largest i Yes
then exchange A[i] A[largest]
MAX-HEAPIFY(A, largest, n)

Dr. Gopi Sanghani #3150703 (ADA)  Unit 2 – Analysis of Algorithm 86

Heap Sort – Algorithm
#  Input: Array A

# Output: Sorted array A 3 4 7 2 1 9

1
Algorithm: Heap_Sort(A[1,…,n]) 7
BUILD-MAX-HEAP(A) 2 3
for i length[A] downto 2 4 3
do exchange A[1] A[i] 4 5
heap-size[A] heap-size[A] – 1 2 1
MAX-HEAPIFY(A, 1, n)

Dr. Gopi Sanghani #3150703 (ADA)  Unit 2 – Analysis of Algorithm 87

Heap Sort Algorithm – Analysis
#  Input: Array A

# Output: Sorted array A
heap-size[A] ← length[A]
for i ← ⌊length[A]/2⌋ downto 1 𝒏/
  𝟐
Algorithm: Heap_Sort(A[1,…,n])
do MAX-HEAPIFY(A, i) 𝐎(𝒍𝒐𝒈
  ⁡𝒏)
BUILD-MAX-HEAP(A)  𝐎(𝒏 𝒍𝒐𝒈 ⁡𝒏)
for i length[A] downto 2
𝒏  −𝟏
do exchange A[1] A[i]
heap-size[A] heap-size[A] – 1
MAX-HEAPIFY(A, 1, n) 𝑶(𝒏
  −𝟏)(𝒍𝒐𝒈𝒏)

 Running time of heap sort algorithm is:

Dr. Gopi Sanghani #3150703 (ADA)  Unit 2 – Analysis of Algorithm 88

Sorting Algorithms
Shell Sort, Radix Sort, Bucket Sort, Counting Sort
Shell Sort - Example
 Sort the following elements in ascending order using shell sort.

80 93 60 12 42 30 68 85 10

Step 1: Divide input array into segments, where Initial Segmenting Gap = 4 (n÷2)

80 93 60 12 42 30 68 85 10
Each segment is
sorted within itself
using insertion sort
10 93 60 12 42 30 68 85 80

Dr. Gopi Sanghani #3150703 (ADA)  Unit 2 – Analysis of Algorithm 90

Shell Sort - Example
 Sort the following elements in ascending order using shell sort.

80 93 60 12 42 30 68 85 10

Step 1: Divide input array into segments, where Initial Segmenting Gap = 4 (n÷2)

10 30 60 12 42 93 68 85 80

Each segment is
10 30 60 12 42 93 68 85 80 sorted within itself
using insertion sort

10 30 60 12 42 93 68 85 80
Dr. Gopi Sanghani #3150703 (ADA)  Unit 2 – Analysis of Algorithm 91
Shell Sort - Example
 Sort the following elements in ascending order using shell sort.

80 93 60 12 42 30 68 85 10

Step 2: Now, the Segmenting Gap = 2 (4 ÷2)

10 30 60 12 42 93 68 85 80

Each segment is
10 30 42 12 60 93 68 85 80 sorted within itself
using insertion sort

10 12 42 30 60 85 68 93 80
Dr. Gopi Sanghani #3150703 (ADA)  Unit 2 – Analysis of Algorithm 92
Shell Sort - Example
 Sort the following elements in ascending order using shell sort.

80 93 60 12 42 30 68 85 10

Step 3: Now, the Segmenting Gap = 1 (2 ÷2)

10 12 42 30 60 85 68 93 80

Each segment is
sorted within itself
10 12 30 42 60 68 80 85 93 using insertion sort

The entire array is sorted now.

Dr. Gopi Sanghani #3150703 (ADA)  Unit 2 – Analysis of Algorithm 93
Shell Sort - Procedure
  Divide the array into several smaller non contiguous segments.
 The distance between successive elements in one segment is called a gap.
 Each segment is sorted within itself using insertion sort.
 Then re-segment into larger segments (smaller gap) and repeat sorting process.
 Continue until only one segment is left, i.e., gap = 1.
 Final sorting finishes the entire array sorting.
 Time complexity of shell sort is

Dr. Gopi Sanghani #3150703 (ADA)  Unit 2 – Analysis of Algorithm 94

Radix Sort
 Radix Sort puts the elements in order by comparing the digits of the numbers.
 Each element in the 𝒏-element array 𝑨 has 𝒅 digits, where digit 1 is the lowest-order digit and
digit 𝑑 is the highest order digit.

Algorithm: RADIX-SORT(A, d)
for i ← 1 to d
do use a stable sort to sort array A on digit i

 Sort following elements in Ascending order using radix sort.

363, 729, 329, 873, 691, 521, 435, 297

Dr. Gopi Sanghani #3150703 (ADA)  Unit 2 – Analysis of Algorithm 95

Radix Sort - Example

3 6 3 6 9 1 5 2 1
7 2 9 5 2 1 7 2 9
3 2 9 3 6 3 3 2 9
8 7 3 8 7 3 4 3 5
6 9 1 4 3 5 3 6 3
5 2 1 2 9 7 8 7 3
4 3 5 7 2 9 6 9 1
2 9 7 3 2 9 2 9 7

Sort on column 1 The2entireSort

Sort on column array
onis sorted3 now.
column

Dr. Gopi Sanghani #3150703 (ADA)  Unit 2 – Analysis of Algorithm 96

Bucket Sort – Introduction
  Sort the following elements in Ascending order using bucket sort.

45 96 29 30 27 12 39 61 91

1. Create empty buckets.

2. Add each input element to appropriate bucket as,
a. Bucket holds values in the half-open interval,

3. Sort each bucket queue with insertion sort.

4. Merge all bucket queues together in order.

 Expected running time is with = size of original sequence. If is then sorting algorithm in

Dr. Gopi Sanghani #3150703 (ADA)  Unit 2 – Analysis of Algorithm 97

Bucket Sort – Example

45 96 29 30 27 12 39 61 91

45 96 29 30 27 12 39 61 91

29
27 39 96
91

12 27
29 30 45 61 91
96
0 1 2 3 4 5 6 7 8 9

Sort each bucket queue with insertion sort

Merge all bucket queues together in order

12 27 29 30 39 45 61 91 96

Dr. Gopi Sanghani #3150703 (ADA)  Unit 2 – Analysis of Algorithm 98

Bucket Sort - Algorithm
# Input: Array A
# Output: Sorted array A
Algorithm: Bucket-Sort(A[1,…,n])
n ← length[A]
for i ← 1 to n do
insert A[i] into bucket B[⌊A[i] ÷ n ⌋]
for i ← 0 to n – 1 do
sort bucket B[i] with insertion sort
concatenate the buckets B[0], B[1], . . ., B[n - 1] together in
order.

Dr. Gopi Sanghani #3150703 (ADA)  Unit 2 – Analysis of Algorithm 99

Counting Sort – Example
 Sort the following elements in Ascending order using counting sort.

3 6 4 1 3 4 1 4 2

Step 1  Given elements are stored in an input array

Index
Elements 3 6 4 1 3 4 1 4 2

Step 2  Define a temporary array . The size of an arrayis equal to the

maximum element in array . Initialize .
Index
Elements 00 00 00 00 00 00

Dr. Gopi Sanghani #3150703 (ADA)  Unit 2 – Analysis of Algorithm 100

Counting Sort – Example
 Sort the following elements in Ascending order using counting sort.
  Input array 3 6 4 1 3 4 1 4 2

Step 3 Update an array C with the occurrences of each value of array 𝐴

Index
Elements 002 001 2 3 0 1

+ +
Step 4 In array 𝐶, from index 2 to 𝑛 add the value with previous element

Index
Elements 22 33 55 88 88 99

Dr. Gopi Sanghani #3150703 (ADA)  Unit 2 – Analysis of Algorithm 101

Counting Sort – Example
 Create an output array 𝐵[1…9]. Start positioning elements of Array 𝐴 𝑡𝑜 𝐵 as shown below.

  Input array 3 6 4 1 3 4 1 4 2

Temporary Array C 201 23 453 7856 8 89

Output Array B 1 1 2 3 3 4 4 4 6

Dr. Gopi Sanghani #3150703 (ADA)  Unit 2 – Analysis of Algorithm 102

Counting Sort - Procedure
 Counting sort assumes that each of the 𝑛 input elements is an integer in the range 0 to 𝑘, for
some integer 𝑘.
 When 𝒌=𝑶(𝒏), the counting sort runs in 𝜽(𝒏) time.
 The basic idea of counting sort is to determine, for each input element 𝑥, the number of
elements less than 𝑥.
 This information can be used to place element 𝑥 directly into its position in the output array.

Dr. Gopi Sanghani #3150703 (ADA)  Unit 2 – Analysis of Algorithm 103

Counting Sort - Algorithm
# Input: Array A
# Output: Sorted array A
Algorithm: Counting-Sort(A[1,…,n], B[1,…,n], k)
for i ← 1 to k do
    c[i] ← 0
for j ← 1 to n do
    c[A[j]] ← c[A[j]] + 1
for i ← 2 to k do
    c[i] ← c[i] + c[i-1]
for j ← n downto 1 do
    B[c[A[j]]] ← A[j]
    c[A[j]] ← c[A[j]] - 1

Dr. Gopi Sanghani #3150703 (ADA)  Unit 2 – Analysis of Algorithm 104

Amortized Analysis
Introduction
 Amortized analysis considers not just one operation, but a sequence of operations on a given
data structure or a database.
 Amortized Analysis is used for algorithms where an occasional operation is very slow, but most
of the other operations are faster.
 The time required to perform a sequence of data structure operations is averaged over all
operations performed.
 In Amortized Analysis, we analyze a sequence of operations and guarantee a worst case
average time which is lower than the worst case time of a particular expensive operation.
 So, Amortized analysis can be used to show that the average cost of an operation is small even
though a single operation might be expensive.

Dr. Gopi Sanghani #3150703 (ADA)  Unit 2 – Analysis of Algorithm 106

Amortized Analysis Techniques
 There are three most common techniques of amortized analysis,
1. The aggregate method
 A sequence of 𝑛 operation takes worst case time 𝑇(𝑛)
 Amortized cost per operation is 𝑇(𝑛)/𝑛
2. The accounting method
 Assign each type of operation an (different) amortized cost
 Overcharge some operations
 Store the overcharge as credit on specific objects
 Then use the credit for compensation for some later operations
3. The potential method
 Same as accounting method
 But store the credit as “potential energy” and as a whole.

Dr. Gopi Sanghani #3150703 (ADA)  Unit 2 – Analysis of Algorithm 107

  
Amortized Analysis - Example Incrementing a Binary Counter

Counter Increment Total  Implementing a -bit binary counter that counts

value [7] [6] [5] [4] [3] [2] [1] [0] cost cost upward from .
0  Use array of bits as the counter where,
0 0 0 0 0 0 0 0 0
1 0 0 0 0 0 0 0 1 1 1
+  is the least significant bit.
2 0 0 0 0 0 0 1 0 2 3
3 0 0 0 0 0 0 1 1 1 4  is the most significant bit.
4 0 0 0 0 0 1 0 0 3 7
5 0 0 0 0 0 1 0 1 1 8
6 0 0 0 0 0 1 1 0 2 10
7 0 0 0 0 0 1 1 1 1 11
8 0 0 0 0 1 0 0 0 4 15
9 0 0 0 0 1 0 0 1 1 16
10 0 0 0 0 1 0 1 0 2 18
11 0 0 0 0 1 0 1 1 1 19

Dr. Gopi Sanghani #3150703 (ADA)  Unit 2 – Analysis of Algorithm 108

  
Amortized Analysis - Example Aggregate Method

Counter Increment Total  The running time of an increment operation is

value [7] [6] [5] [4] [3] [2] [1] [0] cost cost proportional to the number of bits flipped.
0 0 0 0 0 0 0 0 0 0  However, all bits are not flipped at each
1 0 0 0 0 0 0 0 1 1 1 INCREMENT.
2 0 0 0 0 0 0 1 0 2 3  flips at each increment operation;
3 0 0 0 0 0 0 1 1 1 4  flips at alternate increment operations;
4 0 0 0 0 0 1 0 0 3 7  flips only once for 4 successive increment
5 0 0 0 0 0 1 0 1 1 8
6 0 0 0 0 0 1 1 0 2 10 operations;
7 0 0 0 0 0 1 1 1 1 11  In general, bit flips times in a sequence of
8 0 0 0 0 1 0 0 0 4 15 INCREMENTs.
9 0 0 0 0 1 0 0 1 1 16
10 0 0 0 0 1 0 1 0 2 18
11 0 0 0 0 1 0 1 1 1 19

Dr. Gopi Sanghani #3150703 (ADA)  Unit 2 – Analysis of Algorithm 109

Aggregate Method
  For (no. of bits) and (counter value) total number of flips of bit can be given as,

  8 8 8 8
𝐴= 0 + 1 + 2 + 3 Counter Number of bit
2 2 2 2 value flips
0 0000 0
𝐴=8+4 +2+1=𝟏𝟓
 
1 0001 1
2 0010 2
 s can be 3 0011 1
4 0100 3
  𝒌 −𝟏
𝒏 5 0101 1
𝑨=∑ 𝒊 6 0110 2
𝒊=𝟎 𝟐 7 0111 1
8 1000 4
Total Flips = 15

Dr. Gopi Sanghani #3150703 (ADA)  Unit 2 – Analysis of Algorithm 110

Aggregate Method
  Therefore, the total number of flips in the sequence is,

K-1

 Total time
 The amortized cost of each operation is

 
…

Dr. Gopi Sanghani #3150703 (ADA)  Unit 2 – Analysis of Algorithm 111

  
Amortized Analysis - Example Accounting Method
Counter Increment Total  If we charge an amortized cost of to set a bit to .
value [7] [6] [5] [4] [3] [2] [1] [0] cost cost
 When a bit is set we use to pay for the actual
0 0 0 0 0 0 0 0 0 0 setting of the bit and we place the other on the bit
1 0 0 0 0 0 0 0 1 1 1 as a credit.
2 0 0 0 0 0 0 1 0 2 3  At any time point, every in the counter has a of
3 0 0 0 0 0 0 1 1 1 4 credit on it.
4 0 0 0 0 0 1 0 0 3 7  So, no need to charge anything to reset a bit to
5 0 0 0 0 0 1 0 1 1 8  As we can pay for the reset with the on it.
6 0 0 0 0 0 1 1 0 2 10
 At most one bit is set to 1, in an increment
7 0 0 0 0 0 1 1 1 1 11
8 0 0 0 0 1 0 0 0 4 15 operation.
9 0 0 0 0 1 0 0 1 1 16  The amortized cost of an increment operation is at
10 0 0 0 0 1 0 1 0 2 18 most .
11 0 0 0 0 1 0 1 1 1 19  For increment operations, the total amortized
cost is
Running time of an increment operation is proportional to
the number of bits flipped.
Dr. Gopi Sanghani #3150703 (ADA)  Unit 2 – Analysis of Algorithm 113
Potential Method
  Potential method represents the prepaid work as potential energy.
 Potential energy can be released to pay for the future operations.
 Initial data structure
 operations, resulting in with costs .
 A potential function is
 is called the potential of
 Amortized cost of the operation is:
(actual cost + potential change)

 If is the total amortized cost of performing operations on data structure then,

Dr. Gopi Sanghani #3150703 (ADA)  Unit 2 – Analysis of Algorithm 114

Potential Method
  Total actual cost of operations can be bounded as,

 Total actual cost is bounded by total

  amortized
′ cost plus the net drop in potential over an entire
sequence of operations. 𝑻 =𝐓 +𝝋 − 𝝋 𝟎 𝒏

Dr. Gopi Sanghani #3150703 (ADA)  Unit 2 – Analysis of Algorithm 115

Thank You!