Arya Institute Of Engineering and Technology,

Jaipur

Submitted by: Pratibha Sharma

Subject: Analysis of Algorithms
Dept.- Computer Science & Engineering
AIET, Jaipur

1
Algorithm and Complexity
 An algorithm is a specific procedure for solving a well-
defined computational problem.
 The development and analysis of algorithms is fundamental to all
aspects of computer science: artificial intelligence, databases,
graphics, networking, operating systems, security, and so on.
 The (computational) complexity of an algorithm is a measure of the
amount of computing resources (time and space) that a particular
algorithm consumes when it runs.
 Computer scientists use mathematical measures of complexity that
allow them to predict, before writing the code, how fast an
algorithm will run and how much memory it will require.
 Such predictions are important guides for
programmers implementing and selecting algorithms for real-world
applications.
The complexity of an algorithm describes the efficiency of the
algorithm in terms of the amount of the memory required to process
the data and the processing time.
 Complexity of an algorithm is analyzed in two perspectives:
 Time and Space.

 Time Complexity:
 It’s a function describing the amount of time required to run an
algorithm in terms of the size of the input.
 "Time" can mean the number of memory accesses performed, the
number of comparisons between integers, the number of times
some inner loop is executed, or some other natural unit related to
the amount of real time the algorithm will take.
 Space Complexity:
 It’s a function describing the amount of memory an algorithm takes
in terms of the size of input to the algorithm.
 We often speak of "extra" memory needed, not counting the
memory needed to store the input itself. Again, we use natural (but
fixed-length) units to measure this.
 Space complexity is sometimes ignored because the space used is
minimal and/or obvious, however sometimes it becomes as
important an issue as time.
Characteristics of Algorithm
 We can have three cases to analyze an algorithm:
1) Worst Case
2) Average Case
3) Best Case

 Worst Case Analysis (Usually Done)

In the worst case analysis, we calculate upper bound on
running time of an algorithm. We must know the case that
causes maximum number of operations to be executed.

 Average Case Analysis (Sometimes done)

In average case analysis, we take all possible inputs and
calculate computing time for all of the inputs. Sum all the
calculated values and divide the sum by total number of
inputs.We must know (or predict) distribution of cases.

 Best Case Analysis

In the best case analysis, we calculate lower bound on
running time of an algorithm. We must know the case that
causes minimum number of operations to be executed.
Asymptotic Notations
 In Asymptotic Analysis, we evaluate the performance of an
algorithm in terms of input size (we don’t measure the actual
running time). We calculate, how the time (or space) taken by an
algorithm increases with the input size.
 we estimate the efficiency of an algorithm asymptotically.
 Time function of an algorithm is represented by T(n), where n is
the input size.
 Different types of asymptotic notations are used to represent the
complexity of an algorithm. Following asymptotic notations are
used to calculate the running time complexity of an algorithm.
 O − Big Oh
 Ω − Big omega
 θ − Big theta
 o − Little Oh
 ω − Little omega
.
Analysis of Iterations
 1) O(1): Time complexity of a function (or set of statements) is
considered as O(1) if it doesn’t contain loop, recursion and call to
any other non-constant time function.
 // set of non-recursive and non-loop statements
 For example swap() function has O(1) time complexity.
A loop or recursion that runs a constant number of times is also
considered as O(1).
 // Here c is a constant
 for (int i = 1; i <= c; i++)
 {
 // some O(1) expressions
 }
 2) O(n): Time Complexity of a loop is considered as
O(n) if the loop variables is incremented / decremented
by a constant amount.
 For example following functions have O(n) time
complexity.
 // Here c is a positive integer constant
 for (int i = 1; i <= n; i += c)
 {
 // some O(1) expressions
 }
 for (int i = n; i > 0; i -= c)
 {
 // some O(1) expressions
 }
 O(nc): Time complexity of nested loops is equal
to the number of times the innermost statement
is executed.
 For example the following sample loops have
O(n2) time complexity

for (int i = 1; i <=n; i += c)

{
for (int j = 1; j <=n; j += c)
{
// some O(1) expressions
}
}
 O(Logn) :Time Complexity of a loop is
considered as O(Logn) if the loop variables
is divided / multiplied by a constant amount.
 for (int i = 1; i <=n; i *= c)
{
 // some O(1) expressions
}
 for (int i = n; i > 0; i /= c)
 {
 // some O(1) expressions
}
 For example Binary Search(refer iterative
implementation) has O(Logn) time
complexity.
 5) O(LogLogn) :Time Complexity of a loop is
considered as O(LogLogn) if the loop variables is
reduced / increased exponentially by a constant
amount.
 // Here c is a constant greater than 1
 for (int i = 2; i <=n; i = pow(i, c))
 {
 // some O(1) expressions
 }
 //Here fun is sqrt or cuberoot or any other
constant root
 for (int i = n; i > 1; i = fun(i))
 {
 // some O(1) expressions
 }
Recurrence Relation
 How to calculate time complexity of
recursive functions?

 Time complexity of a recursive function

can be written as a mathematical
recurrence relation.

 To calculate time complexity, we must

know how to solve recurrences.
Recurrence Relation
 If we get running time on an input of size
n as a function of n in the form of the
running time on inputs of smaller sizes
then equation representing such relation
is known as recurrence relation.
 T(n)=T(n-1)+c………Factorial
 T(n)=T(n/2)+c…….. Binary Search
 T(n)=2T(n/2)+n…….. Merge Sort
 Methods to solve Recurrence
Relation
1. Master Method
2. Substitution or Iteration Method
3. Recurrence Tree Method
Master Method
 The Master Method is used for solving
the following types of recurrence
 T (n) = a T(n/b)+ f (n)
 with a≥1 and b≥1 be constant & f(n) be a
function
 Let T (n) is defined on non-negative
integers by the recurrence.
 T (n) = a T(n/b)+ f (n)
 In the function to the analysis of a recursive
algorithm, the constants and function take on the
following significance:

 n is the size of the problem.

 a is the number of sub problems in the recursion.
 n/b is the size of each sub problem. (Here it is assumed that
all sub problems are essentially the same size.)
 f (n) is the sum of the work done outside the recursive calls,
which includes the sum of dividing the problem and the sum
of combining the solutions to the sub problems.
 It is not possible always bound the function according to the
requirement, so we make three cases which will tell us what
kind of bound we can apply on the function.
Master’s Theorem:
 Iteration Method
 It means to expand the recurrence and express it as a summation of terms
of n and initial condition.
 Example1: Consider the Recurrence
 T (n) = 1 if n=1 and T(n) = 2T (n-1) if n>1
Iteration Method
Recurrence Tree Method:
 In this method, we draw a recurrence tree and calculate
the time taken by every level of tree.

 Finally, we sum the work done at all levels.

 To draw the recurrence tree, we start from the given

recurrence and keep drawing till we find a pattern
among levels.

 The pattern is typically a arithmetic or geometric series.

Divide and Conquer
 Divide and Conquer is an algorithm design
paradigm that involves breaking up a larger problem
into non-overlapping sub-problems, solving each of
these sub-problems, and combining the results to
solve the original problems.
 A problem has non-overlapping sub-problems if you
can find its solution by solving each sub-problem
once.
 Divide: This involves dividing the problem into some
sub problem.
 Conquer: Sub problem by calling recursively until
sub problem solved.
 Combine: The Sub problem Solved so that we will
get find problem solution.
 The following are some standard algorithms that follows Divide and
Conquer algorithm.
 Binary Search is a searching algorithm. In each step, the algorithm
compares the input element x with the value of the middle element in
array. If the values match, return the index of the middle. Otherwise, if x is
less than the middle element, then the algorithm recurs for left side of
middle element, else recurs for the right side of the middle element.
 Quicksort is a sorting algorithm. The algorithm picks a pivot element,
rearranges the array elements in such a way that all elements smaller than
the picked pivot element move to left side of pivot, and all greater
elements move to right side. Finally, the algorithm recursively sorts the sub
arrays on left and right of pivot element.
 Merge Sort is also a sorting algorithm. The algorithm divides the array in
two halves, recursively sorts them and finally merges the two sorted
halves.
 Closest Pair of Points The problem is to find the closest pair of points
in a set of points in x-y plane. The problem can be solved in O(n^2) time
by calculating distances of every pair of points and comparing the distances
to find the minimum. The Divide and Conquer algorithm solves the
problem in O(nLogn) time.
 Strassen’s Algorithm is an efficient algorithm to multiply two matrices.
A simple method to multiply two matrices need 3 nested loops and is
O(n^3). Strassen’s algorithm multiplies two matrices in O(n^2.8974) time.
Binary Search
 Search a sorted array by repeatedly dividing the search
interval in half.

 Begin with an interval covering the whole array.

 If the value of the search key is less than the item in the
middle of the interval, narrow the interval to the lower
half. Otherwise narrow it to the upper half.

 Repeatedly check until the value is found or the interval

is empty.
 Procedure binary_search
 A ← sorted array
 n ← size of array
 x ← value to be searched
 Set lowerBound = 1
 Set upperBound = n
 while x not found
 if upperBound < lowerBound
 EXIT: x does not exists.
 set midPoint = lowerBound + ( upperBound - lowerBound ) / 2
 if A[midPoint] < x
 set lowerBound = midPoint + 1
 if A[midPoint] > x
 set upperBound = midPoint - 1
 if A[midPoint] = x
 EXIT: x found at location midPoint
 end while
 end procedure
Time Complexity of Binary Search
Algorithm
Merge Sort
 Like QuickSort, Merge Sort is a Divide and
Conquer algorithm. It divides input array in two halves, calls
itself for the two halves and then merges the two sorted
halves.

 Merge Sort follows the rule of Divide and Conquer to

sort a given set of numbers/elements, recursively, hence
consuming less time.

 In Merge Sort, the given unsorted array with n elements, is

divided into n sub arrays, each having one element, because a
single element is always sorted in itself. Then, it repeatedly
merges these sub arrays, to produce new sorted sub arrays,
and in the end, one complete sorted array is produced.

 In merge sort we follow the following steps:
 We take a variable p and store the starting index
of our array in this. And we take another
variable r and store the last index of array in it.

 Then we find the middle of the array using the

formula (p + r)/2 and mark the middle index as q,
and break the array into two subarrays,
from p to q and from q + 1 to r index.

 Then we divide these 2 subarrays again, just like

we divided our main array and this continues.

 Once we have divided the main array into

subarrays with single elements, then we start
merging the subarrays.
 procedure mergesort( var a as array )
 if ( n == 1 )
 return a
 var l1as array = a[0] ... a[n/2]
 var l2 as array = a[n/2+1] ... a[n]
 l1 = mergesort( l1 )
 l2 = mergesort( l2 )
 return merge( l1, l2 )
 end procedure
 procedure merge( var a as array, var b as array )
 var c as array
 while ( a and b have elements )
 if ( a[0] > b[0] ) add b[0] to the end of c
 remove b[0] from b
 else
 add a[0] to the end of c
 remove a[0] from a
 end if
 end while

 while ( a has elements )

 add a[0] to the end of c
 remove a[0] from a
 end while
 while ( b has elements )
 add b[0] to the end of c
 remove b[0] from b
 end while
 return c
 end procedure
Complexity Analysis
 We assume that we're sorting a total of n elements in the entire array.
 The divide step takes constant time, regardless of the sub array size.
After all, the divide step just computes the midpoint q of the
indices p and r. Recall that in big-Θ notation, we indicate constant time
by Θ(1).
 The conquer step, where we recursively sort two sub arrays of
approximately n/2 and n/2 elements each, takes some amount of time,
but we'll account for that time when we consider the sub problems.
 The combine step merges a total of n elements, taking Θ(n) time.

 We have already dealt with the same function in Recurrence,

Master's Theorem chapters and we know that it is going to
take Θ(nlgn) time.
Applications of Merge Sort

 Inversion Count Problem

 Used in External Sorting
Quick Sort
 The algorithm was developed by a British computer scientist Tony
Hoare in 1959.

 The name "Quick Sort" comes from the fact that, quick sort is
capable of sorting a list of data elements significantly faster (twice
or thrice faster) than any of the common sorting algorithms.

 It is one of the most efficient sorting algorithms and is based on

the splitting of an array (partition) into smaller ones and swapping
(exchange) based on the comparison with 'pivot' element selected.

 Due to this, quick sort is also called as "Partition Exchange" sort.

 Like Merge sort, Quick sort also falls into the category of divide
and conquer approach of problem-solving methodology.
Application

 Commercial computing: Used in various

government and private organizations for the
purpose of sorting various data like sorting of
accounts/profiles by name or any given ID, sorting
transactions by time or locations, sorting files by
name or date of creation etc.

 Numerical computations: Most of the efficiently

developed algorithms use priority queues and intern
sorting to achieve accuracy in all the calculations.

 Information search: Sorting algorithms aid in better

search of information and what faster way exists than
to achieve sorting using quick sort.
Explaination
 A quick sort first selects a value, which is called
the pivot value.

 Although there are many different ways to

choose the pivot value, we will simply use the first
item in the list.

 The role of the pivot value is to assist with

splitting the list.

 The actual position where the pivot value belongs

in the final sorted list, commonly called the split
point, will be used to divide the list for
subsequent calls to the quick sort.
Quick Sort Algorithm

 Using pivot algorithm recursively, we end up

with smaller possible partitions. Each
partition is then processed for quick sort.
We define recursive algorithm for quicksort
as follows −
 Step 1 − Make the left-most index value
pivot
 Step 2 − partition the array using pivot
value
 Step 3 − quicksort left partition recursively
 Step 4 − quicksort right partition
recursively
 here are many different versions of
quickSort that pick pivot in different
ways.
 Always pick first element as pivot.

 Always pick last element as pivot

(implemented below)

 Pick a random element as pivot.

 Pick median as pivot.

EXAMPLE
 def quickSort(alist):
 quickSortHelper(alist,0,len(alist)-1)

 def quickSortHelper(alist,first,last):
 if first<last:

 splitpoint = partition(alist,first,last)

 quickSortHelper(alist,first,splitpoint-1)
 quickSortHelper(alist,splitpoint+1,last)
 def partition(alist,first,last):
 pivotvalue = alist[first]

 leftmark = first+1
 rightmark = last

 done = False
 while not done:

 while leftmark <= rightmark and alist[leftmark] <=

pivotvalue:
 leftmark = leftmark + 1

 while alist[rightmark] >= pivotvalue and rightmark >=

leftmark:
 rightmark = rightmark -1
 if rightmark < leftmark:
 done = True
 else:
 temp = alist[leftmark]
 alist[leftmark] = alist[rightmark]
 alist[rightmark] = temp

 temp = alist[first]
 alist[first] = alist[rightmark]
 alist[rightmark] = temp

 return rightmark
Complexity Analysis
 Total running time of the QUICKSORT
function is going to be the summation of
the time taken by the PARTITION(A,
start, end) and two recursive calls to itself.
The comparison (if start < end) is going
to take a constant amount of time and
thus, we are ignoring it.
Complexity Analysis
 Worst Case Complexity [Big-O]: O(n2)
It occurs when the pivot element picked is either the greatest or the smallest
element.
This condition leads to the case in which the pivot element lies in an extreme
end of the sorted array. One sub-array is always empty and another sub-array
contains n - 1 elements. Thus, quicksort is called only on this sub-array.
However, the quick sort algorithm has better performance for scattered pivots.
 Best Case Complexity [Big-omega]: O(n*log
n)
It occurs when the pivot element is always the middle element or
near to the middle element.
 Let's take a case when the partition is always balanced i.e., every
time, each of the two halves separated by the pivot has no more
than n/2 elements. It means that one of the halves
has ⌊n/2⌋⌊n/2⌋ and another has ⌈n/2⌉−1and ⌈n/2⌉−1 elements.
 Average Case Complexity [Big-
theta]: O(n*log n)
 First, let's imagine that we don't always get evenly
balanced partitions, but that we always get at
worst a 3-to-1 split.
 That is, imagine that each time we partition, one
side gets 3n/43n/43, n, slash, 4 elements and the
other side gets n/4n/4n, slash, 4.
 (To keep the math clean, let's not worry about
the pivot.)
 Then the tree of subproblem sizes and
partitioning times would look like this:
 Advantages
 It is in-place since it uses only a small auxiliary stack.
 It requires only n (log n) time to sort n items.
 It has an extremely short inner loop.
 This algorithm has been subjected to a thorough
mathematical analysis, a very precise statement can be
made about performance issues.

 Disadvantages
 It is recursive. Especially, if recursion is not available,
the implementation is extremely complicated.
 It requires quadratic (i.e., n2) time in the worst-case.
 It is fragile, i.e. a simple mistake in the implementation
can go unnoticed and cause it to perform badly.
Strassen’s Matrix Multiplication
 Given two square matrices A and B of size n x n each, find their multiplication matrix.
 Naive Method
Following is a simple way to multiply two matrices.
 void multiply(int A[][N], int B[][N], int C[][N])
 {
 for (int i = 0; i < N; i++)
 {
 for (int j = 0; j < N; j++)
 {
 C[i][j] = 0;
 for (int k = 0; k < N; k++)
 {
 C[i][j] += A[i][k]*B[k][j];
 }
 }
 }
 }

 Time Complexity of above method is O(N3).

 Divide and Conquer
Following is simple Divide and Conquer method to
multiply two square matrices.

1) Divide matrices A and B in 4 sub-matrices of size N/2

x N/2 as shown in the below diagram.

2) Calculate following values recursively.

ae + bg,
af + bh,
ce + dg
and cf + dh.
 In the above method, we do 8
multiplications for matrices of size N/2 x
N/2 and 4 additions.
 Addition of two matrices takes O(N2)
time. So the time complexity can be
written as
 T(N) = 8T(N/2) + O(N2)
 From Master's Theorem, time complexity
of above method is O(N3)
 which is unfortunately same as the above
naive method.
 In the above divide and conquer method, the
main component for high time complexity is 8
recursive calls.

 The idea of Strassen’s method is to reduce the

number of recursive calls to 7.

 Strassen’s method is similar to above simple
divide and conquer method in the sense that this
method also divide matrices to sub-matrices of
size N/2 x N/2 as shown in the above diagram,
but in Strassen’s method, the four sub-matrices of
result are calculated using following formulae.

Time Complexity of Strassen’s
Method


 Addition and Subtraction of two matrices takes O(N2) time. So

time complexity can be written as

 T(N) = 7T(N/2) + O(N2)

 From Master's Theorem, time complexity of above method is
O(NLog7) which is approximately O(N2.8074)
 Generally Strassen’s Method is not preferred
for practical applications for following
reasons.
1) The constants used in Strassen’s method
are high and for a typical application Naive
method works better.
2) For Sparse matrices, there are better
methods especially designed for them.
3) The submatrices in recursion take extra
space.
4) Because of the limited precision of
computer arithmetic on noninteger values,
larger errors accumulate in Strassen’s
algorithm than in Naive Method