Introduction

Dr. K Shahu Chatrapati

 INTRODUCTION
• What is an Algorithm?
• Performance Analysis
 What is an Algorithm?
• The word algorithm
– Comes from the name of a Persian author, Abu Ja’far
Mohammed ibn Musa al Khowarizmi (825 A.D.)
• Definition: An algorithm is a finite set of instructions that, if
followed, accomplishes a particular task, satisfying the
following criteria:
– Input. Zero or more
– Output. At least one quantity is produced
– Definiteness. Each instruction is clear and unambiguous
– Finiteness. The algorithm terminates after a finite number of
steps.
– Effectiveness. Every instruction must be very basic, so that it
can be carried out, in principle, by a person using only pencil
and paper. In other words, it must be feasible.
 Definition of an Algorithm
• Criteria 1 and 2 require that an algorithm produce one or more outputs and have
zero or more inputs that are externally supplied.
• According to criterion 3, each operation must be definite, meaning that it must be
perfectly clear what should be done. Directions such as “add 6 or 7 to x” or
“compute 5/0” are not permitted because it is not clear which of the two
possibilities should be done or what the result is.
– To achieve the criterion of definiteness, algorithms are written in
programming language. Programming languages are designed so that each
legitimate sentence has a unique meaning. A program is the expression of an
algorithm in a programming language.
• According to fourth criterion, the time for termination should be reasonably
short.
• Criterion 5 requires that each operation be effective. Performing arithmetic on
integers is an example of an effective operation, but arithmetic with real numbers
is not since some values may be expressible only by infinitely long decimal
expansion. Adding two such numbers would violate the effectiveness property.
• Algorithms that are definite and effective are also called computational
procedures
 Distinct Areas in the Study of Algorithms
• How to devise algorithms.
– Creating an algorithm is an art which may never be fully automated.
– We will study various design techniques that are proven to be useful.
– Divide and conquer, dynamic programming, greedy method, backtracking, branch and bound
techniques, etc.,
• How to validate algorithms.
– Validation. Prove that the algorithm computes the correct answer for all possible legal inputs.
– A proof can be very much longer than the program.
• How to analyze algorithms.
– Determining how much computing time and storage an algorithm requires.
– Requires great mathematical skill.
– It allows you to make quantitative judgments about the value of one algorithm over another.
– It allows you to predict whether the software will meet an efficiency constraints that exist.
• How to test a program.
– Debugging. It is a process of executing programs on sample data sets to determine whether
faulty results occur and, if so, to correct them.
– Debugging can only point to the presence of errors, but not to their absence.
– Profiling or performance measurement. It is the process of executing a correct program on
data sets and measuring the time and space it takes to compute the results.
 Performance Evaluation
• Space Complexity: It is the amount of memory the algorithm
needs to run to completion.
• Time Complexity: It is the amount of computer time the
algorithm needs to run to completion.
• Two kinds of performance evaluation:
– Performance analysis: a priori estimate
– Performance measurement: a posteriori testing
 Space Complexity
• Space complexity must be taken seriously for
multi-user systems and in situations where
limited memory is available.
• An algorithm generally requires space for
following components:
– Instruction Space: It is the space required to store the
executable version of the program. This space is fixed.
– Data Space: It is the space required to store all the
constants and variables value.
– Environment Space: It is the space required to store
the environment information needed to resume the
suspended function.
 … Space Complexity
• The space needed by an algorithm is the sum of the
following components:
– A fixed part that is independent of the characteristics of the
inputs and outputs. This part typically includes the instruction
space, space for simple variables and fixed-size component
variables, space for constants, and so on.
– A variable part that consists of the space needed by component
variables whose size is dependent on the particular problem
instance being solved, the space needed by referenced variable,
and the recursion stack space.
• The space requirement S(P) of any algorithm P may
therefore be written as S(P)=c+SP (instance characteristics),
where c is a constant.
 Constant Space Complexity

• The problem instance for the above algorithm is characterized by

the specific values of a, b, and c. Making the assumption that one
word is adequate to store the values of each of a, b, c, and the
result, we see that the space needed by abc is independent of the
instance characteristics.
• Consequently, Sabc (instance characteristics)=0 and S(abc)=4.
 Space Complexity for Iterative Function for Sum

• The problem instances for the above algorithm are characterized

by n, the number of elements to be summed. The space needed
by n is one word. The space needed by a is n words. So we
obtain Ssum(a,n)≥(n+3) (n for a[], one each for n, i and s)
 Space Complexity of Recursive Function for Sum

• As in the case of Sum, the instances are characterized by n.

The recursion stack space includes space for formal
parameters, the local variables and the return address.
Each call to Rsum requires at least three words (including
space for the values of n, the return address, and a pointer
to a[]). Since the depth of recursion is n+1, the recursion
stack space needed is ≥3(n+1)
 Time Complexity
• The time T(P) taken by a program P is the sum of the compile
time and the run (or execution) time.
• The compile time does not depend on the instance
characteristics. Also, a compiled program will be run several
times without recompilation.
• Consequently, we concern ourselves with just the run time of
a program. This run time is denoted by tP (instance
characteristics)
• Many of the factors tP depends on are not known at the time
a program is conceived, therefore, we attempt only to
estimate tP.
 … Time Complexity
• If we knew the characteristics of the compiler to be used, we could determine the
number of additions, subtractions, multiplications, divisions, compares, loads,
stores and so on, and obtain an expression for tP(n) of the form
tP(n)=ca Add(n)+cs Sub(n)+cm Mul(n)+cd Div(n)+…
where n denotes the instance characteristics, and ca, cs, cm, cd, and so on, denote
the time needed for addition, subtraction, multiplication, division, and so on
respectively. Add, Sub, Mul, Div, and so on are functions whose values are the
numbers of additions, subtractions, multiplications, divisions, and so on.
• Obtaining such an exact formula is in itself an impossible task, since the time
needed for an addition, subtraction, multiplication, and so on, depends on the
numbers being added, subtracted, multiplied and so on.
• The value of tP(n) can be obtained experimentally. The program can be typed,
compiled and run on a particular machine and the execution time can be physically
clocked.
• Even with this approach, one could face difficulties. In a multiuser system, the
execution time depends on such factors as system load, the number of other
programs running on the computer at the time program P is run, the
characteristics of these other programs, and so on.
Program Step
Example

• When the statements to count are introduced into Algorithm 1.6, the
result is Algorithm 1.8. The change in the value of count by the time the
program terminates is the number of steps executed by Algorithm 1.6. If
count is zero to start with, then it will be 2n+3 on termination
 Summary of Time Complexity
• The time complexity of an algorithm is given by the number of
steps taken by the algorithm.
• The number of steps itself is the function of the instance
characteristics.
• Any specific instance may have several characteristics (e.g.,
number of inputs, number of outputs, etc.). Usually, the number
of steps is computed as a fraction of some subset of these.
• Once the relevant characteristics (n, m, p, q, r, …) are selected,
we can define what a step is. A step is any computation unit that
is independent of the characteristics (n, m, p, q, i,…).
• Thus, 10 additions can be one step; 100 multiplications can also
be one step; but n additions cannot; nor can m/2 additions, p+q
subtractions and so on.
 … Summary of Time Complexity
• For many algorithms, the time complexity is not dependent solely on the
number of inputs or outputs or some other easily specified characteristic.
• For example, the searching algorithm, may terminate in one step, or it
may take two steps, and so on.
• Consequently, when the chosen parameters are not adequate to
determine the step count uniquely, we can define three kinds of step
counts: best case, worst case, and average.
• Our motivation to determine step counts is to be able to compare the
time complexities of two algorithms that compute the same function and
also to predict the growth in run time as the instance characteristics
change.
• But determining the exact step count (best case, worst case and average)
of an algorithm can be difficult. Determining the step count exactly is not
a worthwhile endeavor, since the notion of step is itself inexact.
 … Summary of Time Complexity
• Because of inexactness of what a step stands for, the
exact step count is not useful for comparative
purposes.
• For most situations, it is adequate to be able to make a
statement like c1n2≤tp(n)≤c2n2 or tQ(n,m)=c1n+c2m,
where c1 and c2 are non-negative constants.
• If we have two algorithms with complexity c1n2+c2n
and c3n respectively, then we know that the one with
complexity c3n will be faster than the one with
complexity c1n2+c2n for sufficiently large values of n.
For small values of n, either algorithm could be faster
(depending on c1, c2, and c3).
 Order of Growth
• When the algorithms are compared with respect to their behavior
for large input size, the measure is known as Order of Growth.
• The order of growth can be estimated by taking into account the
dominant term of the running expression. The dominant term
overpasses the values of the other terms when the input size is
large.
For example:
• Let us consider T(n) = an + b where a>0, If the input size 'n' is
multiplied by 'k' then the dominant term in T(n) i.e. 'an' is also
multiplied by 'k'. This produces T(kn) = k(an)+b and we can say that
T(n) has a linear order of growth.
• Let us consider T(n) = an2+ bn + c where a>0, If the input size 'n' is
multiplied by 'k' ,this produces T(kn) = (k2a)n2+ (kb)n + c, thus the
dominant term is multiplied by k2 and we can say that T(n) has a
quadratic order of growth. and so on for cubic, logarithmic etc.
 Asymptotic Notation (O, Ω, θ)
• In Asymptotic analysis it is considered that an algorithm
'A1' is better than algorithm 'A2' if the order of growth of
the running time of the 'A1' is lower than that of 'A2'.
• Therefore asymptotic efficiency of algorithms are
concerned with how the running time of an algorithm
increases with the size of the input in the limit, as the size
of the input increases without bound.
• Asymptotic Notations: For every algorithm corresponding
to efficiency analysis, we have three basic cases :
– Best Case
– Average Case
– Worst Case
 Asymptotic Notation (O, Ω, θ)
• There are 5 asymptotic notations:
 Big O Notation
• Big O notation is used to describe an asymptotic
upper bound for the magnitude of a function in
terms of function.
• It is the formal method of expressing the upper
bound of an algorithm’s running time.
• It’s a measure of the longest amount of time it
could possibly take for the algorithm to complete.
• The function f(n) = O(g(n)) (read as “f of n is big
oh of g of n”) iff there exist positive constants c
and n0 such that f(n)<=c*g(n) for all n, n>=n0.
 Omega Notation
• Just as big O notation provides an asymptotic
upper bound on a function. Omega notation
provides an asymptotic lower bound.
• Since Omega notation describes a lower bound,
we use it to bind the best-case running time of an
algorithms.
• The function f(n) = Ω(g(n)) (read as “f of n is
omega of g of n”) iff there exist positive constants
c and n0 such that f(n)>=c*g(n) for all n, n>=n0.
 Theta Notation
• The theta notation is used when the function f can be bounded both from above
and below by the same function g.
• The function f(n) =Ѳ(g(n)) (read as “f of n is theta of g of n”) iff there exist
positive constants c1, c2 and n0 such that c1*g(n)<=f(n)<=c2*g(n) for all n, n>=n0.
• When we write f(n) = Ѳ(g(n)), we are saying that f lies between c1 times the
function g and c2 times the function g except possibly when n is smaller than n0.
• Here c1 and c2 are positive constants. Thus g is both a lower and upper bound on
the value of f for suitably large n.
• Another way to view the theta notation is that it says f(n) is both Ω(g(n)) and
O(g(n)).
 Little o Notation
• Little o notation is used to describe an upper bound that cannot be tight. In other
words, loose upper bound of f(n).
• Definition : Let f(n) and g(n) be functions that map positive integers to positive
real numbers. We say that f(n) is ο(g(n)) if for any real constant c > 0, there exists
an integer constant n0 ≥ 1 such that 0 ≤ f(n) < c*g(n).
 Little ω notation
• We use ω to denote a lower bound that is not asymptotically tight.
• Definition : Let f(n) and g(n) be functions that map positive integers to
positive real numbers. We say that f(n) is ω(g(n)) if for any real constant c >
0, there exists an integer constant n0 ≥ 1 such that f(n) > c * g(n) ≥ 0 for
every integer n ≥ n0.