Why Study Algorithms

CS 854 Advance Algorithm Analysis

Anis ur Rahman
Department of Computing
NUST-SEECS
Islamabad
September 10, 2014
1/ 115
Course Information
1
Course Information
2
Algorithm analysis
3
Algorithm analysis
4
Big-O notation
2/ 115
Course Information Course Topics
This course is about
1 Vocabulary for design and analysis of algorithms
Big-Oh notation
high level reasoning about algorithms
general analysis methods (Master Method)
2 Divide and conquer algorithm design paradigm
applied to integer multiplication, sorting, matrix multiplication,
closest pair, etc.
3 Randomization in algorithm design
applied to QuickSort, primality testing, graph partitioning, hashing.
4 Primitives for reasoning about graphs
connectivity information, shortest paths, structure of information
and social networks.
5 Use and implementation of data structures
Heaps, balanced binary search trees, hashing, and many more
3/ 115
Course Information Course Topics
This course is about
1 Vocabulary for design and analysis of algorithms
Big-Oh notation
high level reasoning about algorithms
general analysis methods (Master Method)
2 Divide and conquer algorithm design paradigm
applied to integer multiplication, sorting, matrix multiplication,
closest pair, etc.
3 Randomization in algorithm design
applied to QuickSort, primality testing, graph partitioning, hashing.
4 Primitives for reasoning about graphs
connectivity information, shortest paths, structure of information
and social networks.
5 Use and implementation of data structures
Heaps, balanced binary search trees, hashing, and many more
4/ 115
Course Information Course Topics
This course is about
1 Vocabulary for design and analysis of algorithms
Big-Oh notation
high level reasoning about algorithms
general analysis methods (Master Method)
2 Divide and conquer algorithm design paradigm
applied to integer multiplication, sorting, matrix multiplication,
closest pair, etc.
3 Randomization in algorithm design
applied to QuickSort, primality testing, graph partitioning, hashing.
4 Primitives for reasoning about graphs
connectivity information, shortest paths, structure of information
and social networks.
5 Use and implementation of data structures
Heaps, balanced binary search trees, hashing, and many more
5/ 115
Course Information Course Topics
This course is about
1 Vocabulary for design and analysis of algorithms
Big-Oh notation
high level reasoning about algorithms
general analysis methods (Master Method)
2 Divide and conquer algorithm design paradigm
applied to integer multiplication, sorting, matrix multiplication,
closest pair, etc.
3 Randomization in algorithm design
applied to QuickSort, primality testing, graph partitioning, hashing.
4 Primitives for reasoning about graphs
connectivity information, shortest paths, structure of information
and social networks.
5 Use and implementation of data structures
Heaps, balanced binary search trees, hashing, and many more
6/ 115
Course Information Course Topics
This course is about
1 Vocabulary for design and analysis of algorithms
Big-Oh notation
high level reasoning about algorithms
general analysis methods (Master Method)
2 Divide and conquer algorithm design paradigm
applied to integer multiplication, sorting, matrix multiplication,
closest pair, etc.
3 Randomization in algorithm design
applied to QuickSort, primality testing, graph partitioning, hashing.
4 Primitives for reasoning about graphs
connectivity information, shortest paths, structure of information
and social networks.
5 Use and implementation of data structures
Heaps, balanced binary search trees, hashing, and many more
7/ 115
Course Information Course Topics
Skills Youll Learn
1 Become a better programmer
2 Sharpen your mathematical and analytical skills
3 Start thinking algorithmically
4 Literacy with computer sciences greatest hits
5 Ace your technical interviews
8/ 115
Course Information Course Topics
Who Are You?
1 Ideally, you know some programming.
2 Doesnt matter which language(s) you know.
But you should be capable of translating high-level algorithm
descriptions into working programs in some programming language.
3 Some mathematical experience.
basic discrete math, proofs by induction, etc.
Check out Mathematics for Computer Science, by Eric Lehman and
Tom Leighton.
9/ 115
Course Information Course Topics
Logistics
Class meets.
Monday 18:00 20:00
Thursdays 17:00 18:00
Web Site.
lms.nust.edu.pk
E-mail.
[email protected]
10/ 115
Course Information Course Topics
Source Material
No required textbook. A few of the many good ones:
1 Cormen/Leiserson/Rivest/Stein, Introduction to Algorithms, 2009
(3rd edition).
2 Kleinberg/Tardos, Algorithm Design, 2005.
3 Dasgupta/Papadimitriou/Vazirani, Algorithms, 2006.
4 Mehlhorn/Sanders, Data Structures and Algorithms: The Basic
Toolbox, 2008.
5 Web is the best and greenest textbook
No specic development environment required.
But you should be able to write and execute programs.
11/ 115
Course Information Course Topics
General information
Prerequisites.
CS110, CS250
Assessment.
5-6 Quizzes (15%).
2-3 Assignments (10%).
Term paper (5%).
OHTs (30%).
ESE (40%).
12/ 115
Course Information Course Topics
A barrage of show-of-hands questions
13/ 115
Course Information Course Topics
Tiny assignment for today
Q. A good example problem for showing algorithms with dierent
orders of magnitude is the classic anagram detection problem for
strings. One string is an anagram of another if the second is simply a
rearrangement of the rst. For example, heart and earth are
anagrams; python and typhon are anagrams as well. For the sake of
simplicity, assume that the two strings in question are of equal length
and are made up of the set of 26 lowercase alphabetic characters. Our
goal is to write a boolean function that will take two strings and return
whether they are anagrams.
14/ 115
Algorithm analysis
1
Course Information
2
Algorithm analysis
3
Algorithm analysis
4
Big-O notation
15/ 115
Algorithm analysis Algorithms
Algorithm: background
What is an Algorithm?
In mathematics, computer science, and related subjects, an algorithm
is an eective method for solving a problem expressed as a nite
sequence of instructions.
Abu Abdullah Muhammad bin Musa al-Khwarizmi
a Muslim mathematician.
well-known work: Al-Jabr.
invented algorithms to solve quadratic equations.
word algorithm derived by his Latin name Algorithmi.
16/ 115
Algorithm analysis Algorithms
Algorithm: background
What is an Algorithm?
In mathematics, computer science, and related subjects, an algorithm
is an eective method for solving a problem expressed as a nite
sequence of instructions.
Abu Abdullah Muhammad bin Musa al-Khwarizmi
a Muslim mathematician.
well-known work: Al-Jabr.
invented algorithms to solve quadratic equations.
word algorithm derived by his Latin name Algorithmi.
17/ 115
Algorithm analysis Algorithms
Algorithm: question
Which one is an algorithm?
A recipe for making tomato soup?
A procedure to sort 1000 numbers?
A procedure to recognize a particular face in a crowd?
A procedure to judge beauty?
18/ 115
Algorithm analysis Algorithms
Algorithm: designing algorithms
Four steps:
What is a Problem?
Correctness. is whether the proposed solution works.
Eciency. is the desired running time of the algorithm compared
to some benchmark.
How to express an Algorithm
19/ 115
Algorithm analysis Algorithms
Algorithm: solving a problem
Q. How to solve a new problem?
use basic building blocks
to achieve correctness then to worry about eciency
so simple only a genius could have thought of it" Albert Einstein
20/ 115
Algorithm analysis Algorithms
Algorithm: solving a problem
Q. How to solve a new problem?
use basic building blocks
to achieve correctness then to worry about eciency
so simple only a genius could have thought of it" Albert Einstein
21/ 115
Algorithm analysis Algorithms
Algorithm: what is good design?
A good design corresponds to:
selection of suitable building blocks.
use an appropriate building block.
nd interconnections between these building blocks.
22/ 115
Algorithm analysis Algorithms
Algorithm: designing algorithms
Start with simple building blocks
Use tested tools and techniques
Initial design should be as simple as possible
Worry about correctness
Then worry about eciency
23/ 115
Algorithm analysis Algorithms
Algorithm: designing algorithms
24/ 115
Algorithm analysis Algorithms
Algorithm analysis: socratic questioning
Socratic questioning is systematic, disciplined, and
deep, and usually focuses on fundamental concepts,
principles, theories, issues, or problems.
to pursue thought in many directions
to open up issues and problems
to uncover assumptions
to analyze concepts
to distinguish what we know from what we dont
know
to follow out logical implications of thought, or to
control the discussion
25/ 115
Algorithm analysis Algorithms
Algorithm analysis: problem
Find the eldest person six billion people.
a) We need to perform six billion operations?
b) We can do it by doing less than six billion operations by using
fast computers?
c) We can do it by doing about three billion operations by using
intelligent algorithms?
26/ 115
Algorithm analysis Algorithms
Algorithm: how to express one?
Input
Output
Basic idea in simple language
Complexity calculations
27/ 115
Algorithm analysis Algorithms
Algorithm: how to express one?
MATRIX-MULTIPLY(A,B)
if A.columns B.rows
error "incompatible dimensions"
else
let C be a new A.rows B.columns matrix
for i = 1 to A.rows
for j = 1 to B.columns
c
ij
= 0
for k = 1 to A.columns
c
ij
= c
ij
+ a
ik
b
kj
return C
28/ 115
Algorithm analysis Algorithms
Algorithm: how to express one?
Precondition: An image X of size W H
1 function StaticPathway(X)
2 X

RetinalFilter(X)
3 M
u,v
GaborFilterBank(X

)
4 M
u,v
Interactions(M
u,v
)
5 M
u,v
Normalizations(M
u,v
)
6 M
s
Fusion(M
u,v
)
7 return M
s
29/ 115
Algorithm analysis Algorithms
Algorithm: problem
Input: An Array A of k numbers. A[1...k]. The rst k 1 numbers are
sorted in ascending order, the last number is out of place.
1, 3, 7, 9, 13, 17, 23, 4
Output: An array A[1...k], which is sorted in ascending order.
1, 3, 4, 7, 9, 13, 17, 23
Q. Write the algorithm, verify its correctness.
30/ 115
Algorithm analysis Algorithms
Algorithm: moveMin
for i = k to 1
if A[i] < A[i-1]
SWAP(A[i],A[i-1])
Q. Will it swap till the rst element or stop?
For correctness, it is important to check the entire arrayto be sure.
Also, for simplicity of the algorithm.
Q. Is this enough for moveMin?
This is not a general algorithm to sort any unordered array. It is just a
building block for the sorting algorithm.
31/ 115
Algorithm analysis Algorithms
Algorithm: moveMin
for i = k to 1
if A[i] < A[i-1]
SWAP(A[i],A[i-1])
Q. Will it swap till the rst element or stop?
For correctness, it is important to check the entire arrayto be sure.
Also, for simplicity of the algorithm.
Q. Is this enough for moveMin?
This is not a general algorithm to sort any unordered array. It is just a
building block for the sorting algorithm.
32/ 115
Algorithm analysis Algorithms
Algorithm: moveMin
for i = k to 1
if A[i] < A[i-1]
SWAP(A[i],A[i-1])
Q. Will it swap till the rst element or stop?
For correctness, it is important to check the entire arrayto be sure.
Also, for simplicity of the algorithm.
Q. Is this enough for moveMin?
This is not a general algorithm to sort any unordered array. It is just a
building block for the sorting algorithm.
33/ 115
Algorithm analysis Algorithms
Summary
Algorithms are a nite set of operations to solve a problem.
Initial design of an algorithm should be:
Simple. using known concepts and principles.
Clear. described in plain English, or simple notation.
Correct. solve the problem.
34/ 115
Algorithm analysis
1
Course Information
2
Algorithm analysis
3
Algorithm analysis
4
Big-O notation
35/ 115
Algorithm analysis Overview
Algorithm analysis: background
Why is a algorithm analysis important?
Computer scientists learn by experience.
Learn techniques to solve for one problem, and use it for another.
Algorithm design helps to solve more challenging problems.
For example,
Consider two solutions for one problem.
Both give the correct result.
One might be resource ecient compared to the other.
In this lecture, we will learn analysis techniques to compare solely on
their characteristics,
not on the computer system used, or the implementation.
36/ 115
Algorithm analysis Overview
Algorithm analysis: background
Why is a algorithm analysis important?
Computer scientists learn by experience.
Learn techniques to solve for one problem, and use it for another.
Algorithm design helps to solve more challenging problems.
For example,
Consider two solutions for one problem.
Both give the correct result.
One might be resource ecient compared to the other.
In this lecture, we will learn analysis techniques to compare solely on
their characteristics,
not on the computer system used, or the implementation.
37/ 115
Algorithm analysis Overview
Algorithm analysis: background
Why is a algorithm analysis important?
Computer scientists learn by experience.
Learn techniques to solve for one problem, and use it for another.
Algorithm design helps to solve more challenging problems.
For example,
Consider two solutions for one problem.
Both give the correct result.
One might be resource ecient compared to the other.
In this lecture, we will learn analysis techniques to compare solely on
their characteristics,
not on the computer system used, or the implementation.
38/ 115
Algorithm analysis Overview
Algorithm analysis: background
Without algorithm analysis, there will always be some questions:
Is the implementation correct?
Are there any bugs?
How much faster?
39/ 115
Algorithm analysis Overview
Algorithm analysis: background
As soon as an Analytic Engine exists, it will necessarily guide the future
course of the science. Whenever any result is sought by its aid, the
question will ariseBy what course of calculation can these results be
arrived at by the machine in the shortest time?"
Charles Babbage (1864)
40/ 115
Algorithm analysis Overview
Algorithm analysis: background
It is convenient to have a measure of the amount of work involved in a
computing process, even though it may be a very crude one. We may
count up the number of times that various elementary operations are
applied in the whole process..."
Alan Turing (1947)
41/ 115
Algorithm analysis Overview
Algorithm analysis: the challenge
Q. Will the algorithm fare well for large input?
the program is very slow.
the program is out of memory.
42/ 115
Algorithm analysis Overview
Algorithm analysis: the outcomes
Algorithm analysis is used to:
Predict performance.
Compare algorithms.
Provide guarantees.
The main motivation is to avoid performance bugs.
43/ 115
Algorithm analysis Overview
Algorithm analysis: example
Discrete Fourier transform.
Break down waveform of N samples into periodic components.
Applications. spectral analysis, convolution, data compression, ...
Brute force method. N
2
steps.
FFT algorithm. NlogN steps, enables new technology.
44/ 115
Algorithm analysis Overview
Algorithm analysis: quantitative analysis
What to use quantitative, or qualitative analysis.
As an engineer, for the proposed algorithm:
one must determine the actual costs (memory, time, monetary).
comparison of qualitiesfaster, less memoryare unimportant.
45/ 115
Algorithm analysis Overview
Algorithm analysis: running time
Running time of the program is a function T(N),
represents the number of units of time taken by an algorithm on
any input of size N, or
the total number of statements executed by the program.
For example, a program may have a running time T(N) = cN, where c is
some constant.
the program is linearly proportional to the size of the input on
which it is run.
running time is linear time, or just linear.
46/ 115
Algorithm analysis Overview
Algorithm analysis: empirical analysis
Q. How to time a program?
manual. using a stopwatch
automatic. using some timer function
Run the program for various input sizes and measure running time.
data size time (ms)
250 5
500 8
1000 10
2000 15
4000 30
8000 50
16000 75
47/ 115
Algorithm analysis Overview
Algorithm analysis: data analysis
Plot running time T(N) vs. input size N.
48/ 115
Algorithm analysis Overview
Algorithm analysis: question
What are the dierent factor aecting the
running-time of an algorithm?
49/ 115
Algorithm analysis Overview
Algorithm analysis: running time
Dierent factors inuencing the running time of an algorithm are:
processing power of computer
algorithm used
size of the input
structure of the input
amount of available memory
implementation language
50/ 115
Algorithm analysis Overview
Algorithm analysis: running time
We consider three simplications to analyze algorithms:
without any implementation
=> RAM model
taking the worst possible inputs
=> worst-case running time
ignoring details
=> Big-O or Landau notation
51/ 115
Algorithm analysis Overview
Algorithm analysis: running time
We consider three simplications to analyze algorithms:
without any implementation => RAM model
taking the worst possible inputs => worst-case running time
ignoring details => Big-O or Landau notation
52/ 115
Algorithm analysis Overview
Algorithm analysis: running time
We consider three simplications to analyze algorithms:
without any implementation => RAM model
taking the worst possible inputs => worst-case running time
ignoring details => Big-O or Landau notation
53/ 115
Algorithm analysis RAM model
RAM model: overview
Random access machine.
Considers a very simple computer.
Motivation is to analyze an algorithm on a minimal computer
model.
Q. What do you consider to be a minimal computer model for analysis?
54/ 115
Algorithm analysis RAM model
RAM model: overview
A simple model to nd how an algorithm will perform on a real
computer.
Mouse
Keyboard
Display
CD-ROM
Graphics processor
Working memory
Processor
Operating system
File system
Input/Output
Programming capability
CD-ROM
55/ 115
Algorithm analysis RAM model
RAM model: overview
A simple model to nd how an algorithm will perform on a real
computer.
Mouse
Keyboard
Display
CD-ROM
Graphics processor
Working memory
Processor
Operating system
File system
Input/Output
Programming capability
CD-ROM
56/ 115
Algorithm analysis RAM model
RAM model: question
Simple operations (+,-,*,/) take 1 timestep
Loops comprise a condition plus multiple operations in each
iteration
Read/write to memory is free
57/ 115
Algorithm analysis RAM model
RAM model: question
1 a=1
2 b=2
*
a
# of timesteps = 1
1 s=0
2 while s<10:
3 s+=1
# of timesteps = 21
58/ 115
Algorithm analysis RAM model
RAM model: question
1 a=1
2 b=2
*
a
# of timesteps = 1
1 s=0
2 while s<10:
3 s+=1
# of timesteps = 21
59/ 115
Algorithm analysis RAM model
RAM model: question
1 a=1
2 b=2
*
a
# of timesteps = 1
1 s=0
2 while s<10:
3 s+=1
# of timesteps = 21
60/ 115
Algorithm analysis RAM model
RAM model: the problem
We observe some problem with RAM,
memory access times depends whether from cache or disk.
unlimited memory is unrealistic.
a +b a b.
However, its simplicity makes it an excellent tool to analyze algorithms.
61/ 115
Algorithm analysis Worst case running time
Algorithm analysis: running time
We consider three simplications to analyze algorithms:
without any implementation => RAM model
taking the worst possible inputs => worst-case running time
ignoring details => Big-O or Landau notation
62/ 115
Algorithm analysis Worst case running time
Worst case running time: overview
RAM simply counts operations for any input.
Worst case running time combines notion of running time with
structure of input.
determines good or bad algorithm based on all instances of inputs.
63/ 115
Algorithm analysis Worst case running time
Worst case running time: question
Consider an example,
1 count = 0
2 for char in str:
3 if char == 'a':
4 count += 1
In the example,
N: length of string
A: number of as in str
Running time =
2
N +
1
A +
0 or 1
64/ 115
Algorithm analysis Worst case running time
Worst case running time: question
Consider an example,
1 count = 0
2 for char in str:
3 if char == 'a':
4 count += 1
In the example,
N: length of string
A: number of as in str
Running time =2N +1A +0 or 1
65/ 115
Algorithm analysis Worst case running time
Worst case running time: the cases
Consider a sample problem, e.g. sorting (arranging items in order)
66/ 115
Algorithm analysis Worst case running time
Worst case running time: the cases
The plot illustrates three functions:
Worst-case complexity:
upper bound on cost.
determined by most dicult input.
provides a guarantee for all inputs.
Best-case complexity:
lower bound on cost.
determined by easiest input.
provides a goal for all inputs.
Average-case complexity:
expected cost for random input.
requires a model for random input.
provides a way to predict performance.
67/ 115
Algorithm analysis Worst case running time
Worst case running time: the challenge
Actual data might not match input running time model?
Need to understand input to eectively process it.
Two directions to follow:
Approach 1. design for the worst case.
Approach 2. randomize, depend on probabilistic guarantee.
68/ 115
Algorithm analysis Worst case running time
Worst case running time: the outcomes
Goals.
establish diculty of a problem.
develop optimal algorithms.
Approach.
suppress details in analysis; analyze to within a constant factor.
eliminate variability in input model by focusing on the worst case.
Optimal algorithm.
performance guarantee (to within a constant factor) for any input.
no other algorithm can provide a better performance guarantee.
69/ 115
Algorithm analysis Worst case running time
Worst case running time: question
Q. Which one do you consider is used? And why?
Best case is improbable.
Average case is dicult to determine.
Worst-case only guarantees running times.
70/ 115
Algorithm analysis Worst case running time
Worst case running time: question
Q. Which one do you consider is used? And why?
Best case is improbable.
Average case is dicult to determine.
Worst-case only guarantees running times.
71/ 115
Algorithm analysis Worst case running time
Worst case running time: question
Consider an example,
1 count = 0
2 for char in str:
3 if char == 'a':
4 count += 1
In the example consider,
N: length of string
A: number of as in str
Question?
Worst case running time =
3
N +
0
A +
1
Best case running time =
2
N +
0
A +
1
Average case running time =
?
72/ 115
Algorithm analysis Worst case running time
Worst case running time: question
Consider an example,
1 count = 0
2 for char in str:
3 if char == 'a':
4 count += 1
In the example consider,
N: length of string
A: number of as in str
Question?
Worst case running time =3N +0A +1
Best case running time =2N +0A +1
Average case running time = ?
73/ 115
Algorithm analysis Worst case running time
Worst case running time: the problem
Sample code counts ab substring in string str.
1 count = 0
2 for i in range(N):
3 if str[i] == 'a':
4 if str[i+1] == 'b':
5 count += 1
As a function grows larger,
counting exact running times becomes tedious.
determining complexities becomes dicult.
To further simplify we use Big-O notation.
74/ 115
Algorithm analysis Worst case running time
Algorithm analysis: running time (optional)
According to Donald E. Knuth in the book,The Art of Computer
Programming",
Total running time = sum of cost frequency for all operations.
The mathematical model for analysis is dened as,
analyze the program to determine set of operations.
determine cost and frequency,
frequency. depends on algorithm, input data.
cost. depends on machine, compiler.
75/ 115
Algorithm analysis Worst case running time
Algorithm analysis: the power law
The power law. a N
b
N is the size of the data.
b is the slope, determines the order of growth.
a is a constant.
76/ 115
Algorithm analysis Worst case running time
Algorithm analysis: the power law
System independent factors, or frequencies.
Algorithm.
Input data.

determines exponent b
in power law.
System dependent factors, or costs.
Hardware: CPU, memory, cache, ...
Software: compiler, interpreter, garbage collector, ...
System: operating system, network, other apps, ...
77/ 115
Algorithm analysis Worst case running time
Algorithm analysis: the power law
System independent factors, or frequencies.
Algorithm.
Input data.

determines exponent b
in power law.
System dependent factors, or costs.
Hardware: CPU, memory, cache, ...
Software: compiler, interpreter, garbage collector, ...
System: operating system, network, other apps, ...
78/ 115
Algorithm analysis Worst case running time
Algorithm analysis: the power law
System independent factors, or frequencies.
Algorithm.
Input data.
System dependent factors, or costs.
Hardware: CPU, memory, cache, ...
Software: compiler, interpreter,
garbage collector, ...
System: operating system,
network, other apps, ...

determines constant a
in power law.
79/ 115
Algorithm analysis Worst case running time
Algorithm analysis: mathematical models
In principle, there exist accurate mathematical models for algorithm
analysis.
In practice, such models,
use complicate formulas.
require advanced mathematics.
dened and used only for experts.
80/ 115
Algorithm analysis Worst case running time
Algorithm analysis: mathematical models
An example model could be,
T(N) = c
1
A +c
2
B +c
3
C +c
4
D +c
5
E

A =array access
B =integer add
C =integer compare
D =increment
E =variable assignment
where,
costs [c
1
, c
2
, c
3
, c
4
, c
5
] (depend on machine, compiler).
frequencies [A, B, C, D, E] (depend on algorithm, input).
Much easier and cheaper to determine approximate running times,
can run huge number of experiments
However, it is dicult to get precise measurements.
In this course, we will use approximate models, i.e. for matrix multiplication
T(N) cN
3
.
81/ 115
Algorithm analysis Worst case running time
Algorithm analysis: mathematical models
An example model could be,
T(N) = c
1
A +c
2
B +c
3
C +c
4
D +c
5
E

A =array access
B =integer add
C =integer compare
D =increment
E =variable assignment
where,
costs [c
1
, c
2
, c
3
, c
4
, c
5
] (depend on machine, compiler).
frequencies [A, B, C, D, E] (depend on algorithm, input).
Much easier and cheaper to determine approximate running times,
can run huge number of experiments
However, it is dicult to get precise measurements.
In this course, we will use approximate models, i.e. for matrix multiplication
T(N) cN
3
.
82/ 115
Big-O notation
1
Course Information
2
Algorithm analysis
3
Algorithm analysis
4
Big-O notation
83/ 115
Big-O notation Running time
Algorithm analysis: running time
We consider three simplications to analyze algorithms:
without any implementation => RAM model
taking the worst possible inputs => worst-case running time
ignoring details => Big-O or Landau notation
84/ 115
Big-O notation Running time
Algorithm analysis: quadratic growth
Consider the two functions,
f (n) = n
2
g(n) = n
2
3n +2
Around n = 3, they look very dierent.
85/ 115
Big-O notation Running time
Algorithm analysis: quadratic growth
Around n = 10, they look similar.
86/ 115
Big-O notation Running time
Algorithm analysis: quadratic growth
Around n = 100, they are almost same.
87/ 115
Big-O notation Running time
Algorithm analysis: quadratic growth
Around n = 1000, the dierence is indistinguishable.
88/ 115
Big-O notation Running time
Algorithm analysis: quadratic growth
The absolute dierence is large, such that,
f (1000) =1000000
g(1000) =997002
but the relative dierence is very small,

f (1000) g(1000)
f (1000)

=0.002998 < 0.3%

and this dierence goes to zero as n .
89/ 115
Big-O notation Running time
Algorithm analysis: polynomial growth
To demonstrate with another example,
f (n) = n
6
g(n) = n
6
76n
5
+343n
4
345n
3
+45n
2
344n
Around n = 10, they are very dierent
90/ 115
Big-O notation Running time
Algorithm analysis: polynomial growth
Around n = 100, they are not similar.
91/ 115
Big-O notation Running time
Algorithm analysis: polynomial growth
Around n = 1000, they seem similar.
92/ 115
Big-O notation Running time
Algorithm analysis: polynomial growth
Both function pairs of polynomials are similar, they each had the same
leading term:
n
2
in the rst case
n
6
in the second case
In each case, both functions exhibit the same rate of growth,
one is always proportionally larger than the other.
93/ 115
Big-O notation Asymptotic analysis
Big-O notation
Consider two algorithms to solve some problem,
Algorithm 1
3n
2
+2n +33
Algorithm 2
2
n
5n +5
Here, n is the problem size.
Question
Which one will you prefer?
When answering, you ignore all unnecessary details.
Only n
2
< 2
n
considered.
94/ 115
Big-O notation Asymptotic analysis
Big-O notation
Consider two algorithms to solve some problem,
Algorithm 1
3n
2
+2n +33
Algorithm 2
2
n
5n +5
Here, n is the problem size.
Question
Which one will you prefer?
When answering, you ignore all unnecessary details.
Only n
2
< 2
n
considered.
95/ 115
Big-O notation Asymptotic analysis
Big-O notation
Consider two algorithms to solve some problem,
Algorithm 1
3n
2
+2n +33
f(n)
Algorithm 2
2
n
5n +5
g(n)
Here, n is the problem size.
n
0
and c, where c g(n
0
) f (n
0
)
n

> n
0
, where c g(n) f (n)
f (n) O(g(n)), or f (n) = O(g(n))
It means the function f (n) has growth rate no greater than g(n).
96/ 115
Big-O notation Asymptotic analysis
Big-O notation
Consider two algorithms to solve some problem,
Algorithm 1
3n
2
+2n +33
f(n)
Algorithm 2
2
n
5n +5
g(n)
Here, n is the problem size.
n
0
and c, where c g(n
0
) f (n
0
)
n

> n
0
, where c g(n) f (n)
f (n) O(g(n)), or f (n) = O(g(n))
It means the function f (n) has growth rate no greater than g(n).
97/ 115
Big-O notation Asymptotic analysis
Big-O notation
Upper and lower bounds valid for n > n
0
smooth out the behavior of complex
functions.
98/ 115
Big-O notation Asymptotic analysis
Big-O notation
Formally,
f (n) = O(g(n)) means c g(n) is an upper bound on f (n). Thus,
there exists some constant c such that f (n) is always c g(n), for
large enough n (i.e. , n n
0
for some constant n
0
).
f (n) =(g(n)) means c g(n) is a lower bound on f (n). Thus
there exists some constant c such that f (n) is always c g(n), for
all n n
0
.
f (n) =(g(n)) means c
1
g(n) is an upper bound on f (n) and
c
2
g(n) is a lower bound on f (n), for all n n
0
. Thus there exist
constants c
1
and c
2
such that f (n) c
1
g(n) and f (n) c
2
g(n).
This means that g(n) provides a nice, tight bound on f (n).
99/ 115
Big-O notation Asymptotic analysis
Big-O notation
100/ 115
Big-O notation Asymptotic analysis
Big-O notation
f (n) g(n)
g(n) grows at least as fast as f (n)
5n +2 O(n) => 5n +2 O(n
2
)
12n
2
10 O(n
2
) => 12n
2
10 O(n
3
)
3
n
3n
8
+23 O(3
n
)
3
n
n
3
3n
8
+23 O(3
n
n
3
)
However, it is preferable to keep the bound as tight as possible.
101/ 115
Big-O notation Asymptotic analysis
Big-O notation
f (n) g(n)
g(n) grows at least as fast as f (n)
5n +2 O(n)
=> 5n +2 O(n
2
)
12n
2
10 O(n
2
) => 12n
2
10 O(n
3
)
3
n
3n
8
+23 O(3
n
)
3
n
n
3
3n
8
+23 O(3
n
n
3
)
However, it is preferable to keep the bound as tight as possible.
102/ 115
Big-O notation Asymptotic analysis
Big-O notation
f (n) g(n)
g(n) grows at least as fast as f (n)
5n +2 O(n)
=> 5n +2 O(n
2
)
12n
2
10 O(n
2
)
=> 12n
2
10 O(n
3
)
3
n
3n
8
+23 O(3
n
)
3
n
n
3
3n
8
+23 O(3
n
n
3
)
However, it is preferable to keep the bound as tight as possible.
103/ 115
Big-O notation Asymptotic analysis
Big-O notation
f (n) g(n)
g(n) grows at least as fast as f (n)
5n +2 O(n)
=> 5n +2 O(n
2
)
12n
2
10 O(n
2
)
=> 12n
2
10 O(n
3
)
3
n
3n
8
+23 O(3
n
)
3
n
n
3
3n
8
+23 O(3
n
n
3
)
However, it is preferable to keep the bound as tight as possible.
104/ 115
Big-O notation Asymptotic analysis
Big-O notation
f (n) g(n)
g(n) grows at least as fast as f (n)
5n +2 O(n)
=> 5n +2 O(n
2
)
12n
2
10 O(n
2
)
=> 12n
2
10 O(n
3
)
3
n
3n
8
+23 O(3
n
)
3
n
n
3
3n
8
+23 O(3
n
n
3
)
However, it is preferable to keep the bound as tight as possible.
105/ 115
Big-O notation Asymptotic analysis
Big-O notation
f (n) g(n)
g(n) grows at least as fast as f (n)
5n +2 O(n) => 5n +2 O(n
2
)
12n
2
10 O(n
2
) => 12n
2
10 O(n
3
)
3
n
3n
8
+23 O(3
n
)
3
n
n
3
3n
8
+23 O(3
n
n
3
)
However, it is preferable to keep the bound as tight as possible.
106/ 115
Big-O notation Asymptotic analysis
Big-O notation
The most common Big-O functions are given names:
O(1) => constant
O(ln(n)) => logarithmic
O(n) => linear
O(nln(n)) => log linear
O(n
2
) => quadratic
O(n
3
) => cubic
2
n
, e
n
, 4
n
=> exponential
107/ 115
Big-O notation Asymptotic analysis
Big-O notation
Plot for common Big-O functions:
108/ 115
Big-O notation Asymptotic analysis
Big-O notation
Sample code counts ab substring in string str.
1 count = 0
2 for char in str:
3 if char == 'a':
4 count += 1
Using Big-O simplication,
We require no counting of operations.
It can be simply written as, O(n), a linear algorithm.
109/ 115
Big-O notation Asymptotic analysis
Big-O notation: problem
Given the following code fragment, what is its Big-O running time?
1 test = 0
2 for i in range(n):
3 for j in range(n):
4 test = test + i
*
j
a) O(n)
b) O(n
2
)
c) O(logn)
d) O(n
3
)
110/ 115
Big-O notation Asymptotic analysis
Big-O notation: problem
Given the following code fragment what is its Big-O running time?
1 test = 0
2 for i in range(n):
3 test = test + 1
4
5 for j in range(n):
6 test = test - 1
a) O(n)
b) O(n
2
)
c) O(logn)
d) O(n
3
)
111/ 115
Big-O notation Asymptotic analysis
Big-O notation: problem
Given the following code fragment what is its Big-O running time?
1 i = n
2 while i > 0:
3 k = 2 + 2
4 i = i / 2
5 }
a) O(n)
b) O(n
2
)
c) O(logn)
d) O(n
3
)
112/ 115
Big-O notation Asymptotic analysis
Big-O notation: problem
Running time of a sorting algorithm.
Computer A:
Speed is one billion instructions per second.
Uses Insertion Sort and requires 2n
2
instructions to sort n numbers.
Computer B:
Speed is 10 million instructions per second.
Uses Merge Sort and requires 50nlgn instructions.
113/ 115
Big-O notation Asymptotic analysis
Big-O notation: problem
Computer A takes.
2 (10
7
)
2
instructions
10
10
instructions/second
=20, 000secs(> 5.5hrs)
Computer B takes.
50 (10
7
)lg10
7
instructions
10
7
instructions/second
=1, 163secs(< 20mins)
Computer B runs 17 times faster than computer A,
using an algorithm that grows slowly.
even if uses a slow processor.
114/ 115
Big-O notation Asymptotic analysis
Summary
We will use Landau symbols to describe the complexity of algorithms.
e.g., adding a list of n numbers in linear time (n) algorithm.
An algorithm is said to have polynomial time complexity if its runtime
may be described by O(n
d
) for some xed d 0.
We will consider such algorithms to be ecient.
Problems that have no known polynomial-time algorithms are said to
be intractable.
e.g. Traveling salesman problem. nd the shortest path that visits
n cities.
Best run time: O(n
2
2
n
)
In general, you dont want to implement exponential-time or
exponential-memory algorithms.
115/ 115