Origin of word: Algorithm

• The word comes from the name of the muslim

author Abu Ja’far Mohammad ibn Musa
al-Khowarizmi .
• He was born in the eighth century at
Khwarizm (Kheva), a town south of river Oxus
in present Uzbekistan.
• Uzbekistan, a Muslim country for over a
thousand years, was taken over by the
Russians in 1873.

Algorithms – p. 3
Origin of word: Algorithm

Algorithms – p. 4
Origin of word: Algorithm
• His year of birth is not known exactly.
• Al-Khwarizmi parents migrated to a place
south of Baghdad when he was a child.

Algorithms – p. 5
Origin of word: Algorithm
• It has been established from his contributions
that he flourished under Khalifah Al-Mamun at
Baghdad during 813 to 833 C.E.
• Al-Khwarizmi died around 840 C.E.

Algorithms – p. 6
Origin of word: Algorithm
• Much of al-Khwarizmi’s work was written in a
book titled
al Kitab al-mukhatasar fi hisab al-jabr
wa’l-muqabalah
(The Compendious Book on Calculation by
Completion and Balancing).

Algorithms – p. 7
Origin of word: Algorithm
• It is from the titles of these writings and his
name that the words algebra and algorithm
are derived.
• As a result of his work, al-Khwarizmi is
regarded as the most outstanding
mathematician of his time

Algorithms – p. 8
Algorithm: Informal Definition
• An algorithm is any well-defined
computational procedure that takes some
values, or set of values, as input and
produces some value, or set of values, as
output.
• An algorithm is thus a sequence of
computational steps that transform the input
into output.

Algorithms – p. 9
Algorithms, Programming
• A good understanding of algorithms is
essential for programming.
• However, unlike a program, an algorithm is a
mathematical entity

Algorithms – p. 10
Algorithms, Programming
• Algorithm is independent of a specific
programming language, machine, or compiler.
• Algorithm design is about mathematical
theory behind the design of good programs.

Algorithms – p. 11
Why Study Algorithms
• This course is all about how to design good
algorithms.
• CS301 dealt with how to design good data
structures.
• This is not really an independent issue
because the fastest algorithms are fast
because they use fast data structures, and
vice versa.

Algorithms – p. 12
Why Study Algorithms

• The fact is that many of the courses in

computer science deal with efficient
algorithms and data structures.

Algorithms – p. 13
Why Study Algorithms
They apply these to various applications:
• compilers,
• operating systems,
• databases,
• artificial intelligence,
• computer graphics and vision,
• networks, etc.

Algorithms – p. 14
Why Study Algorithms

• A good understanding of algorithm design is

a central element to a good understanding of
computer science and good programming.

Algorithms – p. 15
Course Outline

This course will consist of four major sections.

1. Mathematical tools necessary for the analysis
of algorithms. This will focus on asymptotics,
summations, recurrences.

Algorithms – p. 16
Course Outline

2. Sorting: this will focus on different strategies

for sorting, and use this problem as a
case-study in different techniques for
designing and analyzing algorithms.

Algorithms – p. 17
Course Outline

3. Collection of various algorithmic problems

and solution techniques: Dynamic
programming, greedy strategy, graphs.

Algorithms – p. 18
Course Outline

4. Introduction to the theory of

NP-completeness.
NP-complete problem are those for which no
efficient algorithms are known, but no one
knows for sure whether efficient solutions
might exist.

Algorithms – p. 19
Criterion for Analyzing Algorithms
• In order to design good algorithms, we must
first agree on the criterion for measuring
algorithms.
• The emphasis is on the design of efficient
algorithm

Algorithms – p. 20
Criterion for Analyzing Algorithms

We will measure algorithms in terms of the

amount of computational resources that the
algorithm requires:
• running time
• memory.

Algorithms – p. 21
Model of Computation
• We want the analysis to be as independent
as possible of the variations in machine,
operating system, compiler, or programming
language.
• Unlike programs, algorithms are to be
understood primarily by people and not
machines.
• Thus gives us quite a bit of flexibility in how
we present our algorithms, and many
low-level details may be omitted

Algorithms – p. 22
Model of Computation
• In order to say anything meaningful about our
algorithms, it will be important for us to settle
on a mathematical model of computation.
• Ideally this model should be a reasonable
abstraction of a standard generic
single-processor machine.
• We call this model a random access machine
or RAM.

Algorithms – p. 23
Random Access Machine
• A RAM is an idealized machine with an
infinitely large random-access memory.
• Instructions are executed one-by-one (there
is no parallelism).
• Each instruction involves performing some
basic operation on two values in the
machine’s memory

Algorithms – p. 24
Basic RAM Operations
Basic operations include
• assigning a value to a variable,
• computing any basic arithmetic operation (+, -
, × and division) on integer values of any size
• performing any comparison or boolean
operations
• accessing an element of an array (e.g. A[10])

Algorithms – p. 25
RAM Model
• We assume that each basic operation takes
the same constant time to execute.
• The RAM model does a good job of
describing the computational power of most
modern (nonparallel) machines.

Algorithms – p. 26
RAM Model

Basic operations in C++ that are assumed to

take up same amount of CPU time.
• x = y;
• z = a + b; z = a - b;
• z = a * b; z = a / b;
• z = w[10];
• (x <= y)

Algorithms – p. 27
Example: 2-dimension maxima
Let us do an example that illustrates how we
analyze algorithms.
• Suppose you want to buy a car.
• You want the pick the fastest car.
• But fast cars are expensive; you want the
cheapest.
• You cannot decide which is more important:
speed or price.

Algorithms – p. 28
Example: 2-dimension maxima
• Definitely do not want a car if there is another
that is both faster and cheaper.
• We say that the fast, cheap car dominates the
slow, expensive car relative to your selection
criteria.
• So, given a collection of cars, we want to list
those cars that are not dominated by any
other.

Algorithms – p. 29
Example: 2-dimension maxima

Here is how we might model this as a formal

problem.
• Let a point p in 2-dimensional space be given
by its integer coordinates, p = (p.x, p.y).
• A point p is said to be dominated by point q if
p.x ≤ q.x and p.y ≤ q.y.

Algorithms – p. 30
Example: 2-dimension maxima
• Given a set of n points, P = {p1, p2, . . . , pn} in
2-space a point is said to be maximal if it is
not dominated by any other point in P.

Algorithms – p. 31
Example: 2-dimension maxima
The car selection problem can be modelled this
way:
For each car we associate (x, y) pair where

• x is the speed of the car and

• y is the negation of the price.

Algorithms – p. 32
Example: 2-dimension maxima
• High y value means a cheap car and low y
means expensive car.
• Think of y as the money left in your pocket
after you have paid for the car.
• Maximal points correspond to the fastest and
cheapest cars.

Algorithms – p. 33
Example: 2-dimension maxima

2-dimensional Maxima:
• Given a set of points P = {p1, p2, . . . , pn} in
2-space,
• output the set of maximal points of P,
• i.e., those points pi such that pi is not
dominated by any other point of P.

Algorithms – p. 34
Example: 2-dimension maxima

14
(7,13)
12 (12,12)
(4,11) (9,10)
10 (14,10)

8 (7,7) (15,7)
6 (11,5)
(2,5)
4 (4,4) (13,3)
2
(5,1)

2 4 6 8 10 12 14 16 18
Algorithms – p. 35
Example: 2-dimension maxima
• Our description of the problem is at a fairly
mathematical level.
• We have intentionally not discussed how the
points are represented.
• We have not discussed any input or output
formats.
• We have avoided programming and other
software issues.

Algorithms – p. 36
Example: 2-dimension maxima
Brute-Force Algorithm:

• Let P = {p1 , p2, . . . , pn} be the initial set of

points.
• For each point pi , test it against all other
points pj .
• If pi is not dominated by any other point, then
output it.

Algorithms – p. 37
Example: 2-dimension maxima
M AX IMA (int
n, Point P[1 . . . n])
1 for i ← 1 to n
2 do maximal ← true
3 for j ← 1 to n
4 do
5 if (i
= j) and (P[i].x ≤ P[j].x) and (P[i].y ≤ P[j].y)
6 then maximal ← false; break
7 if (maximal = true)
8 then output P[i]

Algorithms – p. 38
Example: 2-dimension maxima

Points that dominate (4, 11)

14 (7,13)

12 (12,12)
(4,11)
10 (9,10) (14,10)

8 (7,7) (15,7)
6 (11,5)
(2,5)
4 (4,4) (13,3)
2
(5,1)

2 4 6 8 10 12 14 16 18
Algorithms – p. 39
Example: 2-dimension maxima

Points that dominate (9, 10)

14 (7,13)

12 (12,12)
(4,11)
10 (9,10) (14,10)

8 (7,7) (15,7)
6 (11,5)
(2,5)
4 (4,4) (13,3)
2
(5,1)

2 4 6 8 10 12 14 16 18
Algorithms – p. 40
Example: 2-dimension maxima

Points that dominate (7, 7)

14 (7,13)

12 (12,12)
(4,11)
10 (9,10) (14,10)

8 (7,7) (15,7)
6 (11,5)
(2,5)
4 (4,4) (13,3)
2
(5,1)

2 4 6 8 10 12 14 16 18
Algorithms – p. 41
Example: 2-dimension maxima
The maximal points

14
(7,13)
12 (12,12)
(4,11) (9,10)
10 (14,10)

8 (7,7) (15,7)
6 (11,5)
(2,5)
4 (4,4) (13,3)
2
(5,1)

2 4 6 8 10 12 14 16 18
Algorithms – p. 42
Running Time Analysis
Running Time Analysis

• The main purpose of our mathematical

analysis will be measuring the execution time.
• We will also be concerned about the space
(memory) required by the algorithm.

Algorithms – p. 43
Running Time Analysis
• The running time of an implementation of the
algorithm would depend upon
• the speed of the computer,
• programming language,
• optimization by the compiler etc.

• Although important, we will ignore these

technological issues in our analysis.

Algorithms – p. 44
Analysis of 2-d Maxima

To measure the running time of the brute-force

2-d maxima algorithm, we could

• count the number of steps of the pseudo code

that are executed
• or, count the number of times an element of P
is accessed
• or, the number of comparisons that are
performed.

Algorithms – p. 45
Running Time Analysis
• The running time depends upon the input
size, e.g. n
• Different inputs of the same size may result in
different running time.
• For example, breaking out of the inner loop in
the brute-force algorithm depends not only on
the input size of P but also the structure of the
input.

Algorithms – p. 46
Running Time Analysis: Criteria

Two criteria for measuring running time:

1. Worst-case time:

• Worst-case time is the maximum running time

over all (legal) inputs of size n.

Algorithms – p. 47
Running Time Analysis: Criteria

Worst-case time:

• Let I denote an input instance,

• let |I| denote its length, and
• let T (I) denote the running time of the
algorithm on input I.
• Then
Tworst (n) = max T (I)
|I|=n

Algorithms – p. 48
Running Time Analysis: Criteria

2. Average-case time

• Average-case time is the average running

time over all inputs of size n.

Algorithms – p. 49
Running Time Analysis: Criteria

2. Average-case time

• Let p(I) denote the probability of seeing this

input.
• The average-case time is the weighted sum
of running times with weights being the
probabilities:

Tavg(n) = p(I)T (I)
|I|=n

Algorithms – p. 50
Running Time Analysis
• We will almost always work with worst-case
time.
• Average-case time is more difficult to
compute; it is difficult to specify probability
distribution on inputs.
• Worst-case time will specify an upper limit on
the running time.

Algorithms – p. 51
Analysis of 2-dimension maxima

Analysis of the brute-force maxima algorithm.

• Input size is n
• We will count number of times that any
element of P is accessed.

Algorithms – p. 52
 Analysis of 2-dimension maxima
M AX IMA (int
n, Point P[1 . . . n])
1 for i ← 1 to n n times
2 do maximal ← true
3 for j ← 1 to n n times
4 do
5 if (i
= j)&(P[i].x ≤ P[j].x)&(P[i].y ≤ P[j].y) 4 accesses
6 then maximal ← false break
7 if maximal
8 then output P[i].x, P[i].y 2 accesses

Algorithms – p. 53
Analysis of 2-dimension maxima
• The outer loop runs n times
• For each i iteration, the inner loop runs n
time.
• P is accessed four times in the if statement.
• The output statement accesses P 2 times.
• In the worst case, every point is maximal so
every point is output.

Algorithms – p. 54
Analysis of 2-dimension maxima
Worst-case running time:
Pair of nested summations, one for i-loop and
the other for the j-loop
n
n
 
T (n) = 2+ 4
i=1 j=1

Algorithms – p. 55
Analysis of 2-dimension maxima
Worst-case running time:
n

 n
 
T (n) = 2+ 4
i=1 j=1
n
( j=1 4) = 4n, and so
n

T (n) = (4n + 2)
i=1
= (4n + 2)n = 4n2 + 2n
Algorithms – p. 56
Analysis of 2-dimension maxima
• For small values of n, any algorithm is fast
enough.
• What happens when n gets large?
• Running time does become an issue.
• When n is large, n2 term will be much larger
than the n term and will dominate the running
time.

Algorithms – p. 57
Analysis of 2-dimension maxima
• We will say that the worst-case running time
is Θ(n2).
• This is called the asymptotic growth rate of
the function.
• We will discuss this Θ-notation more formally
later.

Algorithms – p. 58
Summations
• The analysis involved computing a
summation.
• Summation should be familiar but let us
review a bit here.

Algorithms – p. 59
Summations
• Given a finite sequence of values
a1, a2, . . . , an ,
• their sum a1 + a2 + . . . + an is expressed in
summation notation as
n

ai
i=1

• If n = 0, then the sum is additive identity, 0.

Algorithms – p. 60
Summations

Some facts about summation:

• If c is a constant
n
 n

cai = c ai
i=1 i=1

and
n
 n
 n

(ai + bi) = ai + bi
i=1 i=1 i=1

Algorithms – p. 61
Summations

Some important summations that should be

committed to memory.
Arithmetic series:

n

i = 1 + 2 + ... + n
i=1
n(n + 1)
= = Θ(n2)
2

Algorithms – p. 62
Summations

Quadratic series:

n

2 2
i = 1 + 4 + 9 + ... + n
i=1
2n3 + 3n2 + n
= = Θ(n3)
6

Algorithms – p. 63
Summations

Geometric series:

n

i 2 n
x = 1 + x + x + ... + x
i=1
x(n+1) − 1
= = Θ(n2)
x−1
If 0 < x < 1 then this is Θ(1), and if x > 1, then
this is Θ(xn).

Algorithms – p. 64
Summations

Harmonic series: For n ≥ 0

n
 1
Hn =
i
i=1
1 1 1
= 1 + + + . . . + ≈ ln n
2 3 n
= Θ(ln n)

Algorithms – p. 65
A Harder Example

NESTED - LOOPS()
1 for i ← 1 to n
2 do
3 for j ← 1 to 2i
4 do k = j . . .
5 while (k ≥ 0)
6 do k = k − 1 ...

Algorithms – p. 66
A Harder Example

NESTED - LOOPS()
1 for i ← 1 to n
2 do
3 for j ← 1 to 2i
4 do k = j . . .
5 while (k ≥ 0)
6 do k = k − 1 ...

How do we analyze the running time of an

algorithm that has complex nested loop?

Algorithms – p. 67
A Harder Example

NESTED - LOOPS()
1 for i ← 1 to n
2 do
3 for j ← 1 to 2i
4 do k = j . . .
5 while (k ≥ 0)
6 do k = k − 1 ...

The answer is we write out the loops as

summations and then solve the summations.

Algorithms – p. 67
A Harder Example

NESTED - LOOPS()
1 for i ← 1 to n
2 do
3 for j ← 1 to 2i
4 do k = j . . .
5 while (k ≥ 0)
6 do k = k − 1 ...

To convert loops into summations, we work from

inside-out.

Algorithms – p. 67
A Harder Example

NES TED - LOOPS () • Consider the inner

1 for i← 1 to n most while loop.
2 do for j ← 1 to 2i • It is executed for k =
3 do k = j
j, j − 1, j − 2, . . . , 0.
4 while (k ≥ 0) J
5 do k = k − 1

Algorithms – p. 68
A Harder Example

NES TED - LOOPS () • Time spent inside the

1 for i← 1 to n while loop is constant.
2 do for j ← 1 to 2i • Let I() be the time
3 do k = j
spent in the while loop.
4 while (k ≥ 0) J
Thus
5 do k = k − 1
j

I(j) = 1=j+1
k= 0

Algorithms – p. 69
A Harder Example

NES TED - LOOPS () • Consider the middle for

1 for i← 1 to n loop.
2 do for j ← 1 to 2i J • It’s running time is de-
3 do k = j
termined by i.
4 while (k ≥ 0)
5 do k = k − 1

Algorithms – p. 70
A Harder Example

NES TED - LOOPS () • Let M() be the time

1 for i← 1 to n spent in the for loop:
2 do for j ← 1 to 2i J

2i
3 do k = j M(i) = I(j)
4 while (k ≥ 0) j= 1

5 do k = k − 1

Algorithms – p. 71
A Harder Example
2i
 2i

M(i) = I(j) = (j + 1)
j=1 j=1
2i
 2i

= j+ 1
j=1 j=1
2i(2i + 1)
= + 2i
2
= 2i2 + 3i

Algorithms – p. 72
A Harder Example

NES TED - LOOPS () • Finally the outer-most

1 for i← 1 to n J for loop.
2 do for j ← 1 to 2i • Let T () be running time
3 do k = j
of the entire algorithm:
4 while (k ≥ 0)
5 do k = k − 1

n
T (n) = M(i)
i=1

Algorithms – p. 73
A Harder Example
n
 n

T (n) = M(i) = (2i2 + 3i)
i=1 i=1
n n

= 2i2 + 3i
i=1 i=1
2n3 + 3n + n 2
n(n + 1)
=2 +3
6 2
4n3 + 15n2 + 11n
=
6
= Θ(n3)
Algorithms – p. 74
2-dimension Maxima Revisited
Recall the 2-d maxima problem:

• Let a point p in 2-dimensional space be given

by its integer coordinates, p = (p.x, p.y).
• A point p is said to dominated by point q if
p.x ≤ q.x and p.y ≤ q.y.

Algorithms – p. 75
2-dimension Maxima Revisited
• Given a set of n points, P = {p1, p2, . . . , pn} in
2-space a point is said to be maximal if it is
not dominated by any other point in P.
• The problem is to output all the maximal
points of P.

Algorithms – p. 76
2-dimension Maxima Revisited
• We introduced a brute-force algorithm that
ran in Θ(n2) time.
• It operated by comparing all pairs of points.
• Is there an approach that is significantly
better?

Algorithms – p. 77
2-dimension Maxima Revisited
• The problem with the brute-force algorithm is
that it uses no intelligence in pruning out
decisions.
• For example, once we know that a point pi is
dominated by another point pj , we do not
need to use pi for eliminating other points.

Algorithms – p. 78
2-dimension Maxima Revisited
• This follows from the fact that dominance
relation is transitive.
• If pj dominates pi and pi dominates ph then
pj also dominates ph ; pi is not needed.

Algorithms – p. 79
Plane-sweep Algorithm
Plane-sweep Algorithm

• We will sweep a vertical line across the 2-d

plane from left to right.
• As we sweep, we will build a structure holding
the maximal points lying to the left of the
sweep line.

Algorithms – p. 80
Plane-sweep Algorithm
Plane-sweep Algorithm

• When the sweep line reaches the rightmost

point of P, we will have the complete set of
maximal points.
• This approach of solving geometric problems
is called the plane sweep.

Algorithms – p. 81
Plane-sweep Algorithm
Plane-sweep Algorithm:

• Store the existing maximal points in a list

• The points that pi dominates will appear at
the end of the list because points are sorted
by x-coordinate.
• Scan the list left to right.
• Every maximal point with y-coordinate less
than pi will be eliminated from computation.

Algorithms – p. 82
Plane-sweep Algorithm
Plane-sweep Algorithm:

• We add maximal points onto the end of a list

and delete from the end of the list.
• We can thus use a stack to store the maximal
points.
• The point at the top of the stack will have the
highest x-coordinate.

Algorithms – p. 83
Plane-sweep Algorithm
sweep line

14
12 (3,13) (12,12)

10 (4,11) (14,10)
(9,10)
8 (15,7)
(7,7)
6 (2,5)
4 (10,5)
(13,3)
2
(5,1) (2,5)

2 4 6 8 10 12 14 16 18 stack

Algorithms – p. 84
Plane-sweep Algorithm
sweep line

14
12 (3,13) (12,12)

10 (4,11) (14,10)
(9,10)
8 (15,7)
(7,7)
6 (2,5)
4 (10,5)
(13,3)
2
(5,1) (3,13)

2 4 6 8 10 12 14 16 18 stack

Algorithms – p. 85
Plane-sweep Algorithm
sweep line

14
12 (3,13) (12,12)

10 (4,11) (14,10)
(9,10)
8 (15,7)
(7,7)
6 (2,5)
4 (10,5)
(13,3) (4,11)
2
(5,1) (3,13)

2 4 6 8 10 12 14 16 18 stack

Algorithms – p. 86
Plane-sweep Algorithm
sweep line

14
12 (3,13) (12,12)

10 (4,11) (14,10)
(9,10)
8 (15,7)
(7,7)
6 (2,5)
(5,1)
4 (10,5)
(13,3) (4,11)
2
(5,1) (3,13)

2 4 6 8 10 12 14 16 18 stack

Algorithms – p. 87
Plane-sweep Algorithm
sweep line

14
12 (3,13) (12,12)

10 (4,11) (14,10)
(9,10)
8 (15,7)
6 (2,5) (7,7)
(7,7)
4 (10,5)
(13,3) (4,11)
2
(5,1) (3,13)

2 4 6 8 10 12 14 16 18 stack

Algorithms – p. 88
Plane-sweep Algorithm
sweep line

14
12 (3,13) (12,12)

10 (4,11) (14,10)
(9,10)
8 (15,7)
6 (2,5) (7,7)
(9,10)
4 (10,5)
(13,3) (4,11)
2
(5,1) (3,13)

2 4 6 8 10 12 14 16 18 stack

Algorithms – p. 89
Plane-sweep Algorithm
sweep line

14
12 (3,13) (12,12)

10 (4,11) (14,10)
(9,10)
8 (15,7) (10,5)
6 (2,5) (7,7)
(9,10)
4 (10,5)
(13,3) (4,11)
2
(5,1) (3,13)

2 4 6 8 10 12 14 16 18 stack

Algorithms – p. 90
Plane-sweep Algorithm
sweep line

14
12 (3,13) (12,12)

10 (4,11) (14,10)
(9,10)
8 (15,7)
6 (2,5) (7,7)

4 (10,5)
(13,3) (12,12)
2
(5,1) (3,13)

2 4 6 8 10 12 14 16 18 stack

Algorithms – p. 91
Plane-sweep Algorithm

14
12 (3,13) (12,12)

10 (4,11) (14,10)
(9,10)
8
(15,7) (15,7)
6 (2,5) (7,7)
(14,10)
4 (10,5)
(13,3) (12,12)
2
(5,1) (3,13)

2 4 6 8 10 12 14 16 18 stack

Algorithms – p. 92
Plane-sweep Algorithm

PLANE - SWEEP - MAXIMA(n, P[1..n])

1 sort P in increasing order by x;
2 stack s;
3 for i ← 1 to n
4 do
5 while (s.notEmpty() & s.top().y ≤ P[i].y)
6 do s.pop();
7 s.push(P[i]);
8 output the contents of stack s;

Algorithms – p. 93
Analysis of Plane-sweep Algorithm
• Sorting takes Θ(n log n); we will show this
later when we discuss sorting.
• The for loop executes n times.
• The inner loop (seemingly) could be iterated
(n − 1) times.
• It seems we still have an n(n − 1) or Θ(n2 )
algorithm.
• Got fooled by simple minded loop-counting.

Algorithms – p. 94
Analysis of Plane-sweep Algorithm
• The while loop will not execute more n times
over the entire course of the algorithm.
• Why is this?
• Observe that the total number of elements
that can be pushed on the stack is n since we
execute exactly one push each time during
the outer for-loop.

Algorithms – p. 95
Analysis of Plane-sweep Algorithm
• We pop an element off the stack each time
we go through the inner while-loop.
• It is impossible to pop more elements than
are ever pushed on the stack.
• Therefore, the inner while-loop cannot
execute more than n times over the entire
course of the algorithm. (Make sure that you
understand this).

Algorithms – p. 96
Analysis of Plane-sweep Algorithm
• The for-loop iterates n times and the inner
while-loop also iterates n time for a total of
Θ(n).
• Combined with the sorting, the runtime of
entire plane-sweep algorithm is Θ(n log n).

Algorithms – p. 97
Comparison
• How much of an improvement is plane-sweep
over brute-force?
• Ratio of running times:
n2 n
=
nlogn log n

Algorithms – p. 98
Comparison
n
n logn logn
100 7 15
1000 10 100
10000 13 752
100000 17 6021
1000000 20 50171
For n = 1, 000, 000, if plane-sweep takes 1
second, the brute-force will take about 14 hours!.

Algorithms – p. 99
Asymptotic Notation
• You may be asking that we continue to use
the notation Θ() but have never defined it.
• Let’s remedy this now.
• Given any function g(n), we define Θ(g(n)) to
be a set of functions that asymptotically
equivalent to g(n).

Algorithms – p. 100
Asymptotic Notation
Formally
Θ(g(n)) = {f(n) | there exist positive
constants c1, c2 and n0 such that
0 ≤ c1g(n) ≤ f(n) ≤ c2g(n)
for all n ≥ n0}

Algorithms – p. 101
Asymptotic Notation
Formally
Θ(g(n)) = {f(n) | there exist positive
constants c1, c2 and n0 such that
0 ≤ c1g(n) ≤ f(n) ≤ c2g(n)
for all n ≥ n0}
Your response at this point might be, “I am sorry that I
asked!”

Algorithms – p. 101
Asymptotic Notation
• This is written as “f(n) ∈ Θ(g(n))”
• That is, f(n) and g(n) are asymptotically
equivalent.
• This means that they have essentially the
same growth rates for large n.

Algorithms – p. 102
Asymptotic Notation

For example, functions like

• 4n2 ,
• (8n2 + 2n − 3),
2
√
• (n /5 + n − 10 log n)
• n(n − 3)

are all asymptotically equivalent. As n becomes

large, the dominant (fastest growing) term is
some constant times n2 .
Algorithms – p. 103
Asymptotic Notation - Example
• Consider the function
f(n) = 8n2 + 2n − 3
• Our informal rule of keeping the largest term
and ignoring the constant suggests that
f(n) ∈ Θ(n2).
• Let’s see why this bears out formally.

Algorithms – p. 104
Asymptotic Notation - Example

We need to show two things for

f(n) = 8n2 + 2n − 3
• Lower bound: f(n) = 8n2 + 2n − 3 grows
asymptotically at least as fast as n2 ,
• Upper bound: f(n) grows no faster
asymptotically than n2 ,

Algorithms – p. 105
Asymptotic Notation - Example

Lower bound: f(n) grows asymptotically at least

as fast as n2 .
• For this, need to show that there exist positive
constants c1 and n0 , such that f(n) ≥ c1n2 for
all n ≥ n0 .
• Consider the reasoning

f(n) = 8n2 +2n−3 ≥ 8n2 −3 = 7n2 +(n2−3) ≥ 7n2

• Thus c1 = 7.

Algorithms – p. 106
Asymptotic Notation - Example
• We implicitly assumed that 2n ≥ 0 and
n2 − 3 ≥ 0
√
• These are not true for all n but if n ≥ 3, then
both are true.
√
• So select n0 ≥ 3.
• We then have f(n) ≥ c1n2 for all n ≥ n0 .

Algorithms – p. 107
Asymptotic Notation - Example

Upper bound: f(n) grows asymptotically no

faster than n2 .
• For this, we need to show that there exist
positive constants c2 and n0 , such that
f(n) ≤ c2 n2 for all n ≥ n0 .
• Consider the reasoning

f(n) = 8n2 +2n−3 ≤ 8n2 +2n ≤ 8n2+2n2 = 10n2

• Thus c2 = 10.

Algorithms – p. 108
Asymptotic Notation - Example
• We implicitly made the assumption that
2n ≤ 2n2 .
• This is not true for all n but it is true for all
n≥1
• So select n0 ≥ 1.
• We thus have f(n) ≤ c2n2 for all n ≥ n0 .

Algorithms – p. 109
Asymptotic Notation - Example
√
• From lower bound we have n0 ≥ 3
• From upper bound we have n0 ≥ 1.
• Combining the√two, we let n0 be the larger of
the two: n0 ≥ 3.

Algorithms – p. 110
Asymptotic Notation - Example
• In conclusion,
√ if we let c1 = 7, c2 = 10 and
n0 ≥ 3, we have
√
7n2 ≤ 8n2 + 2n − 3 ≤ 10n2 for all n ≥ 3
• We have thus established
0 ≤ c1g(n) ≤ f(n) ≤ c2g(n) for all n ≥ n0

Algorithms – p. 111
Asymptotic Notation - Example

Asymptotic Notation
1e+11
8n^2+2n-3
7n^2
10n^2
8e+10

6e+10
f(n)

4e+10

2e+10

0
0 20000 40000 60000 80000 100000
n

Algorithms – p. 112
Asymptotic Notation - Example
• We have established that f(n) ∈ n2 .
• Let’s show why f(n) is not in some other
asymptotic class.
• First, let’s show that f(n) ∈ Θ(n).

Algorithms – p. 113
Asymptotic Notation - Example

Show that f(n) ∈ Θ(n).

• If this were true, we would have had to satisfy
both the upper and lower bounds.
• The lower bound is satisfied because
f(n) = 8n2 + 2n − 3 does grow at least as fast
asymptotically as n.
• But the upper bound is false. Upper bounds
requires that there exist positive constants c2
and n0 such that f(n) ≤ c2n for all n ≥ n0 .

Algorithms – p. 114
Asymptotic Notation - Example
• Informally we know that f(n) = 8n2 + 2n − 3
will eventually exceed c2n no matter how
large we make c2 .
• To see this, suppose we assume that
constants c2 and n0 did exist such that
8n2 + 2n − 3 ≤ c2n for all n ≥ n0
• Since this is true for all sufficiently large n
then it must be true in the limit as n tends to
infinity.

Algorithms – p. 115
Asymptotic Notation - Example
• If we divide both sides by n, we have


3
lim 8n + 2 − ≤ c2.
n→∞ n
• It is easy to see that in the limit, the left side
tends to ∞.
• So, no matter how large c2 is, the statement is
violated. Thus f(n) ∈ Θ(n).

Algorithms – p. 116
Asymptotic Notation - Example

Let’s show that f(n) ∈ Θ(n3 ).

• The idea would be to show that the lower
bound f(n) ≥ c1n3 for all n ≥ n0 is violated.
• (c1 and n0 are positive constants).
• Informally we know this to be true because
any cubic function will overtake a quadratic.

Algorithms – p. 117
Asymptotic Notation - Example
• If we divide both sides by n3 :


8 2 3
lim + 2 − 3 ≥ c1
n→∞ n n n
• The left side tends to 0. The only way to
satisfy this is to set c1 = 0.
• But by hypothesis, c1 is positive.
• This means that f(n) ∈ Θ(n3).

Algorithms – p. 118
O-notation and Ω-notation
• The definition of Θ-notation relies on proving
both a lower and upper asymptotic bound.
• Sometimes we only interested in proving one
bound or the other.
• The O-notation is used to state only the
asymptotic upper bounds.
• The Ω-notation allows us to state only the
asymptotic lower bounds.

Algorithms – p. 119
O-notation (Big-O)

O(g(n)) = {f(n) | there exist positive

constants c and n0 such that
0 ≤ f(n) ≤ cg(n)
for all n ≥ n0}

Algorithms – p. 120
Ω-notation

Ω(g(n)) = {f(n) | there exist positive

constants c and n0 such that
0 ≤ cg(n) ≤ f(n)
for all n ≥ n0}

Algorithms – p. 121
Θ, O, Ω-notation
The three notations:

Θ(g(n)) : 0 ≤ c1g(n) ≤ f(n) ≤ c2g(n)

O(g(n)) : 0 ≤ f(n) ≤ cg(n)
Ω(g(n)) : 0 ≤ cg(n) ≤ f(n)
for all n ≥ n0

Algorithms – p. 122
Limit Rule

These definitions suggest an alternative way of

showing the asymptotic behavior.

Limit rule for Θ-notation:

f(n)
lim = c,
n→∞ g(n)

for some constant c > 0 (strictly positive but not

infinity) then f(n) ∈ Θ(g(n)).

Algorithms – p. 123
Limit Rule

Limit rule for O-notation:

f(n)
lim = c,
n→∞ g(n)

for some constant c ≥ 0 (nonnegative but not

infinite) then f(n) ∈ O(g(n)).

Algorithms – p. 124
Limit Rule

Limit rule for Ω-notation:

f(n)
lim = 0,
n→∞ g(n)

(either a strictly positive constant or infinity) then

f(n) ∈ Ω(g(n)).

Algorithms – p. 125
Asymptotic Intuition

Here is a list of common asymptotic running

times:
• Θ(1): Constant time; can’t beat it!
• Θ(log n): Inserting into a balanced binary
tree; time to find an item in a sorted array of
length n using binary search.
• Θ(n): About the fastest that an algorithm can
run.
• Θ(n log n): Best sorting algorithms.

Algorithms – p. 126
Asymptotic Intuition
• Θ(n2), Θ(n3): Polynomial time. These
running times are acceptable when the
exponent of n is small or n is not to large,
e.g., n ≤ 1000.
• Θ(2n), Θ(3n): Exponential time. Acceptable
only if n is small, e.g., n ≤ 50.
• Θ(n!), Θ(nn): Acceptable only for really small
n, e.g. n ≤ 20.

Algorithms – p. 127
Divide and Conquer Strategy

Algorithms – p. 128
Divide and Conquer
The divide and conquer is a strategy employed to
solve a large number of computational problems:
• Divide: the problem into a small number of
pieces
• Conquer: solve each piece by applying
divide and conquer to it recursively
• Combine: the pieces together into a global
solution.

Algorithms – p. 129
Merge Sort
• Divide and conquer strategy is applicable in a
huge number of computational problems.
• The first example of divide and conquer
algorithm we will discuss is a simple and
efficient sorting procedure called Merge Sort.

Algorithms – p. 130
Merge Sort
• We are given a sequence of n numbers A,
which we will assume are stored in an array
A[1..n].
• The objective is to output a permutation of
this sequence sorted in increasing order.
• This is normally done by permuting the
elements within the array A.

Algorithms – p. 131
Merge Sort
Here is how the merge sort algorithm works:
• Divide: split A down the middle into two
subsequences, each of size roughly n/2
• Conquer: sort each subsequence by calling
merge sort recursively on each.
• Combine: merge the two sorted
subsequences into a single sorted list.

Algorithms – p. 132
Merge Sort
• The dividing process ends when we have split
the subsequences down to a single item.
• A sequence of length one is trivially sorted.
• The key operation is the combine stage which
merges together two sorted lists into a single
sorted list.
• Fortunately, the combining process is quite
easy to implement.

Algorithms – p. 133
Merge sort
The divide phase: split the list top-down into
smaller pieces

7 5 2 4 1 6 3 0

7 5 2 4 1 6 3 0

7 5 2 4 1 6 3 0

7 5 2 4 1 6 3 0

split
Algorithms – p. 134
Merge sort
The conquer phase: merge the smaller lists
bottom up into larger pieces

0 1 2 3 4 5 6 7

2 4 5 7 0 1 3 6

5 7 2 4 1 6 0 3

7 5 2 4 1 6 3 0

merge
Algorithms – p. 135
Merge Sort Algorithm

MERGE - SORT(
array A, int p, int r)
1 if (p < r)
2 then
3 q ← (p + r)/2
4 MERGE - SORT(A, p, q) // sort A[p..q]
5 MERGE - SORT(A, q + 1, r) // sort A[q + 1..r]
6 MERGE(A, p, q, r) // merge the two pieces

Algorithms – p. 136
Merge Sort Algorithm
ME RGE ( array A, int p, int q int r)
1 int B[p..r]; int i ← k ← p; int j ← q + 1
2 while (i ≤ q) and (j ≤ r)
3 do if (A[i] ≤ A[j])
4 then B[k ++ ] ← A[i ++ ]
5 else B[k ++ ] ← A[j ++ ]
6 while (i ≤ q)
7 do B[k ++ ] ← A[i ++ ]
8 while (j ≤ r)
9 do B[k ++ ] ← A[j ++ ]
10 for i ← p to r
11 do A[i] ← B[i]

Algorithms – p. 137
Analysis of Merge Sort
• First, consider the running time of the
procedure merge(A, p, q, r).
• Let n = r − p + 1 denote the total length of
both left and right sub-arrays, i.e., sorted
pieces.
• The merge procedure contains four loops,
none nested in the other.
• Each loop can be executed at most n times.
• Thus running time of merge is Θ(n)

Algorithms – p. 138
Analysis of Merge Sort
• Let T (n) denote the worst case running time
of MergeSort on an array of length n.
• If we call MergeSort with an array containing
a single item (n = 1) then the running time is
constant.
• We can just write T (n) = 1, ignoring all
constants.

Algorithms – p. 139
Analysis of Merge Sort
• For n > 1, MergeSort splits into two halves,
sorts the two and then merges them together.
• The left half is of size n/2 and the right half
is n/2 .
• How long does it take to sort elements in sub
array of size n/2?
• We do not know this but because n/2 < n
for n > 1, we can express this as T (n/2).

Algorithms – p. 140
Analysis of Merge Sort
• Similarly the time taken to sort right sub array
is expressed as T (n/2 ).
• In conclusion we have

1 if n = 1,
T (n) =
T (n/2) + T (n/2 ) + n otherwise

• This is called recurrence relation, i.e., a

recursively defined function.

Algorithms – p. 141
Solving the Recurrence
Let’s expand the terms.
T (1) = 1
T (2) = T (1) + T (1) + 2 = 1 + 1 + 2 = 4
T (3) = T (2) + T (1) + 3 = 4 + 1 + 3 = 8
T (4) = T (2) + T (2) + 4 = 8 + 8 + 4 = 12
T (5) = T (3) + T (2) + 5 = 8 + 4 + 5 = 17
...

Algorithms – p. 142
Solving the Recurrence

...
T (8) = T (4) + T (4) + 8 = 12 + 12 + 8 = 32
...
T (16) = T (8) + T (8) + 16 = 32 + 32 + 16 = 80
...
T (32) = T (16) + T (16) + 32 = 80 + 80 + 32 = 192

Algorithms – p. 143
Solving the Recurrence
What is the pattern here?
• Let’s consider the ratios T (n)/n for powers of
2:
T (1)/1 = 1 T (8)/8 = 4
T (2)/2 = 2 T (16)/16 = 5
T (4)/4 = 3 T (32)/32 = 6
• This suggests T (n)/n = log n + 1
• Or, T (n) = n log n + n which is Θ(n log n) (use
the limit rule).
Algorithms – p. 144
Eliminating Floor and Ceiling
• Floor and ceilings are a pain to deal with.
• If n is assumed to be a power of 2 (2k = n),
this will simplify the recurrence to

1 if n = 1,
T (n) =
2T (n/2) + n otherwise

Algorithms – p. 145
The Iteration Method
• The iteration method turns the recurrence into
a summation.
• Let’s see how it works.

Algorithms – p. 146
The Iteration Method
Let’s expand the recurrence:
T (n) = 2T (n/2) + n
= 2(2T (n/4) + n/2) + n
= 4T (n/4) + n + n
= 4(2T (n/8) + n/4) + n + n
= 8T (n/8) + n + n + n
= 8(2T (n/16) + n/8) + n + n + n
= 16T (n/16) + n + n + n + n

Algorithms – p. 147
The Iteration Method

If n is a power of 2 then let n = 2k or k = log n.

T (n) = 2kT (n/(2k)) + (n + n + n + · · · + n)

 

k times
= 2kT (n/(2k)) + kn
(log n) (log n)
=2 T (n/(2 )) + (log n)n
= 2(log n)T (n/n) + (log n)n
= nT (1) + n log n = n + n log n

Algorithms – p. 148
Mergesort Recurrence Tree
time to merge

n =n
+
n/2 n/2 2(n/2) = n
+
log(n)+1
n/4 n/4 n/4 n/4 4(n/4) = n levels
+
n/8 n/8 n/8 n/8 n/8 n/8 n/8 n/8 8(n/8) = n
.....
+
1 1 1 1 ...... 1 1 1 1 1 ...... 1 n(n/n) = n

n(log(n)+1)

Algorithms – p. 149
A Messier Example
• The merge sort recurrence was too easy.
• Let’s try a messier recurrence.

1 if n = 1,
T (n) =
3T (n/4) + n otherwise

• Assume n to be a power of 4, i.e., n = 4k and

k = log4 n

Algorithms – p. 150
The Iteration Method
T (n) = 3T (n/4) + n
= 3(3T (n/16) + (n/4) + n
= 9T (n/16) + 3(n/4) + n
= 27T (n/64) + 9(n/16) + 3(n/4) + n
= ...

Algorithms – p. 151
The Iteration Method
n
T (n) = 3 T ( k ) + 3k−1(n/4k−1)
k
4
+ · · · + 9(n/16) + 3(n/4) + n
k−1 i

k n 3
= 3 T( k) + n
4 4i
i=0

Algorithms – p. 152
The Iteration Method
With n = 4k and T (1) = 1
k−1 i

k n 3
T (n) = 3 T ( k ) + n
4 4i
i=0
(log4 n)−1
 3i
= 3log4 nT (1) + i
n
4
i=0
(log4 n)−1
 3i
= nlog4 3 + n
4i
i=0

We used the formula alogb n = nlogb a .

Algorithms – p. 153
The Iteration Method
n remains constant throughout the sum and
3i/4i = (3/4)i ; we thus have
(log4 n)−1
i
 3
3
T (n) = n log4
+n
4
i=0

The sum is a geometric series; recall that for

x = 1
m
xm+1 − 1
xi =
x−1
i=0

Algorithms – p. 154