2.

ALGORITHM
‣ computational tractability
ANALYSIS
‣ asymptotic order of growth
‣ implementing Gale–Shapley
‣ survey of common running
times

Lecture slides by Kevin Wayne

Copyright © 2005 Pearson-Addison Wesley
h t tp:// www.cs . p r i n c e t o n . e d u / ~w a yn e /
klein b e rg - t ardos

Last updated on 1/15/20 6:26 AM

 Fundamentals

• What is an algorithm?
• A systematic approach
• Efficiency in terms of
• Computation
• space
• What is algorithm analysis??
• Different alternatives???
• Sorting
• Buble sort, merge sort, etc.
• Which one is efficient?
Basics of Algorithm Analysis

• Analyzing algorithms for efficiency

• Time
• Space
• Scaling the input
• Concrete footing to traceability
• Explain the mathematical functions to precisely understand
the growth as the input scale
• develop the mathematical functions for the various
algorithms
• The importance of data structure is also explored
2.1 Computational Tractability

• Efficiency????
• (1)An algorithm is efficient if, when implemented, it runs quickly
on real input instances
• Where and how well, it is implemented
• What is Real input instance,
• The scale of the problem to unexpected range
• (2) efficiency that is platform-independent, instance-
independent, and of predictive value with respect to increasing
input sizes
• “size” parameter N; input size, the
function is represented suppressing the
rest of the details
• Stable Matching problem
• N is the preference lists
• 2n preference lists, each is of size n, N = 2n*n= 2n
Worst-Case Running Times and Brute-Force
Search

• Worst-case running times

• a bound on the largest possible running time the algorithm could have over all inputs of a
given size N,
• It may be for a few input instances for some problems, but it holds for many problems
• Average-case running times
• Generate for some random input instances
• How should random input instances selected??
• whether a running-time bound is impressive or weak?
• comparison with brute-force search over the search space of possible solutions
• E.g. S-M problem
• n! perfect matching, brute -force finds a stable matching among thesm
• G-S algorithm takes time proportional to N
• E.g. Searching largest value in a list A={20, 30, 31, 33, 45, 100, 500, 5001, 10000, 10001}
• Brute-force 10 steps
• Binary-search, 4 steps
• Similarly for shortest-path in a graph
• Analytical level conclusion instead of practical implementation
• Design level efforts…
• a compact representation, implicitly specifying a giant search space
• Algorithms that improve substantially on brute-force search nearly always contain a
valuable heuristic idea (intrinsic structure/ computational tractability )that makes them
work;
 Efficiency

3. An algorithm is efficient if it achieves

qualitatively better worst-case performance, at
an analytical level, than
brute-force search.
• To quantify the qualitatively better
 Polynomial Time as a Definition of Efficiency

• To quantify the the qualitatively better

• A good algorithm for such a problem to have a better scaling
property,
• when the input size increases by a constant factor—say, a factor of 2—the
algorithm should only slow down by some constant factor C.

• polynomial running time/polynomial-time algorithm

• There are absolute constants
• c>0
• d>0
• on every input instance of size N,
• running time is bounded by cNd steps (statements), next slide
• If N >> to 2N
• Running time
• >>2d(cNd)
• Efficiency (4): An algorithm is efficient if it has
a polynomial running time.
Polynomial Time as a Definition of Efficiency (cont…)

• an algorithm with running time

• proportional to n100—and hence polynomial Better in practic

• a nonpolynomial running time of n1+.02(log n)

• Which one is better?
Summary????
 2.2 Asymptotic Order of Growth
• ability to express the notion that an algorithm’s worst-case running time on inputs of size
n grows at a rate that is at most proportional to some function f (n).
• function f (n) then becomes a bound on the running time of the algorithm.
• pseudo-code style for analysis
• Suppose f (n) = 1.62n2 + 3.5n + 8 steps
• What is wrong????
• Exhausting activity, we need less detail like it is polynomial or exponential as in Table 2.1
• identify broad classes of algorithms with similar behavior
• Meaningless
• Use pseudo-code style for analysis
• C++/ Java/ computer architecture will be different
• f (n) is expressed to be insensitive to constant factors and low order term
• f (n) grows like n2
 2.2 Asymptotic Order of Growth (cont…)
• Asymptotic Upper Bounds
• Suppose
• T(n) be a function—say, the worst-case running time of a certain algorithm on an input of size n.
• Given another function f (n), we say that T(n) is O(f (n)) (read as “T(n) is order f (n)”) if, for
sufficiently large n, the function T(n) is bounded above by a constant multiple of f (n).
• T(n) = O(f(n)), if there exist constants c > 0 and n0 ≥ 0 so that for all n ≥ n0, we have T(n) ≤ c · f
(n). In this case, we will say that T is asymptotically upperbounded by f .
• Suppose T(n) = pn2 + qn + r for positive constants p, q, and r.
• we notice that for all n ≥ 1, we have qn ≤ qn2, and r ≤ rn2. So we can write
• T(n) = pn2 + qn + r ≤ pn2 + qn2 + rn2 = (p + q + r)n2 = cn2 , T(n) ≤ cn2
• O(·) expresses only an upper bound, not the exact growth rate of the function.
• T(n) = O(n2), T(n) = O(n3), why?, “tighter” and “weaker”
• First, T(n) = O(n3), then later on T(n) = O(n2)
 2.2 Asymptotic Order of Growth (cont…)
• Asymptotic Lower Bounds
• algorithm—say we have just proven that its worst-case running time T(n) is O(n 2)—we want
to show that this upper bound is the best one possible.
• We want to express the notion that for arbitrarily large input sizes n, the function T(n) is at
least a constant multiple of some specific function f (n).
• T(n) = Ω(f (n))) if there exist constants c1 >0 and n0 ≥ 0 so that for all n ≥ n0, we have T(n) ≥
c1 · f (n).
• Suppose T(n) = pn2 + qn + r for positive constants p, q, and r.
• we notice that for all n ≥ 1, we have
• T(n) = pn2 + qn + r ≥ pn2 = (p)n2 = c1n2 , T(n) ≥ c1n2
• T(n) = Ω(n2),
• Tighter and weaker bounds, e.g.
• T(n) ≥ pn2 ≥ (p)n = c1n , T(n) ≥ cn
2.2 Asymptotic Order of Growth (cont…)
• Asymptotically Tight Bounds
• T(n) = O(f (n)), and T(n) = Ω(f (n)), then in a natural sense we’ve found the “right” bound:
T(n) grows exactly like f (n).
• T(n) = Ө(f(n)) f (n) is an asymptotically tight bound for T(n)
• Suppose T(n) = pn2 + qn + r for positive constants p, q, and r.
• T(n) = O(n2), T(n)= Ө (n 2)
• T(n) = Ω(n2),
• Characterize the worst-case performance of an algorithm precisely up to constant factors.
• E.g, T(n) = O(n3), T(n) =Ω(n2)
• an invitation to search for an asymptotically tight bound on the algorithm’s worst-case running time.
• Obtained directly
• if the ratio of functions f (n) and g(n) converges to a positive constant as n goes to infinity, then f (n) =
Ө (g(n)).
• Its proof next slide
2.2 Asymptotic Order of Growth (cont…)
• Asymptotically Tight Bounds
Properties of Asymptotic Growth Rates
• Transitivity
• (2.2)
• (a) If f = O(g) and g = O(h), then f = O(h).
• (b) If f = Ω(g) and g = Ω(h), then f = Ω(h).
• Proof. For (a), we’re given that for some constants c and n0, we have f
(n) ≤ cg(n) for all n ≥ n0. Also, for some (potentially different)
constants c’ and n0’, we have g(n) ≤ c’h(n) for all n ≥ n0’. So consider
any number n that is at least as large as both n0 and n0’. We have f (n)
≤ cg(n) ≤ cc’h(n), and so f (n) ≤ cc’h(n) for all n ≥ max(n0, n0’). This
latter inequality is exactly what is required for showing that f = O(h).
 Sums of Functions
• (2.4) Suppose that f and g are two functions such that for some other
function h, we have f = O(h) and g = O(h). Then f + g = O(h).
• Proof. We’re given that for some constants c and n0, we have f (n) ≤ ch(n)
for all n ≥ n0. Also, for some (potentially different) constants c’ and n0’, we
have g(n) ≤ c’h(n) for all n ≥ n0’. So consider any number n that is at least
as large as both n0 and n0’. We have f (n) + g(n) ≤ ch(n) + c’h(n). Thus f (n)
+ g(n) ≤ (c + c’))h(n) for all n ≥ max(n0, n0’), which is exactly what is
required for showing that f + g = O(h).
Asymptotic Bounds for Some Common Functions

• Polynomials
• f (n) = a0 + a1 n + a2 n2 + . . .+ ad nd for some integer constant d > 0, where the final coefficient a d
> 0 is nonzero
• d is degree
• their asymptotic rate of growth is determined by their “high-order term”, i.e. degree
• polynomials raise n to a fixed exponent
• E.g. T(n) = pn2 + qn + r
• (2.7) Let f be a polynomial of degree d, in which the coefficient a d is positive. Then f = O(nd).
• Proof. We write f = a0 + a1 n + a2 n2 + . . .+ ad nd, where ad > 0. The upper bound is a direct
application of (2.5). First, notice that coefficients aj for j < d may be negative, but in any case we
have ajnj ≤ |aj|nd for all n ≥ 1. Thus each term in the polynomial is O(nd). Since f is a sum of a
constant number of functions, each of which is O(nd), it follows from (2.5) that f is O(nd).
Asymptotic Bounds for Some Common Functions
• Polynomials
• Using O(·) notation, it’s easy to formally define polynomial time: a
polynomial-time algorithm is one whose running time T(n) is O(n d) for some
constant d, where d is independent of the input size.
• d may not an integer
• O(n1.59)
• O(n1/2).
• O(n log n). log n ≤ n for all n ≥ 1, and hence n log n ≤ n2 for all n ≥ 1. In other words, if
an algorithm has running time O(n log n), then it also has running time O(n2), and so it
is a polynomial-time algorithm
Asymptotic Bounds for Some Common Functions
• Logarithms
• logb n =x >> bx=n
• logarithms are very slowly growing functions
• if we round it down to the nearest integer, it is one less than the number of digits in
the base-b representation of the number n.
• for example, 1+ log2 n, rounded down, is the number of bits needed to represent n.)
• (2.8) For every b > 1 and every x > 0, we have logb n = O(nx).
• >> , O(log n), base is not important
Asymptotic Bounds for Some Common Functions

• Exponentials
• f (n) = rn for some constant base r.
• r > 1, a very fast-growing function.
• Exponentials raise a fixed number to n as a power
• (2.9) For every r > 1 and every d > 0, we have nd = O(rn).
• people write “The running time of this algorithm is exponential,”
Without specifying which exponential function they have in mind
What does it mean?
Implementing the Stable Matching Algorithm
Using Lists and Arrays
• What is an algorithm?
• A systematic approach
• Efficiency in terms of
• Computation, number of steps, stored-program concept, data structure for storing the data…
• Space
Implementing the Stable Matching Algorithm
Using Lists and Arrays (Cont…)

• What is data structure??

• List

Single Linked list

Doubly Linked list

• Array
Implementing the Stable Matching Algorithm
Using Lists and Arrays (Cont…)
• Every man and woman
preference list
• Free men/women
• Engaged pair
• For above, need data
structure
• Preprocessing data for data
structure choice
• An array is less good for
dynamically maintaining a list
of elements that changes
over time, such as the list of
free men in the S-M algori
• dynamic set of elements is
• via a linked list
• doubly linked list
• See next slide for deletion
• Linked list cannot find the ith
element of the list in O(1)
time
Implementing the Stable Matching Algorithm
Using Lists and Arrays (Cont…)
• Implementing the Stable
Matching Algorithm
• O(n2) for steps
• we need to be able to
implement each iteration
in constant time
• M and W
• Size n
• Ordered
• Associated number i, mi, wi
• ManPref[m, i] to denote
the ith woman on man
m’s preference list,
• WomanPref[w, i]
• What data structure for
each of the above and
why?
Implementing the Stable Matching Algorithm Using Lists and Arrays
(Cont…)

1. We need to be able to
identify a free man
i. as a linked list
ii. First free man
iii. delete m from the list if he becomes engaged
iv. m’ can be inserted at the front of the list,
v. in constant time.
vi. What if it was array????
2. We need, for a man m, to
be able to identify the
highest-ranked woman to
whom he has not yet
proposed.
i. maintain an extra array Next that indicates for
each man m the position of the next woman he
will propose to on his list.
ii. initialize Next[m]= 1
iii. If a man m needs to propose to a woman, he’ll
propose to w = ManPref[m, Next[m]], and
iv. once he proposes to w, we increment the value of
Next[m] by one, regardless of whether or not w
accepts the proposal.
Implementing the Stable Matching Algorithm Using Lists and Arrays
(Cont…)
• For a woman w, we need to
decide if w is currently
engaged, and if she is, we
need to identify her current
partner.
• maintaining an array Current of length n, where
Current[w] is the woman w’s current partner m’.
• set Current[w] to a special null symbol when we need to
indicate that woman w is not currently engaged; at the
start of thealgorithm,
• Current[w] is initialized to this null symbol for all women
w.
• For a woman w and two men
m and m’, we need to be
able to decide, again in
constant time, which of m or
m’ is preferred by w.
• Keeping the women’s preferences in an array
WomanPref, analogous to the one we used for men,
does not work,
• O(n) to find m and m’ in WomanPref array
we create an n × n array
Ranking, where Ranking[w,m]
contains the rank of man m in
the sorted order of w’s
preferences.
By a single pass through w’s
preference list, we can create
this array in
linear time for each woman,
for a total initial time
investment proportional to
n2.
Then, to decide which of m or
m’ is preferred by w, we
simply compare
the values Ranking[w,m] and
Ranking[w,m’].
This allows us to execute step
(4) in constant time, and
2.4 A Survey of Common Running Times

• there are styles of analysis that recur frequently

• “search space”—the set of all possible solutions
• Search for good performance algorithm
• Encounter a New problem
• one on the running time you hope to achieve,
• and the other on the size of the problem’s natural search space (and hence on the running time of a brute-
force algorithm for the problem).
2.4 A Survey of Common Running Times (cont…)

• Linear Time
• O(n)
• its running time is at most a
constant factor times the size of
the input
• to process the input
• in a single pass,
• spending a constant amount of
time on each item of input
encountered.
2.4 A Survey of Common Running Times (cont…)
• Linear Time
• Merging Two
Sorted Lists,
Figure 2.2
• Another way,
we could just
throw the two
lists together,
ignore the fact
that they’re
separately
arranged in
ascending order,
• and run a sorting algorithm,
• Which one is good????
2.4 A Survey of Common Running Times (cont…)

• O(n log n) Time

• it is the running time of divide-
and-conquer recursive algorithms
• Mergesorting
• Merging done in linear time
• Most of the problems have
sorting, so they have O(nlogn)
• For example, suppose we are
given a set of n time-stamps x1,
x2, . . . , xn on which copies of a
file arrived at a server, and we’d
like to find the largest interval of
time between the first and last of
these time- stamps during which
no copy of the file arrived.
• Solution
• Sort the xi
• Find different between two xi and xj successors
• The largest one is the answer
2.4 A Survey of Common Running Times (cont…)

• Quadratic Time
• Suppose, a plane to find the closest two points
• Bruteforce solution
• Enumerate all the pairs
• compute the distance between each pair
• Smallest distance pair is the answer
• The number of pairs of points = O(n2 )
• It results due to
• performing a search over all pairs of input items and spending constant time per pair.
• nested loops
2.4 A Survey of Common Running Times (cont…)

• Cubic Time
• O(n3) time
 2.
ALGORITHM
‣ computational tractability
ANALYSIS
‣ asymptotic order of growth
‣ implementing Gale–Shapley
‣ survey of common running
times

SECTION 2.1
A strikingly modern thought

“ As soon as an Analytic Engine exists, it will necessarily guide the future

course of the science. Whenever any result is sought by its aid, the
question will arise—By what course of calculation can these results be
arrived at by the machine in the shortest time? ” — Charles Babbage
(1864)

how many times do you

have to turn the crank?

Analytic Engine
3
9
Models of computation: Turing machines

Deterministic Turing machine. Simple and idealistic model.

Running time. Number of steps.

Memory. Number of tape cells
utilized.

Caveat. No random access of memory.

・ Single-tape TM requires ≥ n2 steps
to detect n-bit palindromes.
・ Easy to detect palindromes in ≤ cn
4
0
Models of computation: word RAM

assume w ≥ log2 n
Word RAM.
・ Each memory location and input/output cell stores a w-bit integer.
・ Primitive operations: arithmetic/logic operations, read/write
memory, array indexing, following a pointer, conditional branch, …

constant-time C-style operations

(w = 64)
input … memory

program a[i] i
.
.
.
output …

Running time. Number of primitive operations.

Memory. Number of memory cells utilized.

Caveat. At times, need more refined model

(e.g., multiplying n-bit integers). 4
1
Brute force

Brute force. For many nontrivial problems, there is a natural brute-

force search algorithm that checks every possible solution.
・ Typically takes 2n steps (or worse) for inputs of size n.
・ Unacceptable in practice.

Ex. Stable matching problem: test all n! perfect matchings for stability. 42
Polynomial running time

Desirable scaling property. When the input size doubles, the algorithm
should slow down by at most some multiplicative constant factor C.

Def. An algorithm is poly-time if the above scaling property holds.

There exist constants c > 0 and d > 0 such that,

for every input of size n, the running time of the algorithm
corresponds
is bounded above by c nd primitive computational steps.
to C = 2d

von Neumann Nash Gödel Cobham Edmonds Rabin

(1953) (1955) (1964) (1965) (1966)
(1956)
43
Polynomial running time

We say that an algorithm is efficient if it has a polynomial running time.

Theory. Definition is (relatively) insensitive to model of computation.

Practice. It really works!

・ The poly-time algorithms that people develop have
both small constants and small exponents.
・ Breaking through the exponential barrier of brute force
typically exposes some crucial structure of the problem. n120

Map graphs in polynomial time

Exceptions. Some poly-time algorithms in the wild
Mikkel Thorup

have galactic constants and/or huge exponents.

Department of Computer Science, University of Copenhagen
Universitetsparken 1, DK-2100 Copenhagen East,
Denmark [email protected]

Abstract

Chen, Grigni, and Papadimitriou (WADS’97 and

STOC’98) have introduced a modified notion of planarity,
where two faces are considered adjacent if they share at
least one point. The corresponding abstract graphs are
called map graphs. Chen et.al. raised the question of
whether map graphs can be recognized in polynomial time.

Q. Which would you prefer: 20 n120 or n1 + 0.02 ln n ?

They showed that the decision problem is in NP and
presented a polynomial time algorithm for the special case
where we allow at most 4 faces to intersect in any point — if
only 3 are allowed to intersect in a point, we get the usual
planar graphs.
Chen et.al. conjectured that map graphs can be
recognized in polynomial time, and in this paper, their
conjecture is settled affirmatively.

44
Worst-case analysis

Worst case. Running time guarantee for any input of size n.

・ Generally captures efficiency in practice.
・ Draconian view, but hard to find effective alternative.

Exceptions. Some exponential-time algorithms are used widely in practice

because the worst-case instances don’t arise.

simplex algorithm Linux grep k-means algorithm

45
Other types of analyses

Probabilistic. Expected running time of a randomized algorithm.

Ex. The expected number of compares to quicksort n elements is ~ 2n ln n.

Amortized. Worst-case running time for any sequence of n operations.

Ex. Starting from an empty stack, any sequence of n push and pop
operations takes O(n) primitive computational steps using a resizing array.

Also. Average-case analysis, smoothed analysis, competitive analysis, ...

46
 2.
ALGORITHM
‣ computational tractability
ANALYSIS
‣ asymptotic order of growth
‣ implementing Gale–Shapley
‣ survey of common running
times

SECTION 2.2
Big O notation

Upper bounds. f (n) is O(g(n)) if there exist constants c > 0 and n0 ≥ 0

such that 0 ≤ f (n) ≤ c · g (n) for all n ≥ n0.
c · g(n)

Ex. f (n) = 32n2 + 17n + f

1. choose c = 50, n0 = 1
(n)

・ f (n) is
・ 2).f (n) is neither O(n) nor O(n log
O(n
n). n0 n

Typical usage. Insertion sort makes O(n2) compares to sort n elements.

12
Analysis of algorithms: quiz 1

Let f(n) = 3n2 + 17 n log2 n + 1000. Which of the following are true?

A. f (n) is choose c = 1020, n0 = 1

O(n2).

B. f (n) is choose c = 1020, n0 = 1

O(n3).

C. Both A and B.

D. Neither A nor B.

13
Big O notational abuses

One-way “equality.” O(g(n)) is a set of functions, but computer

scientists often write f (n) = O(g(n)) instead of f (n)  O(g(n)).

Ex. Consider g1(n) = 5n3 and g2(n) = 3n2.

・ We have g1(n) = O(n3) and g2(n) = O(n3).
・ But, do not conclude g1(n) = g2(n).

Domain and codomain. f and g are real-valued functions.

・ The domain is typically the natural numbers: ℕ 

ℝ. Sometimes we extend to the reals: ℝ  input size, recurrence relations

・ ≥0

・ Or restrict to a
ℝ. plotting, limits, calculus

subset.

Bottom line. OK to abuse notation in this way; not OK to misuse it.

50
Big O notation: properties

Reflexivity. f is
O( f ).
Constants. If f is O(g) and c > 0, then c f is O(g).

Products. If f 1 is O(g1) and f 2 is O(g2), then f 1 f 2 is O(g1 g2).

Pf.
・∃ c1 > 0 and n1 ≥0 such that 0 ≤ f 1(n) ≤ c1 · g1(n) for all n ≥ n .
1

・∃ c2 > 0 and n2 ≥0 such that 0 ≤ f 2(n) ≤ c2 · g2(n) for all n ≥ n .

2

・ Then, 0 ≤ f 1(n) · f 2(n) ≤ c1 · c2 · g1(n) · g2(n) for all n ≥ max { n1, n2 }. ▪

c

n0

Sums. If f 1 is O(g1) and f2 is O(g2), then f 1 + f 2 is O(max {g1, g2}).

ignore lower-order terms
Transitivity. If f is O(g) and g is O(h), then f is O(h).

Ex. f (n) = 5n3 + 3n2 + n + 1234 is

O(n3).
51
Big Omega notation

Lower bounds. f (n) is Ω(g(n)) if there exist constants c > 0 and n0 ≥ 0

such that f (n) ≥ c · g(n) ≥ 0 for all n ≥ n0.
f
(n)
Ex. f (n) = 32n2 + 17n + 1. c · g(n)
・ f (n) is both Ω(n2) and choose c = 32, n0 = 1

Ω(n).
・ f (n) is not Ω(n3).
n0 n

Typical usage. Any compare-based sorting algorithm requires Ω(n log n)

compares in the worst case.

Vacuous statement. Any compare-based sorting algorithm requires

at least O(n log n) compares in the worst case.

52
Analysis of algorithms: quiz 2

Which is an equivalent definition of big Omega notation?

A. f (n) is Ω(g(n)) iff g (n) is O( f (n)).

B. f (n) is Ω(g(n)) iff there exists a constant c > 0 such that f (n) ≥ c·
g(n) ≥ 0
for infinitely many n.

C. Both A and B.

D. Neither A nor B.

17
Big Theta notation

Tight bounds. f (n) is Θ(g(n)) if there exist constants c1 > 0, c2 > 0, and n0 ≥
0
c2 · g(n)
such that 0 ≤ c1 · g(n) ≤ f (n) ≤ c2 · g(n) for all n ≥ n0.
f
(n)
Ex. f (n) = 32n2 + 17n + c1 · g(n)
1. choose c1 = 32, c2 = 50, n0 = 1
・ f (n) is
・(n2).f (n) is neither Θ (n) nor
Θ
Θ (n3). n0 n

Typical usage. Mergesort makes Θ(n log n) compares to sort n elements.

between ½ n log2 n
and n log2 n

19
Analysis of algorithms: quiz 3

Which is an equivalent definition of big Theta notation?

A. f (n) is Θ (g(n)) iff f (n) is both O( g (n)) and Ω( g (n)).

f
B. f (n) is Θ (g(n)) lim = for some constant 0 < c < ∞.
n→ ∞ (n)
g(n) c
iff

C. Both A and B.

D. Neither A nor B. counterexample

2n i7 n ib
f (n) =
2p2M
3n i7 n ib
g(n) = n Q//

f (n) is O(n) but limit does not

exist
20
Asymptotic bounds and limits

Proposition. If lim
f
n→∞ g(n) = c for some constant 0 < c < ∞ then f (n) is
(n)
Θ (g(n)).
Pf.
・ By definition of the limit, for any ε > 0, there exists n0 such

that
f
c—ϵ ≤ ≤ c+
(n)
g(n)
ϵ
for all n ≥ n0.
・ Choose ε = ½ c > 0.
1/2 c · g(n) ≤ f (n) ≤ 3/2 c · g(n) for all n ≥ n0.
・ Multiplying by g(n)
・ Thus, f (n) is Θ(g(n)) by definition, with c1 = 1/2 c and c2 = 3/2 c. ▪
yields

Proposition. If lim f
n → ∞ (n)
g(n) = 0 , then f (n) is O(g(n)) but not
Ω(g(n)).
Proposition. If lim f
n → ∞ (n)
g(n) = ∞ , then f (n) is Ω(g(n)) but not
O(g(n)). 21
Asymptotic bounds for some common functions

Polynomials. Let f (n) = a0 + a1 n + … + ad n d with ad > 0. Then, f (n) is

Θ (n d ). Pf.
a0 + a 1 n + . . . +
lim = ad >
n→ ∞ ad n d nd
0
Logarithms. loga n is Θ(logb n) for every a > 1 and every b > 1.
Pf. loga n no need to specify base
logb n = logb a (assuming it is a constant)
1

Logarithms and polynomials. loga n is O(nd ) for every a > 1 and every d >
Pf. 0.
lim log a n =
n→ ∞ nd
0

Exponentials and polynomials. n d is O(rn ) for every r > 1 and every d > 0.

Pf. lim nd = 0
n
n→ ∞ r

Factorials. n! is 2Θ(n log n).

√ n n
Pf. Stirling’s formula: n! ~
22
2πn
Big O notation with multiple variables

Upper bounds. f (m, n) is O(g(m, n)) if there exist constants c > 0, m0 ≥

0, and n0 ≥ 0 such that 0 ≤ f (m, n) ≤ c · g (m, n) for all
n ≥ n0 and m ≥ m0.

Ex. f (m, n) = 32mn2 + 17mn + 32n3.

・ f (m, n) is both O(mn2 + n3) and O(mn3).
・ f (m, n) is O(n3) if a precondition to the problem implies m ≤ n.
・ f (m, n) is neither O(n3) nor O(mn2).

Typical usage. In the worst case, breadth-first search takes O(m + n)

time to find a shortest path from s to t in a digraph with n nodes and m
edges.

23
 2.
ALGORITHM
‣ computational tractability
ANALYSIS
‣ asymptotic order of growth
‣ implementing Gale–Shapley
‣ survey of common running
times

SECTION 2.3
Efficient implementation

Goal. Implement Gale–Shapley to run in O(n 2) time.

GALE–SHAPLEY (preference lists for n hospitals and n students)

INITIALIZE M to empty matching.

WHILE (some hospital h is unmatched)
s  first student on h’s list to whom h has not yet
proposed. IF (s is unmatched)
Add h–s to matching M.
ELSE IF (s prefers h to current partner
hʹ) Replace hʹ–s with h–s in matching
M.
ELSE
s rejects h.

RETURN stable matching M.

25
Efficient implementation

Goal. Implement Gale–Shapley to run in O(n 2) time.

Representing hospitals and students. Index hospitals and students 1, …,

n. Representing the matching.

・ Maintain two arrays student[h] and hospital[s].
- if h matched to s, then student[h] = s and hospital[s] = h
- use value 0 to designate that hospital or student is unmatched

・ Can add/remove a pair from matching in O(1) time.

・ Maintain set of unmatched hospitals in a queue (or

stack).
・ Can find an unmatched hospital in O(1) time.

26
Data representation: making a proposal

Hospital makes a proposal.

・ Key operation: find hospital’s next favorite student.
・ For each hospital: maintain a list of students, ordered by
preference.
・ For each hospital: maintain a pointer to student for next proposal.

next proposal to

hospital h 3 4 1 5 2 null

favorite least favorite

Bottom line. Making a proposal takes O(1) time.

27
Data representation: accepting/rejecting a proposal

Student accepts/rejects a proposal.

・ Does student s prefer hospital h to hospital hʹ ?
・ For each student, create inverse of preference list of

hospitals.

pref[] 1st 2nd 3rd 4th 5th 6th 7th 8th

8 3 7 1 4 5 6 2 student prefers hospital 4 to 6

since rank[4] < rank[6]

1 2 3 4 5 6 7 8
rank[]

4th 8th 2nd 5th 6th 7th 3rd 1st

for i = 1 to n
rank[pref[i]] =
i

Bottom line. After Θ(n2 ) preprocessing time (to create the n ranking arrays),
it takes O(1) time to accept/reject a proposal. 28
Stable matching:
summary
Theorem. Can implement Gale–Shapley to run in O(n2)
time. Pf.
・ Θ(n2) preprocessing time to create the n ranking
arrays.
・ There are O(n2) proposals; processing each proposal
takes O(1) time.

Theorem. In the worst case, any algorithm to find a stable matching

must query the hospital’s preference list Ω(n2) times.

29
 2.
ALGORITHM
‣ computational tractability
ANALYSIS
‣ asymptotic order of growth
‣ implementing Gale–Shapley
‣ survey of common running
times

SECTION 2.4
Constant time

Constant time. Running time is O(1).

bounded by a constant,
Examples. which does not depend on input size n
・ Conditional branch.
・ Arithmetic/logic operation.
・ Declare/initialize a
variable.
・ Access
Follow aelement
link in ai in an array.
linked
・
list.
Compare/exchange two elements in an
array.
・…

31
Linear time

Linear time. Running time is O(n).

Merge two sorted lists. Combine two sorted linked lists A = a1, a2, …, an and
B = b1, b2, …, bn into a sorted whole.

O(n) algorithm. Merge in mergesort.

i  1; j  1.
WHILE (both lists are nonempty)
IF (ai ≤ bj) append ai to output list and
increment i. ELSE append bj to output list
and increment j.
32
Append remaining elements from nonempty list to
TARGET
SUM
TARGET-SUM. Given a sorted array of n distinct integers and an integer T,
find two that sum to exactly T ?

input
−20 10 20 30 35 40 60 70 T = 60
(sorted)

i j

33
Logarithmic time

Logarithmic time. Running time is O(log n).

Search in a sorted array. Given a sorted array A of n distinct integers and an

integer x, find index of x in array.

remaining elements
O(log n) algorithm. Binary search.
・ Invariant: If x is in the array, then x is in A[lo ..
hi]. → k ≤ 1 + log2 n.
・ After k iterations of WHILE loop, (hi − lo + 1) ≤ n / 2k
lo  1; hi  n.
WHILE (lo ≤ hi)
mid  (lo +
hi) / 2.
IF (x < A[mid]) hi  mid −
1. ELSE IF (x > A[mid]) lo  mid +
1. ELSE RETURN mid.
RETURN −1. 35
 Logarithmic time

O(log n)

https://www.facebook.com/pg/npcompleteteens
SEARCH IN A SORTED ROTATED
ARRAY

SEARCH-IN-SORTED-ROTATED-ARRAY. Given a rotated sorted array of n distinct

integers and an element x, determine if x is in the array.

sorted circular array

20
30
95

35
90

50
85

60
80

65
75 67

sorted rotated array

80 85 90 95 20 30 35 50 60 65 67 75

1 2 3 4 5 6 7 8 9 10 11 12

37
Linearithmic time

Linearithmic time. Running time is O(n log n).

Sorting. Given an array of n elements, rearrange them in ascending order.

O(n log n) algorithm. Mergesort.

0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15
M E R G E S O R T E X A M P L E
E M R G E S O R T E X A M P L E
E M G R E S O R T E X A M P L E
E G M R E S O R T E X A M P L E
E G M R E S O R T E X A M P L E
E G M R E S O R T E X A M P L E
E G M R E O R S T E X A M P L E
E E G M O R R S T E X A M P L E
E E G M O R R S E T X A M P L E
E E G M O R R S E T A X M P L E
E E G M O R R S A E T X M P L E
E E G M O R R S A E T X M P L E
E E G M O R R S A E T X M P E L
E E G M O R R S A E T X E L M P
E E G M O R R S A E E L M P T X
A E E E E G L M M O P R R S T X
39
LARGEST EMPTY
INTERVAL

LARGEST-EMPTY-INTERVAL. Given n timestamps x1, …, xn on which copies of a

file arrive at a server, what is largest interval when no copies of file arrive?

40
Quadratic time

Quadratic time. Running time is O(n2).

Closest pair of points. Given a list of n points in the plane (x1, y1), …, (xn, yn),
find the pair that is closest to each other.

O(n2) algorithm. Enumerate all pairs of points (with i < j).

min  ∞.
FOR i=1
TO n
FOR j
= i + 1 TO n
d

 (xi −
xj)2 + (yi
Remark. Ω(n2) seems inevitable,
− yj)2. but this is just an illusion. [see §5.4]

IF (d < 42
Cubic time

Cubic time. Running time is O(n3).

3-SUM. Given an array of n distinct integers, find three that sum to

0.

O(n3) algorithm. Enumerate all triples (with i < j < k).

FOR i =1 TO n
FOR j = i+1 TO n
FOR k= j+1 TO n
IF (ai + aj + ak =
0) RETURN (ai,
aj, ak).

Remark. Ω(n3) seems inevitable, but O(n2) is not hard. [see next slide]
43
3-
SUM
3-SUM. Given an array of n distinct integers, find three that sum to
0.

O(n3) algorithm. Try all triples.

O(n2) algorithm.

44
Polynomial time

Polynomial time. Running time is O(nk) for some constant k > 0.

Independent set of size k. Given a graph, find k nodes such that no

two are joined by an edge.
k is a constant

O(nk) algorithm. Enumerate all subsets of k nodes.

FOREACH subset S of k nodes:

Check whether S is an independent set.
IF (S is an independent set)
RETURN S.
independent set of size 3

・ Check whether S is an independent set of size k takes O(k2)

time. n n(n — 1)(n — 2) × ·· · × (n — k k

= ≤ n
+ 1) k!
・ Number of k-element subsets =
・ O(k n / k!) =
2 k k k(k — 1)(k — 2) × ·· · × 1

O(nk). poly-time for k = 17, but not practical 46

Exponential time

Exponential time. Running time is O(2nk ) for some constant k > 0.

Independent set. Given a graph, find independent set of max

cardinality.

O(n2 2n) algorithm. Enumerate all subsets of n elements.

S*  .
FOREACH subset S of n nodes:
Check whether S is an independent set.
IF (S is an independent set and S >
S*)
independent set of max cardinality
S* 
S. RETURN
S*.

47
Exponential time

Exponential time. Running time is O(2nk ) for some constant k > 0.

Euclidean TSP. Given n points in the plane, find a tour of minimum

length.

O(n 𐄂 n!) algorithm. Enumerate all permutations of length n.

π*  .
FOREACH permutation π of n points:
Compute length of tour corresponding to π.
IF (length(π) < length(π*))
π*  π.
for simplicity, we’ll assume Euclidean
RETURN π*. distances are rounded to nearest integer
(to avoid issues with infinite precision)

48
Analysis of algorithms: quiz 4

Which is an equivalent definition of exponential time?

A. doesn’t include 3n
O( 2n)

B. O(2cn) for some constant c > 0. includes 3n but doesn’t include n! = 2Θ(n log n)

C. Both A and B.

D. Neither A nor B.

49