Module 1: Introduction and Asymptotic Analysis

CS 240 - Data Structures and Data Management

Arne Storjohann
Based on lecture notes by R. Dorrigiv and D. Roche
David R. Cheriton School of Computer Science, University of Waterloo
Fall 2012
Storjohann (CS, UW) CS240 - Module 1 Fall 2012 1 / 48
Course Information - Spring 2013
Instructor: Mark Petrick, DC 3125
mdtpetri [at] uwaterloo.ca
Lectures

Section 001 (for SE): TTh 8:30am-9:50am in MC 2017

Section 002: TTh 10:00am-11:20am in MC 2038

Section 003: TTh 2:30pm-3:50pm in MC 2038

Oce hours: Tues Noon-2:00pm, Thurs 1:00-2:00pm
Storjohann (CS, UW) CS240 - Module 1 Fall 2012 2 / 48
Course Information
ISAs:
Cristian Stae, MC 4065
cs240 [at] student.cs.uwaterloo.ca
Oce hours: tba
Tutorials:
Monday: various times and places
Storjohann (CS, UW) CS240 - Module 1 Fall 2012 3 / 48
Course Information
Course Webpage:

http://www.student.cs.uwaterloo.ca/
~
cs240/s13/
Course newsgroup:

www.piazza.com/class
Main resource: Lectures

Course slides will be posted on the course webpage but are not a
substitute for in-class lectures
Textbooks:

Algorithms in C++, by Robert Sedgewick, Addison-Wesley, 1998

More books on the webpage under Resources

Topics and references for each lecture will be posted on the Webpage
Storjohann (CS, UW) CS240 - Module 1 Fall 2012 4 / 48
Mark Breakdown
Final 50%
Midterm 25%

Tues, June 25, 4:30-6:20pm

Assignments 25%

5 assignments each worth 5%

Approx. every two weeks

Generally due on Wednesdays

No lates allowed

Follow the assignment guidelines

All assignment to be submitted

electronically via Markus

Note: You must pass

the weighted average of
the midterm and the nal
exam to pass the course
Storjohann (CS, UW) CS240 - Module 1 Fall 2012 5 / 48
Electronic Communication in CS240
http://www.student.cs.uwaterloo.ca/
~
cs240/s13/
Primary source for up-to-date information for CS 240.

Lecture slides

Assignments

Course policies
Piazza
A forum that is optimized for asking questions and giving answers.
You must sign up using your uwaterloo email address.

You can post to piazza using a nickname though

Posting solutions to assignments is forbidden.
Email
For private communication between students and course sta.
You should be sending email from your uwaterloo email address.
Storjohann (CS, UW) CS240 - Module 1 Fall 2012 6 / 48
Cheating
Cheating includes not only copying the work of another person
(or letting another student copy your work),
but also excessive collaboration.
Standard penalties: a grade of 0 on the assignment you cheated on,
and a deduction of 5% from your course grade. You will also be
reported to the Associate Dean of Undergraduate Studies.
Do not take notes during discussions with classmates.
Storjohann (CS, UW) CS240 - Module 1 Fall 2012 7 / 48
Courtesy
Please silence cell phones before coming to class.
Questions are encouraged, but please refrain from talking in class - it
is disrespectful to your classmates who are trying to listen to the
lectures.
Think about whether bringing your laptop to class will help you pay
attention or not.
Dont play games or watch videos that will distract your classmates.
Storjohann (CS, UW) CS240 - Module 1 Fall 2012 8 / 48
Advice
Attend all the lectures and pay attention!
Study the slides before the lectures, and again afterwards.
Read the reference materials to get dierent perspectives on the course
material.
Keep up with the course material! Dont fall behind.
If youre having diculties with the course, seek help.
Storjohann (CS, UW) CS240 - Module 1 Fall 2012 9 / 48
Course Objectives: What is this course about?
The objective of the course is to study ecient methods of storing,
accessing, and performing operations on large collections of data.
Typical operations include: inserting new data items, deleting data
items, searching for specic data items, sorting.
Motivating examples: Digital Music Collection, English Dictionary
We will consider various abstract data types (ADTs) and how to
implement them eciently using appropriate data structures.
There is a strong emphasis on mathematical analysis in the course.
Algorithms are presented using pseudocode and analyzed using order
notation (big-Oh, etc.).
Storjohann (CS, UW) CS240 - Module 1 Fall 2012 10 / 48
Course Topics
priority queues and heaps
sorting, selection
binary search trees, AVL trees, B-trees
skip lists
hashing
quadtrees, kd-trees
range search
tries
string matching
data compression
Storjohann (CS, UW) CS240 - Module 1 Fall 2012 11 / 48
CS Background
Topics covered in previous courses with relevant sections in [Sedgewick]:
arrays, linked lists (Sec. 3.23.4)
strings (Sec. 3.6)
stacks, queues (Sec. 4.24.6)
abstract data types (Sec. 4-intro, 4.1, 4.84.9)
recursive algorithms (5.1)
binary trees (5.45.7)
sorting (6.16.4)
binary search (12.4)
binary search trees (12.5)
Storjohann (CS, UW) CS240 - Module 1 Fall 2012 12 / 48
Problems (terminology)
Problem: Given a problem instance, carry out a particular computational
task.
Problem Instance: Input for the specied problem.
Problem Solution: Output (correct answer) for the specied problem
instance.
Size of a problem instance: Size(I ) is a positive integer which is a
measure of the size of the instance I .
Example: Sorting problem
Storjohann (CS, UW) CS240 - Module 1 Fall 2012 13 / 48
Algorithms and Programs
Algorithm: An algorithm is a step-by-step process (e.g., described in
pseudocode) for carrying out a series of computations, given an arbitrary
problem instance I .
Algorithm solving a problem: An Algorithm A solves a problem if, for
every instance I of , A nds (computes) a valid solution for the instance
I in nite time.
Program: A program is an implementation of an algorithm using a
specied computer language.
In this course, our emphasis is on algorithms (as opposed to programs or
programming).
Storjohann (CS, UW) CS240 - Module 1 Fall 2012 14 / 48
Algorithms and Programs
For a problem , we can have several algorithms.
For an algorithm / solving , we can have several programs
(implementations).
Algorithms in practice: Given a problem
1
Design an algorithm / that solves . Algorithm Design
2
Assess correctness and eciency of /. Algorithm Analysis
3
If acceptable (correct and ecient), implement /.
Storjohann (CS, UW) CS240 - Module 1 Fall 2012 15 / 48
Eciency of Algorithms/Programs
How do we decide which algorithm or program is the most ecient
solution to a given problem?
In this course, we are primarily concerned with the amount of time a
program takes to run. Running Time
We also may be interested in the amount of memory the program
requires. Space
The amount of time and/or memory required by a program will
depend on Size(I), the size of the given problem instance I .
Storjohann (CS, UW) CS240 - Module 1 Fall 2012 16 / 48
Running Time of Algorithms/Programs
First Option: experimental studies
Write a program implementing the algorithm.
Run the program with inputs of varying size and composition.
Use a method like clock() (from time.h) to get an accurate
measure of the actual running time.
Plot/compare the results.
Storjohann (CS, UW) CS240 - Module 1 Fall 2012 17 / 48
Running Time of Algorithms/Programs
Shortcomings of experimental studies
We must implement the algorithm.
Timings are aected by many factors: hardware (processor, memory),
software environment (OS, compiler, programming language), and
human factors (programmer).
We cannot test all inputs; what are good sample inputs?
We cannot easily compare two algorithms/programs.
We want a framework that:
Does not require implementing the algorithm.
Is independent of the hardware/software environment.
Takes into account all input instances.
We need some simplications.
Storjohann (CS, UW) CS240 - Module 1 Fall 2012 18 / 48
Running Time Simplications
Overcome dependency on hardware/software
Express algorithms using pseudo-code
Instead of time, count the number of primitive operations
Random Access Machine (RAM) Model:
The random access machine has a set of memory cells, each of which
stores one item (word) of data.
Any access to a memory location takes constant time.
Any primitive operation takes constant time.
The running time of a program can be computed to be the number of
memory accesses plus the number of primitive operations.
This is an idealized model, so these assumptions may not be valid for a
real computer.
Storjohann (CS, UW) CS240 - Module 1 Fall 2012 19 / 48
Running Time Simplications
Overcome dependency on hardware/software
Express algorithms using pseudo-code.
Instead of time, count the number of primitive operations.
Implicit assumption: primitive operations have fairly similar, though
dierent, running time on dierent systems
Simplify Comparisons
Example: Compare 1000000n + 200000000000000 with 0.01n
2
Idea: Use order notation
Informally: ignore constants and lower order terms
Storjohann (CS, UW) CS240 - Module 1 Fall 2012 20 / 48
Order Notation
O-notation: f (n) O(g(n)) if there exist constants c > 0 and n
0
> 0
such that 0 f (n) c g(n) for all n n
0
.
-notation: f (n) (g(n)) if there exist constants c > 0 and n
0
> 0
such that 0 c g(n) f (n) for all n n
0
.
-notation: f (n) (g(n)) if there exist constants c
1
, c
2
> 0 and n
0
> 0
such that 0 c
1
g(n) f (n) c
2
g(n) for all n n
0
.
o-notation: f (n) o(g(n)) if for all constants c > 0, there exists a
constant n
0
> 0 such that 0 f (n) c g(n) for all n n
0
.
-notation: f (n) (g(n)) if for all constants c > 0, there exists a
constant n
0
> 0 such that 0 c g(n) f (n) for all n n
0
.
Storjohann (CS, UW) CS240 - Module 1 Fall 2012 21 / 48
Example of Order Notation
In order to prove that 2n
2
+3n +11 O(n
2
) from rst principles, we need
to nd c and n
0
such that the following condition is satised:
0 2n
2
+ 3n + 11 c n
2
for all n n
0
.
note that not all choices of c and n
0
will work.
Storjohann (CS, UW) CS240 - Module 1 Fall 2012 22 / 48
Example of Order Notation
Prove that 2010n
2
+ 1388n o(n
3
) from rst principles.
Storjohann (CS, UW) CS240 - Module 1 Fall 2012 23 / 48
Complexity of Algorithms
Our goal: Express the running time of each algorithm as a function f (n) in
terms of the input size.
Let T
A
(I ) denote the running time of an algorithm A on a problem
instance I .
An algorithm can have dierent running times on input instances of the
same size.
Average-case complexity of an algorithm
Worst-case complexity of an algorithm
Storjohann (CS, UW) CS240 - Module 1 Fall 2012 24 / 48
Complexity of Algorithms
Average-case complexity of an algorithm: The average-case running
time of an algorithm A is a function f : Z
+
R mapping n (the input
size) to the average running time of A over all instances of size n:
T
avg
A
(n) =
1
[I : Size(I ) = n[

{I :Size(I )=n}
T
A
(I ).
Worst-case complexity of an algorithm: The worst-case running time
of an algorithm A is a function f : Z
+
R mapping n (the input size) to
the longest running time for any input instance of size n:
T
A
(n) = maxT
A
(I ) : Size(I ) = n.
Storjohann (CS, UW) CS240 - Module 1 Fall 2012 25 / 48
Growth Rates
If f (n) (g(n)), then the growth rates of f (n) and g(n) are the
same.
If f (n) o(g(n)), then we say that the growth rate of f (n) is
less than the growth rate of g(n).
If f (n) (g(n)), then we say that the growth rate of f (n) is
greater than the growth rate of g(n).
Typically, f (n) may be complicated and g(n) is chosen to be a very
simple function.
Storjohann (CS, UW) CS240 - Module 1 Fall 2012 26 / 48
Common Growth Rates
Commonly encountered growth rates in analysis of algorithms include the
following (in increasing order of growth rate):
(1) (constant complexity),
(log n) (logarithmic complexity),
(n) (linear complexity),
(n log n)(linearithmic/pseudo-linear complexity),
(n
2
) (quadratic complexity),
(n
3
) (cubic complexity),
(2
n
) (exponential complexity).
Storjohann (CS, UW) CS240 - Module 1 Fall 2012 27 / 48
How Growth Rates Aect Running Time
It is interesting to see how the running time is aected when the size of
the problem instance doubles (i.e., n 2n).
constant complexity: T(n) = c, T(2n) = c.
logarithmic complexity: T(n) = c log n, T(2n) = T(n) + c.
linear complexity: T(n) = cn, T(2n) = 2T(n).
(n log n): T(n) = cn log n, T(2n) = 2T(n) + 2cn.
quadratic complexity: T(n) = cn
2
, T(2n) = 4T(n).
cubic complexity: T(n) = cn
3
, T(2n) = 8T(n).
exponential complexity: T(n) = c2
n
, T(2n) = (T(n))
2
/c.
Storjohann (CS, UW) CS240 - Module 1 Fall 2012 28 / 48
Complexity vs. Running Time
Suppose that algorithms A
1
and A
2
both solve some specied
problem.
Suppose that the complexity of algorithm A
1
is lower than the
complexity of algorithm A
2
. Then, for suciently large problem
instances, A
1
will run faster than A
2
. However, for small problem
instances, A
1
could be slower than A
2
.
Now suppose that A
1
and A
2
have the same complexity. Then we
cannot determine from this information which of A
1
or A
2
is faster; a
more delicate analysis of the algorithms A
1
and A
2
is required.
Storjohann (CS, UW) CS240 - Module 1 Fall 2012 29 / 48
Example
Suppose an algorithm A
1
with linear complexity has running time
T
A
1
(n) = 75n + 500 and an algorithm with quadratic complexity has
running time T
A
2
(n) = 5n
2
. Then A
2
is faster when n 20 (the
crossover point). When n > 20, A
1
is faster.
0
1,500
15
3,000
2,500
2,000
1,000
20 5 10 0
x
500
25
Storjohann (CS, UW) CS240 - Module 1 Fall 2012 30 / 48
O-notation and Complexity of Algorithms
It is important not to try and make comparisons between algorithms
using O-notation.
For example, suppose algorithm A
1
and A
2
both solve the same
problem, A
1
has complexity O(n
3
) and A
2
has complexity O(n
2
).
The above statements are perfectly reasonable.
Observe that we cannot conclude that A
2
is more ecient than A
1
in
this situation! (Why not?)
Storjohann (CS, UW) CS240 - Module 1 Fall 2012 31 / 48
Techniques for Order Notation
Suppose that f (n) > 0 and g(n) > 0 for all n n
0
. Suppose that
L = lim
n
f (n)
g(n)
.
Then
f (n)

o(g(n)) if L = 0
(g(n)) if 0 < L <
(g(n)) if L = .
The required limit can often be computed using lHopitals rule. Note that
this result gives sucient (but not necessary) conditions for the stated
conclusions to hold.
Storjohann (CS, UW) CS240 - Module 1 Fall 2012 32 / 48
An Example
Compare the growth rates of log n and n
i
(where i > 0 is a real number).
Storjohann (CS, UW) CS240 - Module 1 Fall 2012 33 / 48
Example
Prove that n(2 + sin n/2) is (n). Note that lim
n
(2 + sin n/2) does
not exist.
60
x
25 20
0
15 10 5 0
40
20
Storjohann (CS, UW) CS240 - Module 1 Fall 2012 34 / 48
Relationships between Order Notations
f (n) (g(n)) g(n) (f (n))
f (n) O(g(n)) g(n) (f (n))
f (n) o(g(n)) g(n) (f (n))
f (n) (g(n)) f (n) O(g(n)) and f (n) (g(n))
f (n) o(g(n)) f (n) O(g(n))
f (n) o(g(n)) f (n) , (g(n))
f (n) (g(n)) f (n) (g(n))
f (n) (g(n)) f (n) , O(g(n))
Storjohann (CS, UW) CS240 - Module 1 Fall 2012 35 / 48
Algebra of Order Notations
Maximum rules: Suppose that f (n) > 0 and g(n) > 0 for all n n
0
.
Then:
O(f (n) + g(n)) = O(maxf (n), g(n))
(f (n) + g(n)) = (maxf (n), g(n))
(f (n) + g(n)) = (maxf (n), g(n))
Transitivity: If f (n) O(g(n)) and g(n) O(h(n)) then f (n) O(h(n)).
Storjohann (CS, UW) CS240 - Module 1 Fall 2012 36 / 48
Summation Formulae
Arithmetic sequence:
n1

i =0
(a + di ) = na +
dn(n 1)
2
(n
2
).
Geometric sequence:
n1

i =0
a r
i
=

a
r
n
1
r 1
(r
n
) if r > 1
na (n) if r = 1
a
1r
n
1r
(1) if 0 < r < 1.
Harmonic sequence:
H
n
=
n

i =1
1
i
(log n)
Storjohann (CS, UW) CS240 - Module 1 Fall 2012 37 / 48
More Formulae and Miscellaneous Math Facts

n
i =1
i r
i
=
nr
n+1
r 1

r
n+1
r
(r 1)
2

i =1
i
2
=

2
6
for k 0,

n
i =1
i
k
(n
k+1
)
log
b
a =
1
log
a
b
log
b
a =
log
c
a
log
c
b
a
log
b
c
= c
log
b
a
n!

n
n+1/2
e
n

log n! (n log n)
Storjohann (CS, UW) CS240 - Module 1 Fall 2012 38 / 48
Techniques for Algorithm Analysis
Two general strategies are as follows.
Use -bounds throughout the analysis and obtain a -bound for the
complexity of the algorithm.
Prove a O-bound and a matching -bound separately to get a
-bound. Sometimes this technique is easier because arguments for
O-bounds may use simpler upper bounds
(and arguments for -bounds may use simpler lower bounds)
than arguments for -bounds do.
Storjohann (CS, UW) CS240 - Module 1 Fall 2012 39 / 48
Techniques for Loop Analysis
Identify elementary operations that require constant time
(denoted (1) time).
The complexity of a loop is expressed as the sum of the complexities
of each iteration of the loop.
Analyze independent loops separately, and then add the results
(use maximum rules and simplify whenever possible).
If loops are nested, start with the innermost loop and proceed
outwards. In general, this kind of analysis requires evaluation of
nested summations.
Storjohann (CS, UW) CS240 - Module 1 Fall 2012 40 / 48
Example of Loop Analysis
Test1(n)
1. sum 0
2. for i 1 to n do
3. for j i to n do
4. sum sum + (i j )
2
5. sum sum
2
6. return sum
Storjohann (CS, UW) CS240 - Module 1 Fall 2012 41 / 48
Example of Loop Analysis
Test2(A, n)
1. max 0
2. for i 1 to n do
3. for j i to n do
4. sum 0
5. for k i to j do
6. sum A[k]
7. if sum > max then
8. max sum
9. return max
Storjohann (CS, UW) CS240 - Module 1 Fall 2012 42 / 48
Example of Loop Analysis
Test3(n)
1. sum 0
2. for i 1 to n do
3. j i
4. while j 1 do
5. sum sum + i /j
6. j j /2|
7. return sum
Storjohann (CS, UW) CS240 - Module 1 Fall 2012 43 / 48
Design of MergeSort
Input: Array A of n integers
Step 1: We split A into two subarrays: A
L
consists of the rst ,
n
2
|
elements in A and A
R
consists of the last
n
2
| elements in A.
Step 2: Recursively run MergeSort on A
L
and A
R
.
Step 3: After A
L
and A
R
have been sorted, use a function Merge to
merge them into a single sorted array. This can be done in time (n).
Storjohann (CS, UW) CS240 - Module 1 Fall 2012 44 / 48
MergeSort
MergeSort(A, n)
1. if n = 1 then
2. S A
3. else
4. n
L
,
n
2
|
5. n
R

n
2
|
6. A
L
[A[1], . . . , A[n
L
]]
7. A
R
[A[n
L
+ 1], . . . , A[n]]
8. S
L
MergeSort(A
L
, n
L
)
9. S
R
MergeSort(A
R
, n
R
)
10. S Merge(S
L
, n
L
, S
R
, n
R
)
11. return S
Storjohann (CS, UW) CS240 - Module 1 Fall 2012 45 / 48
Analysis of MergeSort
Let T(n) denote the time to run MergeSort on an array of length n.
Step 1 takes time (n)
Step 2 takes time T

,
n
2
|

+ T

n
2
|

Step 3 takes time (n)

The recurrence relation for T(n) is as follows:
T(n) =

,
n
2
|

+ T

n
2
|

+ (n) if n > 1
(1) if n = 1.
Storjohann (CS, UW) CS240 - Module 1 Fall 2012 46 / 48
Analysis of MergeSort
The mergesort recurrence is
T(n) =

,
n
2
|

+ T

n
2
|

+ (n) if n > 1
(1) if n = 1.
It is simpler to consider the following exact recurrence, with
unspecied constant factors c and d replacing s:
T(n) =

,
n
2
|

+ T

n
2
|

+ cn if n > 1
d if n = 1.
Storjohann (CS, UW) CS240 - Module 1 Fall 2012 47 / 48
Analysis of MergeSort
The following is the corresponding sloppy recurrence
(it has oors and ceilings removed):
T(n) =

2 T

n
2

+ cn if n > 1
d if n = 1.
The exact and sloppy recurrences are identical when n is a power of 2.
The recurrence can easily be solved by various methods when n = 2
j
.
The solution has growth rate T(n) (n log n).
It is possible to show that T(n) (n log n) for all n
by analyzing the exact recurrence.
Storjohann (CS, UW) CS240 - Module 1 Fall 2012 48 / 48