COMP3011: Design and Analysis of Algorithms This course:

COMP3011: Design and Analysis of Algorithms

Lecturer:
– Lecture 1: Introduction – Dr. Jesper Jansson
Department of Computing, PolyU
Office: PQ825

Teaching assistant:
Lecturer: Dr. Jesper Jansson Mr. Yushi Li
Office: PQ503
The Hong Kong Polytechnic University
2018-09-06

COMP3011 Lecture 1 1 / 38

Schedule Textbook

Lectures:
T. H. Cormen, C. E. Leiserson,
Thursdays, 10:30 a.m. – 12:20 p.m., PQ306
R. L. Rivest, and C. Stein:
Tutorials: Introduction to Algorithms (Third
Tuesdays, 11:30 a.m. – 12:20 p.m., R503 Edition),
Thursdays, 9:30 a.m. – 10:20 a.m., Y415 The MIT Press, Cambridge,
Thursdays, 4:30 p.m. – 5:20 p.m., P308 Massachusetts, 2009.

Three assignments: 30% of grade

Three quizzes: 30% of grade

Final exam: 40% of grade

COMP3011 Lecture 1 2 / 38 COMP3011 Lecture 1 3 / 38

Course contents

PART I
Introduction to algorithms

COMP3011 Lecture 1 4 / 38 COMP3011 Lecture 1 5 / 38

Algorithms
Example: (from elementary school)
This course:
COMP3011: Design and Analysis of Algorithms

Algorithm = A method for solving a specific, well-defined problem.

(Figure from http://www.mathsisfun.com/numbers/multiplication-long.html)

Note that:
It doesn’t matter which two numbers are given here...the same
method works for any input consisting of two positive integers.
The algorithm can be run by a human, an electronic computer, or ???
The running time, i.e., the number of operations in terms of the input
size, is important. (In fact, faster algorithms than the above exist for
multiplying large numbers! [“Toom-Cook multiplication”, etc.])
COMP3011 Lecture 1 6 / 38 COMP3011 Lecture 1 7 / 38
Algorithm descriptions Motivation
Algorithm = A method for solving a specific, well-defined problem.
People often use pseudocode to describe algorithms formally.
Question:
Sometimes people use text (and pictures) to describe an algorithm.
What are some important practical applications of algorithms?

⇒ In these lectures, we will use both types of descriptions.

COMP3011 Lecture 1 8 / 38 COMP3011 Lecture 1 9 / 38

Text searching Complexity

ggaagacagggatgggctaaagataaagggcctcctccaaaatccagatatgaggaagagctaaggacccgagggaag
tggcctcacgtcttcaacactaccattactgcagctaacttcatagacgtgatcatcacccggcaaattgcctcggac We can measure the efficiency of an algorithm in terms of the amount of
aggctccaagttcccttacaaccctaaggatgcaagggaacgtggctctcaaagttcagactcttctagtagcttgtc
ttcagtcacaggtatgaagcacctagtgatgctattgaggtgataagtcctgccagctcacctgcaccaaattacgcg computational resources it requires.
ccccccgtgaccatccggatgctggaaaaaccacagacctatcagccagagatggttaaggcaaatacagcagaaaat
gagtcccccctgtgccccgtcaacctcaacagtatgaaggaccactgactcattatcgtcccagcaggggatcaccgc Usually expressed as a function of the input size.
tctccacagcaacagtacccgatcgccagccaccactgcccccatcgtcccaggcagagggaatgggacaggtgccat
tctccagccagcaaggcccaggacccatcgactgatcacacttgctgaccacatctgtcaaattatcacacaagatat Also called the algorithm’s “complexity”.
ttaaatgctagatcatcacccaggcagctccgcataagttccctcgcagccttctacttctacattccaaacttcacc
gctttgtcgtccactctgggagcgggagccggacctgtaagaactaaaccgtcaagtcgctacagcccagagtctcag
tctcagacaacagagccgcaccccgctctgtcttgcatcccagaccaggtcctagagtctctccagaaaatcttgtgg Computational resources =
caacagagccgcaccccgctcagtataagctacttgccttacttcttcaccaagcttgaaagcacatcacccatggtt
gaaaggccttcttctcaaagaagcaggaaatttttcgtaagttgaactcctctggtggaggtgactctgatatgggtg Time (total number of computation steps)
tgtctcctctgtacattcagagggtgcagctgctcagccaggaacagagatcttcaacctgccagcagttactacatc
agtaagctcaagaagccattcttttgctgatcctgccagtaatcttggtctagaagacatccagattactttattcgc Space (amount of memory used)
gaccaccacagatcagaaaagctctcatgggaagtttcgatgacaaagttgaagatcatggtgttgtatttcaccatg
cggcgtctcatcctaccacaggcagatgtgggagttgtccctggtagtgctagcacctcagttgtgacgagtagtgag Hardware (# of logical gates, # of parallel processors, etc.)
tatacccaccaatcacttacacggagagatgaaggggacccatcacctcattcaggagtgtgcaaaccaaagctgatc
# of random bits used by a randomized algorithm
Input: A target text T of length n, a pattern text P of length m # of messages sent
Objective: Find all occurrences of P in T .
# of experiments
Naive algorithm: Running time proportional to m · n
More clever algorithms: Running time proportional to m + n
COMP3011 Lecture 1 11 / 38 COMP3011 Lecture 1 12 / 38
Machine model
Model of computation: Random-Access Machine (RAM) Many different types of algorithms:
Based on modern, existing computers.
Constructive vs. non-constructive.
Operations are executed sequentially. Discrete vs. numerical.
Available operations: Deterministic vs. non-deterministic.
Arithmetic, data movement, comparison, subroutine call.
Sequential vs. parallel vs. distributed.
Each operation takes a constant amount of time.
Exact vs. approximation.
Simplification ⇒ Makes it easier to compare algorithms without
worrying about the processor speed or the hardware configuration. etc.
Available data types: Integer, floating point.
Upper limit on each word of data:
Input of size n ⇒ Word size ≤ dc lg ne bits for some constant c ≥ 1

Remark:
If one assumes another model of computation such as a quantum computer
or a DNA computer that allows certain other operations in 1 unit of time
⇒ Some computational problems can be solved much faster!
COMP3011 Lecture 1 13 / 38 COMP3011 Lecture 1 14 / 38

Course goals
What are the properties of a “good” algorithm? 1 To provide students with in-depth knowledge of algorithm design
Gives the correct answer. techniques and tools for measuring the efficiency of algorithms.

Terminates. 2 To introduce advanced algorithms for fundamental computational

Efficient, i.e., uses few computational resources. problems.

(Easy for a human to understand and implement.)

For randomized algorithms, we want to get the correct answer with high
probability or that the running time is low with high probability.

For approximation algorithms, we want to get a solution that is within a

small factor of the optimal.

COMP3011 Lecture 1 15 / 38 COMP3011 Lecture 1 16 / 38

 The Sorting Problem

The Sorting Problem

Input: A sequence of n numbers ha1 , a2 , . . . , an i.
Output: A permutation (reordering) ha10 , a20 , . . . , an0 i of the input sequence
such that a10 ≤ a20 ≤ · · · ≤ an0 .

PART II
Example: (also called a “problem instance”)
Insertion-Sort Input h31, 41, 59, 26, 41, 58i
⇒ Output h26, 31, 41, 41, 58, 59i

Algorithm?

COMP3011 Lecture 1 17 / 38 COMP3011 Lecture 1 18 / 38

Insertion-Sort Insertion-Sort, pseudocode

Idea:

Start with an empty left hand and the cards face down on the table. Example:
Remove one card at a time from the table and insert it into the
correct position in the left hand.
To find the correct position for a card, compare it with each of the
cards already in the hand, from right to left.
Note that at all times, the cards held in the left hand are sorted and
these cards were originally the top cards of the pile on the table.
COMP3011 Lecture 1 19 / 38 COMP3011 Lecture 1 20 / 38
Insertion-Sort, cont. Insertion-Sort, correctness

Now, we want to do two things:

When using a loop invariant to prove an algorithm’s correctness, show that:
Prove the correctness of the algorithm.
1) Initialization: It is true before the first iteration of the loop.
Analyze the time complexity of the algorithm. 2) Maintenance: If it’s true before an iteration of the loop, it’s still true
before the next iteration.
3) Termination: When the loop terminates, the loop invariant yields a
particular property that helps show that the algorithm is correct.

Similar to using mathematical induction (i.e., prove a base case and an

inductive step).

COMP3011 Lecture 1 21 / 38 COMP3011 Lecture 1 22 / 38

Insertion-Sort, correctness Insertion-Sort, time complexity

Next, to analyze the algorithm’s time complexity, T (n), write:
To prove that the algorithm is correct, use the following “loop invariant”:
At the start of each iteration of the outer for-loop, A[1..j − 1]
consists of the elements originally in A[1..j − 1] but in sorted order.

1) Initialization:
j = 2 ⇒ A[1..j − 1] = A[1] is trivially sorted.
2) Maintenance:
key is inserted into the correct position and all larger elements are
shifted one step to the right. where ci = the (constant) time taken by line i and
tj = the number of times the while-loop’s test is executed for that j.
3) Termination: The outer for-loop ends when j = n + 1.
⇒ A[1..j − 1] = A[1..n] is sorted. T (n) = c1 n + c2 (n − 1) + c4 (n − 1) + c5 nj=2 tj + c6 nj=2 (tj − 1) +
P P
⇒P
c7 nj=2 (tj − 1) + c8 (n − 1)

COMP3011 Lecture 1 22 / 38 COMP3011 Lecture 1 23 / 38

Insertion-Sort, time complexity Insertion-Sort, time complexity
Next, to analyze the algorithm’s time complexity, T (n), write: Next, to analyze the algorithm’s time complexity, T (n), write:
. .
. .
. .
where ci = the (constant) time taken by line i and where ci = the (constant) time taken by line i and
tj = the number of times the while-loop’s test is executed for that j. tj = the number of times the while-loop’s test is executed for that j.

T (n) = c1 n + c2 (n − 1) + c4 (n − 1) + c5 nj=2 tj + c6 nj=2 (tj − 1) + T (n) = c1 n + c2 (n − 1) + c4 (n − 1) + c5 nj=2 tj + c6 nj=2 (tj − 1) +

P P P P
⇒P ⇒P
c7 nj=2 (tj − 1) + c8 (n − 1) c7 nj=2 (tj − 1) + c8 (n − 1)

Best case: The input is already sorted. Worst case: The input is in reverse sorted order.
⇒ A[i] ≤ key immediately in the while-loop’s test ⇒ tj = 1 for all j ⇒ Always has
 c A[i]c> key in the while-loop’s test ⇒ tj = j for all j 
⇒ T (n) = (c1 + c2 + c4 + c5 + c8 )n − (c2 + c4 + c5 + c8 ) 5 6 c7  2  c5 c6 c7
⇒ T (n) = + + n + c1 + c2 + c4 + − − + c8 n
2 2 2 2 2 2
⇒ T (n) = an + b for constants a, b −(c2 + c4 + c5 + c8 )
⇒ T (n) is a linear function of n. ⇒ T (n) = an2 + bn + c for constants a, b, c
⇒ T (n) is a quadratic function of n.
COMP3011 Lecture 1 23 / 38 COMP3011 Lecture 1 23 / 38

Insertion-Sort, time complexity Insertion-Sort, summary

Next, to analyze the algorithm’s time complexity, T (n), write: In summary, Algorithm Insertion-Sort is correct and its time
. complexity is:
.
. Best case:
where ci = the (constant) time taken by line i and T (n) = an + b for constants a, b ⇒ T (n) = Θ(n)
tj = the number of times the while-loop’s test is executed for that j. Worst case:
T (n) = an2 + bn + c for constants a, b, c ⇒ T (n) = Θ(n2 )
T (n) = c1 n + c2 (n − 1) + c4 (n − 1) + c5 nj=2 tj + c6 nj=2 (tj − 1) +
P P
⇒P
c7 nj=2 (tj − 1) + c8 (n − 1) Average case:
T (n) = an2 + bn + c for constants a, b, c ⇒ T (n) = Θ(n2 )
Average case: (Suppose the input consists of n positive integers in
{1, 2, . . . , n} chosen uniformly at random.) Remark 1: From here on, by an algorithm’s “time complexity” we usually
When the algorithm has to insert element A[j], on average half of the mean its worst-case time complexity. (It’s a guaranteed upper bound.)
elements in A[1..j − 1] are larger than A[j]. ⇒ tj ≈ j/2 for all j
⇒ T (n) = an2 + bn + c for constants a, b, c Remark 2: Θ-notation is used to express rate of growth.
(A simplification.)
⇒ T (n) is a quadratic function of n.
It will be formally defined in the next lecture.
COMP3011 Lecture 1 23 / 38 COMP3011 Lecture 1 24 / 38
 Merge-Sort, overview

The previous algorithm (Insertion-Sort) is incremental.

A more efficient approach, based on divide-and-conquer: Merge-Sort.

Subproblem: Sort the subarray A[p..r ], where initially p = 1 and r = n.

To sort A[p..r ]:
Divide by splitting A[p..r ] into A[p..q] and A[q + 1..r ], where q is the
PART III
halfway point of [p..r ].
Merge-Sort
Conquer by recursively sorting the subarrays A[p..q] and A[q + 1..r ].
Combine by merging the the sorted subarrays A[p..q] and A[q + 1..r ].
To do this, we will define a subroutine Merge(A, p, q, r ) later.

Base case of the recursion: the subarray has only 1 element.

COMP3011 Lecture 1 25 / 38 COMP3011 Lecture 1 27 / 38

Merge-Sort, pseudocode Procedure Merge

Input: Array A and indices p ≤ q < r .

Requires: Subarray A[p..q] is sorted and subarray A[q + 1..r ] is sorted.

Result: The subarrays are merged into a single sorted array in A[p..r ].

Idea: Think of two piles of cards.

Initial call: Merge-Sort(A, 1, n)
Each pile is sorted and placed face-up (smallest card on top).
Example:
Basic step: Choose the smaller of the two visible cards, remove from
A = h5, 2, 4, 7, 1, 3, 2, 6i
its pile, and put it face-down on the output pile.
Repeat the basic step until one input pile is empty.
Put the cards in the other input pile face-down on the output pile.
If we start with n cards, this takes ≤ n basic steps. ⇒ Θ(n) time
To simplify the code:
Put a “sentinel” card (value: +∞) at the bottom of each input pile.
COMP3011 Lecture 1 28 / 38 COMP3011 Lecture 1 29 / 38
Procedure Merge, pseudocode Procedure Merge, example

COMP3011 Lecture 1 30 / 38 COMP3011 Lecture 1 31 / 38

Procedure Merge, example Procedure Merge, analysis

To prove the correctness of Merge, use a loop invariant.

Try it yourself (or see the textbook for the details).

For the worst-case time complexity analysis, observe that:

The first two for-loops take Θ(n1 + n2 ) = Θ(n) time, where n denotes
the size of the subarray being merged.
The last for-loop makes n iterations, each taking constant time, i.e.,
Θ(n) time in total.

Total time: Θ(n) (a linear function of n).

COMP3011 Lecture 1 31 / 38 COMP3011 Lecture 1 32 / 38

Merge-Sort, time complexity Solving recurrences
The next lecture will introduce general methods for solving recurrences.
Today: Intuitive solution by looking at the recursion tree.

Rewrite the recurrence as:


c, if n = 1,
T (n) = where c is a constant
2T (n/2) + cn, if n > 1.
Write:
Expand the recurrence into a “recursion tree”:
T (n) = the worst-case running time of Merge-Sort on an array of size n.

Then:

Θ(1), if n = 1,
T (n) = “recurrence”
2T (n/2) + Θ(n), if n > 1.

(The notation Θ(1) means “constant time”.)

How do we solve this recurrence?

COMP3011 Lecture 1 33 / 38 COMP3011 Lecture 1 34 / 38

Merge-Sort, summary
Continue expanding until the problem sizes reach 1:
In summary, Algorithm Merge-Sort is correct and its worst-case running
time is: Θ(n lg n)

Recall that Insertion-Sort had worst-case running time: Θ(n2 )

⇒ Merge-Sort is faster than Insertion-Sort for all large enough n.

[Each level has cost cn, Does Insertion-Sort have any advantages over Merge-Sort?
there are lg n + 1 levels]

Total cost: cn lg n + cn = Θ(n lg n)

COMP3011 Lecture 1 35 / 38 COMP3011 Lecture 1 36 / 38
More sorting algorithms will be introduced later in the course. Why is sorting important?

Applications need to sort information.

Example: Display your e-mails in chronological order.
Example: Display your music according to artists’ names in
alphabetical order.
Example: To prepare customer statements, banks need to sort
checks by check number.
Example: The books in a library are ordered and organized in a
certain way.
Example: In computer graphics, determine which objects are closer
to the viewer than others.
Example: A database can use an index to find data quickly.
(In the old days: Phonebooks.)
etc.

COMP3011 Lecture 1 37 / 38 COMP3011 Lecture 1 38 / 38

Why is sorting important?

Applications need to sort information.

Other problems use sorting as a subroutine.

Example: In computational geometry, many problems can be solved
efficiently after preprocessing the input points, e.g., by sorting them
according to their x-coordinates.

Example: In data compression, the Burrows-Wheeler transform

(“compression booster”) sorts all rotations of the input text
lexicographically and concatenates the last characters.

Example: Given an array A of n integers, determine if A contains any

duplicates.
Sort A, then traverse it to check if any consecutive elements are equal.
⇒ Θ(n lg n) time!
COMP3011 Lecture 1 38 / 38