What is algorithm?

Chapter 2: Complexity of Algorithms and

Lower Bounds of Problems ▶ Simply speaking, an algorithm is a computational method that
can be used to solve a problem.
Chin Lung Lu ▶ More importantly, we can implement a program based on an
algorithm such that the program can automatically solve the
Department of Computer Science problem.
National Tsing Hua University

1 2

What is a good algorithm? Time complexity of algorithms

How to measure the running time of an algorithm?

▶ An algorithm is good if it takes a short time and requires a small Method 1:

amount of memory space. Write a program for the algorithm and see how fast it runs.
▶ Traditionally, the needed time is a more important factor to ▶ However, this method is not appropriate, because there are so
determine the goodness of an algorithm. many factors unrelated to the algorithm which can affect the
performance of the program.

3 4
Time complexity of algorithms (cont’d) Time complexity of algorithms (cont’d)
How to measure the running time of an algorithm?

Method 2: ▶ Usually, the time of executing an algorithm is dependent on the

Perform a mathematical analysis to determine the number of all the size of the problem, denoted by n.
steps needed to complete the algorithm.
Example:
▶ In fact, we can choose some particular steps that are the most
The number of data in the sorting problem is the problem size.
time-consuming operations in the algorithm.
▶ Most algorithms need more time to complete when n increases.
Example:
Comparison (or movement) of data in sorting algorithms.

5 6

Time complexity of algorithms (cont’d) Big-Oh notation

Definition:
f (n) = O(g(n)) if and only if there exist two positive constants c
and n0 such that f (n) ≤ cg(n) for all n ≥ n0 .
Example: cg(n)
Suppose that it takes n3 + n steps to run an algorithm. We would
often say that the time complexity of this algorithm is in the order of
n3 .
▶ The reason is that the term n3 dominates n. f (n)
▶ It means that as n becomes very large, n is not so significant
when compared with n3 .
f (n) = O( g(n))
n
n0

f (n) is bounded by cg(n) as n is large enough.

7 8
Big-Oh notation (cont’d) Big-Oh notation (cont’d)
Example 1:
▶ Suppose that it takes n3 + n steps to run an algorithm A.
▶ Then the time complexity of A is O(n3 )
Example 2: Let f (n) = 2n2 − 3n.
▶ Let f (n) = n3 + n.
1. f (n) = O(n2 ) ✓
2. f (n) = O(n3 ) ✓
f (n) = n3 + n
3. f (n) = O(n) ✗
= (1 + n12 )n3 4. f (n) = O(1) ✗
≤ 2n3 for n ≥ 1

▶ We have f (n) ≤ cg(n) for all n ≥ n0 by letting g(n) = n3 ,

c = 2 and n0 = 1.
▶ Hence, the time complexity of A is O(n3 ).
9 10

Big-Oh notation (cont’d) A misunderstanding about big-Oh

Example:
▶ Let A1 and A2 be two algorithms of solving the same problem
Example 3: Let f (n) = 2100 n2 − 3n. and their time complexities be O(n3 ) and O(n), respectively.
1. f (n) = O(n2 ) ✓ ▶ We ask the same person to write two programs, say P1 and P2
2. f (n) = O(n3 ) ✓ respectively, for A1 and A2 and run these two programs under
3. f (n) = O(n) ✗ the same programming environment.
4. f (n) = O(1) ✗
Question:
Would program P2 always run faster than program P1 ?
▶ The answer is not necessarily true.

11 12
A misunderstanding about big-Oh (cont’d) A misunderstanding about big-Oh (cont’d)

▶ It is a common mistake to think that P2 will always run faster

than P1 .
Example:
▶ Let f1 and f2 be the time complexities of algorithms A1 and A2 , ▶ Actually, the constant hidden in O -notation can not be ignored.
respectively. ▶ However, no matter how large this constant, its significance de-
▶ Suppose that f1 = n3 and f2 = 100n. creases as n increases.

1. f1 > f2 for n > 10.

(It means that P2 runs faster than P1 when n is large.)
2. f1 < f2 for n < 10.
(It means that P1 may run faster than P2 when n is small.)

13 14

Significance of order Significance of order (cont’d)

n \ f (n) log2 n n n log2 n n2 2n n!

10 0.003 µs 0.01 µs 0.033 µs 0.1 µs 1 µs 3.63 ms
20 0.004 µs 0.02 µs 0.086 µs 0.4 µs 1 ms 77.1 years
30 0.005 µs 0.03 µs 0.147 µs 0.9 µs 1 sec 8.4 × 1015 yrs
40 0.005 µs 0.04 µs 0.213 µs 1.6 µs 18.3 min ▶ It is very meaningful if we can find an algorithm with lower order
50 0.006 µs 0.05 µs 0.282 µs 2.5 µs 13 days time complexity.
102 0.006 µs 0.1 µs 0.644 µs 10 µs 4 × 1013 yrs ▶ While we may dislike the time-complexity functions, such as
103 0.010 µs 1 µs 9.644 µs 1 ms
n2 , n3 , etc., they are still tolerable when compared with 2n .
104 0.013 µs 10 µs 130 µs 100 ms
105 0.016 µs 0.1 ms 1.67 ms 10 sec
106 0.020 µs 1 ms 19.93 ms 16.7 min
107 0.023 µs 0.01 sec 0.23 sec 1.16 days
108 0.026 µs 0.1 sec 2.66 sec 115.7 days (10−9 sec/op)
109 0.030 µs 1 sec 20.90 sec 31.7 years

15 16
Polynomial and exponential algorithms Three cases of algorithm analyses

▶ For any algorithm, we are interested in its behavior under three

Definition: cases: best case, average case and worst case.
A polynomial-time algorithm is any algorithm with time complexity ▶ Let T (I ) be the running time of an algorithm A for instance I.
O(f (n)), where f (n) is a polynomial function of input size n. Definition (time complexity of A in the worst case):
▶ Examples: O(1), O(log n), O(n) and O(n2000 ) max{T (I ) : all instances I }.
Definition: Definition (time complexity of A in the best case):
An exponential-time algorithm is any algorithm whose time complex- min{T (I ) : all instances I }.
ity can not be bounded by a polynomial function.
▶ Examples: O(2n ) and O(n!) Definition (time complexity of A in the average case):
sum{T (I ) × p(I ) : all instances I }, where p(I ) is the probability of
the occurrence of I.

17 18

Insertion sort algorithm Insertion sort algorithm (cont’d)

Input: A sequence of n numbers x1 , x2 , . . . , xn .

Output: The sorted sequence of x1 , x2 , . . . , xn .
1. for j = 2 to n do /* Outer loop */
2. x = xj
3. i = j−1
4. while x < xi and i > 0 do /* Inner loop */
5. xi+1 = xi
6. i = i−1
7. end while
8. xi + 1 = x
9. end for

19 20
Insertion sort algorithm (cont’d) Insertion sort algorithm (cont’d)
sorted unsorted Example:
z }| { z }| {
Let the input sequence be 7, 5, 1, 4, 3, 2, 6.
x1 ... x j −1 x j x j +1 x j +2 ... xn
Sorted sequence Unsorted sequence
x xj Input data 7,5,1,4,3,2,6
xi Initial state 7 5,1,4,3,2,6
j=2 5, 7 1,4,3,2,6
x1 ... x j −1 xi +1
j=3 1, 5, 7 4,3,2,6
if x < xi j=4 1, 4, 5, 7 3,2,6
xi xi +1 j=5 1, 3, 4, 5, 7 2,6
x1 . . . x j −2 x j −1 j=6 1, 2, 3, 4, 5, 7 6
j=7 1, 2, 3, 4, 5, 6, 7

21 22

Time complexity of insertion sort algorithm Best case of insertion sort algorithm
▶ In our analysis below, we use the number of data movements X
as the time complexity measurement.
▶ Outer loop: x = xj and xi+1 = x (always executed) ▶ The best case of the insertion sort occurs when the input data
▶ Inner loop: xi+1 = xi (not always executed) are already sorted.
▶ Let dj be the number of the data movements for xi in the inner Example:
while loop, that is, dj = {xi : xi > xj , 1 ≤ i < j} . The input data is 1, 2, 3, 4, 5, 6, 7.
Lemma:
n ▶ In this case, we have d2 = d3 = . . . = dn = 0.
X= ∑ ( 2 + dj ) . ▶ Therefore, X = 2(n − 1) = O(n).
j=2
n
▶ That is, X = 2(n − 1) + ∑ dj .
j=2

23 24
Worst case of insertion sort algorithm Average case of insertion sort algorithm

▶ The worst case of the insertion sort occurs when the input data ▶ Assume x1 , x2 , . . . , xj−1 is already a sorted sequence and the
are reversely sorted. next data to be inserted is xj .
▶ Suppose xj is the kth largest number among the j numbers.
Example:
▶ There are k − 1 movements in the inner loop, where 1 ≤ k ≤ j.
The input data is 7, 6, 5, 4, 3, 2, 1.
▶ Moreover, there are 2 movements in the outer loop.
▶ In this case, we have d2 = 1, d3 = 2, . . . , dn = n − 1.
n ▶ The probability that xj is the kth largest among j numbers is 1j .
n(n − 1)
▶ Therefore, ∑ dj = and as a result, we have: ▶ The average number of movement when considering xj is:
j=2
2
(j+3)j
n(n − 1) (n − 1)(n + 4) 2+0 2+1 2+j−1 2 j+3
X = 2(n − 1) + = = O(n2 ) + +...+ = =
2 2 j j j j 2

25 26

Average case of insertion sort algorithm (cont’d) Time complexities of insertion sort algorithm

▶ As a result, the average time complexity of the insertion sort is: Theorem:
! In summary, the time complexities of insertion sort are as follows:
n
j+3 1 n n
(n + 8)(n − 1) ▶ Best case: O(n)
∑ 2 = 2 ∑j+ ∑3 = = O(n2 )
j=2 j=2 j=2
4 ▶ Average case: O(n2 )
▶ Worst case: O(n2 )

27 28
Selection sort algorithm Example of selection sort algorithm
▶ Let the input sequence be 7, 5, 1, 4, 3, 2, 6.
Input: A sequence of n numbers a1 , a2 , . . . , an . Step 1: Find the 1th smallest number
Output: The sorted sequence of a1 , a2 , . . . , an .
a1 a2 a3 a4 a5 a6 a7
1. for j = 1 to n − 1 do /* find the jth smallest number */ j=1 7 5 1 4 3 2 6
2. f = j /* f is a flag */ f ←k ↑f ↑k
3. for k = j + 1 to n do f ←k ↑f ↑k
4. if ak < af , then f = k ↑f ↑k
↑f ↑k
5. end for
↑f ↑k
6. aj ↔ af /* exchange aj with af */ ↑f ↑k
7. end for aj ↔ af ≡ a1 ↔ a3 1 5 7 4 3 2 6

29 30

Example of selection sort algorithm (cont’d) Operations of selection sort algorithm

Step 1: Find the 2nd smallest number

Note that there are two operations in the inner for loop:
a1 a2 a3 a4 a5 a6 a7 1. Comparisons of two elements: “if ak < af ”.
j=2 1 5 7 4 3 2 6
↑f ↑k 2. Change of the flag: “f = k”.
f ←k ↑f ↑k ▶ The number of comparisons of two elements is n(n2−1) , which is
↑f ↑k
a fixed number.
f ←k ↑f ↑k
▶ That is, no matter what the input data are, we always have to
↑f ↑k n(n−1)
aj ↔ af ≡ a2 ↔ a6 1 2 7 4 3 5 6 perform 2 comparisons.

31 32
Time complexities of selection sort Quick sort

▶ The basic idea of quick sort (divide and conquer) is as follows:

Theorem:
X
The time complexities of the selection sort (when measured by the
number of comparisons) are as follows: L1 ≤X Quick sort
▶ Best case: O(n2 )
▶ Average case: O(n2 ) Quick sort X
▶ Worst case: O(n2 )
L2 ≥X Quick sort

33 34

Algorithm of quick sort Example of quick sort

Algorithm: Quicksort(f , l) Iteration 1: Let
Input: A sequence of (l − f + 1) numbers af , af +1 , . . . , al . a1 = 3, a2 = 6, a3 = 1, a4 = 4, a5 = 5, a6 = 2.
Output: The sorted sequence of af , af +1 , . . . , al .
X=3 a1 a2 a3 a4 a5 a6
1. if f ≥ l, then return
i = 1, j = 6 3 6 1 4 5 2
2. X = af , i = f , j = l
(aj = a6 < X) ↑i ↑j
3. while i < j do
a1 ↔ a6 2 6 1 4 5 3
4. while aj ≥ X and i < j do
(ai = a1 < X) ↑i ↑j
5. j = j−1
i = i+1 = 2 2 6 1 4 5 3
6. ai ↔ aj
(ai = a2 > X) ↑i ↑j
7. while ai < X and i < j do
a2 ↔ a6 2 3 1 4 5 6
8. i = i+1
(aj = a6 > X) ↑i ↑j
9. ai ↔ aj
j = j−1 = 5 2 3 1 4 5 6
10. end while
(aj = a5 > X) ↑i ↑j
11. Quicksort(f , j − 1), Quicksort(j + 1, l)

35 36
Example of quick sort (cont’d) Best case of quick sort
▶ The best case occurs when X splits the list right in the middle
Iteration 1 (cont’d): for each round.
X=3 a1 a2 a3 a4 a5 a6 ▶ That is, X produces two sublists that contain the same number
j = j−1 = 4 2 3 1 4 5 6 of elements.
(aj = a4 > X) ↑i ↑j ▶ Each round needs O(n) steps to split the lists.
j = j−1 = 3 2 3 1 4 5 6 ▶ For example, the first round needs cn steps to split the list, where
(aj = a3 < X) ↑i ↑j c is a constant.
a2 ↔ a3 2 1 3 4 5 6 ▶ Moreover, the second round needs 2 · cn 2 = cn steps to split its
(ai = a2 < X) ↑i ↑j lists.
i = i+1 = 3 2 1 3 4 5 6 ▶ Assume n = 2p .
(i = j = 3) i↑j ▶ We then need totally p rounds, where p = log2 n.
(end of iteration 1) ≤3 ≤3 =3 ≥3 ≥3 ≥3
▶ Hence, the total time complexity of the best case is cn log2 n =
O(n log2 n).

37 38

Worst case of quick sort Average case of quick sort

▶ Let T (n) denote the number of steps in the average case for n
elements.
▶ Assume after splitting, the first sub-list contains s − 1 elements
▶ The worst case occurs when the input data are sorted or reversely and the second contains n − s elements, where 1 ≤ s ≤ n.
sorted.
▶ By considering all possible cases, we have:
▶ In this case, we need totally n rounds.
▶ Hence, the time complexity of the worst case is: T (n) = Ave (T (s − 1) + T (n − s)) + O(n)
1≤s≤n
c(n + 1)n
cn + c(n − 1) + . . . + c = = O(n2 ) where O(n) is the number of operations needed for the first
2
splitting operation.
▶ For simplifying computation, we let:

T (n) = Ave (T (s − 1) + T (n − s)) + c(n + 1)

1≤s≤n

39 40
Average case of quick sort (cont’d) Average case of quick sort (cont’d)
▶ We can express Ave1≤s≤n (T (s − 1) + T (n − s)) as follows: ▶ Therefore, we have:
Ave (T (s − 1) + T (n − s))
1≤s≤n nT (n) − (n − 1)T (n − 1) = 2T (n − 1) + 2cn
= n1 (T (0) + T (n − 1) + . . . + T (n − 1) + T (0)) nT (n) = (n + 1)T (n − 1) + 2cn
= n1 (2T (0) + 2T (1) + . . . + 2T (n − 1)) T (n)
= T (n−1)
+ n2c
n+1 n +1
▶ Since T (0) = 0, we have:
▶ Recursively, we have:
T (n) = Ave (T (s − 1) + T (n − s)) + c(n + 1)
T (n−1) T (n−2)
1≤s≤n
n = n−1 + 2c
n
1
= n (2T (1) + 2T (2) + . . . + 2T (n − 1)) + c(n + 1) T (n−2)
= T (n−3)
+ n2c
n−1 n−2 −1
⇔ nT (n) = (2T (1) + 2T (2) + . . . + 2T (n − 1)) + cn(n + 1) ..
.
▶ By substituting n = n − 1 into the above formula, we have: T (1) T (0)
2 = 1 + 2c
2
(n − 1)T (n − 1) = 2T (1) + . . . + 2T (n − 2) + c(n − 1)n
41 42

Average case of quick sort (cont’d) Time complexities of quick sort algorithm
▶ Therefore, we have:
 
T (n) 1
n+1 = 2c n+1 + n1 + . . . + 12
= 2c(Hn+1 − 1) Theorem:
In summary, the time complexities of quick sort are as follows:
▶ Note that Hn = 1
n + 1
n−1 + ... + 1
1 and Hn ∼
= loge n when ▶ Best case: O(n log2 n)
n → ∞. ▶ Average case: O(n log2 n)
▶ Finally, we have: ▶ Worst case: O(n2 )

T (n) = 2c(n + 1)(Hn+1 − 1)

∼
= 2c(n + 1) loge (n + 1) − 2c(n + 1)
= O(n log2 n)

43 44
Lower bound of problem Big-Omega notation
Definition:
f (n) = Ω(g(n)) if and only if there exist two positive constants c
and n0 such that f (n) ≥ cg(n) for all n ≥ n0 .
Definition:
A lower bound of a problem is the least time complexity required for f (n)
any algorithm that can be used to solve this problem.
▶ The time complexity used in the above definition usually refers
to the worst-case time complexity.
▶ Hence, this lower bound is called worst-case lower bound. cg(n)
▶ To describe the lower bound, we shall use a notation Ω.

f (n) = Ω( g(n))
n
n0

45 46

Big-Omega notation (cont’d) Determination of problem lower bounds

Question:
Example: How to determine the lower bounds of a problem?
Let f (n) = 2n2 + 3n.
1. f (n) = Ω(n2 ) ✓ Exhaustive method:
2. f (n) = Ω(n3 ) ✗ 1. Enumerate all possible algorithms.
3. f (n) = Ω(n) ✓ 2. Determine the time complexity of each algorithm.
4. f (n) = Ω(1) ✓ 3. Find the minimum time complexity.

▶ It is impossible to enumerate all possible algorithms.

47 48
Determination of problem lower bounds (cont’d) Upper limit of lower bound

Example: What are the lower bounds of sorting?

1. Ω(1): At least one step to complete any sorting algorithm. ▶ As the lower bound of a problem goes higher and higher, we will
2. Ω(n): Every data element must be examined before it’s sorted. inevitably wonder whether there is an upper limit of the lower
bound?
3. Ω(n log n): This requires a theoretical proof.
Question:
▶ The lower bound of a problem is not unique. Is there any possibility that Ω(n2 ) is a lower bound of the sorting
▶ Ω(n log n) is more significant than Ω(1) and Ω(n). problem?
▶ We like the lower bound to be as high as possible. ▶ Answer: no!
▶ Each higher lower bound is found by theoretical analysis, not by
pure guessing.

49 50

Upper limit of lower bound (cont’d) Lower bound and its upper limit

Case 1:
The highest lower bound of a problem is Ω(n log n) and the time
complexity of the best available algorithm to solve this problem is
▶ The time complexity of the best one among currently available O(n2 ).
algorithms for a problem can be considered as the upper limit of Upper limit of lower bound
the lower bound.
▶ Now, let us consider the following two cases:
Gap

Lower bound

51 52
Lower bound and its upper limit (cont’d) Lower bound and its upper limit (cont’d)
Case 2:
The present lower bound is Ω(n log n) and there is indeed an algo-
rithm with time complexity O(n log n).
In this case, there are three possibilities:
1. The lower bound of the problem is too low.
⇒ We should find a higher lower bound. Upper limit of lower bound
2. The best available algorithm is not good enough. No gap
Lower bound
⇒ We should find a better algorithm.
3. Both the lower bound and the algorithm may be improved.
⇒ We should try to improve both. ▶ In this case, the lower bound and the algorithm cannot be im-
proved any further.
▶ It means that we have found an optimal algorithm to solve the
problem and a truly significant lower bound of this problem.

53 54

Optimal algorithm Big-Theta notation

Definition:
f (n) = Θ(g(n)) if and only if there exist positive constants c1 , c2
and n0 such that c1 g(n) ≤ f (n) ≤ c2 g(n) for all n ≥ n0 }.

Definition: c2 g (n)
An algorithm is optimal if its time complexity is equivalent to a lower
bound of the problem.
f (n)
▶ It means that neither the lower bound nor the algorithm can be
improved further.
c1 g (n)

f (n) = Θ( g(n))
n
n0

55 56
Big-Theta notation (cont’d) Binary decision tree
▶ For many (comparison-based) algorithms, their executions can
be described as binary decision trees.
Example:
Example: Consider the case of insertion sort with the input of 3 different ele-
Let f (n) = 21 n2 − 3n. ments (a1 , a2 , a3 ).
1. f (n) = Θ(n2 ) ✓ ▶ Then there are 6 distinct permutations (instances).
2. f (n) = Θ(n3 ) ✗ a1 a2 a3
3. f (n) = Θ (n) ✗
1 2 3
4. f (n) = Θ (1) ✗ 1 3 2
2 1 3
2 3 1
3 1 2
3 2 1

57 58

Binary decision tree (cont’d) Binary decision tree (cont’d)

a1 > a2 (3, 2, 1)

Algorithm: Insertion sort (revised) a1 : a2

a2 > a3
Input: A sequence of n numbers a1 , a2 , . . . , an .
a1 < a2 (3, 1, 2)
Output: The sorted sequence of a1 , a2 , . . . , an . a2 : a3

1. for j = 2 to n do a1 > a2 a2 < a3 (2, 1, 3)

2. i=j (2, 3, 1)
a1 > a2
3. while ai−1 > ai and i > 1 do a1 : a2
4. ai−1 ↔ ai /* Exchange ai−1 with ai */ a1 : a2
a2 > a3
5. i = i−1 a1 < a2 a1 < a2 (1, 3, 2)
6. end while a2 : a3
7. end for
a2 < a3
(1, 2, 3)

59 60
Binary decision tree (cont’d) Binary decision tree (cont’d)
▶ When we apply the insertion sort to the above set of data, each a1 > a2 (3, 2, 1)
permutation has a distinct response.
a1 : a2
Example 1: a2 > a3
a1 < a2 (3, 1, 2)
Suppose that the input is (a1 , a2 , a3 ) = (2, 3, 1). Then the insertion
a2 : a3
sort behaves as follows.
a1 > a2 a2 < a3 (2, 1, 3)
1. Compare a1 = 2 with a2 = 3.
Since a2 > a1 , no exchange of data elements takes place. a1 > a2 (2, 3, 1) Example 1
a1 : a2
2. Compare a3 = 1 with a2 = 3. a1 : a2
a2 > a3
Since a3 < a2 , we exchange a3 and a2 .
As a result, (a1 , a2 , a3 ) = (2, 1, 3). a1 < a2 a1 < a2 (1, 3, 2)
a2 : a3
3. Compare a2 = 1 with a1 = 2.
Since a2 < a1 , we exchange a2 and a1 . a2 < a3
As a result, (a1 , a2 , a3 ) = (1, 2, 3). (1, 2, 3)

61 62

Binary decision tree (cont’d) Binary decision tree (cont’d)

a1 > a2 (3, 2, 1)

a1 : a2
Example 2: a2 > a3
Suppose that the input is (a1 , a2 , a3 ) = (2, 1, 3). Then the insertion a1 < a2 (3, 1, 2)
sort behaves as follows. a2 : a3

1. Compare a1 = 2 with a2 = 1. a1 > a2 a2 < a3 (2, 1, 3) Example 2

Since a2 < a1 , we exchange a2 and a1 . (2, 3, 1)
a1 > a2
As a result, (a1 , a2 , a3 ) = (1, 2, 3). a1 : a2
2. Compare a3 = 3 with a2 = 2. a1 : a2
a2 > a3
Since a3 > a2 , no exchange of data takes place. a1 < a2 a1 < a2 (1, 3, 2)
As a result, (a1 , a2 , a3 ) = (1, 2, 3). a2 : a3

a2 < a3
(1, 2, 3)

63 64
Lower bound of sorting problem Lower bound of sorting problem (cont’d)
▶ For every sorting algorithm, its corresponding binary decision
tree will have n! leaf nodes, as there are n! permutations.
▶ In general, any sorting algorithm whose basic operation is a
▶ The depth of a balanced binary tree is the smallest.
compare-and-exchange operation can be described by a binary
▶ The depth of the balanced binary tree is ⌈log2 n!⌉.
decision tree.
▶ The action of a sorting algorithm on a particular input data ▶ The minimum number of comparisons to sort in the worst case
is at least ⌈log2 n!⌉.
corresponds to one path from the root to a leaf, where each leaf
node corresponds to a particular permutation. ▶ Hence, the worst-case lower bound of sorting is Ω(n log2 n).
√
n
▶ The length of the longest path from the root to a leaf (called Stirling approximation formula: n! ∼= 2πn n . e
tree depth) is the worst-case time complexity of this algorithm. √
▶ The lower bound of the sorting problem is the smallest depth log2 n! = log2 2π + 12 log2 n + n log2 n − n log2 e
of some tree among all possible binary decision trees modeling ≥ n log2 n − 1.44n
sorting algorithms.  
1.44
= n log2 n 1 − log n
2
≥ 0.28n log2 n for n ≥ 4
65 66

Selection sort (revisited) Knockout sort

▶ Recall that for selection sort, we need n − 1 steps to obtain the

first smallest number, then n − 2 steps to obtain the second
smallest number, and so on (all in worst case). Algorithm: Knockout sort
▶ Hence, O(n2 ) steps are needed for selection sort.
1. Construct a knockout tree.
▶ Since the lower bound of sorting is Ω(n log n), selection sort is 2. Output the smallest number, replace it by ∞, and update the
not optimal. knockout tree.
Observation: 3. Repeat the above step untill all numbers are sorted.
When we try to find the second smallest number, the information we
have extracted by finding the first smallest number is not used at all.

67 68
Knockout sort (cont’d) Knockout sort (cont’d)
3 4
Example:
3 5 4 5
Let the input date be 2, 4, 7, 3, 1, 6, 5, 8.

2 4 3 6 5 4 7 6 5
1

2 1 2 5 ∞ 4 7 3 ∞ 6 5 8 ∞ 4 7 ∞ ∞ 6 5 8

5 6
2 3 1 5 2 3 6 5

7 5 7 6
2 4 7 3 1 6 5 8 2 4 7 3 ∞ 6 5 8
∞ 7 6 5 ∞ 7 6 8
Knockout tree Update of knockout tree
∞ ∞ 7 ∞ ∞ 6 5 8 ∞ ∞ 7 ∞ ∞ 6 ∞ 8

69 70

Knockout sort (cont’d) Knockout sort (cont’d)

▶ Actually, knockout sort is similar to the selection sort.
▶ However, after finding the 1st smallest number, only a small part
of the knockout tree needs to be examined for finding the 2nd
7 8 smallest number.

7 8 ∞ 8
2

∞ 7 ∞ 8 ∞ ∞ ∞ 8
2 5

∞ ∞ 7 ∞ ∞ ∞ ∞ 8 ∞ ∞ ∞ ∞ ∞ ∞ ∞ 8
6 5

∞ 6

71 72
Knockout sort (cont’d) Heap
▶ The first smallest number is found after n − 1 comparisons.
▶ For all of the other selections, only ⌈log2 n⌉ − 1 comparisons Definition:
are needed. A heap is a binary tree satisfying the following conditions:
▶ Therefore, the total number of comparisons is: 1. This tree is a complete binary tree.
(n − 1) + (n − 1)(⌈log2 n⌉ − 1) = O(n log n) 2. Son’s value ≤ parent’s value.
▶ The time complexity of the knockout sort is O(n log n). Properties of complete binary tree:
▶ This complexity is valid for best, average and worst cases.
▶ A complete binary tree is a binary tree in which except possibly
▶ The knockout sort is an optimal sorting algorithm. the last level, every level is completely filled, and all nodes are
▶ The reason that the knockout sort is better than the selection as far left as possible.
sort is that it uses previous information.
▶ However, the knockout sort needs 2n − 1 space (i.e., tree size).

73 74

Heap (cont’d) Heap sort

Example 1: Example 2:

48 48 Algorithm: Heap sort

1. Construct the heap.
37 26 37 26 2. Output the largest number, replace it with the last number and
restore the tree as a heap.
3. Repeat the above step until all the numbers are sorted.
15 4 5 9 15 4

75 76
Heap sort (cont’d) Heap sort (cont’d)

Example:
Consider five numbers 15, 37, 4, 48 and 26 and assume that their Step 1: Output 48 and restore the heap (by replacing 48 with 4).
heap is already constructed.
PSfrag 48 4 37
48
37 26 15 26
37 26

37 26 15 4
15 4

15 4

77 78

Heap sort (cont’d) Heap sort (cont’d)

Step 2: Output 37 and restore the heap (by replacing 37 with 4).

37 26
Step 4: Output 15 and restore the heap.
4

15 4
15 26 15 26 15 4

4 4

Step 3: Output 26 and restore the heap (by replacing 26 with 4). Step 5: Output 4.

26 4 15
Output of heap sort:
The output sequence is 48, 37, 26, 15, 4, which is sorted.
15 4 15 4

79 80
Heap sort (cont’d) Heap sort (cont’d)
▶ We use an array (instead of pointers) to represent a heap.

A (1 )

51
▶ Then we can uniquely determine each node and its descendants
A (2 ) A (3 )
using the following rule:
48 37
A (4 ) A (5 )
The rule to determine the descendants of a node:
The descendants of A(h) are A(2h) and A(2h + 1), if they exist.
17 28 30 25 ▶ Using an array to represent a heap, the entire process of heap
A (6 ) A (7 ) sort can be operated on an array.
6 5 21

A (8 ) A (9 ) A(10)

A(1) A(2) A(3) A(4) A(5) A(6) A(7) A(8) A(9) A(10)
51 48 37 17 28 30 25 6 5 21
81 82

Restore routine of heap sort Restore routine of heap sort (cont’d)

Interchange with larger son if smaller

Algorithm: Restore(i, j)
Input: A(i), A(i + 1), · · · , A(j).
Output: A(i), A(i + 1), · · · , A(j) as a heap.
Assumption: The subtrees rooted at sons of A(i) are heaps.
1. if A(i) is not a leaf and a son of A(i) contains a larger element
than A(i) then
2. Let A(h) be the son of A(i) with the largest element.
Heap Heap 3. Interchange A(i) and A(h)
4. Restore(h, j)
Apply restore recursively 5. end if

83 84
Restore routine of heap sort (cont’d) Restore routine of heap sort (cont’d)

4 48 48
▶ In the Restore(i, j) routine, we use the parameter j to determine
48 26 4 26 37 26 whether A(i) is a leaf or not.
Note:
15 37 15 37 15 4 j j
If i > ⌊ 2 ⌋ (or i > 2 ), then A(i) is a leaf.

Restore(1, 5) Restore(2, 5) Heap

85 86

Restore routine of heap sort (cont’d) Construction of a heap

Example:
j
Let j = 5. Then ⌊ 2 ⌋ = 2 and hence A(3), A(4), A(5) are leaves. ▶ Let A(1), A(2), . . . , A(n) be any complete binary tree.
▶ A(i), where i = 1, 2, . . . , ⌊ n2 ⌋, must be an internal node with
A (1 ) descendants.
▶ A(i), where i = ⌊ n2 ⌋ + 1, . . . , n, must be a leaf node without
j
descendants.
A(⌊ 2 ⌋) A (2 ) A (3 ) ▶ For any complete binary tree, we can gradually transform it into
a heap by repeatedly applying the restore routine on the subtrees
A (4 ) A (5 )
rooted at nodes from A(⌊ n2 ⌋) to A(1).

A( j)

87 88
Construction of a heap (cont’d) Construction of a heap (cont’d)

▶ Note that all leaf nodes can be considered as heaps. ▶ Hence, we start the construction of a heap from restoring the
subtree rooted at A(⌊ n2 ⌋).
4
4 4

15 26
15 26 48 26
Heap
48 37 48 37 15 37

Heap Heap Heap Heap

Heap
▶ So we do not have to perform any operation on leaf nodes. Not a heap

89 90

Construction of a heap (cont’d) Algorithm of constructing a heap

▶ We continue to restore the subtree rooted at root.

Input: A(1), A(2), · · · , A(n).

4
Output: A(1), A(2), · · · , A(n) as a heap.
1. for i = ⌊n/2⌋ down to 1 do
48 26 2. Restore(i, n)
3. end for
Heap
15 37

Heap

Not a heap

91 92
Time complexity of constructing a heap Time complexity of constructing a heap (cont’d)
k

▶ Recall that A(i) is an internal node for i = 1, 2, · · · , ⌊n/2⌋.

Note that ∑ i2i−1 = 2k (k − 1) + 1.
i=0
▶ Recall that A(i) must be a leaf node for i = ⌊n/2⌋ + 1, · · · , n. d−1 d−1 d−1
▶ The depth d of a heap is ⌊log2 n⌋. ∑ 2 ( d − L ) 2L = 2d ∑ 2L − 4 ∑ L2L−1
L=0 L=0 L=0
▶ Each internal node needs two comparisons.
= 2d(2 − 1) − 4(2d−1 (d − 1 − 1) + 1)
d
▶ Each node at level L needs 2(d − L) comparisons.
▶ Each level L has at most 2L nodes. = 2d2d − 2d − 4d2d−1 + 4 · 2d − 4
▶ The total number of comparisons for constructing a heap is: = 4 · 2d − 2d − 4
d−1 = 4 · 2⌊log2 n⌋ − 2⌊log2 n⌋ − 4
∑ 2(d − L)2L = O(n) ≤ 4 · 2log2 n − 2⌊log2 n⌋ − 4
L=0
= 4n − 2⌊log2 n⌋ − 4
≤ 4n
93 94

Algorithm of heap sort Time complexity of heap sort

▶ After deleting a number, 2⌊log2 i⌋ comparisons (in the worst

Input: A heap of A(1), A(2), · · · , A(n). case) are needed to restore the heap if there are i elements
Output: A sorted sequence of A(1), A(2), · · · , A(n). remaining.
▶ Therefore, the total number of comparisons needed to delete all
1. for i = n down to 2 do numbers is:
2. Output A(1)
n−1
3. A(1) = A(i) 2 ∑ ⌊log2 i⌋ = O(n log n) (refer to textbook)
4. Delete A(i) i=1
5. Restore(1, i − 1) ▶ Hence, the time complexity of heap sort is:
6. end for
7. Output A(1) O(n) + O(n log n) = O(n log n)

95 96
Convex hull problem Finding lower bound by problem transformation
Definition:
Given n points in the planes, the convex hull problem is to identify
the vertices of the smallest convex polygon in some order (clockwise
▶ What is the lower bound of the convex hull problem?
or counterclockwise).
▶ It appears rather difficult to find a meaningful lower bound of
the convex hull problem directly.
▶ However, we can easily obtain a very meaningful lower bound by
transforming the sorting problem, whose lower bound is known,
to this problem (denoted by sorting problem ∝ convex hull prob-
lem).

97 98

Sorting problem ∝ convex hull problem Lower bound of convex hull problem
1. Let x1 < x2 < · · · < xn be n sorted numbers.
▶ By solving the convex hull problem for these newly created
2. Create a 2-dimensional point (xi , xi 2 ) for each xi .
points, we can also solve the sorting problem.
y
instance transformation
sorting convex hull
problem problem
( x3 , x32 )
sorting algorithm convex hull algorithm

sorting convex hull

( x2 , x22 ) solution solution

( x1 , x12 ) solution transformation

x

99 100
Lower bound of convex hull problem (cont’d)
▶ Let Ω(sorting(n)) be the lower bound of the sorting problem.
▶ Let T (covex-hull(n)) be the time of an algorithm for solving the
convex hull problem.
▶ Let O(transform(n)) be the cost of problem transformation.
▶ Then, we have:

T (convex-hull(n)) + O(transform(n)) ≥ Ω(sorting(n))

T (convex-hull(n)) ≥ Ω(sorting(n)) − O(transform(n))

= Ω(n log n) − O(n)
= Ω(n log n)
▶ Ω(n log n) is a lower bound of the convex hull problem.

101