2IL50 Data Structures

2018-19 Q3

Lecture 3: Heaps
Solving recurrences

one more time …

Solving recurrences

Easiest: Master theorem

caveat: not always applicable

Alternatively: Guess the solution and use the substitution method to prove
that your guess it is correct.

How to guess:
1. expand the recursion
2. draw a recursion tree
Example
Example(A)
► A is an array of length n
1. n = A.length
2. if n==1
3. then return A[1]
4. else begin
5. Copy A[1…⌈n/2⌉] to auxiliary array B[1...⌈n/2⌉]
6. Copy A[1…⌈n/2⌉] to auxiliary array C[1…⌈n/2⌉]
7. b = Example(B); c = Example(C)
8. for i = 1 to n
9. do for j = 1 to i
10. do A[i] = A[j]
11. return 43
12. end
Example
Example(A) Let T(n) be the worst case running time of Example
on an array of length n.
► A is an array of length n
1. n = A.length Lines 1,2,3,4,11, and 12 take Θ(1) time.
2. if n==1 Lines 5 and 6 take Θ(n) time.
Line 7 takes Θ(1) + 2 T(⌈n/2⌉) time.
3. then return A[1] Lines 8 until 10 take
4. else begin n i n

5.

Copy A[1…⌈n/2⌉] to auxiliary array B[1...

i1 ⌈
jn/2
1
Θ(1) 
⌉] i1
 Θ(i)  Θ(n 2 )
time.

6. Copy A[1…⌈n/2⌉] to auxiliaryIfarray C[1…

n=1 lines ⌈n/2
1,2,3 are⌉]executed,
7. else lines 1,2, and 4 until 12 are executed.
b = Example(B); c = Example(C)
8. for i = 1 to n Θ(1) if n=1
➨ T(n) =
2T(n/2) + Θ(n2) if n>1
9. do for j = 1 to i
10. do A[i] = A[j] ➨ use master theorem …
11. return 43
12. end
The master theorem
Let a and b be constants, let f(n) be a function, and let T(n)
be defined on the nonnegative integers by the recurrence
T(n) = aT(n/b) + f(n)

Then we have:
1. If f(n) = O(nlog a – ε) for some constant ε > 0, then T(n) = Θ(nlog a). b
b

2. If f(n) = Θ(nlog a), then T(n) = Θ(nlog a log n)

b b

3. If f(n) = Ω(nlog a + ε) for some constant ε > 0, and if af(n/b) ≤ cf(n) for
b
some constant c < 1 and all sufficiently large n, then T(n) = Θ(f(n))
Quiz
Recurrence Master theorem?

1. T(n) = 4 T(n/2) + Θ(n3) yes T(n) = Θ(n3)

2. T(n) = 4 T(n/2) + Θ(n) yes T(n) = Θ(n2)

3. T(n) = T(n/2) + 1 yes T(n) = Θ(log n)

4. T(n) = T(n/3) + T(2n/3) + n no T(n) = Θ(n log n)

5. T(n) = 9 T(n/3) + Θ(n2) yes T(n) = Θ(n2 log n)

6. T(n) = √n T(√n) + n no T(n) = Θ(n log log n)

Substitution
1 if n = 1 or n = 2
T(n) =
7T(⌊n/3⌋) + n2 if n > 2

Claim: T(n) = O(n2)

(To show: exist constants c and n0 such that T(n) ≤ cn2 for all n ≥ n0)

Proof: by induction on n

Base case (n=1):

1 ≤ c ⋅ n2 = c ⋅ 12 = c for c ≥ 1

Base case (n=2):

1 ≤ c ⋅ n2 = c ⋅ 22 = 4c for c ≥ 0.25
Substitution
1 if n = 1 or n = 2
T(n) =
7T(⌊n/3⌋) + n2 if n > 2

Claim: T(n) = O(n2)

(To show: exist constants c and n0 such that T(n) ≤ cn2 for all n ≥ n0)

Proof: by induction on n

Inductive step:
IH: Assume that for all 1 ≤ k < n it holds that T(k) ≤ ck2.
Then T(n) = 7T(⌊n/3⌋) + n2
≤ 7 ⋅ c ⋅ (⌊n/3⌋)2 + n2 (by IH)
≤ 7/9 ⋅ cn2 + n2
= cn2 – 2/9 ⋅ cn2 + n2
≤ cn2 (if – 2/9 ⋅ cn2 + n2 ≤ 0)
(for c ≥ 9/2)
Substitution
1 if n = 1 or n = 2
T(n) =
7T(⌊n/3⌋) + n2 if n > 2

Claim: T(n) = O(n2)

(To show: exist constants c and n0 such that T(n) ≤ cn2 for all n ≥ n0)

Proof: by induction on n

Base case (n=1): […]

Base case (n=2): […]

Inductive step:
[…]

Let n0 = 1 and c = 9/2.

By induction it holds that T(n) = O(n2).
Tips
Analysis of recursive algorithms:
find the recursion and solve with master theorem if possible

Analysis of loops: summations

Some standard recurrences and sums:

 T(n) = 2T(n/2) + Θ(n) ➨ T(n) = Θ(n log n)

i 
i1
 ½ n(n+1) = Θ(n2)
n

i
i1
2


 ⅙ n(n+1)(2n+1) = Θ(n3)
Heaps
Event-driven simulation
Stores a set of events, processes first event (highest priority)

Supporting data structure:

 insert event
 find (and extract) event with highest priority
 change the priority of an event
Priority queue
Max-priority queue
abstract data type (ADT) that stores a set S of elements, each with an
associated key (integer value).

Operations
Insert(S, x): inserts element x into S, that is, S ← S ⋃ {x}
Maximum(S): returns the element of S with the largest key
Extract-Max(S): removes and returns the element of S with the
largest key
Increase-Key(S, x, k): give key[x] the value k
condition: k is larger than the current value of key[x]

Min-priority queue …
Implementing a priority queue

Insert Maximum Extract-Max Increase-Key

sorted list
sorted array
(Doubly) linked list
Linked list collection of objects stored in linear order, with objects pointing
to their predecessor and successor

L.head / 6 2 7 9 /

L.head points to the first object L.head=NIL if L is empty

Object x:
 ​x.prev points to the predecessor x.prev=NIL if x is first
 ​x.next points to the successor x.next=NIL if x is last
 ​x.key, x.data

Operations
 Search(L, key)
O(n)
 Insert(L, x)
O(1)
 Delete(L, x)
O(1)
Implementing a priority queue

Insert Maximum Extract-Max Increase-Key

sorted list Θ(n) Θ(1) Θ(1) Θ(n)
sorted array Θ(n) Θ(1) Θ(n)? Θ(n)

Today

Insert Maximum Extract-Max Increase-Key

heap Θ(log n) Θ(1) Θ(log n) Θ(log n)
Max-heap
35

30 19

30 8 11 12

17 2 5

Heap
nearly complete binary tree, filled on all levels except possibly the lowest.
(lowest level is filled from left to right)

Max-heap property: for every node i other than the root

key[Parent(i)] ≥ key[i]
Tree terminology
Binary tree: every node has 0, 1, or 2 children
Root: top node (no parent)
Leaf: node without children
Subtree rooted at node x: all nodes below and including x
Depth of node x: length of path from root to x
Depth of tree: max. depth over all nodes
Height of node x: length of longest path from x to leaf
Height of tree: height of root 35
Level: set of nodes with same depth
30 x 19
Family tree terminology
Left/right child
30 8 11 12
Parent
Grandparent …
17 2 5
Max-heap
35
35

30 19
30 19
30 8 21

30 8 11 12 35

30 19
17 2 5
30 8 12

Heap
nearly complete binary tree, filled on all levels except possibly the lowest.
(lowest level is filled from left to right)

Max-heap property: for every node i other than the root

key[Parent(i)] ≥ key[i]
Properties of a max-heap
Lemma
The largest element in a max-heap is stored at the root.

Proof:
Properties of a max-heap
Lemma
The largest element in a max-heap is stored at the root.

Proof: x root
y arbitrary node
x
z1, z2, …, zk nodes on path z1
between x and y
z2

max-heap property ➨ key[x] ≥ key[z1] ≥ … ≥ key[zk] ≥ key[y]

➨ the largest element is stored at x ■
Storing a heap
How to store a heap?
 Tree structure?
 In an array? 35

30 19

30 8 11 12

17 2 5
Implementing a heap with an array
35

30 19

30 8 11 12

17 2 5 array A[1 … A.length]

35 30 19 30 8 11 12 17 2 5
1 2 3 4
heap-size[A]

A.length = length of array A

heap-size[A] =number of elements in the heap
Implementing a heap with an array
level 0

level 1

level 2, position 3

array A[1 … A.length]

1 2 3 4
heap-size[A]
kth node on level j is stored at position A[2j+k-1]
left child of node at position i = Left(i) = 2i
right child of node at position i = Right(i) = 2i + 1
parent of node at position i = Parent(i) = ⌊ i/2 ⌋
Priority queue
Max-priority queue
abstract data type (ADT) that stores a set S of elements, each with an
associated key (integer value).

Operations
Insert(S, x): inserts element x into S, that is, S ← S ⋃ {x}
Maximum(S): returns the element of S with the largest key
Extract-Max(S): removes and returns the element of S with the
largest key
Increase-Key(S, x, k): give key[x] the value k
condition: k is larger than the current value of key[x]
Implementing a max-priority queue
Set S is stored as a heap in an array A.
Operations: Maximum, Extract-Max, Insert, Increase-Key.

35 30 19 30 8 11 12 17 2 5
1 2 3 4
heap-size[A]
Implementing a max-priority queue
Set S is stored as a heap in an array A.
Operations: Maximum, Extract-Max, Insert, Increase-Key.

35 30 19 30 8 11 12 17 2 5
1 2 3 4
heap-size[A]
Maximum(A)
1. if heap-size[A] < 1
2. then error
3. else return A[1]

running time: O(1)

Implementing a max-priority queue
Set S is stored as a heap in an array A.
Operations: Maximum, Extract-Max, Insert, Increase-Key.

35 30 19 30 8 11 12 17 2 5
1 2 3 4
heap-size[A]
Implementing a max-priority queue
Set S is stored as a heap in an array A.
Operations: Maximum, Extract-Max, Insert, Increase-Key.

35
30

30 19

30
17 8 11 12

17 2 5
Implementing a max-priority queue
Set S is stored as a heap in an array A.
Operations: Maximum, Extract-Max, Insert, Increase-Key.

35
5

30 19
Heap-Extract-Max(A)
1. if heap-size[A] < 1 30 8 11 12
2. then error
3. max = A[1] 17 2 5
4. A[1] = A [heap-size[A]] restore heap property
5. heap-size[A] = heap-size[A]–1
6. Max-Heapify(A,1)
7. return max
Max-Heapify
Max-Heapify(A, i)
► ensures that the heap whose root is stored at position i has the max-heap
property
► assumes that the binary subtrees rooted at Left(i) and Right(i) are max-
heaps
Max-Heapify
Max-Heapify(A, i)
► ensures that the heap whose root is stored at position i has the max-heap
property
► assumes that the binary subtrees rooted at Left(i) and Right(i) are max-
heaps

Max-Heapify(A,1) 10
40
exchange A[1] with largest child
40
10
35 19
Max-Heapify(A,2)
30 10
35 11 12
exchange A[2] with largest child
Max-Heapify(A,5) 17 2 5

A[5] larger than its children ➨ done.

Max-Heapify
Max-Heapify(A, i)
► ensures that the heap whose root is stored at position i has the max-heap
property
► assumes that the binary subtrees rooted at Left(i) and Right(i) are max-
heaps
1. if Left(i) ≤ heap-size[A] and A[Left(i)] > A[i]
2. then largest = Left(i)
3. else largest = i
4. if Right(i) ≤ heap-size[A] and A[Right(i)] > A[largest]
5. then largest = Right(i)
6. if largest ≠ i
7. then exchange A[i] and A[largest]
8. Max-Heapify(A, largest)

running time? O(height of the subtree rooted at i) = O(log n)

Implementing a max-priority queue
Set S is stored as a heap in an array A.
Operations: Maximum, Extract-Max, Insert, Increase-Key.

Insert (A, key)

1. heap-size[A] = heap-size[A] +1
2. A[heap-size[A]] = -∞
3. Increase-Key(A, heap-size[A], key)
Implementing a max-priority queue
Set S is stored as a heap in an array A.
Operations: Maximum, Extract-Max, Insert, Increase-Key.
Implementing a max-priority queue
Set S is stored as a heap in an array A.
Operations: Maximum, Extract-Max, Insert, Increase-Key.

35

30 19

Increase-Key(A, i, key) 30 8 11
42 12
1. if key < A[i]
2. then error 17 2 5
3. A[i] = key
4. while i>1 and A[Parent(i)] < A[i]
5. do exchange A[Parent(i)] and A[i]
6. i = Parent(i)

running time: O(log n)

Building a heap
Build-Max-Heap(A)
► Input: array A[1..n] of numbers
► Output: array A[1..n] with the same numbers, but rearranged, such
that the max-heap property holds
1. heap-size = A.length starting at ⌊A.length / 2⌋ is sufficient
2. for i = A.length downto 1
3. do Max-Heapify(A,i)

Loop Invariant
P(i): nodes i+1, …, n are each the root of a max-heap

Maintenance
P(i) holds before line 3 is executed,
P(i – 1) holds afterwards
Building a heap
Build-Max-Heap(A)
1. heap-size = A.length
2. for i = A.length downto 1
3. do Max-Heapify(A,i) O(height of node i)

∑i O(1 + height of i) height of node i

# edges longest simple downward
= ∑0 ≤ h ≤ log n (# nodes of height h) ∙ O(1+h) path from i to a leaf.

≤ ∑0 ≤ h ≤ log n (n / 2h+1) ∙ O(1+h)

= O(n) ∙ ∑0 ≤ h ≤ log n h / 2h+1
= O(n) h=2

h=1

h=0
The sorting problem
Input: a sequence of n numbers ‹a1, a2, …, an›
Output: a permutation of the input such that ‹ai1 ≤ … ≤ ain›

Important properties of sorting algorithms:

running time: how fast is the algorithm in the worst case

in place: only a constant number of input elements are ever stored
outside the input array
Sorting with a heap: Heapsort
HeapSort(A)
1. Build-Max-Heap(A)
2. for i = A.length downto 2
3. do exchange A[1] and A[i]
4. heap-size[A] = heap-size[A] – 1
5. Max-Heapify(A,1)

Loop invariant
P(i): A[i+1 ... n] is sorted and contains the n – i largest elements,
A[1…i] is a max-heap on the remaining elements

Maintenance
P(i) holds before lines 3-5 are executed,
P(i – 1) holds afterwards

Running time: O(n log n)

Sorting algorithms

worst case running time in place

InsertionSort Θ(n2) yes
MergeSort
Θ(n log n) no
HeapSort yes
Θ(n log n)