Algorithms and Data Structures

Lecture notes: Algorithm paradigms: Greedy

algorithms Part 1

Lecturer: Michel Toulouse

Hanoi University of Science and Technology

[email protected]

19 juin 2024
Outline

Greedy algorithms : an introduction

Greedy algo for making change

Failed greedy strategies for 0-1 knapsack

Greedy choice property

Minimum spanning tree problem

Optimal subtructure
Greedy choice property : proofs
Kruskal’s algorithm
Prim’s algorithm
Greedy algorithms

Greedy algorithms can be used to find solutions to ”combinatorial

optimization problems” such as those seen for dynamic programming :
making change, 0-1 knapsack, shortest path problems, longest
common substring, many others.

Unfortunately, often the solutions found by a greedy algorithm are

feasible but not optimal.
Greedy algorithms find optimal solutions when a combinatorial
optimization problem satisfies the following two conditions :
1. the problem exhibit the optimal substructure property (required for DP
problems) AND
2. the problem satisfies the ”greedy choice property” (a condition that
0-1-knapsack and making change fail to satisfy !)
Greedy algorithms & combinatorial opt problems

The optimal solution to a combinatorial optimization problem consists

to select a subset of objects that optimize a cost function while
satisfying all the constraints of the problem
The way recursive algorithms and dynamic programming algorithms
solve such problems is often quite complicated
On the contrary, the strategy used by greedy algorithms is very simple :

▶ sort the object in increasing or decreasing order along some sorting criterion
(the ”greedy criterion”)
▶ according to the sorted order, consider each object individually and add it to
the subset that makes the solution if adding the object produce a subset that
still satisfy the constraints of the problem
Greedy algorithms

Thus greedy algorithms attempt to construct a solution one object at a

time

At each step, a decision is made to add the object or not in the solution

Once decision is made in one particular step it is never changed in a

later steps (unlike recursive and DP algos).

Greedy algorithms always compute a feasible solution (satisfy all the

constraints), but may fail to compute an optimal solution.
Making change : a greedy algorithm

Greedy algorithms cannot in general be applied to the making change problem.

However there are some problem instances of making change for which greedy work.

The problem is paying back an amount n to a customer using the smallest possible
number of coins.

Here is a greedy recursive algorithm. The greedy criterion consists to return change
always using the largest possible coin ≤ n. Coins are sorted in decreasing order of
their value, they are then selected in this order to return change.

Greedy MakingChange(int n)
if (n == 0) return 0 ;
c = the largest coin that is no larger than n ;
return (⌊n/c⌋) + Greedy MakingChange(n − (⌊n/c⌋ × c)) ;
Example of making change

Assume the coin denominations are 25, 10, 5 and 1 cents and we make
change for 97 cents using the above greedy algorithm.
▶ Choose the largest coin that is no larger than n (here 25 cents is
the largest coin no larger than 97 cents)
▶ Give out ⌊n/c⌋ of the c cent coins (⌊97/25⌋ = three quarters)
▶ Make change recursively for n − ⌊n/c⌋c cents (97 − 3 × 25 = 22
cents)
▶ 10 cents is the largest coin no larger than 22 cents, so we hand out
two dimes and make change for 22 − 20 = 2 cents
▶ a penny is the largest coin no larger than 2 cents, so we hand out
two pennies and we’re done !
Altogether we hand out 3 quarters, 2 dimes and 2 pennies, for a total
of 7 coins
Greedy may fail to solve making change optimally

The greedy algorithm always gives out the fewest number of coins in
change with these problem instances where coins are quarters, dimes,
nickels and pennies.

It doesn’t give out the fewest number of coins if the coins are 15 cents,
7 cents, 2 cents and 1 cent.

Example : For 21 cents the greedy algorithm would

▶ give out one 15-cent coin, remainder 21 − 15 = 6 and
▶ give out three 2-cent coins,
▶ for a total of four coins.
Better : give out three 7-cent coins for a total of three coins !
Greedy algorithms for the 0-1 knapsack
There exists at least three different greedy criteria that can be used for
the 0-1 knapsack problem, but none of them is guarantee to always
return the optimal solution :
1. sort objects in decreasing order according to their value
2. sort objects in increasing order according to their weight
v [i]
3. sort objects in decreasing order according to their density w [i]

Greedy 0-1-Knapsack(int W = 11, int n)

int selected objects[n] = {0} ;
int objects[n] ;
int i, j, w = W
for (i=1 ; i ≤ n, i++)
j = objects[i]
if (w > wj )
selected objects[i] = j
w = w − wj
i ++
Greedy criterion is values

Object i 1 2 3 4 5
Weight wi 2 3 5 6 4
Value vi 6 1 12 8 7
Greedy 0-1-Knapsack(int W = 11,int n)
int selected objects[n] = {0} ; int w = W ; int objects[n] ; int i, j
Sort objects[n] in decreasing order of the object’s values
objects = [3, 4, 5, 1, 2] after sorting
for (i = 1; i ≤ n, i + +)
j = objects[i]
if (w > wj )
selected objects[i] = j
w = w − wj
i = 1, object = 3, selected objects = [3, 0, 0, 0, 0], w = 6
i = 2, object = 4, selected objects = [3, 4, 0, 0, 0], w = 0
Solution is 20
Greedy criterion is weights

Object i 1 2 3 4 5
Weight wi 2 3 5 6 4
Value vi 6 1 12 8 7
Greedy 0-1-Knapsack(int W = 11,int n)
int selected objects[n] = {0} ; int w = W ; int objects[n] ; int i, j
Sort objects[n] in increasing order of the object’s weights
objects = [1, 2, 5, 3, 4] after sorting
for (i = 1; i ≤ n, i + +)
j = objects[i]
if (w > wj )
selected objects[i] = j
w = w − wj
i = 1, object = 1, selected objects = [1, 0, 0, 0, 0], w = 9
i = 2, object = 2, selected objects = [1, 2, 0, 0, 0], w = 6
i = 3, object = 5, selected objects = [1, 2, 5, 0, 0], w = 2
Solution is 14
Greedy criterion is density

Object i 1 2 3 4 5
Weight wi 2 3 5 6 4
Value vi 6 1 12 8 7
Greedy 0-1-Knapsack(int W = 11,int n)
int selected objects[n] = {0} ; int w = W ; int objects[n] ; int i, j
Sort objects[n] in decreasing order of the object’s density wv [i]
[i]
objects = [1, 3, 5, 4, 2] after sorting
for (i = 1; i ≤ n, i + +)
j = objects[i]
if (w > wj )
selected objects[i] = j
w = w − wj
i = 1, object = 1, selected objects = [1, 0, 0, 0, 0], w = 9
i = 2, object = 3, selected objects = [1, 3, 0, 0, 0], w = 4
i = 3, object = 5, selected objects = [1, 2, 5, 0, 0], w = 0
Solution is 25
Problems where greedy strategies fail

The greedy strategy consists :

1. Sort objects according to some greedy criterion
2. Construct a solution by greedily inserting objects in the solution if
the insertion does not violate the constraints of the optimization
problem
3. Stop when no more object can be inserted
We have just seen that the greedy strategy returns 3 different solutions
for the same 0-1 knapsack problem and may fail for some instance of
the coin change problem

The reason greedy strategy fails is because these problems do not

satisfy the greedy choice property (although they satisfy the optimal
substructure property)
Why greedy fail

The problem is to maximize the sum of the nodes along a path in the
tree.
Greedy algorithm makes decisions purely on local information

To succeed, greedy needs a sequence of local decisions that leads to an

optimal solution. In the graph above, such sequence does not exist.

An optimization problem verifies the greedy choice property if there

exists a sequence of local decisions that leads to an optimal solution
Greedy choice property

The ”greedy choice property” means that an optimal solution can be

obtained by making a “greedy” (local) decision at every step
▶ In other words there exists a sequence a greedy choices leading to
the optimal solution

Before attempting to write a greedy algorithm for a problem, one must

prove that the problem exhibit the greedy choice property

This king of proof is often by induction

▶ Assume the greedy algorithm finds an optimal solution for any
problem of size n − 1 or smaller.
▶ Look at any optimal solution for a problem of size n and show
that it can be obtained with a greedy choice apply to the solution
of the smaller problem.
Minimum Spanning Tree (MST)

Let G = (V , E ) be a connected, weighted graph.

A weighted graph is a graph where a real number called weight is

associated with each edge.

A spanning tree of G is a subgraph T of G which is a tree that spans

all vertices of G . In other words, T contains all of the vertices of G .
Minimum Spanning Tree

The weight of a spanning tree T is the sum of the weights of its edges.
That is, X
w (T ) = w (u, v ).
(u,v )∈T

A minimum spanning tree (MST) of G is spanning tree T of minimum

weight.

A minimum spanning tree always exists.

Minimum Spanning Tree : Examples

As for any optimization problem, there could be more than one optimal
solution, here there are two minimum spanning trees.

4 3 2

9 7 4
3 3
G 5
2
6
4 1

3 2 4 3 2

T1 3 3 T2
3 3
2 2

4 1 1
Optimal substructure for MST

MST exhibits the optimal substructure property : an optimal tree is

composed of optimal subtrees
▶ Let T be an MST of G and consider an edge (u, v ) not adjacent
to a leave of T
▶ Removing (u, v ) partitions T into two trees T1 and T2

Claim :
T1 is an MST of G1 = (V1 , E1 ), and T2 is an MST of G2 = (V2 , E2 )

Proof : Note that V1 ∩ V2 = ∅. By construction

w (T ) = w (u, v ) + w (T1 ) + w (T2 ). If T1 or T2 was not a MST then
w (T ) would be suboptimal, a contradiction of the assumption that T
is a MST.
Optimal substructure for MST

Assume (u, v ) = (b, e). After removing (b, e) from the MST we have :
1. T1 = {(e, d), (d, g )} and the graph span by the nodes in T1 is
G1 = {(e, d), (d, g ), (e, g )}
2. T2 = {(a, b), (b, c), (a, f )} and the graph span by the nodes in T2
is G2 = {(a, b), (b, c), (a, f ), (f , c), (f , b)}

It is easy to verify that T1 and T2 are the MST respectively of graphs

G1 and G2
MST possess the greedy choice property

The MST satisfies the greedy choice property if there exists a sequence of edges selected
greedily which results into a MST.

It is easy to find this sequence. Select first an edge with minimum cost in the graph, here
(d, e). This is the first edge in the MST.
Look locally how the MST can be expanded. There are 6 edges adjacent either to d or e.
Select the one with minimum cost (b, e). There are two edges in the MST.
Make a local choice again by selecting the edge that is adjacent to the current partial MST
and has minimum cost, the edge (b, c).

It seems by extending this sequence of local choices a MST will be found...

Greedy choice property for MST : proof by induction
Theorem : The MST satisfies the greedy choice property.

Proof by induction :
Induction hypothesis : Let T be a MST for a graph G , and A be a subset of edges
in T . In the graph above, assume A = {(a, b), (b, c), (a, f )}

Let C be a cut of G which does not include any edge of A. For the example above,
a cut is {(a, d), (b, d), (b, e), (c, e)}

Induction step : Add to A the edge with minimum cost in the cut, i.e. edge (b, e).
After adding (b, e), A is still part of the MST T for G

This shows that a MST can be constructed by extending the partial MST with a
greedy choice on the length of the edges
Greedy choice property for MST : proof by induction

The MST satisfies the greedy choice property. Starting from the base case.

Let A = ∅ initially. Select edge (b, c) to enter A, A = {(b, c)}. The cut is
C1 = {(c, f ), (b, f ), (a, b), (b, d), (b, e), (c, e)}

The shortest edge in C1 is (a, b), A = A ∪ (a, b) = {(b, c), (a, b)}. The new cut
C2 = {(a, f ), (a, d), (c, f ), (b, f ), (b, d), (b, e), (c, e)}

A = A ∪ (a, f ) = {(b, c), (a, b), (a, f )} given (a, f ) is the shortest edge in the cut
C2 . The new cut C3 = {(a, d), (b, d), (b, e), (c, e)}

etc.
Greedy choice property for MST : proof by contradiction

Let G = (V , E ) be an undirected graph. Let ev be the cheapest edge

adjacent to each vertex v .

6 10
B D G

1 7 15

5 9 11 14
A E I
4
8
2 13

C F H
3 12

Theorem :
The minimum spanning tree of G contains every ev
Greedy choice property for MST : proof by contradiction
Proof :

Suppose on the contrary that MST of G does not contain some edge
ev = (u, v ).

Let T = optimal MST of G

By adding ev = (u, v ) to T , we obtain a cycle

Greedy choice property for MST : proof by contradiction

Proof :

By our choice of ev , the weight of (u, v ) must be less than the weight
of (v , w ) in T
If we delete (v , w ) and include ev , we obtain a new spanning tree
cheaper than T , a contradiction
Constructing an MST

The minimum spanning tree problem can be solved optimally using a

greedy algorithm.

There are actually two common greedy algorithms to construct MSTs :

▶ Kruskal’s algorithm
▶ Prim’s algorithm

Both of these algorithms use the same basic ideas, but in a slightly
different fashion.
Kruskal’s Algorithm
Initially makes each node of the graph as a tree with a single node, i.e. a
forest of trees
Then greedily select the shortest edge not making a cycle to be part of the
MST
This edge merge two trees into a single one

Algorithm Kruskal(G )
input : Graph G = (V , E ) ;
output : A minimum spanning tree T ;
Sort E by increasing length ;
n = |V | ;
T = ∅;
For each v ∈ V MakeSet(v)
for i = 1 to |E | do /* Greedy loop */
{u, v } = shortest edge not yet considered ;
if (FindSet(u) ̸= FindSet(v)) then
Union(FindSet(u),
S FindSet(v))
T = T {u, v } ;
return T
Kruskal’s Algorithm Example
for i = 1 to |E | do /* Greedy loop */
{u, v } = shortest edge not yet considered ;
if (FindSet(u) ̸= FindSet(v)) then
Union(FindSet(u),
S FindSet(v))
T = T {u, v } ;
return T

A 5 4 C A 5 4 C
B 1 B 1
4 2 4 2
D
6 E D
6 E
12 2 12 2
9 7 3 8 9 7 3 8
E F G E F G
11 1 11 1

A 5 4 C A 5 4 C
B 1 B 1
4 2 4 2
D 6 E D 6 E
12 2 12 2
9 7 3 8 9 7 3 8
E F G E F G
11 1 11 1
for i = 1 to |E | do /* Greedy loop */
{u, v } = shortest edge not yet considered ;
if (FindSet(u) ̸= FindSet(v)) then
Union(FindSet(u),
S FindSet(v))
T = T {u, v } ;
return T

A 5 4 C A 5 4 C
B 1 B 1
4 2 4 2
D
6 E D
6 E
12 2 12 2
9 7 3 8 9 7 3 8
E F G E F G
11 1 11 1

A 5 4 C
A 5 4 C 4
B 1
B 1 2
4 2 6
6 2 D E
2 D E 12
12 9 7 3 8
9 7 3 8 E F G
E F G 11 1
11 1
for i = 1 to |E | do /* Greedy loop */
{u, v } = shortest edge not yet considered ;
if (FindSet(u) ̸= FindSet(v)) then
Union(FindSet(u),
S FindSet(v))
T = T {u, v } ;
return T

5 4 A 5 4 C
A C B 1
4
B 1 4 2
2 6
D
6 E 12 2 D E
12 2
9 8 9 7 3 8
7 3
E F G
E F G 1
11 1 11

A 5 4 C A 5 4 C
B 1 B 1
4 2 4 2
D
6 E D
6 E
12 2 12 2
9 7 3 8 9 7 3 8
E F G E F G
11 1 11 1
for i = 1 to |E | do /* Greedy loop */
{u, v } = shortest edge not yet considered ;
if (FindSet(u) ̸= FindSet(v)) then
Union(FindSet(u),
S FindSet(v))
T = T {u, v } ;
return T

A 5 4 C A 5 4 C
B 1 B 1
4 2 4 2
D
6 E D
6 E
12 2 12 2
9 7 3 8 9 7 3 8
E F G E F G
11 1 11 1

A 5 4 C 4
B 1
4 2 2 1
6 4
2 D E
12 2
9 7 3 8 9
E F G
11 1 1
Kruskal’s Algorithm : time complexity analysis

Algorithm Kruskal(G )
input : Graph G = (V , E ) ; ▶ Sort edges : O(E log E )
output : A minimum spanning tree T ; ▶ O(n) MakeSet()’s
Sort E by increasing length ; ▶ Aggregate analysis of the second ”for”
n = |V | ; loop yields a cost O(n log n) for the
T = ∅; union operations and 2E FindSet, thus
for each v ∈ V MakeSet(v) O(E + n log n)
for i = 1 to |E | do ▶ If |E | ≈ n2 , in O(E + n log n), E is the
shortest {u, v } not yet considered ; dominant term.
if (FindSet(u) ̸= FindSet(v)) then ▶ However edge sorting cost O(E log E ),
Union(FindSet(u), FindSet(v)) this is the most costly operation of
S Kruskal’s algorithm, O(E log E ) is the
T = T {u, v } ;
asymptotic running time of this algo.
return T
Exercises on Kruskal’s algorithm
Consider the non-oriented weighted graph G below represented using
an adjacency matrix
a b c d e f g h i j k l m
a 1 6 2
b 1 2 3
c 4
d 1 5 3 1
e 1 3 2
f 2
g 4
h 3 1
i 2
j 1
k 1
l 2
m

1. Draw this graph

2. Compute by hand a minimum spanning tree using Kruskal’s
algorithm. Show which edge is selected at each step of the greedy
algorithm and tell whether it is included in the spanning tree or
whether it is rejected
3. Is there more than one minimum spanning for this graph
Exercise on Kruskal’s algorithm

Compute the MST using Kruskal’s algorithm. At each step of the

computation, show the configuration of sub-trees.

1
6 5

1
2 5 5 4

3 3 2
6 4

5 6
6
Prim’s Algorithm
1. Prim selects arbitrary a node A of the graph to be the root of the MST
T
2. Then selects the shortest edge adjacent to A to be the first edge of T
3. Prim grows the MST T by selecting the shortest edge adjacent to a
node of T , not yet in T and not forming a cycle

Prim MST(G , r )
∀u∈G
key [u] = Max Int ;
key [r ] = 0 ;
p[r ] = NIL ;
Q = MinPriorityQueue(V )
while (Q ̸= ∅)
u = ExtractMin(Q) ;
for each v adjacent to u
if ((v ∈ Q) & (w (u, v ) < key [v ]))
p[v ] = u ;
key [v ] = w (u, v ) ;
 a 5 c 4 v a b c d e f g h
f 1
4 2
6 k 0 oo oo oo oo oo oo oo
while (Q ̸= ∅) 12 2 e h
p nil ? ? ? ? ? ? ?
9 7 3 8
u = ExtractMin(Q) ; b d g
11 1 Q=[a,b,c,d,e,f,g,h]
for each v adjacent to u
5 4
if ((v ∈ Q) a
4
c
2
f 1 v a b c d e f g h
6 k 0 12 5 4 oo oo oo oo
& (w (u, v ) < key [v ])) 12 2 e h
p nil a a
9 3 8 a ? ? ? ?
7
p[v ] = u ; b d g
11 1 Q=[d,c,b,e,f,g,h]
key [v ] = w (u, v ) ;
a 5 c 4 v a b c d e f g h
f 1
4 2
e 6 k 0 11 2 4 7 oo 1 oo
12 2 h
9 8 p nil d d a d ? d ?
7 3
b d g
11 1 Q=[g,c,e,b,f,h]

a 5 c 4 v a b c d e f g h
f 1
4 2
6 k 0 11 2 4 3 8 1 oo
12 2 e h
9 8 p nil d d a g g d ?
7 3
b d g
11 1 Q=[c,e,f,b,h]
 a 5 c 4 v a b c d e f g h
f 1
4 2
6 k 0 9 2 4 2 4 1 oo
while (Q ̸= ∅) 12
9
2 e h
p nil c d a c c d ?
7 3 8
u = ExtractMin(Q) ; b d g
11 1 Q=[e,f,b,h]
for each v adjacent to u
5 4
if ((v ∈ Q) a
4
c
2
f 1 v a b c d e f g h
6 k 0 9 2 4 2 4 1 6
& (w (u, v ) < key [v ])) 12 2 e h
p nil c
9 3 8 d a c c d e
p[v ] = u ; 7
b d g
11 1 Q=[f,h,b]
key [v ] = w (u, v ) ;
a 5 c 4 v a b c d e f g h
f 1
4 2
6 k 0 9 2 4 2 4 1 1
12 2 e h
9 8 p nil c d a c c d f
7 3
b d g
11 1 Q=[h,b]
 a 5 c 4 v a b c d e f g h
f 1
4 2
6 k 0 9 2 4 2 4 1 1
while (Q ̸= ∅) 12
9
2 e h
p nil c d a c c d f
7 3 8
u = ExtractMin(Q) ; b d
1
g
Q=[b]
11
for each v adjacent to u
5 4
if ((v ∈ Q) a
4
c
2
f 1 v a b c d e f g h
6 k 0 9 2 4 2 4 1 1
& (w (u, v ) < key [v ])) 12 2 e h
p nil c a c
9 3 8 d c d f
7
p[v ] = u ; b d g
11 1 Q=[]
key [v ] = w (u, v ) ;
Prim’s Algorithm : time complexity analysis
Min Priority Queue :
▶ Min priority queue is the same as the max priority queue except it uses a
MinHeapify() routine as below.
▶ MinHeapify() always return the smallest element of an array as the root of the
heap.
▶ A call to min priority queue returns the root, the smallest element of the queue

MinHeapify(A, i)
l = Left(i) ; r = Right(i) ;
if (l ≤ heap size(A) & A[l] < A[i])
smallest = l ;
else
smallest = i ;
if (r ≤ heap size(A) & A[r] < A[smallest])
smallest = r ;
if (smallest != i)
Swap(A, i, smallest) ;
Heapify(A, smallest) ;
Prim’s Algorithm running time analysis

▶ The initialization of the arrays key cost

O(|V | = n)
Prim MST(G , r ) ▶ Initializing the Min Priority Queue cost the
∀u∈G same as Buildheap(), O(n) for the tight bound
key [u] = Max Int ; ▶ The while loop iterates n times, executing
key [r ] = 0 ; ExtractMin n times
p[r ] = NIL ;
▶ Each extractMin() incurs a MinHeapify
Q = MinPriorityQueue(V )
while (Q ̸= ∅) which cost O(log n) in worst case.
u = ExtractMin(Q) ; ▶ Given the n extractMin() operations,
for each v adjacent to u extractMin() cost n log n in total in the
if ((v ∈ Q) & (w (u, v ) < key [v ])) while loop
p[v ] = u ;
key [v ] = w (u, v ) ;
Prim’s Algorithm running time analysis

▶ Aggregate analysis of the for loop :

▶ Each node u is extracted exactly one
Prim MST(G , r ) time from the queue Q.
∀u∈G ▶ The number of edges is |E |, the for
key [u] = Max Int ;
loop iterates a total of |E | times
key [r ] = 0 ;
p[r ] = NIL ; during the n iterations of the while
Q = MinPriorityQueue(V ) loop
while (Q ̸= ∅)
▶ Each iteration of the for loop may reduce
u = ExtractMin(Q) ;
for each v adjacent to u the value of key [v ].
if ((v ∈ Q) & (w (u, v ) < key [v ])) ▶ Repositioning v in min heap requires
p[v ] = u ;
key [v ] = w (u, v ) ; moving up v in the heap, which cost
O(log n)

The total cost of the for loop is |E | × log n, i.e. O(|E | log n).
while loop is O(n lg n + E lg n). If |E | ≈ n2 , the dominant term of the while loop expression
is O(E lg n), this is the asymptotic cost of Prim’s algo.
Exercise on Prim’s algorithm

Compute the MST using Prim’s algorithm. At each step of the

computation, show the configuration of the partial MST.

1
6 5

1
2 5 5 4

3 3 2
6 4

5 6
6
Exercise on Prim’s algorithm
Compute the MST using Prim’s algorithm. Fill the table at each step.

v a b c d e f
k
P

3
a e
1
9 4

3
b d
1
4
2

c 5
f