Introduction to Algorithms

Greedy Algorithms
Greedy Algorithms
• A greedy algorithm always makes the choice that looks best
at the moment

• My everyday examples
• Playing cards
• Invest on stocks
• Choose a university

• The hope
• A locally optimal choice will lead to a globally optimal solution
Introduction

•Similar to Dynamic Programming

• It applies to Optimization Problem
•When we have a choice to make, make the one that looks best right
now
• Make a locally optimal choice in hope of getting a globally optimal
solution
• Greedy algorithms don’t always yield an optimal solution, but
sometimes they do
• For many problems, it provides an optimal solution much more quickly
than a dynamic programming approach
Elements of the greedy strategy

1. Determine the optimal substructure of the problem.

2. Develop a recursive solution. (For the activity-selection problem,
we formulated recurrence (16.2), but we bypassed developing a
recursive algorithm based on this recurrence.)
3. Show that if we make the greedy choice, then only one sub-
problem remains.
4. Prove that it is always safe to make the greedy choice. (Steps 3 and
4 can occur in either order.)
5. Develop a recursive algorithm that implements the greedy strategy.
6. Convert the recursive algorithm to an iterative algorithm.
Coin changing problem

• An example:
• A cappuccino at Starbucks is$1.50
• Plus tax it is: 1.5*1.08 = $1.62
• Often, we give the cashier 2 $1 notes
• She need to give back, 38 cents as change
• Generally, you never see she gives you 38 pennies.
• What is algorithm here?
Coin changing problem

• Coin changing problem (informal):

• Given certain amount of change: n cents
• The denominations of coins are: 25, 10, 5, 1
• How to use the fewest coins to make this change?
• i.e. n = 25a + 10b + 5c + d, what are the a, b, c, and d, minimizing
(a+b+c+d)
• Can you design an algorithm to solve this problem?
Coin changing problem

• n = 25a + 10b + 5c + d, what are the a, b, c, and d, minimizing

(a+b+c+d)
• How to do it in brute-force?
• At most we use n pennies
• Try all the combinations where a<=n, b<=n, c<=n, d<=n
• Choose all the combinations that n = 25a + 10b + 5c + d
• Choose the combination with smallest (a+b+c+d)

How many combinations? Θ(𝑛4 )

Time complexity is
Coin changing problem

• n = 25a + 10b + 5c + d, what are the a, b, c, and d, minimizing

(a+b+c+d)
• How to do it in divide-and-conquer?
coinD&C( n ){
1. if(n<5) return (a=0, b=0, c=0, d=n, n)
2. if(n==5) return (a=0, b=0, c=1, d=0, 1)
3. if(n==10) return (a=0, b=1, c=0, d=0, 1)
4. if(n==25) return (a=1, b=0, c=0, d=0, 1)
5. s= ;
6. Increase s.a or s.b or s.c or s.d by 1 according to the coin used in the minimum one
7. return (s.a, s.b, s.c, s.sum+1);

What is the recurrence equation? 𝑇 ( 𝑛 ) =𝑇 ( 𝑛 −25 ) +𝑇 ( 𝑛 −10 ) +𝑇 (𝑛 − 5 ) +𝑇 ( 𝑛 −1 ) +1

Time complexity? and ,
 Coin changing problem

• n = 25a + 10b + 5c + d, what are the a, b, c, and d, minimizing

(a+b+c+d)
• How to do it in dynamic programming?
coinDP( n ){
1. If(solution for n in memo) return memo(n);
2. if(n<5) return (a=0, b=0, c=0, d=n, n)
3. if(n==5) return (a=0, b=0, c=1, d=0, 1)
4. if(n==10) return (a=0, b=1, c=0, d=1, 1)
5. if(n==25) return (a=1, b=0, c=0, d=1, 1)
6. s= ;
7. Increase s.a or s.b or s.c or s.d by 1 according to the coin used in the minimum one
8. Put (s.a, s.b, s.c, s.sum+1) in memo as memo(n);
9. return (s.a, s.b, s.c, s.sum+1);
How many sub problems? n
If subproblems are solved, how much time to solve a problem? (1)
Time complexity? =(n)
 Coin changing problem
• n = 25a + 10b + 5c + d, what are the a, b, c, and d, minimizing
(a+b+c+d)
• How to do it by a greedy algorithm?
coinGreedy( n ){
if(n>=25) s = coinGreedy(n-25); s.a++;
else if(n>=10) s = coinGreedy(n-10); s.b++;
else if(n>=5) s = coinGreedy(n-5); s.c++;
else s=(a=0, b=0, c=0, d=n, sum=n); Greedy choice
s.sum++; Always choose the possible largest coin
return s;

It that greedy algorithm correct?

Time complexity? =(n)

If n is large, in most of the subproblems it chooses quarter, so it is much faster than dynamic
programming and in DP
Coin changing problem

• Optimal substructure
• After the greedy choice, assuming the greedy choice is correct, can we get
the optimal solution from sub optimal result?
• 38 cents
• Assuming we have to choose 25
• Is a quarter + optimal coin(38-25) the optimal solution of 38 cents?
• Greedy choice property
• If we do not choose the largest coin, is there a better solution?
Coin changing problem

• For coin denominations of 25, 10, 5, 1

• The greedy choice property is not violated
• For other coin denominations
• May violate it
• E.g. 10, 7, 1
• 15 cents
• How to prove the greedy choice property for denominations 25, 10,
5, 1?
• Optimal structure --- easy to prove
• Greedy choice property
Coin changing problem

• 1. Prove that with coin denominations of “5, 1”, it has the greedy
choice property
• Proof:
Apply greedy choice: n = 5 + 5c + d
In a optimal solution if there is a nickel, the proof is done
If there is no nickel: n = d’=5 + d’’
Need to prove that: 1+d’’ <= d’
d’=5+d’’ > 1+d’’
• For “5, 1”, it has greedy choice property, greedy algorithm works
Coin changing problem

• 2. Prove that with coin denominations of “10, 5, 1”, it has the greedy
choice property
• Proof:
Apply greedy choice: n = 10 + 10b + 5c + d
• In a optimal solution if there is a dime, the proof is done
• If there is no dime : n = 5c’ + d’
• Since 5c’ + d’>=10
• with the conclusion of the previous slide, c’>=2
• 5c’ + d’ = 10 + 5(c’-2) + d’ and c’+d’ > 1+c’-2+d’
• it cannot be a optimal solution
• For “10, 5, 1”, it has greedy choice property, greedy algorithm works
Coin changing problem

• 3. Prove that with coin denominations of “25, 10, 5, 1”, it has the
greedy choice property
• Proof:
Apply greedy choice: n = 25 + 25a + 10b + 5c + d
• In a optimal solution if there is a quarter, the proof is done
• If there is no quarter : n = 10b’+5c’+d’
• Since 10b’+5c’+d’ >= 25
• if 25<=n<30, with the conclusion of previous slide, b’>=2, c’>=1
• 10b’+5c’+d’ = 25 + 10(b’-2) + 5(c’-1) + d’ and b’+c’+d’>1+b’-2+c’-1+d’
• it cannot be a optimal solution
• if n>=30, with the conclusion of previous slide, b’>=3
• 10b’+5c’+d’ = 25 + 10(b’-3) + 5(c’+1) + d’ and b’+c’+d’>1+b’-3+c’+1+d’
• it cannot be a optimal solution

For “25, 10, 5, 1”, it has greedy choice property, greedy algorithm works
Greedy versus dynamic programming

• The 0-1 knapsack problem is the following. A thief robbing a store

finds n items. The ith item is worth i dollars and weighs wi pounds,
where i and wi are integers. The thief wants to take as valuable a load
as possible, but he can carry at most W pounds in his knapsack, for
some integer W . Which items should he take?
• In the fractional knapsack problem, the setup is the same, but the
thief can take fractions of items, rather than having to make a binary
(0-1) choice for each item.
• Calculate value per pound of an item
Greedy Strategy for 0-1?
Minimum Spanning Tree

• Model as a graph:
» Undirected graph G = (V, E)
» Weight w(u, v) on each edge (u, v) ∈ E
» Find T ⊆ E such that
• T connects all vertices
• (T is a spanning tree)
• w(T ) = ∑ w(u, v) is minimized ( u ,v )∈T
Minimum Spanning Tree

• A spanning tree whose weight is minimum over all spanning trees is called
a
Minimum Spanning Tree, or MST
• Example:

• In this example, there is more than one MST

• Replace edge (b,c) by (a,h)
• Get a different spanning tree with the same weight
Minimum Spanning Tree

•Which edges form the Minimum Spanning Tree (MST) of the below
graph?

A
6 4
5 9
H B C

14 2
10
15
G E D
3 8
F
Minimum Spanning Tree

• MSTs satisfy the optimal substructure

property: an optimal tree is composed of
optimal subtrees
» Let T be an MST of G with an edge (u,v) in the
middle
» Removing (u,v) partitions T into two trees T1 and
T2
» Claim: T1 is an MST of G1 = (V1,E1), and T2 is an MST of G2 =
(V2,E2) ( Do V1 and V2 share vertices? Why? )
» Proof: w(T) = w(u,v) + w(T1) + w(T2)
(There can’t be a better tree than T1 or T2, or T would be suboptimal)
Some definitions

• A cut (S, V – S) of an undirected graph G =(V,E) is a partition of V

• We say that an edge (u,v) ϵ E crosses the (S, V – S) if one of its

endpoints is in S and the other is in V - S.
Some definitions

• We say that a cut respects a set A of edges if no edge in A crosses the

cut.
• An edge is a light edge crossing a cut if its weight is the minimum of any
edge crossing the cut.
• Note that there can be more than one light edge crossing a cut in the
case of ties.
• More generally, we say that an edge is a light edge satisfying a given
property if its weight is the minimum of any edge satisfying the property
• Theorem 23.1
Let G = (V,E) be a connected, undirected graph with a real-valued
weight function w defined on E. Let A be a subset of E that is included in
some minimum spanning tree for G, let (S, V – S) be any cut of G that
respects A, and let (u, v) be a light edge crossing (S, V – S). Then, edge (u,
v) is safe for A.
Proof of theorem

Except for the dashed edge (u, v), all edges shown are
in T. A is some subset of
the edges of T, but A cannot contain any edges that
cross the cut (S, V − S), since
this cut respects A. Shaded edges are the path p.
Proof of theorem (1)

Since the cut respects A, edge (x, y) is not in A.

To form T‘ from T :
• Remove (x, y). Breaks T into two components.
• Add (u, v). Reconnects.
So T‘ = T − {(x, y)} ∪ {(u, v)}.
T’ is a spanning tree.
w(T’ ) = w(T ) − w(x, y) + w(u, v)
≤ w(T) ,
since w(u, v) ≤ w(x, y). Since T is a spanning tree, w(T’) ≤ w(T ), and T is an
MST, then T’ must be an MST.
Need to show that (u, v) is safe for A:
• A ⊆ T and (x, y) ∉ A ⇒ A ⊆ T’.
• A ∪ {(u, v)} ⊆ T’ .
• Since T’ is an MST, (u, v) is safe for A.
Generic-MST

So, in GENERIC-MST:
• A is a forest containing connected components. Initially, each
component is a single vertex.
• Any safe edge merges two of these components into one. Each
component is a tree.
• Since an MST has exactly |V| − 1 edges, the for loop iterates |V| − 1
times. Equivalently, after adding |V|−1 safe edges, we.re down to just
one component.
Corollary
If C = (VC, EC) is a connected component in the forest GA = (V, A) and (u,
v) is a light edge connecting C to some other component in GA (i.e., (u,
v) is a light edge crossing the cut (VC, V − VC)), then (u, v) is safe for A.
Proof Set S = VC in the theorem.
Growing An MST
• Some properties of an MST:
 It has |V| -1 edges
 It has no cycles
 It might not be unique
• Building up the solution
» We will build a set A of
edges
» Initially, A has no edges
» As we add edges to A,
maintain a loop invariant:
• Loop invariant: A is a
subset of some MST
» Add only edges that
maintain the invariant
If A is a subset of some
MST, an edge (u, v) is
safe for A
Growing An MST
Kruskal’s Algorithm
Run the algorithm:
Kruskal() 2 19
{ 9
14 17
T = ∅; 8 25
5
for each v ∈
V MakeSet(v); 21 13 1

sort E by increasing edge weight w

for each (u,v) ∈ E (in order)
sorted if FindSet(u) ≠
T = T U {{u,v}};
FindSet(v)
Union(FindSet(u), FindSet(v));
}
Kruskal’s Algorithm
Run the algorithm:
Kruskal() 2 19
{ 9
14 17
T = ∅; 8 25
5
for each v ∈
V MakeSet(v); 21 13 1

sort E by increasing edge weight w

for each (u,v) ∈ E (in order)
sorted if FindSet(u) ≠
T = T U {{u,v}};
FindSet(v)
Union(FindSet(u), FindSet(v));
}
Kruskal’s Algorithm
Run the algorithm:
Kruskal() 2 19
{ 9
14 17
T = ∅; 8 25
5
for each v ∈
V MakeSet(v); 21 13 1?

sort E by increasing edge weight w

for each (u,v) ∈ E (in order)
sorted if FindSet(u) ≠
T = T U {{u,v}};
FindSet(v)
Union(FindSet(u), FindSet(v));
}
Kruskal’s Algorithm
Run the algorithm:
Kruskal() 2 19
{ 9
14 17
T = ∅; 8 25
5
for each v ∈
V MakeSet(v); 21 13 1

sort E by increasing edge weight w

for each (u,v) ∈ E (in order)
sorted if FindSet(u) ≠
T = T U {{u,v}};
FindSet(v)
Union(FindSet(u), FindSet(v));
}
Kruskal’s Algorithm
Run the algorithm:
Kruskal() 2? 19
{ 9
14 17
T = ∅; 8 25
5
for each v ∈
V MakeSet(v); 21 13 1

sort E by increasing edge weight w

for each (u,v) ∈ E (in order)
sorted if FindSet(u) ≠
T = T U {{u,v}};
FindSet(v)
Union(FindSet(u), FindSet(v));
}
Kruskal’s Algorithm
Run the algorithm:
Kruskal() 2 19
{ 9
14 17
T = ∅; 8 25
5
for each v ∈
V MakeSet(v); 21 13 1

sort E by increasing edge weight w

for each (u,v) ∈ E (in order)
sorted if FindSet(u) ≠
T = T U {{u,v}};
FindSet(v)
Union(FindSet(u), FindSet(v));
}
Kruskal’s Algorithm
Run the algorithm:
Kruskal() 2 19
{ 9
14 17
T = ∅; 8 25
5?
for each v ∈
V MakeSet(v); 21 13 1

sort E by increasing edge weight w

for each (u,v) ∈ E (in order)
sorted if FindSet(u) ≠
T = T U {{u,v}};
FindSet(v)
Union(FindSet(u), FindSet(v));
}
Kruskal’s Algorithm
Run the algorithm:
Kruskal() 2 19
{ 9
14 17
T = ∅; 8 25
5
for each v ∈
V MakeSet(v); 21 13 1

sort E by increasing edge weight w

for each (u,v) ∈ E (in order)
sorted if FindSet(u) ≠
T = T U {{u,v}};
FindSet(v)
Union(FindSet(u), FindSet(v));
}
Kruskal’s Algorithm
Run the algorithm:
Kruskal() 2 19
{ 9
14 17
T = ∅; 8? 25
5
for each v ∈
V MakeSet(v); 21 13 1

sort E by increasing edge weight w

for each (u,v) ∈ E (in order)
sorted if FindSet(u) ≠
T = T U {{u,v}};
FindSet(v)
Union(FindSet(u), FindSet(v));
}
Kruskal’s Algorithm
Run the algorithm:
Kruskal() 2 19
{ 9
14 17
T = ∅; 8 25
5
for each v ∈
V MakeSet(v); 21 13 1

sort E by increasing edge weight w

for each (u,v) ∈ E (in order)
sorted if FindSet(u) ≠
T = T U {{u,v}};
FindSet(v)
Union(FindSet(u), FindSet(v));
}
Kruskal’s Algorithm
Run the algorithm:
Kruskal() 2 19
{ 9?
14 17
T = ∅; 8 25
5
for each v ∈
V MakeSet(v); 21 13 1

sort E by increasing edge weight w

for each (u,v) ∈ E (in order)
sorted if FindSet(u) ≠
T = T U {{u,v}};
FindSet(v)
Union(FindSet(u), FindSet(v));
}
Kruskal’s Algorithm
Run the algorithm:
Kruskal() 2 19
{ 9
14 17
T = ∅; 8 25
5
for each v ∈
V MakeSet(v); 21 13 1

sort E by increasing edge weight w

for each (u,v) ∈ E (in order)
sorted if FindSet(u) ≠
T = T U {{u,v}};
FindSet(v)
Union(FindSet(u), FindSet(v));
}
Kruskal’s Algorithm
Run the algorithm:
Kruskal() 2 19
{ 9
14 17
T = ∅; 8 25
5
for each v ∈
V MakeSet(v); 21 13? 1

sort E by increasing edge weight w

for each (u,v) ∈ E (in order)
sorted if FindSet(u) ≠
T = T U {{u,v}};
FindSet(v)
Union(FindSet(u), FindSet(v));
}
Kruskal’s Algorithm
Run the algorithm:
Kruskal() 2 19
{ 9
14 17
T = ∅; 8 25
5
for each v ∈
V MakeSet(v); 21 13 1

sort E by increasing edge weight w

for each (u,v) ∈ E (in order)
sorted if FindSet(u) ≠
T = T U {{u,v}};
FindSet(v)
Union(FindSet(u), FindSet(v));
}
Kruskal’s Algorithm
Run the algorithm:
Kruskal() 2 19
{ 9
14? 17
T = ∅; 8 25
5
for each v ∈
V MakeSet(v); 21 13 1

sort E by increasing edge weight w

for each (u,v) ∈ E (in order)
sorted if FindSet(u) ≠
T = T U {{u,v}};
FindSet(v)
Union(FindSet(u), FindSet(v));
}
Kruskal’s Algorithm
Run the algorithm:
Kruskal() 2 19
{ 9
14 17
T = ∅; 8 25
5
for each v ∈
V MakeSet(v); 21 13 1

sort E by increasing edge weight w

for each (u,v) ∈ E (in order)
sorted if FindSet(u) ≠
T = T U {{u,v}};
FindSet(v)
Union(FindSet(u), FindSet(v));
}
Kruskal’s Algorithm
Run the algorithm:
Kruskal() 2 19
{ 9
14 17?
T = ∅; 8 25
5
for each v ∈
V MakeSet(v); 21 13 1

sort E by increasing edge weight w

for each (u,v) ∈ E (in order)
sorted if FindSet(u) ≠
T = T U {{u,v}};
FindSet(v)
Union(FindSet(u), FindSet(v));
}
Kruskal’s Algorithm
Run the algorithm:
Kruskal() 2 19?
{ 9
14 17
T = ∅; 8 25
5
for each v ∈
V MakeSet(v); 21 13 1

sort E by increasing edge weight w

for each (u,v) ∈ E (in order)
sorted if FindSet(u) ≠
T = T U {{u,v}};
FindSet(v)
Union(FindSet(u), FindSet(v));
}
Kruskal’s Algorithm
Run the algorithm:
Kruskal() 2 19
{ 9
14 17
T = ∅; 8 25
5
for each v ∈
V MakeSet(v); 21? 13 1

sort E by increasing edge weight w

for each (u,v) ∈ E (in order)
sorted if FindSet(u) ≠
T = T U {{u,v}};
FindSet(v)
Union(FindSet(u), FindSet(v));
}
Kruskal’s Algorithm
Run the algorithm:
Kruskal() 2 19
{ 9
14 17
T = ∅; 8 25?
5
for each v ∈
V MakeSet(v); 21 13 1

sort E by increasing edge weight w

for each (u,v) ∈ E (in order)
sorted if FindSet(u) ≠
T = T U {{u,v}};
FindSet(v)
Union(FindSet(u), FindSet(v));
}
Kruskal’s Algorithm
Run the algorithm:
Kruskal() 2 19
{ 9
14 17
T = ∅; 8 25
5
for each v ∈
V MakeSet(v); 21 13 1

sort E by increasing edge weight w

for each (u,v) ∈ E (in order)
sorted if FindSet(u) ≠
T = T U {{u,v}};
FindSet(v)
Union(FindSet(u), FindSet(v));
}
Kruskal’s Algorithm
Run the algorithm:
Kruskal() 2 19
{ 9
14 17
T = ∅; 8 25
5
for each v ∈
V MakeSet(v); 21 13 1

sort E by increasing edge weight w

for each (u,v) ∈ E (in order)
sorted if FindSet(u) ≠
T = T U {{u,v}};
FindSet(v)
Union(FindSet(u), FindSet(v));
}
Kruskal’s Algorithm
Kruskal’s Algorithm
Kruskal’s Algorithm

Spring 2006 Algorithm Networking Laboratory 11-37/62

Kruskal’s Algorithm

Kruskal() What will affect the running time?

{ 1 Sort
O(V) MakeSet()
T = ∅; calls O(E) FindSet()
for each v ∈ calls
V MakeSet(v); O(V) Union() calls
sort E by increasing (Exactly how many
edge weight w Union()s?)
for each (u,v) ∈ E sorted order)
(inif FindSet(u) ≠ FindSet(v)
T = T U {{u,v}};
Union(FindSet(u),
FindSet(v));
}
Prim’s Algorithm
MST-Prim(G, w, r) 6 4
Q = V[G]; 5 9
for each u ∈ Q
key[u] = ∞;
14 2
key[r] = 0; 10
15
П[r] = NULL;
while (Q not
empty) 3 8
u = ExtractMin(Q); Run on example graph
for each v ∈ < key[v])
Adj[u]
if (v ∈ Q and
w(u,v)
П[v] = u;
key[v] =
w(u,v);
Prim’s Algorithm

• MST-Prim(G, w, 6 ∞ 4
5 9
r) Q = V[G]; ∞ ∞ ∞
• for each u ∈ Q
14 2
• key[u] = ∞; 10
• key[r] = 0; 15
∞ ∞ ∞
• П[r] = NULL;
3 ∞ 8
• while (Q not
empty) Run on example graph
•u = ExtractMin(Q); key[v])

for each v ∈ Adj[u]

• if (v ∈ Q and
w(u,v) <

• П[v] = u;

• key[v] =
w(u,v);
Prim’s Algorithm

• MST-Prim(G, w, 6 ∞ 4
5 9
r) Q = V[G]; ∞ ∞ ∞
• for each u ∈ Q
14 2
• key[u] = ∞; 10
• key[r] = 0; 15
r 0 ∞ ∞
• П[r] = NULL;
3 ∞ 8
• while (Q not
empty) Pick a start vertex r
•u = ExtractMin(Q); key[v])

for each v ∈ Adj[u]

• if (v ∈ Q and
w(u,v) <
• П[v] = u;

• key[v] =
w(u,v);
Prim’s Algorithm
MST-Prim(G, w, r) 6 ∞ 4
Q = V[G]; 5 9
for each u ∈ Q
∞ ∞ ∞
key[u] = ∞; 14 2
key[r] = 0; 10
15
П[r] = NULL; u 0 ∞ ∞
while (Q not empty)
3 ∞ 8
u = ExtractMin(Q);
for each v ∈ Black vertices have been removed from Q
Adj[u]
if (v ∈ Q and w(u,v) <
key[v])
П[v] = u;
key[v] = w(u,v);
Prim’s Algorithm

• MST-Prim(G, w, 6 ∞ 4
5 9
r) Q = V[G]; ∞ ∞ ∞
• for each u ∈ Q
14 2
• key[u] = ∞; 10
• key[r] = 0; 15
u 0 ∞ ∞
• П[r] = NULL;
3 3 8
• while (Q not
empty) Black arrows indicate parent
•u = ExtractMin(Q); pointers
for each v ∈ Adj[u] key[v])
• if (v ∈ Q and
w(u,v) <
• П[v] = u;

• key[v] =
w(u,v);
Prim’s Algorithm
MST-Prim(G, w, r) 6 ∞ 4
Q = V[G]; 5 9
14 ∞ ∞
for each u ∈ Q
key[u] = ∞; 14 2
key[r] = 0; 10
15
П[r] = NULL; u 0 ∞ ∞
while (Q not empty)
3 3 8
u = ExtractMin(Q);
for each v ∈ Adj[u]
if (v ∈ Q and w(u,v) key[v])
<
П[v] = u;
key[v] = w(u,v);
Prim’s Algorithm
MST-Prim(G, w, r) 6 ∞ 4
Q = V[G]; 5 9
14 ∞ ∞
for each u ∈ Q
key[u] = ∞; 14 2
key[r] = 0; 10
15
П[r] = NULL; 0 ∞ ∞
while (Q not empty)
3 3 8
u = ExtractMin(Q); u
for each v ∈
if (v ∈ Q and w(u,v)
Adj[u] key[v])
<
П[v] = u;
key[v] = w(u,v);
Prim’s Algorithm
MST-Prim(G, w, r) 6 ∞ 4
Q = V[G]; 5 9
14 ∞ ∞
for each u ∈ Q
key[u] = ∞; 14 2
key[r] = 0; 10
15
П[r] = NULL; 0 8 ∞
while (Q not empty)
3 3 8
u = ExtractMin(Q); u
for each v ∈
if (v ∈ Q and w(u,v)
Adj[u] key[v])
<
П[v] = u;
key[v] = w(u,v);
Prim’s Algorithm
MST-Prim(G, w, r) 6 ∞ 4
Q = V[G]; 5 9
10 ∞ ∞
for each u ∈ Q
key[u] = ∞; 14 2
key[r] = 0; 10
15
П[r] = NULL; 0 8 ∞
while (Q not empty)
3 3 8
u = ExtractMin(Q); u
for each v ∈
if (v ∈ Q and w(u,v)
Adj[u] key[v])
<
П[v] = u;
key[v] = w(u,v);
Prim’s Algorithm
MST-Prim(G, w, r) 6 ∞ 4
Q = V[G]; 5 9
10 ∞ ∞
for each u ∈ Q
key[u] = ∞; 14 2
key[r] = 0; 10
15
П[r] = NULL; 0 8 ∞
while (Q not empty)
3 3 8
u = ExtractMin(Q); u
for each v ∈
if (v ∈ Q and w(u,v)
Adj[u] key[v])
<
П[v] = u;
key[v] = w(u,v);
Prim’s Algorithm
MST-Prim(G, w, r) 6 ∞ 4
Q = V[G]; 5 9
10 2 ∞
for each u ∈ Q
key[u] = ∞; 14 2
key[r] = 0; 10
15
П[r] = NULL; 0 8 ∞
while (Q not empty)
3 3 8
u = ExtractMin(Q); u
for each v ∈
if (v ∈ Q and w(u,v)
Adj[u] key[v])
<
П[v] = u;
key[v] = w(u,v);
Prim’s Algorithm
MST-Prim(G, w, r) 6 ∞ 4
Q = V[G]; 5 9
10 2 ∞
for each u ∈ Q
key[u] = ∞; 14 2
key[r] = 0; 10
15
П[r] = NULL; 0 8 15
while (Q not empty)
3 3 8
u = ExtractMin(Q); u
for each v ∈
if (v ∈ Q and w(u,v)
Adj[u] key[v])
<
П[v] = u;
key[v] = w(u,v);
Prim’s Algorithm
MST-Prim(G, w, r) ∞ u
6 4
Q = V[G]; 5 9
10 2 ∞
for each u ∈ Q
key[u] = ∞; 14 2
key[r] = 0; 10
15
П[r] = NULL; 0 8 15
while (Q not empty)
3 3 8
u = ExtractMin(Q);
for each v ∈
if (v ∈ Q and w(u,v)
Adj[u] key[v])
<
П[v] = u;
key[v] = w(u,v);
Prim’s Algorithm
MST-Prim(G, w, r) ∞ u
6 4
Q = V[G]; 5 9
10 2 9
for each u ∈ Q
key[u] = ∞; 14 2
key[r] = 0; 10
15
П[r] = NULL; 0 8 15
while (Q not empty)
3 3 8
u = ExtractMin(Q);
for each v ∈
if (v ∈ Q and w(u,v)
Adj[u] key[v])
<
П[v] = u;
key[v] = w(u,v);
Prim’s Algorithm
MST-Prim(G, w, r) 4 u
6 4
Q = V[G]; 5 9
10 2 9
for each u ∈ Q
key[u] = ∞; 14 2
key[r] = 0; 10
15
П[r] = NULL; 0 8 15
while (Q not empty)
3 3 8
u = ExtractMin(Q);
for each v ∈
if (v ∈ Q and w(u,v)
Adj[u] key[v])
<
П[v] = u;
key[v] = w(u,v);
Prim’s Algorithm
MST-Prim(G, w, r) 4 u
6 4
Q = V[G]; 5 9
5 2 9
for each u ∈ Q
key[u] = ∞; 14 2
key[r] = 0; 10
15
П[r] = NULL; 0 8 15
while (Q not empty)
3 3 8
u = ExtractMin(Q);
for each v ∈
if (v ∈ Q and w(u,v)
Adj[u] key[v])
<
П[v] = u;
key[v] = w(u,v);
Prim’s Algorithm
u
MST-Prim(G, w, r) 6 4 4
Q = V[G]; 5 9
5 2 9
for each u ∈ Q
key[u] = ∞; 14 2
key[r] = 0; 10
15
П[r] = NULL; 0 8 15
while (Q not empty)
3 3 8
u = ExtractMin(Q);
for each v ∈
if (v ∈ Q and w(u,v)
Adj[u] key[v])
<
П[v] = u;
key[v] = w(u,v);
Prim’s Algorithm
MST-Prim(G, w, r) u 4
6 4
Q = V[G]; 5 9
5 2 9
for each u ∈ Q
key[u] = ∞; 14 2
key[r] = 0; 10
15
П[r] = NULL; 0 8 15
while (Q not empty)
3 3 8
u = ExtractMin(Q);
for each v ∈
if (v ∈ Q and w(u,v)
Adj[u] key[v])
<
П[v] = u;
key[v] = w(u,v);
Prim’s Algorithm
u
MST-Prim(G, w, r) 6 4 4
Q = V[G]; 5 9
5 2 9
for each u ∈ Q
key[u] = ∞; 14 2
key[r] = 0; 10
15
П[r] = NULL; 0 8 15
while (Q not empty)
3 3 8
u = ExtractMin(Q);
for each v ∈
if (v ∈ Q and w(u,v)
Adj[u] key[v])
<
П[v] = u;
key[v] = w(u,v);
Prim’s Algorithm
MST-Prim(G, w, r) 6 4 4
Q = V[G]; 5 9
5 2 9
for each u ∈ Q
key[u] = ∞; 14 u
2
key[r] = 0; 10
15
П[r] = NULL; 0 8 15
while (Q not empty)
3 3 8
u = ExtractMin(Q);
for each v ∈
if (v ∈ Q and w(u,v)
Adj[u] key[v])
<
П[v] = u;
key[v] = w(u,v);
Prim’s Algorithm

Spring 2006 Algorithm

Networking Laboratory 11-61/62
Prim’s Algorithm

Spring 2006 Algorithm

Networking Laboratory 11-62/62
Huffman codes

• Suppose we have a 100,000-character data file that we wish to store

compactly. We observe that the characters in the file occur with the
frequencies given by the figure. That is, only 6 different characters
appear, and the character a occurs 45,000 times.

Variable length codeword

We consider here only codes in which no codeword is also a prefix of some other codeword. Such
codes are called prefix codes.
Huffman codes

• For example, with the variable-length prefix code of Figure 16.3, we

code the 3-character file abc as 0.101.100 = 0101100, where “.”
denotes concatenation.
Huffman codes

• An optimal code for a file is always represented by a full binary tree,

• we can say that if C is the alphabet from which the characters are
drawn and all character frequencies are positive, then the tree for an
optimal prefix code has exactly |C| leaves,
• one for each letter of the alphabet, and exactly |C| - 1 internal nodes
• The number of bits required to encode a file is thus
Constructing a Huffman code
Huffman Code Construction Char Freq
E 125
T 93
•Character count in text.
A 80
O 76
I 73
N 71
S 65
R 61
H 55
L 41
D 40
C 31
U 27

81
 Char Freq
E 125
T 93
Huffman Code Construction A
O
80
76
I 73
N 71
S 65
R 61
H 55
L 41
D 40
C 31
U 27

C U
31 27
82
 Char Freq
E 125
T 93
Huffman Code Construction A
O
80
76
I 73
N 71
S 65
R 61
58
H 55
L 41
D 40

C 31
U 27

58

C U
31 27
83
 Char Freq
E 125
T 93
Huffman Code Construction A
81
80
O 76
I 73
N 71
S 65
R 61
58
H 55

L 41
D 40

81

D L

40 41 58

C U
31 27
84
 Char Freq
E 125
113
Huffman Code Construction T 93
81
A 80
O 76
I 73
N 71
S 65
R 61

58
H 55

81 113

D L H

40 41 58 55

C U
31 27
85
 Char Freq
126
E 125
Huffman Code Construction T
113
93
81
A 80
O 76
I 73
N 71

S 65
R 61

81 126 113

D L R S H

40 41 61 65 58 55

C U
31 27
86
 Char Freq
144
126
Huffman Code Construction E 125
113
T 93
81
A 80
O 76

I 73
N 71

81 126 144 113

D L R S N I H

40 41 61 65 71 73 58 55

C U
31 27
87
 Char Freq
156
144
Huffman Code Construction 126
E 125
113
T 93
81

A 80
O 76

156

A O

80 76 81 126 144 113

D L R S N I H

40 41 61 65 71 73 58 55

C U
31 27
88
 Char Freq
174
156
Huffman Code Construction 144
126
E 125
113

T 93
81

156 174

A O T

80 76 81 93 126 144 113

D L R S N I H

40 41 61 65 71 73 58 55

C U
31 27
89
 Char Freq
238
174
Huffman Code Construction 156
144
126

E 125
113

156 174
238
A O T E

80 76 81 93 126 144 125 113

D L R S N I H

40 41 61 65 71 73 58 55

C U
31 27
90
 Char Freq
270
238
Huffman Code Construction 174
156

144
126

156 174 270

238
A O T E

80 76 81 93 126 144 125 113

D L R S N I H

40 41 61 65 71 73 58 55

C U
31 27
91
 Char Freq
330
270
Huffman Code Construction 238

174
156

330

156 174 270

238
A O T E

80 76 81 93 126 144 125 113

D L R S N I H

40 41 61 65 71 73 58 55

C U
31 27
92
 Char Freq
508
330
Huffman Code Construction
270
238

330 508

156 174 270

238
A O T E

80 76 81 93 126 144 125 113

D L R S N I H

40 41 61 65 71 73 58 55

C U
31 27
93
 Char Freq
838

Huffman Code Construction 508

330

838

330 508

156 174 270

238
A O T E

80 76 81 93 126 144 125 113

D L R S N I H

40 41 61 65 71 73 58 55

C U
31 27
94
Huffman Code Construction
0 Char Freq Fixed Huff
1
E 125 0000 110
T 93 0001 011
0 1 A 0 80 0010 1 000
O 76 0011 001
I 73 0100 1011
0 1 0 1 0 N 1 71 0101 0 1010 1

S 65 0110 1001
A O T E
R 61 0111 1000
0 1 0 1 0 1 0 1
H 55 1000 1111
L 41 1001 0101
D L R S N I H
D 40 1010 0100
0 1
C 31 1011 11100
U 27 1100 11101
C U
Total 838 4.00 3.62
95
Correctness of Huffman’s algorithm

Proof Idea

• Step 1: Show that this problem satisfies the greedy choice property, that is, if
a greedy choice is made by Huffman's algorithm, an optimal solution remains
possible.
• Step 2: Show that this problem has an optimal substructure property, that is,
an optimal solution to Huffman's algorithm contains optimal solution to
subproblems.
• Step 3: Conclude correctness of Huffman's algorithm using step 1 and step 2.
Greedy Choice Property

• Lemma : Let c be an alphabet in which each character c has frequency

f[c]. Let x and y be two characters in C having the lowest frequencies.
Then there exists an optimal prefix code for C in which the codewords
for x and y have the same length and differ only in the last bit.

• Proof: The idea of the proof is to take the tree T representing an

arbitrary optimal prefix code and modify it to make a tree
representing another optimal prefix code such that the characters x
and y appear as sibling leaves of maximum depth in the new tree. If
we can construct such a tree, then the codewords for x and y will
have the same length and differ only in the last bit.
Proof contd

• Let a and b be two characters that are sibling leaves of maximum

depth in T . Without loss of generality, we assume that a,freq ≤  b.freq
and x.freq ≤  y.freq. Since x.freq and y.freq are the two lowest leaf
frequencies, in order, and a.freq and b.freq are two arbitrary
frequencies, in order, we have x.freq ≤  a.freq and y.freq ≤  b.freq.
• As Figure shows, we exchange the positions in T of a and x to produce
a tree T’, and then we exchange the positions in T’ of b and y to
produce a tree T’’
Proof contd

• The cost of a tree is

• The difference in cost between T and T’ is
Optimal substructure
Proof
Proof contd