CSE 301

Combinatorial Optimization

Greedy Algorithms (Part 1)

 Greedy Algorithm
• Greedy algorithms make the choice that looks
best at the moment.
• This locally optimal choice may lead to a globally
optimal solution (i.e. an optimal solution to the
entire problem).

2
When can we use Greedy algorithms?

We can use a greedy algorithm when the following are true:

1) The greedy choice property: A globally optimal solution

can be arrived at by making a locally optimal (greedy) choice.

2) The optimal substructure property: The optimal solution

contains within its optimal solutions to subproblems.

3
 Designing Greedy Algorithms
1. Cast the optimization problem as one for which:
• we make a choice and are left with only one subproblem
to solve
2. Prove the GREEDY CHOICE
• that there is always an optimal solution to the original
problem that makes the greedy choice
3. Prove the OPTIMAL SUBSTRUCTURE:
• the greedy choice + an optimal solution to the resulting
subproblem leads to an optimal solution

4
 Example: Making Change
• Instance: amount (in cents) to return to customer
• Problem: do this using fewest number of coins
• Example:
– Assume that we have an unlimited number of coins of
various denominations:
– 1c (pennies), 5c (nickels), 10c (dimes), 25c (quarters), 1$
(loonies)
– Objective: Pay out a given sum $5.64 with the
smallest number of coins possible.

5
 The Coin Changing Problem
• Assume that we have an unlimited number of coins of various
denominations:
• 1c (pennies), 5c (nickels), 10c (dimes), 25c (quarters), 1$ (loonies)
• Objective: Pay out a given sum S with the smallest number of
coins possible.

• The greedy coin changing algorithm:

• This is a (m) algorithm where m = number of denominations.

while S > 0 do
c := value of the largest coin no larger than S;
num := S / c;
pay out num coins of value c;
S := S - num*c;

6
 Example: Making Change
• E.g.:
$5.64 = $2 +$2 + $1 +
.25 + .25 + .10 +
.01 + .01 + .01 +.01

7
 Making Change – A big problem
• Example 2: Coins are valued $.30, $.20, $.05,
$.01
– Does not have greedy-choice property, since $.40 is
best made with two $.20’s, but the greedy solution will
pick three coins (which ones?)

8
 The Fractional Knapsack Problem
• Given: A set S of n items, with each item i having
– bi - a positive benefit
– wi - a positive weight
• Goal: Choose items with maximum total benefit but with weight at
most W.
• If we are allowed to take fractional amounts, then this is the fractional
knapsack problem.
– In this case, we let xi denote the amount we take of item i

– Objective: maximize

b (x / w )
iS
i i i
– Constraint:

x
iS
i W ,0  xi  wi

9
 Example
• Given: A set S of n items, with each item i having
– bi - a positive benefit
– wi - a positive weight
• Goal: Choose items with maximum total benefit but with total weight at
most W.

“knapsack”

Solution:
Items: P
1 2 3 4 5 • 1 ml of 5
50$
Weight: 4 ml 8 ml 2 ml 6 ml 1 ml
• 2 ml of 3
Benefit: $12 $32 $40 $30 $50 10 ml 40$
Value: 3 4 20 5 50 • 6 ml of 4
($ per ml) 30$
• 1 ml of 10
2
4$
 The Fractional Knapsack Algorithm
• Greedy choice: Keep taking item with highest value (benefit to
weight ratio)
– Since  b ( x / w )  (b / w ) x
iS
i i i
iS
i i i

Algorithm fractionalKnapsack(S, W)
Input: set S of items w/ benefit bi and weight wi; max. weight W
Output: amount xi of each item i to maximize benefit w/ weight at most W

for each item i in S

xi  0
vi  bi / wi {value}
w0 {total weight}
while w < W
remove item i with highest vi
xi  min{wi , W - w}
w  w + min{wi , W - w} 11
 The Fractional Knapsack Algorithm
• Running time: Given a collection S of n items, such that each item i
has a benefit bi and weight wi, we can construct a maximum-benefit
subset of S, allowing for fractional amounts, that has a total weight W in
O(nlogn) time.
– Use heap-based priority queue to store S
– Removing the item with the highest value takes O(logn) time
– In the worst case, need to remove all items

12
 Huffman Codes

• Widely used technique for data compression

• Assume the data to be a sequence of characters

• Looking for an effective way of storing the data

• Binary character code

– Uniquely represents a character by a binary string

13
 Fixed-Length Codes

E.g.: Data file containing 100,000 characters

a b c d e f
Frequency (thousands) 45 13 12 16 9 5

• 3 bits needed

• a = 000, b = 001, c = 010, d = 011, e = 100, f = 101

• Requires: 100,000  3 = 300,000 bits

14
 Huffman Codes

• Idea:
– Use the frequencies of occurrence of characters to
build a optimal way of representing each character

a b c d e f
Frequency (thousands) 45 13 12 16 9 5

15
 Variable-Length Codes
E.g.: Data file containing 100,000 characters

a b c d e f
Frequency (thousands) 45 13 12 16 9 5

• Assign short codewords to frequent characters and

long codewords to infrequent characters
• a = 0, b = 101, c = 100, d = 111, e = 1101, f = 1100
• (45  1 + 13  3 + 12  3 + 16  3 + 9  4 + 5  4)
1,000
= 224,000 bits
16
 Prefix Codes

• Prefix codes:
– Codes for which no codeword is also a prefix of some
other codeword
– Better name would be “prefix-free codes”

• We can achieve optimal data compression using

prefix codes
– We will restrict our attention to prefix codes

17
Encoding with Binary Character Codes

• Encoding
– Concatenate the codewords representing each
character in the file

• E.g.:
– a = 0, b = 101, c = 100, d = 111, e = 1101, f = 1100

– abc = 0  101  100 = 0101100

18
Decoding with Binary Character Codes
• Prefix codes simplify decoding
– No codeword is a prefix of another  the codeword
that begins an encoded file is unambiguous
• Approach
– Identify the initial codeword
– Translate it back to the original character
– Repeat the process on the remainder of the file
• E.g.:
– a = 0, b = 101, c = 100, d = 111, e = 1101, f = 1100
– 001011101 = 0 0  101 1101 = aabe

19
 Prefix Code Representation
• Binary tree whose leaves are the given characters
• Binary codeword
– the path from the root to the character, where 0 means “go to the
left child” and 1 means “go to the right child”
• Length of the codeword
– Length of the path from root to the character leaf (depth of node)
100 100
0 1 0 1
86 14 a: 45 55
0 1 0 0 1
25 30
58 28 14
0 1 0 1
0 1 0 1 0 1
c: 12 b: 13 14 d: 16
a: 45 b: 13 c: 12 d: 16 e: 9 f: 5
0 1
f: 5 e: 9 20
 Optimal Codes
• An optimal code is always represented by a full
binary tree
– Every non-leaf has two children
– Fixed-length code is not optimal, variable-length is
• How many bits are required to encode a file?
– Let C be the alphabet of characters
– Let f(c) be the frequency of character c
– Let dT(c) be the depth of c’s leaf in the tree T
corresponding to a prefix code
B (T )  f ( c )d T ( c ) the cost of tree T
cC

21
 Constructing a Huffman Code
• A greedy algorithm that constructs an optimal prefix code
called a Huffman code
• Assume that:
– C is a set of n characters
– Each character has a frequency f(c)
– The tree T is built in a bottom up manner
• Idea: f: 5 e: 9 c: 12 b: 13 d: 16 a: 45
– Start with a set of |C| leaves
– At each step, merge the two least frequent objects: the frequency of
the new node = sum of two frequencies
– Use a min-priority queue Q, keyed on f to identify the two least
frequent objects
22
 Example
f: 5 e: 9 c: 12 b: 13 d: 16 a: 45 c: 12 b: 13 14 d: 16 a: 45
0 1
f: 5 e: 9

14 d: 16 25 a: 45 25 30 a: 45
0 1 0 1 0 1 0 1
f: 5 e: 9 c: 12 b: 13 c: 12 b: 13 14 d: 16
0 1
f: 5 e: 9

a: 45 55 0 100 1
0 1
a: 45 55
25 30 0 1
0 1 0 1
25 30
c: 12 b: 13 14 d: 16 0 1 0 1
0 1 c: 12 b: 13 d: 16
14
f: 5 e: 9 0 1
f: 5 e: 9 23
 Building a Huffman Code
Alg.: HUFFMAN(C) Running time: O(nlgn)
1. n  C 
2. Q  C O(n)
3. for i  1 to n – 1
4. do allocate a new node z
5. left[z]  x  EXTRACT-MIN(Q)
O(nlgn)
6. right[z]  y  EXTRACT-MIN(Q)
7. f[z]  f[x] + f[y]
8. INSERT (Q, z)
9. return EXTRACT-MIN(Q)
24