Chapter 21

Dynamic Programming

Learning Objectives

1. Understand the basics of dynamic programming and its approach to problem solving.

2. Learn the general dynamic programming notation.

3. Be able to use the dynamic programming approach to solve problems such as the shortest route
problem, the knapsack problem and production and inventory control problems.

4. Understand the following terms:

stages
state variables
principle of optimality
stage transformation function
return function
knapsack problem

21 - 1
Chapter 21

Solutions:

1.
Route Value Route Value
(1-2-5-8-10) 22 (1-3-6-8-10) 26
(1-2-5-9-10) 25 (1-3-6-9-10) 22
(1-2-6-8-10) 24 (1-3-7-8-10) 22
(1-2-6-9-10) 20 (1-3-7-9-10) 21
(1-2-7-8-10) 25 (1-4-5-8-10) 22
(1-2-7-9-10) 24 (1-4-5-9-10) 25
(1-3-5-8-10) 19 (1-4-6-8-10) 27
(1-3-5-9-10) 22 (1-4-6-9-10) 23

The route (1-3-5-8-10) has the smallest value and is thus the solution to the problem.

The dynamic programming approach results in fewer computations because all 16 paths from node 1
to node 10 need not be computed. For example, at node 1 we considered only 3 paths: the one from
1-2 plus the shortest path from node 2 to node 10, the one from 1-3 plus the shortest path from node
3 to node 10, and the one from 1-4 plus the shortest path from node 4 to node 10.

2. a. The numbers in the squares above each node represent the shortest route from that node to node 10.

18 8

2 7 7
11 7
10 5 8
26 19 8 9 10
8 10
1 3 10 8 10
6 17
5 5 6
21 6 6
4 11
4 9

The shortest route is given by the sequence of nodes (1-4-6-9-10).

b. The shortest route from node 4 to node 10 is given by (4-6-9-10).

c.
Route Value Route Value
(1-2-5-7-10) 32 (1-3-6-8-10) 34
(1-2-5-8-10) 36 (1-3-6-9-10) 31
(1-2-5-9-10) 28 (1-4-6-8-10) 29
(1-3-5-7-10) 31 (1-4-6-9-10) 26
(1-3-5-8-10) 35
(1-3-5-9-10) 27

See 1 above for an explanation of how the computations are reduced.

21 - 2
 Dynamic Programming

3. Use 4 stages; one for each type of cargo.

Let the state variable represent the amount of cargo space remaining.

a. In hundreds of pounds we have up to 20 units of capacity available.

Stage 1 (Cargo Type 1)

x1 0 1 2 d1* f1(x1) x0
0-7 0 − − 0 0 0-7
8-15 0 22 − 1 22 0-7
16-20 0 22 44 2 44 0-4

Stage 2 (Cargo Type 2)

x2 0 1 2 d 2* f2(x2) x1
0-4 0 − − 0 0 0-4
5-7 0 12 − 1 12 0-2
8-9 22 12 − 0 22 8-9
10-12 22 12 24 2 24 0-2
13-15 22 34 24 1 34 8-10
16-17 44 34 24 0 44 16-17
18-20 44 34 46 2 46 8-10

Stage 3 (Cargo Type 3)

21 - 3
Chapter 21

x3 0 1 2 3 4 d 3* f3(x3) x2
0-2 0 − − − − 0 0 0-2
3-4 0 7 − − − 1 7 0-1
5 12 7 − − − 0 12 5
6-7 12 7 14 − − 2 14 0-1
8 22 19 14 − − 0 22 8
9 22 19 14 21 − 0 22 9
10 24 19 14 21 − 0 24 10
11 22 29 26 21 − 1 29 8
12 24 29 26 21 28 1 29 9
13 34 31 26 21 28 0 34 13
14-15 34 31 36 33 28 2 36 8-9
16 44 41 38 33 28 0 44 16
17 44 41 38 43 40 0 44 17
18 46 41 38 43 40 0 46 18
19 46 51 48 45 40 1 51 16
20 46 51 48 45 50 1 51 17

Stage 4 (Cargo Type 4)

x4 0 1 2 3 d 4* f4(x4) x3
20 51 49 50 45 0 51 20

Tracing back through the tables we find

State Variable Optimal State Variable

Stage Entering Decision Leaving
4 20 0 20
3 20 1 17
2 17 0 17
1 17 2 1

Load 1 unit of cargo type 3 and 2 units of cargo type 1 for a total return of $5100.

b. Only the calculations for stage 4 need to be repeated; the entering value for the state variable is 18.

x4 0 1 2 3 d 4* f4(x4) x3
18 46 47 42 38 1 47 16

Optimal solution: d4 = 1, d3 = 0, d2 = 0, d1 = 2

21 - 4
 Dynamic Programming

Value = 47

4. a. There are two optimal solutions each yielding a total profit of 186.

# of Employees Return

Activity 1 3 44
Activity 2 2 48
Activity 3 0 46
Activity 4 3 48
186

# of Employees Return

Activity 1 2 37
Activity 2 3 55
Activity 3 0 46
Activity 4 3 48
186

b.

# of Employees Return

Activity 1 1 30
Activity 2 2 48
Activity 3 0 46
Activity 4 3 48
172

5. a. Set up a stage for each possible length the log can be cut into: Stage 1 - 16 foot lengths, Stage 2 - 11
foot lengths, Stage 3 - 7 foot lengths, and Stage 4 - 3 foot lengths.

Stage 1

d1
x1 0 1 d1* f1(x1) x0
0-15 0 − 0 0 x1
16-20 0 8 1 8 x1-16

21 - 5
Chapter 21

Stage 2

d2
x2 0 1 d 2* f2(x2) x1
0-10 0 − 0 0 x2
11-15 0 5 1 5 x2-11
16-20 8 5 0 8 x2

Stage 3

d3
x3 0 1 2 d 3* f3(x3) x2
0-6 0 − − 0 0 x3
7-10 0 3 − 1 3 x3-7
11-13 5 3 − 0 5 x3
14-15 5 3 6 2 6 x3-14
16-17 8 3 6 0 8 x3
18-20 8 8 8 1 8 x3-7

Stage 4

d4
x4 0 1 2 3 4 5 6 d 4* f4(x4) x3
20 8      6 1 9 17

Tracing back through the tableau, we see that our optimal decisions are

d 4* = 1, d 3* = 0,

d 2* = 0, d1* = 1.

The total return per log when this pattern is used is $9.

b. We simply create a stage for every length.

21 - 6
 Dynamic Programming

6. a. In the network representation below each node represents the completion of an assignment. The
numbers above the arcs coming into the node represent the time it takes to carry out the assignment.

Once the network is set up we can solve as we did in section 18.1 for the shortest route problem.
The optimal sequence of assignments is A-G-J-M. The total training period for this solution is 30
months.

b. He should choose H. The training program can then be completed in 17 months.

7. Let each stage correspond to a store. Let the state variable xn represent the number of golf balls
remaining to allocate at store n and dn represent the number allocated to store n.

Stage 1 (Store 1)

d1
x1 0 100 200 300 400 500 d1* f1(x1) x0
0 0 − − − − − 0 0 0
100 0 600 − − − − 100 600 0
200 0 600 1100 − − − 200 1100 0
300 0 600 1100 1550 − − 300 1550 0
400 0 600 1100 1550 1700 − 400 1700 0
500 0 600 1100 1550 1700 1800 500 1800 0

21 - 7
Chapter 21

Stage 2 (Store 2)

d2
x2 0 100 200 300 400 500 d 2* f2(x2) x1
0 0 − − − − − 0 0 0
100 600 500 − − − − 100 600 100
200 1100 1100 1200 − − − 200 1200 0
300 1550 1600 1800 1700 − − 200 1800 100
400 1700 2050 2300 2300 2000 − 200 or 300 2300 200 or 100
500 1800 2200 2750 2800 2600 2100 300 2800 200

Stage 3 (Store 3)

d3
x3 0 100 200 300 400 500 d 3* f3(x3) x2
500 2800 2850 2900 2700 2450 1950 200 2900 300

Robin should ship 200 dozen to store 3, 200 dozen to store 2 and 100 dozen to store 1. He can
expect a profit of $2900.

8. a. Let each of the media represent a stage in the dynamic programming formulation. Further, let the
state variable, xn, represent the amount of budget remaining with n stages to go and the decision
variable, dn, the amount of the budget allocated to media n.

Stage 1 (Internet)

d1
x1 0 1 2 3 4 5 6 7 8 d1* f1(x1) x0
0 0 − − − − − − − − 0 0 0
1 0 24 − − − − − − − 1 24 0
2 0 24 37 − − − − − − 2 37 0
3 0 24 37 46 − − − − − 3 46 0
4 0 24 37 46 59 − − − − 4 59 0
5 0 24 37 46 59 72 − − − 5 72 0
6 0 24 37 46 59 72 80 − − 6 80 0
7 0 24 37 46 59 72 80 82 − 7 82 0
8 0 24 37 46 59 72 80 82 82 7 or 8 82 0 or 1

Stage 2 (Print)

21 - 8
 Dynamic Programming

d2
x2 0 1 2 3 4 5 6 7 8 d 2* f2(x2) x1
0 0 − − − − − − − − 0 0 0
1 24 15 − − − − − − − 0 24 1
2 37 39 55 − − − − − − 2 55 0
3 46 52 79 70 − − − − − 2 79 1
4 59 61 92 94 75 − − − − 3 94 1
5 72 74 101 107 99 90 − − − 3 107 2
6 80 87 114 116 112 114 95 − − 3 116 3
7 82 95 127 129 121 127 119 95 − 3 129 4
8 82 97 135 142 134 136 132 119 95 3 142 5

Stage 3 (Radio)

d3
x3 0 1 2 3 4 5 6 7 8 d 3* f3(x3) x2
0 0 − − − − − − − − 0 0 0
1 24 20 − − − − − − − 0 24 1
2 55 44 30 − − − − − − 0 55 2
3 79 75 54 45 − − − − − 0 79 3
4 94 99 85 69 55 − − − − 1 99 3
5 107 114 109 100 79 60 − − − 1 114 4
6 116 127 124 124 110 84 62 − − 1 127 5
7 129 136 137 139 134 115 86 63 − 3 139 4
8 142 149 146 152 149 139 117 87 63 3 152 5

Stage 4 (Television)

d4
x4 0 1 2 3 4 5 6 7 8 d 4* f4(x4) x3
8 152 159 167 169 164 149 125 94 70 3 169 5

Tracing back through the tables, we find the optimal decision is

d4 = 3 x3 = x4 - d 4 = 8 - 3 = 5
d3 = 1 x2 = x3 - d 3 = 5 - 1 = 4
d2 = 3 x1 = x2 - d 2 = 4 - 3 = 1
d1 = 1 x0 = x1 - d 1 = 1 - 1 = 0

This gives the agency a maximum exposure of 169.

b. We simply redo the calculations for stage 4 with x4 = 6.

21 - 9
Chapter 21

d4
x4 0 1 2 3 4 5 6 d 4* f4(x4) x3
6 127 134 139 134 120 94 70 2 139 4

Tracing back through the tables, we find a new optimal solution of

d4 = 2 x3 = x4 - d 4 = 6 - 2 = 4
d3 = 1 x2 = x3 - d 3 = 4 - 1 = 3
d2 = 2 x1 = x2 - d 2 = 3 - 2 = 1
d1 = 1 x0 = x1 - d 1 = 1 - 1 = 0

This gives the agency a maximum value of 139.

c. We can simply return to the stage 3 table and read off the answers.

If $8,000 was available, the optimal decisions are

d3 = 3
d2 = 3
d1 = 2.

The maximum exposure value is 152.

If $6,000 was available, the optimal decisions are

d3 = 1
d2 = 3
d1 = 2.

The maximum exposure is 127.

9. a.

21 - 10
 Dynamic Programming

d3 d2 d1 r1(d1) + r2(d2) + r3(d3)

100 100 0 295
100 100 100 405
100 100 200 595
100 100 300 695
100 100 400 720
100 300 0 575
100 300 100 685
100 300 200 875
100 300 300 975
100 300 400 1000
100 500 0 825
100 500 100 935
100 500 200 1125
100 500 300 1225
100 500 400 1250
100 600 0 875
100 600 100 985
100 600 200 1175
100 600 300 1275
100 800 0 1150
100 800 100 1260
500 100 0 820
500 100 100 930
500 100 200 1120
500 100 300 1220
500 100 400 1245
500 300 0 1100
500 300 100 1210
500 300 200 1400
500 500 0 1350

The optimal solution is d1 = 200, d2 = 300, d3 = 500.

b. Stage 1

d1
x1 0 100 200 300 400 d1* f1(x1) x0
0 0 − − − − 0 0 0
100 0 110 − − − 100 110 0
200 0 110 300 − − 200 300 0
300 0 110 300 400 − 300 400 0

Stage 2

d2

21 - 11
Chapter 21

x2 0 300 500 600 800 d 2* f2(x2) x1

500 545 700 650 − − 300 700 200
900 545 825 1075 1100 1085 600 1100 300

Stage 3

d3
x3 100 500 d 3* f3(x3) x2
1000 1275 1400 500 1400 500

Tracing back through the tableaus, we see we get the same solution as in (a) with much less effort.

10. The optimal production schedule is given below.

Beginning Ending
Month Inventory Production Inventory
1 10 20 10
2 10 20 0
3 0 30 0

21 - 12