13

Dynamic Programming
Repetition is the only form of permanence that nature can achieve"

13 1. INTRODUCTION
Many decision-making problems involve a process that takes place in several stages
(multistage process) in such a way that at each stage, the process is dependent on the
strategy chosen. Such type of problems are called Dynamic Programming Problems
(D.P.P.). Thus dynamic programming is concerned with the theory of multistage decision
pfocess, i.e., the process in which a sequence of interrelated decisions has to be made.
Mathematically, a D.P.P. is a decision-making problem in n-variables, the problem being
sub-divided into n
sub-problems (segments) each sub-problem being a
decision-making
problem in one variable only. The solution to a D.P.P. is achieved sequentially starting
from one (initial) stage to the next till the final stage is reached

Principle of Optimality
It may be interesting to note that the concept of dynamic programming is largely based
upon the principle of optimality due to Bellman, viz.,
"An optimal policy has the property that whatever the initial state and initial decision
are, the remaining decisions must constitute an optimal policy with regard to the state
resulting from the first decision."
The principle of optimality implies that given the initial state of a system, an optimal
policy for the subsequent stages does not depend upon the policy adopted at the preceding
stages. That is, the effect of a current policy decision on any of the policy decisions of the
preceding stages need not be taken into account at all. It is usually referred to as the
Markovian property of dynamic programming.
13: 2. THE RECURSIVE EQUATION APPROACH
There exist different approaches to solve a D.P.P. To illustrate the one called
'recursive equation approach', we consider the following decision-making situation
There are n machines each of which can perform two different kinds of work. If z
machines work on the first kind of work, commodities worth g (z) are produced and if they
worth h (z) are produced. It is also known
work on the second kind of work, commodities
some imachines get partly inoperative. For the
to the company that after completing a job,
the second kind of work, b (t)
first kind of work. a (z) machines are leftover whereas for
what policy should be adopted for
machines are leftover. If k jobs are to be performed
the total value of the goods produced is maximized?
producing goods in such a way that
Each such problem
We subdivide this problem into several decision-making problems.
number of machines. In the
IS an allocation problem of allocating a job to the optimum
use. Let be the number
of
machines are available for x,
Initial(first) stage all the n machines assigned
to job Z.

machines assigned to job / and

n * be the number of
-

when we

denote the rotal optimal value

of the produced goods
Clearly 0sx Sn. Let fy (n)
 OCARCH
Start with n machines and work in N stages, at each stage determining a se
optimum.
Thus
quential
n ) = max. ls (xi) +h 1). where y1=n-X.
0SSn
Now consider stage 2. The number of machines, that are now available in ora.
" a Gi)+b01). From the principle of optimality, given the curent stage, the optim order,
alternative for the current stage is obtained by optimizing the sum of (a) the optim.
value of the goods produced at all previously considered stages, and (b) the value of go
produced at the current stage. 2oods
Thus, if x machines are allotted a particular job andni -2 some other one in stage 2,
then the value of the
goods produced in this stage is g (z2) + h O), where y =n -x The
optimum allocation at stage 2, therefore,
corresponds to
Ja )max fg (a) +h o) + h la m)+b o1)}. yi n- =

0SxSn
or, in general,
h n) =max, {le ()+h ()] f la (x) +b ())},
+ y =
n-
By similar argument, we have
Sa (n) =max. {ls x)+ h 0)] +fhla (x) +b o]}. y = n-x
0Sx Sn
and
Sn) =max {ls )+h ()) +S- [a (x) +b O)]}. y =n- x, k>1
0 SxSn
This is a recurrence equation for ff s.
This recurrence relation connects the optimal decisions function for
problem with the optimal decision function for the (N- 1) stage the N-stage
sub-problem.
13 3. CHARACTERISTICS OF DYNAMIC PROGRAMMING
The basic features which characterize the
dynamic programming problem are as
follows
(a) The problem can be subdivided into stages with a
policy decision required at each
stage. A stage is a device to sequence the decisions. That is, it
sub-problems such that an optimal solution to the problem decomposes
a problem into
can be obtained from the
optimal solutions to the sub-problems.
(b) Every stage consists of a number of states associated with it. The states are the
different possible conditions in which the system may find itself at that stage of the
problem.
(c) Decision at each stage converts the current stage into state associated with the next
stage.
(d) The state of the system at a stage is described by a set of variables, called state
variables.
(e) When the current state is known, an optimal policy for the remaining stages s
independent of the policy of the previous ones.
T o identify the optimum policy for each state of the system, a recursive
formulated with equation
n stages
remaining, given the optimal policy for each state with (n-)
stages left.
8) Using recursive equation approach each time the solution procedure mOv
backward stage by
stage for obtaining the optimum policy of each state for that particu
stage, till it attains the optimum
policy beginning at the initial stage.
DYNAMIC PROGRAMMING 249

13 4. DYNAMIC PROGRAMMING ALGORITHM

The computational procedure for solving a problem by dynamic programming
approach can be summarized in the following steps
Step 1. Identify the decision variables and specify objective function to be optimized
under certain limitations, if any.
Step 2. Decompose (or divide) the given of smaller
problem into a number

sub-problems (or stages). Identify the state variables at each stage and write down the
transformation function as a function of the state variables and decision variables at the
next stage.
Step 3. Write down a general recursive relationship for computing the optimal policy.
Decide whether forward or backward method is to follow to solve the problem.
Step4. Construct appropriate stages to show the required values of the return function
at each stage.
Step 5. Determine the overall optimal policy or decisions and its value at each stage.
There may be more than one such optimal policy. )
RemarksI. Generally the solution of a recursive equation involves two types of computations, according as
the svstem is continuous or discrete. In the first case the optimal decision at each stage is obtained by using the
usual classical methods of optimization. In the second case, a tabular computational scheme is followed.
2. If the dynamic programming problem is solved by using the recursive equation stating from the firs
throughthe last stage, i.e., obtaining the sequence f his . the computation involved is called the
If the recursive equation formulated in a different way so as to obtain the
forward computational procedure.
>Si. then the computation is known as the backward computational procedure. )
sequenceN IN-1 ...