ON A ROUTING PROBLEM

Author(s): RICHARD BELLMAN

Source: Quarterly of Applied Mathematics , APRIL, 1958, Vol. 16, No. 1 (APRIL, 1958),
pp. 87-90
Published by: Brown University

Stable URL: https://www.jstor.org/stable/43634538

REFERENCES
Linked references are available on JSTOR for this article:
https://www.jstor.org/stable/43634538?seq=1&cid=pdf-
reference#references_tab_contents
You may need to log in to JSTOR to access the linked references.

JSTOR is a not-for-profit service that helps scholars, researchers, and students discover, use, and build upon a wide
range of content in a trusted digital archive. We use information technology and tools to increase productivity and
facilitate new forms of scholarship. For more information about JSTOR, please contact [email protected].

Your use of the JSTOR archive indicates your acceptance of the Terms & Conditions of Use, available at
https://about.jstor.org/terms

is collaborating with JSTOR to digitize, preserve and extend access to Quarterly of Applied
Mathematics

This content downloaded from

88.236.104.59 on Tue, 19 Nov 2024 12:40:02 UTC
All use subject to https://about.jstor.org/terms
 1958] RICHARD BELLMAN 87

ON A ROUTING PROBLEM*

By RICHARD BELLMAN ( The RAND Corporation)

Summary. Given a set of N cities, with every two linked by a road, an

required to traverse these roads, we wish to determine the path from one
another given city which minimizes the travel time. The times are not d
portional to the distances due to varying quality of roads and varying
traffic.
The functional equation technique of dynamic programming, combined with approxi-
mation in policy space, yields an iterative algorithm which converges after at most
(N - 1) iterations.
1. Introduction. The problem we wish to treat is a combinatorial one involving
the determination of an optimal route from one point to another. These problems are
usually difficult when we allow a continuum, and when we admit only a discrete set
of paths, as we shall do below, they are notoriously so.
The purpose of this paper is to show that the functional equation technique of
dynamic programming, [1, 2], combined with the concept of approximation in policy
space, yields a method of successive approximations which is readily accessible to either
hand or machine computation for problems of realistic magnitude. The methoa is dis-
tinguished by the fact that it is a method of exhaustion, i.e. it converges after a finite
number of iterations, bounded in advance.
2. Formulation. Consider a set of N cities, numbered in some arbitrary fashion
from 1 to N, with every two linked by a direct road. The time required to travel from
i to j is not directly proportional to the distance between i and ;, due to road conditions
and traffic. Given the matrix T = (¿tí), not necessarily symmetric, where tu is the
time required to travel from i to j, we wish to trace a path between 1 and N which
consumes minimum time.
Since there are only a finite number of paths available, the problem reduces to
choosing the smallest from a finite set of numbers. This direct, or enumerative, approach
is impossible to execute, however, for values of N of the order of magnitude of 20.
We shall construct a search technique which greatly reduces the time required to
find minimal paths.
3. Functional equation approach. Let us now introduce a dynamic programming
approach. Let
fi - the time required to travel from i to N, i = 1, 2, • • • , N - 1,
using an optimal policy, (3.1)
with fx = 0.
Employing the principle of optimality, we se
of equations
U = Min [t„ + /,], i = 1,2, ... ,N - 1,
,vť (3.2)
Ín - 0.

^Received January 30, 1957.

This content downloaded from

88.236.104.59 on Tue, 19 Nov 2024 12:40:02 UTC
All use subject to https://about.jstor.org/terms
 88 NOTES [Vol. XVI, No. 1

This system differs from

in that we do not have a re
4. Uniqueness. Let us sho
in (3.2).
Assume that {/<} and [ř' } are two solutions, with ft, = F# = 0, and let k be an
index for which /» - Fk achieves it maximum. Then

U = Min [ť„ + /,] (4.1)

Fh = Min [ť*, 4- F,].

Let the minimum in the first equation be as

It is clear, since > 0 for all i, j, that r ^ h
inequalities:

fk = tkr + fr < tk* 4" ft j ^

Fk - tk9 F Ě < tkr + Fr .
These lead to

lĒ - ft < I. - F. (1 3)
k I. - F,.

Since k was an index where fk - Fk achieved its maximum, we must have

U - Fk = 1, - F. , (4.4)
which can only be true if in (2) we have

fk = tka + f9 . (4.5)

Now repeat this procedure for the pair {/

ment, that there must be another pair
Furthermore, p ť* s, and p ^ ki since we h

fk = tk, + t,p + fv . (4.6)

Proceeding in this way, we exhaust the set of po

result that one of the terms in the continued e
Hence /, = F{ for i = 1, 2, • • • , N - 1.
5. Approximation in policy space. Let us now tur
an algorithm for obtaining the solution of the sys
of successive approximations. We choose an init
iteratively, setting

fik+l) = Min (*„ + /n, i = 1, 2, - • .

(5.1)
/rn = o,
for k = 0, 1, 2, • • • , .
The choice of {/,-0)} seems to require some care. Let us then invoke the concept of
approximation in policy space in order to obtain a sequence which is monotone increasing.

This content downloaded from

88.236.104.59 on Tue, 19 Nov 2024 12:40:02 UTC
All use subject to https://about.jstor.org/terms
 1958] RICHARD BELLMAN 89

Perhaps the simplest policy

Define

iī0) = ti!f, i = 1,2, ••• ,N. (5.2)

It follows that /,(I) as defined by
/<" = Min [U, + /H, i = 1, 2, • • • , N - 1
,Vi (5.3)
Û r" - 0,
satisfies the inequality
/!" < fr, i = l,2,---,N. (5.4)
This inequality is immediate when we realize that
time for a path with at most one stop. It follows then
{f'k) } as defined by (5.1), with /-0) as in (5.2), satisfy t
/.a+,) < fi", i = k = 0, 1, 2, • • • (5.5)
It is important to note that there are many other policies we
monotone convergence.*
It follows that
lim = ft , i = 1,2, ••• ,N, (5.6)
h - >oo

furnishing a solution to (3.2).

It is clear from the physical interpretation of this iterative scheme that at most
(¿V - 1) iterations are required for the sequence to converge to the solution.
6. Computational aspects. It is easily seen that the iterative scheme discussed above
is a feasible method for either hand or machine computation for values of N of the
order of magnitude of 50 or lOO.f
For each ¿, we require only the column j = 1,2, • • • , N of the matrix T . Hence,
the memory requirement for a digital computer is small.
7. Monotone increasing convergence. Turning back to (3.2), let us consider the
sequence of approximations defined by
f¡0) = Min ti i , i = 1,2, ... ,N = 1,
jV»

h 4i0) - - o, í' (7
4i0) - o, í'

/ru = Min [<ťí + /H, i = 1,

îV»

/ru = o.
It is clear that > /,<l) • It is not, however, obvious that this method yield
uniformly bounded sequence. To establish this, let us show, inductively, that
ñk)<U, i = 1,2, ••• ,N, k = 0, 1,2, ••• , (7.2)
where {/, ) is the solution of (3.2).

*1 owe this choice of an initial policy to F. Haight.

t Added in proof ( December 1957): After this paper was written, the author was inform
Woodbury and George Dantzig that the particular iterative scheme discussed in Sec. 5
tained by them from first principles.

This content downloaded from

88.236.104.59 on Tue, 19 Nov 2024 12:40:02 UTC
All use subject to https://about.jstor.org/terms
 00 NOTES [Vol. XVI, No. 1

The inequality is certainly t

have

/<4+1) = Min [tti +

i+i ņņ
< u .

It. follows that the sequence [f'h) } converges to I/, } as k - ► ® , furnishing the desired
monotone convergence. Once again, only a finito number of iterations will be required.
It is to bo expected that the first method will converge more rapidly.

Referencia

1. It. JMlman, Dynamic Programming , Princeton tJhlvrrflity Press, 1957

2. K. Bellman, The Theory of Dynamic Programming , J lull. Amer. Math. Soc., 60, 503-

ON THE DAMPED OSCILLATIONS EQUATION

WITH VARIABLE COEFFICIENTS*

By E. V. L AITONE (University of Californiat Berkeley)

A useful upper bound can be given for the oscillatory solutions of the seco
linear differential equation with continuous differentiate coefficients

u"(t) + r>{t)u'(t) + q(t)u(t) = 0 (1)

which commonly occurs in vibration studies and airplane or m
The solutions of Eq. (1) are oscillatory whenever

<f>(t) = (q - p2/4 - pV 2) è m2 > 0 (2)

see Bellman [1] or Kamke [2], and for this case it will be shown that as long
tonic the solutions are bounded in the following manner:

I «(*) I è ^ {</>(0)u(0)J + [u'(0) + P(0)U(0)/2]>r exp{~10

This result can be derived by introducing the unique transformation,
[1] or Kamke [2]

u = v exp (- J P/2 dtj

which preserves the same zeros in the oscillatory solutions. Then Eq. (1) is

f"(0 + *(0K0 - 0, (5)

•Received February 5, 1957.

This content downloaded from

88.236.104.59 on Tue, 19 Nov 2024 12:40:02 UTC
All use subject to https://about.jstor.org/terms