Visit ebookfinal.

com to download the full version and

explore more ebooks or textbooks

Approximate Dynamic Programming Solving the Curses

of Dimensionality Second Edition Warren B.
Powell(Auth.)

_____ Click the link below to download _____

https://ebookfinal.com/download/approximate-dynamic-
programming-solving-the-curses-of-dimensionality-second-
edition-warren-b-powellauth/

Explore and download more ebooks or textbook at ebookfinal.com

Here are some recommended products that we believe you will be
interested in. You can click the link to download.

Programming and Problem Solving with C 3rd Edition Edition

Nell B. Dale

https://ebookfinal.com/download/programming-and-problem-solving-
with-c-3rd-edition-edition-nell-b-dale/

Problem Solving with C The Object of Programming Fourth

Edition Walter J. Savitch

https://ebookfinal.com/download/problem-solving-with-c-the-object-of-
programming-fourth-edition-walter-j-savitch/

Vagabond Vol 29 29 Inoue

https://ebookfinal.com/download/vagabond-vol-29-29-inoue/

Approximate Calculation of Integrals V. I. Krylov

https://ebookfinal.com/download/approximate-calculation-of-integrals-
v-i-krylov/
Hacker s Delight Second Edition Henry S. Warren

https://ebookfinal.com/download/hacker-s-delight-second-edition-henry-
s-warren/

The practical anarchist writings of Josiah Warren 1st ed

Edition Warren

https://ebookfinal.com/download/the-practical-anarchist-writings-of-
josiah-warren-1st-ed-edition-warren/

Java An Introduction to Problem Solving and Programming

7th Edition Walter Savitch

https://ebookfinal.com/download/java-an-introduction-to-problem-
solving-and-programming-7th-edition-walter-savitch/

The Shadow of Sparta 1st Edition Anton Powell

https://ebookfinal.com/download/the-shadow-of-sparta-1st-edition-
anton-powell/

Crusade for Justice The Autobiography of Ida B Wells

Second Edition Ida B. Wells (Editor)

https://ebookfinal.com/download/crusade-for-justice-the-autobiography-
of-ida-b-wells-second-edition-ida-b-wells-editor/
Approximate Dynamic Programming Solving the Curses
of Dimensionality Second Edition Warren B.
Powell(Auth.) Digital Instant Download
Author(s): Warren B. Powell(auth.), Walter A. Shewhart, Samuel S.
Wilks(eds.)
ISBN(s): 9781118029176, 1118029178
File Details: PDF, 7.71 MB
Year: 2011
Language: english
Approximate Dynamic
Programming
Approximate Dynamic
Programming
Solving the Curses of Dimensionality

Second Edition

Warren B. Powell
Princeton University
The Department of Operations Research and Financial Engineering
Princeton, NJ

A JOHN WILEY & SONS, INC., PUBLICATION

Copyright © 2011 by John Wiley & Sons, Inc. All rights reserved

Published by John Wiley & Sons, Inc., Hoboken, New Jersey

Published simultaneously in Canada

No part of this publication may be reproduced, stored in a retrieval system, or transmitted in any form
or by any means, electronic, mechanical, photocopying, recording, scanning, or otherwise, except as
permitted under Section 107 or 108 of the 1976 United States Copyright Act, without either the prior
written permission of the Publisher, or authorization through payment of the appropriate per-copy fee
to the Copyright Clearance Center, Inc., 222 Rosewood Drive, Danvers, MA 01923, (978) 750-8400,
fax (978) 750-4470, or on the web at www.copyright.com. Requests to the Publisher for permission
should be addressed to the Permissions Department, John Wiley & Sons, Inc., 111 River Street,
Hoboken, NJ 07030, (201) 748-6011, fax (201) 748-6008, or online at http://www.wiley.com/go/
permission.

Limit of Liability/Disclaimer of Warranty: While the publisher and author have used their best efforts
in preparing this book, they make no representations or warranties with respect to the accuracy or
completeness of the contents of this book and specifically disclaim any implied warranties of
merchantability or fitness for a particular purpose. No warranty may be created or extended by sales
representatives or written sales materials. The advice and strategies contained herein may not be
suitable for your situation. You should consult with a professional where appropriate. Neither the
publisher nor author shall be liable for any loss of profit or any other commercial damages, including
but not limited to special, incidental, consequential, or other damages.

For general information on our other products and services or for technical support, please contact our
Customer Care Department within the United States at (800) 762-2974, outside the United States at
(317) 572-3993 or fax (317) 572-4002.

Wiley also publishes its books in a variety of electronic formats. Some content that appears in print
may not be available in electronic formats. For more information about Wiley products, visit our web
site at www.wiley.com.

Library of Congress Cataloging-in-Publication Data:

Powell, Warren B., 1955–

Approximate dynamic programming : solving the curses of dimensionality / Warren B. Powell.
– 2nd ed.
p. cm.
Includes bibliographical references and index.
ISBN 978-0-470-60445-8 (cloth)
1. Dynamic programming. I. Title.
T57.83.P76 2011
519.7 03–dc22
2010047227

Printed in the United States of America

oBook ISBN: 978-1-118-02917-6

ePDF ISBN: 978-1-118-02915-2
ePub ISBN: 978-1-118-02916-9

10 9 8 7 6 5 4 3 2 1
 Contents

Preface to the Second Edition xi

Preface to the First Edition xv

Acknowledgments xvii

1 The Challenges of Dynamic Programming 1

1.1 A Dynamic Programming Example: A Shortest Path
Problem, 2
1.2 The Three Curses of Dimensionality, 3
1.3 Some Real Applications, 6
1.4 Problem Classes, 11
1.5 The Many Dialects of Dynamic Programming, 15
1.6 What Is New in This Book?, 17
1.7 Pedagogy, 19
1.8 Bibliographic Notes, 22

2 Some Illustrative Models 25

2.1 Deterministic Problems, 26
2.2 Stochastic Problems, 31
2.3 Information Acquisition Problems, 47
2.4 A Simple Modeling Framework for Dynamic Programs, 50
2.5 Bibliographic Notes, 54
Problems, 54

3 Introduction to Markov Decision Processes 57

3.1 The Optimality Equations, 58
3.2 Finite Horizon Problems, 65

v
vi contents

3.3 Infinite Horizon Problems, 66

3.4 Value Iteration, 68
3.5 Policy Iteration, 74
3.6 Hybrid Value-Policy Iteration, 75
3.7 Average Reward Dynamic Programming, 76
3.8 The Linear Programming Method for Dynamic
Programs, 77
3.9 Monotone Policies*, 78
3.10 Why Does It Work?**, 84
3.11 Bibliographic Notes, 103
Problems, 103

4 Introduction to Approximate Dynamic Programming 111

4.1 The Three Curses of Dimensionality (Revisited), 112
4.2 The Basic Idea, 114
4.3 Q-Learning and SARSA, 122
4.4 Real-Time Dynamic Programming, 126
4.5 Approximate Value Iteration, 127
4.6 The Post-Decision State Variable, 129
4.7 Low-Dimensional Representations of Value Functions, 144
4.8 So Just What Is Approximate Dynamic Programming?, 146
4.9 Experimental Issues, 149
4.10 But Does It Work?, 155
4.11 Bibliographic Notes, 156
Problems, 158

5 Modeling Dynamic Programs 167

5.1 Notational Style, 169
5.2 Modeling Time, 170
5.3 Modeling Resources, 174
5.4 The States of Our System, 178
5.5 Modeling Decisions, 187
5.6 The Exogenous Information Process, 189
5.7 The Transition Function, 198
5.8 The Objective Function, 206
5.9 A Measure-Theoretic View of Information**, 211
5.10 Bibliographic Notes, 213
Problems, 214
contents vii

6 Policies 221
6.1 Myopic Policies, 224
6.2 Lookahead Policies, 224
6.3 Policy Function Approximations, 232
6.4 Value Function Approximations, 235
6.5 Hybrid Strategies, 239
6.6 Randomized Policies, 242
6.7 How to Choose a Policy?, 244
6.8 Bibliographic Notes, 247
Problems, 247

7 Policy Search 249

7.1 Background, 250
7.2 Gradient Search, 253
7.3 Direct Policy Search for Finite Alternatives, 256
7.4 The Knowledge Gradient Algorithm for Discrete
Alternatives, 262
7.5 Simulation Optimization, 270
7.6 Why Does It Work?**, 274
7.7 Bibliographic Notes, 285
Problems, 286

8 Approximating Value Functions 289

8.1 Lookup Tables and Aggregation, 290
8.2 Parametric Models, 304
8.3 Regression Variations, 314
8.4 Nonparametric Models, 316
8.5 Approximations and the Curse of Dimensionality, 325
8.6 Why Does It Work?**, 328
8.7 Bibliographic Notes, 333
Problems, 334

9 Learning Value Function Approximations 337

9.1 Sampling the Value of a Policy, 337
9.2 Stochastic Approximation Methods, 347
9.3 Recursive Least Squares for Linear Models, 349
9.4 Temporal Difference Learning with a Linear Model, 356
9.5 Bellman’s Equation Using a Linear Model, 358
viii contents

9.6 Analysis of TD(0), LSTD, and LSPE Using a Single

State, 364
9.7 Gradient-Based Methods for Approximate Value
Iteration*, 366
9.8 Least Squares Temporal Differencing with Kernel
Regression*, 371
9.9 Value Function Approximations Based on Bayesian
Learning*, 373
9.10 Why Does It Work*, 376
9.11 Bibliographic Notes, 379
Problems, 381

10 Optimizing While Learning 383

10.1 Overview of Algorithmic Strategies, 385
10.2 Approximate Value Iteration and Q-Learning Using
Lookup Tables, 386
10.3 Statistical Bias in the Max Operator, 397
10.4 Approximate Value Iteration and Q-Learning Using
Linear Models, 400
10.5 Approximate Policy Iteration, 402
10.6 The Actor–Critic Paradigm, 408
10.7 Policy Gradient Methods, 410
10.8 The Linear Programming Method Using Basis Functions, 411
10.9 Approximate Policy Iteration Using Kernel Regression*, 413
10.10 Finite Horizon Approximations for Steady-State
Applications, 415
10.11 Bibliographic Notes, 416
Problems, 418

11 Adaptive Estimation and Stepsizes 419

11.1 Learning Algorithms and Stepsizes, 420
11.2 Deterministic Stepsize Recipes, 425
11.3 Stochastic Stepsizes, 433
11.4 Optimal Stepsizes for Nonstationary Time Series, 437
11.5 Optimal Stepsizes for Approximate Value Iteration, 447
11.6 Convergence, 449
11.7 Guidelines for Choosing Stepsize Formulas, 451
11.8 Bibliographic Notes, 452
Problems, 453
contents ix

12 Exploration Versus Exploitation 457

12.1 A Learning Exercise: The Nomadic Trucker, 457
12.2 An Introduction to Learning, 460
12.3 Heuristic Learning Policies, 464
12.4 Gittins Indexes for Online Learning, 470
12.5 The Knowledge Gradient Policy, 477
12.6 Learning with a Physical State, 482
12.7 Bibliographic Notes, 492
Problems, 493

13 Value Function Approximations for Resource Allocation

Problems 497
13.1 Value Functions versus Gradients, 498
13.2 Linear Approximations, 499
13.3 Piecewise-Linear Approximations, 501
13.4 Solving a Resource Allocation Problem Using
Piecewise-Linear Functions, 505
13.5 The SHAPE Algorithm, 509
13.6 Regression Methods, 513
13.7 Cutting Planes*, 516
13.8 Why Does It Work?**, 528
13.9 Bibliographic Notes, 535
Problems, 536

14 Dynamic Resource Allocation Problems 541

14.1 An Asset Acquisition Problem, 541
14.2 The Blood Management Problem, 547
14.3 A Portfolio Optimization Problem, 557
14.4 A General Resource Allocation Problem, 560
14.5 A Fleet Management Problem, 573
14.6 A Driver Management Problem, 580
14.7 Bibliographic Notes, 585
Problems, 586

15 Implementation Challenges 593

15.1 Will ADP Work for Your Problem?, 593
15.2 Designing an ADP Algorithm for Complex Problems, 594
15.3 Debugging an ADP Algorithm, 596
x contents

15.4 Practical Issues, 597

15.5 Modeling Your Problem, 602
15.6 Online versus Offline Models, 604
15.7 If It Works, Patent It!, 606

Bibliography 607

Index 623
 Preface to the Second Edition

The writing for the first edition of this book ended around 2005, followed by a
year of editing before it was submitted to the publisher in 2006. As with everyone
who works in this very rich field, my understanding of the models and algorithms
was strongly shaped by the projects I had worked on. While I was very proud of
the large industrial applications that were the basis of my success, at the time I had
a very limited understanding of many other important problem classes that help to
shape the algorithms that have evolved (and continue to evolve) in this field.
In the five years that passed before this second edition went to the publisher, my
understanding of the field and my breadth of applications have grown dramatically.
Reflecting my own personal growth, I realized that the book needed a fundamen-
tal restructuring along several dimensions. I came to appreciate that approximate
dynamic programming is much more than approximating value functions. After
writing an article that included a list of nine types of policies, I realized that every
policy I had encountered could be broken down into four fundamental classes:
myopic policies, lookahead policies, policy function approximations, and policies
based on value function approximations. Many other policies can be created by
combining these four fundamental classes into different types of hybrids.
I also realized that methods for approximating functions (whether they be pol-
icy function approximations or value function approximations) could be usefully
organized into three fundamental strategies: lookup tables, parametric models, and
nonparametric models. Of course, these can also be combined in different forms.
In preparing the second edition, I came to realize that the nature of the decision
variable plays a critical role in the design of an algorithm. In the first edition, one
of my goals was to create a bridge between dynamic programming (which tended
to focus on small action spaces) and math programming, with its appreciation of
vector-valued decisions. As a result I had adopted x as my generic decision vari-
able. In preparing the new edition, I had come to realize that small action spaces
cover a very important class of problems, and these are also the problems that a
beginner is most likely to start with to learn the field. Also action “a” pervades the
reinforcement learning community (along with portions of the operations research
community), to the point that it is truly part of the language. As a result the second
edition now uses action “a” for most of its presentation, but reverts to x specifically
xi
xii preface to the second edition

for problems where the decisions are continuous and/or (more frequently) vectors.
The challenges of vector-valued decisions has been largely overlooked in the rein-
forcement learning community, while the operations research community that works
on these problems has largely ignored the power of dynamic programming.
The second edition now includes a new chapter (Chapter 6) devoted purely to
a discussion of different types of policies, a summary of some hybrid strategies,
and a discussion of problems that are well suited to each of the different strategies.
This is followed by a chapter (Chapter 7) that focuses purely on the issue of policy
search. This chapter brings together fields such as stochastic search and simulation
optimization. The chapter also introduces a new class of optimal learning strate-
gies based on the concept of the knowledge gradient, an idea that was developed
originally to address the exploration–exploitation problem before realizing that it
had many other applications.
I also acquired a much better understanding of the different methods for approx-
imating value functions. I found that the best way to communicate the rich set of
strategies that have evolved was to divide the material into three chapters. The first
of these (Chapter 8) focuses purely on different statistical procedures for approxi-
mating value functions. While this can be viewed partly as a tutorial into statistics
and machine learning, the focus is on strategies that have been used in the approxi-
mate dynamic programming/reinforcement learning literature. ADP imposes special
demands on statistical learning algorithms, including the importance of recursive
estimation, and the need to start with a small number of observations (which works
better with a low-dimensional model) and transition to a larger number of obser-
vations with models that are high-dimensional in certain regions. Next, Chapter 9
summarizes different methods for estimating the value of being in a state using
sample information, with the goal of estimating the value function for a fixed pol-
icy. Since I have found that a number of papers focus on a single policy without
making this apparent, this chapter makes this very explicit by indexing variables
that depend on a policy with a superscript π . Finally, Chapter 10 addresses the very
difficult problem of estimating the value of being in a state while simultaneously
optimizing over policies.
Chapter 11 of this book is a refined version of the old Chapter 6, which
focused on stepsize rules. Chapter 11 is streamlined, with a new discussion of
the implications of algorithms based on policy iteration (including least squares
policy evaluation (LSPE), least squares temporal differences) and algorithms based
on approximate value iteration and Q-learning. Following some recent research,
I use the setting of a single state to develop a much clearer understanding of the
demands on a stepsize that are placed by these different algorithmic strategy. A
new section has been added introducing a stepsize rule that is specifically optimized
for approximate value iteration.
Chapter 12, on the famous exploration–exploitation problem in approximate
dynamic programming, has been heavily revised to reflect a much more thorough
understanding of the general field that is coming to be known as optimal learning.
This chapter includes a recently developed method for doing active learning in the
presence of a physical state, by way of the concept of the knowledge gradient.
preface to the second edition xiii

While this method looks promising, the general area of doing active learning in the
context of dynamic programs (with a physical state) is an active area of research
at the time of this writing.
A major theme of the first edition was to bridge the gap between disciplines,
primarily reinforcement learning (computer science), simulation, and math pro-
gramming (operations research). This edition reinforces this theme first by adopting
more broadly the notation and vocabulary of reinforcement learning (which has
made most of the contributions to this field) while retaining the bridge to math
programming, but now also including stochastic search and simulation optimization
(primarily in the context of policy search).
The mathematical level of the book continues to require only an understanding
of statistics and probability. A goal of the first edition was that the material would
be accessible to an advanced undergraduate audience. With this second edition
a more accurate description would be that the material is accessible to a highly
motivated and well prepared undergraduate, but the breadth of the material is more
suitable to a graduate audience.

Princeton, New Jersey Warren B. Powell

October 2010
 Preface to the First Edition

The path to completing this book began in the early 1980s when I first started
working on dynamic models arising in the management of fleets of vehicles for
the truckload motor carrier industry. It is often said that necessity is the mother
of invention, and as with many of my colleagues in this field, the methods that
emerged evolved out of a need to solve a problem. The initially ad hoc models and
algorithms I developed to solve these complex industrial problems evolved into
a sophisticated set of tools supported by an elegant theory within a field that is
increasingly being referred to as approximate dynamic programming.
The methods in this book reflect the original motivating applications. I started
with elegant models for which academie is so famous, but my work with industry
revealed the need to handle a number of complicating factors that were beyond the
scope of these models. One of these was a desire from one company to understand
the effect of uncertainty on operations, requiring the ability to solve these large-
scale optimization problems in the presence of various forms of randomness (but
most notably customer demands). This question launched what became a multiple-
decade search for a modeling and algorithmic strategy that would provide practical,
but high-quality, solutions.
This process of discovery took me through multiple fields, including linear and
nonlinear programming, Markov decision processes, optimal control, and stochas-
tic programming. It is somewhat ironic that the framework of Markov decision
processes, which originally appeared to be limited to toy problems (three trucks
moving between five cities), turned out to provide the critical theoretical frame-
work for solving truly industrial-strength problems (thousands of drivers moving
between hundreds of locations, each described by complex vectors of attributes).
The ability to solve these problems required the integration of four major dis-
ciplines: dynamic programming (Markov decision processes), math programming
(linear, nonlinear and integer programming), simulation, and statistics. My desire
to bring together the fields of dynamic programming and math programming moti-
vated some fundamental notational choices (in particular, the use of x as a decision
variable). In this book there is as a result a heavy dependence on the Monte Carlo
methods so widely used in simulation, but a knowledgeable reader will quickly
see how much is missing. The book covers in some depth a number of important
xv
xvi preface to the first edition

techniques from statistics, but even this presentation only scratches the surface
of tools and concepts available from with fields such as nonparametric statistics,
signal processing and approximation theory.

Audience
The book is aimed primarily at an advanced undergraduate/masters audience with
no prior background in dynamic programming. The presentation does expect a first
course in probability and statistics. Some topics require an introductory course in
linear programming. A major goal of the book is the clear and precise presentation
of dynamic problems, which means there is an emphasis on modeling and notation.
The body of every chapter focuses on models and algorithms with a minimum
of the mathematical formalism that so often makes presentations of dynamic pro-
grams inaccessible to a broader audience. Using numerous examples, each chapter
emphasizes the presentation of algorithms that can be directly applied to a variety
of applications. The book contains dozens of algorithms that are intended to serve
as a starting point in the design of practical solutions for real problems. Material for
more advanced graduate students (with measure-theoretic training and an interest
in theory) is contained in sections marked with **.
The book can be used quite effectively in a graduate level course. Several
chapters include “Why does it work” sections at the end that present proofs at an
advanced level (these are all marked with **). This material can be easily integrated
into the teaching of the material within the chapter.
Approximate dynamic programming is also a field that has emerged from several
disciplines. I have tried to expose the reader to the many dialects of ADP, reflect-
ing its origins in artificial intelligence, control theory, and operations research. In
addition to the diversity of words and phrases that mean the same thing—but often
with different connotations—I have had to make difficult notational choices.
I have found that different communities offer unique insights into different
dimensions of the problem. In the main, the control theory community has the most
thorough understanding of the meaning of a state variable. The artificial intelligence
community has the most experience with deeply nested problems (which require
numerous steps before earning a reward). The operations research community has
evolved a set of tools that are well suited for high-dimensional resource allocation,
contributing both math programming and a culture of careful modeling.

W. B. P.
 Acknowledgments

The work in this book reflects the contributions of many. Perhaps most important
are the problems that motivated the development of this material. This work would
not have been possible without the corporate sponsors who posed these problems
in the first place. I would like to give special recognition to Schneider National, the
largest truckload carrier in the United States, Yellow Freight System, the largest
less-than-truckload carrier, and Norfolk Southern Railroad, one of the four major
railroads that serves the United States. These three companies not only posed
difficult problems, they provided years of research funding that allowed me to
work on the development of tools that became the foundation of this book. This
work would never have progressed without the thousands of hours of my two senior
professional staff members, Hugo Simão and Belgacem Bouzaiëne-Ayari, who have
written hundreds of thousands of lines of code to solve industrial-strength problems.
It is their efforts working with our corporate sponsors that brought out the richness
of real applications, and therefore the capabilities that our tools needed to possess.
While industrial sponsors provided the problems, without the participation of
my graduate students, I would simply have a set of ad hoc procedures. It is the
work of my graduate students that provided most of the fundamental insights and
algorithms, and virtually all of the convergence proofs. In the order in which
they joined by research program, the students are Linos Frantzeskakis, Raymond
Cheung, Tassio Carvalho, Zhi-Long Chen, Greg Godfrey, Joel Shapiro, Mike
Spivey, Huseyin Topaloglu, Katerina Papadaki, Arun Marar, Tony Wu, Abraham
George, Juliana Nascimento, Peter Frazier, and Ilya Ryzhov, all of whom are my
current and former students and have contributed directly to the material presented
in this book. My undergraduate senior thesis advisees provided many colorful
applications of dynamic programming, and they contributed their experiences with
their computational work.
The presentation has benefited from numerous conversations with profession-
als in this community. I am particularly grateful to Erhan Çinlar, who taught me
the language of stochastic processes that played a fundamental role in guiding my
notation in the modeling of information. I am also grateful for many conversa-
tions with Ben van Roy, Dimitri Bertsekas, Andy Barto, Mike Fu, Dan Adelman,
Lei Zhao, and Diego Klabjan. I would also like to thank Paul Werbos at NSF
xvii
xviii acknowledgments

for introducing me to the wonderful neural net community in IEEE, which con-
tributed what for me was a fresh perspective on dynamic problems. Jennie Si, Don
Wunsch, George Lendaris and Frank Lewis all helped educate me in the language
and concepts of the control theory community.
For the second edition of the book, I would like to add special thanks to Peter
Frazier and Ilya Ryzhov, who contributed the research on the knowledge gradient
for optimal learning in ADP, and improvements in my presentation of Gittins
indices. The research of Jun Ma on convergence theory for approximate policy
iteration for continuous states and actions contributed to my understanding in a
significant way. This edition also benefited from the contributions of Warren Scott,
Lauren Hannah, and Emre Barut who have combined to improve my understanding
of nonparametric statistics.
This research was first funded by the National Science Foundation, but the
bulk of my research in this book was funded by the Air Force Office of Sci-
entific Research, and I am particularly grateful to Dr. Neal Glassman for his
support through the early years. The second edition has enjoyed continued support
from AFOSR by Donald Hearn, and I appreciate Don’s dedication to the AFOSR
program.
Many people have assisted with the editing of this volume through numerous
comments. Mary Fan, Tamas Papp, and Hugo Simão all read various drafts of
the first edition cover to cover. I would like to express my appreciation to Boris
Defourny for an exceptionally thorough proofreading of the second edition. Diego
Klabjan and his dynamic programming classes at the University of Illinois provided
numerous comments and corrections. Special thanks are due to the students in my
own undergraduate and graduate dynamic programming classes who had to survive
the very early versions of the text. The second edition of the book benefited from
the many comments of my graduate students, and my ORF 569 graduate seminar
on approximate dynamic programming. Based on their efforts, many hundreds of
corrections have been made, though I am sure that new errors have been introduced.
I appreciate the patience of the readers who understand that this is the price of
putting in textbook form material that is evolving so quickly.
Of course, the preparation of this book required tremendous patience from my
wife Shari and my children Elyse and Danny, who had to tolerate my ever-present
laptop at home. Without their support, this project could never have been completed.

W.B.P.
CHAPTER 1

The Challenges of Dynamic

Programming

The optimization of problems over time arises in many settings, ranging from the
control of heating systems to managing entire economies. In between are examples
including landing aircraft, purchasing new equipment, managing blood inventories,
scheduling fleets of vehicles, selling assets, investing money in portfolios, and just
playing a game of tic-tac-toe or backgammon. These problems involve making
decisions, then observing information, after which we make more decisions, and
then more information, and so on. Known as sequential decision problems, they
can be straightforward (if subtle) to formulate, but solving them is another matter.
Dynamic programming has its roots in several fields. Engineering and economics
tend to focus on problems with continuous states and decisions (these communities
refer to decisions as controls), which might be quantities such as location, speed,
and temperature. By contrast, the fields of operations research and artificial intel-
ligence work primarily with discrete states and decisions (or actions). Problems
that are modeled with continuous states and decisions (and typically in continuous
time) are often addressed under the umbrella of “control theory,” whereas problems
with discrete states and decisions, modeled in discrete time, are studied at length
under the umbrella of “Markov decision processes.” Both of these subfields set up
recursive equations that depend on the use of a state variable to capture history in a
compact way. There are many high-dimensional problems such as those involving
the allocation of resources that are generally studied using the tools of mathemati-
cal programming. Most of this work focuses on deterministic problems using tools
such as linear, nonlinear, or integer programming, but there is a subfield known as
stochastic programming that incorporates uncertainty. Our presentation spans all of
these fields.

Approximate Dynamic Programming: Solving the Curses of Dimensionality, Second Edition.

Warren B. Powell.
© 2011 John Wiley & Sons, Inc. Published 2011 by John Wiley & Sons, Inc.

1
2 the challenges of dynamic programming

14
1 2
8 10

q 3 5 r

15
17

Figure 1.1 Illustration of a shortest path problem from origin q to destination r .

1.1 A DYNAMIC PROGRAMMING EXAMPLE:

A SHORTEST PATH PROBLEM

Perhaps one of the best-known applications of dynamic programming is that faced

by a driver choosing a path in a transportation network. For simplicity (and this is
a real simplification for this application), we assume that the driver has to decide
at each node (or intersection) which link to traverse next (we are not going to get
into the challenges of left turns versus right turns). Let I be the set of intersections.
If the driver is at intersection i , he can go to a subset of intersections I+i at a cost
cij . He starts at the origin node q ∈ I and has to find his way to the destination
node r ∈ I at the least cost. An illustration is shown in Figure 1.1.
The problem can be easily solved using dynamic programming. Let

vi = Cost to get from intersection i ∈ I to the destination node r.

We assume that vr = 0. Initially we do not know vi , and so we start by setting

vi = M, where “M ” is known as “big M” and represents a large number. We can
solve the problem by iteratively computing
 
vi ← min vi , min {cij + vj } for all i ∈ I. (1.1)
j ∈I+

Equation (1.1) has to be solved iteratively, where at each iteration, we loop over all
the nodes i in the network. We stop when none of the values vi change. It should
be noted that this is not a very efficient way of solving a shortest path problem.
For example, in the early iterations it may well be the case that vj = M for all
j ∈ I+ . However, we use the method to illustrate dynamic programming.
Table 1.1 illustrates the algorithm, assuming that we always traverse the nodes
in the order (q, 1, 2, 3, r). Note that we handle node 2 before node 3, which is the
reason why, even in the first pass, we learn that the path cost from node 3 to node
r is 15 (rather than 17). We are done after iteration 3, but we require iteration 4 to
verify that nothing has changed.
Shortest path problems arise in a variety of settings that have nothing to do with
transportation or networks. Consider, for example, the challenge faced by a college
the three curses of dimensionality 3

Table 1.1 Path cost from each node to node r after each node has been visited
Cost from Node
Iteration q 1 2 3 r
100 100 100 100 0
1 100 100 10 15 0
2 30 18 10 15 0
3 26 18 10 15 0
4 26 18 10 15 0

freshman trying to plan her schedule up to graduation. By graduation, she must take
32 courses overall, including eight departmentals, two math courses, one science
course, and two language courses. We can describe the state of her academic
program in terms of how many courses she has taken under each of these five
categories. Let Stc be the number of courses she has taken by the end of semester
t in category c = {Total courses, Departmentals, Math, Science, Language}, and
let St = (Stc )c be the state vector. Based on this state, she has to decide which
courses to take in the next semester. To graduate, she has to reach the state S8 =
(32, 8, 2, 1, 2). We assume that she has a measurable desirability for each course
she takes, and that she would like to maximize the total desirability of all her
courses.
The problem can be viewed as a shortest path problem from the state S0 =
(0, 0, 0, 0, 0) to S8 = (32, 8, 2, 1, 2). Let St be her current state at the beginning
of semester t, and let at represent the decisions she makes while determining what
courses to take. We then assume we have access to a transition function S M (St , at ),
which tells us that if she is in state St and takes action at , she will land in state
St+1 , which we represent by simply using

St+1 = S M (St , at ).

In our transportation problem, we would have St = i if we are at intersection i ,

and at would be the decision to “go to j ,” leaving us in the state St+1 = j .
Finally, let Ct (St , at ) be the contribution or reward she generates from being
in state St and taking the action at . The value of being in state St is defined by the
equation

Vt (St ) = max {Ct (St , at ) + Vt+1 (St+1 )} ∀st ∈ St ,

xt

where St+1 = S M (St , at ) and where St is the set of all possible (discrete) states
that she can be in at the beginning of the year.

1.2 THE THREE CURSES OF DIMENSIONALITY

All dynamic programs can be written in terms of a recursion that relates the value
of being in a particular state at one point in time to the value of the states that we
Discovering Diverse Content Through
Random Scribd Documents
 “It’s safe enough to keep on till we work our way to that buoy,” said
Rod. “Dad and I have been as far as that several times in the motor-
boat, but of course only in fine weather.”
“I know the shallow water is all to the north of that buoy,” replied
Jack. “What I’m worrying about now is, where is the schooner?
There doesn’t seem to be any sign of her. Maybe she’s close up to
the shoal.”
For another twenty minutes the sloop ran on, tacking occasionally,
and ever approaching nearer the sound of the bell-buoy, the
melancholy note of which now came clearly enough across the
water.
“Hark!” cried Rod, suddenly.
Swish! came the waves against the prow of the sloop. Overhead
there was the constant song of the wind in the halyards. The only
other thing Jack could hear for a while was the bell-buoy.
“What was it?” he asked after a while.
Just then a gust of wind brought across the water the squawk of a
boat’s fog-horn.
Conch-conch!
“There she goes! That’s the schooner, all right,” cried Jack,
swinging the wheel over and heading in the direction of the sound.
Conch-conch! It was more distinct now, the wind bringing it directly
toward them.
“The tug hasn’t arrived, evidently,” Jack shouted to his companion.
“Didn’t I see something away across there just now?” he added,
pointing over to the starboard bow and peering into the haze.
“Yes, there she is!” he cried a few moments later, as a vague blur
became visible through the rain. “And she’s the Grace and Ella, too.
But what’s happened to her? There isn’t a spar standing!”
Another five-minutes’ manœuvering brought the Sea-Lark within
hail of the distressed vessel, and soon she was under her lee.
 “Ahoy, Captain Jordan!” Jack called. “The tug is coming off for
you.”
“Hello, youngster! Where is she?”
“Don’t know. They telephoned up to Rockmore for her. She’d gone
up there with a tow.”
“Well, if she ain’t quick she won’t be a powerful lot o’ use to us,”
replied Captain Jordan. “We could drift home in another month or
two, with this wind. What are you doing out here? This ain’t your beat
with the ferry.”
“Why, I didn’t know how bad a fix you might be in,” replied Jack.
“We could have carried you ashore if the schooner had been sinking.
What happened to your masts?”
Only a few yards separated the two craft. The little boat was rising
and falling on each wave, Jack keeping her clear, headed up into the
wind.
“A squall struck us about four o’clock this morning and made a
clean sweep. The dories went, and everything, and afore we knew it
the schooner was bumping her bottom out on the shoal. If the tide
hadn’t turned just then we’d ha’ been all broken up by now.”
“Well, there’s nothing I can do for you, is there?” asked Jack. Apart
from the natural desire to serve those on another craft in trouble, the
boy would have been pleased to be of service to Captain Jordan, for
he had been on particularly friendly terms with that stalwart
fisherman ever since he himself first began to potter about the
harbor. Indeed, many a time he had paddled about in one of the
Grace and Ella’s dories, and despite Mr. Barker’s enmity, the lad had
received more than one invitation from Captain Jordan to make a trip
in the schooner to the fishing-grounds, though he had declined to do
so, at the desire of his father.
“You say the tug ain’t coming straight out o’ Greenport?” the
skipper asked.
“No telling when she will be here,” replied Jack. “I don’t think she
can be long, though.”
 “Well, I’ll have to drop my anchor soon as the tide turns again,”
replied the fisherman, “unless you like to give us a tow. We can’t do
a thing now, except wait. We’ve got a big trip o’ fish aboard, too, and
she sprang a leak when she hit the sand. We’ve had to keep on
pumping.”
“I can tow you, all right,” replied Jack, “if you’ll pass me a hawser.
You won’t move so very fast, but I can get you there.”
“Well, that’ll be a salvage job, won’t it?” said Captain Jordan.
“You’d better put a price on it before we start.”
Jack, who had not thought of turning his adventure into a business
affair up to that moment, hardly knew what to reply.
“What’s a fair price, Captain?” he asked.
“You’d better settle that with the owners, after we get ashore.”
Jack laughed outright.
“No, thank you,” he replied. “Mr. Barker and I are not what you’d
call particularly good friends. What’s it worth to you to be taken into
the harbor?”
“If the tug comes along,” replied the captain, “it won’t cost the
owner anything, but there’s no sign of her yet and if we keep on
drifting the way we’re headed we’re due to land up on the Big Popple
Beach before long. I’ll take the first tow I can find and I guess I
wouldn’t kick at five hundred dollars for the job.”
There was a gasp from Rodney. Jack blinked once or twice, and
the smile vanished from his face. Then:
“Let me have that hawser,” he said in a businesslike tone. “Only,
Captain, if I start towing you, I want to finish the job.”
“Glad to have you,” replied Captain Jordan, who, though he did not
say so, was by no means heartbroken at the idea of running up a bill
against his employer, in the circumstances. For, like everybody else
with whom Simon Barker had had any dealings, the captain of the
Grace and Ella had been a victim of Barker’s meanness. Most of the
gear on the schooner was little better than junk. To wring a new set
of sails out of her owner was one of the hardest tasks in the world. It
was just the same with halyards, spars, and everything else aboard
that needed to be renewed from time to time after hard wear and
tear. Captain Jordan was convinced that the loss of the two masts
was due to Barker’s stinginess. Twelve months previously the
skipper had pointed out a defect in the mainmast and suggested that
it would be safer to have the spar replaced, but Barker would not
hear of it. It was that mast which gave way first when the squall hit
them, and the foremast, unable to bear the additional strain, followed
suit. Barker, therefore, was only paying for his own niggardliness.
 CHAPTER X
SALVAGE

T he task which Jack had undertaken was not so formidable as it

might have been, for everything was in his favor. The end of a
manila hawser was cast to him. This he made fast to the quarter-
bitts, and then he headed due east to clear the end of Greenport
breakwater. When once she felt the strain of the hawser holding her
back, the little Sea-Lark tugged and fretted like a greyhound puppy
at its leash, for she had a dead weight of several hundred tons
dragging at her heels. And for a few minutes that dead weight almost
seemed to be anchored securely to the bottom of the ocean. But the
steady pull of the sloop presently began to tell. The schooner moved
sluggishly, reluctantly, but the main point was that she moved, and
before long the Sea-Lark had enough way on to make a steady three
knots an hour, for the wind could not have been blowing from a more
satisfactory quarter.
Exciting moments came when a particularly strong gust swept
down on the two craft. With her boom swung away out, to catch the
full benefit of the wind, the Sea-Lark strained her slender mast now
and again more than Jack cared to contemplate. There was no “give”
to ease her, with that solid weight dragging astern. Sometimes, also,
a following sea hit the sloop viciously, breaking over her quarter, half
drenching the boy at her wheel, and pouring down into the cockpit.
Once, after the sloop had been struck in this manner, Captain Jordan
hailed the Sea-Lark from the deck of the schooner.
Putting his hand to his ear Jack turned to listen.
“I’ll send you a man to give a hand if you like,” the fisherman
shouted down the wind.
Jack could not make his own voice heard in reply, but there was
no mistaking the meaning of the wave of his arm, with which he
signaled back.
“Not if I know it!” he said emphatically to his companion. “What do
you think, Rod?”
“I should say not! Why, we must be almost half-way back now.
There would be no fun in it if we let some one else complete the job.”
“That’s just what I’m worrying about,” observed Jack. “We’re on
the way, all right, and if nobody interrupts us we shall get there safely
enough. But where is that tug? I can’t even hear her squeaking.”
“She must have left Rockmore long ago,” said Rod. “They’d have
some difficulty in locating us while this haze keeps as bad as it is
now, but they’ll find us, sure enough.”
“I wonder what will happen when the tug does bob up,” Jack said.
“If it arrived just now I don’t think I’d have the cheek to refuse to give
up the tow.”
“Why not?” So far as Rod could see they were doing perfectly
satisfactorily.
“Well,” replied Jack with a laugh, “you’ve got to remember that we
aren’t under steam-power, and they are, for one thing. Also, the tug
is Barker’s boat, and perhaps—”
“Rubbish!” declared Rod. “If you take my advice you’ll hang on as
long as the sloop holds together and as long as you’re doing what
you contracted to do. Didn’t the captain say he was glad to have you
give him a tow? And didn’t he agree that if you began to tow you
could finish it?”
“Well, we’ll see,” replied Jack. “I’ve got no complaints at present.
Never heard of such luck!”
“Five hundred dollars seems an awful lot of money for doing a little
bit of a thing like this, though, doesn’t it?” asked Rodney.
“Yes, it sounds a lot to us, but you must remember that the Grace
and Ella is worth thousands of dollars and has a full catch of ground-
fish under her hatches. It isn’t exactly the idea of losing the salvage
money that makes me hope the tug will miss us somehow, though,”
he added with a grim smile.
“It’s quite enough reason,” observed Rod.
“Yes, but I want to keep this tow till we get right into the harbor,”
said Jack. “I can’t very well be in two places at the same time, but I’d
give a month’s earnings at the ferry to be standing near Mr. Simon
Barker and watch his face when he sees this schooner of his being
towed in by the Sea-Lark. It won’t take him more than about a minute
to realize that I’m getting a very nice revenge for something he said
to me only a couple of days ago. Slip up for’ard there, Rod, will you?
I believe I spotted the Point then, and I want to make sure of my
bearings. The haze seems to be clearing off a bit—”
Then something happened which he had been dreading. Across
the water, from the direction of Four Fathom bell-buoy, which the
Sea-Lark had now left full three miles astern, came the sharp, piping
call of the tug.
Four short squeaks, a long one, and finally a short one.
“No mistaking her,” was Rod’s only comment. Much to Jack’s
delight, the captain of the Grace and Ella did not attempt to signal
back. Such a proceeding would, however, have been difficult
anyway, against the wind.
“There’s the Point!” suddenly cried Rod from the bow and a minute
later Jack altered his course a shade, for the weather was clearing
up rapidly and he saw it was necessary to make a little more
southing. At the same time he glanced back over his shoulder. The
squawk of the tug was now insistent.
“I believe we shall manage it yet,” he said delightedly. “Another
twenty minutes or so will take us round the end of the breakwater,
and the tug hasn’t spotted us yet.”
But, as he spoke, the rain ceased, and presently the ocean was
visible for miles.
“Here she comes!” Jack cried as his eyes fell on the distant shape
of the fussy little Simon P. Barker. “Tearing along like a snail! And I
guess her skipper’s roaring mad now that he can see what
happened.”
“Keep going! Keep going!” urged Rod, almost dancing with delight.
“You bet I’m going to keep going!” replied Jack, with the light of
battle in his eyes. “He’d bust his boilers in catching up with us now,
before we’re round the breakwater. Did you ever hear Steve Burke
talk when he’s real mad, Rod?”
“I don’t remember ever even seeing him.”
“Well, you will in a while, and it’s no Quaker meeting he’s coming
to, either! He knows what to expect from Mr. Barker as soon as he
lands ashore, for missing the schooner. Rod, I’m not going to let go
now. Burke wouldn’t dare to run us down. Of course if Captain
Jordan cuts us adrift, that settles it; but my guess is that he won’t.”
“It’s a pretty race, isn’t it!” said Jack, a moment later. “Two
tortoises, and one of them hobbled! Thank goodness Burke daren’t
drive his old tug any faster, or her rotten engines would shake
themselves to pieces.”
Meanwhile the irate skipper of the tug was protesting loudly with
his squeaky siren. The Simon P. Barker wallowed and rolled nearer
and nearer, but the Sea-Lark was fast approaching the safe refuge of
the harbor. Soon the tug was only about a thousand yards astern,
but by then Jack was negotiating the end of the breakwater, and
there only remained the straight run up the harbor to the wharf. The
tug, however, evidently had no intention of relinquishing her claim,
for she fussed up to the heels of the sloop and hailed.
“Let go that hawser!” Burke shouted angrily to Jack.
“Can’t do it. I’m under contract,” replied Jack, steadily.
“Well, I’ll bust your contract, and you too, if you don’t hand me that
hawser mighty quick,” Burke snapped back. His practised hand was
at the wheel, and the tug was now gliding through the water within a
few feet of the Sea-Lark’s side. Jack, however, did not give way an
inch. He was by no means sure that Burke would not bump into him
and secure the tow by main force. But the tug’s skipper knew just a
little too much of sea law to carry his tactics as far as that. At this
stage Captain Jordan put in a word.
“Let the lad alone,” he shouted. “He came first, and I had to give
him the job.”
“And what’s your owner going to say to you?” Burke yelled back to
the captain of the schooner.
“He can say what he likes,” returned Captain Jordan. “I’ve got a
word or two to put into his ear for sending me to sea with rotten
spars, and I’ve a right to arrange for a tow whenever I think it
prudent. Now beat it, or you’ll be having a smash-up yet, and then
there’ll be real money to pay out.”
With no alternative, Burke sidled away from the two vessels,
muttering savagely, but realizing that he could put in no just claim to
complete the tow. By that time Gull Island had been reached, and
Jack could make out figures on the wharves ahead.
“Jordan is a brick for standing up for you like that,” Rod said.
“I thought he would,” replied Jack. “But, you know, a bargain is a
bargain, and we had the law on our side, after all.”
“Our friend Burke looked as though he would like to bite your head
off.”
“He wasn’t very polite about it, was he? But you just wait a few
minutes till we draw up alongside Barker’s wharf. That’s when the
fun will begin.”
They plodded slowly on until Captain Jordan hailed them from the
schooner.
“Stand by to cast that hawser off,” he directed.
“Aye, aye!” replied Jack. “Rod, lay hold of that line, and be ready
to slack away.”
“Let go!” shouted Captain Jordan at last.
The hawser fell with a splash into the water, and Jack brought the
sloop round into the wind, leaving the Grace and Ella with enough
way on to sidle to her berth.
An angry face appeared at the edge of the wharf.
“What d’ye mean by meddling in my affairs?” demanded Mr.
Barker, scowling down on the boys.
“Meddling?” said Jack unruffled. “I was giving her a little tow.
Doesn’t she look as though she needed it? It was quite a salvage
job.”
“Salvage!” Mr. Barker’s surprise and consternation were ludicrous.
“You get no salvage out o’ me, young man! No one asked you to
interfere. What do you suppose I sent my tug out for?”
“Tug didn’t come in time,” put in Captain Jordan. “I had to engage
the sloop to make sure.”
“You engaged her?” queried Barker in horrified tones.
“What else could I do?” retorted Captain Jordan. “The tide would
have turned in another twenty minutes. Maybe you wouldn’t have
minded the schooner being piled up on the sand, but I minded.”
“Then you’ll pay for it,” snarled Barker. “I won’t. No, sir! Not a red
cent! What did you arrange to pay this—this kid?”
“Five hundred dollars, and—”
“Five hundred!” echoed Mr. Barker in agony.
“Why not? He saved you perhaps ten times that much.” Captain
Jordan was becoming annoyed at having to bandy words of this
order from ship to wharf.
Simon Barker leaned over and wagged his finger in the direction of
the captain of the Sea-Lark.
“Don’t you think for one minute,” he spluttered, “that I’ll ever pay
it.”
“We’ll see,” replied Jack, quietly.
There were few patrons for the ferry that day, and after landing
Rodney on the Point Jack returned to town and hurried home to his
dinner. He reached the house before his father was through and
quickly told his story. Mr. Holden was neither hopeful nor helpful.
“Don’t have anything to do with the man,” he begged. “He’ll find
some loophole of escape, and goodness knows there has been
unpleasantness enough already.”
“But, Dad, I earned it!” the boy protested. “Captain Jordan knows
what he is talking about. I’m going to consult a lawyer. Mr. Merrill will
see the thing is settled fairly. What’s the use of letting an old skinflint
like Barker cheat me out of the money?”
Mr. Holden made a vague motion with his hands.
“If you consult Mr. Merrill you’ll have to pay him, remember,” he
said. “Lawyers only cost you money in the end.”
“Maybe,” said Jack, “but you know quite well I should never be
able to get that salvage money without a lawyer’s help. I want your
permission to put the case in his hands, Dad.”
“All right,” said Mr. Holden reluctantly.
Five minutes later Jack was explaining to the jovial little lawyer all
that had happened.
“Well, Mr. Barker hates to part with his cash,” said the man of law.
“But I’ll see him in a day or two when he has calmed down, and
perhaps he will be in a more reasonable frame of mind then.”
Mr. Merrill had the reputation of being able to pour oil on troubled
waters more effectively than any other attorney in the county, but for
once his skill in such matters proved ineffective. Simon Barker was
as obstinate as a mule. He declared he would be sold up and even
go to prison rather than pay five hundred dollars to Jack Holden.
“But,” said the lawyer, reporting the outcome of his interview to
Jack two days later, “he will pay in the end. He can’t help paying. He
can fight, and stand us off for a long time if he wants to, but sooner
or later, Jack, he’s got to come to terms. So sit tight, my boy, and let
Simon and me do the worrying.”
 CHAPTER XI
THE STRUGGLE IN THE DARK

J ack was by now becoming accustomed to sleeping aboard the

Sea-Lark. The slight motion of the sloop as she lay at her
moorings, and the gentle lapping of water against the side of the
craft, instead of keeping him awake, lulled him to sleep. There was a
delightful novelty, also, in sleeping on his boat, and the sense of
impending danger which, during the first night or two, had kept his
ears straining, was beginning to vanish. Both he and George now
dropped off to sleep within a few minutes of the light being
extinguished.
There was one night, however, when George was not able to keep
his chum company, as his father required his assistance very early in
the morning to get a launch into the water on the top of the tide.
Jack, therefore, had to spend the night alone. For an hour or so after
darkness came, he chatted with Cap’n Crumbie in the latter’s cubby
on the wharf, and then climbed down to the sloop with no thought of
impending danger. In the cabin he lighted his lantern, read the last
two chapters of a book, stuck his head out of the companionway to
see what the weather was like, and then slipped off his shoes,
preparatory to turning in. As he placed the shoes under the bunk he
began to chuckle.
“It isn’t necessary,” he muttered, going over to a locker and taking
from it certain articles, including a hammer. “But what a joke if—”
He seemed to find considerable amusement in his thoughts and
once or twice his chuckle developed into a hearty laugh, as for
several minutes he busied himself at an occupation rarely indulged
in by skippers.
“There!” he said, apparently to the hammer, when the task was
finished, “it can’t do any harm, and it may do some good.”
 Five minutes later he was in the strange world of dreamland,
where pirates and redskins became entangled inextricably in
adventures more absorbing than any ever found between the covers
of a book.
Up on the wharf Cap’n Crumbie smoked his pipe in his cubby, and
dozed over an old magazine until his head nodded. For a few
minutes he, too, dropped off to sleep and then, when his pipe fell on
the floor, he sat up with a start. He stretched his arms, yawned, and
arose to pace nor’east and so’west on his favorite beat. There was
but a faint light, for the moon was not due for several hours yet, and
the sky was somewhat cloudy. Due south, three miles away, the red
beacon at the sea end of the long breakwater shone steadily, while,
farther east, the white glare of Greenport Light came at intervals.
Cap’n Crumbie screwed up his eyes as he peered into the darkness
away to the south for signs of an incoming vessel, but saw nothing,
and presently returned to his comfortable chair and the magazine.
He stuffed more tobacco into the bowl of his pipe, sent up prodigious
wreaths of smoke, and fixed his spectacles. For half an hour he
browsed over the printed pages, and then, it being a warm, close
night, his eyelids drooped once more. Somewhere a bell clanged. It
was one o’clock in the morning, and all was well. Cap’n Crumbie
dozed and his pipe fell to the floor with a rattle again. But he did not
move. The watchman was in the midst of forty winks, and they were
very sound winks, too.
The dark form of a dory came skulking along the wharves. There
was something suspicious in the behavior of its occupant. He dipped
his oars into the water so silently that the splash could not have been
heard more than a few yards away. He took a couple of furtive
strokes, and then rested, looking over his shoulder, while the dory
drifted slowly along. Another two strokes. A voice came from the
distance. It was some sailor calling “good night” to a belated visitor,
but the man in the dory gave a start. Again he peered over his
shoulder in the direction of Garnett and Sayer’s wharf, now close by,
and after assuring himself that no eye was on him, dipped the oars
cautiously and sidled toward the Sea-Lark. There was no sound
when he stepped with his bare feet over a thwart to fend the dory
from the sloop. This was done with infinite care, and the small craft
lay still. The man ran his fingers along the rail until they encountered
a projecting nail, and then he nodded silently. With cat-like
movements, he pushed the dory along to the stern of the sloop, and
stood up, ready to hoist himself aboard. No sound greeted his ears
save the lap of water against the side of the vessel. At last he
tightened his grip on the rail, pulled his body cautiously over it, and
crouched for an instant at the companionway. His fingers closed on
the door-knob and with stealthy tread he disappeared into the cabin.
One step he took in the blackness, and then something tripped
him up. It was a strong cord stretched across the bottom of the
steps, a foot from the ground. The man had fallen into a booby trap
of the simplest possible kind, which Jack had set in the event of any
unauthorized callers disturbing his slumbers.
With a startled cry quickly suppressed, the intruder sprawled
forward, clutching wildly, and measured his length on the floor. As he
fell his head came in violent contact with the edge of the table.
In a flash Jack was sitting bolt upright, collecting his wits. Then he
remembered the booby trap, but he did not laugh now. The thing had
evidently done its work.
With a bound he leaped from his berth, grasped his stick, and
landed on the intruder, who was already scrambling to his knees. It
was now the boy’s turn to stumble, but they both recovered at the
same moment.
Neither could see the other, although not more than three feet
separated them as they instinctively sprang apart.
“Stand still, or I’ll shoot!” called Jack. At the same moment he
moved toward the door, but before he could reach it the man was on
him and had borne him to the floor.
Jack struggled frantically, but his opponent was stronger than he.
A hand was thrust over his mouth and a knee pressed heavily on his
chest.
“Keep still, or I’ll crush the life out o’ you!” a voice hissed.
 But Jack had no intention of keeping still in the circumstances.
One of his legs was free, and with it he kicked out lustily, making as
much noise as possible, but doing no harm to his assailant.
The hand on Jack’s mouth pressed still more firmly. Acting on an
inspiration, the boy opened his mouth, and his firm teeth closed on a
finger. There was a sharp cry of pain and the hand was withdrawn,
whereupon Jack instantly shouted at the top of his lungs.
Down came the hand again across his face, his half-raised head
was thrust savagely back against the floor of the cabin, and Jack
was momentarily dazed.
“Now will you keep still?” the voice demanded in a hoarse whisper.
While one hand was being held across his mouth, the boy felt
another close remorselessly over his throat. His left arm was
doubled under him, and although his right hand remained free, he
could do nothing with it save clutch ineffectually at the arm of his
assailant. The pressure on his throat was terrible and he could no
longer breathe. Even then, however, he did not cease to squirm, and
his hand encountered the leg of the cabin table. Desperately he
closed his fingers over this and using the last ounce of his now fast-
dwindling strength, dragged the thing over.
Exactly what happened the boy could not tell, but a moment later
he found himself mixed up in a nightmare of table legs and clutching
hands. His mouth and throat were freed, and he uttered another loud
cry for help, but was immediately silenced by a savage blow on the
head. Those powerful hands gripped him afresh and he felt his
senses going. He was choking again, and there was a painful
buzzing in his ears. Vaguely, as though it were miles away, he
thought he heard a shout, and then a ton weight seemed to be lifted
from his chest. His assailant had gone, and Jack could hear some
one shouting distinctly now.
“Hello! hello, there!” It was Cap’n Crumbie, outside on the wharf.
The boy tried to call back, but for the moment he was too exhausted.
Then the watchman burst into the cabin.
“Hello, Jack! What in thunder’s up?”
 “After him, quick!” Jack answered huskily. “He’s nearly choked me
to death!”
“Are you all right, lad?” Cap’n Crumbie lifted him to his feet.
“Yes, I think so, but—” Jack broke off to scramble out on deck and
peer into the darkness.
The sound of oars could be heard not far away. There was another
dory lying near, and into this the watchman and Jack hustled and
pulled desperately after the retreating craft. It had a considerable
lead, and in the black shadows of the wharves was not visible at first.
Finally, though, “I can see him now,” Jack cried. “Pull hard!”
The man in the other dory was evidently not an expert with the
oars, for he was splashing badly as he urged his boat forward.
“We’ve got him!” said Cap’n Crumbie. “He can’t get away now!”
At that instant the splashing ceased, and the pursuers began to
gain on their quarry more rapidly. They were overhauling the dory at
top speed. Cap’n Crumbie stopped rowing.
“Easy there, lad!” he cautioned breathlessly. “Maybe that feller has
a gun, and there’s no need for us to get shot.”
Only a dozen yards of still water separated the two craft now, but
there was no sound or movement in the other dory, and the pursuers
were drifting nearer all the time.
“Why—why, it’s empty!” cried Jack, disgustedly, as the boat
became more distinct.
Two or three more strokes brought them alongside. It was, as the
boy had surmised, without an occupant—just an empty old dory,
floating idly on the water.
“He must have gone over the side and swum ashore,” said Jack,
chagrined. “We should have had him in another few minutes. Let’s
pull in closer to the wharves and see if there’s any sign of him there.”
They searched accordingly, but without success. Evidently the
fugitive had slipped quietly into the water and escaped.
 “Well, we’ve got his dory, anyway,” said Cap’n Crumbie. “Let’s go
back for her.”
They towed the craft back to Garnett and Sayer’s wharf, where the
watchman lighted a lantern and examined it.
“Huh! It’s Joe Whalen’s dory,” declared Cap’n Crumbie. “The old
man’s been in bed with rheumatism for over a week, and he wouldn’t
hurt nobody, anyhow. This chap that attacked you must have swiped
it. And there ain’t a dog-gone thing in it, neither, ’ceptin’ the oars.
Knew too much to leave his things behind, he did.”
“I’m afraid I shouldn’t have got off so well if I hadn’t thought of
using a string to trip him up,” said the boy. “He would have had me at
his mercy in my bunk if I’d been asleep. Why, what on earth—”
The lad stopped at the entrance of the sloop’s cabin, the lantern in
his hand illuminating the scene of the struggle. Then he stooped and
picked up something that was lying on the floor, half covered by the
overturned table. It was beveled like a chisel at one end.
“That yours?” asked the watchman, puzzled.
“I’ve never seen it in my life before,” replied Jack. “It must have
been dropped by that brute.”
He weighed the bar in his hand, and passed it over to Cap’n
Crumbie.
“I admit I was a bit scared,” he said, “being awakened so suddenly
that way and finding a man under my feet as soon as I jumped out,
but I should have been a good deal more scared if I’d known he had
a thing like that in his hand!”
Cap’n Crumbie ran his fingers through his hair and sat down on
the edge of one of the bunks.
“One good tap on the head with that thing, my lad, and you’d ha’
stopped running ferry-boats for all time,” he said. “That bit o’ string
saved your life, I reckon.”
“It looks like it,” the boy admitted. “I wonder whether the Chief of
Police will believe now that there’s something wrong or will wait until
I’m in the hospital before he does anything!”
“Well, beat it up to the station-house now, and show them that
piece of steel. I won’t leave here till you come back. Whether it’s you
or the sloop this mysterious feller is after, I don’t know, but it ain’t
likely he’ll be around again to-night; and if he is he won’t get the
sloop so long’s I’m alive.”
Jack slipped his shoes on and walked to the police station, where
he found Sergeant Banks on night duty.
“What time will the chief be in?” he asked.
“Eight o’clock,” replied the sergeant. “What’s wrong?”
For answer, Jack placed the bar of steel on the sergeant’s desk.
“Somebody came into the cabin of my boat just now, while I was
asleep, and left that when he went out.”
The sergeant frowned over the piece of steel and then eyed the
boy.
“That’s a cold-chisel, I take it,” he said finally. “Where’d you say
you found it?” Jack told his story briefly, ending with: “This was about
half an hour ago, and you might catch the fellow if you send some of
your men to look for him.”
“Not so fast, young man,” replied the sergeant, pompously. “Let
me get this thing right, and then I’ll decide what action is necessary.
First of all, you were in here the other day complaining of some one
walking across your deck, weren’t you?”
“He came into the cabin that time while my chum and I were
asleep.”
“Well, that’s all there is to that. He didn’t hurt you, did he?”
“No, but he tried to strangle me, to-night. That’s something, isn’t
it?”
The officer elevated his eyebrows and puffed out his cheeks.
Without a doubt he was incredulous.
“But just now you said he attacked you with this bar, didn’t you?”
 “No, I said he evidently meant to attack me with it,” replied the boy,
impatiently. “He must have dropped it when he fell over the string.
And then when Cap’n Crumbie heard me shouting, and answered,
the man ran off without it.”
“Then some one besides you saw the fellow, eh? That might make
a difference, if it’s true.”
“It was too dark for Cap’n Crumbie to see the man. He only heard
me yelling.”
“Well, what happened then?”
“Cap’n Crumbie and I jumped into the dory and chased the chap.
He was in a dory of his own. We pulled after him as hard as we
could, and I think we should have caught him if he hadn’t jumped
overboard and swum to one of the wharves.”
“And his dory disappeared too, eh?”
“No. We found it, but it was empty.”
“Now that’s unfortunate,” said the sergeant, sarcastically. “Seems
like you’re the only person who saw this fellow and you didn’t rightly
see him, either—just heard him.”
“I sort of think I felt him, too,” replied Jack, warmly. “Look at my
throat. Aren’t the marks of the man’s fingers there?”
“Ye-es, but I’ve heard of folks having the nightmare and choking
themselves.”
“You think, then, that I just dreamed the whole thing?” exclaimed
the boy. “Gee, that’s great! Maybe I dreamed that chisel thing there,
too!”
“I don’t say you dreamed. But you might have. I’ll give the story to
the chief when he comes on in the morning, and—”
“And meanwhile the fellow can go home and get his wet clothes
off and—”
“I’ll pass the word around,” said the sergeant, “and we’ll keep a
lookout, but I want to tell you that if we find that you’ve been trying to
put any funny business over, it’ll go hard with you. You’d better come
in and see the chief in the morning and let him hear your story.”
Jack returned to the wharf and Cap’n Crumbie.
“Well, what did they say?” the watchman asked.
“I only saw Sergeant Banks, who was on night duty. He didn’t
seem to believe me, but he’s going to pass the word round among
his men, and I’m to see the chief in the morning.”
“Banks is a barnacle!” declared Cap’n Crumbie. “I’ll go with you
next time, and we’ll see if we can’t get ’em to take a bit o’ notice.”
There was no more sleep for Jack that night. The watchman kept
him company until dawn, and the boy went home as usual for an
early breakfast, after which he and Cap’n Crumbie repaired to the
police station.
“’Morning,” greeted the chief. “Glad to see you, Cap’n. Now, what’s
your version of this business?”
The chief’s manner was much too airy for Cap’n Crumbie, and he
scowled as he advanced to the desk. He grunted as he thrust his
hands into his pockets and faced the man behind it.
“My version’s the same as his, chief,” he replied. “I s’pose you’ve
got the feller by this time, ain’t you?”
“Why, no,” he said. “We haven’t made any arrest yet. There isn’t
much to go on, Cap’n. We watched along the wharves last night, but
didn’t see any one.”
“Guess the feller was home and abed before your men started to
look for him,” grunted the Cap’n. “’Taint likely he’d stand around and
wait to be caught, is it? ’Course, if he hadn’t been in the water he
might, but being sort o’ wet—” Cap’n Crumbie’s labored sarcasm
ended in a derisive snort.
“Come, now, Crumbie, let’s get this thing straight. I understand
that you didn’t actually see the man. That’s so, isn’t it?”
“Yes, that’s so, but—”
Welcome to our website – the ideal destination for book lovers and
knowledge seekers. With a mission to inspire endlessly, we offer a
vast collection of books, ranging from classic literary works to
specialized publications, self-development books, and children's
literature. Each book is a new journey of discovery, expanding
knowledge and enriching the soul of the reade

Our website is not just a platform for buying books, but a bridge
connecting readers to the timeless values of culture and wisdom. With
an elegant, user-friendly interface and an intelligent search system,
we are committed to providing a quick and convenient shopping
experience. Additionally, our special promotions and home delivery
services ensure that you save time and fully enjoy the joy of reading.

Let us accompany you on the journey of exploring knowledge and

personal growth!

ebookfinal.com