Explore the full ebook collection and download it now at textbookfull.

com

Foundations of Deep Reinforcement Learning Theory

and Practice in Python 1st Edition Laura Graesser

https://textbookfull.com/product/foundations-of-deep-
reinforcement-learning-theory-and-practice-in-python-1st-
edition-laura-graesser/

OR CLICK HERE

DOWLOAD EBOOK

Browse and Get More Ebook Downloads Instantly at https://textbookfull.com

Click here to visit textbookfull.com and download textbook now
 Your digital treasures (PDF, ePub, MOBI) await
Download instantly and pick your perfect format...

Read anywhere, anytime, on any device!

Foundations of Deep Reinforcement Learning Theory and

Practice in Python First Edition Laura Graesser

https://textbookfull.com/product/foundations-of-deep-reinforcement-
learning-theory-and-practice-in-python-first-edition-laura-graesser/

textbookfull.com

Deep Reinforcement Learning in Action 1st Edition

Alexander Zai

https://textbookfull.com/product/deep-reinforcement-learning-in-
action-1st-edition-alexander-zai/

textbookfull.com

Grokking Deep Reinforcement Learning First Edition Miguel

Morales

https://textbookfull.com/product/grokking-deep-reinforcement-learning-
first-edition-miguel-morales/

textbookfull.com

Deep Reinforcement Learning in Unity: With Unity ML

Toolkit 1st Edition Abhilash Majumder

https://textbookfull.com/product/deep-reinforcement-learning-in-unity-
with-unity-ml-toolkit-1st-edition-abhilash-majumder/

textbookfull.com
Deep Reinforcement Learning in Unity With Unity ML Toolkit
1st Edition Abhilash Majumder

https://textbookfull.com/product/deep-reinforcement-learning-in-unity-
with-unity-ml-toolkit-1st-edition-abhilash-majumder-2/

textbookfull.com

Deep Learning with Python 1st Edition François Chollet

https://textbookfull.com/product/deep-learning-with-python-1st-
edition-francois-chollet/

textbookfull.com

Reinforcement Learning: With Open AI, TensorFlow and Keras

Using Python 1st Edition Abhishek Nandy

https://textbookfull.com/product/reinforcement-learning-with-open-ai-
tensorflow-and-keras-using-python-1st-edition-abhishek-nandy/

textbookfull.com

Probabilistic Deep Learning with Python 1st Edition Oliver

Duerr

https://textbookfull.com/product/probabilistic-deep-learning-with-
python-1st-edition-oliver-duerr/

textbookfull.com

Applied Reinforcement Learning with Python: With OpenAI

Gym, Tensorflow, and Keras Beysolow Ii

https://textbookfull.com/product/applied-reinforcement-learning-with-
python-with-openai-gym-tensorflow-and-keras-beysolow-ii/

textbookfull.com
Praise for Foundations of Deep
Reinforcement Learning
“This book provides an accessible introduction to deep reinforcement learning covering
the mathematical concepts behind popular algorithms as well as their practical
implementation. I think the book will be a valuable resource for anyone looking to apply
deep reinforcement learning in practice.”
—Volodymyr Mnih, lead developer of DQN

“An excellent book to quickly develop expertise in the theory, language, and practical
implementation of deep reinforcement learning algorithms. A limpid exposition which
uses familiar notation; all the most recent techniques explained with concise, readable
code, and not a page wasted in irrelevant detours: it is the perfect way to develop a solid
foundation on the topic.”
—Vincent Vanhoucke, principal scientist, Google

“As someone who spends their days trying to make deep reinforcement learning methods
more useful for the general public, I can say that Laura and Keng’s book is a welcome
addition to the literature. It provides both a readable introduction to the fundamental
concepts in reinforcement learning as well as intuitive explanations and code for many of
the major algorithms in the field. I imagine this will become an invaluable resource for
individuals interested in learning about deep reinforcement learning for years to come.”
—Arthur Juliani, senior machine learning engineer, Unity Technologies

“Until now, the only way to get to grips with deep reinforcement learning was to slowly
accumulate knowledge from dozens of different sources. Finally, we have a book bringing
everything together in one place.”
—Matthew Rahtz, ML researcher, ETH Zürich
 Foundations
of Deep
Reinforcement
Learning
 The Pearson Addison-Wesley
Data & Analytics Series

Visit informit.com/awdataseries for a complete list of available publications.

T he Pearson Addison-Wesley Data & Analytics Series provides readers with


practical knowledge for solving problems and answering questions with data.
Titles in this series primarily focus on three areas:
1. Infrastructure: how to store, move, and manage data
2. Algorithms: how to mine intelligence or make predictions based on data
3. Visualizations: how to represent data and insights in a meaningful and

compelling way

The series aims to tie all three of these areas together to help the reader build
end-to-end systems for fighting spam; making recommendations; building
personalization; detecting trends, patterns, or problems; and gaining insight
from the data exhaust of systems and user interactions.

Make sure to connect with us!

informit.com/socialconnect
 Foundations
of Deep
Reinforcement
Learning

Theory and Practice

in Python

Laura Graesser
Wah Loon Keng

Boston • Columbus • New York • San Francisco • Amsterdam • Cape Town

Dubai • London • Madrid • Milan • Munich • Paris • Montreal • Toronto • Delhi • Mexico City
São Paulo • Sydney • Hong Kong • Seoul • Singapore • Taipei • Tokyo
Many of the designations used by manufacturers and sellers to distinguish their products are claimed as
trademarks. Where those designations appear in this book, and the publisher was aware of a trademark
claim, the designations have been printed with initial capital letters or in all capitals.

The authors and publisher have taken care in the preparation of this book, but make no expressed or
implied warranty of any kind and assume no responsibility for errors or omissions. No liability is assumed
for incidental or consequential damages in connection with or arising out of the use of the information or
programs contained herein.

For information about buying this title in bulk quantities, or for special sales opportunities (which may
include electronic versions; custom cover designs; and content particular to your business, training goals,
marketing focus, or branding interests), please contact our corporate sales department
at [email protected] or (800) 382-3419.

For government sales inquiries, please contact [email protected].

For questions about sales outside the U.S., please contact [email protected].

Visit us on the Web: informit.com/aw

Library of Congress Control Number: 2019948417

Copyright © 2020 Pearson Education, Inc.

Cover illustration by Wacomka/Shutterstock

SLM Lab is an MIT-licensed open source project.

All rights reserved. This publication is protected by copyright, and permission must be obtained from the
publisher prior to any prohibited reproduction, storage in a retrieval system, or transmission in any form or
by any means, electronic, mechanical, photocopying, recording, or likewise. For information regarding
permissions, request forms and the appropriate contacts within the Pearson Education Global Rights &
Permissions Department, please visit www.pearson.com/permissions.

ISBN-13: 978-0-13-517238-4
ISBN-10: 0-13-517238-1

3 20
For those people who make me feel that anything is possible
—Laura

For my wife Daniela

—Keng
This page intentionally left blank
Contents

Foreword xix

Preface xxi

Acknowledgments xxv

About the Authors xxvii

1 Introduction to Reinforcement Learning 1

1.1 Reinforcement Learning 1
1.2 Reinforcement Learning as MDP 6
1.3 Learnable Functions in Reinforcement
Learning 9
1.4 Deep Reinforcement Learning
Algorithms 11
1.4.1 Policy-Based Algorithms 12
1.4.2 Value-Based Algorithms 13
1.4.3 Model-Based Algorithms 13
1.4.4 Combined Methods 15
1.4.5 Algorithms Covered in This
Book 15
1.4.6 On-Policy and Off-Policy
Algorithms 16
1.4.7 Summary 16
1.5 Deep Learning for Reinforcement
Learning 17
1.6 Reinforcement Learning and Supervised
Learning 19
1.6.1 Lack of an Oracle 19
1.6.2 Sparsity of Feedback 20
1.6.3 Data Generation 20
1.7 Summary 21
x Contents

I Policy-Based and Value-Based

Algorithms 23

2 REINFORCE 25
2.1 Policy 26
2.2 The Objective Function 26
2.3 The Policy Gradient 27
2.3.1 Policy Gradient
Derivation 28
2.4 Monte Carlo Sampling 30
2.5 REINFORCE Algorithm 31
2.5.1 Improving
REINFORCE 32
2.6 Implementing REINFORCE 33
2.6.1 A Minimal REINFORCE
Implementation 33
2.6.2 Constructing Policies with
PyTorch 36
2.6.3 Sampling Actions 38
2.6.4 Calculating Policy
Loss 39
2.6.5 REINFORCE Training
Loop 40
2.6.6 On-Policy Replay
Memory 41
2.7 Training a REINFORCE Agent 44
2.8 Experimental Results 47
2.8.1 Experiment: The Effect of
Discount Factor γ 47
2.8.2 Experiment: The Effect of
Baseline 49
2.9 Summary 51
2.10 Further Reading 51
2.11 History 51

3 SARSA 53
3.1 The Q- and V-Functions 54
3.2 Temporal Difference Learning 56
3.2.1 Intuition for Temporal
Difference Learning 59
 Contents xi

3.3 Action Selection in SARSA 65

3.3.1 Exploration and
Exploitation 66
3.4 SARSA Algorithm 67
3.4.1 On-Policy Algorithms 68
3.5 Implementing SARSA 69
3.5.1 Action Function: ε-Greedy 69
3.5.2 Calculating the Q-Loss 70
3.5.3 SARSA Training Loop 71
3.5.4 On-Policy Batched Replay
Memory 72
3.6 Training a SARSA Agent 74
3.7 Experimental Results 76
3.7.1 Experiment: The Effect of
Learning Rate 77
3.8 Summary 78
3.9 Further Reading 79
3.10 History 79

4 Deep Q-Networks (DQN) 81

4.1 Learning the Q-Function in DQN 82
4.2 Action Selection in DQN 83
4.2.1 The Boltzmann Policy 86
4.3 Experience Replay 88
4.4 DQN Algorithm 89
4.5 Implementing DQN 91
4.5.1 Calculating the Q-Loss 91
4.5.2 DQN Training Loop 92
4.5.3 Replay Memory 93
4.6 Training a DQN Agent 96
4.7 Experimental Results 99
4.7.1 Experiment: The Effect of
Network Architecture 99
4.8 Summary 101
4.9 Further Reading 102
4.10 History 102

5 Improving DQN 103

5.1 Target Networks 104
xii Contents

5.2 Double DQN 106

5.3 Prioritized Experience Replay
(PER) 109
5.3.1 Importance Sampling 111
5.4 Modified DQN Implementation 112
5.4.1 Network Initialization 113
5.4.2 Calculating the
Q-Loss 113
5.4.3 Updating the Target
Network 115
5.4.4 DQN with Target
Networks 116
5.4.5 Double DQN 116
5.4.6 Prioritized Experienced
Replay 117
5.5 Training a DQN Agent to Play Atari
Games 123
5.6 Experimental Results 128
5.6.1 Experiment: The Effect of
Double DQN and PER 128
5.7 Summary 132
5.8 Further Reading 132

II Combined Methods 133

6 Advantage Actor-Critic (A2C) 135

6.1 The Actor 136
6.2 The Critic 136
6.2.1 The Advantage
Function 136
6.2.2 Learning the Advantage
Function 140
6.3 A2C Algorithm 141
6.4 Implementing A2C 143
6.4.1 Advantage
Estimation 144
6.4.2 Calculating Value Loss and
Policy Loss 147
 Contents xiii

6.4.3 Actor-Critic Training Loop 147

6.5 Network Architecture 148
6.6 Training an A2C Agent 150
6.6.1 A2C with n-Step Returns on
Pong 150
6.6.2 A2C with GAE on Pong 153
6.6.3 A2C with n-Step Returns on
BipedalWalker 155
6.7 Experimental Results 157
6.7.1 Experiment: The Effect of n-Step
Returns 158
6.7.2 Experiment: The Effect of λ of
GAE 159
6.8 Summary 161
6.9 Further Reading 162
6.10 History 162

7 Proximal Policy Optimization (PPO) 165

7.1 Surrogate Objective 165
7.1.1 Performance Collapse 166
7.1.2 Modifying the Objective 168
7.2 Proximal Policy Optimization (PPO) 174
7.3 PPO Algorithm 177
7.4 Implementing PPO 179
7.4.1 Calculating the PPO Policy
Loss 179
7.4.2 PPO Training Loop 180
7.5 Training a PPO Agent 182
7.5.1 PPO on Pong 182
7.5.2 PPO on BipedalWalker 185
7.6 Experimental Results 188
7.6.1 Experiment: The Effect of λ of
GAE 188
7.6.2 Experiment: The Effect of
Clipping Variable ε 190
7.7 Summary 192
7.8 Further Reading 192
xiv Contents

8 Parallelization Methods 195

8.1 Synchronous Parallelization 196
8.2 Asynchronous Parallelization 197
8.2.1 Hogwild! 198
8.3 Training an A3C Agent 200
8.4 Summary 203
8.5 Further Reading 204

9 Algorithm Summary 205

III Practical Details 207

10 Getting Deep RL to Work 209

10.1 Software Engineering Practices 209
10.1.1 Unit Tests 210
10.1.2 Code Quality 215
10.1.3 Git Workflow 216
10.2 Debugging Tips 218
10.2.1 Signs of Life 219
10.2.2 Policy Gradient
Diagnoses 219
10.2.3 Data Diagnoses 220
10.2.4 Preprocessor 222
10.2.5 Memory 222
10.2.6 Algorithmic Functions 222
10.2.7 Neural Networks 222
10.2.8 Algorithm
Simplification 225
10.2.9 Problem
Simplification 226
10.2.10 Hyperparameters 226
10.2.11 Lab Workflow 226
10.3 Atari Tricks 228
10.4 Deep RL Almanac 231
10.4.1 Hyperparameter
Tables 231
 Contents xv

10.4.2 Algorithm Performance

Comparison 234
10.5 Summary 238

11 SLM Lab 239

11.1 Algorithms Implemented in SLM Lab 239
11.2 Spec File 241
11.2.1 Search Spec Syntax 243
11.3 Running SLM Lab 246
11.3.1 SLM Lab Commands 246
11.4 Analyzing Experiment Results 247
11.4.1 Overview of the Experiment
Data 247
11.5 Summary 249

12 Network Architectures 251

12.1 Types of Neural Networks 251
12.1.1 Multilayer Perceptrons
(MLPs) 252
12.1.2 Convolutional Neural Networks
(CNNs) 253
12.1.3 Recurrent Neural Networks
(RNNs) 255
12.2 Guidelines for Choosing a Network
Family 256
12.2.1 MDPs vs. POMDPs 256
12.2.2 Choosing Networks for
Environments 259
12.3 The Net API 262
12.3.1 Input and Output Layer Shape
Inference 264
12.3.2 Automatic Network
Construction 266
12.3.3 Training Step 269
12.3.4 Exposure of Underlying
Methods 270
12.4 Summary 271
12.5 Further Reading 271
xvi Contents

13 Hardware 273
13.1 Computer 273
13.2 Data Types 278
13.3 Optimizing Data Types in RL 280
13.4 Choosing Hardware 285
13.5 Summary 285

IV Environment Design 287

14 States 289
14.1 Examples of States 289
14.2 State Completeness 296
14.3 State Complexity 297
14.4 State Information Loss 301
14.4.1 Image Grayscaling 301
14.4.2 Discretization 302
14.4.3 Hash Conflict 303
14.4.4 Metainformation
Loss 303
14.5 Preprocessing 306
14.5.1 Standardization 307
14.5.2 Image Preprocessing 308
14.5.3 Temporal
Preprocessing 310
14.6 Summary 313

15 Actions 315
15.1 Examples of Actions 315
15.2 Action Completeness 318
15.3 Action Complexity 319
15.4 Summary 323
15.5 Further Reading: Action Design in
Everyday Things 324

16 Rewards 327
16.1 The Role of Rewards 327
16.2 Reward Design Guidelines 328
16.3 Summary 332
 Contents xvii

17 Transition Function 333

17.1 Feasibility Checks 333
17.2 Reality Check 335
17.3 Summary 337

Epilogue 338

A Deep Reinforcement Learning Timeline 343

B Example Environments 345

B.1 Discrete Environments 346
B.1.1 CartPole-v0 346
B.1.2 MountainCar-v0 347
B.1.3 LunarLander-v2 347
B.1.4 PongNoFrameskip-v4 348
B.1.5 BreakoutNoFrameskip-v4 349
B.2 Continuous Environments 350
B.2.1 Pendulum-v0 350
B.2.2 BipedalWalker-v2 350

References 353

Index 363
This page intentionally left blank
 Foreword

In April of 2019, OpenAI’s Five bots played in a Dota 2 competition match against 2018
human world champions, OG. Dota 2 is a complex, multiplayer battle arena game where
players can choose different characters. Winning a game requires strategy, teamwork, and
quick decisions. Building an artificial intelligence to compete in this game, with so
many variables and a seemingly infinite search space for optimization, seems like an
insurmountable challenge. Yet OpenAI’s bots won handily and, soon after, went on to win
over 99% of their matches against public players. The innovation underlying this
achievement was deep reinforcement learning.
Although this development is recent, reinforcement learning and deep learning have
both been around for decades. However, a significant amount of new research combined
with the increasing power of GPUs have pushed the state of the art forward. This book
gives the reader an introduction to deep reinforcement learning and distills the work done
over the last six years into a cohesive whole.
While training a computer to beat a video game may not be the most practical thing to
do, it’s only a starting point. Reinforcement learning is an area of machine learning that is
useful for solving sequential decision-making problems—that is, problems that are solved
over time. This applies to almost any endeavor—be it playing a video game, walking down
the street, or driving a car.
Laura Graesser and Wah Loon Keng have put together an approachable introduction to
a complicated topic that is at the forefront of what is new in machine learning. Not only
have they brought to bear their research into many papers on the topic; they created an
open source library, SLM Lab, to help others get up and running quickly with deep
reinforcement learning. SLM Lab is written in Python on top of PyTorch, but readers only
need familiarity with Python. Readers intending to use TensorFlow or some other library
as their deep learning framework of choice will still get value from this book as it
introduces the concepts and problem formulations for deep reinforcement learning
solutions.
This book brings together the most recent research in deep reinforcement learning
along with examples and code that the readers can work with. Their library also works
with OpenAI’s Gym, Roboschool, and the Unity ML-Agents toolkit, which makes this
book a perfect jumping-off point for readers looking to work with those systems.

—Paul Dix, Series Editor

This page intentionally left blank
 Preface

We first discovered deep reinforcement learning (deep RL) when DeepMind achieved
breakthrough performance in the Atari arcade games. Using only images and no prior
knowledge, artificial agents reached human-level performance for the first time.
The idea of an artificial agent learning by itself, through trial and error, without
supervision, sparked something in our imaginations. It was a new and exciting approach to
machine learning, and it was quite different from the more familiar field of supervised
learning.
We decided to work together to learn about this topic. We read books and papers,
followed online courses, studied code, and tried to implement the core algorithms. We
realized that not only is deep RL conceptually challenging, but that implementation
requires as much effort as a large software engineering project.
As we progressed, we learned more about the landscape of deep RL—how algorithms
relate to each other and what their different characteristics are. Forming a mental model of
this was hard because deep RL is a new area of research and the theoretical knowledge had
not yet been distilled into a book. We had to learn directly from research papers and online
lectures.
Another challenge was the large gap between theory and implementation. Often, a deep
RL algorithm has many components and tunable hyperparameters that make it sensitive
and fragile. For it to succeed, all the components need to work together correctly and with
appropriate hyperparameter values. The implementation details required to get this right
are not immediately clear from the theory, but are just as important. A resource that
integrated theory and implementation would have been invaluable when we were learning.
We felt that the journey from theory to implementation could have been simpler than
we found it, and we wanted to contribute to making deep RL easier to learn. This book is
our attempt to do that. It takes an end-to-end approach to introducing deep RL—starting
with intuition, then explaining the theory and algorithms, and finishing with
implementations and practical tips. This is also why the book comes with a companion
software library, SLM Lab, which contains implementations of all the algorithms discussed
in it. In short, this is the book we wished existed when we were starting to learn about this
topic.
Deep RL belongs to the larger field of reinforcement learning. At the core of
reinforcement learning is function approximation; in deep RL, functions are learned using
deep neural networks. Reinforcement learning, along with supervised and unsupervised
learning, make up the three core machine learning techniques, and each technique differs
in how problems are formulated and how algorithms learn from data.
In this book we focus exclusively on deep RL because the challenges we experienced
are specific to this subfield of reinforcement learning. This bounds the scope of the book
xxii Preface

in two ways. First, it excludes all other techniques that can be used to learn functions
in reinforcement learning. Second, it emphasizes developments between 2013 and
2019 even though reinforcement learning has existed since the 1950s. Many of the
recent developments build from older research, so we felt it was important to trace the
development of the main ideas. However, we do not intend to give a comprehensive
history of the field.
This book is aimed at undergraduate computer science students and software engineers.
It is intended to be an introduction to deep RL and no prior knowledge of the subject is
required. However, we do assume that readers have a basic familiarity with machine
learning and deep learning as well as an intermediate level of Python programming. Some
experience with PyTorch is also useful but not necessary.
The book is organized as follows. Chapter 1 introduces the different aspects of a deep
reinforcement learning problem and gives an overview of deep reinforcement learning
algorithms.
Part I is concerned with policy-based and value-based algorithms. Chapter 2 introduces
the first Policy Gradient method known as REINFORCE. Chapter 3 introduces the first
value-based method known as SARSA. Chapter 4 discusses the Deep Q-Networks
(DQN) algorithm and Chapter 5 focuses on techniques for improving it—target
networks, the Double DQN algorithm, and Prioritized Experience Replay.
Part II focuses on algorithms which combine policy-based and value-based methods.
Chapter 6 introduces the Actor-Critic algorithm which extends REINFORCE.
Chapter 7 introduces Proximal Policy Optimization (PPO) which can extend
Actor-Critic. Chapter 8 discusses synchronous and asynchronous parallelization techniques
that are applicable to any of the algorithms in this book. Finally, all the algorithms are
summarized in Chapter 9.
Each algorithm chapter is structured in the same way. First, we introduce the main
concepts and work through the relevant mathematical formulations. Then we describe
the algorithm and discuss an implementation in Python. Finally, we provide a configured
algorithm with tuned hyperparameters which can be run in SLM Lab, and illustrate the
main characteristics of the algorithm with graphs.
Part III focuses on the practical details of implementing deep RL algorithms.
Chapter 10 covers engineering and debugging practices and includes an almanac of
hyperparameters and results. Chapter 11 provides a usage reference for the companion
library, SLM Lab. Chapter 12 looks at neural network design and Chapter 13 discusses
hardware.
The final part of book, Part IV, is about environment design. It consists of Chapters 14,
15, 16, and 17 which treat the design of states, actions, rewards, and transition functions
respectively.
The book is intended to be read linearly from Chapter 1 to Chapter 10. These chapters
introduce all of the algorithms in the book and provide practical tips for getting them to
work. The next three chapters, 11 to 13, focus on more specialized topics and can be read
 Preface xxiii

in any order. For readers that do not wish to go into as much depth, Chapters 1, 2, 3, 4, 6,
and 10 are a coherent subset of the book that focuses on a few of the algorithms. Finally,
Part IV contains a standalone set of chapters intended for readers with a particular interest
in understanding environments in more depth or building their own.
SLM Lab [67], this book’s companion software library, is a modular deep RL
framework built using PyTorch [114]. SLM stands for Strange Loop Machine, in homage
to Hofstadter’s iconic book Gödel, Escher, Bach: An Eternal Golden Braid [53]. The specific
examples from SLM Lab that we include use PyTorch’s syntax and features for training
neural networks. However, the underlying principles for implementing deep RL
algorithms are applicable to other deep learning frameworks such as TensorFlow [1].
The design of SLM Lab is intended to help new students learn deep RL by organizing
its components into conceptually clear pieces. These components also align with how deep
RL is discussed in the academic literature to make it easier to translate from theory to code.
Another important aspect of learning deep RL is experimentation. To facilitate this,
SLM Lab also provides an experimentation framework to help new students design and
test their own hypotheses.
The SLM Lab library is released as an open source project on Github. We encourage
readers to install it (on a Linux or MacOS machine) and run the first demo by following
the instructions on the repository website https://github.com/kengz/SLM-Lab. A
dedicated git branch “book” has been created with a version of code compatible with this
book. A short installation instruction copied from the repository website is shown in
Code 0.1.

Code 0.1 Installing SLM-Lab from the book git branch

1 # clone the repository

2 git clone https://github.com/kengz/SLM-Lab.git
3 cd SLM-Lab
4 # checkout the dedicated branch for this book
5 git checkout book
6 # install dependencies
7 ./bin/setup
8 # next, follow the demo instructions on the repository website

We recommend you set this up first so you can train agents with algorithms as they are
introduced in this book. Beyond installation and running the demo, it is not necessary to
be familiar with SLM Lab before reading the algorithm chapters (Parts I and II)—we give
all the commands to train agents where needed. We also discuss SLM Lab more extensively
in Chapter 11 after shifting focus from algorithms to more practical aspects of deep
reinforcement learning.
xxiv Preface

Register your copy of Foundations of Deep Reinforcement Learning on the InformIT site for
convenient access to updates and/or corrections as they become available. To start the reg-
istration process, go to informit.com/register and log in or create an account. Enter the
product ISBN (9780135172384) and click Submit. Look on the Registered Products tab
for an Access Bonus Content link next to this product, and follow that link to access any
available bonus materials. If you would like to be notified of exclusive offers on new editions
and updates, please check the box to receive email from us.
 Acknowledgments

There are many people who have helped us finish this project. We thank Milan Cvitkovic,
Alex Leeds, Navdeep Jaitly, Jon Krohn, Katya Vasilaky, and Katelyn Gleason for supporting
and encouraging us. We are grateful to OpenAI, PyTorch, Ilya Kostrikov, and Jamromir
Janisch for providing high-quality open source implementations of different components
of deep RL algorithms. We also thank Arthur Juliani for early discussions on environment
design. These resources and discussions were invaluable as we were building SLM Lab.
A number of people provided thoughtful and insightful feedback on earlier drafts of this
book. We would like to thank Alexandre Sablayrolles, Anant Gupta, Brandon Strickland,
Chong Li, Jon Krohn, Jordi Frank, Karthik Jayasurya, Matthew Rahtz, Pidong Wang,
Raymond Chua, Regina R. Monaco, Rico Jonschkowski, Sophie Tabac, and Utku Evci
for the time and effort you put into this. The book is better as a result.
We are very grateful to the Pearson production team—Alina Kirsanova, Chris Zahn,
Dmitry Kirsanov, and Julie Nahil. Thanks to your thoughtfulness, care, and attention to
detail the text has been greatly improved.
Finally, this book would not exist without our editor Debra Williams Cauley. Thank
you for your patience and encouragement, and for helping us to see that writing a book
was possible.
This page intentionally left blank
 About the Authors

Laura Graesser is a research software engineer working in robotics at Google. She has a
masters in computer science from New York University where she specialized in machine
learning.
Wah Loon Keng is an AI engineer who applies deep reinforcement learning to industrial
problems at Machine Zone. He has a background in theoretical physics and computer
science.
Together, they have developed two deep reinforcement learning software libraries and have
given a number of talks and tutorials on the subject.
This page intentionally left blank
 1
Introduction to Reinforcement
Learning

In this chapter we introduce the main concepts in reinforcement learning. We start by

looking at some simple examples to build intuitions about the core components of a
reinforcement learning problem—namely, an agent and an environment.
In particular, we will look at how an agent interacts with an environment to optimize
an objective. We will then define these more formally and define reinforcement learning as
a Markov Decision Process. This is the theoretical foundation of reinforcement learning.
Next, we introduce the three primary functions an agent can learn—a policy, value
functions, and a model. We then see how learning these functions gives rise to different
families of deep reinforcement learning algorithms.
Finally, we give a brief overview of deep learning, which is the function approximation
technique used throughout this book, and discuss the main differences between
reinforcement learning and supervised learning.

1.1 Reinforcement Learning

Reinforcement learning (RL) is concerned with solving sequential decision-making
problems. Many real-world problems—playing video games, sports, driving, optimizing
inventory, robotic control—can be framed in this way. These are things that humans and
machines do.
When solving these problems, we have an objective or goal—such as winning the
game, arriving safely at our destination, or minimizing the cost of building products. We
take actions and get feedback from the world about how close we are to achieving the
objective—the current score, distance to our destination, or price per unit. Reaching our
goal typically involves taking many actions in sequence, each action changing the world
around us. We observe these changes in the world as well as the feedback we receive before
deciding on the next action to take as a response.
Imagine the following scenario: you are at a party where a friend brings out a flag pole
and challenges you to balance it on your hand for as long as possible. If you have never
held a flag pole before, your initial attempts will not be very successful. You may spend the
Exploring the Variety of Random
Documents with Different Content
who survived were afterwards thoroughly converted to the gospel of Jesus
Christ by his advent and the ministry of his servants, so that there was
inaugurated an era of peace and perfect righteousness. For two centuries at
least there was a veritable golden age in the American continents, during
which time the simple laws of righteousness promulgated by the gospel
were all sufficient as a rule of conduct, and men practically forgot the reign
of kings and the reign of judges. When wickedness once more began to
stalk through the land it may be that the hitherto prevailing ecclesiastical
governments gave way to the rule of military chieftains, both among the
Nephites and Lamanites, though among the later such chieftains were
sometimes called "kings."

That the monarchial and republican forms of government described in the

Book of Mormon should be in harmony with the principles of those ancient
political systems, and that the kind of government which obtained after the
advent of Messiah among the Nephites should be in such perfect harmony
with the conditions that obtained in that period, is internal evidence of
marked significance in support of the claims of the Book of Mormon. To
see it in its full strength one should ask himself what would be the state of
the case if the descriptions of monarchial and democratic government were
not in harmony with the restricted ideas of ancient governments, but were
full of modern ideas and refinements of government; and if the facts
existing after the advent of Messiah and the introduction of the Nephite
golden age were utterly at variance with the kind of government that we are
ready to believe then obtained. It should be remembered that if
inconsistencies in the Book of Mormon forms of government would be so
damaging against its claims to being an ancient record, then consistency in
its forms of government should be allowed equal weight in support of its
claims to being an ancient record.

The Events to which Importance is Given in the Book

of Mormon are in Harmony with the Character of the
Writers.
In considering this subject we must bear in mind the purposes for which the
Book of Mormon was written. The purposes are set forth in detail in chapter
III.

Here it will be sufficient to say that the main purpose of the Book of
Mormon is to be a witness for Jesus, the Christ; for the truth of the Gospel
as the power of God unto salvation.

Notwithstanding these purposes are adhered to throughout the work it is

very noticeable, and indeed one cause of complaint against the book, that it
gives great prominence, at least in the parts made up of Mormon and
Moroni's abridgments, to wars; to minute descriptions of battles, the
construction of fortifications, and the affairs of war in general. This
doubtless arises from the fact that Mormon and Moroni were both military
chieftains, and notwithstanding their general purpose was to make
prominent the religious events which happened among the Nephites and
Jaredites, and the hand-dealings of God with those peoples, yet when these
writers came to give an account of wars, it is but to be expected, by the very
nature of things, that they could not refrain from recording those events
which would have such a powerful attraction for them. Involuntarily they
were drawn into a description of those events, and unconsciously gave them
prominence in their narratives. So I say the events to which importance is
given in the Book of Mormon are in harmony with the character of the
writers, a fact which is still further emphasized by the nature of the first part
of the volume. We have seen that 149 of the 157 pages constituting that first
part is written by the first Nephi and his brother Jacob, prophets and priests
of God. In their writings wars are mentioned only in the most incidental
way, but there is an abundance of religious teaching, and prominence is
given to visions, dreams and revelations, and that because those writers
were, in the main, prophets and priests of God. It should also be noted, of
course, that the time in which these earlier writers lived was not so much a
period of warfare as subsequent centuries among the Nephites. It is to be
observed, then, in conclusion upon this point, that the very prominence
given to wars and battle-movements in Mormon's and Moroni's part of the
volume is but in keeping with the nature of things—an additional evidence
of consistency in the work—the events to which importance is given are in
harmony with the character of the writers.
 Complexity in the Structure of the Book of Mormon in
Harmony with the Theory of its Origin.

I hesitated some time before adopting the above as a heading for this
division of the subject, because I was aware, and am still aware of the fact
that it scarcely presents the thought I would have considered; and I know
how easily, by a slight variation, it could be made subject to the smart retort
that the complexity of the structure of the Book of Mormon is in harmony
with the theory of its merely human origin since it is simplicity, not
complexity, which is the sign manual of things divine. Still, for all that, I
have concluded to make use of this faulty title, for want of a better,
confident that when my whole thought under it is developed it will result in
producing evidence for the truth of the claims of the book.

That the structure of the Book of Mormon is complex all who read it know.
The first part of it is made up of the translation of unabridged records, the
small plates of Nephi. The second part is made up of the translation of
abridged books (Mormon's abridgment), Mormon, however, retaining for
the several parts of his abridgment the title of the respective books he
abridged.

I have already pointed out the fact[10] that Mormon's condensed narrative
from the original Nephite records makes up the body of his work; with
occasional direct quotations from the original records, and the whole more
or less confused by his running comments, unseparated from the body of his
work save by the sense of the text. All this is complex enough surely, but
the end is not yet; for within the old Nephite records Mormon had at hand
while doing the work of abridgment, there were still other books. That is,
books within books; as, for instance, the Book of Zeniff within the Book of
Mosiah, which see. [11] Also the account of the church founded by the first
Alma, likewise within the book of Mosiah. Also the account of the
missionary expedition to the Lamanites by the young Nephite princes, sons
of King Mosiah II., within the book of Alma, which see.[12] Mormon,
coming to these books within books, followed that order also in his
abridgment; so that as in the original Nephite records, we have books within
books, so within Mormon's abridgment we have abridged records within
abridged records. Then, as if to cap the climax of complexity in structure,
Mormon writes a book of his own to which he gives his own name. That is,
calls it the Book of Mormon; the last two chapters of which, however, are
written by Moroni. Then follows what may be called the third part of the
Book of Mormon—Moroni's abridgment of the twenty-four plates of Ether,
which gives us so much of the history as we have of the Jaredites. By this
arrangement the history of the first people to occupy the western
hemisphere, (after the flood), comes last in the Book of Mormon; and
Moroni's abridgment of the Jaredite record has much of the complexity of
his father's abridgment of the Nephite records.

Now, with all this before the mind of the reader—whether he regards
Joseph Smith, Solomon Spaulding, or Sidney Rigdon as the author of the
Book of Mormon—I submit to him the question: Would either ingenuity or
stupidity in a modern author suggest such complexity in the structure of a
book as this? Can a parallel case be pointed to in the modern making of
books?

If the Book of Mormon were modern in structure and its author or authors
had the conception that this western world was peopled by a colony coming
from the Euphrates valley, in very ancient times, and subsequently by two
other colonies from Judea, one leaving 600 B. C. and the other shortly
afterwards, in giving the history of those people, would not the modern
author have begun with the most ancient colony and treated the history of
the respective peoples in the order of their occupancy of the western
continents? Then, again: If the Book of Mormon is mere fiction, the idle
coinage of an inventive, modern author, why three migrations?

If the object of the modern author was merely to convey an idea how a
civilized race in ancient times occupied the western world, why would not
the first migration—the Jaredite—have answered all his purposes? Or why
not take the second migration—the Nephite—for the accomplishment of
such a purpose? Why complicate it by bringing in the migration of Mulek's
colony, when the simple treatment of developing the Nephite colony into
national proportions would have been sufficient for the purpose of a work
of fiction? One other question I would submit relative to the Jaredite record
and the strange place it occupies in the Book of Mormon. The plates of
Ether were found by an expedition sent out from Zeniff's colony about 123
B. C., and were translated shortly afterwards by Mosiah II., who was a seer;
that is, he was able to use Urim and Thummim in the translation of strange
languages. Now, why did not Mormon include an abridgment of Mosiah's
translation of the plates of Ether in his abridgment of Nephite records,
allowing it to stand in his collection of plates as his abridgment of the Book
of Zeniff stands within his abridgment of the Book of Mosiah, instead of
passing the matter by and leaving it for his son Moroni to make a
translation direct from the Book of Ether, thus throwing the history of the
first inhabitants of the western world, after the flood, to the very last part of
the record? Candidly, does the complex structure of the Book of Mormon
appeal to one as at all modern in its arrangement? Are modern books so
constructed? And yet, notwithstanding all the complexity in the structure of
the book, each part is so in harmony with every other part, and with the
whole, that really, after all, it is a very simple book, and one readily
understood. It is clear that the very peculiar circumstances under which the
Book of Mormon was compiled by the original Nephite writers, and that
neither the ingenuity nor the stupidity of Joseph Smith, nor of any other
modern writer, is responsible for this peculiar structure of the book. And,
moreover, since the book in its details retains harmonious consistency with
the plan of its structure, must not such a fact be conceded to be an
incidental evidence in favor of its claims?

Footnotes

1. Ether vi: 27. Ibid. ix: 15-22. Ibid. x: 10 et. seq.

2. Ether ix: 14, 15, 21, 22. Ibid. x: 13.

3. Mosiah xxix: 21-23. See also remarks, chapters x, and xiii.

4. Perhaps it may be thought that an exception should be made in the matter

of Lamanite kingdoms, of which I have spoken (chapter xiii) as constituting
at one period of Lamanite history, a sort of confederacy of kingdoms; but
this does not affect the statement of the text which is dealing with the form
of government. I believe myself justified in saying that whether reference is
made to the petty Lamanite kingdoms or the central kingdom to which they
were tributary, the principle in government will be found the same—the
king is the source of all political power, the monarchy is "simple," the
kingly power absolute.

5. Doc. & Cov., Sec. ci: 76-80.

6. Letters of Joseph Smith, from Liberty Prison, under date of March 25,
1839—to the Church of the Latter-day Saints. History of the Church, Vol.
III., p. 304.

7. See De Tocqueville's Constitution of the U. S., Vol. I.

8. See Chapter xiii.

9. Ante pp. 216-7.

10. See Ante Chapter xxxvii.

11. Book of Mosiah, p. 181 (current edition).

12. Book of Mormon, p 283 (current edition).

 CHAPTER XXXIX
INTERNAL EVIDENCES—THE
ORIGINALITY OF THE BOOK OF
MORMON AN EVIDENCE IN SUPPORT
OF ITS CLAIMS.

How far originality may be insisted upon as a necessary element in a book

avowedly containing a revelation from God is an open question; just as how
far originality in a prophet may be insisted upon is. In both cases, however,
it cannot be doubted but that originality would be regarded as evidence of
considerable weight in favor of the divinity of the message of either prophet
or book. Somehow men look for originality in any thing that purports to be
a revelation from God, come how it will. They look for a word "from the
inner fact of things" in a revelation. A new word that shall add somewhat to
the sum of known things, and spoken in a way to attract anew the attention
of men. And yet it must not be forgotten that "every scribe which is
instructed unto the kingdom of heaven * * * bringeth forth out of his
treasure things new and old"[1]—the old, mark you, as well as the new—and
one of olden time doubted even if there really was any new thing under the
sun. "The thing that hath been, it is that which shall be; and that which is
done is that which shall be done: and there is no new thing under the sun. Is
there any thing whereof it may be said, see, this is new? It hath been already
of old time, which was before us."[2]

From all which I conclude that while in a way originality may be regarded
as affording some evidence in favor of the claims of a prophet and his
message, or of a book and its revelation, still originality is not an
indispensable quality in either prophet or book. Contemporary prophets, or
prophets living in succession, may come burdened with the same word of
the Lord, with the same divine message; but the one who speaks secondly
or thirdly, and hence with all claim to originality gone, is none the less
God's messenger; and the word he speaks may not with safety be rejected
for that it lacks the quality of originality. So, too, with books. It would be a
senseless manner of handling the scriptures to reject the books called first
and second Chronicles because they chiefly duplicate the matter of the
books called first and second Kings, and have little originality to commend
them to our acceptance. So with the books of the New Testament. Accepting
for our purpose here the order in which they stand in the commonly
received versions of the New Testament, as the order in which the books
were written, shall the book of Mark be rejected because in the main it deals
with the same matter that engages the attention of Matthew, and there is but
little on the score of its originality of matter to commend it as an inspired
book? The same question could be asked in relation to the book of Luke.
The truth is that God in books as in prophets sometimes requires more than
one for a witness to his message, and hence repeats the revelation in a
number of inspired books, in which case the books merely repeating the
revelation are as truly inspired, as truly scripture as the one in which the
message first appeared, although it could be said that the quality of
originality is wholly wanting.

Since the Book of Mormon feigns the introduction of no new religion, but
gives merely an account of the introduction of the Christian religion in the
western hemisphere, by inspired teachers, both before and after the coming
of Messiah, and by the personal ministry of Messiah after his resurrection;
and as the Christian religion is always the same, in all times and in all lands,
it must have been the same when introduced into America as when taught in
Judea—where is room for originality? Is not originality by the very nature
of the claims of the Book of Mormon excluded? The reader, I believe, will
recognize the force of the question; and I take occasion here to remark that
the point in the question exhibits the weakness of those objections that are
sometimes urged against the Book of Mormon on the score of sameness of
matter in it and the New Testament; and also it exhibits how senseless is the
clamor for the existence of some new moral or religious truth[3] in the Book
of Mormon, not to be found in the Old or New Testaments.

Since, then, the Book of Mormon, so far as it treats of religion, treats of the
Christian religion, it is comparison not contrast that should be made;
sameness, not difference that should be looked for; identity of moral and
religious truths, not differences; accordance with old truths, rather than the
existence of new ones. The Christian religion may not be contrasted with
itself; and as the fullness of the gospel was revealed in the proclamation of
it in Judea, it would be sufficient if a dispensation of the same gospel
proclaimed in America is in strict accordance with that taught in Judea. In
fact this is all that the nature of the case strictly requires. Still, after the
reasonableness of all this is established, there may be claimed for the Book
of Mormon an originality in the fact of the existence of new and important
Christian truths in its pages; as, also an originality of emphasis placed on
certain other Christian truths.

This much that a proper estimate may be formed of the value of originality
as an evidence of the divine authenticity or inspiration of a book; neither
giving an exaggerated value to it on the one hand, nor accounting it of little
or no importance on the other.

I.

Originality of Structure.

In enumerating the several particulars in which the Book of Mormon

manifests originality, I would name its peculiar structure—so at variance
with all modern ideas of book making—pointed out in the treatment of the
last subdivision of chapter xxxviii, and ask the reader to consider that
treatise brought over into this subdivision, and the peculiar structure of the
Book of Mormon made one, and the first, of the evidences of its originality.

II.

Originality in Names.

So also as to names; so far as they are original, I would have that fact
considered as another, the second, evidence of the originality of the Book of
Mormon; and so much of that treatise as deals with the originality of the
names, (see chapter xxxvii) considered as brought over into this
subdivision.

III.

In the Manner of its Coming Forth.

In the manner of its coming forth no less than in its structure and its names,
the Book of Mormon is original. It must be remembered that at the time of
the coming forth of the Book of Mormon such a thing as a new revelation
from God was utterly unlooked for. Indeed it was the consensus of Christian
opinion and teaching that the time of revelation had passed; that the days of
miracles were over; that God in the Christian dispensation to mankind (the
dispensation in which Messiah ministered in person) spoke the final word;
that no more divine communication would be given. Speculating upon this
very subject in connection with the desirability for knowledge respecting
the ancient inhabitants of America, Ethan Smith, in his "View of the
Hebrews; or the Tribes of Israel in America," says, most emphatically:

We are to expect no new revelation from heaven, and the days of

miracles are thought to be past. We probably must look for just such
evidence to exhibit to the world that people so long lost [as the ten
Tribes of Israel], as is in fact exhibited by the natives of America.[4]

It is well to remember that this was said some years before the Book of
Mormon was published, and I repeat that it represents the generally
accepted Christian idea concerning revelation and miracles. Furthermore, it
is notorious that the prime objection urged against the Book of Mormon
was the fact that it claimed to be a new revelation from God; and the
arguments found in the discourses and writings of the early Elders of the
Church clearly prove that the chief contention over the Book of Mormon in
those early days was on this point.[5] It follows, therefore, that Joseph
Smith's account of the manner in which the Book of Mormon was brought
forth and translated was a very original one; for it involved a revelation
from God to make known its existence, and what men call a miracle to
secure its translation. Here, then, was not only originality, but a bold
contradiction of what was supposed to be the most completely settled
doctrine of modern Christendom, viz. that the age of revelations and
miracles had forever passed away. It is scarcely probable that imposters
would move along such lines as these. The proclamation of a new
revelation making known the existence of a new volume of scripture was
the most remarkable innovation upon Christian opinion that the world had
ever witnessed. Orthodoxy stood aghast at the presumption as they called it;
and seemed for a time to forget all other points of controversy in order to
concentrate their attack upon this innovation of their most cherished idea.
They thought the very claim that the Book of Mormon involved a new
revelation from God was sufficient to justify its rejection. Yet never was
opposition so completely demolished in controversy as this sectarian
argument against new and continual revelation. So completely was it
overthrown that we to-day scarcely ever hear it mentioned. With this,
however, I have nothing further to do. My only point at present is that there
was a bold originality in Joseph Smith's account of the coming forth and
translation of the Book of Mormon, which, in addition to contravening the
accepted Christian opinion of the times on the subject of revelation and
miracles, carried with it much weight in support of the claims made for this
American volume of scripture; for surely imposters seeking to foist a book
upon the world either for obtaining fame or money would never be found
moving along lines so diametrically opposite to accepted opinions.

IV.

Its Accounting for the Peopling of America.

In its account of peopling America no less than in its structure and the
manner in which its existence was made known and its translation
accomplished, the Book of Mormon is original. All the books on American
antiquities that could possibly have been accessible to Joseph Smith and his
associates favored the theory of migrations from northeastern Asia by way
of Behring Straits where the Asiatic and American continents approach each
other. See Josiah Priest's American Antiquities, preface. Ethan Smith,
referring to the authorities that he was acquainted with on this subject, says:

All seem to agree that the Indians came from the northwest, and
overspread the continent to the south. * * * * * I forbear to offer any
further remarks upon these testimonies incidentally afforded by this
most celebrated author, [meaning Humboldt]. Let them be duly
weighed by the judicious reader; and he surely cannot doubt but that
the natives of America came from the north over Behring's Straits; and
descended from a people of as great mental cultivation, as were the
ancient family of Israel.[6]

Not only were such the prevailing views at the time Ethan Smith wrote,
1825, but even to this day the same general opinion prevails among
authorities;[7] that is, that America was peopled from Asia by way of
Behring Straits. The migrations of the Book of Mormon, however,
contravene this quite generally accepted theory. While it is supposed that
the Jaredites passed out of the Euphrates valley and wandered several years
eastwardly through Asia, they crossed the Pacific and landed in the south
part of the north continent of America and settled in a district of country
they afterwards called Moron, near what was afterwards the Nephite
province called Desolation, which was in the region of country known to us
as the Central American States.[8] The Nephite colony, as we have seen[9],
landed on the west coast of South America about thirty degrees south
latitude; and Mulek's colony is supposed to have landed somewhere in the
south part of the North American continent. These Book of Mormon
accounts of migrations to the American continents constitute the widest
possible departure from usually accepted theories upon the subject.

V.

The Nativity of Ancient American Peoples.

The Book of Mormon is original with reference to the facts it presents
respecting the nativity of its peoples. On this point, more is sometimes
claimed by believers in the Book of Mormon than is warranted by the facts
in the case. For example, it is sometimes stated that the Israelitish origin of
the native Americans was first asserted by the Book of Mormon. That is not
true. Long before the advent of the Book of Mormon James Adair, whose
work was published in 1775, advanced the theory that the native American
Indians were the Lost Ten Tribes of Israel, and argued for the truth of his
theory at great length.[10] Ethan Smith, in his work we have several times
quoted, advances the theory that the native Indians were the "Ten Lost
Tribes of Israel," the very title of his book—"View of the Hebrews; or the
Tribes of Israel in America"—is the evidence of his holding that theory.

It is therefore a mistake to say that the idea of Israelitish descent of the

native American Indians originated with the Book of Mormon. Indeed the
theory that the native Americans were the Ten Lost Tribes of Israel found
many advocates both in Europe and the United States, especially, I may say,
in the New England states, before 1830. Wherein the Book of Mormon is
original in respect of this matter is that while declaring the Israelitish
descent of the ancient people of America, it directly contravenes the idea
that the native Americans, are the Ten Lost Tribes of Israel, by incidentally
declaring those tribes to be in another part of the world, and Jesus
announcing to the Nephites his intention to appear unto them, and
administer among them.[11] Of course reference to Israelitish descent is here
made to the two last migrations only, that is, to the colony of Lehi, and the
colony of Mulek. The colony of Jared were doubtless of the same race, but
of earlier ancestors, among whom the patriarch Shem. The Book of
Mormon refers to Lehi's colony as made up of descendants of Manasseh
[Lehi] and Ephraim [Ishmael][12] while the colony of Mulek were Jews.

From this it appears that the Book of Mormon is as boldly original in

declaring the nativity of these colonies that peopled America with teeming
millions of their descendants, as it is in its account of the course of their
migrations or the manner in which the Book of Mormon came forth. For, in
limiting the nativity of these colonies to the descendants of Joseph and of
Judah, it as radically contravenes existing opinions upon the subject as it
does in respect of the manner in which the book came forth, and the course
of migration.

VI.

Accounting for the Existence of Christian Ideas in

America.

The Book of Mormon is original in the matter of accounting for the

existence of Christian ideas and doctrines among the native Americans. I
would have this statement so understood as to include all Bible ideas, since
right conceptions of Christianity in its fullness includes the Old Testament
and the dispensation of God to the children of men described therein as part
of the Christian heritage, as well as the specific Christian dispensation
which is described in the New Testament.

The manner in which the Book of Mormon accounts for Christian ideas and
doctrines among native Americans is, first, by detailing the facts of direct
revelation of Christian truths to the ancient inhabitants of America, as, for
instance, in the case of the Prophet Moriancumer among the Jaredites,
where that great prophet is represented as being permitted to stand in the
revealed presence of the preexisting spirit of Jesus Christ, and to hear the
proclamation that in him should all mankind have life and that eternally;
and that as he appeared unto that prophet in the spirit, so would he appear
unto his people in the flesh; and that those who would believe on his name
should become his sons and daughters.[13] Also the revelation of Christian
truths vouchsafed to the first Nephi; who, in vision, some hundreds of years
before the advent of Christ, was permitted to foresee the birth of the
Redeemer, the labors of his forerunner, John the Baptist, who prepared the
way before him, and much of the Judean ministry of Christ, including his
crucifixion, his resurrection, and the establishment of his ministry through
twelve Apostles; so also his advent and ministry among the inhabitants of
the western world,[14] ending in the establishment of the Christian
sacraments, and of the Christian Church, as the sacred depository of
Christian truths. Secondly, the Book of Mormon accounts for the existence
of Christian ideas and doctrines among native American races by declaring
the Nephites to be in possession of the Hebrew scriptures extant among that
people from the beginning up to 600 B. C., including the five books of
Moses, some of the writings of Isaiah and Jeremiah.[15] And also ascribing
to the Jaredites the knowledge of most ancient events through scriptures in
their possession, dealing with events from the Tower of Babel back to the
very days of Adam.[16] It is, then, by most direct means of the revelations of
God to the ancient inhabitants of America and the personal ministration of
Jesus Christ among them and the knowledge imparted by these several
volumes of very ancient scripture that the Book of Mormon accounts for the
existence of Christian ideas and Christian truths among the native
Americans.

There is nothing like this in the theories of men to account for the existence
of these truths in America. In the first place let the reader be assured that it
is quite generally conceded by the very best authorities that ideas closely
analogous to Christian truths are found in the traditions of the native
Americans. "Most ancient and modern authors," says De Roo, "agree in
saying that the Christian religion has been taught on our [the American]
continent at an epoch not so very much anterior to the Columbian
discovery. Bastian establishes the latter opinion by the numerous analogies
he points out between the religious belief and practices of the Christians
and those of American aborigines. Von Humboldt admits the parity to be so
striking as to have given the Spanish missionaries a fine opportunity to
deceive the natives by making them believe that their own was none other
than the Christian religion. 'Not a single American missionary who has,
until this day, left any writing has forgotten to notice the evident vestiges of
Christianity which has in former time penetrated even among the most
savage tribes,' says Dr. de Mier, commenting on Sahagun's History. Quite a
number of ancient writers, such as Garcilasso de la Vega, Solorzano,
Acosta, and others are equally explicit in asserting that several Christian
tenets and practices were found among our aborigines; but they deny their
introduction by Christian teachers, giving, strange to say, to the devil the
honor of spreading the light of Christianity, in spite of his hatred of it."[17]
Later he says:
 No modern student of American antiquity fails to notice the close and
striking resemblances between several leading particulars of Christian
faith, morals, and ceremonies and those of ancient American religions.
Sahagun, who wrote in Mexico about the middle of the sixteenth
century, and took such great pains to be correctly informed in regard to
all religious rites of our aborigines, states already that all the Spanish
missionaries who wrote in America before him had pointed out the
numerous vestiges of Christianity to be found even among the savage
Indian tribes.[18]

Devil propaganda of Christianity was quite a favorite theory with many of

the early Spanish writers, while others advanced the theory that Christian
apostles had evangelized the western hemisphere. Among the latter was the
Archbishop of San Domingo, Davilla Padilli, a royal chronicler who wrote
a book to prove that Christian apostles had formerly preached in the West
Indies. So also Torquemada holds the same opinion, although he admits of
the possibility of the devil teaching Christianity. More modern writers seek
to account for the existence of these Christian analogies in other ways.
Prescott for instance, in his Conquest of Peru, says:

In the distribution of bread and wine at this high festival, [the feast of
Raymi] the orthodox Spaniards who first came into the country saw a
striking resemblance to the Christian communion; as in the practice of
confession and penance, which, in a most irregular form indeed, seems
to have been used by the Peruvians, they discerned a coincidence with
another of the sacraments of the Church. The good fathers were fond
of tracing such coincidences, which they considered as the contrivance
of Satan, who thus endeavored to delude his victims by counterfeiting
the blessed rites of Christianity. Others, in a different vein, imagined
that they saw in such analogies the evidence that some of the primitive
teachers of the gospel, perhaps an apostle himself, had paid a visit to
these distant regions and scattered over them the seeds of religious
truth. But it seems hardly necessary to invoke the Prince of Darkness,
or the intervention of the blessed saints to account for coincidences
which have existed in countries far removed from the light of
Christianity, and in ages, indeed, when its light had not yet risen on the
world. It is much more reasonable to refer such casual points of
 resemblance to the general constitution of man and the necessities of
his moral nature.[19]

Of which I think De Roo very justly remarks: "The Christian mysteries

admitted by the ancient Peruvians and Mexicans could hardly find their
origin in man's constitution; nor are religious practices, like baptism,
fasting, celibacy, and a cloistered life, to be considered as necessities of
man's moral, yet corrupt nature. More reasonable and better historical
causes should be found to account for the presence of Christian faith and
Christian rites in ancient America."[20]

H. H. Bancroft also concedes the existence of rites among native Americans

analogous to those existing among Jews and Christians, but regards them as
mere coincidences. He says:

Many rites and ceremonies were found to exist among the civilized
nations of America that were very similar to certain others observed by
the Jews and Christians in the old world. The innumerable speculators
on the origin of the aboriginal inhabitants of the new world, or at least
on the origin of their civilization, have not neglected to bring forward
these coincidences—there is no good reason to suppose them anything
else—in support of their various theories.[21]

On which De Roo remarks: "Coincidences, so many, so striking, in faith, in

morals, and liturgy! Coincidences, indeed, little short of wonders!"

Nadaillac also would refer these "coincidences" to natural causes. He says

"No dissemination of merely Christian ideas, since the conquest [by the
Spaniards] is sufficient to account for these myths [having in mind the
traditions of the creation, flood, migrations, Christian analogies, etc.],
which appear to have their root in the natural tendencies of the human mind
in its evolution from a savage state."[22]

And so in these various ways men would account for the existence of
Christian ideas and doctrines; but it was reserved for Joseph Smith, the
Prophet of the dispensation of the fullness of times, through the Book of
Mormon, to announce the boldly original idea that knowledge of Christian
truths and doctrines had their origin among native American peoples in
direct revelation to them from God; in the personal ministration of the Lord
Jesus Christ, after his resurrection from the dead; and from being in
possession of ancient scriptures which to the Nephites, no less than to the
Jews, made known God's plan of redemption for mankind through the
personal suffering and resurrection of his Son Jesus Christ. I hold that the
very originality and boldness of these assertions respecting the direct means
by which the people of America in ancient times received their knowledge
of Christian truths, and which so far transcend the timid and tentative
speculations of men, even of the most intelligent and courageous, have
about them an atmosphere of truth that is most convincing; moreover, I
cannot help but believe that originality in respect of such things as are here
set down; structure, names, the manner of coming forth, in its account of
peopling America, the nativity of American people, and lastly this
accounting for the existence of Christian ideas among native American
races, is of a vastly greater importance than originality in mere phraseology
or style of composition.

Footnotes

1. Matt. xiii: 52.

2. Ecclesiastes i: 9, 10.

3. For this clamor see a brief discussion on the Book of Mormon between
the writer and an "Unknown" writer in "The Salt Lake Tribune,"
impressions of Nov. 22, 29; and December 6, 13, 1903. See also "The
Golden Bible" (Lamb), Edition of 1887, p. 207-213. Also the views of the
Rev. Dr. Wm. M. Paden, of the First Presbyterian Church; Salt Lake City,
quoted by the writer in a discourse on "The Fifth Gospel"—Third Nephi
—"Defense of the Faith and the Saints," Vol. I, pp. 373-399.

4. View of the Hebrews, 2nd Edition, (1825) pp. 168, 169.

5. See the works of Orson and Parley P. Pratt; John Taylor's Discussion with
three ministers in France; early volumes of Millennial Star, Spencer's
Letters—in fact all the early Church literature. Of late opponents of the
Book of Mormon have not pressed this point of controversy, since the
sectarian arguments respecting it have been utterly demolished. For a brief
consideration of the various points of that argument see "New Witnesses for
God," Vol. I., Ch. viii.

6. View of the Hebrews, pp. 187, 188

7. See chapter xxix, especially taking account of foot note references.

8. Dictionary of the Book of Mormon, Reynolds, p. 168. And Vol. II, pp.
139-40.

9. Vol. II, pp. 157-8.

10. See this volume, pp. 46-48.

11. See III. Nephi xv, xvi, xvii.

12. The statement here that Ishmael was of Ephraim is set down upon the
authority, first, of inference. The inference is based upon the fact as already
stated that there are promises in the Hebrew scriptures respecting Ephraim
which cannot be realized so far as we know, except through the seed of
Ephraim dwelling upon the land of America, as we have seen in considering
the evidence of the Bible for the truth of the Book of Mormon; and as Lehi
and his family were of the tribe of Manasseh, and Mulek's colony being
Jews, it leaves the family of Ishmael, and perhaps Zoram, the servant of
Laban to introduce the descendants of Ephraim into the western world.
Second, a number of Latter-day Saints, familiarly acquainted with the
Prophet Joseph Smith, declare that in conversation they had known him to
say that in Mormon's abridgment of the book of Lehi (which supplied the
116 pages of manuscript lost by Martin Harris) it was plainly stated that
Ishmael was of the tribe of Ephraim. Among those who heard such remarks
was the late Elder Franklin D. Richards, of the Council of the Apostles and
Church Historian, who records his recollection of the Prophet's statement in
the following manner:

"One day in the autumn of that year, (1843) as I was passing near, (the
"Nauvoo Mansion") it being in warm weather, I observed the door standing
open and the Prophet Joseph inside conversing with one of the brethren,
leaning against the counter. It being a public house, I ventured to walk in,
and scarcely had more than time to exchange usual civilities, when this
brother said: 'Brother Joseph, how is it that we call the Book of Mormon the
Stick of Joseph, in the hands of Ephraim, when the book itself tells us that
Lehi was of the lineage of Manasseh? I cannot find in it about the seed of
Ephraim dwelling on this land at all.' Joseph replied: 'You will recollect that
when Lehi and his family had gone from Jerusalem out into the wilderness,
he sent his son Nephi back to the city to get the plates which contained the
law of Moses and many prophecies of the prophets, and that he also brought
out Ishmael and his family, which were mostly daughters. This Ishmael and
his family were of the lineage of Ephraim, and Lehi's sons took Ishmael's
daughters for wives, and this is how they have grown together, 'a multitude
of nations in the midst of the earth.'

"'If we had those one hundred and sixteen pages of manuscript which
Martin Harris got away with, you would know all about it, for Ishmael's
ancestry is made very plain therein. The Lord told me not to translate it over
again, but to take from Nephi's other plates until I came to the period of
time where the other translation was broken off, and then go on with
Mormon's abridgment again. That is how it came about that Ishmael's
lineage was not given in the Book of Mormon, as well as Lehi's."'—
Frankling D. Richards, "The Contributor," Vol. XVII, p. 425.

13. Ether iii.

14. I. Nephi x: 11, 12.

15. See I. Nephi v: 11.

16. Ether i: 3-6.

17. History of America Before Columbus, P. De Roo, Vol. I., 423, 424.

18. Ibid. p. 517.

19. Conquest of Peru, Vol. I., pp. 96, 97.

20. History of America Before Columbus, Vol. I., pp. 523-4.

21. Native Races, Vol. III., pp. 438-9

22. Prehistoric America, p.531.

Welcome to our website – the ideal destination for book lovers and
knowledge seekers. With a mission to inspire endlessly, we offer a
vast collection of books, ranging from classic literary works to
specialized publications, self-development books, and children's
literature. Each book is a new journey of discovery, expanding
knowledge and enriching the soul of the reade

Our website is not just a platform for buying books, but a bridge
connecting readers to the timeless values of culture and wisdom. With
an elegant, user-friendly interface and an intelligent search system,
we are committed to providing a quick and convenient shopping
experience. Additionally, our special promotions and home delivery
services ensure that you save time and fully enjoy the joy of reading.

Let us accompany you on the journey of exploring knowledge and

personal growth!

textbookfull.com