Download the full version of the textbook now at textbookfull.

com

Hands On Machine Learning with Scikit Learn

and TensorFlow Aurélien Géron

https://textbookfull.com/product/hands-on-machine-
learning-with-scikit-learn-and-tensorflow-
aurelien-geron/

Explore and download more textbook at https://textbookfull.com

 Recommended digital products (PDF, EPUB, MOBI) that
you can download immediately if you are interested.

Hands On Machine Learning with Scikit Learn and TensorFlow

1st Edition Aurélien Géron

https://textbookfull.com/product/hands-on-machine-learning-with-
scikit-learn-and-tensorflow-1st-edition-aurelien-geron/

textbookfull.com

Hands On Machine Learning with Scikit Learn and TensorFlow

Early Release 2nd Edition Aurélien Géron

https://textbookfull.com/product/hands-on-machine-learning-with-
scikit-learn-and-tensorflow-early-release-2nd-edition-aurelien-geron/

textbookfull.com

Hands On Machine Learning with Scikit Learn Keras and

TensorFlow Concepts Tools and Techniques to Build
Intelligent Systems 2nd Edition Aurelien Geron [Géron
https://textbookfull.com/product/hands-on-machine-learning-with-
scikit-learn-keras-and-tensorflow-concepts-tools-and-techniques-to-
build-intelligent-systems-2nd-edition-aurelien-geron-geron/
textbookfull.com

The Future of Risk Management, Volume I: Perspectives on

Law, Healthcare, and the Environment Paola De Vincentiis

https://textbookfull.com/product/the-future-of-risk-management-volume-
i-perspectives-on-law-healthcare-and-the-environment-paola-de-
vincentiis/
textbookfull.com
Integrated History and Philosophy of Science Problems
Perspectives and Case Studies 1st Edition Friedrich
Stadler (Eds.)
https://textbookfull.com/product/integrated-history-and-philosophy-of-
science-problems-perspectives-and-case-studies-1st-edition-friedrich-
stadler-eds/
textbookfull.com

Digitalization in Construction 1st Edition Angenette

Spalink

https://textbookfull.com/product/digitalization-in-construction-1st-
edition-angenette-spalink/

textbookfull.com

The European Convention on Human Rights : a commentary 1st

Edition William A. Schabas

https://textbookfull.com/product/the-european-convention-on-human-
rights-a-commentary-1st-edition-william-a-schabas/

textbookfull.com

Core Competencies of Relational Psychoanalysis A Guide to

Practice Study and Research 1st Edition Roy E. Barsness

https://textbookfull.com/product/core-competencies-of-relational-
psychoanalysis-a-guide-to-practice-study-and-research-1st-edition-roy-
e-barsness/
textbookfull.com

Heterogeneous Computing Hardware Software Perspectives 1st

Edition Mohamed Zahran

https://textbookfull.com/product/heterogeneous-computing-hardware-
software-perspectives-1st-edition-mohamed-zahran/

textbookfull.com
Introduction to Computation and Programming Using Python
With Application to Understanding Data Second Edition John
V. Guttag
https://textbookfull.com/product/introduction-to-computation-and-
programming-using-python-with-application-to-understanding-data-
second-edition-john-v-guttag/
textbookfull.com
Hands-On
Machine Learning
with Scikit-Learn
& TensorFlow
CONCEPTS, TOOLS, AND TECHNIQUES
TO BUILD INTELLIGENT SYSTEMS

Aurélien Géron
Hands-On Machine Learning with
Scikit-Learn and TensorFlow
Concepts, Tools, and Techniques to
Build Intelligent Systems

Aurélien Géron

Beijing Boston Farnham Sebastopol Tokyo

Hands-On Machine Learning with Scikit-Learn and TensorFlow
by Aurélien Géron
Copyright © 2017 Aurélien Géron. All rights reserved.
Printed in the United States of America.
Published by O’Reilly Media, Inc., 1005 Gravenstein Highway North, Sebastopol, CA 95472.
O’Reilly books may be purchased for educational, business, or sales promotional use. Online editions are
also available for most titles (http://oreilly.com/safari). For more information, contact our corporate/insti‐
tutional sales department: 800-998-9938 or [email protected].

Editor: Nicole Tache Indexer: Wendy Catalano

Production Editor: Nicholas Adams Interior Designer: David Futato
Copyeditor: Rachel Monaghan Cover Designer: Randy Comer
Proofreader: Charles Roumeliotis Illustrator: Rebecca Demarest

March 2017: First Edition

Revision History for the First Edition

2017-03-10: First Release

See http://oreilly.com/catalog/errata.csp?isbn=9781491962299 for release details.

The O’Reilly logo is a registered trademark of O’Reilly Media, Inc. Hands-On Machine Learning with
Scikit-Learn and TensorFlow, the cover image, and related trade dress are trademarks of O’Reilly Media,
Inc.
While the publisher and the author have used good faith efforts to ensure that the information and
instructions contained in this work are accurate, the publisher and the author disclaim all responsibility
for errors or omissions, including without limitation responsibility for damages resulting from the use of
or reliance on this work. Use of the information and instructions contained in this work is at your own
risk. If any code samples or other technology this work contains or describes is subject to open source
licenses or the intellectual property rights of others, it is your responsibility to ensure that your use
thereof complies with such licenses and/or rights.

978-1-491-96229-9
[LSI]
 Table of Contents

Preface. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xiii

Part I. The Fundamentals of Machine Learning

1. The Machine Learning Landscape. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3
What Is Machine Learning? 4
Why Use Machine Learning? 4
Types of Machine Learning Systems 7
Supervised/Unsupervised Learning 8
Batch and Online Learning 14
Instance-Based Versus Model-Based Learning 17
Main Challenges of Machine Learning 22
Insufficient Quantity of Training Data 22
Nonrepresentative Training Data 24
Poor-Quality Data 25
Irrelevant Features 25
Overfitting the Training Data 26
Underfitting the Training Data 28
Stepping Back 28
Testing and Validating 29
Exercises 31

2. End-to-End Machine Learning Project. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 33

Working with Real Data 33
Look at the Big Picture 35
Frame the Problem 35
Select a Performance Measure 37

iii
 Check the Assumptions 40
Get the Data 40
Create the Workspace 40
Download the Data 43
Take a Quick Look at the Data Structure 45
Create a Test Set 49
Discover and Visualize the Data to Gain Insights 53
Visualizing Geographical Data 53
Looking for Correlations 55
Experimenting with Attribute Combinations 58
Prepare the Data for Machine Learning Algorithms 59
Data Cleaning 60
Handling Text and Categorical Attributes 62
Custom Transformers 64
Feature Scaling 65
Transformation Pipelines 66
Select and Train a Model 68
Training and Evaluating on the Training Set 68
Better Evaluation Using Cross-Validation 69
Fine-Tune Your Model 71
Grid Search 72
Randomized Search 74
Ensemble Methods 74
Analyze the Best Models and Their Errors 74
Evaluate Your System on the Test Set 75
Launch, Monitor, and Maintain Your System 76
Try It Out! 77
Exercises 77

3. Classification. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 79
MNIST 79
Training a Binary Classifier 82
Performance Measures 82
Measuring Accuracy Using Cross-Validation 83
Confusion Matrix 84
Precision and Recall 86
Precision/Recall Tradeoff 87
The ROC Curve 91
Multiclass Classification 93
Error Analysis 96
Multilabel Classification 100
Multioutput Classification 101

iv | Table of Contents
 Exercises 102

4. Training Models. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 105

Linear Regression 106
The Normal Equation 108
Computational Complexity 110
Gradient Descent 111
Batch Gradient Descent 114
Stochastic Gradient Descent 117
Mini-batch Gradient Descent 119
Polynomial Regression 121
Learning Curves 123
Regularized Linear Models 127
Ridge Regression 127
Lasso Regression 130
Elastic Net 132
Early Stopping 133
Logistic Regression 134
Estimating Probabilities 134
Training and Cost Function 135
Decision Boundaries 136
Softmax Regression 139
Exercises 142

5. Support Vector Machines. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 145

Linear SVM Classification 145
Soft Margin Classification 146
Nonlinear SVM Classification 149
Polynomial Kernel 150
Adding Similarity Features 151
Gaussian RBF Kernel 152
Computational Complexity 153
SVM Regression 154
Under the Hood 156
Decision Function and Predictions 156
Training Objective 157
Quadratic Programming 159
The Dual Problem 160
Kernelized SVM 161
Online SVMs 164
Exercises 165

Table of Contents | v
 6. Decision Trees. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 167
Training and Visualizing a Decision Tree 167
Making Predictions 169
Estimating Class Probabilities 171
The CART Training Algorithm 171
Computational Complexity 172
Gini Impurity or Entropy? 172
Regularization Hyperparameters 173
Regression 175
Instability 177
Exercises 178

7. Ensemble Learning and Random Forests. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 181

Voting Classifiers 181
Bagging and Pasting 185
Bagging and Pasting in Scikit-Learn 186
Out-of-Bag Evaluation 187
Random Patches and Random Subspaces 188
Random Forests 189
Extra-Trees 190
Feature Importance 190
Boosting 191
AdaBoost 192
Gradient Boosting 195
Stacking 200
Exercises 202

8. Dimensionality Reduction. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 205

The Curse of Dimensionality 206
Main Approaches for Dimensionality Reduction 207
Projection 207
Manifold Learning 210
PCA 211
Preserving the Variance 211
Principal Components 212
Projecting Down to d Dimensions 213
Using Scikit-Learn 214
Explained Variance Ratio 214
Choosing the Right Number of Dimensions 215
PCA for Compression 216
Incremental PCA 217
Randomized PCA 218

vi | Table of Contents
 Kernel PCA 218
Selecting a Kernel and Tuning Hyperparameters 219
LLE 221
Other Dimensionality Reduction Techniques 223
Exercises 224

Part II. Neural Networks and Deep Learning

9. Up and Running with TensorFlow. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 229
Installation 232
Creating Your First Graph and Running It in a Session 232
Managing Graphs 234
Lifecycle of a Node Value 235
Linear Regression with TensorFlow 235
Implementing Gradient Descent 237
Manually Computing the Gradients 237
Using autodiff 238
Using an Optimizer 239
Feeding Data to the Training Algorithm 239
Saving and Restoring Models 241
Visualizing the Graph and Training Curves Using TensorBoard 242
Name Scopes 245
Modularity 246
Sharing Variables 248
Exercises 251

10. Introduction to Artificial Neural Networks. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 253

From Biological to Artificial Neurons 254
Biological Neurons 255
Logical Computations with Neurons 256
The Perceptron 257
Multi-Layer Perceptron and Backpropagation 261
Training an MLP with TensorFlow’s High-Level API 264
Training a DNN Using Plain TensorFlow 265
Construction Phase 265
Execution Phase 269
Using the Neural Network 270
Fine-Tuning Neural Network Hyperparameters 270
Number of Hidden Layers 270
Number of Neurons per Hidden Layer 272
Activation Functions 272

Table of Contents | vii

 Exercises 273

11. Training Deep Neural Nets. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 275

Vanishing/Exploding Gradients Problems 275
Xavier and He Initialization 277
Nonsaturating Activation Functions 279
Batch Normalization 282
Gradient Clipping 286
Reusing Pretrained Layers 286
Reusing a TensorFlow Model 287
Reusing Models from Other Frameworks 288
Freezing the Lower Layers 289
Caching the Frozen Layers 290
Tweaking, Dropping, or Replacing the Upper Layers 290
Model Zoos 291
Unsupervised Pretraining 291
Pretraining on an Auxiliary Task 292
Faster Optimizers 293
Momentum optimization 294
Nesterov Accelerated Gradient 295
AdaGrad 296
RMSProp 298
Adam Optimization 298
Learning Rate Scheduling 300
Avoiding Overfitting Through Regularization 302
Early Stopping 303
ℓ1 and ℓ2 Regularization 303
Dropout 304
Max-Norm Regularization 307
Data Augmentation 309
Practical Guidelines 310
Exercises 311

12. Distributing TensorFlow Across Devices and Servers. . . . . . . . . . . . . . . . . . . . . . . . . . . 313

Multiple Devices on a Single Machine 314
Installation 314
Managing the GPU RAM 317
Placing Operations on Devices 318
Parallel Execution 321
Control Dependencies 323
Multiple Devices Across Multiple Servers 323
Opening a Session 325

viii | Table of Contents

Visit https://textbookfull.com
now to explore a rich
collection of eBooks, textbook
and enjoy exciting offers!
 The Master and Worker Services 325
Pinning Operations Across Tasks 326
Sharding Variables Across Multiple Parameter Servers 327
Sharing State Across Sessions Using Resource Containers 328
Asynchronous Communication Using TensorFlow Queues 329
Loading Data Directly from the Graph 335
Parallelizing Neural Networks on a TensorFlow Cluster 342
One Neural Network per Device 342
In-Graph Versus Between-Graph Replication 343
Model Parallelism 345
Data Parallelism 347
Exercises 352

13. Convolutional Neural Networks. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 353

The Architecture of the Visual Cortex 354
Convolutional Layer 355
Filters 357
Stacking Multiple Feature Maps 358
TensorFlow Implementation 360
Memory Requirements 362
Pooling Layer 363
CNN Architectures 365
LeNet-5 366
AlexNet 367
GoogLeNet 368
ResNet 372
Exercises 376

14. Recurrent Neural Networks. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 379

Recurrent Neurons 380
Memory Cells 382
Input and Output Sequences 382
Basic RNNs in TensorFlow 384
Static Unrolling Through Time 385
Dynamic Unrolling Through Time 387
Handling Variable Length Input Sequences 387
Handling Variable-Length Output Sequences 388
Training RNNs 389
Training a Sequence Classifier 389
Training to Predict Time Series 392
Creative RNN 396
Deep RNNs 396

Table of Contents | ix
 Distributing a Deep RNN Across Multiple GPUs 397
Applying Dropout 399
The Difficulty of Training over Many Time Steps 400
LSTM Cell 401
Peephole Connections 403
GRU Cell 404
Natural Language Processing 405
Word Embeddings 405
An Encoder–Decoder Network for Machine Translation 407
Exercises 410

15. Autoencoders. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 411

Efficient Data Representations 412
Performing PCA with an Undercomplete Linear Autoencoder 413
Stacked Autoencoders 415
TensorFlow Implementation 416
Tying Weights 417
Training One Autoencoder at a Time 418
Visualizing the Reconstructions 420
Visualizing Features 421
Unsupervised Pretraining Using Stacked Autoencoders 422
Denoising Autoencoders 424
TensorFlow Implementation 425
Sparse Autoencoders 426
TensorFlow Implementation 427
Variational Autoencoders 428
Generating Digits 431
Other Autoencoders 432
Exercises 433

16. Reinforcement Learning. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 437

Learning to Optimize Rewards 438
Policy Search 440
Introduction to OpenAI Gym 441
Neural Network Policies 444
Evaluating Actions: The Credit Assignment Problem 447
Policy Gradients 448
Markov Decision Processes 453
Temporal Difference Learning and Q-Learning 457
Exploration Policies 459
Approximate Q-Learning 460
Learning to Play Ms. Pac-Man Using Deep Q-Learning 460

x | Table of Contents
 Exercises 469
Thank You! 470

A. Exercise Solutions. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 471

B. Machine Learning Project Checklist. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 497

C. SVM Dual Problem. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 503

D. Autodiff. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 507

E. Other Popular ANN Architectures. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 515

Index. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 525

Table of Contents | xi
 Preface

The Machine Learning Tsunami

In 2006, Geoffrey Hinton et al. published a paper1 showing how to train a deep neural
network capable of recognizing handwritten digits with state-of-the-art precision
(>98%). They branded this technique “Deep Learning.” Training a deep neural net
was widely considered impossible at the time,2 and most researchers had abandoned
the idea since the 1990s. This paper revived the interest of the scientific community
and before long many new papers demonstrated that Deep Learning was not only
possible, but capable of mind-blowing achievements that no other Machine Learning
(ML) technique could hope to match (with the help of tremendous computing power
and great amounts of data). This enthusiasm soon extended to many other areas of
Machine Learning.
Fast-forward 10 years and Machine Learning has conquered the industry: it is now at
the heart of much of the magic in today’s high-tech products, ranking your web
search results, powering your smartphone’s speech recognition, and recommending
videos, beating the world champion at the game of Go. Before you know it, it will be
driving your car.

Machine Learning in Your Projects

So naturally you are excited about Machine Learning and you would love to join the
party!
Perhaps you would like to give your homemade robot a brain of its own? Make it rec‐
ognize faces? Or learn to walk around?

1 Available on Hinton’s home page at http://www.cs.toronto.edu/~hinton/.

2 Despite the fact that Yann Lecun’s deep convolutional neural networks had worked well for image recognition
since the 1990s, although they were not as general purpose.

xiii
Or maybe your company has tons of data (user logs, financial data, production data,
machine sensor data, hotline stats, HR reports, etc.), and more than likely you could
unearth some hidden gems if you just knew where to look; for example:

• Segment customers and find the best marketing strategy for each group
• Recommend products for each client based on what similar clients bought
• Detect which transactions are likely to be fraudulent
• Predict next year’s revenue
• And more

Whatever the reason, you have decided to learn Machine Learning and implement it
in your projects. Great idea!

Objective and Approach

This book assumes that you know close to nothing about Machine Learning. Its goal
is to give you the concepts, the intuitions, and the tools you need to actually imple‐
ment programs capable of learning from data.
We will cover a large number of techniques, from the simplest and most commonly
used (such as linear regression) to some of the Deep Learning techniques that regu‐
larly win competitions.
Rather than implementing our own toy versions of each algorithm, we will be using
actual production-ready Python frameworks:

• Scikit-Learn is very easy to use, yet it implements many Machine Learning algo‐
rithms efficiently, so it makes for a great entry point to learn Machine Learning.
• TensorFlow is a more complex library for distributed numerical computation
using data flow graphs. It makes it possible to train and run very large neural net‐
works efficiently by distributing the computations across potentially thousands
of multi-GPU servers. TensorFlow was created at Google and supports many of
their large-scale Machine Learning applications. It was open-sourced in Novem‐
ber 2015.

The book favors a hands-on approach, growing an intuitive understanding of

Machine Learning through concrete working examples and just a little bit of theory.
While you can read this book without picking up your laptop, we highly recommend
you experiment with the code examples available online as Jupyter notebooks at
https://github.com/ageron/handson-ml.

xiv | Preface
Prerequisites
This book assumes that you have some Python programming experience and that you
are familiar with Python’s main scientific libraries, in particular NumPy, Pandas, and
Matplotlib.
Also, if you care about what’s under the hood you should have a reasonable under‐
standing of college-level math as well (calculus, linear algebra, probabilities, and sta‐
tistics).
If you don’t know Python yet, http://learnpython.org/ is a great place to start. The offi‐
cial tutorial on python.org is also quite good.
If you have never used Jupyter, Chapter 2 will guide you through installation and the
basics: it is a great tool to have in your toolbox.
If you are not familiar with Python’s scientific libraries, the provided Jupyter note‐
books include a few tutorials. There is also a quick math tutorial for linear algebra.

Roadmap
This book is organized in two parts. Part I, The Fundamentals of Machine Learning,
covers the following topics:

• What is Machine Learning? What problems does it try to solve? What are the
main categories and fundamental concepts of Machine Learning systems?
• The main steps in a typical Machine Learning project.
• Learning by fitting a model to data.
• Optimizing a cost function.
• Handling, cleaning, and preparing data.
• Selecting and engineering features.
• Selecting a model and tuning hyperparameters using cross-validation.
• The main challenges of Machine Learning, in particular underfitting and overfit‐
ting (the bias/variance tradeoff).
• Reducing the dimensionality of the training data to fight the curse of dimension‐
ality.
• The most common learning algorithms: Linear and Polynomial Regression,
Logistic Regression, k-Nearest Neighbors, Support Vector Machines, Decision
Trees, Random Forests, and Ensemble methods.

Preface | xv
Part II, Neural Networks and Deep Learning, covers the following topics:

• What are neural nets? What are they good for?

• Building and training neural nets using TensorFlow.
• The most important neural net architectures: feedforward neural nets, convolu‐
tional nets, recurrent nets, long short-term memory (LSTM) nets, and autoen‐
coders.
• Techniques for training deep neural nets.
• Scaling neural networks for huge datasets.
• Reinforcement learning.

The first part is based mostly on Scikit-Learn while the second part uses TensorFlow.

Don’t jump into deep waters too hastily: while Deep Learning is no
doubt one of the most exciting areas in Machine Learning, you
should master the fundamentals first. Moreover, most problems
can be solved quite well using simpler techniques such as Random
Forests and Ensemble methods (discussed in Part I). Deep Learn‐
ing is best suited for complex problems such as image recognition,
speech recognition, or natural language processing, provided you
have enough data, computing power, and patience.

Other Resources
Many resources are available to learn about Machine Learning. Andrew Ng’s ML
course on Coursera and Geoffrey Hinton’s course on neural networks and Deep
Learning are amazing, although they both require a significant time investment
(think months).
There are also many interesting websites about Machine Learning, including of
course Scikit-Learn’s exceptional User Guide. You may also enjoy Dataquest, which
provides very nice interactive tutorials, and ML blogs such as those listed on Quora.
Finally, the Deep Learning website has a good list of resources to learn more.
Of course there are also many other introductory books about Machine Learning, in
particular:

• Joel Grus, Data Science from Scratch (O’Reilly). This book presents the funda‐
mentals of Machine Learning, and implements some of the main algorithms in
pure Python (from scratch, as the name suggests).
• Stephen Marsland, Machine Learning: An Algorithmic Perspective (Chapman and
Hall). This book is a great introduction to Machine Learning, covering a wide

xvi | Preface
 range of topics in depth, with code examples in Python (also from scratch, but
using NumPy).
• Sebastian Raschka, Python Machine Learning (Packt Publishing). Also a great
introduction to Machine Learning, this book leverages Python open source libra‐
ries (Pylearn 2 and Theano).
• Yaser S. Abu-Mostafa, Malik Magdon-Ismail, and Hsuan-Tien Lin, Learning from
Data (AMLBook). A rather theoretical approach to ML, this book provides deep
insights, in particular on the bias/variance tradeoff (see Chapter 4).
• Stuart Russell and Peter Norvig, Artificial Intelligence: A Modern Approach, 3rd
Edition (Pearson). This is a great (and huge) book covering an incredible amount
of topics, including Machine Learning. It helps put ML into perspective.

Finally, a great way to learn is to join ML competition websites such as Kaggle.com

this will allow you to practice your skills on real-world problems, with help and
insights from some of the best ML professionals out there.

Conventions Used in This Book

The following typographical conventions are used in this book:
Italic
Indicates new terms, URLs, email addresses, filenames, and file extensions.
Constant width
Used for program listings, as well as within paragraphs to refer to program ele‐
ments such as variable or function names, databases, data types, environment
variables, statements and keywords.
Constant width bold
Shows commands or other text that should be typed literally by the user.
Constant width italic
Shows text that should be replaced with user-supplied values or by values deter‐
mined by context.

This element signifies a tip or suggestion.

Preface | xvii
 This element signifies a general note.

This element indicates a warning or caution.

Using Code Examples

Supplemental material (code examples, exercises, etc.) is available for download at
https://github.com/ageron/handson-ml.
This book is here to help you get your job done. In general, if example code is offered
with this book, you may use it in your programs and documentation. You do not
need to contact us for permission unless you’re reproducing a significant portion of
the code. For example, writing a program that uses several chunks of code from this
book does not require permission. Selling or distributing a CD-ROM of examples
from O’Reilly books does require permission. Answering a question by citing this
book and quoting example code does not require permission. Incorporating a signifi‐
cant amount of example code from this book into your product’s documentation does
require permission.
We appreciate, but do not require, attribution. An attribution usually includes the
title, author, publisher, and ISBN. For example: “Hands-On Machine Learning with
Scikit-Learn and TensorFlow by Aurélien Géron (O’Reilly). Copyright 2017 Aurélien
Géron, 978-1-491-96229-9.”
If you feel your use of code examples falls outside fair use or the permission given
above, feel free to contact us at [email protected].

O’Reilly Safari
Safari (formerly Safari Books Online) is a membership-based
training and reference platform for enterprise, government,
educators, and individuals.

Members have access to thousands of books, training videos, Learning Paths, interac‐
tive tutorials, and curated playlists from over 250 publishers, including O’Reilly
Media, Harvard Business Review, Prentice Hall Professional, Addison-Wesley Profes‐
sional, Microsoft Press, Sams, Que, Peachpit Press, Adobe, Focal Press, Cisco Press,

xviii | Preface
Visit https://textbookfull.com
now to explore a rich
collection of eBooks, textbook
and enjoy exciting offers!
John Wiley & Sons, Syngress, Morgan Kaufmann, IBM Redbooks, Packt, Adobe
Press, FT Press, Apress, Manning, New Riders, McGraw-Hill, Jones & Bartlett, and
Course Technology, among others.
For more information, please visit http://oreilly.com/safari.

How to Contact Us
Please address comments and questions concerning this book to the publisher:

O’Reilly Media, Inc.

1005 Gravenstein Highway North
Sebastopol, CA 95472
800-998-9938 (in the United States or Canada)
707-829-0515 (international or local)
707-829-0104 (fax)

We have a web page for this book, where we list errata, examples, and any additional
information. You can access this page at http://bit.ly/hands-on-machine-learning-
with-scikit-learn-and-tensorflow.
To comment or ask technical questions about this book, send email to bookques‐
[email protected].
For more information about our books, courses, conferences, and news, see our web‐
site at http://www.oreilly.com.
Find us on Facebook: http://facebook.com/oreilly
Follow us on Twitter: http://twitter.com/oreillymedia
Watch us on YouTube: http://www.youtube.com/oreillymedia

Acknowledgments
I would like to thank my Google colleagues, in particular the YouTube video classifi‐
cation team, for teaching me so much about Machine Learning. I could never have
started this project without them. Special thanks to my personal ML gurus: Clément
Courbet, Julien Dubois, Mathias Kende, Daniel Kitachewsky, James Pack, Alexander
Pak, Anosh Raj, Vitor Sessak, Wiktor Tomczak, Ingrid von Glehn, Rich Washington,
and everyone at YouTube Paris.
I am incredibly grateful to all the amazing people who took time out of their busy
lives to review my book in so much detail. Thanks to Pete Warden for answering all
my TensorFlow questions, reviewing Part II, providing many interesting insights, and
of course for being part of the core TensorFlow team. You should definitely check out

Preface | xix
his blog! Many thanks to Lukas Biewald for his very thorough review of Part II: he left
no stone unturned, tested all the code (and caught a few errors), made many great
suggestions, and his enthusiasm was contagious. You should check out his blog and
his cool robots! Thanks to Justin Francis, who also reviewed Part II very thoroughly,
catching errors and providing great insights, in particular in Chapter 16. Check out
his posts on TensorFlow!
Huge thanks as well to David Andrzejewski, who reviewed Part I and provided
incredibly useful feedback, identifying unclear sections and suggesting how to
improve them. Check out his website! Thanks to Grégoire Mesnil, who reviewed
Part II and contributed very interesting practical advice on training neural networks.
Thanks as well to Eddy Hung, Salim Sémaoune, Karim Matrah, Ingrid von Glehn,
Iain Smears, and Vincent Guilbeau for reviewing Part I and making many useful sug‐
gestions. And I also wish to thank my father-in-law, Michel Tessier, former mathe‐
matics teacher and now a great translator of Anton Chekhov, for helping me iron out
some of the mathematics and notations in this book and reviewing the linear algebra
Jupyter notebook.
And of course, a gigantic “thank you” to my dear brother Sylvain, who reviewed every
single chapter, tested every line of code, provided feedback on virtually every section,
and encouraged me from the first line to the last. Love you, bro!
Many thanks as well to O’Reilly’s fantastic staff, in particular Nicole Tache, who gave
me insightful feedback, always cheerful, encouraging, and helpful. Thanks as well to
Marie Beaugureau, Ben Lorica, Mike Loukides, and Laurel Ruma for believing in this
project and helping me define its scope. Thanks to Matt Hacker and all of the Atlas
team for answering all my technical questions regarding formatting, asciidoc, and
LaTeX, and thanks to Rachel Monaghan, Nick Adams, and all of the production team
for their final review and their hundreds of corrections.
Last but not least, I am infinitely grateful to my beloved wife, Emmanuelle, and to our
three wonderful kids, Alexandre, Rémi, and Gabrielle, for encouraging me to work
hard on this book, asking many questions (who said you can’t teach neural networks
to a seven-year-old?), and even bringing me cookies and coffee. What more can one
dream of?

xx | Preface
 PART I
The Fundamentals of
Machine Learning
 CHAPTER 1
The Machine Learning Landscape

When most people hear “Machine Learning,” they picture a robot: a dependable but‐
ler or a deadly Terminator depending on who you ask. But Machine Learning is not
just a futuristic fantasy, it’s already here. In fact, it has been around for decades in
some specialized applications, such as Optical Character Recognition (OCR). But the
first ML application that really became mainstream, improving the lives of hundreds
of millions of people, took over the world back in the 1990s: it was the spam filter.
Not exactly a self-aware Skynet, but it does technically qualify as Machine Learning
(it has actually learned so well that you seldom need to flag an email as spam any‐
more). It was followed by hundreds of ML applications that now quietly power hun‐
dreds of products and features that you use regularly, from better recommendations
to voice search.
Where does Machine Learning start and where does it end? What exactly does it
mean for a machine to learn something? If I download a copy of Wikipedia, has my
computer really “learned” something? Is it suddenly smarter? In this chapter we will
start by clarifying what Machine Learning is and why you may want to use it.
Then, before we set out to explore the Machine Learning continent, we will take a
look at the map and learn about the main regions and the most notable landmarks:
supervised versus unsupervised learning, online versus batch learning, instance-
based versus model-based learning. Then we will look at the workflow of a typical ML
project, discuss the main challenges you may face, and cover how to evaluate and
fine-tune a Machine Learning system.
This chapter introduces a lot of fundamental concepts (and jargon) that every data
scientist should know by heart. It will be a high-level overview (the only chapter
without much code), all rather simple, but you should make sure everything is
crystal-clear to you before continuing to the rest of the book. So grab a coffee and let’s
get started!

3
Random documents with unrelated
content Scribd suggests to you:
way. That is all, Johnny, darling, ‘the conclusion of the whole
matter,’—just to rest on His love.”
“Mamma,” said Johnny, holding his mother fast in a long, close
hug, “I don’t think I ever loved Him so much as I do to-night; and I
don’t think I’ll ever be really worried, or not long, anyhow, when
things seem to go crosswise again.”
 CHAPTER XXII.
THE WAY OF ESCAPE.

t must have been most beautiful,” said Tiny,

“I wonder if it looked at all like that?” and
she pointed to a large, bright star, which
seemed quite alone in the sky, for the sun
had only just set, and no other star could yet
be seen near this one.
“I think it was much larger, Tiny,” said
Johnny, who was standing close beside her.
“You know if it hadn’t been quite different from the other stars, no
one would have thought it was anything in particular, and the wise
men said, quite positively, ‘We have seen His star in the east, and
are come to worship Him.’ So you see, it must have been different.”
“Yes,” said Tiny, “I didn’t think of that. And how glad they must
have been to see it, for they seemed perfectly certain about what it
meant. They didn’t ask if He really had come, or if the people at
Jerusalem thought He had, but just ‘Where is He?’ And then they
found out right away; I don’t believe they would, if they hadn’t been
so certain.”
“And just think,” said Johnny, “how splendid it must have been for
them to be the first ones to tell the people about it, when they got
back to their ‘own country.’ That was even better than it is to be a
missionary now. I wonder if any of the people they told it to laughed
at them, and didn’t believe them.”
“I don’t see how they could,” said Tiny. “Why, you know everybody
was looking for the Saviour, then; and so when the wise men told
them how He had been born just where the prophets had said He
would be, and that they had really seen Him, how could anybody not
believe them?”
Tiny and Johnny were standing by the library window, waiting for
their mother and Jim, for it was Sunday evening, and time for the
“talk.” The lesson was about the leading of the star, and it seemed to
the children unusually beautiful, although there was never any lack
of interest in these talks. They were growing impatient, when Jim
came in sight, walking fast, as if he were afraid of being late, but
they hastily agreed not to question him; for Johnny had found that
this always annoyed him as nothing else did. He had a keen eye for
“chances” to help his less fortunate neighbors, and more than once,
Johnny had accidentally caught him giving time, and thought, and
even money, although, industrious as he was, he seldom made more
in a day than sufficed his actual needs. But he seemed so thoroughly
disconcerted when anything of this kind was discovered, that Johnny
tried hard to resist the temptation to tease him which was offered by
his sensitiveness on this point.
Mrs. Leslie came down a few minutes after Jim arrived, and a
beautiful talk followed. She had brought an old book about the Holy
Land, which she had recently found at a second-hand book store,
and it described in such good, clear language the state of affairs
throughout the world, and the manners and customs of the people
at the time of the birth of our Saviour, that the children, deeply
interested, felt as if they had never before so clearly realized it all.
And Johnny spoke once more of the happiness of the wise men, in
being the bearers of this great news back to their own country.
“I think it must have been much more interesting to be alive then,
than it is now,” he said, with a little discontent in his voice, “for don’t
you believe, mamma, that it seemed a great deal more wonderful
about the Saviour then, when it was all happening, than it seems
now, after so many, many years?”
“Perhaps it did,” said Mrs. Leslie, “but you know how it was when
the apostles began to tell the good news. Besides being disbelieved,
and persecuted, and imprisoned, and banished, they had to endure
something which, to some people, would be hardest of all—we are
told that they were ‘mocked’; that is what you would call at school,
being made fun of.”
“I never thought of that before,” said Johnny, “I do believe that
must have been the hardest of all! You see, a person can screw
himself up to something pretty bad, like having a tooth out, or being
killed, or anything; but to see a whole lot of people making faces
and laughing at you—do you believe you could ever stand that,
mamma?”
“It would be very hard, and yet it is part of their daily work for
some of our missionaries, at this very day,” said Mrs. Leslie, “I have
heard a missionary who had been preaching and teaching in India
say that nothing delighted some of the natives more than to bait and
worry a teacher until it was next to impossible for him to keep his
temper. And no doubt the wise men had that very thing to contend
with, when they went back to their own country. I think every one
has, at some time or other. And then is, above all other times, the
time to ‘let our light so shine before men that they may glorify our
Father which is in Heaven.’ When people see that the power of God
is a power, it nearly always makes some impression on them. So
here is a chance for every one to ‘make manifest,’ and how beautiful
the blessing is! ‘That which doth make manifest is light.’ We are
allowed to carry to others the Light of the World.”
This was the end of the talk, for that time, and it made more
impression upon Jim and Johnny than it did upon Tiny, for Jim, as
we have said, carried his sensitiveness too far, often—as in the case
of little Taffy—allowing it to hinder him from asking for help for
others, when he had come to the end of his own ability, but not the
needs of the case, and when such help would have been most gladly
and efficiently given; as for Johnny, he was foolishly alive to ridicule,
and many of the slips of temper which he afterwards lamented were
due solely to this cause. A jeering laugh or a mocking speech always
had power to make his face flush and his hands clinch, and the
effect did not always stop there—he often said things for which he
was bitterly sorry as soon as the rush of angry feeling was past. And
somehow it seemed to him that the attacks upon his temper always
took place when he was unusually off his guard, and open to them.

POOR KATY.
 The effect of this talk upon Jim was very marked. He began, from
that time, shyly to take Mrs. Leslie into his confidence, whenever he
felt that she could help him, and he schooled himself to bear,
without wincing, any and all allusions to the various and unobtrusive
acts of kindness which he was able to perform. And he very soon
had the encouragement of finding his usefulness greatly increased,
while he still had the satisfaction of doing many things which were
known only to himself and those whom he helped. To his firm and
resolute character, the plan of the campaign was more than half the
battle, while Johnny, who was naturally more heedless and forgetful,
found great difficulty in keeping his good resolutions where he could
find them in a hurry.
He had, for the time being, quite forgotten this talk about the wise
men, when, one day during the following week, as he was playing
with the boys at recess, a little girl strayed into the playground, with
a basket of apples and cakes, hoping to sell some of her wares to
the schoolboys. Johnny remembered her at once, for she was one of
the many people whom Mrs. Leslie had helped and befriended; she
had found the poor child in great trouble and destitution, a few
months before, and had put her to board with an old woman who
only demanded a very moderate amount of work in payment for the
care which she gave the little girl.
Katy employed her spare time in trying to sell
whatever she could pick up most cheaply,
whenever she had a few cents at her command;
matches, sometimes, and what Tiny called
“dreadful” cakes of soap; very thick china
buttons, blunt pins, or, when she had not enough
even for these investments, a few apples or
oranges, and unpleasant-looking cakes.
She was a solemn and anxious-looking child,
and although, through Mrs. Leslie’s care and
teaching, her clothes were nearly always whole and clean, they had
a look of not belonging to her, and Tiny and Johnny, while they
pitied her very much, and were always willing to help her in any way
they could, did not admire her.
It had never before occurred to her to visit the playground with
her basket, a fact over which Johnny had secretly rejoiced, and it
was with a feeling of dismay quite beyond the occasion that he saw
her come in at the gate. She did not see him, just at first, and he
was attacked, as he afterward told Tiny, with a mean desire to “cut
and run.” Before he could make up his mind to do this, however, she
recognized him, and a smile broke over her solemn countenance.
“Why!” she said, in the drawl which always “aggravated” Johnny,
“I didn’t know you went to school here, Johnny Leslie! I’m right glad
I came in. Don’t you want to buy an apple? And don’t some of these
other boys want to? They’re real nice—I tried one.”
“I haven’t any money here, Katy,” said Johnny, briefly, “and I don’t
believe the other boys have, either. And I wouldn’t come here,
again, if I were you; it’s not a good place to sell things at all—at
least, some things,” he added hastily, as he remembered how a
basketful of pop-corn candy had vanished in that very yard, a few
days before.
Katy’s face grew solemn again, and she was turning to go, with
the meekness which, to Johnny, was another of her offences. But a
few of the boys who were standing near, and who had heard the
conversation, saw how anxious Johnny was to get rid of her, and one
of them called out mockingly, loud enough to be heard all over the
playground,—
“Boys! Here’s a young lady friend of Johnny Leslie’s, with some
wittles to sell! His friends in this crowd ought to patronize her!”
The mischief was done, now; the boys flocked around Katy, and
being, most of them, good-natured fellows, as boys go, they said
nothing unmannerly to her, but they contrived, in their politely
worded remarks, which she did not in the least understand, to sting
Johnny to the verge of desperation. And yet, when he thought it
over afterwards, nothing had been said which was really worth
minding; it was the manner, not the matter, and the mocking
laughter, which had roused him.
“I think your friends are real nice, Johnny Leslie,” said Katy, as she
turned, with her empty basket, and her hand full of small coins, to
leave the yard, “and I won’t come back, if you don’t like me to, but I
don’t see why you don’t!” and she walked dejectedly away.
But before she reached the gate, Johnny had fought his battle—
and won it. He sprang after her, and held open the gate, as he
would have done for his mother, saying, loud enough for every one
to hear him,—
“I’m glad you’ve had such good luck, Katy! Come back every day,
if you like, and you wait for me here after school, and I’ll show you a
first-rate place to buy things, where the man won’t cheat you!”
She thanked him all too profusely, as she went slowly through the
gate, and then he turned, feeling that his face was fiery red, to
receive the volley which he fully expected, and had braced himself to
bear. But it was not exactly the sort of volley for which he was
prepared.
“Hurrah for Johnny Leslie!” called one of the little boys; the others
caught it up with a deafening cheer, and an unusual amount of
“tiger,” and Johnny saw that they were quite in earnest.
And then came back to his mind once more the words which had
so often come there, since he had read the quaint and beautiful
story of “The Pilgrim’s Progress from this world to a better,”—“The
lions were chained.”
The fact was, several of the boys had heard about Katy through
Tiny and their sisters, but they could not, or rather would not, resist
the temptation to tease Johnny, when they saw the foolish
annoyance which her coming had caused him. It has often been
noticed how a word, or even a look, will turn the tide, in affairs like
this, and even in much larger ones, and Johnny’s bold championship
of Katy had done this at once.
 It was a good day for her when she invaded the playground, for
Johnny kept his word about showing her where to buy, and, knowing
as he did the things which would be most likely to sell well, the
result was that, after a few lessons, poor little Katy, who was slow
rather than stupid, began to show real judgment in her purchases.
She was always modest and quiet in her manner to the boys, and
the result of this was that their chaffing never passed the bounds of
harmless fun. They called her “The Daughter of the Regiment,” and
threatened her with dire penalties, should she not always come “first
and foremost” to their playground with her new stock.
“I’ve often thought, Tiny,” said Johnny, long afterward, when Katy
had made and saved enough to buy a second-hand counter, have
shelves put in the front room of the two which she and the old
woman occupied, and start a small but promising business. “I’ve
often thought of how it would have been if I had cut and run. And it
seems to me that the ‘way of escape’—about temptations, you know
—is right straight ahead!”
 CHAPTER XXIII.
THE CIRCULAR CITY.

r. Leslie made a discovery.

He had remarked, early in the spring, that
when he was really rich, when he had five or
six millions of dollars, he was going to build
a city in the form of a very large circle, only
two streets deep, and inside of this circle
was to be an immense farm.
“I shall begin,” he said, “by finding and
buying a ready-made farm, for the farmhouse and barns and orchard
and garden must all be old. I shall put all this in perfect order,
without making it look new. Then I shall build twenty-five Swiss
cottages, each with three rooms and a great deal of veranda. I shall
buy twenty-five excellent tents, and hide them about in the orchard
and shrubberies, and I shall invite my friends, fifty families at a time,
to come and stay a month with me on my farm; and if my friends
should all be used up before the summer is over, I will ask some of
them to nominate some of their friends. And in the meantime,” he
added, dropping his millionaire tone of voice suddenly, “if we can
find the farm and the farmhouse, we will make a beginning by going
there for the summer, and planning the rest out.”
The others laughed at this dreadful coming down, but after that it
became a favorite amusement to make additions to the “circular
city,” and I could not begin to tell you all the plans which were made
for the comfort and happiness and goodness of the “circular
citizens,” as one thought of one thing, and one of another. And the
best of this popular “pretend” was, that it set everybody thinking,
and it was surprising to find how many of the plans for the dream-
city might, in much smaller ways, of course, be carried out without
waiting for all the rest.
For instance, when Tiny said that all the little girls
should have dolls, her mother reminded her that she
knew how to make very nicely those rag dolls which
one makes by rolling up white muslin—a thick roll for
the body, and a thin one for the arms; coarse thread
sewed round where the neck ought to be, the top of
the head “gathered” and covered with a little cap, eyes
and nose and mouth inked, or worked in colored
thread, upon the face, and the fact that the infant has
only one leg concealed by a nice long petticoat and
frock.
Mrs. Leslie promised to supply as many “rags” as Tiny would use,
in the making and dressing of these dolls, and it became the little
girl’s delight to carry one of them in her pocket, when she was going
for a walk, and to give it to the poorest, most unhappy-looking child
she could find. There are very few small girls who do not love to
mother dolls, and Tiny’s heart would feel warm all day, remembering
the joyful change in some little pinched face, and the astonished,—
“For me? For my own to keep?”
And when Johnny said that all the sick people should have flowers
every day, his mother reminded him that the “can’t-get-aways” were
glad even of such common things as daisies and buttercups and
clover blossoms. And after that he took many a long walk to the
fields outside the town, where these could be found.
They had all hoped to go back to Mr. Allen’s for the summer, but
when Mrs. Leslie wrote to ask Mrs. Allen if they could be received,
Mrs. Allen replied, that since Ann had married and left them, half the
house seemed gone, and she really didn’t think she could take any
boarders this summer.
“Perhaps you did not hear that Ann was married,” she wrote; “but
I miss her so, all the time, that I feel as if everybody must know it.
 She’s married a widower with two little
children,—a nice, quiet, pleasant sort of a
man,—but we all told Ann she only took
him because she fell in love with the
children! And she does seem as happy as a
queen, and, for that matter, so does he; but
it provokes me to think how little we set by
her, considering what she was worth, till
after we’d lost her.”
It was a week or two after this letter was
received, that Mr. Leslie made his discovery. He found the
farmhouse, the “very identical” farmhouse, for which he was
longing, and he found it when he was not looking for it, as he was
riding a horse which a friend had lent him.
The gate of the long lane which led up to the house was only half
a mile from the railway station, and only eight miles from the town
where the Leslies lived, and two dear old Quaker people, who “liked
children,” lived there all alone, save for their few servants.
“No, they had never taken boarders,” Friend Mercy said, “and she
was afraid the children—her married boys and girls—might not quite
like it.”
But Mr. Leslie, at her hospitable invitation, dismounted, and tied
his horse and sat down on the “settee,” under the lilac bushes, and
drank buttermilk and ate gingerbread, and I am afraid he talked a
good deal, and the result of it all was, that, just as he was going
away, Friend Mercy said,—
“Well, thee bring thy wife and little ones to-morrow afternoon,
Friend Leslie, and have a sociable cup of tea with us. I will talk with
Isaac in the meantime, and with thy wife when she comes, and—
we’ll see.”
Mr. Leslie had no desire to break his children’s hearts, so, although
it was hard work not to, he did not tell them all that Friend Mercy
and he had said to each other, for fear she should not “see her way
clear” to take them; so he only told of his pleasant call, and of this
magnificent invitation to a real country tea, in the “inner circle”; and
they were so nearly wild over that, that it was a very good thing he
stopped there!
Friend Mercy had suggested the four o’clock train, which would
give the children time for “a good run” before the six o’clock tea. So,
while Tiny and Johnny played in the hay, and sailed boats on the
brook, the older people talked; and the result was, that the Leslies
were to be permitted to come and board in the “inner circle,” until
the end of September.
A little talk which Friend Mercy had with
her husband that evening, after the guests
were gone, and when he said he was
“afraid it wouldn’t work,” will explain this.
“Thee sees,
Isaac,” she said,
“those two dear
little things have
played here half
the afternoon, and
there was no quarrelling, or tale-bearing, or
cruelty. They did not stone the chickens and
geese, nor tease Bowser and the cat; and
when I asked John to drive the cows to the spring—which, I will
confess, I did with a purpose—he used neither stick nor stone. I
would not have any children brought here who would teach bad
tricks to Joseph’s and Hannah’s children, for the world; but with
these I think we should be quite safe. Did thee notice how they put
down the kittens, and came at once, when their father called them
to go to the train? When they obey so implicitly such parents as
these seem to be, there is nothing to fear.”
“Thee has had thy own way too long for me to begin to cross thee
now, I’m afraid, mother,” said Friend Gray, with an indulgent smile.
“So, if thy heart is really set upon it, let
them come! The trouble of it will fall chiefly
on thee, I fear.”
It did not seem to fall very heavily. The
one strong, willing maid-of-all-work
declared she could “do for a dozen like
them.”
Mrs. Leslie and Tiny made the three extra
beds, and dusted the rooms every morning; and both Tiny and
Johnny found various delightful ways of helping “Aunt Mercy and
Uncle Isaac,” as the dear old host and hostess were called by
everybody, before a week was out.
The days went by on swift, sunny wings, and everybody was
growing agreeably fat and brown. But, when they stopped to think
of it, there was a shadow over the children’s joy.
They were in the “inner circle”—even the five or six millions, they
thought, could do no more for them; but, oh, the hundreds and
hundreds who were hopelessly outside!
It was not very long, you may be sure, before Aunt Mercy heard
all about the “circular city”; and although at first she treated the
whole matter as a joke, she soon caught herself making valuable
suggestions. And then, when Tiny and Johnny began to lament to
her about all the “outsiders,” she began to think in good earnest,
and the day before the next market day she spoke, and this is what
she said,—
“Father is going to take some chickens to town, to-morrow, and
there will be a good deal of spare room in the wagon. That’s half. He
passes right by the house where a good city missionary lives. That’s
the other half. And the whole is, that if two little people I know
would pick up all those early apples that the wind blew down last
night, in the orchard, and make some nice big bunches of daisies
and clover, with a sweet-william or a marigold in the middle of each,
 father would leave them at Mr. Thorpe’s
door, to be given round to the poor people.”
Tiny and Johnny went nearly as wild over
this announcement as they had gone over
the news that they were to spend the
summer in the inner circle—and then they
went to work. By great good fortune, two of
the grand-children came that very day, and
asked nothing better than to help; and
when, the next morning, at the appointed
hour, which was five o’clock, these four conspirators brought out
their treasures, there was a barrel of apples, and another barrel of
bouquets.
Uncle Isaac laughed, and said he had no idea
what a “fix” he was getting himself into, when he let
Mercy make that speech, but he took the fruit and
flowers, all the same. And after that, it was really
surprising to see the number of things which, it was
found, “might as well go to those poor little ones as
to the pigs.”
Wild raspberries, dewberries, blackberries,
whortleberries, were all to be had for the picking;
Johnny was told that it was only fair for him to keep
one egg out of every dozen for which he had hunted, and these
eggs, which he at first refused to take, and afterward, when he
found that Aunt Mercy was “tried” about it, accepted, were very
carefully packed, and plainly labelled, “For the sickest children.” Then
a very brilliant idea occurred to Tiny.
“Do the pigs have to eat all that bonny-clabber, Aunt Mercy?” she
asked, one morning, as David, the “hired man,” picked up two
buckets full of the nice white curds, and started for the pig-pen.
“Why no, deary,” Aunt Mercy replied, “I was saying to father, only
yesterday, that I was afraid we were over-feeding them, but we
 don’t know what else to do with it. Had
thee thought of anything, dear?”
“If you really
don’t need it,” said
Tiny, hesitating a
little, “I’ve watched
thee make cottage
cheese till I’m sure
I could do it; and I
wouldn’t be in the
way—I’d be ever so careful, and clear up
everything when I was done. And I thought
dear little round white cheeses, tied up in
clean cloths, would be such lovely things to send! Don’t thee think
so, Aunt Mercy?”
Tiny was trying very hard to learn the “plain language”; she
thought it was so pretty.
“Yes, indeed!” said Aunt Mercy, “and of course thee shall! That’s
one of the best things thee’s thought of, dear. Father shall buy us
plenty of that thin cotton cloth I use for my cheese and butter rags,
the very next time he goes to town, and thee shall have all the spare
clabber, after this.”
“But you must let Johnny and me pay for the cotton cloth, Aunt
Mercy,” said Tiny, earnestly. “We’ve been saving up for the next thing
we could think of, and we’ve forty-five cents.”
Aunt Mercy had her mouth open to say “No indeed!” but she shut
it suddenly, and when it opened again, the words which came out
were,—
“Very well, deary.”
So Johnny cut squares of cheese cloth, which was three cents a
yard at the wholesale place where Uncle Isaac bought it, and Tiny
scalded and squeezed and molded the white curd into delightful little
round cheeses, and then Johnny tied them up in the cloths.
 “And the cloths will be beautiful for dumplings, afterward!” said
Tiny.
“Yes, if they can get the dumplings, poor things!” answered
Johnny, soberly.
“There’s a way to make a crust, if the poor souls only knew it,”
said Aunt Mercy, “that’s real wholesome and good for boiled crust
and very cheap. It’s just to scald the flour till it’s soft enough to roll
out, and put in a little salt. And another way, that’s most as cheap,
and better, is to work flour into hot mashed potatoes, till it makes a
crust that will roll out.”
The next time there was a barrel of “windfall” apples to go, Tiny
and Johnny came to Aunt Mercy, each with a sheet of foolscap paper
and a sharp lead pencil, and Tiny said, “Aunt Mercy, will thee please
tell us, quite slowly, those two cheap ways to make apple-dumpling
crust?”
So Aunt Mercy gave out the recipes as if they were a school
dictation, and each of her scholars made twelve copies. It took a
long time, and was a tiresome piece of work, but it was a fine thing
when it was done!
The twenty-four copies were put in a large yellow envelope,
addressed to “Mr. Thorpe,” and Johnny added a note, in the best
hand he had left, after all that writing,—

“Dear Mr. Thorpe,—Will you please put one of these

recipe papers with each batch of apples you give
away? They are all right.
“Very respectfully,
“T. & J.”

This was the beginning of a most interesting correspondence.

When Uncle Isaac came home the next evening, he brought an
envelope addressed to “T. and J.,” and inside was a card, with “John
Thorpe” on one side of it, and on the other, in a clear, firm hand,—
“God bless you both, my dear T. and J. You will never know how
many sad lives you have gladdened, this summer. Is there any moss
in your land of plenty? Have any of your wild-flowers roots? And may
I not know your names?”
Now this was, as Tiny said, “Too beautiful for anything!” especially
as the early apples and all the berries were about gone, and the
children were beginning to wonder what they could find to send
next.
 CHAPTER XXIV.
THE CIRCULAR CITY, CONTINUED.

hey wrote to Mr. Thorpe. Of course they did!

They promised the moss and roots, and told
him how glad they were that the people had
been pleased with what they sent, and
would he be so very kind as to write and tell
them whether he had heard of anybody who
had tried the apple dumplings?
“And if any of your people are ill, dear Mr.
Thorpe,” wrote Tiny, in her share of the letter, “and there is anything
particular that you would like for them, will you please tell us, and
perhaps it will be something we can send you.”
The answer to this letter was delightfully prompt. Yes, several of
the women who had shared the apples had “tried” the dumplings,
and been much pleased with them. Were there any more nice cheap
dishes? And would it be too much trouble to print the recipes in
large, clear letters? Some of the poor people who could read print
quite easily could not read writing at all. And there was “something
particular.” It was almost impossible for any of “his people” to buy
pure milk, and he felt sure that many little children were suffering
and dying for want of proper food. If he might have only two or
three quarts a week of really pure, sweet milk, he would give it to
those who most needed it.
“But perhaps,” he wrote, “it is not in your power to supply this
want, and if it is not, you must not be troubled. God never asks for
any service which we cannot, with His help, render to Him, and the
knowledge of this should keep us from fretting when we cannot
carry out all our wishes and plans.”
 Tiny and Johnny each received ten cents
a week for spending money, and it did not
take them long to decide that, if Uncle
Isaac would sell them three quarts of milk a
week, and lend them a milk can, they
would send that milk, if it took every cent
of their allowance. Uncle Isaac entered into
the plan with spirit; if they took three
quarts of milk a week “straight along,” he
said, it would only be four cents a quart, and he would lend them a
can, and deliver it, with pleasure.
“But that would be skimmed milk, wouldn’t it, Uncle Isaac?” asked
Tiny, doubtfully.
“Oh no,” he answered, “not at all! It shall either be from the
milking over night, with all the cream on it, or, if Johnny chooses, I’ll
call him in time to milk the three quarts that very morning—perhaps
that would be best, for then some of it would keep till next day, if
Mr. Thorpe could find a cold place for it.”
The children were jubilant. There would still be eight cents a week
left, and they admitted to each other that it would have been “very
bad” to be reduced to “nothing at all a week!” And Johnny agreed at
once to do the milking. He had been learning to milk “for fun,” and
could do it quite nicely.
“And that’s a real blessing, Tiny,” he said, “for the milk will be so
nice and fresh, as Uncle Isaac says, that Mr. Thorpe can keep some
till next day. I do hope he has a refrigerator.”
You will begin to see, by this time, that the things which these
little people were doing by way of sharing their happiness, were not
by any means all play, and that some of them were very downright
work. Picking berries in the hot sun, or even flowers, when one picks
them by the bushel, is not amusing. It always seemed to Johnny, on
the milking mornings, that he had only just fallen asleep when Uncle
Isaac gave him the gentle shaking which had been agreed upon,
 because a knock or call would wake the rest
of the family needlessly early. Very often
most interesting things, such as building a
dam, or digging a pond, or making a house
of fence rails, had to be put aside for hours,
that the “consignment,” whatever it
happened to be that time, might be ready
for Uncle Isaac over night. But how sweet
and happy was the play which followed
their labors of love, and how small their
sacrifices seemed, when they thought of
the little children, crowded, packed, into narrow, foul-smelling courts
and alleys, and, inside of these again, into stifling rooms!
The long rambles, in which Mrs. Leslie always, and Mr. Leslie
sometimes, joined, in search of mosses and wild-flower roots, were
only a delight, and quite paid for the work of printing the simple
rules for cheap cookery, which Aunt Mercy told them from time to
time, as she could remember.
They caught Uncle Isaac, nearly every time that he took one of
their cargoes, slipping in something on his own account—vegetables,
or fruit, or eggs, and even, sometimes, a piece of fresh meat, when
one of his own sheep had been killed to supply the table.
“That’s a first-rate way to make a stew,
that thy Aunt Mercy gave thee yesterday,”
he said, gravely, to Tiny, on one of these
occasions; “but I thought if I took the
mutton, and a few carrots and potatoes,
along with it, it would stand a good deal
better chance of getting made than if I
didn’t!”
And Tiny and Johnny delightedly agreed that it would.
Mr. Leslie came home, one evening, looking a little troubled.
 “I haven’t seen Jim at his usual place for two or three days,” he
said; “and if I could only have remembered the street and number of
his lodgings, I would have made time to go and ask after him.
Please write the address on a card for me, dear, and I’ll go to-
morrow, or send if I can’t go.”
The happy days in the country had by no means made Tiny and
Johnny forget Jim, in the hot and weary city; and, as Mr. Leslie often
saw him at his stand, messages were exchanged, and gifts of fruit
and flowers sent, which cheered his loneliness not a little, for he
missed them more than even they could guess. Aunt Mercy and
Uncle Isaac had heard a good deal about him, too, by this time; and
it so happened that they had come to a decision concerning him that
very day.
So now Aunt Mercy said,—
“I was going to speak to thee of that lad this very evening, Friend
Leslie. Our hired man, David, is obliged to leave us next month, and
I have taken a notion to ask thy young friend to take his place. The
work will not be heavy through the winter, and by spring, with good
care and good food in the meantime, he might well be strong
enough to keep on with David’s work, until our time for hiring extra
help comes. And we think it would be well if he could come at once,
while David is still here to instruct him, and we would pay him half
wages until David leaves. Would thee object to laying our proposal
before him, if thee sees him to-morrow?”
The applause which followed this speech quite embarrassed Aunt
Mercy; but she was made to understand very clearly that Mr. Leslie
would not have the slightest objection to undertaking her mission.
Tiny and Johnny were confident that Jim would come the very
next day; and when Mr. Leslie saw the blank faces which greeted
him as he returned, the next evening, alone, he pretended that he
meant to go back to the office immediately.
“For the office cat is always glad to see me,” he said, “and
especially so when I come alone!”
 He received, immediately, an overwhelming apology and
testimonial, all in one. But when it was over, Tiny asked,—
“Why didn’t Jim come with you, papa, really and truly?”
“Jim is slightly ill at his lodging,” said Mr. Leslie. “It is nothing
serious,” he hastened to add, as he saw the anxious faces. “I took
the doctor to see him, and he says Jim has a slight touch of bilious
fever. He is wretchedly uncomfortable, of course, for the old woman
of the house does as little for him as she decently can; but I gave
her a talking to, and the doctor says, he hopes to have Jim on his
legs again in two or three days, though, of course, he will be rather
weak for a while.”
This news caused much lamentation, which was instantly changed
to joy, when Uncle Isaac said, quietly, and as if it were the only thing
to be said under the circumstances,—
“If thee will give me the address, Friend Leslie, I will drive in for
the lad to-morrow. Mercy can arrange a bed in the bottom of the
spring wagon, and I think the slight risk we shall cause him to run
will be justifiable, under the circumstances. The kitchen-chamber is
vacant, and he can sleep there, until David goes.”
Mr. Leslie clasped the old man’s hand with affectionate warmth,
nor could he help saying softly, so that only Uncle Isaac heard,—
“‘I was a stranger, and ye took Me in; sick, and in prison, and ye
visited Me.’”
Aunt Mercy asked Tiny and Johnny to help her make ready the
kitchen chamber, the next day, and Johnny will never receive any
more delightful flattery than her gentle,—
“Thee is such a carpenter, Johnny, and so handy, that I thought
perhaps thee could bore a gimlet-hole in the floor, here by the bed,
and then fix a piece of twine along one of the rafters in the kitchen,
till it reached the door-bell—no one-ever rings that, thee knows, and
that poor boy may want something, and be too weak to call.”
 So Johnny arranged the bell-pull, while Aunt
Mercy and Tiny tacked up green paper shades,
and white muslin curtains, to the two windows
and spread the straw mattress, first with three
or four folded “comfortables,” and then with
lavender-scented sheets and a white bed-
spread, and put a clean cover on the bureau,
and on the little one-legged and three-footed
table which was to stand by the bed. Two or
three braided rugs were laid upon the floor,
and then, when Tiny had decorated the
bureau with a bunch of the brightest flowers she could find, the
room was all ready, “and too lovely for anything,” as Tiny said.
Jim was afraid, at first, that his new friends would
not understand why he could not, try as he might,
find voice to say anything, when Uncle Isaac and
David carried him upstairs, and gently placed him on
the white bed. There was a lump in his throat which
would not let any words pass it, but he raised his
eyes to Aunt Mercy’s face, with a look which
somehow made her stroke his hot forehead with her
soft, cool hands, and say tenderly,—
“There, my dear, thee is safe and at home, and all
thee has to do is to lie here and get well as fast as thee can!”
He did it, and with everything to help forward his recovery, his
strong young frame soon shook off disease and languor.
Three weeks after he came to the farm, he was “all about again,”
as Aunt Mercy said, and so eager for work, that he soon left David
little to do. And what famous help he was about the “mission!” He
seemed to have an especial faculty for finding the places where shy
mosses and delicate wild-flowers hid; he had “spotted” every nut
tree within five miles before the nuts were ripe, and he packed their
various findings in a way which excited wonder and admiration.
 The “beautiful time” in the inner circle came
to an end at last, or rather, to a pause; nobody
was willing to believe it the end. There were
plans and hopes for next year, and for the
winter which must come first, but, in spite of
all the hopes, nobody looked very cheerful
when the last evening came, and if Mrs. Leslie
and Aunt Mercy did not mingle their tears with
those of Tiny and Johnny, the next morning, it
was only because they felt that they must set
a good example even if nobody were able to
follow it!
And you, who are reading this? Are you trying, ever so little, to
share your happiness? Think about it. No one is too poor to do this.
Those of you who enjoy, every summer, a free, happy holiday in the
country, can be “faithful in much,” and those who are themselves
suffering privation can give, always, love and sympathy, and often
the “helping hand” which does so much beside the actual help it
gives. And remember, dear children who are listening to me, that
with the “Inasmuch as ye did,” comes the far more solemn
“Inasmuch as ye did it not, unto the least of these My brethren, ye
did it not to Me.”
 THE DEAD DOLL
And Other Verses.
By MARGARET VANDEGRIFT.
Author of “Little Helpers,” etc.
1 Vol. Square 8vo. Fully illustrated. Uniform with “Davy and the
Goblin,” etc. $1.50.
A charming collection of wise and witty verses for children, many
of which, like “THE DEAD DOLL,” “THE FATE OF A FACE-MAKER,”
etc., are very popular, and have been copied all over the country;
and are household words in thousands of families, where this
complete and beautiful edition will be eagerly welcomed. Among the
other poems are

THE GALLEY CAT.

SLUMBER-LAND.
AT SUNSET.
WINNING A PRINCESS.
 THE CAT AND THE FIDDLE.
A DREAM OF LITTLE WOMEN.
THE CLOWN’S BABY.
THE KING’S DAUGHTER.

These poems are not only very attractive and interesting to

children, but they also have a great fascination for all who care for
children, and for sweetness and innocence of life.
Sent, postpaid, on receipt of price, by the publishers,
TICKNOR & CO., Boston.

AT CLOSE QUARTERS THE FIRST DAY AT GETTYSBURG.

The Recollections of a Drummer Boy.

 By Rev. HARRY M. KIEFFER,
Late of the 150th Pennsylvania Volunteers.

Copiously illustrated with scenes in camp and field. 1 vol. Square

8vo. Revised and enlarged, and printed from entirely new plates.
$1.50.
A new and enlarged edition of this admirable book, which is
particularly adapted for youths, and should be placed in the hands of
every lad in the country, to impart a knowledge of the old war days.
The position of the author, as a clergyman of the Reformed
Church, gives the book a certain value to all persons interested in
true and pure literature, which is also of the greatest power of
attraction. “The Recollections of a Drummer Boy” has become a very
popular book for Sunday-school libraries; and should be read by all
old soldiers and their children. The great demand for the book has
compelled the publishers to issue this enlarged and beautified new
edition.
“The author describes the war fever and enlistment, the advance
to Virginia, the battles of Chancellorsville, Gettysburg, the
Wilderness, Petersburg, and the end, with a simplicity and
straightforwardness that are full of pathos. The evening camps, the
frugal ‘hard tack,’ the long marches over ‘the sacred soil,’ the
Bucktail cantonments under the dark Virginia pines, the whir of the
long roll, the silent watch of midnight pickets, the songs of the
camp, the moans of the hospital, the white tents on Maryland hills,
the joyous rush of artillery coming into action, the imposing
splendors of Presidential reviews—all these and a thousand other
phases of that exciting era are reproduced here with picturesque
fidelity; and once more its readers are ‘Tenting on the old Camp-
ground.’”—Washington Herald.
Sent, postpaid, on receipt of price, by the Publishers,
TICKNOR & CO., Boston.
 JUAN and JUANITA.
By FRANCES COURTENAY BAYLOR.
Author of “On Both Sides,” etc.
1 vol. Square 4to. With many illustrations $1.50.
Miss Baylor’s charming and “ower true” tale has formed (though
only given in part) the chief attraction of the “St. Nicholas” for a
year, and in its present and complete form will be heartily welcomed,
most of all by those who have already learned to love its little hero
and heroine, and will eagerly look for the full story of their
adventures.
The locale of these events, amid the romantic scenery of Northern
Mexico and Western Texas, is brilliantly and accurately described,
with the ways and habits of the Texans, Mexicans, and Indians. With
these are the records of the young hero and heroine, in and beyond
the Cañon of Roses, and their numerous strange and diverting
adventures, making a volume of rare and permanent interest for
young or old.

THREE GOOD GIANTS.

By FRANÇOIS RABELAIS.
Translated by John Dimitry. With 175 Pictures by Gustave
Doré and Anton Robida.
$1.50. Uniform with “Davy and the Goblin,” etc.
“The present beautiful edition of an amusing book cannot fail to
amuse thousands of little ones, who perhaps in these days are
growing tired of ‘Gulliver’s Travels,’ ‘Robinson Crusoe,’ ‘Alice in
Wonderland,’ and ‘The Arabian Nights.’”—The Week.
“Coleridge classes Rabelais with ‘the great creative minds,
Shakspeare, Dante, and Cervantes.’ In ‘Three Good Giants,’ children,
young and old, will find a story which will vie in delightful interest