Deep Learning Examples With Pytorch And Fastai A

Developers Cookbook Bernhard J Mayr download

https://ebookbell.com/product/deep-learning-examples-with-
pytorch-and-fastai-a-developers-cookbook-bernhard-j-mayr-43818478

Explore and download more ebooks at ebookbell.com

Here are some recommended products that we believe you will be
interested in. You can click the link to download.

Practical Computer Vision Applications Using Deep Learning With Cnns

With Detailed Examples In Python Using Tensorflow And Kivy 1st Edition
Ahmed Fawzy Gad

https://ebookbell.com/product/practical-computer-vision-applications-
using-deep-learning-with-cnns-with-detailed-examples-in-python-using-
tensorflow-and-kivy-1st-edition-ahmed-fawzy-gad-7280142

Practical Computer Vision Applications Using Deep Learning With Cnns

With Detailed Examples In Python Using Tensorflow And Kivy 1st Edition
Ahmed Fawzy Gad

https://ebookbell.com/product/practical-computer-vision-applications-
using-deep-learning-with-cnns-with-detailed-examples-in-python-using-
tensorflow-and-kivy-1st-edition-ahmed-fawzy-gad-61258050

Introduction To Deep Learning With Complexe Python And Tensorflow

Examples Jurgen Brauer

https://ebookbell.com/product/introduction-to-deep-learning-with-
complexe-python-and-tensorflow-examples-jurgen-brauer-7409008

Machine Learning With Python Design And Develop Machine Learning And
Deep Learning Technique Using Real World Code Examples Abhishek
Vijayvargia

https://ebookbell.com/product/machine-learning-with-python-design-and-
develop-machine-learning-and-deep-learning-technique-using-real-world-
code-examples-abhishek-vijayvargia-50457874
Deep Learning In Visual Computing Explanations And Examples Hassan
Ugail

https://ebookbell.com/product/deep-learning-in-visual-computing-
explanations-and-examples-hassan-ugail-48754194

Deep Learning By Example A Handson Guide To Implementing Advanced

Machine Learning Algorithms And Neural Networks Menshawy

https://ebookbell.com/product/deep-learning-by-example-a-handson-
guide-to-implementing-advanced-machine-learning-algorithms-and-neural-
networks-menshawy-7240872

Artificial Intelligence By Example Acquire Advanced Ai Machine

Learning And Deep Learning Design Skills 2nd Edition Denis Rothman

https://ebookbell.com/product/artificial-intelligence-by-example-
acquire-advanced-ai-machine-learning-and-deep-learning-design-
skills-2nd-edition-denis-rothman-10815800

Deep Learning Ian Goodfellow Yoshua Bengio Aaron Courville

https://ebookbell.com/product/deep-learning-ian-goodfellow-yoshua-
bengio-aaron-courville-44886094

Deep Learning In Biology And Medicine Davide Bacciu Paulo J G Lisboa

https://ebookbell.com/product/deep-learning-in-biology-and-medicine-
davide-bacciu-paulo-j-g-lisboa-44899934
Deep Learning Examples with
PyTorch and fastai
A Developers’ Cookbook
Dr. Bernhard J. Mayr, MBA [email protected]

© 2020 – All Rights reserved

Contents
1 Introduction 7
1.1 Prerequisites . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8
1.2 Resources . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8
1.3 Introduction to fastai . . . . . . . . . . . . . . . . . . . . . . . . 8
1.4 Using Google’s Colab-Tool . . . . . . . . . . . . . . . . . . . . . 12
1.5 Install fastai library . . . . . . . . . . . . . . . . . . . . . . . . 15
1.6 First Example: Dog Breed Classifier . . . . . . . . . . . . . . . 16

2 Build an Image Classifier from Scratch 31

2.1 Let’s analyze the Dataset . . . . . . . . . . . . . . . . . . . . . 31
2.2 Create the Model . . . . . . . . . . . . . . . . . . . . . . . . . . 35
2.3 Taking a deeper dive into the Source Code . . . . . . . . . . . . 45

3 SGD – Stochastic Gradient Descent 49

3.1 Model Optimizer . . . . . . . . . . . . . . . . . . . . . . . . . . 51
3.2 Animation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 53

4 Multi-Label Classification 55
4.1 Using the DataBlock API with ColumnReader . . . . . . . . . 56
4.2 Using a different Approach – Instrumentalizing lambdaExpressions . . . . .
. . . . . . . . . . . . . . . . . . . . . . . . 57
4.3 Third Approach – use a Custom get_items Function . . . . . . 58
4.4 Create and Train the Model . . . . . . . . . . . . . . . . . . . . 60

5 Improve our Pet Classifier with Multi-classification 65

6 Cross-Fold-Validation 69
6.1 Create a Baseline for training . . . . . . . . . . . . . . . . . . . 72
6.2 Build the Cross-Validation-Model . . . . . . . . . . . . . . . . . 73

7 FastAI – A Glance on the internal API 77

7.1 The PILImage-Class . . . . . . . . . . . . . . . . . . . . . . . . 77
7.2 The ImageBlock Function . . . . . . . . . . . . . . . . . . . . . 78
7.3 The Points Class . . . . . . . . . . . . . . . . . . . . . . . . . . 79
7.4 Create the PointScaler . . . . . . . . . . . . . . . . . . . . . . 83

8 Image Segmentation 89
8.1 Let’s get the CAMVID Dataset . . . . . . . . . . . . . . . . . . . 89
8.2 Training . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 94

9 ImageWoof Dataset 113

9.1 The Architecture and the new Implementations . . . . . . . . . 116
9.2 Training and Optimizing . . . . . . . . . . . . . . . . . . . . . . 118
9.3 One_Cycle_Fit vs. Fit_Flat_Cosine . . . . . . . . . . . . . . . 119

10 Focus on DataBlock API 123 10.1 Using Dataloaders . . . . . . . . . . . . . . . .

. . . . . . . . . 130

11 Style-Transfer 137
11.1 The pretrained Network . . . . . . . . . . . . . . . . . . . . . . 138
11.2 Create the Loss-Function . . . . . . . . . . . . . . . . . . . . . 140
11.3 Create the Model Architecture . . . . . . . . . . . . . . . . . . 144
11.4 DataLoaders and Learners . . . . . . . . . . . . . . . . . . . . . 147
11.5 Do the Style-Transfer on one Image . . . . . . . . . . . . . . . . 149
11.6 Upscale the Model . . . . . . . . . . . . . . . . . . . . . . . . . 152
11.7 Exporting the Model . . . . . . . . . . . . . . . . . . . . . . . . 152
11.8 Export our Style-Transfer Library . . . . . . . . . . . . . . . . 153

12 Server deployment 155

13 Building your own Learner from any Model architecture 159 13.1
Pretrained PKL File . . . . . . . . . . . . . . . . . . . . . . . . 165

14 Keypoints Detection 169

14.1 DataSet . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 169
14.2 Visualize some of the Images . . . . . . . . . . . . . . . . . . . 172
14.3 Create DataBlock and Train our Model . . . . . . . . . . . . . 175

15 Hybridizing Models 185

15.1 Keypoint Regression . . . . . . . . . . . . . . . . . . . . . . . . 185
15.2 Using our CustomUNET with the PETS Dataset . . . . . . . . 199
15.3 Using our Custom UNET-Learner with the Planets Dataset . . 202

16 Object Detection 205

16.1 The Dataset PASCAL_2007 . . . . . . . . . . . . . . . . . . . . . 205
16.2 The RetinaNet Model . . . . . . . . . . . . . . . . . . . . . . . 207
16.3 The Loss-Function . . . . . . . . . . . . . . . . . . . . . . . . . 212
16.4 Using the Model for Inference . . . . . . . . . . . . . . . . . . . 219

17 Multimodal Learning with the Bengali.ai Dataset from Kaggle 223

CONTENTS

17.1 Preprocessing the Data . . . . . . . . . . . . . . . . . . . . . . 223

17.2 Build the fastai DataBlock . . . . . . . . . . . . . . . . . . . . 227
17.3 Custom Heads . . . . . . . . . . . . . . . . . . . . . . . . . . . 228
17.4 Training the Model . . . . . . . . . . . . . . . . . . . . . . . . . 231

18 Super-resolution GANs 235

18.1 Let’s build the crappified Data . . . . . . . . . . . . . . . . . . 235
18.2 Build our DataBlock . . . . . . . . . . . . . . . . . . . . . . . . 237
18.3 Pre-Trained Generator . . . . . . . . . . . . . . . . . . . . . . . 239
18.4 Build the Critic . . . . . . . . . . . . . . . . . . . . . . . . . . . 240
18.5 Training of the Critic . . . . . . . . . . . . . . . . . . . . . . . . 243

19 Siamese Twins 247 19.1 Creating the Dataloader . . . . . . . . . . . . . . . . . . . .

. . 252

20 Audio 255
20.1 Why is Audio different (to Images)? . . . . . . . . . . . . . . . 255
20.2 The Audio Dataset . . . . . . . . . . . . . . . . . . . . . . . . . 256
20.3 Preparing the Dataset . . . . . . . . . . . . . . . . . . . . . . . 257
20.4 Put the Pieces together – Using the Datablock API . . . . . . . 259
20.5 Training . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 261
20.6 Add Data Augmentation . . . . . . . . . . . . . . . . . . . . . . 263
20.7 Mel Frequency Cepstral Coefficient (MFCC) . . . . . . . . . . . 266
20.8 MFCC + Delta . . . . . . . . . . . . . . . . . . . . . . . . . . . 268

21 Introduction to Tabular Data – How to use Pandas DataFrame271

21.1 The Common Functionalities of DataFrame . . . . . . . . . . . 272
21.2 Working with multiple DataFrames . . . . . . . . . . . . . . . . 276
21.3 Dropping of Columns . . . . . . . . . . . . . . . . . . . . . . . . 278
21.4 Grouping (GroupBy) . . . . . . . . . . . . . . . . . . . . . . . . 279
21.5 Map Function . . . . . . . . . . . . . . . . . . . . . . . . . . . . 280
21.6 Apply Function . . . . . . . . . . . . . . . . . . . . . . . . . . . 281

22 Tabular Data with FastAI 283

22.1 Tabular Pandas . . . . . . . . . . . . . . . . . . . . . . . . . . . 283
22.2 Normalizing the Data . . . . . . . . . . . . . . . . . . . . . . . 285
22.3 Fill missing Values . . . . . . . . . . . . . . . . . . . . . . . . . 286
22.4 The DataLoaders . . . . . . . . . . . . . . . . . . . . . . . . . . 287
22.5 Create the TabularLearner . . . . . . . . . . . . . . . . . . . . . 289
22.6 Inference from our TabularLearner . . . . . . . . . . . . . . . . 293

23 TabularData used with Regression 295

23.1 Import the data . . . . . . . . . . . . . . . . . . . . . . . . . . . 297
23.2 Building the Model . . . . . . . . . . . . . . . . . . . . . . . . . 301
23.3 Now let’s do some Inference on the Test-Data! . . . . . . . . . . 302
23.4 Permutation Importance . . . . . . . . . . . . . . . . . . . . . . 303

24 Ensembling Models with TabularData 311

24.1 Gradient Boosting . . . . . . . . . . . . . . . . . . . . . . . . . 312
24.2 Bring in the fastai library . . . . . . . . . . . . . . . . . . . . 312
24.3 Ensembling . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 316
24.4 Random Forest Model . . . . . . . . . . . . . . . . . . . . . . . 316

25 Bayesian Optimization 319

26 Analyzing Neural Nets with the SHAP Library 325 26.1 Using the
fastshap Library . . . . . . . . . . . . . . . . . . . . 326
27 Using TabNet Library with fastai 329 27.1 Creating the TabNet Model .
. . . . . . . . . . . . . . . . . . . 332
28 Introduction to Natural Language Processing 335 28.1 We will start
with calculating the Sentiment of Movie Reviews 335 29 Resources 343

Chapter 1
Introduction
This book will guide you through a journey on Deep Learning using the most
recent version of the library fast.ai (Version 2). If you install the fastai library
now, you will always get version 2.

If you need a former version for some old tutorials, follow the following link:
echo 'fastai 1.*' >> $CONDA_PREFIX/conda-meta/pinned

You can then install fastai v1. If you no longer need v1, just remove the
$CONDA_PREFIX/conda-meta/pinned file. The repository for version 1 will
be renamed in fastaiv1. There will be bug fixes but no further development.

This book will show you working examples on how to use the fastai
framework for deep learning. Fastai is well suited for developers who are just
starting out with deep learning as well as those who already have some
background on deep learning and are familiar with the pytorch framework.

There are even tutorials on how to migrate your projects from different deep
learning frameworks to fastai in the docs. See https://docs.fast.ai/#Mig rating-
from-other-libraries for further information.

You will get a quick introduction on the fastai framework and why we think
it is currently one of the best frameworks for deep learning. You will also get
an introduction to the google colab platform (an online application with free
access to GPU).
All chapters of this book are designed with a work book style in focus. You
should get the code with a few lines of prosa text describing each variable of
the examples. The book will not describe the theory on the algorithms that
were implemented but just give you as much code as possible. You should try
to replicate the examples and adapt them to suite your needs. You will get
references to papers or tutorials for further reading if you are more interested
in some background knowledge.
The aim of developing this book was to give you an idea on how you can
adapt those examples to fit your personal needs!

1.1 Prerequisites

This book is a collection of working examples on applying deep learning to

real world problems. Therefore you should have some experience in Python
programming and you should be familiar with basic concepts on deep
learning with pytorch.

All the code examples have been developed with Jupyter notebooks. If you
are not familiar with Jupyter, just have a look at the documentation: https:
//jupyter.org/.

If you want to run the code examples on your local machine, be sure to have
a Nvidia GPU with at least 8 GB RAM installed. Also make sure you have
installed jupyter server to run jupyter notebooks.

You can also use google colab and run the examples online, totally free of
charge. (Yes of course google will track all your activities… but who cares?
Well at least not while studying.) Nevertheless, you should always consider
data privacy with real world production projects.

1.2 Resources

While working with fastai, I have always received good advice by using the
official forum: forums.fast.ai as well as the official documentation:
docs.fast.ai.

There is also a discord server on which you can attend live coding sessions
with Jeremy Howard (founder of fastai). Just use this link to get started:
https://discord.com/invite/xnpeRdgv.

1.3 Introduction to fastai

Fastai is a library for easy development of deep neural nets that helps you
make state-of-the-art model architectures more easily accessible. You can
dive deep into the basics and adapt elements from the framework to suite
your needs. The framework is open-source and available via github.

There are four main types of models directly supported by fastai:

• Computer Vision (fastai2.vision)

• Tabular Data (fastai2.tabular)
• Text (fastai2.text)
• Collaborative Filtering (fastai2.collab)

This means you can get going quickly with default models on these classes.
These are called applications in the fastai architecture model. You can,
however, dig deeper if you need more fine grained modifications.

Fastai’s architecture consists of three layers:

• High Level Applications: With the high level applications you can build
state-of-the-art deep learning models with less than ten lines of codes. Most
beginners will start here.
• Mid Layer API: Provides more flexibility on designing your deep learning
applications. This level is especially useful if you need modifications on your
data pre-processing and data loaders or if you plan to use your own learners
and model architectures. With the mid layer api you can also monitor your
training progress with TensorBoard. You can schedule different values of
hyper-parameters for training or try different variants of regularization.
• Foundations: This level provides all the basic stuff if you need to work
with tensors or you want to manage operations directly on your GPU.
Fastai also integrates the concept of Callbacks. This helps you to use the
same training loop for very different deep learning applications. You can
customize your training loop to build GAN networks for example. Callbacks
in fastai are two-way callbacks. They can read anything like data, parameters,
etc. as well as write them. This means you can even do the training of a GAN
with the same training loop.

Some examples on how to use Callbacks are: * Learning Rate Schedulers *

Implement early stopping * Do parallel training * Gradient Clipping

Another concept that shows the great flexibility of fastai is that of a generic
optimizer. You can create your own optimizer by just plugging in statistics,
steppers and optimizers.
These architectures allow you to easily implement any new state-of-the-art
optimizer on your own. The statistics module contains any data that you want
to measure while doing the training. The stepper uses these stats to change
the weights of the net. If you now combine these two modules (stats and
steppers) you are able to implement any of the optimizers.

With the data block api, you are able to get your data into any model. For
using the data block api you need to know the following:

• what type of data you have?

• how do you get your data?
• how do you split the data into training and validation sets?
• how do you put your data into a data bunch?
E.g. when doing handwritten digit classification, you could use this code:

mnist = DataBlock(type=(PILImageBW, Category),

get_items=get_image_files,
splitter=GrandparentSplitter(), get_y=parent_label)

databunch = mnist.databunch(untar_data(URLs.MNIST_TINY))
databunch,show_batch(max_n=9, figsize=(4,4))
With this example we say: ok, we have black/white PILImages as input and a
Category for output. We get our data with a list of files. Let’s split the data
into training and validation sets by using the grandparent path of the files.
We grab the y-labels by using the parent path.
This is enough to use the mnist data set with our models. We can have a look
at our data by using the show_batch method.
Of course the data block api works for all applications of deep learning like
image segmentation, object detection, keypoint analysis, text, etc.

According to the foundation level of fastai, a new object type was added:
Object oriented tensors. With oo tensors you can call methods directly on
your tensor objects. E.g. if you have a tensor object containing an image, you
can now just call tensor.flip_lr to flip the image or .show to display the
image. Now any transform can be defined for different types of tensor
objects.

To pre-process data, you can use the transforms pipeline to easily create a
data transform pipeline.

The lowest level is a Callable Transform. The pipeline does function

composition over transforms. It knows how to set up transforms in the
pipeline. This can be a bit tricky because you have to make sure that all the
previous steps have been applied to the pipeline before setting up the next
step.
Next you can put these into an easily indexed collection called transformed
list and apply your transforms in parallel. Finally you use a data loader that
applies these transforms in parallel.

Generally speaking, all the development of the fast.ai library was done by
highly leveraging parallel computing on the GPU. E.g. most of the computer
vision pipeline that is included with fastai is done in parallel on the GPU,
minimizing computing time.

If you are interested in the facts behind architecture design decisions in

fast.ai, you should watch this video of Jeremy Howard on Youtube:
https://youtu. be/bHVqO5YyNbU

1.4 Using Google’s Colab-Tool

If you want to run the examples in this book and you don’t have a GPU on
your own, you have a few options.

You can start an EC2 instance on Amazon AWS (or any other computing
platform). This will bill you for the time while running the instance. Make
sure you have selected an instance level with a GPU. There are a few pros
and cons though. If you are working on production level projects, you should
consider a secure and reliable infrastructure stack. Then running a GPU
server on a cloud platform will be a good solution.

For learning or working on examples/prototypes, etc. there is another handy

option.

Google provides a tool called Colab that is (kind of) a hosted Jupyter
Notebook instance which is completely free of charge, but of course with
some caveats:

• Google does not reassure access to a GPU but there are hardly times
without GPU access.
• Most of the time your session is paired to old, and rather slow, Tesla K80
GPUs.
• Your session is reset after a maximum runtime of 12 hours. This means all
your data will be lost after 12 hours if you do not upload it to another storage
facility.

You can improve performance by buying google colab pro. For 10 dol lars a
month you get access to faster V100 GPUs and your maximum runtime is
extended to 24 hours.

Opening the following link ( https://colab.research.google.com/), will enable

you to directly open any Jupyter notebook file – even from a github
repository. With colab, you get a GPU instance free of charge.

However, keep in mind the following condition: if you are implementing

projects with colab, the runtime will be reset every 12 hours. To prevent data
loss you should think of an alternative solution. I would suggest you use
Google Drive (gdrive). You can easily store and load data with gdrive. While
doing the training loop with fastai, you could plug in your custom callback
that saves models after each training epoch on your gdrive…

1.4.1 Opening a notebook file from a Github repository

Open google colab at https://colab.research.google.com
Click on File->Open Notebook
Now you can choose from where to load the file. You should see:

• Examples (some examples from colab)

• Recent (your latest notebooks)
• Google Drive (mount your gdrive to directly access notebooks)
• GitHub (you can open notebooks from any publicly available repository)
• Upload (upload a local notebook-file to your colab runtime)

1.4.2 How to change your runtime to use a GPU

For most usages of deep learning we want to use a GPU because computing
takes a lot less time on the GPU rather on a CPU. To change the
processingtype of your runtime follow these steps:

1. Click on runtime
2. Change runtime type
3. Set runtime type to Python 3
4. Use Hardware accelerator -> GPU

On first opening a notebook from a public repository, you will probably get a
warning like this:
You can safely ignore this warning if you consider the repository secure and
click on the run anyway button.

1.4.3 Saving your work

If you are working with a notebook from github, you can open and run the
notebook but you cannot save your work. If you want to keep your work, you
can: – save a copy to your google drive – download your notebook and run
locally (or upload again later)

To store your notebook to gdrive, just click on File->Download .ipynb or

File->Save a copy in Drive
1.5 Install fastai library

If you are using google colab, you do not have to install any additional library
as the fastai imports work out of the box.
If you are running your notebooks locally or on your custom GPU-instance
you will have to install fastai first.

I would advise you to use anaconda or miniconda to manage your python

environments. You can find documentation on those here: * https://www.
anaconda.com/ * https://docs.conda.io/en/latest/miniconda.html *
https://docs.conda.io/en/latest/

This is a link to a discussion on the difference between these concepts on

stack overflow: *
https://stackoverflow.com/questions/45421163/anacondavs-miniconda

I suggest you create an environment for your fastai projects and install the
fastai library afterwards into this environment. You can also read the full
installation guide online: https://docs.fast.ai/

conda install c fastai c pytorch c anaconda fastai gh → anaconda

After installing, we can import all libraries. For the first example we will use
basics, callback, and vision. Then we will try a vision task with fastai v2.
from fastai.basics import * from fastai.vision.all import * from
fastai.callback.all import *

1.5.1 General Procedure on Machine Learning Projects

If we want to create a model with fastai, we will mostly follow the following
path: 1. Make all the DataLoaders 1. Make a Learner with some equipment.
Here you will have to decide which task you want to solve. (See different
types of deep learning tasks) 1. Train the model 1. Evaluate, tune and retrain
your model. I will discuss different techniques on how to do this.

1.6 First Example: Dog Breed Classifier

There are a bunch of datasets which you can download very easily with the
fastai library. One of them contains images of different dog and cat breeds
(12 species of cats and 25 species of dogs). To show the massive
improvements in deep learning, think of a state-of-the-art accuracy of about
59% in the year 2015. We will try to beat this of course.

If you want to get a deeper insight on the implementation of different

functions you can just type help(untar_data). This shows you the help text of
any python function.

Or type ??untar_data to directly show the source code of the function

help(untar_data)
Help on function untar_data in module fastai.data.external:

untar_data(url, fname=None, dest=None, c_key='data', ␣

→force_download=False, extract_func=<function file_extract␣ →at
0x7f1baf730560>)

Download `url` to `fname` if `dest` doesn't exist, and␣ →un-tgz or unzip to

folder `dest`.
??untar_data

1.6.1 Getting the Data

Let’s download the data set and have a look at some of the images. We will
directly use the Path library to set a reference to our data-folder. The URLs
class contains constant values for all the different data sets that ship with the
fastai library.

The data set we will use is called Oxford-IIIT Pet Dataset. You can have a
look at the homepage to get a better understanding of the data: (https:
//www.robots.ox.ac.uk/~vgg/data/pets/)

There are special images for the heads because back in 2015 the best model
for distinguishing the breeds was a combined model with different layers
trained on the whole image and on recognizing just parts of the image like the
head, etc.
We will use the whole image for training.
path = untar_data(URLs.PETS)

If we want to be able to recreate the same results, we should set a random-

seed value:
np.random.seed(2)
After downloading all the images we see that path contains a list of POSIX-
Path objects to all the images:
path.ls()[:2]

(#2) [Path('/home/bmayr/.fastai/data/oxford-iiit-pet/
→crappy'),Path('/home/bmayr/.fastai/data/oxford-iiit-pet/ →images')]

Let’s analyze how many image folders our data set contains:
len(path.ls())
5
path.ls()

(#5) [Path('/home/bmayr/.fastai/data/oxford-iiit-pet/
→crappy'),Path('/home/bmayr/.fastai/data/oxford-iiit-pet/
→images'),Path('/home/bmayr/.fastai/data/oxford-iiit-pet/
→models'),Path('/home/bmayr/.fastai/data/oxford-iiit-pet/
→image_gen'),Path('/home/bmayr/.fastai/data/oxford-iiit-pet/ →annotations')]

The fastai library is not just an add on on top of pytorch. Fastai comes with
many more features and implements state-of-the-art techniques from deep
learning science with the goal to provide a very smart abstraction layer for
building rich, performance models with four to five lines of codes.

But you are also able to dig deep into the basics and use fastai for building
your own, more sophisticated models. For this reason, fastai comes with a
three tier architecture that we have already discussed above.

Just a quick reminder:

On top there are the four different applications (Vision, Text, Tabular, Col
lab).
This layer allows the building of state-of-the-art models with a few lines of
codes.

Below these layers, there is a kind of “ mid-level-layer ” that allows for more
control on building models, especially on handling your data and data-
processing. The mid-layer contains the Data Block API, Learner and Cal
lbacks and Generic Optimizers. We will use this layer in later examples.

Below the mid-layer there is the foundation API. This allows for total control
on models and data flow.

Let’s get back to our starter model. First of all we need to grab all the
filenames from the images to build our DataLoaders. We use a helper
function get_image_files() from fastai that returns a list of all filenames in a
given directory.

file_names = get_image_files(path/'images/')
file_names[:2]

(#2) [Path('/home/bmayr/.fastai/data/oxford-iiit-pet/
→images/Birman_105.jpg'),Path('/home/bmayr/.fastai/data/ →oxford-iiit-
pet/images/Siamese_175.jpg')]

If we have a look at the filenames, we see that the name contains the label
that we want to predict later. E.g. /home/bmayr/.fastai/data/oxford-iiit-
pet/images/Birman_105.jpg - this is the image of a Birman cat.

With fastai we have different options on how to get the labels for our training
data. We can either extract the label from a parent folder path, from a csv-file
or data frame or we use a regular expression to extract the label from the
path.

So let’s grab all characters after the last / to the _: '(.+)_\d+.jpg$'

If you are not familiar with regular expressions – don’t be afraid, you are not
alone. There are plenty of good tutorials on the web and you will find
examples for nearly all possible applications on stack overflow. To test your
regEx you can use one of the many online-editors that show which portion of
a string matches your regular expression. E.g. regex101.com: (https:
//regex101.com/)

pat = r'(.+)_\d+.jpg$'

Next we want to add some basic transforms. Batch-transforms are applied to

each batch on your training data whereas item_transforms will be applied to
any of your items before adding them to different batches.

Transforms are an excellent way of augmenting your training data. E.g. you
can zoom in or out a bit, flip your images, warp, etc. to generate even more
training examples. Fastai implements data augmentation in a very fast way by
doing image data augmentation in parallel on your GPU.

In this example we want to resize our images to 224x224 pixels and

normalize on the imagenet_stats because we will use ResNet34 as a base
model for transfer learning.

Each image will be randomly cropped before being added to a batch.

The batch size will be 64 – you can adjust the value according to your
GPUmemory. The higher the batch size, the more memory will be used on
your GPU.

batch_tfms = [*aug_transforms(size=224, max_warp=0),

Normalize.from_stats(*imagenet_stats)] item_tfms =
RandomResizedCrop(460, min_scale=0.75,
→ ratio=(1.,1.))
bs=64

An ImageDataLoader is the highest possible API to use with fastai. We can

directly use the method from_name_re that allows us to generate data-labels
from the path-name by using our regular expression. We use the regEx
pattern we had defined above.

data = ImageDataLoaders.from_name_re(path,
file_names,
pat,
batch_tfms=batch_tfms, item_tfms=item_tfms, bs=bs)

Now we rebuild this DataLoader with the Data-Block-API. The Data Block
API always requires the same steps: 1. define data blocks 1. get the data
items 1. define a splitter to split into training and validation set 1. get the
labels 1. set item transforms 1. set batch transforms

We set the datablock to ImageBlock because we have a vision task and

Category Block because we want to predict exactly one category. Later we
develop an example on how to predict multi-class labels.

Why do we want to split our data into training and validation data?

We train our model on the training dataset. Yet if we use all the available data
for training, the model could easily overfit, meaning that it just memorizes
the images. E.g. it learns that the third image is a cat, but it doesn’t learn what
exactly specifies a cat. Therefore we recheck after each training epoch on a
dataset what the model has not seen during training. This dataset is called the
validation dataset.

pets = DataBlock(blocks=(ImageBlock, CategoryBlock),

get_items=get_image_files,
splitter=RandomSplitter(),
get_y=RegexLabeller(pat = r'/([^/]+)_\d+.*'), item_tfms=item_tfms,
batch_tfms=batch_tfms)

Now with the DataBlock-object in hand, we can create our dataloaders from
the image filenames. Set the path_img to the images folder and create the
dataloaders.

path_img = path/'images/'
dls = pets.dataloaders(path_img, bs=bs)

By now we have followed all the necessary steps for defining our training
and validation data. It’s always a good idea to have a look at the data before
any training. So we can check the validity of our training examples.

We can use show_batch to print a random batch from our dataloaders. This is
a quick check on data sanity.
dls.show_batch(max_n=9, figsize=(6,7))
Let’s see how many classes there are in our training data. In fastai this is
called the vocab (vocabulary) of our dataloaders.
dls.vocab

(#37) ['Abyssinian','Bengal','Birman','Bombay',
'British_Shorthair','Egyptian_Mau','Maine_Coon',
'Persian','Ragdoll','Russian_Blue'...]

There is a total of 37 different classes.

Fastai uses the class L for representing lists. L was developed for fastai to
improve the capabilities of lists in python. Internally all the classes are
translated to integer, but thanks to L we directly see class-names. If we want
to have a look at the ints, we can call dls.vocab.o2i. o2i means a translation
from objects to integer values.

dls.vocab.o2i
{'Abyssinian': 0,
'Bengal': 1,
'Birman': 2,
'Bombay': 3,
'British_Shorthair': 4,
'Egyptian_Mau': 5,
'Maine_Coon': 6,
'Persian': 7,
'Ragdoll': 8,
'Russian_Blue': 9,
'Siamese': 10,
'Sphynx': 11,
'american_bulldog': 12,
'american_pit_bull_terrier': 13,
'basset_hound': 14,
'beagle': 15,
'boxer': 16,
'chihuahua': 17,
'english_cocker_spaniel': 18, 'english_setter': 19,
'german_shorthaired': 20,
'great_pyrenees': 21,
'havanese': 22,
'japanese_chin': 23,
'keeshond': 24,
'leonberger': 25,
'miniature_pinscher': 26,
'newfoundland': 27,
'pomeranian': 28,
'pug': 29,
'saint_bernard': 30,
'samoyed': 31,
'scottish_terrier': 32,
'shiba_inu': 33,
'staffordshire_bull_terrier': 34, 'wheaten_terrier': 35,
'yorkshire_terrier': 36}

1.6.2 Let’s train the model

For our vision task we will be using a convolution neural network backbone
(Resnet34).

This technique is called Transfer-Learning. We use a pre-trained model

(ResNet) which was trained on the imagenet dataset. It does not match our
prediction classes, so we just cut the last layer and add a dense layer
according to the dataclasses of our dataset.

The idea of transfer learning is to use the power of a deep model and retrain
just the last layer on our specific task. This reduces training time considerably
while producing great results quickly.

First of all we create a Learner object. Fastai provides different learners for
different tasks. We have a vision-task so let’s grab a cnn_learner.
Each learner needs: – DataLoaders for managing training and validation data
– a model defining the architecture – an evaluation metric – a loss function
and – an optimizer

We will also add mixed_precision (fp16) because this reduces memory usage
during training.

learn = cnn_learner(dls,
resnet34,
pretrained=True,
metrics=error_rate).to_fp16()

For a single classification task we use CrossEntropyFlat as loss function. If

we do not use any explicit optimizer, fastai uses Adam as default. So let’s
train for some epochs – we will use default values for parameters like
learning-rate, momentum, weight decay, etc.
learn.fit_one_cycle(4)
epoch train_loss

0 1.885583
1 0.656676
2 0.343732
3 0.253843
valid_loss error_rate time

0.388003 0.121786 00:22

0.308593 0.098106 00:23
0.272366 0.083897 00:22
0.281068 0.083897 00:22

We can save our weights for reloading any time later by just calling save on
our learner object. Optionally we use any name for storage we like.
learn.save('stage_1') Path(‘models/stage_1.pth’)

1.6.3 Analyze the Results

Well, 0.083 error rate is not that bad for our first deep learning attempt. Let’s
get into the model and analyze what went wrong while doing the
classification.

With fastai we have a great opportunity to look into our model’s results by
getting the classification-interpretation. This allows to show the most
mismatched examples and plot a confusion matrix.

interp = ClassificationInterpretation.from_learner(learn) losses, idxs =

interp.top_losses()
assert len(dls.valid_ds)==len(losses)==len(idxs)

This shows the top losses with the predicted and the actual class labels and
the loss-values as well as the probability.
interp.plot_top_losses(9, figsize=(15,10))

Now we print the confusion matrix. A “perfect” model would only have
values on the diagonal because this means that the model has always
predicted the correct label.

Below or above the diagonal are misclassified values.

interp.plot_confusion_matrix(figsize=(12,12), dpi=75) Not too bad for just
doing 4 epochs of training.
We are using training data with a lot of classes. Therefore, the confusion
matrix is a bit overwhelming and it’s hard to figure out which classes our
model mixed up more often.

Let’s have a look at the most confused classes:

interp.most_confused(min_val=4)

[(‘Egyptian_Mau’, ‘Bengal’, 7), (‘staffordshire_bull_terrier’,

‘american_pit_bull_terrier’, 7), (‘Russian_Blue’, ‘British_Shorthair’, 6),
(‘american_pit_bull_terrier’, ‘staffordshire_bull_terrier’, 5), (‘beagle’,
‘basset_hound’, 5), (‘Maine_Coon’, ‘Ragdoll’, 4), (‘Ragdoll’, ‘Birman’, 4),
(‘Siamese’, ‘Birman’, 4), (‘english_cocker_spaniel’, ‘english_setter’, 4)]

These are the classes that were mixed up at least 4 times. Well, I am not a
dog breed expert but after some research on these breeds I have to admit,
these are really hard to differentiate.
1.6.4 Unfreeze the whole Model and retrain

Now we can unfreeze all the layers of the pretrained ResNet model and
retrain the whole model for some epochs to get better results.

We can reload the weights from our last training cycle.

learn.load('stage_1');

learn.unfreeze()
learn.fit_one_cycle(4)

epoch train_loss valid_loss error_rate time

0 1.119134 2.879944 0.575778 00:29

1 1.005705 1.229593 0.342355 00:29
2 0.594478 0.549008 0.167794 00:29
3 0.307384 0.419407 0.117727 00:29

So our error rate went bad. After the first round of training we had an error
rate of about 0.08 but now we got 0.11!

What happened here? This is because this time we had unfreezed all layers of
the model before running the training loop and training such a big model is a
way more difficult task than just retraining the last layer of a given model.
Therefore we have to carefully decide on the training parameters.

Fastai provides a method called lr_find() that helps us in finding a good

learning rate.
Let’s reload our weights and call lr_find() beforehand.
learn.load('stage_1');
learn.lr_find()
SuggestedLRs(lr_min=6.309573450380412e-08,
lr_steep=1.0964781722577754e-06)
We want to select a range on the diagram before the loss rate skyrockets. In
this case a range between 1e-6 and 1e-4 will be a good choice. Yet you will
probably have to try different values to improve the results.

When training the whole model we select a range and not a single value
because we want to train the first layers with a lower learning rate than the
last layers. This is called learning-rate annealing. A concept presented in
Stanford’s CS231: (http://cs231n.github.io/neural-networks-3/#anneal)

learn.unfreeze();
learn.fit_one_cycle(4, lr_max=slice(1e-6, 1e-4))
epoch train_loss valid_loss error_rate time

0 0.240683 0.272126 0.081867 00:29

1 0.207130 0.268688 0.078484 00:29
2 0.186820 0.258128 0.074425 00:29
3 0.165958 0.257685 0.075101 00:29

We see that choosing a better learning rate improved our model’s

performance a lot. Now we get an error rate of 0.067659 - better than after
training only the last layer (0.07).
Now we can again save away our trained weights.
If you are using google colab you should consider copying your weights on
your gdrive or any other persistent cloud storage.
learn.save('stage_2')
Path(‘models/stage_2.pth’)

1.6.5 Retrain with ResNet50

Now we have seen that our architecture works and returns good results. We
can try to use a bigger model for transfer learning and restart using ResNet50.

A bigger model always needs more resources on training but doesn’t

necessarily provide better results! A good idea is to start with a subset of your
training data on a small model. Then you can do rapid prototyping and do not
have to waste time while your model is training. If you think you have
obtained good results, train your model on the complete training data and/or
try a bigger model architecture.

pets = DataBlock(blocks=(ImageBlock, CategoryBlock),

get_items=get_image_files,
splitter=RandomSplitter(),
get_y=RegexLabeller(pat = r'/([^/]+)_\d+.*'), item_tfms=item_tfms,
batch_tfms=batch_tfms)

dls = pets.dataloaders(untar_data(URLs.PETS)/"images", bs=32)

# create the learner

learn = cnn_learner(data,
resnet50,
pretrained=True,
metrics=error_rate)

Downloading: “ https://download.pytorch.org/models/resnet50- 19c8e

357.pth” to /home/bmayr/.cache/torch/hub/checkpoints/resnet50- 19
c8e357.pth

HBox(children=(IntProgress(value=0, max=102502400), HTML(value=”)))

# find learning rate
learn.lr_find()
SuggestedLRs(lr_min=0.00831763744354248,
lr_steep=0.0012022644514217973)

# choose a
learning rate with a high negative slope
learn.fit_one_cycle(5, lr_max=(1e-3))
epoch train_loss valid_loss error_rate time

0 1.197415 0.257222 0.089986 00:48

1 0.480817 0.243021 0.084574 00:47
2 0.285627 0.192219 0.062246 00:47
3 0.159144 0.184564 0.053451 00:47
4 0.123560 0.187495 0.057510 00:47

So with the bigger model architecture, we get an even lower error rate of
0.05. Now we can do unfreezing and/or training for more epochs.

learn.save( 'resnet50')
Path(‘/home/bmayr/.fastai/data/oxford-iiit-pet/models/resnet50.pth’) Again
we can run unfreeze() and retrain the whole model

learn.load( 'resnet50')
learn.unfreeze()
learn.fit_one_cycle(2, lr_max=4e-4)
epoch train_loss valid_loss error_rate time 0 0.800914 0.891309 0.264547
01:01 1 0.403630 0.329750 0.096752 01:00
learn.save('resnet50-stage_2')

) stage_2.pth’)
Well, running the training loop for only two epochs on this large model is
definitely not sufficient. If we want to use this model in a production
environment we will have to do more epochs.

Yet for demonstration purposes the model returns acceptable results with a
minimum number of lines of code.

Chapter 2
Build an Image Classifier from
Scratch
This chapter will guide you through the creation of a deep learning model for
a classification task from scratch.
We will build our own deep learning model to recognize handwritten digits.
To train our classifier we will use the MNIST training dataset. This dataset
consists of grayscaled images of handwritten digits.

You can find some more info on this dataset at wikipedia: https://en.wikip
edia.org/wiki/MNIST_database. We will use the URLs-class from fastai for
easy downloading of all the mnist data.

Now import the vision libraries from fastai and download the mnist dataset.
This time we will use the lowest level api from fastai to access our data.

from fastai.basics import *

from fastai.vision.all import *
from fastai.callback.all import *

path = untar_data(URLs.MNIST)
2.1 Let’s analyze the Dataset

The first step of any ML/DL Project is always an analysis of the dataset. So
let’s have a look at our mnist dataset!

We will use DataSource to create our dataset. DataSource needs a list of

filenames. So let’s grab them! We use the function get_image_files from
fastai that returns a list with PosixPath-objects to all the images within a
given folder.
_Remember: You can get documentation on any function by just calling ??
function_name (-> ??get_image_files)

items = get_image_files(path)
Now let’s look at the data and how it’s organized.

We see that all the files are stored in folders indicating training, validation
and test set.
Within these datasets there are folders for each of the classes like 0, 1, … 9
containing all the image files.
items[[0,40,501]]

(#3) [Path(‘/home/bmayr/.fastai/data/mnist_png/training/3/31274.png’),
Path(‘/home/bmayr/.fastai/data/mnist_png/training/3/54648.png’),
Path(‘/home/bmayr/.fastai/data/mnist_png/training/3/27143.png’)]

Next we will look at how to create a PIL greyscaled image from the filename.
Most of the fastaiv2 API has this kind of create method like PILImageBW.
img = PILImageBW.create(items[0])
img.show()
<matplotlib.axes._subplots.AxesSubplot at 0x7f7297802450>

As can be seen, the dataset is organized in different folders for training and
validation. Therefore, we can use the GrandparentSplitter for splitting the
data into training and validation.
We just have to assign the names of the corresponding folders to our
GrandparentSplitter.

After creating the splitter we can use this splitter object to divide our
filenames into training and validation data. We will have a list consisting of
two lists. One containing the indices for the training data folder and one
containing the indices for the validation data folder.

splitter = GrandparentSplitter(train_name='training', → valid_name='testing')

Let’s recheck this! We see splits contains two objects (training and validation
set). We can index into the list and show the first three elements for training
as well as for validation. Our splitter uses indices <60000 for training and
>=60000 for validation!
splits = splitter(items)
print(len(splits))
print(splits[0][:3])
print(splits[1][:3])

2 [0, 1, 2] [60000, 60001, 60002]

Next step will be to create a DataSource-object. The datasource needs the
following values:

• the filenames
• some transforms
• the splits for training and validation

What do we mean by transforms?

The transforms, at this stage, aren’t meant to be image-transforms like

rotation, cropping, zooming in/out, etc. These transforms define how our data
is meant to be loaded into our datasource and how to get our y-values.
Therefore this parameter is a list with two items. Each of these two items is a
list, containing the transforms for the x and the y-values.

With our data we want:

• to grab the x-values and create a PILImage (like we did manually before)
• for the y-values, to get the labels from the parentfolder and categorize Al l
code that is contained in the transforms wil l be applied automatical ly
during training.
datasource = Datasets(items,
tfms=[[PILImageBW.create],
[parent_label, Categorize]], splits=splits)

After creating the datasource object we can look into it by using the show_at
method. We will have to pass the dataset (either train or valid) and the index
we want to look at.

We see greyscale images along with the correct labels from the datasets.
show_at(datasource.train, 3), show_at(datasource.valid, -3)

(<matplotlib.axes._subplots.AxesSubplot at 0x7f729756a990>,
<matplotlib.axes._subplots.AxesSubplot at 0x7f7297516dd0>)
The next step will be to add some transforms to augment the dataset. This
time we will have to define all the steps manually.
In general we will conduct the following tasks:

• Ensure that all the images are of the same size

• Make sure our outputs are tensors, which our model expects

• Add some augmentation to our data (like rotate, crop, etc.)

All the transforms are applied serially (read from left to right). This means we
can think of transforms to be like pipelines. First convert to tensor, second
apply some cropping, third …. )

The first transforms are those that have to be applied on each item before we
can load it into the batch. These transforms are applied on the CPU.

items_tfms = [ToTensor(),
CropPad(size=34, pad_mode=PadMode.Zeros), RandomCrop(size=28)]
Next step is to apply the batch transforms on the GPU. We call Cuda() to
enable GPU usage, convert the int-values to float-values to increase training
performance and then normalize our data.

With this model we do not need to normalize for the imagenet dataset
because we train our model from scratch.
gpu_tfms = [IntToFloatTensor(), Normalize()]
Now we can create our dataloaders-object. We set a batch-size and apply our
transforms to after_items and to after_batch.

dataloaders = datasource.dataloaders(bs=128,
after_item=items_tfms, after_batch=gpu_tfms)

Now we can show a random batch: dataloaders.show_batch()

Now we want to know what size inputs our model expects. Let’s grab a batch
and look at the shape of our xs and ys.
x_batch, y_batch = dataloaders.one_batch()
x_batch.shape, y_batch.shape, dataloaders.c
(torch.Size([128, 1, 28, 28]), torch.Size([128]), 10)
This means our model expects an input of type tensor with a shape of
1x28x28.

This exactly represents our black and white image (remember, one channel
for each colour – so just 1 for a b/w image) with a resolution of 28x28 pixel.
Earlier we defined our batchsize to have a value of 128.

The output size of one batch is 128 because we have one label for each image
in the batch – a total of 128 labels for each batch.

How many different y-values do we have? This is shown by dataloaders.c

(the c means classes). So how many different classes do we have? Right, we
have 10 different classes - one for each digit.

2.2 Create the Model

A (deep) learning model consists of different layers. Each layer is simply a

matrix multiplication. After several layers we should end up with our given y
labels. For the MNIST image-data we will create a model that resembles the
cnn_learner-object we had used in the introduction chapter.

Now we will use these types of layers:

• convolution layers,
• batch normalization layers,
• activation functions (ReLu) and
• a flatten layer as a top layer

2.2.1 Convolutional Layer

Convolutional layers have shown good results when working with images.
You can even transform different data into images and work with conv-
layers. E.g. you create spectograms of your music files and use these
spectograms like images.

What happens inside convolutional layers is well documented by Adit

Deshpande (see: https://adeshpande3.github.io/A-Beginner%27s-Guide-
ToUnderstanding-Convolutional-Neural-Networks/).

The analogy to a convolutional layer is a flashlight that highlights only one

spot inside a dark room. The flashlight moves from left to right and top to
bottom and scans the whole room. However, you only see one spot at a time.
In terms of convolutional layers we call this flashligh a filter or kernel. The
highlighted spot we see is called a receptive field.

The filter itself consists of an array of floats – called weights (or sometimes
parameters).

Keep in mind that the depth of the filter has to be of the same size as the
depth of the input. In our example we use one color channel – so we use a
depth of 1.

Next step is to start moving the filter around on our pixel array of the image.
We define this moving as convolving around the image.

While convolving, the filter multiplies the values inside the filter with the
original values inside the highlighted receptive field. During the process the
filter does an element-wise multiplication.

All the results of these multiplications are then summed up. The filter keeps
repeating for all the pixels in the image resulting in an activation or feature
map.

In our case (24x24 pixels images) this feature map will have 784 different
locations, that turns into a 28x28 array of pixels.

def conv(number_input: int, number_filter: int) -> nn.Conv3d: return

nn.Conv2d(number_input,
number_filter,
kernel_size=3,
stride=2,
padding=1)

2.2.2 Batch Normalization Layer

While our tensors are being passed through the model we want them to be
normalized. This will improve training speed and will allow each layer to
learn independently because each layer is re-normalized according to its
outputs.
def batch_norm(number_filters: int) -> nn.BatchNorm2d: return
nn.BatchNorm2d(number_filters)

2.2.3 Activation Function

An activation function is used to provide non-linearity to our models. The

activation function works with the weights and a bias during a step called
backpropagation (which we will look at in a later chapter).

These functions take care of the learning capability of our model. The
activation function can activate (return a positive value) or deactivate (return
a value less than zero) nodes within our layers.

The ReLu (or rectified linear unit) simply returns the input value if the value
is greater than zero or it returns zero for all input values equal or less than
zero: https://github.com/Kulbear/deep-learning-nano-foundation/wiki/Re LU-
and-Softmax-Activation-Functions#rectified-linear-units

def relu() -> nn.ReLU: return nn.ReLU(inplace=False)

2.2.4 Flattening Layer

The last layer will be flattened layer. We will simply use Flatten for this
purpose. Remember we can just use double question marks to show the
source code of any function: ??Flatten.

??Flatten

2.2.5 Create Model Architecture

We will create a model with five convolutional layers – each consisting of a

conv_layer, a batch_norm layer and a relu activation (except for the last
layer).

We can use nn.Sequential to build a simply sequential model with our

predefined layers. Just add all the layers. The first convolution layer has one
input_filter and an output size of 8. The second layer therefore must have an
input size of 8 and so on….

The last layer must have an output size of 10 because our data has 10
different classes (digits 0 to 9). We do not apply an activation function on the
last layer but squeeze the result into a flattened layer.
model = nn.Sequential(

conv( 1, 8),
batch_norm(8),
relu(),
conv(8, 16),
batch_norm(16),
relu(),
conv(16, 32),
batch_norm(32),
relu(),
conv(32, 16),
batch_norm(16),
relu(),
conv(16, 10),
batch_norm(10),
Flatten()

Now we can create a learner object by using the Learner class and instead of
using one of the pretrained model architectures (like we did with the
cnn_learner before) we now use our own model. We use a
CrossEntropyLossFlat loss-function because our learner will predict
probabilities for different classes.

learner = Learner(dataloaders,
model,
loss_func=CrossEntropyLossFlat(),
metrics=accuracy)

To check the exact sizes of the layers and the number of weights (parameters)
of our learner we call learner.summary()
learner.summary()
epoch train_loss valid_loss accuracy time 0 None None 00:00

Sequential (Input shape: ['128 x 1 x 28 x 28'])

================================================================
Layer (type) Output Shape Param # Trainable
================================================================
Conv2d 128 x 8 x 14 x 14 80 True
________________________________________________________________
BatchNorm2d 128 x 8 x 14 x 14 16 True
________________________________________________________________
ReLU 128 x 8 x 14 x 14 0 False
________________________________________________________________
Conv2d 128 x 16 x 7 x 7 1,168 True
________________________________________________________________
BatchNorm2d 128 x 16 x 7 x 7 32 True
________________________________________________________________
ReLU 128 x 16 x 7 x 7 0 False
________________________________________________________________
Conv2d 128 x 32 x 4 x 4 4,640 True
________________________________________________________________
BatchNorm2d 128 x 32 x 4 x 4 64 True
________________________________________________________________
ReLU 128 x 32 x 4 x 4 0 False
________________________________________________________________
Conv2d 128 x 16 x 2 x 2 4,624 True
________________________________________________________________
BatchNorm2d 128 x 16 x 2 x 2 32 True
________________________________________________________________
ReLU 128 x 16 x 2 x 2 0 False
________________________________________________________________
Conv2d 128 x 10 x 1 x 1 1,450 True
________________________________________________________________
BatchNorm2d 128 x 10 x 1 x 1 20 True
________________________________________________________________
Flatten 128 x 10 0 False
________________________________________________________________
Total params: 12,126
Total trainable params: 12,126 Total non-trainable params: 0

Optimizer used: <function Adam at 0x7f729d9be5f0> Loss function:

FlattenedLoss of CrossEntropyLoss()

Callbacks:
- TrainEvalCallback
- Recorder
- ProgressCallback

Check for the best learning rate with lr_find(). Remember we look for a value
with a very steep negative slope.
This time we will use a value about 1e-2 for training. After we have chosen a
valid learning rate we can train our model for some epochs.
learner.lr_find()
SuggestedLRs(lr_min=0.33113112449646,
lr_steep=0.3019951581954956)
learner.fit_one_cycle(3, lr_max=1e-2)

epoch train_loss
valid_loss accuracy time

0 0.217761 0.122905 0.964300 00:14

1 0.114047 0.064353 0.980400 00:14
2 0.077312 0.045065 0.987000 00:14

This individual model is developed completely with pytorch. By now we

have not used any code from fastai at all.

2.2.6 Simplify the Model

We can simplify our model by using ConvLayer instead of using our three
layer-types. The ConvLayer from the fastai library contains these three
layers: Conv2d, BatchNorm2d and ReLU. You can just use the double
question marks ??ConvLayer again or define a simple conv2 layer and look
at the output:

ConvLayer(1, 8)

ConvLayer( (0): Conv2d(1, 8, padding=(1, 1), bias=False) (1): tum=0.1,

affine=True, track_running_stats=True) (2): ReLU() )

def conv2(number_in: int, number_filter: int) -> ConvLayer: return

ConvLayer(number_in, number_filter, stride=2) This is exactly the same like
in our first model. Let’s use this and simplify our model:

model = nn.Sequential(
conv2(1, 8),
conv2(8, 16),
conv2(16, 32),
conv2(32, 16),
conv2(16, 10),
kernel_size=(3, 3), stride=(1, 1), BatchNorm2d(8, eps=1e-05, momen

Flatten()
)
Now repeat the same steps as before. Create the learner, find optimal learning
rate and do some training.

learner = Learner(dataloaders,
model,
loss_func=CrossEntropyLossFlat(),
 Another Random Document on
Scribd Without Any Related Topics
John Brown will live in history; but his name will not be found among
the names of those who have wrought for humanity and for
righteousness; or among the names of the martyrs and the saints
who "washed their robes and made them white in the blood of the
Lamb."
"YET SHALL HE LIVE": but it will be as a soldier of fortune, an
adventurer. He will take his place in history as such: and will rank
among adventurers as Napoleon ranks among marshals: as Captain
Kidd among pirates: and as Jonathan Wild among thieves.
APPENDICES
 APPENDIX I
CORRESPONDENCE WITH THE LATE D. W.
WILDER CONCERNING JOHN BROWN
 T
o
p
e
k
a
,
K
a
n
s
a
s
,
D
e
c
.
1
8
t
h
,
1
9
0
2
.
General D. W. Wilder, Hiawatha, Kansas.
My Dear General:
I would like to have you kindly tell me something valuable about
John Brown. I listened to your tribute to his memory, read before the
Historical Society on the 2nd inst. It recalled the admiration which I
entertained for the "Old Hero" throughout the many years of my life;
from young manhood up to about four years ago; when I attempted
to write a sketch of his life. It was in reading up to obtain data for
this sketch that the idol, which my credulity, I suppose, or
imagination had set up, went utterly to pieces in my hands. I read
faithfully what his biographers, Sanborn, and Redpath, and the other
fellows, have written about him, but none of them give up any
valuable facts. They all seem to be long on eulogy. They do overtime
on that. The whole performance is a continuous eulogium; but
historical facts, upon which to predicate a story, or upon which his
"immortal fame" is supposed to rest, are painfully lacking.... These
are some of the things which I went up against when I tried in good
faith to write about him, and they broke me all up, so I had to quit.
John Brown, the "Hero" and "Martyr," is a creation—Charlestown
furnished a simple text and the genius of his generation did the rest.
The brilliant minds of this age have exploited him in literary effects,
in prose, in poetry and oratory. They have placarded him "upon the
walls of time"; but I am compelled to believe that his fame thus
acquired, will not survive. The "why" may "repel the philosophic
searcher," but it cannot "defy" the historical searchers. History has
no enigmas.
I will be very glad indeed to have your opinions on this business.
V
e
r
y
t
r
u
l
y
y
o
u
r
s
 ,
H
i
l
l
P
.
W
i
l
s
o
n
.
In this letter the writer asked Mr. Wilder for his opinion upon Brown's
motives in their relation to several incidents that occurred in his life.
His reply is as follows:[514]
H
i
a
w
a
t
h
a
,
K
a
n
s
a
s
,
D
 e
c
.
2
0
,
1
9
0
2
.
My Dear Wilson:
... You have stood on various platforms and made many political
speeches. Did any of them endorse the sentiments you now hold?
The elder Booth, a man of genius, once staggered up to the
footlights and said to the crowded house: "You are all drunk," and
staggered off.
You think the people of your county, your state, your country and of
the civilized world, including its noblest spirits, do not know a hero,
an emancipator—first of his state, then of his nation. Only one
Kansan has made a speech that thrilled the world and is immortal.
You never read it. Only one Kansan lives in poetry, in song, in human
hearts, and is the constant theme of the historian, the dramatist, the
man of letters. You think he was a fool. The whole world has
pronounced its verdict on John Brown.
Y
o
u
r
s
t
r
u
l
 y
,
D
.
W
.
W
i
l
d
e
r
.
To this letter the writer replied:
T
o
p
e
k
a
,
K
a
n
s
.,
J
a
n
u
a
r
y
3
,
 1
9
0
3
.
My Dear General:
Your letter of the 20th ult., is received. I told you that I had gone the
limit of my vocabulary in expressing my admiration of John Brown. I
read the "speech that thrilled the world." I have read the poetry and
have sung the songs. I make the point that the speeches, the poetry,
and the songs are all there is behind John Brown. When I asked you
about some historical facts, you gave me more oratory. It seems to
have become a habit. If you ever analyze this man's character, you
will reverse your estimate of him.
The world sees Brown fighting, heroically, in the engine-house at
Harper's Ferry, but it does not inquire how he came to be there. It
was his death, and not his life, that gave him renown. Usually it is a
man's life—his actions, that determine his place among men. If it be
true that one unimpeachable fact will set aside the most plausible
opposing theory, then Brown's fame will not survive. The facts of his
life impeach the popular verdict.
Very truly yours,
Hill P. Wilson.
General D. W. Wilder, Hiawatha, Kansas.
 APPENDIX II
RECOLLECTIONS OF THE JOHN BROWN RAID
BY THE HON. ALEXANDER R. BOTELER, A
VIRGINIAN WHO WITNESSED THE FIGHT
Taken from The Century

On entering the room where John Brown was, I found him alone,
lying on the floor on his left side, and with his back turned toward
me. The right side of his face was smeared with blood from a sword
cut on his head, causing his grim and grizzled countenance to look
like that of some aboriginal savage with his war-paint on.
Approaching him I began the conversation with the inquiry:
"Captain Brown, are you hurt anywhere except on the head?"
"Yes, in my side, here," said he, indicating the place with his hand.
I then told him that a surgeon would be in presently to attend to his
wounds, and expressed the hope that they were not very serious.
Thereupon he asked me who I was, and on giving him my name he
muttered as if speaking to himself.
"Yes, yes—I know you now—member of congress—this district."
I then asked the question:
"Captain, what brought you here?"
"To free your slaves," was the reply.
"How did you expect to accomplish it with the small force you
brought with you?"
"I expected help."
"Where, whence, and from whom, Captain, did you expect it?"
"Here and from elsewhere," he answered.
"Did you expect to get assistance from whites here as well as from
the blacks?" was my next question.
"I did," he replied.
"Then," said I, "you have been disappointed in not getting it from
either?"
"Yes," he muttered, "I have—been—disappointed."
Then I asked him who planned his movement on Harper's Ferry, to
which he replied: "I planned it all myself," and upon my remarking
that it was a sad affair for him and the country, and that I trusted no
one would follow his example by undertaking a similar raid, he made
no response. I next inquired if he had any family besides the sons
who accompanied him on his incursion, to which he replied by telling
me he had a wife and children in the State of New York at North
Elba, and on my then asking if he would like to write to them and let
them know how he was, he quickly responded:
"Yes, I would like to send them a letter."
"Very well," I said, "you doubtless will be permitted to do so. But,
Captain," I added, "probably you understand that, being in the
hands of the civil authorities of the State, your letters will have to be
seen by them before they can be sent."
"Certainly," he said.
"Then, with that understanding," continued I. "There will, I am sure,
be no objection to your writing home; and although I have no
authority in the premises, I promise to do what I can to have your
wishes in that respect complied with."
"Thank you—thank you, sir," he said repeating his acknowledgment
for the proffered favor and, for the first time, turning his head
toward me.
In my desire to hear him distinctly, I had placed myself by his side,
with one knee resting on the floor; so that, when he turned, it
brought his face quite close to mine, and I remember well the
earnest gaze of the gray eye that looked straight into mine. I then
remarked:
"Captain, we, too, have wives and children. This attempt of yours to
interfere with our slaves has created great excitement and naturally
causes anxiety on account of our families. Now, let me ask you: Is
this failure of yours likely to be followed by similar attempts to create
disaffection among our servants and bring upon our homes the
horrors of a servile war?"
"Time will show," was his significant reply.
Just then a Catholic priest appeared at the door of the room. He had
been administering the last consolations of religion to Quinn, the
marine, who was dying in the adjoining office; and the moment
Brown saw him he became violently angry, and plainly showed, by
the expression of his countenance, how capable he was of feeling
"hatred, malice, and all uncharitableness."
"Go out of here—I don't want you about me—go out!" was the
salutation he gave the priest, who, bowing gravely, immediately
retired. Whereupon I arose from the floor, and bidding Brown good-
morning, likewise left him.
In the entry leading to the room where Brown was, I met Major
Russell, of the marine corps, who was going to see him, and I
detailed to him the conversation I had just had. Meeting the major
subsequently he told me that when he entered the apartment Brown
was standing up—with his clothes unfastened—examining the wound
in his side, and that, as soon as he saw him, forthwith resumed his
former position on the floor; which incident tended to confirm the
impression I had already formed, that there was a good deal of
vitality left in the old man, notwithstanding his wounds—a fact more
fully developed that evening after I had left Harper's Ferry for home,
when he had his spirited and historic talk with Wise, Hunter and
Vallandigham.
 APPENDIX III
THE CONSTITUTION ADOPTED AT CHATHAM,
CANADA
Copy of the Constitution, adopted at Chatham, Canada, May 8, 1858.
Mason Report, p. 48.

PROVISIONAL CONSTITUTION AND

ORDINANCE FOR THE PEOPLE OF THE UNITED
STATES

PREAMBLE

Whereas, slavery throughout its entire existence in the United

States, is none other than a most barbarous, unprovoked, and
unjustifiable war of one portion of its citizens upon another portion,
the only conditions of which are perpetual imprisonment and
hopeless servitude or absolute extermination; in utter disregard of
those eternal and self-evident truths set forth in our Declaration of
Independence: Therefore,
We, citizens of the United States, and the Oppressed People, who,
by a decision of the Supreme Court are declared to have no rights
which the White Man is bound to respect; together with all other
people degraded by the laws thereof, Do, for the time being ordain
and establish for ourselves, the following PROVISIONAL
CONSTITUTION and ORDINANCES, the better to protect our
Persons, Property, Lives and Liberties; and to govern our actions:

ARTICLE I
 QUALIFICATIONS FOR MEMBERSHIP

All persons of mature age, whether Proscribed, oppressed, and

enslaved Citizens, or of the Proscribed or oppressed races of the
United States, who shall agree to sustain and enforce the Provisional
Constitution and Ordinance of this organization, together with all
minor children of such persons, shall be held to be fully entitled to
protection under the same.

ARTICLE II

BRANCHES OF GOVERNMENT

The provisional government of this organization shall consist of three

branches, viz.: Legislative, Executive, and Judicial.

ARTICLE III

LEGISLATIVE

The legislative branch shall be a Congress or House of

Representatives, composed of not less than five, or more than ten
members, who shall be elected by all the citizens of mature age and
of sound mind, connected with this organization; and who shall
remain in office for three years, unless sooner removed for
misconduct, inability, or death. A majority of such members shall
constitute a quorum.

ARTICLE IV

EXECUTIVE

The executive branch of this organization shall consist of a President

and Vice-President, who shall be chosen by the citizens or members
of this organization, and each of whom shall hold his office for three
years, unless sooner removed by death, or for inability or
misconduct.

ARTICLE V

JUDICIAL

The judicial branch of this organization shall consist of one Chief-

Justice of the Supreme Court, and of four Associate Judges of said
Court; each constituting a Circuit Court. They shall each be chosen in
the same manner as the President, and shall continue in office until
their places have been filled in the same manner by election of the
citizens. Said court shall have jurisdiction in all civil or criminal
causes, arising under this constitution, except breaches of the Rules
of War.

ARTICLE VI

VALIDITY OF ENACTMENTS

All enactments of the legislative branch shall, to become valid during

the first three years, have the approbation of the President and the
Commander-in-Chief of the Army.

ARTICLE VII

COMMANDER-IN-CHIEF

A Commander-in-Chief of the army shall be chosen by the President,

Vice-President, a majority of the Provisional Congress, and of the
Supreme Court, and he shall receive his commission from the
President, signed by the Vice-President, the Chief Justice of the
Supreme Court, and the Secretary of War: and he shall hold his
office for three years, unless removed by death, or on proof of
incapacity of misbehavior. He shall, unless under arrest (and till his
place is actually filled as provided by the constitution) direct all
movements of the army, and advise with any allies. He shall,
however, be tried, removed, or punished, on complaint by the
President, by, at least, three general officers, or a majority of the
House of Representatives, or of the Supreme Court; which House of
Representatives (the President presiding); the Vice President, and
the members of the Supreme Court, shall constitute a court-martial,
for his trial; with power to remove or punish, as the case may
require; and to fill his place as above provided.

ARTICLE VIII

OFFICERS

A Treasurer, Secretary of State, Secretary of War, and Secretary of

the Treasury, shall each be chosen for the first three years, in the
same way and manner as the Commander-in-Chief; subject to trial
or removal on complaint of the President, Vice-President, or
Commander in Chief, to the Chief Justice of the Supreme Court; or
on complaint of the majority of the members of said court, or the
Provisional Congress. The Supreme Court shall have power to try or
punish either of those officers; and their places shall be filled as
before.

ARTICLE IX

SECRETARY OF WAR

The Secretary of War shall be under the immediate directions of the

Commander in Chief; who may temporarily fill his place, in case of
arrest, or of any inability to serve.

ARTICLE X
 CONGRESS OR HOUSE OF REPRESENTATIVES

The House of Representatives shall make ordinances for the

appointment (by the President or otherwise) of all civil officers
except those already named; and shall have power to make all laws
and ordinances for the general good, not inconsistent with this
Constitution and these ordinances.

ARTICLE XI

APPROPRIATION OF MONEY, ETC.

The Provisional Congress shall have power to appropriate money or

other property actually in the hands of the Treasurer, to any object
calculated to promote the general good, so far as may be consistent
with the provisions of this Constitution; and may in certain cases,
appropriate, for a moderate compensation of agents, or persons not
members of this organization, for important service they are known
to have rendered.

ARTICLE XII

SPECIAL DUTIES

It shall be the duty of Congress to provide for the instant removal of

any civil officer or policeman, who becomes habitually intoxicated, or
who is addicted to other immoral conduct, or to any neglect or
unfaithfulness in the discharge of his official duties. Congress shall
also be a standing committee of safety, for the purpose of obtaining
important information; and shall be in constant communication with
the Commander-in-Chief; the members of which shall each, as also
the President and Vice-President, members of the Supreme Court,
and Secretary of State, have full power to issue warrants returnable
as Congress shall ordain (naming Witnesses etc) upon their own
information, without the formality of a complaint. Complaint shall be
made immediately after arrest, and before trial; the party arrested to
be served with a copy at once.

ARTICLE XIII

TRIAL OF PRESIDENT AND OTHER OFFICERS

The President and Vice President may either of them be tried,

removed, or punished, on complaint made by the Chief Justice of the
Supreme Court, by a majority of the House of Representatives,
which House, together with the Associate Judges of the Supreme
Court, the whole to be presided over by the Chief Justice in the
cases of the trial of the Vice President, shall have full power to try
such officers, to remove, or punish as the case may require, and to
fill any vacancy so occurring, the same as in the case of the
Commander-in-Chief.

ARTICLE XIV

TRIAL OF MEMBERS OF CONGRESS

The members of the House of Representatives may, any and all of

them, be tried, and on conviction, removed or punished on
complaint before the Chief Justice of the Supreme Court, made by
any number of members of said House, exceeding one third, which
House, with the Vice President and Associate Judges of the Supreme
Court, shall constitute the proper tribunal, with power to fill such
vacancies.

ARTICLE XV

IMPEACHMENT OF JUDGES

Any member of the Supreme Court, tried, convicted, or punished by

removal or otherwise, on complaint to the President, who shall, in
such case, preside; the Vice-President, House of Representatives,
and other members of the Supreme Court, constituting the proper
tribunal (with power to fill vacancies); on complaint of a majority of
said House of Representatives, or of the Supreme Court; a majority
of the whole having power to decide.

ARTICLE XVI

DUTIES OF PRESIDENT AND SECRETARY OF STATE

The President, with the Secretary of State, shall immediately upon

entering on the duties of their office, give special attention to secure,
from amongst their own people, men of integrity, intelligence, and
good business habits and capacity; and above all, of first rate moral
and religious character and influence, to act as civil officers of every
description and grade, as well as teachers, chaplains, physicians,
surgeons, mechanics, agents of every description, clerks and
messengers. They shall make special effort to induce at the earliest
possible period, persons and families of that description, to locate
themselves within the limits secured by this organization; and shall,
moreover, from time to time, supply the names and residence of
such persons to the Congress, for their special notice and
information, as among the most important of their duties, and the
President is hereby authorized and empowered to afford special aid
to such individuals, from such moderate appropriations as the
Congress shall be able and may deem it advisable to make for that
object.
The President and Secretary of State, and in case of disagreement,
the Vice-President shall appoint all civil officers, but shall not have
power to remove any officer. All removals shall be the result of a fair
trial, whether civil or military.

ARTICLE XVII

FURTHER DUTIES
It shall be the duty of the President and Secretary of State, to find
out (as soon as possible) the real friends, as well as the enemies of
this organization in every part of the country; to secure among
them, innkeepers, private postmasters, private mail contractors,
messengers and agents: through whom may be obtained correct and
regular information, constantly; recruits for the service, places of
deposit and sale; together with needed supplies: and it shall be
matter of special regard to secure such facilities through the
Northern States.

ARTICLE XVIII

DUTIES OF THE PRESIDENT

It shall be the duty of the President, as well as the House of

Representatives, at all times, to inform the Commander-in-Chief of
any matter that may require his attention, or that may affect the
public safety.

ARTICLE XIX

DUTY OF PRESIDENT—CONTINUED

It shall be the duty of the President to see that the provisional

ordinances of this organization, and those made by Congress, are
properly and faithfully executed; and he may in cases of great
urgency call on the Commander-in-Chief of the army, or other
officers for aid; it being, however, intended that a sufficient civil
police shall always be in readiness to secure implicit obedience to
law.

ARTICLE XX

THE VICE-PRESIDENT
The Vice-President shall be the presiding officer of the Provisional
Congress and in case of tie shall give the casting vote.

ARTICLE XXI

VACANCIES

In case of death, removal, or inability of the President, the Vice-

President, and next to him, the Chief Justice of the Supreme Court,
shall be the President during the remainder of the term: and the
place of Chief-Justice thus made vacant shall be filled by Congress
from some of the members of said Court; and places of the Vice-
President and Associate Justice thus made vacant, filled by an
election by the united action of the Provisional Congress and
members of the Supreme Court. All other vacancies, not heretofore
specially provided for, shall, during the first three years, be filled by
the united action of the President, Vice-President, Supreme Court,
and Commander-in-Chief of the Army.

ARTICLE XXII

PUNISHMENT OF CRIMES

The punishment of crimes not capital, except in the case of

insubordinate convicts or other prisoners, shall be (so far as may be)
by hard labor on the public works, roads, etc.

ARTICLE XXIII

ARMY APPOINTMENTS

It shall be the duty of all commissioned officers of the army to name

candidates of merit for office or elevation to the Commander-in-
Chief, who, with the Secretary of War, and, in cases of disagreement,
the President, shall be the appointing power of the army: and all
commissions of military officers shall bear the signatures of the
Commander-in-Chief and the Secretary of War. And it shall be the
special duty of the Secretary of War to keep for constant reference
of the Commander-in-Chief a full list of names of persons nominated
for office, or elevation, by officers of the army, with the name and
rank of the officer nominating, stating distinctly but briefly the
grounds for such notice or nomination. The Commander-in-Chief
shall not have power to remove or punish any officer or soldier; but
he may order their arrest and trial at any time, by court-martial.

ARTICLE XXIV

COURT-MARTIALS

Court martials for Companies, Regiments, Brigades, etc., shall be

called by the chief officer of each command, on complaint to him by
any officer, or any five privates, in such command, and shall consist
of not less than five nor more than nine officers, and privates, one-
half of whom shall not be lower in rank than the person on trial, to
be chosen by the three highest officers in the command, which
officers shall not be a part of such court. The chief officer of any
command shall, of course be tried by a court-martial of the
command above his own. All decisions affecting the lives of persons,
or office of persons holding commission, must, before taking full
effect, have the signature of the Commander-in-Chief, who may also,
on the recommendation of, at least, one-third of the members of the
court martial finding any sentence, grant a reprieve or commutation
of the same.

ARTICLE XXV

SALARIES

No person connected with this organization shall be entitled to any

salary, pay, or emoluments, other than a competent support of
himself and family, unless it be from an equal dividend, made of
public property, on the establishment of peace, or of special
provision by treaty; which provision shall be made for all persons
who may have been in any active civil or military service at any time
previous to any hostile action for Liberty and Equality.

ARTICLE XXVI

TREATIES OF PEACE

Before any treaty of peace shall take effect, it shall be signed by the
President and Vice-President, the Commander-in-Chief, a majority of
the House of Representatives, a majority of the Supreme Court, and
a majority of all general officers of the army.

ARTICLE XXVII

DUTY OF THE MILITARY

It shall be the duty of the Commander-in-Chief, and all officers and

soldiers of the army, to afford special protection when needed, to
Congress, or any member thereof; to the President, Vice-President,
Treasurer, Secretary of State, Secretary of Treasury and Secretary of
War; and to afford general protection to all civil officers, other
persons having right to the same.

ARTICLE XXVIII

PROPERTY

All captured or confiscated property, and all property the product of

the labor of those belonging to this organization and their families,
shall be held as the property of the whole, equally, without
distinction; and may be used for the common benefit, or disposed of
for the same object; and any person, officer or otherwise, who shall
improperly retain, secrete, use, or needlessly destroy such property,
or property found, captured, or confiscated, belonging to the enemy,
or shall willfully neglect to render a full and fair statement of such
property by him so taken or held, shall be deemed guilty of a
misdemeanor and, on conviction, shall be punished accordingly.

ARTICLE XXIX

SAFETY OR INTELLIGENCE FUND

All money, plate, watches or jewelry, captured by honorable warfare,

found, taken or confiscated, belonging to the enemy, shall be held
sacred, to constitute a liberal safety or intelligence fund; and any
person who shall improperly retain, dispose of, hide, use, or destroy
such money or other article above mentioned, contrary to the
provisions and spirit of this article, shall be deemed guilty of theft,
and, on conviction thereof, shall be punished accordingly. The
Treasurer shall furnish the Commander-in-Chief at all times with a
full statement of the condition of such fund and its nature.

ARTICLE XXX

THE COMMANDER-IN-CHIEF AND THE TREASURY

The Commander-in-Chief shall have power to draw from the

Treasury the money and other property of the fund provided for it in
ARTICLE twenty-ninth, but his orders shall be signed also by the
Secretary of War, who shall keep strict account of the same; subject
to examination by any member of Congress, or general officer.

ARTICLE XXXI

SURPLUS OF THE SAFETY OR INTELLIGENCE FUND

It shall be the duty of the Commander-in-Chief to advise the
President of any surplus of the Safety or Intelligence Fund; who shall
have power to draw such surplus (his order being also signed by the
Secretary of State) to enable him to carry out the provisions of
Article Seventeenth.

ARTICLE XXXII

PRISONERS

No person, after having surrendered himself or herself a prisoner,

and who shall properly demean himself or herself as such, to any
officer or private connected with this organization, shall afterward be
put to death, or be subject to any corporal punishment, without first
having had the benefit of a fair and impartial trial: nor shall any
prisoner be treated with any kind of cruelty, disrespect, insult, or
needless severity: but it shall be the duty of all persons, male and
female, connected herewith, at all times and under all
circumstances, to treat all such prisoners with every degree of
respect and kindness the nature of the circumstances will admit of;
and to insist on a like course of conduct from all others, as in the
fear of Almighty God, to whose care and keeping we commit our
cause.

ARTICLE XXXIII

VOLUNTARIES

All persons who may come forward and shall voluntarily deliver up
their slaves, and have their names registered on the Books of the
organization, shall, so long as they continue at peace, be entitled to
the fullest protection of person and property, though not connected
with this organization, and shall be treated as friends, and not
merely as persons neutral.
 ARTICLE XXXIV

NEUTRALS

The persons and property of all non-slaveholders who shall remain

absolute neutral, shall be respected so far as the circumstances can
allow of it; but they shall not be entitled to any active protection.

ARTICLE XXXV

NO NEEDLESS WASTE

The needless waste or destruction of any useful property or article,

by fire, throwing open of fences, fields, buildings, or needless killing
of animals, or injury of either, shall not be tolerated at any time or
place, but shall be promptly and properly punished.

ARTICLE XXXVI

PROPERTY CONFISCATED

The entire and real property of all persons known to be acting either
directly or indirectly with or for the enemy, or found in arms with
them, or found wilfully holding slaves, shall be confiscated and
taken, whenever and wherever it may be found, in either free or
slave States.

ARTICLE XXXVII

DESERTION

Persons convicted, on impartial trial, of desertion to the enemy after

becoming members, acting as spies, or of treacherous surrender of
property, arms, ammunition, provisions, or supplies of any kind,
roads, bridges, persons or fortifications shall be put to death and
their entire property confiscated.

ARTICLE XXXVIII

VIOLATION OF PAROLE OF HONOR

Persons proven to be guilty of taking up arms after having been set

at liberty on parole of honor, or, after the same, to have taken an
active part with or for the enemy, direct or indirect, shall be put to
death and their entire property confiscated.

ARTICLE XXXIX

ALL MUST LABOR

All persons connected in any way with this organization, and who
may be entitled to full protection under it, shall be held as under
obligation to labor in some way for the general good, and any
persons refusing, or neglecting so to do, shall on conviction receive a
suitable and appropriate punishment.

ARTICLE XL

IRREGULARITIES

Profane Swearing, filthy conversation, indecent behavior, or indecent

exposure of person, or intoxication, or quarreling, shall not be
allowed or tolerated, neither unlawful intercourse of the sexes.

ARTICLE XLI

CRIMES
Persons convicted of the forcible violation of any female prisoner
shall be put to death.

ARTICLE XLII

THE MARRIAGE RELATION—SCHOOLS—THE SABBATH

The marriage relation shall be at all times respected, and the families
kept together as far as possible, and broken families encouraged to
re-unite, and intelligence offices established for that purpose,
schools and churches established, as soon as may be, for the
purpose of religious and other instructions; and the first day of the
week regarded as a day of rest and appropriated to moral and
religious instruction and improvement; relief to the suffering,
instruction of the young and ignorant, and the encouragement of
personal cleanliness; nor shall any person be required on that day to
perform ordinary manual labor, unless in extremely urgent cases.

ARTICLE XLIII

CARRY ARMS OPENLY

All persons known to be of good character, and of sound mind and

suitable age, who are connected with this organization, whether
male or female, shall be encouraged to carry arms openly.

ARTICLE XLIV

NO PERSON TO CARRY CONCEALED WEAPONS

No person within the limits of the conquered territory, except

regularly appointed policemen, express officers of the army, mail
carriers, or other fully accredited messengers of the Congress,
President, Vice-President, members of the Supreme Court, or
commissioned officers of the army—and those only under peculiar
circumstances—shall be allowed, at any time, to carry concealed
weapons; and any person not specially authorized so to do, who
shall be found so doing, shall be deemed a suspicious person, and
may be at once arrested by any officer, soldier, or citizen, without the
formality of a complaint or warrant, and may at once be subject to
thorough search, and shall have his or her case thoroughly
investigated; and be dealt with as circumstances, or proof, may
require.

ARTICLE XLV

PERSONS TO BE SEIZED

Persons within the limits of the territory holden by this organization,

not connected with this organization, having arms at all, concealed
or otherwise, shall be seized at once, or taken in charge of by some
vigilant officer; and their case thoroughly investigated: and it shall
be the duty of all citizens and soldiers, as well as officers, to arrest
such parties as are named in this and the preceding Section or
Article, without formality of complaint or warrant: and they shall be
placed in charge of proper officer for examination or for safe
keeping.

ARTICLE XLVI

THESE ARTICLES NOT FOR THE OVERTHROW OF

GOVERNMENT

The foregoing articles shall not be construed so as in any way to

encourage the overthrow of any State Government of the United
States: and look to no dissolution of the Union, but simply to
Amendment and Repeal. And our Flag shall be the same as our
Fathers fought under in the Revolution.

ARTICLE XLVII
 NO PLURALITY OF OFFICES

No two offices specially provided for, by this Instrument, shall be

filled by the same person at the same time.

ARTICLE XLVIII

OATH

Every Officer, civil or military, connected with this organization, shall,

before entering upon the duties of his office, make solemn oath or
affirmation, to abide by and support this Provisional Constitution and
these Ordinances. Also, every Citizen and Soldier, before being fully
recognized as such, shall do the same.
 APPENDIX IV
JOHN BROWN'S AUTOBIOGRAPHY
Written to Henry L. Stearns, son of George L. Stearns, and bearing
date Red Rock, Iowa, July 7, 1857.[515]
John was born May 9th, 1800, at Torrington, Litchfield County,
Connecticut; of poor but respectable parents: a descendant on the
side of his father of one of the company of the Mayflower who
landed at Plymouth 1620. His mother was descended from a man
who came at an early period to New England from Amsterdam, in
Holland. Both his Father's & Mother's Fathers served in the war of
the revolution: His Father's Father died in a barn at New York while
in the service, in 1776.
I cannot tell you of anything in the first Four years of John's life
worth mentioning save that at that early age he was tempted by
Three large Brass Pins belonging to a girl who lived in the family &
stole them. In this he was detected by his Mother; & after having a
full day to think of the wrong: received from her a thorough
whipping. When he was Five years old his Father moved to Ohio;
then a wilderness filled with wild beasts, & Indians. During the long
journey which was performed in part or mostly with an ox team; he
was called on by turns to assist a boy Five years older (who had
been adopted by his Father & Mother) & learned to think he could
accomplish smart things in driving the cows, and riding the horses.
Some times he met with Rattle Snakes which were very large; &
which some of the company generally managed to kill. After getting
to Ohio in 1805 he was for some time rather afraid of the Indians, &
of their Rifles; but this soon wore off; & he used to hang about them
quite as much as was consistent with good manners; & learned a
trifle of their talk. His Father learned to dress Deer Skins, & at 6
years old John was installed a young Buck Skin—He was perhaps
rather observing as he ever after remembered the entire process of
Deer Skin dressing; so that he could at any time dress his own
leather such as Squirl, Raccoon, Cat, Wolf, or Dog Skin; & also
learned to make Whip Lashes: which brought him some change at
times; & was of considerable service in many ways. At Six years old
John began to be quite a rambler in the wild new country finding
birds & Squirels, and sometimes a wild Turkey's nest. But about this
period he was placed in the school of adversity: which my young
friend was a most necessary part of his early training. You may laugh
when you come to read about it; but these were sore trials to John:
whose earthly treasures were very few & small. These were the
beginnings of a severe but much needed course of discipline which
he afterwards was to pass through; & which it is to be hoped has
learned him before this time that the Heavenly Father sees it best to
take all the little things out of his hand which he has ever placed in
them. When John was in his Sixth year a poor Indian boy gave him a
Yellow Marble the first he had ever seen. This he thought a great
deal of; & kept it a good while; but at last he lost it beyond recovery.
It took years to heal the wound; & I think he cried at times about it.
About Five months after this he caught a young Squirrel tearing off
his tail in doing it; & getting severely bitten at the same time himself.
He however held to the little bob tail Squirrel; & finally got him
perfectly tamed, so that he almost idolized his pet. This too he lost;
by wandering away; or by getting killed: & for a year or Two John
was in mourning; and looking at all the Squirrels he could see to try
and discover Bobtail if possible, I must not neglect to tell you of a
very bad & foolish habbit to which John was somewhat addicted. I
mean telling lies: generally to screen himself from blame; or from
punishment. He could not well endure to be reproached; & I now
think had he been oftener encouraged to be entirely frank; by
making frankness a kind of atonement for some of his faults; he
would not have been so often guilty of this fault; nor have been
obliged to struggle so long in after life with so mean a habit.
John was never quarrelsome; but was excessively fond of the
hardest & roughest kind of plays; & could never get enough [of]
them. Indeed when for a short time he was sometimes sent to
School the opportunity it afforded to wrestle & Snow ball & run &
jump & knock off old seedy wool hats; offered to him almost the
only compensation for the confinement & restraints of school. I need
not tell you that with such a feeling & but little chance of going to
school at all: he did not become much of a schollar. He would always
choose to stay at home & work hard rather than be sent to school; &
during the warm season might generally be seen barefooted &
bareheaded: with Buck skin Breeches suspended often with one
leather strap over his shoulder but sometimes with Two. To be sent
off through the wilderness alone to very considerable distances was
particularly his delight; & in this he was often indulged so that by the
time he was Twelve years old he was sent off more than a Hundred
Miles with companies of cattle; & he would have thought his
character much injured had he been obliged to be helped in any
such job. This was a boyish kind of feeling but characteristic
however.
At Eight years old John was left a Motherless boy which loss was
complete & permanent, for notwithstanding his Father again married
to a sensible, inteligent, & on many accounts a very estimable
woman: yet he never adopted her in feeling: but continued to pine
after his own Mother for years. This opperated very unfavorably
uppon him: as he was both naturally fond of females; & withall
extremely diffident; & deprived him of a suitable link between the
different sexes; the want of which might under some circumstances
have proved his ruin.
When the war broke out with England, his Father soon commenced
furnishing the troops with beef cattle, the collecting & driving of
which afforded him some opportunity for the chase (on foot) of wild
steers & other cattle through the woods. During this war he had
some chance to form his own boyish judgement of men & measures:
& to become somewhat familiarly acquainted with some who have
figured before the country since that time. The effect of what he saw
Welcome to our website – the perfect destination for book lovers and
knowledge seekers. We believe that every book holds a new world,
offering opportunities for learning, discovery, and personal growth.
That’s why we are dedicated to bringing you a diverse collection of
books, ranging from classic literature and specialized publications to
self-development guides and children's books.

More than just a book-buying platform, we strive to be a bridge

connecting you with timeless cultural and intellectual values. With an
elegant, user-friendly interface and a smart search system, you can
quickly find the books that best suit your interests. Additionally,
our special promotions and home delivery services help you save time
and fully enjoy the joy of reading.

Join us on a journey of knowledge exploration, passion nurturing, and

personal growth every day!

ebookbell.com