Visit https://ebookfinal.

com to download the full version and

explore more ebooks

RStudio for R Statistical Computing Cookbook

Over 50 practical and useful recipes to help you
perform data analysis with R by unleashing every
native RStudio feature 1st Edition Andrea
Cirillo
_____ Click the link below to download _____
https://ebookfinal.com/download/rstudio-for-r-
statistical-computing-cookbook-over-50-practical-and-
useful-recipes-to-help-you-perform-data-analysis-with-
r-by-unleashing-every-native-rstudio-feature-1st-
edition-andrea-cirillo/

Explore and download more ebooks at ebookfinal.com

Here are some suggested products you might be interested in.
Click the link to download

OpenStack Cloud Computing Cookbook 3rd Edition Over 110

effective recipes to help you build and operate OpenStack
cloud computing storage networking and automation Kevin
Jackson
https://ebookfinal.com/download/openstack-cloud-computing-
cookbook-3rd-edition-over-110-effective-recipes-to-help-you-build-and-
operate-openstack-cloud-computing-storage-networking-and-automation-
kevin-jackson/

R Data Analysis Cookbook 1st Edition Viswa Viswanathan

https://ebookfinal.com/download/r-data-analysis-cookbook-1st-edition-
viswa-viswanathan/

Basic Data Analysis for Time Series with R 1st Edition

Dewayne R. Derryberry

https://ebookfinal.com/download/basic-data-analysis-for-time-series-
with-r-1st-edition-dewayne-r-derryberry/

An Introduction to Statistical Methods and Data Analysis

6th Edition R. Lyman Ott

https://ebookfinal.com/download/an-introduction-to-statistical-
methods-and-data-analysis-6th-edition-r-lyman-ott/
Statistical Analysis with R Essentials For Dummies 1st
Edition Joseph Schmuller

https://ebookfinal.com/download/statistical-analysis-with-r-
essentials-for-dummies-1st-edition-joseph-schmuller/

SAS and R Data Management Statistical Analysis and

Graphics 1st Edition Ken Kleinman

https://ebookfinal.com/download/sas-and-r-data-management-statistical-
analysis-and-graphics-1st-edition-ken-kleinman/

PostgreSQL 9 administration cookbook over 150 recipes to

help you run an efficient PostgreSQL database in the cloud
Second Edition (Online-Ausg.) Riggs
https://ebookfinal.com/download/postgresql-9-administration-cookbook-
over-150-recipes-to-help-you-run-an-efficient-postgresql-database-in-
the-cloud-second-edition-online-ausg-riggs/

Parallel Computing for Data Science With Examples in R C

and CUDA 2nd Edition Norman Matloff

https://ebookfinal.com/download/parallel-computing-for-data-science-
with-examples-in-r-c-and-cuda-2nd-edition-norman-matloff/

Sencha Touch Cookbook 2nd Edition Over 100 hands on

recipes to help you understand the complete Sencha Touch
framework and solve your day to day problems Ajit Kumar
https://ebookfinal.com/download/sencha-touch-cookbook-2nd-edition-
over-100-hands-on-recipes-to-help-you-understand-the-complete-sencha-
touch-framework-and-solve-your-day-to-day-problems-ajit-kumar/
RStudio for R Statistical Computing Cookbook Over 50
practical and useful recipes to help you perform data
analysis with R by unleashing every native RStudio
feature 1st Edition Andrea Cirillo Digital Instant
Download
Author(s): Andrea Cirillo
ISBN(s): 9781784391034, 1784391034
Edition: 1st
File Details: PDF, 7.84 MB
Year: 2016
Language: english
RStudio for R
Statistical Computing
Cookbook

Over 50 practical and useful recipes to help you

perform data analysis with R by unleashing every
native RStudio feature

Andrea Cirillo

BIRMINGHAM - MUMBAI
RStudio for R Statistical Computing Cookbook

Copyright © 2016 Packt Publishing

All rights reserved. No part of this book may be reproduced, stored in a retrieval system,
or transmitted in any form or by any means, without the prior written permission of the
publisher, except in the case of brief quotations embedded in critical articles or reviews.

Every effort has been made in the preparation of this book to ensure the accuracy of the
information presented. However, the information contained in this book is sold without
warranty, either express or implied. Neither the author, nor Packt Publishing, and its dealers
and distributors will be held liable for any damages caused or alleged to be caused directly
or indirectly by this book.

Packt Publishing has endeavored to provide trademark information about all of the
companies and products mentioned in this book by the appropriate use of capitals.
However, Packt Publishing cannot guarantee the accuracy of this information.

First published: April 2016

Production reference: 1250416

Published by Packt Publishing Ltd.

Livery Place
35 Livery Street
Birmingham B3 2PB, UK.

ISBN 978-1-78439-103-4

www.packtpub.com
 Credits

Author Project Coordinator

Andrea Cirillo Shweta H Birwatkar

Reviewer Proofreader
Mark van der Loo Safis Editing

Commissioning Editor Indexer

Kartikey Pandey Rekha Nair

Acquisition Editor Graphics

Vinay Argekar Disha Haria

Content Development Editor Production Coordinator

Deepti Thore Aparna Bhagat

Technical Editor Cover Work

Madhunikita Sunil Chindarkar Aparna Bhagat

Copy Editor
Karuna Narayan
 About the Author

Andrea Cirillo is currently working as an internal auditor at Intesa Sanpaolo banking group.
He gained a lot of financial and external audit experience at Deloitte Touche Tohmatsu and
internal audit experience at FNM, a listed Italian company.

His current main responsibilities involve evaluation of credit risk management models and
their enhancement mainly within the field of the Basel III capital agreement.

He is married to Francesca and is the father of Tommaso, Gianna, and Zaccaria.

Andrea has written and contributed to a few useful R packages and regularly shares insightful
advice and tutorials about R programming.

His research and work mainly focuses on the use of R in the fields of risk management
and fraud detection, mainly through modeling custom algorithms and developing
interactive applications.

This book is the result of a lot of patience by my wife and sons, which left
me with the time to write this book, the time that I should have spend
with them.

By Deepti Thore, my content developer editor at Packt Publishing, who

was so clement with me when, and it happened a lot of time, I missed
my writing deadlines.

By my colleagues who endured my talks about the book every three hours
and when I asked for their opinions about almost every recipe.

To all of you, I would like to say a sincere thank you.

 About the Reviewer

Mark van der Loo is a statistical researcher who specializes in data cleaning methodology
and likes to program in R and C. He is the author and coauthor of several R packages published
on CRAN, including stringdist, validate, deductive, lintools, and several others. In 2012, he
authored Learning RStudio for R Statistical Computing, Packt Publishing, with Edwin de Jonge.
 www.PacktPub.com

eBooks, discount offers, and more

Did you know that Packt offers eBook versions of every book published, with PDF and ePub
files available? You can upgrade to the eBook version at www.PacktPub.com and as a print
book customer, you are entitled to a discount on the eBook copy. Get in touch with us at
[email protected] for more details.

At www.PacktPub.com, you can also read a collection of free technical articles, sign up
for a range of free newsletters and receive exclusive discounts and offers on Packt books
and eBooks.

TM

https://www2.packtpub.com/books/subscription/packtlib

Do you need instant solutions to your IT questions? PacktLib is Packt's online digital book
library. Here, you can search, access, and read Packt's entire library of books.

Why Subscribe?
ff Fully searchable across every book published by Packt
ff Copy and paste, print, and bookmark content
ff On demand and accessible via a web browser
 Table of Contents
Preface v
Chapter 1: Acquiring Data for Your Project 1
Introduction 1
Acquiring data from the Web – web scraping tasks 2
Accessing an API with R 12
Getting data from Twitter with the twitteR package 16
Getting data from Facebook with the Rfacebook package 21
Getting data from Google Analytics 24
Loading your data into R with rio packages 27
Converting file formats using the rio package 31
Chapter 2: Preparing for Analysis – Data Cleansing and Manipulation 33
Introduction 33
Getting a sense of your data structure with R 34
Preparing your data for analysis with the tidyr package 36
Detecting and removing missing values 40
Substituting missing values using the mice package 43
Detecting and removing outliers 47
Performing data filtering activities 48
Chapter 3: Basic Visualization Techniques 59
Introduction 59
Looking at your data using the plot() function 60
Using pairs.panel() to look at (visualize) correlations between variables 67
Adding text to a ggplot2 plot at a custom location 69
Changing axes appearance to ggplot2 plot (continous axes) 74
Producing a matrix of graphs with ggplot2 79
Drawing a route on a map with ggmap 85
Making use of the igraph package to draw a network 88
Showing communities in a network with the linkcomm package 93
i
Table of Contents

Chapter 4: Advanced and Interactive Visualization 99

Introduction 99
Producing a Sankey diagram with the networkD3 package 100
Creating a dynamic force network with the visNetwork package 104
Building a rotating 3D graph and exporting it as a GIF 110
Using the DiagrammeR package to produce a process flow diagram
in RStudio 112
Chapter 5: Power Programming with R 117
Introduction 117
Writing modular code in RStudio 118
Implementing parallel computation in R 120
Creating custom objects and methods in R using the S3 system 123
Evaluating your code performance using the profvis package 126
Comparing an alternative function's performance using the
microbenchmarking package 129
Using GitHub with RStudio 131
Chapter 6: Domain-specific Applications 141
Introduction 142
Dealing with regular expressions 142
Analyzing PDF reports in a folder with the tm package 143
Creating word clouds with the wordcloud package 148
Performing a Twitter sentiment analysis 151
Detecting fraud in e-commerce orders with Benford's law 156
Measuring customer retention using cohort analysis in R 161
Making a recommendation engine 163
Performing time series decomposition using the stl() function 165
Exploring time series forecasting with forecast() 167
Tracking stock movements using the quantmod package 168
Optimizing portfolio composition and maximising returns with
the Portfolio Analytics package 170
Forecasting the stock market 173
Chapter 7: Developing Static Reports 175
Introduction 175
Using one markup language for all types of documents – rmarkdown 177
Writing and styling PDF documents with RStudio 184
Writing wonderful tufte handouts with the tufte package and rmarkdown 186
Sharing your code and plots with slides 188
Curating a blog through RStudio 190

ii
 Table of Contents

Chapter 8: Dynamic Reporting and Web Application Development 197

Introduction 197
Generating dynamic parametrized reports with R Markdown 198
Developing a single-file Shiny app 204
Changing a Shiny app UI based on user input 209
Creating an interactive report with Shiny 213
Constructing RStudio add-ins 216
Sharing your work on RPubs 220
Deploying your app on Amazon AWS with ramazon 222
Index 225

iii
 Preface
Why should you read RStudio for R Statistical Computing Cookbook?

Well, even if there are plenty of books and blog posts about R and RStudio out there, this
cookbook can be an unbeatable friend through your journey from being an average R and
RStudio user to becoming an advanced and effective R programmer.

I have collected more than 50 recipes here, covering the full spectrum of data analysis
activities, from data acquisition and treatment to results reporting.

All of them come from my direct experience as an auditor and data analyst and from
knowledge sharing with the really dynamic and always growing R community.

I took great care selecting and highlighting those packages and practices that have proven
to be the best for a given particular task, sometimes choosing between different packages
designed for the same purpose.

You can therefore be sure that what you will learn here is the cutting edge of the R language
and will place you on the right track of your learning path to R's mastery.

What this book covers

Chapter 1, Acquiring Data for Your Project, shows you how to import data into the R
environment, taking you through web scraping and the process of connecting to an API.

Chapter 2, Preparing for Analysis – Data Cleansing and Manipulation, teaches you how to
get your data ready for analysis, leveraging the latest data-handling packages and advanced
statistical techniques for missing values and outlier treatments.

Chapter 3, Basic Visualization Techniques, lets you get the first sense of your data, highlighting
its structure and discovering patterns within it.

Chapter 4, Advanced and Interactive Visualization, shows you how to produce advanced
visualizations ranging from 3D graphs to animated plots.

v
Preface

Chapter 5, Power Programming with R, discusses how to write efficient R code, making use of
the R objective-oriented systems and advanced tools for code performance evaluation.

Chapter 6, Domain-specific Applications, shows you how to apply the R language to a wide
range of problems related to different domains, from financial portfolio optimization to
e-commerce fraud detection.

Chapter 7, Developing Static Reports, helps you discover the reporting tools available within
the RStudio IDE and how to make the most of them to produce static reports for sharing
results of your work.

Chapter 8, Dynamic Reporting and Web Application Development, displays the collected
recipes designed to make use of the latest features introduced in RStudio from shiny web
applications with dynamic UIs to RStudio add-ons.

What you need for this book

The basic requirements for this book are the latest versions of R and RStudio, which you can
download from the following URLs:

ff For Windows: https://cran.r-project.org/bin/windows/base/

ff For Mac OS X: https://cran.r-project.org/bin/macosx/
ff https://www.rstudio.com/products/rstudio/download/

More software will be needed for a few specific recipes, which will be highlighted in the
Getting Ready section of the respective recipe.

Just a closing note: all the software employed in this book is available for free for personal
use, and the greatest advantage of them is that they are open source and powered by the
R community.

Who this book is for

This book was developed and written keeping in mind an average R and RStudio user who
would like to make the move from good to great in the field of their programming skills on
the language.

If you think you are quite good at R and RStudio but you are still missing something in order
to be great, this book is exactly what you need to read.

vi
 Preface

Sections
In this book, you will find several headings that appear frequently (Getting ready, How to do it,
How it works, There's more, and See also).

To give clear instructions on how to complete a recipe, we use these sections as follows:

Getting ready
This section tells you what to expect in the recipe, and describes how to set up any software or
any preliminary settings required for the recipe.

How to do it…
This section contains the steps required to follow the recipe.

How it works…
This section usually consists of a detailed explanation of what happened in the previous
section.

There's more…
This section consists of additional information about the recipe in order to make the reader
more knowledgeable about the recipe.

See also
This section provides helpful links to other useful information for the recipe.

Conventions
In this book, you will find a number of text styles that distinguish between different kinds of
information. Here are some examples of these styles and an explanation of their meaning.

Code words in text, database table names, folder names, filenames, file extensions,
pathnames, dummy URLs, user input, and Twitter handles are shown as follows: "The plot()
function is one of most powerful functions in base R."

vii
Preface

A block of code is set as follows:

> str(lesmiserables)
'data.frame': 254 obs. of 2 variables:
$ V1: Factor w/ 73 levels "Anzelma","Babet",..: 61 49 55 55 21 33 12
23 20 62 ...
$ V2: Factor w/ 49 levels "Babet","Bahorel",..: 42 42 42 36 42 42 42
42 42 42 ...

Any command-line input or output is written as follows:

install.packages("linkcomm")
library(linkcomm)

New terms and important words are shown in bold. Words that you see on the screen, for
example, in menus or dialog boxes, appear in the text like this: "In order to embed your Sankey
diagram, you can leverage the RStudio Save as Web Page control from the Export menu."

Warnings or important notes appear in a box like this.

Tips and tricks appear like this.

Reader feedback
Feedback from our readers is always welcome. Let us know what you think about this
book—what you liked or disliked. Reader feedback is important for us as it helps us develop
titles that you will really get the most out of.

To send us general feedback, simply e-mail [email protected], and mention the

book's title in the subject of your message.

If there is a topic that you have expertise in and you are interested in either writing or
contributing to a book, see our author guide at www.packtpub.com/authors.

viii
 Preface

Customer support
Now that you are the proud owner of a Packt book, we have a number of things to help you to
get the most from your purchase.

Downloading the example code

You can download the example code files for this book from your account at http://
www.packtpub.com. If you purchased this book elsewhere, you can visit http://www.
packtpub.com/support and register to have the files e-mailed directly to you.

You can download the code files by following these steps:

1. Log in or register to our website using your e-mail address and password.
2. Hover the mouse pointer on the SUPPORT tab at the top.
3. Click on Code Downloads & Errata.
4. Enter the name of the book in the Search box.
5. Select the book for which you're looking to download the code files.
6. Choose from the drop-down menu where you purchased this book from.
7. Click on Code Download.

You can also download the code files by clicking on the Code Files button on the book's
webpage at the Packt Publishing website. This page can be accessed by entering the book's
name in the Search box. Please note that you need to be logged in to your Packt account.

Once the file is downloaded, please make sure that you unzip or extract the folder using the
latest version of:

ff WinRAR / 7-Zip for Windows

ff Zipeg / iZip / UnRarX for Mac
ff 7-Zip / PeaZip for Linux

Downloading the color images of this book

We also provide you with a PDF file that has color images of the screenshots/diagrams used
in this book. The color images will help you better understand the changes in the output. You
can download this file from https://www.packtpub.com/sites/default/files/
downloads/RStudioforRStatisticalComputingCookbook_ColorImages.pdf.

ix
Preface

Errata
Although we have taken every care to ensure the accuracy of our content, mistakes do
happen. If you find a mistake in one of our books—maybe a mistake in the text or the
code—we would be grateful if you could report this to us. By doing so, you can save other
readers from frustration and help us improve subsequent versions of this book. If you find
any errata, please report them by visiting http://www.packtpub.com/submit-errata,
selecting your book, clicking on the Errata Submission Form link, and entering the details of
your errata. Once your errata are verified, your submission will be accepted and the errata will
be uploaded to our website or added to any list of existing errata under the Errata section of
that title.

To view the previously submitted errata, go to https://www.packtpub.com/books/

content/support and enter the name of the book in the search field. The required
information will appear under the Errata section.

Piracy
Piracy of copyrighted material on the Internet is an ongoing problem across all media.
At Packt, we take the protection of our copyright and licenses very seriously. If you come
across any illegal copies of our works in any form on the Internet, please provide us with
the location address or website name immediately so that we can pursue a remedy.

Please contact us at [email protected] with a link to the suspected pirated material.

We appreciate your help in protecting our authors and our ability to bring you valuable content.

Questions
If you have a problem with any aspect of this book, you can contact us at
[email protected], and we will do our best to address the problem.

x
 Acquiring Data for
1
Your Project
In this chapter, we will cover the following recipes:

ff Acquiring data from the Web—web scraping tasks

ff Accessing an API with R
ff Getting data from Twitter with the twitteR package
ff Getting data from Facebook with the Rfacebook package
ff Getting data from Google Analytics
ff Loading your data into R with rio packages
ff Converting file formats using the rio package

Introduction
The American statistician Edward Deming once said:

"Without data you are just another man with an opinion."

I think this great quote is enough to highlight the importance of the data acquisition phase
of every data analysis project. This phase is exactly where we are going to start from. This
chapter will give you tools for scraping the Web, accessing data via web APIs, and importing
nearly every kind of file you will probably have to work with quickly, thanks to the magic
package rio.

All the recipes in this book are based on the great and popular packages developed and
maintained by the members of the R community.

1
Acquiring Data for Your Project

After reading this section, you will be able to get all your data into R to start your data analysis
project, no matter where it comes from.

Before starting the data acquisition process, you should gain a clear understanding of your
data needs. In other words, what data do you need in order to get solutions to your problems?

A rule of thumb to solve this problem is to look at the process that you are investigating—from
input to output—and outline all the data that will go in and out during its development.

In this data, you will surely have that chunk of data that is needed to solve your problem.

In particular, for each type of data you are going to acquire, you should define the following:

ff The source: This is where data is stored

ff The required authorizations: This refers to any form of authorization/authentication
that is needed in order to get the data you need
ff The data format: This is the format in which data is made available
ff The data license: This is to check whether there is any license covering data
utilization/distribution or whether there is any need for ethics/privacy considerations

After covering these points for each set of data, you will have a clear vision of future data
acquisition activities. This will let you plan ahead the activities needed to clearly define
resources, steps, and expected results.

Acquiring data from the Web – web scraping

tasks
Given the advances in the Internet of Things (IoT) and the progress of cloud computing, we
can quietly affirm that in future, a huge part of our data will be available through the Internet,
which on the other hand doesn't mean it will be public.

It is, therefore, crucial to know how to take that data from the Web and load it into your
analytical environment.

You can find data on the Web either in the form of data statically stored on websites (that is,
tables on Wikipedia or similar websites) or in the form of data stored on the cloud, which is
accessible via APIs.

For API recipes, we will go through all the steps you need to get data statically exposed on
websites in the form of tabular and nontabular data.

This specific example will show you how to get data from a specific Wikipedia page, the
one about the R programming language: https://en.wikipedia.org/wiki/R_
(programming_language).

2
 Chapter 1

Getting ready
Data statically exposed on web pages is actually pieces of web page code. Getting them from
the Web to our R environment requires us to read that code and find where exactly the data is.

Dealing with complex web pages can become a really challenging task, but luckily,
SelectorGadget was developed to help you with this job. SelectorGadget is a bookmarklet,
developed by Andrew Cantino and Kyle Maxwell, that lets you easily figure out the CSS selector
of your data on the web page you are looking at. Basically, the CSS selector can be seen as
the address of your data on the web page, and you will need it within the R code that you are
going to write to scrape your data from the Web (refer to the next paragraph).

The CSS selector is the token that is used within the CSS code to identify
elements of the HTML code based on their name.
CSS selectors are used within the CSS code to identify which elements are to
be styled using a given piece of CSS code. For instance, the following script
will align all elements (CSS selector *) with 0 margin and 0 padding:
* {
margin: 0;
padding: 0;
}

SelectorGadget is currently employable only via the Chrome browser, so you will need to install
the browser before carrying on with this recipe. You can download and install the last version
of Chrome from https://www.google.com/chrome/.

SelectorGadget is available as a Chrome extension; navigate to the following URL while

already on the page showing the data you need:
:javascript:(function(){
var%20s=document.createElement('div');
s.innerHTML='Loading…'
;s.style.color='black';
s.style.padding='20px';
s.style.position='fixed';
s.style.zIndex='9999';
s.style.fontSize='3.0em';
s.style.border='2px%20solid%20black';
s.style.right='40px';
s.style.top='40px';
s.setAttribute('class','selector_gadget_loading');
s.style.background='white';
document.body.appendChild(s);

3
Acquiring Data for Your Project
s=document.createElement('script');
s.setAttribute('type','text/javascript');
s.setAttribute('src','https://dv0akt2986vzh.cloudfront.net/
unstable/lib/selectorgadget.js');document.body.appendChild(s);
})();

This long URL shows that the CSS selector is provided as JavaScript; you can make this out
from the :javascript: token at the very beginning.

We can further analyze the URL by decomposing it into three main parts, which are as follows:

ff Creation on the page of a new element of the div class with the document.
createElement('div') statement
ff Aesthetic attributes setting, composed by all the s.style… tokens
ff The .js file content retrieving at https://dv0akt2986vzh.cloudfront.net/
unstable/lib/selectorgadget.js

The .js file is where the CSS selector's core functionalities are actually defined and the place
where they are taken to make them available to users.

That being said, I'm not suggesting that you try to use this link to employ SelectorGadget
for your web scraping purposes, but I would rather suggest that you look for the Chrome
extension or at the official SelectorGadget page, http://selectorgadget.com. Once
you find the link on the official page, save it as a bookmark so that it is easily available
when you need it.

The other tool we are going to use in this recipe is the rvest package, which offers great web
scraping functionalities within the R environment.

To make it available, you first have to install and load it in the global environment that runs
the following:
install.packages("rvest")
library(rvest)

4
 Chapter 1

How to do it...
1. Run SelectorGadget. To do so, after navigating to the web page you are interested
in, activate SelectorGadget by running the Chrome extension or clicking on the
bookmark that we previously saved.
In both cases, after activating the gadget, a Loading… message will appear, and
then, you will find a bar on the bottom-right corner of your web browser, as shown in
the following screenshot:

You are now ready to select the data you are interested in.

5
Acquiring Data for Your Project

2. Select the data you are interested in. After clicking on the data you are going to
scrape, you will note that beside the data you've selected, there are some other
parts on the page that will turn yellow:

This is because SelectorGadget is trying to guess what you are looking at by

highlighting all the elements included in the CSS selector that it considers to
be most useful for you.
If it is guessing wrong, you just have to click on the wrongly highlighted parts and
those will turn red:

6
 Chapter 1

When you are done with this fine-tuning process, SelectorGadget will have correctly
identified a proper selector, and you can move on to the next step.

3. Find your data location on the page. To do this, all you have to do is copy the CSS
selector that you will find in the bar at the bottom-right corner:

This piece of text will be all you need in order to scrape the web page from R.

4. The next step is to read data from the Web with the rvest package. The rvest
package by Hadley Wickham is one of the most comprehensive packages for
web scraping activities in R. Take a look at the There's more... section for further
information on package objectives and functionalities.
For now, it is enough to know that the rvest package lets you download HTML code
and read the data stored within the code easily.
Now, we need to import the HTML code from the web page. First of all, we need to
define an object storing all the html code of the web page you are looking at:
page_source <- read_html('https://en.wikipedia.org/wiki/R_
(programming_language)

This code leverages read_html function(), which retrieves the source code that
resides at the written URL directly from the Web.

7
Acquiring Data for Your Project

5. Next, we will select the defined blocks. Once you have got your HTML code, it is time
to extract the part of the code you are interested in. This is done using the
html_nodes() function, which is passed as an argument in the CSS selector and
retrieved using SelectorGadget. This will result in a line of code similar to the following:
version_block <- html_nodes(page_source,".wikitable th ,
.wikitable td")

As you can imagine, this code extracts all the content of the selected nodes, including
HTML tags.

The HTML language

HyperText Markup Language (HTML) is a markup language that is used to
define the format of web pages.
The basic idea behind HTML is to structure the web page into a format with a
head and body, each of which contains a variable number of tags, which can
be considered as subcomponents of the structure.
The head is used to store information and components that will not be
seen by the user but will affect the web page's behavior, for instance, in a
Google Analytics script used for tracking page visits, the body contains all the
contents which will be showed to the reader.
Since the HTML code is composed of a nested structure, it is common to
compare this structure to a tree, and here, different components are also
referred to as nodes.

Printing out the version_block object, you will obtain a result similar to
the following:
print(version_block)

{xml_nodeset (45)}
[1] <th>Release</th>
[2] <th>Date</th>
[3] <th>Description</th>
[4] <th>0.16</th>
[5] <td/>
[6] <td>This is the last <a href="/wiki/Alpha_test" title="Alpha
test" class="mw-redirect">alp ...
[7] <th>0.49</th>
[8] <td style="white-space:nowrap;">1997-04-23</td>
[9] <td>This is the oldest available <a href="/wiki/Source_code"
title="Source code">source</a ...
[10] <th>0.60</th>
[11] <td>1997-12-05</td>
[12] <td>R becomes an official part of the <a href="/wiki/GNU_
Project" title="GNU Project">GNU ...

8
 Chapter 1
[13] <th>1.0</th>
[14] <td>2000-02-29</td>
[15] <td>Considered by its developers stable enough for production
use.<sup id="cite_ref-35" cl ...
[16] <th>1.4</th>
[17] <td>2001-12-19</td>
[18] <td>S4 methods are introduced and the first version for <a
href="/wiki/Mac_OS_X" title="Ma ...
[19] <th>2.0</th>
[20] <td>2004-10-04</td>

This result is not exactly what you are looking for if you are going to work with this
data. However, you don't have to worry about that since we are going to give your text
a better shape in the very next step.

6. In order to obtain a readable and actionable format, we need one more step:
extracting text from HTML tags.
This can be done using the html_text() function, which will result in a list
containing all the text present within the HTML tags:
content <- html_text(version_block)

The final result will be a perfectly workable chunk of text containing the data needed
for our analysis:
[1] "Release"

[2] "Date"

[3] "Description"

[4] "0.16"

[5] ""

[6] "This is the last alpha version developed primarily by

Ihaka and Gentleman. Much of the basic functionality from the
\"White Book\" (see S history) was implemented. The mailing lists
commenced on April 1, 1997."
[7] "0.49"

[8] "1997-04-23"

[9] "This is the oldest available source release, and compiles

on a limited number of Unix-like platforms. CRAN is started on
this date, with 3 mirrors that initially hosted 12 packages. Alpha
versions of R for Microsoft Windows and Mac OS are made available
shortly after this version."

9
Acquiring Data for Your Project
[10] "0.60"

[11] "1997-12-05"

[12] "R becomes an official part of the GNU

Project. The code is hosted and maintained on CVS."

[13] "1.0"

[14] "2000-02-29"

[15] "Considered by its developers

stable enough for production use.[35]"

[16] "1.4"

[17] "2001-12-19"

[18] "S4 methods are introduced and the first

version for Mac OS X is made available soon after."

[19] "2.0"

[20] "2004-10-04"

[21] "Introduced lazy loading, which enables fast

loading of data with minimal expense of system memory."

[22] "2.1"

[23] "2005-04-18"

[24] "Support for UTF-8 encoding, and the beginnings of

internationalization and localization for different languages."

[25] "2.11"

[26] "2010-04-22"

[27] "Support for Windows 64 bit systems."

[28] "2.13"

[29] "2011-04-14"

[30] "Adding a new compiler function that allows

speeding up functions by converting them to byte-code."

10
 Chapter 1
[31] "2.14"

[32] "2011-10-31"

[33] "Added mandatory namespaces for

packages. Added a new parallel package."

[34] "2.15"

[35] "2012-03-30"

[36] "New load balancing functions. Improved

serialization speed for long vectors."

[37] "3.0"

[38] "2013-04-03"

[39] "Support for numeric index values

231 and larger on 64 bit systems."

[40] "3.1"

[41] "2014-04-10"

[42] ""

[43] "3.2"

[44] "2015-04-16"

[45] ""

There's more...
The following are a few useful resources that will help you get the most out of this recipe:

ff A useful list of HTML tags, to show you how HTML files are structured and how to
identify code that you need to get from these files, is provided at http://www.
w3schools.com/tags/tag_code.asp
ff The blog post from the RStudio guys introducing the rvest package and highlighting
some package functionalities can be found at http://blog.rstudio.
org/2014/11/24/rvest-easy-web-scraping-with-r/

11
Acquiring Data for Your Project

Accessing an API with R

As we mentioned before, an always increasing proportion of our data resides on the Web and
is made available through web APIs.

APIs in computer programming are intended to be APIs, groups of procedures,

protocols, and software used for software application building. APIs expose
software in terms of input, output, and processes.
Web APIs are developed as an interface between web applications and third
parties.
The typical structure of a web API is composed of a set of HTTP request
messages that have answers with a predefined structure, usually in the XML
or JSON format.

A typical use case for API data contains data regarding web and mobile applications, for
instance, Google Analytics data or data regarding social networking activities.

The successful web application If This ThenThat (IFTTT), for instance, lets you link together
different applications, making them share data with each other and building powerful and
customizable workflows:

This useful job is done by leveraging the application's API (if you don't know IFTTT, just
navigate to https://ifttt.com, and I will see you there).

12
 Chapter 1

Using R, it is possible to authenticate and get data from every API that adheres to the OAuth
1 and OAuth 2 standards, which are nowadays the most popular standards (even though
opinions about these protocols are changing; refer to this popular post by the OAuth creator
Blain Cook at http://hueniverse.com/2012/07/26/oauth-2-0-and-the-road-to-
hell/). Moreover, specific packages have been developed for a lot of APIs.

This recipe shows how to access custom APIs and leverage packages developed for
specific APIs.

In the There's more... section, suggestions are given on how to develop custom functions
for frequently used APIs.

Getting ready
The rvest package, once again a product of our benefactor Hadley Whickham, provides
a complete set of functionalities for sending and receiving data through the HTTP protocol
on the Web. Take a look at the quick-start guide hosted on GitHub to get a feeling of rvest
functionalities (https://github.com/hadley/rvest).

Among those functionalities, functions for dealing with APIs are provided as well.

Both OAuth 1.0 and OAuth 2.0 interfaces are implemented, making this package really useful
when working with APIs.

Let's look at how to get data from the GitHub API. By changing small sections, I will point out
how you can apply it to whatever API you are interested in.

Let's now actually install the rvest package:

install.packages("rvest")
library(rvest)

How to do it…
1. The first step to connect with the API is to define the API endpoint. Specifications for
the endpoint are usually given within the API documentation. For instance, GitHub
gives this kind of information at http://developer.github.com/v3/oauth/.
In order to set the endpoint information, we are going to use the oauth_endpoint()
function, which requires us to set the following arguments:
‰‰ request: This is the URL that is required for the initial unauthenticated
token. This is deprecated for OAuth 2.0, so you can leave it NULL in this
case, since the GitHub API is based on this protocol.
‰‰ authorize: This is the URL where it is possible to gain authorization for the
given client.

13
Acquiring Data for Your Project

‰‰ access: This is the URL where the exchange for an authenticated token
is made.
‰‰ base_url: This is the API URL on which other URLs (that is, the URLs
containing requests for data) will be built upon.

In the GitHub example, this will translate to the following line of code:
github_api <- oauth_endpoint(request = NULL,
authorize =
"https://github.com/login/oauth/authorize",
access = "https://github.com/login/oauth/access_token",
base_url =
"https://github.com/login/oauth")

2. Create an application to get a key and secret token. Moving on with our GitHub
example, in order to create an application, you will have to navigate to https://
github.com/settings/applications/new (assuming that you are already
authenticated on GitHub).
Be aware that no particular URL is needed as the homepage URL, but a specific URL
is required as the authorization callback URL.
This is the URL that the API will redirect to after the method invocation is done.
As you would expect, since we want to establish a connection from GitHub to our
local PC, you will have to redirect the API to your machine, setting the Authorization
callback URL to http://localhost:1410.
After creating your application, you can get back to your R session to establish a
connection with it and get your data.

3. After getting back to your R session, you now have to set your OAuth credentials
through the oaut_app() and oauth2.0_token() functions and establish a
connection with the API, as shown in the following code snippet:
app <- oauth_app("your_app_name",
key = "your_app_key",
secret = "your_app_secret")
API_token <- oauth2.0_token(github_api,app)

4. This is where you actually use the API to get data from your web-based software.
Continuing on with our GitHub-based example, let's request some information about
API rate limits:
request <- GET("https://api.github.com/rate_limit", config(token =
API_token))

14
 Chapter 1

How it works...
Be aware that this step will be required both for OAuth 1.0 and OAuth 2.0 APIs, as the
difference between them is only the absence of a request URL, as we noted earlier.

Endpoints for popular APIs

The httr package comes with a set of endpoints that are already
implemented for popular APIs, and specifically for the following websites:
ff LinkedIn
ff Twitter
ff Vimeo
ff Google
ff Facebook
ff GitHub
For these APIs, you can substitute the call to oauth_endpoint() with a
call to the oauth_endpoints() function, for instance:
oauth_endpoints("github")
The core feature of the OAuth protocol is to secure authentication. This is
then provided on the client side through a key and secret token, which are
to be kept private.
The typical way to get a key and a secret token to access an API involves
creating an app within the service providing the API.

The callback URL

Within the web API domain, a callback URL is the URL that is called by the API
after the answer is given to the request. A typical example of a callback URL is
the URL of the page navigated to after completing an online purchase.
In this example, when we finish at the checkout on the online store, an API
call is made to the payment circuit provider.
After completing the payment operation, the API will navigate again to the
online store at the callback URL, usually to a thank you page.

There's more...
You can also write custom functions to handle APIs. When frequently dealing with a particular
API, it can be useful to define a set of custom functions in order to make it easier to interact
with.

15
Acquiring Data for Your Project

Basically, the interaction with an API can be summarized with the following three categories:

ff Authentication
ff Getting content from the API
ff Posting content to the API

Authentication can be handled by leveraging the HTTR package's authenticate() function

and writing a function as follows:
api_auth function (path = "api_path", password){
authenticate(user = path, password)
}

You can get the content from the API through the get function of the httr package:
api_get <- function(path = "api_path",password){
auth <- api_auth(path, password )
request <- GET("https://api.com", path = path, auth)

Posting content will be done in a similar way through the POST function:
api_post <- function(Path, post_body, path = "api_path",password){
auth <- api_auth(pat) stopifnot(is.list(body))
body_json <- jsonlite::toJSON(body)
request <- POST("https://api.application.com", path = path, body =
body_json, auth, post, ...)
}

Getting data from Twitter with the

twitteR package
Twitter is an unbeatable source of data for nearly every kind of data-driven problem.

If my words are not enough to convince you, and I think they shouldn't be, you can always
perform a quick search on Google, for instance, text analytics with Twitter, and read the over
30 million results to be sure.

This should not surprise you, given Google's huge and word-spreaded base of users together
with the relative structure and richness of metadata of content on the platform, which makes
this social network a place to go when talking about data analysis projects, especially those
involving sentiment analysis and customer segmentations.

R comes with a really well-developed package named twitteR, developed by Jeff Gentry,
which offers a function for nearly every functionality made available by Twitter through the API.
The following recipe covers the typical use of the package: getting tweets related to a topic.

16
 Chapter 1

Getting ready
First of all, we have to install our great twitteR package by running the following code:
install.packages("twitteR")
library(twitter)

How to do it…
1. As seen with the general procedure, in order to access the Twitter API, you will need
to create a new application. This link (assuming you are already logged in to Twitter)
will do the job: https://apps.twitter.com/app/new.
Feel free to give whatever name, description, and website to your app that you want.
The callback URL can be also left blank.
After creating the app, you will have access to an API key and an API secret, namely
Consumer Key and Consumer Secret, in the Keys and Access Tokens tab in your
app settings.
Below the section containing these tokens, you will find a section called Your Access
Token. These tokens are required in order to let the app perform actions on your
account's behalf. For instance, you may be willing to send direct messages to all new
followers and could therefore write an app to do that automatically.
Keep a note of these tokens as well, since you will need them to set up your
connection within R.

2. Then, we will get access to the API from R. In order to authenticate your app and use
it to retrieve data from Twitter, you will just need to run a line of code, specifically, the
setup_twitter_oauth() function, by passing the following arguments:
‰‰ consumer_key
‰‰ consumer_token
‰‰ access_token
‰‰ access_secret

You can get these tokens from your app settings:

setup_twitter_oauth(consumer_key = "consumer_key",
consumer_secret = "consumer_secret",
access_token = "access_token",
access_secret = "access_secret")

17
Acquiring Data for Your Project

3. Now, we will query Twitter and store the resulting data. We are finally ready for the
core part: getting data from Twitter. Since we are looking for tweets pertaining to a
specific topic, we are going to use the searchTwitter() function. This function
allows you to specify a good number of parameters besides the search string. You
can define the following:
‰‰ n : This is the number of tweets to be downloaded.
‰‰ lang: This is the language specified with the ISO 639-1 code. You can find a
partial list of this code at https://en.wikipedia.org/wiki/List_of_
ISO_639-1_codes.
‰‰ since – until: These are time parameters that define a range of time,
where dates are expressed as YYYY-MM-DD, for instance, 2012-05-12.
‰‰ locale: This specifies the geocode, expressed as latitude, longitude
and radius, either in miles or kilometers, for example, 38.481157,
-130.500342,1 mi.
‰‰ sinceID – maxID: This is the account ID range.
‰‰ resultType: This is used to filter results based on popularity. Possible
values are 'mixed', 'recent', and 'popular'.
‰‰ retryOnRateLimit: This is the number that defines how many times the
query will be retried if the API rate limit is reached.

Supposing that we are interested in tweets regarding data science with R; we run the
following function:
tweet_list <- searchTwitter('data science with R', n = 450)

Performing a character-wise search with twitteR

Searching Twitter for a specific sequence of characters is possible by
submitting a query surrounded by double quotes, for instance, "data
science with R". Consequently, if you are looking to retrieve tweets
in R corresponding to a specific sequence of characters, you will have to
submit and run a line of code similar to the following:
tweet_list <- searchTwitter('data science with R',
n = 450)

tweet_list will be a list of the first 450 tweets resulting from the given query.
Be aware that since n is the maximum number of tweets retrievable, you may retrieve
a smaller number of tweets, if for the given query the number or result is smaller
than n.

18
 Chapter 1

Each element of the list will show the following attributes:

‰‰ text
‰‰ favorited
‰‰ favoriteCount
‰‰ replyToSN
‰‰ created
‰‰ truncated
‰‰ replyToSID
‰‰ id
‰‰ replyToUID
‰‰ statusSource
‰‰ screenName
‰‰ retweetCount
‰‰ isRetweet
‰‰ retweeted
‰‰ longitude
‰‰ latitude

In order to let you work on this data more easily, a specific function is provided to
transform this list in a more convenient data.frame, namely, the twiLstToDF()
function.
After this, we can run the following line of code:
tweet_df <- twListToDF(tweet_list)

This will result in a tweet_df object that has the following structure:
> str(tweet_df)
'data.frame': 20 obs. of 16 variables:
$ text : chr "95% off Applied Data Science with R -
$ favorited : logi FALSE FALSE FALSE FALSE FALSE FALSE ...
$ favoriteCount: num 0 2 0 2 0 0 0 0 0 1 ...
$ replyToSN : logi NA NA NA NA NA NA ...
$ created : POSIXct, format: "2015-10-16 09:03:32" "2015-10-
15 17:40:33" "2015-10-15 11:33:37" "2015-10-15 05:17:59" ...
$ truncated : logi FALSE FALSE FALSE FALSE FALSE FALSE ...
$ replyToSID : logi NA NA NA NA NA NA ...
$ id : chr "654945762384740352" "654713487097135104"
"654621142179819520" "654526612688375808" ...
$ replyToUID : logi NA NA NA NA NA NA ...

19
Acquiring Data for Your Project
$ statusSource : chr "<a href=\"http://learnviral.com/\"
rel=\"nofollow\">Learn Viral</a>" "<a href=\"https://about.
twitter.com/products/tweetdeck\" rel=\"nofollow\">TweetDeck</
a>" "<a href=\"http://not.yet/\" rel=\"nofollow\">final one kk</
a>" "<a href=\"http://twitter.com\" rel=\"nofollow\">Twitter Web
Client</a>" ...
$ screenName : chr "Learn_Viral" "WinVectorLLC" "retweetjava"
"verystrongjoe" ...
$ retweetCount : num 0 0 1 1 0 0 0 2 2 2 ...
$ isRetweet : logi FALSE FALSE TRUE FALSE FALSE FALSE ...
$ retweeted : logi FALSE FALSE FALSE FALSE FALSE FALSE ...
$ longitude : logi NA NA NA NA NA NA ...
$ latitude : logi NA NA NA NA NA NA ...

After sending you to the data visualization section for advanced techniques, we will
now quickly visualize the retweet distribution of our tweets, leveraging the base R
hist() function:
hist(tweet_df$retweetCount)

This code will result in a histogram that has the x axis as the number of retweets and
the y axis as the frequency of those numbers:

There's more...
As stated in the official Twitter documentation, particularly at https://dev.twitter.com/
rest/public/rate-limits, there is a limit to the number of tweets you can retrieve within
a certain period of time, and this limit is set to 450 every 15 minutes.

20
 Chapter 1

However, what if you are engaged in a really sensible job and you want to base your work on a
significant number of tweets? Should you set the n argument of searchTwitter() to 450
and wait for 15—everlasting—minutes? Not quite, the twitteR package provides a convenient
way to overcome this limit through the register_db_backend(), register_sqlite_
backend(), and register_mysql_bakend() functions. These functions allow you to
create a connection with the named type of databases, passing the database name, path,
username, and password as arguments, as you can see in the following example:
register_mysql_backend("db_name", "host","user","password")

You can now leverage the search_twitter_and_store function, which stores the
search results in the connected database. The main feature of this function is the
retryOnRateLimit argument, which lets you specify the number of tries to be performed by
the code once the API limit is reached. Setting this limit to a convenient level will likely let you
pass the 15-minutes interval:
tweets_db = search_twitter_and_store("data science R",
retryOnRateLimit = 20)

Retrieving stored data will now just require you to run the following code:
from_db = load_tweets_db()

Getting data from Facebook with the

Rfacebook package
The Rfacebook package, developed and maintained by Pablo Barberá, lets you easily
establish and take advantage of Facebook's API thanks to a series of functions.

As we did for the twitteR package, we are going to establish a connection with the API and
retrieve posts pertaining to a given keyword.

Getting ready
This recipe will mainly be based on functions from the Rfacebok package. Therefore, we need
to install and load this package in our environment:
install.packages("Rfacebook")
library(Rfacebook)

21
Acquiring Data for Your Project

How to do it...
1. In order to leverage an API's functionalities, we first have to create an application
in our Facebook profile. Navigating to the following URL will let you create an app
(assuming you are already logged in to Facebook): https://developers.
facebook.com.
After skipping the quick start (the button on the upper-right corner), you can see the
settings of your app and take note of app_id and app_secret, which you will
need in order to establish a connection with the app.

2. After installing and loading the Rfacebook package, you will easily be able to
establish a connection by running the fbOAuth() function as follows:
fb_connection <- fbOauth(app_id = "your_app_id",
app_secret = "your_app_secret")
fb_connection

Running the last line of code will result in a console prompt, as shown in the following
lines of code:
copy and paste into site URL on Facebook App Settings: http://
localhost:1410/ When done press any key to continue

Following this prompt, you will have to copy the URL and go to your Facebook
app settings.
Once there, you will have to select the Settings tab and create a new platform
through the + Add Platform control. In the form, which will prompt you after clicking
this control, you should find a field named Site Url. In this field, you will have to paste
the copied URL.
Close the process by clicking on the Save Changes button.
At this point, a browser window will open up and ask you to allow access permission
from the app to your profile. After allowing this permission, the R console will print out
the following code snippet:
Authentication complete
Authentication successful.

3. To test our API connection, we are going to search Facebook for posts related to data
science with R and save the results within data.frame for further analysis.
Among other useful functions, Rfacebook provides the searchPages() function,
which as you would expect, allows you to search the social network for pages
mentioning a given string.

22
 Chapter 1

Different from the searchTwitter function, this function will not let you specify a
lot of arguments:
‰‰ string: This is the query string
‰‰ token: This is the valid OAuth token created with the fbOAuth() function
‰‰ n: This is the maximum number of posts to be retrieved

The Unix timestamp

The Unix timestamp is a time-tracking system originally developed for
the Unix OS. Technically, the Unix timestamp x expresses the number
of seconds elapsed since the Unix Epoch (January 1, 1970 UTC) and
the timestamp.

To search for data science with R, you will have to run the following line of code:
pages ← searchPages('data science with R',fb_connection)

This will result in data.frame storing all the pages retrieved along with the data
concerning them.
As seen for the twitteR package, we can take a quick look at the like distribution,
leveraging the base R hist() function:
hist(pages$likes)

This will result in a plot similar to the following:

Refer to the data visualization section for further recipes on data visualization.

23
Exploring the Variety of Random
Documents with Different Content
 Our Evil Genius 295
Ships of Fire Again 295
Commodore Seymour’s Visit 296
Nouka and Queen ’Toria 297
The Dog to his Vomit Again 298

CHAPTER X.
FAREWELL SCENES.
The War Fever 303
Forced to the War Council 305
A Truce Among the Chiefs 306
Chiefs and People 308
The Kiss of Judas 309
The Death of Ian 309
The Quivering Knife 310
A War of Revenge 312
In the Thick of the Battle 313
Tender Mercies of the Wicked 315
Escape for Life 316
The Loss of All 317
Under the Tomahawk 318
Jehovah is Hearing 318
The Host Turned Back 320
The War Against Manuman 320
Traps Laid 321
House Broken Up 322
War Against Our Friends 322
A Treacherous Murderer 323
On the Chestnut Tree 324
Bargaining for Life 325
Five Hours in a Canoe 328
Kneeling on the Sands 329
Faimungo’s Farewell 330
“Follow! Follow!” 331
A Race for Life 332
Ringed Round with Death 334
Faint yet Pursuing 336
 Out of the Lion’s Jaws 337
Brothers in Distress 339
Intervening Events 341
A Cannibal’s Taste 341
Pillars of Cloud and Fire 342
Passing by on the Other Side 344
Kapuku and the Idol Gods 344
A Devil Chief 344
“In Perils Oft” 345
Through Fire and Water 345
“Sail O! Sail O!” 349
“Let Me Die” 350
In Perils on the Sea 351
Tannese Visitors 352
The Devil Chief Again 353
Speckled and Spotted 354
Their Desired Haven 355
“I am Left Alone” 355
My Earthly All 356
Eternal Hope 356
Australia to the Rescue 357
For My Brethren’s Sake 358
A New Holy League 358
The Uses of Adversity 359
Arm-chair Critics Again 360
Concluding Note 361
Prospectus of Part Second 362

APPENDIX.
A. The Prayer of the Chiefs of Tanna 367
B. Notes on the New Hebrides 371
 LIST OF ILLUSTRATIONS.

Portrait of John G. Paton Frontispiece.

“The Man That Kills Missi Must First Kill Me” To face 156
p.
Natives Stealing Property „ 160
Natives Returning Stolen Property „ 164
“There They Lay Me on Cocoa-nut Leaves on the „ 172
Ground”
“At Daybreak I Found My House Surrounded” „ 188
The Dangerous Landing Through the Surf „ 234
“Suddenly He Drew A Large Butcher-Like Knife” „ 310
Springing Forward He Caught The Club „ 342

Map of the New Hebrides „ 100

 CHAPTER I.
EARLIER DAYS.

Introductory Note.—Kirkmahoe.—Torthorwald Village.—Our

Villagers.—Nithsdale Scenes.—Our Cottage Home.—Our
Forebears.—An Idyll of the Heart.—A Consecrated Father.—
Accepted Vows.—Happy Sabbath Days.—Golden Autumn of
Life.

What I write here is for the glory of God. For more than twenty years
have I been urged to record my story as a missionary of the Cross;
but always till now, in my sixty-fourth year, my heart has shrunk
from the task, as savouring too much of self. Latterly the conviction
has been borne home to me that if there be much in my experience
which the Church of God ought to know, it would be pride on my
part, and not humility, to let it die with me. I lift my pen, therefore,
with that motive supreme in my heart; and, so far as memory and
entries in my note-books and letters of my own and of other friends
serve or help my sincere desire to be truthful and fair, the following
chapters will present a faithful picture of the life through which the
Lord has led me. If it bows any of my readers under as deep and
certain a confidence as mine, that in “God’s hand our breath is, and
His are all our ways,” my task will not be fruitless in the Great Day.

On the 24th May, 1824, I was born in a cottage on the farm of

Braehead, in the parish of Kirkmahoe, near Dumfries, in the south of
Scotland. My father, James Paton, was a stocking manufacturer in a
small way; and he and his young wife, Janet Jardine Rogerson, lived
on terms of warm personal friendship with the “gentleman farmer,”
so they gave me his name, John Gibson; and the curly-haired child of
the cottage was soon able to toddle across to the mansion, and
became a great pet of the lady there. More than once, in my many
journeyings, have I met with one or another, in some way connected
with that family, and heard little incidents not needing to be
repeated here, showing how beautiful and tender and altogether
human was the relationship in those days betwixt the landlord and
the cottars on his estate. On my last visit to Scotland, sixty years
after, I drove to Braehead in company with my youngest brother
James and my cousin David,—the latter born the same week as I, and
the former nearly twenty years my junior; and we found no cottage,
nor trace of a cottage, but amused ourselves by supposing that we
could discover by the rising of the grassy mound, the outline where
the foundations once had been! Of ten thousand homes in Scotland,
once sweet and beautiful, each a little possible Paradise in its own
well-cultivated plot, this is true to-day; and where are the healthy,
happy peasant boys and girls that such homes bred and reared? They
are sweltering and struggling for existence in our towns and cities. I
am told that this must be—that it is all the result of economic laws;
but I confess to a deepening conviction that it need not be, and that
the loss to the nation as a whole is vital, if not irreparable.
While yet a mere child, five years or so of age, my parents took me
to a new home in the ancient village of Torthorwald, about four and a
quarter miles north from Dumfries, on the road to Lockerbie. At that
time, about 1830, Torthorwald was a busy and thriving village, and
comparatively populous, with its cottars and crofters, large farmers
and small farmers, weavers and shoemakers, cloggers and coopers,
blacksmiths and tailors. Fifty-five years later, when I last visited the
scenes of my youth, the village proper was literally extinct, except for
five thatched cottages where the lingering patriarchs were permitted
to die slowly away,—when they too would be swept into the large
farms, and their garden plots ploughed over, like sixty or seventy
others that had been obliterated! Of course the Village Smithy still
survives, but its sparks are few and fading,—the great cultivators
patronizing rather the towns. The Meal Mill still grinds away,—but
nothing like what it did when every villager bought or cultivated his
few acres of corn, and every crofter and farmer in the parish sent all
his grist to the mill. The Grocer’s Shop still recalls the well-known
name of Robert Henderson; but so few are the mouths now to be fed,
that his warm-hearted wife and universal favourite, the very heroine
of our village life, “Jean Grier,” is retiring from it in disgust, and
leaving it to her son-in-law, declaring that “these Tory landlords and
their big farms hae driven our folks a’ awa’, and spoiled the Schule
and the Shop, the Kirk and the Mill.” And verily the School is robbed
of its children, and the Parish Church of its worshippers, when five
families only are reared where twenty once flourished! Political
economy may curse me, if it will; but I heard with grim satisfaction
that this system of large farming, which extinguishes our village
homes, and sends our peasantry to rear their children in lanes and
alleys, in attics and cellars of populous towns, was proving ruinous at
length to the landlords and factors, who had in many cases cruelly
forced it on an unwilling people for mere selfish gain.
The Villagers of my early days—the agricultural servants, or
occasional labourers, the tradesmen, the small farmers—were,
generally speaking, a very industrious and thoroughly independent
race of people. Hard workers they had to be, else they would starve;
yet they were keen debaters on all affairs both in Church and State,
and sometimes in the “smiddy” or the “kiln,” sometimes in a happy
knot on the “village green,” or on the road to the “kirk” or the
“market,” the questions that were tearing the mighty world beyond
were fought over again by secluded peasants with amazing passion
and bright intelligence.
From the Bank Hill, close above our village, and accessible in a
walk of fifteen minutes, a view opens to the eye which, despite
several easily understood prejudices of mine that may discount any
opinion that I offer, still appears to me well worth seeing amongst all
the beauties of Scotland. At your feet lay a thriving village, every
cottage sitting in its own plot of garden, and sending up its blue
cloud of “peat reek,” which never somehow seemed to pollute the
blessed air; and after all has been said or sung, a beautifully situated
village of healthy and happy homes for God’s children is surely the
finest feature in every landscape! There nestled the Manse amongst
its ancient trees, sometimes wisely, sometimes foolishly tenanted,
but still the “man’s house,” the man of God’s house, when such can
be found for it. There, close by, the Parish School, where rich and
poor met together on equal terms, as God’s children; and we learned
that brains and character make the only aristocracy worth
mentioning. Yonder, amid its graves, that date back on crumbling
stone five hundred years, stands the Village Church; and there, on its
little natural hill, at the end of the village, rises the old tower of
Torthorwald, frowning over all the far-sweeping valley of the Nith,
and telling of days of blood and Border foray. It was one of the many
castles of the Kirkpatricks, and its enormous and imperishable walls
seem worthy of him who wrote the legend of his family in the blood
of the Red Comyn, stabbed in the Greyfriars Church of Dumfries,
when he smote an extra blow to that of Bruce, and cried, “I mak’
siccar.” Beyond, betwixt you and the Nith, crawls the slow-creeping
Lochar towards the Solway, through miles and miles of moss and
heather,—the nearest realization that I ever beheld of a “stagnant
stream.” Looking from the Bank Hill on a summer day, Dumfries
with its spires shone so conspicuous that you could have believed it
not more than two miles away; the splendid sweeping vale through
which Nith rolls to Solway, lay all before the naked eye, beautiful
with village spires, mansion houses, and white shining farms; the
Galloway hills, gloomy and far-tumbling, bounded the forward view,
while to the left rose Criffel, cloud-capped and majestic; then the
white sands of Solway, with tides swifter than horsemen; and finally
the eye rested joyfully upon the hills of Cumberland, and noticed
with glee the blue curling smoke from its villages on the southern
Solway shores. Four miles behind you lie the ruins of the Castle of
the Bruce, within the domains of his own Royal Burgh of
Lochmaben; a few miles in front, the still beautiful and amazing
remains of Caerlaverock Castle, famous in many a Border story; all
around you, scattered throughout the dale of Nith, memories or
ruins of other baronial “keeps,” rich in suggestion to the peasant
fancy! Traditions lost nothing in bulk, or in graphic force, as they
were retold for the thousandth time by village patriarchs around the
kindly peat fire, with the younger rustics gaping round. A high spirit
of patriotism, and a certain glorious delight in daring enterprises,
was part of our common heritage.
There, amid this wholesome and breezy village life, our dear
parents found their home for the long period of forty years. There
were born to them eight additional children, making in all a family of
five sons and six daughters. Theirs was the first of the thatched
cottages on the left, past the “miller’s house,” going up the “village
gate,” with a small garden in front of it, and a large garden across the
road; and it is one of the few still lingering to show to a new
generation what the homes of their fathers were. The architect who
planned it had no ideas of art, but a fine eye for durability! It consists
at present of three, but originally of four, pairs of “oak couples”
(Scotticé kipples), planted like solid trees in the ground at equal
intervals, and gently sloped inwards till they meet or are “coupled” at
the ridge, this coupling being managed not by rusty iron, but by great
solid pins of oak. A roof of oaken wattles was laid across these, till
within eleven or twelve feet of the ground, and from the ground
upwards a stone wall was raised, as perpendicular as was found
practicable, towards these overhanging wattles, this wall being
roughly “pointed” with sand and clay and lime. Now into and upon
the roof was woven and intertwisted a covering of thatch, that defied
all winds and weathers, and that made the cottage marvellously cozy,
—being renewed year by year, and never allowed to remain in
disrepair at any season. But the beauty of the construction was and is
its durability, or rather the permanence of its oaken ribs! There they
stand, after probably not less than four centuries, japanned with
“peat reek” till they are literally shining, so hard that no ordinary nail
can be driven into them, and perfectly capable for service for four
centuries more on the same conditions. The walls are quite modern,
having all been rebuilt in my father’s time, except only the few great
foundation boulders, piled around the oaken couples; and parts of
the roofing also may plead guilty to having found its way thither only
in recent days; but the architect’s one idea survives, baffling time and
change—the ribs and rafters of oak.
Our home consisted of a “but” and a “ben” and a “mid room,” or
chamber, called the “closet.” The one end was my mother’s domain,
and served all the purposes of dining-room and kitchen and parlour,
besides containing two large wooden erections, called by our Scotch
peasantry “box-beds”; not holes in the wall, as in cities, but grand,
big, airy beds, adorned with many-coloured counterpanes, and hung
with natty curtains, showing the skill of the mistress of the house.
The other end was my father’s workshop, filled with five or six
“stocking frames,” whirring with the constant action of five or six
pairs of busy hands and feet, and producing right genuine hosiery for
the merchants at Hawick and Dumfries. The “closet” was a very small
apartment betwixt the other two, having room only for a bed, a little
table, and a chair, with a diminutive window shedding diminutive
light on the scene. This was the Sanctuary of that cottage home.
Thither daily, and oftentimes a day, generally after each meal, we
saw our father retire, and “shut to the door”; and we children got to
understand by a sort of spiritual instinct (for the thing was too sacred
to be talked about) that prayers were being poured out there for us,
as of old by the High Priest within the veil in the Most Holy Place.
We occasionally heard the pathetic echoes of a trembling voice
pleading as if for life, and we learned to slip out and in past that door
on tiptoe, not to disturb the holy colloquy. The outside world might
not know, but we knew, whence came that happy light as of a new-
born smile that always was dawning on my father’s face: it was a
reflection from the Divine Presence, in the consciousness of which he
lived. Never, in temple or cathedral, on mountain or in glen, can I
hope to feel that the Lord God is more near, more visibly walking and
talking with men, than under that humble cottage roof of thatch and
oaken wattles. Though everything else in religion were by some
unthinkable catastrophe to be swept out of memory, or blotted from
my understanding, my soul would wander back to those early scenes,
and shut itself up once again in that Sanctuary Closet, and, hearing
still the echoes of those cries to God, would hurl back all doubt with
the victorious appeal, “He walked with God, why may not I?”
A few notes had better here be given as to our “Forebears,” the
kind of stock from which my father and mother sprang. My father’s
mother, Janet Murray, claimed to be descended from a Galloway
family that fought and suffered for Christ’s Crown and Covenant in
Scotland’s “killing time,” and was herself a woman of a pronouncedly
religious development. Her husband, our grandfather, William
Paton, had passed through a roving and romantic career, before he
settled down to a douce deacon of the weavers of Dumfries, like his
father before him.
Forced by a press-gang to serve on board a British man-of-war, he
was taken prisoner by the French, and thereafter placed under Paul
Jones, the pirate of the seas, and bore to his dying day the mark of a
slash from the captain’s sword across his shoulder for some slight
disrespect or offence. Determining with two others to escape, the
three were hotly pursued by Paul Jones’s men. One, who could swim
but little, was shot, and had to be cut adrift by the other two, who in
the darkness swam into a cave and managed to evade for two nights
and a day the rage of their pursuers. My grandfather, being young
and gentle and yellow-haired, persuaded some kind heart to rig him
out in female attire, and in this costume escaped the attentions of the
press-gang more than once; till, after many hardships, he bargained
with the captain of a coal sloop to stow him away amongst his black
diamonds, and thus, in due time, he found his way home to
Dumfries, where he tackled bravely and wisely the duties of husband,
father, and citizen for the remainder of his days. The smack of the
sea about the stories of his youth gave zest to the talks round their
quiet fireside, and that, again, was seasoned by the warm evangelical
spirit of his Covenanting wife, her lips “dropping grace.”
Of their children, two reproduced the disposition of their father,
and two that of their mother. William took to the soldier’s career,
and died in Spain; May, the only daughter, gave her heart and hand
to John Wood, a jolly and gallant Englishman, who fought at
Waterloo, and lived to see his hundredth birthday. John and James,
the latter being my father, both learned the stocking manufacturing
business of their fathers, and both followed their mother’s piety and
became from their early teens very pronounced and consistent
disciples of the Lord.
On the other side, my mother, Janet Rogerson, had for parents a
father and mother of the Annandale stock. William Rogerson, her
father, was one of many brothers, all men of uncommon strength and
great force of character, quite worthy of the Border rievers of an
earlier day. Indeed, it was in some such way that he secured his wife,
though the dear old lady in after-days was chary about telling the
story. She was a girl of good position, the ward of two unscrupulous
uncles who had charge of her small estate, near Langholm; and while
attending some boarding school she fell devotedly in love with the
tall, fair-haired, gallant young blacksmith, William Rogerson. Her
guardians, doubtless very properly, objected to the “connection”; but
our young Lochinvar, with his six or seven stalwart brothers and
other trusty “lads,” all mounted, and with some ready tool in case of
need, went boldly and claimed his bride, and she, willingly mounting
at his side, was borne off in the light of open day, joyously married,
and took possession of her “but and ben,” as the mistress of the
blacksmith’s abode.
The uncles had it out with him, however, in another way. While he
was enjoying his honeymoon, and careless of mere mundane affairs,
they managed to dispose of all the property of their ward, and make
good their escape with the proceeds to the New World. Having heard
a rumour of some such sale, our young blacksmith on horseback just
reached the scene in time to see the last article—a Family Bible—put
up for auction. This he claimed, or purchased, or seized, in name of
the heiress—but that was all that she ever inherited. It was used
devoutly by her till her dying day, and was adorned with the record
of her own marriage and of the birth of a large and happy family,
whom by-and-by God gave to her.
Janet Jardine bowed her neck to the self-chosen yoke, with the
light of a supreme affection in her heart, and showed in her gentler
ways, her love of books, her fine accomplishments with the needle
and her general air of ladyhood, that her lot had once been cast in
easier, but not necessarily happier, ways. Her blacksmith lover
proved not unworthy of his lady bride, and in her old days found a
quiet and modest home, the fruit of years of toil and hopeful thrift,
their own little property, in which they rested and waited a happy
end. Amongst those who at last wept by her grave stood, amidst
many sons and daughters, her son the Rev. James J. Rogerson,
clergyman of the Church of England, who, for many years thereafter,
and till quite recently, was spared to occupy a distinguished position
at ancient Shrewsbury, and has left behind him there an honoured
and beloved name.
One thing else, beautiful in its pathos, I must record of that dear
old lady. Her son, Walter, had gone forth from her, in prosecution of
his calling, had corresponded with her from various counties in
England, and then had suddenly disappeared; and no sign came to
her, whether he was dead or alive. The mother-heart in her clung to
the hope of his return; every night she prayed for that happy event,
and before closing the door, threw it wide open, and peered into the
darkness with a cry, “Come hame, my boy Walter, your mither
wearies sair;” and every morning, at early break of day, for a period
of more than twenty years, she toddled up from her cottage door, at
Johnsfield, Lockerbie, to a little round hill, called the “Corbie Dykes,”
and, gazing with tear-filled eyes towards the south for the form of her
returning boy, prayed the Lord God to keep him safe and restore him
to her yet again. Always, as I think upon that scene, my heart finds
consolation in reflecting that if not here, then for certain there, such
deathless longing love will be rewarded, and, rushing into long-
delayed embrace, will exclaim, “Was lost and is found.”
From such a home came our mother, Janet Jardine Rogerson, a
bright-hearted, high-spirited, patient-toiling, and altogether heroic
little woman; who, for about forty-three years, made and kept such a
wholesome, independent, God-fearing, and self-reliant life for her
family of five sons and six daughters, as constrains me, when I look
back on it now, in the light of all I have since seen and known of
others far differently situated, almost to worship her memory. She
had gone with her high spirits and breezy disposition to gladden, as
their companion, the quiet abode of some grand or great-grand-
uncle and aunt, familiarly named in all that Dalswinton
neighbourhood, “Old Adam and Eve.” Their house was on the
outskirts of the moor, and life for the young girl there had not
probably too much excitement. But one thing had arrested her
attention. She had noticed that a young stocking maker from the
“Brig End,” James Paton, the son of William and Janet there, was in
the habit of stealing alone into the quiet wood, book in hand, day
after day, at certain hours, as if for private study and meditation. It
was a very excusable curiosity that led the young bright heart of the
girl to watch him devoutly reading and hear him reverently reciting
(though she knew not then, it was Ralph Erskine’s “Gospel Sonnets,”
which he could say by heart sixty years afterwards, as he lay on his
bed of death); and finally that curiosity awed itself into a holy
respect, when she saw him lay aside his broad Scotch bonnet, kneel
down under the sheltering wings of some tree, and pour out all his
soul in daily prayers to God. As yet they had never spoken. What
spirit moved her, let lovers tell—was it all devotion, or was it a touch
of unconscious love kindling in her towards the yellow-haired and
thoughtful youth? Or was there a stroke of mischief, of that teasing,
which so often opens up the door to the most serious step in all our
lives? Anyhow, one day she slipped in quietly, stole away his bonnet,
and hung it on a branch near by, while his trance of devotion made
him oblivious of all around; then, from a safe retreat she watched
and enjoyed his perplexity in seeking for and finding it! A second day
this was repeated; but his manifest disturbance of mind, and his long
pondering with the bonnet in hand, as if almost alarmed, seemed to
touch another chord in her heart—that chord of pity which is so often
the prelude of love, that finer pity that grieves to wound anything
nobler or tenderer than ourselves. Next day, when he came to his
accustomed place of prayer, a little card was pinned against the tree
just where he knelt, and on it these words:—
“She who stole away your bonnet is ashamed of what she did; she
has a great respect for you, and asks you to pray for her, that she may
become as good a Christian as you.”
Staring long at that writing, he forgot Ralph Erskine for one day;
taking down the card, and wondering who the writer could be, he
was abusing himself for his stupidity in not suspecting that some one
had discovered his retreat, and removed his bonnet, instead of
wondering whether angels had been there during his prayer,—when,
suddenly raising his eyes, he saw in front of old Adam’s cottage,
through a lane amongst the trees, the passing of another kind of
angel, swinging a milk-pail in her hand and merrily singing some
snatch of old Scottish song. He knew, in that moment, by a Divine
instinct, as infallible as any voice that ever came to seer of old, that
she was the angel visitor that had stolen in upon his retreat—that
bright-faced, clever-witted niece of old Adam and Eve, to whom he
had never yet spoken, but whose praises he had often heard said and
sung—“Wee Jen.” I am afraid he did pray “for her,” in more senses
than one, that afternoon; at any rate, more than a Scotch bonnet was
very effectually stolen; a good heart and true was there bestowed,
and the trust was never regretted on either side, and never betrayed.
Often and often, in the genial and beautiful hours of the
autumntide of their long life, have I heard my dear father tease “Jen”
about her maidenly intentions in the stealing of that bonnet; and
often with quick mother wit have heard her happy retort, that had his
motives for coming to that retreat been altogether and exclusively
pious, he would probably have found his way to the other side of the
wood, but that men who prowled about the Garden of Eden ran the
risk of meeting some day with a daughter of Eve!
Somewhere in or about his seventeenth year, my father passed
through a crisis of religious experience, and from that day he openly
and very decidedly followed the Lord Jesus. His parents had
belonged to one of the older branches of what now we call the United
Presbyterian Church; but my father, having made an independent
study of the Scotch Worthies, the Cloud of Witnesses, the
Testimonies, and the Confession of Faith, resolved to cast in his lot
with the oldest of all the Scotch Churches, the Reformed
Presbyterian, as most nearly representing the Covenanters and the
attainments of both the first and second Reformations in Scotland.
This choice he deliberately made, and sincerely and intelligently
adhered to; and was able at all times to give strong and clear reasons
from Bible and from history for the principles he upheld. Still his
sympathies and votes always went with the more progressive party in
that ancient Church. He held it to be right that Cameronians, like
other citizens, should exercise the municipal and political franchise,
and he adhered to the “Majority Synod,” which has since been
incorporated with the Free Church of Scotland. While glorying in the
Psalms, he rejoiced to sing other hymns and spiritual songs (thanks
to Ralph Erskine’s “Sonnets,” perhaps, for that!) from his earliest
days, at least everywhere except in the ordinary Public Worship; and
long before he died, though he still held the Psalms to be supreme,
he had learned to hear with glowing delight vast congregations
singing the hymns of modern days, had learned joyfully to join in
these songs of Zion, and was heard often to confess his belief that
God had greatly owned and blessed the ministry of song in the
service of the Gospel.
Besides his independent choice of a Church for himself, there was
one other mark and fruit of his early religious decision, which looks
even fairer through all these years. Family Worship had heretofore
been held only on Sabbath day in his father’s house; but the young
Christian, entering into conference with his sympathising mother,
managed to get the household persuaded that there ought to be daily
morning and evening prayer and reading of the Bible and holy
singing. This the more readily, as he himself agreed to take part
regularly in the same and so relieve the old warrior of what might
have proved for him too arduous spiritual toils. And so began in his
seventeenth year that blessed custom of Family Prayer, morning and
evening, which my father practised probably without one single
omission till he lay on his deathbed, seventy-seven years of age;
when, even to the last day of his life, a portion of Scripture was read,
and his voice was heard softly joining in the Psalm, and his lips
breathed the morning and evening Prayer,—falling in sweet
benediction on the heads of all his children, far away many of them
over all the earth, but all meeting him there at the Throne of Grace.
None of us can remember that any day ever passed unhallowed thus;
no hurry for market, no rush to business, no arrival of friends or
guests, no trouble or sorrow, no joy or excitement, ever prevented at
least our kneeling around the family altar, while the High Priest led
our prayers to God, and offered himself and his children there. And
blessed to others, as well as to ourselves, was the light of such
example! I have heard that, in long after years, the worst woman in
the village of Torthorwald, then leading an immoral life, but since
changed by the grace of God, was known to declare, that the only
thing that kept her from despair and from the hell of the suicide, was
when in the dark winter nights she crept close up underneath my
father’s window, and heard him pleading in family worship that God
would convert “the sinner from the error of wicked ways and polish
him as a jewel for the Redeemer’s crown.” “I felt,” said she, “that I
was a burden on that good man’s heart, and I knew that God would
not disappoint him. That thought kept me out of Hell, and at last led
me to the only Saviour.”
My father had a strong desire to be a minister of the Gospel; but
when he finally saw that God’s will had marked out for him another
lot, he reconciled himself by entering with his own soul into this
solemn vow,—that if God gave him sons, he would consecrate them
unreservedly to the ministry of Christ, if the Lord saw fit to accept
the offering, and open up their way. It may be enough here to say
that he lived to see three of us entering upon and not unblessed in
the Holy Office;—myself, the eldest born; my brother Walter, several
years my junior; and my brother James, the youngest of eleven, the
Benjamin of the flock.
Our place of worship was the Reformed Presbyterian Church at
Dumfries, under the ministry, during most of these days, of Rev.
John McDiarmid—a genuine, solemn, lovable Covenanter, who
cherished towards my father a warm respect, that deepened into
apostolic affection when the yellow hair turned snow-white and both
of them grew patriarchal in their years. The minister, indeed, was
translated to a Glasgow charge; but that rather exalted than
suspended their mutual love. Dumfries was four miles fully from our
Torthorwald home; but the tradition is that during all these forty
years my father was only thrice prevented from attending the
worship of God—once by snow so deep that he was baffled and had
to return; once by ice on the road, so dangerous that he was forced to
crawl back up the Roucan Brae on his hands and knees, after having
descended it so far with many falls; and once by the terrible outbreak
of cholera at Dumfries. All intercourse betwixt the town and the
surrounding villages was publicly prohibited; and the farmers and
villagers, suspecting that no cholera would make my father stay at
home on Sabbath, sent a deputation to my mother on the Saturday
evening, and urged her to restrain his devotions for once! That,
however, was needless; as, where the life of others was at stake, his
very devotion came to their aid. Each of us, from very early days,
considered it no penalty, but a great joy, to go with our father to the
church; the four miles were a treat to our young spirits, the company
by the way was a fresh incitement, and occasionally some of the
wonders of city-life rewarded our eager eyes. A few other pious men
and women of the best evangelical type, went from the same parish
to one or other favourite minister at Dumfries,—the parish church
during all those years being rather miserably served; and when these
God-fearing peasants “forgathered” in the way to or from the House
of God, we youngsters had sometimes rare glimpses of what
Christian talk may be and ought to be. They went to the church, full
of beautiful expectancy of spirit—their souls were on the outlook for
God; they returned from the church, ready and even anxious to
exchange ideas as to what they had heard and received of the things
of life. I have to bear my testimony that religion was presented to us
with a great deal of intellectual freshness, and that it did not repel us
but kindled our spiritual interest. The talks which we heard were,
however, genuine; not the make-believe of religious conversation,
but the sincere outcome of their own personalities. That, perhaps,
makes all the difference betwixt talk that attracts and talk that drives
away.
We had, too, special Bible Readings on the Lord’s Day evening,—
mother and children and visitors reading in turns, with fresh and
interesting question, answer, and exposition, all tending to impress
us with the infinite grace of a God of love and mercy in the great gift
of His dear Son Jesus, our Saviour. The Shorter Catechism was gone
through regularly, each answering the question asked, till the whole
had been explained, and its foundation in Scripture shown by the
proof-texts adduced. It has been an amazing thing to me,
occasionally to meet with men who blamed this “catechizing” for
giving them a distaste to religion; every one in all our circle thinks
and feels exactly the opposite. It laid the solid rock-foundations of
our religious life. After years have given to these questions and their
answers a deeper or a modified meaning, but none of us have ever
once even dreamed of wishing that we had been otherwise trained.
Of course, if the parents are not devout, sincere, and affectionate,—if
the whole affair on both sides is taskwork, or worse, hypocritical and
false,—results must be very different indeed! Oh, I can remember
those happy Sabbath evenings; no blinds drawn, and shutters up, to
keep out the sun from us, as some scandalously affirm; but a holy,
happy, entirely human day, for a Christian father, mother, and
children to spend. How my father would parade across and across
our flag-floor, telling over the substance of the day’s sermons to our
dear mother, who, because of the great distance and because of her
many living “encumbrances,” got very seldom indeed to the church,
but gladly embraced every chance, when there was prospect or
promise of a “lift” either way from some friendly gig! How he would
entice us to help him to recall some idea or other, rewarding us when
we got the length of “taking notes” and reading them over on our
return; how he would turn the talk ever so naturally to some Bible
story, or some martyr reminiscence, or some happy allusion to the
“Pilgrim’s Progress”! And then it was quite a contest, which of us
would get reading aloud, while all the rest listened, and father added
here and there a happy thought, or illustration, or anecdote. Others
must write and say what they will, and as they feel; but so must I.
There were eleven of us brought up in a home like that; and never
one of the eleven, boy or girl, man or woman, has been heard, or ever
will be heard, saying that Sabbath was dull or wearisome for us, or
suggesting that we have heard of or seen any way more likely than
that for making the Day of the Lord bright and blessed alike for
parents and for children. But God help the homes where these things
are done by force and not by love! The very discipline through which
our father passed us was a kind of religion in itself. If anything really
serious required to be punished, he retired first to his closet for
prayer, and we boys got to understand that he was laying the whole
matter before God; and that was the severest part of the punishment
for me to bear! I could have defied any amount of mere penalty, but
this spoke to my conscience as a message from God. We loved him all
the more, when we saw how much it cost him to punish us; and, in
truth, he had never very much of that kind of work to do upon any
one of all the eleven—we were ruled by love far more than by fear.
As I must, however, leave the story of my father’s life—much more
worthy, in many ways, of being written than my own—I may here
mention that his long and upright life made him a great favourite in
all religious circles far and near within the neighbourhood, that at
sick-beds and at funerals he was constantly sent for and much
appreciated, and that this appreciation greatly increased, instead of
diminishing, when years whitened his long, flowing locks and gave
him an apostolic beauty; till finally, for the last twelve years or so of
his life, he became by appointment a sort of Rural Missionary for the
four contiguous parishes, and spent his autumn in literally sowing
the good seed of the Kingdom as a Colporteur of the Tract and Book
Society. His success in this work, for a rural locality, was beyond all
belief. Within a radius of five miles, he was known in every home,
welcomed by the children, respected by the servants, longed for
eagerly by the sick and aged. He gloried in showing off the beautiful
Bibles and other precious books, which he sold in amazing numbers.
He sang sweet Psalms beside the sick, and prayed like the voice of
God at their dying beds. He went cheerily from farm to farm, from
cot to cot; and when he wearied on the moorland roads, he refreshed
his soul by reciting aloud one of Ralph Erskine’s “Sonnets,” or
crooning to the birds one of David’s Psalms. His happy partner, “Wee
Jen,” died in 1865, and he himself in 1868, having reached his
seventy-seventh year,—an altogether beautiful and noble episode of
human existence having been enacted, amid the humblest
surroundings of a Scottish peasant’s home, through the influence of
their united love by the grace of God; and in this world, or in any
world, all their children will rise up at mention of their names and
call them blessed!
 CHAPTER II.
AT SCHOOL AND COLLEGE.

A Typical Scottish School.—A School Prize.—A Wayward Master.—

Learning a Trade.—My Father’s Prayers.—Jehovah Jireh.—
With Sappers and Miners.—Harvest Field.—On the Road to
Glasgow.—A Memorable Parting.—Before the Examiners.—
Killing Work.—Deep Waters.—Maryhill School.—Rough School
Scenes.—Aut Cæsar, Aut Nullus.—My Wages.

In my boyhood Torthorwald had one of the grand old typical Parish

Schools of Scotland, where the rich and the poor met together in
perfect equality, where Bible and Catechism were taught as zealously
as grammar and geography, and where capable lads from the
humblest of cottages were prepared in Latin and Mathematics and
Greek to go straight from their village class to the University bench.
Besides, at that time, an accomplished pedagogue of the name of
Smith, a learned man of more than local fame, had added a Boarding
House to the ordinary School, and had attracted some of the better
class gentlemen and farmers’ sons from the surrounding county, so
that Torthorwald, under his régime, reached the zenith of its
educational fame. In this School I was initiated into the mystery of
letters, and all my brothers and sisters after me, though some of
them under other masters than mine;—my youngest brother James,
trained there under a master named Lithgow, going direct from the
Village School to the University of Glasgow in his fourteenth year!
My teacher punished severely—rather, I should say, savagely—
especially for lessons badly prepared. Yet, that he was in some
respects kindly and tender-hearted, I had the best of reasons to
know. Seeing me not so “braw” as the well-to-do fellows of my year,
and taking a warm interest in me as a pupil, he, concluding probably
that new suits were not so easily got in my home as in some of the
rest, planned a happy and kind-hearted surprise—a sort of
unacknowledged school prize. One evening, when my father was
“taking the books,” and pouring out his heart in family worship, the
door of our house gently opened on the latch, and gently closed
again. After prayer, on rushing to the door, I found a parcel
containing a new suit of warm and excellent clothes,—seeing which
my mother said that “God had sent them to me, and I should
thankfully receive them as from His hand, whoever might have
brought them.” Appearing in them at school next morning, the
teacher cheerily saluted and complimented me on my “braws.” I
innocently told him how they came and what my mother said; and he
laughingly replied,—
“John, whenever you need anything after this, just tell your father
to ‘tak’ the Book,’ and God will send it in answer to prayer!”
Years passed by before I came to know, what the reader has
already guessed, that the good-hearted schoolmaster’s hand lifted
the latch that evening during my father’s prayer.
All his influence, however, was marred by occasional bursts of
fierce and ungovernable temper, amounting to savagery. His
favouritism, too, was sometimes disheartening,—as when I won a
Latin prize for an exercise by the verdict of the second master, yet it
was withheld from me, and prizes were bestowed without merit on
other and especially wealthier boys; so at least I imagined, and it
cooled my ambition to excel. Favouritism might be borne, but not
mere brutality when passion mastered him. Once, after having
flogged me unjustly, on my return only at my mother’s entreaty, he
ran at me again, kicked me, and I fled in pain and terror from his
presence, rushing home. When his passion subsided, he came to my
parents, apologized, and pled with me to return; but all in vain,—
nothing would induce me to resume my studies there. Undoubtedly
at that time I had a great thirst for education, and a retentive
memory, which made all lessons comparatively easy; and, as no
other school was within my reach, it was a great loss that my heart
shrank from this teacher.
Though under twelve years of age, I started to learn my father’s
trade in which I made surprising progress. We wrought from six in
Welcome to our website – the ideal destination for book lovers and
knowledge seekers. With a mission to inspire endlessly, we offer a
vast collection of books, ranging from classic literary works to
specialized publications, self-development books, and children's
literature. Each book is a new journey of discovery, expanding
knowledge and enriching the soul of the reade

Our website is not just a platform for buying books, but a bridge
connecting readers to the timeless values of culture and wisdom. With
an elegant, user-friendly interface and an intelligent search system,
we are committed to providing a quick and convenient shopping
experience. Additionally, our special promotions and home delivery
services ensure that you save time and fully enjoy the joy of reading.

Let us accompany you on the journey of exploring knowledge and

personal growth!

ebookfinal.com