Download the full version of the textbook now at textbookfull.

com

Servers for Hackers Server Administration for

Programmers 1 (2018-06-19) Edition Chris
Fidao

https://textbookfull.com/product/servers-for-
hackers-server-administration-for-
programmers-1-2018-06-19-edition-chris-fidao/

Explore and download more textbook at https://textbookfull.com

 Recommended digital products (PDF, EPUB, MOBI) that
you can download immediately if you are interested.

Servers for Hackers Server Administration for Programmers

1 (2018-06-19) Edition Chris Fidao

https://textbookfull.com/product/servers-for-hackers-server-
administration-for-programmers-1-2018-06-19-edition-chris-fidao-2/

textbookfull.com

Linux Basics for Hackers Occupytheweb

https://textbookfull.com/product/linux-basics-for-hackers-
occupytheweb/

textbookfull.com

ubuntu and Centos linux server administration Md. Tanvir

Rahman

https://textbookfull.com/product/ubuntu-and-centos-linux-server-
administration-md-tanvir-rahman/

textbookfull.com

Time series analysis 1st Edition Palma

https://textbookfull.com/product/time-series-analysis-1st-edition-
palma/

textbookfull.com
Being John Lennon A Restless Life First Pegasus Books
Hardcover Edition Beatles.

https://textbookfull.com/product/being-john-lennon-a-restless-life-
first-pegasus-books-hardcover-edition-beatles/

textbookfull.com

The Ethics of Policing and Imprisonment Molly Gardner

https://textbookfull.com/product/the-ethics-of-policing-and-
imprisonment-molly-gardner/

textbookfull.com

Social Psychology Australian New Zealand Edition Saul

Kassin

https://textbookfull.com/product/social-psychology-australian-new-
zealand-edition-saul-kassin/

textbookfull.com

Medical Decision Making 3rd Edition Harold C. Sox

https://textbookfull.com/product/medical-decision-making-3rd-edition-
harold-c-sox/

textbookfull.com

Open Living with an Expansive Mind in a Distracted World

1st Edition Nate Klemp

https://textbookfull.com/product/open-living-with-an-expansive-mind-
in-a-distracted-world-1st-edition-nate-klemp/

textbookfull.com
Framing Community Disaster Resilience Maureen Fordham

https://textbookfull.com/product/framing-community-disaster-
resilience-maureen-fordham/

textbookfull.com
Servers for Hackers
Server Administration for Programmers

Chris Fidao
This book is for sale at http://leanpub.com/serversforhackers

This version was published on 2018-06-19

This is a Leanpub book. Leanpub empowers authors and publishers with the Lean Publishing
process. Lean Publishing is the act of publishing an in-progress ebook using lightweight tools and
many iterations to get reader feedback, pivot until you have the right book and build traction once
you do.

© 2014 - 2018 Chris Fidao

Tweet This Book!
Please help Chris Fidao by spreading the word about this book on Twitter!
The suggested hashtag for this book is #srvrsforhackers.
Find out what other people are saying about the book by clicking on this link to search for this
hashtag on Twitter:
#srvrsforhackers
Contents

Servers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . i
Video Site . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . i
Book Issues . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ii

Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . iii
Accidental Sysadmin Syndrome . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . iv
Assumptions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . iv

Linux Distributions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . v

The Sandbox . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1
Install Virtualbox and Vagrant . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1
Configure Vagrant . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2
Vagrant Up! . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4
Basic Commands . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9
Basic Software . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10
Review . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11

Security . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12
Users and Access . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13
IP Addresses . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13
Creating a New User . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13
Making Our User a Super User . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14

Setting Up the Firewall: Iptables . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24

Adding these rules . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25
Inserting Rules . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27
Deleting Rules . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28
Saving Firewall Rules . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29
Defaulting to DROP Over ACCEPT . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 30
Logging Dropped Packets . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31
Firewalls with UFW . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 33
CONTENTS

Fail2Ban . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 37
Iptables Integration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 37
Installation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 38
Configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 39

Automatic Security Updates . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 43

Package Managers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 45
Apt . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 46
New (as of 16.04 Xenial, continuing into 18.04 Bionic) . . . . . . . . . . . . . . . . . . . . . . . 46
Installing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 46
Repositories . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 47
Examples . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 47
Searching Packages . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 50

Permissions and User Management . . . . . . . . . . . . . . . . . . 53

Permissions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 54
Checking Permissions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 54
Changing Permissions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 55

User Management . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 58
Creating Users . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 59
Umask & Group ID Bit . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 60
ACL’s . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 63
Running Processes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 65

Webservers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 66
HTTP, Web Servers and Web Sites . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 67
A Quick Note on DNS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 68

DNS & Hosts File . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 70

Xip.io . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 72
Virtual Hosts . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 73

Hosting Web Applications . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 76

Three Actors . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 76

Apache . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 83
Installing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 83
Configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 84
CONTENTS

Virtual Hosts . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 87
Apache and Web Applications . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 91
MPM Configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 111
Security Configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 114
Envvars . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 115

Nginx . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 119
Features . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 119
Installation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 120
Web Server Configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 121
Servers (virtual hosts) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 123
Integration with Web Applications . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 129

PHP . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 137
Installation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 137
Configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 138
PHP-FPM . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 140
PHP in Nginx: Subdirectories . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 150

Server Setup for Multi-Tenancy Apps . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 157

DNS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 157
Multi-Tenancy in Apache . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 159
Multi-Tenancy in Nginx . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 160

SSL Certificates . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 162

SSL Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 163

Using SSL in Your Application . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 163

Creating Self-Signed Certificates . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 165

Creating a Wildcard Self-Signed Certificate . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 167
Apache Setup . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 168
Nginx Setup . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 172
One Server Block . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 173

Extra SSL Tricks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 174

LetsEncrypt . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 175
Web Server Security . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 175
Install Letsencrypt . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 176
Install a Certificate . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 176
Nginx Configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 178
Renewal . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 179
CONTENTS

Multi-Server Environments . . . . . . . . . . . . . . . . . . . . . . . . . . . 181

Implications of Multi-Server Environments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 182
Asset Management . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 182
Sessions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 183
Lost Client Information . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 184
SSL Traffic . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 185
Logs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 186

Load Balancing with Nginx . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 188

Balancing Algorithms . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 188
Configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 189
Mapping Headers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 194

SSL with Nginx . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 200

SSL-Termination . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 200
SSL-Passthru Support . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 200
When to use Pass-Thru . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 201
Load Balancer (Server A, at 52.90.130.140): . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 201
Web Server (Server B, at 52.23.215.245): . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 202

Load Balancing with HAProxy . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 205

Common Setups . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 205
Installation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 205
HAProxy Configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 206
Monitoring HAProxy . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 214
Sample NodeJS Web Server . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 215

SSL with HAProxy . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 217

HAProxy with SSL Termination . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 217
HAProxy with SSL Pass-Through . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 220
Sample NodeJS Web Server . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 222

Web Cache . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 223

Nuts and Bolts of HTTP Caching . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 224

Object Caches . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 224
Web Caches . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 224
Types of HTTP Caches . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 225

An Origin Server . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 227

Testing Caching Mechanisms . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 227

Nginx Web Caching . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 231

CONTENTS

Use Cases . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 231

How It Will Work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 232
Origin Server . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 232
Cache Server . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 233
Proxy Caching . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 237
Example: Caching Specific URIs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 240

Varnish . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 242
Origin Server . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 242
Install Varnish . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 243
Basic Configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 243
Increasing Cache Hit Rate . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 249
Varnish Tools . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 251
Extra Resources . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 253

Logs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 254

Logrotate . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 255
What does Logrotate do? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 255
Configuring Logrotate . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 255
Going Further . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 261

Rsyslog . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 262
Configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 262
Usage . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 266
Should I Use Rsyslog? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 269
Sending To Rsyslog From An Application . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 269

File Management, Deployment & Configuration

Management . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 271
Managing Files . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 272
Copying Files Locally . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 272
SCP: Secure Copy . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 272
Rsync: Sync Files Across Hosts . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 273
Deployment . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 275

Auto-deploy with GitHub . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 277

How it Works . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 277
Node Listener . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 277
Shell Script . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 278
Putting it together . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 279
Firewall . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 280
CONTENTS

Configuration Management with Ansible . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 281

Install . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 281
Managing Servers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 282
Basic: Running Commands . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 283
Basic Playbook . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 285
Roles . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 289
Facts . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 296
Vault . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 298

SSH . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 302
Logging in . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 303

SSH Config . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 304

SSH Tunneling . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 306

Local Port Forwarding . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 306
Remote Port Forwarding . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 308

One-Off Commands & Multiple Servers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 309

Basic Ansible . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 309

Monitoring Processes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 312

A Sample Script . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 313

System Services . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 314

System V Init (SysVinit, SysV) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 314
Systemd . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 314
Upstart . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 317
The Service Command . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 318
Using These Systems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 319

Supervisord . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 320
A Chain of Process Monitors . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 320
Installation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 321
Configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 321
Controlling Processes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 323
Web Interface . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 324

Forever . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 326
Installation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 326
Usage . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 326

Circus . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 328
CONTENTS

Installation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 328
Configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 329
Controlling Processes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 331
Web Interface . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 332
Starting on Boot . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 333

Development and Servers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 337

Serving Static Content . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 338

Built-In . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 338
NodeJS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 338
Dynamic Content . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 339
Visit https://textbookfull.com
now to explore a rich
collection of eBooks, textbook
and enjoy exciting offers!
Servers
Servers can be fun!
Knowing how to setup, run and administer a server can be as empowering as coding itself.
Some applications have needs stretching beyond what managed hosting providers are willing to
support. This shouldn’t stop us from building our applications.
Servers can be hard!
Consumers expect and demand services to be functioning. Downtime can cost real money, and is
often met with frustration and anger.
At the same time, servers are increasingly commodified. Hosting once involved a few, powerful
servers. Now, the modern Cloud consists of many small, cheap virtual machines. The more servers
we have, the more likely we are to run into issues.
The end result is that we need to build for failure. This is a Hard Problem™, and requires us to know
a lot about the servers running our applications.
This book exists because we developers are now faced with System Administration issues. We need
to at least know the basics of what goes into hosting and serving our applications.
So, let’s not get stuck with limiting hosting or broken servers!

Video Site
Since publishing this book, I’ve also collected my newsletter editions, articles and videos to
https://serversforhackers.com1 .
I’ll be continuously adding new videos! These continue to concentrate on topics important to web
servers and web development, from the basic to the complex.
The videos all come with a write-up of the commands and information presented in the video,
usually along with some extra resources. This makes the videos easy to come back to for quick
reference later.
1 https://serversforhackers.com
Servers ii

Servers for Hackers Video Site

Book Issues
All feedback is hugely appreciated! Any questions, comments, issues, stories of glory/valor and
praise can be directed to the Github repository2 created for feedback!
https://github.com/Servers-for-Hackers/the-book
2 https://github.com/Servers-for-Hackers/the-book
Introduction
Accidental Sysadmin Syndrome
You’re a developer.
A server broke, and you’re the only one around to fix it.
You have a special-needs application that requires specific software.
You need to setup a development server, and will spend half of your day trying to get some “simple”
configuration to work.
These are symptoms of Accidental Sysadmin Syndrome.
This book is for developers who find themselves needing or wanting to be a SysAdmin.

Assumptions
This book assumes at least a passing familiarity with the command line. Those who have logged
into the shell and poked around a server before will benefit the most.

If you are new to the command line, concentrate on getting comfortable with Vagrant. This
will help familiarize you with using the command line and servers.
Linux Distributions
There are many distributions of Linux. Some popular ones are Arch, Debian, Ubuntu, Redhat,
CentOS, Fedora and CoreOS.
Many of these distributions are related to each other in some way. For example, some of these
distributions are “downstream” from others.
A downstream Linux distribution includes the upstream’s distribution’s changes, and may add their
own.
For example, Ubuntu is based on Debian and is considered downstream of Debian. CentOS is based
on RedHat and is therefore downstream from RedHat. RedHat sponsors Fedora and so Fedora is
very similar to RedHat and CentOS (although it has a much more rapid release cycle).
Each distribution has opinions about Linux and its use. It would be too cumbersome to cover all
topics for each distribution and so I’ve chosen to concentrate on Ubuntu.

This book concentrates on Debian/Ubuntu, however, the serversforhackers.com3 video site

covers multiple distributions!

Ubuntu is one of the most popular server and desktop distributions. It has a great set of configurations
that come out of the box, allowing us to worry less about configuration until we need to. This makes
it easy to use.
Ubuntu updates software packages quickly relative to some other distributions. However, updating
to the latest software makes it easier to introduce new bugs and version conflicts.
Luckily, Ubuntu’s LTS releases are a good solution to this potential issue.

LTS stands for Long Term Support

LTS versions are released every 2 years but support for them last 5 years. This makes them ideal for
longer-term use.
As major versions are released yearly, only every other major release of Ubuntu is an LTS. The
current LTS is 18.04 - the following LTS release will be 20.04.
Xenial, the codename for Ubuntu 16.04, was released in April of 2016. Support officially ends 5 years
later (April of 2021).
3 https://serversforhackers.com
Linux Distributions vi

Bionic, the codename for Ubuntu 18.04, was released in April of 2018. This will be a relevant server
for at least 2 years, but likely beyond that with support officially ending 5 years later (April of 2023).
Ubuntu LTS releases offer more stability and security, but generally do not prevent us from installing
the latest software when we need to. This makes them ideal candidates for every-day server usage.

Popularity is Relative
RedHat Enterprise (RHEL) is a popular distribution in the enterprise world. Many hosting
companies use CentOS along with cPanel/WHM or Plesk control panels. In the open
source/startup worlds, Ubuntu is one of the most popular distributions of Linux.

Because Ubuntu is closely tied to Debian, most, but certainly not all, topics included here will be
exactly the same for Debian.
For RedHat/CentOS distributions, most topics will have small-to-large differences from what you
read here.
In any case, much of what you learn here will be applicable to all distributions.
I recommend this Rackspace knowledge-base article for more information on the various Linux
distributions: https://community.rackspace.com/products/f/25/t/66774 .

4 https://community.rackspace.com/products/f/25/t/6677
The Sandbox
If you want a sandbox - a place to safely play with a server - this chapter is for you.
The topics of the “Sandbox” section is not necessary to follow along in this book, but it will be
helpful.
You’ll learn how to setup a local server on which you can develop an application or experiment with
new technology. As a bonus, you’ll avoid mucking up your computer with development software!
We’ll briefly cover using Vagrant to setup a local server.
The benefit of Vagrant is that it will let us use a “real” server to test on. You can create a server
also used in production. Virtual servers are also safe - we can thoroughly mess them up, throw them
away and recreate them as much as we need.
Let’s get started with Vagrant!

Install Virtualbox and Vagrant

Virtualbox is a tool for creating Virtual Machines. Vagrant is a tool that lets you easily create and
manage virtual machines.
Vagrant takes care of file sharing, network setup and other sticky topics.

A Virtual Machine is a (guest) computer running inside of your (host) computer. VirtualBox
“virtualizes” hardware by making virtual servers think they are running on real hardware.
A guest computer can be almost anything - Windows, Mac, Linux or other operating
systems.

Here’s some important vocabulary: Your computer is called the “host” machine. Any virtual machine
running within the host machine is called a “guest” machine.

I’ll use the term “virtual machine” with “server” interchangeably, as we’ll be creating Ubuntu
servers (VMs) to learn on.

To get started, the first step is to install Virtualbox and Vagrant. These are available for Windows,
Mac and Linux. Installing them only involves browsing to their websites and downloading/running
their installers. You may need to restart your Windows after installing Vagrant.

For this book, you will need Vagrant version 1.5 or higher. Most versions of Virtualbox
should work, I always update to the latest of these two tools.
 The Sandbox 2

Configure Vagrant
Once you have installed these, we can get started! We’ll get Vagrant going by running the following
commands on our host machine.
On Mac, open up the Terminal app. On Windows, you can use the CMD prompt (or your command
line tool of choice) to run Vagrant commands.
On Mac:

1 mkdir -p ~/Sites/sfh
2 cd ~/Sites/sfh
3 vagrant init bento/ubuntu-18.04

On Windows:

1 mkdir C:\sfh
2 cd C:\sfh
3 vagrant init bento/ubuntu-18.04

From here on, I won’t differentiate between Windows and Mac commands. We’ll mostly be
within a server in any case, so the commands will not vary no matter what type of computer
your host is.

The vagrant init command creates a new file called Vagrantfile. This file is configured to use
Ubuntu LTS server.

See the included video for getting started with Vagrant on Virtualbox.

The Vagrantfile created will look something like this (when all the comments are stripped out):
File: Vagrantfile
1 Vagrant.configure(2) do |config|
2
3 config.vm.box = "bento/ubuntu-18.04"
4
5 end

If you look at your file, you’ll see lots of comments, which show some configurations you can use.
I’ll cover a few that you should know about.
The Sandbox 3

Networking
The basic install of Vagrant will do some “port forwarding”. For example, if Vagrant forwards port
8080 to the server’s port 80, then we’ll go to http://localhost:8080 in your browser to reach the server’s
web server at port 80. This has some side effects.

A side effect of this port forwarding has to do with interacting with web applications. You’ll need
to access web pages in your browser using the port which Vagrant sets up, often “8888”. Instead of
“http://localhost”, you’ll use “http://localhost:8888” in the browser. However, your application may
not be coded to be aware of the non-standard port (8888). The application may redirect to, create
links for or submit forms to standard port 80 instead of the forwarded port!
I like to get around this potential problem by assigning a private-network IP address to my Vagrant
server.
To do this, open up your Vagrantfile and make it look like this:
 The Sandbox 4

File: Vagrantfile
1 Vagrant.configure(2) do |config|
2
3 config.vm.box = "bento/ubuntu-18.04"
4
5 config.vm.network :private_network, ip: "192.168.22.10"
6
7 end

The private_network directive tells Vagrant to setup a private network. Our host and guest machines
can communicate on this network. This assigns the guest server the IP address of 192.168.22.10.
Note that each server should have a unique IP address just in case they are run at the same time.

There are IP address ranges set aside for private networks. Generally you can use 10.0.0.0
- 10.255.255.255, 172.16.0.0 - 172.31.255.255, and 192.168.0.0 - 192.168.255.255.
However, always avoid the lower and upper IP addresses within those ranges, as they are
often reserved.

Vagrant Up!
Once the Vagrantfile changes are saved, we can run the vagrant up command. This will download
the bento/ubuntu-18.04 base server (“box”) and run it with our set configuration.
Visit https://textbookfull.com
now to explore a rich
collection of eBooks, textbook
and enjoy exciting offers!
 The Sandbox 5

1 vagrant up

If Vagrant cannot find the Vagrantfile, you need to cd into the directory containing the
Vagrantfile.

You’ll see some output as Vagrant sets up the Ubuntu server. Once it’s complete, run vagrant status
to see that it’s powered on and running.

1 vagrant status

You should see output similar to this:

1 Current machine states:

2
3 default running (virtualbox)
4
5 The VM is running. To stop this VM, you can run `vagrant halt` to
6 shut it down forcefully, or you can run `vagrant suspend` to simply
7 suspend the virtual machine. In either case, to restart it again,
8 run `vagrant up`.

Our machine, named “default” is running, using VirtualBox.

Now we need to log into this server. Vagrant sets up a way to log in without needing a password
nor SSH key. Try running the command vagrant ssh-config. It will show you some information,
including the port forwarded that you can use to SSH into, and the private key configured.
That’s nice for debugging, but you don’t need that information. We can run the vagrant ssh
command to log directly in.
The Sandbox 6

Congratulations, you’re now inside of a real server! Poke around a bit - try some of these commands
out if they are not familiar to you:

• ll - A built-in alias for the command ls -lah, this will list all files and directories within the
current directory, along with their current size in human-readable format.
• lsb_release -a - A command to show all release information about this server.
• top - A command to show running processes and various system information. Use the ctrl+c
keyboard shortcut to return to the prompt.
• clear - A command to clear currently visible output within your terminal. The keyboard
shortcut ctrl+l will do this as well.
• df -h - See how much hard drive space is used/available.
• free -m - See how much memory (RAM) is used/available. Due to various caching mechanisms,
this output can be a tad confusing. To see the actual amount of available memory, take the value
from the free column in the row that says -/+ buffers/cache.

File Sharing
Vagrant sets up file sharing for you. The default shares the server’s /vagrant directory to the host’s
directory containing the Vagrantfile.
In our example, the host machine’s ∼/Sites/sfh directory is shared with the guest’s /vagrant
directory.
 The Sandbox 7

The tilde ∼ expands to the current user’s home directory. ∼/Sites/sfh expands to
/Users/fideloper/Sites/sfh on my Macintosh.

From within our server, let’s list the contents of the /vagrant:
1 ls -la /vagrant

Its output will be something like this:

1 drwxr-xr-x 1 vagrant vagrant 136 Jun 14 16:56 ./
2 drwxr-xr-x 23 root root 4096 Jun 14 19:33 ../
3 drwxr-xr-x 1 vagrant vagrant 102 Jun 14 16:54 .vagrant/
4 -rw-r--r-- 1 vagrant vagrant 480 Jun 14 16:56 Vagrantfile

We see our Vagrantfile and a hidden .vagrant directory containing some meta data used by
Vagrant.
On my host machine (my Macintosh), I’ll create a new text file in ∼/Sites/sfh named hello.txt:
1 echo "Hello World" > ~/Sites/sfh/hello.txt

Now if I log into the guest server, I’ll see that file is available there as well:
1 # See files in /vagrant
2 cd /vagrant
3 ls -la
4
5 # Output the content of "hello.txt"
6 # with the "cat" command
7 cat /vagrant/hello.txt # Output: "Hello World"

This allows us to edit files from our host machine while running the server software within our
guest server!
A Vagrantfile with the default file sharing configuration in place would look like this:
File: Vagrantfile
1 Vagrant.configure(2) do |config|
2
3 config.vm.box = "bento/ubuntu-18.04"
4
5 config.vm.network :private_network, ip: "192.168.22.10"
6
7 # Share Vagrantfile's directory on the host with /vagrant on the guest
8 config.vm.synced_folder ".", "/vagrant"
9
10 end

Server Network
Let’s check out the network configuration. Within the server, run the command ifconfig:
 The Sandbox 8

1 ifconfig

This usually has a good amount of output:

1 eth0 Link encap:Ethernet HWaddr 08:00:27:aa:0e:10

2 inet addr:10.0.2.15 Bcast:10.0.2.255 Mask:255.255.255.0
3 inet6 addr: fe80::a00:27ff:feaa:e10/64 Scope:Link
4 UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
5 RX packets:558 errors:0 dropped:0 overruns:0 frame:0
6 TX packets:379 errors:0 dropped:0 overruns:0 carrier:0
7 collisions:0 txqueuelen:1000
8 RX bytes:56936 (56.9 KB) TX bytes:48491 (48.4 KB)
9
10 eth1 Link encap:Ethernet HWaddr 08:00:27:ac:ef:d2
11 inet addr:192.168.22.10 Bcast:192.168.22.255 Mask:255.255.255.0
12 UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
13 RX packets:4 errors:0 dropped:0 overruns:0 frame:0
14 TX packets:11 errors:0 dropped:0 overruns:0 carrier:0
15 collisions:0 txqueuelen:1000
16 RX bytes:1188 (1.1 KB) TX bytes:958 (958.0 B)
17
18 lo Link encap:Local Loopback
19 inet addr:127.0.0.1 Mask:255.0.0.0
20 inet6 addr: ::1/128 Scope:Host
21 UP LOOPBACK RUNNING MTU:65536 Metric:1
22 RX packets:12 errors:0 dropped:0 overruns:0 frame:0
23 TX packets:12 errors:0 dropped:0 overruns:0 carrier:0
24 collisions:0 txqueuelen:0
25 RX bytes:888 (888.0 B) TX bytes:888 (888.0 B)

The ifconfig command will one day be replaced by the ip command, but not yet!

The ifconfig command output a lot of content! What are we looking at? Well without getting too
deep into this, we are looking at three networks setup within this server. Each network is called an
“interface”.

• lo - The loopback interface. This is used for internal communication between services within
the server. This is “localhost” - 127.0.0.1
• eth0 and eth1 - These are two additional networks created as well. We can see the IP address
we assigend the server at eth1 - 192.168.22.10. The server also has its own private network,
with the IP address 10.0.2.15 assigned to this machine.

The ifconfig command is a quick way to check the IP address of your server as well as see the
various networks the server is connected to.
 The Sandbox 9

You’ll always see a loopback interface. You’ll usually see an internal network, useful for communi-
cating within a local network such as a data center.
Most server providers will assign a server a public IP address. Servers with a public IP address can
usually be reached by users on the internet.

Basic Commands
We’ll be using the command line for 99.9% of this book. On Mac and most Linux desktop
distributions, this means using the Terminal app.
On Windows, this means using the CMD prompt, or any other shell you might install. I’m personally
partial to Git Bash, which is usually installed alongside Git on Windows. You can run the most
common Linux commands with it.
If you’re not logged into your Vagrant server, log back in using vagrant ssh.
Here are some commands you’ll need to know for getting around a server:
pwd - Print working directory. The “working directory” is the directory you are currently in. When
you first log into a server, you’re usually placed in the user’s “home” directory, most often at
/home/username. In our Vagrant server, we’ll be placed in the /home/vagrant directory when we
log in.
ls - List Directory Contents

1 # List contents of current working directory

2 ls
3
4 # List contents in a list form, with extra information:
5 ls -l
6
7 # List contents, including "hidden" files/folders
8 ls -la
9
10 # Add human-readable file/folder sizes:
11 ls -lah

cd - Change Directory.

1 # Change into the "/home/fideloper/sites/sfh" directory.

2 cd /home/fideloper/sites/sfh
3
4 # Same as above, but with the "~" shortcut
5 # to the current users home directory
6 cd ~/sites/sfh

mkdir - Create a directory

Random documents with unrelated
content Scribd suggests to you:
see him at his convent, as soon as ever I dared to move about, and
promised me a very good reception.
The Abyssinians in making their drinking-horns, show
considerable ingenuity, not so much in the complexity of their
machinery, as in the great simplicity of the few aids they require to
turn out a very neatly made article. A proper ox-horn being selected,
it is cut into such lengths as are required. One of these is then
gradually fixed upon a conical wooden mould; boiling water being
employed to soften the horn, and make it more readily adapt itself
to the shape, it is then laid aside for a few days, when the form
becoming fixed, it is placed in the lathe to receive a series of circular
cut rings, with which the outside is usually ornamented.
The lathe is nothing more than two short sticks placed in the
ground, not more than three inches high above its surface. From the
centre of each end of the mould an arm projects about six inches
long, which is armed with a bit of iron. These iron points are
received in the short stick supports, and the mould, with the horn
upon it, then revolves freely. The workman sits upon the ground,
and with his feet pressed hard against a stick, supports it in this
manner against two stones, placed at a convenient distance in front
of his work. This forms a rest for his cutting instrument, which he
holds in his left hand, and presses against the horn, whilst with his
right he wheels backwards and forwards the mould by a small catgut
string bow, applied and used in the same manner as is the same tool
by many artisans in England.
Not only are drinking-horns thus fashioned (and which, I must
observe, are finished by a piece of round wood being fitted like a
thin cork into the lower and smaller end), but also earrings are
turned from the long black horn of the sala, a species of antelope,
common in Adal and the low countries around Abyssinia. The solid
extremities of the horns only are used, so that not more than two
pair of earrings can be made from one horn, which is at least two
feet long. The earrings are large and clumsy, but, considering the
simple means employed in making them, are not despicable works
of art. Each is turned in two pieces, not at all unlike in form and size
high convex buttons, with small straight shafts projecting from the
inside centres. These shafts are made so that one receives the other,
and the earring thus formed looks like two small wheels connected
by a short axle. To receive them into the ear a very large hole is
required, and the axis of one of the halves being first introduced, the
other is fixed upon it, and the lady then turns round, to ask how the
new ornament looks.
Sometimes I have seen these horn earrings ornamented with an
inlaid star of silver, and many an hour’s labour have I had myself,
letting in little brass studs from an old box-lid into the surface in the
same manner, to please some of my female friends, who would
come begging to have their earrings thus improved in appearance.
Besides these ornaments turned from the sala horn, small black
rings are cut, and I have also seen a neat little bottle, about two
inches long, turned in a very ingenious manner, and which was
intended to hold “col” (the black oxide of antimony), with which the
Mahomedans adorn their eyelids, and the Christians employ as a
medicine, applying it in the same manner. Besides horn earrings, the
Abyssinian women wear large silver ones, sometimes weighing as
much as two or three dollars each. One fashion alone is general in
Shoa, a back and front portion, each of which invariably consists of
three large beads, surmounted by a fourth. These are fixed in the
ear in a similar manner as the horn ones, and look not unlike small
bunches of grapes projecting before and behind.
Whilst I am upon this subject, I may observe that the Shoan
women are exceedingly fond of silver ornaments, and all their riches
consist of such stores. Dollars are only valued as the means of thus
enabling the possessors to adorn themselves or their women, for all
the coin of this sort which enters Shoa ultimately finds its way into
the crucible, except such as falls into the hands of the King, and
which are destined for a less useful end, these being securely
packed in jars, and deposited in caves. One hill, called Kundi, a few
miles to the north of Ankobar, is pierced by numerous subterranean
passages, in which are hidden in this manner immense treasures in
gold and silver. They are kept closed by heavy doors of iron, and the
whole hill, which is surmounted by a church, dedicated to the Virgin,
is under the care of a vast number of priests. I think it not
improbable that some excavated chambers that have been found in
Egypt, and in rocks near Jerusalem, and considered to have been
intended for tombs, were in fact the treasuries of the monarchs of
these countries.
One ornament of silver, and which is worn by the women of Shoa
upon the breast, hanging from the neck by a chain, also of silver, is
in the form of a clasp, three or four inches long, and one inch broad;
upon its front surface not unfrequently is rudely engraved some
simple design in waving lines. Bracelets of silver are sometimes
seen, and with the Mahomedan women, they are invariably of that
metal; but the Christians generally wear plain ones, made of pewter,
with anklets to correspond.
Besides the little unpretending martab of blue silk, the Christian
women, if they can afford it, wear large necklaces of beads, and the
British Political Mission have greatly increased the stock of these
ornaments that is now brought into the market. Those I have seen
were made generally by a succession of loops, consisting of seven or
eight threads of different coloured seed beads, collected at certain
lengths into one string, through a large angular-cut piece of amber.
Eight or ten of these loops formed a long negligee, which,
ornamented with a large tassel of small beads, was a present suited
even for the acceptance of royalty. The Mahomedan women, on the
contrary, wear one string of beads around their necks, formed of a
hundred large and differently coloured beads, among which bright
red ones seem to be preferred. These are divided into lengths by the
interposition of pieces of amber, at least twice as long as those
employed by the Christian women in collecting together into one,
the various bead threads of their necklaces. The silver bracelets of
the Islam are also different in form from those worn by the
Christians, consisting of two or three thick silver wires, twisted upon
each other, and finished at each extremity by a beaten square head.
This is looped around the wrist, where it remains until required as
security for loans, the most important use, I think, of silver articles
in Shoa, amongst all religious denominations. No golden ornaments
are ever observed among the Shoans, for a sumptuary edict of the
Negoos forbids his subjects the use of this metal; the royal family of
course being excepted.
 CHAPTER XXIII.
Wallata Gabriel dismissed.—​Reinstated.—​Comparison of different
races of man.—​Of human varieties.—​Of the process of brewing.—​
Abyssinian ale.—​Ingredients.—​The horn of plenty.
August 14th.—Wallata Gabriel was a very good housekeeper, but
unfortunately, like most other young women in Shoa, and, I believe,
in all Abyssinia, she had a great many followers. Whenever
Walderheros and I walked out, some one or other would always be
manœuvring to get out of the house unobserved on our return; and
although I was rather suspicious of some of her lovers making free
with what little property I had, still I had as yet never missed
anything. I had frequently reminded Walderheros of this weakness in
his wife, but he always, in reply, appealed to me if she was not a
good servant, although, he added, that it was for my convenience
that he recognised her as his wife, and would previously have
divorced her, only he thought that together they were so well
adapted to manage my domestic affairs, that he could not do better
than keep her. Coming in rather inopportunely this afternoon, after a
long walk round the town, I could do nothing else but turn her away
at once; whilst Walderheros expostulated with her paramour, among
other severe things, asking him if he were not ashamed to intrude in
such a manner when the balla bait (the master of the house) was
not at home.
About an hour after I had dismissed Wallata Gabriel, an old lady,
a relation of Walderheros, made her appearance, bringing some of
the sweetly-scented herb called Err-guftah, as a memolagee. On
requesting to know what she required, a long apologizing palliating
intercession, of no ordinary character, was made for my delinquent
housekeeper. She attributed my severity, she said, entirely to my not
knowing Abyssinian customs, and turning to Walderheros, who sat
on the raised rim of the hearth, stirring up the dry ashes with a
stick, she upbraided him for not raising his voice in the behalf of his
lawful wife. I was determined she should not return, and was
dismissing the mediatrix with a positive refusal, when Walderheros
looked at me with a most grievous expression of countenance, and
lifted up the top of the straw bread-basket, to intimate, by its empty
condition, how badly we should be off for dinner if I persisted in not
recalling his wife. I could not help smiling, and the old lady, seeing
me relent, put her head out of the door, and called out “Wallata
Gabriel!” two or three times. The fair penitent very soon appeared,
for she had been sitting in the lane all the time, and came tripping
in, laughing and looking quite happy at being reinstated, and
without the least trace of sorrow or contrition in her countenance.
This apparent lack of morality amongst the Shoans, like their
Church history, is quite beyond my understanding. Yet even as
respects this, a person educated in the more correct principles of
what is considered to constitute social happiness, does not perceive
in Shoa that violence done to propriety, which similar conduct in
many of the southern states in Europe is apt to excite. The loose
habits and indiscriminate intrigue, which displeased me when I
witnessed it among the inhabitants of various countries situated
upon the northern shores of the Mediterranean, only occasioned a
smile when I observed it in Abyssinia. Among the former it was the
pretension and affectation of virtue that made their sins stand in
bolder comparison as vices, than a somewhat similar course of
conduct among the simple, good-natured inhabitants of the latter
country, who have no public opinion to propitiate, or, on the other
hand, to control them, and whose naturally yielding disposition
renders them too prone to indulgence; where also, let it be
recollected, religion applies no curb, for the priests themselves in
Shoa have had the decency to cease preaching that, which they
never pretend to practise.
I was not many weeks upon the banks of the Ganges, and had
not many opportunities of observing the native population of India,
but the impression upon my mind of the moral character of the
people generally of that country, apart from their particular worship,
is, that the Indians, especially the women, possess in a great degree
that moral principle, that delicacy of the mind, which is essentially
the basis of that high sense of honour and personal respect, which
constitute female chastity. I was enabled to draw, by my visit to
India, a very interesting contrast between the women of that
country and those of Shoa. Let me compare two extreme specimens,
which will illustrate more broadly that which I wish to establish; that
important differences in the constitution of the mind are the primary
causes of those varieties in human nature; but which have been
previously determined by differences in the features and form. This
comparison will assist me, as truth, I think, is sometimes strikingly
demonstrated by widely different contrasts; the paradox surprises
and amuses the mind, and its effect in consequence is more
permanent.
We will first, however, cursorily allude to the physical differences
between the Indian girl and the Abyssinian, as a kind of introduction
to their habits. The former is tall, thin, long-waisted, with an angular
configuration of form, her features regular, sharply defined, bright
and placid. She is a Circassian with a dark skin. Turn to the
Abyssinian beauty, her eyes smile uncontrollably as you look. Her
figure is short, plump, and roundly formed, with small, but full
voluptuous features, that appear blended together with an infantile
expression.
The minds of both are uneducated and natural: in this
circumstance of their character, fortunately for the effect of the
comparison, they may be supposed to be alike. But are their
dispositions or conduct the same? How very different. The Indian girl
has considerable personal vanity, is fond of ornaments and show,
and seeks to attract attention by rich clothes, or studied graces.
Even in their national attitudinizing, and the alluring nautch, repose
and quiet seem to distinguish her from the laughing, romping, dress-
neglecting Abyssinian, who, to attract notice, affects the child, and
endeavours to please by artlessness and simplicity. As lovers, the
Indian girl capriciously selects one lord, but the Abyssinian would
consider this to be petty treason against nature, and a crying sin;
she always loves the nearest, and whilst the eyes of that one are
upon her, is reluctantly constant, but considers all engagements
quite at end by absence, however short. Our coquettes, tall girls,
with thin lips and cold sparkling eyes, always remind me of the
Indian beauty, whilst our laughter-loving romps, even in their
features and form, seem to belong to the Abyssinian mould, in some
measure demonstrating the solution of the difficult problem, of
accounting for the origin of those differences in the several varieties
into which ethnologists have divided mankind. Among our own
acquaintances, under external circumstances, exactly alike, nature
produces by the mysterious agency of mental endowments, the
possible mothers of families of man, which, under different
circumstances of situation and of social education, would ultimately
present two nations as distinct in every phenomenon of external
appearance as are the most opposed specimens of the Circassian or
the negro type.
The Shoans are certainly not a virtuous people, according to our
ideas, and if we are to judge them by the standard of our moral
code; but I positively deny that they are an immodest people, except
among those where the dehumanizing influence of Mahomedanism,
by degrading woman to the condition of a slave, has engendered the
disgusting sensuality which characterizes the professors of that
religion, and even these in Abyssinia are as superior as possible in
this respect to the Mahomedans of Arabia and Persia. My opinion as
to the modesty of the Shoan women may not perhaps be sufficiently
understood, to be considered correct; but it was formed by
observing the freedom from all restraint which they appear to enjoy
in their country and among their families. This implies some
confidence on the part of the men, and a woman must be modest to
the extent that society requires, to command such a mark of
deference and respect from the opposite sex.
I have seen sufficient, indeed, to convince me that the youth of
Abyssinia, males and females, whilst influenced by the feelings
natural to that age, are diffident, confiding, and good-natured, and
however they may become altered by the experience of increasing
years, and the education of after life, these moral principles still
prevail, and give a favourable inclination to the practice of virtue and
justice, that renders their social condition productive of much
happiness to themselves, and affords some pleasure to the mind,
that contemplates their character and condition, uninfluenced by the
bias derived from the moral discipline of a very differently
constituted community.
August 15th.—Being unable to go much abroad to extend my
acquaintance with the habits and manners of the Shoans, I was
particularly desirous that my establishment should be entirely
conducted upon the principles of Abyssinian domestic economy. As
this exactly suited the inclination of Walderheros, and as it did not
matter to Goodaloo in what way things were managed, so that he
got a large roll of teff bread in a morning to wrap up in his mekanet
when he went to cut fuel, and his supper in the evening when he re
turned, the proposal met with universal approbation from the
members of my household, and my wishes were attended to in
every particular.
Among other employments that occupied Walderheros and his
wife two or three days every fortnight, was that of brewing, which
was no trifling affair, as a fresh jar of ale, holding at least four
gallons, was broached every day. The process is simple enough as
performed in Shoa, and instead of being obliged to stand some time
after it is made, five or six days is the time that is required to ripen
and fine the beverage, which, if made well, is agreeable and very
strong, with a slight acidity, that reminded me of the oldest ale I had
ever drunk in England.
The ingredients are various, sometimes wheat or barley, or
jowarhee grain, but in the kolla or low countries the latter is
preferred, and as I also found some useful medical effects resulting
from its use, my ale most frequently was made from this grain. The
jowarhee is the durra of the Arabs, and is largely grown in India,
where I think English residents might, by following the Abyssinian
method, always have home-brewed ale in their houses.
When barley is employed for the purpose of brewing, it is first
well dried in the sun, and afterwards broken in a mortar to divest it
in some measure from the coarse outer skin, and which is separated
by the usual process of fining through a grass made sieve. The
prepared grain is then placed in a large earthenware saucer, at least
two feet in diameter, and in the centre about six inches deep. This
being raised upon three supports over a low fire, an attendant keeps
stirring the contents, using for this purpose the small reaping hook
of the country, the convex curve of which scrapes the barley from
the bottom of the saucer, and prevents its burning. Whilst this is
going on, another servant washes the jars intended to receive the
ale, and which, after being well rinsed out, are fumigated by a few
leaves of the bitter gaisho plant, placed upon a little lighted
charcoal, on a broken piece of earthenware, and is introduced
beneath the mouth of the vessel, which is held over it to receive the
ascending smoke.
Gaisho are the leaves of a species of Rhamnus indigenous to
Shoa, for besides being regularly cultivated in favourable situations
between six to eight thousand feet above the level of the sea, I have
also found it growing wild at the base of the hill of Kundi, above the
Tabeeb monastery, in that neighbourhood. These leaves are
serrated, and of the form and size of bay leaves, only of a lighter
green. When used, after being dried in the sun, they are pulverized
in a mortar until a very fine powder of an intensely but not
permanent bitter is produced. It is then ready for the purposes
required, which are similar to those of hops and gentian in brewing
our beer.
After the barley has been well roasted, it is taken out of the pan
and ground into a coarse meal, which, after being slightly wetted
with water, is again exposed to the action of heat in the same
manner as before, until it has become thoroughly scorched; being
kept the whole time well stirred to prevent its burning. During this
process, a small jar containing a thin acid mixture of flour and water,
called wahaka, or leaven, to which the powdered gaisho has been
previously added, has been standing to infuse in the warm wood
ashes. The meal being now removed from the fire, is put into
another jar, and sufficient water being added to make it into a paste,
the wahaka is also added, and the mixture remains for the rest of
the day. On the morrow, the whole contents of the lesser jar are
transferred into one much larger, capable of holding at least thirty
gallons of water, and which is now brought and poured by
successive jarsfull into it until full. This is allowed to stand another
day, when the surface, showing evidences of a certain point in the
process of fermentation having been attained, the whole is then
decanted, and strained through a large straw funnel into a number
of lesser jars, each of which contains from four to five gallons. These
are carefully stoppered by large cakes of a dirty mixture of the
refuse of the strainings of the large jar and of clay, and which are
plastered over the mouths of the jars. In about three days the ale is
ready for use, and if made properly, is most excellent; bright,
sparkling, and potent, it reminded me, by a slight acidity, of the best
October of England. After nine or ten days, Abyssinian ale gets too
sour to be a pleasant draught, which I attribute to the imperfect
covering afforded by the clay plasters which close the jars in which it
is contained.
When jowarhee, or durra is used, the grain, after being reduced
to a fine meal, is made into a paste, or rather thin batter, with the
wahaka. After standing one day and night, it is then made into thin
cakes, as in the usual manner of baking teff bread. These cakes are
afterwards broken up and placed in the large jar, the gaisho and
water being added exactly as in the process where barley is
employed, and when fermentation has somewhat progressed, the
wort is in the same manner strained and decanted into lesser jars.
There is a red variety of jowarhee, or millet, called tallange
largely cultivated in Shoa for brewing the tallah alone, as it is
considered to produce the best description of the beverage. It is said
to be injurious to man eaten in the form of nuffrau, or bread,
although the grain is given to cattle for food. This certainly makes
very fine ale, and should the experiment of making jowarhee beer
succeed in the East, where I hope it will be tried, it will be very easy
to procure some tallange for seed from Abyssinia, should the plant
not exist, as I do not expect it does at present, in India.
In conjunction with all these different grains, and with a mixture
of all, which is sometimes employed in the same brewing, it is not
unusual to add a little real malt called bikkalo, generally in the
proportion of double the quantity of gaisho. To make the bikkalo, a
quantity of barley is placed in a flat dish and well wetted with water,
a large stone being placed upon it. This presses the sprouting grain
into one mass of a wheel-like form, which, when the operation has
proceeded as far as is desired, is taken from the dish, a hole made
through the centre, and it is strung upon a rope, where it hangs to
dry against the wall, and is a common ornament of the interior of
the houses in Shoa. On occasions of brewing, the quantity required
is broken off, and its value as an ingredient is well-known, for a
common Shoan proverb says, “the more bikkalo the better ale.”
The proportions of the different ingredients are generally from
forty to fifty pounds of grain, to which is added one pound of gaisho,
and two pounds of bikkalo. From these quantities are made about
thirty gallons of very good beer, but which, as I have observed
before, will not keep more than ten or twelve days, which is one
reason why ale is brewed generally in such small quantities.
There is no beverage so largely indulged in by the Shoans,
whether Christian or Islam, as tallah. The Hurrahgee people are also
extremely addicted to drinking it, and when they arrive in the
country, every saltpiece that they can get is sure to be spent in ale.
It is, therefore, an essential on all occasions of rejoicings, whether of
a religious character, or at weddings, and even at funerals. In fact,
the number and size of the jars of ale provided for the company
indicates the importance of the feast, or the wealth of the
entertainer, whilst no one to whom the cornucopœia of ancient
mythology is familiar, but detects at once, the origin of that poetical
appendage to divinity, as he contemplates the parties engaged in
celebrating these jovial meetings. Every one bearing in his hand, a
deep drinking-horn, varying in length, from a long span to more than
half a cubit, which, as he drains its contents, is handed to the
servants in charge of the jars of tallah, who quickly replenish it, and
return it to the thirsty soul. Each reveller keeps to his own rude
flagon, and nothing could more strikingly typify agricultural wealth
and rustic happiness, than the representation of one of these
drinking horns; and which, ornamented and embellished by Grecian
and Latin poets, still I believe to have been the original of the
famous horn of plenty; probably derived from some Egyptian
hieroglyphic, which well expressed the condition of man it appears
so naturally to characterize.
 CHAPTER XXIV.
Visited by Ibrahim.—​Map of the Hawash.—​Its effect upon table-land
of Abyssinia.—​Future juncture with the Abi.—​Its early tributaries.—​
Effects of denudation.—​Zui lake.—​Popular tradition.—​Abyssinian
geographical work.—​Galla tribes.
August 16th.—Ibrahim, the retired slave-merchant, who had not
called since I had made the improvements in my house, came in to-
day. He was rather astonished at the transformation I had effected,
gave the table a good shake, sat down in my chair, and tabored with
his fingers against the parchment window. “Ahkeem e moot,” said he
at last, “may the doctor die! if it is not good; you are a tabeeb, and
the house of your Queen is not furnished so well.” The old
gentleman had brought his work with him, a piece of blue sood,
which he was embroidering with green and red silk in a large cross-
bar pattern, and which he told me was intended for a holiday guftah
for his wife. Here I must observe, that although the Islam women in
Shoa usually wear clothes of some common material dyed red, upon
festival days they display very rich headdresses of foreign silk, or
embroidered cotton cloth, such as Ibrahim was now working.
Walderheros placed the low Abyssinian chair for his
accommodation, and then, as was generally the case when Ibrahim
came to see me, a long conversation commenced respecting the
town of Hurrah, of which he was a native, although he had not been
to that city for the last eight or nine years. As usual, we had a map
sketched upon the floor before us, which, however, on this occasion
was not a very complicated one, merely the southern portion of the
Hawash, where it encircles Shoa, and which formed the conclusion
of the course of that river, the northern portion of which, as far as
the ford of Mulkakuyu, I had already received information of from
my Dankalli friends, Ohmed Medina and Ohmedu.
 The principal features of the geography of the country included in
the sketch map, were the three principal streams entering the
Hawash from the scarp of the Abyssinian table-land, all of which
flowed nearly to the south; but the most remarkable and interesting
one was the great indentation in the outline of the high country,
which in this situation seemed to be approaching to a separation into
two parts by the denudation of the sources of the Hawash on the
east, and a corresponding degradation on the west, occasioned by
the action of the waters of the Assabi, or Abiah, the red Nile falling
from the elevated plains of its earlier tributaries to join the Bahr ul
Abiad at Kartoom, where its height above the level of the sea does
not, I believe, exceed three thousand feet.
Surrounding the head of the Hawash, separated only by the
narrow valleys of denudation around its sources, are three elevated
countries, all forming part of the table-land of Abyssinia, and
between which, in the course of ages, this river has intruded itself
by slow degrees, and is still progressing annually farther to the west.
These three countries are Zingero to the south, Enarea to the west,
and Shoa to the north, whilst the corresponding portions of the
scarp are Gurague, Maitcha, and the ancient province of Fatagar, the
more westerly portion of which is now possessed by the Soddo
Gallas.
This now excavated portion of Abyssinia must have been at a
former period one continuous table land, and the countries of
Zingero and of Shoa then could only have been separated by
streams that flowed to the north into the Abi, or to the south into
the Gibbee, the ancient Assabi. The same convulsion which has
determined the peculiar course of the Abi, or Bruce’s Nile, seems to
have influenced the direction of the encroachment of the Hawash
into the limits of the plateau of Abyssinia; and also the position of
the débouché of the Red Nile from its summit to the plains below.
An examination of the map will show a curious correspondence
between the situation of the sources of the Hawash, of the southern
curve of the Abi, and of the break in the table land where that river
joins the Red Nile near Fazuglo. A great geological fault seems to
extend across Abyssinia in the direction of these several points, one
effect of which (that of the great disintegration of the material of the
rocks along its course) appears to me to have favoured the
denudation observed on the eastern and western borders of this
country. To this fracture I also attribute the sudden curve of the Abi
to the west, after flowing nearly due south from lake Dembea; the
physical barrier to its farther continuance in that direction not being
a ridge of hills, or what is generally termed an anticlinal axis, but the
presence of the opposite wall of the disjointed rock, which
characterizes the extension of the fault across the table land. This is
neither unfounded assertion nor rash conclusion, but the deliberate
opinion I have formed by a careful examination of the mighty
operations of nature that appear to have acted upon the surface
geography of Abyssinia from the most remote ages.
Let my reader return with me for a moment to the country of
Adal, an extensive plain, scarcely one thousand feet high above the
level of the sea. Its river, the Hawash, peculiarly its own, distinct in
the non-existence of opposite corresponding water-sheds to identify
it as having formed part of the original surface level of the
surrounding countries: an intruder, in fact, between the opposite
slopes of the river Tacazza to the north and of the river Whabbee to
the south; the countries of which were once continuous, but some
convulsion connected probably with that which has occasioned the
fault across the table land of Abyssinia, has in this position, severed
the country completely; and in the gaping chasm, filled up to a
certain level with the debris, has formed the bed of the Hawash,
which gradually progressing on every side, its wide circumference of
sources encroaches every year upon the elevated lands which
surround it.
A traveller in Adal cannot help noticing the singular character of
the situation of the river Hawash, for he crosses over its bounding
ridge to the east, and has partial opportunities of observing the bluff
scarp-like terminations of the Angotcha, the Abyssinian, and the
Hurrahgee table lands, all of which are being rapidly denuded by the
numerous little tributaries which flow to swell the Hawash. But this
extending operation is most strikingly illustrated in a line with the
fault which has extended from the sea-coast to Fazuglo, in the west
of Abyssinia. Here, to the south of Shoa, the Hawash has already
approached within one day’s journey from the deep valley of the Abi,
and removes annually great portions of the surrounding table land,
which had previously determined the rain drops to flow into that
river, but subsequent to which removal, all falling water must for the
future, aid in swelling the insidious river of the low-land of Adal. The
valleys of numerous small streams, the sides of which, denuded to
the required depth, have been thus gradually opened into, and as
this is naturally aided by the steep fall of the scarp, denudation goes
on rapidly when the first inclination towards the Hawash has been
given to the stream, that had previously meandered upon a nearly
level plain. In this manner I contend, that the valley of the Airahra,
between the narrow ridges of Ankobar, and the edge of the table-
land at Tchakkah, has been acted upon, and that the waters falling
to the west of Ankobar, and which now flow into the Hawash, were
formerly conducted to the Jumma, and so to the Abi, when the two
elevated points mentioned were continuous, as they most certainly
have been.
The geology of Abyssinia also favours these strange alterations of
its own face; for it is composed almost entirely of volcanic rocks,
easily decomposable, the operation, in fact, scarcely requires the aid
of water to occasion it; for the atmosphere alone crumbles the
hardest rocks, in the course of one year, into a stratum of loose
earth; and water appears to be merely the carrying agent, to
remove the soft soil, and expose a fresh surface to the action of the
air. It is this which adds so considerably to the fertility of the
Argobbah counties, situated on the scarp of the Abyssinian table-
land; for every fresh year, virgin earth of the most fertile capability,
is offered spontaneously, for the benefit of the cultivator, to whom,
in this situation, the use of manure is unknown.
Rain, however, aids considerably in removing vast portions of the
table-land; for during the wet season, generally some few days after
the commencement of the rains, and again, near its close, severe
thunder storms, with slight earthquakes, occur; and the devastation
which results, is not so much to be attributed to the latter, as it is to
the previously fallen rain; which, having penetrated to a certain
depth of the easily disintegrated rock, the least agitation brings
down immense quantities, from the nearly perpendicular cliffs. An
earthquake scarcely perceptible, and which, perhaps, is only
consequent upon meteoric explosion, by the reverberating vibrations
being communicated to the loose, yet prominent surfaces of the hilly
scarp; there always precipitates ruinous masses of earth and rock,
whilst not a trace of its effects can be perceived upon the table-land.
This is the real character of all earthquakes in Abyssinia I have
witnessed; and although the death of twelve or fifteen people, have
been consequent, it has only been in different situations of peril, the
proper precaution could have easily obviated, as it was where
denudation had been long undermining the foundation of their
houses, or of those on the terraces above; and which, when a
moment of extraordinary atmospheric commotion occurred, were
shaken from the sides of the valleys into the stream below. No
leaping of the earth, or those violent commotions, which mark these
convulsions in other countries, occur in Shoa. In Ankobar, during the
severest landslips, for they are nothing else, a loose stone building
thirty or forty feet high, and a still more rickety arch built by
Demetrius, although in exposed situations, were not affected in the
least.
The tremour of the earth consequent upon portions of its surface
being detached, was only felt upon the situation on which it
occurred; and were it not for the heavy fall of rocks from the
overhanging table-land, no evidences of a violent convulsion could
be ever observed; so that I am justified, in attributing to external
influences, rather than to internal operations, the occasional
agitations of the earth which are experienced in Abyssinia during the
wet season.
The combined effects, however, of all these disintegrating agents
of the table land of Abyssinia, is to increase farther westward the
course of the Hawash, and we find that in the situation most
favourable for the operation of denudation there is contained, its
most distant sources. Already, by the testimony of M. Rochet
d’Hericourt and Dr. Krapf, the head of the Hawash reaches within
thirty miles of the Abi, the Nile of Bruce, and that in that direction it
will still progress, may be safely assumed, whilst the present order
of things established by nature is continued; and in the course of
time a communication will most certainly be opened between this
river and that of Northern Abyssinia, when probably, by this addition
to its volume of water, and a continual denudation going on also
towards the east, diminishing daily the barrier between it and the
sea; the Hawash will then enter the sea, and open a fresh highway
into the interior of Africa. Geologists may observe in this mighty
operation, something analogous to that to which they attribute other
natural phenomena with which they may be familiar, and the facts
that I have stated, singular as they may appear, are as easily
demonstrated to be true as is the westward progress of the falls of
Niagara towards the lakes of Northern America.
Within the indentation in the table land to the south of Shoa,
Ibrahim placed three principal streams, all of which appear to flow
south from the scarp in that situation. These were, one stream
which separated the Maitcha Gallas from the Soddo Gallas; the
second, called Hashei, which separated the latter from the Abitshoo;
and the third was the Kassam, which flowed through the province of
Bulga. On inquiring the situation of the Zui lake, which, from
previous information, I knew was not far from the Hawash in this
situation, Ibrahim explained to me that it received the waters that
flowed from the opposite scarp to that of Shoa, and which
constituted, with the high land to the north of the Gibbee in this
situation, the country of Gurague. On the other side of the stream of
the Gibbee was Zingero. Zui, called also Lakee, has several small
islands situated in its waters, each of which is inhabited by monks,
but on the largest a very celebrated monastery exists, in which,
according to vulgar ideas, all the wealth and books relative to the
ancient empire of Abyssinia have been concealed since the
celebrated Mahomedan invasion of that country in the sixteenth
century, by Mahomed Grahnè. There may be some truth respecting
the manuscripts that are contained in the monastery of Lake Zui, but
I question much if any treasure is to be found there, for in that case
Sahale Selassee would, before this, have attempted to subdue the
Galla tribes intervening, which could be accomplished in one
campaign, for already, in that direction, the country as far as the
Hawash has submitted to him, and Zui is not more than two days’
journey to the south. That its conquest is intended by the Negoos of
Shoa, I have no doubt, and I think he only postpones it until he can
effect the reduction of the whole of Gurague, at the same time the
inhabitants of which are very much affected towards him, and in fact
consider him to be their monarch. I have witnessed two or three
interesting interviews between parties coming with unsolicited
tribute from Gurague; and when the monarch endeavoured to
induce me to remain with him, he held out the opportunity I should
have of visiting that country in the course of the next two years, by
accompanying him, and which he supposed would be a temptation
for me to stay.
The Negoos himself corroborated the statement of Ibrahim, who
had visited the shores of this lake several times, that there was no
outlet for its waters, but that it was entirely distinct from the river
Hawash. Karissa, a Galla, from Cambat, who when enslaved was first
taken to Gurague, and lived near Zui several years, also told me that
a number of small streams fell into the lake from all sides and that
there was a tradition that a long time ago, the length of which he
had no idea of, all the country now occupied by the lake which is
about fifteen miles in diameter, was possessed by seven chiefs,
whose lands, for their sins, of course, or it would not be an
Abyssinian legend, were swallowed up in one night, with loud
subterranean noises, and stars shooting out of the earth, and that
the next day nothing could be seen but the present lakes, and the
islands it contains. Considering the character of the country, and the
phenomena still witnessed in Adal, whilst the country around Zui
appears to be situated upon the same elevation above the sea; I
have no doubt that this tradition is partly founded upon fact, and
contains the national remembrance of an extensive and appalling
incident connected with some volcanic convulsion, that at a former
period occurred in this situation.
My morning’s lesson in geography terminated with a promise that
Ibrahim should get me the title of a Geez book upon the subject,
which he asserted he had seen in Hurrah, for I must observe he
ridiculed the idea of anything having been preserved during the
invasion of Grahnè into Abyssinia, by being taken to the monastery
of Zui. He stated that in the city of Hurrah, which was then the
capital of the kingdom of Adal, there was at the present day an
entire library which had formed part of the spoil of the conqueror on
that occasion, and that in the same building with the books is
preserved the original silver kettle drums that were formerly carried
before the Emperor. He had also seen a map which had been made
by the orders of Mahomed Grahnè, of the countries he had subdued
from Massoah and Gondah in the north, to Magadish in the south,
and upon which was particularly marked the site of every Christian
temple he had destroyed. A copy of this map could, I think, be easily
obtained by means of our Berberah acquaintance, Shurmalkee,
whose connexion with the city of Hurrah is much more considerable
than it is supposed to be by our Government.
Upon both banks of that part of the Hawash which partially
encircles Shoa, numerous tribes of Galla find sustenance for
immense herds of cattle. Among these, the most important are the
Maitcha and Soddo tribes, situated upon the earliest of its most
western tributaries; next to these, proceeding from the west, is the
Tchukalla; then Lakee, or those living between lake Zui and the
Hawash; to these succeed the Gilla, the Roga, and then the Gallahn,
the chief of which, Shumbo, is a son-in-law of the Negoos, baptized
and married the same day, whilst I was in Shoa. Through his district
lies the safest road to Gurague, and accordingly it is the one
principally taken by slave merchants, who, however, seldom return
that way, preferring a more circuitous one, around the sources of the
Hawash, among the tribes situated upon the table land of Abyssinia.
Adjoining to the Gallahn Galla are the Aroosee, a powerful and
warlike nation of the same people, but who appear to be
considerably in advance of their barbarous brethren. The Aroosee
are large agriculturists, and great quantities of coffee, and of a red
dye, called wurrsee, which is exported from Berberah to India and
Arabia, is produced in their country. They occupy all the district
between Hawash and the north-western streams of the Whabbee.
Where they terminate on the east, the possessions of the Hittoo
Galla commence, who also “drink of the waters of the Hawash,” and
are, it will be recollected the tribe, some of whom attacked the
Kafilah of the Hy Soumaulee, at Dophan, on the occasion of my
coming up to Shoa. On a map of a limited size, it would be
impossible to introduce the names of the numerous tribes of these
people that border upon Shoa to the south, nor would any benefit
arise from the list beyond that which may be obtained by the
general designation, “Galla tribes,” and which I have, therefore,
employed to mark the localities of these people.
 CHAPTER XXV.
No prospect of recovery.—​Slaughter of the goat.—​Manufacture of
skin-bags.—​The process.—​Farming.—​The bark employed.—​
Morocco leather.—​Carcase butchers.—​Process of cutting up meat.
August 21st.—The termination of the fast of Felsat was hailed with
considerable pleasure by the very best of Christians in Shoa, and this
happened on the last day of the interval which dates this chapter, for
increasing want of space compels me to relinquish the usual diurnal
account. As it happened, nothing of importance occurred, except
only that I began to find myself gradually getting weaker and
weaker, and the symptoms of my illness increasing in violence upon
every fresh attack. I attributed this, in a great measure, to the wet
season, which was now most decidedly set in, and for the last three
or four days especially it had rained without intermission. It was a
cheerless time, the moist foggy state of the atmosphere, and the
muddy condition of the road, quite prevented me from taking my
usual walks, and looking at the dripping state of my thatched roof,
or listening to the pattering of the large rain drops against my
parchment window, was all the amusement I had after I had
determined to confine myself more to bed, either to recruit myself
after the severe fever paroxysms, or with the hope of averting in
some measure the force of their attack by a little careful nursing. I
ceased, too, to take any pleasure in the interesting conversations of
Ibrahim, or Sheik Tigh, or, in fact, any one from whom previously I
had ever been most inquiring respecting every subject of importance
or novelty I could think of to ask about. My cup of coffee in the
morning, or a drinking hornful of the warmed ale, was the only thing
I dare indulge in, for solids of any kind had a great tendency to
occasion congestion in the brain, and after eating anything a severe
headache was the certain consequence. I took the hint, and gave up
the honeyed repast at breakfast and the fricassee at night, and
made a point of conciliating as much as possible that irritable viscus
the stomach, that seemed after all to be at the bottom of the evil.
It was a horrible retribution, therefore, for Walderheros to
contemplate, and which, no doubt, will have a beneficial effect upon
his future conduct as regards the respect due to the institutions of
the Church, when, on the morning after the end of the fast, which
was to be a day of great rejoicing, I intimated my determination to
eat no more flesh meat for at least a week. After all my jests about
the folly of fasting, telling my people that during the continuance of
such terms of abstinence, I was a good Mahomedan, and having by
my example on more than one occasion, tempted him and the
others of my household to indulge in food when they ought to have
been observing a stricter discipline—after all this, on the day
appointed for rejoicing, to see all appetite taken from me was so
evidently a judgment from heaven, that I was strongly
recommended to propitiate the Virgin Mary immediately by
bestowing the goat, which the Negoos had sent to me at Myolones,
upon a church dedicated to the Mother of Christ. So disinterested in
fact, was Walderheros, that he went off to procure one of the priests
upon the establishment, and who, when he arrived, had I carried out
my servant’s intentions, would have walked away with the goat
immediately, such was his anxiety for my restoration to the favour of
heaven.
I could not be very well angry with Walderheros, and I was too ill
either to laugh at, or to endeavour to convert the priest, so I
dismissed him with an ahmulah, for his willingness to relieve me of
the supposed ban under which I was laid. When he was gone,
however, the weather having cleared up a little, I directed my
servants to kill the goat, and to ask such of their Christian friends
who lived in Aliu Amba to come to the party on the occasion, as I
wanted it eaten up, that no temptation should exist to divert me
from my resolution not to take any animal food.
 The best butcher in the place, Tinta’s misselannee, who had
always shown himself ready to render assistance whenever I
required some extra hand, could not, of course, be omitted. Gwalior,
another servant of Tinta, and a patient of mine, was also called in at
the death of the doomed goat, which gallantly showed fight,
surrounded, as he was, by a host of hungry enemies, who, besides
seeking the satisfaction of revenge for the indiscriminate tuppings
and bumpings he had given and occasioned among the party, had
had their interest excited by the portions of his venison mutton, that
each, in the mind’s eye, already saw hanging up in a mimosa tree
that grew in my garden, and which formed the shambles generally
on such occasions.
A lot of yelping boys came into the enclosure, and crowded about
the butchers aiding the goat in his attempts to get away, by
attempting to catch him, and of course running in the way of those
who might have been able to do it. A number of women also
thronged in as the stir became faster, and who stood around me as
a kind of body-guard, for the leaping “diabolus” of a goat sometimes
threatened even to make our heads a stepping-stone to fly over the
high enclosure. A long lasso at length being thrown ignobly at his
feet, the next move he made ensnared him by the leg, and the
triumph of his life-hunters was complete. The rope being run around
the trunk of the mimosa, the unwilling goat was dragged, like a
victim of Spanish civil war, backwards to his doom, and a prayer of
peace being muttered by the clerk, Walderheros, the high priest, the
misselannee, cut the throat of their prey, the invocation of the
Trinity, like the Islam “ul Allah,” sanctifying the bloody business of
depriving an animal of life.
It is singular to observe the pertinacity of custom, and how
characteristic of descent particular habits and ceremonies become
long after the separation of different nations from their original root.
The Arabs, the Amhara, or the Abyssinians, and the Jews, all
precede the slaughter of animals for food with some short prayer,
which, differing in form, is still the same custom, and which, I think,
originated at a period antecedent to their dispersion as different
nations into the several countries they now occupy. It has also
continued among them, even changed as these nations are in
religion and social character, the Hebrew trader, the Arab nomade
shepherd, and the Abyssinian agriculturist. Jew, Mahomedan, and
Christian, still retain this evidence of a common origin, but which
marks an ethnological era posterior, I believe, by many centuries to
the more general custom of circumcision common to all these
people, and to many other African nations.[10]
Such a goat as had just been killed, fed up to the high condition
he was in, could not have been bought in the market for less than
ten ahmulahs, two shillings and twopence. The skin alone, however,
is supposed to be worth three ahmulahs; and great care is taken not
to injure it with the point of the knife, when flaying the carcase. To
be of any value, it must be taken off uncut, except around the neck,
and in those situations necessary to enable the butchers to draw the
legs out of the skin. Also, of course, where the first incision is made
to commence the process, and which is a circular cut carried around
both haunches, not many inches from and having the tail for a
centre. The hide is then stripped over the thighs, and two smaller
incisions being made around the middle joint of the hind legs, enable
them to be drawn out. A stick is now placed to extend these
extremities, and by this, for the convenience of the operators, the
whole carcase is suspended from the branch of a tree, and by some
easy pulls around the body, the skin is gradually withdrawn over the
forelegs, which are incised around the knees to admit of their being
taken out; after which, the head being removed, the whole business
concludes by the skin being pulled inside out over the decollated
neck. One of the parties now takes a rough stone and well rubs the
inside surface, to divest it of a few fibres of the subcutaneous
muscle which are inserted into the skin, and after this operation it is
laid aside until the next day; the more interesting business of
attending to the meat calling immediate attention.
These entire skins are afterwards made into sacks by the
apertures around the neck and legs being secured by a double fold
of the skin being sewed upon each other, by means of a slender but
very tough thong. These small seams are rendered quite air-tight,
and the larger orifice around the haunches being gathered together
by the hands, the yet raw skin is distended with air, and the orifice
being then tied up, the swollen bag is left in that state for a few
days until slight putrefaction has commenced, when the application
of the rough stone soon divests its surface of the hair. After this has
been effected, a deal of labour, during at least one day, is required
to soften the distended skin by beating it with heavy sticks, or
trampling upon it for hours together, the labourer supporting himself
by clinging to the bough of a tree over head, or holding on by the
wall of the house. In this manner, whilst the skin is drying, it is
prevented from getting stiff, and still further to secure it from this
evil condition, it is frequently rubbed with small quantities of butter.
When it is supposed that there is no chance of the skin becoming
hard and easily broken, the orifice is opened, the air escapes, and a
very soft flaccid leather bag is produced, but which, for several days
after, affords an amusement to the owner, when otherwise
unemployed, by well rubbing it all over with his hands.
Almost all the produce of the fields is conveyed to the market in
such sacks as these—cotton, grain, and the Berberah pepper. It is
even the only moneybag employed to carry home the salt returns for
the different wares that have been sold. None other could have been
employed by Joseph’s brethren when they loaded their asses and
went down into Egypt; for none are more naturally the resources of
a shepherd people, or better adapted by their form and size for the
little useful animal which seems to have been as universally
employed by the Jews as by the Amhara of the present day. By a
species of gratitude, sincere as it is deserved, hiyah, the word
signifying ass, is used by the latter people as another designation for
friend; and I well remember the mistake of a learner of that
language who went into a great rage by being accosted “hiyah” by
an Amhara friend.
The skins of sheep and of small goats are made into parchment
by being more particularly divested of the fleshy fibres with the
rough stone, and then, after the hairs have been removed by
putrefaction, simply drying in the sun. For this purpose, it is
stretched in a favourable situation, a few inches from the ground, by
a number of small wooden pegs, which are inserted into small
apertures made in the edge of the skin, and it is thus prevented
from becoming corrugated during the process of drying.
In the same manner, the larger hides of cows and oxen are dried,
most frequently before putrefaction has produced any effect upon
the hairs, and which, of course, then remain. This is the general seat
for visitors during the day, and their bed at night, unless a tanned
hide (nit, as it is termed) can be procured, and which is considered
softer and more suitable for a respected guest.
The nit, or leather, is tanned by being made into a kind of trough,
which contains an infusion of the bark of the kantuffa acacia. This
trough is formed by a skin being loosely extended upon four stick
supports, which elevate it about a foot from the ground. The
kantuffa bark, after being well pounded in a mortar, is strewed over
the surface, and the hollow is then filled with cold water, and in the
course of a few days a strong red infusion is made, with which the
whole surface of the skin is frequently washed, and when
evaporation has reduced its contents to a sloppy paste, the sticks
are withdrawn, the ends folded in, and with the contained mass, the
skin then undergoes the usual fatiguing process of treading, until the
evidences of the nit being properly prepared are satisfactory.
The bark of the kantuffa reminded me of that of the red mimosa
of Adal, which produced an astringent gum, something like kino, but
not, I considered, so powerful a drug. This tree, however, was
pointed out to me as being that with the bark of which the Dankalli
tan their affaleetahs, or small water-skins, carried by travellers; for
the larger ones are prepared with the hair left on, by simply drying
in the hot sun, after having been distended with air, to expose them
fully to its influence. It is very probable that the celebrated Morocco
leather, derives its bright red colour from the bark employed in
tanning being obtained either from the kantuffa or the Adal tree, for
both these trees give a very red colour to the skins that are prepared
with their bark. From this I am inclined to believe, that among other
articles of commerce that might be advantageously drawn from the
Barbar states in the north of Africa, a good tanning bark could be
obtained in considerable quantities, and at a very reasonable rate.
Walderheros and the misselannee proceeded to carve the flayed
carcase, not in any systematic manner, as I could observe, but
directed chiefly in the size of the lumps of meat that were cut off by
the character of the individual to whom they were severally
assigned: thus, Tinta got a noble haunch forwarded to him, whilst,
on the other hand, the matrabier, or axe, was called in to aid in
dividing the other into three portions, for as many minor
acquaintances of my servants. In the same manner, a certain
number of ribs were counted for Gwalior, but the mother of
Goodaloo got a great many more of the opposite side, and in this
irregular manner, after a very busy scene of some two or three
hours, except the portions which Walderheros had retained for
himself, the whole of the goat had disappeared by degrees through
the wicket of the inclosure, for the rain that was now commencing
prevented the party from holding the festival in the garden, and I
was a great deal too ill to have it celebrated within my own house.

FOOTNOTES:
[10] A singular fact connected with this custom of making a short
prayer, whilst slaughtering the victim, I gathered from a note in a
recent edition of “Sale’s Koran.” It appears that by a decision of
those learned in the law, which is laid down in that book, animals
killed by the Jews may be partaken of by Mahomedans. A
representation to the Cadi of Cairo having been made, that nearly
all the butchers of that city were followers of the law of Moses,
they were about to be suspended from that employment, when
their Chief Rabbi proved to the satisfaction of their Moslem
judges, that the Koran bids Mahomedans not to refuse food which
has been sanctified to the one true God, which was always done
by those who professed the faith of Abraham and the law of
Moses, when killing animals for food. This circumstance, and also
the disrespect shown by the Whaabbees to the tomb of the
Prophet, and the temple at Mecca, demonstrate to my satisfaction
that education alone is required to show to the Mahomedans, the
absurdity of the false hopes with which their Prophet has
surrounded the worship of the only one God, and of the
inapplicability of his laws to improve or humanize mankind. I
could point out, if this were a proper place, proofs without end, of
the liberality and extreme toleration of learned and enlightened
Mahomedans, and we ought not to attribute the bigotry of
ignorance, alike fierce and cruel in Christian, Mahomedan, and
Jew, to their religious belief, which on examination will be found
to have been originally very similar amongst all these
denominations, and that the greatest differences appear to be in
the several codes of social laws adopted by each.