Download the full version of the textbook now at textbookfull.

com

Release it design and deploy production ready

sofware Nygard

https://textbookfull.com/product/release-it-
design-and-deploy-production-ready-sofware-nygard/

Explore and download more textbook at https://textbookfull.com

 Recommended digital products (PDF, EPUB, MOBI) that
you can download immediately if you are interested.

Release It Design and Deploy Production Ready Software 2nd

Edition Michael T. Nygard

https://textbookfull.com/product/release-it-design-and-deploy-
production-ready-software-2nd-edition-michael-t-nygard/

textbookfull.com

Applied OpenStack Design Patterns Design solutions for

production ready infrastructure with OpenStack components
1st Edition Uchit Vyas (Auth.)
https://textbookfull.com/product/applied-openstack-design-patterns-
design-solutions-for-production-ready-infrastructure-with-openstack-
components-1st-edition-uchit-vyas-auth/
textbookfull.com

Docker in the Trenches Successful Production Deployment 1

(Early Release) Edition Joe Johnston

https://textbookfull.com/product/docker-in-the-trenches-successful-
production-deployment-1-early-release-edition-joe-johnston/

textbookfull.com

The English Grammar Workbook for Adults A Self Study Guide

to Improve Functional Writing First Edition Michael
Digiacomo
https://textbookfull.com/product/the-english-grammar-workbook-for-
adults-a-self-study-guide-to-improve-functional-writing-first-edition-
michael-digiacomo/
textbookfull.com
Private Data and Public Value Governance Green Consumption
and Sustainable Supply Chains 1st Edition Holly Jarman

https://textbookfull.com/product/private-data-and-public-value-
governance-green-consumption-and-sustainable-supply-chains-1st-
edition-holly-jarman/
textbookfull.com

Financial accounting : with International Financial

Reporting Standards (ch1-9 only) Kieso

https://textbookfull.com/product/financial-accounting-with-
international-financial-reporting-standards-ch1-9-only-kieso/

textbookfull.com

Pronouns in Literature: Positions and Perspectives in

Language 1st Edition Alison Gibbons

https://textbookfull.com/product/pronouns-in-literature-positions-and-
perspectives-in-language-1st-edition-alison-gibbons/

textbookfull.com

Transnational Commercial Surrogacy and the Un Making of

Kin in India Anindita Majumdar

https://textbookfull.com/product/transnational-commercial-surrogacy-
and-the-un-making-of-kin-in-india-anindita-majumdar/

textbookfull.com

Oral and Maxillofacial Surgery (Oxford Specialist

Handbooks in Surgery) Carrie Newlands

https://textbookfull.com/product/oral-and-maxillofacial-surgery-
oxford-specialist-handbooks-in-surgery-carrie-newlands/

textbookfull.com
The Eagle and the Trident The United States and Ukraine
Steven Pifer

https://textbookfull.com/product/the-eagle-and-the-trident-the-united-
states-and-ukraine-steven-pifer/

textbookfull.com
 Early praise for Release It! Second Edition

Mike is one of the software industry’s deepest thinkers and clearest communica-
tors. As beautifully written as the original, the second edition of Release It! extends
the first with modern techniques—most notably continuous deployment, cloud
infrastructure, and chaos engineering—that will help us all build and operate
large-scale software systems.
➤ Randy Shoup
VP Engineering, Stitch Fix

If you are putting any kind of system into production, this is the single most im-
portant book you should keep by your side. The author’s enormous experience
in the area is captured in an easy-to-read, but still very intense, way. In this up-
dated edition, the new ways of developing, orchestrating, securing, and deploying
real-world services to different fabrics are well explained in the context of the core
resiliency patterns.
➤ Michael Hunger
Director of Developer Relations Engineering, Neo4j, Inc.

So much ground is covered here: patterns and antipatterns for application re-
silience, security, operations, architecture. That breadth would be great in itself,
but there’s tons of depth too. Don’t just read this book—study it.
➤ Colin Jones
CTO at 8th Light and Author of Mastering Clojure Macros
Release It! is required reading for anyone who wants to run software to production
and still sleep at night. It will help you build with confidence and learn to expect
and embrace system failure.
➤ Matthew White
Author of Deliver Audacious Web Apps with Ember 2

I would recommend this book to anyone working on a professional software project.

Given that this edition has been fully updated to cover technologies and topics
that are dealt with daily, I would expect everyone on my team to have a copy of
this book to gain awareness of the breadth of topics that must be accounted for
in modern-day software development.
➤ Andy Keffalas
Software Engineer/Team Lead

A must-read for anyone wanting to build truly robust, scalable systems.

➤ Peter Wood
Software Programmer
Release It! Second Edition
Design and Deploy Production-Ready Software

Michael T. Nygard

The Pragmatic Bookshelf

Raleigh, North Carolina
Many of the designations used by manufacturers and sellers to distinguish their products
are claimed as trademarks. Where those designations appear in this book, and The Pragmatic
Programmers, LLC was aware of a trademark claim, the designations have been printed in
initial capital letters or in all capitals. The Pragmatic Starter Kit, The Pragmatic Programmer,
Pragmatic Programming, Pragmatic Bookshelf, PragProg and the linking g device are trade-
marks of The Pragmatic Programmers, LLC.
Every precaution was taken in the preparation of this book. However, the publisher assumes
no responsibility for errors or omissions, or for damages that may result from the use of
information (including program listings) contained herein.
Our Pragmatic books, screencasts, and audio books can help you and your team create
better software and have more fun. Visit us at https://pragprog.com.

The team that produced this book includes:

Publisher: Andy Hunt
VP of Operations: Janet Furlow
Managing Editor: Brian MacDonald
Supervising Editor: Jacquelyn Carter
Development Editor: Katharine Dvorak
Copy Editor: Molly McBeath
Indexing: Potomac Indexing, LLC
Layout: Gilson Graphics

For sales, volume licensing, and support, please contact [email protected].

For international rights, please contact [email protected].

Copyright © 2018 The Pragmatic Programmers, LLC.

All rights reserved.

No part of this publication may be reproduced, stored in a retrieval system, or transmitted,

in any form, or by any means, electronic, mechanical, photocopying, recording, or otherwise,
without the prior consent of the publisher.

Printed in the United States of America.

ISBN-13: 978-1-68050-239-8
Encoded using the finest acid-free high-entropy binary digits.
Book version: P1.0—January 2018
 Contents

Acknowledgments . . . . . . . . . . . xi

Preface . . . . . . . . . . . . . . xiii

1. Living in Production . . . . . . . . . . . 1
Aiming for the Right Target 2
The Scope of the Challenge 3
A Million Dollars Here, a Million Dollars There 3
Use the Force 4
Pragmatic Architecture 5
Wrapping Up 6

Part I — Create Stability

2. Case Study: The Exception That Grounded an Airline . . . 9
The Change Window 10
The Outage 12
Consequences 14
Postmortem 14
Hunting for Clues 16
The Smoking Gun 18
An Ounce of Prevention? 20

3. Stabilize Your System . . . . . . . . . . 23

Defining Stability 24
Extending Your Life Span 25
Failure Modes 26
Stopping Crack Propagation 27
Chain of Failure 28
Wrapping Up 30
 Contents • vi

4. Stability Antipatterns . . . . . . . . . . 31
Integration Points 33
Chain Reactions 46
Cascading Failures 49
Users 51
Blocked Threads 62
Self-Denial Attacks 69
Scaling Effects 71
Unbalanced Capacities 75
Dogpile 78
Force Multiplier 80
Slow Responses 84
Unbounded Result Sets 86
Wrapping Up 90

5. Stability Patterns . . . . . . . . . . . 91
Timeouts 91
Circuit Breaker 95
Bulkheads 98
Steady State 101
Fail Fast 106
Let It Crash 108
Handshaking 111
Test Harnesses 113
Decoupling Middleware 117
Shed Load 119
Create Back Pressure 120
Governor 123
Wrapping Up 125

Part II — Design for Production

6. Case Study: Phenomenal Cosmic Powers,
Itty-Bitty Living Space . . . . . . . . 129
Baby’s First Christmas 130
Taking the Pulse 131
Thanksgiving Day 132
Black Friday 132
Vital Signs 134
Diagnostic Tests 135
 Contents • vii

Call In a Specialist 136

Compare Treatment Options 137
Does the Condition Respond to Treatment? 138
Winding Down 139

7. Foundations . . . . . . . . . . . . 141
Networking in the Data Center and the Cloud 142
Physical Hosts, Virtual Machines, and Containers 146
Wrapping Up 153

8. Processes on Machines . . . . . . . . . . 155

Code 157
Configuration 160
Transparency 162
Wrapping Up 170

9. Interconnect . . . . . . . . . . . . 171
Solutions at Different Scales 172
DNS 173
Load Balancing 177
Demand Control 182
Network Routing 186
Discovering Services 188
Migratory Virtual IP Addresses 189
Wrapping Up 191

10. Control Plane . . . . . . . . . . . . 193

How Much Is Right for You? 193
Mechanical Advantage 194
Platform and Ecosystem 197
Development Is Production 199
System-Wide Transparency 200
Configuration Services 206
Provisioning and Deployment Services 207
Command and Control 209
The Platform Players 212
The Shopping List 213
Wrapping Up 213

11. Security . . . . . . . . . . . . . 215

The OWASP Top 10 216
The Principle of Least Privilege 231
 Contents • viii

Configured Passwords 232

Security as an Ongoing Process 233
Wrapping Up 233

Part III — Deliver Your System

12. Case Study: Waiting for Godot . . . . . . . . 237

13. Design for Deployment . . . . . . . . . . 241

So Many Machines 241
The Fallacy of Planned Downtime 242
Automated Deployments 242
Continuous Deployment 246
Phases of Deployment 248
Deploy Like the Pros 261
Wrapping Up 261

14. Handling Versions . . . . . . . . . . . 263

Help Others Handle Your Versions 263
Handle Others’ Versions 270
Wrapping Up 273

Part IV — Solve Systemic Problems

15. Case Study: Trampled by Your Own Customers . . . . 277
Countdown and Launch 277
Aiming for Quality Assurance 278
Load Testing 281
Murder by the Masses 284
The Testing Gap 285
Aftermath 286

16. Adaptation . . . . . . . . . . . . . 289

Convex Returns 289
Process and Organization 290
System Architecture 301
Information Architecture 313
Wrapping Up 324

17. Chaos Engineering . . . . . . . . . . . 325

Breaking Things to Make Them Better 325
Visit https://textbookfull.com
now to explore a rich
collection of eBooks, textbook
and enjoy exciting offers!
 Contents • ix

Antecedents of Chaos Engineering 326

The Simian Army 328
Adopting Your Own Monkey 329
Disaster Simulations 335
Wrapping Up 336

Bibliography . . . . . . . . . . . . 337

Index . . . . . . . . . . . . . . 339
 Acknowledgments
I’d like to say a big thank you to the many people who have read and shared
the first edition of Release It! I’m deeply happy that so many people have
found it useful.

Over the years, quite a few people have nudged me about updating this book.
Thank you to Dion Stewart, Dave Thomas, Aino Corry, Kyle Larsen, John
Allspaw, Stuart Halloway, Joanna Halloway, Justin Gehtland, Rich Hickey,
Carin Meier, John Willis, Randy Shoup, Adrian Cockroft, Gene Kim, Dan
North, Stefan Tilkov, and everyone else who saw that a few things had changed
since we were building monoliths in 2006.

Thank you to all my technical reviewers: Adrian Cockcroft, Rod Hilton, Michael
Hunger, Colin Jones, Andy Keffalas, Chris Nixon, Antonio Gomes Rodrigues,
Stefan Turalski, Joshua White, Matthew White, Stephen Wolff, and Peter
Wood. Your efforts and feedback have helped make this book much better.

Thanks also to Nora Jones and Craig Andera for letting me include your stories
in these pages. The war stories have always been one of my favorite parts of
the book, and I know many readers feel the same way.

Finally, a huge thank you to Andy Hunt, Katharine Dvorak, Susannah

Davidson Pfalzer, and the whole team at The Pragmatic Bookshelf. I appreciate
your patience and perseverance.

report erratum • discuss

 Preface
In this book, you will examine ways to architect, design, and build software
—particularly distributed systems—for the muck and mire of the real world.
You will prepare for the armies of illogical users who do crazy, unpredictable
things. Your software will be under attack from the moment you release it.
It needs to stand up to the typhoon winds of flash mobs or the crushing
pressure of a DDoS attack by poorly secured IoT toaster ovens. You’ll take a
hard look at software that failed the test and find ways to make sure your
software survives contact with the real world.

Who Should Read This Book

I’ve targeted this book to architects, designers, and developers of distributed
software systems, including websites, web services, and EAI projects, among
others. These must be available or the company loses money. Maybe they’re
commerce systems that generate revenue directly through sales or critical
internal systems that employees use to do their jobs. If anybody has to go home
for the day because your software stops working, then this book is for you.

How This Book Is Organized

The book is divided into four parts, each introduced by a case study. Part I:
Create Stability shows you how to keep your systems alive, maintaining system
uptime. Despite promises of reliability through redundancy, distributed systems
exhibit availability more like “two eights” rather than the coveted “five nines.”
Stability is a necessary prerequisite to any other concerns. If your system falls
over and dies every day, nobody cares about anything else. Short-term fixes—
and short-term thinking—will dominate in that environment. There’s no viable
future without stability, so we’ll start by looking at ways to make a stable base.

After stability, the next concern is ongoing operations. In Part II: Design for
Production, you’ll see what it means to live in production. You’ll deal with the
complexity of modern production environments in all their virtualized, con-
tainerized, load-balanced, service-discovered gory detail. This part illustrates

report erratum • discuss

 Preface • xiv

good patterns for control, transparency, and availability in physical data

centers and cloud environments.

In Part III: Deliver Your System, you’ll look at deployments. There are great
tools for pouring bits onto servers now, but that turns out to be the easy part
of the problem. It’s much harder to push frequent, small changes without
breaking consumers. We’ll look at design for deployment and at deployments
without downtime, and then we’ll move into versioning across disparate ser-
vices—always a tricky issue!

In Part IV: Solve Systemic Problems, you’ll examine the system’s ongoing life
as part of the overall information ecosystem. If release 1.0 is the birth of the
system, then you need to think about its growth and development after that.
In this part, you’ll see how to build systems that can grow, flex, and adapt
over time. This includes evolutionary architecture and shared “knowledge”
across systems. Finally, you’ll learn how to build antifragile systems through
the emerging discipline of “chaos engineering” that uses randomness and
deliberate stress on a system to improve it.

About the Case Studies

I included several extended case studies to illustrate the major themes of this
book. These case studies are taken from real events and real system failures
that I have personally observed. These failures were very costly and embar-
rassing for those involved. Therefore, I obfuscated some information to protect
the identities of the companies and people involved. I also changed the names
of the systems, classes, and methods. Only such nonessential details have
been changed, however. In each case, I maintained the same industry,
sequence of events, failure mode, error propagation, and outcome. The costs
of these failures are not exaggerated. These are real companies, and this is
real money. I preserved those figures to underscore the seriousness of this
material. Real money is on the line when systems fail.

Online Resources
This book has its own web page,1 where you can find details about it, download
the source code, post to the discussion forums, and report errata such as
typos and content suggestions. The discussion forums are the perfect place
to talk shop with other readers and share your comments about the book.

Now, let’s get started with an introduction to living in production.

1. https://pragprog.com/titles/mnee2/46

report erratum • discuss

 CHAPTER 1

Living in Production
You’ve worked hard on your project. It looks like all the features are actu-
ally complete, and most even have tests. You can breathe a sigh of relief.
You’re done.

Or are you?

Does “feature complete” mean “production ready”? Is your system really ready
to be deployed? Can it be run by operations and face the hordes of real-world
users without you? Are you starting to get that sinking feeling that you’ll be
faced with late-night emergency phone calls and alerts? It turns out there’s
a lot more to development than just adding all the features.

Software design as taught today is terribly incomplete. It only talks about

what systems should do. It doesn’t address the converse—what systems
should not do. They should not crash, hang, lose data, violate privacy, lose
money, destroy your company, or kill your customers.

Too often, project teams aim to pass the quality assurance (QA) department’s
tests instead of aiming for life in production. That is, the bulk of your work
probably focuses on passing testing. But testing—even agile, pragmatic,
automated testing—is not enough to prove that software is ready for the real
world. The stresses and strains of the real world, with crazy real users, globe-
spanning traffic, and virus-writing mobs from countries you’ve never even
heard of go well beyond what you could ever hope to test for.

But first, you will need to accept the fact that despite your best laid plans,
bad things will still happen. It’s always good to prevent them when possible,
of course. But it can be downright fatal to assume that you’ve predicted and
eliminated all possible bad events. Instead, you want to take action and pre-
vent the ones you can but make sure that your system as a whole can
recover from whatever unanticipated, severe traumas might befall it.

report erratum • discuss

 Chapter 1. Living in Production •2

Aiming for the Right Target

Most software is designed for the development lab or the testers in the QA
department. It is designed and built to pass tests such as, “The customer’s
first and last names are required, but the middle initial is optional.” It aims
to survive the artificial realm of QA, not the real world of production.

Software design today resembles automobile design in the early ’90s—discon-

nected from the real world. Cars designed solely in the cool comfort of the lab
looked great in models and CAD systems. Perfectly curved cars gleamed in
front of giant fans, purring in laminar flow. The designers inhabiting these
serene spaces produced designs that were elegant, sophisticated, clever,
fragile, unsatisfying, and ultimately short-lived. Most software architecture
and design happens in equally clean, distant environs.

Do you want a car that looks beautiful but spends more time in the shop
than on the road? Of course not! You want to own a car designed for the real
world. You want a car designed by somebody who knows that oil changes are
always 3,000 miles late, that the tires must work just as well on the last
sixteenth of an inch of tread as on the first, and that you will certainly, at
some point, stomp on the brakes while holding an Egg McMuffin in one hand
and a phone in the other.

When our system passes QA, can we say with confidence that it’s ready for
production? Simply passing QA tells us little about the system’s suitability
for the next three to ten years of life. It could be the Toyota Camry of software,
racking up thousands of hours of continuous uptime. Or it could be the Chevy
Vega (a car whose front end broke off on the company’s own test track) or the
Ford Pinto (a car prone to blowing up when hit in just the right way). It’s
impossible to tell from a few days or even a few weeks of testing what the next
several years will bring.

Product designers in manufacturing have long pursued “design for manufac-

turability”—the engineering approach of designing products such that they
can be manufactured at low cost and high quality. Prior to this era, product
designers and fabricators lived in different worlds. Designs thrown over the
wall to production included screws that could not be reached, parts that were
easily confused, and custom parts where off-the-shelf components would
serve. Inevitably, low quality and high manufacturing cost followed.

We’re in a similar state today. We end up falling behind on the new system
because we’re constantly taking support calls from the last half-baked project
we shoved out the door. Our analog of “design for manufacturability” is “design

report erratum • discuss

 The Scope of the Challenge •3

for production.” We don’t hand designs to fabricators, but we do hand finished

software to IT operations. We need to design individual software systems, and
the whole ecosystem of interdependent systems, to operate at low cost and
high quality.

The Scope of the Challenge

In the easy, laid-back days of client/server systems, a system’s user base
would be measured in the tens or hundreds, with a few dozen concurrent
users at most. Today we routinely see active user counts larger than the
population of entire continents. And I’m not just talking about Antarctica and
Australia here! We’ve seen our first billion-user social network, and it won’t
be the last.

Uptime demands have increased too. Whereas the famous “five nines” (99.999
percent) uptime was once the province of the mainframe and its caretakers,
even garden-variety commerce sites are now expected to be available 24 by
7 by 365. (That phrase has always bothered me. As an engineer, I expect it
to either be “24 by 365” or be “24 by 7 by 52.”) Clearly, we’ve made tremendous
strides even to consider the scale of software built today; but with the
increased reach and scale of our systems come new ways to break, more
hostile environments, and less tolerance for defects.

The increasing scope of this challenge—to build software fast that’s cheap to
build, good for users, and cheap to operate—demands continually improving
architecture and design techniques. Designs appropriate for small WordPress
websites fail outrageously when applied to large scale, transactional, distribut-
ed systems, and we’ll look at some of those outrageous failures.

A Million Dollars Here, a Million Dollars There

A lot is on the line here: your project’s success, your stock options or profit
sharing, your company’s survival, and even your job. Systems built for QA
often require so much ongoing expense, in the form of operations cost,
downtime, and software maintenance, that they never reach profitability, let
alone net positive cash for the business (reached only after the profits gener-
ated by the system pay back the costs incurred in building it.) These systems
exhibit low availability, direct losses in missed revenue, and indirect losses
through damage to the brand.

During the hectic rush of a development project, you can easily make decisions
that optimize development cost at the expense of operational cost. This makes
sense only in the context of the team aiming for a fixed budget and delivery

report erratum • discuss

 Chapter 1. Living in Production •4

date. In the context of the organization paying for the software, it’s a bad
choice. Systems spend much more of their life in operation than in develop-
ment—at least, the ones that don’t get canceled or scrapped do. Avoiding a
one-time developmental cost and instead incurring a recurring operational
cost makes no sense. In fact, the opposite decision makes much more financial
sense. Imagine that your system requires five minutes of downtime on every
release. You expect your system to have a five-year life span with monthly
releases. (Most companies would like to do more releases per year, but I’m
being very conservative.) You can compute the expected cost of downtime, dis-
counted by the time-value of money. It’s probably on the order of $1,000,000
(300 minutes of downtime at a very modest cost of $3,000 per minute).

Now suppose you could invest $50,000 to create a build pipeline and
deployment process that avoids downtime during releases. That will, at a
minimum, avoid the million-dollar loss. It’s very likely that it will also allow
you to increase deployment frequency and capture market share. But let’s
stick with the direct gain for now. Most CFOs would not mind authorizing an
expenditure that returns 2,000 percent ROI!

Design and architecture decisions are also financial decisions. These choices
must be made with an eye toward their implementation cost as well as their
downstream costs. The fusion of technical and financial viewpoints is one of
the most important recurring themes in this book.

Use the Force

Your early decisions make the biggest impact on the eventual shape of your
system. The earliest decisions you make can be the hardest ones to reverse
later. These early decisions about the system boundary and decomposition
into subsystems get crystallized into the team structure, funding allocation,
program management structure, and even time-sheet codes. Team assignments
are the first draft of the architecture. It’s a terrible irony that these very early
decisions are also the least informed. The beginning is when your team is
most ignorant of the eventual structure of the software, yet that’s when some
of the most irrevocable decisions must be made.

I’ll reveal myself here and now as a proponent of agile development. The
emphasis on early delivery and incremental improvements means software
gets into production quickly. Since production is the only place to learn how
the software will respond to real-world stimuli, I advocate any approach that
begins the learning process as soon as possible. Even on agile projects, deci-
sions are best made with foresight. It seems as if the designer must “use the
force” to see the future in order to select the most robust design. Because

report erratum • discuss

 Pragmatic Architecture •5

different alternatives often have similar implementation costs but radically

different life-cycle costs, it is important to consider the effects of each decision
on availability, capacity, and flexibility. I’ll show you the downstream effects
of dozens of design alternatives, with concrete examples of beneficial and
harmful approaches. These examples all come from real systems I’ve worked
on. Most of them cost me sleep at one time or another.

Pragmatic Architecture
Two divergent sets of activities both fall under the term architecture. One type
of architecture strives toward higher levels of abstraction that are more portable
across platforms and less connected to the messy details of hardware, networks,
electrons, and photons. The extreme form of this approach results in the “ivory
tower”—a Kubrick-esque clean room inhabited by aloof gurus and decorated
with boxes and arrows on every wall. Decrees emerge from the ivory tower and
descend upon the toiling coders. “The middleware shall be JBoss, now and
forever!” “All UIs shall be constructed with Angular 1.0!” “All that is, all that
was, and all that shall ever be lives in Oracle!” “Thou shalt not engage in Ruby!”
If you’ve ever gritted your teeth while coding something according to the “com-
pany standards” that would be ten times easier with some other technology,
then you’ve been the victim of an ivory-tower architect. I guarantee that an
architect who doesn’t bother to listen to the coders on the team doesn’t bother
listening to the users either. You’ve seen the result: users who cheer when the
system crashes because at least then they can stop using it for a while.

In contrast, another breed of architect doesn’t just rub shoulders with the
coders but is one. This kind of architect does not hesitate to peel back the lid
on an abstraction or to jettison one if it doesn’t fit. This pragmatic architect
is more likely to discuss issues such as memory usage, CPU requirements,
bandwidth needs, and the benefits and drawbacks of hyperthreading and
CPU binding.

The ivory-tower architect most enjoys an end-state vision of ringing crystal

perfection, but the pragmatic architect constantly thinks about the dynamics
of change. “How can we do a deployment without rebooting the world?” “What
metrics do we need to collect, and how will we analyze them?” “What part of
the system needs improvement the most?” When the ivory-tower architect is
done, the system will not admit any improvements; each part will be perfectly
adapted to its role. Contrast that to the pragmatic architect’s creation, in
which each component is good enough for the current stresses—and the
architect knows which ones need to be replaced depending on how the stress
factors change over time.

report erratum • discuss

 Chapter 1. Living in Production •6

If you’re already a pragmatic architect, then I’ve got chapters full of powerful
ammunition for you. If you’re an ivory-tower architect—and you haven’t already
stopped reading—then this book might entice you to descend through a few
levels of abstraction to get back in touch with that vital intersection of soft-
ware, hardware, and users: living in production. You, your users, and your
company will be much happier when the time comes to finally release it!

Wrapping Up
Software delivers its value in production. The development project, testing,
integration, and planning...everything before production is prelude. This
book deals with life in production, from the initial release through ongoing
growth and evolution of the system. The first part of this book deals with
stability. To get a better sense of the kind of issues involved in keeping your
software from crashing, let’s start by looking at the software bug that
grounded an airline.

report erratum • discuss

Visit https://textbookfull.com
now to explore a rich
collection of eBooks, textbook
and enjoy exciting offers!
 Part I

Create Stability
 CHAPTER 2

Case Study:
The Exception That Grounded an Airline
Have you ever noticed that the incidents that blow up into the biggest issues
start with something very small? A tiny programming error starts the snowball
rolling downhill. As it gains momentum, the scale of the problem keeps getting
bigger and bigger. A major airline experienced just such an incident. It even-
tually stranded thousands of passengers and cost the company hundreds of
thousands of dollars. Here’s how it happened.

As always, all names, places, and dates have been changed to protect the
confidentiality of the people and companies involved.

It started with a planned failover on the database cluster that served the core
facilities (CF). The airline was moving toward a service-oriented architecture,
with the usual goals of increasing reuse, decreasing development time, and
decreasing operational costs. At this time, CF was in its first generation. The
CF team planned a phased rollout, driven by features. It was a sound plan,
and it probably sounds familiar—most large companies have some variation
of this project underway now.

CF handled flight searches—a common service for any airline application.

Given a date, time, city, airport code, flight number, or any combination
thereof, CF could find and return a list of flight details. When this incident
happened, the self-service check-in kiosks, phone menus, and “channel
partner” applications had been updated to use CF. Channel partner applica-
tions generate data feeds for big travel-booking sites. IVR and self-service
check-in are both used to put passengers on airplanes—“butts in seats,” in
the vernacular. The development schedule had plans for new releases of the
gate agent and call center applications to transition to CF for flight lookup,

report erratum • discuss

 Chapter 2. Case Study: The Exception That Grounded an Airline • 10

but those had not been rolled out yet. This turned out to be a good thing, as
you’ll soon see.

The architects of CF were well aware of how critical it would be to the business.
They built it for high availability. It ran on a cluster of J2EE application
servers with a redundant Oracle 9i database. All the data were stored on a
large external RAID array with twice-daily, off-site backups on tape and on-
disk replicas in a second chassis that were guaranteed to be five minutes old
at most. Everything was on real hardware, no virtualization. Just melted
sand, spinning rust, and the operating systems.

The Oracle database server ran on one node of the cluster at a time, with
Veritas Cluster Server controlling the database server, assigning the virtual
IP address, and mounting or unmounting filesystems from the RAID array.
Up front, a pair of redundant hardware load balancers directed incoming
traffic to one of the application servers. Client applications like the server for
check-in kiosks and the IVR system would connect to the front-end virtual
IP address. So far, so good.

The diagram on page 11 probably looks familiar. It’s a common high-availability

architecture for physical infrastructure, and it’s a good one. CF did not suffer
from any of the usual single-point-of-failure problems. Every piece of hardware
was redundant: CPUs, drives, network cards, power supplies, network
switches, even down to the fans. The servers were even split into different
racks in case a single rack got damaged or destroyed. In fact, a second location
thirty miles away was ready to take over in the event of a fire, flood, bomb,
or attack by Godzilla.

The Change Window

As was the case with most of my large clients, a local team of engineers dedi-
cated to the account operated the airline’s infrastructure. In fact, that team
had been doing most of the work for more than three years when this hap-
pened. On the night the problem started, the local engineers had executed a
manual database failover from CF database 1 to CF database 2 (see diagram).
They used Veritas to migrate the active database from one host to the other.
This allowed them to do some routine maintenance to the first host. Totally
routine. They had done this procedure dozens of times in the past.

I will say that this was back in the day when “planned downtime” was a normal
thing. That’s not the way to operate now.

Veritas Cluster Server was orchestrating the failover. In the space of one
minute, it could shut down the Oracle server on database 1, unmount the

report erratum • discuss

 The Change Window • 11

Virtual IP Address

Hardware Load Balancer

CF App CF App CF App CF App

1 2 3 n

Virtual IP Address

CF CF
Database Heartbeat Database
1 2

SCSI SCSI
RAID 5
Array

filesystems from the RAID array, remount them on database 2, start Oracle
there, and reassign the virtual IP address to database 2. The application
servers couldn’t even tell that anything had changed, because they were
configured to connect to the virtual IP address only.

The client scheduled this particular change for a Thursday evening around
11 p.m. Pacific time. One of the engineers from the local team worked with
the operations center to execute the change. All went exactly as planned.
They migrated the active database from database 1 to database 2 and then
updated database 1. After double-checking that database 1 was updated
correctly, they migrated the database back to database 1 and applied the
same change to database 2. The whole time, routine site monitoring showed
that the applications were continuously available. No downtime was planned
for this change, and none occurred. At about 12:30 a.m., the crew marked
the change as “Completed, Success” and signed off. The local engineer headed
for bed, after working a 22-hour shift. There’s only so long you can run on
double espressos, after all.

Nothing unusual occurred until two hours later.

report erratum • discuss

 Chapter 2. Case Study: The Exception That Grounded an Airline • 12

The Outage
At about 2:30 a.m., all the check-in kiosks went red on the monitoring console.
Every single one, everywhere in the country, stopped servicing requests at
the same time. A few minutes later, the IVR servers went red too. Not exactly
panic time, but pretty close, because 2:30 a.m. Pacific time is 5:30 a.m.
Eastern time, which is prime time for commuter flight check-in on the Eastern
seaboard. The operations center immediately opened a Severity 1 case and
got the local team on a conference call.

In any incident, my first priority is always to restore service. Restoring service

takes precedence over investigation. If I can collect some data for postmortem
analysis, that’s great—unless it makes the outage longer. When the fur flies,
improvisation is not your friend. Fortunately, the team had created scripts
long ago to take thread dumps of all the Java applications and snapshots of
the databases. This style of automated data collection is the perfect balance.
It’s not improvised, it does not prolong an outage, yet it aids postmortem
analysis. According to procedure, the operations center ran those scripts right
away. They also tried restarting one of the kiosks’ application servers.

The trick to restoring service is figuring out what to target. You can always
“reboot the world” by restarting every single server, layer by layer. That’s
almost always effective, but it takes a long time. Most of the time, you can
find one culprit that is really locking things up. In a way, it’s like a doctor
diagnosing a disease. You could treat a patient for every known disease, but
that will be painful, expensive, and slow. Instead, you want to look at the
symptoms the patient shows to figure out exactly which disease to treat. The
trouble is that individual symptoms aren’t specific enough. Sure, once in a
while some symptom points you directly at the fundamental problem, but
not usually. Most of the time, you get symptoms—like a fever—that tell you
nothing by themselves.

Hundreds of diseases can cause fevers. To distinguish between possible

causes, you need more information from tests or observations.

In this case, the team was facing two separate sets of applications that were
both completely hung. It happened at almost the same time, close enough
that the difference could just be latency in the separate monitoring tools that
the kiosks and IVR applications used. The most obvious hypothesis was that
both sets of applications depended on some third entity that was in trouble.
As you can see from the dependency diagram on page 13, that was a big finger
pointing at CF, the only common dependency shared by the kiosks and the
IVR system. The fact that CF had a database failover three hours before this

report erratum • discuss

Random documents with unrelated
content Scribd suggests to you:
to the present time. But there are mai ka wa kahiko mai a hiki i
two points which I wish to keia wa. Aka nae, elua a’u mea
explain concerning this hill of e hoakaka aku ai no keia puu o
Kauiki. First: How it originated; Kauiki; akahi, kona wahi i loaa
secondly, the famous localities mai ai; elua, na wahi kaulana e
near to or connected with this pili ana me keia puu. Nolaila, e
hill. Let us therefore examine nana ae kakou i ka manao
some of the erroneous ideas of kuhihewa o ka poe kahiko.
the olden time.

first: how it akahi, kona wahi i loaa

originated. mai ai.

Olden people differed in this Ua manao like ole ia e ka poe

respect, four accounts of its kahiko, eha wahi i loaa mai ai
origin being given 1. That this hill keia puu. Ua loaa mai keia puu
originated from the placenta of mai ka iewe mai o Hamoa, a o
Hamoa, 77 some claim that it kekahi, mai na makua mai o
originated [546]from the parents Pele ma, [547]a o kekahi hoi, mai
of Pele and her host; others, a Kaihuakala mai laua me
from Kaihuakala 78 and Kahaule kana wahine, a o
Kahaule, 79 his wife; still others, kekahi, mai a Kalalawalu mai; a
from Kalalawalu; and these are penei na manao o lakou:
their accounts: Pele, Hiiaka and
Puuhele were born of their Ua hanauia mai o Pele ame
parents; Kahinalii was their Hiiaka, ame Puuhele, e ko lakou
mother. However, Pele and mau makua, e Kahinalii ka
Hiiaka were born with human makuahine o lakou; eia nae, o
bodies, while Puuhele was a Pele ame Hiiaka ua hanau kino
bloody foetus when she was kanaka maoli ia mai laua, a o
born. The elders despised this Puuhele hoi, ua hanau puu koko
body, because when they beheld mai oia. No ka makemake ole
it it was not a human body, but mai o na kaikuaana i keia kino,
only a foetus of blood; so the two no ka mea, i ka nana ana iho,
conspired and said one to the aole he kino maoli, aka, he puu
other: “It were better for us to koko wale no, nolaila, kuko iho la
throw away our younger sister. laua, a olelo aku la kekahi i
How can we care for it? Of kekahi: “E aho e kiloi kaua i ka
course we would care for it if it pokii o kaua. Auhea hoi ka
had a human body!” So they hoihoi aku, i ku no ka hoi ka
threw away the bloody foetus hoihoi aku ke kino kanaka.” A o
without the knowledge of the ko laua nei kiloi aku la la no ia i
parents. ua puu koko nei me ka ike ole o
na makua i ko laua hoolei ana
mai.

From that time onward this A ma ia manawa mai, ua

bloody foetus kept on traveling; it hoomaka keia puu koko e hele,
met Alenuihaha, that is the a halawai mai oia me
channel between Maui and Alenuihaha, oia kela moana
Hawaii, and Alenuihaha asked it: mawaena o Maui nei ame
“A journey! whither going?” The Hawaii, a ua ninau aku o
bloody foetus replied: “Just going Alenuihaha i ua puu koko nei:
to see what is ahead here; I was “He huakai! e hele mahea?”
cast away by my elders and that Olelo mai ua puu koko nei: “E
is the cause of this journeying.” hele ana ma keia wahi aku nei
la, i kiloi ia mai au e o’u mau
kaikuaana, a oia keia la e hele
nei la.”

After their conversation the A pau aela ka laua nei kamailio

foetus passed on until it landed ana, o ka hoomaka mai la no ia
at Nuu; 80 this place where it o ua puu koko nei e hele mai, a
landed is at Kaupo; she walked pae ana keia i Nuu, aia ma
on in the form of a human being. Kaupo keia wahi ana i pae mai
When Nuu looked he beheld a ai, a hele kino kanaka mai la
most beautiful woman. She kept keia, a i ka ike ana’ku o Nuu, he
right on until she met keu a ka wahine maikai. O ko
Puuomaiai; 81 she also was a ianei hele mai la no ia a halawai
good-looking woman. Puuomaiai me Puuomaiai, he wahine
asked the stranger: “To where maikai no hoi keia. Olelo aku la
are you journeying?” The other o Puuomaiai i ka malihini: “E
replied: “Going along to places hele ana mahea kau huakai?”
ahead of here.” Puuomaiai Olelo mai la kela: “E hele aku
requested that they be friends, ana ma keia wahi aku.” A noi
and they became such. She was mai la o Puuomaiai e hoaikane,
received; they ate, and after a hoaikane iho la laua nei, a
being satisfied they went along. hookipa ia iho la keia, ai laua a
While they were going maona, a hele aku la laua nei. O
Manawainui 82 voiced a chant, ko laua nei hele ana, kani ana ka
thus: Manawainui mele, penei no ia:

A restlessness seizes me on Kau ka lia ia oe e ka malihini,

account of thee, O stranger! Ke hele nei a maeele kuu
It is causing numbness to my manawa,
feelings. He hoa paio paha,
Is it an antagonist? He hoa makaikai paha e?
Or is it a traveling companion?

The foetus turned back and Huli ae la ua puu koko nei i

asked: “Why are you calling from hope, a olelo aku: “Heaha kau o
behind?” Manawainui replied: ke kahea ana mahope?” Pane
“Because I saw you very able in mai o Manawainui: “No kuu ike
walking, that was why I called ana ’ku nei ia oe, i ko mama i ka
you.” “What have you to present hele, nolaila kahea aku nei au ia
that you should call?” “Because I oe.” “Heaha kau makana nui o
saw that you were nothing but a ke kahea ana?” “No kuu ike ana
bloody foetus cast away by your aku nei ia oe, he puu koko wale
elders! and yet here you are no i kiloi ia mai e kou mau
walking! You had better be kaikuaana, nolaila, hele mai nei
named Puuhele.” 83 After their oe, e aho e kapa ia kou inoa o
conversation the two came along Puuhele.” A pau ka laua nei
until they arrived at Hana. kamailio ana, o ko laua nei hele
Puuomaiai said to Puuhele: “I mai la no ia a hiki mai i Hana.
had better return; we have come Olelo aku a Puuomaiai ia
together; there is yet another Puuhele: “E aho e hoi au, ua
time for meeting.” Puuhele hele pu mai la no hoi kaua, eia
consented, at the same time aku no hoi ua wa e hui hou ai.”
saying: “I know not how O ka ae mai la no ia o Puuhele,
prosperous this journey is; I may me kae i mai: “Aole i akaka ko’u
see you again or I may not,” and pono e hele nei, he ike hou paha
Puuhele kissed her friend on the ia oe, aole paha,” a o ka honi
nose. mai la no ia o ua Puuhele i ka
ihu o ke aikane.

She no sooner commenced on O ko ianei hoomaka ana e hele,

her journey again when she met halawai aku la keia me
Kanahaha. Kanahaha first spied Kanahaha, ike e mai la o
Puuhele, however, and she Kanahaha, a o kona make koke
immediately became dead, her iho la no ia, a waiho kahelahela
limbs extended and spread iho la na uha o ua wahine nei. O
apart. When Puuhele came up, ko ia nei hele aku no ia, a ike ua
she found the other dead. This make. O keia wahi o Kanahaha,
place Kanahaha is a hill from he [549]punawainui a hiki i keia
which gushes forth a spring of wa. Hele loa aku aku la no o
water to this day. Puuhele Puuhele a halawai me Lehoula.
[548]passed right on until she met Olelo mai o Lehoula: “Nani hoi
Lehoula. Lehoula said to her: ka wahine maikai o oe,” a
“What a beautiful woman you makemake o Lehoula e hele pu
are!” Lehoula wished to follow meia, a hiki ma Wananalua,
her. Coming to Wananalua, 84 hoohiki iho la o Puuhele ia wahi,
Puuhele made a vow to stay a olelo aku ia Lehoula: “Nani mai
there, and she said to Lehoula: “I la kuu makemake i keia wahi; o i
have a great desire for this hele mai nei e imi i kahi kupono
place; I came along looking for a e noho ai, akahi no a loaa.
good place to locate in, and I Nolaila, e noho au a hiki i ko’u
have just located it. Therefore I wa e make ai,” a o ka hoi mai la
will abide here until my death.” no ia o Lehoula a hiki ma kona
Lehoula returned to her usual wahi mau.
home.

Puuhele lived here. When Noho iho la no hoi o ua Puuhele;

Kaihuakala saw her he came a ike mai la o Kaihuakala o ka
down and met her and spoke iho mai la no ia o Kaihuakala a
reprovingly. “By what right did halawai me ia, a pane kiko e mai
you come here?” Puuhele no o Kaihuakala: “Heaha kou
humbly replied: “I came to see kuleana nui o ka hiki ana mai o
the country, and seeing this onei nei?” Olelo aku la o
beautiful country, I vowed to Puuhele me ka nawaliwali: “I
stay.” Then Puuhele was killed, hele mai nei hoi au i ka
and buried. She used her divine makaikai, a loaa keia wahi
power, the hill rose high. maikai, a hoohiki iho nei e noho.”
Kaihuakala named the hill
Kauiki, and that is the name by Ia manawa, pepehi ia iho la o ua
which it is known to the present o Puuhele a make, kanuia iho la
time; that also is the cause for a nalo. E hao ae ana keia i kona
the song some people use, thus: mana, o ka pii ia o ua puu nei a
kiekie, kapaia ka inoa o ua puu
nei o Kaihuakala, o Puuiki, a
mahope kapaia o Kauiki, a oia
ka inoa e kapaia nei a hiki i keia
wa, a oia iho la ke kumu o ke
mele a kekahi poe; a penei no
ia:
At Kauiki is the mail bag of the Aia i Kauiki ka eke leta a ka
wind, makani,
Being tossed about by the heavy Kahi paialewa ia mai la e ka lau
black rain; awa,
Facing is the surf of Anini to the Huli ka nalu o ke Anini i ka
windward, makani,
Facing so as to head for the I hoohuli no a huli i ka wai o
water of Punahoa. Punahoa.

But according to the idea of Aka, o ka manao o kekahi poe,

some people it was Lalawalu na ka Lalawalu i lawe mai, mai
who brought it from Kahiki; 85 she Kahiki mai, i lawe keiki hanai ia
brought it as her foster child, but mai, a no ka uluhua o ka
because she was vexed at the Lalawalu i ke nahu pinepine o ua
child for constantly nipping her keiki nei i ka waiu, nolaila, kupu
breast, therefore the mother ka manao iloko o ka makuahine
made up her mind to leave it. e haalele ia ia, lawe mai la a pae
She brought it along to Koloa, ma Koloa, i Kauai. Manao ia e
Kauai, and there she wanted to kiloi, aole nae he makemake o
cast it away, but the child did not ua keiki nei e noho malaila.
fancy staying there. She Hoomanawanui mai la no oia i
persevered in carrying the child ka hii ana, a hiki i Kaena ma
until they arrived at Kaena; 86 Oahu. Aole no he makemake o
again the child did not desire to ua keiki nei, nolaila, lawe loa ia
be left there, so it was brought mai a pae ma Kawaipapa ma
along until they landed at Hana i Maui Hikina, a hoonoho
Kawaipapa, 87 Hana, East Maui, ia me ia a hiki i keia wa, a pela
and it was left with him; and iho la kekahi manao o ka poe
there it stands until this day. That kahiko.
was the idea of some olden
people.
 secondly: the famous alua: na wahi kaulana
places connected with e pili ana ma keia puu o
this hill of kauiki. kauiki

On this hill are many famous Aia ma keia puu, he nui na wahi
places; for instance, right on top kaulana, oia hoi, maluna pono o
of this hill was the house in keia puu ka hale o Peapea i pau
which Peapea 88 was consumed ai i ke ahi, i puhiia ai e
by fire, when he was burnt out by Liionaiwaa ma, a oia ka mea i
Liionaiwaa and others; thus the oleloia: “Pau Peapea i ke ahi.”
saying at the present time, Aia hoi ma ka hema iki o keia
“Consumed by fire is Peapea.” A puu he awa pae waa keia, o
little to the south of this hill is a Kaihalulu ka inoa, no ia wahi
famous landing place for canoes, keia olelo e olelo ia nei,
called Kaihalulu (the roaring Kaihalulu i ke alo o Kauiki. Aia
sea); concerning this place is the no hoi malaila na niu a Kane; aia
saying now quoted: “The roaring aku makai ponoi o ia wahi he
sea in the presence of Kauiki.” At pohaku nui iloko o ke kai, ua
the same place, too, are the kapaia ka inoa o ia pohaku o
coconuts of Kane; right makai of Mokuhano. Aia hoi ma ka hikina
this place is a large rock in the ponoi o Kauiki o ka Pueokahi, ka
sea which is called Mokuhano. mea i kapaia ai ka inoa o ia
To the east of Kauiki is wahi, he pueo no na ke alii na
Pueokahi; 89 this place was so Peapea; aia ike ua pueo nei i ka
named on account of an owl nui o kanaka lele mai no ia a kau
belonging to the chief, Peapea. ma ke kikihi puka o ke alii, alaila,
When the bird saw there were ua nui kanaka; a mahope
plenty of people, it flew to the pepehiia a make, a oia ka mea i
door of the chief, indicating a kapaia ai o ka Pueokahi.
multitude. Afterwards it was
killed, and that was why it was
called Pueokahi.
To the north of Kahulili, with its A ma ka akau ponoi no hoi o
foundation right under Kauiki, Kahulili, a malalo pono no o
was what was known as the hair Kauiki kona kumu, ua kapaia oia
of Puuhele. Kaihuakala is mauka na lauoho o Puuhele. Aia mauka
of Kauiki. Kaihuakala is not o Kauiki o Kaihuakala. Aole e ike
usually seen; when Maui is calm, wale ia o Kaihuakala, aia a malie
then that locality is seen. Then o Maui nei alaila, ike ia keia
Papahawahawa stands forth and wahi. Ia wa no, ku mai la o
brags, saying, “Here I have lived, Papahawahawa a akena iho la
and yet this is the first time I me ka i iho hoi, “He noho ae nei
have [550]beheld the calmness of no hoi, akahi no a [551]ike ia ka
Maui; it is indeed clear, for malie a Maui, o ka malie ka ia ke
Kaihuakala can be seen.” [One ike ia aku la o Kaihuakala.” O
must behold] Kaihuakala Kaihuakala kai uka, o Kauiki ka i
mountainward and Kauiki kai, alaila pau i ka makaikai ia na
seaward in order to complete wahi a pau. A oia ka mea i olelo
one’s journey of sightseeing. ia nei e ka poe haku mele,
Thus the saying by composers of penei:
chants:

Maui is calm, for Kaihuakala is Malie o Maui, ua ikea o

seen; Kaihuakala,
Kaihuakala mountainward, O Kaihuakala kai uka o Kauiki
Kauiki to seaward. kai kai,
The surf breaks at Kaihalulu in I poia e ke kui, Kaihalulu i ke alo
the presence of Kauiki; o Kauiki,
Kauiki is nursing Mokuhano, Hii Kauiki ia Mokuhano,
Nursing the sand of Pueokahi, Hii ke one ka Pueokahi,
Nursing the cold wind from Hii waikoloa i Kailiili,
Kailiili, Hii ka nalu ia Keanini,
Nursing the surf of Keanini, Hoea ana i Mokumana,
Including Mokumana; He waa Hawaii e,
A canoe from Hawaii Waikaakihi ke awa,
[Enters] the harbor of Punahoa ka wai oneone ia ka
Waikaakihi. Pueokahi,
With the water at Punahoa, the O Honuaula mauka o Kahulili ka
sand at Pueokahi, iluna,
[With] Honuaulu mauka, [and] Pau Peapea la i ke ahi.
Kahulili on top,
Consumed indeed is Peapea by
fire.

If some people are going to A ina e hele ana kekahi poe ma

Kauiki [let them see] these Kauiki i keia mau wahi i hai ia ’ku
places which have been la, a o keia iho la na mea e pili
mentioned. This is all of what ana i ka hai ia ana mai ia’u e
has been told me by some one kekahi mea i ike i ka moolelo o
who knows the story of Kauiki; Kauiki, he nui aku no, aka, ua
there is much more, but this lawa keia mau wahi mea i
which has been learned by heart panaau no keia manawa.
is sufficient.
[Moanauli.]
[Moanauli.]

A STORY OF PUMAIA. HE MOOLELO NO

PUMAIA.

Pumaia 90 was born at Koloa, Ua hanauia o Pumaia ma Koloa i

Kauai. Malaihi was the father Kauai, o Malaihi ka makuakane,
and Kuhihewa was the mother. o Kuhihewa ka makuahine. Oi
They lived together until the child noho lakou nei a nui ua keiki nei,
had grown up, and his body had a ikaika no hoi kona kino, olelo
acquired strength, when Pumaia aku o Pumaia i ka makuakane:
said to his father, “You will “E oluolu oe e hele au e ike i ka
please let me go to see the club laau a ku’u kupunawahine,” o ka
of my grandmother.” The father ae mai la no ia o ka makuakane.
gave his consent. The son went, O ko ianei hele aku la no ia, a
and arriving at the house of Kiha, hiki ma ka hale o Kiha, ke
his grandmother, he asked, kupunawahine, a olelo aku la:
“Where is the club?” “Here it is,” “Auhea la ka laau?” “Eia ae no,”
said the grandmother. He took it. wahi a ka luahine. O ko ianei
The nets of calabashes 91 were lawe ae lo no ia. O ka paipu,
thereby dashed to the ground ulupa ia iho la ilalo, a okaoka
and broken to pieces. He liilii. O ka hiu ae la no ia i ua laau
shouldered the club and nei a kau i ka a-i, a hoomaka e
commenced his journey. He hele aku. Halawai mai la me ia o
caught up with Wakaina, and the Wakaina, a pane mai o Wakaina:
latter inquired, “Where are you “He huakai e hele ana ihea?” “E
journeying to?” “Going hele ana i ka makaikai,” wahi a
sightseeing,” replied Pumaia. Pumaia. “E aho hoi ha owau
“Please let me go with you.” kekahi me oe.” O ka ae mai la
Pumaia consented and they no ia; o ko laua hele aku la no ia.
went along.

Puukolea stood forth and called Ku ana o Puukolea, a olelo ana:

out, “Whose offspring are you?” “Nawai ke kupua o olua?” “Na
“Our own,” replied Pumaia. “If maua no,” wahi a Pumaia. “A nui
you excel in strength, you are ka ikaika ola olua, hapa ka ikaika
saved; if your strength is make olua ia’u.” O ko lakou nei
meager, I will kill you.” Then they hakaka iho la noia. O ke kino
commenced to fight. That man, na’e o kela kanaka he kino
however, had a dual body, while papalua, o ko laua nei kino hoi
these had the ordinary living he kanaka ola maoli no. Ua
body. Pumaia asked Wakaina, ninau aku o Pumaia ia Wakaina:
“Which of us shall be the first “Owai ka mea mua o kaua e
one to fight with this man?” The hakaka me ianei?” Pane mai la
other replied, “It is better that I kela: “E aho owau.” Ae aku la no
do it.” Pumaia consented and hoi keia; o ko Wakaina hele aku
Wakaina went to fight Puukolea. la no ia e hakaka me Puukolea.
Before they commenced fighting, Mamua nae o ko laua hoomaka
however, he chanted a portion of ana e hakaka, hoopuka ae la oia
a song, thus: [552] i kekahi mau lalani mele, penei:
[553]

Behold! behold! the mere lehua Aia, la, aia la, o ka lehua wale o
of Puuoni, Puuoni,
Struggling with the clouds of the Ke a uume inai la me opua i ka
air, lewa
Now above, now below the rain Maluna malalo ka wai opua.
clouds.

The other spoke up and said, O ka pane e mai la no ia o ua

“Why are you delaying? You are kanaka nei. “E hoololohi ana oe i
provoking me through your ke aha? he mele hoonaukiuki
song!” Wakaina had not kau no’u.” Aole nae i pau ke
completed his song before they mele ana o Wakaina, o ka
commenced to fight. From hakaka iho la no ia; mai ka auina
afternoon to near midnight [did la a kokoke aumoe. Kauoha aku
they fight]. Wakaina said to la o Wakaina ia Pumaia: “O ka
Pumaia: “You simply look on and nana mai no ka kau, aole oe e kii
do not come to aid me; I am mai ia’u la, ua kokoke au la e
nearly dead.” Pumaia jumped in make.” O ka lele mai la no ia o
and fought with the other, while Pumaia, a hakaka me iala; oi nei
Wakaina went to seek food for hoi, hele aku la e huli ai no laua
them. They kept on fighting until nei. Oi nonoke aku no hoi laua
Pumaia was weakened. He nei i ka hakaka, a aneane
thought of his club; so when nawaliwali nae o Pumaia;
Wakaina came back and noonoo keia i ka laau a ianei; a
inquired, “How are you?” he ku ana o Wakaina, a olelo mai,
answered, “I am weak; go and “Pehea oe?” Hai mai kela: “Ua
fetch my club; perhaps it can do nawaliwali au, e kii aku oe i kuu
something.” While they yet laau e lawe mai, malia paha he
spoke the other man appeared in hana ka ia laau.” Ia laua nei e
a different body. They fought on, kamailio ana, ku ana ua kanaka
and before Pumaia could reach nei me ke ano okoa. O ka
for his club, he was killed by this hakaka iho la no ia; aole nae i
man; Wakaina also was killed. lalau aku o Pumaia i ka laau a ia
Their spirits returned to their nei o ka make mai la no ia i ua
parents and were seen by them. kanaka nei; a make pu no hoi
(The parents of Wakaina had me Wakaina. O ka hoi iho la no
gone over and were living ia o na uhane o laua nei a ku
together with Pumaia’s parents.) ana imua o na makua, a i ka ike
While they were idling away their ana mai o na makua o Wakaina
time a chant was voiced by the a me Pumaia, no ka mea, ua hoi
wandering spirits: aku na makua o Wakaina a noho
pu me na makua o Pumaia. Ia
lakou nei e nanea ana, kani ana
ke mele a ua mau uhane haukae
nei, penei:

Fear is creeping over us, Ko mai ana ke ano io maua nei,

Coming for us to go; E kii mai ana ia maua e hele,
We can not; we are held by Aole e hiki ua paa ia Waiauau,
Waiauau. E kii mai ana ia maua e hoola—
Come to give us life! e
Life—indeed. E ola no—e.

When these ghosts finished their A pau ke mele ana a ua mau

chant the parents came out eepa nei, o ka oili koke mai la no
immediately, but they saw ia o na makua; aole nae he ike
nothing. They looked here and iki. Oi nana wale lakou nei ma o
there, but they could not see a maanei, aole no he ike iki ia.
them. Meanwhile the two had
gone as spirits until they met O ka uhaele uhane aku la no ia
Pupuilima. Some people saw o laua nei a halawai me
these two coming, but Pupuilima Pupuilima. Ia wa ike mai na
said: “Those are not men; those kanaka i ka hele aku o keia mau
are ghosts.” “How do you know kanaka elua, ia wa olelo mai la o
those are ghosts?” said the Pupuilima: “Aole kela he mau
others. So they laid wagers. kanaka, he akua kela mau mea.”
Pupuilima then said to them, “I “Mahea kahi i maopopo ai ia oe
will spread an ape 92 leaf on the he akua?” wahi a ua poe nei. O
ground; and if it breaks, then ko lakou nei pili iho la no ia.
they are men, but if it does not, Olelo aku la o Pupuilima ia
then they are spirits.” He spread lakou: “I halii auanei au i ka lau
it. Meanwhile Pumaia said to ape ma ke alanui nei a i nahae,
Wakaina, “There is our death alaila, he kanaka, aka hoi, ina
being prepared.” “How shall we nahae ole, alaila, he devil.” O ko
be saved?” Pumaia explained: ia nei halii iho la no ia. Ia wa,
“Where my feet tread there you pane iho la o Pumaia ia
tread; because I was raised from Wakaina, “Aia ko kaua make la,
my young days until I died, and ke hana ia mai la.” “Pehea la
until I found you; for this process auanei kaua e pakele ai?” Hai
is like that done to Pamano mai la o Pumaia: “Ma ko’u
when by the spreading of an ape wawae e hehi ai, malaila mai no
leaf he was saved.” So when oe e hele ai, no ka mea, ua
they proceeded Pumaia trod on hanai ia wau mai kuu wa uuku a
the ape leaf and it broke through; i ko’u wa i make ai, a loaa oe, no
the friend treaded after him. ka mea o keia hana ana, ua like
They were chased by the me ko Pamano halii ia ana e ka
prophet until they were caught. lau ape, a pakele ke ola.” Ia laua
That is what I have obtained. nei i hele aku ai, o ka hehi iho la
 no ia o Pumaia i ka lau ape, a
nahae, a mahope iho la no hoi
ke keehi ana o ke aikane. O ko
laua nei hahai ia aku ’la no ia e
ua makaula nei a loaa i kahi i
loaa ai.

Oia iho la no kahi mea i loaa ie

mai nei.

While they were going along Ia laua nei e hele ana, ua alualu
they were given chase. They ia mai la nae laua nei. O ko laua
came along until they caught up nei hele maila no ia a halawai
with Pueonuiokona. 93 The owl, me Pueonuiokona, aole nae he
however, did not catch sight of ike mai o ua Pueo nei i ka laua
them while they were coming. nei hele aku. A kaa laua nei
When they had passed ahead mamua, ia manawa halawai mai
the prophet who was chasing la ka makaula e alualu nei ia
them caught up with laua me Pueonuiokona. Pane
Pueonuiokona. The owl asked, aku la ua Pueo nei: “Heaha ka
“What is the cause of this heavy mea i nui ai o ka hanu [555]a
breathing and this perspiring?” kahe hoi ka hou?” Hai aku la
This one answered, “That you keia: “Heaha mai ka hoi kau, he
should be asking [554]‘what’? mau uhane aia la, o ka’u ia e
Spirits! and there they are! I am alualu nei aohe loaa iki; e ake
chasing them, but can not catch ana hoi au o ka lihi launa aku,
them; I have been wishing to get make la hoi ia’u, ua hele mai kuu
near them so that I can kill them, ukiuki a nui ia laua.”
for I am possessed with great
anger towards them.”

When the owl heard what the Ia lohe ana o Pueo i ka olelo a
prophet said, he said to him, ka makaula, ia manawa oia i
“You are a prophet, and I am a olelo aku ai i ua makaula nei:
prophet, still I did not see them; “He makaula oe, a he makaula
and now I hear you saying that if wau, eia nae, aole wau i ike aku
you catch them they die.” Where nei ia laua, a no kuu lohe ana
they were holding this mai nei i kau olelo, ke loaa aku
conversation, however, was on ia oe make.” O kahi nae a laua e
the plain of Kamaomao. While kamailio nei, aia ma ke kula o
the others prepared to come for Kamaomao. Ia laua nei nae e
the spirits, Pumaia said to his hoomakaukau ana e kii i na
friend, “Here comes our death; uhane, olelo aku la o Pumaia i
but we will wait. If the new one ke aikane: “Eia a’e ka make o
gets ahead of the old one then kaua la, aka, i kali auanei kaua a
we have hope for life.” i oi kela mea hou mamua o ka
mea mua, alaila, manao ae ke
ola.”

So they sat and watched the two O ko laua nei noho iho la no ia
prophets. When Pueo distanced nana no laua nei i ua mau
the other, Pumaia said to the makaula nei. A oi no o ua o
friend, “We are now saved; it Pueo mamua, olelo aku o
were better that we go to our Pumaia i ke aikane: “Akahi kaua
parents. It may be that we would a pakele, e aho e uhaele kaua a
be found there.” The friend kahi o na makua o kaua; malia
consented. They came along paha, o loaa ae kaua ilaila.” Ae
Kealia, a large pond even to this mai la ke aikane. O ko laua nei
day. These places above hoomaka mai la no ia e hele ma
mentioned, the plain of Kealia, he loko nui no hoi a hiki i
Kamaomao and Kealia are at the keia wa. O keia mau wahi nae i
eastern isthmus of Maui, hai ia a’e nei, no kula o
connecting East and West Maui. Kamaomao a me Kealia, aia no
ma ka puali hikina o Maui nei,
 alaila pau o Maui Hikina, pau o
Maui Komohana.

Let us leave the ghosts and E waiho ae kakou no ua mau

speak of the prophets. While the akua lapu nei, a e kamailio
prophets were following, and kakou no na makaula. Aia i ua
because Pueonuiokona was the mau makaula e alualu nei, a no
faster, an idea occurred to one of ka mama o Pueonuiokona, kupu
them, and he said to himself, “If I mai la ka manao iloko o kekahi o
catch up to that prophet I shall laua, me ka nune iho iloko ona:
kill him, for he is simply going “Ina e loaa ana ia’u kela
after my enemies to save them.” makaula e make ana oia ia’u, no
The owl went along slowly until ka mea he kii wale i ko’u mau
his fellow prophet thundered enemi e hoola.” Naue malie aku
after him and gave his hand a la nae ua Pueo nei, a halulu ana
jerk. Pueo asked, “What wrong ua makaula nei, a huki mai la i
has a fellow committed that you ka lima o Pueo. A ninau mai la o
should tug at the arm from Pueo: “Heaha iho la ka hewa o
behind and thereby nearly ke kanaka o kau huki ana i ka
tripping me?” The other replied lima mahope, a mai hina wau?”
with blazing eyes, “Who told you Olelo mai kela me ka inoino o na
to go after my enemies with the maka: “Nawai oe i i aku e kii i
intention of saving them? Why ko’u mau enemi, me ko olelo iho
don’t you look for your own, and e hoola? Aole no hoi oe e huli i
on them practice saving?” The mau mea okoa aku nau, ilaila oe
owl replied: “If you excel in e hoola ai.” Pane mai la ua Pueo
strength you can obtain them, nei: “Aia a nui ko ikaika, lilo ia
but if you are weak you can not oe, aka hoi, hapa ko ikaika, aole
have them; because I, even I, e loaa ia oe; no ka mea, owau
am an old resident of this plain. nei la, he kamaaina wau no keia
When spirits from the dead kula. Hele mai na uhane make,
arrive I cause them to revive. na’u no e hoola aku, aole no hoi
Also no prophet comes to this he makaula hiki ma keia kula,
plain; this is the first time that I akahi wale no wau a ike o oe
have seen a prophet on this hookahi ka makaula hiki maanei.
plain, and that is yourself. A no ko olelo ana mai nei ia’u no
Because you complained of my kuu kii e hoola i kela mau mea;
desire to give life to those spirits, heaha la ka hana, aia mamuli o
so be it! By a test of strength will ka ikaika e lilo aku ai oe, a ia’u
[we know] whether you obtain paha laua.”
them or I do.”

They immediately commenced to O ko laua nei hoomaka koke iho

fight. Where they fought was at la no ia i ka hakaka. O ko laua
Kalepolepo, 94 near this isthmus. wahi i hakaka ai, ma Kalepolepo,
The prophet was killed and his kokoke no ma keia puali. A
entrails were disemboweled by make ua makaula nei, kua’i ia ae
Pueonuiokona and placed on the la ka opu o ua makaula nei e
akolea. 95 That shrub used to be Pueonuiokona, a kauhihi ia ae la
plentiful at that place, but it is maluna o ka akolea. He laau nui
destroyed now on account of keia ma kela wahi, a ua lilo nae i
numerous animals. mea ole i keia manawa, no ka
nui o na holoholona.
August 8, 1872. Kiliona.
August 8, 1872. Kiliona.

A STORY OF HE MOOLELO NO
PUUPEHE. PUUPEHE.

Puupehe was the child of O Puupehe he keiki ia na

Kapokoholua and Kapoiliili. On Kapokoholua ame Kapoiliilii. Aia
the island of Kaululaau 96 lived ma ka mokupuni o Kaululaau ka
the subject of this story, the mea nona keia moolelo, ka eueu
champion of that past age. Let hoi oia kau i hala. E huli iki ae
us glance briefly at the kakou no na hana a kona mau
occupation of the parents. While makua. Aia i ka wa a laua e
they were living together the noho nei, ua hoomaka mai kana
woman [556]realized that she [557]wahine e loaa nei mea he
would obtain a beautiful fruit hua maikai no kona puhaka. O
from her womb. The husband ka hana hoi a ke kane o ka hele i
was a fisherman who used to go ka lawaia na laua, a hala kekahi
fishing for them, and he passed manawa loihi, o ka hoomau ana
long periods at his occupation. o kana kane i ka lawaia. I kekahi
One day Kapokoholua sailed out la, ua holo aku ua Kapokoholua
on the ocean in a canoe to fish. nei i ka moana maluna o ka waa
When he had gone Puupehe i ka lawaia, a i kona kaawale ana
commenced a conversation from aku ua hoomaka mai o Puupehe
inside the mother’s womb. When e kamailio ae iloko o ka opu o
Puupehe spoke the mother was kona makuahine. Ia wa a
startled from her sleep, and she Puupehe e kamailio nei, ua
wondered at this strange sound puiwa ae la kona makua mai
which she heard. She went to kona hiamoe ana, a
sleep again; the live one called hoomaopopo iho la keia no keia
out, “Say, my dear mother, do mea kupanaha ana i lohe ai. Hoi
thou awake; father is dead.” The hou iho keia moe no, kani ana
mother then realized that it was kahea a ua eueu nei: “E! ko’u
a human voice, but she could mama aloha, e ala mai oe ua
see no human being. She ran make o papa.” Ia manawa
quickly and stood at the door of hoomaopopo loa iho la keia he
the house and looked round for leo kanaka, koe nae kona ike
the owner of the voice, but she ana aku i ke kino maoli. Holo
could see no one; she thought, wikiwiki aku la keia a ku ana ma
however, that it was her ka puka o ka hale, nana aku la
husband, but she was hoi o ka ike aku i ka mea nona
disappointed. She returned and keia leo; aole nae he ike ia, ua
sat in the house. The child manao nae ia o kana kane, aka
voiced a chant thus: ua hoka nae ia. Hoi mai la no
keia a noho iloko o ka hale. Kani
ana ke oli a ua keiki nei, penei:

How I desire the moisture of the Aloha au o kahi wai a ke kehau,

dew, Oia wai haaheo mai iluna o ka
That water proudly hanging on laau.
the tree; A ua mai ka ua i na pali
When the rain falls on the E hoopiha ana i na kahawai.
precipice O ka waiwai no ia a kuu maka o
It fills up the streams. ka ike aku.
That is the benefit my eyes A ike kuu maka a nehe kuu lima.
behold!
My eyes to behold, my hands to
rustle.

When the child finished chanting A pau ke oli ana a ua keiki nei,
his mother became possessed ua ano e mai la ka makuahine, a
and was greatly troubled. I had ku a pilikia maoli ia. E pono e
better explain shortly about his wehewehe iki aku wau i ke ano o
chanting and falsely stating that kana oli ana a me ke kamailio
his father was dead. It was not hoopunipuni ana ua make ka
true as he chanted. He had gone makuakane; aole he oiaio o ke
to watch his father fishing, and ano o ke oli ana. Ua hele ia e
he had sent for a great number nana i ka lawaia ana o kona
of fish to come and bite the makuakane, a ua kii aku ia i na
hook. He saw that his father had i’a he nui loa e hele e ai i ka
caught a great many fish, but he makau. No ka ike ana ua nui na
needed the second [requisite], i’a i loaa i kona papa, ua koe hoi
the awa root. He knew his ka lua, oia ka awa. Ua ike ia
parents had none; that was why aohe awa a kona mau makua,
he voiced the few lines of song nolaila oia i puana ae ai i keia
above written. mau lalani mele e kau ae la
maluna.

Let us drop what the child did for E waiho iki kakou i na hana a ke
some later time and turn and talk keiki a mahope aku. E huli ae
of the father. While his father kakou a kamailio no ka
was fishing he became very makuakane. A i kona makua e
much interested because he lawaia nei, ua nanea loa ia i ka
caught so many. When he nui o na i’a i loaa iaia, a i kona
glanced shoreward he could not nana ana mai iuka aohe ike ia
see land, because Puupehe had aku o uka, no ka mea ua uhi iho
covered it completely with fog. la o Puupehe i ka ohu a
He thought to himself, “What can nalowale ka aina. I iho la keia
this wonderful thing be? There is iloko ona: “Heaha la hoi keia
now no wind to bring the fog on mea kupanaha, nokamea, aole
to the land!” He had a hoi he makani nana e lawe mai
premonition, however, ka ohu a kau iluna o ka aina.” Ua
concerning his wife, so he halialia wale mai nae na ano o
commenced to pull in his line. kana wahine, hoomaka iho la ia
When it was near the top his line e huki mai i ka aho a kokoke e
was held by a shark. The name pau mai iluna, paa ana ke aho a
of this shark was Puaiki. ianei i ka mano, o ka inoa o keia
mano o Puaiki.

Let me say a few words E kamailio iki ae au no keia i’a

concerning this despicable fish. ino. Ia ike ana oia nei he mano
When he realized that it was a ka mea nana e paa mai la, huki
shark holding on he jerked the mai la ia me ka ikaika, e hao aku
line; the fish took him to a point ana ua i’a nei, kokoke keia
near the cape of Papawai here mawaho ae o ka lae o Papawai
on Maui; from there he again ma Maui nei, malaila aku hiki