Introduction to Software Testing Paul Ammann pdf

download

https://ebookgate.com/product/introduction-to-software-testing-
paul-ammann/

Get Instant Ebook Downloads – Browse at https://ebookgate.com

Instant digital products (PDF, ePub, MOBI) available
Download now and explore formats that suit you...

Introduction to Software Testing 1st Edition Paul Ammann

https://ebookgate.com/product/introduction-to-software-testing-1st-
edition-paul-ammann/

ebookgate.com

Software Testing 4th Edition Paul C. Jorgensen

https://ebookgate.com/product/software-testing-4th-edition-paul-c-
jorgensen/

ebookgate.com

Introduction to Nondestructive Testing A Training Guide

2nd Edition Paul E. Mix

https://ebookgate.com/product/introduction-to-nondestructive-testing-
a-training-guide-2nd-edition-paul-e-mix/

ebookgate.com

Introduction to Nondestructive Testing A Training Guide

Second Edition Paul E. Mix(Auth.)

https://ebookgate.com/product/introduction-to-nondestructive-testing-
a-training-guide-second-edition-paul-e-mixauth/

ebookgate.com
Software Testing 1st Edition Ali Mili

https://ebookgate.com/product/software-testing-1st-edition-ali-mili/

ebookgate.com

Software Testing Third Edition Brian Hambling

https://ebookgate.com/product/software-testing-third-edition-brian-
hambling/

ebookgate.com

How to Reduce the Cost of Software Testing 1st Edition

Matthew Heusser

https://ebookgate.com/product/how-to-reduce-the-cost-of-software-
testing-1st-edition-matthew-heusser/

ebookgate.com

Software Testing Interview Questions Computer Science 1st

Edition S. Koirala

https://ebookgate.com/product/software-testing-interview-questions-
computer-science-1st-edition-s-koirala/

ebookgate.com

Well Testing Project Management Paul J. Nardone

https://ebookgate.com/product/well-testing-project-management-paul-j-
nardone/

ebookgate.com
Introduction to Software Testing

This extensively classroom-tested text takes an innovative approach to

explaining software testing that defines it as the process of applying a few
precise, general-purpose criteria to a structure or model of the software.
The text incorporates cutting-edge developments, including techniques to
test modern types of software such as OO, web applications, and embed-
ded software. This revised second edition significantly expands coverage
of the basics, thoroughly discussing test automaton frameworks, and adds
new, improved examples and numerous exercises. Key features include:
 The theory of coverage criteria is carefully, cleanly explained to help
students understand concepts before delving into practical applica-
tions.
 Extensive use of the JUnit test framework gives students practical
experience in a test framework popular in industry.
 Exercises feature specifically tailored tools that allow students to
check their own work.
 Instructor’s manual, PowerPoint slides, testing tools for students, and
example software programs in Java are available from the book’s
website.

Paul Ammann is Associate Professor of Software Engineering at George

Mason University. He earned the Volgenau School’s Outstanding Teach-
ing Award in 2007. He led the development of the Applied Computer
Science degree, and has served as Director of the MS Software Engi-
neering program. He has taught courses in software testing, applied
object-oriented theory, formal methods for software engineering, web
software, and distributed software engineering. Ammann has published
more than eighty papers in software engineering, with an emphasis
on software testing, security, dependability, and software engineering
education.

Jeff Offutt is Professor of Software Engineering at George Mason Uni-

versity. He leads the MS in Software Engineering program, teaches
software engineering courses at all levels, and developed new courses
on several software engineering subjects. He was awarded the George
Mason University Teaching Excellence Award, Teaching with Technol-
ogy, in 2013. Offutt has published more than 165 papers in areas such
as model-based testing, criteria-based testing, test automaton, empirical
software engineering, and software maintenance. He is Editor-in-Chief
of the Journal of Software Testing, Verification and Reliability; helped
found the IEEE International Conference on Software Testing; and is
the founder of the μJava project.
INTRODUCTION TO
SOFTWARE
TESTING

Paul Ammann
George Mason University

Jeff Offutt
George Mason University
University Printing House, Cambridge CB2 8BS, United Kingdom
One Liberty Plaza, 20th Floor, New York, NY 10006, USA
477 Williamstown Road, Port Melbourne, VIC 3207, Australia
4843/24, 2nd Floor, Ansari Road, Daryaganj, Delhi – 110002, India
79 Anson Road, #06-04/06, Singapore 079906

Cambridge University Press is part of the University of Cambridge.

It furthers the University’s mission by disseminating knowledge in the pursuit of
education, learning, and research at the highest international levels of excellence.

www.cambridge.org
Information on this title: www.cambridge.org/9781107172012
DOI: 10.1017/9781316771273

c Paul Ammann and Jeff Offutt 2017



This publication is in copyright. Subject to statutory exception

and to the provisions of relevant collective licensing agreements,
no reproduction of any part may take place without the written
permission of Cambridge University Press.

First published 2017

Printed in the United States of America by Sheridan Books, Inc.

A catalogue record for this publication is available from the British Library.

Library of Congress Cataloguing in Publication Data

Names: Ammann, Paul, 1961– author. | Offutt, Jeff, 1961– author.
Title: Introduction to software testing / Paul Ammann, George Mason
University, Jeff Offutt, George Mason University.
Description: Edition 2. | Cambridge, United Kingdom; New York, NY, USA:
Cambridge University Press, [2016]
Identifiers: LCCN 2016032808 | ISBN 9781107172012 (hardback)
Subjects: LCSH: Computer software–Testing.
Classification: LCC QA76.76.T48 A56 2016 | DDC 005.3028/7–dc23
LC record available at https://lccn.loc.gov/2016032808

ISBN 978-1-107-17201-2 Hardback

Additional resources for this publication at https://cs.gmu.edu/∼offutt/softwaretest/.

Cambridge University Press has no responsibility for the persistence or accuracy of

URLs for external or third-party Internet Web sites referred to in this publication
and does not guarantee that any content on such Web sites is, or will remain,
accurate or appropriate.
Contents

List of Figures page ix

List of Tables xii
Preface to the Second Edition xiv

Part 1 Foundations 1
1 Why Do We Test Software? 3
1.1 When Software Goes Bad 4
1.2 Goals of Testing Software 8
1.3 Bibliographic Notes 17

2 Model-Driven Test Design 19

2.1 Software Testing Foundations 20
2.2 Software Testing Activities 21
2.3 Testing Levels Based on Software Activity 22
2.4 Coverage Criteria 25
2.5 Model-Driven Test Design 27
2.5.1 Test Design 28
2.5.2 Test Automation 29
2.5.3 Test Execution 29
2.5.4 Test Evaluation 29
2.5.5 Test Personnel and Abstraction 29
2.6 Why MDTD Matters 31
2.7 Bibliographic Notes 33

3 Test Automation 35
3.1 Software Testability 36
3.2 Components of a Test Case 36

v
vi Contents

3.3 A Test Automation Framework 39

3.3.1 The JUnit Test Framework 40
3.3.2 Data-Driven Tests 44
3.3.3 Adding Parameters to Unit Tests 47
3.3.4 JUnit from the Command Line 50
3.4 Beyond Test Automation 50
3.5 Bibliographic Notes 53

4 Putting Testing First 54

4.1 Taming the Cost-of-Change Curve 54
4.1.1 Is the Curve Really Tamed? 56
4.2 The Test Harness as Guardian 57
4.2.1 Continuous Integration 58
4.2.2 System Tests in Agile Methods 59
4.2.3 Adding Tests to Legacy Systems 60
4.2.4 Weaknesses in Agile Methods for Testing 61
4.3 Bibliographic Notes 62

5 Criteria-Based Test Design 64

5.1 Coverage Criteria Defined 64
5.2 Infeasibility and Subsumption 68
5.3 Advantages of Using Coverage Criteria 68
5.4 Next Up 70
5.5 Bibliographic Notes 70

Part 2 Coverage Criteria 73

6 Input Space Partitioning 75
6.1 Input Domain Modeling 77
6.1.1 Interface-Based Input Domain Modeling 79
6.1.2 Functionality-Based Input Domain Modeling 79
6.1.3 Designing Characteristics 80
6.1.4 Choosing Blocks and Values 81
6.1.5 Checking the Input Domain Model 84
6.2 Combination Strategies Criteria 86
6.3 Handling Constraints Among Characteristics 92
6.4 Extended Example: Deriving an IDM from JavaDoc 93
6.4.1 Tasks in Designing IDM-Based Tests 93
6.4.2 Designing IDM-Based Tests for Iterator 94
6.5 Bibliographic Notes 102

7 Graph Coverage 106

7.1 Overview 106
7.2 Graph Coverage Criteria 111
7.2.1 Structural Coverage Criteria 112
7.2.2 Touring, Sidetrips, and Detours 116
7.2.3 Data Flow Criteria 123
7.2.4 Subsumption Relationships Among Graph
Coverage Criteria 130
 Contents vii

7.3 Graph Coverage for Source Code 131

7.3.1 Structural Graph Coverage for Source Code 132
7.3.2 Data Flow Graph Coverage for Source Code 136
7.4 Graph Coverage for Design Elements 146
7.4.1 Structural Graph Coverage for Design Elements 147
7.4.2 Data Flow Graph Coverage for Design Elements 148
7.5 Graph Coverage for Specifications 157
7.5.1 Testing Sequencing Constraints 157
7.5.2 Testing State Behavior of Software 160
7.6 Graph Coverage for Use Cases 169
7.6.1 Use Case Scenarios 171
7.7 Bibliographic Notes 173

8 Logic Coverage 177

8.1 Semantic Logic Coverage Criteria (Active) 178
8.1.1 Simple Logic Expression Coverage Criteria 179
8.1.2 Active Clause Coverage 181
8.1.3 Inactive Clause Coverage 185
8.1.4 Infeasibility and Subsumption 186
8.1.5 Making a Clause Determine a Predicate 187
8.1.6 Finding Satisfying Values 192
8.2 Syntactic Logic Coverage Criteria (DNF) 197
8.2.1 Implicant Coverage 198
8.2.2 Minimal DNF 199
8.2.3 The MUMCUT Coverage Criterion 200
8.2.4 Karnaugh Maps 205
8.3 Structural Logic Coverage of Programs 208
8.3.1 Satisfying Predicate Coverage 212
8.3.2 Satisfying Clause Coverage 213
8.3.3 Satisfying Active Clause Coverage 215
8.3.4 Predicate Transformation Issues 217
8.3.5 Side Effects in Predicates 220
8.4 Specification-Based Logic Coverage 223
8.5 Logic Coverage of Finite State Machines 226
8.6 Bibliographic Notes 231

9 Syntax-Based Testing 234

9.1 Syntax-Based Coverage Criteria 234
9.1.1 Grammar-Based Coverage Criteria 234
9.1.2 Mutation Testing 237
9.2 Program-Based Grammars 241
9.2.1 BNF Grammars for Compilers 241
9.2.2 Program-Based Mutation 242
9.3 Integration and Object-Oriented Testing 259
9.3.1 BNF Integration Testing 259
9.3.2 Integration Mutation 259
9.4 Specification-Based Grammars 266
9.4.1 BNF Grammars 266
viii Contents

9.4.2 Specification-Based Mutation 267

9.5 Input Space Grammars 271
9.5.1 BNF Grammars 271
9.5.2 Mutating Input Grammars 273
9.6 Bibliographic Notes 281

Part 3 Testing in Practice 283

10 Managing the Test Process 285
10.1 Overview 285
10.2 Requirements Analysis and Specification 286
10.3 System and Software Design 287
10.4 Intermediate Design 288
10.5 Detailed Design 288
10.6 Implementation 289
10.7 Integration 289
10.8 System Deployment 290
10.9 Operation and Maintenance 290
10.10 Implementing the Test Process 291
10.11 Bibliographic Notes 291

11 Writing Test Plans 292

11.1 Level Test Plan Example Template 293
11.2 Bibliographic Notes 295

12 Test Implementation 296

12.1 Integration Order 297
12.2 Test Doubles 298
12.2.1 Stubs and Mocks: Variations of Test Doubles 299
12.2.2 Using Test Doubles to Replace Components 300
12.3 Bibliographic Notes 303

13 Regression Testing for Evolving Software 304

13.1 Bibliographic Notes 306

14 Writing Effective Test Oracles 308

14.1 What Should Be Checked? 308
14.2 Determining Correct Values 310
14.2.1 Specification-Based Direct Verification of Outputs 310
14.2.2 Redundant Computations 311
14.2.3 Consistency Checks 312
14.2.4 Metamorphic Testing 312
14.3 Bibliographic Notes 314

List of Criteria 316

Bibliography 318
Index 337
Figures

1.1 Cost of late testing 12

2.1 Reachability, Infection, Propagation, Revealability (RIPR) model 21
2.2 Activities of test engineers 22
2.3 Software development activities and testing levels – the “V Model” 23
2.4 Model-driven test design 30
2.5 Example method, CFG, test requirements and test paths 31
3.1 Calc class example and JUnit test 41
3.2 Minimum element class 42
3.3 First three JUnit tests for Min class 43
3.4 Remaining JUnit test methods for Min class 45
3.5 Data-driven test class for Calc 46
3.6 JUnit Theory about sets 48
3.7 JUnit Theory data values 48
3.8 AllTests for the Min class example 49
4.1 Cost-of-change curve 55
4.2 The role of user stories in developing system (acceptance) tests 60
6.1 Partitioning of input domain D into three blocks 76
6.2 Subsumption relations among input space partitioning criteria 89
7.1 Graph (a) has a single initial node, graph (b) multiple initial nodes, and
graph (c) (rejected) with no initial nodes 108
7.2 Example of paths 108
7.3 A Single-Entry Single-Exit graph 110
7.4 Test case mappings to test paths 110
7.5 A set of test cases and corresponding test paths 111
7.6 A graph showing Node Coverage and Edge Coverage 114
7.7 Two graphs showing prime path coverage 116
7.8 Graph with a loop 117
7.9 Tours, sidetrips, and detours in graph coverage 117
7.10 An example for prime test paths 119
7.11 A graph showing variables, def sets and use sets 124
7.12 A graph showing an example of du-paths 126

ix
x List of Figures

7.13 Graph showing explicit def and use sets 128

7.14 Example of the differences among the three data flow coverage criteria 129
7.15 Subsumption relations among graph coverage criteria 132
7.16 CFG fragment for the if-else structure 133
7.17 CFG fragment for the if structure without an else 133
7.18 CFG fragment for the if structure with a return 133
7.19 CFG fragment for the while loop structure 134
7.20 CFG fragment for the for loop structure 134
7.21 CFG fragment for the do-while structure 135
7.22 CFG fragment for the while loop with a break structure 136
7.23 CFG fragment for the case structure 136
7.24 CFG fragment for the try-catch structure 137
7.25 Method patternIndex() for data flow example 141
7.26 A simple call graph 147
7.27 A simple inheritance hierarchy 148
7.28 An inheritance hierarchy with objects instantiated 149
7.29 An example of parameter coupling 150
7.30 Coupling du-pairs 151
7.31 Last-defs and first-uses 151
7.32 Quadratic root program 153
7.33 Def-use pairs under intra-procedural and inter-procedural data flow 154
7.34 Def-use pairs in object-oriented software 155
7.35 Def-use pairs in web applications and other distributed software 155
7.36 Control flow graph using the File ADT 159
7.37 Elevator door open transition 161
7.38 Watch–Part A 163
7.39 Watch–Part B 164
7.40 An FSM representing Watch, based on control flow graphs of the
methods 165
7.41 An FSM representing Watch, based on the structure of the
software 165
7.42 An FSM representing Watch, based on modeling state variables 167
7.43 ATM actor and use cases 169
7.44 Activity graph for ATM withdraw funds 172
8.1 Subsumption relations among logic coverage criteria 187
8.2 Fault detection relationships 202
8.3 Thermostat class 210
8.4 PC true test for Thermostat class 213
8.5 CC test assignments for Thermostat class 214
8.6 Calendar method 226
8.7 FSM for a memory car seat–Nissan Maxima 2012 227
9.1 Method Min and six mutants 243
9.2 Mutation testing process 246
9.3 Partial truth table for (a ∧ b) 253
9.4 Finite state machine for SMV specification 268
9.5 Mutated finite state machine for SMV specification 269
9.6 Finite state machine for bank example 271
 List of Figures xi

9.7 Finite state machine for bank example grammar 272

9.8 Simple XML message for books 274
9.9 XML schema for books 275
12.1 Test double example: Replacing a component 300
 Tables

6.1 First partitioning of triang()’s inputs (interface-based) 82

6.2 Second partitioning of triang()’s inputs (interface-based) 82
6.3 Possible values for blocks in the second partitioning in Table 6.2 83
6.4 Geometric partitioning of triang()’s inputs (functionality-based) 83
6.5 Correct geometric partitioning of triang()’s inputs (functionality-based) 84
6.6 Possible values for blocks in geometric partitioning in Table 6.5 84
6.7 Examples of invalid block combinations 93
6.8 Table A for Iterator example: Input parameters and characteristics 95
6.9 Table B for Iterator example: Partitions and base case 96
6.10 Table C for Iterator example: Refined test requirements 97
6.11 Table A for Iterator example: Input parameters and characteristics
(revised) 100
6.12 Table C for Iterator example: Refined test requirements (revised) 101
7.1 Defs and uses at each node in the CFG for patternIndex() 139
7.2 Defs and uses at each edge in the CFG for patternIndex() 139
7.3 du-path sets for each variable in patternIndex() 140
7.4 Test paths to satisfy all du-paths coverage on patternIndex() 142
7.5 Test paths and du-paths covered in patternIndex() 143
8.1 DNF fault classes 201
8.2 Reachability for Thermostat predicates 211
8.3 Clauses in the Thermostat predicate on lines 28–30 212
8.4 Correlated active clause coverage for Thermostat 215
8.5 Correlated active clause coverage for cal() preconditions 225
8.6 Predicates from memory seat example 229
9.1 Java’s access levels 261
10.1 Testing objectives and activities during requirements analysis and
specification 287
10.2 Testing objectives and activities during system and software
design 288
10.3 Testing objectives and activities during intermediate design 289
10.4 Testing objectives and activities during detailed design 289

xii
 List of Tables xiii

10.5 Testing objectives and activities during implementation 290

10.6 Testing objectives and activities during integration 290
10.7 Testing objectives and activities during system deployment 291
10.8 Testing objectives and activities during operation and maintenance 291
 Preface to the Second Edition

Much has changed in the field of testing in the eight years since the first edition was
published. High-quality testing is now more common in industry. Test automation
is now ubiquitous, and almost assumed in large segments of the industry. Agile
processes and test-driven development are now widely known and used. Many
more colleges offer courses on software testing, both at the undergraduate and
graduate levels. The ACM curriculum guidelines for software engineering include
software testing in several places, including as a strongly recommended course
[Ardis et al., 2015].
The second edition of Introduction to Software Testing incorporates new features
and material, yet retains the structure, philosophy, and online resources that have
been so popular among the hundreds of teachers who have used the book.

What is new about the second edition?

The first thing any instructor has to do when presented with a new edition of a book
is analyze what must be changed in the course. Since we have been in that situation
many times, we want to make it as easy as possible for our audience. We start with
a chapter-to-chapter mapping.

First Edition Second Edition Topic

Part I: Foundations
Chapter 01 Why do we test software? (motivation)
Chapter 02 Model-driven test design (abstraction)
Chapter 1 Chapter 03 Test automation (JUnit)
Chapter 04 Putting testing first (TDD)
Chapter 05 Criteria-based test design (criteria)
Part II: Coverage Criteria
Chapter 2 Chapter 07 Graph coverage
Chapter 3 Chapter 08 Logic coverage
Chapter 4 Chapter 09 Syntax-based testing
Chapter 5 Chapter 06 Input space partitioning

xiv
 Preface to the Second Edition xv

Part III: Testing in Practice

Chapter 10 Managing the test process
Chapter 11 Writing test plans
Chapter 6 Chapter 12 Test implementation
Chapter 13 Regression testing for evolving software
Chapter 14 Writing effective test oracles
Chapter 7 N/A Technologies
Chapter 8 N/A Tools
Chapter 9 N/A Challenges

The most obvious, and largest change, is that the introductory chapter 1 from
the first edition has been expanded into five separate chapters. This is a significant
expansion that we believe makes the book much better. The new part 1 grew out
of our lectures. After the first edition came out, we started adding more founda-
tional material to our testing courses. These new ideas were eventually reorganized
into five new chapters. The new chapter 011 has much of the material from the first
edition chapter 1, including motivation and basic definitions. It closes with a discus-
sion of the cost of late testing, taken from the 2002 RTI report that is cited in every
software testing research proposal. After completing the first edition, we realized
that the key novel feature of the book, viewing test design as an abstract activity
that is independent of the software artifact being used to design the tests, implied a
completely different process. This led to chapter 02, which suggests how test criteria
can fit into practice. Through our consulting, we have helped software companies
modify their test processes to incorporate this model.
A flaw with the first edition was that it did not mention JUnit or other test
automation frameworks. In 2016, JUnit is used very widely in industry, and is
commonly used in CS1 and CS2 classes for automated grading. Chapter 03 recti-
fies this oversight by discussing test automation in general, the concepts that make
test automation difficult, and explicitly teaches JUnit. Although the book is largely
technology-neutral, having a consistent test framework throughout the book helps
with examples and exercises. In our classes, we usually require tests to be auto-
mated and often ask students to try other “*-Unit” frameworks such as HttpUnit
as homework exercises. We believe that test organizations cannot be ready to apply
test criteria successfully before they have automated their tests.
Chapter 04 goes to the natural next step of test-driven development. Although
TDD is a different take on testing than the rest of the book, it’s an exciting topic for
test educators and researchers precisely because it puts testing front and center—
the tests become the requirements. Finally, chapter 05 introduces the concept of
test criteria in an abstract way. The jelly bean example (which our students love,
especially when we share), is still there, as are concepts such as subsumption.
Part 2, which is the heart of the book, has changed the least for the second edi-
tion. In 2014, Jeff asked Paul a very simple question: “Why are the four chapters in
part 2 in that order?” The answer was stunned silence, as we realized that we had
never asked which order they should appear in. It turns out that the RIPR model,

1 To help reduce confusion, we developed the convention of using two digits for second edition chapters.
Thus, in this preface, chapter 01 implies the second edition, whereas chapter 1 implies the first.
xvi Preface to the Second Edition

which is certainly central to software testing, dictates a logical order. Specifically,

input space partitioning does not require reachability, infection, or propagation.
Graph coverage criteria require execution to “get to” some location in the soft-
ware artifact under test, that is, reachability, but not infection or propagation. Logic
coverage criteria require that a predicate not only be reached, but be exercised in
a particular way to affect the result of the predicate. That is, the predicate must
be infected. Finally, syntax coverage not only requires that a location be reached,
and that the program state of the “mutated” version be different from the original
version, but that difference must be visible after execution finishes. That is, it must
propagate. The second edition orders these four concepts based on the RIPR model,
where each chapter now has successively stronger requirements. From a practical
perspective, all we did was move the previous chapter 5 (now chapter 06) in front of
the graph chapter (now chapter 07).
Another major structural change is that the second edition does not include
chapters 7 through 9 from the first edition. The first edition material has become
dated. Because it is used less than other material in the book, we decided not to
delay this new edition of the book while we tried to find time to write this material.
We plan to include better versions of these chapters in a third edition.
We also made hundreds of changes at a more detailed level. Recent research
has found that in addition to an incorrect value propagating to the output, testing
only succeeds if our automated test oracle looks at the right part of the software
output. That is, the test oracle must reveal the failure. Thus, the old RIP model is
now the RIPR model. Several places in the book have discussions that go beyond
or into more depth than is strictly needed. The second edition now includes “meta
discussions,” which are ancillary discussions that can be interesting or insightful to
some students, but unnecessarily complicated for others.
The new chapter 06 now has a fully worked out example of deriving an input
domain model from a widely used Java library interface (in section 06.4). Our
students have found this helps them understand how to use the input space par-
titioning techniques. The first edition included a section on “Representing graphs
algebraically.” Although one of us found this material to be fun, we both found it
hard to motivate and unlikely to be used in practice. It also has some subtle techni-
cal flaws. Thus, we removed this section from the second edition. The new chapter
08 (logic) has a significant structural modification. The DNF criteria (formerly in
section 3.6) properly belong at the front of the chapter. Chapter 08 now starts with
semantic logic criteria (ACC and ICC) in 08.1, then proceeds to syntactic logic crite-
ria (DNF) in 08.2. The syntactic logic criteria have also changed. One was dropped
(UTPC), and CUTPNFP has been joined by MUTP and MNFP. Together, these
three criteria comprise MUMCUT.
Throughout the book (especially part 2), we have improved the examples, sim-
plified definitions, and included more exercises. When the first edition was published
we had a partial solution manual, which somehow took five years to complete. We
are proud to say that we learned from that mistake: we made (and stuck by!) a rule
that we couldn’t add an exercise without also adding a solution. The reader might
think of this rule as testing for exercises. We are glad to say that the second edition
book website debuts with a complete solution manual.
 Preface to the Second Edition xvii

The second edition also has many dozens of corrections (starting with the errata
list from the first edition book website), but including many more that we found
while preparing the second edition. The second edition also has a better index. We
put together the index for the first edition in about a day, and it showed. This time
we have been indexing as we write, and committed time near the end of the process
to specifically focus on the index. For future book writers, indexing is hard work and
not easy to turn over to a non-author!

What is still the same in the second edition?

The things that have stayed the same are those that were successful in the first
edition. The overall observation that test criteria are based on only four types of
structures is still the key organizing principle of the second edition. The second edi-
tion is also written from an engineering viewpoint, assuming that users of the book
are engineers who want to produce the highest quality software with the lowest pos-
sible cost. The concepts are well grounded in theory, yet presented in a practical
manner. That is, the book tries to make theory meet practice; the theory is sound
according to the research literature, but we also show how the theory applies in
practice.
The book is also written as a text book, with clear explanations, simple but illus-
trative examples, and lots of exercises suitable for in-class or out-of-class work. Each
chapter ends with bibliographic notes so that beginning research students can pro-
ceed to learning the deeper ideas involved in software testing. The book website
(https://cs.gmu.edu/∼offutt/softwaretest/) is rich in materials with solution manuals,
listings of all example programs in the text, high quality PowerPoint slides, and soft-
ware to help students with graph coverage, logic coverage, and mutation analysis.
Some explanatory videos are also available and we hope more will follow. The solu-
tion manual comes in two flavors. The student solution manual, with solutions to
about half the exercises, is available to everyone. The instructor solution manual
has solutions to all exercises and is only available to those who convince the authors
that they are using a book to teach a course.

Using the book in the classroom

The book chapters are built in a modular, component-based manner. Most chapters
are independent, and although they are presented in the order that we use them,
inter-chapter dependencies are few and they could be used in almost any order.
Our primary target courses at our university are a fourth-year course (SWE 437)
and a first-year graduate course (SWE 637). Interested readers can search on those
courses (“mason swe 437” or “mason swe 637”) to see our schedules and how we
use the book. Both courses are required; SWE 437 is required in the software engi-
neering concentration in our Applied Computer Science major, and SWE 637 is
required in our MS program in software engineering2 . Chapters 01 and 03 can be
used in an early course such as CS2 in two ways. First, to sensitize early students to

2 Our MS program is practical in nature, not research-oriented. The majority of students are part-time
students with five to ten years of experience in the software industry. SWE 637 begat this book when
we realized Beizer’s classic text [Beizer, 1990] was out of print.
xviii Preface to the Second Edition

the importance of software quality, and second to get them started with test automa-
tion (we use JUnit at Mason). A second-year course in testing could cover all of
part 1, chapter 06 from part 2, and all or part of part 3. The other chapters in part
2 are probably more than what such students need, but input space partitioning is
a very accessible introduction to structured, high-end testing. A common course in
north American computer science programs is a third-year course on general soft-
ware engineering. Part 1 would be very appropriate for such a course. In 2016 we
are introducing an advanced graduate course on software testing, which will span
cutting-edge knowledge and current research. This course will use some of part
3, the material that we are currently developing for part 4, and selected research
papers.

Teaching software testing

Both authors have become students of teaching over the past decade. In the early
2000s, we ran fairly traditional classrooms. We lectured for most of the available
class time, kept organized with extensive PowerPoint slides, required homework
assignments to be completed individually, and gave challenging, high-pressure
exams. The PowerPoint slides and exercises in the first edition were designed for
this model.
However, our teaching has evolved. We replaced our midterm exam with weekly
quizzes, given in the first 15 minutes of class. This distributed a large component of
the grade through the semester, relieved much of the stress of midterms, encouraged
the students to keep up on a weekly basis instead of cramming right before the exam,
and helped us identify students who were succeeding or struggling early in the term.
After learning about the “flipped classroom” model, we experimented with
recorded lectures, viewed online, followed by doing the “homework” assignments in
class with us available for immediate help. We found this particularly helpful with
the more mathematically sophisticated material such as logic coverage, and espe-
cially beneficial to struggling students. As the educational research evidence against
the benefits of lectures has mounted, we have been moving away from the “sage
on a stage” model of talking for two hours straight. We now often talk for 10 to 20
minutes, then give in-class exercises3 where the students immediately try to solve
problems or answer questions. We confess that this is difficult for us, because we
love to talk! Or, instead of showing an example during our lecture, we introduce
the example, let the students work the next step in small groups, and then share
the results. Sometimes our solutions are better, sometimes theirs are better, and
sometimes solutions differ in interesting ways that spur discussion.
There is no doubt that this approach to teaching takes time and cannot acccomo-
date all of the PowerPoint slides we have developed. We believe that although we
cover less material, we uncover more, a perception consistent with how our students
perform on our final exams.
Most of the in-class exercises are done in small groups. We also encourage stu-
dents to work out-of-class assignments collaboratively. Not only does evidence show

3 These in-class exercises are not yet a formal part of the book website. But we often draw them from
regular exercises in the text. Interested readers can extract recent versions from our course web pages
with a search engine.
 Preface to the Second Edition xix

that students learn more when they work collaboratively (“peer-learning”), they
enjoy it more, and it matches the industrial reality. Very few software engineers
work alone.
Of course, you can use this book in your class as you see fit. We offer these
insights simply as examples for things that work for us. We summarize our current
philosophy of teaching simply: Less talking, more teaching.

Acknowledgments
It is our pleasure to acknowledge by name the many contributers to this text. We
begin with students at George Mason who provided excellent feedback on early
draft chapters from the second edition: Firass Almiski, Natalia Anpilova, Khalid
Bargqdle, Mathew Fadoul, Mark Feghali, Angelica Garcia, Mahmoud Hammad,
Husam Hilal, Carolyn Koerner, Han-Tsung Liu, Charon Lu, Brian Mitchell, Tuan
Nguyen, Bill Shelton, Dzung Tran, Dzung Tray, Sam Tryon, Jing Wu, Zhonghua
Xi, and Chris Yeung.
We are particularly grateful to colleagues who used draft chapters of the sec-
ond edition. These early adopters provided valuable feedback that was extremely
helpful in making the final document classroom-ready. Thanks to: Moataz Ahmed,
King Fahd University of Petroleum & Minerals; Jeff Carver, University of Alabama;
Richard Carver, George Mason University; Jens Hannemann, Kentucky State
University; Jane Hayes, University of Kentucky; Kathleen Keogh, Federation Uni-
versity Australia; Robyn Lutz, Iowa State University; Upsorn Praphamontripong,
George Mason University; Alper Sen, Bogazici University; Marjan Sirjani, Reyk-
javik University; Mary Lou Soffa, University of Virginia; Katie Stolee, North
Carolina State University; and Xiaohong Wang, Salisbury University.
Several colleagues provided exceptional feedback from the first edition: Andy
Brooks, Mark Hampton, Jian Zhou, Jeff (Yu) Lei, and six anonymous review-
ers contacted by our publisher. The following individuals corrected, and in some
cases developed, exercise solutions: Sana’a Alshdefat, Yasmine Badr, Jim Bowring,
Steven Dastvan, Justin Donnelly, Martin Gebert, JingJing Gu, Jane Hayes, Rama
Kesavan, Ignacio Martín, Maricel Medina-Mora, Xin Meng, Beth Paredes, Matt
Rutherford, Farida Sabry, Aya Salah, Hooman Safaee, Preetham Vemasani, and
Greg Williams. The following George Mason students found, and often corrected,
errors in the first edition: Arif Al-Mashhadani, Yousuf Ashparie, Parag Bhagwat,
Firdu Bati, Andrew Hollingsworth, Gary Kaminski, Rama Kesavan, Steve Kinder,
John Krause, Jae Hyuk Kwak, Nan Li, Mohita Mathur, Maricel Medina Mora,
Upsorn Praphamontripong, Rowland Pitts, Mark Pumphrey, Mark Shapiro, Bill
Shelton, David Sracic, Jose Torres, Preetham Vemasani, Shuang Wang, Lance
Witkowski, Leonard S. Woody III, and Yanyan Zhu. The following individuals from
elsewhere found, and often corrected, errors in the first edition: Sana’a Alshde-
fat, Alexandre Bartel, Don Braffitt, Andrew Brooks, Josh Dehlinger, Gordon
Fraser, Rob Fredericks, Weiyi Li, Hassan Mirian, Alan Moraes, Miika Nurmi-
nen, Thomas Reinbacher, Hooman Rafat Safaee, Hossein Saiedian, Aya Salah, and
Markku Sakkinen. Lian Yu of Peking University translated the the first edition into
Mandarin Chinese.
xx Preface to the Second Edition

We also want to acknowledge those who implicitly contributed to the second

edition by explicitly contributing to the first edition: Aynur Abdurazik, Muham-
mad Abdulla, Roger Alexander, Lionel Briand, Renee Bryce, George P. Burdell,
Guillermo Calderon-Meza, Jyothi Chinman, Yuquin Ding, Blaine Donley, Patrick
Emery, Brian Geary, Hassan Gomaa, Mats Grindal, Becky Hartley, Jane Hayes,
Mark Hinkle, Justin Hollingsworth, Hong Huang, Gary Kaminski, John King,
Yuelan Li, Ling Liu, Xiaojuan Liu, Chris Magrin, Darko Marinov, Robert Nils-
son, Andrew J. Offutt, Buzz Pioso, Jyothi Reddy, Arthur Reyes, Raimi Rufai,
Bo Sanden, Jeremy Schneider, Bill Shelton, Michael Shin, Frank Shukis, Greg
Williams, Quansheng Xiao, Tao Xie, Wuzhi Xu, and Linzhen Xue.
While developing the second edition, our graduate teaching assistants at George
Mason gave us fantastic feedback on early drafts of chapters: Lin Deng, Jingjing Gu,
Nan Li, and Upsorn Praphamontripong. In particular, Nan Li and Lin Deng were
instrumental in completing, evolving, and maintaining the software coverage tools
available on the book website.
We are grateful to our editor, Lauren Cowles, for providing unwavering support
and enforcing the occasional deadline to move the project along, as well as Heather
Bergmann, our former editor, for her strong support on this long-running project.
Finally, of course none of this is possible without the support of our families.
Thanks to Becky, Jian, Steffi, Matt, Joyce, and Andrew for helping us stay balanced.
Just as all programs contain faults, all texts contain errors. Our text is no
different. And, as responsibility for software faults rests with the developers,
responsibility for errors in this text rests with us, the authors. In particular, the bibli-
ographic notes sections reflect our perspective of the testing field, a body of work we
readily acknowledge as large and complex. We apologize in advance for omissions,
and invite pointers to relevant citations.
PART 1

Foundations
 1

Why Do We Test Software?

The true subject matter of the tester is not testing, but the design of test cases.

The purpose of this book is to teach software engineers how to test. This knowledge
is useful whether you are a programmer who needs to unit test your own software, a
full-time tester who works mostly from requirements at the user level, a manager in
charge of testing or development, or any position in between. As the software indus-
try moves into the second decade of the 21st century, software quality is increasingly
becoming essential to all businesses and knowledge of software testing is becoming
necessary for all software engineers.
Today, software defines behaviors that our civilization depends on in systems
such as network routers, financial calculation engines, switching networks, the Web,
power grids, transportation systems, and essential communications, command, and
control services. Over the past two decades, the software industry has become much
bigger, is more competitive, and has more users. Software is an essential component
of exotic embedded applications such as airplanes, spaceships, and air traffic control
systems, as well as mundane appliances such as watches, ovens, cars, DVD players,
garage door openers, mobile phones, and remote controllers. Modern households
have hundreds of processors, and new cars have over a thousand; all of them running
software that optimistic consumers assume will never fail! Although many factors
affect the engineering of reliable software, including, of course, careful design and
sound process management, testing is the primary way industry evaluates software
during development. The recent growth in agile processes puts increased pressure
on testing; unit testing is emphasized heavily and test-driven development makes
tests key to functional requirements. It is clear that industry is deep into a revolution
in what testing means to the success of software products.
Fortunately, a few basic software testing concepts can be used to design tests
for a large variety of software applications. A goal of this book is to present these
concepts in such a way that students and practicing engineers can easily apply them
to any software testing situation.
This textbook differs from other software testing books in several respects. The
most important difference is in how it views testing techniques. In his landmark

3
4 Foundations

book Software Testing Techniques, Beizer wrote that testing is simple—all a tester
needs to do is “find a graph and cover it.” Thanks to Beizer’s insight, it became evi-
dent to us that the myriad of testing techniques present in the literature have much
more in common than is obvious at first glance. Testing techniques are typically pre-
sented in the context of a particular software artifact (for example, a requirements
document or code) or a particular phase of the lifecycle (for example, require-
ments analysis or implementation). Unfortunately, such a presentation obscures
underlying similarities among techniques.
This book clarifies these similarities with two innovative, yet simplifying,
approaches. First, we show how testing is more efficient and effective by using a clas-
sical engineering approach. Instead of designing and developing tests on concrete
software artifacts like the source code or requirements, we show how to develop
abstraction models, design tests at the abstract level, and then implement actual
tests at the concrete level by satisfying the abstract designs. This is the exact process
that traditional engineers use, except whereas they usually use calculus and algebra
to describe the abstract models, software engineers usually use discrete mathemat-
ics. Second, we recognize that all test criteria can be defined with a very short list
of abstract models: input domain characterizations, graphs, logical expressions, and
syntactic descriptions. These are directly reflected in the four chapters of Part II of
this book.
This book provides a balance of theory and practical application, thereby
presenting testing as a collection of objective, quantitative activities that can be
measured and repeated. The theory is based on the published literature, and pre-
sented without excessive formalism. Most importantly, the theoretical concepts are
presented when needed to support the practical activities that test engineers follow.
That is, this book is intended for all software developers.

1.1 WHEN SOFTWARE GOES BAD

As said, we consider the development of software to be engineering. And like any
engineering discipline, the software industry has its shares of failures, some spectac-
ular, some mundane, some costly, and sadly, some that have resulted in loss of life.
Before learning about software disasters, it is important to understand the differ-
ence between faults, errors, and failures. We adopt the definitions of software fault,
error, and failure from the dependability community.

Definition 1.1 Software Fault: A static defect in the software.

Definition 1.2 Software Error: An incorrect internal state that is the manifes-
tation of some fault.

Definition 1.3 Software Failure: External, incorrect behavior with respect to

the requirements or another description of the expected behavior.

Consider a medical doctor diagnosing a patient. The patient enters the doctor’s
office with a list of failures (that is, symptoms). The doctor then must discover the
 1 Why Do We Test Software? 5

fault, or root cause of the symptoms. To aid in the diagnosis, a doctor may order tests
that look for anomalous internal conditions, such as high blood pressure, an irregu-
lar heartbeat, high levels of blood glucose, or high cholesterol. In our terminology,
these anomalous internal conditions correspond to errors.
While this analogy may help the student clarify his or her thinking about faults,
errors, and failures, software testing and a doctor’s diagnosis differ in one cru-
cial way. Specifically, faults in software are design mistakes. They do not appear
spontaneously, but exist as a result of a decision by a human. Medical problems
(as well as faults in computer system hardware), on the other hand, are often
a result of physical degradation. This distinction is important because it limits
the extent to which any process can hope to control software faults. Specifically,
since no foolproof way exists to catch arbitrary mistakes made by humans, we can
never eliminate all faults from software. In colloquial terms, we can make soft-
ware development foolproof, but we cannot, and should not attempt to, make it
damn-foolproof.
For a more precise example of the definitions of fault, error, and failure, we need
to clarify the concept of the state. A program state is defined during execution of a
program as the current value of all live variables and the current location, as given
by the program counter. The program counter (PC) is the next statement in the
program to be executed and can be described with a line number in the file (PC = 5)
or the statement as a string (PC = “if (x > y)”). Most of the time, what we mean by
a statement is obvious, but complex structures such as for loops have to be treated
specially. The program line “for (i=1; i<N; i++)” actually has three statements that
can result in separate states. The loop initialization (“i=1”) is separate from the loop
test (“i<N”), and the loop increment (“i++”) occurs at the end of the loop body. As
an illustrative example, consider the following Java method:

/**
* Counts zeroes in an array
*
* @param x array to count zeroes in
* @return number of occurrences of 0 in x
* @throws NullPointerException if x is null
*/
public static int numZero (int[] x)
{
int count = 0;
for (int i = 1; i < x.length; i++)
{
if (x[i] == 0) count++;
}
return count;
}
6 Foundations

Sidebar
Programming Language Independence
This book strives to be independent of language, and most of the concepts in the book are.
At the same time, we want to illustrate these concepts with specific examples. We choose
Java, and emphasize that most of these examples would be very similar in many other
common languages.

The fault in this method is that it starts looking for zeroes at index 1 instead of
index 0, as is necessary for arrays in Java. For example, numZero ([2, 7, 0]) correctly
evaluates to 1, while numZero ([0, 7, 2]) incorrectly evaluates to 0. In both tests the
faulty statement is executed. Although both of these tests result in an error, only
the second results in failure. To understand the error states, we need to identify the
state for the method. The state for numZero() consists of values for the variables x,
count, i, and the program counter (PC). For the first example above, the state at the
loop test on the very first iteration of the loop is (x = [2, 7, 0], count = 0, i = 1, PC = “i <
x.length”). Notice that this state is erroneous precisely because the value of i should
be zero on the first iteration. However, since the value of count is coincidentally
correct, the error state does not propagate to the output, and hence the software
does not fail. In other words, a state is in error simply if it is not the expected state,
even if all of the values in the state, considered in isolation, are acceptable. More
generally, if the required sequence of states is s 0, s1 , s2 , . . . , and the actual sequence
of states is s 0 , s2 , s 3, . . . , then state s2 is in error in the second sequence. The fault
model described here is quite deep, and this discussion gives the broad view without
going into unneeded details. The exercises at the end of the section explore some of
the subtleties of the fault model.
In the second test for our example, the error state is (x = [0, 7, 2], count = 0, i
= 1, PC = “i < x.length”). In this case, the error propagates to the variable count and
is present in the return value of the method. Hence a failure results.
The term bug is often used informally to refer to all three of fault, error, and
failure. This book will usually use the specific term, and avoid using “bug.” A
favorite story of software engineering teachers is that Grace Hopper found a moth
stuck in a relay on an early computing machine, which started the use of bug as
a problem with software. It is worth noting, however, that the term bug has an
old and rich history, predating software by at least a century. The first use of bug
to generally mean a problem we were able to find is from a quote by Thomas
Edison:
It has been just so in all of my inventions. The first step is an intuition, and comes
with a burst, then difficulties arise–this thing gives out and [it is] then that ‘Bugs’–
as such little faults and difficulties are called–show themselves and months of intense
watching, study and labor are requisite.
— Thomas Edison
A very public failure was the Mars lander of September 1999, which crashed
due to a misunderstanding in the units of measure used by two modules created by
separate software groups. One module computed thruster data in English units and
 1 Why Do We Test Software? 7

forwarded the data to a module that expected data in metric units. This is a very
typical integration fault (but in this case enormously expensive, both in terms of
money and prestige).
One of the most famous cases of software killing people is the Therac-25 radi-
ation therapy machine. Software faults were found to have caused at least three
deaths due to excessive radiation. Another dramatic failure was the launch failure
of the first Ariane 5 rocket, which exploded 37 seconds after liftoff in 1996. The
low-level cause was an unhandled floating point conversion exception in an iner-
tial guidance system function. It turned out that the guidance system could never
encounter the unhandled exception when used on the Ariane 4 rocket. That is, the
guidance system function was correct for Ariane 4. The developers of the Ariane 5
quite reasonably wanted to reuse the successful inertial guidance system from the
Ariane 4, but no one reanalyzed the software in light of the substantially differ-
ent flight trajectory of the Ariane 5. Furthermore, the system tests that would have
found the problem were technically difficult to execute, and so were not performed.
The result was spectacular–and expensive!
The famous Pentium bug was an early alarm of the need for better testing,
especially unit testing. Intel introduced its Pentium microprocessor in 1994, and
a few months later, Thomas Nicely, a mathematician at Lynchburg College in Vir-
ginia, found that the chip gave incorrect answers to certain floating-point division
calculations.
The chip was slightly inaccurate for a few pairs of numbers; Intel claimed (proba-
bly correctly) that only one in nine billion division operations would exhibit reduced
precision. The fault was the omission of five entries in a table of 1,066 values (part
of the chip’s circuitry) used by a division algorithm. The five entries should have
contained the constant +2, but the entries were not initialized and contained zero
instead. The MIT mathematician Edelman claimed that “the bug in the Pentium
was an easy mistake to make, and a difficult one to catch,” an analysis that misses
an essential point. This was a very difficult mistake to find during system testing,
and indeed, Intel claimed to have run millions of tests using this table. But the table
entries were left empty because a loop termination condition was incorrect; that is,
the loop stopped storing numbers before it was finished. Thus, this would have been
a very simple fault to find during unit testing; indeed analysis showed that almost
any unit level coverage criterion would have found this multimillion dollar mistake.
The great northeast blackout of 2003 was started when a power line in Ohio
brushed against overgrown trees and shut down. This is called a fault in the power
industry. Unfortunately, the software alarm system failed in the local power com-
pany, so system operators could not understand what happened. Other lines also
sagged into trees and switched off, eventually overloading other power lines, which
then cut off. This cascade effect eventually caused a blackout throughout southeast-
ern Canada and eight states in the northeastern part of the US. This is considered the
biggest blackout in North American history, affecting 10 million people in Canada
and 40 million in the USA, contributing to at least 11 deaths and costing up to $6
billion USD.
Some software failures are felt widely enough to cause severe embarrassment
to the company. In 2011, a centralized students data management system in Korea
miscalculated the academic grades of over 29,000 middle and high school students.
8 Foundations

This led to massive confusion about college admissions and a government inves-
tigation into the software engineering practices of the software vendor, Samsung
Electronics.
A 1999 study commissioned by the U.S. National Research Council and the U.S.
President’s commission on critical infrastructure protection concluded that the cur-
rent base of science and technology is inadequate for building systems to control
critical software infrastructure. A 2002 report commissioned by the National Insti-
tute of Standards and Technology (NIST) estimated that defective software costs
the U.S. economy $59.5 billion per year. The report further estimated that 64% of
the costs were a result of user mistakes and 36% a result of design and development
mistakes, and suggested that improvements in testing could reduce this cost by about
a third, or $22.5 billion. Blumenstyk reported that web application failures lead to
huge losses in businesses; $150,000 per hour in media companies, $2.4 million per
hour in credit card sales, and $6.5 million per hour in the financial services market.
Software faults do not just lead to functional failures. According to a Symantec
security threat report in 2007, 61 percent of all vulnerabilities disclosed were due to
faulty software. The most common are web application vulnerabilities that can be
attacked by some common attack techniques using invalid inputs.
These public and expensive software failures are getting more common and
more widely known. This is simply a symptom of the change in expectations of
software. As we move further into the 21st century, we are using more safety crit-
ical, real-time software. Embedded software has become ubiquitous; many of us
carry millions of lines of embedded software in our pockets. Corporations rely more
and more on large-scale enterprise applications, which by definition have large user
bases and high reliability requirements. Security, which used to depend on cryp-
tography, then database security, then avoiding network vulnerabilities, is now
largely about avoiding software faults. The Web has had a major impact. It fea-
tures a deployment platform that offers software services that are very competitive
and available to millions of users. They are also distributed, adding complexity,
and must be highly reliable to be competitive. More so than at any previous time,
industry desperately needs to apply the accumulated knowledge of over 30 years of
testing research.

1.2 GOALS OF TESTING SOFTWARE

Surprisingly, many software engineers are not clear about their testing goals. Is it
to show correctness, find problems, or something else? To explore this concept, we
first must separate validation and verification. Most of the definitions in this book
are taken from standards documents, and although the phrasing is ours, we try to
be consistent with the standards. Useful standards for reading in more detail are the
IEEE Standard Glossary of Software Engineering Terminology, DOD-STD-2167A
and MIL-STD-498 from the US Department of Defense, and the British Computer
Society’s Standard for Software Component Testing.
Definition 1.4 Verification: The process of determining whether the prod-
ucts of a phase of the software development process fulfill the requirements
established during the previous phase.
 1 Why Do We Test Software? 9

Definition 1.5 Validation: The process of evaluating software at the end of

software development to ensure compliance with intended usage.
Verification is usually a more technical activity that uses knowledge about the
individual software artifacts, requirements, and specifications. Validation usually
depends on domain knowledge; that is, knowledge of the application for which the
software is written. For example, validation of software for an airplane requires
knowledge from aerospace engineers and pilots.
As a familiar example, consider a light switch in a conference room. Verification
asks if the lighting meets the specifications. The specifications might say something
like, “The lights in front of the projector screen can be controlled independently of
the other lights in the room.” If the specifications are written down somewhere and
the lights cannot be controlled independently, then the lighting fails verification,
precisely because the implementation does not satisfy the specifications. Validation
asks whether users are satisfied, an inherently fuzzy question that has nothing to do
with verification. If the “independent control” specification is neither written down
nor satisfied, then, despite the disappointed users, verification nonetheless succeeds,
because the implementation satisfies the specification. But validation fails, because
the specification for the lighting does not reflect the true needs of the users. This is
an important general point: validation exposes flaws in specifications.
The acronym “IV&V” stands for “Independent Verification and Validation,”
where “independent” means that the evaluation is done by non-developers. Some-
times the IV&V team is within the same project, sometimes the same company, and
sometimes it is entirely an external entity. In part because of the independent nature
of IV&V, the process often is not started until the software is complete and is often
done by people whose expertise is in the application domain rather than software
development. This can sometimes mean that validation is given more weight than
verification. This book emphasizes verification more than validation, although most
of the specific test criteria we discuss can be used for both activities.
Beizer discussed the goals of testing in terms of the “test process maturity lev-
els” of an organization, where the levels are characterized by the testers’ goals. He
defined five levels, where the lowest level is not worthy of being given a number.

Level 0 There is no difference between testing and debugging.

Level 1 The purpose of testing is to show correctness.

Level 2 The purpose of testing is to show that the software does not work.

Level 3 The purpose of testing is not to prove anything specific, but to reduce the
risk of using the software.

Level 4 Testing is a mental discipline that helps all IT professionals develop higher-
quality software.

Level 0 is the view that testing is the same as debugging. This is the view that
is naturally adopted by many undergraduate Computer Science majors. In most CS
programming classes, the students get their programs to compile, then debug the
programs with a few inputs chosen either arbitrarily or provided by the professor.
10 Foundations

This model does not distinguish between a program’s incorrect behavior and a mis-
take within the program, and does very little to help develop software that is reliable
or safe.
In Level 1 testing, the purpose is to show correctness. While a significant step
up from the naive level 0, this has the unfortunate problem that in any but the most
trivial of programs, correctness is virtually impossible to either achieve or demon-
strate. Suppose we run a collection of tests and find no failures. What do we know?
Should we assume that we have good software or just bad tests? Since the goal of
correctness is impossible, test engineers usually have no strict goal, real stopping
rule, or formal test technique. If a development manager asks how much testing
remains to be done, the test manager has no way to answer the question. In fact,
test managers are in a weak position because they have no way to quantitatively
express or evaluate their work.
In Level 2 testing, the purpose is to show failures. Although looking for failures
is certainly a valid goal, it is also inherently negative. Testers may enjoy finding the
problem, but the developers never want to find problems–they want the software to
work (yes, level 1 thinking can be natural for the developers). Thus, level 2 testing
puts testers and developers into an adversarial relationship, which can be bad for
team morale. Beyond that, when our primary goal is to look for failures, we are still
left wondering what to do if no failures are found. Is our work done? Is our software
very good, or is the testing weak? Having confidence in when testing is complete is
an important goal for all testers. It is our view that this level currently dominates the
software industry.
The thinking that leads to Level 3 testing starts with the realization that test-
ing can show the presence, but not the absence, of failures. This lets us accept
the fact that whenever we use software, we incur some risk. The risk may be
small and the consequences unimportant, or the risk may be great and the con-
sequences catastrophic, but risk is always there. This allows us to realize that the
entire development team wants the same thing–to reduce the risk of using the
software. In level 3 testing, both testers and developers work together to reduce
risk. We see more and more companies move to this testing maturity level every
year.
Once the testers and developers are on the same “team,” an organization can
progress to real Level 4 testing. Level 4 thinking defines testing as a mental disci-
pline that increases quality. Various ways exist to increase quality, of which creating
tests that cause the software to fail is only one. Adopting this mindset, test engi-
neers can become the technical leaders of the project (as is common in many other
engineering disciplines). They have the primary responsibility of measuring and
improving software quality, and their expertise should help the developers. Beizer
used the analogy of a spell checker. We often think that the purpose of a spell
checker is to find misspelled words, but in fact, the best purpose of a spell checker
is to improve our ability to spell. Every time the spell checker finds an incorrectly
spelled word, we have the opportunity to learn how to spell the word correctly.
The spell checker is the “expert” on spelling quality. In the same way, level 4 test-
ing means that the purpose of testing is to improve the ability of the developers
 1 Why Do We Test Software? 11

to produce high-quality software. The testers should be the experts who train your
developers!
As a reader of this book, you probably start at level 0, 1, or 2. Most software
developers go through these levels at some stage in their careers. If you work in
software development, you might pause to reflect on which testing level describes
your company or team. The remaining chapters in Part I should help you move to
level 2 thinking, and to understand the importance of level 3. Subsequent chapters
will give you the knowledge, skills, and tools to be able to work at level 3. An
ultimate goal of this book is to provide a philosophical basis that will allow read-
ers to become “change agents” in their organizations for level 4 thinking, and test
engineers to become software quality experts. Although level 4 thinking is cur-
rently rare in the software industry, it is common in more mature engineering
fields.
These considerations help us decide at a strategic level why we test. At a more
tactical level, it is important to know why each test is present. If you do not know
why you are conducting each test, the test will not be very helpful. What fact is each
test trying to verify? It is essential to document test objectives and test requirements,
including the planned coverage levels. When the test manager attends a planning
meeting with the other managers and the project manager, the test manager must be
able to articulate clearly how much testing is enough and when testing will complete.
In the 1990s, we could use the “date criterion,” that is, testing is “complete” when
the ship date arrives or when the budget is spent.
Figure 1.1 dramatically illustrates the advantages of testing early rather than
late. This chart is based on a detailed analysis of faults that were detected and fixed
during several large government contracts. The bars marked ‘A’ indicate what per-
centage of faults appeared in that phase. Thus, 10% of faults appeared during the
requirements phase, 40% during design, and 50% during implementation. The bars
marked ‘D’ indicated the percentage of faults that were detected during each phase.
About 5% were detected during the requirements phase, and over 35% during sys-
tem testing. Lastly is the cost analysis. The solid bars marked ‘C’ indicate the relative
cost of finding and fixing faults during each phase. Since each project was different,
this is averaged to be based on a “unit cost.” Thus, faults detected and fixed during
requirements, design, and unit testing were a single unit cost. Faults detected and
fixed during integration testing cost five times as much, 10 times as much during
system testing, and 50 times as much after the software is deployed.
If we take the simple assumption of $1000 USD unit cost per fault, and 100
faults, that means we spend $39,000 to find and correct faults during requirements,
design, and unit testing. During integration testing, the cost goes up to $100,000.
But system testing and deployment are the serious problems. We find more faults
during system testing at ten times the cost, for a total of $360,000. And even though
we only find a few faults after deployment, the cost being 50 X unit means we spend
$250,000! Avoiding the work early (requirements analysis and unit testing) saves
money in the short term. But it leaves faults in software that are like little bombs,
ticking away, and the longer they tick, the bigger the explosion when they finally
go off.
12 Foundations

Figure 1.1. Cost of late testing.

To put Beizer’s level 4 test maturity level in simple terms, the goal of testing is
to eliminate faults as early as possible. We can never be perfect, but every time we
eliminate a fault during unit testing (or sooner!), we save money. The rest of this
book will teach you how to do that.

EXERCISES
Chapter 1.
1. What are some factors that would help a development organization move
from Beizer’s testing level 2 (testing is to show errors) to testing level 4
(a mental discipline that increases quality)?
2. What is the difference between software fault and software failure?
3. What do we mean by “level 3 thinking is that the purpose of testing is to reduce
risk?” What risk? Can we reduce the risk to zero?
4. The following exercise is intended to encourage you to think of testing in a
more rigorous way than you may be used to. The exercise also hints at the
strong relationship between specification clarity, faults, and test cases 1.
(a) Write a Java method with the signature
public static Vector union (Vector a, Vector b)
The method should return a Vector of objects that are in either of the two
argument Vectors.
(b) Upon reflection, you may discover a variety of defects and ambiguities
in the given assignment. In other words, ample opportunities for faults
exist. Describe as many possible faults as you can. (Note: Vector is a Java
Collection class. If you are using another language, interpret Vector as a list.)
(c) Create a set of test cases that you think would have a reasonable chance of
revealing the faults you identified above. Document a rationale for each
test in your test set. If possible, characterize all of your rationales in some
concise summary. Run your tests against your implementation.

1 Liskov’s Program Development in Java, especially chapters 9 and 10, is a great source for students who
wish to learn more about this.
 1 Why Do We Test Software? 13

(d) Rewrite the method signature to be precise enough to clarify the defects
and ambiguities identified earlier. You might wish to illustrate your
specification with examples drawn from your test cases.
5. Below are four faulty programs. Each includes test inputs that result in failure.
Answer the following questions about each program.

/** /**
* Find last index of element * Find last index of zero
* *
* @param x array to search * @param x array to search
* @param y value to look for *
* @return last index of y in x; -1 if absent * @return last index of 0 in x; -1 if absent
* @throws NullPointerException if x is null * @throws NullPointerException if x is null
*/ */
public int findLast (int[] x, int y) public static int lastZero (int[] x)
{ {
for (int i=x.length-1; i > 0; i--) for (int i = 0; i < x.length; i++)
{ {
if (x[i] == y) if (x[i] == 0)
{ {
return i; return i;
} }
} }
return -1; return -1;
} }
// test: x = [2, 3, 5]; y = 2; Expected = 0 // test: x = [0, 1, 0]; Expected = 2
// Book website: FindLast.java // Book website: LastZero.java
// Book website: FindLastTest.java // Book website: LastZeroTest.java

/** /**
* Count positive elements * Count odd or positive elements
* *
* @param x array to search * @param x array to search
* @return count of positive elements in x * @return count of odd/positive values in x
* @throws NullPointerException if x is null * @throws NullPointerException if x is null
*/ */
public int countPositive (int[] x) public static int oddOrPos(int[] x)
{ {
int count = 0; int count = 0;
for (int i=0; i < x.length; i++) for (int i = 0; i < x.length; i++)
{ {
if (x[i] >= 0) if (x[i]%2 == 1 || x[i] > 0)
{ {
count++; count++;
} }
} }
return count; return count;
} }
// test: x = [-4, 2, 0, 2]; Expected = 2 // test: x = [-3, -2, 0, 1, 4]; Expected = 3
// Book website: CountPositive.java // Book website: OddOrPos.java
// Book website: CountPositiveTest.java // Book website: OddOrPosTest.java
14 Foundations

(a) Explain what is wrong with the given code. Describe the fault precisely
by proposing a modification to the code.
(b) If possible, give a test case that does not execute the fault. If not, briefly
explain why not.
(c) If possible, give a test case that executes the fault, but does not result in
an error state. If not, briefly explain why not.
(d) If possible give a test case that results in an error, but not a failure. If not,
briefly explain why not. Hint: Don’t forget about the program counter.
(e) For the given test case, describe the first error state. Be sure to describe
the complete state.
(f) Implement your repair and verify that the given test now produces the
expected output. Submit a screen printout or other evidence that your
new program works.
6. Answer question (a) or (b), but not both, depending on your background.
(a) If you do, or have, worked for a software development company, what
level of test maturity do you think the company worked at? (0: test-
ing=debugging, 1: testing shows correctness, 2: testing shows the program
doesn’t work, 3: testing reduces risk, 4: testing is a mental discipline about
quality).
(b) If you have never worked for a software development company, what
level of test maturity do you think that you have? (0: testing=debugging,
1: testing shows correctness, 2: testing shows the program doesn’t work,
3: testing reduces risk, 4: testing is a mental discipline about quality).
7. Consider the following three example classes. These are OO faults taken
from Joshua Bloch’s Effective Java, Second Edition. Answer the following
questions about each.

class Vehicle implements Cloneable

{
private int x;
public Vehicle (int y) { x = y;}
public Object clone()
{
Object result = new Vehicle (this.x);
// Location "A"
return result;
}
// other methods omitted
}
class Truck extends Vehicle
{
private int y;
public Truck (int z) { super (z); y = z;}
public Object clone()
 1 Why Do We Test Software? 15

{
Object result = super.clone();
// Location "B"
((Truck) result).y = this.y; // throws ClassCastException
return result;
}
// other methods omitted
}
//Test: Truck suv = new Truck (4); Truck co = suv.clone()
// Expected: suv.x = co.x; suv.getClass() = co.getClass()

Note: Relevant to Bloch, Item 11 page 54.

Book website: Vehicle.java, Truck.java, CloneTest.java

public class BigDecimalTest

{
BigDecimal x = new BigDecimal ("1.0");
BigDecimal y = new BigDecimal ("1.00");
// Fact: !x.equals (y), but x.compareTo (y) == 0

Set <BigDecimal> BigDecimalTree = new TreeSet <BigDecimal> ();

BigDecimalTree.add (x);
BigDecimalTree.add (y);
// TreeSet uses compareTo(), so BigDecimalTree now has 1 element

Set <BigDecimal> BigDecimalHash = new HashSet <BigDecimal> ();

BigDecimalHash.add (x);
BigDecimalHash.add (y);
// HashSet uses equals(), so BigDecimalHash now has 2 elements
}
// Test: System.out.println ("BigDecimalTree = " + BigDecimalTree);
// System.out.println ("BigDecimalHash = " + BigDecimalHash);
// Expected: BigDecimalTree = 1; BigDecimalHash = 1

// See Java Doc for add() in Set Interface

// The problem is that in BigDecimal, equals() and compareTo()
// are inconsistent. Let’s suppose we decide that compareTo() is correct,
// and that equals()is faulty.

Note: Relevant to Bloch, Item 12 page 62.

Book website: class BigDecimalTest.java
16 Foundations

class Point
{
private int x; private int y;
public Point (int x, int y) { this.x=x; this.y=y; }

@Override public boolean equals (Object o)

{
// Location A
if (!(o instanceof Point)) return false;
Point p = (Point) o;
return (p.x == this.x) && (p.y == this.y);
}
}
class ColorPoint extends Point
{
private Color color;
// Fault: Superclass instantiable; subclass state extended

public ColorPoint (int x, int y, Color color)

{
super (x,y);
this.color = color;
}

@Override public boolean equals (Object o)

{
// Location B
if (!(o instanceof ColorPoint)) return false;
ColorPoint cp = (ColorPoint) o;
return (super.equals(cp) && (cp.color == this.color));
}
// Tests:
Point p = new Point (1,2);
ColorPoint cp1 = new ColorPoint (1,2,RED);
ColorPoint cp2 = new ColorPoint (1,2,BLUE);
p.equals (cp1); // Test 1: Result = true;
cp1.equals (p); // Test 2: Result = false;
cp1.equals (cp2); // Test 3: Result = false;
// Expected: p.equals (cp1) = true; cp1.equals (p) = true,
// cp1.equals (cp2) = false

Note: Relevant to Bloch, Item 17 page 87.

Book website: Point.java, ColorPoint.java, PointTest.java
 1 Why Do We Test Software? 17

(a) Explain what is wrong with the given code. Describe the fault precisely by
proposing a modification to the code.
(b) If possible, give a test case that does not execute the fault. If not, briefly
explain why not.
(c) If possible, give a test case that executes the fault, but does not result in an
error state. If not, briefly explain why not.
(d) If possible give a test case that results in an error, but not a failure. If not,
briefly explain why not. Hint: Don’t forget about the program counter.
(e) In the given code, describe the first error state. Be sure to describe the
complete state.
(f) Implement your repair and verify that the given test now produces the
expected output. Submit a screen printout or other evidence that your new
program works.

1.3 BIBLIOGRAPHIC NOTES

This textbook has been deliberately left uncluttered with references. Instead, each
chapter contains a Bibliographic Notes section, which contains suggestions for fur-
ther and deeper reading for readers who want more. We especially hope that
research students will find these sections helpful.
Most of the terminology in testing is from standards documents, including
the IEEE Standard Glossary of Software Engineering Terminology [IEEE, 2008],
the US Department of Defense [Department of Defense, 1988, Department
of Defense, 1994], the US Federal Aviation Administration FAA-DO178B,
and the British Computer Society’s Standard for Software Component Testing
[British Computer Society, 2001].
Beizer [Beizer, 1990] first defined the testing levels in Section 1.2. Beizer
described them in terms of the maturity of individual developers and used the term
phase instead of level. We adapted the discussion to organizations rather than indi-
vidual developers and chose the term level to mirror the language of the well-known
Capability Maturity Model [Paulk et al., 1995].
All books on software testing and all researchers owe major thanks to the land-
mark books in 1979 by Myers [Myers, 1979], in 1990 by Beizer [Beizer, 1990], and
in 2000 by Binder [Binder, 2000]. Some excellent overviews of unit testing criteria
have also been published, including one by White [White, 1987] and more recently
by Zhu, Hall, and May [Zhu et al., 1997]. The recent text from Pezze and Young
[Pezze and Young, 2008] reports relevant processes, principles, and techniques from
the testing literature, and includes many useful classroom materials. The Pezze and
Young text presents coverage criteria in the traditional lifecycle-based manner, and
does not organize criteria into the four abstract models discussed in this chapter.
Another recent book by Mathur offers a comprehensive, in-depth catalog of test
techniques and criteria [Mathur, 2014].
Numerous other software testing books were not intended as textbooks, or do
not offer general coverage for classroom use. Beizer’s Software System Testing and
18 Foundations

Quality Assurance [Beizer, 1984] and Hetzel’s The Complete Guide to Software Test-
ing [Hetzel, 1988] cover various aspects of management and process for software
testing. Several books cover specific aspects of testing [Howden, 1987, Marick, 1995,
Roper, 1994]. The STEP project at Georgia Institute of Technology resulted in a
comprehensive survey of the practice of software testing by Department of Defense
contractors in the 1980s [DeMillo et al., 1987].
The information for the Pentium bug and Mars lander was taken from sev-
eral sources, including by Edelman, Moler, Nuseibeh, Knutson, and Peterson
[Edelman, 1997, Knutson and Carmichael, 2000, Moler, 1995, Nuseibeh, 1997,
Peterson, 1997]. The well-written official accident report [Lions, 1996] is our favorite
source for understanding the details of the Ariane 5 Flight 501 Failure. The infor-
mation for the Therac-25 accidents was taken from Leveson and Turner’s deep
analysis [Leveson and Turner, 1993]. The details on the 2003 Northeast Black-
out was taken from Minkel’s analysis in Scientific American [Minkel, 2008] and
Rice’s book [Rice, 2008]. The information about the Korean education informa-
tion system was taken from two newspaper articles [Min-sang and Sang-soo, 2011,
Korea Times, 2011].
The 1999 study mentioned was published in an NRC / PITAC report
[PITAC, 1999, Schneider, 1999]. The data in Figure 1.1 were taken from a NIST
report that was developed by the Research Triangle Institute [RTI, 2002]. The
figures on web application failures are due to Blumenstyk [Blumenstyk, 2006]. The
figures about faulty software leading to security vulnerabilities are from Symantec
[Symantec, 2007].
Finally, Rick Hower’s QATest website is a good resource for current, elemen-
tary, information about software testing: www.softwareqatest.com.
 2

Model-Driven Test Design

Designers are more efficient and effective if they can raise their level of abstraction.

This chapter introduces one of the major innovations in the second edition of Intro-
duction to Software Testing. Software testing is inherently complicated and our
ultimate goal, completely correct software, is unreachable. The reasons are formal
(as discussed below in section 2.1) and philosophical. As discussed in Chapter 1, it’s
not even clear that the term “correctness” means anything when applied to a piece
of engineering as complicated as a large computer program. Do we expect correct-
ness out of a building? A car? A transportation system? Intuitively, we know that
all large physical engineering systems have problems, and moreover, there is no way
to say what correct means. This is even more true for software, which can quickly
get orders of magnitude more complicated than physical structures such as office
buildings or airplanes.
Instead of looking for “correctness,” wise software engineers try to evaluate soft-
ware’s “behavior” to decide if the behavior is acceptable within consideration of a
large number of factors including (but not limited to) reliability, safety, maintain-
ability, security, and efficiency. Obviously this is more complex than the naive desire
to show the software is correct.
So what do software engineers do in the face of such overwhelming complexity?
The same thing that physical engineers do–we use mathematics to “raise our level of
abstraction.” The Model-Driven Test Design (MDTD) process breaks testing into
a series of small tasks that simplify test generation. Then test designers isolate their
task, and work at a higher level of abstraction by using mathematical engineering
structures to design test values independently of the details of software or design
artifacts, test automation, and test execution.
A key intellectual step in MDTD is test case design. Test case design can be the
primary determining factor in whether tests successfully find failures in software.
Tests can be designed with a “human-based” approach, where a test engineer uses
domain knowledge of the software’s purpose and his or her experience to design
tests that will be effective at finding faults. Alternatively, tests can be designed to sat-
isfy well-defined engineering goals such as coverage criteria. This chapter describes

19
20 Foundations

the task activities and then introduces criteria-based test design. Criteria-based test
design will be discussed in more detail in Chapter 5, then specific criteria on four
mathematical structures are described in Part II. After these preliminaries, the
model-driven test design process is defined in detail. The book website has sim-
ple web applications that support the MDTD in the context of the mathematical
structures in Part II.

2.1 SOFTWARE TESTING FOUNDATIONS

One of the most important facts that all software testers need to know is that test-
ing can show only the presence of failures, not their absence. This is a fundamental,
theoretical limitation; to be precise, the problem of finding all failures in a pro-
gram is undecidable. Testers often call a test successful (or effective) if it finds an
error. While this is an example of level 2 thinking, it is also a characterization that is
often useful and that we will use throughout the book. This section explores some
of the theoretical underpinnings of testing as a way to emphasize how important the
MDTD is.
The definitions of fault and failure in Chapter 1 allow us to develop the
reachability, infection, propagation, and revealability model (“RIPR”). First, we
distinguish testing from debugging.

Definition 2.6 Testing: Evaluating software by observing its execution.

Definition 2.7 Test Failure: Execution of a test that results in a software

failure.

Definition 2.8 Debugging: The process of finding a fault given a failure.

Of course the central issue is that for a given fault, not all inputs will “trigger”
the fault into creating incorrect output (a failure). Also, it is often very difficult to
relate a failure to the associated fault. Analyzing these ideas leads to the fault/failure
model, which states that four conditions are needed for a failure to be observed.
Figure 2.1 illustrates the conditions. First, a test must reach the location or loca-
tions in the program that contain the fault (Reachability). After the location is
executed, the state of the program must be incorrect (Infection). Third, the infected
state must propagate through the rest of the execution and cause some output or
final state of the program to be incorrect (Propagation). Finally, the tester must
observe part of the incorrect portion of the final program state (Revealability). If
the tester only observes parts of the correct portion of the final program state, the
failure is not revealed. This is shown in the cross-hatched intersection in Figure 2.1.
Issues with revealing failures will be discussed in Chapter 4 when we present test
automation strategies.
Collectively, these four conditions are known as the fault/failure model, or the
RIPR model.
It is important to note that the RIPR model applies even when the fault is miss-
ing code (so-called faults of omission). In particular, when execution passes through
the location where the missing code should be, the program counter, which is part
of the program state, necessarily has the wrong value.
 2 Model-Driven Test Design 21

Figure 2.1. Reachability, Infection, Propagation, Revealability (RIPR) model.

From a practitioner’s view, these limitations mean that software testing is com-
plex and difficult. The common way to deal with complexity in engineering is to
use abstraction by abstracting out complicating details that can be safely ignored by
modeling the problem with some mathematical structures. That is a central theme
of this book, which we begin by analyzing the separate technical activities involved
in creating good tests.

2.2 SOFTWARE TESTING ACTIVITIES

In this book, a test engineer is an Information Technology (IT) professional who is in
charge of one or more technical test activities, including designing test inputs, pro-
ducing test case values, running test scripts, analyzing results, and reporting results
to developers and managers. Although we cast the description in terms of test engi-
neers, every engineer involved in software development should realize that he or
she sometimes wears the hat of a test engineer. The reason is that each software
artifact produced over the course of a product’s development has, or should have,
an associated set of test cases, and the person best positioned to define these test
cases is often the designer of the artifact. A test manager is in charge of one or more
test engineers. Test managers set test policies and processes, interact with other
managers on the project, and otherwise help the engineers test software effectively
and efficiently.
Figure 2.2 shows some of the major activities of test engineers. A test engineer
must design tests by creating test requirements. These requirements are then trans-
formed into actual values and scripts that are ready for execution. These executable
tests are run against the software, denoted P in the figure, and the results are eval-
uated to determine if the tests reveal a fault in the software. These activities may
22 Foundations

Test
Manager

n
desig Test instantiate Executable
Designs Tests

Test
Engineer
Test Test
Engineer Engineer

execute
P Computer Output Evaluate

Figure 2.2. Activities of test engineers.

be carried out by one person or by several, and the process is monitored by a test
manager.
One of a test engineer’s most powerful tools is a formal coverage criterion. For-
mal coverage criteria give test engineers ways to decide what test inputs to use
during testing, making it more likely that the tester will find problems in the program
and providing greater assurance that the software is of high quality and reliability.
Coverage criteria also provide stopping rules for the test engineers. The technical
core of this book presents the coverage criteria that are available, describes how
they are supported by tools (commercial and otherwise), explains how they can best
be applied, and suggests how they can be integrated into the overall development
process.
Software testing activities have long been categorized into levels, and the
most often used level categorization is based on traditional software process steps.
Although most types of tests can only be run after some part of the software is
implemented, tests can be designed and constructed during all software devel-
opment steps. The most time-consuming parts of testing are actually the test
design and construction, so test activities can and should be carried out throughout
development.

2.3 TESTING LEVELS BASED ON SOFTWARE ACTIVITY

Tests can be derived from requirements and specifications, design artifacts, or
the source code. In traditional texts, a different level of testing accompanies each
distinct software development activity:

 Acceptance Testing: assess software with respect to requirements or users’

needs.
 System Testing: assess software with respect to architectural design and overall
behavior.
 Integration Testing: assess software with respect to subsystem design.
 2 Model-Driven Test Design 23

Requirements Acceptance
Analysis Test Test

Architectural Design System

Design Test
Information

Subsystem Integration
Design Test

Detailed Module
Design Test

Implementation Unit
Test

Figure 2.3. Software development activities and testing levels – the “V Model”.

 Module Testing: assess software with respect to detailed design.

 Unit Testing: assess software with respect to implementation.

Figure 2.3, often called the “V model,” illustrates a typical scenario for testing
levels and how they relate to software development activities by isolating each step.
Information for each test level is typically derived from the associated development
activity. Indeed, standard advice is to design the tests concurrently with each devel-
opment activity, even though the software will not be in an executable form until the
implementation phase. The reason for this advice is that the mere process of design-
ing tests can identify defects in design decisions that otherwise appear reasonable.
Early identification of defects is by far the best way to reduce their ultimate cost.
Note that this diagram is not intended to imply a waterfall process. The synthesis
and analysis activities generically apply to any development process.
The requirements analysis phase of software development captures the cus-
tomer’s needs. Acceptance testing is designed to determine whether the completed
software in fact meets these needs. In other words, acceptance testing probes
whether the software does what the users want. Acceptance testing must involve
users or other individuals who have strong domain knowledge.
The architectural design phase of software development chooses components
and connectors that together realize a system whose specification is intended to
meet the previously identified requirements. System testing is designed to determine
whether the assembled system meets its specifications. It assumes that the pieces
work individually, and asks if the system works as a whole. This level of testing usu-
ally looks for design and specification problems. It is a very expensive place to find
lower-level faults and is usually not done by the programmers, but by a separate
testing team.
The subsystem design phase of software development specifies the structure and
behavior of subsystems, each of which is intended to satisfy some function in the
overall architecture. Often, the subsystems are adaptations of previously developed
24 Foundations

software. Integration testing is designed to assess whether the interfaces between

modules (defined below) in a subsystem have consistent assumptions and communi-
cate correctly. Integration testing must assume that modules work correctly. Some
testing literature uses the terms integration testing and system testing interchange-
ably; in this book, integration testing does not refer to testing the integrated system
or subsystem. Integration testing is usually the responsibility of members of the
development team.
The detailed design phase of software development determines the structure and
behavior of individual modules. A module is a collection of related units that are
assembled in a file, package, or class. This corresponds to a file in C, a package
in Ada, and a class in C++ and Java. Module testing is designed to assess individ-
ual modules in isolation, including how the component units interact with each
other and their associated data structures. Most software development organiza-
tions make module testing the responsibility of the programmer; hence the common
term developer testing.
Implementation is the phase of software development that actually produces
code. A program unit, or procedure, is one or more contiguous program statements,
with a name that other parts of the software use to call it. Units are called functions
in C and C++, procedures or functions in Ada, methods in Java, and subroutines
in Fortran. Unit testing is designed to assess the units produced by the implemen-
tation phase and is the “lowest” level of testing. In some cases, such as when
building general-purpose library modules, unit testing is done without knowledge
of the encapsulating software application. As with module testing, most software
development organizations make unit testing the responsibility of the programmer,
again, often called developer testing. It is straightforward to package unit tests
together with the corresponding code through the use of tools such as JUnit for Java
classes.
Because of the many dependencies among methods in classes, it is common
among developers using object-oriented (OO) software to combine unit and module
testing and use the term unit testing or developer testing.
Not shown in Figure 2.3 is regression testing, a standard part of the mainte-
nance phase of software development. Regression testing is done after changes are
made to the software, to help ensure that the updated software still possesses the
functionality it had before the updates.
Mistakes in requirements and high-level design end up being implemented as
faults in the program; thus testing can reveal them. Unfortunately, the software
faults that come from requirements and design mistakes are visible only through
testing months or years after the original mistake. The effects of the mistake tend
to be dispersed throughout multiple software components; hence such faults are
usually difficult to pin down and expensive to correct. On the positive side, even if
tests cannot be executed, the very process of defining tests can identify a significant
fraction of the mistakes in requirements and design. Hence, it is important for test
planning to proceed concurrently with requirements analysis and design and not
be put off until late in a project. Fortunately, through techniques such as use case
analysis, test planning is becoming better integrated with requirements analysis in
standard software practice.
 2 Model-Driven Test Design 25

Although most of the literature emphasizes these levels in terms of when they
are applied, a more important distinction is on the types of faults that we are looking
for. The faults are based on the software artifact that we are testing, and the software
artifact that we derive the tests from. For example, unit and module tests are derived
to test units and modules, and we usually try to find faults that can be found when
executing the units and modules individually.
One final note is that OO software changes the testing levels. OO software blurs
the distinction between units and modules, so the OO software testing literature
has developed a slight variation of these levels. Intra-method testing evaluates indi-
vidual methods. Inter-method testing evaluates pairs of methods within the same
class. Intra-class testing evaluates a single entire class, usually as sequences of calls
to methods within the class. Finally, inter-class testing evaluates more than one class
at the same time. The first three are variations of unit and module testing, whereas
inter-class testing is a type of integration testing.

2.4 COVERAGE CRITERIA

The essential problem with testing is the numbers. Even a small program has a huge
number of possible inputs. Consider a tiny method that computes the average of
three integers. We have only three input variables, but each can have any value
between -MAXINT and +MAXINT. On a 32-bit machine, each variable has a pos-
sibility of over 4 billion values. With three inputs, this means the method has over
80 Octillion possible inputs!
So no matter whether we are doing unit testing, integration testing, or system
testing, it is impossible to test with all inputs. The input space is, to all practical pur-
poses, infinite. Thus a test designer’s goal could be summarized in a very high-level
way as searching a huge input space, hoping to find the fewest tests that will reveal
the most problems. This is the source of two key problems in testing: (1) how do
we search? and (2) when do we stop? Coverage criteria give us structured, practical
ways to search the input space. Satisfying a coverage criterion gives a tester some
amount of confidence in two crucial goals: (A) we have looked in many corners of
the input space, and (B) our tests have a fairly low amount of overlap.
Coverage criteria have many advantages for improving the quality and reduc-
ing the cost of test data generation. Coverage criteria can maximize the “bang for
the buck,” with fewer tests that are effective at finding more faults. Well-designed
criteria-based tests will be comprehensive, yet factor out unwanted redundancy.
Coverage criteria also provide traceability from software artifacts such as source,
design models, requirements, and input space descriptions. This supports regression
testing by making it easier to decide which tests need to be reused, modified, or
deleted. From an engineering perspective, one of the strongest benefits of coverage
criteria is they provide a “stopping rule” for testing; that is, we know in advance
approximately how many tests are needed and we know when we have “enough”
tests. This is a powerful tool for engineers and managers.
Coverage criteria also lend themselves well to automation. As we will formal-
ize in Chapter 5, a test requirement is a specific element of a software artifact that
a test case must satisfy or cover, and a coverage criterion is a rule or collection
of rules that yield test requirements. For example, the coverage criterion “cover
Other documents randomly have
different content
leave them as we were to go.
The journey from Chieng Kawng was intensely hot; the
thermometer standing at 103° in my howdah by day, and on one
night in my tent at 96°. On the banks of the Mê Ing I found native
white roses in bloom in abundance, and brought home with me a
plant which Mrs. McGilvary greatly prized, for this was the only
native rose I had found in the Lāo territory.
On the way to Mûang Tông I passed the camp of Chao Wieng Sā,
a Nān prince whom I had met in his home on two former visits. He
was overseeing the felling and running of teak timber down the Mê
Ing and the Mê Kōng to Lūang Prabāng. He had received and read a
Siamese New Testament, was quite familiar with the life and
teachings of Jesus, and admired His character. A lawsuit afterwards
brought him to Chiengmai, where I saw a great deal of him. He was
surely a believer at heart. To me he was willing to confess that his
only hope was in Jesus Christ, but was not ready to make a public
profession of his faith. I love to think of many such whom I have
met as like the Gamaliels, the Nicodemuses, and the Josephs of
Christ’s day.
 MRS. McGILVARY
1893

At Mûang Tông, as soon as I dismounted from my elephant an

officer met me to enquire who I was, and to escort me to the public
sālā. I soon learned that he was the brother of another officer whom
I had found on the road to Chieng Rāi the year before, unable to
travel and, apparently, sick unto death with fever. His company could
not linger indefinitely in the forest, and so had left him there with
two men to watch him, and probably to see him die. A dose of
calomel, and the quinine which I left with instructions as to its use,
seem to have cured his fever and enabled him to reach his home in
safety. He was himself now absent, but his brother’s heart had been
opened to friendship, and he did all that he could for my comfort. At
night he invited his friends to the sālā to meet me, and we had an
interesting evening. In all these places Nān Suwan and Noi Siri
would often be heard talking to the audience after I had retired, and
until sleep closed my eyes.
During our absence from Chieng Rāi a case of oppression, or, at
least, of evident injustice, on the part of the Court, had led our
friend the governor to take all Christians under his personal
protection as his own dependents. The kindness was well meant,
and we thanked him for it. But I doubted its wisdom. The only
scheme under which Christianity can really establish itself in all
lands, is to have Christians stand on precisely the same level before
the law as Buddhists or Brahmans or the followers of any other
religion.
From Chieng Rāi the elders were sent on to Cha Pū Kaw’s village
to see how the Mūsôs were getting on. I followed them in a day or
two. When I reached the chapel at Mê Kawn, the elders had
returned from the Mūsô village with a glowing account of their
constancy. This the testimony of Noi Tāliya and of all the Lāo
Christians confirmed. They had not missed a single Sunday service;
old and young alike came, and mothers, as before, bringing their
children tied on their backs. They had shamed the Lāo Christians by
their earnestness, getting to the chapel first, studying hard, and
returning home late.
On Saturday morning the whole village came down, and we spent
the day together. They remained that night as the guests of the Lāo.
The next day, Sunday, was largely given up to their instruction. They
all had renounced the worship of spirits; they all accepted Jesus as
their Saviour; they were all diligently learning to read and to sing.
Their conduct was most consistent; they had a good reflex influence
upon the church; and their conversion was an astonishment to the
non-Christian community.
These Mūsôs had all come, expecting to join the church. They had
been taught that public baptism—confessing Christ before men—was
the consummating act, the external seal of their initiation into the
privileges of the church. Although we impressed upon them that
they were not saved by the mere ceremony of baptism, yet
somehow they felt that without it they were not quite in the church,
and hence probably not quite safe from the spirits. Since it would be
nearly a year before they would have another opportunity, it seemed
unwise not to receive some of them at this time. The greatest doubt
was about Cha Waw. Yet he felt that more than any other he needed
whatever protection and assistance the church could afford him. He
had begun with his whole strength to break the chains of his opium
habit, to seek pardon and be saved. He felt confident that with God’s
help he would succeed.
The final decision was that, in order to bind them to the service of
Christ, they were all to appear before the session and make their
profession; but that only the two old men should be received into
full communion, and that one grandson from each family be baptized
as non-communing members. It was thought best to let the others
wait till our next visit; though I have never been satisfied that they
should not all have been admitted that day. Three of these Mūsô
boys accompanied me to Chiengmai on my return, and entered the
Boys’ School. It is not at all surprising that, in surroundings so
different from those of their mountain homes, they presently grew
lonesome and homesick. But they were satisfactory pupils, and
remained in school long enough to get a good start in reading and
singing.
Cha Waw, after a manful struggle, finally succeeded in breaking
away entirely from his opium—by the help of prayer and of quinine,
as he always believed and affirmed. When the non-Christian
tribesmen with their opium pipes visited his village, he was
accustomed to go down to the elders at Mê Kawn, to be away from
temptation, and under Christian influence. He lived a number of
years after this to attest the reality of his victory—the only case I
have ever known where the victory was surely won.
That year there was a famine among all the hill tribes. The upland
rice was almost entirely cut off by a plague of rats. I do not believe
in “rice Christians”; but when people are famishing with hunger, I
believe in feeding them, whether they are Christians or not. These
did not ask either for money or for any other aid. But when I left
them, I made arrangement with the Lāo elders to furnish them with
sixty buckets of rice, for which I paid ten rupees in advance. They
were very grateful for the aid.
The days spent among the Mūsôs that week were inspiring.
Glowing visions arose before us of a new tribe brought into the
Christian church, of which these were the first-fruits. On this whole
tour, indeed, only nine adults and seventeen children were baptized.
But in addition to the opening of work among the Mūsôs, we had for
the first time preached the Gospel beyond the borders of the
kingdom of Siam; and our longing eyes were turned toward the
Sipsawng Pannā, and beyond the great river. By this time the rains
had already begun to fall. A new season was needed to fulfil our
desires.
Much as I always enjoy my long tours, when my work is done and
my face at last is turned homewards, the gait of my sadaw seems
distressingly slow. On reaching Chiengmai I found all in fair health,
and all departments of work in full operation. But while I was still on
my way, word reached me of the death of Mrs. Briggs in Lakawn,
only a month and nine days after that of Mrs. Phraner. So
unexpected was it that I was not even aware that she had been ill.
In answer to my request for a few particulars from Dr. Briggs, I have
received the following, which I know he will excuse me for
transferring to these pages:
“Mrs. Alice Hamilton Briggs was from Truro, Nova Scotia. Although
within a year of graduation, she gave up her medical course and
accompanied her husband to the Lāo mission in answer to the call of the
Board. When she bade good-bye to the Secretaries of the Board, Dr.
Gillespie said to her, ‘It is a pleasure to see you so robust and strong. In
this respect you are better off than your husband. There have been so
many missionary women who have broken down on the field, that we are
glad to see that you have a reserve of health.’
“Before leaving American shores, however, Mrs. Briggs contracted a
slight cough which developed in severity during the voyage. On her
arrival in Siam it became apparent that the case was one of pulmonary
tuberculosis. The disease seemed to respond to treatment, and for
months improvement was marked. Up to within twenty-four hours of
death Mrs. Briggs was so hopeful of a return to health that she refused to
allow her family at home to know of her condition. On Saturday she was
cutting out a new dress for herself. On Sunday night she passed away.
Dr. Briggs was spending the evening with her, when a call came to attend
a child said to be dying just across the road. The doctor said he would be
back soon. A few minutes later he was called back too late even to hear
a last word of farewell.”

The event most interesting to us as a family during the fall of this

year, 1891, was the arrival of our son Evander with his young bride,
and our daughter Margaret, to carry on the work begun by their
parents. Our son had made special preparation for translating the
Scriptures into the Lāo language, then the most pressing need of the
mission.
 XXX

AMONG THE MŪSÔ VILLAGES—FAMINE

For the tour of 1892 I was to have the company of Dr. McKean as
long as he could be spared from Chiengmai, which would greatly
enhance the value of the trip. We had also three native evangelist-
assistants, and, last, but not least, we were well supplied with
Scriptures and tracts in the Lāo dialect. Our start was made on
January 5th.
Our first two Sundays and the intervening week we spent in
Wieng Pā Pāo, where we established ourselves in the new chapel
which the people themselves had built since our last tour. We
observed the Week of Prayer with two chapel services daily, and
house-to-house and heart-to-heart work in the intervals. The church
was formally organized with thirty-six adult members and thirty
children, three ruling elders, and two deacons.
From Wieng Pā Pāo we moved on to the village of Mê Kawn, the
centre of our very interesting work of the previous year among the
Mūsô tribe. The Sunday we spent there was a red-letter day in our
missionary life. Of it Dr. McKean writes: “This has been a blessed
day. All [of the Mūsôs] desire baptism. Two boys baptized last year
were admitted to the communion. Eleven other adults and seven
children were baptized, making twenty-two Mūsôs now members of
the visible church. One Lāo girl was received on confession, and
three Lāo children were baptized. Our Christian Mūsôs were out in
full force. A Mūsô officer and others not Christians attended from
another village. Before this we had visited these people in their
homes. We found that they had built a good chapel for their
worship, a better building than either of their own houses. They had
been very diligent in observing the Sabbath, in studying the
catechism, and in worship.”
We could not have been better pleased with our first success. The
exclusion of this little group from the large villages made it possible
and easy for all of them to become Christians. The whole-hearted
zeal with which they entered the church awakened strong hopes for
the conversion of their race. Cha Pū Kaw’s knowledge of the Lāo
tongue was above the average even of their head men. It would be
a long time before we could have another such interpreter and
assistant. And he was nearly, or quite, seventy years old; so that
whatever he was to do in teaching his people must be done soon. It
was, therefore, thought best to make a strong effort through him
and his family during that season.
At our next stopping-place, Nāng Lê, we came near having a
serious casualty. Our boys were out on a deer hunt, and one of them
bethought him of a novel expedient for getting the game. He
climbed a tree, and had the grass fired on the other side of the open
space. The grass was tall and dry, and the wind blew strong towards
him. He became so engrossed in looking for the deer that he forgot
the fire, till it was too late to flee. He could climb beyond the actual
flames; but meanwhile the whole air had become like the breath of a
furnace. When, at last, the fire had swept past him, and he was able
to descend, he was a mass of blisters. The swiftness of the rush of
the fire alone saved his life. Had it been slower, he could not have
escaped suffocation.
From Nāng Lê we visited a very large Mūsô village. It was a steep
foot-climb of four solid hours, and, to make it longer, our guide
missed the way. The first sign of human life we saw was a Mūsô girl
alone watching a clearing. She fled for dear life, till, recognizing Cha
Pū Kaw’s Mūsô speech, she stopped long enough to point the way to
the village. Her fleet steps outran ours, and when we reached the
village, the people were already assembling to see the unwonted
sight of the white foreigners. But the community was greatly
disturbed over another matter. One of their leading officers, it
seemed, was accused of being the abode of a demon that had
caused an epidemic of disease. The authorities were hourly waiting
for an order from the court in Chieng Rāi to expel him and his family
by force from the province. They had heard of Cha Pū Kaw’s
conversion, and were anxious to hear from himself his reasons
therefor—which he gave and enforced till late in the night. They
were expecting, however, on the morrow a regular conflict which
might result in bloodshed, and they evidently preferred that we
should not be there. The head Pū Chān was several days’ journey
distant. They would confer together among themselves and with
him, would let us know the result, and would invite us up again
before we left their neighbourhood.
About midnight a fierce storm of wind and rain broke upon us to
our great discomfort. Our thin tent afforded but poor protection. We
doubled up our bedding over our clothes, and sat upon the pile
under our umbrellas, and laughed at the novelty of our situation and
the poor prospect of a night’s sleep. But later the storm passed off,
and we did get a little sleep. Our visit to that group of Mūsô villages
was evidently not well timed. We took the advice of their officers,
and returned to Nāng Lê.
Two days later we reached Chieng Sên. Here we received a mail
from home, with news that Mrs. McKean was not well, and other
members of the station needed the doctor’s presence. It was
expressed as “the unanimous judgment of the station that he should
return immediately.” We had planned a regular campaign in the
Mūsô districts on both sides of the Mê Kōng—the sort of trip in
which the medical missionary finds his best opportunity. But the
recall was so imperative that it could not be ignored. So I was left to
continue the work alone.
The Mūsô tribe was about equally numerous in the mountain
ranges on both sides of the big river. On the east side there were
eleven villages. It seemed advisable to take that section first,
because they were under Chieng Sên rulers, of whose cordial and
sincere interest in our work we were sure. Sên Chai, the head man
of the large village nearest to the city, was a friend of Nān Suwan,
and was strongly inclined to embrace our religion; but felt the
difficulty of breaking the tribal bond. Before this I had made him a
visit of two or three days, and saw clearly that our only chance of
accomplishing anything was to gain all the head men of the eleven
villages. It was actually easier to win over the whole as a unit than
to win it piecemeal. This was a formidable task to undertake, but
with God’s blessing on the labours of Cha Pū Kaw and Nān Suwan, it
seemed not impossible.
We set out for the first village one morning shortly after ten
o’clock. It was four o’clock when we stopped for rest at the first
cluster of houses on the outskirts of the settlement. The news of our
arrival soon reached the main village. When we started again we
met Sên Chai with a regular serenade-party of men and boys with
native reed instruments, blowing their plaintive dirge-like music, to
welcome us and escort us in. Soon the population was all assembled
—the maidens in their best sarongs, the mothers and grandmothers
each with an urchin strapped to her back by her scarf, the men
coming in from their work, and the inevitable crowd of children. Cha
Pū Kaw was already answering their questions, with Nān Suwan’s
sympathetic aid. They were respectfully shy, but there was no
cringing. Sên Chai invited the local Pū Chān and all the villagers to
assemble after their evening meal to hear the new doctrines. We
first had worship with singing, and prayer by Cha Pū Kaw. It was the
first time they had heard the Great Spirit addressed in their own
Mūsô tongue. There were frequent exclamations of delight that they
were able to understand every word.
And then, before that motley crowd, drinking with them their
native tea from an earthen teapot, the men seated close around, or
reclining as they smoke their pipes, the women and children walking
about or sitting on the ground—we tell of God the great Spirit, the
Creator, and Father of all—the Bible, His message to men—the
incarnation, life, and death of Christ, and redemption through His
blood. Before we get through you will hear man after man say, “I
believe that. It is true.” One man takes up the story from Cha Pū
Kaw’s mouth and repeats it to another—a story that till now he
himself had never heard. Another says, “Nān Suwan has told us this
before, but now we hear it from the father-teacher.”
Before we retired that night Sên Chai said to us, with the approval
of most of his village, “Go on to Sên Bun Yūang and the head men
of the other villages. If they agree, we will all accept Christianity.
One village cannot accept it alone. If we do not ‘kin waw’ with them
—join in their New Year’s feast—we shall be treated as enemies by
the whole tribe.”
So, next morning, we set out to find the great Pū Chān—the
religious head of the province. On our way to his village we fell in
with a man to whom Cha Pū Kaw was speaking with great
earnestness. I found on approaching him that he was not a Mūsô,
but a Kūi—of a tribe which we had planned to visit later. He was the
Pū Chān of his village. He had already invited us through Cha Pū
Kaw to change our plan, and visit his village first. It was nearer than
the village we were intending to visit, and we were already tired
enough with our climb to be willing to stop at the nearest place.
The village was a large one, as mountain villages go—of twenty-
five or thirty houses, and from two hundred and fifty to three
hundred souls—in general not unlike the Mūsô villages we had seen.
The Kūi language also, while different from the Mūsô, is cognate
with it, so that Cha Pū Kaw could still act fairly well as our
interpreter. His talk with the Pū Chān on the way had already laid a
good foundation for our work in the evening, when curiosity and
interest in our errand brought the whole village together to hear Cha
Pū Kaw’s new doctrine from his own lips. The news of his conversion
had already reached them, and he had made a good impression on
the religious head of the village.—And, then, it was something new
to see the Mūsô boys able to read and to sing. Nān Suwan and Cha
Pū Kaw led in prayer, the one in Lāo and the other in Mūsô. Then
our religion was explained in its two leading ideas—rejection of the
spirit-cult, and acceptance of Jesus for the pardon of sin and the life
eternal. Questions were asked and answered.
At last the Pū Chān suggested that, while we continued our
reading and singing with the women and children, he and the men,
with Cha Pū Kaw, withdraw to a neighbouring house and talk the
matter over. It was evident that they would be more at their ease by
themselves, unawed by the presence of the foreign teacher. For
some two hours the debate continued. I could hear their earnest
voices from the neighbouring house, with only now and then a Lāo
word that I could understand. Then they returned to make their
report. With oriental politeness, they expressed their gratitude to the
“great teacher” who had come so far and at such expense, and had
brought with him a fellow-mountaineer of theirs, to teach them,
creatures of the jungle, the way to happiness. They had talked these
matters over, and understood them somewhat, but not fully. Some
were greatly pleased with the teachings, and believed them true. But
they could not yet come as an entire village, and they dared not
separate. Next morning we parted as friends. They were glad that
we had found the way to their village. “Be sure to come again!” That
I thought surely I should do; but this proved to be my only visit.
At the Sên Lūang’s village, where the great Pū Chān lived, we had
the same experience—a good reception, many apparently interested
and anxious to escape their own spirit-worship. A number of the
head men said, “If such and such a village accepts the Jesus-
religion, we will.” But no one could be found to face the clan and
make a start.
Thinking that our native evangelists might get at the heart of the
people all the better if left to do it alone, and being anxious to get
my mail from home, I went down on Saturday to Nān Suwan’s to
spend the Sunday there with the Christians. On Tuesday, to my
disappointment, the evangelists returned to me discouraged. They
were convinced that in the district east of the Mê Kōng River, no
break in the solidarity of the clan could be accomplished that season.
 But it was important not to leave these people with the impression
that we had abandoned them. I had left Sên Chai’s village with the
promise to return. So I went up with the Mūsô Christian boys, and
spent a last night with them. The village again assembled, and we
had an interesting evening. The Sên was greatly disappointed that
none of the other villages would join him. But the New Year was at
hand, when the clan must be unbroken. They would wait another
year, and try to get the other villages to join them. On the whole, I
was encouraged. When we left them we were escorted out of the
village to the music of their plaintive flutes, more like a victorious
than a vanquished army.
After a day or two with the Chieng Sên church, we visited the
ridge to the southeast of that city, between it and Chieng Kawng.
Our experience there was but a repetition of that from which we
were just come—cordial receptions, night audiences, manifest
interest, individual believers, anxious consultations, promises for the
next year; but the tribal bond was too strong to be broken.
But Cha Pū Kaw was anxious that we should not pass by his own
mountain villages on the Mê Kok. So we turned southward again
toward Chieng Rāi. This, moreover, was one of those famine years,
such as we have already encountered in our story, and shall
encounter yet again; many people were on the verge of starvation.
In places we could not get food for our own men. And famine was
beginning to be followed by disease and death. This was a serious
obstacle to our work.
Another serious obstacle was the use of opium, which became
more prevalent the further west we went along the Mê Kok range
towards Mûang Fāng. We presently reached villages where the
poppy was cultivated, until, in the last village, men, women, and
boys, and sometimes even girls, were its slaves. Fevers and
dysentery prevail during the rainy season. These people have a very
scanty pharmacopœia, and no antidotes whatever for these
diseases. Opium in some form is probably their surest remedy. Many
persons told me that they began by using it in sickness. As sickness
recurred the habit grew, until they were fast bound in its chains.
These facts largely determined the character of the instruction we
gave, and made our tour a kind of anti-opium crusade. Encouraged
and disappointed at every village, I was still tempted on by visions of
capturing some large village that would prove a more effective
entering wedge for the tribe than Cha Pū Kaw’s poor little hamlet.
The six weeks so spent were at the time the most novel and
exciting, as well as most arduous, of all my missionary experiences
so far.
We took both the old Mūsô men as assistants, and the younger
ones as carriers for our equipment. Our first day’s journey was a fair
sample of what we had to do continually. In many places it would be
a misnomer to speak of the track we travelled as a path. We left the
plain in the morning, and it was half-past two in the afternoon when
we reached the first summit. It was five o’clock when, desperate
with thirst, we came upon a flowing brook. There was, then, still
another hard climb before we saw our long looked-for first village
ahead. And, in general, because of the habit these people have of
planting their villages upon the very highest points where they can
get water, the journey from one of these villages to another in plain
sight, and, apparently, but a short distance away, would take hours
of the hardest travel. Sometimes we would walk weary hours
through rain, or through bushes as wet as rain, to visit a village;
only to walk back again after sitting three hours in wet clothes trying
in vain to awaken some interest in old or young.
One of the most interesting, and, at the same time, one of the
saddest, cases we met was that of Mûn Kamprai, the head man of a
village which clearly bore the impress of his character in the
intelligence and industry of its inhabitants. From opium he had kept
entirely aloof until, only a few years before this time, under the
stress of a severe illness, he began to take it. The poor man now
realized that he was becoming a wreck, but seemed to have no will-
power left to make the effort to break away from the habit. He was
much interested, however, in his two fellow-tribesmen whom I had
brought as my assistants; and Cha Waw’s example seemed to afford
him a faint gleam of hope. If we would stop a week and teach his
people, and would stand by to aid him, he would try. If successful,
he would surely become a Christian—and then his village would be
the one we had been hoping for to free itself from the tribal bond,
and become Christian.
The experiment was, indeed, pathetic. Removing all temptation,
he began with a desperate determination to succeed. We
encouraged him with human sympathy and the hope of divine aid.
We pushed as far as we dared the use of a tonic which Dr. McKean
had given me for such cases; and it aided him perceptibly. He held
out manfully for several days. But, at last, in an evil hour, he could
endure the torture no longer, and before we knew it, he had
resumed the use of the drug. For two nights he had not slept. In his
own expressive language, it was not his eyes, but his heart that
could not sleep. Poor man! his sufferings must have been as near
those of the infernal regions as it is possible to experience in the
body. And then his absolute wreck of mind, and the contempt he felt
for himself when he gave up the struggle as hopeless!
We spared no labour to reach the homes of these people, or their
hearts. We tried to become Mūsôs to the Mūsôs that we might win
them. Sometimes we had to sleep in their huts—on a floor raised
two or three feet from the ground, which the dogs shared with the
family, while the pigs and goats were on the ground beneath. In the
centre was a raised fireplace on which the native teapot always
boiled. Sleeping-mats or thin bedding lay about on the floor, and on
this, before bedtime, some of the inmates would lie down and fall
asleep even while listening to the conversation.—But everywhere the
tribal bond was too strong to be broken.
 MŪSÔ PEOPLE AND HUT NEAR CHIENG RAI

By this time the rains had set in. The trails—and the leeches that
infested them—were getting worse and worse. Soon the torrent-
streams would become impassable. We must return while yet we
could. Our six weeks’ wanderings we retraced in four days of
constant tramping. It had been a hard trip for all of us. I myself had
a touch of fever. It seemed good on reaching our camp to have once
more the luxury of a chair and a table. And then to be on the
sadaw’s back travelling homewards, and to meet a good mail on the
way! My three-score and fourth birthday was spent in the forest, and
I reached home safely on the 18th of May, after an absence of
nearly five months.

The peninsula of Farther India is largely exempt from the terrible

scourge of famine which has become almost chronic in Hindustan,
its greater neighbour on the west. There the population is so
numerous that the normal production of food is just sufficient to
supply its needs. Even a local or a partial failure of the crops must
produce distress. Siam, on the contrary, is happy in that it not only
produces an abundant supply for its own people, but is a granary for
the surrounding countries. The worst that has ever been
experienced in Lower Siam in years of greatest scarcity, has been
the necessity of checking the export of rice. The annual floods there
cover the whole country, so that a general failure of crops is,
humanly speaking, impossible.
In the northern states the land is higher; and considerable
portions of it, being above inundation, are directly dependent upon
the seasonal and local rains. But with a population by no means
dense, this very diversity of the cultivated areas is a source of safety.
A season of heavy rainfall which drowns the lowland rice, is apt to
prove exceptionally good for the uplands. And, on the other hand, a
season of light rainfall, which cuts short the upland crop, is apt to be
a good season for the flooded areas. And in considerable sections of
the country there is the chance that a second crop in the same
season may make good the loss of the first. There is a further
security also in the fact that, until communication with the coast
becomes such as to make exportation profitable, the excess of
fruitful years remains unconsumed in the country, to supply the need
of less fruitful ones. It thus comes about that scarcity amounting to
a real famine cannot result from the failure of crops in any single
year. It requires two consecutive failures to produce extensive
suffering among the very poor, and three to result in a real famine.
This last, however, was the case in 1892. In 1890 there was a light
crop throughout the land, with less excess than usual to be stored.
In 1891 the crop was lighter still. In the eastern provinces,
particularly in Lakawn and Prê, there was very little rice to be
reaped. Famine conditions began there long before the time for
harvest. People were scattering off in squads or by families into
Chiengmai and the northern provinces, begging a daily morsel. They
were poverty-stricken as well as famishing. The distress led the
brethren in Lakawn to make an appeal to friends in the United
States for a famine fund. Quite a liberal response, amounting to
several thousand dollars, was made to this call, largely by the friends
of the Lāo mission. The relief was almost as timely for the
missionaries as it was for the famishing people. Otherwise they
scarcely could have lived through the long strain on their nerves and
sympathies caused by the constant sight of sufferings which they
could not even in part relieve.
The province of Chiengmai could have met its own needs until the
new crop came in, had it not been for the constant draft upon its
reserves to meet the demands of Lakawn and Prê. But, between
high prices offered and pity for the less fortunate, those reserves
were steadily drained away, until, during the latter months of the
year, famine was upon us in Chiengmai, too. Bands of men from
destitute villages, maddened by hunger and unable to buy food,
began to roam about the country by night, or, sometimes, by day,
and seize rice wherever any little remnant of it could be found. The
authorities were powerless to restrain them or to keep order. The
condition of the more destitute provinces can better be imagined
than described.
At last the relief committee in Lakawn were asked if they could not
spare us a small portion of their fund, for it seemed that their
condition could not be much worse than ours. A letter from Dr. W. A.
Briggs brought us three hundred rupees, but with the following
caveat—the italics are his:
“Wherever we can reach the absolutely starving, that is a place to
invest. We do not pretend to relieve all the suffering. Now, if the need in
Chiengmai, or in the district mentioned, is so great that people are
actually dying from starvation, and those now living are living on such
stuff as the sample enclosed (cocanut-husks, leaves, bark, etc.), with
never a grain of rice, then I would advise you to form a Famine
Committee, and go into the business as we have done. The actual
starvation must be attended to, no matter where it is. But our saddest
experience is within Prê. Some one should be sent there at once.”

The scenes reported from Prê were harrowing. I will not pain the
reader by dwelling upon them. One happy result followed the efforts
of the brethren who went to the relief of that district. While
administering to bodily wants, they preached the Gospel, making
such an impression that there was a strong demand for a permanent
station there—which was established the next year, with Dr. and Mrs.
Briggs as pioneer missionaries.
It should be stated that, toward the last, the Siamese government
sent up supplies of rice; but, because of the distance and the
difficulty of transportation, not much reached the suffering people in
time to help them; and much was lost in passing through the hands
of so many officials.
 XXXI

CHIENG RUNG AND THE SIPSAWNG PANNĀ

At the Annual Meeting of the mission in December, 1892, the

broad field of Tai peoples north of the frontier of Siam was
discussed, and Rev. Robert Irwin and myself were appointed to
make a tour into that region as long and as far as in our judgment
might be deemed wise. The tour occupied nearly five months—from
January 3d to May 25th, 1893. This time we went fairly well supplied
with portions of Scriptures and tracts, and a good outfit of medicine.
Of quinine we carried a hundred ounces, and returned with less than
twenty-five. We relied on the medicines for the welcome they never
yet had failed to win for us. And Mr. Irwin had a cornet which did
excellent service throughout the tour. For riding I had my big
“sadaw” elephant, and Mr. Irwin had a pony; so we could exchange
mounts at our convenience. I pass over the earlier portion of our
route, already so often described, and the two weeks spent among
the hill-tribes visited on previous trips.
The chief object of our trip was to visit, in their ancient homes,
two northern tribes of the Lāo race—the Kôn and the Lû—from
which very many of our parishioners in the southern provinces
derived their origin. For, under conditions which lasted very nearly
down to our own time, there was almost constant predatory warfare
going on in this northern country—stronger states raiding the
weaker, and sweeping away the entire population of the districts
they overran, to plant them in their own realms. Thus whole villages,
and even entire districts, in the Lāo provinces of Siam, are peopled
by the descendants of such colonies of captives. We found it
unadvisable to attempt both visits in the same season, and the Lû
were the more accessible, living on the nearer slopes of the Mê Kông
valley. We went up on the west of the river along the edge of the
British territory, now known as the South Shan States, and beyond it
into Chinese territory, as far as Chieng Rung;[15] then, returning, we
made a somewhat wider sweep to the east of the river, through
French Indo-China; finally recrossing the river at Chieng Lāp, where
we struck once more our outgoing trail.
15. This name appears on some maps as Chieng Hung, initial r in the North
being generally pronounced as h.—Ed.
After leaving Mûang Len, the utmost point of a former trip, we
travelled awhile by a fine road along the summit of a ridge so
regular as to seem almost like an artificial embankment, and
affording noble views over the valley. At Wieng Mai, a recent
offshoot of Mûang Yawng, we spent a most interesting Saturday and
Sunday. Here the Prince-Governor sent to ask if he should not put up
a sālā to shelter us during our stay. In the morning we preached in
the market-place, and afterwards I distributed medicine and talked
with the people till noon, when I had to flee away to rest under the
shade of a big tree by the river. The people seemed hungry for the
bread of life. I could not supply all the requests made for copies of
the Scriptures.
Mûang Yawng, the older and larger city, we reached on Monday
forenoon, after a two hours’ ride. An officer met us at the gate, and
showed us to the sālā. When the Chao Mawm heard of our arrival,
he sent for us, meeting us at the door. We had a very interesting
interview, but he was not inclined to talk on the subject of religion.
He told me that the city and district had been entirely depopulated in
1809 by a force from Chiengmai, when “nothing was left behind but
the ground.”[16] It had recovered itself, however, and its population
was now larger than that of Lampūn. With Nān Suwan I visited the
market and the Court. At the latter place I learned that the British
Commissioner would arrive the next day. Knowing that everything
would be in confusion, we decided to move on the next morning.
16. This incident is a striking illustration of the methods of warfare in those days.
The expedition in question was directed against the Burmese, who had
established themselves in Mûang Yāng some sixty miles or more to the
northwest. On its way it passed through Mûang Yawng, where it was loyally
received. But being defeated at Mûang Yāng, it fell back upon Mûang Yawng,
and there gathered up all the inhabitants and swept them off to Chiengmai
to prevent their falling into the hands of the enemy!—Ed.
From this point on, our elephant was everywhere an object of
great interest. Sometimes the people climbed trees to get a better
view of him. A long day’s march brought us to Mûang Yū,
picturesquely situated on high bluffs, with deep gorges running
down to the Mê Lūi. Here we remained only overnight, leaving early
the next morning for Mûang Lūi, which we reached about noon. That
evening we had a large attendance at worship, the governor and
officials remaining till after eleven o’clock. The original population of
both these districts, as well as that of Mûang Yawng, are now
scattered throughout the provinces of Chiengmai and Lampūn.
Next morning we crossed the beautiful stream on a raft, while our
elephant took the ford. During the forenoon we came upon Captain
Davis of the Commissioner’s staff, who had been sent to make a
detour by Mûang Sing, and was then on his way to join his party. He
was resting by the roadside, ill with fever, and was glad to get from
me some quinine.
The following day, Saturday, brought us to Mûang Lūang, the
largest and most important place in the valley and the southernmost
of the old Sipsawng Pannā confederacy. The valley population is
wholly Lû. There is scarcely a Ngīu (Western Shan) to be found east
of the Keng Tung watershed. Here were the best roads we had seen
anywhere in Farther India, with a real arched bridge of stone across
the stream at the entrance to the city. Early next morning we were
awakened by a noisy crowd about our tent, anxious to see us. It was
the great market day, so, instead of attempting a regular service in
camp, we chose the market-place. There, whether reading or
speaking, we always had some attentive listeners.
On Monday our road lay for many miles along the summit of a low
ridge on which at intervals were fifteen large villages, just at the
edge of the long fertile plain, where are the rice-fields that feed the
country. I never saw in all my touring anything quite to equal that
row of villages. It seemed too bad to pass through so many without
even stopping.

GROUP OF YUNNAN LĀO

On the fourth day from Mûang Lūang we reached Chieng Rung,

the limit of our northward journey. Its location is strikingly beautiful,
on a high steep bluff overlooking the Mê Kōng River, which sweeps
in a majestic curve about its base. It is in Chinese territory, and is
ruled by a Chao Fā appointed from Yunnan. An officer from Yunnan
was there at the time collecting tribute. The influence of the English
was already felt there. Mûang Chê, to the west, had rebelled against
the Chao Fā, who thereupon sent out an expedition which captured
and brought away some three hundred families of the inhabitants.
But England cannot allow border warfare to go on along her frontier.
An English officer appeared on the scene, and the thing was
stopped.
At Chieng Rung we were still in the midst of an area of Lāo-
speaking people—an area which extended far beyond on every side.
I gave a portion of Scripture to a Lû whose home was ten days’
journey northward; and others to men from as far to the east and to
the west.
We had an interview with the Chao Fā by previous appointment.
At the door the officer suggested that we pull off our shoes. We
replied that it was not our custom, and was unnecessary. He looked
very doubtful, but said no more, and we walked in. The Chao Fā
received us courteously. We took him to be a man of no great
strength of character, about forty years of age, and somewhat
weakened by the use of opium. He asked whether we had not some
antidote to enable him to stop its use. He listened attentively to our
statement of the object of our coming, and said, “You are merit-
makers, and that is a good work.”
When we called at the court, the presiding officer had a wise
suggestion as to how we might further our purpose and establish
our religion in the place—a suggestion evidently not originating with
himself, but from a higher source. “The favour of the Chao Fā,” said
he, “will be necessary and all-sufficient. I see you have a fine
elephant. Just make a present of him to the Chao Fā. He will be
delighted, and your road will be all smooth.” I told him that I was an
old man, far from home, and dependent on the elephant. So I could
not part with him. This same suggestion was pressed upon us
several times afterwards, by the highest officials, and quite up to the
hour of our departure; though its form was modified from a gift to a
sale. I became at last a little anxious about the result, and was
somewhat relieved when we actually got away without loss of the
elephant.
 I may mention at this point an incident of this trip which never
came to my knowledge till thirteen years later, showing how we
were providentially spared from what would have put a sudden and
tragic end to our tour and to our lives. When Dr. S. C. Peoples and
Dr. W. C. Dodd were in Keng Tung in March, 1907, the presiding
officer of the Court told them that he had met Dr. McGilvary and Mr.
Irwin on their way to Chieng Rung; that when the people of Chieng
Rung first heard that some foreigners from the south were en route
to their capital, they planned to kill and plunder them. But when
they saw that the foreigners rode elephants and were accompanied
by carriers, they decided that this was probably the advance guard
of a formidable army, which it might not be well to attack. And then,
he said, the kindness of the missionaries so completely won their
hearts, that all thought of murder and plunder was given up.
Our return was to be through the region to the east of the Mê
Kōng. Its northern cities still belonged to the Sipsawng Pannā. But
the rest of it was territory recently ceded by Siam to France. The
governing race—the people of the plains—were everywhere Tai,
speaking the Lāo language and using the Lāo literature. On its
mountain ridges dwelt numerous hill-tribes, especially the Kamu and
the Lamēt.
The route we were to take crosses the river two days’ journey
south of Chieng Rung; so we had at first to retrace our steps. We
left the city on Monday, March 13th, safe from unseen plots, and
with our elephant. On the second day, after leaving our upward road
to strike across to the river, we entered unexpectedly a large village,
where we met with a reception ludicrously hostile. At every door
men were standing with guns in their hands. We were surprised;
but, supposing that it might be muster-day or something of the sort,
we passed innocently along, without challenge, to the monastery,
where we dismounted and began to unload. Then guns were laid
aside and the head man and villagers came up to see us and to offer
assistance. They had heard that foreigners were coming with
elephants and men, whether for peace or war no one knew. So they
had taken the precaution to be ready. When they found out our
peaceful errand, they were ashamed. We had a pleasant visit and
worship with them that evening.
The next stage of our road was bad. In some places we had to cut
our way through, and there were difficult passages of brook-beds
and gorges. We reached the river at Chieng Hā in a pouring rain,
and it rained again at night. The next day was the Buddhist sacred
day, and we were awakened early by the crowd of merit-makers and
worshippers—the women and girls, as usual, in their head-dresses
and gay colours, and all anxious to see the elephant and the white
faces.
It was 10:30 that morning before we got away. Ourselves, our
men, the saddles and luggage, were carried over by the ferry. Nān
Suwan alone faced the deep river on the sadaw to guide him
through. At the first plunge all of the elephant save his trunk, and
half of the rider, went out of sight. Thence on they went, now up
and now down, till they struggled out on the further shore. Such an
effort is very exhausting to the animal, and he has to have a good
rest and breathing-spell after it.
Mûang Ham, on the eastern bank, is larger than its neighbour on
the west. Its governor was a Chao Mawm, next in rank to the Chao
Fā of Chieng Rung, and his wife was the Chao Fā’s sister. I had a
long talk on religion with the wife. It was a new thought to her that
any one could be greater than the Buddha, though he was neither
Creator nor Saviour, but only a man. It is unnecessary continually to
state what was everywhere the case throughout this trip; namely,
that we had good audiences and interested hearers. We left in every
place some books in the hands of those most likely to use them;
though we could have used to advantage many more, if we had had
them.
From Mûang Ham two days’ march brought us on a Saturday to
Mûang Nūn, the most important city on our route, and, therefore, a
most desirable place to spend Sunday. The city is in the valley of the
Nam Bān. It has well paved streets, and a very large monastery on
an eminence above, where we camped. The abbot gave us a hearty
Welcome to Our Bookstore - The Ultimate Destination for Book Lovers
Are you passionate about books and eager to explore new worlds of
knowledge? At our website, we offer a vast collection of books that
cater to every interest and age group. From classic literature to
specialized publications, self-help books, and children’s stories, we
have it all! Each book is a gateway to new adventures, helping you
expand your knowledge and nourish your soul
Experience Convenient and Enjoyable Book Shopping Our website is more
than just an online bookstore—it’s a bridge connecting readers to the
timeless values of culture and wisdom. With a sleek and user-friendly
interface and a smart search system, you can find your favorite books
quickly and easily. Enjoy special promotions, fast home delivery, and
a seamless shopping experience that saves you time and enhances your
love for reading.
Let us accompany you on the journey of exploring knowledge and
personal growth!

ebookgate.com