Introduction to Software Testing (2nd edition)

Chapter 1: Why Do We Test Software?

Authors: Paul Ammann & Jeff Offutt

Lecturer: Shirin Baghoolizadeh
 Testing in the 21st Century

• Software defines behavior

• network routers, finance, switching networks, other infrastructure

• Today’s software market:

• is much bigger
• is more competitive
• has more users
• Embedded Control Applications
• airplanes, air traffic control
• spaceships
• watches
• ovens
• remote controllers

• Agile processes put increased pressure on testers

• Programmers must unit test – with no training or education!
• Tests are key to functional requirements – but who builds those tests ?
2
Software is a Skin that Surrounds Our Civilization

Quote due to Dr. Mark Harman

3
 Software Faults, Errors & Failures

• Software Fault : A static defect in the software

• Software Error : An incorrect internal state that is the manifestation of some

fault

• Software Failure : External, incorrect behavior with respect to the requirements

or other description of the expected behavior

Faults in software are design mistakes. They do not appear

spontaneously, but exist as a result of a decision by a human.

4
 Fault and Failure Example

• A patient gives a doctor a list of symptoms

• Failures
• The doctor tries to diagnose the root cause, the ailment
• Fault
• The doctor may look for anomalous internal conditions
(high blood pressure, irregular heartbeat, bacteria in the
blood stream)
• Errors

Most medical problems result from external attacks (bacteria,

viruses) or physical degradation as we age.
Software faults were there at the beginning and do not “appear”
when a part wears out.
5
 A Concrete Example
A program state is defined during execution of a program as the
current value of all live variables and the current location, as given
by the program counter. Fault: Should start
searching at 0, not 1
public static int numZero (int [ ] arr) Test 1
[ 2, 7, 0 ]
{ // Effects: If arr is null throw NullPointerException
Expected: 1
// else return the number of occurrences of 0 in arr Actual: 1
int count = 0; Error: i is 1, not 0, on the
for (int i = 1; i < arr.length; i++) first iteration Test 2
{ Failure: none [ 0, 2, 7 ]
if (arr [ i ] == 0) Expected: 1
{ Actual: 0
count++;
} Error: i is 1, not 0
} Error propagates to the variable count
return count; Failure: count is 0 at the return statement
}
6
 The Term Bug
• Bug is used informally
• Bug Sometimes means fault, sometimes error, and sometimes
failure … often the speaker doesn’t know what it means !

BUG

“It has been just so in all of my inventions. The

first step is an intuition, and comes with a burst,
then difficulties arise—this thing gives out and
[it is] then that 'Bugs'—as such little faults and
difficulties are called—show themselves and
months of intense watching, study and labor are
requisite. . .” – Thomas Edison

7
 Spectacular Software Failures
• NASA’s Mars lander: September 1999, crashed due to a THERAC-25 design
units integration fault
- misunderstanding in the units of measure used by two modules created by
separate software groups. One module computed thruster data in English units and
forwarded the data to a module that expected data in metric units.

• THERAC-25 radiation machine : Poor testing of safety-

critical software can cost lives : 3 patients were killed
- due to excessive radiation
• Ariane 5 explosion : About 370 million $ lost
- exploded 37 seconds after liftoff in 1996. The low-level cause was an unhandled
floating point conversion exception in an inertial guidance system function.
It turned out that the guidance system could never encounter the unhandled
exception when used on the Ariane 4 rocket
Ariane 5:
• Intel’s Pentium FDIV fault : Public relations nightmare exception-handling
bug

We need our software to be dependable

Testing is one way to assess dependability 8
 Northeast Blackout of 2003

508 generating
units and 256
power plants shut
down

Affected 10 million
people in Ontario,
Canada
Affected 40 million
people in 8 US
states
Financial losses of
$6 Billion USD

The alarm system in the energy management system failed due

to a software error and operators were not informed of the power
overload in the system
9
 Spectacular software Failures
• Boeing A220 : Engines failed after software
update allowed excessive vibrations
• Boeing 737 Max : Crashed due to overly
aggressive software flight overrides (MCAS)

• Toyota brakes : Dozens dead, thousands of crashes

• Healthcare website : Crashed repeatedly

on launch—never load tested

• Northeast blackout : 50 million people, $6

billion USD lost … alarm system failed
Software testers try to find faults before
the faults find users
10
 Validation & Verification (IEEE)
• Validation : The process of evaluating software at the end of software
development to ensure compliance with intended usage

• Verification : The process of determining whether the products of a given phase

of the software development process fulfill the requirements established during
the previous phase

Validation usually depends on domain knowledge; that is, knowledge of the

application for which the software is written. For example, validation of software
for an airplane requires knowledge from aerospace engineers and pilots.

Verification is usually a more technical activity that uses knowledge about the
individual software artifacts, requirements, and specifications.

11
Validation and Verification

12
 IV&V

• IV&V stands for “independent verification and validation” where

“independent” means that the evaluation is done by non-developers:

• Sometimes the IV&V team is within the same project,

• Sometimes the same company,

• And sometimes it is entirely an external entity.

13
 Testing Goals Based on Test Process Maturity

Beizer discussed the goals of testing in terms of the “test process maturity levels” of
an organization, where the levels are characterized by the testers’ goals:

 Level 0 : There’s no difference between testing and debugging

 Level 1 : The purpose of testing is to show correctness
 Level 2 : The purpose of testing is to show that the software doesn’t
work

 Level 3 : The purpose of testing is not to prove anything specific,

but to reduce the risk of using the software
 Level 4 : Testing is a mental discipline that helps all IT professionals
develop higher quality software

14
 Level 0 Thinking

• Testing is the same as debugging

• This is what we teach undergraduate CS majors

• Does not help develop software that is reliable or safe

15
 Level 1 Thinking
• Purpose is to show correctness
• Correctness is impossible to achieve
• What do we know if no failures?
• Good software or bad tests?

• Test engineers have no:

• Strict goal
• Real stopping rule
• Formal test technique
• Test managers are powerless

16
 Level 2 Thinking

• Purpose is to show failures

• Looking for failures is a negative activity

• Puts testers and developers into an adversarial relationship

• What if there are no failures?

17
 Level 3 Thinking

• Testing can only show the presence, but not the absence, of failures

• Whenever we use software, we incur some risk

• Risk may be small and consequences unimportant

• Risk may be great and consequences catastrophic

• Testers and developers cooperate to reduce risk

18
 Level 4 Thinking
• A mental discipline that increases quality

• Testing is only one way to increase quality

• Test engineers have primary responsibility to measure and improve

software quality

• Their expertise should help the developers

We hope to teach you to become “change agents” in your

workplace …
Advocates for level 4 thinking

19
 Tactical Goals : Why Each Test ?
If you don’t know why you’re conducting each test, it won’t be
very helpful

• Written test objectives and test requirements must be documented

• What are your planned coverage levels?

• How much testing is enough?

• Common objective – spend the budget … test until the ship-date …

20
 Cost of Late Testing

60
50
40
Fault origin (%)
30
Fault detection (%)
20
10 Unit cost (X)

SoftwareEngineering Institute; Carnegie Mellon University; Handbook CMU/SEI-96-HB-002

21
 Cost of Not Testing

• Testing is the most time consuming and expensive part of

software development

• Not testing is even more expensive

• If we have too little testing effort early, the cost of testing

increases

A tester’s goal is to eliminate faults as early as possible

22