Rapid software testing


Although the value of the class is greater because of the people who helped us, the sole responsibility for the
content of this class belongs to the authors, James Bach and Michael Bolton. This class is essentially a long
editorial opinion about testing. We have no authority to say what is right. There are no authorities in our field.
We don’t peddle best practices. We don’t believe in them.

What we will do is share our experiences and challenge your mind. We hope you will challenge us back.
You’ll find the class more rewarding, we think, if you assume that our purpose is to make you stronger,
smarter and more confident as a tester. Our purpose is NOT to have you listen to what the instructor says and
believe it, but to listen and then think for yourself.

All good testers think for themselves. In a way, that’s what testing is about. We look at things differently than
everyone else so that we can find problems no one else will find.


If any of these assumptions don’t fit, this class might not be for you.

We don’t make any assumptions about how experienced you are. The class can work well for novices or
experts.

It’s the desire to be very good at software testing that compels you to want to take a class. Most of the people
in the world who work in software testing have never taken a testing class, read a testing book, attended a
testing conference, participated in a test user group. They still get to keep their jobs, and they might even be
okay at testing. A class like this becomes important when you want to be confident and great at software
testing.

As a supplement to this course, consider the Black Box Software Testing course available through
http://www.testingeducation.org. This course, developed by Cem Kaner and James Bach, is set up for self-
study. It’s composed of more than 40 hours of video lectures, and includes detailed course notes, quizzes,
self-tests, supplementary reading materials, and links to more. The Association for Software Testing
(http://associationforsoftwaretesting.org) offers instructor-led versions of the course to its members.


Rapid testing is a mind set–a particular way of looking at the world; and a skill set—a particular set of things
that we practice and get better at.

We’re advocates of the context-driven approach to software testing.

We’ve applied these approaches in financial institutions, in court cases, in testing of medical devices, in
commercial shrink-wrapped software, to games… The methodology is designed to adapt to any kind of testing
context.

Rapid testing involves considerations of skill, personal integrity, improved focus on the underlying need for
testing tasks, improved appreciation for the stakeholders of testing tasks, and knowledge of the possible
techniques and tools that could be brought to bear to improve efficiency.


Without scrutiny, this would be easy to do. You could test badly, but no one would ever know or care. What
makes testing hard is that someone, perhaps your employer, perhaps the users, or perhaps you yourself, will be
judging your work. Maybe this scrutiny will be indirect, such as someone unhappy with the product and
cursing your company for building it so poorly. Or perhaps it will be the most intimate form of scrutiny–
which is your feeling of pride or disappointment in your own work.

If you want your testing to pass muster, you need to work at it.


You will get some practice in thinking like an expert tester. That will mean some success and some failures.
For the successes, we want you to feel good—strong and powerful; that’s a feeling that an expert tester gets.
Alas, from time to time, expert testers also get the opportunity to feel embarrassed by failure. There are many
traps in the exercises. We want to give you the experience of being trapped, of recognizing the traps, and of
working your way out of them. This class is intended to provide you with a safe place to do that, so you can
recognize similar situations and respond appropriately when you’re on the job.

We also expect you to challenge our expertise. That’s part of developing your own expertise, but it’s also
fundamental to our development. Any ―expert‖ can be wrong, and ―expert advice‖ is always wrong in some
context, so we don’t get upset if you argue. On the contrary; we’re delighted if you argue. Challenging our
statements shows us that you’re thinking about how things might work—or not—in your own environment.
The conversation is often very enlightening.


The first question about quality is always ―whose opinions matter?‖ If someone who doesn’t matter thinks
your product is terrible, you don’t necessarily change anything. So, an important bug is something about a
product that really bugs someone important.

One implication of this is that, if we see something that we think is a bug and we’re overruled, we don’t
matter. That’s a fact: testers don’t matter. That is to say, our standards are not the standards to which the
product must adhere; we don’t get to make decisions about quality. This can be hard for some testers to accept,
but it’s the truth. We provide information to managers; they get to make the hard decisions. They get the big
bucks, and we get to sleep at night.


Cem Kaner prefers this definition:
“Software testing is a technical investigation for the purpose of revealing the quality of a software product on
behalf of stakeholders.”

Kaner’s definition means the same thing as our definition in every material respect. However, we sometimes
prefer more concise wording.

One surprising implication of both our definitions is that you can test a product that doesn’t yet exist in
operable form. The questioning process that precedes what we call test execution is still part of testing if it
serves the process of test design and test execution.


By ―modeling‖, we mean that we must build a representation in our minds of what the product must do. If our
model is incorrect or limited, then we automatically will not test what should be tested.

The first four items on the list are usually called ―test design‖. The latter five items are usually called ―test
execution‖.


Key ideas about heuristics:

•A heuristic is not an edict. Heuristics require guidance and control of skilled practitioner.
•Heuristics are context-dependent.
•Heuristics may be useful even when they contradict each other– especially when they do!
•Heuristics can substitute for complete and rigorous analysis.

Types of heuristics:
•Guideword Heuristics: Words or labels that help you access the full spectrum of your knowledge and
experience as you analyze something.

•Trigger Heuristics: Ideas associated with an event or condition that help you recognize when it may be time to
take an action or think a particular way. Like an alarm clock for your mind.

•Subtitle Heuristics: Help you reframe an idea so you can see alternatives and bring out assumptions during a
conversation.

•Heuristic Model: A representation of an idea, object, or system that helps you explore, understand, or control
it.

•Heuristic Procedure or Rule: A plan of action that may help solve a class of problems.


•Cognitive Heuristics: Heuristics built into your mind by virtue of how human brains work. These show themselves as
systematic biases in our senses (optical illusions) and decision-making.


Everyday examples of heuristics:

•It’s dangerous to drink and drive.
•A bird in hand is worth two in the bush.
•Nothing ventured, nothing gained.
•Sometimes people stash their passwords near their computers. Try looking there.
•Stores are open later during the Holidays.
•If your computer is behaving strangely, try rebooting. If it’s very strange, reinstall Windows.
•If it’s a genuinely important task, your boss will follow-up, otherwise, you can ignore it.


Our colleague, Jeremy Kominar, is an expert amateur magician. We asked him to provide us with some
remarks on the links between software testing and magic. Here’s an edited version of what he had to say--we
can’t quote him exactly, because he asked us not to give away his secrets!

•Learning to perform magic causes the magician to be aware of not only his own perspective, but also (and
more importantly) others’ perception of the trick. This becomes invaluable for testers because it helps us to
recognize that the interests and perspectives of many stakeholders must be considered when testing software.
The trick may ―look good‖ or ―work‖ from your angle, but there are always other angles to cover.

•When learning to perform magic tricks, you generally want to simplify things--minimize the number of
moves and figure out how you can reach the same end result without as many steps to get there. In a testing
context this idea can be used for creating different usage scenarios—taking multiple paths and variables to
attain the same goal. (We might want to minimize the number of steps, but we might also want to vary the
paths to shake out more bugs.)

•Learning to perform magic and learning to figure out magic tricks require us to develop observational skills,
and to recognize things that deceive us. Most non-magicians fail to consider possibilities that come easily to
magicians; devices, coins, or cards can be gimmicked. Non-magicians only think about normal cards. You
need to think outside of the box to be a magician or to figure out how the magician does his work. This kind
of reasoning and deduction are key assets to being a tester. There isn’t really such a thing as magic but there
clearly is such a thing as deception. As the magician becomes more experienced and more wise to the
practice, he should be able to reverse-engineer tricks by using patterns that he already knows about
magic. Once you’ve seen one kind of trick, similar tricks aren’t so mysterious because you have heuristics that
you can use to recognize how it’s done.


Jerry Weinberg has suggested that ―it works‖ may mean ―We haven't tried very hard to make it fail, and we
haven't been running it very long or under very diverse conditions, but so far we haven't seen any failures,
though we haven't been looking too closely, either.‖ In this pessimistic view, you have to be on guard for
people who say it works without checking even once to see if it could work.


Note that we’re saying ―system‖ here, instead of ―product‖. A product could behave perfectly as a part of
some system, but when something else within the system changes, that change could undermine the product’s
usefulness or perceived quality.

Seeing the product as a component or participant of some larger system opens our eyes to a greater range of
potential problems.


Fixed Markers. Some programmers use the hex values 0xDEADBEEF as a flag to indicate uninitialized data.
Statistical Markers. If you create input that has a specific statistical property, then you may be able to detect
missing or corrupted data by examining the statistical properties of the output.
Self-referential data. Counterstrings are strings that identify their lengths. *3*5*7*9*12*15* is an example
of a counterstring. (Satisfice’s PERLCLIP tool creates counterstrings of arbitrary lengths.) Another example is
using JAMES_FIRSTNAME and BACH_LASTNAME as sample data in the first name and last name fields of
a database.
Easy Input Regions. A number times one is always itself; a number plus zero is always itself; a multiple of
ten always ends in 0, and so on.
Outrageous values. Examples include letters in fields that expect only numbers, extremely long strings in
fields that expect short ones, huge numbers where modest ones are expected, or zero or null values where
some positive value is required. Each example should trigger some kind of error handling.
Idempotent input. Take some data and process it in some way, such that the output from the first run can be
used as input for subsequent runs but shouldn’t go through additional changes. Examples include spell-
checking (after all the corrections have been accepted for the first run, no further changes should be suggested)
or code formatters (after the code has been formatted once, the same code should pass through the process
unchanged).
Match. Given the same input data and the same function on that data, we would typically expect the output to
be the same. This approach is sometimes used to check various kinds of encoding or checksums.
Progressive mismatch. Check for consistency in a function by varying the input in some limited or controlled
way, and observe the effect on the output. This is an exploratory approach, sometimes used as a code-breaking
technique.


Try pronouncing CRUSSPIC STMPL; sometimes we think of it as the name of an Eastern European hockey
player, but however you think of it, it makes this list easier to remember. The point is to be able to cycle
through the list at any time to ask a new question about something that we might value in the product, or
something that might represent a vulnerability if it were missing or broken.


―State Conservation‖ means that you minimize disturbances to the product state, maintaining it as much as
possible while using the product.


Some people prefer a more restrictive definition of procedure. Cem Kaner considers a sequential set of steps to
be an important aspect of procedure. He would prefer that we use the word ―test implementation‖ instead of
procedure, in this section.

However, in the Rapid Testing methodology, we usually opt for the most inclusive definitions that still express
the key ideas, since that makes the conversation easier. In our view, the word procedure as it is commonly
used is not limited only to sequential instruction sets.

As always, it’s up to you what words you use.


We’ll come back to these later. For now, note that we’re more likely to catch different bugs with different
nets; a diverse set of test techniques will find a diverse set of bugs.


Testing is itself a form of risk analysis: risk-based testing begins with a rumor of risk and transforms
that into information.

Even if testing reveals a problem that cannot be reproduced, you have learned something about risk.

Exploratory testing is a popular and powerful approach to analyzing risk. Scripted testing is better
suited for monitoring known risk.


Happy Path: Use the product in the most simple, expected, straightforward way, just as the most optimistic programmer might imagine users to
behave. Perform a task, from start to finish, that an end-user might be expected to do. Look for anything that might confuse, delay, or irritate a
reasonable person.
Documentation Tour: Look in the online help or user manual and find some instructions about how to perform some interesting activity. Do those
actions. Improvise from them. If your product has a tutorial, follow it. You may expose a problem in the product or in the documentation; either
way, you’ve found something useful. Even if you don’t expose a problem, you’ll still be learning about the product.
Sample Data Tour: Employ any sample data you can, and all that you can—the more complex or extreme the better. Use zeroes where large
numbers are expected; use negative numbers where positive numbers are expected; use huge numbers where modestly-sized ones are expected; and
use letters in every place that’s supposed to handle numbers. Change the units or formats in which data can be entered. Cha llenge the assumption
that the programmers have thought to reject inappropriate data.
Variables Tour: Tour a product looking for anything that is variable and vary it. Vary it as far as possible, in every dimension possible. Identifying
and exploring variations is part of the basic structure of my testing when I first encounter a product.
Complexity Tour: Tour a product looking for the most complex features and using challenging data sets. Look for nooks and crowds where bugs ca n
hide.
File Tour: Have a look at the folder where the program's .EXE file is found. Check out the directory structure, including subs. Loo k for READMEs,
help files, log files, installation scripts, .cfg, .ini, .rc files. Look at the names of .DLLs, and extrapolate on the funct ions that they might contain or the
ways in which their absence might undermine the application.
Me nus and Windows Tour: Tour a product looking for all the menus (main and context menus), menu items, windows, toolbars, icons, and other
controls.
Ke yboard and Mouse Tour: Tour a product looking for all the things you can do with a keyboard and mouse. Run through all of the keys on the
keyboard. Hit all the F-keys. Hit Enter, Tab, Escape, Backspace. Run through the alphabet in order. Combine each key with Sh ift, Ctrl, and Alt.
Also, click on everything.
Inte rruptions: Start activities and stop them in the middle. Stop them at awkward times. Perform stoppages using cancel buttons, O/S level interrupts
(ctrl-alt-delete or task manager), arrange for other programs to interrupt (such as screensavers or virus checkers). Also try suspending an activity and
returning later.
Unde rmining: Start using a function when the system is in an appropriate state, then change the state part way through (for instance, dele te a file
while it is being edited, eject a disk, pull net cables or power cords) to an inappropriate state. This is similar to interruption, except you are expecting
the function to interrupt itself by detecting that it no longer can proceed safely.
Adjustments: Set some parameter to a certain value, then, at any later time, reset that value to something else without resetting or recreating the
containing document or data structure.
Dog Piling: Get more processes going at once; more states existing concurrently. Nested dialog boxes and non-modal dialogs provide opportunities to
do this.
Continuous Use: While testing, do not reset the system. Leave windows and files open. Let disk and memory usage mount. You're hoping that the
system ties itself in knots over time.
Fe ature Interactions: Discover where individual functions interact or share data. Look for any interdependencies. T our them. Stress them. I once
crashed an app by loading up all the fields in a form to their maximums and then traversing to the report generator. Look for places where the
program repeats itself or allows you to do the same thing in different places.
Click for Help: At some point, some users are going to try to bring up the context-sensitive help feature during some operation or activity. Does the


product’s help file explain things in a useful way, or does it offend the user’s intelligence by simply restating what’s already on the screen?
Is help even available at all?


Input Constraint Attack: Discover sources of input and attempt to violate constraints on that input. For instance, use a geometrically expanding
string in a field. Keep doubling its length until the product crashes. Use special characters. Inject noise of any kind into a system and see what
happens. Use Satisfice’s PerlClip utility to create strings of arbitrary length and content; use PerlClip’s counterstring feature to create a string that
tells you its own length so that you can see where an application cuts off input.
Click Frenzy: Ever notice how a cat or a kid can crash a system with ease? Testing is more than "banging on the keyboard", but that phrase wasn't
coined for nothing. Try banging on the keyboard. Try clicking everywhere. I broke into a touchscreen system once by poking ev ery square centimeter
of every screen until I found a secret button.
Shoe Test: This is any test consistent with placing a shoe on the keyboard. Basically, it means using auto -repeat on the keyboard for a very cheap
stress test. Look for dialog boxes so constructed that pressing a key leads to, say, another dialog box (perhaps an error message) that also has a button
connected to the same key that returns to the first dialog box. That way you can place a shoe (or Coke can, as I often do, but sweeping off a cowboy
boot has a certain drama to it) on the keyboard and walk away. Let the test run for an hour. If there’s a resource or memory leak, this kind of test will
expose it.
Blink Te st: Find some aspect of the product that produces huge amounts of data or does some operation very quickly. For instance, look a long log
file or browse database records very quickly. Let the data go by too quickly to see in detail, but notice trends in length or look or shape of the data.
Some bugs are easy to see this way that are hard to see with detailed analysis. Use Excel’s conditional formatting feature t o highlight interesting
distinctions between cells of data.
Error Message Hangover: Make error messages happen and test hard after they are dismissed. Often developers handle errors poorly.
Re source Starvation: Progressively lower memory, disk space, display resolution, and other resources until the product collapses, or gracefully (we
hope) degrades.
Multiple Instances: Run a lot of instances of the app at the same time. Open the same files. Manipulate them from different windows.
Crazy Configs: Modify the operating system’s configuration in non-standard or non-default ways either before or after installing the product. Turn
on ―high contrast‖ accessibility mode, or change the localization defaults. Change the letter of the system hard drive. Consider that the product has
configuration options, too—change them or corrupt them in a way that should trigger an error message or an appropriate default behavior.
Cheap Tools: Learn how to use InCtrl5, Filemon, Regmon, AppVerifier, Perfmon, and Process Explorer, and T ask Manager (all of which are fre e).
Have these tools on a thumb drive and carry it around. Also, carry a digital camera. I now carry a tiny 3 megapixel camera and a tiny video camera.
Both fit into my coat pockets. I use them to record screen shots and product behaviors. While it’s not cheap, you can usually find Excel on most
Windows systems; use it to create test matrices, tables of test data, charts that display performance results, and so on. Use the World-Wide Web
Consortium’s HTML Validator at http://validator.w3c.org. Pay special attention to tools that hackers use; these tools can be used for good as well as
for evil. Netcat, Burp Proxy, wget, and fuzzer are but a few examples.


Detailed procedural documentation is expensive and largely unnecessary.

Tutorial documentation is also usually unnecessary, but if you do it, then keep it separate from the working
documents.

Rapid software testing

More Related Content

What's hot (20)

Viewers also liked (20)

Similar to Rapid software testing (20)

More from Sachin MK (20)

Recently uploaded (20)

Rapid software testing