Unit 4.

1
What is Testing?
"Testing is the process of executing a program' with the intent of finding errors".

Software testing is the process of testing the software product.

Effective testing will contribute to the delivery &higher quality software

products, more satisfied users, lower maintenance costs, more accurate, and
reliable results.

Testing is an important discipline, and consumes significant, amount of effort.

A proper strategy is required to carry out testing activities systematically and

effectively.
Thus, strategy provides a framework or set of activities, which are essential for
the success of the project .

This may Include planning, designing of test cases, execution of program with
test cases interpretation of the outcome and finally collection and management
of data.

It is a very expensive process and consumes one third to one-half of the cost of
a typical development project.

It is a specialized discipline requiring unique skills.

Why should we Test?

Although software testing is itself an expensive activity, yet launching of software

without testing may lead to cost potentially much higher than that of testing, specially
in systems where human safety is involved.

No one would think of allowing automatic pilot software into service without the most
rigorous testing.

In so-called life critical systems, economics must not be the prime consideration while
deciding whether a product should be released to a customer.

It is not possible to test the software for all possible combinations of input cases.

Testing therefore continues to the point where it is considered that the costs of the
testing processes significantly outweigh the returns.

Hence, when to release the soft-ware in the market, is a very important decision.
Who should do the Testing?
The testing requires the developers to find errors from their software. It is very
difficult for software developer to point out errors from own creations.

Many organizations have made a distinction between development and testing

phase by making different people responsible for each phase. This has an
additional advantage.

Therefore, most of the times, testing persons are different from development
persons for the overall benefit of the system.

Developers provide guide-lines during testing, however, whole responsibility is

owned by testing persons.
What should we Test?

We should test the program's responses to every possible input. It means, we

should test for all valid and invalid inputs.

Hence, complete testing is just not possible, although, we may wish to do so.
Another dimension is to execute all possible paths of the program.

A program path can be traced through the code from the start of the program to
program termination.

Two paths differ if the program executes different statements in each, or executes
the same statements but different order. A program may have many paths.
Error, Mistake, Bug, Fault and Failure

People make errors. A good synonym is mistake. This may be a syntax error or
misunderstanding of specifications. Sometimes, there are logical errors. When
developers make mistakes while coding, we call these mistakes "bugs".

Errors propagate from one phase to another with higher severity. A requirement
error may be magnified during design , and amplified still more during coding. If
it could not be detected prior to release it may have serious implications in the
field. An error may lead to one or more faults.

It is more precise to say that a fault is the representation of an error, where

representation is the mode of expression, such as narrative text, data flow
diagrams, ER diagrams, source code etc.
Defect is a good synonym for fault. If fault is in Source code, we call it a bug.

A failure occurs when a fault executes. It is the departure of the output of

program from the expected output. Hence failure is dynamic. The program has
to execute for a failure to occur.

A fault may lead to many failures.

A particular fault may cause different failures, depending on how it has been
exercised.
Test, Test case and Test Suite
Test and Test case terms are used interchangeably. In practice, both are same
and are treated as synonyms.

Test case describes an input description and an expected output description.

Every test case will have an identification.

Inputs are of two types: pre conditions (circumstances that hold prior to test
case execution) and the actual inputs that are identified by some testing
methods.

Expected outputs are also of two types: post conditions and actual outputs.
During testing, we set necessary preconditions, give required inputs to program,
and compare the observed output with expected output to know the outcome
of a test case.

If expected and observed outputs are different, then, there is a failure and it
must be recorded properly in order to identify the cause of failure.

If both are same, then, there is no failure and program behaved in the expected
manner.

A good test case has a high probability of finding an error.

The test case designer's main objective is to identify good test cases.
The template for a typical test case is given in Fig. 8.2.
Test cases are valuable and useful—at least as valuable as source code. They
need to be developed, reviewed, used, managed, and saved. The set of test
cases is called a test suite.

We may have a test suite of all possible test case. We may have a test suite of
effective/good test cases. Hence any combination of test cases may generate a
test suite.
 Alpha, Beta and Acceptance Testing

Customer involvement is required before delivering the final product. The above
mentioned three terms are related to customer's involvement in testing but have
different meanings.
 Acceptance testing
This term is used when the software is developed for a specific customer. A series of tests are
conducted to enable the customer to validate all requirements.

These tests are conducted by the end user/customer and may range from adhoc tests to well
planned systematic series of tests.

Acceptance testing may be conducted for few weeks or months.

The discovered error will be fixed and better quality software will be delivered to the
customer.

Acceptance Testing is of twotypes

• Alpha Testing
• Beta Testing
Alpha and beta testing
The terms alpha and beta testing are used when the software is developed as a
product for anonymous customers.

The alpha tests are conducted at the developer's site by a customer. These test's
are conducted in a controlled environment. Alpha testing may be started when
formal testing process is near completion.

The beta tests are conducted by the customers/end users at their sites. Unlike
alpha testing, developer is not present here. Beta testing is conducted in a real
environment that cannot be controlled by the developer. Customers are expected
to report failures, if any, to the company.
After receiving such failure reports, developers modify the code and fix the bug
and prepare the product for final release.

Most of the companies are following this practice firstly, they send the beta
release of their product for few months.

Many potential customers will use the product and may sent their views about
the product.
FUNCTIONAL TESTING
Functional testing refers to testing, which involves only observation of the
output for certain input values.

There is no attempt to analyze the code, which produces the output.

We ignore the internal structure of the code.

Therefore, functional testing is also referred to as black box testing in which

contents of the black box are not known.
Functionality of the black box is understood completely in terms of its inputs and outputs as
shown in figure.

Here, we are interested in functionality rather than internal structure of the code.

Many times we operate more effectively with black box knowledge.

For example, most people successfully operate automobiles with only black box knowledge.
Following are main techniques to black box testing
• Equivalence Class Partitioning
• Boundary Value Analysis
• Cause Effect Graphs.
1.1Boundary Value Analysis

Experience shows that test cases that are close to boundary conditions have a
higher chances of detecting an error.

Here boundary condition means, an input value may be on the boundary, just
below the boundary (upper side) or just above the boundary (lower side).

Suppose, we have an input variable x with a range from 1-100.

The boundary values are 1, 2, 99 and 100.

Consider a program with two input variables x and y. These input variables have
specified boundaries as:

Hence both the inputs x and y are bounded by two intervals [a, b] and [c, d]
respectively.

For input x, we may design test cases with values a and b, just above a and also
just below b.

Similarly for input y, we may have values c and d, just above c and also just
below d.
These test cases will have more chances to detect an error.
Any point within the inner rectangle is a legitimate input to the program.

For input x, we may design test cases with values a and b, just above a and also
just below b.

Similarly for input y, we may have values c and d, just above c and also just
below d.
The basic idea of boundary value analysis is to use input variable values at their
minimum, just above minimum, a nominal value, just below their maximum and
at their maximum.

Thus, boundary value analysis test cases are obtained by holding the values of all
but one variable at their nominal values and letting that variable assume its
extreme values.
The boundary value analysis test cases for our program with two inputs variables
(x and y) that may have any value from 100-300 are:

(200, 100), (200, 101), (200, 200), (200, 299), (200, 300), (100, 200), (101, 200),
(299, 200) and (300, 200).

This input domain is shown in Fig. 8.5. Each dot represent a test case and inner
rectangle is the domain of legitimate inputs.

Thus, for a program of n variables, boundary value analysis yields 4n + 1 test

cases.
(200, 100), (200, 101), (200, 200), (200, 299), (200, 300),
(100, 200), (101, 200), (299, 200) and (300, 200).
1.2 Robustness testing

It is nothing but the extension of boundary value analysis.

Here, we would like to see, what happens when the extreme values are exceeded
with a value slightly greater than the maxi-mum and a value slightly less than
minimum.

It means, we want to go outside the legitimate boundary of input domain.

This type of testing is quite common in electric and electronic circuits.

This extended form of boundary value analysis is called robustness testing and
shown in Fig. 8.6.
There are four additional test cases which are outside the legitimate input domain.
Hence total test cases in robustness testing are 6n + 1, where n is the number of
input variables.
So, 13 test cases are:
(200, 99), (200, 100), (200, 101), (200, 200), (200, 299), (200, 300), (200, 301), (99,
200), (100, 200), (101, 200), (299, 200), (300, 200), (301, 200).
1.3 Worst-case testing

If we like to see what happens when more than one variable has -an extreme value.

In electronic circuits analysis, this is called "worst case analysis".

It is more thorough in the sense that boundary value test cases are a proper subset of
worst case test cases.

It requires more effort.

Worst case testing for a function of n variables generates 5n test cases as opposed to 4n
+ 1 test cases for boundary value analysis.

Our two variable example will have 52 = 25 test cases and are given in Table 8.1.
Boundary value analysis works well for the programs with independent input
values.

Here input values should be truly independent.

This technique does not make sense for Boolean variables where extreme values
are TRUE and FALSE, but no clear choice is available for other values like
nominal, just above boundary and just below boundary.
1.4 Equivalence Class Testing
In this method, input domain of a program is partitioned into a finite number of
equivalence classes such that one can reasonably assume that the test of a
representative value of each class is equivalent to a test of any other value.

That is, if one test case in a class detects an error, all other test cases in the class
would be expected to find same error.

Conversely, if a test case did not detect an error, we would expect that no other
test cases in the class would find an error.
Two steps are required in implementing this method:

1. The equivalence classes are identified by taking each input

condition and partitioning it into valid and invalid classes.

For example, if an input condition specifies a range of values

from 1-999, we identify one valid equivalence class [1 < item <
999], and two invalid equivalence classes [item < 1 ] and [ item
> 999].
2. Generate the test cases using the equivalence classes identified in
the previous step.

This is performed by writing test cases covering all the valid

equivalence classes.

Then a test case is written for each invalid equivalence class so that
no test contains more than one invalid class. This is to ensure that no
two invalid classes mask each other.
The idea is to choose at least one element from each equivalent class.

In the triangle problem we would certainly have a test case for equivalent
triangle, and we may take (50, 50, 50) as inputs for a test case.

If we do this, we would not expect to learn much from test cases such as (40, 40,
40) and (100, 100, 100).

These test cases may be treated as redundant test cases.

We should -not forget to have equivalent classes for invalid inputs.

This is often best source of bugs.

We should test different types of invalid inputs in order to get more errors.

As an example, for a program that is supposed to accept any number between 1 and
99, there are at least four equivalence classes from input side.

The classes are:

(i) Any number between 1 and 99 is valid input.
(ii) Any number less than 1. This include 0 and all negative numbers.
(iii) Any number greater than 99.
(iv) If it is not a number, it should not be accepted.

Most of the time, equivalence class testing defines classes of the input domain.
However, equivalence classes should also be defined for output domain. Hence, we
should design equivalence classes based on input and output domains.
Cause Effect Graphing Technique
One weakness of boundary value analysis and equivalence partitioning is that
these do not explore combinations of input circumstances.

These consider only single input conditions. However, combinations of inputs

may result in interesting situations.

These situations should be tested. If we consider all valid combinations of

equivalence classes, then we will have large number of test cases, many of which
will not be useful for revealing any new errors.
For example, if there are "n" different input conditions, such that any
combination of the input conditions is valid, we will have 2n test cases.

Cause effect graphing is a technique that aids in selecting, in a systematic way, a

high-yield set of test cases.

It has a beneficial effect in pointing out incompleteness and ambiguities in the

specifications.
1. The causes and effects in the specifications are identified.

A cause is a distinct input condition or an equivalence class of input conditions.

An effect is an output condition or a system transformation (a lingering effect

that an input has on the state of the program or system).

For instance, if a transaction to a program causes a master file to be updated,

the alteration to the master file is a system transformation; a confirmation
message would be an output condition.

Causes and effects are identified by reading the specification word by word and
underlining words or phrases that describe causes and effects. Each cause and
effect is assigned a unique number.
2. The semantic content of the specification is analyzed and transformed into a
Boolean graph linking the causes and effects. This is the cause effect graph.

3. The graph is annotated with constraints describing combinations of causes

and/or effects that are impossible because of syntactic or environmental
constraints.

4. By methodically tracing state conditions in the graph, the graph is converted

into a limited entry decision table. Each column in the table represents a test
case.

5. The columns in the decision table are converted into test cases. The basic
notation for the graph is shown in Fig. 8.8.
Think of each node as having the value 0 or 1; 0 represents the 'absent state' and
1 represents the present state. The identity function states that if c1 is 1, e1 is 1;
else e1 is 0.

The NOT function states that if c1 is 1, e1 is 0 else, e1 is 1. The OR function states

that if c1 or c2 or c3 is 1, e1 is 1; else e1 is 0. The AND function states that if both
c1 and c2 are 1, e1 is 1; else e1 is 0.
The AND and OR functions are allowed to have any number of inputs.

Myers explained this effectively with the following example.

1. The characters in column 1 must be an A or B. The character in column 2

must be a digit. In this situation, the file update is made.

2. If the character in column 1 is incorrect, message x is issued.

3. If in column 2 is not a digit, message y is issued.

The causes are

c1. character in column1 is A

c2: character in column 1 is B
c3: character in column 2 is a digit

and the effects are

e1: update made

e2: message x is issued
e3: message y is issued

The cause-effectgraph is shown in Fig. 8.9.

Test Cases