Black-box Testing

 Test cases are designed using only

functional specification of the
software:
 without any knowledge of the internal
structure of the software.
 For this reason, black-box testing is
also known as functional testing.

1
Black-box Testing

 There are essentially two main

approaches to design black box
test cases:
 Equivalence class partitioning
 Boundary value analysis

2
Equivalence Class
Partitioning

 Input values to a program are

partitioned into equivalence classes.
 Partitioning is done such that:
 program behaves in similar ways to
every input value belonging to an
equivalence class.

3
Why define equivalence
classes?

 Test the code with just one

representative value from each
equivalence class:
 as good as testing using any other
values from the equivalence classes.

4
Equivalence Class
Partitioning

 How do you determine the

equivalence classes?
 examine the input data.
 few general guidelines for
determining the equivalence
classes can be given

5
Equivalence Class
Partitioning
 If the input data to the program is
specified by a range of values:
 e.g. numbers between 1 to 5000.
 one valid and two invalid equivalence
classes are defined.
1 5000

6
Equivalence Class
Partitioning
 If input is an enumerated set of
values:
 e.g. {a,b,c}
 one equivalence class for valid input
values
 another equivalence class for invalid
input values should be defined.

7
Example
 A program reads an input value
in the range of 1 and 5000:
 computes the square root of the
input number
SQR
T

8
Example (cont.)
 There are three equivalence classes:
 the set of negative integers,
 set of integers in the range of 1 and
5000,
 integers larger than 5000.

1 5000

9
Example (cont.)
 The test suite must include:
 representatives from each of the
three equivalence classes:
 a possible test suite can be:
{-5,500,6000}.
1 5000

10
E-commerce website discount
Purchases between Rs.500 and Rs. 1000 ->
10% discount
Purchases between Rs. 1001 and Rs. 2000 ->
20% discount
Purchases between Rs. 2001 and Rs. 5000 ->
30% discount
Purchases above Rs. 5000 -> 40% discount
11
12
13
14
15
Boundary Value Analysis

 Some typical programming errors occur:

 at boundaries of equivalence classes
 might be purely due to psychological
factors.
 Programmers often fail to see:
 special processing required at the
boundaries of equivalence classes.

16
Boundary Value Analysis

 Programmers may improperly

use < instead of <=
 Boundary value analysis:
 select test cases at the
boundaries of different
equivalence classes.
17
18
19
Example

 For a function that computes the

square root of an integer in the
range of 1 and 5000:
 test cases must include the
values: {0,1,5000,5001}.
1 5000

20
21
22
23
24
25
26
27
28
29
30
31
32
Levels of Testing

 Software products are tested at

three levels:
 Unit testing
 Integration testing
 System testing

33
Unit testing
 During unit testing, modules are
tested in isolation:
 If all modules were to be tested
together:
 it may not be easy to determine
which module has the error.

34
Unit testing

 Unit testing reduces debugging

effort several folds.
 Programmers carry out unit
testing immediately after they
complete the coding of a module.

35
Integration testing

 After different modules of a system

have been coded and unit tested:
 modules are integrated in steps
according to an integration plan
 partially integrated system is
tested at each integration step.

36
System Testing

 System testing involves:

 validating a fully developed system
against its requirements.

37
Integration Testing

 Develop the integration plan by

examining the structure chart :
 big bang approach
 top-down approach
 bottom-up approach
 mixed approach

38
Big bang Integration
Testing
 Big bang approach is the simplest
integration testing approach:
 all the modules are simply put
together and tested.
 this technique is used only for
very small systems.

39
Big bang Integration
Testing
 Main problems with this approach:
 if an error is found:
 it is very difficult to localize the error
 the error may potentially belong to any of the
modules being integrated.
 debugging errors found during big bang
integration testing are very expensive to
fix.

40
41
Bottom-up Integration
Testing
 Integrate and test the bottom level
modules first.
 A disadvantage of bottom-up
testing:
 when the system is made up of a
large number of small subsystems.
 This extreme case corresponds to the
big bang approach.
42
43
Top-down integration
testing
 Top-down integration testing starts with
the main routine:
 and one or two subordinate routines in the
system.
 After the top-level 'skeleton’ has been
tested:
 immediate subordinate modules of the
'skeleton’ are combined with it and tested.

44
45
Mixed integration testing

 Mixed (or sandwiched)

integration testing:
 uses both top-down and bottom-
up testing approaches.
 Most common approach

46
az

47
Integration Testing
 In top-down approach:
 testing waits till all top-level
modules are coded and unit
tested.
 In bottom-up approach:
 testing can start only after bottom
level modules are ready.

48
Phased versus Incremental
Integration Testing

 Integration can be incremental or

phased.
 In incremental integration testing,
 only one new module is added to
the partial system each time.

49
Phased versus Incremental
Integration Testing

 In phased integration,
 a group of related modules are
added to the partially integrated
system each time.

50
Phased versus Incremental
Integration Testing
 Phased integration requires less
number of integration steps:
 compared to the incremental integration
approach.
 However, when failures are detected,
 it is easier to debug if using incremental
testing
 since errors are very likely to be in the
newly integrated module.

51
System Testing

 There are three main kinds of

system testing:
 Alpha Testing
 Beta Testing
 Acceptance Testing

52
Alpha Testing

 System testing is carried out by

the test team within the
developing organization.

53
Beta Testing

 System testing performed by a select

group of friendly customers.

54
Acceptance Testing

 System testing performed by the

customer himself:
 to determine whether the system
should be accepted or rejected.

55
System Testing

 During system testing, in addition

to functional tests:
 performance tests are performed.

56
Performance Testing

 Addresses non-functional
requirements.
 May sometimes involve testing hardware
and software together.
 There are several categories of
performance testing.

57
Stress testing

 Evaluates system performance

 when stressed for short periods of
time.
 Stress testing
 also known as endurance testing.

58
Stress testing
 Stress tests are black box tests:
 designed to impose a range of
abnormal and even illegal input
conditions
 so as to stress the capabilities of the
software.

59
Stress Testing
 If the requirements is to handle a
specified number of users, or
devices:
 stress testing evaluates system
performance when all users or
devices are busy simultaneously.

60
Stress Testing
 If an operating system is supposed to
support 15 multiprogrammed jobs,
 the system is stressed by attempting to run 15
or more jobs simultaneously.
 A real-time system might be tested
 to determine the effect of simultaneous arrival
of several high-priority interrupts.

61
Volume Testing
 Addresses handling large amounts of
data in the system:
 whether data structures (e.g. queues,
stacks, arrays, etc.) are large enough to
handle all possible situations
 Fields, records, and files are stressed to
check if their size can accommodate all
possible data volumes.

62
Configuration Testing
 Analyze system behavior:
 in various hardware and software
configurations specified in the requirements
 sometimes systems are built in various
configurations for different users
 for instance, a minimal system may serve a
single user,
 other configurations for additional users.

63
Compatibility Testing

 These tests are needed when the

system interfaces with other
systems:
 check whether the interface
functions as required.

64
Compatibility testing
Example
 If a system is to communicate
with a large database system to
retrieve information:
 a compatibility test examines speed
and accuracy of retrieval.

65
Recovery Testing

 These tests check response to:

 presence of faults or to the loss of
data, power, devices, or services
 subject system to loss of resources
 check if the system recovers properly.

66
Maintenance Testing

 Verify that:
 all required artifacts for
maintenance exist
 they function properly

67
Documentation tests

 Check that required documents

exist and are consistent:
 user guides,
 maintenance guides,
 technical documents

68
Documentation tests

 Sometimes requirements specify:

 format and audience of specific
documents
 documents are evaluated for
compliance

69
Usability tests

 All aspects of user interfaces are

tested:
 Display screens
 messages
 report formats
 navigation and selection problems

70
Regression Testing

 Does not belong to either unit

test, integration test, or system
test.
 In stead, it is a separate dimension
to these three forms of testing.

71
Regression testing

 Regression testing is the running of

test suite:
 after each change to the system or after
each bug fix
 ensures that no new bug has been
introduced due to the change or the bug
fix.

72
Regression testing

 Regression tests assure:

 the new system’s performance is at
least as good as the old system
 always used during phased system
development.

73
How many errors are still
remaining?
 Seed the code with some known
errors:
 artificial errors are introduced into
the program.
 Check how many of the seeded
errors are detected during testing.

74
Error Seeding
 Let:
 N be the total number of errors in the
system
 n = non-seeded errors found during
testing.
 S be the total number of seeded
errors,
 s of the seeded errors be found during
testing.
75
Error Seeding

 n/N = s/S
 N = S n/s
 remaining defects:
N - n = n ((S - s)/ s)

76
Example

 100 errors were introduced.

 90 of these errors were found during
testing
 50 other errors were also found.
 Remaining errors=
50 (100-90)/90 = 6

77
A software was tested using the error seeding strategy in which 20 errors were
seeded in the code. When the code was tested using the complex test suite,
16 of the seeded errors were detected. The same test suite also detected 200
non-seeded errors. What is the estimated number of unseeded errors in the
code after this testing?

78