Module 5.

1
■ Software Testing Strategies

1
Software Testing

Testing is the process of exercising a

program with the specific intent of
finding errors prior to delivery to the
end user.

2
 Testing

• One of the practical methods commonly used to detect the

presence of errors (failures) in a computer program is to test it
for a set of inputs. The output
is correct?

I1, I2, I3, Our program

…, In, … Expected results
=?
Obtained results

“Inputs
”
 Testing Strategy
A strategy for software testing provides a road map that describes the
steps to be conducted as part of testing, when these steps are
planned and then undertaken, and how much effort, time, and
resources will be required.

Therefore, any testing strategy must incorporate test planning, test

case design, test execution, and resultant data collection and
evaluation.

A software testing strategy should be flexible enough to promote a

customized testing approach. At the same time, it must be rigid
enough to encourage reasonable planning and management tracking
as the project progresses.

4
Testing Objectives
The Major Objectives of Software Testing:

- Uncover as many as errors (or bugs) as possible in a given timeline.

- Demonstrate a given software product matching its requirement

specifications.

- Validate the quality of a software testing using the minimum cost

and efforts.

- Generate high quality test cases, perform effective tests, and issue
correct and helpful problem reports.

5
What Testing Shows
errors
requirements conformance

performance

an indication
of quality

6
Strategic Approach
■ To perform effective testing, you should conduct
effective technical reviews. By doing this, many errors
will be eliminated before testing commences.
■ Testing begins at the component level and works
"outward" toward the integration of the entire computer-
based system.
■ Different testing techniques are appropriate for different
software engineering approaches and at different points
in time.
■ Testing is conducted by the developer of the software
and (for large projects) an independent test group.
■ Testing and debugging are different activities, but
debugging must be accommodated in any testing
strategy.

7
V&V
■ Verification refers to the set of tasks that ensure
that software correctly implements a specific
function.
■ Validation refers to a different set of tasks that
ensure that the software that has been built is
traceable to customer requirements. Boehm
[Boe81] states this another way:
■ Verification: "Are we building the product right?"
■ Validation: "Are we building the right product?"

8
Who Tests the Software?

developer independent tester

Understands the system Must learn about the system,
but, will test "gently" but, will attempt to break it
and, is driven by "delivery" and, is driven by quality

9
 Who Tests the Software?
For every software project, there is an inherent conflict
of interest that occurs as testing begins.

Developer treads lightly, designing and executing tests

that will demonstrate that the program works, rather than
uncovering errors.

Unfortunately, errors will be present. And, if the software

engineer doesn’t find them, the customer will!

10
 Who Tests the Software?
The role of an independent test group (ITG) is to remove the inherent
problems associated with letting the builder test the thing that has been
built. Independent testing removes the conflict of interest that may
otherwise be present.

However, you don’t turn the program over to ITG and walk away. The
developer and the ITG work closely throughout a software project to
ensure that thorough tests will be conducted. While testing is conducted,
the developer must be available to correct errors that are uncovered.

The ITG is part of the software development project team in the sense
that it becomes involved during analysis and design and stays involved
(planning and specifying test procedures) throughout a large project.

11
 TEST CASE

▪Documentation plays a critical role in testing.

▪ There are different levels of documentation, like:
1. Test Script: A line-by-line description of all the
actions and data needed to perform a test.
2. Test Case: Describes a specific idea that is to be
tested, without detailing the exact steps to be taken
or data to be used.
3. Test Scenario: It is a simple description of an
 TEST CASE FORMAT

• Test Case Name: A test case should have a name or title

that is self-explanatory.
• Test Case Description: The description should tell the
tester what they’re going to test in brief.
• Pre-Conditions: Any assumptions that apply to the test
and any preconditions that must be met prior to the test
being executed should be listed here.
• Test Case Steps: The test steps should include the
necessary data and information on how to execute the
test. The steps should be clear and brief, without leaving
out essential facts.
 TEST CASE FORMAT

• Test Data: It’s important to select a data set that gives

sufficient coverage. Select a data set that specifies not
only the positive scenarios but negative ones as well.
• Expected Result: The expected results tell the tester
what they should experience as a result of the test steps.
• Actual Result: They specifies how the application
actually behaved while test cases were being executed.
• Comments: Any other useful information such as
screenshots that tester want’s to specify can be included
here.
 EXAMPLE OF TEST CASE

Title: Login Page – Authenticate user on Hotmail.com

Description: A user should be able to log in at hotmail.com.
Precondition: The user must have an email address and
password that is previously registered.
Assumption: The browser supports hotmail.com
Test Steps:
1. Navigate to hotmail.com
2. Enter the email address of the registered user in the
’email’ field.
3. Enter the password of the registered user
4. Click the ‘Next’ button.
5. Click ‘Log in’
EXAMPLE OF TEST CASE
Testing Strategy
System engineering

Analysis modeling
Design modeling

Code generation Unit test

Integration test
Validation test

System test

17
 Testing Strategy
Organizing software testing:
To be done by the developers and an independent testing group.
A software testing strategy: Viewed as a spiral.

Software engineering S
Requirements R
Design D
Code C
U Unit test
I Integration test
V Validation test

ST System Test
To develop🡪 spiral in to test 🡪 spiral out 18
Testing Strategy

19
Testing Strategy
■ We begin by ‘testing-in-the-small’ and move toward
‘testing-in-the-large’
■ For conventional software
■ The module (component) is our initial focus
■ Integration of modules follows
■ For OO software
■ our focus when “testing in the small” changes from an
individual module (the conventional view) to an OO class
that encompasses attributes and operations and implies
communication and collaboration

20
Strategic Issues
■ Specify product requirements in a quantifiable manner long
before testing commences.
■ State testing objectives explicitly.
■ Understand the users of the software and develop a profile for
each user category.
■ Develop a testing plan that emphasizes “rapid cycle testing.”
■ Build “robust” software that is designed to test itself
■ Use effective technical reviews as a filter prior to testing
■ Conduct technical reviews to assess the test strategy and test
cases themselves.
■ Develop a continuous improvement approach for the testing
process.

21
What is Error, bug, defect
and failure

“A mistake in coding is called error ,error found by

tester is called defect, defect accepted by
development team then it is called bug ,build does
not meet the requirements then it Is failure.”
Examples
1.Error: For example: Let me explain with an example. If the requirement mentions 5 Plus
3 Divided by 4; the programmer codes to add 5 and 3 first; and then divides the result [8]
by 4 to arrive at result 2.
2.But if the actual/intended application requirement is to first divide 3 by 4 and then add
5; expecting a result 5.75 then this is a classic Design/Code Error.

Bug: Assume that the programmer has actually understood the intended requirement
correctly but has declared/stored the result as an integer to show a result of 5 instead of
5.75 – this is called a Bug. Here the programmer has to declare the variable as “double or
float” and not “int”.
Defect: A defect is any of the above but found/un-covered in the “released-application” -
running in the “Production Servers” [gone “Live”] or may be in “Beta-Release”
Fault: When the application runs without any issues in the intended Production
Environment under normal circumstances, but misbehaves after continuously running for
about a month or after performing about a million transactions, then we call this condition
as Fault.
 Criteria for Completion of Testing

A classic question arises every time software testing is discussed:

“When are we done testing—how do we know that we’ve tested enough?”

Sadly, there is no definitive answer to this question,

“You’re never done testing; the burden simply shifts from you (the
software engineer) to the end user.” Every time the user executes a
computer program, the program is being tested.

“You’re done testing when you run out of time or you run out of money.”

One approach is to divide the test results into various severity levels
Then consider testing to be complete when certain levels of errors no longer
occur or have been repaired or eliminated
 Testing Strategy

■ Unit testing
■ Exercises specific paths in a component's control structure to ensure
complete coverage and maximum error detection
■ Components are then assembled and integrated
■ Integration testing
■ Focuses on inputs and outputs, and how well the components fit together
and work together
■ Validation testing
■ Provides final assurance that the software meets all functional, behavioral,
and performance requirements
■ System testing
■ Verifies that all system elements (software, hardware, people, databases)
mesh properly and that overall system function and performance is
achieved
Unit Testing

module
to be
tested

results

software
engineer
test cases

26
 Unit Testing
■ Focuses testing on the function or software module
■ Concentrates on the internal processing logic and data structures
■ Is simplified when a module is designed with high cohesion
■ Reduces the number of test cases
■ Allows errors to be more easily predicted and uncovered
■ Concentrates on critical modules and those with high cyclomatic
complexity when testing resources are limited

27
Targets for Unit Test Cases
■ Module interface
■ Ensure that information flows properly into and out of the
module
■ Local data structures
■ Ensure that data stored temporarily maintains its integrity during
all steps in an algorithm execution
■ Boundary conditions
■ Ensure that the module operates properly at boundary values
established to limit or restrict processing
■ Independent paths (basis paths)
■ Paths are exercised to ensure that all statements in a module have
been executed at least once
■ Error handling paths
■ Ensure that the algorithms respond correctly to specific error
conditions

28
 Common Computational Errors
in Execution Paths

■ Misunderstood or incorrect arithmetic precedence

■ Mixed mode operations (e.g., int, float, char)
■ Incorrect initialization of values
■ Precision inaccuracy and round-off errors
■ Incorrect symbolic representation of an expression (int vs. float)

29
Other Errors to Uncover
■ Comparison of different data types
■ Incorrect logical operators or precedence
■ Expectation of equality when precision error makes equality
unlikely (using == with float types)
■ Incorrect comparison of variables
■ Improper or nonexistent loop termination
■ Failure to exit when divergent iteration is encountered
■ Improperly modified loop variables
■ Boundary value violations

30
Problems to uncover in
Error Handling
■ Error description is unintelligible or ambiguous
■ Error noted does not correspond to error encountered
■ Error condition causes operating system intervention prior to
error handling
■ Exception condition processing is incorrect
■ Error description does not provide enough information to
assist in the location of the cause of the error

31
Testing Procedure
■ Unit testing is normally considered as an adjunct to the coding step.
■ The design of unit tests can occur before coding begins or after source
code has been generated.
■ A review of design information provides guidance for establishing test
cases that are likely to uncover errors.
■ Each test case should be coupled with a set of expected results.
■ Because a component is not a stand-alone program, driver and/or stub
software must often be developed for each unit test.

32
Drivers and Stubs for
Unit Testing
■ Driver
■ A simple main program that accepts test case data, passes such data to the
component being tested, and prints the returned results
■ Stubs
■ Serve to replace modules that are subordinate to (called by) the
component to be tested
■ It uses the module’s exact interface, may do minimal data manipulation,
provides verification of entry, and returns control to the module
undergoing testing
■ Drivers and stubs both represent overhead
■ Both must be written but don’t constitute part of the installed software
product

33
Unit Testing

34
Unit Test Environment
driver
interface
local data structures

Module boundary conditions

independent paths
error handling paths

stub stub

test cases

RESULTS
35
 Integration Testing
“If they all work individually, why do you doubt that
they’ll work when we put them together?”

The problem, of course, is “putting them together”—interfacing.

Examples - Data can be lost across an interface;

1.one component can have an inadvertent, adverse effect on another;

2.Sub-functions, when combined, may not produce the desired major
function;
3.individually acceptable imprecision may be magnified to unacceptable
levels;
4.global data structures can present problems

36
UNIT TESTING
 Integration Testing
Integration testing is a systematic technique for constructing
the software architecture while at the same time conducting
tests to uncover errors associated with interfacing.

The objective is to take unit-tested components and build a

program structure that has been dictated by design.

38
Integration Testing Strategies
Options:
• the “big bang” approach
• an incremental construction strategy

39
 Integration Testing - Big Bang
1. All components are combined in advance.
2. The entire program is tested as a whole.
3. A set of errors is encountered.
4. Correction is difficult because isolation of causes is
complicated by the vast expanse of the entire program.
5. Once these errors are corrected, new ones appear and
the process continues in a seemingly endless loop.

40
 Integration Testing - Incremental
1. Incremental integration is the Solution to the issues
encountered with big bang approach.
2. The program is constructed and tested in small
increments,
3. errors are easier to isolate and correct;
4. interfaces are more likely to be tested completely; and a
systematic test approach may be applied.

• Top Down
• Bottom up
• Regression
• Smoke 41
 Top Down Integration
1. Top-down integration testing is an incremental approach
to construction of the software architecture.

2. Modules are integrated by moving downward through

the control hierarchy, beginning with the main control
module(main program).

3. Modules subordinate (and ultimately subordinate) to the

main control module are incorporated into the structure
in either a depth-first or breadth-first manner.

42
Top Down Integration A

B C D

E F G

Test
Tes Test A, A, B,
tA B, C, D C, D,
E, F, G

Layer I Layer I + II All

43
Layers
Top Down Integration

44
Top Down Integration – Depth First
1. depth-first integration integrates all components on a
major control path of the program structure. Selection
of a major path is somewhat arbitrary and depends on
application-specific characteristics.
2. For example, selecting the left-hand path, components
M1, M2 , M5 would be integrated first.
3. Next, M8 or (if necessary for proper functioning of M2)
M6 would be integrated.
4. Then, the central and right-hand control paths are built.

45
Top Down Integration – Breadth First
1. Breadth-first integration incorporates all components
directly subordinate at each level, moving across the
structure horizontally.
2. From the figure, components M2, M3, and M4 would be
integrated first.
3. The next control level, M5, M6, and so on, follows.

46
Top Down Integration – Steps
The integration process is performed in a series of five steps:
1. The main control module is used as a test driver and stubs are substituted
for all components directly subordinate to the main control module.
2. Depending on the integration approach selected (i.e., depth or breadth first),
subordinate stubs are replaced one at a time with actual components.
3. Tests are conducted as each component is integrated.
4. On completion of each set of tests, another stub is replaced with the real
component.
5. Regression testing (discussed later in this section) may be conducted to
ensure that new errors have not been introduced.

47
Top Down Integration – Pros
1. The top-down integration strategy verifies major control or decision
points early in the test process.
2. In a “well-factored” program structure, decision making occurs at
upper levels in the hierarchy and is therefore encountered first.
3. If major control problems do exist, early recognition is essential.
4. If depth-first integration is selected, a complete function of the
software may be implemented and demonstrated.
5. Early demonstration of functional capability is a confidence builder
for all stakeholders.
48
Top Down Integration – Cons
1. Top-down strategy sounds relatively uncomplicated, but in
practice, logistical problems can arise.
2. The most common of these problems occurs when processing at
low levels in the hierarchy is required to adequately test upper
levels.
3. Stubs need to be created to substitute for modules that have not
been built or tested yet; this code is later discarded
4. Writing stubs is difficult: Stubs must allow all possible conditions to
be tested.
5. Large number of stubs may be required, especially if the lowest
level of the system contains many methods.
6. Some interfaces are not tested separately.

49
 Bottom-Up Integration
1. Bottom-up integration testing, as its name implies,
begins construction and testing with atomic modules
(i.e., components at the lowest levels in the program
structure).
2. Because components are integrated from the bottom
up, the functionality provided by components
subordinate to a given level is always available and the
need for stubs is eliminated.

50
 Bottom-Up Integration - Steps

1. Low-level components are combined into clusters

(sometimes called builds) that perform a specific software
sub-function.
2. A driver (a control program for testing) is written to
coordinate test case input and output.
3. The cluster is tested.
4. Drivers are removed and clusters are combined moving
upward in the program structure. 51
 Bottom-Up Integration A

B C D

Tes
tE E F G
Test B,
E, F
Tes
tF
Tes Test
tC A, B,
C, D,
E, F, G
Tes Test
tG D,G 52
 INTEGRATION TESTING- Example

let us consider the Yahoo.com page.

 INTEGRATION TESTING- Example

▪In Top Down Integration, the order of integration could be:

1. Yahoo Homepage with Mail
2. Yahoo Homepage with News
3. Yahoo Homepage with Shopping
4. Yahoo Homepage with Finance
5. Mail with Inbox
6. Mail with Sent
7. Mail with Contacts
▪ If News module is not fully coded, it is replaced by a
skeletal implementation of the component called
a STUB. For example, a static page with this a message
this Page is still under construction. A Stub is called by
other components.
 INTEGRATION TESTING- Example

▪In Bottom-Up Integration, the order of Integration could

be:
1. Contacts with Mail
2. Sent with Mail
3. Inbox with Mail
4. Mail with Yahoo Homepage
5. News with Yahoo Homepage
6. Finance with Yahoo Homepage
7. Shopping with Yahoo Homepage
▪ Opposite of top-down, the components here are integrated
in a bottom-up order. If the ‘Yahoo Homepage’ is yet to be
fully built, special components called ‘DRIVERS’ are used
to test the integration. The Drivers are later replaced by the
Bottom-Up Integration

56
 Bottom-Up Integration
1. Components are combined to form clusters 1, 2, and 3.
2. Each of the clusters is tested using a driver (shown as a
dashed block).
3. Components in clusters 1 and 2 are subordinate to Ma.
Drivers D1 and D2 are removed and the clusters are
interfaced directly to Ma.
4. Similarly, driver D3 for cluster 3 is removed prior to
integration with module Mb.
5. Both Ma and Mb will ultimately be integrated with
component Mc, and so forth.

57
Bottom-Up Integration – Pros/Cons
1. Con:
a) Tests the most important subsystem (user interface)
last
b) Drivers needed

2. Pro
a) No stubs needed
b) Useful for integration testing of the following systems
i. Object-oriented systems
ii. Real-time systems
iii. Systems with strict performance requirements.
58
 Sandwich Testing

1. Combines top-down strategy with bottom-up strategy

2. The system is viewed as having three layers
a) A target layer in the middle
b) A layer above the target
c) A layer below the target
3. Testing converges at the target layer.

59
 Sandwich Testing A

B C D
Tes
tA
E F G
Test
Tes A,B,C, D
tE
Test
Test B, E, A, B,
Tes F C, D,
tF E, F, G

Test D,G
Tes
tG 60
Sandwich Testing – Pros/ Cons
1. Top and Bottom Layer Tests can be done in parallel

2. Problem: Does not test the individual subsystems and

their interfaces thoroughly before integration

3. Solution: Modified sandwich testing strategy

61
Modified Sandwich Testing
1. Test in parallel:
a) Middle layer with drivers and stubs
b) Top layer with stubs
c) Bottom layer with drivers
2. Test in parallel:
a) Top layer accessing middle layer (top layer replaces
drivers)
b) Bottom accessed by middle layer (bottom layer
replaces stubs).

62
Modified Sandwich Testing
A

B C D
Tes
tA Test
Tes A,C E G
tC F
Te
st Test
Te
B A, B,
Test B,
st E, F
C, D,
Te
E E, F,
st G
F
Tes
tD Test
Tes D,G
63
tG
 Regression Testing
■ Each time a new module is added as part of integration
testing,the software changes.
■ New data flow paths are established, new I/O may occur,
and new control logic is invoked.
■ These changes may cause problems with functions that
previously worked flawlessly.
■ In the context of an integration test strategy, regression
testing is the reexecution of some subset of tests that have
already been conducted to ensure that changes have not
propagated unintended side effects.

64
 Regression Testing
■ software configuration (the program, its documentation,
or the data that support it) is changed.
■ Regression testing helps to ensure that changes (due to
testing or for other reasons) do not introduce unintended
behavior or additional errors.
■ Regression testing may be conducted manually, by re-
executing a subset of all test cases or using automated
capture/playback tools.
■ Capture/playback tools enable the software engineer to capture
test cases and results for subsequent playback and comparison.

65
 Regression Test Suite
■ The regression test suite (the subset of tests to be executed) contains three different
classes of test cases:
■ A representative sample of tests that will exercise all software functions.
■ Additional tests that focus on software functions that are likely to be affected by
the change.
■ Tests that focus on the software components that have been changed.
■ As integration testing proceeds, the number of regression tests can grow quite large.
■ Therefore, the regression test suite should be designed to include only those tests
that address one or more classes of errors in each of the major program functions.
■ It is impractical and inefficient to re-execute every test for every program function
once a change has occurred.

66
 Smoke Testing
■ Taken from the world of hardware
■ Power is applied and a technician checks for sparks, smoke, or other
dramatic signs of fundamental failure
■ Designed as a pacing mechanism for time-critical projects
■ Allows the software team to assess its project on a frequent basis
■ Includes the following activities
■ The software is compiled and linked into a build
■ A series of breadth tests is designed to expose errors that will keep the build
from properly performing its function
• The goal is to uncover “show stopper” errors that have the highest likelihood of
throwing the software project behind schedule
■ The build is integrated with other builds and the entire product is smoke
tested daily
• Daily testing gives managers and practitioners a realistic assessment of the
progress of the integration testing
■ After a smoke test is completed, detailed test scripts are executed

67
 Smoke Testing - Advantages
■ Smoke testing provides a number of benefits when it is applied on
complex, time critical software projects:
■ Integration risk is minimized. Because smoke tests are conducted daily,
incompatibilities and other show-stopper errors are uncovered early, thereby
reducing the likelihood of serious schedule impact when errors are
uncovered.
■ The quality of the end product is improved. Because the approach is
construction (integration) oriented, smoke testing is likely to uncover
functional errors as well as architectural and component-level design errors.
If these errors are corrected early, better product quality will result.

68
 Smoke Testing - Advantages
■ Error diagnosis and correction are simplified. Like all integration testing
approaches, errors uncovered during smoke testing are likely to be
associated with “new software increments”—that is, the software that has
just been added to the build(s) is a probable cause of a newly discovered
error.
■ Progress is easier to assess. With each passing day, more of the software has
been integrated and more has been demonstrated to work. This improves
team morale and gives managers a good indication that progress is being
made.

69
 Validation Testing

■ Validation testing follows integration testing

■ Focuses on user-visible actions and user-recognizable output
from the system
■ Demonstrates conformity with requirements
■ Designed to ensure that
■ All functional requirements are satisfied
■ All behavioral characteristics are achieved
■ All performance requirements are attained
■ Documentation is correct
■ Usability and other requirements are met (e.g., transportability,
compatibility, error recovery, maintainability)
Validation Testing

■ After each validation test

■ The function or performance characteristic conforms to specification and is
accepted
■ A deviation from specification is uncovered and a deficiency list is created
■ A configuration review or audit ensures that all elements of the
software configuration have been properly developed, cataloged,
and have the necessary detail for entering the support phase of the
software life cycle

71
COMPARISON OF VERIFICATION VALIDATION

VERIFICATION VALIDATION
1. It includes checking 1. It includes testing and
documents, design, codes and validating the actual product.
programs.
2. Verification is the static 2. Validation is the dynamic
testing. testing.
3. Methods used in 3. Methods used in validation
verification are reviews, are Black Box Testing, White
walkthroughs, inspections Box Testing and non-
and desk-checking. functional testing.
4. It checks whether the 4. It checks whether the
software conforms to software meets the
specifications or not. requirements and
COMPARISON OF VERIFICATION VALIDATION

VERIFICATION VALIDATION
5. It comes before 5. It comes after verification.
validation.
6. It does not include the 6. It includes the execution of
execution of the code. the code.
7. It can find the bugs in the 7. It can only find the bugs that
early stage of the could not be found by the
development. verification process.
8. Verification is done by 8. Validation is executed on
QA team to ensure that the software code with the help of
software is as per the testing team.
specifications in the SRS
Alpha and Beta Testing
■ Alpha testing
■ Conducted at the developer’s site by end users
■ Software is used in a natural setting with developers watching
intently
■ Testing is conducted in a controlled environment
■ Beta testing
■ Conducted at end-user sites
■ Developer is generally not present
■ It serves as a live application of the software in an environment
that cannot be controlled by the developer
■ The end-user records all problems that are encountered and
reports these to the developers at regular intervals
■ After beta testing is complete, software engineers make
software modifications and prepare for release of the software
product to the entire customer base

74
Customer Acceptance Testing
1. A variation on beta testing, called customer
acceptance testing, is sometimes performed when
custom software is delivered to a customer under
contract.
2. The customer performs a series of specific tests in an
attempt to uncover errors before accepting the
software from the developer.
3. In some cases (e.g., a major corporate or
governmental system) acceptance testing can be
very formal and encompass many days or even
weeks of testing.

75
 System Testing
1. Software is only one element of a larger computer-based
system.
2. Ultimately, software is incorporated with other system
elements (e.g., hardware, people, information), and a
series of system integration and validation tests are
conducted.
3. These tests fall outside the scope of the software
process and are not conducted solely by software
engineers.
4. However, steps taken during software design and testing
can greatly improve the probability of successful software
integration in the larger system
76
 System Testing
1. System testing is actually a series of different tests
whose primary purpose is to fully exercise the computer-
based system.
2. Although each test has a different purpose, all work to
verify that system elements have been properly
integrated and perform allocated functions.
a) Recovery Testing
b) Security Testing
c) Stress Testing
d) Performance Testing
e) Deployment Testing

77
 System Testing - Recovery
1. Many computer-based systems must recover from faults and
resume processing with little or no downtime. In some cases, a
system must be fault tolerant; that is, processing faults must not
cause overall system function to cease.
2. In other cases, a system failure must be corrected within a specified
period of time or severe economic damage will occur.
3. Recovery testing is a system test that forces the software to fail in a
variety of ways and verifies that recovery is properly performed.
4. If recovery is automatic (performed by the system itself), re-
initialization, check-pointing mechanisms, data recovery, and restart
are evaluated for correctness.
5. If recovery requires human intervention, the mean-time-to-repair
(MTTR) is evaluated to determine whether it is within acceptable
limits.
78
 System Testing - Security
1. Any computer-based system that manages sensitive information or
causes actions that can improperly harm (or benefit) individuals is a
target for improper or illegal penetration.
2. Penetration spans a broad range of activities: hackers who attempt
to penetrate systems for sport, disgruntled employees who attempt
to penetrate for revenge, dishonest individuals who attempt to
penetrate for illicit personal gain.
3. Security testing attempts to verify that protection mechanisms built
into a system will, in fact, protect it from improper penetration.
4. During security testing, the tester plays the role(s) of the individual
who desires to penetrate the system. Anything goes!

79
 System Testing - Security
1. The tester may attempt to acquire passwords hrough external clerical
means;
a) may attack the system with custom software designed to break down
any defenses that have been constructed;
b) may overwhelm the system, thereby denying service to others;
c) may purposely cause system errors, hoping to penetrate during
recovery;
d) may browse through insecure data, hoping to find the key to system
entry.
2. Given enough time and resources, good security testing will ultimately
penetrate a system.
3. The role of the system designer is to make penetration cost more than the
value of the information that will be obtained

80
 System Testing - Stress
1. Stress tests are designed to confront programs with abnormal
situations.
2. Stress testing executes a system in a manner that demands
resources in abnormal quantity, frequency, or volume. For example,
a) special tests may be designed that generate ten interrupts per
second, when one or two is the average rate,
b) input data rates may be increased by an order of magnitude to
determine how input functions will respond,
c) test cases that require maximum memory or other resources are
executed,
d) test cases that may cause thrashing in a virtual operating
system are designed,
e) test cases that may cause excessive hunting for disk-resident
data are created.
81
 System Testing - Sensitivity
1. A variation of stress testing is a technique called sensitivity testing.
2. In some situations (the most common occur in mathematical
algorithms), a very small range of data contained within the bounds
of valid data for a program may cause extreme and even erroneous
processing or profound performance degradation.
3. Sensitivity testing attempts to uncover data combinations within valid
input classes that may cause instability or improper processing.

82
 System Testing - Performance
1. For real-time and embedded systems, software that provides
required function but does not conform to performance requirements
is unacceptable.
2. Performance testing is designed to test the run-time performance of
software within the context of an integrated system.
3. Performance testing occurs throughout all steps in the testing
process.
4. Even at the unit level, the performance of an individual module may
be assessed as tests are conducted. However, it is not until all
system elements are fully integrated that the true performance of a
system can be ascertained.

83
 System Testing - Performance
1. Performance tests are often coupled with stress testing and usually
require both hardware and software instrumentation.
2. That is, it is often necessary to measure resource utilization
1. (e.g., processor cycles) in an exacting fashion.
2. External instrumentation can monitor execution intervals, log
events (e.g., interrupts) as they occur, and sample machine
states on a regular basis.
3. By instrumenting a system, the tester can uncover situations that
lead to degradation and possible system failure.

84
 System Testing - Deployment
1. In many cases, software must execute on a variety of platforms and
under more than one operating system environment.
2. Deployment testing, sometimes called configuration testing,
exercises the software in each environment in which it is to operate.
3. In addition, deployment testing examines all installation procedures
and specialized installation software (e.g., “installers”) that will be
used by customers, and all documentation that will be used to
introduce the software to end users.
4. A more thorough deployment test might encompass combinations of
Web browsers with various operating systems (e.g., Linux, Mac OS,
Windows).
5. Because security is a major issue, a complete set of security tests
would be integrated with the deployment test.

85
Debugging: A Diagnostic Process

86
 Debugging: A Diagnostic Process
1. Debugging occurs as a consequence of successful testing. That is,
when a test case uncovers an error, debugging is the process that
results in the removal of the error.
2. Although debugging can and should be an orderly process, it is still
very much an art.
3. As a software engineer, you are often confronted with a
“symptomatic” indication of a software problem as you evaluate the
results of a test.
4. That is, the external manifestation of the error and its internal cause
may have no obvious relationship to one another.
5. The mental process that connects a symptom to a cause is
debugging.

87
The Debugging Process

These slides are designed to accompany Software Engineering: A Practitioner’s Approach, 7/e
(McGraw-Hill 2009). Slides copyright 2009 by Roger Pressman. 88
Why is Debugging so Difficult?
■ The symptom and the cause may be geographically remote
■ The symptom may disappear (temporarily) when another error is corrected
■ The symptom may actually be caused by nonerrors (e.g., round-off accuracies)
■ The symptom may be caused by human error that is not easily traced
■ The symptom may be a result of timing problems, rather than processing
problems
■ It may be difficult to accurately reproduce input conditions, such as asynchronous
real-time information
■ The symptom may be intermittent such as in embedded systems involving both
hardware and software
■ The symptom may be due to causes that are distributed across a number of tasks
running on different processes
Debugging Strategies
■ Objective of debugging is to find and correct the cause of a
software error
■ Bugs are found by a combination of systematic evaluation,
intuition, and luck
■ Debugging methods and tools are not a substitute for careful
evaluation based on a complete design model and clear source
code
■ There are three main debugging strategies
■ Brute force
■ Backtracking
■ Cause elimination

90
Strategy #1: Brute Force
■ Most commonly used and least efficient method
■ Used when all else fails
■ Involves the use of memory dumps, run-time traces, and
output statements
■ Leads many times to wasted effort and time

91
Strategy #2: Backtracking
■ Can be used successfully in small programs
■ The method starts at the location where a symptom has been
uncovered
■ The source code is then traced backward (manually) until the
location of the cause is found
■ In large programs, the number of potential backward paths
may become unmanageably large

92
 Strategy #3: Cause Elimination

■ Involves the use of induction or deduction and introduces the concept of

binary partitioning
■ Induction (specific to general): Prove that a specific starting value is true; then
prove the general case is true
■ Deduction (general to specific): Show that a specific conclusion follows from a
set of general premises
■ Data related to the error occurrence are organized to isolate potential
causes
■ A cause hypothesis is devised, and the aforementioned data are used to
prove or disprove the hypothesis
■ Alternatively, a list of all possible causes is developed, and tests are
conducted to eliminate each cause
■ If initial tests indicate that a particular cause hypothesis shows promise,
data are refined in an attempt to isolate the bug

93
Consequences of Bugs
infectious
damage

catastrophic
extreme
serious
disturbing

annoying
mild

Bug Type

Bug Categories: function-related bugs,

system-related bugs, data bugs, coding bugs,
design bugs, documentation bugs, standards
violations, etc.
94
Correcting the Error
■ Is the cause of the bug reproduced in another part of the program? In
many situations, a program defect is caused by an erroneous
pattern of logic that may be reproduced elsewhere.
■ What "next bug" might be introduced by the fix I'm about to make?
Before the correction is made, the source code (or, better, the
design) should be evaluated to assess coupling of logic and
data structures.
■ What could we have done to prevent this bug in the first place? This
question is the first step toward establishing a statistical
software quality assurance approach. If you correct the process
as well as the product, the bug will be removed from the
current program and may be eliminated from all future
programs.

95
 Summary
■ Software testing accounts for the largest percentage of technical effort in the
software process.

■ Regardless of the type of software you build, a strategy for systematic test
planning, execution, and control begins by considering small elements of the
software and moves outward toward the program as a whole.

■ Unlike testing (a systematic, planned activity), debugging can be viewed as

an art. Beginning with a symptomatic indication of a problem, the debugging
activity must track down the cause of an error.

96