Software Testing and Quality Assurance

Theory and Practice

Chapter 11
System Test Design

Software Testing and QA Theory and Practice (Chapter 11: System Test Design) © Naik & Tripathy 1
 Outline of the Chapter
• Test Design Factors
• Requirement Identification
• Test Objective Identification
• Modeling a Test Design Process
• Modeling Test Results
• Test Design Preparedness Metrics
• Test Case Design Effectiveness

Software Testing and QA Theory and Practice (Chapter 11: System Test Design) © Naik & Tripathy 2
 What is system testing ?
• System testing of software or hardware is testing conducted
on a complete, integrated system to evaluate the system's
compliance with its specified requirements.
• System testing is carried out by specialists testers or
independent testers.
• System testing should investigate both functional and non-
functional requirements of the testing.

Software Testing and QA Theory and Practice (Chapter 11: System Test Design) © Naik & Tripathy 3
What is Functional requirement?
It describes what a software system should do.
It describes the behavior of the system as it relates to the
system’s functionality.
Examples:
• Business Rules
• Transaction corrections, adjustments and cancellations
• Administrative functions
• Authentication
• Authorization levels

Software Testing and QA Theory and Practice (Chapter 11: System Test Design) © Naik & Tripathy 4
• External Interfaces
• Certification Requirements
• Reporting Requirements
• Legal or Regulatory Requirements

Software Testing and QA Theory and Practice (Chapter 11: System Test Design) © Naik & Tripathy 5
What is Non-Functional requirement?
It place constraints on how the system will do so.
It elaborates a performance characteristic of the system.
Examples:
• Performance
• Scalability
• Capacity
• Availability
• Reliability

Software Testing and QA Theory and Practice (Chapter 11: System Test Design) © Naik & Tripathy 6
• Recoverability
• Maintainability
• Serviceability
• Security
• Regulatory
• Manageability
• Environmental
• Data Integrity
• Usability
• Interoperability

Software Testing and QA Theory and Practice (Chapter 11: System Test Design) © Naik & Tripathy 7
Types of System Testing

• Black box testing

• White box testing
• Gray box testing

Software Testing and QA Theory and Practice (Chapter 11: System Test Design) © Naik & Tripathy 8
Black box testing

• A method of software testing that verifies the functionality

of an application without having specific knowledge of the
application’s code/internal structure.
• It is also known as functional testing.
• Tests are based on requirements and functionality.

Software Testing and QA Theory and Practice (Chapter 11: System Test Design) © Naik & Tripathy 9
White box testing

• It is a method of testing software that tests internal

structures or workings of an application, as opposed to its
functionality.
• also known as clear box testing, glass box testing,
transparent box testing, and structural testing.

Software Testing and QA Theory and Practice (Chapter 11: System Test Design) © Naik & Tripathy 10
Gray box testing

• It is a combination of white-box testing and black-box

testing.
• The aim of this testing is to search for the defects if any
due to improper structure or improper usage of applications
• For a complete software examination, both white box and
black box tests are required.

Software Testing and QA Theory and Practice (Chapter 11: System Test Design) © Naik & Tripathy 11
 Characteristics of Testable Requirements
• One way to determine the requirement description is testable is as
follows:
• Take the requirement description “The system must perform X.”

• Encapsulate the requirement description to create a test objective:

“Verify that the system performs X correctly.”

• Review this test objective and find out if it is possible to execute it

assuming that the system and the test environment are available

• If the answer to the above question is yes, then the requirement

description is clear and detailed for testing purpose
• Otherwise, more work needs to be done to revise or supplement the
requirement description

Software Testing and QA Theory and Practice (Chapter 11: System Test Design) © Naik & Tripathy 12
 Characteristics of Testable Requirements
• The following items must be analyzed during the review of
requirements:
– Safety
– Security
– Completeness
– Correctness
– Consistency
– Clarity
– Relevance
– Feasibility
– Verifiable
– Traceable

Software Testing and QA Theory and Practice (Chapter 11: System Test Design) © Naik & Tripathy 13
 Characteristics of Testable Requirements
• A functional specification provides
– a precise description of the major functions the system must fulfill the
requirements
– explanation of the technological risks involved
– external interfaces with other software modules
– data flow such as flowcharts, transaction sequence diagrams, and finite-state
machines describing the sequence of activities
– fault handling, memory utilization and performance estimates
– any engineering limitation

Software Testing and QA Theory and Practice (Chapter 11: System Test Design) © Naik & Tripathy 14
 Characteristics of Testable Requirements
• The following are the objectives that are kept in mind while
reviewing a functional specification:
– Achieving requirements
– Correctness
– Extensible
– Comprehensive
– Necessity
– Implementable
– Efficient
– Simplicity
– Consistency with existing components
– Limitations

Software Testing and QA Theory and Practice (Chapter 11: System Test Design) © Naik & Tripathy 15
Characteristics of Testable Requirements

Table 11.4: Characteristics of testable functional specification.

Software Testing and QA Theory and Practice (Chapter 11: System Test Design) © Naik & Tripathy 16
 Test Objective Identification
• The question “What do I test?” must be answered with another
question: “What do I expect the system to do?”
• The first step in identifying the test objective is to read, understand,
and analyze the functional specification
• It is essential to have a background familiarity with the subject area,
the goals of the system, business processes, and system users, for a
successful analysis
• One must critically analyze requirements to extract the inferred
requirements that are embedded in the requirements
• An inferred requirement is one that a system is expected to support,
but not explicitly stated
• Inferred requirements need to be tested just like the explicitly stated
requirements

Software Testing and QA Theory and Practice (Chapter 11: System Test Design) © Naik & Tripathy 17
 Test Objective Identification
• The test objectives are put together to form a test group or a
subgroup after they have been identified
• A set of (sub) groups of test cases are logically combined to form a
larger group
• A hierarchical structure of test groups as shown in Figure 11.2 is
called a test suite
• It is necessary to identify the test groups based on test categories, and
refine the test groups into sets of test objectives
• Individual test cases are created for each test objective within the
subgroups
• Test groups may be nested to an arbitrary depth
• The test grouping may be used to aid system test planning and
execution that are discussed in Chapters 12 and 13, respectively

Software Testing and QA Theory and Practice (Chapter 11: System Test Design) © Naik & Tripathy 18
 Test Objective Identification

Figure 11.2: Test Suite Structure

Software Testing and QA Theory and Practice (Chapter 11: System Test Design) © Naik & Tripathy 19
 Modeling a Test Design Process
• One test case is created for each test objective
• Each test case is designed as a combination of modular components called test
steps
• Test cases are clearly specified so that testers can quickly understand, borrow, and
re-use the test cases
• Figure 11.6 illustrate the life-cycle model of a test case in the form of a state
transition diagram
• One can easily implement a database of test cases using the test cases schema
shown in Table 11.6

Figure 11.6: State-transition diagram of a test case.

Software Testing and QA Theory and Practice (Chapter 11: System Test Design) © Naik & Tripathy 20
 Modeling a Test Design Process

Table 11.6: Test case schema summary.

Software Testing and QA Theory and Practice (Chapter 11: System Test Design) © Naik & Tripathy 21
 Modeling Test Results
• A test suite schema shown in Table 11.7 can be used for testing a
particular release
• The schema requires a test suite id, a title, an objective and a list of
test cases to be managed by the test suite
• The idea is to gather a selected number of released test cases and
repackage them to form a test suite for a new project
• The results of executing those test cases are recorded in a database
for gathering and analyzing test metrics
• The result of test execution is modeled by using a state-transition
diagram as shown in Figure 11.7
• The corresponding schema is given in Table 11.8

Software Testing and QA Theory and Practice (Chapter 11: System Test Design) © Naik & Tripathy 22
 Modeling Test Results

Table 11.7: Test suite field summary.

Software Testing and QA Theory and Practice (Chapter 11: System Test Design) © Naik & Tripathy 23
 Modeling Test Results

Table 11.8: Test result schema summary.

Software Testing and QA Theory and Practice (Chapter 11: System Test Design) © Naik & Tripathy 24
 Modeling Test Results

Figure 11.7: State transition diagram of a test case results

Software Testing and QA Theory and Practice (Chapter 11: System Test Design) © Naik & Tripathy 25
 Test Design Preparedness Metrics
• Metrics are tracked
– To know if a test project is progressing according to schedule
– Plan the next project more accurately

• The following metrics are monitored

– Preparation status of test cases (PST)
– Average time spent (ATS) in test case design
– Number of available test (NAT) cases
– Number of planned test (NPT) cases
– Coverage of a test suite (CTS)

Software Testing and QA Theory and Practice (Chapter 11: System Test Design) © Naik & Tripathy 26
 Test Case Design Effectiveness
• The objective of the test design effectiveness metric is to
– Measure the defect revealing ability of the test suite
– Use the metric to improve the test design process
• A metric commonly used in the industry to measure test case design
effectiveness is the Test Case Design Yield (TCDY)
NPT
TDCY  100%
NPT  Number of TCE

• TCE stand for Test Case Escaped

• During testing new defects are found for which no test cases had
been planned
• For these new defects, new test cases are designed, which is known
as Test Case Escaped metric

Software Testing and QA Theory and Practice (Chapter 11: System Test Design) © Naik & Tripathy 27