SOFTWARE QUALITY ENGINEERING

LECTURE#4: QUALITY ASSURANCE

 LECTURE OVERVIEW
 Quality Assurance (QA)
 Classification Scheme for QA as Dealing with Defects
1. Defect Prevention
2. Defect Reduction (Defect Detection and Removal)
3. Defect Containment
 DEALING WITH PRE-/POST-RELEASE DEFECTS

 Pre-release
 Defect prevention, detection, removal
 Several approaches including development methodologies, programming
techniques, debugging tools, testing, beta tests, etc.
 DEALING WITH PRE-/POST-RELEASE DEFECTS
 After release
 Defect containment (activities aim at minimizing the negative impact of these
remaining faults)
 Dormant defects (The faults left in the finished software products are often
called “dormant defects”, which may stay dormant for some time, but have the
potential of causing problems to customers and users of the products)
 Known /unknown defects
 Bug reports
 Cost factor
GRAPHICAL DEPICTION OF THE CLASSIFICATION
SCHEME

 Forming a series of barriers represented by dotted broken lines. Each barrier removes
or blocks defect sources, or prevents undesirable consequences.
 QA AS BARRIERS
 Dealing with errors, faults, and failures
 Removing or blocking defect sources
 Preventing undesirable consequences
 A CLASSIFICATION SCHEME
 QA activities as attempting to prevent, eliminate, reduce, or contain various
specific problems associated with different aspects of defects
 QA alternatives classify these into three generic categories
 DEFECT PREVENTION
 Defect prevention activities assume that there are known error sources or
missing /incorrect actions that result in fault injections
 If human misconceptions are the error sources, education, and training can
help us remove these error sources
 If imprecise designs and implementations that deviate from product
specifications or design intentions are the causes for faults, formal methods
can help us prevent such deviations
 If non-conformance to selected processes or standards is the problem
that leads to fault injections, then process conformance or standard
enforcement can help use prevent the injection of related faults
 If certain tools or technologies can reduce fault injections under
similar environments, they should be adopted
 FORMAL METHODS
 Formal methods provide a way to eliminate certain error sources and
verify the absence of related faults
 Include formal specification and formal verification
 In Formal specification, an unambiguous set of product specifications is
given so that customer requirements, as well as environmental constraints
and design intentions, are correctly reflected, thus reducing the chances
of accidental fault injections
 Formal verification checks the conformance of software design or code
against these formal specifications, thus ensuring that the software is
fault-free with respect to its formal specifications
 OTHER DEFECT PREVENTION TECHNIQUES
 Software methodologies or technologies can also help reduce the chances
of fault injections. Many of the problems with low-quality “fat software”
could be addressed by disciplined methodologies and return to essentials
for high-quality “lean software” (Wirth, 1995)
 Lean programming is a paradigm for the development and
production of computer programs that are based on the principles
of optimizing efficiency and minimizing waste
 Similarly, the use of the information-hiding principle (Parnas, 1972) can
help reduce the complexity of program interfaces and interactions among
different components, thus reducing the possibility of related problems.
 OTHER DEFECT PREVENTION TECHNIQUES
 A better-managed process can also eliminate many systematic problems
 For example, not having a defined process or not following it for system configuration
management may lead to inconsistencies or interface problems among different
software components.
 Ensuring appropriate process definition and conformance helps eliminate some
such error sources. Enforcement of selected standards for certain types of products
and development activities also reduces fault injections
 Specific software tools can also help reduce the chances of fault injections
 For example, a syntax-directed editor that automatically balances out each open
parenthesis, “{”, with a close parenthesis, “}”, can help reduce syntactical problems in
programs written in the C language
 DEFECT REDUCTION

 Inspection
 Testing
 Other techniques
 DEFECT REDUCTION

 Software inspections are critical examinations of software artifacts by

human inspectors aimed at discovering and fixing faults in the software
systems. Inspection is a well-known QA alternative familiar to most
experienced software quality professionals.
 The earliest and most influential work in software inspection is Fagan
inspection (Fagan, 1976)
 Various other variations have been proposed and used to effectively
conduct an inspection under different environments
 INSPECTION
 Inspections are critical reading and analysis of software code or other
software artifacts, such as designs, product specifications, test plans, etc.
 Inspections are typically conducted by multiple human inspectors,
through some coordination process. Multiple inspection phases or
sessions might be used
 Faults are detected directly in inspection by human inspectors,
either during their individual inspections or various types of group
sessions
 Identified faults need to be removed as a result of the inspection
process, and their removal also needs to be verified
 INSPECTION

 The inspection processes vary, but typically include some planning

and follow-up activities in addition to the core inspection activity.
 The formality and structure of inspections may vary, from very
informal reviews and walkthroughs to formal inspection
 Informal reviews:
 Self-conducted reviews/walkthroughs
 Independent reviews (and their coordination)
 Formal inspections:
 Fagan inspection
 Gilb inspection
 INSPECTION
 Inspection is most commonly applied to code, but it could also be applied
to requirement specifications, designs, test plans and test cases, user
manuals, and other documents or software artifacts
 Inspection can be used throughout the development process, particularly
early in the software development before anything can be tested
 Consequently, inspection can be an effective and economical QA
alternative because of the much-increased cost of fixing late defects as
compared to fixing early ones
 TESTING

 Testing is one of the most important parts of QA and the most

commonly performed QA activity.
 Testing involves the execution of software and the observation of the
program behaviour or outcome
 If a failure is observed, the execution record is then analyzed to locate and
fix the fault(s) that caused the failure
 WHEN CAN A SPECIFIC TESTING ACTIVITY BE
PERFORMED AND RELATED FAULTS BE DETECTED

 Testing is an execution-based QA activity, a prerequisite to actual

testing is the existence of the implemented software units, components, or
system to be tested, although preparation for testing can be carried out in
earlier phases of software development
 Testing can be divided into various sub-phases starting from the coding phase
up to post-release product support, including: unit testing, integration testing,
system testing, acceptance testing, beta testing, etc.
 The observation of failures can be associated with these individual sub-
phases, and the identification and removal of related faults can be associated
with corresponding individual units, components, or the complete system
 WHAT TO TEST, AND WHAT KIND OF FAULTS ARE
FOUND?
 External specifications (black-box)/functional
 When black-box testing is performed, failures related to specific external
functions can be observed, leading to corresponding faults being
detected and removed. The emphasis is on reducing the chances of
encountering functional problems by target customers.
 Internal implementation (white/clear-box)/structural
 When white-box testing is performed, failures related to internal
implementations can be observed, leading to corresponding faults being
detected and removed. The emphasis is on reducing internal faults so
that there is less chance for failures later on no matter what kind of
application environment the software is subjected to.
 WHEN, OR AT WHAT DEFECT LEVEL, TO STOP
TESTING?

 When to stop testing:

 Coverage information (higher coverage information means the higher quality
or lower levels of defects)
– Checklists are used to make containing major functions and usage scenarios are tested
before product release
 Product reliability goals can be used as a more objective criterion to
stop testing.
– Use of this criterion requires the testing to be performed under an environment that
resembles actual usage by target customers so that realistic reliability assessment can be
obtained, resulting in the so-called usage-based statistical testing
 OTHER TECHNIQUES FOR DEFECT REDUCTION

 Timing and performance analysis for real-time systems

 Accident analysis and reconstruction using software fault trees and event
trees for safety critical systems, can locate and remove related defects
 Risk identification
 DEFECT CONTAINMENT

 Due to large size and high complexity of most software systems, the
defect reduction activities can only reduce the number of faults to a fairly
low level, but not completely eliminate them
 For software systems where failure impact is substantial, such as many
real-time control software sub-systems used in medical, nuclear,
transportation, and other embedded systems, this low defect level and
failure risk may still be inadequate
 DEFECT CONTAINMENT
 Few faults may be triggered under rare conditions or unusual dynamic
scenarios, making it unrealistic to attempt to generate the huge number of
test cases to cover all these conditions or to perform exhaustive inspection
based on all possible scenarios.
 Break the causal relation between these faults and the resulting failures,
thus “tolerating” these faults
 Or contain the failures by reducing the resulting damage
SAFETY ASSURANCE AND FAULT CONTAINMENT

 For safety critical systems, the primary concern is our ability to prevent
accidents from happening, where an accident is a failure with a severe
consequence.
 Even low failure probabilities for software are not tolerable in such systems
if these failures may still likely lead to accidents.
 Specific techniques for safety critical systems based on analysis of hazards
 Hazard: a precondition to accident
SAFETY ASSURANCE AND FAULT CONTAINMENT
 Hazard elimination through simplification, decoupling, elimination of
specific human errors, and reduction of hazardous materials or conditions
 Hazard reduction through design for controllability (for example, automatic
pressure release in boilers), use of locking devices (for example, hardware
software interlocks), redundancy
SAFETY ASSURANCE AND FAULT CONTAINMENT
 Hazard control through reducing exposure, isolation, and containment (for
example, barriers between the system and the environment), and fail-safe
design (fail in a safe state without causing further damages).
 Damage control through escape routes, safe abandonment of products,
materials, and devices for limiting physical damages to equipment's or
people. These techniques reduce the severity of accidents, thus limiting
the damage caused by these accidents and related software failures
 DIFFERENT TYPES OF PEER REVIEWS
 Code review is systematic examination (often as peer review) of computer
source code
 Pair programming is a type of code review where two persons develop
code together at the same workstation
 Inspection Technical review is a is a very formal type of peer review where
the reviewers are following a well-defined process to find defects