Software Quality Assurance and

Management
(SQA)
Quality Assurance and
Management
Lectures # 1 & 2
 Quality
• Synonyms
– Excellence, superiority, class, eminence, value,
worth
• Antonym
– Inferiority
• Everyone claims to manufacture / develop /
sell / market “good” quality products /
services
 Software Quality
• Quality as it relates to software
• Difficult to define
• Good software quality characteristics can be
identified
• Bad or undesirable characteristics can also be
identified
• Relates to all aspects of software (requirements /
design / code / tests / documents / training)
 Software Quality Definitions - 1
• Low levels of defects when deployed,
ideally approaching zero
• High reliability, or the capability of running
without crashes or strange results
• A majority of clients with high user-
satisfaction when surveyed
 Software Quality Definitions - 2
• A structure that can minimize “bad fixes” or
insertion of new defects during repairs
• Effective customer support when problems
do occur
• Rapid repairs for defects, especially for
high-severity defects
 Beyond Absence of Defects
• Sense of beauty
• Sense of fitness for purpose
• Sense of elegance that goes beyond the
simple absence of overt flaws
• Has well-formed requirements
• Robust
 Why Software Quality? - 1
• Reduces time to market for new products
• Enhances market share compared to direct
competitors
• Minimizes “scrap and rework” expenses
• Attracts and keeps “top-gun” personnel
• Minimizes the risk of serious litigation
 Why Software Quality? - 2
• Minimizes the risk of serious operating
failures and delays
• Minimizes the risk of bankruptcy or
business failures, which may be attributed
directly to poor quality or poor software
quality
 Achieving Software Quality
• “For a software application to achieve high
quality levels, it is necessary to begin
upstream and ensure that intermediate
deliverables and work products are also of
high quality levels. This means that the
entire process of software development
must itself be focused on quality”
– Capers Jones
 Six Software Defect Origins
• Requirements
• Design
• Source code
• User manuals/training material
• “Bad fixes” or mistakes made during
repairs
• Flawed test cases used by the application
 Six Root Causes of Poor
Software Quality
• Inadequate training of managers and staff
• Inadequate defect and cost measurement
• Excessive schedule pressure
• Insufficient defect removal
• High complexity levels
• Ambiguous and creeping requirements and
design (feature race & gimmicks)
Six Software Defect Elimination
Strategies
• Effective defect prevention
• High levels of defect removal efficiency
• Accurate defect prediction before the
project begins
• Accurate defect tracking during
development
• Useful quality measurements
• Ensuring high levels of user-satisfaction
 Software Defects in Six
Application Size Ranges
• 1 function point or 125 C statements
• 10 function points or 1,250 C statements
• 100 function points or 12,500 C statements
• 1,000 function points or 125,000 C statements
• 10,000 function points or 1,250,000 C statements
• 100,000 function points or 12,500,000 C
statements
 Software Quality in Six
Sub- Industries
• Systems software that controls physical devices
• Information systems that companies build for their
own use
• Outsource or contract software built for clients
• Commercial software built by vendors for lease or
sale
• Military software built following various military
standards
• End-user software built for private use by computer
literate workers or managers
 Quality Attributes
• Quality attributes set is a way to represent
customer quality requirements
• Ask your current and prospective customers
about their definition of quality
• Develop a quality assurance program based
on the requirements of your customers
 Product-Specific Attributes
• Ease of use
• Documentation
• Defect tolerance how much the product is reluctant
• Defect frequency
• Defect impact
• Packaging
• Price versus reliability
• Performance
Organization-Specific Attributes
• Service and support
• Internal processes
 Achieving High Levels of
Software Quality - 1
• Enterprise-wide quality programs
• Quality awareness and training methods
• Quality standards and guidelines
• Quality analysis methods
• Quality measurement methods
• Defect prevention methods
• Non-test defect removal methods
 Achieving High Levels of
Software Quality - 2
• Testing methods
• User-satisfaction methods
• Post-release quality control
Quality Assurance Organizations
• No quality assurance 60%
• Token quality assurance 20%
• Passive quality assurance 15%
• Active quality assurance 5%
– i.e Nasa
 Best in Class Quality Results - 1
• Quality measurements
• Defect prevention
• Defect and quality estimation automation
• Defect tracking automation
• Complexity analysis tools
• Test coverage analysis tools
• Formal inspections
Best in Class Quality Results - 2
• Formal testing by test specialists
• Formal quality assurance group
• Executive and managerial understanding of
quality
 Two Components of
Software Quality Improvement
• Reductions in total defect potentials using
methods of defect prevention
• Improvements in cumulative removal
efficiency levels
 Categories of Software Defects
• Errors of commission: something wrong is
done
• Errors of omission: something left out by
accident
• Errors of clarity and ambiguity: different
interpretations
• Errors of speed and capacity
 Software Defect Prevention
Req. Design Code Document Perf.
Defects Defects Defects Defects Defects

JAD Excellent Good N/A Fair Poor

Prototypes Excellent Excellent Fair N/A Excellent

Structured Fair Good Excellent Fair Fair

Methods

CASE Tools Fair Good Fair Fair Fair

Blueprints / Excellent Excellent Excellent Excellent Good

Reusable Code

QFD Good Excellent Fair Poor Good

 Joint Application Development
• Users are active participants in the
requirements sessions
• Both client and MIS sides agree on
uninterrupted time commitments
• JAD-based requirements are more
“complete” versus the traditional
requirements
 Quality Function Deployment
• QFD is a very formal, structured group
activity involving clients and development
personnel
• During QFD sessions, user’s quality criteria
are exhaustively enumerated and defined.
This is followed by the product’s quality
response to these requirements
 Quality Laggards - 1
• No software quality measurement program
of any kind
• No usage of formal design and code
inspections
• No knowledge of the concepts of defect
potentials and defect removal efficiency
• Either no quality assurance group or a
group that is severely understaffed
 Quality Laggards - 2
• No trained testing specialists available
• Few or no automated quality assurance
tools
• No quality and reliability estimation
capability
• Minimal or no software project
management tools available
 Quality Laggards - 3
• No automated risk assessment or
avoidance capability
• From a low of one to a high of perhaps four
distinct testing stages
• No test library or test-case management
tools available
• No complexity analysis tools utilized
 Quality Laggards - 4
• Defect potentials averaging more than 6
defects per function point  not yet !
• Defect removal efficiency averaging less
than 80%
• Executive and managerial indifference (and
ignorance) of quality matters
Project Management Approaches
and High Software Quality - 1
• Use of automated project estimation
methods
• Use of automated project planning methods
• Use of early and automated estimates of
software defect potentials
• Use of early and automated estimates of
software defect removal efficiency
• Formal risk-analysis
Project Management Approaches
and High Software Quality - 2
• Provision of adequate time for pre-test
inspections
• Historical quality data from similar projects
available
• Milestone tracking automated and thorough
• Defect tracking automated and thorough
• Management focus concentrated on
achieving excellent results
Project Management Approaches
and Poor Software Quality
• Exact opposite of the project management
approaches correlating with high software
quality
 SQA Group’s Activities - 1
• Preparation of an SQA plan for a project
– Evaluations to be performed
– Audits and reviews to be performed
– Standards that are applicable to the project
– Procedures for error reporting and tracking
– Documents to be produced by the SQA group
– Amount of feedback provided to the software
project team
 SQA Group’s Activities - 2
• Participation in the development of the
project’s software process description
• Review of software engineering activities to
verify compliance with the defined software
process
• Audit of designed software work products
to verify compliance with those defined as
part of the software process
 SQA Group’s Activities - 3
• Ensure that deviations in software work and
work products are documented and handled
according to a documented procedure
• Record any noncompliance and reports to
senior management
Costs of Software Quality - 1

• Defects prevention costs

• User satisfaction optimization costs
• Data quality defect prevention costs
• Data quality defect removal costs
• Quality awareness/training costs
• Non-test defect removal costs
• Testing defect removal costs
 Costs of Software Quality - 2
• Post-release customer support costs
• Litigation and damage award costs
• Quality savings from reduced scrap/rework
• Quality savings from reduced user
downtime
• Quality value from reduced time-to-market
intervals
 Costs of Software Quality - 3
• Quality value from enhanced
competitiveness
• Quality value from enhanced employee
morale
• Quality return on investment
 Cost Per Defect Hazards
• Test cases must be created whether there
are many bugs, only a few, or none at all
• Test cases must be run whether there are
any bugs or not, although tests will be run
more often for buggy software
• During testing, programmers are waiting
(and getting paid) for bugs to be found
 Data Quality - 1
• Extremely important to understand issues of
data quality
• Data results in (useful | useless) information
• Usually, governments are holders of largest
data banks i.e NADRA (are they consistent?)
• Companies are increasingly using data to
their advantage over competitors
 Data Quality - 2
• Data warehouses present a unique challenge
to keep data consistent
– Data can’t be lost.
– So ,Inactive data is placed in warehouses so
that query results run fast & database should
not become bulky.
• Another problem is the interpretation of
data
 Defect Discovery
• Defects are discovered by developers &
testers (usually) before release
• Defects are discovered by customers and
users (usually) after release
• Defects discovered after release can be
embarrassing for the development team
 Defect Discovery by Customers
• Rule 1: Defect discovery is directly related
to the number of users
• Rule 2: Defect discovery is inversely related
to the number of defects
Quality Measurement Questions
• What should be measured for quality?
• How often should quality measurement be
taken and reported?
Quality Measurement Categories
• Measurement of defects or bugs in software
– 100% of software projects
• Measurement of user-satisfaction levels
– Only for software projects where clients can be
queried and actually use the software
consciously
 Software Defect Quality
Measurements - 1
• Defect volumes (by product, by time period,
by geographic region)
• Defect severity levels
• Special categories (invalid defects(defect
exists but you are interpreting it wrong),
duplicates, un-duplicatable problems)
• Defect origins (i.e., requirements, design,
code, documents, or bad fixes)
 Software Defect Quality
Measurements - 2
• Defect discovery points (i.e., inspections,
tests, customer reports, etc.)
• Defect removal efficiency levels
• Normalized data (i.e., defects per function
point or per KLOC)
• Causative factors (i.e., complexity, creeping
requirements, etc.)
 Software Defect Quality
Measurements - 3
• Defect repair speeds or intervals from the
first report to the release of the fix
 Software User-Satisfaction
Quality Measurements - 1
• User perception of quality and reliability
• User perception of features in the software
product
• User perception of ease of learning
• User perception of ease of use
• User perception of customer support
• User perception of speed of defect repairs
 Software User-Satisfaction
Quality Measurements - 2
• User perception of speed of adding new
features
• User perception of virtues(good things) of
competitive products
• User perception of the value versus the cost
of the package
 Who Measures User-
Satisfaction?
• Marketing or sales organization of the
software company
• User associations
• Software magazines
• Direct competitors
• User groups on the internet, etc.
• Third-party survey groups
Gathering User-Satisfaction Data
• Focus groups of customers
• Formal usability laboratories
• External beta tests
• Requests from user associations for
improvements in usability
• Imitation(repitition,copy) of usability
features of competitive or similar products
by other vendors
 Barriers to Software Quality
Measurement
• Lack of understanding of need to measure
quality
• Often technical staff shies away from
getting their work measured
• Historically, “lines of code” or LOC and
“cost per defect” metrics have been used,
which are a poor way of measuring
software quality
 Defect Prevention and Removal
• Both defect prevention and removal
techniques are used by the “best-in-the-
class” companies
• Defect prevention is very difficult to
understand, study, and quantify
• Both non-test and testing defect removal
techniques must be applied
 Defect Prevention Approaches
• Formal requirements analysis, i.e., JAD
• Formal risk-analysis early in the
development
• Prototyping
• Structured or formal specification methods
• Structured programming methods
• Certified reusable design and code
components
 Non-Test Defect Removal
Methods
• Requirement inspections
• Design inspections
• Code inspections
• Test-case inspections
• Test plan reviews
• User documentation editing or reviews
Testing Defect Removal Methods
• Unit test by individual programmers
• New function testing
• Regression testing
• Performance testing
• Integration testing
• System testing
• Field test (external beta test)
 Defect Removal
• Not all defects are equal when it comes to
removal
• Requirements errors, design problems, and
“bad fixes” are particularly difficult
 Software Defect Origins &
Defect Removal Effectiveness
Req. Design Code Document Perf.
Defects Defects Defects Defects Defects

Reviews /
Inspections Fair Excellent Excellent Good Fair

Prototypes Good Fair Fair N/A Good

Testing (all
forms) Poor Poor Good Fair Excellent

Correctness
Proofs Poor Poor Good Fair Poor
 Defect Removal Efficiency
• Accumulation of defect statistics for errors
found prior to delivery, and then for a
predetermined period after deployment
(usually one year)
• US averages: 85%
• Best projects in best US companies: 99%
 Defect Repair Rates - 1
• Reported data on defect repair rates is not
consistent
• Defect repair rates usually decline as
cyclomatic and essential complexity
increases
• Defect repair rates increase with experience
in application, language, inspections,
structured design and coding methods
 Defect Repair Rates - 2
• Defect repair rates are higher for
maintenance specialists than others during
maintenance phase
• For coding errors, they correlate with
comment density. IBM’s study concluded
that 18% comment density is ideal
• It also found, flow charts had no impact, but
good error messages had great impact
 Defect Seeding
• Willful insertion of errors into a software
deliverable prior to a review, inspection, or
testing activity
• It is the quickest way of determining defect
removal efficiency
• May be considered unpleasant by many
 Defect Severity Levels
• Most software defect tracking systems
include a multi-tier “severity level” scale
• For example,
– Severity level 1: total failure of application
– Severity level 2: failure of major function(s)
– Severity level 3: minor problem
– Severity level 4: cosmetic problem(look & feel
or interface problem)
 Defect Tracking
• It is important to use an accurate and
automated defect tracking system
• Defect tracking tools
– Tracking defects by severity level and by origin
– Routing defects to appropriate repair facility
– Keeping records of duplicate defects
– Invalid defects
– Repair information against defects
 Software Quality Personnel
• Unfortunately are under-paid
• Usually are let go first in times of crisis
• “Top-gun” SQA personnel and managers
with proven track record are in high
demand from companies that have active
QA programs
Economics of Software Quality -
1
• High quality software applications have
shorter development schedules than low
quality applications because they do not get
hung up in integration and testing due to
excessive defect levels
• That is No rework is required if the quality
is properly maintained
Economics of Software Quality -
2
• High quality software applications have
lower development and maintenance costs
than low quality applications. This is
because the cumulative costs of finding and
fixing bugs is often the major cost driver for
software projects
Economics of Software Quality -
3
• High quality software applications have
better reliability levels and longer mean
times to failure than low quality
applications
• High quality commercial software packages
have larger market shares than low quality
commercial software packages
Economics of Software Quality -
4
• High quality software achieves better user-
satisfaction ratings than low quality
software
• High quality software projects score better
on employee morale surveys than do low
quality software projects
Economics of Software Quality -
5
• High quality software produced under contract or
an outsource agreement has a much lower
probability of ending up in court for breach of
contract or malpractice litigation than low quality
software
• High quality software benefits or augments the
performance levels of users, while poor quality
tends to degrade worker performance
Economics of Software Quality -
6
• Poor quality software can trigger truly
massive unplanned expense levels. Denver
airport example
 Error-Prone Modules
• Those modules, which contain majority of
reported defects
• By isolating and repairing error-prone
modules, it is possible to make significant
quality improvements for comparatively
low costs
• Continuous analysis of factors that lead to
error-prone modules is required
 Why Error-Prone Modules?
• Excessive schedule pressure on the
programmers
• Poor training or lack of experience in
structured methods
• Rapidly creeping requirements which
trigger late changes
• High complexity levels with cyclomatic
ranges greater than 15
 “Good Enough” Software
Quality - 1
• Rather than striving for zero-defect levels or
striving to exceed in 99% in defect removal
efficiency, it is better to ship software with
some defects still present in order to speed
up or shorten time to market intervals
• Developed by the fact that major
commercial software companies have latent
software bugs in their released products
 “Good Enough” Software
Quality - 2
• Major commercial software companies have
cumulative defect removal efficiency of
95% (and 99% on their best projects)
• This concept is very hazardous for ordinary
companies, which usually have their defect
removal efficiency level between 80%-85%
• Quality will be decrease for these
companies
 Object-Oriented Quality Levels
• OO technology is being adopted world-
wide with a claim that it produces better
quality software products
• OO technology has a steep learning curve,
and as a result it may be difficult to achieve
high quality software
• More data needs to be reported
• UML may play a significant role
Orthogonal Defect Reporting - 1
• The traditional way of dealing with software
defects is to simply aggregate(combine) the
total volumes of bug reports, sometimes
augmented(increased) by severity levels and
assertions(confident statement) as to whether
defects originated in requirements, design,
code, user documents, or as “bad fixes”
Orthogonal Defect Reporting - 2
• In the orthogonal defect classification
(ODC) method, defects are identified using
the following criteria
– Detection method (design/code inspection, or any of a variety of testing
steps)

– Symptom (system completely down, performance downgraded, or questionable data

integrity)

– Type (interface problems, algorithm errors, missing function, or documentation error)

– Trigger (start-up / heavy utilization / termination of application, or installation)
– Source (version of the projects using normal configuration control identification)
 Outsourcing and Software
Quality
• Outsourcing in software industry is done in
a variety of ways
• Every situation introduces new challenges
for development of high quality software
• Software quality metrics must be mentioned
in the outsourcing contract
 Quality Estimating Tools - 1
• Estimating defect potentials for bugs in five
categories (requirements, design, coding,
documentation, and bad fixes)
• Estimating defect severity levels into four
categories, ranging from 1 (total or
catastrophic failure) to severity 4 (minor or
cosmetic problem)
 Quality Estimating Tools - 2
• Estimating the defect removal efficiency
levels of various kinds of design reviews,
inspections, and a dozen kinds of testing
against each kind and severity of defects
• Estimating the number and severity of latent
defects present in a software application
when it is delivered to users
 Quality Estimating Tools - 3
• Estimating the number of user-reported
defects on an annual basis for up to 20 years
• Estimating the reliability of software at
various intervals using mean-time to failure
(MTTF) and/or mean-time between failures
(MTBF) metrics
 Quality Estimating Tools - 4
• Estimating the “stabilization period” or
number of calendar months of production
before users can execute the application
without encountering severe errors.
• Estimating the efforts and costs devoted to
various kinds of quality and defect removal
efforts such as inspections, test-case
preparation, defect removal, etc.
 Quality Estimating Tools - 5
• Estimating the number of test cases and test
runs for all testing stages
• Estimating maintenance costs for up to 20
years for fixing bugs (also for additions)
• Estimating special kinds of defect reports
including duplicates and invalid reports
which trigger investigative costs but no
repair costs
 Litigation and Quality
• Relevant factors for software quality
– Correctness, reliability, integrity, usability,
maintainability, testability, understandability
• Irrelevant factors for software quality
– Efficiency, flexibility, portability, reusability,
interoperability, security
• It is important to narrow down the scope of
quality definition similar to hardware
warranties
 Schedule Pressure and Quality
• Healthy pressure
– Motivates and keeps morale of the personnel
high
• Excessive pressure
– Has serious negative impact on the morale of
personnel
– Can lead to low quality software
 Quality Process Metrics
• Defect arrival rate
• Test effectiveness
• Defects by phase
• Defect removal effectiveness
• Defect backlog
• Backlog management index
• Fix response time
• Percent delinquent fixes
• Defective fixes
 Product Metrics
• Defect density
• Defects by severity
• Mean time between failures
• Customer-reported problems
• Customer satisfaction
 Function Point Metric - 1
• It was developed at IBM and reported to
public in 1979
• It is a way of determining the size of a
software application by enumerating and
adjusting five visible aspects that are of
significance to both users and developers
 Function Point Metric - 2
• Inputs that enter the application (i.e., Input
screens, forms, commands, etc.)
• Outputs that leave the application (i.e.,
Output screens, reports, etc.)
• Inquiries that can be made to the application
(i.e., Queries for information)
• Logical files maintained by the application
(i.e., Tables, text files, etc.)
 Function Point Metric - 3
• Interfaces between the application and
others (i.e., shared data, messages, etc.)
 Function Point Metric - 4
• Once the raw total of these five factors has
been enumerated, then an additional set of
14 influential factors are evaluated for
impact using a scale that runs from 0 (no
impact) to 5 (major impact)
 Quality Related Terms
• See Jones’ book pages: 329-336
 Points to Take Away
• Quality is free, but it needs investment and
commitment
• Without quality products, sooner or later,
you will be out of business
• Think software, think quality
 References
• Software Quality: Analysis and Guidelines
for Success by Capers Jones
• Customer-Oriented Software Quality
Assurance by Frank Ginac
• A Practitioner’s Approach to Software
Engineering by Roger Pressman