JOURNAL OF COMPUTER SCIENCE AND ENGINEERING, VOLUME 13, ISSUE 2, JUNE 2012

Software Quality Framework

Malik F. Saleh

Abstract— In this paper we present a general software quality model, providing the possibility to describe different concepts
related to quality. We show that our quality model is able to integrate the various concepts found in quality standards and
different quality models. Furthermore, we provide different quality views related to software quality, enabling consistency and
continuity of quality-related information. The most influential factor for the developers of software is the customer perception.
We connect the developer with the customer to derive a common interpretation for quality. This paper introduces a model for
software quality by connecting and integrating the different views of software quality. In addition, it connects the customer view
with the developer view of software quality and it treats software as a product.

Index Terms— Software quality, quality framework, software engineering, software assurance.

——————————  ——————————

1 INTRODUCTION

P roducing software with high quality is a challenging

task. Many software projects fail before they are
completed while other projects fail after deployment.
applied to any software product by tailoring it to a specific
purpose [5].

If you are developing software then quality is often on In this paper we present a general software quality
your mind. But what does software quality really mean? model, while describing very different concepts related to
When users encounter errors in software applications, the quality. We show that our quality model can integrate
general public of users blames the errors on the system various characteristics found in quality standards and
making no distinction between hardware, software or different quality models. Furthermore, we show that this
users input. Are developers to be blamed for all aspect of quality model is able to describe the interrelations of dis-
software quality? ciplines, like requirements engineering and software test,
to software quality. With this quality model, we provide
Assuring high quality of software is crucial. The dis- different quality views common in concepts related to
ciplines of software engineering are all aiming to produce software quality, enabling consistency and continuity of
software that assures quality [1]. Although quality is cru- quality-related information.
cial, it a highly complex topic which required the discip-
lines of software engineering to develop their own quality
assurance approaches. Many quality metrics have been
2 BACKGROUND AND PROBLEM
proposed. Various metrics can be proposed to evaluate STATEMENT
the way the quality is affected [2] and many are being Different approaches have been developed to assure
used such as the number of lines of code (LOC). In fact, software quality. A good definition of software quality
many companies have built their own quality frameworks must measure quality in a meaningful way. This measure
based on product metrics. [3]. Quality standards were should consider all stages of the software product such as
created such as ISO standards. Software metrics have software development, distribution, manufacturing, the
been proved to reflect software quality, and thus have user experience, and maintenance. The literature points
been widely used in software quality evaluation methods out many of the problems that software quality faces:

•
[2].
The customer plays an important role in software quali- There is no comprehensive framework to de-
ty. Many definitions of software quality have been pub- scribe all different concepts relating to software

•
lished. In general, most definitions include the phrase cus- quality in a common way [1].
tomer satisfaction [4, 5]. According to ISO 8402, quality is There does not exist a rigorous method for mea-
the total characteristics of an entity that bear on its ability suring customer perception of product quality
to satisfy stated and implied needs. ISO/IEC 9126 defines [4]. Surveys of customer opinion cannot be con-
a quality model that comprises six characteristics of soft- sidered as measurements, since they are based on
ware product quality. These characteristics are: Functio- non-objective assignments that vary according to
nality, Reliability, Usability, Efficiency, Maintainability, user judgments. A measure, however, should be
and Portability. This quality model is generic and can be an empirical assignment of a number to an entity

•
to characterize its specific attribute.
————————————————
• M.F. Saleh is with Prince Mohammd Bin Fahd University, P.O Box 1664
There are many requirements that affect the qual-
Al Khobar 31952 Saudi Arabia. ity of software. For example operation require-
ment measure the software functionalities with-
out the necessary non-functional requirements.
Therefore, the quality characteristics are treated
2

as technical issues related to the design and test- The developer view of software quality is different
ing of the software product [6]. While the prod- from the customer view. For example the customer inter-
uct revision requirement is affected by how the prets the quality of operation as meeting the requirement
software will be changed. It is determined by while the developers are using different factors. The de-
how easily bugs can be found and corrected and veloper view of quality in the software is affected by
the efforts needed to modify the operational many factors. This model stresses on three primary ones:
software. Finally, software portability is deter- 1. The code - It is measured by its correctness and
mined by how easily the software can be used on reliability
different platforms and it is determined by its 2. The data - It is measured by the application inte-

•
reusability factor. grity
Quality in software projects is discussed in the 3. Maintainability – it has different measures the
context of tolerated errors in software and the simplest is the Mean Time To Change
meanings of "safety-critical" and "mission-
critical". In addition, software quality is dis- 3.2.1 Correctness and Reliability
cussed in the context of software contribution to Correctness and Reliability determine how well a sys-
the larger functionality and the quality of prod- tem fulfils the customer’s overall objectives; measuring
ucts and services [7]. correctness involves keeping track of reported defects and
using a simple quantitative approach. The defects per
These problems are rather unconnected and not inte- thousand lines of code (KLOC) can be used by the devel-
grated. They lack a common foundation and they lack a opers to track the number of lines affected by defects
common quality measure, which leads to loss of quality. when making correction to the code. The percentage of
Finally, there is no comprehensive framework to describe the affected lines to the total lines of code is the measured
all the different views of concepts relating to software to be used. However, this measure is almost impossible to
quality. interpret. It must be complemented by the measure of
functional point analysis.
3 SOFTWARE QUALITY FRAMEWORK
Errors that may appear to be difficult are found to be
This paper introduces a model for software quality by simple when dissected into their components and placed
connecting and integrating the different views of software into their appropriate categories. Function Point Analysis
quality. This model connects the customer view with the (FPA) is a method to break systems into smaller compo-
developer view of software quality and it treats software nents, so they can be better understood and analyzed [8].
as a product. The software product view reviews the cha- Developers need to combine both the KLOC and FPA to
racteristics of a product that bear on its ability to satisfy analyze the correctness of the code. FPA is used to pro-
stated and implied needs. duce functional decomposition of software and it can be
used to size software applications. Combining both
This model is also a framework that describes all the measures will derive a more accurate measure than just
different concepts relating to quality in a common way using the KLOC alone. Correcting errors also involves
measured by qualitative scale that can be understood and using both measures. For example, an application that
interpreted in a common way. Therefore the most in- consists of one million lines of code is broken down into
fluential factor for the developers is the customer percep- twenty function points. The average for each function
tion. This model connects the developer with the cus- point is 100 thousands line of code. If developers record-
tomer to derive a common interpretation for quality. ed errors that affected 1000 lines per error in five function
points then the measure is as follow:
3.1 Developers View
Table 1
Validation and verification are two independent methods Measuring Correctness using FPA and LOC
used together for checking that a software product meets No. LOC Per No. LOC Error
the requirements and that it fulfills its intended purpose. FP Error Affected
Validation checks that the product design satisfies the
1 100,000 1 1000 1%
Function Points

intended usage and verification checks for errors in the

2 100,000 2 2000 2%
software. The primary concern for developers is in the
design and engineering processes involved in producing 3 100,000 3 3000 3%
software. Quality can be measured by the degree of con- 4 100,000 4 4000 4%
formance to predetermined requirement and standards, 5 100,000 5 5000 5%
and deviations from these standards can lead to poor 1 Million 15 Er- 15K 1.5%
quality and low reliability. While validation and verifica- rors
tion are used by the developers to improve the software, Average Errors 3%
the two methods don’t represent a quantifiable quality Therefore, the average correctness for the affected
measurement function points is 97. The overall correctness for the soft-
ware is: 100% - 1.5% = 98.5%
 3

This method has many advantages: Therefore, the overall integrity for the software is: 100% -
a. Finding the strength and weaknesses in the 1.18% = 98.82 Integrity.
application
b. Defining what to re-engineer 3.1.3 Maintainability
c. Estimating the overall effort and schedule of
maintenance It determines how easily bugs can be found and fixed.
3.1.2 Integrity There is no clear way of measuring maintainability, but it
It is measured by considering the proportion of attacks can be seen as the ease with which the software can be
as opposed to valid uses of the software. A quantitative corrected or perfected. A simple measure is mean time to
measure is used by grouping the same types of attacks change (MTTC) [9, 10].
together and assigning a value ranging from 1 (Perfect) to
0 (weakness) to the attack group. Mean time to change requires keeping track of reported
The following measure is used to derive the overall In- bugs. The average of the time it takes to analyze a re-
tegrity of the software: ported bug, design the correct modification, developers
implementing the change, testing the software, and dis-
(1) tributing the change to the customer. This quality factor
affects user perception of the software. The lower the
mean time the more maintainable the software product is
(2) perceived by the user

3.2 User View

The highest quality in software is expected by the users

The software integrity is therefore measured by grouping when they acquire software. The quality is different when
all types of integrity attacks end users develop their software. End-user programming,
a phrase popularized by [11] which is “programming to
(3) achieve the result of a program primarily for personal,
rather than public use.” The important distinction here is
that software itself is not primarily intended for use by a
large number of users with varying needs. For example, a
For example, an application has one thousand entries for teacher may write a grades spreadsheet to track students’
user access in the log file. Of these, one hundred of the en- test scores. In these end-user programming situations, the
tries were attacks, grouped as follow: program is a means to an end that could be used to ac-

•
complish a goal. In contrast to end-user programming,
50 Denial of Service (DOS) attacks. Five suc- professional programming has the goal of producing

•
cessful DOS attacks software for others to use. For example, the moment a
20 password guessing attacks. Two success- novice Web developer moves from designing a web page

•
ful attacks for himself to designing a Web page for others, the nature

•
10 accidental attacks. Two successful attacks of this activity has changed. The moment this shift in in-
20 failed logon. Four accounts were locked tent occurs, the developer must plan and design for a
broader range of possible uses, increasing the importance
Table 2 of design and testing, and the prevalence of potential
Measuring Integrity bugs [12] therefore, increasing the importance of the
No. Type Threat Security Integrity software quality.
attacks

1 Denial of 50 45/50 = (1-0.05) * Users perceive software quality as a fit between their
goals and software's functionality. The better the quality,
Service /1000 = 0.90 (1 – 0.90)
the more likely the user will be satisfied with the soft-
0.05 = 0.095
ware. When the quality is bad, developers must meet user
Active Attacks

2 Password 20 18/20 = (1 – 0.02)

needs or face a diminishing demand for their software
Guessing /1000 = 0.9 * (1 – 0.9) [13]. Therefore, the user sees quality as fitness for purpose
0.02 = 0.098 [7]. Avoiding complexity and keeping software simple,
3 Accidental 10 / 2 / 10 = (1 – 0.01) considerably lessens the implementation risk of software.
attacks 1000 = 0.2 * (1 – 0.2) In some instances, users abandoned the implementation
0.01 = 0.792 of a complex software because the software developers
4 Attempted 20 / 16/20 = (1 – 0.02) were expecting the users to change their business and to
failed lo- 1000 = 0.8 * (1 – 0.8) go with the way the software works [14].
gon 0.02 = 0.196
Total 1.18 Measuring quality of the software is therefore determined
4

by whether the software is accomplishing the goal for the The second view dealt with the customer when using the
user and the number of errors or unexpected behavior of software. The final view treated the software as a product.
the software that the users face during the software nor- This framework handled the perception of quality in a
mal use. Complex systems inherently present unique manner that quantifies quality into measurable indicators.
risks due to tightly linked interdependencies of business This perception of quality was comprehensive and it de-
processes. Users need to have an understanding of com- scribes all different concepts relating to software quality
plexity risks when planning and conducting assurance of in a common way that can be understood by the num-
the reliability of these complex software [15]. Prototyping bers.
can help simplify the complexity of the software. Proto- Customer survey was introduced for measuring customer
typing is a critical component in evolutionary develop- perception of product quality. Surveys of customer opi-
ment [16]. Prototyping is recommended for clarity in un- nion were empirical by assigning a number to an entity to
derstanding system requirements and in planning sys- characterize its specific attribute. This survey measures
tems architecture [17] and thus can help in improving the perception of the customer, which changes over time.
software quality. Therefore, the survey is repeated many times.
This model examined the different requirements that af-
The user experience with the software can be a mo- fect the quality of software including the technical, non-
mentary experience. Hassenzahl (2008) argued that the technical requirement, and the software revision re-
user experience is primarily an evaluation of feelings quirement. The technical requirement and the revision
while interacting with a product or service. By that, the requirement were measured in the developer view while
user experience shifts attention from the product and ma- the non-technical requirements were measured in the
terials to humans and feelings and it changes over time. customer view.
Tracking the user experience, therefore, is very challeng- Finally, this model discussed quality in the context of
ing. Tracking the user attitude toward the software re- stakeholder (the developer and the user) satisfactions
quires evaluating the software while the user interacts instead of the context of tolerated errors in software. In
with the software and this attitude can change from time addition, software quality is discussed in the context of
to time. Therefore, the user experience should not be eva- software contribution to the larger functionality and the
luated based on a single outstanding moments and it quality of products.
should be done continually over different time periods
[16]. APPENDIX A
A simple survey can be used to capture the perceived Customer Satisfaction Survey
user view of the quality of the software. Appendix 1 lists The scales ask you to indicate whether these practices are satisfacto-
some questions that quantify the answers. The items in ry and to show how well this is done.
this survey were adopted from [14]. These items were Use a five-point scale ranging from never (1) to always (5).
also used by [18] and by [19]. Each item was scored using
a five-point scale ranging from never (1) to always (5). All Description Rating
items were presented such that the greater the score, the 1. To what extent does the software quality as-
greater the satisfaction of the particular item surance function have a management report-
ing channel separate from the software devel-
opment project management?
3.3 Product View
The product view sees quality as tied to inherent charac- 2. To what extent is a formal procedure used in
teristics of the product. The Free Dictionary defines the the management review of each software de-
product quality as “the collection of features and charac- velopment prior to making contractual com-
teristics of a product that contribute to its ability to meet
mitments?
given requirements “ [20]. Product quality can be meas-
ured by the value-based view which sees the quality as
3. To what extent is a formal procedure used to
dependent on the amount a customer is willing to pay for
make estimates of software size?
it. To the users, a high-quality product is one that satisfies
their preferences and expectations while meeting their
requirement. End-users satisfaction of the product 4. To what extent is a formal procedure used to
represents wiliness to learn, use, upgrade the product, produce software development schedules?
and when asked to participate in rating the product, a
positive rating is casted. 5. To what extent are formal procedures applied
to estimating software development costs?

4 CONTRIBUTION AND CONCLUSION 6. To what extent are profiles of software size

A well-rounded model was introduced that covered the maintained for each software configuration
three views of software, the first view dealt with the in- item over time?
ception and development of software by the developers.
 5

7. To what extent are statistics on software design 24. To what extent is a mechanism used for verify-
errors gathered? ing that the samples examined by software
quality assurance are truly representative of
8. To what extent are statistics on software code the work performed?
and test errors?
25. To what extent is a mechanism used for man-
9. To what extent do software development first- aging and supporting the introduction of new
line managers sign off on their schedules and technologies?
cost estimates?
26. To what extent is test coverage measured and
10. To what extent is a mechanism used for con- recorded for each phase of functional testing?
trolling changes to software requirements?
27. To what extent are analyses of errors con-
11. To what extent is a mechanism used for con- ducted to determine their process related
trolling changes to the code? (Who can make causes?
changes and under what circumstances?)
28. To what extent is the error data from code re-
12. To what extent is there a software engineering views and tests analyzed to determine the like-
process group function? ly distribution and characteristics of the errors
remaining in the product?
13. To what extent is there a required software
engineering training program for software de- 29. Keep the system simple
veloper?
30. Hide complexity
14. To what extent is a formal training program
required for design and code review leader? 31. Avoid changes

15. To what extent does the software organization 32. Obtain user participation
use a standardized software development
process? 33. Obtain user commitment

16. To what extent does the software organization 34. Providing training to users
use a standardized and documented software
development process on each project? 35. Providing on-going assistance

17. To what extent does senior management have 36. Use a prototype approach for system devel-
a mechanism for the regular review of the sta- opment
tus of software development projects?
37. Tailor system to people’s capabilities
18. To what extent are the action items resulting
from design reviews tracked to closure? 38. Reliable software

19. To what extent are the action items resulting 39. Efficient cost of software operations
from code reviews tracked to closure?
40. Wide range of outputs that can be generated
20. To what extent is a mechanism used for ensur- and queries that can be answered
ing compliance with the software engineering
standards? 41. Overall responsive software to users

21. To what extent are internal software design 42. Ability to meet project goals
reviews conducted?
43. Adherence to schedule
22. To what extent is a mechanism used for con-
trolling changes to software design? 44. Adherence to budget

23. To what extent are software code reviews con-

ducted?
6

ACKNOWLEDGMENT Malik F. Saleh is a professot at Prince Mohamamd Bin Fahd University in

the Kingdom of Saudi Arabia with over 15 years of teaching experience.
The author wishes to thank Prince Mohammad Bin Fahd Dr. Saleh Chaired the MIS department and he has a leadership role in the
Univerisyt for the support provided in publishing this maintenance of academic standards and in the development of educational
policy and of curriculum areas within the University.
research.

REFERENCES
[1] Lochmann, K. and A. Goeb, A unifying model for software quality, in Pro-
ceedings of the 8th international workshop on Software quality. 2011, ACM:
Szeged, Hungary. p. 3-10
[2] Stroggylos, K. and D. Spinellis, Refactoring--Does It Improve
Software Quality?, in Proceedings of the 5th International Workshop
on Software Quality. 2007, IEEE Computer Society. p. 10
[3] Basili, V.R., L. Briand, and W.L. Melo, A validation of object-
oriented design metrics as quality indicators. IEEE Trans. Softw.
Eng., 1996. 22(10): p. 751-761
[4] Xenos, M. and D. Christodoulakis, Measuring perceived software
quality. Information and Software Technology, 1997. 39(6): p.
417-424
[5] Jung, H.-W., S.-G. Kim, and C.-S. Chung, Measuring Software
Product Quality: A Survey of ISO/IEC 9126. IEEE SOFTWARE,
2004. 21(5): p. 88-92
[6] Chung, L., et al., Non-Functional Requirements in Software Engi-
neering. 2000, Boston: Kluwer Academic Publishers
[7] Kitchenham, B. and S.L. Pfleeger, Software Quality: The elusive
Target. IEEE Software, 1996. 13(1): p. 12-21
[8] Longstreet, D. Function Points Analysis Training Course. 2012;
Available from: http://www.softwaremetrics.com
[9] Bhatt, P., G. Shroff, and A.K. Misra, Dynamics of Software Mainten-
ance. ACM SIGSOFT Software Engineering Notes, 2004. 29(5)
[10] Akingbehin, K. and B. Maxim. A Three-Layer Model for Software
Engineering Metrics. in Proceedings of the Seventh ACIS Interna-
tional Conference on Software Engineering, Artificial Intelligence,
Networking, and Parallel/Distributed Computing. 2006. Las Vegas,
Nevada, USA
[11] Nardi, B.A., A Small Matter of Programming: Perspectives on End
User Computing. . The MIT Press, 1993
[12] Ko, A.J., et al., The State of the Art in End-User Software Engineering.
ACM Computing Surveys, 2011. 43(3)
[13] Robinson, W.N., Seeking Quality through User-Goal Monitoring.
IEEE Softw., 2009. 26(5): p. 58-65
[14] Subramanian, G.H., J.J. Jiang, and G. Klein, Software quality and IS
project performance improvements from software development process ma-
turity and IS implementation strategies Journal of Systems and Software,
2007. 80(4): p. 616-627
[15] Wright, S. and A. Wright, Information system assurance for enterprise
resource planning systems: unique risk considerations. Journal of Informa-
tion Systems, 2002. 16(1): p. 99–113
[16] Pressman, R., Software Engineering: A Practitioner's Approach Vol. 6.
2004: McGraw-Hill
[17] Boehm, B.W. and P.N. Papaccio, Understanding and controlling software
costs. IEEE Transactions on Software Engineering, 1988. 14(10): p.
1462–1477
[18] Robey, D., L.A. Smith, and L.R. Vijayasarathy, Perceptions of conflict
and success in information system development projects. Journal of Man-
agement Information Systems, 1993. 10(1): p. 123–139
[19] Henderson, J.C. and A.S. Lee, Managing I/S design teams: a control
theories perspective. . Management Science, 1992. 36(6): p. 757–777
[20] TheFreeDictionary. Product quality TheFreeDictionary 2012; Availa-
ble from:
http://encyclopedia2.thefreedictionary.com/Product+quality