International Journal of Digital Information and Wireless Communications (IJDIWC) 4(1): 95-102

The Society of Digital Information and Wireless Communications, 2014 (ISSN: 2225-658X)

Developing Usable Software Product Using Usability Risk Assessment Model

Jayaletchumi T. Sambantha Moorthy, Suhaimi bin Ibrahim, Mohd Naz’ri Mahrin

Universiti Teknologi of Malaysia (UTM)
Kuala Lumpur, Malaysia.
[email protected], [email protected], [email protected]

ABSTRACT Usability has been recognized as an important

quality factor in various quality model [6][7][8]
Usability is an important factor in ensuring and has always been present even in the very first
development of quality and usable software model of software quality known as McCall
product. Ignorance and unawareness about the Factors Criteria Metrics (FCM) proposed by
concept of usability and failure to address usability McCall in [9].
during software development process has led to
usability problems in software product. Many Study has shown that ignorance and unawareness
efforts has been suggested in literature to overcome on most referenced usability standard, ISO 9241-
usability problem in software products but current
11 among industries [10] and governmental
practices faces challenges in reducing these
usability problems. Alternatively, the concept of organizations [11] in Malaysia, has created
risk management can be used to control usability various usability problems in software products
problems even though these problems cannot be [12]. Existence of usability problems are also
eliminated totally. The concept of risk management reflected when usage level of software products is
is important to deal with usability problem before it low [13]. A survey by Malaysian Administrative
occurs. Unfortunately, there is still lack of proper Modernization and Management Planning Unit
definition of usability risk and a proper model to (MAMPU) in 2011 had reported that the usage of
identify, analyze and prioritize potential usability online services in government is only 40% of the
risk during Software Development Lifecycle overall online services provided [14] and this
(SDLC). This paper presents comprehensive study result shows that users are exposed to higher
on the need for Usability Risk Assessment Model to
usability problem in using online services[15] [16]
reduce usability problems in software products.
[17]. Example of reported usability problems are
such as high number of broken links and slower
KEYWORDS
accessibility speed [49]; less usability activities in
Usability; Usability Problem; Usability Risk; Risk product designs; limited skills and knowledge on
Management; Software Risk Assessment. usability among the designers and management;
unawareness on various activities of usability
engineering life-cycle and inappropriately used
1 MOTIVATION
usability methods [16]. Generally, existence of
Software quality has emerged as important aspects usability problem could lead to failure of a system.
of software development process, as it might result
in serious consequences such as financial loss and To increase the rate of successful software project,
reputation loss [1]. Quality improvement after it is crucial to reduce usability problems. Usability
development of software is not recommended professionals have expressed that by integrating
because it only increases the cost and is almost usability closely with software development
remaking the product [2]. Poor software quality process [18], usability problems can be reduced.
has been identified as the key factor for success or There are many efforts to integrate a formal
failure of a software product [3][4][5]. usability process, standard, techniques and
practices into software development process to

95
 International Journal of Digital Information and Wireless Communications (IJDIWC) 4(1): 95-102
The Society of Digital Information and Wireless Communications, 2014 (ISSN: 2225-658X)

improve the interaction and quality of the systems potential usability risks at earlier phases of
[19][20]. However, software developers face development process. If development team
difficulties when new usability practices are continues to develop software products without
introduced [21]. Some usability approaches are identifying, analyzing and prioritizing usability
only integrated in requirement and design phase risks, the chances of producing less usable
[22]. In fact, their practical implementation is software products are higher.
largely missing. Usability practices as well are not
part of requirement engineering [19], so In relation to this, development of Usability Risk
developers are often given an incomplete, Assessment Model could guide development team
confusing, and sometimes contradictory on identifying, analysing and prioritizing potential
requirement. As a result, many development teams usability risks that could arise during SDLC in
are facing difficulties in avoiding and minimizing order to produce more usable software products
usability problems. with less usability problems.

Besides this, various usability evaluation activities This paper is structured as follows. Section 2
such as inspection, empirical testing, and metrics reviews existing studies on usability, usability
for usability standards in computing has been problem, usability risk, software risk assessment
integrated into software development process to processes and software risk assessment models. A
measure and improve usability of software [23]. conceptual view of proposed Usability Risk
However, it only evaluates a completed system Assessment Model is illustrated in Section 3.
and does not intervene at earlier stages of Section 4 explains on contribution and Section 5
development process [24]. An International includes conclusion and future work.
standard, ISO 13407 [25] also had proposed a
framework for integration of usability in all phases 2 RELATED WORKS
of software development process. Even so, current
usability engineering practices had failed to reduce Existence of usability problems creates many
usability problems in software products. quality problems in software and contributes to its
failure. Risk management approach can be used to
Alternatively, studies have shown that problems in overcome usability problem in software by
software products can also be controlled using identifying and analysing usability risk earlier in
Software Risk Management methods, even though the SDLC.
these problems cannot be eliminated totally
[26][27]. Using Software Risk Management, 2.1 Usability
problems in software products are dealt before it
occurs, so that risk control activities can be Usability is considered as one of the significant
planned and implemented as needed. factor of software product quality. Even, five most
mentioned quality models, McCall [9], Boehm [6],
Tim Altom [28] had suggested that usability FURPS [30], Dromey [8] and ISO 1926/2001 [7]
should be portrayed as risk management and not had defined usability as significant factor in
as an abstraction, to avoid later arguments, improving product acceptability and reliability,
encourages discussion and allows management to increasing user satisfaction, and it is also
see the benefit of usability in software products. financially beneficial to companies [31]. Usability
Study has also shown that usability problems can is the best factor that balances between technical
be considered as a significant usability risk factor and human aspects of a software product which is
[13]. However, there are great ignorance on important in defining quality [32]. Hence, it’s
managing usability risk compared to managing important to ensure usability characteristics are
other risks such as technology risk, market risk integrated well in software products to ensure its
and money risk [32]. Furthermore, there is little success.
effort in identifying, analyzing and prioritizing

96
 International Journal of Digital Information and Wireless Communications (IJDIWC) 4(1): 95-102
The Society of Digital Information and Wireless Communications, 2014 (ISSN: 2225-658X)

2.2 Usability Problem for development team in developing a usable

software product.
Many studies had revealed that usability problems
causes quality problems in software product 2.4 Software Risk Assessment Processes
[33][34]. Potential problems lying in interface
design, operating process or product structure also For several decades, risk management has been a
contributes to usability problems which resulted in popular approach in non-software domain and it
lower effectiveness, efficiency and difficulty of has been adopted in software domain since the last
use for end users [35]. Websites as well faces huge few years [43].
usability problems which has been identified as
factors for lower usage level [15][36][37][38]. According to Boehm [44], two main processes in
Discussion above shows that it is extremely software risk management are software risk
important to reduce frequency of usability assessment and software risk control. The first
problem to improve user-experience and quality of process in software risk management is software
software product [39]. risk assessment with activities of risk
identification, risk analysis and risk prioritization.
2.3 Usability Risk The second process in software risk assessment is
risk control and it involves risk management
The term usability risk was first introduced in the planning, risk monitoring and risk resolution.
context of e-commerce and World Wide Web
services [29]. Generally, usability risk is a chosen Usability Risk Assessment Model focuses only on
action or activity that leads to a loss or an adverse the software risk assessment activities namely,
outcome which could impact the usability of a Identification. Analyze and Prioritization.
software product. It is related to user acceptance Furthermore, since usability risk has not been
and meeting user’s requirement. Usability risk identified as an entity previously, it is more
contributes to negative user experiences which appropriate to conduct the process of risk
lead to software product failure [40]. However, assessment before risk control can be performed.
search for the term ‘usability risk’ in the literature
has shown that this term is not widely used and 2.5 Software Risk Assessment Models
some studies related to mobile application has
mentioned this term in their studies [40][41][42]. Some popular published software risk assessment
Most of the current researches are associated with models are Software Risk Assessment Model
usability problem and not usability risk. (SRAM) [45], Software Risk Assessment and
Estimation Model (SRAEM) [46], Risk
Usability problem can also be perceived as a Identification, Mitigation and Avoidance Model
risk factor in producing usable software product for Handling Software Risk (RIMAM) [47], and
since it is a threat to an optimal user experience Software Risk Assessment and Evaluation Process
[13]. User experience, in general includes many using Model Based Approach (SRAEP) [48].
aspect such as human factors, design, ergonomics,
Human Computer Interaction (HCI), accessibility, All four assessment model explains the
marketing as well as usability. Hence, usability perspective of software development and not
problem is recognized as a risk factor for software product itself. These models are lacks in
producing usable software product. practical guidelines. There is also a lack in
standard framework or model for assessing and
Since studies on usability risk is still lacking, we managing software product risk based on
believe that it is important to define usability risk attributes of quality, particularly usability. This
as a single entity, either in form of checklist, creates extensive need for Usability Risk
model or in other appropriate form, as a guidance Assessment Model.

97
 International Journal of Digital Information and Wireless Communications (IJDIWC) 4(1): 95-102
The Society of Digital Information and Wireless Communications, 2014 (ISSN: 2225-658X)

3 CONCEPTUAL MODEL clearly stated the initial intention of the software

development projects. Potential usability risks are
As the needs for a Usability Risk Assessment selected from the list of usability risks stored in
Model is clearly described in the previous section, Usability Risk Repository. Identification of
a proposed model of usability risk assessment is potential usability risks is done with two steps:
suggested in this section. Figure 1 illustrates the a) Identification of usability risks mentioned in
proposed model of usability risk assessment. the literature. Search in several digital
databases such as Institute of Electrical and
Electronics Engineers (IEEE), Association for
User (Risk Manager/ Project) Computing Machinery (ACM), Springer,
MANAGER) Science Direct, Web of Science and Google
Tender/RFP Scholar will be conducted to identify existing
documents
Usability Risk usability risks in government applications and
Usability Risk Repository systems.
Identification
Usability Risk analysis b) Identification of usability risk factors from
usability attributes defined in a quality model
Likelihood Impact
Assessment Assessment which will be used to construct questionnaire
to identify usability risks from the industry.
Usability Risk Exposure Level
In 2012, Aman Kumar, Arvind and Hardeep
[2] suggested that factors affecting the quality
Usability Risk of software can be identified from attributes
Classification defined in software quality models. This study
considers attributes of usability as factors in
producing usable software products. These
Usability Risk usability attributes are subjected to risks that a
Prioritization software product might have troubles with.
Scenario which affects the ability to achieve
these attributes is considered as potential
usability risk during software development
Requirement Design
Implementation
Testing Evaluation process. Based on usability attributes defined
Analysis And
Maintenance in quality model, namely Efficiency,
Satisfaction, Comprehensibility and Safety, a
Software Development Lifecycle (SDLC) list of usability risk factors will be derived and
used as constructs to design and develop a
Figure 1: Conceptual view of Usability Risk questionnaire which will be distributed to the
Assessment Model industry to collect information on usability
risk, from the perspective of industry.
In Figure 1, important elements and activities in
the proposed Usability Risk Assessment Model Potential usability risks identified from a) and b)
are shown and its description is as given below: will be reconciled so that risks with the same
meaning are considered only once. The output will
(i) Usability Risk Identification be a list of potential usability risk that could occur
during SDLC.
This model facilitates user in identifying potential
usability risk in a software development projects
by analysing initial tender, Request for Proposal
(RFP) document or any other documents that

98
 International Journal of Digital Information and Wireless Communications (IJDIWC) 4(1): 95-102
The Society of Digital Information and Wireless Communications, 2014 (ISSN: 2225-658X)

(ii) Usability Risk Analysis Initially, each expert will be given a brief
explanation on the goal of the session and how the
Once potential usability risks have been identified, outcome of the session contributes to the
these risks will be analyze using risk analysis development Usability Risk Assessment Model.
technique to determine the likelihood and impact Then, the first round questionnaire will be
of each usability risks towards phases in SDLC. distributed to the experts to collect information on
likelihood of each usability risk for analysis. Since
Likelihood and impact of each usability risks are Delphi method is done in an iterative manner, the
identified using the Delphi method. Classification information from first round of survey will be fed
of usability risk based on SDLC phases is also back to the experts for comments and as a basis
determined from this method. for the second round which is to determine the
impact of usability risk to SDLC phases. This
Delphi is a method to iteratively gather process is repeated in third round until common
experiences, knowledge, and opinions of the understandings between experts are achieved. All
experts on an issue or development process under questionnaires in this survey is developed using
study by using normally by interview or survey five point Likert scale.
[50]. It uses a number of questionnaire rounds,
feedback from responses of experts and gives The advantage of this method is that all
opportunity to the experts to modify their knowledge from experts is evaluated and
responses and anonymity of responses are assured, commented by other experts, producing
collectively agreed information on likelihood,
In this study, using Delphi method, a three- impact, and classification and mitigation plan of
iteration questionnaire survey will be conducted usability risks on SDLC phases.
with experts. The first round gives opportunity for
the experts to suggest new potential usability risks Then, the exposure level for each potential
and determine the likelihood of each usability usability risk is calculated based on likelihood and
risks. The second round determines the impact of impact of each usability risks. Each usability risk
each usability risks to SDLC phases. The third will be classified according to phases in SDLC
round involves the activities of experts to classify and correlation analysis will be done to determine
and suggest mitigation plan for each usability relationship between potential usability risk and
risks. phases in SDLC.

Experts are selected based on their experiences in (iii) Usability Risk Prioritization
software development and/or risk management.
Experts with more than 10 years of experience in Prioritization of each potential usability risk is
dealing with software development projects or done by sorting respective risk exposure level in
who are seek advice from when dealing with descending order to know the impact of usability
software development projects, are chosen. There risk in each phase in SDLC. Higher the risk
is no rules on the number of experts that is exposure, more priority and attention should be
required in a Delphi survey since the decision on given to that particular usability risk because it has
the number of experts needed is taken after more impact in creating less usable software
consideration of factors such as time and expenses product.
[51]. The representation of experts is assessed
with the qualities and experiences acquired than Prioritization is important because it gives insight
its numbers [53]. Study has suggest that a suitable of critical and noncritical usability risk based on
expert size from 4 to 3000 [50][51][52] but for the SDLC phases, among usability risks and in overall
context of this research, five experts will be software product development.
chosen due to time factor.

99
 International Journal of Digital Information and Wireless Communications (IJDIWC) 4(1): 95-102
The Society of Digital Information and Wireless Communications, 2014 (ISSN: 2225-658X)

5 CONTRIBUTIONS of risk management. The idea of this concept is

that if usability risks can be identified and
The proposed model the proposed model could managed well, the overall chances of reducing risk
help project managers, quality managers, risk of failure and producing usable software product
management team and software development team could be increased. Since studies on usability risk
in identifying and describing potential usability is still lacking, this will be a good approach for
risk that impacts and effects SDLC phases and development team to be responsive towards
delivery of software development objectives. By potential usability risk that need be managed
eliminating these risks, customer’s expectations on during software development process.
quality can be met while saving the costs and time.
In future, suitable methods to implement each
Theoretically, this model contributes to the area of activities suggested in the proposed Usability Risk
knowledge in Software Engineering Body of Assessment Model will be determine in detail.
Knowledge (SWEBOK) and Project Management Current practices in developing a risk assessment
Body of Knowledge (PMBOK) as the concept of model will also be investigated for this purpose.
usability risk assessment has not been explored
before and would contribute to development of 7 REFERENCES
more usable software products.
1. O. K. Hussain, T. S. Dillon, F. K. Hussain, and E. J.
A list of important elements and activities in Chang, “Understanding Risk and Its Importance in
Business Activities,” in Risk Assessment and
software risk assessment models is identified and Management in the Networked Economy, Springer Berlin
used as a basis for development of Usability Risk Heidelberg, 1-29, 2013.
Assessment Model. Gaps and lacking in current 2. A. K. Sharma, A. Kalia, and H. Singh, “An Analysis of
software risk assessment models are identified too Optimum Software Quality Factors,” IOSR Journal of
and will be overcome in the proposed model. Engineering, vol 2(4), 663-669, 2012.
3. J. Verner, J. Sampson, and N. Cerpa, “What factors lead
to software project failure?,”in IEEE Second
Usability Risk Assessment Model also consists of International Conference on Research Challenges in
two other sub contributions: Information Science, 2008, RCIS 2008, 71-80, 2008.
4. R. Kaur and J. Sengupta, “Software Process Models and
Analysis on Failure of Software Development Projects,”
 List of potential usability risk which influences
arXiv preprint arXiv:1306.1068, 2013.
SDLC to increase understanding and awareness 5. K. Schwaber, and J. Sutherland, Software in 30 Days:
on existence of potential usability risks among How Agile Managers Beat the Odds, Delight Their
development team. Customers, and Leave Competitors in the Dust. Wiley,
2012.
B. W. Boehm, J. R. Brown, H. Kaspar, M. Lipow, G. J.
 Exposure level of each usability risk which 6.
MacLeod, and M. J. Merrit, “Characteristics of software
describes the importance to overcome usability quality,” ,1, North-Holland Publishing Company, 1978.
risks. 7. Software Engineering-Product Quality-Part 1: Quality
Model, ISO/IEC 9126 -1, 2001.
Usability risk is also incorporated to SDLC phases 8. R. G. Dromey, “A model for software product quality,”
IEEE Transactions on Software Engineering,” 21(2),
to show the impact of usability risk during 146-162, 1995.
software development and this incorporation 9. J. A. McCall, P. K. Richards, and G. F. Walters, “Factors
serves as guidance for development team to in software quality,” General Electric, National
develop more usable software product. Technical Information Service, 1977.
10. I. Hussein, M.Mahmud, and A. W. Yeo, “HCI practices
in Malaysia: A reflection of ICT professionals'
6 CONCLUSION AND FUTURE RESEARCH perspective,” 2010 IEEE International Symposium in
Information Technology (ITSim), 3, 1549-1554, 2010.
This paper proposes a concept in ensuring high 11. A. W. Yeo, P. C. Chiu, T. Y. Lim, P. P. Tan, T. Lim, and
usability in software products by using the concept I. Hussein, “Usability in Malaysia,” In Global Usability,
Springer London, 211-222, 2011.

100
 International Journal of Digital Information and Wireless Communications (IJDIWC) 4(1): 95-102
The Society of Digital Information and Wireless Communications, 2014 (ISSN: 2225-658X)

12. S. K. Dubey, and A. Rana, “Analytical Roadmap to 27. Z. Jun-guang, and X. Zhen-chao, “Method study of
Usability Definitions and Decompositions,” in software project risk management,” in International
International Journal of Engineering Science and Conference on Computer Application and System
Technology, 2(9), 4723-4729, 2010. Modeling (ICCASM), IEEE, 8, 8-9, October 2010.
13. A. Inversini, L. Cantoni, and D. Bolchini, “Connecting 28. T. Altom, “Usability as risk management,” Interactions,
Usages with Usability Analysis through the User 14(2), 16-17, 2007.
Experience Risk Assessment Model: A Case Study in the 29. A. B. Platt, “The usability risk,” in Proceedings of the
Tourism Domain,” In Design, User Experience, and 18th IEEE Symposium on Reliable Distributed Systems,
Usability. Theory, Methods, Tools and Practice,” 283- IEEE, 396-400, Proceedings of the First Asia-Pacific
293, Springer Berlin Heidelberg, 2011. Conference on Quality Software, IEEE, 297-305,1999.
14. Modernization and Management Planning Unit 30. R. B. Grady, and D. L. Caswell, “Software metrics:
(MAMPU), “Kajian Baseline Perkhidmatan E-Kaunter establishing a company-wide program,” 1987.
& Paperless Government,” 2011. 31. M. Simes-Marques and I. Nunes, “Usability of
15. W. A. M. Isa, M. R. Suhami, N. I. Safie, and S. S. interfaces,” 155-170, 2012.
Semsudin, “Assessing the usability and accessibility of 32. M. Sangeetha, C. Arumugam, P.G. Sapna, and S. K. A
Malaysia e-government website,” American Journal of KM, “Cooperative approach to ensure software product
Economics and Business Administration,3(1),40-46, quality,” International Journal of Computer Science, 8.
2011. vol. 8 (4), 598, 2011.
16. T. Jokela, “Performance rather than capability problems. 33. S. U. Farooq and S. M. K. Quadri, “Quality Practices in
insights from assessments of usability engineering Open Source Software Development Affecting Quality
processes,” in Product Focused Software Process Dimensions,” Trends in Information Management, 7(2),
Improvement,115-127, Springer Berlin Heidelberg, 2005. 2012.
17. A. Seffah, M. Donyaee, R. B. Kline, and H. K. Padda, 34. L. Hua, and Y. Gong, “Usability Evaluation of a
“Usability measurement and metrics: A consolidated Voluntary Patient Safety Reporting System:
model,” Software Quality Journal,14(2), 159-178, 2006. Understanding the Difference between Predicted and
18. G. Lindgaard, “Usability testing and system evaluation: Observed Time Values by Retrospective Think-Aloud
A guide for designing useful computer systems,” Protocols,” in Human-Computer Interaction.
London: Chapman & Hall,. 221-246, 1994. Applications and Services, Springer Berlin Heidelberg,
19. J. Heiskari, M. Kauppinen, M. Runonen, and T. 94-100, 2013.
Mannisto, “Bridging the gap between usability and 35. L. Liang, X. Deng, and Y. Wang, “Usability
requirements engineering,” in 17th IEEE International Measurement Using a Fuzzy Simulation Approach,” in
Requirements Engineering Conference, 303-308, 2009. International Conference on Computer Modeling and
20. H. Fischer, “Integrating usability engineering in the Simulation, 2009. ICCMS'09, IEEE, 88-92, 2009.
software development lifecycle based on international 36. S. Lee and J. E. Cho, “Usability Evaluation of Korean e-
standards,” in Proceedings of the 4th ACM SIGCHI Government Portal,” in Universal Access in Human-
symposium on Engineering Interactive Computing Computer Interaction, Applications and Services,
Systems, ACM, 321-324, 2012. Springer Berlin Heidelberg, 64-72, 2007.
21. P. Carlshamre, and M. Rantzer, “Business: 37. M. Abdul Aziz, W. A. R. Wan Mohd Isa, and N. Nordin,
Dissemination of Usability: Failure of a Success Story,” “Assessing the accessibility and usability of Malaysia
Interactions, vol 8(1), 31-41, 2001. Higher Education Website,” in 2010 International
22. A. Seffah, R. Djouab, and H. Antunes, “Comparing and Conference on User Science and Engineering(i-USEr),
reconciling usability-centered and use case-driven IEEE, 203-208, 2010.
requirements engineering processes,” in Australian 38. E. N. Asiimwe and N. Lim, “Usability of government
Computer Science Communications, IEEE Computer websites in Uganda,” Electronic Journal of
Society, 23, 5, 132-139, 2001. e-Government, vol 8(1), 1-12, 2010.
23. S. R. Humayoun, “Incorporating usability evaluation in 39. J. Sauro. (2010, September). How common are usability
software development environments, ” KI-Künstliche problems [Online]. Available:
Intelligenz, vol 26(2), 197-200, 2012. http://www.measuringusability.com/problem-
24. C. Lallemand, “Toward a closer integration of usability frequency.php
in software development: a study of usability inputs in a 40. P. Ketola, “Integrating usability with concurrent
model-driven engineering process,” in Proceedings of the engineering in mobile phone development,” Academic
3rd ACM SIGCHI symposium on Engineering interactive Dissertation, Department of Computer and Information
computing systems, ACM, 299-302, 2011. Sciences, University of Tampere, Finland 2002.
25. Human-centred design processes for interactive systems, 41. C. Jones, “Assessment and Control of Software Risk,”
ISO 13407, 1999. Yourdon Press619, 1994.
26. R. S. MBA and J. C. Arias, “Review of Risk 42. B. Jin, S. Ko, J. Mun, and Y. G. Ji, “A study for usability
Management Methods,” Volume 4-Number 1-January risk level in physical user interface of mobile phone,” in
2011-Semiannual Publication, vol 4(1), 59, 2011.

101
 International Journal of Digital Information and Wireless Communications (IJDIWC) 4(1): 95-102
The Society of Digital Information and Wireless Communications, 2014 (ISSN: 2225-658X)

Usability and Internationalization. HCI and Culture,

Springer Berlin Heidelberg, 327-335, 2007.
43. C. M. Subhas, K. Vinod, and K. Uma, “Different
techniques for risk management in software engineering:
a review,” ASAC 2006 Eric Sprott School of Business
Carleton University Banff, Alberta, 2006.
44. B. W. Boehm, “Software risk management: principles
and practices,” Software, IEEE, 8(1), 32-41, 1991.
45. S. W. Foo and A.Muruganantham, “Software risk
assessment model,” in Proceedings of the 2000 IEEE
International Conference on Management of Innovation
and Technology, 2000. ICMIT, IEEE,.2, 536-544, 2000.
46. D. Gupta, and M. Sadiq, “Software risk assessment and
estimation model,” in International Conference on
Computer Science and Information Technology, 2008.
ICCSIT'08, IEEE, 963-967, 2008.
47. B. Shahzad and A. S. Al-Mudimigh, “Risk Identification,
Mitigation and Avoidance Model for Handling Software
Risk,” in 2010 Second International Conference on
Computational Intelligence, Communication Systems and
Networks (CICSyN), IEEE, 191-196, 2010.
48. M. Sadiq, M. K. I. Rahmani, M. W. Ahmad and S. Jung,
“Software risk assessment and evaluation process
(SRAEP) using model based approach,” in 2010
International Conference on Networking and Information
Technology (ICNIT), IEEE, 171-177, 2010.
49. Isa, W.A.R.W.M., Suhami, M.R., Safie, N.I., &
Semsudin, S.S. Assessing the Usability and Accessibility
of Malaysia E-Government Website. American Journal
of Economics and Business Administration, 3(1), 40-46,
2011.
50. Lilja, Kari K, Laakso, Kimmo, & Palomki, J. Using the
Delphi method. Paper presented at the 2011 Proceedings
of PICMET'11: Technology Management in the Energy
Smart World (PICMET), 2011.
51. Thangaratinam, Velayutham, & Selvambigai, Shakila..
Health technology assessment in maternal and perinatal
medicine: delphi survey of practice, systematic reviews of
evidence and meta analyses. University of Birmingham,
2011.
52. Schmidt, Roy, Lyytinen, Kalle, Keil, Mark, & Cule, Paul.
Identifying software project risks: an international Delphi
study. Journal of management information systems,
17(4), 5-36, 2001.
53. Powell, Catherine. The Delphi technique: myths and
realities. Journal of advanced nursing, 41(4), 376-382,
2003.

102