Computer Science and Information Technologies

Vol. 5, No. 1, March 2024, pp. 7~12

ISSN: 2722-3221, DOI: 10.11591/csit.v5i1.pp7-12  7

The impact of usability in information technology projects

Freddy Richard Putra Hulu, Teguh Raharjo, Tiarma Simanungkalit

Faculty of Computer Science, Universitas Indonesia, Jakarta, Indonesia

Article Info ABSTRACT

Article history: Achieving success in information system and technology (IS/IT) projects is a
complex and multifaceted endeavour that has proven difficult. The literature
Received Oct 23, 2023 is replete with project failures, but identifying the critical success factors
Revised Nov 24, 2023 contributing to favourable outcomes remains challenging. The triad of Time-
Accepted Jan 05, 2024 Cost-Quality is widely accepted as key to achieving project success. While
time and cost can be quantified and measured, quality is a more complex
construct that requires different metrics and measurement approaches.
Keywords: Utilizing the PRISMA Methodology, this study initiated a comprehensive
search across literature databases and identified 142 relevant articles
IT project pertaining to the specified keywords. A subset of ten articles was deemed
Project success suitable for further examination through rigorous screening and eligibility
Success factor assessments. Notably, a primary finding indicates that despite recognizing
Systematic literature review usability as a critical element, there is a tendency to neglect usability
Usability enhancements due to time and resource constraints. Regarding the influence
of usability on project success, the active involvement of end-users emerges
as a pivotal factor. Moreover, fostering the enhancement of Human-
Computer Interaction (HCI) knowledge within the development team is
essential. Failure to provide good usability can lead to project failure,
undermining user satisfaction and adoption of the technology.
This is an open access article under the CC BY-SA license.

Corresponding Author:
Freddy Richard Putra Hulu
Faculty of Computer Science, Universitas Indonesia
10430 Salemba, Jakarta, Indonesia
E-mail: [email protected]

1. INTRODUCTION
Extensive research has been conducted on information technology (IT) projects’ success and failure
factors, revealing the inherent risk involved in such endeavors [1]–[3]. It is well established that achieving
success in IT projects is a complex and challenging process that requires careful planning, execution, and
management. Despite the significant efforts invested in IT project management, the failure rates still need to
be lowered. Therefore, understanding the critical factors that impact project success, such as usability, is
essential for achieving favorable outcomes.
However, it is important to note that successful information system and technology (IS/IT) projects
are not a rare phenomenon [4], [5]. While failures in IS/IT projects have been widely studied, critical success
factors that contribute to favorable outcomes have also been extensively researched. It is imperative to
identify and prioritize these success factors to mitigate risks and ensure the success of IS/IT projects.
Within the triad of cost, time, and quality, quantifying quality is often the most challenging, as it
encompasses a range of factors that vary across projects. While cost and time can be more easily measured,
the measurement of quality is multifaceted and can include aspects such as maintainability and usability. The
present study will focus on usability as a critical quality factor in the success of IS/IT projects. Despite being
an integral aspect of computing since its inception, the contribution of usability to project success has yet to

Journal homepage: http://iaesprime.com/index.php/csit

2  ISSN: 2722-3221

be fully explored. This paper aims to shed light on this important but often overlooked aspect of IT project
management.
The following research questions will be answered throughout this systematic literature review:
− How is Usability positioned in IS/IT Project?
− How to improve the quality of usability factor in the IT Project Success?
This paper is structured as follows: section 1 provides an introduction to the topic, while section 2
discusses the background and related works (meta-studies) in the field of usability and IS/IT projects. In
section 3, the research questions and methodology will be presented in detail. Sections 4 through 6 will cover
the results for each of the research questions. Finally, section 7 will conclude the paper

2. LITERATURE REVIEW
This section will explore literature relevant to the management of IS/IT projects concerning
usability. We will also examine how usability is an essential aspect of system acceptability. Furthermore, we
will review literature focusing on the success criteria for IS/IT projects, including project management
success, system quality, and project product success.

2.1. Background
Firstly, research about usability and usability evaluation, including the user experience evaluation, is
easily found in academic databases. The methodology of usability evaluation also developed into many kinds
of user interaction, such as e-government [6], mobile application [7], chatbot [8], and virtual reality [9]. In
addition to that, Systematic Literature Review about the usability and user experience also is a common
article published in computer-science-related journals and publications.
On the other hand, IS/IT projects have also been the subject of extensive research, which includes
investigations into success factors, failure rates, and systematic literature reviews. Such research is readily
accessible, and the findings can be leveraged to enhance project management practices. Although the
researchers have not yet come to one conclusion about what is the most critical success or failure factors, it’s
worth noting that likely most parts of the IT project have been researched.
However, the research about the roles of usability in IS/IT projects has yet to be researched to a
considerable extent. This is an important area of research, as usability is a key factor in the success of any
IS/IT project. This research aims to fill this gap by reviewing the literature on usability and IS/IT projects
from the past decade and synthesizing the main findings.

2.2. Usability
Usability is defined in ISO/IEC 25010 as “the degree to which a product or system can be used by
specified users to achieve specified goals with effectiveness, efficiency, and satisfaction in a specified
context of use” [10], [11]. While another definition describes usability as being “easy to learn, useful
(provide necessary functionality), easy to use and pleasant to use” [12]–[14]. Despite the various definitions,
usability is widely acknowledged as being context-dependent. Another scholar emphasizes that usability
needs to be referenced to a particular context [15].
Usability is a component of user experience (UX), which is part of practical acceptability according
to Nielsen’s model [16]. The components of usability are shown in Figure 1. The figure also shows the
position of usability related to the system and practical acceptability.
Measuring usability presents challenges due to its multidimensional nature, influenced by various
factors such as the user’s task, the product itself, and the specific environment [17], [18]. To address this,
researchers often adopt an approach that involves engaging participants to perform tasks using the system,
and then assessing key factors like the time taken to complete tasks, the rate of successful scenario
completions, and the time required for error recovery [19]. This task-oriented evaluation allows for a more
comprehensive understanding of usability and its impact on the overall user experience, making it a valuable
method in usability research.
In the context of project management, usability is considered a non-functional requirement (NFR)
and part of the quality factor [20], which is one of the three common criteria used to measure project success
[21], [22]. Previous research has demonstrated that usability can improve satisfaction, productivity, and
technology acceptance, ultimately contributing to the success of IS/IT projects [8]. Therefore, understanding
and prioritizing usability as a critical quality factor is essential for achieving favorable outcomes in IT
projects.
To strengthen the argument for how usability success contributes to the overall success of an IS/IT
project, prior research has demonstrated that improved usability leads to enhanced user satisfaction, increased
productivity, and better technology acceptance [9]. Furthermore, assessing usability poses unique challenges

Comput Sci Inf Technol, Vol. 5, No. 1, March 2024: 7-12

Comput Sci Inf Technol ISSN: 2722-3221  3

due to its multidimensional nature, which is influenced by various factors specific to the user’s task, the
product, and the environment in which it is used, making it more complex than traditional one-dimensional
product or user characteristics [10], [11]. This nuanced understanding of usability’s impact on the user
experience underscores its crucial role in the success of IS/IT projects and highlights the need for tailored
approaches in evaluating and enhancing usability.
In conclusion, usability is a critical quality factor in IS/IT projects that can impact project success.
Usability is challenging to measure due to its multidimensional nature and its dependence on context.
However, assessing usability can be achieved by asking participants to perform tasks using the system and
measuring factors such as task completion time, success rate, and error recovery time. Prioritizing usability as
an NFR can improve satisfaction, productivity, and technology acceptance, ultimately contributing to the
success of IS/IT projects.

Figure 1. Model of the system acceptability attributes by nielsen (recreated)

2.3. IS/IT project

A project is an endeavor undertaken to create a unique product, service, or other result [23]. This
endeavor has to be temporary. Hence, we could easily distinguish the project from operational activities.
IS/IT Project is not defined far from the project definition as well. Bannerman presents a definition for
“IT Project”, “Discrete and unique activities that serve as vehicles of multidimensional IT-based change”
[24]–[26].
The word “multidimensional” serves well in today’s dynamics of business and other sectors of the
world. Businesses are not only the ones who benefit from IS/IT development but every other organization,
from the government to the private sector, from small organizations to large enterprises. Nowadays, IT
Projects become a vehicle of transformation and growth for the business. The global IT Investment in 2022
was estimated by Gartner to grow and reaches 4,4 trillion dollars. It only grows from previous years amid the
pandemic.
Nevertheless, there is complexity in every IS/IT Project that has been kicked off. This is the one that
makes an IS/IT Project unique. Moreover, one project had a different complexity from another, and this not
only makes it unique but also makes it prone to failure [3], [27]. But as it was mentioned before, the failure
and success of IT Projects have been widely researched in empirical studies and systematic reviews. This
paper is not research on IT Projects and failure but is more focused on the usability in IS/IT Projects.
Although IT project success is important for the overall IT success [28], there are no formal criteria
for project success. However there is a criterion that is stated very well by Iriarte and Bayona [25]. The
criteria are divided into two groups which are project management success and project product success. Each
group has three criteria, and there are system quality criteria that fall into both groups. The list of project
success criteria is shown in Figure 2.
A systematic literature review comprising 39 studies conducted by Iriarte and Bayona identified
several critical success factors in IS/IT projects [25]. These factors include the capability and skills of team
members, user involvement, top management support, consultant knowledge and technical expertise,
organizational maturity and culture, internal communication, project organization structure and
responsibilities, effective planning, and the ability to handle changes and deviations. Lamprou identified
additional critical success factors, including project finance/funding and economics, team competency and
effectiveness, and project managers’ capability and experience [5].
However, research has also shown that inadequate attention to these critical success factors can lead
to project failure. Inadequate management support, lack of technical know-how among team members, and
lack of project managers’ capability and experience have been identified as factors contributing to project

The impact of usability in information technology projects (Freddy Richard Putra Hulu)
4  ISSN: 2722-3221

failure. Other factors identified in the literature include poor supplier selection, data-related technical
problems, and insufficient user involvement and commitment [29]. The causes of project failure can vary
depending on the project lifecycle or phase, geography, and type of measured failure.
Recent research has highlighted the need for customized approaches to understanding project failure
in IS/IT projects. Gupta has suggested the importance of researching project failure prediction under dynamic
scenarios [2], while Money has proposed a forensic analysis framework to comprehensively understand the
causes of project failure in IS/IT projects [30]. These approaches are based on the principle that a
comprehensive understanding of each project is necessary to understand why, where, when, how, and what
causes project failure. Another research shows that IT project failure comes from the neglect of Non-
Functional Requirements in the Development phase [31]. The findings of these studies emphasize the
importance of identifying and addressing critical success factors and potential failure points to improve
project outcomes in IS/IT projects.
According to Figure 2, which outlines the criteria for project success, system quality is an essential
component of both project management success and project product success. System quality is a measure of
users’ perceptions of adaptability, availability, reliability, response time, and usability [32], [33]. Specific
usability characteristics, such as appropriateness, recognizability, learnability, and operability, are also
associated with improved system quality across four factors, including custom software quality,
documentation, time and budget, and new package functionalities [33].
Given these findings, it is evident that usability is a crucial factor in achieving success in IS/IT
projects. By improving system quality, usability can positively impact project management success and
project product success. Therefore, it is crucial to prioritize usability as an essential component of IS/IT
projects to improve the overall success of the project.

Figure 2. Project success criteria by iriarte (recreated)

3. METHOD
This paper employs the preferred reporting items for systematic reviews and meta-analyses
(PRISMA) methodology to conduct a literature review. PRISMA is a widely-used methodology in the
academic world for conducting systematic reviews or meta-analyses on a topic. The PRISMA methodology
comprises four key steps: identification, screening, eligibility, and inclusion [34]. Table 1 outlines the
inclusion and exclusion criteria employed in this study. The straightforward description of the method
depicted in Figure 3. Using this methodology, this paper aims to provide a comprehensive review of the
existing literature on usability and its impact on IS/IT project success.

Table 1. Inclusion and exclusion criteria

Inclusion Criteria Exclusion Criteria
1. Either title, abstract, or keywords related to this paper’s topic 1. Paper in a language other than English
2. Articles about IS/IT Projects and Usability 2. Unpublished articles
3. Published in the range of ten years (2012 – 2022) 3. Duplicates articles
4. Journal articles and conferences proceedings 4. Lecturer Note, review articles, reports
5. Able to answer the RQ 5. Systematic literature review and Meta-analysis
6. The whole paper is available for the authors

Comput Sci Inf Technol, Vol. 5, No. 1, March 2024: 7-12

Comput Sci Inf Technol ISSN: 2722-3221  5

Figure 3. PRISMA methodology flowchart

3.1. Identification
This PRISMA Method was applied to a literature review from five of the academic databases. These
academic databases were chosen since they are both prominent and peer-reviewed. The academic databases
are ACM Digital Library (https://dl.acm.org/), Emerald Insights (https://www.emerald.com/insight/),
IEEExplore (https://ieeexplore.ieee.org/Xplore/home.jsp), ScienceDirect, and Scopus
(https://www.scopus.com/search/form.uri).
In searching for the related studies, these main keywords were used, “Information System Projects”
AND “usability”. Next, the term “Information System Projects” expanded into similar words and terms such
as Information Technology as well as the abbreviation. Thus, the produced keywords are as follows, “IT
Project”, “IT Projects”, “Information Technology Project”, “Information Technology Projects”, “IS Project”,
“IS Projects”, “Information System Project”, “Information System Projects”. On the other hand, the usability
keyword is not modified or added with other related terms such as User Experience or User Interface. These
two other keywords will skew the query results since it has different meaning and academic topics.

3.2. Screening
The screening process is an important step in the PRISMA methodology as it aims to remove
duplicate results from various databases. In some cases, articles may be indexed in two or more databases,
resulting in duplication. To identify and remove these duplicates, researchers must manually read and
compare the title, abstract, and keywords of each article. This ensures that only unique articles are included in
the study and eliminates any potential bias or error resulting from the inclusion of duplicate results.
During the screening process, any literature that cannot be retrieved in full text is excluded. This
step is necessary as it allows researchers to determine the quality, details, and methodology of the articles.
Only literature that can be retrieved in full text can be properly evaluated and, therefore, cannot be included
in the study. By excluding these articles, researchers can ensure that the study is based on a thorough and
reliable analysis of the existing literature.

3.3. Eligibility
The third step in the PRISMA methodology is the eligibility check, which involves reading the
entire article that passed the screening step. During this step, researchers evaluate the article based on the
inclusion criteria, which typically involves assessing whether the article can answer at least one of the
research questions. If an article meets the inclusion criteria, it will be selected and included as a reference for
the research. By conducting a thorough eligibility check, researchers can ensure that the selected articles are
relevant and reliable and can provide valuable insights into the topic being studied.

The impact of usability in information technology projects (Freddy Richard Putra Hulu)
6  ISSN: 2722-3221

3.4. Included
After passing the first three steps, the last step in PRISMA is to determine how the articles act as a
source for this current systematic review. In the PRISMA model in Figure 3, there are two articles that are
included from other sources, whilst both obtained from the same database source as the database searching.
These two articles were selected from dl.acm.org and scopus.com since it has the same literature relation and
connection to this current research. The final results of the method finally included 10 literatures, which
shown in the Table 2 from identification process up to the included.

Table 2. Comparison of number of researches by sources

No. Source Identification Screening Eligibility Included
1 ACM Digital Library 117 137 11 10
2 Emerald 3
3 IEEExplore 7
4 ScienceDirect 15
5 Scopus 20
Total 212

4. RESULTS AND DISCUSSION

In this section, the selected studies will be analyzed to facilitate comparisons between articles. The
analysis will begin by examining the project management methodology, including whether the traditional
waterfall or Agile methodology was used. The next focus will be the product developed in the project, with a
brief analysis of the project results (whether it failed or succeeded) if applicable. However, the most critical
aspect of the analysis will be the positioning of usability in the project, including whether the project
management plan explicitly mentions usability as a key factor. Furthermore, Table 3, which contains the
titles of the articles to be reviewed, will provide a comprehensive overview of the literature sources
considered in this analysis.
Given the crucial role of usability in IS/IT project success, it is essential to understand how it is
positioned in each study. By examining how usability is incorporated into the project management plan,
researchers can gain insights into the best practices for integrating usability into IS/IT projects. This analysis
will contribute to a more comprehensive understanding of how usability impacts project success and provide
valuable insights for future research in this area.

Table 3. Included articles to be reviewed

No. Published Year Article Title Reference:
1 2013 The impact of end-user participation in IT projects on product [35]
usability
2 2014 Lean UX: The Next Generation of User-Centered Agile [36]
Development?
3 2016 Hard lessons learned: Delivering usability in IT projects [37]
4 2018 Adapting Lightweight User-Centered Design with the Scrum-Based [38]
Development Process
5 2018 Configurations of user involvement and participation in relation to [39]
information system project success
6 2018 Delivering Usability in IT Products: Empirical Lessons from the [40]
Field
7 2019 Integrating HCI Perspective into a Mobile Software Development [41]
Team: Strategies and Lessons from the Field
8 2020 IT Project Failures, Causes and Cures [3]
9 2021 Human Factors Methods Applied to a Healthcare Information [42]
Technology Project
10 2021 Information System Project Success Based on User Experience at [43]
the Village Administration Office

To simplify the presentation of the selected articles, they will be summarized in Table 4 based on
the main problem of project management related to usability. The table will also include information on the
type of research conducted to show how the research was carried out. Most of the selected articles are based
on empirical research, with two articles focusing on specific case studies and one article using a qualitative
comparative analysis approach. By presenting the information in a clear and concise format, the results of the
literature review can be easily understood and used to inform future research in this area.

Comput Sci Inf Technol, Vol. 5, No. 1, March 2024: 7-12

Comput Sci Inf Technol ISSN: 2722-3221  7

Table 4. Articles main problem

No. Type of Research Project Management Problems Related to Usability
1 Empirical Research Study Case with 15 IT Project managers about end-user
participation in the final product usability.
Problems identified: Involving user in IT Project would
increase the cost, whilst PM need to undertake decision to
minimize cost.
2 Case Study Agile, which has gained popularity in the past decade, has
yet to reach a similar de facto standard status as the old
waterfall process for usability testing.
Lean UX philosophy is identified by Design thinking, lean
production, and Agile development.
3 Empirical Research Most software vendors manage projects by encountering
numerous issues in providing usability in IT solutions and
successfully completing a project within a given time
frame.
Customers will perceive a low-end usability product as
low quality as a whole.
The risks associated with the failure to provide usability
were assigned to the project manager or the entire
company in charge of the project.
4 Empirical research of two projects There are several things that could be improved in
managing IT Projects in developed countries.
The vendors who can meet the demand for IT projects
lack skills and unrealized benefits, as well as an
inadequate education and training support infrastructure
and the absence of national guidelines and standards.
The project owner failed to take explicit usability
requirements into account in the call for tender documents
5 Qualitative Comparative analysis Even though it has been suggested that UIP is important
in requirement analysis or during testing to influence
project success positively, there is a lack of knowledge
about the appropriate timing for UIP.
Proposing how to involve users in an IS/IT Project
6 Empirical Research It is not easy to reveal the factors that impact software
product usability and the success of an entire project.
The aim is to determine factors with a positive or negative
impact on delivering usability in software products and
the success or failure of an entire project.
7 Case Study A small company does not have enough resources for
HCI, whilst a software engineer is more concerned with
the performance and capabilities of the developed
software. This problem increases if the software
development method is Agile.
8 Empirical Research There are five failed projects that have been investigated,
and one of the problems is usability which needs to be
addressed.
Need to perform the usability test better, e.g., only asked
about the user interface with a few users.
Did not check usability with real users because the
assumption was that since the expert had designed it, it
was good.
9 Case Study The availability of patient demographic data and poor
quality of clinical referrals.
Healthcare quality improvement projects often use a
narrow range of metrics that provide just one measure of
systems.
10 Case Study IS/IT Implementation projects come into a fail because of
a lot of factors, and User Experience is one of them.
The quality system is the most influential factor in
information system project success based on User
Experience, while the quality of the system itself is
influenced by the constructs of usability, affect, and user
value.

Once the main problems identified in each selected article have been analyzed, Table 5 will be used
to summarize the research findings and recommendations. This summary will serve as a basis for the analysis
related to the research questions in this paper. The column numbering of Table 5 corresponds to Table 3,
which lists the articles to be reviewed. By presenting the research findings and recommendations in a concise
and organized manner, this table will facilitate the comparison and synthesis of the results, providing insights
into the best practices for incorporating usability into IS/IT projects.

The impact of usability in information technology projects (Freddy Richard Putra Hulu)
8  ISSN: 2722-3221

Table 5. Research recommendations

No. Findings Recommendations
1 The study of user participation in IT projects Improving the success rate of an IS/IT Project (particularly in
demonstrated its significance in ensuring developing an Information system for an organization), it is important to
product usability. involve real end-user.
Collected data shows that product usability
increased in projects where end users were
involved.
The findings of an examination of the factors
influencing (high) product usability.
This analysis reveals that the following
factors influenced the user participation
variable: group size, project participation
stage, and group size, with group size having
the greatest impact on product usability.
2 Involving users in the classic waterfall Since Lean UX attempts to incorporate user testing into the scrum sprint
method does not suitable for the Agile or other Agile Timebox, it is important to involve users in carrying out
method. regular testing of the deliverables or the increment of the sprint.
3 The survey of projects also revealed that The author proposes a set of rules that could help the IT Project deliver
formalization of project documentation better usability.
ensures high usability. From the ten rules developed, rules number 1,2,3,4,7,9, and 10 contain
direct guidelines to improve the quality of usability. For this paper, rules
number 1 and number 2 will be summarized as "the better the
formalized documentation, and the higher the involvement of the end-
user, then the higher the usability quality.”
4 There is a real problem regarding IT project There is a need to develop formal standards and guidelines for the
usability, especially in developing countries. government to improve the IT Investment of the government itself. In
Lack of IT skill and experience, combined this case, these are standards and guidelines for usability or other quality
with the organization’s failure to state the attributes for Information System Development. While the author
quality attribute such as usability on the involves in the project as a Human-Computer Interaction researcher,
procurement document, lead to failure in IT joined the project team, and made usability tests and recommendations,
Project. the usability quality of the software was improved.
5 User involvement is important, but only some During the user requirements analysis phase, project managers should
phases of the project need the end user. This ensure that they focus their attention on user participation. They should,
paper concludes that the best phase for in particular, ensure that actual end-users of the IS participate. This
involving the user is the requirement phase should help to improve the software’s perceived usability.
and the testing phase.
6 There are several factors that contribute to the We should bring software developers and UI designers closer together to
project’s failure, and usability is one of the ensure that a reported and approved change is both possible to
factors. Usability issues have happened in all implement and does not degrade usability.
types of project management methods, such We believe that software architecture should include not only the
as waterfall or Agile. technical aspects required to develop and implement a product (namely
functional requirements) but also usability interaction with users as part
of the non-functional requirements.
7 To verify the impact of changes on usability At the end of the case study, the end-users said that there is a better
and user experience, we conducted an dialogue and understanding between clients and developers (IT
interaction flow review. vendors). Moreover, the improvement is also measured in the design
This analysis focused on user interface and quality, usability, and user experience. In addition to that, the feeling of
interaction elements, including validation security and bravery in the development while there is an HCI expert
elements such as multiple features, UI style, joining the development team.
overall usability, and user ratings.
8 IT Project failure is preventable, whether in The author develops a causes and cures model to map any kind of IT
cost, time, and quality. One of the most Project fail factor to researched-based cures. Organizations could apply
reported key fail factors is the failure to the cures throughout the project to avoid or mitigate the risks.
predict the system integration and false
estimation of the human factor (the
developers and the end-users)
9 The improvement of the healthcare IT Project The involvement of the HCI Experts in the hospital-based improvement
could be reached through usability project has led to better product usability. The observed improvement is
improvement of the Information System used. in the quality of clinical referrals and the availability of patient data,
particularly demographic data.
10 Inexperienced users in rural areas could be The most influential factor of IS/IT Projects based on User Experience is
benefited from Information System in a better the quality system, and this quality is heavily influenced by Usability as
way if the usability of the system is good. well as the effect and User Values.
The authors developed a model that combines Government or other policymakers in rural areas regarding Information
the User Experience model with Information technology should pay a better understanding and attention to the aspect
System Project Success. of User Experience for every IS/IT Project kicked off.

4.1. RQ1: How the usability positioned in IT project?

The usability factor is already well-understood as a critical component of product quality in IS/IT
project development. However, project managers often find themselves in a position where they must balance

Comput Sci Inf Technol, Vol. 5, No. 1, March 2024: 7-12

Comput Sci Inf Technol ISSN: 2722-3221  9

the achievement of project goals in terms of time, cost, and quality. In other words, quality must compete
with the other two factors, as achieving better quality often requires additional time and resources, including
those needed to improve usability.
Previous research suggests that some projects need to conduct usability tests on their products,
despite recommendations to invest up to 10% of the budget in usability [44]. Unfortunately, this study was
unable to determine whether this recommendation was applied in the projects reviewed. Moreover, it remains
to be seen why many projects do not include usability requirements or allocate sufficient time and resources
for usability testing and user involvement.

4.2. RQ2: How to improve the quality of usability factor in the IT project success?
The involvement of end-users in the project is widely recognized as a key factor in achieving project
success, as stated in the literature review. However, the involvement must be done in the correct manner, as
involving end-user managers is not suitable. End-user managers are not the actual end-users who will
regularly use the system. Moreover, the involvement of end-users must be timed appropriately and taken into
consideration during the construction phase.
Furthermore, in addition to involving end-users, improving the development team's HCI knowledge
or skill is also important. This can be achieved by providing additional resources to the team, such as hiring
HCI experts or training software engineers in the basics of user experience. Such efforts will lead to the
development of better products that meet end-users needs and expectations.
In addition to project management and organizational efforts, it is also necessary for governments to
develop and establish formalized standards and guidelines. These standards and guidelines can help small
organizations that lack the resources for usability testing to improve the quality of their products. Ultimately,
this can lead to the delivery of better products to clients and end-users.

4.3. Discussion
This subsection aims to contextualize the present study within the available body of evidence by
providing a discussion that compares and contrasts the findings with those of similar studies. In doing so, we
seek to elucidate the contribution of this research to the field of usability in IS/IT project management.
Furthermore, the limitations of this study will be presented, along with suggestions for future research that
could help fill the gaps in the current body of knowledge regarding usability in IS/IT projects.

4.2.1. Interpretation of results in the context of other available evidence

The primary discovery of our study is that involving end-users and usability experts is crucial in
enhancing the usability of the project’s end product. Improved usability implies better system quality,
ultimately increasing the likelihood of success in IS/IT projects. In addition to these findings, we have
identified several other factors that should be considered by project members, such as formalized
documentation, standards and guidelines, and communication and collaboration among designers,
developers, and end-users.
Our findings are consistent with a previous study conducted by Vince Bruno in 2011, which
investigated the importance of usability in IS/IT projects. One of the primary outcomes of Bruno’s study was
the significance of creating a “usability mindset” in key project stakeholders through a collaborative
approach that highlights involvement in usability activities to attain a positive usability outcome [45]. In the
research, he suggested creating a shared usability vision for the project or promoting a usability mindset that
extends beyond the project and across the organizational culture to other projects. The concept of a usability
vision is connected with usability standards and guidelines, as they foster shared understanding and
expectation of the desired usability outcome of a project. Usability goals promote a usability mindset,
extending beyond the project context mindset [46].
Four major themes emerged from Bruno’s research: usability mindset, collaborative approach,
usability practice, and project constraints [45]. The most significant relationships found in the study are the
two robust connections between the themes of a collaborative approach and a usability mindset. The research
emphasized the importance of developing a symbiotic relationship between involvement and a usability
mindset to achieve a beneficial usability outcome. Another noteworthy finding is the consistency of our
results concerning the roles of usability in overall project success. Bruno’s research conducted in 2013
reveals that usability is an iterative process that facilitates stakeholder involvement, articulates usability goals
and requirements, and avoids technological constraints. These critical aspects are instrumental in achieving a
successful usability outcome in a project [47].
Another study by Bruno proposed role differentiation between Usability consultants or experts and
usability practitioners in IS/IT projects. The research revealed that there should be usability managers who
focus on strategic usability issues and improving stakeholder collaboration, usability consultants who
concentrate more on complying with usability activities within a process, and organizational practitioners
The impact of usability in information technology projects (Freddy Richard Putra Hulu)
10  ISSN: 2722-3221

who focus more on nurturing and educating usability understanding within the organization and stakeholders
involved in an IS/IT project. Furthermore, the study showed that an iterative usability process, stakeholder
involvement, articulating usability goals and requirements, and avoiding technological constraints are critical
to achieving a successful usability outcome in a project.
While our research and the previous study both recommend the participation of end-users, usability
experts, and practitioners, there are differences in emphasis. The previous research emphasizes the
involvement of usability practitioners rather than end-users. In contrast, our selected research shows that five
studies suggest end-user’ involvement and two studies suggest the involvement of usability experts.
In alignment with the broader objectives identified within the realm of IT project management,
recent studies corroborate the significance of user engagement and satisfaction in determining project success
[48]. Husam AlFahl’s research, particularly relevant in this context, identifies three pivotal constructs –
application design, application utility, and user interface structure – as the strongest predictors of user’s
continued intention to utilize a mobile application and their loyalty towards it. This finding underscores a
parallel in our research, reinforcing the notion that the sustained engagement of end users is a crucial, albeit
often underemphasized, objective in the lifecycle of IT projects. Emphasizing these aspects can significantly
contribute to the holistic success and long-term viability of IT initiatives.

4.2.2. Limitation and future research

The research in this paper has certain limitations, such as a relatively small number of studies
included. To address this, it may be necessary to expand the search to include other databases and sources of
information. Additionally, the research may not have been conducted in a thorough or rigorous manner, and
the authors acknowledge that there may be more information to be gained from the selected previous research
studies. In future research, it would be beneficial to extend the search for academic literature beyond the
sources used in this paper.

5. CONCLUSION
In conclusion, this paper makes a substantial contribution by highlighting the crucial role of quality
enhancement through usability in the realm of IS/IT projects. The success of an IS/IT project should not be
measured solely in terms of time, financial resources, and technical excellence. Instead, it requires in-depth
consideration of user experience. Even if a project is executed within its allocated resources and schedule, its
long-term viability can still be compromised if end-users are not provided with a satisfying experience. This
research proposes three strategic approaches for improving usability: user engagement, the involvement of
HCI experts, and the adoption of formalized usability standards and guidelines. Organizations have the
option of implementing incremental improvements in usability or can seek the expertise of HCI professionals
for a more standardized approach. It is essential to understand that IS/IT projects involve significant
investments and carry inherent risks of failure. Therefore, comprehensive measures are imperative to prevent
yet another project failure.

ACKNOWLEDGEMENTS
The author extends sincere gratitude to Della Maulidiya for her invaluable insights during the
research and preparation of this manuscript. Additionally, heartfelt thanks are extended to the editors of the
CSIT Journal for their meticulous review and constructive feedback, which significantly enhanced the quality
of this work.

REFERENCES
[1] A. Nizam, “Software Project Failure Process Definition,” IEEE Access, vol. 10, pp. 34428–34441, 2022, doi:
10.1109/ACCESS.2022.3162878.
[2] S. K. Gupta, A. Gunasekaran, J. Antony, S. Gupta, S. Bag, and D. Roubaud, “Systematic literature review of project failures:
Current trends and scope for future research,” Comput. Ind. Eng., vol. 127, pp. 274–285, Jan. 2019, doi:
10.1016/j.cie.2018.12.002.
[3] S. Lauesen, “IT Project Failures, Causes and Cures,” IEEE Access, vol. 8, pp. 72059–72067, 2020, doi:
10.1109/ACCESS.2020.2986545.
[4] M. Ayat, M. Imran, A. Ullah, and C. W. Kang, “Current trends analysis and prioritization of success factors: a systematic
literature review of ICT projects,” Int. J. Manag. Proj. Bus., vol. 14, no. 3, pp. 652–679, Apr. 2021, doi: 10.1108/IJMPB-02-
2020-0075.
[5] A. Lamprou and D. G. Vagiona, “Identification and Evaluation of Success Criteria and Critical Success Factors in Project
Success,” Glob. J. Flex. Syst. Manag., vol. 23, no. 2, pp. 237–253, Jun. 2022, doi: 10.1007/s40171-022-00302-3.

Comput Sci Inf Technol, Vol. 5, No. 1, March 2024: 7-12

Comput Sci Inf Technol ISSN: 2722-3221  11

[6] R. Lyzara, B. Purwandari, M. F. Zulfikar, H. B. Santoso, and I. Solichah, “E-Government Usability Evaluation: Insights from A
Systematic Literature Review,” in Proceedings of the 2nd International Conference on Software Engineering and Information
Management, Bali Indonesia: ACM, Jan. 2019, pp. 249–253. doi: 10.1145/3305160.3305178.
[7] Sunardi, G. F. P. Desak, and Gintoro, “List of Most Usability Evaluation in Mobile Application: A Systematic Literature
Review,” in 2020 International Conference on Information Management and Technology (ICIMTech), Bandung, Indonesia:
IEEE, Aug. 2020, pp. 283–287. doi: 10.1109/ICIMTech50083.2020.9211160.
[8] R. Ren, M. Zapata, J. W. Castro, O. Dieste, and S. T. Acuna, “Experimentation for Chatbot Usability Evaluation: A Secondary
Study,” IEEE Access, vol. 10, pp. 12430–12464, 2022, doi: 10.1109/ACCESS.2022.3145323.
[9] C. Tuena et al., “Usability Issues of Clinical and Research Applications of Virtual Reality in Older People: A Systematic
Review,” Front. Hum. Neurosci., vol. 14, p. 93, Apr. 2020, doi: 10.3389/fnhum.2020.00093.
[10] ISO.org, “Ergonomics of human-system interaction.” 2018. Accessed: Mar. 06, 2022. [Online]. Available:
https://www.iso.org/obp/ui/#iso:std:iso:9241:-11:ed-2:v1:en
[11] N. Bevan and J. Earthy, “Benefiting from ISO Standards,” pp. 51–69, Dec. 2017, doi: 10.1002/9781118976005.ch3.
[12] J. D. Gould and C. Lewis, “Designing for usability—key principles and what designers think,” pp. 50–53, Dec. 1983, doi:
10.1145/800045.801579.
[13] N. Bevana, J. Kirakowskib, and J. Maissela, “What is usability,” in Proceedings of the 4th International Conference on HCI,
Citeseer, 1991, p. 24.
[14] J. Nielsen, Usability engineering, 2nd ed. Boston San Diego New York [etc.]: Academic press, 1993.
[15] J. R. Lewis, “The System Usability Scale: Past, Present, and Future,” Int. J. Human–Computer Interact., vol. 34, no. 7, pp. 577–
590, Jul. 2018, doi: 10.1080/10447318.2018.1455307.
[16] J. Nielsen, “The usability engineering life cycle,” Computer, vol. 25, no. 3, pp. 12–22, Mar. 1992, doi: 10.1109/2.121503.
[17] N. Bevan, J. Carter, and S. Harker, “ISO 9241-11 Revised: What Have We Learnt About Usability Since 1998?,” pp. 143–151,
Aug. 2015, doi: 10.1007/978-3-319-20901-2_13.
[18] J. Sauer, A. Sonderegger, and S. Schmutz, “Usability, user experience and accessibility: towards an integrative model,”
Ergonomics, vol. 63, no. 10, pp. 1207–1220, Oct. 2020, doi: 10.1080/00140139.2020.1774080.
[19] J. Whiteside, J. Bennett, and K. Holtzblatt, “Chapter 36 - Usability Engineering: Our Experience and Evolution,” in Handbook of
Human-Computer Interaction, M. HELANDER, Ed., Amsterdam: North-Holland, 1988, pp. 791–817. doi:
https://doi.org/10.1016/B978-0-444-70536-5.50041-5.
[20] P. Karhapää et al., “Strategies to manage quality requirements in agile software development: a multiple case study,” Empir.
Softw. Eng., vol. 26, no. 2, p. 28, Mar. 2021, doi: 10.1007/s10664-020-09903-x.
[21] T. Olsson, S. Sentilles, and E. Papatheocharous, “A systematic literature review of empirical research on quality requirement s,”
Requir. Eng., Feb. 2022, doi: 10.1007/s00766-022-00373-9.
[22] K. E. S. de Souza et al., “An Evaluation Framework for User Experience Using Eye Tracking, Mouse Tracking, Keyboard Input,
and Artificial Intelligence: A Case Study,” Int. J. Human–Computer Interact., vol. 38, no. 7, pp. 646–660, Apr. 2022, doi:
10.1080/10447318.2021.1960092.
[23] Project Management Institute, Ed., A guide to the project management body of knowledge / Project Management Institute, Sixth
edition. in PMBOK guide. Newtown Square, PA: Project Management Institute, 2017.
[24] P. L. Bannerman and A. Thorogood, “Celebrating IT Projects Success: A Multi-domain Analysis,” in 2012 45th Hawaii
International Conference on System Sciences, Maui, HI, USA: IEEE, Jan. 2012, pp. 4874–4883. doi: 10.1109/HICSS.2012.147.
[25] C. Iriarte and S. Bayona, “IT projects success factors: a literature review,” Int. J. Inf. Syst. Proj. Manag., vol. 8, no. 2, pp. 49–78,
Oct. 2021, doi: 10.12821/ijispm080203.
[26] A. Gemino, B. Horner Reich, and P. M. Serrador, “Agile, Traditional, and Hybrid Approaches to Project Success: Is Hybrid a
Poor Second Choice?,” Proj. Manag. J., vol. 52, no. 2, pp. 161–175, Apr. 2021, doi: 10.1177/8756972820973082.
[27] D. L. Hughes, N. P. Rana, and A. C. Simintiras, “The changing landscape of IS project failure: an examination of the key
factors,” J. Enterp. Inf. Manag., vol. 30, no. 1, pp. 142–165, Feb. 2017, doi: 10.1108/JEIM-01-2016-0029.
[28] A. R. Peslak, “Information Technology Project Management and Project Success,” Int. J. Inf. Technol. Proj. Manag., vol. 3, no.
3, pp. 31–44, Jul. 2012, doi: 10.4018/jitpm.2012070103.
[29] Y. K. Dwivedi et al., “IS/IT Project Failures: A Review of the Extant Literature for Deriving a Taxonomy of Failure Factors,” in
Grand Successes and Failures in IT. Public and Private Sectors, vol. 402, Y. K. Dwivedi, H. Z. Henriksen, D. Wastell, and R.
De’, Eds., in IFIP Advances in Information and Communication Technology, vol. 402. , Berlin, Heidelberg: Springer Berlin
Heidelberg, 2013, pp. 73–88. doi: 10.1007/978-3-642-38862-0_5.
[30] W. H. Money, S. H. Kaisler, and S. J. Cohen, “Understanding Failed Software Projects through Forensic Analysis,” J. Comput.
Inf. Syst., vol. 0, no. 0, pp. 1–14, 2021, doi: 10.1080/08874417.2021.1950076.
[31] M. Shahid and K. A. Tasneem, “Impact of Avoiding Non-functional Requirements in Software Development Stage,” Am. J. Inf.
Sci. Comput. Eng., vol. 3, no. 4, pp. 52–55, 2017.
[32] “The DeLone and McLean Model of Information Systems Success: A Ten-Year Update,” J. Manag. Inf. Syst., vol. 19, no. 4, pp.
9–30, Apr. 2003, doi: 10.1080/07421222.2003.11045748.
[33] D. Russo, P. Ciancarini, T. Falasconi, and M. Tomasi, “A Meta-Model for Information Systems Quality: A Mixed Study of the
Financial Sector,” ACM Trans Manage Inf Syst, vol. 9, no. 3, Sep. 2018, doi: 10.1145/3230713.
[34] M. J. Page et al., “The PRISMA 2020 statement: an updated guideline for reporting systematic reviews,” BMJ, vol. 372, 2020.
[35] K. Redlarski, “The Impact of End-User Participation in IT Projects on Product Usability,” in Proceedings of the International
Conference on Multimedia, Interaction, Design and Innovation, in MIDI ’13. New York, NY, USA: Association for Computing
Machinery, 2013. doi: 10.1145/2500342.2500353.
[36] L. A. Liikkanen, H. Kilpiö, L. Svan, and M. Hiltunen, “Lean UX: The next Generation of User-Centered Agile Development?,”
in Proceedings of the 8th Nordic Conference on Human-Computer Interaction: Fun, Fast, Foundational, in NordiCHI ’14. New
York, NY, USA: Association for Computing Machinery, 2014, pp. 1095–1100. doi: 10.1145/2639189.2670285.
[37] K. Redlarski and P. Weichbroth, “Hard lessons learned: Delivering usability in IT projects,” in 2016 Federated Conference on
Computer Science and Information Systems (FedCSIS), Sep. 2016, pp. 1379–1382.
[38] D. Teka, Y. Dittrich, and M. Kifle, “Adapting Lightweight User-Centered Design with the Scrum-Based Development Process,”
in Proceedings of the 2018 International Conference on Software Engineering in Africa, in SEiA ’18. New York, NY, USA:
Association for Computing Machinery, 2018, pp. 35–42. doi: 10.1145/3195528.3195530.
[39] P. Haake et al., “Configurations of user involvement and participation in relation to information system project success,” vol.
10816 LNCS, pp. 87–102, 2018, doi: 10.1007/978-3-319-91563-0_6.
[40] P. Weichbroth, “Delivering Usability in IT Products: Empirical Lessons from the Field,” Int. J. Softw. Eng. Knowl. Eng., vol. 28,
no. 07, pp. 1027–1045, Jul. 2018, doi: 10.1142/S0218194018500298.
The impact of usability in information technology projects (Freddy Richard Putra Hulu)
12  ISSN: 2722-3221

[41] T. Darin, N. Carneiro, R. Andrade, R. Castro, and R. Almeida, “Integrating HCI Perspective into a Mobile Software
Development Team: Strategies and lessons from the field,” in Proceedings of the 21st International Conference on Human-
Computer Interaction with Mobile Devices and Services, Taipei Taiwan: ACM, Oct. 2019, pp. 1–10. doi:
10.1145/3338286.3347545.
[42] M. Woodward, N. De Pennington, and L. Morgan, “Human Factors Methods Applied to a Healthcare Information Technology
Project,” vol. 222 LNNS, pp. 475–481, 2021, doi: 10.1007/978-3-030-74611-7_65.
[43] D. Yuniarto, E. Firmansyah, A. Saeppani, and D. I. Junaedi, “Information System Project Success based on User Experience at
the Village Administration Office,” in 2021 9th International Conference on Cyber and IT Service Management (CITSM), Sep.
2021, pp. 1–7. doi: 10.1109/CITSM52892.2021.9588978.
[44] J. Nielsen, “Return on Investment for Usability,” Nielsen Norman Group. Accessed: Mar. 27, 2022. [Online]. Available:
https://www.nngroup.com/articles/return-on-investment-for-usability/
[45] V. Bruno, “Improving Usability Outcomes in IS Projects: the Views of Usability Practitioners,” Doctor of Philosophy (PhD),
RMIT University, Australia, 2011. Accessed: Apr. 17, 2023. [Online]. Available:
https://researchrepository.rmit.edu.au/esploro/outputs/doctoral/Improving-usability-outcomes-in-IS-projects/9921861563301341
[46] V. Bruno and M. Dick, “Improving Usability Outcomes for each of the Usability Practitioner Roles,” Electron. J. Inf. Syst. Eval.,
vol. 16, no. 3, pp. 174–188, 2013.
[47] V. Bruno and M. Dick, “Making usability work in industry: an Australian practitioner perspective,” in Proceedings of the 19th
Australasian conference on Computer-Human Interaction: Entertaining User Interfaces, Adelaide Australia: ACM, Nov. 2007,
pp. 261–264. doi: 10.1145/1324892.1324948.
[48] H. AlFahl, “The Usability of Management Information Systems Applications,” Int. J. E-Serv. Mob. Appl., vol. 15, no. 1, pp. 1–
13, Mar. 2023, doi: 10.4018/IJESMA.320502.

BIOGRAPHIES OF AUTHORS

Freddy Richard Putra Hulu is a Master of IT student at Faculty of Computer

Science of University of Indonesia. He received the degree in Information System from the
same university at 2018. His research interest are in area Human-Computer Interface and
related subjects. He works for Indonesian State Own Company since 2010 as an IT Analyst
and involved in several project on various size. In 2021 he rewarded LPDP full scholarship at
Universitas Indonesia. He can be contacted at e-mail: [email protected].

Teguh Raharjo got his Ph.D. in computer science from Faculty of Computer
Science, University of Indonesia. His research is focused to develop enterprise agile
framework. He is a lecture in this university. His research interest includes software
engineering and IT project management. He is also a professional working as a Senior Project
Manager at IBM Indonesia. He got a professional certification for PMP, PMI-ACP, Prince2,
Prince2 Agile, Certified Scrum Master (CSM), and SAFe Agilist (SA). He can be contacted at
e-mail: [email protected].

Tiarma Simanungkalit is a Partner in IBM Indonesia. Tiarma graduated from

the Bandung Institute of Technology with a Bachelor of Industrial Engineering, then she
completed her Master of Information Technology (cumlaude) from University of Indonesia.
She happily shares her knowledge as Assistant Lecturer in the Faculty of Computer Science,
University of Indonesia. She has (co)-authored several publication papers in digital and
technology topics - finding and learning new interesting things on digital and technology
research out there. She can be contacted at e-mail: [email protected].

Comput Sci Inf Technol, Vol. 5, No. 1, March 2024: 7-12