See discussions, stats, and author profiles for this publication at: https://www.researchgate.

net/publication/327424387

A Usability Testing of a Mobile Print shop Booking and Design Application

Article  in  Journal of Advanced Research in Dynamical and Control Systems · August 2018

CITATIONS READS
11 795

2 authors:

Azham Hussain Emmanuel Mkpojiogu

Universiti Utara Malaysia Universiti Utara Malaysia
253 PUBLICATIONS   2,134 CITATIONS    134 PUBLICATIONS   1,487 CITATIONS   

SEE PROFILE SEE PROFILE

Some of the authors of this publication are also working on these related projects:

Usability Evaluation Model for oBike Mobile Application: Novice vs Expert Users View project

Mobile Usability View project

All content following this page was uploaded by Emmanuel Mkpojiogu on 04 September 2018.

The user has requested enhancement of the downloaded file.

Jour of Adv Research in Dynamical & Control Systems, Vol. 10, 10-Special Issue, 2018

A Usability Testing of a Mobile Print shop

Booking and Design Application
1*
Azham Hussain, School of Computing, Universiti Utara Malaysia, Sintok, Kedah, Malaysia, [email protected]
2
Emmanuel O.C. Mkpojiogu, School of Computing, Universiti Utara Malaysia, Sintok, Kedah, Malaysia
3
Khairi Suleiman, School of Computing, Universiti Utara Malaysia, Sintok, Kedah, Malaysia
Abstract—This study presents the usability evaluation of a mobile print shop and design shopping application. The
study was conducted using usability testing methodology to measure the effectiveness, efficiency and satisfaction of
the application. Ten (10) test participants were arbitrarily selected. Some of the participants were either novices or
had some experience on the use of related applications. The study’ result reveals that usability score of the
application was 66%. This implies that the application’s usability was below average. is. This indicates the presence
of usability issues that should be improved upon.
Index Terms—Print shop and Design App, Mobile Usability.

I. Introduction
The revolution of the usage of the Internet has transformed the ways of people behavior in purchasing goods and
services. People nowadays are very busy with their daily life routines resulting in their not being able to shop in
stores. For such people, shopping online is their best alternative. The rapid growth in the development of mobile
phones has led to people’s habitual and daily usage of mobile devices to satisfy their needs. Business card and others
print shop products can be booked for and purchased via mobile phones. This has occasioned the ease at which
people who are so much attached to their mobiles use it and for instance, produce paperless tickets. According to the
report on global e-commerce from [1], more than half of global online purchasing has been accounted for and are as
follows: 58% for fashion products, an average of 55% representing travel products or services, while book, music
and stationery represents 50%, meanwhile 43% have been accounted for IT and mobile and event for tickets
represents 41% of the total global respondents.
Mobile applications have provided ease in purchasing online and this attracts customers. The growth of the
current mobile phone technology has opened and widened the opportunity for airlines, travel and tourism companies
to profit from the technology. According to Nielsen Mobile Wallet Syndicated Report in 2016 [2], Canada’ smart
phone owners represent a large population (76%) that use mobile phones for purchasing-related activities. It was
also highlighted in 2016 by a Bronto report [3] that 64% of Americans use their mobile phones to shop more often
than the usual means of shopping. The purchasing of print shop products and services are the second most likely
online shopping purchased product or service based on the combination of surveys mentioned above. The
company’s products’ tremendous accomplishment can be interpreted by their user-friendly mobile interface. Similar
applications have been introduced in the market in purchasing and designing print shop product. However,
customers or users are not attracted to use any application with poor usability.
Furthermore, heuristic evaluation has been applied to mobile applications due to the fact that it is one of the best
methods of the usability evaluation in use [4]. It is also the most favorite method in Human-Computer Interaction
(HCI) [5]. However, no usability techniques have being used to evaluate mobile applications for print shop booking
and design even though usability evaluation techniques have been applied in many other applications and systems.
The issues found in the mobile print shop booking and design application is to be analyzed for the purpose
improving the interface o the application [6].
This study aims to test a mobile print shop application with real users. A study conducted by [7] had shown that
the methods of usability inspection needed to be combined with the methods of usability testing, that is, heuristic
evaluation can be combined with direct user’ test. The application of heuristic evaluation finds out issues in an
application based on the evaluation of experts. The issues found through the heuristic evaluation would be used to
develop the test scenario for the usability test. When the issues are found by the experts, thereafter users would be
used for the testing. Developing the test scenario for a whole application is time-consuming and unnecessary as the
application may have so many pages or screens. Meanwhile, to find the issues based on the use of real users, the
1359

*Corresponding Author: Azham Hussain, Email: [email protected]

Article History: Received: April 19, 2018, Accepted: May 22, 2018
 A Usability Testing of a Mobile Print shop Booking and Design Application

usability test would have to be conducted. The evaluator would observe users during the test with regard to their task
completion. Also, the use of heuristics method is to prove that issues found by experts should also be found as well
by actual users so that the result of the usability can be reliable and validated.

II. Background Studies

A. Usability Characteristics of the Online Shopping Websites
In [8], the usability characteristics of online shopping website were described. The most pertinent usability
characteristics that need to be considered are in the design of the interfaces. For HCI experts, designing an interface
is the main focus so as to produce a good design and interface which can fulfill the requirement of users. According
to [9], in developing the e-commerce website, the layout, navigation, design, content, and performance are five
characteristics that need to be considered and also not to be left out is the Internet connection on the device can be
unstable and should also be an issue during the designing the application.
B. Existing Methods Applied on Online Shopping Websites
The usability evaluation methods (UEMs) used by [10] applied the inquiry method in the evaluation of online
shopping websites. The authors applied the inquiry method by conducting unstructured interviews with 20
interviewers and administering questionnaires to them. Unfortunately, the design prototype was not tested with real
users. Usability can be defined as the extent to which a system or product is used by specified user in order to
achieve effectiveness, efficiency and user satisfaction [6]. The accuracy of users in completing their tasks without
error can be defined as effectiveness. While the definition of efficiency is hinged on how fast users complete their
tasks consistently, accurately, and completely. Finally, the positive attitude of users towards the use of a system
together with the comfort with which they accomplish their tasks on the system according to ISO 9241-11 [6], can
be defined as user satisfaction [12-18].
C. Types of Usability Evaluation Methods (UEMs)
An important parameter in software development process is usability evaluation. It enables the production of
usable products [19]. By allowing users to interact with the interface design, the evaluation is conducted using task
completion metrics, errors, time used, and subjective satisfaction questionnaire etc [20]. Possible errors and
weaknesses of the system can be identified by the usability evaluation.
Heuristic Evaluation
One of the inspection techniques is heuristic evaluation that is conducted by usability experts or evaluators in
order to find out some issues on the website or application’s interface using a set of usability guidelines [7]. To
reduce cost, produce results in short time, and to avoid the recruiting of users that is time consuming, heuristic
evaluation is employed [10]. Based on [19], however, heuristic evaluation results cannot be completely trusted
because of the non involvement of actual users. According to [21] the reasons for conducting heuristic evaluation
are as follows:
1. To identify possible issues in an application in the early stage of the application’s development before the
conducting of usability test.
2. As there are many pages or screens to be evaluated during usability test, so conducting heuristics evaluation
would help determine the possible issues found in the usability test later.
Usability Test
A user’s opinions and responses or reaction after using a product or system is known as user experience [21].
Petrie and Bevan [21] say that the reasons to conduct usability evaluation are to:
1. Provide evidence of usability issues found by real user.
2. Provide the evidence for developers or managements for possible improvement of the interface.
In addition, multiple observers are allowed to be involved in the evaluation, at one session of the usability testing
and as the test is conducted in a controlled environment. Based on the literature study, no researchers have
conducted heuristic evaluation and/ or usability test on mobile print shop and design applications. This study
conducts usability test to find usability issues using real users. A study conducted by [22] had shown that the
methods of usability inspection needed to be combined with methods of usability test [7]. The issues found on the
heuristic evaluation would be used to develop the test scenario for a usability test. Moreover, when the issues are
found by the experts, thereafter, only the users would be used for the test. Developing the test scenario for a whole
application would be time-consuming and not necessary as the application has so many pages or screens.
Jour of Adv Research in Dynamical & Control Systems, Vol. 10, 10-Special Issue, 2018

Meanwhile, to find issues using real users, the usability test would be conducted. The evaluator would observe users
during the test with respect to their task completion. Using heuristics method proves that the issues found by the
experts should also be found as well by actual users so as to validate the result of the usability.

III. Methodology
The protocol used for this study is as follows: 1) literature review related to the usability evaluation methods
(UEMs); 2) usability test implementation; 3) metrics measurement (ISO 9241-11, 1998); 4) observation and
recommendations; 5) discussion and conclusion of the study [8][22-23]. Test scenarios were created from issues
found in the report of a heuristic evaluation done earlier [24-25], before the usability test is conducted. In addition,
users were asked before the test started to fill a demographic questionnaire. During the test, the users were asked to
perform the test scenario given. On the other hand, the user’s performance would be evaluated by the evaluators
observing and analyzing what users were doing. The measuring of the usability metrics follows. Three usability
metrics [6] were measured: effectiveness, efficiency and satisfaction. The measurement of satisfaction metric was
done by administering a satisfaction questionnaire. The questionnaire is structure using a 5-points Likert scale that
ranged from 1 to 5. The evaluated application was Achik.biz mobile app.
All the questions were designed in order to get feedbacks on the users’ feelings about the Achik.biz application.
The usability score of the application was calculated in percentage by averaging the three of usability metrics:
effectiveness, efficiency and satisfaction. Users were closely observed by the evaluators during the test session. The
usability issues found were elaborated in detail with their severity levels. More so, the study used observations and
questionnaires for data collection [26-27]. Uses selected as participants for the test were either novice users with no
previous experience about the application or users that have experienced in other mobile print shop and design
applications. Ten (10) users were purposively chosen [22].
The three usability metrics, namely, effectiveness, efficiency and satisfaction were applied [21] in conducting the
usability test on this mobile print shop and design application. During the observation phase of the usability test, the
metrics of effectiveness and efficiency were measured. Meanwhile, the satisfaction questionnaire measures the
satisfaction metric. The questionnaires were distributed to the users after the test was conducted. The scores of
effectiveness, efficiency and satisfaction metrics were captured and the average of these three scores was calculated
to achieve the application’s overall usability score.

IV. Results
Of 10 participants used in the test, more than half of were female (80%) and the others were male (20%).
Majority of the participants were Malays (80%), the other participants were divided among the Chinese (10%), and
Indians (10%). With regard to age, majority were aged 21 to 30 years old (60%), while others were 31 to 40 years
(40%). The results indicate that 100% of the participants used touch screen mobile phone daily and 20% of them
were familiar with the mobile print shop and design application and have also bought at least a print shop product on
their phone. The other 80% were not familiar with the mobile print shop and design application.
A. Effectiveness Measurement

*Corresponding Author: Azham Hussain, Email: [email protected]

Article History: Received: April 19, 2018, Accepted: May 22, 2018
1361
 A Usability Testing of a Mobile Print shop Booking and Design Application

Figure 1. Application’s Effectiveness

It can be concluded from the observations made and findings obtained that the mobile print shop and design was
not ease to use as such as the results show that no user successfully completed Task #3 (select item and design).
Some users were not able to complete the payment process, because it was hanging at the payment page as the task
requested. Regarding Task #1, only user 4, 5, 9 and 10 completed the task. The login button that does not work as
expected caused the problem even when users have already keyed in the input correctly. From the observation of
Task #3, there were no users who were able to complete the task. User 1 encountered a network application issue,
user 4 encountered a page server error, while, user 3 encountered an unavailable ATM transfer payment method and
user 8 decided to give up on the task. In addition, 80% of users successfully completed Task #7. Only two users
partially completed the task and took so much time to do so.
There were 32 tasks criteria with 10 attempts per task and 320 of the total attempts. 197 attempts of the tasks
criteria \were completed successfully and 50 tasks criteria were partially successful. However, there were 73 tasks
criteria that were completed unsuccessfully which was calculated as 73 x 0% = 0. Furthermore, to calculate the
effectiveness rating of this set of tasks criteria the following equation was used:
Effectiveness (%) = (Yes + (Partial*0.5)) / Total*100% (1)
In Figure 2., 4 users, users 1, 3, 6, and 7 had scores below 66%, thus were below average, that is, C [2]. Using
Eq. (1), the effectiveness rating of the Achik.biz application with 10 participants was 68.9%. This result indicates
that the effectiveness of the application was average or C.
B. Efficiency Measurement

Figure 2. Application’s Efficiency

Only 2 participants selected the right step at the first try in the purchase of print shop product. In addition, 8 out
of 10 users took much time to complete Task #2. In other words, it can be said that the print shop and design
application was not efficient. Another issue is on Task #5, only 2 participants were able, to select the menu at the
first try, and only 2 participants did not take much time to complete the task. The application was also not easy to
learn as observed. As observed from Figure 2, 5 users, users 2, 3, 6, 7, and 9, had score below 66% that is, an
average, C. the efficiency rating of Achik.biz application with 10 participants was 66.56%.
C. Satisfaction Measurement
Using a 5-point Likert scale with “strongly disagree” representing 1 and “strongly agree” representing 5, each
question was answered by 10 participants who offered a possible positive response factor of 60 points. Thus, there
were 600 points or 100% satisfaction for 12 questions. In order, to get the satisfaction rating for the Achik.biz
application, the following equation was used..
Satisfaction (%) = Answer / Total*100% (2)
Jour of Adv Research in Dynamical & Control Systems, Vol. 10, 10-Special Issue, 2018

Figure 3. Application’s Satisfaction

Using Eq. (2), the usability testing with 10 participants had the satisfaction rating for Achik.biz as approximately
61.67%. All the users who had high satisfaction scores were users that had never bought ticket on their mobile. In
other word, they do not have any prior experience with any others mobile purchase and design application with good
usability. However, users who have previous experience in purchasing and designing print shop products on their
mobile (user 3), had their satisfaction score very low. That is, the users felt that the application was not easy to use
compared to their favorite mobile booking application such as Vistaprint.com.
D. Usability Score
The mean usability score was computed by averaging the three metrics: effectiveness, efficiency and satisfaction
in percentage form, The usability score for the Achik.biz application can be derived from the following equation:
Usability (%) = (Effectiveness + Efficiency + Satisfaction) / 3 (3)
From Eq. (3), the usability testing with 10 users showed that the usability level for the Achik.biz app was
approximately 66.67%. According to [22], the average usability score is 68%. Thus, the usability of Achik.biz is
below average or C [28]. There are probably some serious problems on the application’s usability that need to be
addressed.

V. Conclusion
The usability measurements for the three metrics were observably related. It means that the results of the
observations and questionnaire were strongly interrelated. However, the study indicates that the users’ perceptions
toward application may be somewhat different with the satisfaction results. Most of the participants answered
neutral or agree to the first post-test questionnaire of “I would use this application again”. However, the satisfaction
results showed that the score was below average (62%). The users were not satisfied with the application as adduced
from the results. Another issue was about users’ understanding about the test scenario. It was observed that some
users were very confident in completing their tasks. Users felt that they had completed the task correctly but in
actual terms of completion, it was a failure. Some users got confused while searching for a product even though the
task was clearly mentioned and were asked to search using the icon search on the application. The results show that
the test scenario was not well understood by users. Prior to the test execution, in order to make users understand
what next to be done, more briefings are needed in future tests. Furthermore, before the test execution is enforced,
the design of the test scenario should be tested many times. Overall, the application has weak and poor usability (as
its performance is below average (66%)). This shows that the application certainly has some serious issues that need
to be fixed. However, in order, to validate and generalize these results, more frequent testing with different operating
systems and users are needed so as to present different platforms for comparison.

References
[1] “A Look at the Evolving E-Commerce Landscape”, Retrieved December 02, 2017, from
http://www.nielsen.com/my/en/insights/news/2017/a-look-at-the-evolving-e-commerce-landscape.html,
2017.

*Corresponding Author: Azham Hussain, Email: [email protected]

Article History: Received: April 19, 2018, Accepted: May 22, 2018
1363
 A Usability Testing of a Mobile Print shop Booking and Design Application

[2] “Canadian Mobile Wallet Users Set Mobile Payment Trends”, Retrieved December 02, 2017, from
http://www.nielsen.com/ca/en/insights/news/2017/canadian-mobile-wallet-users-set-mobile-payment-
trends.html, 2017.
[3] “Younger consumers turn increasingly to mobile devices for buying as well as browsing”. Retrieved
December 02, 2017, from https://www.digitalcommerce360.com/2016/04/25/younger-consumers-turn-
increasingly-mobile-devices-buying/, 2017.
[4] J. Mankoff, A.K. Dey, G. Hsieh, J. Kientz, S. Lederer, and M. Ames, “Heuristic evaluation of ambient
displays”. In Proceedings of the SIGCHI Conference on Human factors in Computing Systems, ACM,
2003, pp. 169-176.
[5] C. Forsell, and J. Johansson, “An heuristic set for evaluation in information visualization”. In Proceedings
of the International Conference on Advanced Visual Interfaces, ACM, 2010, pp. 199-206.
[6] ISO 9241-11. “Ergonomic requirements for office work with visual display terminals (vdts)–part 11:
Guidance on usability”. ISO Standard 9241-11: 1998. International Organization for Standardization, 1998.
[7] A. Holzinger, “Usability engineering methods for software developers”. Communications of the ACM,
48(1), 2005, pp. 71-74.
[8] R. Inostroza, C. Rusu, S. Roncagliolo, and V. Rusu, “Usability heuristics for touchscreen-based mobile
devices: update”. In Proceedings of the 2013 Chilean Conference on Human-Computer Interaction, ACM,
2013, pp. 24-29.
[9] R. Molich, and C. Wilson, “Tips and tricks for avoiding common problems in usability test facilitation”. In
CHI'08 extended abstracts on Human factors in computing systems, ACM, 2008, pp. 2379-2382).
[10] F. Gündüz, and A.S.K. Pathan, “On the key factors of usability in small-sized mobile touch-screen
application”. Int. J. Multimed. Ubiquitous Eng, 8(3), 2013, pp. 115-138.
[11] A. Hussain, E.O.C. Mkpojiogu and Z. Hussain, “Usability evaluation of a web-based health awareness
portal on Smartphone devices using ISO 9241-11 model,” JurnalTeknologi (Sciences & Engineering), 77
(4), 2015, pp.1-5.
[12] A. Hussain, E.O.C. Mkpojiogu, H. Almazini, and H. Almazini, “Assessing the usability of shazam mobile
app”. Proceedings of the 2nd International Conference on Applied Science and Technology (ICAST’17),
Kedah, Malaysia, AIP Conference Proceedings, 2017.
[13] A. Hussain, E.O.C. Mkpojiogu, J. Musa, and S. Mortada, “A user experience evaluation of amazon kindle
mobile application.” Proceedings of the 2nd International Conference on Applied Science and Technology
(ICAST’17), Kedah, Malaysia. AIP Conference Proceedings, 2017.
[14] A. Hussain, E.O.C. Mkpojiogu, N.M. Fadzil, and N.M. Hassan, N.M. “The UX of amila pregnancy on
mobile device.” Proceedings of the 2nd International Conference on Applied Science and Technology
(ICAST’17), Kedah, Malaysia. AIP Conference Proceedings, 2017.
[15] Hussain, A., AbdRazak, M.N.F., Mkpojiogu, E.O.C. and Hamdi, M.M.F. (2017). “UX evaluation of a
video streaming application with teenage users.” Journal of Telecommunication, Electronic & Computer
Engineering (JTEC), 9 (2-11), 2017, pp.129-131.
[16] A. Hussain, M. Isam, and E.O.C. Mkpojiogu, “A UX assessment of a mobile recommender app for
household electrical energy savings”. Journal of Telecommunication, Electronic & Computer Engineering
(JTEC), 9 (2-11) 2017.
[17] A. Hussain, H.A. Razak, and E.O.C. Mkpojiogu, “The perceived usability of automated testing tools for
mobile applications”. Journal of Engineering Science and Technology (JESTEC), 4, 89-97, 2017.
[18] A. Hussain, E.O.C. Mkpojiogu, and N.M.D. Jasin, “Usability metrics and methods for public transportation
mobile applications: a systematic literature review”. Journal of Engineering Science and Technology
(JESTEC), 4, 98-105, 2017.
[19] F. Gündüz, and A.S.K. Pathan, “Usability improvements for touch-screen mobile flight booking
application: A case study”. In 2012 International Conference on Advanced Computer Science Applications
and Technologies (ACSAT), IEEE, 2012, pp. 49-54.
[20] A, Dillon, “Beyond usability: process, outcome and affect in human-computer interactions”. Canadian
Journal of Library and Information Science, 2002.
[21] H. Petrie, and N. Bevan, “The evaluation of accessibility, usability, and user experience”. In The universal
access handbook, CRC Press, 2009, pp. 1-16.
[22] A. Kaikkonen, A Kekäläinen, M. Cankar, T. Kallio, and A. Kankainen, “Usability testing of mobile
applications: A comparison between laboratory and field testing”. Journal of Usability Studies, 1(1),
2005,pp. 4-16.
[23] J. Nielsen, “First rule of usability? Don't listen to users”. The Alertbox: Current Is-sues in Web Usability.
See:< http://www. useit. com/alertbox/20010805. html, 2001.
View publication stats

Jour of Adv Research in Dynamical & Control Systems, Vol. 10, 10-Special Issue, 2018

[24] A. Hussain, E.O.C. Mkpojiogu, and K. Suleiman, “A heuristic evaluation of a mobile print and design
shopping application,” JTEC, 2018.
[25] A. Hussain, E.O.C. Mkpojiogu, and K. Suleiman, “A heuristic evaluation of achik.biz mobile shopping
app” AIP conf., Proc., 2018.
[26] J. Nielsen, “Enhancing the explanatory power of usability heuristics”. In Proceedings of the SIGCHI
conference on Human Factors in Computing Systems, ACM, 1994, pp. 152-158.
[27] M. Ismail, M.N. Diah, S. Ahmad, N.A.M. Kamal, and M.K.M. Dahari, “Measuring usability of
educational computer games based on the user success rate”. In 2011 International Symposium on
Humanities, Science & Engineering Research (SHUSER), IEEE, 2011, pp. 56-60.
[28] A. Bangor, P. Kortum, and J. Miller, “Determining what individual SUS scores mean: Adding an adjective
rating scale”. Journal of Usability Studies, 4(3), 2009, pp. 114-123.

*Corresponding Author: Azham Hussain, Email: [email protected]

Article History: Received: April 19, 2018, Accepted: May 22, 2018
1365