Bartlett et al.

Research Involvement and Engagement (2021) 7:32

https://doi.org/10.1186/s40900-021-00273-2

RESEARCH ARTICLE Open Access

Evaluating human-centred design for

public health: a case study on developing a
healthcare app with refugee communities
Rebeccah Bartlett1* , Jacqueline A. Boyle1, Jessica Simons Smith2, Nadia Khan1, Tracy Robinson3 and
Rohit Ramaswamy2

Abstract
Background: Australian women from migrant and refugee communities experience reduced access to sexual and
reproductive healthcare. Human-centred design can be a more ethical and effective approach to developing health
solutions with underserved populations that are more likely to experience significant disadvantage or social
marginalisation. This study aimed to evaluate how well Shifra, a small Australian-based not-for-profit, applied
human-centred design when developing a web-based application that delivers local, evidence-based and culturally
relevant health information to its non-English speaking users.
Methods: This study undertook a document review, survey, and semi-structured interviews to evaluate how well
Shifra was able to achieve its objectives using a human-centred design approach.
Results: A co-design process successfully led to the development of a web-based health app for refugee and
migrant women. This evaluation also yielded several important recommendations for improving Shifra’s human-
centred design approach moving forward.
Conclusions: Improving refugees’ access to sexual and reproductive health is complex and requires innovative and
thoughtful problem solving. This evaluation of Shifra’s human-centred design approach provides a helpful and
rigorous guide in reporting that may encourage other organisations undertaking human-centred design work to
evaluate their own implementation.
Keywords: Human-centred design, Design thinking, Refugee health, Evaluation

* Correspondence: [email protected]
1
Monash Centre for Health Research and Implementation – MCHRI, School of
Public Health and Preventive Medicine, Monash University, Melbourne,
Australia
Full list of author information is available at the end of the article

© The Author(s). 2021 Open Access This article is licensed under a Creative Commons Attribution 4.0 International License,
which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give
appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if
changes were made. The images or other third party material in this article are included in the article's Creative Commons
licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons
licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain
permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/.
The Creative Commons Public Domain Dedication waiver (http://creativecommons.org/publicdomain/zero/1.0/) applies to the
data made available in this article, unless otherwise stated in a credit line to the data.
Bartlett et al. Research Involvement and Engagement (2021) 7:32 Page 2 of 13

Plain English summary

Australian women from non-English speaking migrant and refugee communities face reduced access to sexual and
reproductive healthcare and many then go on to experience poor health outcomes as a result. There is an urgent
need for new approach to improve access to healthcare for underserved communities, one that centres these
women in the process of finding, developing and disseminating the solutions themselves. Human-centred design
can be a more ethical and effective methodology in working with communities to develop these health solutions.
This study aimed to evaluate how well Shifra, a small Australian-based not-for-profit focused on improving access
to healthcare for refugees and new migrants, undertook human-centred design approach when developing a
Smartphone app that delivers local, safe and culturally relevant health information to non-English speaking
Australians. The authors interviewed refugees, health and social sector experts and computer programmers involved
in creating Shifra to evaluate how well they used human-centred design to achieve its goals. This evaluation found
that Shifra’s approach was successful whilst also highlighting several important recommendations for improving
collaborative efforts with refugee communities. These findings could help other projects also seeking to undertake
an authentic community co-design process.

Introduction obtaining an understanding of their needs, desires,

During 2017, Australia became home to 16,757 refu- and experiences, which often transcends that which
gees [1]. Women and children, who comprised 87% the people themselves actually realised” [11]. Human-
[1, 2] of these new arrivals, face significant health centred design, design thinking, co-design, co-
challenges, including limited access to quality sexual production and co-creation are all terms that are
and reproductive health (SRH) services [3, 4]. Poor often used interchangeably despite having nuanced
SRH care has intergenerational consequences, affect- differences in application and outcome. Each of these
ing health and psychosocial outcomes for both approaches focus on addressing complex problems
mothers and their children [5]. Conversely, access to and designing solutions with the end user communi-
quality SRH services improves a number of health ties (i.e. beneficiaries) [12, 13]. Design thinking is a
outcomes in women including prevention and man- specific set of stages within the HCD approach which
agement of high-risk pregnancies, reduction in un- help to guide problem solving teams through the
planned pregnancies and abortions, reduction in whole experience as it diverges and converges the In-
obstetric complications, decreased anaemia and im- spiration, Ideation and Implementation phases in an
proved nutrition for both mother and baby [6, 7]. iterative manner. Co-design, short for collaborative
Despite these benefits, within Australia, women from design is the process of design thinking steps that in-
migrant and refugee communities report less SRH cludes generative research (i.e. learning from end
awareness and experience reduced access to SRH- users) and development design (i.e. creating solutions
specific care as well as culturally-relevant support that with end users) [14]. This second, development design
could assist them to make evidence-based decisions stage, is often termed co-production and together
about their own health and service utilisation [2]. A with co-design, these two stages form co-creation.
new approach to improve access to healthcare for Studies have demonstrated promise that using HCD
underserved communities, particularly for women when developing health interventions can improve
from refugee and migrant backgrounds, is needed, health outcomes for diverse populations [15–17] and
one that centres these women in the process of find- that solutions developed using this approach result in
ing, developing and disseminating the solutions increased uptake of services [18]; produce higher
themselves. quality products and interventions; and that these
Addressing public health problems through human- products and interventions increased beneficiary satis-
centred design (HCD) can be a more ethical and ef- faction [19]. Importantly, this approach allows for the
fective approach to developing solutions with under- development of locally-driven, contextually-
served populations that are more likely to experience appropriate information that is crucial for meeting
significant disadvantage or social marginalisation [8, the health literacy needs of this population. However,
9]. HCD utilises multidisciplinary teams to approach a scoping review analysing 21 different studies for use
the problem-solving process through three distinct of HCD in global health across various geographies
phases: Inspiration, Ideation and Implementation [10]. and populations was unable to draw definitive conclu-
HCD utilises “techniques which communicate, inter- sions about the effectiveness, because of the hetero-
act, empathise, and stimulate the people involved, geneity of implementation, application areas and
Bartlett et al. Research Involvement and Engagement (2021) 7:32 Page 3 of 13

contexts [20]. There remains a lack of understanding included refugee end users, subject matter experts
regarding how best to achieve and evaluate a success- (SMEs) from different, partner organisations that
ful HCD-driven solution [20] and how to destabilise focus on health for multicultural communities, user
power structures inherent to the HCD process itself. experience (UX) students, and computer program-
Maya Goodwill (2020) argues five different yet inter- mers. Given the sensitive nature of the content for
related forms of power exist within the design this app, SMEs were used to support and reinforce
process. These include privilege, access power, goal the refugees’ opinions when designs may be seen by
power, role power and rule power” [21]. These power some in the community as confronting. This hap-
differentials are present no matter how well- pened in one instance, where a refugee end user
intentioned the design process is and there is a grow- noted that icons used to reference herpes were in-
ing need to evaluate the implementation of programs appropriate. The designers questioned this, an SME
that apply HCD principles, particularly when working was independently consulted in the same session and
with communities affected by the legacy of colonisa- supported the initial opinion that the image was in-
tion and systemic bias. Implementation evaluations appropriate and should not be used. Following this
not only assess a program’s deliverables against session, the designer’s supervisor was notified of the
intended goals but also identify the strengths and issue and the refugee who made the initial observa-
weaknesses of a the implementation process, inform- tion was debriefed and reassured that her opinion
ing replication and efforts to scale [22]. To date, there was valid and paramount to the development of a
are limited studies or reports detailing an implemen- sensitive and quality app.
tation evaluation of an entire HCD project. Most Local partnerships were integral to completing many
studies address only one aspect of the design process of the design steps. In 2017, Shifra collaborated with an
such as planning [18, 23], prototyping [24] or asses- undergraduate UX class from Monash University’s
sing stakeholder engagement [25] but do not evaluate School of Art Design and Architecture. A semester-long
the entire approach. This study aimed to evaluate the process to design a digital health solution to bridge the
HCD approach that Shifra, a small Melbourne-based gap in refugees’ access to SRH services resulted in five
not-for-profit focused on improving access to health- prototypes. Shifra’s founder (RB) then selected two de-
care for refugees and new migrants, undertook in de- signs to combine and develop further in conjunction
veloping a web-based application to deliver local, with the co-designers to incorporate end users’ values
evidence-based and culturally relevant SRH informa- and cultural beliefs [27]. The Shifra team used a combin-
tion to its users. Future papers will assess the relative ation of design thinking approaches developed by IDEO,
success of the Shifra app in achieving its intended Stanford’s d-School and Mummah et al. (2016) [10, 24].
outcomes related to improving SRH literacy within IDEO’s Field Guide to Human Centred Design [10]
non-English speaking refugee communities. and the Stanford d-School’s Process Guide influenced
the development of the Empathise, Define, Ideate, Proto-
Context type and Test steps [28] however the external assessor
The Shifra web-app (herein simply referred to as an renamed the Test step “Launch and Share”, to ensure
“app”), provides high quality, rights-based information appropriate dissemination of any product or early re-
on family planning, pregnancy and newborn health, search findings as per Fig. 1 [24]. Given the importance
sexuality and sexual health, as well as mental health, of evaluating public health interventions yet the lack of
family violence and adolescent health. Written and robust methodology surrounding those that are co-
video resources provide information on accessing designed, the Shifra team planned for process evaluation
health services in Australia and cover topics such as to be undertaken regularly and as objectively as possible.
healthcare rights and responsibilities, accessing trans-
lating and interpreting services, public and private in- Shifra’s human-centred design process
surance, as well as clinic locations [26]. The app was
originally designed for English and Arabic-speaking Empathise & define Empathy sessions between Arabic-
communities living in Melbourne and is in the speaking refugees, refugee advocates and healthcare
process of being translated into other languages. As workers took place with final year UX students from
Shifra is committed to working in partnership with Monash University throughout the first half of 2017. Re-
refugee and migrant communities to create products searchers undertook CBPR activities and group surveys
that are both ethical and sustainable [26], the team to better understand barriers and enablers to accessing
chose to apply a HCD approach, using design think- healthcare for women from these refugee backgrounds.
ing methods to prototype and eventually develop its Partner organisation, Multicultural Centre for Women’s
digital health intervention. Co-designers for the app Health (MCWH), referred four refugee end users and
Bartlett et al. Research Involvement and Engagement (2021) 7:32 Page 4 of 13

Fig. 1 Design Thinking Process Adaptation for Shifra

two others were recruited using snowball-sampling tech- Methods

niques. MCWH also assisted in connecting the CBPR re- Evaluation questions
searchers to women interested in helping the Shifra The evaluation was designed and conducted by an exter-
team improve their understanding of the healthcare nal assessor (JSS) to reduce bias and focused on the fol-
journey of different women from within these communi- lowing three questions:
ties in Melbourne.
1. To what extent did Shifra complete all the steps of
Ideate & design UX students designed a solution the design thinking process shown in Fig. 1?
based upon the insights gained during these empathy 2. To what extent did the final Shifra app incorporate
sessions. These designs were ideated then iterated the contributions of all co-designers?
with end users and other key stakeholders over sev- 3. To what extent were the co-designers satisfied with
eral sessions throughout Monash University’s first se- the process?
mester in 2017. After selecting the winning design,
Shifra’s founder participated in a hackathon hosted by The first question assisted Shifra’s team to understand
Random Hacks of Kindness, a not-for-profit company how faithful to design thinking principles the initial co-
that connects business analysts, programmers and UX design sessions were. The answer to this question, deter-
designers with social impact organisations for week- mined through a scoring rubric, could help the team im-
end long prototyping meetups. Several computer pro- prove future co-design endeavours. Shifra could only
grammers worked on Shifra at the event and score a full 12 points if it adequately addressed the fol-
continued to develop the technological component of lowing criteria:
the app until the next hackathon five months later.
The beta version of this app was developed, tested  End users in co-design sessions were engaged, felt
and iterated with Arabic-speaking refugees over the respected, were compensated and were representa-
next five months. tive of the whole target population
SMEs vetted health information and simplified content  Empathy exercises were undertaken to understand
into plain language for accuracy and accessibility. After the lived of experience end users
development of the initial prototype, a more advanced  Learnings from empathy exercises were compiled,
version was user tested with two different groups of co- brainstorming solutions sessions were done in
designers. First, Arabic-speaking women tested the beta teams, end user insights guided the creation of the
version of the app through a partnership with a local adult solution, additional information was gathered from
education program and neighbourhood house, located in end users if necessary
Melbourne’s inner-city suburbs where one in four people  Group consensus was obtained on the problem to
are from migrant or refugee backgrounds. Again, the app’s be addressed/solved. Group consensus was obtained
content was user tested with SMEs for accuracy and ac- regarding which solutions to prototype to solve the
cessibility. Online SMEs from around the world com- identified problem
pleted functionality and basic content testing via a  Multiple iterations of prototypes and/or MVPs were
Qualtrics online survey software and a group of local created and tested with end user population
SMEs then met in person to review and edit the health  End user feedback was incorporated into subse-
content, ensuring it was evidence-based before simplifying quent iterations of solution
the information further into plain English prior to Arabic  Solution was validated with subject matter experts
translation. The six people who attended user testing ses- and/or existing literature
sions and the all SMEs were recruited using snowball-  Product was launched
sampling techniques.  User testing was completed to understand users’ ex-
perience and satisfaction
Launch & share Shifra launched the beta version of the  User feedback was incorporated into plans for fu-
app in August 2017 with an event attended by co- ture iterations
designers, supporters, and funders.  Process or product results were shared with
Bartlett et al. Research Involvement and Engagement (2021) 7:32 Page 5 of 13

program staff, co-designers, and wider community design steps used (Fig. 1) appear linear, the design
process is fluid and the steps listed did not neces-
A score of < 3 equated with poor evaluation, 4–6 with sarily occur in a stepwise fashion. The rubric as-
adequate, 7–9 good and > 10 was equivocal to excellent sesses the level of completion of each step of the
though feedback should always include room for design thinking process on a scale from 0 (non-ex-
improvement. istent) to 3 (full completion) with a maximum score
The second question was intended to determine the possible of 12.
extent of co-designer involvement, a hallmark of the A survey (Table 3 in Appendix) and a semi-
HCD approach. The third and final question would not structured interview guide (Table 4 in Appendix)
only assess co-designer satisfaction but also inform modelled on the IDEO Field Guide to Human
decision-making about how to structure future design Centred Design and mHealth evaluation guidelines
sessions. [10, 29, 30] were created to answer the second and
third evaluation questions as well as to obtain clarifi-
Ethics approval cation and confirmation of the data obtained through
Monash University provided ethics approval prior the document review.
to data collection (Project ID number 13811: These questions explored end user representation,
Evaluating the process and product of Shifra’s co-designers’ understanding of the co-design sessions
mHealth intervention). As the external evaluator and design thinking methods used, issues around
was from The University of North Carolina-Chapel communication (i.e., language barriers, role clarifica-
Hill (UNC), the Institutional Review Board from tion and understanding HCD goals), and co-
UNC also reviewed the evaluation plan and deter- designers’ levels of satisfaction, using a Likert scale,
mined no additional ethics approval requirements (Table 3 in Appendix) regarding their involvement
for this project (UNC-Chapel Hill Study #18–1449) in the process.
Table 1.
Survey and interview procedures
Data collection approach The external evaluator (JSS) engaged three groups of
The primary data for the first question (completion co-designers to complete surveys and semi-structured
of designing thinking steps) involved a thorough re- interviews. All co-designers were approached however
view of all of Shifra’s organisational documents in- not all responded or accepted the invitation to be in-
cluding student design reports, community based volved in the evaluation. This included four refugee
participatory research (CBPR) results, meeting notes end users (out of six originally involved in the pro-
from prototyping events, Qualtrics data from user ject), three UX designers or computer programmers
testing, emails between Shifra staff and the com- (UX/programmers) and six SMEs (including one
puter programmers regarding app updates and re- funding representative) totalling 13 respondents and
quests, as well as launch event information. Since representing approximately 80% of those involved. All
there is a notable lack of validated tools evaluating SMEs and UX/programmers were fluent in English
HCD projects, a maturity rubric was designed to even if it was not their native language. End users
synthesize the findings from the document review. considered themselves “conversational-level” English
This rubric was developed through consultation speakers. Ten of the respondents identified as women.
with two experts in the field of implementation sci- All co-designers had at least a bachelor’s-level educa-
ence (RR, JAB), two reproductive and indigenous tion. Shifra compensated the refugee end users for
health experts (JAB, RB), one HCD expert (RR) and their time participating in the evaluation through
one participatory research expert (TR). Several iter- store-bought gift cards.
ations of feedback from the expert panel were used The co-designers who participated in the evaluation
to improve the usability, completeness, and level of represented the larger co-design groups involved in
detail of the rubric (Table 2). While the rubric and creating the Shifra app in relation to gender identity,

Table 1 Co-designer involvement

Co-designer category Process involvement Evaluation involvement Involvement details
n= n=
Refugee 6 4 Attended 2 design and 1–2 test sessions.
UX/ Programmer 3 3 Attended all design and test sessions
SME 7 6 Attended 1 design and 1 test session
Bartlett et al. Research Involvement and Engagement (2021) 7:32 Page 6 of 13

Table 2 Shifra HCD evaluation score

Comments Points
Empathise and define
Moderately attempted: (2 points) • The organisational documents provided evidence of compensation, 2
End users were engaged, felt respected, and were compensated for end user interviews and ethnographic work, CBPR and empathy
their participation in co-design sessions. (Compensation may be fi- exercises but there was no documentation regarding whether co-
nancially via cash or gift cards, transportation costs to get to ses- designers felt respected or recognised for their contribution. Instead,
sions, provision of childcare during sessions, or other means.) this information came from the surveys.
Empathy exercises were undertaken to understand lived experience • All co-designers selected ‘agree’ or ‘strongly agree’ to feeling
of end users. respected and if they would encourage others to participate in a co-
design session with the Shifra team.
• The representativeness of the end user co-designers was also not ap-
parent in any of the organisational documents but emerged as a
theme during the interviews.
• Co-designers’ views on how representative the end users were var-
ied by the type of co-designer group the participant came from.
• SMEs and UX/programmers generally agreed that there was enough
end user representation at the co-design sessions. As one computer
programmer said: “We focused bringing on more and more people
from the refugee and migrant community which is really good … I
don’t think we had a shortage of that diversity … in terms of cultural
background it was quite well represented.”
• The end users themselves felt that there were groups within the
Arabic-speaking population that were unrepresented. Some of the
suggested groups include individuals who did not attend university,
Arabic speakers with no or low English proficiency, middle and late
middle age individuals, people with different levels of proficiency
with mobile technology, and refugees who had just arrived to
Australia compared to refugees who have been living in Australia for
some time.
• A more representative end user population would have accrued a
higher score.
Ideate and design
Minimally attempted: (1 point) • An extensive document review verified that the Shifra team did not 1
Learnings from empathy exercises were compiled. have a pre-conceived idea of what the prototype would be, and that
Pre-determined solution was minimally modified in response to the app’s features and structures arose in response to insights gained
what was learned from end users. from the end user co-designers.
• The ideation stage took place within the design student teams and
was tested at intervals with end users.
• Surveys and interviews with the co-designers revealed that one quar-
ter of all survey participants marked ‘neither agree nor disagree’ to
whether Shifra confirmed group consensus for either the problem
statement or the solutions to be prototyped. The remaining three
quarters indicated they ‘agree’ or ‘strongly agree’ that Shifra did in
fact, achieve this.
• Whilst the issue of consensus did not emerge as a theme across all
interviews, one computer programmer articulated the problem this
way: “I feel like sometimes there was a bit of disconnect between what
[Shifra staff] wanted and maybe necessarily what the refugees wanted
… they wanted a resource where they could find health information,
locations specifically of hospitals, GPs, pharmacies … the actual health
information, they would rather go directly to the source, rather than …
reading it online.”
• Given the conflicting information received from the surveys and the
interviews and the lack of documentation, the evaluator reported
finding difficulty scoring Shifra in this area.
• A lower score of 1 out of 3 was given to draw attention to this issue
in the future.
• More thorough record keeping during this design stage will shed
light on this process and the methods used in future HCD projects.
Prototype
Satisfied: (3 points) • End users from different backgrounds tested and contributed to 3
Multiple iterations of prototypes and/or MVPs created and tested iterations of the app on multiple occasions and continue to do so to
with end user population. this day.
End user feedback incorporated into subsequent iterations of • SME co-designers and their feedback lead to tangible changes in the
solution. app appearance, language accessibility and functionality.
Solution validated with subject matter experts and/or existing • Organisational documents alone verified these requirements
Bartlett et al. Research Involvement and Engagement (2021) 7:32 Page 7 of 13

Table 2 Shifra HCD evaluation score (Continued)

Comments Points
literature.
Launch and share
Satisfied: (3 points) • Shifra held a product release in August 2017 which included co- 3
Product launched. designers, partners, and funders and presented preliminary findings
User testing completed to understand users’ experience and from the CBPR projects.
satisfaction.
User feedback incorporated into plans for future iterations.
Process or product results shared with program staff, co-designers,
and wider community.
Total score (max score of 12): 9

education level, and English proficiency. Interviews And as users … they take our notes and they try to
with each participating co-designer were consented to improve. And we share, really. We share as a big
in advance and then again in person when they were group, as a teamwork. And we share our ideas to-
completed at a location of the co-designer’s choosing. gether (Source: refugee end user).
All Arabic-speaking end users declined the use of an
interpreter. The survey was administered before the The surveys and interviews revealed important learn-
interview, with three exceptions: one SME declined to ings for the Shifra team when using HCD.
answer the survey, and two phone interviewees com-
pleted the surveys after the interview. The surveys
Communication
were analysed using Qualtrics online survey software.
Interviews were recorded and transcribed. Transcripts First was the issue of communication. Three of the
were de-identified, coded inductively with a hierarch- four end users mentioned that language was a barrier,
ical framework using NVivo 12 software by the exter- despite assistance from other community members
nal evaluator and a research assistant (NK), herself a who acted as interpreters during the co-design ses-
daughter of immigrants, who requested to work on sions and their own self-assessment as being profi-
this project because of its migrant women’s health cient in English. Role clarification was also a need
focus. reported by co-designers from all groups. An end
user expressed how she came to understand her role
Results in the co-design session:
Shifra scored 9 out of 12 for fidelity to the design
thinking process with a notable need for improvement After some time, I could realise what’s going on,
around the ideation stage (Table 2). Feedback ob- and understand what I had to do. It wasn’t clear in
tained through both the interviews and surveys veri- the beginning. Like when I went there, I didn’t
fied that the Shifra team did complete all the steps of know why I’m going there. I just know that I want
the design thinking approach. All survey respondents to be part of this, this is what I really wanted to do,
selected ‘agree’ or ‘strongly agree’ to statements asses- and yeah, after some time I could understand what’s
sing the collaborative nature of the group work and going on, but nobody explained me how.
that they felt safe sharing their opinions (Table 3 in (Source: refugee end user).
Appendix). Over 90% reported that they would par-
ticipate in another Shifra co-design session and that Fragmentation of involvement
they would recommend participation to a friend or There was fragmentation of co-designers’ experiences
family member. In the interviews, a feeling of enjoy- during the sessions. Many spoke about not under-
ment from participating in the co-design sessions standing the project’s entire process, wanting to be
clearly emerged. All 13 evaluation participants re- more involved but not receiving further invitations,
ported feeling valued, appreciated, and/or respected or the need to build on previous sessions with end
during the co-design sessions. users. While end users were involved at every step
of the co-design, different individuals participated at
… the students, they were very enthusiastic. They different points and in different ways. Very few end
take our notes and they try to discuss with us … user co-designers were a part of the process from
Bartlett et al. Research Involvement and Engagement (2021) 7:32 Page 8 of 13

beginning to end. This led to feelings of disconnec- Maybe we will use this program Shifra and
tion and confusion, as one co-designer put it: maybe never we will use this program because
we can search … what we need by Google …
First, I was really interested, but after some time but that program Shifra, it’s good for different
when I found like nobody’s calling you back, so I level of the people (refugee end user).
said no I don’t want to waste my time on this. But
it’s something really helpful and I really like the idea Increased representation from within the end-user
of helping new arrivals from refugee and migrant group should be considered for future HCD
backgrounds. endeavours.
Finally, a survey question assessed whether co-
SMEs who facilitated meetings between Shifra staff designers felt that there was enough time for relation-
and end users also felt the desire to be more involved ship building during the co-design sessions. End users
throughout the entire HCD project. One SME responded most negatively to the statement, with half
commented: stating that they ‘strongly disagree’ or ‘disagree’ that
there was enough time for this.
I think it would be good to have a follow up
consultation on working on actual usage … be- Discussion
cause we haven’t sort of touched base again This evaluation found that a co-design process was
with those women to say, have you used it? … successfully applied to the development of a web-
it was almost still in the design stage, and based app for refugee and migrant women in repro-
things hadn’t quite been finished. So, I think ductive health. This evaluation also yielded several
we could have a follow up that says, this is important recommendations for improving Shifra’s
the latest version of the product, let’s have a HCD approach moving forward, findings that can be
play around with it, what do you think now? I applied to other projects seeking to undertake an au-
think that would be really timely (Source: thentic community co-design process. First, with so
SME). many people of diverse backgrounds contributing to
the project, clear communication about roles and ex-
A third area for consideration in future co-design pectations is critical. More attention to facilitator
session is the importance of diversity and represen- training, identification of session goals, following up
tation within end-user groups. The SMEs and UX/ with consistent communication, and seeking end user
programmer generally agreed that there was suffi- and SME feedback would help to reduce future con-
cient end user representation at the co-design fusion [10]. Second, it is important to set realistic ex-
sessions: pectations and role clarifications with co-designers.
Design is a non-linear and creative process, which
We focused bringing on more and more people can inadvertently contribute to confusion about the
from the refugee and migrant community which is co-designer’s purpose and the project’s goals [31].
really good … I don’t think we had a shortage of Care should be taken in advance to explain this and
that diversity … in terms of cultural background it answer questions from participants not familiar with
was quite well represented (Source: UX/ the concept.
programmer). Third, it is important not to view all end users as
interchangeable [10]. UX/programmers and some
The end users themselves, however, felt that there SMEs saw refugees at all the meetings and viewed
were groups within the Arabic-speaking population that that as enough end user participation. End users how-
were unrepresented. Some of the suggested groups in- ever, felt there were other voices from their commu-
clude individuals who did not attend university, Arabic nity that needed engagement. For example, the fact
speakers with no or low English proficiency, middle and that all the end users spoke some English meant that
late middle age individuals, people with different levels co-design sessions could proceed without certified in-
of proficiency with mobile technology, and refugees who terpreters, but it also meant that the voices and expe-
had just arrived to Australia compared to refugees who riences of refugees with low English proficiency were
have been living in Australia for some time. The risks of missing. There is also the issue of inherent bias. As
only collaborating with end users who are university- mentioned, one refugee end user’s opinion was dis-
educated, recently resettled and English-proficient were missed by a designer until it was supported by an
summarised by one end user: SME. This was not tolerated by the Shifra team and
Bartlett et al. Research Involvement and Engagement (2021) 7:32 Page 9 of 13

the designer’s supervisor was notified to mentor the usefulness moving forward. Including operational defi-
designer in question. Ultimately though, this shows nitions of each of the three stages would clarify the
how HCD in and of itself can be flawed and active expectations, especially for anyone who is trying to
steps need to be taken to reduce, and where possible use the rubric to guide future HCD-driven projects.
remove, power structures that pervade everyday life. Additionally, some of the requirements were impos-
Sasha Costanza-Chock’s (2020) notes that “Design sible to assess using existing documentation alone.
justice asks whether the affordances of a designed ob- For example, the evaluator was unable to ascertain
ject or system disproportionally reduce opportunities whether end users felt respected from organisational
for already oppressed groups of people while enhan- documents alone and all three data sources were re-
cing the life opportunities of dominant groups, inde- quired to complete the rubric. As mentioned, future
pendently of whether designers intend this outcome.” papers will assess the influence or impact of the app
[32] Increasing the influence of end users needs a on user SRH literacy and determine the relative suc-
purposeful and planned approach, one that most pro- cess of the health intervention. Questions specific to
jects, including Shifra’s, needs to improve upon in the the user’s experience of the app may demonstrate
future. areas where refugee voices came through strongest
There will naturally be trade-offs in any public and others where their perspectives needed to be ele-
health project since limited resources are an unchan- vated more. Likewise, co-designers’ perceptions of
ging reality, but several process adjustments could whether they felt respected and valued during co-
address this issue. This evaluation demonstrates that design sessions should be included all future feedback
the practice of engaging end users across all em- forms, surveys and interview guides. This is especially
pathy, design, and prototyping stages and into prod- important in helping reduce implicit bias within the
uct development is possible [31]. Collaborating with design process that may be overlooked or go un-
multiple migrant and refugee advocate organisations acknowledged due to inherent power structures rein-
to use diverse sampling techniques will help to en- forced by unchecked privilege, however unintentional
gage a more representative sample in the future co- they may be.
design sessions. Using multiple data sources (document review, sur-
Finally, by setting aside adequate time to develop veys, and interviews) should have helped to verify an-
collaborative relationships amongst all co-design swers. Instead, each data source provided unique and
groups the HCD process is an opportunity to give isolated findings and there proved inconsistency be-
power and control back to the end user population tween interviews and the survey responses concerning
for whom one is designing the health intervention communication. For example, while the majority
[33]. Placing greater effort into building relationships (75%) of co-designers responded ‘agree’ or ‘strongly
as a part of the co-design session is especially import- agree’ to the survey statement about whether roles
ant with Shifra’s partner refugee communities [26]. and responsibilities had been clearly explained, many
When properly implemented, the intent of HCD is to anecdotes arose during the interviews around the fact
provide public health organisations a pathway to shar- that co-designers did not actually understand their
ing (and where applicable, handing over) power in role in the overall project or at specific co-design ses-
order to achieve true citizen participation and control sions. Similarly, one of the main themes of the semi-
[25]. Failure to apply HCD principles in an authentic structured interviews was how fragmented the co-
or purposeful way usually results in tokenism, and de- designers felt their involvement was though there was
velopment of solutions that are unsustainable [33]. It no way to verify this finding in the document review
can also “exacerbate social exclusion and destroy trust or with the surveys. One way to address this problem
systems” when done poorly [34]. A project cannot is by considering evaluation methods alongside HCD
utilise HCD without a power dynamic shift that en- planning meetings. Collecting co-designer surveys
sures the end user, not the UX designer, computer throughout the project can help staff understand their
programmer or community organisation, is in the pri- experiences ‘in the moment’ and offers an opportunity
mary decision-making role [11]. for a more agile response if needed, whilst also offer-
ing an opportunity to compare experiences and feed-
Future considerations back later following end of project co-designer
There are several important considerations when evaluations. Shifra did not have any documentation
planning for evaluation of any HCD-driven projects. regarding training materials on design thinking facili-
First, while this version of the rubric was helpful in tation or how facilitators had introduced the co-
gaining a deeper understanding of Shifra’s HCD ap- designers to various methodologies at different points
proach, several iterations will increase the tool’s in the app’s development, making it difficult to verify
Bartlett et al. Research Involvement and Engagement (2021) 7:32 Page 10 of 13

these findings. Future documentation on roles, design design thinking and co-design efforts in public health is
thinking goals and checking in with co-designers fre- challenging due to its abstract, creative, and iterative na-
quently would benefit all involved. ture and because results are specific to the local context.
There are no definitive guidelines providing specific pa-
Strengths and limitations rameters for assessing a HCD-driven project and these
This paper adds to the existing literature regarding terminologies are frequently interchanged despite their
the rigorous use of HCD in public health. This imple- differences in application and outcome. The methods
mentation evaluation provides an important guide to identified in this paper are a first attempt to benchmark
purposefully working with end user communities to this innovative approach and will need to be refined in
design better health interventions. Engaging in evalu- the future.
ation work increases the transparency of organisations
and helps them demonstrate their commitment to the Conclusion
HCD ethos. This paper and the rubric are helpful Improving refugees’ access to sexual and reproductive
tools for organisations attempting to evaluate their health is complex and multidimensional and requires
use of design thinking methods, and can assist them innovative and thoughtful problem solving. HCD is
to plan, prepare for, and execute successful co-design one way to address complex problems in, ideally, a
sessions. Additionally, the evaluation included mul- more ethical and effective way and it is how Shifra
tiple sources of data including surveys, interviews, chose to approach the development of its solution to
and records. While Shifra’s data collection methods this problem among Arabic-speaking refugees in Mel-
will be improved as a result of this evaluation, mul- bourne, Australia. The surveys and interviews revealed
tiple sources of data give a more full, rich, and accur- that end user, SME, and UX/programmer co-
ate picture of co-designers’ experiences and the designers enjoyed participating in the co-design ses-
methods themselves [35]. The richness of the data sions, felt respected and welcomed, and saw their
obtained is especially important since HCD is nebu- contributions reflected in the final product. Oppor-
lous by nature and there is a lack of demonstrated, tunities for growth include engaging a more diverse
rigorous evaluation [20, 36]. Finally, the utilisation of end user population and communicating expectations
an external assessment increased the objectivity of and results more clearly during and after the co-
this evaluation [37]. design sessions. A comprehensive process evaluation
There were several limitations to this evaluation study. benefits the field by providing an example of how to
First, the small sample size and convenience sampling of assess an organisation’s ability to follow all the HCD
co-designers introduced selection bias into the results. It steps based on individualised contexts and will ad-
is possible that the co-designers who did not respond to vance knowledge on the effectiveness of HCD in de-
a request to participate in the evaluation would have veloping solutions that are aligned with the needs of
provided different answers and perspectives. There were the target audience. It also provides an opportunity to
only four end users involved in this evaluation, limiting explore implicit bias and inherent power structures
the feedback and perspectives of those most important present in HCD methodologies that may be uninten-
to both the mission and process being undertaken in de- tional and may, therefore, go unchecked. This evalu-
veloping the Shifra app. Second, the evaluation took ation of Shifra’s HCD approach provides a helpful
place 9–16 months after most of the co-design sessions, and rigorous guide in reporting that may encourage
making results vulnerable to recall bias. Several inter- other organisations undertaking HCD work to evalu-
viewees mentioned at different points that the sessions ate their own implementation. Such organisations
had happened so long ago it took effort to remember should explore in advance, how they plan to evaluate
and answer the questions. Third, a yet to be validated not only the design steps but also co-designers’ per-
rubric guided part of the evaluation. Despite expert feed- ceptions around their role and the contributions they
back and iteration on this rubric before its application, made to the end product. Determining how to assess
there needs to be more use of the checklist to ensure satisfaction with both process and product needs
that it is both reliable and generalisable. Fourth, within thoughtful consideration to ensure co-designers and
the research team, only the research assistant (NK) had evaluators are reflecting and measuring the same out-
any personal connection to the migrant or refugee ex- come respectively. Finally, utilising mixed methodolo-
perience. Future research on migrant or refugee commu- gies has the potential to reveal inconsistent answers
nities must include funding to support people from across the different sources of data being examined
migrant and refugee backgrounds to play an active, and so care needs to be taken to ensure evaluative and
ideally lead, role in designing, conducting, analysing and probing questions in the semi-structured interview
reporting on said research. Finally, evaluating HCD, process, add clarity and reduce confusion.
Bartlett et al. Research Involvement and Engagement (2021) 7:32 Page 11 of 13

Appendix

Table 3 Participant Survey Feedback Form

At the co-design session … Strongly Disagree Neither Agree Strongly N/
disagree disagree nor agree A
agree
1 The group work felt collaborative
2 My contribution was valued
3 People attended who do not usually have representation (i.e. people with
refugee and migrant backgrounds, people with diverse sexual orientation, people
experiencing health accessibility issues)
4 The voices of refugee and/or migrant end users were heard
5 The roles and responsibilities of my participation were clearly defined
6 I understood both the processes and the language used
7 My time participating was compensated appropriately
8 There was a commitment by Shifra staff to develop consensus on what the end
product should include
9 All co-designers were kept informed of any changes
10 I felt respected by Shifra staff
11 I felt respected by all co-design partners
12 Shifra staff offered me an opportunity for skill development and capability
building
13 Shifra staff offered me co-design training and resources
14 There were strategies to involve people with different communication needs
15 There was enough time to allow relationship building
16 Shifra staff made attempts to reduce any power imbalance (e.g. between health
professionals and refugees)
17 I felt safe sharing my opinions
18 Refugee and/or migrant end users helped shape the common agenda
19 I would participate in another co-design session with Shifra staff
20 I would encourage others to participate in a co-design session with Shifra staff
21 I see my ideas and contributions reflected in the final Shifra website/app
Comments/Feedback:

Table 4 Semi-structured interview guide

1 Could you elaborate on how you felt your contribution to the co-design sessions was (or was not) recognised and valued?
2 Were there stakeholders who were not represented at the co-design sessions? If so, who else should have been invited?
3 Was there any confusion regarding your role, the processes or any language barriers at the co-design sessions that could have been made sim-
pler/easier to understand? If so, what were these?
4 Were there ways that the Shifra team or other participants made you feel respected or disrespected/unwelcome during this co-design
experience?
5 Do you feel that any power imbalances between co-design participants were addressed e.g. do you feel that your opinion mattered and that it
was safe for you to communicate your thoughts and experiences if you wanted to?
6 How could this co-design experience be improved in the future?
7 Do you have any other feedback you wish to share?
Bartlett et al. Research Involvement and Engagement (2021) 7:32 Page 12 of 13

Abbreviations 3. Austin J, Guy S, Lee-Jones L, McGinn T, Schlecht J. Reproductive health: a

CBPR: Community based participatory research; HCD: Human centred design; right for refugees and internally displaced persons. Reprod Health Matters.
MCWH: Multicultural Centre for Women’s Health; SRH: Sexual and 2008;16(31):10–21. https://doi.org/10.1016/S0968-8080(08)31351-2.
reproductive health; SME: Subject matter experts; UNC: University of North 4. Casey SE, Chynoweth SK, Cornier N, Gallagher MC, Wheeler EE. Progress and
Carolina; UX: User experience gaps in reproductive health services in three humanitarian settings: mixed-
methods case studies. Confl Heal. 2015;9(Suppl 1 Taking Stock of
Acknowledgements Reproductive Health in Humanitarian):S3.
The authors recognise and thank all co-designers involved in this project and 5. Habibov N, Fan L. The effect of maternal healthcare on the probability of
its evaluation. We are especially grateful to those who have refugee experi- child survival in Azerbaijan. Biomed Res Int. 2014;2014:1–6. https://doi.org/1
ences for taking the time to be involved and all attempts will be made to 0.1155/2014/317052.
disseminate these findings to them. 6. Cohen S. The broad benefits of investing in sexual and reproductive health.
The authors also recognise Douglas Holmes, Hannah Korsmeyer and Sara La New York: Guttmacher Institute; 2004.
Lone for their contribution to the evaluation plan and rubric design of this 7. Barot S. Sexual and reproductive health and rights are key to global
paper. development: the case for ramping up investment. New York: Guttmacher
Institute; 2015.
8. Roy MJ, Donaldson C, Baker R, Kerr S. The potential of social enterprise to
Authors’ contributions
enhance health and well-being: a model and systematic review. Soc Sci
JSS undertook the initial research. NK assisted with data analysis. RB drafted
Med. 2014;123:182–93. https://doi.org/10.1016/j.socscimed.2014.07.031.
this paper based on the research and managed all edits and version control.
JAB and TR reviewed the paper and offered feedback. RR provided 9. Iipito Mendonca HNKaVZ, I. Youth empowerment: the role of service design
and mobile technology in accessing reproductive health information. Paper
supervisory guidance and offered feedback. All authors read and approved
presented at: Proceedings of the 13th Participatory Design Conference:
the final manuscript.
Short Papers, Industry Cases, Workshop Descriptions, Doctoral Consortium
papers, and Keynote abstracts 2014.
Authors’ information 10. IDEO.org. The field guide to human centered design. In:2015: http://www.
Not Applicable. designkit.org/resources/1. Accessed 14 Aug 2019.
11. Giacomin J. What is human Centred design? Des J. 2015;17(4):606–23.
Funding 12. Kindig S. Solving problems with human-centred design: Gates Cambridge;
Not applicable. 2015.
13. Brown TaW J. Design thinking for social innovation. Standford Soc Innov
Availability of data and materials Rev. 2010;8(1):31–5.
The data that support the findings of this study are available from author, 14. Sanders EBN, Stappers PJ. Co-creation and the new landscapes of design.
JSS, but restrictions apply to the availability of these data, which were used CoDesign. 2008;4(1):5–18. https://doi.org/10.1080/15710880701875068.
under license for the current study, and so are not publicly available. Data 15. Koehly LM, Morris BA, Skapinsky K, Goergen A, Ludden A. Evaluation of the
are however available from the authors upon reasonable request and with families SHARE workbook: an educational tool outlining disease risk and
permission of author, JSS. healthy guidelines to reduce risk of heart disease, diabetes, breast cancer
and colorectal cancer. BMC Public Health. 2015;15(1):1120. https://doi.org/1
0.1186/s12889-015-2483-x.
Declarations
16. Parmar V. Disseminating maternal health information to rural women: a user
Ethics approval and consent to participate centered design framework. Paper presented at: AMIA Symposium 2010.
Monash Health Project ID number 13811: Evaluating the process and 17. Catalani C, Green E, Owiti P, Keny A, Diero L, Yeung A, et al. A clinical
product of Shifra’s mHealth intervention. decision support system for integrating tuberculosis and HIV care in Kenya:
The University of North Carolina, the Institutional Review Board from UNC- a human-centered design approach. PLoS One. 2014;9(8):e103205. https://
Chapel Hill also reviewed the evaluation plan and determined no additional doi.org/10.1371/journal.pone.0103205.
ethics approval requirements for this project (UNC Study #18–1449). 18. Itad. Evaluation of The Hewlett Foundation’s strategy to apply human-
centered design to improve family planning and reproductivehealth
services in Sub-Saharan Africa: Hewlett Foundation; 2018.
Consent for publication 19. Steen M, Manschot M, De Koning N. Benefits of co-design in service design
All authors consent to publication. projects. Int J Des. 2011;5(2):53–60.
20. Bazzano AN, Martin J, Hicks E, Faughnan M, Murphy L. Human-centred
Competing interests design in global health: a scoping review of applications and contexts. PLoS
Conflict of Interest: First author (RB) founded the Shifra app, however an One. 2017;12(11):e0186744. https://doi.org/10.1371/journal.pone.0186744.
external evaluator (JSS) was used to limit bias in evaluating Shifra’s HCD 21. Goodwill M. A social designer’s field guide to power literacy. https://crea
process otherwise no authors report competing interests. tivecommons.org/licenses/by-nc/4.0/2020. Accessed 23 Mar 2021.
22. Saunders RP, Evans MH, Joshi P. Developing a process-evaluation plan for
Author details assessing health promotion program implementation: a how-to guide.
1
Monash Centre for Health Research and Implementation – MCHRI, School of Health Promot Pract. 2005;6(2):134–47. https://doi.org/10.1177/15248399042
Public Health and Preventive Medicine, Monash University, Melbourne, 73387.
Australia. 2Public Health Leadership Program, Gillings School of Global Public 23. Palmer VJ, Piper D, Richard L, et al. Balancing opposing forces—a nested
Health, University of North Carolina at Chapel Hill, Chapel Hill, USA. 3School process evaluation study protocol for a stepped wedge designed cluster
of Nursing, Midwifery & Indigenous Health, Faculty of Science, Charles Sturt randomized controlled trial of an experience based codesign intervention.
University, Bathurst, Australia. Int J Qual Methods. 2016;15(1):1609406916672216.
24. Mummah SA, Robinson TN, King AC, Gardner CD, Sutton S. IDEAS (integrate,
Received: 16 February 2021 Accepted: 15 April 2021 design, assess, and share): a framework and toolkit of strategies for the
development of more effective digital interventions to change health
behavior. J Med Internet Res. 2016;18(12):e317. https://doi.org/10.2196/jmir.
References 5927.
1. Negussie F. Access to health care: refugee women’s perspective: a story 25. Hendricks S, Conrad N, Douglas TS, Mutsvangwa T. A modified stakeholder
map. 2018; https://unc.maps.arcgis.com/apps/MapJournal/index.html?a participation assessment framework for design thinking in health
ppid=4acccf8ca7504d5e95025069a7353690. Accessed 1 May 2019. innovation. Healthc (Amst). 2018;6(3):191–6. https://doi.org/10.1016/j.hjdsi.2
2. Sexual and reproductive health data report. Multicultural Centre for 018.06.003.
Women’s Health; 2016. 26. Shifra. About us. 2018; www.shifra.app. Accessed 14 Aug 2019.
Bartlett et al. Research Involvement and Engagement (2021) 7:32 Page 13 of 13

27. Hussain S. Toes that look like toes: Cambodian children’s perspectives on
prosthetic legs. Qual Health Res. 2011;21(10):1427–40. https://doi.org/10.11
77/1049732311411058.
28. d.School S. An introduction to design thinking process guide. https://
dschoolold.stanford.edu/sandbox/groups/designresources/wiki/36873/atta
chments/74b3d/ModeGuideBOOTCAMP2010L.pdf. Accessed 14 Aug 2019.
29. Marcolino MS, Oliveira JAQ, D'Agostino M, Ribeiro AL, Alkmim MBM, Novillo-
Ortiz D. The impact of mHealth interventions: systematic review of
systematic reviews. JMIR Mhealth Uhealth. 2018;6(1):e23. https://doi.org/1
0.2196/mhealth.8873.
30. Bloomfield GS, Vedanthan R, Vasudevan L, Kithei A, Were M, Velazquez EJ.
Mobile health for non-communicable diseases in Sub-Saharan Africa: a
systematic review of the literature and strategic framework for research.
Glob Health. 2014;10(1):49. https://doi.org/10.1186/1744-8603-10-49.
31. Brown TaK B. Change by design: how design thinking transforms
organizations and inspires innovation: Harper Collins eBooks; 2009.
32. Costanza-Chock S. Design justice: community-led practices to build the
worlds we need: MIT Press; 2020. https://doi.org/10.7551/mitpress/12255.
001.0001.
33. Arnstein SR. A ladder of citizen participation. J Am Inst Plann. 1969;35(4):
216–24. https://doi.org/10.1080/01944366908977225.
34. Burkett I. An introduction to co-design: Centre for social impact; 2012.
35. Brownson RC, Fielding JE, Maylahn CM. Evidence-based public health: a
fundamental concept for public health practice. Annu Rev Public Health.
2009;30(1):175–201. https://doi.org/10.1146/annurev.publhealth.031308.1
00134.
36. Donetto S, Tsianakas V, Robert G. Using experience-based Co-design (EBCD)
to improve the quality of healthcare: mapping where we are now and
establishing future directions. London: National Nursing Research Unit; 2014.
37. Pattyn, VaB M. Outsource versus In-house? An identification of
organizational conditions influencing the choice for internal or external
evaluators. Can J Program Eval. 2013;28(2):43–57.

Publisher’s Note
Springer Nature remains neutral with regard to jurisdictional claims in
published maps and institutional affiliations.