Digital Interventions for Health Systems Strengthening

WHO guideline
recommendations
on digital
interventions
for health system
strengthening

page ii
WHO guideline: recommendations on digital interventions for health system strengthening
ISBN 978-92-4-155050-5
© World Health Organization 2019
Some rights reserved. This work is available under the Creative Commons Attribution-NonCommercial-ShareAlike
3.0 IGO licence (CC BY-NC-SA 3.0 IGO; https://creativecommons.org/licenses/by-nc-sa/3.0/igo).
Under the terms of this licence, you may copy, redistribute and adapt the work for non-commercial purposes, provided
the work is appropriately cited, as indicated below. In any use of this work, there should be no suggestion that WHO
endorses any specific organization, products or services. The use of the WHO logo is not permitted. If you adapt the work,
then you must license your work under the same or equivalent Creative Commons licence. If you create a translation of
this work, you should add the following disclaimer along with the suggested citation: “This translation was not created
by the World Health Organization (WHO). WHO is not responsible for the content or accuracy of this translation. The
original English edition shall be the binding and authentic edition”.
Any mediation relating to disputes arising under the licence shall be conducted in accordance with the mediation rules
of the World Intellectual Property Organization.
Suggested citation. WHO guideline: recommendations on digital interventions for health system strengthening.
Geneva: World Health Organization; 2019. Licence: CC BY-NC-SA 3.0 IGO.
Cataloguing-in-Publication (CIP) data. CIP data are available at http://apps.who.int/iris.
Sales, rights and licensing. To purchase WHO publications, see http://apps.who.int/bookorders. To submit requests for
commercial use and queries on rights and licensing, see http://www.who.int/about/licensing.
Third-party materials. If you wish to reuse material from this work that is attributed to a third party, such as tables,
figures or images, it is your responsibility to determine whether permission is needed for that reuse and to obtain
permission from the copyright holder. The risk of claims resulting from infringement of any third-party-owned
component in the work rests solely with the user.
General disclaimers. The designations employed and the presentation of the material in this publication do not imply
the expression of any opinion whatsoever on the part of WHO concerning the legal status of any country, territory, city or
area or of its authorities, or concerning the delimitation of its frontiers or boundaries. Dotted and dashed lines on maps
represent approximate border lines for which there may not yet be full agreement.
The mention of specific companies or of certain manufacturers’ products does not imply that they are endorsed
or recommended by WHO in preference to others of a similar nature that are not mentioned. Errors and omissions
excepted, the names of proprietary products are distinguished by initial capital letters.
All reasonable precautions have been taken by WHO to verify the information contained in this publication. However,
the published material is being distributed without warranty of any kind, either expressed or implied. The responsibility
for the interpretation and use of the material lies with the reader. In no event shall WHO be liable for damages arising
from its use.

page iiiWHO guideline recommendations on digital interventions for health system strengthening
Contents
Foreword . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . v
Acknowledgements . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . vi
Abbreviations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . viii
Executive summary ix
Background . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ix
Objectives of the guideline . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xi
Target audience . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .xii
Implementation context . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xiii
Methods . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xiv
Summary of recommendations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xix
1. Introduction 1
1.1 Background . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .1
1.2 Role of digital health in health system strengthening
and universal health coverage . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2
1.3 Objectives of this guideline . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .7
1.4 Target audience . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8
1.5 Linkages with other WHO resources . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9
1.6 Context and the enabling environment . . . . . . . . . . . . . . . . . . . . . . . . . . 10
1.7 Linkages to broader digital health architecture . . . . . . . . . . . . . . . . . . . . . 11
1.8 Living guidelines approach . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12
2. Methods 13
2.1 Identification of priority questions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13
2.2 Scoping of interventions and outcomes . . . . . . . . . . . . . . . . . . . . . . . . . . 14
2.3 Evidence retrieval . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21
2.4 Assessment, synthesis and grading of the evidence . . . . . . . . . . . . . . . . . . . 23
2.5 Roles and responsibilities of contributors . . . . . . . . . . . . . . . . . . . . . . . . . 26
2.6 Consolidation of evidence . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28
2.7 Decision-making and formulation of recommendations . . . . . . . . . . . . . . . . 29
2.8 Document preparation and peer review . . . . . . . . . . . . . . . . . . . . . . . . . . 31
2.9 Presentation of the guideline . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32

page iv
3. Evidence and recommendations 33
3.1 Cross-cutting acceptability and feasibility findings . . . . . . . . . . . . . . . . . . .33
3.2 Accountability coverage: birth and death notification via mobile devices . . . . . . 38
3.3 Availability of commodities: stock notification and commodity . . . . . . . . . . . . 44
3.4 Accessibility of health facilities and human resources for health:
client-to-provider telemedicine . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 48
3.6 Accessibility of health facilities and human resources for health:
provider-to-provider telemedicine . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 52
3.7 Contact and continuous coverage: targeted client communication
for behaviour change related to sexual, reproductive, maternal, newborn,
child and adolescent health . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 56
3.8 Effective coverage: Health worker decision support . . . . . . . . . . . . . . . . . . . 63
3.9 Multiple points of coverage: digital tracking of clients’ health status and
services combined with decision-support and targeted client communication . . . 67
3.10 Effective coverage: digital provision of training and educational content
to health workers via mobile devices/mobile learning . . . . . . . . . . . . . . . . . . 73
4. Implementation considerations 77
4.1 Linking the recommendations across the health system . . . . . . . . . . . . . . . . 77
4.2 Implementation componets . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 79
4.3 Overarching implementation considerations . . . . . . . . . . . . . . . . . . . . . . . 81
5. Future research 86
5.1 Overarching research gaps . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 86
5.2 Considerations for the design of future evaluations . . . . . . . . . . . . . . . . . . . 88
6. Disseminating and updating the guideline 89
6.1 Dissemination and implementation of the guideline . . . . . . . . . . . . . . . . . . 89
6.2 Updates and living guidelines approach . . . . . . . . . . . . . . . . . . . . . . . . . . 90
Glossary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 91
References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 92
Annexes 96
Annex 1: Classification of digital health interventions and health system challenges . . 97
Annex 2: Priority questions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 99
Annex 3: Contributors . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 107
Annex 4. Summary of declarations of interest . . . . . . . . . . . . . . . . . . . . . . . . . 112
Annex 5: Evidence maps and illustrative research questions . . . . . . . . . . . . . . . . 114

page vWHO guideline recommendations on digital interventions for health system strengthening
Foreword
Human health has only ever improved because of advances in technology. From the
development of modern sanitation to the advent of penicillin, anesthesia, vaccines and
magnetic resonance imaging, science, research and technology have always been key
drivers of better health.
It’s no different today. Advances in technology are continuing to push back the
boundaries of disease. Digital technologies enable us to test for diabetes, HIV and malaria
on the spot, instead of sending samples off to a laboratory. 3-D printing is revolutionizing
the manufacture of medical devices, orthotics and prosthetics. Telemedicine, remote care
and mobile health are helping us transform health by delivering care in people’s homes and strengthening care
in health facilities. Artificial intelligence is being used to give paraplegic patients improved mobility, to manage
road traffic and to develop new medicines. Machine learning is helping us to predict outbreaks and optimize
health services.
Propelled by the global ubiquity of mobile phones, digital technologies have also changed the way we manage our
own health.Today we have more health information – and misinformation – at our fingertips than any generation
in history. Before we ever sit down in a doctor’s office, most of us have Googled our symptoms and diagnosed
ourselves – perhaps inaccurately. Similarly, digital technologies are being used to improve the training and
performance of health workers, and to address a diversity of persistent weaknesses in health systems.
Harnessing the power of digital technologies is essential for achieving the Sustainable Development Goals,
including universal health coverage and the other “triple billion” targets in WHO’s 13th General Programme of
Work. Such technologies are no longer a luxury; they are a necessity.
A key challenge is to ensure that all people enjoy the benefits of digital technologies for everyone. We must
make sure that innovation and technology helps to reduce the inequities in our world, instead of becoming
another reason people are left behind. Countries must be guided by evidence to establish sustainable
harmonized digital systems, not seduced by every new gadget.
That’s what this guideline is all about.
At the Seventy-First World Health Assembly, WHO’s Member States asked us to develop a global strategy on
digital health. This first WHO guideline establishes recommendations on digital interventions for health system
strengthening and synthesizes the evidence for the most important and effective digital technologies, primarily
those which can be accessed on mobile devices.
The nature of digital technologies is that they are evolving rapidly; so will this guideline. As new technologies
emerge, new evidence will be used to refine and expand on these recommendations. WHO is significantly
enhancing its work in digital health to ensure we provide our Member States with the most up-to-date
evidence and advice to enable countries to make the smartest investments and achieve the biggest gains in
health. Ultimately, digital technologies are not ends in themselves; they are vital tools to promote health, keep
the world safe, and serve the vulnerable.
Dr Tedros Adhanom Ghebreyesus
Director-General, World Health Organization

page vi
Acknowledgements
The World Health Organization (WHO) is grateful for the contributions that many individuals and
organizations have made over several years to the development of this guideline.
This guideline was coordinated by Garrett Mehl, Lale Say and Tigest Tamrat of the WHO
Department of Reproductive Health and Research, in collaboration with departments across WHO.
Marita Sporstøl Fønhus, Claire Glenton and Simon Lewin from the Norwegian Institute of Public
Health provided methodological support as members of the technical team and prepared the
evidence-to-decision frameworks.
The following WHO staff and consultants contributed to the guideline development process at
various stages (in alphabetical order): Onyema Ajuebor, Virginia Arnold, Ian Askew, Venkatraman
Chandra-Mouli, Doris Chou, Giorgio Cometto, Theresa Diaz, Dennis Falzon, Mary Eluned Gaffield,
Jan Grevendonk, Lianne Gonsalves, Per Hasvold, Lisa Hedman, Michelle Hindin, Cheryl Johnson,
Surabhi Joshi, Maki Kajiwara, Edward Kelley, Etienne Langlois, Doris Ma Fat, Martin Meremikwi,
Manjulaa Narasimhan, Olufemi Oladapo, Kathryn O’Neill, Maeghan Orton, Sameer Pujari, Knut
Staring, Anneke Schmider, Hazim Timimi, Özge Tunçalp, Wilson Were, Teodora Wi and Diana Zandi.
The following WHO regional advisers were also consulted: Navreet Bhataal, Jun Gao, Rodolfo
Gomez, Mark Landry, Derrick Muneene, Mohammed Hassan Nour, David Novillo and Leopold
Ouedraogo.
WHO extends sincere thanks to members of the guideline development group (GDG): Smisha
Agarwal, Pascale Allotey, Fazilah Shaik Allaudin, Subhash Chandir, Shrey Desai, Vajira H.W.
Dissanayake, Frederik Frøen, Skye Gilbert, Rajendra Gupta, Robert Istepanian, Oommen John,
Karin Källander, Gibson Kibiki, Yunkap Kwankam, Alain Labrique, Amnesty LeFevre, Alvin Marcelo,
Patricia Mechael, Marc Mitchell, Thomas Odeny, Hermen Ormel, Olasupo Oyedepo, Caroline Perrin,
Kingsley Pereko, Anshruta Raodeo, Chris Seebregts, Lavanya Vasudevan and Hoda Wahba.
WHO is especially grateful to Alain Labrique for serving as GDG chair in both the scoping and final
consultations, and to Pascale Allotey for co-chairing the final GDG meeting.
WHO appreciates the feedback provided by James BonTempo, Carolyn Florey, Kelly L’Engle,
Liz Peloso, Dykki Settle and Chaitali Sinha during the scoping consultation of the guideline
development process.
WHO is grateful to the following colleagues from partnering United Nations organizations, who
contributed to the guideline process during the technical consultations: Sean Blaschke, Hani
Eskandar, Maria Muniz, Remy Mwamba, Vincent Turmine and Sylvia Wong.
WHO thanks Nicholas Henschke and Nicola Maayan for coordinating the commissioned systematic
reviews and the authors of the Cochrane systematic reviews used in this guideline: Smisha Agarwal,
Heather Ames, Josip Car, Caroline Free, Daniela Gonçalves Bradley, Priya Lall, Willem Odendaal,
Melissa Palmer, Rebecca Rees, Sasha Shepperd, Lorainne Tudor Car and Lavanya Vasudevan.

page viiWHO guideline recommendations on digital interventions for health system strengthening
WHO extends its gratitude to the following members of the external review group for their peer
review of this guideline: Patricia Garcia, Teng Liaw, Steve Ollis, Xenophon Santas and Maxine
Whittaker.
WHO acknowledges the following observers, who represented various organizations: David Heard,
Carl Leitner, Ingvil Von Mehren Saeterdal, Merrick Schaefer, Adele Waugaman and William Weiss.
Special thanks to Susan Norris and the broader WHO Guidelines Review Committee.
WHO thanks Rebecca Richards-Diop and Jessica Stone-Weaver from RRD Design for the creative
direction and design.
This work was funded by the Department for International Development (DFID), the Norwegian
Agency for Development Cooperation (Norad), United States Agency for International
Development (USAID), and the UNDP-UNFPA-UNICEF-WHO-World Bank Special Programme of
Research, Development and Research Training in Human Reproduction (HRP), a co-sponsored
programme executed by WHO. The views of the funding bodies have not influenced the content of
this guideline.
Editing and proofreading: Green Ink, United Kingdom (greenink.co.uk)
Design and layout: RRD Design LLC (rrddesign.co)

page viii
Abbreviations
AeHIN Asia eHealth Information Network
CERQual confidence in the evidence from reviews
of qualitative research
CHW community health worker
CRVS civil registration and vital statistics
DHA digital health atlas
EMTCT elimination of mother-to-child
transmission
GDG guideline development group
GRADE Grading of Recommendations
Assessment, Development and
Evaluation
ICT information and communications
technology
IHE Integrating the Healthcare Enterprise
ITS interrupted time series
ITU International Telecommunication Union
LMIS logistics management information
system
mHealth mobile health
mLearning mobile learning
NIPH Norwegian Institute of Public Health
NRS non-randomized studies
OpenHIE Open Health Information Exchange
PICO population (P), intervention (I),
comparator (C), outcome (O)
RCT randomized controlled trial
RHR reproductive health and research (WHO
department)
SDG Sustainable Development Goal
SDS service delivery and safety (WHO
department)
SMS short message service
SRH sexual and reproductive health
SRMNCAH sexual, reproductive, maternal, newborn,
child and adolescent health
TB tuberculosis
TCC targeted client communication
UHC universal health coverage
UNDP United Nations Development Programme
UNFPA United Nations Population Fund
UNICEF United Nations Children’s Fund
USAID United States Agency for International
Development
USSD unstructured supplementary service data
WHO World Health Organization

page ixWHO guideline recommendations on digital interventions for health system strengthening
Executive summary
Background
Digital health, or the use of digital technologies for health, has become a salient field of practice
for employing routine and innovative forms of information and communications technology
(ICT) to address health needs. The term digital health is rooted in eHealth, which is defined as
“the use of information and communications technology in support of health and health-related
fields”. Mobile health (mHealth) is a subset of eHealth and is defined as “the use of mobile wireless
technologies for health”. More recently, the term digital health was introduced as “a broad umbrella
term encompassing eHealth (which includes mHealth), as well as emerging areas, such as the use
of advanced computing sciences in ‘big data’, genomics and artificial intelligence”.
The World Health Assembly Resolution on Digital Health unanimously approved by WHO Member
States in May 2018 demonstrated a collective recognition of the value of digital technologies to
contribute to advancing universal health coverage (UHC) and other health aims of the Sustainable
Development Goals (SDGs). This resolution urged ministries of health “to assess their use of
digital technologies for health […] and to prioritize, as appropriate, the development, evaluation,
implementation, scale-up and greater use of digital technologies,... Furthermore, it tasked WHO
with providing normative guidance in digital health, including through the promotion of evidence-
based digital health interventions.
Amid the heightened interest, digital health has also been characterized by implementations
rolled out in the absence of a careful examination of the evidence base on benefits and harms. The
enthusiasm for digital health has also driven a proliferation of short-lived implementations and
an overwhelming diversity of digital tools, with a limited understanding of their impact on health
systems and people’s well-being. This concern was highlighted most notably in the consensus
statement of the WHO Bellagio eHealth Evaluation Group, which opened by stating: “To improve
health and reduce health inequalities, rigorous evaluation of eHealth is necessary to generate
evidence and promote the appropriate integration and use of technologies.” While recognizing the
innovative role that digital technologies can play in strengthening the health system, there is an
equally important need to evaluate their contributing effects and ensure that such investments do
not inappropriately divert resources from alternative, non-digital approaches.

page x
Role of digital health in health system
strengthening and Universal Health Coverage
The goal of UHC is to ensure the quality, accessibility and affordability of health services. However,
shortfalls remain in ensuring access to all who need health services and in ensuring that they are
delivered with the intended quality without causing financial hardship to the people accessing
them. The Tanahashi framework published by WHO in 1978 provides a time-tested model for
understanding health system performance gaps and how they prevent the intended coverage,
quality and affordability of health services. This cascading model illustrates how health systems
lose performance because of challenges at successive levels, each dependent on the previous level.
Health system challenges – such as geographical inaccessibility, low demand for services, delayed
provision of care, low adherence to clinical protocols and costs to individuals/patients – contribute
to accumulated losses in health system performance. These shortfalls limit the ability to close the
gaps in coverage, quality and affordability, and undermine the potential to achieve UHC.
Figure 1 Layers of UHC achievement affected by health system performance
This adapted Tanahashi model illustrates that each health system performance layer builds
on the components below it but also falls short (dotted lines) of the optimal, desired level
(Figure 1). Digital health interventions could contribute to efforts to address challenges that limit
achievement of that health system goal.
Source: adapted from Tanahashi, 1978.

page xiWHO guideline recommendations on digital interventions for health system strengthening
Digital technologies provide concrete opportunities to tackle health system challenges, and
thereby offer the potential to enhance the coverage and quality of health practices and services.
Digital health interventions may be used, for example, to facilitate targeted communications
to individuals in order to generate demand and broaden contact coverage. Digital health
interventions may also be targeted to health workers to give them more immediate access
to clinical protocols through, for example, decision-support mechanisms or telemedicine
consultations with other health workers. The range of ways digital technologies can be used to
support the needs of health systems is wide, and these technologies continue to evolve due to the
inherently dynamic nature of the field. A starting point for categorizing the different ways that
digital technologies are being used to overcome defined health system challenges is provided by
WHO’s Classification of digital health interventions v1.0.
A digital health intervention is defined here as a discrete functionality of digital technology that is
applied to achieve health objectives and is implemented within digital health applications and ICT
systems, including communication channels such as text messages.
Objectives of the guideline
The key aim of this guideline is to present recommendations based on a critical evaluation of
the evidence on emerging digital health interventions that are contributing to health system
improvements, based on an assessment of the benefits, harms, acceptability, feasibility,
resource use and equity considerations. For the purposes of this version of the guideline, the
recommendations examine the extent to which digital health interventions, primarily available
via mobile devices, are able to address health system challenges along the pathway to UHC. By
reviewing the evidence of different digital interventions against comparative options, as well as
assessing the risks, this guideline aims to equip health policy-makers and other stakeholders with
recommendations and implementation considerations for making informed investments into
digital health interventions.
This guideline urges readers to recognize that digital health interventions are not a substitute
for functioning health systems, and that there are significant limitations to what digital health
is able to address. Digital health interventions should complement and enhance health system
functions through mechanisms such as accelerated exchange of information, but will not
replace the fundamental components needed by health systems such as the health workforce,
financing, leadership and governance, and access to essential medicines. An understanding of
which health system challenges can realistically be addressed by digital technologies, along with
an assessment of the ecosystem’s ability to absorb such digital interventions, is thus needed to
inform investments in digital health. Additionally, the adoption of the recommendations in this
guideline should not exclude or jeopardize the provision of quality non-digital services in places
where there is no access to the digital technologies or they are not acceptable or affordable for
target communities.

page xii
The recommendations in this guideline represent a subset of prioritized digital health
interventions accessible via mobile devices, and this guideline will gradually include a broader set
of emerging digital health interventions over subsequent versions. This includes recommendations
on the following topics:
ȺȺ birth notification via mobile devices
ȺȺ death notification via mobile devices
ȺȺ stock notification and commodity management via mobile devices
ȺȺ client1
-to-provider telemedicine
ȺȺ provider-to-provider telemedicine
ȺȺ targeted client communication via mobile devices
ȺȺ digital tracking of patients’/clients’ health status and services via mobile devices
ȺȺ health worker decision support via mobile devices
ȺȺ provision of training and educational content to health workers via mobile devices (mobile
learning-mLearning)
Target audience
The primary target audiences for this guideline are decision-makers in ministries of health,
public health practitioners and other stakeholders who will benefit from an understanding of
which digital health interventions have an evidence base to address health system needs. This
guideline may also prove beneficial to organizations that invest resources into digital health as
implementation and development partners. This document aims to strengthen evidence-based
decision-making on digital approaches by governments and partner institutions, encouraging the
mainstreaming and institutionalization of effective digital interventions.
1 Although WHO’s Classification of digital health interventions v1.0 uses the term “client”, the terms “individual” and “patient” may be used
interchangeably, where appropriate.
The systematic reviews included accessibility via mobile devices to ensure that these digital
interventions are applicable in low resource settings where extensive computerized systems
may not be available or feasible. However, the recommended interventions can be deployed
through any digital device, including stationary devices, such as desktop computers, and does
not preclude them from being used on non-mobile digital devices.

page xiiiWHO guideline recommendations on digital interventions for health system strengthening
Foundational Layer: ICT and Enabling Environment
LEADERSHIP & GOVERNANCE
STRATEGY &
INVESTMENT
SERVICES &
APPLICATIONS
LEGISLATION,
POLICY&
COMPLIANCE
WORKFORCE
STANDARDS &
INTEROPERABILITY
INFRASTRUCTURE
Health Content
Information that is aligned with
recommended health practices
or validated health content
Digital Health
Interventions
A discrete function of digital
technology to achieve health
sector objectives
Digital Applications
ICT systems and communication
channels that facilitate delivery
of the digital interventions and
health content
+ +
Implementation context
Digital health has the potential to help address problems such as distance and access, but still
shares many of the underlying challenges faced by health system interventions in general,
including poor management, insufficient training, infrastructural limitations, and poor access to
equipment and supplies. These considerations need to be addressed in addition to the specific
implementation requirements introduced by digital health.
Digital health interventions are applied within a country context and a health system, and their
implementation is made possible by a number of factors including: (i) the health domain area
and associated content; (ii) the digital intervention or functionality provided; (iii) the software
and communication channels for delivering the digital health intervention; and mediated by
(iv) a foundational layer of the ICT and the enabling environment (see Figure 2). Furthermore,
these components need to be made appropriate to the local context and ensure effective
implementation through reflection on the behaviour and organizational changes that would
also be required. Lastly, digital health interventions are intended to fit into an overall digital
health architecture. While the unit of analysis for this guideline focuses on the value of specific
digital interventions, there is an equally important need to support a cohesive approach to
implementation, in which different digital interventions can leverage one another, as opposed to
operating as isolated initiatives.
Figure 2 Components contributing to digital health implementations

page xiv
As the context may drive the eventual impact of the digital health interventions, the broader
health system and enabling environment become especially critical. There is considerable value
in assessing the ecosystem in a given context or country, in reviewing health system needs and
tempering expectations based on the ICT and enabling environment available within a setting. In
the absence of a robust enabling environment, there is the risk of a proliferation of unconnected
systems and a severe impact on the effectiveness and sustainability of the health intervention.
Methods
The development of this guideline followed the methods described in the second edition of
the WHO handbook for guideline development. This institution-wide process at WHO entailed
the identification of critical questions and outcomes, retrieval of the evidence, assessment
and synthesis of that evidence, the formulation of recommendations, and planning for the
implementation, dissemination, impact evaluation and updating of the guideline.
The guideline development process also included two rounds of online surveys and three in-person
consultations. These consultations included (i) an advisory meeting in February 2016 to establish
the goal of the guideline in light of other WHO resources and to determine underlying framework;
(ii) a scoping meeting in September 2016 to prioritize and draft the critical questions and
outcomes; and (iii) a final meeting in June 2018 to review the synthesized evidence and formulate
recommendations. Online surveys were used before and after the September scoping meeting to
inform the refinement and prioritization of the questions.
Scope of interventions and outcomes
The scoping process resulted in priority questions across the following digital health interventions
prioritized for evidence review within the guideline (included in Annex 2). The definitions of the
interventions included in this guideline are provided in Table 1.

page xvWHO guideline recommendations on digital interventions for health system strengthening
Table 1 Definitions of included digital health interventions
Digital health
intervention
Definition
Synonyms and
other descriptors
Birth notification
via mobile devices
Digital approaches to support the
notification of births, to trigger the
subsequent steps of birth registration
and certification, and to compile vital
statistics
ȺȺ Birth event alerts
ȺȺ Enabling health workers and
community to transmit alerts/
notifications when a birth has
occurred
Death
notification via
mobile devices
notification of deaths, to trigger the
subsequent steps of death registration
statistics, including cause-of-death
information
ȺȺ Death surveillance
ȺȺ Death event alert
communities to transmit alerts/
notifications when a death has
occurred
Stock
notification
and commodity
management via
mobile devices
Digital approaches for monitoring
and reporting stock levels, and
consumption and distribution of
medical commodities. This can include
the use of communication systems (e.g.
SMS) and data dashboards to manage
and report on supply levels of medical
commodities
ȺȺ Stock-out prevention and monitoring
ȺȺ Alerts and notifications of stock levels
ȺȺ Restocking coordination
ȺȺ Logistics management and
coordination
Client-to-
provider
telemedicine
Provision of health services at a
distance; delivery of health services
where clients/patients and health
workers are separated by distance
ȺȺ Consultations between remote client/
patient and health worker
ȺȺ Clients/patients transmit medical
data (e.g. images, notes and videos) to
health worker
Provider-
to-provider
telemedicine
Provision of health- services at a
distance; delivery of health services
where two or more health workers are
separated by distance
ȺȺ Consultations for case management
between health workers
ȺȺ Consulting with other health workers,
particularly specialists, for patient
case management and second opinion

page xvi
Digital health
intervention
Definition
Synonyms and
other descriptors
Targeted client
communication
via mobile devices
(targeted
communication
to individuals)
Transmission of customized health
information for different audience
segments (often based on health status
or demographic categories). Targeted
client communication may include:
i. transmission of health-event alerts
to a specified population group;
ii. transmission of health information
based on health status or
demographics;
iii. alerts and reminders to clients;
iv. transmission of diagnostic results (or
of the availability of results).
ȺȺ Notifications and reminders for
appointments, medication adherence,
or follow-up services
ȺȺ Health education, behaviour change
communication, health promotion
communication based on a known
client’s health status or clinical history
ȺȺ Alerts for preventive services and
wellness
ȺȺ Notification of health events to
specific populations based on
demographic characteristics
Health worker
decision support
via mobile devices
Digitized job aids that combine an
individual’s health information with the
health worker’s knowledge and clinical
protocols to assist health workers
in making diagnosis and treatment
decisions
ȺȺ Clinical decision support systems
(CDSS)
ȺȺ Job aid and assessment tools to
support service delivery, may or may
not be linked to a digital health record
ȺȺ Algorithms to support service delivery
according to care plans and protocol
Digital tracking
of patients’/
clients’ health
status and
services within
a health record
(digital tracking)
Digitized record used by health workers
to capture and store health information
on clients/patients in order to follow-
up on their health status and services
received. This may include digital
service records, digital forms of paper-
based registers for longitudinal health
programmes and case management
logs within specific target populations,
including migrant populations.
ȺȺ Digital versions of paper-based
registers for specific health domains
ȺȺ Digitized registers for longitudinal
health programmes, including tracking
of migrant populations’ benefits and
health status
ȺȺ Case management logs within specific
target populations, including migrant
population
Provision of
training to
health workers
via mobile devices
(mobile learning/
mLearning)
The management and provision of
education and training content in
electronic form for health professionals.
In contrast to decision support, health
worker training does not need to be
used at the point of care.
ȺȺ mLearning, eLearning, virtual learning
ȺȺ Educational videos, multimedia
learning and access to clinical and
non-clinical guidance for training
reinforcement
Source: adapted from Classification of digital health interventions v1.0 (WHO, 2018).

page xviiWHO guideline recommendations on digital interventions for health system strengthening
The interventions included in this guideline are those prioritized through the process described
above from the wider range of digital interventions available. Figure 3 depicts which interventions
were reviewed in this guideline, as well as interventions that were excluded at the scoping stage.
1.1 Targeted client
communication
1.4 Personal health
tracking
1.7 Client financial
transactions
1.5 Citizen based
reporting
1.6
On-demand
information
services to clients
1.2 Untargeted client
communication
1.3 Client to client
communication
1.1.1
Transmit health event
alerts to specific
population group(s)
1.1.2
Transmit targeted health
information to client(s)
demographics
1.1.3 Transmit targeted alerts
and reminders to client(s)
1.1.4
Transmit diagnostics
result, or availability of
result, to client(s)
1.4.1 Access by client to own
medical records
1.4.2
Self monitoring of
health or diagnostic data
by client
1.4.3 Active data capture/
documentation by client
1.7.1
Transmit or manage out
of pocket payments by
client(s)
1.7.2
Transmit or manage
vouchers to client(s) for
health services
1.7.3
Transmit or manage
incentives to client(s) for
health services
1.5.1 Reporting of health system
feedback by clients
1.5.2 Reporting of public health
events by clients
1.6.1 Client look-up of health
information
1.2.1
Transmit untargeted
health information to an
undefined population
1.2.2
Transmit untargeted
health event alerts to
undefined group
1.3.1 Peer group for clients
1.0
Clients
2.1
Client
identification and
registration
2.5 Health worker
communication
2.6 Referral
coordination
2.7
Health worker
activity planning
and scheduling
2.8 Health worker
training
2.9
Prescription
and medication
management
2.10
Laboratory and
Diagnostics
Imaging
Manangement
2.2 Client health
records
2.3 Health worker
decision support
2.4 Telemedicine
2.1.1 Verify client
unique identity
2.1.2 Enrol client for health
services/clinical care plan
2.5.1
Communication from
health worker(s) to
supervisor
2.5.2
Communication and
performance feedback to
health worker(s)
2.5.3
Transmit routine news and
workflow notifications to
health worker(s)
2.5.4
Transmit non-routine
health worker(s)
2.5.5 Peer group for health
workers
2.6.1 Coordinate emergency
response and transport
2.6.2
Manage referrals between
points of service within
health sector
2.6.3 Manage referrals between
health and other sectors
2.7.1 Identify client(s) in need
of services
2.7.2 Schedule health worker's
activities
2.8.1 Provide training content to
health worker(s)
2.8.2 Assess capacity of health
worker(s)
2.9.1 Transmit or track
prescription orders
2.9.2 Track client's medication
consumption
2.9.3 Report adverse drug events
2.10.1 Transmit diagnostic result
to health worker
2.10.2 Transmit and track
diagnostic orders
2.10.3 Capture diagnostic results
from digital devices
2.10.4 Track biological specimens
2.2.1
Longitudinal tracking
of clients’health status
and services
2.2.2 Manage client’s structured
clinical records
2.2.3
Manage client’s
unstructured
clinical records
2.2.4
Routine health indicator
data collection and
management
2.3.1
Provide prompts and
alerts based according
to protocol
2.3.2 Provide checklist
according to protocol
2.3.3 Screen clients by risk or
other health status
2.4.1
Consultations between
remote client and health
worker
2.4.2
Remote monitoring of
client health or diagnostic
data by provider
2.4.3 Transmission of medical
data to health worker
2.4.4
Consultations for case
management between
health worker(s)
2.0
Health Workers
3.1 Human resource
management
3.4 Civil Registration
andVital Statistic
3.6 Equipment and
asset management
3.7 Facility
management
3.5 Health
financing
3.2 Supply chain
management
3.3 Public health
event notification
3.1.1
List health workforce
cadres and related
identification information
3.1.2 Monitor performance of
health worker(s)
3.1.3
Manage certification/
registration of health
worker(s)
3.1.4 Record training credentials
of health worker(s)
3.4.1 Notify birth event
3.4.2 Register birth event
3.4.3 Certify birth event
3.4.4 Notify death event
3.4.5 Register death event
3.4.6 Certify death event
3.6.1 Monitor status of
health equipment
3.6.2
Track regulation and
licensing of medical
equipment
3.7.1 List health facilities and
related information
3.7.2 Assess health facilities
3.5.1 Register and verify client
insurance membership
3.5.2 Track insurance billing and
claims submission
3.5.3 Track and manage
insurance reimbursement
3.5.4
Transmit routine payroll
payment to health
worker(s)
3.5.5
Transmit or manage
incentives to health
worker(s)
3.5.6 Manage budget and
expenditures
3.2.1
Manage inventory and
distribution of health
commodities
3.2.2 Notify stock levels of
health commodities
3.2.3 Monitor cold-chain
sensitive commodities
3.2.4 Register licensed drugs
and health commodities
3.2.5 Manage procurement
of commodities
3.2.6
Report counterfeit or
substandard drugs
by clients
3.3.1
Notification of public
health events from
point of diagnosis
3.0
Health System Managers
4.1
Data collection,
management,
and use
4.3 Location
mapping
4.4 Data exchange and
interoperability
4.2 Data
coding
4.1.1
Non-routine data
collection and
management
4.1.2 Data storage and
aggregation
4.1.3 Data synthesis and
visualization
4.1.4
Automated analysis of
data to generate new
information or predictions
on future events
4.3.1 Map location of health
facilities/structures
4.3.2 Map location of
health events
clients and households
health worker
4.4.1 Data exchange
across systems
4.2.1 Parse unstructured data
into structured data
4.2.2
Merge, de-duplicate, and
curate coded datasets or
terminologies
4.2.3 Classify disease codes or
cause of mortality
4.0
Data Services
Figure 3 Interventions targeted in the guideline
Key: solid orange outline = full inclusion; dotted orange outline = partial inclusion
Source: WHO Classification of digital health interventions v1.0

page xviii
Scoping considerations regarding health domains and
delivery channels
Considering the diversity of the uses of ICT in health, the guideline process established that it was
also necessary to define the scope of the prioritized questions in relation to (i) health domains; (ii)
types of digital device (i.e. mobile devices); and (iii) delivery channels for the interventions (e.g.
SMS text messaging, multimedia applications, voice calls, interactive voice response).
Health domains
During the scoping consultations described above, the domains to be covered by the guideline
were determined, and they are presented in Table 2.
Table 2 Health domains covered by the guideline
Digital health intervention
Health domains included in
systematic review
Birth notification via mobile devices All – no restrictions
Death notification via mobile devices All – no restrictions
Stock notification and commodity
management via mobile devices
All – no restrictions
Client-to-provider telemedicine All – no restrictions
Provider-to-provider telemedicine All – no restrictions
Targeted client communication via
mobile devices (targeted communication
to individuals)
Sexual, reproductive, maternal, newborn, child
and adolescent health
Targeted client communication for
noncommunicable diseases was not included in
this version but has been prioritized for the next
update of this guideline
Health worker decision support
via mobile devices
Digital tracking of patients’/clients’ health
status and services (digital tracking)
Provision of training to health workers
via mobile devices (mLearning)

page xixWHO guideline recommendations on digital interventions for health system strengthening
Devices
Mobile devices are now used widely in almost all settings, and this has been the primary driver
for research and investment in digital health efforts across low- and middle-income countries.
The mobile nature of these devices also offers unique opportunities for service delivery. Given
the current and growing importance of mobile devices for delivering digital health interventions,
particularly in low- and middle-income countries, it was decided that this guideline would focus
on digital health interventions that were accessible via mobile devices. This decision was also
based on the need to define clear parameters for the systematic reviews.
Presentation of the guideline
For each recommendation, a summary of the evidence is given in Chapter 3 on the positive and
negative effects of the intervention, its acceptability and feasibility, the equity, gender and human
rights impacts, resource use, and on any other considerations reviewed at the GDG meeting.
The language that was used to interpret the evidence on effects is consistent with the approach
recommended by the Cochrane EPOC Group. Where the WHO team identified any existing
WHO recommendations relevant to this guideline, these were integrated into the text, and in all
instances transcribed exactly as published in the respective source guidelines. Where needed,
additional remarks are included to contextualize these recommendations, and citations for the
source documents are given for more details.
Summary of recommendations
Expected
Contribution
to universal health
coverage (UHC)
Digital health
intervention
Recommendation
Accountability
coverage
Recommendation 1
Birth notification
via mobile devices
WHO recommends the use of birth notification via mobile devices
under these conditions:
ȺȺ in settings where the notifications provide individual-level data
to the health system and/or a civil registration and vital statistics
(CRVS) system, and
ȺȺ the health system and/or CRVS system has the capacity to respond
to the notifications.
(Recommended only in specific contexts or conditions)
Responses by the health system including the capacity to accept the
notifications and trigger appropriate health and social services, such
as initiating of postnatal services.
Responses by the CRVS system include the capacity to accept the
notifications and to validate the information, in order to trigger the
subsequent process of birth registration and certification.

page xx
Expected
Contribution
to universal health
coverage (UHC)
Digital health
intervention
Recommendation
Accountability
coverage
Recommendation 2
Death notification
via mobile devices
WHO recommends the use of death notification via mobile devices
under these conditions:
ȺȺ in the context of rigorous research, and
ȺȺ in settings where the notifications provide individual-level data to
the health system and/or a CRVS system, and
ȺȺ the health system and/or CRVS system has the capacity to respond
to the notifications.
(Recommended only in the context of rigorous research and in specific
contexts or conditions)
Responses by the health system include the capacity to accept the
notifications and trigger appropriate health and social services.
Responses by the CRVS system include the capacity to accept the
notifications and to validate the information, in order to trigger the
subsequent process of death registration and certification.
Availability of
commodities and
equipment
Recommendation 3
Stock notification
and commodity
management via
mobile devices
WHO recommends the use of stock notification and commodity
management via mobile devices in settings where supply chain
management systems have the capacity to respond in a timely and
appropriate manner to the stock notifications.
Availability of
human resources
for health
Recommendation 4
Client-to-provider
telemedicine
WHO recommends the use of client-to-provider telemedicine to
complement, rather than replace, the delivery of health services and
in settings where patient safety, privacy, traceability, accountability
and security can be monitored.
In this context, monitoring includes the establishment of standard
operating procedures that describe protocols for ensuring patient
consent, data protection and storage, and verifying provider licensing
and credentials.
Availability of
human resources
for health
Effective coverage
Recommendation 5
Provider-to-provider
telemedicine
WHO recommends the use of provider-to-provider telemedicine in
settings where patient safety, privacy, traceability, accountability and
security can be monitored.
In this context, monitoring includes the establishment standard
operating procedures of that describe protocols for ensuring patient
consent, data protection and storage, and verifying provider licensing
and credentials.

page xxiWHO guideline recommendations on digital interventions for health system strengthening
Expected
Contribution
to universal health
coverage (UHC)
Digital health
intervention
Recommendation
Contact coverage
Continuous
coverage
Recommendation 6
Targeted client
communication via
mobile devices
WHO recommends targeted client communication via mobile devices
for health issues regarding sexual, reproductive, maternal, newborn,
and child health under the condition that potential concerns about
sensitive content and data privacy can be addressed
Effective coverage Recommendation 7
Health worker
decision support via
mobile devices
WHO recommends the use of decision support via mobile devices for
community and facility-based health workers in the context of tasks
that are already defined within the scope of practice for the health
worker.
Effective coverage
Accountability
coverage
Recommendation 8
Digital tracking of
clients’ health status
and services (digital
tracking) combined
with decision
support
WHO recommends digital tracking of clients’ health status and
services, combined with decision support under these conditions:
ȺȺ in settings where the health system can support the
implementation of these intervention components in an integrated
manner; and
ȺȺ for tasks that are already defined as within the scope of practice for
the health worker.
Effective coverage
Accountability
coverage
Continuous
coverage
Recommendation 9
Digital tracking
combined with:
(a) decision
support and
(b) targeted client
communication
WHO recommends the use of digital tracking combined with decision
support and targeted client communication under these conditions:
ȺȺ where the health system can support the implementation of these
intervention components in an integrated manner;
ȺȺ for tasks that are already defined as within the scope of practice for
the health worker; and
ȺȺ where potential concerns about data privacy and transmitting
sensitive content to clients can be addressed.
Effective coverage Recommendation 10
Digital provision
of training and
educational content
to health workers
via mobile devices/
mobile learning
(mLearning)
WHO recommends the provision of learning and training content
via mobile devices /mLearning to complement, rather than replace,
traditional methods of delivering continued health education and
post-certification training
(Recommended)

page xxii
While the recommendations included in this guideline are based on distinct digital interventions,
they all contribute to the health systems’needs in different but interlinked ways. For health system
managers, the recommendation on digital stock notification aims to drive availability of commodities
at the point of services. From the clients’and patients’perspectives, this would include ability
to access health information and services more immediately, such as through client to provider
telemedicine and targeted client communication. Likewise, health workers need to be accessible and
adhere to practices for delivering high-quality care, through interventions such as decision support
and mLearning. Figure 4 illustrates the linkages across the different recommendations and the
interlinked ways that these digital interventions can cohesively address health system needs.
Health workers can provide
appropriate and high quality care
Births are notified
and accounted
for to receive services
Individuals
can access
health services
and information
Health workers
are knowledgeable
about which services
to provide
Deaths are
notified and
accounted for
Health workers
are accessible
Health commodities
and supplies are
available at the
point of care
Health workers can
follow-up to ensure
individuals receive
appropriate services
Recommendation 1
Birth notification
Recommended in specific conditions
ACCOUNTABILITY
Recommendation 2
ACCOUNTABILITY
Death notification
Recommended in the context
of rigorous research and
specific conditions
Figure 4 Linkages of the recommendations across the health system
Recommendation 6
Targeted client
communication
DEMAND
Client-to-provider
telemedicine
SUPPLY
Recommendation 4
SUPPLY
Stock notification &
commodity management
SUPPLY
Recommendation 3
QUALITY
telemedicine
Recommendation 5
Recommendation 7
QUALITY
Health worker
decision support
digital tracking +
decision support
QUALITY
Recommendation 8
Recommendation 9
digital tracking +
decision support & targeted
client communication
QUALITY
Recommendation 10
provision of training
and educational content
Recommended
QUALITY

WHO guideline recommendations on digital interventions for health system strengthening
1. Introduction
1.1 Background
Digital health, or the use of digital technologies for health, has become a salient field of practice
for employing routine and innovative forms of information and communications technology (ICT)
to address health needs. The term digital health is rooted in eHealth, which is defined as “the use
of information and communications technology in support of health and health-related fields” (1).
Mobile health (mHealth) is a subset of eHealth and is defined as “the use of mobile wireless
technologies for public health” (2,3). More recently, the term digital health was introduced as
“...a term encompassing eHealth (which includes mHealth), as well as emerging areas, such as the
use of advanced computing sciences in ‘big data’, genomics and artificial intelligence” (3,4).
Digital health has attracted substantial interest from the medical and public health community,
most notably in low- and middle-income countries, where mobile communication has opened a
new channel for overcoming geographical inaccessibility of health care. Over a thousand digital
health deployments have been recorded since 2008 (5), representing a fraction of the uses of digital
health that may exist but are not formally documented. Governments, donors and multilateral
institutions have also recognized the potentially transformative role of digital technologies for
health system strengthening. In a joint document published in 2015, the World Bank Group, the
United States Agency for International Development (USAID) and the World Health Organization
(WHO) advocated the “use of the digital revolution to scale up health interventions and engage
civil society” (6).
The World Health Assembly Resolution on Digital Health unanimously approved by Member
States in May 2018 demonstrated a collective recognition of the value of digital technologies to
contribute to advancing universal health coverage (UHC) and other health aims of the Sustainable
Development Goals (SDGs) (4). This resolution urged ministries of health
to assess their use of digital technologies for health […] and to prioritize, as appropriate, the development,
evaluation, implementation, scale-up and greater use of digital technologies, as a means of promoting equitable,
affordable and universal access to health for all, including the special needs of groups that are vulnerable in the
context of digital health (4).
Furthermore, it tasked WHO with providing normative guidance in digital health, including
“through the promotion of evidence-based digital health interventions” (4).

Amid all the heightened interest, digital health has also been characterized, however, by
implementations being widely rolled out in the absence of careful examination of the evidence
base on benefits and harms (7). The enthusiasm for digital health has also driven a proliferation
of short-lived implementations and an overwhelming diversity of digital tools, with a limited
understanding of their impact on health systems and people’s well-being. This concern was
highlighted most notably in the consensus statement of the WHO Bellagio eHealth Evaluation
Group, which opened by stating: “To improve health and reduce health inequalities, rigorous
evaluation of eHealth is necessary to generate evidence and promote the appropriate integration
and use of technologies” (8). While recognizing the innovative role that digital technologies can
play in strengthening the health system, there is an equally important need to evaluate their
contributing effect to ensure that such investments do not inappropriately divert resources from
alternative, non-digital approaches.
1.2 Role of digital health in health system
strengthening and universal health coverage
UHC aims to ensure the quality, accessibility and affordability of health services. However,
shortfalls remain in ensuring access to all who need health services and in ensuring that they are
delivered with the intended quality without causing financial hardship to the people accessing
them (9). The Tanahashi framework published by WHO in 1978 provides a time-tested model of
understanding health system performance gaps and how they prevent the intended coverage,
quality and affordability of health services to individuals (10). This cascading model illustrates how
health systems lose performance because of challenges at successive levels, each dependent on
the previous level. Health system challenges – such as geographical inaccessibility, low demand
for services, delayed provision of care, low adherence to clinical protocols and costs to individuals/
patients – contribute to incremental losses in health system performance that cumulatively
impact on the health of individuals. These shortfalls limit the ability to close the gaps in coverage,
quality and affordability, and undermine the potential to achieve UHC (Figure 1.1).

Figure 1.1 Layers of UHC achievement affected by health system performance
This adapted Tanahashi (10) model illustrates that each health system performance layer builds on
the components below it but also falls short (dotted lines) of the optimal, desired level. Digital health
interventions could contribute to efforts to address challenges that limit achievement of that health
system goal (11).

To deliver effective and affordable coverage of health services to all, this guideline extends the
conceptual foundation of the Tanahashi framework, as follows (11).
ȺȺ Accountability – Accountability coverage represents the proportion of people in the target
population (registered a subset of the total population) in the health system (for example,
through civil registration and vital statistics mechanisms, population censuses, the issuance
of national or health identifiers), which importantly establishes the different population
denominators of health care provision.
ȺȺ Supply comprises the availability of commodities and equipment, of human resources and of
health facilities, and facilitates access to appropriate services with qualified health workers in
geographically accessible health facilities, where and when patients need them. Even where
health services are available, there may be barriers to accessing them for target populations.
ȺȺ Demand – driving demand and increasing access can ensure that gaps in contact coverage
(i.e. the gap between the total availability of services and the actual contact that individuals
have with facilities, health workers and services) do not further undermine health system
performance. Individuals often need multiple interactions and follow-up with the health
system for health interventions to be effective, and continuous coverage defines the extent to
which the full course of the interventions is achieved.
ȺȺ Quality is related to effective coverage and can be undermined by gaps that that result when
health interventions are delivered suboptimally, such as when health workers do not abide by
treatment protocols.
ȺȺ Affordability – direct and indirect costs to the patient can have catastrophic financial
effects. Efforts made to ensure individuals are protected from impoverishment due to health
interventions are reflected in the affordability layer as improved financial coverage.
Digital technologies introduce novel opportunities to address health system challenges,
and thereby offer the potential to enhance the coverage and quality of health practices and
services (Figure 1.2) (11,12). Digital health interventions may be used, for example, to facilitate
targeted communications to individuals through reminders and health promotion messaging
in order to stimulate demand for services and broaden access to health information. Digital
health interventions may also be targeted to health workers to give them more immediate
access to clinical protocols through, for example, decision-support mechanisms or telemedicine
consultations with other health workers.

Figure 1.2 Linking health system challenges to digital health interventions,
implemented through digital applications
A digital health intervention is defined here as a discrete functionality of digital technology that
is applied to achieve health objectives (13). The range of digital health interventions is broad, and
the software and technologies – digital applications – that make it possible to deliver these digital
interventions continue to evolve within the inherently dynamic nature of the field. A starting
point for categorizing the different digital health interventions being used to overcome defined
health system challenges is provided by WHO’s Classification of digital health interventions v1.0 (13),
summarized in Figure 1.3.
Health System
ChallengesȺȺ poor demand
for services
ȺȺ failure to follow
guidelines
ȺȺ commodities stockout
ȺȺ insufficient workforce
ȺȺ inaccessibility of facilities
Quality & Coverage
of Health Intervention
Intervention of
known efficacy
Digital Health
Interventions

Figure 1.3 Examples of how digital health interventions may address
health system challenges, implemented through ICT systems
As an example, digital applications and ICT systems (such as logistics management information systems) are
implemented and apply digital health interventions (such as to notify stock levels of health commodities) to
address health system challenges (such as insufficient supply of commodities) and achieve health objectives
(maintain consistent availability of commodities).
Source: WHO, 2018 (13)
Health System
Challenge (HSC)
Need or problem to be addressed
» Client
communication
system
Digital Health Intervention (DHI)
Digital functionality for
addressing the HSC
Numbered interventions relate to WHO’s Classification
of digital health interventions v1.0 (7)
Applications and ICT systems
Software systems and communication
channels that deliver one or more of the
digital health interventions
Insufficient
supply of
commodities
Lack of access
to information
or data
Loss to
follow-up
of clients
» Logistics
Management
Information
System
» Health Management
Information System
(HMIS)
» Electronic
Medical Record
» Electronic
Medical Record
1.1.3 Transmit targeted alerts and reminders
to client(s)
3.2.1 Manage inventory and distribution
of health commodities
3.2.2 Notify stock levels of health commodities
2.2.1 Longitudinal tracking of clients’health
status and services
4.1.2 Data storage and aggregation
4.1.3 Data synthesis and visualizations
2.2.4 Routine health indicator data collection
and management
Lastly, digital health interventions are applied within a country context and a health system, and
their implementation is made possible by a number of factors (Figure 1.4). These include: (i) the
health domain area and associated content; (ii) the digital intervention itself (i.e. the functionality
provided); (iii) the hardware, software and communication channels for delivering the digital
health intervention; and mediated within (iv) a foundational layer of the ICT and enabling
environment, characterized by the country infrastructure, leadership and governance, strategy
and investment, legislation and policy compliance, workforce, standards and interoperability, and
common services and other applications.

STRATEGY &
INVESTMENT
SERVICES &
APPLICATIONS
LEGISLATION,
POLICY &
COMPLIANCE
WORKFORCE
STANDARDS &
INTEROPERABILITY
INFRASTRUCTURE
Health Content
Digital Health
Interventions
sector objectives
health content
+ +
Figure 1.4 Components of digital health implementations
1.3 Objectives of this guideline
This guideline responds to the 2018 World Health Assembly Resolution on Digital Health,
requesting WHO to provide Member States with normative guidance to inform the adoption of
evidence-based digital health interventions. Within the Resolution, Member States specifically
request:
… that WHO builds on its strengths, by developing guidance for digital health, including, but not limited to, health
data protection and usage, on the basis of its existing guidelines and successful examples from global, regional and
national programmes, including through the identification and promotion of best practices, such as evidence-based
digital health interventions and standards (4).
The key aim of this guideline is to present recommendations based on a critical evaluation of
the evidence on emerging digital health interventions that are contributing to health system
improvements, including an assessment of the benefits, harms, acceptability, feasibility,
resource use and equity considerations. For the purposes of the guideline, the recommendations
examine the extent to which digital health interventions available via mobile devices are able to
address health system challenges at different layers of coverage along the pathway to UHC. By
reviewing the evidence of different digital interventions, as well as assessing the risks against
comparative options, this guideline aims to equip health policy-makers and other stakeholders
with recommendations and implementation considerations for making informed investments into
digital health interventions.

This guideline urges readers to recognize that digital health interventions are not a substitute for
functioning health systems, and that there are significant limitations to what digital health is able
to address. Digital health interventions should complement and enhance health system functions
through mechanisms such as accelerating exchange of information. However, digital health will
not replace the fundamental components needed by health systems such as the health workforce,
financing, leadership and governance, and access to essential medicines (14). An understanding of
what health system challenges can realistically be addressed by digital technologies, along with
an assessment of the ecosystem’s ability to absorb such digital interventions, is needed to inform
investments in digital health.
This guideline reviewed the following interventions:
ȺȺ stock notification and commodity management via mobile devices across all health conditions
ȺȺ client1
-to-provider telemedicine across all health conditions
ȺȺ provider-to-provider telemedicine across all health conditions
ȺȺ targeted client communication (TCC) via mobile devices (spread across five population groups
for sexual, reproductive, maternal, newborn, child and adolescent health [SRMNCAH])
ȺȺ health worker decision support via mobile devices across all health conditions
ȺȺ digital tracking of patients’/clients’health status and services via mobile devices across all
health conditions
ȺȺ provision of training to health workers via mobile devices (mLearning) across all health
conditions.
1.4 Target audience
The primary target audience for this guideline is decision-makers in ministries of health and
public health practitioners, to aid them to develop a better understanding of which digital health
interventions have an evidence base to address health system needs. This guideline may also prove
beneficial to organizations that invest resources into digital health systems as implementation
and development partners. This document aims to strengthen evidence-based decision-making on
digital approaches by governments and partner institutions, encouraging the mainstreaming and
institutionalization of effective digital interventions within supportive digital systems.
2 Although WHO’s Classification of digital health interventions v1.0 uses the term “client” (13), the terms “individual” and “patient” may be used
The systematic reviews included accessibility via mobile devices to ensure that these digital
interventions are applicable in low resource settings where extensive computerized systems
may not be available or feasible. However, the recommended interventions can be deployed
through any digital device, including stationary devices, such as desktop computers, and does
not preclude them from being used on non-mobile digital devices.

1.5 Linkages with other WHO resources
WHO has published several resources on digital health, yet to date has not released normative
guidelines detailing recommendations about which digital health interventions are supported by
demonstrable evidence for addressing specific health system challenges.
Several WHO clinical and public health guidelines have been developed that include
recommendations for digital technologies alongside other interventions, such as medication
adherence and supporting community health workers. These include:
ȺȺ 2016 Consolidated guidelines on the use of antiretroviral drugs for treating and preventing HIV
infection (15)
ȺȺ 2017 update of Guidelines for treatment of drug-susceptible tuberculosis and patient care (16)
ȺȺ 2018 WHO guideline on health policy and system support to optimize community health worker
programmes (17).
Within these examples, digital health interventions are embedded as part of a package of
recommended options. This guideline, by contrast, will make explicit recommendations on the
added value of specific digital interventions while also including the recommendations of those
previous WHO guidelines, where relevant.
Other WHO resources on digital health, detailed below, include the National eHealth Strategy
Toolkit published jointly with the International Telecommunication Union (ITU), reports from the
Global Observatory for eHealth, the Classification of digital health interventions v1.0, the Digital
Health Atlas and the Be He@lthy, Be Mobile initiative.
ȺȺ The WHO/ITU National eHealth Strategy Toolkit is a foundational resource to guide policy-
makers at ministries of health in establishing national eHealth/digital health strategies, which
are necessary for national governance and a supportive ecosystem for digital health (18).
ȺȺ The WHO Global Observatory for eHealth reports are based on periodic surveys conducted
among Member States on their use of eHealth. The most recent eHealth report was in 2016
and featured survey responses from 125 countries (1). A similar report focused on mHealth was
conducted in 2011 (2).
ȺȺ The WHO Classification of digital health interventions v1.0 provides a shared language to describe
the uses of digital technology for health, specifying discrete digital capabilities applicable to
clients, health workers, health system managers, and data services (13).
ȺȺ The WHO Digital Health Atlas is a website-based technology registry for systematically
tracking national and subnational digital health activities, in order to equip governments,
technologists, implementers and donors to better coordinate implementations, monitor their
functionality and geographical growth, and establish gaps against which to collaboratively
target investments (19).
ȺȺ The Be He@lthy, Be Mobile initiative represents a collaboration between WHO and ITU to harness
mobile technologies for communication on noncommunicable disease (NCD) risk factors (20).

1.6 Context and the enabling environment
The maturity of the ecosystem, comprising the enabling and ICT environments, has a critical
influence on the relevance and impact of the recommended digital health interventions. The
enabling environment is defined as the attitudes, actions, policies and practices that support
the effective and efficient functioning of organizations and programmes. For digital health,
this includes factors such as the leadership, governance mechanisms, regulatory and policy
frameworks, strategy and financial investment, workforce capacity, standards and interoperability,
and sociocultural considerations – as articulated within the pillars of the WHO/ITU eHealth
Strategy Toolkit (18). The ICT environment consists of the infrastructure and the mechanisms for
executing the digital health intervention, such as the hardware and digital applications.
There is considerable value in assessing the ecosystem in a given context or country, reviewing
health system needs and tempering expectations and plans for adoption of different interventions
based on the ICT and enabling environments available within a setting. In the absence of a robust
enabling environment, there is the risk of a proliferation of unconnected systems and a severe
impact on the effectiveness and sustainability of digital tools. To help assess ecosystem readiness
and the maturity of the ecosystem, several resources exist, including the WHO Score assessment
tool (21), MEASURE Evaluation’s Health Information Systems Interoperability Maturity Toolkit (22),
the Partnership for Maternal, Newborn and Child Health’s ICT planning workbook (23) and the
Global Digital Health Index (24).
As with any introduction of innovations and new approaches, digital health interventions require
changes in behaviour and transitions to new practices. One example is moving away from
entrenched paper-based systems to digital approaches. Implementations will succeed only if the
digital health intervention is taken up by users, adds value, and facilitates the desired change or
action. As such, implementers must be aware of the motivations, barriers and resistance to the
disruption of the status quo that may affect the fidelity of deployment and understand that this
will temper the possible benefit of digital health interventions.
The adoption of the recommendations in this guideline should not exclude or jeopardize the
provision of quality health services in places where there is no access to the digital interventions,
or because they are not acceptable or affordable for target communities. Additionally, in contexts
where the ecosystem may not be mature enough to accommodate specific digital health
interventions, there should be a focus on strengthening the health system and addressing
gaps in the enabling environment to facilitate the implementation of these recommendations in
the future.

1.7 Linkages to the broader digital health
architecture
Digital health interventions are intended to integrate with and fit into an overall digital health
architecture. The digital health architecture provides an overview or blueprint to describe how
different digital applications (software and ICT systems) and related functionalities would interact
with each other within a given context (25). While the unit of analysis for this guideline focuses on
the value of specific digital interventions, there is an equally important need to support a cohesive
approach to implementation, in which different digital interventions can operate together, rather
than as duplicative and isolated implementations. Stakeholders will benefit from a thorough
review of guidance found in the following complementary sources.
ȺȺ The WHO/ITU National eHealth Strategy Toolkit (18) gives government agencies a framework
and methods for developing a national eHealth vision, an action plan and a monitoring
framework – critical elements for establishing an enabling environment.
ȺȺ The ITU Digital Health Platform Handbook: Building a Digital Information Infrastructure
(Infostructure) for Health (25) provides guidance for ensuring investments into digital health
systems are systematically planned as part of an enterprise architecture that establishes core
systems (such as health management information systems, logistics management information
systems and electronic medical records) and common functionalities (such as registries, data
exchange, terminology services) that are interoperable and reusable across different health
programme areas.
ȺȺ The Principles for Digital Development (26) are nine living concepts designed to help
implementers integrate established best practices into digital programmes, facilitate
the avoidance of common pitfalls and encourage the adoption of approaches that have
demonstrated value over time. These include principles of designing with users, understanding
the ecosystem, reuse of and improvement upon existing digital solutions, and addressing
privacy and security concerns.
ȺȺ The Principles of Donor Alignment for Digital Health (27) offer ministries of health the tools
to hold signatory donors and technical partners accountable for making investments in
digital health that align in a coordinated way with the national digital health strategies
that support national health strategies. This document also calls for a heightened focus on
architecture, standards, investment frameworks, privacy protection and detailed operational
and monitoring plans.
ȺȺ The forthcoming WHO Planning and Costing Guide for Digital Interventions for Health
Programmes serves as an implementation guide for ministries of health to operationalize
these recommendations into a costed plan for their health programmes. The Guide provides
a systematic approach to assessing health system gaps and needs, a stepwise approach to
identifying appropriate digital health interventions within the digital ecosystem, and the
planning tools for costing implementation, which are appropriate within and across health
programme areas within a ministry of health.

ȺȺ Resources available from Integrating the Healthcare Enterprise (IHE) (28), including
standards-based tools and services (resources) to improve the way digital systems in health
care function and interoperate, to support patient and population care.
Communities of practice focused on strengthening capacity and digital health implementation
through knowledge-sharing and coordination include (in alphabetical order):
ȺȺ African Alliance of Digital Health Networks (African Alliance) (29)
ȺȺ Asia eHealth Information Network (AeHIN) (30)
ȺȺ Global Digital Health Network (31)
ȺȺ Health Data Collaborative, Digital Health and Interoperability Working Group (32)
ȺȺ Open Health Information Exchange (OpenHIE community of practice) (33).
1.8 Living guidelines approach
This guideline includes recommendations on a list of prioritized digital health interventions
accessible via mobile devices, representing a subset of a much larger set of digital interventions.
This guideline aims to incorporate a broader set of emerging digital health interventions gradually
in subsequent versions. The WHO Classification of digital health interventions v1.0 (13), provides
a starting point to tackle the evolving nature of digital health and to identify interventions for
future inclusion in updated guidelines. This version applies WHO Guidelines Review Committee
procedures (34) to a priority list of emerging digital innovations, while also acknowledging that
future guideline versions will need to incorporate the evidence for additional digital health
interventions. This approach to updating WHO guidelines is known as “living guidelines”.
The living guidelines approach also facilitates the updating of existing recommendations as new
evidence becomes available and the inclusion of additional health domains that might not have
been reflected in this initial release. For example, the evidence and recommendations for the
digital health intervention of targeted client communication (TCC) was restricted to specific
health areas and a subsequent version of the guideline will expand on this area to include the
use of TCC for noncommunicable diseases. Chapter 6 (Disseminating and updating the guideline)
also details the living guidelines approach for updating and broadening the set of digital health
interventions falling under a WHO guideline development process.

2. Methods
The development of this guideline followed the methods described in the second edition of the
WHO handbook for guideline development (35). This institution-wide process at WHO entailed
the identification of critical questions and outcomes, retrieval of the evidence, assessment
and synthesis of that evidence, the formulation of recommendations, and planning for the
implementation, dissemination, impact evaluation and updating of this guideline.
The guideline development process also included two rounds of online surveys and three in-person
consultations. These consultations included (i) an advisory meeting in February 2016 to establish the goal of the
guideline in light of other WHO resources and to determine underlying frameworks; (ii) a scoping meeting in
September 2016 to prioritize and draft the critical questions and outcomes; and (iii) a final meeting in June 2018
to review the synthesized evidence and formulate recommendations. Online surveys were used before and after
the September scoping meeting to inform the refinement and prioritization of the questions.
2.1 Identification of priority questions
Process for defining the scope of interventions and outcomes
The initial advisory meeting in February 2016 was used to explore the strategic direction of this
guideline, including defining the objectives and the framing of digital health interventions. Since
there were no preceding WHO guidelines with defined terminologies for specific digital health
interventions, this meeting examined frameworks and standardized classifications that could be
leveraged for the formulation of the priority questions. These included the WHO Classification
of digital health interventions v1.0 (13), which would serve as the source for prioritizing the
interventions (see below and Figure 2.1). The health system challenges outlined in the same source
(see Annex 1) informed the development of outcomes.

Following the advisory meeting, the responsible officer’s team at WHO compiled a set of questions
using the standard PICO (population, intervention, comparator, outcomes) format. This initial
set of questions was reviewed during a virtual consultation in June 2016 with participants from
the February advisory meeting, as well as by technical focal points across WHO, to ensure the
appropriateness of the outcomes. The draft questions then underwent further revisions during the
scoping meeting in September 2016, conducted in person with global technical experts.
Prioritization of interventions and outcomes
To supplement the scoping meeting, WHO circulated two rounds of virtual surveys across global
and regional networks, including the Asia eHealth Information Network (AeHIN) (30), the Global
Digital Health Network (31), Health Information For All (36) and the Implementing Best Practices
(IBP) Initiative (37). The first survey was conducted in August 2016 to obtain a general sense of
priority interventions and outcomes prior to the scoping meeting in September 2016. During the
scoping meeting, the panel of technical experts further refined and prioritized the questions.
Following the in-person scoping meeting, WHO distributed a second survey to prioritize the
revised questions. This survey asked respondents to rank outcomes and interventions along a
nine-point scale based on how critical the questions were for decision-making, where a rating
of 1 indicated that the outcome was not important and a rating of 9 indicated that the outcome
was critical (6). Over 300 respondents from all WHO regions participated across the two surveys.
Findings from this second survey helped to narrow down the final list of priority questions.
2.2 Scoping of interventions and outcomes
The scoping process focused on nine digital health interventions that were prioritized for evidence
review (see Annex 2 for the questions in the PICO format):
ȺȺ stock notification and commodity management via mobile devices
ȺȺ client1
-to-provider telemedicine
ȺȺ provider-to-provider telemedicine
ȺȺ targeted client communication via mobile devices (spread across five population groups)
ȺȺ health worker decision support via mobile devices
ȺȺ digital tracking of patients’/clients’ health status and services via mobile devices
ȺȺ provision of training to health workers via mobile devices (mobile learning/mLearning).
1 Although WHO’s Classification of digital health interventions v1.0 (13) uses the term “client” , the terms “individual” and “patient” may be used

Table 2.1 Definitions of included digital health interventions
Digital health
intervention
Definition Synonyms and other descriptors
Birth
notification
The capture and onward transmission
of minimum essential information
on the fact that a birth has occurred,
with that transmission of information
being sufficient to support eventual
registration and certification of the
vital event.
notification of births, to trigger the
subsequent steps of birth registration
statistics (13,38)
ȺȺ Birth event alerts
notifications when a birth has
occurred
Death
notification
The capture and onward transmission
of minimum essential information on
the fact that a death has occurred,
with that transmission of information
being sufficient to support eventual
registration and certification of the
vital event.
notification of deaths, to trigger the
subsequent steps of death registration
statistics, including cause-of-death
information (13,38)
ȺȺ Death surveillance
ȺȺ Death event alert
notifications when a death has
occurred
Stock
notification
and commodity
management
Digital approaches for monitoring
and reporting stock levels, and
consumption and distribution of
medical commodities. This can include
the use of communication systems (e.g.
SMS) and data dashboards to manage
and report on supply levels of medical
commodities (13).
ȺȺ Stock-out prevention and monitoring
ȺȺ Alerts and notifications of stock levels
ȺȺ Restocking coordination
ȺȺ Logistics management and
coordination
Client-to-
provider
telemedicine
Provision of health services at a
distance; delivery of health care
services where clients/patients and
health workers are separated by
distance (13,18)
ȺȺ Consultations between remote client/
individual and health worker
ȺȺ Clients/individuals contact health
workers to receive clinical guidance on
health issue
ȺȺ Clients/individuals transmit medical
data (e.g. images, notes and videos) to
health worker

Digital health
intervention
Definition Synonyms and other descriptors
Provider-
to-provider
telemedicine
Provision of health services at
a distance; delivery of health
care services where two or more
heath workers are separated by
distance (13,18)
ȺȺ Consultations for case management
between health workers
ȺȺ Consulting other health workers,
including specialists, for patient case
management and second opinion
Targeted client
communication
(targeted
communication
to individuals
and patients)
Transmission of customized health
information for different audience
segments (often based on health status
or demographic categories). Targeted
client communication may include
i. transmission of health-event alerts
to a specified population group;
ii. transmission of health information
demographics;
iii. alerts and reminders to clients; and
iv. transmission of diagnostic results (or
of the availability of results) (13,39).
ȺȺ Notifications and reminders for
appointments, medication adherence,
or follow-up services
ȺȺ Notification of health events to
specific populations based on
demographic characteristics
ȺȺ Health education, behaviour change
communication, health promotion
communication based on a known
client’s health status or clinical history
ȺȺ Alerts for preventive services and
wellness
Health worker
decision support
Digitized job aids that combine an
individual’s health information with the
health worker’s knowledge and clinical
protocols to assist health workers
in making diagnosis and treatment
decisions (13,18)
ȺȺ Clinical decision support systems
(CDSS)
ȺȺ Job aid and assessment tools to
support service delivery, may or may
not be linked to a digital health record
ȺȺ Algorithms to support service delivery
according to care plans and guidelines
Digital tracking
of patients’/
clients’ health
status and
services (digital
tracking)
Digitized record used by health workers
to capture and store health information
on clients/patients in order to follow-
up on their health status and services
received (13,18). This may include
digital service records, digital forms of
paper-based registers for longitudinal
health programmes (40), and case
management logs within specific
populations.
ȺȺ Digital versions of paper-based
registers for specific health domains
ȺȺ Digitized registers for longitudinal
health programmes including tracking
of migrant populations’ benefits and
health status
ȺȺ Case management logs within specific
population
Provision of
training and
educational
content to
health workers
(mobile
learning/
mLearning)
The management and provision
of education and training
content in digital form for health
professionals (13,18). In contrast to
decision support, mLearning does not
need to be used at the point of care.
ȺȺ mLearning, eLearning, virtual learning
ȺȺ Educational videos, multimedia
learning and access to clinical
guidance for training reinforcement
Source: adapted from Classification of digital health interventions v1.0 (13)

The interventions included in this guideline are those prioritized through the process described
above from the wider range of digital interventions available (13) (Figure 2.1). The excluded digital
health interventions can be readily identified for subsequent updates to this guideline (see
section 6.3).
Digital health interventions excluded during the scoping process for this version of the
guideline are:
ȺȺ untargeted client communication (e.g. transmitting health information to an undefined
population);
ȺȺ client-to-client communication (e.g. peer communication);
ȺȺ citizen-based reporting (e.g. reporting of health system feedback or public health events
by clients);
ȺȺ on-demand information services to clients;
ȺȺ client financial transactions (e.g. transmission of vouchers to clients for health services);
ȺȺ client identification and registration (e.g. verifying unique ID);
ȺȺ health worker activity planning and scheduling (e.g. client looking up health information);
ȺȺ prescription and medication management (e.g. tracking client’s medical consumption);
ȺȺ laboratory and diagnostics imaging management (e.g. transmitting diagnostic orders);
ȺȺ human resource management (e.g. monitoring performance of health workers);
ȺȺ public health event notification (e.g. notification of public health event for point of diagnosis);
ȺȺ health financing (e.g. registering and verifying insurance membership);
ȺȺ equipment and asset management (e.g. monitoring status of health equipment);
ȺȺ facility management (e.g. assessing health facilities);
ȺȺ data collection management and use (e.g. non routine data collection, data visualization);
ȺȺ data coding (e.g. classifying disease codes or cause of mortality);
ȺȺ location mapping (e.g. mapping location of health events);
ȺȺ data exchange and interoperability (e.g. facilitating data exchange across systems).

Figure 2.1 Interventions targeted in the guideline
1.1 Targeted client
communication
1.4 Personal health
tracking
transactions
1.5 Citizen based
reporting
1.6
On-demand
information
services to clients
communication
communication
1.1.1
alerts to specific
population group(s)
1.1.2
demographics
1.1.4
medical records
1.4.2
Self monitoring of
health or diagnostic data
by client
1.7.1
client(s)
1.7.2
Transmit or manage
health services
1.7.3
Transmit or manage
health services
feedback by clients
events by clients
information
1.2.1
Transmit untargeted
1.2.2
Transmit untargeted
undefined group
1.0
Clients
2.1
Client
identification and
registration
2.5 Health worker
communication
2.6 Referral
coordination
2.7
Health worker
activity planning
and scheduling
2.8 Health worker
training
2.9
Prescription
and medication
management
2.10
Laboratory and
Diagnostics
Imaging
Manangement
2.2 Client health
records
2.3 Health worker
decision support
2.4 Telemedicine
2.1.1 Verify client
unique identity
2.5.1
Communication from
health worker(s) to
supervisor
2.5.2
Communication and
health worker(s)
2.5.3
health worker(s)
2.5.4
health worker(s)
workers
2.6.2
health sector
of services
activities
health worker(s)
worker(s)
prescription orders
consumption
to health worker
diagnostic orders
2.2.1
and services
clinical records
2.2.3
Manage client’s
unstructured
clinical records
2.2.4
data collection and
management
2.3.1
Provide prompts and
to protocol
other health status
2.4.1
worker
2.4.2
data by provider
2.4.4
management between
health worker(s)
2.0
Health Workers
3.1 Human resource
management
andVital Statistic
3.6 Equipment and
asset management
3.7 Facility
management
3.5 Health
financing
3.2 Supply chain
management
3.3 Public health
event notification
3.1.1
cadres and related
health worker(s)
3.1.3
worker(s)
of health worker(s)
health equipment
3.6.2
equipment
related information
claims submission
3.5.4
payment to health
worker(s)
3.5.5
Transmit or manage
worker(s)
expenditures
3.2.1
commodities
health commodities
of commodities
3.2.6
substandard drugs
by clients
3.3.1
health events from
point of diagnosis
3.0
4.1
Data collection,
management,
and use
4.3 Location
mapping
interoperability
4.2 Data
coding
4.1.1
Non-routine data
collection and
management
aggregation
visualization
4.1.4
on future events
health events
health worker
4.4.1 Data exchange
across systems
4.2.2
terminologies
cause of mortality
4.0
Data Services
Key: solid orange outline = full inclusion; dotted orange outline = partial inclusion
Source: WHO Classification of digital health interventions v1.0

Scoping health domains and delivery channels
In addition to delineating the specific digital health intervention, the guideline development
process established that it was also necessary to define the scope of the prioritized questions in
relation to (i) health domains; (ii) types of digital device (e.g. mobile devices); and (iii) delivery
channels for the interventions (e.g. digital applications, SMS text messaging, voice calls, interactive
voice response, etc.).
Health domains
During the scoping consultations described above, the following decisions were made on the
domains (Table 2.2) to be covered by the guideline.
ȺȺ Digital health interventions targeting health workers, health system managers and health systems
more broadly: The guideline questions regarding these interventions were not restricted to a
specific health condition and were aimed to be inclusive of all health domains and services
provided at the primary care level. This decision was made because these interventions, such as
notification of stock levels, or decision support, were recognized as having functions that cut
across multiple health domains and were often implemented across a whole health system. The
systematic reviews commissioned for these interventions extracted information on the health
domains covered in order to conduct subgroup analyses where appropriate, and to highlight
any potential differences across health domains.
ȺȺ Digital health interventions primarily targeting clients/individuals: This guideline includes one
intervention – targeted client communication (TCC) – that is typically linked with or directed
to health behaviours associated with specific health topics, such as completing treatment for
sexually transmitted infections or returning for family planning appointments. Consequently,
it was decided that the scope of the guideline question for this specific intervention focusing
on clients’ use of services needed to specify the range of health topics.
In this first version of the guideline, the population focus for the intervention of TCC was
sexual, reproductive, maternal, newborn, child and adolescent health (SRMNCAH) since this
was the entry point for initiating the guideline development process. A planned update to this
guideline will include TCC for additional health domains, including noncommunicable diseases.

Table 2.2 Scope of health domains included in systematic reviews on
digital health interventions
Digital health intervention
Health domains included in
systematic review
Birth notification via mobile devices All – no restrictions
Death notification via mobile devices All – no restrictions
Stock notification and commodity
Client-to-provider telemedicine All – no restrictions
Provider-to-provider telemedicine All – no restrictions
Targeted client communication via mobile
devices (targeted communication to
individuals)
Sexual, reproductive, maternal, newborn, child and
adolescent health
Targeted client communication for
noncommunicable diseases was not included in
this version but has been prioritized for the next
update of this guideline
Digital tracking of patients’/clients’ health
status and services
Health worker decision support via mobile
devices
Provision of training to health workers via
mobile devices (mLearning)
Devices
Mobile devices are now used widely in almost all settings (40), and this has been the primary
driver for research and investment in digital health efforts across low- and middle-income
countries. The mobile nature of these devices also offers unique opportunities for service
delivery. Given the current and growing importance of mobile devices for delivering digital health
interventions, particularly in low- and middle-income countries, it was decided that this guideline
would primarily focus on digital health interventions that were accessible via mobile devices.
This decision was also based on the practical consideration to define clear parameters for the
systematic reviews.
The phrase “accessible via mobile devices” was chosen explicitly to indicate that interventions
may be used across a variety of digital devices, but that there should at least be a way to engage
with the digital intervention through a mobile interface. These devices range from different types
of mobile phones and smartphones, to tablets and other point-of-care handheld devices. The

searches for the systematic reviews required that interventions have, at a minimum, a mobile
component, and could also have additional ways of engaging with the information, including
through desktop computers. While this guideline focuses on interventions accessible via mobile
devices as the inclusion criteria for primary studies, the evidence retrieval included information on
linkages to non-mobile digital systems. For example, the review on stock management included
information on the linkages to web-based data dashboards accessed on desktop computers for
visualizing the data at district and national levels.
Although the systematic reviews included accessibility via mobile devices to ensure that these
digital interventions are applicable in low resource settings where extensive computerized
systems may not be available, it does not preclude the recommendations from being used on non-
mobile digital devices, such as desktop computers.
Delivery channels
The guideline development process did not place any restrictions on the delivery channels for the
included digital health interventions – whether the interventions would be delivered via voice
messaging, text messaging or interactive voice response, for example.
2.3 Evidence retrieval
Two main types of evidence were considered for this guideline:
ȺȺ evidence on the effectiveness digital health interventions based on randomized controlled
trials (RCTs), non-randomized studies (NRS), controlled before-and-after studies (CBAs) and
interrupted time series studies (ITSs); and
ȺȺ evidence on factors affecting the acceptability, feasibility and implementation of digital health
interventions based on qualitative studies.
The evidence on resource use and cost-effectiveness was confined to that found in the studies
included in the reviews of effectiveness, including RCTs and NRS. No further searches for cost-
effectiveness evidence were undertaken. Additional information about resource requirements was
gathered through an assessment of programme documents and discussions with implementers.
This assessment provided detailed information from the health system perspective on the major
cost drivers for implementing each intervention, to inform the guideline development group’s
(GDG) discussions regarding the resources required. The compiled set of evidence was presented
in the evidence-to-decision frameworks (see Web Supplement 1).
The Web Supplements are available at www.who.int/reproductivehealth/publications/
digital-interventions-health-system-strengthening/en/).

Evidence on the effectiveness of digital health interventions
Cochrane systematic reviews were used as the primary source of evidence on the effectiveness
of digital health interventions. Using the priority questions agreed on during the scoping process,
the WHO steering group commissioned new Cochrane reviews or identified existing or ongoing
Cochrane reviews. When ongoing Cochrane reviews were identified, the authors were invited to
collaborate with the technical team (see Annex 3 of this guideline document) to ensure that the
reviews would be as relevant as possible for the guideline.
The search strategies to identify relevant studies, and the specific criteria for study inclusion and
exclusion, are described within the individual systematic reviews (see Web Supplements 2G-2L).
Most of the included reviews were based on the methods recommended by the Cochrane Effective
Practice and Organisation of Care (EPOC) (42) and Consumers and Communication (43) groups.
Evidence on factors affecting the acceptability, feasibility
and implementation of digital health interventions
Systematic reviews of qualitative studies were the primary source of evidence on factors affecting
the acceptability, feasibility and implementation of digital health interventions. Using the priority
questions agreed on during the scoping process, the WHO steering group commissioned one new
Cochrane review of qualitative studies and identified two ongoing reviews. When ongoing reviews
were identified, the authors were invited to collaborate with the GDG (see Annex 3 to ensure
that the reviews were as relevant as possible for the guideline. These three systematic reviews of
qualitative studies covered the following topics:
ȺȺ health workers’ perceptions and experiences of digital health interventions in primary care
(Web Supplement 2A)
ȺȺ health workers’ and students’ perceptions and experiences of mLearning (Web Supplement 2B)
ȺȺ clients’ perceptions and experiences of targeted client communication (Web Supplement 2C).
In addition, two of the Cochrane reviews of effectiveness commissioned for the WHO guideline
development process also included a secondary objective focused on identifying factors
influencing the implementation of the interventions in question. For this objective, the reviews
included studies of any design that reported quantitative, qualitative or descriptive data. The two
systematic reviews identified covered the following topics:
ȺȺ tracking health commodity inventory and notifying stock levels via mobile devices
(Web Supplement 2D)
ȺȺ birth and death notification via mobile devices (Web Supplement 2E).

Descriptions of the search strategies to identify the qualitative studies, the specific criteria for
inclusion and exclusion of qualitative studies, and the databases searched are included in each
of the individual systematic reviews. Similar information is available in each of the individual
systematic reviews that included a secondary objective on identifying the factors influencing the
implementation of the interventions in question.
Finally, an overview of systematic reviews was commissioned to explore the factors influencing
the acceptability, feasibility and implementation of telemedicine interventions. This overview
included reviews that fulfilled the PRISMA Group’s definition of a systematic review (44) and that
included qualitative studies, surveys or mixed-method studies. The details of the methods used
are available in the overview report (Web Supplement 2F).
Cross-cutting factors affecting the acceptability, feasibility
and implementation of digital health interventions
To identify common factors affecting acceptability, feasibility and implementation that cut across
the digital health interventions included in this guideline, an overarching analysis of findings was
undertaken using the findings from the systematic reviews of qualitative studies, the overview of
systematic reviews and the mixed-methods analyses done alongside the reviews of effectiveness.
2.4 Assessment, synthesis and grading
of the evidence
Assessment of risk of bias/methodological limitations of
primary studies included in the reviews
For the effectiveness studies included in the systematic reviews of the effects of interventions,
the risk of bias was assessed using the explicit criteria outlined in the Cochrane handbook for
systematic reviews of interventions (45), and the guidance from the Cochrane EPOC group (42). Each
included study was assessed and rated by the review authors as being at low, high or unclear risk
of bias for each risk-of-bias domain. These assessments provided an overall risk of bias for each
included study and each outcome, where appropriate.
For the qualitative studies included in the qualitative evidence syntheses, the methodological
limitations were assessed by applying a quality appraisal framework to each study. The adaptation
of the Critical Appraisal Skills Programme’s quality assessment tool was used for qualitative
studies (46).

Two of the Cochrane reviews included a secondary objective focused on identifying factors
influencing the implementation of the interventions in question, and included studies of any
design that reported quantitative, qualitative or descriptive data. For these additionally included
studies, the methodological limitations were assessed using the “ways of evaluating important and
relevant data” (WEIRD) tool for the critical appraisal of programme descriptions, implementation
descriptions and other mainly descriptive types of evidence (47,48).
For the effectiveness studies, qualitative studies and other studies included in the assessment of
implementation factors for two of the reviews, no studies were excluded based on an assessment
of the risk of bias or of the methodological limitations, but instead this information was used
to assess the certainty of the review findings, as part of the Grading of Recommendations
Assessment, Development and Evaluation (GRADE) or Confidence in the Evidence from Reviews
of Qualitative Research (GRADE-CERQual) approaches (49–52) (see the last subsection in section
2.4). An adapted version of the ‘Enhancing transparency in reporting the synthesis of qualitative
research’ (ENTREQ) statement was used for the criteria to judge the methodological limitations of
the included systematic reviews (53).
Synthesis of evidence
For systematic reviews of the effects of interventions, meta-analyses were conducted to estimate
an overall effect for outcomes if the intervention characteristics and outcome measures were
sufficiently similar across the included studies – that is, if the interventions, participants and the
underlying question were similar enough for statistical pooling to be feasible. Where interventions
and outcomes were not sufficiently similar to allow meta-analysis, results were reported using
a structured narrative summary. Subgroup analyses were planned to focus on factors such as
study setting, health care setting, provider type and intervention characteristics, but there were
generally insufficient data to allow them to be conducted. Summary tables were created for the
main comparisons and included the most important outcomes, the findings for each and the
assessment of the certainty of this evidence.
For the syntheses of the qualitative evidence, the data were analysed to identify themes. Findings
were then compiled for each theme. Details of the analytical approaches used for each synthesis
are described in Web Supplement 2. Summary tables were created to include each synthesis
finding and an assessment of the confidence in the evidence for it.

For the overview of systematic reviews to explore the factors influencing the acceptability,
feasibility and implementation of telemedicine interventions, three authors analysed the data
using a thematic approach. The details of this approach are found in Web Supplement 2.. Summary
tables were developed to include each overview finding and an assessment of the confidence in
the evidence for it.
Where possible, evidence from the reviews on effectiveness, qualitative evidence syntheses and
systematic review of systematic reviews was used to highlight the impacts of the interventions on
gender, equity and human rights.
Assessment of the certainty of review evidence
The GRADE approach was used to assess the certainty of the evidence on the effectiveness of
the interventions for all the outcomes identified in the PICO questions, and a GRADE evidence
profile was prepared for each outcome for each review comparison (52). Based on this approach,
the certainty of evidence for each outcome was rated as high, moderate, low or very low. Within
the GRADE approach, RCTs were considered to provide high-certainty evidence, while NRS and
observational studies were considered to provide low-certainty evidence. The evidence for each
outcome was then downgraded when justified by the assessments of risk of bias, inconsistency,
imprecision, indirectness and publication bias. This grading was undertaken by the review authors
in collaboration with the technical team. The final assessment was based on a consensus among
the review authors.
The GRADE-CERQual approach was used to assess the confidence that should be placed in
each review finding from (i) the qualitative evidence syntheses; (ii) the secondary analysis of
factors influencing the implementation of the interventions included in two of the Cochrane
reviews; and (iii) the telemedicine systematic review of systematic reviews. In the GRADE-
CERQual approach, confidence in the evidence is based on the following four components: the
methodological limitations of included studies; the coherence of the review finding; the adequacy
of the data contributing to a review finding; and the relevance of the included studies to the
review question (47,49). After assessing each of the four components, a judgement was made
about the overall confidence in the evidence supporting each review finding. All findings started
as high confidence and were then downgraded if there were important concerns about any of the
CERQual components. The overall confidence was judged as high, moderate, low or very low. This
grading was undertaken by the review teams in collaboration with the technical team. The final
assessment was based on consensus among the review authors.

2.5 Roles and responsibilities of contributors
The guideline development process was guided by the WHO steering group, the technical team,
the GDG, the external review group, and external partners and observers (see Annex 3 for the list of
contributors in these main groups). An advisory group representing global experts also contributed
to the guideline development process prior to establishing the formal GDG.
WHO steering group
The WHO steering group comprised WHO staff members and consultants representing WHO
regional offices, and WHO departments, including the following (in alphabetical order):
ȺȺ Alliance for Health Policy and Systems Research
ȺȺ Essential Medicines and Health Products
ȺȺ Global TB Programme
ȺȺ Health Workforce
ȺȺ HIV/AIDS
ȺȺ Immunization, Vaccines and Biologicals
ȺȺ Information, Evidence and Research
ȺȺ Maternal, Newborn, Child and Adolescent Health
ȺȺ Prevention of Noncommunicable Diseases
ȺȺ Reproductive Health and Research
ȺȺ Service Delivery and Safety
The steering group, whose members are listed in Annex 3, contributed to the scoping of the
guideline, drafting of the questions in PICO format, and interpretation of the findings from the
systematic reviews. The steering group will also oversee the dissemination of the guideline
(Chapter 6).

Technical team
The technical team, whose members are listed in Annex 3, comprised guideline methodologists
from the Norwegian Institute of Public Health, the systematic review team, and systematic
reviewers from Cochrane Response, an evidence consultancy unit operated by Cochrane. The
technical team provided guidance on formulating the priority guideline questions so as to ensure
that these questions could then be addressed by systematic reviews. The technical team also
collaborated with the WHO steering group in developing the systematic review protocols; in
undertaking and managing the systematic reviews; in appraising the evidence from systematic
reviews using the GRADE methodology (for reviews of intervention effectiveness) and the GRADE-
CERQual methodology (for qualitative evidence syntheses) (49–52); in populating the evidence-
to-decision frameworks; in supporting meeting processes for the GDG; and in preparing the final
guideline document.
Additional support for undertaking the systematic reviews was provided by the Cochrane EPOC
Group and the Cochrane Consumers and Communication Group, including in relation to scoping
the priority guideline questions and ensuring that the systematic reviews followed standard
Cochrane methods and processes.
Guideline development group
The GDG comprised 28 external (non-WHO) international stakeholders with expertise in research
and implementation for digital health interventions, including health programme managers,
government representatives, researchers and implementers. The members of the group, who are
listed in Annex 3 were identified in a way that ensured geographical representation and gender
balance. Their short biographies were published at the WHO website for public review and
comment prior to the first GDG meeting.
Selected members of the group participated in a scoping meeting held in September 2016 (see
the beginning of section 2.1) and provided input into the final version of the priority guideline
questions and outcomes that guided the evidence review. The GDG examined and interpreted the
evidence and formulated the final evidence-based recommendations at a face-to-face meeting in
June 2018. The group also reviewed and approved the final guideline document.
External review group
An external review group of six additional expert stakeholders (listed in Annex 3) peer-reviewed
the final guideline document to identify any factual errors, and commented on the clarity of the
language, contextual issues and implications for implementation. It was not within the remit of
this group to change any recommendations formulated by the GDG.

Declarations of interest by external contributors
In accordance with the second edition of the WHO handbook for guideline development (35), all
GDG, technical team and external review group members were required to complete and submit
a WHO declaration-of-interests form before engaging in the guideline process. The standard WHO
form for declaration of interests was completed and signed by each expert and sent electronically
to the responsible technical officer. The WHO steering group assessed the declarations and
determined whether any identified conflict warranted one of several actions: exclusion from the
GDG, exclusion from deliberations and voting in one or more of the topic areas, inclusion in all
of evidence review sessions but exclusion from final voting on recommendations or no action
required. In addition, all experts were instructed to notify the responsible technical officer of any
change in relevant interests during the course of the guideline development process, for a review
of conflicts of interest accordingly. See Annex 4 for a summary of the declaration-of-interest
statements and how any conflicts were managed. Additionally, the responsible officer’s team also
posted on the WHO website the names and brief biographies of GDG members.
2.6 Consolidation of evidence
The technical team supervised and finalized the preparation of the evidence profiles and evidence
summaries. These were then consolidated into an evidence-to-decision framework for each
guideline question. The evidence-to-decision frameworks (see Web Supplement 1) provided
explicit and systematic presentations of the evidence for each question on the following criteria.
ȺȺ Effectiveness – the evidence on the critical outcomes was summarized to answer the
questions: “What are the desirable and undesirable effects of the intervention/option?” and
“What is the certainty of the evidence on effects?”
ȺȺ Acceptability – this criterion addressed the question: “Is the intervention/option acceptable
to clients and health workers?”
ȺȺ Feasibility – factors such as the resources, infrastructure and training requirements
determine the feasibility of implementing an intervention. The question addressed was: “Is
it feasible for the relevant stakeholders to implement the intervention/option?”
ȺȺ Resource use – this criterion addressed the questions: “What are the resources associated
with the intervention/option?” and “Is the intervention/option cost-effective?”
ȺȺ Gender, equity and rights – this criterion encompassed evidence or considerations on
whether or not an intervention would reduce health inequities. The question addressed was:
“What is the anticipated impact of the intervention/option on equity?”
For each guideline question, judgements were made on the impact of the intervention under these
criteria, to guide the GDG’s recommendation decisions.

2.7 Decision-making and formulation of
recommendations
The WHO steering group provided the evidence-to-decision frameworks, including evidence
summaries, GRADE evidence profiles, and other documents related to each guideline question, to
the GDG in advance of the final in-person GDG meeting. The purpose of the final GDG meeting
was to reach a majority decision on each recommendation, including its direction and conditions,
based on the evidence and implementation experiences presented. During this face-to-face
meeting in June 2018, and under the leadership of the GDG co-chairs (Annex 3), GDG members
collectively reviewed the frameworks and contributed to the drafting of the recommendations.
The GDG meetings were guided by the following process: (i) presentation of the evidence-to-
decision frameworks for the specific interventions by the relevant systematic review teams;
(ii) discussion followed by indicative voting on the different components of the evidence-to-
decision frameworks (effectiveness, acceptability, feasibility, resource use, gender, equity and
rights); (iii) discussion followed by voting to determine the category of recommendation (see the
recommendation categories below); and (iv) a discussion on any conditions. The views of the GDG
were gauged based on online voting before moving towards a decision on a recommendation for
each guideline question.
Based on the discussions and voting process, the responsible officer’s team at WHO drafted
the recommendations during the meeting and presented these to the GDG for its remarks on
the research priorities and implementation considerations. GDG members were invited to a
subsequent webinar in October 2018 for any clarifications needed ahead of reviewing the draft
guideline document.
Finally, the technical team had also drafted implementation considerations for each intervention,
based on the findings of the evidence syntheses and the gaps identified in the evidence base. The
GDG and the WHO steering group added to these implementation considerations during the GDG
meeting and subsequent review of this document.

Recommendation categories
In line with other published WHO guidelines (54–56), GDG members voted to classify each
recommendation into one of the following categories:
ȺȺ recommended – the intervention or option should be implemented;
ȺȺ not recommended – the intervention or option should not be implemented;
ȺȺ recommended only in specific contexts or conditions – the intervention or option is applicable
only to the condition, setting or population specified in the recommendation, and should be
implemented only in these contexts; or
ȺȺ recommended only in the context of rigorous research – there are important uncertainties about
the intervention or option; in such instances, implementation can still be undertaken on a
large scale, provided that it takes the form of research that is able to address unanswered
questions and uncertainties related to effectiveness of the intervention and its acceptability
and feasibility.
What do we mean by a recommendation
“only in the context of rigorous research”?
The recommendation category “Recommended only in the context of rigorous research” is
used in this guideline when the evidence reviewed for a guideline question demonstrated
important uncertainties or left unanswered questions about the intervention.
Where uncertainties relate to the effectiveness of an intervention, future research should
ideally compare people who are exposed to the option with people who are not, and
include a baseline assessment. These comparison groups should be as similar as possible
to ensure that the effect of an intervention is assessed rather than the effect of other
factors. Programmes evaluated without a comparison group or baseline assessment
are generally at a higher risk of bias and so may not measure the true effect of an
intervention. RCTs are the most robust way to assess the effectiveness of an intervention.
Randomization may not be feasible though for some kinds of intervention (for example,
interventions that can be implemented only across a whole jurisdiction) – in these cases,
other study designs should be considered, such as interrupted time series analyses or
controlled before-and-after studies.
Where unanswered questions or uncertainties are linked to the acceptability or feasibility
of the intervention, future research should include well-conducted qualitative studies, and
quantitative designs such as surveys, to explore these issues.

Voting process
Voting on the recommendations was conducted electronically while the GDG meeting was in
session, such that GDG members were blinded to the reactions of their peers. The GDG co-
chairs announced the voting results while the recommendation was being discussed. Majority
decision was defined as the agreement of two thirds or more of the GDG, provided that those
who disagreed did not feel strongly about their position. Strong disagreements would have
been recorded in this guideline; no such disagreements occurred in the GDG meeting. The GDG
determined any contexts for the recommendations by the same process of majority decision,
based on discussions about the balance of evidence on the effects (benefits and harms) of the
interventions across different contexts.
The WHO steering group, systematic review team and observers were not eligible to vote. If the
issue to be voted on involved primary research or systematic reviews conducted by any of the
participants who had declared a conflict of interest, those individuals were allowed to participate
in the discussion but were not allowed to vote on the issue in question.
2.8 Document preparation and peer review
Following the final GDG meeting, the responsible technical officer from the WHO steering group
prepared a draft of the full guideline document that reflected as accurately as possible the
deliberations and decisions of the GDG. Other members of the steering group and the technical
team provided comments on the draft document before it was sent electronically to the GDG
members for further comments and to the external review group for peer review. The technical
team reviewed the feedback provided by the GDG and the external review group and revised the
draft guideline as needed. After the GDG meetings and external peer review, further modifications
to the document by the steering group and technical team were limited to corrections of factual
error and improvements in language to address any lack of clarity. The revised final version was
returned electronically to the GDG members for their approval.

2.9 Presentation of the guideline
The recommendations are presented in the executive summary of this guideline. For each
recommendation, a summary of the evidence is given in Chapter 3 on the positive and negative
effects of the intervention, its acceptability and feasibility, the equity, gender and human rights
impacts, resource use, and on any other considerations reviewed at the GDG meeting. The
language that was used to interpret the evidence on effects is consistent with the approach
recommended by the Cochrane EPOC Group (42). Where the WHO steering group identified any
existing WHO recommendations relevant to this guideline, these were integrated into the text, and
in all instances transcribed exactly as published in the respective source guidelines. Where needed,
additional remarks are included to contextualize these recommendations, and citations of the
source documents are given for more details.

3. Evidence and
recommendations
This guideline provides nine evidence-based recommendations on the digital health
interventions that were prioritized during the scoping process (see sections 2.1 and 2.2).
These recommendations are made with the expectation that their implementation is grounded
in an understanding of the ecosystem readiness and maturity, as outlined in Chapter 4. Although
the systematic reviews included accessibility via mobile devices to ensure that these digital
interventions are applicable in low resource settings where extensive computerized systems may
not be available, it does not preclude the recommended interventions from being used on non-
mobile digital devices, such as desktop computers. For each of the digital health interventions
reviewed in this guideline, this chapter elaborates on the following components:
ȺȺ background information on the specific digital health intervention
ȺȺ an overview of the specific evidence
ȺȺ the recommendation along with a justification and remarks
ȺȺ specific implementation considerations.
Overall gaps in the evidence are described in Chapter 5; specific gaps and research questions for each of the
interventions is detailed in Annex 5. In addition, Web Supplement 1 contains the evidence-to-decision frameworks
and elaborates on the specific findings for each intervention as it relates to its effectiveness, acceptability,
feasibility, resource use, and gender, equity and human rights concerns.The Web Annexes cited here are available
at www.who.int/reproductivehealth/publications/digital-interventions-health-system-strengthening/en/
3.1 Cross-cutting acceptability and
feasibility findings
Most of the digital health interventions in this guideline are targeted at or expected to be used
by health workers. The following findings point to factors that influence the acceptability and
feasibility of digital interventions used by health workers. These findings are based on qualitative
evidence syntheses and overviews of digital health interventions for health workers in primary
care (Web Supplement 2A); mLearning (Web Supplement 2B) stock notification and tracking
commodities (Web Supplement 2D), and birth and death notification (Web Supplement 2E).

Acceptability for health workers
Factors that may increase acceptability
Digital health interventions allow health workers to expand their range of tasks as well as take
on tasks previously assigned to higher-level workers. This can be experienced as satisfying and
fulfilling, both for those to whom tasks are shifted, as well as to those from whom tasks are
shifted (moderate confidence, Web Supplement 2A). Health workers working in rural and remote
contexts particularly appreciate the efficiency of digital health technologies as these allow them
to offer services through the device (moderate confidence, Web Supplement 2A). Health workers
are likely to perceive digital health technologies to be more efficient because of the increased
speed with which they allow them to work (moderate confidence, Web Supplement 2A). These
technologies are also likely to save travelling time for health workers in both urban and rural
settings, allowing them to spend more time with their clients1
in urban areas or to provide services
remotely to clients in rural areas (moderate confidence, Web Supplement 2A). Health workers may
appreciate the portability of digital health technologies because this allows them to be flexible,
to work when convenient, and not have to be office-bound to access information (low confidence,
Web Supplement 2A). Health workers, particularly lay health workers in low- and middle-income
settings, also perceive digital health technologies as allowing them to better coordinate the
delivery of care through connecting them to other people and sectors in the health system and
to clients and communities (moderate confidence, Web Supplement 2A).
Some health workers also report that digital health technologies raise their social status and
increase the trust and respect they receive in communities. This is in part due to the device
itself but is also because they use these devices to access health workers at higher levels of care.
Community health workers, feel that the devices increase the respect they receive from health
professionals and from the community (moderate confidence, Web Supplements 2A and 2E).
Similar findings are seen among health workers in training, although there is also some concern
that clients/patients and colleagues might regard their use of mobile devices as unprofessional
because of their association with recreation (low confidence, Web Supplement 2B ).
Factors that may decrease acceptability
Some health workers do not experience digital health interventions as efficient as these
interventions do not reduce their workload and in some cases increase their workload
(moderate confidence, Web Supplement 2A), making them less likely to accept these interventions
(moderate confidence, Web Supplement 2F). Health workers may perceive digital health
interventions as increasing their workload when it means maintaining two systems (i.e. digital and
paper-based), when there are staff shortages, when the addition of the digital health intervention
to current work is not understood and appreciated by supervisors, or when they themselves
perceive the intervention as peripheral to their work. While some health workers do not object to
the additional work, others expect to be remunerated for it (low confidence, Web Supplements 2A
and 2E).

Health workers may also be concerned about loss, damage and theft and may complain about
having to carry both a personal and a work phone (low confidence, Web Supplements 2A and 2B).
In some settings, health workers use their personal mobile phones and Internet access for work
purposes, although this use is not necessarily formalised and health worker expenses are not
always covered (low confidence, Web Supplements 2A and 2E). This can include expenses for
air time or for charging their phone. Health workers may see these personal costs as a burden.
However, they may feel a moral imperative to assist their clients by using their own phones despite
the personal costs this incurs (low confidence, Web Supplement 2A).
Health workers’ perceptions and experiences of digital health interventions are likely to be shaped
by their pre-existing digital literacy. Health workers who manage well have positive views about
the use of mobile devices. However, health workers who struggle to use these technologies have
negative perceptions about its usefulness, may not understand the information generated by these
technologies, and are also anxious about making errors. In some instances, poor digital literacy
threatens job security (high confidence, Web Supplement 2A). However, even technologically
more competent users are reported as needing support and repeat training in the use of the
programmes and devices (low confidence, Web Supplement 2B).
Feasibility for health worker
Many health workers, particularly in rural and remote areas, experience logistical challenges when
using digital health technologies, including poor network connectivity and access to electricity
to charge their mobile phones (high confidence, Web Supplements 2A, 2B, 2D, 2E and 2F). In some
instances, poor connectivity also results in client dissatisfaction because it creates delays in
receiving health services (high confidence, Web Supplement 2A).
Health workers want easy-to-use, reliable equipment and ongoing technical support (high
confidence, Web Supplements 2A, 2D and 2F). They also feel that the use of these technologies
can be expanded to a wider range of settings, services, and illnesses (high confidence, Web
Supplement 2A). However, health workers often report usability issues, and poor integration with
other digital systems (high confidence, Web Supplements 2C and 2F). Although the introduction
of digital health interventions into existing healthcare systems may be important, this requires
many changes and may be difficult to achieve (low confidence, Web Supplement 2F). For instance,
institutional support and local champions may be considered important for ensuring integration
into existing systems, but staff reorganization and the breakdown of existing partnerships may
undermine this support (low confidence, Web Supplement 2F).

Health workers may experience a number of problems with the design of the programmes or
of the device itself, including programmes in languages they are not proficient in, inaccurate
rendering of the local language font, small screens, devices being ill-suited for note-taking,
and SMS character limitations (low confidence, Web Supplement 2A and 2B). Although the
involvement of staff and clients in the planning, design and implementation of the digital system
is considered important by health workers (moderate confidence, Web Supplements 2A and
2D), this is not always done (moderate confidence, Web Supplement 2F). Health workers may be
dissatisfied with digital health when technology changes are too rapidly introduced, or when their
expectations of the technologies are not met (low confidence, Web Supplement 2A).
Some stakeholders are also concerned about the confidentiality of medical information and
data security (moderate confidence, Web Supplement 2F). Health workers may try to protect
clients’ confidential information when using digital health devices, in particular when the
information concerns stigmatised conditions such as HIV/AIDS (low confidence, Web Supplement
2A). Achieving informed consent for sharing records and images can also be challenging,
particularly in settings with low levels of basic literacy or digital literacy (moderate confidence,
Web Supplement 2F).
Training is important for staff acceptance and system use (high confidence, Web Supplements 2A,
2B, 2D, 2E and 2F). While some health workers experience difficulties in understanding and using
digital health technologies, health workers and trainers feel that training and familiarity with
these technologies can help overcome these difficulties. Some health workers feel hampered in
learning to use mobile health technologies if it is not also used by their clinical mentors (moderate
confidence, Web Supplement 2A). This may be particularly important as health workers requiring
technical support may receive this support from higher level staff or from peers (low confidence,
Web Supplement 2A). Supportive supervision is also considered important for staff acceptance
and system use (moderate confidence, Web Supplement 2D).
Digital systems can make it possible to track and monitor health workers’ activities. Health
workers may feel that this changes how they work and may make their work more visible. Some
health workers may perceive this as positive, but it may leave other health workers with the sense
of “big brother watching”. Supervisors may feel that this allows them to be more aware of the
work of lower level health workers and to address problems (low confidence, Web Supplements 2A
and 2D).
Even where challenges tied to the design and usability of digital systems and devices are
addressed, these systems may not be able mitigate a number of broader health systems
challenges, for example, an underlying lack of medical commodities (low confidence, Web
Supplement 2D).

Acceptability and feasibility for clients/individuals
The following findings point to factors that are likely to influence the acceptability and feasibility
of digital health interventions targeted at or expected to be used by clients/patients. These
findings are summarized based on overviews and qualitative evidence syntheses related targeted
client communication (Web Supplement 2C) and telemedicine (Web Supplement 2F). More
detailed descriptions on the acceptability and feasibility findings are available within the sections
focused on the specific interventions.
Some individuals describe targeted communication and telemedicine services via mobile devices
in positive terms. For instance, some clients appreciate the fact that someone is taking the time to
send them messages as this can make them feel like someone is interested in their situation and
invested in their well-being. These clients describe the messages as providing support, guidance
and information, and giving a sense of direction, reassurance and motivation (moderate
confidence, Web Supplement 2C). Similarly, some clients using telemedicine services see these
as offering reassurance and a sense of safety and appreciate the increased access and the
consistency and continuity of care that it can offer (low confidence, Web Supplement 2F). Some
clients also feel that telemedicine services have increased their independence and self-care (low
confidence, Web Supplement 2F).
However, individuals who are dealing with health conditions that are often stigmatised or very
personal (e.g. HIV, family planning and abortion care) worry that their confidential health
information will be disclosed or their identity traced due to their participation in targeted
communication programmes (high confidence, Web Supplement 2C). Some individuals using
telemedicine services prefer face-to-face contact (low confidence, Web Supplement 2F).
Additionally, individuals believe there should be little or no charge tied to digital health
programmes, such as joining the programme, downloading apps, or charges related to sending and
receiving SMS/phone calls (high confidence, Web Supplement 2C).
Targeted communication and telemedicine services can potentially increase access for some
groups of individuals. For instance, telemedicine services can give individuals who speak minority
languages access to health workers who speak this language (high confidence, Web Supplement
2F); and may save money and reduce the burden of travel for clients with caring or work
responsibilities, living far from health care facilities or with few funds (low confidence, Web
Supplements 2C and 2F).
However, access to and use of these services can be particularly difficult for some individuals.
These include individuals with poor access to network services, electricity (high confidence, Web
Supplement 2C) or mobile devices (moderate confidence, Web Supplements 2A and 2C); clients
who speak minority languages, have low literacy or digital literacy skills (moderate confidence,
Web Supplement 2C) or hearing impairments (high confidence, Web Supplement 2A). Clients
with stigmatized health conditions may also be particularly concerned about the privacy of their
information (high confidence, Web Supplement 2C).

Accountability coverage
The proportion of those in the target population registered into the health system
3.2 Accountability coverage:
birth and death notification
via mobile devices
Background
A global scale-up plan for strengthening civil registration and vital statistics (CRVS) systems has
been developed by the World Bank and WHO with the goal of achieving “universal civil registration
of births, deaths and other vital events, including reporting cause of death, and access to legal
proof of registration for all individuals by 2030” (57). A key component of this plan is to prioritize
and strengthen the linkages between CRVS systems and health (57–59). This includes the use
of digital information systems to strengthen CRVS systems and expanding the coverage of
registration services among underserved populations, such as people residing in rural areas (57–60).
In these respects, the global proliferation of mobile phones and cellular network connectivity (41)
is increasingly being leveraged, especially in resource-limited settings, to drive the development
and use of digital civil registration systems (11,12,60–63).
Notification is the capture and onward transmission of minimum essential information on the
fact of birth or death has occurred, and represents the first step in the process leading to eventual
registration and certification of the vital event. Increasing the efficiency of birth and death
notification as well as promoting linkages between the health and civil registry sectors (many
births are first known in the health sector) can strengthen civil registration processes and the
use of health services (61,62). Digital mechanisms to facilitate notifications may enhance these
linkages as well as catalysing civil registration. Furthermore, added to their ability to conduct
notifications, the increased access to mobile devices among community-based individuals such
as vaccination programme workers, community health workers and village elders can potentially
expand the coverage of civil registration systems to underserved rural and remote regions (60–63).
For birth notifications, other information related to the birth may be transmitted via mobile
phones in the form of phone calls, inputs to an interactive voice response or unstructured
supplementary service data (USSD) system, SMS text messages, messages from mobile device-
based applications (apps) or calls or messages to publicly known short codes or access numbers.
The content of the birth notification may vary by country or implementation, but may include
the name of the child born, the name and address of the parents, the place and date of birth, and
details of birth outcomes.

Similarly, for death notifications, information related to the death may be transmitted via mobile
phone calls, inputs to an interactive voice response or USSD system, SMS text messages, messages
from apps, or calls or messages to publicly known short codes or access numbers. The content of
the death notification may vary by country or implementation, but may include the name of the
deceased, the name and address of a relative, the place and date of death, and details of the cause
of death.
This guideline question reviewed the added value of the notification of birth and death events via
mobile devices as an additional channel for supporting the establishment of a CRVS system and
strengthening linkages to it.
Overview of the evidence
The following is a summary of the evidence on birth and death notification via mobile devices.
Web Supplement 1 provides the full evidence-to-decision framework for this intervention,
detailing the available evidence on effectiveness, acceptability, feasibility, resource use and
implications for equity, gender and rights.
Effectiveness
ȺȺ Births: There is limited evidence on the effectiveness of using mobile devices for birth
notification as the certainty of this evidence was assessed as very low.
ȺȺ Deaths: No evidence on effectiveness was identified for death notification via mobile devices.
Acceptability
The qualitative evidence suggests the intervention is probably acceptable to health workers
and enables them to be more proactive in identifying pregnancies and coordinating emergency
services. They report earning more trust and respect from their communities due to the ability
to communicate with and coordinate emergency services. Conversely, acceptability for clients of
birth notification may be reduced by sociocultural norms, such as the extent to which stillbirths,
births to unmarried mothers or maternal deaths are acknowledged in communities. The evidence
also points to the potential costs of notification as a barrier and to the need to demonstrate the
advantages of birth or death notification to communities.
Feasibility
The qualitative evidence highlights several feasibility issues including, the need for adequate local
staffing and for strong underlying health and civil registration system infrastructure, resources
and processes. Health workers’competing priorities and lack of adequate incentives may affect
the successful adoption of these strategies. Inadequate attention is sometimes given to legal
frameworks governing civil registration, and governments may need to modify these frameworks to
allow to allow new types of health care cadre and other key informants to notify births and deaths.
Strong underlying health and civil registration system infrastructure, resources and processes are
necessary to achieve the impact of using mobile devices for birth and death notification.

Resource use
No evidence on resource use was identified. Resource use considerations are listed within the
evidence-to-decision framework in Web Supplement 1.
Gender, equity and human rights
The qualitative evidence indicates that while birth and death notification via mobile devices can
help to reach under-registered populations, there may be inequities in the implementation of this
intervention that are related to the availability of supportive infrastructure (network connectivity,
for example), literacy in the use of information and communications technologies (ICT), and
available funding resources.
Recommendation and justification/remarks
Birth notification via mobile devices
recommendation 1
WHO recommends the use of birth notification via mobile devices under these conditions:
ȺȺ in settings where the notifications provide individual-level data to the health system and/or a civil
registration and vital statistics (CRVS) system, and
ȺȺ the health system and/or CRVS system has the capacity to respond to the notifications.
Responses by the health system include the capacity to accept the notifications and trigger appropriate
health and social services, such as initiating of postnatal services.
Responses by the CRVS system include the capacity to accept the notifications and to validate the
information, in order to trigger the subsequent process of birth registration and certification.
Death notification via mobile devices
(Recommended only in the context of rigorous research and
in specific contexts or conditions)
recommendation 2
WHO recommends the use of death notification via mobile devices under these conditions:
ȺȺ in the context of rigorous research, and
ȺȺ in settings where the notifications provide individual-level data to the health system and/or a CRVS
system, and
ȺȺ the health system and/or CRVS system has the capacity to respond to the notifications.
Responses by the health system include the capacity to accept the notifications and trigger appropriate
health and social services.
Responses by the CRVS system include the capacity to accept the notifications and to validate the
information, in order to trigger the subsequent process of death registration and certification.

justification/remarks
Birth notification
ȺȺ The guideline development group (GDG) acknowledged the limited evidence but emphasized that
birth notification represents a vital first step in a care cascade that can ultimately lead to increased
and timely access to health services and other social services. The GDG also believed that the use of
mobile devices to perform this task was likely to provide a more expedient means of effecting the
notification and subsequent health services.
ȺȺ GDG members noted that while birth notification should not be viewed as a substitute for legal
birth registration, it could provide an opportunity to accelerate the registration by linking birth
notifications to national civil registration systems. The GDG also recognized that digital notification
of births could facilitate providing newborns with legal identity and future access to health and other
social services.
Death notification
ȺȺ The GDG remarked that a lack of information on deaths, especially deaths outside of facilities,
exacerbates data gaps in understanding the rates and causes of mortality.
ȺȺ The GDG therefore decided, while noting the limited evidence, to recommend death notification via
mobile devices in the context of rigorous research and where notifications can be linked to health
and/or CRVS systems.
ȺȺ The GDG noted that while data on deaths and causes of death are very useful for health planning,
they expressed concerns about adding the responsibility of CRVS-related functions to already poorly
resourced, understaffed and overburdened primary care health systems.
ȺȺ The GDG also recognized the sociocultural sensitives of communities notifying about deaths through
digital devices and recommended that further research be taken to understand these considerations.
Remarks that apply to both birth and death notification
ȺȺ It should also be noted that increases in the notification of births and deaths would also require that
civil registration services have, in turn, the capacity to manage a higher demand for registration and
certification services.
ȺȺ The ability for the health system and/or CRVS system to respond and act appropriately on the birth
and death notification was seen as a critical component for successful implementation. If such
linkages are not in place, the notification of birth and death events would not add any value and
would incur an additional cost to the system.

Implementation considerations
The specific implementation considerations that emerged from the literature and the GDG’s
deliberations for this intervention are listed below, organized where appropriate against the
framework outlined in the WHO/ITU National eHealth strategy toolkit (18). This is not an exhaustive
list of considerations; additional implementation resources and policy documents should be
consulted before taking up the recommendations.
Legislation, policy and compliance
ȺȺ The implementation of birth and death notifications needs to be in the context of national
policies, laws and guidelines. This may require modifications of legal frameworks to include
mobile notification in established practice and to enable cadres of informants such as
community health workers and community leaders to conduct the notifications if current
policies do not already provide for this.
ȺȺ Consider whether changes to legal frameworks will be needed to allow birth and death
notification to occur via mobile device or be carried out by new groups of health workers or
other cadres, as mentioned above, and how this would be linked to the issuance of birth/death
certificates. For example, consider whether there will need to be any modifications to existing
processes to accommodate signatures and approvals currently conducted on paper-based
forms. This review and modification should take place in the context of a broader legal review
of CRVS-related laws and regulations and would require collaboration among the institutions
that cover the health sector, civil registration sector and the local governments.
ȺȺ Consider the specific data storage, privacy and confidentiality issues. Implementers should
understand, for example, the implications and necessary regulations if the database of notified
births and deaths is also being held by mobile network operators, and the potential for
commercial uses of the data. Additionally, a relevant authority needs to ensure the right to data
protection by monitoring and enforcing a set of data protection laws.
Services and applications
ȺȺ Consider establishing mechanisms to prevent duplicate notifications. Unique identification
can be used to address this (for example, by issuing national identities; possibly identification
of the parents). Where national IDs are not available, consider an interim measure of IDs being
provided by health facilities, drawing from codes in master facility lists. Implementers may also
want to consider local de-duplication processes, such as using routine coordination meetings
across health workers to de-duplicate birth/death notifications before they are transmitted to
the civil registrar.

Workforce
ȺȺ When developing birth and death notification systems, consider mechanisms to ensure the
completeness of the data, and whether demand-generation activities are needed to incentivize
reporting by explaining its benefits. Implementers should be aware, however, of any reporting
targets placed on health workers, and ensure birth and death data are validated before being
released to the civil registration system.
ȺȺ Consider how best to ensure the quality and timeliness of birth and death data, for instance
by checking on low performers identified through digital performance data or spot checks.
Other ways to help improve data quality include standardizing the definitions associated with
reporting birth and death events, such as for stillbirths, and making these definitions accessible
to those inputting the data.
ȺȺ Implementers should note that increases in notification would in turn require that the health
system and civil registration services were prepared to absorb higher demand for registration.
This is a potential bottleneck in the registration and validation process and could deter
populations from continuing notifications.
Infrastructure
ȺȺ Consider how to improve accessibility and shorten the connection between the health workers
or communities providing the notifications and the CRVS sector undertaking the registration.
Consider, for instance, increasing the number and proximity of registration service points, and
look at the use of digital systems to speed up the registration process at these points.
Considerations for equity, gender and human rights
ȺȺ Explore sociocultural barriers associated with communicating about births/deaths and address
the way these dynamics will influence notifications via digital devices.
ȺȺ Consider linking birth notification to health services that have high coverage, such as
immunization services or health facilities that offer very high rates of institutional delivery. It
is important, however, to consider whether an increase in notifications can be absorbed by the
civil registration system.

Availability of commodities and equipment
Ensuring availability of commodities and equipment
3.3 Availability of commodities:
stock notification and commodity
Background
The availability of health commodities at point of services is critical to strengthening the quality
of care and supporting the pillars of universal health coverage (UHC) (64). Health commodities
include health products, and health and medical supplies that may be needed for the provision of
health services, including medicines, vaccines, medical supplies such as contraceptives dressings,
needles and syringes, and laboratory/diagnostic consumables (65,66). Various high-level initiatives,
including the UN Commission on Life-Saving Commodities for Women’s and Children’s Health,
have advocated equitable access to life-saving medicines and other health commodities (67,68).
Stock-outs of critical medical commodities remain an issue, however, particularly in rural settings,
where infrastructural limitations and geographical barriers can obstruct access to commodities at
the point of care.
The rapid global expansion of mobile devices has emerged as providing a potential opportunity for
mitigating the challenges of commodity distribution and stock-outs. Approaches can include the
use of communication systems such as text messaging (SMS) and data dashboards to manage and
report on supply levels. Specific examples by which mobile tools may be used to improve supply-
chain management include to track inventories of health commodities, notify their stock levels,
forecast demand for commodities, monitor cold chain-sensitive commodities, and manage the
distribution of health commodities (13).
Although broader initiatives to strengthen logistics management information systems are ongoing
(69), this guideline question reviewed the added value of extending the systems via mobile devices
to address commodity management at primary health care levels.

The following is a summary of the evidence on stock notification and commodity management
via mobile devices. Web Supplement 1 provides the full evidence-to-decision framework for this
intervention, detailing the available evidence on effectiveness, acceptability, feasibility, resource
use and implications for equity, gender and rights.
Effectiveness
There is limited available evidence on the effectiveness of and resources required as the certainty
of the evidence was assessed as very low.
Acceptability
The qualitative evidence suggests that access to digital data on stock availability at all levels
of the health system may be useful by health system managers as it allows them to respond to
anticipated stock shortages and ensure ongoing supply of needed health commodities. Staff at the
subnational levels may be concerned, however, about the data at their level becoming available
simultaneously with those at the national level since this would take away their opportunity to
contextualize the data or to explain shortcomings in stock availability.
Feasibility
Barriers to optimal implementation of stock notification and commodity management via
mobile devices include an underlying lack of stock at national or district level and a mismatch
between national ordering routines and local needs. The qualitative evidence on the feasibility of
digital health interventions, more broadly, also highlights challenges including those of network
connectivity, access to electricity, usability of the device, sustaining training and support to health
workers using the digital tools, and system integration.
Resource use
The qualitative evidence on gender, equity and human rights concerning digital health
interventions suggests health workers based in peripheral facilities and rural communities may
find these interventions helpful in overcoming geographical barriers and linking to the broader
health system, including when communicating about stock levels. Health workers in these settings
may be more likely to experience poor network coverage and access to electricity, though, and may
have lower levels of training and literacy in the use of technologies and fewer resources, including
limited access to the mobile devices that may be needed.

stock notification and commodity
Recommendation 3
WHO recommends the use of stock notification and commodity management via mobile
devices in settings where supply chain management systems have the capacity to respond
in a timely and appropriate manner to the notifications.
Justification/remarks
ȺȺ Despite the limited evidence on effectiveness and the identified feasibility barriers,
the guideline development group (GDG) felt that the use of mobile devices was likely
to provide a more expedient means of effecting stock notifications and ensuring
the subsequent availability of commodities at the point of services. This, in turn,
may increase the ability of health services to manage health issues in a timely and
appropriate way.
ȺȺ The GDG also assessed stock notification via mobile devices to be a relatively low-risk
intervention with potentially high impact, including the potential to save resources
through an improved allocation of commodities and reduced wastage. The GDG further
believed that the availability of timely stock data would increase transparency and
promote accountability.
ȺȺ Addressing the identified barriers to implementation as well as ensuring responsiveness
to the stock notifications were seen as critical ways to build trust and drive the effective
use of the digital intervention. If there are no mechanisms for health managers to
respond to the incoming data, or a lack of infrastructure or financial resources to
purchase new commodities, the gathering of stock data and issuance of notifications
would not add any value and would incur an additional cost to the system.
ȺȺ Although the condition within this recommendation requires that the health system be
responsive to the stock notifications, the GDG also remarked the importance of building
the capacity of weaker health systems so that this intervention may be used effectively.
Linkage with other WHO recommendations
This discussion aligns with recommendation 15 of the WHO guideline on health policy and
system support to optimize community health worker programmes, which recommends the use
of mobile health technology to support supply chain functions, including adequate reporting,
to enhance the availability of health commodities (17).

consulted before taking up recommendations.
ȺȺ Ensure there is no harm or reprisal to health workers for reporting stock-outs or wastage;
instead, the emphasis should be on explaining the benefits of reporting stock-outs so that they
can be addressed. To motivate continued reporting, ensure that some action is possible when
stock-outs are reported.
Standards and interoperability
ȺȺ Prioritize integrating notifications with existing data reporting systems, including national or
subnational information management systems where available, such as supply chain, logistics
and warehouse management information systems. Consider integrating the stock notification
system with a data dashboard that displays the notification, receipt of commodity at the
station and action taken among other data for ensuring transparency.
Workforce
ȺȺ Consider the need for training at all levels of the health care system, including the training of
health workers to send stock reports, of support staff such as cold-chain technicians to manage
stock and of facility workers to assess stock levels. Training should be reinforced by the basic
processes of inventory management and stock distribution. Since the management staff at
national and subnational levels make decisions on whether or not, according to the data, to
supply health facilities and health workers with stock replenishments, the introduction of the
digital system should also be accompanied with refresher training on the basic processes of
supply chain management. Training should include the use of the technology, such as the use of
text messages for the notification and the use of data dashboards.
ȺȺ When designing digital systems for stock notification, consider how the system can be made
easy to use, with effective display of the data through fact sheets and simple graphical and
tabular illustrations.
ȺȺ Ensure that the digital systems and ordering routines are flexible enough to respond to local
needs. For instance, where systems deal with quarterly stock orders, ensure they can also
accommodate unexpected or seasonal needs.

Availability of human resources
Ensuring availability of human resources
Accessibility of health facilities
Ensuring access to health facilities
3.4 Accessibility of health facilities and
human resources for health:
client-to-provider telemedicine
Background
Despite progress in addressing health workforce shortages, challenges in the equitable access to
health workers serves as a major hindrance to achieving the full requirements of effective coverage
of human resources for health (70). Geographical inaccessibility and the preference of health
workers for working in urban environments are among some of the well-documented reasons for
imbalances in the distribution of health workers (71). While there is a wide range of ongoing efforts
to reduce inequities in access to health workers, including incentives and alternative approaches
to training, digital approaches such as telemedicine have also been explored as a mechanism of
making health services available to underserved communities (71).
Within the WHO/ITU National eHealth strategy toolkit, telemedicine is defined as supporting
“the provision of health care services at a distance” (18). Although other definitions elaborate on
telemedicine as the use of ICT for medical diagnostic, monitoring and therapeutic purposes at
a distance (72–75), the driving principle is centered on the provision of remote clinical support
as a means of overcoming geographical barriers (72). Telemedicine can function between clients
and health workers who are separated by distance, as well as among health workers based in
different locations. The type of exchange between these actors varies and may include remote
consultations, remote monitoring of vital signs or diagnostic data, and the transmission of medical
files such as images for review, commonly referred to as “store and forward” (72–75).
In 2010, WHO reported extensively on the global status of telemedicine, including factors affecting
its uptake in low- and middle-income settings (72). In more recent years, the emergence of mobile
technologies has shifted the landscape, triggering new considerations for connecting clients
and health workers (3). This guideline question builds on this preceding resource from WHO and
examined the evolved use of telemedicine via mobile devices between clients and health workers.

The following is a summary of the evidence on client-to-provider telemedicine. Web Supplement 1
provides the full evidence-to-decision framework for this intervention, detailing the available
evidence on effectiveness, acceptability, feasibility, resource use and implications for equity,
gender and rights.
Effectiveness
The evidence on effectiveness suggests that this intervention may improve some outcomes,
such as fewer unnecessary clinical visits, reduced mortality among individuals with heart-
related conditions, exclusive breastfeeding, and increase health-related quality of life assessed
1–6 months after the intervention. However, it may make little or no difference on other
outcomes, such as hospital admissions for heart-related conditions or older individuals receiving
home-based care.
Acceptability
The qualitative evidence suggests that health workers appreciate the ability to offer immediate
care, to follow up on missing clients and offer informed care, advice and emotional support to
clients, even when physical contact is not possible. However, health workers feel that some cases
still warrant face-to-face contact and are also concerned that loss of face-to-face communication
will change the health worker–client relationship and lead to poorer quality care. Health workers
may also be concerned about having to work beyond their clinical capacity and about potential
issues of clinical liability.
From the client’s perspective, the qualitative evidence suggests these individuals may appreciate
being able to communicate with health workers from their homes and see telemedicine services
as offering reassurance and increased access and the consistency and continuity of care that it can
offer. Some clients may also feel that telemedicine services have increased their independence
and self-care, although some health workers may be concerned about clients’ ability to manage
their own conditions.
Feasibility
The qualitative evidence on the feasibility of digital health interventions, in general, highlighted
challenges related to network connectivity, access to electricity, usability of the device, sustaining
training and support to health workers using the digital tools, concerns about data privacy and
obtaining informed consent.
Resource use
The evidence on resource use was assessed as having very low certainty. Resource use
considerations are listed within the evidence-to-decision framework in Web Supplement 1.

This intervention may positively impact on equity by facilitating access to health services,
particularly for individuals who speak minority languages. It also may reduce the burden of
travel, particularly for people with caring or work responsibilities and those living far from health
facilities. However, access to telemedicine services may be difficult for other groups, though,
including people with hearing impairments or poor digital literacy.
Client-to-provider telemedicine
Recommendation 4
WHO recommends client-to-provider telemedicine:
ȺȺ under the condition that it complements, rather than replaces, face-to-face
delivery of health services; and
ȺȺ in settings where patient safety, privacy, traceability, accountability and security
can be monitored.
In this context, monitoring includes the establishment standard operating procedures that
describe protocols for ensuring patient consent, data protection and storage, and verifying
health worker licenses and credentials.
The guideline development group (GDG) felt that despite the mixed available evidence on
effectiveness spanning a wide range of health conditions, client-to-provider telemedicine has
the potential to expand access to health services. It may also potentially reduce the burden of
travel and decrease inequities for populations that have difficulties in accessing health services
through conventional approaches.
ȺȺ This recommendation recognizes that while telemedicine may enhance access to health
services, it should not be used to replace or detract from efforts to strengthen the health
workforce.
ȺȺ The establishment of standard operating procedures and mechanisms to ensure
patient safety, privacy, traceability and accountability of services was deemed to be
a necessary condition to mitigate the potential risks and harms of implementing this
recommendation.

ȺȺ Clarify the legal framework for the implementation of telemedicine, including relating to
the licensing and regulation of telemedicine health workers. The legal framework for remote
consultation should also consider cross-border consultations in which the health worker is
based in another country or jurisdiction.
ȺȺ Clarify clinical protocols to explain what can and cannot be done in the remote consultation.
For example, determine what type of cases still warrant face-to-face contact. Consider whether
it is possible or desirable for clients to meet health workers in person before connections are
made over digital services.
ȺȺ Explore whether changes in regulations are necessary to support any changes needed to health
workers’ scopes of practice. Develop policies and protocols to clarify the liability issues of
health workers using telemedicine systems.
ȺȺ Explore reimbursement models and mechanisms of integrating client-to-provider telemedicine
within existing service delivery models.
ȺȺ Ensure that there are mechanisms for documenting and tracing past exchanges and decisions
made during consultations.
Workforce
ȺȺ Ensure that use of the technology does not impact negatively on the relationship between
client and health worker, particularly when users are learning about the technology and how to
operate the devices. Extensive training on the technology and operating the device should be
done before introducing the system for use directly with clients.
ȺȺ Ensure that health workers remain able to use their own skills, judgement and knowledge
within the changed context.
ȺȺ Develop guidelines in collaboration with health workers that protect them from clients
contacting them outside of normal working hours, such as in the context of emergencies or
other considerations. If this contact is encouraged or expected, how can it best be managed to
avoid overwhelming the health worker? Will health workers be compensated for this type of
client support?
ȺȺ Involve the relevant professional bodies as well as the health workers and clients in the
planning, design and implementation of the telemedicine programme to ensure that their
needs and concerns are met, such as to educate health workers on the legal frameworks
governing telemedical exchanges.

ȺȺ Pay special attention to the needs, preferences and circumstances of particularly disadvantaged
or hard-to-reach groups, including people with low literacy or few digital literacy skills, people
with limited control over or access to mobile devices, people speaking minority languages,
migrant populations in new settings, and people with disabilities such as sight or hearing
impairment.
ȺȺ Consider how services can be made available to people with disabilities such as sight or hearing
impairments, with poor access to electricity or poor network coverage, who cannot afford
mobile devices or charges to use them, and people who have limited autonomy, for example
because their access to devices is controlled by another person. Strategies to increase access
to telemedicine in these cases may be provided through public kiosks or outreach through
community health workers, as examples.
ȺȺ Consider using telemedicine to link clients who speak minority languages to health workers
who also speak the language.
Availability of human resources
Ensuring availability of human resources
Accessibility of health facilities
Ensuring access to health facilities
3.6 Accessibility of health facilities and
human resources for health:
provider-to-provider telemedicine
Background
Access to qualified health workers with the appropriate competencies, skills and behaviours is an
even greater obstacle to improving health outcomes than the availability of health workers (70,71).
Geographical inaccessibility and the unequal distribution of health workers also contribute to
limitations in the effective coverage of human resources for health (62). Digital approaches, most
notably telemedicine between different types of health workers, have emerged as a potential
way to overcome the barriers of long distances to qualified health workers and shortages in their
numbers.
Provider-to-provider telemedicine, as with client-to-provider telemedicine, facilitates the
provision of health services at a distance and is primarily used to link less skilled health workers
with more specialist ones (72). The communication between health workers may be made for a

variety of reasons, including to get assistance with diagnoses, to remotely monitor clients’ health
status through vital signs and to conduct case-management consultations. This communication
between health workers may occur asynchronously through the exchange of video and image files
to be reviewed later (also referred to as store-and-forward exchanges) or synchronously in real-
time exchanges (13,18,72–75).
Although telemedicine is one of the more established forms of ICT-enabled health service delivery
(72), this guideline question expands on the existing evidence base, particularly in light of the
advances in facilitating health workers’ exchanges via mobile devices.
The following is a summary of the evidence on provider-to-provider telemedicine. Web
Supplement 1 provides the full evidence-to-decision framework for this intervention, detailing
the available evidence on effectiveness, acceptability, feasibility, resource use and implications for
equity, gender and rights.
Effectiveness
The evidence suggests that provider-to-provider telemedicine may improve health worker
performance, reduce the time for clients to receive appropriate care or follow-up, and decrease
length of stay among individuals visiting the emergency department. However, the intervention
may make little or no difference to other health status and well-being outcomes such as clinical
improvements in individuals.
Acceptability
The qualitative evidence suggests that health workers appreciate the opportunity to communicate
with each other and reduce their professional isolation. In particular, lower-level health workers
noted how telemedicine services allowed them to access advice from higher-level health workers,
which they saw as enabling better quality of care and client satisfaction. While some health workers
may perceive provider-to-provider telemedicine as supportive, others may note challenges in
collaboration, and concerns about liability and loss of control during the provision of care.
Feasibility
The qualitative evidence on the feasibility of digital health interventions, in general, highlights
training and support to health workers using the digital tools, concerns about data privacy and
obtaining informed consent.
Resource use
The evidence on resource use was assessed as having very low certainty. Resource use
considerations are listed within the evidence-to-decision framework in Web Supplement 1.

The qualitative evidence on provider-to-provider telemedicine suggests that this intervention may
improve equity by enabling health workers to facilitate access to higher-level care on behalf of
their clients. Yet poor access to the digital technology, or the personal expenses associated with its
use, may exclude some health workers, and thereby their clients, from these services.
Provider-to-provider telemedicine
Recommendation 5
WHO recommends provider-to-provider telemedicine in settings where patient safety,
privacy, traceability, accountability and security can be monitored.
In this context, monitoring includes the establishment of standard operating procedures that
describe protocols for ensuring patient consent, data protection and storage, and verifying
health worker licenses and credentials.
ȺȺ The guideline development group (GDG) noted that provider-to-provider telemedicine
has the potential to improve access to quality care and to reduce the isolation of health
workers working in remote settings.
ȺȺ Although the cost of the telemedicine system may vary depending on the modality used
(exchange of image files, voice calls, remote monitoring), the GDG felt that provider-to-
provider telemedicine could support care delivery by peripheral health workers.
ȺȺ Due to concerns about liability issues, the GDG suggested that standard operating
procedures/protocols be established to ensure patient safety, privacy, traceability
and accountability of services and to mitigate the potential harms of implementing
provider-to-provider telemedicine.
ȺȺ It was also noted that the nature of telemedicine is changing and that a wide range of
delivery channels are being used across health workers to facilitate communication
exchanges.

ȺȺ Explore whether changes to licensing and legislation are necessary to support any changes in
health workers’ scopes of practice. Clarify liability issues for health workers using telemedicine
systems and determine what can and cannot be done during remote consultations; the
approach should not be a substitute for the adequate training of health workers.
ȺȺ Ensure a clear legal framework for the implementation of telemedicine, including the licensing
and regulation of care health workers using it. Additional clarifications are also required in
cases of cross-border telemedicine, in which consultations are occurring across different
jurisdictions.
ȺȺ Ensure that there are mechanisms for documenting and tracing past exchanges and decisions
made during consultations.
Interoperability and standards
ȺȺ The use of telemedicine requires that the health worker can access the patient’s relevant
clinical history. Integration with digital health record systems that can be accessed by the
health worker and in which the patient’s identity can be verified may be considered as a way to
facilitate continuity of care.
Workforce
ȺȺ Ensure that the distribution of roles and responsibilities between different health workers is
clear, including through regulations and job descriptions.
ȺȺ Explore whether changes to health worker salaries or incentives are needed to reflect any
changes in scopes of practice.
ȺȺ Build trust between professionals considering establishing links between facilities across
institutions, for example through twinning programmes.
ȺȺ Develop protocols to educate health workers in the use of the technology. (More details in
Chapter 4.3 – ‘Overarching implementation considerations’)

Continuous coverage
The extent to which clients receive the full course of intervention required to be effective
Contact coverage Proportion of clients who have contact with relevant facilities,
providers and services among the target population
3.7 Contact and continuous coverage:
targeted client communication
for behaviour change
related to sexual,reproductive,maternal,newborn,
child and adolescent health
Background
Targeted client communication2
– defined as the transmission of health content or information
to a specific audience based on their health status or demographic profile (13) – represents an
approach for engaging with individuals to increase their knowledge about health and health-
seeking behaviours, about where to find or how to access services, or for helping to retain
them within health services when follow-up is needed. This includes the transmission of health
information to individuals about health promotion, for spreading awareness of services and
behaviours, transmission of reminders about services or treatments to encourage adherence to
recommended practice, and transmission of notifications about diagnostic results (13). Using
registered phone numbers or other contact information, the delivery of health content to
individuals can be via a range of digital channels, including text messaging, voice, interactive voice
response, multimedia applications and games (apps on mobile devices), and social media.
Several WHO guidelines have explored the use of targeted client communication via mobile
devices as a potential tool to improve medication adherence. Most notably, the 2016 Consolidated
guidelines on the use of antiretroviral drugs include a recommendation on the use of text messaging
as part of a package of interventions to support adherence to antiretroviral therapy (15). Similarly,
the 2017 Guidelines for treatment of drug-susceptible tuberculosis and patient care also recommend
the use of text messages and voice calls to support health education and treatment adherence (16).
Building on this previous work, this guideline question reviews the use of targeted client
communication via mobile devices across a broader range of health topics and populations of
interest for sexual, reproductive, maternal, newborn, child and adolescent health (SRMNCAH).
Note that the use of targeted client communication in the prevention and management of
noncommunicable diseases will be examined in a subsequent version of this guideline.

The following is a summary of the evidence on targeted client communication via mobile devices.
Web Supplement 1 provides the full evidence-to-decision framework for this intervention,
Effectiveness
The evidence on effectiveness suggests targeted client communication may have positive impacts
on some behaviours and health outcomes, such as: oral contraception use by adolescents, modern
contraception use by adults, adherence to antiretroviral medications, attendance of antenatal care
appointments, taking iron and folate tablets during pregnancy, skilled birth attendance, receipt of
childhood vaccinations, and attendance of HIV appointments among exposed children.
However, the evidence also indicates that targeted client communication may make little or no
difference to other outcomes, such as: health status as assessed by CD4 count and adherence to
prenatal antiretroviral medication.
The evidence on targeted client communication also suggests the intervention has some
unintended negative consequences, such as women experiencing physical violence in the context
of receiving targeted communications for sexual and reproductive health (SRH) services.
The certainty of the evidence was assessed as very low for some outcomes such as: adherence
to antiretroviral medication and attendance for STI/HIV testing among adolescents, breast and
cervical cancer screening; and women’s attendance for neonatal appointments.
Acceptability
The qualitative evidence suggests that targeted client communication is generally acceptable to
individuals, but that some population subgroups have concerns about the confidentiality of health
information, particularly for sensitive health issues such as HIV infection and other aspects of SRH.
Some clients describe digital targeted client communication programmes as providing them with
support and connectedness. The fact that someone is taking the time to send them messages can
make clients feel like someone is interested in their situation, invested in their well-being and
cares about them. Some clients describe this as leading to feelings of encouragement, increased
self-confidence and self-worth, and describe the messages as providing support, guidance and
information, giving a sense of direction, reassurance and motivation. Some clients also feel that
the sense of caring and support that they receive from health workers through these types of
programmes has a positive influence on their relationship with their health worker.

However, clients who are dealing with health conditions that are often stigmatised or personal
(e.g. HIV, family planning and abortion care) worry that their confidential health information will
be disclosed, or their identity traced due to their participation in these types of programmes. This
was noted particularly for vulnerable populations, including adolescents and pregnant women
living with HIV, in which the transmission of sensitive health information could disclose their
health status or compromise their privacy when seeking health information and services.
Clients’ perceptions and experiences of digital targeted client communication are influenced
by characteristics of the content; the format; and the delivery mechanisms. The evidence also
indicates that access to and use of targeted client communication may be particularly difficult for
certain groups of individuals, such as people with low literacy or with limited or controlled access
to mobile devices.
Feasibility
The qualitative evidence on the feasibility highlights a number of constraints. These include
reliable network connectivity, access to electricity and mobile devices, and the availability of
mechanisms to obtain informed consent when enrolling clients into the service. Health systems
may experience challenges when attempting to communicate with clients who regularly change
their phone numbers without informing the health worker or clients who have poor access to a
mobile device.
Resource use
The evidence suggests targeted client communication via mobile devices may use fewer resources
than non-digital interventions.
The qualitative evidence suggests targeted client communication may be particularly difficult
for certain population groups, including individuals with poor access to network services
or electricity; with limited or controlled access to mobile devices, particularly women and
adolescents; individuals who speak minority languages or have low literacy skills or low digital
literacy skills; or individuals with conditions that cause them to be particularly concerned about
the confidentiality of information exchanged via digital devices.

Targeted client communication
via mobile devices
Recommendation 6
WHO recommends targeted client communication via mobile devices for behaviour change
regarding sexual, reproductive, maternal, newborn and child health, under the condition
that concerns about sensitive content and data privacy are adequately addressed.
Examples of ways to address sensitive content and data privacy include ensuring that
individuals are actively made aware of how to opt out of receiving the targeted client
communication.
ȺȺ The guideline development group (GDG) considered this intervention to offer the
potential to improve health behaviours and reduce inequities among individuals with
access to mobile devices. The GDG, however, highlighted that measures should be
taken to address inequities in access to mobile devices so that further inequity is not
perpetuated in accessing health information and services, including mechanisms
to ensure individuals who do not have access to mobile devices can still receive
appropriate services.
ȺȺ The GDG also raised the need to address potential concerns about sensitive content
and data privacy, including potential negative unintended consequences. This could be
done, for example, through mechanisms that actively allow individuals to opt out of
services.

Linkages with other WHO recommendations
The GDG noted that WHO has previously made recommendations related to targeted
client communication for improving HIV and tuberculosis medication adherence, which
contributed to the considerations for this recommendation. These previous recommendations
are listed below.
In the Consolidated guidelines on the use of antiretroviral drugs for treating and preventing HIV
infection (15), the following interventions demonstrated benefit (all with moderate-quality
evidence) in improving adherence and viral suppression:
ȺȺ peer counsellors
ȺȺ mobile phone text messages
ȺȺ reminder devices
ȺȺ cognitive-behavioural therapy
ȺȺ behavioural skills training/medication adherence training.
In the Guidelines for treatment of drug-susceptible tuberculosis and patient care (16), one or
more of the following treatment adherence interventions (complementary and not mutually
exclusive interventions) may be offered to patients on tuberculosis treatment or to health
workers:
ȺȺ tracers* and/or digital medication monitor (conditional recommendation, very low
certainty in the evidence)
ȺȺ material support to the patient (conditional recommendation, moderate certainty in the
evidence)
ȺȺ psychological support to the patient (conditional recommendation, low certainty in the
evidence)
ȺȺ staff education (conditional recommendation, low certainty in the evidence)
ȺȺ fixed-dose combinations and once-daily regimens (moderate-quality evidence).
This guideline also makes the following recommendations on options offered to patients on
tuberculosis treatment.
a. Community- or home-based directly observed treatment is recommended over health
facility-based directly observed treatment or unsupervised treatment (conditional
recommendation, moderate certainty in the evidence).
b. Directly observed treatment administered by trained lay health workers or health care
workers is recommended over directly observed treatment administered by family
members or unsupervised treatment (conditional recommendation, very low certainty
in the evidence).
c. Video-observed treatment may replace directly observed treatment when the video
communication technology is available, and it can be appropriately organized and
operated by health workers and patients (conditional recommendation, very low
certainty in the evidence).
* Tracers refer to communications with the patient, including via home visits, SMS text messages
or voice telephone calls.

ȺȺ Ensure that clients are actively made aware of how to opt out of receiving the targeted client
communication. Attention needs to be paid to ensure that consenting procedures clearly
communicate to the clients the intended uses of their data, including to the intentions to
continue contacting them, over what period of time, and their right “to be forgotten”, or opt out.
Procedures need to be in place to ensure that participants are not unduly pressured to provide
personal information.
ȺȺ Ensure that individuals know the messages are coming from a trusted sender such as a
government or health institution, health worker or other familiar entities worthy of their
attention.
ȺȺ Ensure that any sensitive content or personal data transmitted and stored are held on a secure
server with protocols in place for destroying the data when appropriate.
ȺȺ Effective digital communication relies on behaviour change to achieve the intended
impact. Such communication should be conducted in the context of a comprehensive
communications strategy so that messages received through mobile devices are reinforced by
other mechanisms. For example, digital messages should be consistent with the information
communicated by health workers, print media and other sources. Further considerations to
review when developing content for digital communication include the following.
→→ Consider the languages used for the content to reach the target audiences, including
whether they are in active spoken or written use.
→→ Ensure that messages are clear and simple. Avoid jargon, technical terms and shortened
forms of text. Consider testing to ensure that messages are understood as intended and that
any necessary colloquial translations are used.
→→ Consider the tone of the messages and whether clients are likely to perceive them as
friendly and motivational as opposed to shaming or frightening.
→→ Consider how the content can be tailored to the client, for instance by using their name,
local information or personalized reminders.

ȺȺ Consider whether to include two-way communication with clients to enable their interaction
and response to the health system.
ȺȺ Ensure that the content of the communication reflects the reality of the available commodities
and services. For example, encouraging women to seek family planning at their nearest health
facility is appropriate if a full range of contraception and advice is available there, including the
relevant commodities.
Infrastructure
ȺȺ Ensure the mode of content delivery is appropriate for the setting’s network connectivity.
For example, in contexts with low connectivity coverage, not all populations may be reached
through digital channels making use of multimedia or mobile app-based communications.
Consider offering messages in a variety of formats (text, audio and video) depending on the
setting and infrastructural limitations.
Equity and sociocultural considerations
ȺȺ Pay attention to the circumstances of people who have poor access to electricity or poor
network coverage, people who cannot afford a mobile device or voice and data charges and
people who have limited autonomy, for example because their access to phones is controlled
by another person. For the latter case, the GDG felt that the programme should target content
accordingly and ensure that users were not put at increased risk.
ȺȺ Develop concurrent initiatives where such inequity exists so that individuals who do not have
access to mobile devices can still receive appropriate services.
ȺȺ Pay particular attention to the needs, preferences and circumstances of especially
disadvantaged or hard-to-reach groups, including people with low literacy or few digital
literacy skills, people speaking minority languages, migrant populations in new settings,
people affected by emergency situations and people with disabilities such as sight or hearing
impairment. Also consider any demographic characteristics, sexual identity or preferences
that could put a targeted population at greater risk and ensure that the way the information is
provided and accessed is sensitive to this.
ȺȺ Ensure there are little, or no charges tied to the programme, for instance those associated with
downloading apps or sending or receiving the content. Implementers may need to negotiate
with mobile network operators and other partners to determine options for subsidizing
communication costs or employing voucher systems.

Effective coverage
The proportion of individuals receiving satisfactory health services among the target population
3.8 Effective coverage:
Background
Quality of care, defined as the “degree to which health services for individuals and populations
increase the likelihood of desired health outcomes and are consistent with current professional
knowledge”, is a foundational component of universal health coverage (76). Quality of care has
consistently been documented as suboptimal, particularly across low- and middle-income
countries. Commonly cited reasons for poor quality of care have included health workers’
inaccurate diagnosis, inappropriate or unnecessary treatment, inadequate or unsafe clinical
practices, along with a range of other systemic issues such as insufficient commodities and
infrastructural limitations (76).
Although low quality of care stems from numerous deeply rooted health system challenges,
decision support tools that offer guidance to health workers have been leveraged as a mechanism
to augment adherence to recommended clinical practices (77–80). In their digital form, decision
support systems for health workers are defined as electronic systems designed to aid directly
in decision-making, in which characteristics of individual patients are used to generate
patient-specific assessments or recommendations that are then presented to clinicians for
consideration (13,18). Digital decision support for health workers (13), also referred to as clinical
decision support systems (CDSS), may be used for a wide range of clinical interactions, including
diagnosis and treatment, to facilitate appropriate referrals, minimize errors in medication
prescription, and ensure the provision of thorough and accurate care (79). Functionally, decision
support tools may be designed to guide health workers through algorithms and rules based on
clinical protocols, provide the health worker with checklists for case management and referrals,
screen clients by risk or other health status and to assist in health worker activity planning and
scheduling (13).

The use of decision-support tools has been well established and is supported by some emerging
evidence (80). However, over the last decade, health worker decision support has transitioned
from being operated on fixed computerized systems to mobile devices, which provide unique
opportunities for point-of-care assessment, diagnosis and management. Furthermore, most
health care systems in low- and middle-income countries, especially in rural areas, do not have the
required infrastructure for desktop computer-based decision support systems and are increasingly
investing in making these tools accessible via mobile devices.
This guideline question will explore the added value of digital decision support tools available at
primary health care levels and accessible to health workers via mobile devices. Furthermore, as
the function of this digital health intervention is broadly applicable across programmatic areas,
the guideline question will explore the use of such digital job aids across health conditions within
primary care settings.
The following is a summary of the evidence for decision support for health workers via mobile
devices. Web Supplement 1 provides the full evidence-to-decision framework for this intervention,
Effectiveness
There is limited evidence on the effectiveness of health worker decision support via mobile
devices directed to clinical health workers. For the intervention directed to community health
workers, the evidence suggests that this may have positive effects on individuals taking prescribed
medication but may make little or no difference to the individuals’ overall health status. When
directed to community health workers, decision support may make little or no difference to clients’
satisfaction with the information they receive.
Acceptability
The qualitative evidence suggests health workers find the intervention useful and reassuring for
guiding the delivery of care. However, some health workers perceive algorithms as too prescriptive,
and are concerned that they may lose their clinical competencies by blindly following treatment
algorithms. The evidence also suggests that clients find the intervention acceptable and enables
health workers to be more thorough when providing care.
Feasibility
The qualitative evidence on the feasibility of digital health interventions, in general, highlights
training and support to health workers using the digital tools.

Resource use
No evidence on resource use was identified.
The evidence on gender, equity and human rights on digital health interventions broadly suggests
health workers based in peripheral facilities and rural communities may find these interventions
helpful in overcoming geographical barriers and linking to the broader health system, including
to access clinical guidance. Health workers in these settings may, though, be more likely to
experience poor network coverage and access to electricity, may have lower levels of training and
literacy with digital technology, and may have fewer resources, including having limited access to
mobile devices.
accessible via mobile devices
Recommendation 7
WHO recommends the use of health worker decision support via mobile devices
in the context of tasks that are already defined as within the scope of practice for these
health workers.
ȺȺ The GDG expressed that the use of health worker decision support tools when used on
mobile devices may improve provision of services point of care. The GDG noted, however,
that decision support should not be used for tasks that are beyond the current scope of
practices as this may introduce the risk of health workers delivering services for which
they have not received adequate training, or of overwhelming the health workers with an
expanded set of tasks.
ȺȺ The GDG highlighted the importance of ensuring the validity of the underlying
information, such as the algorithms and decision-logics.
ȺȺ The GDG also acknowledged additional literature that was not assessed as part of this
guideline, on decision support systems via fixed/stationary digital devices. The GDG
felt that this evidence suggested the potential of such tools in improving patient/client
outcomes could be extrapolated to mobile use, which may offer additional opportunities
for settings where the infrastructure for fixed devices is weak.

Workforce
ȺȺ Health workers may find it helpful in increasing the acceptability to clients/patients of using
digital decision support if they explain that they will be using a digital device and seek clients’
permission before using them. Clients should also be made aware that the information from
the counselling may be saved and used at future visits to improve quality and continuity. Any
concerns with acceptability may be mitigated by, for example, health workers showing the
client the inputs and results or listening to the messages or videos together with them so that
the device does not become a barrier in the consultation.
ȺȺ Before using the decision support system, implementers should assess health workers’ skills
and knowledge to ensure that they have adequate capacity to obtain accurate data before
input, to avoid erroneous outputs.
ȺȺ Referral linkages might need to be strengthened to support possible increases in the number
of patients seeking care for previously undetected needs now being revealed by the decision
support system.
ȺȺ Check the relevance and quality of the decision support content (such as algorithms) and that
it aligns with evidence-based clinical guidance, such as WHO or national guidance. Engaging
expert groups on the clinical/health topic area may also be necessary as existing guidance may
not have sufficient clarity.
ȺȺ Ensure adequate time for testing all paths of the algorithm with any changes to the software.
This type of validation can be done through mechanisms such as an independent review and
using mock cases to test the intended output from the algorithms. Also consider built-in
mechanisms to update content remotely as algorithms evolve.
ȺȺ Both health workers and clients should understand that the support provided is based on
existing guidelines and policy. While health workers may deviate from the recommendations,
they should be clear about their rationale for doing so. Where possible, enable cases to be
documented in which health workers feel they need to deviate from the guidance proposed by
the decision support system.
ȺȺ Ensure that use of the device does not impact negatively on the relationship between patient
and health worker, particularly when the provider is learning to use the device. As above, this
may be helped, for example, by health workers showing patients the inputs and results or
listening to the messages or videos together with them so that the device does not put up
a barrier. Finally, pay attention to user experience so that correct use of the system is easy
for health workers and does not demand more time compared with alternative approaches
without it.

ȺȺ For the ease of viewing the patient’s health history, decision support tools are often integrated
with digital health records. See section 3.8 for the evidence and discussion surrounding the
combination of decision support with digital tracking of clients’ health status and services.
3.9 Multiple points of coverage:
digital tracking of clients’
health status and services
combined with decision-support and
targeted client communication
Background
The use of paper-based systems in the delivery of health services introduces a clerical burden
on health workers. Additionally, the ability for health workers to keep track of clients effectively,
and follow up on services, whether within the facility or in the community, is essential to the
continuity of care (12).
Digital tracking is the use of a digitized record to capture and store health information on clients in
order to follow-up on their health status and services received (13,40,81). This may include digital
forms of paper-based registers and case management logs within specific target populations,
as well as electronic patient records linked to uniquely identified individuals. Digital tracking
makes possible the registration and follow-up of services and may be done through an electronic
medical record (EMR) or other digital forms of health records. Digital tracking aims to reduce
lapses in continuity of care by stimulating timely follow-up visits and may incorporate decision
support tools to guide health workers at the point of care in executing clinical protocols, delivering
appropriate care, scheduling upcoming services and following checklists for appropriate case
management.
Digital tracking and decision support systems may also be linked with demand-side interventions
to engage clients/patients, such as through targeted client communication via mobile devices.
Targeted client communication in this context is defined as the transmission of targeted health
Effective coverage
Continuous coverage
The extent to which clients receive the full course of intervention required to be effective
Accountability coverage
The proportion of those in the target population registered into the health system

content or reminders to a specified population or to individuals within a predefined health or
demographic group (13).
This guideline has sought to understand the benefit of an integrated package consisting of three
different digital health interventions, to support health worker practices as well as to stimulate
client-side demand for health services and stimulate behaviour change.
This guideline reviewed the following intervention combinations:
(a) digital tracking with decision support
(b) digital tacking with targeted client communication
(c) digital tracking with decision support and targeted client communication.
The following is a summary of the evidence on the digital tracking of clients’ health status and
services (shortened to digital tracking), in combination with health worker decision support
and targeted client communication. Web Supplement 1 provides the full evidence-to-decision
framework for this intervention, detailing the available evidence on effectiveness, acceptability,
feasibility, resource use and implications for equity, gender and rights.
Effectiveness
(a) Digital tracking and decision support: The evidence on the effectiveness of digital tracking
combined with decision support suggests it may improve health service use and health outcomes,
such as: attendance of antenatal care appointments, taking iron tablets during pregnancy,
immediate breastfeeding, receipt of the third dose of polio vaccine, and use of postpartum
contraception six months after birth.
However, digital tracking combined with decision support probably makes little to no difference
on other outcomes, such as: the proportion of children under five who are vaccinated, proportion
of women who give birth in a facility, women breastfeeding exclusively for six months, or on the
proportion of women using contraception within six months of birth.
There was limited evidence on the effect of digital tracking combined with decision support on the
use of emergency visits for children under five and on the timeliness of receiving services, as the
certainty of this evidence was assessed as very low.
(b) Digital tracking with targeted client communication: No evidence was identified for this
intervention combination.
(c) Digital tracking with decision support and targeted client communication: There was limited
evidence in demonstrating the effectiveness of combining digital tracking with both decision
support and targeted client communication, as the certainty of this evidence was assessed
as very low.

Acceptability
The qualitative evidence suggests that most health workers see advantages to digital technologies
compared with paper-based systems. These include quicker recording of required client data and
services delivered, easier access to client data, easy correction of recording mistakes, and not
having to carry paper registers. Health workers are often reluctant, however, to use digital tracking
when they have to maintain both digital and paper-based systems, since this increases their work
burden.
Feasibility
There was limited evidence documenting the feasibility of these integrated interventions
specifically. Challenges have been highlighted, however, by the qualitative evidence on the
feasibility of digital health interventions in general, including those of network connectivity,
access to electricity, usability of the device, sustaining training and support to the health workers
using the digital tools, and system integration.
Resource use
The qualitative evidence on these digital health interventions suggests health workers based
in peripheral facilities and rural communities may find the interventions useful in overcoming
geographical barriers and linking to the broader health system. Health workers in these settings
may also, however, be more likely to experience poor network coverage and poor access to
electricity, may have lower levels of training and literacy with technology, and may have fewer
resources, including having poorer access to mobile devices.
digital tracking of clients’ health status
and services (digital tracking) combined
with decision support
Recommendation 8
WHO recommends the use of digital tracking with decision support under these conditions:
ȺȺ in settings where the health system can support the implementation of these
intervention components in an integrated manner; and
ȺȺ for tasks that are already defined as within the scope of practice for the health worker.

digital tracking combined with decision
support and targeted client communication
Recommendation 9
WHO recommends the use of digital tracking combined with both decision support and
targeted client communication under these conditions:
ȺȺ in settings where the health system can support the implementation of these
intervention components in an integrated manner; and
ȺȺ for tasks that are already defined as within the scope of practice for the health worker;
and
ȺȺ where potential concerns about data privacy and transmitting sensitive content to
clients can be addressed.
ȺȺ The guideline development group (GDG) recognized that this intervention package may
pose challenges, particularly in settings in which the health system may not be able to
manage the infrastructural and technical complexity of such a multifaceted intervention.
The GDG also felt that the intervention may require substantial upfront resource use but
believed that the intervention may reduce costs in the long term by transitioning away
from inflexible paper-based systems.
ȺȺ Despite the risk of increasing complexity by implementing a system with multiple digital
components, the GDG believed that implementing these interventions in an integrated
manner offered opportunities to (i) reduce health workers’ time spent on redundant
activities such as reporting; (ii) increase the timeliness and responsiveness of health
workers by linking data from client health tracking systems to the actions recommended
from decision support tools; and (iii) provide a more holistic view of the client and their
interactions with the health system.
ȺȺ While there is value in a multi-pronged digital intervention that simultaneously targets
supply side factors (i.e. decision support to health workers), and demand-side factors
(i.e. targeted client communication), the technical and human resource requirements
for such an intervention should be considered. The GDG suggests the three components
be implemented in a gradual manner, particularly in settings where the enabling
environment and infrastructure may not be sufficiently mature to support such a
multifaceted intervention.
ȺȺ In line with the separate recommendation on targeted client communication via mobile
devices (see section 3.6 for more detail), the GDG’s recommendation to combine it into
digital tracking is conditional on measures being taken to address inequities in access
to mobile devices and address concerns about sensitive content. Similarly, the inclusion
of the decision support component will require alignment to the tasks and scope of
practice for health workers to avoid potential harms and added burden (see section 3.7
for more detail).

These findings align with recommendation 11 of the WHO guideline on health policy and sys-
tem support to optimize community health worker programmes,which suggests that practising
community health workers“document the services they are providing and that they collect,
collate and use health data on routine activities, including through relevant mobile health
solutions” (17).
ȺȺ Accurate client/patient identification to facilitate the digital tracking of health services
across different facilities and health workers requires adequate policy and legal processes
and protections. This can include the use of a card-based or biometric-based identifier, as an
example, and having telecommunications infrastructure that is available consistently across
facilities and programmes.
Infrastructure
ȺȺ Consider whether the digital tracking would have adequate infrastructural support to
be maintained over time. The start-up costs and infrastructural requirements of a digital
tracking system tend to be higher than for paper-based interventions. When used appropriately
and effectively, the costs of digital interventions are amortized, and cost-savings may
materialize in the long run. However, in contexts where basic health infrastructure is limited,
including in human resources, digital tracking systems may be very resource-intensive to set
up and maintain.
ȺȺ The digital tracking should be linked to a system that provides a unique identity for each
individual. Such unique IDs help health workers search for clients, reduce the potential for
duplicate registration of clients in community and facility systems and ensure continuity
of care. This unique ID could, in turn, be linked to a local or national ID system to provide a
foundational digital identity that can facilitate longitudinal follow-up and linkages across
different levels of the health system and digital health interventions.

Workforce
ȺȺ Consider phasing implementations to avoid overburdening health workers. For example,
consider introducing integrated packages only once health workers have already been
implementing at least one of the interventions and are familiar with digital technologies.
ȺȺ Focus on introductory and ongoing training of health workers in using these tools, including
support for technical troubleshooting during the provision of care. Health workers may have
challenges in using technology during the provision of services, which can negatively impact
the quality of care, or result in the technology not being used. Use metrics to assess health
workers’ use of the digital system and identify opportunities to reinforce training.
Equity and sociocultural considerations
ȺȺ Inequities may be reduced for populations included within the digital tracking system because
it helps to ensure that they receive services. Inequities may arise, however, for those outside
of the digital tracking system whose service provision might not be accounted for. Such
inequity needs to be monitored during implementation. The problem can be addressed by first
enumerating the target population and so increasing the accuracy of the denominator by which
populations are eligible for services.
ȺȺ The digital tracking of individuals’ health status may be controversial in some circumstances,
for example among migrants or other groups who lack firm legal status in particular settings.
The extent to which such groups may trust tracking depends on who is doing the tracking
and how the information is likely to be used. It is important to take these concerns, and
local policies on digital identities, into account when designing a programme to ensure it
does no harm.

Effective coverage
3.10 Effective coverage:
digital provision of training
to health workers via mobile devices/mobile learning
Background
Broadly defined as the management and provision of educational and training content in digital
form for health professionals, electronic learning (eLearning) has emerged as one approach to
increasing health workers’ access to training and educational resources (18). More recently, the
widespread reach of mobile devices has prompted the use of such technologies to deliver training
content to health workers, also known as mobile learning (mLearning). Such training content may
be exchanged using channels such as SMS text messaging, the multimedia messaging service,
applications (“apps”), games, and other forms of digital modality (82). In particular, low- and
middle-income countries and remote areas with limited ICT infrastructure and geographical
barriers may seek to leverage mobile devices to maximize access to educational content and
continuing medical education (82).
Although the use of digital tools for strengthening the health workforce is referenced in several
WHO resources (15,70,71,83), these do not examine the specific considerations on digital health
worker training via mobile devices. This guideline question assesses the potential contributions
and implications of providing digital training and educational content via mobile devices/
mLearning, as part of complementary efforts to support workforce needs for in-service training
and continued education.

The following is a summary of the evidence on the provision of digital training and educational
content for health workers via mobile devices/mLearning. Web Supplement 1 provides the
full evidence-to-decision framework for this intervention, detailing the available evidence on
effectiveness, acceptability, feasibility, resource use and implications for equity, gender and rights.
Effectiveness
The evidence suggests that this intervention may increase health workers’ knowledge. However,
the effects of this intervention on other outcomes, including health workers’ performance, skills
and attitudes, is uncertain because there is no direct evidence, or the evidence is of very low
certainty.
Acceptability3
The qualitative evidence from medical and nursing students suggests that these users see a
number of advantages to mLearning tools, including the ease and portability of accessing materials
and ability to personalize content to their own needs. They may have some concerns, however, for
instance about the validity and accuracy of the information, as well as potential negative effects
when used during patient interactions.
Feasibility
The qualitative evidence on the feasibility of digital health interventions highlights challenges
related to network connectivity, access to electricity, usability of the device, sustaining training
and support to health workers using the digital tools.
Resource use
The qualitative evidence on digital health interventions broadly suggests health workers based
in peripheral facilities and rural communities may find these interventions helpful in overcoming
geographical barriers and linking to the broader health system. However, health workers in
these settings may also be more likely to experience poor network coverage and access to
electricity, may have lower levels of training and literacy with digital technology, and may have
fewer resources, including poorer access to the mobile devices that may be needed for some
programmes.
3 The systematic review of mLearning specifically explored factors influencing implementation of mLearning among both pre- and post-qualified
health workers. However, this review only included studies on nursing and medical students. The technical team extrapolated findings from this
review that would be relevant for health workers.

Digital provision of training and
educational content for health workers
via mobile devices/mLearning
Recommended
Recommendation 10
WHO recommends digital provision of training and educational content for health workers
via mobile devices/mLearning under the condition that it complements rather than replaces
traditional methods of delivering continued health education and in-service training.
ȺȺ Despite the availability of evidence primarily focused on improving health worker
knowledge, the guideline development group (GDG) felt that the potential benefits of the
intervention outweighed the potential harms.
ȺȺ The GDG also noted that mLearning offered an additional delivery channel for continuing
health education, and thereby expanding access to in-service training resources and
professional development opportunities to a broader set of health workers.
ȺȺ The GDG also considered the potential for cost savings for continued health education,
when compared with the costs of expanding face-to-face in-service training.
ȺȺ It should be noted that this intervention only applies to post-certification health workers
and used in the context of in-service training and continued health eduction.
The WHO guideline on health policy and system support to optimize community health worker
programmes suggests an emphasis on face-to-face learning for pre-service community health
workers, to be supplemented by eLearning on aspects where it is relevant (17).

Infrastructure
ȺȺ Consider network capacity and coverage especially if mLearning materials may be videos which
can my time consuming to download in certain settings.
ȺȺ Consider if health workers can earn credits for continuing education using these materials, as a
way of increasing their uptake.
Workforce
ȺȺ To increase the acceptability of mLearning devices, it may be important to improve awareness
among staff and supervisors about the value of portable devices and to develop ground rules or
codes of conduct for when and how devices should be used.
ȺȺ Similarly, it may be helpful to give patients explanations of device use, and to ask patients’
permission before using a device. Ensure also that use of devices does not impact negatively on
ȺȺ the relationship between health workers and clients, particularly if being used in the context of
service delivery, and especially when health workers are learning to use the devices.
ȺȺ Involve the relevant professional bodies, including national certification or institutional boards,
to ensure that the content of the mLearning programmes aligns with the current scopes of
practice and national training curriculums for health workers.
ȺȺ Ensure that the information is from a source that is considered trustworthy and credible by
health workers in your setting. For example, the information loaded on the mLearning system
should be based on validated content or should align with national or WHO clinical guidance.
ȺȺ Consider which types of training content are best delivered via mLearning channels and which
through other or mixed channels, including through in-person training.
ȺȺ Where available, mLearning materials should be curated and accredited as formal training
courses.
ȺȺ Ensure that the programme is user-tested among health workers, both those in practice and
those in training, to ensure that their needs and concerns are met.
ȺȺ Ensure that health workers can easily store content for future reference.
ȺȺ Consider how health workers can tailor the content to suit their specific needs. For instance,
develop content in a modular format so that health workers can select information for
particular review.

4. Implementation
considerations
Digital health has the potential to help address problems such as distance and access, but still
shares many of the underlying challenges faced by health system interventions in general,
including poor governance, insufficient training, infrastructural limitations, and poor access
to equipment and supplies. These considerations need to be addressed in addition to specific
requirements introduced by digital health. As the context will moderate the eventual impact
of digital health interventions, the broader health system and enabling environment become
especially critical.
4.1 Linking the recommendations
across the health system
While the recommendations included in this guideline are based on distinct digital interventions,
they all contribute to the health systems’ needs in different but interlinked ways. For health
system managers, the recommendation on digital stock notification aims to drive availability of
commodities at the point of services. From the clients’ and patients’ perspectives, this would
include ability to access health information and services more immediately, such as through
client to provider telemedicine and targeted client communication. Likewise, health workers need
to be accessible and adhere to practices for delivering high-quality care, through interventions
such as decision support and mLearning. Figure 4.1 illustrates the linkages across the different
recommendations and the interlinked ways that these digital interventions can cohesively
address health system needs.

Health workers can provide
appropriate and high quality care
Births are notified
and accounted
for to receive services
Individuals
can access
health services
and information
Health workers
are knowledgeable
about which services
to provide
Deaths are
notified and
accounted for
Health workers
are accessible
Health commodities
and supplies are
available at the
point of care
Health workers can
follow-up to ensure
individuals receive
appropriate services
Recommendation 1
Birth notification
ACCOUNTABILITY
Recommendation 2
ACCOUNTABILITY
Death notification
Recommended in the context
of rigorous research and
specific conditions
Recommendation 6
Targeted client
communication
DEMAND
Client-to-provider
telemedicine
SUPPLY
Recommendation 4
SUPPLY
Stock notification &
commodity management
SUPPLY
Recommendation 3
QUALITY
telemedicine
Recommendation 5
Recommendation 7
QUALITY
Health worker
decision support
digital tracking +
decision support
QUALITY
Recommendation 8
Recommendation 9
digital tracking +
decision support & targeted
client communication
QUALITY
Recommendation 10
provision of training
Recommended
QUALITY
Figure 4.1 Linkages of the recommendations across the health system

4.2 Implementation componets
Digital health implementations rely on a host of factors, and their success is often mediated by
issues intrinsic to the design of the implementation, as well as external factors related to the
enabling and ICT environment. The implementation of digital health interventions is broadly
predicated on the following critical components:
i. appropriate and accurate health content and information aligned with recommendation
practices (e.g. from health programme guidelines or evidence-based normative practices);
ii. the digital health intervention, consisting of the discrete digital functionality being applied to
achieve the health objectives; this guideline focuses on different digital health interventions;
iii. digital applications, which represent the software and communication channels that facilitate
the delivery of digital interventions combined with health content (e.g. text messaging,
software and information and communications technology [ICT] systems, or smartphone
applications “apps”); and
iv. ICT and enabling environment (e.g. governance, infrastructure, legislation and policies,
workforce, interoperability and digital architecture). See Figure 4.2 below, which was also
introduced in section 1.2 about the role of digital health in health system strengthening and
universal health coverage.
Gaps within these different implementation components can jeopardize the quality and impact of
the implementation. For example, the delivery of inaccurate health information poses a risk on the
health outcomes that may result from this implementation. Likewise, the inappropriate selection
of hardware, software and communication channels may present challenges to the usability and
reach of the implementation. Additionally, limitations in the maturity of the ICT and enabling
environment can prevent the uptake of the intervention and potentially strain the health system
by diverting resources and inducing fragmentation of services. Furthermore, these implementation
components should be designed appropriate to the local context based on intended user needs, in
reflection of the absorptive capacity of the health system, and the behavioural and organizational
changes that would be required to adapt to these digital interventions.

STRATEGY &
INVESTMENT
SERVICES &
APPLICATIONS
LEGISLATION,
POLICY, &
COMPLIANCE
WORKFORCE
STANDARDS &
INTEROPERABILITY
INFRASTRUCTURE
Health Content
Digital Health
Interventions
sector objectives
health content
+ +
Figure 4.2 Components contributing to digital health implementations

4.3 Overarching implementation considerations
Implementations need to be made appropriate to the local needs, intended users, and overall
ecosystem comprised of the ICT and enabling environment. The National eHealth strategy
toolkit produced jointly by WHO and ITU (18) provides useful considerations for assessing the ICT
and enabling environment and can be used to help countries in determining their readiness to
adopt the digital health interventions.
Table 4.1 Components of the ICT and enabling environment
Components of ICT and
enabling environment
Description
Leadership and
governance
This includes coordination mechanisms at the national level,
alignment with health goals and political support, and awareness and
engagement from stakeholders
Strategy and investment
This includes aligning financing with health priorities and ensuring
funding to achieve the objectives of the strategy
Legislation, policy and
compliance
This includes a legal, policy and enforcement policy environment to
establish trust and protection for individuals and industry
This includes the systems and functionalities that need to be in place
to enable stakeholders to access, use and share health information
Infrastructure
This includes the physical infrastructure, core services and
hardware (such as networks) that underpin a national digital health
environment. An example is identification authentication services
Standards and
interoperability
This includes the standards that enable consistent and accurate
collection and exchange of health information across health systems
and services
Workforce
This includes the available education and training programmes for
health workforce capacity-building in digital health
Source: Adapted from WHO/International Telecommunication Union National eHealth strategy toolkit (18)
In addition to considerations surrounding the ICT and enabling environment, the following cross-
cutting implementation issues were identified from systematic reviews of the global evidence.
These considerations have been mapped to the different components in Table 4.1.
Note that the following section is not intended to be an exhaustive list of implementation
considerations, but rather aims to highlight the issues commonly cited during the evidence
syntheses conducted for the guideline or identified by the guideline development group.
Implementers should seek more comprehensive implementation resources before designing and
implementing recommended digital health interventions.

Leadership and governance
ȺȺ Involve health workers, facility staff and other users in the design, user testing and
implementation of the programme, and include them in decisions about changes to the
programme. Ensure stakeholder consultation and engagement throughout the process.
Strategy and investment
ȺȺ Assess how the programme will be integrated into existing health care systems, including
how it might change workflows and the delivery of services. For example, how will the daily
routines of health workers need to change to include digital technologies? Will there be tasks or
activities, such as manual tabulation of data, that will no longer be required?
ȺȺ As with introducing new interventions, develop policies for change management to optimize
acceptability, feasibility, and overall uptake. This requires an understanding of the users of
the digital intervention and others targeted by it, their perceptions and interactions with the
intervention, and the context in which the intervention is implemented.
ȺȺ Put systems in place to ensure data privacy, ownership, access, integrity and protection of
patient information. Ensure that these systems meet national legal standards. Also ensure
that these systems meet the concerns of clients4
and that health workers, clients and other
stakeholders are aware of and able to use these systems. This is particularly important in
contexts where individual health information has financial value and may be particularly
vulnerable (information used for reimbursement in a health insurance scheme, for example),
where more rigorous enforcement is needed. Security is needed to address not only risks to
patient confidentiality, but also risks to data integrity such as unauthorized alteration of data.
ȺȺ Develop systems for ensuring informed consent among all populations, including those with
limited literacy.
ȺȺ Establish a plan or processes to replace manual/paper-based systems – to reduce the burden of
operating a dual system.

ȺȺ Review the potential to establish linkages with foundational digital infrastructure – such as to
registries of the health workforce, health facilities and clients – to effectively combine different
digital health interventions across various implementations. Determine ways to leverage
existing common digital architecture, such as identity authentication systems and terminology
services, which collectively or in part can make implementation of a digital interventions far
less burdensome and harmonized systems possible.
ȺȺ Use data standards to facilitate exchange of health information and linkages across different
digital systems. Increasingly these digital health interventions may be implemented in settings
where existing digital systems may already be in place. Global bodies such as Health Level
Seven (HL7), Integrating the Healthcare Enterprise (IHE), WHO (e.g. International Classification
of Diseases, ICD) and the International Organization for Standardization (ISO) have established
standards, which are a set of rules that enable information to be shared and processed in a
uniform and consistent manner (24,84–86). These standards allow implementers to align on
common data models and naming protocols, which can then facilitate exchange of information
across components of the digital health ecosystem and prevent siloed and unscalable
implementations.
Workforce, including training, supervision, and support
ȺȺ Deliver training to health workers on the use of devices before the programme is rolled out with
clients and patients. Also ensure easy availability of in-service training, refresher training, and
training in connection with updates to the software or devices.
ȺȺ When designing the programme and planning health worker training, pay particular attention
to the needs of health workers who are not familiar with digital technologies. Make an effort to
ensure that the requirements of the new programme do not threaten their job security.
ȺȺ Ensure that training and support is available through different channels, including individual
training sessions, online and through peers. Also ensure that health workers have ongoing and
easily accessible technical support to reinforce the training.
ȺȺ Ensure that supervisors are familiar with the programme and devices and receive appropriate
training. Where possible, equip supervisors with devices to enable them to be more engaged
and aware of how the digital system functions.
ȺȺ Continuously monitor how the programme is affecting health worker roles and daily activities.
Is it reducing or increasing workloads? For instance, is the health worker expected to maintain a
new digital system in addition to other, paper-based or non-digital systems? If additional work
is expected, at least in a transition phase, will health workers have time to manage it and will
they be compensated?

Infrastructure
ȺȺ Assess whether health workers are likely to have reliable network connectivity and access
to electricity in all their work settings. Put systems in place to deal with situations where
connectivity or electricity may be lacking or unreliable. This may include the provision of solar
chargers or enabling the digital system to function without Internet or data connection.
ȺȺ Put systems in place to replace health workers’ lost, broken or stolen mobile devices. The
consequences of lost devices should be clearly communicated, with efforts to limit misuse;
this could form part of a contractual agreement. Where health workers are expected to
use their own devices for work purposes, ensure that they incur no personal costs and that
organizational applications are compatible.
ȺȺ The quality of the information or health content to be delivered digitally, including its design
and presentation, is as critical as it would be in non-digital formats. This is particularly the
case for interventions that leverage health content to improve skills and competence, such as
decision support, mLearning and targeted client communication. Algorithms, learning modules
and other forms of health content should reflect and reinforce evidence-based clinical and
public health recommendations found in national protocols and normative guidelines.
ȺȺ Although equity, gender and human rights are not a component of the WHO/ITU National
eHealth strategy toolkit (18), this guideline recognizes their importance.
ȺȺ Programmes should account for the inequities in programme design, and proactively
develop and implement alternative means of providing services to those who would be left
out by digital only. The adoption of recommendations in this guideline should not exclude
or jeopardize the provision of quality non-digital health services where access to digital
technologies are not available, acceptable or affordable for target communities.
ȺȺ Particular attention needs to be paid to the needs, preferences and circumstances of
particularly disadvantaged or hard-to-reach groups, including people with low literacy or
digital literacy skills, people speaking minority languages, migrant populations in new settings,
people affected by emergency situations, or people with disabilities such as sight or hearing
impairment.

Lastly, implementations should also be guided by the Principles for Digital Development (26):
ȺȺ design with the user
ȺȺ understand the existing ecosystem
ȺȺ design for scale
ȺȺ build for sustainability
ȺȺ be data-driven
ȺȺ use open standards, open data, open source and open innovation
ȺȺ reuse and improve
ȺȺ address privacy and security, and
ȺȺ be collaborative.

5. Future research
This chapter on future research highlights crosscutting evidence gaps observed across
a range of interventions in relation to effectiveness, resource use and cost-effectiveness,
and gender, equity and rights. In addition, specific research questions are provided for each
of the interventions, based on the gaps identified through the evidence-to-decision
framework and GDG.
5.1 Overarching research gaps
The following sections describe the overarching research priorities identified through this
guideline process. These reflect the general areas in which the available evidence was found
to be of low or very low certainty or confidence, or where no direct evidence was identified.
Where studies were available, in some cases the certainty or confidence of the evidence was
affected by poor reporting of outcomes, studies with small numbers of participants, and limited
representation across different settings.
Annex 6 maps the state of evidence and its gaps based on the findings from the effectiveness
reviews for the included digital health interventions.
Effectiveness
For many of the interventions, the available evidence on effectiveness was sparse. Future research
should measure health system process improvements that may immediately result from the
digital intervention, such as health workers’ adherence to recommended practice, as well as
related distal health outcomes. Researchers should be realistic about the extent to which digital
health interventions can impact on distal health outcomes, which are often affected by a variety
of factors beyond the interaction with the digital intervention. Additionally, effectiveness studies
need to include ways of concurrently monitoring technological performance (for example, do
messages reach intended individuals?) and behavioural performance or user engagement (e.g. do
individuals who get messages listen to or read them, and subsequently act on them?).

Resource use and cost-effectiveness
The studies included in the systematic reviews of the effectiveness of the digital interventions
considered by the guideline identified limited evidence on the resources used to implement
these interventions. Costing studies should assess costs over a longer period, with appropriate
accounting of amortization and maintenance of equipment and the continuous user support
required. Future research should explore the cost-effectiveness, and potential for cost savings of
the identified intervention and additional savings achieved through combining interventions.
Gender, equity and rights
Further research needs to encompass a wider range of contexts and populations, including
populations with poor access to digital or conventional health services, in order to better
understand and mitigate any potential negative impacts on gender, equity and rights. Key research
questions include how digital health interventions can help to reduce disparities in linking to
the wider health system and whether these interventions may create further inequities in some
settings as a consequence of poor network coverage, limited control of mobile devices, or a lack of
other resources. Research should also explore unintentional exacerbation of inequities based on
who has access to digital devices, and who has access to network connectivity.
Implementation research
Due to the strong focus on integrated health systems and interoperability, future research
should also examine the synergies across different combinations of digital health interventions
to determine which packages of interventions are most effective and cost-effective. Addressing
this question is important given the potential complexity of implementing packages of digital
interventions and the costs of establishing and maintaining these systems. Specific questions
include the following.
ȺȺ What is the feasibility and effectiveness of combining different digital health interventions?
ȺȺ What are the non-digital health and supporting interventions (for example, enhanced
transportation, supervision) that should be packaged together with digital health interventions
to ensure their effectiveness, acceptability and feasibility?
ȺȺ What are the minimum requirements of a country’s enabling environment (infrastructure,
governance, workforce, interoperability and standards) to support the different recommended
digital health interventions?
ȺȺ How can the fidelity (i.e. the roll out of all the critical components of the intervention as
intended) of implementation at scale be facilitated?
Frameworks such as RE-AIM (Reach, Effectiveness, Adoption, Implementation, and Maintenance)
may be useful in structuring the implementation research (87).

5.2 Considerations for the design of
future evaluations
The GDG also identified several issues related to the design of future evaluations of digital health
interventions, including the following:
ȺȺ Health system focused digital interventions, such as stock management and birth and death
notification, are often complex in the number of components, behaviours targeted, and
organizational levels involved (89). These factors may make designs such as randomized
controlled trials for evaluating the effectiveness of these interventions difficult to apply. Other
designs may therefore need to be considered, such as controlled before-and-after studies,
stepped-wedge randomized controlled trials and interrupted time series studies.
ȺȺ While there is value in evaluating changes in client/patient health outcomes, intermediate
outcomes are also critical for the evaluation of digital health interventions. For example, the
effect of decision support on client/patient health outcomes are influenced not only by the
information delivered through the digital system, but also by a host of other factors, including
access to medicines, their cost, family support, and biomedical factors such as whether the
individual responds appropriately to recommended treatments or has comorbidities. A logical
framework of how the digital intervention functions may be helpful in understanding the
pathways through which the intervention influences a targeted behaviour or health system
challenge and in selecting appropriate outcomes along these pathways.
ȺȺ Digital technologies provide new opportunities to capture research data for measuring
the effectiveness of implementations in real time, thus facilitating the ability to conduct
evaluations more rapidly. Incorporating the research data collection needs for primary and
secondary outcomes of interest at the design stage can ensure that the data needed to
measure these outcomes is captured alongside the implementation.
ȺȺ Rapid changes in digital technologies and the iterative approaches often used for software
development may force digital health interventions to evolve during evaluation periods, which
may pose challenges for the evaluation process. Detailed process evaluations running alongside
impact evaluations may be helpful in understanding the effects of incremental changes in the
digital interventions over time.
ȺȺ Future research efforts should establish common metrics and tools for assessing the
effectiveness and cost-effectiveness of digital health interventions

6. Disseminating and
updating the guideline
6.1 Dissemination and implementation
of the guideline
This guideline will be available in its full version, as well as a condensed form that includes the
executive summary, implementation considerations, and future research priorities. WHO will
disseminate this guideline as much as possible through regional offices and networks and existing
large-scale, global convenors, including the Asia eHealth Information Network (AeHIN) (30),
the Global Digital Health Network (31), the Health Data Collaborative’s Digital Health and
Interoperability Working Group (90), and the IBP Initiative (37) among other peer-learning groups
and communities of practice. WHO will also convene regional consultations at policy-maker
gatherings and with digital health working groups. Additionally, structured webinars will be used
to share the recommendations and maximize the reach of these evidence-based findings on digital
health interventions.
It is equally important that this guideline is shared with public health practitioners who have
a limited experience with digital health. Where possible, WHO will identify opportunities for
presentation panels at conferences for clinicians and public health practitioners across different
domains, including health systems strengthening, and digital innovations and UHC, with an
emphasis on conferences that focus on low- and middle-income countries.
WHO has also developed a complementary implementation guide, the Planning and costing
guide for digital interventions for health programmes, to help implementers and health planners
in Ministries of Health select, plan for, cost and implement the recommended digital health
interventions in accordance with identified local health needs, the enabling environment and
available technologies. This implementation guide will provide a stepwise process to ensuring that
the implementation of the recommended digital health interventions fits in meeting the identified
needs, and within appropriate contexts.

6.2 Updates and living guidelines approach
This guideline will be subject to a phased approach that treats them as living guidelines,
supporting the review of new evidence for specific questions on digital health interventions.
This will ensure that new evidence is brought to the guideline development group (GDG) for
review. The first planned update to the guideline will be to include the use of targeted client5
communication for noncommunicable diseases. A virtual GDG will be convened for formulating
recommendations based on the evidence tables prepared for this additional priority question.
Associated recommendations will be included in version 1.1 of the guideline.
This guideline document recognizes the need to monitor the rapidly evolving nature of digital
health, systematically through a continuous scanning and review of the literature and innovation
pipelines. The first major update to the guideline is likely to be needed within 18 to 24 months of
this initial dissemination, to accommodate new evidence for the existing recommendations and
any emerging evidence related to other innovations in the WHO classifications.
WHO’s Classification of digital health interventions v1.0 (13) provides the grounding for this living-
guidelines approach, to help determine which additional interventions will need the deliberations
of the GDG, and to help establish the questions for systematic review and the subsequent
synthesis of evidence and development of recommendations. Scans of the evolving evidence base
and collaboration with WHO’s Innovations Hub (92) will also assist WHO in its vigilance to identify
any emerging digital innovations that may warrant review by the GDG but were not reflected in
the original classification scheme.
This guideline recognizes that the innovative approach of a living guideline is critical for ensuring
Member States stay informed in the rapidly evolving field of digital health. WHO will continue to
work closely with the Secretariat of the WHO Guidelines Review Committee (34) to ensure that
this process adds value and is tested and refined.

Glossary
Client An individual who is a potential or current user of health services; may also be referred to as
patient or non-patient who uses health information and services. (Although WHO’s Classification of
digital health interventions v1.0 uses the term “client” (13), the terms “individual” and “patient” may
be used interchangeably, where appropriate.)
eHealth The use of information and communications technology (ICT) in support of health and health-
related fields, including health care services, health surveillance, health literature, and health
education, knowledge and research. mHealth is a component of eHealth (1).
Enabling
environment
Attitudes, actions, policies and practices that stimulate and support the effective and efficient
functioning of organizations, individuals and programmes or projects. The enabling environment
includes legal, regulatory and policy frameworks, and political, sociocultural, institutional and
economic factors.
Digital health An overarching term that comprises eHealth (which includes mHealth), and emerging areas, such
as the use of computing sciences in the fields of artificial intelligence, big data and genomics (3,4).
Digital health
architecture
An overview or blueprint used to design and describe how different digital applications (software
and ICT systems) and other core functionalities will interact with each other within a given
context (25).
Digital health
application
The software, ICT systems, and communication channels used in the health sector, such as
a software being used for health management information systems (HMIS) or an interactive
messaging application (“app”) (25).
Digital health
intervention
A discrete function of a digital technology to achieve health sector objectives. The WHO
Classification of digital health interventions v1.0 provides an overview of the range of digital health
interventions identified in the literature and implementation practices (13).
Table 2.1 lists definitions of the specific digital health interventions included in this guideline.
Digital health
ecosystem
The combined set of digital health components representing the enabling environment,
foundational architecture and ICT capabilities available in a given context or country.
Evidence-
to-decision
framework
A framework to assist people making and using evidence-informed recommendations and
decisions. Their main purpose is to help decision-makers use evidence in a systematic and
transparent way. When used in a WHO guidelines context, evidence-to-decision frameworks
inform guideline development group (GDG) members about the comparative pros and cons of the
interventions being considered, ensure that GDG members consider all the important criteria for
making a decision, provide GDG members with a concise summary of the best available evidence
about each criterion to inform their judgments, help help the GDG members to structure and
document their discussions and to identify any reasons for disagreement, making the process and
the basis for their decisions transparent.
Interoperability The ability of multiple ICT systems and software applications to communicate with one another,
exchange data and use the information that has been exchanged.
mHealth The use of mobile and wireless technologies to support health objectives (2,3).

References
1. Global diffusion of eHealth:
making universal health coverage
achievable. Report of the third global
survey on eHealth. Geneva: World
Health Organization; 2016 (http://
apps.who.int/iris/bitstream/hand
le/10665/252529/9789241511780-eng.
pdf, accessed 19 November 2018).
2. mHealth: new horizons for health
through mobile technologies. Geneva:
World Health Organization; 2011
(Global Observatory for eHealth series,
volume 3; http://www.who.int/goe/
publications/goe_mhealth_web.pdf,
accessed 24 May 2018).
3. mHealth: use of appropriate digital
technologies for public health: report
by the director-general. Geneva: World
Health Organization; 2018 (A71/20;
http://apps.who.int/gb/ebwha/
pdf_files/WHA71/A71_20-en.pdf,
accessed 21 November 2018).
4. Agenda item 12.4. Digital health
resolution. In: Seventy-first World
Health Assembly, Geneva, 26
May 2018. Geneva: World Health
Organization; 2018 (A71/VR/7;
http://apps.who.int/gb/ebwha/
pdf_files/WHA71/A71_R7-en.pdf,
5. mHealth deployment tracker. In:
GSMA [website]. London: GSM
Association; 2018 (https://www.gsma.
com/mobilefordevelopment/m4d-
tracker/mhealth-deployment-tracker,
accessed 28 October 2018).
6. The roadmap for health measurement
and accountability. Geneva and
Washington (DC): World Bank Group,
United States Agency for International
Development and World Health
Organization; 2015 (www.who.int/
hrh/documents/roadmap4health-
measurement_accountability.pdf,
accessed 5 January 2018).
7. Tomlinson M, Rotheram-Borus MJ,
Swartz L, Tsai AC. Scaling up mHealth:
where is the evidence? PLoS Med.
2013;10(2):e1001382. doi:10.1371/
journal.pmed.1001382.
8. Fraser H, Bailey C, Mehl G, Sinha
C. The role of evaluation in global
eHealth. In: Global health delivery
online [website]. Cambridge, MA:
Harvard University; 2011 (https://www.
ghdonline.org/tech/discussion/the-
role-of-evaluation-in-global-ehealth,
9. Universal health coverage (UHC). In:
World Health Organization [website].
Geneva: World Health Organization;
2017 (http://www.who.int/news-
room/fact-sheets/detail/universal-
health-coverage-(uhc), accessed
20 September 2018).
10. Tanahashi T. Health service coverage
and its evaluation. Bull World Health
Organ. 1978;56(2):295–303 (http://
www.who.int/iris/handle/10665/261736,
11. Mehl G, Labrique A. Prioritizing
integrated mHealth strategies
for universal health coverage.
Science. 2014;345(6202):1284–7.
doi:10.1126/science.1258926.
12. Labrique AB, Vasudevan L, Kochi
E, Fabricant R, Mehl G. mHealth
innovations as health system
strengthening tools: 12 common
applications and a visual framework.
Glob Health Sci Pract. 2013;1(2):160–71.
doi:10.9745/GHSP-D-13-00031.
13. Classification of digital health
interventions v1.0: a shared language
to describe the uses of digital
technology for health. Geneva: World
Health Organization; 2018 (WHO/
RHR/18.06; http://apps.who.int/iris/
bitstream/handle/10665/260480/
WHO-RHR-18.06-eng.pdf, accessed 21
November 2018).
14. Monitoring the building blocks
of health systems: a handbook of
indicators and their measurement
strategies. Geneva: World Health
Organization; 2010.
15. Consolidated guidelines on the use
of antiretroviral drugs for treating
and preventing HIV infection:
recommendations for a public health
approach, second edition. Geneva:
(http://apps.who.int/iris/bitstream/
handle/10665/208825/9789241549684_
eng.pdf, accessed 4 May 2018).
16. Guidelines for treatment of drug-
susceptible tuberculosis and patient
care, 2017 update. Geneva: World
le/10665/255052/9789241550000-eng.
17. WHO guideline on health policy
and system support to optimize
community health worker
programmes. Geneva: World
le/10665/275474/9789241550369-eng.
18. National eHealth strategy toolkit.
Geneva: World Health Organization
and International Telecommunication
Union; 2012 (https://apps.who.
int/iris/handle/10665/75211,
accessed 2 September 2015)
19. Digital health atlas [website]. Geneva:
World Health Organization; no
date (https://digitalhealthatlas.org,
20. International Telecommunication
Union (ITU) and World Health
Organization. Be he@lthy, be mobile.
Geneva: ITU; 2014 (https://www.itu.int/
en/ITU-D/ICT-Applications/eHEALTH/
Be_healthy/Documents/Be_Healthy_
Be_Mobile_Annual_Report%202013-
2014_Final.pdf, accessed 6 March 2019).
21. Standards and tools to strengthen
country monitoring and evaluation. In:
2019 (https://www.who.int/healthinfo/
topics_standards_tools, accessed 5
March 2019).
22. Health information systems
interoperability maturity toolkit.
In: MEASURE Evaluation [website].
Chapel Hill (NC): University of North
Carolina at Chapel Hill; (https://www.
measureevaluation.org/resources/
tools/health-information-systems-
interoperability-toolkit, accessed 27
October 2018).

23. Partnership for Maternal, Newborn
and Child Health. Information and
communication technologies for
women’s and children’s health: a
planning workbook. Geneva: World
Health Organization; 2014 (https://
www.who.int/pmnch/knowledge/
publications/ict_mhealth.pdf,
24. Global Digital Health Index [website].
Washington (DC): Global Digital
Health Index; no date (https://www.
digitalhealthindex.org, accessed 30
October 2018).
25. International Telecommunication
Union. Digital health platform
handbook: building a digital
information infrastructure
(infostructure) for health. Geneva: ITU;
2018.
26. Principles. In: Principles for digital
development [website]. Principles
for Digital Development; no date
(https://digitalprinciples.org/principles,
27. Department of Computer and
Systems Sciences/DSV, SPIDER.
The principles of donor alignment
for digital health [website]. Kista:
Stockholm University; no date (http://
digitalinvestmentprinciples.org,
accessed 6 March 2018).
28. Resources. In: Integrating the
Healthcare Enterprise (IHE) – IHE
International [website]. Oak Brook
(IL): IHE International; no date
(https://www.ihe.net/resources,
29. African Alliance [website]. African
Alliance of Digital Health Networks;
no date (https://www.africanalliance.
digital, accessed 21 November 2018).
30. #WeAreAeHIN [website]. Asia
eHealth Information Network;
no date (http://www.aehin.org,
31. Global Digital Health Network
[website]. Baltimore (MD): Johns
Hopkins University; 2018 (https://
www.globaldigitalhealthnetwork.org,
32. Health Data Collaborative –
data for health and sustainable
development [website]. Health
Data Collaborative; 2018 (https://
www.healthdatacollaborative.org,
33. Communities. In: OpenHIE
[website]. OpenHIE; no date
(https://ohie.org/#communities,
34. Guidelines Review Committee
(GRC). In: World Health Organization
[website]. Geneva: World Health
Organization; 2018 (http://www.
who.int/publications/guidelines/
guidelines_review_committee,
35. WHO handbook for guideline
development – 2nd edition. Geneva:
(http://www.who.int/publications/
guidelines/handbook_2nd_ed.pdf,
36. Health Information For All
[website]. Charlbury: Global
Healthcare Information Network;
no date (https://www.hifa.org,
37. IBP initiative [website]. Geneva:
Implementing Best Practices
Consortium; 2018 (http://www.
ibpinitiative.org, accessed 21 November
2018).
38. Civil registration and vital statistics
(CRVS). Geneva: World Health
Organization; 2017 (http://www.who.
int/healthinfo/civil_registration,
39. Hawkins RP, Kreuter M, Resnicow K,
Fishbein M, Dijkstra A. Understanding
tailoring in communicating
about health. Health Educ Res.
2008;23(3):454–66.
40. Frøen JF, Myhre SL, Frost MJ, Chou
D, Mehl G, Say L, et al. eRegistries:
Electronic registries for maternal
and child health. BMC Pregnancy
Childbirth. 2016;16:11.
41. ICT Data and Statistics Division,
Telecommunication Development
Bureau. ICT facts and figures. Geneva:
International Telecommunications
Union; 2015 (https://www.itu.int/
en/ITU-D/Statistics/Documents/
facts/ICTFactsFigures2015.pdf,
42. EPOC resources for review authors.
In: Cochrane Effective Practice and
Organisation of Care (EPOC) [website].
Cochrane; 2018 (http://epoc.cochrane.
org/epoc-specificresources-review-
authors, accessed 22 September 2018).
43. Consumers and communication group
resources for authors. In: Cochrane
Consumers and Communication
[website]. London: Cochrane; 2018
(https://cccrg.cochrane.org/author-
resources, accessed 27 November 2018).
44. Moher D, Liberati A, Tetzlaff J, Altman
DG, The PRISMA Group. Preferred
reporting items for systematic
reviews and meta-analyses: the
PRISMA statement. PLoS Med.
2009;6(7):e1000097. doi:10.1371/journal.
pmed.1000097.
45. Higgins JPT, Green S, editors. Cochrane
handbook for systematic reviews of
interventions, version 5.1.0 (updated
March 2011). London: The Cochrane
Collaboration; 2011: section 8.5
(http://handbook-5-1.cochrane.org,
46. CASP checklist: 10 questions to help
you make sense of a qualitative
research. In: CASP [website]. Oxford:
Critical Appraisal Skills Programme;
no date (https://casp-uk.net/wp-
content/uploads/2018/01/CASP-
Qualitative-Checklist-2018.pdf,
47. Lewin S, Langlois E, Tunçalp Ö,
Portela A, the COMMVAC Project
Team. The WEIRD (Ways of Evaluating
Important and Relevant Data) tool.
Questions to guide assessment/critical
appraisal of programme descriptions,
implementation descriptions and other
mainly descriptive types of evidence.
Norway: Norwegian Institute of Public
Health; 2018.
48. Lewin S. 5th global symposium on
HSR: invitation to a discussion on the
WEIRD (ways of evaluating important
and relevant data) tool. In: Health
information for all [website]. Oxford:
Global Healthcare Information
Network; 2018 (https://www.hifa.org/
dgroups-rss/5th-global-symposium-
hsr-invitation-discussion-weird-
ways-evaluating-important-and-0,
49. Lewin S, Glenton C, Munthe-Kaas H,
Carlsen B, Colvin CJ, Gulmezoglu M,
et al. Using qualitative evidence in
decision making for health and social
interventions: an approach to assess
confidence in findings from qualitative
evidence syntheses (GRADE-CERQual).
PLoS Med. 2015;12(10):e1001895.
doi:10.1371/journal.pmed.1001895.
50. Lewin S, Booth A, Glenton C, Munthe-
Kaas HM, Rashidian A, Wainwright
M, et al. Applying GRADE-CERQual
to qualitative evidence synthesis
findings: introduction to the series.
Implement Sci. 2018;13(Suppl 1):2.

51. Lewin S, Bohren M, Rashidian A,
Munthe-Kaas H, Glenton C, Colvin
CJ, et al. Applying GRADE-CERQual to
qualitative evidence synthesis findings,
paper 2: how to make an overall
CERQual assessment of confidence
and create a Summary of Qualitative
Findings table. Implement Sci.
2018;13(Suppl 1):10.
52. GRADE [website]. The GRADE
Working Group; 2018 (http://
gradeworkinggroup.org,
53. Tong A, Flemming K, McInnes E,
Oliver S, Craig J. Enhancing
transparency in reporting the synthesis
of qualitative research: ENTREQ.
BMC Med Res Methodol. 2012;12:181.
doi: 10.1186/1471-2288-12-181.
54. Optimizing health worker roles to
improve access to key maternal
and newborn health interventions
through task shifting. Geneva:
(http://apps.who.int/iris/bitstream/
handle/10665/77764/9789241504843_
eng.pdf, accessed 26 November 2018).
55. Antenatal care for a positive
pregnancy experience. Geneva: World
le/10665/250796/9789241549912-eng.
56. WHO recommendations: intrapartum
care for a positive childbirth
experience. Geneva: World Health
Organization; 2018 (https://apps.
who.int/iris/bitstream/hand
le/10665/260178/9789241550215-eng.
pdf, accessed 7 March 2019).
57. World Bank, World Health Organiza-
tion. Global civil registration and vital
statistics: scaling up investment plan
2015–2024. Washington, DC: World
Bank Group; 2014: xiv (88351; http://
documents.worldbank.org/curated/
en/457271468148160984/pdf/883510W-
P0CRVS000Box385194B00PUBLIC0.pdf,
58. Muzzi M. UNICEF good practices in
integrating birth registration into
health systems (2000-2009). New York
(NY): United Nations Children’s Fund;
2009 (www.unicef.org/protection/files/
Birth_Registration_Working_Paper.pdf,
59. World Health Organization, Health
Metrics Network (HMN). Civil
registration and vital statistics 2013:
challenges, best practice and design
principles for modern systems.
2013 (www.who.int/healthinfo/civil_
registration/crvs_report_2013.pdf,
accessed 20 December 2017).
60. The future for women and children:
UNICEF and WHO joint statement on
strengthening civil registration and
vital statistics (CRVS). New York (NY)
and Geneva: United Nations Children’s
Fund and World Health Organization;
2018 (http://www.who.int/healthinfo/
civil_registration/WHO_UNICEF_
Statement_CRVS_2018.pdf, accessed
25 November 2018).
61. Jackson D, Wenz K, Muniz M, Abouzahr
C, Schmider A, Braschi MW, et al.
Civil registration and vital statistics
in health systems. Bull World Health
Organ. 2018;96(12):861-3. doi: 10.2471/
BLT.18.213090.
62. Phillips DE, AbouZahr C, Lopez AD,
Mikkelsen L, De Savigny D, Lozano
R, et al. Are well functioning civil
registration and vital statistics
systems associated with better
health outcomes? Lancet.
2015;386(10001):1386-94.
63. Oomman N, Mehl G, Berg M, Silverman
R. Modernising vital registration
systems: why now? Lancet.
2013;381(9875):1336-7.
64. Delivering quality health services:
a global imperative for universal
health coverage. Geneva: World
Health Organization, Organisation
for Economic Co-operation and
Development, and The World Bank;
2018.
65. Agarwal S, Tamrat T, Fønhus MS,
Henschke N, Bergman H, Mehl GL, et al.
Tracking health commodity inventory
and notifying stock levels via mobile
devices (protocol). Cochrane Database
Syst Rev. 2018;(1):CD012907.
66. Module 7: managing the supply of
drugs and commodities. In: Lamptey
PR, Zeitz P, Larivee C, editors. Strategies
for expanded and comprehensive
response (ECR) to a National HIV/AIDS
epidemic: a handbook for designing
and implementing HIV/AIDS programs.
Arlington (VA): Family Health
International; 2001: 97–114 (http://
pdf.usaid.gov/pdf_docs/Pnacn557.pdf,
67. Ki-moon, B. Global strategy for
women’s and children’s health. New
York (NY): United Nations; 2010
(http://www.who.int/pmnch/topics/
maternal/20100914_gswch_en.pdf,
68. Kimbwarata J, editor, UN Commission
on Life-Saving Commodities for
Women’s and Children’s Health.
A toolkit for implementing
health services. New York (NY):
UN Commission on Life-Saving
Commodities; 2015 (http://www.
lifesavingcommodities.org/wp-
content/uploads/2015/08/Toolkit-for-
Health-Services-2.pdf, accessed 29
May 2018).
69. Facility and Community Data Working
Group: Logistics Management
Information Sub-group: DRAFT
Terms of Reference. Health Data
Collaborative; no date (https://www.
healthdatacollaborative.org/fileadmin/
uploads/hdc/Documents/Working_
Groups/LMIS_TORs_and_Workplan.
pdf, accessed 28 May 2018).
70. World Health Organization. A
universal truth: no health without
a workforce. Geneva: World Health
Organization; 2014 (http://www.who.
int/workforcealliance/knowledge/
resources/GHWA-a_universal_truth_
report.pdf, accessed 25 November
2018).
71. World Health Organization. In-
creasing access to health workers
in remote and rural areas through
improved retention: global policy
recommendations. Geneva: World
apps.who.int/iris/bitstream/han-
dle/10665/44369/9789241564014_eng.
72. Telemedicine opportunities and
developments in Member States:
report on the second global survey
on eHealth. Geneva: World Health
Organization; 2010 (Global Observatory
for eHealth series, volume 2; http://
apps.who.int/iris/bitstream/
handle/10665/44497/9789241564144_
eng.pdf, accessed 26 November 2018).
73. Wootton R. Telemedicine support
for the developing world. J Telemed
Telecare. 2008;14(3):109-14.
74. Hersh WR, Hickam DH, Severance
SM, Dana TL, Krages KP, Helfand
M. Telemedicine for the Medicare
population: update. Evidence Report/
Technology Assessment. 2006;131:1-41.

75. Gonçalves BDC, Buckley BS, Fønhus
MS, Glenton C, Henschke N, Lewin S,
et al. Mobile-based technologies to
support client to healthcare provider
communication and management of
care (Protocol). Cochrane Database
of Systematic Reviews 2018; 1. Art.
No. CD012928. doi: 10.1002/14651858.
CD012928.
76. Delivering quality health services:
a global imperative for universal
health coverage. Geneva: World
Health Organization, Organisation
for Economic Co-operation
and Development, and The
World Bank; 2018 (http://apps.
who.int/iris/bitstream/hand
le/10665/272465/9789241513906-eng.
77. Moja L, Kwag KH, Lytras T, Bertizzolo L,
Pecoraro V, Rigon G, et al. Effectiveness
of computerized decision support
systems linked to electronic health
records: a systematic review and meta-
analysis. American Journal of Public
Health 2014;104(12):e12-22.
78. Agarwal S, Tamrat T, Glenton C,
Lewin S, Henschke N, Maayan N, et
al. Decision-support tools via mobile
devices to improve quality of care in
primary healthcare settings (protocol).
Cochrane Database Syst Rev.
2018;(2):CD012944.
79. Kawamoto K, Houlihan CA, Balas
EA, Lobach DF. Improving clinical
practice using clinical decision support
systems: A systematic review of trials
to identify features critical to success.
BMJ. 2005;330(7494):765.
80. Bright TJ, Wong A, Dhurjati R, Bristow
E, Bastian L, Coeytaux RR, et al. Effect
of clinical decision-support systems:
a systematic review. Ann Intern Med.
2012;157(1):29-43.
81. Agarwal S, Vasudevan L, Tamrat T,
Glenton C, Lewin S, Bergman H, et
al. Digital tracking, provider decision
support systems, and targeted client
communication via mobile devices
to improve primary health care
(Protocol). Cochrane Database Syst
Rev. 2018;(1):CD012925.
82. Tudor Car L, Riboli-Sasco EF, Marcano
Belisario JS, Nikolaou CK, Majeed
A, Zary N, Car J. Mobile learning
for delivering health professional
education (Protocol). Cochrane
Database Syst Rev. 2015;(9):CD011861.
83. World Health Organization. Global
strategy on human resources for
health: workforce 2030. Geneva: World
www.who.int/hrh/resources/global_
strategy_workforce2030_14_print.pdf,
84. Introduction to HL7 standards. In:
HL7 International [website]. Ann
Arbor (MI): Health Level Seven
International (http://www.hl7.org/
implement/standards/index.cfm,
85. Classification of diseases (ICD). In:
no date (https://www.who.int/
classifications/icd, accessed 7 March
2019).
86. ISO International Organization for
Standardization [website]. Geneva:
International Organization for
Standardization; no date (https://www.
iso.org, accessed 7 March 2019).
87. About RE-AIM. In: RE-AIM [website].
University of Nebraska Medical Center;
no date (http://www.re-aim.org/about,
88. The Lives Saved Tool [website].
Baltimore (MD): Johns Hopkins
Bloomberg School of Public Health; no
date (https://www.livessavedtool.org,
89. Lewin S, Hendry M, Chandler J,
Oxman AD, Michie S, Shepperd S,
et al. Assessing the complexity of
interventions within systematic
reviews: development, content and
use of a new tool (iCAT_SR). BMC Med
Res Methodol. 2017;17:76. doi: 10.1186/
s12874-017-0349-x.
90. Digital Health & Interoperability
Working Group. In: Health Data
Collaborative – data for health and
sustainable development [website].
Health Data Collaborative; 2018
(https://www.healthdatacollaborative.
org/how-we-work/digital-health-
interoperability-working-group,
91. World Health Organization and PATH.
Planning and costing guide for digital
interventions for health programmes.
forthcoming.
92. Innovation. In: World Health
Organization [website]. Geneva: World
Health Organization; no date (https://
www.who.int/topics/innovation,

Annex 1.
Classification of digital health interventions and
Health system challenges
Source: Classification of digital health interventions v1.0: a shared language to describe the uses of digital technology for health. Geneva: World Health
Organization; 2018 (WHO/RHR/18.06; http://apps.who.int/iris/bitstream/handle/10665/260480/WHO-RHR-18.06-eng.pdf, accessed 21 November 2018)
1.1 Targeted client
communication
1.4 Personal health
tracking
transactions
1.5 Citizen based
reporting
1.6
On-demand
information
services to clients
communication
communication
1.1.1
alerts to specific
population group(s)
1.1.2
demographics
1.1.4
medical records
1.4.2 Self monitoring of health
or diagnostic data by client
1.7.1
client(s)
1.7.2
Transmit or manage
health services
1.7.3
Transmit or manage
health services
feedback by clients
events by clients
information
1.2.1
Transmit untargeted
1.2.2
Transmit untargeted
undefined group
1.0
Clients
2.1
Client
identification and
registration
2.5 Health worker
communication
2.6 Referral
coordination
2.7
Health worker
activity planning
and scheduling
2.8 Health worker
training
2.9
Prescription
and medication
management
2.10
Laboratory and
Diagnostics
Imaging
Manangement
2.2 Client health
records
2.3 Health worker
decision support
2.4 Telemedicine
2.1.1 Verify client
unique identity
2.5.1
Communication from
health worker(s) to
supervisor
2.5.2
Communication and
health worker(s)
2.5.3
health worker(s)
2.5.4
health worker(s)
workers
2.6.2
health sector
of services
activities
health worker(s)
worker(s)
prescription orders
consumption
to health worker
diagnostic orders
2.2.1
and services
clinical records
2.2.3
Manage client’s
unstructured
clinical records
2.2.4
data collection and
management
2.3.1
Provide prompts and
to protocol
other health status
2.4.1
worker
2.4.2
data by provider
2.4.4
management between
health worker(s)
2.0
Health Workers
3.1 Human resource
management
andVital Statistic
3.6 Equipment and
asset management
3.7 Facility
management
3.5 Health
financing
3.2 Supply chain
management
3.3 Public health
event notification
3.1.1
cadres and related
health worker(s)
3.1.3
worker(s)
of health worker(s)
health equipment
3.6.2
equipment
related information
claims submission
3.5.4
payment to health
worker(s)
3.5.5
Transmit or manage
worker(s)
expenditures
3.2.1
commodities
health commodities
of commodities
3.2.6
substandard drugs
by clients
3.3.1
health events from
point of diagnosis
3.0
4.1
Data collection,
management,
and use
4.3 Location
mapping
interoperability
4.2 Data
coding
4.1.1
Non-routine data
collection and
management
aggregation
visualization
4.1.4
on future events
health events
health worker
4.4.1 Data exchange
across systems
4.2.2
terminologies
cause of mortality
4.0
Data Services

3.2.1 Manage inventory and distribution
of health commodities
1 Information 3 Quality 6 Efficiency
7 Cost
8 Accountability
2 Availability 4 Acceptability
5 Utilization
1.1 Lack of population
denominator
1.2 Delayed reporting
of events
1.3 Lack of quality/
reliable data
1.4 Communication
roadblocks
1.5 Lack of access to
information or data
1.6 Insufficient utilization of
data and information
1.7 Lack of unique identifier
3.1 Poor patient experience
3.2 Insufficient health
worker competence
3.3 Low quality health
commodities
3.4 Low health worker
motivation
3.5 Insufficient continuity
of care
3.6 Inadequate supportive
supervision
3.7 Poor adherence to
guidelines
6.1 Inadequate workflow
management
6.2 Lack of or inappropriate
referrals
6.3 Poor planning and
coordination
6.4 Delayed provision of care
6.5 Inadequate access to
transportation
7.1 High cost of manual
processes
7.2 Lack of effective resource
allocation
7.3 Client-side expenses
7.4 Lack of coordinated payer
mechanism
8.1 Insufficient patient
engagement
8.2 Unaware of service
entitlement
8.3 Absence of community
feedback mechanisms
8.4 Lack of transparency in
commodity transactions
8.5
Poor accountability
between the levels of
the health sector
8.6 Inadequate understanding
of beneficiary populations
2.1 Insufficient supply
of commodities
of services
of equipment
2.4 Insufficient supply of
qualified health workers
4.1 Lack of alignment with
local norms
4.2
Programs which do not
address individual beliefs
and practices
5.1 Low demand for services
5.2 Geographic inaccessibility
5.3 Low adherence to
treatments
5.4 Loss to follow up
Health System Challenges

Annex 2.
Priority questions
Priority questions in the PICO format (population, intervention, comparator, outcomes) identified during the
guideline development process (see section 2.1).
Population Intervention Comparator Outcomes
Health
domains
of focus in
systematic
reviews
Key
informants,
health
workers, civil
registrar and
health focal
points
Birth
notification via
mobile devices;
Death
notification via
mobile devices
Standard
practice,
non-digital
intervention
1. [Information/Data] Change in data access
and use, and in time between reporting of
data and appropriate action
2. [Efficiency] Change in time between birth
and initiation of newborn and child health
services
3. [Use/Demand] Change in patients’/clients’
use of primary care services
4. [Information/Data] Change in number of
children and age of children whose births
are registered by linking birth notification
to health services with higher demand-side
application, such as immunization
5. Unintended consequences
6. Clients’ and health workers’ satisfaction
with/acceptability of the digital health
intervention
7. Resource use/cost/cost-effectiveness
All – no
restrictions
Health
workers in
primary care,
management
staff
Stock
notification
and commodity
management
Standard
practice,
non-digital
intervention
1. [Information/Data] Change in data access
and use, and in time between receipt/
reporting of data and appropriate action
2. [Resource allocation] Change in the
availability of essential commodities
through better planning of health services/
resource allocation (also wastage, stock-outs,
availability at point of care)
3. [Information/Data] Change in the quality of
data about stock management (accuracy,
timeliness, completeness of data)
4. [Efficiency] Change in health workers’ time
spent on administrative tasks
5. Health workers’ satisfaction with/
acceptability of the digital health
intervention
All – no
restrictions

Health
domains
of focus in
systematic
reviews
Individuals
contacting
health
workers (any
health issue)
Client-to-
provider
telemedicine
Standard
practice,
non-digital
intervention
1. [Use/Demand] Change in clients’ use of
primary care services
2. [Efficiency] Change in time between
presentation and appropriate management
by provider, includes change in time for
clients to receive/access health services and
information
3. [Use] Change in service linkages for clients,
including referrals
4. [Health-related outcomes ] Change in
patients’/clients’ health and well-being
6. Health workers’ and clients’ satisfaction
intervention
All – no
restrictions
Lay/
community
health
workers and
professional
health
workers for
clients’health
(any health
issue)
Provider-
to-provider
telemedicine
Standard
practice,
non-digital
intervention
1. [Use] Change in clients’ use of primary care
services
2. [Quality] Change in health workers’
adherence to recommended/clinical practice,
guidelines or protocols (e.g. providing the
service at the recommended time, referral as
recommended)
3. [Quality] Change in providers’ ability for
screening and prioritizing groups of clients
4. [Efficiency] Change in time between
presentation and appropriate management,
including time for referral services
5. [Quality/Efficiency] Change in health
workers’ interpersonal collaboration and
coordination of care, including emergency
transport services
6. [Health-related outcomes] Change in
intervention
All – no
restrictions

Health
domains
of focus in
systematic
reviews
Adolescent
and youth
populations
(aged 10–24
years)
Targeted client
communication
Standard
practice,
non-digital
intervention
1. [Knowledge] Change in adolescents’ and
youths’ knowledge about health behaviours
for sexual and reproductive health (SRH); and
their knowledge about the existence of SRH
services
2. [Knowledge] Change in adolescents’ and
youths’ awareness or knowledge about their
entitlement to SRH services
3. [Attitude] Change in adolescents’ and
youths’ attitudes and norms, self-efficacy,
empowerment or intent with regard to an
SRH behaviour or service or health issue
4. [Use/Behaviour] Change in adolescents’ and
youths’ targeted behaviour regarding SRH
health (includes, e.g. adherence to protocols,
retention in care, treatment completion, etc.)
5. [Use/Demand] Change in adolescents’
and youths’ use of SRH services, including
complementary services
6. [Efficiency] Change in timeliness of receiving
and accessing SRH services and information
(e.g. contraceptive options, partner
notification, receipt of test results, etc.)
adolescents’ and youths’ health and well-
being (includes surrogate health outcomes
such as CD4 count, treatment for sexually
transmitted infections (STIs), unintended
pregnancy)
9. Satisfaction with/acceptability of the digital
health intervention among adolescents and
youths
SRH for
adolescents

Health
domains
of focus in
systematic
reviews
Adult users/
potential
users of SRH
services (to
contrast
with focus on
adolescents
above)
Targeted client
communication
Standard
practice,
non-digital
intervention
1. [Use/Behaviour] Change in targeted
behaviour regarding SRH (includes, e.g.
adherence to protocols, retention/loss
to follow-up, treatment completion,
appointment attendance, etc.)
2. [Use/Demand] Change in use of SRH services,
including complementary services
(e.g. partner notification, receipt of test
results, etc.)
4. [Health-related outcomes] Change in health
and well-being (includes surrogate health
outcomes such as CD4 count, STI treatment,
unintended pregnancy)
6. Adults’ satisfaction with/acceptability of the
digital health intervention
SRH for
adults/non-
adolescent
populations
Pregnant
women,
postpartum
women
and their
partners/
support
health
workers
Targeted client
communication
Standard
practice,
non-digital
intervention
behaviour regarding SRH (includes, e.g.
adherence to protocols, retention/loss
to follow-up, treatment completion,
appointment attendance, etc.)
2. [Use/Demand] Change in clients’ use of SRH
services, including complementary services
(e.g. partner notification, receipt of test
results, etc.)
outcomes such as CD4 count, STI treatment,
unintended pregnancy)
6. Pregnant/postpartum women’s satisfaction
intervention
Maternal
and newborn
health

Health
domains
of focus in
systematic
reviews
Pregnant
women and
breastfeeding
women living
with HIV,
and their
partners/
support
health
workers
Targeted client
communication
Standard
practice,
non-digital
intervention
behaviours regarding elimination of mother-
to-child transmission (EMTCT) (includes
adherence to protocols, retention of mother–
infant pairs, antiretroviral adherence)
9. [Use/Demand] Change in use of EMTCT
services, including complementary services
or accessing EMTCT information or services
(e.g. receipt of test results, infant diagnosis
and initiation of prophylactics)
outcomes such as CD4 count)
13. Pregnant/postpartum women’s satisfaction
intervention
Maternal
and newborn
health; EMTCT
of HIV and
syphilis
Parents
and other
caregivers
of children
under the age
of five years
Targeted client
communication
Standard
practice,
non-digital
intervention
behaviours regarding newborn and child
health (e.g. adherence to protocols, retention
in services/vaccination follow-up)
2. [Use/Demand] Change in use of newborn
and child health care services, including
complementary services
3. [Efficiency] Change in timeliness of receiving/
accessing newborn and child health services/
information (e.g. reporting of adverse drug/
vaccination effects)
newborn and child health and well-being (e.g.
diarrhoeal incidence, malaria, immunization
rate)
6. Parents’/caregivers’ satisfaction with/
intervention
Child health

Health
domains
of focus in
systematic
reviews
Lay/
community
health
workers and
professional
health
workers for
clients’health
Decision
support
Standard
practice,
non-digital
intervention
2. [Quality] Change in health workers’ skills/
ability to undertake the tasks assigned or
provide services
3. [Quality] Change in providers’ adherence to
recommended practice or clinical practice
recommended)
4. [Quality] Change in providers’ ability for
screening and prioritizing groups of clients
5. [Use] Change in patient loss to follow-up/
discontinuation of services
6. [Efficiency/Quality] Change in time between
including time for referrals and service
linkages
intervention
All – no
restrictions

Health
domains
of focus in
systematic
reviews
Lay/community
health
workers and
professional
health
workers for
clients’ health
Digital tracking
of client’s health
status and
services (within
a health record)
combined with
decision support
Digital tracking
of client’s health
status and
services (within
a health record)
combined
with decision
support and
targeted client
communication
Standard
practice,
non-digital
intervention
1. [Information/Data] Change in the quality
of data about services provided (accuracy,
timeliness, completeness of data)
2. [Use/Demand] Change in patients’/clients’ use
of primary care services
3. [Quality] Change in health workers’ adherence
to recommended practice or clinical practice
recommended, etc.)
4. [Quality] Change in screening and
prioritization of groups of patients
5. [Quality] Change in patient loss to follow-up/
discontinuation, and service linkage
6. [Quality] Change in time between
presentation and appropriate management
7. [Efficiency] Change in health workers’ time
spent on administrative tasks
acceptability of the intervention
For the combination of digital tracking,
decision support and targeted client
communication:
1. [Use/Behaviour] Change in clients’ targeted
behaviours (e.g. adherence to protocols,
retention in services/vaccination follow-up)
services
3. [Efficiency] Change in clients’ timeliness of
receiving/accessing services
All – no
restrictions

Health
domains
of focus in
systematic
reviews
Lay/community
health
workers and
professional
health
workers for
clients’ health
(any health
issue)
mLearning Standard
practice,
non-digital
intervention
1. [Use/Demand] Change in patients’ use of
2. [Quality/Attitude] Change in health workers’
attitudes and norms, motivation, self-efficacy,
empowerment, responsiveness to clients’
needs with regard to health service delivery/
health issue
3. [Quality] Change in health workers’ skills/
ability to undertake the tasks assigned or
provide services
4. [Quality] Change in health workers’ adherence
to recommended practice or clinical practice
recommended)
5. [Efficiency/Quality] Change in time between
including time for referral services and service
linkages
7. Health workers’ and patients’ satisfaction
intervention
All – no
restrictions
EMTCT: elimination of mother-to-child transmission; SRH: sexual and reproductive health; STI: sexually transmitted infection

Annex 3.
Contributors
Guideline development group
Pascale Allotey (co-chair)
Director
United Nations University International Institute for
Global Health
Malaysia
Alain Labrique (co-chair)
Director, Global mHealth Initiative
Johns Hopkins Bloomberg School of Public Health
United States of America (USA)
Smisha Agarwal
Associate
Population Council
USA
Fazilah Shaik Allaudin
Senior Deputy Director
Planning Division
Ministry of Health
Malaysia
Subhash Chandir
Director, Maternal and Child Health
Interactive Research and Development (IRD)
Pakistan
Shrey Desai
Head, Community Outreach
Society for Education Welfare and Action – Rural (SEWA
Rural)
India
Vajira H.W. Dissanayake
Past President
Health Informatics Society of Sri Lanka
Sri Lanka
Frederik Frøen
Head of Research/Chief Scientist
Norwegian Institute of Public Health
Norway
Skye Gilbert
Deputy Director, Digital Health
PATH
USA
Rajendra Gupta
Adviser
Ministry of Health and Family Welfare
India
Robert Istepanian
Institute of Global Health Innovation
Faculty of Medicine, Imperial College
United Kingdom of Great Britain and Northern Ireland
Oommen John
Senior Research Fellow
The George Institute for Global Health
India
Karin Källander
Senior Research Adviser
Malaria Consortium
United Kingdom
Gibson Kibiki
Executive Secretary
East African Health Research Commission
United Republic of Tanzania
S. Yunkap Kwankam
Executive Director
International
Society for Telemedicine and eHealth (ISfTeH)
Switzerland
Amnesty E. LeFevre
Honorary Associate Professor
University of Cape Town
South Africa

Alvin Marcelo
Executive Director
Asia eHealth Information Network
Philippines
Patricia Mechael
Principal and Policy Lead; Co-founder
HealthEnabled
USA
Marc Mitchell
Founder, D-tree International
Adjunct Lecturer, Harvard T.H. Chan School of Public
Health
USA
Thomas Odeny
Research Scientist
Kenyan Medical Research Institute
Kenya
Hermen Ormel
Senior Adviser, Global Health
KIT Royal Tropical Institute
The Netherlands
Olasupo Oyedepo
Director
African Alliance of Digital Health Networks
Nigeria
Caroline Perrin
Division of eHealth and Telemedicine
Geneva University Hospitals
Switzerland
Dr Kingsley Pereko
Country Coordinator, People’s Health Movement
University of Cape Coast, School of Medical Sciences
Ghana
Anshruta Raodeo
Director, Standing Committee on Sexual and Reproductive
Health including HIV/AIDS (SCORA)
International Federation of Medical Students’ Association
India
Chris Seebregts
Chief Executive Officer
Jembi Health Systems
South Africa
Lavanya Vasudevan
Research Scholar
Center for Health Policy and Inequalities Research
Duke Global Health Institute
USA
Hoda Wahba
Vice President
Ain Shams University Virtual Hospital
Egypt
External review group
Patricia Garcia
Professor, School of Public Health
Universidad Peruana Cayetano Heredia
Peru
Teng Liaw
Professor of General Practice and Head of WHO
Collaborating Centre on eHealth
University of New South Wales, Sydney
Australia
Steve Ollis
Senior Digital Health Advisor
Maternal and Child Survival Program
USA
Xenophon Santas
Associate Director for Informatics and Information
Resources, Center for Global Health Leadership
Centers for Disease Control and Prevention
USA
Maxine Whittaker
Dean, College of Public Health
Medical and Veterinary Sciences
James Cook University
Australia

External partners and observers
David Heard
Head of Digital Health
Novartis Foundation
Switzerland
Carl Leitner
Deputy Director
Digital Square
USA
Ingvil Von Mehren Saeterdal
Head of Section, Global Health
Norway
Merrick Schaefer
Digital Health Lead
U.S. Global Development Lab
United States Agency for International Development
(USAID)
USA
Chaitali Sinha
Senior Programme Specialist
International Development Research Centre
Canada
Adele Waugaman
Digital Health Adviser
USAID
USA
William Weiss
Monitoring and Evaluation Specialist
Bureau for Global Health
USAID
USA
United Nations agencies
Sean Blaschke
Technology for Development Business Analyst
Eastern and Southern Africa Regional Office
United Nations Children’s Fund (UNICEF)
Kenya
Hani Eskandar
ICT Applications Coordinator
Telecommunication Development Bureau (BDT)
International Telecommunication Union (ITU)
Switzerland
Remy Mwamba
Statistics and Monitoring Specialist
Implementation Research and Delivery Unit, Health Section
UNICEF
USA
Vincent Turmine
Digital Health Deployment Specialist
Innovations, West and Central Africa Regional Office
UNICEF
Senegal
Sylvia Wong
Innovations Lead
United Nations Population Fund
USA

Technical team
Marita Sporstøl Fønhus
Researcher, Global Health Unit
Claire Glenton
Senior Researcher, Global Health Unit
Simon Lewin
Senior Researcher, Global Health Unit
Systematic review team
Nicholas Henschke (coordination)
Senior Systematic Reviewer
Cochrane Response
Cochrane, London
United Kingdom
Nicola Maayan (coordination)
Systematic Reviewer
Cochrane Response
Cochrane, London
United Kingdom
Smisha Agarwal
Associate
Population Council
USA
Heather Ames
Researcher, Global Health Unit
Norway
Daniela Gonçalves Bradley
Systematic Reviewer
Nuffield Department of Population Health
University of Oxford
United Kingdom
John Eyers
Information Specialist
Independent Consultant
United Kingdom
Caroline Free
Professor of Primary Care and Epidemiology London
School of Hygiene and Tropical Medicine
United Kingdom
Melissa Palmer
Researcher
London School of Hygiene and Tropical Medicine
United Kingdom
Sasha Shepperd
Professor
Nuffield Department of Population Health
University of Oxford
Lavanya Vasudevan
Visiting Scholar
Duke Global Health Institute
USA

WHO steering group
WHO headquarters
Department of Reproductive Health and
Research
Ian Askew
Venkatraman Chandra-Mouli
Doris Chou
Mary Eluned Gaffield
Lianne Gonsalves
Garrett Mehl
Manjulaa Narasimhan
Olufemi Oladapo
Lale Say
Tigest Tamrat
Özge Tunçalp
Teodora Wi
Department of Service Delivery and Safety
Maki Kajiwara
Edward Kelley
Diana Zandi
Department of Innovation, Evidence and
Research
Doris Ma Fat
Kathryn O’Neill
Knut Staring
Global TB Programme
Dennis Falzon
Hazim Timimi
Department of HIV/AIDS
Cheryl Johnson
Prevention of Noncommunicable Diseases
Virginia Arnold
Per Hasvold
Surabhi Joshi
Sameer Pujari
Health Workforce
Onyema Ajuebor
Giorgio Cometto
Essential Medicines and Health Products
Lisa Hedman
Alliance for Health Policy and Systems
Research
Etienne Langlois
Maternal, Newborn, Child and Adolescent
Health
Theresa Diaz
Martin Meremikwi
Wilson Were
Immunizations, Vaccines and Biologicals
Jan Grevendonk
WHO regional offices
Regional Office for Africa
Derrick Muneene
Leopold Ouedraogo
Regional Office for the Americas/
Pan American Health Organization (PAHO)
Rodolfo Gomez
David Novillo
Regional Office for the Eastern Mediterranean
Mohamed Hassan Nour
Regional Office for South-East Asia
Mark Landry
Regional Office for the Western Pacific
Navreet Bhataal
Jun Gao

Annex 4.
Summary of declarations of interest
Name Declarations of interest Management
Dr Smisha Agarwal Led the development of Cochrane reviews contributing to
this guideline. Received research support from the GSM
Association (GMSA) to provide monitoring and evaluation
support to its deployment of digital health programmes in
up to 10 countries
Excluded from final voting
on the recommendations
about interventions
related to these three
systematic reviews
Dr Pascale Allotey None declared No further action taken
Dr Fazilah Shaik Allaudin Both with payments from the World Health Organization
(WHO), conducted a mission to provide technical assistance
to Nepal and participated in a consultation of experts on
eHealth for integrated service delivery in the WHO Western
Pacific Region
No further action taken
Dr Subhash Chandir None declared No further action taken
Dr Shrey Desai None declared No further action taken
Professor Vajira H.W.
Dissanayake
In my position as president of the Health Informatics
Society of Sri Lanka (HISSL) as well as other similar
positions, I have been involved in promoting digital health
Dr Frederik Frøen None declared No further action taken
Professor Patricia Garcia None declared No further action taken
Ms Skye Gilbert None declared No further action taken
Dr Gibson Kibiki None declared No further action taken
Mr Rajendra Gupta None declared No further action taken
Professor Robert
Istepanian
None declared No further action taken
Professor Teng Liaw None declared No further action taken
Dr Oommen John None declared No further action taken
Dr Karin Källander I often speak to the media and in conferences and
meetings, giving statements about the role of digital health
interventions for health care provision in Africa and Asia.
Research and project support from the Bill & Melinda Gates
Foundation, Comic Relief and the United Nations Children’s
Fund (UNICEF)

Name Declarations of interest Management
Dr Alain Labrique Research grants received from the Aetna Foundation, the
Bill & Melinda Gates Foundation, Johnson & Johnson, the
UBS Optimus Foundation and WHO
Dr Amnesty LeFevre None declared No further action taken
Dr Alvin Marcelo Member of the Philippines National eHealth Technical
Working Group representing academia and, in this position,
consulted by government agencies as an expert on eHealth.
I have stated my views on the importance of digital health
for achieving and measuring universal health coverage. The
University of the Philippines contributes to my working
group salary
Dr Patricia Mechael Received research support from the Bill & Melinda Gates
Foundation; Gavi, the Vaccine Alliance; Johnson & Johnson;
The ELMA Philanthropies; Royal Philips of the Netherlands,
UNICEF; and the United States Agency for International
Development
Dr Marc Mitchell Receives 50% of employment salary from D-tree
International
Excluded from discussion
and voting on decision
support
Mr Steve Ollis None declared No further action taken
Dr Thomas Odeny None declared No further action taken
Dr Hermen Ormel None declared No further action taken
Mr Olasupo Oyedepo Serve as project director for a programme providing
technical assistance to the Government of Nigeria to
operationalize its national eHealth strategy
Caroline Perrin None declared No further action taken
Mr Kingsley Pereko None declared No further action taken
Dr Anshruta Raodeo None declared No further action taken
Dr Chris Seebregts None declared No further action taken
Dr Lavanya Vasudevan Conducted a systematic review that contributed to this
guideline. Receives employment and salary support from
Aetna, the National Institutes of Health of the United States
of America (NIH) and WHO
Excluded from final voting
on the recommendations
about the interventions in
this systematic review
Dr Hoda Wahba None declared No further action taken
Dr Maxine Whittaker None declared No further action taken

Annex 5.
Evidence maps and illustrative research questions
The tables below illustrates the general trends in the evidence found in the effectiveness reviews,
demonstrating low and very low certainty evidence across most interventions. For more details
on the specific interventions and outcomes, please review the summary of findings in Web
Supplement 1.
In addition, specific research gaps and accompanying illustrative research questions are listed
Table A5.4. These questions should be addressed using rigorous methods.

Table A5.1 Effectiveness evidence for client interventions
Digital
intervention
Unintended
consequences Resource use
Satisfaction and
acceptability
Utilization of
health services
Health behaviour,
status and well-being
TCC –
adolescents
TCC – adults
TCC – pregnant
+ postpartum
TCC – pregnant
+ postpartum
with HIV
TCC –
children <5
Client-to-
provider
telemedicine
TCC stands for targeted client communication. This intervention was reviewed across five population groups.
This table does not reflect information on satisfaction and acceptability obtained from qualitative reviews.
The comparison for all interventions reflected on these tables is standard care. Please see Web Supplement 1 for other comparison groups for TCC.
Table A5.2 Effectiveness evidence for health worker (HW) interventions
Digital
intervention
Unintended
Satisfaction/
acceptability
HW
performance
HW skills/
attitudes
HW
knowledge
Clients’
utilization
of health
services
Clients’ health
behaviour,
health status/
well-being
Provider-to-
provider
telemedicine
Decision support
Decision support
+ digital tracking
Decision support
+ digital tracking
+ TCC
mLearning
Table A5.3 Effectiveness evidence for Health system interventions
Digital
intervention
Unintended
Satisfaction/
acceptability
Coverage of
birth/death
notification
Timeliness of
birth /death
notification
Coverage of
newborn or
child health
services
Timeliness of
newborn or
child health
services
Availability of
commodities
Quality and
timeliness
of stock
management
Birth
notification
Death
notification
Stock
notification
Key
UNKNOWN LITTLE OR NO DIFFERENCE POSITIVE EFFECT NEGATIVE EFFECT
Not applicable/Not measured May make little or no difference
(low certainty evidence)
May have benefits
May lead to harm
Uncertain effect because of
very low certainty evidence
Probably makes
little or no difference
(moderate certainty evidence)
Probably has benefits
Probably leads to harm
no incidence
No evidence identified
Makes little or no difference
(high certainty evidence)
no incidence
Has benefits
no incidence
Leads to harm
no incidence
Size of bubbles reflects the number of studies contributing to the outcome1-3
7-10 4-6

Intervention-specific research gaps
Table A5.4 outlines the specific research gaps, with illustrative research questions, identified for
each of the interventions included in the guideline. These research questions should be addressed
using rigorous methods.
Table A5.4 Research gaps
Intervention
Evidence-
to-decision
domain Research gaps and illustrative research questions
Birth and
death
notification Effectiveness
ȺȺ What is the effect of birth and death notification on the quality and timeliness
of birth and death reporting or on the accountability for responding to the
data?
ȺȺ Does notification by mobile devices lead to more timely and complete legal
registration, in the case of births, increased coverage and timeliness of health
and other social services (e.g. vaccination), or in the case of deaths, increased
recording of the causes?
Acceptability
ȺȺ What is the acceptability of birth and death notification via mobile devices,
rather than through standard practices of notification? Research should include
how these interventions interact with the sociocultural norms and needs of
different communities regarding births and deaths and their notification.
Feasibility
ȺȺ What are the legal, ethical, data security and policy requirements for allowing
new groups of people or cadres of health worker to notify births and deaths?
What types of modification to existing legal frameworks would be needed to
implement birth and death notification by mobile devices at national scale?
ȺȺ What are the most appropriate ways to train health workers and other people
designated to use birth and death notification?
ȺȺ In what ways do birth (and infant death) notification provide opportunities to
link maternal health records with child health outcomes?
Resource use
ȺȺ See overarching research gaps in section 5.1
Gender, equity
and rights
ȺȺ How does this intervention increase or decrease health-related disparities? Are
there population groups or settings that may not be able to benefit from this
intervention, and how can this be addressed?

Intervention
Evidence-
to-decision
Stock
notification
and
commodity
management
Effectiveness
ȺȺ What is the effect of stock notification and commodity management via mobile
devices on improved availability/reduced stock-out of commodities at the
point of care?
ȺȺ What are the health system conditions that contribute to the effectiveness
of this intervention (for example, supervision of health workers, effective
transport of products, drug access/purchase policies)?
ȺȺ Future research should also be conducted across a range of settings.
Acceptability
ȺȺ No research gaps identified
Feasibility
ȺȺ How can digital stock notification and commodity management systems be
implemented so that they are aligned closely with both national ordering
routines and local needs, and are also supported by well-functioning national
and subnational commodity management?
ȺȺ What can be learnt from practices in logistics management information
systems used outside of the health sector that may be applicable to primary
health care settings?
Resource use
ȺȺ What are the potential cost savings from introducing digital stock notification,
for example through reducing the need for buffer stock and improving the
accuracy of stock need forecasts?
Gender, equity
and rights

Intervention
Evidence-
to-decision
Client1
-to-
provider
telemedicine Effectiveness
ȺȺ What types of digital channel used in facilitating client-to-provider
telemedicine are most effective (for example, transfer of images, voice, text,
and other delivery channels)? Under which circumstances should these
different channels be used?
ȺȺ Future research should include the following outcomes:
ȺȺ use of health services
ȺȺ health behaviour, status and well-being
ȺȺ health worker and client satisfaction
ȺȺ unintended consequences, including the specific risks and safety concerns for
implementing telemedicine different health domains or conditions.
Acceptability
ȺȺ How does this intervention influence health workers’ ability to communicate
or explain information to clients, including issues of liability? Linked to this, in
what ways does this intervention change interactions between clients/patients
and health workers?
ȺȺ Further research in low- and middle-income settings is especially needed.
Feasibility
ȺȺ What mechanisms can address identified implementation barriers, such as
concerns about data privacy obtaining informed consent, and challenges
in network connectivity that may compromise the quality of information
exchanged (e.g. loss of quality of image files, interrupted connection)?
Resource use
ȺȺ What are the resources needed to implement client-to-provider telemedicine,
and what is the cost-effectiveness of this intervention? This should include
research on the cost-effectiveness of different delivery channels, such as voice-
based consultations, image exchanges and other modalities to facilitate client-
to-provider telemedicine for different health issues.
Gender, equity
and rights
ȺȺ How does this intervention increase or decrease health-related disparities? Are
there population groups or settings that may not be able to be able to benefit
from this intervention, and how can this be addressed?

Intervention
Evidence-
to-decision
Provider-
to-provider
telemedicine Effectiveness
ȺȺ What are the conditions that contribute to the effectiveness of provider-to-
provider telemedicine?
ȺȺ Future research should include the following outcomes:
»» health worker performance and adherence to recommended practice, quality
of care provision
»» health behaviour, status and well-being
»» health worker and client satisfaction
»» unintended consequences, including the specific risks and safety concerns
for implementing telemedicine different health domains or conditions.
Acceptability
ȺȺ How is provider-to-provider telemedicine perceived by health workers to
influence inter-professional interactions and collaboration?
Feasibility
ȺȺ What are the potential barriers to implementing these interventions, and
how can these be mitigated? Such barriers include, for example, challenges
in connectivity and its resulting consequences on the quality of information
exchange (e.g. loss of quality of image files, interrupted connections).
Resource use
ȺȺ What are the resources needed to implement provider-to-provider
telemedicine, and what is the cost effectiveness of this intervention?
This should include research on the cost-effectiveness of different delivery
channels, such as voice-based consultations, image exchanges and other
modalities, to facilitate provider-to-provider telemedicine for different
health issues.
Gender, equity
and rights

Intervention
Evidence-
to-decision
Targeted
client
communication Effectiveness
ȺȺ How does the frequency, dose, delivery channel and overall exposure to
content of targeted client communication affect behaviour change and health
outcomes?
ȺȺ Future research on effectiveness should consider the following outcomes:
»» use of health services
»» health behaviour, status and well-being
»» satisfaction with services
»» knowledge and attitudes (for adolescent populations)
»» unintended consequences.
Acceptability
ȺȺ Most studies to date have asked people about their views were they to
receive targeted communications via mobile devices, while some studies have
evaluated people’s experiences within pilot projects or randomized trials.
Future research should focus on the views of participants involved in national-
scale targeted client communication programmes.
ȺȺ What is the acceptability of different formats and delivery mechanisms across
different sociocultural contexts and population groups, such as adolescents?
Feasibility
ȺȺ What strategies can be used to address privacy concerns and to mitigate any
potential negative effects of transmitting sensitive health content, including
ways to enforce consent and the ability to opt out of programmes?
ȺȺ What ways can be used to maintain contact with clients who regularly change
their phone numbers, or who have limited or shared access to mobile devices?
Resource use
ȺȺ What is the cost-effectiveness of different delivery channels, such as voice, text
messages, USSD, and smartphone applications?
Gender, equity
and rights
ȺȺ What strategies can be used to ensure equal access to and use of targeted
client communication services for all groups, including people with poor access
to mobile devices and/or poor network coverage, people who speak minority
languages and people with low literacy or poor technological literacy and skills?
ȺȺ Future research assessing the effectiveness of targeted client communication
using mobile devices should make efforts to ensure that disadvantaged
populations are included. Trials should avoid excluding, wherever possible,
participants on the basis of mobile device ownership, literacy levels, language
or participation in formal health care programmes.
Other ȺȺ Where possible, research should take an integrated approach that includes
outcomes across the continuum of care in pregnancy, childbirth and child
health, as well as across sexual and reproductive health in general.

Intervention
Evidence-
to-decision
Health
worker
decision
support
Effectiveness
ȺȺ What is the effectiveness of health worker decision support via mobile devices
across different settings, health domains, levels of health care, and among
health workers with different levels of training? Future research should focus
on these outcomes:
»» health worker performance and adherence to recommended practice, quality
of care provision
»» clients’/patients’ use of services
»» clients’/patients’ health behaviour, status and well-being
Acceptability
ȺȺ How is decision support via mobile devices perceived by health workers and
clients, and how does it influence their interactions in the provision of services?
Feasibility
ȺȺ What mechanisms can be used to validate the health content within decision
support systems, to ensure that the recommended clinical practices are
congruent with the best available evidence?
Resource use
Gender, equity
and rights
Other ȺȺ What mechanisms can be used to ensure that decision support tools evolve
with new clinical evidence and subsequent policy changes? The development
of the clinical algorithms used within decision support systems is presently an
inexact science. Further research is needed to identify best practice, to develop
and refine these algorithms both in terms of their clinical effectiveness and
their ease of use and acceptability for health workers and clients. The use of
artificial intelligence for the development of decision support systems is an
emerging area that may help to refine algorithms, but more research is needed
on acceptability, feasibility and ethics.

Intervention
Evidence-
to-decision
Digital
tracking
with decision
support and
targeted client
communication
Effectiveness
ȺȺ What is the effectiveness of digital tracking across different settings and health
domains? Research should focus on these outcomes:
»» health worker performance and adherence to recommended practices;
quality of care provision
»» clients’/patients’ use of health services, including follow-up services
»» quality of data on the services provided
»» clients’/patients’ health behaviour, status and well-being
Acceptability
ȺȺ What approaches can be used to minimize the dual burden on health workers
of operating paper and digital systems?
Feasibility
ȺȺ What are the policy requirements for transitioning from paper to digital
systems for client health records, including the establishment and
institutionalized use of unique identification mechanisms?
ȺȺ What are the implementation approaches and requirements for maintaining
a longitudinal client record across the continuum of care and for ensuring
linkages of records across different facilities?
ȺȺ How should service delivery be planned for those individuals and communities
who opt out of tracking when digital tracking systems are implemented at
scale?
Resource use
ȺȺ What are the resources needed to implement and maintain digital tracking
combined with health worker decision support and/or targeted client
communication?
ȺȺ Future research should also identify the potential savings from removing
or reducing the costs of printing and assess the cost-effectiveness of these
interventions. Modelling approaches such as the Lives Saved Tool (88) may be
helpful.
Gender, equity
and rights
ȺȺ How can digital tracking be implemented among marginalized populations,
such as migrants and displaced populations, which may not be included within
a unique identification system?
Other ȺȺ What are the key feasibility, acceptability, resource use and equity
considerations linked to incorporating emerging technologies that use
biometric identification data to uniquely identify each client, including infants?
This includes technologies such as facial recognition and fingerprint and optical
scanning.

Intervention
Evidence-
to-decision
mLearning
Effectiveness
ȺȺ What are the health system conditions that contribute to the effectiveness of
mLearning? Research should include these outcomes:
»» health worker skills and attitudes, including long-term effects on these
outcomes
»» health worker performance and adherence to recommended practice; quality
of care provision
»» client health behaviours
Acceptability
ȺȺ No research gaps identified
Feasibility
ȺȺ What are the potential barriers to implementing this intervention, including
potential losses to the per diem remuneration received by health workers when
shifting from face-to-face to mLearning modalities?
Resource use
ȺȺ What are the resources needed to implement mLearning, and what is the
cost-effectiveness of these interventions? Research should consider the cost-
effectiveness across different mLearning delivery channels.
ȺȺ Resource use and cost-effectiveness was recognized as a cross-cutting research
gap across all of the examined digital health interventions.
Gender, equity
and rights

World Health Organization
Avenue Appia 20
1211 Geneva 27
Switzerland
ISBN 978-92-4-155050-5

Digital Interventions for Health Systems Strengthening

More Related Content

What's hot (20)

Similar to Digital Interventions for Health Systems Strengthening (20)

More from Prof. Rajendra Pratap Gupta (20)

Recently uploaded (20)

Digital Interventions for Health Systems Strengthening