Received: 24 December 2021 Revised: 23 March 2022 Accepted: 12 April 2022

DOI: 10.1002/lrh2.10311

RESEARCH REPORT

Conceptualizing learning health systems: A mapping review

Josefien de Bruin1,2 | Cheryl Bos1 | Jeroen Nathan Struijs1,3 |

Hanneke Wil-Trees Drewes1 | Caroline Astrid Baan2

1
Department of Quality of Care and Health
Economics, National Institute for Public Health Abstract
and the Environment, Center for Nutrition, Introduction: Health systems worldwide face the challenge of increasing population
Prevention and Health Services, Bilthoven, the
Netherlands health with high-quality care and reducing health care expenditure growth. In pursuit
2
Tranzo, Tilburg School of Social and for a solution, regional cross-sectoral partnerships aim to reorganize and integrate
Behavioral Sciences, Tilburg University,
services across public health, health care and social care. Although the complexity of
Tilburg, the Netherlands
3
Department of Public Health and Primary regional partnerships demands an incremental strategy, it is yet not known how
Care/LUMC-Campus The Hague, Leiden learning works within these partnerships. To understand learning in regional cross-
University Medical Centre, The Hague, the
Netherlands sectoral partnerships for health, this study aims to map the concept Learning Health
System (LHS).
Correspondence
Caroline Astrid Baan, Tranzo, Tilburg School of Methods: This mapping review used a qualitative text analysis approach. A literature
Social and Behavioral Sciences, Tilburg search was conducted in Embase and was limited to English-language papers published
University, Tilburg, the Netherlands.
Email: [email protected] in the period 2015-2020. Title-abstract screening was performed using established
exclusion criteria. During full-text screening, we combined deductive and inductive cod-
ing. The concept LHS was disentangled into aims, design elements, and process of learn-
ing. Data extraction and analysis were performed in MAX QDA 2020.
Results: In total, 155 articles were included. All articles used the LHS definition of the Insti-
tute of Medicine. The interpretation of the concept LHS varied widely. The description of
LHS contained 25 highly connected aims. In addition, we identified nine design elements.
Most elements were described similarly, only the interpretation of stakeholders, data infra-
structure and data varied. Furthermore, we identified three types of learning: learning as 1)
interaction between clinical practice and research; 2) a circular process of converting rou-
tine care data to knowledge, knowledge to performance; and performance to data; and 3)
recurrent interaction between stakeholders to identify opportunities for change, to reveal
underlying values, and to evaluate processes. Typology 3 was underrepresented, and the
three types of learning rarely occurred simultaneously.
Conclusion: To understand learning within regional cross-sectoral partnerships for
health, we suggest to specify LHS-aim(s), operationalize design elements, and choose
deliberately appropriate learning type(s).

KEYWORDS
conceptualization, learning health system, types of learning

This is an open access article under the terms of the Creative Commons Attribution-NonCommercial-NoDerivs License, which permits use and distribution in any
medium, provided the original work is properly cited, the use is non-commercial and no modifications or adaptations are made.
© 2022 The Authors. Learning Health Systems published by Wiley Periodicals LLC on behalf of University of Michigan.

Learn Health Sys. 2023;7:e10311. wileyonlinelibrary.com/journal/lrh2 1 of 10

https://doi.org/10.1002/lrh2.10311
 23796146, 2023, 1, Downloaded from https://onlinelibrary.wiley.com/doi/10.1002/lrh2.10311 by Capes, Wiley Online Library on [16/03/2025]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License
2 of 10 de BRUIN ET AL.

TABLE 1 Evolving definitions of learning health system

Etheredge 2007 LHS is a system aimed at increasing the value of

1 | I N T RO DU CT I O N health care without draconian cost cutting.
Rapid learning from new evidence for practice
and policy, adapting electronic health records
Worldwide health systems are challenged to provide high-quality,
(EHRs) in clinical settings aimed at integrating
accessible, and affordable care.1 During the last decade, multiple clinical, financial, and administrative data.”13
transformational programs have emerged in various countries among
Institute of The LHS generates and applies the best evidence
which Accountable Care Organizations (U.S.),2 Vancouver City Pro- Medicine for the collaborative healthcare choices of
grams (Canada),3 Vanguards (UK),4 Gesundes Kinzigtal (Germany),5 2007 each patient and provider; drives the process
and the Population Health Management initiatives (the Netherlands).6 of discovery as a natural outgrowth of patient
care; and ensures innovation, quality, safety
In general, these initiatives comprise regional cross-sectoral partner-
and value in healthcare.”14
ships for health, which aim to reorganize and integrate services across
Institute of Within a LHS science, informatics, incentives,
public health, health care, social care, and wider public services to Medicine and culture are aligned for continuous
increase population health with high-quality care and a reduction in 2011 improvement and innovation, with best
health care expenditure growth.7 practices seamlessly embedded in the delivery
process and new knowledge captured as an
These partnerships are complex due to the interdependence of
integral by-product of the delivery
multiple factors contributing to health and well-being itself, the large experience.”12
number of involved stakeholders with potentially conflicting interests,
and the interrelatedness within the health system environment includ-
ing existing legislations and regulations.8 Complexity in regional cross-
sectoral partnerships demands an incremental strategy in order to strive to reveal the underlying vision and various perspectives on LHS.
remain sight of the formulated goals, while at the same time meeting In this article, in order to understand learning within regional partner-
9,10
upcoming challenges. Hence, learning cycles and emergent learn- ships, the overall aim is to get more insight in the use of the concept
ing are considered as a prerequisite in cross-sectoral partnerships for LHS. Therefore, we focus on the following research questions: which
health.6,7,11 aims are formulated regarding LHS? How are the LHS-design ele-
Up until now, it is not known how to operationalize “learning” ments interpreted? And last, what processes of learning are applied?
within regional cross-sectoral partnerships. To understand how
regional partnerships for health learn, the concept of Learning Health
System was expected to be closely related. Originally introduced to 2 | METHODS
shorten the time from proven effective drugs in evidence-based medi-
cine to clinical practice, the definition of LHS has been evolved over As the aim of this study is to clarify the use of the concept LHS, a
time into “within a LHS, science, informatics, incentives, and culture are mapping review was conducted. The purpose of a mapping review is
aligned for continuous improvement and innovation, with best practices to categorize, classify, and characterize patterns, trends or themes
seamlessly embedded in the delivery process and new knowledge cap- with regard to a specific review question.25 Unlike a systematic
12
tured as an integral by-product of the delivery experience” ; see full review, a mapping review does not appraise the findings, but merely
definitions in Table 1. The development of the concept LHS has been aims to examine the range, nature and evolution of a topic area.26 We
an effort by all. Several (semi-) governmental agencies (eg, AHRQ and chose the mapping review approach as our aim to clarify the concept
PCORI) and multi-stakeholder networks (eg, Learning Health Commu- LHS corresponds to the purpose of mapping review to “map out” and
15–17
nity) have built upon on and contributed to existing work. Fur- thematically understand the pre-existing topic, including visual syn-
thermore, the journal of Learning Health System and the academic thesis of the data.27 Moreover, pre-existing literature on LHS is
Department of Learning Health Sciences at the University of Michigan numerous and highly diverse in article types. This mapping review
Medical School have played a role in shaping operational views of the used a 3-step approach: composing the search strategy, selecting rele-
LHS vision. In the scientific literature, several models describe differ- vant studies, and data-extraction and analysis.
ent angles of the concept LHS such as data infrastructure,18 data
architecture,19,20 LHS classification,21 and value-creating operationa-
lization.22 A recent study by Easterling et al. clarified the operationali- 2.1 | Search strategy
zation of LHS within organizations and revealed an LHS-taxonomy and
five bodies of work.23 The search was conducted using Embase and was limited to English-
However, an overview of the concept LHS is lacking. As the Insti- language papers published between January 2015 and May 2020.
tute of Medicine put a broad definition on LHS in place to enable Although the early development and historical context shaped the
context-independent implementation, LHS is now interpreted understanding of the concept LHS, the scope of this study focused on
widely.20,24 Hence, we aim to clarify the use of the concept LHS–by the most recent literature to reflect the current use of the concept
studying the description of LHS in scientific literature. As such, we LHS. A concise search strategy was developed in collaboration with a
 23796146, 2023, 1, Downloaded from https://onlinelibrary.wiley.com/doi/10.1002/lrh2.10311 by Capes, Wiley Online Library on [16/03/2025]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License
de BRUIN ET AL. 3 of 10

scientific librarian to identify studies matching the following search process of learning within LHS. Qualitative analysis lead to a framework
term: learning health*[tiab] OR LH*[tiab]. In this way, the search strat- of “process of learning.” We used this framework to study the
egy included all possible terminology of learning health system such (co)-occurrence of the process of learning within the included articles.
as learning healthcare systems and learning health networks and did Lastly, the code “design element” was applied when a component within
not exclude any articles based on spelling differences. the LHS paragraph was stated as important for operationalization or
developing LHS, such as “stakeholders” or “data.”
During data extraction and data analysis, two researchers (JB,
2.2 | Study selection ChB) first executed a pilot of 20 articles, reducing researcher's bias.
After both researchers independently coded 10 articles, the coded
Title-abstract screening was performed by two researchers (JB and articles were crosschecked and discussed to develop a codebook. To
ChB). In preparation, one researcher (JB) developed draft in- and finalize the codebook and to ascertain the replicability of the
exclusion criteria in close collaboration with the research group. In a researchers strategy, 10 additional articles were coded together. Pro-
50-articles pilot screening by JB and ChB, the inclusion and exclusion posed methods and preliminary findings were discussed within and
criteria were clarified and finalized, see Table 2. The researchers then agreed upon by the research group. Using the established codebook
screened all articles independently and, in case of noncongruent opin- and methods, one researcher (JB) extracted the data and analyzed the
ions, discussed the abstracts until an agreement was reached. Articles remaining articles. Due to the explorative study design, both the main
were excluded in full-text screening if a) full-text were not available, researcher and the research group both reflected on findings and
b) they were abstract-only (eg, conference papers), c) they did not added codes when needed. Even though saturation was reached after
explicitly describe the concept LH*, or d) they were not available in 70 articles, all included studies were analyzed to ensure no viewpoint
English. was missed.
Data-extraction and analysis were performed in MAX QDA. To
visualize the code aims, the option Code Matrix in MAX QDA
2.3 | Data extraction and analysis was used.

This mapping review used a qualitative text analysis approach. We

systematically extracted data using the search term Learning Health 3 | RE SU LT S
and LH; and subsequently coded the entire paragraph. For analysis,
we chose to combine deductive and inductive coding. We used 3.1 | Study selection
deductive coding with the codes “definition,” “aim,” “process of
learning,” and “design elements” to differentiate between the sub The Embase-search yielded 631 articles. During title abstract
questions; inductive coding to stick to the articles' description of the screening, 319 articles were excluded. After data extraction and
concept LHS. Text was coded as “definition” when cited in italic or analysis, 164 articles were excluded resulting in a total of
presented between brackets. The code “aim” was used when the goal 155 included articles (Figure 1). For the entire list of included arti-
of LHS was described. This could either be explicitly, for instance, cles, see Data S1.
when words as goal or aim were used: “the goal of LHS is” or “the aim
is to,” or implicitly using the words to or for within the description.
The code “process of learning” was used when authors described the 3.2 | Formulated aims

Overall, 25 aims were described in relation to LHS in the included arti-

TABLE 2 Inclusion and exclusion criteria in title abstract
cles (Table 3). The most prevalent aims were accelerating research,
screening
clinical decision-making, and improving quality of care. To test
Inclusion criteria whether distinct clusters of aims for LHS could be identified, Figure 2
• Presence of learning health* (LH*)
shows the relatedness of the aim-codes that co-occur in one article
• LH* in a OECD Country
• Published after January 2015 for a minimum of four times. In Figure 2, the various nodes lack any
Exclusion criteria thematic congruence. Consequently, we are not able to interpret the
• Absence of LH* (#1) LHS-aims as clusters.
• LH* is only part of the institute's name, the author's title or the
conflict of interest statement. (#2)
• LH* explicitly described in a non-OECD Country (#3)*
• Abstract not available (#4) 3.3 | Design elements
• LH* only mentioned in conclusion as a possible application for a
specific method/analysis/tool (#5) In the description of the concept LHS, nine design elements were
• LH* only described as data source for effect studies (#6)
found (Table 4). Most design elements were described identically;
*Source: https://www.oecd.org/about/members-and-partners/. however, the interpretation of the three elements stakeholders, data
 23796146, 2023, 1, Downloaded from https://onlinelibrary.wiley.com/doi/10.1002/lrh2.10311 by Capes, Wiley Online Library on [16/03/2025]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License
4 of 10 de BRUIN ET AL.

FIGURE 1 Flowchart study

Total hits: n=631
selection
Excluded

Absence of LH (n=58)
LH explicitly described in a non-OECD country (n=16)
Abstract not available (n=74)
LH in conclusion as a possible applicaon for a specific
method/analysis/tool (n=107)
Full text: n=319 LH described as a datasource for effect studies (n=57)

Excluded

Full text not available (n=69)

LH not described (n=44)
Abstracts only (n=50)
Included in review: n=155 Full text not in English (n=1)

T A B L E 3 List of aims in relation to the concept Learning Health of data for (end-) users as well, for instance via patient and providing fac-
System (LHS); ranked on occurrence ing data-dashboards, (eg, Reference 33) or via prognostic models and/or
• accelerate research/formulate hypothesis 28 clinical decision support tools.(eg, References 30,34,38,39) In addition,
• clinical decision-making29 several articles stressed that the data infrastructure included a supportive
• quality of care30 system. Friedman et al describes the supportive system as “the technolo-
• clinical outcomes/patient outcomes24
gies, policies, and standards comprising these [supportive] services constitute
• care delivery/health services15
• reducing healthcare cost15 the infrastructure for the LHS.”40 Maddox et al. specifies this supportive
• value /cost-effective care12 system by “data oversight, which encompasses data governance, regulation,
• innovation11 privacy protection, and data security, has an essential supporting function
• system improvement/transformation11
for the LHS data infrastructure.”41 Furthermore, the focus of the element
• patient health10
• learning10 data infrastructure varied considerably. LHS might be complaint-based;
• population health10 (eg, Reference 42) disease-based43; or patient-based.29
Last, the description of the design element “data” varied widely.
Note: Aims occurred less than 10 times: safety (n = 9), improvement, not-
specified (n = 9), patient experience (n = 5), efficiency (n = 5), policy and The basic and most narrow interpretation of LHS data was routine
management (n = 5), personalized care (n = 5), public health (n = 4), care data, for example, “based on data flowing from routine care.”22
equity (n = 3), panel management (n = 3), data-sharing (n = 2), pilots Several articles expended routine care data with other data (sources)
(n = 1), stakeholder input (n = 1), consumer education (n = 1), solving
such as health-related data (not specified32), patient-reported outcomes,
health-related problems (n = 1), providers in general (n = 1), access to
services (n = 1). (eg, References 43-45), experience of care (patient and professional,35)
social determinants of health, (eg, Reference 46) patient generated data,
infrastructure, and data varied considerably. As such, we will discuss (eg, References 33,47) and geospatial data (eg, Reference 18). These
these three elements in more detail here. additional data (sources) could be linked, for instance, as Steels et Van
First, although many articles stressed the importance of “(multi-) Staa described: “To link data across multiple agencies including health
stakeholder engagement” in LHS, differences are found in the kind of (physical and mental), social care, criminal justice, housing and education
stakeholders to be engaged and their role. The included articles to develop a more complete Learning Health System.”48 Rubinstein et
described various kind of stakeholders: ranging from patients – Warner described the LHS data-sources explicitly, including patient
patients, family members, patient advocates; providers – care providers and generated data: “the data originate in a variety of sources, including
clinicians; to payers, policy makers –healthcare administrators, policy electronic health records (EHRs), claims databases, pharmaceutical clear-
makers; and other experts – researchers, technology experts, health system inghouses, and clinical trial databases. The newest and potentially richest
leaders, thought leaders on continuous improvement, health service man- source, as measured by kilobytes generated, is patient-generated health
agers, and planners. Moreover, the roles of patients and care providers data.”47 All findings related to the element “data” were of quantita-
differ substantially between articles. In some articles, patients and care tive nature, we did not find (explicit) use of qualitative data.
providers were not described as stakeholder, in others as active
participators,36,37 or even co-creators.(eg 31).
Second, the often-mentioned design element “data-infrastruc- 3.4 | Process of learning
ture” was occasionally extended with a so-called “support system.” In
general, data infrastructure was interpreted as the linkage and/or stor- Using inductive analysis of the code “process of learning,” we have
age of different data-sources. Some articles described the availability identified three types of learning (Table 5):
 23796146, 2023, 1, Downloaded from https://onlinelibrary.wiley.com/doi/10.1002/lrh2.10311 by Capes, Wiley Online Library on [16/03/2025]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License
de BRUIN ET AL. 5 of 10

F I G U R E 2 The co-occurrence of Learning Health System (LHS) aims. The smaller the distance between the codes, the more often they
simultaneously occur. The line between two codes show co-occurrence in at least 4 documents

3.4.1 | Interaction between clinical practice and what is known (quality improvement). Bringing these functions together
research to create active, mutual learning cycles (…) has been labeled a ‘learning
health system’.”58 We call this type of learning “interaction between
The first type of learning describes learning as an intermittent infor- clinical practice and research.”
mation exchange between clinical practice and the research domain.
This type of learning is seen as high potential for accelerating research
and implementing knowledge into practice, (eg, References 45-51) for 3.4.2 | Continuous circular routine care data-driven
instance via clinical induced research priorities,(eg, References 52,53); process
via generating and testing hypothesis without randomised controlled
trials, (eg, Reference 54), and/or to evaluate treatment effectiveness For the second type, learning within LHS is described as a (technol-
in specific subgroups, that cannot be studied adequately in random- ogy-aided) continuous circular process of converting (routine care)
ized, controlled trials.(eg, References 55,56) Guise et al. stressed the data to knowledge, knowledge to performance, and performance to
two-way interaction between research and clinical practice with data. In this type of learning, the information stream, data infrastruc-
improving health as ultimate aim: “As such, the LHS concept requires ture, and data are key. Friedman et al described continuous learning
that evidence generation not be an end in itself; efforts to generate evi- cycle. “Learning cycles can occur at various speeds and levels of scale but
dence must be accompanied by equally emphasized efforts to apply it to invariably consist of three core processes, namely1 converting data to
57
improve health.” Teare et al. combined research and quality knowledge (D2K),2 applying knowledge to influence performance (K2P),
improvement to form an LHS, “improving health and services requires and3 documenting changes in performance to generate new data
both better knowledge (research) and better action to adapt and use (P2D).”40 Even though the levels of learning cycles are specified “at
 23796146, 2023, 1, Downloaded from https://onlinelibrary.wiley.com/doi/10.1002/lrh2.10311 by Capes, Wiley Online Library on [16/03/2025]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License
6 of 10 de BRUIN ET AL.

T A B L E 4 List of codes “design element” in relation to Learning T A B L E 5 Typology of learning within Learning Health
Health System (LHS)* System (LHS)

Design elements Example Type of learning Example

Financial “The last area that a learning health system must Learning as (intermittent) Thus research priorities are
incentives have in place is the alignment of incentives. information exchange between aligned with key issues
When the system is based on fee-for-service, the clinical domain and the clinicians face in everyday
full waiting rooms and even fuller procedure research domain practice, and research on those
scheduling is the only way providers can pay issues informs best practice.
the bills and live at the lifestyle level they (Price-Haywood-2015-Clinical
desire.(…)”31 comparative effect, P. 3:5317)
Data- “LHS foundational elements, such as harnessing Learning as a (technology-aided) “Learning cycles can occur at
infrastructure contemporary technology and data support continuous circular process of various speeds and levels of
structures, enhances capacity to collect and converting (routine care) data scale but invariably consist of
use data and evidence (…).”32 to knowledge; knowledge to three core processes, namely1
Policy- “Upscaling a learning health system for palliative performance; and converting data to knowledge
infrastructure care will require intelligent navigation of performance to data. Central (D2K),2 applying knowledge to
several domains: (…) establishment of policies is the information stream. influence performance (K2P),
that favor culture change and reward and3 documenting changes in
measured performance; (…).”33 performance to generate new
data (P2D).”40
Technology “LHS foundational elements, such as harnessing “In a LHS, knowledge is obtained
contemporary technology and data support continuously through routine
structures, enhances capacity to collect and clinical documentation at the
use data and evidence (…).”32 point of care (POC) and turned
Data “At the centre of a LHS ethos is routine capture, into guidance through clinical
transformation and dissemination of data and decision support (CDS), with a
knowledge, with various uses (.)”28 resulting vast repository of data
Learning “As such, LHSs incorporate continuous learning on treatment effectiveness to
at the system, enhance RCTs and evidence-
organizational, departmental, and individual based medicine.”44
levels, in cycles or loops moving from data to Learning as recurrent interaction “shared learning between centers
knowledge and then from knowledge to between stakeholders to with the intention of driving
practice and back again.”20 reveal/discuss underlying the emergence of a nationwide
Evidence and “The LHS brings together the elements of values, to evaluate processes, community of practice”
measurement stakeholder engagement, clinician leadership, and to identify opportunities (Wildman 2019)
best available evidence and measurement, IS for change, and share best-
rigour with integration of HCI in an iterative practices. Human interaction
model that learns from success and failures.”34 and the exchange of
experiences and ideas are
Culture (change) “Organizational culture plays a crucial role in central.
supporting an effective learning health system
approach. Specifically, organizations need to
learn as they go and not be afraid to ‘fail,’ to
foster a spirit of curiosity and courage.”35 3.4.3 | Recurrent interaction between stakeholders
Stakeholders “Importantly, the patients, clinicians, and for collaborative learning
communities are at the center of the model,
indicating engagement and the alignment of
For the third type, learning within LHS is interpreted as (recurrent)
care with their priorities.”36
interaction between stakeholders to identify opportunities for change,
*Due to the qualitative nature of this analysis, the ferquency of “design
for joint goal setting, to reveal underlying values, to evaluate pro-
elements” is not specified, nor could the list be ranked on occurrence.
cesses, and to share best practices. In this type of learning, human
interaction and the exchange of ideas and experiences are central. For
the system, organizational, departmental, and individual level.”20,40,59 instance, Hirsch et al. described activities as “ABI [adaptive biomedical
Most articles did not discriminate between these levels when describ- information] involves bringing stakeholders together to set shared objec-
ing the concept of LHS. In addition, some articles took a more clinical tives, foster trust, structure decision-making, and manage expectations
perspective. Serena et al. described “In a LHS, knowledge is obtained through rapid-cycle feedback loops that maximize product knowledge
continuously through routine clinical documentation at the point of care and reduce uncertainty in a continuous, adaptive, and sustainable learn-
(POC) and turned into guidance through clinical decision support (CDS), ing healthcare system.”60 Menear et al. went a step further and stated
with a resulting vast repository of data on treatment effectiveness to that networks and learning communities contribute to the culture
enhance RCTs and evidence-based medicine.”44 We call this type of switch in LHS “Networks and learning communities that foster trusting
learning “the continuous circular routine care data-driven process.” relations between diverse stakeholders can nurture cultures in which
 23796146, 2023, 1, Downloaded from https://onlinelibrary.wiley.com/doi/10.1002/lrh2.10311 by Capes, Wiley Online Library on [16/03/2025]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License
de BRUIN ET AL. 7 of 10

FIGURE 3 The (co-)occurrence of learning types

learning and improvement is ingrained within their normal operations, analyzed the interpretation of LHS with respect to the aims, interpre-
though fully realizing such culture shifts is considered one of the most tation of design elements, and processes of learning. We found that
22
challenging tasks of LHS implementation.” We call this type of learn- the concept is associated with 25 highly interconnected aims showing
ing “recurrent interaction between stakeholders for collaborative no specific clustering. Furthermore, nine elements were cited as
learning.” design elements; we found that variation occurred in the interpreta-
Analysis of the included articles demonstrates the (co-) occur- tion of the elements “stakeholder engagement,” “data infrastructure,”
rence of the learning types (Figure 3). Figure 3 shows that the majority and “data.” Lastly, three types of learning processes were identified.
of the articles (n = 92) used the second type of learning “the continu- The majority of the articles focused on the second type of learning
ous circular (routine-care) data-driven process.” The first type of learn- “continuous circular (routine care) data-driven process.” The third type
ing “interaction between clinical practice and the research domain” was of learning “recurrent interaction with stakeholders for collaborative
represented in more than one third of the articles (n = 65), and the learning” was underrepresented.
third type of learning ‘recurrent interaction between stakeholders for Putting these results in perspective, we are—as far as we know—
collaborative learning’ occurred the least (n = 34). In total, twelve arti- the first to use a qualitative text analysis approach mapping current
cles applied all three typologies of learning. These twelve articles var- LHS interpretations to understand learning within regional cross-
ied in LHS-scope, focus, scientific background and author. Other sectoral partnerships. In addition to existing literature,20,23,24 this
combinations of learning types were described as well, such as Teare study provides in-depth insights in how (widely) the LHS concept is
et al. relating to both typology 1 and typology 3. “Improving health interpreted. Furthermore, the result of this mapping review is similar
and services requires both better knowledge [research] and better action to that of other health system concepts, among which population
to adapt and use what is known [quality improvement]. Bringing these health management.61
functions together to create active, mutual learning cycles, which tap the Looking in more detail, the extensive number and interrelatedness
experience and expertise of health service users, service providers, of LHS aims shows readiness for the use of the concept LHS but is a
researchers and people skilled in facilitating quality improvement, has point of concern as well. This broad use of LHS aims can possibly be
been labelled a “learning health system.”58
explained by the author's background and by the differences in under-
lying health system goals. For instance, Menear et al. explicitly
described the translation of the concept LHS—based on USA-system
4 | DISCUSSION goals—to the public-funded Canadian health system.22 Our point of
concern is that - as a result of the high number and interrelatedness
This mapping review conceptualizes the term LHS to understand of LHS aims—the concept LHS might become diluted and possibly
learning from a regional cross-sectoral partnership perspective. We become a catchall. In that case, the potential of LHS might not be
 23796146, 2023, 1, Downloaded from https://onlinelibrary.wiley.com/doi/10.1002/lrh2.10311 by Capes, Wiley Online Library on [16/03/2025]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License
8 of 10 de BRUIN ET AL.

realized.24,62 Translating our insights to learning within regional cross- regional partnerships to remain sight of complexity while keeping the
sectoral partnerships, we would recommend regional partnerships to next steps tangible enough to keep moving forward. Furthermore,
formulate clear and specific aims. as transition scholars underscore the importance of multistakeholder
The three elements stakeholders, data, and data infrastructure engagement and a multilevel approach in understanding and
provide food for thought. Interestingly, the role and responsibilities of governing system transition,68,69 typology 3 might be essential in
stakeholders varied, whereas several articles did not even described commitment and alignment of stakeholders at various levels and at
patients and clinicians as stakeholders at all. This is in contrast with various sectors within regional cross-sectoral partnerships for health.
the IOM model of LHS that put patients, clinicians, and communities This study has limitations that need to be considered when inter-
at the core of LHS.63 Furthermore, the multiple data sources and preting the results. One is that the search string was limited to articles
selected measures may contribute to the challenges in LHS compara- published after 2015. Yet, we do not expect that an extended publica-
bility. This might be caused by the different measures selected in vari- tion dates would affect the results of this study, as the variation of
ous countries,64 and by methodological reasons as the chosen data LHS-interpretation is quite wide. In addition, this mapping review had
sources, measures, and analysis might affect LHS-prediction models a qualitative focus and did therefore not aim to be exhaustive. Second,
and clinical decision support tools (learning type 2). Data sources, in order to conceptualize the use of the term LHS, we only analyzed
selected measures, and analyses should thus be chosen carefully. the written description of the term LHS and did not focus on the visu-
Moreover, although we did not find (explicit) data sources of qualita- alizations of LHS models. As our aim was to gain insights in the possi-
tive nature, one might hypothesize that a combination of quantitative ble use of the concept LHS, we decided not to study the theoretical
and qualitative data provides added value for learning. As research visualizations, but focus on the written LHS descriptions by the wider
has embraced mixed-methods design for several reasons among public. However, it might be interesting to compare the different LHS
which understanding context, providing explanations for research visualizations and relate the results to current study.
findings, and confirming and testing hypothesis65,66; this mixed- For further development of learning within regional partnerships,
method approach might be applied in learning processes as well. Last, we recommend a more in-depth understanding of (emergent) learning
as the process of learning is described to occur in various cycles and at within real-life learning health systems. For instance, it might be bene-
various scales (namely individual, organizational, and system),20,21,40 ficial to study how to increase the learning capacity of regional part-
we suggest regional partnerships to operationalize learning and con- nerships in real life LHS, providing leverage points for facilitators. A
current multilayered data infrastructure for all stakeholders, including realist study can present program theories on which strategies increase
patients, clinicians, representatives of healthcare organizations, and regional partnerships' learning capacity, in which context, and why?70 In
policy makers. addition, it could be valuable to focus on the role of patients. Several
The identified types of learning offer opportunity for learning studies showed that patient and citizen engagement in regional part-
strategies within regional cross-sectoral partnerships. Easterling et al. nerships is challenging.71,72 As such, we must learn how to engage
identified five bodies of work in operationalizing LHS within organiza- (with) patients, citizens and communities in real-life LHS. Moreover, it
tions.23 A direct comparison with current article is hindered due to would be interesting to analyze the three identified learning types
differences in both applied focus of articles (operationalization vs con- within regional partnerships over time. For instance by using the
ceptualization), and scope (LHS within organization vs cross sectoral 5-phase model A pathway for transforming health and well-being
partnerships). However, the three types of learning show congruence through regional stewardship as described by Rethink Health73 and also
with the five bodies of work. Our mapping review yielded only twelve observed in the Netherlands.6 Last but not least, it would be illuminat-
articles that combined the three typologies of learning. The focus on ing to reflect upon the short-and long-term effect of the COVID-19
the T2 “continuous circular (routine care) data-centered process” and pandemic on learning within health systems. The COVID-19 pandemic
relatively underrepresented T3 “recurrent interaction between stake- showcases the importance of a learning health system.74 Nevertheless,
holder for collaborative learning” might be explained by context or the question remains: what is the long-term effect of the COVID-19
path-dependency. For instance, available technologies for data infra- pandemic upon learning (within) health systems?
structure and data-sharing, up-to-date electronic health records, and To conclude, this study showed extensive variation in LHS inter-
positive experience with data sharing may impact the (first) preferred pretations and simultaneously provides leverage points for under-
typology of learning.67 In addition, it may be more ambitious to evolve standing learning within regional cross-sectoral partnerships for
in collaborative learning when a health system has emphasized com- health. Although a concept such as LHS is dynamic and evolving, it is
petition. For regional partnerships, all three types of learning are of of importance to create shared language to facilitate operationa-
importance and might complement each other. Learning type lization. We recommend regional partnerships to describe the shared
1 enables learning, innovation, and discovery (scientific dimension); LHS-aims clearly, operationalize the design elements and choose
learning type 2 addresses data integration (technical dimension), and deliberately appropriate learning type(s). We believe that the distin-
learning type 3 focuses on building a community (the social dimen- guished types of learning provide opportunities for reflection on
20
sion). As the regional partnerships' underlying transformational chal- learning within regional cross-sectoral partnerships. Learning from
lenge* is highly complex, the (interplay of the) three processes of successes and failures is crucial to see full-impact of regional cross-
learning could portray additional perspectives. These will enable sectoral partnerships for health.
 23796146, 2023, 1, Downloaded from https://onlinelibrary.wiley.com/doi/10.1002/lrh2.10311 by Capes, Wiley Online Library on [16/03/2025]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License
de BRUIN ET AL. 9 of 10

CONF LICT OF IN TE RE ST 19. McLachlan S, Dube K, Kyrimi E, LAGOS FN. Learning health systems
The authors declare that there is no conflict of interest. and how they can integrate with patient care. BMJ Health Care Inf.
2019;26(1):e100037.
20. Lessard L, Michalowski W, Fung-Kee-Fung M, Jones L, Grudniewicz
ENDNOTE A. Architectural frameworks: defining the structures for implementing
* To reorganize and integrate services across public health, health care, learning health systems. Implementation Sci: IS. 2017;12(1):78.
social care and wider public services to increase population health with 21. McLachlan S, Potts HWW, Dube K, et al. The Heimdall framework for
high-quality care and a reduction in health care expenditure growth supporting characterisation of learning health systems. J Innov Health
Inf. 2018;25(2):77‐87.
22. Menear M, Blanchette MA, Demers‐Payette O, Roy D. A framework
RE FE R ENC E S for value‐creating learning health systems. Health Res Policy Syst.
1. OECD. Fiscal Sustainability of Health Systems: Bridging Health and 2019;17(1):79.
Finance Perspectives. Paris: OECD, Organisation for Economic Co‐ 23. Easterling D, Perry AC, Woodside R, Patel T, Gesell SB. Clarifying the
operation and Development; 2015. concept of a learning health system for healthcare delivery organiza-
2. Alley DE, Asomugha CN, Conway PH, Sanghavi DM. Accountable tions: implications from a qualitative analysis of the scientific litera-
Health Communities‐‐Addressing Social Needs through Medicare and ture. Learn Health Syst. 2021;6(2):e10287.
Medicaid. N Engl J Med. 2016;374(1):8‐11. 24. Budrionis A, Bellika JG. The learning healthcare system: where are we
3. Vancouver Co. A Healthy City for All; Vancouver: City of Vancouver; now? A systematic review. J Biomed Inform. 2016;64:87‐92.
2014:2018‐2025. 25. Booth A, Sutton A, Papaioannou D. Systematic Approaches to a Suc-
4. Office NA. Developing New Care Models Through NHS Vanguards. cessful Literature Review. London: Sage; 2016.
London: National Audit Office; 2018. 26. Paré GKS. Handbook of eHealth Evaluation: an Evidence‐Based
5. Pimperl A, Schulte A, Daxer C, Roth M, Hildebrandt H. Balanced Approach. Victoria (BC): University of Victoria; 2017.
scorecard‐ansatz: case study gesundes Kinzigtal. Monitor Ver- 27. Grant MJ, Booth A. A typology of reviews: an analysis of 14 review
sorgungsforschung. 2013;01:26‐30. types and associated methodologies. Health Inf Libr J. 2009;26:91‐
6. van Vooren NJE, Steenkamer BM, Baan CA, Drewes HW. Trans- 108. doi:10.1111/j.1471‐1842.2009.00848.x
forming towards sustainable health and wellbeing systems: eight guid- 28. Sadler E, Porat T, Marshall I, et al. Shaping innovations in long‐term
ing principles based on the experiences of nine Dutch population care for stroke survivors with multimorbidity through stakeholder
health management initiatives. Health Policy. 2020;124(1):37‐43. engagement. PLoS ONE. 2017;12(5):e0177102.
7. Steenkamer B, Drewes H, Putters K, van Oers H, Baan C. 29. Bravata DM, Myers LJ, Homoya B, et al. The protocol‐guided rapid
Reorganizing and integrating public health, health care, social care evaluation of veterans experiencing new transient neurological symp-
and wider public services: a theory‐based framework for collaborative toms (PREVENT) quality improvement program: rationale and
adaptive health networks to achieve the triple aim. J Health Serv Res methods. BMC Neurol. 2019;19(1):294.
Policy. 2020;25(3):187‐201. 30. Séroussi B, Soualmia LF, Holmes JH. Transforming data into knowl-
8. Foundation TH. Evidence scan: Complex Adaptive System 2010. edge: how to improve the efficiency of clinical care? Yearb Med
9. Best A, Greenhalgh T, Lewis S, Saul JE, Carroll S, Bitz J. Large‐system Inform. 2017;26(1):4-6.
transformation in health care: a realist review. Milbank Q. 2012;90(3): 31. Felkey BG, Fox BI. Are you practicing in a learning health system?
421‐456. Hosp Pharm. 2017;52(1):82‐83.
10. Khan S, Vandermorris A, Shepherd J, et al. Embracing uncertainty, 32. Mullins CD, Wingate LT, Edwards HA, Tofade T, Wutoh A.
managing complexity: applying complexity thinking principles to Transitioning from learning healthcare systems to learning health care
transformation efforts in healthcare systems. BMC Health Serv Res. communities. J Comp Eff Res. 2018;7(6):603-614.
2018;18(1):192. 33. Kamal AH, Kirkland KB, Meier DE, Morgan TS, Nelson EC, Pantilat
11. Steenkamer B, de Weger E, Drewes H, Putters K, Van Oers H, Baan SZ. A person‐centered, registry‐based learning health system for palli-
C. Implementing population health management: an international ative care: a path to coproducing better outcomes, experience, value,
comparative study. J Health Organ Manag. 2020;34:273‐294. and science. J Palliat Med. 2018;21(S2):S61‐S67.
12. Institute of Medicine (US). In: Grossmann C, Powers B, JM MG, eds. 34. Melder A, Robinson T, McLoughlin I, Iedema R, Teede H. An overview
Digital Infrastructure for the Learning Health System: The Foundation for of healthcare improvement: unpacking the complexity for clinicians
Continuous Improvement in Health and Health Care: Workshop Series and managers in a learning health system. Intern Med J. 2020;50(10):
Summary. Washington (DC): National Academies Press (US); 2011. 1174‐1184.
13. Etheredge LM. A rapid‐learning health system. Health Aff (Millwood). 35. Kuluski K, Guilcher SJT. Toward a person‐Centred learning health
2007;26(2):w107‐w118. system: understanding value from the perspectives of patients and
14. Institute of Medicine Roundtable on Evidence‐Based M. In: Olsen L, caregivers. HealthcarePapers. 2019;18(4):36‐46.
Aisner D, JM MG, eds. The Learning Healthcare System: Workshop 36. Moore JL, Shikako‐Thomas K, Backus D. Knowledge translation in
Summary. Washington (DC): National Academies Press (US); 2007. rehabilitation: a shared vision. Pediatr Phys Ther. 2017;29(Suppl 3):
15. About Learning Health Systems. Content last reviewed May 2019. S64‐S72. doi:10.1097/PEP.0000000000000381
Agency for Healthcare Research and Quality, Rockville, MD. https:// 37. Brannon E, Wang T, Lapedis J, et al. Towards a learning health system
www.ahrq.gov/learning-health-systems/about.html [Accessed 23 to reduce emergency department visits at a population level. AMIA
March 2022]. Annu Symp Proc AMIA Symp. 2018;2018:295‐304.
16. Pcori.org. Homepage | PCORI. [online]. 2022. Available at: https:// 38. Paton C, Karopka T. The role of free/libre and open source software
www.pcori.org/ [Accessed 23 March 2022]. in learning health systems. Yearb Med Inform. 2017;26(1):53‐58.
17. Learning Health Community. Learning Health Community. [online]. 39. Porat T, Marshall IJ, Sadler E, et al. Collaborative design of a decision
2022. Available at: https://www.learninghealth.org/ [Accessed 23 aid for stroke survivors with multimorbidity: a qualitative study in the
March 2022] UKengaging key stakeholders. BMJ Open. 2019;9(8):e030385.
18. Friedman CP, Rubin JC, Sullivan KJ. Toward an information infrastruc- 40. Friedman CP, Rubin J, Brown J, et al. Toward a science of learning
ture for Global Health improvement. Yearb Med Inform. 2017;26(1): systems: a research agenda for the high‐functioning learning health
16-23. system. J Am Med Inform Assoc. 2015;22(1):43‐50.
 23796146, 2023, 1, Downloaded from https://onlinelibrary.wiley.com/doi/10.1002/lrh2.10311 by Capes, Wiley Online Library on [16/03/2025]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License
10 of 10 de BRUIN ET AL.

41. Maddox TM, Albert NM, Borden WB, et al. The learning healthcare 61. Steenkamer BM, Drewes HW, Heijink R, Baan CA, Struijs JN. Defin-
system and cardiovascular care: a scientific statement from the Amer- ing population health management: a scoping review of the literature.
ican Heart Association. Circulation. 2017;135(14):e826‐e857. Popul Health Manag. 2017;20(1):74‐85.
42. Bhandari RP, Feinstein AB, Huestis SE, et al. Pediatric‐collaborative 62. Lessard L, Grudniewicz A, Sauré A, Szczotka A, King J, Fung‐Kee‐
health outcomes information registry (Peds‐CHOIR): a learning health Fung M. Type and use of digital technology in learning health sys-
system to guide pediatric pain research and treatment. Pain. 2016; tems: a scoping review protocol. BMJ Open. 2019;9(5):e026204.
157(9):2033‐2044. 63. Committee on the Learning Health Care System in A, Institute of M.
43. Manohar N, Perkins T, Laurion R, Oliver B. Patient‐reported out- In: Smith M, Saunders R, Stuckhardt L, McGinnis JM, eds. Best Care
comes in multiple sclerosis: an overview. J Clin Outcomes Manag. at Lower Cost: the Path to Continuously Learning Health Care in Amer-
2019;26(6):277‐288. ica. Washington (DC): National Academies Press (US) Copyright
44. Serena TE, Fife CE, Eckert KA, Yaakov RA, Carter MJ. A new 2013 by the National Academy of Sciences. All rights reserved;
approach to clinical research: integrating clinical care, quality 2013.
reporting, and research using a wound care network‐based learning 64. Hendrikx RJP, Drewes HW, Spreeuwenberg M, Ruwaard D, Struijs
healthcare system. Wound Repair Regen. 2017;25(3):354‐365. JN, Baan CA. Which triple aim related measures are being used to
45. Ryerson AB, Eheman C, Styles T, Rycroft R, Snyder C. Connecting the evaluate population management initiatives? Int Comp Anal Health
dots: linking the National Program of cancer registries and the needs Policy. 2016;120(5):471‐485.
of survivors and clinicians. Am J Prev Med. 2015;49(6):S528‐S535. 65. Greene JC, Caracelli VJ, Graham WF. Toward a conceptual frame-
46. Leung TI, Dumontier M. FAIR principles for clinical practice guidelines work for mixed‐method evaluation designs. Educ Eval Policy Anal.
in a learning health system. Stud Health Technol Inform. 2019;264: 1989;11(3):255‐274.
1690‐1691. 66. Bryman A. Integrating quantitative and qualitative research: how is it
47. Rubinstein SM, Warner JL. CancerLinQ: origins, implementation, and done? Qual Res. 2006;6(1):97‐113.
future directions. JCO Clin Cancer Inf. 2018;2018(2):1‐7. 67. Davis AR, Editor a conceptual framework for understanding path
48. Steels S, Van Staa T. Evaluation protocol of the implementation of a dependency and technology option evaluation when valuing IT
learning healthcare system in clinical practice: the connected health opportunities. 2015.
cities programme in the north of England. BMJ Open. 2019;9(6): 68. Loorbach D, Rotmans J. Managing transitions for sustainable develop-
e025484. ment. In: Olsthoorn X, Wieczorek AJ, eds. Understanding Industrial
49. Coury DL, Murray DS, Fedele A, Hess T, Kelly A, Kuhlthau KA. The Transformation: Views from Different Disciplines. Dordrecht: Springer
autism treatment network: bringing best practices to all children with Netherlands; 2006:187‐206.
autism. Pediatrics. 2020;145:13‐19. 69. Rotmans J, Loorbach D. Complexity and transition management. J Ind
50. Meldolesi E, Van Soest J, Damiani A, et al. Standardized data collec- Ecol. 2009;13(2):184‐196.
tion to build prediction models in oncology: a prototype for rectal 70. Pawson R, Greenhalgh T, Harvey G, Walshe K. Realist review‐‐a new
cancer. Future Oncol. 2016;12(1):119‐136. method of systematic review designed for complex policy interven-
51. Ethier JF, Curcin V, McGilchrist MM, et al. eSource for clinical trials: tions. J Health Serv Res Policy. 2005 Jul;10(Suppl 1):21‐34. doi:10.
implementation and evaluation of a standards‐based approach in a 1258/1355819054308530
real world trial. Int J Med Inform. 2017;106:17‐24. 71. de Weger EJ, van Vooren NJE, Drewes HW, Luijkx KG, Baan CA.
52. Bindman AB. Learning healthcare systems: a perspective from the US. Searching for new community engagement approaches in the Nether-
Public Health Res Pract. 2019;29(3):2931920. lands: A realist qualitative study. BMC Public Health. 2020;508. doi:
53. Price‐Haywood EG. Clinical comparative effectiveness research through 10.1186/s12889‐020‐08616‐6
the lens of healthcare decisionmakers. Ochsner J. 2015;15(2):154‐161. 72. van Vooren HWDNJE, de Weger E, Bos C, Vugts MAP, Baan CA,
54. Walkey AJ, Bor J, Cordella NJ. Novel tools for a learning health sys- Struijs JN. Update lerende evaluatie Juiste Zorg op de Juiste Plek. 2021.
tem: a combined difference‐in‐difference/regression discontinuity 73. Rippel. (n.d.). Rethink Health_Regional Stewardship. Rethink Health ‐
approach to evaluate effectiveness of a readmission reduction initia- a Ripple Effect. Retrieved December 12, 2021, from https://
tive. BMJ Qual Saf. 2020;29(2):161‐167. rethinkhealth.org/stewards-pathway/#1
55. Sanders JC, Showalter TN. How big data, comparative effectiveness 74. Romanelli RJ, Azar KMJ, Sudat S, Hung D, Frosch DL, Pressman AR.
research, and rapid‐learning health‐care systems can transform Learning Health System in Crisis: Lessons From the COVID‐19 Pan-
patient care in radiation oncology. Front Oncol. 2018;8(MAY):155. demic. Mayo Clin Proc Innov Qual Outcomes. 2021;5(1):171‐176. doi:
56. Gaveikaite V, Filos D, Schonenberg H, van der Heijden R, Maglaveras 10.1016/j.mayocpiqo.2020.10.004
N, Chouvarda I. Learning healthcare systems: scaling‐up integrated
care programs. Stud Health Technol Inform. 2018;247:825‐829.
57. Guise JM, Savitz LA, Friedman CP. Mind the gap: putting evidence SUPPORTING INF ORMATION
into practice in the era of learning health systems. J Gen Intern Med. Additional supporting information may be found in the online version
2018;33(12):2237‐2239.
of the article at the publisher's website.
58. Teare GF, Keller M, Hall D. Bringing together research and quality
improvement: the Saskatchewan approach. Healthc Q. 2018;21(SP):
56‐60.
59. McLachlan S, Potts HWW, Dube K, et al. The Heimdall framework for How to cite this article: de Bruin J, Bos C, Struijs JN,
supporting characterisation of learning health systems. J Innov Health Drewes HW-T, Baan CA. Conceptualizing learning health
Inform. 2018;25(2):77‐87. systems: A mapping review. Learn Health Sys. 2023;7(1):
60. Hirsch G, Trusheim M, Cobbs E, et al. Adaptive biomedical innovation:
e10311. doi:10.1002/lrh2.10311
evolving our global system to sustainably and safely bring new medi-
cines to patients in need. Clin Pharmacol Ther. 2016;100(6):685‐698.