Bullock et al.

Implementation Science (2021) 16:18

https://doi.org/10.1186/s13012-021-01082-7

RESEARCH Open Access

Understanding the implementation of

evidence-informed policies and practices
from a policy perspective: a critical
interpretive synthesis
Heather L. Bullock1* , John N. Lavis1,2, Michael G. Wilson1,2, Gillian Mulvale3 and Ashleigh Miatello1

Abstract
Background: The fields of implementation science and knowledge translation have evolved somewhat
independently from the field of policy implementation research, despite calls for better integration. As a result,
implementation theory and empirical work do not often reflect the implementation experience from a policy lens
nor benefit from the scholarship in all three fields. This means policymakers, researchers, and practitioners may find
it challenging to draw from theory that adequately reflects their implementation efforts.
Methods: We developed an integrated theoretical framework of the implementation process from a policy
perspective by combining findings from these fields using the critical interpretive synthesis method. We began with
the compass question: How is policy currently described in implementation theory and processes and what aspects
of policy are important for implementation success? We then searched 12 databases as well as gray literature and
supplemented these documents with other sources to fill conceptual gaps. Using a grounded and interpretive
approach to analysis, we built the framework constructs, drawing largely from the theoretical literature and then
tested and refined the framework using empirical literature.
Results: A total of 11,434 documents were retrieved and assessed for eligibility and 35 additional documents were
identified through other sources. Eighty-six unique documents were ultimately included in the analysis. Our
findings indicate that policy is described as (1) the context, (2) a focusing lens, (3) the innovation itself, (4) a lever of
influence, (5) an enabler/facilitator or barrier, or (6) an outcome. Policy actors were also identified as important
participants or leaders of implementation. Our analysis led to the development of a two-part conceptual
framework, including process and determinant components.
Conclusions: This framework begins to bridge the divide between disciplines and provides a new perspective
about implementation processes at the systems level. It offers researchers, policymakers, and implementers a new
way of thinking about implementation that better integrates policy considerations and can be used for planning or
evaluating implementation efforts.
Keywords: Implementation science, Public policy, Evidence-based health care, Systematic review, Critical
interpretive synthesis

* Correspondence: [email protected]
1
Department of Health Research Methods, Evidence and Impact, McMaster
University, 1280 Main Street West, Hamilton, Ontario L8S 4L6, Canada
Full list of author information is available at the end of the article

© The Author(s). 2021 Open Access This article is licensed under a Creative Commons Attribution 4.0 International License,
which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give
appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if
changes were made. The images or other third party material in this article are included in the article's Creative Commons
licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons
licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain
permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/.
The Creative Commons Public Domain Dedication waiver (http://creativecommons.org/publicdomain/zero/1.0/) applies to the
data made available in this article, unless otherwise stated in a credit line to the data.
Bullock et al. Implementation Science (2021) 16:18 Page 2 of 24

in the implementation process appears to be under-

Contributions to the literature
theorized. When policy is included in conceptual work,
 This study unpacks the implementation of evidence- it is often identified as a contextual variable [7, 8] rather
informed policies and practices through the systematic de- than being central to the implementation concept itself.
It is also often presented as a broad category of “policy”,
velopment of new theory drawing from three distinct fields
rather than as a variable that is specific and therefore
of scholarship: policy implementation, implementation sci-
measurable in empirical work. This lack of conceptual
ence, and knowledge translation, answering a call from im- clarity and empirical work about policy and other
plementation researchers for more integration. policy-related structural constructs has been noted by
 The conceptual framework views implementation from the several researchers. For example, a systematic review of
“outer context” and includes (1) a model describing the measures assessing the constructs affecting implementa-
process of implementation and (2) a framework that tion of health innovations makes specific reference to
identifies the policy-related determinants of implementation the “relatively few” measures available to assess struc-
tural constructs, which they define as “political norms,
success.
policies and relative resources/socio-economic status”
 This conceptual framework provides researchers,
[9]. As a result, the field of public policy appears to have
policymakers, and implementers with a new way of thinking on the one hand, a challenge of too many policy-related
about implementation and can be used for planning or implementation variables, and on the other hand, the
evaluating implementation efforts. fields of knowledge translation and implementation ap-
pear to have too few.
In recent years some researchers have recognized
Background these silos in scholarship and have called for more im-
Implementation has captured the attention of public plementation research that integrates public policy and
policy scholars for well over 50 years [1], yet remains implementation science and knowledge translation per-
relatively under-studied compared to other stages of spectives [10]. For example, Johansson concludes that
policy-making. The reasons for this are many and in- implementation problems could be better understood
clude challenges with isolating implementation from through the inclusion of research in public administra-
other parts of the policy process and a lack of agreement tion, with more focus on issues such as resource alloca-
about conceptual underpinnings [2]. This then leads to tion, priorities, ethical considerations, and the
challenges in identifying relevant explanatory variables distribution of power between actors and organizational
and analysts often must resort to a “long list of variables boundaries [11].
that are potentially useful” [2]. Even once decisions re- In addition to these challenges, much of the seminal
garding these challenges have been made, the complex, policy scholarship on implementation from both the
multi-level, and multi-faceted nature of implementation public policy and knowledge translation and implemen-
creates difficulties designing and conducting high-quality tation literatures come from the USA [12–15]. This has
empirical research that can offer useful generalizations resulted in a concentration of theoretical and empirical
to those interested in improving the process of imple- works that reflect the governance, financial and delivery
mentation and thus achieving better policy results [2]. arrangements that are particular to the USA [16, 17] and
Research on implementation has also independently that may not always readily apply in other contexts.
come into sharp focus through the related fields of These differences are particularly marked when it comes
knowledge translation and implementation science. Con- to the policy domain of health given the differences of
ceptual work on implementation from these fields has the US system compared to most others [18]. One not-
increased at a seemingly exponential rate to the point able exception to this is the European contributions to
where there is a great deal of focus on sorting and classi- the “second generation” of policy scholarship on imple-
fying the many frameworks, models, and theories and mentation, which adopted the perspective of those at the
providing guidance toward their use [3–6]. The empir- “coal face” of policy implementation [19].
ical literature is also rapidly increasing, with over 6200 In response to these challenges, the objective of our
systematic reviews on consumer-targeted, provider- study was to develop an integrated theoretical frame-
targeted, and organization-targeted implementation work of the implementation process from a policy per-
strategies in the health field alone (based on a search of spective by combining findings from the public policy,
www.healthsystemsevidence.org). implementation science, and knowledge translation
Despite the large number of models, theories, and fields. By integrating knowledge from these fields using a
frameworks being generated in the knowledge transla- critical interpretive synthesis approach, we specifically
tion and implementation science fields, the role of policy examine how policy considerations are described in
Bullock et al. Implementation Science (2021) 16:18 Page 3 of 24

implementation theories, frameworks, and processes policy considerations are most likely to be an important
from existing published and gray literature. Our goal factor. Implementation of a single evidence-based prac-
was to generate a theoretical framework to foster an im- tice (unless across a large-scale) or implementation in a
proved understanding of the policy contributions to im- single organization were excluded, as was research that
plementation that can be used in future studies to focused on behavior change at the individual level.
generate testable hypotheses about large-scale system
implementation efforts. Electronic search strategy
Using the compass question, and in consultation with a
Methods librarian, we constructed a table of Boolean-linked key
Study design words and then tested several search strategies (Table 1).
Given the broad goal of this study, the question of inter- The search was then conducted in October 2020 for the
est, and the scope of potentially applicable literature time period of January 2000–September 2020 using the
from discrete fields that could inform this work, we se- following 12 databases: ASSIA, CINAHL (via EBSCO),
lected a critical interpretive synthesis (CIS) approach. EMBASE (via Ovid), ERIC, Health Star (via Ovid),
Drawing from the techniques of meta-ethnography com- MEDLINE (via Ovid), PAIS Index, PolSci, PsychINFO,
bined with traditional systematic review processes, CIS Social Sciences Abstracts, Social Services Abstracts, and
employs an inductive and interpretive technique to crit- Web of Science. The dates for the policy databases
ically inspect the literature and develop a new (PolSci and Social Sciences Abstracts) were extended to
conceptualization of the phenomenon of interest. Unlike 1973 to ensure key conceptual articles would be re-
traditional systematic reviews that often focus on ques- trieved, such as the seminal work by Sabatier and Maz-
tions of effectiveness, CIS is helpful in generating mid- manian in 1980 [14]. A gray literature search was also
range theories with strong explanatory power [20, 21]. conducted using Health Systems Evidence (which in-
This is suitable for our goal of developing a conceptual dexes policy documents related to health system ar-
framework that better integrates findings from diverse rangements and implementation strategies, as well as
fields and affords the opportunity to critically inspect systematic reviews). Similar search strings were used
both individual studies and the literature from each field across all databases with minor adjustments to ensure
as a whole in terms of the nature of the assumptions searches were optimized. We prioritized sensitivity
underlying each field, and what has influenced their pro- (comprehensiveness) over specificity (precision) in our
posed solution [22]. The method begins with a compass search strategy.
question, which evolves throughout the course of the re-
view [22, 23]. Our compass question was as follows: Article selection
How is policy currently described in implementation We excluded articles based on their titles and abstracts
theory and processes and what aspects of policy are im- if they did not fit within the study scope or if they were
portant for implementation success? not conceptual or empirical works. We created add-
itional inclusion criteria that were based on the follow-
Review scope ing questions: (1) Is there a moderate (or greater)
Our review casts a very broad net in terms of implemen- chance that the article will shed light on the role of pol-
tation processes and theories. While our main focus is icy in an implementation process or on the outcomes of
on large-scale implementation efforts in health, behav- the process? (2) Does the article describe implementa-
ioral health, and human services areas that are not spe- tion efforts at a community or systems level? And (3)
cific to a particular condition, we also drew from other does the article identify actors at the government,
large-scale implementation theories and empirical work, organizational or practice level such as policy entrepre-
such as from the field of environmental science, that neurs who may be central to policy implementation ef-
may yield important insights toward a more integrated forts? Any articles that did not meet at least one of these
framework of implementation. We drew from two key criteria were excluded.
sources of literature: (1) existing frameworks, models, Complementary to the formal search and in keeping
and theories (public policy, implementation science and with the inductive strategies that are part of the CIS
knowledge translation) and (2) empirical studies that re- process, we also conducted hand searches of the refer-
port on specific implementation processes. ence lists of relevant publications and searched the au-
Given our interest in implementation processes from a thors’ personal files to identify further articles and
policy perspective, we limited our review to implementa- theoretically sampled additional articles to fill conceptual
tion frameworks, models, theories and empirical reports gaps as the analysis proceeded.
that describe implementation efforts at a community or After completing the searches, an Endnote database
systems level (e.g., city, province/state or country) where was created to store and manage results. Once
Bullock et al. Implementation Science (2021) 16:18 Page 4 of 24

Table 1 Search terms

Implementation Government Organizational level Practice level Evidence terms (with and without
terms level dashes)
implement* AND policy OR organizational polic* OR “clinical AND “evidence-based practice*”
guideline”
“knowledge strategy policy and procedures “practice “evidence-informed practice*”
translation” manual guideline”
“knowledge mobili*” “action plan” procedures manual scope of “evidence-informed policy”
practice
“evidence-based policy”

duplicates were removed, a random selection of two per- interpretive categories including “synthetic constructs”
cent of the articles was independently screened by two developed by the review team from the article and add-
reviewers (H.B. and A.M.) who were blinded to each itional notes on how the article contributed to the devel-
other’s ratings and used the same inclusion criteria. The opment of the conceptual model. Additionally, the data
reviewers classified each title and abstract as “include”, extraction form for the conceptual articles included a
“exclude”, or “uncertain”. Inter-rater agreement was de- classification of the type of framework according to Nil-
termined using the kappa statistic. This process was sen’s taxonomy of implementation models, theories, and
undertaken to improve the methodological rigor by en- frameworks [3].
hancing trustworthiness and stimulating reflexivity, not In the fourth and final stage, we initially focused on
to establish a quantitative assessment per se [24]. Any the conceptual literature and used it as a base from
discrepancies were then discussed between reviewers which to build our integrated conceptual model. We de-
until consensus was reached. Next, one reviewer veloped the synthetic constructs by reviewing the con-
assessed the remaining titles and abstracts. Articles clas- tent from each article that addressed the compass
sified as “include” or “uncertain” were kept for full text question and interpreting the underlying evidence using
review. a constant comparative method to ensure that the emer-
The full text of the remaining articles was then ging synthetic constructs were grounded in the data,
assessed by one reviewer. Articles were excluded at this similar to a grounded theory approach [25]. These syn-
stage if they did not provide detailed insight into the thetic constructs were then used to begin to build the
compass question. Articles were also sorted according to conceptual model and an accompanying graphic repre-
whether they were a conceptual contribution (i.e., pre- sentation of it. We then critiqued the emerging con-
sented a model, theory, framework or theoretical con- structs to identify gaps in the evidence and emerging
cept on implementation) or an empirical contribution constructs.
(i.e., used qualitative, quantitative, review or other re- Using this emerging model, we purposively sampled
search methods to present new findings, or an analysis additional conceptual literature to fill the gaps that we
of implementation). identified and to ensure we incorporated as many rele-
vant concepts as possible. We did this by consulting re-
Data analysis and synthesis views of existing models, theories, and frameworks [2–6]
Our data analysis proceeded in four stages. First, while to identify additional relevant concepts not captured by
screening and assessing the articles for inclusion, we our search strategy and by hand searching the reference
noted some general observations of how policy was in- sections of some seminal conceptual papers [7, 26].
corporated in the literature from each field of interest Once saturation of the conceptual literature was
(policy/public administration, implementation, and reached, we purposively sampled a subset of the empir-
knowledge translation). Second, we classified articles ac- ical literature and used this subset to “test” the model
cording to how policy was portrayed in implementation and add additional detail to the theoretical constructs
theory and processes. Third, we constructed a data ex- gleaned from empirical report. We used a similar data
traction template for conceptual and empirical studies extraction template with the exceptions of removing the
that included (1) descriptive categories (the author(s), descriptive category of model or theory name and the
the name of the model, theory or framework (if pro- interpretive classification using the Nilsen taxonomy [3],
vided), year of publication, author location, focus of the but adding the descriptive category of “methodology”. If
article, and whether a graphic or visual aid was in- our model did not capture findings from the empirical
cluded), (2) content from the article that addressed the studies, we revised it and re-tested. This process contin-
compass question regarding how policy is portrayed and ued until saturation was reached and additional empir-
what aspects are important for success, and (3) ical studies yielded no further insights into our model.
Bullock et al. Implementation Science (2021) 16:18 Page 5 of 24

The methods reported here are based on a protocol with an initial sample of 10% of the articles. We also
developed prior to initiating the study. The protocol and noted that nine of the articles related to a large, multi-
a note about the four ways that the reported methods year national implementation study [27–35]. Because
differed from the protocol are available upon request. this was the largest and most comprehensive account of
the role of policy in large-scale implementation efforts
Results identified through our search, we included these as a
Search results and article selection sub-group for data extraction. This approach led to data
Our database search retrieved 16,107 documents and 11, extraction for 34 empirical articles.
434 unique documents once duplicates were removed. In addition to these two approaches, we sampled articles
The review of titles and abstracts was completed inde- that filled in conceptual gaps as our model developed. This
pendently by two reviewers on a random sample (n = process resulted in the retrieval of an additional 26 concep-
171) of the documents. The Kappa score was 0.72 indi- tual articles and 3 empirical articles. In total, 86 unique
cating substantial agreement. Figure 1 provides a flow documents were included with two of these documents
diagram outlining the search strategy. Following these used in both the conceptual and empirical data extraction
criteria for the remaining titles and abstracts resulted in (Tables 2 and 3). While our process was inclusive of English
1208 documents included for full text review. The full language publications from any country, the majority of ar-
text review excluded an additional 940 documents leav- ticles were conducted by US researchers (n = 57), with the
ing 268 potentially relevant documents (excluded docu- others coming mainly from other Western countries (the
ments and the rationale for exclusion are available upon UK (n = 8), the Netherlands (n = 7), Australia (n = 5),
request). Of these, 23 conceptual documents, 243 empir- Canada (n = 2), Sweden (n = 2), Germany (n = 2), and
ical documents, and two documents that included both Europe, China, and OECD (n = 1)). Articles covered a
conceptual and empirical elements were included for the range of topics including health and health care, public
data extraction and analysis phase. We sampled and ex- health, mental health and addictions, children and youth,
tracted data on all of the conceptual articles. For the em- social care, justice, and climate change, among others. The
pirical articles, we chose a maximum variation sampling conceptual documents included all of the categories of the-
approach based on the subject matter and article topic ories, models and frameworks identified by Nilsen [3], with

Fig. 1 Literature search and study selection flow diagram

Bullock et al. Implementation Science (2021) 16:18 Page 6 of 24

Table 2 Overview of included conceptual literature

Author Year Author Topic area Focus Name of model/theory/framework Framework type
location [3]
Aarons et al. [12] 2011 USA Public services Implementation Conceptual model of global factors affecting Determinants
for children implementation in public service sectors framework
and families
Bauman et al. [36] 2006 Australia Physical activity Supra-National A Six-Step Framework for International Physical Process model
(majority) Activity Dissemination
Bowen et al. [37] 2010 USA HIV Organizational/ Rogers-Rütten Framework Determinants +
program evaluation
framework
Bowen and Zwi 2005 Australia Public health Knowledge Evidence-informed Policy and Practice Pathway Determinants
[38] translation framework
Bruns et al. [39] 2008 USA Children and System (state) No name per se but addresses dimensions of Determinants
youth state EBP implementation effort framework
Burris et al. [40] 2012 USA Public health System (law) No name per se but unified framework Determinants
integrating public health law and public health framework
systems and services
Campos and 2019 USA Health Policy, politics and No name but identifies six different directions for Process model
Reich [41] stakeholders different stakeholders
Chaudoir et al. 2013 USA Health System (measures A multi-level framework predicting implementa- Determinants
[9]a of determinants) tion outcomes framework
Cherney and 2011 Australia Evidence-based System Components of a Support Delivery System: ‘9Cs’ Determinants
Head [42] policy/practice framework
Chin and 2011 USA Health System A Conceptual Model for Specifically Addressing Implementation
Goldmann [43] Disparities 6 Key Levels of Influence theory
Damschroder 2009 USA Health Organizational Consolidated Framework for Advancing Determinants
et al. [7] Implementation Research (CFIR) framework
Domitrovich et al. 2008 USA Schools Implementation No name per se but identified as factors that can Determinants
[44] quality affect implementation quality: a multi-level framework
model
Evans and Davies 1999 UK and UK Policy transfer Policy Policy transfer Determinants
[45], and and framework
Dolowitz and 2000
March [46]
Feldstein and 2008 USA Healthcare Research to PRISM (Practical, Robust Implementation and Process model
Glasgow [47] practice Sustainability Model)
implementation
Fleuren et al. [48] 2014 Netherlands Healthcare Organizational/ No name per se but “Framework representing Determinants
program the innovation process and related categories of framework
determinants”
Godfrey [49] 2011 USA Mental health System Hypothesized factors that influence ACT Determinants
implementation framework
Green et al. [50] 2006 USA Physical activity Knowledge Push-Pull Capacity Model Process model
translation
Greenhalgh et al. 2004 UK Healthcare Organizational Diffusion of Innovations in Service Organizations Determinants
[26] framework
Greig et al. [51] 2012 UK Healthcare Implementation Activity Theory Classic theory
activity/practices
Harris et al. [52] 2012 USA Health Organizational Health Promotion Resource Center Dissemination Process model
promotion Framework
Harvey and Kitson 2016 Australia Health services Implementation Integrated Promoting Action on Research Determinants
[53] Implementation in Health Services (I-PARIHS) framework
Hendriks et al. 2013 Netherlands Public health Policy Behavior Change Ball Implementation
[54] (childhood theory
obesity)
Hill and Hupe 2003 UK and Policy Policy No model/theory or framework but discussed N/A
[55] Netherlands implementation ‘the multi-layer problem’
Bullock et al. Implementation Science (2021) 16:18 Page 7 of 24

Table 2 Overview of included conceptual literature (Continued)

Author Year Author Topic area Focus Name of model/theory/framework Framework type
location [3]
Hill and Hupe 2002 UK and Policy Policy N/A (book) Determinants
[56] Netherlands implementation framework
Hodges and 2013 USA Children and Policy (local) Multilevel framework for local policy Determinants
Ferreira [57] families development and implementation framework
Howlett [58] 2004 Canada Policy Policy N/A Other (most
implementation (instruments) closely resembles
Classic Theory)
Hupe [59] 2011 Netherlands Policy Explaining policy Thesis of incongruent implementation Determinants
implementation implementation framework
Hupe and Hill 2016 Netherlands Policy Policy N/A N/A
[60] and UK implementation
Jansen [61] 2010 Netherlands Public health Disconnections 3 niches of public health Process model +
between policy, determinants
practice and framework
research
Jilcott et al. [62] 2007 USA Public health Evaluating policy Applying the RE-AIM framework to assess the Evaluation
implementation public health impact of policy change framework
Johansson [11] 2010 Sweden Human services Policy N/A N/A
Leeman et al. [63] 2012 USA Obesity Policy Center TRT’s evaluation framework Evaluation
prevention framework
Lipsky [13] 1980 USA Social services Policy and Street-Level Bureaucracy Implementation
individual theory
Matland [64] 1995 USA Policy Policy Ambiguity-Conflict Model of Implementation Implementation
implementation theory
Mendel et al. [65] 2008 USA Mental health Organizational/ Framework of Dissemination in Health Services Process
community Intervention Research framework (2nd)
Michie [66] 2011 UK Behavior Individual The Behaviour Change Wheel Implementation
change (EBPs) theory +
determinants
framework
Moulton and 2017 USA Public service Policy The Strategic Action Field Framework Implementation
Sandfort [67] interventions theory
Pettigrew and 1992 UK Business Organizational/ Understanding strategic change: three essential Classic theory
Whip [68] firm dimensions (Warwick Framework)
Proctor et al. [69] 2011 USA Mental health Implementation Conceptual Model of Implementation Research Evaluation
outcomes framework
Raghavan et al. 2008 USA Mental health Policy A Policy Ecology of Implementation Determinants
[70] framework
Rutten et al. [71] 2003 Germany/ Health Policy Determinants of policy analysis Determinants
Europe promotion framework +
classic theory
Sabatier and 1980 USA Policy Policy Framework of Analysis for the Implementation of Determinants
Mazmanian [14] implementation Public Policy framework +
process model
Schoenwald et al. 2008 USA Mental health System Conceptual model for the MacArthur research Determinants
[72]a network on youth mental health child STEPs framework
initiative on evidence-based practice in clinics
and systems
Shortell [73] 2004 USA Health care System N/A; levels and associated assumptions about Implementation
change theory
Spoth et al. [74] 2013 USA Public health/ Population Translation Science to Population Impact (TSci Process model
prevention Impact) framework
Strehlenert et al. 2015 Sweden Health and Policy Conceptual Model for Evidence-Informed Policy Process model
[75] social care Formulation and Implementation
Bullock et al. Implementation Science (2021) 16:18 Page 8 of 24

Table 2 Overview of included conceptual literature (Continued)

Author Year Author Topic area Focus Name of model/theory/framework Framework type
location [3]
Thomann et al. 2017 Germany Policy Policy Extended Accountability Regimes Framework Implementation
[76] implementation theory
VanDeusen Lukas 2007 USA Heath care Organizational Framework for Organizational Transformation Classic theory
et al. [77]
Viennet and Pont 2017 International Education Policy Education Policy Implementation Framework Determinants
[78] framework and
implementation
theory
Wandersman 2016 USA Empowerment Innovation and Getting to Outcomes Process model
et al. [79] evaluation system interface
Wisdom et al. [80] 2014 USA Innovation System N/A Determinants
adoption framework
a
Also included in empirical literature

the Determinants Framework type being most common. 2. Focusing lens, signaling to systems what the
The empirical articles employed a wide array of methods priorities should be (i.e., referring to policy
that fall into the broad categories of qualitative, quantita- statements or attention by policymakers as a signal
tive, and mixed methods. about what is important to prioritize)
3. Innovation itself—the implementation object (i.e.,
General observations the “thing” being implemented is policy, such as
Through this process, we noted several general observa- new legislative policy on tobacco cessation)
tions regarding the characteristics of existing literature. 4. Lever of influence in the implementation process
In terms of the scholarly disciplines, most of the imple- (i.e., policy is identified as at least one of the factors
mentation science literature focused on the influencing the implementation process)
organizational or service provider levels with an em- 5. Enabler/facilitator or barrier to implementation
phasis on changing practice, often by introducing an (moderating variable) (i.e., while policy is identified
evidence-informed policy or practice (EIPP). The know- as being external to the implementation effort, it is
ledge translation literature included policymakers as a later found to be a barrier or facilitator to
target audience for research evidence, but the focus was implementation)
on the agenda setting or policy formulation stages of the 6. Outcome—the success of the implementation
policy cycle, as opposed to the implementation of an process is at least partially defined and measured by
EIPP. Here, the scholarship focused on strategies to in- a change in policy.
crease the use of evidence in policy decision-making. 7. Policy actors as important participants or leaders in
The public policy literature included theory describing implementation
“top-down”, “bottom-up”, and integrated approaches to
implementing an EIPP. The object of implementation in Theoretical framework
this area was the policy itself, rather than a specific pro- Our approach to developing the theoretical framework
gram or practice. There was often no clear articulation was twofold. The findings from our analysis suggested
of independent and dependent policy-related implemen- constructs that addressed both the process of implemen-
tation variables across any field, although many articles tation and the factors underpinning the success or fail-
did partially address this. ure of implementation. We therefore first developed a
process model [3] that describes the steps in the process
How policy is described in implementation theory and of translating EIPPs into effectively embedded system
processes changes. Next, we developed a determinants framework,
Our coding based on the compass question resulted in which specifies the types of policy determinants (inde-
the following characterization of how policy is described pendent variables) that affect implementation outcomes
in implementation theory and processes: (dependent variables). This two-part theoretical frame-
Policy is described as follows: work achieves two goals: (1) the process model is most
useful in describing the process of implementation from
1. Context in which implementation occurs (i.e., only a policy perspective and (2) the determinants framework
briefly citing a policy as the reason for is most useful for understanding and explaining policy-
implementation) related influences on implementation outcomes.
Bullock et al. Implementation Science (2021) 16:18 Page 9 of 24

Table 3 Overview of Included Empirical Literature

Author Year Author Topic area Level of focus Methodology
location
Bax et al. [81] 2010 Netherlands Road safety System Policy analysis
Beidas et al. 2016 USA Mental health Stakeholder Qualitative interviews
[82]
Brodowski et al. 2013 USA Child abuse prevention System Descriptive case study
[83]
Brownson et al. 2015 USA Public health System (state + local) Cross-sectional survey
[84]
Chaudoir et al. 2013 USA Health System (measures of Systematic review and criterion-validity assessment
[9]a determinants)
Cheadle et al. 2009 USA Physical activity promotion Community Evaluation—uncontrolled prospective design
[85]
Culotta et al. 2016 USA Climate change Regional Case study/policy analysis
[86]
Evans [87] 2013 UK Health Policy Mixed methods survey and in-depth interviews
Fleuren et al. 2014 Netherlands Prevention child health care/ Innovation Systematic review + Delphi study
[88] schools determinants
Gotham et al. 2008 USA Addictions System (state) Case study
[89]
Grace et al. [90] 2015 Australia Mental health Policy Policy analysis (document analysis)
Grundy and 2011 Canada Employment Policy Policy analysis
Smith [91]
Hargreaves 2013 USA Home visiting Systems Mixed methods
et al. [92]
Haug et al. [93] 2010 Europe Climate change Policy Literature review
Horner et al. 2014 USA School behavioral supports Multi-state EIPP Descriptive evaluation
[94]
Monroe-DeVita 2012 USA Mental health EIPP Literature Review
et al. [95]
Painter [96] 2010 USA Mental health Policy Single case study (document analysis + secondary
data analysis of single provider)
Perla et al. [97] 2013 US and UK Healthcare System Scan of literature using modified Delphi technique
Powell et al. 2012 USA Health and mental health EIPP Narrative review
[98]
Powell et al. 2014 USA Mental health EIPP Systematic review
[99]
Powell et al. 2015 USA Health and mental health EIPP Delphi process
[100]
Rhoades et al. 2012 USA Prevention (of crime and System (state level) Case description
[101] delinquency)
Rieckmann 2011 USA Addictions Policy Mixed methods (survey and key informant
[102] interviews)
Rieckmann 2015 USA Addictions Policy Mixed methods (survey and key informant
[103] interviews)
Rubin [104] 2016 USA Alignment of implementation Systems Intro to special issue (review of articles)
and public systems
Schoenwald 2008 USA Mental health System Structured survey (national sample)
et al [72]a
Yamey [105] 2012 USA Health in LMICs System Key informant interviews
Zhang and 2016 China Administrative policy transfer Policy Policy analysis
Marsh [106]
National Implementing Evidence-Based Practices Project articles (53 sites; 8 states), n = 9b
Bullock et al. Implementation Science (2021) 16:18 Page 10 of 24

Table 3 Overview of Included Empirical Literature (Continued)

Author Year Author Topic area Level of focus Methodology
location
Bond et al. 2009 USA Mental health System (multi-state) Mixed methods
[27]
Finnerty et al. 2009 USA Mental health Policy/system Instrument development and testing
[28]
Isett et al. 2007 USA Mental health System (multi-state) Qualitative (interviews)
[29]
Isett et al. 2008 USA Mental health System (multi-state) Case study (site visits + semi-structured interviews)
[30]
Jones et al. 2014 USA Mental health System (multi-state) Semi-structured interviews (state leaders)
[31]
Mancini et al. 2009 USA Mental health Innovation (2 states) Mixed methods (fidelity measurement + interviews,
[32] surveys, and site visits)
Peterson 2014 USA Mental health System (multi-state) Longitudinal analysis (descriptive)
et al. [33]
Rapp et al. 2005 USA Mental health System (multi-state) Descriptive
[34]
Rapp et al. 2010 USA Mental health System (state) Descriptive
[35]
a
Also included in conceptual
b
Nine articles described individually in subsequent rows

Part 1—process model ideas (values, evidence, etc.), interests (interest groups,
Figure 2 depicts this novel process model focusing on civil society, etc.), institutions (existing rules and institu-
one policy or system level. What follows is a narrative tional structures), and external factors (natural disaster,
description of the model. change in economic conditions) that affect the interpret-
Policy is shaped as it moves through systems. The ation of the policy package [107, 108]. This context af-
process through which policy travels from one level to fects how a problem is defined, whether it has the
another is known as policy transfer [36, 45, 46]. Each attention of decision makers and whether it is up for ac-
policy level is nested in a context that includes existing tive decision-making. This aligns with the “problem

Fig. 2 Process model of implementation from a policy perspective depicting the process at one policy level
Bullock et al. Implementation Science (2021) 16:18 Page 11 of 24

definition” and “agenda setting” stages of the policy cycle context surrounding each level (prevailing ideas, inter-
but is also described as part of the “exploration phase” ests, institutions, and external events) influences the ac-
in implementation science [12, 109]. Once a decision ceptability and ultimate success of implementation.
has been reached that something should be done to ad- Finally, the overall implementation approach may need
dress a given issue, attention shifts to the “policy devel- to shift over time in response to a constantly evolving
opment” stage of the policy cycle, which aligns with the context. For example, one study found it necessary to
“adoption decision and preparation” stage of implemen- change the implementation approach for a road safety
tation. It is during the policy development/adoption de- program in respond to changes in policy authority [81].
cision and preparation stage that the policy package gets
developed. Outcomes
The process of implementation is undertaken in order
Policy package to lead to outcomes, which can be separated and mea-
A policy package usually includes a mix of policy levers sured at different levels. Proctor et al. [69]e identifies
or instruments, including legal and regulatory instru- three separate outcomes: (1) implementation outcomes,
ments, economic instruments, voluntary instruments, or (2) service outcomes, and (3) recipient-related outcomes.
information and education instruments [58, 110]. The Along with these outcomes, our model includes policy-
policy package can also include some implementation and systems-level outcomes. These can be evaluated ac-
guidance such as a description of the overall implemen- cording to the policy outputs (i.e., enforcement variables,
tation strategy architecture, the major streams of activity, change of perspective of street-level staff), policy out-
timing of events and milestones, and roles and comes (i.e., unemployment levels, life-expectancy of
responsibilities. population) or indices of policy system change (i.e., ad-
The level of ambiguity of the policy package in terms ministrative re-organization, privatization) [56]. While
of its goals and means of attaining them, and the the measures and levels will vary depending on the size,
amount of conflict among actors with respect to the pol- scale, and focus of implementation, there is broad agree-
icy package are important to help characterize the im- ment that outcomes should be clearly defined a priori
plementation process and to explain its outcomes. and precisely measured. Evaluation findings regarding
According to Matland [64] the consideration of ambigu- outputs and outcomes can dynamically feed back into
ity and conflict leads to four types of implementation the implementation process as it unfolds. This creates
processes: (1) administrative implementation occurs feedback loops and the process becomes very dynamic
when there is low policy ambiguity and low policy con- and multi-directional.
flict (e.g., eradication of small pox), (2) political imple-
mentation occurs when there is low ambiguity but high Part 2—determinants framework
levels of conflict (e.g., public transit), (3) experimental Figure 3 presents an overview of our determinants
implementation occurs when there is high ambiguity but framework and the relationship among the determinants.
low conflict (e.g., Head Start programs for young chil- Our findings point to three sets of policy-related factors
dren), and (4) symbolic implementation occurs when that affect the process, outputs, and outcomes of imple-
both ambiguity and conflict are high and policies only mentation: (1) policy instruments and strategies, (2) de-
have a referential goal and differing perspectives on how terminants of implementation, and (3) policy actors,
to translate the abstract goal into instrumental actions including their characteristics, relationships, and context.
(e.g., establishing youth employment agencies). Collectively, these feed into the process of implementa-
tion that proceeds in an iterative fashion along the
Implementation process stages: exploration, installation/preparation, initial im-
The policy implementation process can start at any level, plementation, and full implementation/sustainment [12,
move in any direction and can “skip” levels. Power also 109]. The types of policy influences vary according to
shifts as implementation proceeds through levels [29, the stage of implementation [12]. The process of imple-
56]. The level with the most implementation activity mentation leads to a variety of outputs and outcomes as
tends to have the most power. This is true not only for described above.
different levels of governance, but as implementation
cascades across organizations, through “street level bu- Policy instruments and strategies
reaucrats” [13] and on to the end-user or target popula- Policy instruments and strategies are the most common
tion (the “recipient”) of the implementation process. set of factors mentioned in the literature and we found
Policy decisions at one level become context for other evidence for each of the instrument types described here,
levels. Implementation activities at one level can exert although with varying levels of detail. Policy instruments
either direct or indirect effects on another level. The can be applied to implementation in differing ways, often
Bullock et al. Implementation Science (2021) 16:18 Page 12 of 24

Fig. 3 Determinants framework of implementation from a policy perspective

with two or three levers used concurrently to implement policy package cannot be evaluated based on its intrinsic
a single initiative or strategy [90]. In order to classify characteristics alone [56]. Instead, it is important to
these strategies in a meaningful way, we drew on and examine whether the policy selected is an appropriate
adapted elements of a mutually exclusive and collectively “fit” with the problem [91], well-justified [78], and
exhaustive framework that identifies key features of aligned with existing context [12, 88].
health and social systems [107] and honed in on strat- II—Policy formulation process. This is the shape given
egies that are particularly important for implementation to a policy by the initial formation processes [45]. It in-
(Table 4). These include strategies focused on the gov- cludes who in government is responsible for formulating
ernance arrangements, financial arrangements, service the policy, their legitimacy and the extent to which there
delivery arrangements, and implementation-related sup- is opportunity to provide feedback, how much feedback
ports in systems. We then divided these strategies ac- is given, and the responsiveness in terms of adjustments
cording to the intended “target” of implementation. made [45].
Common targets of implementation from a policy per- III—Vertical public administration and thickness of
spective include the whole system, organizations, the hierarchy. Vertical public administration is the term
workforce or service providers, consumers, and the used to identify the layers in the policy transfer process.
innovation itself (the EIPP to be implemented). We wish It refers to separate governments exercising their au-
to note, however, that because policy-related variables thority with relative autonomy [45]. Policies generated
have not necessarily been treated with the same specifi- outside of a socio-political level may be more or less ac-
city as other types of implementation variables, the most ceptable to that level. Within a given layer, a particular
common strategies do not reflect the full array of strat- policy area may require the mobilization of any number
egies that could be employed. of institutions, departments, or agencies, and these agen-
cies must act in a coordinated, interdependent fashion,
Determinants termed “thickness of the hierarchy” [55].
Our framework identifies eight categories of determi- IV—Networks/inter-organizational relationships. The
nants (see “Determinants” box and elsewhere in Fig. 3). existence and nature of the relationships between paral-
Each of these categories represents a suite of factors that lel organizations who must collaborative in order to
are hypothesized to independently affect implementation achieve effective implementation and who do not have a
outcomes. These determinants are described briefly hierarchical relationship [45].
below and in more detail in Table 5. V—Implementing agency responses. The factors affect-
I—Characteristics of the evidence-informed policy or ing the responses of implementing agencies can be di-
practice (EIPP). The success or failure of a particular vided into issues related to the overall characteristics of
Bullock et al. Implementation Science (2021) 16:18 Page 13 of 24

Table 4 Policy-related strategies and examples of those strategies for implementation according to type of target
Target Strategy Examples References
System Policy authority (governance • Centralization/decentralization of policy authority (e.g., [7, 12, 27–31, 38, 39, 45,
arrangement) creating a regional infrastructure with some policy authority 80, 100]
to oversee implementation)
• Accountability of the state sector’s role in implementation
(e.g., develop system-wide performance indicators or targets,
monitor performance and fidelity, evaluate, report results pub-
licly, consider enforcement strategies)
• Leadership for implementation (through the appointment of
state sector leaders, dedicated resources, garnering support
for innovation and its implementation)
• Stewardship of the non-state sector’s role in implementation
(e.g., constructing formal opportunities for non-state sector in
oversight of implementation, contracting with non-state sec-
tor for implementation-related activities, fostering networks
and linkages across different types of organizations who are
engaged in implementation)
Funding system infrastructure (financial • Dedicate resources for system infrastructure to support [9, 27, 28, 31, 32, 35, 39,
arrangement) implementation (e.g., intermediaries, technical assistance 49, 74, 82, 83, 94, 97,
centers, backbone organizations, facilitators) 100, 101]
• Create funding sources that align with time needed for
effective implementation and scaling
Re-designing system to meet needs • Consider impacts of implementation on availability of care/ [7, 14, 26, 34, 37, 50, 53,
(delivery arrangement and service and plan for scaling-up across the geographical area or 54, 73, 89, 96, 100]
implementation-related supports) population
• Assess possible impacts on other services (e.g., wait times) in
response to implementing innovation
With what supports service is provided • Create or change system-wide record systems or information [29, 94, 97, 100]
(delivery arrangement) and communication technologies to support implementation
Organization Organizational authority (governance • Management approaches in support of optimal [7, 34, 72, 89, 97, 100]
arrangement) implementation, including: developing data collection
systems, developing and monitoring performance indicators,
quality improvement plans, use of scorecards, or public
reporting
• Develop and deploy appropriate organizational leadership for
implementation oversight and engagement
• Include innovation as part of accreditation processes
• Engage in networks/multi-institutional arrangements in
support of implementation
Funding organizations (financial • Provide service grants or contract with organizations to [12, 28, 29, 34, 35, 70, 72,
arrangement) support implementation or to offset additional administrative 94, 96, 100, 103, 104,
costs of implementing an innovation (e.g., training, data 111]
infrastructure changes, workforce stability impacts)
• Prospective payments to cover lag-time costs when beginning
to implement an innovation
• Targeted payments or penalties based on organizational
performance related to innovation (e.g., changing
reimbursement rate structure so that providers of high fidelity
receive modestly higher per unit rate)
• Targeted payments or penalties based on client outcomes
• Shift organizational funding models to support
implementation (e.g., from fee-for-service to no-risk managed
care arrangements)
Where service is provided (delivery • Adjust sites of service delivery in response to an innovation [90, 100]
arrangement) • Consider how the physical structure, facilities, and equipment
can support innovation during implementation and ensure
appropriate supply (supply chain management)
• Adjustments to the organizational scale in response to an
innovation (e.g., number of beds, units of service)
With what supports service is provided • Change organizational record systems or other information [97, 100]
(delivery arrangement) and communication technologies to support implementation
Organization-targeted implementation • Develop educational materials, hosting educational meetings, [7, 9, 28, 29, 35, 52, 53,
supports (Implementation-related training, or outreach visits tailored to organizations 65, 82, 86, 100]
supports) • Develop and disseminate program or organizational service
standards
Bullock et al. Implementation Science (2021) 16:18 Page 14 of 24

Table 4 Policy-related strategies and examples of those strategies for implementation according to type of target (Continued)
Target Strategy Examples References
• Provision of technical assistance and other forms of
implementation support
• Support development and maintenance of inter-
organizational collaboratives, communities of practice, and
other forms of inter-organizational communication/learning
• Consider non-monetary awards, incentives, and disincentives
for organizations (e.g., exemplary program award)
Workforce/ Professional authority (governance • Create or alter training and licensure requirements [27, 28, 34, 70, 72, 89, 95,
service arrangement) • Change scope of practice to reflect innovation 100]
provider • Alter where providers can practice geographically and in what
systems (public vs private)
• Continuing competence (e.g., provide training and continuing
education unit credits for innovation or disallow certain
courses for credit)
• Professional liability (e.g., change liability laws)
• Alter university curricula to include knowledge of innovation
Remunerating providers (financial • Reimbursement for program participation, extra efforts in [7, 27, 32, 34, 47, 52, 65,
arrangement) applying the innovation, or lost time due to training 69, 70, 73, 80, 82, 88, 90,
• Increase reimbursement rate 100, 104]
• Changing the way providers are reimbursed to encourage
implementation (e.g., from fee-for-service to capitation)
• Loan forgiveness
• Targeted payments or penalties for performance
• Targeted payments or penalties based on outcomes
• Review and align fiscal and billing policies and incentives for
providers
• Make billing easier for providers
By whom service is provided (delivery • Assess and improve workplace conditions for providers to [7, 45, 49, 54, 57, 66, 79,
arrangement) foster implementation 89, 100]
• Extend the role of a particular provider within their existing
scope of practice
• Shift tasks between types of providers
• Optimize the performance of the workforce in their current
roles by creating, disseminating, and monitoring guidelines or
standards of care for service providers
Workforce-targeted implementation • Develop educational materials, hosting educational meetings, [28–30, 34, 39, 65, 70, 83,
supports (implementation-related training, or outreach visits 89, 100]
supports) • Engage local opinion leaders
• Reminders and prompts
• Audit and feedback
• Coaching
• Develop either tailored or multi-faceted interventions to sup-
port implementation
• Consider non-monetary awards, incentives, and disincentives
for workforce
Consumer Consumer and stakeholder involvement • Consumer protection (laws, complaints management) [70, 72, 89, 100]
(governance arrangement) • Consumer, family, and stakeholder engagement in
implementation and monitoring
Incentivizing consumers (financial • Alter consumer/patient fees [29, 90, 100]
arrangement) • Consider disincentives that may exist for consumers to be
successful (e.g., some employment programs)
• Subsidies for private health insurance
Consumer-targeted implementation • Information or education provision [45, 54, 66, 70]
supports (implementation-related • Behavior change support
supports) • Skills and competencies development
• Communication and decision-making facilitation
Innovation Commercial authority (governance • Adjust licensure and registration requirements to support [54, 66]
arrangement) implementation
• Consider pricing and purchasing
• Establish voluntary agreements on advertising
Purchasing products and services • Changes to the scope and nature of insurance plans: [32, 49, 70, 72, 79, 84, 89,
(financial arrangement) extending or ending insurance coverage 95, 100]
• Adjust list of covered/reimbursed services and products
Bullock et al. Implementation Science (2021) 16:18 Page 15 of 24

Table 4 Policy-related strategies and examples of those strategies for implementation according to type of target (Continued)
Target Strategy Examples References
• Change restrictions or caps on coverage/reimbursement for
innovation and related supports
• Change mechanisms for billing
• Prior approval requirements

the agencies and the behavior of front-line or street-level implementation literature. These include two types of
staff [13, 56]. special interests: (1) implementing agencies, organiza-
VI—Attributes and responses from those affected by tions or programs that are responsible for implementing
EIPP. Attributes include the diversity of target group be- the EIPP (e.g., hospitals, schools), and 2 street-level bu-
havior and the target group as a percentage of the popu- reaucrats who, due to the relatively high degree of dis-
lation [14]. Responses include thing like impacts on cretion in their jobs, and therefore discretion over the
workforce stability [12]. dispensation of public benefits or sanctions to citizens,
VII—Timing/sequencing. As implementation is a can be critical to realizing any large-scale implementa-
process that unfolds over time, it does not always align tion efforts. There are also three expert sub-types that
with the cycles to which it is subject and the time con- are particularly visible during implementation: (1) field
straints inherent therein [86, 87]. Additionally, the exter- or practice leaders who can be influential in supporting
nal context in which implementation occurs is ever practice change among professionals, (2) innovation de-
changing and “quintessentially unstable”, and success velopers/disseminators who have developed the EIPP to
hinges on the ability to perceive those changes and take be implemented and who may contribute or adapt tools
the necessary actions to adjust along the way [68]. In and other types of support to encourage successful im-
Fig. 3, timing/sequencing is placed outside of the deter- plementation, and (3) intermediaries/technical assistance
minants box to reflect its importance across all of the providers who are organizations, programs, or individ-
other elements. uals that work “in between” policymakers, funders, and
VIII—External environment or policy context. Much of front-line implementers, to facilitate effective implemen-
the literature identified factors outside the policy area of tation drawing on expertise in implementation.
focus that may influence implementation (Fig. 3, outside There are also three categories of actor-related vari-
the hatched line). Many authors referred to this gener- ables that are important: (1) actor characteristics, (2)
ally as the “political and social climate”, as unmodifiable actor relationships, and (3) the context in which the ac-
or macro “context”, or as “socio-economic conditions” tors are embedded (Fig. 4). First, the characteristics of
[9, 14, 38, 40, 52, 70, 75, 80]. We organized this deter- the policy actors (either individual- or organizational-
minant using (1) the 3I+E framework [113] and (2) a level) such as their knowledge of the implementation
taxonomy of health and social system arrangements context, their legitimacy, power and control, and their
[114]. leadership in the context of the implementation effort
In general, these categories of determinants should be are often cited as being critical to the success in large-
viewed as interactive and not completely discrete [56] scale implementation initiatives. Second, the relation-
and the inter-relationship among the determinants is ships policy actors have with other actors, such as the
key [45]. level of shared values and beliefs or the coordination
and alignment of actors and their activities, can be pre-
Policy actors dictive of successful implementation. Finally, the context
Our analysis revealed a wide range of policy actors who of the actors, such as the sustainment of political will
are important for implementation. In an attempt to cre- and commitment and the stability of the actors them-
ate a category of variables that is analytically useful selves can predict the long-term success of
across contexts, we first divided the types of policy ac- implementation.
tors into the broad categories of political actors, bureau-
cratic actors, special interests, and experts [115]. To Discussion
provide more specificity, we further divided these into a Our study represents one of the first comprehensive at-
non-exhaustive list of actor sub-types that were fre- tempts to answer the call of scholars to integrate the
quently mentioned in the literature and included exam- fields of implementation science, knowledge translation,
ples of the types of roles they tend to assume in and policy implementation in an effort to build a more
implementation (Table 6). While many of the sub-types comprehensive and accurate understanding of imple-
are commonly identified in other phases of the policy mentation. By integrating conceptual and empirical
cycle, some receive particular attention in the works from all three fields, the resultant two-part
Bullock et al. Implementation Science (2021) 16:18 Page 16 of 24

Table 5 Determinants of implementation from a policy perspective and the factors that characterize the determinants
Determinant Description Factors that characterize determinant
I. Characteristics of the 1. Not possible to predict the success or failure of a 1. Relative advantage [26, 37]
EIPP particular policy package based on its intrinsic characteristics 2. Compatibility [26, 37]
alone [56] 3. Complexity of goals and ease of implementation [26, 37,
2. Need to examine questions such as whether the policy 67]
selected: 4. Obligations [26, 37]
(a) Is an appropriate fit with the problem [78, 91] 5. Resources [26, 37]
(b) Aligned with existing context [12, 88] 6. Existing relationship with state and provider [103]
7. Level of ambiguity of the EIPP [64]
8. Level of conflict among stakeholders [64]
9. Interaction of policy characteristics with other
determinants [56]
II. Policy formulation 1. Shape given to a policy by the initial formation processes 1. Government actors responsible for formulating policy [56]
process has an impact on its implementation [56] 2. Perceived legitimacy of government actors [56]
2. Depending on the implementation approach, the 3. Extent to which there is opportunity to provide feedback
government may distribute responsibility for some or almost [56]
all of the policy formulation process to other stakeholders 4. Responsiveness of policymakers to feedback [56]
[12] 5. Level of involvement of non-governmental actors [12, 78,
3. Level of involvement of service organizations, street-level 97]
bureaucrats and recipients may influence the confidence in, 6. Adequacy of planning for implementation (consideration
and support of, the policy decision and improve the chances of resources for implementation) [97]
for successful implementation [97] 7. Constraints experienced during formulation [12, 26]
III. Vertical public 1. Vertical Public Administration 1. Number of socio-political levels [56]
administration and (a) Term used to identify the layers in the policy transfer 2. Acceptability of policy generated outside of a particular
thickness of hierarchy process socio-political level [114]
(b) Refers to separate governments exercising their authority 3. Appropriateness of socio-political level [86]
with relative autonomy [56] 4. Thickness of each socio-political level (number and com-
(c) Policies generated outside of a socio-political level may plexity of institutions, departments, or agencies and their co-
be more or less acceptable to that level ordination and interdependence) [55]
2. Thickness of Hierarchy
(a) Number and complexity of institutions, departments, or
agencies at a particular socio-political level
(b) The thicker the hierarchy, the more managerial
competence, professionalism, and governance skills are
required by public servants in order to support effective
implementation [59]
IV. Networks/inter- 1. Reflects the existence and nature of the relationships 1. Degree of coordination among:
organizational between parallel organizations who must collaborative in (a) Systems [80]
relationships order to achieve effective implementation and who do not (b) Organizations [86]
have a hierarchical relationship [56] (c) Donors /other funders [84]
2. Better connections among stakeholders also increases the (d) Leaders [12]
opportunity for rapid diffusion and informal spread of 2. Formality (formal or informal) [72]
innovation, facilitating implementation 3. Network type (e.g., policy or inter-organizational) [53]
4. Coherence and strength of connections [112]
V. Implementing 1. Factors affecting the responses of implementing agencies 1. Overall characteristics of the agencies:
agency responses can be divided into: (a) Level of organizational control [56, 76]
(a) Issues related to the overall characteristics of the (b) Rate of staff turnover [88]
agencies (c) Organizational decision-making processes [88]
(b) Behavior of front-line or street-level staff [56] (d) Extent of policy and behavior-related change required
2. Overall “health” of organizations and how front-line/street- [14, 67]
level bureaucrats use their discretion and power impact im- (e) Attitudes of the agencies [14, 47]
plementation success (f) Resources of the agencies (e.g., minimum “investment
threshold” in implementation infrastructure [97] or cost-
absorptive capacity of agency to absorb additional costs as-
sociated with implementation [12] or certainty of funding
[49])
(g) Impetus for chang e[77] (e.g., external mandates may
increase an agency’s predisposition (i.e. motivation), but not
its capacity to adopt an innovation; mandates may divert
activity away from other innovations or locally generated
priorities [26])
(h) Perception of implementation approach (e.g. if approach
is punitive, mandatory, or “top down”) [82]
2. Behavior of front-line or street-level staff
(a) Level of discretion and level of relative autonomy from
organizational authority affect the amount of interpretation
of EIPP [13, 76]
Bullock et al. Implementation Science (2021) 16:18 Page 17 of 24

Table 5 Determinants of implementation from a policy perspective and the factors that characterize the determinants (Continued)
Determinant Description Factors that characterize determinant
(b) Competing accountabilities (e.g., state, market,
professional, societal) [76]
(c) Power distribution between actors at the front-line,
agency, and political levels [30]
(d) Personal characteristics including their knowledge, skills,
and perceived support from colleagues [88]
VI. Attributes and 1. The characteristics of the people affected by the EIPP, 1. Diversity of target group behavior [14]
responses from those their response to it, and the impact of the responses 2. Target group as percentage of the population [14]
affected by EIPP 2. Most evident when those affected are powerful, such as 3. Impacts on stability of the workforce and responses to
in regulatory policy when those regulated are large instability [12]
organizations [56]
VII. Timing/sequencing 1. Implementation processes at scale require adequate time, 1. Balance of predictability and adaptiveness to changing
which does not always align with the cycles they are subject circumstances [68, 93]
to and some authors have identified the lack of time or 2. The simultaneous address of system levers (including
short-term opportunism as barriers to effective implementa- policy changes, measurement systems, and regulatory
tion [e53, 54] mechanisms) [97]
2. The sequencing of activities and alignment of 3. Timing and pace of cycles, such as political, policy, and
implementation with other cycles is also important funding cycles [104]
4. Specific aspects of time that impact implementation:
(a) The phased structure of the implementation process
[104]
(b) When and how the implementation efforts are initiated
[104]
(c) Timeframes for funding and leadership support [104]
(d) The need to demonstrate the impacts early
(e) Return on investment of time and money [104]
VIII. External 1. Factors outside of the policy area of focus may influence 1. 3I+E framework
environment or policy implementation (a) Ideas (e.g., the interplay between beliefs and values of
context 2. Can be referred to generally as the “political and social policymakers and research evidence in a general way [38])
climate”, as unmodifiable or macro “context” or as “socio- (b) Interests (e.g., the political culture and the depth of social
economic conditions” [9, 14, 38, 40, 52, 70, 75, 80] cleavages [58])
3. While most included articles did not address these (c) Institutions (e.g., relevant policies from other areas that
determinants in depth, an overall examination of extracted “may represent potentially powerful contextual effects” [74,
data suggested two theoretical frameworks would be useful 78])
for classifying and understanding these determinants: (d) External factors (e.g., technology and technological
(a) 3I+E framework that identifies the institutions, interests, changes [14, 40]) economic forces operating in the overall
ideas, and external events that help explain what influences society [60], and environmental (in)stability [53, 67]
policy choices [113] 2. Taxonomy of health and social system arrangements [114].
(b) Taxonomy of health and social system arrangements (a) Governance arrangements that are not specific to the
classified according to the governance, financial, and EIPP being implemented but are still relevant to
delivery arrangements [114] understanding implementation outcomes (e.g., centralization
4. These broader context and system arrangements may be and power distribution of government [30, 72] or the form
critically important in explaining implementation outcomes of governance structures (omnibus/discrete) [72])
and these frameworks provide some logic and organization (b) Financial arrangements (e.g., private/public contractual
to potential variables relations, reimbursement rates and mechanisms [72], and
existing resource distribution [30])
(c) Delivery arrangements—referred to more generally in the
health-focused articles as “health(care) system and services
context” [38, 75] or “medical delivery system” [40]

theoretical framework provides additional clarity regard- differences between the fields that made it clear to us
ing the process of implementation viewed from a policy the extent of the lack of integration up to this point. Per-
perspective and identifies a number of policy-related de- haps not surprisingly, the area of public health seemed
terminants that can be tested empirically in the future. to be the most fertile ground for integration. This is
A key strength of our study was the methodological likely due to their focus on population-level concerns re-
approach we took to theory building. quiring system-wide implementation of EIPPs and a di-
First was the comprehensiveness of the search strategy, verse implementation ecosystem. The search strategy
which aimed to identify scholarship from more than one was part of the mixed methods approach of the CIS,
academic discipline and across wide range of topics be- which blended the rigor of a systematic search method-
yond health. The literature identified through the search ology that is explicit and replicable, with the inductive,
process revealed some interesting parallels and unique iterative, and purposive sampling techniques from
Bullock et al. Implementation Science (2021) 16:18 Page 18 of 24

Table 6 Types of policy actors identified in implementation

Actor Sub-type (non- Role description Role in implementation (non-exhaustive) References
exhaustive)
Political Politicians • Represent citizens (in a democracy) through • Most important level of elected officials is [34, 36, 41, 50,
actors popular consensus. the level where most policy authority rests 54, 61, 63, 65,
• Mandate to create laws and policies with for area of implementation 80, 82, 83, 94,
varying levels of authority • Develop and pass laws/policies (e.g., 95, 101]
• Can be supra-national, national/federal, state/ mandating a particular EIPP)
provincial, regional, local/municipal • Provide leadership and focus
• Source of funding for implementation
(organizations, providers, and/or consumer
levels)
Other elected • Similar to elected politicians but mandate is • If policy authority rests at their level, they [54]
officials limited to a particular policy domain and may develop and pass laws
(often) limited geographic jurisdiction (e.g., • Enforce laws/polices from other levels
sheriff, judge, school board trustee) • Interpret/adapt laws/policies for their
implementation
• Provide leadership and focus
• Source of funding for implementation
(organizations, providers, and/or consumer
levels)
Bureaucratic Executive • Departments or ministries who specialize in a • Support policy development, including [34, 61, 82, 84,
actors departments unique area of government responsibility implementation considerations 89]
(e.g., health) • Operationalize policy/law passed by
• Responsible for carrying out the “vision” of politicians
an elected official with leadership for that • May allocate tasks, responsibilities and define
portfolio (e.g., Minister of Health) competencies for implementation
• Not elected nor formally tied to a particular • Monitor policy implementation and track
political party outputs or outcomes
• Source of funding for implementation
(organizations, providers, and/or consumer
levels)
Boards and • Often operate semi-independently from gov- • Regulation and enforcement [27, 28, 34, 65,
agencies of ernment but are appointed by them • Interpretation of policies/laws 80, 82]
government • In most cases, they deal exclusively with one • Monitor policy implementation and track
particular sub-field of responsibility in which outputs or outcomes
the demand for public services is especially • May have the ability to apply penalties for
high (e.g., food inspection agency, state non-compliance
mental health authority) • May allocate tasks, responsibilities, and
define competencies for implementation
Self-governing • Bodies that regulate the conduct of their • Can set or change: scope of practice, [82]
regulatory own members (such as admissions and training, and licensure requirements, or
agencies discipline) and are empowered to do so by professional liability to support
the appropriate level of government and implementation
their members (e.g., medicine, law) • Develop/adopt guidelines or standards
• Regulators are drawn from the membership • Monitor quality and safety and continued
competence of professionals during
implementation
Judicial system • System of courts that provide a formal • Interpret/re-interpret laws through rulings [40]
mechanism for interpretation and application that may affect how they are implemented
of laws in the name of the state and resolves • Define/re-define public policies through
disputes legal challenges
Special Implementing • Organizations or programs that are • Interpretation of policies/laws [28, 34, 41, 43,
interests agencies responsible for implementing the laws or • Develop or adapt organizational policies and 50, 83, 87, 89,
policies developed (e.g., hospitals, schools, procedures to support implementation 91, 95, 101]
child welfare agencies, industry) • Training and support for workforce
• Location(s) where the majority of the • Provide or manage funds to support
implementation takes place implementation
• Monitor and evaluate implementation at
organizational level
Street-level • The schools, police and welfare departments, • Interpretation of policies/laws [13, 41, 43]
bureaucrats lower courts, legal services offices, and other • Often the parties responsible for changing
agencies whose workers interact with and their behaviors or practices during
have wide discretion over the dispensation implementation
of benefits or the allocation of public
sanctions [13]
Bullock et al. Implementation Science (2021) 16:18 Page 19 of 24

Table 6 Types of policy actors identified in implementation (Continued)

Actor Sub-type (non- Role description Role in implementation (non-exhaustive) References
exhaustive)
• Have (1) relatively high degree of discretion
and (2) relative autonomy from
organizational authority [13]
Insurers • Organizations or government bodies that • Have the ability to change the risk pool by [50, 65, 82]
manage risk by pooling risk across a group insuring more or fewer people (scope and
of individuals and providing coverage to nature of insurance plan)
them for needed services • Can adjust the list of covered/reimbursed
• Managed care organizations are a specific organizations, providers, services, and
type of insurer in health care that monitor products
and control the provision of care in an effort • Can change billing/reimbursement processes
to increase quality through regulating the to facilitate implementation
choices of providers and patients • Engagement and potential influence with
political and bureaucratic actors (feedback
loops) regarding implementation and scaling
Donors/ • Organizations that raise and allocate funds • Funding and/or in-kind implementation sup- [41, 105]
foundations based on a specific mandate that they ports (e.g., human resources)
identify • May have funded an innovation and now
have a vested interest in seeing it
implemented or scaled (bring leadership and
focus, implementation, and scaling expertise)
• Engagement and potential influence with
political and bureaucratic actors (feedback
loops) to support implementation and
scaling
Government • Organizations or businesses that are run • Interpretation of policies/laws
corporations independently from government but are still • Develop or adapt organizational policies and
ultimately accountable to them procedures to support implementation
Unions • Organized associations of workers created to • Negotiate contractual relationships with [34, 41]
promote and protect their interests in the implementing organizations on behalf of
workplace members (can influence the ease of
implementation)
• Engagement and potential influence with
political and bureaucratic actors (feedback
loops) regarding implementation and scaling
Experts Scientists/ • Individuals or research programs that • Share or contribute research expertise [35, 101]
researchers systematically gather, analyze, and use concerning the problem, the innovation, the
research and other evidence through implementation or the evaluation of the
processes, such as theorizing, synthesizing, implementation effort and any expected
and hypothesis testing, to gain and share outcomes
understanding and knowledge • Engagement and potential influence with
political and bureaucratic actors (feedback
loops) to support implementation and
scaling
Field or practice • Individuals who belong to a service • Share or contribute practice expertise [31, 38, 42]
leaders/champions providing community and are viewed as concerning the problem, the innovation, the
leaders or champions of an innovation and implementation or the evaluation of the
its implementation implementation effort and any expected
outcomes
• Act as champions for implementation to
members of their service providing
community and to other policy actors
• Engagement and potential influence with
political and bureaucratic actors (feedback
loops) to support implementation and
scaling
Patients or • Individuals who bring personal knowledge or • Share or contribute lived experience of the [28, 35, 43, 45,
persons with lived experience of a problem, condition, or problem, the innovation, the 57, 83, 100]
experience and service and who are the intended implementation or the evaluation of the
families/carers beneficiaries or ultimate “targets” of implementation effort and any expected
implementation outcomes
• Individuals who are family members or carers
to individuals who bring personal knowledge
or experience of a problem, condition, or
Bullock et al. Implementation Science (2021) 16:18 Page 20 of 24

Table 6 Types of policy actors identified in implementation (Continued)

Actor Sub-type (non- Role description Role in implementation (non-exhaustive) References
exhaustive)
service
Innovation/ • Organizations, programs or individuals who • Synthesize knowledge about innovation and [52, 101]
developers and have developed a process, program, or package it in ways that are “usable”
disseminators product to be implemented • Actively seek opportunities for innovation to
be adopted in policy and/or practice
• Provide expertise about the innovation
during implementation process
• Adapt innovation and materials as needed
during implementation process
Intermediaries and • Organizations, programs, or individuals that • Translate policy intention for implementing [9, 31, 42, 44,
technical work “in between” policymakers, funders, and agencies 45, 53, 83, 86,
assistance front-line implementers, to facilitate effective • Provide technical assistance to 89, 100, 101]
providers implementation drawing on expertise in implementing agencies (e.g., guidance on
implementation implementation process, coaching, decision
• Also known as purveyor organizations, support, monitoring, and evaluation)
backbone organizations, or central bodies • Provide mechanism for communication
charged with coordination between service delivery, policy systems, and
innovation developer (if applicable)
Other Media • Individuals and organizations that • Monitor implementation and communicate [34, 65]
communicate information through a variety facts or perceptions of the process and
of channels, including formal media outlets outcomes to the public
and social media outlets • Provides feedback loop for political actors,
bureaucratic actors, special interests, and
experts regarding implementation

Fig. 4 Characteristics, relationships, and the context of policy actors important for implementation
Bullock et al. Implementation Science (2021) 16:18 Page 21 of 24

qualitative review methods to build mid-range theory. framework elements and unpack them further for add-
The result is a theoretical framework that is clearly itional precision and clarity, drawing from multiple fields
linked to the literature, which should instill some confi- of scholarship. Our framework also offers some much-
dence in the academic community regarding its ground- needed policy variables that have been lacking in the im-
ing. Critical interpretive synthesis is a relatively new plementation science and knowledge translation fields,
approach but is growing in popularity for these reasons. which could be incorporated as part of a suite of vari-
Despite the merits of our approach, we did identify ables in implementation research.
some challenges. First, we believe the literature from
public policy may be underrepresented for several rea- Conclusions
sons: (1) search terms did not retrieve as much from Our study represents an early effort at integrating the
those fields (it could be that there are terms used more fields of public policy, implementation science, and
commonly in those fields that would have increased knowledge translation. We have learned that there is in-
yield), (2) the disciplinary approach to the scholarship in deed a great deal that each of the fields can learn from
public policy often means the articles were less explicit the other to advance our understanding of policy- and
about methods and this meant that more were excluded systems-level implementation efforts and hope that these
as not being “high yield”, and (3) more of that scholar- efforts are followed by more interdisciplinary research in
ship is captured through other media (e.g., books) and order to truly bridge this divide.
while some of these were included, our approach was
not as sensitive to retrieving these types of documents. Acknowledgements
Not applicable
We also did not include all of the empirical articles for
data extraction and we may have missed a key theme or Authors’ contributions
framework component. While we believe this is unlikely HLB was responsible for conceiving of the focus and design of the study
(with support from JNL) and for completing all data collection, analysis, and
because we continued to sample until saturation was interpretation. JNL also contributed to the analysis during ongoing iterative
reached, it is still possible something was missed. Finally, cycles of interpretation and synthesis that led to the development of the
there were few documents from low- and middle- final theoretical model. AM independently assessed a sub-sample of the doc-
uments for eligibility and worked with me to refine the inclusion criteria. HLB
income countries included in the final sample. Specific drafted the manuscript, and JNL, MGW, and GM provided comments and
efforts to include relevant documents from LMICs in fu- suggestions that were incorporated into revisions. All of authors approved
ture may enrich and refine the model. the final version of the manuscript.
As a result of this research, policymakers and practi-
Funding
tioners looking to use a conceptual model to guide their This study was partially supported through a doctoral scholarship from the
implementation activities have two additional options P.E. Trudeau Foundation.
that they can be confident draw from existing theory
Availability of data and materials
and empirical works. Large-scale implementation en- Not applicable
deavors or those that have started small and are looking
to scale-up should at least be mindful of the critical roles Ethics approval and consent to participate
Not applicable
of policy during the process and what policy-related fac-
tors may be important for success. Those planning im- Consent for publication
plementation activities can consider the elements Not applicable
presented in the framework as factors that may require
Competing interests
consideration and adjustment prior to implementing The authors declare they have no competing interests.
something new. Our work supports thinking beyond the
program or practice levels and unpacks policy consider- Author details
1
Department of Health Research Methods, Evidence and Impact, McMaster
ations that may have influence on, or affect the effective- University, 1280 Main Street West, Hamilton, Ontario L8S 4L6, Canada.
ness of, a program or practice. Furthermore, the 2
McMaster Health Forum, Hamilton, Canada. 3DeGroote School of Business,
inclusion of policy-related outputs and outcomes in our McMaster University, Burlington, Canada.
framework offers policymakers and practitioners the op- Received: 8 August 2020 Accepted: 7 January 2021
tion of additional indicators of success on which they
can measure and report.
Like any new theoretical contribution, our framework References
1. Pressman JL, Wildavsky AB. Implementation. How great expectations in
would benefit from further refinement and testing by Washington are dashed in Oakland. Berkeley, CA: University of California
the research community. Future research could adopt Press; 1973.
the process model to guide a policy-intensive implemen- 2. Hill M, Hupe P. Implementing public policy: governance in theory and in
practice. 3rd ed. London, UK: SAGE; 2014.
tation effort and test it to determine its usefulness in 3. Nilsen P. Making sense of implementation theories, models and frameworks.
such efforts. Researchers could also select particular Implement Sci. 2015;10:53.
Bullock et al. Implementation Science (2021) 16:18 Page 22 of 24

4. Moullin JC, Sabater-Hernandez D, Fernandez-Llimos F, Benrimoj SI. A 29. Isett KR, Burnam MA, Coleman-Beattie B, Hyde PS, Morrissey JP,
systematic review of implementation frameworks of innovations in Magnabosco J, et al. The state policy context of implementation issues for
healthcare and resulting generic implementation framework. Heal Res evidence-based practices in mental health. Psychiatr Serv. 2007;58(7):914–21.
Policy Syst. 2015;13:16. 30. Isett KR, Burnam MA, Coleman-Beattie B, Hyde PS, Morrissey JP,
5. Tabak RG, Khoong EC, Chambers DA, Brownson RC. Bridging research and Magnabosco JL, et al. The role of state mental health authorities in
practice: models for dissemination and implementation research. Am J Prev managing change for the implementation of evidence-based practices.
Med. 2012;43(3):337–50. Community Ment Heal J. 2008;44(3):195–211.
6. Mitchell SA, Fisher CA, Hastings CE, Silverman LB, Wallen GR. A thematic 31. Jones AM, Bond GR, Peterson AE, Drake RE, McHugo GJ, Williams JR. Role of
analysis of theoretical models for translational science in nursing: mapping state mental health leaders in supporting evidence-based practices over
the field. Nurs Outlook. 2010;58(6):287–300. time. J Behav Heal Serv Res. 2014;41(3):347–55.
7. Damschroder LJ, Aron DC, Keith RE, Kirsh SR, Alexander JA, Lowery JC. 32. Mancini AD, Moser LL, Whitley R, McHugo GJ, Bond GR, Finnerty MT, et al.
Fostering implementation of health services research findings into practice: Assertive community treatment: facilitators and barriers to implementation
a consolidated framework for advancing implementation science. in routine mental health settings. Psychiatr Serv. 2009;60(2):189–95.
Implement Sci. 2009;4:50. 33. Peterson AE, Bond GR, Drake RE, McHugo GJ, Jones AM, Williams JR.
8. Wandersman A, Chien VH, Katz J. Toward an evidence-based system for Predicting the long-term sustainability of evidence-based practices in
innovation support for implementing innovations with quality: tools, mental health care: an 8-year longitudinal analysis. J Behav Heal Serv Res.
training, technical assistance, and quality assurance/quality improvement. 2014;41(3):337–46.
Am J Community Psychol. 2012;50(3–4):445–59. 34. Rapp CA, Bond GR, Becker DR, Carpinello SE, Nikkel RE, Gintoli G. The role of state
9. Chaudoir SR, Dugan AG, Barr CHI. Measuring factors affecting mental health authorities in promoting improved client outcomes through
implementation of health innovations: a systematic review of structural, evidence-based practice. Community Ment Health J. 2005;41(3):347–63.
organizational, provider, patient, and innovation level measures. Implement 35. Rapp CA, Goscha RJ, Carlson LS. Evidence-based practice implementation in
Sci. 2013;8:22. Kansas. Community Ment Health J. 2010;46(5):461–5.
10. Nilsen P, Stahl C, Roback K, Cairney P. Never the twain shall meet?--a 36. Bauman AE, Nelson DE, Pratt M, Matsudo V, Schoeppe S. Dissemination of
comparison of implementation science and policy implementation research. physical activity evidence, programs, policies, and surveillance in the
Implement Sci. 2013;8:63. international public health arena. Am J Prev Med. 2006;31(4 Suppl):S57–65.
11. Johansson S. Implementing evidence-based practices and programmes in 37. Bowen SAK, Saunders RP, Richter DL, Hussey J, Elder K, Lindley L. Assessing
the human services: lessons from research in public administration. Eur J levels of adaptation during implementation of evidence-based
Soc Work. 2010;13(1):109–25. interventions: introducing the Rogers-Rutten framework. Heal Educ Behav.
12. Aarons GA, Hurlburt M, Horwitz SMC. Advancing a conceptual model of 2010;37(6):815–30.
evidence-based practice implementation in public service sectors. Adm 38. Bowen S, Zwi AB. Pathways to “evidence-informed” policy and practice: a
Policy Ment Heal. 2011 Jan;38(1):4–23. framework for action. PLoS Med. 2005;2(7):600–5.
13. Lipsky M. Street-level bureaucracy: dilemmas of the individual in public 39. Bruns EJ, Hoagwood KE, Rivard JC, Wotring J, Marsenich L, Carter B. State
services. New York: Russell Sage Foundation; 1980. implementation of evidence-based practice for youths, part II:
14. Sabatier P, Mazmanian D. The implementation of public policy: a framework Recommendations for research and policy. J Am Acad Child Adolesc
of analysis. Policy Stud J. 1980;8(4):538–60. Psychiatry. 2008;47(5):499–504.
15. Van Meter DS, Van Horn CE. The policy implementation process: a 40. Burris S, Mays GP, Scutchfield FD, Ibrahim JK. Moving from intersection to
conceptual framework. Adm Soc. 1975;6(4):445–88. integration: public health law research and public health systems and
16. O’Toole LJ Jr. Research on policy implementation: assessment and services research. Milbank Q. 2012;90:375–408.
prospects. J Public Adm Res Theor. 2000;10(2):263–88. 41. Campos PA, Reich MR. Political analysis for health policy implementation.
17. Saetren H. Implementing the third generation research paradigm in policy Heal Syst Reform. 2019;5(3):224–35.
implementation research: an empirical assessment. Public Policy Adm. 2014; 42. Cherney A, Head B. Supporting the knowledge-to-action process: a systems-
29(2):84–105. thinking approach. Evid Policy. 2011;7(4):471–88.
18. Papanicolas I, Woskie LR, Jha AK. Health care spending in the United States 43. Chin MH, Goldmann D. Meaningful disparities reduction through research
and other high-income countries. JAMA. 2018;319(10):1024–39. and translation programs. JAMA. 2011;305(4):404–5.
19. Hjern B. Implementation research—the link gone missing. J Public Policy. 44. Domitrovich CE, Bradshaw CP, Poduska JM, Hoagwood K, Buckley JA, Olin S,
1982;2(3):301–8. Romanelli LH, Leaf PJ, Greenberg MT, Ialongo NS. Maximizing the
20. Flemming K. Synthesis of quantitative and qualitative research: an example implementation quality of evidence-based preventive interventions in
using Critical Interpretive Synthesis. J Adv Nurs. 2010;66(1):201–17. schools: A conceptual framework. Adv Sch Ment Health Promot. 2008;1(3):
21. Wilson MG, Ellen ME, Lavis JN, Grimshaw JM, Moat KA, Shemer J, et al. 6–28.
Processes, contexts, and rationale for disinvestment: a protocol for a critical 45. Evans M, Davies J. Understanding policy transfer: a multi-level, multi-
interpretive synthesis. Syst Rev. 2014;3(1):143. disciplinary perspective. Public Adm. 1999;77(2):361–85.
22. Dixon-Woods M, Cavers D, Agarwal S, Annandale E, Arthur A, Harvey J, et al. 46. Dolowitz DP, Marsh D. Learning from abroad: the role of policy transfer in
Conducting a critical interpretive synthesis of the literature on access to contemporary policy-making. Governance. 2000;13(1):5–23.
healthcare by vulnerable groups. BMC Med Res Methodol. 2006;6(1):35. 47. Feldstein AC, Glasgow RE. A practical, robust implementation and
23. Entwistle V, Firnigl D, Ryan M, Francis J, Kinghorn P. Which experiences of sustainability model (PRISM) for integrating research findings into practice.
health care delivery matter to service users and why? A critical interpretive Jt Comm J Qual Patient Saf. 2008;34(4):228–43.
synthesis and conceptual map. J Health Serv Res Policy. 2012;17(2):70–8. 48. Fleuren M, Wiefferink K, Paulussen T. Determinants of innovation within
24. Patton MQ. Enhancing the quality and credibility of qualitative analysis. health care organizations: literature review and Delphi study. Int J Qual
Health Serv Res. 1999;34(5 Pt 2):1189. Health Care. 2004;16(2):107–23.
25. Charmaz K. Constructing grounded theory: A practical guide through 49. Godfrey JL. Re-implementing assertive community treatment: one agency’s
qualitative analysis 2nd ed. London: SAGE; 2014. challenge of meeting state standards. Diss Abstr Int Sect B Sci Eng. 2011;
26. Greenhalgh T, Robert G, Macfarlane F, Bate P, Kyriakidou O. Diffusion of 72(4-B):2434.
innovations in service organizations: systematic review and 50. Green LW, Orleans CT, Ottoson JM, Cameron R, Pierce JP, Bettinghaus EP.
recommendations. The Milbank Quarterly. 2004;82(4):581–629. Inferring strategies for disseminating physical activity policies, programs, and
27. Bond GR, Drake RE, McHugo GJ, Rapp CA, Whitley R. Strategies for practices from the successes of tobacco control. Am J Prev Med. 2006;31(4
improving fidelity in the National Evidence-Based Practices Project. Res Soc Suppl):S66–81.
Work Pract. 2009;19(5):569–81. 51. Greig G, Entwistle VA, Beech N. Addressing complex healthcare problems in
28. Finnerty MT, Rapp CA, Bond GR, Lynde DW, Ganju V, Goldman HH. The diverse settings: insights from activity theory. Soc Sci Med. 2012;74(3):305–12.
State Health Authority Yardstick (SHAY). Community Ment Heal J. 2009;45(3): 52. Harris JR, Cheadle A, Hannon PA, Forehand M, Lichiello P, Mahoney E,
228–36. Snyder S, Yarrow J. A framework for disseminating evidence-based health
Bullock et al. Implementation Science (2021) 16:18 Page 23 of 24

promotion practices. Prev Chronic Dis. 2012;9:E22. Epub 2011 Dec 15. PMID: 76. Thomann E, Hupe P, Sager F. Serving many masters: public accountability in
22172189; PMCID: PMC3277406. private policy implementation. Governance. 2018;31(2):299–319.
53. Harvey G, Kitson A. PARIHS revisited: from heuristic to integrated framework 77. Lukas CV, Holmes SK, Cohen AB, Restuccia J, Cramer IE, Shwartz M, et al.
for the successful implementation of knowledge into practice. Implement Transformational change in health care systems: an organizational model.
Sci. 2016;11(1):33. Health Care Manage Rev. 2007;32(4):309–20.
54. Hendriks A, Jansen M, Gubbel J, De Vries N, Paulussen T, Kremers S. 78. Viennet R, Pont B. Education policy implementation. OECD Education
Proposing a conceptual framework for integrated local public health policy, Working Papers Series. 2017.
applied to childhood obesity--the behavior change ball. Implement Sci. 79. Wandersman A, Alia K, Cook BS, Hsu LL, Ramaswamy R. Evidence-based
2013;8(1):46. interventions are necessary but not sufficient for achieving outcomes in
55. Hill M, Hupe P. The multi-layer problem in implementation research. Public each setting in a complex world: empowerment evaluation, getting to
Manag Rev. 2003;5(4):471–90. outcomes, and demonstrating accountability. Am J Eval. 2016;37(4):544–61.
56. Hill M, Hupe P. Implementing public policy: governance in theory and in 80. Wisdom JP, Chor KHB, Hoagwood KE, Horwitz SM. Innovation adoption: a
practice. 1st ed. London, UK: Sage; 2002. review of theories and constructs. Adm Policy Ment Heal. 2014;41(4):480–
57. Hodges S, Ferreira K. A multilevel framework for local policy development and 502.
implementation. In: Child and family advocacy: bridging the gaps between 81. Bax C, de Jong M, Koppenjan J. Implementing evidence-based policy in a
research, practice, and policy. New York, NY: Springer Science; 2013. p. 205–15. network setting: road safety policy in the Netherlands. Public Adm. 2010;
58. Howlett M. Beyond good and evil in policy implementation: Instrument 88(3):871–84.
mixes, implementation styles, and second generation theories of policy 82. Beidas RS, Stewart RE, Adams DR, Fernandez T, Lustbader S, Powell BJ, et al.
instrument choice. Policy Soc. 2004;23(2):1–7. A multi-level examination of stakeholder perspectives of implementation of
59. Hupe PL. The thesis of incongruent implementation: revisiting Pressman evidence-based practices in a large urban publicly-funded mental health
and Wildavsky. Public Policy Adm. 2011;26(1):63–80. system. Adm Policy Ment Heal. 2016;43(6):893–908.
60. Hupe PL, Hill MJ. ‘And the rest is implementation. ’Comparing approaches 83. Brodowski ML, Counts JM, Gillam RJ, Baker L, Collins VS, Winkle E, et al.
to what happens in policy processes beyond Great Expectations. Public Translating evidence-based policy to practice: a multilevel partnership using
Policy Adm. 2016;31(2):103–21. the interactive systems framework. Fam Soc J Contemp Soc Serv. 2013;94(3):
61. Jansen MW, van Oers HA, Kok G, de Vries NK. Public health: disconnections 141–9.
between policy, practice and research. Heal Res Policy Syst. 2010;8(1):37. 84. Brownson RC, Allen P, Jacob RR, Harris JK, Duggan K, Hipp PR, et al.
62. Jilcott S, Ammerman A, Sommers J, Glasgow RE. Applying the RE-AIM Understanding mis-implementation in public health practice. Am J Prev
framework to assess the public health impact of policy change. Ann Behav Med. 2015;48(5):543–51.
Med. 2007;34(2):105–14. 85. Cheadle R, LoGerfo JP, Schwartz S, Harris JRAE. Promoting sustainable
63. Leeman J, Sommers J, Vu M, Jernigan J, Payne G, Thompson D, et al. An community change in support of older adult physical activity: evaluation
evaluation framework for obesity prevention policy interventions. Prev findings from the Southeast Seattle Senior Physical Activity Network (SESP
Chronic Dis. 2012;9(6):E120. AN). J Urban Heal. 2009:1–9.
64. Matland RE. Synthesizing the implementation literature: the ambiguity-conflict 86. Culotta D, Wiek A, Forrest N. Selecting and coordinating local and regional
model of policy implementation. J Public Adm Res Theory. 1995;5(2):145–74. climate change interventions. Environ Plan C-Government Policy. 2016;34(7):
65. Mendel P, Meredith LS, Schoenbaum M, Sherbourne CD, Wells KB. 1241–66.
Interventions in organizational and community context: a framework for 87. Evans BA, Snooks H, Howson H, Davies M. How hard can it be to include
building evidence on dissemination and implementation in health services research evidence and evaluation in local health policy implementation?
research. Adm Policy Ment Heal. 2008;35(1–2):21–37. Results from a mixed methods study. Implement Sci. 2013;8.
66. Michie S, van Stralen MM, West R. The behaviour change wheel: a new 88. Fleuren MAH, Paulussen TGWM, Dommelen P, Van Buuren S. Towards a
method for characterising and designing behaviour change interventions. measurement instrument for determinants of innovations. Int J Qual Heal
Implement Sci. 2011;6(1):42. Care. 2014;26(5):501–10.
67. Moulton S, Sandfort JR. The strategic action field framework for policy 89. Gotham HJ, White MK, Bergethon HS, Feeney T, Cho DW, Keehn B. An
implementation research. Policy Stud J. 2017;45(1):144–69. implementation story: moving the GAIN from pilot project to statewide use.
68. Pettigrew A, Whipp R. Managing change and corporate performance. In J Psychoactive Drugs. 2008;40(1):97–107.
European industrial restructuring in the 1990s. London: Palgrave Macmillan; 90. Grace FC, Meurk CS, Head BW, Hall WD, Carstensen G, Harris MG, et al. An
1992. pp. 227–6265. analysis of policy levers used to implement mental health reform in
69. Proctor E, Silmere H, Raghavan R, Hovmand P, Aarons G, Bunger A, et al. Australia 1992-2012. BMC Health Serv Res. 2015;15(1):479.
Outcomes for implementation research: conceptual distinctions, 91. Grundy J, Smith M. Evidence and equity: Struggles over federal
measurement challenges, and research agenda. Adm Policy Ment Heal. employment equity policy in Canada, 1984-95. Can Public Adm. 2011;54(3):
2011;38(2):65–76. 335–57.
70. Raghavan R, Bright CL, Shadoin AL. Toward a policy ecology of 92. Hargreaves M, Cole R, Coffee-Borden B, Paulsell D, Boller K. Evaluating
implementation of evidence-based practices in public mental health infrastructure development in complex home visiting systems. Am J Eval.
settings. Implement Sci. 2008;3(1):26. 2013 Jun;34(2):147–69.
71. Rütten A, Lüschen G, von Lengerke T, Abel T, Kannas L, Diaz JAR, et al. 93. Haug C, Rayner T, Jordan A, Hildingsson R, Stripple J, Monni S, et al.
Determinants of health policy impact: a theoretical framework for policy Navigating the dilemmas of climate policy in Europe: evidence from policy
analysis. Soz Präventivmed Soc Prev Med. 2003;48(5):293–300. evaluation studies. Clim Change. 2010;101(3–4):427–45.
72. Schoenwald SK, Chapman JE, Kelleher K, Hoagwood KE, Landsverk J, 94. Horner RH, Kincaid D, Sugai G, Lewis T, Eber L, Barrett S, et al. Scaling
Stevens J, et al. A survey of the infrastructure for children’s mental health up school-wide positive behavioral interventions and supports:
services: implications for the implementation of empirically supported experiences of seven states with documented success. J Posit Behav
treatments (ESTs). Adm Policy Ment Heal. 2008 Mar;35(1–2):84–97. Interv. 2014;16(4):197–208.
73. Shortell SM. Increasing value: a research agenda for addressing the 95. Monroe-DeVita M, Morse G, Bond GR. Program fidelity and beyond: multiple
managerial and organizational challenges facing health care delivery in the strategies and criteria for ensuring quality of assertive community
United States. Med Care Res Rev. 2004;61(3 Suppl):12S–30S. treatment. Psychiatr Serv. 2012.
74. Spoth LA, Greenberg M, Leaf P, Brown CH, Fagan A, Catalano RF, et al. 96. Painter K. Legislation of evidence-based treatments in public mental health:
Addressing core challenges for the next generation of type 2 translation analysis of benefits and costs. Soc Work Public Health. 2009;24(6):511–26.
research and systems: the Translation Science to Population Impact (TSci 97. Perla RJ, Bradbury E, Gunther-Murphy C. Large-scale improvement
Impact) framework. Prev Sci. 2013;14(4):319–51. initiatives in healthcare: a scan of the literature. J Healthc Qual. 2013;
75. Strehlenert H, Richter-Sundberg L, Nyström ME, Hasson H. Evidence- 35(1):30–40.
informed policy formulation and implementation: a comparative case study 98. Powell BJ, McMillen JC, Proctor EK, Carpenter CR, Griffey RT, Bunger AC,
of two national policies for improving health and social care in Sweden. et al. A compilation of strategies for implementing clinical innovations in
Implementation Sci. 2015;10(1):169. health and mental health. Med Care Res Rev. 2012;69(2):123–57.
Bullock et al. Implementation Science (2021) 16:18 Page 24 of 24

99. Powell BJ, Proctor EK, Glass JE. A systematic review of strategies for
implementing empirically supported mental health interventions. Res Soc
Work Pr. 2014;24(2):192–212.
100. Powell B, Waltz TJ, Chinman MJ, Damschroder LJ, Smith JL, Matthieu MM,
et al. A refined compilation of implementation strategies: results from the
Expert Recommendations for Implementing Change (ERIC) project.
Implement Sci. 2015;10(1):21.
101. Rhoades BL, Bumbarger BK, Moore JE. The role of a state-level prevention
support system in promoting high-quality implementation and
sustainability of evidence-based programs. Am J Community Psychol. 2012;
50(3–4):386–401.
102. Rieckmann AE. Cassidy, Elaine F.; McCarty, Dennis TR. K. Employing policy
and purchasing levers to increase the use of evidence-based practices in
community-based substance abuse treatment settings: reports from single
state authorities. Eval Program Plann. 2011;34(4):366–74.
103. Rieckmann T, Abraham A, Zwick J, Rasplica C, McCarty D. A longitudinal
study of state strategies and policies to accelerate evidence-based practices
in the context of systems transformation. Health Serv Res. 2015;50(4):1125–
45.
104. Rubin RMM, Hurford MOO, Hadley T, Matlin S, Weaver S, Evans AC.
Synchronizing watches: the challenge of aligning implementation science
and public systems. Adm Policy Ment Heal. 2016;43(6):1023–8.
105. Yamey G. What are the barriers to scaling up health interventions in low
and middle income countries? A qualitative study of academic leaders in
implementation science. Glob Heal. 2012;8:11.
106. Zhang Y, Marsh D. Learning by doing: the case of administrative policy
transfer in China. Policy Stud. 2016;37(1):35–52.
107. Lavis JN, Rottingen JA, Bosch-Capblanch X, Atun R, El-Jardali F, Gilson L,
et al. Guidance for evidence-informed policies about health systems: linking
guidance development to policy development. PLoS Med. 2012;9(3):
e1001186.
108. Shearer JC, Abelson J, Kouyate B, Lavis JN, Walt G. Why do policies change?
Institutions, interests, ideas and networks in three cases of policy reform.
Heal Policy Plan. 2016;31(9):1200–11.
109. Fixsen DL, Naoom SF, Blase KA, Friedman RM, Wallace F, Network TNIR.
Implementation Research: A Synthesis of the Literature. Tampa, FL:
University of South Florida, Louis de la Parte Florida Mental Health Institute;
2005.
110. Treasury Board of Canada Secretariat. Assessing, selecting, and
implementing instruments for government action. Ottawa: Governement of
Canada; 2007.
111. VanDeusen LC, Engle R, Holmes S, Parker V, Petzel R, Nealon Seibert M, et al.
Strengthening organizations to implement evidence-based clinical practices.
Health Care Manage Rev. 2010;35(3):235–45.
112. Pal LA. Beyond policy analysis: public issue management in turbulent times.
5th ed: Nelson Education; 2014.
113. Lavis JN. Studying health-care reforms. In: Lazar H, Lavis J, Forest P-G,
Church J, editors. Paradigm freeze: why it is so hard to reform health care in
Canada. Kingston: McGill-Queen’s University Press; 2013.
114. Lavis JN, Wilson MG, Moat KA, Hammill AC, Boyko JA, Grimshaw JM, et al.
Developing and refining the methods for a “one-stop shop” for research
evidence about health systems. Heal Res Policy Syst. 2015;13(1):10.
115. Dente B. Who Decides? Actors and Their Resources. In: Understanding
Policy Decisions. SpringerBriefs in Applied Sciences and Technology. Cham:
Springer; 2014. https://doi.org/10.1007/978-3319-02520-9_2

Publisher’s Note
Springer Nature remains neutral with regard to jurisdictional claims in
published maps and institutional affiliations.