Aarons et al.

Implementation Science 2014, 9:45

http://www.implementationscience.com/content/9/1/45
Implementation
Science

RESEARCH Open Access

The implementation leadership scale (ILS):

development of a brief measure of unit level
implementation leadership
Gregory A Aarons1,2*, Mark G Ehrhart3 and Lauren R Farahnak1,2,3

Abstract
Background: In healthcare and allied healthcare settings, leadership that supports effective implementation of
evidenced-based practices (EBPs) is a critical concern. However, there are no empirically validated measures to
assess implementation leadership. This paper describes the development, factor structure, and initial reliability and
convergent and discriminant validity of a very brief measure of implementation leadership: the Implementation
Leadership Scale (ILS).
Methods: Participants were 459 mental health clinicians working in 93 different outpatient mental health programs
in Southern California, USA. Initial item development was supported as part of a two United States National
Institutes of Health (NIH) studies focused on developing implementation leadership training and implementation
measure development. Clinician work group/team-level data were randomly assigned to be utilized for an
exploratory factor analysis (n = 229; k = 46 teams) or for a confirmatory factor analysis (n = 230; k = 47 teams). The
confirmatory factor analysis controlled for the multilevel, nested data structure. Reliability and validity analyses were
then conducted with the full sample.
Results: The exploratory factor analysis resulted in a 12-item scale with four subscales representing proactive leadership,
knowledgeable leadership, supportive leadership, and perseverant leadership. Confirmatory factor analysis supported an
a priori higher order factor structure with subscales contributing to a single higher order implementation leadership factor.
The scale demonstrated excellent internal consistency reliability as well as convergent and discriminant validity.
Conclusions: The ILS is a brief and efficient measure of unit level leadership for EBP implementation. The availability of
the ILS will allow researchers to assess strategic leadership for implementation in order to advance understanding of
leadership as a predictor of organizational context for implementation. The ILS also holds promise as a tool for leader and
organizational development to improve EBP implementation.

Introduction providers and increased staffing to support monitoring of

The adoption, implementation, and sustainment of evidenced- implementation-related activities [3]. Although there are
based practices (EBPs) are becoming increasingly import- calls for increased attention to organizational context in
ant for health and allied healthcare organizations and EBP dissemination and implementation [4,5], there are
providers, and widespread adoption of EBPs holds prom- gaps in examining how organizational context affects EBP
ise to improve quality of care and patient outcomes [1,2]. implementation. Most relevant for this research is the
Considerable resources are being allocated to increase need for development of measures to assess organizational
the implementation of EBPs in community care set- constructs likely to impact implementation process and
tings with support for activities such as training service outcomes. One organizational factor in need of greater at-
tention is that of leadership for EBP implementation [6].
* Correspondence: [email protected] Leaders can positively or negatively impact the capacity
1
Department of Psychiatry, University of California, San Diego, La Jolla, CA, to foster change and innovation [7-10] and therefore are
USA instrumental in facilitating a positive climate for innovation
2
Child and Adolescent Services Research Center, San Diego, CA, USA
Full list of author information is available at the end of the article and positive attitudes toward EBP during implementation

© 2014 Aarons et al.; licensee BioMed Central Ltd. This is an Open Access article distributed under the terms of the Creative
Commons Attribution License (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use, distribution, and
reproduction in any medium, provided the original work is properly credited.
Aarons et al. Implementation Science 2014, 9:45 Page 2 of 10
http://www.implementationscience.com/content/9/1/45

[6,11]. Although the role of leadership in EBP implementa- into such behaviors can be garnered from existing lit-
tion is often discussed, it is rarely empirically examined. erature demonstrating that strategically-focused lead-
The limited empirical research in this area supports the ership predicts the achievement of specific goals. For
presence of a relationship between general leadership abil- example, a recent meta-analysis confirmed the relative
ity and implementation of innovative practices [12], but fo- advantage of strategic leadership—compared to general
cuses less on identifying specific behaviors that leaders leadership—for specific organizational change initiatives
may enact to facilitate EBP implementation. To stimulate [24]. Recent organizational research in climate for cus-
and support additional empirical work in this area, there is tomer service [25] and climate for safety [26,27] has
a need for brief and efficient measures to assess specific ac- shown that strategically-focused leadership is a critical
tions leaders may engage in to influence the success of im- precursor to building a strategic climate, which subse-
plementation efforts in their organizations or programs. quently predicts strategic outcomes such as increased cus-
Both implementation and leadership theories emphasize tomer satisfaction or decreased accidents, respectively.
the importance of leadership in supporting implementa- Although more than 60 implementation strategies were
tion of innovative practices such as EBP. For example, identified in a recent review of the implementation litera-
implementation scholars have asserted the importance ture [28], few focused on leadership as an implementation
of leadership in terms of obtaining funding, dispersing re- factor and none focused mainly on leader development to
sources, and enforcing policies in support of implementa- support EBP implementation. Of those identified, extant
tion [13]. Research from the Collaboration for Leadership strategies involve the recruitment and training of leaders
in Applied Health Research and Care has addressed and involving leaders at different organizational levels
the importance of leaders serving as clinical opinion [29-31]. Hence, we argue that leadership focused on a spe-
leaders, managing implementation projects, fostering cific strategic imperative, such as adoption and use of EBP,
organizational learning climates, and obtaining senior can influence employee attitudes and behavior regarding
management support [14]. Other research suggests that the imperative. This is consistent with research demon-
managers are responsible for interpreting research evi- strating that leader and management support for imple-
dence, applying it to organizational contexts, and making mentation is a significant and strong predictor of positive
research-informed implementation decisions [15]. Wei- implementation climate [32]. Thus, there is a need to
ner’s organizational theory of innovation implementation identify those behaviors that leaders may enact to create a
suggests that leaders play a critical role in creating readi- strategic EBP implementation climate in their teams and
ness for change, ensuring innovation-values fit, and de- better facilitate the implementation and sustainment of
veloping plans, practices, structures, and strategies to EBP.
support implementation [16]. The goals of the present study were to develop a scale
There is also empirical evidence for the importance of that focused on strategic leadership for EBP implemen-
leadership in predicting the success of implementation ef- tation and to examine its factor structure, reliability, and
forts. For example, transformational leadership (i.e., the convergent and discriminant validity. We drew from
degree to which a leader can inspire and motivate others) strategic climate and leadership theory, implementation
has been shown to predict employees’ reported use research and theory, implementation climate literature,
of an innovative practice being implemented in their and feedback from subject matter experts to develop
organization [12,17]. Consistent with transactional lead- items for the implementation leadership scale (ILS) to
ership (e.g., providing contingent rewards) [18] perceived extend work on management support for implementa-
support from one’s supervisor has been associated with tion. In particular, we focused on leader behaviors re-
employees’ participation in implementation [19]. Much of lated to organizational culture and climate embedding
the empirical research on leadership and implementation mechanisms that promote strategic climates [33]. In line
has focused on identifying mechanisms through which with this literature, items were developed to assess the
leaders affect implementation. These include a positive degree to which a leader is proactive with regard to EBP
organizational climate [20], supportive team climate implementation, leader knowledge of EBP and implemen-
[21], and positive work attitudes [22]. Research has tation, leader support for EBP implementation, leader per-
also focused on the role of leaders in influencing em- severance in the EBP implementation process, and leader
ployee attitudes toward EBP [11] and commitment to attention to and role modeling effective EBP implementa-
organizational change [23]. tion. Through a process of exploratory factor analysis
Although general leadership is held to play an import- (EFA) followed by confirmatory factor analysis (CFA), we
ant role in implementation, research in this area has not expected to find empirical support for the conceptual
necessarily outlined specific behaviors that leaders may areas identified above. We also expected the final scale
enact in order to strategically influence followers to sup- and subscales to demonstrate high internal consistency re-
port the larger goal of successful implementation. Insight liability. In regard to convergent validity, we expected that
Aarons et al. Implementation Science 2014, 9:45 Page 3 of 10
http://www.implementationscience.com/content/9/1/45

the derived leadership scale would have moderate to high Procedure

correlations with other measures of leadership (i.e., trans- The study was approved by the appropriate Institutional
formational and transactional leadership). Finally, in Review Boards prior to clinician recruitment and informed
regard to discriminant validity, we expected to find low to consent was obtained prior to administering surveys. The
moderate correlations between the derived leadership research team first obtained permission from agency ex-
scale and a measure of general organizational climate. ecutive directors or their designees to recruit their clini-
cians for participation in the study. Clinicians were then
Method contacted either via email or in-person for recruitment to
Item generation the study. Data were collected using online surveys or in-
Item generation and domain identification proceeded in person paper-and-pencil surveys.
three phases. First, as part of a study focused on develop- For online surveys, each participant was e-mailed an
ing an intervention to improve leadership for evidence- invitation to participate including a unique username
based practice implementation [18], the investigative team and password as well as a link to the web survey. Partici-
developed items based on review of literature relating pants reviewed informed consent and after agreeing to
leader behaviors to implementation and organizational cli- participate were able to access the survey and proceed to
mate and culture change [32,33]. Second, items were the survey items. Once participants logged in to the online
reviewed for relevance and content by subject matter ex- survey, they were able to answer questions and could pause
perts, including a mental health program leader, an EBP and resume at any time. The online survey took approxi-
trainer and Community Development Team consultant mately 30 to 40 minutes to complete and incentive vouchers
from the California Institute for Mental Health, and four ($15 USD) were sent by email after survey completion.
mental health program managers. Third, potential items In-person data collection occurred for those teams in
were reviewed by the investigative team and program which in-person data collection was preferred or would
managers for face validity and content validity. Twenty- be more efficient. Paper surveys were administered during
nine items were developed that represented five potential meetings at each of the participating program locations.
content domains of implementation leadership: proactive In most cases, the research team reserved one hour for
EBP leadership, leader knowledge of EBP, leader support data collection during a regular clinical work group or
for EBP, perseverance in the face of EBP implementation team meeting. Research staff obtained informed consent,
challenges, and attention and role modeling related to handed out surveys to all eligible participants, checked the
EBP implementation. returned surveys for completeness, and then provided an
incentive voucher to each participant. For participants not
Participants present at in-person meetings, paper surveys were pro-
Participants were 459 mental health clinicians working in vided and were returned to the research team in pre-paid
93 different outpatient mental health programs in Southern envelopes.
California, USA. Of the 573 clinicians eligible to participate Teams were identified in close collaboration with agency
in this research, 459 participated (80.1% response rate). Par- administrators. It was of utmost importance that team
ticipant mean age was 36.5 years (SD = 10.7; Range = 21 members shared a single direct supervisor to properly ac-
to 66) and the majority of respondents were female (79%). count for dependence in the data for variables pertaining
The racial/ethnic distribution of the sample was 54% to leadership. It was also important that participants com-
Caucasian, 23.4% Hispanic, 6.7% African American, 5% pleted the survey questions pertaining to leadership based
Asian American, 0.5% American Indian, and 10% ‘other’. on the proper supervisor as identified by the agency
Participants had worked in the mental health services administrators. Participants completing the online sur-
field for a mean of 8.5 years (SD = 7.7; Range = 1 week vey selected their supervisor from a dropdown menu
to 43 years), in child and/or adolescent mental health of supervisors within their agency in the beginning of
services for a mean of 7.5 years (SD = 7.6; Range = 1 week the survey. The supervisor’s name was then automatically
to 43 years), and in their present agency for 3.4 years inserted into all questions regarding leadership in order
(SD = 4.3; Range = 1 week to 28.1 years). Highest level to ensure clarity in the target of all leadership ques-
of education consisted of 7% Ph.D./M.D. or equiva- tions. The research team verified the identified leader
lent, 68% master’s degree, 6.5% graduate work but no de- with organization charts.
gree, 12.2% bachelor’s degree, 3% some college but no For in-person data collection, participants were given
degree, and 0.7% no college. The primary discipline of the paper-and-pencil surveys with their supervisor’s name
sample was 47% marriage and family therapy, 26% social pre-printed on the front page of the survey and in sec-
work, 16% psychology, 3% child development, 2% human tions pertaining to general leadership and implementation
relations, 1% nursing, and 4.8% other (e.g., drug/alcohol leadership. Participants were instructed to answer all lead-
counseling, probation, psychiatry). ership questions about the supervisor whose name was
Aarons et al. Implementation Science 2014, 9:45 Page 4 of 10
http://www.implementationscience.com/content/9/1/45

printed on their survey. In cases where a participant noted Work group/team-level data was randomized within
that they reported to a different supervisor, this was clari- organization to be utilized for either the EFA (n = 229;
fied and the survey was adjusted if deemed appropriate. k = 46 teams) or CFA (n = 230; k = 47 teams). Exploratory
factor analysis was used to derive and evaluate the factor
Measures structure of the scale using IBM SPSS. Principal axis fac-
Implementation leadership scale (ILS) toring was selected for factor extraction because it allows
Item development for the ILS is described above. All 29 for consideration of both systematic and random error
ILS items were scored on a 0 (‘not at all’) to 4 (‘to a very [37] and Promax oblique rotation was utilized for factor
great extent’) scale. rotation as we assumed that derived factors would be cor-
related. Item inclusion or exclusion was based on an itera-
Multifactor leadership questionnaire (MLQ) tive process in which items with relatively low primary
The MLQ [34] is one of the most widely researched loadings (e.g., < 0.40) or high cross-loadings (e.g., > .30)
measures of leadership in organizations. The MLQ in- were removed [37]. The number of factors to be retained
cludes the assessment of transformational leadership, was determined based on parallel analysis, factor loadings,
which has been found in numerous studies to be associ- and interpretability of the factor structure as indicated in
ated with organizational performance and success (in- the rotated solution. Parallel analysis is among the better
cluding attitudes toward EBP), as well as transactional methods for determining the number of factors based on
leadership [11]. The MLQ has good psychometric prop- simulation studies [38]. Parallel analysis was based on esti-
erties including internal consistency reliability and con- mation of 1,000 random data matrices with values that
current and predictive validity. All items were scored on correspond to the 95th percentile of the distribution of
a 0 (‘not at all’) to 4 (‘frequently, if not always’) scale. random data eigenvalues [39,40]. The random values were
Transformational leadership was measured with four then compared with derived eigenvalues to determine
subscales: idealized influence (α = 0.87, 8 items), inspir- the number of factors. Confirmatory factor analysis was
ational motivation (α = 0.91, 4 items), intellectual stimula- conducted using Mplus [41] statistical software adjusting
tion (α = 0.90, 4 items), and individualized consideration for the nested data structure using maximum likelihood
(α = 0.90, 4 items). The MLQ also includes one subscale estimation with robust standard errors (MLR), which ap-
identified as best representing transactional leadership: propriately adjusts standard errors and chi-square values.
contingent reward (α = 0.87, 4 items). Missing data were handled through full information max-
imum likelihood (FIML) estimation. Model fit was assessed
Organizational climate using several empirically supported indices: the compara-
The Organizational Climate Measure (OCM) [35] con- tive fit index (CFI), the Tucker-Lewis index (TLI), the root
sists of 17 scales capturing the four domains of the com- mean square error of approximation (RMSEA), and the
peting values framework [36]: human relations, internal standardized root mean square residual (SRMR). CFI and
process, open systems, and rational goal. We utilized the TLI values greater than 0.90, RMSEA values less than
autonomy (α = 0.67, 5 items) scale from the human rela- 0.10, and SRMR values less than 0.08 indicate accept-
tions domain, the formalization scale (α = 0.77, 5 items) able model fit [41-44]. Type two error rates tend to be
from the internal process domain, and the efficiency low when multiple fit indices are used in studies where
(α = 0.80, 4 items) and performance feedback (α = 0.79, sample sizes are large and non-normality is limited, as in
5 items) scales of the rational goal domain [35] as mea- the present study [45].
sures for assessing discriminant validity of the ILS. All Reliability was assessed by examining Cronbach’s alpha
OCM items were scored on a 0 (‘definitely false) to 3 (‘def- internal consistency for each of the subscales and the
initely true’) scale. total scale. Item analyses were also conducted, including
an examination of inter-item correlations and alpha if
Statistical analyses item removed. Convergent and discriminant validity were
In order to determine whether the data represented a assessed by computing Pearson Product Moment Correla-
unit-level construct (in this case, clinical treatment work tions of ILS subscale and total scale scores with MLQ and
groups or teams), we examined intraclass correlations OCM subscale scores.
(ICCs) and the average correlation within group (awg) for
each item. Agreement indices are used to assess the ap- Results
propriate level of aggregation for nested data. Higher Examination of distributions for all scale items indicated
levels of agreement suggest that the higher level of ag- that data were generally normally distributed with no ex-
gregation is supported. This is relevant for the current treme skewness. Thus, we treated variables as continu-
study as clinicians were working within clinical work ous in our analyses. The presence of missing data was
groups or teams led by a single supervisor. minimal. For example, among the 459 respondents, only
Aarons et al. Implementation Science 2014, 9:45 Page 5 of 10
http://www.implementationscience.com/content/9/1/45

26 (6%) had any missing data. Of those with missing statistics have the advantage over rwg [48,49] of not being
data, 17 of the 26 (65%) had missing information on scale and sample size dependent, and not assuming a uni-
only one item, two had two or three items missing (8%), form distribution [48,49]. Values of awg greater than 0.60
and the remaining 7 (27%) had more than three items represent acceptable agreement and values of 0.80 and
missing. For the EFA, we used bivariate (rather than list- above represent strong agreement [48-50]. As shown in
wise) deletion in order to minimize the number of ex- Table 1, considering ICCs and awg, ILS items and scales
cluded cases and used FIML estimation to address missing should be considered as representing unit-level (i.e., clin-
values in the CFA. ical work group or team) constructs in this study.

Aggregation analyses Exploratory factor analysis

We first examined the amount of dependency among An iterative approach was taken to conducting the factor
observations within groups using intraclass correlations analyses and item reduction. In the first iteration and
(ICC, type 1) [46]. As shown in Table 1, the ICCs indi- consistent with our hypotheses, five factors were speci-
cated a moderate degree of dependency among service fied and all 29 items were included. The EFA results
provider responses within the same team. Nevertheless, showed that no items met the factor loading criteria for
the true variance tends to be underestimated whenever a proposed fifth factor (i.e., no loadings > 0.40). That,
ICCs take on non-zero values, an effect that is magnified coupled with the parallel analysis, suggested a four factor
with increasing average cluster size [47]. However, the solution. Thus, we conducted the next EFA specifying
average cluster size was relatively small in this study four factors. The results suggested the removal of 15
(mean = 6), mitigating this concern. items. Thirteen items were removed because of low pri-
We next examined the average agreement within clinical mary factor loadings and/or high cross loadings, and
work group for individual items and scales using awg(1) two items were removed because of overlapping content
and awg(J), respectively [48-50]. awg ranges from 1 to −1, with other items. Thus, 14 items were retained. The next
with awg(1) calculated as one minus the quotient of two EFA included 14 items and specified four factors. Based
times the observed variance divided by the maximum pos- on those results, two additional items were removed due
sible variance, and awg(J) is the sum of awg(1) values for to statistical (i.e., lower relative factor loadings) and con-
items divided by the number of items for a scale. These ceptual (i.e., item content less directly consistent with

Table 1 Implementation leadership scale, subscale and item statistics

EFA factor loadings
ILS items, subscales, and total Mean sd ICC awg ev v α 1 2 3 4
1. Proactive leadership 2.12 1.25 0.25 0.68 9.50 79.0% 0.95
Established clear standards for implementation of EBP 2.16 1.33 0.67 0.96 0.02 0.06 −0.08
Developed a plan to facilitate EBP implementation 2.12 1.29 0.70 0.95 0.00 −0.05 0.05
Removed obstacles to implementation of EBP 2.09 1.30 0.67 0.75 0.02 0.07 0.12
2. Knowledgeable leadership 2.56 1.18 0.26 0.72 0.76 6.3% 0.96
Knows what he/she is taking about when it comes to EBP 2.58 1.22 0.73 0.09 0.94 −0.03 −0.02
Is knowledgeable about EBP 2.59 1.20 0.71 −0.06 0.87 0.09 0.06
Is able to answer staff questions about EBP 2.50 1.25 0.71 0.30 0.85 0.06 0.04
3. Supportive leadership 2.63 1.15 0.22 0.69 0.49 4.1% 0.95
Supports employee efforts to use EBP 2.63 1.21 0.70 0.02 0.10 0.84 0.02
Supports employee efforts to learn more about EBP 2.67 1.18 0.72 −0.03 −0.01 0.83 0.16
Recognizes and appreciates employee efforts 2.59 1.25 0.67 0.17 0.17 0.69 −0.08
4. Perseverant leadership 2.36 1.25 0.29 0.69 0.37 3.1% 0.96
Perseveres through the ups and downs of implementing 2.37 1.29 0.69 0.05 0.07 0.05 0.81
Carries on through the challenges of implementing EBP 2.38 1.31 0.69 0.10 0.02 0.11 0.78
Reacts to critical issues regarding implementation of EBP 2.32 1.30 0.69 0.29 0.18 0.09 0.44
Implementation leadership scale total 2.42 1.12 0.29 0.70 0.98
Note: N = 459 for means and standard deviations and ICC; n = 229 for other EFA derived statistics; sd = Standard deviation; ICC = intraclass correlation; awg = average
within group correlation; ev = initial eigenvalue; v = variance accounted for before rotation; α = Cronbach’s alpha; bold font for means and sd indicate the overall scale
mean and sd; bold font for EFA factor loadings indicates the scale on which the items load.
Aarons et al. Implementation Science 2014, 9:45 Page 6 of 10
http://www.implementationscience.com/content/9/1/45

other items) criteria. The final EFA included 12 items 0.94, and all factor loadings were statistically significant
with three items loading on each of four factors. (p’s < 0.001).
Table 1 displays the factor means, item means, ICC,
awg, initial eigenvalues, variance accounted for by each Convergent validity
factor, internal consistency reliabilities, and rotated fac- Table 3 shows that, as predicted, the ILS scale scores
tor loadings. Internal consistencies were high, ranging had moderate to high correlations with MLQ subscales
from 0.95 to 0.98. Item analyses indicated that inter- representing transformational and transactional leader-
item correlations were high, (range = 0.83 to 0.92) and ship. Correlations ranged from 0.62 to 0.75 indicating
the alpha for the subscales would not be improved by re- convergent validity. The magnitude of the correlations
moving any items. As shown in Table 2, factor correla- suggests that leadership is being assessed by the ILS and
tions ranged from 0.73 to 0.80, suggesting a higher order that transformational leaders are likely to perform the
implementation leadership factor. The results of the behaviors necessary for effective EBP implementation,
CFA testing this higher order factor structure are pro- but not so high as to suggest that the MLQ and ILS
vided in the next section. Subscale labels were created scales are measuring identical constructs.
based on an examination of the items and factor load-
ings presented in Table 1. The first factor was labeled Discriminant validity
‘Proactive Leadership’ as it indicated the degree to which Table 3 shows the results of the discriminant validity
the leader anticipates and addresses implementation analyses. As predicted, the ILS scale scores had low cor-
challenges. Factor two addressed ‘Knowledgeable Leader- relations with OCM subscales representing aspects of gen-
ship’ or the degree to which a leader has a deep under- eral organizational climate. Correlations ranged from
standing of EBP and implementation issues. Factor three 0.050 to 0.406 indicating strong support for the discrimin-
was labeled ‘Supportive Leadership’ because it repre- ant validity of the ILS in contrast to general organizational
sented the leader’s support of clinicians’ adoption and climate.
use of EBP. Finally, factor four reflected ‘Perseverant
Leadership’ or the degree to which the leader is consist- Discussion
ent, unwavering, and responsive to EBP implementation The current study describes the development of the first
challenges and issues. measure of strategic leadership for evidence-based prac-
tice implementation, the ILS. We used an iterative
Confirmatory factor analysis process to develop items representing implementation
Confirmatory factor analysis was used with a sample in- leadership and then used quantitative data reduction
dependent of the EFA sample in order to evaluate the techniques to develop a brief measure that may be easily
factor structure identified in the EFA above. In addition, and efficiently used for research and applied purposes.
because we proposed a higher-order factor model in Such brief measures are needed to improve the effi-
which each subscale was considered an indicator of an ciency of services and implementation research [51].
overall implementation leadership latent construct, we Although we originally proposed five factors of imple-
evaluated the higher order model. We also controlled mentation leadership, quantitative analyses supported a
for the nested data structure (i.e., clinicians within clinical four-factor model. The identified factors correspond to
work groups or teams). The higher order factor model four of the original five subdomains originally conceived
demonstrated excellent fit as indicated by multiple fit indi- by the research team. The factors or subscales of the ILS
cators (n = 230; χ2(50) = 117.255, p < 0.001; CFI = 0.973, represent Proactive Leadership, Supportive Leadership,
TLI = 0.964; RMSEA = 0.076; SRMR = 0.034). Figure 1 dis- Knowledgeable Leadership, and Perseverant Leadership.
plays the standardized factor loadings for the higher-order The factor that was not supported in these analyses had
factor model. First-order factor loadings ranged from 0.90 to do with the events and practices that leaders pay de-
to 0.97, second-order factor loadings ranged from 0.90 to liberate attention to as well as the extent to which a
leader models effective EBP implementation. It may be
Table 2 Implementation leadership scale factor that these behaviors are more akin to a strategic climate
intercorrelations for EBP implementation and thus may have been less
Factor 1 2 3 4 relevant for the core focus on leadership in the ILS. In
1. Proactive leadership 1.0
addition, employees may not consciously recognize the
specific targets of their leaders’ intentions. Conversely,
2. Knowledgeable leadership 0.73 1.0
it may be that the items that were developed did not
3. Supportive leadership 0.75 0.79 1.0 sufficiently capture this aspect of leadership. Future
4. Perseverant leadership 0.79 0.77 0.80 1.0 studies should examine the degree to which leader at-
Note: N = 229; All correlations, p < 0.001. tention and role modeling can be captured through
Aarons et al. Implementation Science 2014, 9:45 Page 7 of 10
http://www.implementationscience.com/content/9/1/45

Figure 1 Second-order confirmatory factor analysis factor loadings for the implementation leadership scale. Note: n = 230; All factor
loadings are standardized and are statistically significant, p < 0.001; χ2(50) = 117.255, p < 0.001; CFI = 0.973, TLI = 0.964; RMSEA = 0.076; SRMR = 0.034.

the development of measures of organizational cli- phases (i.e., qualitative item generation, exploratory fac-
mate for EBP implementation. tor analysis, confirmatory factor analysis, reliability as-
The ILS demonstrated strong internal consistency reli- sessment, validity assessment) of this line of research.
ability, convergent validity, and discriminant validity. However, the item and scale development was based on
Given that the ILS is very brief (i.e., 12 items), adminis- extant literature as well as investigator and practitioner
tration and use in health services and implementation knowledge and experience with leadership development
studies can be very efficient with little respondent bur- and EBPs in community-based mental health service set-
den. It generally takes less than five minutes to complete tings. Further research is needed to determine the utility
scales of this length. The practicality of this brief scale is of the measure for research and practice in this and
consistent with calls for measures that can be utilized in other health and allied health care settings and contexts.
real-world settings where the efficiency of the research This study raises additional directions for future re-
process is paramount [52]. search. The factor analytic approach utilized here was
This is the first scale development study for imple- highly rigorous. Not only did we randomize respondent
mentation leadership and thus represents the first few data, but we randomly assigned data at the work group/

Table 3 Pearson product moment correlations of implementation leadership scale scores with multifactor leadership
questionnaire [convergent validity] and organizational climate measure [discriminant validity] scores
Implementation leadership scales
Proactive Knowledge Support Perseverant ILS total
MLQ
Transformational leadership
Intellectual stimulation 0.628** 0.698** 0.718** 0.699** 0.736**
Inspirational motivation 0.655** 0.683** 0.708** 0.705** 0.741**
Individualized consideration 0.618** 0.665** 0.705** 0.672** 0.715**
Idealized influence 0.658** 0.715** 0.721** 0.708** 0.753**
Transactional leadership
Contingent reward 0.631** 0.644** 0.684** 0.649** 0.702**
Climate
Autonomy 0.050 0.136* 0.121* 0.080 0.103*
Formalization 0.161** 0.147* 0.175** 0.156** 0.176**
Efficiency 0.227** 0.225** 0.281** 0.248** 0.268**
Feedback 0.327** 0.376** 0.406** 0.346** 0.392**
Note: N = 459; MLQ = Multifactor Leadership Questionnaire; Climate = General Organizational Climate Measure; *p < 0.05, **p < 0.01.
Aarons et al. Implementation Science 2014, 9:45 Page 8 of 10
http://www.implementationscience.com/content/9/1/45

team level to either the EFA or CFA analyses. Thus, Conclusions

there is no overlap in team membership across the two The current study builds on previous research by extend-
phases of this study. In addition, our examination of ing the general concept of leadership to a new construct:
scale reliability and convergent and discriminant validity strategic leadership for EBP implementation. This study
in this study confirmed expected relationships between suggests that effective leaders of EBP implementation
the ILS and other constructs. For example, the moderate should be proactive, knowledgeable, supportive, and perse-
to high correlations with other leadership scales affirms verant in the implementation process. The extent to which
that there is some overlap between implementation lead- these newly identified aspects of EBP leadership can impact
ership and effective general leadership (i.e., transform- individual factors (e.g., employee behaviors), organizational
ational and transactional leadership) but that unique factors (e.g., implementation climate), and implementation
aspects of leadership are also being captured. On the outcomes should be the subject of future studies [54].
other hand, we had only one other measure of leadership More immediately, strategies for improving leadership
in the study and future research should examine the de- knowledge, skills, abilities, and behaviors in order to pro-
gree to which other conceptual approaches and mea- mote strategic climates that will improve the efficiency of
sures of leadership are associated with the ILS and its EBP implementation should be developed and tested. In
subscales [53]. In addition, the low association with addition, the extent to which leadership influences fidelity
general organizational climate suggests that the ILS and adoption of EBPs should be examined to increase our
dimensions are distinct from common measures of understanding of the complex ways in which leader-
general organizational climate. Future research should ship may affect clinician behavior in healthcare organiza-
examine the association of ILS scales with other mea- tions. Pursuing such a research agenda has the potential
sures of organizational climate and strategic climate to improve the efficiency and effectiveness of imple-
for EBP implementation. mentation efforts and to improve the reach and public
The ILS may help to inform our understanding of the health impact of evidence-based treatments and practices.
influences and effects of leadership focused on EBP im-
plementation. The ILS could also be utilized as a meas- Additional files
ure to identify leaders or to identify areas to develop in
existing leaders. This is in keeping with an implemen- Additional file 1: Implementation Leadership Scale (ILS).
tation strategy recently developed and pilot tested by Additional file 2: Implementation Leadership Scale Scoring
the authors focused specifically on leadership and Instructions.
organizational change for implementation (LOCI) [18].
The LOCI implementation strategy utilizes data to sup- Competing interests
port leader development and cross-level congruence of GAA is an Associate Editor of Implementation Science; all decisions on this
paper were made by another editor. The authors declare that they have no
leadership and organizational strategies that support a other competing interests.
first-level leader in creating a positive EBP implementation
climate and implementation effectiveness [32,54]. Such Authors’ contributions
strategies address calls for leadership and organizational GAA and MGE were study principal investigators and contributed to the
theoretical background and conceptualization of the study, item
change strategies to facilitate EBP implementation and development, study design, writing, data analysis, and editing. LRF
sustainment [18]. contributed to the item development, study design, data collection, writing,
The ILS is a brief tool that may be used in imple- and editing. All authors read and approved the final manuscript.
mentation research to assess the extent to which
Acknowledgements
leaders support their staff in implementing EBP. The Preparation of this paper was supported by National Institute of Mental
ILS and scoring instructions can be found in Health grants R21MH082731 (PI: Aarons), R21MH098124 (PI: Ehrhart),
Additional files 1 and 2, or may be obtained from R01MH072961 (PI: Aarons), P30MH074678 (PI: Landsverk), R25MH080916 (PI:
Proctor), and by the Child and Adolescent Services Research Center (CASRC)
GAA. After establishing this baseline level of imple- and the Center for Organizational Research on Implementation and
mentation leadership, researchers and/or organizations Leadership (CORIL). The authors thank the community-based organizations,
may apply this knowledge to the identification of clinicians, and supervisors that made this study possible.
The Implementation Leadership Scale (ILS) and scoring instructions are
areas for implementation leadership development. Be- available from GAA at no cost or may be obtained as additional files
cause the scale is comprised of behaviorally focused accompanying this article.
items, results of the assessment may be used to guide
Author details
leadership development. Thus, not only does this 1
Department of Psychiatry, University of California, San Diego, La Jolla, CA,
measure allow for assessment of implementation USA. 2Child and Adolescent Services Research Center, San Diego, CA, USA.
3
leadership, it has the potential to serve as a develop- Department of Psychology, San Diego State University, San Diego, CA, USA.
mental tool to improve both leadership and EBP imple- Received: 31 July 2013 Accepted: 26 March 2014
mentation success within organizations. Published: 14 April 2014
Aarons et al. Implementation Science 2014, 9:45 Page 9 of 10
http://www.implementationscience.com/content/9/1/45

References 23. Hill N, Seo M, Kang J, Taylor M: Building employee commitment to

1. Hoagwood K: Family-based services in children’s mental health: a change across organizational levels: the influence of hierarchical
research review and synthesis. J Child Psychol Psyc 2005, 46:690–713. distance and direct managers’ transformational leadership. Organ Sci
2. Aarons GA, Hurlburt M, Horwitz SM: Advancing a conceptual model of 2012, 23:758–777.
evidence-based practice implementation in public service sectors. 24. Hong Y, Liao H, Hu J, Jiang K: Missing link in the service profit chain:
Adm Policy Ment Hlth 2011, 38:4–23. a meta-analytic review of the antecedents, consequences, and
3. Magnabosco JL: Innovations in mental health services implementation: moderators of service climate. J Appl Psychol 2013, 98:237–267.
a report on state-level data from the U. S. evidence-based practices 25. Schneider B, Ehrhart MG, Mayer DM, Saltz JL, Niles-Jolly K: Understanding
project. Implement Sci 2006, 1:1–11. organization-customer links in service settings. Acad Manage J 2005,
4. Chaffin M: Organizational culture and practice epistemologies. Clin Psychol-Sci 48:1017–1032.
Pr 2006, 13:90–93. 26. Barling J, Loughlin C, Kelloway EK: Development and test of a model
5. Kessler ML, Gira E, Poertner J: Moving best practice to evidence-based linking safety-specific transformational leadership and occupational
practice in child welfare. Fam Soc 2005, 86:244–250. safety. J Appl Psychol 2002, 87:488–496.
6. Aarons GA, Sommerfeld DH: Leadership, innovation climate, and attitudes 27. Zohar D: Modifying supervisory practices to improve subunit safety: a
toward evidence-based practice during a statewide implementation. leadership-based intervention model. J Appl Psychol 2002, 87:156–163.
J Am Acad Child Psy 2012, 51:423–431. 28. Powell BJ, McMillen JC, Proctor EK, Carpenter CR, Griffey RT, Bunger AC,
7. Damanpour F, Schneider M: Phases of the adoption of innovation in Glass JE, York JL: A compilation of strategies for implementing clinical
organizations: effects of environment, organization and top managers. innovations in health and mental health. Med Care Res Rev 2012,
Brit J Manage 2006, 17:215–236. 69:123–157.
8. Jung DI, Chow C, Wu A: The role of transformational leadership in 29. Leeman J, Baernholdt M, Sandelowski M: Developing a theory-based
enhancing organizational innovation: hypotheses and some preliminary taxonomy of methods for implementing change in practice. J Adv Nurs
findings. Leadership Quart 2003, 14:525–544. 2007, 58:191–200.
9. Gumusluoglu L, Ilsev A: Transformational leadership, creativity, and 30. Massoud M, Nielsen G, Nolan K, Nolan TW, Schall MW, Sevin C: A Framework
organizational innovation. J Bus Res 2009, 62:461–473. for Spread: From Local Improvements to System-wide Change. Cambridge,
10. Scott SG, Bruce RA: Determinants of innovative behavior: a path model of MA: Institute for Healthcare Improvement; 2006.
individual innovation in the workplace. Acad Manage J 1994, 37:580–607. 31. Wensing M, Bosch MC, Grol R: Selecting, tailoring, and implementing
11. Aarons GA: Transformational and transactional leadership: association knowledge translation interventions. In Knowledge Translation in Health
with attitudes toward evidence-based practice. Psychiatr Serv 2006, Care: Moving from Evidence to Practice. Edited by Straus S, Tetroe J, Graham
57:1162–1169. ID. Oxford, England: Wiley-Blackwell; 2009:94–113.
12. Michaelis B, Stegmaier R, Sonntag K: Shedding light on followers’ 32. Klein KJ, Conn AB, Sorra JS: Implementing computerized technology:
innovation implementation behavior: the role of transformational an organizational analysis. J Appl Psychol 2001, 86:811–824.
leadership, commitment to change, and climate for initiative. J Manage 33. Schein E: Organizational Culture and Leadership. San Francisco, CA: John
Psychol 2010, 25:408–429. Wiley and Sons; 2010.
13. Aarons GA, Horowitz JD, Dlugosz LR, Ehrhart MG: The role of 34. Bass BM, Avolio BJ: MLQ: Multifactor leadership questionnaire (Technical
organizational processes in dissemination and implementation research. Report). Binghamton University, NY: Center for Leadership Studies; 1995.
In Dissemination and Implementation Research in Health: Translating science 35. Patterson MG, West MA, Shackleton VJ, Dawson JF, Lawthom R, Maitlis S,
to practice. Edited by Brownson RC, Colditz GA, Proctor EK. New York, NY: Robinson DL, Wallace AM: Validating the organizational climate measure:
Oxford University Press; 2012. links to managerial practices, productivity and innovation. J Organ Behav
14. Harvey G, Fitzgerald L, Fielden S, McBride A, Waterman H, Bamford D, Kislov 2005, 26:379–408.
R, Boaden R: The NIHR collaboration for leadership in applied health 36. Quinn R, Rohrbaugh J: A spatial model of effectiveness criteria: towards a
research and care (CLAHRC) for Greater Manchester: combining competing values approach to organizational analysis. Manage Sci 1983,
empirical, theoretical and experiential evidence to design and evaluate a 29:363–377.
large-scale implementation strategy. Implement Sci 2011, 6:96. 37. Fabrigar LR, Wegener DT, MacCallum RC, Strahan EJ: Evaluating the use of
15. Kyratsis Y, Ahmad R, Holmes A: Making sense of evidence in management exploratory factor analysis in psychological research. Psychol Methods
decisions: the role of research-based knowledge on innovation adoption 1999, 4:272–299.
and implementation in healthcare. Study protocol. Implement Sci 2012, 38. Zwick WR, Velicer WF: Comparison of five rules for determining the
7:22. number of components to retain. Psychol Bull 1986, 99:432–442.
16. Weiner BJ: A theory of organizational readiness for change. Implement Sci 39. Patil VH, Singh SN, Mishra S, Donovan T: Efficient theory development and
2009, 4:67. factor retention criteria: a case for abandoning the ‘Eigenvalue Greater
17. Michaelis B, Stegmaier R, Sonntag K: Affective commitment to change Than One’ criterion. J Bu Res 2008, 61(2):162–170.
and innovation implementation behavior: the role of charismatic 40. Horn JL: A rationale and test for the number of factors in factor analysis.
leadership and employees’ trust in top management. J Change Manage Psychometrika, 1965, 30:179-85.
2009, 9:399–417. 41. Dunn G, Everitt B, Pickles A: Modeling Covariances and Latent Variables Using
18. Aarons GA, Ehrhart MG, Farahnak LR, Hurlburt M: Leadership and EQS. London: Chapman & Hall/CRC Press; 1993.
organizational change for implementation (LOCI): A mixed-method pilot 42. Hu L-T, Bentler PM: Fit indices in covariance structure modeling:
study of a leadership and organization development intervention for sensitivity to underparameterized model misspecification. Psychol Methods
evidence-based practice implementation. Manuscript submitted for 1998, 3:424–453.
publication. In review. 43. Hu L-T, Bentler PM: Cutoff criteria for fit indexes in covariance structure
19. Sloan R, Gruman J: Participation in workplace health promotion analysis: conventional criteria versus new alternatives. Struct Equ Modeling
programs: the contribution of health and organizational factors. Health 1999, 6:1–55.
Educ Behav 1988, 15:269–288. 44. Kelloway EK: Using Lisrel for Structural Equation Modeling: A Researcher’s
20. Aarons GA, Sommerfeld DH, Willging CE: The soft underbelly of Guide. Thousand Oaks, CA: Sage; 1998.
system change: the role of leadership and organizational climate in 45. Guo Q, Li F, Chen X, Wang W, Meng Q: Performance of fit indices in
turnover during statewide behavioral health reform. Psychol Serv different conditions and selection of cut-off values. Acta Psychologica
2011, 8:269–281. Sinica 2008, 40:109–118.
21. Bain PG, Mann L, Pirola-Merlo A: The innovative imperative: the relationships 46. Shrout PE, Fleiss JL: Intraclass correlations: uses in assessing rater
between team climate, innovation, and performance in research and reliability. Psychol Bull 1979, 86:420–428.
development teams. Small Group Res 2001, 32:55–73. 47. Cochran WG: Sampling Techniques. 3rd edition. New York: Wiley and Sons;
22. Kinjerski V, Skrypnek BJ: The promise of spirit at work: increasing job 1977.
satisfaction and organizational commitment and reducing turnover and 48. James LR, Demaree RG, Wolf G: Estimating within-group interrater reliability
absenteeism in long-term care. J Gerontol Nurs 2008, 34:17–25. with and without response bias. J Appl Psychol 1984, 69:85–98.
Aarons et al. Implementation Science 2014, 9:45 Page 10 of 10
http://www.implementationscience.com/content/9/1/45

49. James LR, Demaree RG, Wolf G: rwg: an assessment of within-group interrater
agreement. J Appl Psychol 1993, 78:306–309.
50. Brown RD, Hauenstein NMA: Interrater agreement reconsidered:
an alternative to the rwg indices. Organ Res Methods 2005, 8:165–184.
51. Lagomasino IT, Zatzick DF, Chambers DA: Efficiency in mental health
practice and research. Gen Hosp Psychiat 2010, 32:477–483.
52. Chambers DA, Wang PS, Insel TR: Maximizing efficiency and impact in
effectiveness and services research. Gen Hosp Psychiat 2010, 32:453–455.
53. Kouzes JM, Posner BZ: The Leadership Challenge. San Francisco, CA:
Jossey-Bass; 2007.
54. Proctor E, Silmere H, Raghavan R, Hovmand P, Aarons GA, Bunger A, Griffey
R, Hensley M: Outcomes for implementation research: conceptual
distinctions, measurement challenges, and research agenda. Adm Policy
Ment Hlth 2011, 38:65–76.

doi:10.1186/1748-5908-9-45
Cite this article as: Aarons et al.: The implementation leadership scale
(ILS): development of a brief measure of unit level implementation
leadership. Implementation Science 2014 9:45.

Submit your next manuscript to BioMed Central

and take full advantage of:

• Convenient online submission

• Thorough peer review
• No space constraints or color figure charges
• Immediate publication on acceptance
• Inclusion in PubMed, CAS, Scopus and Google Scholar
• Research which is freely available for redistribution

Submit your manuscript at

www.biomedcentral.com/submit