European Management Journal xxx (2016) 1e8

Contents lists available at ScienceDirect

European Management Journal

Choosing PLS path modeling as analytical method in European

a r t i c l e i n f o a b s t r a c t

Article history: Today, there is heightened controversy about the value of partial least squares (PLS) path modeling as a
Received 30 December 2015 quantitative research method, including within the domain of European management research. Critical
Received in revised form lines of argument within the management and psychology literature assert that there is no reason to use
19 May 2016
PLS path modeling at all. At the same time, authors using PLS path modeling continue to advance
Accepted 24 May 2016
fallacious arguments to justify their choice of method. This paper identifies flaws on both sidesdinvalid
Available online xxx
arguments in favor of using PLS path modeling and invalid arguments opposing its usedwithin the
context of a unifying framework and a realist philosophy of science.
Keywords:
Structural equation modeling
© 2016 Elsevier Ltd. All rights reserved.
Partial least squares path modeling
Scientific realism
Conceptual variable
Proxy variable
Measurement error

You are in Rome. Something wonderful is waiting for you in it.” Referring to PLS path modeling, Ro €nkko€ and Evermann (2013, p.
Brussels. Unfortunately, there are only two flights available to 19) assert, “… it is very difficult to justify its use for theory testing
you. One goes to Frankfurt, and the other goes to Paris. As you over [factor-based] SEM…” Writing in a psychological journal,
ponder this choice, some people confront you. “Frankfurt?” they Ro€ nkko€ , McIntosh, and Antonakis (2015, p. 82) conclude, “… PLS
scoff. “Why fly to Frankfurt? Brussels is in BelgiumdFrankfurt is should not be adopted as a tool for psychological research.”
not even in the same country. It would be ridiculous to fly to Perhaps the most recent and impactful contribution in this vein
Frankfurt, because the city you are trying to reach is Brussels.” is an editorial from the editors in chief of Journal of Operations
They confidently conclude, “Frankfurt doesn't work. Fly to Management (Guide & Ketokivi, 2015, p. vii), who warned, “We are
Paris.” desk rejecting practically all PLS-based manuscripts, because we
have concluded that PLS has been without exception the wrong
modeling approach in the kinds of models OM researchers use.”
However, Guide and Ketokivi went further, clarifying that desk
1. Introduction rejection was primarily a response to researchers making incorrect
claims about PLS path modeling (p. vii): “Consequently, we will
Perhaps there has always been controversy between different automatically desk reject a manuscript that makes incorrect claims
approaches to structural equation modeling (SEM), ever since about the applicability of the estimator (obviously, any estimator,
Herman Wold unveiled a composite-based alternative to Karl not just PLS).”
€reskog's common factor-based innovation. In the last few
Jo Guide and Ketokivi's (2015) editorial points to two different
yearsdperhaps in response to a new vibrancy within the partial problems related to the use and understanding of PLS path
least squares (PLS) path modeling communitydthe tenor of this modeling. Today, too many researchers offer a flawed rationale for
controversy has become sharper. Antonakis, Bendahan, Jacquart, choosing PLS path modeling as their method, citing strengths or
and Lalive (2010, p. 1103) declared, “… there is no use for PLS advantages for PLS path modeling that do not exist. At the same
whatsoever… We thus strongly encourage researchers to abandon time, critics offer flawed reasons to avoid PLS path modeling. Some
of these critical arguments falsely ascribe advantageous properties
to the factor-based approach to SEM that do not exist, while some
E-mail address: [email protected].

http://dx.doi.org/10.1016/j.emj.2016.05.006
0263-2373/© 2016 Elsevier Ltd. All rights reserved.

Please cite this article in press as: Rigdon, E. E., Choosing PLS path modeling as analytical method in European management research: A realist
perspective, European Management Journal (2016), http://dx.doi.org/10.1016/j.emj.2016.05.006
2 E.E. Rigdon / European Management Journal xxx (2016) 1e8

are based on flawed evidence about the performance of PLS path the observed variables found in datasets. Structural equation
modeling. modeling explicitly incorporates a third type of variable. In this
The aim of this paper is to review and correct both types of framework, the common factors in factor-based SEM and the
errorsdboth alleged strengths or advantages and alleged weak- composites in composite-based SEM are together classed as “proxy
nesses of PLS path modeling which have not been supported with variables” (Wickens, 1972; Woolridge, 2009), defined as variables
valid evidence, despite publication in well-regarded academic that stand in for other variables which may be unobservable or
journals. This paper presents an alternative understanding of simply unavailable. Factor-based SEM and composite-based SEM
structural equation modeling, one which is consistent with aspects use common factors or composites to represent unobserved con-
of factor-based and composite-based approaches to SEM which ceptual variables. These representations are formed out of the
both users and critics of PLS path modeling have tended to ignore. observed variables in a dataset. Because the representations are
It is easy enough to find oneself embracing the beliefs and biases formed out of data, they share the strengths and weaknesses of
of one school of thought, to the point where contrary arguments and data, to some degree. As such, the common factors or composites in
perspectives seem not only wrong but nonsensical and even SEM's statistical models are not equivalent to or identical with the
dangerous. Unsuspecting researchers may quietly succumb to a conceptual variables that populate theoretical models. So, beyond
“methodological tribalism” (Saunders & Bezzina, 2015, p. 298). the important statistical differences between the factor-based and
While Saunders and Bezzina (2015) studied the implications of a composite-based approaches to SEM, there is a crucial underlying
broad qualitative vs quantitative orientation among (primarily) Eu- similarity.
ropean management researchers, the same divide arises between
researchers with differing quantitative backgrounds. As with 3. Flawed arguments in favor of PLS path modeling
Saunders and Bezzina (2015, p. 303), this is not a call for “method-
ological relativism,” that is, for simply withholding judgment. Rather, As Guide and Ketokivi's (2015) editorial confirms, a great many
the aim here is to overcome misunderstandings by embracing a management research manuscripts that employ PLS path modeling
pluralism of quantitative methodologies which are all rigorously do a very poor job of justifying their choice of statistical method.
consistent with a single framework. This paper is written in the hope Unfortunately, many of the invalid arguments employed by these
that it will (a) help researchers to make better design and methods papers date to the origin of PLS path modeling, and they are
choices, (b) help writers to avoid crucial errors in explaining their stubbornly repeated in books and methodological manuscripts,
choices, and (c) help to move the SEM dialog forward. despite contrary evidence, some of which is common knowledge in
the factor-based SEM community. Granted, much statistical prac-
2. A framework for understanding structural equation tice is justified using invalid arguments, when researchers even
modeling bother to offer a justification (Gigerenzer, 2004; Wasserstein &
Lazar, 2016). Taking an approach that is less common or that de-
Some disputes are particularly immune to resolution because viates from the norm is more likely to provoke questions. Giger-
the different sides are operating on the basis of fundamentally enzer (e.g., 2004) points out how null hypothesis significance
different assumptions, philosophies or worldviews. In such cases, testing with p-values is mindlessly executed in paper after paper-
the “Yes, it is” from the one side and the “No, it isn't” from the other dand mindlessly described in textbook after textbook. When was
side may both be correct (or incorrect) because the two positions the last time a writer was asked to justify use of a p-value? Re-
are referring to different realities. It may therefore be helpful to viewers have even been known to demand themdwithout
briefly outline a framework within which one might understand providing a reason, and without regard to the potential to
arguments about different approaches to structural equation mislead (Ziliak & McCloskey, 2008). But when the editors of Basic
modeling. and Applied Social Psychology took the opposite course and banned
Statistical tools that fall within the family of SEM methods p-values from their journal (Trafimow & Marks, 2015), there was an
potentially could be used for a variety of purposes, but here the uproardfollowed by thoughtful reconsideration at the highest
focal purpose is better understanding of the behavior of unob- levels (Wasserstein & Lazar, 2016). It is probably true that many
served conceptual variables. From a scientific realist perspective manuscripts employing the common factor-based approach to SEM
(Chakravartty, 2007; Haig & Evers, 2016; Leplin, 1984), these un- either fail to justify their choice of technique or provide a justifi-
observed conceptual variables are defeasibly real entities, like the cation incorporating invalid arguments (Cliff, 1983). But the factor-
microorganisms that were once invisible or the subatomic particles based approach is the most widely known approach to SEM, and so
that remain beyond perception today. While they may remain applications of that approach may face less scrutiny. Regardless,
unobservable themselves, these conceptual variables are of interest researchers who make unjustified claims in support of the use of
because of their direct or indirect causal consequences in the PLS path modeling can expect an unwelcoming reception for their
observable world. From an empiricist perspective, in contrast, only work.
the observable phenomena would be considered real, with unob-
served conceptual variables being no more than labels for certain 3.1. Low sample size
observed empirical regularities (Creath, 2014). An operationalist, in
the tradition of P. W. Bridgman, Edwin Boring or S. S. Stevens, might Data weaknesses were a primary motivation for Wold to create a
define a “conceptual variable” to be nothing more than the appli- composite-based alternative to factor-based SEM. Wold touted the
cation of a particular quantitative methodology (like factor anal- ability of PLS path modeling to “work”dthat is, to produce
ysis) to data collected under a particular protocol (Chang, 2009). parameter estimates and standard errorsdeven when there were
But again, for the scientific realist, unobservable conceptual vari- fewer cases or observations than variables, and when the available
ables are themselves real and are independent of data, of statistical observations were not even mutually independent (Wold, 1988).
procedure, and of the researcher, though mistakes and mis- This stood in contrast to maximum likelihood-based factor analysis,
understandings can lead to incorrect beliefs about particular con- which can fail entirely when sample size is low. In the context of
ceptual variables. linear regression, it has been noted that the same statistical method
So far, this framework has described two types of variablesdthe can imply different minimum sample sizes depending on the
unobserved conceptual variables found in theoretical models and particular purpose or criterion that is being pursueddwhether the

Please cite this article in press as: Rigdon, E. E., Choosing PLS path modeling as analytical method in European management research: A realist
perspective, European Management Journal (2016), http://dx.doi.org/10.1016/j.emj.2016.05.006
 E.E. Rigdon / European Management Journal xxx (2016) 1e8 3

research is looking for cross-validated R2, or for sufficient statistical estimation in particular but with factor-based SEM broadly:
power, or for something else (Maxwell, 2000, pp. 434e5). Still, low “CBSEM [factor-based SEM] generally requires a multivariate
sample size was cited by Ioannidis (2005) as a primary driver in the normal distribution of the sample data” (Peng & Lai, 2012, p. 470).
publication of false scientific findingsdregardless of analytical For many years now, there have been a variety of procedures
method. available for estimating the parameters of factor-based models. The
Given the focus on predictive validity that is widely shared distributional requirements underlying these various procedures
among PLS path modeling users, the simulations of Dana and have ranged from strict to loose to almost none. ML estimation it-
Dawes (2004) are especially relevant. Their study examined out- self is somewhat robust to modest violations of multinormality,
of-sample R2 in regression. The authors created large populations especially in regard to parameter estimates (Finney & DiStefano,
of simulation data and then for each population, (1) drew a sample, 2013). While the earliest of these techniquesdBrowne's (1984)
(2) estimated model parameters from the sample, (3) predicted the weighted least squares approachdrequired infeasible sample
dependent variable at population level using the parameter values sizes numbering in the thousands, more recent alternatives have
estimated from the sample. They found that ordinary least squares been shown to work well at sample sizes as small as 200 (Lei & Wu,
(OLS) regression weights produced the highest out-of-sample R2 2012; Yuan & Bentler, 1998). Moreover, there is little cause to
only when both sample size was quite large and true predictability compare ML estimation of factor model parameters with PLS esti-
(the true population R2 for the dependent variable) was high. At the mation of composite model parameters. The factor-based model
other end, when sample size was small and/or true predictability and the composite-based model are two different models. Even if
was low, simple unit or equal weightsdthat is, just summing the ML estimation did a poor job of estimating a factor-based model in
(standardized) predictorsdoutperformed regression weights. a certain situation, this failure would argue for a change of esti-
Becker et al.'s (2013) simulations confirmed this result in the mation method, not for a change from factor-based SEM to
context of PLS path modeling. Consistent with Dana and Dawes' composite-based SEM.
(2004) results for regression, Becker et al. (2013) found that, at
low sample sizes, researchers would do as well to simply sum their 3.3. Formative/reflective
multiple indicators and forget about weights. Becker et al.'s (2013)
simulations demonstrated bias in PLS parameter estimates when “Reflective measurement” is a term used to describe a situation
sample size is low. Yes, PLS path modeling will produce parameter where a set of observed variables are jointly dependent upon
estimates even when sample size is very small, but reviewers and another variable which is not itself observed. With appropriate
editors can be expected to question the value of those estimates, constraints on the residual variances of the observed variables, this
beyond simple data description. With respect to both composite- arrangement describes a common factor model, so it is easy to
based and factor-based approaches to SEM, if sample size is understand how “reflective measurement” might represent the
small, the best course is to get more data. norm in factor-based SEM. The reverse arrangement, where the
Sometimes (as an anonymous reviewer has pointed out), more unobserved variable is modeled as dependent on the observed
data are not obtainable. Sometimes, indeed, evidence is not avail- variables, is then known as “formative measurement.” The factor-
able to allow a researcher to make an impactful contribution in a based SEM literature struggles mightily with formative measure-
given area. Before the advent of CERN's Large Hadron Collider, it ment, on both practical and conceptual grounds (e.g., Bollen &
was impossible to obtain experimental evidence for the existence Bauldry, 2011; Diamantopoulos & Winklhofer, 2001; Edwards,
of the Higgs boson subatomic particle (http://home.cern/topics/ 2011). Given this, researchers sometimes argue that they must
higgs-boson). Before the upgrading of the Laser Interferometer use PLS path modeling because “their concepts are formative.”
Gravitational-Wave Observatories (LIGO), no researcher had been A realist perspective clarifies these issues significantly. If con-
able to demonstrate the existence of gravitational waves (Abbott ceptual variables transcend data, then it is impossible to “form”
et al. 2016). In other cases, a population itself may be countably conceptual variables out of data. By definition, conceptual variables
finite. In some areas of statistics, adjustment procedures may be are themselves the causes of data. Both the factor-based and the
available to account for this deviation from the standard assump- composite-based approaches to SEM form proxies, not conceptual
tion of an infinite population (e.g., Royall, 1970). However, the variables, out of data. Because PLS path modeling is a composite-
generalizability of findings is limited to the population from which based method, it creates proxies as weighted composites (Rigdon,
data are sampled. Making a significant contribution will require 2012), consisting of one or more variables. In the factor-based
that this population be of particular interest in itself. So it will be approach to SEM, proxies are formed as common factors (except
the nature of the population that justifies the small sample size, and for single indicators), even when it may appear otherwise (Rigdon
not the small sample size that justifies the choice of PLS path et al. 2014).
modeling. Whether PLS path modeling performs better than Among its estimation options, PLS path modeling includes two
alternative approaches in analysis of data from finite populations is which have been known for decades as “Mode A” and “Mode B.”
a little-explored research area. Many writers associate Mode A estimation with “reflective mea-
surement” and associate Mode B with “formative measurement”
3.2. Non-normal data (e.g., Hair, Hult, Ringle, & Sarstedt, 2014, pp. 42e43). This is an
illusion. Both modes create composite proxies, because PLS path
€reskog's (e.g., 1969) creation of an inferential basis for factor
Jo modeling cannot do anything else. Using Mode A instead of Mode B
analysis initially relied on maximum likelihood (ML) estimation. means using correlation weights (Waller & Jones, 2010) instead of
€reskog (1969) proved analytically that, when its assumptions
Jo OLS regression weights (Becker, Rai, & Rigdon, 2013; Rigdon, 2012).
held, no method yielding unbiased estimates could be more sta- Unlike OLS regression weights, correlation weights ignore collin-
tistically efficient than ML estimation. In creating PLS path earity among predictors. This means that users of correlation
modeling, Herman Wold was pushing back against the distribu- weights do not experience parameters having unexpected signs
tional assumptions that supported ML estimation (Dijkstra, 2010), due to the impact of collinearity on weights, and will not be misled
which included an assumption of conditional multivariate into deleting indicators based on collinearity-driven signs and
normality (Finney & DiStefano, 2013). Since those early days, PLS inflated standard errors. As confirmed by Becker et al. (2013) in the
users have linked the multinormality assumption not with ML PLS context, Dana and Dawes (2004) demonstrated that, while

Please cite this article in press as: Rigdon, E. E., Choosing PLS path modeling as analytical method in European management research: A realist
perspective, European Management Journal (2016), http://dx.doi.org/10.1016/j.emj.2016.05.006
4 E.E. Rigdon / European Management Journal xxx (2016) 1e8

correlation weights yield a somewhat lower in-sample R2 than OLS being a “deliberate approximation” to ML factor analysis, accepting
regression weights, they yield a higher out-of-sample R2 when less accuracy in exchange for greater speed and relaxed assump-
sample size and true predictability are moderate, potentially tions. A number of studies have used simulation to demonstrate
covering a much larger range of practice than the special conditions this bias (e.g., Aguirre-Urreta & Marakas, 2013; McDonald, 1996;
required for OLS regression weights to excel. So there can be good Ro€nkko€ & Evermann, 2013). As various PLS path modeling texts
reason to choose Mode A or Mode B within a PLS path model, but note, estimates of factor model loadings will tend to be biased
this has nothing to do with a choice between “formative” and upwards (away from 0), while estimates of paths between factors
“reflective.” Thus, researchers do face real choices, between com- will tend to be biased downwards (toward 0).
mon factor proxies and composite proxies, and between regression However, all of these simulations, and perhaps Wold's own
weighted composites and correlation weighted composites. In thinking, were flawed (Wold, 1982, p. 25). Some of these studies
contrast, the terms “formative” and “reflective” only obscure the were flawed in multiple relevant ways. One flaw is common to all. A
statistical reality. simulation must begin by specifying a populationda true state
from which data are sampled. Even though these studies aimed to
3.4. Exploratory evaluate the performance of composite-based PLS path modeling, all
of the simulations noted here began by defining a truth consistent
Wold recognized that factor-based SEM required the researcher with a common factor model. Thus, these simulations evaluated PLS
to have a well-developed statistical model in mind. He also saw that models that were misspecified relative to the population. Statistical
there were circumstances where data were available but the prior methods in general perform less well when the model is mis-
knowledge was not. A poor factor model may produce no results specified, and the same is true of PLS path modeling. Factor-based
whatsoever, while a PLS path modeling analysis is very likely to SEM itself is known to perform less well when the model being
produce parameter estimates and bootstrapped standard errors. estimated is inconsistent with the population (e.g., Hu & Bentler,
Thus, Wold (1985, p. 589) recommended his composite-based 1998), but scholars have repeatedly chosen to evaluate PLS path
method for situations that were “data-rich but theory-primitive.” modeling using discrepant populations.
This association between PLS path modeling and exploratory Simulations employing a correct population have shown that
analysis continues in the literature (Hair et al. 2014, p. 2, 14). PLS path modeling estimates are consistent (Becker et al. 2013).
From a realist perspective, there is a substantial problem with Becker et al.'s (2013) simulations first defined composite-based
using any SEM method in an exploratory context. Findings drawn populations, and then examined the performance of PLS path
from SEM analysis face a substantial validity challenge, relating modeling in analyzing samples drawn from those populations.
specifically to the proxies that represent unobserved conceptual Becker et al. (2013) found the bias in PLS path modeling estimates
variables. For findings to be judged valid, the proxies themselves approaching 0 as sample size increased. Even in these simulations,
must be valid representations of particular conceptual variables. In at lower sample sizes, PLS parameter estimates showed clear bias,
an exploratory context, a researcher may have no idea what sorts of with the nature of that bias varying based on simulation conditions.
conceptual variables may be at work, and thus may be in no posi- Thus, Becker et al.'s (2013) results provided further evidence
tion to make any statements, pro or con, about the validity of the against the use of PLS path modeling at low sample sizes. No
proxy variables in a model. So, as in the case of low sample size, PLS simulations using composite-based populations have obtained re-
analysis can be executed in an exploratory environment, but it is sults contradicting those of Becker et al. (2013).
unlikely to lead to a significant contributiondunless the Individual simulation studies critical of PLS path modeling have
dataset alone is so interesting and unique that data description included additional design errors. Aguirre-Urreta and Marakas
itself amounts to a contribution. That could happen, if data are rare, (2013) criticized the behavior of PLS path modeling in the context
difficult to obtain, and (within the scope of this journal) relate of “formative measurement.” Aguirre-Urreta and Marakas (2013)
directly to constituencies and issues of especially keen or urgent created the population for their simulation using a common fac-
interest to European management researchers. More likely, re- tor model. To minimize the problems that can be encountered
searchers who attribute their choice of PLS path modeling to the when modeling a “formative” relationship with a factor-based
exploratory nature of their research will be declaring, in effect, that model, Aguirre-Urreta and Marakas (2013) specified a population
their work has no contribution to make. where a set of observed variables, along with another common
factor, served as predictors of a focal common factor. Statistical
4. Flawed arguments against the use of PLS path modeling identification for this model (a rather involved issue for factor-
based SEM) was achieved by including other common factors that
Clearly, a substantial number of researchers are building justi- were exclusively dependent on the focal common factor, making it
fications for choosing PLS path modeling out of flawed arguments. a second order factor. The PLS path model that the authors esti-
Just as clearly, a phalanx of incorrect or unsupported claims are mated, however, was different (Rigdon et al. 2014). In the PLS path
being presented to back a rejection of PLS path modeling as an model, the observed variables, modeled as predictors in the popu-
analytical method. Some of these invalid arguments are tied to lation, were now components of a compositeda composite which,
weaknesses in the factor-based approach which the factor-based in turn, was predicted by another composite. In the population, the
SEM community has long ignored, while some of these issues observed variables and a common factor were predictors jointly,
have only become clear in recent years. The position here is not but in the PLS path model the predictor composite was the only
necessarily that these arguments are incorrect but that they are predictor of a composite defined by the observed variables. These
invalid as arguments favoring the factor-based approach to SEM differences between models, compounded with the factor-based
over PLS path modeling, particularly from a realist perspective. nature of the simulation population, invalidated Aguirre-Urreta
and Marakas' (2013) conclusions regarding the behavior of PLS
4.1. Biased parameter estimates path modeling.
Ro€nkko
€ and Evermann's (2013; Ro €nkko € et al. 2015) simulations
From the birth of PLS path modeling, it has been known and included a different design flaw. Ro €nkko € and Evermann's (2013)
acknowledged that the method yields biased estimates of factor simulations appeared to demonstrate that PLS parameter esti-
model parameters. Wold (1982, p. 27, 28) spoke of his method as mates could be not only inconsistent but bimodal, a shocking

Please cite this article in press as: Rigdon, E. E., Choosing PLS path modeling as analytical method in European management research: A realist
perspective, European Management Journal (2016), http://dx.doi.org/10.1016/j.emj.2016.05.006
 E.E. Rigdon / European Management Journal xxx (2016) 1e8 5

deviation from the bell-shaped distribution that researchers would term appears to mean either (a) a conceptual variable in a theo-
be inclined to expect. Ro € nkko
€ and Evermann's (2013) results sug- retical model which is believed to affect the behavior of other
gested that PLS path modeling could hardly be trusted as a statis- variables, or (b) a common factor. If by “latent variable,” someone
tical method at all. intends the second meaning, then of course it is correct to say that
Ro€ nkko
€ and Evermann's (2013) results were obtained by spec- PLS path modeling, a composite-based method, is not a factor-
ifying a model that violated the known conditions under which the based method, but this on its face appears to be mere methodo-
PLS path modeling estimation algorithm works. This algorithm logical tribalism. It would be no less pointless to assert that
requires that every composite proxy must be correlated with at composite-based PLS path modeling is better than the factor-based
least one other composite (e.g., Rigdon, 2013). The PLS path approach because the factor-based approach is not composite-
modeling algorithm alternates what are called “inner proxies” and based.
“outer proxies.” The inner proxy for a given composite is formed On the other hand, if “latent variable” refers to an unobserved
from other composites that have a direct relationship with the conceptual variable, then the argument is incorrect. From a scien-
given composite, in the statistical model. If a composite is uncor- tific realist perspective, conceptual variables are as crucially
related with all other composites in the model, the algorithm fails. important to PLS path modeling as they are to the factor-based
Ro€ nkko€ and Evermann (2013) specified a population model with approach to SEM. Without conceptual variables, a proxy is no
two common factors, with the population correlation between more than some function of the data. There can be no question of
them set at 0. Then they attempted to estimate PLS path models validity, because the proxy, formed from the dataset, represents
using these data. The choice of population ensured that the PLS nothing beyond the dataset. The composite-based approach and
estimation algorithm would only function when random sampling the factor-based approach alike cannot be anything more than
error pushed the actual correlation between composites away from exercises in data description. It may well be that, in practice, many
0. Hence the simulation produced a bimodal distribution, with applications of both approaches are no more than this, but the two
estimates on the positive side and estimates on the negative side, approaches are equally stripped of their potential to build knowl-
and little in between. PLS path modeling can certainly accommo- edge if their proxies stand only for themselves.
date 0 paths between composites, but it cannot accommodate a It has also been argued that the patterns of correlation captured
composite that is orthogonal to all other composites in a model. by common factor models imply a common cause, so that common
Far from demonstrating the untrustworthy nature of PLS path factor models provide evidence for the existence of unobserved
modeling, Ro €nkko € and Evermann's simulations showed what conceptual variables in a way that composite-based models do not.
happens when you “break” a statistical method, asking it to work However, identical patterns can be induced by processes that lack
outside of its boundary conditions. Ro € nkko
€ and Evermann (2013) this common cause (van der Maas et al. 2006, 2014), though the
could easily have demonstrated a similar limitation in factor- statistical models implied by these complex processes may be
based SEM, if that had been their purpose. Their simulation beyond the power of existing software to estimate. Moreover, given
model included three observed variables loading on each of the two that factor models are generally no more than approximations to
common factors. If the model had instead specified that four data from real situations (Jo€reskog, 1969; MacCallum, Browne, &
observed variables loaded on one factor while two loaded on the Cai, 2007), common factor proxies cannot be generally assumed
other, then the zero population correlation would have caused to carry greater significance than composite proxies in regard to the
factor-based SEM to fail, just as PLS path modeling failed, because existence or nature of conceptual variables. Moreover, the multiple
the factor model would have been under-identified (Bollen, 1989). meanings associated with the terms “latent variable” and
When a statistical model is under-identified, parameter estimates “construct” make these terms no more useful than “formative” and
are inconsistent and other negative consequences follow. The “reflective.”
“three indicator rule” for identification of factor models specifies
that a model will be identified if each common factor has at least 4.3. No overall fit test
three “congeneric” indicators (loading on that factor alone, with no
residual correlations). The “two indicator rule” specifies that the € reskog's (e.g., 1969) maximum likelihood factor analysis
Jo
model can be identified with only two congeneric indicators per became an inferential statistical method thanks to an overall test
factor as long as each factor is correlated with some other variable statistic that, under assumptions, follows a central c2 distribution.
in the model. Ro €nkko€ and Evermann included just enough in- This statistic can be used to formally test the null hypothesis that
dicators for each factor in the population model so that estimation the data-based covariance matrix of the observed variables is equal
of the factor model would not fail when the factor correlation was to the model-implied covariance matrix, within sampling error. PLS
0, and so that it would seem that PLS path modeling, by contrast, path modeling, being a composite-based method more closely tied
was unreliable. If either common factor had had only two indicators to regression, offers no such test. Therefore it is argued that PLS
instead of three, factor-based SEM would have also failed, perhaps path modeling is not in position to falsify and reject models in the
even more spectacularly. In sum, the undesirable behavior attrib- same way as the factor-based approach to SEM. Critics of PLS path
uted to PLS path modeling in each simulation study should actually modeling have suggested that this is a fatal flaw in PLS path
be attributed to design flaws in the studies themselves, not to a modeling, making it fundamentally less suitable for serious
fundamental weakness in PLS path modeling. research.
Of course, the factor-based SEM community itself does not think
4.2. Not a latent variable method much of the c2 statistic. Fit evaluation in factor-based SEM tends to
focus on alternative fit indices, not on the c2 itself. The null hy-
Factor-based SEM is said to be distinguished from other multi- pothesis is assumed to be false (MacCallum et al. 2007), so that, “If a
variate statistical methods because it is a technique that models sufficiently large sample were obtained this c2 statistic would, no
“latent variables.” By contrast, it is said that PLS path modeling “is doubt, indicate that any such non-trivial hypothesis was statisti-
not a latent variable method,” and on this basis that it is not cally untenable.” (Jo€ reskog, 1969, p. 200).
structural equation modeling at all (Ro € nkko
€ et al. 2015). Understanding structural equation modeling as involving con-
“Latent variable,” like the noun “construct” (Michell, 2013), is a ceptual variables, observed variables and proxies, moreover, leads
term with multiple potential meanings. In different usages, the to an even more fundamental question about the advantage, if any,

Please cite this article in press as: Rigdon, E. E., Choosing PLS path modeling as analytical method in European management research: A realist
perspective, European Management Journal (2016), http://dx.doi.org/10.1016/j.emj.2016.05.006
6 E.E. Rigdon / European Management Journal xxx (2016) 1e8

that accrues to factor-based SEM on the basis of the c2 test statistic. approach to structural equation modeling protects users from
Statistical analysis is not an end in itself. The realist aims to reach measurement error.
conclusions about the conceptual variables in a theoretical model. This position runs counter to the vast bulk of factor-based SEM
Structural equation modeling facilitates this process to the extent literature, but the roots of the argument go as far back as Wilson
that the proxies (common factors or composites) in statistical (1928). A researcher might represent a factor model linking p
models, built from observed variables, are valid representations of observed variables yi (i ¼ 1 to p) to a single common factor F as:
the conceptual variables. If they are, then researchers can learn
about the behavior of the conceptual variables through the
yi ¼ li F þ εi
behavior of the proxies. If the proxies are not valid, then the sta-
tistical model is uninformative regarding the conceptual variables. The p terms εi (i ¼ 1 to p) are specific factors or residuals, the
The c2 test statistic tells the researcher essentially nothing about part of each yi not accounted for by the common factor, F. These
this most fundamental validity question. What does it indicate? terms have often been described as “measurement errors” (e.g.,
According to Jo € reskog (1969, p. 201), “If a value of c2 is obtained, € reskog, 1983). No such terms figure in PLS path modeling, because
Jo
which is large compared to the number of degrees of freedom, this PLS path modeling is a method of composites and not factors. With
is an indication that more information can be extracted from the “measurement error” thus explicitly represented in the factor
data.” So the c2 might indicate whether or not the common factors model and not in PLS path modeling, researchers could easily
in a model account for all systematic variance among the observed believe that the factor-based approach addresses measurement
variables. But that is not the same thing as indicating whether the error while the composite-based approach does not.
proxies are valid representations of particular conceptual variables. Yet the failure of the factor-based approach to address or ac-
It might well be that valid proxies could be formed while leaving a count for genuine measurement error is apparent on both con-
substantial amount of the observed variables' systematic variance ceptual and analytical grounds. Consider again the conceptual
un-accounted for. Similarly, a good c2 value, indicating that the framework featuring observed variables, conceptual variables and
common factors do account for all systematic variation in the proxies. Defining the εi terms in the factor model as “measurement
observed variables, offers no reason to believe that those factors are errors” implies that the common factor F is that which is to be
valid proxies for particular conceptual variablesdbecause the same measured. Yet the original purpose of the research effort was to
c2 value applies to the statistical model, no matter what conceptual learn about the behavior of a conceptual variable, not to learn about
variables are hypothetically being studied. In terms of the validity a common factor. These two different goals could be pursued
of the factor proxies employed in a statistical model, the c2 statistic simultaneously if the conceptual variable and its common factor
is no more than a very sophisticated and powerful answer to the proxy were identical or equivalent. For that matter, the factor
wrong question. model residuals would indeed be “measurement errors” if it could
be assumed that the common factor proxy was identical to the
4.4. Measurement error conceptual variable that it was intended to represent.
For common factors, such an identity is essentially impossible
One of the seemingly most compelling criticisms of PLS path due to the phenomenon of factor indeterminacy (Guttman, 1955;
modeling is that the method fails to address measurement error. Mulaik, 2010; Scho €nemann & Steiger, 1976). With a composite,
The presence of measurement error can be shown to induce bias in given the values of the composite's observed variable components
parameter estimates (e.g., Rigdon, 1994), not just in parameters and a set of weights, it is easy to determine the one correct value for
directly linked to a single error-tainted observed variable but across the composite. With a common factor, however, this is almost al-
an entire model. Factor-based SEM, it is argued, has a built-in ways impossible. There is no one best value for a common factor,
mechanism accounting for measurement errorda mechanism except at a hypothetical limit, even if data and model parameters
which composite-based PLS path modeling lacks, leaving PLS path are fully known. A given common factor can be represented as a
models vulnerable to the negative consequences of measurement combination of two parts (Guttman, 1955; Scho €nemann & Steiger,
error in a way that factor-based structural equation models are not. 1976). One part is, indeed, a function of model parameters and
Being composite-based, PLS path modeling partially reduces the the model's observed variables. However, the other part is arbitrary,
impact of any random variance within individual components capable of taking on an infinity of different values even when the
(Rigdon, 2012). The variance of a sum of components is equal to (a) values of observed variables and model parameters are all fixed.
the sum of the components' individual variances plus (b) twice the The scope of variation permitted depends on the number of in-
sum of the covariances among the components. Random variance is dicators in the model and the strength of their relationship with the
universally orthogonal. The correlated parts of the components common factor. In general, however, the correlation between two
thus are counted three times (once in the original variance and different realizations of the same common factor, with the same
twice in the covariances), while random variance is represented observed variables and same parameters, need not be high, nor
only once. If the components are weighted, as in PLS path modeling, even positive (Guttman, 1955).
then the variance of the sum is similarly weighted. The weighting in The correlation between different realizations of the same
PLS path modeling is aimed at maximizing correlations. Again, with indeterminate common factor, in turn, governs the relationship
random variance being orthogonal, components that are high on between that common factor and any variable not explicitly
random variance will tend to be under-weighted, and this will tend included in the factor model (Steiger, 1979). The correlation be-
to further reduce the influence of random variance. However, tween common factor and external variable can only be defined in
random variance will not be completely eliminated in this way, so terms of a range. The greater the degree of indeterminacy, the
that the PLS path modeling user will still be exposed to some share wider the range of possible correlations between common factor
of any negative consequences. and external variable. The conceptual variable, for which the
The “measurement error argument” figures prominently in the common factor proxies, is one such external variable. The concep-
literature critical of PLS path modeling (e.g., Antonakis et al. 2010; tual variable is clearly not a part of the factor model itselfdif it
Ro€ nkko
€ & Evermann, 2013). Nevertheless, the measurement error were, then there would be no need for a proxy. Thus, the extent to
argument is not a valid objection to the use of PLS path modeling, which a researcher can establish the correlation between common
because neither the factor-based approach nor the composite-based factor proxy and conceptual variable is a function of the common

Please cite this article in press as: Rigdon, E. E., Choosing PLS path modeling as analytical method in European management research: A realist
perspective, European Management Journal (2016), http://dx.doi.org/10.1016/j.emj.2016.05.006
 E.E. Rigdon / European Management Journal xxx (2016) 1e8 7

factor's indeterminacy. The researcher cannot claim evidence for a compare different empirical representations with the conceptual
perfect correlation, implying equality between proxy and concep- variables being represented, then they will be able to determine
tual variable, unless indeterminacy is 0. Zero indeterminacy can whether one set of representations is better than another.
only be approached when a common factor is associated with a
very large number of strong indicators, something that is never Acknowledgements
observed in practice.
If the common factor and the conceptual variable are not iden- I thank the special issue editorial team for their guidance and
tical, then the observed variable residuals are not “measurement two exceptional though anonymous reviewers for their challenging
errors,” because it is the conceptual variable that is being measured, and constructive comments. This paper is based in part on pre-
not the common factor proxy. A composite proxy will also fail to be sentations delivered in 2015 at the Modern Modeling Methods
identical to the conceptual variable it represents, because, at a Conference at the University of Connecticut and the 2nd Interna-
minimum, the composite will be tainted by random variance. Of tional Symposium on Partial Least Squares Path Modeling at Uni-
course, any kind of proxy may be affected by systematic error, as versidad de Sevilla. I thank the organizers for granting those
well. Whether a particular proxy is affected by error to a greater or a opportunities and thank many listeners for their kind feedback.
lesser degree must be an empirical question, but there seems no This research did not receive any specific grant from funding
basis for arguing that factor proxies will have any overall advantage agencies in the public, commercial, or not-for-profit sectors, though
over composite proxies. This is why a concern about “measurement travel to the conferences named above was supported by the
error” does not amount to a valid argument for favoring a factor- Marketing Department and Robinson College of Business, Georgia
based approach over a composite-based approach like PLS path State University.
modeling.
References
5. Conclusion
Abbott, B. P., et al., LIGO Scientific Collaboration and Virgo Collaboration. (2016).
As a family of statistical methods, structural equation modeling Observation of gravitational waves from a binary black hole merger. Physical
Review Letters, 116, 061102. http://journals.aps.org/prl/abstract/10.1103/
is still young. The factor-based approach to SEM spread rapidly PhysRevLett.116.061102.
across the social sciences (Bentler, 1986), while PLS path modeling Aguirre-Urreta, M. I., & Marakas, G. M. (2013). Partial least squares and models with
emerged later and developed more slowly, perhaps due to the formatively measured endogenous constructs: A cautionary note. Information
Systems Research, 25, 761e778.
absence of strong software. Regardless, much of the “received
Antonakis, J., Bendahan, S., Jacquart, P., & Lalive, R. (2010). On making causal claims:
wisdom” on SEM has a limited evidentiary base for support. More A review and recommendations. The Leadership Quarterly, 21, 1086e1120.
than that, the emergence of SEM invited researchers to think in Becker, J.-M., Rai, A., & Rigdon, E. E. (2013). Predictive validity and formative
new ways, at new levels of abstraction. Analogies and heuristics are measurement in structural equation modeling: Embracing practical relevance.
In Thirty fourth international conference on information systems (ICIS), Milan.
powerful tools in helping people to come to grips with things that Bentler, P. M. (1986). Structural modeling and Psychometrika: An historical
are new and difficult to understand. While useful, however, heu- perspective on growth and achievements. Psychometrika, 51, 35e51.
ristics can also be misleading. The more fundamental technology of Bollen, K. A. (1989). Structural equations with latent variables. New York: John Wiley
and Sons.
null hypothesis significance testing is itself widely misunderstood, Bollen, K. A., & Bauldry, S. (2011). Three Cs in measurement models: Causal in-
with textbooks enshrining mistakes as basic principles and dicators, composite indicators, and covariates. Psychological Methods, 16,
infecting whole generations of researchers with falsehoods and 265e284.
Browne, M. W. (1984). Asymptotically distribution-free methods for analysis of
confusion (Gigerenzer, 2004; Ziliak & McCloskey, 2008). Regarding covariance structures. British Journal of Mathematical and Statistical Psychology,
factor analysis, Stewart (1981, p. 51) noted a similar state: “So 37, 62e83.
widespread are current misconceptions about factor analysis in the Chakravartty, A. (2007). A metaphysics for scientific realism. Cambridge, UK: Cam-
bridge University.
marketing community that even its defenders and some prominent
Chang, H. (2009). Operationalism. In E. N. Zalta (Ed.), Stanford encyclopedia of phi-
reviewers perpetuate misinformation.” The same might be said of losophy. http://plato.stanford.edu/archives/fall2009/entries/operationalism/.
both factor-based and composite-based approaches to SEM today. Cliff, N. (1983). Some cautions concerning the application of causal modeling
methods. Multivariate Behavioral Research, 18, 115e126.
Saunders and Bezzina's (2015) “methodological tribalism” appears
Creath, R. (2014). Logical empiricism. In E. N. Zalta (Ed.), Stanford encyclopedia of
to separate quantitative methods communities just as much as it philosophy. http://plato.stanford.edu/archives/spr2014/entries/logical-
separates quantitative from qualitative. empiricism/.
It should be clear to European management researchers that PLS Dana, J., & Dawes, R. M. (2004). The superiority of simple alternatives to regression
for social science predictions. Journal of Educational and Behavioral Statistics, 29,
path modeling is not a panacea for flaws in research design or 317e331.
execution. It does not multiply a small sample size into a large one. Diamantopoulos, A., & Winklhofer, H. M. (2001). Index construction with formative
It does not transform a poorly conceived approach into a piercing, indicators: An alternative to scale development. Journal of Marketing Research,
38, 269e277.
insightful analysis. At the same time, PLS path modeling is not a Dijkstra, T. K. (2010). Latent variables and indices: Herman wold's basic design and
flawed analytical method. It may be misunderstood, but probably is partial least squares. In V. Esposito Vinzi, W. W. Chin, J. Henseler, & H. Wang
no more so than the factor-based approach to SEM, or any other (Eds.), Handbook of partial least squares: Concepts, methods and applications (pp.
23e46). Berlin: Springer-Verlag.
sophisticated data analysis technique. Edwards, J. R. (2011). The fallacy of formative measurement. Organizational Research
So, if PLS path modeling is a valid tool for structural equation Methods, 14, 370e388.
modeling, when should researchers choose this composite-based Finney, S. J., & DiStefano, C. (2013). Nonnormal and categorical data in structural
equation modeling. In G. R. Hancock, & R. O. Mueller (Eds.), Structural equation
approach, and when should researchers choose the factor-based modeling: A second course (pp. 439e492). Charlotte, NC: Information Age.
approach? Answers to this question in the literature tend to Gigerenzer, G. (2004). Mindless statistics. Journal of Socio-Economics, 33, 587e606.
retreat to the same false beliefs and invalid arguments rejected Guide, V. D. R., & Ketokivi, M. (2015). Notes from the editors: Redefining some
methodological criteria for the journal. Journal of Operations Management, 37,
here. There may, indeed, be some set of specific circumstances
veviii.
where one approach or the other has a predictable advantage, but Guttman, L. (1955). The determinacy of factor score matrices with implications for
that will require future research. Comparisons of different ap- five other basic problems of common-factor theory. British Journal of Statistical
proaches should be based not on the particulars of the statistical Psychology, 8, 65e81.
Haig, B. D., & Evers, C. W. (2016). Realist inquiry in social science. London: Sage.
models, which are different, but on their common outcome, proxies Hair, J. F., Jr., Hult, G. T. M., Ringle, C. M., & Sarstedt, M. (2014). A primer on partial
for conceptual variables. When researchers have the ability to least squares structural equation modeling (PLS-SEM). Thousand Oaks, CA: Sage.

Please cite this article in press as: Rigdon, E. E., Choosing PLS path modeling as analytical method in European management research: A realist
perspective, European Management Journal (2016), http://dx.doi.org/10.1016/j.emj.2016.05.006
8 E.E. Rigdon / European Management Journal xxx (2016) 1e8

Hu, L.-t., & Bentler, P. M. (1998). Fit indices in covariance structure modeling: Aguirre-Urreta and Marakas. Information Systems Research, 25, 780e784.
Sensitivity to underparameterized model misspecification. Psychological €nkko
Ro € , M., & Evermann, J. (2013). A critical examination of common beliefs about
Methods, 3, 424e453. partial least squares path modeling. Organizational Research Methods, 16,
Ioannidis, J. P. A. (2005). Why most published research findings are false. PLoS 425e448.
Medicine, 2(8), e124. http://journals.plos.org/plosmedicine/article?id¼10.1371/ €nkko
Ro € , M., McIntosh, C. M., & Antonakis, J. (2015). On the adoption of partial least
journal.pmed.0020124. squares in psychological research: Caveat emptor. Personality and Individual
€reskog, K. G. (1969). A general approach to confirmatory maximum likelihood
Jo Differences, 87, 76e84.
factor analysis. Psychometrika, 34, 183e201. Royall, R. M. (1970). On finite population sampling theory under certain linear
€reskog, K. G. (1983). Factor analysis as an errors-in-variables model. In H. Wainer,
Jo regression models. Biometrika, 57, 377e387.
& S. Messick (Eds.), Principles of modern psychological measurement: A festschrift Saunders, M. N. K., & Bezzina, F. (2015). Reflections on conceptions of research
for Frederick M. Lord (pp. 185e196). Hillsdale, NJ: Lawrence Erlbaum. methodology among management academics. European Management Journal,
Lei, P.-W., & Wu, Q. (2012). Estimation in structural equation modeling. In 33, 297e304.
R. H. Hoyle (Ed.), Handbook of structural equation modeling (pp. 164e179). New Scho€nemann, P. H., & Steiger, J. H. (1976). Regression component analysis. British
York: Guilford. Journal of Mathematical and Statistical Psychology, 29, 175e189.
Leplin, J. (1984). Scientific realism. Berkeley, CA: University of California. Steiger, J. H. (1979). The relationship between external variables and common
van der Maas, H. L. J., Dolan, C. V., Grasman, R. P. P. P., Wicherts, J. M., factors. Psychometrika, 44, 93e97.
Huizenga, H. M., & Raijmakers, M. E. J. (2006). A dynamic model of general Stewart, D. W. (1981). The application and misapplication of factor analysis in
intelligence: The positive manifold of intelligence by mutualism. Psychological marketing research. Journal of Marketing Research, 18, 51e62.
Review, 113, 842e861. Trafimow, D., & Marks, M. (2015). Editorial. Basic and Applied Social Psychology, 37,
van der Maas, H. L. J., Kan, K.-J., & Borsboom, D. (2014). Intelligence is what the 1e2.
intelligence test measures. Seriously. Journal of Intelligence, 2, 12e15. Waller, N., & Jones, J. (2010). Correlation weights in multiple regression. Psycho-
MacCallum, R. C., Browne, M. W., & Cai, L. (2007). Factor analysis models as ap- metrika, 75, 58e69.
proximations. In R. Cudeck, & R. C. MacCallum (Eds.), Factor analysis at 100: Wasserstein, R. L., & Lazar, N. A. (2016). The ASA's statement on p-values: Context,
Historical developments and future directions (pp. 153e175). Mahwah, NJ: Law- process and purpose. The American Statistician. http://dx.doi.org/10.1080/
rence Erlbaum. 00031305.2016.1154108.
Maxwell, S. E. (2000). Sample size and multiple regression analysis. Psychological Wickens, M. R. (1972). A note on the use of proxy variables. Econometrica, 40,
Methods, 5, 434e458. 759e761.
McDonald, R. P. (1996). Path analysis with composite variables. Multivariate Wilson, E. B. (1928). Review of ‘the abilities of man, their nature and measurement’.
Behavioral Research, 31, 239e270. Science, 67, 244e248.
Michell, J. (2013). Constructs, inferences and mental measurement. New Ideas in Wold, H. O. (1982). Soft modeling: The basic design and some extensions. In
Psychology, 31, 13e21. K. G. Jo€reskog, & H. O. Wold (Eds.), Systems under indirect observation: Causality,
Mulaik, S. A. (2010). Foundations of factor analysis (2nd ed.). Boca Raton, FL: structure, prediction (Part II) (pp. 1e54). Amsterdam: North-Holland.
Chapman & Hall/CRC. Wold, H. O. (1985). Partial least squares. In S. Kotz, & N. L. Johnson (Eds.), Ency-
Peng, D. X., & Lai, F. (2012). Using partial least squares in operations management clopedia of statistical sciences (vol. 6, pp. 581e591). New York: Wiley.
research: A practical guideline and summary of past research. Journal of Oper- Wold, H. O. (1988). Predictor specification. In S. Kotz, & N. L. Johnson (Eds.), Ency-
ations Management, 30, 467e480. clopedia of statistical sciences (vol. 8, pp. 587e599). New York: Wiley.
Rigdon, E. E. (1994). Demonstrating the effects of unmodeled random measurement Woolridge, J. M. (2009). On estimating firm-level production functions using proxy
error. Structural Equation Modeling, 1, 375e380. variables to control for unobservables. Economic Letters, 104, 112e114.
Rigdon, E. E. (2012). Rethinking partial least squares path modeling: In praise of Yuan, K.-H., & Bentler, P. M. (1998). Normal theory based test statistics in structural
simple methods. Long Range Planning, 45, 341e358. equation modeling. British Journal of Mathematical and Statistical Psychology, 51,
Rigdon, E. E. (2013). Partial least squares path modeling. In G. R. Hancock, & 289e309.
R. O. Mueller (Eds.), Structural equation modeling: A second course (2nd ed., pp. Ziliak, S. T., & McCloskey, D. N. (2008). The cult of statistical significance: How the
81e116). Charlotte, NC: Information Age. standard error costs us jobs, justice and lives. Ann Arbor, MI: University of
Rigdon, E. E., Becker, J.-M., Rai, A., Ringle, C. M., Diamantopoulos, A., Karahanna, E., Michigan.
et al. (2014). Conflating antecedents and formative indicators: A comment on

Please cite this article in press as: Rigdon, E. E., Choosing PLS path modeling as analytical method in European management research: A realist
perspective, European Management Journal (2016), http://dx.doi.org/10.1016/j.emj.2016.05.006