0% found this document useful (0 votes)
7 views

The_General_Method_of_Theory-Building_Research_in_

The document discusses the challenges and methodologies of theory-building research in applied disciplines, particularly human resource development (HRD). It emphasizes the importance of making the logic of theory-building explicit and accessible, proposing a five-phase method for this process. The author argues that good theory should be practical and relevant, serving to explain and improve actions in real-world contexts.
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
0% found this document useful (0 votes)
7 views

The_General_Method_of_Theory-Building_Research_in_

The document discusses the challenges and methodologies of theory-building research in applied disciplines, particularly human resource development (HRD). It emphasizes the importance of making the logic of theory-building explicit and accessible, proposing a five-phase method for this process. The author argues that good theory should be practical and relevant, serving to explain and improve actions in real-world contexts.
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
You are on page 1/ 23

See discussions, stats, and author profiles for this publication at: https://www.researchgate.

net/publication/249631291

The General Method of Theory-Building Research in Applied Disciplines

Article in Advances in Developing Human Resources · August 2002


DOI: 10.1177/1523422302043002

CITATIONS READS
380 27,261

1 author:

Susan Alma Lynham


Colorado State University
35 PUBLICATIONS 2,066 CITATIONS

SEE PROFILE

All content following this page was uploaded by Susan Alma Lynham on 18 March 2016.

The user has requested enhancement of the downloaded file.


Advances in Developing Human Resources
http://adh.sagepub.com

The General Method of Theory-Building Research in Applied Disciplines


Susan A. Lynham
Advances in Developing Human Resources 2002; 4; 221
DOI: 10.1177/1523422302043002

The online version of this article can be found at:


http://adh.sagepub.com/cgi/content/abstract/4/3/221

Published by:

http://www.sagepublications.com

On behalf of:

Academy of Human Resource Development

Additional services and information for Advances in Developing Human Resources can be found at:

Email Alerts: http://adh.sagepub.com/cgi/alerts

Subscriptions: http://adh.sagepub.com/subscriptions

Reprints: http://www.sagepub.com/journalsReprints.nav

Permissions: http://www.sagepub.com/journalsPermissions.nav

Citations http://adh.sagepub.com/cgi/content/refs/4/3/221

Downloaded from http://adh.sagepub.com at Tartu University Library on October 15, 2009


䉱 Chapter 1
Advances in Developing Human Resources August 2002

The General Method of


Theory-Building Research
in Applied Disciplines
Susan A. Lynham

The problem and the solution. One of the challenges of


theory-building research in applied disciplines is making the
logic used to build the theory explicit and accessible to the user
of the developed theory. Although different methods of theory
building advocate different theory-building research processes,
there is an inherently generic nature to theory building. This
chapter acts as a foundation for the journal by highlighting strat-
egies commonly used in building theory and offers a generic,
five-phase method of theory-building research.

I passionately believe in the need for and utility of good theory. As a result,
one familiar statement that dumbfounds me is, “Well, that is all very well in
theory, but it does not work like that in practice or in the real world.” State-
ments of this nature are informed by a number of deeply held, and generally
erroneous, assumptions about the nature and utility of theory. Some of these
false assumptions include the following:

• that theory is disconnected and removed from practice,


• that the process of theory construction happens in isolation of the real
world,
• that those who engage in theory building or development are not the
same as those who engage in practice or in the real world, and
• that usefulness and application are optional outcomes of theory.

What is the purpose of good theory other than to describe and explain how
things actually work and, in so doing, to help us improve our actions in this
world? Some will contend that theory is largely idealistic (Kaplan, 1964). How-
OD

The outcomes of this chapter were informed and improved on through the generous and support-
ive guidance, editing, and other helpful suggestions offered by a number of people. In particular, my
sincere thanks to Dr. Richard A. Swanson, Dr. Yvonna S. Lincoln, Dr. David A. Erlandson, Dr. Jean
Madsen, students in the fall 2001 and spring 2002 advanced human resource development theory
course at the University of Minnesota, and participants in the 2002 Academy of Human Resource
Development theory-building preconference.
Advances in Developing Human Resources Vol. 4, No. 3 August 2002 221-241
Copyright 2002 Sage Publications

Downloaded from http://adh.sagepub.com at Tartu University Library on October 15, 2009


222 Advances in Developing Human Resources August 2002

ever, it can just as easily be argued that good theory in applied disciplines is
about as realistic as it comes (Dubin, 1978; Kaplan, 1964; Lewin, 1951;
Lynham, 2000b; Swanson, 1997; Van de Ven, 1989). Think about it: How many
theories do you hold about the world around you and how that world works?
How do these theories inform you of what things work, and do not work, in day-
to-day actions? Every time we encounter a new issue, we first experience it, and
then we try to observe and understand how that issue presents itself and works.
Next, we begin to develop a system of ideas, informed from our experience and
knowledge of the world and the issue, about how to address the issue. Then, we
put those ideas to the test by applying them to the issue. If these ideas work, then
the issue or problem gets satisfactorily addressed. If not, we go back to our own
internal drawing boards and begin the process of problem-solution formulation
and application all over again. In effect, what we are continuously doing is
developing informed knowledge frameworks about how to act on things in our
world, thereby formulating ways in which to understand and address issues and
problems in the world around us (Alvesson & Deetz, 2000). These informed
knowledge and experience frameworks that we apply to our world are simply
personal theories-in-use (Argyris & Schon, 1974, 1996). Think about them as
theories-in-practice. Each of our lives is informed by many theories-in-practice.
They are put into practice or use precisely because they help us to understand,
explain, anticipate, know, and act in the world in better and more informed ways,
and to better ends and outcomes. Theories therefore have a very practical role in
our everyday lives.
Sure, we can hold and develop grandiose and idealistic theories of how
the world might be and work. Argyris and Schon (1996) called these idealis-
tic, speculative conceptions of espoused theories. However, espoused and
unconfirmed theories of the world and phenomena within the world are less
of what we are interested in as applied theorists and cannot be classified as
real theory. In an applied discipline such as human resource development
(HRD), theory is required to be of practical value (Kaplan, 1964; Lynham,
2000b; Mott, 1996; Swanson, 1997, 1999; Van de Ven, 1989). By virtue of
its application nature, good theory is of value precisely because it fulfills
one primary purpose. That purpose is to explain the meaning, nature, and
challenges of a phenomenon, often experienced but unexplained in the
world in which we live, so that we may use that knowledge and understand-
ing to act in more informed and effective ways (Campbell, 1990; Lewin,
1951; Strauss & Corbin, 1990; Van de Ven, 1989; Whetten, 1989).
Theory is described as “a coherent description, explanation and represen-
tation of observed or experienced phenomena” (Gioia & Pitre, 1990, p. 587).
Theory building is the ongoing process of producing, confirming, applying,
and adapting theory (Lynham, 2000b). In a way, to live life successfully we
are all obliged to engage in theory building, that is, in processes by which we
observe, experience, think about, and understand and act in our worlds, and

Downloaded from http://adh.sagepub.com at Tartu University Library on October 15, 2009


Lynham / THE GENERAL METHOD 223

we do so continuously. However, these theories-in-practice are not always


explicit and often occur in the form of implicit, unconscious knowledge on
the part of the theorist. As such, these theories that we put into use in our
daily lives are no more, or less, than personal theories-in-practice and are
seldom made explicit by the holder and user of those theories. For example,
how many times has a parent or trusted friend given you advice about what
works and what does not, about what you should or should not do about
something, but when questioned about what he or she actually knows and
how it all works, you get the response: “I just know; trust me, I have had lots
of experience with this.”
As the recipients of such personal theories-in-practice, we are faced with
two choices. The first is one of a leap of faith—to apply the advice given and
hope that it will have the same results for you as it did for the advisor. The
second is the choice of inquiry and discovery—to develop our own explana-
tions for the issue at hand and how to deal with it. If both are pursued on only
a personal front, then it is unlikely that the wisdom of either will be transmit-
ted to anyone else. And next time we are asked the same question by some-
one facing a similar issue, our response is likely to mimic that of our original
advisor: “I just know; trust me.” The point here is that an important function
and characteristic of theory building is to make these explanations and
understandings of how the world is and works explicit and, by so doing, to
make transferable, informed knowledge for improved understanding and
action in the world tacit rather than implicit.
Theory building can be described as “the purposeful process or recurring
cycle by which coherent descriptions, explanations, and representations of
observed or experienced phenomena are generated, verified, and refined”
(Lynham, 2000b, p. 161). Good theory building should result in two kinds of
knowledge: outcome knowledge, usually in the form of explanative and pre-
dictive knowledge, and process knowledge, for example, in the form of
increased understanding of how something works and what it means
(Dubin, 1976). Good theory and theory building should also reflect two
important qualities: rigor and relevance (Marsick, 1990a), or what are also
termed validity and utility (Van de Ven, 1989). Theory building achieves
these two desired knowledge outputs and empirical qualities by use of what
Kaplan (1964) called “the logic-in-use” and the “reconstructed logic” (p. 8),
that is, by following a logical cognitive style in the development and appli-
cation of the theory and by explicitly reconstructing, or making explicit, that
logic-in-use.
It is the purpose of this monograph to present multiple possible methods,
or logics-in-use, for generating, confirming, and refining theory in HRD and
other applied disciplines. It is intended that these explicit representations
and descriptions of theory building will be useful to practitioners, research-
ers, and educators in learning about, engaging in, and evaluating the traits

Downloaded from http://adh.sagepub.com at Tartu University Library on October 15, 2009


224 Advances in Developing Human Resources August 2002

and outcomes of HRD and other applied theory-building endeavors. It is the


aim of this first chapter to provide a contextual overview and reconstruction
of the general logic-in-use embedded in the nature and challenges of the
journey of theory building. Specifically, this chapter first presents some
considerations common to theory-building inquiry in applied disciplines.
Second, it describes theory building as a five-phase, general, and recursive
process. Third, it briefly highlights why theory-building research is impor-
tant to the HRD profession, together with some of the challenges associated
with building applied theory. Finally, it offers concluding comments on
some of the key points raised in the chapter.

General Considerations of
Theory-Building Research
Before considering the generic methodological components of theory
building, it might be helpful to highlight and discuss considerations general
to theory-building research. The first is the notion of the multiple purposes
of theory-building research methods. Second is a brief presentation and
description of two commonly used strategies in theory building. And finally,
consideration is given to the requirement of expertise in both knowledge of
and experience with the phenomenon that is the focus of the theory-building
endeavor.

The Multiple Purposes of


Theory-Building Inquiry

Theory-building research is a method of scholarly inquiry (Gall, Borg, &


Gall, 1996; Kaplan, 1964; Swanson, 1997). Just like any other form of schol-
arly inquiry, theory building can involve varied and various logics-in-use
and can be engaged in from multiple research paradigms (Kaplan, 1964).
There is no one supreme method of theory building, and nor should there be
(Gioia & Pitre, 1990; Kuhn, 1970; Lynham, 2000b; Marsick, 1990b;
Swanson, 1997; Swanson, Lynham, Ruona, & Torraco, 2000; Thomas,
1997). Rather, the specific theory-building research method employed
should be dictated by the nature of the theory building being engaged in, and
not by the preferred inquiry methodology of the researcher-theorist or the
practitioner-theorist. It is therefore less important that we support one spe-
cific theory-building research method over another than that we view
applied theory-building research as a necessary and helpful form of schol-
arly inquiry in developing and expanding our understanding of and ability to
explain, anticipate, and act on related phenomena, issues, and problems.
Like any form of inquiry, theory-building research is used for numerous
purposes, and these intended purposes influence the nature and require-

Downloaded from http://adh.sagepub.com at Tartu University Library on October 15, 2009


Lynham / THE GENERAL METHOD 225

ments of the theory-building method employed. Habermas’s (Hultgren &


Coomer, 1989) three-perspective classification of scholarly inquiry is infor-
mative in considering the various purposes and nature of theory building in
HRD. He highlighted three broad modes of inquiry in the social or human
sciences, namely, empirical-analytical, interpretive, and critical science
research. When applied to theory building, this framework can be used to
provide a general, comparative overview of the contrasting empirical char-
acteristics of three dominant modes of theory-building inquiry or research
(McLean, 2001) (see Table 1).
As indicated earlier, and informed by Table 1, the question is not one of
whether we should engage in multimethod theory-building research in
HRD. Rather, it is one of when is theory-building research the most justifiable
means to address the phenomena or problem, and therefore, what theory-
building methods, or combinations thereof, are the most suited to the issue
under inquiry?

Two Common Strategies


Used in Theory Building

Because HRD is of an applied nature, theory-building methods must be


capable of dealing with issues of application (Campbell, 1990; Dubin, 1976,
1978; Lynham, 1998, 2000b; Swanson, 1988, 1997, 2000; Torraco, 1994,
1997, 2000). This monograph highlights and discusses a number of research
methods particularly well suited to and relevant for use in theory building in
HRD and other applied disciplines. Beyond these applied methods of theory
building, it is worth considering two strategies common to theory building
(Reynolds, 1971). The first is one of a research-to-theory strategy, whereas
the second is one of a theory-to-research strategy (Reynolds, 1971).
The research-to-theory strategy, also termed the research-then-theory strat-
egy, is related to “deriving the laws of nature from a careful examination of all
the available data” (Reynolds, 1971, p. 140). Francis Bacon referred to the out-
come of this theory-building strategy as interpretations of nature (Reynolds,
1971). As described by Reynolds (1971), the essences of this research-to-theory
strategy are as follows:

1. Select a phenomenon and list all the characteristics of the phenomenon,


2. measure all the characteristics of the phenomenon in a variety of situa-
tions (as many as possible),
3. analyze the resulting data carefully and determine if there are any sys-
tematic patterns among the data “worthy” of further attention, and
4. once significant patterns have been found in the data, formalization of
these patterns as theoretical statements constitutes the laws of nature
(axioms, in Bacon’s terminology). (p. 140)

Downloaded from http://adh.sagepub.com at Tartu University Library on October 15, 2009


226

TABLE 1: The Contrasting Features of Empirical-Analytical, Interpretive, and Critical Science Approaches to Theory-Building
Research
Downloaded from http://adh.sagepub.com at Tartu University Library on October 15, 2009

View of Area of Assumption Desired


Theory-Building Human Interest About Empirical HRD-Related
Inquiry and Application Knowledge Purpose Outcome

Empirical- • Work • Observational data are • To explain, predict, • Generalizable laws and
analytical • Technical, that is, considered the foundation and control explanations of organi-
about practice affected of knowledge zational and human
through newly developed • Generalizations character- behavior
means to achieve ized by empiricism
established ends
Interpretive • Interaction (language) • Constructed meanings of • To make sense of, • Common meanings and
• Practical, that is, about stakeholders are considered understand, and clarifying interpretations
policy and practice the foundation of knowledge interpret of organizational and
informed through human actions and
interpretations of daily experiences
events and contexts
Critical • Power (reason) • Constructed meanings of • To enlighten and • Underlying, hidden, or
• Emancipatory, that is, stakeholders are considered emancipate through unreflected choices
about policy and practice the foundation of knowledge the process of surfaced to inform
changed through critique • Critique of ideologies believed critique and reasoned human and
and recovering self- to promote needed social identifying potential organizational choice
reflection to unite theory change, which is open and
and practice ongoing
Note: HRD = human resource development.
Lynham / THE GENERAL METHOD 227

Also frequently referred to as the Baconian approach, this research-to-theory


theory-building strategy requires, according to Reynolds (1971), two important
conditions, namely, “a relatively small number of variables to measure during
data collection” and “that there be a few significant patterns to be found in the
data” (p. 140). The dominant ontology of this theory-building strategy is a quan-
titative one. As a result, the corresponding assumptions about knowledge (epis-
temology) that underlie and govern the research-to-theory strategy are also of a
quantitative nature (for example, that the real world is objective and external to
the researcher; that the truth is out there to be discovered through careful,
methodical, and comprehensive inquiry by the researcher; and that the purpose
of research is the discovery of universal, causal laws to enable causal explana-
tion). Of a predominantly deductive nature, this research-to-theory strategy is
thought to be well suited to the pure sciences, where the purpose of theory build-
ing is to develop large, generalizable laws of nature that explain how phenomena
in the natural, objective world within which we live can be expected to work and
potentially be predicted and controlled.
The second strategy for building theory is that of theory to research, or what
Reynolds (1971) called the “theory-then-research strategy” (p. 144). In this
approach to theory building, theory is made explicit through the continuous,
reiterative interaction between theory construction and empirical inquiry
(Kaplan, 1964; Reynolds, 1971). Reynolds highlighted the essences of this
theory-building strategy as follows:

1. Develop an explicit theory in either axiomatic or process description


form;
2. select a statement generated by the theory for comparison with the
results of empirical research;
3. design a research project to “test” the chosen statement’s correspon-
dence with empirical research;
4. if the statement derived from the theory does not correspond with the
research results, make appropriate changes in the theory or the research
design and continue with the research; and
5. if the statement from the theory corresponds with the results of the
research, select further statements for testing or attempt to determine the
limitations of the theory. (p. 144)

This theory-to-research strategy was made popular by Karl Popper, in which


“he suggests that scientific knowledge would advance most rapidly through the
development of new ideas [conjectures] and attempts to falsify them with empir-
ical research [refutations]” (Reynolds, 1971, p. 144). Often more inclusive of
qualitative research, this strategy is informed by corresponding assumptions
about the nature of scientific knowledge, for example, that there is no “real
world” or “one truth” but rather that knowledge about human behavior is created
in the minds of individuals, “that science is a process of inventing descriptions of
phenomena” (Reynolds, 1971, p. 145), that there are multiple and divergent real-
ities and therefore “truths,” and that the purpose of science is one of interpretive

Downloaded from http://adh.sagepub.com at Tartu University Library on October 15, 2009


228 Advances in Developing Human Resources August 2002

discovery and explanation of the nature and meaning of phenomena in the world
in which we live and experience life (Hultgren & Coomer, 1989). Of an interac-
tive inductive-deductive nature, this theory-to-research strategy is well suited to
the applied nature of the behavioral and human sciences (Lynham, 2000b;
Reynolds, 1971).
The significance of these two theory-building strategies lies not in the
need to choose one above the other. Rather, their value to the theorist is in the
insight that they provide regarding the virtuous, systemic nature of the inter-
action among three elements critical to applied theory building, namely, the
development and accumulation of a system of coherent, disciplined, and rig-
orous knowledge and explanation (theory); the conduct of focused and dis-
ciplined scholarly inquiry and discovery (research); and the resulting
informed and improved action that ensues from the application of the out-
comes of the first two elements in practice (practice). The concept of a virtu-
ous cycle (also noted by Egan in Chapter 3) is informed by systems theory
and refers to a positive, reinforcing relationship of interdependence among
the components of the system concerned (Kauffman, 1980; Senge, 1990;
Von Bertalanffy, 1968). This growth cycle of theory-research-practice (see
Figure 1) is fundamental to building rigorous and relevant applied theory
(Dubin, 1978). The expertise required for successful applied theory build-
ing must therefore relate to the virtuous nature of applied theory building
and will be discussed more in the next section of this chapter.

Toward a General Theory-


Building Research Method

The theory-to-research theory-building strategy, coupled with the


applied nature and emphasis of numerous theory-building methods
described in this monograph, demands that the theorist have expertise of
both the phenomenon central to the theory as well as of the theory-building
method itself (Campbell, 1990; Cohen, 1991; Dubin, 1976; Gioia & Pitre,
1990; Hearn, 1958; Patterson, 1986; Reynolds, 1971; Van de Ven, 1989).
Applied theory-building methods therefore require the theorist to interact
with and be influenced and informed by both her or his experience of the
phenomenon in practice and her or his acquired knowledge/mastery of the
phenomenon in theory. In this way, both knowledge of and knowledge about
the phenomenon central to the theory are brought together through the theory-
building process and are ordered according to the internal logic, or logic-in-
use, and informed imagination of the theorist (Cohen, 1991; Dubin, 1978;
Reynolds, 1971; Weick, 1995). This continuous and iterative conversation
in applied theory construction, between knowledge and experience of the
phenomenon that is the focus of the theory, facilitates the accumulation of
relevant and rigorous theoretical knowledge of the phenomenon in the expe-

Downloaded from http://adh.sagepub.com at Tartu University Library on October 15, 2009


Lynham / THE GENERAL METHOD 229

RESEARCH
Focused, disciplined
empirical inquiry THEORY
and discovery The development and
accumulation of a system of
Reinforcing coherent, disciplined and rigorous
knowledge and explanation

PRACTICE
Informed and
improved action

FIGURE 1: The Growth Cycle of Applied Theory-Building

rienced world and is the focus of the theory and the theory-building method
itself (see Figure 2).
Informed by Figures 1 and 2, a useful way of conceptualizing the research
method for applied theory building is as a recursive system of five distinct
phases:

• conceptual development,
• operationalization,
• application,
• confirmation or disconfirmation, and
• continuous refinement and development (of the theory).

This five-phase depiction of the method of applied theory building is presented


in Figure 3.
From an overall perspective, applied theory-building research consists of
two broad components, namely, theorizing to practice and practice to theo-
rizing. Each of these components produces distinct in-process outputs that
guide the applied theory-building research and, ultimately, result in a trust-
worthy, rigorous, and relevant theory for improved action (Denzin & Lincoln,
2000; Marsick, 1990a; Van de Ven, 1989). An essential output from the the-
orizing component of theory building is a coherent and informed theoretical
framework, which encapsulates and “contains” the explanation of the phe-
nomenon, issue, or problem that is the focus of the theory. Key outputs from

Downloaded from http://adh.sagepub.com at Tartu University Library on October 15, 2009


230 Advances in Developing Human Resources August 2002

Building applied theory requires two


kinds of expertise…

… Of the phenomenon
Knowledge ... Expertise Experience …
… Of the theory
building research
method being used
When one starts on the When one starts on the
theory this component is in theory this component is in
the background of the foreground of attention.
attention. It is then moved It is then moved to the
to the foreground of the background of the attention
attention of the researcher- of the researcher-theorist.
theorist.

FIGURE 2: The Recursive Nature of Practical and Theoretical Expertise Inherent in Applied
Theory-Building Research

the practice components of theory building are carefully obtained data/


findings and experiential knowledge that are used to confirm, or disconfirm,
and further refine and develop the existing theory and to enhance the utility
of the theory in practice. The five phases of the applied theory-building
research method take place within this larger two-component theory-building
frame indicated in Figure 3.
It is important to note that these five phases do not necessarily need to be
pursued in the order in which they appear in Figure 3. However, each com-
plete applied theory-building research effort, regardless of the specific
theory-building method employed by the researcher-theorist, involves, in some
form, these five general theory-building phases, namely, conceptual devel-
opment, operationalization, application, confirmation or disconfirmation,
and continuous refinement and development.
Furthermore, it is important to bear in mind that an applied theory is
never considered complete but rather “true until shown otherwise” (Cohen,
1991; Dubin, 1978; Kaplan, 1964; Reynolds, 1971; Root, 1993). As such,
the theory is always “in progress,” and further research related to the theory
is used to refine and increase confidence, or not, in the existing theory—
hence the nature of the cyclical phase of applied theory building, namely,
continuous refinement and development. Which phase is actually carried
out first in the theory-building process is dependent on the theory-building
method being employed by the researcher-theorist. The remaining chapters
in this monograph reflect this multiple entry point option that characterizes
theory building in applied disciplines.

Downloaded from http://adh.sagepub.com at Tartu University Library on October 15, 2009


Lynham / THE GENERAL METHOD 231

The environment in which we live, observe and experience the world.

Theorizing
to
Practice

DEDUCTIVE Conceptual Operationalization DEDUCTIVE


Development

Continuous
Refinement
and
Development

Confirmation or
Application Disconfirmation

INDUCTIVE INDUCTIVE
Practice
to
Theorizing

FIGURE 3: The General Method of Theory-Building Research in Applied Disciplines

The following sections provide a brief description of each of the five


phases of the general method of applied theory-building research as por-
trayed in Figure 3. It should be noted that the discussion sequences the
phases from the perspective of a theorizing-to-practice strategy of applied
theory-building research. Using a practice-to-theorizing strategy of theory
building does not change the occurrence of these five phases but rather what
makes for the appropriate sequencing of each phase in the applied theory-
building research process.

Conceptual Development

Conceptual development requires that the theorist formulate initial ideas


in a way that depicts current, best, most informed understanding and expla-
nation of the phenomenon, issue, or problem in the relevant world context
(Dubin, 1978; Lynham, 2000b). The purpose of this phase is therefore to
develop an informed conceptual framework that provides an initial under-
standing and explanation of the nature and dynamics of the issue, problem,
or phenomenon that is the focus of the theory.

Downloaded from http://adh.sagepub.com at Tartu University Library on October 15, 2009


232 Advances in Developing Human Resources August 2002

The process of conceptual development varies according to the theory-


building method employed by the theorist. However, at a minimum this pro-
cess will include the development of the key elements of the theory, an ini-
tial explanation of their interdependence, and the general limitations and
conditions under which the theoretical framework can be expected to oper-
ate. The output of this phase is an explicit, informed, conceptual framework
that often takes the form of a model and/or metaphor that is developed from
the theorist’s knowledge of and experience with the phenomenon, issue, or
problem concerned (Dubin, 1978; Kaplan, 1964).
The phase of conceptual development is one of two phases that dominate
the theorizing component of theory-building research. Here, the theorist con-
ducts theoretical inquiry into the phenomenon, issue, or problem core to the
theory. Starting the journey at this point is often more typical of quantitative-
type (or experimental) theory-building research methods, for example, the
hypothetico-deductive method and meta-analysis (Cohen, 1991; Dubin,
1976, 1978; Hearn, 1958; Kaplan, 1964; Patterson, 1986; Reynolds, 1971).
More qualitatively oriented theory-building research methods, for example,
case study, grounded theory, and social constructivist approaches, typically
begin with inquiry in the application phase and then use the results of such
inquiry to inform the development of the conceptual framework of the the-
ory (Eisenhardt, 1989, 1995; Stake, 1994; Strauss & Corbin, 1998). Regard-
less of the sequencing of the conceptual development phase of theory build-
ing, the development of an informed conceptual framework is fundamental
to all theory-building research. This theoretical framework is essentially the
core explanatory container of any theory.

Operationalization

The purpose of the operationalization phase of theory-building research


is essentially an explicit connection between the conceptualization phase
and practice. The operationalization of a theory needs to be confirmed and/
or tested in its real-world context. In order for the theoretical framework to
evoke trust and confidence, the initial explanation of the phenomenon, prob-
lem, or issue embedded in the framework must be applied to and empirically
confirmed in the world in which the phenomenon, issue, or problem occurs.
To achieve this necessary confirmation, the theoretical framework must be
translated, or converted, to observable, confirmable components/elements.
These components/elements can be in the form of, for example, confirmable
propositions, hypotheses, empirical indicators, and/or so-called knowledge
claims (Cohen, 1991), and are addressed through appropriate inquiry meth-
ods, depending on the theory-building method being employed by the
theorist.

Downloaded from http://adh.sagepub.com at Tartu University Library on October 15, 2009


Lynham / THE GENERAL METHOD 233

Operationalization reaches toward an overlap between the theorizing and


practice components of the theory-building research process. A primary
output of the theorizing component of theory-building research in applied
disciplines is therefore an operationalized theoretical framework, that is, an
informed theoretical framework that has been converted into components or
elements that can be further inquired into and confirmed through rigorous
research and relevant application.

Confirmation or Disconfirmation

The confirmation or disconfirmation phase falls within the practice com-


ponent of applied theory building. This theory-building phase involves the
planning, design, implementation, and evaluation of an appropriate
research agenda and studies to purposefully inform and intentionally con-
firm or disconfirm the theoretical framework central to the theory. When
adequately addressed, this third phase results in a confirmed and trustwor-
thy theory that can then be used with some confidence to inform better
action and practice.

Application

A theory that has been confirmed in the contextual world to which it


applies (i.e., operationalized) and has, at least to some extent, gone through
inquiry in the practical world is not enough. A theory must also be threaded
through the application phase. The application of the theory to the problem,
phenomenon, or issue in the world of practice is in the practice component
of the general theory-building research method. Application of the theory
enables further study, inquiry, and understanding of the theory in action.
An important outcome of this application phase of theory building is
therefore that it enables the theorist to use the experience and learning from
the real-world application of the theory to further inform, develop, and
refine the theory. It is in the application of a theory that practice gets to judge
and inform the usefulness and relevance of the theory for improved action
and problem solving (Lynham, 2000b). And it is through this application
that the practical world becomes an essential source of knowledge and expe-
rience for ongoing development of applied theory (Ruona & Lynham, 1999;
Swanson, 1997).

Ongoing Refinement and Development

Because a theory is never “complete,” it is necessary that the theory be


continually refined and developed (Cohen, 1991; Root, 1993). This recur-

Downloaded from http://adh.sagepub.com at Tartu University Library on October 15, 2009


234 Advances in Developing Human Resources August 2002

sive nature of applied theory-building research requires the ongoing study,


adaptation, development, and improvement of the theory in action and
ensures that the relevance and rigor of the theory are continuously attended
to and improved on by theorists through further inquiry and application in
the real world. This continuing phase marks a further overlap between the
practice and theorizing components of applied theory-building research.
This phase addresses the responsibility of continuous attention to the trust-
worthiness and substantive quality of the theory that is the burden of the the-
orist (Dubin, 1978; Van de Ven, 1989). The intentional outcome of this
phase is thus to ensure that the theory is kept current and relevant and that it
continues to work and have utility in the practical world. It also ensures that
when the theory is no longer useful, or is found to be “false,” that it is shown
to be as such and adapted or discarded accordingly.

Limitations of the General


Method of Theory Building
Like all multidimensional models presented in a two-dimensional media,
this five-phase method of theory-building research in applied disciplines is
much less programmatic than is apparent in Figure 3. These phases of
applied theory building are not so much linear as they are necessary. The
process of applied theory building can begin with any one of these phases
and progress in a much less orderly way than this model might suggest.
Where one begins and ends with applied theory-building research is less rel-
evant than the acknowledgment that all of the five phases presented in the
method are necessary and required for the outcome of a relevant, useful, and
trustworthy research-based theory.
Each specific theory-building research method in the following chapters
has its own distinctive characteristics, process, and requirements. However,
common to all these methods are the five phases of the general method of
theory-building research. It is believed that this five-phase method and con-
ceptualization of applied theory building can be used as the generic and
informative organizer and guide to those involved in theory building in HRD
and applied disciplines. It can also be used as a means to compare and con-
trast specific research methods and their contributions to the general method
(as indicated in Table 2 of chapter 7). This general method of applied theory-
building research can further be used to serve and address the acknowledged
importance and challenges of theory building in HRD and other applied
disciplines.

Downloaded from http://adh.sagepub.com at Tartu University Library on October 15, 2009


Lynham / THE GENERAL METHOD 235

The Importance and Challenges


of Theory Building in HRD
The past decade, and particularly the past few years, has seen increased
recognition by HRD scholars of the importance of theory building to the
profession (Chalofsky, 1996, 1998; Gradous, 1989; Hansen, 1998; Hatcher,
1999; Lynham, 2000a, 2000b; Marsick, 1990a, 1990b; Mott, 1996, 1998;
Passmore, 1990; Shindell, 1999; Swanson, 1997, 1999, 2000; Swanson &
Holton, 2001; Torraco, 1997, 1999). Three compelling such points of pro-
fessional import are offered by Lynham (2000b), namely, that theory build-
ing can play an important role in advancing professionalism and maturity in
the field, that theory building can help to dissolve the tension between
research and practice in HRD, and that theory building can enable the devel-
opment of multiple and inclusive methods of research for HRD theory and
practice.
Torraco (1997), a notable scholar of theory building in HRD, provided fur-
ther support to the importance of theory building to the profession. He high-
lighted nine “prominent roles served by theory . . . in the context of human
resource development” (p. 117), namely,

• “a means by which new research data can be interpreted and coded


for future use,”
• “a means for responding to new problems that have no previously
identified solutions strategy,”
• “a means for identifying and defining applied problems,”
• “a means for prescribing or evaluating solutions to applied
problems,”
• a way of telling “us that certain facts among the accumulated knowl-
edge are important and others are not,”
• a means of giving “old data new interpretations and new meaning,”
• a means by which to identify “important new issues and prescribe the
most critical research questions that need to be answered to maxi-
mize understanding of the issue,”
• a means of providing “members of a professional discipline with a
common language and a frame of reference for defining boundaries
of their profession,” and
• a means “to guide and inform research so that it can, in turn, guide devel-
opment efforts and improve professional practice.” (pp. 117-119)

Theory, and by association theory building, therefore acts to improve and


protect HRD research and practice and does so by providing a means of rigor and

Downloaded from http://adh.sagepub.com at Tartu University Library on October 15, 2009


236 Advances in Developing Human Resources August 2002

relevance for reducing both atheoretical practice (Swanson, 1997) and


nonscientific research (Lynham, 2000b). Having recognized the importance of
theory building to the profession, it is, however, necessary to recognize that the
task of theory building in HRD and other applied disciplines does not come
without certain challenges (Dubin, 1978; Gioia & Pitre, 1990; Glaser & Strauss,
1967; Hansen, 1998; Jacobs, 1997, 1999; Klein, Tosi, & Cannella, 1999; Kuhn,
1970; Lynham, 2000b; Marsick, 1990a, 1990b; Morgeson & Hofmann, 1999;
Mott, 1998; Torraco, 1997; Van de Ven, 1989). The first of these challenges is
that of having to deal with the pressure that theory building puts on the rela-
tionship between the researcher and the practitioner and the second is of the need
to recognize that the outcomes of theory-building research are enriched by
building theory from multiple research perspectives and methods (Lynham,
2000b).

Conclusions
This chapter attempted to present an overview of the general method of
theory-building research in applied disciplines. Specifically, it presented
some considerations general to theory building as well as a framework of
five core phases of the general theory-building research process. The fol-
lowing chapters present specific methods of theory building considered to
be particularly well suited to applied disciplines.
A common myth associated with theory is that theory is all good and well,
but it seldom can be expected to work in the real world. It has been recog-
nized that in an applied field like that of HRD, theory is good precisely
because of its utility in practice. No one underscores the utility of good the-
ory more than Lewin (1945, 1951), who long since coined the notion that
there is nothing quite as practical as good theory. This utility-relevance
requirement of theory in an applied field has been increasingly echoed by
HRD and related scholars of theory building.
Although relevance-utility is seen as a necessary condition of HRD the-
ory, it is also agreed that good applied theory must be extended to include the
conditions of empirical rigor and trustworthiness. It is this dual condition of
what Marsick (1990a) referred to as rigor-relevance that makes theory
building useful in reducing the occurrence of atheoretical practice
(Swanson, 1997) and related nonscientific inquiry (Lynham, 2000b).
Another misconception commonly associated with theory building is not
only that the task of this empirical endeavor is primarily the responsibility of
the HRD academic-researcher but that the origins of theory come essentially
from research. Swanson (1997) allayed this concern, however, and provided
us with clear logic and evidence of the multiple practice-development-
research origins of theory and the corresponding researcher-practitioner
nature of theory builders.

Downloaded from http://adh.sagepub.com at Tartu University Library on October 15, 2009


Lynham / THE GENERAL METHOD 237

What does appear to be common to theory-building research in applied


disciplines, regardless of the origins and interest of the theory builders, is
the virtuous, systemic nature of the relationship between HRD theory,
inquiry, and practice. This systemic nature of applied theory building is
fundamental to understanding and being able to participate in the general
process of theory building and can be framed by way of five interdepen-
dent, interacting phases of theory building, namely, conceptual develop-
ment, operationalization, confirmation, application, and continuous refine-
ment and development of the theory. This generic applied framework of
theory-building research is further useful in that it informs and makes the
logic-in-use embedded in multiple research methods of theory building
explicit and, in turn, accessible to the HRD theorist. This framework also
helps to address one of the current difficulties of theory building in HRD and
other applied disciplines, namely, the generally perceived inaccessibility
and often academically viewed nature of theory-building methods—a com-
mon deterrent to the aspiring practitioner-theorist.
It is further evident that theory-building research methods are of a duo
deductive-inductive nature. Although some theory-building research meth-
ods may begin with deduction, at some point they become informed by
induction. With other theory-building methods, the relationship between
deduction and induction may be the other way around. What is important in
theory building inquiry, whether one starts with theory and then moves to
research and/or application, or vice versa, is that the choice of specific theory-
building research methods should be based on the nature of the phenome-
non, issue, or problem that is the focus of the theory-building endeavor, and
not by the theorist’s preferred specific method of theory building. It is also
increasingly acknowledged that multiple methods of theory building can
and should be used to develop theory in fields of the applied nature of HRD.
Just as each specific method of applied theory building is a way of develop-
ing insight, understanding, and possible explanation of the phenomenon,
issue, or problem, so it is a way of not doing so (Passmore, 1997).
By developing integrated, inclusive, and multiple-methods perspective
and approaches to building theory in HRD, the profession has a better
chance that the resulting theories will reflect the rigor-relevance character-
istic of good applied theory. In turn, these theories are likely to result in
better outcomes and understanding for improved HRD research, practice,
and education.

References
Alvesson, M., & Deetz, S. (2000). Doing critical management research. London: Sage.
Argyris, C., & Schon, D. A. (1974). Theory in practice: Increasing professional effective-
ness. San Francisco: Jossey-Bass.

Downloaded from http://adh.sagepub.com at Tartu University Library on October 15, 2009


238 Advances in Developing Human Resources August 2002

Argyris, C., & Schon, D. A. (1996). Organizational learning II. Theory, method, and
practice. New York: Addison-Wesley.
Campbell, J. P. (1990). The role of theory in industrial and organizational psychology. In
M. D. Dunnette & L. M. Hough (Eds.), Handbook for industrial and organizational
psychology (Vol. 1, pp. 39-73). Palo Alto, CA: Consulting Psychologists Press.
Chalofsky, N. E. (1996, Winter). Professionalization comes from theory and research:
The why instead of the how to. New Directions for Adult and Continuing Education,
p. 7.
Chalofsky, N. E. (1998). Professionalization comes from theory and research: The “why”
instead of the “how-to.” In R. J. Torraco (Ed.), Proceedings of the Academy of Human
Resource Development annual conference (pp. 666-670). Baton Rouge, LA: Acad-
emy of Human Resource Development.
Cohen, B. P. (1991). Developing sociological knowledge: Theory and method (2nd ed.).
Chicago: Nelson-Hall.
Denzin, N. K., & Lincoln, Y. S. (2000). Handbook of qualitative research (2nd ed.).
Thousand Oaks, CA: Sage.
Dubin, R. (1976). Theory building in applied areas. In M. D. Dunnette (Ed.), Handbook
of industrial and organizational psychology (pp. 17-39). Chicago: Rand McNally.
Dubin, R. (1978). Theory building (Rev. ed.). New York: Free Press.
Eisenhardt, K. M. (1989). Building theories from case study research. In G. P. Huber &
A. H. Van de Ven (Eds.), Longitudinal field research methods: Studying processes of
organizational change (pp. 65-90). Thousand Oaks, CA: Sage.
Eisenhardt, K. M. (1995). Building theories from case study research. In G. P. Huber &
A. H. Van de Ven (Eds.), Longitudinal field research methods (pp. 65-90). Thousand
Oaks, CA: Sage.
Gall, M. D., Borg, W. R., & Gall, J. P. (1996). Educational research (6th ed.). White
Plains, NY: Longman.
Gioia, D. A., & Pitre, E. (1990). Multiparadigm perspective on theory building. Academy
of Management Review, 15(4), 584-602.
Glaser, B., & Strauss, A. (1967). The discovery of grounded theory: Strategies of qualita-
tive research. London: Weidenfeld and Nicholson.
Gradous, D. B. (1989). Preface to the monograph. In D. B. Gradous (Ed.), Systems the-
ory applied to human resource development: Theory-to-practice monograph series
(pp. 1-6). Alexandria, VA: American Society for Training and Development.
Hansen, C. D. (1998). HRD theory building through qualitative research. In R. J. Torraco
(Ed.), Proceedings of the Academy of Human Resource Development annual confer-
ence (pp. 281-284). Baton Rouge, LA: Academy of Human Resource Development.
Hatcher, T. (1999). Reorienting the theoretical foundations of human resource develop-
ment: Building a sustainable profession and society. In P. K. Kuchinke (Ed.), Proceed-
ings of the Academy of Human Resource Development annual conference (pp. 202-
208). Baton Rouge, LA: Academy of Human Resource Development.
Hearn, G. (1958). Theory building in social work. Toronto, Canada: University of
Toronto Press.

Downloaded from http://adh.sagepub.com at Tartu University Library on October 15, 2009


Lynham / THE GENERAL METHOD 239

Hultgren, F. H., & Coomer, D. L. (Eds.). (1989). Alternative modes of inquiry in home
economics research. Peoria, IL: Glencoe.
Jacobs, R. L. (1997). HRD partnerships for integrating HRD research and practice. In
R. A. Swanson & E. F. Holton III (Eds.), Human resource development handbook:
Linking research and practice (pp. 47-61). San Francisco: Berrett-Koehler.
Jacobs, R. L. (1999). Partnership research: Ensuring more useful HRD collaborations. In
P. K. Kuchinke (Ed.), Proceedings of the Academy of Human Resource Development
annual conference (pp. 874-879). Baton Rouge, LA: Academy of Human Resource
Development.
Kaplan, A. (1964). The conduct of inquiry: Methodology for behavioral science. San
Francisco: Chandler.
Kauffman, D. L. (1980). Systems 1: An introduction to systems thinking. Minneapolis,
MN: Future Systems, Inc.
Klein, K. J., Tosi, H., & Cannella, A. A., Jr. (1999). Multilevel theory building: Benefits,
barriers, and new developments. Academy of Management Review, 24(2), 243-248.
Kuhn, T. (1970). The structure of scientific revolutions (2nd ed.). Chicago: University of
Chicago Press.
Lewin, K. (1945). The research center for group dynamics at Massachusetts Institute of
Technology. Sociometry, 8, 126-135.
Lewin, K. (1951). Field theory in social sciences. New York: Harper Row.
Lynham, S. A. (1998). The development and evaluation of a model of responsible leader-
ship for performance: Beginning the journey. Human Resource Development Interna-
tional, 1(2), 207-220.
Lynham, S. A. (2000a). The development of a theory of responsible leadership for perfor-
mance (Tech. Rep.). St. Paul: University of Minnesota, Human Resource Develop-
ment Research Center.
Lynham, S. A. (2000b). Theory building in the human resource development profession.
Human Resource Development Quarterly, 11(2), 159-178.
Marsick, V. J. (1990a). Altering the paradigm for theory building and research in human
resource development. Human Resource Development Quarterly, 1(1), 5-24.
Marsick, V. J. (1990b). Responding to Professor Passmore’s response. Human Resource
Development Quarterly, 1(1), 29-34.
McLean, G. A. (2001). Contrasting three modes of inquiry. Unpublished manuscript.
Morgeson, F. P., & Hofmann, D. A. (1999). The structure and function of collective con-
structs: Implications for multilevel research and theory development. Academy of
Management Review, 24(2), 249-265.
Mott, V. J. (1996). Knowledge comes from practice: Reflective theory building in prac-
tice. In R. Rowden (Ed.), Workplace learning: Debating five critical questions of the-
ory and practice (New Directions for Adult and Continuing Education No. 72, pp. 57-
63). San Francisco: Jossey-Bass.
Mott, V. J. (1998). Professionalization and reflective theory building in HRD. In R. J.
Torraco (Ed.), Proceedings of the Academy of Human Resource Development annual
conference (pp. 671-676). Baton Rouge, LA: Academy of Human Resource Development.

Downloaded from http://adh.sagepub.com at Tartu University Library on October 15, 2009


240 Advances in Developing Human Resources August 2002

Passmore, D. L. (1990). Invited reaction: Pick a paradigm, any paradigm. Human


Resource Development Quarterly, 1(1), 25-28.
Passmore, D. L. (1997). Ways of seeing: Disciplinary bases of research in HRD. In R. A.
Swanson & E. F. Holton III (Eds.), Human resource development handbook: Linking
research and practice (pp. 199-214). San Francisco: Berrett-Koehler.
Patterson, C. H. (1986). Theories of counseling and psychotherapy (4th ed.). San Fran-
cisco: Harper & Row.
Reynolds, P. D. (1971). A primer in theory construction. New York: Macmillan.
Root, M. (1993). Philosophy of social science: The methods, ideals, and politics of social
inquiry. Cambridge, MA: Blackwell.
Ruona, W. E., & Lynham, S. A. (1999). Towards a philosophical framework for thought
and practice. In P. K. Kuchinke (Ed.), Proceedings of the Academy of Human
Resource Development annual conference (pp. 209-216). Baton Rouge, LA: Acad-
emy of Human Resource Development.
Senge, P. M. (1990). The fifth discipline: The art and practice of the learning organiza-
tion. New York: Doubleday & Currency.
Shindell, T. (1999). The need for theoretical model building in HRD. In P. K. Kuchinke
(Ed.), Proceedings of the Academy of Human Resource Development annual confer-
ence (pp. 217-220). Baton Rouge, LA: Academy of Human Resource Development.
Stake, R. E. (1994). Case studies. In N. K. Denzin & Y. S. Lincoln (Eds.), Handbook of
qualitative research (pp. 236-247). Thousand Oaks, CA: Sage.
Strauss, A., & Corbin, J. (1990). Basics of qualitative research: Grounded theory proce-
dures and techniques. Thousand Oaks, CA: Sage.
Strauss, A., & Corbin, J. (1998). Basics of qualitative research: Grounded theory proce-
dures and techniques (2nd ed.). Thousand Oaks, CA: Sage.
Swanson, R. A. (1988). Research and development (and other life and death matters).
Performance Improvement Quarterly, 1(1), 69-82.
Swanson, R. A. (1997). HRD research: Don’t go to work without it. In R. A. Swanson &
E. F. Holton III (Eds.), Human resource development handbook: Linking research and
practice (pp. 3-20). San Francisco: Berrett-Koehler.
Swanson, R. A. (1999). Foundations of performance improvement and implications for
practice. Advances in Developing Human Resources, 1, 1-45.
Swanson, R. A. (2000). Theory and other irrelevant matters. Human Resource Develop-
ment International, 3(3), 273-278.
Swanson, R. A., & Holton, E. F., III. (2001). Foundations of human resource develop-
ment. San Francisco: Berrett-Koehler.
Swanson, R. A., Lynham, S. A., Ruona, W.E.A., & Torraco, R. J. (2000). Theory building
research in HRD—Pushing the envelope! In P. Kuchinke (Ed.), Proceedings of the
Academy of Human Resource Development 2000 annual conference (pp. 1125-1130).
Baton Rouge, LA: Academy of Human Resource Development.
Thomas, G. (1997). What’s the use of theory? Harvard Educational Review, 67(1), 75-104.
Torraco, R. J. (1994). The development and validation of a theory of work analysis.
Unpublished doctoral dissertation, University of Minnesota, St. Paul.

Downloaded from http://adh.sagepub.com at Tartu University Library on October 15, 2009


Lynham / THE GENERAL METHOD 241

Torraco, R. J. (1997). Theory-building research methods. In R. A. Swanson & E. F.


Holton III (Eds.), Human resource development handbook: Linking research and
practice (pp. 114-137). San Francisco: Berrett-Koehler.
Torraco, R. J. (Ed.). (1999). Performance improvement: Theory and practice. Advances
in Developing Human Resources, 1.
Torraco, R. J. (2000). A theory of knowledge management. Advances in Developing
Human Resources, 5, 38-62.
Van de Ven, A. H. (1989). Nothing is quite so practical as a good theory. Academy of Man-
agement Review, 14(4), 486-489.
Von Bertalanffy, L. (1968). General systems theory (Rev. ed.). New York: George
Braziller.
Weick, K. E. (1995). What theory is not, theorizing is. Administrative Science Quarterly,
40(3), 385-390.
Whetten, D. A. (1989). What constitutes a theoretical contribution? Academy of Manage-
ment Review, 14(4), 490-495.

Downloaded from http://adh.sagepub.com at Tartu University Library on October 15, 2009

View publication stats

You might also like