See discussions, stats, and author profiles for this publication at: https://www.researchgate.

net/publication/225969619

Developmental research methods: Creating

knowledge from instructional design and
development practice

Article  in  Journal of Computing in Higher Education · March 2005

DOI: 10.1007/BF02961473

CITATIONS READS

44 14,453

2 authors:

Rita C. Richey James D. Klein

Wayne State University Florida State University
22 PUBLICATIONS   1,069 CITATIONS    50 PUBLICATIONS   1,386 CITATIONS   

SEE PROFILE SEE PROFILE

All content following this page was uploaded by Rita C. Richey on 15 May 2014.

The user has requested enhancement of the downloaded file.

Journal of Computing in Higher Education
Spring 2005 Vol. 16(2), 23-38.

Developmental Research Methods:

Creating Knowledge from
Instructional Design and
Development Practice
Rita C. Richey
Wayne State University
James D. Klein
Arizona State University

ABSTRACT

HE PURPOSE OF THIS ARTICLE is to provide direction to

T those entertaining a developmental research project. There are

two categories of developmental research, both of which are
examined here. The two types vary in terms of the extent to which
the conclusions resulting from the research are generalizable or con-
textually specific. This article describes developmental research in
terms of the traditional stages of planning, conducting, and reporting
a research project--problem definition, literature reviews, and research
procedures. Examples of adapting a literature review to a develop-
mental study are identified. Problem definition procedures encompass
focusing on and framing the problem, and identifying the limitations
of the research. Research procedures are adapted to developmental
studies, particularly with respect to identifying the study's participants,
creating a research design, and collecting and analyzing the data.
Finally, issues with respect to reporting developmental research are
discussed. (Keywords: developmental research, evaluation research
instructional design, research methods, theory to practice)

23
 DEVELOPMENTAL RESEARCH METHODS

INTRODUCTION

OR MANY, the design, development and evaluation of instruc-

F tional products and programs are considered to be the heart of

instructional technology. It is also critical to those engaged in
a broad range of computer applications. Consequently, it follows that
a substantial part of our research should address design processes and
tools. In other words, we should be studying design, development,
and evaluation as well as doing it. Of course, this is not a new
suggestion; some scholars have heeded it, even though the innovators
often put more effort into doing rather than studying.
The body of research literature that directly pertains to instruc-
tional development is known as developmental research: "the system-
atic study of designing, developing and evaluating instructional pro-
grams, processes and products that must meet the criteria of internal
consistency and effectiveness" (Seels & Richey, 1994, p. 127). De-
velopmental research seeks to create knowledge grounded in data
systematically derived from practice. It is a pragmatic type of research
that offers a way to test "theory" that has been only hypothesized and
to validate practice that has been perpetuated essentially through
unchallenged tradition. In addition, it is a way to establish new pro-
cedures, techniques, and tools based upon a methodical analysis of
specific cases. As such, developmental research can have a function
of either creating generalizable conclusions or statements of law, or
producing context-specific knowledge that serves a problem solving
function.
There are two categories of developmental research, referred to
as Type 1 and Type 2 (Richey, Klein, & Nelson, 2004). They vary
in terms of the extent to which the conclusions resulting from the
research are generalizable or contextually specific. Type 1 develop-
mental studies focus upon a given instructional product, program,
process, or tool. They reflect an interest in identifying either general
development principles or situation-specific recommendations. Typi-
cally Type 1 studies address not only product design and develop-

24
 Richey and Klein

ment, but evaluation as well. At times they may validate a particular

design or development technique or tool. Type 2 studies, on the other
hand, focus upon a given design, development, or evaluation model
or process. They may involve constructing and validating unique design
models and processes, as well as identifying those conditions that
facilitate their successful use.
Developmental research is different from the design-based research
that has been recently discussed. This research emphasizes the study
of learning as a result of designing unique instructional interventions
(The Design-Based Research Collective, 2003). It is also different from
traditional research in instructional psychology, media comparison, and
message design. However, well-conceived developmental research can
offer a direct response to the many demands for research that is
prescriptive and meets the pressing needs of practitioners (Richey,
1997).
The aim of this article is to provide some direction to those
entertaining a developmental research project. In essence, it is a
discussion of establishing the credibility of a given developmental study
by assuring authenticity and methodological precision. Since many
developmental research topics can be addressed in a number of dif-
ferent ways, this paper may also help the reader recognize the poten-
tial of a given problem to be addressed as a developmental topic.
Below, we describe developmental research in terms of the traditional
stages of planning, conducting, and reporting a research project:
9 Problem definition;
9 Literature reviews; and
9 Research procedures.
DEFINING THE RESEARCH PROBLEM
The authenticity of a developmental research project often de-
pends upon the problem selected. Is the problem one that is common
to many designers and developers? Is it one that is currently critical
to the profession? Does the problem reflect realistic constraints and
conditions typically faced by designers? Does the problem pertain to

25
 DEVELOPMENTAL RESEARCH METHODS

cutting-edge technologies and processes? Answers to these questions

not only predict interest in the project, but also whether the research
is viewed as relevant. "Explorations of research relevance are typi-
cally examinations of shared perceptions, the extent to which research-
ers' notions of relevance are congruent with the perceptions and needs
of practitioners" (Richey, 1998, p. 8). This is particularly true with
developmental research where the object of such research is clearly
not simply knowledge, but knowledge that practitioners can use.
Focusing the Problem. Given a relevant research topic, the project
must first be given a "developmental twist." This begins in the prob-
lem definition stage. It is done by focusing the research problem on
a particular aspect of the design, development, or evaluation process,
as opposed to focusing on a particular variable that impacts learning.
The problem definition stage must also establish research param-
eters. At minimum, this involves determining whether the research
will be conducted as the design and development is occurring or
whether retrospective data will be collected on a previously devel-
oped program or set of materials. Then the scope of the study must
be established. How much of the design and development process
will be addressed? Will the research address:
9 All parts of the design of the instruction?
9 The development (or part of the development) of the instruc-
tion?
9 The evaluation of the instruction? If so, will formative, and
summative, and confirmative evaluation be addressed?
9 The revision and retesting of the instruction?
Developmental studies often are structured in phases. For example,
comprehensive Type 1 studies may have an analysis phase, design
phase, a development phase, and a try-out and evaluation phase.
Another organization of a Type 1 study would include phases directed
toward first analysis, then prototype development and testing, and
finally prototype revision and retesting. A study by McKenney (2002)
is an example of this type of research. McKenney examined the
development of a computer program to support curriculum materials

26
 Richey and Klein

development in the context of secondary science and mathematics

education in southern Africa. As with any traditional research project,
the study was guided by predetermined questions and was embedded
into a conceptual framework supported by current literature on cur-
riculum development, teacher professional development, and computer-
based performance support. The project was a careful and extensive
documentation of the phases of ISD--needs and context analysis,
design, development, and formative evaluation of several prototypes,
and summative evaluation.
Type 2 studies may have a model construction phase, a model
implementation phase, and a model validation phase. For example,
Tracey's (2002) research was conducted in four phases. The first phase
was a literature review that was synthesized into a multiple intelli-
gences instructional design model. The second phase was a Delphi
study that resulted in a modified model with expert approval. The
third phase involved controlled testing of the model implementation.
Two versions of a given instructional workshop were designed and
developed, one by a pair of designers using a traditional design model
and another by a design team using the multiple intelligences model.
Finally, in phase four, the two instructional products were implemented
and compared in a real-life training situation to complete the valida-
tion process.
Framing the Problem. Seels (1994) describes typical processes
one uses to explain the goals of a developmental research project. For
example, research questions, rather than hypotheses, commonly serve
as the organizing framework for developmental studies. This tactic is
appropriate if there is not a firm base in the literature which one can
use as a basis for formulating a hypothesis. This is often the case
with such research, especially if the problem focuses on emerging
technologies. However, research questions are also more appropriate
for qualitative research, a common developmental methodology.
Identifying Limitations. Because developmental research is of-
ten context-specific, one must be particularly concerned with the limi-
tations or unique conditions that may be operating in a particular study.
Such limitations will affect the extent to which one may generalize

27
 DEVELOPMENTAL RESEARCH METHODS

the conclusions of the study. The results may be applicable only in

the situation studied, or to others with similar characteristics, rather
than being generalizable to a wider range of instructional environ-
ments.
T H E R E V I E W OF R E L A T E D LITERATURE
Typically, literature reviews in experimental studies concentrate
upon the specific variables being studied, usually the independent and
dependent variables. This orientation may not prove useful in many
developmental studies. The goal of the literature review, however,
remains the same as with other types of research projects. It is to
establish the conceptual foundations of the study.
The literature review in Type 1 developmental studies may ad-
dress topics, such as:
9 Procedural models which might be appropriate for the task
at hand;
9 Characteristics of similar effective instructional products,
programs, or delivery systems;
9 Factors which have impacted the use of the target develop-
ment processes in other situations;
9 Factors impacting the implementation and management of
the target instructional product, program, or delivery system
in other situations.
Literature reviews in Type 2 studies may address topics, such as:
9 A description of models (either formally published or those
currently in use) similar to the one being studied, including
their strengths and weaknesses;
9 Research on the targeted process (for example, task analysis
or evaluation of organizational impact); or
9 Research on factors impacting the use of a given model or
process (for example, factors that facilitate the use of rapid
prototyping models).

28
 Richey and Klein

Both types of studies often address the methodology of developmental

research itself in the literature review.
In developmental studies directed toward innovative instructional
environments or innovative design and development processes, it would
not be unusual to find little research in the literature that is directly
relevant. In such cases the researcher must still identify literature that
is relevant to the foundational theory of the project, even though the
link may be indirect. For example, there is literature on factors that
affect the use of computers and other media in instruction, but there
may be little on factors related to the specific use of virtual reality
as a delivery system. In developmental research the conceptual flame-
work for the study may be found in literature from actual practice
environments (for example, an evaluation report) as well as from tra-
ditional research literature directed toward theory construction.
THE RESEARCH PROCEDURES
Often developmental research occurs in natural work environments.
This tends to enhance the credibility of the research, as well as create
methodological dilemmas for the researcher. Nonetheless, whether the
research is conducted during the design and development process or
retrospectively, the best research pertains to actual projects, rather than
simulated or idealized projects. For example, a study by Jones and
Richey (2000) involved an examination of two actual projects of a
design and development firm (one involving stand-up training and the
other computer-based training).
Perhaps it is the "real life" aspect of developmental research that
results in studies that frequently take even more time to complete
than other types of research. There are often more changes in one's
research plans and procedures as a result of unanticipated events than
is typical in other types of research. Consequently, detailed research
procedures and timelines are most important.
Participants. There are often multiple types of participants in a
given developmental research project, and if the study is conducted
in phases, the participants may vary among phases. The nature of the

29
 DEVELOPMENTAL RESEARCH METHODS

participating populations tends to vary with the type of developmen-

tal research being conducted. Typical populations include:
9 Designers, developers, and evaluators;
9 Clients;
9 Instructors and/or program facilitators;
9 Organizations;
9 Design and development researchers and theorists; as well as
9 Learners and other types of users.
Table 1 shows the array of persons that most commonly participate
in these projects and the various phases of the project in which they
tend to contribute data.
Table 1.
Common Participants in Developmental Research Studies

For example, participants in the Tracey (2002) study included

researchers and theorists in model development, designers in model
use, and learners and instructors in model validation. This was a Type
2 study. The participants in Jones and Richey (2000), another Type

30
 Richey and Klein

2 study, were designers, developers, and clients. This study primarily

addressed model use. McKenney's research (2002) was a Type 1 study.
In the phase addressing the construction and prototyping of an elec-
tronic design tool, the participants were developers, users, evaluators,
and a variety of ID experts. Then, in the phase evaluating and vali-
dating the tool, participants were users (pre-service and in-service
teachers and curriculum developers, i.e., designers) and experts.
Research Design. It is not uncommon for a developmental re-
search project to also utilize multiple research methodologies and
designs, with different designs again being used for different phases
of the project. Table 2 presents a summary of those research methods
that are most frequently used in the various types and phases of
developmental research.
Table 2.
Common Research Methods Employed
in Developmental Research Studies

31
 DEVELOPMENTAL RESEARCH METHODS

This table highlights some typical methodology patterns used in

developmental studies. In Type 1 studies critical design and develop-
ment processes are often explicated using case study methods. Inter-
views, observations, and document analysis are techniques that can
be used to gather the case study data, to document the processes used
and the conditions under which they are employed. Corry, Frick, and
Hansen's (1997) case study on the design and usability of a univer-
sity Web site interviewed prospective users during the needs analysis
phase, used think-aloud protocols and surveys while testing the pro-
totype for usability. Finally, they conducted online testing of site
usability.
Evaluation research techniques are often employed in Type 1
studies to determine the effectiveness of the resulting product or the
particular techniques used during the design and development project.
As with all evaluation research, a variety of data collection techniques
are possible. In Fischer, Savenye, and Sullivan's (2002) Type 1 study,
they conducted a formative evaluation of a computer-based training
course directed toward the implementation of a new university finan-
cial system. In the Klein et al. (2000) Type 1 study, they conducted
a needs assessment to determine the optimal content and delivery
method of a graduate level educational technology course. This study
involved document analysis, interviews, and surveys.
Sometimes, a full experiment is even constructed to test the
product or technique. For example, Borras (1993) developed a com-
puter-based multimedia program for improving college students' spo-
ken French. She field-tested the program using experimental methods
to evaluate the various versions of the program.
In Type 2 research, models of the full design and development
process (or of a particular part of the process) are constructed in a
variety of ways, including the following:
9 By conducting surveys of designers and developers in re-
gards to projects in which they have been involved, such
as Shellnut's (1999) study of motivation design or Phillip's
(2000) study of organizational impact evaluation techniques;
or

32
 Richey and Klein

9 By synthesizing models from the literature, such as Adamski's

(1998) reviews of instructional technology, human factors,
and aviation literature in his efforts to devise an initial model
for designing job performance aids for use in high risk
settings; or
9 By arriving at a consensus of respected experts in the field
using Delphi techniques, such as Tracey's (2002) methods
of finalizing her multiple intelligences design model; or
9 By conducting experiments to validate particular design and
development models, such as Tracey's (2002) and Adamski's
(1998) projects.
Developmental researchers are commonly confronted by method-
ological dilemmas. One is the need to account for contextual vari-
ables, especially in those studies taking place in a natural work set-
ting. For example, to what extent does the client's design experience
and sophistication, or designer expertise, or time pressures impact the
success of a particular project? Since it is typically not possible to
control such factors in the research design, the researcher is then
obligated to carefully describe (and measure, if possible) these vari-
ables in an effort to account for their impact.
Another common situation that is potentially problematic is when
the researcher is also a participant in the study, for example, when
the researcher is also the designer or developer. While this situation
is not preferable, it is not unusual. Care must be taken to insure ob-
jectivity through consistent, systematic data collection techniques and
the collection of corroborating data, if possible. Often structured logs
and diaries completed by several project participants according to a
regularly established schedule create a structure that facilitates the
generation of reliable and comparable data.
Another frequent problem is maintaining the integrity of recall
data. Many studies rely upon self-reports of past projects. Others use
structured interviews of participants. Using previously prepared docu-
ments or data from others involved in the same project facilitates a
triangulation process to validate the data collected.

33
 DEVELOPMENTAL RESEARCH METHODS

Collecting and Analyzing Data. Data collection in a develop-

mental study takes a variety of forms depending upon the focus of
the research. The validity of the conclusions is often dependent upon
the richness of the data set as well as the quality of the research
design. Typical types of data collected in developmental research relate
tO:

9 Documentation of the design, development, and evaluation

tasks, including profiling the design and development con-
text and collecting data: work time and expenses, problems
encountered and decisions made, adjustments made in the
original plans, designer reactions and attitudes, or records
of concurrent work patterns;
9 Documentation of the conditions under which the develop-
ment and implementation took place, including factors such
as equipment and resources available, participant expertise
and background, or time and client constraints; and
9 Identifying the results of pre-design needs assessments,
formative, summative and confirmative evaluations, includ-
ing documentation of the target populations and the imple-
mentation context, and measures of learning, transfer, and
the impact of the intervention on the organization.
As with all research projects, participants must be fully informed
of the nature of the research, not be coerced to be involved, and be
assured that the data will be both anonymous and confidential. Not
only must the participants be informed and give their consent, but
often written consent from their organizations is required as well. Data
may be sensitive and proprietary in nature, and researchers must be
attuned to these issues and their ramifications.
Data analysis and synthesis in a developmental study is not unlike
that of other research projects. There are likely to be descriptive data
presentations, and qualitative data analyses using data from documen-
tation, interviews, and observations. Traditional quantitative data
analyses techniques are used as well.

34
 Richey and Klein

REPORTING DEVELOPMENTAL RESEARCH DATA

There has been an increase in journal articles, conference papers,
and doctoral dissertations reporting on developmental studies in re-
cent years (see Richey, Klein, & Nelson, 2004 for a review). A
particular issue in writing a developmental research report is deciding
how much data to include. While length is typically not an issue for
doctoral dissertations, data sets are often too massive even to include
in a dissertation appendix. Furthermore, journals do not provide enough
space for detailed documentation of developmental research data.
In response to this dilemma, some researchers are using Web sites
as data repositories. For example, full results of a needs assessment
may be included in a Web site, or complete copies of electronic tools
can be provided, or full transcripts of designer interviews can be
presented. Assuming these Web sites are stable and accurate, this
solution allows for full disclosure of data that should prove valuable
to practitioners and researchers alike. With full data sets, practitioners
should be to apply general lessons to their own work environments.
With full data sets, researchers should have opportunities for second-
ary analysis of data (an option seldom available to researchers in this
field), as well as opportunities to fully replicate the research in other
settings.

CONCLUSIONS

HE CRUX OF DEVELOPMENTAL RESEARCH is to sys-

T tematically examine our products, tools, processes, and models

in order to provide reliable, usable information to both prac-
titioners and theorists. It is research, however, that is intricately
connected to real world practice. It creates a loop with practice in-
forming research and research, in turn, informing practice.
With increasing interest in instructional technologies in higher
education and other settings, it is essential that we systematically collect
empirical data to better inform this practice. As evidenced by the

35
 DEVELOPMENTAL RESEARCH METHODS
i

increase in journal articles, conference papers, and doctoral disserta-

tions reporting on developmental studies, this seems to be an increas-
ingly c o m m o n opinion. There is growing interest in this type of
research.
While developmental research is only one of several types of
research methods that can provide practitioners with usable data, its
focus on the design, development, and evaluation of instructional
products and processes is unique. We hope that this article will pro-
vide direction to scholars who want to conduct developmental research,
as well as encourage practitioners to support and participate in such
research. We look forward to new knowledge to guide the field whether
it is based in a specific context or is generalizable to a variety of
settings.

REFERENCES

Adamski, A.J. (1998). The development of a systems design model for job perfor-
mance aids: A qualitative developmental study. (Doctoral dissertation, Wayne State
University, 1998). Dissertation Abstracts International-A, 59(03), 789.

Borras, I. (1993). Developing and assessing practicing spoken French: A multimedia

program for improving speaking skills. Educational Technology Research and
Development, 41 (4), 91-103.

Corry, M.D., Frick, T.W., & Hansen, L. (1997). User-centered design and usability
testing of a web site: An illustrative case study. Educational Technology Research
and Development, 45(4), 65-76.

The Design-Based Research Collective (2003). Design-based research: An emerging

paradigm for educational inquiry. Educational Researcher, 32(1), 5-8.

Fischer, K.M., Savenye, W.C., & Sullivan, H.J. (2002). Formative evaluation of
computer-based training for a university financial system. Performance Improvement
Quarterly, 15(1), 11-24.

Jones, T.S., & Richey, R.C. (2000). Rapid prototyping in action: A developmental
study. Educational Technology Research and Development, 48(2), 63-80.

36
 Richey and Klein

Klein, J.D., Brinkerhoff, J., Koroghlanian, C., Brewer, S., Ku, H., & MacPherson-
Coy, A. (2000). The foundations of educational technology: A needs assessment.
Tech Trends, 44(6), 32-36.

McKenney, S. (2002). Computer-based support for science education materials

development in Africa: Exploring potentials. (Doctoral dissertation, Universiteit
Twente (The Netherlands, 2001). Dissertation Abstracts International-C, 63(03),
355.

Phillips, J.H. (2000). Evaluating training programs for organizational impact. (Doc-
toral dissertation, Wayne State University, 2000). Dissertation Abstracts Interna-
tional-A, 61(03), 840.

Richey, R.C. (1997) Research on instructional development. Educational Technol-

ogy Research and Development, 45(3), 91-100.

Richey, R.C. (1998) The pursuit of useable knowledge in instructional technology.

Educational Technology Research and Development, 46(4), 7-22.

Richey, R.C., Klein, J., & Nelson, W. (2004) Developmental research: Studies of
instructional design and development. In D. Jonassen (Ed.) Handbook of Research
for Educational Communications and Technology (2nd ed.) (pp. 1099-1130). Mahwah,
NJ: Lawrence Erlbaum Associates, Inc.

Seels, B. (1994). An advisor's view: Lessons learned from developmental research

dissertations. A paper presented at the 1994 Annual Meeting of the Association for
Educational Communications and Technology in Nashville, TN.

Seels, B.B., & Richey, R.C. (1994). Instructional technology: The definition and
domains of the field. Washington, DC: Association for Educational Communications
and Technology.

Shellnut, B.J. (1999). The influence of designer and contextual variables on the
incorporation of motivational components to instructional design and the perceived
success of a project. (Doctoral dissertation, Wayne State University, 1999). Disser-
tation Abstracts International-A, 60(06), 1993.

Tracey, M.W. (2002). The construction and validation of an instructional design

model for incorporating multiple intelligences. (Doctoral dissertation, Wayne State
University, 2001). Dissertation Abstracts International-A, 62(12), 4135.

37
 DEVELOPMENTAL RESEARCH METHODS

ABOUT THE AUTHORS

Rita C. Richey is a Professor of Instructional Technology at Wayne

State University where she has taught for over thirty years. Her research
interests are in instructional design research and theory and competency
modeling. She has written or edited ten books, one of which received
the Association of Educational Communications & Technology's 1995
Outstanding Book Award and 1996 Brown Publication Award. Dr. Richey
is a past Vice-President for Research and Development for the Interna-
tional Board of Standards for Training, Performance and Instruction and
a past President of AERA's Special Interest Group on Instructional
Technology. She has received four awards from Wayne State Univer-
sity-the President's Award for Excellence in Teaching, a Distinguished
Faculty Fellowship, the Outstanding Graduate Mentor Award, and the
award for Outstanding Scholarly Achievement by Women Faculty. In
addition, she has been inducted into the Wayne State University Acad-
emy of Scholars. In recognition of her career's work, Dr. Richey re-
ceived the AECT Distinguished Service Award in 2000. Author's present
address: Wayne State University, 381 Education, Detroit., MI 48202. E-
mail: rrichey @wayne.edu
James D. Klein is a Professor in the Educational Technology Program
at Arizona State University, Tempe. He serves as a member and treasurer
of the International Board of Standards for Training, Performance and
Instruction. Previously, he served as development editor of Educational
Technology Research & Development and as president of the Design and
Development Division, and the Research and Theory Division of the
Association for Educational Communications and Technology. He has
been recognized as an outstanding alumnus of the Instructional Systems
Program at Florida State University and for his service to the Design and
Development Division. Dr. Klein's research, teaching, and consulting
activities are in the areas of instructional design, strategies for active
learning, and performance improvement. Author's present address: Ari-
zona State University, Division of Psychology in Education, PO Box
870611, Tempe, AZ 85287-0611. E-mail: [email protected]

38
View publication stats