Psychological Review

V O L U M E 87 NUMBER 3 MAY 1980

Verbal Reports as Data

K. Anders Ericsson and Herbert A. Simon
Carnegie-Mellon University

The central proposal of this article is that verbal reports are data. Accounting
for verbal reports, as for other kinds of data, requires explication of the mech-
anisms by which the reports are generated, and the ways in which they are
sensitive to experimental factors (instructions, tasks, etc.). Within the theoret-
ical framework of human information processing, we discuss different types of
processes underlying verbalization and present a model of how subjects, in re-
sponse to an instruction to think aloud, verbalize information that they are
attending to in short-term memory (STM). Verbalizing information is shown
to affect cognitive processes only if the instructions require verbalization of
information that would not otherwise be attended to. From an analysis of
what would be in STM at the time of report, the model predicts what can
reliably be reported. The inaccurate reports found by other research are shown
to result from requesting information that was never directly heeded, thus
forcing subjects to infer rather than remember their mental processes.

After a long period of time during which This concern for understanding the course
stimulus-response relations were at the focus of cognitive processes has revived interest in
of attention, research in psychology is now finding means to increase the temporal density
seeking to understand in detail the mecha- of observations of behavior to reveal in
nisms and internal structure of cognitive pro- greater detail intermediate stages of the pro-
cesses that produce these relations. In the cesses. Increasingly, investigators record the
limiting case, we would like to have process direction of the subject's gaze (eye move-
models so explicit that they could actually ments) and the intermediate behavior (mak-
produce the predicted behavior from the in- ing moves or other physical manipulations of
formation in the stimulus. stimulus material) that precedes the solution
or criterion performance. Since data on inter-
mediate processing are costly to gather and
This research was supported by Research Grant analyze, it is important to carefully consider
MH-07722 from the National Institute of Mental
Health.
how such data can be interpreted validly and
We would like to thank Bill Chase and Robert what contribution they can make to our un-
Neches for their valuable comments on earlier ver- derstanding of the phenomena under study.
sions of this article.
Requests for reprints should be sent to K. Anders
Ericsson, Department of Psychology, Carnegie-Mel-
Doubts About Verbal Data
lon University, Schenley Park, Pittsburgh, Pennsyl- One method frequently used to gain in-
vania 15213 (before September 1), or Department of
Psychology, Muenzinger Building, University of Colo- formation about the course and mechanisms
rado, Boulder, Colorado 80309 (after September 1). of cognitive processes is to probe verbally the

Copyright 1980 by the American Psychological Association, Inc. 0033-295X/80/8703-0215$00.75

215
216 K. ANDERS ERICSSON AND HERBERT A. SIMON

subject's internal states. However, since the purpose in obtaining verbal reports is mainly
triumph of behaviorism over introspectively to generate hypotheses and ideas, investi-
oriented competing viewpoints, verbal reports gators need not concern themselves (and gen-
have been suspect as data. More precisely, erally have not concerned themselves) with
behaviorism and allied schools of thought methodological questions about how such
have been schizophrenic about the status of data are to be collected. As a result, there is
verbalizations as data. On the one hand, ver- little published literature on such methodo-
bal responses (or keypunches that are psy- logical issues; the data-gathering and data-
chologically indistinguishable from verbal re- analysis methods actually used vary tremen-
sponses, except that they are made with the dously, and the details of these methods are
finger instead of the mouth) provide the basic sketchily reported in research publications
behavioral data in standard experimental that make use of such data.
paradigms. In the concept attainment experi- This state of affairs is wholly unsatisfac-
ment, the subjects say (or signal) yes or no tory if we are to make rapid and continuing
when a possible instance is presented to them. progress in understanding human cognitive
In a problem-solving experiment, they report processes. First, no clear guidelines are pro-
the answer when they find it. In a rote ver- vided to distinguish illegitimate "introspec-
bal learning experiment, they say "DAX" tion" from the numerous forms of verbal out-
when the stimulus syllable "CEF" is pre- put (see the earlier examples) that are rou-
sented. The actual performance measures tinely treated as hard data, such as passing
commonly used—latencies and numbers of the chain-and-transit test. On what theoreti-
items correct—are derived from these re- cal or practical grounds do we distinguish be-
sponses, and the former depend for their tween a subject's yes or no in a concept at-
validity on the veridicality of the latter. tainment experiment and the assertions that
On the other hand, modern psychology has the hypothesis being entertained is "small
been vague about the use that can be made yellow circle"? Second, no distinction is made
of verbalizations produced by the subject between such diverse forms of verbalization
along the route to solution or final response. as "thinking aloud" protocols, retrospective
Even more dubious is the status of subject re- responses to specific probes, and the classical
sponses to experimenter probes or retrospec- introspective reports of trained observers. All
tive answers to experimenter questions about are jointly and loosely condemned as "intro-
prior behavior. All of these sorts of verbal be- spection."
havior are frequently dismissed as variants of
the discredited process of introspection (Nis- Evolving a Methodology for Verbal Reporting
bett & Wilson, 1977). Introspection, it has
been generally agreed, may be useful for the To end this confusion, we must extend our
discovery of psychological processes; it is analyses of the tasks that our subjects are
worthless for verification. As Lashley (1923) performing to incorporate the processes they
said in a vigorous and widely cited attack on are using to produce their verbal responses.
the method, "introspection may make the The expansion of theories to include a theory
preliminary survey, but it must be followed of the measuring instruments is commonplace
by the chain and transit of objective measure- in physics. Experiments that involve weigh-
ment" (p. 352). ing objects require at least a rudimentary
theory of the pan balance. In the same way,
Unsatisfactory Methodology lor Verbal experiments that record verbal responses of
Reporting any kind need at least a rudimentary theory
of how subjects produce such responses—in
The notion that verbal reports provide pos- what memories the response information has
sibly interesting but only informal informa- been stored, what demands the response
tion to be verified by other data has had a makes on short-term memory, whether re-
significant effect on the ways in which ver- sponses can go on in parallel with other be-
balizations are collected and analyzed. If the haviors, and so on. Nor does this require-
 VERBAL REPORTS AS DATA 217

merit of a theory of the response mechanism status as a direct observation; models that
involve us in a vicious circle. Such a theory can regenerate the verbalizations (or encoded
must be developed and tested simultaneously aspects of them) can be constructed and
with our theories of task performance. In evaluated without such assumptions (Ander-
fact, such a theory, correct or incorrect, is son, Note 1).
implicit when we treat the subjects' verbal
responses as veridical in standard laboratory Types of Verbalizing Procedures
paradigms.
In this article we reexamine the validity of The only common feature among the whole
verbal reports as data. We propose some range of techniques used to obtain verbal
means for moving from informal analysis of data is that the subject responds orally to an
verbalized information toward objective pro- instruction or probe. Because of the flexibility
cedures for collecting and analyzing them of language, there are virtually no limits to
that would satisfy Lashley's (1923) chain- the probes we can insert and the questions we
and-transit test. A main goal is to demon- can ask subjects that will elicit some kind of
strate that results from studies that are often verbal response.
cited against the use of verbalized informa- We propose a model of the cognitive pro-
tion can be understood in terms of the meth- cesses that generate subjects' verbal responses.
ods used to collect and analyze the verbaliza- This model should be seen as a hypothesis
tions. about cognition on par with other hypotheses
about cognitive processes. An example will
Plan and Scope make this point clear. Suppose that subjects
in a problem-solving experiment are asked
With the advent of human information- whether they used subgoals to solve the prob-
processing theory, detailed models of mem- lem or solved it directly. If they assert that
ory and problem solving (e.g., Anderson & they used subgoals, this would hardly be
Bower, 1973; Newell & Simon, 1972; Simon, conclusive evidence that they did, for it is
1979) have been put forward that demon- easy to propose models of their cognitive
strate the sensitivity of behavior to task in- processes that would permit them to generate
structions, types of stimuli, and other crucial this answer without consulting memory traces
factors in the experimental design. In this of the solution process to search for one or
article we use the theoretical framework of more subgoals among them. On the other
human information-processing theory to pro- hand, if a subject, in reply, at once described
pose a model for the verbalization processes one or more specific subgoals, and these were
of subjects instructed to think aloud, to give both relevant to the problem and consistent
retrospective verbal reports, or to produce with other evidence of the solution process,
other kinds of verbalizations in response to then it would be more difficult to construct
experimenters' instructions. We will then use a model of the cognitive processes that would
the model to analyze the thinking aloud produce this information without hypothesiz-
method and other procedures using verbaliza- ing that it was stored in, and accessible from,
tion in the light of the criticisms that have the subject's memory of the steps taken in
been made of them. solving the problem.
We will conceive of the recorded verbaliza- The analogy to performance behavior,
tions as data—exactly like latencies, eye fixa- whose veridicality is commonly accepted
tions, sequences of moves, and so on—to be without question, is clear. It would never
accounted for by a corresponding model, cross a researcher's mind simply to ask sub-
which generates them literally or on the level
of encoded patterns or information content. 1
This means that we will not assume that the Because the phrase "attended to" is often awk-
ward stylistically, we will sometimes use "heeded" in-
verbalized description accurately reflects the stead. So, we will say, more or less synonymously,
internal structure of processes or of heeded 1 that information was "attended to," was "heeded,"
information, or that it has any privileged or was "stored in STM."
218 K. ANDERS ERICSSON AND HERBERT A. SIMON

jects to raise their hands when they had However, in the situations of primary im-
solved a problem, without reporting the solu- portance to us, the two tasks are highly in-
tion, no matter how much the experimenter terrelated. In favorable situations, the addi-
thought he or she could trust them. The best tional cognitive load imposed by the instruc-
evidence that they have actually reached a tion to verbalize may be negligible. In our
solution is their ability to report it. Con- subsequent review of experimental results,
versely, it is hard to imagine a model of the we will come back to the paradigm of dual
cognitive processes that could report the solu- tasks in order to discuss the interference gen-
tion unless it had actually been found. The erated by the added verbalizing task. Note
relation of report to outcome of the process is that even in the case of retrospective verbal-
so obvious that it never occurs to us to ques- ization, the subject's performance may de-
tion the inference about process that we are pend heavily on how much incidental mem-
drawing from the behavior. The procedures we orizing he or she does while performing the
propose in this article to infer internal pro- initial task.
cesses from a wider range of overt verbal be- We will mainly consider situations in which
haviors are simply an extension of the pro- the verbalizing is supposed to be subordinate
cedures that we already use every day in the to, and passively dependent on, the ongoing
laboratory. In both cases they involve build- cognitive process (see Panel B of Figure 1),
ing and testing alternative models of the cog- as it only involves verbalization of heeded
nitive processes that are going on, using stan- information already generated by the task-
dard paradigms for generating and testing directed processes. However, we also discuss
hypotheses. situations in which the verbalization is pri-
We begin with a classification of different mary and must follow requirements of form
species of verbalization. Producing verbaliza- and content imposed, for example, by instruc-
tions may be the subject's primary task, or tions (see Panel C of Figure 1).
only incidental to the "real" task that he or
she is addressing. The verbalization may Concurrent and Retrospective Verbalization
either be concurrent with task performance or
retrospective. Various kinds of intermediate One of the primary distinctions made in
processes may intervene between the internal human information-processing models is be-
representation of information and its ver- tween different types of storage systems used
balization. The subject may report about to retain information (Atkinson & Shiffrin,
specific events or may be asked to make gen- 1968; Simon, 1979, chap. 2.3). In whatever
eralizations. These and other variations in way the differences among memories may be
the circumstances under which verbalization conceptualized, what is remembered, and how
takes place can have a significant effect on well, will generally depend critically on the
what is verbalized and on the interpretation interval between the moment of acquisition
of the verbal data. and the moment of recall. This interval is an
important consideration in classifying ver-
Relative Primacy of the Two Tasks balization procedures.
If information is verbalized at the time
When subjects verbalize concurrently, they the subject is attending to it, we will label
generally must do two things, namely, per- the procedure concurrent verbalization. If a
form the task that is being studied and pro- subject is asked about cognitive processes
duce the verbalizations. In the extreme case that occurred at an earlier point in time, we
in which the verbalization information is will label the procedure retrospective ver-
totally unrelated to the main task and the balization.
purpose is to study the interference between
the two (Peterson, 1969), the tasks and their Receding Before Verbalization
respective heeded information may be viewed
as entirely separate and distinct. (See Panel Various kinds of processes, and especially
A of Figure 1.) receding processes, may intervene between
 VERBAL REPORTS AS DATA 219

Cognitive Task -directed

processes cognitive
procesees

f t
Heeded Information

Verbalization

Requirement of verbalization*
having specified form and content

Heeded Information —I Heeded information

t
Verbalization Task-directed
cognitive proci

Verbalization C

Figure 1. Three possible relations between task-directed cognitive processes and verbalization.
([Panel A] Verbalization and task-directed processes are separate and distinct. [Panel B] Ver-
balization of heeded information is generated by task-directed processes. [Panel C] Require-
ments for verbalization modify task-directed processes.)

the time the information was heeded by the verbal descriptions of nonverbal stimuli for
central processor (CP) and the time a cor- their own future use, the format is compact
responding verbalization is generated. When and incorporates many idiosyncratic referents.
information is reproduced in the form in When verbalizations are generated to com-
which it was acquired from the central pro- municate the information to another person,
cessor, we will speak of direct or Level 1 additional processing is required to find un-
verbalization. When one or more mediating derstandable referents (Werner & Kaplan,
processes occurs between attention to the in- 1963).
formation and its delivery, we will speak of 2. Intermediate scanning or filtering pro-
encoded Level 2 or Level 3 verbalization. A cesses (Level 3 verbalization). When the task
number of different kinds of intermediate instructions ask for verbalization of only a
processes exist between access and verbaliza- selected type of attended content, it is neces-
tion that modify the information that is sary to postulate additional processes that
heeded. Among the important kinds are the test recurrently if the heeded information
following. matches the desired type. A typical example
1. Intermediate receding into verbal code of such instructions occurs in commentary
(Level 2 verbalization). This occurs when driving experiments in which the subjects are
the internal representation in which the in- asked to report all perceived traffic hazards
formation is originally encoded is not in ver- while they are driving a car (Soliday & Al-
bal code but has to be translated into that len, 1972).
form. Werner and Kaplan (1963) have shown 3. Intermediate inference or generative pro-
that when subjects generate verbalizations or cesses (Level 3 verbalization). The situation
220 K. ANDERS ERICSSON AND HERBERT A. SIMON

is even more complicated if the experimenter curate than observers were in identifying the
is interested in particular aspects of the situa- important situational factors that actually
tion that a subject would not ordinarily ver- determined the behavior.
balize or attend to. The issue of whether the
instruction to verbalize calls for information Forms of Probing
that normally is not heeded by the subjects One of the most direct and widely used
in performing the cognitive activity under methods to gain information about subjects'
study is central and directly related to the internal states is to instruct them to think
occurrence of intermediate inference and gen- aloud or talk aloud. With this procedure, the
erative processes. Since we will return to this heeded information may be verbalized either
issue in more depth, only a brief summary through direct articulation or by verbal en-
will be given here of the types of information coding of information that was originally
that are likely to require additional mediating stored in a nonverbal code. With the instruc-
processing for their generation. tion to verbalize, a direct trace is obtained
In addition to verbalizing their ongoing of the heeded information, and hence, an in-
thinking, subjects are sometimes asked for direct one of the internal stages of the cogni-
verbal descriptions of their motor activities, tive process.
for example, what objects are moved where, In a related procedure, the subjects are
or where they are looking. When this in- probed, concurrently with their performance
formation is not directly heeded, as is often of a task, for specific information, usually of
the case, the subject is required to observe a kind that they presumably need to guide
his or her own internal processes or overt be- their succeeding behavior. Typical examples
havior to generate the information. of concurrent probing are requests to sub-
Experimenters are often interested in the jects to report the hypotheses they are using
subjects' reasons for their overt behavior and in concept learning and discrimination learn-
consequently ask the subjects to verbalize ing.
their motives and reasons, which in the nor- A third class of verbalization procedures,
mal case for many activities may not be avail- which we have called retrospective verbaliza-
able directly or even at all. tion, probes the subject for information after
Similarly, in studies that use retrospective the completion of the task-induced processes.
verbalization, subjects are seldom asked what For example, subjects may be asked to report
they can remember about specific instances of just after the process has been completed.
their cognitive processes. Rather, they are Another form of retrospective probing is a
generally asked to retrospect about their method, which we call interpretive probing,
thought processes in experiments with many in which subjects are probed at the comple-
trials or to answer general questions, and tion of an experimental session consisting of
thus must try to synthesize all the available a large number of different trials. This pro-
information after selective recall. cedure is sometimes justified as eliminating
The intermediate processes investigated by any possibility that the probing will affect
Tversky and Kahneman (1973) for judg- the "real" data of the experiment.
ments on frequency and the probability of
events fall in this general category. Events Particular and General Reports
that were recalled readily were judged by If the purpose of retrospective probing
subjects to be representative and frequent, were to recover memory traces of subjects'
but this led to large estimating errors, for processes, then the appropriate instruction
frequency and representativeness are not the would be to ask them to recall their specific
only determiners of availability for recall. thought processes during particular trials of
Similarly, Nisbett and Wilson (1977) have the experiment. For at least two different
shown that subjects verbalizing retrospec- reasons, such a procedure is rarely used. First,
tively in a variety of settings about the mo- after a series of trials, a subject's memory for
tives for their behavior were no more ac- internal states of individual cognitive pro-
 VERBAL REPORTS AS DATA 221

cesses will be poor and lacking in detail. More- proposed by Flanagan (1954), the subjects
over, there is a tendency over time for recur- were always asked to report their memory
rent cognitive processes to become automated, for specific events, for example,
so that the accessible intermediate states of
the processes for the later trials of the ex- pilots returning from combat were asked "to think
periment become few or nonexistent. of some occasion during combat flying in which you
Second, many experimenters are interested personally experienced feelings of acute disorienta-
tion or strong vertigo." They were then asked to
primarily in general characteristics of the describe what they "saw, heard, or felt that brought
thought processes, and not in the episodic on the experience." (Flanagan, 19S4, p. 329)
details of the individual trials. Such experi-
menters probe their subjects with questions From these considerations, we can see that
such as, "How did you do these tasks?" interpretive probing, unlike the critical inci-
Such questions implicitly or explicitly re- dent technique, cannot be relied on to pro-
quest a general, rather than specific, inter- duce data stemming directly from the sub-
pretation of how the subject was performing jects' actual sequences of thought processes.
the tasks in question. The former procedures encourage or even re-
There are several different ways in which quire subjects to speculate and theorize about
subjects might arrive at descriptions of their their processes, rather than leaving the the-
general procedures, as distinct from reports ory-building part of the enterprise to the ex-
on specific behaviors during individual trials. perimenter. There is no reason to suppose
One possibility is that the subjects are aware that the subjects themselves will or can be
of the general procedures, or "programs," aware of the limitations of the data they are
they are using, use essentially the same pro- providing. Moreover, the variety of inference
grams on all trials, and can recall and report and memory processes that might be involved
these directly, without reference to the spe- in producing the reports make them extremely
cific behavior they produced. Another pos- difficult to interpret or to use as behavioral
sibility is that subjects can remember some data.
parts, or even complete episodes, of their In some studies, subjects are even asked
processes during particular trials, and that how they would behave if the conditions of
they attempt to generalize this information the experiment were altered in some way. We
into a general procedure, which they then will refer to this procedure as probing for
report. A different possibility is that subjects hypothetical states. For example, in a study
remember some specific tasks, regenerate (by by Reed and Johnsen (1977), subjects were
redoing them) the processes used for these asked how they would solve a problem if it
tasks, and use this information to infer the were presented to them again. Subjects in a
general procedures they may have used. Fi- study of Nisbett and Wilson (1977) were
nally, the subjects may be drawing on a asked how they would react to a story if some
variety of kinds of prior information, such passages had not been presented.
as general knowledge on how one ought to do Finally, subjects may be asked questions
these tasks, to generate a verbal report de- that can be answered without reference to
scribing a general procedure or strategy. In
the context of the experiment—what we will
this case, the verbal reports may not bear
any close relation to the actual cognitive call probing for general states. Many investi-
processes used in the tasks (Nisbett & Wil- gators (Watson, 1920) do not seem to dis-
son, 1977). tinguish between verbal reports given about
It is interesting to note that in areas of a just preceding cognitive process, and verbal
applied psychological research in which ver- probing to elicit general procedures and meth-
bal questioning of subjects has a long tradi- ods outside the context of specific processes
tion, subjects are usually asked about specific (e.g., how one hits a golf ball). The theoreti-
events rather than for general information or cal interpretation of these two modes of ver-
conclusions. In the critical incident technique balization should be quite different.
222 K. ANDERS ERICSSON AND HERBERT A. SIMON

Directed or Specialized Probing is frequent and relatively well motivated.

When subjects are asked to report on im-
Verbal probes differ not only along the
mediately preceding cognitive processes of
dimensions of concurrency versus retrospec-
relatively short duration, specific probes are
tion and the generality or particularity of the
more questionable and less useful. In a logi-
events that are to be reported, but also in the
cal sense, the experimenter gets just as much
comprehensiveness of the topics that are to
information from the subject in the third as
be reported. In many studies, the investigator
in the first two of the following three cases.
is interested in only some particular aspect of
subjects' behaviors. Then the verbal probe (1) Directed probe 1
may be constructed to induce the subjects to Question: Did you use X as a subgoal?
generate information specifically relevant to Answer: Yes
(2) Directed probe 2
the hypotheses under consideration. To help Question: Did you use any subgoals? If so,
subjects retrieve the desired information from which ?
memory, and to induce greater completeness Answer: Yes, I used X.
of the verbal reports, the question or verbal (3) Undirected probe
Verbal report: . . . I was first trying to get X
probe often contains contextual information. and I ... when I attained X ...
To guard against subjectivity in analyzing
verbal reports, the investigator often supplies The replies in all three cases provide evidence
subjects with a fixed set of alternative re- that the subject used X as a subgoal, yet the
sponses. In contrast, a general instruction to evidence is stronger in the third case than in
give verbal reports typically asks subjects to the second, and in the second than in the first.
tell everything they can remember or are The verbalization of Case 1 could easily be
thinking of while performing the task. generated by processes independent of any
In most cases, verbosity and absence of memory for the actual thought processes.
selectivity in subjects' reports is not an im- Comparing Cases 2 and 3, the former com-
portant problem. What the subject reports is municates to subjects the information that
likely to be less, rather than more, than we the experimenter expects them to report. It
should like to hear. In no study known to us may encourage subjects to try to infer or
using general instructions has the investigator guess what kind of information the experi-
complained that subjects have reported too menter will accept, and to generate informa-
much information from actual memory. tion accordingly.
One common difficulty in probing for spe- In many cases, other criteria are available
cific information, especially when the subjects for estimating the validity of the reports. An
are offered a fixed set of alternative answers, analysis of the task (Newell & Simon, 1972)
is to know that the questions conform to the will often provide strong indications of the
internal representations that the subjects are adequacy of verbalized information, especially
employing in their thought. Probes for types in cases with a large number of logical pos-
of information that subjects do not have di- sibilities of response.
rectly accessible, or probes that provide in- Finally, different kinds of probes may have
adequate sets of alternatives, may force sub- different effects on the subsequent behavior of
jects to intermediate and inferential process- subjects. The request for a certain type of
ing, and hence produce verbal reports that information may serve as a hint to subjects
are not closely related to the actual thought about what aspects of the task are important.
process. Moreover, when specific, fixed-alter- Subjects may also alter their normal mode of
native probes are used, there is no way to processing in order to be able to give the re-
detect from subjects' responses that this has quested information to the experimenter on
occurred. subsequent trials.
Providing contextual information and
prompts to subjects may aid recall from long- The Processing Model
term memory, and in studies of long- Our purpose in presenting a specific pro-
term memory the use of prompts and context cessing model is to aid us in interpreting
 VERBAL REPORTS AS DATA 223

verbal data obtained from subjects and the (transfered to STM) before it can be re-
relation of their verbal to their other be- ported.
havior. Since the data (including the verbal This general picture is compatible with all
data) are gathered to test theories about the sorts of specific hypotheses that have been
human information-processing system, we are put forth with respect to the details of the
engaged in something of a bootstrap opera- mechanisms. For example, some theorists pro-
tion. We need a model to interpret data that pose that what we call "short-term memory"
are to be used to test the model. is not a separate, specialized store but simply
Under these circumstances our data-in- a portion of LTM that is currently and tem-
terpretation model should be as simple as porarily activated. Some theorists believe that
possible, and it must not incorporate com- information in STM extinguishes with pas-
ponents that are themselves bones of theoreti- sage of time, unless rehearsed; others believe
cal contention. The model should be robust, that it is lost only when replaced. In general,
that is, compatible with a wide range of these differences of detail do not affect the
alternative assumptions about human infor- model at the level of specificity required for
mation processing. our purposes. The important hypothesis for
The specifications we are about to present us is that due to the limited capacity of STM,
are simple and robust in this sense, and, in- only the most recently heeded information
deed, summarize the core that is common to is accessible directly. However, a portion of
most current information-processing theories the contents of STM are fixated in LTM
of cognition. Of course, they are not entirely before being lost from STM, and this portion
neutral, for they would be hard to reconcile can, at later points in time, sometimes be
with an extreme version of behaviorism that retrieved from LTM.
denied the relevance of central processes to We assume that any verbalization or verbal
the explanation of behavior. But they do not report of the cognitive process would have to
represent the view of any particular "sect" be based on a subset of the information in
within the general information-processing these memories. From this and the earlier
tradition. mentioned hypotheses, the taxonomy of ver-
The most general and weakest hypothesis balization procedures shown in Table 1 fol-
we require is that human cognition is infor- lows in a straightforward fashion. The tax-
mation processing: that a cognitive process onomy provides us with a theoretical founda-
can be seen as a sequence of internal states tion for some of the distinctions we have
successively transformed by a series of in- already made in types of verbalization.
formation processes. An important and more The two dimensions of Table 1 represent
specific assumption is that information is two major distinctions. First, the time of ver-
stored in several memories having different balization is important in determining from
capacities and accessing characteristics: sev- what type of memory the information is likely
eral sensory stores of short duration, a short- to be drawn. Second, we make a distinction
term memory (STM) with limited capacity between procedures in which the verbaliza-
and/or intermediate duration, and a long- tion is a direct articulation or explication of
term memory (LTM) with large capacity and the stored information and procedures in
relatively permanent storage, but with slow which the stored information is input to in-
fixation and access times compared with the termediate processes, such as abstraction and
other memories. inference, and the verbalization is a product
Within the framework of this information- of this intermediate processing.
processing model, it is assumed that informa-
tion recently acquired (attended to) by the Detailed Specification
central processor is kept in STM and is di- We must now specify more fully the com-
rectly accessible for further processing (e.g., ponents, which we have just sketched, of the
for producing verbal reports), whereas in- information-processing system that carry out
formation from LTM must first be retrieved the processes of verbalization. The model
224 K. ANDERS ERICSSON AND HERBERT A, SIMON

Table 1
A Classification of Different Types of Verbalization Procedures as a Function of Time of
Verbalization {Rows) and the Mapping From Heeded to Verbalized Information (Columns)

Relation between heeded and verbalized information

Intermediate processing
Time of verbalization Direct one to one Many to one Unclear No relation
While information is Talk aloud
attended Think aloud
While information is Concurrent Intermediate inference and generative processes
still in short-term probing
memory
After the completion Retrospective Requests for Probing Probing
of the task-directed probing general reports hypothetical general states
processes states

draws on a variety of sources that are sum- of links that associate these nodes to others
marized in Newell and Simon (1972, chap. that have already been activated. Informa-
14) and Simon (1979, chap. 2.3). tion accessed by association is then also repre-
Few of the model's specifications are con- sented by pointers in STM. Thus, informa-
troversial. It makes no real difference, for tion can be brought into STM from sensory
example, whether we assume a single homo- stimuli via the recognition process, or from
geneous memory with different modes of ac- LTM via the association process. Association
tivation (e.g., Anderson, 1976; Shiffrin & processes are much slower than direct recogni-
Schneider, 1977) or several discrete mem- tion processes, requiring at least several hun-
ory stores (sensory stores, STM, and LTM). dred milliseconds for each associative step.
The important matters, which can be de- Associative processes may use STM to store
scribed in either terms, relate to the amounts intermediate steps. So, for example, in re-
and kinds of information that can be re- calling a name that is not immediately ac-
tained and the conditions for accessing them cessible, a person may use a sequence of cues
and reporting them verbally. We will use the to find an associative path, step by step, to
conventional model of multiple memories in the sought-for name. Such processes may last
our description. tens of seconds, or even minutes, and may
Recognition. Information received from leave numerous intermediate symbols in STM,
the sensory organs resides for a short time in where they are temporarily available for ver-
memories (iconic and echoic) associated with bal reports.
the different senses. During this time, por- Short-term memory. The CP, which con-
tions of the sensory information are directly trols and regulates the nonautomatic cogni-
recognized and encoded with the aid of in- tive processes, determines what small part of
formation already stored in LTM. Recogni- the information in sensory stimuli and LTM
tion associates the stimulus, or some part of finds its way into STM. This is the informa-
it, with existing patterns in LTM and stores tion that is heeded or attended to. The amount
in STM "pointers" to those familiar patterns. of information that can reside in STM at
Intermediate stages of the direct recognition one time is limited to a small number (four?)
process, which may take only 50-100 msec, of familiar patterns (chunks). Each chunk
do not use STM to store their products. is represented by one symbol or pointer to
Long-term memory. The LTM may be information in LTM (Simon, 1979, chap.
pictured as an enormous collection of inter- 2.2). As new information is heeded, informa-
related nodes. Nodes can be accessed either tion previously stored in STM may be lost.
by recognition, as just explained, or by way When a cognitive task (e.g., mental addi-
 VERBAL REPORTS AS DATA 225

tion of a column of figures) is being carried tion to be less complete than reports of an
out, the typical chunks in STM are pointers orderly process that is induced by the succes-
to the operands, operators, and outputs of the sive contents of STM itself (e.g., a thought
operations that are being performed. Thus, in sequence during which goals in STM are
adding 3 to 4, pointers corresponding to the guiding the thought processes).
symbols "3," "4," "PLUS," and "7" might Fixation. New information is retained in
at some time be present in STM. Since, in STM during the time the CP is attending to
our culture, adding two digits involves a di- it. To create an LTM representation of new
rect reference to LTM ("table look up"), no information that can later be recalled, associa-
further details of the process would be heeded tions must be built up by coding and imaging,
in STM or available for verbal reports. On as well as new tests and branches in the recog-
the other hand, if the task were to multiply nition network. Processing of the order of
17 by 45, STM might hold, at various points 8-10 sec is required to assemble each new
in the process "45," "17," "7," "TIMES," chunk from its familiar components in STM,
"3" (the carry in.multiplying 45 by 7), "315" and to store it in LTM as a new chunk (Si-
(the first intermediate product), "45," "1," mon, 1979, chap. 2.2, 2.3).
"TIMES," "PLUS," "765." Automation. As particular processes be-
We hold no brief for the details of the come highly practiced, they become more and
earlier description, which is intended merely more fully automated (Shiffrin & Schneider,
as an example of the kinds of information we 1977). Automation means that intermediate
would expect to be heeded in STM, and to steps are carried out without being inter-
be available, potentially, for concurrent or preted, and without their inputs and outputs
retrospective reports. The specific details using STM. The automation of performance
would depend on the particular strategies is therefore analogous to executing a com-
subjects used and the nature of the chunks puter algorithm in compiled instead of in-
stored in LTM (Simon, 1979, chap. 2.4). terpretive mode. Automation and compil-
STM would symbolize the process only down ing have two important consequences. They
to some modest level of detail (corresponding greatly speed up the process (typically, by
to elementary processes of a sec or 2 in dura- an order of magnitude), and they make the
tion), and we would not expect to find infor- intermediate products unavailable to STM,
mation there about simple, automated pro- hence unavailable also for verbal reports.
cesses (e.g., the processes of retrieval from
LTM or recognition processes), much less Verbalization Processes
about neuronal events. Thus, the architec- Within the context of this general model,
ture of the control apparatus (CP) determines verbalization processes produce (externalize)
the fineness of grain of the representation of information that is in STM. In the case of
processes in STM. thinking-aloud instructions, the information
Control of attention. The flow of attention verbalized will then be some portion of the
is diverted, from time to time, by interrup- information currently being attended to. The
tions through the higher control mechanism. verbal production process takes two rather
Intermediate stages in these interruptions, different forms, depending on whether the
not being symbolized in STM, are not re- STM chunks already denote symbols in the
portable. Sudden movements in peripheral verbal mode. For information that can be
vision, loud noises, and emotions operating represented as a string of phonemes, that is,
through the reticular system are important aural information, the model assumes that
causes of interruption and shift in attention attending to that information or activating
(Simon, 1979, chap. 1.3). Even though in- the corresponding structure in LTM allows
formation heeded immediately before or after the information to be vocalized by automatic
a shift in attention may sometimes allow sub- verbal translation without making additional
jects to give a relatively clear account of the demands on STM or the CP.
interruption, we would expect such informa- Intermediate processes. When information
226 K. ANDERS ERICSSON AND HERBERT A. SIMON

in STM is not verbally encoded (e.g., visual is confused with the events being queried, and
imagery), making a verbal report requires, hence is inaccurate. The degree to which
according to the model, the corresponding retrospective verbalization must rely on re-
verbal representations of the information (in trieval from LTM can be minimized by study-
the simplest case, names or labels) to be ing cognitive processes of short duration,
evoked. The recoding processes will make at where the verbal responses lag the task pro-
least modest demands on processing capacity cesses by only a brief interval.
and processing time. This means that some
heeded information may not be vocalized Empirical Evidence on Verbalization
when other task-directed processes take prior- In the remainder of this article, we will
ity and interrupt the verbal encoding and develop in more detail our model of verbaliza-
production processes. tion and test it against empirical evidence. We
The verbal-encoding processes involved in will show that the empirical findings are com-
thinking aloud evoke a verbal reference to, patible with the assumptions of our model,
and occasionally an explication of, the heeded and that the model provides guidelines for
structure in STM. These processes would not the interpretation of verbal reports gathered
be predicted to change the information at- under various procedures.
tended to in the way that requested explana- Since the relevant literature is voluminous,
tions would. (When subjects are asked for we cannot review it all explicitly within the
explanations, the verbal reports cannot be compass of a journal article. Instead, we will
generated without extending the information select representative studies for discussion,
and relations heeded.) Hence, thinking aloud, leaving a more exhaustive survey to other
as distinguished from explanation, will not papers (Ericsson & Simon, in press; Erics-
change the structure and course of the task son & Simon, Note 2, Note 3). However, in
processes, although it may slightly decrease our process of selection we will be careful not
the speed of task performance. to screen out studies that are troublesome for
Retrospective reports. The most general our theoretical framework.
retrospective verbalizing instruction asks the The classical issues we will discuss in the
subject to report everything he or she can next three sections are (a) the effects on the
remember about the cognitive process studied. cognitive processes of the instruction to ver-
If the subject is asked immediately after per- balize and of probes, (b) the completeness of
forming the process, the model predicts that verbal reports, and (c) the consistency of
some previously heeded information will still verbal reports with other empirical data on
be in STM, permitting direct reporting by behavior. These are the central questions that
the processes described earlier, and facilitat- must be settled if verbal reports are to be
ing retrieval of additional information stored used as data in psychological research. In ad-
in LTM in episodic associations that were dition, in experiments in which verbal reports
formed when the information was heeded. are obtained, it is important to raise two
The control process in retrieving previously further questions: (d) the generalizability
heeded information from LTM, however, may and validity of the verbalized information
be rather variable. Since it is outside the and (e) the design of objective methods for
scope of this article to review the existing encoding and analyzing think-aloud protocols.
evidence on factors affecting the efficiency We have addressed these latter questions in
of retrieval, we will limit ourselves to a few several working papers (Ericsson & Simon,
comments. In situations in which similar in- Note 2, Note 3) and will not pursue them
formation is attended to over and over, as in further here.
experiments with factorially designed stimu-
lus material, the model would predict that Effects of Probes and Instructions
retrieval of specific items will be hampered by to Verbalize
extensive interference. Retrieval may not only
fail to access previously heeded information It is often asserted that the mere procedure
but may on occasion access information that of eliciting verbal reports changes the course
 VERBAL REPORTS AS DATA 227

and structure of the cognitive processes that When the information being processed in
are under study. If that were so, the verbal order to perform the main task is not verbal
data, even if they reflected accurately the or propositional, the model predicts that the
cognitive processes going on during verbaliza- performance may be slowed down, and the
tion (and hence provided significant and valid verbalization may be incomplete but that the
data about cognition), would give an inac- course and structure of the task-performance
curate picture of the "normal" course of those process will remain largely unchanged. Sev-
processes. eral kinds of tasks fall in this category.
In studies in which the probing and the The performance may be highly automated,
verbal reporting take place at the end of the hence may not make much use of STM. This
experiment, it may be argued that since the case includes acts of recognizing familiar
subjects cannot be aware of the fact that they stimuli, and more generally, many kinds of
are subsequently going to be asked to report tasks after long practice. For such tasks, it
on their processes, the reporting task cannot is most likely that the thinking-aloud proto-
affect those processes. However, in studies cols will be very sketchy, but that the pro-
in which the subjects are told explicitly in cesses will not be slowed down or altered. We
their initial instruction that they will be would also expect a more frequent injection
questioned about general (Morgan, 1934) or of metastatements (explicit statements about
specific (Rommetveit, 1965) aspects of the the process itself) replacing statements about
experiment, the possibility that these instruc- inputs and outputs in the protocols.
tions will affect their cognitive processes dur- Similar predictions can be made for tasks
ing the experiment cannot be ruled out on with a large motor-perceptual component and
logical grounds. The possibility of induced tasks employing complex visually encoded
effects is even greater when the probing pro- stimuli. If the task performance is not highly
cedure requires subjects to give reports peri- automated, then the subjects, in their en-
odically during the experimental sessions. deavors to obey the thinking-aloud instruc-
Nevertheless, the greatest concern about tions, may take time to translate their inputs
possible effects of verbalization on the course and outputs into verbal form, and to report
of the cognitive processes arises when the them, but at the expense of slowing down their
verbalization is concurrent with the task per- performance of the task.
formance. For that reason we will concentrate, In studies in which subjects are not merely
in this section, mainly on concurrent verbali- asked to think aloud but are asked for spe-
zation. cific kinds of information—for example, the
reasons for their subsequent actions and
Predictions for Concurrent Verbalization moves (Gagne & Smith, 1962; Wilder &
Our model of concurrent verbalizing as- Harvey, 1971)—their efforts to obey the in-
sumes that the verbalizations involve either structions would be predicted to have more
direct articulation of information stored in a substantial effects on task performance. The
language (verbal) code 2 (Level 1 verbaliza- model would predict that these effects would
tion); articulation or verbal receding of non- be especially prominent if the subjects were
propositional information without additional asked to produce information that would not
processing (Level 2 verbalization); or articu- normally be available to them during their
lation after scanning, filtering, inference, or performance of the task. If the information
generative processes have modified the infor- requested was information that would nor-
mation available (Level 3 verbalization). mally be available, then the model would pre-
When the subjects articulate information dict that the effects of the verbalization in-
directly that is already available to them, structions would be relatively minimal. The
the model predicts that thinking aloud will
not change the course and structure of the 2
Hereafter, we will refer to information stored in
cognitive processes. Nor will verbalization un- memory in essentially propositional form as "ver-
der these conditions slow down these processes. bally encoded."
228 K. ANDERS ERICSSON AND HERBERT A. SIMON

prediction depends, therefore, on the process with 20 control subjects. The subjects were
used to perform the task; and conversely, the divided into two matched groups on the basis
degree to which verbalization changes task of 10 preliminary problems, and were then
performance can be used as a cue to determine given IS experimental problems for which the
what that process is. experimental group was asked to think aloud.
In studies using probes, the same distinction Finally, 5 problems were given for which all
can be made between probes requesting spe- subjects were instructed to be silent, to allow
cific information to be verbalized (and occa- exploration of aftereffects of thinking aloud.
sionally specifying even the information's The stimuli in Karpf's (1973) experiment
form and completeness) and more general and were slides, each containing a pair of letters
nondirective probes. The latter would affect varied along eight dimensions, such as form
the process less than the former. The effects (A or U), size (large or small), color (black
on task performance would be especially large or white), shape of the border surrounding
if the probes rested on a theory or concep- the letter (circle or square), and texture of
tual framework that was not a veridical or line under the letter (solid or dotted). The
adequate description of the information avail- letters in each pair were discriminated by
able directly to the subjects. simple hypotheses, involving a single dimen-
It is worth emphasizing again that the rela- sion. Although it would be expected that sub-
tion between main task and reporting task jects would process these stimuli in pictorial
is mutual. The predictions of the model can rather than verbal mode, it is fairly easy to
be used to test hypotheses about the task pro- recode them verbally; hence, the model would
cesses, just as predictions from the latter can predict that thinking aloud would affect, at
be used to test hypotheses about the model most, only the speed of the task performance.
of verbalization. Once we have acquired some This is, indeed, what Karpf reported. No
confidence in the verbalization model, we find reliable differences were found between the
that differences between subjects in thinking- thinking-aloud group and the control group,
aloud and silent conditions have implications for either the experimental problems or the
for the processes that are being used to per- final problems, in numbers of problems solved
form the main task. correctly. However, the thinking-aloud group
In considering empirical studies that em- took about 50% more time than the control.
ploy thinking-aloud procedures, we will be Supporting evidence comes from a study by
concerned primarily with studies that meet Roth (1966), who found that verbalization
the criteria of Level 2 verbalization, for most had no effect on the effectiveness of task per-
of the experiments in the literature deempha- formance—but in his experiment also, there
size speed and instruct the subject to "take was no effect on speed of performance. (For
your time and concern yourself with perform- similar findings, see Carroll & Payne, 1977;
ance." This does not mean that verbalizing Feldman, 1959; Kazdin, 1976; Johnson &
cannot remain at Level 1, even for complex Russo, Note 4).
tasks, but that in most cases the additional In a study of mental multiplication, Dan-
information obtainable when receding is per- sereau and Gregg (1966) asked a subject to
mitted is judged to be more important than verbalize each step during the solution pro-
strict invariance of performance. cess. Moreover, whenever the subject re-
mained silent too long, the experimenter urged
Studies oj Level 1 and 2 Verbalization him to talk. On a wide range of problems
Of the experiments explicitly designed to varying in difficulty, no reliable differences in
study the effects of verbalizing, only a few speed of performance were found between a
have exposed the experimental and control silent control condition and the verbalizing
groups to identical conditions. An important condition. In a subsequent study, Dansereau
example is a study of discrimination learning (1969) reported that with increased practice,
by Karpf (1973), in which he compared 40 two of his faster subjects gave overt verbali-
subjects who were instructed to think aloud zations only of intermediate results and the
 VERBAL REPORTS AS DATA 229

initial problem, and they reported introspec- could be compared (Ericsson, 197Sa, 197Sb).
tively that complete verbalization interfered Various subgoals can be defined for the task
with retrieving information. To what extent (e.g., arranging the tiles in the first row cor-
these subjects felt that verbalizing slowed rectly). No differences were found in the sub-
them down is not reported, nor is a compari- jects' attainments of such subgoals, nor in
son provided with performance under a silent the structure of their search trees. However,
control condition. for the first several problems, the verbalizing
A number of other studies allow compari- subjects made a larger average number of
sons between a silent and a vocalizing group, moves than did the silent subjects. Thus, it
for identical or similar tasks, but without appears that there was some tendency for the
complete comparability between experimental silent subjects to do more planning and think-
and control conditions. For example, for sub- ing ahead than the verbalizing subjects.
jects discovering proofs in prepositional logic, Whether this difference was due to the ver-
Newell and Simon (1972) compared the num- balizing, to a feeling of "irreversibility" in
ber of solutions attained and the detailed typing the move to the computer, or to some
solution paths of their seven think-aloud sub- other cause cannot be determined from the
jects with the solutions (collected by different experiment.
investigators at Yale) to the same two prob- This sample of studies comparing problem-
lems by 64 subjects under silent conditions. solving behavior under thinking-aloud and
The data that could be compared between the silent conditions illustrates that the model
two conditions were the actual steps taken predicts correctly that verbalization does not
while searching for the proof—the entire affect the behavioral manifestations of the
search tree, including both the correct paths thought processes when the conditions for
and the unsuccessful attempts. When the de- Level 1 verbalization are satisfied; generally,
tailed structures of the search trees were it only affects the speed of performance when
compared between the two groups, no differ- the stimuli are nonverbal but easily receded
ences were found. Both groups explored es- for verbal reporting. Where other effects of
sentially the same parts of the problem space verbalization are found (as in the Ericsson
with about the same relative frequencies (and experiment), they can reasonably be attrib-
found the correct solutions about as often). uted to other differences between the silent
The stimuli here were symbolic expressions and thinking-aloud conditions. These findings
that are easily described in words. suggest that the internal structure of the
Similar results were obtained by Ericsson thought processes also is not changed as a
(1975a, 197Sb) in a study of problem solv- result of the verbalizing activity.
ing with the 8-puzzle, a task in which small,
numbered tiles must be manipulated into a Studies Not Meeting Level 2 Conditions
desired arrangement. The task has a strong When the criteria for Level 2 verbalizing
visual-perceptual component, but moves are are not met, so that the subject is asked to
easily encoded in symbolic form (e.g., "Move verbalize information that would not be
7 up," "Move 3 left.") so that it is not dif- heeded in the normal course of processing or
ficult to verbalize them. that could not easily be encoded in a verbal
Two separate experiments were conducted code, our model predicts that the course and
for the thinking-aloud and silent conditions, structure of the cognitive processes may be
but the same sequence of puzzles was pre- changed by the verbalization.
sented to the subjects in both conditions. In Not only is it possible that verbalization
the thinking-aloud experiment, the subjects will change the thought process, but con-
also had to tell the experimenter which tiles versely, it is likely that the nature of the
to move; in the silent condition, the subjects, task will cause the subjects to give a different,
sitting alone, pressed keys on a teletype to and generally less complete, account of their
cause the computer to make the moves. The processes than under Level 2 conditions. We
sequence of moves made in the two conditions will postpone to the next main section of this
230 K. ANDERS ERICSSON AND HERBERT A. SIMON

article the discussion, of the completeness of (1957) found that requiring subjects to tell
verbalizations, and will be concerned here the experimenter what they were going to
only with how the attempt to verbalize will do before they started to manipulate the
affect task performance. switches significantly decreased the number of
Verbalization of perceptual-motor processes. overt trials to solution (but the correspond-
The problems of verbalizing perceptual-motor ing solution times are not reported).
processes are most clearly visible in problem Most of these results could be explained by
situations in which the problem is represented the hypothesis that in response to the instruc-
physically (e.g., the disks and pegs of the tion to "think" or verbalize, the subjects did
Tower of Hanoi puzzle), and performance in- not change the structure of their processing,
volves manipulation of this physical repre- but simply substituted nonobservable covert
sentation. Because verbalizations are often moves for the overt moves (the measured in-
quite sketchy in tasks of these kinds, experi- dex of performance). However, it is reason-
menters sometimes change the task to increase able to assume also that an internal repre-
verbalization. For example, if the subjects sentation generated for the covert processing
must instruct the experimenter to make the improves memory and the organization of the
manipulations for them, instead of making processing.
them themselves, they are thereby forced to Verbalization can be encouraged not only
form an internal representation of the moves by constraining the manipulations of the
that can be encoded verbally. When this is problem material but also by instructing sub-
done, more of the content of the thought jects to verbalize the motives or reasons for
processes is, in fact, verbalized. their actions. Our model predicts that such
To increase verbalization of content, some instructions will likely change the course of
experimenters using the thinking-aloud method processing. In general, when the thinking-
change the task by constraining the manipula- aloud instructions do not require verbaliza-
tions partially (Durkin, 1937) or wholly tions of motives or reasons, the protocols do
(Benjafield, 1971), thereby forcing the sub- not contain them. Three studies in which
jects to form an internal representation of the subjects were specifically instructed to make
content that can be encoded verbally. When such reports illustrate the effects that may
this is done, more of the content of the be expected.
thought processes is, in fact, verbalized. The study by Gagne and Smith (1962)
From a behavioral point of view, constrain- with the Tower of Hanoi problem was aimed
ing manipulation significantly changes the at investigating the effects of different ver-
task in that the overt moves have now be- balization instructions on performance during
come covert and are no longer amenable to some training tasks (two-disk to five-disk
direct observation. The subjectively perceived problems) and on transfer to a similar but
polarity between thinking and manipulation, more complex task (six-disk problem). One
mentioned earlier, may very well correspond of the two factorially combined manipulations
to this difference between covert and overt during the training tasks required the subjects
trials. Duncan (1963) showed that giving to state verbally a reason for each move. This
subjects an explicit instruction to think re- requirement greatly improved performance
sulted in significantly fewer overt trials in a on the transfer task, both as to number of
switch-setting task but an actual increase in moves required and time taken to find a solu-
solution time. Hence, the covert processing, tion. As a second manipulation, the subjects
in the condition that induced more planning in one pair of conditions were also instructed
before manipulation, took longer than the cor- to search for a general principle behind the
responding overt trials. A similar trade-off different versions of the problem. This in-
between speed and quality of performance was struction did not affect performance on the
noted by Shipstone (1960) in a concept-learn- transfer task.
ing task, with an instruction to disregard In the training tasks, the overt verbaliza-
speed and concentrate on what to do. Ray tion group produced more efficient solutions
 VERBAL REPORTS AS DATA 231

(solutions having fewer moves), indicating the test problem. The hypothesis that the
that the instruction to verbalize the reasons "think-aloud" instruction would influence per-
induced more deliberate planning, in addition formance by interacting with the presence or
to its effect on transfer. Although no formal absence of the experimenter was not borne
record was kept of the time taken for each out. The extent to which these effects were
move, the experimenters judged this to be mediated by differences in solution times can-
longer for the overt verbalization group, but not be discussed, since the published report
they reported that this extra time was " 'filled' gives no information about these times.
with time, taken up entirely with the act of These three studies show that as predicted,
verbalization" (Gagne & Smith, 1962, p. 17). a requirement to verbalize reasons and mo-
They suggested that the instruction to ver- tives has substantial effects on both im-
balize the reasons for the moves affected per- mediate performance and learning, and that
formance by forcing the subjects to think. generating verbalized reasons brings about
In a follow-up study with the same basic changes, at least in manipulative tasks, in the
design, Wilder and Harvey (1971) investi- course of the processes. Here, as in the studies
gated whether the overt verbalization was discussed earlier, we do not know to what ex-
crucial, or if equivalent results could be ob- tent forcing subjects to give reasons for their
tained with a firm instruction to state the actions causes them to substitute unrecorded
reasons covertly; in addition, they checked covert trials and planning for overt trials. The
the time taken to achieve solutions during the negative result of the Davis et al. (1968) ex-
training tasks. The results showed no differ- periment is most readily interpreted as show-
ence between the overt and covert verbalizing ing that in a problem-solving situation with
conditions but a clear reduction in the num- a heavy cognitive load, initial instructions
ber of moves in both those conditions, as may be disregarded by subjects unless they
compared with a control condition. The time are monitored by the experimenter. An anal-
taken to solution did not differ among the ysis of the content of the verbalizations, not
three conditions, during either the training provided by the authors, would be required
tasks or the final task. This finding eliminates to test this explanation. Finally, we may con-
the hypothesis that the advantage in transfer jecture that the richness of alternative strat-
shown for the verbalization condition in this egies for the Tower of Hanoi problem prob-
experiment was attributable to extra learning ably increases, in comparison with other tasks,
time during the training sessions. the sensitivity of thought processes to instruc-
In another follow-up of Gagne and Smith tions to verbalize reasons.
(1962), Davis, Carey, Foxman, and Tarr Verbalization of visual encodings. There
(1968) included the presence of the experi- is compelling evidence to support the distinc-
menter as an additional dimension in a fac- tion between a visual representation or code
torial design. In this study the subjects were and a verbal or symbolic representation or
instructed at the beginning of the experiment code when subjects are presented with draw-
to verbalize their reasons, but the instruction ings or pictures. An instruction to describe a
was not repeated. (In the earlier experiments, visual scene verbally should require a verbal
the subjects were closely monitored during receding of the picture, which will imply ex-
the entire session to make sure they followed tensive processing. Our model predicts that
the instructions.) In the Davis et al. study, this additional processing may have three
verbalization had no effect on the training kinds of effects: It may slow down perform-
task (five-disk problem), but the "think- ance of the main task, it may change the
aloud" subjects required significantly fewer structure and course of performance of the
moves on the test problem (six-disk prob- main task, and it may influence what is re-
lem), even though no subjects were asked to membered about the task and is later avail-
verbalize on the latter task. The experimen- able to retrospective verbalization. The mag-
ter's presence facilitated performances in both nitude of the effects will depend on how fully
conditions on the training problem but not on the subjects carry out the instruction to ver-
232 K. ANDERS ERICSSON AND HERBERT A. SIMON

balize, and on how difficult it is to describe formance in intelligence tests was investigated.
the visual scene in words. We have already For example, Resting, using the Figure Rea-
seen that when the latter is relatively easy, soning Test, reported that 13- to 17-year-old
the only detectable effect of verbalization may subjects who always had to say aloud how
be to slow task performance. the figures were alike or different performed
The task of viewing a novel visual scene significantly better than the subjects who
has been studied to detect effects of imposing were asked to say the same thing to them-
the additional task of describing the scene selves silently, but they were also significantly
verbally. A study by Freund (described in slower. In this study, there were also two
Loftus & Bell, 1975) showed that subsequent other conditions with additional verbal inter-
recognition of scenes was much improved by ference tasks: One group of subjects had to
the verbalizing requirement. On the other say "eins, eins, . . ." rhythmically, whereas
hand, when an unrelated verbal task, such as another group had to sing "la, la, ... " while
counting backwards by threes, is imposed, solving the test items. These conditions
the subsequent recognition of scenes deteri- yielded the same performance (in terms of
orates as compared with normal viewing, but number of correct solutions and time re-
not to a chance level (Loftus, 1972; studies quired) as the silent verbalizing condition,
of Freund and Szewczuk described in Loftus which suggests that the same processes,,
&Bell, 1975). mainly nonverbal in character, were used in
Evidence supporting the distinctness of the all three conditions.
visual and verbal codes comes from an in- To test the hypothesis that the instruction
vestigation by Schuck and Leahy (1966) on to verbalize made subjects assume a more
fragmenting visual images. They found that analytic problem-solving style, Hofgen (cited
subjects reporting the disappearances verbally in Merz, 1969) compared performance on
tended to report omissions of meaningful parallel forms of the Figure Reasoning Test
complete segments, whereas control subjects between a group that had previously verbal-
who traced the disappearance on an outline ized on an initial form and a control group
of the image did not. that had not. The verbalizing group per-
A number of studies provide evidence of formed significantly less well when not re-
large individual differences in preferences for quired to verbalize than before, but still
perceptual versus verbal processing (Ericsson somewhat better than the control group,
& Simon, Note 3). Our model would predict whose performance hardly differed between
that requiring subjects to verbalize explana- the two occasions.
tions in a task with a complex visual stimulus Inner speech (measured by electrical ac-
would cause subjects with preference for per- tivity in the speech apparatus) during per-
ceptual processing to alter their strategies formance on Raven's Progressive Matrices
and hence their performances. For example, items without overt verbalization increased
in a study by Brunk, Collister, Swift, and with difficulty of the items (Sokolov, 1972).
Stayton (1958), subjects were given an initial Analyzing the protocols from verbal recon-
test of the Vygotsky type and then a second, structions of the problem-solving process,
similar test. In one condition on the second Sokolov showed that the simple items were
test, each subject was "requested to tell why solved in a predominantly visual way, whereas
he placed each block where he did" (p. 238). with the more difficult problems verbal desig-
In a control condition, no such explanation nations of some features of the figures were
was requested. The correlation of subject used to aid solution. In outlining a scheme
scores between initial and second test was for the interplay of visual and verbal pro-
significantly lower under the instruction to cesses in solving such problems, Sokolov
explain than under the control condition, as pointed to the influence of verbalization in
the model would predict. attending to features that would have gone
In a series of studies reported in Merz unnoticed in the purely visual analysis. In this
(1969), the effect of verbalization on per- interpretation, the directed verbalization pro-
 VERBAL REPORTS AS DATA 233

vides the subject with additional noticed fea- occur in the verbalizing condition. The num-
tures, which in turn facilitate performance. ber of irrelevant dimensions of the stimuli
There is little evidence as to whether di- was varied, although the instructions indi-
rected verbalizing has a general or a differen- cated which two features were relevant to the
tial effect on subjects. A general effect is sug- solution in each case. The requirement to ver-
gested by the fact that the variance of the balize hypotheses significantly improved per-
performance with verbalizing is equal to or formance (i.e., number of responses to cri-
less than the variance of performance under terion), but only for the first problem. Bower
control conditions in the studies of Resting and King found that variation of the number
and of Waszak and Hofgen (cited in Merz, of irrelevant features or dimensions affected
1969). The investigation by Sokolov indicates only the initial problem, suggesting that the
differences between subjects, leading Sokolov verbalizing of hypotheses helped the subjects
to propose a differential reliance on verbal initially to ignore the irrelevant attributes. It
and visual processing, but this result is not should be noted that no training trials were
incompatible with the idea that there may also used in this study.
be general effects on all subjects. In a cue-probability learning task, Brehmer
(1974) required one group of subjects to de-
Other Concurrent Verbalization Studies scribe the rule underlying their predictions,
just after each prediction was made but be-
None of the studies that remain to be dis-
fore feedback was received. The subjects' de-
cussed employed highly manipulative tasks
scriptions were to be so explicit that another
or pictorial stimuli.
subject could understand and use it; if they
In a study on clinical judgment (Baranow-
did not meet this standard, the experimenter
ski, Note 5), the subjects (who were psychol-
prompted for more information. Explanations
ogists) made two successive series of judg-
like "I guessed" or "I remembered from the
ments. On the first occasion, all subjects
previous trial" were accepted as verbal de-
performed the task under identical conditions.
scriptions. An analysis of variance showed no
On the second occasion, the subjects were
significant effect or interactions associated
divided into (a) a group working under the
with verbalization. In a subsequent study
same instructions as on the first occasion and
(Brehmer, Kuylenstierna, & Liljergren, 197S),
(b) a group instructed to verbalize and moni-
the subjects wrote down their current hy-
tored by the experimenter, who asked the
potheses in a booklet at the beginning of the
subjects questions whenever "a particular
test blocks, without any significant effect on
profile could use more explanation" (p. 21).
performance.
There was no difference between control and
verbalizing conditions (measured by variance According to our model, requiring verbal
accounted for by linear and nonlinear models). explanations of behavior should not alter the
However, the cross-validated linear models normal processes unless the information re-
over the two occasions accounted for signifi- quired for the verbalizations would not other-
cantly less variance for the verbalizing group wise be generated. Unfortunately, there is
than for the control, suggesting that the in- little evidence for the tasks used in the above
struction to verbalize changed the utilization cited studies about the content of undirected
or subjective weights of the cue variables. verbalizations. In the cue-probability experi-
In a concept-learning study by Bower and ment of Brehmer (1974) with very simple
King (1967), one group of subjects was re- stimuli (a straight line varying in length),
the number required for explaining the rule
quired to verbalize their hypotheses before
classifying the stimuli, but a control group was most likely consciously generated even
was not. In preparation for the experiment, in the silent condition. That the effects of
the subjects described the stimuli to ensure verbalization were limited to the first trial in
that subject and experimenter agreed in their the Bower and King (1967) experiment could
descriptions. Under these circumstances, we be attributed to the fact that verbalizing
would expect that no further encoding need helped the subjects ignore irrelevant features.
234 K. ANDERS ERICSSON AND HERBERT A. SIMON

Alternatively, one might speculate that ver- explicitness called for by the instruction to
balizing may have speeded up the generation verbalize. Sowder (1974), examining the ef-
of an internal representation, thus making the fects of various sorts of verbalizations of
subjects more independent of their direct per- learned generalizations, found no differences
ceptions. In the study of Baranowski (Note as compared with a control condition. He also
5), unlike the other studies, the subjects were cited two studies in which no effects of pro-
highly skilled. In our model, verbal explana- ducing written descriptions of learned gen-
tion of automated activities would be cum- eralizations were found. Sowder proposed that
bersome and would change the course of the important difference between his study
the processing from a largely perceptual (rec- and Hendrix's (1947) was that Hendrix re-
ognition) to a more cognitive one. In support quired complete specification of the content
of this hypothesis, the time taken by the of her subjects' verbalization (quantifiers, do-
clinical psychologists to perform the task main, and so on), whereas he did not.
with verbalization was two or three times the Rommetveit (1960, 1965) and Rommetveit
time taken in the silent condition. and Kvale (1965a, 1965b) studied concept
formation in a situation in which 12- to 13-
Ejects From Retrospective Verbalization year-old subjects played on a wheel of fortune
with different pictures being displayed when
With this summary of effects of concurrent
the subjects were to win or lose, respectively.
verbalization (or the absence of such effects),
They found that instructing the subjects that
we turn now to the topic of retrospective ver- they were subsequently to describe the differ-
balization. We will first consider some experi-
ences between the "win" and "lose" figures,
ments that are often cited to support the idea as opposed to just playing on the wheel, in-
that attempts to verbalize information may
fluenced subjects' retrospective descriptions
change and deform it, and hence affect sub-
of the two figures. Other procedural varia-
sequent task behavior.
tions, such as demonstrating before the ex-
Hendrix (1947) showed that an instruction
periment how the figures differed (Rommet-
to describe a concept or principle verbally
veit, 1965), tended to eliminate a (correct)
after learning it caused a decrement in abil-
tendency toward associating roundness with
ity to use the concept in a transfer situation.
good figures. In these studies, therefore, the
These results were substantiated in subse-
effective variable is not verbalization per se
quent work by Phelan (1965). Careful anal-
but directing the cognitive processes by the
ysis shows that these studies do not address instructions. Without such direction, verbali-
the question of verbalization, as such, but
zation seems to have no effect on the cognitive
rather verbalization of explicit and logical processes.
concepts. There are two issues. The first is
that if the subjects do not normally organize A number of studies have not found any ef-
what they learn in these experiments in ver- fects from instructions to give verbal reports.
balizable concepts and general principles, then In a series of studies of probabilistic infer-
verbalization forces them to generate such ence, Brehmer (1974) and his co-workers
concepts and principles from whatever infor- have investigated the effects of asking their
mation is currently available to them. The subjects to describe retrospectively their hy-
reformulation may not at all reflect the way potheses about the relations between the cue
in which the learning was actually encoded. and the criterion. The subjects were asked to
For example, Phelan found that the verbal describe, explicitly enough for someone else
descriptions of certain pictorial stimuli tended to make the predictions, their rules for arriv-
to contain discriminative features different ing at the prediction from the cue value.
from those that defined the concept the sub- However, subjects were free to report that
jects had learned. In our discussion of in- they were guessing, were remembering from
completeness of verbal reports, we will return earlier trials, and so on. In a factorial study
to this issue. (Brehmer, 1974) in which subjects gave de-
The second issue relates to the detail and scriptions from trial to trial, no main effects
 VERBAL REPORTS AS DATA 235

or interactions could be attributed to ver- balized. In our model, as in most theories of

balization. the structure of the human information-pro-
Two other studies in which the verbalized cessing system, a distinction is made between
descriptions were generated prior to blocks fast automatic processes that are not neces-
of test trials also failed to find effects of ver- sarily conscious (and that are often thought
balization (Brehmer, 1974; Brehmer, Kuy- to proceed in parallel) and the slow serial pro-
lenstierna, & Liljergren, 1974, 1975). In. a cesses that are executed under cognitive con-
concept acquisition study using auditory trol—a distinction, that is, between preatten-
stimuli, Wilson and Spellacy (1972) found no tive and focally attended processes (Neisser,
difference in number of trials or errors to cri- 1967), perceptual and cognitive processes
terion between a control group and a group (Simon, 1979), and automatic and cognitively
of subjects who told the experimenter what controlled processes (Shiffrin & Schneider,
rule they used. Asking subjects what they 1977). With increase in experience with a
thought was the correct solution before each task, the same process may move from cog-
trial in a discrimination-learning experiment nitively controlled to automatic status, so that
was not found to change the proportion of what is available for verbalization to the
correct responses (Karpf & Levine, 1971). novice may be unavailable to the expert. Sev-
The proportion of correct placement of cards eral types of processes generally occur auto-
in two piles was not related to whether or not matically, in this sense, and rapidly (in a
the subjects were verbalizing the rule they matter of tens or hundreds of milliseconds):
were using (Dulaney & O'Connell, 1963; perceptual-encoding processes (recognition),
Verplanck, 1962). memory retrieval processes, and motor pro-
The fact that encoding and reporting ver- cesses.
bally take time creates a procedural difference,
which is important in some studies, between Completeness of Verbal Reports
the verbal reporting and control conditions.
Although time may not be important in many When subjects do not verbalize information
tasks, it is known to be an important variable that the investigator has strong reasons to
in long-term memory phenomena. Boersma, assume they would need to have available in
Conklin, and Carlson (1966) allowed their order to perform the task, it is reasonable to
subjects in the verbal report condition an ad- conclude that the protocols provide only an
ditional minute to specify their encoding incomplete record of the process. For example,
strategy for each stimulus. Retention scores Rees and Israel (1935) found that subjects
were superior for the verbal report condition, could acquire a set for solving anagram prob-
but the experimental design confounds the lems in a specific way without reporting the
effect of the additional time allowed for ver- similarity of their solutions and of the struc-
balization with effects of generating a written tures of the anagrams. Similarly, many con-
description of the encoding strategy. cept-formation studies have shown that sub-
jects can display consistent and accurate be-
Discussion havior without always being able to report
Even though the empirical data from sys- verbally the concepts employed by the ex-
tematic studies investigating the effects of perimenter (Heidbreder, 1934, 1936; Smoke,
verbalizing are relatively modest, the results 1932). With respect to learning, many in-
of these studies consistently support our vestigators have shown subjects to be unable
model's prediction that producing verbal re- to report reinforcement contingencies used by
ports of information directly available in the experimenter (e.g., Greenspoon, 1955) or
prepositional form does not change the course the use of mediating associations in paired-
and structure of the cognitive processes. How- associate learning (e.g., Bugelski & Scharlock,
ever, instructions that require subjects to re- 1952).
code information in order to report it may What kinds of incompleteness in reporting
affect these processes. Our model assumes that would be predicted in the framework of our
only information in jocal attention can be ver- model of verbalizing? The model proposes
236 K. ANDERS ERICSSON AND HERBERT A. SIMON

that verbal reports are based on the informa- also, it is clear that we normally have access
tion currently in STM or on information pre- only to certain higher level intermediate re-
viously in STM that has been fixated in, and sults. In a recent article, Broadbent (1977)
can be retrieved from, LTM. Within the presented empirical evidence for a hierarchi-
model, we can identify three different causes cal organization of such processes. Most
of incompleteness of reports: (a) The infor- studies of perceptual-motor processes using
mation is not heeded, hence not stored in thinking-aloud protocols and other verbal re-
STM, hence not accessible for verbal report- ports have examined problem solving with
ing, (b) Not all the information available in puzzles. In tasks allowing physical manipula-
STM at the time of the report is actually tion, Klinger (1974) found a relatively high
reported, (c) Not all of the information pre- frequency of higher level verbal evaluations
viously available in STM has been retained in of unverbalized solution attempts (e.g., "Yep,"
LTM, or is retrievable from LTM. "Dammit," etc.), and of verbalizations of at-
tention-control processes (e.g., "Let's see,"
Unavailability of Information in STM "Where was I?", etc.). When engaged in per-
ceptual-motor manipulation, subjects did not
Under a variety of circumstances, informa- verbalize, and appeared not to be aware of,
tion about ongoing cognitive processes may the lower level content or structure of their
simply not be available in STM. Several types thought processes. For example, Ruger (1910)
of cognitive processes, like perceptual-encod- found that subjects could often solve one of
ing processes, motor processes, and LTM his mechanical puzzles several times, yet they
direct retrieval processes, appear not to use provided only a limited high-level account for
short-term memory for storage at intermedi- the intermediate steps leading to the solution.
ate stages of processing, but only for the final It has been suggested that physical manipula-
product. On this point, model and empirical tion is different from thinking in not employ-
studies are in full agreement. ink any internal (i.e., STM) representation
Generally, we recognize familiar faces, (Durkin, 1937).
words, and objects directly, that is, without There appears to be a close (negative) re-
storing in STM the features extracted from lation between degree of practice and aware-
the stimuli and used for discrimination. There ness of intermediate stages of a process. The
is evidence, also, for direct recognition of early work of Watt and others suggested that
more complex patterns and relationships, espe- the conscious content disappeared with ex-
cially when the presentation is visual. Clapa- tended practice and growing automaticity of
rede (1933) found that his thinking-aloud
the processes (Woodworth, 1938). More re-
subjects did not report intermediate stages cently, Dean and Martin (1966) found that
when generating interpretations and hypothe- overlearning in paired-associate learning leads
ses for complex visual stimuli. Similar findings to a decrease in the number of reported medi-
are reported for subjects noticing relationships ating associations. The work of Schneider
in geometry problems (Henry, 1934). and Shiffrin (1977; Shiffrin& Schneider, 1977)
In recall and retrieval of familiar informa-
suggests that there are clear differences be-
tion, unless it requires problem solving with tween automatic and controlled processing,
the aid of successive associations, we fre-
in terms of speed and of accessibility for
quently find processes that leave only the modification and learning. From all this evi-
final product as trace in STM. The phenome-
dence, it seems necessary to postulate, as
non is so familiar that it appears not to have we have, that many highly overlearned pro-
been tested by experiment. There is ample cesses operate automatically without leaving
evidence, however, from introspective reports any more trace than their final result in
directed at the issue of the existence of image- STM.3
less thought for Woodworth (1938) to reach
the conclusion that "what is imageless is not
thought as much as recall" (p. 787). 3
This effect of automation may be explained
With respect to perceptual-motor processes, thus: Before overlearning has occurred, processes
 VERBAL REPORTS AS DATA 237

We may distinguish between automatic pro- several investigators have shown that with
cesses that subjects already possessed prior practice, subjects change from attending to
to an experiment, as part of their cognitive the meaning of the presented information to
skills, and processes whose intermediate stages recognition of invariant perceptual character-
became more automatic, and hence less re- istics of the display (Quinton & Fellows,
portable, during the course of the experiment. 197S) or purely formal properties of the
In the case of the latter, reports obtained presented information (Wood & Shotter,
from the automated processes at the end of 1973). Wood, Shotter, and Godden (1974)
the experiment will not give useful informa- gave direct experimental evidence of this
tion about intermediate states of which the transition in showing a marked relation be-
subjects were aware at the beginning of the tween subjects' ability to answer unexpected
experiment. We will later provide some con- questions—that would require attending to
crete examples of this phenomenon. the meaning of the presented information—
An often cited study by Rees and Israel and practice.
(1935) demonstrated that subjects could
solve a long series of anagrams having iden- Failure to Report STM Contents
tical structure and would select, in anagrams
Subjects tend to stop verbalizing or to ver-
with multiple solutions, the solution corre-
balize incompletely in conditions in which
sponding to the common structure, without
they are giving indications of being under a
reporting awareness of that structure. The
high cognitive load. Such indications may
anagrams all used the simple permutation,
take the form of reorganizations of the prob-
S4123, which permitted rapid solution. It is
lem representation or strategy (Durkin,
thus not unreasonable to suppose that these
1937), or direct expressions of feeling diffi-
anagrams were solved by an "automatic" pro-
culty (Johnson, 1964). On the other hand,
cess that did not leave intermediate results
in situations in which the subject is not
in STM. In a study by Sargent (1940), using
judged to be performing major task-directed
thinking-aloud and retrospective reports, sub-
processes, verbalization tends to be relatively
jects were generally unable to report inter-
complete. Ericsson (197Sa) calculated the
mediate states when they achieved fast solu-
association between regularly observable fea-
tions to anagrams. More support for this
interpretation is given by the findings of Rees tures of the process (e.g., reversal of a pair
of moves) and specific kinds of verbalization
and Israel that subjects recognized the simi-
and found a very high correlation (e.g., on
larity of anagrams when a more complex
each occurrence of a reversal, the subject
permutation of the letters was used. In fact,
verbalized a negative evaluation of the origi-
the only anagram pattern that was not re-
nal moves).
ported by subjects was the one depicted Apart from cases in which the subjects rely
earlier. In experiments in which subjects are on automatic processes, there are other forms
asked to do similar tasks over and over, the of incompleteness in reports in which infor-
subjects' latencies become markedly shorter mation that was once in focal attention (in
with practice. Relying on the subjects' retro- STM) is not verbalized. Our model asserts
spective reports along with other evidence, that the verbal report will be based on in-
formation that is available to the subject at
the time of the report. From this assumption,
have to be interpreted, with substantial feedback we expect information in STM to be reported
from intermediate processing stages into STM. Over-
learning amounts to compiling these processes, so in full, whereas information in LTM will be
that fewer tests are performed when they are being differentially accessible for various reasons.
executed, hence less information is stored at inter- From extensive research on information stored
mediate stages in STM. Experience with compiling in STM, it is clear that such information is
in computer languages shows that automation typi-
cally speeds up a process by an order of magnitude,
easily obliterated. In a matter of a few sec-
at the expense of making it less flexible, and its in- onds, the contents of STM can be destroyed
termediate stages less available for report. or made inaccessible by requiring subjects
238 K. ANDERS ERICSSON AND HERBERT A. SIMON

to perform certain types of processes, for ex- thinking-aloud protocols. Of this kind of in-
ample, repeatedly subtracting 7 from a given sight, Durkin (1937) says:
number (Brown-Peterson paradigm). With
When it occurs, it comes with an onrush that makes
any shift in the focus of attention, of which it seem very sudden—an "out of the blue" experi-
this paradigm provides an example, the pre- ence. But it can always be found to have developed
vious contents of STM become unavailable. gradually. The suddenness must be regarded as due
If an intermediate result in a sequence of to the concealment of the background. It does not
bring in a new kind of process, (p. 81)
processes causes a direct execution of other
processes that make full demands on STM, For geometry proofs (Henry, 1934), and
the intermediate result may reside for only for a variety of "insight" problems (Bui-
a brief moment in STM, and may be lost brook, 1932), the thinking-aloud protocols
before being reported. Under thinking-aloud showed that the progress to solution was
conditions, it has been observed (Duncker, either gradual or was determined by trial
1945) that information that leads to the di- and error. In neither case was it necessary to
rect recognition of the appropriate action of- postulate additional kinds of processes.
ten tends not be be verbalized. Similar ob- Insight or the illumination of a creative
servations have been made (de Groot, 1965) idea was described by many early investiga-
about the reports of chess grandmasters con- tors as the result of a period of unconscious
sidering possible moves in chess positions. work, or incubation, following preliminary
A frequently cited study by Maier (1931) work in becoming familiar with the problem
on subjects' retrospective reports about a hint (preparation). During this period of alleged
given during solution of the pendulum prob- incubation, according to these accounts, the
lem gives some evidence for the same phe- scientist or inventor has laid the problem
nomenon. Subjects who described the solution aside in favor of other activities. However, it
as emerging in a single step did not report has been suggested (Woodworth, 1938) that
any memory of the hint. A result of Maier's during the period of incubation, the scientists
study that is less often recognized was that will occasionally lapse into thinking about
all subjects who mentioned more than one the problem, even though their main activities
step in the solution of the problem reported may be different.4 Woodworth (1938) cited a
that the hint had been administered. study by Platt and Baker that suggests that
The two related mechanisms mentioned subjects are not aware of the durations of these
thus far—absence of intermediate stages of unplanned episodes of concentrated thought
acts of recognition from STM and failure to on the problem. Clearly, such unanticipated
report transient contents of STM—are fully thought processes, if they occur, will be very
adequate to account for the phenomena of difficult to retrieve in retrospect. Generally
sudden "insight" that are the subject of so such episodes are terminated, often abruptly,
many anecdotes in the literature of creativity by external demands, as, for example, in a
(Nisbett & Wilson, 1977, pp. 240-241). The driving situation.
studies cited in support of sudden insight From research on daydreaming (Singer,
are based on retrospective accounts of pur- 1975) and undirected thought (Klinger,
portedly real creative acts, often reported 1971), there are suggestions that such thought
many years after the event. Fortunately, a episodes are difficult to recall fully unless the
number of studies have addressed this topic retrospective reports are obtained shortly
in a more controlled experimental environ- thereafter, or unless the subjects label or re-
ment. Durkin (1937) sought to create favor- hearse the thought content for subsequent
able circumstances for "insight" with subjects recall. Studies using concurrent verbalization
thinking aloud while solving block puzzles. (Bertini, Lewis, & Witkin, 1964; Kazdin,
Even though subjects occasionally reported
insights, the background steps leading to the 4
A different explanation of incubation, based on
emergence of the insightful ideas could al- forgetting of STM contents, has been proposed by
ways be determined from the concurrent Simon (1977, pp. 296-299).
 VERBAL REPORTS AS DATA 239

1976; Klinger, 1971) provide detailed and 1936; Phelan, 196S; Smoke, 1932). Although
informative accounts of undirected thought these studies have been cited in support of
processes. In sum, thinking that is not closely the notion that the verbalized information is
related to the external environment can some- incomplete, a more reasonable interpretation
times be retrieved with situational clues but is that the subjects cannot formulate the con-
seldom otherwise, except when it is verbalized cept as it is denned by the experimenter, al-
concurrently with the thought process. though they can differentiate instances from
noninstances in a test run. However, correct
Incompleteness in Retrieval from LTM selection of instances can also be mediated
by processes like memorizing exemplars or
We consider next the situation in which a
a set of correlated discrimination features
subject is probed for information that is not
without using a rule expressed in terms of
available in STM at the time of probing. Then
common features.
the information must be retrieved from LTM.
Memory retrieval is fallible and sometimes When Smoke (1932) asked subjects to
leads to accessing other related, though in- identify the concept used by the experimenter
appropriate, information. Further, the in- from verbal descriptions, he found in four
formation that can be recalled depends on experiments that 2Q%-25% of the verbal de-
what cues and probes are provided. Hence, scriptions of the successful subjects were de-
the completeness of the information retrieved fective, and "usually too inclusive" (p. 20).
will vary with the probing procedure. This study, and other related ones, do not
Verbalizing rules in concept attainment. address the incompleteness problem as we
In reviewing evidence for incompleteness in would like to state it: Is the verbal report a
reports, we will be concerned primarily with complete or sufficient description of the in-
studies of concept attainment and learning. formation the subject actually has and uses?
Such processes can hardly be carried out In Smoke's (1932) study the subjects who
automatically, since they require reprogram- were classified as unable to verbalize the con-
ming of responses to stimuli; hence, the cept could "almost invariably" (p. 20) draw
stimuli should be attended to and thus avail- two instances of the concept from memory
able in STM, at least before extended prac- correctly, but this fact does not imply that
tice. We can thus be reasonably sure that they had a complete and correct (though un-
the information corresponding to these changes verbalizable) criterion for making the selec-
in behavioral regularities has at least at one tions.
time resided in STM. To test our hypotheses The question of completeness has been put
about the less complete report of information to a more direct test in an interesting study
from LTM, as compared with STM, we will by Wilson (1974, 1975). In his study of con-
review evidence allowing us to infer what in- cept learning, the subjects wrote down their
formation would reside in STM at the time rules for positive instances during each trial
of a verbal report. In particular, we will be and were then asked to sort a test series of
concerned with empirical studies showing that instances. Wilson then assessed the informa-
subjects can learn to behave in agreement tion transmitted in these verbal descriptions
with concepts or rules without being able to by having the same subjects a week later
verbalize the concepts or the stimulus at- make re-sorts from the descriptions. In ad-
tributes they are responding to. We will also dition, naive subjects who did not participate
discuss some other studies that have examined in the concept-learning experiment were asked
whether experimentally controlled factors fa- to sort the test series on the basis of the in-
cilitating learning are mediated by reportable dividual verbal descriptions provided by the
intermediate states. original subjects. The results showed that the
In concept learning, it has been observed sorts made by subjects after a week's delay
that subjects can select appropriate instances agreed less closely with their original sorts
in test trials without being able to state the during the concept-learning experiment than
concepts they are using (Heidbreder, 1934, they did with the sorts made by the naive
240 K. ANDERS ERICSSON AND HERBERT A. SIMON

subjects. This is evidence that the subjects peatedly for poor documentation of probing
had more information at the time of the procedures and brief postexperimental inter-
original experiment than they gave in their views, which are sometimes not given after
verbal reports. It should be pointed out, how- the learning trials but after an additional
ever, that the correspondence between all the series of extinction trials (Spielberger, 1962).
sorts was high, especially considering that the This type of experiment does not contradict
verbal rule was not always applicable to all the possibility that subjects retain in STM
instances in the test series. the information about the reinforcement con-
In a second experiment, Wilson (1974, tingency until it is lost during overlearning
197S) found that the degree of incomplete- or extinction trials. To ensure as complete a
ness of descriptions varied with the stage of verbal report as possible, the probing should
concept learning at which the verbalization occur just after the last learning trials, prefer-
of the rule was obtained. For rules verbalized ably before the subject is told that the experi-
either at the beginning of the experiment or ment is over, to ensure that the critical in-
at criterion, almost complete agreement was formation remains in STM and does not re-
obtained between sorts, but more discrep- quire retrieval from LTM. Many of the
ancies were found for rules verbalized at in- studies of this first type seem little concerned
termediate stages of acquisition. Wilson at- with these considerations. Furthermore, the
tributed these results to subjects' difficulties verbal probes used have been global ques-
in verbalizing the complex hypotheses they tions, like "What did you think the experi-
entertained during intermediate learning ment was about?" (Brewer, 1974). In our
stages, and this interpretation was further framework, such verbal probes are not aimed
supported by the greater length of the verbal at eliciting retrospective memory of the sub-
descriptions for these stages. jects' own cognitive processing, but rather at
One alternative explanation, suggested pre- encouraging them to generate hypotheses
viously, would be that the instructions used about the experiment, which may or may not
by Wilson (to verbalize a rule—the average be related to those processes.
number of words per rule was about eight) A second type of study, which has re-
were inadequate to tap the subjects' informa- sponded to the earlier mentioned criticisms by
tion about positive instances. Another possible probing subjects just after the last learning
interpretation is that instances in the test block of trials and explicitly asking them
series may have served as cues for retrieval for their memories of the cognitive processes
and recognition of previously presented items during the preceding trials, has generally failed
that were not available to the subjects when to find evidence for learning without reported
they generated the verbal report. Some sup- awareness of the reinforcement contingency.
porting evidence is given by a study on dis- The studies of this type have been criticized,
crimination learning by Frankel, Levine, and in turn, for asking questions that are too
Karpf (1970), in which the subjects could specific, and hence suggesting awareness to
give a retrospective description of "on what the subjects. We have already discussed this
basis they had responded" (p. 346) that de- issue in the section on effects of probing. The
scribed more than 90% of their responses. need for specific probes is not well docu-
Learning without awareness. The contro- mented, but a general motivation for it is
versy over whether learning can and does oc- given in Dulany (1962, p. 114). Another im-
cur without awareness, as evidenced by ver- portant consideration is that the subjects
bal reports from subjects, has recently been often report contingencies that although not
reviewed by Brewer (1974). Drawing on his identical to the one the experimenter rein-
work, our primary aim will be to interpret forces, are correlated with it (Dulany, 1962).
the differences, primarily methodological, that Here, failure to report the experimenter's
version of the contingency may simply mean
distinguish studies of learning without aware- that this is not the version the subject is
ness from studies without such learning. using, and may not at all imply incomplete-
One type of study has been criticized re- ness in the report of the contents of STM.
 VERBAL REPORTS AS DATA 241

In a third type of study, subjects are asked likely to change hypotheses in response to
to verbalize during conditioning experiments negative feedback, the assumption of stable
according to a technique suggested by De- hypotheses is less tenable for probabilistically
Nike (1964), in which the subject writes determined feedback conditions.
down "any thoughts that come to you that In a recent study by Williams (1977), all
have any relation to the experiment" (p. S23). subjects who successfully learned the dis-
Using this procedure, DeNike was able to ex- crimination were assessed (by a procedure of
tend the claim of consistency between written the similar to Spielberger, 1962) to have been
thoughts and behavior in agreement with the aware of the reinforcement. However, in a
reinforcement contingency, finding that sub- second experiment with probabilistic rela-
jects' behaviors changed on the trial in which tions, Williams found evidence of learning
they wrote down their first correct hypoth- without awareness (assessed by the same
esis about the reinforcement. procedure). During the experiment, subjects
Also using DeNike's technique, Kennedy were explicitly told that "All the sentences
(1970, 1971) found that the behaviors I said 'correct' to met the same necessary re-
changed before subjects wrote down that they quirements" (p. 93). These instructions might
were confident of their hypotheses or before have encouraged subjects to abandon cor-
their verbal hypotheses were confirmed. rect and correlated hypotheses when they
Brewer (1974) pointed out that Kennedy's encountered probabilistically determined neg-
finding, that behavioral changes were associ- ative feedback. Hence, the subjects may very
ated with trying out or modifying verbal hy- well have entertained correct or correlated
potheses, does not challenge, but supports the hypotheses during the trials evidencing learn-
validity of the verbalized information. ing, but they may have discarded them when
There is evidence from comparing different negative feedback was encountered. A study
probing techniques with each other that the by O'Connell (196S) shows clearly that if
written thoughts elicited by DeNike's tech- the subjects verbalize their hypotheses on
nique are incomplete as compared with in- each trial, the verbalized reports account for
formation obtained using Dulany's postex- their behavior even in a nondeterministic en-
perimental questionnaire (Sallows, Dawes, vironment with partial reinforcement.
& Lichtenstein, 1971). There is also some evi- There is a related issue of awareness of
dence that the incompleteness may be caused mediating associations. A series of studies
by the requirement of written responses. Sil- (Bugelski & Scharlock, 195 2; Horton &
veira (1972) found a marked difference in Kjeldergaard, 1961; Russel & Storms, 1955)
number and character of responses elicited have found that paired-associate learning can
during a creativity test between written and be facilitated by prior exposure of the sub-
oral response conditions. The process of writ- jects to the proper mediating associations.
ing the responses or ideas, as contrasted with Subjects who first learned lists of paired as-
giving them aloud, was found to be linked to sociates of Types A-B and B-C learned lists
evaluation and censorship. of Type A-C faster than control groups. In
It has been assumed implicitly that the all of these studies, informal postexperimental
contents of STM after the last learning block questioning gave no evidence that the sub-
are representative of the contents during the jects were aware of using any mediating B-
preceding trials. This is a reasonable assump- list items. "None of the Ss [subjects] was
tion for concept learning and for experiments able to report any correct appreciation of
with deterministic reinforcement schedules, the nature of the experiment and most as-
in which there is no inducement for the sub- suredly did not verbalize a pattern of A-B,
ject to reject a correct hypothesis. Several B-C, A-C in learning the third list" (Bugel-
studies have shown that subjects are unlikely ski & Scharlock, 1952, p. 366). This was in-
to change hypotheses in response to positive terpreted as evidence for unconscious media-
feedback (Heidbreder, 1924; Karpf & Levine, tion in learning.
1971). However, since subjects are highly However, the previously cited studies can
242 K. ANDERS ERICSSON AND HERBERT A. SIMON

be criticized on the same grounds as the might have affected the retrievability of the
studies purporting to demonstrate concept mediating links. However, the most obvious
learning without awareness; typically, we difference, in our view, lies in the probing
find different results in more recent studies procedure.
that probe for retrospective information in a Other studies alleging incompleteness. We
more controlled and ambitious manner. In a will mention briefly several miscellaneous
study by Dean and Martin (1966), the sub- studies claiming incompleteness of retrospec-
jects, after reaching criterion for the third tive reports. Rees and Israel (193S) found
list, read each stimulus of the last list and that anagrams were solved faster if the solu-
were then asked to tell the experimenter ex- tion words were sampled from specific types
actly what came to their minds when they of words, like "nature words" (names of
saw the syllable on the screen. The subjects plants and trees). Subjects reported in the
were then shown the entire list of paired as- postexperimental questioning that they noticed
sociates and were asked how the list was the relation between the solution words, but
learned. Dean and Martin found that a ma- most of them also reported that they did not
jority of the subjects reported using at least actively use that information.
one mediating term from the previously In a study of the effect of reversal in dis-
learned facilitating lists. An analysis of learn- crimination learning, Walk (19S2) found that
ing rates for each of the reported mediation the reversal had a short-term effect on the
types (other than A-B-C mediation) clearly behavior of the reversed group that was not
suggested that the effective difference among matched by verbal evidence of awareness of
groups was attributable to the occurrence or the reversal. With the pictorial stimuli, Walk
nonoccurrence of A-B-C mediation. When also found that subjects were occasionally
Dean and Martin had one experimental group unable to define the reasons for their correct
overlearn the paired-associate list for 10 extra selections.
trials, they found a significantly lower oc- In a study of concept formation, Heid-
currence of reported mediation, thus suggest- breder (1924) found that subjects referred
ing that direct and automatic processing had to aspects of figural stimuli presented earlier
developed as a result of additional practice. that they had not previously considered in the
In a study following a procedure similar to retrospective reports, as illustrated by a quote
that in Horton and Kjeldergaard (1961), from a verbal report, "I've been wondering
Horton (1964) used a direct question: "Did if the ones I've had right haven't always had
you notice any relationship between the pairs more lines in the figure I marked. I think
you just completed and the ones you learned they have but I'm not sure" (p. 136).
earlier in the experiment?" Horton assessed
three levels of awareness, where the highest Discussion
level required naming the actual mediating Under a variety of circumstances, verbal
items. He found a consistent relation between reports may omit information that subjects
mediation (effect of previously learned fa- use to perform the task. Evidence of the
cilitating paired-associate lists) and assessed nature of the omissions is consistent with
awareness for a variety of experimental ma- the predictions of our model. The intermedi-
nipulations. ate stages of immediate recognition processes
There are many differences between the and the detailed steps of perceptual-motor
studies reporting awareness of mediating items processes are not generally recorded in STM;
and those that report no awareness. Horton hence, they are not reported. Processes that
(1964) suggested that the difference between have been so often repeated as to have be-
his study and Horton and Kjeldergaard come automated are less often and less fully
(1961) stemmed from the factors in the ex- reported.
perimental situation stimulating awareness. When subjects give indications that they
The two groups of studies also differ in the are working under a heavy cognitive load,
strictness of the learning criterion, which they tend to stop verbalizing or they provide
 VERBAL REPORTS AS DATA 243

less complete verbalizations. The contents of and inconsistent verbal reports have been ob-
STM can be obliterated, hence, omissions served. In most of the cases in which incon-
caused in reports, by requiring subjects to sistency has been observed or claimed, the
perform intervening tasks concurrently with verbal reports were retrospective.
their verbal reporting.
The evidence for inability to report the Potential Sources oj Inconsistency
bases for sudden insights is mainly anecdotal
Within our model, inconsistent verbal re-
and is refuted by the few laboratory studies
that have been made of insight phenomena. ports could be produced by two processes.
First, cues used to access LTM, if too gen-
Nor is the evidence, all of it anecdotal and
eral, could retrieve information related to,
retrospective, of the unreportability of thought
but not identical with, the information that
processes during incubation convincing. Al-
was actually sought. There is evidence that
ternative explanations of these phenomena
subjects have information (in the form of
are available that are consistent with our
differential confidence) about the correctness
model.
of LTM retrievals (see Montague, 1972);
When clear probes are used for specific
this information could be tapped to decrease
retrospective memory and when reports are
the number of retrieval errors stemming from
requested immediately after the last trial(s),
this cause.
informative verbal reports can usually be
The second source of inconsistent informa-
obtained, although perhaps not always in
tion, already discussed in some detail, is the
the case of complex pictorial stimuli. The
use by subjects of intermediate processes to
failure of subjects to report some information
infer missing information and to fill out and
does not demonstrate the uselessness of ver-
generalize incomplete memories before re-
bal protocols. Incompleteness of reports may
sponding. An example is provided in the
make some information unavailable, but it
study of Rommetveit and Kvale (196Sb),
does not invalidate the information that is
discussed earlier. Patterns were displayed to
present. In an often cited remark, Duncker
signal to subjects playing a wheel of fortune
(1945) observed that "a protocol is relatively
whether they would win or not. When the
reliable only for what it positively contains,
experimenters asked a boy to describe the
but not for that which it omits" (p. 11).
differences between positive and negative pat-
terns, he said he did not know, although he
Consistency of Verbal Reports With
had been able to anticipate the rewards cor-
Other Behavior
rectly on previous trials. When pressed by
So far we have been concerned with the the experimenters, he finally attempted a
failure of subjects' reports to contain in- verbal description and gave one that was in-
formation that one would expect was heeded consistent with the actual signals. In this and
and hence was in STM at some time. In this many other similar accounts, it seems ap-
section we will be concerned with verbal re- propriate to attribute the error to absence of
ports that are inconsistent with other sources the information from memory, rather than
of data—primarily with observable nonverbal inconsistency between memory contents and
behavior. verbal reports of them. When information is
Claims that verbalized information is in- not in memory, it cannot be reported verbally.
consistent with other behavior are often made In a review article, Smedslund (1969)
in general and sweeping terms without pro- claims that verbal reports did not provide
viding specific evidence. When evidence is useful information on rapid mental processes
provided, it is often anecdotal, resting on the in an arithmetic task he had studied (Smeds-
premise that if one can produce a single case lund, 1968). He could describe two reports
of a clearly inaccurate or inconsistent verbal that could be proven inconsistent—one in-
report, then verbal reports are wholly inad- consistent with the subject's performance and
missible as data. In this section we will con- speed in solving the test items, the other in-
sider under what circumstances inaccurate consistent with the types of items actually
244 K. ANDERS ERICSSON AND HERBERT A. SIMON

given. In the original article, however, Smeds- artifacts of the original experiment. First,
lund does not mention inaccuracy in the ver- consider the case in which correct placement
bal reports and actually quotes those reports was reinforced. Making a correction for guess-
to support some of his general results. In ing (the subjects had a SO-SO chance of
Smedslund's study, the conditions for verbal placing the card in the correct pile when
reporting were far from optimal, for the sub- they did not know the rule), we can estimate
jects were asked after a relatively large num- that subjects knew the correct answer in
ber of trials to tell how they solved the tasks. 43.6% of the trials—a percentage very close
Even under these unpromising circumstances, to the 48.4% in which they stated the cor-
the verbalized information, with a few excep- rect rule.
tions, seemed to satisfy the author as basically Second, with respect to the reinforcement
consistent with his observations of subjects' of rules, Dulany and O'Connell (1963) found
performance. that the correct rules used by Verplanck and
Provided that our procedure for analyzing Oskamp were ambiguous for the card illustra-
data can handle occasional errors in the ver- tions they employed. In fact, naive subjects,
bal information, and provided that these er- who were told the rules explicitly, generated
rors can be minimized by appropriate pro- the same proportion of misplacements as was
cedures (including recognition of the cir- recorded in the original experiment.
cumstances under which subjects cannot be In a detailed analysis of the rules the sub-
expected to remember certain information), jects verbalized on each trial, Dulany and
anecdotal evidence of the sort just cited need O'Connell (1963) found that on all but 11
not shake our confidence in the validity and of 34,408 trials, the subjects put the card
legitimacy of verbal reports. where they said they were going to. Hence,
Dulany and O'Connell impeached rather
Inconsistency With Concurrent Verbalization thoroughly the evidence put forth by Ver-
In an often cited study (Verplanck, 1962), planck and Oskamp for believing that the
Verplanck and Oskamp claimed to have shown verbalized rules were inconsistent with the
that verbalized rules are dissociated from behaviors.
Numerous studies provide documented sup-
the behavior they were supposed to control.
port for consistency between verbalized rules,
This study is the only one we have found
concepts, and hypotheses, and immediately
claiming inconsistency of concurrent verbal
preceding and succeeding behavior, before sub-
reports with behavior. By having the subjects
verbalize the rules they were following in jects receive feedback. In Schwartz (1966),
sorting illustrated cards, the experimenters where subjects were asked their reasons for
could reinforce either the verbal rule or the placing a card as they did, reasons consistent
placement of cards (i.e., behavior). To make with placements were given on all but 2 of
1,962 trials. Even more impressive, Frankel
the contingencies less noticeable, the partial
reinforcement followed the criterion trials. et al. (1970) obtained retrospective reports
When correct placements were reinforced, from subjects on the basis of their responses
to four earlier discrimination-learning prob-
the subjects placed cards correctly in 11.8%
lems with 30 nonfeedback trials each and
of the trials, but they stated a correct or cor-
found that subjects could provide reports in
related rule on only 48A% of the trials. When
more than 90% of the sequences of trials.
the correct statement of the rule was rein-
forced, the subjects stated a correct or cor-
The, Nisbett-Wilson Literature Review
relation rule on 92.8% of the trials but placed
the cards correctly on only 76.8% of the In a recent, extensive review of studies
trials. permitting comparison of retrospective verbal
In a replication and analysis of the Ver- reports with behavior, Nisbett and Wilson
planck-Oskamp experiment, Dulany and (1977) have stated conclusions that appear
O'Connell (1963) showed that the previously at first sight to be almost diametrically op-
mentioned results could be attributed to two posite to those reached in this article. Since
 VERBAL REPORTS AS DATA 245
the article by Nisbett and Wilson has re- erations (p. 255)," that is, they hold in
ceived widespread attention, it is important STM and can access the symbols that are
to consider how their findings are to be recon- inputs and outputs to such operations.
ciled with ours. We may compare this list of "private facts"
Nisbett and Wilson (1977) summarized and intermediate results that according to
their main empirical findings as follows: Nisbett and Wilson are accessible to subjects
with the kinds of verbalizations we have been
People often cannot report accurately on the effects considering in our review of the evidence. The
of particular stimuli on higher order, inference-based
individuals know, they say, their focus of at-
responses. Indeed, sometimes they cannot report on
the existence of critical stimuli, sometimes cannot re- tention, their current sensations, their emo-
port on the existence of their responses, and some- tions, their evaluations, and their plans. They
times cannot even report that an inferential process know the intermediate results of their mental
of any kind has occurred, (p. 233) operations. But these are exactly the kinds of
information that according to our model and
First, we call attention to the frequent use, the evidence we have examined, are held in
in their summary, of the qualifiers "often" STM and are available for verbal reports.
and "sometimes." Nisbett and Wilson cited a Only one kind of item that we have consid-
large number of experiments that support ered is missing from this definition of "con-
their conclusions, but they did not investigate tent": awareness of ongoing processes. If
in detail the conditions under which these there is a discrepancy between our account
conclusions do and do not hold. Moreover, and that of Nisbett and Wilson, it lies in
they did not propose a definite model of the that domain.
cognitive processes as a framework for in- Unfortunately, the studies reviewed by
terpreting the findings they surveyed. Their Nisbett and Wilson provide little data as to
theoretical interpretations of these findings what information is heeded during the
are entirely informal, resting heavily on an thought processes and what information is
undefined distinction between introspective accessible from STM and LTM at the time
access to "content" and to "process," or, as of the verbal report. Nisbett and Wilson
they alternatively state it, between access to (1977) simply assert that the subjects, when
"private facts" and to "mental processes." asked questions about their cognitive pro-
Nisbett and Wilson's (1977) summary of the cesses, frequently do not base their answers
kinds of information to which subjects do on memory for specific events at all, but
have access is this: "theorize" about their processes.
We do indeed have direct access to a great storehouse When reporting on the effects of stimuli, people may
of private knowledge. . . . The individual knows a not interrogate a memory of the cognitive processes
host of personal historical facts; he knows the focus that operated on the stimuli; instead, they may
of his attention at any given point of time; he base their reports on implicit, a priori theories about
knows what his current sensations are and has what the causal connection between stimulus and response,
almost all psychologists and philosophers would as- (p. 233)
sert to be "knowledge" at least quantitatively superior
to that of observers concerning his emotions, evalua- In reviewing the studies cited by Nisbett
tions, and plans. Given that the individual does and Wilson, we can profitably raise the ques-
possess a great deal of accurate knowledge . . . it
becomes less surprising that people would persist in tion of why and when subjects do not consult
believing that they have, in addition, direct access their memories of cognitive processes in an-
to their own cognitive processes. The only mystery swering questions about those processes. It is
is why people are so poor at telling the difference easy to draw the erroneous conclusion that
between private facts that can be known with near
certainty and mental processes to which there may this independence of the verbal answers to
be no access at all. (p. 255) the questions about cognitive processes from
the actual course and results of those pro-
Nisbett and Wilson (1977) also observed cesses implies a general lack of accessible
that subjects "are often capable of describing memory for such processes, or even an un-
intermediate results of a series of mental op- awareness of the information while the process
246 K. ANDERS ERICSSON AND HERBERT A. SIMON

was actually going on. But we have seen Moreover, in most of the studies reviewed
that such a sweeping conclusion is contra- by Nisbett and Wilson, the time lag between
dicted by the evidence from concurrent ver- task and probe was sufficiently great as to
balization. make it unlikely that the relevant informa-
Drawing on our taxonomies of types of tion remained in STM. When the probe is
verbalization and of techniques for probing not a good retrieval cue for the relevant
for information, we will show that in the aspects of the memory of the process (see our
studies reviewed by Nisbett and Wilson, very discussion in the next paragraph), the sub-
different procedures were used from those ject must attempt, through conscious pro-
that according to our model, would elicit valid cessing, to secure a sufficiently complete recall
retrospective reports of processes. Before we for giving the appropriate answer. Such re-
discuss these differences, we want to point trieval from LTM requires considerable time
out that the studies reviewed in Nisbett and and effort, and we would claim that subjects,
Wilson were neither designed for nor primar- unless explicitly instructed to provide a rela-
ily concerned with determining subjects' tively complete recall, would be highly un-
memories of their cognitive processes. It would likely to do so, especially if other processing
be preferable to test the implications of our alternatives were available to them.
model against a new set of studies directly The second difference concerns the types of
designed to assess the information heeded information that the studies reviewed by
by subjects and the information reported Nisbett and Wilson probed for, and the rela-
retrospectively for the types of cognitive tion of those types of information to informa-
processes investigated in this research. Until tion stored about specific instances of process-
such studies have been carried out, we can ing. Our model predicts that information can
only speculate on what information about the be recovered by probes only under conditions
cognitive processes is heeded and stored in in which that same information would be
accessible form. accessed by undirected concurrent or retro-
On the basis of the distinctions made in spective reports. Information about what such
our taxonomy of verbalization and probing undirected reports would provide is lacking
procedures, we find three important differ- for most types of cognitive processes pertinent
ences between the retrospective verbalization to the Nisbett-Wilson review. However, we
procedures that our model would recom- find for many of the studies in that review
mend and the procedures used in studies re- that our taxonomy and model would predict
viewed by Nisbett and Wilson (1977). In failure to obtain from the probes verbal in-
some of these studies, the questions presented formation about particular instances of pro-
to subjects contain considerable background cesses. For example, in between-subjects de-
information that would make it feasible for signs, subjects obviously cannot answer from
subjects to generate answers without con- memory of their processes why they behaved
sulting their memories of the cognitive pro- differently from subjects in another experi-
cesses. With questions like, "I noticed that mental condition because the processes did
you took more shock than average. Why do not include such a comparison. Hence, this
you suppose you did? (Nisbett & Wilson, information can be derived, if at all, only by
1977, p. 237)," it is not even clear to us, nor comparing the descriptions of the processes
probably to the subjects, that memory for by different sets of subjects in the two condi-
the cognitive process should be the informa- tions. In other studies the subjects were asked
tion source for the answer. If subjects can how they would have reacted if the experi-
generate their answers without consulting their mental conditions had been different in a
memories of the cognitive process (Nisbett specified respect. Such probing for hypotheti-
& Wilson, 1977, showed that control subjects cal states can never tap subjects' memories
could do exactly that), this might often be for their cognitive processes, since the in-
more efficient than retrieving information formation was never in memory. In still
from memory. other studies, subjects were asked, explicitly
 VERBAL REPORTS AS DATA 247

or implicitly, to summarize or generalize the generalize incomplete or missing memories.

processes they used, rather than reporting the Studies in the literature provide examples of
processes used on each trial. both.
The third difference between the studies On the other hand, the single study, by
cited and our model concerns the time inter- Verplanck and Oskamp (cited in Verplanck,
val between the execution of cognitive pro- 1962), purporting to find inconsistency be-
cesses and the probe asking them to be re- tween concurrent verbalizations and behav-
ported. We have already discussed this point ior, does not stand up under careful analysis.
earlier and in the section on incompleteness. When the data were reanalyzed, they failed
Our review of "learning without awareness" to support the claims of inconsistency. Mean-
and "memory for mediating associations" while, a number of subsequent studies have
showed some of the procedural conditions in shown a high level of congruence between
such experiments that consistently determined verbal reports and other behavioral measures
how available or unavailable memories would in a variety of experimental settings.
be and how complete or incomplete the A well-known review article by Nisbett
reports were. and Wilson (1977) is sometimes thought to
A few of the studies reviewed by Nisbett provide evidence discrediting verbal reports
and Wilson (1977), including Maier's (1931) as data. A close examination of the specific
study of problem solving and the studies of studies analyzed in their article shows that
learning without awareness, have already been the instances cited of inconsistency between
discussed earlier in some detail, and have verbal reports and data all refer to experi-
been found to be consistent with our model. mental situations and procedures where our
In sum, after exploring the implications of model would predict that veridical reports
the studies reviewed by Nisbett and Wilson could hardly be expected. In fact, Nisbett
(1977) for our model, we find no difficulty and Wilson's own detailed summary of the
in reconciling the experimental findings with conditions under which verbal reports can be
our model. In most cases, the studies re- assumed to be valid are consistent with the
viewed by Nisbett and Wilson are fully con- conclusions we have reached in this article.
sistent with the model; in other cases they For more than half a century, and as the
involve verbalization or probing procedures result of an unjustified extrapolation of a
for which our model does not make strong justified challenge to a particular mode of
predictions. Further discussion of the Nisbett- verbal reporting (introspection), the verbal
Wilson conclusions, agreeing generally with reports of human subjects have been thought
our critique but making some additional suspect as a source of evidence about cogni-
points as well, can be found in a recent article tive processes. In this article we have under-
by Smith and Miller (1978). taken to show that verbal reports, elicited
with care and interpreted with full under-
General Discussion standing of the circumstances under which
they were obtained, are a valuable and thor-
When verbal reports are collected con- oughly reliable source of information about
currently with other records of behavior, it cognitive processes. It is time to abandon the
becomes possible to check the consistency of careless charge of "introspection" as a means
the reports with the other behavior. Evidences for disparaging such data. They describe hu-
of inconsistency can be found only under ex-
man behavior that is as readily interpreted
perimental conditions in which such incon-
sistency would be predicted by the model. as any other human behavior. To omit them
With our model, inconsistent retrospective when we are carrying the "chain and transit
reports can be produced as a result of probes of objective measurement" is only to mark
that are too general to elicit the information as terra incognita large areas on the map of
actually sought, and as a result of subjects' human cognition that we know perfectly well
use of inferential processes to fill out and how to survey.
248 K. ANDERS ERICSSON AND HERBERT A. SIMON

Reference Notes Brewer, W. F. There is no convincing evidence for

operant or classical conditioning in adult humans.
1. Anderson, J. R. Personal communication, 1978. In W. B. Weimer & D. S. Palermo (Eds.), Cogni-
2. Ericsson, K. A., & Simon, H. A. Retrospective tion and symbolic processes. Hillsdale, N.J.: Erl-
verbal reports as data (CIP Working Paper No. baum, 1974.
388). Unpublished manuscript, Carnegie-Mellon Broadbent, D. E. Levels, hierarchies and the locus
University, 1978. of control. Quarterly Journal of Experimental Psy-
chology, 1977, 29, 181-201.
3. Ericsson, K. A., & Simon, H. A. Thinking-aloud Brunk, L., Collister, G., Swift, C., & Stayton, S. A
protocols as data: Effects of verbalisation (CIP correlation study of two reasoning problems. Jour-
Working Paper No. 397). Unpublished manuscript, nal of Experimental Psychology, 1958, 55, 236-241.
Carnegie-Mellon University, 1979. Bugelski, B. R., & Scharlock, D. P. An experimental
4. Johnson, E. J., & Russo, J. E. What is remem- demonstration of unconscious mediated association.
bered after a purchase decision? (CIP Working Journal of Experimental Psychology, 1952, 44, 334-
Paper No. 379). Unpublished manuscript, Car- 338.
negie-Mellon University, 1978. Bulbrook, M. E. An experimental inquiry into the
5. Baranowski, T. Clinical judgment as a cognitive existence and nature of "insight." American Jour-
process: A comparison of linear regression and nal of Psychology, 1932, 44, 409-453.
decision net models. Unpublished manuscript, Uni- Carroll, J. S., & Payne, J. W. Judgments about crime
versity of Kansas, 1975. and the criminal: A model and a method for in-
vestigating parole decisions. In B. D. Sales (Ed.),
Perspectives in law and psychology: The criminal
References justice system (Vol. 1). New York: Plenum Press,
1977.
Anderson, J. R. Language, memory, and thought. Claparede, E. La genese de 1'hypothese. Archives de
Hillsdale, N.J.: Erlbaum, 1976. Psychologie, 1933, 24, 1-155.
Anderson, J. R., & Bower, G. H. Human associative Dansereau, D. F. An information processing model
memory. Washington, D.C.: V. H. Winston, 1973. of mental multiplication (Doctoral dissertation,
Atkinson, R. C., & Shiffrin, R. Human memory: A Carnegie-Mellon University, 1969). Dissertation
proposed system and its control processes. In K. Abstracts, 1969, 30, 1916B. (University Microfilms
Spence & J. Spence (Eds.), The psychology of No. 69-15, 746)
learning and motivation (Vol. 2). New York: Aca- Dansereau, D., & Gregg, L. W. An information pro-
demic Press, 1968. cessing analysis of mental multiplication. Psycho-
Benjafield, J. Evidence for a two-process theory of nomic Science, 1966, 6, 71-72.
problem-solving. Psychonomic Science, 1971, 23, Davis, J. H., Carey, M. H., Foxman, P. N., & Tarr,
397-399. D. B. Verbalization, experimenter presence, and
Bertini, M., Lewis, H. B., & Witkin, H. A. Some pre- problem solving. Journal of Personality and Social
liminary observations with an experimental pro- Psychology, 1968, 8, 299-302.
cedure for the study of hypnagogic and related Dean, S. J., & Martin, R. B. Reported mediation as
phenomena. Archive di Psicologia, Neurologia e a function of degree of learning. Psychonomic Sci-
Psichiatria, 1964,6,493-534. ence, 1966, 4, 231-232.
Boersma, F. J., Conklin, R. C., & Carlson, J. E. de Groot, A. D. Thought and choice in chess. The
Effects of reporting associative strategies on the Hague: Mouton, 1965.
retention of paired-associates. Psychonomic Sci-
DeNike, L. D. The temporal relationship between
ence, 1966, S, 463-464.
Bower, A. C., & King, W. L. The effect of number awareness and performance in verbal conditioning.
Journal of Experimental Psychology, 1964, 68,
of irrelevant stimulus dimensions, verbalization,
521-529.
and sex on learning bi-conditional classification
rules. Psychonomic Science, 1967, 8, 453-454. Dulany, D. E. The place of hypotheses and inten-
Brehmer, B. Hypotheses about relations between tions: An analysis of verbal control in verbal con-
scaled variables in the learning of probabilistic in- ditioning. In C. W. Eriksen (Ed.), Behavior and
ference tasks. Organizational Behavior and Human awareness. Durham, N.C.: Duke University Press,
Performance, 1974,11, 1-27. 1962.
Brehmer, B., Kuylenstierna, J., & Liljergren, J. E. Dulany, D. E., & O'Connell, D. C. Does partial rein-
Effects of functional form and cue validity on the forcement dissociate verbal rules and the behavior
subjects' hypotheses in probabilistic inference tasks. they might be presumed to control? Journal of
Organisational Behavior and Human Performance, Verbal Learning and Verbal Behavior, 1963, 2, 361-
1974, 11, 338-354. 372.
Brehmer, B., Kuylenstierna, J., & Liljergren, J. E. Duncan, C. P. Effects of instructions and informa-
Effects of information about the probabilistic na- tion on problem solving. Journal of Experimental
ture of the task on learning of uncertain inference Psychology, 1963, 65, 321-327.
tasks. Umea Psychological Reports (No. 90). Umea, Duncker, K. On problem solving. Psychological
Sweden: Department of Psychology, University of Monographs, 1945, 58(5, Whole No. 270).
Umea, 1975. Durkin, H. E. Trial-and-error, gradual analysis and
 VERBAL REPORTS AS DATA 249

sudden reorganization: An experimental study of Therapy and Experimental Psychiatry, 1976, 7,

problem solving. Archives of Psychology, 1937, 210. 213-219.
Ericsson, K, A, Instruction to verbalize as a means Kennedy, T. D. Verbal conditioning without aware-
to study problem solving processes with the 8- ness: The use of programmed reinforcement and
puzzle: A preliminary study (No. 458). Reports recurring assessment of awareness. Journal of Ex-
from the Department of Psychology. Stockholm, perimental Psychology, 1970, 84, 487-494.
Sweden: University of Stockholm, 1975, (a) Kennedy, T. D. Reinforcement frequency, task char-
Ericsson, K. A. Problem-solving behaviour with the acteristics and interval of awareness assessment as
8-puzzle. IV: Process in terms of sequences of factors in verbal conditioning without awareness,
moves (No. 448). Reports from the Department Journal of Experimental Psychology, 1971,88, 103-
of Psychology. Stockholm, Sweden: University of 112.
Stockholm, 1975. (b) Klinger, E. Structure and functions of fantasy. New
Ericsson, K. A., & Simon, H. A. Sources of evidence York: Wiley, 1971.
on cognition: A historical overview. In T. V. Mer- Klinger, E. Utterances to evaluate steps and control
luzzi, C. R. Glass, & M. Genest (Eds.), Cognitive attention distinguish operant from respondent
assessment. New York: Guilford Press, in press, thought while thinking out loud. Bulletin of the
Feldman, J. An analysis of predictive behavior in a Psychonomic Society, 1974, 4, 44-45.
two-choice situation. Unpublished doctoral disser- Lashley, K. S. The behavioristic interpretation of
tation, Carnegie Institute of Technology, 19S9. consciousness II. Psychological Review, 1923, 30,
Flanagan, J. C. The critical incident technique. Psy- 329-353.
chological Bulletin, 1954, 51, 327-358. Loftus, G. R. Eye fixations and recognition memory
Frankel, F., Levine, M., & Karpf, D. Human dis- for pictures. Cognitive Psychology, 1972, 3, 525-
crimination learning: A test of the blank-trials 551.
assumption. Journal of Experimental Psychology, Loftus, G. R., & Bell, S. M. Two types of informa-
1970, 85, 342-398. tion in picture memory. Journal of Experimental
Gagne, R. H., & Smith, E. C. A study of the effects Psychology: Human Learning and Memory, 1975,
of verbalization on problem solving. Journal of 1, 103-113.
Experimental Psychology, 1962, 63, 12-18. Maier, N. R. F. Reasoning in humans, II: The solu-
Greenspoon, J. The reinforcing effect of two spoken tion of a problem and its appearance in conscious-
sounds on the frequency of two responses. Ameri- ness. Journal of Comparative Psychology, 1931,12,
can Journal of Psychology, 1955, 68,409-416. 181-194.
Heidbreder, E. An experimental study of thinking. Merz, F. Der Einfluss des Verbalisierens auf die
Archives of Psychology, 1924, 73. Leistung bei Intelligenzaufgaben, Zeitschrift fur
Heidbreder, E. A study of the evolution of concepts. Experimented und Angewandte Psychologie, 1969,
Psychological Bulletin, 1934, 31, 673. (Abstract) 16, 114-137.
Heidbreder, E. Language and concepts. Psychological Montague, W. E. Elaborative strategies in verbal
Bulletin, 1936, 33, 724. (Abstract) learning and memory. In G. Bower (Ed.), The
Hendrix, G. A new clue to transfer of training, Ele- psychology of learning and motivation (Vol. 6).
mentary School Journal, 1947, 48, 197-208. New York: Academic Press, 1972.
Henry, L. K. The role of insight in the analytic Morgan, C. L. Characteristics of problem-solving be-
thinking of adolescents. Studies in education, Uni- havior of adults. Studies in Education, University
versity of Iowa Studies, 1934, 9, 65-102. of Iowa Studies, 1934, 9, 105-143.
Horton, D. L. The effects of meaningfulness, aware- Neisser, U. Cognitive psychology. Englewood Cliffs,
ness and type of design in verbal mediation. Jour- N.J.: Prentice-Hall, 1967.
nal of Verbal Learning and Verbal Behavior, 1964, Newell, A., & Simon, H. A. Human problem solv-
3, 187-194. ing. Englewood Cliffs, N.J.: Prentice-Hall, 1972.
Horton, D. L., & Kjeldergaard, P. K. An experimental Nisbett, R. E., & Wilson, T. D. Telling more than
analysis of associative factors in mediated general- we can know: Verbal reports on mental processes.
izations. Psychological Monographs, 1961, 7J(11, Psychological Review, 1977, 84, 231-259.
Whole No. 515). O'Connell, D. C. Concept learning and verbal con-
Johnson, E. S. An information processing model of trol under partial reinforcement and subsequent
one kind of problem solving. Psychological Mono- reversal or nonreversal shift. Journal of Experi-
graphs, 1964, 78(4, Whole No. 581). mental Psychology, 1965, 69, 144-151.
Karpf, D. A. Thinking aloud in human discrimination Peterson, L. R. Concurrent verbal activity. Psycho-
learning (Doctoral dissertation, State University logical Review, 1969, 76, 376-386.
of New York at Stony Brook, 1972). Dissertation Phelan, J. G. A replication of a study on the effects
Abstracts International, 1973, 33, 6111B. (Univer- of attempts to verbalize on the process of concept
sity Microfilms No. 73-13, 625) attainment. Journal of Psychology, 1965, 59, 283-
Karpf, D., & Levine, M. Blank-trial probes and in- 293.
trotacts in human discrimination learning. Journal Quinton, G., & Fellows, B. J. "Perceptual" strategies
of Experimental Psychology, 1971, 90, 51-55. in the solving of three-term series problems. British
Kazdin, A. E. Assessment of imagery during covert Journal of Psychology, 1975, 66, 69-78.
modeling of assertive behavior. Journal of Behavior Ray, W. S. Verbal compared with manipulative solu-
250 K. ANDERS ERICSSON AND HERBERT A. SIMON

tion of an apparatus-problem. American Journal tion timing and length on problem solution and
of Psychology, 1957, 77, 289-290. quality of problem processing (Doctoral disserta-
Reed, S. K., & Johnsen, J. A. Memory for problem tion, University of Oregon, 1971). Dissertation
solutions. In G. Bower (Ed.), The psychology oj Abstracts International, 1972, 32, 5500B. (Univer-
learning and motivation (Vol. 11). New York: sity Microfilms No. 72-9, 560)
Academic Press, 1977. Simon, H. A. Models of discovery. Boston: Reidel,
Rees, H. J., & Israel, H. E. An investigation of the 1977.
establishment and operation of mental sets. Psy- Simon, H. A. Models of thought. New Haven: Yale
chological Monographs, 1935, 46((>, Whole No. University Press, 1979.
210). Singer, J. L. The inner world of daydreaming,. New
Rommetveit, R. Stages in concept formation and York: Harper & Row, 1975.
levels of cognitive functioning. Scandinavian Jour- Smedslund, J. Mental processes involved in rapid
nal of Psychology, 1960,1, 115-124. logical reasoning. Scandinavian Journal of Psy-
Rommetveit, R. Stages of concept formation. II: chology, 1968, 9, 187-205.
Effects of an extra intention to verbalize the con- Smedslund, J. Psychological diagnostics. Psychological
cept and stimulus predifferentiation. Scandinavian Bulletin, 1969, 71, 237-248.
Journal of Psychology, 1965, 6, 59-64. Smith, E. R., Miller, F. S. Limits on perception of
Rommetveit, R., & Kvale, S. Stages in concept for- cognitive processes: A reply to Nisbett and Wilson.
mation. Ill: Further inquiries into the effects of Psychological Review, 1978, 85, 355-362.
an extra intention to verbalize. Scandinavian Jour- Smoke, K. L. An objective study of concept forma-
nal of Psychology, 1965, 6, 65-74. (a) tion. Psychological Monographs, 1932, 42(4, Whole
Rommetveit, R., & Kvale, S. Stages in concept for- No. 191).
mation. IV: A temporal analysis of effects of an Sokolov, A. N. Inner speech and thought. New York:
extra intention to verbalize. Scandinavian Journal Plenum Press, 1972.
of Psychology, 1965, 6, 75-79. (b) Soliday, S. M., & Allen, J. A. Hazard perception in
Roth, B. The effect of overt verbalization on prob- automobile drivers: Age differences. (Report from
lem solving (Doctoral dissertation, New York Uni- Highway Safety Research Center). Chapel Hill:
versity, 1965). Dissertation Abstracts, 1966, 27, University of North Carolina, 1972.
9S7B. (University Microfilms No. 65-9, 321) Sowder, L. The influence of verbalization of dis-
Ruger, H. A. The psychology of efficiency. Archives covered numerical- or sorting-task generalizations
of Psychology, 1910, 15. on short-term retention in connection with the
Russel, W. A., & Storms, L. H. Implicit verbal chain- Hendrix hypothesis. Journal for Research in
ing in paired associate learning. Journal of Experi- Mathematics Education, 1974, S, 167-176.
mental Psychology, 1955, 49, 287-293. Spielberger, C. D. The role of awareness in verbal
Sallows, G. O., Dawes, R. H., & Lichtenstein, E. conditioning. In C. W. Eriksen (Ed.), Behavior
Subjective value of the reinforcer (RSv) and per- and awareness. Durham, N.C.: Duke University
formance: Crux of the S-R versus cognitive media- Press, 1962.
tion controversy. Journal of Experimental Psychol- Tversky, A., & Kahneman, D. Availability: A heuris-
ogy, 1971, 89, 274-281. tic for judging frequency and probability. Cogni-
Sargent, S. S. Thinking processes at various levels of tive Psychology, 1973, 5, 207-232.
difficulty. Archives of Psychology, 1940, 249, 5-58. Verplanck, W. S. Unaware of where's awareness:
Schneider, W., & Shiffrin, R. M. Controlled and auto- Some verbal operants—notates, monents and no-
matic human information processing: I. Detec- tants. In C. W. Eriksen (Ed.), Behavior and
tion, search, and attention. Psychological Review, awareness—A symposium oj research and inter-
1977, 84, 1-66. pretation. Durham, N.C.: Duke University Press,
Schuck, J. R., & Leahy, W. R. A comparison of 1962.
verbal and non-verbal reports of fragmenting vis- Walk, R. D. Effects of discrimination reversal on
ual images. Perception fr Psychophysics, 1966, 1, human discrimination learning. Journal of Experi-
191-192. mental Psychology, 1952, 44, 410-419.
Schwartz, S. H. Trial-by-trial analysis of processes Watson, J. B. Is thinking merely the action of lan-
in simple and disjunctive concept-attainment tasks. guage mechanisms? British Journal of Psychology,
Journal oj Experimental Psychology, 1966, 72, 456- 1920, ;7, 87-104.
465. Werner, H., & Kaplan, B. Symbol formation. New
Shiffrin, R. M., & Schneider, W. Controlled and York: Wiley, 1963.
automatic human information processing: II. Per- Wilder, L., & Harvey, D. J. Overt and covert ver-
ceptual learning, automatic attending and a gen- balization in problem solving. Speech Monographs,
eral theory. Psychological Review, 1977, 84, 127- 1971, 38, 171-176.
189. Williams, B. W. Verbal operant conditioning with-
Shipstone, E. I. Some variables affecting pattern out subjects awareness of reinforced conditions.
conception. Psychological Monographs, 1960, 74(17, Canadian Journal of Psychology, 1977, 31, 90-101.
Whole No. 504). Wilson, A. The verbal report of the concept in con-
Silveira, J. M. Incubation: The effects of interrup- cept-learning research (Doctoral dissertation, Uni-
 VERBAL REPORTS AS DATA 251
versity of Victoria, 1973). Dissertation Abstracts Journal of Experimental Psychology, 1973, 25, 504-
International, 1974, 35, 1097B. 510.
Wilson, A. The inference of covert hypotheses by Wood, D. J., Shotter, J. D., & Godden, D. An in-
verbal reports in concept-learning research. Quar- vestigation of the relationships between problem
terly Journal of Experimental Psychology, 1975, solving strategies, representation and memory.
27, 313-322. Quarterly Journal oj Experimental Psychology,
Wilson, A., & Spellacy, F. Cognitive process in audi- 1974, 26, 252-257.
tory concept acquisition. Psychonomic Science, Woodworth, R. S. Experimental psychology. New
1972, 28, 383-34S. York: Holt, 1938.
Wood, D. J., & Shotter, J. D. A preliminary study
of distinctive features in problem solving. Quarterly Received January 3, 1980 •