Journal of Vision (2003) 3, 49-63 http://journalofvision.

org/3/1/6/ 49

Visual memory and motor planning in a natural task

Center for Visual Science, University of Rochester,
Mary M. Hayhoe Rochester, NY, USA

Center for Visual Science, University of Rochester,

Anurag Shrivastava Rochester, NY, USA

Center for Neural Science, Rochester Institute of Technology,

Ryan Mruczek Rochester, NY, USA

Center for Neural Science, Rochester Institute of Technology,

Jeff B. Pelz Rochester, NY, USA

This paper investigates the temporal dependencies of natural vision by measuring eye and hand movements while
subjects made a sandwich. The phenomenon of change blindness suggests these temporal dependencies might be
limited. Our observations are largely consistent with this, suggesting that much natural vision can be accomplished with
“just-in-time” representations. However, we also observe several aspects of performance that point to the need for some
representation of the spatial structure of the scene that is built up over different fixations. Patterns of eye-hand
coordination and fixation sequences suggest the need for planning and coordinating movements over a period of a few
seconds. This planning must be in a coordinate frame that is independent of eye position, and thus requires a
representation of the spatial structure in a scene that is built up over different fixations.

Keywords: natural tasks, saccades, eye-hand coordination

Simons, 1996). Many of these, and more recent studies,

Introduction have been reviewed by Simons and Levin (1997) and
One of the fundamental issues in visual perception is Simons (2000), and this insensitivity to changes has been
how visual mechanisms operate over time scales longer described as “change blindness.” Since detection of a
than a single fixation. Visual operations are normally change requires a comparison of the information in
embedded in the context of extended behavioral different fixations, change blindness has been interpreted
sequences. However, we have limited understanding of as evidence that only a small part of the information in
how visual processes operate in the service of natural, the scene is retained across fixations. Irwin suggests that it
ongoing, behavior. A central aspect of vision in its natural is limited by the capacity of working memory, that is, to a
context is how we make the transition from the small number of individual items whose identity is
computations within a fixation to those that operate remembered better than their location (Irwin, 1996).
between fixations. To what extent does the current Thus memory from prior fixations is primarily semantic
computation depend on information acquired in in nature, suggesting a large degree of independence of
previous fixations, or are visual operations within a the visual computations within individual fixations.
fixation essentially independent? This question has It is not clear, however, how much we can generalize
traditionally been addressed in the context of integration these findings to natural vision. What information in a
of information across saccadic eye movements: whether scene do observers actually need, and how much of this
there is such an integrated representation of a visual information persists past a given fixation? Although some
scene, and what the contents of that representation might studies have examined change blindness in the real world
be (Irwin, 1991; Rayner & Pollatsek, 1983). The (Simons & Levin, 1997), most paradigms examine a
conclusion from a large body of previous work is that single visual or motor operation over repeated trials.
representation of information acquired in prior fixations Visual function in this context may be fundamentally
is very limited. Evidence for limited memory from prior different from active participation in a real scene.
fixations is provided by the finding that observers are Observers almost certainly fine-tune their behavior to the
extremely insensitive to changes in the visual scene during experimental demands. For example observers are very
an eye movement, film cut, or similar masking stimulus sensitive to the probabilistic structure of the trials, and
(Hendersen, 1992; Hochberg, 1986; Irwin, 1991; Irwin, match the distribution of attention to expected events
Zacks, & Brown, 1990; Pollatsek & Rayner,1992; (Mack & Rock, 1996). In natural behavior, the observer
O'Regan, 1992; Rensink, O'Regan, & Clark, 1997; performs a sequence of different computations, whose

DOI 10:1167/3.1.6 Received March 7, 2002; published February 3, 2003 ISSN 1534-7362 © 2003 ARVO
Hayhoe, Shrivastava, Mruczek, & Pelz 50

initiation and timing is controlled by the observer, not by and priming of pop-out might be mechanisms that guide
the experimenter. This active initiation of behaviors is attention and eye movements in scenes.
likely to be important. For example, viewing a picture of a The goal of the present investigation was to examine
scene is very different from acting within that scene, the fixation patterns in natural behavior, in order to gain
simply because the observer needs different information. insights about the way that natural behavior might
Some evidence for the importance of observer actions is depend on information in prior fixations. We recorded
given by Wallis and Büthoff (2000), who showed that eye and hand movements while making a sandwich. This
drivers and passengers in a virtual environment have task was modeled on a similar one used by Land, Mennie,
different sensitivity to changes in the scene. Other and Rusted (1999), who recorded fixation patterns while
evidence also suggests the importance of the immediate observers made a cup of tea. Like tea-making, the
task in determining what is detected (Folk, Remington, & sandwich making task allows the observer considerable
Johnston, 1992; Hayhoe, Bensinger, & Ballard, 1998). flexibility while still providing an explicit set of behavioral
Another difference that is likely to be important is goals for the observer. Some explicit manifestation of the
the nature of the stimulus array. Investigations of change observer’s goals is required for understanding the
blindness typically involve viewing either two-dimensional behavior. The focus of the current investigation was to
pictorial representations of scenes or simple arrays of understand the temporal dependencies of natural
letters or geometric figures. These displays differ from behavior. Thus, to what extent is visual information that
normal scenes in their spatial structure. One difference is was acquired in prior fixations needed for performing the
spatial scale. The visual angle subtended by an image of a task. In particular, to what extent is such information
room in a typical experimental display, for example, is needed for guiding eye and hand movements? Our
very different from being in a real room, and it is not observations confirm earlier studies demonstrating the
clear how such infidelities in spatial scale might affect transient, task-specific nature of the information extracted
observers’ representations of the spatial structure of the within a fixation (Ballard, Hayhoe, & Pelz, 1995; Hayhoe
scene. Depth information introduces an additional level Bensinger, & Ballard, 1998; Land et al, 1999). Thus
of spatial complexity in normal vision and poses a greater much visual processing is accomplished within a fixation.
challenge for the visuo-motor apparatus. Because of this, change blindness may not be much of a
Both active participation and spatial structure are limitation in normal performance, because much of the
likely to be important for understanding the visual information in a scene is not in current use. However, we
representations that are used to locate targets for the eyes also observe several aspects of performance that point to
and hands, and to coordinate the movements of eyes, the need for some representation of the spatial structure
head, hands, and body. These behaviors are important of the scene that is built up over different fixations.
requirements of any situation, and Chun and Nakayama Patterns of eye-hand coordination and fixation sequences
(2000) pointed out the potential importance of implicit suggest the need for planning and coordinating
memory structures for guiding attention and eye movements over a period of a few seconds. This planning
movements around a scene. They argue that such must be in a coordinate frame that is independent of eye
guidance requires continuity of visual representations position and thus requires a representation of the spatial
across different fixation positions. In contrast to the structure in a scene that is built up over different
findings of the change blindness experiments, there is fixations.
evidence to suggest that subjects do in fact build an
implicit memory representation of the spatial structure of
the environment. Chun and Jiang (1998) showed that Methods
visual search is facilitated (by 60-80 msec) by prior
exposure to visual contexts associated with the target. Subjects wore an eye-tracker mounted on the head,
They suggest that this reflects sensitivity to the redundant and were seated at a table with the items required for
structure in a scene, which remains invariant across making a sandwich. They were thus free to make natural
multiple gaze points. It seems likely that observers are movements. No instructions were given except to make a
sensitive to this invariance. Other evidence for an peanut butter and jelly sandwich and to pour a glass of
influence of prior views is “priming of popout.” This is soda. Observations were made on 11 subjects. Seven of
the reduction of both search latencies and saccade the subjects made the sandwich with the layout
latencies to locations or features that have been recently demonstrated in Figure 1 (top).
presented (Maljkovic & Nakayama, 1994; McPeek, The necessary items were laid out on the table in
Skavenski, & Nakayama, 2000). Such mechanisms do not front of the observer, with a few background items
require conscious intervention, and exhibit greater irrelevant to the task. Four subjects made the sandwich
memory capacity, longer durability, and greater with a more cluttered layout, as shown in Figure 1
discriminability than explicit short-term visual memory. (bottom), where a number of arbitrarily chosen irrelevant
Chun & Nakayama proposed that both contextual cueing items (other food items, tools, silverware) were
Hayhoe, Shrivastava, Mruczek, & Pelz 51

over a central 40° field. Both pupil and first Purkinje

image centroids are recorded, and horizontal and vertical
eye-in-head position is calculated based on the vector
difference between the two centroids. This technique
reduces artifacts due to any movement of the headband
with respect to the head. (Errors in reported eye position
caused by movement of the headband with respect to the
head were less than 0.1°, measured over a sequence of
movements at a peak velocity of 60°/sec.) Both trackers
provide a video record of eye position. The ISCAN
headband held a miniature “scene-camera” to the left of
the subject’s head, aimed at the scene. The ASL’s scene
camera was mounted so as to be coincident with the
observer’s line of sight. The tracker creates a cursor,
indicating eye-in-head position, that is merged with the
video from the scene-camera, providing a video record of
the scene from the subject’s perspective on the scene-
monitor, with the cursor indicating the intersection of the
subject’s gaze with the working plane. Because the scene-
camera moves with the head, the eye-in-head signal
indicates the gaze point with respect to the world. Head
movements appear on the record as full field image
motion. (Because the ISCAN scene camera was not
coaxial with the line of sight, calibration of the video
signal was strictly correct for only a single plane.
Calibration was close to the plane of the table, so the
parallax error was significant when subjects lifted objects
out of that plane toward the body.)
The eye tracker was calibrated for each subject before
each trial. The subject was seated at the work surface,
with all items within reach. At this distance, the plate
close to the observer subtended about 20° of visual angle,
and the peanut butter and jelly subtended about 7°. All
the items were within about a 90° region. Calibration was
Figure 1. The two scene layouts used in the experiment. performed using a nine-point grid, over a region of about
50° by 40°. (This region moves with the subject’s head.)
Following data collection, which took about two minutes
interspersed with the items required for the task. Before per subject, the video records were analyzed on a frame-
the experiment, the layout was occluded by a cardboard by-frame basis, recording the time of initiation and
sheet showing the calibration points. Following termination of each eye and hand movement, the
calibration, this was withdrawn, and the subjects location of the fixations, the nature of the hand actions,
immediately began the task. The research followed the and periods of track loss. These detailed records formed
World Medical Association Declaration of Helsinki and the basis of the summary statistics described below. For 4
was approved by the University of Rochester Research of the subjects, the image of the eye provided by the
Subjects Review Board. Informed consent was obtained tracker was superimposed on the record from the scene
from the subjects. camera either as a transparent overlay, or in the top
corner of the scene image. The eye image is shown in
Monitoring Eye Position Figure 2. Two crosshairs indicate the tracker’s calculation
of center of the pupil and corneal reflection. If either of
Monocular (left) eye position was monitored with these signals is lost, the corresponding crosshair
either an Applied Science Laboratories Model 501 or an disappears. This provides a mechanism for checking the
ISCAN eyetracker. The ISCAN was used with the scene video for transient track losses and blinks. The
uncluttered scene, and the ASL was used in the cluttered movement of the eye can also be seen in the eye image,
scene. Both are headband mounted, video-based, IR providing an additional source of information for
reflection eyetrackers. The eye position signal was identifying fixations and measuring their duration.
sampled at 60 Hz and had a real time delay of 50 msec.
The accuracy of the eye-in-head signal is approximately 1°
Hayhoe, Shrivastava, Mruczek, & Pelz 52

Figure 2.The eye image with cross-hairs indicating pupil center

and cormeal reflection.
Movie 1. A segment of the experimental task.

the jar, the right hand also moves toward the jar to
Results coordinate with the left hand in screwing it on the jar,
and so on. Thus the first fixation is for guiding knife
putdown. This requires both directional and distance
Task Specific Fixations information for controlling the arm and computing the
In agreement with Land et al’s (1999) observations on contact forces. The fixation on the lid on the table is
tea-making, we found that fixation patterns were highly required to guide pickup. This involves computing
directed, and used to acquire specific information just as information to control the grasp, including the position,
it was needed for the momentary task. A description of a orientation, and size of the lid, and perhaps recalling
small segment of the task is shown in Figure 3 and Movie from memory information about surface friction and
1. At the beginning of the segment, the subject is fixating weight to plan the forces. The intervening fixation on the
the completed sandwich on the plate, guiding the knife to jar may provide information for the future movement to
cut the sandwich with the right hand, and the left hand place the lid on the jar. The final fixation on the rim of
steadies the bread. Gaze is then transferred to the edge of the jar initially guides the direction and posture of the left
the plate to guide placement of the knife with the right hand to contact the jar, then the right hand movement
hand. The left hand simultaneously begins to move and posture to the jar, and then the lid placement and
toward the lid of the jelly jar on the table. the screwing action. Thus fixations are tightly locked to
While the right hand completes placement of the the task, and their role is well-defined. Fixations on task-
knife, the eye fixates the jelly jar briefly, then fixates the relevant objects were typically close in time to their use in
lid to guide pickup with the left hand. The eye then the task. For example, in this subject, except for fixations
returns to the jelly jar to guide the lid towards the jar. Just immediately on viewing the scene, the soda was not
before the left hand, holding the lid, makes contact with fixated until the subject was about to pour a glass.

Sandwich Plate Jelly Lid Jelly Jar

EYE

RH Holding Knife Knife to Pause Move to Screw on

Plate Jelly Lid Lid
LH
Steady Hand to Lid on Lid to Jar Screw on
Sandwich Table Lid

0 400 800 1200 1600 1800 2200 2600 2800 3200 3400 3800

time (msec)

Figure 3. Sequence of eye and hand movements shown in Movie 1.

Hayhoe, Shrivastava, Mruczek, & Pelz 53

Fixations appear to play a specific role, depending on impossible to identify small saccades within a radius of
momentary task context. The locations of the fixations on about 1.5° around fixation. If these were present, the
the objects were different for different actions, for number of long fixations would be overestimated.
example, subjects fixate the middle of the jar for grasping Although it is impossible to know what the role of
with the hand in a vertical posture, and the rim for individual fixations is from such observations, it appears
putting on the lid, with the hand in a horizontal posture. that, to a first approximation, the fixation durations are
This suggests that the visual information being extracted determined by the momentary task demands. Gaze often
controls the pre-shaping in one case, and the orientation departs just at the point a hand movement is complete, or
of the lid in the other. To the extent that information is there is no longer need for visual guidance. An example
obtained at the moment it is needed, visual computations of this is given in Figure 3, and in the accompanying
depend only on the information available within that video, where the eye departed from controlling knife
fixation. placement when the knife was close to the plate, and the
In the first scene, there were a number of objects remainder of the movement could be controlled using
surrounding the work area, as shown in Figure 1 top, a somatosensory information. The eye then arrived to guide
monitor, camera, tools etc. Subjects rarely fixated these lid pickup just as the left hand approached the lid.
background items. This occurred on only 0.02 of the Similar time-locking of fixations to critical stages of the
fixations (one or two per subject). In the second, actions was observed by Johansson et al (2001). This is, of
cluttered, scene, shown in Figure 1 bottom, a variety of course, an incomplete description of determinants of
irrelevant objects were present, interspersed with the fixation duration. In a number of instances vision may
items needed for the task. There was an approximately not be providing critical information for the ongoing
equal number of irrelevant as relevant items. In this case, action. For example, screwing the cap on the soda bottle
an average of 0.2 +/- 0.04 of the fixations were made on can be completed under proprioceptive control and it is
irrelevant items. not obvious what role is being played by fixation during
such periods.
Fixation Durations
The duration of each fixation was calculated from the 12
A
video transcriptions. The frequency distributions are 10

shown in Figures 4 and 5. Data for the three subjects in 8

Figure 4 were recorded with the image of the eye provided
by the tracker, superimposed on the record from the 6

scene camera. This allowed careful monitoring of the 4

fixation durations measurements, since a transient track 2

loss sometimes results in a deviation of the cursor
position, and thus appears like the termination of a 0

fixation. Movement of the eye during the track loss could 12

be observed directly in the eye image. The distributions 10

B

are quite similar for the different subjects. The data for
the seven subjects in Figure 5 did not have the eye image
Frequency

available on the video record. It is therefore possible that 6

these data are partially contaminated by transient track 4

losses. This should not be a major factor, however, as
2
segments of the tape where the cursor disappeared,
indicating a track loss, were eliminated from the analysis. 0

The most distinctive feature of these distributions is their 12

C
wide spread. Fixations range from under 100 msec to over 10

1500 msec. There is some variation between subjects, but 8

most of the distributions have a mode between 100 and
6
200 msec, which is less than for reading or picture
viewing (Henderson & Hollingworth, 1999). The very 4

long fixations are usually associated with some prolonged 2

action of the hands that required continuous guidance, 0

such as spreading, scooping out peanut butter, pouring, 500 1000 1500 2000

or undoing the tie on the bread bag. Land et al (1999) Fixation Duration (ms)
observed a similarly wide spread of fixation durations in
their tea-making task. For the long fixations, it is Figure 4. Distribution of fixation durations for 3 subjects using
important to note that the noise in the tracker made it the eye image.
Hayhoe, Shrivastava, Mruczek, & Pelz 54

20
A B
15

10

Frequency 0

20
C D
15

10

0
500 1000 1500 2000 500 1000 1500 2000

Frequency Duration (ms)

Figure 5. Distribution of fixation durations for another 4 subjects without the eye image.

An interesting feature of the distributions is the fixations occur on a variety of occasions and are not
frequency of very short fixations. All subjects show a limited to the interval before a corrective saccade.
number of fixations of two to four video frames (66 – 133
Table 1. Categories of Fixations Less than 100 msec.
msec). The measurement of the short fixations for these
subjects is thus highly reliable, within the temporal Frequency Type of fixation
resolution limits of the video record (30 Hz). The short 0.24 guide reach
fixations do not appear to play a single specialized role.
0.1 corrective
We examined all the fixations of 100 msec or less and
attempted to categorize them according to the context. 0.07 in path
The frequency of the various categories is shown in Table 0.12 occlusion
1. Very short fixations have previously been observed 0.37 other
between the primary and corrective saccade to a
remembered target location (Becker & Fuchs, 1969). In These short fixations are of interest because the time
our experiment only about 0.1 of the short fixations to program a saccade is reliably found to be in the 200-
could be potentially classified as preceding a corrective 250 msec range. Consequently these brief fixations must
saccade. We classified corrective saccades as those where be part of pre-programmed sequences of saccades (Becker
the eye landed on an object (for example the bottle), and & Jurgens, 1979). This planning must be done in a
then moved to an adjacent location on the object (the spatial, not retinal coordinate frame, and the partially
neck), followed by an action involving the object programmed second saccade updated for the first
(pouring). The occurrence of the action suggests that the movement. Thus pre-programmed sequences of saccades
second fixation locus is the intended target, though this is point to the existence of a representation in spatial
not known with any certainty. Of the other short coordinates, independent of eye position.
fixations, 0.24 occurred while guiding some kind of
reaching movement, either for picking up or placement;
0.07 were on a task-relevant object located more or less
Initial Fixations
on the path of the saccade, between the pre-saccadic The strict capacity limits on the information that can
position and the location of the next item to be be retained across fixation positions raise the question of
manipulated. Notably these objects were ones needed at whether there is some representation of scenes that is
some other point in the task. For example, a brief fixation built up over time. O’Regan & Levy-Schoen (1983) and
might occur on the knife, positioned between plate and Irwin (Irwin, 1991; Irwin et al, 1990) suggest there is
bread, as the subject moved from plate to bread to open some sparse, post-categorical description of the objects
the bread bag. About 0.12 of the fixations occurred and their locations in a scene accumulated over different
following some kind of occlusion of the point of interest eye positions. This seems plausible, since in most ordinary
by a hand or following a blink. The remaining 0.37 of the environments observers have prolonged exposure to the
fixations could not be obviously classified. Thus the scene, and multiple opportunities to accumulate
Hayhoe, Shrivastava, Mruczek, & Pelz 55

information, despite the capacity limits of visual short-

term memory. However, it is not known what subjects do
in natural viewing. We were therefore interested to
observe what subjects look at when they first view a novel
scene. Do they in fact make a series of exploratory eye
movements as we might expect if they are building a
representation of scene layout for later use? We therefore
examined the fixations made by subjects after the scene
was initially exposed by removing the calibration display,
and before the first reaching movement, which indicated
that they had begun the task. We found that on the
initial exposure, subjects scan the scene and make a series
of fixations on the objects, before the first reaching
movement is initiated. Eac of the 11 subjects made
between 3 and 21 fixations on this initial exposure. The
mean number of fixations was 8.9 +/- 1.5, as shown in
Table 2.
Movie 2. Initial fixations of one subject.
Table 2. Number and Duration of Pre-Task Fixations, and
Frequency of Fixations on Irrelevant Objects.
Eye-Hand Latencies
Pre-task fixations 8.9 +/- 1.5
Little is known about the targeting of reaches in
Fixation duration 197 msec +/- 26
Pre-task natural behavior. A straightforward way in which the
Irrelevant objects 0.17 +/- 0.04 target of a reach might be selected is for the subject to
During task visually search the peripheral retina for the desired object,
Irrelevant objects 0.48 +/- 0.07 and then to program both the reach and the
Pre-task accompanying saccade on the basis of this information.
Experiments on the relative timing of eye and hand
An example of one subject’s fixations on first view is movements to a target reveal eye-hand latencies close to
given in Figure 6. This subject makes a series of short zero, consistent with this speculation (Abrams et al,
fixations on the bread, the peanut butter, in between the 1990). However, in a typical experiment the target is
peanut butter and the jelly, two fixations on the bread, usually presented at the onset of the trial, and there is
then on the jelly, between soda and jelly, and then to the little opportunity to locate the target ahead of time,
bread bag to guide the first reaching movement. A second unlike the natural world, where objects are continuously
subject’s initial fixations are shown in Movie 2. In the available. When the target is continuously present
case of subjects who used the cluttered scene, these initial observers have the opportunity to plan for the arm
fixations were distributed fairly equally between relevant movement. Such planning is an essential component of
and irrelevant objects (0.48 +/- 0.07, on irrelevant motor behavior, and allows speedier movements as well as
objects). During task performance, however, the coordination with other movements, such as the other
proportion of fixations on irrelevant objects went down hand or the body. We measured the latency between eye
to 0.16 +/- 0.04. This suggests that subjects are doing and hand movements for all the reaches that subjects
something different in the initial fixations. The initial make. The initiation of both eye and hand movements
fixations were typically quite short (mean 197 msec +/- was taken from the video record, using the first frame on
26). Thus the information being acquired in these which a translation could be detected. The frequency
fixations does not take extensive visual analysis. distributions of eye-hand latencies for seven subjects are

coke+ fixate on
plate pb jar pb/jelly bread bread bag jelly jelly jar bread bag
to guide right hand

0 200 400 600 800 1000 1200 1400 1600 1800 2000 2200 2400 2600 2800

time (msec)

Figure 6. Sequence of fixations made by a subject on first viewing the scene.

Hayhoe, Shrivastava, Mruczek, & Pelz 56

14 14

A B
12 12

10 10

8 8

6 6

4 4

2 2

0 0

14 14

C D
12 12

10 10

8 8

6 6

4 4

2 2

0 0

14 14

E F
12 12

10 10

8 8

6 6

4 4

2 2

0 0
-1000 -800 -600 -400 -200 0 200 400 600 800 1000
14
Time (ms)
G
12

10

0
-1000 -800 -600 -400 -200 0 200 400 600 800 1000

Time (ms)

Figure 7. Eye-hand latency distributions for 7 subjects. The hand leads for negative values.

shown in Figure 7. (For a small number of the reaches the the control of putdown actions. These reaches must have
hand was not visible in the video at the beginning of the been controlled using either peripheral vision, visual
movement and these were omitted.) memory, or perhaps somatosensory information about
Most (0.87 +/ - 0.03) of the reaching movements were the height of the table. Even when the reach was
accompanied by a fixation on the target. When the hand accompanied by a fixation, there was substantial flexibility
movement was not accompanied by a fixation, it was in the stage of the reach when the fixation occurred. On a
almost always for the purpose of placing an object on the number of occasions, 0.19 =/- 0.02, a substantial fraction
table. Only a very few of the pickup actions were not of the movement was accomplished without fixation on
accompanied by fixation at some stage of the movement. the target. Although the predominant strategy is for eye
This suggests that foveal information was less critical for and hand to depart close together in time, all subjects
Hayhoe, Shrivastava, Mruczek, & Pelz 57

LOOK AHEAD LOOK AHEAD

top of top mid pb

EYE bread bag pb plate
plate
pb jar
pb jar
jelly jar

RH move slices put bread down on move pb jar

pull slices out of bag separate slices move toward pb
to left hand plate inwards

LH move towards bread steady bread bag

move in to grasp bread from
separate slices
slice move
pause
move
get bread right hand down inward to pb

0 200 400 600 800 1000 1200 1400 1600 1800 2000 2200 2400 2600 2800 3000 3200 3400 3600 3800

time (msec)

Figure 8. Sequence of eye and hand movements during a segment of the task, showing look-ahead fixations on the peanut butter jar,
and later on the jelly jar.

show a number of movements where the reach was msec later while screwing on the lid, and then fixates it
initiated well ahead of, or later than, the eye movement, for the third time 3,660 msec after the first look, this time
as shown in Figure 7. Presumably, these reaches could be maintaining fixation until the reach to the jelly is
completed without further visual input, or with initiated. These fixations on objects that were to be
peripheral guidance. Similarly, the eye frequently fixates picked up shortly afterwards may indicate that the subject
the object for as much as a second before the initiation of is planning a reach, and is looking to the object to acquire
the reach. These large lags and leads result from the its spatial location for guiding the next movement.
interweaving of visual control of the two hands, with Another example of lookahead fixations is shown in
some movements starting while the eye is supervising the Figure 8. It seems likely that the spatial memory
other hand's action. An example of this can be seen in information facilitates the targeting of the saccade and
Figure 3, where the movement of the left hand towards perhaps initiates programming the reach. Similar “look-
the lid begins at the same time that the eye and right ahead” fixations were observed by Pelz et al (2001) in a
hand move to put down the knife, about 800 msec after hand-washing context. As subjects approached the wash
the start of the record. The eye does not move to the lid basin they fixated the tap, soap, and paper towels in
until about 600 msec later (at 1400 msec), after the right sequence, before returning to fixate the tap to guide
hand movement is complete. These long relative latencies contact with the hand.
suggest that the next eye or hand movement may be
planned as much as a second ahead of time. For example,
if fixation of an object is required for final guidance of
the reach, the fixation must be planned to some extent
when the reach is initiated, so as to be there when
needed. Since several fixations intervene between the eye
and hand movement to the object, this planning must
occur in a representation that is independent of eye
position. This can be seen in Figure 3. While the left
hand moves to the jelly lid, a fixation is made on the
plate, then on the jelly, before the saccade to the jelly lid
is initiated. While these arguments are indirect, they
make a plausible case for visual representations that span
fixations and are maintained over a period of a second or
more, to coordinate visually guided movements.

Fixations Prior to Reaching

Reaching behavior provides another clue that a Movie 3. An example of a reach preceded by a fixation.
representation of the locations of objects is preserved
across fixations. In 11 subjects examined, we found that
0.3 (+/- 0.06) of the reaches that subjects made to pick up Discussion
objects were preceded by a fixation on that object in the In general, these observations suggest that the visual
recent past (less than 8 sec). (This is prior to the fixation operations within a given fixation are highly specific to
on the object during the actual reach.) An example of this the immediate task. The dependence of fixation location
is given in Movie 3. The subject fixates the jelly while on the immediate task was also observed by Land in the
picking up the peanut butter jar lid, fixates it again 1300 tea-making task (Land, Mennie & Rusted, 1999). Land et
Hayhoe, Shrivastava, Mruczek, & Pelz 58

al described performance as a sequence of “object related saccade, the degree of pre-planning of the next saccade,
acts”. Thus the sequence: pick up an object, move it to a and the time taken by the hands to complete other
new location, put the object down, would constitute an aspects of the task such as a manipulation or a reach.
object-oriented action, where fixation would be required Thus different visual goals require different
for picking up, then for targeting the location for computations. While such task dependence is to some
placement and guiding the placement. To pick up an degree inevitable, the extent to which fixation durations
object, observers typically fixate the point on the object vary moment by moment during task performance
where the hand makes contact. Similar step-by-step underscores the overriding control of visual operations by
control of hand actions by fixation at a specific locus in the internal agenda rather than the properties of the
the scene has been demonstrated under more controlled stimulus, and the range of different kinds of visual
circumstances by Johansson et al (2001) in a task where information that can be extracted from the same visual
subjects picked up a bar, moved it past an obstacle, and stimulus. The intrinsic salience of scene objects does not
used it to contact a switch. Fixations clustered at critical appear to be a major factor in attracting fixations in
loci for each segment of the movement, moving on to the normal vision, and models that depend entirely on
next locus as the action was completed. Other natural salience, such as that of Itti & Koch (2000) cannot be
behaviors, such as driving, playing cricket, and table generally applicable. The specificity of the information
tennis also reveal stereotyped fixation patterns for extracted within a fixation suggests a large degree of
acquisition of information critical to the momentary task independence of the visual computations within
needs. In driving, Land has shown that drivers reliably individual fixations, to the extent that the particular
fixate the tangent point of the curve to control steering information extracted does not depend on information
around the curve (Land & Lee, 1994). In cricket, players from prior fixations. This is consistent with the body of
exhibit very precise fixation patterns, fixating the bounce work indicating limited memory across fixation positions.
point of the ball just ahead of its impact (Land & For at least some proportion of the task, observers appear
McLeod, 2000). A similar pattern is seen in table tennis to access the information explicitly at the point of
(Land & Furneaux, 1997). In a task where observers copy fixation, at the time when it is needed, as opposed to
a pattern of colored blocks, Ballard et al (1995) showed relying on information from prior fixations. This
that block color and location are acquired in separate behavior is consistent with O’Regan’s suggestion that the
fixations on the pattern, just before block pickup and scene serves as a kind of external memory that can be
placement, respectively. quickly accessed when needed (O’Regan, 1992; Ballard et
The specificity of the information acquired in al, 1995).
different fixations is indicated not only by the ongoing
hand actions and the point in the task, but also by the Integrated Representations for Motor
durations of the fixations, which vary over a wide range. It
appears that a large component of this variation depends Planning
on the particular information required for that point in However, some aspects of natural behavior cannot be
the task, fixation being terminated when the particular accounted for this way. Land & Furneaux (1997) noted
information is acquired. In addition to the current the need for some kind of visual buffer both in driving,
observations, other evidence suggests that ongoing task is where the current information controls the steering
a primary factor in fixation duration. In the block-copying action about 800 msec later, and in piano playing, where
task, fixations for acquiring block location took about 75 the fixations lead the note played by about a second. In
msec longer than those for acquiring color (Hayhoe et al, the tea making task, Land et al (1999) also noted a
1998). In addition, different distributions of fixation number of instances where objects were found more
duration are observed for reading than for viewing easily when they had been fixated a few seconds
pictorial representations of scenes (Henderson & previously. The current observations provide further
Hollingworth, 1999; Viviani, 1991). Pelz et al (2000) evidence that memory across fixations is needed as a basis
observed different distributions for three phases of a for motor planning and coordination. First, observers
model-building task. There were three phases of the task: consistently scan the scene with a small number of brief
reading the instructions, searching for the pieces, and fixations before beginning the task. It seems plausible that
putting the pieces together, each with a characteristic this provides information about the identity and location
distribution of fixation durations. Epelboim et al (1995) of objects in the scene. The existence of a coarse scene
also observed shorter fixation durations for tapping than representation has been postulated by O’Regan, Irwin,
simply for looking at a sequence of lights on a table. The and coworkers (O’Regan & Levy-Schoen, 1983; O’Regan,
argument that the time required for acquisition of the 1992; Irwin 1991; Irwin Zacks & Brown, 1990). Ullman
currently needed information can, of course, be made in (1984) also suggested the need for such a representation,
only the most general terms. In any particular instance, which he suggested was extracted using some kind of
the duration of a fixation will depend on a variety of general-purpose routine. To this point, however, there has
other factors, such as the time to program the next been no evidence that observers in fact construct such a
Hayhoe, Shrivastava, Mruczek, & Pelz 59

representation in normal viewing. The scanning behavior et al (2000) also demonstrated concurrent programming
observed here hints at such a general-purpose of saccades in a similar situation where two targets were
representation. However, it would be necessary to observe in competition. Saccades to the wrong stimulus were
scanning as a common occurrence when observers view often followed by a second saccade to the correct stimulus
novel scenes, for this argument to have much force. with a very brief inter-saccadic interval. The frequency of
Other evidence shows that information about the spatial very short fixations we observe in the sandwich task
organization of scenes is preserved across fixations. For indicate that pre-programming, or concurrent
example, De Graef and Verfaille show encoding of spatial programming of more than one saccade, is a common
relationships of “bystander” objects that are not the target occurrence in ordinary movements, and not restricted to
of a saccade (de Graef et al, 2001; Verfaille et al, 2001). particular experimental situations. The significance of pre-
Melcher & Kowler (2001) showed memory for both the planning is that programming of the second (and
identity and location of about 8 objects in multiple scenes subsequent) saccade in a sequence must initially occur in
following inspection periods of a few seconds. a reference frame that is independent of the eye, and the
It seems likely that one function of an integrated second saccade is using information acquired prior to the
representation is for targeting (and planning) eye and immediately preceding fixation. This implies the existence
hand movements. In normal viewing, the target is of some form of spatial memory representation that is
frequently present in the peripheral retina, and can be precise enough to support saccadic targeting. McPeek &
located on the basis of stimulus features, so it is not Keller (2002) observed that neurons in the superior
obvious that spatial memory from a prior fixation would colliculus show activity related to preparation of the
be useful in target selection. However, other evidence second saccade even while the first saccade is still in
supports the idea that prior fixations facilitate target progress. Thus neural activity for more than one saccade
selection. For example, Epelboim et al (1995) found that can be maintained concurrently, even at levels close to the
the time taken to tap a specified sequence of colored motor output, and the neural activity for the second
lights arrayed on a table rapidly decreased as the task was saccade must be able to take into account the eye
repeated. Zelinsky et al (1997) found faster search times displacement by the first saccade. Freedman et al (1996)
and fewer saccades for target objects when subjects were have demonstrated that cells in the superior colliculus
given a pre-view of the spatial array prior to a search task. code gaze position in space (or with respect to the body),
McPeek & Nakayama (1999), showed that saccades to not retinal error. Thus the intrinsic organization of the
colored targets have shorter latency if a target of the same saccadic system appears to be in spatial coordinates.
color has been presented on the previous trial. A similar Under special conditions, the latency of saccades
result has been found in Frontal Eye Field neurons of evoked by a flashed stimulus can be in the 70-130 msec
monkeys by Bichot & Schall (1999). Chun & Jiang range, comparable to the fixation durations observed
(1998,1999), also showed that visual search is facilitated here. These are called express saccades (Fischer &
by prior exposure to the spatial context. Ramsberger, 1984). These are seen when the fixation
The behavior of observers in this study is consistent point is turned off before the stimulus appears, and
with the suggestion that a spatial memory representation usually involves substantial practice. Usually the stimulus
is used in targeting eye and hand movements. The is in one of two positions left or right of fixation. These
fixation distributions revealed an unexpectedly large saccades are commonly thought to be a special kind of
number of very short fixations in the range 70-130 msec. saccade that by-passes high level cortical control
This is much shorter than the time normally required to mechanisms (Fischer & Breitmeyer, 1987). However, the
program a new saccade. Saccade's evoked by a sudden high frequency of very brief fixations in natural contexts
onset typically occur with a latency of 200-250 msec. Thus suggests that planning is a fundamental aspect of saccade
these very short fixations show that observers must pre- programming, and that the short latencies observed in the
program two or more of the saccades. Zingale & Kowler express saccade paradigm are simply a result of motor
(1987) have demonstrated that saccades can be pre- planning. This is consistent with the suggestion of Kowler
programmed by showing that the latency to initiate a (1991), and is supported by recordings of the Superior
sequence of saccades increased with the number of Colliculus, that indicate some metrical preparation
saccades in the sequence. Very brief fixations have also indicated by increased activity in build-up neurons
been observed in circumstances where two targets are in (Munoz & Wurtz, 1995).
competition, such as a double-step task (Becker & Reaching movements also suggest the existence of a
Jurgens, 1979). Theeuwes and colleagues (Theeuwes et al, spatial representation independent of eye position. On a
1998, 1999; Irwin et al, 2000) observed short fixations to number of occasions a reaching movement was initiated
a distractor stimulus that was suddenly presented when up to a second ahead of the eye movement to the target.
subjects were preparing to saccade to a search target. They As indicated in Figure 3, several fixations could intervene
interpreted these brief fixations as the consequence of a between the initiation of the movement and the grasp.
concurrently programmed second saccade to the target, This means that the programming and control of the
which terminated the fixation on the distractor. McPeek reach was accomplished with the eye in two or more
Hayhoe, Shrivastava, Mruczek, & Pelz 60

positions with respect to the scene, and the reach must be

guided either by a spatial memory representation, or a Conclusions
visual representation that is independent of eye position. In conclusion, examination of eye and hand
This is interesting because neurophysiological evidence movements in natural behavior suggests that much of
from cells in the intraparietal sulcus suggests that what the visual system has to do is computed at the
reaching movements are programmed in an eye-centered moment it is needed for the particular task, and does not
coordinate frame (Batista et al, 1999). The above evidence appear to be heavily dependent on information acquired
suggests, instead, that this coordinate frame must be in prior gaze positions, in agreement with prior work
exocentric, rather than eye-centered. (Ballard et al, 1995). Thus the limitations of short term
In general, the relationship between the eye and hand memory, and the related susceptibility to change
movements is much more flexible than expected on the blindness may not be much of a limitation for normal
basis of previous experimental work. The wide range of visual function. However, there must be some scene
eye –hand latencies is very different from the usual single representation that corresponds to perceptual awareness.
trial experiments where the eye-hand latency is usually O’Regan (1992) and Irwin (1991) have postulated that
close to zero (Abrams et al, 1990). This difference is there is some integrated representation of the scene, but
presumably a consequence of the opportunity for motor suggest that the representation of spatial information is
planning afforded by the continuous presence of the imprecise and that the representation is semantic in
scene, as well as the need to interleave control of the two nature. The evidence presented here, however, supports
hands. Despite the wide range of relative latencies, it is the suggestion of Chun & Nakayama (2000) that the
interesting to note the predominance of latencies close to spatial information cannot be imprecise, but must be able
zero, suggesting a preference for simultaneously initiated, to support high precision movements.
synergistic movements. Land et al (1999) also measured
eye-hand latencies in their tea-making task. They also
observed a number of long lead times for the hand, Acknowledgments
although their distribution was strongly biased toward This research was supported by National Institutes of
positive values, where the eye leads the hand. Health Grants EY-05729 and RR-09283 Thanks to Chris
The frequent looks to objects, a few seconds prior to Chizk for assistance with the experiments. Commercial
reaching for them, are suggestive of movement planning relationships: None.
in a spatial coordinate frame. These fixations on objects
that were to be picked up shortly afterwards may indicate
that the subject is looking to the object to acquire its References
spatial location for guiding the next movement. As
described above, similar “look-ahead” fixations were Abrams, R., Meyer, D., & Kornblum, S. (1990). Eye-hand
observed by Pelz et al (2001) in a hand-washing context. coordination: Oculomotor control in rapid aimed
The high frequency of these look-ahead fixations in the limb movements. Journal Experimental Psychology:
present task, as well as in hand-washing and tea-making Human Perception and Performance, 15, 248-267.
Land et al (1999), suggest that this is a ubiquitous aspect [PubMed]
of natural behavior. Pelz et al interpreted these look- Ballard, D, Hayhoe, M., & Pelz, J. (1995). Memory
ahead fixations in terms of their perceptual role, representations in natural tasks. Cognitive
suggesting that they provide continuity of perceptual Neuroscience, 7, 66-80.
experience. It also seems likely that fixating the location Batista, A., Buneo, C., Snyder, L., & Andersen, R.
of a future target facilitates the programing of the saccade, (1999). Reach plans in eye-centered coordinates.
and perhaps initiates programming the reach (McPeek & Science, 285, 257-260. [PubMed]
Nakayama, 1999; Chun & Nakayama, 2000; Zelinsky et
al, 1997). It is known that accurate saccades can be made Becker, W., & Fuchs, A. F. (1969). Further properties of
on the basis of memory for stimulus location when the the human saccadic system: Eye movements and
original stimulus is no longer present (eg Miller, 1980; correction saccades with and without visual fixation
Gnadt et al, 1991; Hayhoe et al, 1992; Colby, 1998). points. Vision Research, 9, 1248-1258. [PubMed]
However, in normal viewing, the target is continuously Becker, W., & Jurgens, R. (1979). An analysis of the
present in the peripheral retina, and can be located on saccadic system by means of double-step stimuli.
the basis of stimulus features, so it is not obvious that Vision Research, 19, 967-983. [PubMed]
spatial memory would be useful in target selection. Its Bichot, N.P. & Schall, J.D. (1999) Effects of similarity
usefulness becomes more apparent when the need for and history on neural mechanisms of visual
motor planning is taken into account. In the case of selection. Nature Neuroscience, 2, 549-554. [PubMed]
reaching movements, the slower velocity of the arm
relative to eye movements makes early initiation of the
arm movements particularly useful.
Hayhoe, Shrivastava, Mruczek, & Pelz 61

Chun, M. M., & Jiang, Y. (1998). Contextual cueing: Hayhoe, M., Lachter, J., & Moeller, P. (1992). Spatial
Implicit learning and memory of visual context memory and integration across saccadic eye
guides spatial attention. Cognitive Psychology, 36, 28- movements. In K. Rayner (Ed), Eye movements and
71. [PubMed] visual cognition: Scene perception and reading (pp. 130-
Chun, M., & Jiang, Y. (1999). Top-down attentional 145). New York: Springer-Verlag.
guidance based on implicit learning of visual Henderson, J. M. (1992). Visual attention and eye
covariation. Psychological Science, 10, 360-365. movement control during reading and picture
Chun, M., & Nakayama, K. (2000). On the functional viewing. In K. Rayner (Ed.), Eye movements and visual
role of implicit visual memory for the adaptive cognition (pp.261-283).Berlin: Springer.
deployment of attention across scenes. Visual Henderson, J., & Hollingworth, A. (1999). The role of
Cognition, 7, 65-82. fixation position in detecting scene changes across
Colby, C. L. (1998) Action-oriented spatial reference saccades. Psychological Science, 10, 438-443.
frames in cortex. Neuron, 20: 15-24. [PubMed] Hochberg, J. (1986). Representation of motion and space
De Graef, P., Verfaille, K., & Lamote, C. (2001). in video and cinematic displays. In K. Boff, L.
Transsaccadic coding of object position: Effects of Kauffman, & J. Thomas (Eds), Handbook of perception
saccadic status and allocentric reference frame. and human performance (Vol. 1, pp. 22.21-22.64). New
Psychologica Belgica, 41, 29-54. York: Wiley.
Epelboim, J., Steinman, R., Kowler, E., Edwards, M., Irwin, D. E. (1991). Information integration across
Pizlo, Z., Erkelens, C., & Collewijn, H. (1995) The saccadic eye movements. Cognitive Psychology, 23, 420-
function of visual search and memory in sequential 456. [PubMed]
looking tasks. Vision Research, 35, 3401-3422. Irwin, D. (1992). Memory for position and identity across
[PubMed] eye movements. Journal Experimental Psychology:
Fischer, B., & Breitmeyer, B. (1987). Mechanisms of Learning, Memory, & Cognition. 18, 307-317.
visual attention revealed by saccadic eye movements. Irwin, D. (1996) Integrating information across saccadic
Neuropsychologia, 25, 78-83. [PubMed] eye movements. Current Directions in Psychological
Fischer, B., & Ramsperger, E. (1984). Human express Science, 5, 94-100.
saccades: extremely short reaction times of goal Irwin,D.& Gordon, R. (1998) Eye movements, attention,
directed eye movements. Experimental Brain Research, and trans-saccadic memory. Visual Cognition, 5, 127-
57, 191-195. [PubMed] 155.
Folk, C., Remington, R., & Johnston, J. (1992). Irwin, D. E., Zacks, J. L., & Brown, J. S. (1990). Visual
Involuntary covert orienting is contingent on memory and the perception of a stable visual
attentional control settings. Journal Experimental environment. Perception and Psychophysics, 47, 35-46.
Psychology: Human Perception and Performance, 18, [PubMed]
1030-1044. [PubMed] Irwin, D., Colcombe, A., Kramer, A., & Hahn, S., (2000)
Freedman, E., Stanford, T., & Sparks, D. (1996) Attentional and oculomotor capture by onset,
Combined eye-head gaze shifts produced by electrical luminance, and color singletons. Vision Research, 40,
stimulation of the superior colliculus in rhesus 1443-1458. [PubMed]
monkeys. J. Neurophysiol. 76, 927-952. [PubMed]. Itti, L. & Koch, C. (2000) A saliency-based search
Gibson, B., & Jiang, Y. (1998). Surprise! An unexpected mechanism for overt and covert shifts of visual
color singleton does not capture attention in visual attention. Vision Research, 40, 1489-1506. [PubMed]
search. Psychological Science, 9, 176-182. Jiang, Y., Olson, I. R., & Chun, M. M. (2000)
Gnadt, J., Bracewell, R., & Andersen, R. (1991) Organization of visual short-term memory. Journal
Sensorimotor transformation during eye movements Experimental Psychology: Learning, Memory, and
to remembered visual targets. Vision Research, 31, Cognition. 26, 683-702. [PubMed]
693-715. [PubMed] Johansson, R., Westling, G., Backstrom, A., Flanagan, J.
Hayhoe, M. M. (2000). Vision using routines: a R. (2001) Eye-hand coordination in object
functional account of vision. Visual Cognition, 7, 43- manipulation. J. Neuroscience, 21, 6917-6932.
64. [PubMed]
Hayhoe, M., Bensinger, D., & Ballard (1998). Task
constraints in visual working memory. Vision
Research, 38, 125-137. [PubMed]
Hayhoe, Shrivastava, Mruczek, & Pelz 62

Kowler, E. (1991) The role of visual and cognitive Miller, J. (1980) The information used by the perceptual
processes in the control of eye movement. In E. and oculomotor systems regarding the amplitude of
Kowler (Ed.), Eye movements and their role in visual and saccadic and pursuit eye movements. Vision Research,
cognitive processes (Reviews of Oculomotor Research, 20, 59-68. [PubMed]
Vol. 4, pp. 1-70). Amsterdam: Elsevier. Munoz, D. & Wurtz, R. (1995) Saccade-related activity in
Land, M. F., & Lee, D. N. (1994). Where we look when monkey superior colliculus: II. Spread of activity
we steer. Nature, 369, 742-744. [PubMed] during saccades. Journal of Neurophysiology, 73, 2334-
Land, M., & Furneaux, S. (1997) The knowledge base of 2348. [PubMed]
the oculomotor system. Philosophical Transactions, O'Regan, J. K. (1992). Solving the “real” mysteries of
Royal Society of London, Series B, 352, 1231-1239. visual perception: The world as an outside memory.
[PubMed] Canadian Journal Psychology, 46, 461-488. [PubMed]
Land, M. F. & McLeod, P. (2000) From eye movements O'Regan, J. K., & Levy-Schoen, A. (1983). Integrating
to actions: How batsmen hit the ball. Nature visual information from successive fixations: Does
Neuroscience, 3, 1340-1345. [PubMed] trans-saccadic fusion exist? Vision Research, 23, 765-
Land, M., Mennie, N., & Rusted, J. (1999). Eye 769. [PubMed]
movements and the roles of vision in activities of O'Regan, J. K., Rensink, R. A., & Clark, J. J. (1999).
daily living: making a cup of tea. Perception, 28, 1311- Change-blindness as a result of “mudsplashes.”
1328. [PubMed] Nature, 398, 34. [PubMed]
Land, M. (1996) The time it takes to process visual O'Regan, J. K., Deubel, H., Clark, J., & Rensink, R. A.
information while steering a vehicle [Abstract]. (2000). Picture changes during blinks: Looking
Investigative Ophthalmology & Visual Science, 37, S525. without seeing and seeing without looking. Visual
Levin, D., & Simons, D. (1997). Failure to detect changes Cognition, 7, 191-211.
to attended objects in motion pictures. Psychonomic Pelz, J. B., Canosa, R., Babcock, J., Kucharczyk, D., Silver,
Bulletin & Review, 4, 501-506. A., & Konno, D. (2000). Portable Eyetracking: A
Mack, A., & Rock, I. (1996). Inattentional blindness. Study of Natural Eye Movements, Proceedings SPIE
Cambridge, MA: MIT Press. Volume 3959, Human vision and electronic imaging V
(pp. 566-582). Bellingham, WA: SPIE. [Abstract]
McConkie, G., & Currie, C. (1996). Visual stability
across saccades while viewing complex pictures. Pelz, J. B., & Canosa, R., (2001). Oculomotor Behavior
Journal of Experimental Psychology: Human Perception and perceptual strategies in complex tasks. Vision
and Performance, 22, 563-581. [PubMed] Research, 41, 3587-3596. [PubMed]
McPeek, R., & Keller, E. (2001a). Short-term priming, Pollatsek, A., & Rayner, K. (1992). In K. Rayner (Ed.),
concurrent processing, and saccade curvature during Eye movements and visual cognition: Scene perception and
a target selection task in the monkey. Vision Research, reading (pp. 166-191). New York: Springer-Verlag.
41, 785-800. [PubMed] Pylyshyn, Z. (1989). The role of location indices in spatial
McPeek, R.M., & Keller, E.L. (2002) Superior colliculus perception: A sketch of the FINST spatial-index
activity related to concurrent processing of saccade model. Cognition, 32, 65-97. [PubMed]
goals in a visual search task. Journal of Neurophysiology. Rayner, K., & Pollatsek, A. (1983) Is visual information
87, 1805-1815. [PubMed] integrated across saccades? Perception & Psychophysics,
McPeek, R., Maljkovic, V., & Nakayama, K. (1999). 34, 39-48. [PubMed]
Saccades require focal attention and are facilitated Rensink, R. A., O'Regan, J. K., & Clark, J. J. (1997). To
by a short-term memory system. Vision Research, 39, see or not to see: The need for attention to perceive
1555-1565. [PubMed] changes in scenes. Journal Psychological Science,. 8,
McPeek, R., Skavenski, A., & Nakayama, K. (2000). 368-373.
Concurrent processing of saccades in visual search. Simons, D. (1996). In sight, out of mind: When object
Vision Research, 40, 2499-2516. [PubMed] representations fail. Psychological Science, 7, 301-305.
Maljkovic, V., & Nakayama, K. (1994) Priming of pop- Simons, D. J. (2000). Change blindness and visual
out: I Role of features Memory & Cognition, 22, 657- memory [Special issue]. Visual Cognition. 7,
672. [PubMed] Psychology Press, Hove, UK.
Melcher, D., & Kowler, E. (2001) Visual scene memory Simons, D., & Levin, D. (1997). Change blindness.
and the guidance of saccadic eye movements. Vision Trends in Cognitive Science, 1, 261-267.
Research, 41, 3597-3611. [PubMed]
Hayhoe, Shrivastava, Mruczek, & Pelz 63

Simons, D., & Levin, D. (1998). Failure to detect changes Wallis, G., & Büthoff, H. (2000). What’s scene and not
to people in real-world interactions. Psychonomic seen: Influences of movement and task upon what
Bulletin & Review, 5, 644-649. we see. Visual Cognition, 7, 175-190.
Theeuwes, J., Kramer, A., Hahn, S., & Irwin, D. (1998). Ullman, S. (1984). Visual routines. Cognition, 18, 97-157.
Our eye do not always go where we want them to go: [PubMed]
capture of the eyes by new objects. Psychological Viviani, P. (1991). In E. Kowler (Ed.), Eye movements and
Science, 9, 379-385. their role in visual and cognitive processes (Reviews of
Theeuwes, J., Kramer, A., Hahn, S., Irwin, D., & oculomotor Research, Vol. 4, pp. 1-70). Amsterdam:
Zelinsky, G. (1999). Influence of attentional capture Elsevier.
on oculomotor control. J. Experimental Psychology: Zelinsky, G, Rao, R. Hayhoe, M. & Ballard, D. (1997)
Human Perception & Performance, 25, 1595-1608. Eye movements reveal the spatiotemporal dynamics
[PubMed] of visual search. Psychological Science, 8, 448-453.
Verfaille, K., De Graef, P., Germeys, F., Gysen, V., & Zingale, C. M., & Kowler, E. (1987). Planning sequences
Van Eccelpoel, C. (2001). Selective transsaccadic of saccades. Vision Research, 27, 1327-1341.
coding of object and event-diagnostic information. [PubMed]
Psychologica Belgica, 41, 89-114.