See discussions, stats, and author profiles for this publication at: https://www.researchgate.

net/publication/3208983

Recent advances in augmented reality. IEEE Comput Graphics Appl

Article in IEEE Computer Graphics and Applications · December 2001

DOI: 10.1109/38.963459 · Source: IEEE Xplore

CITATIONS READS
3,342 22,214

6 authors, including:

Yohan Baillot Reinhold Behringer

42 PUBLICATIONS 4,829 CITATIONS

Knorr-Bremse Group
84 PUBLICATIONS 4,735 CITATIONS
SEE PROFILE
SEE PROFILE

Simon Julier Blair Macintyre

University College London Georgia Institute of Technology
231 PUBLICATIONS 35,079 CITATIONS 150 PUBLICATIONS 12,211 CITATIONS

SEE PROFILE SEE PROFILE

All content following this page was uploaded by Reinhold Behringer on 06 November 2013.

The user has requested enhancement of the downloaded file.

Survey

Ronald Azuma
HRL Laboratories

Yohan Baillot
NRL Virtual Reality Lab/ITT Advanced Engineering

Reinhold Behringer
Rockwell Scientific

Steven Feiner
Columbia University

Recent Advances in Simon Julier

NRL Virtual Reality Lab/ITT Advanced Engineering

Augmented Reality Blair MacIntyre

Georgia Institute of Technology

W hat is augmented reality? An AR system

supplements the real world with virtual
(computer-generated) objects that appear to coexist in
The beginnings of AR, as we define it, date back to
Sutherland’s work in the 1960s, which used a see-
through HMD to present 3D graphics.2 However, only
the same space as the real world. While many researchers over the past decade has there been enough work to refer
broaden the definition of AR beyond this vision, we define to AR as a research field. In 1997, Azuma published a sur-
an AR system to have the following properties: vey3 that defined the field, described many problems,
and summarized the developments up to that point.
■ combines real and virtual objects in a real environment; Since then, AR’s growth and progress have been remark-
■ runs interactively, and in real time; and able. In the late 1990s, several conferences on AR began,
■ registers (aligns) real and virtual objects with each including the International Workshop and Symposium
other. on Augmented Reality, the International Symposium on
Mixed Reality, and the Designing Augmented Reality
Note that we don’t restrict this Environments workshop. Some well-funded organiza-
definition of AR to particular dis- tions formed that focused on AR, notably the Mixed Real-
In 1997, Azuma published a play technologies, such as a head- ity Systems Lab in Japan and the Arvika consortium in
mounted display (HMD). Nor do we Germany. A software toolkit (the ARToolkit) for rapidly
survey on augmented reality limit it to our sense of sight. AR can building AR applications is now freely available at http://
potentially apply to all senses, inclu- www.hitl.washington.edu/research/shared_space/. Be-
(AR). Here we provide a ding hearing, touch, and smell. Cer- cause of the wealth of new developments, this field needs
tain AR applications also require an updated survey to guide and encourage further re-
complement to that survey, removing real objects from the per- search in this exciting area.
ceived environment, in addition to Our goal here is to complement, rather than replace,
denoting the rapid adding virtual objects. For example, the original survey by presenting representative examples
an AR visualization of a building that of the new advances. We refer you to the original survey3
technological advancements stood at a certain location might for descriptions of potential applications (such as medical
remove the building that exists there visualization, maintenance and repair of complex equip-
since then. today. Some researchers call the task ment, annotation, and path planning); summaries of AR
of removing real objects mediated or system characteristics (such as the advantages and dis-
diminished reality, but we consider it a subset of AR. advantages of optical and video approaches to blending
Milgram1 defined a continuum of real-to-virtual envi- virtual and real, problems in display focus and contrast,
ronments, in which AR is one part of the general area of and system portability); and an introduction to the cru-
mixed reality (Figure 1). In both augmented virtuality, in cial problem of registration, including sources of regis-
which real objects are added to virtual ones, and virtu- tration error and error-reduction strategies.
al environments (or virtual reality), the surrounding
environment is virtual, while in AR the surrounding Enabling technologies
environment is real. We focus on AR and don’t cover One category for new developments is enabling tech-
augmented virtuality or virtual environments. nologies. Enabling technologies are advances in the basic

Mixed reality
1 Milgram’s reality–virtuality con-
tinuum. (Adapted from Milgram
and Kishino.1) Real Augmented Augmented Virtual
environment reality virtuality environment

34 November/December 2001 0272-1716/01/$10.00 © 2001 IEEE

technologies needed to build com- Camera 2 Concepts of
pelling AR environments. Examples Display Display optical see-
of these technologies include dis- Combiner through (left)
Opaque mirror
plays, tracking, registration, and (semi-transparent and video see-
calibration. mirror) through dis-
plays (right).
Displays
We can classify displays for viewing the merged virtu-
al and real environments into the following categories:
head worn, handheld, and projective.

Head-worn displays (HWD). Users mount this

type of display on their heads, providing imagery in 3 Minolta
front of their eyes. Two types of HWDs exist: optical see- eyeglass display

Courtesy of Hiroaki Ueda, Minolta

through and video see-through (Figure 2). The latter with holograph-
uses video capture from head-worn video cameras as a ic element.
background for the AR overlay, shown on an opaque dis-
play, whereas the optical see-through method provides
the AR overlay through a transparent display.
Established electronics and optical companies (for
example, Sony and Olympus) have manufactured color,
liquid crystal display (LCD)-based consumer head-worn
displays intended for watching videos and playing video
games. While these systems have relatively low resolu-
tion (180,000 to 240,000 pixels), small fields of view objects to be augmented. In the simplest case, the inten-
(approximately 30 degrees horizontal), and don’t sup- tion is for the augmentations to be coplanar with the
port stereo, they’re relatively lightweight (under 120 surface onto which they project and to project them
grams) and offer an inexpensive option for video see- from a single room-mounted projector, with no need for
through research. Sony introduced 800 × 600 resolution, special eyewear. Generalizing on the concept of a multi-
stereo, color optical see-through displays (later discon- walled Cave automatic virtual environment (CAVE),
tinued) that have been used extensively in AR research. Raskar and colleagues8 show how multiple overlapping
A different approach is the virtual retinal display,4 projectors can cover large irregular surfaces using an
which forms images directly on the retina. These dis- automated calibration procedure that takes into account
plays, which MicroVision is developing commercially, surface geometry and image overlap.
literally draw on the retina with low-power lasers whose Another approach for projective AR relies on head-
modulated beams are scanned by microelectro- worn projectors, whose images are projected along the
mechanical mirror assemblies that sweep the beam hor- viewer’s line of sight at objects in the world. The target
izontally and vertically. Potential advantages include objects are coated with a retroreflective material that
high brightness and contrast, low power consumption, reflects light back along the angle of incidence. Multi-
and large depth of field. ple users can see different images on the same target
Ideally, head-worn AR displays would be no larger projected by their own head-worn systems, since the
than a pair of sunglasses. Several companies are devel- projected images can’t be seen except along the line of
oping displays that embed display optics within con- projection. By using relatively low output projectors,
ventional eyeglasses. MicroOptical produced a family of nonretroreflective real objects can obscure virtual
eyeglass displays in which two right-angle prisms are objects. However, projectors worn on the head can be
embedded in a regular prescription eyeglass lens and heavy. Figure 4 (next page) shows a new, relatively light-
reflect the image of a small color display, mounted facing weight prototype that weighs less than 700 grams.9
forward on an eyeglass temple piece.5 The intention of One interesting application of projection systems is
the Minolta prototype forgettable display is to be light in mediated reality. Coating a haptic input device with
and inconspicuous enough that users forget that they’re retroreflective material and projecting a model of the
wearing it.6 Others see only a transparent lens, with no scene without the device camouflages the device by
indication that the display is on, and the display adds less making it appear semitransparent10 (Figure 5, next
than 6 grams to the weight of the eyeglasses (Figure 3). page). Other applications using projection displays
include a remote laser pointer control11 and a simula-
Handheld displays. Some AR systems use hand- tion of a virtual optical bench.12
held, flat-panel LCD displays that use an attached cam-
era to provide video see-through-based augmentations.7 Problem areas in AR displays. See-through dis-
The handheld display acts as a window or a magnifying plays don’t have sufficient brightness, resolution, field of
glass that shows the real objects with an AR overlay. view, and contrast to seamlessly blend a wide range of
real and virtual imagery. Furthermore size, weight, and
Projection displays. In this approach, the desired cost are still problems. However, there have been
virtual information is projected directly on the physical advances on specific problems. First, in conventional

IEEE Computer Graphics and Applications 35

Survey

optical see-through displays, virtual objects can’t com- Second, most video see-through displays have a par-
pletely occlude real ones. One experimental display allax error, caused by the cameras being mounted away
addresses this by interposing an LCD panel between the from the true eye location. If you see the real world
optical combiner and the real world, blocking the real- through cameras mounted on top of your head, your
world view at selected pixels13 (Figure 6). view is significantly different from what you’re normal-
ly used to, making it difficult to adapt to the display.14
The MR Systems Lab developed a relatively lightweight
4 Experimental (340 grams) video graphics array (VGA) resolution video
head-worn see-through display, with a 51-degree horizontal field of
projective dis- view, in which the imaging system and display system
play using optical axes are aligned for each eye.15 Finally, most dis-
lightweight plays have fixed eye accommodation (focusing the eyes
optics. (Cour- at a particular distance). Some prototype video and opti-
tesy of Jannick cal see-through displays can selectively set accommo-
Rolland, Univer- dation to correspond to vergence by moving the display
sity of Central screen or a lens through which it’s imaged. One version
Florida, and can cover a range of .25 meters to infinity in .3 seconds.16
Frank Biocca,
Michigan State New tracking sensors and approaches
University.) Accurately tracking the user’s viewing orientation and
position is crucial for AR registration. Rolland et al.17
give a recent overview of tracking systems. For prepared,
indoor environments, several systems demonstrate
excellent registration. Typically such systems employ
hybrid-tracking techniques (such as magnetic and video
sensors) to exploit strengths and compensate weak-
5 Projection nesses of individual tracking technologies. A system
display used to combining accelerometers and video tracking demon-
camouflage a strates accurate registration even during rapid head
haptic input motion.18 The Single Constraint at a Time (Scaat) algo-
device. The rithm also improved the tracking performance. Scaat
haptic input incorporates individual measurements at the exact time
device normally they occur, resulting in faster update rates, more accu-
doesn’t reflect rate solutions, and autocalibrated parameters.19 Two
projected new scalable tracking systems, InterSense’s Constella-
graphics (top). tion20 and 3rdTech’s HiBall,21 can cover the large indoor
The haptic input environments needed by some AR applications.
device coated Visual tracking generally relies on modifying the envi-
with retro- ronment with fiducial markers placed in the environment
Courtesy Tachi Lab, University of Tokyo

reflective mate- at known locations. The markers can vary in size to

rial appears improve tracking range,22 and the computer-vision tech-
transparent niques that track by using fiducials can update at 30 Hz.23
(bottom). While some recent AR systems demonstrate robust and
compelling registration in prepared, indoor environ-
ments, there’s still much to do in tracking and calibration.
Ongoing research includes sensing the entire environ-
ment, operating in unprepared environments, minimiz-
ing latency, and reducing calibration requirements.

Environment sensing. Effective AR requires

6 Images photographed through knowledge of the user’s location and the position of all
optical see-through display sup- other objects of interest in the environment. For example,
porting occlusion. (a) Transparent it needs a depth map of the real scene to support occlu-
overlay. (b) Transparent overlay sion when rendering. One system demonstrates real-
rendered taking into account real- time depth-map extraction using several cameras, where
world depth map. (c) LCD panel the depth map is reprojected to a new viewing location.24
opacifies areas to be occluded. Kanade’s 3D dome drives this concept to its extreme with
(d) Opaque overlay created by 49 cameras that capture a scene for later virtual replay.25
opacified pixels. (Courtesy of
Kiyoshi Kiyokawa, Communications Outdoor, unprepared environments. In out-
Research Lab.) door and mobile AR applications, it generally isn’t prac-
tical to cover the environment with markers. A hybrid

36 November/December 2001
compass/gyroscope tracker provides motion-stabilized
orientation measurements at several outdoor locations26
(Figure 7). With the addition of video tracking (not in
real time), the system produces nearly pixel-accurate
results on known landmark features. 27,28 The Townwear
system29 uses custom packaged fiber-optic gyroscopes 7 Motion-
for high accuracy and low drift rates.30 Either the Glob- stabilized labels
al Positioning System (GPS) or dead reckoning tech- annotate the
niques usually track the real-time position outdoors, Phillips Tower,
although both have significant limitations (for example, as seen from
GPS requires a clear view of the sky). two different
Ultimately, tracking in unprepared environments viewpoints.

Courtesy HRL Laboratories

may rely heavily on tracking visible natural features
(such as objects that already exist in the environment,
without modification).31 If a database of the environ-
ment is available, we can base tracking on the visible
horizon silhouette32 or rendered predicted views of the
surrounding buildings, which we then match against
the video.33 Alternately, given a limited set of known
features, a tracking system can automatically select
and measure new natural features in the environ-
ment.34 There’s a significant amount of research on
recovering camera motion given a video sequence with
no tracking information. Today, those approaches
don’t run in real time and are best suited for special
effects and postproduction. However, these algorithms 8 RV-Border
can potentially apply to AR if they can run in real time Guards, an AR
and operate causally (without using knowledge of game.
Courtesy MR Systems Lab

what occurs in the future). In one such example,35 a

tracking system employs planar features, indicated by
the user, to track the user’s change in orientation and
position.

Low latency. System delays are often the largest

source of registration errors. Predicting motion is one
way to reduce the effects of delays. Researchers have
attempted to model motion more accurately36 and switch tion to automatically measure and compensate for
between multiple models.37 We can schedule system changing calibration parameters.19,44
latency to reduce errors38 or minimize them altogether
through careful system design.39 Shifting a prerendered Interfaces and visualization
image at the last instant can effectively compensate for In the last five years, a growing number of researchers
pan-tilt motions.40 Through image warping, such cor- have considered how users will interact with AR appli-
rections can potentially compensate for delays in 6D cations and how to effectively present information on
motion (both translation and rotation).41 AR displays.

Calibration and autocalibration User interface and interaction

AR systems generally require extensive calibration to Until recently, most AR prototypes concentrated on
produce accurate registration. Measurements may displaying information that was registered with the
include camera parameters, field of view, sensor offsets, world and didn’t significantly concern themselves with
object locations, distortions, and so forth. The AR com- how potential users would interact with these systems.
munity uses well-established basic principles of camera Prototypes that supported interaction often based their
calibration and developed many manual AR calibration interfaces on desktop metaphors (for example, they pre-
techniques. One way to avoid a calibration step is to sented on-screen menus or required users to type on key-
develop calibration-free renderers. Since Kutulakos and boards) or adapted designs from virtual environments
Vallino introduced their approach of calibration-free AR research (such as using gesture recognition or tracking
based on a weak perspective projection model,42 Seo 6D pointers). In certain applications, such techniques
and Hong have extended it to cover perspective projec- are appropriate. In the RV-Border Guards game,45 for
tion, supporting traditional illumination techniques.43 example, users combat virtual monsters by using ges-
Another example obtains the camera focal length35 tures to control their weapons and shields (Figure 8).
without an explicit metric calibration step. The other However, it’s difficult to interact with purely virtual
way to reduce calibration requirements is autocalibra- information. There are two main trends in AR interac-
tion. Such algorithms use redundant sensor informa- tion research:

IEEE Computer Graphics and Applications 37

Survey

Courtesy Jun Rekimoto, Sony Computer Science Laboratories

9 Heteroge-
neous AR sys-
tems using
projected (top)
and see-
through hand-
held (bottom)
displays.

10 Heteroge-
neous displays
in Emmie, com-
bining head-
worn,
projected, and
private flat-
screen displays.
11 The Studierstube (top) and Magic Book (bottom)
collaborative AR systems, with two users wearing see-
through HMDs. (Courtesy Dieter Schmalstieg, Vienna
Courtesy A. Butz, T. Höllerer, S. Feiner, B. MacIntyre, C. Beshers, Columbia Univ.
University of Technology, and Mark Billinghurst, Human
Interface Technologies Lab.)
■ using heterogeneous devices to leverage the advan-
tages of different displays and
■ integrating with the physical world through tangible
interfaces.

Different devices best suit different interaction tech-

niques, so using more than one device lets an appropri-
ate device be used for each interaction task. For
example, a handheld tablet interacts well with a text
document.46 In the Augmented Surfaces system47 (Fig- Courtesy Hirokazu Kato
ure 9), users manipulate data through a variety of real
and virtual mechanisms and can interact with data
through projective and handheld displays. Similarly, the
Emmie system48 mixes several display and device types
and lets information be moved between devices to
improve interaction (Figure 10). AR systems can also 12 User wields a real paddle to pick up, move, drop,
simulate the benefits of multiple devices. In the Studier- and destroy models.
stube system, the Personal Interaction Panel (PIP)49 is a
tracked blank physical board the user holds, upon which
virtual controls or parts of the world are drawn (Figure real paddle to manipulate furniture models in a proto-
11). The haptic feedback from the physical panel gives type interior design application.50 Through pushing, tilt-
similar benefits as a handheld display in this case. ing, swatting, and other motions, the user can select
Tangible interfaces support direct interaction with pieces of furniture, drop them into a room, push them
the physical world by emphasizing the use of real, phys- to the desired locations, and remove them from the
ical objects and tools. In one example, the user wields a room (Figure 12). In the AR2 Hockey system, two users

38 November/December 2001
 14 Virtual and
real occlusions.
The brown cow
and tree are
virtual; the rest
is real.

Courtesy INRIA

occluded by real objects is to use a probabilistic function

that gradually fades out the hidden virtual object along
the edges of the occluded region, making registration
errors less objectionable.54
Courtesy Naval Research Lab

Data density. If we augment the real world with

large amounts of virtual information, the display may
become cluttered and unreadable. Unlike other applica-
tions that must deal with large amounts of information,
AR applications must also manage the interaction
between the physical world and virtual information, with-
out changing the physical world. Julier et al.55 use a fil-
13 Data filtering to reduce density problems. Unfiltered tering technique based on a model of spatial interaction
view (top) and filtered view (bottom), from Julier et al.55 to reduce the amount of information displayed to a min-
imum while keeping important information in view (Fig-
ure 13). The framework takes into account the goal of the
play an air hockey game by moving a real object that user, the relevance of each object with respect to the goal,
represents the user’s paddle in the virtual world.51 and the position of the user to determine whether each
Researchers have explored other interaction possi- object should be shown. In a complementary approach,
bilities. For example, the Magic Book52 depicts live VR Bell et al.56 model the environment and track certain real
environments on the pages of a book and lets one or entities, using this knowledge to ensure that virtual infor-
more users enter one of the VR environments. When mation isn’t placed on top of important parts of the envi-
one user switches from AR into the immersive VR world ronment or other information.
depicted on the page, the other AR users see an avatar
appear in the environment on the book page (Figure Advanced rendering
11). The Magic Book takes advantage of video see- For some applications, virtual augmentations should
through AR displays, letting the users’ views of the be indistinguishable from real objects. While high-quality
world be completely blocked out when they’re in the renderings and compositions aren’t currently feasible in
VR environment. real time, researchers are studying the problem of pho-
torealistic rendering in AR and of removing real objects
Visualization problems from the environment (for example, mediated reality).
Researchers are beginning to address fundamental
problems of displaying information in AR displays. Mediated reality. The problem of removing real
objects goes beyond extracting depth information from
Error estimate visualization. In some AR sys- a scene; the system must also segment individual objects
tems, registration errors are significant and unavoid- in that environment. Lepetit discusses a semiautomat-
able. For example, the measured location of an object ic method for identifying objects and their locations in
in the environment may not be known accurately the scene through silhouettes.57 In some situations, this
enough to avoid a visible registration error. Under such technique enables the insertion of virtual objects and
conditions, one approach to rendering an object is to deletion of real objects without an explicit 3D recon-
visually display the area in screen space where the object struction of the environment (Figure 14).
could reside, based on expected tracking and measure-
ment errors.53 This guarantees that the virtual repre- Photorealistic rendering. A key requirement for
sentation always contains the real counterpart. Another improving the rendering quality of virtual objects in AR
approach when rendering virtual objects that should be applications is the ability to automatically capture the

IEEE Computer Graphics and Applications 39

Survey

rect occlusion can ameliorate some depth perception

problems.63 Consistent registration plays a crucial role
in depth perception, even to the extent that accurately
determining the eyepoint location as the eye rotates
can affect perception. An analysis of different eye-
point locations to use in rendering an image con-
cluded that the eye’s center of rotation yields the best
position accuracy, but the center of the entrance pupil
yields higher angular accuracy.66

Courtesy Nassir Navab, Siemens Corporate Research

■ Adaptation. User adaptation to AR equipment can
negatively impact performance. One study investi-
gating the effects of vertically displacing cameras
above the user’s eyes in a video see-through HMD
showed that subjects could adapt to the displacement,
but after removing the HMD, the subjects exhibited
a large overshoot in a depth-pointing task.14
■ Fatigue and eye strain. Uncomfortable AR displays
may not be suitable for long-term use. One study
found that binocular displays, where both eyes see
the same image, cause significantly more discomfort,
both in eyestrain and fatigue, than monocular or
15 Two-dimensional shop floor plans and a 3D pipe model superimposed stereo displays.65
on an industrial pipeline.
New applications
We’ve grouped the new applications into three areas:
environmental illumination and reflectance information. mobile, collaborative, and commercial applications.
Three examples of work in this area are an approach that Before discussing these further, though, we would like
uses ellipsoidal models to estimate illumination para- to briefly highlight representative advances in the more
meters,58 photometric image-based rendering,59 and traditional areas of assembly, inspection, and medical
high dynamic range illumination capturing.60 applications.
Curtis et al.67 describe the verification of an AR system
Human factors studies and perceptual problems for assembling aircraft wire bundles. Although limited
Experimental results from human factors, perceptu- by tracking and display technologies, their tests on actu-
al studies, and cognitive science61 can help guide the al assembly-line workers prove that their AR system lets
design of effective AR systems. Drascic62 discusses 18 workers create wire bundles that work as well as those
different design issues that affect AR displays. The issues built by conventional approaches. This paper also empha-
include implementation errors (such as miscalibration), sizes the need for iterative design and user feedback.
technological problems (such as vertical mismatch in In their research,68 Navab and his colleagues take
image frames of a stereo display), and fundamental lim- advantage of 2D factory floor plans and the structural
itations in the design of current HMDs (the accommo- properties of industrial pipelines to generate 3D models
dation-vergence conflict). Rolland and Fuchs offer a of the pipelines and register them with the user’s view
detailed analysis of the different human factors in con- of the factory, obviating the need for a general-purpose
nection with optical and see-through HMDs for medical tracking system (Figure 15). Similarly, they take advan-
applications.63 Finally, we need to better understand tage of the physical constraints of a C-arm x-ray machine
human factors related to the effects of long-term use of to automatically calibrate the cameras with the machine
AR systems. Some significant factors include and register the x-ray imagery with the real objects.69
Fuchs and his colleagues are continuing work on med-
■ Latency. Delay causes more registration errors than ical AR applications, refining their tracking and display
all other sources combined. For close range tasks, a techniques to support laparoscopic surgery.70 New med-
simple rule of thumb is that one millisecond of delay ical AR applications are also being explored. For exam-
causes one millimeter of error.64 More importantly, ple, Weghorst71 describes how to use AR to help treat
delay can reduce task performance. Delays as small akinesia (freezing gait), one of the common symptoms
as 10 milliseconds can make a statistically significant of Parkinson’s disease.
difference in the performance of a task to guide a ring
over a bent wire.65 Mobile applications
■ Depth perception. Accurate depth perception is a dif- With advances in tracking and increased computing
ficult registration problem. Stereoscopic displays help power, researchers are developing mobile AR systems.
with depth perception, but current display technolo- These may enable a host of new applications in naviga-
gies cause additional problems (including accom- tion, situational awareness, and geolocated information
modation-vergence conflicts, or low resolution and retrieval.
dim displays causing objects to appear further away Researchers have been investigating mobile AR
than they really are62). Rendering objects with cor- research systems operating in well-prepared indoor

40 November/December 2001
 17 Battlefield
Augmented
Reality System,
a descendent of
the Touring
Machine. (Cour-
tesy of the
Naval Research
Lab.)
16 A view through a see-through HMD shows a 3D
model of demolished building at its original location.
(Courtesy T. Höllerer, S. Feiner, J. Pavlik, Columbia Univ.)

environments for some time. NaviCam, for example,

augments the video stream collected by a handheld
video camera.7 A set of fiducials—used to find the type
of objects in view and to place the augmentation with-
out knowing the user’s absolute position—populate the
environment. The system provides simple information,
such as a list of new journals on a bookshelf. Starner et
al. are considering the applications and limitations of
AR for wearable computers.72 Using an approach simi-
lar to NaviCam, they use virtual tags for registering
graphics and consider the problems of finger tracking
(as a surrogate mouse) and facial recognition.
The “New tracking sensors and approaches” section 18 Two views
describes strategies for tracking in various outdoor of a combined
environments; here we focus on examples of outdoor augmented and
applications. virtual environ-
The first outdoor system was the Touring Machine.46 ment. (Courtesy
Developed at Columbia University, this self-contained Wayne Piekars-
system includes tracking (a compass, inclinometer, and ki, Bernard
differential GPS), a mobile computer with a 3D graphics Gunther, and
board, and a see-through HMD. The system presents the Bruce Thomas,
user with world-stabilized information about an urban University of
environment (the names of buildings and departments South
on the Columbia campus). The AR display is cross-refer- Australia.)
enced with a handheld display, which provides detailed
information. More recent versions of this system render
models of buildings that previously existed on campus,
display paths that users need to take to reach objectives,
and play documentaries of historical events that occurred
at the observed locations73 (Figure 16). The Naval
Research Lab (NRL) developed a similar system—the Bat-
tlefield Augmented Reality System (Figure 17)— to help
during military operations in urban environments.55 The
system goal is to augment the environment with dynam-
ic 3D information (such as goals or hazards) usually con-
veyed on 2D maps. Recently, the system also provides ARQuake,76 designed using the same platform, blends
tools to author the environment with new 3D informa- users in the real world with those in a purely virtual envi-
tion that other system users see in turn.74 ronment. A mobile AR user plays as a combatant in the
In the same area, Piekarski75 is developing user inter- computer game Quake, where the game runs with a vir-
action paradigms and techniques for interactive model tual model of the real environment. When GPS is unavail-
construction in a mobile AR environment. This system able, the system switches to visual tracking derived from
also lets an outdoor user see objects (such as an aircraft) the ARToolkit. The recently started Archeoguide project
that only exist in a virtual military simulator (Figure 18). is developing a wearable AR system for providing tourists

IEEE Computer Graphics and Applications 41

Survey

Surfaces47), users are unencum-

bered, can see each other’s eyes, and
will see the same augmentations.
However, this approach is limited to
adding virtual information to the
projected surfaces.
Tracked, see-through displays can
alleviate this limitation by letting 3D
graphics be placed anywhere in the
environment. Examples of collabo-
rative AR systems using see-through
displays include both those that use
19 AR in sports see-through handheld displays
broadcasting. (such as Transvision82) and see-
The annotations through head-worn displays (such
on the race cars as Emmie,48 Magic Book,52 and
and the yellow Studierstube83). An example of mul-
first down line tiple-system collaboration is the
are inserted integration of mobile warfighters
into the broad- (engaged with virtual enemies via
cast in real time. AR displays) collaborating with
(Courtesy of units in a VR military simulation.75,84
NASCAR and A significant problem with collo-
Sportvision, top cated, collaborative AR systems is
and bottom, ensuring that the users can establish
respectively.) a shared understanding of the virtu-
al space, analogous to their under-
standing of the physical space.
Because the graphics are overlaid
independently on each user’s view of
the world, it’s difficult to ensure that
each user clearly understands what
other users are pointing or referring
to. In Studierstube, the designers
attempt to overcome this problem by
rendering virtual representations of
the physical pointers, which are vis-
ible to all participants (Figure 11).
Numerous system designers have
suggested the benefits of adaptive
with information about a historic site in Olympia, interfaces tailored to each user’s interests and skills. The
Greece.77 Rekimoto discusses creating content for wear- ability to personalize the information presented to each
able AR systems.78 user also lets AR systems present private information to
Mobile AR systems must be worn, which challenges individuals without fearing that others will see it. In the
system designers to minimize weight and bulk. With Emmie system, Butz and his colleagues discuss the
current technology, one approach is to move some of notion of privacy management in collaborative AR sys-
the computation load to remote servers, reducing the tems and present an approach to managing the visibili-
equipment the user must wear.79,80 ty of information using the familiar metaphors of lamps
and mirrors.48
Collaborative applications Another form of collaborative AR is in entertainment
Many AR applications can benefit from having multi- applications. Researchers have demonstrated a number
ple people simultaneously view, discuss, and interact with of AR games, including AR air hockey,51 collaborative
the virtual 3D models. As Billinghurst and Kato discuss,81 combat against virtual enemies,45 and an AR-enhanced
AR addresses two major issues with collaboration: pool game.85

■ seamless integration with existing tools and practices Commercial applications

and Recently, AR has been used for real-time augmenta-
■ enhancing practice by supporting remote and collo- tion of broadcast video, primarily to enhance sporting
cated activities that would otherwise be impossible. events and to insert or replace advertisements in a scene.
An early example is the FoxTrax system, which high-
By using projectors to augment the surfaces in a col- lights the location of a hard-to-see hockey puck as it
laborative environment (such as Rekimoto’s Augmented moves rapidly across the ice.86

42 November/December 2001
 Figure 19 shows two current examples of AR in sports.
In both systems, the environments are carefully modeled
ahead of time, and the cameras are calibrated and pre-
cisely tracked. For some applications, augmentations are
added solely through real-time video tracking. Delaying
the video broadcast by a few video frames eliminates the
registration problems caused by system latency. Fur- 20 Virtual
thermore, the predictable environment (uniformed play- advertising. The
ers on a green, white, and brown field) lets the system Pacific Bell and
use custom chroma-keying techniques to draw the yel- Pennsylvania
low line only on the field rather than over the players. Lottery ads are
With similar approaches, advertisers can embellish AR augmenta-
broadcast video with virtual ads and product place- tions.
ments (Figure 20).

Future work
Apart from the few commercial examples described in
the last section, the state of the art in AR today is com-
parable to the early years of VR—many research sys-
tems have been demonstrated but few have matured
beyond lab-based prototypes.87 We’ve grouped the
major obstacles limiting the wider use of AR into three
Courtesy Pacific Video Image

themes: technological limitations, user interface limi-

tations, and social acceptance issues.

Technological limitations
Although we’ve seen much progress in the basic
enabling technologies, they still primarily prevent the
deployment of many AR applications. Displays, track-
ers, and AR systems in general need to become more
accurate, lighter, cheaper, and less power consuming.
By describing problems from our common experi- research in AR can reduce these difficulties through
ences in building outdoor AR systems, we hope to improved tracking in unprepared environments and cal-
impart a sense of the many areas that still need ibration-free or autocalibration approaches to minimize
improvement. Displays such as the Sony Glasstron are set-up requirements.
intended for indoor consumer use and aren’t ideal for
outdoor use. The display isn’t very bright and com- User interface limitations
pletely washes out in bright sunlight. The image has a We need a better understanding of how to display data
fixed focus to appear several feet away from the user, to a user and how the user should interact with the data.
which is often closer than the outdoor landmarks. The Most existing research concentrates on low-level per-
equipment isn’t nearly as portable as desired. Since the ceptual issues, such as properly perceiving depth or how
user must wear the PC, sensors, display, batteries, and latency affects manipulation tasks. However, AR also
everything else required, the end result is a cumber- introduces many high-level tasks, such as the need to
some and heavy backpack. identify what information should be provided, what’s the
Laptops today have only one CPU, limiting the appropriate representation for that data, and how the
amount of visual and hybrid tracking that we can do. user should make queries and reports. For example, a user
Operating systems aimed at the consumer market aren’t might want to walk down a street, look in a shop window,
built to support real-time computing, but specialized and query the inventory of that shop. To date, few have
real-time operating systems don’t have the drivers to studied such issues. However, we expect significant
support the sensors and graphics in modern hardware. growth in this area because research AR systems with suf-
Tracking in unprepared environments remains an ficient capabilities are now more commonly available.
enormous challenge. Outdoor demonstrations today For example, recent work suggests that the creation and
have shown good tracking only with significant restric- presentation of narrative performances and structures
tions in operating range, often with sensor suites that may lead to more realistic and richer AR experiences.88
are too bulky and expensive for practical use. Today’s
systems generally require extensive calibration proce- Social acceptance
dures that an end user would find unacceptably com- The final challenge is social acceptance. Given a sys-
plicated. Many connectors such as universal serial bus tem with ideal hardware and an intuitive interface, how
(USB) connectors aren’t rugged enough for outdoor can AR become an accepted part of a user’s everyday
operation and are prone to breaking. life, just like a mobile phone or a personal digital assis-
While we expect some improvements to naturally tant (PDA)? Through films and television, many people
occur from other fields such as wearable computing, are familiar with images of simulated AR. However, per-

IEEE Computer Graphics and Applications 43

Survey

suading a user to wear a system means addressing a Special Interest Group on Computer–Human Interaction
number of issues. These range from fashion (will users (SIGCHI 98), ACM Press, New York, 1998, pp. 542-549.
wear a system if they feel it detracts from their appear- 13. K. Kiyokawa, Y. Kurata, and H. Ohno, “An Optical See-
ance?) to privacy concerns (we can also use the tracking through Display for Mutual Occlusion of Real and Virtu-
required for displaying information for monitoring and al Environments,” Proc. Int’l Symp. Augmented Reality
recording). To date, little attention has been placed on 2000 (ISAR 00), IEEE CS Press, Los Alamitos, Calif., 2000,
these fundamental issues. However, these must be pp. 60-67.
addressed before AR becomes widely accepted. ■ 14. F.A. Biocca and J.P. Rolland, “Virtual Eyes Can Rearrange
Your Body: Adaptation to Visual Displacement in See-
Acknowledgements Through, Head-Mounted Displays,” Presence: Teleopera-
Part of this work was funded by the US Office of Naval tors and Virtual Environments. vol. 7, no. 3, June 1998, pp.
Research contracts N00014-99-3-0018, N00014-00-1- 262-277.
0361, N00014-99-1-0249, N00014-99-1-0394, N00014- 15. A. Takagi et al., “Development of a Stereo Video See-
99-0683, and the US National Science Foundation Grant through HMD for AR Systems,” Proc. Int’l Symp. Augment-
IIS-00-82961. ed Reality 2000 (ISAR 00), IEEE CS Press, Los Alamitos,
Calif., 2000, pp. 68-77.
16. T. Sugihara and T. Miyasato, “A Lightweight 3D HMD with
Accommodative Compensation,” Proc. 29th Soc. Informa-
References tion Display (SID 98), San Jose, Calif., 1998, pp. 927-930.
1. P. Milgram and F. Kishino, “A Taxonomy of Mixed Reality 17. J.P. Rolland, L.D. Davis, and Y. Baillot, “A Survey of Track-
Visual Displays,” IEICE Trans. Information Systems, vol. ing Technologies for Virtual Environments,” Fundamentals
E77-D, no. 12, 1994, pp. 1321-1329. of Wearable Computers and Augmented Reality, W. Barfield
2. I. Sutherland, “A Head-Mounted Three-Dimensional Dis- and T. Caudell, eds., Lawrence Erlbaum, Mahwah, N.J.,
play,” Fall Joint Computer Conf., Am. Federation of Infor- 2001, pp. 67-112.
mation Processing Soc. (AFIPS) Conf. Proc. 33, Thompson 18. Y. Yokokohji, Y. Sugawara, and T. Yoshikawa, “Accurate
Books, Washington, D.C., 1968, pp. 757-764. Image Overlay on Video See-through HMDs Using Vision
3. R. Azuma, “A Survey of Augmented Reality,” Presence: Tele- and Accelerometers,” Proc. IEEE Virtual Reality 2000, IEEE
operators and Virtual Environments. vol. 6, no. 4, Aug. 1997, CS Press, Los Alamitos, Calif., 2000, pp. 247-254.
pp. 355-385. 19. G. Welch and G. Bishop, “Scaat: Incremental Tracking with
4. H.L. Pryor, T.A. Furness, and E. Viirre, “The Virtual Reti- Incomplete Information,” Proc. ACM Siggraph 97, ACM
nal Display: A New Display Technology Using Scanned Press, New York, 1997, pp. 333-344.
Laser Light,” Proc. 42nd Human Factors Ergonomics Soc., 20. E. Foxlin, M. Harrington, and G. Pfeifer, “Constellation: A
Santa Monica, Calif., 1998, pp. 1570-1574. Wide-Range Wireless Motion-Tracking System for Aug-
5. M.B. Spitzer et al., “Eyeglass-Based Systems for Wearable mented Reality and Virtual Set Applications,” Proc. Sig-
Computing,” Proc. 1st Int’l Symp. Wearable Computers, graph 98, ACM Press, New York, 1998, pp. 371-378.
(ISWC 97), IEEE CS Press, Los Alamitos, Calif., 1997, pp. 21. G. Welch et al., “High-Performance Wide-Area Optical
48-51. Tracking—The HiBall Tracking System,” Presence: Teleop-
6. I. Kasai et al., “A Forgettable Near Eye Display,” Proc. 4th erators and Virtual Environments. vol. 10, no. 1, Feb. 2001,
Int’l Symp. Wearable Computers (ISWC 2000), IEEE CS pp. 1-21.
Press, Los Alamitos, Calif., 2000, pp. 115-118. 22. Y. Cho, J. Lee, and U. Neumann, “A Multi-Ring Fiducial Sys-
7. J. Rekimoto, “NaviCam: A Magnifying Glass Approach to tem and an Intensity-Invariant Detection Method for Scal-
Augmented Reality,” Presence: Teleoperators and Virtual able Augmented Reality,” Proc. Int’l Workshop Augmented
Environments, vol. 6, no. 4, Aug. 1997, pp. 399-412. Reality (IWAR 98), A.K. Peters, Natick, Mass., 1998, pp.
8. R. Raskar, G. Welch, and W-C. Chen, “Table-Top Spatially- 147-165.
Augmented Realty: Bringing Physical Models to Life with 23. F. Sauer et al., “Augmented Workspace: Designing an AR
Projected Imagery,” Proc. 2nd Int’l Workshop Augmented Testbed,” Proc. Int’l Symp. Augmented Reality 2000 (ISAR
Reality (IWAR 99), IEEE CS Press, Los Alamitos, Calif., 00), IEEE CS Press, Los Alamitos, Calif., 2000, pp. 47-53.
1999, pp. 64-71. 24. Y. Ohta et al., “Share-Z: Client/Server Depth Sensing for
9. H. Hua et al., “An Ultra-light and Compact Design and See-through Head Mounted Displays,” Proc. 2nd Int’l Symp.
Implementation of Head-Mounted Projective Displays,” Mixed Reality (ISMR 2001), MR Systems Lab, Yokohama,
Proc. IEEE Virtual Reality 2001, IEEE CS Press, Los Alami- Japan, 2001, pp. 64-72.
tos, Calif., 2001, pp. 175-182. 25. T. Kanade et al., “Virtualized Reality: Digitizing a 3D Time-
10. M. Inami et al., “Visuo-Haptic Display Using Head-Mount- Varying Event As Is and in Real Time,” Proc. Int’l Symp.
ed Projector,” Proc. IEEE Virtual Reality 2000, IEEE CS Mixed Reality (ISMR 99), Springer-Verlag, Secaucus, N.J.,
Press, Los Alamitos, Calif., 2000, pp. 233-240. 1999, pp. 41-57.
11. S. Mann, “Telepointer: Hands-Free Completely Self Con- 26. R. Azuma et al., “A Motion-Stabilized Outdoor Augment-
tained Wearable Visual Augmented Reality without Head- ed Reality System,” Proc. IEEE Virtual Reality, IEEE CS
wear and without Any Infrastructural Reliance,” Proc. 4th Press, Los Alamitos, Calif., 1999, pp. 252-259.
Int’l Symp. Wearable Computers (ISWC 2000), IEEE CS 27. R. Azuma et al., “Tracking in Unprepared Environments
Press, Los Alamitos, Calif., 2000, pp. 177-178. for Augmented Reality Systems,” Computers and Graphics,
12. J. Underkoffler and H. Ishii, “Illuminating Light: An Optical vol. 23, no. 6, Dec. 1999, pp. 787-793.
Design Tool with a Luminous-Tangible Interface,” Proc. ACM 28. S. You, U. Neumann, and R. Azuma, “Hybrid Inertial and

44 November/December 2001
 Vision Trcking for Augmented Reality Registration,” Proc. Int’l Symp. Augmented Reality 2000 (ISAR 00), IEEE CS
IEEE Virtual Reality, IEEE CS Press, Los Alamitos, Calif., Press, Los Alamitos, Calif., 2000, pp. 159-164.
1999, pp. 260-267. 45. T. Ohshima et al., “RV-Border Guards: A Multi-Player
29. K. Satoh et al., “Townwear: An Outdoor Wearable MR Sys- Mixed Reality Entertainment,” Trans. Virtual Reality Soc.
tem with High-Precision Registration,” Proc. 2nd Int’l Symp. Japan, vol. 4, no. 4, 1999, pp. 699-705.
Mixed Reality (ISMR 2001), MR Systems Lab, Yokohama, 46. S. Feiner et al., “A Touring Machine: Prototyping 3D Mobile
Japan, 2001, pp. 210-211. Augmented Reality Systems for Exploring the Urban Envi-
30. K. Sawada, M. Okihara, and S. Nakamura, “A Wearable ronment,” Proc. 1st Int’l Symp. Wearable Computers (ISWC
Attitude Measurement System Using a Fiber Optic Gyro- ‘97), IEEE CS Press, Los Alamitos, Calif., 1997, pp. 74-81.
scope,” Proc. 2nd Int’l Symp. Mixed Reality (ISMR 2001), 47. J. Rekimoto and M. Saitoh, “Augmented Surfaces: A Spa-
MR Systems Lab, Yokohama, Japan, 2001, pp. 35-39. tially Continuous Workspace for Hybrid Computing Envi-
31. U. Neumann and S. You, “Natural Feature Tracking for ronments,” Proc. ACM SIGCHI 99, 1999, ACM Press, New
Augmented Reality,” IEEE Trans. Multimedia, vol. 1, no. 1, York, pp. 378-385.
Mar. 1999, pp. 53-64. 48. A. Butz et al., “Enveloping Users and Computers in a Col-
32. R. Behringer, “Registration for Outdoor Augmented Real- laborative 3D Augmented Reality,” Proc. 2nd Int’l Work-
ity Applications Using Computer Vision Techniques and shop Augmented Reality (IWAR 99), IEEE CS Press, Los
Hybrid Sensors,” Proc. IEEE Virtual Reality, IEEE CS Press, Alamitos, Calif., 1999, pp. 35-44.
Los Alamitos, Calif., 1999, pp. 244-251. 49. Zs. Szalavári and M. Gervautz, “The Personal Interaction
33. V. Coors, T. Huch, and U. Kretschmer, “Matching Build- Panel—A Two-Handed Interface for Augmented Reality,”
ings: Pose Estimation in an Urban Environment,” Proc. Int’l Proc. 18th Eurographics, Eurographics Assoc., Aire-la-Ville,
Symp. Augmented Reality 2000 (ISAR 00), IEEE CS Press, Switzerland, 1997, pp. 335-346.
Los Alamitos, Calif., 2000, pp. 89-92. 50. H. Kato et al., “Virtual Object Manipulation of a Table-Top
34. B. Jiang, S. You, and U. Neumann, “Camera Tracking for AR Environment,” Proc. Int’l Symp. Augmented Reality 2000
Augmented Reality Media,” Proc. IEEE Int’l Conf. Multime- (ISAR 00), IEEE CS Press, Los Alamitos, Calif., 2000, pp.
dia Expo 2000, IEEE CS Press, Los Alamitos, Calif., 2000, 111-119.
pp. 1637-1640. 51. T. Ohshima et al., “AR2 Hockey: A Case Study of Collabo-
35. G. Simon, A.W. Fitzgibbon, and A. Zisserman, “Markerless rative Augmented Reality,” Proc. IEEE Virtual Reality Ann.
Tracking Using Planar Structures in the Scene,” Proc. Int’l Int’l Symp. (VRAIS 98), IEEE CS Press, Los Alamitos, Calif.,
Symp. Augmented Reality 2000 (ISAR 00), IEEE CS Press, 1998, pp. 268-275.
Los Alamitos, Calif., 2000, pp. 120-128. 52. M. Billinghurst, H. Kato, and I. Poupyrev, “The Magic-
36. Y. Akatsuka and G.A. Bekey, “Compensation for End to End Book—Moving Seamlessly between Reality and Virtuali-
Delays in a VR System,” Proc. Virtual Reality Ann. Int’l ty,” IEEE Computer Graphics and Applications, vol. 21, no.
Symp. (VRAIS 98), IEEE CS Press, Los Alamitos, Calif., 3, May/June 2001, pp. 2-4.
1998, pp. 156-159. 53. B. MacIntyre and E.M. Coelho, “Adapting to Dynamic Reg-
37. L. Chai et al., “An Adaptive Estimator for Registration in istration Errors Using Level of Error (LOE) Filtering,” Proc.
Augmented Reality,” Proc. 2nd Int’l Workshop Augmented Int’l Symp. Augmented Reality 2000 (ISAR 00), IEEE CS
Reality (IWAR 99), IEEE CS Press, Los Alamitos, Calif., Press, Los Alamitos, Calif., 2000, pp. 85-88.
1999, pp. 23-32. 54. A. Fuhrmann et al., “Occlusion in Collaborative Augment-
38. M.C. Jacobs, M.A. Livingston, and A. State, “Managing ed Environments,” Computers and Graphics, vol. 23, no. 6,
Latency in Complex Augmented Reality Systems,” Proc. Dec. 1999, pp. 809-819.
1997 Symp. Interactive 3D Graphics, ACM Press, New York, 55. S. Julier et al., “Information Filtering for Mobile Augment-
1997, pp. 49-54. ed Reality,” Proc. Int’l Symp. Augmented Reality 2000 (ISAR
39. M. Regan et al., “A Real Time Low-Latency Hardware Light- 00), IEEE CS Press, Los Alamitos, Calif., 2000, pp. 3-11.
Field Renderer,” Proc. Siggraph 99, ACM Press, New York, 56. B. Bell, S. Feiner, and T. Höllerer, “View Management for
1999, pp. 287-290. Virtual and Augmented Reality,” Proc. ACM Symp. User
40. R. Kijima, E. Yamada, and T. Ojika, “A Development of Interface Software and Technology (UIST 2001), ACM Press,
Reflex HMD-HMD with Time Delay Compensation Capa- New York, 2001, pp. 101-110.
bility,” Proc. 2nd Int’l Symp. Mixed Reality (ISMR 2001), 57. V. Lepetit and M.-O. Berger, “Handling Occlusions in Aug-
MR Systems Lab, Yokohama, Japan, 2001, pp. 40-47. mented Reality Systems: A Semi-Automatic Method,” Proc.
41. W.R. Mark et al., “Post-Rendering 3D Warping,” Proc. 1997 Int’l Symp. Augmented Reality 2000 (ISAR 00), IEEE CS
Symp. Interactive 3D Graphics, ACM Press, New York, 1997, Press, Los Alamitos, Calif., 2000, pp. 137-146.
pp. 7-16. 58. J. Stauder, “Augmented Reality with Automatic Illumina-
42. K.N. Kukulakos and J.R. Vallino, “Calibration-Free Aug- tion Control Incorporating Ellipsoidal Models,” IEEE Trans.
mented Reality,” IEEE Trans. Visualization and Computer Multimedia, vol. 1, no. 2, June 1999, pp. 136-143.
Graphics, vol. 4, no. 1, Jan.-Mar. 1998, pp. 1-20. 59. Y. Mukaigawa, S. Mihashi, and T. Shakunaga, “Photomet-
43. Y. Seo and K-S. Hong, “Weakly Calibrated Video-Based ric Image-Based Rendering for Virtual Lighting Image Syn-
Augmented Reality: Embedding and Rendering through thesis,” Proc. 2nd Int’l Workshop Augmented Reality (IWAR
Virtual Camera,” Proc. Int’l Symp. Augmented Reality 2000 99), IEEE CS Press, Los Alamitos, Calif., 1999, pp. 115-124.
(ISAR 00), IEEE CS Press, Los Alamitos, Calif., 2000, pp. 60. P. Debevec, “Rendering Synthetic Objects into Real Scenes:
129-136. Bridging Traditional and Image-Based Graphics with Glob-
44. B. Hoff and R. Azuma, “Autocalibration of an Electronic al Illumination and High Dynamic Range Photography,” Proc.
Compass in an Outdoor Augmented Reality System,” Proc. Siggraph 98, ACM Press, New York, 1998, pp. 189-198.

IEEE Computer Graphics and Applications 45

Survey

61. U. Neumann and A. Majoros, “Cognitive, Performance, and ed Reality First Person Application,” Proc. 4th Int’l Symp.
Systems Issues for Augmented Reality Applications in Man- Wearable Computers (ISWC 2000), 2000, pp. 139-146.
ufacturing and Maintenance,” Proc. IEEE Virtual Reality 77. D. Stricker et al., “Design and Development Issues for
Ann. Int’l Symp. (VRAIS 98), IEEE CS Press, Los Alamitos, Archeoguide: An Augmented Reality based Cultural Her-
Calif., 1998, pp. 4-11. itage On-Site Guide,” Proc. Int’l Conf. Augmented Virtual
62. D. Drascic and P. Milgram, “Perceptual Issues in Aug- Environments and 3D Imaging (ICAV3D 2001), Publishing
mented Reality,” Proc. SPIE, Stereoscopic Displays VII and ZITI, Greece, 2001, pp. 1-5, http://www.ziti.gr.
Virtual Systems III, SPIE Press, Bellingham, Wash., vol. 78. J. Rekimoto, Y. Ayatsuka, and K. Hayashi, “Augmentable
2653, 1996, pp. 123-134. Reality: Situated Communication through Physical and
63. J.P. Rolland and H. Fuchs, “Optical Versus Video See- Digital Spaces,” Proc. 2nd Int’l Symp. Wearable Computers,
through Head-Mounted Displays in Medical Visualization,” IEEE CS Press, Los Alamitos, Calif., 1998, pp. 68-75.
Presence: Teleoperators and Virtual Environments, vol. 9, 79. R. Behringer et al., “A Wearable Augmented Reality Test-
no. 3, June 2000, pp. 287-309. bed for Navigation and Control, Built Solely with Com-
64. R.L. Holloway, “Registration Error Analysis for Augment- mercial-off-the-Shelf (COTS) Hardware,” Proc. Int’l Symp.
ed Reality,” Presence: Teleoperators and Virtual Environ- Augmented Reality 2000 (ISAR 00), IEEE CS Press, Los
ments. vol. 6, no. 4, Aug. 1997, pp. 413-432. Alamitos, Calif., 2000, pp. 12-19.
65. S.R. Ellis et al., “Factors Influencing Operator Interaction 80. S. Mann, “Wearable Computing: A First Step Toward Per-
with Virtual Objects Viewed via Head-Mounted See- sonal Imaging,” Computer, vol. 30, no. 2, Feb. 1997,
through Displays: Viewing Conditions and Rendering pp. 25-32.
Latency,” Proc. Virtual Reality Ann. Int’l Symp. (VRAIS 97), 81. M. Billinghurst and H. Kato, “Collaborative Mixed Reali-
IEEE CS Press, Los Alamitos, Calif., 1997, pp. 138-145. ty,” Proc. Int’l Symp. Mixed Reality (ISMR 99), Springer-Ver-
66. L. Vaissie and J. Rolland, “Accuracy of Rendered Depth in lag, Secaucus, N.J., 1999, pp. 261-284.
Head-Mounted Displays: Choice of Eyepoint Locations,” 82. J. Rekimoto, “Transvision: A Hand-Held Augmented Real-
Proc. SPIE AeroSense 2000, SPIE, Bellingham, Wash., vol. ity System for Collaborative Design,” Proc. Virtual Systems
4021, 2000, pp. 343-353. and Multimedia (VSMM 96), Int’l Soc. Virtual Systems and
67. D. Curtis et al., “Several Devils in the Details: Making an Multimedia, Gifu, Japan, 1996, pp. 85-90.
AR Application Work in the Airplane Factory,” Proc. Int’l 83. Zs. Szalavári et al., “Studierstube: An Environment for Col-
Workshop Augmented Reality (IWAR 98), A.K. Peters, Nat- laboration in Augmented Reality,” Virtual Reality—Systems,
ick, Mass., 1998, pp. 47-60. Development and Application, vol. 3, no. 1, 1998, pp. 37-49.
68. N. Navab et al., “Scene Augmentation via the Fusion of 84. S. Julier et al., “The Software Architecture of a Real-Time
Industrial Drawings and Uncalibrated Images with a View Battlefield Visualization Virtual Environment,” Proc. IEEE
to Markerless Calibration,” Proc. 2nd Int’l Workshop Aug- Virtual Reality 99, IEEE CS Press, Los Alamitos, Calif., 1999,
mented Reality (IWAR 99), IEEE CS Press, Los Alamitos, pp. 29-36.
Calif., 1999, pp. 125-133. 85. T. Jebara et al., “Stochasticks: Augmenting the Billiards
69. N. Navab, A. Bani-Hashemi, and M. Mitschke, “Merging Experience with Probabilistic Vision and Wearable Com-
Visible and Invisible: Two Camera-Augmented Mobile C- puters,” Proc. 1st Int’l Symp. Wearable Computers (ISWC
arm (CAMC) Applications,” Proc. 2nd Int’l Workshop Aug- 97), 1997, IEEE CS Press, Los Alamitos, Calif., pp. 138-145.
mented Reality (IWAR 99), IEEE CS Press, Los Alamitos, 86. R. Cavallaro, “The FoxTrax Hockey Puck Tracking System,”
Calif., 1999, pp. 134-141. IEEE Computer Graphics and Applications, vol. 17, no. 2,
70. H. Fuchs et al., “Augmented Reality Visualization for Mar./Apr. 1997, pp. 6-12.
Laparoscopic Surgery,” Proc. 1st Int’l Conf. Medical Image 87. F.P. Brooks, Jr., “What’s Real About Virtual Reality?,” IEEE
Computing and Computer-Assisted Intervention (MICCAI Computer Graphics and Applications, vol. 16, no. 6,
98), Springer-Verlag, Heidelberg, Germany, 1998, pp. 934- Nov./Dec. 1999, pp. 16-27.
943. 88. B. MacIntyre et al., “Ghosts in the Machine: Integrating 2D
71. S. Weghorst, “Augmented Reality and Parkinson’s Disease,” Video Actors into a 3D AR System,” Proc. 2nd Int’l Symp.
Comm. ACM, vol. 40, no. 8, Aug. 1997, pp. 47-48. Mixed Reality (ISMR 2001), MR Systems Lab, Yokohama,
72. T. Starner et al., “Augmented Reality through Wearable Japan, 2001, pp. 73-80.
Computing,” Presence: Teleoperators and Virtual Environ-
ments, vol. 6, no. 4, Aug. 1997, pp. 386-398.
73. T. Höllerer et al., “Exploring MARS: Developing Indoor and
Outdoor User Interfaces to a Mobile Augmented Reality Ronald Azuma is a senior research
System,” Computers and Graphics, vol. 23, no. 6, Dec. 1999, staff computer scientist at HRL Labs.
pp. 779-785. He received a BS in electrical engi-
74. Y. Baillot, D. Brown, and S. Julier, “Authoring of Physical neering and computer science from
Models Using Mobile Computers,” to appear in Proc. Int’l the University of California at Berke-
Symp. Wearable Computers, IEEE CS Press, Los Alamitos, ley in 1988, and MS and PhD degrees
Calif., 2001. in computer science from the Univer-
75. W. Piekarski, B. Gunther, and B. Thomas, “Integrating Vir- sity of North Carolina at Chapel Hill in 1990 and 1995,
tual and Augmented Realities in an Outdoor Application,” respectively. His research interests are in augmented real-
Proc. 2nd Int’l Workshop Augmented Reality (IWAR 99), ity, virtual environments, and visualization. He is program
IEEE CS Press, Los Alamitos, Calif., 1999, pp. 45-54. co-chair of the International Symposium on Augmented
76. B. Thomas et al., “ARQuake: An Outdoor/Indoor Augment- Reality (ISAR 2001).

46 November/December 2001
 Yohan Baillot is a software engi- Blair MacIntyre is an assistant
neer at ITT Industries working as a professor in the College of Comput-
consultant at the Naval Research ing at the Georgia Institute of Tech-
Lab, Washington, D.C. He received nology. He received BMath and
an MS degree in electrical engineer- MMath degrees from the University
ing in 1996 from ISIM, France, and of Waterloo in 1989 and 1991, and
an MS in computer science in 1999 received a PhD in computer science
from the University of Central Florida, Orlando. His from Columbia University in 1999. His research interests
research interests are in computer graphics, 3D displays, are in computer graphics and human–computer interac-
body tracking, 3D reconstruction, and mobile augmented tion, with a focus on the design and implementation of
reality. He is a member of the IEEE Computer Society. augmented reality systems.

Readers may contact Azuma at HRL Labs, 3011 Malibu

Reinhold Behringer is a senior Canyon Rd. MS RL96, Malibu, CA 90265, email
scientist at Rockwell Scientific and is [email protected].
working on projects related to aug-
mented reality, human–computer For further information on this or any other computing
interaction, and computer vision. He topic, please visit our Digital Library at http://comput-
received his MA in physics from er.org/publications/dlib.
SUNY Buffalo in 1998 and his diplo-
ma in physics from Würzburg University, Germany, in
1990. He received his PhD in engineering from the Uni-
versity of Bundeswehr, München, Germany in 1996. He
was the chair of IWAR 99 and co-chair of IWAR 98 and
ISAR 2000.

Steven K. Feiner is a professor of

computer science at Columbia Uni-
versity, where he directs the Comput-
er Graphics and User Interfaces Lab. Announcing
He received a PhD in computer science Two New
from Brown University in 1987. He is
the coauthor of Computer Graphics:
Resources for
Principles and Practice (Addison-Wesley, 1990), and is an Mobile, Wireless,
associate editor of ACM Transactions on Graphics. He was and Distributed
the program co-chair of IEEE Virtual Reality 2000, is the
general chair of IEEE Information Visualization 2001, and Applications
is the symposium co-chair of the International Symposium
on Augmented Reality 2001. In 1991 he received an Office IEEE Pervasive Computing IEEE Transactions on
of Naval Research Young Investigator Award. Strategies for mobile, wireless, and Mobile Computing
distributed applications, including State-of-the-art research papers
• Mobile computing on topics such as
• Wireless networks • Mobile computing
Simon Julier is a research scientist • Security, scalability, • Wireless networks
for ITT Industries at the Naval and reliability • Reliability and quality
Research Lab, Washington, D.C. He • Intelligent vehicles assurance
received a DPhil from the Robotics and environments • Distributed systems
Research Group, Oxford University, • Pervasive computing architecture
United Kingdom. He is a technical applications • High-level protocols
lead on the Battlefield Augmented
Reality System (Bars) Project. His research interests
include mobile augmented reality and large-scale distrib-
uted data fusion.

http://computer.org/pervasive http://computer.org/tmc

IEEE Computer Graphics and Applications 47

View publication stats