Chapter 2:

Displays
Augmented Reality – Principles and Practice

http://www.augmentedrealitybook.org
Multi-modal displays
• Audio displays
• Haptic displays
• Olfactory displays
• Gustatory displays

www.augmentedrealitybook.org Displays 2
• Haptic displays
Haptics is the science of applying tactile sensation to human interaction with
computers. A haptic device is one that involves physical contact between the computer
and the user, usually through an input/output device, such as a joystick or data gloves,
that senses the body's movements.

www.augmentedrealitybook.org Displays 3
Visuo-Haptic Registration
The stylus of a PHANToM haptic device is highlighted by visual AR

Image: Ulrich Eck and Christian Sandor

www.augmentedrealitybook.org Displays 4
• Haptic displays
Tactile Feedback is a type of Haptic Feedback. Haptic feedback is generally divided into two
different classes: Tacticle and Kinesthetic.
The difference between the two is quite complex, but at a high level:
Kinesthetic: The things you feel from sensors in your muscles, joints, tendons. Weight, stretch,
joint angles of your arm, hand, wrist, fingers, etc. Imagine holding a coffee-mug  in your hand.
Kinesthetic feedback tells your brain the approximate size of the mug, it's weight, and how you
are holding it relative to your body.
Tactile: The things you feel in your 'fingers' etc., or on the surface. The tissue (for example in your
fingers), has a number of different sensors embedded in the skin and right underneath it. They
allow your brain to feel things such as vibration, pressure, touch, texture etc.

Hapticwww.augmentedrealitybook.org
Feedback is a combination of both Tactile Displays
and Kinesthetic Feedback. 5
• Olfactory displays
An olfactory display is a computer-controlled device that generates scented air with the
intended component and concentration of odor material and provides it to the human
olfactory organ.
An olfactory display, which is the olfactory counterpart to a visual display, is controlled by
a computer or information equipment and provides smells to a human user.

www.augmentedrealitybook.org Displays 6
www.augmentedrealitybook.org Displays 7
Olfactory Display
MetaCookie: An olfactory display is combined with visual
augmentation of a plain cookie to provide the illusion of a
flavored cookie (chocolate, in the inset).

Image: Takuji Narumi

www.augmentedrealitybook.org Displays 8
• Gustatory displays
“edible user interfaces”
Food science does provide a basis for the development of gustatory displays.
"Meta Cookie" system to change the perceived taste of a cookie by overlaying visual and
olfactory information onto a real cookie which an AR marker pattern.

www.augmentedrealitybook.org Displays 9
Visual Displays
• See-through displays
• Optical see-through
• Video see-through
• Spatial Augmented Reality

www.augmentedrealitybook.org Displays 10
 Visual Displays
One of the most promising and challenging future uses of head-
mounted displays (HMDs) is in applications where virtual environments
enhance rather than replace real environments. To obtain an enhanced
view of the real environment, the user wears a see-through HMD to see
3D computer generated objects superimposed on his/her real-world
view.

www.augmentedrealitybook.org Displays 11
Optical See-Through Displays
An optical see-through display uses an optical element to combine a
user’s view of the real world with computer-generated images

Pose Sensors Generated Virtual Imagery

(potentially stereoscopic)

Real World

Optical
Combiners

www.augmentedrealitybook.org Displays 12
Video See-Through Displays
A video see-through display captures the real world with a video camera and electronically modifies
the resulting image using a graphics processor to deliver a combined real + virtual image to the user

Pose Sensors (optional)

Image
Sensors

Monitors Digital
Combiner

Generated
Virtual Imagery
(potentially
stereoscopic)

www.augmentedrealitybook.org Displays 13
Spatial Augmented Reality
Generated Projector Spatial projection casts images
Virtual Imagery directly onto real-world objects
Image No combiner unit is required
Sensors

Pose Sensors
(optional)

www.augmentedrealitybook.org Displays 14
 Non-See-Through Displays

The Rift is a binocular HMD intended for immersive computer games. It is under development by Oculus, which
was acquired by Facebook in 2014 for $2 billion, raising the interest in HMD technology worldwide

The Samsung Gear VR is an example of a

head-mounted display that uses a
smartphone (here: Galaxy S6) as the
main I/O and computational engine
www.augmentedrealitybook.org Displays 15
See-Through Display Taxonomy
See-Through
Displays

Monocular Binocular

Video Optical Video Optical

See-Through See-Through See-Through See-Through

Single Dual Monoscopic Stereoscopic

Camera Camera Overlays Overlays

Example
Products Monoscopic Stereoscopic Monoscopic Stereoscopic
E.g.: smartphone- or E.g.: Microvision Overlays Overlays Overlays Overlays
tablet-based Nomad,
hand-held AR DigiLens DL40, E.g.: Vuzix iWear VR920 E.g.: Microsoft HoloLens,
Also: Google Glass in TacEye ST, E.g.: Trivisio with Possible, but no E.g.: Canon COASTAR, Epson Moverio BT-200,
VST mode Vuzix M2000AR ARVision iWear CamAR clear advantage Vuzix Wrap 1200DXAR E.g.: Lumos DK-40 Vuzix STAR 1200XLD

www.augmentedrealitybook.org Displays 16
 Occluder Shadows
Projector-
based
illumination

The occlusion shadows technique uses controlled

illumination to blank out those portions of the real Pose Tra
cker
world where opaque graphics should be visible Occluded
part masked
out from
illumination

Virtual Real

Image: Oliver Bimber

www.augmentedrealitybook.org Displays 17
 Optical See-Through with Real Occlusion

Virtual image

Real image

LCD panel
Half-silvered
mirror

The ELMO HMD uses an additional LCD panel

between display and optical combiner for pixel-wise Image: Kiyoshi Kiyokawa
blocking of occluded real-world objects

www.augmentedrealitybook.org Displays 18
Spatial Display Model

www.augmentedrealitybook.org Displays 19
Coordinate Systems
Eye
Local object coordinates
coordinates

Perspective transformation
• Calibrate offline
• For both camera and display

Model transformation View transformation

• Track for moving objects, • Track for moving objects,
if there are static objects as well if there are no static objects
• Track for moving observer

Global world
coordinates

www.augmentedrealitybook.org Tracking 20
 Display Coordinate Systems
The spatial model of most AR displays can be defined as the spatial relationship of up to five components:
the user’s eye, the display, the camera, an object to be augmented, and the world.

Object World
World to camera to camera to object
Camera
Display to display World to camera
to eye
World to display
World to eye

Each coordinate transformation can be fixed and calibrated, tracked dynamically, or left unconstrained.

www.augmentedrealitybook.org Displays 21
 VISUAL Displays

• Near Eye Display

• Handheld Display
• Stationary Display
• Projected Display

www.augmentedrealitybook.org Displays 22
Display Space Taxonomy
AR displays can be categorized according to the distance from eye to display

Head-mounted Hand-held Stationary Projected

display display display display

Head space Body space World space

www.augmentedrealitybook.org Displays 23
Display Mounting

Helmet-mounted Clip-on Visor Display

Rockwell Collins SimEye Google Glass Epson Moverio

www.augmentedrealitybook.org Displays 24
Optical See-Through Examples

Sony Glasstron LDI-D100B,

retrofit on a custom mount as part of the
Columbia MARS system

Image: Columbia University

www.augmentedrealitybook.org Displays 25
See-Through Display with Optical Prism
eye

Coupled-in optics
Coupled-out optics
Image
source

The Light-Guide optical element technology by Lumus

propagates an image through a special optical prism

Image: Jens Grubert

www.augmentedrealitybook.org Displays 26
Optical See-Though

T
C

T
T

Without eye tracking With eye tracking

www.augmentedrealitybook.org Displays 27
Video See-Though

C T T T
C C

Without eye tracking With eye tracking

www.augmentedrealitybook.org Displays 28
Video See-Through with Half-Silvered
Mirror

Example of VST HMD using cameras

above the eyes with mirror optics.

Design by Andrei State, 2005.

Image: Andrei State, UNC Chapel Hill

www.augmentedrealitybook.org Displays 29
Oculus Rift with Stereo Video See-Through
The AR-Rift, a modified Oculus Rift with two video cameras

Image: William Steptoe

www.augmentedrealitybook.org Displays 30
Hand-Held Display

T
C

A handheld AR display can be built from an

unmodified smartphone or tablet computer

Image: Daniel Wagner

www.augmentedrealitybook.org Displays 31
 User-Perspective Hand-Held Display
Handheld display with device perspective Handheld display with user perspective

T T
C

Image: Domagoj Baričević

www.augmentedrealitybook.org Displays 32
Desktop AR
A desktop AR display can be built using the eyeball-in-
T hand metaphor, in which the camera is tracked and its
recordings are fed to the display. In the application
depicted here ([Lee and Höllerer 2007]), we are tracking
the camera relative to an object (user’s hand), which is
recognized as a marker and subsequently augmented.

Often, the camera is stationary, covering a working

T
volume, in which augmentations can occur. Again, we are
C tracking the camera relative to a moving object
(checkerboard pattern)

www.augmentedrealitybook.org Displays 33
Video-See Through Magic Mirror The user (=box) must be tracked
with respect to the camera.
T
C
T

C
Display always shows the user,
independent of viewing angle.

T
C

C
Display behaves like real mirror.
Image: Matthias Straka and Stefan Hauswiesner
www.augmentedrealitybook.org Displays 34
Optical See-Through Magic Mirror
Andy Wilson of Microsoft Research showing the HoloFlector

Image: Microsoft Research

www.augmentedrealitybook.org Displays 35
Virtual Showcase T
C

The Virtual Showcase is a stationary optical see-through C

display intended for exhibitions, museums, and showrooms

Image: Oliver Bimber

www.augmentedrealitybook.org Displays 36
Transparent Display
Samsung Transparent Smart Window display, showcased at CES 2012

www.augmentedrealitybook.org Displays 37
Immaterial Display
Dual-sided
interactive
FogScreen

Two FogScreens in an
L-shaped configuration
produce a depth-fused
3D rendering for a
People can augment each other and tracked observer
interact through the FogSscreen

www.augmentedrealitybook.org Displays 38
Spatial Augmented Reality
Spatial AR can be used to turn generic objects into textured models

C
View-independent
spatial AR

Image: Michael Marner

www.augmentedrealitybook.org Displays 39
View-Dependent Spatial Augmented Reality
View-dependent spatial AR requires tracking the user,
but can present free-space 3D objects
T
Projector

C
Head tracker

Virtual
object

Image: Oliver Bimber

www.augmentedrealitybook.org Displays 40
 Spatial Augmented Reality with Projector
Array
Multiple projectors can be Projectors
combined to minimize pixels
projected out of focus
C
T

The geometry of the projection surface

needs to be known (here: a display
calibrated to the world)

Projection Surface

www.augmentedrealitybook.org Displays 41
Head-Mounted Projective Display
incident light

diffusion Retro-reflective materials send incident rays

reflection retro-reflection back to the illuminating source, so they work
well with head-mounted projector displays
Lambertian reflector
Mirror reflector Retro-reflector
(e.g. unfinished wood)

Spatial relationship
C Spatial relationship C schematics for HMPDs with
schematics for HMPDs T head tracking - virtual
C without head tracking C objects are stable in space,
while the viewer is moving

www.augmentedrealitybook.org Displays 42
Dynamic Shader Lamps Dynamic shader lamps
T
deliver spatial AR on
Painting with light on real surfaces tracked objects

Animatronic character with animated facial projection

Image: Michael Marner Image: Greg Welch, UNC Chapel Hill

www.augmentedrealitybook.org Displays 43
Steerable Projector
Everywhere Projector Display

T
A steerable, tracked projector
can display images anywhere

Image: Claudio Pinhanez, IBM Research

www.augmentedrealitybook.org Displays 44