Introduction to Augmented Reality

Prof. A S Rao

Robotics & CAD/CAM Laboratory

Mechanical Engineering Department
VEERMATA JIJABAI TECHNOLOGICAL INSTITUTE MUMBAI-
400019

-1-
 Definition of Virtual Reality …

Real Virtuality

Virtuality/ Reality continuum

Real Augmented Augmented Virtual

Environment Reality (AR) Virtuality (AV) Environment

Mixed Reality Spectrum

[Milgram & Kishikino 94]

-2-
Components of an AR system :

-3-
What is "Augmented Reality“

• Coexistence of real and

real virtual objects.

• Additional context
dependent information in
the field of view of the
user.

• Usually markers are used

for aligning both.

• This is also known as

“Extended Reality”

virtual

-4-
Applications: Automotive Development:
Crash test of an automobile

Crash-Test:

Comparison simulation data

with real data

-5-
Applications: Automotive Development:
Training on the assembly of a door lock

-6-
 Applications: Assembly of wires (loom) in airplanes

Without AR

D. Mizell, Boeing 1998 With AR

-7-
 Applications: Service applications

Benefits in service applications:

• Failure analysis
• Repair manuals
• Remote expertise
• World-wide access to knowledge -8-
Applications: Architecture presentation

Benefits:

• Effect of virtual models

in real environment

• Superimposition of the
virtual bridge in a video

• Here stereo-vision is not

possible. However, if the
scene is shot from
several angles, each
video can be superimposed
with the bridge at
appropriate angles.

-9-
 Applications: Medicine

UNC CMU + MIT AI-Lab

Shadyside + Brigham
Hospital Women‘s
Hospital
VR: The model of the patient is available on which the doctor can rehearse the operation.
AR: The zone to be operated is highlighted and the model of the tumor constructed from CT scan etc. is projected onto
the actual patient. Now it is real.
RE: for capturing the geometry.
RP&T: for physical realization.

Note: The matching of real and virtual objects is very vital in medical applications!

-10-
Applications: Others

• Sports and games

• Design evaluation

• Visualization for
researchers

• Video conference

-11-
 Applications: Others

Aqua Gauntlet (MRL Japan) AR2 Hockey (MRL Japan)

-12-
 Two Types of Augmented Reality Systems
Real time AR:

• Virtual objects are merged with the real

environment in which the user is present.
Real time AR
• The line of sight is variable.

Augmented Video:

• Virtual objects are integrated into a video of

real environment.

• The line of sight is fixed.

Augmented
Video

-13-
 Principle of Augmented Reality

Virtual Objects Real Objects

Merging

Display

-14-
 Augmented Reality System

Virtual Real Objects

Objects

Positioning

Visualization
computation
Mixing
Interactio
n
Display

-15-
 Augmented Reality System ...

Virtual Real Objects

Objects
Determination
Positioning of the line of
sight of the
Visualization user …
computation
Mixing
Interacti
on
Display

-16-
 Augmented Reality System ...

Virtual Real Objects

Objects through:
• Mechanical tracking
Positioning
• Electromagnetic
tracking
Visualization
computation • Optical tracking
Mixing
Interacti • Photogrammetry-based
tracking
Display
• Others

-17-
Tracking: Mechanical tracking

Boom

-18-
Tracking: Electromagnetic tracking

electromagnetic
sensor

-19-
Tracking: Optical tracking

Infrared-camera and
optical markers

-20-
Tracking: Photogrammetry-based tracking

Camera
The camera is easily
transportable and can
be used anywhere.

-21-
Cameras for the photogrammetry-based Tracking

-22-
 Other sensors
• GPS
• Ultrasonic
• Hybrid tracking...

Ultrasonic

GPS

-23-
 Tracking: Hybrid tracking

Hybrid tracking is required

where the range of the
tracking is not adequate for
the required accuracy.
o Rough tracking: GPS
o Fine tracking:
Photogrammetry-based
Tracking

Rough tracking with GPS

Photogrammetry-based
tracking for exact
Positioning

-24-
 Positioning the Virtual Camera

Virtual Real Objects

Objects Transformation of
coordinate
Positioning systems:
The coordinate
Visualization
computation system of the virtual
Mischen environment must
Interacti be matched with the
Display coordinate system
of the real
environment

-25-
 Positioning the Virtual Camera …

World Coordinate System & Object Coordinate System

Y Object coordinates system:

Coordinate system attached to the
object (example: Pencil in Frankfurt)
Z

World Coordinate System

y
X = coordinate system in
Z which the objects are
present
= coordinate system of the
scene
X

-26-
Augmented Reality System – Marker-Based Tracking

Virtual Real Objects

Objects • Picture-based
Tracking
Positioning • Marker-based
Tracking
Visualization • Character-based
computation Tracking
Mixing
Interac • Purely picture-
ti based Tracking
Display

Camera

-27-
 Marker-based Tracking

Marker-based tracking is based on well distinguishable marker samples:

-28-
 Marker-based Tracking

Virtual
Object

On the basis the position

of the characteristic points
the position of the virtual
objects is computed
Camera takes up the Picture is processed to
view of the user extract the characteristic
sample points

-29-
Augmented Reality System – Character-based Tracking

Virtual Real Objects

Objects • Picture-based
Tracking
Positioning • Marker-based
Tracking
Visualization • Character-based
computation Tracking
Mixing
Interac • Purely picture-
ti based Tracking
Display

Kamera

-30-
Character-based Tracking

-31-
Character-based Tracking ...

On the basis special

characters the line of sight
of the viewer can be
determined

-32-
 Character-based Tracking ...

Position of the virtual

object is computed on
the basis certain
characters

-33-
Character-based Tracking ...

... and
superimposed.

-34-
Augmented Reality System – Purely Picture-based Tracking

Virtual Real Objects

Objects • Picture-based
Tracking
Positioning
• Marker-based
Tracking
Visualization
computation • Characteristic-
based Tracking
Mixing
Interac • Purely picture-
ti based Tracking
Display

Kamera

-35-
Purely Picture-based Tracking

In this method, the user does not have to identify

the characters for tracking. The system software is
built with the adequate pattern recognition
knowledge to choose prominent features common
to the real and imaginary worlds.
Certain photographs of known objects taken from
certain positions in the environment are used to
calibrate the environment.

-36-
 Purely Picture-based Tracking ...

User view

The user view is

compared with the
calibration pictures
There exists no match
between these two
pictures.

-37-
 Purely Picture-based Tracking ...

User view

There exists substantial

match between these two
pictures.

-38-
 Purely Picture-based Tracking ...

0.1

0.7

User view
Augmentation

0.8

Calibration pictures
Compute the
Matching.

-39-
 Picture-based Tracking : Comparison

Marker-based Tracking:
Advantages:
• Very reliable, insensitive to changes in shading and lighting conditions
• Markers are essential.
• Suitable for applications such as medicine where high accuracy is required.
Limitation:
• Attaching markers is always not possible

Character-based and purely picture-based Tracking:

Advantages:
• No markers are required
• Suitable for outdoor scenes
Limitation:
• Less accurate.
• Sensitive to changes in shading and lighting conditions
-40-
Augmented Reality System : Visual Computation

Virtual Real Objects

Objects • Determination of the
position and the line of
Positioning sight of the user
• Adjustment of the
Visual Computation virtual camera

Mixing
Interacti
on
Display

-41-
 Augmented Reality System : Visual Computation ...

Virtual Real Objects

Objects

Positioning

Visual Computation
Filters
Briefcases Mixing
Interactio
Renders n
Display
(see visualization pipeline)

-42-
 Augmented Reality System : Visual Computation ...

Virtual Real Objects

Objects

Positioning

Visual Computation
Filters
Briefcases Mixing
Interactio
The visible
Renders n
region of the Display
(see virtual world
visualization pipeline)

is computed

-43-
 Augmented Reality System : Visual Computation ...

Virtual Real Objects

Objects

Positioning

Visual Computation
Filters
Briefcases Mixing
Interactio
Renders n
Allocation of special data Display
values (such as text and icons)
on certain graphic elements:

-44-
 Augmented Reality System : Visual Computation ...

Virtual Real Objects

Objects

Positioning

Visual Computation
Filters
Briefcases Mixing
Interactio
Renders n
Display
(see visualization pipeline)

•Hidden line removal

•Lighting

-45-
 Rendering: Hidden Line Removal

For computing the hidden regions, one needs depth information!

Model-basis:

• Use of the existing models or model

• 3D-Reconstruction

Picture-basis:

• Depth picture

-46-
 Hidden Line Removal – Model-basis

Photography of the city Frankfurt and a model reconstructed by modeling

-47-
Hidden Line Removal – Picture-basis

Admission of a depth picture with the z-Cam

-48-
Hidden Line Removal – Picture-basis ...

Pair of stereo
pictures

Depth picture /
Disparity-map created
from the pair of
stereo picture

-49-
Hidden Line Removal Computation

• One uses the depth information

about the material scene

• One can compute the hidden

ranges of the virtual objects

• and removed

-50-
 Augmented Reality System : Visual Computation

Virtual Real Objects

Objects

Positioning

Visual Computation
Filters
Briefcases Mixing
Interactio
Renders n
Display
(see visualization pipeline)
Special AR Techniques:
•Covering
•Lighting

-52-
Augmented Reality System : Display

Virtual Real Objects

Objects

Positioning

Visual Computation

Mixing
Interactio
n
Display

-53-
 Displays ...

Two kinds of displays can be used for real time AR:

• Optical-See-Through (partial-transparency)

• Video-See-Through (Video-picture)

-54-
Display devices

Optical-See-
Through

-55-
Displays – Principle of Optical „See-Through“

Monitor Graphics

Optical Combiner

Real scene
Virtual scene

-56-
 Displays

Optical „See Through“

 Simple and cheap

 Direct view of the real world

 Good resolution and no delay

 Less optical distortion

-57-
 Displays

Video-See-Through

-58-
Displays – Principle of Video-See-Through

Graphics

Monitor Combiner

Camera

Real scene
Virtual scene

-59-
 Displays

Video-See-Through

 Mismatch does not occur due to change in

helmet position etc.

 Digitized scene of the real world

- Flexibility with scene composition

- co-ordinating the delays possible

- additional registration possibilities

-60-
 Display - Trends

(See-Through) HMD MicroOptical: Prisma Microvision: Laser-Retinal Display

(Laser projection on retina)

-61-
 Hand-guided Displays

Binocular

Boom

-62-
 Augmented Reality System

Virtual Real Objects

Objects

Positioning

Visualization
computation
Mixing (combining)
Interacti
on
Display

-63-
 Mixing (combining)

The applied procedure depends on the kind of the display:

•Optical-See-Through (partial transparency):

- The visible parts of the virtual object are changed

•Video-See-Through (Video picture):

- Visible ranges of the virtual object overwrite appropriate pixels

of the video stream

-64-
Augmented Reality System – Interaction

Virtual Real Objects

Objects

Positioning

Visualization
computation
Mixing (combining)
Interacti
on
Display

-65-
Interaction

• Phonetic input is possible.

• Hand input devices such as keyboard are

used.

• Long distance communications through

internet, intranet and telephone are done.

• Depending on the bandwidth available for

communication, optimization is done. For
instance, when the bandwidth is low, the
user can speak which is converted into text
using voice synthesizer, transmitted over
long distance and it is converted back to
voice

-66-
 Interactions devices

Employment of the
appropriate devices
dependent on:

- Display

- task/ environment
Cyber
stick
Pen

Space mouse Keyboard

for the Arm

Head set

Twiddle
r

-67-
 Interactions devices – Advantages and Limitations

Speech input:

+ Hands free

- Prone for interferences and distortion

- Requires high bandwidth

Space mouse:

+ Navigation in 3D

- Precise navigation is relatively poor

Twiddler and Arm keyboard

+ Very handy

- It takes some time to get used to these.

-68-
 Augmented Reality System

Virtual Real Objects

Objects

Positioning

Visualization
computation
Mixing (combining)
Interacti
on
Display

-69-
 Augmented Reality & Virtual Reality

Augmented Reality and Virtual Reality are closely related techniques.

Both make use of

• Stereo-vision und

• Tracking tools.

-70-
Augmented Reality & Virtual Reality:
Projection-based Augmented Reality

Uses the advantages of immersive

projection technology for applications of
AR:
• high resolution through up and/or back
projection
• simple Stereo-goggles instead of HMDs -
higher acceptance
• stationary installation, problem-free Tracking

-71-
 Augmented Reality & Virtual Reality:
Projection-based Augmented Reality ...

Semi-transparent mirror permits the

combination of reality and virtuality
New applications:
• Augmented Design Review
• Annotations/ superposition over
physical models

-72-
Augmented Reality & Virtual Reality:
AR-Setups with multiple users

Semi-transparent mirrors permit multi-

user AR-Scenario:
• The perspectives corresponding to many viewers have to be
projected on the projection surface.
• The physical object being located in a semitransparent
mirror prism or cone can be overlaid with virtual
information.

-73-
 Augmented Reality & Virtual Reality: Virtuality Continuum (VC)

Applications Appn-1 Appn-2 ... Appn-n

Applications Programming Interface (new API‘s) • A kernel system can be

evolved for AR and VR.
VR VC AR

Mixed Reality Driver

New I/O peripherals (including sensors, tracking, etc..)

-74-
 Augmented Reality & Virtuelle Reality:

Mixing of AR and VR is described as virtuality continuum:

Virtuality Continuum (VC)

Real Augmented Augmented Virtual

Environment Reality (AR) Virtuality (AV) Environment

Mixed Reality
[Milgram & Kishikino 94]

-75-
 1985
1990
Visualisation

1995
Virtuelle Reality

Augmented Reality
today
Conclusions

Mixed Reality
-76-
 Augmented Reality URLs:

- Recent Advances in Augmented Reality:

http://www.cs.unc.edu/~azuma/cga2001.pdf

- Reinhold Behringer„s Page: http://www.Augmented-Reality.org

- Jim Vallino„s Page: http://www.se.rit.edu/~jrv/research/ar

- ARVIKA-Project-Page: http://www.arvika.de

-77-
 Literature:

1. Azuma, R. T.; Survey of Augmented Reality; Presence, Special Issue on Augmented

Reality, volume 6, 1997

2. Milgram, P.; Kishino, F.; A taxonomy of mixed reality visual displays; IEICE
Transactions on Information Systems, 1994

3. Feiner; MacIntyre; Seligmann; Knowledge-Based Augmented Reality; Communications

of the ACM, 1996

4. Faugeras, O.; Three-Dimensional Computer Vision; MIT-Press 1993

5. Tsai; An Efficient an Accurate Camera Calibration Technique for 3D Machine

Vision; Proc. CVPR, 1986

6. Weng, J.; Ahuja, N.; Huang, T.S.; Camera Calibration with Distortion Models an
Accuracy Evaluation; IEEE Transaction on Pattern Analysis and Machine
Intelligence, 1993

-78-