Lecture 16

Stereo and 3D Vision

UW CSE vision faculty

Stereoscopes: A 19th Century Pastime

Public Library, Stereoscopic Looking Room, Chicago, by Phillips, 1923

Teesta suspension bridge-Darjeeling, India

Woman getting eye exam during immigration procedure at Ellis

Island, c. 1905 - 1920 , UCR Museum of Phography

Mark Twain at Pool Table", no date, UCR Museum of Photography

Anaglyphs
Anaglyphs provide a stereoscopic 3D effect when
viewed with 2-color glasses (each lens a
chromatically opposite color, usually red and cyan).
http://en.wikipedia.org/wiki/Anaglyph_image

A free pair of red-blue stereo glasses can be ordered from Rainbow Symphony Inc
http://www.rainbowsymphony.com/freestuff.html

3D Movies

How do we get 3D from Stereo Images?

Perception of depth arises from disparity of a given 3D point in
your right and left retinal images
3D point

left image

right image

disparity: the difference in image location of the same 3D

point when projected under perspective to two different cameras
d = xleft - xright

Recall (from Lecture 5): Perspective Projection

real image
point E

This is the axis of the

real image plane.
camera lens

O
f

image of point
B in front image

D
xi

B
3D object
point

O is the center of projection.

This is the axis of the front
image plane, which we use.
xi = x
f
z
(from similar triangles)

Z
(Note: For convenience, we orient Z axis as above and use f instead of f as in lecture 5)

Projection for Stereo Images

Simple Model: Optic axes of 2 cameras are parallel
image planes

camera L

xl

baseline
camera R

f
xr
x-b

X
z
f

x
xl

z
f

P=(x,z)
x-b
=
xr

z
f

(from similar triangles)

y =
yl

y
yr

Y-axis is
perpendicular
to the page.

3D from Stereo Images: Triangulation

For stereo cameras with parallel optical axes, focal length f,
baseline b, corresponding image points (xl,yl) and (xr,yr), the
location of the 3D point can be derived from previous slides
equations:
Depth z = f*b / (xl - xr) = f*b/d
x = xl*z/f or b + xr*z/f
y = yl*z/f or yr*z/f

This method of
determining depth
from disparity d is
called triangulation.

Note that depth is inversely proportional to disparity

Depth z = f*b / (xl - xr) = f*b/d

x = xl*z/f or b + xr*z/f
y = yl*z/f or yr*z/f

Two main problems:

1. Need to know focal length f, baseline b
- use prior knowledge or camera calibration

2. Need to find corresponding point (xr,yr) for

each (xl,yl) Correspondence problem

Solving the stereo correspondence problem

1. Cross correlation or SSD using small windows.

dense

2. Symbolic feature matching, usually using segments/corners.

sparse
3. Use the newer interest operators, e.g., SIFT.

sparse

Given a point in the left image, do you need

to search the entire right image for the
corresponding point?

Epipolar Constraint for Correspondence

Epipolar plane = plane connecting C1, C2, and point P P
z1

y1

z2
y2

P1

C1

P2

epipolar
plane
x

C2

.
Epipolar plane cuts through image planes forming an epipolar
line in each plane
Match for P1 (or P2) in the other image must lie on epipolar line

Epipolar Constraint for Correspondence

Match for P1 in the other image must lie on epipolar line
So need search only along this line
P

P1

C1

C2
Epipolar line

What if the optical axes of the 2

cameras are not parallel to each other?

Epipolar constraint still holds

P

e1 and e2 are the epipolar

lines for point P

y1
y2
P1

e2
e1

P2

x2

x1

C1
C2
But the epipolar lines may no longer be horizontal

Java demo: http://www.ai.sri.com/~luong/research/Meta3DViewer/EpipolarGeo.html

Example
Yellow epipolar
lines for the three
points shown on
the left image

(from a slide by Pascal Fua)

Given a point P1 in left image on epipolar line e1, can find

epipolar line e2 provided we know relative orientations of
cameras Requires camera calibration (see lecture 5)

Alternate approach: Stereo image rectification

Reproject image planes onto a common

plane parallel to the line between optical centers
Epipolar line is horizontal after this transformation
Two homographies (3x3 transforms), one for each
input image reprojection, is computed. See:
C. Loop and Z. Zhang. Computing Rectifying Homographies for
Stereo Vision. IEEE Conf. Computer Vision and Pattern Recognition,
1999.

Example

After rectification, need only search for matches along

horizontal scan line
(adapted from slide by Pascal Fua)

Your basic stereo algorithm

For each epipolar line

For each pixel in the left image
compare with every pixel on same epipolar line in right image
pick pixel with minimum match cost

Improvement: match windows

A good survey and evaluation: http://vision.middlebury.edu/stereo/

Matching using Sum of Squared Differences (SSD)

Stereo matching based on SSD

SSD
dmin
Best matching disparity

Problems with window size

Input stereo pair

W=3

Effect of window size W

Smaller window
+ Good precision, more detail
Sensitive to noise

Larger window
+ Robust to noise
Reduced precision, less detail

W = 20

Example depth from stereo results

Data from University of Tsukuba

Scene

Ground truth

Results with window-based stereo matching

Window-based matching
(best window size)

Ground truth

Better methods exist...

State of the art method:

Ground truth

Boykov et al., Fast Approximate Energy Minimization via Graph Cuts,

International Conference on Computer Vision, 1999

For the latest and greatest: http://www.middlebury.edu/stereo/

State of the art: Stereo as energy minimization

What defines a good stereo correspondence?

1. Match quality

Want each pixel to find a good match in the other image

2. Smoothness

If two pixels are adjacent, they should (usually) be displaced

about the same amount i.e., have similar disparities

Stereo as energy minimization

Expressing this mathematically
1. Match quality

Want each pixel to find a good match in the other image

2. Smoothness

If two pixels are adjacent, they should has similar disparities

We want to minimize

Stereo as energy minimization

We want to minimize:

Image as a graph with

disparity labels

This is a special type of energy

function known as an MRF
(Markov Random Field)

Effective and fast algorithms

have been recently developed:
Graph cuts, belief
propagation.
for more details (and code):
http://vision.middlebury.edu/
MRF/

Min-cost graph cut

yields a labeling of
each pixel with best
disparity

Stereo reconstruction pipeline

Steps

Calibrate cameras
Rectify images
Compute disparity
Estimate depth

What will cause errors?

Camera calibration errors

Poor image resolution
Occlusions
Violations of brightness constancy (specular reflections)
Large motions
Low-contrast image regions

Example Application: Robot Navigation

Nomad robot searches for meteorites in Antartica

http://www.frc.ri.cmu.edu/projects/meteorobot/index.html

Stereo also used in Mars Rover (Clark Olsons guest lecture)

Anaglyph from Mars Rover

What if 3D object has little or no texture?

Matching points might be difficult or impossible

Can we still recover depth information?

Idea: Use structured light!

Disparity between laser points on the same scanline in the images

determines the 3-D coordinates of the laser point on object
http://www.cs.wright.edu/~agoshtas/stereorange.html

Recovered 3D Model

http://www.cs.wright.edu/~agoshtas/stereorange.html

Actually, we can make do with just 1 camera

Virtual image plane
intersecting laser beam at
same distance as cameras
image place

Angle
b
From calibration of both camera and light projector, we can
compute 3D coordinates laser points on the surface

The Digital Michelangelo Project

Object
Direction of travel
Laser sheet
CCD image plane
Cylindrical lens
Laser

CCD
http://graphics.stanford.edu/projects/mich/

Optical triangulation
Project a single stripe of laser light
Scan it across the surface of the object

Laser scanned models

The Digital Michelangelo Project, Levoy et al.

Laser scanned models

The Digital Michelangelo Project, Levoy et al.

Laser scanned models

The Digital Michelangelo Project, Levoy et al.

Laser scanned models

The Digital Michelangelo Project, Levoy et al.

Laser scanned models

The Digital Michelangelo Project, Levoy et al.

A cool stereo application: Video View Interpolation

http://research.microsoft.com/users/larryz/videoviewinterpolation.htm

Now, for Project 2 voting results

3rd Place Winners (3-way tie)

Alex Eckerman and Mike Chung (52 votes)

John Lyon and James George (52 votes)

Aron Ritchie and Andrew Reusch (52 votes)

2nd Place Winner

Paramjit Singh Sandhu and Zhen Wang (63 votes)

Drumroll please

1st Place Winner

Brice Johnson and Will Johnson (68 votes)

Next Time: Guest lecture by Richard Ladner

Things to do:
Work on Project 4