Computer Vision and Image Processing

– Fundamentals and Applications

Image Formation in a Stereo Vision Setup

n
ya
hu
.B
.K
M
Course Instructor: Dr. M.K. Bhuyan
Professor

Department of Electronics and Electrical Engineering,

Indian Institute of Technology Guwahati, India.
M
.K
.B
hu
ya
n
 n
ya
hu
.B
.K
M

Introduction to binocular stereo vision 3

M
.K
.B
hu
ya
n
M
.K
.B
hu
ya
n
M
.K
.B
hu
ya
n
Image Formation in a Stereo Vision Setup

n
ya
hu
.B
.K
M
Image formation in a stereo vision setup (Epipolar geometry)

n
ya
hu
.B
.K
M
M
.K
.B
hu
ya
n
M
.K
.B
hu
ya
n
M
.K
.B
hu
ya
n
M
.K
.B
hu
ya
n
STEREO IMAGES RECTIFICATION

n
ya
hu
.B
.K
M
STEREO IMAGES BEFORE AND AFTER RECTIFICATION

n
ya
hu
.B
.K
M
M
.K
.B
hu
ya
n
Elementary Stereo Geometry in the Rectified Configuration

n
ya
hu
.B
.K
M

Triangulation is a process of obtaining the three-dimensional

scene point from its projected points in the two image planes.
 Disparity is the horizontal distance
between the points U and U/

From the similar right-angled triangle

UCB and CPA

n
ya
hu
.B
.K
M
The remaining coordinates of the considered three dimensional point can be obtained as:
Two subproblems

• Matching (hardest)
• Finding corresponding elements in the two images

• Reconstruction

n
ya
hu
.B
• Establishing 3-D coordinates from the 2-D image

.K
M
correspondences found during matching
Prerequisites
• Camera model parameters must be known:

• External parameters:
• Positions, orientations

n
ya
hu
.B
• Internal parameters:

.K
M
• Focal length, image center, distortion, etc..
 Basics of Stereo Correspondence

n
ya
hu
.B
.K
M

Low gray level values correspond to low disparity values, whereas

high gray level values correspond to higher disparity values
Scene Point, P, imaged at
pL and pR

pL = 9
pR = 3

Disparity, d = pR – PL = 6

n
ya
hu
.B
.K
M
Disparity is the amount by
which the two images of P are
displaced relative to each
other

Mircea Nicolescu, University of Nevada at Reno

 Accurate Image Segmentation using Three-dimensional
Information

n
ya
hu
.B
.K
M

(a) Left image; (b) right image; (c) disparity map; (d) colour-based segmentation; and (e)
disparity map-based segmentation
Reconstruction
− Having found the corresponding points,
we can compute the disparity map
− Disparity maps are commonly expressed in pixels
ie number of pixels between corresponding points in two images
− Disparity map can be converted to a 3D map of the scene if the
geometry of the imaging system is known
− Critical parameters: Baseline, camera focal length, pixel size

n
ya
hu
.B
.K
M
M
.K
.B
hu
ya
n
 n
ya
hu
.B
.K
M

Introduction to binocular stereo vision 25

Goal: disparity map
• Disparity:
• The horizontal displacement between corresponding
points
• Closely related to scene depth

n
ya
hu
.B
.K
M
M
.K
.B
hu
ya
n
M
.K
.B
hu
ya
n
The matching problem
• Which image entities should be matched?
• Two main approaches
• Pixel/area-based (lower-level)
• Feature-based (higher-level)

n
ya
hu
.B
.K
M
Matching challenges
• Scene elements do not always look the same in the two images
• Camera-related problems
• Image noise, differing gain, contrast, etc..
• Viewpoint-related problems:

n
ya
hu
• Perspective distortions

.B
.K
M
• Occlusions
• Specular reflections
Choice of camera setup
• Baseline
• distance between cameras (focal points)

n
• Trade-off

ya
hu
.B
.K
M
• Small baseline: Matching easier
• Large baseline: Depth precision better
More matching heuristics
• Always valid:
• (Epipolar line)
• Uniqueness
• Minimum/maximum disparity

n
ya
hu
.B
.K
M
• Sometimes valid:
• Ordering
• Local continuity (smoothness)
 Stereo Matching Constraints and
Assumptions
▪ Epipolar constraint: Matching point of a
pixel in the left image lies in the
corresponding epipolar line in the right

n
ya
hu
.B
image.

.K
M
▪ Uniqueness constraint: In most cases, a
pixel from the first image can correspond to
only one pixel in the second image.
M
.K
.B
hu
ya
n
▪ Photometric compatibility constraint: Intensity
values of the pixels in a region in the left image
and its corresponding matching region in the right
image only slightly differ in intensity values.

n
ya
hu
.B
.K
▪ Geometric similarity constraint: These build on

M
the observation that geometric characteristics of
the features (such as length or orientation of a line
segment, contours or regions) found in the first
and second images do not differ much.
▪ Ordering constraint: This constraint says that for
regions having similar depth, the order of the
pixels in the left image and the order of their
matching pixels in the right image are the same.

n
ya
hu
.B
.K
M

Illustration of ordering constraint in two scenarios

▪ Disparity continuity constraint: This
constraint states that there is an abrupt
change in disparity values at the object
boundaries, whereas the disparity values do
not change significantly for smooth
regions.

n
ya
hu
.B
.K
M
▪ Disparity limit constraint: This constraint
imposes a global limit on the maximum
allowable disparity value between the
stereo images.
Issues Related to Accurate Disparity
Map Estimation

n
ya
hu
.B
.K
M
M
.K
.B
hu
ya
n
M
.K
.B
hu
ya
n
M
.K
.B
hu
ya
n
M
.K
.B
hu
ya
n
M
.K
.B
hu
ya
n
M
.K
.B
hu
ya
n
M
.K
.B
hu
ya
n
M
.K
.B
hu
ya
n
M
.K
.B
hu
ya
n
M
.K
.B
hu
ya
n
M
.K
.B
hu
ya
n
M
.K
.B
hu
ya
n
Area-based matching
• Finding pixel-to-pixel correspondences
• For each pixel in the left image, search for the most similar pixel in the right
image
• Using neighbourhood windows

n
ya
hu
.B
.K
M
 Intensity-based Methods
• Match image sub-windows between the two images
(e.g., using correlation).
left right

n
ya
hu
.B
.K
M
• How to choose the window size W?
– Too small a window may not capture enough image structure, and may
be too noise sensitive (i.e., less distinctive, many false matches).
– Too large a window makes matching less sensitive to noise (desired)
but also harder to match
Area-based matching
• Similarity measures for two windows
• SAD (sum of absolute differences)
• SSD (sum of squared differences)
• CC (cross-correlation)

n
ya
hu
.B
• …

.K
M
M
.K
.B
hu
ya
n
Feature-based matching
• Matching features:
• Edge points
• lines
• corners

n
ya
hu
.B
• …

.K
M
• Sparse reconstruction sets
• Best if scene type is known a priori
M
.K
.B
hu
ya
n
 Intensity-based vs feature-based approaches

• Intensity-based methods
– Provide a dense disparity map.
– Need textured images to work well.
– Sensitive to illumination changes.

n
ya
hu
.B
.K
M
• Feature-based methods:
– Faster than correlation-based methods.
– Provide sparse disparity maps.
– Relatively insensitive to illumination changes.
Three or more viewpoints
• More matching information
• Additional epipolar constraints
• More confident matches

n
ya
hu
.B
.K
M
M
.K
.B
hu
ya
n
Summary
Summary
• Stereo vision:
• A method for 3-D analysis of a scene using images from two or more
viewpoints
• Two subproblems:

n
ya
hu
.B
• Matching

.K
M
• Reconstruction
• Most difficult part: Matching
• Two main approaches:
• Area based: Dense reconstruction
• Feature based: Sparse reconstruction