Projective Geometry and

Camera Models

Computer Vision
Jia-Bin Huang, Virginia Tech
Many slides from S. Seitz and D. Hoiem
Administrative stuffs
• HWs
• HW 1 will be back this week
• HW 2 due 11:55 PM on Oct 3.
• Submit your results on shape alignment: link
• Frequently asked questions for HW 2

• Think about your final projects

• work in groups of 2-4
• should evince independent effort to learn about a new
topic, try something new, or apply to an application of
interest
• Proposals will be due Oct 23
HW 2: Feature Tracking
function featureTracking
% Main function for feature tracking
folder = '.\images';
im = readImages(folder, 0:50);

tau = 0.06; % Threshold for harris corner detection

[pt_x, pt_y] = getKeypoints(im{1}, tau); % Prob 1.1: keypoint detection

ws = 7; % Tracking ws x ws patches
[track_x, track_y] = ... % Prob 1.2 Keypoint tracking
trackPoints(pt_x, pt_y, im, ws);

% Visualizing the feature tracks on the first and the last frame
figure(2), imagesc(im{1}), hold off, axis image, colormap gray
hold on, plot(track_x', track_y', 'r');
 HW 2 – Feature/Keypoint detection

• Compute second moment matrix

 I x2 ( D ) I x I y ( D )
 ( I ,  D )  g ( I )   
 I x I y ( D ) I y ( D ) 
2

• Harris corner criterion

har  det[ ( I , D)]   [trace( ( I , D)) 2 ] 

g ( I x2 ) g ( I y2 )  [ g ( I x I y )]2   [ g ( I x2 )  g ( I y2 )]2

• Threshold   0.04

• Non-maximum suppression
function [keyXs, keyYs] = getKeypoints(im, tau)
% im: input image; tau: threshold
% keyXs, keyYs: detected keypoints, with dimension [N] x [1]

% 0. Smooth image (optional)

% 1. Compute image gradients. Hint: can use “gradient”

% 2. Compute Ix*Ix, Iy*Iy, Ix*Iy

% 3. Smooth Ix*Ix, Iy*Iy, Ix*Iy (locally weighted average)

% 4. Compute Harris corner score. Normalize the max to 1

% 5. Non-maximum suppression. Hint: use “ordfilt2”

% 6. Find positions whose values larger than tau. Hint: use

“find”
end
function [track_x, track_y] = trackPoints(pt_x, pt_y, im, ws)
% Tracking initial points (pt_x, pt_y) across the image sequence
% track_x: [Number of keypoints] x [nim]
% track_y: [Number of keypoints] x [nim]

% Initialization
N = numel(pt_x); % Number of keypoints
nim = numel(im); % Number of images
track_x = zeros(N, nim);
track_y = zeros(N, nim);
track_x(:, 1) = pt_x(:);
track_y(:, 1) = pt_y(:);

for t = 1:nim-1 % Tracking points from t to t+1

[track_x(:, t+1), track_y(:, t+1)] = ...
getNextPoints(track_x(:, t), track_y(:, t), im{t}, im{t+1}, ws);
end
Iterative L-K algorithm
Original (x,y) position

1. Initialize (x’,y’) = (x,y) It = I(x’, y’, t+1) - I(x, y, t)

2. Compute (u,v) by

2nd moment matrix for feature

patch in first image displacement

3. Shift window by (u, v): x’=x’+u; y’=y’+v;

4. Recalculate It
5. Repeat steps 2-4 until small change
• Use interpolation for subpixel values
function [x2, y2] = getNextPoints(x, y, im1, im2, ws) % Iterative Lucas-Kanade feature tracking
% (x, y) : initialized keypoint position in im1; (x2, y2): tracked keypoint positions in im2
% ws: patch window size

% 1. Compute gradients from Im1 (get Ix and Iy)

% 2. Grab patches of Ix, Iy, and im1.
Hint 1: use “[X, Y] = meshgrid(-hw:hw,-hw:hw);” to get patch index, where hw = floor(ws/2);
Hint 2: use “interp2” to sample non-integer positions.

for iter = 1:numIter % 5 iterations should be sufficient

% Check if tracked patch are outside the image. Only track valid patches.
% For each keypoint
% - grab patch1 (centered at x1, y1), grab patch2 (centered at x2,y2)
% - compute It = patch2 – patch1
% - grab Ix, Iy (centered at x1, y1)
% - Set up matrix A and vector b
% - Solve linear system d = A\b.
% - x2(p)=x2(p)+d(1); y2(p)=y2(p)+d(2); -> Update the increment
end
HW 2 – Shape Alignment

• Global transformation (similarity, affine, perspective)

• Iterative closest point algorithm
Fitting an affine transformation
 m1 
 
   m2  
x yi 0 0 1 0  m3   xi 
 i  
0 0 xi yi 0 1 m4   yi 
    
   t1 
 
 t 2 
• Linear system with six unknowns
• Each match gives us two linearly
independent equations: need at least three
to solve for the transformation parameters
• 
function Tfm = align_shape(im1, im2)
% im1: input edge image 1
% im2: input edge image 2
% Output: transformation Tfm [3] x [3]

% 1. Find edge points in im1 and im2. Hint: use “find”

% 2. Compute initial transformation (e.g., compute translation and scaling
by center of mass, variance within each image)

for i = 1: 50

% 3. Get nearest neighbors: for each point find corresponding

% 4. Compute transformation T based on matches

% 5. Warp points p according to T

end
HW 2 – Local Feature Matching

• Implement distance ratio test feature matching

algorithm
function featureMatching
im1 = imread('stop1.jpg');
im2 = imread('stop2.jpg');
% Load pre-computed SIFT features
load('SIFT_features.mat');
% Descriptor1, Descriptor2: SIFT features
% Frame1, Frame2: position, scale, rotation

% For every descriptor in im1, find the 1st nearest neighbor

and the 2nd nearest neighbor in im2.

% Compute distance ratio score.

% Threshold and get the matches: a 2 x N array of indices

that indicates which keypoints from image1 match which
points in image 2
figure(1), hold off, clf
plotmatches(im2double(im1),im2double(im2),Frame1,Frame2,mat
ches); % Display the matched keypoints
 Review: Interpreting Intensity
• Light and color
–What an image records
• Filtering in spatial domain
• Filtering = weighted sum of neighboring pixels
• Smoothing, sharpening, measuring texture
• Filtering in frequency domain
• Filtering = change frequency of the input image
• Denoising, sampling, image compression
• Image pyramid and template matching
• Filtering = a way to find a template
• Image pyramids for coarse-to-fine search and multi-
scale detection
• Edge detection
• Canny edge = smooth -> derivative -> thin ->
threshold -> link
 Review: Correspondence and Alignment
• Interest points
• Find distinct and repeatable points in images
• Harris-> corners, DoG -> blobs
• SIFT -> feature descriptor
• Feature tracking and optical flow
• Find motion of a keypoint/pixel over time
• Lucas-Kanade:
• brightness consistency, small motion, spatial coherence
• Handle large motion:
• iterative update + pyramid search
• Fitting and alignment
• find the transformation parameters that
best align matched points
• Object instance recognition
• Keypoint-based object instance recognition and search
Perspective and 3D Geometry
• Projective geometry and camera models
• What’s the mapping between image and world coordiantes?
• Single view metrology and camera calibration
• How can we measure the size of 3D objects in an image?
• How can we estimate the camera parameters?
• Photo stitching
• What’s the mapping from two images taken without camera
translation?
• Epipolar Geometry and Stereo Vision
• What’s the mapping from two images taken with camera
translation?
• Structure from motion
• How can we recover 3D points from multiple images?
Next two classes:
Single-view Geometry
How tall is this woman?

How high is the camera?

What is the camera

rotation?

What is the focal length of

the camera?

Which ball is closer?

Today’s class
Mapping between image and world coordinates

• Pinhole camera model

• Projective geometry Vertical vanishing

point

• Vanishing points and lines Vanishing

(at infinity)

line

Vanishing
Vanishing
point

• Projection matrix
point

x  K R t  X
 Image formation

Let’s design a camera

–Idea 1: put a piece of film in front of an object
–Do we get a reasonable image?
Slide source: Seitz
 Pinhole camera

Idea 2: add a barrier to block off most of the rays

• This reduces blurring
• The opening known as the aperture

Slide source: Seitz

 Pinhole camera
f

f = focal length
c = center of the camera

Figure from Forsyth

Camera obscura: the pre-camera
• First idea: Mo-Ti, China (470BC to 390BC)
• First built: Alhazen, Iraq/Egypt (965 to 1039AD)

Illustration of Camera Obscura Freestanding camera obscura at UNC Chapel Hill

Photo by Seth Ilys
Camera Obscura used for Tracing

Lens Based Camera Obscura, 1568

First Photograph

Oldest surviving photograph Photograph of the first photograph

• Took 8 hours on pewter plate

Joseph Niepce, 1826 Stored at UT Austin

Niepce later teamed up with Daguerre, who eventually created Daguerrotypes

Dimensionality Reduction Machine (3D to 2D)

3D world 2D image

Point of observation

Figures © Stephen E. Palmer, 2002

 Slide source: Seitz

Projection can be tricky…

 Slide source: Seitz

Projection can be tricky…

Making of 3D sidewalk art: http://www.youtube.com/watch?v=3SNYtd0Ayt0

Projective Geometry
What is lost?
• Length

Who is taller?

Which is closer?
Length is not preserved

A’

C’

B’

Figure by David Forsyth

Projective Geometry
What is lost?
• Length
• Angles

Parallel?

Perpendicular?
Projective Geometry
What is preserved?
• Straight lines are still straight
Vanishing points and lines
Parallel lines in the world intersect in the image at a
“vanishing point”
Vanishing points
 Vanishing points and lines
Vanishing Point Vanishing Point
o o
Vanishing Line

• The projections of parallel 3D lines intersect at a vanishing point

• The projection of parallel 3D planes intersect at a vanishing line
• If a set of parallel 3D lines are also parallel to a particular plane, their vanishing
point will lie on the vanishing line of the plane
• Not all lines that intersect are parallel
• Vanishing point <-> 3D direction of a line
• Vanishing line <-> 3D orientation of a surface
 Vanishing points and lines

Vertical vanishing
point
(at infinity)

Vanishing
line

Vanishing
Vanishing
point
point

Slide from Efros, Photo from Criminisi

 Vanishing points and lines

Photo from online Tate collection

Note on estimating vanishing points

Use multiple lines for better accuracy

… but lines will not intersect at exactly the same point in practice
One solution: take mean of intersecting pairs
… bad idea!
Instead, minimize angular differences
Vanishing objects
Projection:
world coordinatesimage coordinates

X 
Optical
Center
. P  Y 
 
(u0, v0) Y  Z 

. f
. Z
X

v
Camera Center
. u
(tx, ty, tz)

 X
u   f Z
p 
v    f Y

 Z
Homogeneous coordinates
• Is this a linear transformation?
– no—division by z is nonlinear

Converting to homogeneous coordinates

homogeneous image homogeneous scene

coordinates coordinates

Converting from homogeneous coordinates

Homogeneous coordinates

Invariant to scaling
 x   kx 
     kx
  w
x
k  y    ky    ky    y 
kw

 w kw  kw   w 
Homogeneous Cartesian
Coordinates Coordinates

Point in Cartesian is ray in Homogeneous

Basic geometry in homogeneous coordinates
ui 
• Append 1 to pixel coordinate to get homogeneous  
coordinate pi   vi 
• Line equation:  𝑙𝑖𝑛 𝑒⊤ 𝑝 =0
𝑖
 1 
𝑎𝑢+𝑏𝑣 +𝑐= 0
   𝑙𝑖𝑛 𝑒 ⊤
𝑖 =[ 𝑎 𝑏 𝑐 ]

• Line given by cross product of two points

 𝑙𝑖𝑛 𝑒 = 𝑝 × 𝑝
𝑖𝑗 𝑖 𝑗

• Intersection of two lines given by cross product of the

lines qij  line i  line j
• Three points lies on the same line
  𝑝⊤ ( 𝑝 × 𝑝 ) =0
𝑘 𝑖 𝑗

• Three lines intersect at the same point

 𝑙𝑖𝑛𝑒⊤ ( 𝑙𝑖𝑛𝑒 ×𝑙𝑖𝑛𝑒 ) =0
𝑘 𝑖 𝑗
Another problem solved by homogeneous
coordinates

Intersection of parallel lines

Cartesian: (Inf, Inf) Cartesian: (Inf, Inf)
Homogeneous: (1, 1, 0) Homogeneous: (1, 2, 0)
Interlude: where can this be useful?
Applications

Object Recognition (CVPR 2006)

Applications

Single-view reconstruction (SIGGRAPH 2005)

Applications

Getting spatial layout in indoor scenes (ICCV 2009)

Applications

Image Completion [SIGGRAPH14]

- Revealing unseen pixels
 Applications
Inserting synthetic objects into images: http://vimeo.com/28962540
Applications
Creating detailed and complete 3D scene models from a single view
Two-mins break
 Perspective Projection Matrix
• Projection is a matrix multiplication using homogeneous
coordinates

 x
f 0 0 0    f x 
0   y   x y
 f 0 0   f y (f , f )
z z z
 0 0 1 0    z  divide by the third
1  coordinate

In practice: lots of coordinate transformations…

Camera to World to 3D
2D Perspective
= pixel coord. camera coord. point
point projection matrix
trans. matrix trans. matrix (4x1)
(3x1) (3x4)
(3x3) (4x4)
 Slide Credit: Saverese

Projection matrix
R,T
jw

kw
Ow
iw

x  K R t  X
x: Image Coordinates: (u,v,1)
K: Intrinsic Matrix (3x3)
R: Rotation (3x3)
t: Translation (3x1)
X: World Coordinates: (X,Y,Z,1)
 Projection matrix

Intrinsic Assumptions Extrinsic Assumptions

• Unit aspect ratio • No rotation
• Optical center at (0,0) • Camera at (0,0,0)
• No skew K

x
u   f 0 0 0  
x  K  I 0 X
y
wv    0 f 
0 0  
    z 
1   0 0 1 0  
1 
Slide Credit: Saverese
Remove assumption: known optical
center
Intrinsic Assumptions Extrinsic Assumptions
• Unit aspect ratio • No rotation
• No skew • Camera at (0,0,0)

 x
u   f 0 u0 0  
x  K  I 0 X
y
wv    0 f v0 
0  
    z 
1   0 0 1 0  
1 
Remove assumption: square pixels

Intrinsic Assumptions Extrinsic Assumptions

• No skew • No rotation
• Camera at (0,0,0)

 x
u   0 u0 0  
y
x  K  I 0 X wv    0  v0 
0  
    z 
1   0 0 1 0  
1 
Remove assumption: non-skewed
pixels
Intrinsic Assumptions Extrinsic Assumptions
• No rotation
• Camera at (0,0,0)

x
u   s u0 0  
x  K  I 0 X
y
wv    0  v0 
0  
    z 
1   0 0 1 0  
1 

Note: different books use different notation for parameters

Oriented and Translated Camera
R
jw

t kw
Ow
iw
 Allow camera translation

Intrinsic Assumptions Extrinsic Assumptions

• No rotation

 x
u   0 u0  1 0 0 t x   
x  K I t X
  y
wv    0  
v0 0 1 0 t y  
     z 
1   0 0 1  0 0 1 t z   
1 
 Slide Credit: Saverese

3D Rotation of Points

Rotation around the coordinate axes, counter-clockwise:

1 0 0 
Rx ( )  0 cos   sin  
0 sin  cos  
p’
 cos  0 sin  
g
R y (  )   0 1 0 
p
y  sin  0 cos  
cos   sin  0
Rz ( )   sin  cos  0
z
 0 0 1
Allow camera rotation

x  K R t X

x
u   s u0   r11 r12 r13 tx   
y
wv    0  v0  r21 r22 r23 
ty  
     z 
1   0 0 1  r31 r32 r33 t z   
1 
Degrees of freedom

x  K R t X

5 6
x
u   s u0   r11 r12 r13 tx   
y
wv    0  v0  r21 r22 r23 
ty  
     z 
1   0 0 1  r31 r32 r33 t z   
1 
Vanishing Point = Projection from
Infinity
x
 y  x  xR 
p  K R t      
 p  KR y  p  K y R 
z    
   z   z R 
0 

fx R
u   f 0 u0   xR  u  u0
zR
wv    0 f v0   y R  
    
1   0 0 1   z R  fy R
v  v0
zR
Scaled Orthographic Projection
• Special case of perspective projection
• Object dimensions are small compared to distance to
camera
Image World

x
u   f 0 0 0  
v    0  y
w f 0 0  
• Also called “weak perspective”     z 
1   0 0 0 s   
1 
Slide by Steve Seitz
Orthographic Projection - Examples
Orthographic Projection - Examples
Applications in object detection
Far field: object appearance doesn’t change as objects translate

Near field: object appearance changes as objects translate

 Beyond Pinholes: Radial Distortion
• Common in wide-angle lenses or for
special applications (e.g., security)
• Creates non-linear terms in
projection
• Usually handled by through solving
for non-linear terms and then
correcting image

Corrected Barrel Distortion

Image from Martin Habbecke

Things to remember Vertical vanishing
point
(at infinity)
Vanishing
line

• Vanishing points and Vanishing Vanishing

point point
vanishing lines

• Pinhole camera model

and camera projection
matrix x  K R t X

• Homogeneous
coordinates
Next class
Applications of camera model and
projective geometry

• Recovering the camera intrinsic and extrinsic

parameters from an image

• Measuring size in the world

• Projecting from one plane to another