Image Features:

Local Descriptors

Sanja Fidler CSC420: Intro to Image Understanding 1 / 57

Local Features
Detection: Identify the interest points.
Description: Extract a feature descriptor around each interest point.
Matching: Determine correspondence between descriptors in two views.

[Source: K. Grauman]
Sanja Fidler CSC420: Intro to Image Understanding 2 / 57
The Ideal Feature Descriptor

Repeatable: Invariant to rotation, scale, photometric variations

Distinctive: We will need to match it to lots of images/objects!

Compact: Should capture rich information yet not be too

high-dimensional (otherwise matching will be slow)

Efficient: We would like to compute it (close-to) real-time

Sanja Fidler CSC420: Intro to Image Understanding 3 / 57

Invariances

[Source: T. Tuytelaars]

Sanja Fidler CSC420: Intro to Image Understanding 4 / 57

Invariances

[Source: T. Tuytelaars]

Sanja Fidler CSC420: Intro to Image Understanding 4 / 57

What If We Just Took Pixels?
The simplest way is to write down the list of intensities to form a feature
vector, and normalize them (i.e., mean 0, variance 1).
Why normalization?
But this is very sensitive to even small shifts, rotations and any affine
transformation.

Sanja Fidler CSC420: Intro to Image Understanding 5 / 57

Tones Of Better Options

SIFT
PCA-SIFT
GLOH
HOG
SURF
DAISY
LBP
Shape Contexts
Color Histograms

Sanja Fidler CSC420: Intro to Image Understanding 6 / 57

Tones Of Better Options

SIFT TODAY
PCA-SIFT
GLOH
HOG
SURF
DAISY
LBP
Shape Contexts
Color Histograms

Sanja Fidler CSC420: Intro to Image Understanding 7 / 57

SIFT Descriptor [Lowe 2004]
SIFT stands for Scale Invariant Feature Transform
Invented by David Lowe, who also did DoG scale invariant interest
points
Actually in the same paper, which you should read:
David G. Lowe
Distinctive image features from scale-invariant
keypoints
International Journal of Computer Vision, 2004
Paper: http://www.cs.ubc.ca/~lowe/papers/ijcv04.pdf

Sanja Fidler CSC420: Intro to Image Understanding 8 / 57

SIFT Descriptor
1 Our scale invariant interest point detector gives scale ρ for each
keypoint

[Adopted from: F. Flores-Mangas]

Sanja Fidler CSC420: Intro to Image Understanding 9 / 57
SIFT Descriptor
2 For each keypoint, we take the Gaussian-blurred image at
corresponding scale ρ

[Adopted from: F. Flores-Mangas]

Sanja Fidler CSC420: Intro to Image Understanding 9 / 57
SIFT Descriptor
3 Compute the gradient magnitude and orientation in neighborhood of
each keypoint

[Adopted from: F. Flores-Mangas]

Sanja Fidler CSC420: Intro to Image Understanding 9 / 57
SIFT Descriptor
3 Compute the gradient magnitude and orientation in neighborhood of
each keypoint

Sanja Fidler CSC420: Intro to Image Understanding 9 / 57

SIFT Descriptor
4 Compute dominant orientation of each keypoint. How?

[Adopted from: F. Flores-Mangas]

Sanja Fidler CSC420: Intro to Image Understanding 9 / 57
SIFT Descriptor: Computing Dominant Orientation
Compute a histogram of gradient orientations, each bin covers 10◦

[Adopted from: F. Flores-Mangas]

Sanja Fidler CSC420: Intro to Image Understanding 10 / 57
SIFT Descriptor: Computing Dominant Orientation
Compute a histogram of gradient orientations, each bin covers 10◦
Orientations closer to the keypoint center should contribute more

[Adopted from: F. Flores-Mangas]

Sanja Fidler CSC420: Intro to Image Understanding 10 / 57
SIFT Descriptor: Computing Dominant Orientation
Compute a histogram of gradient orientations, each bin covers 10◦
Orientations closer to the keypoint center should contribute more
Orientation giving the peak in the histogram is the keypoint’s orientation

[AdoptedSanja
from:
Fidler F. Flores-Mangas]
CSC420: Intro to Image Understanding 10 / 57
SIFT Descriptor
4 Compute dominant orientation

[Adopted from: F. Flores-Mangas]

Sanja Fidler CSC420: Intro to Image Understanding 11 / 57

SIFT Descriptor
5 Compute a 128 dimensional descriptor: 4 × 4 grid, each cell is a
histogram of 8 orientation bins relative to dominant orientation

[Adopted from: F. Flores-Mangas]

Sanja Fidler CSC420: Intro to Image Understanding 11 / 57
SIFT Descriptor: Computing the Feature Vector
Compute the orientations relative to the dominant orientation

[Adopted from: F. Flores-Mangas]

Sanja Fidler CSC420: Intro to Image Understanding 12 / 57

SIFT Descriptor: Computing the Feature Vector
Compute the orientations relative to the dominant orientation

[Adopted from: F. Flores-Mangas]

Sanja Fidler CSC420: Intro to Image Understanding 12 / 57

SIFT Descriptor: Computing the Feature Vector
Compute the orientations relative to the dominant orientation
Form a 4 × 4 grid. For each grid cell compute a histogram of orientations for
8 orientation bins spaced apart by 45◦

[Adopted from: F. Flores-Mangas]

Sanja Fidler CSC420: Intro to Image Understanding 12 / 57
SIFT Descriptor: Computing the Feature Vector
Compute the orientations relative to the dominant orientation
Form a 4 × 4 grid. For each grid cell compute a histogram of orientations for
8 orientation bins spaced apart by 45◦

[Adopted from: F. Flores-Mangas]

Sanja Fidler CSC420: Intro to Image Understanding 12 / 57
SIFT Descriptor: Computing the Feature Vector
Compute the orientations relative to the dominant orientation
Form a 4 × 4 grid. For each grid cell compute a histogram of orientations for
8 orientation bins spaced apart by 45◦
Form the 128 dimensional feature vector

[Adopted from: F. Flores-Mangas]

Sanja Fidler CSC420: Intro to Image Understanding 12 / 57
SIFT Descriptor: Post-processing

The resulting 128 non-negative values form a raw version of the

SIFT descriptor vector.
To reduce the effects of contrast or gain (additive variations are
already removed by the gradient), the 128-D vector is normalized to
unit length: fi = fi /||fi ||

Sanja Fidler CSC420: Intro to Image Understanding 13 / 57

SIFT Descriptor: Post-processing

The resulting 128 non-negative values form a raw version of the

SIFT descriptor vector.
To reduce the effects of contrast or gain (additive variations are
already removed by the gradient), the 128-D vector is normalized to
unit length: fi = fi /||fi ||
To further make the descriptor robust to other photometric
variations, values are clipped to 0.2 and the resulting vector is once
again renormalized to unit length.

Sanja Fidler CSC420: Intro to Image Understanding 13 / 57

SIFT Descriptor: Post-processing

The resulting 128 non-negative values form a raw version of the

SIFT descriptor vector.
To reduce the effects of contrast or gain (additive variations are
already removed by the gradient), the 128-D vector is normalized to
unit length: fi = fi /||fi ||
To further make the descriptor robust to other photometric
variations, values are clipped to 0.2 and the resulting vector is once
again renormalized to unit length.
Great engineering effort!

Sanja Fidler CSC420: Intro to Image Understanding 13 / 57

SIFT Descriptor: Post-processing

The resulting 128 non-negative values form a raw version of the

SIFT descriptor vector.
To reduce the effects of contrast or gain (additive variations are
already removed by the gradient), the 128-D vector is normalized to
unit length: fi = fi /||fi ||
To further make the descriptor robust to other photometric
variations, values are clipped to 0.2 and the resulting vector is once
again renormalized to unit length.
Great engineering effort!
What is SIFT invariant to?

Sanja Fidler CSC420: Intro to Image Understanding 13 / 57

SIFT Descriptor: Post-processing

The resulting 128 non-negative values form a raw version of the

SIFT descriptor vector.
To reduce the effects of contrast or gain (additive variations are
already removed by the gradient), the 128-D vector is normalized to
unit length: fi = fi /||fi ||
To further make the descriptor robust to other photometric
variations, values are clipped to 0.2 and the resulting vector is once
again renormalized to unit length.
Great engineering effort!
What is SIFT invariant to?

Sanja Fidler CSC420: Intro to Image Understanding 13 / 57

Properties of SIFT
Invariant to:
Scale
Rotation
Partially invariant to:
Illumination changes (sometimes even day vs. night)
Camera viewpoint (up to about 60 degrees of out-of-plane rotation)
Occlusion, clutter (why?)
Also important:
Fast and efficient – can run in real time
Lots of code available

Sanja Fidler CSC420: Intro to Image Understanding 14 / 57

Examples

Figure: Matching in day / night under viewpoint change

[Source: S. Seitz]
Sanja Fidler CSC420: Intro to Image Understanding 15 / 57
Example

Figure: NASA Mars Rover images with SIFT feature matches

[Source: N. Snavely]

Sanja Fidler CSC420: Intro to Image Understanding 16 / 57

PCA-SIFT

The dimensionality of SIFT is pretty high, i.e., 128D for each keypoint
Reduce the dimensionality using linear dimensionality reduction
In this case, principal component analysis (PCA)
Use 10D or so descriptor

[Source: R. Urtasun]

Sanja Fidler CSC420: Intro to Image Understanding 17 / 57

Gradient location-orientation histogram (GLOH)
Developed by Mikolajczyk and Schmid (2005): variant of SIFT that uses a
log-polar binning structure instead of the four quadrants.
The spatial bins are 11, and 15, with eight angular bins (except for the
central region), for a total of 17 spatial bins and 16 orientation bins.
The 272D histogram is then projected onto a 128D descriptor using PCA
trained on a large database.

[Source: R. Szeliski]
Sanja Fidler CSC420: Intro to Image Understanding 18 / 57
Other Descriptors

SURF
DAISY
LBP
HOG
Shape Contexts
Color Histograms

Sanja Fidler CSC420: Intro to Image Understanding 19 / 57

Local Features
Detection: Identify the interest points.
Description: Extract feature descriptor around each interest point.
Matching: Determine correspondence between descriptors in two views.

[Source: K. Grauman]
Sanja Fidler CSC420: Intro to Image Understanding 20 / 57
 Image Features:

Matching the Local Descriptors

Sanja Fidler CSC420: Intro to Image Understanding 21 / 57

Matching the Local Descriptors
Once we have extracted keypoints and their descriptors, we want to match the
features between pairs of images.
Ideally a match is a correspondence between a local part of the object on
one image to the same local part of the object in another image
How should we compute a match?

Figure: Images from K. Grauman

Sanja Fidler CSC420: Intro to Image Understanding 22 / 57

Matching the Local Descriptors
Once we have extracted keypoints and their descriptors, we want to match the
features between pairs of images.
Ideally a match is a correspondence between a local part of the object on
one image to the same local part of the object in another image
How should we compute a match?

Figure: Images from K. Grauman

Sanja Fidler CSC420: Intro to Image Understanding 22 / 57

Matching the Local Descriptors
Simple: Compare them all, compute Euclidean distance

Sanja Fidler CSC420: Intro to Image Understanding 23 / 57

Matching the Local Descriptors
Simple: Compare them all, compute Euclidean distance

Sanja Fidler CSC420: Intro to Image Understanding 23 / 57

Matching the Local Descriptors
Find closest match (min distance). How do we know if match is reliable?

Sanja Fidler CSC420: Intro to Image Understanding 23 / 57

Matching the Local Descriptors
Find also the second closest match. Match reliable if first distance “much”
smaller than second distance

Sanja Fidler CSC420: Intro to Image Understanding 23 / 57

Matching the Local Descriptors
Compute the ratio:
∗
||fi − f 0 i ||
φi =
||fi − f 0 ∗∗
i ||
∗ ∗∗
where f 0 i is the closest and f 0 i second closest match to fi .

Sanja Fidler CSC420: Intro to Image Understanding 23 / 57

Which Threshold to Use?
Setting the threshold too high results in too many false positives, i.e.,
incorrect matches being returned.
Setting the threshold too low results in too many false negatives, i.e., too
many correct matches being missed

Figure: Images from R. Szeliski

Sanja Fidler CSC420: Intro to Image Understanding 24 / 57
Which Threshold to Use?
Threshold ratio of nearest to 2nd nearest descriptor
Typically: φi < 0.8

Figure: Images from D. Lowe

[Source: K. Grauman]
Sanja Fidler CSC420: Intro to Image Understanding 25 / 57
Applications of Local Invariant Features

Wide baseline stereo

Motion tracking
Panorama stitching
Mobile robot navigation
3D reconstruction
Recognition
Retrieval

[Source: K. Grauman]
Sanja Fidler CSC420: Intro to Image Understanding 26 / 57
Wide Baseline Stereo

[Source: T. Tuytelaars]

Sanja Fidler CSC420: Intro to Image Understanding 27 / 57

Recognizing the Same Object

[Source: K. Grauman]
Sanja Fidler CSC420: Intro to Image Understanding 28 / 57
Motion Tracking

Figure: Images from J. Pilet

Sanja Fidler CSC420: Intro to Image Understanding 29 / 57

Now What

Now we know how to extract scale and rotation invariant features

We even know how to match features across images
Can we use this to find Waldo in an even more sneaky scenario?

Sanja Fidler CSC420: Intro to Image Understanding 30 / 57

Now What

Now we know how to extract scale and rotation invariant features

We even know how to match features across images
Can we use this to find Waldo in an even more sneaky scenario?

template

Waldo on the road

Sanja Fidler CSC420: Intro to Image Understanding 30 / 57
Now What

Now we know how to extract scale and rotation invariant features

We even know how to match features across images
Can we use this to find Waldo in an even more sneaky scenario?

template

He comes closer... We know how to solve this

Sanja Fidler CSC420: Intro to Image Understanding 30 / 57
Now What

Now we know how to extract scale and rotation invariant features

We even know how to match features across images
Can we use this to find Waldo in an even more sneaky scenario?

template

Someone takes a (weird) picture of him!

Sanja Fidler CSC420: Intro to Image Understanding 30 / 57
Find My DVD!
More interesting: If we have DVD covers (e.g., from Amazon), can
we match them to DVDs in real scenes?

Sanja Fidler CSC420: Intro to Image Understanding 31 / 57

 Matching Planar Objects In New
Viewpoints

Sanja Fidler CSC420: Intro to Image Understanding 32 / 57

What Kind of Transformation Happened To My DVD?

Sanja Fidler CSC420: Intro to Image Understanding 33 / 57

What Kind of Transformation Happened To My DVD?
Rectangle goes to a parallelogram (almost but not really, but let’s believe
that for now)

Sanja Fidler CSC420: Intro to Image Understanding 34 / 57

All 2D Linear Transformations
Linear transformations are combinations of
Scale,
Rotation
Shear
Mirror
 0   
x a b x
=
y0 c d y

[Source: N. Snavely]

Sanja Fidler CSC420: Intro to Image Understanding 35 / 57

All 2D Linear Transformations
Properties of linear transformations:
Origin maps to origin
Lines map to lines

Sanja Fidler CSC420: Intro to Image Understanding 36 / 57

All 2D Linear Transformations
Properties of linear transformations:
Origin maps to origin
Lines map to lines
Parallel lines remain parallel

Sanja Fidler CSC420: Intro to Image Understanding 36 / 57

All 2D Linear Transformations
Properties of linear transformations:
Origin maps to origin
Lines map to lines
Parallel lines remain parallel
Ratios are preserved

Sanja Fidler CSC420: Intro to Image Understanding 36 / 57

All 2D Linear Transformations
Properties of linear transformations:
Origin maps to origin
Lines map to lines
Parallel lines remain parallel
Ratios are preserved
Closed under composition

Sanja Fidler CSC420: Intro to Image Understanding 36 / 57

All 2D Linear Transformations
Properties of linear transformations:
Origin maps to origin
Lines map to lines
Parallel lines remain parallel
Ratios are preserved
Closed under composition
 0     
x a b e f i j x
=
y0 c d g h k l y

Sanja Fidler CSC420: Intro to Image Understanding 36 / 57

All 2D Linear Transformations
Properties of linear transformations:
Origin maps to origin
Lines map to lines
Parallel lines remain parallel
Ratios are preserved
Closed under composition
 0     
x a b e f i j x
=
y0 c d g h k l y

What about the translation?

[Source: N. Snavely]
Sanja Fidler CSC420: Intro to Image Understanding 36 / 57
All 2D Linear Transformations
Properties of linear transformations:
Origin maps to origin
Lines map to lines
Parallel lines remain parallel
Ratios are preserved
Closed under composition
 0     
x a b e f i j x
=
y0 c d g h k l y

What about the translation?

[Source: N. Snavely]
Sanja Fidler CSC420: Intro to Image Understanding 36 / 57
Affine Transformations
Affine transformations are combinations of
Linear transformations, and
Translations  0     
x a b x e
= +
y0 c d y f
same as:
 
 0   x
x a b e  
= y
y0 c d f
1

Sanja Fidler CSC420: Intro to Image Understanding 37 / 57

Affine Transformations
Affine transformations are combinations of
Linear transformations, and
Translations  0     
x a b x e
= +
y0 c d y f
same as:
 
 0   x
x a b e  
= y
y0 c d f
1

Sanja Fidler CSC420: Intro to Image Understanding 37 / 57

Affine Transformations
Affine transformations are combinations of
Linear transformations, and
Translations  
 0   x
x a b e  
= y
y0 c d f
1
Properties of affine transformations:
Origin does not necessarily map to origin
Lines map to lines
Parallel lines remain parallel
Ratios are preserved
Closed under composition
Rectangles go to parallelograms
[Source: N. Snavely]
Sanja Fidler CSC420: Intro to Image Understanding 38 / 57
2D Image Tranformations

These transformations are a nested set of groups

Closed under composition and inverse is a member

Sanja Fidler CSC420: Intro to Image Understanding 39 / 57

What Transformation Happened to My DVD?

Affine transformation approximates viewpoint changes for roughly

planar objects and roughly orthographic cameras (more about
these later in class)
DVD went affine!

Sanja Fidler CSC420: Intro to Image Understanding 40 / 57

Computing the (Affine) Transformation
Given a set of matches between images I and J
How can we compute the affine transformation A from I to J?
Find transform A that best agrees with the matches

[Source: N. Snavely]
Sanja Fidler CSC420: Intro to Image Understanding 41 / 57
Computing the (Affine) Transformation
Given a set of matches between images I and J
How can we compute the affine transformation A from I to J?
Find transform A that best agrees with the matches

[Source: N. Snavely]
Sanja Fidler CSC420: Intro to Image Understanding 41 / 57
Computing the Affine Transformation

Let (xi , yi ) be a point on the reference (model) image, and (xi0 , yi0 ) its match
in the test image
An affine transformation A maps (xi , yi ) to (xi0 , yi0 ):
 
 0   xi
xi a b e  
= yi
yi0 c d f
1
We can rewrite this into a simple linear system:
 
a
 b   0 
 

xi yi 0 0 1 0   c  = xi0

0 0 xi yi 0 1   d 
 yi
 e 
f
Sanja Fidler CSC420: Intro to Image Understanding 42 / 57
Computing the Affine Transformation

Let (xi , yi ) be a point on the reference (model) image, and (xi0 , yi0 ) its match
in the test image
An affine transformation A maps (xi , yi ) to (xi0 , yi0 ):
 
 0   xi
xi a b e  
= yi
yi0 c d f
1
We can rewrite this into a simple linear system:
 
a
 b   0 
 

xi yi 0 0 1 0   c  = xi0

0 0 xi yi 0 1   d 
 yi
 e 
f
Sanja Fidler CSC420: Intro to Image Understanding 42 / 57
Computing the Affine Transformation
But we have many matches:
 

..
 a 
..

.  b  .
xi0
    
 xi yi 0 0 1 0 
 c   
=
yi0
  
 0 0 xi yi 0 1   d   
  
.. ..
 
 e 
. .
| {z } f | {z }
P P0
| {z }
a

For each match we have two more equations

How many matches do we need to compute A?

Sanja Fidler CSC420: Intro to Image Understanding 43 / 57

Computing the Affine Transformation
But we have many matches:
 

..
 a 
..

.  b  .
xi0
    
 xi yi 0 0 1 0 
 c   
=
yi0
  
 0 0 xi yi 0 1   d   
  
.. ..
 
 e 
. .
| {z } f | {z }
P P0
| {z }
a

For each match we have two more equations

How many matches do we need to compute A?

Sanja Fidler CSC420: Intro to Image Understanding 43 / 57

Computing the Affine Transformation
But we have many matches:
 

..
 a 
..

.  b  .
xi0
    
 xi yi 0 0 1 0 
 c   
=
yi0
  
 0 0 xi yi 0 1   d   
  
.. ..
 
 e 
. .
| {z } f | {z }
P P0
| {z }
a

For each match we have two more equations

How many matches do we need to compute A?
6 parameters → 3 matches
But the more, the better (more reliable)
How do we compute A?

Sanja Fidler CSC420: Intro to Image Understanding 43 / 57

Computing the Affine Transformation
But we have many matches:
 

..
 a 
..

.  b  .
xi0
    
 xi yi 0 0 1 0 
 c   
=
yi0
  
 0 0 xi yi 0 1   d   
  
.. ..
 
 e 
. .
| {z } f | {z }
P P0
| {z }
a

For each match we have two more equations

How many matches do we need to compute A?
6 parameters → 3 matches
But the more, the better (more reliable)
How do we compute A?

Sanja Fidler CSC420: Intro to Image Understanding 43 / 57

Computing the Affine Transformation
 

..
 a 
..

.  b  .
xi0
    
 xi yi 0 0 1 0   c   
=
yi0
  
 0 0 xi yi 0 1   d   
  
.. ..
 
 e 
. .
| {z } f | {z }
P P0
| {z }
a

If we have 3 matches, then computing A is really easy:

a = P−1 P0

If we have more than 3, then we do least-squares estimation:

min ||Pa − P0 ||22
a,b,··· ,f

Which has a closed form solution:

a = (PT P)−1 PT P0

Sanja Fidler CSC420: Intro to Image Understanding 44 / 57

Image Alignment Algorithm: Affine Case
Given images I and J
1 Compute image features for I and J
2 Match features between I and J
3 Compute affine transformation A between I and J using least squares on the
set of matches

Is there a problem with this?

[Source: N. Snavely]

Sanja Fidler CSC420: Intro to Image Understanding 45 / 57

Image Alignment Algorithm: Affine Case
Given images I and J
1 Compute image features for I and J
2 Match features between I and J
3 Compute affine transformation A between I and J using least squares on the
set of matches

Is there a problem with this?

[Source: N. Snavely]

Sanja Fidler CSC420: Intro to Image Understanding 45 / 57

Robustness

!"#$%&'()

%*$%&'()

[Source: N. Snavely]

Sanja Fidler CSC420: Intro to Image Understanding 46 / 57

Simple Case
Lets consider a simpler example ... Fit a line to the points below!

!"#$%&'()*+,)-)%+.&),#),/&0&)1-,-2#+.,0) 3&-0,)045-"&0)6,)

How can we fix this?

[Source: N. Snavely]

Sanja Fidler CSC420: Intro to Image Understanding 47 / 57

Simple Case
Lets consider a simpler example ... Fit a line to the points below!

!"#$%&'()*+,)-)%+.&),#),/&0&)1-,-2#+.,0) 3&-0,)045-"&0)6,)

How can we fix this?

[Source: N. Snavely]

Sanja Fidler CSC420: Intro to Image Understanding 47 / 57

Simple Idea: RANSAC

Take the minimal number of points to compute what we want. In the line
example, two points (in our affine example, three matches)

By “take” we mean choose at random from all points

Sanja Fidler CSC420: Intro to Image Understanding 48 / 57

Simple Idea: RANSAC

Take the minimal number of points to compute what we want. In the line
example, two points (in our affine example, three matches)

By “take” we mean choose at random from all points

Fit a line to the selected pair of points

Sanja Fidler CSC420: Intro to Image Understanding 48 / 57

Simple Idea: RANSAC

Take the minimal number of points to compute what we want. In the line
example, two points (in our affine example, three matches)

By “take” we mean choose at random from all points

Fit a line to the selected pair of points

Count the number of all points that “agree” with the line: We call the
agreeing points inliers

Sanja Fidler CSC420: Intro to Image Understanding 48 / 57

Simple Idea: RANSAC

Take the minimal number of points to compute what we want. In the line
example, two points (in our affine example, three matches)

By “take” we mean choose at random from all points

Fit a line to the selected pair of points

Count the number of all points that “agree” with the line: We call the
agreeing points inliers

“Agree” = within a small distance of the line

Sanja Fidler CSC420: Intro to Image Understanding 48 / 57

Simple Idea: RANSAC

Take the minimal number of points to compute what we want. In the line
example, two points (in our affine example, three matches)

By “take” we mean choose at random from all points

Fit a line to the selected pair of points

Count the number of all points that “agree” with the line: We call the
agreeing points inliers

“Agree” = within a small distance of the line

Repeat this many times, remember the number of inliers for each trial

Among several trials, select the one with the largest number of inliers

This procedure is called RAndom SAmple Consensus

Sanja Fidler CSC420: Intro to Image Understanding 48 / 57

Simple Idea: RANSAC

Take the minimal number of points to compute what we want. In the line
example, two points (in our affine example, three matches)

By “take” we mean choose at random from all points

Fit a line to the selected pair of points

Count the number of all points that “agree” with the line: We call the
agreeing points inliers

“Agree” = within a small distance of the line

Repeat this many times, remember the number of inliers for each trial

Among several trials, select the one with the largest number of inliers

This procedure is called RAndom SAmple Consensus

Sanja Fidler CSC420: Intro to Image Understanding 48 / 57

RANSAC for Line Fitting Example

1 Randomly select minimal

subset of points
2 Hypothesize a model

[Source: R. Raguram]

Sanja Fidler CSC420: Intro to Image Understanding 49 / 57

RANSAC for Line Fitting Example

1 Randomly select minimal

subset of points
2 Hypothesize a model
3 Compute error function

[Source: R. Raguram]

Sanja Fidler CSC420: Intro to Image Understanding 49 / 57

RANSAC for Line Fitting Example

1 Randomly select minimal

subset of points
2 Hypothesize a model
3 Compute error function
4 Select points consistent
with model

[Source: R. Raguram]

Sanja Fidler CSC420: Intro to Image Understanding 49 / 57

RANSAC for Line Fitting Example

1 Randomly select minimal

subset of points
2 Hypothesize a model
3 Compute error function
4 Select points consistent
with model
5 Repeat hypothesize and
verify loop

[Source: R. Raguram]

Sanja Fidler CSC420: Intro to Image Understanding 49 / 57

RANSAC for Line Fitting Example

1 Randomly select minimal

subset of points
2 Hypothesize a model
3 Compute error function
4 Select points consistent
with model
5 Repeat hypothesize and
verify loop

[Source: R. Raguram]

Sanja Fidler CSC420: Intro to Image Understanding 49 / 57

RANSAC for Line Fitting Example

1 Randomly select minimal

subset of points
2 Hypothesize a model
3 Compute error function
4 Select points consistent
with model
5 Repeat hypothesize and
verify loop
6 Choose model with
largest set of inliers

[Source: R. Raguram]

Sanja Fidler CSC420: Intro to Image Understanding 49 / 57

RANSAC for Line Fitting Example

1 Randomly select minimal

subset of points
2 Hypothesize a model
3 Compute error function
4 Select points consistent
with model
5 Repeat hypothesize and
verify loop
6 Choose model with
largest set of inliers

[Source: R. Raguram]

Sanja Fidler CSC420: Intro to Image Understanding 49 / 57

Translations

[Source: N. Snavely]
Sanja Fidler CSC420: Intro to Image Understanding 50 / 57
RAndom SAmple Consensus

!"#"$%&!"#&'(%$)&(%&*(+,-'.&$-/+%&$"%$#&'&

[Source: N. Snavely]
Sanja Fidler CSC420: Intro to Image Understanding 51 / 57
RAndom SAmple Consensus

!"#"$%&'()%*"+&,'%$*&'%&+'(-),.&$)/(%&!"#!$%&&

[Source: N. Snavely]
Sanja Fidler CSC420: Intro to Image Understanding 51 / 57
RAndom SAmple Consensus

!"#$"#%#&'%#()*+,)-.*%/0#&%#&'%&01&'+#%*"23'(%.4%0*,0'(+%

[Source: N. Snavely]
Sanja Fidler CSC420: Intro to Image Understanding 51 / 57
RANSAC
All the inliers will agree with each other on the translation vector; the
(hopefully small) number of outliers will (hopefully) disagree with each other
RANSAC only has guarantees if there are < 50% outliers

Sanja Fidler CSC420: Intro to Image Understanding 52 / 57

RANSAC
All the inliers will agree with each other on the translation vector; the
(hopefully small) number of outliers will (hopefully) disagree with each other
RANSAC only has guarantees if there are < 50% outliers
”All good matches are alike; every bad match is bad in its own way.” –
[Tolstoy via Alyosha Efros]

[Source: N. Snavely]

Sanja Fidler CSC420: Intro to Image Understanding 52 / 57

RANSAC
All the inliers will agree with each other on the translation vector; the
(hopefully small) number of outliers will (hopefully) disagree with each other
RANSAC only has guarantees if there are < 50% outliers
”All good matches are alike; every bad match is bad in its own way.” –
[Tolstoy via Alyosha Efros]

[Source: N. Snavely]

Sanja Fidler CSC420: Intro to Image Understanding 52 / 57

RANSAC
Inlier threshold related to the amount of noise we expect in inliers
Often model noise as Gaussian with some standard deviation (e.g., 3 pixels)

Sanja Fidler CSC420: Intro to Image Understanding 53 / 57

RANSAC
Inlier threshold related to the amount of noise we expect in inliers
Often model noise as Gaussian with some standard deviation (e.g., 3 pixels)
Number of rounds related to the percentage of outliers we expect, and the
probability of success we’d like to guarantee

Sanja Fidler CSC420: Intro to Image Understanding 53 / 57

RANSAC
Inlier threshold related to the amount of noise we expect in inliers
Often model noise as Gaussian with some standard deviation (e.g., 3 pixels)
Number of rounds related to the percentage of outliers we expect, and the
probability of success we’d like to guarantee
Suppose there are 20% outliers, and we want to find the correct answer with
99% probability

Sanja Fidler CSC420: Intro to Image Understanding 53 / 57

RANSAC
Inlier threshold related to the amount of noise we expect in inliers
Often model noise as Gaussian with some standard deviation (e.g., 3 pixels)
Number of rounds related to the percentage of outliers we expect, and the
probability of success we’d like to guarantee
Suppose there are 20% outliers, and we want to find the correct answer with
99% probability
How many rounds do we need?

[Source: R. Urtasun]

Sanja Fidler CSC420: Intro to Image Understanding 53 / 57

RANSAC
Inlier threshold related to the amount of noise we expect in inliers
Often model noise as Gaussian with some standard deviation (e.g., 3 pixels)
Number of rounds related to the percentage of outliers we expect, and the
probability of success we’d like to guarantee
Suppose there are 20% outliers, and we want to find the correct answer with
99% probability
How many rounds do we need?

[Source: R. Urtasun]

Sanja Fidler CSC420: Intro to Image Understanding 53 / 57

How many rounds?
Sufficient number of trials S must be tried.
Let p be the probability that any given correspondence is valid and P be the
total probability of success after S trials.

Sanja Fidler CSC420: Intro to Image Understanding 54 / 57

How many rounds?
Sufficient number of trials S must be tried.
Let p be the probability that any given correspondence is valid and P be the
total probability of success after S trials.
The likelihood in one trial that all k random samples are inliers is p k

Sanja Fidler CSC420: Intro to Image Understanding 54 / 57

How many rounds?
Sufficient number of trials S must be tried.
Let p be the probability that any given correspondence is valid and P be the
total probability of success after S trials.
The likelihood in one trial that all k random samples are inliers is p k
The likelihood that S such trials will all fail is

1 − P = (1 − p k )S

Sanja Fidler CSC420: Intro to Image Understanding 54 / 57

How many rounds?
Sufficient number of trials S must be tried.
Let p be the probability that any given correspondence is valid and P be the
total probability of success after S trials.
The likelihood in one trial that all k random samples are inliers is p k
The likelihood that S such trials will all fail is

1 − P = (1 − p k )S

The required minimum number of trials is

log(1 − P)
S=
log(1 − p k )

Sanja Fidler CSC420: Intro to Image Understanding 54 / 57

How many rounds?
Sufficient number of trials S must be tried.
Let p be the probability that any given correspondence is valid and P be the
total probability of success after S trials.
The likelihood in one trial that all k random samples are inliers is p k
The likelihood that S such trials will all fail is

1 − P = (1 − p k )S

The required minimum number of trials is

log(1 − P)
S=
log(1 − p k )

The number of trials grows quickly with the number of sample points used.

Sanja Fidler CSC420: Intro to Image Understanding 54 / 57

How many rounds?
Sufficient number of trials S must be tried.
Let p be the probability that any given correspondence is valid and P be the
total probability of success after S trials.
The likelihood in one trial that all k random samples are inliers is p k
The likelihood that S such trials will all fail is

1 − P = (1 − p k )S

The required minimum number of trials is

log(1 − P)
S=
log(1 − p k )

The number of trials grows quickly with the number of sample points used.

[Source: R. Urtasun]
Sanja Fidler CSC420: Intro to Image Understanding 54 / 57
How many rounds?
Sufficient number of trials S must be tried.
Let p be the probability that any given correspondence is valid and P be the
total probability of success after S trials.
The likelihood in one trial that all k random samples are inliers is p k
The likelihood that S such trials will all fail is

1 − P = (1 − p k )S

The required minimum number of trials is

log(1 − P)
S=
log(1 − p k )

The number of trials grows quickly with the number of sample points used.

[Source: R. Urtasun]
Sanja Fidler CSC420: Intro to Image Understanding 54 / 57
RANSAC pros and cons
Pros
Simple and general
Applicable to many different problems

Sanja Fidler CSC420: Intro to Image Understanding 55 / 57

RANSAC pros and cons
Pros
Simple and general
Applicable to many different problems
Often works well in practice

Sanja Fidler CSC420: Intro to Image Understanding 55 / 57

RANSAC pros and cons
Pros
Simple and general
Applicable to many different problems
Often works well in practice
Cons
Parameters to tune

Sanja Fidler CSC420: Intro to Image Understanding 55 / 57

RANSAC pros and cons
Pros
Simple and general
Applicable to many different problems
Often works well in practice
Cons
Parameters to tune
Sometimes too many iterations are required

Sanja Fidler CSC420: Intro to Image Understanding 55 / 57

RANSAC pros and cons
Pros
Simple and general
Applicable to many different problems
Often works well in practice
Cons
Parameters to tune
Sometimes too many iterations are required
Can fail for extremely low inlier ratios

Sanja Fidler CSC420: Intro to Image Understanding 55 / 57

RANSAC pros and cons
Pros
Simple and general
Applicable to many different problems
Often works well in practice
Cons
Parameters to tune
Sometimes too many iterations are required
Can fail for extremely low inlier ratios
We can often do better than brute-force sampling
[Source: N. Snavely, slide credit: R. Urtasun]

Sanja Fidler CSC420: Intro to Image Understanding 55 / 57

RANSAC pros and cons
Pros
Simple and general
Applicable to many different problems
Often works well in practice
Cons
Parameters to tune
Sometimes too many iterations are required
Can fail for extremely low inlier ratios
We can often do better than brute-force sampling
[Source: N. Snavely, slide credit: R. Urtasun]

Sanja Fidler CSC420: Intro to Image Understanding 55 / 57

Ransac Verification

[Source: K. Grauman, slide credit: R. Urtasun]

Sanja Fidler CSC420: Intro to Image Understanding 56 / 57
Summary – Stuff You Need To Know

To match image I and J under affine transformation:

Compute scale and rotation invariant keypoints in both images
Compute a (rotation invariant) feature vector in each keypoint (e.g., SIFT)
Match all features in I to all features in J
For each feature in reference image I find closest match in J
If ratio between closest and second closest match is < 0.8, keep match
Do RANSAC to compute affine transformation A:
Select 3 matches at random
Compute A
Compute the number of inliers
Repeat
Find A that gave the most inliers

Sanja Fidler CSC420: Intro to Image Understanding 57 / 57