CS231A Course Notes 3: Epipolar Geometry

Kenji Hata and Silvio Savarese

1 Introduction
Previously, we have seen how to compute the intrinsic and extrinsic param-
eters of a camera using one or more views using a typical camera calibration
procedure or single view metrology. This process culminated in deriving
properties about the 3D world from one image. However, in general, it is not
possible to recover the entire structure of the 3D world from just one image.
This is due to the intrinsic ambiguity of the 3D to the 2D mapping: some
information is simply lost.

Figure 1: A single picture such as this picture of a man holding up the

Leaning Tower of Pisa can result in ambiguous scenarios. Multiple views of
the same scene help us resolve these potential ambiguities.

For example, in Figure 1, we may be initially fooled to believe that the

man is holding up the Leaning Tower of Pisa. Only by careful inspection
can we tell that this is not the case and merely an illusion based on the
projection of different depths onto the image plane. However, if we were able
to view this scene from a completely different angle, this illusion immediately
disappears and we would instantly figure out the correct scene layout.

1
 The focus of these lecture notes is to show how having knowledge of geom-
etry when multiple cameras are present can be extremely helpful. Specifically,
we will first focus on defining the geometry involved in two viewpoints and
then present how this geometry can aid in further understanding the world
around us.

2 Epipolar Geometry

Figure 2: The general setup of epipolar geometry. The gray region is the
epipolar plane. The orange line is the baseline, while the two blue lines are
the epipolar lines.

Often in multiple view geometry, there are interesting relationships be-

tween the multiple cameras, a 3D point, and that point’s projections in each
of the camera’s image plane. The geometry that relates the cameras, points
in 3D, and the the corresponding observations is referred to as the epipolar
geometry of a stereo pair.
As illustrated in Figure 2, the standard epipolar geometry setup involves
two cameras observing the same 3D point P , whose projection in each of
the image planes is located at p and p0 respectively. The camera centers
are located at O1 and O2 , and the line between them is referred to as the
baseline. We call the plane defined by the two camera centers and P the
epipolar plane. The locations of where the baseline intersects the two image

2
Figure 3: An example of epipolar lines and their corresponding points drawn
on an image pair.

planes are known as the the epipoles e and e0 . Finally, the lines defined by
the intersection of the epipolar plane and the two image planes are known as
the epipolar lines. The epipolar lines have the property that they intersect
the baseline at the respective epipoles in the image plane.

Figure 4: When the two image planes are parallel, then the epipoles e and
e0 are located at infinity. Notice that the epipolar lines are parallel to the u
axis of each image plane.

An interesting case of epipolar geometry is shown in Figure 4, which

occurs when the image planes are parallel to each other. When the image
planes are parallel to each other, then the epipoles e and e0 will be located at
infinity since the baseline joining the centers O1 , O2 is parallel to the image
planes. Another important byproduct of this case is that the epipolar lines
are parallel to an axis of each image plane. This case is especially useful

3
and will be covered in greater detail in the subsequent section on image
rectification.
In real world situations, however, we are not given the exact location of
the 3D location P , but can determine its projection in one of the image planes
p. We also should be able to know the cameras locations, orientations, and
camera matrices. What can we do with this knowledge? With the knowledge
of camera locations O1 , O2 and the image point p, we can define the epipolar
plane. With this epipolar plane, we can then determine the epipolar lines1 .
By definition, P ’s projection into the second image p0 must be located on the
epipolar line of the second image. Thus, a basic understanding of epipolar
geometry allows us to create a strong constraint between image pairs without
knowing the 3D structure of the scene.

Figure 5: The setup for determining the essential and fundamental matrices,
which help map points and epipolar lines across views.

We will now try to develop seamless ways to do map points and epipolar
lines across views. If we take the setup given in the original epipolar geometry
framework (Figure 5), then we shall further define M and M 0 to be the camera
projection matrices that map 3D points into their respective 2D image plane
locations. Let us assume that the world reference system is associated to the
first camera with the second camera offset first by a rotation R and then by
a translation T . This specifies the camera projection matrices to be:

M 0 = K0 R T
   
M =K I 0 (1)
1
This means that epipolar lines can be determined by just knowing the camera centers
O1 , O2 and a point in one of the images p

4
3 The Essential Matrix
In the simplest case, let us assume that we have canonical cameras, in
which K = K 0 = I. This reduces Equation 1 to

M0 = R T
   
M= I 0 (2)
Furthermore, this means that the location of p0 in the first camera’s refer-
ence system is RT p0 − RT T . Since the vectors RT p0 − RT T and RT T lie in the
epipolar plane, then if we take the cross product of RT T × (RT p0 − RT T ) =
RT T × RT p0 = RT (T × p0 ), we will get a vector normal to the epipolar
plane. This also means that p, which lies in the epipolar plane is normal to
RT (T × p0 ), giving us the constraint that their dot product is zero:

(RT (T × p0 ))T p = 0
(3)
(T × p0 )T Rp = 0

From linear algebra, we can introduce a different and compact expression

for the cross product: we can represent the cross product between any two
vectors a and b as a matrix-vector multiplication:
  
0 −az ay bx
a × b =  az 0 −ax  by  = [a× ]b (4)
−ay ax 0 bz

Combining this expression with Equation 3, we can convert the cross

product term into matrix multiplication, giving

([T× ]p0 )T Rp = 0
p0T [T× ]T Rp = 0 (5)
0T
p [T× ]Rp = 0

The matrix E = [T× ]R is known as the Essential Matrix, creating a com-

pact expression for the epipolar constraint:

p0T Ep = 0 (6)

The Essential matrix is a 3 × 3 matrix that contains 5 degrees of freedom. It

has rank 2 and is singular.
The Essential matrix is useful for computing the epipolar lines associated
with p and p0 . For instance, `0 = Ep gives the epipolar line in the image of
camera 2. Similarly ` = E T p0 gives the epipolar line in the image of camera 1.
Other interesting properties of the essential matrix is that its matrix product

5
with the epipoles equate to zero: Ee = E T e0 = 0. Because for any point x
(other than e) in the image of camera 1, the corresponding epipolar line in
the image of camera 2, l0 = Ex, contains the epipole e0 . Thus e0 satisfies
e0T (Ex) = (e0T E)x = 0 for all the x, so e0T E = 0. Similarly Ee = 0.

4 The Fundamental Matrix

Although, we derived a relationship between p and p0 when we have canon-
ical cameras, we should be able to find a more general expression when the
cameras are no longer canonical. Recall that gives us the projection matrices:

M 0 = K0 R T
   
M =K I 0 (7)
First, we must define pc = K −1 p and p0c = K 0−1 p0 to be the projections of
P to the corresponding camera images if the cameras were canonical. Recall
that in the canonical case:

p0T
c [T× ]Rpc = 0 (8)

By substituting in the values of pc and p0c , we get

p0T K 0−T [T× ]RK −1 p = 0 (9)

The matrix F = K 0−T [T× ]RK −1 is known as the Fundamental Matrix,

which acts similar to the Essential matrix from the previous section but
also encodes information about the camera matrices K, K 0 and the relative
translation T and rotation R between the cameras. Therefore, it is also
useful in computing the epipolar lines associated with p and p0 , even when the
camera matrices K, K 0 and the transformation R, T are unknown. Similar to
the Essential matrix, we can compute the epipolar lines `0 = F p and ` = F T p0
from just the Fundamental matrix and the corresponding points. One main
difference between the Fundamental matrix and the Essential matrix is that
the Fundamental matrix contains 7 degrees of freedom, compared to the
Essential matrix’s 5 degrees of freedom.
But how is the Fundamental matrix useful? Like the Essential matrix, if
we know the Fundamental matrix, then simply knowing a point in an image
gives us an easy constraint (the epipolar line) of the corresponding point in
the other image. Therefore, without knowing the actual position of P in 3D
space, or any of the extrinsic or intrinsic characteristics of the cameras, we
can establish a relationship between any p and p0 .

6
Figure 6: Corresponding points are drawn in the same color on each of the
respective images.

4.1 The Eight-Point Algorithm

Still, the assumption that we can have the Fundamental matrix, which is
defined by a matrix product of the camera parameters, seems rather large.
However, it is possible to estimate the Fundamental matrix given two images
of the same scene and without knowing the extrinsic or intrinsic parameters of
the camera. The method we discuss for doing so is known as the Eight-Point
Algorithm, which was proposed by Longuet-Higgins in 1981 and extended
by Hartley in 1995. As the title suggests, the Eight-Point Algorithm assumes
that a set of at least 8 pairs of corresponding points between two images is
available.
Each correspondence pi = (ui , vi , 1) and p0i = (u0i , vi0 , 1) gives us the epipo-
lar constraint p0T
i F pi = 0. We can reformulate the constraint as follows:
 
F11
F12 
 
F13 
 
 F21 
 
 0 0 0 0 0 0
ui ui vi ui ui ui vi vi vi vi ui vi 1  F22  = 0
 (10)
F23 
 
F31 
 
F32 
F33

Since this constraint is a scalar equation, it only constrains one degree of

freedom. Since we can only know the Fundamental matrix up to scale, we

7
require eight of these constraints to determine the Fundamental matrix:
 
 0 0 0 0 0 0
 F11
u1 u1 v1 u1 u1 u1 v1 v1 v1 v1 u1 v1 1  
u2 u02 v2 u02 u02 u2 v20 v2 v20 v20 u2 v2 1 F12 
u3 u03 v3 u03 u03 u3 v30 v3 v30 v30 u3 v3 1 F13 
  
u4 u04 v4 u04 u04 u4 v40 v4 v40 v40 u4 v4 1 F21 
  
u5 u05 v5 u05 u05 u5 v50 v5 v50 v50 u5 v5 1 F22  = 0 (11)
  
 F
u6 u0 v6 u0 u0 u6 v 0 v6 v 0 v 0 u6 v6 1  23 
 
6 6 6 6 6 6
u7 u07 v7 u07 u07 u7 v70 v7 v70 v70 u7 v7 1 F31 
  
F32 
u8 u08 v8 u08 u08 u8 v80 v8 v80 v80 u8 v8 1
F33
This can be compactly written as

Wf = 0 (12)

where W is an N × 9 matrix derived from N ≥ 8 correspondences and f is

the values of the Fundamental matrix we desire.
In practice, it often is better to use more than eight correspondences and
create a larger W matrix because it reduces the effects of noisy measurements.
The solution to this system of homogeneous equations can be found in the
least-squares sense by Singular Value Decomposition (SVD), as W is rank-
deficient. SVD will give us a estimate of the Fundamental matrix F̂ , which
may have full rank. However, we know that the true Fundamental matrix
has rank 2. Therefore, we should look for a solution that is the best rank-2
approximation of F̂ . To do so, we solve the following optimization problem:

minimize kF − F̂ kF
F (13)
subject to det F = 0

This problem is solved again by SVD, where F̂ = U ΣV T , then the best

rank-2 approximation is found by
 
Σ1 0 0
F = U  0 Σ2 0 V T (14)
0 0 0

4.2 The Normalized Eight-Point Algorithm

In practice, the standard least-squares approach to the Eight-Point Algo-
rithm is not precise. Often, the distance between a point pi and its corre-
sponding epipolar line `i = F T p0 will be very large, usually on the scale of

8
10+ pixels. To reduce this error, we can consider a modified version of the
Eight-Point Algorithm called the Normalized Eight-Point Algorithm.
The main problem of the standard Eight-Point Algorithm stems from the
fact that W is ill-conditioned for SVD. For SVD to work properly, W should
have one singular value equal to (or near) zero, with the other singular values
being nonzero. However, the correspondences pi = (ui , vi , 1) will often have
extremely large values in the first and second coordinates due to the pixel
range of a modern camera (i.e. pi = (1832, 1023, 1)). If the image points
used to construct W are in a relatively small region of the image, then each
the vectors for pi and p0i will generally be very similar. Consequently, the
constructed W matrix will have one very large singular value, with the rest
relatively small.
To solve this problem, we will normalize the points in the image before
constructing W . This means we pre-condition W by applying both a trans-
lation and scaling on the image coordinates such that two requirements are
satisfied. First, the origin of the new coordinate system should be located
at the centroid of the image points (translation). Second, the mean square
distance of the transformed image points from the origin should be 2 pix-
els (scaling). We can compactly represent this process by a transformation
matrices T, T 0 that translate by the centroid and scale by the scaling factor
2
mean distance
for each respective image.
Afterwards, we normalize the coordinates:

qi = T pi qi0 = T 0 p0i (15)

Using the new, normalized coordinates, we can compute the new Fq using
the regular least-squares Eight Point Algorithm. However, the matrix Fq is
the fundamental matrix for the normalized coordinates. For it to be usable
on regular coordinate space, we need to de-normalize it, giving

F = T 0T Fq T (16)

Ultimately, this new Fundamental matrix F gives good results in real-world

applications.

5 Image Rectification
Recall that an interesting case for epipolar geometry occurs when two images
are parallel to each other. Let us first compute the Essential matrix E in the
case of parallel image planes. We can assume that the two cameras have the
same K and that there is no relative rotation between the cameras (R = I.

9
In this case, let us assume that there is only a translation along the x axis,
giving T = (Tx , 0, 0). This gives
 
0 0 0
E = [T× ]R = 0 0 −Tx  (17)
0 Tx 0

Once E is known, we can find the directions of the epipolar lines associ-
ated with points in the image planes. Let us compute the direction of the
epipolar line ` associated with point p0 :
   0  
0 0 0 u 0
` = E T p0 = 0 0 Tx  v 0  =  Tx  (18)
0 −Tx 0 1 −Tx v 0

We can see that the direction of ` is horizontal, as is the direction of `0 , which

is computed in a similar manner.

Figure 7: The process of image rectification involves computing two homo-

graphies that we can apply to a pair of images to make them parallel.

If we use the epipolar constraint p0T Ep = 0, then we arrive at the fact

that v = v 0 , demonstrating that p and p0 share the same v-coordinate. Con-
sequently, there exists a very straightforward relationship between the cor-
responding points. Therefore, rectification, or the process of making any
two given images parallel, becomes useful when discerning the relationships
between corresponding points in images.

10
Figure 8: The rectification problem setup: we compute two homographies
that we can apply to the image planes to make the resulting planes parallel.

Rectifying a pair of images does not require knowledge of the two camera
matrices K, K 0 or the relative transformation R, T between them. Instead, we
can use the Fundamental matrix estimated by the Normalized Eight Point
Algorithm. Upon getting the Fundamental matrix, we can compute the
epipolar lines `i and `0i for each correspondence pi and p0i .
From the set of epipolar lines, we can then estimate the epipoles e and e0 of
each image. This is because we know that the epipole lies in the intersection
of all the epipolar lines. In the real world, due to noisy measurements, all the
epipolar lines will not intersect in a single point. Therefore, computing the
epipole can be found by minimizing the least squared error of fitting a point
to all the epipolar lines. Recall that each epipolar line can be represented
as a vector ` such that all points on the line (represented in homogeneous
coordinates) are in the set {x|`T x = 0}. If we define each epipolar line as
 T
`i = `i,1 `i,2 `i,3 , then we can we formulate a linear system of equations
and solve using SVD to find the epipole e:
 
`T1
 .. 
 . e = 0 (19)
T
`n

After finding the epipoles e and e0 , we will most likely notice that they
are not points at infinity along the horizontal axis. If they were, then, by
definition, the images would already be parallel. Thus, we gain some insight
into how to make the images parallel: can we find a homography to map an
epipole e to infinity along the horizontal axis? Specifically, this means that we
want to find a pair of homographies H1 , H2 that we can apply to the images
to map the epipoles to infinity. Let us start by finding a homography H2 that

11
maps the second epipole e0 to a point on the horizontal axis at infinity (f, 0, 0).
Since there are many possible choices for this homography, we should try to
choose something reasonable. One condition that leads to good results in
practice is to insist that the homography acts like a transformation that
applies a translation and rotation on points near the center of the image.
The first step in achieving such a transformation is to translate the second
image such that the center is at (0, 0, 1) in homogeneous coordinates. We
can do so by applying the translation matrix

1 0 − width
 
2
T = 0 1 − height2
 (20)
0 0 1

After applying the translation, we apply a rotation to place the epipole on

the horizontal axis at some point (f, 0, 1). If the translated epipole T e0 is
located at homogeneous coordinates (e01 , e02 , 1), then the rotation applied is
e0 e0
 
α √ 021 02 α √ 022 02 0
e1 +e2 e1 +e2
e02 e01
 
R = −α √
 α √ 0  (21)
e02 02
1 +e2 e02 02
1 +e2

0 0 1

where α = 1 if e01 ≥ 0 and α = −1 otherwise. After applying this rotation,

notice that given any point at (f, 0, 1), bringing it to a point at infinity on
the horizontal axis (f, 0, 0) only requires applying a transformation
 
1 0 0
G =  0 1 0 (22)
1
−f 0 1

After applying this transformation, we finally have an epipole at infinity, so

we can translate back to the regular image space. Thus, the homography H2
that we apply on the second image to rectify it is

H2 = T −1 GRT (23)

Now that a valid H2 is found, we need to find a matching homography H1

for the first image. We do so by finding a transformation H1 that minimizes
the sum of square distances between the corresponding points of the images
X
arg min kH1 pi − H2 p0i k2 (24)
H1
i

12
Although the derivation2 is outside the scope of this class, we can actually
prove that the matching H1 is of the form:

H1 = HA H2 M (25)

where F = [e]× M and  

a1 a2 a3
HA =  0 1 0  (26)
0 0 1
with (a1 , a2 , a3 ) composing the elements of a certain vector a that will be
computed later.
First, we need to know what M is. An interesting property of any 3 × 3
skew-symmetric matrix A is A = A3 up to scale. Because any cross product
matrix [e]× is skew-symmetric and that we can only know the Fundamental
matrix F up to scale, then

F = [e]× M = [e]× [e]× [e]× M = [e]× [e]× F (27)

By grouping the right terms, we can find that

M = [e]× F (28)

Notice that if the columns of M were added by any scalar multiple of e, then
the F = [e]× M still holds up to scale. Therefore, the more general case of
defining M is
M = [e]× F + ev T (29)
 
for some vector v. In practice, defining M by setting v T = 1 1 1 works
very well.
To finally solve for H1 , we need to compute the a values of HA . Recall that
we want to find a H1 , H2 to minimize the problem posed in Equation 24. Since
we already know the value of H2 and M , then we can substitute p̂i = H2 M pi
and p̂0i = H2 p0i and the minimization problem becomes
X
arg min kHA p̂i − p̂0i k2 (30)
HA
i

In particular, if we let p̂i = (x̂i , ŷi , 1) and p̂0i = (x̂0i , ŷi0 , 1), then the mini-
mization problem can be replaced by:
X
arg min (a1 x̂i + a2 ŷi + a3 − x̂0i )2 + (ŷi − ŷi0 )2 (31)
a
i

2
If you are interested in the details, please see Chapter 11 of Hartley & Zisserman’s
textbook Multiple View Geometry

13
Since ŷi − ŷi0 is a constant value, the minimization problem further reduces
to X
arg min (a1 x̂i + a2 ŷi + a3 − x̂0i )2 (32)
a
i

Ultimately, this breaks down into solving a least-squares problem W a = b

for a where    
x̂1 ŷ1 1 x̂01
W =
 .. 
b =  ... 
 
(33)
. 
x̂n ŷn 1 x̂0n
After computing a, we can compute HA and finally H1 . Thus, we gen-
erated the homographies H1 , H2 to rectify any image pair given a few corre-
spondences.

14