Globally Optimal Consensus Set Maximization Through Rotation Search
Globally Optimal Consensus Set Maximization Through Rotation Search
i
y
i
(2a)
s.t. y
i
(Ru
i
, v
i
) y
i
, i = 1 . . . N (2b)
y
i
{0, 1}, i = 1 . . . N (2c)
R SO(3) (2d)
When y
i
= 1, the inlier/outlier constraint at Eq (2b) corresponds to Eq (1b).
When y
i
= 0, the constraint is simply eliminated (i.e. always true since > 0).
Globally Optimal Consensus Set Maximization Through Rotation Search 3
System (2) still constitutes a challenging optimization problem. First, it con-
tains two families of unknowns: binary variables (y) and continuous geometric
model parameters (R). Second, it is non-linear because of the distance and the
unknowns (y and R) are multiplied to each other at Eq (2b). Third, it is non-
convex. Finally, fullling the rotation constraint at Eq (2d) requires specic care.
3 Rotation Search Algorithm
In this section, we explain our algorithm to solve system (2), and thus system (1),
in a globally optimal way. It is based on branch-and-bound framework and a ro-
tation space search. We start by a brief introduction to branch-and-bound, then
explain how to compute the bounds and nally present our search procedure.
3.1 Branch-and-bound
Branch-and-bound (noted B&B) is a general framework for global optimization
[13]. The main idea is the following: the denition space of the model to be
estimated is partitioned into smaller sub-spaces that are iteratively discarded or
rened. When it is certain, via a feasibility test, that a sub-space does not contain
the optimal solution, then this sub-space is discarded. On the contrary, if it might
contain the solution, then it is partitioned into smaller sub-spaces. Iteratively,
the size of the sub-spaces decreases and the estimated solution converges to the
optimal solution. The process stops when the desired accuracy level is reached.
B&B has gained popularity quite recently in computer vision [1420].
3.2 Bounds
The feasibility test is generally conducted by the computation of bounds, that is
the best (and optionally the worst) solution that can be obtained in a sub-space.
In the following, we explain how to compute the bounds for our application.
Computation By adopting the angle-axis representation, any rotations can be
represented by a point in a ball B
into a set of cubes. The question becomes: how to compute the bounds of
the inlier cardinality that can be obtained for any rotations in a cube D. The
answer is: the lower and upper bounds are the solutions of the systems (3) and
(4) respectively, where
R is the (known) rotation corresponding to the center of
the cube D and is the half length side of D (cf proof below).
max
y
N
i
y
i
(3a)
s.t. y
i
(
Ru
i
, v
i
) y
i
(3b)
y
i
{0, 1}, i = 1 . . . N (3c)
max
y
N
i
y
i
(4a)
s.t. y
i
(
Ru
i
, v
i
) y
i
(
+
3
)
(4b)
y
i
{0, 1}, i = 1 . . . N (4c)
4 Jean-Charles Bazin, Yongduek Seo and Marc Pollefeys
The unknowns of the systems (3) and (4) are the binary y. It is interesting to
note that an Integer Programming solver, which is computationally expensive,
does not have to be run. Indeed, we can simply nd wether y
i
is 1 or 0 by
checking whether a matching pair veries the inlier distance constraint or not:
(
Ru
i
, v
i
) for the lower bound at system (3)
(
Ru
i
, v
i
) +
Ru, Ru))
Ru
i
, v
i
) (R
opt
u
i
, v
i
) +(
Ru
i
, R
opt
u
i
) (5)
+
3 (6)
It means that if a match is an inlier by R
opt
with respect to the residual threshold
, then it is also an inlier by
R with respect to the residual threshold +
3.
Therefore system (4) provides at least c inliers, i.e. u c.
Finally, B&B is guaranteed to converge to the globally optimal solution when
the bounds are valid [13]. Since we proved the bounds are valid, it proves, in
turn, that our proposed B&B-based algorithm is globally optimal.
3.3 Search procedure
Our B&B procedure is formalized in Alg1 using a breadth-rst search. The cube
list L is simply initialized with the rotation ball B
repeat
Subdivision ( /2) of each cube D
i
of L
for each cube D
i
of L do
Get the rotation R
D
i
of the cube center
Compute the lower l
i
and upper u
i
bounds (cf Section 3.2)
end for
l
= max
i
l
i
, i
= arg max
i
u
i
, u
= u
i
, R
= R
D
i
)
Return: R
. Figure 3-(b,c)
illustrate the registration of the two sets by the estimated rotation (without any
additional renement, e.g. [22], in order to show the quality of our B&B solu-
tion). Figure 4 compares the number of inliers obtained by the proposed method
1
in rare cases, it might happen that the optimal number of inliers (returned by our
algorithm) is higher than the number of synthesized inliers because some outliers
might create a new dominant model, especially for high outlier ratio.
2
http://graphics.stanford.edu/data/3Dscanrep
Globally Optimal Consensus Set Maximization Through Rotation Search 7
1 2 3 4 5 6
0
10
20
30
Convergence of bounds
Iterations
U
p
p
e
r
a
n
d
l
o
w
e
r
b
o
u
n
d
s
upper
lower
(a)
5 10 15 20 25 30
0
5
10
15
20
Nb of detected inliers
p
e
r
c
e
n
t
a
g
e
o
f
e
x
p
e
r
i
m
e
n
t
s
distribution of the nb of inliers
(b)
1 2 3 4 5 6
10
4
10
2
10
0
10
2
Convergence of volume
Iterations
r
e
m
a
i
n
i
n
g
v
o
l
u
m
e
(
i
n
%
)
(c)
1 2 3 4 5 6
10
1
10
2
10
3
evolution of the nb of cubes
Iterations
N
u
m
b
e
r
o
f
c
u
b
e
s
(d)
Fig. 2. Experimental results on synthesized data. (a): convergence of the bounds of
the proposed B&B approach. (b): distribution of the number of inliers detected by
RANSAC on the same problem instance. (c): convergence of the volume of the search
space. (d): evolution of the number of cubes.
and RANSAC. Our method converges to 8 inliers, whereas RANSAC spans be-
tween 2 and 8 inliers. RANSAC performs poorly on this dataset because of the
large proportion of outliers (8 inliers out of 1191 points) and the data noise. In
contrast our approach can handle this challenging case without any diculties.
4.3 Real data: panorama stitching
We now present results in the context of panoramic image stitching [5]. When
the scene is far away, two images are related by the innite homography H
.
When the intrinsic calibration matrix K is known [1], H
reduces to a rotation
that we aim to estimate. To obtain correspondences between the two images, we
extracted and matched SIFT features [23] and applied K
1
to compute the asso-
ciated ray matches. Figure 5-a illustrates the inlier and outlier matches found by
the proposed approach. Once the inliers are found, any reprojection error mini-
mization method can be applied [5, 22]. The nal stitching result is shown in Fig-
ure 5-b. Figure 5-(c,d) compare the number of inliers obtained by the proposed
8 Jean-Charles Bazin, Yongduek Seo and Marc Pollefeys
(a)
(b) (c)
Fig. 3. Registration result between two 3D point sets of the Bunny model. (a): inlier
(green lines) and outlier (red lines) matches detected by the proposed approach. (b,c):
3D views after 3D registration of the two 3D point sets by the estimated rotation,
without any additional renement.
2 4 6 8
0
20
40
60
80
100
120
Convergence of bounds
Iterations
U
p
p
e
r
a
n
d
l
o
w
e
r
b
o
u
n
d
s
upper
lower
1 2 3 4 5 6 7 8
0
20
40
60
80
Nb of detected inliers
p
e
r
c
e
n
t
a
g
e
o
f
e
x
p
e
r
i
m
e
n
t
s
distribution of the nb of inliers
Fig. 4. Experimental results for the Bunny model. (a): convergence of the bounds of
the proposed B&B approach. (b): distribution of the number of inliers by RANSAC.
approach and RANSAC. Our approach nds 654 inliers while RANSAC spans
between 651 and 654 inliers and manages to detect 654 inliers in less than 1%
of the RANSAC runs.
Globally Optimal Consensus Set Maximization Through Rotation Search 9
(a)
(b)
2 4 6 8
0
200
400
600
800
Convergence of bounds
Iterations
U
p
p
e
r
a
n
d
l
o
w
e
r
b
o
u
n
d
s
upper
lower
(c)
651 652 653 654
0
20
40
60
80
100
Nb of detected inliers
p
e
r
c
e
n
t
a
g
e
o
f
e
x
p
e
r
i
m
e
n
t
s
distribution of the nb of inliers
(d)
Fig. 5. Result of panoramic image stitching. (a): inlier (green lines) and outlier (red
lines) matches detected by our approach between the two input images. (b): stitching
of the two images to build a panoramic view. (c): convergence of the bounds of our
B&B approach. (d): distribution of the number of inliers detected by RANSAC.
10 Jean-Charles Bazin, Yongduek Seo and Marc Pollefeys
4.4 Real data: line clustering
Given a set of input lines, line clustering aims to nd which line belongs to which
vanishing point (VP) and estimate the VPs [24]. Very recently, Bazin et al. [25]
proposed a method to maximize the number of clustered lines in this context. It
provides interesting results but is based on interval analysis [26], which provides
loose bounds and, similarly to the big-M method [11], is subject to numerical
inaccuracies. The formulation of system (2) can be modied so that our frame-
work can be applied for line clustering. In the following, we consider intrinsically
calibrated images and orthogonal vanishing points (Manhattan world [6, 27]).
Mathematical formulation: let n
i
be the normal vector of the i
th
input
line with i = 1 . . . N and v
j
be the j
th
VP with j = 1 . . . M where 1 M 3. A
line-VP pair composed of (n
i
, v
j
) is considered an inlier if
(n
i
, v
j
)
2
.
We say that n
i
is matched (clustered) to v
j
and we note (n
i
, v
j
) M
I
. The
consensus set maximization can be written:
max
M
I
,{v}
card(M
I
) (7a)
s.t.
(n
i
, v
j
)
2
, (i, j) M
I
M (7b)
and v
j
= 1, j and v
T
j
v
k
= 0, j, k with j = k (7c)
As observed by Bazin et al. [6], this system can be simplied by representing
the set of M orthogonal VPs by a single rotation R: v
j
= Re
j
where e
j
represents
the j
th
axis of the canonical basis, i.e. e
1
= (1, 0, 0), e
2
= (0, 1, 0) and e
3
=
(0, 0, 1). Additionally, we introduce the clustering variables y
i,j
set to 1 if the
i
th
line is clustered to the j
th
VP and to 0 otherwise. This permits to dene the
following system:
max
y,R
N
i
M
j
y
i,j
(8a)
s.t. y
i,j
(n
i
, Re
j
)
2
y
i,j
, i = 1 . . . N, j = 1 . . . M (8b)
y
i,j
{0, 1}, i = 1 . . . N, j = 1 . . . M (8c)
0
M
j
y
i,j
1, i = 1 . . . N (8d)
R SO(3) (8e)
The (optional) constraint at Eq (8d) enforces that a line can belong to at most
one VP. To solve system (8), we employ a strategy similar to Section 3.2 where
the rotation cube center is used. Concretely, the variables y
i,j
are set to 0 or 1
by simply testing the constraints:
(n
i
,
Re
j
)
2
(n
i
,
Re
j
)
2
= arg min
j
(n
i
, Re
j
)
2