CS 294-167 Spring 2022

Geometry and Learning for 3D Vision

Yi Ma

UC Berkeley

Invitation to 3D vision
 Lecture Seven

Multiple-View Reconstruction from Scene Knowledge

“Mathematics is the art of giving the same name to different things.”

– Henri Poincare

3/11/22
 INTRODUCTION: Scene knowledge and symmetry

Symmetry is ubiquitous in man-made or natural

environments
3/11/22
 INTRODUCTION: Scene knowledge and symmetry
Parallelism (vanishing point) Orthogonality

Congruence Self-similarity

3/11/22
 INTRODUCTION: Wrong assumptions

Ames room illusion

Ames Room, by Adelbert Ames, Jr., 1946

3/11/22
 INTRODUCTION: Wrong assumptions

Perspective projection Orthogonal projection

Necker’s cube illusion

3/11/22
 INTRODUCTION: Related literature
Mathematics: Hilbert 18th problem: Fedorov 1891, Hilbert 1901,
Bieberbach 1910, George Polya 1924, Weyl 1952

Cognitive Science: Figure “goodness”: Gestalt theorists (1950s),

Garner’74, Chipman’77, Marr’82, Palmer’91’99…

Computer Vision (isotropic & homogeneous textures): Gibson’50,

Witkin’81, Garding’92’93, Malik&Rosenholtz’97, Leung&Malik’97…

Detection & Recognition (2D & 3D): Morola’89, Forsyth’91,

Vetter’94, Mukherjee’95, Zabrodsky’95, Basri & Moses’96,
Kanatani’97, Sun’97, Yang, Hong, Ma’02

Reconstruction (from single view): Kanade’81, Fawcett’93,

Rothwell’93, Zabrodsky’95’97, van Gool et.al.’96, Carlsson’98,
Svedberg and Carlsson’99, Francois and Medioni’02, Huang,
Yang, Hong, Ma’02,03, Zhou and Ma’2021.

3/11/22
 Multiple-View Reconstruction from Scene Knowledge

SYMMETRY & MULTIPLE-VIEW GEOMETRY

• Fundamental types of symmetry
• Equivalent views
• Symmetry based reconstruction

MUTIPLE-VIEW MULTIPLE-OBJECT ALIGNMENT

• Scale alignment: adjacent objects in a single view
• Scale alignment: same object in multiple views

ALGORITHMS & EXAMPLES

• Building 3-D geometric models with symmetry
• Symmetry extraction, detection, and matching
• Camera calibration

SUMMARY: Problems and future work

3/11/22
 SYMMETRY & MUTIPLE-VIEW GEOMETRY

• Why does an image of a symmetric object give away its structure?

• Why does an image of a symmetric object give away its pose?

• What else can we get from an image of a symmetric object?

3/11/22
 Equivalent views from rotational symmetry

90O

3/11/22
 Equivalent views from reflectional symmetry

3/11/22
 Equivalent views from translational symmetry

3/11/22
 GEOMETRY FOR SINGLE IMAGES – Symmetric Structure

Definition. A set of 3-D features S is called a symmetric structure

if there exists a nontrivial subgroup G of E(3) that acts on it such
that for every g in G, the map

is an (isometric) automorphism of S. We say the structure S has a

group symmetry G.

3/11/22
 GEOMETRY FOR SINGLE IMAGES – Multiple “Equivalent” Views

3/11/22
 GEOMETRY FOR SINGLE IMAGES – Symmetric Rank Condition

Solving g0 from Lyapunov equations:

with g’i and gi known.

3/11/22
 THREE TYPES OF SYMMETRY – Reflective Symmetry

Pr

3/11/22
 THREE TYPES OF SYMMETRY – Rotational Symmetry

3/11/22
 THREE TYPES OF SYMMETRY – Translatory Symmetry

3/11/22
 SINGLE-VIEW GEOMETRY WITH SYMMETRY – Ambiguities

“(a+b)-parameter” means there are an a-parameter family of ambiguity in

R0 of g0 and a b-parameter family of ambiguity in T0 of g0.

Pr
Pr
N

3/11/22
 Symmetry-based reconstruction (reflection)

2 (1)
Reflectional symmetry
1(2)

3 (4) 4 (3)

Virtual camera-camera

Isosceles trapezoid
3/11/22
 Symmetry-based reconstruction

2 (1) Epipolar constraint

1(2)

3 (4) 4 (3)

Homography

3/11/22
 Symmetry-based reconstruction (algorithm)

2 pairs of symmetric image points

Recover essential matrix or homography

Decompose or to
obtain

Solve Lyapunov equation

to obtain and then .

3/11/22
 Symmetry-based reconstruction (reflection)

3/11/22
 Symmetry-based reconstruction (rotation)

3/11/22
 Symmetry-based reconstruction (translation)

3/11/22
 ALIGNMENT OF MULTIPLE SYMMETRIC OBJECTS

3/11/22
 Correct scales within a single image
Pick the image of a point
on the intersection line

3/11/22
 Correct scale within a single image

3/11/22
 Correct scales across multiple images

3/11/22
 Correct scales across multiple images

3/11/22
 Correct scales across multiple images

3/11/22
 Image alignment after scales corrected

3/11/22
 ALGORITHM: Building 3-D geometric models

1. Specify symmetric objects and correspondence

3/11/22
 Building 3-D geometric models

2. Recover camera poses and scene structure

3/11/22
 Building 3-D geometric models

3. Obtain 3-D model and rendering with images

3/11/22
APPLICATIONS – Photo Editing (Beckman Institute, UIUC)
APPLICATIONS – Photo Editing (Beckman Institute, UIUC)
APPLICATIONS – Photo Editing (Beckman Institute, UIUC)
APPLICATIONS – Human Face Features from a Single View

INS canonical views (3/4 view)

APPLICATIONS – Symmetric Curves and Surfaces
APPLICATIONS – Unmanned Aerial Vehicles (UAVs)

Rate: 10Hz
Accuracy: 5cm, 4o

Berkeley Aerial Robot (BEAR) Project

 ALGORITHM: Symmetry detection and matching

Extract, detect, match symmetric objects across

images, and recover the camera poses.

3/11/22
 Segmentation & polygon fitting

1.Color-based segmentation (mean shift)

2. Polygon fitting

3/11/22
 Symmetry verification & recovery
3. Symmetry verification (rectangles,…)

4. Single-view recovery

3/11/22
 Symmetry-based matching

5. Find the only one set of camera poses that

are consistent with all symmetry objects

3/11/22
 MATCHING OF SYMMETRY CELLS – Graph Representation
Cell in image 1 # of possible matching

(1,2,1, g)
3 1
2 Cell in image 2 Camera transformation

1
3 2

The problem of finding the largest set of

36 possible matchings matching cells is equivalent to the problem
of finding the maximal complete subgraphs
(cliques) in the matching graph.
3/11/22
 Camera poses and 3-D recovery
Side view Top view Generic view

Length ratio Reconstruction Ground truth

Whiteboard 1.506 1.51
Table 1.003 1.00

3/11/22
 Multiple-view matching and recovery (Ambiguities)

3/11/22
 Multiple-view matching and recovery (Ambiguities)

3/11/22
 ALGORITHM: Calibration from symmetry

Calibrated homography

Uncalibrated homography

(vanishing point)

3/11/22
 ALGORITHM: Calibration from symmetry

Calibration with a rig is also simplified: we only need

to know that there are sufficient symmetries, not
necessarily the 3-D coordinates of points on the rig.

3/11/22
 SUMMARY: Multiple-View Geometry + Symmetry

Multiple (perspective) images = multiple-view rank condition

Single image + symmetry = “multiple-view” rank condition

Multiple images + symmetry = rank condition + scale correction

Matching + symmetry = rank condition + scale correction

+ clique identification

3/11/22
 SUMMARY
Multiple-view 3-D reconstruction in presence of symmetry

• Symmetry based algorithms are accurate, robust, and simple.

• Methods are baseline independent and object centered.

• Alignment and matching can and should take place in 3-D space.

• Camera self-calibration and calibration are simplified and linear.

Related applications

• Using symmetry to overcome occlusion.

• Reconstruction and rendering with non-symmetric area.

• Large scale 3-D map building of man-made environments.

3/11/22