ITC Faculty / Department of Earth Observation Science / Earth Observation with UAVs

STRUCTURE FROM
MOTION (SFM)

SINGLE IMAGE ORIENTATION

SLIDES BY B. ALSADIK
CONTENTS
• Camera settings
• Pinhole camera model
• Collinearity equations
• Projection matrix
• Direct linear transformation
• Intrinsic camera matrix
• 3D Rotation
• Extrinsic parameters & camera pose
CAMERA TYPES
• RGB: DSLR, compact, MILC
• Multispectral
• Thermal

(A) DJI Zenmuse X7; (B) MAPIR Survey3 (also

avaliable in multispectral option); (C) PhaseOne iXU-
RS 1000; (D) Sony ILCE-QX1; (E) senseFly S.O.D.A

UAV thermal cameras

UAV Multispectral cameras

CAMERA SENSORS
• Sensor Size: Larger sensors will have better light
gathering ability at the same resolutions while smaller
sensors will need greater exposure times to achieve
the same effective outcome.
For UAV inspections it is important to have a high-
resolution camera, with a larger sensor which allows to:
• Provide more details about any point of the inspected
object.
• Increase image contrast to help getting better 3D
models and point clouds.
CAMERA SETTINGS
• Depth of field: Depth of field (DOF) refers to the
areas of the photograph both in front and behind
the focus point which remains ‘sharp’ (in focus).

• Aperture: Aperture refers to the size of the

opening in the lens that controls the amount of
light reaching the sensor. The relation between
aperture and the focal length is called f-number
or f-stop=focal length / aperture diameter Adjust
the aperture value helps to match lighting
conditions to effectively expose sensor and
obtain the sharpest images.

http://www.canonoutsideofauto.ca/play/
FIELD OF VIEW FOV

• Preferable cameras will have a quick autofocus and

short focal length so when you are above 20m
everything will be in the infinity focus region.

• Shorter focal length lenses provide a wider field of

view FOV but offer less magnification. On the other
hand, longer focal lengths offer a smaller FOV, but
provides larger magnification.

Try to experience the change of the focal length on the image FOV
https://camerasim.com/camerasim-free-web-app/
PINHOLE CAMERA MODEL
The pinhole camera model describes the mathematical relationship between
the coordinates of a 3D point and its projection onto the image plane

Please check the video describing the pinhole camera principle

https://www.youtube.com/watch?v=hhWVJ4SmkF0
 PINHOLE CAMERA GEOMETRY
Mathematically, the central perspective projection modeled by the
pinhole camera can be formulated using trigonometric proportion

� 𝑌𝑌,
𝑋𝑋, � 𝑍𝑍are
̅ the world coordinates.
𝑥𝑥, 𝑦𝑦, are the image coordinates.

From trigonometry:
𝑋𝑋� 𝑌𝑌�
𝑥𝑥 = 𝑓𝑓( � ) ; 𝑦𝑦 = 𝑓𝑓( �)
𝑍𝑍 𝑍𝑍
𝑂𝑂

• The distance from the optical center to the image plane

is the focal length 𝑓𝑓

� 𝑌𝑌,
• 𝑋𝑋, � and 𝑍𝑍̅ are the world coordinates to the origin 𝑝𝑝. 𝑝𝑝 = 𝑝𝑝𝑝𝑝𝑝𝑝𝑝𝑝𝑝𝑝𝑝𝑝𝑝𝑝𝑝𝑝𝑝𝑝 𝑝𝑝𝑝𝑝𝑝𝑝𝑝𝑝𝑝𝑝
located at the camera perspective center 𝑂𝑂 (lens). = 𝑖𝑖𝑖𝑖𝑖𝑖𝑖𝑖𝑖𝑖 𝑐𝑐𝑐𝑐𝑐𝑐𝑐𝑐𝑐𝑐𝑐𝑐 𝑤𝑤𝑤𝑤𝑤𝑤𝑤 𝑐𝑐𝑐𝑐𝑐𝑐𝑐𝑐𝑐𝑐𝑐𝑐𝑐𝑐𝑐𝑐𝑐𝑐𝑐𝑐 𝑢𝑢0 , 𝑣𝑣0
MATHEMATICAL MODELS OF THE PINHOLE
CAMERA
Collinearity equations is based on the geometric
condition that the object point 𝐴𝐴 𝑋𝑋𝐴𝐴 , 𝑌𝑌𝐴𝐴 , 𝑍𝑍𝐴𝐴 , its
projection on the image 𝑥𝑥𝑎𝑎 , 𝑦𝑦𝑎𝑎 and the camera
lens (𝑋𝑋𝑜𝑜 , 𝑌𝑌𝑜𝑜 , 𝑍𝑍𝑜𝑜 ) are all collinear (green dotted line)

Exterior orientation parameters

𝒓𝒓𝟏𝟏𝟏𝟏 𝒓𝒓𝟏𝟏𝟏𝟏 𝒓𝒓𝟏𝟏𝟏𝟏 𝑋𝑋𝑜𝑜
R= 𝒓𝒓𝟐𝟐𝟐𝟐 𝒓𝒓𝟐𝟐𝟐𝟐 𝒓𝒓𝟐𝟐𝟐𝟐 , T= 𝑌𝑌𝑜𝑜
𝒓𝒓𝟑𝟑𝟑𝟑 𝒓𝒓𝟑𝟑𝟑𝟑 𝒓𝒓𝟑𝟑𝟑𝟑 𝑍𝑍𝑜𝑜
𝑟𝑟𝑟𝑟𝑟𝑟𝑟𝑟𝑟𝑟𝑟𝑟𝑟𝑟𝑟𝑟 𝑖𝑖𝑖𝑖 𝑎𝑎 𝑓𝑓𝑓𝑓𝑓𝑓𝑓𝑓𝑓𝑓𝑓𝑓𝑓𝑓𝑓𝑓 𝑜𝑜𝑜𝑜(𝜔𝜔,𝜑𝜑,𝑘𝑘) 𝑡𝑡𝑡𝑡𝑡𝑡𝑡𝑡𝑡𝑡𝑡𝑡𝑡𝑡𝑡𝑡𝑡𝑡𝑡𝑡𝑡𝑡

𝒙𝒙𝒐𝒐 , 𝒚𝒚𝒐𝒐 , 𝒇𝒇: 𝒊𝒊𝒊𝒊𝒊𝒊𝒊𝒊𝒊𝒊𝒊𝒊𝒊𝒊𝒊𝒊 𝒑𝒑𝒑𝒑𝒑𝒑𝒑𝒑𝒑𝒑𝒑𝒑𝒑𝒑𝒑𝒑𝒑𝒑𝒑𝒑

WHAT IS EXTERIOR ORIENTATION?
• Both the camera position and angular orientation at
the moment of taking the image are called the exterior
orientation parameters (extrinsic).
• The extrinsic parameters represent a rigid
transformation from 3-D world coordinate system to
the 3-D camera’s coordinate system.
• There are six extrinsic parameters 3 for rotation R
and 3 for translation T.
• The determination of the extrinsic parameters will
define the camera pose.

Exterior orientation parameters

𝒓𝒓𝟏𝟏𝟏𝟏 𝒓𝒓𝟏𝟏𝟏𝟏 𝒓𝒓𝟏𝟏𝟏𝟏 𝑋𝑋𝑜𝑜
R= 𝒓𝒓𝟐𝟐𝟐𝟐 𝒓𝒓𝟐𝟐𝟐𝟐 𝒓𝒓𝟐𝟐𝟐𝟐 , T= 𝑌𝑌𝑜𝑜
𝒓𝒓𝟑𝟑𝟑𝟑 𝒓𝒓𝟑𝟑𝟑𝟑 𝒓𝒓𝟑𝟑𝟑𝟑 𝑍𝑍𝑜𝑜
𝑟𝑟𝑟𝑟𝑟𝑟𝑟𝑟𝑟𝑟𝑟𝑟𝑟𝑟𝑟𝑟 𝑖𝑖𝑖𝑖 𝑎𝑎 𝑓𝑓𝑓𝑓𝑓𝑓𝑓𝑓𝑓𝑓𝑓𝑓𝑓𝑓𝑓𝑓 𝑜𝑜𝑜𝑜(𝜔𝜔,𝜑𝜑,𝑘𝑘) 𝑡𝑡𝑡𝑡𝑡𝑡𝑡𝑡𝑡𝑡𝑡𝑡𝑡𝑡𝑡𝑡𝑡𝑡𝑡𝑡𝑡𝑡
WHAT IS INTERIOR ORIENTATION?
• The internal geometry of a camera as it existed at the moment of data capture is
defined by its interior orientation
• camera parameters are mainly the focal length 𝑓𝑓, principal point 𝑝𝑝. 𝑝𝑝 coordinates, and
lens distortion parameters.
• The camera parameters are represented by a special matrix called the intrinsic
camera matrix K.
• K matrix represents the transformation from image coordinates to pixel coordinates

𝑢𝑢 𝑥𝑥 𝑢𝑢 𝑓𝑓 0 𝑥𝑥0 𝑥𝑥
𝑣𝑣 = 𝐾𝐾 𝑦𝑦 𝑜𝑜𝑜𝑜 𝑣𝑣 = 0 𝑓𝑓 𝑦𝑦0 𝑦𝑦
1 1 1 0 0 1 1

𝒙𝒙𝒐𝒐 , 𝒚𝒚𝒐𝒐 , 𝒇𝒇: 𝒊𝒊𝒊𝒊𝒊𝒊𝒊𝒊𝒊𝒊𝒊𝒊𝒊𝒊𝒊𝒊 𝒑𝒑𝒑𝒑𝒑𝒑𝒑𝒑𝒑𝒑𝒑𝒑𝒑𝒑𝒑𝒑𝒑𝒑𝒑𝒑

EXTERIOR AND INTERIOR ORIENTATION
• The determination of the extrinsic parameters will define the camera pose.
• The determination of the intrinsic parameters will define the camera calibration
parameters.
𝑦𝑦
𝑢𝑢
Interior orientation

𝑣𝑣 𝑥𝑥
3D to 2D transformation
K matrix

Z
3D to 3D transformation
Rotation + Translation
Exterior orientation

WORLD X
SYSTEM
ROTATION IN 3D
• Understanding the 3D rotation is essential in studying the topic of
photogrammetry
• The rotation can be clockwise if a left-handed system is adopted or
anticlockwise if a right-handed system
• The orientation of the camera coordinate system with respect to the world
system can be represented by an orthonormal rotation matrix R
• Point 𝑎𝑎 can be transformed from 𝑥𝑥 𝑎𝑎 = 𝑋𝑋𝑎𝑎 𝑌𝑌𝑎𝑎 𝑍𝑍𝑎𝑎 𝑡𝑡 to point 𝑏𝑏 as 𝑥𝑥 𝑏𝑏 =
𝑋𝑋𝑏𝑏 𝑌𝑌𝑏𝑏 𝑍𝑍𝑏𝑏 𝑡𝑡
𝑥𝑥 𝑎𝑎 = 𝑅𝑅𝑎𝑎𝑎𝑎 𝑥𝑥 𝑏𝑏
ROTATION IN 3D
Euler angles are typically denoted as omega 𝜔𝜔, phi 𝜑𝜑, and kappa 𝑘𝑘. The 𝜔𝜔 angle is
applied around the 𝑋𝑋-axis, 𝜑𝜑 around the 𝑌𝑌-axis, and 𝑘𝑘 around the 𝑍𝑍-axis
𝑅𝑅𝑘𝑘

𝑅𝑅𝜔𝜔 𝑅𝑅𝜑𝜑
cos 𝜑𝜑 0 − sin 𝜑𝜑 cos 𝑘𝑘 sin 𝑘𝑘 0 1 0 0
𝑅𝑅𝜑𝜑 = � 0 1 0 �, 𝑅𝑅𝑘𝑘 = �− sin 𝑘𝑘 cos 𝑘𝑘 0� 𝑅𝑅𝜔𝜔 = �0 cos 𝜔𝜔 sin 𝜔𝜔 �
sin 𝜑𝜑 0 cos 𝜑𝜑 0 0 1 0 − sin 𝜔𝜔 cos 𝜔𝜔

- In the 𝑍𝑍𝑍𝑍𝑍𝑍 sequence, angles will be:

𝑐𝑐𝑐𝑐𝑐𝑐𝑐𝑐 𝑐𝑐𝑐𝑐𝑐𝑐𝑐𝑐 𝑐𝑐𝑐𝑐𝑐𝑐𝑐𝑐 𝑠𝑠𝑠𝑠𝑠𝑠𝑠𝑠 + 𝑠𝑠𝑠𝑠𝑠𝑠𝑠𝑠 𝑠𝑠𝑠𝑠𝑠𝑠𝑠𝑠 𝑐𝑐𝑐𝑐𝑐𝑐𝑐𝑐 𝑠𝑠𝑠𝑠𝑠𝑠𝑠𝑠 𝑠𝑠𝑠𝑠𝑠𝑠𝑠𝑠 − 𝑐𝑐𝑐𝑐𝑐𝑐𝑐𝑐 𝑠𝑠𝑠𝑠𝑠𝑠𝑠𝑠 𝑐𝑐𝑐𝑐𝑐𝑐𝑐𝑐
𝑅𝑅𝑍𝑍𝑍𝑍𝑍𝑍 = 𝑅𝑅𝑘𝑘 𝑅𝑅𝜑𝜑 𝑅𝑅𝜔𝜔 = �−𝑐𝑐𝑐𝑐𝑐𝑐𝑐𝑐 𝑠𝑠𝑠𝑠𝑠𝑠𝑠𝑠 𝑐𝑐𝑐𝑐𝑐𝑐𝑐𝑐 𝑐𝑐𝑐𝑐𝑐𝑐𝑐𝑐 − 𝑠𝑠𝑠𝑠𝑠𝑠𝑠𝑠 𝑠𝑠𝑠𝑠𝑠𝑠𝑠𝑠 𝑠𝑠𝑠𝑠𝑠𝑠𝑠𝑠 𝑠𝑠𝑠𝑠𝑠𝑠𝑠𝑠 𝑐𝑐𝑐𝑐𝑐𝑐𝑐𝑐 + 𝑐𝑐𝑐𝑐𝑐𝑐𝑐𝑐 𝑠𝑠𝑠𝑠𝑠𝑠𝑠𝑠 𝑠𝑠𝑠𝑠𝑠𝑠𝑠𝑠�
𝑠𝑠𝑠𝑠𝑠𝑠𝑠𝑠 −𝑠𝑠𝑠𝑠𝑠𝑠𝑠𝑠 𝑐𝑐𝑐𝑐𝑐𝑐𝑐𝑐 𝑐𝑐𝑐𝑐𝑐𝑐𝑐𝑐 𝑐𝑐𝑐𝑐𝑐𝑐𝑐𝑐
SPATIAL RESECTION – IMAGE POSE
• Spatial resection is the process of computing the six
exterior parameters.
𝒓𝒓𝟏𝟏𝟏𝟏 𝒓𝒓𝟏𝟏𝟏𝟏 𝒓𝒓𝟏𝟏𝟏𝟏 𝑋𝑋𝑜𝑜
R= 𝒓𝒓𝟐𝟐𝟐𝟐 𝒓𝒓𝟐𝟐𝟐𝟐 𝒓𝒓𝟐𝟐𝟐𝟐 , T= 𝑌𝑌𝑜𝑜
𝒓𝒓𝟑𝟑𝟑𝟑 𝒓𝒓𝟑𝟑𝟑𝟑 𝒓𝒓𝟑𝟑𝟑𝟑 𝑍𝑍𝑜𝑜
𝑟𝑟𝑟𝑟𝑟𝑟𝑟𝑟𝑟𝑟𝑟𝑟𝑟𝑟𝑟𝑟 𝑖𝑖𝑖𝑖 𝑎𝑎 𝑓𝑓𝑓𝑓𝑓𝑓𝑓𝑓𝑓𝑓𝑓𝑓𝑓𝑓𝑓𝑓 𝑜𝑜𝑜𝑜(𝜔𝜔,𝜑𝜑,𝑘𝑘) 𝑡𝑡𝑡𝑡𝑡𝑡𝑡𝑡𝑡𝑡𝑡𝑡𝑡𝑡𝑡𝑡𝑡𝑡𝑡𝑡𝑡𝑡

• Spatial resection can be applied using the nonlinear

collinearity equations.
• a minimum of three reference points should be
measured in the single image.
PROJECTION MATRIX
• Projection matrix specifies how a pinhole camera applies a projective mapping of 3D points
in the world to 2D points in an image.
• We can represent the ideal pinhole camera shown previously using homogeneous
coordinates by:

𝑋𝑋� �
𝑥𝑥 = 𝑓𝑓 ⎫ 𝑥𝑥 𝑓𝑓 0 0 0 𝑋𝑋
̅
𝑍𝑍 in homogenous coordinates→ �𝑦𝑦� = � 𝑌𝑌�
0 𝑓𝑓 0 0� � ̅ �
𝑌𝑌� ⎬ 1 𝑍𝑍
𝑦𝑦 = 𝑓𝑓 ⎭ 0 0 1 0
𝑍𝑍 ̅ 1
𝑥𝑥, 𝑦𝑦 are the coordinates related to the center of the image at the principal point and we
need to translate the image coordinates to the pixel coordinates origin at the top left
corner. 𝑦𝑦
� 𝑢𝑢𝑝𝑝𝑝𝑝𝑖𝑖𝑝𝑝 𝑝𝑝𝑖𝑖𝑓𝑓𝑝𝑝 𝑝𝑝𝑝𝑝𝑝𝑝𝑝𝑝𝑖𝑖𝑝𝑝
𝑥𝑥 𝑓𝑓 0 𝑥𝑥0 0 𝑋𝑋 [0,0]
�𝑦𝑦� = �0 𝑌𝑌�
𝑓𝑓 𝑦𝑦0 0� � ̅ � 𝑥𝑥𝑎𝑎 , 𝑦𝑦𝑎𝑎
1 𝑍𝑍 𝑢𝑢𝑎𝑎 , 𝑣𝑣𝑎𝑎
0 0 1 0
1 𝑥𝑥 𝑤𝑖𝑖𝑝𝑝𝑖𝑖𝑤𝑝𝑝
𝑤𝑖𝑖𝑝𝑝𝑖𝑖𝑤𝑝𝑝 𝑥𝑥𝑜𝑜 , 𝑦𝑦𝑜𝑜

𝑤𝑤𝑝𝑝𝑐𝑐𝑝𝑝𝑤 𝑤𝑤𝑝𝑝𝑐𝑐𝑝𝑝𝑤
PROJECTION MATRIX

• 𝑋𝑋� 𝑌𝑌� 𝑍𝑍̅ coordinates are the world's three-dimensional coordinates of points that have been
rotated by 𝑅𝑅 and translated by 𝑡𝑡 into the camera frame

𝑥𝑥 𝑋𝑋
𝑓𝑓 0 𝑥𝑥0
𝑌𝑌
�𝑦𝑦� = �0 𝑓𝑓 𝑦𝑦0 � [��
𝑅𝑅3×3
���−𝑅𝑅
��� ��] � �
��𝑡𝑡�3×1
3×3
� 𝑍𝑍
1 �������
0 0 1 3×4
3×1
�1
3×3
4×1

𝑓𝑓 0 𝑥𝑥0 𝑟𝑟11 𝑟𝑟12 𝑟𝑟13 𝑡𝑡𝑥𝑥 𝑝𝑝11 𝑝𝑝12 𝑝𝑝13 𝑝𝑝14

�0 𝑓𝑓 𝑦𝑦0 � �𝑟𝑟21 𝑟𝑟22 𝑟𝑟23 𝑡𝑡𝑦𝑦 � 𝑝𝑝21 𝑝𝑝22 𝑝𝑝23 𝑝𝑝24
𝑝𝑝31 𝑝𝑝32 𝑝𝑝33 𝑝𝑝34
0 0 1 𝑟𝑟31 𝑟𝑟32 𝑟𝑟33 𝑡𝑡𝑧𝑧
PROJECTION MATRIX
Projection matrix specifies how a pinhole camera applies a projective mapping of 3D points in
the world to 2D points in an image.

2D 3D
point = Intrinsic matrix Rotation matrix Translation matrix point
K(3x3) R(3x3) T(3x4) X(4x1)
(3x1)

P matrix

P is a 3x4 Matrix=12 elements

𝑥𝑥 𝑝𝑝11 𝑝𝑝12 𝑝𝑝13 𝑝𝑝14 𝑋𝑋
11 degree of freedom:
𝑦𝑦 𝑌𝑌
= 𝑝𝑝21 𝑝𝑝22 𝑝𝑝23 𝑝𝑝24 5 from intrinsic matrix K
1 𝑝𝑝31 𝑝𝑝32 𝑝𝑝33 𝑝𝑝34 𝑍𝑍 3 from rotation R
1 3 from translation T

Camera pose can be solved when finding the projection matrix P.

 WHY COMPUTING CAMERA POSE?
• Each camera projection matrix along a trajectory defines the UAV motion.
• In SfM and visual odometry, we need to compute the camera motion.
Camera motion = multiple image/camera pose (Pi) along its trajectory.

trajectory
When camera motion (multiple
P1 P2 P3 P4
poses) is defined, the structure of
the scene is reconstructed from the
Z images.
Then mapping of the scene can be
applied.
Y
More details will be given in the
SfM lecture.
WORLD X
SYSTEM
DIRECT LINEAR TRANSFORMATION DLT
• There are different techniques to solve the projection matrix
either linear or nonlinear methods.
• DLT equations for one observed point can be formulated as Z
follows:
𝑝𝑝1
Y
𝑋𝑋 𝑌𝑌 𝑍𝑍 1 0 0 0 0 − 𝑥𝑥𝑥𝑥 − 𝑥𝑥𝑥𝑥 − 𝑥𝑥𝑥𝑥 − 𝑥𝑥 𝑝𝑝2 0
� �� ⋮ � = � �
0 0 0 0 𝑋𝑋 𝑌𝑌 𝑍𝑍 1 − 𝑦𝑦𝑦𝑦 − 𝑦𝑦𝑦𝑦 − 𝑦𝑦𝑦𝑦 − 𝑦𝑦 0
𝑝𝑝12
• With six measured image points, twelve equations can be X
formed in the DLT which are the minimum number of points to
determine the parameters of the projection matrix.
• The DLT solution of the form 𝐴𝐴𝐴𝐴 = 0 using singular value
decomposition 𝑺𝑺𝑺𝑺𝑺𝑺 to solve such an equation system by
decomposing matrix 𝐴𝐴.
WHAT HAVE WE LEARNED?
• Some useful camera settings and its relation to have good images.
• The geometry of a single image , 3D to 2D.
• The rotation in 3D space and some rotational variants.
• Deamination of camera pose using linear and nonlinear methods.
Thank you