UAV PHOTOGRAMMETRY

FOR MAPPING AND 3D MODELING

- CURRENT STATUS AND FUTURE PERSPECTIVES -
Fabio REMONDINO, Francesco Nex
3D Optical Metrology (3DOM)
Bruno Kessler Foundation (FBK), Trento, Italy
Email: [email protected]

Luigi Barazzetti, Marco Scaioni

BEST Department
Politecnico di Milano, Italy
Email: luigi.barazzetti><marco.scaioni>@polimi.it

Daniele Sarazzi
Zenit srl, Italy
Email: [email protected]
F. Remondino UAV PHOTOGRAMMETRY: Current status and future perspectives
 UAV Photogrammetry
Aerial photogrammetry with non-traditional platforms carrying data capturing sensors for
geomatics and robotics applications

Platforms: balloons, airship, kite, model helicopter (electric or engine), fixed/rotary wing
aircraft, solar powered, etc.

Platforms with navigation units on board, digital camera or active sensors (laser scanner,
Kinect)

Payload constraint

Different open-source systems

[Neubronner, 1903] [Whittlesley, 1970] [Wester-Ebbinghaus, 1980] [Eisenbeiss, 2004]

F. Remondino UAV PHOTOGRAMMETRY: Current status and future perspectives
 UAV applications and features
Applications:

3D surveying and mapping

Archaeological documentation

Monitoring (environment, fires, energy fluxes, high risky situations, etc.)

Precision farming

3D city modeling

Possibility to fly everywhere and whenever we wish (almost )

Combination of terrestrial & aerial techniques and processing methodologies

Point of interest for different communities: photogrammetry, surveying, robotics,

computer vision, artificial intelligence, space domain, etc.

F. Remondino UAV PHOTOGRAMMETRY: Current status and future perspectives

 Photogrammetric pipeline with UAV mages

Flight planning (designing, requirements, system performances, etc.)

Image acquisition (autonomous, manual, GSM-based, waypoint navigation, etc.)

Image triangulation

DSM generation / Feature extraction / Ortho-image generation

F. Remondino UAV PHOTOGRAMMETRY: Current status and future perspectives

 Image triangulation

UAV image blocks have different network geometries depending on the characteristics of the
scene / object

In some cases the block is a traditional aerial image block with GPS/INS data AAT
techniques can be used

However,
Direct geo-referencing with traditional UAVs images is still not feasible
The image block can become a close-range image block with convergent images, wide baseline,
scale variations, etc.
No commercial solution for automated tie point extraction human interaction still needed

In-house developed solution for automated tie point extraction: ATiPE

[Barazzetti, L., Remondino, F., Scaioni, M., 2010: Fully automated UAV image-based sensor orientation. Int. Archives of
Photogrammetry, Remote Sensing and Spatial Information Sciences, Vol. 38(1), Calgary, Canada]

Open source solution: Apero (www.tapenade.gamsau.archi.fr, www.micmac.ign.fr)

[Pierrot-Deseilligny, M. and Clery, I., 2011: APERO, An Open Source Bundle Adjustment Software for Automatic Calibration
and Orientation of Set of Images. IAPRS&SIS, Vol. 38(5/W16), Trento, Italy]
F. Remondino UAV PHOTOGRAMMETRY: Current status and future perspectives
 Image triangulation - tie point extraction
ATiPE Automated Tie Point Extraction

Starting from the images and their calibration parameters, the method allows the automatic
orientation of correspondences from image blocks for the successive bundle adjustment

The method is based on the extraction and matching of scale invariant features
(SIFT/SURF), image location refinement (LSM) and tie point distribution improvement

[Barazzetti, L., Remondino, F., Scaioni, M., 2010: Fully automated UAV image-based sensor orientation. Int. Archives of
Photogrammetry, Remote Sensing and Spatial Information Sciences, Vol. 38(1). ISPRS Commission I Symposium,
Calgary, Canada]
F. Remondino UAV PHOTOGRAMMETRY: Current status and future perspectives
 Image triangulation - tie point extraction
INPUT DATA
The whole matching procedure is made up of images calibration parameters

COMPRESSION & RGB2GRAY

several steps
VISIBILITY MAP CREATION
Images can be compressed to speed up the from images from GPS/INS & DSM

elaboration (coarse-to-fine) approach CREATION OF M IMAGE PAIRS

The image points are detected and matched Image pair i = 1

with the SIFT or SURF operators between KEYPOINT DETECTION (SIFT/SURF)

image pairs KEYPOINT MATCHING (quadratic/kd-tree)

The robust estimation (RANSAC, LMedS, OUTLIER DETECTION

robust estimation fundamental (essential) matrix
MAPSAC) of the epipolar geometry helps in
False
i=M
removing wrong correspondences True
IMAGE PAIR CONCATENATION
The matched pairs are combined in order to
REDUCTION / BETTER DISTRIBUTION
obtain the tie points for the whole image block
IMAGE COORDINATES

BUNDLE ADJUSTMENT
F. Remondino UAV PHOTOGRAMMETRY: Current status and future perspectives
 Image triangulation - tie point extraction

The descriptor comparisons must be repeated for each image pair

N images (N2-N)/2 combinations O(N2) !!!

To limit the number of combinations we create a visibility map

using low-res images:
+ it always works
- it needs another (fast) elaboration

Using GNSS/INS + DTM

+ faster
- it works for aerial-like blocks

Example - block with 70 images

no visibility map: > 2400 combinations
with the visibility map: 1179 combinations
image

connection
F. Remondino UAV PHOTOGRAMMETRY: Current status and future perspectives
 Image triangulation - tie point extraction
Visibility map

The method was tested on

4 blocks featuring different
characteristics (e.g.
network geometry,
convergent images, wide
baselines, )

3160 2415 496 3741 Total pairs

1179 507 160 897 Estimated pairs
F. Remondino UAV PHOTOGRAMMETRY: Current status and future perspectives
 Image triangulation - tie point extraction
INPUT DATA
The whole matching procedure is made up of images calibration parameters

COMPRESSION & RGB2GRAY

several steps
VISIBILITY MAP CREATION
Images can be compressed to speed up the from images from GPS/INS & DSM

elaboration (coarse-to-fine) approach CREATION OF M IMAGE PAIRS

The image points are detected and matched Image pair i = 1

with the SIFT or SURF operators between KEYPOINT DETECTION (SIFT/SURF)

image pairs KEYPOINT MATCHING (quadratic/kd-tree)

The robust estimation (RANSAC, LMedS, OUTLIER DETECTION

robust estimation fundamental (essential) matrix
MAPSAC) of the epipolar geometry helps in
False
i=M
removing wrong correspondences True
IMAGE PAIR CONCATENATION
The matched pairs are combined in order to
REDUCTION / BETTER DISTRIBUTION
obtain the tie points for the whole image block
IMAGE COORDINATES

BUNDLE ADJUSTMENT
F. Remondino UAV PHOTOGRAMMETRY: Current status and future perspectives
 Image triangulation - tie point extraction

The reduction of the number of TPs is carried out by dividing the images into regular cells

For each cell only the point with the best multiplicity is stored

Better distribution on the images, better point multiplicity easier outlier removal

Before

After

F. Remondino UAV PHOTOGRAMMETRY: Current status and future perspectives

 Image triangulation - Example

Image block is made up of 80 convergent images (Nikon D200, 20 mm lens) randomly

distributed in front of the faade of a spire of the Dome

Helicopter (Hirobo SST-Eagle-Freyer) manually piloted (bad GPS signal in that area)

GPS/INS data not available visibility map from images

F. Remondino UAV PHOTOGRAMMETRY: Current status and future perspectives

 Image triangulation - Example
The procedure for image matching is robust under different viewing conditions

Convergent images

Scale variation

Camera poses after the bundle

F. Remondino UAV PHOTOGRAMMETRY: Current status and future perspectives

 Image triangulation - Example
Block with some GCPs available for accuracy check

GSD = ca 2 cm

The precision in the normal case is given by:

Z2
z 2 im 8 cm
cb
Bundle adjustment with 3 GCPs and 5 Check Points

RMSEx = 4 cm ; RMSEy = 3 cm ; RMSEz = 7 cm

However, the covariance matrix of the bundle solution shows the

following theoretical precisions:

x = 0.6 cm ; y = 0.6 cm ; z = 2.3 cm

!!! Very often is difficult to manually measure the GCP in the images !!!

F. Remondino UAV PHOTOGRAMMETRY: Current status and future perspectives

 Image matching and DSM generation
Automated DSM generation for mapping, documentation, monitoring, visualization issues

Commercial (SAT-PP, 4Dixplorer AG) vs free solutions

Open-source solution: MicMac (www.tapenade.gamsau.archi.fr, www.micmac.ign.fr)

Multi-image multi-resolution approach

[Pierrot-Deseilligny, M., Paparoditis N., 2006. A multiresolution and optimization-based image matching approach: an
application to surface reconstruction from SPOT5-HRS stereo imagery. IAPRS&SIS, Vol.36(1/W41), Antalya, Turkey]
F. Remondino UAV PHOTOGRAMMETRY: Current status and future perspectives
 Image matching and DSM generation
Microdrone MD-200

19 images

Pentax Optio A40 (8 mm lens, 12 Mpx,

pixel size 1.9 mm)

DSM 17 with mil. pts

F. Remondino UAV PHOTOGRAMMETRY: Current status and future perspectives

 Further examples
Maya archaeological site in Copan, Honduras

Image acquisition with UAV model helicopter equipped with a Nikon SRL camera (12 Mpx)

Flying height ca 100 m

GSD ca 1 cm

F. Remondino UAV PHOTOGRAMMETRY: Current status and future perspectives

 Further examples

F. Remondino UAV PHOTOGRAMMETRY: Current status and future perspectives

 Further examples

Archaeological area of Pava (Siena, Italy), 40 images, ca 40x50 m

Microdrone MD4-200, Pentax Optio A40 (8 mm lens, 12 Mpx, pixel size 1.9 mm)

Flying height ca 35 m

DSM @ 5 cm resolution

Mosaic of the area

An image of the dataset
F. Remondino UAV PHOTOGRAMMETRY: Current status and future perspectives
F. Remondino UAV PHOTOGRAMMETRY: Current status and future perspectives
 Further examples

DSM @ 5 cm resolution

Entire processing in ca 2 days

orthophoto
DSM used for excavation volume computation

Ortho used for GIS issues

F. Remondino UAV PHOTOGRAMMETRY: Current status and future perspectives
 Further examples

Archaeological area in Montalcino (ca 250x350 m), 52 images

Microdrone MD4-200, Pentax Optio A40 (8 mm lens, 12 Mpx, pixel size 1.9 mm)

Flying height ca 105 m

18 GCP with DGPS and total station

F. Remondino UAV PHOTOGRAMMETRY: Current status and future perspectives

 Further examples

Flight planning:
Shapefile of the AOI
Overlap across/along track
Camera parameters
Required GSD
DTM of the area

F. Remondino UAV PHOTOGRAMMETRY: Current status and future perspectives

 Further examples

Image orientation and DSM generation

F. Remondino UAV PHOTOGRAMMETRY: Current status and future perspectives

 Further examples

DSM and derived contours

F. Remondino UAV PHOTOGRAMMETRY: Current status and future perspectives
 Further examples

Textured 3D model
F. Remondino UAV PHOTOGRAMMETRY: Current status and future perspectives
 Further examples
Precision Farming Winery area
- Pentax Optio A40 for the images in the visible spectrum and a Sigma DP1 for the images in
the NIR spectrum

wine yard area NIR false colors estimated NDVI index

Thermal application
- MD4-100 with IR camera to track animals

F. Remondino UAV PHOTOGRAMMETRY: Current status and future perspectives

 Conclusions and outlook

UAV are a promising alternative for imaging and 3D modeling

Automated methodology for UAV image triangulation, DSM generation and

orthophoto production user interaction still required for geo-referencing

Navigation data are strictly necessary for autonomous flight

Navigation data not strictly necessary for automated orientation, but can be
used to:
create the visibility map (reduce the computational time in finding the image
correspondences)

estimate the bundle solution

From the data processing point of view, the solutions are available

F. Remondino UAV PHOTOGRAMMETRY: Current status and future perspectives 28

 Conclusions and outlook
Technology made great advantages from the platforms and control point of view
but for geomatics applications we still need:

Reliability of the systems/platforms in the fields and for mapping applications

Quality of the acquired images Payload issue

Regulation for the flights

Real-time triangulation (direct geo-referencing)

Longer flying time and more autonomy

F. Remondino UAV PHOTOGRAMMETRY: Current status and future perspectives 29

 Thank you

F. Remondino UAV PHOTOGRAMMETRY: Current status and future perspectives

 Image triangulation - Example

To verify the accuracy of the triangulation methodology, 4 blocks were employed

The precision of the bundle solution is good and acceptable

Few outliers were easily removed with data snooping techniques

Computational issues due to the huge numbers of TPs but reduction method very helpful

Block 4 was impossible to orient without tie point reduction method

F. Remondino UAV PHOTOGRAMMETRY: Current status and future perspectives