3D Computer Vision

Prof. Didier Stricker

Kaiserlautern University
http://ags.cs.uni-kl.de/
DFKI – Deutsches Forschungszentrum für Künstliche Intelligenz
http://av.dfki.de

1
Augmented Vision
• Head: Prof. Didier Stricker
• Founded in July 2008
• 60 PhDs and fulltime researchers
• 3 strongly connected research areas

Computer Vision

Virtual and
Body Sensor
Augmented
Networks
Reality

2
3D Computer Vision
Lecture:
• Room: 01-006
• Time: Wednesday, 10:00 to 11:30
• In-person lecture

• “SWS: 2V+1Ü“
• Credit points: 4 CP
• Language: English
3D Computer Vision
Exercise:
• Room: 11-243
• Time: Thursday, 15:30 to 17:00 (tomorrow!)
• Q&A-Style session with tutors

• Quizzes
• Programming exercise
• Final presentation
3D Computer Vision
Quizzes:
• After every lecture (11:30) in OLAT.
• Usually 1 week time (check deadlines in OLAT).

Exercise:
• Large programming task
• Available on 30th of October (11:30)
• Deadline on 16th of January (23:59)
• Final presentation of your solution
• (Optional) Introduction to Python & Jupyter Notebook
• After THIS lecture on OLAT
3D Computer Vision
Exam:
• 28.02.2025, 13:30, 46-210 tentative
• 27.03.2025, 13:30, 46-210 tentative

Eligibility (not negotiable!):

• 11/13 Quizzes must be passed successfully!
• Automatic feedback
• Unlimited attempts
• Programming exercise must be passed successfully!
• Final presentation
Contact

• Prof. Didier Stricker

• Tutors:
• Marcel Rogge
• Katharina Bendig

• All materials will be available in OLAT

• We will announce any updates via OLAT (turn on notifications)
• Please ask questions in the weekly Q&A sessions or OLAT forum
3D Computer Vision: lecture

• Topics:
Camera pose
• Introduction: what is a camera? images
• Camera model and camera calibration
• Fitting and parameter estimation
• 2D-image transformation (mapping) and panorama
• Two cameras: epipolar geometry and triangulation
3D reconstruction
• Multiple view reconstruction
• Depth maps and multiple view stereo reconstruction
• Structured light: laser, coded light

Texturing Texturing 8
Overview

• Introduction: application of computer vision

• The pinhole projection model

• Qualitative properties
• Perspective projection

• The real camera (cameras with lenses)

• Depth of focus
• Field of view
• Lens aberrations

• Digital cameras
• Types of sensors
• Color
9
Augmented Vision: Maintenance

1998-2000
Augmented Reality: Marker-less tracking

Augmented Vision for Maintenance Tourism

2004 2002
Augmented Things
The universal interface for the Internet of Things

Tracking of an industrial machine

Augmented Things

23.10.2024 13
Autonomous Driving

14
Body tracking

23.10.2024 15
Example

Input images Structure From Motion

3D model

Multiple View Stereo

17
3D Reconstruction from spherical images

http://youtu.be/Hg792_Mh1Cg
Theta camera 360° (RICOH)
• Image stack (1 cm, 60 images)

23.10.2024 Copyright © 2015 Augmented Vision - DFKI 19

23.10.2024 20
23.10.2024 21
The Golden Age of Computer Vision

Source: http://cvpr2019.thecvf.com

23.10.2024 9,161 full submissions at CVPR in 2023 22

Achievements

• Until early 2000

• Great achievements at theoretical level (projective geometry, scale-space theory, ...)

• But a big lack of robustness („if it works, it is not computer vision“)

• Today

• “Detection of associations“ (Peral,‘18) works very well

• Several problems seem to be solved (e.g. face detection, ..)
• But: reality is pretty complex and expectations are very high

23.10.2024 23
Overview

• Introduction: application of computer vision

• The pinhole projection model

• Qualitative properties
• Perspective projection

• The real camera (cameras with lenses)

• Depth of focus
• Field of view
• Lens aberrations

• Digital cameras
• Types of sensors
• Color
24
Today: The Camera

25
How do we see the world?

• Let’s design a camera

• Idea 1: put a piece of film in front of an object
• Do we get a reasonable image?

26
Slide by Steve Seitz
Pinhole camera

• Add a barrier to block off most of the rays

• This reduces blurring
• The opening known as the aperture

27
Slide by Steve Seitz
Pinhole camera model

• Pinhole model:
• Captures pencil of rays – all rays through a single point
• The point is called Center of Projection (focal point)
• The image is formed on the Image Plane

28
Slide by Steve Seitz
Dimensionality Reduction Machine (3D to 2D)

3D world 2D image

Point of observation

What have we lost?

• Angles
• Distances (lengths)
Slide by A. Efros 29
Figures © Stephen E. Palmer, 2002
Projection properties

• Many-to-one: any point along the same ray maps to the same point in the
image

• Points → points
• But the projection of points on the focal plane is undefined

• Lines → lines (collinearity is preserved)

• But a line through focal point projects to a point

• Planes → planes (or half-planes)

• But plane through focal point projects to line
30
Projection properties
• Parallel lines converge at a vanishing point
• Each direction in space has its own vanishing point
• But parallels parallel to the image plane remain
parallel
• All directions in the same plane have vanishing
points on the same line

How do we construct the vanishing point/line? 31

One-point perspective

• Masaccio, Trinity, Santa

Maria Novella, Florence,
1425-28

• First consistent use of

perspective in Western
art?

32
Perspective distortion
• Problem for architectural photography: converging verticals

33
Source: F. Durand
Perspective distortion
• What does a sphere project to?

34
Image source: F. Durand
Perspective distortion
• What does a sphere project to?

35
Perspective distortion
• The exterior columns appear bigger
• The distortion is not due to lens flaws
• Problem pointed out by Da Vinci

36
Slide by F. Durand
Perspective distortion: People

37
Modeling projection

• The coordinate system

• We will use the pinhole model as an approximation
• Put the optical center (O) at the origin
• Put the image plane (Π’) in front of O

38
Source: J. Ponce, S. Seitz
Modeling projection
•Projection equations
• Compute intersection with Π’ of ray from P = (x,y,z) to O
• Derived using similar triangles

• We get the projection by throwing out the last coordinate:

x y
( x, y , z ) → ( f ' , f ' ) 39
z z Source: J. Ponce, S. Seitz
Overview

• Introduction: application of computer vision

• The pinhole projection model

• Qualitative properties
• Perspective projection

• The real camera (cameras with lenses)

• Depth of focus
• Field of view
• Lens aberrations

• Digital cameras
• Types of sensors
• Color
40
Building a real camera

41
Camera Obscura

• Basic principle known to

Mozi (470-390 BCE),
Aristotle (384-322 BCE)
• Drawing aid for artists:
described by Leonardo da
Vinci (1452-1519)

Gemma Frisius, 1558

42
Source: A. Efros
Home-made pinhole camera
35mm-pinhole-camera

Why so
blurry?

43
Slide by A. Efros http://www.debevec.org/Pinhole/
Shrinking the aperture

• Why not make the aperture as small as possible?

• Less light gets through
• Diffraction effects…
44
Slide by Steve Seitz
Shrinking the aperture

Diffraction effects

45
The reason for lenses

46
Slide by Steve Seitz
Adding a lens

• A lens focuses light onto the film

• Rays passing through the center are not deviated

47
Slide by Steve Seitz
Adding a lens

focal point

• A lens focuses light onto the film

• Rays passing through the center are not deviated
• All parallel rays converge to one point on a plane
located at the focal length f

48
Slide by Steve Seitz
Adding a lens

“circle of
confusion”

• A lens focuses light onto the film

• There is a specific distance at which objects are “in
focus”
• other points project to a “circle of confusion” in the image

49
Slide by Steve Seitz
Thin lens formula
D’ D
f

Frédo Durand’s slide 50

Thin lens formula

Similar triangles everywhere!

D’ D
f

Frédo Durand’s slide 51

Thin lens formula

Similar triangles everywhere! y’/y = D’/D

D’ D
f
y
y’

Frédo Durand’s slide 52

Thin lens formula
Similar triangles everywhere! y’/y = D’/D
y’/y = (D’-f)/f
D’ D
f
y
y’

Frédo Durand’s slide 53

Thin lens formula
1 1 1 Any point satisfying the thin lens
+ =
D’ D f equation is in focus.
D’ D
f

Frédo Durand’s slide 54

Depth of Field

http://www.cambridgeincolour.com/tutorials/depth-of-field.htm

Slide by A. Efros 55
How can we control the depth of field?
• Changing the aperture size affects depth of field
• A smaller aperture increases the range in which the object is approximately in focus
• But small aperture reduces amount of light – need to increase exposure
At f/5.6, the flowers are isolated from the background.

f / 5.6
At f/32, the background is distracting.

f / 32
56
Flower images from Wikipedia http://en.wikipedia.org/wiki/Depth_of_field Slide by A. Efros
Varying the aperture

Large aperture = small DOF Small aperture = large DOF 57

Slide by A. Efros
Field of View (Zoom)

58
Slide by A. Efros
Field of View

f
f

FOV depends on focal length and size of the camera retina

Smaller FOV = larger Focal Length Slide by A. Efros

59
Field of View / Focal Length

Large FOV, small f

Camera close to car

Small FOV, large f

Camera far from the car 60
Sources: A. Efros, F. Durand
Same effect for faces

wide-angle standard telephoto

61
Source: F. Durand
Real lenses

62
Lens Flaws: Chromatic Aberration

• Lens has different refractive indices for different

wavelengths: causes color fringing

Near Lens Center Near Lens Outer Edge

63
Lens flaws: Vignetting

64
Radial Distortion
• Caused by imperfect lenses
• Deviations are most noticeable for rays that pass through the
edge of the lens

No distortion Pin cushion Barrel

65
Overview

• Introduction: application of computer vision

• The pinhole projection model

• Qualitative properties
• Perspective projection

• The real camera (cameras with lenses)

• Depth of focus
• Field of view
• Lens aberrations

• Digital cameras
• Types of sensors
• Color
66
Digital camera

• A digital camera replaces film with a sensor array

• Each cell in the array is light-sensitive diode that converts photons to electrons
• Two common types
• Charge Coupled Device (CCD)
• Complementary metal oxide semiconductor (CMOS)
• http://electronics.howstuffworks.com/digital-camera.htm

67
Slide by Steve Seitz
CCD vs. CMOS
• CCD: transports the charge across the chip and reads it at one corner of the array. An analog-to-digital converter (ADC) then turns each
pixel's value into a digital value by measuring the amount of charge at each photosite and converting that measurement to bin ary form

• CMOS: uses several transistors at each pixel to amplify and move the charge using more traditional wires. The CMOS signal is digital, so
it needs no ADC.

http://electronics.howstuffworks.com/digital-camera.htm

68
http://www.dalsa.com/shared/content/pdfs/CCD_vs_CMOS_Litwiller_2005.pdf
High Dynamic Range Images
• CCD / CMOS have a given dynamic range for capturing the luminance of the scene – LDR cameras (Low Dynamic
Range)
• Problem: The dynamic range of the scene illumination is higher than the dynamic of the camera
• It results:
• Underexposed areas
• Saturation areas

69
High Dynamic Range Images
• High dynamic range image
- Standard RGB images are encoded over 8 bit integer (255 values)
- HDR: 12-, 16- or 32-bit floating point

See: www.OpenEXR.com

70
Color sensing in camera: Color filter array

Bayer grid
Estimate missing
components from
neighboring values
(demosaicing)

Why more green?

Human Luminance Sensitivity Function

71
Source: Steve Seitz
Color sensing in camera: Prism

• 3 CCD camera
• requires three chips and precise alignment
• More expensive
CCD(R)

CCD(G)

CCD(B) 72
Issues with digital cameras
• Noise
• low light is where you most notice noise
• light sensitivity (ISO) / noise tradeoff
• stuck pixels

• Resolution: Are more megapixels better?

• requires higher-quality lens
• Noise issues

• In-camera processing
• Over-sharpening can produce halos

• RAW vs. compressed

• File size vs. quality tradeoff

• Blooming
• Charge overflowing into neighboring pixels

• Color artefacts
• Purple fringing from microlenses, artefacts from Bayer patterns
• white balance 73
Slide by Steve Seitz
How to build your own camera

• Nice project and video (pinehole, lense, sensor)

• https://www.youtube.com/watch?v=PaXweP73NT4

• https://www.instructables.com/DIY-Image-Sensor-and-Digital-Camera/

74
Historical context
• Pinhole model: Mozi (470-390 BCE), Aristotle (384-322 BCE)
• Principles of optics (including lenses):
Alhacen (965-1039 CE) Alhacen’s notes

• Camera obscura: Leonardo da Vinci (1452-1519), Johann Zahn

(1631-1707)
• First photo: Joseph Nicephore Niepce (1822)
• Daguerréotypes (1839)
• Photographic film (Eastman, 1889)
Niepce, “La Table Servie,” 1822
• Cinema (Lumière Brothers, 1895)
• Color Photography (Lumière Brothers, 1908)
• Television (Baird, Farnsworth, Zworykin, 1920s)
• First consumer camera with CCD: Sony Mavica (1981)
• First fully digital camera: Kodak DCS100 (1990)
75
CCD chip
 Thank You!
Do not forget the QUIZ!!!

76