Geometric optics and camera, practically

Václav Hlaváč

Czech Technical University in Prague

Czech Institute of Informatics, Robotics and Cybernetics, also Center for Machine Perception
160 00 Prague 6, Jugoslávských partyzánů 1580/3, Czech Republic
http://people.ciirc.cvut.cz/hlavac, [email protected]
Courtesy: Pavel Krsek, Vladimír Smutný. Prerequisite: Image formation lecture.

Outline of the talk:

Motivation, lens of a camera. Depth of focus. Depth of field.



Why are lenses needed? Lenses.



Geometric optics as a simplified model. Lenses abberations.



Lens from a physical point of view. Camera, radiometric image formation.



 Entire image acquisition chain, overview
2/1
A look at an entire chain: from the observed property of interest through radiance L and
irradiance E to an electrical signal and finally to a digital image.

Two image acquisition options:

Direct observation – there is one-to-one correspondence between a point in a 3D scene and its


2D image (e.g., a ray in the projective transformation).

Indirect observation – provides also a spatially dependent radiance L but there is no


one-to-one correspondence between 3D and 2D information (e.g., radar, tomography, spectral

imaging techniques, magnetic resonance).
Single-lens reflex camera, a cross-section
3/1
 Basic elements of a camera
4/1

Photographic lens. Fixed or varying



focal length. Diaphragm with varying Analog/digital converter.


aperture. Composed from several optical
Computer converting raw image data to


lenses, single optical elements.
a viewable representation.
Shutter. Mechanic or electronic.


LCD display for viewing images. LCD


Optical viewer (absent with cheaper stands for Liquid Crystal Display.


cameras).
Memory medium, often removable.


Matrix of light sensitive sensors. CCD


Energy source, battery or rechargeable


or CMOS.
battery.
Amplifier modifying the signal from


sensors.
 Improving parts/functions of the camera
5/1
Automatic exposition setting, i.e. joint setting of the diaphragm opening and shutter speed.


Automatic focusing. Q: What physical principles are used for it?



Image stabilization minimizing effect of a thrashing hand.



Built-in flash.


Ability to capture a video sequence.



Image resolution and its compression can be set.



Image capturing in a RAW mode.



The camera contains a processor dealing with basic image processin/analysis operations, e.g,


human face detection which serves for automatic selection of image points used for automatic
focusing.
 The job of a (photographic) lens
6/1

The optical system (photographic lens) focuses the incoming energy (photons) and creates


the image on the image sensor.

The optical lens is a single, optically transparent device shaped to allow the light transmission


and refraction while creating a desired optical outcome.

The measured physical entity is the irradiance [W m−2] (informally brightness or gray value


from a human perception point of view).

The lens should mimic the ideal perspective transformation as much as possible (also


projective transformation, pin-hole model).

We will constrain to geometric optics in this simplified optics explanation. We leave wave and


quantum optics models aside.

 Approximation by geometric optics
7/1

It is one of several possible approximations.

Assumptions:
The involved wavelengths of the electromagnetic irradiation (here a frequency sub-band of it


= light) are very small as compared to sizes of used optical and mechanical elements.
The energy of photons (from the quantum theory point of view) are small with respect to


energetic sensitivity of involved sensors.

Geometric optics is a rough approximation. Geometric optics is important for daily life technology.
It is also interesting from the point of view of the historic development of opinions in physics.

Recommended reading: Feynman R.P, Leighton R.B., Sands M.: Feynman Lectures on
Physics, 3 volumes, (1963-1965).
 A pin-hole camera
8/1
15th century, the architect Filippo Brunelleschi from Italian Florence (1377-1446),


a tool for drawing perspective images.

16th century, pin-hole camera, in Latin camera obscura.


1822 The Frenchman J.-N. Niepce added a photographic plate to the pinhole camera ⇒ the


first photograph was born.

(str
aigh
t lin
e) r
a y

pinhole
three-dimensional
scene image p
lane

Convention: We consider left to right direction of light.

 The size of a hole in a pin-hole camera
9/1
The interplay of contradictory phenomena.

a. The bigger hole passes more light but blurs the image.

b. The small hole causes diffraction and the image will be blurred too.

c. The optimum exists, in which the image is least blurred. Example: For f =100 [mm] and λ=500
[nm], the optimal diameter of the hole is 0.32 [mm].

a b c
 Why are lenses used?
10/1
The pin-hole camera:
Collects only a few photons (light).


Troubles due to diffraction due to the hole.



No abberations (almost).


The lens:
Collects more photons (light).


Have to be in focus.


Suffers from abberations.


 The (optical) lens from a physics point of view
11/1
Behavior of a lens was explained by the Dutch mathematician Willebrord


van Roijen Snell (1580–1626), who formulated the light refraction law on
the boundary of separation of two contacting substances in the year 1621. a1

n1 sin α2
n= = , where n is the refractive index. n1
n2 sin α1
n2
n for a yellow light λ=589 [nm] on the boundary between the vacuum and


X:
X = air 1,0002; water 1.333; crown glass (a small diffusion of light, a a2
small refraction index) 1.517; lead optical glass 1.655; diamond 2.417.
There is an elegant derivation of the Snell’s refraction law, which uses


(approximate) Fermatt’s principle of the shortest time from the year 1650,
see Feynman’s Lectures on Physics.
 Lens
12/1
A lens forms an image by focusing light from the scene.


Different rays of an incident light beam are refracted through different angles, so that an


entire beam of parallel rays can be caused to converge on, or to appear to diverge from, a
single point.
 The lens, mathematical model
13/1

object plane image plane

principal point image focal

point
d

object focal
point

z f f z’
principal plane

1 1 1
The thin lens equation in a Newtonian form = +
f z0 + f f + z
or rewritten to a simpler expression f 2 = z z 0
 A single lens, derivation, idea: similar 4
14/1

y y

y’ y’

z f f z’ z f f z’

y0 z0 + f y0 z0
= =
y z+f y f
By connecting the two equations:
z0 + f z0
=
z+f f
0 0
f (z + f ) = z (z + f )
0 2 0 0
fz + f = zz + f z
2 0
f = zz
 Ray diagrams
15/1
Ray diagrams simplify lens operation by It is useful to distinguish between two categories of images (also in
considering three distinct cases: ray diagrams):
Real images - are produced from actual rays of light coming


to a focus (e.g. a film projected onto a screen);
Virtual images - are produced from where rays of light appear


to be coming from (e.g. a magnifying glass image).

A. A ray passing through the optical centre of We will show diagrams representing image formation from an object
the lens; positioned. Let F be the focal point on the object side (left).
B. A ray parallel to the principal axis, which
object positioned between F and the lens


refracts through the lens, passing through
the principal focus; object positioned at F

C. A ray passing through the principal focus
object positioned between F and 2F


(on the same side as the object) and being

refracted through the lens, emerging object positioned at 2F from the lens


parallel to the principal axis.

 Arrangements, positioning of the object
16/1
Magnifying glass. Object between F and the Projector. Object between F and 2F .



lens; virtual, upright, magnified

1:1 Copy machine. Object at 2F .


Similar image to the image above. This time the
Object at F ; parallel rays, image at infinity = no image is the same distance behind the lens as the


image object is in front. As before, the image is inverted

and real, but is the same size as the object.
Camera lens. Distance to the object > 2F ,


usually  F .
Normal lens, wide-angle lens, telephoto lens.
 Arrangements, telescope, microscope
17/1
Refracting astronomical telescope Compound microscope
Used to see large and distant objects. Large Used to see close, very small objects. Small



objective aperture. objective aperture.
Focal length of the eye piece lens is smaller than Focal length of the eye piece lens is grater than



focal length of the objective. focal length of the objective.
Objective focal length of the objective is large for Objective focal length of the objective is large for



higher magnification. higher magnification.
 Thick (composed) lens
the approximation of the optical system 18/1

image
focal point

object
focal point

z f f z’
principal plane 1 principal plane 2
 A compound lens, implementation example
19/1

The compound lens is used to suppress optical abberations.



Principal optical abberations: vignetting (natural, optical, mechanical), chromatic, spherical



abberations, coma, astigmatism, and geometric (radial/tangencial) distortions. We will treat

optical abberations later in this lecture.
 (Object space) telecentric lens
20/1
Only chief rays used, i.e. oblique rays passing through the center of the aperture stop placed


at focal point. These rays are ≈ parallel to the optical axis.

The input lens has to have bigger diameter than measured object.


Useful when measured object changes its position or the object is ‘thick’ because


magnification is independent to the distance from the lens.

Collimator
Chip

Light source Lens

Diaphragm
Object
 Diaphragm (also aperture stop)
21/1
A thin planar opaque object perpendicular to


Exit pupil - image of the aperture as

seen from the image trough the lens
the optical axis with an aperture at its center. Object
plane
Image
plane

The diaphragm role is to stop part of the light



Object in
a scene

energy (rays) from reaching the focal plane.

Optic
axis

The aperture size is adjustable either manually Projected



Physical object
aperture

or it is motorized with commercial lenses in

cameras and video cameras.
The adjustable aperture is often called the iris


diaphragm, analogically to the iris in human

eye. The adjustable diaphragm often consists
of adjustable blades.

One possible construction of the diaphragm.

Diaphragm 2, examples
22/1

f2.8 f8 f22
 Depth of focus
23/1
Depth of focus explains, why is it possible to shift the image plane to the right a little in the


direction of optical axis in the image space and still have the image in focus. It is because of a
finite size of one pixel on a sensor or a definite size of the grain in the film.
The dispersive circle has a diameter ε.


aperture
stop

field of focus
principal
point
d

image
focal point
f z

object space image space

 Depth of field
24/1

Depth of field determines the range of distances from the center of projection in the object space,
in which the objects are shown in focus. This is the parameter, which is of practical interest for the
photographer.

aperture stop

depth of field
circle of the
admissible defocus

object space image space

Aperture stop influence to the field of focus (1)
25/1

a large aperture, a small depth of field a small aperture, a large depth of field
Aperture stop influence to the field of focus (2)
26/1

f =1,8 f =8 f =22

Lens Canon EF 50 mm f/1.8 II with maximum aperture 1:1.8.

Influence of the focal length to the depth of field
27/1
 Parameters of lenses (1)
28/1
Focal length – fixed, adjustable (zoom) manually or motorized.
Diaphragm
Aperture is the maximum diameter of a light beam that


can pass through an optical system. The size of aperture is

controlled by the size of diaphragm.
Diaphragm can be fixed, adjustable manually or motorized.


Lens connecting
C – the distance between the back of the lens and the chip


is approx. 17 mm.
CS – approx 12 mm, the other parameters are the same.


Lens for C mount can be adjusted to CS mount by an



extension ring 5 mm thick. It does not work from CS to C.

 Parameters of lenses (2)
29/1

Focusing – Fix focus (e.g., web cameras, mobile phones),

manual or motorized focusing.
Distances in which object is in focus – can be changed
by extension rings in the expense of deteriorated optical
properties.
Format – which is the biggest chip usable; 1”, 2/3”, 1/2”,
1/3”, 1/4”.
Thread for a filter – e.g., a clear filter is used to protect
the lens.
Radial distortion – is not given in technical sheets but it is
important for measurement applications. Lenses with
short focal length have typically bigger radial distortions
(several pixels).
Courtesy Markus Kohlpaintner (figure)
 Natural vignetting
30/1
The term cos4 α describes a systematic optical abberation


image
plane
named natural vignetting.

The derivation of the related Irradiance equation can be
found in the lecture about image capturing from the f
physics point of view.
The natural vignetting phenomenon describes the


situation, in which the rays refracted at higher angle α

with respect to the optical axis.
original
This error (natural vignetting) is more pronounced with


wide angle lenses than with telephoto ones.

Natural vignetting is a systematic error. It can be undone


if the camera and the scene is radiometrically calibrated.

vignetting
 Optical vignetting
31/1
The compound lens thickness thickness


ranges from several millimeters to

several centimeters. This is the reason,
that why not all rays can hit the lens
aperture opening.
The phenomenon is more pronounced


for a more open aperture stops.

 Mechanical vignetting
32/1
Only inattentive users suffer from mechanical vignetting.


The lens hood must match the particular lens..


 Chromatic abberations
33/1
Chromatic aberrations are caused by the dependence of the lens refraction index on the light wavelength.


This property is desired in the prism for the light decomposition. However, it is undesirable for lenses.
The lens brings different colors (wavelength) to a focus at different points on optical axis.


Two types: Longitudial/axial (Fig. line 1) and lateral/traverse (Fig. line 2). The aberration is more pronounced at


the margins of the image.

 Chromatic abberations, doublet lens correction
34/1

The abberation is rectified while manufacturing



the lens.
A pair of doublets is used as a building element,


i.e. the lens is composed of two pieces from two

different materials, the crown and the lead (flint)
glasses, optical glasses with, respectively,
relatively low and high refraction indices.

Doublet lens correcting for longitudinal

chromatic abberation.
Chromatic lateral abberation, a practical view
35/1

Near to the optical axis Far from the optical axis

Center of the image Image rims
 Chromatic abberation, extreme illustration
36/1

Bad quality lens of a peephole Projected sunset

in the US motel. on the opposite side of a dark roon.
 Astigmatism
37/1
Astigmatism is an optical aberration that occurs when rays lying in two perpendicular planes


to the optical axis (sagittal = horizontal; tangential = vertical) have different foci.
This causes blur in one direction that is absent in the other direction. If we focus the sensor


for the sagittal plane, we see circles become ellipses in the tangential direction and vice versa.

Courtesy http://www.opto-e.com/
 Coma
38/1
Coma (from Latin comet) applies to rays entering the lens at an slanted angle. These rays do


not quite converge at the focal plane.

Coma can be demonstrated by tilting a single lens under the sunlight. If the lens optical axis


directs to sun, the projected sun is circular. If the optical axis is tilted the projected sun has
elongated shape, like a comet.
Coma is positive when off-axis rays focus furthest from the axis, and negative when they are


closest.

Courtesy http://www.opto-e.com/
 Spherical aberration
39/1
Spherical lenses are very common because they are relatively easy to manufacture.


The spherical shape is not ideal for perfect imaging. Collimated rays entering the lens at


different distances from the optical axis will converge to different points at optical axis,
causing an overall loss of focus.
The blur effect increases towards the edge of the lens.


To reduce the problem, aspherical lenses are often used.



Courtesy http://www.opto-e.com/
 Petzvald field curvature
40/1
Field curvature aberration describes the fact that parallel rays reaching the lens from different


directions do not focus on a plane, but rather on a curved (Petzval) surface (rotational
paraboloid).
This causes radial defocusing, i.e. for a given sensor position, only a circular crown will be in


focus. The aberration manifests itself as geometric radial distortion described next.

Courtesy http://www.opto-e.com/
 Radial (geometric) distortion
41/1
It is the prevalent distortion. It is pronounced more


with wide-angle lenses.

(x0, y 0) are uncorrected point coordinates measured in


the image; (x, y) are corrected coordinates; (x0, y0)

are coordinates of the principal point; (∆x, ∆y ) are
pincushion barrel
elements
p of the correction, and r is the radius,
r = (x0 − x0)2 + (y 0 − y0)2.
The distortion is often approximated by a polynomial


(x,y)
of the even order (why?), often only 2nd order. (x’,y’) Dy
Dx
r
(x0 ,y0 )
∆x = (x0 − x0) (κ1r2 + κ2r4 + κ3r6) , x

∆y = (y 0 − y0) (κ1r2 + κ2r4 + κ3r6) .

y
 Radial distortion, practical illustration
42/1

barrel without distortion pincushion

Radiometric image formation in a camera
43/1

Courtesy: Sergey Alexandrov

Low brightness dynamic range illustration
44/1

Luminance [cd/m2]: night with Moon light 10−2;

indoor lighting 102; day light (overcast/clear sky) 103/105.

Courtesy: Sergey Alexandrov

 Principle of photoconversion in semiconductors
45/1

Incoming radiation (photons) in converted in the semiconductor mass into charge couples,


electron-hole.
The semiconductor is in a static electric field. The Electron-hole couples are converted into a


short current impulse.

The current impulse must be amplified and processed. E.g., in a CCD element the impulse is


used to charge a capacitor.

 Photodiode and MOS structure
46/1

Cross cut of two main principles for current generation and storing the charge.
CCD architectures
47/1
 CCD chip, properties of the technology
48/1

+ Linearity: CCD sensors explore conversion of a photon to the couple electron-hole. The
obtained charge is integrated in a capacitor.
+ Low noise: is given by the integral character of the measurement. Uncooled chip with TV
read-out has SNR approx. 60 dB.
+ Efficiency: Current sensors have hight energetic efficiency approx. 40%, i.e. every third
photon generates one couple electron-hole.
– Read-out: only from the whole chip at once.
– Limited range of intensities: is given by the maximal capacity of individual capacitors..
 CMOS chip, properties of the technology
49/1

http://www.ims-chips.de/products/vision/hdrc alt/hdrc ima.html

http://www.imec.be/bo
http://www.vector-international.be/C-Cam/cmosccd.html
+ Logarithmic sensitivity: CMOS sensors are based on the photo diode principle. They
measure a current in a read-out instance.
+ Read-out: possible in arbitrary order, e.g. only the region of interest can be read-out.
+ Camera and processor on the same chip: CMOS technology is well mastered
(processors, memory). Smart cameras.
– Higher noise:
 Cameras, user’s view (1)
50/1

Spatial resolution: number of pixels in a row and in a column. TV CCIR/PAL 768×576.



TV RS170/NTSC 640×484. Non-television cameras also 2000×2000, keep increasing.

Resolution in intensity: given in bits for digital cameras, output typically 8 bits also 12


bits. For analog cameras – SNR, usually >50 dB.

Sensitivity: v lux. Should be recalculated according to used diaphragm and AGC.


AGC: Automatic Gain Control; yes/no, can be switched off?, manual control of gain.


Shutter: commonly from 1/50 s to 1/10000 s.


 Cameras, user’s view (2)
51/1

Format: size of the photosensitive chip. Given either in inches of the equivalent vidicon tube


diameter or in mm. 1/2” corresponds to 4.8×6.5 mm.

Shape of a pixel: square pixel vs. non-square pixel.


Output for automatic diaphragm:



AWB: Automatic White Balance. Changes ratio of R and B with respect to G.



Gama correction: fixed/adjustable. Direct signal γ = 1. Typically γ = 0, 45 (enhances



black). Compensates intensity conversion function of the CRT (Cathode Ray Tube) and
adjusts it to the sensitivity of a human eye.
Lens thread: C mount / CS mount.

 Interlaced/non-interlaced scanning
52/1

1 768 1 768

1 1

2 2

3 3

4 4

574 574

575 575

576 576

Interlaced. Non-interlaced.
 Signal, interlaced/non-interlaced scanning
53/1

field frame frame

odd even odd even odd even odd even

… ~ … …
1 3 5 575 2 4 6 576 1 2 3 576

Interlaced. Non-interlaced.
 Electronic shutter
54/1

Shortened exposition is used either if there is too much light or if fast events have to be captured.

I exposition

t
frame (noninterlaced)
field (interlaced)
 Flash light and suppression of ambient light
55/1
I
The shutter time is shortened.


ambient light intensity The instant of the flash is set when the shutter


t is open.
I
flash intensity
The ration between the integral of ambient in-


tensity during the shutter opening and integral
t of the flash intensity gives the influence of am-
I
bient light.
exposition

LED are often used as cheap ‘flash light’.


t
I

flash contribution

ambient light contribution

t
 Color cameras setups
56/1

Manual change of color filters in front of the monochromatic camera lens.



Three chip cameras – an incoming light is divided to a appropriate chip using color filters and


semitransparent mirrors.
One chip camera has filters directly on a chip. Spatial resolution in color resolution is smaller


than coresponds to the number of pixels.

Arrangement of color filters in single chip cameras
57/1

R G R G C Y C Y

G B G B M G M G

R G R G C Y C Y

G B G B M G M G

Additive color model. Subtractive color model.

 Color scanner
58/1
scanned document glass

Illuminant mirror lens chip movement direction

c
 (str
aigh
t lin
e) r
ay

pinhole
three-dimensional
scene image p
lane
a b c
a1

n1
n2

a2
 object plane image plane

principal point image focal

point
d

object focal
point

z f f z’
principal plane
y

y’

z f f z’
y

y’

z f f z’
 image
focal point

object
focal point

z f f z’
principal plane 1 principal plane 2
 Collimator
Chip

Light source Lens

Diaphragm
Object
 Exit pupil - image of the aperture as
seen from the image trough the lens
Image
Object
plane
plane

Object in
a scene

Optic
axis

Projected
Physical object
aperture
 aperture
stop

field of focus
principal
point
d

image
focal point
f z

object space image space

 aperture stop

depth of field
circle of the
admissible defocus

object space image space

 image
plane

f
pincushion barrel
 (x,y)
(x’,y’) Dy
Dx
r
(x0 ,y0 )
x

y
 1 768

574

575

576
 1 768

574

575

576
field frame

odd even odd even odd even odd even

… ~ …
1 3 5 575 2 4 6 576
 frame

…
1 2 3 576
I exposition

t
frame (noninterlaced)
field (interlaced)
I

ambient light intensity

t
I
flash intensity

t
I

exposition

t
I

flash contribution

ambient light contribution

t
R G R G

G B G B

R G R G

G B G B
C Y C Y

M G M G

C Y C Y

M G M G
 scanned document glass

Illuminant mirror lens chip movement direction