2.

3 The Digital Camera

EECS 4422/5323 Computer Vision !1 J. Elder

74 Image
Computer Sensing
Vision: Pipeline
Algorithms (Simplified)
and Applications (September 3, 2010 draf

Figure 2.23 Image sensing pipeline, showing the various sources of noise as well as typic
digital post-processing steps.
EECS 4422/5323 Computer Vision !2 J. Elder
 Outline
❖ The Sensor

❖ Sampling & Aliasing

❖ Colour Coding

EECS 4422/5323 Computer Vision !3 J. Elder

 Outline
❖ The Sensor

❖ Sampling & Aliasing

❖ Colour Coding

EECS 4422/5323 Computer Vision !4 J. Elder

 Sensor
❖ CCD (Charge-Coupled Device)

๏ Photons accumulated in each active well.

๏ Then charge transferred from well to well (“bucket brigade”) until deposited at sense
amplifiers

❖ CMOS (Complementary Metal Oxide Semiconductors)

๏ photons directly affect conductivity of a photodetector

๏ Each photodetector can be selectively gated and amplified

74 Computer Vision: Algorithms and Applications (September 3, 2010

๏ Read out using multiplexing scheme

❖ Most digital cameras now use CMOS.

Figure 2.23 Image sensing pipeline, showing the various sources of noise as well as ty
EECS 4422/5323 Computer Vision !5 digital post-processing steps. J. Elder
 Shutter Speed
❖ Measured in fractions of a second (e.g., 1/125, 1/60, 1/30,…)

❖ Controls the amount of light integrated by the sensor

❖ Faster shutter speeds prevent ‘camera shake’ and reduce motion

blur but will be noisier unless scene is well illuminated.

❖ Need to use a tripod for slower shutter speeds!

EECS 4422/5323 Computer Vision !6 J. Elder

 Sampling Pitch & Fill Factor
❖ Sampling pitch is the physical spacing between adjacent sensor cells.

❖ For a fixed chip size, smaller pitch means higher resolution (good!) but less light per
pixel (bad!)

Detector aperture width

Fill factor !
Sampling pitch

EECS 4422/5323 Computer Vision !7 J. Elder

 Chip Size
❖ Chip widths can vary from around ~1/4” to ~1.1”

❖ Generally this is less than the 35mm width of a standard film frame.

❖ Our understanding of focal lengths (e.g., a standard 50mm lens) is based on using
35mm film

❖ To adapt this to a digital camera we must scale by the ratio of the sensor widths.

EECS 4422/5323 Computer Vision !8 J. Elder

umber pixel coordinates using integer values, say [0, W ) ⇥ [0, H), the focal length
ra center (cx , cy ) in (2.59) can be expressed as pixel values. How do these quan-
Focal Lengths
to the more familiar focal lengths used by photographers?
2.10 illustrates the relationship between the focal length f , the sensor width W ,
❖ Focal length can be measured either in pixels or in mm.
d of view ✓, which obey the formula 2.1 Geometric primitives and transformations

 1
✓ W W ✓
tan = or f = tan . (2.60) W/2
2 2f 2 2
θ/2 f Z
tional film cameras, W = 35mm, and hence f is also expressed in millimeters. (x,y,1)
Optical centre
work with digital images, it is more convenient to express W in pixels so that the (X,Y,Z)
h f can be used directly in the calibration matrix K as in (2.59).
r possibility is to scale the pixel coordinates so that theyFigure
go from2.10 [Central along showingSensor
1, 1)projection, planebetween the 3D and 2
the relationship
image dimension and [ a 1 , a 1 ) along the shorter axis, nates,where
p and x,a as well as the
1 is the relationship between the focal length f , image wid
the field of view ✓.
ct ratio (as opposed to the sensor cell aspect ratio introduced earlier). This can be
❖ Example: What focal length would give me the equivalent of a 50mm lens for the
FLIR BlackFly S BFS-PGE-122S6C-C? Figure 2.9 shows how these quantities can be visualized as part of a simplifie
model. Note that now we have placed the image plane in front of the nodal point (
center of the lens). The sense of the y axis has also been flipped to get a coordina
❖ Sensor width: 1.1” = 27.94mm compatible with the way that most imaging libraries treat the vertical (row) coordi
tain graphics libraries, such as Direct3D, use a left-handed coordinate system, whic
to some confusion.

A note on focal lengths

The issue of how to express focal lengths is one that often causes confusion in imp
computer vision algorithms and discussing their results. This is because the fo
depends on the units used to measure pixels.
If we number pixel coordinates using integer values, say [0, W ) ⇥ [0, H), the fo
f and camera center (cx , cy ) in (2.59) can be expressed as pixel values. How do th
EECS 4422/5323 Computer Vision !9 J. Elder
tities relate to the more familiar focal lengths used by photographers?
 Analog Gain
❖ May be controlled through automatic gain control logic

❖ Can also be adjusted through ISO setting

❖ Higher gain allows faster shutters speeds (less motion blur) and/or smaller apertures
(greater depth of field).

❖ But at the expense of higher sensor noise!

74 Computer Vision: Algorithms and Applications (September 3, 2010 d

Figure 2.23 Image sensing pipeline, showing the various sources of noise as well as ty
digital post-processing steps.

EECS 4422/5323 Computer Vision !10 J. Elder

Light falling on an imaging sensor is usually picked up by an active sensing area,
 Sensor Noise
❖ May include

๏ fixed pattern noise

๏ dark current noise

๏ shot noise

๏ amplifier noise

๏ quantization noise

❖ Increases with sensor gain

❖ Can be estimated (Assignment 1) by

๏ Measuring variability when irradiance is constant

๏ Differencing two images taken in rapid succession

EECS 4422/5323 Computer Vision !11 J. Elder

 Outline
❖ The Sensor

❖ Sampling & Aliasing

❖ Colour Coding

EECS 4422/5323 Computer Vision !12 J. Elder

 Sampling & Aliasing
❖ The optical signal is continuous, containing arbitrarily high spatial frequencies.

❖ The sensor is spatially sampling this signal at discrete locations determined by the
sampling pitch.

❖ If the image is not low-pass filtered, aliasing will result: high frequency content will
be inextricably mixed with low frequency content in the digital image.

❖ Example: sampling rate fs = 2

78 Computer Vision: Algorithms and Applications (Sep

− sin ( 2π (5 / 4)x )
* =
sin ( 2π (3 / 4)x )
f = 3/4 f = 5/4

Figure 2.24 Aliasing of a one-dimensional signal: The blue sine wav

Vision sine wave at f = 5/4 !1have
EECS 4422/5323 Computer red 3 the same digital samples, when sampled
J. Elder
 Nyquist Limit
❖ Shannon’s sampling theorem: sampling rate must be at least twice the maximum
frequency in the signal.
78 Computer Vision: Algorithms and Applications (Septem
fs ≥ 2 fmax

− sin ( 2π (5 / 4)x )
* =
sin ( 2π (3 / 4)x )
f = 3/4 f = 5/4

Figure 2.24 Aliasing of a one-dimensional signal: The blue sine wave

red sine wave at f = 5/4 have the same digital samples, when sampled a
convolution with a 100% fill factor box filter, the two signals, while no
magnitude, are still aliased in the sense that the sampled red signal loo
lower magnitude version of the blue signal. (The image on the right is s
visibility. The actual sine magnitudes are 30% and 18% of their origina

Harry Nyquist (1889 - 1976) Claude Shannon (1916 - 2001)

from its instantaneous samples
EECS 4422/5323 Computer Vision !14
must be at least twice the highest frequenc
J. Elder
 Effect of the Fill Factor
❖ Each pixel is actually the result of integrating light over a small square, the size of
which is determined by the sampling pitch and fill factor.

❖ This serves to attenuate high frequencies.

❖ However, the Fourier transform of this ‘boxcar’ filter falls only as 1/f, and thus high
frequencies, while attenuated, are still present and cause aliasing.
2.3 The digital camera 79

Original
(a)Image Boxcar with(b)
25% fill factor Boxcar with (c)
100% fill factor High-quality lowpass filter
(d)

Figure 2.25 Aliasing of a two-dimensional signal: (a) original full-resolution image; (b)
downsampled 4⇥ with a 25% fill factor boxSubsampled by a factor of 44⇥ with a 100% fill
filter; (c) downsampled
factor
EECS box filter;
4422/5323 Computer downsampled 4⇥ with a high-quality
(d)Vision !15 9-tap filter. Notice how the higher
J. Elder
 Point Spread Function (PSF)
❖ The pre-filtering of the optical signal is determined by:

๏ The optical system (diffraction, focal blur)

๏ The integration area (sampling pitch and fill factor)

๏ Integrated optical anti-aliasing filters

❖ If together these filters adequately attenuate frequencies above the Nyquist limit,
visible aliasing will be minimal.

EECS 4422/5323 Computer Vision !16 J. Elder

 End of Lecture
Sept 19, 2018

EECS 4422/5323 Computer Vision !17 J. Elder

 Outline
❖ The Sensor

❖ Sampling & Aliasing

❖ Colour Coding

EECS 4422/5323 Computer Vision !18 J. Elder

 Colour Sampling
❖ Natural scenes reflect light rays over a wide continuum of wavelengths.

❖ Yet most colour cameras have only 3 discrete types of sensor elements tuned to 3
different colours (wavelengths): red, green and blue.

2.3 The digital camera

❖ Similarly, most colour displays have 3 distinct types of light-emitting elements, also
emitting at red, green and blue wavelengths.

❖ Why is this? Why should this be sufficient?

EECS 4422/5323 Computer Vision !19

(a) J. Elder
 Trichromacy
❖ The human retina has (at most) 3 distinct photoreceptive cone types, each tuned to a
specific band of wavelengths.

❖ This means that human colour vision is 3-dimensional

❖ Any 3 colour vectors that span this 3D space are sufficient to generate the entire space
of colours that we experience.

Cone sensitivity

Human cone mosaic Wavelength (nm)

EECS 4422/5323 Computer Vision !20 J. Elder

 CIE RGB Representation
❖ Colour representation standard formed in 1931

❖ Based on human behavioural colour matching to monochromatic test colours.

❖ Subjects adjusted the relative amplitudes of 3 monochromatic primaries:

๏ Red (700.0nm)

๏ Green (546.1nm)

๏ Blue (435.8nm)
82 Computer Vision: Algorithms and Applications (Septemb
❖ Note that reproducing pure spectra in the blue-green range requires a negative amount of red
light!
0.4 2.0

r 1.8
0.3 g 1.6

b 1.4
0.2 1.2
1.0
0.8
0.1
0.6
0.4
0.0
360 400 440 480 520 560 600 640 680 720 760 0.2
0.0
-0.1 360 400 440 480 520 560 600

EECS 4422/5323 Computer Vision (a) !21 (b)

J. Elder
 certain pure spectra in the blue–green range, a negative amount of red light has to be added,
i.e., a certain amount of red has to be added to the color being matched in order to get a color
CIE XYZ Representation
match. These results also provided a simple explanation for the existence of metamers, which
are colors with different spectra that are perceptually indistinguishable. Note that two fabrics
❖ To or
avoid
paintthis negative
colors light problem,
that are metamers thelight
under one CIEmay
created a new
no longer XYZ
be so standard
under differentbased on a
lighting.
linear transformation of the RGB standard.
Because of the problem associated with mixing negative light, the CIE also developed a
❖ In the
newXYZ representation,
color space called XYZ,the Y channel
which contains corresponds to (achromatic)
all of the pure spectral luminance.
colors within its positive
octant. (It also maps the Y axis to the luminance, i.e., perceived relative brightness, and maps
❖ Note that,
pure unlike
white the CIE (equal-valued)
to a diagonal RGB space, the XYZThe
vector.) dimensions are ‘imaginary’
transformation from RGB toprimary
XYZ is
colours
given having
by no physical reality.
2 3 2 32 3
X 0.49 0.31 0.20 R
6 7 1 6 76 7
4 Y 5 = 4 0.17697 0.81240 0.01063 54 G 5. (2.103)
Computer Vision: Algorithms and Applications (September 3, 2010 draft)
0.17697
Z 0.00 0.01 0.99 B
While the official definition of the CIE XYZ standard has the matrix normalized so that the
2.0
Y value corresponding
1.8 to pure red is 1, a more commonly used form is to omit the leading
r x
g 1.6 y
b 1.4
z
1.2
1.0
0.8
0.6 In MATLAB:
0.4 • rgb2xyz(rgb)
520 560 600 640 680 720 760 0.2 • xyz2rgb(xyz)
0.0
360 400 440 480 520 560 600 640 680 720 760

(a)
EECS 4422/5323 Computer Vision
(b)
!22 J. Elder
 obtain the resulting (x̄( ), ȳ( ), z̄( )) curves shown in Figure 2.28b. Notice how all three
spectra (color matching functions) now have only positive values and how the ȳ( ) curve
Chromaticity
matches that of the luminance Coordinates
perceived by humans.
If we divide the XYZ values by the sum of X+Y+Z, we obtain the chromaticity coordi-
nates
digital camera x=
X
, y=
Y
, z=
Z
, (2.104)
X +Y +Z X +Y +Z X +Y +Z
which sum up to 1. The chromaticity coordinates discard the absolute intensity of a given
color sample and just represent its pure color. If we sweep the monochromatic color pa-
rameter in Figure 2.28b from = 380nm to = 800nm, we obtain the familiar chromaticity
diagram shown in Figure 2.29. This figure shows the (x, y) value for every color value per-
ceivable by most humans. (Of course, the CMYK reproduction processMonochromatic colours
in this book does not
actually span the whole gamut of perceivable colors.) The outer curved rim represents where
all of the pure monochromatic color values map in (x, y) space, while the lower straight line,
which connects the two endpoints, is known as the purple line.
A convenient representation for color values, when we want to tease apart luminance
and chromaticity, is therefore Yxy (luminance plus the two most distinctive chrominance
components).

L*a*b* color space

While the XYZ color space has many convenient properties, including the ability to separate
luminance from chrominance, it does not actually predict how well humans perceive differ-
ences in color or luminance.
2.29 CIE chromaticity
EECS 4422/5323 Computer Vision
diagram, showing
!23
colors and their correspondi
J. Elder
 L*a*b* Space
❖ Human luminance/colour sensitivity is roughly logarithmic

๏ We can perceive relative differences of about 1%.

84 Since XYZ space

❖ is linear
Computer withAlgorithms
Vision: the amplitude of the stimulus,
and Applications it does 3,
(September not2010 draft)
predict human perception of colour and luminance differences.
Because the response of the human visual system is roughly logarithmic (we can perceive
❖ L*a*b
relative space isdifferences
luminance a nonlinearof remapping of XYZ
about 1%), the space that
CIE defined renders re-mapping of the
a non-linear
XYZ differences in L*a*b*
space called luminance
(alsoand chrominance
sometimes more perceptually
called CIELAB), uniform.in luminance
where differences
or chrominance are more perceptually uniform.19 100

The L* component of lightness is defined as NB: Error in textbook 80

✓ ◆ 60
Y
L⇤ = 116f ,−16 (2.105)

L*
40
Yn
20

where Yn is the luminance value for nominal white (Fairchild 2005) and 0
0 20 40 60 80 100
( Y
t1/3 t> 3
f (t) = (2.106)
t/(3 ) + 2 /3 else,
2 In MATLAB:
• rgb2lab(rgb)
Yn =root δ = 6 / 29. • lab2rgb(lab)
is a finite-slope approximation towhere
the cube 100, with = 6/29. The resulting 0 . . . 100 scale
• xyz2lab(xyz)
roughly measures equal amounts of lightness perceptibility. • lab2xyz(lab)
EECS 4422/5323 Computer Vision !24 J. Elder
In a similar fashion, the a* and b* components are defined as
 Appendix A: Spectral Response Curves

FL2-03S2M Colour Cameras

❖ Spectral sensitivities vary from camera to camera.

❖ It’s the job of the camera firmware to convert these proprietary sensor responses to
standard colour values.

❖ For some professional and scientific cameras, the manufacturer provides the spectral
responses.

Spectral response curves for Point Grey Flea2

FL2-03S2C

EECS 4422/5323 Computer Vision Revised 10-Jun-11

Copyright (c) 2008, 2009, 2010, 2011 Point Grey
!25 48 J. Elder
Research Inc.
 Colour Filter Arrays
❖ Colour camera sensors consist of a mosaic of sensing elements covered by different
coloured filters.

86 design is the
❖ The most common Computer
Bayer Vision:
pattern,Algorithms andof
consisting
74
Applications (September 3, 2010 draft)
Computer Vision: Algorithms and Applications (September 3, 2010 dr

๏ 25% red G R G R rGb Rgb rGb Rgb

B G B G rgB rGb rgB rGb

๏ 50% green
G R G R rGb Rgb rGb Rgb

๏ 25% blue B G B G rgB rGb rgB rGb

Figure 2.23 Image sensing pipeline, showing the various sources of noise as well as typ

❖ The greater density of green elements (a)

reflects the fact that digital post-processing steps. (b)
Light falling on an imaging sensor is usually picked up by an active sensing area, i
Figure 2.30 Bayer RGB pattern: (a) color filter array layout; (b) interpolated pixel values,
grated for the duration of the exposure (usually expressed as the shutter speed in a fractio

๏ perceived luminance depends primarily

valueson the green channel a second, e.g., 125 , 60 , 30 ), and then passed to a set of sense amplifiers . The two main ki
1 1 1

with unknown (guessed) shown as lower case. of sensor used in digital still and video cameras today are charge-coupled device (CCD)
complementary metal oxide on silicon (CMOS).
In a CCD, photons are accumulated in each active well during the exposure time. Th
in a transfer phase, the charges are transferred from well to well in a kind of “bucket briga
๏ visual acuity is The
far greater for luminance
most commonly than colour
used pattern in color cameras today is the Bayer pattern (Bayer
until they are deposited at the sense amplifiers, which amplify the signal and pass i
an analog-to-digital converter (ADC).10 Older CCD sensors were prone to blooming, w
1976), which places green filters over half of the sensors (in a checkerboard pattern), and red
charges from one over-exposed pixel spilled into adjacent ones, but most newer CCDs h
anti-blooming technology (“troughs” into which the excess charge can spill).
and blue filters
❖ Interpolation of missing colourover the remaining
values ones (Figure
at each pixel known 2.30). The reason that there are twice as many
as demosaicing.
In CMOS, the photons hitting the sensor directly affect the conductivity (or gain) o
photodetector, which can be selectively gated to control exposure duration, and locally
green filters as red and blue is because the luminance signal is mostly determined by green
plified before being read out using a multiplexing scheme. Traditionally, CCD sen
values and the visual system is much more sensitive to high frequency detail in luminance
outperformed CMOS in quality sensitive applications, such as digital SLRs, while CM
was better for low-power applications, but today CMOS is used in most digital cameras.
than in chrominance (a fact that is!26exploited in color image compression—see Section 2.3.3).
EECS 4422/5323 Computer Vision J. Elder
The main factors affecting the performance of a digital image sensor are the shutter spe
 White Balance
❖ The colour of the irradiance received from a surface depends upon both the colour of the
surface material and the colour of the illuminant.

❖ Standard colour systems assume a specific illuminant (e.g., daylight)

❖ If the illuminant deviates from this standard, the resulting photo (out of context) may look
oddly coloured.

❖ White balance is an attempt to reduce this effect by moving the white point of the image
closer to pure white (equal RGB values).

❖ Can achieve this by scaling the R, G74and B valuesComputer

by different amounts
Vision: Algorithms and(Assignment 1). 3, 2010
Applications (September

EECS 4422/5323 Computer Vision !27 J. Elder

 White Balance Example

EECS 4422/5323 Computer Vision !28 J. Elder

 White Balance Results

Original White Balanced

EECS 4422/5323 Computer Vision !29 J. Elder

brightness was characterized by a number called gamma ( ), since
responds to the more noise-sensitive region of the visual system.

B=V , Gamma
(2.110)
wist, i.e., they use a general 3 ⇥ 3 color transform matrix.21 Exer-
compensate Cameras
❖ for typically
this effect,
me of these issues. compress
the electronics theTV
in the intensity
camera(luminance)
would of pixel values through an
ance Y through inverse ‘gamma
an inverse function’:
gamma,
1
Y =Y ,
0
(2.111)
and white television,
where γthe
! phosphors in the CRT used to display
2.2.
= 0.45.
on-linearly to their input voltage. The relationship between the
gnal through this non-linearity before transmission had a beneficial
rightness wasThis
❖ roughly by
characterized cancels the called
a number gamma function
gamma applied to RGB values by display
( ), since
uring transmission
systems(remember,
prior to these were analog days!) would be
rendering:
e gamma at the receiver) in the darker regions of the signal where
22
B=V , (2.110)
e 2.31). (Remember that our visual system is roughly sensitive to
inance.) ❖forHowever
ompensate this effect,the
thenonlinear
electronicsrelationship between
in the TV camera wouldencoded RGB values and physical
nce Y throughintensities
an inverse complicates
gamma, physics-based computer vision algorithms, which often
emember the early days of color
assume accesstelevision
to will naturally
linear think of the
luminance hue adjustment
values.
could produce truly0 bizarre1 results.
Y =Y , (2.111)
companding was the basis of the Dolby noise reduction systems used with audio

0.45.

Output intensity
nal through this non-linearity before transmission had a beneficial
ring transmission (remember, these were analog days!) would be
gamma at the receiver) in the darker regions of the signal where
2.31).22 (Remember that our visual system is roughly sensitive to
nance.)
member the early days of color television will naturally think of the hue adjustment Input intensity
could produce truly bizarre results.
EECS 4422/5323 Computer Vision !30 J. Elder
ompanding was the basis of the Dolby noise reduction systems used with audio
 Compression
❖ All compression algorithms start by separating luma and chroma channels so that
luma can be encoded with higher fidelity.

❖ Block transform stage then breaks image into disjoint blocks (e.g., 8 x 8 pixels) and
codes each using a discrete cosine transform (DCT), which approximates an efficient
coding (principal components) strategy.

❖ Resulting DCT coefficients then coded using a variation of Huffman coding.

❖ Video coding uses predictive (difference) encoding between frames, compensating for
estimated motion in the image.

DCT Basis Functions

EECS 4422/5323 Computer Vision !31 J. Elder

 Outline
❖ The Sensor

❖ Sampling & Aliasing

❖ Colour Coding

EECS 4422/5323 Computer Vision !32 J. Elder