Imaging system fundamentals

Gerald C. Holst

Downloaded From: http://opticalengineering.spiedigitallibrary.org/ on 12/28/2016 Terms of Use: http://spiedigitallibrary.org/ss/termsofuse.aspx

Optical Engineering 50(5), 052601 (May 2011)

Imaging system fundamentals

Gerald C. Holst
JCD Publishing Company
2932 Cove Trail
Winter Park, Florida 32789
Email: [email protected]

Abstract. Point-and-shoot, TV studio broadcast, and thermal infrared

imaging cameras have significantly different applications. A parameter
that applies to all imaging systems is F/d, where F is the focal ratio,
is the wavelength, and d is the detector size. F/d uniquely defines the
shape of the camera modulation transfer function. When F/d<2, aliased
signal corrupts the imagery. Mathematically, the worst case analysis assumes that the scene contains all spatial frequencies with equal amplitudes. This quantifies the potential for aliasing and is called the spurious
response. Digital data cannot be seen; it resides in a computer. Cathode
ray tubes, flat panel displays, and printers convert the data into an analog
format and are called reconstruction filters. The human visual system is
an additional reconstruction filter. Different displays and variable viewing
distance affect the perceived image quality. Simulated imagery illustrates
different F/d ratios, displays, and sampling artifacts. Since the human visual system is primarily sensitive to intensity variations, aliasing (a spatial
frequency phenomenon) is not considered bothersome in most situations.

C 2011 Society of Photo-Optical Instrumentation Engineers (SPIE). [DOI: 10.1117/1.3570681]

Subject terms: sampling; modulation transfer function; imaging system; aliasing;

reconstruction; spurious response.
Paper 110144TR received Feb. 14, 2011; revised manuscript received Mar. 5,
2011; accepted for publication Mar. 7, 2011; published online May 10, 2011.

1 Introduction
Point-and-shoot, TV studio broadcast, and thermal infrared
imaging cameras have significantly different applications.
Common to all is an optical system and detector array which
are linked together by F/d, where F is the focal ratio, is
the wavelength of interest, and d is the detector size. In the
frequency domain, it is the ratio of the detector cutoff to the
optical cutoff. F/d uniquely defines the shape of the camera
modulation transfer function (MTF).
The MTF is the primary parameter used for system design,
analysis, and specifications. It describes how sinusoidal patterns propagate through the system. Because any scene can
be decomposed into a Fourier series, the MTF approach indicates how imagery will appear on the display. In general,
images with higher MTFs are judged as having better image
quality. However, there is no single ideal MTF shape that
provides the best image quality.
Sampling is an inherent feature of all electronic imaging systems. The scene is spatially sampled in both directions by the discrete detector locations. It creates ambiguity
in target edges and produces moire patterns when viewing
periodic targets. Aliasing becomes obvious when image features approach the detector size. It distorts the image and the
amount of distortion is scene dependent. It is pronounced
when viewing periodic structures and these are rare in nature. Aliasing is seldom reported when reproducing natural
scenery.
Mathematically, worst case analysis assumes that the
scene contains all spatial frequencies with equal amplitudes.
This quantifies the potential for aliasing and is called the spurious response. However, real scenes have a limited spectrum
and image quality is a subjective measure. This means there
is no method of validating the theory with imagery. MTF
C 2011 SPIE
0091-3286/2011/$25.00 

Optical Engineering

theory and sampling issues are just two slices through the
multidimensional image quality space. They provide guidance for camera design but do not uniquely quantify image
quality.
Assuming the detectors are in a rectangular lattice, the fill
factor (Fig. 1) is the ratio of areas
Fill factor = FF =

d H dV
AD
=
.
dCCH dCCV
APIXEL

(1)

The photosensitive area (AD ) is dH dV . Larger detectors

collect more photons (higher sensitivity). Unfortunately, the
current trend is to make smaller detectors. The detector
center-to-center spacing (pitch) defines the sampling frequency (uS = 1/dCCH and vS = 1/dCCV ) and pixel area (APIXEL
= dCCH dCCV ). Note that the detector size and pixel size can
be quite different and this leads to confusion when describing
system performance.
For convenience, a one-dimensional (horizontal) approach is used with FF = 1. Then d = dH = dCC = dCCH .
The equations and graphs are easily modified for finite fill
factors.
As the world becomes digital, we tend to ignore linear
system theory (developed for analog systems) and sampling
theory (analog-digital-analog conversion). The analog output
of each detector is immediately quantized. The digital data is
processed (image processing), digitally transmitted, and then
sent to a digital display. We cannot see digital data. It must be
transformed into analog data. Each display medium [cathode
ray tube (CRT), flat-panel display, or printer] modifies the
image in a different way.
2 Linear Shift Invariant Systems
Four conditions must be met to achieve a linear shift invariant (LSI) system: 1. the radiation is incoherent; 2.
signal processing is linear; 3. the system mapping is

052601-1

May 2011/Vol. 50(5)

Downloaded From: http://opticalengineering.spiedigitallibrary.org/ on 12/28/2016 Terms of Use: http://spiedigitallibrary.org/ss/termsofuse.aspx

Holst: Imaging system fundamentals

For small increments, this becomes the convolution integral

dH

i(x, y)

Detector

 

Pixel

dCCV

dV

and is symbolically represented by the two-dimensional convolution operator **

i(x, y) = o(x, y) h SPATIAL (x, y).

dCCH
Fig. 1 Detector and pixel relationship.

single-valued; and 4. the image is spatially invariant. An

LSI system only modifies the amplitude and phase of the
target.
Nonlinear image processing, present in nearly every imaging system, violates the one-to-one mapping requirement. For
convenience, image processing will be considered a linear
process. Single-valued mapping only occurs with non-noisy
and nonquantized systems. No system is truly noiseless, but
can be approximated as one when the signal-to-noise ratio
(SNR) is high.
A sampled-data system may be considered globally shiftinvariant on a macroscale. As a target moves from the top
to the bottom of the field of view, the image also moves
from the top to the bottom. On a microscale, moving a point
source across a single detector does not change the detector
output. An imaging system system is not shift-invariant on a
microscale.
In spite of these disclaimers, an imaging system is treated
as quasilinear over a restricted operating region to take advantage of the wealth of mathematical tools available. For
mathematical convenience, an electronic imaging system is
characterized as a linear spatial-temporal system with respect
to both time and two spatial dimensions. Although space is
three-dimensional, an imaging system displays only two dimensions.
2.1 Linear System Theory
An object can be thought of as the sum of an infinite array
of impulses located inside the object boundaries. Thus, an
object can be decomposed into a two-dimensional array of
weighted Dirac delta functions, (x x ), (y y )
o(x, y) =

o(x  , y  ) (x x  )(y y  )x  y  .

(2)

I (u, v) = O(u, v)HSPATIAL (u, v).

i(x, y) =
h SPATIAL {o(x  , y  ) (x x  )(y y  )x  y  }.

x  = y  =

(3)
Optical Engineering

(6)

HSPATIAL (u, v) is the optical transfer function and is usually

labeled as OTF(u, v). The MTF is the magnitude and the phase
transfer function (PTF) is the phase of the complex-valued
OTF. Symbolically
OTFSPATIAL (u, v) = MTFSPATIAL (u, v) e j PTF(u,v) .

(7)

Spatial frequency can be defined in image-space (at the

focal plane) with units of cycles/mm or in object-space (cycles/mrad). They are related by u = uo /f l. To maintain dimensionality, if uo is measured in cycles/mrad then the focal
length, f l, must be measured in meters to obtain u in cycles/mm.
The MTF and PTF alter the image as it passes through
the system. For LSI systems, the PTF is of no special interest since it only indicates a spatial or temporal shift with
respect to an arbitrarily selected origin. An image where the
MTF is drastically altered is still recognizable whereas large
nonlinearities in the PTF can destroy recognizability.
The point spread function is assumed to be separable
in Cartesian coordinates (taken as horizontal and vertical).
Separability1 reduces the analysis so that complex calculations that include cross-terms are not required. Separability
in Cartesian coordinates requires that
h SPATIAL (x, y) = h SPATIAL (x)h SPATIAL (y).

(8)

Separability in polar coordinates requires

An optical system produces an image and the process is

symbolically represented by the operator hSPATIAL {}

(5)

The function hSPATIAL (x, y) is the optical systems response

to an input impulse. The resulting image is the point spread
function (PSF). Equation (4) is simply the summation of all
the impulse responses. If i(x, y) passes through another LSI
system, i  (x, y) = i(x, y) h SPATIAL (x, y). As the number
of LSI systems increases, multiple convolution calculations
become tedious.
Since convolution and multiplication are Fourier transform pairs, convolution in space becomes a multiplication in
the frequency domain

x  = y  =

o(x  , y  )h SPATIAL {(x x  )(y y  )} d x  dy  , (4)

h SPATIAL (r, ) = h SPATIAL (r )h SPATIAL ( ).

(9)

The PSF of an aberration-free optical system can be characterized by a function that is separable in polar coordinates.
The detector is assumed to be rectangular. Its PSF is separable
in Cartesian coordinates, but is not separable in polar coordinates. The collective PSF of the detector and the optics is not
separable in either polar or Cartesian coordinates. The errors

052601-2

May 2011/Vol. 50(5)

Downloaded From: http://opticalengineering.spiedigitallibrary.org/ on 12/28/2016 Terms of Use: http://spiedigitallibrary.org/ss/termsofuse.aspx

Holst: Imaging system fundamentals

HSYSTEM (u) = HSPATIAL (u)HELECTRONICS ( f e u),

MTFSYSTEM (u, v) =

0.6
0.4
0.2
0

0.2

0.4

0.6

0.8

Spatial frequency (u/uC)

Fig. 2 Optics MTF for a clear (unobscured) circular aperture.

MTF is
MTFSYSTEM = MTFOPTICS MTFDETECTOR MTFDISPLAY .
(13)
Camera manufacturers have no control over how an observer will process the imagery and therefore their analyses
generally omit the display MTF. The perceived MTF depends
upon display characteristics and the human visual system
(HVS) interpretation
MTFPERCEIVED = MTFSYSTEM MTFHVS ,

(11)

and is used for predicting image quality metrics. These metrics will not be discussed here. However, MTFHVS plays an
important role when viewing imagery (discussed in Sec. 5).

A system is composed of many components that respond

to spatial and temporal signals. Here lies the advantage of
working in the frequency domain. If multiple LSI components exist in the spatial and/or electronic domains, the
individual MTFs can be multiplied together. Equivalently,
multiple convolutions in space or time are equivalent to multiplications (or cascading) in the frequency domain. For independent MTFs
n 
m


0.8

(10)

and
HSYSTEM (v) = HSPATIAL (v).

MTF OPTICS

associated with separability are usually small2 and therefore

most analyses use the Cartesian separability approximation.
The detector array is assumed to be composed of rectangular (or square) detectors spaced in a rectangular (or square)
grid (Fig. 1). Any other spacing (e.g., hexagonal) can only
be analyzed on a case-by-case basis.3
The electronic imaging system response consists of both
an optical response and electronic response. Time and
spatial coordinates are treated separately: hSYSTEM (x, y, t)
= hSPATIAL (x, y)hELECTRICAL (t). This is reasonable. Optical
elements do not generally change with time and therefore
are characterized only by spatial coordinates. Similarly, electronic circuitry exhibits only temporal responses. The detector provides the interface between the spatial and temporal
components, and its response depends on both temporal and
spatial quantities. The conversion of two-dimensional optical information to a one-dimensional electrical response
assumes a linear photodetection process. Implicit in the detector response is the conversion from input photon flux to
output voltage (or amps).
The electronic circuitry is assumed to modify the horizontal signal only (although this depends on the system design).
With appropriate scaling, the electronic frequencies can be
converted into spatial frequencies. This is symbolically represented by fe u:

MTFSPATIAL (i, u, v)

(14)

2.2 Optics MTF

A complex optical system is replaced with a simple lens
that has the equivalent focal length. For an aberration-free,
radially symmetric optical system, OTFOPTICS is the same
in the horizontal and vertical directions. Since the OTF is
positive, it is labeled as the MTF. In the horizontal direction,
the diffraction-limited MTF for circular aperture (Fig. 2) is

 
 2
u
2 1 u
u
.
MTFOPTICS (u) =

1
cos

uC
uC
uC

i=1 j=1

MTFELECTRONICS ( j, f e u).

When coupled with the three-dimensional noise

parameters4 the MTF uniquely defines system performance.
The MTF determines how the system responds to spatial frequencies. It does not contain any signal intensity
information.
Image formation is straight forward. Over the region that
linear system theory is valid, the scene is transformed into
its frequency components O(u, v). Each frequency is then
multiplied by MTFSYSTEM (u, v) to provide ISYSTEM (u, v).
Then the inverse transform provides ISYSTEM (x, y).
While an imaging system is composed of many subsystems, generally the MTF is dominated by the optics, detector, electronic filters, digital filters, and display
medium. Adding electronic and digital filters to the analysis obscures the fundamentals of image creation. Here,
the electronic and digital filter MTFs are assumed to be
unity over the spatial frequencies of interest. The basic
Optical Engineering

(15)

(12)

The image-space optics cutoff frequency is uC = D/(f l)

= 1/(F), where D is the aperture diameter and F
= f l/D. Because the cutoff frequency is wavelength dependent, Eq. (15) and Fig. 2 are only valid for noncoherent
monochromatic light. The extension to polychromatic light
is lens-specific. Most lens systems are color corrected (achromatized) and therefore there is no simple way to apply this
simple formula to predict the MTF. As an approximation to
the polychromatic MTF, the average wavelength is used to
calculate the cutoff frequency: AVE = (MAX + MIN )/2.
2.3 Detector MTF
The detector OTF cannot exist by itself. Rather, the detector
OTF must also have the optical MTF to make a complete
imaging system. In the horizontal direction, the OTF of a
single rectangular detector is
OTFDETECTOR (u) =

052601-3

sin( du)
.
du

(16)
May 2011/Vol. 50(5)

Downloaded From: http://opticalengineering.spiedigitallibrary.org/ on 12/28/2016 Terms of Use: http://spiedigitallibrary.org/ss/termsofuse.aspx

Holst: Imaging system fundamentals

OTFDETECTOR

0.8
0.6
0.4
0.2
0
-0.2

Fig. 4 Replicated spectra. The ideal reconstruction filter response is

unity up to fN and zero thereafter. It eliminates the replicated spectra
leaving only the analog base band.

-0.4

Spatial frequency (u/uD)

Fig. 3 Detector OTF.

The OTF is equal to zero when u = k/d (Fig. 3). The first
zero (k = 1) is considered the detector cutoff frequency, uD .
It is customary to plot the OTF up to uD . This is probably
done because it is unknown what effect the negative OTF has
on overall image quality. Since the OTF is positive up to uD ,
it is called the MTF. Plotting the MTF up to the first zero
erroneously suggests that the detector does not respond to
frequencies greater than uD . The optical system cutoff limits
the absolute highest spatial frequency that can be faithfully
imaged and not uD . Nevertheless most analyses (and that
considered here) consider the response up to uD only.
2.4 Display MTF
Display specifications are a mix of CRT terminology, video
transmission standards, alphanumeric character legibility,
and graphics terminology. CRTs are low cost with high resolution, wide color gamut, and high luminance. Flat panels do
not have all these attributes. Nevertheless, flat panel displays
will probably replace all CRTs in the near future.
Flat panel displays are assumed to have rectangular pixels. Usually the number of pixels matches the number of
detectors. When referred to image space,
OTFDISPLAY (u) =

sin ( dCC u)
.
dCC u

(17)

As with the detector, the display response is (erroneously)

plotted up to the first zero. When FF = 1 the display MTF is
identical to detector MTF.
3 Sampling
Sampling is an inherent feature of all electronic imaging
systems. The scene is spatially sampled in both directions
by the discrete detector locations. Sampling theory states
that the frequency can be unambiguously recovered for all
input frequencies below Nyquist frequency. After aliasing,
the original signal can never be recovered. The mathematics suggests that aliasing is an extremely serious problem.
Objection-ability depends upon the scene, F/d, dCC , display medium, and viewing distance.
3.1 Sampling Theorem
The sampling theorem as introduced by Shannon5 was applied to information theory. He stated that if a time-varying
function v(t) contains no frequencies higher than fMAX (Hz), it
is completely determined by giving its ordinates at a series of
points spaced 1/(2fMAX ) seconds apart. The original function
Optical Engineering

can be reconstructed by an ideal low-pass filter. Shannons

work is an extension of others,6 and the sampling theorem
is often called the ShannonWhittaker theorem. Reference 7
provides an in-depth discussion on sampling effects.
If sampling occurs every T seconds, the sampling frequency is fS = 1/T. The resultant signal is
v SAMPLE (t) = v(t)s(t),

(18)

where s(t) is the sampling function is equal to (t nT). Since

multiplication in one domain is represented as convolution
in the other, the sampled frequency spectrum is
VSAMPLE ( f e ) = V ( f e ) S( f e ),

(19)

where V( fe ) is the amplitude spectrum of the band-limited

analog signal and S( fe ) is the Fourier transform of the sampler. The transform S( fe ) is a series of impulses at nfS and
is called a comb function. When convolved with V( fe ), the
resultant is a replication of V( fe ) about nfS (n = to
+ ). Equivalently, the sampling frequency interacts with
the signal to create sum and difference frequencies. Any input frequency, fo , will appear as nfS fo after sampling.
Figure 4 illustrates a band-limited system with frequency
components replicated by the sampling process. The base
band ( fH to fH ) is replicated about nfS . To avoid distortion,
the lowest possible sampling frequency is that value where
the first replicated spectrum just adjoins the base band. This
leads to the sampling theorem that a band-limited system
must be sampled at twice the highest frequency ( fS 2fH ).
Nyquist frequency is defined as fN = fS /2.
After digitization, the data reside in data arrays (e.g., a
computer memory location). The signal must be converted
(reconstructed) into an analog signal to be useful. If the original signal was oversampled ( fS 2fH ) and if the reconstruc-

ALIASED OUTPUT

2T

INPUT

3T

4T

5T

6T

Fig. 5 An undersampled sinusoid will appear as a lower frequency

after ideal reconstruction.

052601-4

May 2011/Vol. 50(5)

Downloaded From: http://opticalengineering.spiedigitallibrary.org/ on 12/28/2016 Terms of Use: http://spiedigitallibrary.org/ss/termsofuse.aspx

Holst: Imaging system fundamentals

MTF

Replicated spectra

uS

2uS

3uS

Relative spatial frequency

Fig. 6 Overlapping spectra. The amount of aliasing (and hence image quality) is related to the amount of overlap.

Fig. 8 Reconstruction with a flat panel display (left) and a CRT (right).
See footnote on page 8.

3.2 Aliasing
To apply the above sampling theory to imaging systems, let
fe u, fS uS , and T dCC . As the sampling frequency
decreases, the first replicated spectrum starts to overlap the
base band (Fig. 6). It is the summation of these spectra that
distort the image.
A bar pattern consists of an infinite number of frequencies. While the fundamental may be less than the Nyquist
frequency, higher-order terms will not. These higher-order
terms are aliased and distort the signal. In Fig. 7 the input bar
pattern fundamental is 1.68 uN and the aliased fundamental
is 0.32 uN . Since higher-order frequencies are present, the
reconstructed bars appear more triangular than sinusoidal.
Since aliasing occurs at the detector, the signal must be
band-limited by the optical system to prevent it. This is
achieved by designing a system with F/d 2 or by using an optical low pass filter (OLPF).9 While monochrome
aliasing is tolerable, color aliasing is bothersome. Single chip
color arrays always have a window over the array. The first
impression is that the window is designed to protect the array. This is an ancillary feature. The window is actually a
birefringent crystal that acts as an OPLF. The OPLF reduces
the MTF and reduces image contrast. Since color aliasing is
unacceptable, the reduced MTF is a small penalty to pay.

3.3 Reconstruction
Digital data cannot be seen because it resides in a computer memory. Any attempt to view a digital image requires
a reconstruction filter.10 Most imaging systems rely on the
display medium and HVS to produce a perceived continuous image (discussed in Sec. 5). Display media include laser
printers, half-toning, fax machines, CRTs, and flat panel displays. The display medium creates an image by painting a
series of light spots on a screen or ink spots on paper. The
spot acts as a low pass reconstruction filter. Each display
medium has a different spot size and shape resulting in different frequency responses. The perceived imagery will be
different on each display type.
A flat panel display is not an ideal reconstruction filter.
It passes significant frequency components above uN and
this makes the image blocky or pixelated. A CRT will remove the higher frequencies (above uN ) but also attenuates
the in-band frequencies to create a somewhat blurry image
(Fig. 8). The ideal reconstruction filter abruptly drops to zero
at uN . As illustrated in Fig. 9, a sharp drop in one-domain
produces ringing in the other (Gibbs phenomenon). In these
two figures, the optics and detector MTFs are unity over the
spatial frequencies of interest. This emphasizes how different
reconstruction filters affect image quality.
Moire patterns occur when periodic targets are viewed.
Some published articles provide resolution charts. The sampling artifacts become more noticeable when viewing targets
at an angle with respect to the array axis. Figure 10 illustrates
various artifacts when viewing a TV resolution chart. It compares a Bayer pattern with the Kodak TrueSense color filter
array (CFA) and their respective demosaicking algorithms.

Fig. 7 Aliasing. Input (left) and aliased output (right). An ideal reconstruction filter was used. Imagery created (Ref. 8) by MAVIISS.

Fig. 9 Reconstruction with an ideal reconstruction filter. See footnote

on page 8.

tion filter limits frequencies to fN , then the reconstructed

image can be identical to the original image.
Within an overlapping band ( fS < 2 fH ), there is an ambiguity in frequency. It is impossible to tell whether the reconstructed frequency resulted from an input frequency of fo or
nfS fo (Fig. 5).

Optical Engineering

052601-5

May 2011/Vol. 50(5)

Downloaded From: http://opticalengineering.spiedigitallibrary.org/ on 12/28/2016 Terms of Use: http://spiedigitallibrary.org/ss/termsofuse.aspx

Holst: Imaging system fundamentals

MTFPOST(u)
MTFPRE(u)

MTF

MTFPRE(uS-u)

MTFPOST(u)MTFPRE(u)

MTFPOST(u)MTFPRE(uS-u)

uN

Relative spatial frequency

(a)

(b)

(c)

(d)

Fig. 12 Practical reconstruction filter. MTFPOST (u)MTFPRE (uS u )

was created by the sampling process. It was not in the original scene.

(e)

Fig. 10 Moire patterns vary according to the CFA type and demosaicking algorithm. Details are provided in Ref. 11. The heavy horizontal lines represent the estimated TV resolution afforded by each
camera. The on-line PDF version is in color where color aliasing can
be seen. This figure has been enlarged. At normal viewing distances, the individual pixels cannot be resolved. View this figure from
several feet.

3.5 Spurious Response

In one dimension, the reconstructed image12 is
I (u) = MTFPOST (u)

MTFPRE (nu S u)O(nu S u).

n=0

This figure illustrates the difficulty in quantifying resolution.

It illustrates distorted edges and periodic structures.

(20)

3.4 Resampling
Nearly all analyses focus on the spatial sampling created
by the detector array. If the camera output is analog, then
a frame capture board redigitizes the image for computer
processing. The board can digitize the analog signal at a rate
that is different than uS . Additional samplers in the overall
system can add new sampling artifacts.
Imagery imported into word documents or other programs
are automatically interpolated to provide a smooth continuous image. As the image is enlarged interpolation smoothes
it. This is not so when the image is resampled without interpolation. In Fig. 11, an enlarged portion of a scene captured
by a 10 Mpixel point-and-shoot camera was inserted as a
desktop picture. The flat panel display resampled the scene.
Since the flat panel elements (12801024) did not align with
the camera pixels (36642748), resampling artifacts become
obvious: straight lines appear as a stair step (jaggies).

MTFPRE contains all the MTFs up to the sampler (the detector). For this paper, MTFPRE = MTFOPTICS MTFDETECTOR .
MTFPOST represents all the filters after the sampler. For this
paper MTFPOST = MTFDISPLAY . Equation (20) can be written
as
I (u) = MTFPOST (u)MTFPRE (u)O(u)
+ MTFPOST (u)

MTFPRE (nu S u)O(nu S u).

n=1

(21)
The first term is the image spectrum when no sampling is
present and is called the direct response. Sampling created
the remaining terms and these may be considered an aliasing
metric. If uS 2uH and the reconstruction filter response is
zero for all frequencies greater than uS uH (see Fig. 4), the
second term is zero.
Considering the first fold back frequency (n = 1) and
assuming O(u) = 1, Schade13 defined the spurious response
as

MTFPOST (u)MTFPRE (u S u) du
.
Spurious response = 0 
0 MTFPOST (u)MTFPRE (u) du
(22)

Fig. 11 Original image (left) and image seen on the WindowXP desktop background scene (right). Each picture is 210245 pixels (HV).
The arrows point to the most obvious sampling artifacts. Careful examination reveals numerous others. The on-line PDF version is in
color.
Optical Engineering

The highest scene spatial frequency is limited by the optical cutoff. The upper limit in the denominator is uC . The
highest spatial frequency in the aliased signal (numerator)
is limited by the reconstruction filter. The assumption that
the scene contains all frequencies [O(u) = 1] with equal amplitude is for mathematical convenience. Perhaps Schades
spurious response should be called the potential for aliasing metric. Figure 12 illustrates the spurious response when
a practical post-reconstruction filter is used. The spurious

052601-6

May 2011/Vol. 50(5)

Downloaded From: http://opticalengineering.spiedigitallibrary.org/ on 12/28/2016 Terms of Use: http://spiedigitallibrary.org/ss/termsofuse.aspx

Holst: Imaging system fundamentals

0.7

MWIR design

0.6

MTF (u N)

LWIR design

50

d (m)

40
F/d = 2

Visible

30

0.5
0.4
0.3
0.2
0.1
0

20

0.5

1.5

F /d

10

No aliasing

Fig. 14 MTFOPTICS (uN )MTFDETECTOR (uN ) when FF = 1.

0
0

10

20

30

40

50

F (m)
Fig. 13 Design space for visible, MWIR, and LWIR cameras (FF =
1). There is no aliasing when F/d 2.

response value depends critically upon the reconstruction

filter used.

4 Optics-Detector Subsystem
Image metrics may be described in the spatial domain where
the optical blur diameter is compared to the detector size or in
the frequency domain where the detector cutoff is compared
to the optics cutoff. Either comparison provides an image
quality metric14 that is a function of F/d. Table 1 summarizes the two limiting cases. Since the transition from one
region to the other is gradual, it is difficult to select an F/d
value that separates the two regions. It is nominally set at F/d
= 1. From a sampling viewpoint, the important parameter15
is F/dCC (also called F/ and Q). When FF = 1, F/d
= F/dCC . Figure 13 illustrates overall design space for
visible, mid-wave infrared (MWIR) and long-wave infrared
(LWIR) cameras. As listed in Table 2, the more popular designs are detector-limited.
The MTF at Nyquist frequency is often used as a measure of performance (Fig. 14). As MTF(uN ) increases, image
quality should increase. Unavoidably, as the MTF increases,
aliasing also increases and image quality suffers. Figure 15
provides 256256 pristine images. Figures 1618 illustrate
the imagery for three different F/d ratios with FF = 1.
Figure 8 illustrates F/d = 0.55. All images were recon-

structed with a flat-panel display. As evident in these figures,

image distortion is more obvious with periodic targets and
straight lines. Low F/d imagery exhibits the most aliasing.
In Fig. 16 the 3-bar targets appear as one, two, or distorted
3 bars. High F/d imagery (Fig. 18) has no aliasing or distortion. But the imagery is blurry because the MTF is so
low.
5 Viewing Distance
Whether the image is displayed on a CRT, flat-panel display, or printed, the viewing distance significantly affects
perceived quality. As illustrated in Fig. 19, as the target frequency increases, the perceived modulation decreases. As the
distance increases a small object will eventually subtend an
angle smaller than that which can be resolved by the HVS.
If several objects are close together, they are perceived as
one where the saturation, hue, and brightness are an average
of all the objects. In this context, the HVS is an additional
reconstruction filter.
5.1 Flat Panel Displays
For monochrome systems, the minimum viewing distance
occurs when the individual display pixels are barely perceptible. At closer distances, the pixels become visible and
this interferes with image interpretation. The flat panel pixel
consists of three color elements (red, green, and blue). At normal viewing distances, the eyes MTF attenuates the spatial
frequencies associated with the display pixels thus producing a smooth continuous image. Assuming a display pixel
pitch of 0.26 mm and a typical viewing distance of 0.355 m
(14 in.), each pixel subtends 0.73 mrad and each color element subtends 0.24 mrad. The HVS can resolve 0.29 mrad
(equivalent to 20/20 vision). Thus at 0.355 m, each pixel is

Table 1 Optics-limited versus detector-limited performance.

Table 2 F/d for F = 2.
F/d
<1

>1

System
performance
Detector-limited

Optics-limited

Optical Engineering

Spatial domain

Frequency domain

Airy disk much

smaller than
detector

Optical cutoff much

greater than the detector cutoff

Airy disk much

larger than detector

Optical cutoff much

less than the detector cutoff

Spectral
response

Detector

Typical
d (m)

AVE
(m)

F/d

Visible

CMOS

2.2

0.55

1.1

0.5

MWIR

InSb

18

4.0

8.0

0.44

LWIR

Microbolometer

18

12

24

0.75

052601-7

May 2011/Vol. 50(5)

Downloaded From: http://opticalengineering.spiedigitallibrary.org/ on 12/28/2016 Terms of Use: http://spiedigitallibrary.org/ss/termsofuse.aspx

Holst: Imaging system fundamentals

Table 3 Digital still camera prints (assuming 300 pixels per inch is
acceptable).

Fig. 15 Pristine imagery.

Application

Typical array size (HV)

Print size (inches)

Cell phone

640480

2.11.6

Point-and-shoot
6.3 Mpixel

28972173

9.67.2

Point-and-shoot
10 Mpixel

36502738

12.19.1

Professional SLR
40 Mpixel

73025477

24.318.2

resolved and each color element is not (desired result). At

0.154 m (6 in.), each color element subtends 0.56 mrad and
can be just perceived. When too close, the individual color
spots become visible and the impression of full color is lost.
For all the imagery in this paper, it is suggested that you
move several feet away to achieve the same visual angle
that would normally exist. At normal viewing distance,
the HVS attenuates the amplitudes of the spatial frequencies associated with the blockiness and the imagery appears
continuous.
Fig. 16 Imagery when F/d = 0.2. See footnote on this page.

Fig. 17 Imagery when F/d = 1.0. See footnote on this page.

5.2 Prints
Color printing suffices with four colors (red, green, blue, and
black). There are no grays. It is either ink or no ink. For
lighter colors, a small ink spot is surrounded by a white area.
To increase saturation, the ink spot is larger. For fixed pixel
size the white area is smaller. The HVS blends the ink spot
and white area to have a perceived saturation.
Printed imagery is considered excellent when there are
more than 300 dots/in (dpi). Photos look good when there are
about 300 pixels/in. To allow for color printing, the printer
should provide about 3 times more dots, or at least 900 dpi.
The values in Table 3 assume 300 pixels/in creates an excellent image. Larger images can be created. If pixel replication is used, at some point they will start to be blocky
(you can see the individual pixels). However, blocky images are rarely seen because software always interpolates
the data to create a smooth image. At small viewing distances this smoothed image may appear blurry. In contrast to
wet-film cameras, image enlargement does not provide more
resolution. Resolution is fixed by the pixel size and focal
length.
6 Summary
Performance for all cameras (point-and-shoot, TV studio
broadcast, and thermal infrared imaging) can be described
by an MTF with F/d being an important design metric.
The difference between detector-limited and optics-limited
The 256256 image (Fig. 15) was downsampled to 3232 detectors. The

Fig. 18 Imagery when F/d = 2.0. See footnote on this page.

Optical Engineering

imagery is enlarged so that your eye MTF and the printing do not significantly affect the image quality. This allows you to see the distortion created
by sampling and the system MTF degradation. Image quality depends upon
viewing distance. View the images at several feet to view to simulate normal
distance (same visual angle).
052601-8

May 2011/Vol. 50(5)

Downloaded From: http://opticalengineering.spiedigitallibrary.org/ on 12/28/2016 Terms of Use: http://spiedigitallibrary.org/ss/termsofuse.aspx

Intensity

Perceived Intensity

Holst: Imaging system fundamentals

Distance (x)

Distance (x)

Fig. 19 Perceived modulation (right) when viewing a sweep frequency target (left). Very high spatial frequencies are perceived as a uniform
gray.

is important. When detector-limited (F/d < 1), changes in

the optical system will have little effect on image quality.
Likewise, in the optics-limited region (F/d > 1), changing
the detector size will have minimal effect on image quality.
While Fig. 13 illustrates overall design space, most imaging
systems today are detector-limited (Table 2).
The MTF will always be used for lens design. It will be
used for scientific and military system optimization. It is
of lesser concern for commercial applications. The MTF at
Nyquist frequency is often used as a measure of performance
(Fig. 14). As MTF(uN ) increases, image quality should increase. Unavoidably, as the MTF increases, aliasing also increases and image quality suffers. This suggests there may be
an optimum MTF, or, equivalently an optimum F/d. This is
not so. Cameras are designed for a specific application with
F/d being a secondary consideration.
There is no aliasing when F/d 2. This limiting condition is known as critical sampling or Nyquist sampling of
the optical blur in the astronomy community.16 The latter
term is not good terminology since the blur diameter consists
of all frequencies up to the optical cutoff. When F/d 2
the imagery replicates the scene exactly. This may be an important consideration for medical imaging where a sampling
artifact could be construed as a medical abnormality or a
space probe where it is impossible to obtain ground truth.
Having F/d 2 may be overly restrictive. In-band MTFs
(frequencies less than the Nyquist frequency) are reduced in
amplitude (Fig. 18). If the SNR is sufficiently high, a boost
circuit can increase the MTF and sharpen the image. Note
that if F/d < 2, the boost circuit will also increase the aliased
signal [(MTFPOST (u)MTFPRE (uS u) in Fig. 12]. If too much
boost is used, the amplified aliased signal can degrade image
quality.
Sampling theory suggests that an ideal reconstruction
filter could be used. Although unrealizable, it can be approximated by a high order Butterworth filter. This works
well if there is no aliasing. With aliasing, the sharp cutoff filter creates undesirable ringing (Fig. 9). Flat panel displays are not ideal reconstruction filters and will not produce
ringing.
Aliasing was quantified by the spurious response. The
information in the higher frequencies has been aliased to
lower frequencies. But it is not known how to interpret this information. Mathematically, worst case analysis assumes that
the scene contains all spatial frequencies with equal amplitude. Real scenes have a limited spectrum and image quality
is a subjective measure. This means there is no method of
Optical Engineering

validating the theory with imagery. MTF theory and sampling

issues are just two slices through the multidimensional image
quality space. They provide guidance for camera design but
do not uniquely quantify image quality. The perceived image
quality depends upon F/d, dCC , display medium, and viewing distance. Changing any one or all of these parameters
affects the perceived image quality.
Nearly every visible digital camera aliases the scene
(Fig. 13). Is this really bad? Sampling artifacts are seen routinely. The amount of aliasing is scene specific and may or
may not be bothersome. It becomes apparent when viewing
test patterns (Figs. 10 and 16), picket fences, plowed fields,
railroad tracks, and Venetian blinds. In fact, while aliasing is
present (Fig. 11), the imagery may be considered excellent.
The color rendition on the flat panel display is excellent. The
scene content is exciting and the few jagged lines are just
ignored. On the other hand, a printed image is enlarged to
1620 in. (see Table 2) and looks wonderful.
Why is aliasing acceptable? The eye is primarily sensitive to intensity variations and less so to frequencies. Therefore, sampling artifacts in imagery are usually tolerable.
We have become accustomed to the aliasing. Simply watch
television. Pay attention to the folks wearing clothing with
narrow stripes or small patterns. Note the changing patterns
as they move. Stripes will change in shape, width, and color.
In contrast, the ear is a frequency detector and any distortion is immediately obvious. The sampling theorem must
be strictly followed for auditory-based processes but not for
imagery.
Different display media create different images. Image
quality is scene dependent. Go to any store selling TVs and
observe the difference between identical TVs. Note the
differences at different viewing distances. As the distance
decreases, the differences become more apparent. Simply
stated, all displays are designed for an assumed viewing distance. The printed image will never be the same as that seen
on the flat panel display.
Finally, wet-film based cameras did not alias the scene.
With technological advancements in digital cameras, image
quality has decreased. This is the trade-off between instant
imagery, ability to digitally transmit the imagery, and image quality. Current trends are to build smaller detectors
because it leads to lower cost and lighter cameras. Staring
array detectors create photoelectrons that are stored in charge
wells. But the charge well capacity decreases as the pixel area
decreases. The next generation camera will have a smaller
dynamic range, smaller SNR, and poorer image quality.

052601-9

May 2011/Vol. 50(5)

Downloaded From: http://opticalengineering.spiedigitallibrary.org/ on 12/28/2016 Terms of Use: http://spiedigitallibrary.org/ss/termsofuse.aspx

Holst: Imaging system fundamentals

References
1. R. G. Driggers, P. Cox, and T. Edwards, Introduction to Infrared and
Electro-Optical Systems, pp. 2125, Artech House, Norwood, MA
(1999).
2. R. H. Vollmerhausen and R. G. Driggers, Analysis of Sampled Imaging
Systems, SPIE Tutorial Text TT39, pp. 2831, Bellingham, WA (2000).
3. O. Hadar, A. Dogariu, and G. D. Boreman, Angular dependence of
sampling modulation transfer function, Appl. Opt. 3628, 72107216
(1997).
4. G. C. Holst, Electro-optical Imaging System Performance, 5th ed.,
pp. 362374, JCD Publishing Company, Winter Park, FL (2008).
5. C. E. Shannon, Communication in the Presence of Noise, Proceedings
of the IRE Vol. 37, pp. 1021, Institute of Electrical and Electronics
Engineers, New York (1949).
6. A. B. Jerri, The Shannon Sampling Theorem Its Various Extensions
and Applications: A Review, Proceedings of the IEEE, Vol. 85(11),
pp. 15651595, Institute of Electrical and Electronics Engineers, New
York (1977).
7. G. C. Holst, Sampling, Aliasing, and Data Fidelity, JCD Publishing,
Winter Park, FL (1998).
8. MAVIISS (MTF based Visual and Infrared System Simulator) is
an interactive software program available from JCD Publishing at
www.JCDPublishing.com.
9. G. C. Holst and T. L. Lomheim, CMOS/CCD Sensors and Camera
Systems, 2nd ed., pp. 287289, JCD Publishing Company, Winter Park,
FL (2011).
10. G. C. Holst, Are reconstruction filters necessary?, In Infrared Imaging
Systems: Design, Analysis, Modeling, and Testing XVII, G. C. Holst,
Ed., Proc. SPIE 6207, 62070K (2006).

Optical Engineering

11. D. Tamburrino, J. M. Speigle, D. J. Tweet, and J.-Jan Lee, 2PFC (two

pixels, full color): Image sensor demosaicing and characterization,
J. Electron. Imaging 19(2), 021103 (2010).
12. G. C. Holst, Electro-Optical Imaging System Performance, 5th ed.,
pp. 197199, JCD Publishing Company, Winter Park, FL (2008).
13. O. H. Schade, Sr., Image reproduction by a line raster process,
in Perception of Displayed Information, L. C. Biberman, Ed., pp. 233
278, Plenum Press, New York (1973).
14. G. C. Holst, Imaging system performance based upon F/d, Opt. Eng.
46, 103204 (2007).
15. R. D. Fiete, Image quality and FN/ for remote sensing systems,
Opt. Eng. 38, 12291240 (1999).
16. M. Clampin, Ultraviolet-optical charge-coupled devices for space instrumentation, Opt. Eng. 41, 11851191 (2002).

Gerald C. Holst is an independent consultant for imaging system analysis and testing. His varied background includes serving
as a technical liaison to NATO, research scientist for DoD, and a member of the Martin
Marietta (now Lockheed Martin) senior technical staff. He has planned, organized, and
directed the internationally acclaimed SPIE
conference Infrared Imaging Systems: Design, Analysis, Modeling and Testing since
1990. He is author of over 30 journal articles
and 6 books. He is a SPIE fellow and a member of OSA.

052601-10

May 2011/Vol. 50(5)

Downloaded From: http://opticalengineering.spiedigitallibrary.org/ on 12/28/2016 Terms of Use: http://spiedigitallibrary.org/ss/termsofuse.aspx