CS 534 Spring 2003: Ahmed Elgammal,

Rutgers University

CS 534: Computer Vision

Linear Filters

Spring 2004
Ahmed Elgammal
Dept of Computer Science
Rutgers University

CS 534 – Ahmed Elgammal 1

Outlines
• What are linear filters
• Smoothing and convolution
• Image noise
• Smoothing with a Gaussian
• Differentiation and convolution
• Fourier Transform, Discrete Fourier Transform
• Sampling and Aliasing

CS 534 – Ahmed Elgammal 2

1
CS 534 Spring 2003: Ahmed Elgammal,
Rutgers University

Digital image
• Assume we use a gray-level image
• Digital image: a two-dimensional light intensity function
f(x,y) where x and y denote spatial coordinates, the value
of f at any point is proportional to the brightness (gray
level) of the image at that point.
• A digital image:
– is discretized in the spatial domain
– Is discretized in the brightness domain. f(x,y)

CS 534 – Ahmed Elgammal 3

Linear Filters
General process:
• Form new image whose pixels are a weighted sum of original pixel
values, using the same set of weights at each point.
Properties
• Output is a linear function of the input
• Output is a shift-invariant function of the input (i.e. shift the input
image two pixels to the left, the output is shifted two pixels to the left)
• This operation is called convolution

weights

Original image Filtered image

CS 534 – Ahmed Elgammal 4

2
CS 534 Spring 2003: Ahmed Elgammal,
Rutgers University

Smoothing and convolution

• The convolution of two functions, f(x) and g(x) is defined
as ∞
h(x) = ∫ g(x' )f(x−x' )dx' =g(x)∗f(x)
−∞
• When the functions f and g are discrete and when g is
nonzero only over a finite range [-n,n] then this integral is
replaced by the following summation:

n
h(i) = ∑ g( j) f (i + j)
j =− n

• g is called kernel function f

g

CS 534 – Ahmed Elgammal 5

Example of 1-d convolution

1 2 3 4 5 6

f 8 7 8 22 23 12 10 11 9 5 6 4

g 1/13 1 3 5 3 1
n

∑ g( j) f (i + j)
1 3 5 3 1

1 3 5 3 1
h(i) =
j =− n

h 12 17 18
2
h(4) = ∑ g( j) f ( 4 + j)
j =−2

= g(−2) f ( 2) + g(−1) f (3) + g( 0) f (4) + g(1) f ( 5) + g(2) f ( 6)

CS 534 – Ahmed Elgammal 6

3
CS 534 Spring 2003: Ahmed Elgammal,
Rutgers University

Smoothing and convolution

• These integrals and summations extend simply to functions of two

variables:
n n
h(i, j) = f (i, j)∗g = ∑ ∑ g(k ,l) f (i + k, j + l)
k = − n l =− n

• Convolution computes the weighted sum of the gray levels in each nxn
neighborhood of the image, f, using the matrix of weights g.
• Convolution is a so-called linear operator because
– g*(af1 + bf 2) = a(g*f 1) + b(g*f 2)
• Convolution is shift invariant.

CS 534 – Ahmed Elgammal 7

2-D convolution

1 1
h(5,5) = ∑ ∑ g (k,l) f (5 + k,5 + l)
k = −1l = −1

= g( −1, −1) f ( 4,4) + g(−1,0) f (4,5) + g(−1,1) f (4, 4)

+ g(0,−1) f (5,4) + g(0,0) f (5,5) + g(0,1) f (5,6)
+ g(1,−1) f (6,4) + g(1,0) f (6,5) + g(1,1) f (6,6)

CS 534 – Ahmed Elgammal 8

4
CS 534 Spring 2003: Ahmed Elgammal,
Rutgers University

Smoothing and convolution

CS 534 – Ahmed Elgammal 9

Example: Smoothing by Averaging

Flat kernel: all weights equal 1/N

CS 534 – Ahmed Elgammal 10

5
CS 534 Spring 2003: Ahmed Elgammal,
Rutgers University

Smoothing with a Gaussian

• Smoothing with an average
actually doesn’t compare at all
well with a defocussed lens
– Most obvious difference is that
a single point of light viewed
in a defocussed lens looks like
a fuzzy blob; but the averaging
process would give a little
square.

• A Gaussian gives a good model of

a fuzzy blob

CS 534 – Ahmed Elgammal 11

An Isotropic Gaussian
• The picture shows a smoothing
kernel proportional to

  x2 + y2  
exp  −  
  2σ 2  

(which is a reasonable model of a

circularly symmetric fuzzy
blob)

CS 534 – Ahmed Elgammal 12

6
CS 534 Spring 2003: Ahmed Elgammal,
Rutgers University

Smoothing with a Gaussian

CS 534 – Ahmed Elgammal 13

Gaussian smoothing
• Advantages of Gaussian filtering
– rotationally symmetric (for large filters)
– filter weights decrease monotonically from central peak, giving
most weight to central pixels
– Simple and intuitive relationship between size of σ and the
smoothing.
– The Gaussian is separable:

(x2 +y2) x2 y2
− 2 − 2 − 2
e 2σ =e 2σ ∗e 2σ

CS 534 – Ahmed Elgammal 14

7
CS 534 Spring 2003: Ahmed Elgammal,
Rutgers University

Advantage of seperability
• First convolve the image with a one dimensional horizontal
filter
• Then convolve the result of the first convolution with a one
dimensional vertical filter
• For a kxk Gaussian filter, 2D convolution requires k2
operations per pixel
• But using the separable filters, we reduce this to 2k
operations per pixel.

CS 534 – Ahmed Elgammal 15

Separability

2 3 3 11
1 2 1 3 5 5 18

4 4 6 18

1 11

2 18 65
1 18

1 x 1 2 1 1 2 1 2 3 3 =2 + 6 + 3 = 11

2 = 2 4 2 3 5 5 = 6 + 20 + 10 = 36
1 1 2 1 4 4 6 = 4 + 8 + 6 = 18

65
CS 534 – Ahmed Elgammal 16

8
CS 534 Spring 2003: Ahmed Elgammal,
Rutgers University

Advantages of Gaussians
• Convolution of a Gaussian with itself is another Gaussian
– so we can first smooth an image with a small Gaussian
– then, we convolve that smoothed image with another small
Gaussian and the result is equivalent to smoother the original
image with a larger Gaussian.
– If we smooth an image with a Gaussian having sd σ twice, then we
get the same result as smoothing the image with a Gaussian
having standard deviation (2σ)1/2

CS 534 – Ahmed Elgammal 17

Noise
• Simplest noise model
– independent stationary additive Gaussian noise
– the noise value at each pixel is given by an independent draw from
the same normal probability distribution

fobserved ( x, y) = f ( x, y) + N (0, σ 2 )
• Issues
– this model allows noise values that could be greater than maximum
camera output or less than zero
– for small standard deviations, this isn’t too much of a problem - it’s
a fairly good model
– independence may not be justified (e.g. damage to lens)
– may not be stationary (e.g. thermal gradients in the ccd)

CS 534 – Ahmed Elgammal 18

9
CS 534 Spring 2003: Ahmed Elgammal,
Rutgers University

sigma=1

CS 534 – Ahmed Elgammal 19

sigma=16

CS 534 – Ahmed Elgammal 20

10
CS 534 Spring 2003: Ahmed Elgammal,
Rutgers University

The response of a linear filter to noise

• Do only stationary independent • Variance:
additive Gaussian noise with – recall
zero mean (non-zero mean is • variance of a sum of random
easily dealt with) variables is sum of their
variances
• Mean:
• variance of constant times
– output is a weighted sum of random variable is constant^2
inputs times variance
– so we want mean of a weighted – then if σ2 is noise variance and
sum of zero mean normal kernel is K, variance of
random variables response is
– must be zero
fobserved ( x , y ) = f ( x, y ) + N (0, σ 2 )
g ∗ f observed = g ∗ f + g ∗ N (0, σ 2 ) σ 2 ∑ K u,v
2

u,v

N ′( 0, σ ′2 )
CS 534 – Ahmed Elgammal 21

CS 534 – Ahmed Elgammal 22

11
CS 534 Spring 2003: Ahmed Elgammal,
Rutgers University

Differentiation and convolution

• Recall

∂f  f (x + ε, y) f (x, y )
= lim − 
∂x ε→ 0  ε ε 
• Now this is linear and shift invariant, so must be the result of a
convolution.
• We could approximate this as 0 0 0 
 
∂f f (xn+1 , y )− f (xn , y) G =  1 0 − 1
≈ 0 0 0 
∂x ∆x  
(which is obviously a convolution; it’s not a very good way to do things, as
we shall see)

CS 534 – Ahmed Elgammal 23

Finite differences

CS 534 – Ahmed Elgammal 24

12
CS 534 Spring 2003: Ahmed Elgammal,
Rutgers University

Finite differences responding to noise

Increasing noise ->

(this is zero mean additive gaussian noise)

CS 534 – Ahmed Elgammal 25

Finite differences and noise

• Finite difference filters respond strongly to noise
– obvious reason: image noise results in pixels that look very
different from their neighbours
• Generally, the larger the noise the stronger the response
• What is to be done?
– intuitively, most pixels in images look quite a lot like their
neighbors
– this is true even at an edge; along the edge they’re similar, across
the edge they’re not
– suggests that smoothing the image should help, by forcing pixels
different to their neighbors (=noise pixels?) to look more like
neighbors

CS 534 – Ahmed Elgammal 26

13
CS 534 Spring 2003: Ahmed Elgammal,
Rutgers University

Fourier Transform
• Fourier Transform as a change of basis
∞
− j 2πux
F (u ) = ∫ f ( x) e
−∞
dx
e − j 2πux = cos(2πux ) − j sin( 2πux)
∞
j 2πux
f ( x) = ∫ F (u ) e
−∞
du

FT

Temporal or spatial domain Frequency domain

• F(u) is composed of an infinite sum of sine and cosine terms

• Inverse transform: reconstruction of the signal as infinite sum of sine
and cosine terms
CS 534 – Ahmed Elgammal 27

• Since of a real function is generally complex, we use

magnitude and phase
F (u ) = (R 2 (u ) + I 2 (u) )
1/ 2

F ( u) = R(u) + jI (u)
φ (u ) = tan −1 I (u ) / R (u )

f(x) |F(u)|2
Power Spectrum
Lower frequencies ⇒ narrower power spectrum

x u

Higher frequencies ⇒ wider power spectrum

CS 534 – Ahmed Elgammal 28

14
CS 534 Spring 2003: Ahmed Elgammal,
Rutgers University

Important Properties:
• Fourier transform is linear

F ( g ( x ) + h( x )) = F ( g( x )) + F (h( x ))
F (kg ( x )) = k F ( g ( x ))
• FT and Convolution
f ( x) ∗ g ( x ) ⇔ F ( u)G ( u)
f ( x ) g ( x ) ⇔ F ( u ) ∗ G ( u)

• FT of a Gaussian is a Gaussian

CS 534 – Ahmed Elgammal 29

Discrete Fourier Transform

• If we discretize f(x) using uniformly spaced samples f(0), f(1),…,f(N-1)
, we can obtain FT of the sampled function
N −1
F (u ) = ∑ f ( x ) e
− j 2π ux / N

x =0
N −1

∑ F (u ) e
− j 2π ux / N
f ( x) =
u =0

• Important Property:
Periodicity F(u)=F(u+N)

One period
CS 534 – Ahmed Elgammal 30

15
CS 534 Spring 2003: Ahmed Elgammal,
Rutgers University

2-dimension
∞
In two dimension − j 2π ( ux + vy)
F (u , v ) = ∫ ∫ f ( x, y ) e dxdy
−∞
∞
j 2 π ( ux+ vy)
f ( x, y ) = ∫ ∫ F (u, v ) e dudv
−∞

− j 2π ( ux + vy)
e = cos(2π (ux + vy )) + j sin( 2π (ux + vy ))
• These terms are sinusoids on the x,y plane whose
orientation and frequency are defined by u,v

CS 534 – Ahmed Elgammal 31

• The FT tries to represent all images as a summation of

cosine-like images

Images of pure cosines

• Center of the image:the origin of the frequency
coordinate system
• u-axis: (left to right) the horizontal component of
frequency
• v-axis: (bottom-top) the vertical component of
frequency
• Center dot (0,0) frequency : image average

FT

• high frequencies in the vertical direction will cause bright dots

away from the center in the vertical direction.
• high frequencies in the horizontal direction will cause bright dots
away from the center in the horizontal direction.

CS 534 – Ahmed Elgammal 32

16
CS 534 Spring 2003: Ahmed Elgammal,
Rutgers University

• Since images are real numbers (not complex) FT image is

symmetric around the origin.

FT: symmetry FT is shift invariant

CS 534 – Ahmed Elgammal 33

• In general, rotation of the image results in equivalent

rotation of its FT

Why it is not the case ?

• Edge effect !
• FT always treats an image as if it were part of a periodically replicated array of
identical images extending horizontally and vertically to infinity
• Solution: “windowing” the image

CS 534 – Ahmed Elgammal 34

17
CS 534 Spring 2003: Ahmed Elgammal,
Rutgers University

• notice a bright band going to high

frequencies perpendicular to the
strong edges in the image
• Anytime an image has a strong-
contrast, sharp edge the gray
values must change very rapidly.
It takes lots of high frequency
power to follow such an edge so
there is usually such a line in its
magnitude spectrum.

CS 534 – Ahmed Elgammal 35

• What happens if we swap the magnitude spectra ?

• Phase spectrum holds the spatial information (where things are),
• Phase spectrum is more important for perception than magnitude spectrum.

CS 534 – Ahmed Elgammal 36

18
CS 534 Spring 2003: Ahmed Elgammal,
Rutgers University

Sampling and Aliasing

• Differences between continuous and discrete images
• Images are sampled version of a continuous brightness
function.
successful sampling

unsuccessful sampling
CS 534 – Ahmed Elgammal 37

Sampling and Aliasing successful sampling

• Sampling involves loss of

information
• Aliasing: high spatial
frequency components
appear as low spatial
frequency components in
the sampled signal unsuccessful sampling

Java applet from: http://www.dsptutor.freeuk.com/aliasing/AD102.html

CS 534 – Ahmed Elgammal 38

19
CS 534 Spring 2003: Ahmed Elgammal,
Rutgers University

Aliasing
• Nyquist theorem: The
sampling frequency must
be at least twice the highest
frequency present for a
signal to be reconstructed
from a sampled version.
(Nyquist frequency)

CS 534 – Ahmed Elgammal 39

Sampling, aliasing, and DFT

• DFT consists of a sum of copies of the FT of the original signal shifted
by the sampling frequency:
– If shifted copies do not intersect: reconstruction is possible.
– If shifted copies do intersect: incorrect reconstruction, high frequencies
are lost (Aliasing)

CS 534 – Ahmed Elgammal 40

20
CS 534 Spring 2003: Ahmed Elgammal,
Rutgers University

Sampling, aliasing, and DFT

Notices: Interferences between FT copies

CS 534 – Ahmed Elgammal 41

Smoothing and Resampling

• In many vision application we want to reduce image size by half by
resampling
• We need to reduce the aliasing effect
• Spatial frequencies above the new sampling frequency should be removed.
• Multiply the DFT by a Gaussian will do
• Equivalently: convolve the original image with a Gaussian.
• Choice of σ is critical:
– Large σ : Less aliasing, more information lost
– Small σ : More aliasing, less information lost

CS 534 – Ahmed Elgammal 42

21
CS 534 Spring 2003: Ahmed Elgammal,
Rutgers University

Sources
• Forsyth and Ponce, Computer Vision a Modern approach:
chapter 7.
• R. Gonzalez and P. Wintz, Digital Image Processing,
second edition.
• Slides by:
– D. Forsyth
• Fourier transform images from Prof. John M. Brayer @
UNM
http://www.cs.unm.edu/~brayer/vision/fourier.html
• Aliasing java applets available at
http://klebanov.homeip.net/~pavel/fb/java/la_applets/Aliasing/
http://www.dsptutor.freeuk.com/aliasing/AliasingDemo.html

CS 534 – Ahmed Elgammal 43

22