Computer Vision

Lecture # 4
Image Filtering & Its Applications
 Today: Image Filters Review

Smooth/Sharpen Images... Find edges... Find waldo…

 Image neighborhoods
• Q: What happens if we reshuffle all pixels within the
images?

• A: Its histogram won’t change.

Point-wise processing unaffected.

• Need to measure properties relative to small

neighborhoods of pixels
Images as functions

Source: S. Seitz
Images as functions
• We can think of an image as a function, f, from
R2 to R:
• f( x, y ) gives the intensity at position ( x, y )
• Realistically, we expect the image only to be defined over a
rectangle, with a finite range:
– f: [a,b] x [c,d]  [0, 1.0]

• A color image is just three functions pasted

together. We can write this as a “vector-valued”
function: r ( x, y )  
f ( x , y )   g ( x, y ) 
 
 b( x, y ) 
Source: S. Seitz
 Digital images
• In computer vision we operate on digital (discrete) images:
• Sample the 2D space on a regular grid
• Quantize each sample (round to nearest integer)
• Image thus represented as a matrix of integer values.

2D

1D

Adapted from S. Seitz

Motivation: noise reduction

• We can measure noise in multiple images of the same

static scene.
• How could we reduce the noise, i.e., give an estimate of
the true intensities?
Common types of noise
– Salt and pepper noise:
random occurrences of
black and white pixels

– Impulse noise: random

occurrences of white pixels

– Gaussian noise: variations

in intensity drawn from a
Gaussian normal
distribution

Source: S. Seitz
Gaussian noise

>> noise = randn(size(im)).*sigma;

>> output = im + noise;

Fig: M. Hebert
Effect of
sigma on
Gaussian
noise:

Image shows
the noise
values
themselves.
Effect of
sigma on
Gaussian
noise:

Image shows
the noise
values
themselves.
Effect of
sigma on
Gaussian
noise:

Image shows
the noise
values
themselves.
 Effect of
sigma on
Gaussian
noise:

sigma=1 This shows

the noise
values added
to the raw
intensities of
an image.
 Effect of
sigma on
Gaussian
noise

sigma=16 This shows

the noise
values added
to the raw
intensities of
an image.
Motivation: noise reduction

• How could we reduce the noise, i.e., give an estimate of

the true intensities?
• What if there’s only one image?
First attempt at a solution
• Let’s replace each pixel with an average of all
the values in its neighborhood
• Assumptions:
– Expect pixels to be like their neighbors
– Expect noise processes to be independent from
pixel to pixel
First attempt at a solution
• Let’s replace each pixel with an average of all
the values in its neighborhood
• Moving average in 1D:

Source: S. Marschner
Weighted Moving Average
• Can add weights to our moving average
• Weights [1, 1, 1, 1, 1] / 5

Source: S. Marschner
Weighted Moving Average
• Non-uniform weights [1, 4, 6, 4, 1] / 16

Source: S. Marschner
Moving Average In 2D

0 0 0 0 0 0 0 0 0 0

0 0 0 0 0 0 0 0 0 0 0

0 0 0 90 90 90 90 90 0 0

0 0 0 90 90 90 90 90 0 0

0 0 0 90 90 90 90 90 0 0

0 0 0 90 0 90 90 90 0 0

0 0 0 90 90 90 90 90 0 0

0 0 0 0 0 0 0 0 0 0

0 0 90 0 0 0 0 0 0 0

0 0 0 0 0 0 0 0 0 0

Source: S. Seitz
Moving Average In 2D

0 0 0 0 0 0 0 0 0 0

0 0 0 0 0 0 0 0 0 0 0 10

0 0 0 90 90 90 90 90 0 0

0 0 0 90 90 90 90 90 0 0

0 0 0 90 90 90 90 90 0 0

0 0 0 90 0 90 90 90 0 0

0 0 0 90 90 90 90 90 0 0

0 0 0 0 0 0 0 0 0 0

0 0 90 0 0 0 0 0 0 0

0 0 0 0 0 0 0 0 0 0

Source: S. Seitz
Moving Average In 2D

0 0 0 0 0 0 0 0 0 0

0 0 0 0 0 0 0 0 0 0 0 10 20

0 0 0 90 90 90 90 90 0 0

0 0 0 90 90 90 90 90 0 0

0 0 0 90 90 90 90 90 0 0

0 0 0 90 0 90 90 90 0 0

0 0 0 90 90 90 90 90 0 0

0 0 0 0 0 0 0 0 0 0

0 0 90 0 0 0 0 0 0 0

0 0 0 0 0 0 0 0 0 0

Source: S. Seitz
Moving Average In 2D

0 0 0 0 0 0 0 0 0 0

0 0 0 0 0 0 0 0 0 0 0 10 20 30

0 0 0 90 90 90 90 90 0 0

0 0 0 90 90 90 90 90 0 0

0 0 0 90 90 90 90 90 0 0

0 0 0 90 0 90 90 90 0 0

0 0 0 90 90 90 90 90 0 0

0 0 0 0 0 0 0 0 0 0

0 0 90 0 0 0 0 0 0 0

0 0 0 0 0 0 0 0 0 0

Source: S. Seitz
Moving Average In 2D

0 0 0 0 0 0 0 0 0 0

0 0 0 0 0 0 0 0 0 0 0 10 20 30 30

0 0 0 90 90 90 90 90 0 0

0 0 0 90 90 90 90 90 0 0

0 0 0 90 90 90 90 90 0 0

0 0 0 90 0 90 90 90 0 0

0 0 0 90 90 90 90 90 0 0

0 0 0 0 0 0 0 0 0 0

0 0 90 0 0 0 0 0 0 0

0 0 0 0 0 0 0 0 0 0

Source: S. Seitz
Moving Average In 2D

0 0 0 0 0 0 0 0 0 0

0 0 0 0 0 0 0 0 0 0 0 10 20 30 30 30 20 10

0 0 0 90 90 90 90 90 0 0 0 20 40 60 60 60 40 20

0 0 0 90 90 90 90 90 0 0 0 30 60 90 90 90 60 30

0 0 0 90 90 90 90 90 0 0 0 30 50 80 80 90 60 30

0 0 0 90 0 90 90 90 0 0 0 30 50 80 80 90 60 30

0 0 0 90 90 90 90 90 0 0 0 20 30 50 50 60 40 20

0 0 0 0 0 0 0 0 0 0 10 20 30 30 30 30 20 10

0 0 90 0 0 0 0 0 0 0 10 10 10 0 0 0 0 0

0 0 0 0 0 0 0 0 0 0

Source: S. Seitz
 Correlation filtering
Say the averaging window size is 2k+1 x 2k+1:

Attribute uniform weight Loop over all pixels in neighborhood around

to each pixel image pixel F[i,j]

Now generalize to allow different weights depending on

neighboring pixel’s relative position:

Non-uniform weights
 Correlation filtering

This is called cross-correlation, denoted

Filtering an image: replace each pixel with a linear combination of

its neighbors.

The filter “kernel” or “mask” H[u,v] is the prescription for the

weights in the linear combination.
 Averaging filter
• What values belong in the kernel H for the moving average
example?

0 0 0 0 0 0 0 0 0 0

0
0

0
0

0
0

90
0

90
0

90
0

90
0

90
0

0
0

0
1 1 1 0 10 20 30 30

0
0

0
0

0
90

90
90

90
90

90
90

90
90

90
0

0
0

0
1 ?1 1
0 0 0 90 0 90 90 90 0 0
1 1 1
0 0 0 90 90 90 90 90 0 0

0 0 0 0 0 0 0 0 0 0

0 0 90 0 0 0 0 0 0 0
“box filter”
0 0 0 0 0 0 0 0 0 0
Smoothing by averaging
depicts box filter:
white = high value, black = low value

original filtered
 Gaussian filter
• What if we want nearest neighboring pixels to have the
most influence on the output?

0 0 0 0 0 0 0 0 0 0
This kernel is an
0 0 0 0 0 0 0 0 0 0
approximation of a
0 0 0 90 90 90 90 90 0 0
Gaussian function:
0 0 0 90 90 90 90 90 0 0 1 2 1
0 0 0 90 90 90 90 90 0 0
2 4 2
0 0 0 90 0 90 90 90 0 0
1 2 1
0 0 0 90 90 90 90 90 0 0
0 0 0 0 0 0 0 0 0 0
0 0 90 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0

Source: S. Seitz
Smoothing with a Gaussian
Gaussian filters
• What parameters matter here?
• Size of kernel or mask
– Note, Gaussian function has infinite support, but discrete filters
use finite kernels

σ = 5 with 10 σ = 5 with 30
x 10 kernel x 30 kernel
Gaussian filters
• What parameters matter here?
• Variance of Gaussian: determines extent of
smoothing

σ = 2 with 30 σ = 5 with 30
x 30 kernel x 30 kernel
Filtering an impulse signal
What is the result of filtering the impulse signal (image) F
with the arbitrary kernel H?

0 0 0 0 0 0 0
0 0 0 0 0 0 0
a b c

?
0 0 0 0 0 0 0
d e f
0 0 0 1 0 0 0
g h i
0 0 0 0 0 0 0
0 0 0 0 0 0 0
0 0 0 0 0 0 0
Filtering an impulse signal
What is the result of filtering the impulse signal (image) F
with the arbitrary kernel H?

0 0 0 0 0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0 0 0 0 0
a b c
0 0 0 0 0 0 0 0 0 0a 0b 0c 0 0
d e f
0 0 0 1 0 0 0 0 0 0d 1e 0f 0 0
g h i
0 0 0 0 0 0 0 0 0 0g 0h 0i 0 0
0 0 0 0 0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0 0 0 0 0
Practice with linear filters

?
0 0 0
0 1 0
0 0 0

Original

Source: D. Lowe
Practice with linear filters

0 0 0
0 1 0
0 0 0

Original Filtered
(no change)

Source: D. Lowe
Practice with linear filters

?
0 0 0
0 0 1
0 0 0

Original

Source: D. Lowe
Practice with linear filters

0 0 0
0 0 1
0 0 0

Original Shifted left

By 1 pixel

Source: D. Lowe
Practice with linear filters

-
0 0 0 1 1 1
0
0
2
0
0
0
1
1
1
1
1
1
?
(Note that filter sums to 1)
Original

Source: D. Lowe
Practice with linear filters

-
0 0 0 1 1 1
0 2 0 1 1 1
0 0 0 1 1 1

Original Sharpening filter

- Accentuates differences with local
average

Source: D. Lowe
Denoising

Gaussian
Filter

Additive Gaussian Noise

Slide: Hoiem
Reducing Gaussian noise

Smoothing with larger standard deviations suppresses noise, but also blurs the
image
Source: S. Lazebnik
Reducing salt-and-pepper noise by
Gaussian smoothing

3x3 5x5 7x7

Alternative idea: Median filtering
• A median filter operates over a window by
selecting the median intensity in the window

• Is median filtering linear?

Source: K. Grauman
Median filter
• What advantage does median filtering have
over Gaussian filtering?
– Robustness to outliers

Source: K. Grauman
Median filter
Salt-and-pepper noise Median filtered

• MATLAB: medfilt2(image, [h w])

Source: M. Hebert
 Median vs. Gaussian filtering
3x3 5x5 7x7

Gaussian

Median
Edge detection
• Goal: map image from 2d array of pixels to a set of curves
or line segments or contours.
• Why?

Figure from J. Shotton et al., PAMI 2007

• Main idea: look for strong gradients, post-process

 What can cause an edge?

Depth discontinuity:
Reflectance change: object boundary
appearance
information, texture

Cast shadows

Change in surface
orientation: shape
Contrast and invariance
Recall : Images as functions

• Edges look like steep cliffs

Source: S. Seitz
Derivatives and edges
An edge is a place of rapid change in the image
intensity function.
intensity function
image (along horizontal scanline)

Source: L. Lazebnik
 Differentiation and convolution
For 2D function, f(x,y), the partial derivative is:

f ( x, y) f ( x   , y)  f ( x, y)
 lim
x  0 
For discrete data, we can approximate using finite differences:
f ( x, y) f ( x  1, y)  f ( x, y)

x 1
To implement above as convolution, what would be the associated
filter?
Partial derivatives of an image

f ( x, y ) f ( x, y )
x y

-1 ? 1
or
-1 1 1 -1
Which shows changes with respect to x?
(showing flipped filters)
Assorted finite difference filters

>> My = fspecial(‘sobel’);
>> outim = imfilter(double(im), My);
>> imagesc(outim);
>> colormap gray;
Image gradient
The gradient of an image:

The gradient points in the direction of most rapid change in intensity

The gradient direction (orientation of edge normal) is given by:

The edge strength is given by the gradient magnitude

Slide credit S. Seitz

Thresholding
• Choose a threshold value t
• Set any pixels less than t to zero (off)
• Set any pixels greater than or equal to t to one (on)
Original image
Gradient magnitude image
Thresholding gradient with a lower threshold
Thresholding gradient with a higher threshold
Canny edge detector
• Filter image with derivative of Gaussian
• Find magnitude and orientation of gradient
• Non-maximum suppression:
– Thin multi-pixel wide “ridges” down to single pixel width
• Linking and thresholding (hysteresis):
– Define two thresholds: low and high
– Use the high threshold to start edge curves and the low
threshold to continue them

• MATLAB: edge(image, ‘canny’);

• >>help edge

Source: D. Lowe, L. Fei-Fei

The Canny edge detector

original image (Lena)

The Canny edge detector

norm of the gradient

The Canny edge detector

thresholding
The Canny edge detector

Problem:
pixels along
this edge
didn’t survive
the
thresholding

thinning
(non-maximum suppression)
Hysteresis thresholding
• Check that maximum value of gradient value is
sufficiently large
– drop-outs? use hysteresis
• use a high threshold to start edge curves and a low
threshold to continue them.

Source: S. Seitz
Hysteresis thresholding

original image

high threshold low threshold hysteresis threshold

(strong edges) (weak edges)
Source: L. Fei-Fei
Object boundaries vs. edges

Background Texture Shadows

Edge detection is just the beginning…
image human segmentation gradient magnitude

Berkeley segmentation database:

http://www.eecs.berkeley.edu/Research/Projects/CS/vision/grouping/segbench/

Much more on segmentation later in term…

Source: L. Lazebnik
Template matching
• Filters as templates:
Note that filters look like the effects they are intended to
find --- “matched filters”

• Use normalized cross-correlation score to find a given

pattern (template) in the image.
– Szeliski Eq. 8.11
• Normalization needed to control for relative brightnesses.
Template matching

Template (mask)

Scene

A toy example
Template matching

Template

Detected template
Template matching

Detected template Correlation map

Where’s Waldo?

Template

Scene
Where’s Waldo?

Template

Scene
Where’s Waldo?

Detected template Correlation map

Template matching

Template

Scene

What if the template is not identical to some

subimage in the scene?
Template matching

Template

Detected template

Match can be meaningful, if scale, orientation, and

general appearance is right.
Template matching
• Goal: find in image

• Main challenge: What is a

good similarity or distance
measure between two
patches?
– Correlation
– Zero-mean correlation
– Sum Square Difference
– Normalized Cross
Correlation

Slide: Hoiem
Matching with filters
• Goal: find in image
• Method 0: filter the image with eye patch
h[ m, n]   g[ k , l ] f [ m  k , n  l ]
k ,l
f = image
g = filter

What went wrong?

Input Filtered Image

Slide: Hoiem
Matching with filters
• Goal: find in image
• Method 1: filter the image with zero-mean eye
h[ m, n]   ( f [ k , l ]  f ) ( g[ m  k , n  l ] )
k ,l mean of f

True detections

False
detections

Input Filtered Image (scaled) Thresholded Image

Slide: Hoiem
 Matching with filters
• Goal: find in image
• Method 3: Normalized cross-correlation
mean template mean image patch

 ( g[k , l ]  g )( f [m  k , n  l ]  f m ,n )
h[ m, n]  k ,l
0.5
 2
  ( g[ k , l ]  g )  ( f [ m  k , n  l ]  f m,n ) 
2

 k ,l k ,l 

Matlab: normxcorr2(template, im)

Slide: Hoiem
Matching with filters
• Goal: find in image
• Method 3: Normalized cross-correlation

True detections

Input Normalized X-Correlation Thresholded Image

Slide: Hoiem
Matching with filters
• Goal: find in image
• Method 3: Normalized cross-correlation

True detections

Input Normalized X-Correlation Thresholded Image

Slide: Hoiem
Q: What if we want to find larger or smaller eyes?

A: Image Pyramid
Review of Sampling

Gaussian
Filter Sample
Low-Pass Low-Res
Image
Filtered Image Image

Slide: Hoiem
Gaussian pyramid

Source: Forsyth
Template Matching with Image Pyramids

Input: Image, Template

1. Match template at current scale

2. Downsample image

3. Repeat 1-2 until image is very small

4. Take responses above some threshold, perhaps

with non-maxima suppression

Slide: Hoiem
Coarse-to-fine Image Registration
1. Compute Gaussian pyramid
2. Align with coarse pyramid
3. Successively align with finer
pyramids
– Search smaller range

Why is this faster?

Are we guaranteed to get the same

result?

Slide: Hoiem
Laplacian filter

unit impulse
Gaussian Laplacian of Gaussian

Source: Lazebnik
2D edge detection filters

Laplacian of Gaussian

Gaussian derivative of Gaussian

is the Laplacian operator:

Laplacian pyramid

Source: Forsyth
Computing Gaussian/Laplacian Pyramid

Can we reconstruct the original

from the laplacian pyramid?

http://sepwww.stanford.edu/~morgan/texturematch/paper_html/node3.html
 The simplest wavelet transform:
the Haar transform
1 1 0.5 0.5
U= U-1=
1 -1 0.5 -0.5

The simplest set of functions:

96
Haar transform
The simplest set of functions:

1 1 0.5 0.5
U= U-1=
1 -1 0.5 -0.5

To code a signal, repeat at several locations:

1 1 1 1
1 -1 1 -1
1 1 1 1
1 -1 1 -1
U= U-1= ½ 1 1
1 1
1 -1 1 -1
1 1 1 1
1 -1 1 -1
97
Recursive matrix construction of Haar transform
A3 A2 A1

A3 A2 A1 (A3 A2 A1)-1

98
 2D Haar transform
1 1
Basic elements: 1 1 1 -1
1 -1

99
 2D Haar transform
1 1
Basic elements: 1 1 1 -1
1 -1

1 1 1 Low pass
1 1 = 2
1 1 1

100
 2D Haar transform
1 1
Basic elements: 1 1 1 -1
1 -1

1 1 1 Low pass
1 1 = 2
1 1 1

1
1 -1 =
1

1
1 1 =
-1

1
1 -1 =
-1
101
 2D Haar transform
1 1
Basic elements: 1 1 1 -1
1 -1

1 1 1 Low pass
1 1 = 2
1 1 1

1 1 -1
1 -1 = 2
1 1 -1

1 1 1 2
1 1 =
-1 -1 -1

1 1 -1
1 -1 = 2
-1 -1 1
102
 2D Haar transform
1 1
Basic elements: 1 1 1 -1
1 -1

1 1 1 Low pass
1 1 = 2
1 1 1

High pass
1 1 -1
1 -1 = 2 vertical
1 1 -1

High pass
1 1 1 2 horizontal
1 1 =
-1 -1 -1

1 1 -1
1 -1 = 2 High pass
-1 -1 1 diagonal
103 103
2D Haar transform
Sketch of the Fourier transform

1 1 2
1 1

1 -1
2
1 -1

1 1 2
-1 -1

1 -1
2
-1 1
104
2D Haar transform
Sketch of the Fourier transform
Horizontal low pass,
Vertical low-pass
1 1 2
1 1
Horizontal high
1 -1 pass, vertical low-
2 pass
1 -1

Horizontal low
1 1 2 pass, vertical
high-pass
-1 -1

1 -1 Horizontal high
2 pass, vertical high
-1 1 pass

105 105
 Simoncelli and Adelson, in “Subband coding”, Kluwer, 1990.

Pyramid cascade

106 106
Wavelet/QMF representation

1 -1

1 -1

1 1 1 -1

-1 -1 -1 1

Same number of pixels!

107
Image representation

• Pixels: great for spatial resolution, poor access to

frequency

• Fourier transform: great for frequency, not for spatial

info

• Pyramids/filter banks: balance between spatial and

frequency information

Slide: Hoiem
Major uses of image pyramids

• Compression

• Object detection
– Scale search
– Features

• Detecting stable interest points

• Registration
– Course-to-fine

Slide: Hoiem
 Acknowledgements

 Computer Vision A modern Approach by Frosyth

 CSCI 1430: Introduction to Computer Vision by James Tompkin
Material in these slides has been taken from, the following resources

 Statistical Pattern Recognition: A Review – A.K Jain et al., PAMI (22) 2000
 Pattern Recognition and Analysis Course – A.K. Jain, MSU
 Pattern Classification” by Duda et al., John Wiley & Sons.
 Digital Image Processing”, Rafael C. Gonzalez & Richard E. Woods, Addison-Wesley, 2002
 Machine Vision: Automated Visual Inspection and Robot Vision”, David Vernon, Prentice Hall,
1991
 www.eu.aibo.com/
 Advances in Human Computer Interaction, Shane Pinder, InTech, Austria, October 2008
 Computer Vision A modern Approach by Frosyth

110