COMP/ECE/ENGI: COMPUTER

VISION

Lecture 4: Image coordinates, resizing

COMPUTER VISION www.mun.ca

 Images

Moire Patterns
https://www.sandlotscience.com/moire-pattern/
Coming up…

Image sensors

Image sampling

Image scaling

Filtering

Aliasing
Mystery:
The case
of the
Soiled Van Gogh
Our hero

Was up late last night...

The Van Gogh was just too big

he needed to find some way

to fit it on the slide

Or else!
Eureka! Throw away every other row and column
Eureka! Throw away every other row and column

But wait!
Something is amiss...
 Soiled! Who
did it?
 Another example

Throw away even rows

and columns
 Another example

Throw away even rows

and columns
 Another example

Throw away even rows

and columns
 Another example

Throw away even rows

and columns

1/2
Another example

1/2
Another example

1/2
What went wrong?
We’re implicitly treating the image as a function f(x,y) that maps position to light

f(x,y)
y
What went wrong?

But cameras don’t measure that function

Sub-sampling Camera

What if we throw out every other row/column?

Sub-sampling Camera

What if we throw out every other row/column?

Sub-sampling Camera

Here’s a particularly bad example

Camera sensors actually have this problem

read-out
electronics

CMOS sensor

photodiode

CMOS sensor
Photosensitive area (fill factor) often < 50%
 Hmmm…
Tell me more about camera sensors
https://sites.google.com/a/globalsystemsscience.org/digital-earth-watch/tools/digital-cameras-overview/what-happens-to-the-near-infared-entering-the-camera
Signal to Noise Ratio (SNR)

You want as much light as you can get

Some strategies:
● Bigger sensors
● Larger fill factor
● Minimize off time (shutters)
● Long exposures (HDR)
Full frame sensor gathers 50x more light than phone sensor
CMOS sensor

photodiode

https://micro.magnet.fsu.edu/primer/digitalimaging/cmosimagesensors.html http://www.ysctech.com/digital-microscope-CCD-camera-info.html
CMOS fabrication
CMOS vs. CCD

CCD (Charged Coupled Device) CMOS

+ Larger fill factor + Random access to pixels
- Harder to read out image + Cheaper to manufacture
- More expensive - Lower fill factor
https://meroli.web.cern.ch/lecture_cmos_vs_ccd_pixel_sensor.html
Shutters

CCD requires a shutter CMOS

- Stop exposing while data read off + Can expose all the time (rolling shutter)
- Causes some new artifacts

https://meroli.web.cern.ch/lecture_cmos_vs_ccd_pixel_sensor.html
Rolling shutter
 Fascinating!
But what does that tell us
about the Van Gogh?
Camera sensing every other pixel

read-out
electronics

CMOS sensor

photodiode

CMOS sensor
Photosensitive area (fill factor) often < 50%
f(x)
f(x)
g(x)

f(x)
f(x)
 g(x) is an “alias” of
Aliasing

f(x)

f(x)
 g(x) ~=
Aliasing

f(x)

f(x)

No aliasing in this case

Sampling and the Nyquist rate

< 1 sample per cycle > 10 samples per cycle

To avoid aliasing
● ≥ two samples per cycle

This minimum sampling rate is called the Nyquist rate

But this only works for sine waves, right?

Fourier transform
● Decomposes any signal or image into weighted sum of sines and cosines

To avoid aliasing
● sampling rate ≥ 2 * max frequency present in the image (Nyquist rate)
Low pass filtering (optical)

low pass filter

“antialias filter”
“blurring filter”
CMOS sensor

To avoid aliasing
● Low pass filtering: block the high frequencies, but allow the low frequencies to pass through
with antialias filter No antialias filter

From Dave Etchells, Imaging Resource

Going back to image resizing...

What’s the right way to do this?

● We want a digital version of this low-pass filter

CMOS sensor
Image as array of pixels

F[x,y]
An array of numbers (“pixels”)
x,y are integer column/row indices
Image as array of pixels

F[x,y]
An array of numbers (“pixels”)
x,y are integer column/row indices
Image as 2D function

0 1 2
x 3 4 5
0

2
y
3

f(x,y)
a continuous function
x,y are floats
 Relation of array and function representations
x
f(-½ ,-½) 0 1 2 3 4 5
0

1 F[x,y] gives value at center of that pixel

● f(x,y) = F[x,y] when x,y are integers
2 ● So top-left of pixel grid is at (-½, -½)
y ● Important for image resizing
3
calculations, e.g.,
4 (x’,y’) = (-½,-½) + scale*(x+½,y+½)
Leaves the top left corner fixed during
5 resizing -- try plugging in (x,y) = (-½,-½)

f(x,y)
a continuous function
x,y are floats
We’ll use both representations

0 1 2
x 3 4 5
0

2
y
3

F[x,y] f(x,y)
An array of numbers (“pixels”) A continuous function
x,y are integer column/row indices x,y are floats
 Going back to the resizing problem

Mean intensity over

the region to be
shrunk down

1/2
Mean filtering

0 0 0 0 0 0 0 0 0 0

0 0 0 0 0 0 0 0 0 0

0 0 0 90 90 90 90 90 0 0

0 0 0 90 90 90 90 90 0 0

0 0 0 90 90 90 90 90 0 0

0 0 0 90 0 90 90 90 0 0 80 90

0 0 0 90 90 90 90 90 0 0 50 50

0 0 0 0 0 0 0 0 0 0 30 30

0 0 90 0 0 0 0 0 0 0 10

F[x,y G[x,y
0 0 0 0 0 0 0 0 0 0
What about the borders?

Multiple options…

Option 1: don’t evaluate borders of G

Option 2: pad F with extra rows/columns at left/right/top/bottom borders, and fill in using various options, e.g.
Cross-correlation filtering

Let’s write this down as an equation (assume mean window (2k+1)x(2k+1))

We can generalize this idea by allowing weights for different neighboring pixels

This is called a cross-correlation and denoted

H is called the “filter,” “kernel,” or “mask”

The above allows negative filter indices. Need to use H[u+k,v+k] when you implement.
Mean filtering

0 0 0 0 0 0 0 0 0 0
What’s the kernel for a 3x3 mean filter?
0 0 0 0 0 0 0 0 0 0

0 0 0 90 90 90 90 90 0 0

0 0 0 90 90 90 90 90 0 0
1/ 1/ 1/
0 0 0 90 90 90 90 90 0 0 9 9 9
0 0 0 90 0 90 90 90 0 0
1/ 1/ 1/

H[u,v1/
0 0 0 90 90 90 90 90 0 0 9 9 9

9] 9
0 0 0 0 0 0 0 0 0 0
1/ 1/
0 0 90 0 0 0 0 0 0 0 9

F[x,y
0 0 0 0 0 0 0 0 0 0
Gaussian filtering Less weight to pixels further from center

0 0 0 0 0 0 0 0 0 0

0 0 0 0 0 0 0 0 0 0

0 0 0 90 90 90 90 90 0 0

0 0 0 90 90 90 90 90 0 0
1 2 1
0 0 0 90 90 90 90 90 0 0
2 4 2
0 0 0 90 0 90 90 90 0 0

H[u,v
0 0 0 90 90 90 90 90 0 0 1 2 1

]
0 0 0 0 0 0 0 0 0 0

0 0 90 0 0 0 0 0 0 0

F[x,y
0 0 0 0 0 0 0 0 0 0 Discrete approx above
You’ll do a more accurate version in HW2
Mean vs. Gaussian
Exercise -- why does mean filtering look “boxy”?

0 0 0 0 0 0 0

0 0 0 0 0 0 0

0 0 0 0 0 0 0 1 1 1 1 1 1

0 0 0 1 0 0 0 1 1 1 1 1 1

0 0 0 0 0 0 0 1 1 1

H[u,v
1 1 1
0 0 0 0 0 0 0

]
F[x,y G[x,y
0 0 0 0 0 0 0

] ]
Exercise -- why does mean filtering look “boxy”?

0 0 0 0 0 0 0 0 0 0 0 0 0 0

0 0 0 0 0 0 0 0 0 0 0 0 0 0

0 0 0 0 0 0 0 1 1 1 0 0 1 1 1 0 0

0 0 0 1 0 0 0 1 1 1 0 0 1 1 1 0 0

0 0 0 0 0 0 0 0 0 1 1 1 0 0

H[u,v
1 1 1
0 0 0 0 0 0 0 0 0 0 0 0 0 0

]
F[x,y G[x,y
0 0 0 0 0 0 0 0 0 0 0 0 0 0

] ]
Exercise: filtering an impulse

0 0 0 0 0 0 0

0 0 0 0 0 0 0

0 0 0 0 0 0 0 a b c
0 0 0 1 0 0 0 d e f
0 0 0 0 0 0 0

H[u,v
g h i
0 0 0 0 0 0 0

]
F[x,y G[x,y
0 0 0 0 0 0 0

] ]
Exercise: filtering an impulse

0 0 0 0 0 0 0

0 0 0 0 0 0 0

0 0 0 0 0 0 0 a b c i h g

0 0 0 1 0 0 0 d e f f e d

0 0 0 0 0 0 0 c b a

H[u,v
g h i
0 0 0 0 0 0 0

]
F[x,y G[x,y
0 0 0 0 0 0 0

] ]
result flips horizontally and vertically!
Filtering demo

http://matlabtricks.com/post-5/3x3-convolution-kernels-with-online-demo#demo

https://setosa.io/ev/image-kernels/
Convolution

A convolution is a cross-correlation where the filter is flipped both horizontally and vertically before being
applied to the image

It is written

Suppose H is a Gaussian. How does convolution differ from cross-correlation?

Suppose F is an impulse function (previous slide). What will G look like?

Low pass filtering
What is the digital version of this filter?

?
CMOS sensor
Low pass filtering
What is the digital version of this filter?

?
CMOS sensor
Gaussian does a pretty good job
Filtered sub-sampling

Solution: low-pass filter the image first, then sub-sample

● Filter size should double for each ½ size reduction
Prefiltered sub-sampling
No prefiltering
Temporal aliasing
How about up-resing?

Making the image bigger

One example: Debayering

https://en.wikipedia.org/wiki/Bayer_filter

¼ of pixels see red light (e.g.)

• Q: how do you get red at every pixel?
• A: Need to interpolate -- called debayering
General resizing

Resample the image to a new pixel grid

x x

y y

f(x,y) f(x,y)
General resampling

Resample the image to a new pixel grid

x x

y y

f(x,y) f(x,y)
General resampling

x
0 1 2 3 4
0
1 How to interpolate?

2 Evaluate f between grid points

y
3
4

f(x,y)
Nearest neighbor

x
0 1 2 3 4
0
1

2
y
3
4

f(x,y)
Bilinear interpolation

x
0 1 2 3 4
0
1

2
y
3
4
q = V1A1 + V2A2 + V3A4 + V4A4?

f(x,y)
Bilinear interpolation via filtering

F[x]

1
0.5 h
0 1 1.5 2 3 4
x
Bilinear interpolation via filtering

F[x]

1
0.5 h
0 1 2 3 4
x
2D interpolation filters

Bicubic even better

● fit 3rd degree polynomial surface to pixels in neighborhood
● Can also be implemented by convolution filter
Bicubic vs bilinear
High pass filter

Suppose H is a low pass filter and F is an image that we want to filter

This gives us enough information to create a high pass filtered version of F

highpass(F) = F - H⊗F

⊗
Filters are linear operators

That means

H1 ⊗ F + H2 ⊗ F = (H1 + H2) ⊗ F

This means we can implement our high pass operator as a filter:

F-H⊗F = I⊗F-H⊗F
= (I - H) ⊗ F
Filtering demo

http://matlabtricks.com/post-5/3x3-convolution-kernels-with-online-demo#demo

https://setosa.io/ev/image-kernels/
Image gradient

The gradient of an image:

The gradient points in the direction of most rapid change in intensity

Image gradient

How can we differentiate a digital image?

How can we write this as a cross-correlation filter?

-1 1

H[u,v
]
Filtering demo

http://matlabtricks.com/post-5/3x3-convolution-kernels-with-online-demo#demo

https://setosa.io/ev/image-kernels/
The Sobel operator

Better approximations of the gradient exist

-1 0 1 1 2 1
⅛ -2 0 2 ⅛ 0 0 0
-1 0 1 -1 -2 -1
The standard definitions of these Sobel filters omit the ⅛ term
● ⅛ not needed for edge detection
● ⅛ is needed to correctly compute gradient values
Low pass filtering
What is the digital version of this filter?

?
Is the Gaussian really a low-pass filter?

If so why?
CMOS sensor
And can we do better?
Fourier Transform

Fourier transform
● Decomposes any signal or image into weighted sum of sines and cosines
Fourier Series

We want to get this function

Following slides from Alyosha Efros

Fourier Series

≈ +

We want to get this function

Following slides from Alyosha Efros

Fourier Series

≈ +

We want to get this function

=
Fourier Series

≈ +

We want to get this function

=
Fourier Series

≈ +

We want to get this function

=
Fourier Series

≈ +

We want to get this function

=
Fourier Series

We want to get this function We’ll get there in the limit

Fourier transform

f(x) |F(𝟂)|
signal Fourier transform of signal
Fourier transform as change of basis

* =
The math

Where are the sines and cosines?

The result is a complex function

We’ve been showing only the amplitude A so far:

Phase is also encoded:

Nice tutorial on Fourier Series
Nice tutorial on Fourier transform
2D Fourier Transform
What does a sine look like in 2D?

http://sharp.bu.edu/~slehar/fourier/fourier.html#filtering
Fourier Transform of an image

f(x,y) |f(𝟂)|
2D Fourier Transform
Low pass filtering in the Fourier domain

h(𝟂) |f(𝟂)|
Low pass filtering in the Fourier domain

h(𝟂) |f(𝟂)|

This is a product of two functions, not a convolution

Low pass filtering in the Fourier domain

f(x,y) |h(𝟂) f(𝟂)| FT-1[h(𝟂)f(𝟂)]

The Convolution Theorem

FT[h﹡f ] = FT[h] FT[f ]

Convolution in the spatial domain = multiplication in the Fourier domain

FT-1[h f ] = FT-1[h ]﹡FT-1[f ]

Works for inverse Fourier transforms too:
Applying the convolution theorem

FT-1 =

FT-1
﹡ FT-1
Applying the convolution theorem

FT-1 =

FT-1
﹡ FT-1
Applying the convolution theorem

FT-1 =

﹡
The “ideal” low pass filter

﹡ =
sinc[u,v] F[x,y] G[x,y]

Wait a minute, it still looks soiled!

Remember what happens when you filter an impulse

0 0 0 0 0 0 0

0 0 0 0 0 0 0

0 0 0 0 0 0 0 i h g

0 0 0 1 0 0 0 f e d

0 0 0 0 0 0 0 c b a

H[u,v
0 0 0 0 0 0 0

]
F[x,y G[x,y
0 0 0 0 0 0 0

] ]
This phenomenon with sinc known as “ringing”
Be careful what you wish for...

The “ideal” low-pass filter is not that good

● produces ringing artifacts
● requires infinite size
● seldom what you want anyway

What went wrong?

● Our goal is to remove high frequencies
● Not to pass through all low frequencies untouched
Fade out the high frequencies

f(x,y) |h(𝟂) f(𝟂)| FT-1[h(𝟂)f(𝟂)]

Applying the convolution theorem

FT-1 =

﹡
Gaussian(𝛔
)

Gaussian(1/𝛔)
And that’s why filtering with a Gaussian works...

Lesson learned!
● Before you downsample an image, filter with a Gaussian (or similar filter)!
The case
of the
Soiled Van Gogh

SOLVED!
Epilogue
Lots of other types of filters: median

Median pixel value in nxn window

Lots of other types of filters: bilateral

original mean filtered Bilateral filtered

you’ll (optionally) do this in HW2