Faculty of Information Technology

Ton Duc Thang University

August 2023
 Image transformations
 As with any function, we can apply operators to an image

g (x,y) = f (x,y) + 20 g (x,y) = f (-x,y)

 We‟ll talk about a special kind of operator, convolution (linear filtering)

@2023 Pham Van Huy and Trinh Hung Cuong 2

 Question: Noise reduction
 Given a camera and a still scene, how can you reduce noise?

Source: S. Seitz
Take lots of images and
average them! What‟s the next
best thing?
@2023 Pham Van Huy and Trinh Hung Cuong 3
 Image filtering
 Modify the pixels in an image based on some function of a local
neighborhood of each pixel

Some
10 5 3 function
4 5 1 7
1 1 7

Local image data Modified image data

Source: L. Zhang

@2023 Pham Van Huy and Trinh Hung Cuong 4

 Image filtering

@2023 Pham Van Huy and Trinh Hung Cuong 5

@2023 Pham Van Huy and Trinh Hung Cuong 6
 Linear filtering
 One simple version: linear filtering (cross-correlation, convolution)
 Replace each pixel by a linear combination of its neighbors

 The prescription for the linear combination is called the “kernel” (or “mask”,
“filter”)

10 5 3 0 0 0
4 6 1 0 0.5 0 8
1 1 8 0 1 0.5

Local image data kernel Modified image data

Source: L. Zhang

@2023 Pham Van Huy and Trinh Hung Cuong 7

 YTB

@2023 Pham Van Huy and Trinh Hung Cuong 8

 Cross-correlation
 Let F be the image, H be the kernel (of size 2𝑘 + 1 × 2𝑘 + 1), and G be the
output image:

This is called a cross-correlation operation:

 When the aperture is partially outside the image, the operation interpolates outlier
pixel values according to the specified border mode (refers [1] )

@2023 Pham Van Huy and Trinh Hung Cuong 9

 Cross-correlation (ct)
 How similar the kernel is to the image at any point [2]
 Used for image alignment and simple image matching

 Refers [3] [4] more about template matching and normalized cross-correlation.

@2023 Pham Van Huy and Trinh Hung Cuong 10

@2023 Pham Van Huy and Trinh Hung Cuong 11 11
@2023 Pham Van Huy and Trinh Hung Cuong 12
 Ex. 1
 Apply cross-correlation operation into the following image F:

F H G
1 2 3 1 1 1 ? ? ?
9 9 9
4 5 6 1 1 1 ? 48 ?
9 9 9 9

8 9 10 1 1 1 ? ? ?
9 9 9

@2023 Pham Van Huy and Trinh Hung Cuong 13

 Ex. 2
 Apply normalized cross-correlation operation to locate the best matching of
H in the image F (zero padding for the border pixels):
F H G
1 1 1 1 1 1 3 3 2 47 ? ? ? ? ?

1 20 2 2 2 1 3 3 2 ? ? ? ? ? ?

1 2 3 3 2 1 2 2 2 ? ? ? ? ? ?

1 2 3 3 2 1 ? ? ? ? ? ?

1 2 2 2 2 1 ? ? ? ? ? ?

1 1 1 1 1 1 ? ? ? ? ? ?

@2023 Pham Van Huy and Trinh Hung Cuong 14

 https://www.youtube.com/watch?app=desktop&v=_HATc2zAhcY

@2023 Pham Van Huy and Trinh Hung Cuong 15

 Convolution
 Same as cross-correlation, except that the kernel is “flipped” (horizontally
and vertically)

This is called a convolution operation:

 Convolution is commutative and associative

@2023 Pham Van Huy and Trinh Hung Cuong 16

 2D Convolution
 g(x,y) = h(x,y) * f(x,y)
 f, g: input/output
 h: mask/filter/kernel

 Flip the mask (horizontally and vertically) only

once
 Slide the mask onto the image.
 Multiply the corresponding elements and then
add them
 Repeat this procedure until all values of the
image has been calculated.

@2023 Pham Van Huy and Trinh Hung Cuong 17

 Example
 http://www.songho.ca/dsp/convolution/convolution2d_example.html

@2023 Pham Van Huy and Trinh Hung Cuong 18

 CONVOLUTION

Adapted from F. Durand

https://www.allaboutcircuits.com/uploads/article
s/Fig2_2D_Conv.jpg

@2023 Pham Van Huy and Trinh Hung Cuong 19

 Convolution applications
 Blur image
 Remove noise
 Sharpening
 Smoothing
 Edge detection
 ...

https://www.geeksforgeeks.org/python-opencv-filter2d-function/

@2023 Pham Van Huy and Trinh Hung Cuong 20

 Ex. 3
 Apply convolution operation into the following image F:

F H G
1 2 3 -1 0 1 ? ? ?

4 5 6 -2 0 2 ? ? ?

8 9 10 -1 0 1 ? ? ?

@2023 Pham Van Huy and Trinh Hung Cuong 21

@2023 Pham Van Huy and Trinh Hung Cuong 22
 Linear filters:
examples

0 0 0
=
* 0
0
1
0
0
0

Original Identical image

Source: D. Lowe

@2023 Pham Van Huy and Trinh Hung Cuong 23

 Linear filters:
examples

0 0 0
=
* 1
0
0
0
0
0

Original Shifted left By 1 pixel

Source: D. Lowe

@2023 Pham Van Huy and Trinh Hung Cuong 24

 Linear filters:
examples

1 1 1

=
* 1
1
1
1
1
1

Original Blur (with a mean filter)

Source: D. Lowe

@2023 Pham Van Huy and Trinh Hung Cuong 25

 Mean
filtering
0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0 0 10 20 30 30 30 20 10
0 0 0 90 90 90 90 90 0 0 0 20 40 60 60 60 40 20
0 0 0 90 90 90 90 90 0 0 0 30 60 90 90 90 60 30
1 1 1
1
1
1
1
1
1
* 0
0
0
0
0
0
0
0
0
90
90
90
90 90
0 90
90 90
90
90
90
90
90
90
0
0
0
0
0
0
= 0
0
0
30
30
20
50
50
30
80
80
50
80
80
50
90
90
60
60 30
60 30
40 20

0 0 0 0 0 0 0 0 0 0 10 20 30 30 30 30 20 10

0 0 90 0 0 0 0 0 0 0 10 10 10 0 0 0 0 0

0 0 0 0 0 0 0 0 0 0

@2023 Pham Van Huy and Trinh Hung Cuong 26

 Ex. 4
 Apply the filtering (cross-correlation) into the following image F (zero padding
at the borders):
F H G
1 1 1 1 1 1 0 0 0 ? ? ? ? ? ?

1 180 180 180 180 1 1 0 0 ? ? ? ? ? ?

1 180 96 96 180 1 0 0 0 ? ? ? ? ? ?

1 180 96 96 180 1 ? ? ? ? ? ?

1 180 180 180 180 1 ? ? ? ? ? ?

1 1 1 1 1 1 ? ? ? ? ? ?

@2023 Pham Van Huy and Trinh Hung Cuong 27

 Ex. 5
 Apply the filtering (cross-correlation) into the following image F (zero padding
at the borders):
F H G
1 1 1 1 1 1 0 0 0 ? ? ? ? ? ?

1 180 180 180 180 1 0 0 1 ? ? ? ? ? ?

1 180 96 96 180 1 0 0 0 ? ? ? ? ? ?

1 180 96 96 180 1 ? ? ? ? ? ?

1 180 180 180 180 1 ? ? ? ? ? ?

1 1 1 1 1 1 ? ? ? ? ? ?

@2023 Pham Van Huy and Trinh Hung Cuong 28

 Linear filters:
examples

0 0 0 1 1 1
- =
* 0
0
2 0
0 0
1
1
1 1
1 1

Sharpening
Original
filter
(accentuates
Source: D. Lowe edges)

@2023 Pham Van Huy and Trinh Hung Cuong 29

 SHARPENING

Source: D. Lowe

@2023 Pham Van Huy and Trinh Hung Cuong 30

 Sharpening
 emphasizes differences in adjacent pixel values
 accentuating the edges of the image
 add contrast to edges

https://i.stack.imgur.com/XXBUN.png

@2023 Pham Van Huy and Trinh Hung Cuong 31

 Ex. 6
 Apply the filtering (cross-correlation) into the following image F (zero padding
at the borders):
F H G
20 20 20 20 20 20 0 -1 0 ? ? ? ? ? ?

20 120 120 120 120 20 -1 5 -1 ? ? ? ? ? ?

20 120 20 20 120 20 0 -1 0 ? ? ? ? ? ?

20 120 20 20 120 20 ? ? ? ? ? ?

20 120 120 120 120 20 ? ? ? ? ? ?

20 20 20 20 20 20 ? ? ? ? ? ?

@2023 Pham Van Huy and Trinh Hung Cuong 32

 SMOOTHING WITH BOX FILTER
REVISITED

Source: D. Forsyth

@2023 Pham Van Huy and Trinh Hung Cuong 33

 GAUSSIAN KERNEL

https://theailearner.com/2019/05/0
6/gaussian-blurring/

Source: C. Rasmussen

@2023 Pham Van Huy and Trinh Hung Cuong 34

 Discrete approximation of the Gaussian kernels

https://www.researchgate.net/figure/Discrete-approximation-of-the-Gaussian-kernels-
3x3-5x5-7x7_fig2_325768087

@2023 Pham Van Huy and Trinh Hung Cuong 35

 Gaussian blur
 Use a weighted mean: the values near the center pixel will have a higher
weight
 probably get a less blurred image but a natural blurred image because it handles the
edge values very well

https://theailearner.com/tag/gaussian-filter/

@2023 Pham Van Huy and Trinh Hung Cuong 36

 Gaussian kernel - properties
 Gaussian kernel is linearly separable: can break any 2-d filter into two 1-d filters
 Applying multiple successive Gaussian kernels is equivalent to applying a single, larger
Gaussian blur

 Gaussian kernel weights(1-D) can be obtained quickly using the Pascal‟s Triangle

https://theailearner.com/tag/gaussian-filter/
@2023 Pham Van Huy and Trinh Hung Cuong 37
 Gaussian blur – code

@2023 Pham Van Huy and Trinh Hung Cuong 38

 Gaussian filters

=1 =5 = 10 = 30
pixel pixels pixels pixels

@2023 Pham Van Huy and Trinh Hung Cuong 39

 Sharpening revisited

– =
original smoothed (5x5) detail sharpened
Source: S. Lazebnik

Let‟s add it back:

+ = F+𝛼 𝐹−𝐻∗𝐹
α image blurred
original detail image
@2023 Pham Van Huy and Trinh Hung Cuong 40
 Sharpen filter

unfiltered

filtered

@2023 Pham Van Huy and Trinh Hung Cuong 41

 Convolution in the real world
Camera shake

= *
Source: Fergus, et al. “Removing Camera Shake from a Single Photograph”, SIGGRAPH 2006

Source: http://lullaby.homepage.dk/diy-
Bokeh: Blur out-of-focus regions of an image. camera/bokeh.html

@2023 Pham Van Huy and Trinh Hung Cuong 42

 Rank filters
 Rank filters assign the k-th value of the gray levels from the window
consisting of M pixels sorted in ascending order [code]
 The special cases k = 1, k = M (MIN and MAX filter) : erosion and dilation
 k = (M + 1)/2 : median filter

 Generalisation of flat dilation/erosion: in lieu of min or max value in window,

use the k-th ranked value
 Increases robustness against noise
 Best-known example: median filter for noise reduction

 Concept useful for both gray-level and binary images

 All rank filters are commutative with thresholding

@2023 Pham Van Huy and Trinh Hung Cuong 43

 Rank filters - benefits
 image quality enhancement, e.g., image smoothing, sharpening
 image pre-processing, e.g., noise reduction, contrast enhancement
 feature extraction, e.g., border detection, isolated point detection
 image post-processing, e.g., small object removal, object grouping, contour
smoothing

@2023 Pham Van Huy and Trinh Hung Cuong 44

 Median filter

@2023 Pham Van Huy and Trinh Hung Cuong 45

 Median filter

https://www.researchgate.net/figure/Graphic-Depiction-of-Rank-Order-Filter-
Operation_fig6_268373873

@2023 Pham Van Huy and Trinh Hung Cuong 46

 Majority filter: example

Binary image with

3x3 majority filter 20% „Salt&Pepper‟ noise 3x3 majority filter
5% „Salt&Pepper‟ noise

@2023 Pham Van Huy and Trinh Hung Cuong 47

 Median filter: example

Original 5% „Salt&Pepper‟ 3x3 median 7x7 median

image noise filtering filtering
@2023 Pham Van Huy and Trinh Hung Cuong 48
 Example: non-uniform lighting compensation

Original image Dilation (local max) Rank filter

1632x1216 pixels 61x61 structuring element 10st brightest pixel
61x61 structuring element

@2023 Pham Van Huy and Trinh Hung Cuong 49

 References
1. https://docs.opencv.org/4.x/d4/d86/group__imgproc__filter.html
2. https://www.youtube.com/watch?app=desktop&v=kGHz-cEyjiE
3. https://docs.opencv.org/4.x/d4/dc6/tutorial_py_template_matching.html
4. https://www.youtube.com/watch?app=desktop&v=kGHz-cEyjiE
5. https://vincmazet.github.io/bip/filtering/convolution.html

@2023 Pham Van Huy and Trinh Hung Cuong 50