CSC 447

Digital Image
Processing

11/7/2022 1
[email protected]
 Image
Segmentation I
Fundamental Steps of DIP
Wavelet and
Multi- Compression
resolution
Output is generally images

Color Image Morphological

Output is generally attributes

Processing Processing

Knowledge base
Restoration Segmentation

Representation
Enhancement and
Description

Object
Acquisition Recognition
Problem Domain
Contents

1. Basics of Image Segmentation

2. Edge-based Segmentation
3. Thresholding
4. Region-based Segmentation
Basics of Image Segmentation
• Segmentation is to subdivide an image into
its constituent regions or objects.
• Typically the first step in any automated
computer vision application.
Basics of Image Segmentation – (cont.)
• Automated segmentation is a very difficult
problem in general.

• The results of segmentation are very

important in determining the eventual
success or failure of image analysis.
Basics of Image Segmentation – (cont.)
Examples
Basics of Image Segmentation – (cont.)
Examples
Basics of Image Segmentation – (cont.)
Examples
Basics of Image Segmentation – (cont.)
• Monochrome image segmentation algorithms are
generally based on:
1- Edge-based Approaches
Use the boundaries of regions to segment the image
by detecting abrupt changes in intensity
(discontinuities) → edge detection
2- Region-based Approaches
Use similarity among pixels to partition an image
into regions according to a set of predefined criteria.
→ thresholding, region growing, merging, splitting…
Basics of Image Segmentation – (cont.)
 Edge-based

11/7/2022 12
Edge-based Approaches
- Discontinuities
Point, line, edge

- Discontinuity Detectors  f 
st
 x 
1 derivative (Gradient). f =  
nd  f 
2 derivative (Laplacian).
 y 
 2
f  2
f
 f = 2 + 2
2

 x  y
Recall - Sharpening Filters
1st Derivative Example in 1D:

A B
Recall - Sharpening Filters
1st Derivative Example in 1D:
Image Strip

8
7
6
5
4
3
2
1
0

1st Derivative

8
6
4
2
0
-2
-4
-6
-8
Recall - Sharpening Filters
2nd Derivative:
The formula for the 2nd derivative of a function
is as follows:
 f
2
= f ( x + 1) + f ( x − 1) − 2 f ( x )
 x
2

Simply takes into account the values both

before and after the current value.
Sharpening Filters – (cont.)
2nd Derivative Example in 1D:
8
Image Strip

7
6
5
4
3
2
1
0

2nd Derivative

10

-5

-10

-15
 8
7
6
5

Profile 4
3
2
1
0
1st Derivative

8
6
4
2
0

1st -2
-4
-6
-8
2nd Derivative

10

-5

2nd -10

-15
Edge-based Approaches – (cont.)
- Isolated Points
Which detector?
Edge-based Approaches – (cont.)
- Isolated Points
Laplacian + threshold
Edge-based Approaches – (cont.)
- Lines
Which detector?
Edge-based Approaches – (cont.)
- Lines
Laplacian + deal with double-line effect.
Edge-based Approaches – (cont.)
- Lines
Gradient for each direction?
Edge-based Approaches – (cont.)
- Lines
Gradient for each direction?

NO!
Edge-based Approaches – (cont.)
- Lines
Masks for specific direction + threshold the
absolute value
Edge-based Approaches – (cont.)
- Lines
Edge-based Approaches – (cont.)
- Edge Models
Edge-based Approaches – (cont.)
- Ramp Edge Model

• Magnitude of 1st derivative.

• Zero crossing of 2nd derivative.
Edge-based Approaches – (cont.)
• Derivative-based edge detectors are
extremely sensitive to noise.
Edge-based Approaches – (cont.)
• Image smoothing should be a serious
consideration prior to the use of derivatives
in applications where noise is likely to be
present.
• Another approach is to threshold the
gradient image.
• Sometimes, both smoothing and
thresholding of gradient are used.
Edge-based Approaches – (cont.)
Examples Original Image Horizontal Gradient Component

Vertical Gradient Component Combined Edge Image

Edge-based Approaches – (cont.)
Examples Smoothed Image Horizontal Gradient Component

Vertical Gradient Component Combined Edge Image

Edge-based Approaches – (cont.)
- Marr-Hildreth Edge Detector
• The Laplacian is too sensitive to noise, so
usually when used for edge detection, it is
combined with a smoothing Gaussian filter
𝒙𝟐 +𝒚𝟐
−
𝑮 𝒙, 𝒚 = 𝒆 𝟐𝝈𝟐.
The mask can be of arbitrary size and is called
The Laplacian of a Gaussian (LoG):
𝟐 𝟐 𝟐 𝒙𝟐 +𝒚𝟐
𝟐
𝒙 + 𝒚 − 𝟐𝝈 −
𝛁 𝑮 𝒙, 𝒚 = 𝒆 𝟐𝝈𝟐
𝝈𝟒
Edge-based Approaches – (cont.)
- Marr-Hildreth Edge Detector
Edge-based Approaches – (cont.)
- Marr-Hildreth Edge Detector
Edge-based Approaches – (cont.)
- Canny Edge Detector
• A good approximation to the optimal step
edge detector and is based on the 1st
derivative of a Gaussian
𝒅 𝒙𝟐 −𝒙 − 𝟐𝒙𝟐
− 𝟐
𝒆 𝟐𝝈 = 𝒆 𝟐𝝈
𝒅𝒙 𝝈𝟐
• Involves smoothing, computing the gradient,
computing edge angles, and thresholding.
• Trade-off between improved performance
and speed.
Edge-based Approaches – (cont.)
- Canny Edge Detector
Edge-based Approaches – (cont.)
- Fundamental steps for edge detection:
1- Image smoothing
2- Edge points detection
3- Edge localization
- Local Processing
- Regional Processing
- Global Processing (Hough Transform)
 Hough Transform

11/7/2022 39
Edge-based Approaches – (cont.)
- Hough Transform
• Based on whether a set of candidate pixels
lie on curves of a specified shape.
• Once detected, these curves form edges or
boundaries of regions.
Edge-based Approaches – (cont.)
- Hough Transform - Line
• A line in the 𝒙𝒚-plane can be expressed by
the slope-intercept form
𝒚 = 𝒂𝒙 + 𝒃
• Where
𝒂 = slope
𝒃 = 𝒚-intercept
Edge-based Approaches – (cont.)
- Hough Transform - Line
• On the other hand, the line can be
represented in the parameter space as:
𝒃 = −𝒂𝒙 + 𝒚
Edge-based Approaches – (cont.)
- Hough Transform - Line
• Hence, principal lines in 𝒙𝒚-plane can be
found by identifying points in 𝒂𝒃-plane
where large number of lines intersect.
Edge-based Approaches – (cont.)
- Hough Transform - Line
Edge-based Approaches – (cont.)
- Hough Transform - Line
• For each edge point (𝒙𝒌 , 𝒚𝒌 ), loop every
possible 𝒂𝒊 , solve for 𝒃𝒋 , and accumulate
𝒃 = −𝒂𝒙 + 𝒚
𝑨(𝒊, 𝒋).

𝑨(𝒊, 𝒋)
Edge-based Approaches – (cont.)
- Hough Transform - Line
• Pixels, which lie on the line, need not all be
contiguous (HT works even with occlusions).
Edge-based Approaches – (cont.)
- Hough Transform - Line
• Problem!

Vertical lines
have slope ~ ∞
and
𝒃 = −𝒂𝒙 + 𝒚
fails
Edge-based Approaches – (cont.)
- Hough Transform - Line
• Use the normal representation of a line
(polar coordinates)
𝒄𝒐𝒔 𝜽 𝝆
𝐲= − 𝒙+
𝒔𝒊𝒏 𝜽 𝒔𝒊𝒏 𝜽
Edge-based Approaches – (cont.)
- Hough Transform - Line
• A line can be represented in the parameter
space as: 𝛒 = 𝒙 𝒄𝒐𝒔 𝜽 + 𝒚 𝒔𝒊𝒏 𝜽
Edge-based Approaches – (cont.)
- Hough Transform - Line
• Subdivide 𝝆𝜽-space into accumulative cells
such that 𝜽𝒎𝒊𝒏 , 𝜽𝒎𝒂𝒙 : −𝟗𝟎 ≤ 𝜽 ≤ 𝟗𝟎
−𝑫 ≤ 𝝆 ≤ 𝑫 D=minimum distance between
opposite corners of image

𝑨(𝒊, 𝒋)
Edge-based Approaches – (cont.)
- Hough Transform - Line
• Number of subdivisions (steps of 𝜽 and 𝝆)
determines accuracy of colinearity of points.

𝑨(𝒊, 𝒋)
Edge-based Approaches – (cont.)
- Hough Transform - Line
• For each edge point (𝒙𝒌 , 𝒚𝒌 ), loop every
possible 𝜽𝒊 , solve for 𝝆𝒋 , and accumulate
𝑨(𝒊, 𝒋).

𝑨(𝒊, 𝒋)
Edge-based Approaches – (cont.)
- Hough Transform - Line
• Finally, examine continuity of pixels in high
count accumulator cells; usually by the
distance between disconnected pixels.
• Gaps are bridged if length of gap is less than
a certain value.
Edge-based Approaches – (cont.)
- Hough Transform - Line
Example
Edge-based Approaches – (cont.)
- Hough Transform - Line
Extract the two
edges of the
principal runway.
Edge-based Approaches – (cont.)
- Hough Transform - Line
Edge-based Approaches – (cont.)
- Hough Transform vs Probabilistic HT
Edge-based Approaches – (cont.)
- Hough Transform - Circular
Other Segmentations Methods
Active Contours
Snakes, Scissors, Level Sets
Split and Merge
Watershed, region split/merge, graph
Mean Shift
Graph Cuts
Examples
(a) active contours
(b) level sets
(c) graph-based merging
(d) mean shift (e) texture and intervening
contour-based normalized cuts
(f) binary MRF solved using graph cuts

Richard Szeliski, Computer Vision Algorithms and

Applications, Springer 2010. Ch. 5
Examples – (cont.)
Active Contours - Snakes

(a) the “snake pit” for interactively controlling shape; (b) lip tracking.

Richard Szeliski, Computer Vision Algorithms and Applications, Springer 2010. Ch. 5
Examples – (cont.)
Active Contour - Scissors

(a) as the mouse traces the white path, the scissors follow the orange path along
the object boundary (the green curves show intermediate positions).
(b) regular scissors can sometimes jump to a stronger (incorrect) boundary.
(c) after training to the previous segment, similar edge profiles are preferred.

Richard Szeliski, Computer Vision Algorithms and Applications, Springer 2010. Ch. 5
Examples – (cont.)
Active Contours - Level Sets

The initial circles evolve towards an accurate segmentation of foreground and background,
adapting their topology as they evolve. (Grayscale).

Richard Szeliski, Computer Vision Algorithms and Applications, Springer 2010. Ch. 5
Examples – (cont.)
Active Contours - Level Sets

The initial circles evolve towards an accurate segmentation of foreground and background,
adapting their topology as they evolve. (Color).

Richard Szeliski, Computer Vision Algorithms and Applications, Springer 2010. Ch. 5
 Thresholding

11/7/2022 65
Thresholding
• Thresholding is usually a first or intermediate
step in any segmentation approach.

• Segmenting an image is based on direct

intensity values or properties of these
values.
 Original Image Thresholded Image

Threshold Too Low Threshold Too High

Thresholding – (cont.)
• Thresholding based on the histogram of the
image differentiates between modes.

• When intensity distribution of objects and

background are sufficiently distinct, a single
values can be used to segment.
Thresholding – (cont.)
Thresholding can be viewed as
• 𝑻 = 𝑻[𝒙, 𝒚, 𝒑 𝒙, 𝒚 , 𝒇(𝒙, 𝒚)]
𝒑(𝒙, 𝒚): some neighborhood property.
𝑻 = 𝑻[𝒇(𝒙, 𝒚)] T is applied to the entire image Single constant
→ Global thresholding threshold

𝑻 = 𝑻[𝒑 𝒙, 𝒚 , 𝒇(𝒙, 𝒚)] 𝑇 depends on properties of

neighborhood → Local thresholding Variable
thresholds
𝑻 𝑇 depends on spatial coordinates
= 𝑻[𝒙, 𝒚, 𝒑 𝒙, 𝒚 , 𝒇(𝒙, 𝒚)]→ Dynamic/adaptive thresholding
Thresholding – (cont.)
Basic Global Thresholding
• The success of this technique depends very
strongly on how well the histogram can be
partitioned. 1 if f ( x, y)  T
g ( x, y ) = 
0 if f ( x, y)  T
Thresholding – (cont.)
Basic Global Thresholding
• Automatic procedure:
– Select initial 𝑻.
– Segment using 𝑻 → 𝑮𝟏 and 𝑮𝟐.
– Compute means 𝒎𝟏 and 𝒎𝟐.
– Compute new threshold 𝑻 = ½(𝒎𝟏 + 𝒎𝟐).
– Repeat until the difference between successive
𝑻 is < ∆𝑻.
• Choice of 𝑻.
Thresholding – (cont.)
Basic Global Thresholding
• Thresholding viewed as decision-theory
problem aims to minimize average error
incurred in assigning pixels to two or more
groups (classes).
• Bayes decision rule.
• Requires
– PDF of intensity levels of each class.
– Probability that each class occurs in an application
Thresholding – (cont.)
Basic Global Thresholding
• Thresholding viewed as decision-theory
problem aims to minimize average error
incurred in assigning pixels to two or more
groups (classes).
• Bayes decision rule.
Not trivial
• Requires → assume the form
– PDF of intensity levels of each class.of PDF, e.g. Gaussian
– Probability that each class occurs in an application
Thresholding – (cont.)
Basic Global Thresholding
• Thresholding viewed as decision-theory
problem aims to minimize average error
incurred in assigning pixels to two or more
groups (classes). Not trivial
• Bayes decision rule. → assume the form
of PDF, e.g.
• Requires Gaussian
– PDF of intensity levels of each class.→ Still not practical
– Probability that each class occurs in an application
Thresholding – (cont.)
Optimum Global Thresholding (Otsu)
• A measure of region homogeneity is the variance
(i.e., regions with high homogeneity will have low
variance).
• Otsu’s method selects the threshold by minimizing
the within-class variance of the two groups of
pixels separated by the thresholding operator.
• Equivalently, it maximize the between-class
variance.
• Section 10.3.3 for reference and math. model.
Thresholding – (cont.)
Optimum Global Thresholding (Otsu)
• 𝝈𝟐 = 𝝈𝟐 𝒘𝒊𝒕𝒉𝒊𝒏 + 𝝈𝟐 𝒃𝒆𝒕𝒘𝒆𝒆𝒏

• Within-class variance = weighted sum of the

variances of the two classes 𝝈𝟐 𝒃𝒆𝒕𝒘𝒆𝒆𝒏
C2
C1
𝝈𝟐 𝑶 𝝈𝟐 𝑩
T
Thresholding – (cont.)
Optimum Global Thresholding (Otsu)
Minimize the within-class variance

a lot of
computations
Thresholding – (cont.)
Optimum Global Thresholding (Otsu)
Easier to minimize the between-class variance

- Usually done with a simple iterative approach.

- Other optimization methods might be used.
Check end of slides and textbook
Section 10.3.3 for more details
Note
Thresholding – (cont.)
Drawbacks of Otsu’s Method
• Assumes the histogram is bimodal.
• Fails when the two classes are very unequal (i.e.,
have very different sizes).
– In this case sigmaB may have two maxima.
– The correct maximum is not necessary the global one.
– The selected threshold should correspond to a valley of
the histogram.
• Does not work well with variable illumination.
Thresholding – (cont.)
Global Thresholding Problems
• Single value thresholding only works for
bimodal histograms.
• Images with other kinds of histograms need
more than a single threshold.
Thresholding – (cont.)
Global Thresholding Problems
• Noise

Gaussian noise
with zero mean
and different
standard
deviations
Thresholding – (cont.)
Global Thresholding Problems
• Noise
• Solutions:
- Smoothing
Thresholding – (cont.)
Global Thresholding Problems
• Non-uniform illumination or reflectance
Thresholding – (cont.)
Global Thresholding Problems
• Non-uniform illumination or reflectance

Non-uniform
illumination
can result from
multiplying by
ramp pattern.
Thresholding – (cont.)
Global Thresholding Problems
• Non-uniform illumination or reflectance
Thresholding – (cont.)
Global Thresholding Problems
• Non-uniform illumination or reflectance
• Solutions:
- Correct the non-uniform shading pattern by
its inverse (requires device).
- Correcting non-uniform pattern: background
subtraction, top-hat filtering, etc.
- Or: Variable thresholding.
Thresholding – (cont.)
Variable Thresholding
• Image Partitioning
– Subdivide image into non-overlapping
rectangles.
– Small enough so that illumination of each is
approximately uniform.
– Estimate a different threshold to segment each
subimage.
Thresholding – (cont.)
Variable Thresholding
• Image Partitioning
Iterative
global

Otsu’s
method
Otsu’s on each
method partition
Thresholding – (cont.)
Variable Thresholding
• Image Partitioning
- Further subdivisions
may be required for
better results.
Thresholding – (cont.)
Variable Thresholding
• Image Partitioning

Global Thresholding Local Thresholding

Thresholding – (cont.)
Variable Thresholding
• Local Image Properties
– Threshold according to the mean and standard
deviation of a neighborhood.
– 𝑻𝒙𝒚 = 𝒂𝝈𝒙𝒚 + 𝒃𝒎𝒙𝒚
– 𝑻𝒙𝒚 = 𝒂𝝈𝒙𝒚 + 𝒃𝒎𝑮

𝟏 𝒊𝒇 𝒇 𝒙, 𝒚 > 𝑻𝒙𝒚
– 𝒈 𝒙, 𝒚 = ൝
𝟎 𝒊𝒇 𝒇 𝒙, 𝒚 ≤ 𝑻𝒙𝒚
Thresholding – (cont.)
Variable Thresholding
• Local Image Properties
Thresholding – (cont.)
Variable Thresholding
• Moving Average
– Scanning is carried out line by line in zigzag pattern to reduce
illumination bias.
1 𝑘+1
– 𝑚(𝑘 + 1) = σ𝑖=𝑘+2−𝑛 𝑧𝑖
𝑛
– 𝑇𝑥𝑦 = 𝑏𝑚𝑥𝑦

1 𝑖𝑓 𝑓 𝑥, 𝑦 > 𝑇𝑥𝑦
– 𝑔 𝑥, 𝑦 = ൝
0 𝑖𝑓 𝑓 𝑥, 𝑦 ≤ 𝑇𝑥𝑦
– Works well when the objects of interest are small (or thin) with
respect to the image size, a condition satisfied in images of
handwritten text.
Thresholding – (cont.)
Variable Thresholding
• Moving Average
Next Lecture
Region-based Segmentation + Morphology

Assignment
Check book sections and associated problems
Chapter 10 1, 2, 2.1, 2.2, 2.3, 2.4, 2.5, 2.7(Hough only)

Associated 2, 3, 4, 5, 6, 7, 12, 23, 26, 27, 28, 29, 36, 39

problems

11/7/2022 96
References
- Gonzalez and Woods, Digital Image Processing.
- Richard Szeliski, Computer Vision Algorithms and
Applications, Springer 2010. (Chapter 5)
http://szeliski.org/Book/
- Peters, Richard Alan, II, “Image Segmentation
(PointLineEdge)", Lectures on Image Processing, Vanderbilt
University, Nashville, TN, April 2008, Available on the web at
the Internet Archive,
http://www.archive.org/details/Lectures_on_Image_Processing.
-http://opencv-python-
tutroals.readthedocs.org/en/latest/py_tutorials/py_imgproc/py_houghlines/py_houg
hlines.html
11/7/2022 97