Modul 3

Image Segmentation
 Image Segmentation
Objectives
• To separate different objects of interest from
other regions e.g from the background.
• First step towards image analysis (i.e. extracting
information from an image and also for pattern
recognition.
 Image Segmentation
• Example – segmenting circular objects only

Befor Afte
e r
 Image Segmentation
Basic approach of segmentation
1. Based on discontinuity of gray-level values –
based on abrupt changes in intensity such as
edges.
2. Based on similarity of gray-level values – based
on partitioning an image into regions similar in
accordance to a set of predefined criteria. This
includes thresholding, region growing, and region
splitting and merging.
 Detection of Discontinuities
• 3 types of discontinuities
– Points
– Lines
– Edges
• Most common is to use masks, window or
template. Normally of size 3x3.
• Similar to convolution in process.

5
 Detection of Discontinuities
w1 w2 w3

w4 w5 w6
w7 w8 w9

The response of the mask will be

R = w1z1 + w2z2 + … + w9z9

where zk is the gray level of the pixel associated with mask coefficient wk.
The response of the mask is defined with respect to its center location.

6
 Point Detection
• Very straight forward.
• If a pixel centered by the mask has response |R| ≥
T – where T is a nonnegative threshold value,
then a point is detected.

-1 -1 -1
A mask used for detecting
-1 8 -1 isolated points different
-1 -1 -1 from a constant background

7
Point Detection

Point detected

8
 Line Detection
• To detect a line (one pixel thick) oriented
horizontally, the following mask should be used
-1 -1 -1

2 2 2
-1 -1 -1

• Maximum response when the line passed through

the middle row of the mask.
9
 Line Detection
Example
When the mask is point
0 1 0 1 1
centered at
1 1 2 0 R = 10 + 10 +10 - (1 +1 + 2)
1 2
5 5 5 5 - (1 + 1 + 0)
= 30 - 4 -2 = 24
1 1 0 0 2
when at
0 0 0 1 R = 30 - (1+2+0)-(1+0+0)
= 26
10
 Line Detection
Other orientation of line
-1 -1 2 -1 2 -1 Vertica
-1 2 -1 -1 2 -1 l
(90°)
2 -1 -1 -1 2 -1

+ 45° 2 -1 -1

- 45° -1 2 -1

-1 -1 2
11
 Line Detection
• In normal practise
– Let R1, R2, R3, & R4 be the response to the line at 0°,
45°, 90° and -45°.
– all these masks are run through the image.
– whichever Ri i = 1,2,3,4, gives the maximum indicates
that a line associated with that angle has been
detected.

12
Line Detection

13
 Edge Detection
• Most common approach for detecting meaningful
discontinuities in gray level.
• An edge is a set of connected pixels that lie on the
boundary between two regions
• Fundamentally an edge is a ‘local’ concept
whereas boundary is a more global idea.
• An ideal edge sharp transition in gray levels.

14
 Image Segmentation
Ideal
Edges

Horizontal
profile
 Edge Detection
• In practice, optics, sampling, and other image
acquisition imperfections yield edges that are
blurred.
• The degree of blurring is determined of the quality
of the image acquisition system, the sampling
rate, and the illumination under which the image
is acquired.
• As a result, edges are more closely modeled as
having a ‘ramplike’ profile

16
 Image Segmentation
Practical
Edges

Horizontal
profile
 Edge Detection

• The basic idea underlying most edge-detection

techniques is the computation of a local derivative
operator.

18
 Edge Detection

Edge detection by Horizontal

derivative operators: profile
(a) Light stripe on a dark
background; 1st derivative
(b) Dark stripe on a light
background. Note that
the second derivative
has a zero crossing at
the location of each 2nd derivative
edge.
19
 Edge Detection
• From the first derivative of (a)
– Positive at leading edge transition.
– Negative at trailing edge transition.
– Zero at constant level.
• From the second derivative of (a)
– positive for transition in the dark side
– negative for transition in the bright side
– zeros in constant area

20
 Edge Detection
• Contain zero crossing at the midpoint of transition
in gray level. Location is at the edge. Useful in
detecting the location of edge.
• Summary
– Magnitude of the 1st derivative - detects the presence
of edge.
– Sign of the 2nd derivative - indicates whether the pixel
belongs to dark area or bright area.

21
 Edge Detection
– In case of a noise free edge, 1st and 2nd derivatives give
good and clear result.
– In presence of noise we will not get as good as
expected when using the 1st derivative.
– The result is even worst when using 2nd derivative.

22
 Edge Detection

Noise free image 1st derivative image 2nd derivative image

23
 Edge Detection

Noise image with 1st derivative image 2nd derivative image

gaussian noise
σ = 0.001

24
 Edge Detection

Noise image with 1st derivative image 2nd derivative image

gaussian noise
σ = 0.01

25
 Edge Detection

Noise image with 1st derivative image 2nd derivative image

gaussian noise
σ = 0.1

26
 Edge Detection
Gradient Operator
• Used in taking the 1st derivative.
• The gradient of an image f (x,y) is given by the vector

• From vector analysis, the gradient vector points in the

direction of a maximum rate of change of f at
coordinates (x,y).

27
 Edge Detection
• In edge detection an important quantity is the
magnitude of this vector, simply referred to as the
gradient and denoted as ∇f. i.e.
∇f = mag(∇ f ) = [Gx2 + Gy2]½ ①
• In practise, approximation is adequate
∇f ≈ |Gx| + |Gy| ②
• Eq. ② is simpler to implement in dedicated
hardware.

28
 Edge Detection
• Another important quantity is the gradient
direction and given by
α(x,y) = tan-1(Gy/Gx)
the angle is measured with respect to the x-axis.
• The direction of an edge is perpendicular to the
direction of the gradient vector at that point.

29
 Edge Detection
• How to approximate ① and ② in digital image?
• Consider first an image region below.
• There are many ways to approximate gradient at
point z5

z1 z2 z3 x

z4 z5 z6

z7 z8 z9 y
30
 Edge Detection
Simplest way
z5 – z8 y direction
z5 – z6 x direction
thus
∇f ≈ [(z5 – z8)2 + (z5 – z6)2]½
or
∇f ≈ |(z5 – z8)| + |(z5 – z6)|

31
 Edge Detection
• Another way - use cross-differences
∇f ≈ [(z9 – z5)2 + (z8 – z6)2]½
or
∇f ≈ |(z9 – z5)| + |(z8 – z6)|
in masks form we have
-1 0 0 -1
0 1 1 0

These are called as the Roberts operator

32
 Edge Detection
• However, the above mask is awkward to
implement and sensitive to noise.

33
 Edge Detection
3 x 3 operators
• Extending the above approximation using 3x3 window

∇f ≈ [{(z7+z8+z9) – (z1+z2+z3)}2 +
{(z3+z6+z9) – (z1+z4+z7)}2]½ z1 z2 z3
or
z4 z5 z6
∇f ≈ |(z7+z8+z9) - (z1+z2+z3)| +
z7 z8 z9
|(z3+z6+z9) - (z1+z4+z7)|

34
 Edge Detection
Thus
Gx = (z3+z6+z9) – (z1+z4+z7)
Gy = (z7+z8+z9) – (z1+z2+z3)
in mask form (Prewitt Operator)
-1 -1 -1 -1 0 1

1/3 0 0 0 1/3 -1 0 1
1 1 1 -1 0 1

Gy Gx
35
 Edge Detection
• Another most common edge operator is called the
Sobel operator with the following masks:

-1 -2 -1 -1 0 1

¼ 0 0 0 ¼ -2 0 2
1 2 1 -1 0 1

Gy Gx
36
 Edge Detection
• This operator has the smoothing effect which
reduce the noise.
• Thus
Gx = {(z3 + 2z6 + z9) - (z1 + 2z4 + z7)}
Gy = {(z7 + 2z8 + z9) - (z1 + 2z2 + z3)}

37
 Edge Detection
• Example of Edge Response
– Suppose the image region is as follow
i
0 1 2 3 4
j
0 0 0 0 0 5

1 0 0 0 5 5

2 0 0 5 5 5

3 0 5 5 5 5
4 5 5 5 5 5
38
 Edge Detection
– Applying the Sobel operator for i,j = 1 to 3.
– skip rows and columns 0 and 4 because the mask
cannot cover properly
0 0 0 0 0

0 7 21 21 0
Results obtained
using Eq. ①
0 21 21 7 0

0 21 7 0 0
0 0 0 0 0
39
 Edge Detection

0 0 0 0 0

0 10 30 30 0 Results obtained
0 30 30 10 0 using Eq. ②
0 30 10 0 0
0 0 0 0 0

40
Edge Detection

41
 Edge Detection
• In general the image is first smoothed as to
average out the effect of noise.
• A simple lowpass filter is normally used.
• However, the effect will be slightly loss on the
details of the edges

42
Edge Detection

43
 Edge Detection
• It is also possible to modify the 3x3 masks so that
they have their strongest response along the
diagonal directions.
• Example for Prewitt.

0 1 1 -1 -1 0

Gy -1 0 1 Gx -1 0 1
-1 -1 0 0 1 1

44
 Edge Detection
• Example for Sobel.

0 1 2 -2 -1 0

-1 0 1 -1 0 1
-2 -1 0 0 1 2

Gy Gx

45
Edge Detection

46
 Edge Detection
Laplacian
• The 2nd order derivative can be carried out by the
Laplacian of a 2-D function i.e.

• The approximation in digital image is given by

∇2f = 4z5 - (z2 + z4 + z6 + z8)

47
 Edge Detection
Basic requirements
i. center pixel must be positive while outer pixels
must be negative.
ii. sums of coefficients must be zero because
Laplacian is a derivative.
– mask of Laplacian for 4-connectivity
0 -1 0
-1 4 -1
0 -1 0
48
 Edge Detection
• Laplacian can also be approximated using the
diagonal neighbors.
∇2f = 8z5 - (z1 + z2 + z3 + z4 + z6 + z7 + z8 + z9)
– mask of Laplacian for 8-connectivity

-1 -1 -1

-1 8 -1
-1 -1 -1

49
 Edge Detection
• However, Laplacian is seldomly used in its general
form for edge detection because of its high
sensitivity to noise.
• This is because Laplacian is based on a
second-order derivatives.
• In addition its magnitude component produces
double edges which complicates segmentation
process.
• It also unable to detect edge direction.

50
 Edge Detection
• In segmentation, Laplacian is normally used to
– detect location of edges from its zero-crossing
property, and
– to determine whether a pixels is on the dark or light
side of an edge.
• In the 1st category, the Laplacian is combined with
smoothing as a precursor to finding edges via
zero-crossings.

51
 Edge Detection
• Consider the following function

where r2 = x2 + y2 and σ is the standard dev.

• Convolving this function with an image blurs the
image, with degree of blurring determined by σ .

52
 Edge Detection
• The Laplacian of h (2nd derivative of h w.r.t. r) is

• This function is commonly known as the Laplacian of

Gaussian (LoG) since the h is in the form of Gaussian.
• It is also sometimes called the Mexican hat function
because of its shape.

53
 Edge Detection
3-D plot of
LoG
Black = -
White = +
Gray = 0

a b
c d
Cross section 0 0 -1 0 0
showing zero
0 -1 -2 -1 0
crossings.
-1 -2 16 -2 -1
0 -1 -2 -1 0
0 0 -1 0 0
5 x 5 mask approximation to the
shape in (a)
54
 Edge Detection
• The 5x5 mask is an approximation and is not unique.
• Center is positive and surrounded by negative
coefficients values which sum up to zero. This is to
ensure that constant areas will have zero response
from this mask.
• For this small size mask, only images that are
essentially noise free will produce desirable outcomes.
• Same effect can be achieved by convolving an image
with the Gaussian smoothing function and then
compute the Laplacian.

55
 Edge Detection
Original Edge image –
image – using Sobel
angiogram

Gaussian -1 -1 -1
smoothing Laplacian
filter -1 8 -1 mask
-1 -1 -1

LoG image LoG image – thresholded Zero crossings 56

 Edge Detection
• Other types of Edge Detectors
• Kirch detector - corresponds to 45° and -45° edge.
• Frei&Chen - orthogonal mask.
• Canny - mathematical-based operator.
• Hexagonal-based - adopted from hexagonal
arrangement.

57
 Edge Linking
• In practice, could not get clear edges. Exist some
breaks in the boundary from non-uniform
illuminations.
• need some algorithm for linking and other
boundary detection procedures.
• Two types
– local processing (using mask)
– global processing (entire image)

58
 Thresholding
• Simplest form of segmentation.
• Usually done at histogram level.
• suppose the following histogram f(x,y)
corresponds to lights objects on a dark
background.
 Thresholding

T
• One obvious way to extract the objects from the
background is to select a threshold T that could
separate the two modes.
 Thresholding
• Thresholding normally results in binary image.
• Mathematically, the operation can be expressed as

• Any point for which f(x,y) > T is called an object point

(white). Otherwise the point is called a background
point (black).
• Since T is applied to the whole image, it is called a
global threshold.
 Thresholding

T T
• Suppose if we have the following
1 2 histogram for f(x,y).
• Now we have 3 dominant modes (two light objects on
a dark background).
• Involves multilevel thresholding (not a binarized
image)
 Thresholding
• Need to classify a point to one object class if T1 <
f(x,y) ≤ T2, to the other object class if f(x,y) > T2,
and to the background if f(x,y) ≤ T1.
 Thresholding
• In general, segmentation problems requiring multiple
thresholds are best solved using region growing
method as will be discussed later.
• From the discussion, thresholding can be viewed as an
operation that involves tests against a function T of
the following form
T = T[x,y,p(x,y),f(x,y)]
where f(x,y) is the gray level of point (x,y) and p(x,y)
denotes some local property of this point – for
example, the average gray level of a neighborhood
centered on (x,y).

64
 Thresholding
• If T depends on some small neighborhood, i.e T
depends on both f(x,y) and p(x,y) it is called a local
threshold.
• If in addition, T depends on the spatial
coordinates x and y, the threshold is called
dynamic or adaptive.

65
 Thresholding
Basic Global Thresholding
• Global threshold works fine in highly controlled
environment like in industrial inspection
applications where illumination can be controlled.
• Global threshold can be obtained manually i.e.
visual inspection of the histogram, or
• Obtained automatically as follows:

66
 Thresholding
1. Select an initial estimate of T
2. Segment the image using T. This will produce two groups
of pixels: G1 consisting of all pixels with values ≤ T.
3. Compute the average gray level values μ1 and μ2 for the
pixels in regions G1 and G2.
4. Compute a new threshold value:

5. Repeat steps 2 through 4 until the difference in T

successive iterations is smaller than a predefined
parameter T0.

67
 Thresholding
How to choose initial threshold?
• If the background and the object occupy
comparables areas in the image, set T as an
average gray level of the image.
• If the object is much smaller than the background
(or vice-versa) set T as mid-value between the
maximum and minimum gray levels.
• T0 is used to stop the algorithm and is useful when
speed of iteration is an important issue.
 Thresholding
Original image

• Example of global threshold

Global threshold image

Histogram of the original image T = 128

69
 Thresholding
• Example of global threshold
T = 90

70
 Adaptive Thresholding
• Uneven illumination causes difficulty in partitioning
histogram so that a global threshold is achieved.
• One approach to handle this situation is to divide
original image into subimages and then utilize
different a different threshold to segment each image.
• Key Issues:
– How to subdivide the image?
– How to estimate the threshold for each subimage?

71
 Adaptive Thresholding
a b
Example (from Gonzalez c d
et.al 2002)
– Consider the following
image which cannot be
thresholded using single
global threshold (a).
– Figure (b) shows the result
of a single threshold.
– Figure (c) shows the
partition into subimages

72
 Adaptive Thresholding
Example (from Gonzalez et.al 2002)
– All the subimages that did not contain a boundary between
object and background had variances of less than 75. All
subimages containing boundaries had variances in excess of 100.
– Each subimage with variance greater than 100 was segmented
using a threshold computed for that subimage using the
previous algorithm.
– The initial value for T in each case was selected as the point
midway between the minimum and maximum gray levels in the
subimage.
– All subimages with variance less than 100 were treated as one
composite image which was segmented using a single threshold
estimated using the same algorithm.

73
 Adaptive Thresholding
Example (from Gonzalez et.al 2002)
– The result of segmentation is shown in (d).
– Except the two subimages, the improvement obtained
in (d) over (b) is evident.
– The boundary between object and background in this
two subimages was small and dark. Thus the resulting
histogram was almost unimodal.

74
 Adaptive Thresholding
Example (from Gonzalez
et.al 2002)
– (a) shows the top two
subimages.

b
a
c
e d f
75
 Adaptive Thresholding
Example (from Gonzalez et.al 2002)
– The histogram for the top image (b) clearly bimodal with
well-defined peaks and valley but the improper segmented
subimage has histogram which is almost unimodal with no
clear distinction between object and background.
– (d) shows how the subimage is further subdivided into
much smaller subimages.
– (e) shows the histograms of the top left small subimage.
This subimage contains the transition between object and
background and has clearly bimodal histogram and
therefore should be segmentable
– (f) shows the result of the segmentation

76
 Optimal Threshold
Obtaining Optimal Threshold
• How to choose threshold?
• Suppose an image contains only 2 principal gray-level regions and
denote this gray values as z.
• z can be viewed as random quantities and the histogram then
can be considered as an estimate of the probability density
function p(z).
• The overall density function is the sum or mixture of two
densities, one for the light and the other for the dark regions.
• If the form of densities is known or assumed, optimal threshold
(in terms of minimum error) separating the two regions can be
determined.

77
 Optimal Threshold
• Consider the following PDF shown below

• Assume the larger PDF corresponds to the background

while the smaller one corresponds to the gray levels of
objects.
• The mixture PDF describing the overall gray-level variation
is.
78
 Optimal Threshold

• P1 and P2 are the probabilities of occurrence of

the two classes of pixels. That is, P1 is the
probability that a random pixel with value z is an
object pixel and similarly P2 is the probability that
the pixel is a background pixel.
• These probabilities must satisfy the following
constraint
P1 + P2 = 1.

79
 Optimal Threshold
• Recall that an image is segmented (in this case) by
classifying as background all pixels with gray levels greater
than a threshold T. All other pixels are called object pixels.
• Main objective then is to select the value of T that
minimizes the average error in making decisions that a
given pixel belongs to an object of to the background.
• Recall that the probability of a random variable having a
value in the interval [a,b] is the integral of its probability
density function from a to b, which is the area under the
PDF curve between these two limits.

80
 Optimal Threshold
• Hence, the probability of erroneously classifying a
background point as an object point is

• This is the area under the curve of p2(z) to the left of the
threshold.
• Similarly, the probability of erroneously classifying an
object point as a background point is

• This is the area under the curve of p1(z) to the right of T.

81
 Optimal Threshold
• Thus, the overall error
E(T) = P2E1(T) + P1E2(T)
• Note:
– Quantities E1 and E2 are weighted by the probability of
occurrence of object or background pixels.
– Subscripts are opposite

82
 Optimal Threshold
• Explanation
– Consider the extreme case in which background points
are known never to occur. In this case P2=0. The
contribution to the overall error (E) of classifying a
background point as an object point (E1) should be
zeroed out because background points are known
never to occur. i.e. P2 E1 = 0
– If background and object are equally likely to occur,
then the weights are P1= P2 = 0.5

83
 Optimal Threshold
• To find T that will minimize the error need to
differentiate E(T) with respect to T (using
Leibnitz’s rule) and set the result equal to 0. Thus
we have
P1p1(z) = P2p2(z)
• Solving the above equation will give the optimum
T.
• Note: if P1 = P2, then the optimum T is where the
curves for p1 (z) and p2 (z) intersect.

84
 Optimal Threshold
• To analytically solve T, p1(z) and p2(z), the two
PDFs must be known.
• Estimating these densities are normally not
feasible.
• Instead use densities whose parameters are
reasonably simple to obtain.
• One of the principal densities used is the Gaussian
densities

85
 Optimal Threshold
then
p(z) = P1p1(z) + P2p2(z)

where
– μ1 & μ2 - mean values of the 2 brightness level
– σ1 & σ2 - standard deviations about the mean

86
 Optimal Threshold
• Applying this to the Gaussian density, taking
logarithms, and simplifying, gives the quadratic
equation
AT2 + BT + C = 0
where

87
 Optimal Threshold
• Because of quadratic terms, 2 solutions are
possible
• If the variances are equal however, then

• If the prior probability are equal, P1 = P2 or σ= 0,

then optimal threshold is given by

88
 Optimal Threshold
• i.e. T is equal to the average of the mean.
• To enhance T further, we need to adjust p(z).
Hence we need to establish some error criterion.
Typically

Adjust this iteratively

but computationally
expensive
h(z) is the actual histogram

89
 Boundary Characteristic
Threshold
• Based on using gradient & Laplacian.
• Idea - to consider pixel values on and near
boundary location.
• The probability between dark region & white
region are similar.
• The more the valley is pronounced the easier T
can be chosen.

90
 Boundary Characteristic
Threshold
• Thus used

0,+, & -, represent 3 distinct levels

T is a threshold (for an edge)
∇f and ∇2f are computed are every point
(x,y)

91
 Boundary Characteristic
Threshold

• From previous result for Laplacian, near the edge

dark region will have + values bright region will
have – values
• Thus
▪ + indicates pixels on the dark side of edge
▪ – indicates pixels on the bright side of edge

92
 Boundary Characteristic
Threshold
• This information can then be used to generate a
segmented, binary image where 1’s correspond to
object of interest and 0’s correspond to background.
• To generate this, notice
▪ transition from a light background to dark object is
characterized by – followed by +
▪ interior will be composed either by 0 or +
▪ finally, transition from object back to background is
characterized by + followed by –.

93
 Boundary Characteristic
Threshold
• In a sequence form, we could have
(…)(–,+)(0 or +)(+, –)(…)
(…) - represents combination of +, –, and 0
• Set the combination between (–,+) and (+, –) as 1,
while the rest to 0 → binary image

94
 Boundary Characteristic
Threshold
• Example

Original image

Segmented image

95
 Region Based Segmentation
• Let R represent the entire image region.
• Partition R into n subregions, R1, R2, …, Rn.
• Thus
(a)

(b) Ri = is a connected region, i = 1,2,…,n

(c) for all i and j, i ≠ j

96
Region Based Segmentation
(d) P(Ri) = TRUE for i=1,2,…,n
- P(Ri) is a logical predicate over the point in set Ri
(e) FALSE for i≠j
• Indicates that region Ri and Rj are different in the sense of
predicate P. Example gray level values

97
 Region Based Segmentation
Region Growing by Aggregation
• Start with a “seed” point.
• Then append to each seed point those
neighboring pixels that have similar properties
(such as gray level, texture, color).

98
 Region Based Segmentation
Example 1

(a) (b)

(a) Original image array

(b) segmentation result using
an absolute difference of less
than 3 between intensity levels
(c) result using an absolute
difference of less than 8 (c)
99
 Region Based Segmentation
Example 2

100
 Region Based Segmentation
• 3 main problems with this simple technique
(i) selecting initial seeds that properly represent regions
of interest.
(ii) selecting suitable properties for including points in
various regions during the growing process.
(iii) stopping rule - when to stop growing

101
 Region Splitting & Merging
• Subdivide an image into a set of arbitrary,
disjointed regions.
• Merge and/or split the regions by using basic
formulation given.
• One example of the algorithm
– Subdivide R into smaller & smaller quadrant regions so
that for any region Ri, P(Ri) = TRUE
– That is, if P(R)=FALSE, subdivide the quadrant into
subquadrant.

102
 Region Splitting & Merging
– This is called quadtree splitting technique

R2 R
R1
R41 R42 R1 R4
R3
R43 R44 R2 R3
Partitioned image

R41 R42 R43 R44

Corresponding quadtree
103
 Region Splitting & Merging
– To allow for merging, check
– P(Rj∪Ri) = TRUE or not
– if TRUE merge Rj and Ri i
• Example

(a (b (c (d
) ) ) )
104
End of Lecture

105