Image

Video
Processing
Laboratory

CH10

Image Segmentation

1
 Image
Basics of image segmentation Video
Processing
Laboratory

• Input: image => Output: attributes

• Segmentation of nontrivial image is one of the most
difficult tasks in image processing

Ex. Separate the sky, clouds, and

mountains

IVP Lab., CSIE@Tunghai University 2

 Image
Two approaches Video
Processing
Laboratory

™ Two basic properties for monochrome image

segmentation:
• discontinuity- partition an image based on abrupt
changes in gray level, the principal areas are detection
of isolated points, detection of lines and edges in an
image.
• similarity- thresholding, region growing, and region
spliting and merging.

IVP Lab., CSIE@Tunghai University 3

 Image
Detection of discontinuity Video
Processing
Laboratory

• Three basic types of gray-level discontinuity

• Points, lines, and edges
• Method: run a mask through the image

Input: 3x3 region z1, z2 , z3 , … , z9

Output: z5 -> R
9
R = ∑ wi zi
i =1

3x3 mask

IVP Lab., CSIE@Tunghai University 4

 Image
Point detection Video
Processing
Laboratory
• Point detection mask
1. Mask operation over whole image
2. Threshold: |R| ≥ T
™ The idea is that the gray level of an isolated point
will be quite different from the gray level of its
neighbors.

渦輪葉片 X 光 pointIVPmask
Lab., CSIE@Tunghai University
thresholding 5
 Image
Line detection Video
Processing
Laboratory

1. Line detection mask

2. Which orientation?
A thin line: 0 0 0 Hori. 45o Veri. -45o
121 R1=8 R2=2 R3=2 R4=2
000

Thresholding: |Ri| ≥ |Rj| ≥ T

IVP Lab., CSIE@Tunghai University 6
 Image
Example: single-line detection Video
Processing
Laboratory

Input: binary image absolute value

after –45o mask |R| ≥ T

1 Detect line that are one pixel thick

2 Isolated “line points”

IVP Lab., CSIE@Tunghai University 7

 Image
Definition of edges? Video
Processing
Laboratory

• Intuition of edge: set of connected pixels that line one

the boundary between two regions
• Model of digital edge

IVP Lab., CSIE@Tunghai University 8

 Image
Video
Processing
Laboratory
Two factors affects acquired edges
• Blur 模糊化
• Sampling rate, illumination condition

zoom

• Noise 雜訊

IVP Lab., CSIE@Tunghai University 9

 Image
Video
Processing
Method of edge detection: derivatives Laboratory

Edge detection
method

Magnitude of 1st
derivatives ≥T

1. Sign determines
Dark or light region

2. Zero-crossing
IVP Lab., CSIE@Tunghai University 10
 Image
Methods: 1st derivatives Video
Processing
Laboratory

• 1st derivatives
⎡ ∂f ⎤
⎢ ∂x ⎥
∇f = ⎢ ∂f ⎥
2-D gradient ⎢ ⎥
⎢⎣ ∂y ⎥⎦
0 .5
⎡ ∂f 2 ∂f 2 ⎤
Magnitude = ⎢ ( ) + ( ) ⎥
⎣ ∂x ∂y ⎦
∂f ∂f
≈ +
∂x ∂y

∂f / ∂y
−1
Direction = tan ( ) ∂f ∂f
∂f / ∂x
∂x ∂y
IVP Lab., CSIE@Tunghai University 11
 Image
Example: Sobel filter Video
Processing
Laboratory

Original: 1200x1600

Hor.
abs

abs
+

Ver. IVP Lab., CSIE@Tunghai University 12

 Image
Effects of noise Video
Processing
Laboratory

Unwanted
Detailed
Structure !

IVP Lab., CSIE@Tunghai University 13

 Image
Video
Processing
1st 2nd Laboratory

Noisy
edges Fail to detect

IVP Lab., CSIE@Tunghai University 14

 Image
Reduce noise: Smoothing Video
Processing
Laboratory

Smooth with 5x5 average filter

Hor.
abs

abs
+

Ver. IVP Lab., CSIE@Tunghai University 15

 Image
Method: 2nd derivative Video
Processing
Laboratory

Edge detection
method 2nd derivative mask

Magnitude of 1st
derivatives ≥T

1. Sign determines
Dark or light region

2. Zero-crossing
IVP Lab., CSIE@Tunghai University 16
 Image
Video
Processing
Laboratory
Edge Detection (cont.)

™ Laplacian-
• The Laplacian of a 2-D function f(x, y) is a second derivative defined
as 2
∂ f ∂ f
2
∇2 f = + 2
∂x 2
∂y
• The spatial mask for Laplacian is
∇ 2 f = 4z5 - (z2 + z4 + z6 + z8).

0 -1 0
-1 4 -1
0 -1 0

IVP Lab., CSIE@Tunghai University 17

 Image
Video
Processing
Laboratory
Edge Detection (cont.)

• The Laplacian is unsuitable for edge detection because

‘sensitive to noise
’produce double edges
“unable to detect edge direction
• The Laplacian is usually used to detect whether a pixel is on the
dark or light side of an image.
• A more general use of Laplacian is in finding the location of edges
using its zero-crossing property by convolving an image with the
Laplacian of a 2-D gaussian function
⎛ x2 + y2 ⎞
h ( x , y ) = exp ⎜ − ⎟
⎝ 2σ ⎠
2

• where σ is the standard deviation.

IVP Lab., CSIE@Tunghai University 18

 Image
Video
Processing
Laboratory

IVP Lab., CSIE@Tunghai University 19

 Image
Video
Processing
Laboratory
Edge Detection (cont.)

• Let r2 = x2 + y2, the Laplacian of h is

⎛ r2 − σ 2 ⎞ ⎛ r2 ⎞
∇ h=⎜
2
⎟ exp⎜ − 2⎟
⎝ σ 2
⎠ ⎝ 2σ ⎠
• See Fig. 10.14 for the shape of a Laplacian function
(classical Mexican hat shape).
• See Fig. 10.15 for illustartion.
• Note edge detection by gradient operations work well
in cases involving images with sharp intensity
transition and relatively low noise; while zero
crossings offer an alternative in cases when edges are
blurry or when a high noise content is present.

IVP Lab., CSIE@Tunghai University 20

 Image
Video
Processing
Chapter 10 Laboratory

Image Segmentation

IVP Lab., CSIE@Tunghai University 21

 Image
Video
Processing
Chapter 10 Laboratory

Image Segmentation

IVP Lab., CSIE@Tunghai University 22

 Image
Video
Processing
Laboratory
Edge Linking and Boundary Detection

™ Objective- to assemble edge pixels, which is yielded

by edge detection algorithms and seldom characterize
a boundary completely because of noise, breaks in the
boundary from nonuniform illumination and other
effects that introduce spurious intensity discontinuities,
into meaningful boundaries.
™ Approaches-
)local processing
)global processing via the Hough Transform

IVP Lab., CSIE@Tunghai University 23

 Image
Motivation for edge linking Video
Processing
Laboratory

• Edge detection: find pixels lying on edges

=> Link edge points

Sobel: horizontal

Sobel: vertical
original
Goal:找出車牌方塊

IVP Lab., CSIE@Tunghai University 24

 Image
Edge linking methods Video
Processing
Laboratory

• Local processing
• Analyze pixels in a small region (3x3 or 5x5)
• Gradient magnitude:
∇ f ( x , y ) − ∇ f ( x0 , y 0 ) ≤E
• Gradient direction:
α ( x , y ) − α ( x0 , y 0 ) ≤ A
• Global processing
• Given n (edge) points in an image
• Find all possible lines by every pairs of points
• For each line, find subsets of points that close to it

IVP Lab., CSIE@Tunghai University 25

 Image
Example: Local processing Video
Processing
Laboratory

Sobel: vertical

Vertical scan

Criterion:
1. Grad. mag. > 25
2. Grad. Direction
difference < 15o

Horizontal scan

Sobel: horizontal
IVP Lab., CSIE@Tunghai University 26
 Image
Video
Processing
Laboratory
Global Processing via the Hough
Transform

™ Useful for detecting and allocating curve of specified shape (lines,

circles, and ellipses, etc.).
™ Advantages of Hough Transform-
• immune to noise
• tolerating gaps
• applicable to partial shapes
™ However, the shape must be analytic, i.e., the shape must be
describable using an equation.

IVP Lab., CSIE@Tunghai University 27

 Image
Video
Processing
7.2.2 Global Processing via the Hough Laboratory
Transform

™ Idea: (點共線 線共點)

• Consider a point (xi, yi) and the general equation of a
straight line in slope-intercept form, yi = axi + b.

• Rewriting the above equation as b = -xia + yi and

considering the ab plane (called parameter space)
yields the equation of a single line for a fixed pair (xi, yi).

• Furthermore, a second point (xj, yj) also has a line in

parameter space associated with (xj, yj) , this line will
intersect the line associated with (xi, yi) at (a’, b’), where
a’ is the slope and b’ the intercept of the line containing
(xi, yi) and (xj, yj) in the xy plane.

IVP Lab., CSIE@Tunghai University 28

 Image
Video
Processing
Laboratory

IVP Lab., CSIE@Tunghai University 29

 Image
Video
Processing
Laboratory
7.2.2 Global Processing via the Hough Transform
™ Implementation (accumulator cells):
• The parameter space is subdivided into accumulator cells (see Fig.
10.18), where (amin, amax) and (bmin, bmax) are the expected ranges
of slope and intercept values.
• For each point (xk, yk) in the image plane, let a equal each possible
value ai in the range (amin, amax) and compute its corresponding bj
using the equation b = -xka + yk, increment the accumulator cell
count A(i, j).
• A value of M in A(i, j) corresponds to M points in the xy plane lying
on the line y = aix + bj.
• If there are n image points and the a axis is subdivided into K
values, the method involves nK computations.

IVP Lab., CSIE@Tunghai University 30

 Image
Video
Processing
Laboratory

IVP Lab., CSIE@Tunghai University 31

 Image
Video
Processing
Laboratory
7.2.2 Global Processing via the Hough Transform
™ Drawback of the slope-intercept form, e.g., y = ax + b,
is that both the slope and intercept approach infinity
as the line approaches the vertical.
™ To solve the above problem is to use normal
representation:
x cosθ + y sinθ =ρ. (see Fig. 10.19)
™ The range of angle θ is [-90o, 90o], measured with
respect to x axis.
Ï horizontal line: θ = 0o and ρ = x intercept.
vertical line: θ = 90o and ρ = positive y intercept
or θ = -90o and ρ = negative y intercept

IVP Lab., CSIE@Tunghai University 32

 Image
Video
Processing
Laboratory

IVP Lab., CSIE@Tunghai University 33

 Image
Video
Processing
Chapter 10 Laboratory

Image Segmentation

IVP Lab., CSIE@Tunghai University 34

 Image
Video
Processing
Chapter 10 Laboratory

Image Segmentation

IVP Lab., CSIE@Tunghai University 35

 Image
Video
Processing
Graph -Theoretic Technique
Graph-Theoretic Laboratory

IVP Lab., CSIE@Tunghai University 36

 Image
Video
Processing
Laboratory

IVP Lab., CSIE@Tunghai University 37

 Image
Video
Processing
Laboratory

IVP Lab., CSIE@Tunghai University 38

 Image
Video
Processing
Chapter 10 Laboratory

Image Segmentation

IVP Lab., CSIE@Tunghai University 39

 Image
Foundation of thresholding Video
Processing
Laboratory

• Idea: object and background pixels have gray levels

grouped into two dominant modes

Original image histogram

IVP Lab., CSIE@Tunghai University 40
 Image
Foundation of thresholding Video
Processing
Laboratory

• Input f(x,y), given threshold T

⎧1 if f ( x, y ) > T
g ( x, y ) = ⎨
⎩0 if f ( x, y ) ≤ T

IVP Lab., CSIE@Tunghai University 41

 Image
Issues of thresholding Video
Processing
Laboratory

• Selection of threshold T ?
• Complex environment – illumination
• Multiple thresholds – more than one object
• Global threshold
• Local threshold

Thresholding as a multi-variable function:

g(x,y) = T[ f(x,y), x, y, p(x,y) ]

Adaptive: Depend on Local: local property func.

position

IVP Lab., CSIE@Tunghai University 42

 Image
Video
Processing
Thresholding Laboratory

IVP Lab., CSIE@Tunghai University 43

 Image
Video
Processing
Chapter 10 Laboratory

Image Segmentation

IVP Lab., CSIE@Tunghai University 44

 Image
Video
Processing
Chapter 10 Laboratory

Image Segmentation

IVP Lab., CSIE@Tunghai University 45

 Image
1. Automatic selection of T Video
Processing
Laboratory

1. Select an initial T
G1 G2
• Average gray level
• Mean of max. and min. gray level

2. Segment the image using T

T
μ1 μ2
3. Calculate mean of G1 and G2
T2
4. New threshold: T2 = 0.5(μ1 + μ2)
5. Repeat steps 2~4 until difference in
successive T is small
IVP Lab., CSIE@Tunghai University 46
 Image
Example: automatically select T Video
Processing
Laboratory

Initial: gray level mean

3 iterations
T = 125.4

fingerprint

IVP Lab., CSIE@Tunghai University 47

 Image
Video
Processing
Chapter 10 Laboratory

Image Segmentation

IVP Lab., CSIE@Tunghai University 48

 Image
Video
Processing
Chapter 10 Laboratory

Image Segmentation

IVP Lab., CSIE@Tunghai University 49

 Image
Video
Processing
Optimal Thresholding Laboratory

IVP Lab., CSIE@Tunghai University 50

 Image
Video
Processing
Laboratory
Optimal Thresholding Example

IVP Lab., CSIE@Tunghai University 51

 Image
Video
Processing
Boundary Characteristics Laboratory

Eq. 10.3-6

IVP Lab., CSIE@Tunghai University 52

 Image
Video
Processing
Laboratory

IVP Lab., CSIE@Tunghai University 53

 Image
Video
Processing
Laboratory
Multispectral Thresholding Example

IVP Lab., CSIE@Tunghai University 54

 Image
Video
Processing
REGION-ORIENTED SEGMENTATION Laboratory

‰ Let R represent the entire image region

‰ Partitions R into n subregions, R1, R2, ..., Rn, such that
n
1. UR i = R
i =1
2. Ri is a connected region, i = 1, 2, ..., n

3. Ri I R j = φ for all i and j , i ≠ j

4. P(R;) = TRUE for i = 1, 2, ..., n, and

5, P ( R i U R j ) = FALSE for i ≠ j
where P(Ri) is a logical predicate over the points in set R;

IVP Lab., CSIE@Tunghai University 55

 Image
Video
Processing
Laboratory
Region Growing by Pixel Aggregation
‰ Starts with a set of "seed" points
‰ Appending to each seed point those neighboring pixels
with similar properties (such as gray level, texture, color)

Three problems for region growing

1. Selection of initial seeds
2. Selection of suitable properties
3. The formulation of a stopping rule
‰ Selecting starting points can be based on the nature of the
problem
‰ For example, in military applications of infrared imaging,
targets of interest are hotter (and thus appear brighter) than the
background. Choosing the brightest pixels as starting point
IVP Lab., CSIE@Tunghai University 56
 Image
Video
Processing
Laboratory

IVP Lab., CSIE@Tunghai University 57

 Image
Video
Processing
Laboratory

IVP Lab., CSIE@Tunghai University 58

 Image
Video
Processing
Laboratory
Region Splitting and Merging
‰ Let R represent the entire image region and select a
predicate P
‰ For a square image, subdivide it successively into
smaller and smaller quadrant regions so that, for any
region Ri , P(Ri) = TRUE
‰ If P(Ri) = FALSE, divide the image into quadrants
‰ If P is FALSE for any quadrant, subdivide that
quadrant into subquadrants, and so on
‰ This particular splitting technique has a convenient
representation in the form of a so-called quad tree (that
is, a tree in which each node has exactly four
descendants), as illustrated in Fig. 10.42

IVP Lab., CSIE@Tunghai University 59

 Image
Video
Processing
Laboratory

IVP Lab., CSIE@Tunghai University 60

 Image
Video
Processing
Morphological Watersheds Laboratory

Watershed lines

Regional minimum Dam

Catchment basins

IVP Lab., CSIE@Tunghai University 61

 Image
Video
Processing
Chapter 10 Laboratory

Image Segmentation

IVP Lab., CSIE@Tunghai University 62

 Image
Video
Processing
Chapter 10 Laboratory

Image Segmentation

IVP Lab., CSIE@Tunghai University 63

 Image
Video
Processing
Laboratory

IVP Lab., CSIE@Tunghai University 64

 Image
Video
Processing
Chapter 10 Laboratory

Image Segmentation

IVP Lab., CSIE@Tunghai University 65

 Image
Video
Processing
Laboratory

IVP Lab., CSIE@Tunghai University 66

 Image
Video
Processing
Laboratory

IVP Lab., CSIE@Tunghai University 67

 Image
Video
Processing
Laboratory

THE USE OF MOTION IN SEGMENTATION

‰ Detecting changes between two image frames f(x, y, ti) and f(x,
y, tj) is to compare the two images pixel by pixel.

‰ Suppose that we have a reference image containing only

stationary components, comparing this image against a
subsequent image having the same environment but including a
moving object results in the difference of the two images
canceling the stationary components, leaving only nonzero entries
that correspond to the nonstationary image components.

‰A difference image between two images taken at times ti and tj

may be defined as

IVP Lab., CSIE@Tunghai University 68

 Image
Video
Processing
Laboratory

⎧1 if f ( x, y , t i ) − f (c, y , t j ) > θ
d ij = ⎨ ( 7 . 5 . 1)
⎩0 otherwise

‰ In dynamic image analysis, all pixels with dij(x, y)=

1 are considered as the result of object motion
‰ This approach is applicable only if the two images
are registered and the illumination is relatively
constant within the bounds established by θ.
‰ In practice, 1-valued entries in dij(x, y) often arise as
a result of noise. Typically, these entries are isolated
points in the difference image

IVP Lab., CSIE@Tunghai University 69

 Image
Video
Processing
Laboratory

‰ A simple approach to their removal is to form 4- or 8-

connected regions of 1 's in dij(x, y) and then ignore any region
that has less than a predetermined number of entries
‰ Although it may result in ignoring small and/or slow-
moving objects, this approach improves the chances that the
remaining entries in the difference image actually are the result
of motion.

IVP Lab., CSIE@Tunghai University 70

 Image
Video
Processing
Laboratory

Accumulative differences
‰ A difference image often contains isolated noise. Although
the number of these entries can be reduced or completely
eliminated by a thresholded connectivity analysis, this
filtering process can also remove small or slow-moving
objects

IVP Lab., CSIE@Tunghai University 71

 Image
Video
Processing
Laboratory
‰ To address this problem by considering changes at a
pixel over several frames thus introducing a "memory"
into the process
‰ The basic idea is to ignore changes that occur only
sporadically over a frame sequence

Implementation
‰ Consider a sequence of image frames f(x,y,t1) , ...,
f(x, y, tn)
‰ Let f(x, y,t1) be the reference image
‰ An accumulative difference image is formed by
comparing this reference image with every subsequent
image in the sequence

IVP Lab., CSIE@Tunghai University 72

 Image
Video
Processing
Laboratory
‰ A counter for each pixel in the accumulative
image is incremented every time a difference
occurs at that pixel between the reference and an
image in the sequence

IVP Lab., CSIE@Tunghai University 73

 Image
Video
Processing
Laboratory
Accumulative difference image (ADI)
Eq. 10.6-2~4

IVP Lab., CSIE@Tunghai University 74

 Image
Video
Processing
Laboratory

IVP Lab., CSIE@Tunghai University 75