Review- Image segmentation

BBM 413
Fundamentals of • Goal: identify groups of pixels that go together

Image Processing

Erkut Erdem
Dept. of Computer Engineering
Hacettepe University

Segmentation – Part 2

Slide credit: S. Seitz, K. Grauman

Review- The goals of segmentation Review- What is segmentation?

• Separate image into coherent “objects”
• Clustering image elements that “belong together”

– Partitioning
• Divide into regions/sequences with coherent internal properties
– Grouping
• Identify sets of coherent tokens in image
image human segmentation

http://www.eecs.berkeley.edu/Research/Projects/CS/vision/grouping/segbench/
Slide credit: S. Lazebnik Slide credit: Fei-Fei Li
Review- K-means clustering Review - K-means: pros and cons
• Basic idea: randomly initialize the k cluster centers, and Pros
iterate between the two steps we just saw.
• Simple, fast to compute
1. Randomly initialize the cluster centers, c1, ..., cK • Converges to local minimum of
2. Given cluster centers, determine points in each cluster within-cluster squared error
• For each point p, find the closest ci. Put p into cluster i
3. Given points in each cluster, solve for ci
• Set ci to be the mean of points in cluster i Cons/issues
4. If ci have changed, repeat Step 2
• Setting k?
Properties
• Sensitive to initial centers
• Will always converge to some solution
• Can be a “local minimum” • Sensitive to outliers
• does not always find the global minimum of objective function: • Detects spherical clusters
• Assuming means can be computed

Slide credit: S. Seitz Slide credit: K Grauman

Mean shift clustering and

Segmentation methods
segmentation
• Segment foreground from background
• An advanced and versatile technique for clustering-based
• Histogram-based segmentation segmentation
• Segmentation as clustering
– K-means clustering
– Mean-shift segmentation
• Graph-theoretic segmentation
– Min cut
– Normalized cuts
• Interactive segmentation

http://www.caip.rutgers.edu/~comanici/MSPAMI/msPamiResults.html

D. Comaniciu and P. Meer, Mean Shift: A Robust Approach toward Feature Space Analysis,
PAMI 2002.
Slide credit: S. Lazebnik
 Mean shift algorithm
Finding Modes in a Histogram
• The mean shift algorithm seeks modes or local maxima of
density in the feature space

Feature space
image (L*u*v* color values)

• How Many Modes Are There?

– Easy to see, hard to compute

Slide credit: S. Seitz Slide credit: S. Lazebnik

Mean shift algorithm

Mean Shift Algorithm Mean shift
1. Choose a search window size. Search
window
2. Choose the initial location of the search window.
3. Compute the mean location (centroid of the data) in the search window. Center of
4. Center the search window at the mean location computed in Step 3. mass
5. Repeat Steps 3 and 4 until convergence.
The mean shift algorithm seeks the mode or point of highest density of a data distribution:

Two issues:
(1) Kernel to interpolate density
based on sample positions.
(2) Gradient ascent to mode.
Mean Shift
vector

Slide credit: B. Freeman and A. Torralba Slide credit: Y. Ukrainitz & B. Sarel
Mean shift Mean shift
Search Search
window window

Center of Center of
mass mass

Mean Shift Mean Shift

vector vector

Slide credit: Y. Ukrainitz & B. Sarel Slide credit: Y. Ukrainitz & B. Sarel

Mean shift Mean shift

Search Search
window window

Center of Center of
mass mass

Mean Shift Mean Shift

vector vector

Slide credit: Y. Ukrainitz & B. Sarel Slide credit: Y. Ukrainitz & B. Sarel
Mean shift Mean shift
Search Search
window window

Center of Center of
mass mass

Mean Shift
vector

Slide credit: Y. Ukrainitz & B. Sarel Slide credit: Y. Ukrainitz & B. Sarel

Mean shift clustering/segmentation

Mean shift clustering
• Find features (color, gradients, texture, etc)
• Cluster: all data points in the attraction basin of a mode • Initialize windows at individual feature points
• Perform mean shift for each window until convergence
• Attraction basin: the region for which all trajectories lead to the
same mode • Merge windows that end up near the same “peak” or mode

Slide credit: Y. Ukrainitz & B. Sarel Slide credit: S. Lazebnik

 Window in image domain Apply mean shift jointly in the image
(left col.) and range (right col.) domains
1 Intensities of pixels within
image domain window
2
0 1
Center of mass of pixels within
both image and range domain 3
windows 0 1
Window in
range domain
4
Center of mass of pixels within
both image and range domain
windows
5 6
0 1
7 Comaniciu and Meer, IEEE PAMI vol. 24, no. 5, 2002
Slide credit: B. Freeman and A. Torralba
0 1 Slide credit: B. Freeman and A. Torralba

Mean shift segmentation results More results

http://www.caip.rutgers.edu/~comanici/MSPAMI/msPamiResults.html
Slide credit: S. Lazebnik Slide credit: S. Lazebnik
More results Mean shift pros and cons
• Pros
– Does not assume spherical clusters
– Just a single parameter (window size)
– Finds variable number of modes
– Robust to outliers
• Cons
– Output depends on window size
– Computationally expensive
– Does not scale well with dimension of feature space

Slide credit: S. Lazebnik Slide credit: S. Lazebnik

Graph-Theoretic Image Segmentation

Segmentation methods
• Segment foreground from background
Build a weighted graph G=(V,E) from image
• Histogram-based segmentation
• Segmentation as clustering V: image pixels
– K-means clustering
– Mean-shift segmentation E: connections between
• Graph-theoretic segmentation pairs of nearby pixels
• Min cut
• Normalized cuts Wij : probability that i &j
• Interactive Segmentation belong to the same
region

Segmentation = graph partition

Slide credit: B. Freeman and A. Torralba
 A Weighted Graph and its
Graphs Representations
Representation

a b c d e Affinity Matrix
a é0 1 0 0 1ù a é 1 .1 .3 0 0 ù
a ê1
b b
ê 0 0 0 0úú b ê.1 1 .4 0 .2ú
ê ú
c ê0 0 0 0 1ú ê.3 .4 1 .6 .7ú
c ê ú W=
ê ú
e d
ê0 0 0 0 1ú c ê 0 0 .6 1 1 ú
e
e êë1 0 1 1 0úû 6 êë 0 .2 .7 1 1 úû
d
d Wij : probability that i &j
Adjacency Matrix
belong to the same
region
Slide credit: B. Freeman and A. Torralba * From Khurram Hassan-Shafique CAP5415 Computer Vision 2003 Slide credit: B. Freeman and A. Torralba * From Khurram Hassan-Shafique CAP5415 Computer Vision 2003

Segmentation by graph partitioning Affinity between pixels

Similarities among pixel descriptors

j Wij = exp(-|| zi – zj ||2 / σ2)
wij σ = Scale factor…
it will hunt us later
i

A B C

• Break graph into segments

– Delete links that cross between segments
– Easiest to break links that have low affinity
• similar pixels should be in the same segments
• dissimilar pixels should be in different segments

Slide credit: S. Seitz Slide credit: B. Freeman and A. Torralba

Affinity between pixels Scale affects affinity

Similarities among pixel descriptors • Small σ: group only nearby points

Wij = exp(-|| zi – zj ||2 / σ2) • Large σ: group far-away points
σ = Scale factor…
Interleaving edges it will hunt us later
Wij = 1 - max Pb
Line between i and j
With Pb = probability of boundary

Slide credit: B. Freeman and A. Torralba Slide credit: S. Lazebnik

British Machine Vision Conference, pp. 103-108, 1990

Example eigenvector

points
Wij = exp(-|| zi – zj ||2 / s2)
With an appropriate s
eigenvector

W=

The eigenvectors of W are:

Affinity matrix
Three points in feature space

The first 2 eigenvectors group the points

as desired…
Slide credit: B. Freeman and A. Torralba Slide credit: B. Freeman and A. Torralba
Example eigenvector Graph cut

points A B

• Set of edges whose removal makes a graph disconnected

eigenvector
• Cost of a cut: sum of weights of cut edges
• A graph cut gives us a segmentation
– What is a “good” graph cut and how do we find one?
Affinity matrix

Slide credit: B. Freeman and A. Torralba Slide credit: S. Seitz

Segmentation methods Minimum cut

• Segment foreground from background
• Histogram-based segmentation A cut of a graph G is the set of edges S such that
• Segmentation as clustering removal of S from G disconnects G.
– K-means clustering
– Mean-shift segmentation
Cut: sum of the weight of the cut edges:
• Graph-theoretic segmentation
• Min cut cut(A,B) = ∑ W(u,v),
• Normalized cuts u∈A,v ∈B

• Interactive segmentation with A ∩ B = ∅

Slide credit: B. Freeman and A. Torralba

 Minimum cut Minimum cut
• We can do segmentation by finding the minimum cut in a graph • We can do segmentation by finding the minimum cut in a graph
– Efficient algorithms exist for doing this – Efficient algorithms exist for doing this

Minimum cut example Minimum cut example

Slide credit: S. Lazebnik Slide credit: S. Lazebnik

Drawbacks of Minimum cut Segmentation methods

• Segment foreground from background
• Weight of cut is directly proportional to the number of edges
in the cut. • Histogram-based segmentation
• Segmentation as clustering
– K-means clustering
– Mean-shift segmentation
• Graph-theoretic segmentation
• Min cut
Cuts with • Normalized cuts
lesser weight
than the • Interactive segmentation
ideal cut
Ideal Cut

Slide credit: B. Freeman and A. Torralba * Slide from Khurram Hassan-Shafique CAP5415 Computer Vision 2003
 Normalized cuts Normalized cut
Write graph as V, one cluster as A and the other as B • Let W be the adjacency matrix of the graph
• Let D be the diagonal matrix with diagonal entries
cut(A,B) cut(A,B) D(i, i) = Σj W(i, j)
Ncut(A,B) = +
assoc(A,V) assoc(B,V) • Then the normalized cut cost can be written as
yT ( D - W ) y
cut(A,B) is sum of weights with one end in A and one end in B yT Dy
cut(A,B) = ∑ W(u,v), where y is an indicator vector whose value should be 1 in
u∈A,v ∈B the ith position if the ith feature point belongs to A and a
with A ∩ B = ∅ negative constant otherwise
assoc(A,V) is sum of all edges with one end in A.
assoc(A,B) = ∑ W(u,v)
€ u∈A,v ∈B

A and B not necessarily disjoint

J. Shi and J. Malik. Normalized cuts and image segmentation. PAMI 2000
Slide credit: B. Freeman and A. Torralba J. Shi and J. Malik. Normalized cuts and image segmentation. PAMI 2000 Slide credit: S. Lazebnik

Normalized cut Normalized cut algorithm

• Finding the exact minimum of the normalized cut cost is NP-
complete, but if we relax y to take on arbitrary values, then we
can minimize the relaxed cost by solving the generalized
eigenvalue problem (D − W)y = λDy
• The solution y is given by the generalized eigenvector
corresponding to the second smallest eigenvalue
• Intitutively, the ith entry of y can be viewed as a “soft”
indication of the component membership of the ith feature
– Can use 0 or median value of the entries as the splitting point
(threshold), or find threshold that minimizes the Ncut cost

J. Shi and J. Malik. Normalized cuts and image segmentation. PAMI 2000 Slide credit: S. Lazebnik Slide credit: B. Freeman and A. Torralba
 Boundaries of image regions defined
Global optimization
by a number of attributes
• In this formulation, the segmentation becomes a global process. – Brightness/color
• Decisions about what is a boundary are not local (as in Canny – Texture
edge detector) – Motion
– Stereoscopic depth
– Familiar configuration

Slide credit: B. Freeman and A. Torralba

[Malik] Slide credit: B. Freeman and A. Torralba

Example
Affinity:

brightness Location
N pixels = ncols * nrows

W=

Slide credit: B. Freeman and A. Torralba Slide credit: B. Freeman and A. Torralba
 Brightness Image Segmentation Brightness Image Segmentation

http://www.cs.berkeley.edu/~malik/papers/SM-ncut.pdf Slide credit: B. Freeman and A. Torralba http://www.cs.berkeley.edu/~malik/papers/SM-ncut.pdf Slide credit: B. Freeman and A. Torralba

Results on color segmentation

http://www.cs.berkeley.edu/~malik/papers/SM-ncut.pdf Slide credit: B. Freeman and A. Torralba http://www.cs.berkeley.edu/~malik/papers/SM-ncut.pdf Slide credit: B. Freeman and A. Torralba
Example results Results: Berkeley Segmentation Engine

http://www.cs.berkeley.edu/~fowlkes/BSE/
Slide credit: S. Lazebnik Slide credit: S. Lazebnik

Normalized cuts: Pro and con Segmentation methods

• Pros • Segment foreground from background
– Generic framework, can be used with many different • Histogram-based segmentation
features and affinity formulations
• Segmentation as clustering
• Cons – K-means clustering
– High storage requirement and time complexity – Mean-shift segmentation
– Bias towards partitioning into equal segments • Graph-theoretic segmentation
• Min cut
• Normalized cuts

• Interactive segmentation

Slide credit: S. Lazebnik

 Intelligent Scissors [Mortensen 95] Intelligent Scissors
• Approach answers a basic question • Basic Idea
– Q: how to find a path from seed to mouse that follows object – Define edge score for each pixel
boundary as closely as possible? • edge pixels have low cost
– Find lowest cost path from seed to mouse

mouse

seed
Mortensen and Barrett, Intelligent
Scissors for Image Composition,
Proc. 22nd annual conference on Questions
Computer graphics and interactive
techniques, 1995
• How to define costs?
• How to find the path?
Slide credit: S. Seitz Slide credit: S. Seitz

Path Search (basic idea) How does this really work?

• Treat the image as a graph
• Graph Search Algorithm
– Computes minimum cost path from seed to all other pixels
q
c
p

Graph
• node for every pixel p
• link between every adjacent pair of pixels, p,q
• cost c for each link
Note: each link has a cost
• this is a little different than the figure before where each
Slide credit: S. Seitz
pixel had a cost Slide credit: S. Seitz
Defining the costs Defining the costs
• Treat the image as a graph

q q
c c
p p

Want to hug image edges: how to define cost of a link? • c can be computed using a cross-correlation filter
• the link should follow the intensity edge – assume it is centered at p
– want intensity to change rapidly to the link
• c - |difference of intensity to link| • Also typically scale c by its length
– set c = (max-|filter response|)
• where max = maximum |filter response| over all pixels in the image
Slide credit: S. Seitz Slide credit: S. Seitz

Defining the costs Dijkstra’s shortest path algorithm

link cost
q 4 9 5
1 1 c 1
1 0 3
3 2 3
w -1
-1 -1 p
Algorithm
1. init node costs to , set p = seed point, cost(p) = 0
2. expand p as follows:
• c can be computed using a cross-correlation filter
for each of p s neighbors q that are not expanded
– assume it is centered at p
» set cost(q) = min( cost(p) + cpq, cost(q) )

• Also typically scale c by its length

– set c = (max-|filter response|)
• where max = maximum |filter response| over all pixels in the image
Slide credit: S. Seitz Slide credit: S. Seitz
Dijkstra’s shortest path algorithm Dijkstra’s shortest path algorithm
4 9 5 4 9 5

4 9 5 3 2 5 4 9 5
1 1 0 3 3 2 1 1 0 3 3
3 2 3 3 4 3 3 2 3

3 2 3 3 2 3
Algorithm Algorithm
1. init node costs to , set p = seed point, cost(p) = 0 1. init node costs to , set p = seed point, cost(p) = 0
2. expand p as follows: 2. expand p as follows:
for each of p s neighbors q that are not expanded for each of p s neighbors q that are not expanded
» set cost(q) = min( cost(p) + cpq, cost(q) ) » set cost(q) = min( cost(p) + cpq, cost(q) )
» if q s cost changed, make q point back to p » if q s cost changed, make q point back to p
» put q on the ACTIVE list (if not already there) » put q on the ACTIVE list (if not already there)
3. set r = node with minimum cost on the ACTIVE list
4. repeat Step 2 for p = r
Slide credit: S. Seitz Slide credit: S. Seitz

Dijkstra’s shortest path algorithm Segmentation by min (s-t) cut

4 3 6 5 min cut

3 2 5 4 9 5
3 2 1 1 0 3 3
3 4 3 3 2 3 s
t
4 3 2 3
Algorithm
1. init node costs to , set p = seed point, cost(p) = 0 • Graph
– node for each pixel, link between pixels
2. expand p as follows:
– specify a few pixels as foreground and background
for each of p s neighbors q that are not expanded
• create an infinite cost link from each bg pixel to the t node
» set cost(q) = min( cost(p) + cpq, cost(q) ) • create an infinite cost link from each fg pixel to the s node
» if q s cost changed, make q point back to p
– compute min cut that separates s from t
» put q on the ACTIVE list (if not already there)
– how to define link cost between neighboring pixels?
3. set r = node with minimum cost on the ACTIVE list
4. repeat Step 2 for p = r Y. Boykov and M-P Jolly, Interactive Graph Cuts for Optimal Boundary &
Slide credit: S. Seitz Region Segmentation of Objects in N-D images, ICCV, 2001. Slide credit: S. Seitz
 Do we need recognition to take the next step in performance?
Random Walker
• Compute probability that a random walker arrives at seed

L. Grady, Random Walks for Image Segmentation, IEEE T-PAMI, 2006

http://cns.bu.edu/~lgrady/Random_Walker_Image_Segmentation.html Slide credit: B. Freeman and A. Torralba

Top-down segmentation Top-down segmentation

Normalized
cuts

Top-down
segmentation

• E. Borenstein and S. Ullman, Class-specific, top-down segmentation, • E. Borenstein and S. Ullman, Class-specific, top-down segmentation,
ECCV 2002 ECCV 2002
• A. Levin and Y. Weiss, Learning to Combine Bottom-Up and Top- • A. Levin and Y. Weiss, Learning to Combine Bottom-Up and Top-
Down Segmentation, ECCV 2006. Down Segmentation, ECCV 2006.
Slide credit: S. Lazebnik Slide credit: S. Lazebnik
 Motion segmentation

Input sequence Image Segmentation Motion Segmentation

Input sequence Image Segmentation Motion Segmentation

A. Barbu, S.C. Zhu. Generalizing Swendsen-Wang to sampling arbitrary

posterior probabilities, IEEE TPAMI, 2005. Slide credit: K. Grauman