0% found this document useful (0 votes)
22 views58 pages

IT5409 - Ch6-Motion Detection and Tracking-Eng v1.0

The document discusses motion detection and tracking in computer vision. It covers topics like motion detection approaches, optical flow, Lucas-Kanade algorithm, Horn-Schunck algorithm, and background subtraction for static scene object detection and tracking. Motion can be detected by measuring changes in speed or vector of an object. Optical flow and change detection are common approaches to motion detection.

Uploaded by

Bui Minh Duc
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
0% found this document useful (0 votes)
22 views58 pages

IT5409 - Ch6-Motion Detection and Tracking-Eng v1.0

The document discusses motion detection and tracking in computer vision. It covers topics like motion detection approaches, optical flow, Lucas-Kanade algorithm, Horn-Schunck algorithm, and background subtraction for static scene object detection and tracking. Motion can be detected by measuring changes in speed or vector of an object. Optical flow and change detection are common approaches to motion detection.

Uploaded by

Bui Minh Duc
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
You are on page 1/ 58

Computer Vision

Chapter 6: Motion detection and Tracking

1
Course Content
• Chapter 1. Introduction
• Chapter 2. Image formation, acquisition and digitization
• Chapter 3. Image Processing
• Chapter 4. Feature detection and matching
• Chapter 5. Segmentation
• Chapter 6. Motion object detection and tracking
• Chapter 7. Object recognition and deep learning
Motion Detection and Tracking
• Motion detection
• Approaches of motion detection
• Moving object tracking

3
Motion Detection
• Action of sensing physical movement in a given area
• Motion can be detected by measuring change in speed or vector
of an object
• Goals of motion detection
‒ Identify moving objects
‒ Detection of unusual activity patterns
‒ Computing trajectories of moving objects

• Applications of motion detection


‒ Indoor/outdoor security
‒ Real time crime detection
‒ Traffic monitoring
‒ Many intelligent video analysis systems are based on motion detection

4
Approaches to Motion Detection
• Optical Flow
‒ Compute motion within region or the frame as a whole

• Change detection
‒ Detect objects within a scene
‒ Track object across a number of frames

5
Optical flow vs. motion field
• Optical flow does not always correspond to motion field

• Optical flow is an approximation of the motion field


• The error is small at points with high spatial gradient
under some simplifying assumptions
Estimating optical flow

• Assume the image intensity I is constant

Time = t Time = t+dt

( x, y )
( x + dx, y + dy )
𝐼0 (𝑥, 𝑦, 𝑡) ≈ 𝐼1 (𝑥 + 𝛿𝑥, 𝑦 + 𝛿𝑦, 𝑡 + 𝛿𝑡)
Optical flow constraint
𝐼(𝑥, 𝑦, 𝑡) ≈ 𝐼(𝑥 + 𝛿𝑥, 𝑦 + 𝛿𝑦, 𝑡 + 𝛿𝑡)
𝜕𝐼 𝜕𝐼 𝜕𝐼
𝐼(𝑥, 𝑦, 𝑡) ≈ 𝐼(𝑥, 𝑦, 𝑡) + 𝛿𝑥 + 𝛿𝑦 + 𝛿𝑡
𝜕𝑥 𝜕𝑦 𝜕𝑡

𝜕𝐼 𝛿𝑥 𝜕𝐼 𝛿𝑦 𝜕𝐼
+ + = 0, and let 𝛿𝑡 → 0
𝜕𝑥 𝛿𝑡 𝜕𝑦 𝛿𝑡 𝜕𝑡
𝐼𝑥 𝑢 + 𝐼𝑦 𝑣 + 𝐼𝑡 = 0
𝐼𝑥 𝑢 + 𝐼𝑦 𝑣 = −𝐼𝑡 ,
𝑢
𝐼𝑥 𝐼𝑦 = −𝐼𝑡 , ∇𝐼 𝑇 𝐮 = 𝐛; 𝐴𝐮 = 𝐛
𝑣
𝐸(𝑢, 𝑣) = (𝐼𝑥 𝑢 + 𝐼𝑦 𝑣 + 𝐼𝑡 )2
Lucas-Kanade algorithm
Assume a single velocity for all pixels within an image patch

𝐸(𝑢, 𝑣) = ෍ (𝐼𝑥 (𝑥, 𝑦)𝑢 + 𝐼𝑦 (𝑥, 𝑦)𝑣 + 𝐼𝑡 )2


𝑥,𝑦∈Ω

Minimizing 𝐴𝐮 − 𝐛 2
𝐮 = (𝐴𝑇 𝐴)−1 𝐴𝑇 𝐛
෍ 𝐼𝑥2 ෍ 𝐼𝑥 𝐼𝑦 ෍ 𝐼𝑥 𝐼𝑡
𝑢
=−
𝑣
Hessian ෍ 𝐼𝑥 𝐼𝑦 ෍ 𝐼𝑦2 ෍ 𝐼𝑦 𝐼𝑡

(෍ ∇𝐼 ⋅ ∇𝐼 𝑇 ) →
𝑢 = − ෍ ∇𝐼 ⋅ 𝐼𝑡

LHS: sum of the 2x2 outer product of the gradient vector


Matrix form
𝐸(𝐮 + Δ𝐮) = ෍ 𝐼1 (𝐱 𝑖 + 𝐮 + Δ𝐮) − 𝐼𝑜 (𝐱 𝑖 ) 2

𝑖
2
≈ ෍ 𝐼1 (𝐱 𝑖 + 𝐮) + 𝐉1 (𝐱 𝑖 + 𝐮)Δ𝐮 − 𝐼𝑜 (𝐱 𝑖 )
𝑖
2
= ෍ 𝐉1 (𝐱 𝑖 + 𝐮)Δ𝐮 + 𝑒𝑖
𝑖
𝜕𝐼1 𝜕𝐼1
𝐉1 (𝐱 𝑖 + 𝐮) ≈ ∇𝐼1 (𝐱 𝑖 + 𝐮) = , (𝐱 𝑖 + 𝐮)
𝜕𝑥 𝜕𝑦
𝑒𝑖 = 𝐼1 (𝐱 𝑖 + 𝐮) − 𝐼𝑜 (𝐱 𝑖 )
Matrix form
𝐴Δ𝐮 = 𝐛
𝐴 = ෍ 𝐉1𝑇 (𝐱 𝑖 + 𝐮)𝐉1 (𝐱 𝑖 + 𝐮)
𝑖

𝐛 = − ෍ 𝑒𝑖 𝐉1 (𝑥𝑖 + 𝑢)
𝑖

෍ 𝐼𝑥2 ෍ 𝐼𝑥 𝐼𝑦 ෍ 𝐼𝑥 𝐼𝑡
𝐴= , 𝐛=−
෍ 𝐼𝑥 𝐼𝑦 ෍ 𝐼𝑦2 ෍ 𝐼𝑦 𝐼𝑡

𝜕𝐼1 𝜕𝐼1
𝐉1 (𝐱 𝑖 + 𝐮) ≈ ∇𝐼1 (𝐱 𝑖 + 𝐮) = , (𝐱 𝑖 + 𝐮)
𝜕𝑥 𝜕𝑦
𝑒𝑖 = 𝐼1 (𝐱 𝑖 + 𝐮) − 𝐼𝑜 (𝐱 𝑖 )

𝐉1 (𝐱 𝑖 + 𝐮) ≈ 𝐉0 (𝐱 𝑖 )

for computational efficiency


Computing gradients in X-Y-T
y
1
𝐼𝑥 = [(𝐼𝑖+1,𝑗,𝑘 + 𝐼𝑖+1,𝑗,𝑘+1 + 𝐼𝑖+1,𝑗+1,𝑘 + 𝐼𝑖+1,𝑗+1,𝑘+1 )
4𝛿𝑥 time
−(𝐼𝑖,𝑗,𝑘 + 𝐼𝑖,𝑗,𝑘+1 + 𝐼𝑖,𝑗+1,𝑘 + 𝐼𝑖,𝑗+1,𝑘+1 )]

j+1
likewise for Iy and It
k+1

k
x
i i+1
The aperture problem
෍ 𝐼𝑥 𝐼𝑡
Let 𝐴 = ෍ ∇𝐼 ⋅ ∇𝐼 𝑇 , and 𝐛 = −
෍ 𝐼𝑦 𝐼𝑡

• Algorithm: At each pixel compute u by solving 𝐴𝐮 = 𝐛


• A is singular if all gradient vectors point in the
same direction
‒ e.g., along an edge
‒ of course, trivially singular if the summation is over a
single pixel or there is no texture
‒ i.e., only normal flow is available (aperture problem)
• Corners and textured areas are OK
Error functions
• Robust error function
𝑥2
𝐸(𝐮) = ෍ 𝜌(𝐼(𝐱 𝑖 + 𝐮) − 𝐼(𝐱)), 𝜌(𝑥) =
1 + 𝑥 2 /𝑎2
𝑖
• Spatially varying weights

2
𝐸(𝐮) = ෍ 𝑤0 (𝐱 𝑖 )𝑤1 (𝐱 𝑖 + 𝐮) 𝐼(𝐱 𝑖 + 𝐮) − 𝐼(𝐱 𝑖 )
𝑖
• Bias and gain: images taken with different exposure
𝐼(𝐱 + 𝐮) = (1 + 𝛼)𝐼(𝐱) + 𝛽, 𝛼 is the gain and 𝛽 is the bias
2
𝐸(𝐮) = ෍ 𝐼(𝐱 𝑖 + 𝐮) − (1 + 𝛼)𝐼(𝐱 𝑖 ) − 𝛽
𝑖

• Correlation (and normalized cross correlation)


E (u) =  I (xi ) I (xi + u)
i
Horn-Schunck algorithm
• Global method with smoothness constraint to solve
aperture problem
• Minimize a global energy functional with calculus of
variations
2
𝐸=න 𝐼𝑥 𝑢 + 𝐼𝑦 𝑣 + 𝐼𝑡 + 𝛼 2 ∇𝑢 2
+ ∇𝑣 2
𝑑𝑥𝑑𝑦

𝜕𝐿 𝜕 𝜕𝐿 𝜕 𝜕𝐿
− − =0
𝜕𝑢 𝜕𝑥 𝜕𝑢𝑥 𝜕𝑦 𝜕𝑢𝑦
𝜕𝐿 𝜕 𝜕𝐿 𝜕 𝜕𝐿
− − =0
𝜕𝑣 𝜕𝑥 𝜕𝑣𝑥 𝜕𝑦 𝜕𝑣𝑦
where 𝐿 is the integrand of the energy function
Horn-Schunck algorithm
𝐼𝑥 (𝐼𝑥 𝑢 + 𝐼𝑦 𝑣 + 𝐼𝑡 ) − 𝛼 2 Δ𝑢 = 0
𝐼𝑦 (𝐼𝑥 𝑢 + 𝐼𝑦 𝑣 + 𝐼𝑡 ) − 𝛼 2 Δ𝑣 = 0
𝜕2 𝜕2
where Δ = 2 + 2 is the Laplace operator
𝜕𝑥 𝜕𝑦

Δ𝑢(𝑥, 𝑦) = 𝑢(𝑥,lj 𝑦) − 𝑢(𝑥, 𝑦)


(𝐼𝑥2 + 𝛼 2 )𝑢 + 𝐼𝑥 𝐼𝑦 𝑣 = 𝛼 2 𝑢lj − 𝐼𝑥 𝐼𝑡
𝐼𝑥 𝐼𝑦 𝑢 + (𝐼𝑦2 + 𝛼 2 )𝑣 = 𝛼 2 𝑣lj − 𝐼𝑦 𝐼𝑡
Horn-Schunck algorithm
(u , v )
• Iterative scheme v

• Yields high density flow


• Fill in missing information in the homogenous
regions
• More sensitive to noise than local methods
Background Subtraction
• Uses a reference background image for
comparison purposes.
• Current image (containing target object) is
compared to reference image pixel by pixel.
• Places where there are differences are detected
and classified as moving objects.

Motivation: simple difference of two images


shows moving objects

18
a. Original scene b. Same scene later

Subtraction of scene a from scene b Subtracted image with threshold of 100

19
Static Scene Object Detection and
Tracking
• Model the background and subtract to obtain
object mask
• Filter to remove noise
• Group adjacent pixels to obtain objects
• Track objects between frames to develop
trajectories

20
Approaches to Background Modeling

• Background Subtraction

• Statistical Methods
(e.g., Gaussian Mixture Model, Stauffer and Grimson 2000)

Background Subtraction:
1. Construct a background image B as average of few
images
2. For each actual frame I, classify individual pixels as
foreground if |B-I| > T (threshold)
3. Clean noisy pixels

21
22
Background Subtraction

Background Image Current Image

23
Statistical Methods

• Pixel statistics: average and standard deviation of


color and gray level values

• Gaussian Mixture Model


‒ Model the color values of a particular pixel as a mixture of
Gaussians
‒ Multiple adaptive Gaussians are necessary to cope with
acquisition noise, lighting changes, etc.
‒ Pixel values that do not fit the background distributions
(Mahalanobis distance) are considered foreground

24
Gaussian Mixture Model

Block 44x42 Pixel 172x165


Temple1 - RGB Distribution of Pixel 172x165

200

150

100

B
50

0
250
Temple1 - RG Distribution of Pixel 172x165 200
250
200
250 150
150
G 100 100
R

200

150
G

100

50
R-G-B Distribution

0
0 50 100 150 200 250
R-G Distribution R

25
Detection of Moving Objects Based on
Local Variation

For each block location (x,y) in the video plane


• Consider texture vectors in a symmetric window [t-W,
t+W] at time t
• Compute the covariance matrix
• Motion measure is defined as
the largest eigenvalue of the covariance matrix

26
Dynamic Distribution Learning and Outlier
Detection
f (t ) − mean(t − 1)
(1)
 C1 Detect Outlier
std (t − 1)

f (t ) − mean(t − 1)
(2)  C2 Switch to a nominal state
std (t − 1)

(3) mean(t ) = u  mean(t − 1) + (1 − u )  f (t )


Update the estimates of mean and standard
deviation only when the outliers are not
(4)
std (t ) =  2 (t ) detected

(5)  2 (t ) = u   2 (t − 1) + (1 − u )  ( f (t ) − mean(t − 1)) 2

27
Moving object tracking
• Model of object motion
• Kalman filter
• Mean Shift
Model based tracking
• Mathematical model of objects’ motions:
‒ position, velocity (speed, direction), acceleration
• Can predict objects’ positions
Motion
Model

Predict
Update? Location

Verify
Location

29
Simple Motion Model

• Newton’s laws
1
s(t ) = s + ut + a t
0
2

2
• s = position
• u = velocity
• a = acceleration
‒ all vector quantities
‒ measured in image co-ordinates

30
Prediction
• Can predict position at time t knowing
‒ Position
‒ Velocity
‒ Acceleration
• At t=0

31
Uncertainty
• If some error in a - Da or u - Du
• Then error in predicted position - Ds

Ds(t ) = s0 + Dut + Da t 2
1
2

32
Verification
• Is the object at the predicted location?
‒ Matching
• How to decide if object is found
‒ Search area
• Where to look for object

33
Object Matching
• Compare
‒ A small bitmap derived from the object vs.
‒ Small regions of the image
• Matching?
‒ Measure differences

34
Search Area
• Uncertainty in knowledge of model parameters
‒ Limited accuracy of measurement
‒ Values might change between measurements
• Define an area in which object could be
‒ Centred on predicted location, s  Ds

35
Update the Model
• Is the object at the predicted location?
• Yes
‒ No change to model
• No
‒ Model needs updating
‒ Kalman filter is a solution
• Mathematically rigorous methods of using uncertain
measurements to update a model

36
Kalman filter
• Matrix description of system state, model and
measurement
• Progressive method

prediction

correction
• Proper dealing with noise
Kalman filter
• Advantages of using KF in particle tracking
• Progressive method
‒ No large matrices has to be inverted
• Proper dealing with system noise
• Track finding and track fitting
• Detection of outliers
• Merging track from different segments
Kalman filter assumptions

• Linear system
‒ System parameters are linear
function of parameters at some
previous time
‒ Measurements are linear function
of parameters
• White Gaussian noise
 KF is the
‒ White: uncorrelated in time
optimal filter
‒ Gaussian: noise amplitude
Kalman filter
• Relates
‒ Measurements y[k]
• e.g. positions
‒ System state x[k]
• Position, velocity of object, etc
‒ Observation matrix H[k]
• Relates system state to measurements
‒ Evolution matrix A[k]
• Relates state of system between epochs
‒ Measurement noise n[k]
‒ Process noise v[k]

42
Mathematically

How do observations relate to model?

yk  = Hk xk  + nk 


 xk 
   xk 
 x k  1 0 0 0   + nk 
=
 yk  0 0 1 0  yk 
   
 
 y k 

43
Prediction of System State
• Relates system states at epochs k and k+1

xˆ k + 1 | k  = Ak xk | k  + vk 


 xˆk + 1 | k  1 T 0 0  xk | k 
 xˆk + 1 | k   0  
1 0 0   xk | k 
 = + vk 
 yˆ k + 1 | k  0 0 1 T   yk | k 
ˆ    
 y k + 1 | k   0 0 0 1  y k | k 

44
Prediction of Observation

• From predicted system state, estimate what


observation should occur:

yˆ k + 1 | k  = Hk xˆ k + 1 | k  + nk 


 xˆk + 1 | k 
 
 xk + 1 | k  1 0 0 0  xk + 1 | k 
ˆ ˆ

 yˆ k + 1 | k  = 0 0 1 0  yˆ k + 1 | k  + nk 
   
ˆ 
 y k + 1 | k 
45
Updating the Model

• Predict/estimate a measurement yˆ k + 1 | k 
• Measurement
yk + 1
• Predict state of model
• How does the new measurement xˆ k + 1 | k 
contribute to updating the model?

46
Updating the Model

xˆ k + 1 | k + 1 = xˆ k + 1 | k  + Gk + 1Dyk + 1 | k 
Dyk + 1 | k  = yk + 1 − yˆ k + 1 | k 
• G is Kalman Gain
‒ Derived from A, H, v, n.

(
G = Ck | k H HC k | k  H + n
T T −1
)
Ck + 1 | k  = Ck | k  − GHCk | k 
C = system covariance

47
Example
• Tracking two corners of a minimum bounding box
• Matching using colour
• Image differencing to locate target

48
Condensation Tracking
• So far considered single motions
• What if movements change?
‒ Bouncing ball
‒ Human movements
• Use multiple models
‒ plus a model selection mechanism

49
Selection and Tracking
• Occur simultaneously
• Maintain
‒ A distribution of likely object positions plus weights
• Predict
‒ Select N samples, predict locations
• Verify
‒ Match predicted locations against image
‒ Update distributions

50
Tracking Using Hidden Markov
Models
• Hidden Markov model describes
‒ States occupied by a system
‒ Possible transitions between states
‒ Probabilities of transitions
‒ How to recognise a transition

51
Mean-shift

• The mean-shift algorithm is an efficient approach to


tracking objects whose appearance is defined by
distributions (histograms). (not limited to only color)
• Motivation – to track non-rigid objects, (like a walking
person), it is hard to specify an explicit 2D parametric
motion model.

• Appearances of non-rigid objects can sometimes be


modeled with color distributions
Intuitive Description
Region of
interest

Center of
mass

Mean Shift
vector
Objective : Find the densest region
Distribution of identical billiard balls
Stolen from: www.wisdom.weizmann.ac.il/~deniss/vision_spring04/files/mean_shift/mean_shift.ppt
Mean Shift Vector
• Given:
Data points and approximate location of the mean of
this data:

• Task:
Estimate the exact location of the mean of the data
by determining the shift vector from the initial mean.
Mean Shift Vector
Mean-Shift Object Tracking
Target Representation

Choose a Represent the


Choose a model by its
reference
feature space PDF in the
target model
feature space

0.35

0.3

0.25
Quantized

Probability
0.2

Color Space 0.15

0.1

0.05

0
1 2 3 . . . m

color

Stolen from: www.cs.wustl.edu/~pless/559/lectures/lecture22_tracking.ppt


Mean-Shift Object Tracking
PDF Representation

Target Model Target Candidate


(centered at 0) (centered at y)
0.35 0.3

0.3 0.25
0.25
Probability

0.2

Probability
0.2
0.15
0.15
0.1
0.1

0.05 0.05

0 0
1 2 3 . . . m 1 2 3 . . . m
color color

m m
q = qu u =1..m q u =1 p ( y ) =  pu ( y )u =1..m p u =1
u =1 u =1

Similarity f ( y ) = f  q , p ( y ) 
Function:   Q is the target histogram,
P is the object histogram
Stolen from: www.cs.wustl.edu/~pless/559/lectures/lecture22_tracking.ppt (depends on location y)
Mean-Shift Object Tracking
Target Localization Algorithm

Start from
the position Search in the Find best
of the model model’s candidate by
in the current neighborhood maximizing a
frame in next frame similarity func.

q p ( y) f  p ( y ) , q 
Stolen from: www.cs.wustl.edu/~pless/559/lectures/lecture22_tracking.ppt
Mean Shift
• Mean-Shift in tracking task:
➢ track the motion of a cluster of interesting features.
• 1. choose the feature distribution to represent an object (e.g.,
color + texture),
• 2. start the mean-shift window over the feature distribution
generated by the object
• 3. finally compute the chosen feature distribution over the next
video frame
‒ Starting from the current window location, the mean-shift algorithm
will find the new peak or mode of the feature distribution, which
(presumably) is centered over the object that produced the color and
texture in the first place.
‒ In this way, the mean-shift window tracks the movement of the object
frame by frame.
Thank you for
your attention!

You might also like