IT5409 - Ch6-Motion Detection and Tracking-Eng v1.0
IT5409 - Ch6-Motion Detection and Tracking-Eng v1.0
1
Course Content
• Chapter 1. Introduction
• Chapter 2. Image formation, acquisition and digitization
• Chapter 3. Image Processing
• Chapter 4. Feature detection and matching
• Chapter 5. Segmentation
• Chapter 6. Motion object detection and tracking
• Chapter 7. Object recognition and deep learning
Motion Detection and Tracking
• Motion detection
• Approaches of motion detection
• Moving object tracking
3
Motion Detection
• Action of sensing physical movement in a given area
• Motion can be detected by measuring change in speed or vector
of an object
• Goals of motion detection
‒ Identify moving objects
‒ Detection of unusual activity patterns
‒ Computing trajectories of moving objects
4
Approaches to Motion Detection
• Optical Flow
‒ Compute motion within region or the frame as a whole
• Change detection
‒ Detect objects within a scene
‒ Track object across a number of frames
5
Optical flow vs. motion field
• Optical flow does not always correspond to motion field
( x, y )
( x + dx, y + dy )
𝐼0 (𝑥, 𝑦, 𝑡) ≈ 𝐼1 (𝑥 + 𝛿𝑥, 𝑦 + 𝛿𝑦, 𝑡 + 𝛿𝑡)
Optical flow constraint
𝐼(𝑥, 𝑦, 𝑡) ≈ 𝐼(𝑥 + 𝛿𝑥, 𝑦 + 𝛿𝑦, 𝑡 + 𝛿𝑡)
𝜕𝐼 𝜕𝐼 𝜕𝐼
𝐼(𝑥, 𝑦, 𝑡) ≈ 𝐼(𝑥, 𝑦, 𝑡) + 𝛿𝑥 + 𝛿𝑦 + 𝛿𝑡
𝜕𝑥 𝜕𝑦 𝜕𝑡
𝜕𝐼 𝛿𝑥 𝜕𝐼 𝛿𝑦 𝜕𝐼
+ + = 0, and let 𝛿𝑡 → 0
𝜕𝑥 𝛿𝑡 𝜕𝑦 𝛿𝑡 𝜕𝑡
𝐼𝑥 𝑢 + 𝐼𝑦 𝑣 + 𝐼𝑡 = 0
𝐼𝑥 𝑢 + 𝐼𝑦 𝑣 = −𝐼𝑡 ,
𝑢
𝐼𝑥 𝐼𝑦 = −𝐼𝑡 , ∇𝐼 𝑇 𝐮 = 𝐛; 𝐴𝐮 = 𝐛
𝑣
𝐸(𝑢, 𝑣) = (𝐼𝑥 𝑢 + 𝐼𝑦 𝑣 + 𝐼𝑡 )2
Lucas-Kanade algorithm
Assume a single velocity for all pixels within an image patch
Minimizing 𝐴𝐮 − 𝐛 2
𝐮 = (𝐴𝑇 𝐴)−1 𝐴𝑇 𝐛
𝐼𝑥2 𝐼𝑥 𝐼𝑦 𝐼𝑥 𝐼𝑡
𝑢
=−
𝑣
Hessian 𝐼𝑥 𝐼𝑦 𝐼𝑦2 𝐼𝑦 𝐼𝑡
( ∇𝐼 ⋅ ∇𝐼 𝑇 ) →
𝑢 = − ∇𝐼 ⋅ 𝐼𝑡
𝑖
2
≈ 𝐼1 (𝐱 𝑖 + 𝐮) + 𝐉1 (𝐱 𝑖 + 𝐮)Δ𝐮 − 𝐼𝑜 (𝐱 𝑖 )
𝑖
2
= 𝐉1 (𝐱 𝑖 + 𝐮)Δ𝐮 + 𝑒𝑖
𝑖
𝜕𝐼1 𝜕𝐼1
𝐉1 (𝐱 𝑖 + 𝐮) ≈ ∇𝐼1 (𝐱 𝑖 + 𝐮) = , (𝐱 𝑖 + 𝐮)
𝜕𝑥 𝜕𝑦
𝑒𝑖 = 𝐼1 (𝐱 𝑖 + 𝐮) − 𝐼𝑜 (𝐱 𝑖 )
Matrix form
𝐴Δ𝐮 = 𝐛
𝐴 = 𝐉1𝑇 (𝐱 𝑖 + 𝐮)𝐉1 (𝐱 𝑖 + 𝐮)
𝑖
𝐛 = − 𝑒𝑖 𝐉1 (𝑥𝑖 + 𝑢)
𝑖
𝐼𝑥2 𝐼𝑥 𝐼𝑦 𝐼𝑥 𝐼𝑡
𝐴= , 𝐛=−
𝐼𝑥 𝐼𝑦 𝐼𝑦2 𝐼𝑦 𝐼𝑡
𝜕𝐼1 𝜕𝐼1
𝐉1 (𝐱 𝑖 + 𝐮) ≈ ∇𝐼1 (𝐱 𝑖 + 𝐮) = , (𝐱 𝑖 + 𝐮)
𝜕𝑥 𝜕𝑦
𝑒𝑖 = 𝐼1 (𝐱 𝑖 + 𝐮) − 𝐼𝑜 (𝐱 𝑖 )
𝐉1 (𝐱 𝑖 + 𝐮) ≈ 𝐉0 (𝐱 𝑖 )
j+1
likewise for Iy and It
k+1
k
x
i i+1
The aperture problem
𝐼𝑥 𝐼𝑡
Let 𝐴 = ∇𝐼 ⋅ ∇𝐼 𝑇 , and 𝐛 = −
𝐼𝑦 𝐼𝑡
2
𝐸(𝐮) = 𝑤0 (𝐱 𝑖 )𝑤1 (𝐱 𝑖 + 𝐮) 𝐼(𝐱 𝑖 + 𝐮) − 𝐼(𝐱 𝑖 )
𝑖
• Bias and gain: images taken with different exposure
𝐼(𝐱 + 𝐮) = (1 + 𝛼)𝐼(𝐱) + 𝛽, 𝛼 is the gain and 𝛽 is the bias
2
𝐸(𝐮) = 𝐼(𝐱 𝑖 + 𝐮) − (1 + 𝛼)𝐼(𝐱 𝑖 ) − 𝛽
𝑖
𝜕𝐿 𝜕 𝜕𝐿 𝜕 𝜕𝐿
− − =0
𝜕𝑢 𝜕𝑥 𝜕𝑢𝑥 𝜕𝑦 𝜕𝑢𝑦
𝜕𝐿 𝜕 𝜕𝐿 𝜕 𝜕𝐿
− − =0
𝜕𝑣 𝜕𝑥 𝜕𝑣𝑥 𝜕𝑦 𝜕𝑣𝑦
where 𝐿 is the integrand of the energy function
Horn-Schunck algorithm
𝐼𝑥 (𝐼𝑥 𝑢 + 𝐼𝑦 𝑣 + 𝐼𝑡 ) − 𝛼 2 Δ𝑢 = 0
𝐼𝑦 (𝐼𝑥 𝑢 + 𝐼𝑦 𝑣 + 𝐼𝑡 ) − 𝛼 2 Δ𝑣 = 0
𝜕2 𝜕2
where Δ = 2 + 2 is the Laplace operator
𝜕𝑥 𝜕𝑦
18
a. Original scene b. Same scene later
19
Static Scene Object Detection and
Tracking
• Model the background and subtract to obtain
object mask
• Filter to remove noise
• Group adjacent pixels to obtain objects
• Track objects between frames to develop
trajectories
20
Approaches to Background Modeling
• Background Subtraction
• Statistical Methods
(e.g., Gaussian Mixture Model, Stauffer and Grimson 2000)
Background Subtraction:
1. Construct a background image B as average of few
images
2. For each actual frame I, classify individual pixels as
foreground if |B-I| > T (threshold)
3. Clean noisy pixels
21
22
Background Subtraction
23
Statistical Methods
24
Gaussian Mixture Model
200
150
100
B
50
0
250
Temple1 - RG Distribution of Pixel 172x165 200
250
200
250 150
150
G 100 100
R
200
150
G
100
50
R-G-B Distribution
0
0 50 100 150 200 250
R-G Distribution R
25
Detection of Moving Objects Based on
Local Variation
26
Dynamic Distribution Learning and Outlier
Detection
f (t ) − mean(t − 1)
(1)
C1 Detect Outlier
std (t − 1)
f (t ) − mean(t − 1)
(2) C2 Switch to a nominal state
std (t − 1)
27
Moving object tracking
• Model of object motion
• Kalman filter
• Mean Shift
Model based tracking
• Mathematical model of objects’ motions:
‒ position, velocity (speed, direction), acceleration
• Can predict objects’ positions
Motion
Model
Predict
Update? Location
Verify
Location
29
Simple Motion Model
• Newton’s laws
1
s(t ) = s + ut + a t
0
2
2
• s = position
• u = velocity
• a = acceleration
‒ all vector quantities
‒ measured in image co-ordinates
30
Prediction
• Can predict position at time t knowing
‒ Position
‒ Velocity
‒ Acceleration
• At t=0
31
Uncertainty
• If some error in a - Da or u - Du
• Then error in predicted position - Ds
Ds(t ) = s0 + Dut + Da t 2
1
2
32
Verification
• Is the object at the predicted location?
‒ Matching
• How to decide if object is found
‒ Search area
• Where to look for object
33
Object Matching
• Compare
‒ A small bitmap derived from the object vs.
‒ Small regions of the image
• Matching?
‒ Measure differences
34
Search Area
• Uncertainty in knowledge of model parameters
‒ Limited accuracy of measurement
‒ Values might change between measurements
• Define an area in which object could be
‒ Centred on predicted location, s Ds
35
Update the Model
• Is the object at the predicted location?
• Yes
‒ No change to model
• No
‒ Model needs updating
‒ Kalman filter is a solution
• Mathematically rigorous methods of using uncertain
measurements to update a model
36
Kalman filter
• Matrix description of system state, model and
measurement
• Progressive method
prediction
correction
• Proper dealing with noise
Kalman filter
• Advantages of using KF in particle tracking
• Progressive method
‒ No large matrices has to be inverted
• Proper dealing with system noise
• Track finding and track fitting
• Detection of outliers
• Merging track from different segments
Kalman filter assumptions
• Linear system
‒ System parameters are linear
function of parameters at some
previous time
‒ Measurements are linear function
of parameters
• White Gaussian noise
KF is the
‒ White: uncorrelated in time
optimal filter
‒ Gaussian: noise amplitude
Kalman filter
• Relates
‒ Measurements y[k]
• e.g. positions
‒ System state x[k]
• Position, velocity of object, etc
‒ Observation matrix H[k]
• Relates system state to measurements
‒ Evolution matrix A[k]
• Relates state of system between epochs
‒ Measurement noise n[k]
‒ Process noise v[k]
42
Mathematically
43
Prediction of System State
• Relates system states at epochs k and k+1
44
Prediction of Observation
• Predict/estimate a measurement yˆ k + 1 | k
• Measurement
yk + 1
• Predict state of model
• How does the new measurement xˆ k + 1 | k
contribute to updating the model?
46
Updating the Model
xˆ k + 1 | k + 1 = xˆ k + 1 | k + Gk + 1Dyk + 1 | k
Dyk + 1 | k = yk + 1 − yˆ k + 1 | k
• G is Kalman Gain
‒ Derived from A, H, v, n.
(
G = Ck | k H HC k | k H + n
T T −1
)
Ck + 1 | k = Ck | k − GHCk | k
C = system covariance
47
Example
• Tracking two corners of a minimum bounding box
• Matching using colour
• Image differencing to locate target
48
Condensation Tracking
• So far considered single motions
• What if movements change?
‒ Bouncing ball
‒ Human movements
• Use multiple models
‒ plus a model selection mechanism
49
Selection and Tracking
• Occur simultaneously
• Maintain
‒ A distribution of likely object positions plus weights
• Predict
‒ Select N samples, predict locations
• Verify
‒ Match predicted locations against image
‒ Update distributions
50
Tracking Using Hidden Markov
Models
• Hidden Markov model describes
‒ States occupied by a system
‒ Possible transitions between states
‒ Probabilities of transitions
‒ How to recognise a transition
51
Mean-shift
Center of
mass
Mean Shift
vector
Objective : Find the densest region
Distribution of identical billiard balls
Stolen from: www.wisdom.weizmann.ac.il/~deniss/vision_spring04/files/mean_shift/mean_shift.ppt
Mean Shift Vector
• Given:
Data points and approximate location of the mean of
this data:
• Task:
Estimate the exact location of the mean of the data
by determining the shift vector from the initial mean.
Mean Shift Vector
Mean-Shift Object Tracking
Target Representation
0.35
0.3
0.25
Quantized
Probability
0.2
0.1
0.05
0
1 2 3 . . . m
color
0.3 0.25
0.25
Probability
0.2
Probability
0.2
0.15
0.15
0.1
0.1
0.05 0.05
0 0
1 2 3 . . . m 1 2 3 . . . m
color color
m m
q = qu u =1..m q u =1 p ( y ) = pu ( y )u =1..m p u =1
u =1 u =1
Similarity f ( y ) = f q , p ( y )
Function: Q is the target histogram,
P is the object histogram
Stolen from: www.cs.wustl.edu/~pless/559/lectures/lecture22_tracking.ppt (depends on location y)
Mean-Shift Object Tracking
Target Localization Algorithm
Start from
the position Search in the Find best
of the model model’s candidate by
in the current neighborhood maximizing a
frame in next frame similarity func.
q p ( y) f p ( y ) , q
Stolen from: www.cs.wustl.edu/~pless/559/lectures/lecture22_tracking.ppt
Mean Shift
• Mean-Shift in tracking task:
➢ track the motion of a cluster of interesting features.
• 1. choose the feature distribution to represent an object (e.g.,
color + texture),
• 2. start the mean-shift window over the feature distribution
generated by the object
• 3. finally compute the chosen feature distribution over the next
video frame
‒ Starting from the current window location, the mean-shift algorithm
will find the new peak or mode of the feature distribution, which
(presumably) is centered over the object that produced the color and
texture in the first place.
‒ In this way, the mean-shift window tracks the movement of the object
frame by frame.
Thank you for
your attention!