Computer Vision

Chapter 6: Motion detection and Tracking

1
Course Content
• Chapter 1. Introduction
• Chapter 2. Image formation, acquisition and digitization
• Chapter 3. Image Processing
• Chapter 4. Feature detection and matching
• Chapter 5. Segmentation
• Chapter 6. Motion object detection and tracking
• Chapter 7. Object recognition and deep learning
Motion Detection and Tracking
• Motion detection
• Approaches of motion detection
• Moving object tracking

3
Motion Detection
• Action of sensing physical movement in a given area
• Motion can be detected by measuring change in speed or vector
of an object
• Goals of motion detection
‒ Identify moving objects
‒ Detection of unusual activity patterns
‒ Computing trajectories of moving objects

• Applications of motion detection

‒ Indoor/outdoor security
‒ Real time crime detection
‒ Traffic monitoring
‒ Many intelligent video analysis systems are based on motion detection

4
Approaches to Motion Detection
• Optical Flow
‒ Compute motion within region or the frame as a whole

• Change detection
‒ Detect objects within a scene
‒ Track object across a number of frames

5
Optical flow vs. motion field
• Optical flow does not always correspond to motion field

• Optical flow is an approximation of the motion field

• The error is small at points with high spatial gradient
under some simplifying assumptions
 Estimating optical flow

• Assume the image intensity I is constant

Time = t Time = t+dt

( x, y )
( x + dx, y + dy )
𝐼0 (𝑥, 𝑦, 𝑡) ≈ 𝐼1 (𝑥 + 𝛿𝑥, 𝑦 + 𝛿𝑦, 𝑡 + 𝛿𝑡)
Optical flow constraint
𝐼(𝑥, 𝑦, 𝑡) ≈ 𝐼(𝑥 + 𝛿𝑥, 𝑦 + 𝛿𝑦, 𝑡 + 𝛿𝑡)
𝜕𝐼 𝜕𝐼 𝜕𝐼
𝐼(𝑥, 𝑦, 𝑡) ≈ 𝐼(𝑥, 𝑦, 𝑡) + 𝛿𝑥 + 𝛿𝑦 + 𝛿𝑡
𝜕𝑥 𝜕𝑦 𝜕𝑡

𝜕𝐼 𝛿𝑥 𝜕𝐼 𝛿𝑦 𝜕𝐼
+ + = 0, and let 𝛿𝑡 → 0
𝜕𝑥 𝛿𝑡 𝜕𝑦 𝛿𝑡 𝜕𝑡
𝐼𝑥 𝑢 + 𝐼𝑦 𝑣 + 𝐼𝑡 = 0
𝐼𝑥 𝑢 + 𝐼𝑦 𝑣 = −𝐼𝑡 ,
𝑢
𝐼𝑥 𝐼𝑦 = −𝐼𝑡 , ∇𝐼 𝑇 𝐮 = 𝐛; 𝐴𝐮 = 𝐛
𝑣
𝐸(𝑢, 𝑣) = (𝐼𝑥 𝑢 + 𝐼𝑦 𝑣 + 𝐼𝑡 )2
Lucas-Kanade algorithm
Assume a single velocity for all pixels within an image patch

𝐸(𝑢, 𝑣) = ෍ (𝐼𝑥 (𝑥, 𝑦)𝑢 + 𝐼𝑦 (𝑥, 𝑦)𝑣 + 𝐼𝑡 )2

𝑥,𝑦∈Ω

Minimizing 𝐴𝐮 − 𝐛 2
𝐮 = (𝐴𝑇 𝐴)−1 𝐴𝑇 𝐛
෍ 𝐼𝑥2 ෍ 𝐼𝑥 𝐼𝑦 ෍ 𝐼𝑥 𝐼𝑡
𝑢
=−
𝑣
Hessian ෍ 𝐼𝑥 𝐼𝑦 ෍ 𝐼𝑦2 ෍ 𝐼𝑦 𝐼𝑡

(෍ ∇𝐼 ⋅ ∇𝐼 𝑇 ) →
𝑢 = − ෍ ∇𝐼 ⋅ 𝐼𝑡

LHS: sum of the 2x2 outer product of the gradient vector

 Matrix form
𝐸(𝐮 + Δ𝐮) = ෍ 𝐼1 (𝐱 𝑖 + 𝐮 + Δ𝐮) − 𝐼𝑜 (𝐱 𝑖 ) 2

𝑖
2
≈ ෍ 𝐼1 (𝐱 𝑖 + 𝐮) + 𝐉1 (𝐱 𝑖 + 𝐮)Δ𝐮 − 𝐼𝑜 (𝐱 𝑖 )
𝑖
2
= ෍ 𝐉1 (𝐱 𝑖 + 𝐮)Δ𝐮 + 𝑒𝑖
𝑖
𝜕𝐼1 𝜕𝐼1
𝐉1 (𝐱 𝑖 + 𝐮) ≈ ∇𝐼1 (𝐱 𝑖 + 𝐮) = , (𝐱 𝑖 + 𝐮)
𝜕𝑥 𝜕𝑦
𝑒𝑖 = 𝐼1 (𝐱 𝑖 + 𝐮) − 𝐼𝑜 (𝐱 𝑖 )
Matrix form
𝐴Δ𝐮 = 𝐛
𝐴 = ෍ 𝐉1𝑇 (𝐱 𝑖 + 𝐮)𝐉1 (𝐱 𝑖 + 𝐮)
𝑖

𝐛 = − ෍ 𝑒𝑖 𝐉1 (𝑥𝑖 + 𝑢)
𝑖

෍ 𝐼𝑥2 ෍ 𝐼𝑥 𝐼𝑦 ෍ 𝐼𝑥 𝐼𝑡
𝐴= , 𝐛=−
෍ 𝐼𝑥 𝐼𝑦 ෍ 𝐼𝑦2 ෍ 𝐼𝑦 𝐼𝑡

𝜕𝐼1 𝜕𝐼1
𝐉1 (𝐱 𝑖 + 𝐮) ≈ ∇𝐼1 (𝐱 𝑖 + 𝐮) = , (𝐱 𝑖 + 𝐮)
𝜕𝑥 𝜕𝑦
𝑒𝑖 = 𝐼1 (𝐱 𝑖 + 𝐮) − 𝐼𝑜 (𝐱 𝑖 )

𝐉1 (𝐱 𝑖 + 𝐮) ≈ 𝐉0 (𝐱 𝑖 )

for computational efficiency

Computing gradients in X-Y-T
y
1
𝐼𝑥 = [(𝐼𝑖+1,𝑗,𝑘 + 𝐼𝑖+1,𝑗,𝑘+1 + 𝐼𝑖+1,𝑗+1,𝑘 + 𝐼𝑖+1,𝑗+1,𝑘+1 )
4𝛿𝑥 time
−(𝐼𝑖,𝑗,𝑘 + 𝐼𝑖,𝑗,𝑘+1 + 𝐼𝑖,𝑗+1,𝑘 + 𝐼𝑖,𝑗+1,𝑘+1 )]

j+1
likewise for Iy and It
k+1

k
x
i i+1
The aperture problem
෍ 𝐼𝑥 𝐼𝑡
Let 𝐴 = ෍ ∇𝐼 ⋅ ∇𝐼 𝑇 , and 𝐛 = −
෍ 𝐼𝑦 𝐼𝑡

• Algorithm: At each pixel compute u by solving 𝐴𝐮 = 𝐛

• A is singular if all gradient vectors point in the
same direction
‒ e.g., along an edge
‒ of course, trivially singular if the summation is over a
single pixel or there is no texture
‒ i.e., only normal flow is available (aperture problem)
• Corners and textured areas are OK
Error functions
• Robust error function
𝑥2
𝐸(𝐮) = ෍ 𝜌(𝐼(𝐱 𝑖 + 𝐮) − 𝐼(𝐱)), 𝜌(𝑥) =
1 + 𝑥 2 /𝑎2
𝑖
• Spatially varying weights

2
𝐸(𝐮) = ෍ 𝑤0 (𝐱 𝑖 )𝑤1 (𝐱 𝑖 + 𝐮) 𝐼(𝐱 𝑖 + 𝐮) − 𝐼(𝐱 𝑖 )
𝑖
• Bias and gain: images taken with different exposure
𝐼(𝐱 + 𝐮) = (1 + 𝛼)𝐼(𝐱) + 𝛽, 𝛼 is the gain and 𝛽 is the bias
2
𝐸(𝐮) = ෍ 𝐼(𝐱 𝑖 + 𝐮) − (1 + 𝛼)𝐼(𝐱 𝑖 ) − 𝛽
𝑖

• Correlation (and normalized cross correlation)

E (u) =  I (xi ) I (xi + u)
i
Horn-Schunck algorithm
• Global method with smoothness constraint to solve
aperture problem
• Minimize a global energy functional with calculus of
variations
2
𝐸=න 𝐼𝑥 𝑢 + 𝐼𝑦 𝑣 + 𝐼𝑡 + 𝛼 2 ∇𝑢 2
+ ∇𝑣 2
𝑑𝑥𝑑𝑦

𝜕𝐿 𝜕 𝜕𝐿 𝜕 𝜕𝐿
− − =0
𝜕𝑢 𝜕𝑥 𝜕𝑢𝑥 𝜕𝑦 𝜕𝑢𝑦
𝜕𝐿 𝜕 𝜕𝐿 𝜕 𝜕𝐿
− − =0
𝜕𝑣 𝜕𝑥 𝜕𝑣𝑥 𝜕𝑦 𝜕𝑣𝑦
where 𝐿 is the integrand of the energy function
 Horn-Schunck algorithm
𝐼𝑥 (𝐼𝑥 𝑢 + 𝐼𝑦 𝑣 + 𝐼𝑡 ) − 𝛼 2 Δ𝑢 = 0
𝐼𝑦 (𝐼𝑥 𝑢 + 𝐼𝑦 𝑣 + 𝐼𝑡 ) − 𝛼 2 Δ𝑣 = 0
𝜕2 𝜕2
where Δ = 2 + 2 is the Laplace operator
𝜕𝑥 𝜕𝑦

Δ𝑢(𝑥, 𝑦) = 𝑢(𝑥,lj 𝑦) − 𝑢(𝑥, 𝑦)

(𝐼𝑥2 + 𝛼 2 )𝑢 + 𝐼𝑥 𝐼𝑦 𝑣 = 𝛼 2 𝑢lj − 𝐼𝑥 𝐼𝑡
𝐼𝑥 𝐼𝑦 𝑢 + (𝐼𝑦2 + 𝛼 2 )𝑣 = 𝛼 2 𝑣lj − 𝐼𝑦 𝐼𝑡
Horn-Schunck algorithm
(u , v )
• Iterative scheme v

• Yields high density flow

• Fill in missing information in the homogenous
regions
• More sensitive to noise than local methods
Background Subtraction
• Uses a reference background image for
comparison purposes.
• Current image (containing target object) is
compared to reference image pixel by pixel.
• Places where there are differences are detected
and classified as moving objects.

Motivation: simple difference of two images

shows moving objects

18
 a. Original scene b. Same scene later

Subtraction of scene a from scene b Subtracted image with threshold of 100

19
Static Scene Object Detection and
Tracking
• Model the background and subtract to obtain
object mask
• Filter to remove noise
• Group adjacent pixels to obtain objects
• Track objects between frames to develop
trajectories

20
Approaches to Background Modeling

• Background Subtraction

• Statistical Methods
(e.g., Gaussian Mixture Model, Stauffer and Grimson 2000)

Background Subtraction:
1. Construct a background image B as average of few
images
2. For each actual frame I, classify individual pixels as
foreground if |B-I| > T (threshold)
3. Clean noisy pixels

21
22
Background Subtraction

Background Image Current Image

23
Statistical Methods

• Pixel statistics: average and standard deviation of

color and gray level values

• Gaussian Mixture Model

‒ Model the color values of a particular pixel as a mixture of
Gaussians
‒ Multiple adaptive Gaussians are necessary to cope with
acquisition noise, lighting changes, etc.
‒ Pixel values that do not fit the background distributions
(Mahalanobis distance) are considered foreground

24
 Gaussian Mixture Model

Block 44x42 Pixel 172x165

Temple1 - RGB Distribution of Pixel 172x165

200

150

100

B
50

0
250
Temple1 - RG Distribution of Pixel 172x165 200
250
200
250 150
150
G 100 100
R

200

150
G

100

50
R-G-B Distribution

0
0 50 100 150 200 250
R-G Distribution R

25
Detection of Moving Objects Based on
Local Variation

For each block location (x,y) in the video plane

• Consider texture vectors in a symmetric window [t-W,
t+W] at time t
• Compute the covariance matrix
• Motion measure is defined as
the largest eigenvalue of the covariance matrix

26
Dynamic Distribution Learning and Outlier
Detection
f (t ) − mean(t − 1)
(1)
 C1 Detect Outlier
std (t − 1)

f (t ) − mean(t − 1)
(2)  C2 Switch to a nominal state
std (t − 1)

(3) mean(t ) = u  mean(t − 1) + (1 − u )  f (t )

Update the estimates of mean and standard
deviation only when the outliers are not
(4)
std (t ) =  2 (t ) detected

(5)  2 (t ) = u   2 (t − 1) + (1 − u )  ( f (t ) − mean(t − 1)) 2

27
Moving object tracking
• Model of object motion
• Kalman filter
• Mean Shift
Model based tracking
• Mathematical model of objects’ motions:
‒ position, velocity (speed, direction), acceleration
• Can predict objects’ positions
Motion
Model

Predict
Update? Location

Verify
Location

29
Simple Motion Model

• Newton’s laws
1
s(t ) = s + ut + a t
0
2

2
• s = position
• u = velocity
• a = acceleration
‒ all vector quantities
‒ measured in image co-ordinates

30
Prediction
• Can predict position at time t knowing
‒ Position
‒ Velocity
‒ Acceleration
• At t=0

31
Uncertainty
• If some error in a - Da or u - Du
• Then error in predicted position - Ds

Ds(t ) = s0 + Dut + Da t 2
1
2

32
Verification
• Is the object at the predicted location?
‒ Matching
• How to decide if object is found
‒ Search area
• Where to look for object

33
Object Matching
• Compare
‒ A small bitmap derived from the object vs.
‒ Small regions of the image
• Matching?
‒ Measure differences

34
Search Area
• Uncertainty in knowledge of model parameters
‒ Limited accuracy of measurement
‒ Values might change between measurements
• Define an area in which object could be
‒ Centred on predicted location, s  Ds

35
Update the Model
• Is the object at the predicted location?
• Yes
‒ No change to model
• No
‒ Model needs updating
‒ Kalman filter is a solution
• Mathematically rigorous methods of using uncertain
measurements to update a model

36
Kalman filter
• Matrix description of system state, model and
measurement
• Progressive method

prediction

correction
• Proper dealing with noise
Kalman filter
• Advantages of using KF in particle tracking
• Progressive method
‒ No large matrices has to be inverted
• Proper dealing with system noise
• Track finding and track fitting
• Detection of outliers
• Merging track from different segments
Kalman filter assumptions

• Linear system
‒ System parameters are linear
function of parameters at some
previous time
‒ Measurements are linear function
of parameters
• White Gaussian noise
 KF is the
‒ White: uncorrelated in time
optimal filter
‒ Gaussian: noise amplitude
Kalman filter
• Relates
‒ Measurements y[k]
• e.g. positions
‒ System state x[k]
• Position, velocity of object, etc
‒ Observation matrix H[k]
• Relates system state to measurements
‒ Evolution matrix A[k]
• Relates state of system between epochs
‒ Measurement noise n[k]
‒ Process noise v[k]

42
Mathematically

How do observations relate to model?

yk  = Hk xk  + nk 

 xk 
   xk 
 x k  1 0 0 0   + nk 
=
 yk  0 0 1 0  yk 
   
 
 y k 

43
Prediction of System State
• Relates system states at epochs k and k+1

xˆ k + 1 | k  = Ak xk | k  + vk 

 xˆk + 1 | k  1 T 0 0  xk | k 
 xˆk + 1 | k   0  
1 0 0   xk | k 
 = + vk 
 yˆ k + 1 | k  0 0 1 T   yk | k 
ˆ    
 y k + 1 | k   0 0 0 1  y k | k 

44
Prediction of Observation

• From predicted system state, estimate what

observation should occur:

yˆ k + 1 | k  = Hk xˆ k + 1 | k  + nk 

 xˆk + 1 | k 
 
 xk + 1 | k  1 0 0 0  xk + 1 | k 
ˆ ˆ

 yˆ k + 1 | k  = 0 0 1 0  yˆ k + 1 | k  + nk 
   
ˆ 
 y k + 1 | k 
45
Updating the Model

• Predict/estimate a measurement yˆ k + 1 | k 
• Measurement
yk + 1
• Predict state of model
• How does the new measurement xˆ k + 1 | k 
contribute to updating the model?

46
Updating the Model

xˆ k + 1 | k + 1 = xˆ k + 1 | k  + Gk + 1Dyk + 1 | k 
Dyk + 1 | k  = yk + 1 − yˆ k + 1 | k 
• G is Kalman Gain
‒ Derived from A, H, v, n.

(
G = Ck | k H HC k | k  H + n
T T −1
)
Ck + 1 | k  = Ck | k  − GHCk | k 
C = system covariance

47
Example
• Tracking two corners of a minimum bounding box
• Matching using colour
• Image differencing to locate target

48
Condensation Tracking
• So far considered single motions
• What if movements change?
‒ Bouncing ball
‒ Human movements
• Use multiple models
‒ plus a model selection mechanism

49
Selection and Tracking
• Occur simultaneously
• Maintain
‒ A distribution of likely object positions plus weights
• Predict
‒ Select N samples, predict locations
• Verify
‒ Match predicted locations against image
‒ Update distributions

50
Tracking Using Hidden Markov
Models
• Hidden Markov model describes
‒ States occupied by a system
‒ Possible transitions between states
‒ Probabilities of transitions
‒ How to recognise a transition

51
Mean-shift

• The mean-shift algorithm is an efficient approach to

tracking objects whose appearance is defined by
distributions (histograms). (not limited to only color)
• Motivation – to track non-rigid objects, (like a walking
person), it is hard to specify an explicit 2D parametric
motion model.

• Appearances of non-rigid objects can sometimes be

modeled with color distributions
Intuitive Description
Region of
interest

Center of
mass

Mean Shift
vector
Objective : Find the densest region
Distribution of identical billiard balls
Stolen from: www.wisdom.weizmann.ac.il/~deniss/vision_spring04/files/mean_shift/mean_shift.ppt
Mean Shift Vector
• Given:
Data points and approximate location of the mean of
this data:

• Task:
Estimate the exact location of the mean of the data
by determining the shift vector from the initial mean.
Mean Shift Vector
Mean-Shift Object Tracking
Target Representation

Choose a Represent the

Choose a model by its
reference
feature space PDF in the
target model
feature space

0.35

0.3

0.25
Quantized

Probability
0.2

Color Space 0.15

0.1

0.05

0
1 2 3 . . . m

color

Stolen from: www.cs.wustl.edu/~pless/559/lectures/lecture22_tracking.ppt

Mean-Shift Object Tracking
PDF Representation

Target Model Target Candidate

(centered at 0) (centered at y)
0.35 0.3

0.3 0.25
0.25
Probability

0.2

Probability
0.2
0.15
0.15
0.1
0.1

0.05 0.05

0 0
1 2 3 . . . m 1 2 3 . . . m
color color

m m
q = qu u =1..m q u =1 p ( y ) =  pu ( y )u =1..m p u =1
u =1 u =1

Similarity f ( y ) = f  q , p ( y ) 
Function:   Q is the target histogram,
P is the object histogram
Stolen from: www.cs.wustl.edu/~pless/559/lectures/lecture22_tracking.ppt (depends on location y)
Mean-Shift Object Tracking
Target Localization Algorithm

Start from
the position Search in the Find best
of the model model’s candidate by
in the current neighborhood maximizing a
frame in next frame similarity func.

q p ( y) f  p ( y ) , q 
Stolen from: www.cs.wustl.edu/~pless/559/lectures/lecture22_tracking.ppt
Mean Shift
• Mean-Shift in tracking task:
➢ track the motion of a cluster of interesting features.
• 1. choose the feature distribution to represent an object (e.g.,
color + texture),
• 2. start the mean-shift window over the feature distribution
generated by the object
• 3. finally compute the chosen feature distribution over the next
video frame
‒ Starting from the current window location, the mean-shift algorithm
will find the new peak or mode of the feature distribution, which
(presumably) is centered over the object that produced the color and
texture in the first place.
‒ In this way, the mean-shift window tracks the movement of the object
frame by frame.
Thank you for
your attention!