JOURNAL OF LATEX CLASS FILES, VOL. 14, NO.

8, AUGUST 2021 1

Exploring the Random Impulses Vehicle Trajectory

Model for Dimensionality Reduction and Motion
Extraction from Aerial Videos
Thinh Hoang Dinh, Student Member, IEEE, Vincent Martinez, Pierre Maréchal, and Daniel Delahaye

Abstract—Intelligent Transportation Systems (ITS) hold great

potential for improving the safety, efficiency, and comfort of road
users. These applications heavily rely on processing trajectory
data, a significant component of which is the road vehicle trajec-
tory model. This paper presents a kinematic statistical model for MEAN TRAJECTORY
∫∫ TRAJECTORY
DATASET

(NGSIM 101)

RANDOM IMPULSES
large-scale ITS that is more suitable for batch processing of data, RANDOM IMPULSES TRAJECTORY MODEL

especially in systems with limited hardware capabilities such as

vehicular communication modems and embedded systems. Our
proposed model approximates slipping velocity and acceleration COVARIANCE KERNEL COVARIANCE KERNEL

as random impulses originating from stationary distributions,

simplifying to approximated forms. We verify the validity of
the model through analysis and comparison with the computed
covariance kernel using the standard NGSIM 101 dataset. EIGENFUNCTIONS EIGENFUNCTIONS
We present two applications of this model. First, our derived
Functional Principal Component Analysis (FPCA) basis allows
for effective dimensionality reduction, a key step for various MODEL VALIDATION
MACHINE LEARNING TASKS
machine learning tasks such as trajectory prediction and anomaly • ANOMALY DETECTION
detection. Second, we apply the model to filter and smooth • PREDICTION
• DATA COMPRESSION
trajectories captured from aerial videos. We achieve this through
DIMENSIONALITY REDUCTION
a combination of object detection using a deep neural network,
and a Two-Filter scheme, which includes a Gaussian Mixture
Kalman Filter (GMKF) for forward filtering, and a Backward FILTER & SMOOTHER
DESIGN
Information Filter for smoothing. We provide theoretical results
regarding the estimation of the dataset size required for reliable AERIAL VIDEO SMOOTHED TRAJECTORY
AND VELOCITY
estimation of the model’s mean trajectory. Our numerical results,
based on the NGSIM 101 dataset, demonstrate that highway
trajectories can be significantly reduced to vectors with very few Fig. 1: Overview of the paper.
dimensions for the first application. For the second application,
the combination of filtering and smoothing yields a smoothed
estimate of velocity with a confidence interval up to two times
smaller than that achieved by filtering alone. We believe that efficiency. Consequently, processing this data is crucial for
the proposed model exhibits potential for application within the realizing intelligent transportation, as its successful analysis
broader context of large-scale traffic systems. Its simplicity and could assist in traffic congestion control [1], promote au-
low computational complexity make it suitable for implementing
stringent-constrained data processing at the edges. tonomous driving [2], and enhance passenger comfort. Nu-
merous trajectory models have been proposed in the litera-
Index Terms—ITS, road trajectories, Random Impulses model,
ture, including car-following and lane-changing models [3],
FPCA, motion extraction, Kalman Filter, Information Filter.
[4], in addition to deep learning generative models [5]–[7].
Termed as interaction-aware [8], these models incorporate
I. I NTRODUCTION inter-vehicle interactions, often requiring substantial compu-

T HE newest generation of vehicles and roadside units, tational resources. They are ideally suited for tasks such as
equipped with sensors and communication systems, gen- path optimization and collision avoidance, where performance
erate a substantial amount of data, which holds vast potential evaluation is primarily assessed from the standpoint of each
for improving vehicular safety and urban traffic infrastructure individual vehicle. Conversely, kinematic models, despite re-
Thinh Hoang Dinh is with Artificial and Natural Intelligence Toulouse
ceiving less attention, offer an alternative approach. While
Institute (ANITI), Université de Toulouse, Toulouse, France. these models are typically simpler due to their lack of explicit
Vincent Martinez is with NXP Semiconductors Toulouse, Toulouse, France. modeling of inter-vehicle interactions, their application to col-
Pierre Maréchal is with Toulouse Institute of Mathematics, Toulouse,
France.
lective tasks, such as optimizing data transmission in vehicular
Daniel Delahaye is with École Nationale de l’Aviation Civile, Toulouse, networks or batch trajectory smoothing at scale, often leads to
France. converging average performance with interaction-aware mod-
Manuscript received April 19, 2021; revised August 16, 2021. els. For instance, when designing a vehicular communication
modem,
0000–0000/00$00.00 predicting all surrounding vehicle trajectories using
© 2021 IEEE
JOURNAL OF LATEX CLASS FILES, VOL. 14, NO. 8, AUGUST 2021 2

an optimality solver that accounts for all relative distances implementation reference for these feature trackers by using
and velocities to optimize transmission parameters may not the GMKF for the lateral component. We go a step further
be necessary. In this particular scenario, a simpler model and propose a smoother for offline processing of video data,
performing well on average would be adequate, despite the leading to better estimates thanks to the assimilation of future
possibility of its performance being less optimal for certain measurements compared to filtering-based solutions alone.
vehicles compared to others. The paper unfolds as follows: Section II reviews various
The latter model category is valuable for its ability to trajectory models and relevant literature on trajectory estima-
handle large-scale trajectory information. Distinguished by its tion from aerial videos. Section III introduces the “Random
simplicity, ease of use, and minimal hardware requirements, Impulses” Model and Section IV justifies its validity through
it is optimally suited for edge computing applications, such the analytical derivation of the covariance kernel. Section V
as in smart traffic lights or vehicular communication modems. expounds on the model’s eigenfunctions for trajectory dimen-
In the emerging landscape of urban ITS, such as Vehicle-to- sionality reduction, a crucial component for anomaly detection
Everything (V2X) networks and smart road infrastructure, the in Advanced Driver Assistance Systems (ADAS). Section VI
accessibility of such a model becomes crucial. It can consid- showcases the model’s secondary application in filtering and
erably enhance the design of safe and efficient applications, smoothing trajectory data from aerial videos. The seventh
underlining its transformative potential in this rapidly evolving section that follows will provide various discussions around
field. the assumptions and the results, and the paper concludes with
This paper presents the “Random Impulses” trajectory Section VII.
model, tailored to meet the specific requirements of a kine-
matic statistical model in batch data processing contexts. At
II. R ELATED W ORK
its core, this model perceives individual vehicle trajectories
as variations of a “mean trajectory”. This mean trajectory Lefevre et al. [8] categorized trajectory models into three
represents the safety distance-keeping by drivers, coupled with types: physics-based, maneuver-based, and interaction-aware.
a slow-varying random component that captures the influence Interaction-aware models, including car-following and lane-
of various factors considered to be Markovian. This approach changing models, define motion as functions of surrounding
simplifies the interactive effects exerted by surrounding ve- vehicles’ movement [21]–[24]. Studies also considered ego-
hicles, while preserving the simplicity of the more basic motion due to multi-agent influences using Dynamic Bayesian
kinematic models that assume constant velocity or acceleration Networks [25]. Conversely, maneuver-based models presume
[9]–[11]. The model’s simplicity and computational efficiency a finite number of “maneuvers” that drivers can perform,
make it particularly attractive for deployment in large-scale typically learned from a clustering algorithm or modeled by a
ITS, especially those operating under hardware constraints. Gaussian Process [26], [27].
The considerable value of this model becomes more appar- Deep Generative models were extensively explored as well.
ent in the subsequent sections of this paper, where we demon- Chandra et al. [28] utilized a hybrid LSTM-CNN to model
strate its applications through various application frameworks road agents’ dynamics, shapes, and behaviors. Their model
related to the Karhunen-Loève Transform (KLT) - also known could predict multiple agent behaviors simultaneously. How-
as FPCA - and an approximated linear state-space model. The ever, the inherent fading gradient problem of RNNs led to the
KLT has strong ties to Principal Component Analysis (PCA), a introduction of the attention mechanism [29]. Messaoud et al.
technique that enjoys widespread use in the machine learning [30] incorporated global and partial attention to surrounding
community for data dimension reduction. The intricate nature vehicles into their model. Despite their success, these models
of the lateral impulses distribution necessitates the use of a were often considered too complex for trajectory data process-
Gaussian Mixture implementation for the filter and smoother. ing in systems with limited computational power.
Achieving these applications with car-following or constant Kinematic models, like those assuming constant velocity
velocity models would have posed substantial challenges, or acceleration [9]–[11], [31], align closest with the Random
which likely contributes to their limited implementation in Impulses model. These models are frequently chosen for appli-
edge processing facilities. This stark contrast underscores the cations involving Kalman filters but their utility for prediction
transformative contribution of the ”Random Impulses” model is limited due to disregarding road curvature and distance-
to the field. keeping tendencies.
Furthermore, the field of high-resolution trajectory extrac- While these models are useful for object tracking applica-
tion from aerial videos has also witnessed tremendous devel- tions, they provide deterministic trajectories without a confi-
opments, with a special mention of the NGSIM dataset [12]. dence interval. For instance, several trackers like SORT [17]–
This method has inspired other approaches, such as using [20] extensively used Kalman Filter and its variants. Yet, these
video streams from Unmanned Aerial Vehicles (UAVs) for models are suboptimal for batch offline applications. Chen
traffic monitoring [13]–[16]. Kalman and Particle Filters have et al. [15] attempted to smooth trajectories using wavelets
been widely used in vehicle tracking models such as SORT but without maintaining the same probabilistic framework for
[17] and its variants [18]–[20]. Nevertheless, the process tracking. Furthermore, like other methods, they didn’t provide
and measurement models employed in these trackers often confidence intervals.
employ random walk models without justification. This paper The Random Impulses model aims to enhance the kinematic
also provides the reasoning behind, and the corresponding models by integrating a component that mirrors self-regulatory
JOURNAL OF LATEX CLASS FILES, VOL. 14, NO. 8, AUGUST 2021 3

y
RAIM MODEL
P

∫∫
T3

T4 P2

y(t)
P1
MEAN TRAJECTORY RANDOM IMPULSES

RYIM MODEL T1

O
w(t) w

MEAN TRAJECTORY
∫ (a) (b)
Fig. 3: (a) Definition of the P -segment coordinate system.
Wt , Yt are relative coordinates, and absolute coordinates re-
RANDOM IMPULSES
quire road topology be taken into account. (b) An example
of two road segments P1 - a rectangle, and P2 - an L-shape
Fig. 2: An overview of the RAIM and RYIM Models for
road segments, which are shaded polygons at an intersection.
longitudinal and lateral components of the trajectory time
Here, both trajectories T1 and T2 are considered valid for the
series. The “mean” component represents the self-regulatory
road segment P1 , since vehicles enter and leave at the same
effect of distance-maintaining behavior for safety assurance.
places. However, for road segment P2 only the trajectory T3
The Random Impulses model the effects of various stochastic
is considered valid because T4 leaves P2 prematurely.
factors.

distance-keeping behavior, thereby bridging the gap between

kinematic and interaction-aware models.

III. R ANDOM I MPULSES M ODEL Fig. 4: All vehicles in the segment (blue) are considered to be
In this section, we formulate the Random Impulses Models following a virtual vehicle (green). If the following is exact,
for both the longitudinal and lateral components of the vehicle the separation between each vehicle is maintained, which is
trajectory. The former represents the “fast-slow” characteris- the optimal scenario.
tics of speeding, while the latter characterizes lane-changing
behaviors. For reasons that will become apparent, we will refer
to these models as the Random Acceleration Impulses Model a “virtual vehicle” in a “leisurely” manner. By “leisurely,” we
(RAIM) and the Random Yawing Impulses Model (RYIM), mean that the process of following is not exact. Nevertheless,
respectively (Fig. 2). Consider the trajectory of vehicle i, any vehicle whose motion deviates substantially from the
(i) (i) (i) trajectory of the virtual vehicle runs an increased risk of
denoted as Wt , Yt . Here, Wt specifies the vehicle’s
relative position with respect to the left roadside, and Yt
(i) colliding with other cars, as illustrated in Fig. 4. Formally, let
(i)
indicates the longitudinal distance the vehicle has traveled {Yt }N i=1 be the set of Nv longitudinal trajectories of vehicles
v

(i)
into the segment (Fig. 3a). We conceptualize each vehicle’s traveling on the road segment P , Yt ∈ RR+ . Let Y¯t be
trajectory as an independent realization of two corresponding the trajectory of the virtual vehicle. Statistically, we consider
(i)
random processes, Zt and Xt . These processes will be defined {Yt }N v
i=1 to be independent sample paths of a stochastic
in Equations (1) and (5). process:
{Xt , X(t, ω)}, (1)
A. Random Acceleration Impulses Model (RAIM) where the sample function is defined as:
Consider a road segment with several lanes, all directing ve-
hicles in the same direction. While vehicles might temporarily X(·, ω) : R+ → R,
stop within the segment due to, for example, a traffic light, we over the probability space (Ω, A, P ) indexed by t ∈ R+ . The
simplify our model by assuming all vehicles enter and exit the RAIM model assumes:
segment at the same location (Figure 3b).
To avoid a collision, the motion of any vehicle entering E [Xt ] = Y¯t ,
a road segment is influenced by nearby cars. Similarly, the
movement of these proximate vehicles also impacts the motion and the “residual deviation” of the trajectory is generated
of other neighboring cars. This interaction results in a complex from doubly integrating path-wisely the random acceleration
set of interlocking equations describing vehicle relationships impulses model At :
that defy an easy solution. However, for simplicity, let’s Z t Z s
consider all vehicles in a given road segment as following Xt = E [Xt ] + Aτ dτ ds. (2)
s=0 τ =0
JOURNAL OF LATEX CLASS FILES, VOL. 14, NO. 8, AUGUST 2021 4

Acceleration
0.05
State

0.00
0.05
0
0 20 40 60 80 100 120 0 50 100 150 200
Timestep Timestep

Fig. 5: One sample path of Lt with renewal intervals I0,k Fig. 6: One sample path of At with renewal intervals I0,k given
given by Γ(α = 5, β = 0.5) and I1,k given by Γ(α = by Γ(α = 5, β = 0.5), I1,k given by Γ(α = 30, β = 0.5) and
30, β = 0.5). Lt = 1 marks the period where there is Xk ∼ N (0, 0.052 ).
acceleration activity, and Lt = 0 otherwise. There are 11
regeneration points in this exemplary sample path located at
t ∈ {3, 22, 24, 46, 48, 72, 75, 85, 88, 100, 105}. y
0.8

Probability density
v
We now describe the process At . For each k ∈ {0, 1}, let 0.6
{Ik,i }i , i ∈ N∗ be a sequence of independent random variables 0.4
vy Θ
(with respect to i), called renewal intervals associated with the
state k, given by distribution functions: 0.2

Fk (u) = P (Ik,i ≤ u), k ∈ {0, 1}, i ∈ N∗ . O vx x 0.0

2 0 2
Slipping velocity
Equipped with two sequences of renewal intervals correspond-
ing to state k = 0 and k = 1, we define a sequence of (a) (b)
regeneration points corresponding to moments where there is Fig. 7: (a) Illustration of the yawing angle θ, the lateral
a jump from state 0 to state 1 and vice-versa: velocity vx and the longitudinal velocity vy . (b) Probability
density of the Slipping Velocity Distribution with three modes
WI = {ωi }, i ∈ N∗ ,
indicating changing to the left lane, lane keeping and changing
such that: to the right lane.
(
ωi−1 + I1, i+1 if i is odd,
ωi = 2

ωi−1 + I0, 2i if i is even. B. Random Yawing Impulses Model (RYIM)

This allows us to formally define a Markov renewal process We now shift our focus onto the lateral component of
Lt , which is the acceleration activity process Lt as follows: the trajectory. As before, we assume that the set of lateral
(i)
+∞
trajectories {Wt }N v
i=1 are independent path samples of a
X
stochastic process Zt . The RYIM model assumes:
Lt = 1 ωi ≤t≤ωi+1 × δ{i≡0[2]} . (3)
i=1 Z s=t
Informally, the acceleration activity process is composed of Zt = E [Zt ] + Bs ds. (5)
s=0
alternating periods of idleness and acceleration. The lengths
of these periods are determined by the renewal intervals of
Introduce the coordinate system as depicted in Fig. 7(a),
I0,· and I1,· respectively (Fig. 5). The RAIM represents this
where Oy is aligned with the centre marking. With a small
physical reality, suggesting that drivers following a virtual
yawing angle θ, it follows that vx ≈ vθ and vy ≈ v. The
vehicle but intermittently apply and release the gas or brake
dynamics of θ can be attributed to the driver’s steering wheel
pedals over random periods of time, at random times. This
input, and this gives a physical interpretation to the RYIM
process results in the vehicle’s motion deviating from that of
model. As a vehicle navigates a road, small adjustments to
the virtual vehicle.
the steering wheel by the driver keep the vehicle in its lane,
Building upon (3), the RAIM model further assumes that leading to θ oscillating around zero. Occasionally, a significant
acceleration impulses areidentically independent samples from shift in θ might be observed when the driver decides to change
a stationary distribution function Fa (u), resulting in the ac- lanes.
celeration impulses model At :
In comparison with the RAIM model, there are still 2 states
+∞
X {0, 1} which correspond to lane-keeping and lane-changing
At = Mi × 1 ωi ≤t≤ωi+1 × δ{i≡0[2] ), Mi ∼ Fa (u) respectively. The only difference is that Zt only has one
i=1 integration of the random impulses process rather than two,
(4)
as shown in (2).
A path sample is shown in Fig. 6. It is also worth mentioning
As above, the “slipping activity process” St is defined as:
that the RAIM model is a particular case of the more general
Piecewise Markov Process and the Markov Renewal Process. +∞
Other interesting properties, including expended and remain- X

St = 1 ωi ≤t≤ωi+1 × δ{i≡0[2]} . (6)
ing times in a segment, can be further found in [32]. i=1
JOURNAL OF LATEX CLASS FILES, VOL. 14, NO. 8, AUGUST 2021 5

where
Lat. velocity
0.5
0.0
0.5 X
0 100 200 300 400 500 Zt = E[Zt ] + Mk I1,k if ωi ≤ t ≤ ωi+1 ,
Timestep
1≤k≤b 2i c
Fig. 8: The slipping impulses generated by process Bt with
renewal intervals I0,k given by Γ(α = 30, β = 0.25), I1,k and
given by Γ(α = 150, β = 0.25) and Ri ∼ N (0, 0.52 ). X
Zt = E[Zt ] + Mk I1,k if ωi ≤ t ≤ ωi+1 .
1≤k≤b i+1
2 c
The slipping impulse model is then defined as
+∞
X We define Fπ (u) as the product distribution of Flc (u) and
Bt = Ri ×11ωi ≤t≤ωi+1 ×δ{i≡0[2]}
in which Ri ∼ Flc (u). Fk (u). Obviously Fπ (u) is stationary if the component distri-
i=1
(7) butions are stationary. Let Qk = Mk I1,k then Qk ∼ Fπ (u).
The distribution Flc dictates the behavior of drivers in terms The lower bound process could be rewritten into a more
of lane-keeping and lane-changing, while the distributions suggestive form:
I·,k manage the frequency and reaction rates of drivers. The X
sample paths of the slipping activity process, denoted as St , Zt = E[Zt ] + Qk , (9)
1≤k≤ζt
are shown in Fig. 5, and of Bt are presented in Fig. 8, with
its corresponding Flc showcased in Fig. 7(b).
where ζt is the the number of regenerations before t of the
process I0 + I1 , {I1,k + I0,k }k . In other words, the process
IV. VALIDATION OF THE M ODEL THROUGH C OMPUTATION
bounds are essentially renewal-reward processes, whose re-
OF THE C OVARIANCE K ERNEL AND ITS E IGENFUNCTIONS
ward function is Qk , and the holding times I1,k +I0,k indicate
The RAIM and RYIM models operate on the assumption the time between two consecutive renewals into state 1. Let
that the residual components of the trajectory time series, once ρ , ω
k 2k+1 , then {ρ1 , ρ2 , . . . } are the times when I0 + I1
the mean trajectories have been subtracted, originate from a regenerates. From the Key Renewal Theorem
Markov process. However, the validity of such assumptions
still needs verification. To address this, we analytically derive h
the covariance kernel of the processes Xt , Zt and the corre- E[ζτ +h ] − E[ζτ ] → as τ → +∞, (10)
µ
sponding eigenfunctions. The stationary process assumption
could be deemed valid if the convergence of the covariance where
function can be established via the law of large numbers.
µ = E[I0 ] + E[I1 ].
A. RYIM Model’s Covariance Kernel and Eigenfunctions Equation (10) implicitly assumes that the process has been
We study the time integration of Bt in Eq. (7). Analysis for running long enough. We also assume that the longer the run
the acceleration process (4) should be similar. time is, the more renewals into state 1 there will be, with
+∞
X probability one:
Bt = Ri × 1 ωi ≤t≤ωi+1 × δ{i≡0[2]} . (8)
i=1 ζt < ζs ⇒ t < s a.s. (11)
Proposition 1. The covariance kernel for the process Zt
defined in (5) converges to the covariance kernel for the Let t = τ + t0 , s = τ + s0 , we can calculate the covariance
Wiener process if the regeneration time I0 + I1 goes to zero. function of Zt :
Proof. The RYIM model assumes: Cov(t, s) = E[(Zt − E[Zt ])(Zs − E[Zs ])]
Z t
Zt= E[Zt ] + Bs ds = E[E[(Zt − E[Zt ])(Zs − E[Zs ])]|ζt , ζs ]
 
 0
X X
E[(Qi − E[Qi ])(Qj − E[Qj ])]
X


 E[Z t ] + Mk I1,k if ωi ≤ t ≤ ωi+1 and = E

 i 1≤i≤ζt 1≤j≤ζs


 1≤k≤ 2

 i is even; = E [min(ζt , ζs ) Var[Qi ]]
= X = E [min(ζτ +t0 , ζτ +s0 ) Var[Qi ]]
E[Zt ] +
 Mk I1,k + M i+1 (t − ωi ) if

 2 1

 1≤k≤ i−1
2
→ min(ρζt , ρζs ) Var[Qi ] (12)
µ



ω ≤ t ≤ ω
i i+1 and i is odd.
where we have used the law of total probability to go from the
We define Zt , Zt as the almost sure upper and lower bounds
second line to the third line, independence of Qi to go from
of Zt respectively:
the third line to the fourth, and (10) to go to the final line if
Zt ≤ Zt ≤ Zt a.s. t < s implies ζt < ζs (11).
JOURNAL OF LATEX CLASS FILES, VOL. 14, NO. 8, AUGUST 2021 6

0 = 0 / RSID
0

100 100
min(s+µ,t+µ) min(s+µ,t+µ)
200 200

t
300

t
300
min(s,t) min(s,t)
400 400

500 500
t t 100 0 100 200 300 400 500 600 100 0 100 200 300 400 500 600
ρ1 ρ2 ρ3 ρ4 ρ5 ρ6 ρ1 ρ2 ρ3 ρ4 ρ5 ρ6 s s

s s
(a) (b)
(a) (b) Fig. 10: (a) Computed covariance kernel from the NGSIM
Fig. 9: (a) depicts one instance of µ min(ζt , ζs ) Var[Qi ]|ζt , ζs , dataset (b) Predicted covariance kernel from the RYIM model.
with s fixed, as the blue graph. The black dots are regeneration
points. The expectation over all ζt , ζs , when τ → +∞ gives 0.08 1 1.00
2
0.06 3
the covariance function shown on (b). The lower bound of 4
0.75

Roadwise coordinate (m)

0.04 5 0.50
the process coincides with min(s, t), while the upper bound 0.02 0.25
0.00 0.00
coincides with min(s + µ, t + µ). Note the relative position 0.02 0.25
1
of the regeneration points with respect to the two bounds 0.04 0.50 2
3
0.06 0.75 4
min(s, t) and min(s + µ, t + µ). 1.00 5
0 100 200 300 400 500 0 100 200 300 400 500
Timestep (hundreth of a sec) t

(a) (b)
By definition, ρζt0 = t0 , Cov(t, s) =
(1/µ) min(t, s) Var[Qi ]. Similarly for Zt : Fig. 11: (a) KL basis obtained by numerically eigendecom-
posing the empirical covariance kernel (b) KL basis predicted
Cov (t, s) = E[(Zt − E[Zt ])(Zs − E[Zs ] by the RYIM model. Functions that are inverted vertically are
= E[E[(Zt − E[Zt ])(Zs − E[Zs ])]|ζt , ζs ] the same since the corresponding coefficients can be replaced
  with an opposite sign.
X X
= E E[(Qi − E[Qi ])(Qj − E[Qj ])]
1≤i≤ζt +1 1≤j≤ζs +1 the regeneration period of the slipping activity process not
= E [min(ζt + 1, ζs + 1) Var[Qi ]] being sufficiently small, and from the fact that there is a finite
= E [min(ζτ +t + 1, ζτ +s + 1) Var[Qi ]]
0 0 number of lanes for vehicles to change into. Nevertheless,
1 the structural similarity provides justification for the model’s
→ min(ρζt +µ , ρζs +µ ) Var[Qi ] (13) validity, as made more apparent in the respective visualiza-
µ
tions of their eigenfunctions bearing the harmonic shape with
and we also have Cov (t, s) = (1/µ) min(t+µ, s+µ) Var[Qi ]. similar frequencies (Fig. 11).
Figure 9 shows the covariance functions’ upper and lower
bounds. Obviously, if I0 + I1 regenerates fast enough, µ → 0 B. RAIM Model’s Covariance Kernel and Eigenfunctions
and we can approximate: The derivation of the covariance kernel and eigenfunctions
1 of RAIM is more involved.
Cov[(t, s)] ≈ min(t, s) Var[Qi ]. (14)
µ Proposition 2. Under the assumptions made in Proposition
1, the covariance kernel eigenfunctions are given by (23) and
the covariance kernel can be computed from (19).
Because the covariance kernel’s eigenfunctions are also the
KL basis [33], we will use the terms interchangeably. Proof. Consider the RAIM given by Eq. (2):
Z t Z s
Corollary 1. The KL decomposition for Zt is also the same Xt = E [Xt ] + (Au du) ds. (16)
as the Wiener process is given by: s=0 u=0

+∞ Restricting to t ∈ [0, 1], and since At and Bt are similar, we

√
  
X 1 can reuse Equation (15):
Zt = E[Zt ] + βk 2 sin k− πt . (15)
2 +∞  !
k=1 Z 1 √

X 1
where αk are normally distributed Gaussian random vari- Xt = E [Xt ] + Z0 + βk 2 sin k− πt dt.
0 2
k=1
ables. (17)
Proof. See [34]. Without any loss of generality, we can assume Z0 = 0. A
direct integration gives:
The model’s validity is confirmed by Fig. 10. There is +∞
−βk √
    
negligible cross-correlation between widely separated time
X 1
Xt = E [Xt ] + 2 cos k − πt − 1 .
steps, though it appears to be slightly more significant than k=1
k − 12 π 2
in the analytical kernel. This discrepancy might result from (18)
JOURNAL OF LATEX CLASS FILES, VOL. 14, NO. 8, AUGUST 2021 7

0 0
While (18) is a series expansion of Xt , it is far from being
100
the eigenfunctions (or the KLT). This is evident from the fact 100

that {cos ((k − 1/2) πt)−1}k do not even form an orthogonal 200 200

t
basis. To avoid cluttering of notations, we denote: 300
300

400
√
    
1 400
ek (t) = 2 cos k− πt − 1 . 500
2 100 0 100 200 300 400 500 600 100 0 100 200 300 400 500 600
s s

Then (18) can be rewritten as: (a) (b)

+∞
X −βk Fig. 12: The covariance kernels from (a) RAIM and (b)
Xt = E [Xt ] + ek (t).
k=1
k − 12 π NGSIM dataset.

The covariance function can be calculated directly as: 1 1

0.075
0.075 2 2
3 0.050 3
R(s, t) = E[(Xt − E[Xt ])(Xs − E[Xs ])] (19) 0.050 4
5 0.025
4
5
0.025
+∞ X
+∞ 0.000
0.000
X E[βk βl ] 0.025
= ek (t)el (s), (20) 0.025
0.050
(k − 1/2)2 π 2 0.050
0.075
k=1 l=1
0.075
0.100
and independence of βk leads to: 0 100 200
t
300 400 500 0 100 200
t
300 400 500

+∞
X E[βk2 ] (a) (b)
R(s, t) = ek (t)ek (s).
(k − 1/2)2 π 2 Fig. 13: (a) KL basis obtained from RAIM (b) KL basis
k=1

Using (15), we can further simplify the equation to: obtained numerically from NGSIM dataset. Note that some
components in (a) and (b) were inverted, but they are the
+∞
X 1 same since we can invert the sign of the corresponding score
R(s, t) = ek (t)ek (s). (21) component.
(k − 1/2)4 π 4
k=1

The KL basis is solution to the Fredholm equation:

Z 1
R(s, t)φ(t)dt = λφ(s), (22) +∞ +∞ +∞
X ci XX ei (s)
t=0 λ 2 2
ei (t) = Aij cj .
i=1
π (i − 1/2) i=1 j=1
(i − 1/2)2 π 2
which is a special case of the Fredholm integral equation of
the second kind with separable kernel. In general, there are (24)
infinitely many solutions corresponding to the zero eigenvalue Because ei (t) 6= 0, we can match the coefficients of ei (t):
of the integral operator of kernel R(s, t). But it is of our
interest to find solutions with the largest positive eigenvalues Ac = λc, (25)
which are written under the form: where A = [Aij ], c = [c1 , c2 , . . . ]> . Hence, an eigendecom-
+∞
X ci position of A should reveal the eigenvectors of coefficients c
φ(t) = ei (t). (23) and the KLT basis is given by (23). The coefficients of the
i=1
π 2 (i − 1/2)2
first 3 KL basis functions are given in Table I.
Substitution into (22) gives:
Z 1 X +∞
! c1 c2 c3 c4 c5 c6 c7
1 -9.99e-01 -3.21e-02 -1.78e-03 -8.45e-04 -2.05e-04 -1.28e-04 -5.03e-05
λφ(s) = ei (s)ei (t) -9.42e-01 3.30e-01 5.48e-02 4.71e-03 3.85e-03 9.49e-04 8.11e-04
t=0 i=1
(i − 1/2)4 π 4 -0.72 0.55 -0.39 -0.10 -0.01 -0.01 0
 
+∞
c TABLE I: 7 first coefficients of the first 3 KL basis to be used
j
X
 e (t) dt
2 (j − 1/2)2 j in Equation (23).
j=1
π
+∞ X
+∞ Z The striking similarity between the covariance kernels pre-
X ei (s) ei (s) ej (s)
= cj dt. dicted by RAIM and those numerically computed from the
(i − 1/2)2 π 2 (i − 1/2)2 π 2 (j − 1/2)2 π 2
i=1 j=1 datasets is readily observable (Fig. 12). Their eigenfunctions
Let: also display a remarkable resemblance (Fig. 13). These obser-
Z vations provide robust support for the assumptions presented
ei (s) ej (s) in Proposition 1.
Aij = ds
(i − 1/2)2 π 2 (j − 1/2)2 π 2
 
ei (s) ej (s) V. F UNCTIONAL PCA (FPCA) A PPLICATIONS
, , .
(i − 1/2)2 π 2 (j − 1/2)2 π 2
Dimensionality reduction, a cornerstone of machine learn-
Then: ing, promises innovative solutions in predictive modeling,
JOURNAL OF LATEX CLASS FILES, VOL. 14, NO. 8, AUGUST 2021 8

1.0 DIRECT KLT 1.0 DIRECT KLT

Normalized Mean Squared Error

Normalized Mean Squared Error

RAIM RYIM
0.8 DFT 0.8 DFT
FPCA DCT DCT
0.6 B-SPLINES 0.6 B-SPLINES

Fig. 14: In the decomposition of trajectories using FPCA, 0.4 0.4

each original time series is treated as a functional—equivalent 0.2 0.2
to a R150 vector in the discrete domain—as depicted by a
curve in the left graph. Each of these curves is subsequently 0.0 0.0
0 2 4 6 8 0 2 4 6 8
transformed into a 2D point, which can be thought of as Components Components
equivalent to a R2 vector. The horizontal axis is β1 , and the
other axis is β2 . (a) (b)
Fig. 15: Average N-terms basis restriction error of longitudinal
component (a) and lateral component (b). RAIM and RYIM
anomaly detection, and classification tasks. Among these symbolize the predicted basis functions from the RAIM and
methods, PCA stands out. Extensively utilized in fields like RYIM models respectively.
biology and engineering, PCA effectively mitigates the curse
of dimensionality, enhancing the performance of inference
80 140
methods in low-dimensional space. Original
It turns out that the eigenfunctions introduced in Eq. (15) 70 120 KLT

Position (mean subtracted)

Position (mean subtracted)

and (23) are also the basis functions required to perform 60 100
dimensionality reduction (Fig. 14) [35], [36]. In particular, let 50 80
f (t) be the time series that is either the longitudinal or lateral 40 60
component of the trajectory, and ek (t) representing the k-th 30 40
eigenfunction as denoted by (15) or (23). Then: 20 20
Z 10 Original 0
βk = ek (t)f (t)dt (26) 0 KLT
t 20
0 50 100 150 200 0 50 100 150 200
Then, given some N ∈ N , f (t) ∈ R∞ can be approximated
∗
Time Time
by a truncated series:
Fig. 16: Approximation with 9 components of the longitudinal
N
X component of two exemplary trajectories. The mean compo-
f (t) ≈ f˜N (t) , βk ek (t), (27)
nent was subtracted for better visualization.
k=1

in other words, f (t) can now be represented by a vector β =

[β1 , . . . , βN ] with a finite dimension of N . decomposition. However, according to Proposition 1, these
We illustrate multiple applications of this dimensionality re- basis functions remain invariant provided the prescribed as-
duction method, encompassing data compression, identifying sumptions are maintained.
anomalies, and completing trajectories, providing comparisons To demonstrate the approximation power, we compare the
whenever feasible. normalized mean squared error when performing the decom-
position using the analytical basis functions (KLT transform)
A. Dimensionality Reduction provided by (15) and (23), the basis functions computed
Definition 1 (N-terms basis restriction error and Normalized numerically from the learning set, and other harmonic linear
mean Squared Error). Let the N-terms approximated func- transforms such as Discrete Fourier Transform (DFT) and
tional of f (t) be given by (27), we define the N-terms basis Discrete Cosine Transform (DCT). We also include the Cubic
restriction error as: Splines Basis for reference due to its popularity, even though
Z +∞  2 B-splines is not typically considered a linear transform.
N = f (t) − f˜N (t) dt, (28) The remarkable efficacy of the RAIM and RYIM models
t=−∞ is clearly demonstrated in Fig. 15, where they exhibit the
and the related normalized mean squared error: quickest error convergence for the longitudinal component of
N the trajectories. Moreover, they rank second for the speediest
˜N = R +∞ (29) convergence on the lateral component. Although it may seem
−∞
f (t)2 dt
counterintuitive given that the KLT should offer the optimal
Then it has been shown in [37] that the FPCA decompo- convergence rate [37], we postulate that the convergence
sition whose score coefficients given by (26) minimize the conditions for the lateral component weren’t as perfectly met
N-terms basis restriction error. as they were for the longitudinal component, an assertion
Typically, in the traditional FPCA workflow, the basis backed by Fig. 11. It’s noteworthy to mention that this
functions must be computed from a sample set, referred to error convergence rate was even faster than when using the
as the learning dataset, and then applied to carry out the empirical basis numerically computed from the covariance
JOURNAL OF LATEX CLASS FILES, VOL. 14, NO. 8, AUGUST 2021 9

Original 2
0.0 KLT
POSTERIOR
0.2 1 DISTRIBUTION
0.4 PRIOR
Position

Position
DISTRIBUTION
0.6 0
0.8
1
1.0
Original
1.2 2 KLT
0 50 100 150 200 0 50 100 150 200
Time Time
Fig. 18: Illustration of the estimation process: the observations
Fig. 17: Approximation with 9 components of the lateral
provide information to “narrow down” the distribution of the
component of two exemplary trajectories.
value of a in the FPCA domain, which explains the more
“pointy” look of the posterior distribution.
kernel. This phenomenon can be attributed to the inherent
numerical errors and variances resulting from working with
a sampled training set. However, it’s anticipated that this “fill in the blanks” of the unobserved part of the longitudinal
discrepancy will diminish as more data is incorporated into trajectory, using partial observations obtained up to a specific
the training set. timestep.
Figures 16 and 17 demonstrate the approximation of trajec- Formally, let y1:n be the set of n noisy observations iden-
tories using only nine FPCA scores. It is immediately obvious tically and independently obtained from a vehicle’s trajectory
that FPCA is comparable to the role of DFT and DCT in whose measurement model is y = ŷ+y where y ∼ N (0, σy2 )
image processing where the truncated basis similarly reveals i
and D = {y1:q ; 1 ≤ i ≤ m} be the training set of m
a denoising ability. There’s a direct relationship between the trajectories whose length is q > n. We would like to find
number of components used in the transformation and the the best estimate for yn+1:q . Furthermore, let a ∈ RN be
smoothness of the trajectory - fewer components result in the corresponding FPCA score vector of the trajectory y1:q
smoother trajectories. computed from (26).
The general idea is to fit a distribution (presumably Gaus-
B. Data Compression sian in this example) on the FPCA score vectors corresponding
Drawing parallels with renowned digital data formats such to the trajectories of D, giving us the prior distribution
as MP3 and JPEG, the KLT carries immense potential akin to p(a) = N (0, Σa ). Then, we compute the posterior:
the DCT for data compression. This potential is particularly
n
evident in the context of vehicular network. For instance, p(y1:n |a)p(a) Y
p(a|y1:n ) = ∝a p(yi |a)p(a) (30)
instead of transmitting vehicle coordinates for each of the 150 p(y1:n ) i=1
time steps, it becomes feasible to transmit only 2-3 FPCA
score coefficients, resulting in a far more efficient scheme.
In the context that all distributions are Gaussians, the
Details can further be found in our previous work [38],
maximum-a-posteriori coincides with the minimum mean
where we successfully demonstrated the implementation of
square (MMSE) estimate of a is obtained from Algorithm 1.
such transform in devising a data compression strategy for the
Figure 19 provides an example of our attempt to complete a
Cooperative Awareness Message (CAM) transmission within
the V2X network setting.
The earlier implementation, however, was marred by prac- Algorithm 1 Trajectory Representation Estimation
tical challenges. Specifically, it required continuous collection 1: φ ← [ek [0], 1 ≤ k ≤ q]
of trajectories in the learning set and recurrent computation 2: µ0 ← 0q , [0, 0, . . . , 0] (q zeroes)
of the basis functions. This paper addresses these challenges 3: Σ0 ← Σa
and offers a significant improvement: we demonstrate that a 4: for i = 1 to n do
single, unique basis set can be utilized, which simplifies the 5: Σ0 ← (Σ−1 2
a + 1/σy (φi ⊗ φi ))
−1

process considerably. This development makes the approach 0 0 2 >

6: µ ← µ + 1/σy (yi − µi )φi
more amenable for hard coding on embedded platforms. 7: a∗ ← Σ0 µ0
8: end for
C. Trajectory Prediction and Completion
FPCA simplifies the dataset by reducing its dimensions, longitudinal trajectory. Remarkably, the prediction error stays
thereby facilitating a more intuitive and straightforward infer- within acceptable bounds for as long as 90 timesteps, or 9
ence process. To illustrate, we propose a Bayesian inference seconds. This duration notably surpasses the typical timeframe
framework as an example. This framework is designed to delivered by the Constant Velocity model.
JOURNAL OF LATEX CLASS FILES, VOL. 14, NO. 8, AUGUST 2021 10

where xk = [u, v, s, r, u̇, v̇, ṡ]> ; here, s and r symbolize the

Prediction

Coordinate (ft)
Ground truth scale, and u, v signify the horizontal and vertical coordinates
200 Observed
of the target center, respectively. However, the model makes
some questionable assumptions. For instance, the assumption
0
0 20 40 60 80 100 120 140 about the nature of the model being perturbed by random walk
Timestep (1/10s) noise is not adequately justified. Additionally, the characteri-
0 Prediction error zation of Q as a full-rank covariance matrix invites scrutiny
regarding its appropriateness.
Error (ft)

5 We now propose a novel, comprehensive filtering and

10 smoothing framework based on the previously validated RAIM
40 60 80 100 120 140 and RYIM models. Despite extensive research on filtering,
Timestep (1/10s)
smoothing has been relatively under-explored. To the best of
Fig. 19: The trajectory completion is illustrated for obser- our knowledge, this is the first study to suggest a smoothing
vations up to timestep 30. The upper graph presents the framework for this application.
completed trajectory, while the lower one displays the error
between the actual and predicted trajectories. A. Approximated Form for Lateral Component Dynamics
Let µx,k be the mean of the lateral component of the
1200 600 trajectory at time k. The RYIM model suggests the following
1000 approximated model:
Roadwise coordinate (ft)

400
800
FPCA score 2

200 xk+1 = xk + µx,k+1 − µx,k + ηx,k

600
yk = yk + ζx,k (32)
400 0
200 where:
200
0 ζx,k ∼i.i.d. N (0, σx2 )
0 50 100 4000 2000 0 2000 2 2 2
Timestep (1/10s) FPCA score 1 ηx,k ∼i.i.d. αl N (µv,l , σv,l ) + αc N (0, σv,c ) + αr N (µv,r , σv,r )

(a) (b) In other words, ηx,k samples from a Gaussian Mixture

composing of three modes corresponding to switching to
Fig. 20: (a) The speeding vehicle’s longitudinal component left lane, lane-keeping and switching to right lane (Fig. 7
(b) The corresponding FPCA scores. KNN could easily single (b)), rather than from a unimodal Gaussian distribution. By
out the FPCA representation of this speeding vehicle from the definition:
rest. E[ηx,k ] = 0 ⇒ µv,l = −µv,r (33)
and
D. Anomaly Detection αl + αc + αr = 1 (34)
Anomaly detection tasks are also straightforward as algo-
rithms such as K-nearest-neighbor (KNN) and clustering can
B. Lateral Component Filtering
be effectively performed on this domain (Fig. 20). While this
concept has been partially explored in previous studies like In the following, the filtering expressions for the particular
[39], [40], the utilization of the same basis functions for case of RYIM model will be presented. For a more general
different traffic scenarios to decompose trajectories is a novel case of Gaussian Mixture process and measurement models,
finding. This feature, akin to data compression applications, we refer to [41]. Recall the fundamental equations of the
allows us to bypass the learning process, favoring a hard-coded Bayesian filtering problem are:
transformation, making these results more impactful. p(yk |xk )p(xk |y1:k−1 )
p(xk |y1:k ) = (35)
p(yk |y1:k−1 )
VI. F ILTERING AND S MOOTHING OF V EHICLE Z
T RAJECTORIES E XTRACTED FROM A ERIAL V IDEOS p(xk+1 |y1:k ) = p(xk+1 |xk )p(xk |y1:k )dxk (36)
In the domain of extracting high-resolution vehicle tra-
The equations can be developed into a recursive algorithm,
jectories, established tracking frameworks such as the Sim-
with q being the length of the trajectory and p(x1 ) being the
ple Online and Realtime Tracking (SORT) [17] commonly
prior distribution of the vehicle’s lateral position. The algo-
employ methods such as Kalman or Particle Filter for data
rithm consists of two steps: in the measurement assimilation
assimilation, in conjunction with the Hungarian algorithm to
step, we go from:
associate measurements with their respective targets. SORT,
for instance, presupposes the following process model: Nk|k−1
X
l 2
p(xk |y1:k−1 ) = wk|k−1 N (x̂lk|k−1 , σk|k−1
l
) (37)
xk+1 = xk + k , k ∼i.i.d. N (0, Q) (31) l=1
JOURNAL OF LATEX CLASS FILES, VOL. 14, NO. 8, AUGUST 2021 11

to: C. Smoothing of Lateral Component

Nk|k
X 2
p(xk |y1:k ) = l
wk|k N (x̂lk|k , σk|k
l
) (38) For a more precise estimation, the distribution of the state
l=1 vector can be tailored based on future observations, rather than
solely relying on the observations up until the estimation time.
by using the Kalman update for each individual component l
This leads to a smoother, which is usually formulated as:
of the predictive mixture p(xk |y1:k−1 ). In the prediction step,
we go from the previous posterior distribution p(xk |y1:k ) to p(xk+1 |xk )p(xk+1 |y1:q )
Z
the predictive distribution: p(xk |y1:q ) = p(xk |y1:k ) dxk+1
p(xk+1 |y1:k )
Nk+1|k (40)
X
l 2 The conventional form described is utilized by the renowned
p(xk+1 |y1:k ) = wk+1|k N (x̂lk+1|k , σk+1|k
l
) (39)
l=1
Rauch-Tung-Streibel (RTS) Smoother. However, as evidenced
in [43], an analytical solution doesn’t exist when a Gaus-
Given that the process model’s Gaussian mixture consists
sian mixture is involved as in the case of RYIM model.
of three components, the quantity of components in the
Consequently, an alternative method known as the Two-Filter
posterior will increase exponentially as iterations progress.
approach should be applied:
To counteract this, we implement the Kullback-Leibler (KL)
GMM Reduction, as outlined in Algorithm 1 of [42]. The main p(yk+1:q |xk )p(xk |y1:k )
idea is to pairwise merge the components that result in the p(xk |y1:q ) = (41)
p(yk+1:q |y1:k )
KL divergence staying bounded to some threshold limit. The
filtering procedure for RYIM model is depicted in Algorithm
2. p(yk+1:q |xk ) =
Z
Algorithm 2 GMKF. Abbreviations: LC - Left Change, LK - p(yk+1 |xk+1 )p(yk+2:q |xk+1 )p(xk+1 |xk )dxk+1
Lane Keeping, RC - Right Change, Comp - Mixture Compo- (42)
nent.
Initialize N1|0 = 1, x̂11|0 = x1 , p11|0 = σx21 . A unimodal The precise formulas of the Two-Filter, applied to a general
prior of the lateral position N (x1 , σx21 ) Gaussian mixture for both process and measurement cases,
2: for 1 ≤ k ≤ q do
are detailed in [42]. A standout feature is the utilization of the
. Measurement assimilation (Eq. 35) information form to circumvent computational singularity tied
4: Nk|k = Nk|k−1 . Unimodal measurement model to the uniform distribution prior. In Algorithm 3, we introduce
for 1 ≤ l ≤ Nk|k−1 do the expressions for the Backward Information Filter, specific
to the RYIM model, which noticeably incorporates the mean
6: elk = xk − x̂lk|k−1 . Prediction error
2 2 trajectory as the control input of the process model. Owing
σkl = σk|k−1
l
+ σx2 to this unique aspect, we’ve named this algorithm Smoothing
2 2
8: Kkl = σk|k−1
l
/σkl . Kalman gain over Expectation, or SmoothiE.
x̂k|k = x̂k|k−1 + Kkl elk
l l
. Kalman update The notations used in Algorithm 3 are as follows:
10: l
σk|k
2
= (1 − Kkl )σk|k−1
l 2
. Posterior comp. var p(yk+2:q |xk+1 ) is the Gaussian mixture whose lth component

2 l −2
 is characterized by the information Llk+1|k+2 , slk+1|k+2 is the
wlk|k exp −1/2elk /σk
l information vector and tk+1|k+2 is a helpful variable for the
wk|k = √ l
2πσk
12: l
Normalize {wk|k } . Posterior comp. weight weight normalization.
end for
14: . Prediction (Eq. 36) D. Approximated Form for Longitudinal Component Dynam-
for 1 ≤ s ≤ Nk|k do ics
16: for 1 ≤ j ≤ 3 do . 3 modes: LC, LK, RC.
Nk+1|k = 3Nk|k Let zk represent longitudinal position and let µy,k stand for
18: l = 3(s − 1) + j the longitudinal mean time series value at timestep k.
x̂lk+1|k = µx,k+1 − µx,k + x̂lk|k . Position for
LC, LK, RC respectively zk+1 = zk + vk
2 l 2
20: l
σk+1|k = σk|k + σp2 where σp2 = σv,l
2
if j = zk+1 − µy,k+1 = zk − µy,k+1 − µy,k + µy,k + vk
2 2
1, σv,c if j = 2 and σv,r if j = 3
l s Let z̃k , zk − µy,k :
wk+1|k = wk|k αp where αp = αl if j = 1, αc
if j = 2 and αr if j = 3 z̃k+1 − z̃k = µy,k − µy,k+1 + vk
22: end for
end for ṽk = −v k + vk
24: Perform Mixture Component pruning using KL Diver-
where v k is the numerical derivative of the mean trajectory, or
gence (see Algorithm 1 of [42])
the mean velocity timeseries. The RAIM model then assumes:
end for
ṽk+1 = ṽk + ηv,k , ηv,k ∼i.i.d. N (0, σv2 ) (43)
JOURNAL OF LATEX CLASS FILES, VOL. 14, NO. 8, AUGUST 2021 12

Algorithm 3 Smoothing of lateral component (SMOOTHIE).

Initialize L1q|q+1 = 0, sq|q+1 = 0, tq|q+1 = 0, Nq = 1
for q − 1 ≥ k ≥ 1 do
. Measurement assimilation (Eq. 41)
for 1 ≤ l ≤ Nk+1 do
Llk+1|k+1 = Llk+1|k+2 + 1/σx2
ok+1 = −yk+1
slk+1|k+1 = slk+1|k+2 + ok+1 /σx2
tlk+1|k+1 = tlk+1|k+2 + o2k+1 /σx2 + ln(2πσx2 )
(a) (b)
end for
. Backward prediction (Eq. 42) Fig. 21: (a) An original frame from the NGSIM 101 dataset
for 1 ≤ l ≤ Nk+1 do and (b) the frame whose perspective is corrected.
for 1 ≤ i ≤ 3 do
s = 3(l − 1) + i
σp2 = σv,l
2 2
if i = 1, σv,c 2
if i = 2 and σv,r if yk for the longitudinal position at time k:
i=3 yk = zk + ζy,k , ζy,k ∼i.i.d. N (0, σy2 ) (45)
αp = αl if i = 1, αc if i = 2 and αr if i = 3
s 1 Obviously, the longitudinal component is a much simpler
Ok+1 = Ll +1/σ 2
k+1|k+1 p
1 than the lateral component since all distributions are unimodal
Lsk|k+1 = s
− Ok+1
σp2 (1 /σp2 )
Gaussians. This results in a familiar state-space model:
i
bk = µx,k+1 −µx,k +µv,l if i = 1, µx,k+1 −µx,k         
if i = 2, µx,k+1 − µx,k + µv,r if i = 3 zk+1 1 1 zk 0 0
= + (v k+1 − v k ) + (46)
vk+1 0 1 vk 1 ηv,k
bi
    
1   zk
ssk|k+1 = 2 bik + s
Ok+1 slk+1|k+1 − k2 yk = 1 0 + ζy,k (47)
σp σp vk

2 E. Filtering and Smoothing of Longitudinal Component

tsk|k+1 = tlk+1|k+1 − Ok+1s
slk+1|k+1 + ln σp2 − 2 ln αp
  The aforementioned state-space model enables conventional
s
− ln |Ok+1 | + bik /σp2 bik + 2Ok+1
s
slk+1|k+1 − Ok+1
s
bik /σp2 implementation of the Kalman Filter [44] for the longitu-
dinal component. When considering smoothing, one might
end for be inclined to apply the Two-Filter Approach, as detailed in
end for preceding sections.
. Convert to state-covariance form Equation (46) implies a process model covariance matrix of
for each comp l of the predictive mixture k|k + 1 do  
2 0 0
σ l = 1/Ll Q= (48)
2
µ = −σ l sl
l 0 σv2
2 2
w = 0.5(ln |2πσ l | + Ll µl − tl )
l
which does not have full-rank, complicating calculations like
end for s
Ok+1 in Algorithm 3. We suggest utilizing the RTS smoother
Perform Mixture Component pruning using KL Diver- to avoid this, as it doesn’t require inversion of the process
gence (see Algorithm 1 of [42]) covariance matrix.
. Convert back to information form
for each comp l of the predictive mixture k|k + 1 do F. Experimental Results
2
Ll = 1/σ l
We derive vehicle trajectories from the NGSIM 101
sl = −Ll µl
2 2 2 dataset’s aerial footage. Due to unavailable camera calibration
tl = µl /σ l + ln |2πσ l | − 2wl
information, we manually rectified perspective (Fig. 21), using
end for
pixel coordinates directly. While this doesn’t yield measure-
end for
ments in meters, it suffices to demonstrate the proposed filter
and smoother’s effectiveness.
The full trajectory extraction framework is presented in Fig.
As the result, the full process model for the longitudinal 22. The car positions and bounding boxes on the perspectived
state vector z is: transformed frame are detected with YOLOv8 [45] trained
( with 200 manually collected examples. The Precision-Recall
zk+1 = zk + vk Curve is shown in Fig. 23.
vk+1 = v k+1 + ṽk + ηy,k = (v k+1 − v k ) + vk + ηv,k Raw trajectory extracted from the CSRT tracker, a direct
(44) application of Algorithm 2 and 3 for the lateral component,
along with the KF and RTS smoother for the longitudinal
With a slight abuse of notation of the measurement being component reveal the results shown in Fig. 24-26. Here we
JOURNAL OF LATEX CLASS FILES, VOL. 14, NO. 8, AUGUST 2021 13

600
Filter MMSE

Longitudinal pos (y)

CSRT
500 Smoother MMSE
TRACKER 400 Raw
300
YOLOv8 CONVOLUTIONAL OBJECT DETECTION
200
NEURAL NETWORK
0 10 20 30 40 50
Timestep
FILTERED TRAJECTORY
GMKF Fig. 25: Estimation of longitudinal position y.
KF

0
Filter MMSE

Longitudinal vel (v)

RAW TRAJECTORY
5 Smoother MMSE

SMOOTHIE 10
15
0 10 20 30 40
Timestep

Fig. 26: Estimation of velocity v.

SMOOTHED TRAJECTORY

Fig. 22: Overview of the trajectory extraction framework. with approximately a twofold decrease in the confidence
interval of the velocity. The method’s uniqueness lies in
the provision of the smoothed MMSE, coupled with the
availability of the confidence interval, distinguishing it from
conventional denoising techniques where a confidence interval
is not usually obtainable. In scenarios where the vehicle’s
velocity does not factor into the application, employing the
filtered result could be sufficient for the sake of simplicity.

G. Estimation of the Mean Trajectory

As a prerequisite to both RAIM and RYIM models, the
Fig. 23: Precision-recall characteristics of the deep convolu- mean trajectory has to be computed for both the lateral and
tional neural network for detection of cars in video frames. longitudinal components.
Proposition 3. The unbiased estimate of the mean trajectory
is the empirical mean of the trajectory samples. The growth
have used the following parameters: σx = σy = 0.3, σv = rate of the sample set size is proportional to n2 , where n
0.2, αl = 0.05, µl = −1.5, σv,l = 0.1, αv,c = 0.9, σv,c = signifies the length of the mean trajectory.
0.1, αv,r = 0.05, µv,r = 1.5, σv,r = 0.1.
The data clearly demonstrates that both the filter and the Proof. From the approximated RYIM model (32):
smoother produced similar outcomes concerning the lateral
and longitudinal position of the vehicle. Nonetheless, the xk+1 = xk + µx,k+1 − µx,k + ηx,k + ζx,k
smoother’s performance displayed a significant enhancement
then with n being the length of the trajectory, we have:
n−1
70
X
Filter MMSE xn − x1 = µx,n − µx,1 + (ηx,k + ζx,k )
Smoother MMSE
Lateral pos (x)

k=1
65 Raw
(i) (i)
60 Suppose the set of samples for xn − x1 is {xn − x1 }m
i=1 ,
using Central Limit Theorem (CLT):
0 10 20 30 40 50 Pm (i) (i)
Timestep i=1 (xn − x1 ) − m(µx,n − µx,1 )
√
m
Fig. 24: Estimation of lateral position x.
→D N (0, (n − 1)(V ar[ηx,k ] + V ar[ζx,k ]))
JOURNAL OF LATEX CLASS FILES, VOL. 14, NO. 8, AUGUST 2021 14

or: With the validation of these models and the availability of

m
! covariance kernels, the applications of FPCA can be extended.
1 X (i)
(x(i)
n − x1 ) − (µx,n − µx,1 ) Conventionally, FPCA necessitates the derivation of basis
m functions from a training set, raising the question of the
 i=1 
n−1 robustness of these functions in relation to time and traffic con-
→D N 0, (V ar[ηx,k ] + V ar[ζx,k ])
m ditions. This paper introduces a novel finding from Proposition
1, asserting that the basis functions remain invariant under
In other words, the size of the dataset required for estimation
mild conditions of impulse regeneration time and amplitude
of the mean lateral component is proportional to the length of
samples from a stationary distribution. These assumptions
the trajectory:
appear valid for highway scenarios, and we speculate that
m = O(n) (49)
they likely hold for most traffic scenarios, provided conditions
For the longitudinal component, we have: do not rapidly change, such as with the sudden presence of
n−1
reckless vehicles.
The implications of these findings are significant; the learn-
X
yn − y1 = vk
k=1
ing phase can be entirely bypassed, and the same basis
functions can be hard-coded into signal processors. This can
We also have: lead to considerable memory and cost savings for vehicular
k=1
X and ITS applications.
vk − v1 = µv,k − µv,1 + ηv,j Moreover, this paper presents a comprehensive probabilistic
j=1
framework for filtering and, more importantly, smoothing of
Then it is not hard to see that: trajectories from aerial videos. While the position estimates
n−1
X n−1
X k−1
X of the vehicles don’t differ substantially between filtering and
yn − y1 = (v1 + µv,k − µv,1 ) + ηv,j + ζy,n smoothing, velocity estimation proves much more effective, as
k=1 k=1 j=1 indicated by a confidence interval reduction of approximately
50%. This paper also provides the theoretical background, jus-
Notice that E v1 = µv,1 , and that both v1 and ζy,n are
Pn−1 Pk−1 tification, and reference implementation for the use of random
dominated by the k=1 j=1 ηv,j term if n is sufficiently
walks as process noise models for vehicle tracking. While the
large, using CLT again:
! application is currently limited to trajectory extraction from
n−1 aerial videos, we believe these principles could be applicable
1 (i) (i)
X
(yn − y1 ) − µv,k in various automotive contexts, such as radar tracking and
m
k=1 antenna beamforming.
 
(n − 2)(n − 1) However, we acknowledge several unaddressed shortcom-
→D N 0, V ar[ηv ]
2m ings that warrant further research. First, a more detailed
So the size of the sample set m grows with the square of experimental or numerical study that compares filtering and
the length of the vehicle trajectory: smoothing results against other common trackers is needed.
Second, further refinement of the models is necessary, partic-
m = O(n2 ) (50) ularly for the lateral component. For instance, Fig. 10 suggests
a possible “feedback” effect that could explain the “rugby
football”-like shape of the contours.
VII. D ISCUSSION
VIII. C ONCLUSION
The validation of the Random Impulses Model demonstrates
that the Markovian assumption regarding random acceleration In conclusion, this paper introduces a kinematic statistical
and yawing impulses is reasonable, although the assumption model for road vehicle trajectory and investigates its potential
is more valid for the longitudinal component than the lateral. in various applications. These include FPCA, as well as the
The models further reveal that the discrepancy originates from approximated forms for probabilistic filtering and smoothing
the less frequent regeneration of yawing impulses compared to of trajectories derived from aerial video extraction.
acceleration impulses. Consequently, the approximated models The model’s assumptions were validated through the com-
(32) and (44) generally hold, but to varying degrees. putation of covariance kernels, while a convergence study
The models also offer a more intuitive approximation of illuminated the approximation power of the RYIM and RAIM
the underlying physical processes. Rather than modeling the models. The simplicity of these models, combined with the
driver’s reactions as functions of various external factors, they theoretical developments presented in this paper, advocate for
represent cumulative distance-keeping behavior using a single increased confidence in the use of the derived FPCA basis
component, namely the mean trajectory. The residual compo- functions within ITS and vehicular embedded platforms.
nent reflects the variations in inputs, leading to a significantly Moreover, the filtering and smoothing framework enriches
simpler model. However, despite this simplicity, the model still existing literature on the use of random walks as a process
maintains the powerful characteristics seen in car-following noise model. Ultimately, this research not only contributes to
models, such as a long prediction horizon. the current understanding of vehicle trajectory modeling, but
JOURNAL OF LATEX CLASS FILES, VOL. 14, NO. 8, AUGUST 2021 15

also paves the way for further exploration and application in [19] H. Fu, L. Wu, M. Jian, Y. Yang, and X. Wang, “Mf-sort: Simple online
this field. and realtime tracking with motion features,” in Image and Graphics:
10th International Conference, ICIG 2019, Beijing, China, August 23–
25, 2019, Proceedings, Part I 10. Springer, 2019, pp. 157–168.
[20] K.-C. Liu, Y.-T. Shen, and L.-G. Chen, “Simple online and realtime
ACKNOWLEDGMENTS tracking with spherical panoramic camera,” in 2018 IEEE International
Conference on Consumer Electronics (ICCE). IEEE, 2018, pp. 1–6.
This research was supported by the AI Interdisciplinary [21] P. G. Gipps, “A behavioural car-following model for computer simula-
Institute ANITI, funded by the French ”Investing for the tion,” Transportation Research Part B: Methodological, vol. 15, no. 2,
Future – PIA3” program (Grant n°ANR-19-PI3A-0004). We pp. 105–111, 1981.
[22] K. I. Ahmed, “Modeling drivers’ acceleration and lane changing behav-
also acknowledge Google Colab’s generosity in providing free ior,” Ph.D. dissertation, Massachusetts Institute of Technology, 1999.
GPU for YOLOv8 training. [23] K. Aghabayk, M. Sarvi, and W. Young, “A state-of-the-art review of
car-following models with particular considerations of heavy vehicles,”
Transport reviews, vol. 35, no. 1, pp. 82–105, 2015.
R EFERENCES [24] J. Han, H. Shi, L. Chen, H. Li, and X. Wang, “The car-following model
and its applications in the v2x environment: A historical review,” Future
[1] H. Yu and M. Krstic, “Traffic congestion control for aw–rascle–zhang Internet, vol. 14, no. 1, p. 14, 2021.
model,” Automatica, vol. 100, pp. 38–51, 2019. [25] G. Agamennoni, J. I. Nieto, and E. M. Nebot, “A bayesian approach
[2] E. Yurtsever, J. Lambert, A. Carballo, and K. Takeda, “A survey of for driving behavior inference,” in 2011 IEEE Intelligent Vehicles
autonomous driving: Common practices and emerging technologies,” Symposium (IV). IEEE, 2011, pp. 595–600.
IEEE access, vol. 8, pp. 58 443–58 469, 2020. [26] J. Joseph, F. Doshi-Velez, A. S. Huang, and N. Roy, “A bayesian
[3] Y. Li and D. Sun, “Microscopic car-following model for the traffic flow: nonparametric approach to modeling motion patterns,” Autonomous
the state of the art,” Journal of Control Theory and Applications, vol. 10, Robots, vol. 31, pp. 383–400, 2011.
no. 2, pp. 133–143, 2012. [27] G. Aoude, J. Joseph, N. Roy, and J. How, “Mobile agent trajectory pre-
[4] S. Moridpour, M. Sarvi, and G. Rose, “Lane changing models: a critical diction using bayesian nonparametric reachability trees,” in Infotech@
review,” Transportation letters, vol. 2, no. 3, pp. 157–173, 2010. Aerospace 2011, 2011, p. 1512.
[5] Y. Ji, L. Wang, W. Wu, H. Shao, and Y. Feng, “A method for lstm-based [28] R. Chandra, U. Bhattacharya, A. Bera, and D. Manocha, “Traphic:
trajectory modeling and abnormal trajectory detection,” IEEE Access, Trajectory prediction in dense and heterogeneous traffic using weighted
vol. 8, pp. 104 063–104 073, 2020. interactions,” in Proceedings of the IEEE/CVF Conference on Computer
[6] Y. Wang, G. Li, K. Li, and H. Yuan, “A deep generative model Vision and Pattern Recognition, 2019, pp. 8483–8492.
for trajectory modeling and utilization,” Proceedings of the VLDB [29] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez,
Endowment, vol. 16, no. 4, pp. 973–985, 2022. Ł. Kaiser, and I. Polosukhin, “Attention is all you need,” Advances in
[7] G. Zhu, Y. Sang, W. Chen, and L. Zhao, “When self-attention and neural information processing systems, vol. 30, 2017.
topological structure make a difference: Trajectory modeling in road [30] K. Messaoud, I. Yahiaoui, A. Verroust-Blondet, and F. Nashashibi,
networks,” in Web and Big Data: 6th International Joint Conference, “Attention based vehicle trajectory prediction,” IEEE Transactions on
APWeb-WAIM 2022, Nanjing, China, November 25–27, 2022, Proceed- Intelligent Vehicles, vol. 6, no. 1, pp. 175–185, 2020.
ings, Part III. Springer, 2023, pp. 381–396. [31] A. Barth and U. Franke, “Where will the oncoming vehicle be the next
[8] S. Lefèvre, D. Vasquez, and C. Laugier, “A survey on motion prediction second?” in 2008 IEEE Intelligent Vehicles Symposium. IEEE, 2008,
and risk assessment for intelligent vehicles,” ROBOMECH journal, pp. 1068–1073.
vol. 1, no. 1, pp. 1–14, 2014. [32] A. Kuczura, “Piecewise markov processes,” SIAM Journal on Applied
[9] S. Ammoun and F. Nashashibi, “Real time trajectory prediction for colli- Mathematics, vol. 24, no. 2, pp. 169–181, 1973.
sion risk estimation between vehicles,” in 2009 IEEE 5th International [33] P. Deheuvels and G. Martynov, “Karhunen-Loeve expansions for
Conference on Intelligent Computer Communication and Processing. weighted Wiener processes and Brownian bridges via Bessel functions,”
IEEE, 2009, pp. 417–422. in High Dimensional Probability III. Springer, 2003, pp. 57–93.
[10] N. Kaempchen, K. Weiss, M. Schaefer, and K. C. Dietmayer, “Imm [34] R. Durrett, Probability: theory and examples. Cambridge university
object tracking for high dynamic driving maneuvers,” in IEEE Intelligent press, 2019, vol. 49.
Vehicles Symposium, 2004. IEEE, 2004, pp. 825–830. [35] T. H. Dinh, V. Martinez, and D. Delahaye, “Recognition of outlying
[11] A. Polychronopoulos, M. Tsogas, A. J. Amditis, and L. Andreone, driving behaviors: A data-driven perspective with applications to v2x
“Sensor fusion for predicting vehicles’ path for collision avoidance collective perception,” in 2021 IEEE Vehicular Networking Conference
systems,” IEEE Transactions on Intelligent Transportation Systems, (VNC). IEEE, 2021, pp. 52–59.
vol. 8, no. 3, pp. 549–562, 2007. [36] H. L. Shang, “A survey of functional principal component analysis,”
[12] V. G. Kovvali, V. Alexiadis, and L. Zhang PE, “Video-based vehicle AStA Advances in Statistical Analysis, vol. 98, pp. 121–142, 2014.
trajectory data collection,” Tech. Rep., 2007. [37] J. Brown, Jr, “Mean square truncation error in series expansions of
[13] J. Apeltauer, A. Babinec, D. Herman, and T. Apeltauer, “Automatic random functions,” Journal of the Society for Industrial and Applied
vehicle trajectory extraction for traffic analysis from aerial video data,” Mathematics, vol. 8, no. 1, pp. 28–32, 1960.
The International Archives of Photogrammetry, Remote Sensing and [38] T. H. Dinh, V. Martinez, and D. Delahaye, “Spherical codec for v2x
Spatial Information Sciences, vol. 40, no. 3, p. 9, 2015. cooperative awareness trajectory compression: A preliminary study,” in
[14] N. Balamuralidhar, S. Tilon, and F. Nex, “Multeye: Monitoring system 2023 IEEE Vehicular Technology Conference (VTC). IEEE, 2023.
for real-time vehicle detection, tracking and speed estimation from uav [39] G. Jarry, D. Delahaye, F. Nicol, and E. Feron, “Aircraft atypical approach
imagery on edge-computing platforms,” Remote Sensing, vol. 13, no. 4, detection using functional principal component analysis,” Journal of Air
p. 573, 2021. Transport Management, vol. 84, p. 101787, 2020.
[15] X. Chen, Z. Li, Y. Yang, L. Qi, and R. Ke, “High-resolution vehicle tra- [40] G. Jarry, D. Delahaye, and E. Feron, “Trajectory approach analysis:
jectory extraction and denoising from aerial videos,” IEEE Transactions A post-operational aircraft approach analysis tool,” in SID 2019, 9th
on Intelligent Transportation Systems, vol. 22, no. 5, pp. 3190–3202, SESAR Innovation Days, 2019.
2020. [41] A. G. Wills, J. Hendriks, C. Renton, and B. Ninness, “A bayesian
[16] E. Barmpounakis and N. Geroliminis, “On the new era of urban traffic filtering algorithm for gaussian mixture models,” arXiv preprint
monitoring with massive drone data: The pneuma large-scale field arXiv:1705.05495, 2017.
experiment,” Transportation research part C: emerging technologies, [42] M. P. Balenzuela, J. Dahlin, N. Bartlett, A. G. Wills, C. Renton, and
vol. 111, pp. 50–71, 2020. B. Ninness, “Accurate gaussian mixture model smoothing using a two-
[17] A. Bewley, Z. Ge, L. Ott, F. Ramos, and B. Upcroft, “Simple online filter approach,” in 2018 IEEE Conference on Decision and Control
and realtime tracking,” in 2016 IEEE international conference on image (CDC). IEEE, 2018, pp. 694–699.
processing (ICIP). IEEE, 2016, pp. 3464–3468. [43] G. Kitagawa, “The two-filter formula for smoothing and an implementa-
[18] N. Wojke, A. Bewley, and D. Paulus, “Simple online and realtime tion of the gaussian-sum smoother,” Annals of the Institute of Statistical
tracking with a deep association metric,” in 2017 IEEE international Mathematics, vol. 46, pp. 605–623, 1994.
conference on image processing (ICIP). IEEE, 2017, pp. 3645–3649. [44] G. Welch, G. Bishop et al., “An introduction to the kalman filter,” 1995.
JOURNAL OF LATEX CLASS FILES, VOL. 14, NO. 8, AUGUST 2021 16

[45] Ultralytics. Ultralytics yolov8. [Online]. Available: https://docs.ultralyti

cs.com/

Thinh Hoang Dinh is currently pursuing his PhD

at the ENAC Optimization Laboratory in Toulouse,
France. His research focuses on the application
of artificial intelligence in the field of transporta-
tion, aiming to improve efficiency and sustainability
within this sector.

Vincent Martinez received the M.Sc. degree in

mathematical and statistical modeling from INSA
Toulouse, France. He is working as a Senior Princi-
pal Engineer at NXP Semiconductors since 2006.
He is currently a Wireless Signal Processing and
Physical Layer Expert, and is contributing to various
V2X standardization bodies.

Pierre Maréchal is Professor of Mathematics at the

University of Toulouse III Paul Sabatier. Although
his initial training was in engineering, he obtained a
PhD in Image processing in 1997, and then turned
to mathematics through a postdoc at Simon Fraser
University. His areas of interest are mainly, but not
exclusively, optimization and inverse problems.

Daniel Delahaye leads the Optimization and Ma-

chine Learning Team at ENAC Research Laboratory
and chairs the ”AI for ATM and Large Scale Urban
Mobility” research at ANITI in Toulouse. After earn-
ing his engineering degree and master’s from ENAC
and National Polytechnic Institute of Toulouse, he
obtained his Ph.D. in automatic control and later
tenure in applied mathematics at ENAC. He has
worked at the French Civil Aviation Study Center
and MIT. His research, conducted in collaboration
with MIT, Georgia Tech, and NASA, centers on AI
and mathematical optimization for airspace and aircraft trajectory optimiza-
tion.