International Journal of Computer Vision 51(2), 91–109, 2003

c 2003 Kluwer Academic Publishers. Manufactured in The Netherlands.

!

Dynamic Textures

GIANFRANCO DORETTO
Computer Science Department, University of California, Los Angeles, CA 90095
[email protected]

ALESSANDRO CHIUSO
Dipartimento di Ingegneria dell’Informazione, Università di Padova, Italy 35131
[email protected]

YING NIAN WU
Statistics Department, University of California, Los Angeles, CA 90095
[email protected]

STEFANO SOATTO
Computer Science Department, University of California, Los Angeles, CA 90095
[email protected]

Received May 1, 2001; Revised February 21, 2002; Accepted July 3, 2002

Abstract. Dynamic textures are sequences of images of moving scenes that exhibit certain stationarity properties
in time; these include sea-waves, smoke, foliage, whirlwind etc. We present a characterization of dynamic
textures that poses the problems of modeling, learning, recognizing and synthesizing dynamic textures on a
firm analytical footing. We borrow tools from system identification to capture the “essence” of dynamic tex-
tures; we do so by learning (i.e. identifying) models that are optimal in the sense of maximum likelihood
or minimum prediction error variance. For the special case of second-order stationary processes, we iden-
tify the model sub-optimally in closed-form. Once learned, a model has predictive power and can be used
for extrapolating synthetic sequences to infinite length with negligible computational cost. We present exper-
imental evidence that, within our framework, even low-dimensional models can capture very complex visual
phenomena.

Keywords: textures, dynamic scene analysis, 3D textures, minimum description length, image com-
pression, generative model, prediction error methods, ARMA model, subspace system identification,
canonical correlation, learning

1. Introduction well as upon its material properties (reflectance) and

on the light distribution of the environment. It is well
Consider a sequence of images of a moving scene. known that the joint reconstruction of photometry and
Each image is an array of positive numbers that de- geometry is an intrinsically ill-posed problem: from any
pend upon the shape, pose and motion of the scene as (finite) number of images it is not possible to uniquely
92 Doretto et al.

recover all unknowns (shape, motion, reflectance and assumptions. All we require is that the model be capa-
light distribution). Traditional approaches to scene ble of predicting future measurements. In a sense, we
reconstruction rely on fixing some of the unknowns look for an “explanation” of the image data that allows
either by virtue of assumption or by restricting us to recreate and extrapolate it. It can therefore be
the experimental conditions, while estimating the thought of as the compressed version or the “essence”
others.1 of the sequence of images.
However, such assumptions can never be validated
from visual data, since it is always possible to con-
struct scenes with different photometry and geometry 1.1. Contributions of this Work
that give rise to the same images.2 The ill-posedness of
the most general visual reconstruction problem and the This work addresses several aspects in the field of dy-
remarkable consistency in the solution as performed namic (or time-varying) textures. On the issue of rep-
by the human visual system reveals the importance of resentation, we present a novel definition of dynamic
priors for images (Zhu et al., 1997); they are necessary texture that is general (even the simplest instance can
to fix the arbitrary degrees of freedom and render the capture the statistics of a sequence of images {I (t)} that
problem well-posed (Kirsch, 1996). In general, one can are a second-order stationary process3 with an arbi-
use the extra degrees of freedom to the benefit of the trary covariance sequence) and precise (it allows mak-
application at hand: one can fix photometry and esti- ing analytical statements and drawing from the rich
mate geometry (e.g. in robot vision), or fix geometry literature on system identification). On learning, we
and estimate photometry (e.g. in image-based render- propose two criteria: total likelihood or prediction error.
ing), or recover a combination of the two that satisfies Under the hypothesis of second-order stationarity, we
some additional optimality criterion, for instance the give a closed-form sub-optimal solution of the learn-
minimum description length of the sequence of video ing problem. On recognition, we show how textures
data (Rissanen, 1978). alike tend to cluster in model space, therefore assess-
Given this arbitrariness in the reconstruction and ing the potential to build a recognition system based
interpretation of visual scenes, it is clear that there on this framework (Saisan et al., 2001). On synthesis,
is no notion of a true interpretation, and the crite- we show that even the simplest linear dynamical model
rion for correctness is somewhat arbitrary. In the case (first-order ARMA4 model with white zero-mean IID5
of humans, the interpretation that leads to a correct Gaussian input) captures a wide range of dynamic tex-
Euclidean reconstruction (that can be verified by other tures. Our algorithm is simple to implement, efficient
sensory modalities, such as touch) has obvious appeal, to learn and fast to simulate; it allows generating in-
but there is no way in which the correct Euclidean finitely long sequences from short input sequences and
interpretation can be retrieved from visual signals to control the parameters in the simulation (Doretto and
alone. Soatto, 2002).
Therefore, in this paper we will analyze sequences Although in our experiments we only consider sim-
of images of moving scenes solely as visual signals. ple choices of input distributions, more general classes
“Interpreting” and “understanding” a signal amounts can be taken into account by using particle filtering
to inferring a stochastic model that generates it. The techniques and more general classes of filter banks.
“goodness” of the model can be measured in terms of We use linear dynamical systems because they capture
the total likelihood of the measurements or in terms second-order stationarity. Several extensions can be de-
of its predicting power: a model should be able to vised, although no closed-form solutions are available.
give accurate predictions of future signals (akin to Some of these results may be useful for video com-
so-called prediction error methods in system identi- pression and for image-based rendering and synthesis
fication). Such a model will involve a combination of of image sequences.
photometry, geometry and dynamics and will be de-
signed for maximum-likelihood or minimal prediction
error variance. Notice that we will not require that the 1.2. Prior Related Work
reconstructed photometry or geometry be correct (in
the Euclidean sense), for that is intrinsically impos- Statistical inference for analyzing and understanding
sible without involving (visually) non-verifiable prior general images has been extensively used for the last
 Dynamic Textures 93

two decades (Mumford and Gidas, 1998). The statisti- cess involves morphing techniques to smooth out vi-
cal characterization of textures was pioneered by Julesz sual discontinuities. Another example is the work of
four decades back (Julesz, 1962). Following that, there Wei and Levoy (2000), where they synthesize temporal
has been extensive work in the area of 2D texture anal- textures by generating each new pixel, in the 3D spatio-
ysis, recognition and synthesis. Most of the approaches temporal space of a video sequence, by searching, in
use statistical models (Heeger and Bergen, 1995; Zhu the original sequence, a pixel neighborhood that best
et al., 1997; Popat and Picard, 1993; Portilla and matches its companion in the synthesized output. Pro-
Simoncelli, 1999; de Bonet and Viola, 1998; Paget cedural techniques result in a relatively quick solution
and Longstaff, 1996; Cross and Jain, 1983; Hassner for the purpose of synthesis. Within this framework,
and Sklansky, 1981) while few others rely on the simulation is generated without explicitly inferring
deterministic structural models (Efros and Leung, a model, which results in lack of flexibility for other
1999; Wei and Levoy, 2000). Another distinction is purposes such as editing, classification, recognition, or
that some work directly on the pixel values while compression.
others project image intensity onto a set of basis There has been comparatively little work in the spe-
functions.6 cific area of image-based techniques that rely on a
There have been many physics-based algorithms model. The problem of modeling dynamic textures
which target specific dynamic textures (Ebert et al., has been first addressed by Nelson and Polana (1992),
1994; Fournier and Reeves, 1986; Peachey, 1986). where they classify regional activities of a scene char-
Some simulations have been performed using par- acterized by complex, non-rigid motion. The same
ticle systems (Reeves, 1983; Sims, 1990). In these problem has been later considered by Saisan et al.
approaches a model of the scene is derived from first (2001).
principles, then approximated, and finally simulated. Bar-Joseph et al. (2001) use multi-resolution anal-
Such techniques have been successfully applied] for ysis (MRA) tree merging for the synthesis and merg-
synthesizing sequences of natural phenomena such as ing of 2D textures and extends the idea to dynamic
smoke, fire etc. (see for instance Stam and Fiume textures. For 2D textures new MRA trees are con-
(1995) and references therein), but also walking gaits structed by merging MRA trees obtained from the
(Hodgins and Wooten (1998) and references), and me- input; the algorithm is different from De Bonet’s
chanical systems (Barzel (1992) and references). The (de Bonet and Viola, 1998) that operates on a single
main advantage of these techniques is the extent in texture sample. The idea is extended to dynamic tex-
which the synthesis can be manipulated, resulting in tures by constructing MRA trees using a 3D wavelet
great editing power. While physics-based models are transform. Impressive results were obtained for the 2D
the most principled and elegant, they have the disad- case, but only a finite length sequence is synthesized af-
vantage of being computationally expensive and often ter computing the combined MRA tree. Our approach
highly customized for particular textures, therefore not captures the essence of a dynamic texture in the form
allowing automatic ways of inferring new models for of a dynamical model, and an infinite length sequence
a large class of dynamic textures. can be generated in real time using the parameters com-
An alternative to physics-based techniques are puted off-line and, for the particular case of linear dy-
image-based ones. In this framework, new texture namic textures, in closed-form.
movies are generated using images without building Szummer and Picard’s work (1996) on temporal tex-
a physical model of the process that generates the ture modeling uses a similar approach towards cap-
scene. Among these approaches, one can distinguish turing dynamic textures. They use the spatio-temporal
two subclasses, the so-called “procedural” techniques auto-regressive model (STAR), which imposes a neigh-
that forego the use of a model altogether and gener- borhood causality constraint even for the spatial do-
ate synthetic images by clever concatenation or repeti- main. This severely restricts the textures that can be
tion of image data, and image-based techniques that captured and does not allow to capture rotation, ac-
rely on a model, albeit not a physical one. As ex- celeration and other simple non translational motions.
ample of the first subclass, the work of Schödl et al. It works directly on the pixel intensities rather than
(2000), addresses the problem by finding transition a smaller dimensional representation of the image. We
points in the original sequence where the video can incorporate spatial correlation without imposing causal
be looped back in a minimally invasive way. The pro- restrictions, as will be clear in the coming sections,
94 Doretto et al.

and can capture more complex motions, including ones IID IID
with x(0) = x0 , v(t) ∼ q(·) unknown, w(t) ∼ pw (·)
where the STAR model is ineffective (see Szummer and
given, such that I (t) = φ(x(t)). To the best of our
Picard (1996), from which we borrow some of the data
knowledge, the characterization of a dynamic texture
processed in Section 5).
as the output of an ARMA model is novel.10
We want to make it clear that this definition ex-
plains what we mean by dynamic textures. It could
2. Representation of Dynamic Textures be argued that this definition does not capture the in-
tuitive notion of a dynamic texture, and that is indeed
What is a suitable definition of texture? For a single possible. As showed in Section 5, however, we have
image, one can say a texture is a realization from a found that the model (1) captures most of what our
stationary stochastic process with spatially invariant intuition calls dynamic textures, and even visual phe-
statistics (Zhu et al., 1997). This definition captures the nomena that are beyond the purpose of this modeling
intuitive notion of texture. For a sequence of images framework. Furthermore, one can easily generalize the
(time-varying texture), individual images are clearly definition to an arbitrary non-linear model of the form
not independent realizations from a stationary distri- x(t + 1) = f (x(t), v(t)), leading to the concept of a
bution, for there is a temporal coherence intrinsic in non-linear dynamic texture.
the process that needs to be captured. The underlying
assumption, therefore, is that individual images are re-
2.2. Filters and Dimensionality Reduction
alizations of the output of a dynamical system driven
by an independent and identically distributed (IID) pro-
The definition of dynamic texture above entails a choice
cess. We now make this concept precise as an operative
of filters φα , α = 1 . . . n. These filters are also inferred
definition of dynamic texture.
as part of the learning process for a given dynamic
texture.
There are several criteria for choosing a suitable class
2.1. Definition of Dynamic Texture of filters, ranging from biological motivations to com-
putational efficiency. In the simplest case, we can take
Let {I (t)}t=1...τ , I (t) ∈ Rm , be a sequence of τ images. φ to be the identity, and therefore look at the dynamics
Suppose that at each instant of time t we can mea- of individual pixels11 x(t) = I (t) in (1). We view the
sure a noisy version of the image, y(t) = I (t) + w(t), choice of filters as a dimensionality reduction step, and
where w(t) ∈ Rm is an independent and identically seek for a decomposition of the image in the simple
distributed sequence drawn from a known distribu- (linear) form
tion,7 pw (·), resulting in a positive measured sequence
n
y(t) ∈ Rm , t = 1 . . . τ . We say that the sequence # .
{I (t)} is a (linear) dynamic texture if there exists a I (t) = xi (t)θi = C x(t), (2)
i=1
set of n spatial filters φα : R → Rm , α = 1 . . . n
and a stationary distribution q(·) such that, defining8 where C = [θ1 , . . . , θn ] ∈ Rm×n and {θi } can be an
x(t) ∈ Rn such that I (t) = φ(x(t)), we have x(t) = orthonormal basis of L2 , a set of principal components,
k nv
i=1 Ai x(t − i) + Bv(t), with v(t) ∈ R an IID real-
!
or a wavelet filter bank, for instance.
9
ization from the density q(·), for some choice of ma- An alternative non-linear choice of filters can be ob-
trices, Ai ∈ Rn×n , i = 1, . . . , k, B ∈ Rn×n v and initial tained by processing the image with a filter bank, and
condition x(0) = x0 . Without loss of generality, we representing it as the collection of positions of the max-
can assume k = 1 since we can redefine the state of the imal response in the passband (Mallat, 1989). In this
above model x(t) to be [x(t)T x(t −1)T . . . x(t −k)T ]T . paper we will restrict our attention to linear filters.
Therefore, a linear dynamic texture is associated to an
auto-regressive moving average process (ARMA) with
unknown input distribution 3. Learning Dynamic Textures

Given a sequence of noisy images {y(t)}t=1...τ , learning

x(t + 1) = Ax(t) + Bv(t)
"
(1) the dynamic texture amounts to identifying the model
y(t) = φ(x(t)) + w(t) parameters A, B, C and the distribution of the input
 Dynamic Textures 95

q(·) in the model (1). This is a system identification as

problem (Lindquist and Picci, 1979), where one has to .
infer a dynamical model from a time series. However, Â, B̂, Ĉ, q̂ = lim arg min E'y(t + 1) − C x̂(t + 1|t)'2
t→∞
in the literature of dynamical systems, it is commonly subject to (1). (3)
assumed that the distribution of the input is known. In
the context of dynamic textures, we have the additional Unfortunately, explicit forms of the one-step predictors
complication of having to infer the distribution of the are available only under restricted assumptions, for in-
input along with the dynamical model. The learning, stance linear models driven by white Gaussian noise
or system identification, problem can then be posed as which we consider in Section 4. For details the reader
follows. is referred to (Ljung, 1987).

3.1. Maximum Likelihood Learning 3.3. Representation of the Driving Distribution

The maximum-likelihood formulation of the dynamic So far we have managed to defer addressing the fact that
texture learning problem can be posed as follows: the unknown driving distribution belongs, in principle,
to an infinite-dimensional space, and therefore some-
given y(1), . . . , y(τ ), find thing needs to be said about how this issue is dealt with
Â, B̂, Ĉ, q̂(·) = arg max log p(y(1), . . . , y(τ )) algorithmically.
A,B,C,q We consider three ways to approach this problem.
subject to (1) and v(t) ∼ q.
IID One is to transform this into a finite-dimensional infer-
ence problem by choosing a parametric class of den-
sities. This is done in the next section, where we pos-
The inference method depends crucially upon what
tulate that the unknown driving density belongs to a
type of representation we choose for q. Note that the
finite-dimensional parameterization of a class of expo-
above inference problem involves the hidden variables
nential densities, and therefore the inference problem
x(t) multiplying the unknown parameter A and realiza-
is reduced to a finite-dimensional optimization. The ex-
tions v(t) multiplying the unknown parameter B, and
ponential class is quite rich and it includes, in particular,
is therefore intrinsically non-linear even if the origi-
multi-modal as well as skewed densities, although with
nal state-space model is linear. In general, one could
experiments we show that even a single Gaussian model
use iterative techniques that alternate between estimat-
allows achieving good results. When the dynamic tex-
ing (sufficient statistics of) the conditional density of
ture is represented by a second-order stationary process
the state and maximizing the likelihood with respect
we show that a closed-form sub-optimal solution can
to the unknown parameters, in a fashion similar to the
be obtained.
expectation-maximization (EM) algorithm (Dempster
The second alternative is to represent the density
et al., 1977). In order for such iterative techniques to
q via a finite number of fair samples drawn from it;
converge to a unique minimum, canonical model real-
the model (1) can be used to represent the evolution
izations need to be considered, corresponding to par-
of the conditional density of the state given the mea-
ticular forms for the matrices A and B. We discuss
surements, and the density is evolved by updating the
such realizations in Section 4, where we also present
samples so that they remain a fair realization of the con-
a closed-form sub-optimal solution for a class of
ditional density as time evolves. Algorithms of this sort
dynamic textures.
are called “particle filters” (Liu et al., 2000), and in par-
ticular the CONDENSATION filter (Blake and Isard,
3.2. Prediction Error 1998) is the best known instance in the Computer
Vision community.
As an alternative to maximum-likelihood, one can con- The third alternative is to treat (1) as a semi-
sider estimating the model that results in the least pre- parametric statistical problem, where one of the “pa-
diction error, for instance in the sense of mean square. rameters” (q) lives in the infinite-dimensional manifold
.
Let x̂(t + 1|t) = E[x(t + 1)|y(1), . . . , y(t)] be the best of probability densities that satisfy certain regularity
one-step predictor that depends upon the unknown pa- conditions, endowed with a Riemannian metric (cor-
rameters A, B, C, q. One can then pose the problem responding to the Fisher’s information matrix), and to
96 Doretto et al.

design gradient descent algorithms with respect to the choice of basis of the state space (because it has been
natural connection, as it has been done in the context of fixed).
independent component analysis (ICA) by Amari and While there are many possible choices of canonical
Cardoso (1997). This avenue is considerably more la- models (see for instance Kailath, 1980), we are inter-
borious and we are therefore not considering it in this ested in one that is “tailored” to the data, in the sense
study. explained below. Since we are interested in data di-
mensionality reduction, we will make the following
4. A Closed-Form Solution for Learning assumptions about the model (4):
Second-Order Stationary Processes
m ( n; rank(C) = n, (5)
It is well known that a second-order stationary process
with arbitrary covariance can be modeled as the output and choose the canonical model that makes the columns
of a linear dynamical system driven by white, zero- of C orthonormal:
mean Gaussian noise (Ljung, 1987). In our case, we
will therefore assume that there exist a positive integer C T C = In , (6)
n, a process {x(t)}, x(t) ∈ Rn , with initial condition
x0 ∈ Rn and symmetric positive-definite matrices Q ∈ where In is the identity matrix of dimension n × n. As
Rn×n and R ∈ Rm×m such that we will see shortly, this assumption results in a unique
model that is tailored to the data in the sense of defining
"
x(t + 1) = Ax(t) + v(t) v(t) ∼ N (0, Q); x(0) = x0 a basis of the state-space such that its covariance is
y(t) = C x(t) + w(t) w(t) ∼ N (0, R) asymptotically diagonal (see Eq. (11)).
(4) The problem we set out to solve can then be for-
mulated as follows: given measurements of a sample
for some matrices A ∈ Rn×n and C ∈ Rm×n . The prob- path of the process: y(1), . . . , y(τ ); τ ( n, estimate
lem of system identification consists in estimating the Â, Ĉ, Q̂, R̂, a canonical model of the process {y(t)}.
model parameters A, C, Q, R from the measurements Ideally, we would want the maximum-likelihood
y(1), . . . , y(τ ). Note that B and v(t) in the model (1) solution:
are such that B B T = Q, and v(t) ∼ N (0, In v ) where
In v is the identity matrix of dimensions n v × n v . Â(τ ), Ĉ(τ ), Q̂(τ ), R̂(τ ) = arg max p(y(1) . . . y(τ )).
A,C,Q,R
(7)
4.1. Uniqueness and Canonical Model Realizations
Asymptotically optimal solutions of this problem, in
The first observation concerning the model (4) is that the maximum-likelihood sense, do exist in the literature
the choice of matrices A, C, Q is not unique, in the of system identification theory (Ljung, 1987). In par-
sense that there are infinitely many such matrices that ticular, the subspace identification algorithm N4SID,
give rise to exactly the same sample paths y(t) start- described in Van Overschee and De Moor (1994), is
ing from suitable initial conditions. This is immedi- available as a Matlab toolbox. The main reason why in
ately seen by substituting A with TAT −1 , C with C T −1 Section 4.2 we propose a sub-optimal solution of the
and Q with TQT T , and choosing the initial condition problem is because, given the dimensionality of our
T x0 , where T ∈ GL(n) is any invertible n × n ma- framework, the N4SID algorithm requires a memory
trix. In other words, the basis of the state-space is ar- storage far beyond the capabilities of the current state
bitrary, and any given process has not a unique model, of the art workstations. The result derived in Section
.
but an equivalence class of models R = {[A] = 4.2 is a closed-form sub-optimal solution in the sense
TAT −1 , [C] = CT −1 , [Q] = TQT T , | T ∈ GL(n)}. of Frobenius that takes 30 seconds to run on a common
In order to be able to identify a unique model of the PC when m = 170 × 110 and τ = 140.
type (4) from a sample path y(t), it is therefore nec- Before presenting the solution of the learning prob-
essary to choose a representative of each equivalence lem (7), we point out an unspoken hypothesis that has
class: such a representative is called a canonical model been made so far in the paper, i.e. the fact the frame-
realization, in the sense that it does not depend on the work we propose entails the filtering in space and time
 Dynamic Textures 97

to be separable, which means that we perform filter- .

where v̂(t) = x̂(t + 1) − Â(τ )x̂(t). Should Q̂ not be
ing in space and time in two separate stages. The rea- full rank, its dimensionality can be further reduced
son for this choice is nothing else than computational by computing the SVD Q̂ = U Q % Q U QT where % Q =
simplicity of the resulting algorithm. diag{σ Q (1), . . . , σ Q (n v )} with n v ≤ n, and letting B̂
T
be such that B̂ B̂ = Q̂.
4.2. Closed-Form Solution In the algorithm above we have assumed that the or-
der of the model n was given. In practice, this needs to
. . be inferred from the data. Following Arun and Kung
Let Y1τ = [y(1), . . . , y(τ )] ∈ Rm×τ and X 1τ =
. (1990), we propose to determine the model order em-
[x(1), . . . , x(τ )] ∈ Rn×τ with τ > n, and W1τ =
[w(1), . . . , w(τ )] ∈ Rm×τ , and notice that pirically from the singular values σ1 , σ2 , . . . , by choos-
ing n as the cutoff where the singular values drop below
Y1τ = C X 1τ + W1τ ; C ∈ Rm×n ; C T C = I, (8) a threshold. A threshold can also be imposed on the
difference between adjacent singular values.
by our assumptions (5) and (6). Now let Y1τ = U %V T ; Notice that the model we describe in this paper
U ∈ Rm×n ; U T U = I ; V ∈ Rτ ×n , V T V = I be the can also be used to perform denoising of the origi-
singular value decomposition (SVD) (Golub and Van nal sequence. It is immediate to see that the denoised
Loan, 1989) with % = diag{σ1 , . . . , σn }, and {σi } be sequence is given by
the singular values, and consider the problem of find- .
Iˆ(t) = Ĉ x̂(t), (13)
ing the best estimate of C in the sense of Frobenius:
Ĉ(τ ), X̂ (τ ) = arg minC,X 1τ 'W1τ ' F subject to (8). It fol- where Ĉ is the estimate of C and x̂(t) is obtained from
lows immediately from the fixed rank approximation x̂(t + 1) = Â x̂(t) + B̂ v̂(t).
property of the SVD (Golub and Van Loan, 1989) that
the unique solution is given by 4.3. Asymptotic Properties

Ĉ(τ ) = U X̂ (τ ) = %V T (9) The solution given above is, strictly speaking, incorrect
because the first SVD does not take into account the fact
 can be determined uniquely (assuming distinct singu- that the state X (τ ) has a very particular structure (i.e.
lar values), again in the sense of Frobenius, by solving it is the state of a linear dynamical model). We chose
the following linear problem: Â(τ ) = arg min A 'X 2τ − to give up completeness in the exposition, and there-
.
AX 1τ −1 ' F , where X 2τ = [x(2), . . . , x(τ )] ∈ Rn×τ fore reported the simplest instance of the algorithm, in
which is trivially done in closed-form using the state favor of clarity and simplicity of exposition. It is pos-
estimated from (9): sible, however, to adapt the algorithm to take this into
account while still achieving a closed-form solution
Â(τ ) = %V T D1 V (V T D2 V )−1 % −1 (10) that can be proven to be asymptotically efficient, i.e.
to approach the maximum-likelihood solution. The re-
where D1 = [ Iτ0−1 00 ] and D2 = [ Iτ0−1 00 ]. Notice that sulting algorithm is exactly N4SID, and its asymptotic
Ĉ(τ ) is uniquely determined up to a change of sign of properties have been studied by Bauer et al. (1999) and
the components of C and x. Also note that Chiuso and Picci (to appear). Such an optimal algo-
τ rithm, however, is computationally expensive, and the
1# gain in the quality of the final model, for the experi-
E[x̂(t)x̂ T (t)] ≡ lim x̂(k)x̂ T (k)
τ →∞ τ ments reported below, is marginal.
k=1
1 1
= %V T V % = % 2 ,
∼ (11)
τ τ
5. Experiments
which is diagonal as mentioned in Section 4.1. Finally,
the sample input noise covariance Q can be estimated We coded the algorithm described in Section 4 using
from Matlab: learning a graylevel sequence of 140 frames
τ −1
with m = 170 × 110 takes about 30 seconds on a desk-
1 # top PC (1 GHz), while it takes about 5 minutes for 150
Q̂(τ ) = v̂(i)v̂ T (i) (12)
τ − 1 i=1 color frames and m = 320 × 220. Synthesis can be
98 Doretto et al.

Figure 1. Matlab code implementation of the closed-form sub-

optimal learning algorithm proposed in Section 4 (function dytex),
and the synthesis stage (function synth). In order to perform sta-
ble simulations, the synthesis function assumes that the poles of
the linear system (i.e. the eigenvalues of Ahat) are within the unit
circle.

performed at frame rate. The Matlab routines imple-

menting the learning and synthesis algorithms are re-
ported in Fig. 1. The dimension of the state n and input
Figure 2. Spiraling-water. The figure shows how an “infinite
n v is given as an input argument. In our implementa- length” texture sequence is synthesized from a typically “short” in-
tion, we have used τ between 50 and 150, n between put texture sequence by just drawing samples from v(t). The data set
10 and 50 and n v between 10 and 30. used comes from the MIT Temporal Texture database. The particular
structure of this sequence (spiraling-water synthesized using
n = 20 principal components, τ = 120, m = 85 × 65), is amongst
the ones that cannot be captured by the STAR model (Szummer and
5.1. Synthesis Picard, 1996).

Figure 2 illustrates the fact that an “infinite length”

texture sequence can be synthesized from a typically Figures 3 to 7 show the behavior of the algorithm
“short” input sequence by just drawing IID samples on a representative set of experiments (the training se-
v(t) from a Gaussian distribution. The frames belong quences of Figs. 3 to 6 have been borrowed from the
to the spiraling-water sequence. From a 120 MIT Temporal Texture database13 ). In each case, on
frame-long training sequence a 300 frames synthe- the first row we show a few images from the origi-
sized sequence12 of dimensions 85 × 65 pixels has nal dataset, on the second row we show their com-
been generated using n = 20 principal components. pressed version (see Section 5.3), and on the third
This sequence has been shown in Szummer and Picard row we show a few extrapolated samples. On the
(1996) as an example where the STAR model fails in last row we show the overall compression error as
capturing non-translational motion. Our model, on the a function of the dimension of the state space (left)
other hand, has no difficulty in capturing the spatio- as well as the prediction error as a function of the
temporal statistics of the input sequence, as shown in length of the learning set (right). For very regular se-
Fig. 2. quences, the prediction error decreases monotonically;
 Dynamic Textures 99

Figure 3. River. From top to bottom: Samples of the original sequence, corresponding samples of the compressed sequence (compression
ratio: 2.53), samples of extrapolated sequence (using n = 50 components, τ = 120, m = 170 × 115), compression error as a function of the
dimension of the state space n, and extrapolation error as a function of the length of the training set τ . The data set used comes from the MIT
Temporal Texture database.

however, for highly complex scenes (e.g. talking- question is how long should the input sequence be in
face, smoke), it is not monotonic. In these simula- order to capture the temporal dynamics of the process?
tions x0 was set to x̂(1), which explains why the first To answer this question experimentally we plot the pre-
100 frames of the 300 synthesized frames resemble diction error at the bottom right of Figs. 3 to 7 as a func-
the ones of the training sequence. Notice that the ex- tion of the length, τ , of the input (training) sequence.
ample of the talking face (Fig. 7) has been included This means that for each length, τ , we predict the frame
to show the output of the proposed technique when τ + 1 (not part of the training set) and compute the pre-
the underlying hypothesis of the input sequence be- diction error per pixel in gray levels. We do so many
ing a realization of a second-order stationary process times in order to infer the statistics of the prediction er-
is violated. Of course the model fails to capture the ror, i.e. mean and variance at each τ . Figure 8 shows an
non-stationary nature of the sequence, giving rise to a error-bar plot including mean and standard deviation of
synthesized sequence that shows some artifacts. This the prediction error per pixel for the steam sequence.
is, therefore, an example where our technique fails. The average error decreases and becomes stable after
Nevertheless, it is surprising to note that such a simple approximately 80 frames. Notice that the plot of Fig. 8
model can be pushed so far in modeling complex visual has also the meaning of model verification in the sense
phenomena. that this plot of the prediction error validates a posteri-
As explained in Section 4, we choose the model or- ori the model inferred with a sub-optimal solution, and
der n and learn the parameters of the model. A crucial is informative for challenging the model.
100 Doretto et al.

Figure 4. Smoke. From top to bottom: Samples of the original sequence, corresponding samples of the compressed sequence (compression
ratio: 2.53), samples of extrapolated sequence (using n = 30 components, τ = 150, m = 170 × 120), compression error as a function of the
dimension of the state space n, and extrapolation error as a function of the length of the training set τ . The data set used comes from the MIT
Temporal Texture database.

Another important parameter to compare various Soatto, 2002). Notice that, in certain cases correspond-
texture synthesis models is the time it takes to synthe- ing to certain natural phenomena (e.g. the sequence
size them. It is well established that models using Gibbs of smoke-far) it may be possible, and actually cor-
sampling (Zhu et al., 1997) and other methods to draw rect, for the learning process to return a marginally
samples from complex distributions are computation- stable system, capturing the “repetitive” nature of the
ally intensive. Moreover, there is always uncertainty input sequence. In fact, the system captures the ex-
on whether the samples have converged. Determinis- act nature of the ongoing process of the training set
tic methods to extend and extrapolate sequences have and generalizes it in time during a synthetic simula-
to go back and query the input texture in one way or tion. Analytically, the poles of the training sequences
another to obtain information that generates the next can be very close to the unit circle, and for the case of
frame (Efros and Leung, 1999; Wei and Levoy, 2000).14 “unstable” dynamic textures, in order to make stable
In our model, learning is performed in closed-form (30 synthesis simulations, we just relocate the unstable sys-
seconds for 100 graylevel samples), and synthesis is tem poles within the unit circle. To accomplish this task
instantaneous (frame-rate), even in our Matlab imple- we found that by simply reducing to 0.99 the distance
mentation. Moreover, we can even control the number of the unstable poles from the origin (while maintain-
of parameters to obtain a particular synthesis speed, ing their phase constant), we obtain stable synthesized
and change the model parameters (e.g. the eigenvalues sequences that very well resemble the original training
of Â) to manipulate the original dynamics (Doretto and set.
 Dynamic Textures 101

Figure 5. Steam. From top to bottom: Samples of the original sequence, corresponding samples of the compressed sequence (compression
ratio: 2.53), samples of extrapolated sequence (using n = 30 components, τ = 120, m = 176 × 96), compression error as a function of the
dimension of the state space n, and extrapolation error as a function of the length of the training set τ . The data set used comes from the MIT
Temporal Texture database.

Finally, Figs. 9 and 10 show some more results on tor y(t), at time t, in Eq. (1), contains the three unfolded
synthesis. Here, the dimension of the state has been set RGB channels ordered one after the other. Represen-
to n = 50, and x0 has been drawn from a zero-mean tation of color in a more suitable space (Hunt, 1996)
Gaussian distribution with covariance inferred from the may lead to a more efficient use of our model in terms
estimated state X̂ (τ ). For the experiments in Fig. 9, the of its ability to capture information from the training
training sequences have been borrowed again from the sequence.
MIT Temporal Texture database, the length of these
sequences ranges from τ = 100 to τ = 150 frames, and 5.2. Recognition
the synthesized sequences are 300 frames long. For the
experiments in Fig. 10, the training sets are color se- According to our definition in Section 2.1, each tex-
quences that have been captured by the authors except ture is characterized by an ARMA model. Therefore,
for the fire sequence that comes from the Artbeats in order to compare textures, we need to first define a
Digital Film Library.15 The length of the sequences base measure in the space of linear dynamical systems,
is τ = 150 frames, the frames are 320 × 220 pixels, and then to characterize probability distributions in that
and the synthesized sequences are 300 frames long. space.
The extension of the learning algorithm to the case of Defining an appropriate base measure in the space
color images can be done in several ways. The one we of ARMA models is not trivial, since each model en-
used for our experiments implies that the column vec- tails a combination of an input density and state and
102 Doretto et al.

Figure 6. Toilet. From top to bottom: Samples of the original sequence, corresponding samples of the compressed sequence (compression
ratio: 2.53), samples of extrapolated sequence (using n = 30 components, τ = 120, m = 170 × 115), compression error as a function of the
dimension of the state space n, and extrapolation error as a function of the length of the training set τ . The data set used comes from the MIT
Temporal Texture database.

output transition matrices that have a very particular define the model (1). Now, let p1 and p2 be two p.d.f.s
Riemannian structure (they are not a linear space). We that correspond to two different dynamic textures. The
should define an inner product in the space of models K-L divergence, KL( p1 ' p2 ), between p1 and p2 is de-
.
(which involves Stiefel manifolds), and a distance as fined as KL( p1 ' p2 ) = limτ →∞ KLτ ( p1 ' p2 ) where
the length of the geodesic joining two models. This is KLτ ( p1 ' p2 ) = τ1 E p1 [log( p1 ( I¯τ1 )/ p2 ( I¯τ1 ))] and E p1 [·]
beyond the scope of this paper, and we refer the reader is the expectation taken with respect to p1 .
to Bissacco et al. (2001) and Saisan et al. (2001) for In Fig. 11 we display the distance, the quantity
details on how to do so. Here we compute the Kullback- KLτ ( p1 ' p2 ), between different dynamic textures plot-
Leibler divergence between different realizations of the ted against the length τ . We have taken different re-
textures and show how similar textures cluster together alizations of the textures river and steam and
in model space. The problem is formalized as follows. have computed the distance of the former realiza-
Let I (t), t = 0, 1, . . . be an infinitely long sequence tions against themselves and the latter.16 It is evi-
of images. This can be modeled as a stochastic pro- dent that alike textures tend to cluster together. There-
cess which takes values in a subset of Rm for an m- fore, in principle, a comprehensive database of param-
.
dimensional image. Let I¯τ = (I (1), I (2), . . . , I (τ )) be eters learned from commonly occurring dynamic tex-
a sequence of images and let p( I¯τ ) be the correspond- tures can be maintained and a new temporal sequence
ing probability density function (p.d.f.). The p.d.f. can be categorized after learning its parameters and
p( I¯τ ) is completely determined by the parameters that computing the distance. Notice that it would not be
 Dynamic Textures 103

Figure 7. Talking-face. From top to bottom: Samples of the original sequence, corresponding samples of the compressed sequence (compression
ratio: 2.53), samples of extrapolated sequence (using n = 40 components, τ = 125, m = 105 × 170), compression error as a function of the
dimension of the state space n, and extrapolation error as a function of the length of the training set τ . This sequence is chosen to challenge the
model, since it violates the hypothesis of being a realization of a second-order stationary process. The result shows some artifacts, meaning that
not all the information has been captured by the second-order statistics.

easy to build a recognition framework for dynamic pro- 5.3. Compression

cesses using procedural techniques like Schodl et al.
(2000) and Wei and Levoy (2000). An extensive as- In this section we present a preliminary comparison
sessment of the recognition capabilities of our system between storage requirements for the estimated para-
as well as extensions to non-global representations are meters relative to the original space requirement of the
beyond the scope of the paper. Some of these issues texture sequences, to get an estimate of the sequence
are discussed in Saisan et al. (2001) and Bissacco et al. compression capabilities of our model. A thorough as-
(2001). sessment of the compression capabilities of this model
104 Doretto et al.

Figure 8. Model verification: To verify the quality of the model learned, we have used a fixed number of principal components in the
representation (20) and considered sub-sequences of the original data set of length varying from 10 to 120. We have used such sub-sequences to
learn the parameters of the model in the Maximum-Likelihood sense, and then used the model to predict the next image. The average prediction
error per pixel is shown as a function of the length of the training sequence (for the steam sequence), expressed in gray levels within a range
of 256 levels. The average error per pixel decreases and becomes stable after about 80 frames. Mean and standard deviation for 100 trials are
shown as an error-bar plot.

is a research program in its own right. Our intention, tive compression power comes especially when long
here, is to point out the potential of the model for sequences are considered, i.e. when τ ( n, since the
compression, as a further motivation for the model. matrix C is responsible for the higher storage oc-
The storage requirement of the original dataset is cupancy while the other components are negligible
O(mτ ), while the components of the model that are nec- (it is enough to notice that the sequence has to be
essary to re-create an approximation of the sequence 15000 frames long to have n v τ = mn, when m = 100×
are A, C, Q and the input sequence v(t). Therefore, one 100).
would need n 2 numbers (for Â), m × n − n(n − 1)/2 Of course, a more systematic evaluation of the poten-
(for Ĉ, counting the orthogonality constraints), n × n v tial of this model for compression is due. We would like
numbers for Q̂ and, finally, n v × τ numbers for the to point out that our algorithm provides compression
input sequence v̂(t). Thus, the storage requirement based on the temporal characteristics, and therefore it
of our model is O(mn + n 2 + nn v + n v τ ), where operates on top of MPEG encoding and provides fur-
n + m, τ > n, and n v is the effective rank of Q̂. If ther compression. For very long sequences (large τ ),
we consider the fact that typical values for accept- the algorithm presented above can be modified in order
able quality are in the order of n = 30 and n v = 20, to avoid computing the SVD of a very large matrix. In
it is immediate to convince ourselves that the effec- particular, the model can be identified from a shorter
 Dynamic Textures 105

Figure 9. Fountain, plastic, river-far, smoke-far. (a) fountain sequence (τ = 100, m = 150 × 90), (b) plastic sequence (τ = 119,
m = 190 × 148), (c) river-far sequence (τ = 120, m = 170 × 115), (d) smoke-far sequence (τ = 150, m = 170 × 115). For each of
them the top row are samples of the original sequence (borrowed from the MIT Temporal Texture database), the bottom row shows samples
of the synthesized sequence. All the data are available on-line at http://www.cs.ucla.edu/∼doretto/projects/dynamic-
textures.html.
106 Doretto et al.

Figure 10. Fire, color-fountain, ocean, water [COLOR]. (a) fire sequence (τ = 150, m = 360 × 243), (b) color-fountain sequence
(τ = 150, m = 320 × 220), (c) ocean sequence (τ = 150, m = 320 × 220), (d) water sequence (τ = 150, m = 320 × 220). For each of them
the top row are samples of the original sequence, the bottom row shows samples of the synthesized sequence. All the data are available on-line
at http://www.cs.ucla.edu/∼doretto/projects/dynamic-textures.html.
 Dynamic Textures 107

Figure 11. The figure demonstrates that textures belonging to the same class tend to cluster together in the sense of Kullback-Leibler. In
particular for this figure distances are computed amongst three realizations of the river sequence and three of the steam sequence w.r.t.
the former. The graphs on top refer to “steam w.r.t. river” type of distances and the ones below refer to the “river w.r.t. river” type. The K-L
divergences are computed using Monte-Carlo methods.

subsequence, and then the identified model can be white zero-mean Gaussian noise can capture com-
used to compute the input (in innovation form (Van plex visual phenomena. The algorithm is simple to
Overschee and De Moor, 1993)) using a simple linear implement, efficient to learn and fast to simulate.
Kalman filter. For real-time transmission or broadcast- The generative model we propose can be used as a
ing, the innovation can be estimated in real-time using a compact representation of the data, and is suitable for
Kalman filter and transmitted in lieu of the sequence of control/manipulation, and recognition. Therefore, the
images, after the initial model is identified and transmit- framework presented in this paper stimulates several
ted to the receiver. This would ensure real-time coding areas of future investigation, ranging from video com-
and decoding—after an initial batch—for applications pression, classification, recognition to image-based
such as teleconferencing or remote video broadcasting. rendering, synthesis, and editing of image sequences.

6. Discussion Acknowledgments

We have introduced a novel representation of This research is supported by NSF grants IIS-9876145,
dynamic textures and associated algorithms to per- IIS-9877127, IIS-0208197, ECS-0200511 and DMS-
form learning and synthesis of sequences from train- 0072538, Italian Space Agency and MIUR National
ing data. Somewhat surprisingly, even the simplest Project “Identification and Control for Industrial Sys-
choice of a first-order ARMA model driven by tems.” We wish to thank Prabhakar Pundir for his
108 Doretto et al.

assistance in reviewing current literature and with data 13. ftp://whitechapel.media.mit.edu/pub/szummer/tem-

collection. A shorter version of this paper appeared poral-texture/
in the Proc. of the Intl. Conf. on Computer Vision, 14. In Wei and Levoy (2000) for each new pixel a search is
conducted for a similar neighborhood pattern in the original
Vancouver, BC, July 2001. texture.
15. http://www.artbeats.com.
16. We obtain a similar plot if we compute the distance of the latter
against the former although the K-L divergence by definition is
Notes
not commutative.

1. For instance, in stereo and structure from motion one assumes

that (most of) the scene has Lambertian reflection properties, and References
exploits such an assumption to establish correspondence and
estimate shape. Similarly, in shape from shading one assumes Amari, S. and Cardoso, J. 1997. Blind source separation—
constant albedo and exploits changes in irradiance to recover semiparametric statistical approach. IEEE Transactions on Signal
shape. Processing, 45:692–700.
2. For example, a sequence of images of the sea at sunset could have Arun, K.S. and Kung, S.Y. 1990. Balanced approximation of
been originated by a very complex and dynamic shape (the sur- stochastic systems. SIAM Matrix Analysis and Applications, 42–
face of the sea) with constant reflection properties (homogeneous 68.
material, water), but also by a very simple shape (e.g. the plane Bar-Joseph, Z., El-Yaniv, R., Lischinski, D., and Werman, M. 2001.
of the television monitor) with a non-homogeneous radiance (the Texture mixing and texture movie synthesis using statistical learn-
televised spatio-temporal signal). Similarly, the appearance of a ing. IEEE Transactions on Visualization and Computer Graphics,
moving Lambertian cube can be mimicked by a spherical mir- 7(2):120–135.
ror projecting a light distribution to match the albedo of the Barzel, R. 1992. Physically-Based Modeling for Computer Graphics:
cube. A Structured Approach. Academic Press, San Diego, CA.
3. A stochastic process is stationary (of order k) if the joint statistics Bauer, D., Deistler, M., and Scherrer, W. 1999. Consistency and
(up to order k) are time-invariant. For instance a process {I (t)} asymptotic normality of some subspace algorithms for systems
.
is second-order stationary if its mean I¯ = E[I (t)] is constant without observed inputs. Automatica, 35(7):1243–1254.
and its covariance E[(I (t1 ) − I¯)(I (t2 ) − I¯)] only depends upon Bigun, J. and du Buf, J.M. 1994. N -folded symmetries by complex
t 2 − t1 . moments in gabor space and their application to unsupervised
4. ARMA stands for auto-regressive moving average. texture segmentation. IEEE Transactions on Pattern Analysis and
5. IID stands for independent and identically distributed. Machine Intelligence, 16(1):80–87.
6. Most common methods use Gabor filters (Bigun and du Buf, Bissacco, A., Chiuso, A., Ma, Y., and Soatto, S. 2001. Recogni-
1994), and steerable filters (Freeman and Adelson, 1991; tion of human gaits. In Proceedings of IEEE Computer Society
Simoncelli et al., 1992; Heeger and Bergen, 1995). One could Conference on Computer Vision and Pattern Recognition, Kauai,
also infer and choose the best filters as part of the learning process HA.
(Zhu et al., 1997). Blake, A. and Isard, M. 1998. Active Contours. Springer Verlag:
7. This distribution can be inferred from the physics of the imag- Berlin.
ing device. For CCD sensors, for instance, a good approxima- Chiuso, A. and Picci, G. To appear. Asymptotic variance of subspace
tion is a Poisson distribution with intensity related to the av- techniques. In Journal of Econometrics.
erage photon count. The reason for including the term w(t) in Cross, G. and Jain, A. 1983. Markov random field texture models.
modeling dynamic textures is because it is necessary for the IEEE Transactions on Pattern Analysis and Machine Intelligence,
covariance sequence of y(t) to be arbitrary. Moreover, stan- 5:25–40.
dard results in stochastic realization (Ljung, 1987) state that, de Bonet, J. and Viola, P. 1998. Multiresolution sampling procedure
under some additional hypotheses (see Section 4), second- for analysis and synthesis of texture images. In Proceedings IEEE
order stationary processes can be represented as the output of Conf. Computer Vision and Pattern Recognition.
models of the type (1), which include the output noise term Dempster, A.P., Laird, N.M., and Rubin, D.B. 1977. Maximum like-
w(t). lihood from incomplete data via the EM algorithm. J. R. Statist.
8. φ(·) indicates the combination of the output of the n filters Soc. B, 39:185–197.
{φα } respectively applied to each of the n state components; Doretto, G. and Soatto, S. 2002. Editable dynamic tex-
see Section 2.2 for details. tures. Technical Report 020001, UCLA Computer Science
9. In our experiments we have m = 320 × 220 for the color se- Department.
quences, and m = 170 × 110 for the graylevel sequences, while Ebert, D., Carlson, W., and Parent, R. 1994. Solid spaces and inverse
n ranges from 10 to 50, and n v ranges from 10 to n. particle systems for controlling the animation of gases and fluids.
10. Fitzgibbon (2001), which appeared after this paper was submit- The Visual Computer, 10(4):179–190.
ted, also advocates the use of AR models for registering non-rigid Efros, A. and Leung, T. 1999. Texture synthesis by non-parametric
scenes. sampling. In Seventh International Conference on Computer
11. As an anonymous reviewer remarked, this would result in an Vision, Corfu, Greece.
approach similar to Video Textures (Schodl et al. (2000)). Fitzgibbon, A.W. 2001. Stochastic rigidity: Image registration for
12. Movies available online at http://www.cs.ucla.edu/ nowhere-static scenes. In Proc. of the Intl. Conf. on Comp. Vis.,
∼doretto/projects/dynamic-textures.html. pp. 662–670.
 Dynamic Textures 109

Fournier, A. and Reeves, W. 1986. A simple model of ocean waves. Peachey, D. 1986. Modeling waves and surf. In SIGGRAPH
In ACM SIGGRAPH Proceedings, pp. 75–84. Conference Proceedings, pp. 65–74.
Freeman, W. and Adelson, E. 1991. The design and use of steer- Popat, K. and Picard, R. 1993. Novel cluster-based probability
able filters. IEEE Transactions on Pattern Analysis and Machine model for texture synthesis, classification, and compression. In
Intelligence, 13(9):891–906. Proceedings SPIE Visual Communications and Image Processing,
Golub, G. and Van Loan, C. 1989. Matrix Computations, 2nd edn. Boston.
Johns Hopkins University Press: Baltimore, MD. Portilla, J. and Simoncelli, E. 1999. Texture representation and
Hassner, M. and Sklansky, J. 1981. The use of markov random fields synthesis using correlation of complex wavelet coefficient
as models of texture. In Image Modeling. Academic Press: San magnitudes. In CSIC, Madrid.
Diego, CA. Reeves, W. 1983. Particle systems: A technique for modeling a class
Heeger, D. and Bergen, J. 1995. Pyramid-based texture analysis /syn- of fuzzy objects. ACM Trans. Graphics, 2:91–108.
thesis. In ACM SIGGRAPH Conference Proceedings. Rissanen, J. 1978. Modeling by shortest data description. Automat-
Hodgins, J.K. and Wooten, W.L. 1998. Animating human athletes. ica, 14:465–471.
In Robotics Research: The Eighth International Symposium, Y. Saisan, P., Doretto, G., Wu, Y.N., and Soatto, S. 2001. Dynamic
Shirai and S. Hirose (Eds.). Springer-Verlag: Berlin, Germany, texture recognition. In Proceedings of IEEE Computer Society
pp. 356–367. Conference on Computer Vision and Pattern Recognition, Kauai,
Hunt, R.W. 1996. The Reproduction of Colour, 5th edn. Fisher Books. HA.
Julesz, B. 1962. Visual pattern discrimination. IRE Trans. Info Schödl, A., Szeliski, R., Salesin, D., and Essa, I. 2000. Video textures.
Theory, IT-8. In Proceedings of ACM SIGGRAPH Conference, New Orleans,
Kailath, T. 1980. Linear Systems. Prentice Hall: Englewood Cliffs, LA.
NJ. Simoncelli, E., Freeman, W., Adelson, E., and Heeger, D. 1992.
Kirsch, A. 1996. An Introduction to the Mathematical Theory of Shiftable multi-scale transforms. IEEE Trans. Information Theory,
Inverse Problems. Springer-Verlag: New York. 38(2):587–607.
Lindquist, A. and Picci, G. 1979. The stochastic realization problem. Sims, K. 1990. Particle animation and rendering using data parallel
SIAM J. Control Optim, 17:365–389. computation. Computer Graphics, 24(4):405–413.
Liu, J., Chen, R., and Logvinenko, T. 2000. A theoretical framework Stam, J. and Fiume, E. 1995. Depicting fire and other gaseous phe-
for sequential importance sampling and resampling. Technical nomena using diffusion processes. In SIGGRAPH Conference Pro-
Report, Stanford University, Department of Statistics. ceedings, pp. 129–136.
Ljung, L. 1987. System Identification—Theory for the User. Prentice Szummer, M. and Picard, R.W. 1996. Temporal texture modeling. In
Hall: Englewood Cliffs, NJ. IEEE International Conference on Image Processing, Lausanne,
Mallat, S. 1989. A theory of multiresolution signal decomposi- Switzerland, vol. 3.
tion: The wavelet representation. IEEE Transactions on Pattern Van Overschee, P. and De Moor, B. 1993. Subspace algorithms
Analysis and Machine Intelligence, 11:674–693. for the stochastic identification problem. Automatica, 29:649–
Mumford, D. and Gidas, B. 1998. Stochastic models for generic im- 660.
ages. Technical Report, Division of Applied Mathematics, Brown Van Overschee, P. and De Moor, B. 1994. N4sid: Subspace algo-
University. rithms for the identification of combined deterministic-stochastic
Nelson, R.C. and Polana, R. 1992. Qualitative recognition of motion systems. Automatica, 30:75–93.
using temporal texture. Computer Vision, Graphics, and Image Wei, L.Y. and Levoy, M. 2000. Fast texture synthesis using
Processing. Image Understanding, 56(1):78–89. tree structured vector quantization. In SIGGRAPH Conference
Paget, R. and Longstaff, D. 1996. A nonparametric multiscale Proceedings.
markov random field model for synthesising natural textures. In Zhu, S., Wu, Y., and Mumford, D. 1997. Minimax entropy princi-
Fourth International Symposium on Signal Processing and its ple and its application to texture modeling. Neural Computation,
Applications, 2:744–747. 9:1627–1660.