A Spectral Algorithm for Learning Hidden Markov Models

Daniel Hsu Sham M. Kakade Tong Zhang

UC San Diego Toyota Technological Institute at Chicago Rutgers University
[email protected] [email protected] [email protected]

Abstract The situation is in many ways analogous to learning mix-

ture distributions with samples from the underlying distri-
Hidden Markov Models (HMMs) are one of the bution. There, the general problem is believed to be hard.
most fundamental and widely used statistical tools However, much recent progress has been made when certain
for modeling discrete time series. In general, learn- separation assumptions are made with respect to the compo-
ing HMMs from data is computationally hard (un- nent mixture distributions (e.g. [Das99, DS07, VW02, CR08,
der cryptographic assumptions), and practitioners BV08]). Roughly speaking, these separation assumptions
typically resort to search heuristics which suffer imply that with high probability, given a point sampled from
from the usual local optima issues. We prove that the distribution, we can recover which component distribu-
under a natural separation condition (bounds on tion generated this point. In fact, there is prevalent sentiment
the smallest singular value of the HMM param- that we are often only interested in clustering the data when
eters), there is an efficient and provably correct such a separation condition holds. Much of the theoretical
algorithm for learning HMMs. The sample com- work here has been on how small this separation need be in
plexity of the algorithm does not explicitly depend order to permit an efficient algorithm to recover the model.
on the number of distinct (discrete) observations— We present a simple and efficient algorithm for learning
it implicitly depends on this quantity through spec- HMMs under a certain natural separation condition. We pro-
tral properties of the underlying HMM. This makes vide two results for learning. The first is that we can ap-
the algorithm particularly applicable to settings with proximate the joint distribution over observation sequences
a large number of observations, such as those in of length t (here, the quality of approximation is measured
natural language processing where the space of ob- by total variation distance). As t increases, the approxima-
servation is sometimes the words in a language. tion quality degrades polynomially. Our second result is on
The algorithm is also simple: it employs only a approximating the conditional distribution over a future ob-
singular value decomposition and matrix multipli- servation, conditioned on some history of observations. We
cations. show that this error is asymptotically bounded—i.e. for any
t, conditioned on the observations prior to time t, the error in
predicting the t-th outcome is controlled. Our algorithm can
1 Introduction be thought of as ‘improperly’ learning an HMM in that we do
not explicitly recover the transition and observation models.
Hidden Markov Models (HMMs) [BE67] are the workhorse However, our model does maintain a hidden state representa-
statistical model for discrete time series, with widely diverse tion which is closely (in fact, linearly) related to the HMM’s,
applications including automatic speech recognition, natural and can be used for interpreting the hidden state.
language processing (NLP), and genomic sequence model- The separation condition we require is a spectral condi-
ing. In this model, a discrete hidden state evolves according tion on both the observation matrix and the transition matrix.
to some Markovian dynamics, and observations at particular Roughly speaking, we require that the observation distribu-
time depend only on the hidden state at that time. The learn- tions arising from distinct hidden states be distinct (which
ing problem is to estimate the model only with observation we formalize by singular value conditions on the observa-
samples from the underlying distribution. Thus far, the pre- tion matrix). This requirement can be thought of as being
dominant learning algorithms have been local search heuris- weaker than the separation condition for clustering in that the
tics, such as the Baum-Welch / EM algorithm [BPSW70, observation distributions can overlap quite a bit—given one
DLR77]. observation, we do not necessarily have the information to
It is not surprising that practical algorithms have resorted determine which hidden state it was generated from (unlike
to heuristics, as the general learning problem has been shown in the clustering literature). We also have a spectral condition
to be hard under cryptographic assumptions [Ter02]. Fortu- on the correlation between adjacent observations. We believe
nately, the hardness results are for HMMs that seem divorced both of these conditions to be quite reasonable in many prac-
from those that we are likely to encounter in practical appli- tical applications. Furthermore, given our analysis, exten-
cations. sions to our algorithm which relax these assumptions should
be possible. tory) do not asymptotically diverge. This result was proven
The algorithm we present has both polynomial sample in [EDKM07] when an approximate model is already known.
and computational complexity. Computationally, the algo- Roughly speaking, the reason this error does not diverge is
rithm is quite simple—at its core is a singular value decom- that the previous observations are always revealing informa-
position (SVD) of a correlation matrix between past and fu- tion about the next observation; so with some appropriate
ture observations. This SVD can be viewed as a Canonical contraction property, we would not expect our errors to di-
Correlation Analysis (CCA) [Hot35] between past and fu- verge. Our work borrows from this contraction analysis.
ture observations. The sample complexity results we present Among recent efforts in various communities [ARJ03,
do not explicitly depend on the number of distinct observa- VWM07, ZJ07, CC08], the only previous efficient algorithm
tions; rather, they implicitly depend on this number through shown to PAC-learn HMMs in a setting similar to ours is due
spectral properties of the HMM. This makes the algorithm to [MR06]. Their algorithm for HMMs is a specialization of
particularly applicable to settings with a large number of ob- a more general method for learning phylogenetic trees from
servations, such as those in NLP where the space of observa- leaf observations. While both this algorithm and ours rely
tions is sometimes the words in a language. on the same rank condition and compute similar statistics,
they differ in two significant regards. First, [MR06] were
1.1 Related Work not concerned with large observation spaces, and thus their
algorithm assumes the state and observation spaces to have
There are two ideas closely related to this work. The first the same dimension. In addition, [MR06] take the more am-
comes from the subspace identification literature in control bitious approach of learning the observation and transition
theory [Lju87, OM96, Kat05]. The second idea is that, rather matrices explicitly, which unfortunately results in a less sta-
than explicitly modeling the hidden states, we can represent ble and less sample-efficient algorithm that injects noise to
the probabilities of sequences of observations as products of artificially spread apart the eigenspectrum of a probability
matrix observation operators, an idea which dates back to the matrix. Our algorithm avoids recovering the observation and
literature on multiplicity automata [Sch61, CP71, Fli74]. transition matrix explicitly1 , and instead uses subspace iden-
The subspace identification methods, used in control the- tification to learn an alternative representation.
ory, use spectral approaches to discover the relationship be-
tween hidden states and the observations. In this literature,
the relationship is discovered for linear dynamical systems 2 Preliminaries
such as Kalman filters. The basic idea is that the relation- 2.1 Hidden Markov Models
ship between observations and hidden states can often be dis-
covered by spectral/SVD methods correlating the past and The HMM defines a probability distribution over sequences
future observations (in particular, such methods often do a of hidden states (ht ) and observations (xt ). We write the set
CCA between the past and future observations). However, of hidden states as [m] = {1, . . . , m} and set of observations
algorithms presented in the literature cannot be directly used as [n] = {1, . . . , n}, where m ≤ n.
to learn HMMs because they assume additive noise models Let T ∈ Rm×m be the state transition probability matrix
with noise distributions independent of the underlying states, with Tij = Pr[ht+1 = i|ht = j], O ∈ Rn×m be the obser-
and such models are not suitable for HMMs (an exception vation probability matrix with Oij = Pr[xt = i|ht = j], and
is [ARJ03]). In our setting, we use this idea of performing a ~π ∈ Rm be the initial state distribution with ~πi = Pr[h1 =
CCA between past and future observations to uncover infor- i]. The conditional independence properties that an HMM
mation about the observation process (this is done through satisfies are: 1) conditioned on the previous hidden state, the
an SVD on a correlation matrix between past and future ob- current hidden state is sampled independently of all other
servations). The state-independent additive noise condition events in the history; and 2) conditioned on the current hid-
is avoided through the second idea. den state, the current observation is sampled independently
The second idea is that we can represent the probability from all other events in the history. These conditional inde-
of sequences as products of matrix operators, as in the lit- pendence properties of the HMM imply that T and O fully
erature on multiplicity automata [Sch61, CP71, Fli74] (see characterize the probability distribution of any sequence of
[EDKM05] for discussion of this relationship). This idea states and observations.
was re-used in both the Observable Operator Model of [Jae00] A useful way of computing the probability of sequences
and the Predictive State Representations of [LSS01], both is in terms of ‘observation operators’, an idea which dates
of which are closely related and both of which can model back to the literature on multiplicity automata (see [Sch61,
HMMs. In fact, the former work by [Jae00] provides a non- CP71, Fli74]). The following lemma is straightforward to
iterative algorithm for learning HMMs, with an asymptotic verify (see [Jae00, EDKM07]).
analysis. However, this algorithm assumed knowing a set of
‘characteristic events’, which is a rather strong assumption Lemma 1. For x = 1, . . . , n, define
that effectively reveals some relationship between the hid- Ax = T diag(Ox,1 , . . . , Ox,m ).
den states and observations. In our algorithm, this problem
is avoided through the first idea. 1
In Appendix A, we discuss the key step in [MR06], and also
Some of the techniques in the work in [EDKM07] for show how to use their technique in conjunction with our algorithm
tracking belief states in an HMM are used here. As discussed to recover the HMM observation and transition matrices. Our algo-
earlier, we provide a result showing how the model’s condi- rithm does not rely on this extra step—we believe it to be generally
tional distributions over observations (conditioned on a his- unstable—but it can be taken if desired.
For any t: with complicated mixing conditions for Markov chains in
our sample complexity calculation, as these conditions are
Pr[x1 , . . . , xt ] = ~1>
m Axt . . . Ax1 ~
π. not essential to the main ideas we present. In practice, how-
Our algorithm learns a representation that is based on this ever, one should use the full sequences to form the proba-
observable operator view of HMMs. bility estimation tables required by our algorithm. In such
scenarios, a single long sequence is sufficient for learning,
2.2 Notation and the effective sample size can be simply discounted by
As already used in Lemma 1, the vector ~1m is the all-ones the mixing rate of the underlying Markov chain.
vector in Rm . We denote by x1:t the sequence (x1 , . . . , xt ), Our goal is to derive accurate estimators for the cumula-
and by xt:1 its reverse (xt , . . . , x1 ). When we use a sequence tive (joint) distribution Pr[x1:t ] and the conditional distribu-
as a subscript, we mean the product of quantities indexed by tion Pr[xt |x1:t−1 ] for any sequence length t. For the condi-
the sequence elements. So for example, the probability cal- tional distribution, we obtain an approximation that does not
culation in Lemma 1 can be written ~1> depend on t, while for the joint distribution, the approxima-
m Axt:1 ~
π . We will use
~ht to denote a probability vector (a distribution over hidden tion quality degrades gracefully with t.
states), with the arrow distinguishing it from the random hid-
den state variable ht . Additional notation used in the theo- 3 Observable Representations of Hidden
rem statements and proofs is listed in Table 1. Markov Models
2.3 Assumptions A typical strategy for learning HMMs is to estimate the ob-
We assume the HMM obeys the following condition. servation and transition probabilities for each hidden state
(say, by maximizing the likelihood of a sample). However,
Condition 1 (HMM Rank Condition). ~π > 0 element-wise, since the hidden states are not directly observed by the learner,
and O and T are rank m. one often resorts to heuristics (e.g. EM) that alternate be-
The conditions on ~π and T are satisfied if, say, the Markov tween imputing the hidden states and selecting parameters O b
chain specified by T is ergodic and ~π is its stationary distri- and Tb that maximize the likelihood of the sample and cur-
bution. The condition on O rules out the problematic case in rent state estimates. Such heuristics can suffer from local
which some state i has an output distribution equal to a con- optima issues and require careful initialization (e.g. an accu-
vex combination (mixture) of some other states’ output dis- rate guess of the hidden states) to avoid failure.
tributions. Such a case could cause a learner to confuse state However, under Condition 1, HMMs admit an efficiently
i with a mixture of these other states. As mentioned before, learnable parameterization that depends only on observable
the general task of learning HMMs (even the specific goal quantities. Because such quantities can be estimated from
of simply accurately modeling the distribution probabilities data, learning this representation avoids any guesswork about
[Ter02]) is hard under cryptographic assumptions; the rank the hidden states and thus allows for algorithms with strong
condition is a natural way to exclude the malicious instances guarantees of success.
created by the hardness reduction. This parameterization is natural in the context of Ob-
The rank condition of O can be relaxed through a simple servable Operator Models [Jae00], but here we emphasize
modification of our algorithm that looks at multiple observa- its connection to subspace identification.
tion symbols simultaneously to form the probability estima-
tion tables. For example, if two hidden states have identical 3.1 Definition
observation probability in O but different transition probabil-
Our HMM representation is defined in terms of the following
ity in T , then they may be differentiated by using two con-
vector and matrix quantities:
secutive observations. Although our analysis can be applied
in this case with minimal modifications, for clarity, we only [P1 ]i = Pr[x1 = i]
state our results for an algorithm that estimates probability
[P2,1 ]ij = Pr[x2 = i, x1 = j]
tables with rows and columns corresponding to single obser-
vations. [P3,x,1 ]ij = Pr[x3 = i, x2 = x, x1 = j] ∀x ∈ [n],
2.4 Learning Model where P1 ∈ Rn is a vector, and P2,1 ∈ Rn×n and the
Our learning model is similar to those of [KMR+ 94, MR06] P3,x,1 ∈ Rn×n are matrices. These are the marginal proba-
for PAC-learning discrete probability distributions. We as- bilities of observation singletons, pairs, and triples.
sume we can sample observation sequences from an HMM. The representation further depends on a matrix U ∈ Rn×m
In particular, we assume each sequence is generated start- that obeys the following condition.
ing from the same initial state distribution (e.g. the stationary
distribution of the Markov chain specified by T ). This setting Condition 2 (Invertibility Condition). U > O is invertible.
is valid for practical applications including speech recogni-
tion, natural language processing, and DNA sequence mod- In other words, U defines an m-dimensional subspace
eling, where multiple independent sequences are available. that preserves the state dynamics—this will become evident
For simplicity, this paper only analyzes an algorithm that in the next few lemmas.
uses the initial few observations of each sequence, and ig- A natural choice for U is given by the ‘thin’ SVD of P2,1 ,
nores the rest. We do this to avoid using concentration bounds as the next lemma exhibits.
Lemma 2. Assume ~π > 0 and that O and T have column hidden states (though the following lemma shows that they
rank m. Then rank(P2,1 ) = m. Moreover, if U is the ma- are linearly related). As per Lemma 3, the initial state is
trix of left singular vectors of P2,1 corresponding to non-zero ~b1 = (U > O)~π .
singular values, then range(U ) = range(O), so U ∈ Rn×m
obeys Condition 2. Generally, for any t ≥ 1, given observations x1:t−1 with
Pr[x1:t−1 ] > 0, we define the internal state as:
Proof. Using the conditional independence properties of the
HMM, entries of the matrix P2,1 can be factored as ~bt = ~bt (x1:t−1 ) = Bxt−1:1~b1
.
m X
m ~b> Bx ~b
X ∞ t−1:1 1
[P2,1 ]ij = Pr[x2 = i, x1 = j, h2 = k, h1 = `]
k=1 `=1
The case t = 1 is consistent with the general definition of ~bt
Xm X m because the denominator is ~b> ~ ~ > > −1 (U > O)~π =
∞ b1 = 1m (U O)
= Oik Tk` ~π` [O> ]`j ~1>
m~π = 1. The following result shows how these inter-
k=1 `=1 nal states can be used to compute conditional probabilities
Pr[xt = i|x1:t−1 ].
so P2,1 = OT diag(~π )O> and thus range(P2,1 ) ⊆ range(O).
The assumptions on O, T , and ~π imply that T diag(~π )O> Lemma 4 (Conditional Internal States). Assume the condi-
has linearly independent rows and that P2,1 has m non-zero tions in Lemma 3. Then, for any time t:
singular values. Therefore 1. (Recursive update of states) If Pr[x1:t ] > 0, then
O = P2,1 (T diag(~π )O> )+ Bxt~bt
~bt+1 = ,
(where X + denotes the Moore-Penrose pseudo-inverse of a ~b> Bx ~bt
∞ t

matrix X), which in turn implies range(O) ⊆ range(P2,1 ). 2. (Relation to hidden states)
Thus rank(P2,1 ) = rank(O) = m, and also range(U ) =
range(P2,1 ) = range(O). ~bt = (U > O) ~ht (x1:t−1 )

where [~ht (x1:t−1 )]i = Pr[ht = i|x1:t−1 ] is the condi-

Our algorithm is motivated by Lemma 2 in that we com-
tional probability of the hidden state at time t given the
pute the SVD of an empirical estimate of P2,1 to discover a
observations x1:t−1 ,
U that satisfies Condition 2. We also note that this choice for
U can be thought of as a surrogate for the observation matrix 3. (Conditional observation probabilities)
O (see Remark 5).
Pr[xt |x1:t−1 ] = ~b> ~
∞ B x t bt .
Now given such a matrix U , we can finally define the
observable representation: Remark 5. If U is the matrix of left singular vectors of P2,1
corresponding to non-zero singular values, then U acts much
~b1 = U > P1 like the observation probability matrix O in the following

+ sense:
~b∞ = >
P2,1 U P1
Bx = U > P3,x,1


U > P2,1

+
∀x ∈ [n] . Let ~bt = ~bt (x1:t−1 ) and ~ht = ~ht (x1:t−1 ). Then
Pr[xt = i|x1:t−1 ] = [U~bt ]i = [O~ht ]i .
3.2 Basic Properties
The following lemma shows that the observable representa- To see this, note that U U > is the projection operator to
range(U ). Since range(U ) = range(O) (Lemma 2), we
tion {~b∞ , ~b1 , B1 , . . . , Bn } is sufficient to compute the prob-
abilities of any sequence of observations. have U U > O = O, so U~bt = U (U > O)~ht = O~ht .

Lemma 3 (Observable HMM Representation). Assume the 3.3 Proofs

HMM obeys Condition 1 and that U ∈ Rn×m obeys Condi- Proof of Lemma 3. The first claim is immediate from the fact
tion 2. Then: P1 = O~π . For the second claim, we write P1 in the follow-
ing unusual (but easily verified) form:
1. ~b1 = (U > O)~π .
P1> = ~1> π )O>
m T diag(~
2. ~b> ~ > > −1 .
∞ = 1m (U O) = ~1> > −1
(U > O)T diag(~π )O>
m (U O)
3. Bx = (U > O)Ax (U > O)−1 ∀x ∈ [n]. = ~1> > −1 >
m (U O) U P2,1 .
4. Pr[x1:t ] = ~b> ~
∞ Bxt:1 b1 ∀t ∈ N, x1 , . . . , xt ∈ [n]. The matrix U > P2,1 has linearly independent rows (by the
assumptions on ~π , O, T , and the condition on U ), so
In addition to joint probabilities, we can compute condi-
tional probabilities using the observable representation. We ~b> = P1> (U > P2,1 )+
∞
do so through (normalized) conditional ‘internal states’ that
= ~1> >
m (U O)
−1
(U > P2,1 ) (U > P2,1 )+
depend on a history of observations. We should emphasize
that these states are not in fact probability distributions over = ~1> >
m (U O)
−1
.
To prove the third claim, we first express P3,x,1 in terms of
Ax : Algorithm L EARN HMM(m, N ):
Inputs: m - number of states, N - sample size
P3,x,1 = OAx T diag(~π )O> Returns: HMM model parameterized by
= OAx (U > O)−1 (U > O)T diag(~π )O> {bb1 , bb∞ , B
bx ∀x ∈ [n]}
= OAx (U > O)−1 U > P2,1 . 1. Independently sample N observation triples
> (x1 , x2 , x3 ) from the HMM to form empirical esti-
Again, using the fact that U P2,1 has full row rank,
mates Pb1 , Pb2,1 , Pb3,x,1 ∀x ∈ [n] of P1 , P2,1 , P3,x,1

+ ∀x ∈ [n].
Bx = U > P3,x,1 U > P2,1



+ 2. Compute the SVD of Pb2,1 , and let U b be the matrix of
= (U > O)Ax (U > O)−1 U > P2,1 U > P2,1


left singular vectors corresponding to the m largest
= (U > O)Ax (U > O)−1 . singular values.
The probability calculation in the fourth claim is now readily 3. Compute model parameters:
seen as a telescoping product that reduces to the product in b > Pb1 ,
(a) bb1 = U
Lemma 1. (b) bb∞ = (Pb> U b )+ P1 ,
2,1

(c) B b > Pb3,x,1 (U

bx = U b > Pb2,1 )+ ∀x ∈ [n].
Proof of Lemma 4. The first claim is a simple induction. The
second and third claims are also proved by induction as fol-
lows. The base case is clear from Lemma 3 since ~h1 = ~π and Figure 1: HMM learning algorithm.
~b1 = (U > O)~π , and also ~b> Bx ~b1 = ~1> Ax ~π = Pr[x1 ].
∞ 1 m 1
For the inductive step,

~bt+1 Bxt~bt Bxt (U > O)~ht (U > O)Axt ~ht • Given an observation xt , the ‘internal state’ update is:
= = =
~b> Bx ~bt Pr[xt |x1:t−1 ] Pr[xt |x1:t−1 ]
∞ t
Bbx bbt
>Pr[ht+1 = ·, xt |x1:t−1 ] bbt+1 = t
.
= (U O) bb> B
∞ x bt
Pr[xt |x1:t−1 ]
b b
t

Pr[ht+1 = ·|x1:t ] Pr[xt |x1:t−1 ] • To predict the conditional probability of xt given x1:t−1 :
= (U > O)
Pr[xt |x1:t−1 ]
b> b b
> ~
= (U O) ht+1 (x1:t ) c t |x1:t−1 ] = Pb∞ Bxt bt .
Pr[x
b> b b
x b∞ Bx bt
(the first three equalities follow from the first claim, the in-
ductive hypothesis, and Lemma 3), and Aside from the random sampling, the running time of the
~b> Bx ~bt+1 = ~1> Ax ~ht+1 = Pr[xt+1 |x1:t ] learning algorithm is dominated by the SVD computation of
∞ t+1 m t+1 an n×n matrix. The time required for computing joint prob-
(again, using Lemma 3). ability calculations is O(tm2 ) for length t sequences—same
as if one used the ordinary HMM parameters (O and T ). For
conditional probabilities, we require some extra work (pro-
4 Spectral Learning of Hidden Markov portional to n) to compute the normalization factor. How-
Models ever, our analysis shows that this normalization factor is al-
ways close to 1 (see Lemma 13), so it can be safely omitted
4.1 Algorithm in many applications.
The representation in the previous section suggests the al-
gorithm L EARN HMM(m, N ) detailed in Figure 1, which 4.2 Main Results
simply uses random samples to estimate the model param- We now present our main results. The first result is a guar-
eters. Note that in practice, knowing m is not essential be- antee on the accuracy of our joint probability estimates for
cause the method presented here tolerates models that are observation sequences. The second result concerns the ac-
not exactly HMMs, and the parameter m may be tuned using curacy of conditional probability estimates — a much more
cross-validation. As we discussed earlier, the requirement delicate quantity to bound due to conditioning on unlikely
for independent samples is only for the convenience of our events. We also remark that if the probability distribution is
sample complexity analysis. only approximately modeled as an HMM, then our results
The model returned by L EARN HMM(m, N ) can be used degrade gracefully based on this approximation quality.
as follows:
4.2.1 Joint Probability Accuracy
• To predict the probability of a sequence: Let σm (M ) denote the mth largest singular value of a matrix
M . Our sample complexity bound will depend polynomially
c 1 , . . . , xt ] = bb>
Pr[x ∞ Bxt . . . Bx1 b1 . on 1/σm (P2,1 ) and 1/σm (O).
b b b
 Also, define To interpret this quantity γ, consider any two distribu-
tions over hidden states ~h, b
h ∈ Rm . Then kO(~h − b h)k1 ≥
 
X 
(k) = min Pr[x2 = j] : S ⊆ [n], |S| = n − k , ~ b ~
γkh − hk1 . Regarding h as the true hidden state distribution
and bh as the estimated hidden state distribution, this inequal-
 
j∈S
(1) ity gives a lower bound on the error of the estimated obser-
and let vation distributions under O. In other words, the observation
n0 () = min{k : (k) ≤ /4}. process, on average, reveal errors in our hidden state estima-
In other words, n0 () is the minimum number of observa- tion. [EDKM07] uses this as a contraction property to show
tions that account for about 1 − /4 of the total probability how prediction errors (due to using an approximate model)
mass. Clearly n0 () ≤ n, but it can often be much smaller do not diverge. In our setting, this is more difficult as we
in real applications. For example, in many practical appli- do not explicitly estimate O nor do we explicitly maintain
cations, the frequencies of observation symbols observe a distributions over hidden states.
power law (called Zipf’s law) of the form f (k) ∝ 1/k s , We also need the following assumption, which we dis-
where f (k) is the frequency of the k-th most frequently ob- cuss further following the theorem statement.
served symbol. If s > 1, then (k) = O(k 1−s ), and n0 () = Condition 3 (Stochasticity Condition). For all observations
O(1/(1−s) ) becomes independent of the number of obser- x and all states i and j, [Ax ]ij ≥ α > 0.
vations n. This means that for such problems, our analysis
below leads to a sample complexity bound for the cumula- Theorem 7. There exists a constant C > 0 such that the
tive distribution Pr[x1:t ] that can be independent of n. This following holds. Pick any 0 < , η < 1. Assume the HMM
is useful in domains with large n such as natural language obeys Conditions 1 and 3, and
processing. 
m · n0 () m
N ≥C· 2 +
Theorem 6. There exists a constant C > 0 such that the  σm (O)2 σm (P2,1 )2 σm (O)2 σm (P2,1 )4
following holds. Pick any 0 < , η < 1 and t ≥ 1. Assume 
m (log(2/α))4 (log(2/α))4

1
the HMM obeys Condition 1, and · 2 2+ + · log .
 α 4 α 4 γ 4 α10 γ 4 η
t2
 
m · log(1/η) m · n0 () · log(1/η)
N ≥ C· 2 · + . With probability at least 1 − η, then the model returned by
 σm (O)2 σm (P2,1 )4 σm (O)2 σm (P2,1 )2
L EARN HMM(m, N ) satisfies, for any time t,
With probability at least 1 − η, the model returned by the
algorithm L EARN HMM(m, N ) satisfies KL(Pr[xt |x1 , . . . , xt−1 ] || Pr[x
c t |x1 , . . . , xt−1 ])
X " #
| Pr[x1 , . . . , xt ] − Pr[x
c 1 , . . . , xt ]| ≤ . Pr[xt |x1:t−1 ]
= Ex1:t ln ≤ .
x1 ,...,xt c t |x1:t−1 ]
Pr[x
The main challenge in proving Theorem 6 is understand- To justify our choice of error measure, note that the prob-
ing how the estimation errors accumulate in the algorithm’s lem of bounding the errors of conditional probabilities is
probability calculation. This would have been less problem- complicated by the issue of that, over the long run, we may
atic if we had estimates of the usual HMM parameters T and have to condition on a very low probability event. Thus we
O; the fully observable representation forces us to deal with need to control the relative accuracy of our predictions. This
more cumbersome matrix and vector products. makes the KL-divergence a natural choice for the error mea-
4.2.2 Conditional Probability Accuracy sure. Unfortunately, because our HMM conditions are more
In this section, we analyze the accuracy of our conditional naturally interpreted in terms of spectral and normed quan-
c t |x1 , . . . , xt−1 ]. Intuitively, we might hope tities, we end up switching back and forth between KL and
predictions Pr[x L1 errors via Pinsker-style inequalities (as in [EDKM07]). It
that these predictive distributions do not become arbitrarily is not clear to us if a significantly better guarantee could be
bad over time (as t → ∞). The reason is that while estima- obtained with a pure L1 error analysis (nor is it clear how to
tion errors propagate into long-term probability predictions do such an analysis).
(as evident in Theorem 6), the history of observations con- The analysis in [EDKM07] (which assumed that approx-
stantly provides feedback about the underlying hidden state, imations to T and O were provided) dealt with this problem
and this information is incorporated using Bayes’ rule (im- of dividing by zero (during a Bayes’ rule update) by explic-
plicitly via our internal state updates). itly modifying the approximate model so that it never assigns
This intuition was confirmed by [EDKM07], who showed the probability of any event to be zero (since if this event oc-
that if one has an approximate model of T and O for the curred, then the conditional probability is no longer defined).
HMM, then under certain conditions, the conditional predic- In our setting, Condition 3 ensures that true model never as-
tion does not diverge. This condition is the positivity of the signs the probability of any event to be zero. We can relax
‘value of observation’ γ, defined as this condition somewhat (so that we need not quantify over
γ = inf kO~v k1 . all observations), though we do not discuss this here.
~ v k1 =1
v :k~
√ We should also remark that while our sample complex-
Note that γ ≥ σm (O)/ n, so it is guaranteed to be positive ity bound is significantly larger than in Theorem 6, we are
by Condition 1. However, γ can be much larger than what also bounding the more stringent KL-error measure on con-
this crude lower bound suggests. ditional distributions.
 m, n Number of states and observations The following lemma bounds these errors with high prob-
n0 () Number of significant observations ability as a function of the number of observation samples
O, T , Ax HMM parameters used to form the estimates.
P1 , P2,1 , P3,x,1 Marginal probabilities
Pb1 , Pb2,1 , Pb3,x,1 Empirical marginal probabilities Lemma 8. If the algorithm independently samples N obser-
1 , 2,1 , 3,x,1 Sampling errors [Section 5.1] vation triples from the HMM, then with probability at least
1 − η:
Ub Matrix of m left singular vectors of Pb2,1
eb∞ , B ex , eb1 True observable parameters using U b r r
1 3 1
[Section 5.1] 1 ≤ ln +
bb∞ , B bx , bb1 Estimated observable parameters using U b N η N
r r
δ ∞ , ∆x , δ 1 Parameter errors [Section 5.1] 1 3 1
∆
P 2,1 ≤ ln +
x ∆x [Section 5.1] N η N
σm (M ) m-th largest singular value of matrix M s r !
~bt , bbt True and estimated states [Section 5.3]
X k 3 k
3,x,1 ≤ min ln + + 2(k)
~ht , b ht , gbt b > O)−1~bt , (U
(U ht /(~1>
b > O)−1bbt , b k N η N
m ht ) x
b
[Section 5.3] r r
1 3 1
A
bx b > O)−1 B
(U bx (Ub > O) [Section 5.3] + ln +
N η N
γ, α inf{kOvk1 : kvk1 = 1}, min{[Ax ]i,j }
where (k) is defined in (1).
Table 1: Summary of notation.
The rest of the analysis estimates how the sampling er-
4.2.3 Learning Distributions -close to HMMs rors affect the accuracies of the model parameters (which in
Our L1 error guarantee for predicting joint probabilities still turn affect the prediction quality).
holds if the sample used to estimate Pb1 , Pb2,1 , Pb3,x,1 come Let U ∈ Rn×m be matrix of left singular vectors of P2,1 .
from a probability distribution Pr[·] that is merely close to The first lemma implies that if Pb2,1 is sufficiently close to
an HMM. Specifically, all we need is that there exists some P2,1 , i.e. 2,1 is small enough, then the difference between
tmax ≥ 3 and some m state HMM with distribution PrHMM [·] projecting to range(U b ) and to range(U ) is small. In particu-
such that: >
lar, U O will be invertible and be nearly as well-conditioned
b
1. PrHMM satisfies Condition 1 (HMM Rank Condition), as U > O.
2. ∀t ≤ tmax , x1:t | Pr[x1:t ] − PrHMM [x1:t ]| ≤ HMM (t),
P
Lemma 9. Suppose 2,1 ≤ ε · σm (P2,1 ) for some ε < 1/2.
3. HMM (2)  21 σm (P2,1
HMM
). Let ε0 = 22,1 /((1 − ε)σm (P2,1 ))2 . Then:
The resulting error of our learned model Pr
c is
X 1. ε0 < 1,
| Pr[x1:t ] − Pr[x
c 1:t ]|
x1:t b > Pb2,1 ) ≥ (1 − ε)σm (P2,1 ),
X 2. σm (U
HMM HMM
≤  (t) + |Pr [x1:t ] − Pr[x
c 1:t ]|
x1:t b > P2,1 ) ≥ √1 − ε0 σm (P2,1 ),
3. σm (U
for all t ≤ tmax . The second term is now bounded as in The-
orem 6, with spectral parameters corresponding to PrHMM . b > O) ≥ √1 − ε0 σm (O).
4. σm (U

5 Proof ideas
Now we will argue that the estimated parameters bb∞ , B
bx , bb1
We outline the main ideas for proving Theorems 6 and 7. are close to the following true parameters from the observ-
Full proofs can be found in a technical report available from able representation when Ub is used for U :
arXiv (http://arxiv.org/abs/0811.4413).
Throughout this section, we assume the HMM obeys Con- eb∞ > b + b > O)−>~1m ,
= (P2,1 U ) P1 = (U
dition 1. Table 1 summarizes the notation that will be used
throughout the analysis in this section. B
ex = b > P3,x,1 )(U
(U b > P2,1 )+

5.1 Estimation Errors b > O)Ax (U

= (U b > O)−1 ∀x ∈ [n]
Define the following sampling error quantities: eb1 b > P1 .
= U
1 = kPb1 − P1 k2
By Lemma 3, as long as U b > O is invertible, these parameters
2,1 = kPb2,1 − P2,1 k2 eb∞ , B
ex , eb1 constitute a valid observable representation for
3,x,1 = kPb3,x,1 − P3,x,1 k2 the HMM.
 Define the following errors of the estimated parameters: We introduce the following notation. Let the unnormal-
ized estimated conditional hidden state distributions be
b> > b
δ∞ = (U O) (b∞ − eb∞ )

b > O)−1bbt ,
ht = (U
∞
b

b > >b ~
= (U O) b∞ − 1m ,

∞
and its normalized version,
 
b > −1 b b > O)
∆x = (U O) Bx − B ex (U ht /(~1>
gbt = b m ht ).
b
1

b > −1 b b >
= (U O) Bx (U O) − Ax ,
Also, let
1 A b > O)−1 B
bx = (U b > O).
bx (U
X
∆ = ∆x This notation lets us succinctly compare the updates made
x by our estimated model to the updates of the true model. Our
δ1
b > −1 b
= (U
b > −1b
O) (b1 − eb1 ) = (U O) b1 − ~π .
algorithm never explicitly computes these hidden state distri-
1 1 butions gbt (as it would require knowledge of the unobserved
We can relate these to the sampling errors as follows. O). However, under certain conditions (namely Conditions 1
and 3 and some estimation accuracy requirements), these dis-
Lemma 10. Assume 2,1 ≤ σm (P2,1 )/3. Then: tributions are well-defined and thus we use them for sake of
  analysis.
2,1 1 The following lemma shows that if the estimated param-
δ∞ ≤ 4 · + ,
σm (P2,1 )2 3σm (P2,1 ) eters are accurate, then the state updates behave much like
√ the true hidden state updates.
8 m
∆x ≤ √ · ·
3 σm (O) ~ ∈ Rm and any
Lemma 13. For any probability vector w

2,1 3,x,1
 observation x,
Pr[x2 = x] · + ,
σm (P2,1 )2 3σm (P2,1 )
 
X 
√  P   bb> (U
b > O)A ~ − 1 ≤ δ∞ + δ∞ ∆ + ∆ and
bx w 
8 m 2,1 x 3,x,1
 ∞
∆ ≤ √ · · + , 
x

3 σm (O) σm (P2,1 )2 3σm (P2,1 )
√
2 m [A
bx w]
~i ~ i − ∆x
[Ax w]
δ1 ≤ √ · · 1 . ≥
3 σm (O) bb> (U
b > O)A bx w~ ~
1 >A w
m x ~ + δ ∞ + δ ∞ ∆x + ∆ x
∞

5.2 Proof of Theorem 6 for all i = 1, . . . , m. Moreover, for any non-zero vector
~ ∈ Rm ,
w
We need to quantify how estimation errors propagate in the ~1>
m Ax w
b ~ 1
probability calculation. Because the joint probability of a ≤ .
length t sequence is computed by multiplying together t ma- bb> (U
∞
b > O)Abxw
~ 1 − δ∞
trices, there is a danger of magnifying the estimation errors
exponentially. Fortunately, this is not the case: the following A consequence of Lemma 13 is that if the estimated pa-
lemma (readily proved by induction) shows that these errors rameters are sufficiently accurate, then the state updates never
accumulate roughly additively. allow predictions of very small hidden state probabilities.

b > O is invertible. For any time t: Corollary 14. Assume δ∞ ≤ 1/2, maxx ∆x ≤ α/3, δ1 ≤
Lemma 11. Assume U α/8, and maxx δ∞ + δ∞ ∆x + ∆x ≤ 1/3. Then [b
gt ]i ≥ α/2
X
b > O)−1 B

bx bb1 − B
 for all t and i.
(U t:1
ex eb1
t:1
1
x1:t Lemma 13 and Corollary 14 can now be used to prove the
t t
≤ (1 + ∆) δ1 + (1 + ∆) − 1. contraction property of the KL-divergence between the true
hidden states and the estimated hidden states. The analysis
All that remains is to bound the effect of errors in bb∞ . shares ideas from [EDKM07], though the added difficulty is
Theorem 6 will follow from the following lemma combined due to the fact that the state maintained by our algorithm is
with the sampling error bounds of Lemma 8. not a probability distribution.

Lemma 12. Assume 2,1 ≤ σm (P2,1 )/3. Then for any t, Lemma 15. Let ε0 = maxx 2∆x /α + (δ∞ + δ∞ ∆x +
∆x )/α + 2δ∞ . Assume δ∞ ≤ 1/2, maxx ∆x ≤ α/3, and
maxx δ∞ + δ∞ ∆x + ∆x ≤ 1/3. For all t, if gbt ∈ Rm is a
X  
Pr[x1:t ] − Pr[x
c 1:t ]
x1:t
probability vector, then
δ∞ + (1 + δ∞ ) (1 + ∆)t δ1 + (1 + ∆)t − 1 . γ2


≤
KL(~ht+1 ||b
gt+1 ) ≤ KL(~ht ||b
gt )−
KL(~ht ||b
gt )2 +ε0 .
2 2
5.3 Proof of Theorem 7 2 ln α

In this subsection, we assume the HMM obeys Condition 3 Finally, the recurrence from Lemma 15 easily gives the
(in addition to Condition 1). following lemma.
Lemma 16. Let ε0 = maxx 2∆x√ /α + (δ∞ +√δ∞ ∆x + [CR08] Kamalika Chaudhuri and Satish Rao. Learning
∆x )/α+2δ∞ and ε1 = maxx (δ∞ + mδ∞ ∆x + m∆x )/α. mixtures of product distributions using correla-
Assume δ∞ ≤ 1/2, maxx ∆x ≤ α/3, and maxx δ∞ + tions and independence. In COLT, 2008.
δ∞ ∆x + ∆x ≤ 1/3. Also assume
[Das99] Sanjoy Dasgupta. Learning mixutres of Gaus-
α4 γ 2
r
ε0 α 1 1 sians. In FOCS, 1999.
δ1 ≤ 2
≤ ≤ , ε0 ≤ 2 , and ε1 < .
8γ 8 2 128 ln α2 2


[DLR77] A. P. Dempster, N. M. Laird, and D. B. Rubin.
Then for all t, Maximum likelihood from incomplete data via
s the EM algorithm. Journal of the Royal Statis-

2 tical Society, Series B, 39(1):1–38, 1977.
2 ln α2 ε0
KL(~ht ||b
gt ) ≤ and
γ2 [DS07] Sanjoy Dasgupta and Leonard Schulman. A
probabilistic analysis of EM for mixtures
KL(Pr[xt |x1:t−1 ] || Pr[x
c t |x1:t−1 ])
of separated, spherical Gaussians. JMLR,
s
2
2 ln α2 ε0 8(Feb):203–226, 2007.
≤ + δ∞ + δ∞ ∆ + ∆ + 2ε1 .
γ2 [EDKM05] Eyal Even-Dar, Sham M. Kakade, and Yishay
Mansour. Planning in POMDPs using multi-
Theorem 7 follows by combining the previous lemma plicity automata. In UAI, 2005.
and the sampling error bounds of Lemma 8.
[EDKM07] Eyal Even-Dar, Sham M. Kakade, and Yishay
Acknowledgments Mansour. The value of observation for monitor-
The authors would like to thank John Langford and Rus- ing dynamic systems. In IJCAI, 2007.
lan Salakhutdinov for earlier discussions on using bottleneck
methods to learn nonlinear dynamic systems. The lineariza- [Fli74] M. Fliess. Matrices deHankel. J. Math. Pures
tion of the bottleneck idea was the basis of this paper. We Appl., 53:197–222, 1974.
also thank Yishay Mansour for pointing out hardness results
for learning HMMs. This work was completed while the first [Hot35] H. Hotelling. The most predictable criterion.
author was an intern at TTI-C in 2008. Journal of Educational Psychology, 1935.
[Jae00] Herbert Jaeger. Observable operator models for
References discrete stochastic time series. Neural Comput.,
[ARJ03] S. Andersson, T. Ryden, and R. Johansson. Lin- 12(6), 2000.
ear optimal prediction and innovations repre- [Kat05] T. Katayama. Subspace Methods for System
sentations of hidden markov models. Stochastic Identification. Springer, 2005.
Processes and their Applications, 108:131–149,
2003. [KMR+ 94] M. Kearns, Y. Mansour, D. Ron, R. Rubinfeld,
R. Schapire, and L. Sellie. On the learnability of
[BE67] Leonard E. Baum and J. A. Eagon. An inequal- discrete distributions. In STOC, pages 273–282,
ity with applications to statistical estimation for 1994.
probabilistic functions of Markov processes and
to a model for ecology. Bull. Amer. Math. Soc., [Lju87] L. Ljung. System Identification: Theory for
73(3):360–363, 1967. the User. NJ: Prentice-Hall Englewood Cliffs,
1987.
[BPSW70] Leonard E. Baum, Ted Petrie, George Soules,
and Norman Weiss. A maximization technique [LSS01] Michael Littman, Richard Sutton, and Satin-
occurring in the statistical analysis of proba- der Singh. Predictive representations of state.
bilistic functions of Markov chains. Annals of In Advances in Neural Information Processing
Mathematical Statistics, 41(1):164–171, 1970. Systems 14 (NIPS), pages 1555–1561, 2001.
[BV08] S. Charles Brubaker and Santosh Vempala. [MR06] E. Mossel and S. Roch. Learning nonsingu-
Isotropic PCA and affine-invariant clustering. lar phylogenies and hidden Markov models.
In FOCS, 2008. Annals of Applied Probability, 16(2):583–614,
2006.
[CC08] G. Cybenko and V. Crespi. Learning hid-
den markov models using non-negative ma- [OM96] P. V. Overschee and B. De Moor. Subspace
trix factorization. Technical report, 2008. Identification of Linear Systems. Kluwer Aca-
arXiv:0809.4086. demic Publishers, 1996.
[CP71] J.W Carlyle and A. Paz. Realization by stochas- [Sch61] M.P. Schützenberger. On the definition of a
tic finite automaton. J. Comput. Syst. Sci., 5:26– family of automata. Inf. Control, 4:245–270,
40, 1971. 1961.
 [Ter02] Sebastiaan Terwijn. On the learnability of hid- Now we can form O from the diagonals of Ox . Since O
den Markov models. In International Collo- has full column rank, O+ O = Im , so it is now easy to also
quium on Grammatical Inference, 2002. recover ~π and T from P1 and P2,1 :
[VW02] Santosh Vempala and Grant Wang. A spectral O+ P1 = O+ O~π = ~π
algorithm for learning mixtures of distributions.
In FOCS, 2002. and
O+ P2,1 (O+ )> diag(~π )−1
[VWM07] B. Vanluyten, J. Willems, and B. De Moor. A
new approach for the identification of hidden = O+ (OT diag(~π )O> )(O+ )> diag(~π )−1
markov models. In Conference on Decision and = T.
Control, 2007.
Note that because [MR06] do not allow more observa-
[ZJ07] MingJie Zhao and Herbert Jaeger. The error tions than states, they do not need to work in a lower dimen-
controlling algorithm for learning OOMs. Tech- sional subspace such as range(U ). Thus, they perform an
nical Report 6, International University Bre- eigen-decomposition of the matrix
men, 2007. !
X X
−1
gx P3,x,1 P3,1 = (OT ) gx Ox (OT )−1 ,
x x
A Recovering the Observation and
Transition Matrices and then use the eigenvectors to form the matrix OT . Thus
they rely on the stability of the eigenvectors, which depends
We sketch how to use the technique of [MR06] to recover heavily on the spacing of the eigenvalues.
the observation and transition matrices explicitly. This is an
extra step that can be used in conjunction with our algorithm.
Define the n × n matrix [P3,1 ]i,j = Pr[x3 = i, x1 = j].
Let Ox = diag(Ox,1 , . . . , Ox,m ), so Ax = P T Ox . Since
P3,x,1 = OAx T diag(~π )O> , we have P3,1 = x P3,x,1 =
OT T diag(~π )O> . Therefore
U > P3,x,1 = U > OT Ox T diag(~π )O>
= (U > OT )Ox (U > OT )−1
(U > OT )T diag(~π )O>
= (U > OT )Ox (U > OT )−1 (U > P3,1 ).
The matrix U > P3,1 has full row rank, so it follows that

(U > P3,1 )(U > P3,1 )+ = I,

and thus
(U > P3,x,1 )(U > P3,1 )+ = (U > OT ) Ox (U > OT )−1 .
Since Ox is diagonal, the eigenvalues of (U > P3,x,1 )(U > P3,1 )+
are exactly the observation probabilities Or,1 , . . . , Or,m .
Define i.i.d. random variables gx ∼ N (0, 1) for each x.
It is shown in [MR06] that the eigenvalues of
X
gx (U > P3,x,1 )(U > P3,1 )+
x
!
X
>
= (U OT ) gx Ox (U > OT )−1 .
x

will be separated with high probability (though the separa-

tion is roughly on the same order as the failure probabil-
ity; this is the main source of instability with this method).
Therefore an eigen-decomposition will recover the columns
of (U > OT ) up to a diagonal scaling matrix S, i.e. U > OT S.
Then for each x, we can diagonalize (U > P3,x,1 )(U > P3,1 )+ :

(U > OT S)−1 (U > P3,x,1 )(U > P3,1 )+ (U > OT S) = Ox .