Subspace-Based Signal Analysis Using Singular Value Decomposition

ALLE-JAN VAN DER VEEN, STUDENT MEMBER, IEEE, ED F. DEPRETTERE, SENIOR MEMBER, IEEE, AND A. LEE SWINDLEHURST, MEMBER, IEEE

In this paper, we present a unified approach to the (related) problems of recovering signal parameters from noisy observations and the identification of linear system model parameters fiom observed inputloutput signals, both using singular value decomposition (SVD) techniques. Both known and new SVD-based identification methods are classified in a subspace-orientedscheme. The singular value decomposition of a matrix constructedfr-om the observed signal data provides the key step to a robust discrimination between desired signals and disturbing signals in terms of signal and noise subspaces. The methods that are presented are contrasted by the way in which the subspaces are determined and how the signal or system model parameters are extracted fiom these subspaces. Typical examples such as the direction-of-arrival problem and system identification from inputloutput measurements are elaborated upon, and some extensions to time-varying systems are given.

I. INTRODUCTION The analysis of time series is a fundamental problem in almost all scientific disciplines. In engineering parlance, time series are called signals and their analysis generally serves at least one of two possible purposes. First, the signals themselves are of prime interest and are to be recognized or recovered by the analysis procedure, as for example in communication applications. Secondly, the signals bear information pertinent to the physical dynamical systems that produced them, or to the hypothetical dynamical systems that could have produced them. In the latter case, the analysis of the signal should provide the unknown system parameters. A typical example of the first class of problems is the following. Consider a number of signals si( t ) , modulated by a known carrier frequency, and suppose that only a number of unknown linear combinations ~ k ( t of these )
Manuscript received March 9, 1992; revised March 4,1993. This review was supported in part by the commission of the EC under the ESPRIT BRA program 6632 (NANA2). Dr. Swindlehurst was supported by the National Science Foundation under Grant MIP-9 1 IO 112. A. J. van der Veen and E. F. Deprettere are with the Department of Electrical Engineering, Delft University of Technology, 2628 CD Delft, The Netherlands. A. L. Swindlehurst is with the Department of Electrical and Computer Engineering, Brigham Young University, Provo, UT 84602. IEEE Log Number 921 1776.

signals have been received at sensors located at different points. We assume that each of the coefficients of these linear combinations is a known function of both the (known) sensor positions and some (unknown) parameter 4; of each signal. The objective is to reconstruct the original signals from the received signals, which will be possible if we first determine the actual values of the parameters #i, and subsequently identify the pairs (4i, s ; ( t ) )for each of the signals. We can think of the 4i as being spatial directions from which the signals of interest s ; ( t ) are received. As an example of the second class of problems, suppose we have recorded two signals, u ( t )and y(t), where u(t) is a test signal that is applied at some point in a system, and y( t) is a response signal measured at some other point in the system. If we represent the system mathematically as the mapping u ( t ) + y(t) = T ( u ( t ) )where T satisfies certain , causality and linearity constraints, then the problem may be stated as one of using U ( ; ) and y(t) to either identify the map T, or to find a map T of low complexity that is close, in some sense, to T. It is instructive and useful to notice that the two problems alluded to above are sometimes quite similar. For example, if the mapping T of our second example is a causal, linear and time-invariant operator, then it is in fact a matrix multiplicative operator that is completely determined by the response h ( t ) due to a unit impulse excitation u ( t ) = 6 ( t ) . This impulse response and all of its timeshifted versions constitute the rows of the matrix map. Moreover, if the system is finite, meaning that it can be described by a difference equation of finite order, then this impulse response must be a linear combination of a number of exponentially decaying functions of time, where the exponential factors are the unknown parameters to be determined first. The description of this signal (a weighted sum of elementary signals described by a single parameter) is very similar to the description of each of the received signals in the first example, and the two problems may even become identical in certain specific application scenarios. What we observe here is that the impulse response h(t), much as was the case with the

0018-9219/93$03.00 0 1993 IEEE

PROCEEDINGS OF THE IEEE, VOL. 81, NO 9. SEPTEMBER 1993
1277

Authorized licensed use limited to: IEEE Editors in Chief. Downloaded on August 17, 2009 at 19:20 from IEEE Xplore. Restrictions apply.

recorded signals 2 , t ) , explicitly reveals parameters, in 1( particular the poles of the presumed system model that directly or indirectly define a realization for the model. The determination of the realization parameters of a predefined model is called system identijication. System identification techniques can also be used to determine signal models as well. For example, a signal composed of a sum of damped complex exponentials may be thought of as the output of a certain linear system in response to a known or presumed excitation. Identifying this system will then provide a model for the signal. Whether the objective is to recover a signal, to model a signal, or to identify a linear system, the choice of the structure of the signal (or the model of the system) plays a crucial role. Surely, a priori knowledge of the signal properties must be incorporated into the model, but we must also account for uncertainties in a proper way, that is to say, in such a way that they do not introduce modeling artifacts. But even when these choices have been made successfully, the subsequent signal analysis can be carried out along many different routes, and its success will depend on three important additional choices: 1) the kind of realization that we have in mind, 2) the analysis strategy, and 3) the tightness of the coupling between the analysis procedure and the system realization. What comes into play here are aspects of numerical stability, minimality, and tightness of approximation. Numerical stability guarantees robustness of the analysis procedure, minimality avoids artifacts due to opaque dependencies between excess parameters, and tightness of approximation has to do with convergence of the analysis procedure. The ideal situation occurs when the analysis procedure directly constructs a realization of the model that has been chosen to have a necessary and sufficient number of parameters, and to have low sensitivity with respect to perturbations of its parameters. In all practical applications, the observed signals are corrupted versions of the observations that we would expect under ideal circumstances. The unavoidable contaminations are commonly called noise, and they obstruct the extraction of the true or desired parameters from the analysis of the observed signals. Consequently, the goal of any given identification method is to find the signal model parameters that best match the noise-corrupted observations. Commonly used approaches include maximum likelihood estimation (estimation of the parameters of the model that, in a probabilistic sense, most likely produced the observed signal) and least squares error minimization (yielding the parameters of the model that optimally approximates the observed signal in terms of minimal energy of the difference signal). For an overview of many such identification methods, see [ 11-[3]. In practice, therefore, the choice of the signal or system model has to be complemented by the choice of a noise model and an optimization criterion. For example, in terms of the two classes of applications mentioned above, and with the assumption that the noise is additive, the noise could be due to interfering signals that are received from directions outside the focus area, or it could be due to receiver equipment noise (class 1). On the other hand, it
1278

could be part of the impulse response corresponding to higher order modes that are not of interest (class 2). The selection of the signal or system model, the noise model, and the optimization criterion will in general depend on any a priori available knowledge, desired accuracy, etc., or in short on a number of design variables. Choosing values for these variables may be quite difficult, and an optimal choice may only be possible by trial and error. This makes identification as much an a t as it is a science. r In this paper, we will focus on signals and systems that fit deterministic state-space models. State-space models cover causal and finite systems that may be neither linear nor time-invariant. If they are linear and time-invariant, then they are closely related to constant coefficient difference equations relating input and output signals. In a functiontheoretic framework, these models in tum become rational (expressed by a ratio of two polynomials), and are also called pole-zero models. However, while such models are global input/output characterizations of the system, statespace models also take the intemal system behavior into account by describing the current output as a function of a current intemal state and the current input, and by describing the next state as a function of the current state and the current input. A linear, time-invariant system is simply one for which these functions are themselves linear and time-invariant. The order of the state-space model is the dimension of the state vector, or more precisely, that of the state space, and is a measure of the systems memory capacity. In this paper, we will only be concemed with linear state-space models, and we will require that all signals (input, output, and state signals) belong to certain normed spaces. The analysis of these signals and their models is done through extensive use of linear algebra. Signals are represented as (possibly infinite-length) vectors, and the state-space model is taken to be a matrix map from the input space and state space to the output and state spaces. The observations from which such a map is to be identified do not in general include the (intemal) state signals, so estimation of the model order becomes an essential part of the identification problem. The presence of noise tums this problem into a difficult one, since noise tends to reveal itself as an increased state-space dimension. In order to discriminate against noise, our approach will essentially be the following. We collect the observed signal or signals in a so-called observation matrix, which will often inherit a certain (Hankel) structure from the natural ordering imposed by the state-space model. Decomposing the column (range) space of this matrix into a dominant and a subordinate part reveals which of its subspaces can be attributed to the noise-free signal or signals and which can be attributed to the noise. We will assume that these two subspaces are orthogonal to each other, which implies that in terms of inner products, the noise-free signals and disturbances are independent of one another. The dominant subspace is due to the signals and is referred to as the signal subspace, while the other is referred to as the noise subspace.
PROCEEDINGS OF THE IEEE. VOL. 81, NO. 9, SEFTEMBER 1993

Authorized licensed use limited to: IEEE Editors in Chief. Downloaded on August 17, 2009 at 19:20 from IEEE Xplore. Restrictions apply.

The designated tool used to decompose the range space of the observation matrix into these two complementary subspaces is the singular value decomposition (SVD). The SVD is computationally very robust and allows for highresolution discrimination against noise contamination. Once the signal subspace has been determined, the model parameters are extracted from it. This approach gives rise to a number of subspace-based approaches, and we will be interested in understanding the basic differences between them. Again, these approaches correspond to different model assumptions, specific design parameters, or altemative ways of computing what are essentially the same quantities. Associated with each of these approaches is a certain algorithm: a computational scheme. However, we will focus on the basic principles of subspace modeling-also called low-rank approximation-rather than dwelling on the algorithmic details. We will strive to provide a unified description of low-rank approximation methods, while at the same time pointing out the particularities of each of the approaches with respect to the generic solution. The paper can be divided into two main parts. In the first, the generic problem we are considering is described, and several relevant applications are presented. The second part of the paper is concerned with various classes of algorithms that have been developed over the years for these applications. Linking the two parts of the paper is a discussion of the SVD, which is both a theoretical and computational tool used in the analysis of the data models and the development of appropriate algorithms. In the first part of the paper, Section 11 presents an introduction to linear system realization theory, which can be viewed as identification in the absence of noise. The shift-invariance strucrure present in the data matrices is shown to be a crucial property. Section I11 illustrates the presence of such shift-invariant data structures in four identification scenarios: realization theory for time-varying systems, pole estimation from input-output measurements, direction-of-arrivalestimation in antenna array applications, and harmonic retrieval of sinusoidal signals. Section IV then contains the intermediate discussion of the properties of the SVD that we will use in this paper. The second part of the paper consists of Sections V-IX, and contains details conceming the actual identification algorithms under consideration. An overview of these algorithms is given in Section V, which leads to a classification of the available methods into three classes, which are subsequently treated in Sections VI-VIII. The methods in Section VI (a.0. TAM, ESPRIT) are algebraic and are based on the single-shift structure observed between two submatrices of the data matrix. The methods in Section VI1 (Min-Norm, AAK) are in a sense intermediate; while they can be described using submatrices as in Section VI, they are based on the analytic (i.e., polynomial) properties of one vector selected from the noise subspace orthogonal to the signal subspace. This is elaborated upon in Section VIII, where the analytic properties of the full noise subspace (or equivalently, the full signal subspace) are taken into account (Max Likelihood, MUSIC, Weighted Subspace
VAN DER VEEN cf al.: SUBSPACE-BASED SIGNAL ANALYSIS

Fitting, MODE). The general objective in these approaches is to find a low-rank subspace with a shift structure that has minimal distance to the true signal space, or equivalently, that is as orthogonal to the noise subspace as possible. To conclude the paper, Section IX gives a review of recent work on the statistical accuracy and computational load of the above algorithms. Several parts of the contents of this paper have appeared in separate tutorials and books, in particular the material on the SVD and elementary system theory. In the context of signal processing, introductory texts on SVD and linear prediction methods can be found in [4], 5 ] . During the [ review of this paper, a related tutorial by Rao and Arun on subspace-based model identification was published [6]. Obviously, there is some overlap between their paper and ours. The present paper gives more details conceming the classification of single shift-invariant methods, and also features some maximum-likelihood and Hankel-norm approximation methods. In addition, we consider an application to time-varying systems, and model identification from input/output data.
A. Notation

Throughout this paper, the superscript * denotes complex conjugate transpose and the superscript denotes the ordinary matrix transpose. The superscript A is used either to denote a low-rank approximant of a matrix, or the reduction of a matrix to a smaller size by omitting some rows or columns. The ith column (or sometimes row) of a matrix X is denoted by z . addition, for the polynomial i In we will constructed from a vector U = [ U I uz use the notation U(.) = u*a(z) = U1 U p ..., with a(.) = [I z 2 ...IT , for z E C. For a one-sided infinite matrix (operator) H , we denote byHT the operator H with its top row removed. Likewise, H is the operator H minus its first column. For a finite matrix H of size (L 1) x N, H() is the L x N matrix containing the first L rows of H , and H() is the matrix containing the last L rows of H. The matrix Id is the identity matrix of size d x d. The range of a matrix H of size L x N is the space { H z : z E C N } , which is a subspace in the Euclidean space C L . The kemel of H is the subspace { E C N : Hs = O}. z Projectors onto subspaces are denoted by n. Tr (F)denotes the trace of a matrix F, i.e., the sum of the diagonal entries of F. Eig (F) denotes the diagonal matrix containing the eigenvalues X i of F.

...IT, +

11. INTRODUCTION TO LINEAR SYSTEM REALIZATION THEORY The realization problem for linear systems is already a fairly old subject. A state-space approach to this problem was introduced by Nerode [7], and was subsequently formalized by Ho and Kalman [8]. The realization scheme is based on the analysis of certain subspaces spanned by inputs in the past in combination with outputs in the future. In the mid- 1970s, the SVD was introduced as
1279

Authorized licensed use limited to: IEEE Editors in Chief. Downloaded on August 17, 2009 at 19:20 from IEEE Xplore. Restrictions apply.

a tool to identify these subspaces in a numerically stable way, and for obtaining an approximate realization of lower order than the true system order [9]-[ 1 11. This section will introduce some system theoretic notions with relevance to subspace-based system realization theory. Section I11 will apply this theory to a few standard identification scenarios that will be used throughout this paper. More background material on linear systems theory can be found in the books by Kailath [12] and Rugh [13].

u.,

-5
Y.1
y1

U1

A . System Operator Consider a causal linear time-invariant (LTI) system with system transfer operator T , mapping an input vector (sequence) that represents an input signal
(a)

'k+l

(b)

to a corresponding output sequence

y = [...

51-1

Yo

Y1

-1'

Fig. 1. LTI state-space model. (a) Mapping of an input sequence { U , } to an output sequence { y t } using an intermediate state sequence { x t } . The state dimension is d = 2. Due to causality, the signal flow is from top to bottom. The delay operator z denotes a time shift here. (b) The operation at a particular time instant IC is a linear map from input U k and current state x k to output yk and next state x1;+1.

such that y = Tu. For simplicity of notation, we consider systems with only one input and one output, although the general case follows easily along the same lines. We take the input and output sequences to be of finite energy, IIuIIi = u*u 5 M < 00, so that they are elements of the Hilbert space l z (see, e.g., [14]), and we take T to be a bounded (stable) operator acting from to &. Associated with T is its impulse response

B. State-Space Representation The familiar state-space model used to describe causal LTI systems is
xk+l

= A X k -k

BUk
DUk

yk = C X k

which is the response of the system to a unit impulse applied at time 0. The operator T has a matrix representation such that y = Tu fits the usual rules for matrix-vector multiplications:

in which xk is the state vector (assumed to have d entries), A is a d x d matrix, B and CT are d x 1 vectors, and D is a scalar (see Fig. 1). The integer d is called the state dimension or system order. All finite-dimensional linear systems can be described in this way. The realizarion problem is to find a state-space representation that matches a given system operator T, i.e.

such that the impulse response of the state-space system

h = [... 0 D

C B CAB C A 2 B

...IT

(2)

The ith column contains the impulse response due to an impulse at time i. Note that the above relationship relies on the linearity of the system. The input U can be thought of as consisting of a sum of impulses, one for each time instant i, weighted by u .The output of i the system is then the weighted sum of the responses to these impulses. This description is equivalent to the familiar convolution sum y = h*u, defined by yi = CEOk u i - k . h Because of time invariance, the matrix representation has a Toeplitz structure: it is constant along diagonals. It is lower triangular due to causality.
1280

matches the impulse response of T . In principle, there exist an infinite number of state-space realizations for a given system. For example, the state vector xk might contain some states that are not observed in the output or that are never excited by the input. Hence, we will limit our attention to minimal state-space models, that is, models for which the state dimension d is minimal. It is well known that for minimal systems, in order to have h E &, the eigenvalues of A must be smaller than 1 in absolute value, although eigenvalues on the unit circle are allowed in some applications. Even for minimal systems, the representation ( 1 ) is not at all unique. An equivalent system representation (yielding the same input-output relationship) is obtained by applying a state transformation R (an invertible d x d matrix) to
PROCEEDINGS OF THE IEEE, VOL. 81, NO. 9, SEFTEMBER 1993

Authorized licensed use limited to: IEEE Editors in Chief. Downloaded on August 17, 2009 at 19:20 from IEEE Xplore. Restrictions apply.

define a new state vector x ; = Rxk. The equivalent system is x ; , ~= Ax; BUk

. .
-2

+ Y k = CX; + DUk
[-l

where the new state space quantities are given by

11 The eigenvalues of A remain invariant under this transformation since R - l A R is a similarity transformation [ 151. The eigenvalues of A are directly related to the poles of the system, a fact that is easily verified if these poles are distinct. Under the assumption of distinct poles, another way to describe linear systems is via a partial fraction expansion of the z-transform of the impulse response h

u.1

A B [C D ] =

A B R 1][C D][

YO

Y1

where $ i = 1,. . . , d, are the d poles of the system, and , ; ri, i = 1,. . . , d , their respective residues. A corresponding state-space realization is

Fig. 2. Applying inputs up to t = -1 and recording outputs from t = 0 on yields information about the state at t = 0 . From this, a state space realization can be derived.

Another way to obtain this decomposition is to start from a given realization {A, B , C ,D } and apply an appropriate state similarity transformation that will diagonalize the A matrix: A = R@ R-l. This is an eigenvalue decomposition of A, and the entries of @ are the eigenvalues of A. A sufficient condition for the existence of this decomposition (i.e., an invertible R) is that the poles of the system be distinct [ 121.

H=

-hi h2 h3

h2 h3

h3

...

.*.

C . Hankel Operator We now tum to the realization problem: given a system transfer operator T (or equivalently an impulse response h), how can a state-space model that realizes this transfer operator be determined? The solution to the realization problem in a subspace context calls for the Hankel operator, which we define presently. The idea is to apply inputs only up to time t = - 1 (called the past with respect to the present time instant t = 0) and measure the resulting outputs from t = 0 on (the future; see Fig. 2). Writing y = Tu,we have

1.

(5)

X X

...
=

X
X

0
X

yo
511
Y2

. . h3 h2
h3

hl h2 h3

X X X X X

Clearly, H has a factorization H = OC. C is called the controllability operator and 0 is called the observability operator, and for a minimal realization they have by definition full rank d. Since H is an
1281

VAN DER VEEN et al.: SUBSPACE-BASED SIGNAL ANALYSIS

Authorized licensed use limited to: IEEE Editors in Chief. Downloaded on August 17, 2009 at 19:20 from IEEE Xplore. Restrictions apply.

outer product of rank d matrices, it must be of rank d itself. Even for minimal realizations, there is of course an ambiguity in this factorization. With R an invertible d x d, matrix, we can also factor H as H = O'C' = OR . R-'C, corresponding to a statespace model that has undergone a state transformation by R as described above. Factorizations modulo R lead to equivalent systems. 2 ) H has a shifi-invariance structure. Denote by H the T operator H with its top row deleted. Likewise, denote by H the operator H with its first column deleted. ' Shift-invariance means that the range (column space) of the shifted operator is contained in the range of the original operator. This property can be deduced directly from the Hankel structure in (5)

where the subscript (:, 1) denotes the first column of the associated operator, and (1,:) the first row. Various issues emerge here to make this realization scheme feasible in practice. First, we are only willing to do computations on matrices of finite size. In particular, H should have finite size. This issue can be dealt with relatively easily. Suppose we have available a top-left ( L 1) x N window of the infinitely dimensioned H :

H L + I , N=

I.
:

hi h2

h2 h3

...

~ L + Ih ~ + 2 . ' . ~ N + L

H T H '

= OTC = OC'

0A.C = O.AC.
=

=I

r c i
CA

i I

. [ B A B A 2 B ... A N - l B ]
(7)

Thus it is seen that shifting H upwards or to the left is equivalent to a multiplication by A in the center of the factorization. There is a physical interpretation of this shift invariance. Just as the range of H contains all possible outputs of the system from t = 0 on, due to inputs that last until t = -1, the range of H contains all possible outputs of the system T from t = 1 on, due to inputs that stop at t = -1. Because of the time-invariance of the system, this is the same as stating that HT contains the outputs of the system from t = 0 on, due to all inputs that stop at t = -2. This set of inputs is a subspace in the set of all inputs in the past, and hence the resulting set of future outputs (the range of HT) must be a subspace contained in the original set of future outputs (the range of H).

Define U ( ' ) and 0(2) by

and as before, let d be the rank of H . If L and N are equal is to or larger than d, then the rank of H L + ~ , N also equal to d, and in particular O(') and are of full rank d. The shift-invariance property in this finite-size case is now

O(2) = O ( 1 ) A

A = 0(1)+0(2)

(8)

D . Realization Scheme Using the above two properties of the Hankel operator H-i.e., that it is of finite rank with some minimal factorization H = OC, and that it is shift-invariant-we will show how to obtain a state-space realization as in (1) from a given transfer operator T. 1) Given T, construct the Hankel operator H as in ( 5 ) . Determine the rank d of the operator, and a factorization H = OC, where (3 and C are of full rank d. The SVD is a robust tool for doing this, as will be discussed later. 2) At this point, we know that C and C? have the shiftinvariant structure of (6). Use this property to derive

OA=OT

A=O+OT

where O+ is the pseudo-inverse of O such that O+O = I d . Because 0 is of full row rank d, we have O+ = (O*C?)-'O*. This determines A . The matrices B , C and D follow simply as

and A = 0 ( 1 ) + 0 ( the same matrix as obtained in the is 2 ) infinite case. A second issue is how to handle an inaccurate T. This is more difficult to treat, and in fact is the subject of most of the remaining part of the paper. Suppose that T is corrupted by additive noise, or altematively, that we have measured an impulse response sequence h which contains additive noise. The matrix H will thus be constructed from noisy measurements, and will therefore have full rank in general. H will also have high rank if T represents a system of high order for which a "reduced order" model is desired. In both cases (system identification and model reduction) the objective is to find an approximate system with Hankel matrix H of low rank d that, in some suitable norm, is as close to the original noisy H as possible. In essence, the problem is to determine the optimal (or close-to-optimal) positions of the poles of the approximating system, or in other words to estimate the d x d diagonal matrix = eig ( A ) given a finite extent of the impulse response. At first, we will consider only the shift-invariant structure in the observability matrix 0. The key problem (and also the major distinction between the various algorithms) is how to enforce the shift-invariant structure present in the original or noisy O to be present in the approximation too.

B D
1282

=
= =

C(:,l)

O(1,:) ho

E . Discussion There is a subspace theory underlying the low-rank and shift-invariance properties that we have used implicitly. We
PROCEEDINGS OF THE IEEE. VOL. 81, NO. 9, SEPTEMBER 1993

Authorized licensed use limited to: IEEE Editors in Chief. Downloaded on August 17, 2009 at 19:20 from IEEE Xplore. Restrictions apply.

assumed the existence of a model as in (l), and used the resulting properties to derive the structure of C and 0 as in (6). A proof of the existence of this model starts from some system transfer operator T and its Hankel operator H. We briefly touch upon this subject. Let the minimal system order be d , and let H = OC with 0 and C of full rank d. The output state space N O is the subspace defined by

Flo = {y+

: y+ = H u - ,

all U- E

a,}.

A. Realization of a Time-Varying System The purpose of this section is to give a brief introduction to realization theory for time-varying systems, primarily to demonstrate the generality of the subspace concept. The derivation is very similar to the time-invariant case, and a more detailed discussion along these lines can be found in [18], [19]. Consider again an input sequence U E C2, which is mapped by an operator T to a corresponding output sequence y = Tu,where
U =

Ho is the subspace of all possible outputs in the future

that can be reached by inputs in the past. Mathematically, Ffo is the range (column space) of H , and the d columns of 0 constitute a minimal basis for it. Likewise, define the input null space M and input state space 7-l as

[...

U-1

U0

U1

y=

[...

Y-1

Yo

Y1

...I -1.

T is assumed to be bounded and causal, and hence has a matrix representation

M = {U- : y, = Hu- = O} H=ML.

M is the kernel of H and consists of all inputs in the past

that yield zero output in the future. FI is the orthogonal complement of M and is equal to the column space of H*, or the conjugate transpose of the row range space of H. The d columns of C* constitute a minimal basis for

I...
h;
xk+l

h2l

h22

. .

IFI.
Using the above spaces 3-1 and 80and making use of the assumption that they are of finite dimension d , it is possible to formally derive that there must exist a state-space model in the form of (1). We omit this derivation, but remark that are crucial in the derivation is the fact that H and ,YO shiftinvariant; e.g., the space $,is contained in &. It follows that their bases must also be shift-invariant, and hence that there must be some matrix A to express the shifted basis in terms of the original: 0 T = OA. This gives rise to the now familiar structures of C and 0, and is the content of the abstract realization theory in [16], [17].
111. APPLICATIONSSUBSPACE-BASED OF REALIZATION THEORY In this section, we discuss a number of related identification problems that rely on the same type of low-rank and shift-invariance properties described in the previous section. We first discuss the realization problem for time-varying systems, and show that the resulting time-varying Hankel operator is of low rank and has a shift-invariance property which can be used to determine a time-varying state-space realization. A second application is system identification using input-output data. In this problem, the impulse response is not specified, but instead a measured collection of inputs and their corresponding outputs is given. The third application is the direction-of-arrivalestimation problem, in which one attempts to determine the incidental directions of a number of narrowband plane wave signals impinging on an antenna array. Finally, in the fourth application, we discuss the classical harmonic retrieval problem, where one attempts to determine the frequencies and decay factors of multiple cisoids.
VAN DER VEEN et al.: SUBSPACE-BASED SIGNAL ANALYSIS

As before, the ith column of T is the response of the system to an impulse applied at time t = i, but because the system is time-varying, these impulse responses can change with time. We have thus lost the Toeplitz structure of T. A time-varying state-space realization has the form
= Akxk = Ckxk

Yk

+ Bkuk +DkUk

in which xk is the state vector at time k (taken to have d k entries; the state dimensions need not be constant now), Ak is a dk+l x d k (possibly nonsquare) matrix, Bk is a dk+l x 1 vector, c k is a 1 x d k vector, and Dk is a scalar. Note that, with time-varying state dimensions, the Ak matrices are no longer square matrices, and hence they do not have the eigenvalue decompositions which were used in the time-invariant case to compute the poles of the system. Nonetheless, it is possible to compute time-varying state realizations for a given time-varying system transfer operator T, as the next paragraph will show. Suppose a time-varying system transfer operator T is given, for which we want to determine a time-varying statespace realization. The approach is as in the time-invariant case. Denote a certain time instant as current time, apply all possible inputs in the past with respect to this instant, and measure the corresponding outputs in the future, from the current time instant on (see Fig. 3). As in the timeinvariant case, we select in this way a lower-left submatrix of T. For example, for the current time t = 2,

X X

... ...

x x
h2,--1

Yz
Y3
Y4

x x h2o
h30

0
x
h21
X

U- 1

U0

h31
h4l

x x

x x

U1

..

0 0

Authorized licensed use limited to: IEEE Editors in Chief. Downloaded on August 17, 2009 at 19:20 from IEEE Xplore. Restrictions apply.

where for a minimal system both 0 2 and Cz are of full rank d z . c k and 0 k can be regarded as time-varying controllability and observability matrices. 2 ) H k has shif-invariant properties. For example

Y2

=03c3

Fig. 3. Principle of the identificationof a time-varying state-space model. In this picture, the current time is t = 2, all possible inputs up till time t = 1 (the past) are. applied, and the rm corresponding output sequences are. recorded f o time t = 2 on (the future). This yields H 2 , a Hankel operator at instant t = 2. This should be done in tum for all t .

Denote by submatrix:

H2

the one-sided infinite equivalent of this

In analogy with the time-invariant case, we call H2 the Hankel operator at time 2, although in reality it does not possess an antidiagonal Hankel structure. By doing this in tum for all time, we obtain from T a sequence of Hankel operators Hk, viz.,

T=

:.;.;
0
h43

The shift-invariance pro rty is now reflected by the fact that the range of Hk is contained in the column space of Hk+1. This can also be seen from (9). The physical interpretation is the same as in the timeinvariant case; i.e., the range of H i contains the output sequences from t = 3 on, due to inputs in the past up to t = 1, whereas the range of H3 contains the output sequences from t = 3 on, due to inputs that run up to t = 2. The latter set of inputs properly contains the former, hence the range of H3 contains the range of H l . The above properties form the ingredients for obtaining a realization of a given time-varying transfer operator T: 1) First construct Hankel operators Hk from T . Compute the rank d k of each Hankel operator; this is the system order at time instant k. Compute a decomposition Hk = O k C k into full rank d k factors 0 , and c k . 1 1 The columns of 0 ,form a basis for the output State space at time k, and likewise the columns of C; form a basis for the input state space at time k. 2) Having obtained c k and 0 , for all time instants k, 1 apply the shift-invariance property

T.

-. . (9)

H1

H3

Although we have lost the Hankel structure, we retain the following two important properties: 1) Hk has rank d k equal to the minimal system order at time instant k . While this can be derived formally, it also follows from inspection of Fig. 3. For example, at time 2 we have that the realization satisfies

It is instructive to perform the above recipe on a numerical example. Consider the finite lower triangular matrix

H2=

c 4 A 3 A 2 [B1 AlBO

.I

AlAOB-1

.-I

T=

[y:
1/24

li3 1/12

1 1/4

=02cz
I284

Finite matrices form a special class of time-varying systems. The Hankel matrices with nonvanishing dimensions
PROCEEDINGS OF THE IEEE, VOL. 81, NO. 9, SEPTEMBER 1993

Authorized licensed use limited to: IEEE Editors in Chief. Downloaded on August 17, 2009 at 19:20 from IEEE Xplore. Restrictions apply.

1
A

1
Y1

1
Yz

1
Y3

Y4

an algebraic framework of polynomial rings. However, many results, in particular on controllability, detectability, stabilizability, etc., have been discussed by a number of authors without using these specialized mathematical means (see, e.g., Anderson and Moore [23] and references therein, and Gohberg et al. [24]) by time-indexing the state-space matrices {A, B , C, D} as above.

Fig. 4. Time-varying state realization of a finite matrix.

are

H = 2

[ :f ] [
=

H3 = 1/12 1/24 1 = 1;4][1/3 ID 1/6

H = [1/4 4
1/12 1/24]

1/24 ~-

1;3] 1/2 1/12

1/61

Since raiik(Hk) = 0 for k < 2 and k > 4, no states are needed at these points in time. One state is needed for :1:2 and one for xa, because raiik(H2) = raiik(H4) = 1. Finally, also only one state is needed for ~ 3 because , raiik(H3) = 1. In fact, this is (for this example) the only nontrivial rank condition: if one of the entries in H3 would have been different, then two states would have been needed. The realization algorithm leads to the sequence of realization matrices

[2 E:] [
=

1/3

1/3

where the . indicates entries that actually have dimension 0 because the corresponding states do not exist. The corresponding realization is depicted in Fig. 4, and it is not difficult to see that it indeed computes the matrix-vector The above example of the derivamultiplication y = Tu. tion of a computational network shows how system theory can be used to obtain efficient algorithms for linear algebra problems (in this case matrix-vector multiplications of lower triangular matrices, but also inversion, Cholesky factorization, etc., is possible) [ 191. Although the development of a time-varying state-space theory started in the 1950s (or even earlier), the realization approach presented here is fairly recent, and based on [ 181. Some other important approaches that parallel the given presentation can be found in the monograph by Feintuch and Saeks [20], in which a Hilbert resolution space approach is taken, and in recent work by Kamen et al. [21], [22], where time-varying systems are put into
VAN DER VEEN et al.: SUBSPACE-BASED SIGNAL ANALYSIS

B. Realization from Inputloutput Measurements In Section 11, we assumed that impulse response measurements h, of the system to be identified were somehow available. In many practical situations, however, instead of the impulse response one is given only a segment of the response of the system to some known nonimpulsive input sequence. A deconvolution operation could be used to determine the impulse response, from which the system can subsequently be identified, but this does not yield a very convincing algorithm because the deconvolution operation itself needs some estimate of the system parameters. We would like to use the Hankel approach of the previous section, where we obtained a realization by applying all possible inputs in the past (inputs that are zero from t = 0 on), and determined the range of the corresponding output sequences from t = 0 on. We first look at a slightly different scenario. Suppose we have applied a collection of N independent input sequences {U,}, = 0 , . .. , N - 1, but have measured only a finite 1 segment of the corresponding output sequences 2,. say from time t = 0 to t = L, with d 5 L << N . We denote the known part of each y, by y,, which thus is an (L + 1)-dimensional vector. Likewise, U, is defined to be the segment of U , from time t = 0 to t = L, which will be the only part of each input sequence that will be used in the algorithm. Because the input sequences are not zero from t = 0 on, we cannot apply the Hankel approach directly. However, the system is linear, and hence we can construct new input sequences by taking linear combinations of the given sequences, and compute the corresponding output sequences by applying the same linear combinations to the original output sequences. In particular, if we choose the linear combinations such that all known future segments of the input sequence U, become zero vectors, then we have in fact constructed an input that lives entirely in the past (is zero from t = 0), with corresponding output sequences known only from t = 0 up to t = L. This leads to a transformation

;[

... ... y;

O yL+l

... ...

,O
YR.-1

(10)

in which Q is an N x N matrix representing the appropriate we linear combinations. Note that for independent {U,}, cannot expect to make all U: zero; L 1 independent nonzero U: will remain. From the analysis in Section 11, it is clear that the output vectors y,,, . . . . yi.-l are contained

I285

Authorized licensed use limited to: IEEE Editors in Chief. Downloaded on August 17, 2009 at 19:20 from IEEE Xplore. Restrictions apply.

in the output state space restricted to t E [O, L ] ; i.e.,

r c i

(1 1) where Xo is an unknown d x ( N - L - 1 ) matrix that can be regarded as containing the initial states (at time t = 0) due to the portion of each of the new set of inputs in the (unknown) past. Only if X O is of full rank d will the above decomposition determine O L + ~ to a up state transformation, and in this case we arrive at a model identification problem that is slightly less restricted than that associated with (7), since in (1 1) only O L + ~ a has shift-invariance property. From this shift invariance, we can obtain A and C as before. The determination of B and D is more involved now, and requires a least squares fit of the given input-output relations (we omit the details) [25]. A few remarks are in place. First, the appropriate transformation Q in (10) can be conveniently computed via a QR (or rather LQ) factorization:

A set of N inputs { t i i } that satisfies condition 1) for all possible input state spaces H of a certain rank is called persistently exciting. We will not discuss precise conditions for a set of inputs (or a single input, from which a set of N inputs is constructed by considering shifts as in (13)) to be persistently exciting. In practice, however, if one takes N >> d and ensures that the span of the past inputs has dimension N , one can be almost sure that the rank of X O is equal to d. Typically, this will be the case when a stochastic input (zero mean white noise) is applied to the system. Alternatively, one can construct a deterministic input sequence which also has this property. As a simple example illustrating the above, consider the system described by the first-order difference equation
yk = a k
-

Yk-1

which has a (trivial) state model in which xk = Y k - 1 is the state, and where A = a is the pole to be identified. Suppose that we have applied the input sequence U = [ . . . , 1: 2 , 1 , 1 , ...I, which resulted in the output sequence y = [ . . . , 1 , 2 + a , 1 2n cy2, 1 (Y 2a2 a3, . . . I . With L = 1 and N = 3, the Hankel matrices constructed on the data according to (13) are

+ +

where the matrices R11 and R22 are lower triangular matrices of dimension ( L 1 ) x ( L l ) , and [Ql Qz] are the first 2 ( L 1) columns of the unitary matrix Q having dimension N x N . Consequently

[;I= [;:I

If
r
=

1+2a

2 1 1+2a 1+2a+a2

1 1 1 1 2 0 cy2 1+a+2cy2+cy3

+ +

Taking linear combinations of the columns to zero the third column of U leads to

and it is seen from (1 1) that R22 must have rank d and a range space that spans that of O L + ~ . Hence it is shiftinvariant, and A can be determined from R22 as A = Rig)+ ( 2 ) R22 . Secondly, when only one input-utput sequence is given, of length N L say, then we can use the time invariance of the system to construct a set of N independent input-output sequences of length L , as

I1
r 1

1 +2 a

2 1 1+2cy 1 2cy

+ + cy2

5 2

+ 3a3

so that [ Y : + ~ ]can be written as

(cf. (11)). The above technique thus yields C = 1 and A = cy. The material in this section is primarily based on recent work of Verhaegen [26]-[28], whose subspace model identification scheme was in tum inspired by De Moor et al. [25], [29], and Moonen [30], [31]. It is also possible to derive a combined stochastic/deterministicidentification scheme [32], [33].

Finally, it is essential that XO in ( 1 1) has full rank d. In order to realize this, the set of inputs should be sufficiently rich. More precisely, we must have 1) the part of the inputs for t < 0 should span at least the input state space H (which is unknown); and 2) L 2 d, N 2 L 1 d.

+ +

C . Direction of Arrival Estimation The third application arises in antenna array signal processing, and concems the estimation of the angles of amval of d narrow-band plane waves impinging upon an antenna array. This is the so-called direction-of-arrival (DOA) estimation problem (see Fig. 5). For simplicity, the narrow-band signals S k ( t ) associated with each plane wave are modeled as complex-valued sinusoids s k ( t ) = .iik(t) e x p (j27rf t ) , where j = & , is the amplitude ii t ) k( of the signal (assumed to be slowly time-varying), and f its
PROCEEDINGS OF THE IEEE, VOL. 81, NO. 9, SEFTEMBER 1993

I286

Authorized licensed use limited to: IEEE Editors in Chief. Downloaded on August 17, 2009 at 19:20 from IEEE Xplore. Restrictions apply.

and A ( @ ) by
r l 1

Fig. 5. Direction of arrival estimation. Shown is a uniform linear array consisting of four sensors, and two impinging signals. The angle-of-arrival 0, of signal s, is computed from an estimate of the phase shift corresponding to the distance A sin ( 0 , ) .

A has a structure that is known as Vandermonde structure, and its column space is clearly shift-invariant. Letting A(1) represent the first L rows of A, and A(2) last L rows the (and likewise for X ( l ) and X ( ' ) ) , we have
A(2)= d(1)@
and
X(1) X(2)

center frequency. The assumption of complex (or analytic) signals is supported by the fact that most antenna receivers decompose the received signals into both in-phase and quadrature components. An analytic signal model is convenient here since, for narrow-band signals, it allows a time delay to be represented as multiplication by a complex exponential. Consequently, corresponding to each angle of incidence 6 k is a complex constant $ k of unit modulus that represents the phase shift due to the propagation delay Tk of a plane wave signal between two neighboring sensors of the array separated by a distance A. ThUS S k ( t - ?-k) = S k ( t ) $ ? ) k , with $ k = exp(j2rfA sin(6k)). We will parameterize the DOA problem in $k rather than 6k. Assuming that the sensors and associated receiver hardware are approximately linear, the array output signal at the ith sensor, xi(t), is given as a weighted sum of the d input signals:
d

= =

A(1)S A(2)S =

A(l)@S.

(17)

As before, the equation = A(')@ illustrates the shiftinvariant structure present in the array due to the uniform distribution of its (identical) sensors. If no two of the d signals s k @ ) are fully correlated, then S is of full rank d.' As before, a decomposition of X into minimal rank-d factors is not unique, and will not reveal the Vandermonde structure. We may obtain the decomposition as
X U )

X(2)

@1)c o(2)C =

O(1)AC

A(I)R-I.RS

= A(l)[email protected]

z ; ( t )= C a i ( $ k ) s k ( t ) ,
k=l

i = I,..., L + 1

(14)

where R is some unknown invertible d x d matrix, playing the role of a similarity transformation. However, 0 is shift-invariant, and A can be determined as in (8): A = d1)+d2) = R@R-l, so that the eigenvalues of A are equal to the 4 k . A related shift structure arises if, instead of a ULA, the array is known to be composed of two identical but otherwise arbitrary subarrays. In this case, A(@) satisfy will

A(@) =

where U ; ( & ) represents the response of the ith sensor to a signal arriving from the direction associated with &, and we have assumed that there are a total of L 1 sensors. Suppose that N samples are taken at time instants t l , . . . ,t ~and collect the data z ; ( t j )into a ( L 1) x N , matrix X with entries Xi,j = x;(tj). Because of (14), X may be decomposed into the product of a ( L 1)x d matrix A ( @ ) a d x N matrix S and

[2@]

X = A(@ ) S

(15)

where the lcth row of S contains the samples S k ( t j ) , @ = diag($q,...,$d) is a diagonal matrix containing the parameters $k that are to be identified, A ( @ ) = [a(41)... a(&)] is a matrix with columns of the form a(&) = [ul($k) ... U L + 1 ( & ) I T , which is the array response vector due to a signal impinging from direction $k. This vector depends only on the geometrical construction of the array and the directional response of the sensors. For a uniform linear array (ULA) of identical equispaced sensors, a($)is given by a(4)= [ 1 $ $2 ... dLIT,
VAN DER VEEN
(If

for some full rank do. This kind of block-shift structure is the parameterization assumed by the so-called ESPRIT algorithm [34]-[36]. Techniques for exploiting this structure are described in Section VI. The matrix X above will drop rank if either the array response matrix A(@)or the signal matrix S has rank less than d. When A ( @ ) rank less than d, the array is has referred to as being ambiguous, and the signal parameters $k are not identifiable. This corresponds in some sense to an unobservable linear system. This type of rank deficiency can be avoided by proper array design, or in cases where the signal location parameters are restricted to some subset of possible phase delays. For example, the ULA described above is guaranteed to be unambiguous if and only if A < X/2, where X is the wavelength of the narrow-band signals. When S is rank-deficient, it usually indicates that
I Note that perfect sinusoidal signals of the same frequency are the same, up to a difference in phase and amplitude, and consequently S will have rank 1. The rank condition is satisfied if ( t )is not constant but slowly time-varying, and the sampling time is long enough.

U/,:

SUBSPACE-BASED SIGNAL ANALYSIS

1287

Authorized licensed use limited to: IEEE Editors in Chief. Downloaded on August 17, 2009 at 19:20 from IEEE Xplore. Restrictions apply.

some subset of the signals are perfectly coherent; that is, (at least) one of the signals is just a scaled and delayed version of another signal. This type of situation arises when the multipath phenomenon is present, such as occurs when both a direct-path signal and one (or more) reflections are received by the array. Unlike the case of an ambiguous array, the location parameters 4 k are often still identifiable when S is rank deficient [37], [38].

where we allow the 4 k and r k to possibly be complex. If we let 4 k = e a k + j w k , the time-domain version of the impulse response of (19) can be written as
d

hn k=l

rke(akfjuk)n

(21)

D.Harmonic Retrieval
The relationship between the Hankel decomposition H = OC in (6) and the decomposition X = A(@)Sin (15) is not coincidental. The Hankel matrix decomposition can also be written in terms of Vandermonde matrices if the poles of the system are distinct. Under this condition, recall the partial fraction expansion of the z-transform of the impulse response in (3)
03

h ( z ) = C h n z n = ro
0

+ +

rkz
~

k=l
d

-4kz
rkz(1

= TO

4kz

+ 48z2 +

which is just a sum of d damped exponential signals. Thus the problem of determining the poles of a linear system from observations of its impulse response can be recast as one of estimating the frequencies and decay factors of multiple exponential signals. This latter problem is referred to as harmonic retrieval, and has been studied by researchers for many years in fields as diverse as economics, zoology, and physics, not to mention engineering. One of the earliest written accounts of such work was given by the Baron de Prony in the late eighteenth century [39]. Comparing (20) with (15), we see that the matrix A ( @ ) defined here is analogous to the array manifold in the DOA estimation problem, and will be unambiguous, (i.e., full rank d) if L 1 > d and W k < T . When N - 1 > d, the Nyquist assumption W k < T also can be shown to guarantee that S is full rank d.

k=l

(19)

- 1

...

1-

41
=

4 :

r1

..*
rd

IV. SINGULAR VALUE DECOMPOSITION In the previous section, the notions of subspace, column space, rank, and factorization of matrices have been introduced conceptually, and it was noted that the singular value decomposition (SVD) of matrices is a robust tool for computing them. In sections to follow we will make extensive use of this tool, and therefore we shall take a closer look at it in this section. For a more detailed account (and an overview of algorithms for its computation) we refer to [15]. Tutorial information as well as related technical papers on the subject of SVD and signal processing are provided by [4] and the series [401, [41].
A . Subspaces Starting with a given matrix X of size L x N and with entries in 43, one may want to know how many columns (rows) of this matrix are nonparallel or independent of each other. We will assume throughout that the dimensions L and N are finite (however, most of the results will still hold when the dimensions are not finite, provided X is a so-called compact operator, i.e., when the sum of its squared entries is bounded). If there are d 5 L 5 N independent columns in X, then this matrix is said to have a d-dimensional range or column space, which is a subspace of the L-dimensional Euclidian space C L . The rank of the matrix is the dimension of this subspace. If d = L, then the matrix is of full rank, and for d < L it is rank-deficient. Now C L is spanned by the columns of any unitary matrix in C L L, the Euclidean space of square, complex-valued Ldimensional matrices. The same holds for C N of which the row space of X is a d-dimensional subspace: the columns of any N x N unitary matrix in C N x Nspan the vector space C N . Assuming d 5 L 5 N , we can choose a unitary U such that the d-dimensional column space of X is spanned
PROCEEDINGS OF THE IEEE, VOL. E l , NO. 9. SEPTEMBER 1993

- : . . 424

1288

Authorized licensed use limited to: IEEE Editors in Chief. Downloaded on August 17, 2009 at 19:20 from IEEE Xplore. Restrictions apply.

by a subset of d columns of say the first d columns, which together form a matrix U:
d L-d
t ) c )

q,

U = L l (U

U').

the column space of X and are called the left singular vectors. Similarly, the d columns of V are called the right singular vectors and span the row space of X (or the column space of X*) . In terms of these (sometimes much) smaller matrices, the SVD of X can also be written in "economy" size

Since U is a unitary matrix, we shall have 1) From U*U = I L :

x = &$*

(22)

( a ) U*U = I d ( b ) U*UL = 0
(c)

(U')*ii'

=1L-d.

2) From UU* = I L :
( d ) UU*

+ U'(OL)*

= IL

where I d is the identity matrix of order d, and similarly for IL and 1 L - d . Relations (a)-(d) tell us that any vector z E C L can be decomposed into two mutually orthogonz$ vecto!s 3 and ZLin the spaces spanned by the columns of U and UL, respectively. These two spaces are d-dimensional and ( L - d)-dimensional orthogonal subspaces in C L , and their direct sum is equal to C L . Therefore, the orthogonal complement in C L of the column space of X ',s spanned by the columns of the matrix UL. The matrices UU* = IIc and U*(UL)* = I in the above relation ( d ) are the : orthogonal projectors onto the column space of X and its orthogonal complement in C L , respectively. That is; 3 = II,z and kL = IItz. The unitary matrix V can be similarly decomposed:
d N-d
t ) t )

where 9 is a d x d diagonal matrix containing 01, . . . (Td. This form of the SVD better reveals that X is actually of rank d: it is constructed from a product of rank-d matrices. The SVD of X makes the various spaces (range and kernel) associated with X explicit. So doeSAany decomposition of X as X = UE,V*, where U and V are any matrices whose columns span the column and row spaces of X, respectively, and where E, is an invertible d x d matrix. The property that makes the SVD special is the fact that E, is a diagonal matrix, so that a decoupling is obtained: with U; the ith column of U, and vi likewise for V , X can be written as a sum of rank-1 isometric matrices uiv:, scaled by (T;:
d

x=
i=l

.(;; ;U.)

and we also have oiu;= xva ,

0;v; = x*u;.

V=,T(V

VL).

Here, the matrices V V * = II, and V * ( V L ) * = I are : orthogonal projectors onto the original subspaces in C N spanned by the columns of V and V I , respectively. The columns of V* span the kernel of X ,i.e., the space of input vectors z for which Xz = 0.

B . SVD In terms of the above discussion of subspaces, the singular value decomposition of the L x N matrix X, which we assume to have rank d, is obtained by making a certain well-defined choice for U and V, which then gives rise to the following decomposition [ 151:

x = [U 09c

[(&*I

This makes it possible to rank the vectors in the column space and row space of X: the most important direction in the column space is u1,with scale 01, and is reached by applying X to the vector V I . The second most important direction is u2,etc. This ranking will in turn lead to optimal low-rank approximants of X (see below). In the mapping a E C N + b E C L : b = Xu,b will automatically be a vector in the column range of X,and will be nonzero if and only if a has a component in the row space of X; i.e., if and only if II,a is nonzero. On the other hand, b will be identically zero if and only if a is orthogonal to the row space of X. Therefore, the space spanned by the vectors vd+l,"' 1~~ in v* is called the null space (or kernel) of X. Vectors a in this space are mapped to zero by one of the zero singular values of X. The SVD of X reveals the behavior of the map b = Xu:a is rotated in N-space (by V * ) ,then scaled (by the entries of C: L - d components are projected to zero), and finally rotated in L-space (by U) to give b.

where C is an L x N diagonal matrix containing the singular i values ( ~ of X.These are positive numbers ordered such that
(TI

2 (T2 2 ' ' . 2 (Td > (Td+l

= ' . . = (Tt = 0 .

Note that only d singular values are nonzero. The d columns of U corresponding to these nonzero singular values span
VAN DER VEEN et al.: SUBSPACE-BASED SIGNAL ANALYSIS

C . The Effect of Noise Suppose that X is an L x N matrix with rank d 5 L. As before, denote the SVD of X as X = UCV*= UCV*. In this subsection, we will briefly study the effect of adding noise to X on its SVD. The perturbation theory of the SVD is partially based on the link of the SVD with eigenvalue decompositions:

x = u c v * * xx* = U C 2 U *
1289

Authorized licensed use limited to: IEEE Editors in Chief. Downloaded on August 17, 2009 at 19:20 from IEEE Xplore. Restrictions apply.

so that the singular values of X are the positive square roots of the eigenvalues of X X * , while its left singular values are the eigenvectors of X X * . Suppose that X is perturbed by some noise matrix V: X = X V . We first consider the case where the entries of V are generated by uncorrelated, zero mean, white-noise processes with variance a, so that the variance E(VV*)is asymptotically (for N -+ 00) given by E ( V V * / N )= a 2 1 ~ . Under the same conditions

Fig. 6. Construction of the left singular vectors and values of the z matrix S = [ l 221, where 11 and 2 2 have equal lengths.

E(XX*/N) = E ( X X * / N )+ 2 1 L
so that, for large N , the SVD of X is given by

Xx U(C2 + Na21)12V*
for some unitary matrix V. This expression shows that, for large N and small a,the singular values of X increase by an amount approximately equal to a n , while the left singular vectors of X remain the same. X is now of full rank, and its L - d smallest singular values are no longer zero, but equal to a n . In theory, we can recover X X * by subtracting N a 2 1 from (E). This should set the (L - d ) smallest singular values back to zero. F e range space of X , as estimated from X, is spanned by U, the left singular vectors corresponding to the dlargest singular values of X. It is not possible to recover V (or X ) , because the length of the columns of V is equal to N , and hence these vectors do not participate in the averaging effect of increasing N . For more general V, and in case N is not extremely large, one can show that the singular values of X are raised by an amount on the order of 11 V 11, the largest singular value of V . However, in this case the singular vectors are also perturbed. The amount of the perturbation in the subspace which they span can be estimated (see, e.g., [42], [43]), and is again in the order of 11 V 11. The effect on the singular vectors themselves can be much larger if the corresponding singular values are close [42]. Summarizing, the singular values and the subspace spanned by the left singular vectors are (for reasonably large N) relatively insensitive to added perturbations on the entries of the matrix, and hence the SVD is numerically reliable and robust. The SVD thus provides a good estimate of the numerical rank of a matrix, and is useful for quantifying how close a matrix is to being low-rank. The noise threshold depends on the smallest singular value of the original matrix. This singular value is related to the smallest vector that can be constructed with linear combinations of the columns of the matrix, or the smallest distance of one column of the matrix to the column range of the remaining columns. Obviously, it will be small when the columns are more or less aligned, as displayed in Fig. 6. This figure shows the construction of the left singular vectors of a matrix X = [q 221, whose columns 2 1 and 2 2 are of equal length. The largest singular vector u1 is in the direction of the sum of 2 1 and 2 2 , i.e., the common direction of the two vectors, and the corresponding singular value 01 is equal to 01 = 11 21 2 2 II/&. On the other hand, the smallest singular vector uz is dependent on

the difference 2 2 - 2 1 , as is its corresponding singular value: 02 = 1122 - 2 1 I//&. If 2 1 and 2 2 become more aligned, then a 2 will be smaller and X will be closer to a 2 singular matrix. Clearly, u is the most sensitive direction for perturbations on 2 1 and 2 2 . The relevance of this observation is that the resolution of subspace-based parameter estimation algorithms depends on the smallest singular value of the matrix of observations, in relation to the noise level. For example, in the previous section, the observation matrix consisted of linear combinations of vectors of the form U(+) = [l I$ I$2 ... 4L-1]T. If two directions, or two poles, are close together, then $1 x 4 2 and ~ ( 4 1 points in about the same direction ) as u ( $ 2 ) , which will be the direction of u1. The smallest singular value, 0 2 , is dependent on the difference of the directions of ~ ( 4 1and ~ ( $ 2 ) . ) With a noise matrix V added, detecting the presence of two signals will in general become difficult if 02 is approximately the same or smaller than 11 V 11, the noise level. This is because the structure of V determines how much a2 is increased: a; 5 ( a i ) 2 5 0; IIV112, and because the second direction is only visible if a > i 1) V 1). Note that in the commonly assumed case where V is generated by independent identically distributed noise processes such that E ( V V * / N ) = a21, then, for large enough N, all of the singular values squared increase by the same amount IIV1I2. In such cases, 0; > IIVII automatically, and detection of the second signal is always possible. It is also important to note that the smallest singular value is strongly dependent on the value of L, the length of the observation vectors. If L is increased, and a(42) becomes then the difference between ~ ( 4 1 ) more pronounced, so that 0 becomes larger and the 2 resolution increases. This effect is stronger than the effect of increasing N , the number of observation vectors. In the latter case, the purpose is to average out the noise. For illustration, consider the following small numerical experiment. Let 41= 1, $2 = exp ( . O . l ) , and construct j . matrices X L , N from unitary linear combinations of the columns of [u($l) a(+a)].For L , N 2 2, these matrices have rank 2. The two nonzero singular values of X L , N for some values of L, N are given in Table 1. It is seen that doubling L almost triples the smallest singular value, whereas doubling N only increases the singular values by which is because the matrices have larger size. a factor In the latter case, the ratio between 02 and the noise level is not increased because the perturbation matrix would also have twice its original size, which leads to an increase in the noise level of the same factor

a,

a.

1290

PROCEEDINGS OF THE IEEE, VOL. 81, NO. 9. SEPTEMBER 1993

Authorized licensed use limited to: IEEE Editors in Chief. Downloaded on August 17, 2009 at 19:20 from IEEE Xplore. Restrictions apply.

L=3 N = 3 L=6 N = 3

U1

= 3.44 = 0.44 = 4.73 = 1.29

L =3

TI

= 4.86 = 0.63

up
U1

N =6

~2

~2

D.Pseudo-Inverse
Consider a rank-d L x N matrix X . In general, since X may be rank-deficient or nonsquare, the inverse of X does not exist; i.e., for a given vector b, we cannot always find a vector a such that b = X a . A related notion is the (Moore-Penrose) pseudo-inverse of X , denoted by X + , which can be defined in terms of the economy size SVD of X (see (22)) as

x+ = pg-10
This pseudo-inverse satisfies the properties
1. x x + x = x 2. X + X X + =

x+

3. 4.

x x + = n, x+x = nr

which constitute the Moore-Penrose inverse in the traditional way. These equations show that, in order to make the problem b = X a solvable, a solution can be forced to an approximate problem by projecting b onto the column space of X : b = II&, after which b = X u has solution a = X+b. The projection can also be done implicitly by just taking a = X+b: from properties 1 and 3 of the list above, we have that a = X+b = X + X X + b = X+b. It can be shown that this solution a is the solution of the (least squares) minimization problem

it can be shown that X is the rank-d approximation of X that minimizes (1 X - X the Frobenius norm of the difference E = X - X. The Frobenius norm of a matrix is the sum of the squares of its entries, and can be shown to be equal to the sum of the squares of its singular values (because this norm is rotationally invariant). X is called the rank-d Least Squares (LS) approximation to X : it retains the d most important singular values and vectors of X , and sets the remaining L - d singular values to zero. Hence 11 E 1(; = oi+l ... U:. A typical LS application is the following. Suppose that a vector b is given, and we want to find a vector a such that b = X a . We saw above that a (least squares minnorm) solution is obtained by setting a = X+b. However, since X + = VC+U*, small singular values of X play an important role in X + : the pseudo-inverse of the full-rank matrix can lead to numerical instabilities. A more reliable solution is obtained by setting the small singular values o,f X equal to zero, thus obtaining an LS approximation X of X . The vector a is then obtained by computing a as the least squares min-norm solution of b = X a (that is, a = X+b). Now, suppose that instead of a single vector b we are given an(L x N)-dimensional matrix Y, the columns of which are not all in the column space of the matrix X . We want to force solution: to X A = Y. Clearly, we can use a LS approximation Y = IIgY to force the columns of Y to be in the d-dimensional column space of X . This is reminiscent to the, LS application above, but just one way to arrive at X and Y having a common column space. There is an other way, called Total Least Squares (TLS) which is effectively described as projecting both X and Y onto some d-dimensional subspace that lies between them, and that is closest to the column spaces of the two matrices. To implement this method, we compute the SVD

(IF,

+ +

where a is chosen to have minimal norm if there is more than one solution (the latter requirement translates to a =

[X Y ] = [U

&a).
E. L and TLS Approximations S Suppose that X has full rank L. In this case, od+l, . . . ,UL are nonzero, and the SVD of X can be written as

oqc (v*)* [ v* = fi2 [q V;] + U 9 1 ( V 1 ) *

F d take the TGS, !column :pace) approximati9ns to be X = n , X = UCV; and Y = II,Y = UCV,*, where VI and fi are the-partitions of V corresponding to X and Y, respectively. X and Y are in fact solutions to

x = ucv*= U5i.* + U l g l p l
where U contains the first d left singular vectors of X, corresponding to, the d largest singular values which are collected in E. CL contains the LA- d remaining singular values, which are now nonzero. U contains the d most important vectors (directions) in the column space of X . Hence, a rank-d approximation X of X is obtained by putting and A satisfying X A = Y is obtained as A = X + Y . This A is the TLS solution of X A x Y.

x
A ^

ucv* = nu- x nv
^ A n

(23)

where I, = UU* and I = V V * are the projectors I I onto the approximated column space and row space of X , respectively. If X is any rank-d L x N matrix, then
VAN DER VEEN er al.: SUBSPACE-BASED SIGNAL ANALYSIS

F. The Matrix Pencil Problem To close this section we consider the following eigenvalue problem. Let X and Y be two (full-rank) matrices of dimension L x N (L 5 N), and let X be a complex scalar. The matrix pencil problem is to determine values of X for which the rank of the matrix Y - AX is L - 1 instead of L. Y - A is called a matrix pencil, and those X special values of X are called the rank-reducing numbers of
1291

Authorized licensed use limited to: IEEE Editors in Chief. Downloaded on August 17, 2009 at 19:20 from IEEE Xplore. Restrictions apply.

the pencil. When X and Y are not of full rank, the matrix pencil problem is to find the values of X for which the rank of the pencil drops one in comparison to the usual rank of the pencil. In cases where X and Y are full-rank square L x L matrices, the matrix pencil problem reduces to an ordinary eigenvalue problem. There are L rank-reducing numbers XI,. . . ,XL, and they are known as the generalized eigenvalues (GE's) of the matrix pair (Y,X ) . The GE's of (Y,X ) are those values of X for which there exists a nontrivial vector z such that Y z = X X z . Since, under the present assumptions, X is invertible, these are just the solutions to the ordinary eigenvalue problem ( X - ' Y ) z = Xz. We now turn to the more general problem that will be encountered in the next section, where X and Y are rank-d L x N matrices. For convenience, we require X and Y to have the same column space and row space. These amount to conditions for the existence of nontrivial A. If X and Y were to have disjoint column spaces, then the rank of Y - AX can only decrease if the rank of AX decreases, i.e., if X = 0. A similar result holds for the row spaces. We will show that the solution of the matrix pencil problem can be given in terms of the pseudo-inverse! of X and Y as introduced before. Call UxCxV: and UyCyV,* the "economy-size'' SVD's of X and Y, respectively. Since by assumption X and Y span the same column y d row spaces, we ,can express Y-in terms of U and V,: say , Y = U,E,V;, with Ey = U,*YV, a d x d matrix. Hence

is 1) to find the pole locations (or equivalently a), and 2) to determine a matching state-space model (i.e., to find the corresponding zeros of the system). In the input-output identification application, H is not a Hankel matrix but its column space is still shift-invariant. In the DOA application, the second description given above is more natural since A corresponds to the array response matrix and S to the incoming signals. S has a shift-invariant structure only if the sampling period is constant. The purpose in DOA estimation is 1) to find the directions of arrival (or equivalently a), and 2) to reconstruct the signal matrix S (signal copy). For the sake of discussing these applications within a unified framework, and to present algorithms that are valid for both system identification and DOA estimation, we will use the description [H = A(@)S] in most of the remainder of the paper, and focus on finding a. Once a, and hence A(@), known, it is a straightforward is matter to determine a corresponding S from H, e.g., by setting S = A+H. The algorithms in Section I1 were based on noise-free conditions. In general, however, H is corrupted by noise, which is assumed to be additive:

H = A(@)S V .
The noise incorporates all undesired components of the data. Depending on the problem at hand and on the chosen solution strategy, the noise is assumed to be either stationary zero-mean white (as in the system identification and DOA problem), or to encompass unwanted higher order components of an actual system response (modes to be neglected in model reduction problems). The problem we will consider in the remaining part of this paper thus reads as follows: Given a matrix H which contains noise-corrupted observations of a system, determine a d x d diagonal matrix using the model

- AX = ~ , ( E Y Xk,)V, -

and thus the problem is reduced to the square pencil problem: the rank-reducing numbefls of the pencil X - A Y are given by the d eigenvalues of C;'E,. It can be shown that these solutions are precisely the nonzero entries in eig ( X + Y ) or eig ( Y X + ) . Indeed

X'Y

= Vxgglc:. UxEyV:
= V, . C i ' E ,
, . A

H=A(@)S +V,

. V, .

A ( @ = [a(4i) . . . 4 4 4 ) a($) = [ I 4 d2 ...

I
$"I'
(24)

From the property that the nonzero eigenvalues of the product ( A B ) of two matrices A and B are equal to the nonzero eigenvalues of ( B A ) ,the result follows.

v.

OVERVIEW OF IDENTIFICATION SCHEMES

A. The Model

In the realization theory of Sections I1 and 111, we have seen that there are two Hankel matrix decompositions that are in fact equivalent if the system poles are distinct:

( l ) H = oc,
( 2 ) H = A(@)S
in which A has a Vandermonde structure parametrized by the diagonal matrix @ with entries di, and in which 0 ,C, A, S are shift-invariant. In fact, the second description is a special case of the first. The purpose of identification
I292

inwhichHisofsize(L+l)xNwithN> L>d,Aand S are of full rank d, and the matrix V represents additive noise. In this problem statement, H need not be Hankel, and hence no shift-invariant structure in S is presumed. The column space of AS is referred to as the signal subspace (which is the output state space in system theory), and its orthogonal complement is referred to as the noise subspace. The presence of the noise term means that H will actually be of full rank. An important issue that we have not dealt with thus far is that of model order determination. With white noise present and N approaching infinity, the extra singular values due to the noise are all the same and presumably small, and d can be found by simply counting the multiplicity of the smallest singular value of H and subtracting from L 1. However, with probability one, for finite N none of the singular values of H will be repeated, and hence some other method is required to estimate d. Put simply, the

PROCEEDINGS OF THE IEEE, VOL. 81, NO. 9, SEPTEMBER 1993

Authorized licensed use limited to: IEEE Editors in Chief. Downloaded on August 17, 2009 at 19:20 from IEEE Xplore. Restrictions apply.

strategy is to look for a break in the pattern of singular values of H , attributing the larger ones to the signal and the small ones to the noise. The detection of such a break has been well studied and a number of techniques have been developed, most of them being based on the asymptotic distribution of the covariance matrix related to H under the assumption of white Gaussian noise. These include the classical sequential hypothesis test [U]-[46], Akaikes Information Criterion (AIC) [47], Rissanens Minimum Description Length (MDL) principle [48], [49], and the refinements of Zhao et al. [50]. Specific applications to DOA estimation have been studied in [51]-[55]. It is beyond the scope of this paper to study the model order determination problem in any detail, so we will just assume in what follows that d has been correctly determined by some method.

B. Solution Outline A number of strategies for solving the identification problem in (24) have been proposed. They differ primarily in the degree of structure that is imposed on the solution. 1) Subspace Fitting: These methods seek to match the data with the true model; i.e., they minimize 11 H A(@)S11; in the Frobenius norm over all possible S and Vandermonde matrices d ( @of rank d. ) Equivalently, they may be thought of as finding a model (with shift-invariance properties) whose column vectors are most orthogonal to the estimated noise subspace of the data. Weighted versions of this minimization have recently been proposed (Weighted Subspace Fitting, MODE) which provide minimum variance parameter estimates. 2) Single Shift-Invariant Methods: In contrast to the Subspace Fitting techniques, these methods impose only a single shift-invariance property on the data, in the sense that the observation matrix H is partitioned into two matrices X = H ( l ) and Y = H ( 2 ) ,with X containing the first L rows of H, and Y the last L rows. The problem is then recast as one of finding @ from

X=dLSN

+ Vi Y =dL@SN + V2.

(25)

thought of as intermediary between them. These methods are also based on the shift-invariant structure of (25), but they can equivalently be described as methods that find vectors orthogonal to a particular vector selected from the noise subspace (see the discussion below). Subspace fitting techniques are described insection VIII. In these techniques, the problem is to determine a ddimensional column space (range) of d that has the required Vandermonde-like structure and is closest to the column space of H. By ignoring any (shift-invariant) structure that S might possess, the minimization is linear in the parameters of S . Consequently, the problem can be made more compact by deriving from H a rankd signal subspace, and then finding a rank-d matrix A with Vandermonde structure whose column space is as close as possible to the signal subspace (or equivalently, which is as orthogonal as possible to the noise subspace). Though Subspace Fitting techniques can provide estimates of minimum variance, such techniques are more difficult to implement since in general they require a multidimensional (gradient) search over the parameter space. This drawback is mitigated by the fact that the computationally efficient Single Shift-Invariant methods can be used to obtain an accurate starting point for the search. Approaches to the Single Shift-Invariance problem (25) are motivated by the exact relationships present in the noisefree case. They give rise to the matrix pencil techniques that we already have encountered in Section IV, in which the pencil Y - AX is studied for varying values of A. Without noise, it readily follows from the structure of (25) that the diagonal entries of @ are the rank-reducing numbers of the pencil Y - AX, i.e., those values of A for which the pencil drops rank. This is because Y - AX = d ( @ AI)S. A slightly more general way to describe these methods is by defining an L x L matrix F that satisfies F X = Y. Since in the noise-free case X and Y are of rank d, F is not unique; it can have rank anywhere from d to L. For any of the possible choices of F , it can be shown that d of the eigenvalues of F are equal to the entries of the diagonal matrix @. Indeed, since A and S are rectangular mamces of full rank d, they have pseudo-inverses d + ,S+ such that

We will describe a number of methods (e.g., TAM, ESPRIT) that determine @ from ( X , Y ) using only the above decomposition (25), hence ignoring any further (shift-invariant) structure that A or S might possess. These methods are thus valid for any application in which an X and Y which obey (25) are somehow obtained, but for which no further information on the underlying structure is known. In particular, in the ESPRIT algorithm for DOA estimation, X and Y typically contain data from two identical sensor subarrays. H is then obtained by stacking X and Y , thus having size 2L x N . 3 ) Orthogonal Vector Methods: This class of techniques is related to the above two strategies, and can be
VAN DER VEEN et al.: SUBSPACE-BASED SIGNAL ANALYSIS

(dropping subscripts for ease of notation) and hence the equation F X = Y results in

FdS=d@S

@=d+Fd.

It readily follows that a subset of the eigenvalues of F form the entries of @. If F is taken to be rank d (e.g., the LS solution F = Y X + ) , then F has L - d zero eigenvalues and @ is equal to the d nonzero eigenvalues of F . If there is noise, X and Y will have full rank L. We will consider two classes of solutions to solve the problem in this case. In Section IV, the algebraic structure present in (25) is exploited; i.e., these methods rely on the fact that X and Y
1293

Authorized licensed use limited to: IEEE Editors in Chief. Downloaded on August 17, 2009 at 19:20 from IEEE Xplore. Restrictions apply.

should ideally have the same (d-dimensional) column space and row space. By SVD *analysis on X and Y , rank-d LS or TLS approximations X and Y are obtained that satisfy this property, without retaining any (Hankel) ,structure that might be present in X and Y . Setting F X = Y , and solving for F in a Least Squares sense, the entries of iP are obtained as the d nonzero eigenvalues of F . These methods are also known as Principal Component methods because the column/row spaces of X , Y contain the d strongest components in the column/row spaces of X , Y , and are obtained by projecting X , Y onto these principal subspaces. In many identification contexts (except DOA estimation with sensor doublets), the fact that Y has many entries in common with X is in principle not used in finding or projecting onto these subspaces. However, this fact can be exploited in the derivation of algorithms that are more computationally efficient. Section VI1 describes the Orthogonal Vector methods as an intermediary between Single Shift-Invariance and Subspace Fitting techniques. They can be written in the same style as the Single Shift-Invariance methods, operating on X and Y in (25) above, and using the single shift-invariance between them to obtain a different F, now having full rank L and a special structure. @ is obtained by selecting an appropriate set of d eigenvalues from the L eigenvalues of F . On the other hand, it can be shown that these methods compute a rank-L Vandermonde matrix A that is precisely orthogonal to one selected vector U in the noise subspace of H. Due to the structure of A, this reduces the problem to finding the roots of the polynomial U(.) associated with this vector. Taking A of rank d and maximally orthogonal (in some Least Squares sense) to all vectors in the noise subspace instead of just one, a connection with the Subspace Fitting class of techniques is obtained.

procedure described in the previous section which covers almost all algorithms in this and the next section is given in the following list. 1) Using the LS or TLS approximation algorithms of Section IV, estimate from the row space of X and/or Y a common d-dimensional row space, i.e., the row space of S.Let I, represent the orthogonal projector I onto this space. 2) Estimate from the column (range) space of X and/or Y a d-dimensional common column space, i.e., the column space of A. Let IIc represent the orthogonal projector onto this space. 3) Apply these projectors to X and Y to obtain the rank-d approximants

x = IIc x n,
Y

=ITc Y

n,.

VI. LEAST SQUARES SINGLE SHIFT-INVARIANT METHODS In the Single Shift-Invariant (or Principal Component) methods described in this section, the (L 1) rows of the data matrix H are arranged into two matrices X and Y , with X = H ( l ) consisting of the first L rows of H , and Y = H ( 2 ) consisting of its last L rows. As was already stated in the previous section, the first step in this class of solutions is to find rank d approximants X and Y to X and Y , and then invoke the (dS,A@S) structure of (25) to estimate iP. Any additional shift-invariance structure that might be present in X and Y is not used, and also not retained by this rank reduction. The approximation is performed by projections onto subspaces spanned by the d most important singular vectors derived from SVD analysis of X and/or Y , and the approximation norm is the Frobenius norm. In LS solutions, the projection operators are constructed from either the X data or the Y data, in a way that closely follows the definitions of II, and I, I in Section IV. In Total Least Squares (TLS) solutions, the subspaces, and hence the projection operators, are obtained from both the X and Y data [56], [15]. An outline of the

Next, find any matrix F such that F X = Y , and set the entries of the diagonal matrix @ equal to the e nonzero eigenvalues of F. These eigenyalues the rank-reducing numbers of the pencil (Y - A X ) . The solution is by no, mean: unique. Each of the projections used to obtain X and Y can be done in either LS or TLS sense, giving rise to at least four different, though clo2ely rejated solutions. In addition, a matrix F such that F X = Y cannot only be found in LS sense, in which case it will have rank d, but also in a predictor: form of full rank L, in which the first L - 1 rpws of X are just copied by F to Y , and the last row of Y is constructed by F as a linear combination of the rows of X . Although in the latter case F is of full rank L, only d eigenvalues are relevant to the solution and somehow these d eigenvalues must be separated from the rest. This fact can give rise to problems. Three of the four LS/TLS methods which lead to rank-d estimates of F have appeared in the literature, and are discussed below. Predictor methods are discussed in Section VII. Principal Component methods were introduced by Moore in 1978 (see [lo], [57]), who analyzed such methods on the Grammians of intemally balanced systems. This work is related to the Principal Hankel Component analysis discussed here. Related papers are by Zeiger and McEwen [9] and by Pemebo and Silverman [58]. In the past decade, several major contributions in this field have appeared in the publications of Kung er al. [ll], [59]-[61], in which infinite-data Principal Component algorithms and related covariance methods are discussed, with applications to state-space and harmonic retrieval problems. This research has led to a covariance-based method referred to as TAM, the direct-data variant of which is related to the LS-LS and TLS-LS cases discussed below. In another series of publications, Roy, Paulraj, and Kailath [34]-[36] have devised a comparable harmonic retrieval algorithm called ESPRIT, corresponding to the TLS-TLS case discussed below. Since then, a number of authors [62]-[65] have investigated the relationship between TAM and ESPRIT, and concluded that their statistical performance is asymptotically (i.e., for
PROCEEDINGS OF THE IEEE, VOL. 81, NO. 9, SEWEMBER 1993

1294

Authorized licensed use limited to: IEEE Editors in Chief. Downloaded on August 17, 2009 at 19:20 from IEEE Xplore. Restrictions apply.

N -+ 00) equivalent. Other authors have popularized the use of a pencil description for the same type of problem [66]-[68]. The classification below is both a summary and unification of the underlying concepts in the above publications, and does not precisely follow any of them in particular.
A . LS-LS Algorithm In the LS-LS type algorithms, both II,, the projector I, onto the common row space, and I, the projector onto the common column space, are determined from an SVD of X only (hence Least Squares). Following the outline above, the algorithm is in principle as follows: Determine the SVD of X:

B. TLS-LS Algorithm In the TLS-LS ^algorithm,a d-dimensional common row space for X and Y is obtained by SVD analysis of the full data matrix H of which X = H ( l ) and Y = H ( 2 ) are submatrices. This determines the projector IT, onto the row space. The projector TI, is the projector onto the column space of X n , , but is never explicitly formed because the projection is done implicitly in the computation of eig (F), as in the LS-LS case. The outline of the algorithm is as follows: 1) With X = H ( l ) and Y = H ( 2 ) , compute the SVD of the full data matrix H:

H=UCV*

fi=UgV*

X = ucv*.
The rank-d LS approximant X of x is X = UW*, where U and V contain the d singular veFtors corresponding to the d largest singular values C in E. The LS projectors onto these subspaces are

where V * represents the common d-dimensiona! rpw space of X and Y in the TLS sense, i.e., II, = VV*. Project X and Y onto this row space (hence TLS):

r;r = X

V V * = UgQ*

Y =y p v *=

u2w*,

n, = UU*
rI, = V V *
and the LS-LS approximations of X and Y are X = (qU* X

where U 1 consists of the first L rows of U , and U2*= U ( 2 )consists of the last L rows of U . Hence U1 and U 2 are matrices of size L x d, and in fact X and Y are just subm$riFes of H . 2) Set F = Y X + . Then F = U2UT, and
= eig(F) = eig( UT02)

pp* = UkQ*

y = uu*y Q Q * .

Put FX = Y , and solve for F in the LS sense:

F=

= uU*y V 9 - 1 0 * .

Compute @ = eig(F), discarding zero eigenvalues. Using the fact that the nonzero eigenvalues of a matrix product (AB) are equal to the nonzero eigenvalues of the product (BA), we can obtain @ as

It is thus seen that the actual computations needed in the LS-LS case amount to 1) computing the SVD of X, and 2) computing @ = eig (U*YVC-).This shows that the projection of ,Y onto the column space of 2 is in fact not needed (Y need not be computed) because this is a side effect of computing ( Y X + ) .The projection of Y onto the row space of X can also be omitted because the computation of eig(YX+) will impljcitly project Y onto the row space and column space of X (see Section IV). The LS-LS algorithm is akin to the direct matrix pencil algorithm described by Hua and Sarkar [67], [68], although in their approach Y is constructed from an SVD on Y, rather than by projections based on X. This has the conceptual advantage that X and, Y are treated equally. @ is then determined as @ = eig(YX+) as before.
VAN DER VEEN rr al.: SUESPACE-BASED SIGNAL ANALYSIS

(discarding zero eigenvalues). By construction, X and Y share the same row space. Again, the computation of eig (F)implicitly projects the columns of Y onto the column space of X in the LS sense. The above method is known in the DOA context as the LS-ESPRIT algorithm. As before, X and Y typically contain data from two identical sensor subarrays, H is obtained by stacking X and Y , thus having size 2L x N . The method also encompasses the diret-data TAM method for harmonic retrieval in [59], although the description of the compu!ation is slightly different here. It is observed in [59] that U; can be computed without inverting matrices because U 1 is almos: an isometry. To, s: this, denote by e U L the last row of U, and note that U;U1 u i u ~ Id. = Elaborating on this formula, it follows from UTU1 = I d that

U; = ( I d - U;uL)-lu;

The TLS-LS algorithm (as well as the LS-LS algorithm) is suitable for efficient SVD updating techniques [69], [70], which can be adapted to yield on-line estimates of @ for an increasing number of samples N .

C . TLS-TLS Algorithm In the above two algorithms, the actual choiceAofF results in an implicit L S projection of the columns of Y onto the column space of X when the eigenvalues are computed.
1295

Authorized licensed use limited to: IEEE Editors in Chief. Downloaded on August 17, 2009 at 19:20 from IEEE Xplore. Restrictions apply.

In the TLS-TLS method, an explicit projection isdone in the TLS sense by projecting the columns of X and Y onto a subspace that lies between the column space of X and the column space of Y. This subspace is obtained by computing the SVD of a matrix H1 = [X Y ]and retaining the d left singular vectors that correspond to the d largest singular values. Although this extra projection gives rise to results which are slightly different from TLS-LS, and presumably more accurate, the difference with the TLS-LS case for system identification is only marginal if N is large. The algorithm is given below. 1) With X = H(l)and Y = H(), denote H1 = [X Y ] , H = H . Compute the SVDs of H ; and H2, and 2 denote their rank-d approximants by HI and H2:

H = UlClV; 1
H 2

-+

H1 = U1k1V;
= U2k2V;

= UzCzV,*

+H 2

In this step, the common column andw spaces of X !r and Y are determined explicitly to be U1 and V v d :; the projectors onto these subspaces are IIc = U1U; and II, = V2V;. 2 ) Define X = U1U: . X . V 2 V ; =: UlE,V; Y = U1U: . Y . V2V; =: UIEyP; where E, and E, are d x d the following full rank matrices:

Multiplication by 2 2 can even be omitted, since this will not affect a. The above algorithm only requires computation of the SVDs of Hland H2, follFwed by the computation of the eigenvalues of the pair (UTU11, U,*U21). If X and Y are Hunkel matrices (as in system identification),then X and Y have all but two columns in common. It seems in this case better to omit the duplicate columns in H I , but in doing so HI and H2 differ in only one row and one column, and for large N the difference between the TLS-TLS and TLS-LS algorithms is negligible. If the Hankel assumption is not used, then the above algorithm is a modification of the more sequential TLS-ESPRIT algorithm of [35], [36] in which the projection onto the common column space is done first, and the common row space is then determined from the resulting smaller matrices. In the ESPRIT context, H is obtained by stacking X and Y . Because the noise on X is now unrelated to the noise on Y , the difference between the LS and the TLS variant can be significant.

D.Pro-ESPRIT
For completeness, and to indicate that there exists a litany of algorithms that are all based on (repeated) rankd truncations of matrices constructed from X and Y , we mention an algorithm based on Procrustes rotations [71], called Pro-ESPRIT. The algorithm can basically be formulated as follows [72]. Starting from data matrices X and Y as- before, compute (independent) rank-d approximations X, Y using SVDs:

E, = U: X V 2 E, = U: Y V 2 .

(26)

With these definitions, X and Y are rank d and share common column and row spaces obtained by (TLS) projections onto both the column space spanned by U1 and the row space spanned by V,*. They reflect the structure of the assumed model (25) in the sense that they are weighted outer products of rank-d rectangular matrices, the weights being the d x d matrices E, and E,. 3) Set F = Y X + , then
rr;.
r;.-i

Then the rank reducing numbers of Y - X X are equal to those of the rank-d pencil

QUgzQ:

and
@ = eig(E,E;l)

The computation of E, and E, in (26) can be done efficiently by defining the matrices U11 and U21 to be the first and last L rows of U2, respectively, so that

x = u11c2v;
Y = u21c2v; .
Substituting this in the definitions of E, and E, in (26) and using the fact that C2 V;V2= [& OIT, we obtain

E, = U:U1122

E, = UTU2122.
1296

with Qu = U T U 2 and Qw = V ; V 2 . Under noise-free conditions, Qu and Qv are unitary matrices. With noise they are not, but can beAreplacedA(approximated) their closest by unitary matrices Q, and Q.. This is called a Procrustes approximation, and Qu and Qv can be computed via SVDs of Qu and Qw by setting all singular values equal to one. Hence @ is determined from th,e $ r,ank reducing numbers of the approximated pencil QuC2Q: - XC1. In [72] it is shown that, under certain conditions, this algorithm yields resu!ts identiFa1 to those that would be obtained by replacing U1 and U2 in (27) with approximating unitary matrices sharing a common d-dimensional space. Thjs apprqximation is obtained via an SVD of [U1 U Z ]and VI , and V are approximated in the same fashion. The resulting 2 algorithm can be viewed as yet another (two-step) variant of the algorithms mentioned above, where a common ddimensional subspace of the column spaces of X and Y is determined in two successive steps.
PROCEEDINGS OF THE IEEE, VOL. 81, NO. 9, SEPTEMBER 1993

Authorized licensed use limited to: IEEE Editors in Chief. Downloaded on August 17, 2009 at 19:20 from IEEE Xplore. Restrictions apply.

E. Discussion It is a difficult matter to decide which of the above algorithms is to be preferred. They are all closely related, and their differences tend to disappear when N is large since they are all asymptotically (for large N) equivalent to a first-order approximation [72]. The variance of the estimated parameters is, however, smaller for the various TLS implementations. If a parallel array of processors is used, then there is not a dramatic difference in the number of operations between the LS-LS and TLS-TLS algorithm (less than a factor 2), because on a parallel array it takes about the same number of operations to compute the SVD of a matrix as it takes to apply the resulting U and V matrices to a second matrix. Pro-ESPRIT requires roughly twice the amount of computation, and is not necessarily more accurate. SVD updating techniques are very promising for an on-line (or real-time) implementation of the TLS-LS algorithm. In these techniques, estimates of @ are calculated for increasing values of N by updating the SVD's of X and Y obtained from some previous value of N [70].
VII. ORTHOGONAL VECTOR METHODS
A. Introduction

matrix (see (16)) u*H=O

++
w

u*d(@) 0 =

~(4i)0 =

(i = l , . . . , d )

(29)

which states that if U is in the left null space of H , then the d elements q5i on the diagonal of @ must be solutions of the equation u ( z ) = 0. Hence, the polynomials U(.) derived from all possible vectors U in the left null space of H have d roots in common, and in the noise-free case the choice of U in this null space is of no particular importance. In comparison with the previous paragraph, we see that U must be proportional to [g - 1]*.The equality between the eigenvalues of a matrix in bottom companion form (F in (28)) and the roots of the polynomial constructed from the last row of this matrix is a well-known result in linear systems theory [12]. Orthogonal Vector methods are sometimes called prediction methods. This is because when H is a Hankel matrix built from a time series h k , the entries g; of g can be thought of as the coefficients of a linear prediction filter (moving average filter)

As before, we assume that an (L 1) x N data matrix H is given, and let X = H ( ' ) represent the matrix constructed from the first L rows of H , and Y = represent the matrix containing the last L rows. The last (unique) row of Y is denoted yL. In contrast to the Least Squares Single Shift-Invariant methods of the previous section, Orthogonal Vector methods exploit the fact that Y is a shifted version of X, so that, in the relation F X = Y, F can be chosen to be an L x L matrix with the following structure: -0
1 0

F=

... ...
0
9

This reflects the fact that all but the last rows of Y are just copies of rows of X . The last row y L of Y is obtained as a linear combination of rows of X, gX = yL, and hence [g - 1]H= 0. Consequently, [9 - 1]* could be any vector in the left null space of H. As mentioned in the problem outline in Section V, @ = eig (F)has L eigenvalues, only d of which are relevant. In the noise-free case, the valid eigenvalues are independent of L. The remaining ( L - d ) eigenvalues depend on the particular choice of g. An alternative approach leading to the same result takes the analytic structure of H into account. With the definition a) = [l z z2 . . .IT, we can associate with the vector ( . U = [u1 2 ~ 2 ...IT a polynomial u ( z ) = u*a(z)= '111 U 2 z . . .. The basic property used by all Orthogonal Vector Methods is the trivial (noise-free) relationship that H = A(@)Ssatisfies when A(@) a Vandennonde-type is

that predicts a new data sample h k + l from knowledge of the preceding L samples { h k , . . . ,h k - ~ + 1 } , for k = L to k = N L - 1. In writing out the equations, it is seen that g predicts yL from a linear combination of the rows of X by minimizing the error (yL - gX). For such a matched filter, the zeros di of the prediction-error filter -1 G ( z ) are the zeros of U(.) in (29), and hence equal to the poles of the system that generated the data because the inverse prediction-error filter will have the original data sequence as its impulse response. There are many variants of such linear prediction methods. For example, when the data are known to be sinusoidal, as in the harmonic retrieval problem, then doing both a forward (as above) and a backward prediction (predicting h k from { h k + l , . . . , h k + ~ } ) yields improved accuracy. When covariance data are used instead of direct data, then the resulting relationships are known as the Yule-Walker equations; they are solved in precisely the same way as in the sequel to this section [73], [741. a noisy data matrix, then an approximation If H fi = UCV* may be formed from the SVD representation H = U W * . In this way, !he column space of H - i s split into a signal subspace U a?d a noise subspace U ' which is the left null space of H. There exist a number of Orthogonal Vector methods, each of which differs from the others in the actual choice sf the vector U in the noise subspace. Because, with noise, H no longer has a left factor A with Vandermonde structure, property (29) above is no longer valid and different selections of vectors U in the noise subspace lead to different solutions. A few of the possibilities are discussed in the subsections which follow. One of the problems associated with these methods is that only d of the L roots of the polynomial U(.) are of interest; namely, those that correspond to the system

VAN DER VEEN er al.: SUBSPACE-BASED SIGNAL ANALYSIS

1297

Authorized licensed use limited to: IEEE Editors in Chief. Downloaded on August 17, 2009 at 19:20 from IEEE Xplore. Restrictions apply.

poles. Apart from the computational overhead incurred in obtaining L roots (in comparison with the order-d eigenvalue calculations of the previous section), one is also faced with the problem of how to select these d roots. Each of the methods below has its own rationale behind its selection criteria. A few observations are indicative in this respect. One is that if H were noise-free, then the residues ri of the underlying model h ( z ) (in (3)) that correspond to the L - d extra roots would be zero, and hence these extra poles would be unobserved in the model. If H does contain noise, one might assume that these spurious residues are still small. Another observation is that for rank-d Hankel matrices H that are corrupted by additive white noise, the L - d 1 smallest singular values of H would be equal. The theory of Adamjan, Arov, and Krein (AAK) in [75] states that if the (d 1)st through (L 1)st singular values of H are equal, then the polynomials constructed from any of the corresponding singular vectors have d roots in common. Hence, for both the white noise and noise-free cases, all polynomials constructed from vectors that are in the noise subspace of H have d roots in common, and the results of each of the methods discussed below should asymptotically be the same. Below, a brief overview is given of four Orthogonal Vector methods: Pad6 approximation (as in [59]), KumareSan-Tufts (KT) Min-norm method with and without rank reduction, and AAK Hankel-norm model reduction. The first two methods are included for historical reasons. The different methods are characterized by the choice of the representative vector from the noise subspace (as in [76]) since the roots of the polynomial constructed from this vector are directly related to the poles of the approximating system.

that the restriction N = L is removed. The vector g is computed by trying to solve the overdetermined system of equations g X = yL for 9. With noise present, the null space of H* = [ X * YE] will contain no vectors at all; the row yL is not contained in the row space of X . However, after projecting yL onto the row space of X , resulting in y L = y L X f X , the null space of I?* = [X spans This g is also the solution precisely one vector: [g - 1]*. to the minimization problem

and is determined explicitly as g = G L X + .Note that the L s methods of Section VI with d = L yield precisely the same result since no actual rank reduction is done; i.e.,F = Y X + is the same as that obtained here. Pisarenkos method [77] for harmonic retrieval operates on a covariance matrix constructed on the given data but is essentially the same method (see 1601). These methods are still very sensitive to perturbations in X due to noise.

D.Kumaresan-Tufts Minimum-Norm Method

The Min-norm method proposed by Kumaresan-Tufts [78]-[83] is a modification of the above method to make it more robust for the separation of closely spaced sinusoids in the presence of noise. It amounts to the following three steps: 1) A solution to g X = yL is forced by reducing H = [X YE] to rank d. This can be done in two ways. The classical LS way would compute a rank-d approximation X from a? SVD of X , and project yL onto the row space of X to obtain an yL such that g X = yL has solutions 9:

B . Padk Approximation In this class of methods, the data matrix X is square and of full rank, so N = L and d = L. Hence the order of the system determines the size of the data matrices to be used, and vice versa. The vector g is defined by g x = yL
=+

= UCV + X = u g v *
yL = yL

v*v
* a

g = y p .

This is the counterpart of the LS-LS method of the previous section. A TLS method (cf. the TLS-LS method of Section VI), would compute the SVD of H and derive X , y L from the rank-d approximation I? as follows:

With F constructed from g as before (see (28)), we have F X = Y and @ = eig ( F ) . The approximating system which results is of degree d = L. Since this method uses all data without rank reduction, it is very sensitive to perturbations in X and yL [59], and the number of measurements directly determines the degree of the approximating system. The noise subspace is defined in this case by the null space of H* = [ X * YE], which has dimension one, and is spanned by the vector [g - 1]*. A related method is the classical method of Prony [39] for sinusoidal data.

which yields

2 = x v*v = olcl,cv* yL = y v*v = ( U ) L C V *

L A

C . Kumaresan-Tufts Method Without Rank Reduction In the Kumaresan-Tufts method without rank reduction, it is assumed that the L x N matrices X and Y satisfy N > L. Since no rank reduction is done, we still have d = L. In comparison with the Prony method, it is seen
1298

where ( 0 ) the last row of U. is ~ 2) The system g X = yL is now underdetermined, and the noise subspace of I? has dimension L - d 1. Of the many possible vectors [9 - 1]* in this subspace, choose the one with minimum norm 11 g 112, i.e., choose g = yLX+ = y L X + as in the previous case. If the TLS approach is used in the above step, then we can show that in fact

[g

-1 1

- (UJI)@*

PROCEEDINGS OF THE IEEE, VOL. 81, NO. 9, SEPTEMBER 1993

Authorized licensed use limited to: IEEE Editors in Chief. Downloaded on August 17, 2009 at 19:20 from IEEE Xplore. Restrictions apply.

in whifh U L spansnthe noise subspace defi?ed via U = [U U l l , and ( U L ) is the last row of U L (see ~ also [84]). This determines precisely which vector of the noise subspace is chosen. 3) The estimated d poles are a subset of the eigenvalues of F, with F as in (28). We could also compute the roots of the polynomial associated with [9 - 1]*, leading to the same result. In comparison with the previous method, the rank reduction to order d in combination with a null space vector of dimension larger than d greatly improves the previous two methods [82]. The choice o,f g to have minimal norm among all vectors g that satisfy gX = GL forces the extra L - d+ 1 eigenvalues of F to lie regularly spaced on a circle of minimal radius within the unit disc [78]-[80], [85]. This property can be used to select the d desired eigenvalues.

E, AAK Hankel-Norm Approximations The ultimate goal of the methods considered in this paper is, given a (full-rank) matrix H, to-find a rank d approximating structured matrix fi = A(@)S that minimizes in an appropriate norm the difference H - fi. In Section VI, the minimizing norm was taken at first to be the Frobenius norm:
min fi rank d

fill$.

(30)

i.e., it is the approximation in Lz norm of the Hankel matrix associated with h(z).Unlike the Frobenius norm, the Hankel-norm approximation allows the d vectors spanning the range of H to have components outside the range spanned by the first d singular vectors of H without penalty on the n o m of the error, because the norm only measures the largest singular value. This enables H to take on a Hankel structure, something that the SVD methods of Sections VI and VI1 were not able to achieve. We can summarize the main results [59], [75], [86]-[91] as follows, favoring vector notations over polynomial descriptions, when possible, for better comparison with the previous methods. Given a matrix H of infinite size, representing a stable high-order system, let H = UCV* and denote the (d+ 1)st column of U by Ud+l. Wit: U = [U 611as before, U d + l is the first column of U-, the noise subspace. The corresponding singular value and right singular vector of are denoted g d + l and ud+1. The polynomial 1ld+l(z) constructed from U d + l has pretisely d stable roots & inside the unit circle. If H is a rank-d Hankel matrix approximating H according to (31), then the minimum error (1 H 1211= s u p U l l u * ~ U * H I Iequals O d + l and is attained by the corresponding left singular vector ud+l O H : f
U:+1(H

However, the minimizing fi does not have the required shift-invariance structure. By ignoring this fact, and using properties that H would have in the noise-free case, we were able to derive a reduced-order model that does possess shift-invariance structure and is presumably not too far away from H . Unfortunately, to date no bound has been found to quantify this error. In Section VIII, techniques will be discussed that do solve the above minimization problem in the Frobenius norm, taking the structure of the approximant into account. This is a highly nonlinear operation, leading to complicated search techniques. Under certain conditions, however, it can be shown that a structured solution can be found when a different norm is applied. Such a norm is the Hankel norm. In a celebrated paper, Adamjan, Arov, and Krein [75] have demonstrated that, when H is a Hankel matrix of infinite dimensions, but of finite rank and bounded Lz norm, there exists a unique Hankel matrix H that is the solution to a related minimization problem: min
H
rank

-H

) = gd+lV:+]

1 1 -fill

(31)

in which the matrix

L2

(operator) norm is minimized

sup ( ( H z - i i z ( 1 2 . I12 112=1

IIH-HI( =

Recall that the LZ norm of a matrix is in fact equal to its largest singular value. The use of this norm leads to a socalled Hankel-norm approximation of the impulse response vector h on which H was built, or its polynomial h ( z ) ;
VAN DER VEEN
CI

where u d + 1 is the (d+l)st right singular vecto? Since Cd+lU:+l, we must have U : + ~ H= 0. Hence the columns of 1 are all orthogonal to ?&+I, 2 or, in the context of the previous section, U d + l is in the noise subspace associated with H . Combining the above two properties, it is concluded that the d stable roots of Ud+l(z) define the best rank-d Hankel approximant in the Lz norm. above properties are derived only for infinitedimensional Hankel matrices. If a high-order (stable) model of h ( z ) = b ( z ) / a ( z ) is known, for example in the form of a high-order state-space model, then the theory can be extended to operate on Hankel matrices of finite size that are larger than or equal to the model order, thereby obtaining the same results [591, [86], 1881, [89], [92]. The singular values and vectors of the infinite-dimensional Hankel matrix can then also be easily computed [93]. If operating on Hankel matrices that are windowed (finite) versions of infinite Hankel matrices (as is the case throughout this paper), then the above theory is no longer applicable, although the solution is continuous when the rank of the matrix is finite and the dimension is larger than the rank. However, in general it can easily happen that Ud+l(Z) has more or fewer roots than d in the open unit disc, especially if the data are corrupted by noise, and hence the rank of the underlying infinite matrix is not finite. One way to avoid these problems is first to derive a high-order stable model using other techniques, and then use an extension of the AAK theory that works on finite-size state-space models to

u:+~H =

ol.: SUBSPACE-BASED SIGNAL ANALYSIS

1299

Authorized licensed use limited to: IEEE Editors in Chief. Downloaded on August 17, 2009 at 19:20 from IEEE Xplore. Restrictions apply.

obtain a rank-d reduced-order model fi. Fecise formulas appear in [94], [95, p. 4521. Since this H is obtained by a two-step process, it will be a suboptimal solution to (31). However, it will approach the optimal solution as N , L + 00. AAK Hankel-norm model reduction methods can also be extended to the time-varying context [96].

F. Root MUSIC: A Link to Subspace Fitting Techniques To link the methods of this section with the Subspace Fitting techniques of the next section, we briefly discuss here a derivation of (root)-MUSIC. In the introduction to this section, we noticed that the basic form of the Orthogonal Vector methods is simply
u*H=0

which are most orthogonal to the noise subspace. Note that MUSIC cannot force the null spectrum to be zero since it only uses vectors o(4) from the array manifold in its search; i.e., instead of rooting a polynomial as above, it finds points on the unit circle where the sum of squared polynomials is minimized. On the other hand, root-MUSIC finds the exact roots of this polynomial, and then estimates 4i, i = 1,. .. ,d by projecting these roots onto the unit circle. VIII. SUBSPACEF I " G TECHNIQUES In this section, the class of Subspace Fitting techniques for solving the direction-of-arrival estimation problem is considered. The discussion follows the framework of Viberg and Ottersten [97] .and Stoica et al. [98], [99], whose recent work provides an enlightening overview of the DOA estimation problem and new results on the asymptotic behavior of the estimate errors. The generic subspace fitting problem considered in [97] is the ^followi?g: given some representation of the data M , find @ and T such that

U(&)

=0

(32)

which means that for a selected U in the left null space of

fi, the roots of U(.) are viable estimates of di. However, as fi does not have the Vandermonde structure, different
choices of U in this null space will lead to, different estimates {di}. Because the left null space of H is, by definition, spanned by fil, we can write U* = wUl* for some row vector w of dimension L - d 1. For example, in the AAK approach w = [l0 . ..01 selects the first vector in the noise subspace, while for for the Kumaresan-Tufts TLS method, w = (U')L is the last row in UL. Now, using the notation a($) := [l 4 4' ... (32) is equivalent to the polynomial equation in q5

@,T argminip,T 11 A - A(cP)T 1; = 4

,...

(34)

w P * a ( 4 )= 0 .

for T of suitable size, and with A(@) T of rank d. In and the sequel, we will often write just A instead of A(@). Due to the special structure of A, this is a nonlinear optimization problem, separable however into a linear part in T and a nonlinear part in A. Substituting the solution of the linear part, T = A + M , back into (34) gives

Orthogonal Vector methods select one specific vector w, and search for the roots of the above expression. In this context, the idea behind the well-known DOA estimation algorithm MUSIC is not to select a single^w,but instead to work with the full polynomial null space UL*a.In particular, root-MUSIC exploits the fact that in the noise-free case, as well as in the infinite-data white-noise case, all entries of the column vector of polynomials U l * a ( z ) UL(z) have d roots in common. The root-MUSIC algorithm, as a spectral estimation method, then makes the assumption that these roots lie on the unit circle, and e2timates them by rooting the sum of squared polynomials U'*(z-')U*(z), retaining only the d roots in the unit disc with modulus nearest unity (only roots inside the unit circle need be considered since the squaring operation forces conjugate reciprocal roots). To connect this Orthogonal Vector method with the Subspace Fitting methods of the next section, note that the root-MUSIC technique was derived from the MUSIC algorithm, which obtains parameter estimates by minimizing the so-called MUSIC null-spectrum:

i = argminip 11 (I- n A ) M ;1
= argmaxipTr( II,((a)

MM* )

(35)

in which HA(@) = dd+ is the LS projector onto the column space of A ( @ )and Tr denotes the trace operator.2 , Several popular DOA estimation algorithms may be cast in the form of (35). These include the deterministic maximum-likelihood method [79], [98], [ 1001-[ 1041, multidimensional MUSIC [35], [105], as well as Weighted Subspace Fitting (WSF) [97]. The MODE algorithm of Stoica et al. also has a related interpretation [98], [99]. In our discussion of identification methods so far, we have been able to avoid the notion of covariance matrices. However, the Subspace Fitting techniques have been introduced in the literature in a statistical framework, and hence the analysis is traditionally not done directly on the data, but rather on the covariance matrix of the data. There are Strong links between the two, and it is possible to avoid the notion of covariance altogether (as we have done in the preceding sections), but in the discussion of the present section the use of covariance matrices avoids certain complications. Denoting the ((L 1)x N)-dimensional output data matrix H of (24) by H = H N = A(@)SN V N ,the relevant

i = 1,..., d

(33)
2Recall that the trace of a matrix is defined as the sum of its diagonal entries. We will use some of its properties: 1) the trace of a projection operator is equal to the dimension of the subspace on which it projects, 2) Tr(-4B) = Tr(B.4). for matrices -4and D of compatible size, 3) (1 A 1; = Tr (..l*A).
PROCEEDINGS OF THE IEEE, VOL. 81, NO. 9, SEF'TEMBER I993

for q& on the unit disc. It can be seen that MUSIC attempts to find, one at a time, d vectors a(4i) from the array manifold which most closely fit the signal subspace, or
1300

Authorized licensed use limited to: IEEE Editors in Chief. Downloaded on August 17, 2009 at 19:20 from IEEE Xplore. Restrictions apply.

covariance matrices are defined as Signal covariance: P = lim - SN Sg

N-+w

substitution into (37) reduces the minimization problem to

1

N N

6 = arg m a + Tr ( nd(@) ) . RN
(36)

(38)

Output covariance: R = lim

N+w

H N H; .

With the assumption that the additive noise matrix is a realization of a stationary, zero-mean white Gaussian process with spatial covariance a21, we have R = A ( @ ) A * ( @ ) a21 P if the noise and signals are uncorrelated. If A ( @ ) Pis full rank d, it is easily seen that the L 1 - d smallest eigenvalues of R are all equal to a2.This fact is reflected in our notation for the eigenvalue decomposition of R

R = E,A,E.:

+ E,A,E;
AIL= a21

= EJEQ

+a

2 ~

where E = [E, E,,] is unitary

The algorithm resulting from implementation of either of the two above extremization problems is referred to as deterministic, or conditional, Maximum Likelihood (ML) [79], [98], [loo]-[104]. Since A ( @ )is nonlinear in the entries of @, its computation requires in general a complicated multidimensional search over the parameter space. Asymptotic properties of the deterministic ML method are given in [97]-[99]. Note that deterministic ML can be cast in the Subspace Fitting framework of (34) if the matrices M and T are chosen to be H N and S N ,respectively. Using asymptotic arguments, another connection with Subspace Fitting can be made [97]. For large N , we have A, -+ a21 and . a21. As the trace of a2nd is a constant, RN + E J E , it can be omitted from the optimization and, from (38), it then follows that the ML solution is asymptotically (for large N ) equivalent to the solution obtained from
^

A = A,

- a21

6 = arg min+,T 11 E,A1/2- A(@)T1;

and E, and E,, are isometries of rank d and ( L 1 - d ) , respectively. Since the column space of E, is equal to that of d(@)P, is referred to (as above) as the signal subspace. it The column space of E,, is correspondingly referred to as the noise subspace. Since in practical applications we cannot allow N 00, the above quantities must be estimated using finite sample averages. Thus the sample covariance RN of the data is computed as in (36) by removing the limit statement. Estimates of the signal and noise subspaces are then simply obtained by performing an eigenvalue decomposition o n R N , and these estimates will be denoted as E, and E,. Comparing R N with H N = U C V * and its rank-d approximation H N = U C V * , where U = [U as usual, we can identify U = Es,U 1 = E,,, and g 2 / N = Ag. This provides the link between the SVD of a data matrix and the eigenvalue decomposition of the estimate of its covariance matrix. An estimate of c2 can be obtained by simply averaging the L 1 - d smallest eigenvalues OfRN. The remainder of this section is devoted to a brief overview of the various Subspace Fitting methods, based on specific choices of M in (34) and (35).
--f

= arg maX+Tr(fld(@)k,Ak,*) .

(39)

This is again an instance of the generic-S_ubspaceFitting problem in (34) and (35) for M = E,A1/2 and T of dimension d x d. Using the weighting A, (39) minimizes the distance of the d-dimensional shift-invariant subspace of A to the signal subspace E,. In going from the formulation of (37) to that of (39), we see that the minimization has been made more compact; i.e., it involves d columns of data instead of N .

UL]

B . ESPRIT and MI-ESPRIT In Section VI, it was mentioned that the (TLS) ESPRIT algorithm [35] was a special case of the TLS-TLS principal component approach. It has recently been noted [97], [ 1061 that ESPRIT also has a Subspace Fitting interpretation. In particular, it can be shown that the ESPRIT algorithm is equivalent to the following least squares minimization problem:

A . Deterministic Maximum Likelihood If we assume that the columns of V N are stationary, independent, zero-mean, circular, complex Gaussian random vectors, and that the signals corresponding to the matrix SN are deterministic (as in the pole estimation problem), then maximizing the log likelihood of the data H N with respect to @ andSN can be shown [102], [lo31 to be equivalent to the following minimization problem:

where E l and E 2 contain the rows of Es corresponding to the two identical subarrays; e.g., for the uniform linear array described in (16), two maximally overlapped subarrays will yield El = E ( 1 ) equal to the first L roAwsof E , and E2 = containing the last L rows of E. The obvious connection with (34) is made by describing A ( @ )as in (18) and letting

M=

6, = arg niin+,sN )I H N - d(@)sN . sk1;

[$I.

(37). If instead of just two subarrays, the array is composed of multiple identical subarrays, a similar Subspace Fitting
1301

The solution of the linear part gives SN = d+HN, and

VAN DER VEEN
et

al.: SUBSPACE-BASED SIGNAL ANALYSIS

Authorized licensed use limited to: IEEE Editors in Chief. Downloaded on August 17, 2009 at 19:20 from IEEE Xplore. Restrictions apply.

approach may be formulated. Letting E represent the rows i ofE, corresponding to the ith subarray, and @i the diagonal matrix of phase delay factors due to propagation of the (plane) wave from the reference to the ith subarray, the most natural extension of ESPRIT is given by the following minimization problem:

described by replacing M with E, in (34):

6 = arg min, 11 E, - d(@)T1; = arg max, Tr( HA(@) fisk:) .

(43)

The motivation for the terminology "one-dimensional" and "multi-dimensional" MUSIC becomes clear when comparing (43) and (42).

@ = arg

a1

E. Weighted Subspace Fitting (WSF) In the Weighted Subspace Fitting method of Viberg and Ottersten [97], the optimality criterion is defined as (cf. (34) and (39))

where we have assumed a total of p 1 identical subarrays. Algorithms based on this approach have been developed in [107], [lo61 for the case where @i = ai, in [lo81 for and the two-dimensional (azimutWelevation) case. When = ai above, a generalized Vandermonde structure results as evidenced by the multiple-shift structure in the signal subspace. The algorithm for this case is referred to as Multiple Invariance (MI) ESPRIT. One drawback relative to (41) that should be mentioned is that the elegant "closed-form" SVD solution of ESPRIT is not applicable; minimizing (41) requires a nonlinear multidimensional search when p > 1.

6 = arg mina 11 kSW1/' - d(@)T 11%

= arg m m a Tr ( U() ,@

E,wE,*

1.

(44)

c. MUSIC
Although the Subspace Fitting paradigms of (34) and (35) are inherently multidimensional, similar one-dimensional formulations are also possible. For example, if the MUSIC [51], [lo91 cost function introduced in (33) is normalized it by dividing by a*($i)a($i), may be re-written as
$i

In this method, a positive definite weighting matrix W is introduced. We showed earlier that the deterministic ML method corresponds to the case where W = A1/'. Viberg and Ottersten have shown [97] that W can be chosen to asymptotically (for large N) minimize the estimation error variance of the parameters &, and that the optimal choice or a consistent estimate thereof. for W is WO,, = This choice for W has also been shown to make WSF statistically eficient; i.e., the WSF estimates asymptotically achieve the Cramtr-Rao lower bound on the variance of the estimation error under the assumption that the signal waveforms are Gaussian random processes [ 1131.

= arg max+ Tr ( &($)

E,E,* ) ,

(42)

where IIa($) is the projection onto the vector a($).The only difference between (42) and (35) above with M = Es is that while (35) implements a search for all of the parameters simultaneously, MUSIC searches for them one at a time. Thus MUSIC can be classified as a onedimensional Subspace Fitting technique. The asymptotic properties of MUSIC have been studied, a.o., in [98], [110]-1121. One of the interesting results of these studies is that deterministic ML and MUSIC have equivalent asymptotic performance if the sources are uncorrelated and of equal power.

F. Method of Direction of Arrival Estimation (MODE) Using the orthogonality of the, estimated signal and noise subspaces defined by E, and E,, an algorithm that is in some sense a dual of the Subspace Fitting approach in (43) can be developed. In this approach, one estimates the parameters @ as those for which d(@)provides the worst fit (i.e., most orthogonal) to the estimated noise subspace. Such an approach has been formulated in [99] by considering a criterion function of the form

6 = arg min, 11 l?cd(@)W,"2 11%

= arg min, Tr ( X E , E d W1 ) c

(45)

D.Multidimensional MUSIC
Although relatively simple to compute, MUSIC does not give accurate results if the signals are highly correlated. This is primarily because the parameter search is done one dimension at a time. Schmidt [51] hinted at a multidimensional (MD) counterpart to MUSIC that would overcome this difficulty, and Cadzow independently developed such an algorithm [ 1051. The resulting algorithm, which has been referred to by several authors as MD-MUSIC, can be
1302

The estimation error covariance is shown in [99], [114] to be minimized by the weighting W1,opt = (A*Ud)-l, where U = E, A2AT1 E,*, and the resulting algorithm using this weighting is referred to as MODE. It can easily be shown that both WSF and MODE yield results with identical asymptotic second-order error statistics [ 1 151. Note also that the MUSIC algorithm is equivalent to (45) when W1 = I, and that deterministic ML is asymptotically equivalent to (45) when W:" = S or W1 = P [ 1141.

G . Identification via Subspace Fitting While the description of the above algorithms has been couched in the problem of DOA estimation, the subspace fitting concept may also be directly applied to the pole estimation (i.e., system identification) problem. To see this,
PROCEEDINGS OF THE IEEE, VOL. 81, NO. 9, SEPTEMBER 1993

Authorized licensed use limited to: IEEE Editors in Chief. Downloaded on August 17, 2009 at 19:20 from IEEE Xplore. Restrictions apply.

recall (12), where it is shown that the column space of the matrix R22QZ is equivalent to that of the observability matrix 0. Without measurement noise, there will exist a full rank d x d matrix T satisfying

E=0T
where E represents the d principal components of R 2 2 Q z . With noise, R22Q; is full-rank, and we are led by the subspace fitting results above to consider the minimization problem [ 1 161

A , c = arg minA,cll
= arg maxA.C Tr &(A,

* . .

- OT

1;

C) kWk*) (46)

where A and C are the matrices of the state-space model upon which 0 depends. Because of the special shift structure inherent in 0, we see that the minimization problem of (46) is isomorphic to that of the MI-ESPRIT algorithm described by (41). As with MI-ESPRIT, implementation of (46) is somewhat more difficult than for the single-shift invariance methods of Section VI. Whereas in the latter case the estimates are obtained directly via one or two SVDs, solving (46) requires some type of search technique. However, since single-shift methods can be used to efficiently obtain an accurate initial estimate, a Newton-like gradient search will rapidly converge to the desired solution. Details of a Gauss-Newton implementation can be found in [ 1061. One might immediately assume that the weighting matrix W could be chosen to minimize the variance of the parameter estimates, as does the WSF algorithm. Strictly speaking, however, the optimality of WO,, has only been derived for the case where the observations (columns) in H are independent (as is the case in the DOA estimation problem). In the pole estimation problem, the Hankel structure of H violates this assumption. However, simulations indicate that the weighting nonetheless has the desired effect of reducing the variance of the pole estimates. Ix. PROPERTIES OF THE IDENTIFICATION METHODS The previous three sections have introduced perhaps an overwhelming number of algorithms and methods, all computing approximately the same quantity. How does one go about selecting an appropriate algorithm for a given application? Usually, the tradeoff that must be addressed in answering this question comes down to estimation accuracy versus ease of implementation and computational complexity. As a general rule, recent literature conveys that among the identification methods mentioned in the previous sections the best estimation performance is obtained by the optimal Subspace Fitting methods (e.g., WSF, MODE), whereas the most computationally efficient solutions are obtained by the Single Shift-Invariant methods of Section VI. However, since there are often other variables and tradeoffs to consider, the question above is often not so easily answered. For example, in the array processing context, if the source signals are highly correlated (e.g., due to
VAN DER VEEN
rf

specular multipath: the same source is observed directly as well as via reflections), then one of the multidimensional Subspace Fitting methods must be selected. On the other hand, these methods require full knowledge of the sensor array geometry and sensor properties (i.e., the array must be calibrated), while ESPRIT exploits the special doublet structure of the array and does not require precise locations and response properties of the sensors. To conclude the paper, we will briefly describe these tradeoffs in more detail. Our focus will be on the DOA estimation problem, since this is where most of the research in this area has been conducted. Because of the large number of techniques discussed, it is impractical to conduct and present the results of extensive simulation studies in this paper. Instead, we choose to qualitatively describe the results that others have obtained in various performance analyses. We refer the interested reader to the papers cited in this section for the actual numerical results of such simulation studies.

A. Performance Analyses In the past several years, there has been considerable interest in investigating the statistical properties of the various methods mentioned in the previous three sections. In particular, the goal of this work has most often been to derive theoretical expressions for the variance of the pole or DOA estimates obtained by these algorithms. Since this is very difficult to do in general, the theoretical studies are usually limited to the large sample case (i.e., large N), and hence can be considered to hold only asymptotically. The picture can be completed by numerical examples for finite N . It should be noted that since these studies have concentrated on the DOA estimation problem and its corresponding assumption of independent noise samples, their results are not directly applicable to the system identification problem since the additive noise has (by construction) a Hankel structure that cannot be regarded as a set of L x N truly independent random variables. For the DOA application, most of the algorithms mentioned in this paper have been investigated, and more or less final results have been published [97], [98], [117], [118], which we summarize below. The results have been obtained for signals modeled as stationary stochastic processes, with temporally uncorrelated zero-mean jointly Gaussian distributions. The noise is assumed to be a zero-mean temporally uncorrelated white Gaussian process that is also uncorrelated with the signals (there are a few other more minor conditions). It has been shown that ESPRIT, Deterministic Maximum Likelihood, MUSIC, WSF, etc., are all asymptotically unbiased; that is, the estimated parameters converge to the true parameters as N + CO with probability one. However, the second-order performance (estimation error variance) of these algorithms can be very different, and it is usually this second-order performance that is used to evaluate them. This evaluation is often conducted with respect to the so-called Cram&-Rao Bound (CRB), which provides a lower bound on the estimation error variance of any unbiased estimator.
1303

al.: SUBSPACE-BASED SIGNAL ANALYSIS

Authorized licensed use limited to: IEEE Editors in Chief. Downloaded on August 17, 2009 at 19:20 from IEEE Xplore. Restrictions apply.

For historical reasons, the MUSIC algorithm was the first to have its performance extensively analyzed [98], [ 1lo]-[ 1121. Among other results obtained in these papers, it has been shown that MUSIC is a large sample realization of the deterministic Maximum Llkelihood (ML) method if the signals are uncorrelated ( P diagonal) [98]. Under this condition, both algorithms asymptotically achieve the deterministic signal CRB, where by asymptotic we mean for both large N and L. For finite L , however, neither method is statistically efficient. For correlated signals, deterministic ML will generally outperform MUSIC. In cases where the signals are highly correlated, MUSIC will often fail to resolve all d of the signals; that is, there will be fewer than d local maxima in the MUSIC spectrum of (42). This loss of resolution can also occur when the signal-to-noise ratio is very low, or if the signals arrive from nearly coincident directions. One of the advantages of the Orthogonal Vector formulation of MUSIC, i.e., root-MUSIC, is that it does not exhibit this loss-of-resolution threshold effect3. Above the threshold, however, both MUSIC and root-MUSIC yield estimates with identical asymptotic variance [ 1121. When compared with the other Orthogonal Vector methods, rootMUSIC has the lowest estimation error variance that can be achieved by selecting only one orthogonal vector [ 1171 from the noise subspace; in particular, it has a lower variance than Pisarenko, Min-Norm, and AAK. The same has been observed in [ 1191 via other methods. As with Orthogonal Vector methods, the Single ShiftInvariant techniques of Section VI are guaranteed by construction to always produce the correct number of parameter estimates. However, these algorithms will also fail when the signals are perfectly coherent, or nearly so. In this case, a failure is manifest by one of the estimates taking on what is essentially a random value. Among other results obtained for the Single Shift-Invariant methods of Section VI, it has been shown that TLS-ESPRIT and LS-ESPRIT are asymptotically equivalent 1641, [ 1 181, although for small N TLS-ESPRIT has slightly better empirical performance. It has also been shown that TLS-ESPRIT is in general asymptotically less accurate that MUSIC [ 1201, although comparing the two algorithms is somewhat unfair since they rely on a different set of assumptions about the sensor array. In particular, MUSIC requires much more information about the array, and hence its superior performance is to be expected. A recent nonasymptotic comparison between Orthogonal Vector methods (MUSIC, Min-Norm) and Single Shift-Invariant techniques (TAM, ESPRIT) has appeared in [43], [121], and supports the above asymptotic results. In these papers, closed-form expressions for firstorder approximations of the perturbation of the signal and noise subspaces are derived. One of the greatest advantages of the multidimensional Subspace Fitting methods of Section VI11 is their ability to provide accurate parameter estimates in the presence
Strictly speaking, root-MUSIC does have a performance threshold that results when the algorithm chooses a spurious root from its polynomial. However, this effect is manifest well beyond the MUSIC threshold.
1304

of perfectly coherent signals. Of these methods, WSF and MODE possess the smallest estimation error, and in fact both methods asymptotically achieve the CRB under the Gaussian signal and noise model [97]. Thus both WSF and MODE can be thought of as large sample realizations of the Maximum Likelihood method for stochastic signals [ 1021, [122]. An important result derived in [113], [I141 states that asymptotically, deterministic ML is statistically less efficient than WSF, MODE, and stochastic ML, independent of whether one assumes the signals are random or not. The performance difference between these algorithms and deterministic ML can be quite large in difficult cases involving highly correlated, closely spaced signals at low signal-to-noise ratios. Our discussion thus far in this section has implicitly focused on algorithm performance degradations due to additive noise. Another important practical consideration is the sensitivity of the algorithms to various modeling assumptions, the most important of which is the assumption of a perfectly uniform linear array of identical sensors (or, in the general case, a perfectly calibrated array response). Such analyses have been carried out for many of the algorithms discussed thus far, including MUSIC [123]-[125], ESPRIT [126], [127], deterministic ML [128], and Subspace Fitting algorithms in general [129]-[ 1321. One of the surprising results to come out of these studies is the fact that, under the assumption of simple Gaussian perturbations to the array response and infinite data ( N + . CO), MUSIC yields lower variance estimates than MODE, WSF, MD-MUSIC, and deterministic ML [125], [132]. A Subspace Fitting minimization of the form (44) can yield performance equivalent to MUSIC in such cases, but it requires a weighting matrix W quite different from that of WSF. In the context of system identification, theoretical studies comparing several Matrix Pencil and Orthogonal Vector methods have been carried out in [68], [72], [Sl], [120], [ 1331, for the harmonic retrieval problem. As already noted above, in this problem the noise matrix has a Hankel structure, and its columns cannot be regarded as being independent. This fact makes the analysis somewhat more difficult, although some results have been obtained. For example, the conclusion of the study in [120] is that MUSIC and ESPRIT perform almost equally, although usually ESPRIT is slightly better (this contrasts with the DOA problem). For signals with unknown damping factors, the Single Shift-Invariant methods of Section VI are less sensitive to noise than the Orthogonal Vector methods [68], [72]. A significant increase in accuracy for these methods is obtained by increasing L, because the error variance is proportional to l / ( L 3 N )[120]. This is interesting because for a given set of data, one is free to choose the blocking factor N I L of the Hankel matrix constructed on the data, as long as d 5 N , L. Note, however, that the computational complexity is also proportional to L 3 , and that we still require N >> L. For the special case of only one signal, it has been derived [68] that the best choice for the pencil method is ( N - L ) / 3 5 L 5 2 ( N - L ) / 3 . For model
PROCEEDINGS OF THE IEEE, VOL. 81, NO. 9, SEPTEMBER 1993

Authorized licensed use limited to: IEEE Editors in Chief. Downloaded on August 17, 2009 at 19:20 from IEEE Xplore. Restrictions apply.

reduction, the noise due to unwanted high-order modes is actually deterministic, and cannot be modeled as white noise; hence, the statistical results obtained in the DOA context are not necessarily valid. In fact, one wants to have a bound on the modeling error 11 h ( z ) - h ( z )11 in some suitable norm. At present, only the AAK method provides such a bound (in terms of the Hankel norm).

R has a block decomposition into four blocks, such that R12 and R22 both have small Frobenius norms, and the smallest singular value of R11 is of the order of the smallest singular value of X that one does not want to neglect. In
this way, one still obtains a decomposition of the range space of X into a signal subspace and a noise subspace. The URV decomposition can be updated and downdated at lower computational cost than the SVD, which makes it a useful tool for adaptive subspace tracking algorithms.

B . Computational Aspects Although WSF, MODE, and stochastic ML are optimal in the sense of minimum asymptotic estimation error variance, the minimization of their various error criteria can only be achieved by iterative, nonlinear optimization procedures. These procedures are necessarily complex, and must be given initial estimates of reasonable quality to guarantee convergence. In [53], a Gauss-Newton descent method is proposed that can be used for both WSF and other ML techniques, and that requires O ( L d 2 )operations per iteration. Compared with the fact that the computation of the SVD for an (L x N) matrix requires O ( L 2 ( N 20L)) operations, the cost of each Gauss-Newton iteration is relatively small. The number of iterations required for convergence depends of course on the quality of the initial estimates. When ESPRIT is used to obtain the starting point, adequate convergence can be expected in two to three iterations. A number of empirical studies [53], [ 1151 have indicated that WSF has better convergence properties than both deterministic and stochastic ML. In comparison with Subspace Fitting and Orthogonal Vector methods (OVM), Single Shift-Invariant methods (such as ESPRIT) are computationally more attractive. The number of operations required for the SVD part of these algorithms is the same as for Subspace Fitting and OVM, but the eigenvalue computations can be done on d x d matrices in the SSI class, while the OVM requires the solution of a larger L x L eigenvalue problem, after which the d valid eigenvalues must be selected. Because of the regularity of the operations, the Single Shift-Invariant methods are amenable to implementation on parallel arrays of processors, of which the basic operation is a Jacobi (plain) rotation [ 1341. In many signal processing applications, the identification problem is solved several times, using new data as it becomes available, and discarding the older data. There is recent interest in developing efficient updating techniques, which will result in an on-line processor array that can update the pole or angle estimates each time a new sample vector is received (updating) and an old vector is discarded (downdating). One such updating scheme, based on an approximate SVD that will converge for stationary signals, is reported in [70]. To alleviate the cost of computing the SVD, alternative but computationally less demanding decompositions of the form X = UE,V*, where E, is not diagonal any more, are gaining interest. Recent developments are the rankrevealing QR factorization [135] which can be updated [ 1361, and the rank-revealing URV decomposition [ 1371, where E, =: R is upper-triangular. In this decomposition,

REFERENCES
L. Ljung, System Identification - Theory for the User. Englewood Cliffs, NJ: Prentice Hall, 1987. T. Soderstrom and P. Stoica, System Identification. Englewood Cliffs, NJ: Prentice Hall, Inc., 1989. L. Ljung and T. Sijderstrom, Theory and Practice of Recursive Identification. Cambridge, MA: MIT Press, 1983. L. L. Scharf, The SVD and reduced-rank signal processing, in SVD and Signal Processing, II: Algorithms, Analysis and Applications, R. J. Vaccaro, Ed. New York: Elsevier, 1991, pp. 3-31. S. Haykin, Adaptive Filter Theory. Englewood Cliffs, NJ: Prentice-Hall, 1991. D.V. Bhaskar Rao and K.S. Arun, Model based processing of signals: A state space approach, Proc. IEEE, vol. 80, pp. 283-309, Feb. 1992. A. Nerode, Linear automaton transformations, Proc. Amer. Math. Soc., vol. 9, pp. 541-544, 1958. B. L. Ho and R. E. Kalman, Effective construction of linear, state-variable models from input/output functions, Regelungstechnik, vol. 14, pp. 545-548, 1966. H. P. Zeiger and A. J. McEwen, Approximate linear realizations of given dimension via Hos algorithm, IEEE Trans. Automat. Cont., vol. AC-19, p. 153, Apr. 1974. B. C. Moore, Singular value analysis of linear systems, in Proc. IEEE Conf Dec. Control, 1979, pp. 66-73. S. Y. Kung, A new identification and model reduction algorithm via singular value decomposition, in 12th Asilomar Conf on Circuits, Systems and Comp.(Asilomar, CA), Nov. 1978, pp. 705-714. T. Kailath, Linear Systems. Englewood Cliffs, NJ: Prentice Hall, 1980. W. J. Rugh, Linear System Theory. A Graduate Course. Englewood Cliffs, NJ: Prentice Hall, 1992. P. R. Halmos, Introduction to Hilbert Space. New York: Chelsea Pub. Co., 1951. G. Golub and C. F. Van Loan, Matrix Computations. Baltimore, M D Johns Hopkins Univ. Press, 1984. R. E. Kalman, P. L. Falb, and M. A. Arbib, Topics in Mathematical System Theory (Int. Series in Pure and Applied Mathematics). New York: McGraw-Hill, 1970. P.A. Fuhrmann, Linear Systems and Operators in Hilbert Space. New York: McGraw-Hill, 1981. A. J. van der Veen and P. M. Dewilde, Time-varying system theory for computational networks, in Algorithms and Parallel VLSI Architectures, II, P. Quinton and Y. Robert, Eds. New York: Elsevier, 1991, pp. 103-127. , Time-varying computational networks: Realization, orthogonal embedding and structural factorization,in Proc. SPIE, Advanced Signal Processing Algorithms. Architectures, and Implementations, III, F. T. Luk, Ed., vol. 1770, pp. 164-177, July 1992. A. Feintuch and R. Saeks, System Theory: A Hilbert Space Approach. New York: Academic Press, 1982. E. W. Kamen, P. P. Khargonekar, and K. R. Poolla, A transferfunction approach to linear time-varying discrete-time systems, SIAM J. Contr. Optimization, vol. 23, no. 4, pp. 550-565, 1985. E. W. Kamen, The poles and zeros of a linear time-varying system, Lin. Alg. Applications, vol. 98, pp. 263-289, 1988. B. D. 0. Anderson and J. B. Moore, Detectability and stabilizability of time-varying discrete-time linear systems, SIAM J. Contr. Optimization, vol. 19, no. 1, pp. 20-32, 1981.

VAN DER VEEN et al.: SUBSPACE-BASED SIGNAL ANALYSIS

I305

Authorized licensed use limited to: IEEE Editors in Chief. Downloaded on August 17, 2009 at 19:20 from IEEE Xplore. Restrictions apply.

[24] I. Gohberg, M. A. Kaashoek, and L. Lerer, Minimality and realization of discrete time-varying systems, in Time Variant Systems andlnterpolation, I. Gohberg, Ed., vol. OT 56. Basel, Switzerland: Birkhguser Verlag, 1992, pp. 261-296. [25] B. De Moor, Mathematical concepts and techniques for modeling of static and dynamic systems, Ph.D. dissertation, Kath. Univ. Leuven, Belgium, 1988. [26] M. Verhaegen and E. F. Deprettere, Subspace model identification, in Algorithms and Parallel V U I Architectures, vol. B, E. F. Deprettere and A. J. van der Veen, Eds. New York: Elsevier, 1991, pp. 13-32. [27] M. Verhaegen and P. M. Dewilde, Subspace model identification. Part 1: The output error state space model identification class of algorithms, Int. J. Contr., vol. 56, no. 5, pp. 1187-1210, 1992. [28] -, Subspace model identification. Part 2: Analysis of the elementary output-error state-space model identification algorithm, Int. J. Contr., vol. 56, no. 5, pp. 1211-1241, 1992. [29] B. De Moor, M. Moonen, L. Vandenberghe, and J. Vandewalle, A geometrical approach for the identification of state space models with singular value decomposition, in Proc. IEEE ICASSP (New York, NY), vol. 4, 1988, pp. 2244-2247. [30] M. Moonen, B. De Moor, L. Vandenberghe, and J. Vandewalle, On- and off-line identification of linear state-space models, Int. J. Contr., vol. 49, no. 1, pp. 219-232, 1989. [31] M. Moonen and J. Vandewalle, QSVD approach to on- and off-line state space identification, Int. J. Contr., vol. 50, no. 1, pp. 1133-1 146, 1990. [32] P. Van Overschee, B. De Moor, and J. Suykens, Subspace algorithms for system identification and stochastic realization, in Recent Advances in Mathematical Theory of Systems, Control, Networks and Signal Processing (Proc. Int. Symp. MTNS-91), vol. I, H. Kimura and S. Kodama, Eds. MITA Press, Japan, 1992, pp. 589-594. [33] P. Van Overschee and B. De Moor, Two subspace algorithms for the identification of combined deterministic and deterministic-stochastic systems, in Proc. IEEE Conf. on Decision Control (Tucson, AZ), Dec. 1992, pp. 511-516. (Also to appear in Automatica (Special Issue on Statistical Signal Processing and Control). . [34] R. Roy, A. Paulraj, and T. Kailath, ESPRIT-A subspace rotation approach to estimation of parameters of cisoids in noise, IEEE Trans. Acoust., Speech, Signal Processing, vol. ASSP-34, no. 5, pp. 134C1342, 1986. [35] R. Roy, ESPRIT, Ph.D. dissertation, Stanford Univ., Stanford, CA, 1987. [36] R. Roy and T. Kailath, ESPRIT- Estimation of signal parameters via rotational invariance techniques, IEEE Trans. Acoust., Speech, Signal Processing, vol. 37, pp. 984-995, July 1989. [37] Y. Bresler and A. Macovski, On the number of signals resolvable by a uniform linear array, IEEE Trans. Acoust., Speech, Signal Processing, vol. ASSP-34, pp. 1361-1375, Dec. 1986. [38] M. Wax and I. Ziskind, On unique localization of multiple sources by passive sensor arrays, IEEE Trans. Acoust.. Speech, Signal Processing, vol. 37, pp. 996lOoQ. July 1989. [39] R. Prony, Essai experimental et analytique sur les lois de la dilabilite des fluides elastiaues ..., J. de IEcole Polvtechniaue. . vol. 1, no. 2, pp. 2 k 7 6 , i795. [40] E. F. Deprettere, Ed., SVD and Signal Processing: Algorithms, Applications and Architectures. Amsterdam, The Netherlands: North-Holland, 1988. [41] R. J. Vaccaro, Ed., SVD and Signal Processing, II: Algorithms, Analysis and Applications. New York: Elsevier, 1991. [42] G. W. Stewart, Perturbation theory for the singular value decomposition, in SVD and Signal Processing, 11: Algorithms, Analysis and Applications, R. J. Vaccaro, Ed. New York: Elsevier, 1991, pp. 99-109. [43] F. Li and R. J. Vaccaro, Performance degradation of DOA estimators due to unknown noise fields, IEEE Trans. Signal Processing, vol. 40, pp. 686-690, Mar. 1992. [44] M. S. Bartlett, Tests of significance in factor analysis, Erifish J. Psych. (Statist. Sect.), vol. 3, pp. 77-85, 1950. [45] -, A note on the multiplying factors for various x2 approximations, J. Roy. Statist. Soc., Ser. E , vol. 16, pp. 296298, 1954. 1461 T. W. Anderson, Asymptotic theory for principal component analysis, Ann. Math. Statist., vol. 34, pp. 122-148, 1963.
1306

[47] H. Akaike, Information theory and an extension of the Maximum Likelihood principle, in Proc. 2nd Int. Symp. on Information Theory, 1973, pp. 267-281. [48] J. Rissanen, Modeling by shortest data description, Automatica, vol. 14, pp. 465471, 1978. [49] -, A universal prior for the integers and estimation by Minimum Description Length, Annals Statist., vol. 11, pp. 417-431, 1983. [50] L. C. Zhao, P. R. Krishnaiah, and 2. D. Bai, Onthe detection of the number of signals when the noise covariance matrix is arbitrary, J. Multivuriate Anal., vol. 20, no. 1, pp. 1-25, 1986. [51] R. 0. Schmidt, A signal subspace approach to multiple emitter location and spectral estimation, Ph.D. dissertation, Stanford University, Stanford, CA, 1981. [52] M. Wax and T. Kailath, Detection of signals by information theoretic criteria, IEEE Trans. Acoust., Speech, Signal Processing , vol. ASSP-33, pp. 387-392, Apr. 1985. [53] M. Viberg, B. Ottersten, and T. Kailath, Detection and estimation in sensor arrays using weighted subspace fitting, IEEE Trans. Signal Processing, vol. 39, pp. 2436-2449, Nov. 1991. 1541 M. Wax and 1. Ziskind, Detection of the number of coherent signals by the MDL principle, IEEE Trans. Acoust., Speech, Signal Processing, vol. 37, pp. 1190-1 196, Aug. 1989. [55] M. Wax, Detection and localization of multiple sources via the stochastic signals model, IEEE Trans. Signal Processing, vol. 39, pp. 2450-2456, Nov. 1991. [56] S. Van Huffel and J. Vandewalle, The Total Least Squares problem: Computational Aspects and Analysis. Philadelphia, PA: SIAM, 1991. [57] B. C. Moore, Principal component analysis in linear systems: Controllability, observability, and model reduction, IEEE Trans. Automat. Contr., vol. AC-26, pp. 17-32, Feb. 1981. [58] L. Pemebo and L. M. Silverman, Balanced systems and model reduction, in Proc. IEEE Conf. Dec. Control, 1979, pp. 865-867. [59] S. Y. Kung and D. W. Lin, Recent progress in linear system model-reduction via Hankel matrix approximation, in Proc. ECCTD Circuit Theory and Design, (The Hague), 1981, pp. 222-233. [60] S. Y. Kung, K. S. A m , and D. V. Bhaskar Rao, Statespace and singular-value decomposition-based approximation methods for the harmonic retrieval problem, J. Opt. Soc. Amer., vol. 73, pp. 1799-1811, Dec. 1983. [61] S. Y. Kung, C. K. Lo, and R. Foka, A Toeplitz approximation approach to coherent source direction finding, in Proc. IEEE ICASSP, 1986, pp. 193-196. [62] S. Mayrargue, ESPRIT and TAM (Toeplitz approximation method) are theoretically equivalent, in Proc. IEEE ICASSP, vol. 4 (New York), 1988, pp. 24562459. [63] S. Mayrargue and J. P. Jouveau, A new application of SVD to harmonic retrieval, in SVD and Signal Processing, E. F. Deprettere, Ed. Amsterdam, The Netherlands: North-Holland, 1988, pp. 467472. [64] D. V. Bhaskar Rao and K. V. S. Hari, Performance analysis of ESPRIT and TAM in determining the direction of arrival of plane waves in noise, IEEE Trans. Acoust., Speech, Signal Processing, vol. 37, pp. 19%1995, Dec. 1989. [65] D. V. Bhaskar Rao, Relationship between matrix pencil and state space based harmonic retrieval methods, IEEE Trans. Acoust., Speech, Signal Processing, vol. 38, pp. 177-179, Jan. 1990. [66] H. Ouibrahim, D. D. Weiner, and T. K. Sarkar, A general approach to direction finding, in Proc. IEEE MILCON, 1986, pp. 41.4.1-41.4.5. [67] Y. Hua and T. K. Sarkar, Matrix pencil method and its performance, in Proc. IEEE ICASSP, 1988, pp. 2476-2479. [68] -, Matrix pencil method for estimating parameters of exponentially damped/undamped sinusoids in noise, IEEE Trans. Acoust., Speech, Signal Processing, vol. 38, pp. 814-824, May 1990. [69] M. Moonen, P. Van Dooren, and J. Vandewalle, An SVD updating algorithm for subspace tracking, SIAM J. Matrix Anal. Appl., vol. 13, no. 4, 1992. [70] M. Moonen, F. Van Poucke, and E. Deprettere, Parallel and adaptive high resolution direction finding, in Proc. SPIE: Advanced Signal Processing Algorithms, Architectures and Implementations III, F. Luk, Ed., 1992, pp. 219-230. [71] M. D. Zoltowski and D. Stavrinides, Sensor array signal
PROCEEDINGS OF THE IEEE, VOL. 81, NO. 9. SEITEMBER 1993

Authorized licensed use limited to: IEEE Editors in Chief. Downloaded on August 17, 2009 at 19:20 from IEEE Xplore. Restrictions apply.

processing via a Procrustus rotations based eigenandysis of the ESPIUT data pencil, IEEE Trans. Acoust., Speech, Signal Processing, vol. 37, pp. 832-861, June 1989. [72] Y. Hua and T. K. Sarkar, On SVD for estimating generalized eigenvalues of singular matrix pencil in noise, IEEE Trans. Signal Processing, vol. 39, pp. 892-900, Apr. 1991. [73] Y. T. Chan and R. P. Langford, Spectral estimation via the high-order Yule-Walker equations, IEEE Trans. Acoust., Speech, Signal Processing, vol. ASSP-30, pp. 689-698, Oct. 1982. 741 P. Stoica, R. L. Moses, T. Sijderstrom, and J. Li, Optimal highorder Yule-Walker estimation of sinusoidal frequencies, IEEE Trans. Signal Processing, vol. 39, pp. 1360-1368, June 1991. 751 V. M. Adamjan, D. Z. Arov, and M. G. Krein, Analytic properties of Schmidt pairs for a Hankel operator and the generalized Schur-Takagi problem, Mat. USSR Sbornik, vol. 15, no. 1, pp. 31-73, 1971. 761 M. Bouvet and H. Clergeot, Eigen- and singular value decomposition techniques for the solution of harmonic retrieval problems, in SVD and Signal Processing, E. F. Deprettere, Ed. Amsterdam, The Netherlands: North-Holland, 1988. [77] V. Pisarenko, The retrieval of harmonics from a covariance function, Geophys. J . Roy. Astron. Soc., vol. 33, pp. 347-366, 1973. [78] R. Kumaresan and D. W. Tufts, Estimating the parameters of exponentially damped sinusoids and pole-zero modeling in noise, IEEE Trans. Acoust., Speech, Signal Processing, vol. ASSP-30, pp. 833-840, Dec. 1982. [79] D. W. Tufts and R. Kumaresan, Estimation of frequencies of multiple sinusoids: Making Linear Prediction perform like Maximum Likelihood, Proc. IEEE, vol. 70, pp. 975-989, Sept. 1982. [80] R. Kumaresan, On the zeros of the Linear Prediction-errorfilter for deterministic signals, IEEE Trans. Acoust., Speech, Signal Processing, vol. ASSP-31, pp. 217-220, Feb. 1983. [81] B. Porat and B. Friedlander, Onthe accuracy of the KumarseanTufts method for estimating complex damped exponentials, IEEE Trans. Acoust., Speech, Signal Processing, vol. ASSP-35, pp. 231-235, Feb. 1987. [82] F. Li, R. J. Vaccaro, and D. W. Tufts, Min-Norm Linear Prediction for arbitrary sensor arrays, in Proc. IEEE ICASSP (Glasgow), May 1989, pp. 2613-2616. [83] D. W. Tufts, R. J. Vaccaro, and A. C. Kot, Analysis of estimation of signal parameters by Linear Prediction at high SNR using matrix approximations, in Proc. IEEE ICASSP (Glasgow), May 1989, pp. 2194-2197. [84] E. M. Dowling and R. D. DeGroat, The equivalence of the Total Least Squares and Minimum Norm methods, IEEE Trans. Signal Processing, vol. 39, pp. 1891-1892, Aug. 1991. [85] G. Cybenko, Locations of zeros of predictor polynomials, IEEE Trans. Acoust., Speech, Signal Processing, vol. ASSP-27, no. 1, pp. 235-237, 1982. [86] A. Bultheel, Recursive rational approximation, Ph.D. dissertation, Catholic University Louvain, Louvain, Belgium, 1979. [87] K. Glover, All optimal Hankel-norm approximations of linear multivariable systems, Int. J. Contr., vol. 39, no. 6, pp. 1 1 15-1 193, 1984. [88] Y. V. Genin and S. Y. Kung, A two-variable approach to the model reduction problem with Hankel norm criterion, IEEE Trans. Circuits Syst., vol. CAS-28, pp. 912-924, Sept. 1981. [89] Ph. Delsarte, Y. Genin, and Y. Kamp, On the role of the Nevanlinna-Pick problem in circuit theory and design, Circuit Theory Appl., vol. 9, pp. 177-187, 1981. [90] M. H. Gutknecht, J. 0. Smith, and L. N. Trefethen, The CarathCodory-FejCr method for recursive digital filter design, IEEE Trans. Acoust., Speech, Signal Processing, vol. ASSP-3 1, pp. 1417-1426, Dec. 1983. [91] P. M. Dewilde and E. F. Deprettere, Singular value decomposition: an introduction, in SVD and Signal Processing, E. F. Deprettere, Ed. Amsterdam, The Netherlands: North-Holland, 1988, pp. 3 4 1 . [92] L. M. Silverman and M. Bettayeb, Optimal approximation of linear systems, in Proc. 1980 Joint Autom. Control Conf.. 1980. [93] H. Ozbay, Computing the singular values and vectors of a Hankel operator, in SVD and Signal Processing, 11: Algorithms, Analysis and Applications, R. J. Vaccaro, Ed. New York: Elsevier, 1991, pp. 455469. [94] S. Y. Kung and D. W. Lin, A state-space formulation for
VAN DER VEEN et al.: SUBSPACE-BASED SIGNAL ANALYSIS

optimal Hankel-norm approximations, IEEE Trans. Automat. Contr., vol. AC-26, pp. 942-946, Aug. 1981. [95] J. A. Ball, I. Gohberg, and L. Rodman, Interpolation o f R a tional Matrix Functions, vol. OT 45 of Operator Theory: Advances and Applications. Basel, Switzerland: Birkhauser Verlag, 1990. [96] P. M. Dewilde and A. J. van der Veen, On the Hankel-norm approximation of upper-triangular operators and matrices, Integral Equations and Operator Theory, vol. 17, no. 1, pp. 1-45, 1993. [97] M. Viberg and B. Ottersten, Sensor array processing based on subspace fitting, IEEE Trans. Signal Processing, vol. 39, pp. 1110-1121, May 1991. [98] P. Stoica and A. Nehorai, MUSIC, Maximum Likelihood and CramCr-Rao bound, IEEE Trans. Acoust., Speech, Signal Processing, vol. 37, pp. 720-741, May 1989. [99] P. Stoica and K. C. Shaman, Maximum Likelihood methods for direction-of-arrival estimation, IEEE Trans. Acoust., Speech, Signal Processing, vol. 38, pp. 1132-1 143, July 1990. [lo01 D. W. Tufts and R. Kumaresan, Improved spectral resolution 11, in Proc. IEEE ICASSP (Denver, CO), Apr. 1980, pp. 592-597. [ 1011 R. Kumaresan, L. L. Scharf, and A. K. Shaw, An algorithm for pole-zero modeling and spectral analysis, IEEE Trans. Acoust., Speech, Signal Processing, vol. ASSP-34, pp. 637-640, June 1986. [ 1021 J. F. Bohme, Estimation of spectral parameters of correlated signals in wavefields, Signal Processing, vol. 11, pp. 329-337, Dec. 1986. [ 1031 M. Wax, Detection and estimation of superimposed signals, Ph.D. dissertation, Stanford University, Stanford, CA, 1985. [lo41 Y. Bresler and A. Macovski, Exact Maximum Likelihood parameter estimation of superimposed exponential signals in noise, IEEE Trans. Acoust., Speech, Signal Processing, vol. ASSP-34, pp. 1081-1089, Oct. 1986. [lo51 J. A. Cadzow, A high resolution direction-of-arrivalalgorithm for narrow-band coherent and incoherent sources, IEEE Trans. Acoust., Speech, Signal Processing, vol. 36, pp. 965-979, July 1988. [ 1061 A. Swindlehurst,B. Ottersten, R. Roy, and T. Kailath, Multiple invariance ESPRIT, IEEE Trans. Signal Processing, vol. 40, pp. 867-881, Apr. 1992. [lo71 R. Roy, B. Ottersten, A. L. Swindlehurst, and T. Kailath, Multiple invariance ESPRIT, in Proc. 22-nd Asilomar Conf. Sign., Syst., Computing, 1988, pp. 583-587. [ 1081 A. Swindlehurst and T. Kailath, Azimuth/elevation direction finding using regular array geometries, IEEE Trans. Aerosp. Electron. Syst., vol. 29, pp. 145-156, Jan. 1993. [lo91 G. Bienvenue and L. Kopp, Optimality of high resolution array processing using the eigensystem approach, IEEE Trans. Acoust., Speech, Signal Processing, vol. ASSP-31, no. 5. pp. 1235-1 247, 1983. [llO] A. J. Barabell, Improving the resolution performance of eigenstructure-based direction finding algorithms, in Proc. IEEE ICASSP, 1983, pp. 336-339. [ 1111 M. Kaveh and A. J. Barabell, The statistical performance of the MUSIC and minimum-norm algorithms in resolving plane waves in noise, IEEE Trans. Acoust., Speech, Signal Processing, vol. ASSP-34, pp. 331-341, Apr. 1986. (Corrections in vol. ASSP-34, no. 6, 1986.) [112] D. V. Bhaskar Rao and K. V. S. Hari, Performance analysis of root-MUSIC, IEEE Trans. Acoust., Speech, Signal Processing, vol. 37, pp. 1939-1949, Dec. 1989. [113] B. Ottersten, M. Viberg, and T. Kailath, Analysis of subspace fitting and ML techniques for parameter estimation from sensor array data, IEEE Trans. Signal Processing, vol. 40, pp. 590-600, Mar. 1992. [ 1141 P. Stoica and A. Nehorai, Performance study of conditional and unconditional direction-of-arrival estimation, IEEE Trans. Acoust., Speech, Signal Processing, vol. 38, pp. 1783-1795, Oct. 1990. [115] B. Ottersten, M. Viberg, P. Stoica, and A. Nehorai, Exact and large sample ML techniques for parameter estimation and detection in array processing, in Radar Array Processing, S. Haykin, Ed. New York: Springer-Verlag, 1991, ch. 4. [116] A. Swindlehurst, R. Roy, B. Ottersten, and T. Kailath, A subspace fitting approach for identification of linear state space models, submitted to IEEE Trans. Automat. Contr., 1993.
1307

Authorized licensed use limited to: IEEE Editors in Chief. Downloaded on August 17, 2009 at 19:20 from IEEE Xplore. Restrictions apply.

[117] P. Stoica and A. Nehorai, MUSIC, Maximum Likelihood, and Cram&-Rao bound: Further results and comparisons, IEEE Trans. Acoust., Speech, Signal Processing, vol. 38, pp. 2140-2150, Dec. 1990. [ 1181 B. Ottersten, M. Mberg, and T. Kailath, Performance analysis of the Total Least Squares ESPRIT algorithm, IEEE Trans. Signal Processing, vol. 39, pp. 1122-1135, May 1991. [119] H. Krim, P. Forster, and J. G. Proakis, Operator approach to performance analysis of root-MUSIC and root-Min-Norm, IEEE Trans. Signal Processing, vol. 40,pp. 1687-1696, July 1992. 201 P. Stoica and T. Soderstrom, Statistical analysis of MUSIC and subspace rotation estimates of sinusoidal frequencies, IEEE Trans. Signal Processing, vol. 39, pp. 1836-1847, Aug. 1991. 211 F. Li and R. J. Vaccaro, Performance analysis of state-space realization (TAM) and ESPRIT algorithms for DOA estimation, IEEE Trans. Antennas Propagat., vol. 39, pp. 418-423, Mar. 1991. 221 W. J. Bangs, Array processing with generalized beamformers. Ph.D. dis&tation,-Yale University, fiew Haven, CT, 1971. [ 1231 K. M. Wong, R. S. Walker, and G. Niezgoda, Effects of random sensor motion on bearing estimation by the MUSIC algorithm, Proc. Inst. Elec. Eng., vol. 135, pt. F, pp. 233-250, June 1988. [124] B. Friedlander, A sensitivity analysis of the MUSIC algorithm, IEEE Trans. Acoust., Speech, Signal Processing, vol. 38, pp. 1740-1751, Oct. 1990. [125] A. L. Swindlehurst and T. Kailath, A performance analysis of subspace-based methods in the precence of model errors -Part 1: The MUSIC algorithm,lEEE Trans. Signal Processing, vol. 40, pp. 1758-1774, July 1992. [126] F. Li, R. Vaccaro, and D. Tufts, Unified performance analysis of subspace-based estimation algorithms, in Proc. IEEE ICASSP (Albuquerque, NM), 1990, vol. 5, pp. 2575-2578. [I271 A. Swindlehurst and T. Kailath, On the sensitivity of the ESPRIT algorithm to non-identical subarrays, Sadhand, Academy Proc. in Eng. Sciences, vol. 15, pp. 197-212, Nov. 1990. [128] B. Friedlander, Sensitivity of the Maximum Likelihood direction finding algorithm, IEEE Trans. Aerosp. Electron. Syst., vol. 26, pp. 953-968, NOV. 1990. [129] F. Li and R. Vaccaro, Statistical comparison of subspace based DOA estimation algorithms in the presence of sensor errors, in Proc. 5th Acoust., Speech, Signal Processing Spectral Estimation Workshop (Rochester, NY), Oct. 1990, pp. 327-331. [ 1301 A. Swindlehurst and T. Kailath, An analysis of subspace fitting algorithms in the presence of sensor errors, in Proc. IEEE ICASSP (Albuquerque, NM), 1990, vol. 5, pp. 2647-2650. [ 13I] A. Swindlehurst, Robust algorithms for direction-finding in the presence of model errors, in Proc. 5th Acoust., Speech, Signal Processing Workshop on Spectral Estimation and Modeling (Rochester, NY), Oct. 1990, pp. 362-366 . [132] A. Swindlehurst and T. Kailath, A performance analysis of subspace-based methods in the presence of model errors-Part 2: Multidimensional algorithms, IEEE Trans. Signal Processing, vol. 41, pp. 2882-2890, Sept. 1993. [133] Y. Hua and T. K. Sarkar, Perturbation analysis of TK method for harmonic retrieval Droblems. IEEE Trans. Acoust.. Soeech. Signal Processing, vol: 36, pp. 228-240, Feb. 1988. 341 A. J. van der Veen and E. F. Deprettere, Parallel VLSI matrix pencil algorithm for high resolution direction finding, IEEE Trans. Acoust., Speech, Signal Processing, vol. 39, pp. 383-394, Feb. 1991. 351 T. F. Chan, Rank revealing QR factorizations, Lin. Alg. Appl., vol. 88/89, pp. 67-82, 1987. 361 C. H. Bischof and G. M. Schroff, On updating signal subspaces, IEEE Trans. Signal Processing, vol. 40, pp. 96-105, Jan. 1992.

[137] G. W. Stewart, An updating algorithm for subspace tracking, IEEE Trans. Signal Processing, vol. 40, pp. 1535-1541, June 1992.

Alle-Jan van der Veen (Student Member, IEEE) was bom in The Netherlands in 1966. He graduated (cum laude) from the Department of Electrical Engineering, Delft University of Technology, in 1988. He is currently with the Network Theory Section at the Delft University, where he has recently received the Ph.D. degree (cum laude). His research interests are in the areas of system theory, in particular system identification, model reduction, and time-varying system theory, and in parallel algorithms for linear algebra. He has organized two workshops in the area of signal processing, and is the co-editor of the book Algorithms and Parallel VLSI Architectures.

Ed F. Deprettere (Senior Member, IEEE) was bom in Roeselare, Belgium, on August IO, 1944. He received the M.S. degree from Ghent State University, Ghent, Belgium, in 1968, and the Ph.D. degree from Delft University of Technology (DUT), Delft, The Netherlands, in 1981. In 1970, he became a Research Assistent and Lecturer at the DUT, where he is now Associate Professor in the Department of Electncal Engineering, Network Theory Section, Signal Processing Group. His current research interests are in modem signal processing: algorithms, VLSI architectures, and applications, and in methodologies for the mapping of parallel signal processing algorithms, network graphs, and numerical computations onto silicon. He is the editor of the books SVD and Signal Processing Algorithms, Architectures and Applications and Algorithms and Parallel VLSI Architectures. He is on the editorial board of the IEEE Transacrions on Signal Processing, the Journal of VLSI Signal Processing, and Integration, the VLSI Journal. He coauthored a paper that received a 1989 IEEE SP award.

A. Lee Swindlehurst was bom on March IO, 1960 in Boulder City, NV. He received the B.S.

and M.S. degrees in electrical engineering from Brigham Young University, Provo, UT, in 1985 and 1986, respectively, and the Ph.D. degree also in electrical engineering from Stanford University, Stanford, CA, in 1991. He is currently an Assistant Professor in the Electrical and Computer Engineering Department at Bngham Young University. His research interests are in the general areas of signal processing, estimation,and control theory. In particular, his published research has focussed on problems in sensor array signal processing, state-space system identification, and bispectral estimation.

I308

PROCEEDINGS OF THE IEEE, VOL. 81, NO. 9. SEPTEMBER 1993

Authorized licensed use limited to: IEEE Editors in Chief. Downloaded on August 17, 2009 at 19:20 from IEEE Xplore. Restrictions apply.