AGC

DSP Signal Spaces

 The purpose of this part of the course is to introduce
the basic concepts behind generalised Fourier
Analysis
 The approach is taken via vector spaces and least
squares approximation
 Modern Signal Processing is based in a substantial
way on the theory of vector spaces. In this course we
shall be concerned with the discrete–time case only

Professor A G Constantinides© 1
AGC
DSP Signal Spaces
 In order to compare two signals we need to define a
measure of “distance” known as a metric.
 A metric is a function d ( x, y ) that produces a
scalar value from two signals such that

 1)
d ( x, y )  d ( y , x )
 2) d ( y, x)  0
 3) d ( x, x )  0
 4) d ( x, z )  d ( x, y )  d ( y , z )
Professor A G Constantinides© 2
AGC
DSP Signal Spaces
 There are many metrics that can be used.
 Examples:
 1) If we have a set of finite numbers representing
signals then a metric may be
n
d1 ( x, y )   xi  yi
i 1

 This is known as the l1 metric or the Manhattan

distance.

Professor A G Constantinides© 3
AGC
DSP Signal Spaces
 2) Another metric for the same set is
n 1/ 2

d 2 ( x, y )   xi  yi 
2
i 1 
This is called the l2 metric or the Euclidean
distance
 3) Yet another form is the lp metric
n 1/ p

d p ( x, y )   xi  yi 
p
i 1 
Professor A G Constantinides© 4
AGC
DSP Signal Spaces
 4) As the integer p  the last form becomes
d  ( x, y )  max xi  yi
i 1, 2 ,..., n

This is called the l metric or distance.

 In channel coding we use the Hamming Distance as a
metric
n
d H ( x, y )   xi  yi 
i 1
where  is the modulo-2 addition of two binary
vectors

Professor A G Constantinides© 5
AGC
DSP Signal Spaces
 When the set of vectors which we use is defined
along with an appropriate metric then we say we
have a metric space.

 There are many metric spaces as seen from the

earlier discussion on metrics.

 (In the case of continuous time signals we say we

have function spaces)

Professor A G Constantinides© 6
AGC
DSP Vector Spaces
 We think of a vector x as an assembly of elements
xarranged
i i  1as
,2,3,4,...
 x1 
x 
x   2
.
x 
 n

The length may be in certain cases infinite

Professor A G Constantinides© 7
AGC
DSP Vector Spaces
 A linear vector space S over a set of scalars R is
formed when a collection of vectors is taken together
with an addition operation and a scalar multiplication,
and satisfies the following:
 1) S is a group under addition ie the following are
satisfied
 a) for any x and y in S then x  y is also in S

b) there is a zero identity element ie
 c) for every element there is another such that their sum is
zero x0x
 d) addition is associative ie
(x  y )  z  x  (y  z )
Professor A G Constantinides© 8
AGC
DSP Vector Spaces
 2) For any pair of scalars a, b and
R x, y  S
ax  S
a (bx)  (ab)x
(a  b)x  ax  bx
a (x  y )  ax  ay
 3) There is a unit element in the set of scalars R such
that
1x  x
(The set of scalars is usually taken as the set of real
numbers)

Professor A G Constantinides© 9
AGC
DSP Linear Combination
 Often we think of a signal as being composed of
other simpler (or more convenient to deal with)
signals. The simplest composition is a linear
combination of the form
m
x[n]   ci pi [n]
i 1

 Where { pi [ n]} i  1,2,3,4,... are the simpler

signals, and the coefficients are in the scalar set.

Professor A G Constantinides© 10
AGC
DSP Vector space …

Professor A G Constantinides© 11
AGC
DSP Vector space …

Professor A G Constantinides© 12
AGC
DSP Linear Independence
 If there is no nonzero set of coefficients
{ci } i  1,2,3,4,...
m
such that  ci pi [ n]  0
i 1

then we say that the set of vectors

{ pi [n]} i  1,2,3,4,...

is linearly dependent

Professor A G Constantinides© 13
AGC
DSP Linear Independence
 Examples:
 2  3 4
T
 1) p1
p 2   1 6  2
T

p 3  1 6 2
T

Observe that
4p1  5p 2  3p 3  0

ie the set is linearly dependent

Professor A G Constantinides© 14
AGC
DSP Linear Independence
 Examples
 2) p1 t
p2 1  t

the set is linearly independent

Professor A G Constantinides© 15
AGC
DSP Linear Independence

Professor A G Constantinides© 16
AGC
DSP The Span of Vectors
 The Span of a vector set is the set of all possible
vectors that can be reached (ie made through linear
combinations) from the given set.

 That is there exists a set {ci } i  1,2,3,4,...

such that
m
x[n]   ci pi [n]
i 1

Professor A G Constantinides© 17
AGC
DSP The Span of Vectors
 Example: The vectors below are in 3-D real vector
space.
p1  1 1 0
T

p 2  0 1 0
T

 Their linear combination forms

x  c1 c1  c2 0
T

which is essentially a vector in the plane of the given

two vectors.

Professor A G Constantinides© 18
AGC
DSP Basis and Dimension
 If { pi [ n]} i  1,2,3,4,... is a selection of
linearly independent vectors from a vector space
such that they span the entire space then we say the
selected vectors form a (complete) basis.
 The number of elements in the basis is the cardinality
of the basis
 The least number of independent vectors to span a
given vector space S is called the dimension of the
vector space, usually designated as
dim(S )
Professor A G Constantinides© 19
AGC
DSP IMPORTANT!

 Every vector space has a basis.

 Thus for many purposes whatever

operations we want to do in the vector
space can be done with the basis.

Professor A G Constantinides© 20
AGC
DSP
Basis

Professor A G Constantinides© 21
AGC
DSP Vector Spaces
 Let us start with the intuitive notion that we can
represent a signal as

x[n]   ci pi [n]
i 0
 This representation is called a projection of x[n],the
signal, into the linear vector space
{ pi [n]} i  1,2,3,4,...
 The vectors above are linearly independent and can
span any signal in the space

Professor A G Constantinides© 22
AGC
DSP Vector Spaces
 Examples are seen in Matrix Theory and typically in
Fourier Analysis.

 The length of a vector is known as the norm.

 We can select any convenient norm, but for
mathematical tractability and convenience we select
the second order norm.
 A real valued function x is the norm of x

Professor A G Constantinides© 23
AGC
DSP Norm
 A real valued function x is the norm of x when
it satisfies
 Positivity x 0

 Zero length x 0 if x0

 Scaling x   x


Triangle inequality xy  x  y

Professor A G Constantinides© 24
AGC
Induced norm/Cauchy
DSP Schwartz inequality

 Induced norm of the space follows from the inner

product as
2
x, x  x
 The l2 norm is represented as x 2
 The following condition (Cauchy-Schwartz) is
satisfied by an induced norm (eg l2 )
x, y  x 2 y 2
Professor A G Constantinides© 25
AGC
DSP Inner Product
 The inner product of two vectors in a scalar, and has
the following properties

*
 x, y  x, y

 if the vectors are real then x, y  y , x

  x, y   y , x

Professor A G Constantinides© 26
AGC
DSP Inner Product

x  y , z  x, z  y , z


x, x  0 x  0

 x, x  0 x  0

 In finite-dimensional real space

x, y  xT y   xi yi
Professor A G Constantinides© 27
AGC
DSP Direction of Vectors

From the two and three dimensional cases we define
the angle  between two vectors to be given from
x, y
cos 
x2y2
 If y  ax the vectors are colinear

 If x, y  0 the vectors are orthogonal x y

(the zero vector is orthogonal to all vectors)

Professor A G Constantinides© 28
AGC
DSP Orthonormal
 A set of vectors [p1 , p 2 , p 3 ,...p m ] is
orthonormal when

p i , p j   ij
 (Pythagoras)
If x  y then the indiced norms satisfy
2 2 2
xy  x  y

Professor A G Constantinides© 29
AGC
DSP Weighted inner product
 Very often we want a weighted inner product which
we define as
H
x, y W
 y Wx

where W is a Hermitian matrix

For the induced norm to be positive for yx we
must have for all non-zero vectors
H
x, x W  x Wx  0
This means that W must be positive definite

Professor A G Constantinides© 30
AGC
DSP Example
 Let
1  2
x1    x2   
1 1 
 Clearly  2
xT1 x 2  1 1   3  0
 while
1 
 2  2  2  2  2
 2 2  x 2  1 1 2 2  1   0
T
x1
    
Professor A G Constantinides© 31
AGC
DSP Example
 Note that the inner product in the previous example
cannot serve as a norm as for any
a 
x 
a 
we have T
x Wx  0
 This violates the conditions for a norm

Professor A G Constantinides© 32
AGC
DSP Complete spaces
 If every signal in a signal space is reachable (ie can
be spanned) by a set of vectors then the space is
complete and the reaching a complete set
 This means that there will be no left over part of a
given signal expressed as an appropriate linear
combination of basis vectors
 For example a Fourier series reaches all periodic
signals in the least square sense, and hence the set
of all complex exponentials is a complete set

Professor A G Constantinides© 33
AGC
DSP Hilbert spaces
 Complete linear vector spaces with induced l2
norms are known as Hilbert Spaces
 In signal processing we are mainly interested in finite
energy signals ie in Hilbert spaces
 If the norm above is orther than the second then we
have Banach Spaces. (Such norms are useful in
approximating signals and system functions as we
shall see later)

Professor A G Constantinides© 34
AGC
DSP Orthogonal subspaces
 Let S be a inner product signal (vector) space and V
and W be subspaces of S.
 Then V and W are orthogonal if for every v  V
and w  W we have v, w  0
 In above the set of all vectors orthogonal to a
subspace V is called the orthogonal complement
of the of the subspace denoted by V 

Professor A G Constantinides© 35
AGC
DSP Inner-sum spaces
 Let S be a inner product signal (vector) space and V
and W be subspaces of S with v  V and w  W
 Then V +W is the inner sum space that consists of
all vectors x  w  v

 Example: Let S be the set of all 3-tuples in GF(2)

and
v  1 0 1 S w  0 0 1 S
Then
x  w  v  1 0 0 S
Professor A G Constantinides© 36
AGC
DSP Example …
 Let A  span0 1 0
B  span1 0 0
 Then
A  0 0 0, 0 1 0
B  0 0 0, 1 0 0
 Note that since vectors in one of these are pairwise
orthogonal to the vectors in the other the two
subspaces are are orthogonal

Professor A G Constantinides© 37
AGC
DSP Example …
 The orthogonal complement of A is found by
observing the the second member of the set is
orthogonal to all vectors having as their first entry a
zero. Thus
A  0 0 0, 0 1 0, 0 0 1, 0 1 1



 Thus observe that B  A
 And the pair-wise sum
A  B  0 0 0, 0 1 0, 1 0 0, 1 1 0

Professor A G Constantinides© 38
AGC
DSP Disjoint spaces
 If two linear vector spaces of the same dimensionality
have only the zero vector in common they are called
disjoint.
 Two disjoint spaces are such that one is the algebraic
complement of the other
 Their sum is the entire vector space

Professor A G Constantinides© 39
AGC
DSP Disjoint spaces …
 Let S be a inner product signal (vector) space and V
and W be subspaces of S

 Then for every x  W  V there exist unique

vectors
v V w W
xwv
such that
if and only if the two sbspaces are disjoint.
(ie if they are not disjoint a vector may be generated
form a pair-wise sum of more than one pair)

Professor A G Constantinides© 40
AGC
DSP Projections
 From above a pictorial representation can be
produced as

V
v x

W
w
S

Professor A G Constantinides© 41
AGC
DSP Projections
 Let S be a inner product signal (vector) space and V
and W be subspaces of S
 We can think of v  V and w  W as being the
projections of x  w  v in the component sets.
P : S V
We introduce the projection operator
x  S we have


such that for any

Px  v
x
 That is the operation returns that component of
that lies in V

Professor A G Constantinides© 42
AGC
DSP Projections
 Thus if x is already in V the operation does not
change the value of x
 Thus
P(Px)  Px
 This gives us the definition
A linear tranformation is a projection if
P
P2  P
(Idempotent operator)

Professor A G Constantinides© 43