Linear Least Mean Squared Error (LLMSE, MMSE) Filter

LLMSE (MMSE or Wiener) filter

Assumption
1. h(t) is causal and stable.
2. S(t), N(t): uncorrelated
3. S(t), N(t): jointly w.s.s. (and each is zero-mean w.s.s.)
Cost function
causal. is Assume
J? minimizes that (t) ) ( is What : Q
] | ) ( ) ( [|
2
o
o
h
h t h
t y t s E J
=
+ =
) (t x
) (t s
) (t h
) ( ) ( ) (
0 0
t N t s t y + =
) (t N
0
2 * *
0
* *
0 0
*
0
*
0 0
( ) ( ) ( )
[| ( ) | 2Re[ ( ) ( ) ( ) ]
( ) ( ) ( ) ( ) ]
(0) 2Re[ ( ) ( ) ]
( ) ( ) ( )
Replace by
( ) (0)
ss sx
xx
o
y t h x t d
J E s t s t h x t d
h h x t x t d d
R R h d
h h R d d
h h h
J J

=
= + + +

= + +

+
= +

2
2
where
C D +
*
0 0
0
*
0 0
0
0
Re[ ( )[ ( ) ( ) ( ) ] ]
( ) ( ) ( )
In order to have a minimum at 0, we have
( ) | 2 0
( ) ( ) ( ),
sx xx
xx
xx sx
C h R R h d d
D h h R d d
J C
R h d R

=

=

= + +
=
=

= =

= +

0
This is a necessary condition for optimum.

{ }
2
2
2
2 * 2 2
0
-It is also sufficient.
C 0
D E X( ) h( )d 0
J( ) J(0)
- General (nonlinear) solution
Minimize J E[|Y-Z| ] ,
Y Y Y
J( ) J J E[|Y-Z Y| ]
[| | ] 2 Re( [( ) ]) [| | ]
where X Z N
E Y Z E Y Z Y E Y

=
=

= = +
+
= + = +
= + +

*
0
2 2
( ) 2Re( [( ) ]) 0
It is both necessary & sufficient since J [| | ] 0
J E Y Z Y
E Y

= =

N Z X + =
Y
h

* * *
*
* *
*
0
- Optimum linear filter
( 0
[( ) ] 0 ,
[( ) ] 0 : Wiener-Hopf Eq.
Let Z s(t )
X X(t- )
h X h( )X(t- )d
( ) ( ) ( )
Other propert
ss
xx sx
R
Y X h
Y X h
E[ Y Z) Y ]
E h X Z X h h
E h X Z X
R h d R

=
=
=
=
=
=
= +

= +

*
*
ies of E[(Y-Z) Y ] 0, Y
Y can be anything.
1) Let Y 1
E[(Y-Z)] 0
[ ] [ ] unbiased estimator. E Y E Z

=
=
=
=
error. Est. Input
Principle ity Orthogonal

|
*
2
2 2
2 2
2 2
2
*
*
( ) ( )
0
2) Let Y Y
[( ) ] 0
[ ] [ ]
J E[Z ] 2 [ ] [ ]
E[Z ] [ ] 0
E[Y ] [ ]
Note that Max J E[Z ]
3) Explicit expression of Y
E[(Y-Z) Y ]
f(x,z) (y-z) y dxdz
f(x)[ f(z|
z x
x z
fx x f z x
E Y Z Y
E Y E YZ
But E YZ E Y
E Y
E Z

=
=
=
= +
=

=
=
=

*
x z
0
[ | ]
x)(y-z)dz] y 0, y
y f(z|x)dz zf(z|x)dz
[ | ] in general, non-linear system.
E z x
dx
y E z x

=
=
=
=
=

0
( Realizable Causal Stable )
filter: Remove the assumption of causal h:
( )
( )
( )
( ) ( ) ( )
( ) ( ) ( ) ( ) ( ) ( )
( ) (
o j
sx
u
xx
xx ss nn xx ss nn
sx ss
Unrealizable
H e
x t s t n t
R R R
R R

= +

=
= +
= + = +
=

) ( ) ( )
( )
( )
( ) ( )
) Noiseless case : ( ) 0
( )
) ( ) ( )
( )
( ) 0
( )
) ( ) ( )
( )
sx ss
o j
ss
u
ss nn
nn
o j
u
nn ss
o j
ss
u
nn
nn ss
o j
u
H e
i
H e
ii
H e
iii
H e

=
=
+
=
=
>>

<<

) prediction pure : 0 0, (
estimation : 0
prediction : 0
nn
= >

>
*
ss
*
2
' go back to J(0).
J(0) [R (0) ( ) ( ) ]
using F.T.
1
[ ( ) ( ) ( )]
2
( ) ( ) ( )
1

2 ( )
( ) ( ) 1
If s, n are unco
2 ( ) ( )
o
sx
j o
ss sx
ss xx sx
xx
ss nn
ss nn
Let s
R h d
e H d
d
d

= +
=

=
=
+

rrelated.
xx
0
0 -
sx xx
Realizable filter R ( ) ( ) ( )
h( ) 0, 0. (causal.)
( ) R ( ) R ( ) ( )
( ) 0, 0.
Using the same tech. for the M.F. realizable derivat
sx
h d R
Assume
g h d
where g

= +
= <
= +
=

0
ion,
( ) 1
H ( )
( ) ( )
( ) 1 1

( ) 2 ( )
o j
ss
r
xx xx
j t j t
ss
xx xx
e
e e d dt

+
+

(
=
(


=
`
)

*
ss
0 0 0
0
*
of realizable filter (Assume h is optimal)
J R (0) 2Re[ ( ) ( ) ] ( ) ( ) ( )
sing Inverse F.T., ( causal.)
1
J [ ( ) 2Re[ ( ) ( ) ] ( )
2
s
sx xx
j
ss sx xx
MSE
R h d h h R d d
U
H e

= + +

= +

2
( ) ]
s nn
H d

*
2
2
*
( )
*
sx
1
( ) 1 ( ) ( ) ( )
2
( ) ( ) ( ) ( )
- J as a function of .
Define G( ) R ( ) (
j
ss nn
noise
distortion
j j
sx sx
H
H e H d
H e d H e d
h

=


= +
`

)
| |
|
=
|
\ .
= +

{ }
0
*
xx
0 0
* 2
0
ss
) By Wiener-Hopf. Eq.
h ( ) R ( ) ( )
E | ( ) ( ) |
J R (0) ( ) 0 increasing function of .
d
h d d
X h d
G

=
=
=

*
- 0 -
*
-
( )
0
( )
G( ) is a non-increasing function.
pf) G( ) ( ) ( ) ( )
1
( ) ( )
2
( ) 1 1
( )
( ) 2 ( )
sx
j
sx
j t j t
sx
xx xx
k t
R h d
R H e d d
H e e d

+
+

= +

= +
= +
=

*
*
0
( )
( )
2 2
0
( )
Define K( ) .
( )
1
Using I.F.T., k(t) ( )
2
1
( ) ( ) ( )
2 [ ( )]
|k(t )| |k( ) |
j
sx
xx
sx
xx
j t
j t
j
sx
xx
e
dt
K e d
e
G k t R e d d dt
dt t dt

+

=
=

=
=
= + +
= + =

decreasing ftn.

{ }
nn
( )
0
( )
( )
2
m 0
Pure prediction : 0, 0.
1 1
( ) ( )
( ) 2
Discrete time
min. E |s(k )-y(k)|
Wiener-Hopf . eqn.
( ) ( ) ( ), m 0.
ss
ss
j t j t
u ss
ss
xx sx
H e e d dt
R m n h n R m

+ +
+

=

=
> =
=

+
= +

) k ( h
) k ( s
) k ( N
) k ( x
) k ( N ) k ( s ) k ( y
0 0
+ =
{ } { }

=
=
= =
= =
=
=
=
(

=
=
=
+
+
Eqn. Hopf. - Wiener : ) ( ) ( ) (
)] ( l)x - E[x(n h(l) )] ( E[s(n)x
1. - p , ... 0,1, k , 0 )] ( E[e(n)x principle, ity orthogonal By
) ( ) ( ) ( J
l) - h(l)x(n y(n)
0. Let
p taps of # : e) (realizabl Wiener FIR
) (
) (
) (
1
) ( H
Re
) (
) (
) ( H
1 - p
0 l
) ( R
* *
*
2 2
1 - p
0 l
r
u
xx
k r l k R l h
k n k n
k n
n y n s E n e E Minimize
z
z
z
z
z
alizable
z
z
z
z
le Unrealizab
sx xx
l k
xx
sx
xx
xx
sx
4 4 4 3 4 4 4 2 1

sx x
H
sx
H
sx
sx
sx sx
r
sx
sx
h
R
xx xx
xx xx xx
xx xx xx
r R r
h r
l r l h
n s l n x l h n s E
l n x n e E l h n
r r h
p r
r
p h
h
r p r
p r r r
p r r r
sx
x
1
ss
ss
*
ss
0
* * *
1
x x
) 0 ( r
) 0 ( r
) ( ) ( ) 0 ( r
)] ( )] ( ) ( ) ( [[
)] ( ) ( [ ) ( )] ( E[e(n)s
] l)] - h(l)x(n - ) E[e(n)[s(n J
MSE
R h R
) 1 (
) 0 (
) 1 (
) 0 (
) 0 ( ... ) 1 (

) 2 ( ... ) 0 ( ) 1 (
) 1 ( ... ) 1 ( ) 0 (

= =
=

=
=
=
=
=
=

= =
(
(
(
(

=
(
(
(
(

(
(
(
(
(

4 4 3 4 4 2 1
43 42 1
M
M
43 42 1
M
M
4 4 4 4 4 4 3 4 4 4 4 4 2 1
M M
(

+
+
=
(

=
(

+
+
= =
+ = + =

= =


=

= =

2
2 2
2 2 2
2
2
k
ss sx
2
k
nn ss xx
2
nn
2
nn
1
2
1 1 1
ss
k
ss
1
) 1 (
1
) 1 ( h
) 0 ( h
1
) 1 ( h
) 0 ( h
1
1
) k ( r ) k ( r
) k ( r r ) k ( r
Wiener? FIR tap - 2
ed uncorrelat : n , s
) k ( ) k ( r
) z 1 )( z 1 (
1
z 1
1
z 1
1
) k ( r

.) Eg
] 1 [
) 1 (
1
1
h r 1
) l ( r ) l ( h ) 0 ( r MSE
FIR for MSE
) z (
) z (
) z (
1
) z ( H
IIR Realizable
)) z z ( 1 ( - 1
- 1

) z 1 )( z 1 ( ) 1 (
1
) z 1 )( z 1 (
1
) z 1 )( z 1 (
1
) z (
) z (
) z ( H
IIR ble Unrealiza
2 2 2 2
2 2 2
H
sx
sx ss
xx
ss
xx
r
1 2 2 2
2
1 2 2
2
2
1
2
1
2
xx
ss
u
+ +
+
=
=

=

+ + +

=
+

=
+




=

+
+

2 2 2
2 2
2
2 2 2
2 2 2 2 2 2 2
) 1 (
1
) 1 (
] 1 [ ) (1
+
+
=
+
+ + +
=