DSP

Chapter-6 : Wiener Filters and the LMS Algorithm

Marc Moonen
Dept. E.E./ESAT-STADIUS, KU Leuven
[email protected]
www.esat.kuleuven.be/stadius/
 Part-III : Optimal & Adaptive Filters

Chapter-6 Wieners Filters & the LMS Algorithm

• Introduction / General Set-Up
• Applications
• Optimal Filtering: Wiener Filters
• Adaptive Filtering: LMS Algorithm
Chapter-7
– Recursive Least Squares Algorithms
• Least Squares Estimation
• Recursive Least Squares (RLS)
• Square Root Algorithms
• Fast RLS Algorithms

DSP 2016 / Chapter-6: Wiener Filters & the LMS Algorithm 2 / 32

 Introduction / General Set-Up

1. ‘Classical’ Filter Design

3

Norbert Wiener (1894-1964)

I nt r oduct i on : Opt i m al and adapt i ve fi l t er s

See Part-II
1. ‘Classical’ filter design
lowpass/ bandpass/ notch filters/ ...

‘Optimal’
2. ‘Optimal’
2. Filter
filter designDesign filter input

• signals are viewed as realizations of

stochastic processes (H249-HB78)
• filter optimisation/ design in a filter filter parameters

statistical sense based on a priori

filter output
statistical infor mation
+
→ Wiener filters
error desired signal

DSP 2016 / Chapter-6: Wiener Filters & the LMS Algorithm 3 / 32

 Introduction / General Set-Up

I nt r oduct i on : Opt i m al and adapt i ve fi l t er s

P r ot ot y p e opt i m al fi l t er i ng set -up :

filter input
Design filter such that for a given
(i.e. ‘statistical info available’)
input signal, filter output signal is filter filter parameters
‘optimally close’ (to be defined)
to a given ‘desired output signal’. filter output
+

error desired signal

DSP 2016 / Chapter-6: Wiener Filters & the LMS Algorithm 4 / 32

 Introduction / General Set-Up
5

I nt r oduct i on : Opt i m al and adapt i ve fi l t er s

when a priori statistical information is not available :

‘Adaptive’
3.Adaptive
3. filter sFilters
• self-designing
• adaptation algorithm to monitor environment
• properties of adaptive filters :
convergence/ tracking
numerical stability/ accuracy/ robustness
computational complexity
hardware implementation

DSP 2016 / Chapter-6: Wiener Filters & the LMS Algorithm 5 / 32

 Introduction / General Set-Up

I nt r oduct i on : Opt i m al and adapt i ve fi l t er s

P r ot ot y p e adapt i ve fi l t er i ng set -up :

filter input
Basic operation involves 2 processes :
1. filtering process adaptive filter parameters
2. adaptation process filter

adjusting filter parameters to filter output

(time-varying) environment +
adaptation is steered by error signal
error desired signal

• Depending on theapplication, either thefilter parameters, thefilter output

or the error signal is of interest

DSP 2016 / Chapter-6: Wiener Filters & the LMS Algorithm 6 / 32

 Applications

10

I nt r oduct i on : A ppl i cat i ons

sy st em i dent i fi cat i on/ m odel i ng

plant input

‘plant’ can be any system

adaptive plant
filter

error plant output

Optimal/adaptive filter provides mathematical model for input/output-

behavior of the plant

DSP 2016 / Chapter-6: Wiener Filters & the LMS Algorithm 7 / 32

 Applications

echo path

near-end signal

near-end signal
+ echo

DSP 2016 / Chapter-6: Wiener Filters & the LMS Algorithm 8 / 32

 Applications

17

I nt r oduct i on : A ppl i cat i ons

ex am pl e : interference cancellation

reference sensor
noise

adaptive noise source

filter

+ signal source
signal signal
+ residual noise primary sensor
+ noise

DSP 2016 / Chapter-6: Wiener Filters & the LMS Algorithm 9 / 32

 Applications

18

I nt r oduct i on : A ppl i cat i ons

ex am pl e : acoustic noise cancellation

noise source
noise

adaptive
filter reference sensors

+
signal source
signal signal primary sensor
+ residual noise + noise

DSP 2016 / Chapter-6: Wiener Filters & the LMS Algorithm 10 / 32

 Applications

11

I nt r oduct i on : A ppl i cat i ons

ex am pl e : channel identification

training sequence training sequence

0,1,1,0,1,0,0,... 0,1,1,0,1,0,0,...

radio channel
adaptive
filter base station antenna

+ mobile receiver

DSP 2016 / Chapter-6: Wiener Filters & the LMS Algorithm 11 / 32

 Applications

20
Inverse modeling :
I nt r oduct i on : A ppl i cat i ons

ex am pl e : channel equalization (training mode)

training sequence
0,1,1,0,1,0,0,...

radio channel

base station antenna

mobile receiver

adaptive
filter

+ training sequence
0,1,1,0,1,0,0,...

DSP 2016 / Chapter-6: Wiener Filters & the LMS Algorithm 12 / 32

 Optimal Filtering : Wiener Filters

26

Opt i m al fi l t er i ng/ W i ener fi l t er s

P r ot ot y p e opt i m al fi l t er r ev i si t ed
Have to decide on 2 things..
filter structure ? filter input
1
→ FIR filters
(= pragmatic choice)
filter filter parameters

2 cost function ?
filter output
→ quadratic cost function
+
(= pragmatic choice)
error desired signal

DSP 2016 / Chapter-6: Wiener Filters & the LMS Algorithm 13 / 32

 Optimal Filtering : Wiener Filters

PS: Shorthand notation uk=u[k], yk=y[k], dk=d[k], ek=e[k],

Filter coefficients (‘weights’) are wl (replacing bl of

For adaptive filters wl also have a time index wl[k]

27

Opt i m al fi l t er i ng/ W i ener fi l t er s

1
FIR filters (= tapped-delay line filter/ ‘transversal’ filter)
filter input u[k]
N−1
L
yk = å wl .uk-l =l =w0 T .u k = uTk .w
yk = wl · uk− l u[k] u[k-1] u[k-2] u[k-3]

w0[k] w1[k] w2[k] w3[k]

l=0 yk = w T · u k

previous chapters)
e[k] filter output y[k]
where +

d[k]
wTT =é
w 0 w 1 …2 w ùN − 1
error desired signal
w w w ··· w
w =ë 0
b w

0 L û a+bw a

uT = uk uk− 1 uk− 2 · · · uk− N + 1

Tk é
u k = ë uk uk-1 … uk-L ùû

DSP 2016 / Chapter-6: Wiener Filters & the LMS Algorithm 14 / 32

 Optimal Filtering : Wiener Filters

I nt r oduct i on : Opt i m al and adapt i ve fi l t er s

N ot e : generalization to I I R (infinite impulse response)

is non-trivial filter input

• convergence problems
0

• stability problems a1[k] a2[k] a3[k]

b0[k] b1[k] b2[k] b3[k]

filter output
+

error desired signal

b w
a+bw a

N ot e : generalization to non-linear filters not treated here

DSP 2016 / Chapter-6: Wiener Filters & the LMS Algorithm 15 / 32

 Optimal Filtering : Wiener Filters

28

PS: iCan
Opt m al generalize FIR
fi l t er i ng/ W filterfi lto
i ener t er‘multi-channel
s FIR filter’
example: see page 11
Multi-channel FIR filters
filter input 1 filter input 2 filter input 3

w0[k] w1[k] w2[k] w3[k] w4[k] w5[k] w6[k] w7[k] w8[k] w9[k] w10[k] w11[k]

0 0 0

+ + +

error e[k] desired signal d[k]

b w
a+bw a

DSP 2016 / Chapter-6: Wiener Filters & the LMS Algorithm 16 / 32

 Optimal Filtering : Wiener Filters

29
PS: Special case of ‘multi-channel FIR filter’ is ‘linear combiner’
Opt i m al fi l t er i ng/ W i ener fi l t er s

‘special case’ : linear combiner

filter input 1 filter input 2 filter input 3

w0[k] w1[k] w2[k]

y = u
T
T
wk
ykk= w k· u +

where error e[k] desired signal d[k]

b w

é −L1 ù
a+bw a

uT= ê uu0k0k u1k uu1k3k · …

T
u = · · uNu
k k ú
kk ë û
FIR filter may then also be viewed as special case of ‘linear combiner’
where input signals are delayed versions of each other

DSP 2016 / Chapter-6: Wiener Filters & the LMS Algorithm 17 / 32

 Optimal Filtering : Wiener Filters

2
30

Opt i m al fi l t er i ng/ W i ener fi l t er s

Quadratic cost function :

minimum mean-square error (MMSE) criter ion

(w) = E {e } = E {( d ) } = E {(d - u w) }
2
-y
2 T
J 2
2 2 T 2
M SE (w ) = E{ kek } = E{ |dkk − ykk | } = E{ |dkk − w k u k | }
JMSE

E{ x} is ‘expected value’ (mean) of x

DSP 2016 / Chapter-6: Wiener Filters & the LMS Algorithm 18 / 32

 Optimal Filtering : Wiener Filters

35

Opt i m al fi l t er i ng/ W i ener fi l t er s

MMSE cost function can

Wiener filter solution: filter input u[k]

be expanded as…
filter filter parameters

filter output y[k]

J MSE (w) = E {ek2 } +

{( )}
error e[k] desired signal d[k]
2} 2
= E dk - u kTw 2
J M SE (w ) = E{ |ek | T

= E{ |dk − w · u k | }
= E{ d2 T T T
k } + w E{ u k u k } w − 2w E{ u k dk } .
X̄ uu X̄ du
X̄ uu = correlation matr ix X̄ du = cross-correlation vector

DSP 2016 / Chapter-6: Wiener Filters & the LMS Algorithm 19 / 32

 Optimal Filtering : Wiener Filters

34

Correlation
Everything you matrix hasabout
need to know a special structure…
stochastic processes (I V)

for a stationary discrete-time stochastic process { uk } :

aut ocor r el at i on coeffi ci ent s : x̄ uu(δ) = E{ uk · uk− δ}

cor r el at i on m at r i x :
with u T
k = uk uk− 1 uk− 2 · · · uk− N + 1

x̄ uu (0) x̄ uu (1) x̄ uu (2) . . . x̄ uu (N − 1)

x̄ uu (1) x̄ uu (0) x̄ uu (1) . . . x̄ uu (N − 2)
X̄ uu = E{ u k · u T
k} = x̄ uu (2) x̄ uu (1) x̄ uu (0) . . . x̄ uu (N − 3)
... ...
x̄ uu (N − 1) x̄ uu (N − 2) x̄ uu (N − 3) ... x̄ uu (0)

• i.e. symmetric & Toeplitz & non-negative definite

DSP 2016 / Chapter-6: Wiener Filters & the LMS Algorithm 20 / 32

 Optimal Filtering : Wiener Filters

36

MMSE
Opt i m alcost
fi l t erfunction canfi l t er s
i ng/ W i ener
be expanded as…(continued)
J M SE (w ) = E{ d2 T T T
k } + w E{ u k u k } w − 2w E{ u k dk } .
X̄ uu X̄ du

cost function is convex, with a (mostly) unique minimum,

obtained by setting the gradient equal to zero:
∂J M SE (w )
0= [ ]w = w W F = [2X̄ uuw − 2X̄ du]w = w W F
∂w
Wiener-Hopf equations :

X̄ uu · w W F = X̄ du → w W F = X̄ − 1
uu X̄ du .....simple enough!

This is the ‘Wiener Filter’ solution

DSP 2016 / Chapter-6: Wiener Filters & the LMS Algorithm 21 / 32

 Optimal Filtering : Wiener Filters

38

How do we
Everything yousolve
need to the
knowWiener–Hopf equations?
about matrices and vector s (I )

( L+1
solving linear systems (N linear
linear equations
equations inin
NL+1 unknowns)
unknowns):

12 3 1
· wWF = → wWF =
34 7 1

O(L33)) arithmetic operations

requires O(N

O(L22)) arithmetic operations if X̄ uu is Toeplitz

requires O(N
• Schur algorithm
• Levinson-Durbin algorithm
= used intensively in applications, e.g. in speech codecs, etc.
details omitted, see Appendix

DSP 2016 / Chapter-6: Wiener Filters & the LMS Algorithm 22 / 32

 Adaptive Filtering: LMS Algorithm

How do we solve the Wiener–Hopf equations?

Alternatively, an iterative steepest

descent algorithm can be used

This will be the basis for the derivation of the

Least Mean Squares (LMS)
adaptive filtering algorithm…

Bernard Widrow 1965 (https://www.youtube.com/watch?v=hc2Zj55j1zU)

DSP 2016 / Chapter-6: Wiener Filters & the LMS Algorithm 23 / 32

 Adaptive Filtering: LMS Algorithm

How do we compute the Wiener filter?

1) Cfr supra: By solving Wiener-Hopf equations (L+1 equations in L+1 unknowns)

2) Can also apply iterative procedure to minimize MMSE criterion, e.g.

here n is iteration index

μ is ‘stepsize’ (to be tuned..)

DSP 2016 / Chapter-6: Wiener Filters & the LMS Algorithm 24 / 32
 Adaptive Filtering: LMS Algorithm
Bound on stepsize ?

DSP 2016 / Chapter-6: Wiener Filters & the LMS Algorithm 25 / 32

 Adaptive Filtering: LMS Algorithm
Convergence speed?

 small λ_i implies slow convergence

DSP λ_min
2016 <<λ_max
/ Chapter-6: (hence
Wiener Filters & thesmall μ) implies *very* slow convergence
LMS Algorithm 26 / 32
 Adaptive Filtering: LMS Algorithm

as follows
Replace n+1 by n for convenience…
w(n) = w(n -1)+ m.(E{uk .dk }- E{uk .uTk }.w(n -1))
Then replace iteration index n by time index k
k
(i.e. perform 1 iteration per sampling interval)

w[k] = w[k -1]+ m.(E{uk .dk }- E{uk .uTk }.w[k -1])

Then leave out expectation operators
(i.e. replace expected values by instantaneous estimates)

wLMS [k] = wLMS [k -1]+ m.uk .(dk - u .wLMS [k -1]) T

k
‘a priori error’

DSP 2016 / Chapter-6: Wiener Filters & the LMS Algorithm 27 / 32

 Adaptive Filtering: LMS Algorithm

wLMS [k] = wLMS [k -1]+ m.uk .(dk - u .wLMS [k -1]) T

k
Simple algorithm, can even draw
signal flow graph (=realization)…

DSP 2016 / Chapter-6: Wiener Filters & the LMS Algorithm 28 / 32

 Adaptive Filtering: LMS Algorithm

Whenever LMS has reached the WF solution, the expected value of u k .(dk - uTk.wLMS [k -1])
(=estimated gradient in update formula) is zero, but the instantaneous value is generally non-
DSP 2016
zero / Chapter-6:
(=noisy), andWiener
henceFilters
LMS& the
willLMS Algorithm
again move away from the WF solution! 29 / 32
 Adaptive Filtering: LMS Algorithm

[¥]

[¥] [¥]
LL
ål i
i=0

å
LL
i=0
li L L
0.2
m<
L.E{uk2 }
means step size has to be much smaller…!

DSP 2016 / Chapter-6: Wiener Filters & the LMS Algorithm 30 / 32

 Adaptive Filtering: LMS Algorithm

LMS is an extremely popular algorithm

many LMS-variants have been developed (cheaper/faster/…)…
(see p.35)
m
w NLMS [k] = w NLMS [k -1] + .u k .(dk - uTk.w NLMS [k -1])
a + uTk .u k

K is block index, LB is block size

m
.å u(K-1). LB +i .(d(K-1). LB +i - uT(K-1).
LB
w BLMS [K ] = w BLMS [K -1]+ ( LB +i .w BLMS [K -1])
L i=1

DSP 2016 / Chapter-6: Wiener Filters & the LMS Algorithm 31 / 32

 Adaptive Filtering: LMS Algorithm

= LMS with normalized step size

(mostly used in practice)

mm
w
wNLMS
NLMS[k] = w NLMS
(k +1) = w[k -1]+
(k) + .u .(d
.u -
k k -k .w
k k .(d u T T
u k .w
NLMS [k -1])
(k))
NLMS
aa+ + T T
u ku.uk .u
k k
NLMS

DSP 2016 / Chapter-6: Wiener Filters & the LMS Algorithm 32 / 32