UTFPR - CPGEI

ETH0000 - Estimation Theory

Lecture Notes
Lecture 02

Daniel Rodrigues Pipa

[email protected]

October 5, 2019

2019-10-05 Prof. Daniel R. Pipa 1/41

 2 Minimum Variance Unbiased Estimation

2 Minimum Variance Unbiased Estimation

2019-10-05 Prof. Daniel R. Pipa 2/41

 2 Minimum Variance Unbiased Estimation 2.3 Unbiased Estimators

Introduction
Definition
An estimator is said unbiased if

E(θ̂) = θ a<θ<b

where (a, b) is the range of possible values of θ.

Example 2.1
N −1
1 󰁛
 = x[n]
N
n=0
has expected value
󰀣 󰀤
󰀓 󰀔 N −1
1 󰁛
N −1
1 󰁛
E Â = E x[n] = E (x[n]) = A
N N
n=0 n=0

for all A and is, therefore, unbiased.

2019-10-05 Prof. Daniel R. Pipa 3/41
 2 Minimum Variance Unbiased Estimation 2.3 Unbiased Estimators

PDF of the estimator

󰀃 󰀄
The PDF of the previous estimator is N A, σ 2 /N (proof for homework).

Unbiased estimators usually have symmetric PDF centered about the true
value of θ, although this is not necessary.
󰀓 󰀔
The restriction that E θ̂ = θ for all θ is an important one. Let
θ̂ = g (x), where x = [x[0] x[1] · · · x[N − 1]]T , it asserts that
󰀓 󰀔 󰁝
E θ̂ = g (x) p (x; θ) dx for all θ.

This may hold for some values of θ, but not others.

2019-10-05 Prof. Daniel R. Pipa 4/41
 2 Minimum Variance Unbiased Estimation 2.3 Unbiased Estimators

Biased Estimator

Example 2.2
N −1
1 󰁛
Ǎ = x[n]
2N
n=0

has expected value

󰀣 󰀤
󰀓 󰀔 N −1
1 󰁛
N −1
1 󰁛 1
E Â = E x[n] = E (x[n]) = A
2N 2N 2
n=0 n=0
=A if A = 0
∕= A if A ∕= 0

is clearly a biased estimator.

2019-10-05 Prof. Daniel R. Pipa 5/41

 2 Minimum Variance Unbiased Estimation 2.3 Unbiased Estimators

Unbiasedness

Definition
The bias of an estimator is defined as

b(θ) = E(θ̂) − θ.

◮ Unbiasedness does not necessary mean good estimation. It only

guarantees that the true value will be attained on the average.
◮ Biased estimators → systematic errors → poor estimators.
Suppose multiple estimates {θ̂1 , θ̂2 , · · · , θ̂n } of the same parameter are
available. We may try to average the estimates to improve performance
n
1󰁛
θ̂ = θ̂i .
n
i=1

2019-10-05 Prof. Daniel R. Pipa 6/41

 2 Minimum Variance Unbiased Estimation 2.4 Minimum Variance Criterion

Optimal estimators
In searching for optimal estimators, we need to adopt some optimality
criterion. A natural one is the mean square error (MSE), define as
󰀓 󰀔 󰀗󰀓 󰀔2 󰀘
mse θ̂ = E θ̂ − θ .

This measures the average mean squared deviation of the estimator from
the true value.
We may rewrite the MSE as
󰀓 󰀔 󰀗󰀓 󰀔2 󰀘
mse θ̂ = E θ̂ − θ
󰀝󰁫󰀓 󰀓 󰀔󰀔 󰀓 󰀓 󰀔 󰀔󰁬2 󰀞
=E θ̂ − E θ̂ + E θ̂ − θ
󰀓 󰀔
= var θ̂ + b2 (θ)

2019-10-05 Prof. Daniel R. Pipa 7/41

 2 Minimum Variance Unbiased Estimation 2.4 Minimum Variance Criterion

Unrealizable estimators
Consider a modified DC estimator
N −1
1 󰁛
Ǎ = a x[n]
N
n=0

for some constant

󰀃 󰀄 a. Let’s try to 󰀃find󰀄 a which results in the minimum
MSE. With E Ǎ = aA and var Ǎ = a2 σ 2 /N , we have

󰀃 󰀄 a2 σ 2
mse Ǎ = + (a − 1)2 A2
N
Differentiating the MSE

d 󰀃 󰀄 2aσ 2 A2
mse Ǎ = + 2 (a − 1) A2 aopt =
da N A2 + σ 2 /N
The estimator is not realizable because the optimal value of a depends on
the unknown parameter A.
2019-10-05 Prof. Daniel R. Pipa 8/41
 2 Minimum Variance Unbiased Estimation 2.5 Existence of MVU Estimator

Existence of MVU Estimator I

The MVU estimator may not exist: the variance of the estimator must be
minimum among all unbiased estimators for all range of the parameter.

2019-10-05 Prof. Daniel R. Pipa 9/41

 2 Minimum Variance Unbiased Estimation 2.6 Finding the MVU Estimator

Finding the MVU Estimator

Approaches discussed throughout the course:

1. Determine the Cramer-Rao lower bound (CRLB), a theoretical lower limit for the
variance of any unbiased estimator, and check if some estimator satisfies it. If the
variance of an estimator equals the CRLB, then it is the MVU estimator.
2. Apply the Rao-Blackwell-Lehmann-Scheffe (RBLS) theorem if no estimator equals
CRLB, since a MVU may still exist (e.g. θ1 ). The procedure finds a sufficient
statistics (efficient use data) and then an unbiased estimator.
3. Further restrict the estimator to be linear. Find the BLUE. This is somethings a
severe restriction and only for particular data sets it produces the MVU estimator.

2019-10-05 Prof. Daniel R. Pipa 10/41

 2 Minimum Variance Unbiased Estimation 2.7 Extension to a Vector Parameter

Extension to a Vector Parameter

If θ = [θ1 θ2 · · · θp ]T is a vector of unknown parameters, then we say that
󰁫 󰁬T
an estimator θ̂ = θˆ1 θˆ2 · · · θˆp is unbiased if
󰀓 󰀔
E θ̂i = θi a i < θ i < bi
for i = 1, · · · , p. By defining
󰀵 󰀓 󰀔󰀶
E θ̂1
󰀓 󰀔 󰀹 󰀺
󰀹 .. 󰀺
E θ̂ = 󰀹 . 󰀺
󰀷 󰀓 󰀔󰀸
E θ̂p
we can equivalently define an unbiased estimator to have the property
󰀓 󰀔
E θ̂ = θ
󰀓 󰀔
A MVU estimator has the additional property that var θ̂i , i = 1, · · · , p
is minimum among all unbiased estimators.
2019-10-05 Prof. Daniel R. Pipa 11/41
 3 Cramer-Rao Lower Bound

3 Cramer-Rao Lower Bound

2019-10-05 Prof. Daniel R. Pipa 12/41

 3 Cramer-Rao Lower Bound 3.3 Estimator Accuracy Consideration

Estimator Accuracy Consideration

Estimator Accuracy

◮ All information is in the observed data and the underlying PDF.

◮ The more the PDF is influenced by the unknown parameter, the
better we should be able to estimate it.

Example 3.1 - PDF Dependence on Unknown Parameter

For a DC level estimation, suppose a single sample is observed
󰀃 󰀄
x [0] = A + w [0] , w[0] ∼ N 0, σ 2
󰀓 󰀔
and an unbiased estimator  = x [0] , var  = σ 2 .
The estimator accuracy improves as σ 2 decreases.

2019-10-05 Prof. Daniel R. Pipa 13/41

 3 Cramer-Rao Lower Bound 3.3 Estimator Accuracy Consideration

Estimator Accuracy Consideration cont.

Alternative interpretation
󰀗 󰀘
1 1
pi (x[0]; A) = 󰁴 exp − 2 (x[0] − A)2
2πσi2 2σi

Definition
When the PDF is viewed as a function of the unknown parameter, with x
fixed, it is termed likelihood function. The log-likelihood function is the
natural logarithm of the likelihood function.

From the previous example

√ 1
ln p (x[0]; A) = − ln 2πσ 2 − (x[0] − A)2 .
2σ 2

2019-10-05 Prof. Daniel R. Pipa 14/41

 3 Cramer-Rao Lower Bound 3.3 Estimator Accuracy Consideration

Estimator Accuracy Consideration cont.

Fixing x[0] = 3 and plotting the PDF against A for different σ.

If σ = 1/3, A > 4 is highly unlikely. If σ = 1, the PDF depends less on A.

Intuitively, the “sharpness” of the likelihood function determines how

accurately we can estimate the unknown parameter.

To quantify this notion, we derive the Cramer-Rao Lower Bound (CRLB).

2019-10-05 Prof. Daniel R. Pipa 15/41

 3 Cramer-Rao Lower Bound 3.4 Cramer-Rao Lower Bound

Cramer-Rao Lower Bound

It is assume that the PDF p (x; θ) satisfies the “regularity” condition
󰀗 󰀘
∂ ln p (x; θ)
E =0 for all θ
∂θ

Theorem
Cramer-Rao Lower Bound: the variance of any unbiased estimator θ̂ must satisfy
󰀓 󰀔 1
var θ̂ ≥ 󰀗 2
󰀘
∂ ln p (x; θ)
−E
∂θ2

An unbiased estimator may be found that attains the CRLB for all θ if and only if
∂ ln p (x; θ)
= I (θ) (g (x) − θ)
∂θ
for some functions g and I. The estimator, which is the MVU, is θ̂ = g (x) and
the minimum variance is 1/I (θ).
2019-10-05 Prof. Daniel R. Pipa 16/41
 3 Cramer-Rao Lower Bound 3.4 Cramer-Rao Lower Bound

Example 3.2 - CRLB for the previous example

First, note that
󰀗 󰀘 󰁝 2
∂ 2 ln p (x; θ) ∂ ln p (x; θ)
E 2
= p (x; θ) dx
∂θ ∂θ2

Then
∂ ln p (x[0]; A) 1
= 2 (x[0] − A)
∂A σ󰀗 󰀘
∂ 2 ln p (x[0]; A) 1 ∂ 2 ln p (x[0]; A) 1
2
=− 2 −E 2
= 2
∂A σ ∂A σ

Finally 󰀓 󰀔
var  ≥ σ 2 .
󰀓 󰀔
Because the estimator was unbiased and had var  = σ 2 , it attained
the CRLB and was the MVU estimator.
2019-10-05 Prof. Daniel R. Pipa 17/41
 3 Cramer-Rao Lower Bound 3.4 Cramer-Rao Lower Bound

Efficient estimators

Definition
An unbiased estimator that attains the CRLB is said to be efficient.
An MVU estimator may or may not be efficient, as it requires only that
the variance be less than any other unbiased estimators.

2019-10-05 Prof. Daniel R. Pipa 18/41

 6 Best Linear Unbiased Estimators

6 Best Linear Unbiased Estimators

2019-10-05 Prof. Daniel R. Pipa 19/41

 6 Best Linear Unbiased Estimators 6.3 Definition of the BLUE

Introduction
Consider the cases where
◮ we do not know the PDF of the data. The CRLB or Sufficient
Statistic cannot be applied; or
◮ We know the PDF, but the MVU cannot be found.
We may resort to suboptimal estimators.
Alternative approach
Restrict the estimator to be linear in the data and find the estimator that
is unbiased and has minimum variance.
Advantages
◮ We need knowledge of only the first and second moments of PDF.
◮ We do not need knowledge of the complete PDF.
◮ If the performance of the estimator meets our system requirements,
its use may be adequate for the problem.
2019-10-05 Prof. Daniel R. Pipa 20/41
 6 Best Linear Unbiased Estimators 6.3 Definition of the BLUE

Definition of the BLUE I

A linear estimator is given by
N
󰁛 −1
θ̂ = an x[n] = aT x
n=0

where the an ’s are constants to be determined.

Definition
The best linear unbiased estimator (BLUE) is the linear unbiased estimator
that has the minimum variance.
The BLUE may also be the MVU estimator for some problems, suboptimal
for others, or totally inadequate. For example, estimating the power of
WGN 󰀣N −1 󰀤 N −1
󰀓 󰀔 󰁛 󰁛
󰁦2
E σ =E an x[n] = an E (x[n]) = 0
n=0 n=0
2019-10-05 Prof. Daniel R. Pipa 21/41
 6 Best Linear Unbiased Estimators 6.3 Definition of the BLUE

Definition of the BLUE II

However, we can use the BLUE in the transformed data y[n] = x2 [n].

2019-10-05 Prof. Daniel R. Pipa 22/41

 6 Best Linear Unbiased Estimators 6.4 Finding the BLUE

Finding the BLUE

The unbiased constraint is

󰀣N −1 󰀤 N −1
󰀓 󰀔 󰁛 󰁛
E θ̂ = E an x[n] = an E (x[n]) = aT E (x) = θ (6.2)
n=0 n=0

Letting a = [a0 a1 · · · aN −1 ]T the variance is

󰀓 󰀔 󰁫󰀃 󰀄2 󰁬 󰁫󰀃 󰀄2 󰁬
var θ̂ = E aT x − aT E (x) = E aT (x − E (x))
󰁫 󰁬
= E aT (x − E (x)) (x − E (x))T a = aT Ca. (6.3)

The vector a of weights is found by minimizing (??) subject to the

constraint of (??).

2019-10-05 Prof. Daniel R. Pipa 23/41

 6 Best Linear Unbiased Estimators 6.4 Finding the BLUE

Unbiased Constraint
In order to satisfy the unbiased constraint, E (x[n]) must be linear in θ or

E (x[n]) = s[n]θ

where the s[n]’s are known. This is because a linear combination of the
E (x[n])’s must yield θ. Additionally, writing x[n] as

x[n] = E (x[n]) + [x[n] − E (x[n])]

and viewing x[n] − E (x[n]) as noise, we have

x[n] = θs[n] + w[n]

Application of BLUE
Amplitude estimation of known signals in noise. To generalize, we will
need nonlinear transformations.
2019-10-05 Prof. Daniel R. Pipa 24/41
 6 Best Linear Unbiased Estimators 6.4 Finding the BLUE

Scalar case
The unbiasedness constraint results in
N
󰁛 −1 N
󰁛 −1
an E (x[n]) = an s[n]θ = θ
n=0 n=0
N
󰁛 −1
an s[n] = 1 aT s = 1.
n=0
Now, we use Lagrange multipliers to minimize the variance subject to the
constraint
min J(a) = aT Ca + λ(aT s − 1)
∂J λ
= 2Ca + λs = 0 a = − C−1 s
∂a 2
In the equatlity constraint
λ λ 1 sT C−1
− sT C−1 s = 1 − = T −1 θ̂ = x
2 2 s C s sT C−1 s
2019-10-05 Prof. Daniel R. Pipa 25/41
 6 Best Linear Unbiased Estimators 6.5 Extension to a Vector Parameter

Extension to a Vector Parameter I

Considering a p × 1 vector parameter to be estimated linearly from the data
󰀵 T󰀶
N −1 a1
󰁛 󰀹 .. 󰀺
T
θ̂i = ain x[n] = ai x A=󰀷 . 󰀸 θ̂ = Ax
n=0 T
ap
where A is p × N . Extending directly that scalar case, the variances are
󰀓 󰀔
var θ̂i = aTi Cai

The unbiasedness condition

󰀓 󰀔
E θ̂ = AE (x) = θ

requires that E (x) depends linearly with the parameter

󰀅 󰀆
E (x) = Hθ = h1 θ1 h2 θ2 · · · hp θp
where H is a known or given matrix.
2019-10-05 Prof. Daniel R. Pipa 26/41
 6 Best Linear Unbiased Estimators 6.5 Extension to a Vector Parameter

Derivation of BLUE I
The unbiasedness condition becomes

AHθ = θ AH = I aTi hj = δij

To minimize the variance subject to the unbiasedness constraint, we

employ Lagrange multipliers for the equality constraint and define a cost
function (the Lagrangian)
p
󰁛 󰀃 󰀄
Ji = aTi Cai + λij aTi hj − δij
j=1

where λi = [λi1 λi2 · · · λip ]T . Taking the gradient of the Lagrangian

󰁛 p
∂Ji
= 2Cai + λij hi = 2Cai + Hλi
∂ai
j=1

2019-10-05 Prof. Daniel R. Pipa 27/41

 6 Best Linear Unbiased Estimators 6.5 Extension to a Vector Parameter

Derivation of BLUE II
Setting the gradient to zero yields
1 1 1
ai = − C−1 Hλi AT = − C−1 HΛ A = − ΛT HT C−1
2 2 2
Substituting in the unbiasedness condition
1 1 󰀃 󰀄−1
− ΛT HT C−1 H = I − HT C−1 HΛ = I Λ = −2 HT C−1 H
2 2
The optimal matrix A is
󰀃 󰀄−1 T −1
A = HT C−1 H H C

and the BLUE is

󰀃 󰀄−1 T −1
θ̂ = Ax = HT C−1 H H C x.

2019-10-05 Prof. Daniel R. Pipa 28/41

 6 Best Linear Unbiased Estimators 6.5 Extension to a Vector Parameter

Derivation of BLUE III

The covariance of θ̂ is
󰀗󰀓 󰀓 󰀔󰀔 󰀓 󰀓 󰀔󰀔T 󰀘
Cθ̂ = E θ̂ − E θ̂ θ̂ − E θ̂

If we assume, as in the scalar case, that

x = E (x) + [x − E (x)] = Hθ + w

then
󰀓 󰀔 󰀃 󰀄−1 T −1
θ̂ − E θ̂ = HT C−1 H H C (Hθ + w) − θ
󰀃 T −1 󰀄−1 T −1
= H C H H C w

2019-10-05 Prof. Daniel R. Pipa 29/41

 6 Best Linear Unbiased Estimators 6.5 Extension to a Vector Parameter

Derivation of BLUE IV

and
󰁫󰀃 󰀄−1 T −1 󰀃 T −1 󰀄−1 󰁬
T −1 T −1
Cθ̂ = E H C H H C ww C H H C H
󰀃 T −1 󰀄−1 T −1 󰀃 󰀄−1
= H C H H C CC−1 H HT C−1 H
󰀃 󰀄−1
= HT C−1 H

with the minimum variances given by diagonal elements of Cθ̂ or

󰀓 󰀔 󰀅 󰀆
var θ̂i = Cθ̂ ii .

2019-10-05 Prof. Daniel R. Pipa 30/41

 6 Best Linear Unbiased Estimators 6.5 Extension to a Vector Parameter

Gauss-Markov Theorem

Theorem
If the data are of the general linear model form

x = Hθ + w

where H is a known N × p (measurement) matrix, θ is a p × 1 vector of

parameters to be estimated, and w is an N × 1 noise vector with zero
mean and covariance C (the PDF of w is otherwise arbitraty), then the
BLUE of θ is 󰀃 󰀄−1 T −1
θ̂ = HT C−1 H H C x.
Additionally, if w ∼ N (0, C), then data is truly Gaussian and the BLUE is
also the MVU estimator.

2019-10-05 Prof. Daniel R. Pipa 31/41

 6 Best Linear Unbiased Estimators Problems

Homework Assignments I

1. The data {x[0], x[1], · · · , x[N − 1]} are observed where

󰀃 x[n]’s
󰀄 are
independent and identically distributed (IID) as N 0, σ 2 . We wish
to estimate the variance σ 2 as
N −1
ˆ2
1 󰁛 2
σ = x [n].
N
n=0

Is this an unbiased estimator? Find the variance of σˆ2 and examine

what happens as N → ∞.
󰀓 󰀔 󰀓 󰀔
2. Show, detailing every step, that mse θ̂ = var θ̂ + b2 (θ).
󰀃 󰀄
3. Prove that the PDF of  given in Example 2.1 is N A, σ 2 /N .

2019-10-05 Prof. Daniel R. Pipa 32/41

Alternative form of CRLB
Noting that
󰀕 󰀖′ 󰀕 󰀖2
′′ f′ f ′′ f − f ′ f ′ f ′′ f′ f ′′ 󰀅 ′ 󰀆2
(ln f ) = = 2
= − = − (ln f )
f f f f f

Then 󰀥 ∂ 2 p(x;θ) 󰀕
󰀗 󰀘 󰀖2 󰀦
∂ 2 ln p (x; θ) ∂θ 2 ∂ ln p (x; θ)
E =E −
∂θ2 p (x; θ) ∂θ
But
󰀥 ∂ 2 p(x;θ) 󰀦 󰁝 ∂ 2 p(x;θ) 󰁝
∂θ 2 ∂θ 2 ∂2 ∂2
E = p (x; θ) dx = 2 p (x; θ) dx = 1=0
p (x; θ) p (x; θ) ∂θ ∂θ2

Therefore 󰀗 󰀘 󰀥󰀕 󰀖2 󰀦
∂ 2 ln p (x; θ) ∂ ln p (x; θ)
E = −E
∂θ2 ∂θ
This expression may be useful for theoretical work.
2019-10-05 Prof. Daniel R. Pipa 33/41
Fisher Information I

Definition
The Fisher Information of the data x is defined as
󰀗 2 󰀘
∂ ln p (x; θ)
I (θ) = −E
∂θ2

It measures the amount of information that an observation x carries about

an unknown parameter θ.

It has important properties for an information measure

1. Nonnegativity;
2. Additive for independent observations. That is, the CRLB for N IID
observations is 1/N times that for one observation.

2019-10-05 Prof. Daniel R. Pipa 34/41

Fisher Information II
Verification: for independent observations
N
󰁛 −1
ln p (x; θ) = ln p (x[n]; θ)
n=0

This results in
󰀗 󰀘 N
󰁛 −1 󰀗 2 󰀘
∂ 2 ln p (x; θ) ∂ ln p (x[n]; θ)
−E =− E
∂θ2 ∂θ2
n=0

and finally for identically distributed observations

I (θ) = N i (θ)

where 󰀗 󰀘
∂ 2 ln p (x[n]; θ)
i (θ) = −E
∂θ2
is the Fisher information for one sample.
2019-10-05 Prof. Daniel R. Pipa 35/41
Regularity conditions for the CRLB

From the axiom of probabilty Noting that

󰁝 1 ′
′ ′
p (x; θ) dx = 1 (ln u) = u u′ = (ln u) u
u
We get
Taking the partial derivative
󰁝
󰁝 ∂ ln p (x; θ)
∂ ∂ p (x; θ) dx = 0
p (x; θ) dx = 1 ∂θ
∂θ ∂θ
which is
Assuming that we can swap 󰀗 󰀘
differentiation and integration (special ∂ ln p (x; θ)
E = 0.
case of Leibniz’s integral rule) ∂θ
󰁝 Important: if this condition is not
∂p (x; θ)
dx = 0 satisfied, the CRLB cannot be applied.
∂θ

2019-10-05 Prof. Daniel R. Pipa 36/41

Derivation of the CRLB I

Derivation from unbiasedness

󰁝 󰀓 󰀔
θ̂ − θ p (x; θ) dx = 0
󰁝 󰀓 󰀔
∂ ∂
θ̂ − θ p (x; θ) dx = 0
∂θ ∂θ
󰁝
Leibniz ∂ 󰀓 󰀔
→ θ̂ − θ p (x; θ) dx = 0
integral rule ∂θ
󰁝 󰁝 󰀓 󰀔 ∂
− p (x; θ) dx + θ̂ − θ p (x; θ) dx = 0
∂θ
󰁝 󰀓 󰀔 ∂ ln p (x; θ)
θ̂ − θ p (x; θ) dx = 1
∂θ
󰁝 󰁫󰀓 󰀔󰁳 󰁬 󰀗 ∂ ln p (x; θ) 󰁳 󰀘
θ̂ − θ p (x; θ) · p (x; θ) dx = 1
∂θ

2019-10-05 Prof. Daniel R. Pipa 37/41

Derivation of the CRLB II
Using Cauchy-Schwarz inequality
󰀗󰁝 󰀓 󰀔2 󰀘 󰀥󰁝 󰀕 󰀖 󰀦
∂ ln p (x; θ) 2
θ̂ − θ p (x; θ) dx · p (x; θ) dx ≥ 1
∂θ
󰁿 󰁾󰁽 󰂀 󰁿 󰁾󰁽 󰂀
var(θ̂ ) I(θ)

Defining
󰀥󰀕 󰀖2 󰀦 󰀗 󰀘
∂ ln p (x; θ) ∂ 2 ln p (x; θ)
I (θ) = E = −E
∂θ ∂θ2

The Cramer-Rao lower bound is

󰀓 󰀔 1
var θ̂ ≥
I (θ)

2019-10-05 Prof. Daniel R. Pipa 38/41

 3.5 General CRLB for Signals in White Noise

Observation Model and PDF I

A typical case is a signal embedded in WGN

x[n] = s[n; θ] + w[n] n = 0, 1, · · · , N − 1.

which has the likelihood function

󰀫 N −1
󰀬
1 1 󰁛
p (x; θ) = N exp − 2 (x[n] − s[n; θ])2
(2πσ 2 ) 2 2σ
n=0

Differentiating produces
N −1
∂ ln p (x; θ) 1 󰁛 ∂s[n; θ]
= 2 (x[n] − s[n; θ])
∂θ σ ∂θ
n=0

N −1
󰀫 󰀕 󰀖 󰀬
∂ 2 ln p (x; θ) 1 󰁛 ∂ 2 s[n; θ] ∂s[n; θ] 2
= 2 (x[n] − s[n; θ]) −
∂θ2 σ ∂θ2 ∂θ
n=0
2019-10-05 Prof. Daniel R. Pipa 39/41
 3.5 General CRLB for Signals in White Noise

Observation Model and PDF II

Taking the expected values yields

󰀗 󰀘 N −1 󰀕 󰀖
∂ 2 ln p (x; θ) 1 󰁛 ∂s[n; θ] 2
E =− 2
∂θ2 σ ∂θ
n=0

so that finally
󰀓 󰀔 σ2
var θ̂ ≥ N −1 󰀕 󰀖2
󰁛 ∂s[n; θ]
∂θ
n=0

That is, signals that change rapidly with the unknown parameter results in
accurate estimators.

2019-10-05 Prof. Daniel R. Pipa 40/41

 3.5 General CRLB for Signals in White Noise

Example 3.5 - Sinusoidal Frequency Estimation

Assuming
1
s[n; f0 ] = A cos (2πf0 n + φ) 0 < f0 <
2
where the amplitude and phase are assumed known. The CRLB becomes
󰀓 󰀔 σ2
var fˆ0 ≥ 󰁓 N −1 2
A2 n=0 [2π sin (2πf0 n + φ)]
Plotting for an SNR A2 /σ 2 = 1, N = 10 and φ = 0.

2019-10-05 Prof. Daniel R. Pipa 41/41