Bayesian Inference

Loukia Meligkotsidou

National and Kapodistrian University of Athens

MSc in Statistics and Operational Research,

Department of Mathematics

Loukia Meligkotsidou, University of Athens Bayesian Inference

Conjugate Analysis for the Normal Model

Let x = (x1 , . . . , xn ) be an i.i.d sample from the N(µ, τ −1 )

distribution with both parameters unknown.

The likelihood of the observations is

n
( )
τX
f (x | µ, τ ) ∝ τ n/2 exp − (xi − µ)2 .
2
i=1

The conjugate prior distribution for (µ, τ ) is of the form

f (µ, τ ) = f (τ ) f (µ | τ )
where f (τ ) ≡ Gamma (a, b) and
f (µ | τ ) ≡ N ξ, (cτ )−1 . That is
n τc o
f (µ, τ ) ∝ τ a−1 exp {−τ b} × τ 1/2 exp − (µ − ξ)2 .
2

Loukia Meligkotsidou, University of Athens Bayesian Inference

Joint and Conditional Posteriors
The joint posterior distribution of (µ, τ ), obtained by Bayes
theorem, is

f (µ, τ | x) ∝ f (µ, τ ) f (x | µ, τ )
n
( " #)
n+1
+a−1 1X c
∝ τ 2 exp −τ (xi − µ)2 + (µ − ξ)2 + b .
2 2
i=1

The conditional posterior distribution of τ given µ is given by

f (τ | x, µ) ∝ f (µ, τ | x) (as a function of τ )

" n ( #)
n+1 1 X c
∝ τ 2 +a−1 exp −τ (xi − µ)2 + (µ − ξ)2 + b ,
2 2
i=1

which is a Gamma(P, Q)P distribution with parameters

n
P = n+1
2 + a and Q = 1
2
2 c 2
i=1 (xi − µ) + 2 (µ − ξ) + b.
Loukia Meligkotsidou, University of Athens Bayesian Inference
Joint and Conditional Posteriors
The conditional posterior distribution of µ given τ is

f (µ | x, τ ) ∝ f (µ, τ | x) (as a function of µ)

( " n #)
τ X 2 2
∝ exp − (xi − µ) + c(µ − ξ)
2
i=1
n
( )
τ 2 X τc 2
∝ exp − nµ + τ µ xi − µ + τ cµξ
2 2
i=1
 
τ (n + c) 2
∝ exp − µ + τ (nx̄ + cξ)µ
2
 
τ (n + c) 2 nx̄ + cξ
∝ exp − [µ + 2 µ] ,
2 n+c
nx̄+cξ
which is a Normal(B, D 2 ) distribution with mean B = n+c and
variance D 2 = τ −1 (n + c)−1 .
Loukia Meligkotsidou, University of Athens Bayesian Inference
Marginal Posterior of τ
Exact Bayesian inference is based on the marginal posteriors.
Z
f (τ | x) = f (µ, τ | x)dµ
Z ∞ n
n+1
+a−1 τX τc
∝ τ 2 exp{− (xi − µ)2 − (µ − ξ)2 − τ b}dµ
−∞ 2 2
i=1
n
n 1 X c
= τ 2 +a−1 exp{−τ [ xi2 + ξ 2 + b]}
2 2
i=1
Z ∞ n
τ (n + c) 2 X
× τ 1/2 exp{− µ + τ( xi + c)µ}dµ
−∞ 2
i=1
n
n 1X cξ 2 (nx̄ + cξ)2
∝ τ 2 +a−1 exp{−τ [ xi2 + +b− ]},
2 2 2(n + c)
i=1
This is again a Gamma(P 0 , Q 0 ) density, with parameters
2
P 0 = n2 + a and Q 0 = 21 ni=1 xi2 + c2 ξ 2 + b − (nx̄+cξ)
P
2(n+c) .
Loukia Meligkotsidou, University of Athens Bayesian Inference
Marginal Posterior of µ

The marginal posterior of µ is obtained by integrating the joint

posterior distribution over τ , i.e.
Z
f (µ | x) = f (µ, τ | x)dτ
Z ∞ n
n+1
+a−1 1X c
∝ τ 2 exp{−τ [ (xi − µ)2 + (µ − ξ)2 + b]}dτ
0 2 2
i=1
" n
#−( n+1 +a)
2
1X c
∝ (xi − µ)2 + (µ − ξ)2 + b .
2 2
i=1

By some calculus manipulation, it can be shown that the

normalised version of this formula is a non-standardized (three
parameter) Student’s-t probability density function.

Loukia Meligkotsidou, University of Athens Bayesian Inference

The non-standardized Student’s-t distribution

The standard Student’s-t distribution can be generalized to a three

parameter location-scale family, introducing a location parameter µ
and a scale parameter σ , through the relation X = µ + σT , where
T ∼ t(ν).

The resulting non-standardized Student’s-t distribution has p.d.f.

" 2 #− ν+1
Γ( ν+1 2

) 1 x − µ
f (x | ν, µ, σ) = ν √2 1+
Γ( 2 ) πνσ ν σ

Here, σ does not correspond to a standard deviation. It simply sets

the overall scaling of the distribution. The mean and variance of
the non-standardized Student’s t distribution are, respectively,
ν
E (X ) = µ, for ν > 1, and V (X ) = σ 2 ν−2 for ν > 2.

Loukia Meligkotsidou, University of Athens Bayesian Inference

Marginal Likelihood

Z Z
f (x) = f (x | µ, τ )f (µ | τ )f (τ )dµdτ
τ µ
n
Z Z ( )
n+1 n τ X
= (2π)− 2 τ 2 exp − (xi − µ)2
τ µ 2
i=1
b a a−1 τc
× τ exp{−τ b}(2π)−1/2 (cτ )1/2 exp{− (µ − ξ)2 }dµdτ
Γ(a) 2

Loukia Meligkotsidou, University of Athens Bayesian Inference

Marginal Likelihood

Z Z
f (x) = f (x | µ, τ )f (µ | τ )f (τ )dµdτ
τ µ
n
Z Z ( )
n+1 n τ X
= (2π)− 2 τ 2 exp − (xi − µ)2
τ µ 2
i=1
b a a−1 τc
× τ exp{−τ b}(2π)−1/2 (cτ )1/2 exp{− (µ − ξ)2 }dµdτ
Γ(a) 2
n b
a Z
n τ X τc 2
= (2π)− 2 c 1/2 τ 2 +a−1 exp{− xi2 − ξ − τ b}
Γ(a) τ 2 2
n
"Z #
1 1 τ (n + c) X
× (2π)− 2 τ 2 exp{− µ2 + τ ( xi + c)µ}dµ dτ
µ 2
i=1

Loukia Meligkotsidou, University of Athens Bayesian Inference

Marginal Likelihood

Therefore,
ba 1 n 1
f (x) = (2π)− 2 c 2 (n + c)− 2
Γ(a)
n
1 X 2 cξ 2 (nx̄ + cξ)2
Z
n
+a−1
× τ 2 exp{−τ [ xi + +b ]}dτ
τ 2 2 2(n + c)
i=1

Loukia Meligkotsidou, University of Athens Bayesian Inference

Marginal Likelihood

Therefore,
ba 1 n 1
f (x) = (2π)− 2 c 2 (n + c)− 2
Γ(a)
n
1 X 2 cξ 2 (nx̄ + cξ)2
Z
n
+a−1
× τ 2 exp{−τ [ xi + +b ]}dτ
τ 2 2 2(n + c)
i=1
ba 1 − n2 1
= (2π) c 2 (n + c)− 2
Γ(a)
" n #−( n +a)
2
n 1 X 2 cξ 2 (nx̄ + cξ)2
× Γ( + a) xi + +b− .
2 2 2 2(n + c)
i=1

Loukia Meligkotsidou, University of Athens Bayesian Inference

Exercise 1.1
The rock strata A and B are difficult to distinguish in the field.
Through careful laboratory studies it has been determined that the
only characteristic which might be useful in aiding discrimination is
the presence or absence of a particular brachipod fossil. The
probabilities of fossil presence are found to be as follows.

Stratum Fossil present Fossil absent

A 0.9 0.1
B 0.2 0.8

It is also known that rock type A occurs about four times as often
as type B. If a sample is taken, and the fossil found to be present,
calculate the posterior distribution of rock types.
If the geologist always classifies as A when the fossil is found to be
present, and classifies as B when it is absent, what is the
probability she will be correct in a future classification?
Loukia Meligkotsidou, University of Athens Bayesian Inference
Solution

Denote
A: rock stratum A, B: rock stratum B and F: fossil is present.

We are given that Pr(F | A) = 0.9, Pr(F c | A) = 0.1,

Pr(F | B) = 0.2, Pr(F c | B) = 0.8 and Pr(A) = 4 Pr(B).

To obtain the posterior distribution of rock types, after finding the

fossil in the sample, we need to calculate Pr(A | F ) and Pr(B | F ).

Loukia Meligkotsidou, University of Athens Bayesian Inference

Solution

Denote
A: rock stratum A, B: rock stratum B and F: fossil is present.

We are given that Pr(F | A) = 0.9, Pr(F c | A) = 0.1,

Pr(F | B) = 0.2, Pr(F c | B) = 0.8 and Pr(A) = 4 Pr(B).

To obtain the posterior distribution of rock types, after finding the

fossil in the sample, we need to calculate Pr(A | F ) and Pr(B | F ).

Pr(A) + Pr(B) = 1 ⇒ 4 Pr(B) + Pr(B) = 1 ⇒ Pr(B) = 0.2.

Loukia Meligkotsidou, University of Athens Bayesian Inference

Solution

Denote
A: rock stratum A, B: rock stratum B and F: fossil is present.

We are given that Pr(F | A) = 0.9, Pr(F c | A) = 0.1,

Pr(F | B) = 0.2, Pr(F c | B) = 0.8 and Pr(A) = 4 Pr(B).

To obtain the posterior distribution of rock types, after finding the

fossil in the sample, we need to calculate Pr(A | F ) and Pr(B | F ).

Pr(A) + Pr(B) = 1 ⇒ 4 Pr(B) + Pr(B) = 1 ⇒ Pr(B) = 0.2.

Prior: Pr(A) = 0.8 Pr(B) = 0.2

Likelihood: Pr(F | A) = 0.9 Pr(F | B) = 0.2
Prior x likelihood:
Pr(A) Pr(F | A) = 0.72 Pr(B) Pr(F | B) = 0.04

Loukia Meligkotsidou, University of Athens Bayesian Inference

Solution

Law of total probability:

Pr(F ) = Pr(A) Pr(F | A) + Pr(B) Pr(F | B) = 0.76

Loukia Meligkotsidou, University of Athens Bayesian Inference

Solution

Law of total probability:

Pr(F ) = Pr(A) Pr(F | A) + Pr(B) Pr(F | B) = 0.76
Pr(A) Pr(F |A) 72
Posterior: Pr(A | F ) = Pr(F ) = 76
Pr(B) Pr(F |B) 4
and Pr(B | F ) = Pr(F ) = 76 .

Loukia Meligkotsidou, University of Athens Bayesian Inference

Solution

Law of total probability:

Pr(F ) = Pr(A) Pr(F | A) + Pr(B) Pr(F | B) = 0.76
Pr(A) Pr(F |A) 72
Posterior: Pr(A | F ) = Pr(F ) = 76
Pr(B) Pr(F |B) 4
and Pr(B | F ) = Pr(F ) = 76 .

Probability of correct classification:

Pr(correct) = Pr(A, F ) + Pr(B, F c )

= Pr(A) Pr(F | A) + Pr(B) Pr(F c | B)
= 0.72 + 0.16 = 0.88.

Loukia Meligkotsidou, University of Athens Bayesian Inference

Exercise 1.4

A seed collector who has acquired a small number of seeds from a

plant, has a prior belief that the probability θ of germination of
each seed is uniform over the range 0 ≤ θ ≤ 1. She experiments by
sowing two seeds and finds that they both germinate.
i. Write down the likelihood function for θ deriving from this
observation, and obtain the collector’s posterior distribution of
θ
ii. Compute the posterior probability that θ is less than one half.

Loukia Meligkotsidou, University of Athens Bayesian Inference

Exercise 1.4

A seed collector who has acquired a small number of seeds from a

plant, has a prior belief that the probability θ of germination of
each seed is uniform over the range 0 ≤ θ ≤ 1. She experiments by
sowing two seeds and finds that they both germinate.
i. Write down the likelihood function for θ deriving from this
observation, and obtain the collector’s posterior distribution of
θ
ii. Compute the posterior probability that θ is less than one half.

Solution.
(i.) The likelihood function is
 
2 2
f (X = 2 | θ) = θ (1 − θ)2−2 = θ2
2

Loukia Meligkotsidou, University of Athens Bayesian Inference

Exercise 1.4-Solution

Using the uniform prior f (θ) = 1, 0 ≤ θ ≤ 1 we obtain the

posterior distribution f (θ | x) = f (θ)f (x|θ)
f (x) = 3θ2 .
R1
Note that f (x) = 0 θ2 dθ = 31 .

Loukia Meligkotsidou, University of Athens Bayesian Inference

Exercise 1.4-Solution

Using the uniform prior f (θ) = 1, 0 ≤ θ ≤ 1 we obtain the

posterior distribution f (θ | x) = f (θ)f (x|θ)
f (x) = 3θ2 .
R1
Note that f (x) = 0 θ2 dθ = 31 .
R 0.5
(ii.) The prior probability is P (θ < 1/2) = 0 1dθ = 0.5.
The posterior probabilityR is
0.5
P (θ < 1/2 | X = 2) = 0 3θ2 dθ = 0.8.

Loukia Meligkotsidou, University of Athens Bayesian Inference

Exercise 1.5

A posterior distribution is calculated up to a normalizing constant

as
f (θ | x) ∝ θ−3 ,
for θ > 1. Calculate the normalizing constant of this posterior and
the posterior probability of θ < 2.

Loukia Meligkotsidou, University of Athens Bayesian Inference

Exercise 1.5

A posterior distribution is calculated up to a normalizing constant

as
f (θ | x) ∝ θ−3 ,
for θ > 1. Calculate the normalizing constant of this posterior and
the posterior probability of θ < 2.

Solution.
R
Since f (θ | x) dθh = 1,ithus,
R∞ 1 ∞
cθ−2
1 c θ3 dθ = 1 ⇔ −2 = 1 ⇔ c = 2.
1

Loukia Meligkotsidou, University of Athens Bayesian Inference

Exercise 1.5

A posterior distribution is calculated up to a normalizing constant

as
f (θ | x) ∝ θ−3 ,
for θ > 1. Calculate the normalizing constant of this posterior and
the posterior probability of θ < 2.

Solution.
R
Since f (θ | x) dθh = 1,ithus,
R∞ 1 ∞
cθ−2
1 c θ3 dθ = 1 ⇔ −2 = 1 ⇔ c = 2.
1

The posterior probability

R2 is: R2
P (θ < 2 | x) = 1 f (θ | x) dθ = 1 2θ−3 dθ = 3/4.

Loukia Meligkotsidou, University of Athens Bayesian Inference

Exercise

a. Show that Jeffrey’s prior is consistent across 1-1 parameter

transformations.
b. Suppose that X | θ ∼ Binomial (n, θ). Find Jeffrey’s prior for
the corresponding posterior distribution of θ
c. Now suppose that φ = 1θ . What is the Jeffrey’s prior for φ?

Loukia Meligkotsidou, University of Athens Bayesian Inference

Exercise

a. Show that Jeffrey’s prior is consistent across 1-1 parameter

transformations.
b. Suppose that X | θ ∼ Binomial (n, θ). Find Jeffrey’s prior for
the corresponding posterior distribution of θ
c. Now suppose that φ = 1θ . What is the Jeffrey’s prior for φ?
Solution.  
 dθ 
a. We need to show that JΦ φ = JΘ θ  dφ . It is sufficient to show
 2
 dθ 
that IΦ φ = IΘ θ  dφ 
We have that, d logdφ f (x|Φ)
= d log f dθ
(x|θ(φ)) dθ(φ)
dφ . Hence, 
 2    2
d log f (x|φ) d log f (x|θ(φ)) dθ(φ)
I (φ) = E dφ = E dθ dφ =
 2   2  2
d log f (x|θ) dθ  dθ 
E dθ dφ = I (θ)  dφ 

Loukia Meligkotsidou, University of Athens Bayesian Inference

Solution
b.  
n x
f (x | θ) = θ (1 − θ)n−x
x
and L(θ) = log f (x | θ) = x log (θ) + (n − x) log (1 − θ) + c

dL(θ) x n−x
= −
dθ θ 1−θ
d 2 L(θ) x n−x
=− 2 −
dθ 2 θ (1 − θ)2
 
Since E (x) = nθ ⇒ I (θ) = − nθ
θ2
− n−nθ
(1−θ)2
= n 1−θ+θ
θ(1−θ) =
nθ−1 1 − θ−1 ⇒ J (θ) ∝ θ−1/2 (1 − θ)−1/2

Loukia Meligkotsidou, University of Athens Bayesian Inference

Solution
b.  
n x
f (x | θ) = θ (1 − θ)n−x
x
and L(θ) = log f (x | θ) = x log (θ) + (n − x) log (1 − θ) + c

dL(θ) x n−x
= −
dθ θ 1−θ
d 2 L(θ) x n−x
=− 2 −
dθ 2 θ (1 − θ)2
 
Since E (x) = nθ ⇒ I (θ) = − nθ
θ2
− n−nθ
(1−θ)2
= n 1−θ+θ
θ(1−θ) =
nθ−1 1 − θ−1 ⇒ J (θ) ∝ θ−1/2 (1 − θ)−1/2

c.
1 n−x
   
n −x
f (x | φ) = φ 1−
x φ
n
with E (X ) = φ
Loukia Meligkotsidou, University of Athens Bayesian Inference
Solution
 
L (φ) = −x log (φ) + (n − x) log φ−1φ +c =
−n log (φ) + (n − x) log (φ − 1) + c

dL(φ) x −x n
= −
dφ φ−1 φ
2
d L(φ) n−x n
2
=− 2
+ 2
dφ (φ − 1) φ
 2 
d L(φ) n − n/φ n nφ − n (φ − 1)
−1
I (φ) = −E 2
= 2
− 2 = 2
∝ φ3 − φ2
dφ (φ − 1) φ φ (φ − 1)
−1/2
⇒ J (φ) ∝ φ3 − φ2

Loukia Meligkotsidou, University of Athens Bayesian Inference

Solution
 
L (φ) = −x log (φ) + (n − x) log φ−1φ +c =
−n log (φ) + (n − x) log (φ − 1) + c

dL(φ) x −x n
= −
dφ φ−1 φ
2
d L(φ) n−x n
2
=− 2
+ 2
dφ (φ − 1) φ
 2 
d L(φ) n − n/φ n nφ − n (φ − 1)
−1
I (φ) = −E 2
= 2
− 2 = 2
∝ φ3 − φ2
dφ (φ − 1) φ φ (φ − 1)
−1/2
⇒ J (φ) ∝ φ3 − φ2

• Using the result

  from a. we can find the Jeffrey’s prior for φ as:
θ−1/2 (1 − θ)−1/2 φ12 ∝
 dθ  1
J(φ) = J (θ)  dφ  = B(1/2,1/2)
 −1/2 −1/2
θ−1/2 (1 − θ)−1/2 φ12 = φ1/2 1 − φ1 1
φ2
= φ−3/2 (φ−1)
φ−1/2
=
−1/2
φ−1 (φ − 1)−1/2 = φ3 − φ2


Loukia Meligkotsidou, University of Athens Bayesian Inference
Exercise 2.1
In each of the following cases, derive the posterior distribution:
a. x1 , x2 , . . . xn are a random sample from the distribution with
probability function
f (x | θ) θx−1 (1 − θ) ; x = 1, 2, . . .
with the Beta (p, q) prior distribution
θp−1 (1 − θ)q−1
f (θ) = , 0 ≤ θ ≤ 1.
B (p, q)
b. x1 , x2 , . . . xn are a random sample from the distribution with
probability density function
e −θ θx
f (x | θ) = , x = 0, 1, . . .
x!
with the prior distribution
f (θ) = e −θ , θ ≥ 0.
Loukia Meligkotsidou, University of Athens Bayesian Inference
Exercise 2.1-Solution

a. We know that f (θ | x) ∝ f (θ) f (x | θ).

The likelihood
Qn function is: Pn
f (x | θ) = i=1 θ xi −1 (1 − θ) = (1 − θ)n θ i=1 xi −n .

Loukia Meligkotsidou, University of Athens Bayesian Inference

Exercise 2.1-Solution

a. We know that f (θ | x) ∝ f (θ) f (x | θ).

The likelihood
Qn function is: Pn
f (x | θ) = i=1 θ xi −1 (1 − θ) = (1 − θ)n θ i=1 xi −n .
The posterior distribution is:
Pn
f (θ|x) ∝ θp−1 (1 − θ)q−1 (1 − θ)n θ−n θ i=1 xi
Pn
∝θ i=1 xi +p−n−1 (1 − θ)q+n−1 ≡ Beta (P, Q)

Loukia Meligkotsidou, University of Athens Bayesian Inference

Exercise 2.1-Solution

a. We know that f (θ | x) ∝ f (θ) f (x | θ).

The likelihood
Qn function is: Pn
f (x | θ) = i=1 θ xi −1 (1 − θ) = (1 − θ)n θ i=1 xi −n .
The posterior distribution is:
Pn
f (θ|x) ∝ θp−1 (1 − θ)q−1 (1 − θ)n θ−n θ i=1 xi
Pn
∝θ i=1 xi +p−n−1 (1 − θ)q+n−1 ≡ Beta (P, Q)

b. Likewise, Pn
xi
e −θ θxi e −nθ
f (x | θ) = ni=1 Qnθ i=1
Q
xi ! = i=1 xi !
and
Pn Pn
f (θ|x) ∝ e −nθ θ i=1 xi e −θ = e −(n+1)θ θ i=1 xi ≡ Gamma(p, q).

Note:Y ∼ Beta(p, q) ⇔ f (y ) = 1
B(p,q)
y p−1 (1 − θ)q−1 and
q
p
Y ∼ Gamma (p, q) ⇔ f (y ) = Γ(p)
e −qy y p−1 .

Loukia Meligkotsidou, University of Athens Bayesian Inference

Exercise 2.2

The proportion, θ, of defective items in a large shipment is

unknown, but the expert assessment assigns θ of the Beta(2, 200)
prior distribution. If 100 items are selected at random from the
shipment, and 3 are found to be defective, what is the posterior
distribution of θ?

If another statistician, having observed the 3 defectives, calculated

the posterior distribution as being a beta distribution with mean
4/102 and variance 0.0003658, then what prior distribution had
she used?

Loukia Meligkotsidou, University of Athens Bayesian Inference

Exercise 2.2-Solution

Prior distribution: f (θ) ∝ θ (1 − θ)199 , 0 ≤ θ ≤ 1.

Loukia Meligkotsidou, University of Athens Bayesian Inference

Exercise 2.2-Solution

Prior distribution: f (θ) ∝ θ (1 − θ)199 , 0 ≤ θ ≤ 1.

Likelihood:
100
θ3 (1 − θ)97 ∝ θ3 (1 − θ)97 , 0 ≤ θ ≤ 1.


f (x | θ) = 3

Loukia Meligkotsidou, University of Athens Bayesian Inference

Exercise 2.2-Solution

Prior distribution: f (θ) ∝ θ (1 − θ)199 , 0 ≤ θ ≤ 1.

Likelihood:
100
θ3 (1 − θ)97 ∝ θ3 (1 − θ)97 , 0 ≤ θ ≤ 1.


f (x | θ) = 3

Posterior: f (θ | x) ∝ θ3 (1 − θ)97 θ (1 − θ)199 = θ4 (1 − θ)296 ≡

Beta(5, 297).

Loukia Meligkotsidou, University of Athens Bayesian Inference

Exercise 2.2-Solution

Prior distribution: f (θ) ∝ θ (1 − θ)199 , 0 ≤ θ ≤ 1.

Likelihood:
100
θ3 (1 − θ)97 ∝ θ3 (1 − θ)97 , 0 ≤ θ ≤ 1.


f (x | θ) = 3

Posterior: f (θ | x) ∝ θ3 (1 − θ)97 θ (1 − θ)199 = θ4 (1 − θ)296 ≡

Beta(5, 297).

P
Let θ | x ∼ Beta (P, Q) ⇔ E (θ | x) = P+Q = 4/102 and V (θ |
x) = (P+Q)2PQ
(P+Q+1)
= 0.0003658
After some straight forward algebra: P = 4 and Q = 98.
Also, P = p + x =⇒ 4 = p + 3 =⇒ p = 1 and
Q = q + n − x =⇒ 98 = q + 3 − 100 =⇒ q = 1.
=⇒ The statistician used Beta(1, 1) ≡ U(0, 1) prior distribution.

Loukia Meligkotsidou, University of Athens Bayesian Inference

Exercise 2.3

The diameter of a component from a long production run varies

according to a Normal (θ, 1) distribution. An engineer specifies
that the prior distribution for θ is Normal (10, 0.25). In one
production run 12 components are sampled and found to have a
sample mean diameter of 31/3. Use this information to find the
posterior distribution of mean component diameter. Hence
calculate the probability that this is more than 10 units.

Loukia Meligkotsidou, University of Athens Bayesian Inference

Exercise 2.3

The diameter of a component from a long production run varies

according to a Normal (θ, 1) distribution. An engineer specifies
that the prior distribution for θ is Normal (10, 0.25). In one
production run 12 components are sampled and found to have a
sample mean diameter of 31/3. Use this information to find the
posterior distribution of mean component diameter. Hence
calculate the probability that this is more than 10 units.

Solution. 2
−θ)
For known σ: f (xi | θ) = √ 1 exp{− (xi2σ 2 } ∝
2πσ 2
θ2
xi θ
exp{− 2σ1 2 xi2 + σ2
2σ 2
} ∝ exp{− 2σ1 2 θ2 + σ12 xi θ}
−
Likelihood: f (x | θ) = ni=1 f (xi | θ) ∝
Q
Qn θ2 1 n 2 θ Pn
i=1 exp{− 2σ 2 + σ 2 xi θ} = exp{− 2σ 2 θ + σ 2 i=1 xi }

Loukia Meligkotsidou, University of Athens Bayesian Inference

Exercise 2.3-Solution

Prior distribution:
f (θ) ∝ exp{− 2d1 2 (θ − b)2 } ∝ exp{− 2d1 2 θ2 + 1
d2
bθ}

Loukia Meligkotsidou, University of Athens Bayesian Inference

Exercise 2.3-Solution

Prior distribution:
f (θ) ∝ exp{− 2d1 2 (θ − b)2 } ∝ exp{− 2d1 2 θ2 + 1
d2
bθ}
Posterior distribution:
f (θ | x) ∝ f (θ) f
(x | θ) ∝ Pn

exp − 2σn 2 − 2d1 2 θ2 + d12 b + 1
σ2 i=1 θ =
exp − 2D1 2 θ2 + DB2 θ


Loukia Meligkotsidou, University of Athens Bayesian Inference

Exercise 2.3-Solution

Prior distribution:
f (θ) ∝ exp{− 2d1 2 (θ − b)2 } ∝ exp{− 2d1 2 θ2 + 1
d2
bθ}
Posterior distribution:
f (θ | x) ∝ f (θ) f
(x | θ) ∝ Pn

exp − 2σn 2 − 2d1 2 θ2 + d12 b + 1
σ2 i=1 θ =
exp − 2D1 2 θ2 + DB2 θ


In our problem: Pn
xi
d 2 = 0.25, b = 10, i=1
n = 31
3
Substituting, B = 10.25 and D = 16−1 = 0.0625
R∞
Calculate P(θ > 10) = 10 f (θ | x) dθ.

Loukia Meligkotsidou, University of Athens Bayesian Inference

Exercise 2.4

The number of defects in a single roll of magnetic tape has a

Poisson (θ) distribution. The prior distribution for θ is Γ(3, 1).
When 5 rolls of this tape are selected at random, the number of
defects found on each are 2,2,6,0 and 3 respectively. Determine
the posterior distribution of θ.

Solution.
Prior:
q p p−1 −qθ
f (θ) = Γ(p) θ e

Loukia Meligkotsidou, University of Athens Bayesian Inference

Exercise 2.4

The number of defects in a single roll of magnetic tape has a

Poisson (θ) distribution. The prior distribution for θ is Γ(3, 1).
When 5 rolls of this tape are selected at random, the number of
defects found on each are 2,2,6,0 and 3 respectively. Determine
the posterior distribution of θ.

Solution.
Prior:
q p p−1 −qθ
f (θ) = Γ(p) θ e
Likelihood: Q
x −θ
f (x | θ) = ni=1 θ ixei !

Loukia Meligkotsidou, University of Athens Bayesian Inference

Exercise 2.4

The number of defects in a single roll of magnetic tape has a

Poisson (θ) distribution. The prior distribution for θ is Γ(3, 1).
When 5 rolls of this tape are selected at random, the number of
defects found on each are 2,2,6,0 and 3 respectively. Determine
the posterior distribution of θ.

Solution.
Prior:
q p p−1 −qθ
f (θ) = Γ(p) θ e
Likelihood: Q
x −θ
f (x | θ) = ni=1 θ ixei !
Posterior:
−qθ e −nθ θ ni=1 xi = e −(q+n)θ θ ni=1 xi +p−1 ≡
P P
f (θ | x) ∝ θp−1 e
Gamma (p + ni=1 xi , q + n) = Gamma(16, 6)
P

Loukia Meligkotsidou, University of Athens Bayesian Inference

Exercise 2.8

(i) Observations y1 , y2 , . . . , yn are obtained from independent

random variables which are normally distributed, each with
the same (known) variance σ 2 but with respective means
x1 θ, x2 θ, . . . , xn θ. The values of x1 , x2 , . . . , xn are known but θ
is unknown. Show that the likelihood, given a single
observation yi is of the form
 
1 2 2 1
f (yi | θ) ∝ exp − 2 xi θ + 2 yi xi θ
2σ σ

(ii) Given the prior distribution for the unknown coefficient θ may
be described as normal with mean b and variance σ 2 /α2 ,
show that the posterior distribution of θ is proportional to
n n
( " ! # " ! # )
1 X X
exp − a2 + xi2 /σ 2 θ2 + a2 b + yi xi /σ 2 θ
2
i=1 i=1

Loukia Meligkotsidou, University of Athens Bayesian Inference

Exercise 2.8

(iii) Use this to write down the posterior mean of θ. Show that it
may be written as
Pn
yi xi
θ̂ = wb + (1 − w ) Pi=1
n n
i=1 xi

and obtain an expression for w.

Loukia Meligkotsidou, University of Athens Bayesian Inference

Exercise 2.8

(iii) Use this to write down the posterior mean of θ. Show that it
may be written as
Pn
yi xi
θ̂ = wb + (1 − w ) Pi=1
n n
i=1 xi

and obtain an expression for w.

Solution. n o
(i) f (yi | θ) = √ 1 2 exp − 2σ1 2 (yi − xi θ)2 ∝
2πσ
exp − 2σ1 2 yi2 + xi2 θ2 − 2xi yi θ ∝ exp − 2σ1 2 xi2 θ2 − 2xi yi θ



(ii)Prior: n o n o
a2 2 a2 2 a2
f (θ) = √ 1 2 exp − 2σ 2 (θ − b) ∝ exp − 2σ 2
θ + σ2
bθ
2πσ

Loukia Meligkotsidou, University of Athens Bayesian Inference

Exercise 2.8-Solution

Likelihood: Q
f (y | θ) = ni=1 f (yi | θ) ∝ ni=1 exp − 2σ1 2 xi2 θ2 + 1
Q 
yxθ
σ2 i i
=
n o
θ2 Pn 2+ θ
Pn
exp − 2σ 2 x
i=1 i σ i=1 xi yi

Posterior: Pn Pn
xi2
n  2    o
2 + a2 b + i=1 xi yi
f (θ | y ) ∝ exp − 21 σa 2 + i=1 σ 2 θ σ 2 σ 2 θ ≡
 2 Pn 2
−1
xi
N(B, D) where D = σa 2 + i=1 σ2
and
xi2 −1 a2
Pn n
a2 b+ ni=1 xi yi
 2   P  P
i=1 xi yi
B = σa 2 + i=1 σ2 σ2
b + σ2
= a2 + n x 2
P =
Pn i=1 i
yi xi
wb + (1 − w ) Pi=1
n n
i=1 xi
a 2
where w = a2 + ni=1 xi2
P

Loukia Meligkotsidou, University of Athens Bayesian Inference

Exercise 3.1
Which of the following densities belong to the exponential family
f1 (x | θ) = θ2θ x −(θ+1) for x > 2
f2 (x | x) = θx θ−1 exp{−x θ } for x > 0
In each case calculate the conjugate prior if the density belongs to
the exponential family.

Solution.
A density belongs to Exponential Family of distributions if it can
be written in the form of f (x | θ) = h (x) g (θ) exp {t(x)c(θ)}

Loukia Meligkotsidou, University of Athens Bayesian Inference

Exercise 3.1
Which of the following densities belong to the exponential family
f1 (x | θ) = θ2θ x −(θ+1) for x > 2
f2 (x | x) = θx θ−1 exp{−x θ } for x > 0
In each case calculate the conjugate prior if the density belongs to
the exponential family.

Solution.
A density belongs to Exponential Family of distributions if it can
be written in the form of f (x | θ) = h (x) g (θ) exp {t(x)c(θ)}
f1 (x | θ) = θ2θ exp {− (θ + 1) log x}
Exponential family with:
h(x) = 1
g (θ) = θ2θ
t(x) = log x
c(θ) = − (θ + 1)
Loukia Meligkotsidou, University of Athens Bayesian Inference
Exercise 3.1-Solution

In this case, we choose prior of the form

f (θ) ∝ (g (θ))d exp {bc (θ)}, hence
f (θ) ∝ θd 2dθ exp {− (θ + 1) b} = θd exp {dθ log 2 − (θ + 1) b} =
θd exp {(dθ log 2 − b) θ} ≡ Gamma(α, β)
with α = d + 1 and β = b − d log 2

Loukia Meligkotsidou, University of Athens Bayesian Inference

Exercise 3.1-Solution

In this case, we choose prior of the form

f (θ) ∝ (g (θ))d exp {bc (θ)}, hence
f (θ) ∝ θd 2dθ exp {− (θ + 1) b} = θd exp {dθ log 2 − (θ + 1) b} =
θd exp {(dθ log 2 − b) θ} ≡ Gamma(α, β)
with α = d + 1 and β = b − d log 2

f2 (x | x) = θ exp (θ − 1) log x − x θ


⇓
Does not belong to the Exponential family.

Loukia Meligkotsidou, University of Athens Bayesian Inference

Exercise 3.2
Find the Jeffreys prior for θ in the geometric model:
f (x | θ) = (1 − θ)x−1 θ x = 1, 2, . . .
Note: E (X ) = 1/θ.

Loukia Meligkotsidou, University of Athens Bayesian Inference

Exercise 3.2
Find the Jeffreys prior for θ in the geometric model:
f (x | θ) = (1 − θ)x−1 θ x = 1, 2, . . .
Note: E (X ) = 1/θ.

Solution.
J (θ) ∝ |I (θ)|1/2 , 
  2 
d 2 L(θ) dL(θ)
where I (θ) = −E dθ2
=E dθ

Loukia Meligkotsidou, University of Athens Bayesian Inference

Exercise 3.2
Find the Jeffreys prior for θ in the geometric model:
f (x | θ) = (1 − θ)x−1 θ x = 1, 2, . . .
Note: E (X ) = 1/θ.

Solution.
J (θ) ∝ |I (θ)|1/2 , 
  2 
d 2 L(θ) dL(θ)
where I (θ) = −E =E dθ2
and L (θ) =
dθ
h i
log f (x | θ) = log (1 − θ)x−1 θ = (x − 1) log (1 − θ) + log θ

Loukia Meligkotsidou, University of Athens Bayesian Inference

Exercise 3.2
Find the Jeffreys prior for θ in the geometric model:
f (x | θ) = (1 − θ)x−1 θ x = 1, 2, . . .
Note: E (X ) = 1/θ.

Solution.
J (θ) ∝ |I (θ)|1/2 , 
  2 
d 2 L(θ) dL(θ)
where I (θ) = −E =E dθ2
and L (θ) = dθ
h i
log f (x | θ) = log (1 − θ)x−1 θ = (x − 1) log (1 − θ) + log θ
dL(θ) 1 (x−1)
dθ = θ − (1−θ)

Loukia Meligkotsidou, University of Athens Bayesian Inference

Exercise 3.2
Find the Jeffreys prior for θ in the geometric model:
f (x | θ) = (1 − θ)x−1 θ x = 1, 2, . . .
Note: E (X ) = 1/θ.

Solution.
J (θ) ∝ |I (θ)|1/2 , 
  2 
d 2 L(θ) dL(θ)
where I (θ) = −E =E dθ2
and L (θ) =
dθ
h i
log f (x | θ) = log (1 − θ)x−1 θ = (x − 1) log (1 − θ) + log θ
dL(θ) 1 (x−1)
dθ = θ − (1−θ)
d 2 L(θ) (x−1)
dθ2
= − θ12 − (1−θ)2

Loukia Meligkotsidou, University of Athens Bayesian Inference

Exercise 3.2
Find the Jeffreys prior for θ in the geometric model:
f (x | θ) = (1 − θ)x−1 θ x = 1, 2, . . .
Note: E (X ) = 1/θ.

Solution.
J (θ) ∝ |I (θ)|1/2 , 
  2 
d 2 L(θ) dL(θ)
where I (θ) = −E =E dθ2
and L (θ) =
dθ
h i
log f (x | θ) = log (1 − θ)x−1 θ = (x − 1) log (1 − θ) + log θ
dL(θ) 1 (x−1)
dθ = θ − (1−θ)
d 2 L(θ) (x−1)
dθ2
= − θ12 − (1−θ)2
  1
1 (x−1) 1 −1 1
I (θ) = E θ2
+ (1−θ)2 = θ 2 +
θ
(1−θ)2
= θ2 (1−θ)
⇒ Jeffrey’s prior
∝ 1
= (1 − θ)−1/2 θ−1 .
θ(1−θ)1/2
Loukia Meligkotsidou, University of Athens Bayesian Inference
Exercise 3.3

Suppose x has the Pareto distribution Pareto(a, b), where a is

known but b is unknown. So,

f (x|b) = bab x −b−1 ; (x > a, b > 0).

Find the Jeffreys prior and the corresponding posterior distribution

for b.

Loukia Meligkotsidou, University of Athens Bayesian Inference

Exercise 3.3

Suppose x has the Pareto distribution Pareto(a, b), where a is

known but b is unknown. So,

f (x|b) = bab x −b−1 ; (x > a, b > 0).

Find the Jeffreys prior and the corresponding posterior distribution

for b.

Solution.

log-likelihood: `(b) = log b + b log a − (b + 1) log x

d` 1
= + log a − log x
db b

Loukia Meligkotsidou, University of Athens Bayesian Inference

Solution

d` 1
= + log a − log x
db b
2 d 2`
 
d ` 1 1
2
=− 2 ⇒E − 2 = 2
db b db b

Loukia Meligkotsidou, University of Athens Bayesian Inference

Solution

d` 1
= + log a − log x
db b
2 d 2`
 
d ` 1 1
2
=− 2 ⇒E − 2 = 2
db b db b
 2 
d `
Jeffreys’ prior: J(b) ∝ |E − db 2 |1/2 = b1

Loukia Meligkotsidou, University of Athens Bayesian Inference

Solution

d` 1
= + log a − log x
db b
2 d 2`
 
d ` 1 1
2
=− 2 ⇒E − 2 = 2
db b db b
 2 
d `
Jeffreys’ prior: J(b) ∝ |E − db 2 |1/2 = b1
Posterior:
1
f (b | x) ∝ f (x|b)J(b) ∝ bab x −b−1 = ab x −b−1
b
 a b x
∝ ab x −b = = exp{−b log( )}.
x a
The posterior distribution is exponential with rate log( xa ).

Loukia Meligkotsidou, University of Athens Bayesian Inference

Exercise 3.5

You are interested in estimating θ, the probability that a drawing

pin will land point up. Your prior belief can be described by a
mixture of Beta distributions:
Γ(a + b) a−1 Γ(p + q) p−1
f (θ) = θ (1 − θ)b−1 + θ (1 − θ)q−1 .
2Γ(a)Γ(b) 2Γ(p)Γ(q)

You throw a drawing pin n independent times, and observe x

occasions on which the pin lands point up. Calculate the posterior
distribution for θ.

Loukia Meligkotsidou, University of Athens Bayesian Inference

Solution

Let X denote the number of times that the drawing pin lands
point up.
X ∼ Binomial(n, θ)

Likelihood: f (x | θ) ∝ θx (1 − θ)n−x

Loukia Meligkotsidou, University of Athens Bayesian Inference

Solution

Let X denote the number of times that the drawing pin lands
point up.
X ∼ Binomial(n, θ)

Likelihood: f (x | θ) ∝ θx (1 − θ)n−x

Posterior: f (θ | x) ∝ f (x | θ)f (θ)

Γ(a+b) x+a−1 Γ(p+q) x+p−1
∝ 2Γ(a)Γ(b) θ (1 − θ)n−x+b−1 + 2Γ(p)Γ(q) θ (1 − θ)n−x+q−1

Loukia Meligkotsidou, University of Athens Bayesian Inference

Solution

Let X denote the number of times that the drawing pin lands
point up.
X ∼ Binomial(n, θ)

Likelihood: f (x | θ) ∝ θx (1 − θ)n−x

Posterior: f (θ | x) ∝ f (x | θ)f (θ)

Γ(a+b) x+a−1 Γ(p+q) x+p−1
∝ 2Γ(a)Γ(b) θ (1 − θ)n−x+b−1 + 2Γ(p)Γ(q) θ (1 − θ)n−x+q−1
n o
Γ(a+b) Γ(x+a)Γ(n−x+b) Γ(n+a+b) x+a−1 (1
= 2Γ(a)Γ(b) Γ(n+a+b) Γ(x+a)Γ(n−x+b) θ − θ)n−x+b−1
n o
Γ(p+q) Γ(x+p)Γ(n−x+q) Γ(n+p+q) x+p−1 (1
+ 2Γ(p)Γ(q) Γ(n+p+q) Γ(x+p)Γ(n−x+q) θ − θ)n−x+q−1

Loukia Meligkotsidou, University of Athens Bayesian Inference

Solution

Γ(a+b) Γ(x+a)Γ(n−x+b) Γ(p+q) Γ(x+p)Γ(n−x+q)

Let α = 2Γ(a)Γ(b) Γ(n+a+b) and β = 2Γ(p)Γ(q) Γ(n+p+q) .

Loukia Meligkotsidou, University of Athens Bayesian Inference

Solution

Γ(a+b) Γ(x+a)Γ(n−x+b) Γ(p+q) Γ(x+p)Γ(n−x+q)

Let α = 2Γ(a)Γ(b) Γ(n+a+b) and β = 2Γ(p)Γ(q) Γ(n+p+q) .

Posterior:
 
α Γ(A + B) A−1
f (θ | x) = θ (1 − θ)B−1
α + β Γ(A)Γ(B)
 
β Γ(P + Q) P−1
+ θ (1 − θ)Q−1 ,
α + β Γ(P)Γ(Q)

where A = x + a, B = b + n − x, P = p + x and Q = q + n − x.

Loukia Meligkotsidou, University of Athens Bayesian Inference

Solution

Γ(a+b) Γ(x+a)Γ(n−x+b) Γ(p+q) Γ(x+p)Γ(n−x+q)

Let α = 2Γ(a)Γ(b) Γ(n+a+b) and β = 2Γ(p)Γ(q) Γ(n+p+q) .

Posterior:
 
α Γ(A + B) A−1
f (θ | x) = θ (1 − θ)B−1
α + β Γ(A)Γ(B)
 
β Γ(P + Q) P−1
+ θ (1 − θ)Q−1 ,
α + β Γ(P)Γ(Q)

where A = x + a, B = b + n − x, P = p + x and Q = q + n − x.

That is the posterior distribution of θ is a mixture of Beta

distributions with updated parameters and updated mixing
proportions.

Loukia Meligkotsidou, University of Athens Bayesian Inference

Exercise 4.3

(a) Observations x1 and x2 are obtained of random variables X1

and X2 , having Poisson distributions with respective means θ
and φθ, where φ is a known positive coefficient. Show, by
evaluating the posterior density of θ, that the Gamma (p, q)
family of prior distributions of θ is a conjugate or this data
mode.
(b) Now suppose that φ is also an unknown parameter with prior
density f (φ) = 1/ (1 + φ)2 , and independent of θ. Obtain the
joint posterior distribution of θ and φ and show that the
marginal posterior distribution of φ is proportional to
φx2
(1 + φ)2 (1 + φ + q)x1 +x2 +p

Loukia Meligkotsidou, University of Athens Bayesian Inference

Exercise 4.3-Solution

x
e −θ θx1 e −θφ (θφ) 2
(a) f (x1 , x2 | θ) = x1 ! x2 ! ∝ e −θ(1+φ) θx1 +x2

Loukia Meligkotsidou, University of Athens Bayesian Inference

Exercise 4.3-Solution

x
e −θ θx1 e −θφ (θφ) 2
(a) f (x1 , x2 | θ) = x1 ! x2 ! ∝ e −θ(1+φ) θx1 +x2
Prior:
q p p−1 −qθ
f (θ) = Γ(p) θ e ∝ θp−1 e −qθ ≡ Gamma(p, q)

Loukia Meligkotsidou, University of Athens Bayesian Inference

Exercise 4.3-Solution

x
e −θ θx1 e −θφ (θφ) 2
(a) f (x1 , x2 | θ) = x1 ! x2 ! ∝ e −θ(1+φ) θx1 +x2
Prior:
q p p−1 −qθ
f (θ) = Γ(p) θ e ∝ θp−1 e −qθ ≡ Gamma(p, q)
Posterior:
f (θ | x1 , x2 ) = θx1 +x2 +p−1 e −(1+φ+q)θ ≡
Gamma(x1 + x2 + p, 1 + φ + q) =⇒ Gamma family is
conjugate for this model.
Posterior Mean = x1+φ+q 1 +x2 +p

Loukia Meligkotsidou, University of Athens Bayesian Inference

Exercise 4.3-Solution

x
e −θ θx1 e −θφ (θφ) 2
(a) f (x1 , x2 | θ) = x1 ! x2 ! ∝ e −θ(1+φ) θx1 +x2
Prior:
q p p−1 −qθ
f (θ) = Γ(p) θ e ∝ θp−1 e −qθ ≡ Gamma(p, q)
Posterior:
f (θ | x1 , x2 ) = θx1 +x2 +p−1 e −(1+φ+q)θ ≡
Gamma(x1 + x2 + p, 1 + φ + q) =⇒ Gamma family is
conjugate for this model.
Posterior Mean = x1+φ+q 1 +x2 +p

(b) Joint Posterior:

f (φ, θ | x1 , x2 ) ∝ f (φ) f (θ) f (x1 , x2 | φ, θ) ∝
1
(1+φ)2
θp−1 e −qθ e −θ(1+φ) θx1 θx2 φx2 =
1
(1+φ)2
θp+x1 +x2 −1 φx2 e −(1+φ+1)θ

Loukia Meligkotsidou, University of Athens Bayesian Inference

Exercise 4.3-Solution

Marginal Posterior:

Z
f (φ, x1 , x2 ) = f (φ, θ | x1 , x2 ) dθ
∞
φx2
Z
∝ θp+x1 +x2 −1 φx2 e −(1+φ+1)θ dθ
0 (1 + φ)2
φx2 Γ(p + x1 + x2 )
=
(1 + φ) (1 + φ + q)x1 +x2 +p
2

φx2 1
∝ .
(1 + φ) (1 + φ + q)x1 +x2 +p
2

Loukia Meligkotsidou, University of Athens Bayesian Inference

Exercise 5.1

I have been offered an objet d’art at what seems a bargain price of

£100. If it is not genuine, then it is worth nothing. If it is genuine
I believe I can sell it immediately for £300. I believe there is a 0.5
chance that the object is genuine. Should I buy the object?

An art ‘expert’ has undergone a test of her reliability in which she

has separately pronounced judgement — ‘genuine’ or ‘counterfeit’
— on a large number of art subjects of know origin. From these it
appears that she has probability 0.8 of detecting a counterfeit and
probability 0.7 of recognising a genuine object. The expert charges
£30 for her services. Is it to my advantage to pay her for an
assessment?

Loukia Meligkotsidou, University of Athens Bayesian Inference

Solution

1. The parameter space is Θ = {θ1 , θ2 }, where θ1 and θ2

correspond to the object being genuine and counterfeit
respectively;
2. The set of actions is A = {a1 , a2 } where a1 and a2 correspond
to buying and not buying the object respectively;
3. The loss function is
L(θ, a) θ1 θ2
a1 -200 100
a2 0 0

Loukia Meligkotsidou, University of Athens Bayesian Inference

Solution

1. The parameter space is Θ = {θ1 , θ2 }, where θ1 and θ2

correspond to the object being genuine and counterfeit
respectively;
2. The set of actions is A = {a1 , a2 } where a1 and a2 correspond
to buying and not buying the object respectively;
3. The loss function is
L(θ, a) θ1 θ2
a1 -200 100
a2 0 0

The decision strategy is to evaluate the expected loss for each

action and choose the action which has the minimum expected
loss.

Loukia Meligkotsidou, University of Athens Bayesian Inference

Solution

The expected loss is first calculated based on the prior distribution

of θ: f (θ1 ) = 0.5 and f (θ2 ) = 0.5.

Loukia Meligkotsidou, University of Athens Bayesian Inference

Solution

The expected loss is first calculated based on the prior distribution

of θ: f (θ1 ) = 0.5 and f (θ2 ) = 0.5.

f (θ) 0.5 0.5

L(θ, a) θ1 θ2 E [L(θ, a)]
a1 -200 100 0.5 × (−200) + 0.5 × 100 = −50
a2 0 0 0.5 × 0 + 0.5 × 0 = 0

Loukia Meligkotsidou, University of Athens Bayesian Inference

Solution

The expected loss is first calculated based on the prior distribution

of θ: f (θ1 ) = 0.5 and f (θ2 ) = 0.5.

f (θ) 0.5 0.5

L(θ, a) θ1 θ2 E [L(θ, a)]
a1 -200 100 0.5 × (−200) + 0.5 × 100 = −50
a2 0 0 0.5 × 0 + 0.5 × 0 = 0

So, under the prior, the best action is to buy (according to

minimisation of expected loss - here maximization of expected
profit). The expected profit of this action is £50.

Loukia Meligkotsidou, University of Athens Bayesian Inference

Solution

The expected loss is first calculated based on the prior distribution

of θ: f (θ1 ) = 0.5 and f (θ2 ) = 0.5.

f (θ) 0.5 0.5

L(θ, a) θ1 θ2 E [L(θ, a)]
a1 -200 100 0.5 × (−200) + 0.5 × 100 = −50
a2 0 0 0.5 × 0 + 0.5 × 0 = 0

So, under the prior, the best action is to buy (according to

minimisation of expected loss - here maximization of expected
profit). The expected profit of this action is £50.

Suppose I pay the expert. Her possible conclusions about the

object (observations) are x1 = ’says genuine’ and
x2 = ’says counterfeit’.

Loukia Meligkotsidou, University of Athens Bayesian Inference

Solution

θ1 θ2
f (x1 |θ) 0.70 0.20
Likelihoods f (x2 |θ) 0.30 0.80

Loukia Meligkotsidou, University of Athens Bayesian Inference

Solution

θ1 θ2
f (x1 |θ) 0.70 0.20
Likelihoods f (x2 |θ) 0.30 0.80
Prior f (θ) 0.50 0.50

Loukia Meligkotsidou, University of Athens Bayesian Inference

Solution

θ1 θ2
f (x1 |θ) 0.70 0.20
Likelihoods f (x2 |θ) 0.30 0.80
Prior f (θ) 0.50 0.50
f (x1 , θ) 0.35 0.10 0.45 f (x1 )
Joints f (x2 , θ) 0.15 0.40 0.55 f (x2 )

Loukia Meligkotsidou, University of Athens Bayesian Inference

Solution

θ1 θ2
f (x1 |θ) 0.70 0.20
Likelihoods f (x2 |θ) 0.30 0.80
Prior f (θ) 0.50 0.50
f (x1 , θ) 0.35 0.10 0.45 f (x1 )
Joints f (x2 , θ) 0.15 0.40 0.55 f (x2 )
a1 a2
f (θ|x1 ) 7/9 2/9 -1200/9 0 Expected
Posteriors f (θ|x2 ) 3/11 8/11 200/11 0 Losses

E (L(θ, a1 ) | x1 ) = f (θ1 | x1 )L(θ1 , a1 ) + f (θ2 | x1 )L(θ2 , a1 ) =

7 2 1200
9 × (−200) + 9 × 100 = − 9
E (L(θ, a1 ) | x2 ) = f (θ1 | x2 )L(θ1 , a1 ) + f (θ2 | x2 )L(θ2 , a1 ) =
3 8 200
11 × (−200) + 11 × 100 = 11
E (L(θ, a2 ) | x1 ) = E (L(θ, a2 ) | x2 ) = 0
Loukia Meligkotsidou, University of Athens Bayesian Inference
Solution

Bayes Decision Rule: d(x1 ) = a1 , d(x2 ) = a2 .

Loukia Meligkotsidou, University of Athens Bayesian Inference

Solution

Bayes Decision Rule: d(x1 ) = a1 , d(x2 ) = a2 .

Bayes Risk:
1200
BR(d) = ρ(d(x1 ), x1 )f (x1 )+ρ(d(x2 ), x2 )f (x2 ) = − ×0.45+0×0.55 = −60
9

Loukia Meligkotsidou, University of Athens Bayesian Inference

Solution

Bayes Decision Rule: d(x1 ) = a1 , d(x2 ) = a2 .

Bayes Risk:
1200
BR(d) = ρ(d(x1 ), x1 )f (x1 )+ρ(d(x2 ), x2 )f (x2 ) = − ×0.45+0×0.55 = −60
9

That is the profit associated with our decision rule is £60. The
gain of £10 does not worth the £30 cost of the expert’s services.

Loukia Meligkotsidou, University of Athens Bayesian Inference

Exercise 5.2

Consider a decision problem with two actions, α1 ana α2 and a loss

function which depends on a parameter θ, with 0 ≤ θ ≤ 1. The
loss function is
(
0 α = α1 .
L (θ, α) =
2 − 3θ α = α2 .

Assume a Beta (1, 1) prior for θ, and an observation

X ∼ Binomial (n, θ). The posterior distribution is
Beta (x + 1, n − x + 1). Calculate the expected loss of each action
and the Bayes rule.

Loukia Meligkotsidou, University of Athens Bayesian Inference

Exercise 5.2

Consider a decision problem with two actions, α1 ana α2 and a loss

function which depends on a parameter θ, with 0 ≤ θ ≤ 1. The
loss function is
(
0 α = α1 .
L (θ, α) =
2 − 3θ α = α2 .

Assume a Beta (1, 1) prior for θ, and an observation

X ∼ Binomial (n, θ). The posterior distribution is
Beta (x + 1, n − x + 1). Calculate the expected loss of each action
and the Bayes rule.

Loukia Meligkotsidou, University of Athens Bayesian Inference

Exercise 5.2-Solution

Prior:
f (θ) = 1, 0 ≤ θ ≤ 1
Likelihood:

f (x | θ) = nθ θx (1 − θ)n−x

Loukia Meligkotsidou, University of Athens Bayesian Inference

Exercise 5.2-Solution

Prior:
f (θ) = 1, 0 ≤ θ ≤ 1
Likelihood:

f (x | θ) = nθ θx (1 − θ)n−x

Posterior:
f (θ | x) ∝ θx (1 − θ)n−x ≡ Beta(x + 1, n − x + 1)

Loukia Meligkotsidou, University of Athens Bayesian Inference

Exercise 5.2-Solution

Prior:
f (θ) = 1, 0 ≤ θ ≤ 1
Likelihood:

f (x | θ) = nθ θx (1 − θ)n−x

Posterior:
f (θ | x) ∝ θx (1 − θ)n−x ≡ Beta(x + 1, n − x + 1)

Expected loss underR the two actions:

1
E (L (θ, α1 ) | x) = 0 0dθ = 0

Loukia Meligkotsidou, University of Athens Bayesian Inference

Exercise 5.2-Solution

Prior:
f (θ) = 1, 0 ≤ θ ≤ 1
Likelihood:

f (x | θ) = nθ θx (1 − θ)n−x

Posterior:
f (θ | x) ∝ θx (1 − θ)n−x ≡ Beta(x + 1, n − x + 1)

Expected loss underR the two actions:

1
E (L (θ, α1 ) | x) = 0 0dθ = 0
R1
E (L (θ, α2 ) | x) = 0 (2 − 3θ) f (θ | x)dθ = 2 − 3E (θ | x) =
3(x+1)
2 − x+1+n−x+1 = 2 − 3(x+1)
n+2

Loukia Meligkotsidou, University of Athens Bayesian Inference

Exercise 5.2-Solution

Prior:
f (θ) = 1, 0 ≤ θ ≤ 1
Likelihood:

f (x | θ) = nθ θx (1 − θ)n−x

Posterior:
f (θ | x) ∝ θx (1 − θ)n−x ≡ Beta(x + 1, n − x + 1)

Expected loss underR the two actions:

1
E (L (θ, α1 ) | x) = 0 0dθ = 0
R1
E (L (θ, α2 ) | x) = 0 (2 − 3θ) f (θ | x)dθ = 2 − 3E (θ | x) =
3(x+1)
2 − x+1+n−x+1 = 2 − 3(x+1)
n+2

We prefer the action α2 if E (L (θ, α2 ) | x) < E (L (θ, α1 ) | x) ⇒

2 − 3(x+1)
n+2 < 0 ⇒ · · · ⇒ x ≥ 3
2n+1

Loukia Meligkotsidou, University of Athens Bayesian Inference

Exercise 5.3

(a) For a parameter θ with a posterior distribution described by the

Beta(P, Q) distribution, find the posterior mode in terms of P and
Q and compare it with the posterior mean.

Solution.

f (θ | x) ∝ θP−1 (1 − θ)Q−1
log f (θ | x) = (P − 1) log θ + (Q − 1) log(1 − θ) + c

Loukia Meligkotsidou, University of Athens Bayesian Inference

Exercise 5.3

(a) For a parameter θ with a posterior distribution described by the

Beta(P, Q) distribution, find the posterior mode in terms of P and
Q and compare it with the posterior mean.

Solution.

f (θ | x) ∝ θP−1 (1 − θ)Q−1
log f (θ | x) = (P − 1) log θ + (Q − 1) log(1 − θ) + c
d log f (θ|x) Q−1
dθ = 0 ⇒ P−1
θ − 1−θ = 0 ⇒
P−1 Q−1
θ = 1−θ ⇒ P − 1 − (P − 1)θ = (Q − 1)θ ⇒
P−1
(P + Q − 2)θ = P − 1 ⇒ θ = P+Q−2 [posterior mode]

Loukia Meligkotsidou, University of Athens Bayesian Inference

Exercise 5.3

(a) For a parameter θ with a posterior distribution described by the

Beta(P, Q) distribution, find the posterior mode in terms of P and
Q and compare it with the posterior mean.

Solution.

f (θ | x) ∝ θP−1 (1 − θ)Q−1
log f (θ | x) = (P − 1) log θ + (Q − 1) log(1 − θ) + c
d log f (θ|x) Q−1
dθ = 0 ⇒ P−1
θ − 1−θ = 0 ⇒
P−1 Q−1
θ = 1−θ ⇒ P − 1 − (P − 1)θ = (Q − 1)θ ⇒
P−1
(P + Q − 2)θ = P − 1 ⇒ θ = P+Q−2 [posterior mode]
P
Posterior mean: E (θ | x) = P+Q [closer to 1/2 than mode]

Loukia Meligkotsidou, University of Athens Bayesian Inference

Exercise 5.4

The parameter θ has a Beta(3, 2) posterior density. Show that the

interval [5/21, 20/21] is a 94.3% highest posterior density region
for θ.

Solution.

Loukia Meligkotsidou, University of Athens Bayesian Inference

Exercise 5.4

The parameter θ has a Beta(3, 2) posterior density. Show that the

interval [5/21, 20/21] is a 94.3% highest posterior density region
for θ.

Solution.
R region Cα (x) is a 100(1 − α)% credible region for θ if
A
Cα (x)
f (θ | x) dθ = 1 − α. It is an HPD region if Cα (x) = {θ : f (θ | x) ≥ γ}.

Loukia Meligkotsidou, University of Athens Bayesian Inference

Exercise 5.4

The parameter θ has a Beta(3, 2) posterior density. Show that the

interval [5/21, 20/21] is a 94.3% highest posterior density region
for θ.

Solution.
R region Cα (x) is a 100(1 − α)% credible region for θ if
A
Cα (x)
f (θ | x) dθ = 1 − α. It is an HPD region if Cα (x) = {θ : f (θ | x) ≥ γ}.

1
In our case, f (θ | x) = B(3,2) θ2 (1 − θ) = 12θ2 (1 − θ) , θ ∈ [0, 1] .
h ib
2 (1 − θ) dθ = 0.943 ⇔ 12 θ3 − θ4
Rb
a 12θ 3 4 = 0.943 ⇔
a
4b 3 − 3b 4 − 4a3 + 3a4 = 0.943.

Loukia Meligkotsidou, University of Athens Bayesian Inference

Exercise 5.4

The parameter θ has a Beta(3, 2) posterior density. Show that the

interval [5/21, 20/21] is a 94.3% highest posterior density region
for θ.

Solution.
R region Cα (x) is a 100(1 − α)% credible region for θ if
A
Cα (x)
f (θ | x) dθ = 1 − α. It is an HPD region if Cα (x) = {θ : f (θ | x) ≥ γ}.

1
In our case, f (θ | x) = B(3,2) θ2 (1 − θ) = 12θ2 (1 − θ) , θ ∈ [0, 1] .
h ib
2 (1 − θ) dθ = 0.943 ⇔ 12 θ3 − θ4
Rb
a 12θ 3 4 = 0.943 ⇔
a
4b 3 − 3b 4 − 4a3 + 3a4 = 0.943.
Moreover, 12a2 (1 − a) = 12b 2 (1 − b) = γ.

Loukia Meligkotsidou, University of Athens Bayesian Inference

Exercise 5.4

The parameter θ has a Beta(3, 2) posterior density. Show that the

interval [5/21, 20/21] is a 94.3% highest posterior density region
for θ.

Solution.
R region Cα (x) is a 100(1 − α)% credible region for θ if
A
Cα (x)
f (θ | x) dθ = 1 − α. It is an HPD region if Cα (x) = {θ : f (θ | x) ≥ γ}.

1
In our case, f (θ | x) = B(3,2) θ2 (1 − θ) = 12θ2 (1 − θ) , θ ∈ [0, 1] .
h ib
2 (1 − θ) dθ = 0.943 ⇔ 12 θ3 − θ4
Rb
a 12θ 3 4 = 0.943 ⇔
a
4b 3 − 3b 4 − 4a3 + 3a4 = 0.943.
Moreover, 12a2 (1 − a) = 12b 2 (1 − b) = γ.
Solving the two-equation system, we derive a = 5/21 and
b = 20/21.

Loukia Meligkotsidou, University of Athens Bayesian Inference

Exercise 5.6
A parameter θ has a posterior density that is Gamma(1, 1).
Calculate the 95% HPD
√ region for θ. Now consider the
transformation φ = 2θ. Obtain the posterior density of φ and
explain why the highest posterior density region for φ is not
obtained by transforming the interval for θ in the same way.

Loukia Meligkotsidou, University of Athens Bayesian Inference

Exercise 5.6
A parameter θ has a posterior density that is Gamma(1, 1).
Calculate the 95% HPD
√ region for θ. Now consider the
transformation φ = 2θ. Obtain the posterior density of φ and
explain why the highest posterior density region for φ is not
obtained by transforming the interval for θ in the same way.

Solution.
The posterior density of θ is decreasing (Exponential(1)). Thus,
the 95% HPD region is an interval of the form [0,b] satisfying
R b −θ
0 e = 0.95 ⇔ 1 − e −b = 0.95 =⇒ C0.05 (x) = [0, 3] .

Loukia Meligkotsidou, University of Athens Bayesian Inference

Exercise 5.6
A parameter θ has a posterior density that is Gamma(1, 1).
Calculate the 95% HPD
√ region for θ. Now consider the
transformation φ = 2θ. Obtain the posterior density of φ and
explain why the highest posterior density region for φ is not
obtained by transforming the interval for θ in the same way.

Solution.
The posterior density of θ is decreasing (Exponential(1)). Thus,
the 95% HPD region is an interval of the form [0,b] satisfying
R b −θ
0 e = 0.95 ⇔ 1 − e −b = 0.95 =⇒ C0.05 (x) = [0, 3] .
√ 2
Now, we have that φ = 2θ ⇔ θ = φ2 . Posterior density of φ:
 
 dθ  1 2
fφ (φ | x) = fθ (φ)   = φe − 2 φ .
dφ

Loukia Meligkotsidou, University of Athens Bayesian Inference

Exercise 5.6
A parameter θ has a posterior density that is Gamma(1, 1).
Calculate the 95% HPD
√ region for θ. Now consider the
transformation φ = 2θ. Obtain the posterior density of φ and
explain why the highest posterior density region for φ is not
obtained by transforming the interval for θ in the same way.

Solution.
The posterior density of θ is decreasing (Exponential(1)). Thus,
the 95% HPD region is an interval of the form [0,b] satisfying
R b −θ
0 e = 0.95 ⇔ 1 − e −b = 0.95 =⇒ C0.05 (x) = [0, 3] .
√ 2
Now, we have that φ = 2θ ⇔ θ = φ2 . Posterior density of φ:
 
 dθ  1 2
fφ (φ | x) = fθ (φ)   = φe − 2 φ .
dφ
Since the posterior density of φ is not monotonic, the credibility
interval will be of the form [a, b] with a 6= 0. Hence, it is not a
transformation of the credibility interval for θ.
Loukia Meligkotsidou, University of Athens Bayesian Inference
Exercise 5.8

Consider a sample x1 , . . . , xn consisting of independent draws from

a Poisson random variable with mean θ. Consider the hypothesis
test, with Null hypothesis

H0 : θ = 1

against an alternative hypothesis

H1 : θ 6= 1

Assume a prior probability of 0.95 for H0 and a Gamma prior

q p p−1
f (θ) = θ exp{−qθ},
Γ(p)

under H1 .
(a) Calculate the posterior probability of H0 .

Loukia Meligkotsidou, University of Athens Bayesian Inference

Solution
P
Likelihood: f (x | θ) = Q1 θ xi e −nθ
xi !

Loukia Meligkotsidou, University of Athens Bayesian Inference

Solution
P
Likelihood: f (x | θ) = Q1 θ xi e −nθ
xi !

H0 : P(H0 | x) ∝ P(H0 )P(x | H0 )

1
= P(H0 )f (x | θ = 1) = 0.95 Q e −n
xi !

Loukia Meligkotsidou, University of Athens Bayesian Inference

Solution
P
Likelihood: f (x | θ) = Q1 θ xi e −nθ
xi !

H0 : P(H0 | x) ∝ P(H0 )P(x | H0 )

1
= P(H0 )f (x | θ = 1) = 0.95 Q e −n
xi !

H1 : P(H1 | x) ∝ P(H1 )P(x | H1 )

Z
= P(H1 ) f (x | θ)f (θ | H1 )dθ
Z ∞
1 P q p p−1 −qθ
= 0.05 Q θ xi e −nθ θ e dθ
0 xi ! Γ(p)
∞ P
1 qp
Z
= 0.05 Q θ xi +p−1 e −(n+q)θ dθ
xi ! Γ(p) 0
1 q p Γ( xi + p)
P
= 0.05 Q P
xi ! Γ(p) (n + q) xi +p
Loukia Meligkotsidou, University of Athens Bayesian Inference
Exercise 5.8 - Solution

p
P
q Γ( xP
i +p)
Let α = 0.95e −n and β = 0.05 Γ(p) (n+q) xi +p
.

α
Then P(H0 | x) = α+β .

Loukia Meligkotsidou, University of Athens Bayesian Inference

Exercise 5.8 - Solution

p
P
q Γ( xP
i +p)
Let α = 0.95e −n and β = 0.05 Γ(p) (n+q) xi +p
.

α
Then P(H0 | x) = α+β .

(b) Assume n = 10, ni=1 xi = 20, and p = 2q. What is the

P
posterior probability of H0 for each of p = 2, 1, 0.5, 0.1. What
happens to this posterior probability as p → 0?

Loukia Meligkotsidou, University of Athens Bayesian Inference

Exercise 5.8 - Solution

p
P
q Γ( xP
i +p)
Let α = 0.95e −n and β = 0.05 Γ(p) (n+q) xi +p
.

α
Then P(H0 | x) = α+β .

(b) Assume n = 10, ni=1 xi = 20, and p = 2q. What is the

P
posterior probability of H0 for each of p = 2, 1, 0.5, 0.1. What
happens to this posterior probability as p → 0?

For n = 10, ni=1 xi = 20, and p = 2q:

P
p
α = 0.95e −10 and β = 0.05 (p/2) Γ(20+p)
Γ(p) (10+p/2)20+p .

Loukia Meligkotsidou, University of Athens Bayesian Inference

Exercise 6.1
A random sample x1 , . . . , xn is observed from a Poisson(θ)
distribution.The prior on θ is a Gamma(g , h). Show that the
predictive distribution for a future observation y from this
Poisson(θ) distribution is
  y  G
y +G −1 1 1
f (y | x) = 1− y = 0, 1, . . .
G −1 1+H 1+H

What is this distribution?

Loukia Meligkotsidou, University of Athens Bayesian Inference

Exercise 6.1
A random sample x1 , . . . , xn is observed from a Poisson(θ)
distribution.The prior on θ is a Gamma(g , h). Show that the
predictive distribution for a future observation y from this
Poisson(θ) distribution is
  y  G
y +G −1 1 1
f (y | x) = 1− y = 0, 1, . . .
G −1 1+H 1+H

What is this distribution?

Solution R
f (y | x) = f (y | θ) f (x | θ) dθ

Loukia Meligkotsidou, University of Athens Bayesian Inference

Exercise 6.1
A random sample x1 , . . . , xn is observed from a Poisson(θ)
distribution.The prior on θ is a Gamma(g , h). Show that the
predictive distribution for a future observation y from this
Poisson(θ) distribution is
  y  G
y +G −1 1 1
f (y | x) = 1− y = 0, 1, . . .
G −1 1+H 1+H

What is this distribution?

Solution R
f (y | x) = f (y | θ) f (x | θ) dθ
−θ x hg g −1
Posterior: f (θ | x) ∝ ni=1 e xi !θ i
Q
Γ(g ) θ exp {−hθ}

Loukia Meligkotsidou, University of Athens Bayesian Inference

Exercise 6.1
A random sample x1 , . . . , xn is observed from a Poisson(θ)
distribution.The prior on θ is a Gamma(g , h). Show that the
predictive distribution for a future observation y from this
Poisson(θ) distribution is
  y  G
y +G −1 1 1
f (y | x) = 1− y = 0, 1, . . .
G −1 1+H 1+H

What is this distribution?

Solution R
f (y | x) = f (y | θ) f (x | θ) dθ
−θ x hg g −1
Posterior: f (θ | x) ∝ ni=1 e xi !θ i Γ(g
Q
)θ exp {−hθ}
Mariginal likelihood:
R ∞ Qn e −θ θxi hg g −1
f (x) = 0 i=1 xi ! Γ(g ) θ exp {−hθ} dθ =
Pn Pn
e −nθ θ i=1 ∞
i=1 xi +g −1 dθ
R
Q n
xi !Γ(g ) 0 exp {− (n + h) θ} θ
i=1

Loukia Meligkotsidou, University of Athens Bayesian Inference

Exercise 6.1- Solution

Pn Pn
e −nθ θ i=1 Γ i=1 x +g
Pn i
= Q n x +g
i=1 xi !Γ(g ) (n+h) i=1 i

Qn e −θ θxi hg g −1
i=1 xi ! Γ(g ) θ exp {−hθ}
⇒ f (θ | x) = Pn Pn
e −nθQ θ i=1 Γ( i=1 xi +g )
Γ(g ) ni=1 xi ! (n+h) ni=1 xi +g
P

Pn
(n + h) i=1 xi +g Pni=1 xi +g −1
= θ exp {− (h + n) θ}
Γ ( ni=1 xi + g )
P

Loukia Meligkotsidou, University of Athens Bayesian Inference

Exercise 6.1- Solution

Predictive distribution
R
f (y | Px) = f (y | θ)f (θ | x)dθ =
n
(n+h) i=1 xi +g R ∞ e −θ θy
Pn
Pn θ i=1 xi +g −1 exp {− (h + n) θ} =
Γ( i=1 xi +g ) 0 y !

Loukia Meligkotsidou, University of Athens Bayesian Inference

Exercise 6.1- Solution

Predictive distribution
R
f (y | Px) = f (y | θ)f (θ | x)dθ =
n
(n+h) i=1 xi +g R ∞ e −θ θy
Pn
Pn θ i=1 xi +g −1 exp {− (h + n) θ} =
Γ( i=1 xi +g ) 0 y !
Pn
(n+h) i=1 xi +g R ∞
Pn
Pn θ i=1 xi +g +y −1 exp {− (h + n + 1) θ} =
Γ( i=1 xi +g )y ! 0

Loukia Meligkotsidou, University of Athens Bayesian Inference

Exercise 6.1- Solution

Predictive distribution
R
f (y | Px) = f (y | θ)f (θ | x)dθ =
n
(n+h) i=1 xi +g R ∞ e −θ θy
Pn
Pn θ i=1 xi +g −1 exp {− (h + n) θ} =
Γ( i=1 xi +g ) 0 y !
Pn
(n+h) i=1 xi +g R ∞
Pn
Pn θ i=1 xi +g +y −1 exp {− (h + n + 1) θ} =
Γ( i=1 xi +g )y ! 0
Pn Pn
(n+h) i=1 xi +g Γ( i=1P xi +g +y )
Pn n
Γ( i=1 xi +g )y ! (h+n+1) i=1 xi +g +y
=
y  Pni=1 xi +g
( ni=1 xi +g +y −1)!
P 
1 n+h
=
( ni=1 xi +g −1)y ! h+n+1 n+h+1
P

G −1+y

1 y
  
1
 G 
1

G −1 1+H 1 − H+1 ≡ NegativeBinomial 1+H , G

Loukia Meligkotsidou, University of Athens Bayesian Inference

Exercise 6.3

A random sample x1 , . . . , xn is observed from a N(θ, σ 2 )

distribution with σ 2 known, and a normal prior for θ is assumed,
leading to a posterior distribution N(B, D 2 ) for θ. Show that the
predictive distribution for a further observation, y , from the
N(θ, σ 2 ) distribution, is N(B, D 2 + σ 2 ).

Loukia Meligkotsidou, University of Athens Bayesian Inference

Exercise 6.3

A random sample x1 , . . . , xn is observed from a N(θ, σ 2 )

distribution with σ 2 known, and a normal prior for θ is assumed,
leading to a posterior distribution N(B, D 2 ) for θ. Show that the
predictive distribution for a further observation, y , from the
N(θ, σ 2 ) distribution, is N(B, D 2 + σ 2 ).

Solution
√ 1 exp − 2D1 2 (θ − B)2

Posterior: f (θ | x) =
2πD 2
R
Predictive: f (y | x) = f (y | θ)f (θ | x)dθ
R∞
= −∞ √ 1 2 exp − 2σ1 2 (y − θ)2 √ 1 2 exp − 2D1 2 (θ − B)2 dθ
 
2πσ 2πD

Loukia Meligkotsidou, University of Athens Bayesian Inference

Exercise 6.3 - Solution

R∞
√ 1 exp − 2σ1 2 (y − θ)2 √ 1
exp − 2D1 2 (θ − B)2 dθ
 
= −∞ 2πσ 2 2πD 2

Loukia Meligkotsidou, University of Athens Bayesian Inference

Exercise 6.3 - Solution

R∞
√ 1 exp − 2σ1 2 (y − θ)2 √ 1 2 exp − 2D1 2 (θ − B)2 dθ
 
= −∞ 2πσ 2 2πD
n o
1 y2
1
R∞ yθ θ2 θ2 θB B2
= 2πσD −∞ exp − [
2 σ2 − 2 σ2
+ σ2
+ D2
− 2 D2
+ D2
] dθ

Loukia Meligkotsidou, University of Athens Bayesian Inference

Exercise 6.3 - Solution

R∞
√ 1 exp − 2σ1 2 (y − θ)2 √ 1 2 exp − 2D1 2 (θ − B)2 dθ
 
= −∞ 2πσ 2 2πD
n o
1 y2
1
R∞ yθ θ2 θ2 θB B2
= 2πσD −∞ exp − [
2 σ2 − 2 σ2
+ σ2
+ D2
− 2 D2
+ D2
] dθ
n −2 −2 o
2 − 2θ y σ −2 +BD −2 + ( y σ −2 +BD −2 )2 ] dθ
R∞ σ +D
= −∞ exp − 2 [θ σ −2 +D −2 σ −2 +D −2

(σ −2 y +BD −2 )2
n o
1 y2 B2
× 2πσD exp − 2σ 2 − 2D 2
+ 2(σ −2 +D −2 )

Loukia Meligkotsidou, University of Athens Bayesian Inference

Exercise 6.3 - Solution

R∞
√ 1 exp − 2σ1 2 (y − θ)2 √ 1 2 exp − 2D1 2 (θ − B)2 dθ
 
= −∞ 2πσ 2 2πD
n o
1 y2
1
R∞ yθ θ2 θ2 θB B2
= 2πσD −∞ exp − [
2 σ2 − 2 σ2
+ σ2
+ D2
− 2 D2
+ D2
] dθ
n −2 −2 o
2 − 2θ y σ −2 +BD −2 + ( y σ −2 +BD −2 )2 ] dθ
R∞ σ +D
= −∞ exp − 2 [θ σ −2 +D −2 σ −2 +D −2

(σ −2 y +BD −2 )2
n o
1 y2 B2
× 2πσD exp − 2σ 2 − 2D 2
+ 2(σ −2 +D −2 )
√
2π 1
= √
σ −2 +D −2
× 2πσD ×
n 2 2
o
B 2 σ2
exp − 2σ2 D 2 (σ1−2 +D −2 ) [ y σD2 + y 2 + D2
+ B 2 − σ 2 D 2 (σ −2 y + BD −2 )2 ]

Loukia Meligkotsidou, University of Athens Bayesian Inference

Exercise 6.3 - Solution

2 2
B 2 σ2
=√ 1
exp{− 2(D 21+σ2 ) [ y σD2 + y 2 + D2
+ B2
2π(D 2 +σ 2 )

− σ 2 D 2 σ −4 y 2 − σ 2 D 2 B 2 D −4 − 2σ 2 D 2 σ −2 yBD −2 ]}

Loukia Meligkotsidou, University of Athens Bayesian Inference

Exercise 6.3 - Solution

2 2
B 2 σ2
=√ 1
exp{− 2(D 21+σ2 ) [ y σD2 + y 2 + D2
+ B2
2π(D 2 +σ 2 )

− σ 2 D 2 σ −4 y 2 − σ 2 D 2 B 2 D −4 − 2σ 2 D 2 σ −2 yBD −2 ]}
1
=√ exp{− 2(D 21+σ2 ) (y 2 + B 2 − 2yB)}
2π(D 2 +σ 2 )

1
=√ exp{− 2(D 21+σ2 ) (y − B)2 }
2π(D 2 +σ 2 )

Therefore y | x ∼ N(B, D 2 + σ 2 )

Loukia Meligkotsidou, University of Athens Bayesian Inference

Exercise 6.5
Observations x = (x1 , x2 , . . . , xn ) are made of independent random
variables X = (X1 , X2 , . . . , Xn ) with Xi having uniform distribution
1
f (xi |θ) = ; 0 ≤ xi ≤ θ.
θ
Assume that θ has an improper prior distribution
1
f (θ) = ; θ ≥ 0.
θ

(a) Show that the posterior distribution of θ is given by

nM n
f (θ|x) =; θ ≥ M,
θn+1
where M = max(x1 , x2 , . . . , xn ).
(b) Show that θ has posterior expectation
n
E (θ|x) = M.
n−1
Loukia Meligkotsidou, University of Athens Bayesian Inference
Solution

(a) Likelihood:Q
f (x | θ) = θ1n ni=1 I (0 ≤ xi ≤ θ) = 1
θn I (0 ≤ M ≤ θ),
where M = max(x1 , . . . , xn )
Prior: f (θ) = 1θ
1
Posterior: f (θ | x) = cf (x | θ)f (θ) = c θn+1 I (θ ≥ M)

Loukia Meligkotsidou, University of Athens Bayesian Inference

Solution

(a) Likelihood:Q
f (x | θ) = θ1n ni=1 I (0 ≤ xi ≤ θ) = 1
θn I (0 ≤ M ≤ θ),
where M = max(x1 , . . . , xn )
Prior: f (θ) = 1θ
1
Posterior: f (θ | x) = cf (x | θ)f (θ) = c θn+1 I (θ ≥ M)
−n ∞
R∞ 1 h i
dθ = 1 ⇒ c θ−n
R
f (θ | x)dθ = 1 ⇒ c M θn+1 =1⇒
M

Loukia Meligkotsidou, University of Athens Bayesian Inference

Solution

(a) Likelihood:Q
f (x | θ) = θ1n ni=1 I (0 ≤ xi ≤ θ) = 1
θn I (0 ≤ M ≤ θ),
where M = max(x1 , . . . , xn )
Prior: f (θ) = 1θ
1
Posterior: f (θ | x) = cf (x | θ)f (θ) = c θn+1 I (θ ≥ M)
−n ∞
R∞ 1 h i
dθ = 1 ⇒ c θ−n
R
f (θ | x)dθ = 1 ⇒ c M θn+1 =1⇒
M
−n nM n
c Mn = 1 ⇒ c = nM n ⇒ f (θ | x) = θn+1
, θ ≥ M.

Loukia Meligkotsidou, University of Athens Bayesian Inference

Solution

(a) Likelihood:Q
f (x | θ) = θ1n ni=1 I (0 ≤ xi ≤ θ) = 1
θn I (0 ≤ M ≤ θ),
where M = max(x1 , . . . , xn )
Prior: f (θ) = 1θ
1
Posterior: f (θ | x) = cf (x | θ)f (θ) = c θn+1 I (θ ≥ M)
−n ∞
R∞ 1 h i
dθ = 1 ⇒ c θ−n
R
f (θ | x)dθ = 1 ⇒ c M θn+1 =1⇒
M
−n nM n
c Mn = 1 ⇒ c = nM n ⇒ f (θ | x) = θn+1
, θ ≥ M.

(b) Posterior Expectation: h i∞

R∞ R∞ nM n θ−n+1 nM
E (θ | x) = M θf (θ | x)dθ = M θn dθ = nM n −n+1 M = n−1

Loukia Meligkotsidou, University of Athens Bayesian Inference

Exercise 6.5

(c) Verify the posterior probability:

1
Pr(θ > t M|x) = for any t ≥ 1.
tn
(d) A further, independent, observation Y is made from the same
distribution as X . Show that the predictive distribution of Y
has density
 
1 n 1
f (y |x) = ; y ≥ 0.
M n + 1 [max(1, y /M)]n+1

Loukia Meligkotsidou, University of Athens Bayesian Inference

Solution
R∞ h i∞
nM n θ−n 1
(c) Pr(θ > tM | x) = tM θn+1 dθ = nM n −n tM = tn , t≥1

Loukia Meligkotsidou, University of Athens Bayesian Inference

Solution
R∞ h i∞
nM n θ−n 1
(c) Pr(θ > tM | x) = tM θn+1 dθ = nM n −n tM = tn , t≥1

(d) Likelihood of future observation: f (y |θ) = 1θ ; 0 ≤ y ≤ θ.

nM n
Z Z
1
f (y | x) = f (y | θ)f (θ | x)dθ = I (θ ≥ y ) n+1 I (θ ≥ M)dθ
θ θ

Loukia Meligkotsidou, University of Athens Bayesian Inference

Solution
R∞ h i∞
nM n θ−n 1
(c) Pr(θ > tM | x) = tM θn+1 dθ = nM n −n tM = tn , t≥1

(d) Likelihood of future observation: f (y |θ) = 1θ ; 0 ≤ y ≤ θ.

nM n
Z Z
1
f (y | x) = f (y | θ)f (θ | x)dθ = I (θ ≥ y ) n+1 I (θ ≥ M)dθ
θ θ
Z ∞ n
 n
∞
nM nM
= n+2
dθ = θ−n−1
max(M,y ) θ −n − 1 max(M,y )

Loukia Meligkotsidou, University of Athens Bayesian Inference

Solution
R∞ h i∞
nM n θ−n 1
(c) Pr(θ > tM | x) = tM θn+1 dθ = nM n −n tM = tn , t≥1

(d) Likelihood of future observation: f (y |θ) = 1θ ; 0 ≤ y ≤ θ.

nM n
Z Z
1
f (y | x) = f (y | θ)f (θ | x)dθ = I (θ ≥ y ) n+1 I (θ ≥ M)dθ
θ θ
Z ∞ n
 n
∞
nM nM
= n+2
dθ = θ−n−1
max(M,y ) θ −n − 1 max(M,y )
nM n 1
=
n + 1 [max(M, y )]n+1

Loukia Meligkotsidou, University of Athens Bayesian Inference

Solution
R∞ h i∞
nM n θ−n 1
(c) Pr(θ > tM | x) = tM θn+1 dθ = nM n −n tM = tn , t≥1

(d) Likelihood of future observation: f (y |θ) = 1θ ; 0 ≤ y ≤ θ.

nM n
Z Z
1
f (y | x) = f (y | θ)f (θ | x)dθ = I (θ ≥ y ) n+1 I (θ ≥ M)dθ
θ θ
Z ∞ n
 n
∞
nM nM
= n+2
dθ = θ−n−1
max(M,y ) θ −n − 1 max(M,y )
nM n 1
=
n + 1 [max(M, y )]n+1
nM n 1
= n+1
n+1M [max(M/M, y /M)]n+1

Loukia Meligkotsidou, University of Athens Bayesian Inference

Solution
R∞ h i∞
nM n θ−n 1
(c) Pr(θ > tM | x) = tM θn+1 dθ = nM n −n tM = tn , t≥1

(d) Likelihood of future observation: f (y |θ) = 1θ ; 0 ≤ y ≤ θ.

nM n
Z Z
1
f (y | x) = f (y | θ)f (θ | x)dθ = I (θ ≥ y ) n+1 I (θ ≥ M)dθ
θ θ
Z ∞ n
 n
∞
nM nM
= n+2
dθ = θ−n−1
max(M,y ) θ −n − 1 max(M,y )
nM n 1
=
n + 1 [max(M, y )]n+1
nM n 1
= n+1
n+1M [max(M/M, y /M)]n+1
 
1 n 1
= , y ≥ 0.
M n + 1 [max(1, y /M)]n+1

Loukia Meligkotsidou, University of Athens Bayesian Inference