Random Variable

Generation
Prof. Nicholas Zabaras

Email: [email protected]
URL: https://www.zabaras.com/

October 8, 2020

Statistical Computing and Machine Learning, Fall 2020, N. Zabaras 1

 Contents
 Need for sampling methods, Sampling from a discrete distribution

 Reverse sampling for continuous distributions

 Transformation Methods, Box-Muller Algorithm , Sample from the Gaussian

 Rejection Sampling, Accept/Reject Algorithm, Examples

Following closely:
 C. Robert, G. Casella, Monte Carlo Statistical Methods (Ch.. 1, 2, 3.1, & 3.2) (google books, slides, video)
 J. S. Liu, MC Strategies in Scientific Computing (Chapters 1 & 2)
 J-M Marin and C. P. Robert, Bayesian Core (Chapter 2)
 Statistical Computing & Monte Carlo Methods, A. Doucet (course notes, 2007)
Statistical Computing and Machine Learning, Fall 2020, N. Zabaras 2
 Goals
 The goals for today’s lecture include:

 Learning how to sample from a discrete distribution

 Learn how to use reverse sampling for continuous distributions

 Understand how transformation methods can be used to sample the

Gaussian and other distributions

 Understand rejection sampling and the accept/reject algorithm

Statistical Computing and Machine Learning, Fall 2020, N. Zabaras 3

 The Problem of Interest
 Let 𝒙 be a vector of 𝑘 −random variables and 𝜋 𝒙 the underlying distribution
function. In the context of Bayesian analysis, this will be the posterior
distribution.
 Goal is to evaluate 𝔼𝜋 𝑓 𝒙

  f ( x )    f ( x ) ( x )dx

 ‫ 𝒙𝑑 𝒙 𝜋 ׬‬is the normalizing constant (often not known).

 In most algorithms we do not need to know this normalization constant.

Statistical Computing and Machine Learning, Fall 2020, N. Zabaras 4

 Monte Carlo Integration: Review
 Monte Carlo integration evaluates 𝔼𝜋 𝑓(𝐱ሻ by drawing samples ሼ𝒙𝑖 , 𝑖 =

N
1
  f ( x )  
N
 f (x ), x
i 1
i i i.i.d .

Statistical Computing and Machine Learning, Fall 2020, N. Zabaras 5

 Sampling From an Arbitrary Distribution
 Let us assume that we need to estimate
  f ( x)    f ( x) ( x) dx, x  X

 The basic steps in MC are the following:

 Generate i.i.d samples But how do

x i ~  ( x) we sample from
an arbitrary
𝑁 distribution?
 Evaluate 𝑓መ =
1
෍𝑓(𝐱 𝑖 ሻ
𝑁
𝑖=1
 We have shown that this estimator is unbiased:
𝔼 𝑓መ = 𝔼𝜋 𝑓(𝐱ሻ
 The variance of the estimator is:
𝑉𝑎𝑟 𝑓(𝐱ሻ
𝑉𝑎𝑟 𝑓መ =
𝑁
Statistical Computing and Machine Learning, Fall 2020, N. Zabaras 6
 Sampling From an Arbitrary Distribution
 Consider an arbitrary probability density 𝜋 𝒙

 Monte Carlo approximation is given by

𝑁
1 𝑖 𝑖.𝑖.𝑑.
𝜋ො 𝑁 𝐱 = ෍ 𝛿 𝑿 𝑖 𝐱 , 𝑤ℎ𝑒𝑟𝑒 𝑿 ~𝜋
𝑁
𝑖=1
But how do
 For any function f :X  we sample from
𝑁 an arbitrary
1 distribution?
𝔼𝜋ෝ 𝑁 𝑓 = ෍𝑓(𝑿 𝑖 ൯ ≅ 𝔼𝜋 𝑓
𝑁
𝑖=1

or more precisely:

𝑉𝑎𝑟𝜋 𝑓
𝔼𝑿 𝔼𝜋ෝ 𝑁 𝑓 = 𝔼𝜋 𝑓 𝑎𝑛𝑑 𝑉𝑎𝑟𝑿 𝔼𝜋ෝ 𝑁 𝑓 =
𝑁

Statistical Computing and Machine Learning, Fall 2020, N. Zabaras 7

 Sampling Uniform Random Variables
 Most sampling algorithms rely on generating uniform random variables in
[0, 1].

 We only have algorithms for generating pseudo-random numbers which look

like they are i.i.d. U [0, 1].

 There are several standard uniform random number generators available.

Statistical Computing and Machine Learning, Fall 2020, N. Zabaras 8

 Sampling From a Discrete Distribution
 Consider X  1, 2,3 and

 ( X  1)  1/ 6,  ( X  2)  2 / 6,  ( X  3)  1/ 2
 Define the cdf of 𝑋 for 𝑥 ∈ [0, 3] as
3
FX ( x)    ( X  i ) (i  x)
i 1

and its inverse for 𝑢 ∈ [0, 1]

FX1 (u)  inf  x  X ; FX ( x)  u

 We will show that 𝐹𝑋−1 (𝑢ሻ is a sample from 𝜋 ∙ .

Statistical Computing and Machine Learning, Fall 2020, N. Zabaras 9

 Sampling from a Discrete Distribution
 Consider the distribution of a discrete random variable

 To sample from this discrete distribution, sample 𝑢~𝒰[0,1]

Histogram of generated samples

 Define the following: 0.25

0.2

k
Pk   p j  [0,1], P0  0, P1  p1 , P  1
0.15

0.1
j 1
0.05

 Obviously 0
1 2 3 4 5

Pk  Pr  X  k 
𝑃2 = 𝑝1 + 𝑝2 = Pr 𝑋 ≤ 2
 Set 𝑥 = 𝑘, where 𝑃1 ≤ 𝑢 ≤ 𝑃2
𝑃2

k  min k  1, 2,3,...,  such that u  Pk 

𝑢 𝑘 = min 𝑘 𝑠.t. 𝑢 ≤ 𝑃k = 2
𝑃1 𝑝2
𝑝1

MatLab Implementation
Statistical Computing and Machine Learning, Fall 2020, N. Zabaras 10
 Sampling from a Discrete Distribution
 Consider a discrete random variable 𝑋 taking values ሼ1,2,3, … , ∞} with

probability Pr  X  k   p , with p 1 k i 1
i

 We want by sampling 𝑢~𝒰[0,1] to generate samples 𝑥~𝑝(𝑋ሻ

k
 Let Pk  [0,1], s.t. Pk  Pr  X  k    p j
j 1

 Draw 𝑢~𝒰[0,1].

 Then set 𝑥 = 𝑘, where k  min k  1, 2,3,...,  such that u  Pk 

Pk

Note : Pr  x  k   Pr  Pk 1  u  Pk    1du 𝑃2
Pk 1 𝑢 𝑃1 𝑝2
k k 1 𝑝1
 Pk  Pk 1   p j   p j  pk , i.e. the desired distribution
j 1 j 1

Statistical Computing and Machine Learning, Fall 2020, N. Zabaras 11

 Sampling from a Continuous Distribution
 Assume the distribution has a density 𝜋 𝒙 , then the CDF takes the form:
 x
Note : FX  x  = Pr  X  x  =  π  u   u  x  du   π  u  du
 

 Algorithm: 𝑢~𝒰 0,1 , and then set 𝑋 = 𝐹𝑋−1 𝑢 . Do we have 𝑋~𝜋?

Proof : Pr  X  x   Pr  u  FX  x   since FX is non - decreasing

1
  (u  FX ( x))du since u U  
0

 FX ( x) which is precisely the CDF of 

Statistical Computing and Machine Learning, Fall 2020, N. Zabaras 12

Reverse Sampling for Continuous Distributions
 Simplest method
 Used frequently for non-uniform random number generation
 Sample a random number 𝑢 from 𝒰[0,1ሻ
 Set 𝑥 = 𝑦, where 𝑢 = 𝐹𝑋 (𝑦ሻ (i.e. x  FX1 (u) )
 Simple, but you need the functional form of 𝐹.

Indeed : Pr  x  y   Pr u  FX ( y ) 
FX ( y )

 0
1du  FX ( y )

Sampling using the inverse of the CDF

Statistical Computing and Machine Learning, Fall 2020, N. Zabaras 13

 Sampling from a Continuous Distribution
 The distribution and the CDF of a normal distribution

Statistical Computing and Machine Learning, Fall 2020, N. Zabaras 14

 Inverse Method: Exponential Distribution
 Consider the exponential distribution with parameter l:
 ( x)  l e  l x [0,  )
 The CDF of 𝑋 is:
x  0 if x  0
FX ( x)    ( z )dz   l x

1  e if x  0
 Thus the inverse CDF is:
l x 1
1 e u  x log(1  u )  FX1 (u )
l
1
 Inverse method: 𝑢~𝒰[0,1], then X   log(1  U ) ~   Exp (l )
l
1
 Similarly if 𝑢~𝒰[0,1], then X   log U ~ Exp (l ) . Indeed:
l

Pr  X  x   Pr   log U  l x   Pr U  e  l x   1  e  l x  X ~ Exp (l )

Statistical Computing and Machine Learning, Fall 2020, N. Zabaras 15

Inverse Method: Exponential Distribution
Exponential(l): 𝝅(𝒙ሻ = 𝝀𝒆−𝝀𝒙 , 𝑭𝑿(𝒚ሻ = 𝟏 – 𝒆−𝝀𝒚 , 𝒙 = – ln(𝟏 − 𝒖ሻ/𝝀

As 𝑭(𝒚ሻ is area under 𝝅(𝒚ሻ, 𝒚 = 𝑭−𝟏 (𝒖ሻ

prescribes that
• Choose 𝒖 = (𝟎, 𝟏], then find value
𝒚 that has that fraction 𝒖 of area to
the left of 𝒚, or 𝒖 = 𝑭(𝒚ሻ
• Return that value of 𝒙 = 𝒚.

2.5

1.5

0.5
MatLab Implementation
0
1 2 3 4
Statistical Computing and Machine Learning, Fall 2020, N. Zabaras 16
 Inverse Method: The Max Function
 Consider 𝑁 i.i.d. random variables 𝑋𝑖 ~𝑓𝑋 (with CDF 𝐹𝑋). We are interested to
sample from the following distribution:
Z  max  X 1 ,..., X N 
 You can use an inverse approach as follows:
FZ ( z )  Pr  X 1  z ,..., X N  z 
N
  Pr  X i  z    FX ( z ) 
N

i 1

 Thus for any 𝑈~𝒰[0,1], we can sample 𝑍~𝑓𝑍 as follows:

Z  FZ1 (U )  FX1 (U 1/ N )

Statistical Computing and Machine Learning, Fall 2020, N. Zabaras 17

 Inverse Method: Limitations
 Practical and simple method for univariate distributions.

 Limited to cases where the inverse cdf has an analytical form that can be
tabulated.

 Its practical use is very limited.

Statistical Computing and Machine Learning, Fall 2020, N. Zabaras 18

 Transformation Methods
 We have seen an example of the transformation method:
1
If u ~ U [0,1]  X   log(1  U ) ~ Exp (l )
l
1
If u ~ U [0,1]  X   log U ~ Exp (l )
l
 We use the fact that  is related to other transformations easier to sample
from.
 These methods are specific to some distributions, e.g.
 
X i ~ Exp (1) then : Y  2 X i ~  22 , Y    X i ~ Gamma ( ,  ),
i 1 i 1



Xi
Y  i 1
~ Be ( ,  )


i 1
Xi

Statistical Computing and Machine Learning, Fall 2020, N. Zabaras 19

 Transformation Methods
 Starting with samples from the uniform distribution, these transformations are
very simple to apply:

Y  2 log U i  ~  22 ,
i 1

Y     log U i  ~ Gamma ( ,  )
i 1



log U i 
Y  i 1
~ Be ( ,  )


i 1
log U i 

 With this approach, we cannot generate Gamma random variables with a non-
integer shape parameter 𝑎.

Statistical Computing and Machine Learning, Fall 2020, N. Zabaras 20

 Box Muller Algorithm to Sample Gaussians
 Consider (𝑥, 𝑦ሻ coordinates X 1 ~ N (0,1) and X 2 ~ N (0,1), then the polar coordinates
(𝑅, 𝜃ሻ of this point are independent and distributed according to
1 𝐼𝑓 𝑋1 ~𝒩 0,1 , … , 𝑋𝑛 ~𝒩(0,1ሻ,
R  X  X ~ Exp ( )
2 2 2
1 2
2 then 𝑋12 +. . +𝑋𝑛2 ~𝓍𝑛2 . For 𝑛 = 2,
1
 ~ U [0, 2 ] 2
𝓍𝑛=2 ~ℰ𝓍𝓅 .
2

 It is simple to simulate R  2log(U1 ) and  ~ 2U 2 , where

U1 ~ U [0,1] and U 2 ~ U [0,1]. Then
X 1  R cos   2 log(U1 ) cos(2 U 2 )
0.3
X 2  R sin   2 log(U1 ) sin(2 U 2 )

 By construction 𝑋1 and 𝑋2 are two 0.2

independent 𝒩(0,1ሻ random 0.1

variables. A plot of one of them is

0
shown here. -4 -2 0 2 4
MatLab Implementation

Statistical Computing and Machine Learning, Fall 2020, N. Zabaras 21

 Box Muller Algorithm to Sample Gaussians
 Let us see a simpler approach to sample from a spherical Gaussian. We
1
sample 𝑧1 , 𝑧2 uniformly from a unit circle: 𝑝 𝑧1 , 𝑧2 = 𝕀(𝑧 𝑖𝑛𝑠𝑖𝑑𝑒 𝑢𝑛𝑖𝑡 𝑐𝑖𝑟𝑐𝑙𝑒ሻ. This can
𝜋
be done by sampling uniformly on the cube −1 1 2 and disregarding the samples
outside the circle.
𝜕 𝑧1 ,𝑧2 𝜕 𝑟𝑐𝑜𝑠𝜃,𝑟𝑠𝑖𝑛𝜃
 Consider polar coordinates 𝑟, 𝜃 . You can show that = = 𝑟.
𝜕 𝑟,𝜃 𝜕 𝑟,𝜃

 Define 𝑦1 = −2𝑙𝑛𝑟 2 1/2 𝑐𝑜𝑠𝜃, 𝑦

2 = −2𝑙𝑛𝑟 2 1/2 𝑠𝑖𝑛𝜃. With simple algebra you can
𝜕 𝑦1 ,𝑦2 2
show that = − .
𝜕 𝑟,𝜃 𝑟
 The distribution of 𝑦1 , 𝑦2 can now be computed as:
−1
𝜕 𝑧1 , 𝑧2 𝜕 𝑦1 , 𝑦2 1 𝑟 𝑟2 1 1 2
𝑝 𝑦1 , 𝑦2 = 𝑝 𝑧1 , 𝑧2 = 𝑟 = = 𝑒𝑥𝑝 − 𝑦1 + 𝑦22
𝜕 𝑟, 𝜃 𝜕 𝑟, 𝜃 𝜋 2 2𝜋 2𝜋 2
 Thus we obtain the desired result:
1 1 2 1 1 2
𝑝 𝑦1 , 𝑦2 = 𝑒𝑥𝑝 − 𝑦1 𝑒𝑥𝑝 − 𝑦2
2𝜋 2 2𝜋 2
Statistical Computing and Machine Learning, Fall 2020, N. Zabaras 22
Sampling from the Bivariate Normal Density
To sample from the bivariate density,

1) Generate two, uncorrelated, standard normal variates, 𝑧1 and 𝑧2.

z ~ N    0,  2  1

2) Compute the correlated 𝑋1 and 𝑋2

X 1  1   1 z1
X 2  2   2  z1  z2 1   2 
 
3) 𝑋1 and 𝑋2 will have means 𝜇1 and 𝜇2, standard deviations 𝜎1 and 𝜎2, and correlation 𝜌

Statistical Computing and Machine Learning, Fall 2020, N. Zabaras 23

Sampling from the Bivariate Normal Density
Example 1. data are sampled from the standard bivariate normal distribution

Example 2. data are sampled from the bivariate normal distribution with

Mean vector {1 , 2 }  {1,1}

Covariance 1.44 0.3 

matrix  
 0.3 0.25 

Statistical Computing and Machine Learning, Fall 2020, N. Zabaras 24

 Sampling from the Multivariate Gaussian
 In principle, the inverse approach can work in high-dimensions:

 Sample 𝒖~𝒰[0,1]𝑑
1
 Set x  FX (u)
 For the multivariate N (  , ) , we can apply the following technique:
 Draw 𝑧𝑖~𝒩(0,1ሻ i.i.d.
 Introduce the Cholesky decomposition of :   SS T
 Set x =  + Sz
Indeed :  x -   x -    =  Szz T S T  
T
 
 S  zz T  S T  SS T  

Statistical Computing and Machine Learning, Fall 2020, N. Zabaras 25

Sampling from the Bivariate Normal Density
Validation of the algorithm

Exact pdf PDF with bivariate samples PDF with univariate samples

The pdf generated from

samples compared with the
true standard bivariate
Gaussian pdf

A MatLab implementation can be downloaded from here.

Statistical Computing and Machine Learning, Fall 2020, N. Zabaras 26
Sampling from the Bivariate Normal Density

A MatLab implementation can be downloaded from here.

Statistical Computing and Machine Learning, Fall 2020, N. Zabaras 27
Sampling from the Bivariate Normal Density

A MatLab implementation can be downloaded from here.

Statistical Computing and Machine Learning, Fall 2020, N. Zabaras 28
 Rejection Sampling
 More generally, we would like to sample from 𝜋(𝑥ሻ defined on X only known
up to a proportionality constant,    *

 The method relies on samples generated from a proposal distribution 𝑞(𝑥ሻ on

𝑋. 𝑞 might also be known up to a normalizing constant, q  q *

 We need 𝑞(𝑥ሻ to dominate 𝜋(𝑥ሻ, i.e.

 *( x)
M  sup  
x X q *( x)

 This implies that  *( x)  0  q *( x)  0 but also that the tails of q *( x) must be

thicker than the tails of  *( x).

Statistical Computing and Machine Learning, Fall 2020, N. Zabaras 29

 Accept/Reject Algorithm
 More generally, we would like to sample from 𝜋(𝑥ሻ, but it’s easier to sample
from a proposal distribution 𝑞(𝑥ሻ
 𝑞(𝒙ሻ satisfies 𝜋 ∗ (𝒙ሻ < 𝑀’ 𝑞∗ (𝒙ሻ for all 𝒙 for some 𝑀’ ≥ 𝑀
 Procedure:
 Sample 𝑦 from 𝑞(𝑦ሻ and 𝑢 from 𝒰[0,1]
 Accept (set 𝑥 = 𝑦) with probability 𝜋 ∗ (𝑦ሻ/ 𝑀’𝑞 ∗ (𝑦ሻ (i.e. if 𝑢 ≤ 𝜋 ∗ (𝑦ሻ / 𝑀’
𝑞 ∗ (𝑦ሻ)
 Reject otherwise and repeat.
 The accepted 𝑥 (𝑖ሻ are samples from 𝜋(𝑥ሻ!
 If 𝑀’ is too large, we will rarely accept samples.
B. D. Flury, Rejection Sampling made easy, SIAM review, 1990
J. Halton, Reject the rejection technique, J. Scientific Computing, 1992

Statistical Computing and Machine Learning, Fall 2020, N. Zabaras 30

 Rejection Sampling
Reject
 *( y )
M ' q *( y )
M ' q *( y )
M ' q *( y )u  *( y )

y
 Set 𝑖 = 1 Accept
 Repeat until 𝑖 = 𝑁
 Sample y ~ q( y ) and u ~ U(0,1)
 If u  M' *( y)
q *( y )
then accept (set 𝑥 (𝑖ሻ = 𝑦) and increment the

counter 𝑖
 Otherwise, reject

The distribution 𝜋(𝑦ሻ needs to be known only up to a normalizing

constant:
 *( y )
 ( y)  , Z    *( y )dy
Z
Statistical Computing and Machine Learning, Fall 2020, N. Zabaras 31
 Rejection Sampling
 Step 1:Sample y ~ q *( y) and u ~ U(0,1)
 *( y )
 Step 2: Set 𝑥 = 𝑦 if u  M ' q *( y) . Otherwise return to step 1.
1 if the proposed sample y is accepted
Define : J  
0 otherwise
1  *( y ) Z
Indeed : Pr[ J  1]   Pr[ J  1| y ]q( y )dy   q *( y ) dy  
X q *( y)dy M ' q *( y) M '  q *( y )dy
X
Distribution of accepted x :
p ( y = x and J = 1) p ( J = 1 | y = x ) p ( y = x )
p ( y = x | J = 1)   
p ( J = 1) p ( J = 1)
M '  q *( y )dy
 *( x)  *( x)
 q( x) X
   ( x)
M ' q *( x) p ( y= x ) Z Z
Pr[ J 1| y  x ] Pr[ J 1]

X  *( y)dy
Note : From Pr[Y accepted ]   large M ' leads to very low acceptance
M '  q *( y )dy
X
Statistical Computing and Machine Learning, Fall 2020, N. Zabaras 32
 Rejection Sampling
 We can alternatively show that Pr Y  x | Y accepted   Pr  X  x  x  X

 Indeed, it is clear that:

1 x  π *( y ) 
Pr(Y  x and Y accepted) =    u   q( y ) 1dydu
0 
 M'q *( y ) 
x


xπ *( y )
q ( y )dy =

π *( y )dy
 M'q *( y )
M'  q *( y )dy
X
  *( y)dy
 Using Bayes rule and Pr[Y accepted ]  X

M '  q *( y )dy
X
x

π *( y )dy
M'  q *( y )dy
Pr(Y ≤ x and Y accepted) x
Pr(Y ≤ x | Y accepted) =  X
   ( y ) dy
X
Pr(Y accepted) 
π *( y )dy
M'  q *( y )dy
X
Statistical Computing and Machine Learning, Fall 2020, N. Zabaras 33
 Accept/Reject Efficiency
 The acceptance probability Pr(𝑌 𝑎𝑐𝑐𝑒𝑝𝑡𝑒𝑑ሻ is a measure of efficiency.

 The expected number of trials before accepting a candidate is

1
Pr(Y is accepted )
 The number of trials before success is thus an unbiased estimate of
1/Pr(𝑌 𝑖𝑠 𝑎𝑐𝑐𝑒𝑝𝑡𝑒𝑑ሻ.

 This is important as we will see later when discussing the Metropolis Hastings
algorithm.

Statistical Computing and Machine Learning, Fall 2020, N. Zabaras 34

 Accept/Reject Efficiency
 The number of trials before a candidate sample is accepted is a geometric
distribution.

 Indeed note that:

Pr  k th proposal is accepted   (1   ) k 1 

X  *( y)dy
  Pr(Y is accepted ) 
M '  q *( y )dy
X

 The mean of the geometric distribution is 1/𝛾, thus the number of trials before
success is thus an unbiased estimate of 1/Pr(𝑌 𝑖𝑠 𝑎𝑐𝑐𝑒𝑝𝑡𝑒𝑑ሻ.

Statistical Computing and Machine Learning, Fall 2020, N. Zabaras 35

 Rejection Sampling: Example
 Consider as a target PDF 𝜋(𝑥ሻ = 𝜋 ∗ (𝑥ሻ = ℬℯ𝓉𝒶(3,2ሻ
 We take as proposal 𝑞 = 𝑞∗ ~𝑈[0,1] and 𝑀’ = 𝑀 = 48/27
2.5
For Be ( x; ,  ), q  U [0,1]
Mq ( x) x 1 (1  x)  1
find : M  sup
2

x 0,1 1
1.5 2.5

1 2

values of the weights * 0.5N

0.5 1.5

0 1
0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1

0.5
 For a C++ implementation of
this example see here. 0
0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1

 Two MatLab implementations are value of x

also available.
Statistical Computing and Machine Learning, Fall 2020, N. Zabaras 36
 Example: Accept-Reject Sampling
 In this example, the target and proposal distributions are:
 * ( x)  exp   x 2 / 2 sin 2 (6 x)  3cos 2 ( x) sin 2 (4 x)  1
q* ( x)  exp   x 2 / 2 

 The proposal distribution is a standard normal.

 The maximum of each of the trigonometric functions inside the target is 1
 Hence a reasonable upper bound for the ratio of the target to the proposal is 5
  * ( x) 
sup  * 5
 q ( x ) 
M '5
The adjoining figure illustrates the sampling procedure. Link to MatLab implementation
1. Plot the target distribution and 5 times the proposal
distribution to show the manner in which the proposal
envelopes the target
2. Generate a sample x from the normal distribution
3. Evaluated acceptance probability as
 * ( x)
5q * ( x )
4. Generate a uniform random number 𝑈. If it is less than
the acceptance probability we accept the sample.
5. Plot the point as
( x,5Uq* ( x))

This picture resulted from 500 samples drawn from the

proposal distribution
Statistical Computing and Machine Learning, Fall 2020, N. Zabaras 37
 Example: Accept-Reject Sampling
 The target distribution and the proposal distribution are given as
 ( x)  exp   x / 2 sin (6 x)  3cos ( x) sin (4 x)  1
* 2 2 2 2

q ( x)  exp   x / 2 
* 2

 The proposal distribution is a standard normal.

 The maximum of each of the trigonometric functions inside the target is 1
 Hence a reasonable upper bound for the ratio of the target to the proposal
  ( x) 
*
is 5 sup 
* 5
 q ( x) 
M '5 Link to MatLab implementation

The figure shows the

histogram plot obtained from
5000 samples drawn from
the proposal distribution

Statistical Computing and Machine Learning, Fall 2020, N. Zabaras 38