CS303: Mathematical Foundations for AI

Sampling
28 Mar 2025
Recap

• Previous

1 / 20
Recap

• Previous
↭ Change of Variables (Discrete)

1 / 20
Recap

• Previous
↭ Change of Variables (Discrete)
↭ Change of Variables (Continuous)

1 / 20
Recap

• Previous
↭ Change of Variables (Discrete)
↭ Change of Variables (Continuous)
• Inverse CDF Sampling/Inverse transform Sampling/Smirnov Sampling/

1 / 20
Recap

• Previous
↭ Change of Variables (Discrete)
↭ Change of Variables (Continuous)
• Inverse CDF Sampling/Inverse transform Sampling/Smirnov Sampling/
• Rejection Sampling

1 / 20
Recap

• Previous
↭ Change of Variables (Discrete)
↭ Change of Variables (Continuous)
• Inverse CDF Sampling/Inverse transform Sampling/Smirnov Sampling/
• Rejection Sampling
• Monte Carlo Sampling

1 / 20
References

• Murphy, Kevin P. Machine learning: a probabilistic perspective. MIT

press, 2012. (link)
• Bishop, Christopher M., and Nasser M. Nasrabadi. Pattern recognition
and machine learning.(link)
• Sampling (Resource 1) (Video Lectures)

2 / 20
Sampling Uniform Distribution

How to generate X → Uni f [0, 1]?

3 / 20
Sampling Uniform Distribution

How to generate X → Uni f [0, 1]?

Pseudo Random Number Generators (PRNG)

Xn+1 = ( aXn + c) mod m

Mersenne Twister

3 / 20
Sampling Non-Uniform Distribution

How to sample any distribution assuming we can sample from Uniform Distribution?

4 / 20
Inverse transform sampling

Inverting the CDF!

4 / 20
Inverse transform sampling

Case 1: CDF is invertible

4 / 20
Inverse transform sampling

Case 1: CDF is invertible

• Consider a random variable X : Ω ↑ R

4 / 20
Inverse transform sampling

Case 1: CDF is invertible

• Consider a random variable X : Ω ↑ R
• FX is invertible i.e.,

↓ F ↔1 : (0, 1) ↑ R s.t. F ↔1 ( F ( x )) = x, ↗ x ↘ R

4 / 20
Inverse transform sampling

Case 1: CDF is invertible

• Consider a random variable X : Ω ↑ R
• FX is invertible i.e.,

↓ F ↔1 : (0, 1) ↑ R s.t. F ↔1 ( F ( x )) = x, ↗ x ↘ R

4 / 20
Inverse transform sampling

Case 1: CDF is invertible

• Consider a random variable X : Ω ↑ R
• FX is invertible i.e.,

↓ F ↔1 : (0, 1) ↑ R s.t. F ↔1 ( F ( x )) = x, ↗ x ↘ R

Theorem 5

If U → Uni f (0, 1) then F ↔1 (U ) → F i.e., F ↔1 (U ) has a CDF F.

4 / 20
Invertible CDF

• X → Uni f ( a, b) (viz)

5 / 20
Invertible CDF

• X → Uni f ( a, b) (viz)
• X → Normal (µ, σ) ?

5 / 20
CDF

• Exponential
• Discrete

6 / 20
Inverse Transform Sampling

Case 2: CDF is not invertible

7 / 20
Inverse Transform Sampling

Case 2: CDF is not invertible

• Consider a random variable X : Ω ↑ R

7 / 20
Inverse Transform Sampling

Case 2: CDF is not invertible

• Consider a random variable X : Ω ↑ R
• G : (0, 1) ↑ R
G ( a) = in f { x ↘ R : a ≃ F ( x )}

7 / 20
Inverse Transform Sampling

Case 2: CDF is not invertible

• Consider a random variable X : Ω ↑ R
• G : (0, 1) ↑ R
G ( a) = in f { x ↘ R : a ≃ F ( x )}

7 / 20
Inverse Transform Sampling

Case 2: CDF is not invertible

• Consider a random variable X : Ω ↑ R
• G : (0, 1) ↑ R
G ( a) = in f { x ↘ R : a ≃ F ( x )}

Theorem 10
If U → Uni f (0, 1) then G (U ) → F i.e., G (U ) has a CDF F.

7 / 20
Inverse CDF

Examples
• Discrete
xi 1 2 3 4
pi 0.1 0.4 0.2 0.3

8 / 20
Inverse CDF

Examples
• Discrete
xi 1 2 3 4
pi 0.1 0.4 0.2 0.3

↭ Generate U → Uni f (0, 1)

8 / 20
Inverse CDF

Examples
• Discrete
xi 1 2 3 4
pi 0.1 0.4 0.2 0.3

↭ Generate U → Uni f (0, 1)

↭ Determine k s.t. ∑kj=↔11 p j ≃ U < ∑kj=1 p j , return X = xk

8 / 20
Inverse CDF

Examples
• Discrete
xi 1 2 3 4
pi 0.1 0.4 0.2 0.3

↭ Generate U → Uni f (0, 1)

↭ Determine k s.t. ∑kj=↔11 p j ≃ U < ∑kj=1 p j , return X = xk
↭ viz

8 / 20
Inverse CDF

Examples
• Discrete
xi 1 2 3 4
pi 0.1 0.4 0.2 0.3

↭ Generate U → Uni f (0, 1)

↭ Determine k s.t. ∑kj=↔11 p j ≃ U < ∑kj=1 p j , return X = xk
↭ viz
• X → λe↔λx (Exponential)

8 / 20
Box-Muller Algorithm

How to generate Normal Distribution?

9 / 20
Box-Muller Algorithm

How to generate Normal Distribution?

Reference

9 / 20
Rejection Sampling

Sampling from a complicated distribution or in higher dimensional space

10 / 20
Rejection Sampling

Sampling from a complicated distribution or in higher dimensional space

• Uniform on A where A is complicated!

10 / 20
Rejection Sampling

Sampling from a complicated distribution or in higher dimensional space

• Uniform on A where A is complicated!

Mandelbrot Set

10 / 20
Rejection Sampling

Uniform Distribution
• If A ⇐ B

11 / 20
Rejection Sampling

Uniform Distribution
• If A ⇐ B
• Y1 , Y2 , . . . are independently and uniformly drawn from B

11 / 20
Rejection Sampling

Uniform Distribution
• If A ⇐ B
• Y1 , Y2 , . . . are independently and uniformly drawn from B
• X = YK where K = min{k : Yk ↘ A}

11 / 20
Rejection Sampling

Uniform Distribution
• If A ⇐ B
• Y1 , Y2 , . . . are independently and uniformly drawn from B
• X = YK where K = min{k : Yk ↘ A}
• Then X → Uni f ( A)

11 / 20
Rejection Sampling

Uniform Distribution
• If A ⇐ B
• Y1 , Y2 , . . . are independently and uniformly drawn from B
• X = YK where K = min{k : Yk ↘ A}
• Then X → Uni f ( A)
• Note that K is a random variable

11 / 20
Rejection Sampling

Sampling from a complicated Non-uniform distribution f on R d

12 / 20
Rejection Sampling

Sampling from a complicated Non-uniform distribution f on R d

• Choose a proposal distribution q

12 / 20
Rejection Sampling

Sampling from a complicated Non-uniform distribution f on R d

• Choose a proposal distribution q
↭ there exist c > 0 where cq( x ) ⇒ f ( x ) for all x

12 / 20
Rejection Sampling

Sampling from a complicated Non-uniform distribution f on R d

• Choose a proposal distribution q
↭ there exist c > 0 where cq( x ) ⇒ f ( x ) for all x
↭ Easy to sample from

12 / 20
Rejection Sampling

Sampling from a complicated Non-uniform distribution f on R d

• Choose a proposal distribution q
↭ there exist c > 0 where cq( x ) ⇒ f ( x ) for all x
↭ Easy to sample from
• Sample from X → q

12 / 20
Rejection Sampling

Sampling from a complicated Non-uniform distribution f on R d

• Choose a proposal distribution q
↭ there exist c > 0 where cq( x ) ⇒ f ( x ) for all x
↭ Easy to sample from
• Sample from X → q
• Given X, we sample Y → Uni f (0, cq( X ))

12 / 20
Rejection Sampling

Sampling from a complicated Non-uniform distribution f on R d

• Choose a proposal distribution q
↭ there exist c > 0 where cq( x ) ⇒ f ( x ) for all x
↭ Easy to sample from
• Sample from X → q
• Given X, we sample Y → Uni f (0, cq( X ))
• Reject as follows !
Y ≃ f ( X ), Z=X
reject

12 / 20
Rejection Sampling

Sampling from a complicated Non-uniform distribution f on R d

• Choose a proposal distribution q
↭ there exist c > 0 where cq( x ) ⇒ f ( x ) for all x
↭ Easy to sample from
• Sample from X → q
• Given X, we sample Y → Uni f (0, cq( X ))
• Reject as follows !
Y ≃ f ( X ), Z=X
reject

• Z→ f

12 / 20
Rejection Sampling (Intuition)

• Rejecting uniformly the area about f and below cq

13 / 20
Rejection Sampling (Intuition)

• Rejecting uniformly the area about f and below cq

• Choosing a good q

13 / 20
Rejection Sampling

• If X → q and Y | X → Uni f (0, cq( x )) then ( X, Y ) → Uni f ( B)

where B = {( x, y) : x ↘ R d , 0 < y < cq( x )}

14 / 20
Rejection Sampling

• If X → q and Y | X → Uni f (0, cq( x )) then ( X, Y ) → Uni f ( B)

where B = {( x, y) : x ↘ R d , 0 < y < cq( x )}
• Viz

14 / 20
Expectation

!
∑ x x f ( x ); X is discrete
E[X] = "∞
↔∞ x f ( x ) dx; X is continuous

15 / 20
Expectation

!
∑ x x f ( x ); X is discrete
E[X] = "∞
↔∞ x f ( x ) dx; X is continuous
Linearity: E [ aX + Y ] = aE [ X ] + E [Y ]

15 / 20
Examples

Find the expectation of

+2
• Y, f (y) = y25 ; y = {1, 2, 3, 4, 5}
• X → Geometric(λ)

16 / 20
Monte Carlo Sampling

How to approximate expectations?

• without access to the distribution’s closed form
• sum or integral is intractable

17 / 20
Monte Carlo Sampling

18 / 20
Monte Carlo Sampling

Goal: Approximate E [ f ( X )]

19 / 20
Monte Carlo Sampling

Goal: Approximate E [ f ( X )]
• If X1 , X2 , . . . , Xn are sampled from a distribution p

19 / 20
Monte Carlo Sampling

Goal: Approximate E [ f ( X )]
• If X1 , X2 , . . . , Xn are sampled from a distribution p
• Xi⇑ s are independent and identically distributed

19 / 20
Monte Carlo Sampling

Goal: Approximate E [ f ( X )]
• If X1 , X2 , . . . , Xn are sampled from a distribution p
• Xi⇑ s are independent and identically distributed
• The approximate expectation is given by,

1 n
n i∑
µ̂ = f ( Xi )
=1

19 / 20
Monte Carlo Sampling

Goal: Approximate E [ f ( X )]
• If X1 , X2 , . . . , Xn are sampled from a distribution p
• Xi⇑ s are independent and identically distributed
• The approximate expectation is given by,

1 n
n i∑
µ̂ = f ( Xi )
=1

• µ̂ is an estimator for E[ f ( X )]

19 / 20
Unbiased Estimator

The sample mean µ̂ is an

• Unbiased estimator
E [µ̂] = E [ f ( X )]

20 / 20
Unbiased Estimator

The sample mean µ̂ is an

• Unbiased estimator
E [µ̂] = E [ f ( X )]

• Consistent estimator (σ2 ( f ( X )) < ∞)

Pr(|µ̂n ↔ E [ f ( X )]| < ε) ↑ 1

20 / 20
Unbiased Estimator

The sample mean µ̂ is an

• Unbiased estimator
E [µ̂] = E [ f ( X )]

• Consistent estimator (σ2 ( f ( X )) < ∞)

Pr(|µ̂n ↔ E [ f ( X )]| < ε) ↑ 1

1 2
• n σ ( f ( X )) ↑0

20 / 20