PROBABILITY & STATISTICS

Introduction

Sampling Chapter 7: Point Estimation of Parameters

Distributions

General concepts
of point Learning objectives
estimation

1. Introduction
Methods of point
estimation 2. Sampling Distributions
Method of
moments 3. General Concepts Of Point
Method of
maximum 4. Estimation Methods Of Point Estimation
likelihood

Summary

07/06/2023 Department of Mathematics 1/31

 SET UP PROBLEM
Introduction
Introduction

Sampling Let X is a random variable with probability

Distributions distribution f(x), which is characterized by the
General concepts unknown θ = (θ1, θ2, …, θk)
of point
estimation

Methods of point For example, X ~ N(µ, σ2) then θ = (µ, σ2).

estimation
Method of
moments
Method of
maximum
likelihood
How to “determine” the values of θ?

Summary

07/06/2023 Department of Mathematics 2/31

 PROBLEMS IN ENGINEERING
Introduction
Introduction

Sampling
Distributions In engineering, we often need to estimate
1. The mean μ of a single population.
General concepts
of point
2. The variance σ2 (or standard deviation ) of a single
estimation
population.
Methods of point
estimation
3. The proportion p of items in a population that belong to
Method of a class of interest.
moments
Method of
4. The difference in means of two populations, μ1 – μ2 .
maximum
5. The difference in two population proportions, p1 – p2 .
likelihood

Summary

07/06/2023 Department of Mathematics 3/31

 PROBLEMS IN ENGINEERING
Introduction
Introduction
The important results on point estimation
Sampling
Distributions 1. For μ , the estimate is the ̂ = x , sample mean.
General concepts 2. For σ2 , the estimate is ˆ 2 = s 2 , the sample variance.
3. For p, the estimate is pˆ = x / n , the sample proportion.
of point
estimation

Methods of point
4. For μ1 – μ2 , the estimate is ˆ1 − ˆ2 = x1 − x2
estimation
ˆ1 − pˆ 2
5. For p1 – p2 , the estimate is p
Method of
moments
Method of
maximum
likelihood

Summary

07/06/2023 Department of Mathematics 4/31

 SAMPLING DISTRIBUTIONS
Introduction
Definition Random Sample
Sampling
Sampling
Distributions The random variables X1, …., Xn are called a random sample of
Distributions
size n if
General concepts
of point
• The Xi’s are independent
estimation • Every Xi has the same probability distribution
Methods of point
estimation Definition Statistic
Method of
moments • A statistic ̂ is any function of the observations X1, …, Xn:
Method of
ˆ = h( X ,..., X )

maximum 1 n
likelihood • The probability distribution of a statistic is called a sampling
distribution.
Summary

07/06/2023 Department of Mathematics 5/31

 SAMPLING DISTRIBUTIONS
Introduction

Sampling
Sampling
Example Two important statistic
Distributions
Distributions • Sample mean X
General concepts
X 1 + ... + X n
of point X=
estimation n

Methods of point • Sample variance S2

estimation
Method of
moments
( X − X )2
+...+ ( X − X )2
Method of S2 = 1 n
maximum n −1
likelihood

Summary

07/06/2023 Department of Mathematics 6/31

 SAMPLING DISTRIBUTIONS
Introduction

Sampling
Sampling
Theorem
Distributions
Distributions In above example, if (X1, …., Xn) is a random sample of
General concepts size n take from a normal distribution N(μ, σ2) then
of point
estimation • X has a normal distribution N(μ, σ2/n)

Methods of point ( n − 1) S 2
estimation
• 2 has a chi-square distribution with n-1 degrees of
Method of
moments freedom (see pages 273-274).
Method of
maximum
likelihood

Summary

07/06/2023 Department of Mathematics 7/31

 SAMPLING DISTRIBUTIONS
Introduction

Sampling
Theorem Central Limit Theorem
Sampling
Distributions
Distributions Let (X1, …., Xn) is a random sample of size n take from a
population with mean  and finite variance σ2, and if X
General concepts
of point is the sample mean, the limiting form of the distribution
estimation of
X −
Methods of point Z =
estimation  / n
Method of
moments
Method of
As n →  , is the standard normal distribution.
maximum
likelihood
Remark: The normal approximation for X depends on
Summary the sample size n.
07/06/2023 Department of Mathematics 8/31
 SAMPLING DISTRIBUTIONS
Introduction

Sampling
Sampling
Distributions
Distributions

General concepts
of point
estimation

Methods of point
estimation
Method of Figure 7-1
moments Distributions of
Method of average scores
maximum
likelihood
from throwing
dice.
Summary

07/06/2023 Department of Mathematics 9/31

 SAMPLING DISTRIBUTIONS
Introduction
Definition Point Estimate
Sampling
Sampling
Distributions
Distributions
• A point estimate of some population parameter θ is a
single numerical value ˆ of a statistic ̂ .
General concepts
of point • The statistic ̂ is called the point estimator.
estimation

Methods of point
estimation
Method of
Two steps to find point estimation:
moments
Method of
Step 1. Determine ̂ by using the theoretical results.

Steps 2. Calculate ˆ from the experimental data.

maximum
likelihood

Summary

07/06/2023 Department of Mathematics 10/31

 SAMPLING DISTRIBUTIONS
Introduction

Sampling
Sampling
Example
Distributions
Distributions Market researchers use the number of sentences per
General concepts
advertisement as a measure of readability for magazine
of point advertisements. The following represents a random sample of
estimation the number of sentences found in 15 advertisements. Find a
point estimate of the population mean, .
Methods of point
estimation 9 20 18 16 9 9 11 13 22 16 5 18 6 6 5
Method of
ˆ X 1 + ... + X 15
moments Step 1.  =
Method of 15
maximum Step 2. A point estimate for  is
likelihood
9 + 20 + ... + 6 + 5 183
ˆ = = = 12. 2
Summary 15 15
07/06/2023 Department of Mathematics 11/31
 UNBIASED ESTIMATOR
Introduction
Definition Unbiased Estimator
Sampling
Distributions The point estimator ̂ is an unbiased estimator for the
parameter θ if
General concepts
General concepts
of
of point
point ˆ =
E ()
estimation
estimation
If the estimator is not unbiased, then the difference
Methods of point
estimation
ˆ − )
E (
is called the bias of the estimator ̂.
Method of
moments
Method of
maximum
likelihood

Summary

07/06/2023 Department of Mathematics 12/31

 UNBIASED ESTIMATOR
Introduction

Sampling Example
Distributions
Suppose that X is a random variable with mean μ and
General concepts
General concepts variance σ2. Let X1, …, Xn be a random sample of size n
of
of point
point from the population represented by X. Show that the
estimation
estimation sample mean X and sample variance S2 are unbiased
estimators of μ and σ2, respectively.
Methods of point
estimation
Method of
moments For the sample mean.
Method of
maximum
X 1 + ... + X n EX 1 + ... + EX n
likelihood E( X ) = E = =
n n
Summary

07/06/2023 Department of Mathematics 13/31

 UNBIASED ESTIMATOR
Introduction
Now consider the sample variance. We have
Sampling
Distributions

General concepts
General concepts
of
of point
point
estimation
estimation

Methods of point
estimation
Method of
moments
Method of
maximum
likelihood

Summary Hence E (S 2 ) =  2
07/06/2023 Department of Mathematics 14/31
 MINIMUM VARIANCE UNBIASED ESTIMATOR
Introduction
Definition Minimum Variance Unbiased Estimator
Sampling
Distributions If we consider all unbiased estimators of θ, the one with
the smallest variance is called the minimum variance
General concepts
General concepts unbiased estimator (MVUE).
of
of point
point
estimation
estimation

Methods of point
estimation
Method of Figure 7-5
moments The sampling
Method of distributions of
maximum
likelihood
two unbiased
estimators
Summary ˆ and 
 ˆ .
1 2
07/06/2023 Department of Mathematics 15/31
 MINIMUM VARIANCE UNBIASED ESTIMATOR
Introduction

Sampling Theorem
Distributions
Let (X1, …., Xn) is a random sample of size n take from a
General concepts
General concepts normal distribution N(μ, σ2) , the sample mean X is the
of
of point
point MVUE for μ.
estimation
estimation

Methods of point
estimation
Method of
moments
Method of
maximum
likelihood

Summary

07/06/2023 Department of Mathematics 16/31

 STANDARD ERROR
Introduction
Definition Minimum Variance Unbiased Estimator
Sampling
Distributions The standard error of an estimator ̂ is its standard
deviation, given by
General concepts
General concepts
of
of point
point  ˆ = V ( ˆ )
estimation
estimation
If the standard error involves unknown parameters that
Methods of point can be estimated, substitution of those values into  ̂
estimation produces an estimated standard error, denoted by ˆ ̂
Method of
moments
Method of Remark
maximum
likelihood If the random sample of size n take from a normal
distribution N(μ, σ2) then X has a normal distribution N(μ,
Summary σ2/n).
07/06/2023 Department of Mathematics 17/31
 STANDARD ERROR
Introduction

Sampling
Standard error of X is
Distributions


General concepts
General concepts  Xˆ =
of
of point
point n
estimation
estimation

If we did not know σ then the estimated standard error of

Methods of point
estimation X would be
Method of
moments s
Method of ˆ Xˆ =
maximum n
likelihood

Summary

07/06/2023 Department of Mathematics 18/31

 STANDARD ERROR
Introduction
Exercise
Sampling A new method of measuring the thermal conductivity of
Distributions
Armco iron: Using a temperature of 1000F and a power
General concepts
General concepts input of 550 watts, the following 10 measurements of
of
of point
point thermal conductivity (in Btu/hr-ft-0F) were obtained:
estimation
estimation
41.60, 41.48, 42.34, 41.95, 41.86,
Methods of point
estimation 42.18, 41.72, 42.26, 41.81, 42.04
Method of
moments
a, Find a point estimate of the mean thermal conductivity
Method of at 1000F and 550 watts.
maximum
likelihood b, Find the estimated standard error of X

Summary

07/06/2023 Department of Mathematics 19/31

 METHOD OF MOMENTS
Introduction
Definition The kth moments
Sampling
Distributions Let (X1, …., Xn ) be a random sample from random variable
X with the probability distribution f (x)
General concepts
of point • The kth moment of X are denoted by Mk := E(Xk)
estimation
• The corresponding kth sample moment is
Methods of point
estimation X k
+ ... + X k

Method of
Method of Xk = 1 n

moments
moments n
Method of
maximum
likelihood

Summary

07/06/2023 Department of Mathematics 20/31

 METHOD OF MOMENTS
Introduction
Definition The kth moments
Sampling
Distributions Let (X1, …., Xn ) be a random sample from either a
probability mass function or probability density
General concepts
function with m unknown parameters (θ1, θ2, …,θm).
of point
The moment estimators  ˆ , ..., 
ˆ are found the
estimation 1 m
following system
Methods of point
estimation M 1 = X 1
Method of
Method of 
moments
moments .......... ...
Method of M = X m
maximum  m
likelihood

Summary

07/06/2023 Department of Mathematics 21/31

 METHOD OF MOMENTS
Introduction

Example
Sampling
Distributions Suppose that X1, …, Xn be a random sample of size n
from a normal distribution N(μ, σ2). Find the moment
General concepts estimators of μ and σ2.
of point
estimation
We have to solve following system
Methods of point
 X 1 + ... + X n
M 1 = X  EX =
estimation 1
Method of
Method of
 or n
M 2 = X 2 
+ +
moments
moments 2 2
Method of  EX 2 = 1
X ... X n

maximum  n
likelihood

Summary Here E(X) = μ and E(X2) = V(X) +(EX)2 = σ2 + μ2

07/06/2023 Department of Mathematics 22/31
 METHOD OF MOMENTS
Introduction
Hence, by solving the above system we have
Sampling
Distributions n n

General concepts  i
X 2
− n( X ) 2
 i
( X − X ) 2

of point  = X, 2 = i =1
= i =1

estimation n n
Methods of point Finally, the moment estimators are
estimation
Method of
Method of n
moments
moments
Method of  i
( X − X ) 2
Biased

ˆ = X ,  = i =1 estimator
maximum ˆ 2
likelihood n
Summary Unbiased
estimator
07/06/2023 Department of Mathematics 23/31
 METHOD OF MAXIMUM LIKELIHOOD
Introduction
Definition Maximum likelihood Estimator
Sampling
Distributions Suppose that X is a random variable with probability
distribution f (x, θ ), where θ is a single unknown
General concepts parameter. Let x1, x2, , xn be the observed values in a
of point
estimation
random sample of size n. Then the likelihood function
of the sample is
Methods of point
estimation L(θ) = f(x1, θ) f(x2, θ)…f(xn, θ)
Method of
moments
The maximum likelihood estimator (MLE) of θ is the
Method of
Method of value of θ that maximizes the L(θ).
maximum
maximum
likelihood

Summary

07/06/2023 Department of Mathematics 24/31

 METHOD OF MAXIMUM LIKELIHOOD
Introduction

Example
Sampling
Distributions Let X is a Bernoulli random variable. The probability
mass function is
General concepts
of point  p x (1 − p)1− x , x = 0,1
estimation
f ( x, p) = 
0 otherwise
Methods of point Find the maximum likelihood estimator of p.
estimation
Method of
moments The likelihood function of a random sample of size n is
Method of
Method of
maximum
maximum
likelihood

Summary

07/06/2023 Department of Mathematics 25/31

 METHOD OF MAXIMUM LIKELIHOOD
Introduction

Sampling
Distributions

General concepts
of point
estimation

Methods of point
estimation
Method of
moments
Method of
Method of
The maximum likelihood estimator of p is
maximum
+ ... + X
maximum
ˆ X
P =
likelihood 1 n

Summary
n
Is this a unbiased estimator?
07/06/2023 Department of Mathematics 26/31
 METHOD OF MAXIMUM LIKELIHOOD
Introduction
Exercise
Sampling
Distributions Let X is a random variable, which has probability mass function
is
p , x = −1
General concepts

of point f ( x ) = 1 − p , x =1
estimation 0
 , otherwise
Methods of point Find the maximum likelihood estimator of p.
estimation
Method of
 1−2x 1+ x
moments Solution:
f (x) =  p (1 − p) 2
, x = −1,1
Method of
Method of
maximum
maximum 0 , otherwise
likelihood

Summary
ˆP = 1 − X 1 + ... + X n
2 2n
07/06/2023 Department of Mathematics 27/31
 METHOD OF MAXIMUM LIKELIHOOD
Introduction

Sampling
Invariance Property
Distributions
Let ˆ 1 ,..., ˆ k be the maximum likelihood estimators of the
General concepts parameters 1 ,..., k . Then the maximum likelihood
estimator of any function h(1,...,k ) of these parameters is
of point
estimation
the same function h(ˆ 1 ,..., ˆ k ) of the estimators ˆ 1 ,..., ˆ k .
Methods of point
estimation
Method of
moments
Method of
Method of
maximum
maximum
likelihood

Summary

07/06/2023 Department of Mathematics 28/31

 METHOD OF MAXIMUM LIKELIHOOD
Introduction
Exercise
Sampling Let X be normally distributed N(μ, σ2), where both μ and
Distributions
σ2 are unknown.
General concepts
(a) Find the maximum likelihood estimator of μ and σ2.
of point
estimation (b) Find the maximum likelihood estimator of θ = μ2σ2.
Methods of point
estimation n

 i
Solution:
Method of ( X − X ) 2
moments
Method of ˆ = X , ˆ 2 = i =1
Method of
maximum
n
maximum n
likelihood
 ( X i − X )2
Summary ˆ = ( X ) 2 i =1
n
07/06/2023 Department of Mathematics 29/31
 SUMMARY
Introduction
We have studied:
Sampling
Distributions
1. Sampling distributions
2. General concepts of point estimation: Unbiased Estimator
General concepts
of point 3. Methods of point estimation
estimation
• Method of Moments
Methods of point
estimation • Method of Maximum Likelihood
Method of
moments
Method of
Homework: Read slides of the next lecture.
maximum
likelihood

Summary
Summary

07/06/2023 Department of Mathematics 30/31