Lecture 4: ECON-UA 266 - Intro to

Econometrics
Part I - Univariate analysis

Sahar Parsa

1
Outline

Step 1: Define the population relation of interest/parameter of

interest: The population linear regression model

Step 2: Define estimator: Ordinary Least Squares

(OLS) estimator

Step 3: Define the properties of the estimator under

the underlying assumptions

Step 4: Inference analysis

2
Solution OLS estimator
Solve for the OLS estimator

Formally, let (𝑌𝑖 , 𝑋𝑖 ) be 𝑖.𝑖.𝑑., the Ordinary least squares

problem minimizes the sum of squared residuals:

𝑁
min ∑ 𝑒2𝑖
𝑎,𝑏
𝑖
⇔
𝑁
min ∑(𝑌𝑖 − 𝑎 − 𝑏𝑋𝑖 )2
𝑎,𝑏
𝑖

3
Derive the estimator

FOC with respect to

𝑁
(𝑎 ∶) − 2 ∑𝑖=1 [𝑌𝑖 − 𝛼𝑂𝐿𝑆 − 𝛽𝑂𝐿𝑆 𝑋𝑖 ] = 0

FOC with respect to

𝑁
(𝑏 ∶) − 2 ∑𝑖=1 [𝑌𝑖 − 𝛼𝑂𝐿𝑆 − 𝛽𝑂𝐿𝑆 𝑋𝑖 ]𝑋𝑖 = 0

Two equations and two unknowns (𝛼𝑂𝐿𝑆 , 𝛽𝑂𝐿𝑆 )

4
Derive the estimator

Take the first equation (FOC with respect to 𝛼𝑂𝐿𝑆 )

𝑁 𝑁
∑ 𝑌𝑖 ∑ 𝑋𝑖
𝛼𝑂𝐿𝑆 = 𝑖=1 − 𝛽𝑂𝐿𝑆 𝑖=1
𝑁 𝑁

𝛼𝑂𝐿𝑆 = 𝑌 ̄ − 𝛽𝑂𝐿𝑆 𝑋̄

5
Derive the estimator

Substitute 𝛼𝑂𝐿𝑆 in the second equation:

𝑁
∑[𝑌𝑖 − [𝑌 ̄ − 𝛽𝑂𝐿𝑆 𝑋]̄ − 𝛽𝑂𝐿𝑆 𝑋𝑖 ]𝑋𝑖 = 0
𝑖=1
⇔
𝑁
∑[(𝑌𝑖 − 𝑌 ̄ ) − 𝛽𝑂𝐿𝑆 (𝑋𝑖 − 𝑋)]𝑋
̄ 𝑖 =0
𝑖=1

6
Derive the estimator

𝑁
∑𝑖=1 𝑌𝑖 𝑋𝑖
𝑆 − 𝑌 ̄ 𝑋̄
𝛽𝑂𝐿𝑆 = 𝑌2𝑋 = 𝑁
𝑁
𝑆𝑋 ∑𝑖=1 𝑋𝑖2
𝑁 − 𝑋̄ 2
𝛼𝑂𝐿𝑆 = 𝑌 ̄ − 𝛽𝑂𝐿𝑆 𝑋̄

7
OLS predicted values and residuals

The OLS predicted/fitted values become:

𝑌𝑖̂ = 𝛼𝑂𝐿𝑆 + 𝛽𝑂𝐿𝑆 𝑋𝑖

The OLS residual is given by:

𝑒𝑖 = 𝑌𝑖 − 𝑌𝑖̂

8
Difference between OLS estimator and the popula-
tion regression model

1. Population intercept and slope

𝛼 = 𝔼[𝑌𝑖 ] − 𝛽𝔼[𝑋𝑖 ]
𝐶𝑜𝑣[𝑌𝑖 𝑋𝑖 ]
𝛽=
𝑉 𝑎𝑟[𝑋𝑖 ]
2. OLS intercept and slope

𝛼𝑂𝐿𝑆 = 𝑌 ̄ − 𝛽𝑂𝐿𝑆 𝑋̄
𝑆
𝛽𝑂𝐿𝑆 = 𝑌2𝑋
𝑆𝑋
9
Difference between estimator and population

## Warning: Using `size` aesthetic for lines was deprec

## i Please use `linewidth` instead.
## This warning is displayed once every 8 hours.
## Call `lifecycle::last_lifecycle_warnings()` to see w
## generated.
log of wage on education in years

2
monthly salary

10
Population regression error and OLS residual

There is a difference between the population regression error

and the OLS residual.

1. Difference between the population 𝑌𝑖 and population

regression function: 𝜀𝑖 = 𝑌𝑖 − 𝛼 − 𝛽𝑋𝑖

• Unobserved
• Random variable

2. Difference between the sample 𝑌𝑖 and the predicted line

𝑌𝑖̂ : 𝑒𝑖 = 𝑌𝑖 − 𝛼𝑂𝐿𝑆 − 𝛽𝑂𝐿𝑆 𝑋𝑖

• Observed data
• Depends on the sample
• Itself a statistics
11
Algebraic Properties
Algebraic properties

The OLS estimator benefits from a number of algebraic

properties that follow from the first order condition.

12
#1 Sample average of the residuals
𝑁
∑ 𝑖 𝑒
𝑒̄ = 𝑁𝑖
= 0: The sample average of the residuals are
(always) equal to zero.

Why? Because of the first order condition with respect to 𝑎:

𝑁
∑[𝑌𝑖 − 𝛼𝑂𝐿𝑆 − 𝛽𝑂𝐿𝑆 𝑋𝑖 ] = 0
𝑖=1

⇔
𝑁
∑ 𝑒𝑖 = 0
𝑖=1
𝑁
∑𝑖 𝑒𝑖
Hence: 𝑁 = 0.

13
#1 Sample average of the residuals

Is it always true? No.

It only holds when we allow an intercept. If we FORCE the

intercept to be zero, then it might not hold.

14
#1 Sample average of the residuals
log of wage on education in years

3.0

2.5
monthly salary

2.0

1.5

1.0

9 12 15 18
number of years of education
Data Wooldridge

15
#2 Residuals are orthogonal to 𝑋𝑖

The OLS residuals (𝑒1 , 𝑒2 , ⋯ , 𝑒𝑁 ) are orthogonal to

(𝑋1 , 𝑋2 , ⋯ , 𝑋𝑁 ).

From the first order condition with respect to 𝑏:

𝑁
∑[𝑌𝑖 − 𝛼𝑂𝐿𝑆 − 𝛽𝑂𝐿𝑆 𝑋𝑖 ]𝑋𝑖 = 0
𝑖=1

⇔
𝑁
∑ 𝑒𝑖 𝑋𝑖 = 0
𝑖=1

16
#2 Residuals are orthogonal to 𝑋

Remember that the sample covariance between 𝑒𝑖 and 𝑋𝑖 is:

𝑁
∑ 𝑒𝑖 𝑋𝑖
𝑆𝑒𝑋 = 𝑖=1 − 𝑒𝑋̄ ̄
𝑁
We know from the previous property that 𝑒 ̄ = 0, hence:
𝑁
∑ 𝑒𝑖 𝑋𝑖
𝑆𝑒𝑋 = 𝑖=1 =0
𝑁
because
𝑁
∑ 𝑒𝑖 𝑋𝑖 = 0
𝑖=1
from the FOC.
17
Decomposition in two orthogonal components

The OLS estimator decomposes 𝑌𝑖 in two orthogonal

components:
𝑌𝑖 = 𝑌𝑖̂ + 𝑒𝑖

1. 𝑌𝑖̂ depends on 𝑋𝑖

2. 𝑒𝑖 residual

We know both components are uncorrelated from the previous

algebraic results.

18
Decomposition in two orthogonal components

The regression model passes through the mean of the data,

i.e., 𝑌 ̄ and 𝑋̄ :

𝑌 ̄ = 𝛼𝑂𝐿𝑆 + 𝛽𝑂𝐿𝑆 𝑋̄

19
SST, SSE, and SSR

1. The total sum of squares (SST):

𝑁
𝑆𝑆𝑇 = ∑(𝑌𝑖 − 𝑌 ̄ )2
𝑖

2. The explained sum of squares (SSE):

𝑁 𝑁
̄
𝑆𝑆𝐸 = ∑(𝑌𝑖̂ − 𝑌 ̂ )2 = ∑(𝑌𝑖̂ − 𝑌 ̄ )2
𝑖 𝑖
̄
In your assignment, show that 𝑌 ̂ = 𝑌 ̄ .

3. The residual sum of squares (SSR):

𝑁 𝑁
𝑆𝑆𝑅 = ∑(𝑒𝑖 − 𝑒)̄ 2 = ∑(𝑒𝑖 )2
𝑖 𝑖
20
SST, SSE, and SSR

We can decompose (and you need to show in your assignment)

that

𝑆𝑆𝑇 = 𝑆𝑆𝐸 + 𝑆𝑆𝑅

21
Relation with other statistics
Relation between 𝛽𝑂𝐿𝑆 and the coefficient of corre-
lation

Remember the coefficient of correlation:

𝑆𝑋𝑌
𝑟𝑌 𝑋 = 2 𝑆2
√𝑆 𝑋 𝑌
The OLS estimator is:

𝑆𝑌 𝑋
𝛽𝑂𝐿𝑆 = 2
𝑆𝑋
Hence,
𝑆𝑌 𝑋 𝑆 𝑆 𝑆
𝛽𝑂𝐿𝑆 = = 𝑌 𝑋 𝑌 = 𝑟𝑌 𝑋 𝑌
𝑆𝑋 𝑆𝑋 𝑆𝑋 𝑆𝑌 𝑆𝑋 𝑆𝑋
22