The Linear Regression Model

(Ideal) Randomized controlled experiment: control all other variables that impact Y except X
Data Types:
Experimental data come from experiments designed to evaluate a treatment or policy or to investigate a causal effect.
Observational Data: data obtained by observing actual behavior outside an experimental setting
- Cross-sectional data is data collected for multiple entities (person, firm, country) for a single period.
- Time series data are data for a single entity collected at multiple time periods.

Simple Linear Regression: The functional form for the line: y = a ∗x + b or y = B1 * x + B0 + ui

- Panel data, also called longitudinal data, are data for multiple entities in which each entity is observed at two or more time periods.
slope in the simple linear regression model
 ”a or B1” is called slope
Slope shows how much would y (dependent variable) increase, if x (independent variable) increased by one unit *
 ”b OR BO” is called intercept. Slope:
Intercept shows the value of y (dependent variable) when the value of x (independent variable) is set to zero *
- Y= dependent variable on average
- X= independent variable
β0+β1*Xi : represents the population regression function.
ui is the regression error (disturbance term).
Criteria to choose best line: SSR=i=1∑n(Yi−b0−b1Xi)2 Intercept:

-Total Errors are Smallest (distance from errors to is closest to line)

-Unique Solution: (Y= x^2) Square each error, see which sum is the smallest: Ordinary Least Squares (OLS)
In Ordinary Least Squares (OLS) regression, the sample regression line always passes through the point (Xˉ,Yˉ), where Xˉ is the mean of the independent variable and Yˉ is the mean of
the dependent variable.
Steps
1. For all observations, sum X, divide by N= sample mean (X-bar)
2. Take observed x value for each i, subtract X-bar, square them, then sum = Denominator

1. Sum Ys, divide by N (Y-bar)

2. Take observed Y value, subtract Y-bar, square them, then sum

1. (X- X-bar)*(Y- Y-bar), then SUM = Numerator.

(BEST B1) Beta1_Hat = Numerator / Denominator
(BEST B0) Beta0_Hat = Y-bar – Beta1_hat * x-bar
Plug-Ins: Definitions
 B1(slope)= is a true population parameter, the slope of the population regression line
 B1hat= the OLS estimator

 U(error)= represents the deviation of observations from the population regression line
 Uhat= difference between Y and its predicted value Which of the following is the objective function that OLS minimizes?
 E(Y/X)= expected value of Y given values of x ∑i=1n( Yi−b0−b1Xi)^2
 Yhat= OLS predicted value of Y for given values of x The OLS sample regression line passes through the point (Xˉ,Yˉ): Yes
 OLS Residual = Actual - sample avg
Measures of Fit and Prediction Accuracy.
- Y= B0 + B1*X
T (y->x) = Total Variation of Y that comes from X
T(y) Total Variation of Y
- X-bar(mean) = sum everything /N TSS= ESS + SSR
Variance = E (X1-X-bar)^2
N R^2 is unitless, SER has same units as dependent variable
Regression R2(Goodness of Fit): the fraction of the sample variance of Y explained by X. (*Unit-Free) R^2 always between 0 and 1
- Yi = ˆYi + ˆui . R2 is defined as the ratio of the sample variance of Y-hat (predicted value) to the sample variance of Y
- If the independent variable explains none of the variations in the dependent variable, then the value of R^2 will be 0.
- The explained sum of squares (ESS) is the sum of squared deviations of the predicted value, Y-hat-i , from its average, and the total sum of squares
(TSS) is the sum of squared deviations of Yi from its average

▶ The Standard Error of the Regression: The standard error of the regression (SER) is an estimator of the standard deviation of the regression error ui :
Higher R^2 the better

: the estimated slope in the regression of growth on x

Probability
Random variable: A random variable is a numerical summary of a random outcome. X=aY+b, a=scaling factor
Discrete random variable takes on only a discrete set of values,
Var(X)=a^2⋅Var(Y
Continuous random variable takes on a continuum of possible values. (any number)
Probability distribution of a discrete random variable is the list of all
)
possible values of the variable and the probability that each value will occur.
The cumulative probability is the probability that the random variable is less than or equal to a particular value.
The cumulative probability distribution is the list of all possible values and corresponding cumulative probabilities
The cumulative probability distribution of a continuous random variable is the probability that the random variable is less
than or equal to a particular value. Similar to discrete case

Expected Value (sometimes called mean or average) is one of the central tendency measures (others being mode and median)
- The expected value of a discrete random variable is computed as a weighted average of the possible outcomes of
that random variable, where the weights are the probabilities of that outcome
linear transformation of a random variable
- Mu or u: is the mean x and sigma
- σx: standard deviation
- variance = SD^2 SD = square root of variance
Then the expected value and the standard deviation of Y are given as: 0 and 1
The Variance
- The variance and standard deviation measure the dispersion or the “spread” of a probability distribution.
The variance of a RV Y, denoted var(Y), is the expected value of the square of the deviation of Y from its mean:
- Large variance = wider distribution
Steps | ex: E(M) Expected Value or Mean: μX=E[X ]= ∑Xi⋅P(Xi) = 0*0.8 + 1*0.10 + 2*0.06 + 3*0.03 + 4*0.01 = 0.35
1. Find difference between for each variable and mean
2. Square difference
3. Multiply by respected probability
The Normal Distribution
Denoted N(μ, σ2)
- μ: mean
- σ^2: variance
The area under the curve in any interval corresponds to the probability that the RV gets
values in this interval. The area under the curve sums to 1.
 (μ − 1.96σ, μ + 1.96σ) interval contains 95% of the area under the distribution graph
Standard Normal Distribution has mean 0 and variance 1, and is usually denoted by Z: Z N(0, 1).
 Its cumulative distribution is usually denoted by Φ. Thus, Pr ( Z ≤ c)=Φ(c).
For given z score we must find in table (1)pick the integer and the first decimal from the first column, and the second decimal from the first row. The number in the intersection of the corresponding row and
column is the cumulative probability corresponding to the z score of interest.
Less Than (or equal to) = take Z Z= β^ 1 - β1 / σβ1
Greater Than = subtract by 1

X – random variable x Example: X – N (3,4) Pz (x<5) mean = 3 variance = 4

Mx or μ – mean x 5-3 = 2 = 1 = Pz (Z< 1) = 0.8413
Ox or σ - standard deviation of x square root 4 2 Z = X - mean
SD

EXAMPLE: Z- N(0,1) Pz (z < 1.96) (Probability of z less than 1.96) (Less Than = take Z) = 0.975
Pz (z > 1.52) (Greater Than = subtract by 1) = 1 - .9357 = 0.0643
Pz (.81 < Z < 1.96) = .975 - .7910 = .184
Pz (-1 < Z < 1.96) = .975 – (1-.8413)
P(1 > 1.96) = just regular value

Random Sampling
- When Y1, Y2 . . . Yn are drawn from the same distribution and are independently distributed, they are said to be independently and identically distributed (i.i.d.).
The Sampling Distribution of the OLS Estimators
Because the OLS estimators ˆβ0 and ˆβ1 are computed from a randomly drawn sample, the estimators themselves are random variables with a probability distribution—the sampling distribution—that describes
distribution—the sampling distribution—that describes the values they could take over different possible random samples. OLS passes through(Xbar,Ybar) Objective Function OLS Minimizes
- ˆβ0 and ˆβ1 are unbiased estimators of Bo and B1
- The sampling distribution of ˆβ0 and ˆβ1: is well approximated by the bivariate normal distribution if the sample is sufficiently large.
1. Central Limit Theorem
The law of large numbers states that, under general conditions, ˆβ1 will be ”close” to β1 with very high probability when n is large.
- Specifically: E ( ˆβ0) = β0, and E ( ˆβ1) = β1
The larger is n and/or the variance of Xi the more precise will the OLS estimators be.

X2 = variable that follows standard normal distribution

Estimators and Their Properties

The ˆβ1 is an example of the estimators. It is an estimator of population slope.
Xbar= sum of X1
N
B^ = Sum of (x-xbar) (y1-ybar)
Sum of (x1-xbar)^2
 β1 as an estimator of β1 possesses all three characteristics of a “good” estimator.
 ˆβ1 is unbiased, consistent and has the lowest variance among all the linear estimators of β1
 ˆβ1 is BLUE - best linear unbiased estimator
An estimator is a function of a sample of data to be drawn randomly from a population.
An estimate is the numerical value of the estimator when it is actually computed using data from a specific sample.
- An estimator is a random variable because of randomness in selecting the sample, while an estimate is a nonrandom number.
- There might be more estimators for the same population parameters
- A “good” estimator must possess the following characteristics: unbiasedness, consistency, and efficiency.
- The estimator is unbiased if E( ˆβ1) =β1, where E( ˆβ1Y ) is the mean of the sampling distribution of ˆβ1.; otherwise ˆβ1 is biased.
- UNBIASED IF: its average value, over repeated sampling for the same sample size, is equal to the population value
If the probability that ˆβ1 is within a small interval of the true value β1 approaches to 1 as the sample size increases, then ˆβ1 is consistent. (increasing sample size will increase accuracy)
- Efficient – choose unbiased estimator with SMALLER variance
Hypothesis Testing (look at population parameters)
 The hypothesis that we want to test is called the null hypothesis and is denoted H0. For ex. we want to test whether β1(NOT Bhat) =1.5. Then, H0: βeduc. 1 =1.5$. In general, H0: β1= βh 1 where βh
1 is the hypothesized value.
 (does not equal, NEVER use accept)
 The hypothesis that we compare the null hypothesis to is called the alternative hypothesis and is denoted H1. The alternative hypothesis holds if the null hypothesis is not true.
 If we have enough evidence against the null evidence, then we reject the null hypothesis. If we do not have enough evidence then we fail to reject the null hypothesis. We never accept the null
hypothesis
 The p-value, also called the significance probability, is the probability of drawing a statistic at least as adverse to the null hypothesis as the one you actually computed in your sample, assuming the
null hypothesis is correct.
How to calculate the p-Value
 State the null hypothesis.
 Calculate the test statistics based on sample data.
 Standardize the test statistics.
 Determine the distribution of the standardized test statistics.
 Calculate the p-value.
 Make the decision

T stats=Z stats
T statistic = Estimator – hypothesized value (Coefficient of Slope)
Standard Error of Estimator
small p-value = reject null (less than 0.05)
large p-value = fail to reject null (higher than 0.05)
Larger (in absolute terms) Test statistics = smaller p value
Smaller test statistics= higher p value
Negative = same (b/c absolute value takes both – and positive, multiply by 2)

P value: includes AREA IN BOTH ENDS= ABSOLUTE VALUE Two tail= * 2

- | x | > 1 (less than -1 AND greater than 1) multiply by 2 for answer
EXAMPLE: Y=Bo+B1*(X). B1hat=5. Ho: B1=2
H1: B1=/(doesn’t equal) 2 -> P2(B1sac-hat>5) = P2 (B1hat – 2) = 5 – 2 = P2(z > 1.5) = 1 – 0.9332(z) = .066 =
0.66*(2) = 0.132
(O) SE 2
- Z= X- Mx
Ox
X – random varuavke x
Mx – mean x
Ox- standard deviation of x

Hypothesis formula for example:

- Ho: B1=1.5
- H1: Bo=/(doesn’t equal) 1.5
- SE(B1hat) = 0.008
 - Bactual = 1.52
-
B (beta) is actual coefficient B^ (beta hat) is the estimator of B, estimated P-value represents the probability of observing data as extreme, or more extreme than
(greater than), the data actually observed, assuming the null hypothesis is true