An econometric model for Linear Regression using Statistics

International Research Journal of Engineering and Technology (IRJET) e-ISSN: 2395-0056
Volume: 10 Issue: 04 | Apr 2023 www.irjet.net p-ISSN: 2395-0072
© 2023, IRJET | Impact Factor value: 8.226 | ISO 9001:2008 Certified Journal | Page 886
An econometric model for Linear Regression using Statistics
Renvil Dsa1, Remston Dsa2
1Renvil Dsa Fr. Conceicao Rodrigues College of Engineering, Mumbai, India
2Remston Dsa New York University, New York, USA
---------------------------------------------------------------------***---------------------------------------------------------------------
Abstract - This research paper discusses the econometric
modeling approach of linear regression using statistics.
Linear regression is a widely used statistical technique for
modeling the relationship between a dependent variable
and one or more independent variables. The paper begins by
introducing the concept of linear regression and its basic
assumptions.
The univariate and multivariate linear regression models
are discussed, and the coefficients of the regression models
are derived using statistics. The matrix form of the Simple
Linear Regression Model is presented, and the properties of
the Ordinary Least Squares (OLS) estimators are proven.
Hypothesis testing for multiple linear regression is also
discussed in the matrix form.
The paper concludes by emphasizing the importance of
understanding the econometric modeling approach of linear
regression using statistics. Linear regression is a powerful
tool for predicting the values of the dependent variable
based on the values of the independent variables, and it can
be applied in various fields, including economics, finance,
and social sciences. The paper's findings contribute to the
understanding of the linear regression model's practical
application and highlight the need for rigorous statistical
analysis to ensure the model's validity and reliability.
Key Words: econometric model, linear regression,
statistics, univariate regression, multivariate
regression, OLS estimators, matrix form, hypothesis
testing.
1. INTRODUCTION
Linear regression is a powerful statistical modeling
technique widely used to analyze the relationship between
a dependent variable and one or more independent
variables. In a simple linear regression model, the
dependent variable is assumed to be a linear function of
one independent variable, while in a multiple linear
regression model, the dependent variable is a linear
function of two or more independent variables.
The coefficients of the linear regression model are
represented by beta (β) values, which are constants that
determine the slope and intercept of the regression line.
The beta values are estimated using the Ordinary Least
Squares (OLS) method, which involves minimizing the
sum of squared errors between the predicted values and
the actual values of the dependent variable.
The matrix form of the linear regression model is a useful
tool for understanding the relationship between the
dependent and independent variables. The matrix form
allows for a more efficient calculation of the beta values
and the variance-covariance matrix of the beta values.
Partial derivatives are used to derive the expected values
and variation of the beta values. The expected values of
the beta values are equal to the true beta values, and the
variation of the beta values can be used to derive
confidence intervals and hypothesis tests for the beta
values.
The relationship between the beta values and the normal
distribution is important in understanding the properties
of the OLS estimators. The beta values are normally
distributed, and their variance-covariance matrix can be
used to derive confidence intervals and hypothesis tests.
Hypothesis testing is an essential component of linear
regression analysis. The null hypothesis is typically that
the beta value is equal to zero, indicating that there is no
relationship between the independent variable and the
dependent variable. The t-test and F-test are commonly
used to test hypotheses about the beta values.
The ANOVA matrix is a useful tool for decomposing the
total variation in the dependent variable into the
explained variation (sum of squared regression) and
unexplained variation (sum of squared error). The sum of
squared total is the sum of squared deviations of the
dependent variable from its mean.
The relationship between the sum of squared regression,
sum of squared error, and sum of squared total with the
mean squared error and the degrees of freedom is
important in interpreting the results of the linear
regression analysis.
In summary, linear regression is a powerful statistical
modeling technique that can be used to analyze the

relationship between a dependent variable and one or
more independent variables. The beta values, matrix form
of the model, partial derivative concept, expected values
and variation of beta values, relationship with normal
distribution, hypothesis testing, ANOVA matrix, formulas
for sum of squared regression, sum of squared error, sum
of squared total, and relationship with mean squared
error, t-test, and F-test with degrees of freedom are
important concepts in understanding and interpreting the
results of linear regression analysis.
2. METHODS
2.1 Deriving the coefficient of the regression
model for the simple linear regression model
The important result that is needed to derive ̂ and ̂
Suppose, is the sample mean of y then is defined as 1
over n times the summation from i = 1 to n of
= ∑
It also means that is equal to the summation from
i = 1 to n of
= ∑ ------------- ( a )
Similarly,
is the sample mean of y and is defined as 1 over n times
the summation from i = 1 to n of
= ∑
It also means that is equal to the summation from
i = 1 to n of .
= ∑ ------------ ( b )
There are some important results that we need to know
using the summation operator,
∑ ( - )( -
) =
The written L.H.S. can also
be written as:
∑ - --------
(1)
∑ ( - ) --------
(2)
∑ ( - ) --------
(3)
Table 1: Different forms of ∑ ( - )( - ) [1]
Derive: ∑ = ∑ -
L.H.S. = ∑
= ∑ ( - - + )
= ∑ - ∑ - ∑ +
From equation (a) and equation (b)
= ∑ - + -
= ∑ -
= R.H.S [Equation (1)]
Derive: ∑ = ∑ ( - )
L.H.S. = ∑
= ∑ ( - - + )
= ∑ - ∑ - ∑ +
From equation (b)
∑ - - ∑ +
∑ - ∑
∑ ( - )
= R.H.S [Equation (2)]

Derive: ∑ = ∑ ( - )
L.H.S. = ∑
= ∑ ( - - + )
= ∑ - ∑ - ∑ +
From equation (a)
∑ - ∑ - +
∑ - ∑
∑ ( - )
= R.H.S. [Equation (3)]
Deriving the formula for ̂ and ̂ [2]
1. S.S.E: The sum of squared error is equal to the
summation from i = 1 to n of the square of the difference
between and ̂
2. is the real observation and ̂ is the predicted
observation
3. The goal is to minimize the distance between the
real observation and the predicted observation.
4. The equation is a quadratic equation and the
objective is to find the minimum point and not the
maximum point.
S.S.E = ∑ ( ̂ )
substituting ̂ = ̂ ̂
S.S.E = ∑ ( ̂ ̂ )
Applying Partial Derivative with respect to ̂
̂
= 2 ∑ ( ̂ ̂ )
(-1)
To find the maxima or minima,
Equivalently, the derivative equals 0.
0 = - 2∑ ( ̂ ̂ )
0 = ∑ - ̂ - ̂ ∑
from equation (a) and equation (b)
0 = - ̂ - ̂
0 = - ̂ - ̂
̂ = - ̂
Applying Partial Derivative with respect to ̂
= - 2 ∑ ( ̂ ̂ )
To find the maxima or minima,
Equivalently, the derivative equals 0.
= - 2 ∑ ( ̂ ̂ )
0 = ∑ - ̂ ∑ - ̂ ∑
From equation (b)
0 = ∑ - ̂ - ̂ ∑
As ̂ = - ̂
0 = ∑ - ̂ - ̂ ∑
0 = ∑ - + ̂ - ̂ ∑
From equation (1)
0 = ∑ + ̂ - ∑ )
- ̂ - ∑ ) = ∑
̂ = ∑
∑
As ∑ = (∑ )
̂ = ∑
∑
2.2 Matrix form of the Simple Linear Regression
Model

Here, is a 2 x 2 symmetric matrix
det( ∑ ∑ ∑
= n∑ - [∑
= n∑ -
= n(∑ - n )
= ( ∑ - n ) = ∑
det( ) = n∑ = n
̂
̂ X
Scalar form of Y is
= +
For Y, the intercept, is the slope, and is the error
term epsilon. This is known as the scalar form. Suppose
there are n data points, then i ranges from 1 to n
The Matrix Form of Y is
[3]
2.3 SLR Matrix Form Proof of Properties of
OLS(Ordinary Least Squares) Estimators
The first assumption is that the expected value of epsilon
is equal to zero for all i and these i are in the range of i = 1,
2,..., n, where n is the number of data points.
The mathematical representation is stated below.
E[ ] = 0 i, where i = 1, 2, 3,..., n.
The anticipated value of vector epsilon is going to be zero
vector or in rows, so it's essentially epsilon, the expected
value of epsilon is going to be simply a vector of zero,
where there will be n rows, So this is the first assumption
that will be evaluated when working with the simple
linear regression model in matrix form.
Expected value and Variance of [4]
E[ ] =
The second assumption is about the variance of the
epsilon
Var[] i. The epsilon vector's variance matrix is equal to
sigma squared times the identity matrix, with dimensions
of n by n.

Var[ ] =
So the variance of epsilon is equal to sigma squared on all
diagonal terms up to n rows, and all these entries should
be zero other than diagonal terms.
Var[ ] = I
The third assumption is that the epsilon values are
uncorrelated. If they are uncorrelated then that means
that for any Epsilon values and , the variance of
and is equal to zero. This also means that the
covariances of and are equal to zero. Thus,
where i j, the covariance of and is equal to zero.
As ’s are uncorrelated
The mathematical representation of the above third
assumption is:
Var( ) = 0
Cov( ) = 0
Cov( ) = 0 i
As the epsilon of i's( ’s) are uncorrelated with each other,
and based on these three assumptions, when combined
with the fourth assumption, allow for a very good
powerful construction of the simple linear regression
model, from which various results can be derived.
The fourth and last assumption is that the error terms are
normally distributed.
The epsilon i( ) follows the normal distribution with
some mean( ) and some variance( ) .
N( , ) [5]
From Assumptions 1 and 2, we know that = E[ ] =
and Var[ ] = where is the standard deviation.
Also, from assumption 3 where i, where i = 1, 2, 3,........., n
Cov( ) = 0 i which means these epsilon are
independent and are identically distributed.
Hence,
N( , )
Hence, ̂ = Y - ̂
̂ = Y - ̂
Derive and show that E[ ̂] =
As ̂ =
E[ ̂] = E[ ]
Note: is not stochastic, it’s not random at all, it
is all just constant known values that are multiplied by
each other.
E[ ̂] = E[ ]
As, Y = X +
E[Y] = E[X + ]
= E[X ] + E[ ]
= E[X ] ----(From Assumption 1, E[ ] = 0)
= X
Hence,
E[ ̂] = E[ ]
= ( X)
As ( X) = where X is symmetric matrix
E[ ̂] =
E[ ̂] =
Derive and show that Var[ ] = ̂
Var[ ̂] = Var[ ]
Let B =
Var[B ]
= B Var[ ]
= B Var[X + ] = B Var[ ] + B Var[ ]
Here, in Var[ ] is not random and continuous hence
we can assume that Var[ ] = 0

From Assumption 2 where Var[ ] = I
Var[ ̂] = B I
= I
As is symmetric, = X
= X
Var[ ̂] = I
= I X
=
As ( X) = where X is symmetric matrix
= I
=
As are unobservable, cannot be computed hence we
assume ̂is an estimator of
Hence, Var[ ̂] = ̂
2.4 Hypothesis testing for Multiple Linear
Regression- Matrix form
Y = X
here, ̂ = Y - ̂ where Y above in the equation is the actual
observation and ̂ is the fitted observation.
The following equation gives the sum of squared
errors(SSE):
SSE = ̂ ̂
Therefore, the above function can be written as:
SSE = ̂ ̂
To fit the regression model, the goal should be to minimize
the SSE,
minimize(SSE) = minimize(̂ ̂)
= minimize[ ̂ ̂ ]
Here, ̂ =
where
E[ ̂] =
Var[ ̂]= ̂
̂
̂ = ̂
= MSE = = ̂ ̂
Here, p regressors of interest 1 is for and n is the
number of samples in the population
Let Se be the Standard Error
Hence,
Se( ̂) =
√ ̂
̂
= √ ̂
Here, Se( ̂ ) =
√ ̂

Hypothesis to be tested: : ̂ = 0 vs : ̂ 0
[6]
Test Statistic = ̂ ̂
̂
= ̂
̂
= ̂
√̂
Hypothesis testing on a linear combination of regression
parameters in the matrix form of the linear regression
problem.
Below is the general formula of the regression model that
is in matrix form:
Y = X
Scalar representation of the regression parameters:
Y = + + + + ……….+ .
Linear combination:
Let ̂ = + + +...............+ .
The , , ,...., in the above equation represent the
constants.
̂ ̂
Hypothesis to be tested: :̂ = vs : ̂
Expected value of ̂ :
E[̂ ] = E[̂ ̂] = E[ ̂] = = L
is the row vector of the constants.
Variance of ̂ :
̂ = = c = ̂ c
Formulation of the test statistics is a t-test where the
parameter of interest (̂) follows the normal distribution
with mean of L and variance of ̂ c.
̂ N( , ̂ c )
Test statistics T:
T = ̂
̂
: It is the value of the linear combination under the null
hypothesis
where ̂ = √ ̂
Then test statistics becomes,
T =
√̂
ANOVA Matrix Form [7]:
SSR : Sum of squares Regression : ∑ (̂ )
SSE : Sum of squares Errors : ∑ ̂
SST : Sum of Squared Total : ∑ ( )
Source of
variation
Sum of
Squares
degree
s of
freedo
m
Mean
Square
F
statistic
Regresion SSR p MSE = F =
Errors SSE n-p-1 MSE =
=
̂

Total SST = SSR
+ SSE
n-1 MST =
Table 2: ANOVA matrix form [8]
F =
Hypothesis to be tested: : = = = =...=
vs : F [9]
f F > then reject else accept
3. CONCLUSIONS
In conclusion, this research paper has explored the
econometric modeling approach of linear regression using
statistics. The paper has presented the fundamental
concepts of linear regression, such as the basic
assumptions and the definition of the dependent and
independent variables.
The paper has then discussed the univariate and
multivariate linear regression models, and the coefficients
of the regression models have been derived using
statistical methods. The matrix form of the Simple Linear
Regression Model has been presented, and the properties
of the Ordinary Least Squares (OLS) estimators have been
proven.
Finally, the paper has discussed hypothesis testing for
multiple linear regression in the matrix form. The
importance of understanding the econometric modeling
approach of linear regression using statistics has been
emphasized, and the paper's findings contribute to the
understanding of the practical application of the linear
regression model.
Overall, linear regression is a powerful tool for predicting
the values of the dependent variable based on the values
of the independent variables. It can be applied in various
fields, including economics, finance, and social sciences.
Rigorous statistical analysis is necessary to ensure the
model's validity and reliability, and this paper has
provided insights into the statistical methods used in
linear regression modeling.
REFERENCES
[1] Introduction to Linear Regression Analysis, 5th
Edition" by Montgomery, Peck, and Vining
[2] Applied Linear Regression, 3rd Edition" by Weisberg
[3] Matrix Approach to Simple Linear Regression" by Kato
and Hayakawa
[4] Linear Regression Analysis: Theory and Computing"
by Liu and Wu
[5] Linear Models with R, 2nd Edition" by Faraway
[6] Applied Regression Analysis and Generalized Linear
Models, 3rd Edition" by Fox
[7] Statistical Models: Theory and Practice" by Freedman,
Pisani, and Purves
[8] Statistical Inference" by Casella and Berger
[9] Probability and Statistics for Engineering and the
Sciences, 9th Edition" by Devore

An econometric model for Linear Regression using Statistics

Recommended

More Related Content

Similar to An econometric model for Linear Regression using Statistics (20)

More from IRJET Journal (20)

Recently uploaded (20)

An econometric model for Linear Regression using Statistics