aktuwallah.

com

REGRESSION ANALYSIS

In our discussion of regression analysis, we will first focus our discussion on simple linear

regression and then expand to multiple linear regression. The reason for this ordering is not because

simple linear regression is so simple, but because we can illustrate our discussion about simple

linear regression in two dimensions and once the reader has a good understanding of simple linear

regression, the extension to multiple regression will be facilitated. It is important for the reader to

understand that simple linear regression is a special case of multiple linear regression. Regression

models are frequently used for making statistical predictions -- this will be addressed at the end of

this chapter.

Simple Linear Regression

Simple linear regression analysis is used when one wants to explain and/or forecast the variation in

a variable as a function of another variable. To simplify, suppose you have a variable that exhibits

variable behavior, i.e. it fluctuates. If there is another variable that helps explain (or drive) the

variation, then regression analysis could be utilized.

An Example

Suppose you are a manager for the Pinkham family, which distributes a product
whose sales volume varies from year to year, and you wish to forecast the next
years’ sales volume. Using your knowledge of the company and the fact that its
marketing efforts focus mainly on advertising, you theorize that sales might be a
linear function of advertising and other outside factors. Hence, the model’s
mathematical function is:

SALESt = B0 + B1 ADVERTt + Error

Where: SALESt represents Sales Volume in year t

ADVERTt represents advertising expenditures in year t
B0 and B1 are constants (fixed numbers)
and Errort is the difference between the actual sales volume
value in year t and the fitted sales volume value in year t

36
 aktuwallah.com

Note: the Errort term can account for influences on sales volume other than advertising.

Ignoring the error term one can clearly see that what is being proposed is a linear equation (straight
line) where the SALESt value depends on the value of ADVERTt. Hence, we refer to SALESt as the
dependent variable and ADVERTt as the explanatory variable.
To see if the proposed linear relationship seems appropriate we gather some data and plot the data
to see if a linear relationship seems appropriate. The data collected is yearly, from 1907 - 1960,
hence, 54 observations. That is for each year we have a value for sales volume and a value for
advertising expenditures, which means we have 54 pairs of data.

Year Advert Sales

1907 608 1016

1908 451 921
. . .
. . .
. . .
. . .
1959 644 1387
1960 564 1289

To get a feel for the data, we plot (called a scatter plot) the data as is shown as Figure 1. (Hereafter,
the scatter plot will be called plot.)

Scatter Plot of Sales vs. Advertising

3900

3400

2900
Sales
2400

1900

1400

900

0 0.4 0.8 1.2 1.6 2

Advertising (X 1000)

37
 aktuwallah.com

Figure 1. Scatter Plot of Sales vs. Advertising

As can be seen, there appears to be a fairly good linear relationship between sales (SALES) and

advertising (ADVERT) (at least for advertising less than 1200 ~ note scaling factor for ADVERT x

1000). At this point, we are now ready to conclude the specification phase and move on to the

estimation phase where we estimate the best fitting line.

Summary: For a simple linear regression model, the functional relationship is: Yt = B0 + B1 Xt + Et

and for our example the dependent variable Yt is SALESt and the explanatory variable is ADVERTt.

We suggested our proposed model in the example based upon theory and confirmed it via a visual

inspection of the scatter plot for SALESt and ADVERTt. Note: In interpreting the model we are

saying that SALES depends upon ADVERT in the same time period and some other influences,

which are accounted for by the ERROR term.

Estimation

We utilize the computer to perform the estimation phase. In particular, the computer will calculate

the “best” fitting line, which means it will calculate the estimates for B 0 and B1. The results are

Regression Analysis - Linear model: Y = a + b*X

--------------------------------------- --------------------------------------
Dependent variable: sales
Independent variable: advert
--------------------------------------- --------------------------------------
Standard T
Parameter Estimate Error Statistic P-Value
--------------------------------------- --------------------------------------
Intercept 488.833 127.439 3.83582 0.0003
Slope 1.43459 0.126866 11.3079 0.0000
--------------------------------------- --------------------------------------

Analysis of Variance
--------------------------------------- --------------------------------------
Source Sum of Squares Df Mean Square F-Ratio P-Value
--------------------------------------- --------------------------------------
Model 1.50846E7 1 1.50846E7 127.87 0.0000
Residual 6.13438E6 52 117969.0
--------------------------------------- --------------------------------------
Total (Corr.) 2.1219E7 53

Correlation Coefficient = 0.843149

R-squared = 71.0901 percent
Standard Error of Est. = 343.466

38
 aktuwallah.com

Table 1.

Since B0 is the intercept term and B1 represents the slope we can see that the fitted line is:

SALESt = 488.8 + 1.4 ADVERTt

The rest of the information presented in Table 1 can be used in the diagnostic checking phase that

we discuss next.

Diagnostic Checking

Once again the purpose of the diagnostic checking phase is to evaluate the model’s adequacy. To

do so, at this time we will restrict our analysis to just a few pieces of information in Table 1.

First of all, to see how well the estimated model fits the observed data, we examine the R-squared

(R2) value, which is commonly referred to as the coefficient of determination. The R 2 value denotes

the amount of variation in the dependent variable that is explained by the fitted model. Hence, for

our example, 71.09 percent of the variation in SALES is explained by our fitted model. Another

way of viewing the same thing is that the fitted model does not explain 28.91 percent of the

variation in SALES.

A second question we are able to address is whether the explanatory variable, ADVERTt, is a

significant contributor to the model in explaining the dependent variable, SALESt. Thus, for our

example, we ask whether ADVERTt is a significant contributor to our model in terms of explaining

SALESt. The mathematical test of this question can be denoted by the hypothesis:

H0 : 1 = 0
H1 : 1  0

which makes sense, given the previous statements, when one remembers that the model we

proposed is: SALESt = B0 + B1 ADVERTt + ERRORt

39
 aktuwallah.com

Note: If B1 = 0, (i.e. the null hypothesis is true), then changes in ADVERTt will not produce a

change in SALESt. From Table 1, we note that the p-value (probability level) for the hypothesis test,

which resides on the line labeled slope, is 0.00000 (truncation). Since the p-value is less than  =.

05, we reject the null hypothesis and conclude that ADVERTt is a significant explanatory variable

for the model, where SALESt is the dependent variable.

An Example

To further illustrate the topic of simple linear regression and the model building
process, we consider another model using the same data set. However, instead of
using advertising to explain the variation in sales, we hypothesize that a good
explanatory variable is to use sales lagged one year. Recall that our time series data is
in yearly intervals, hence, what we are proposing is a model where the value of sales
is explained by its amount one time period (year) ago. This may not make as much
theoretical sense [to many] as the previous model we considered, but when one
considers that it is common in business for variables to run in cycles, it can be seen to
be a valid possibility.

Plot of sales vs lag(sales,1)

3900

3400

2900
sales

2400

1900

1400

900
900 1400 1900 2400 2900 3400 3900

lag(sales,1)

Figure 2. Plot of Sales vs. Lag(Sales,1)

Looking at Figure 2 as shown above, one can see that there appears to be a linear relationship

between sales and sales one time period before. Thus the model being specified is:

40
 aktuwallah.com

SALESt = B0 + B1 SALESt-1 + Errort

Where: SALESt represents sales volume in year t

SALESt-1 represents sales volume in year t-1
B0 and B1 are constants (fixed numbers)
and Errort is the difference between the actual sales volume value in year t and the fitted sales
volume value in year t

Estimation

Using the computer, (StatGraphics software), we are able to estimate the parameters B 0 and B1 as is

shown in Table 2.

Regression Analysis - Linear model: Y = a + b*X

-----------------------------------------------------------------------------
Dependent variable: sales
Independent variable: lag(sales,1)
-----------------------------------------------------------------------------
Standard T
Parameter Estimate Error Statistic P-Value
-----------------------------------------------------------------------------
Intercept 148.303 98.74 1.50196 0.1393
Slope 0.922186 0.050792 18.1561 0.0000
-----------------------------------------------------------------------------

Analysis of Variance
-----------------------------------------------------------------------------
Source Sum of Squares Df Mean Square F-Ratio P-Value
-----------------------------------------------------------------------------
Model 1.77921E7 1 1.77921E7 329.64 0.0000
Residual 2.75265E6 51 53973.5
-----------------------------------------------------------------------------
Total (Corr.) 2.05447E7 52

Correlation Coefficient = 0.9306

R-squared = 86.6017 percent
Standard Error of Est. = 232.322

hence, the fitted model is:

SALESt = 148.30 + 0.92 SALESt-1

41
 aktuwallah.com

Diagnostic Checking

In evaluating the attributes of this estimated model, we can see where we are now able to fit the

variation in sales better, as R2, the amount of explained variation in sales, has increased from 71.09

percent to 86.60 percent. Also, as one probably expects, the test of whether SALESt-1 does not have

a significant linear relationship with SALESt is rejected. That is, the p-value for

H0: 1 = 0
H1: 1  0

is less than alpha (.00000 < .05). There are other diagnostic checks that can be performed but we

will postpone those discussions until we consider multiple linear regression. Remember: simple

linear regression is a specific case of multiple linear regression.

Update

At this point, we have specified, estimated and diagnostically checked (evaluated) two simple linear

regression models. Depending upon one’s objective, either model may be utilized for explanatory

or forecasting purposes.

Using Model

As discussed previously, the end result of regression analysis is to be able to explain the variation

of sales and/or to forecast value of SALESt. We have now discussed how both of these end results

can be achieved.

Explanation

As suggested by Table 1 and 2, when estimating the simple linear regression models, one is

calculating estimates for the intercept and slope of the fitted line (B 0 and B1 respectively). The

interpretation associated with the slope (B1) is that for a unit change in the explanatory variable it

42
 aktuwallah.com

represents the respective change in the dependent variable along the forecasted line. Of course, this

interpretation only holds in the area where the model has been fitted to the data. Thus usual

interpretation for the intercept is that it represents the fitted value of the dependent variable when

the independent (explanatory) variable takes on a value of zero. This is correct only when one has

used data for the explanatory variable that includes zero. When one does not use values of the

explanatory variable near zero, to estimate the model, then it does not make sense to even attempt to

interpret the intercept of the fitted line.

Referring back to our examples, neither data set examined values for the explanatory variables

(ADVERTt and SALESt-1) near zero, hence we do not even attempt to give an economic

interpretation to the intercepts. With regards to the model:

SALESt = 488.83 + 1.43 ADVERTt

the interpretation of the estimated slope is that a unit change in ADVERT ($1,000) will generate, on

the average, a change of 1.43 units in SALESt ($1,000). For instance, when ADVERTt increases

(decreases) by $1,000 the average effect on SALESt will be an increase (decrease) of $1,430. One

caveat, this interpretation is only valid over the range of values considered for ADVERT, which is

the range from 339 to 1941 (i.e., minimum and maximum values of ADVERT).

Forecasting

Calculating the point estimate with a linear regression is a very simple process. All one needs to do

is substitute the specific value of the explanatory variable, which is being forecasted, into the fitted

model and the output is the point estimate.

For example, referring back to the model:

SALESt = 488.8 + 1.4 ADVERTt

if one wishes to forecast a point estimate for a time period when ADVERT will be 1200 then the

point estimate is:

43
 aktuwallah.com

2168.8 = 488.8 + 1.4 (1200)

Deriving a point estimate is useful, but managers usually find more information in confidence

intervals. For regression models, there are two sets of confidence intervals for point forecasts that

are of use as shown in Figure 3 on the next page.

Regression of Sales on Advertising

3900

3400

2900
Sales
2400

1900

1400

900

0 0.4 0.8 1.2 1.6 2

(X 1000)
Advertising

Figure 3. Regression of Sales on Advertising

Viewing Figure 3 as shown1, one can see two sets of dotted lines, each set being symmetric about
the fitted line. The inner set represents the limits (upper and lower) for the mean response for a
given input, while the other set represents the limits of an individual response for a given input. It is
the outer set that most managers are concerned with, since it represents the limits for an individual
value. For right now, it suffices to have an intuitive idea of what the confidence limits represent
and graphically what they look like. So for an ADVERT value of 1200 (input), one can visually see
that the limits are approximately 1500 and 2900. (The values are actually 1511 and 2909.) Hence,
when advertising is $1,200 for a time period ( ADVERTt = 1,200) then we are 95 percent confident
that sales volume (SALESt) will be between approximately 1,500 and 2,900.

1
Figure 3 was obtained by selecting Plot of Fitted Line under the Graphical Options icon.

44
 aktuwallah.com

MARKET MODEL - Stock Beta’s

An important application of simple linear regression, from business, is used to calculate the ß of a

stock2. The ß’s are measures of risk and used by portfolio managers when selecting stocks.

The model used (specified) to calculate a stock ß is:

Rj,t =  +  Rm,t + t

Where: Rj,t is the rate of return for the jth stock in time periodt
Rm,t is the market rate of return in time periodt
t is the error term in time periodt
 and  are constants

To illustrate the above model, we will use data that resides in the data file SLR.SF3. In particular,

we will calculate ‘s for Anheuser Busch Corporation, the Boeing Corporation, and American

Express using the New York Stock Exchange (NYSE - Finance) as the “market” portfolio. The data

in the file SLR.SF3 has already been converted from monthly values of the individual stock prices

and dividends to represent the monthly rate of returns (starting with June 1995).

For all three stocks, the model being specified and estimated follows the form stated in the equation

shown above, the individual stocks rate of returns will be used as the dependent variable and the

NYSE rate of returns will be used as the independent variable.

2
For an additional explanation on the concept of stock beta’s, refer to the Appendix.

45
 aktuwallah.com

1. Anheuser Busch Co. (AnBushr)

Using the equation, the model we specify is AnBushRt =  +  DJIAVGRt + t.

The estimation results are shown below in Table 3:

Regression Analysis - Linear model: Y = a + b*X

--------------------------------------- --------------------------------------
Dependent variable: ROR Anheuser
Independent variable: DJIAVGR
--------------------------------------- --------------------------------------
Standard T
Parameter Estimate Error Statistic P-Value
--------------------------------------- --------------------------------------
Intercept 0.0122761 0.00738469 1.66238 0.1018
Slope 0.411569 0.153443 2.68223 0.0095
--------------------------------------- --------------------------------------

Analysis of Variance
--------------------------------------- --------------------------------------
Source Sum of Squares D f Mean Square F-Ratio P-Value
--------------------------------------- --------------------------------------
Model 0.021278 1 0.021278 7.19 0.0095
Residual 0.171541 58 0.00295761
--------------------------------------- --------------------------------------
Total (Corr.) 0.192819 59

Correlation Coefficient = 0.332193

R-squared = 11.0352 percent
Standard Error of Est. = 0.0543839

Table 3

As shown in the estimation results, the estimated  for Anheuser Busch Co. is 0.411. Note that with

a p-value of 0.00254, the coefficient of determination, R-squared, is only 11.04 percent, which

indicates a poor fit of the data. However, at this point we only wish to focus on the estimated .

46
 aktuwallah.com

2. The Boeing Co.

The model we specify, using equation (1) is BoeingRt =  +  DJIAVGRt + t

The results appear below in Table 4.

Regression Analysis - Linear model: Y = a + b*X

--------------------------------------- --------------------------------------
Dependent variable: ROR Boeing
Independent variable: DJIAVGR
--------------------------------------- --------------------------------------
Standard T
Parameter Estimate Error Statistic P-Value
--------------------------------------- --------------------------------------
Intercept -0.00894732 0.00840995 -1.0639 0.2918
Slope 1.08754 0.174746 6.22357 0.0000
--------------------------------------- --------------------------------------

Analysis of Variance
--------------------------------------- --------------------------------------
Source Sum of Squares D f Mean Square F-Ratio P-Value
--------------------------------------- --------------------------------------
Model 0.148573 1 0.148573 38.73 0.0000
Residual 0.22248 58 0.00383585
--------------------------------------- --------------------------------------
Total (Corr.) 0.371053 59

Correlation Coefficient = 0.63278

R-squared = 40.041 percent
Standard Error of Est. = 0.0619343

Table 4

Note that the estimated  for The Boeing Co. is 1.08 while the R2 value is 40.04percent.

47
 aktuwallah.com

3. American Express

The model we specify, using the equation is as follows:

AmExpRt =  +  DJIAVGRt + t

which can be estimated using StatGraphics

The results appear in Table 5:

Regression Analysis - Linear model: Y = a + b*X

--------------------------------------- --------------------------------------
Dependent variable: ROR Amer Express
Independent variable: DJIAVGR
--------------------------------------- --------------------------------------
Standard T
Parameter Estimate Error Statistic P-Value
--------------------------------------- --------------------------------------
Intercept 0.00844987 0.00672318 1.25683 0.2139
Slope 1.39088 0.139698 9.95637 0.0000
--------------------------------------- --------------------------------------

Analysis of Variance
--------------------------------------- --------------------------------------
Source Sum of Squares D f Mean Square F-Ratio P-Value
--------------------------------------- --------------------------------------
Model 0.243012 1 0.243012 99.13 0.0000
Residual 0.142185 58 0.00245146
--------------------------------------- --------------------------------------
Total (Corr.) 0.385197 59

Correlation Coefficient = 0.794278

R-squared = 63.0877 percent
Standard Error of Est. = 0.0495123

Table 5

The estimation results indicate that the  is 1.39, with an R-squared value of 63.09 percent.

48
 aktuwallah.com

Summary

Using monthly values from June 1995 to June 2000, we utilized simple linear regression to estimate

the ‘s of Anheuser Busch Co. (0.411), the Boeing Co. (1.09), and American Express (1.39). Note

that the closer the ‘s are to 1.0, the closer the stocks move with the market. What does that imply

about Anheuser Busch Corporation, the Boeing Corporation, and American Express?

The risk contribution to a portfolio of an individual stock is measured by the stock’s beta

coefficient. Analysts review the market outlooks - if the outlook suggests a market decline, stocks

with large positive coefficients might be sold short. Of course, the historical measure of  must

persist at approximately the same level during the forecast period. (Additional discussion about

stock betas appears in the Appendix.)

49
 aktuwallah.com

Multiple Linear Regression

Referring back to the Pinkham data, suppose you decided that ADVERTt contained information

about SALESt that lagged value of SALESt (i.e. SALESt-1) did not, and vice versa, and that you

wished to regress SALESt on both ADVERTt and SALESt-1; the solution would be to use a multiple

regression model. Hence, we need to generalize our discussion of simple linear regression models

by now allowing for more than one explanatory variable, hence the name multiple regression.

[Note: more than one explanatory variable, hence we are not limited to just two explanatory

variables.]

Specification: Going back to our example, if we specify a multiple linear regression model where

SALESt is again the dependent variable and ADVERTt and SALESt-1 are the explanatory variables,

then the model is:

SALESt = B0 + B1 ADVERTt + B2 SALESt-1 + ERRORt

where: B0, B1, and B2 are parameters (coefficients).

Estimation: To obtain estimates for B0, B1, and B2 via StatGraphics, the criterion of least squares

still applies, the mathematics employed involves using matrix algebra. It suffices for the student to

understand what the computer is doing on an intuitive level; i.e. the best fitting line is being

generated. The results from the estimation phase are shown in Table 7.

50
 aktuwallah.com

Multiple Regression Analysis

-----------------------------------------------------------------------------
Dependent variable: sales
-----------------------------------------------------------------------------
Standard T
Parameter Estimate Error Statistic P-Value
-----------------------------------------------------------------------------
CONSTANT 138.691 95.6602 1.44982 0.1534
lag(sales,1) 0.759307 0.0914561 8.30242 0.0000
advert 0.328762 0.155672 2.11189 0.0397
-----------------------------------------------------------------------------

Analysis of Variance
-----------------------------------------------------------------------------
Source Sum of Squares Df Mean Square F-Ratio P-Value
-----------------------------------------------------------------------------
Model 1.80175E7 2 9.00875E6 178.23 0.0000
Residual 2.52722E6 50 50544.3
-----------------------------------------------------------------------------
Total (Corr.) 2.05447E7 52

R-squared = 87.699 percent

R-squared (adjusted for d.f.) = 87.2069 percent
Standard Error of Est. = 224.821
Mean absolute error = 173.307
Durbin-Watson statistic = 0.916542

Table 7

Diagnostic Checking

We still utilize the diagnostic checks we discussed for simple linear regression. We are now going

to expand that list and include additional diagnostic checks, some require more than one

explanatory variable but most also pertain to simple linear regression. We waited to introduce some

of the checks [that also pertain to simple linear regression] because we didn’t want to introduce too

much at one time and most of the corrective measures involve knowledge of multiple regression as

an alternative model.

The first diagnostic we consider involves focusing on whether any of the explanatory variables

should be removed from the model. To make these decision(s) we test whether the coefficient

associated with each variable is significantly different from zero, i.e. for the ith explanatory variable:

H0: i = 0
H1: i  0

51
 aktuwallah.com

As discussed in simple linear regression this involves a t-test. Looking at Table 7, the p-value for

the tests associated with determining the significance for SALESt-1 and ADVERT1 are 0.0000 and

0.0397, respectively, we can ascertain that neither explanatory variable should be eliminated from

the model. If one of the explanatory variables had a p-value greater than  =. 05, then we would

designate that variable as a candidate for deletion from the model and go back to the specification

phase.

Another attribute of the model we are interested in is the R2 adjusted value that in Table 7 is 0.8721,

or 87.21 percent. Since we are now considering multiple linear regression models, the R 2 value that

we calculate represents the amount of variation in the dependent variable ( SALESt) that is explained

by the fitted model, which includes all of the explanatory variables jointly (ADVERTt and SALESt-

). At this point we choose to ignore the adjusted (ADJ) factor included in the printout.
1

Since we have already asked the question if anything should be deleted from the model the next

question that should be asked if there is anything that is missing from the model, i.e. should we add

anything to the model. To answer this question we should use theory but from an empirical

perspective we look at the residuals to see if they have a pattern, which as we discussed previously

would imply there is information. If we find missing information for the model (i.e. a pattern in the

residuals), then we go back to the specification phase, incorporate that information into the model

and then cycle through the 3 phase process again, with the revised model. We will illustrate this in

greater detail in our next example. However, the process involved is very similar to that which we

employed earlier in the semester. We illustrate the residual analysis with a new example.

Example

The purpose behind looking at this example is to allow us to work with some cross sectional data

and also to look in greater detail at analyzing the residuals. The data set contains three variables

52
 aktuwallah.com

that have been recorded by a firm that presents seminars. Each record focuses on a seminar with the

fields representing:

 number of people enrolled (ENROLL)

 number of mailings sent out (MAIL)
 lead time (in weeks) of 1st mailing (LEAD)
The theory being suggested is that the variation in the number of enrollments is an approximate

linear function of the number of mailings and the lead-time. As recommended earlier, we look at

the scatter plots of the data to see if our assumptions seem valid. Since we are working with two

explanatory variables, a three dimensional plot would be required to see all three variables

simultaneously, which can be done in StatGraphics with the PLOTTING FUNCTIONS, X-Y-Z

LINE and SCATTER PLOT options (note the dependent variable is usually Z). See Figure 7 for

this plot.

Plot of enroll vs lead and mail

59

49
enroll

39

29
18
19 12 15
3 6 9 mail
0 4 8 12 0
16 20 24
lead

Figure 7. Plot of Enroll vs. Mail & Lead

This plot provides some insight, but for beginners, it is usually more beneficial to view multiple

two-dimensional plots where the dependent variable ENROLL is plotted against the different

explanatory variables, as is shown in Figures 8 and 9.

53
 aktuwallah.com
Plot of enroll vs mail
59

49

enroll
39

29

19
0 3 6 9 12 15 18

mail

Figure 8. Plot of Enroll vs. Mail

Plot of enroll vs lead

59

49
enr
oll 39

29

19
0 4 8 12 16 20 24
lead

Figure 9. Plot of Enroll vs. Lead

Looking at Figure 9, which plots ENROLL against LEAD, we notice that there is a dip for the largest

LEAD values which may economically suggest diminishing returns i.e. at a point the larger lead

time is counterproductive. This suggest that ENROLL and LEAD may have a parabolic relationship.

Since the general equation of a parabola is:

y = ax2 + bx + c

we may want to consider including a squared term of LEAD in the model. However, at this point

we are not going to do so, with the strategy that if it is needed, we will see that when we examine

the residuals, as we would have ignored some information in the data and it will surface when we

analyze the residuals. (In other words we wish to show that if a term should be included in a model,

but is not identified, one should be able to identify it as missing when examining the residuals of a

model estimated without it.)

54
 aktuwallah.com

Specification
Thus the model we tentatively specify is:
ENROLLi = B0 + B1 MAILi + B2 Leadi + ERRORi
Estimation
Multiple Regression Analysis
-----------------------------------------------------------------------------
Dependent variable: enroll
-----------------------------------------------------------------------------
Standard T
Parameter Estimate Error Statistic P-Value
-----------------------------------------------------------------------------
CONSTANT 14.8523 2.1596 6.87733 0.0000
lead 0.627318 0.165436 3.79191 0.0008
mail 1.27378 0.233677 5.45103 0.0000
-----------------------------------------------------------------------------

Analysis of Variance
-----------------------------------------------------------------------------
Source Sum of Squares Df Mean Square F-Ratio P-Value
-----------------------------------------------------------------------------
Model 1985.35 2 992.674 56.87 0.0000
Residual 453.824 26 17.4548
-----------------------------------------------------------------------------
Total (Corr.) 2439.17 28

R-squared = 81.3943 percent

R-squared (adjusted for d.f.) = 79.9631 percent
Standard Error of Est. = 4.17789
Mean absolute error = 3.33578
Durbin-Watson statistic = 1.03162

Table 8

Note that MAIL and LEAD are both significant, since their p-values are 0.0000 and 0.0008,

respectively. Hence, there is no need at this time to eliminate either from the model. Also, note that

R2adj is 79.96 percent.

To see if there is anything that should be added to the model, we analyze the residuals to see if they

contain any information. Utilizing the graphics options icon, one can obtain a plot of the

standardized residuals versus lead (select residuals versus X). Plotting against the predicted values

55
 aktuwallah.com

is similar to looking for departures from the fitted line. For our example since we entertained the

idea of some curvature (parabola) when plotting ENROLL against LEAD, we now plot the residuals

against LEAD. This plot is shown as Figure 10.

Residual Plot
Studentized residual

2.8

1.8

0.8

-0.2

-1.2

-2.2
0 4 8 12 16 20 24

lead

Figure 10. Residual Plot for Enroll against Lead

What we are looking for in the plot is whether there is any information in LEAD that is missing

from the fitted model. If one sees the curvature that still exists, then it suggests that one needs to

add another variable, actually a transformation of LEAD, to the model. Hence we go back to the

specification phase, based upon the information just discovered, and specify the model as:

ENROLLi = B0 + B1 MAILi + B2 Lead + B3 (LEAD)2i + ERRORi

The estimation of the revised model generates the output presented in Table 9.

56
 aktuwallah.com
Multiple Regression Analysis
--------------------------------------- --------------------------------------
Dependent variable: enroll
--------------------------------------- --------------------------------------
Standard T
Parameter Estimate Error Statistic P-Value
--------------------------------------- --------------------------------------
CONSTANT 0.226184 2.89795 0.0780495 0.9384
lead 4.50131 0.675669 6.66201 0.0000
mail 0.645073 0.189375 3.40633 0.0022
lead * lead -0.132796 0.022852 -5.81115 0.0000
--------------------------------------- --------------------------------------

Analysis of Variance
--------------------------------------- --------------------------------------
Source Sum of Squares D f Mean Square F-Ratio P-Value
--------------------------------------- --------------------------------------
Model 2246.12 3 748.707 96.96 0.0000
Residual 193.053 25 7.72211
--------------------------------------- --------------------------------------
Total (Corr.) 2439.17 28

R-squared = 92.0853 percent

R-squared (adjusted for d.f.) = 91.1356 percent
Standard Error of Est. = 2.77887
Mean absolute error = 2.081
Durbin-Watson statistic = 1.121

Table 9

Diagnostic Checking

At this point we go through the diagnostic checking phase again. Note that all three explanatory

variables are significant and that the R2adj value has increased to 91.13 percent from 79.96 percent.

For our purposes at this point, we are going to stop our discussion of this example, although the

reader should be aware that the diagnostic checking phase has not been completed. Residual plots

should be examined again, and other diagnostic checks we still need to discuss should be

considered.

Before we proceed however, it should be pointed out that the last model is still a multiple linear

regression model. Many students think that by including the squared term, to incorporate the

57
 aktuwallah.com

curvature, that we may have violated the linearity condition. This is not the case, as when we say

“linear” it is linear with regards to the coefficients. An intuitive explanation of this is to think like

the computer, all LEAD2 represents is the squared values of LEAD, therefore, the calculations are

the same as if LEAD2 was another explanatory variable.

The next three multiple regression topics we discuss will be illustrated with the data that was part of

a survey conducted of houses in Eugene, Oregon, during the 1970’s. The variables measured

(recorded), for each house, are sales price (price), square feet (sqft), number of bedrooms (bed),

number of bathrooms (bath), total number of rooms (total), age in years (age), whether the house

has an attached garage (attach), and whether the house has a nice view (view).

Dummy Variables

Prior to this current example, all the regression variables we have considered have been either ratio

or interval data, which means they are non-qualitative variables. However, we now want to

incorporate qualitative variables into our analysis. To do this we create dummy variables, which are

binary variables that take on values of either zero or one. Hence, the dummy variable (attach) is

defined as:

attach = 1 if garage is attached to house

0 otherwise (i.e. not attached)
and
view = 1 if house has a nice view
0 otherwise

Note that each qualitative attribute (attached garage and view) cited above has two possible

outcomes (yes or no) but there is only 1 dummy variable for each. That is because there must

always be, at maximum, one less dummy variable than there are possible outcomes for the

particular qualitative attribute. We mention this because there are going to be situations, for other

examples, where one wants to incorporate a qualitative attribute that has more than two possible

58
 aktuwallah.com

outcomes in the analysis. For example, if one is explaining sales and has quarterly data, they might

want to include the season as an explanatory variable. Since there are four seasons (Fall, Winter,

Spring, and Summer) there will be three (four minus one) dummy variables. To define these three

dummy variables, we arbitrarily select one season to “withhold” and create dummy variables for

each of the other seasons. For example, if summer was “withheld” then our three dummy variables

could be

D1 = 1 if Fall
0 otherwise
D2 = 1 if Winter
0 otherwise
D3 = 1 if Spring
0 otherwise

Now, what happens when we withhold a season is not that we ignore the season, but the others are

being compared with what is being withheld.

Outliers

When an observation has an undue amount of influence on the fitted regression model (coefficients)

then it is called an outlier. Ideally, each observation has an equal amount of influence on the

estimation of the fitted lines. When we have an outlier, the first question one needs to ask is “Why

is that observation an outlier?” The answer to that question will frequently dictate what type of

action the model builder should take.

One reason an observation may be an outlier is because of a recording (inputting) error. For

instance, it is easy to mistakenly input an extra zero, transpose two digits, etc. When this is the

cause, then corrective action can clearly be taken. Don’t always assume the data is correct!

Another source is because of some extra ordinary event that we do not expect to occur again. Or

the observation is not part of the population we wish to make interpretation/forecasts about. In

these cases, the observation may be “discarded.”

59
 aktuwallah.com

If the data is cross-sectional, then the observation may be eliminated, thereby decreasing the

number of observations by one. If the data is times series, by “discarding the impact” of the

observation one does not eliminate observations since doing so may effect lagging relationships,

however one can set the dummy variable equal to one (1) for that observation, zero (0) otherwise.

At other times, the outcome, which is classified as an outlier, is recorded correctly, may very well

occur again, and is indeed part of the concerned population. In this case, one would probably want

to leave the observation in the model construction process. In fact, if an outlier or set of outliers

represents a source of specific variation then one should incorporate that specific variation into the

model via an additional variable. Keep in mind, just because an observation is an outlier does not

mean that it should be discarded. These observations contain information that should not be

ignored just so “the model looks better.”

Now that we have defined what an outlier is and what action to take/not take for outlier, the next

step is to discuss how to determine what observations are outliers. Although a number of criteria

exist for classifying outliers, we limit our discussion to two specific criteria - standardized residuals

and leverage.

The theory behind using standardized residuals is that outliers are equated with observations which

have large residuals. To determine what is large, we standardize the residuals and then use the rule

that any standardized residual outside the bounds of -2 to 2 is considered an outlier. [Why do we

use -2 and 2? Could we use -3 and 3?]

The theory behind the leverage criteria is that a large residual may not necessarily equate with an

outlier. Hence, the leverage value measures the amount of influence that each observation has on

the set of estimates. It’s not intuitive, but can be shown mathematically, that the sum of the

leverage points is equal to the number of B coefficients in the model (P). Since there are N

60
 aktuwallah.com

observations, under ideal conditions each observation should have a leverage value of P/N. Hence,

using our criteria of large being outside two standard deviation, the decision rule for declaring

outliers by means of leverage values is to declare an observation as a potential outlier if its leverage

value exceeds 2*P/N. StatGraphics employs a cut off of 3 * P/N.

To illustrate, identifying outliers, we estimate the model:

Pricei = B0 + B1 SQFTi + B2 BED + Error

Multiple Regression Analysis

--------------------------------------- --------------------------------------
Dependent variable: price
--------------------------------------- --------------------------------------
Standard T
Parameter Estimate Error Statistic P-Value
--------------------------------------- --------------------------------------
CONSTANT -15.4038 7.34394 -2.09749 0.0414
sqft 3.52674 0.269104 13.1055 0.0000
bed 7.64828 2.78697 2.7443 0.0086
--------------------------------------- --------------------------------------

Analysis of Variance
--------------------------------------- --------------------------------------
Source Sum of Squares Df Mean Square F-Ratio P-Value
--------------------------------------- --------------------------------------
Model 29438.5 2 14719.3 140.65 0.0000
Residual 4918.52 47 104.649
--------------------------------------- --------------------------------------
Total (Corr.) 34357.0 49

R-squared = 85.6841 percent

R-squared (adjusted for d.f.) = 85.0749 percent
Standard Error of Est. = 10.2298
Mean absolute error = 7.19612
Durbin-Watson statistic = 1.682

Table 10

With the results being shown in Table 10, in our data set of houses, clearly some houses are going

to influence the estimate more than others. Those with undue influences will be classified as

potential outliers. Again, the standardized residuals outside the bounds -2, +2 (i.e. absolute value

greater than 2), and the leverage values greater than 3 3/50 (P = 3 since we estimated the coefficient

for two (2) explanatory variables and the intercept and n = 50 since there were 50 observations) will

61
 aktuwallah.com

be flagged. After estimating the model we select the "unusual residuals" and "influential points"

options under the tabular options icon. Note that from tables 11 and 12 observations 8, 42, 44, 47,

49 and 50 are classified as outliers.

Unusual Residuals
--------------------------------------------------------------
Predicted Studentized
Row Y Y Residual Residual
--------------------------------------------------------------
44 111.3 85.482 25.818 2.73
47 115.2 92.1828 23.0172 2.40
49 129.0 89.2508 39.7492 5.03
--------------------------------------------------------------

Table 11

Influential Points
------------------------------------------------
Mahalanobis
Row Leverage Distance DFITS
------------------------------------------------
8 0.0816156 3.28611 0.560007
42 0.144802 7.14775 0.58652
49 0.0947427 4.04401 1.62728
50 0.339383 23.6798 0.0932134
------------------------------------------------
Average leverage of single data point = 0.06

Table 12

Once the outliers are identified one then needs to decide what, if anything, needs to be modified in

the data or model. This involves checking the accuracy of the data and/or determining if the

62
 aktuwallah.com

outliers represent a specific source of variation. To ascertain any sources of specific variation one

looks to see if there is anything common in the set, or subset, of observations flagged as outliers. In

Table 113 one can see that some of the latter observations (42, 44, 47, 49, and 50) were flagged.

Since the data ( n = 50) was entered by ascending price, one can see that the higher priced homes

were flagged. As a result, for this example, the higher priced homes are receiving a large amount of

influence. Hence, since this is cross-sectional data, one might want to split the analysis into two

models - one for “lower” priced homes and the second for “higher” priced homes.

Multicollinearity

When selecting a set of explanatory variables for a model, one ideally would like each explanatory

variable to provide unique information that is not provided by the other explanatory variable(s).

When explanatory variables provide duplicate information about the dependent variable, then we

encounter a situation called multicollinearity. For example, consider our house data again, where

the following model is proposed:

Price = B0 + B1 SQFT + B2 BATH + B3 TOTAL + ERROR

Clearly there is a relationship among the three (3) explanatory variables. What problems might this
create? To answer this, consider the estimation results, which are shown on the following page.

3
StatGraphics also used two other techniques for identifying outliers (Mahalanobis Distribution and DIFTS), which we
have elected not to discuss since from an intuitive level they are similar to the standardized residual/leverage criteria.

63
 aktuwallah.com

Multiple Regression Analysis

-----------------------------------------------------------------------------
Dependent variable: price
-----------------------------------------------------------------------------
Standard T
Parameter Estimate Error Statistic P-Value
-----------------------------------------------------------------------------
CONSTANT -42.6274 9.50374 -4.48533 0.0000
sqft 3.02471 0.296349 10.2066 0.0000
bath -10.0432 3.49189 -2.87614 0.0061
total 10.7836 2.06048 5.23351 0.0000
-----------------------------------------------------------------------------

Analysis of Variance
-----------------------------------------------------------------------------
Source Sum of Squares Df Mean Square F-Ratio P-Value
-----------------------------------------------------------------------------
Model 30780.2 3 10260.1 131.95 0.0000
Residual 3576.84 46 77.7575
-----------------------------------------------------------------------------
Total (Corr.) 34357.0 49

R-squared = 89.5892 percent

R-squared (adjusted for d.f.) = 88.9102 percent
Standard Error of Est. = 8.81802
Mean absolute error = 5.89115
Durbin-Watson statistic = 1.53269

Table 13

If one were to start interpreting the coefficients individually and noticed that bath has a negative

coefficient, they might come to the conclusion that one way to increase the sales price is to

eliminate a bathroom. Of course, this doesn’t make sense, but it does not mean the model is not

useful. After all, when the BATH is altered so are the TOTAL and SQFT. So a problem with

multicollinearity is one of interpretation when other associated changes are not considered. One

important fact to remember, is that just because multicollinearity exists, does not mean the model

can not be used for meaningful forecasting, provided the forecasts are within the data region

considered for constructing the model.

64
 aktuwallah.com

Predicting Values with Multiple Regression

Regression models are frequently used for making statistical predictions. A multiple regression

model is developed, by the method of least squares, to predict the values of a dependent, response

variable based on two or more independent, explanatory variables.

Research data can be classified as cross-sectional data or as time series data. Cross-sectional data

has no time dimension, or it is ignored. Consider collecting data on a group of subjects. You are

interested in their age, weight, height, gender, and whether they tend to be left-handed. The time

dimension in collecting the data is not important and would probably be ignored; even though

researchers tend to collect the data within a reasonably short time period.

Time series data is a sequence of observations collected from a process with equally spaced periods

of time. For example, in collecting sales data, the data would be collected weekly with the time (the

specific week of the year) and sales being recorded in pairs.

Using Cross-sectional Data for Predictions

When using regression models for making predictions with cross-sectional data, it is imperative that

you use only the relevant range of the predictor variable(s). When predicting the value of the

response variable for a given value of the explanatory variable, one may interpolate within the

range of the explanatory variables. However, contrary to when using time series data, one may not

extrapolate beyond the range of the explanatory variables. (To predict beyond the range of an

explanatory variable is to assume that the relationship continues to hold true below and/or above the

range -- something that is not known nor can it be determined. To make such an interpretation is

meaningless and, at best, subject to gross error.)

65
 aktuwallah.com

An Example: Using a Regression Model to Predict

Consider the following research problem - a real estate firm is interested in developing a model to

predict, or forecast, the selling price of a home in a local community. Data was collected on 50

homes in a local community over a three week period.

The data can consist of both qualitative and quantitative values. Quantitative variables are

measurable whereas qualitative variables are descriptive. For example: your height, a quantitative

value, is measurable whereas the color of your hair, a qualitative variable, is descriptive.

For our real estate example, the dependent variable (selling price) and the explanatory variables

(square feet, number of bathrooms, and total number of rooms) are all quantitative variables. None

of the data are qualitative variables.

Table 13. Variable With Range of Values

Variables Range of Values

Price (selling) ($1000) 30.6 - 165
Square feet (100 ft2) 8 - 40
Number of Bathrooms 1-3
Total number of rooms 5 - 12

As a review, the multiple regression model can be expressed as:

Yi = 0 + 1X1 + 2X2 + 3X3 + i

The slope, i, known as a net regression coefficient, represents the unit change in Y per unit

change in Xi taking into account (or, holding constant) the effect of the remaining explanatory

variables. In our real estate problem, b1 , where X1 is in square feet, represents the unit change

selling price per unit change in square feet, taking into account the effect of number of bedrooms,

and total number of rooms.

66
 aktuwallah.com

The resulting model fitting equation is shown in Table 14.

Multiple Regression Analysis

--------------------------------------- --------------------------------------
Dependent variable: price
--------------------------------------- --------------------------------------
Standard T
Parameter Estimate Error Statistic P-Value
--------------------------------------- --------------------------------------
CONSTANT -42.6274 9.50374 -4.48533 0.0000
sqft 3.02471 0.296349 10.2066 0.0000
bath -10.0432 3.49189 -2.87614 0.0061
total 10.7836 2.06048 5.23351 0.0000
--------------------------------------- --------------------------------------

Analysis of Variance
--------------------------------------- --------------------------------------
Source Sum of Squares D f Mean Square F-Ratio P-Value
--------------------------------------- --------------------------------------
Model 30780.2 3 10260.1 131.95 0.0000
Residual 3576.84 46 77.7575
--------------------------------------- --------------------------------------
Total (Corr.) 34357.0 49

R-squared = 89.5892 percent

R-squared (adjusted for d.f.) = 88.9102 percent
Standard Error of Est. = 8.81802
Mean absolute error = 5.89115
Durbin-Watson statistic = 1.53269

Table 14

Multiple regression analysis is conducted to determine whether the null hypothesis, written as H o: i

= 0 (with i = 0 - 3), can be rejected. If the null hypothesis can be rejected, then there is sufficient

evidence of a relationship (or, an association) between the response variable and the explanatory

variables in the sample. Table 14 also displays the resulting analysis of variance (ANOVA) for the

multiple regression model using the explanatory variables listed in Table 12.

67
 aktuwallah.com

The ANOVA for the full multiple regression shows a p-value equal to 0.0000, thus Ho can be

rejected (because the p-value is less than  of 0.05). Since the null hypothesis may be rejected,

there is sufficient evidence of a relationship (or, an association) between selling price and the three

explanatory variables in the sample of 50 houses.

CAUTION: As stated, when using regression models for making predictions with
cross-sectional data, use only the relevant range of the explanatory variable(s). To
predict outside the range of an explanatory variable is to assume that the relationship
continues to hold true below and/or above the range -- something that is not known
nor can be determined. To make such an interpretation is meaningless and, at best,
subject to gross error.

Suppose one wishes to obtain a point estimate, along with confidence intervals for both the

individual forecasts and the mean, for a home with the following attributes

1500 square feet, 1 bath, 6 total rooms.

To do this using Statgraphics, alls one needs to do is add an additional row of data to the data file

(HOUSE.SF). In particular one would insert a 15 in the sqft column (remember that the square feet

units is in 100 's), a 1 in the bath column and a 6 in the total column. We leave the other columns

blank, especially the price column, since Statgraphics will treat it as a missing value and hence

estimate it. To see the desired output, one runs the regression, using the additional data points, goes

to the tabular options icon and selects the "report" option. Table 15 shows the forecasting results

for our example.

Regression Results for price

------------------------------------------------------------------------------------------------------
Fitted Stnd. Error Lower 95.0% CL Upper 95.0% CL Lower 95.0% CL Upper 95.0% CL
Row Value for Forecast for Forecast for Forecast for Mean for Mean
------------------------------------------------------------------------------------------------------
51 57.4014 9.1313 39.021 75.7818 52.6282 62.1746
------------------------------------------------------------------------------------------------------

Table 15

68
 aktuwallah.com

Summary
In the introduction to this section, cross-sectional data and time series data were defined. With

cross-sectional data, the time dimension in collecting the data is not important and can be ignored;

even though researchers tend to collect the data within a reasonably short time period. When

predicting the value of the response variable for a given value of the explanatory variable with

cross-sectional data, a researcher is restricted to interpolating within the range of the explanatory

variables. However, a researcher may not extrapolate beyond the range of the explanatory variables

because it cannot be assumed that the relationship continues to hold true below and/or above the

range since such an assumption cannot be validated. Cross-sectional forecasting is stationary, it

does not change over time.

On the other hand, time series data is a sequence of observations collected from a process with

equally spaced periods of time. Contrary to the restrictions placed on cross-sectional data, when

using time series data a major purpose of forecasting is to extrapolate beyond the range of the

explanatory variables. Time series forecasting is dynamic, it does change over time.

Practice Problem
As part of your job as personnel manager for a company that produces an industrial

product, you have been assigned the task of analyzing the salaries of workers

involved in the production process. To accomplish this, you have decided to

develop the “best” model, utilizing the concept of parsimony, to predict their weekly

salaries. Using the personnel files, you select, based on systematic sampling, a

sample of 49 workers involved in the production process. The data, entered in the

file COMPANY, corresponds to their weekly salaries, lengths of employment, ages,

gender, and job classifications.

69
 aktuwallah.com

a. ^ = _________________________________________________________
y
b. H0: ______________________ H1: _______________________
p-value: ___________________ Decision: __________________
c. In the final model, state the value and interpret for R2adj. R2adj: ________ %
d. In the final model, state the value and interpret for b 1 . b1 = ________
e. Predict the weekly salaries for the following employees:

Category Employee #1 Employee #2

Length of employment (in months) 10 125
Age (in years) 23 33
Gender female male
Job classification technical clerical

Employee 95% LCL ^

y 95% UCL

#1

#2

[Check documentation on file to ascertain gender coding for female and male. Also check for
proper coding for job classification.]

70
 aktuwallah.com

Stepwise Regression

When there exists a large number of potential explanatory variables, a good exploratory technique

one can utilize is known as stepwise regression. This technique involves introducing or deleting

variables one at a time. There are two general procedures under the umbrella of stepwise regression

-- forward selection and backwards elimination. A hybrid of both forward selection and

backwards elimination exists and is generally known as stepwise.

In the sections below, we describe the three (3) procedures cited above. In order to follow the

discussion, we first need to review the t- test for regression coefficients. Recall that for the model

Yi =  0 + 1 X1,i + 2 X2,i + ......... +  k Xk,i + t

the t-test for: H0 : k = 0

H1 : k  0

actually tests whether the variable Xk should be included in the model. If one rejects H0, then the

decision is to keep Xt in the model, whereas if one does not reject H0 the decision is to eliminate

Xt from the model. Since rejecting H0 is usually done when either t  -2.0 or t  2.0, one can see

that having a variable in the model is equated to having a t-value with an absolute value greater

than 2. Likewise, if a variable has a corresponding t-value, which is equal to or less than 2 in

absolute terms, it should be eliminated from the model.

To simplify the programming for the stepwise procedures, the software packages generally rely on

the fact that squaring a distribution gives one an F distribution. Hence, the discussion above about

the t value and whether to keep or eliminate the corresponding variable can be expressed as:

If the F-statistic ( F = t 2) is greater than 4.0 , then the corresponding variable should
be included in the model. If the F-statistic is less than 4.0, then the corresponding
variable should not be included in the model.

Given this background information, we now discuss the three (3) stepwise procedures.

71
 aktuwallah.com

Forward Selection
This procedure starts with no explanatory variables in the model, only a constant. It then calculates

an F-statistic for each variable and focuses its attention on that variable with the highest F-value. If

the highest F-value is greater than 4.0, then the corresponding variable is inserted into the model.

If the highest F-value is less than 4.0, then the process stops. Assuming the first variable is inserted

in the model, an F-statistic is then calculated for each of the variables not in the model, conditioned

upon the fact that the first variable selected is in the model. The procedure then focuses on the

variable with the highest F-value and asks whether the F-value is greater than 4.0. If the answer is

yes, the associated variable is inserted into the model and the process continues by calculating an F-

statistic for each of the variables not included in the model, conditioned upon the fact that the first

two variables selected are included in the model. Once again, the procedure focuses attention on

that variable with the largest F-value and determines whether it is larger than 4.0. If the answer is

yes the associated variable is inserted into the model and the process continues by calculating an F-

statistic for each of the variables not included in the model, conditioned upon the fact that the first

three variables selected are included in the model. This process continues on until finally either all

of the variables have been included in the model or none of the remaining variables are significant.

Backward Elimination
This procedure starts with all of the explanatory variables in the model and successively drops one

variable at a time. Given all of the explanatory variables in the model, the “full” regression is run

and an F-statistic for each explanatory variable is calculated. The attention now focuses on the

variable with the smallest F-value. If the F-value is less than 4.0, then that variable is eliminated

from the model and a new regression model is estimated. From this “smaller regression” F-

statistics are examined and again the attention now focuses on that variable with the smallest F-

value. If the F- value is less than 4.0, then that variable is eliminated from the model and a new

72
 aktuwallah.com

regression model is estimated. This process continues on until either all of the explanatory

variables have been eliminated from the model or all of the remaining explanatory variables are

significant.

Stepwise
This procedure is a hybrid of forward selection and backwards elimination. It operates the same

as forward selection, except at each stage the possibility of deleting a variable, as in backward

elimination is considered. Hence, a variable that enters at one stage may be eliminated at a later

stage (due to multicollinearity)

Summary

Generally all three stepwise procedures will provide the same model. Under extreme collinear

conditions (explanatory variables) the final results may be different. Keep in mind that stepwise

procedures are good exploratory techniques, to provide the model builder with some insight. One

should not be fooled into thinking that stepwise models are the best because the “computer

generates the models.” Stepwise procedures fail to consider things such as outliers, residual

patterns, autocorrelation, and theoretical considerations.

73
 aktuwallah.com

RELATIONSHIPS BETWEEN SERIES

When building models one frequently desires to utilize variables that have significant linear

relationships. In this section we discuss correlation as it pertains to cross sectional data,

autocorrelation for a single time series (demonstrated in the previous chapter), and cross

correlation, which deals with correlations of two series. Hopefully, the reader will note the

relationship between correlation, autocorrelation, and cross correlation.

Correlation

As we mentioned previously, when we talk of statistical correlation we are discussing a value which

measures the linear relationship between two variables. The statistic

rxy 
  x  x y  y
i i

Sx Sy

where Sy and Sx represent the sample standard deviation of Y and X respectively, measures the

strength of the linear relationship between the variables Y and X. Again we are not going to dwell

on the mathematics, but will be primarily concerned with the interpretation.

To interpret the correlation coefficient, it is important to note that the denominator is included so

that values generated are not sensitive to the choice of metrics (i.e. inches vs. feet, ounces vs.

pounds, cents vs. dollars, etc.). As a result, the range of possible values for the correlation

coefficients range from -1.0 to 1.0.

Since the denominator is always a positive value, one can interpret the sign of the correlation

coefficient as the indicator of relationship of how X and Y move together. For instance, if the

correlation coefficient is positive, this indicates that positive (negative) changes in X tend to

accompany positive (negative) changes in Y (i.e. X and Y move in the same direction). Likewise, a

74
 aktuwallah.com

negative correlation value indicates that positive (negative) changes in X tend to accompany

negative (positive) changes in Y (i.e. X and Y move in opposite directions).

The absolute value of the correlation coefficient indicates how strong of a linear relationship two

variables have. The closer the absolute value is to 1.0 the stronger the linear relationship.

To summarize we consider the plots in Figure 1, where we show five different values for the

correlation coefficient. Note that (1) the sign indicates whether the variables move in the same

direction and (2) the absolute value indicates the strength of the linear relationship.

Autocorrelation

As indicated by its name, the autocorrelation function will calculate the correlation coefficient for

a series and itself in previous time periods. Hence, when analyzing one series and determining how

(linear) information is carried over from one time period to another, we will rely on the

autocorrelation function.

The autocorrelation function is defined as:

 x t 
 x xt  k  x 
r(k ) 
Sx t Sx t  k

where again Sx and Sx(t-k) are the sample standard deviations of Xt and Xt-k ; which if you think

about it are the same value. Hence when you substitute X t and Xt-k into the correlation equation for

Y and X you can see the similarity. The one difference is with the time element component and

hence the inclusion of k. What k represents is the “lag” factor. So when one calculates r(1), that is

the sample autocorrelation of a time series variable and itself 1 time period ago, r(2) is the sample

autocorrelation of a time series variable and itself 2 time periods ago, r(3) is the sample

autocorrelation of a time series variable and itself 3 time periods ago, etc.

75
 aktuwallah.com

To illustrate the value of the autocorrelation function, consider the series TSDATA.BUBBLY

(StatGraphics data sample), which represents the monthly champagne sales volume for a firm. The

plot of this series shows a strong seasonality component as shown on the next page in Figure 2.

TSDATA.bubbly

15

12

9
data
6

0
03/97 12/98 09/99 06/00 03/01
Time

Figure 2. Time Sequence Plot for Bubbly Data

The autocorrelation function can be displayed numerically, Table 1, below:

Table 1. Estimated autocorrelations for TSDATA.bubbly
----------------------------------------------------------------
Lag Estimate Stnd.Error Lag Estimate Stnd.Error
----------------------------------------------------------------
1 .48933 .10911 2 .05787 .13269
3 -.15498 .13299 4 -.25001 .13512
5 -.03906 .14052 6 .03647 .14065
7 -.03773 .14076 8 -.24633 .14088
9 -.18132 .14592 10 -.00307 .14858
11 .37333 .14858 12 .80455 .15935
13 .40606 .20200 14 .02545 .21150
15 -.17323 .21153 16 -.24418 .21322
17 -.05609 .21652 18 .02920 .21669
19 -.03339 .21674 20 -.20632 .21680
21 -.14682 .21913 22 -.01295 .22029
23 .27869 .22030 24 .60181 .22446
----------------------------------------------------------------

The autocorrelation function can also be displayed graphically (where dotted lines -- symmetric

about 0 -- represent the significance limits) as shown in Figure 3.

76
 aktuwallah.com

Estimated Autocorrelations

0.5

coefficient
0

-0.5

-1

0 5 10 15 20 25
lag

Figure 3. Estimated Autocorrelations

By analyzing the display, the autocorrelation at lags 1, 11, 12, 13, and 24 are all significant ( =

0.05). Hence, one can conclude that there is a linear relationship between sales in the current time

period and itself and 1, 11, 12, 13, and 24 time periods ago. The values at 1, 11, 12, 13, and 24 are

connected with a yearly cycle (every 12 months).

Stationarity

The next topic we wish to discuss in this section is the cross correlation function, which will be used

to examine the relationship between two series displaced by k time periods. This will allow us to

begin identifying leading indicators. However in order to discuss the cross correlation function, we

first need to review what it means for a series to be stationary. This discussion is necessary because

the interpretation of the cross correlation function only makes useful sense if both series involved

are stationary.

77
 aktuwallah.com

Recall, a series is stationary if it has a constant mean and variance. Common departures from

stationarity (i.e. non-stationary series) are shown below:

When a series is nonstationary because of a changing variance, one can treat this problem by taking

logs of the data [logs in this course will be natural logs (Ln), not common logs (base 10)]. When a

series is nonstationary due to a changing mean then one can take differences to treat that problem.

If seasonality exists then one may in addition to taking differences of consecutive time periods, take

seasonal differences.

If a nonstationary series has a nonconstant mean and a nonconstant variance then differences and

logs may both be required to achieve a transformation to a stationary series. When taking both logs

and differences one must take the logs first (i.e. treat the nonconstant variance and the attack the

nonconstant mean). Why?

Cross Correlation

With the knowledge discussed in the autocorrelation section and the stationarity section, we are now

prepared to discuss the cross correlation function, which as we said before is designed to measure

the linear relationship between two series when they are displaced by k time periods. The cross

correlation function is shown below. (The formula is shown on extra large type to highlight the

components of the formula.)

78
 aktuwallah.com

rxy  k  
   Y t Y  X t k X 
SYt S X t  k

To interpret what is being measured in the cross correlation function one needs to combine what we

discussed about the correlation function and the autocorrelation function. Again note, like in the

autocorrelation function, that k can take on integer values, only now k can take on positive and

negative values.

For instance, let Y represent SALES and X represent ADVERTISING for a firm. If k = 1, then we

are measuring the correlation between SALES in time period t and ADVERTISING in time period t-

1. i.e. we are looking at the correlation between SALES in a time period and ADVERTISING in the

previous time period. If k = 2, we would be measuring the correlation in SALES in time period t

and ADVERTISING two time periods prior. What if k = 3, k = 4, ....? Note that when k is zero we

are considering the relationship of ADVERTISING in the same time periods.

When k takes on negative values then our interpretations are the same as above, except that now we

are looking at cases were Y (SALES) are leading indicators for X (ADVERTISING). This is the

“opposite” of what we were doing with the positive values for k. Note the cross correlation

function is not symmetric about 0. i.e.

rxy (k)  rxy (-k) for all x,y, k  0

An Example

To illustrate the cross correlation function, we consider the data TSDATA.units and
TSDATA.leadind. This data is sample data from Statgraphics and resides on the
network.

The joint plot of units and leadind, is shown in Figure 4 on the following page.
Note how leadind “leads” units. And how both series are nonstationary. Given at

79
 aktuwallah.com

least one of the series is nonstationary, the cross correlation function will be
meaningless if it is applied to the original data. Since both series can be transformed
to stationary series by simple differences (verify this), we will apply the cross
correlation function to the differenced series for both series.

Time Sequence Plot units

leadind

270 14.7

13.7
250

12.7
230
11.7

210
10.7

190 9.7

0 30 60 90 120 150

Time

Figure 4. Time Sequence Plot of Lead and Lag Indicators

Looking at the CCF (cross correlation plot) plot displayed in Figure 5 on the next page, we can see

significant cross correlation values at lags 2 and 3. Given leadind was the input (X t-k) value and

units is the output (Yt) value, we can conclude that leadind is a leading indicator of units by 2 and

3 time periods. So a change in leadind will result in a change in units two and three time periods

later. Note it takes two time periods for a change in leadind to show up in units.

(Note: for a situation where it is of interest to determine whether advertising leads sales, then
advertising would be the input and sales would be the output.)

80
 aktuwallah.com

Estimated Cross-Correlations

0.5

coefficient
0

-0.5

-1

-13 -8 -3 2 7 12 17

lag

Figure 5. Estimated Cross-Correlations

Questions:

 Does units lead leadind?

 What do you think would be the relationship between sales and advertising for a
firm?
 In the units/leadind example, what does the CCF value for k = 0 mean?

81
 aktuwallah.com

Mini-Case

Herr Andres Lüthi owns a bank in Bern, Switzerland. One of Herr Lüthi’s requirements of his

employees is they must continually solicit unnumbered accounts from foreign investors. Herr Lüthi

prefers to call such accounts “CDs” because they have time limits similar to certificates of deposits

used in the United States.

Being very computer literate, Herr Lüthi created a file, CD, to store his data. In this historical file,

he maintains data of the sales volume of CDs, volume, for his bank. All the data is maintained on a

monthly basis. Included in the data set are call (the number of cold calls Herr Lüthi’s employees

made each month during the period January 1990 through July 1995), rate (the average rate for a

CD), and mail (the number of mailings Herr Lüthi sent out to potential customers). Because of the

excellent services provided by the bank, it is the norm for customers to roll their CDs over into new

CDs when their original CDs expire.

It should be noted that several years ago Herr Lüthi took many of his employees on an extended ski

vacation. Records show that the ski vacation was in 1992, February through May. The few non-

skiers, who opted to take their holidays in Spain, continued soliciting CDs. They were, of course,

credited with any walk-in traffic and any roll over accounts.

You were recently offered a position at Herr Lüthi’s bank. As part of your responsibilities, you are

to construct a regression model that can be used to analyze the bank’s performance with regards to

selling CDs. When the Board of Directors met last week, they projected the following for next

month:
Number of Cold Calls 900
Average Rate for a CD 3.50
Number of Mailings 4,500

Prepare your analysis for Herr Lüthi.

82
aktuwallah.com

83
aktuwallah.com

84