Correlation_and_Regression-3.ppt

Correlation and Regression
The test you choose depends on level of measurement:
Independent Dependent StatisticalTest
Dichotomous Interval-ratio Independent Samples t-test
Dichotomous
Nominal Nominal CrossTabs
Dichotomous Dichotomous
Nominal Interval-ratio ANOVA
Dichotomous Dichotomous
Interval-ratio Interval-ratio Correlation and
Dichotomous OLS Regression

•Correlation is a statistic that assesses the
strength and direction of linear association of
two interval-ratio variables . . . It is created
through a technique called “regression”
•Bivariate regression is a technique that fits a
straight line as close as possible between all the
coordinates of two interval-ratio variables
plotted on a two-dimensional graph--to
summarize the relationship between the
variables

• For example:
A sociologist may be interested in the relationship between education
and self-esteem or Income and Number of Children in a family.
Independent Variables
Education
Family Income
Dependent Variables
Self-Esteem
Number of Children

•For example:
•May expect: As education increases, self-esteem increases
(positive relationship).
•May expect: As family income increases, the number of
children in families declines (negative relationship).
Education
Family Income
Dependent Variables
Self-Esteem
Number of Children
+
-

•For example:
•Null Hypothesis: There is no relationship between education
and self-esteem.
•Null Hypothesis: There is no relationship between family
income and the number of children in families.
•Ho: b = 0 “b” is a symbol for a statistic
•Ha: b ≠ 0 that describes the relationship
Education
Family Income
Dependent Variables
Self-Esteem
Number of Children

•Let’s look at the relationship between income and
number of children.
•Regression will start with plotting the coordinates in
your data (although you will hardly ever “plot” your
data in reality).
•Some data:
Case: 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25
Children (Y): 2 5 1 9 6 3 1 0 3 7 7 2 4 2 1 0 1 2 4 3 0 1 2 5 7
Income 1=$10K (X): 3 4 9 5 4 12 14 10 1 4 3 11 4 9 13 10 7 5 2 5 15 11 8 3 2

Y
X
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15
1
2
3
4
5
6
7
8
9
10
Case: 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25
Children (Y): 2 5 1 9 6 3 1 0 3 7 7 2 4 2 1 0 1 2 4 3 0 1 2 5 7
Income 1=$10K (X): 3 4 9 5 4 12 14 10 1 4 3 11 4 9 13 10 7 5 2 5 15 11 8 3 2
Plotted coordinates for income and children
Can you see a relationship?

Y
X
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15
1
2
3
4
5
6
7
8
9
10
Case: 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25
Children (Y): 2 5 1 9 6 3 1 0 3 7 7 2 4 2 1 0 1 2 4 3 0 1 2 5 7
Income 1=$10K (X): 3 4 9 5 4 12 14 10 1 4 3 11 4 9 13 10 7 5 2 5 15 11 8 3 2
Well, the slope of a
line fitted to the
points could tell us
the nature of the
relationship!

Y
X
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15
1
2
3
4
5
6
7
8
9
10
Case: 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25
Children (Y): 2 5 1 9 6 3 1 0 3 7 7 2 4 2 1 0 1 2 4 3 0 1 2 5 7
Income 1=$10K (X): 3 4 9 5 4 12 14 10 1 4 3 11 4 9 13 10 7 5 2 5 15 11 8 3 2
Is it positive?

Y
X
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15
1
2
3
4
5
6
7
8
9
10
Case: 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25
Children (Y): 2 5 1 9 6 3 1 0 3 7 7 2 4 2 1 0 1 2 4 3 0 1 2 5 7
Income 1=$10K (X): 3 4 9 5 4 12 14 10 1 4 3 11 4 9 13 10 7 5 2 5 15 11 8 3 2
Is it negative?

Y
X
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15
1
2
3
4
5
6
7
8
9
10
Case: 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25
Children (Y): 2 5 1 9 6 3 1 0 3 7 7 2 4 2 1 0 1 2 4 3 0 1 2 5 7
Income 1=$10K (X): 3 4 9 5 4 12 14 10 1 4 3 11 4 9 13 10 7 5 2 5 15 11 8 3 2
Is there no
relationship?

Y
(# Children)
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15
1
2
3
4
5
6
7
8
9
10
What is the slope of a “fitted line?”
The slope is the change in Y along the line
as you go up one on X while following the
line (rise over run).
Slope = 0, No relationship!
X
(Income)

Y
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15
1
2
3
4
5
6
7
8
9
10
Slope = 0.5, Positive Relationship!
1
0.5
Y
(# Children)
X
(Income)

Y
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15
1
2
3
4
5
6
7
8
9
10
Slope = -0.5, Negative Relationship!
1
0.5

•The mathematical equation for a line:
Y = mx + b
Where: Y = the line’s position on the
vertical axis at any point
X = the line’s position on the
horizontal axis at any point
m = the slope of the line
b = the intercept with theY axis,
where X equals zero

• The statistics equation for a line:
Y = a + bx
Where: Y = the line’s position on the
vertical axis at any point (estimated
value of dependent variable)
X = the line’s position on the
horizontal axis at any point (value of
the independent variable for which you
want an estimate ofY)
b = the slope of the line (called the coefficient)
a = the intercept with theY axis,
where X equals zero
^
^

• The next question:
How do we draw the line???
• Our goal for the line:
Fit the line as close as possible to all the data points for all values of
X.

Y
X
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15
1
2
3
4
5
6
7
8
9
10
Case: 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25
Children (Y): 2 5 1 9 6 3 1 0 3 7 7 2 4 2 1 0 1 2 4 3 0 1 2 5 7
Income 1=$10K (X): 3 4 9 5 4 12 14 10 1 4 3 11 4 9 13 10 7 5 2 5 15 11 8 3 2
How do we
minimize the
distance between a
line and all the
data points?

•How do we minimize the distance between a line and
all the data points?
•You already know of a statistic that minimizes the
distance between itself and all data values for a
variable--the mean!
•The mean minimizes the sum of squared deviations-
-it is where deviations sum to zero and where the
squared deviations are at their lowest value. (Y -
Y-bar)2
•

•Let’s “fit the line” to the place where squared
deviations from the line (vertically) are at their
lowest value (across all X’s).
•Minimize this: (Y - Y)2 Y = line
•Minimizing the sum of squared errors gives you the
unique, best fitting line for all the data points. It is the
line that is closest to all points.
^ ^

Y
X
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15
1
2
3
4
5
6
7
8
9
10
Case: 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25
Children (Y): 2 5 1 9 6 3 1 0 3 7 7 2 4 2 1 0 1 2 4 3 0 1 2 5 7
Income 1=$10K (X): 3 4 9 5 4 12 14 10 1 4 3 11 4 9 13 10 7 5 2 5 15 11 8 3 2
(Y - Y)2= ?
(9 - 4)2 = 25
(Y - Y)2= ?
(1 - 2)2 = 1
^
^

• (Y -Y)2 aka “sum of squared errors”
•There is a simple, elegant formula for “discovering”
the line that minimizes the sum of squared errors—
You don’t have to memorize!
((X - X)(Y -Y))
b = (X - X)2 a =Y - bX Y = a + bX
•This is the method of least squares, it gives our least
squares estimate and indicates why we call this
technique “ordinary least squares” or OLS regression
^
^

u
a
M
1
R
q u
s t
u
E
t
m
P
a
O
b
4
1
4
3
0 a
6
3
8
0
4
R
R
T
M
1
m
u a
d
e
u
F
S i
P
a
D
b
i c
a
4
0
0
4
9
0
0
(
I N
M
1
B
E
d a
i c
e t
d
e
f
s
t
i g
D
a
In fact, this is the output that SPSS would give you for the
data values:
Y = a + bX
^

Y
X
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15
1
2
3
4
5
6
7
8
9
10
Case: 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25
Children (Y): 2 5 1 9 6 3 1 0 3 7 7 2 4 2 1 0 1 2 4 3 0 1 2 5 7
Income 1=$10K (X): 3 4 9 5 4 12 14 10 1 4 3 11 4 9 13 10 7 5 2 5 15 11 8 3 2
The fitted line for our
example has the equation:
Y = 6 - .4X
If you were to draw any other line,
it would not
minimize (Y - Y)2
^
^

Y
X
0 1
1
2
3
4
5
6
7
8
9
10
Considering that our line minimizes  (Y - Y)2,
where would the regression cross data points for two
groups in a dichotomous independent variable?
^
0=Men: Mean = 6
1=Women: Mean = 4

Y
X
0 1
1
2
3
4
5
6
7
8
9
10
The difference of means will be the slope.
This is the same number that is tested for
significance in an independent samples t-test.
^
0=Men: Mean = 6
1=Women: Mean = 4
Slope = -2 ; Y = 6 – 2X

Y
X
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15
1
2
3
4
5
6
7
8
9
10
Case: 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25
Children (Y): 2 5 1 9 6 3 1 0 3 7 7 2 4 2 1 0 1 2 4 3 0 1 2 5 7
Income 1=$10K (X): 3 4 9 5 4 12 14 10 1 4 3 11 4 9 13 10 7 5 2 5 15 11 8 3 2
Getting back to interval-ratio independent
variables, the line is fitted among the
minimized squared vertical distance of all
data points from itself cumulatively for all
values of X.
(Y - Y)2
Y - Y
^
^

•Y = a + bX This equation gives the conditional mean
ofY at any given value of X.
• So… In reality, our line gives us the expected mean ofY given each
value of X
• The line’s equation tells you how the mean on your dependent variable
changes as your independent variable goes up.
^
Y
^
X
Y Y = Average
for Y at
each level
of X
^

• As you know, every mean has a distribution around it--so
there is a standard deviation. This is true for conditional
means as well. So, you also have a conditional standard
deviation.
• “Conditional Standard Deviation” or “Root Mean Square Error” equals
“approximate average deviation from the line.”
SSE  (Y -Y)2
• = n - 2 = n - 2
Y
^
X
Y
^
^

• The Assumption of Homoskedasticity:
• The variation around the line is the same no matter the X.
• The conditional standard deviation is for any given value of X.
• If there is a relationship between X andY, the conditional standard deviation is
going to be less than the standard deviation ofY--if this is so, you have
improved prediction of the mean value ofY by taking into account each level of
X.
• If there were no relationship, the conditional standard deviation would be the
same as the original, and the regression line would be flat at the mean ofY.
Y
X
Y Conditional
standard
deviation
Original
standard
deviation

•So guess what?
•We have a way to determine how much our
understanding ofY is improved when taking X into
account—it is based on the fact that conditional
standard deviations should be smaller thanY’s
original standard deviation.

•Proportional Reduction in Error
• Let’s call the variation around the mean inY “Error 1.”
• Let’s call the variation around the line when X is considered “Error 2.”
• But rather than going all the way to standard deviation to determine
error, let’s just stop at the basic measure, Sum of Squared Deviations.
• Error 1 (E1) =  (Y –Y)2 also called “Sum of Squares”
• Error 2 (E2) =  (Y –Y)2 also called “Sum of Squared Errors”
Y
Y Error 2
Error 1


•Proportional Reduction in Error
• To determine how much taking X into consideration reduces the variation
inY (at each level of X) we can use a simple formula:
E1 – E2 Which tells us the proportion or
E1 percentage of original error that
is Explained by X.
• Error 1 (E1) =  (Y –Y)2
• Error 2 (E2) =  (Y –Y)2
Y
X
Y
Error 2
Error 1


r2 = E1 - E2
E1
= TSS - SSE
TSS
=  (Y –Y)2 -  (Y –Y)2
 (Y –Y)2

r2 is called the “coefficient of
determination”…
It is also the square of the
Pearson correlation
Y
X
Y Error 2
Error 1

• R2
• Is the improvement obtained by using X (and drawing a line through the
conditional means) in getting as near as possible to everybody’s value for
Y over just using the mean forY alone.
• Falls between 0 and 1
• 1 means an exact fit (and there is no variation of scores around the regression
line)
• 0 means no relationship (and as much scatter around the line as in the originalY
variable and a flat regression line (slope = 0) through the mean ofY)
• Would be the same for X regressed onY as forY regressed on X
• Can be interpreted as the percentage of variability inY that is explained
by X.
• Some people get hung up on maximizing R2, but this is too bad because any
effect is still a finding—a small R2 only indicates that you haven’t told the
whole (or much of the) story of the relationship between your variables.

u
a
M
1
R
q u
s t
u
E
t
m
P
a
O
b
4
1
4
3
0 a
6
3
8
0
4
R
R
T
M
1
m
u a
d
e
u
F
S i
P
a
D
b
i c
a
4
0
0
4
9
0
0
(
I N
M
1
B
E
d a
i c
e t
d
e
f
s
t
i g
D
a
Back to the SPSS output:
r2
 (Y – Y)2 -  (Y – Y)2
 (Y – Y)2

71.194 ÷ 154.64 = .460

ANOVA
b
71.194
1
71.194
19.623
.000a
83.446
23
3.628
154.640
24
Regression
Residual
Total
Model
1
Sum of
Squares
df
Mean
Square
F
Sig.
Predictors: (Constant), INCOME
a.
Dependent Variable: CHILD
b.
X
Y
Mean
Q: So why did I see an ANOVA Table?
A: Levels of X can be thought of like
groups in ANOVA
…and the squared distance from the
line to the mean (Regression SS) is
equivalent to BSS—group mean to
big mean (but df = 1)
…and the squared distance from the
line to the data values on Y (Residual
SS) is equivalent to WSS—data value
to the group’s mean
… and the ratio of these forms an F
distribution in repeated sampling
If F is significant, X is explaining
some of the variation in Y.
BSS
WSS
TSS

Y
X
0 1
1
2
3
4
5
6
7
8
9
10 Using a dichotomous independent variable,
the ANOVA table in bivariate regression will
have the same numbers and ANOVA results
as a one-way ANOVA table would (and
compare this with an independent samples t-
test).
^
0=Men: Mean = 6
1=Women: Mean = 4
Slope = -2 ; Y = 6 – 2X
Mean = 5 BSS
WSS
TSS

Descriptive:
•The equation for your line is
a descriptive statistic. It tells
you the real, best-fitted line
that minimizes squared
errors.
Inferential:
•But what about the
population? What can we
say about the relationship
between your variables in
the population???
•The inferential statistics are
estimates based on the best-
fitted line.
Recall that statistics are divided between descriptive
and inferential statistics.

•The significance of F, you already understand.
• The ratio of Regression (line to the mean ofY) to Residual (line to data
point) Sums of Squares forms an F ratio in repeated sampling.
• Null: r2 = 0 in the population. If F exceeds critical F, then your variables have
a relationship in the population (X explains some of the variation inY).
Most extreme
5% of F’s
F = Regression SS / Residual SS

• What about the Slope (called “Coefficient”)?
• The slope has a sampling distribution that is normally distributed.
• So we can do a significance test.
-3 -2 -1 0 1 2 3
z-scores

Population Slope of relationship between
two interval-ratio variables
Slope
values
-3 -2 -1 0 1 2 3 Flippy Ruler

Conducting aTest of Significance for the slope of the Regression Line
By slapping the sampling distribution for the slope over a guess of the
population’s slope, Ho, we can find out whether our sample could have been
drawn from a population where the slope is equal to our guess.
1.Two-tailed significance test for -level = .05
2.Critical t = +/- 1.96
3.To find if there is a significant slope in the population,
Ho:  = 0
Ha:   0  (Y –Y )2
4.Collect Data n - 2
5.Calculate t (z): t = b – o s.e. =
s.e.  ( X – X )2
6.Make decision about the null hypothesis
7.Find P-value


u
a
M
1
R
q u
s t
u
E
t
m
P
a
O
b
4
1
4
3
0 a
6
3
8
0
4
R
R
T
M
1
m
u a
d
e
u
F
S i
P
a
D
b
i c
a
4
0
0
4
9
0
0
(
I N
M
1
B
E
d a
i c
e t
d
e
f
s
t
i g
D
a
Back to the SPSS output:
Of course, you get the
standard error and
t on your output,
…and the p-value too!

Y
X
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15
1
2
3
4
5
6
7
8
9
10
Case: 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25
Children (Y): 2 5 1 9 6 3 1 0 3 7 7 2 4 2 1 0 1 2 4 3 0 1 2 5 7
Income 1=$10K (X): 3 4 9 5 4 12 14 10 1 4 3 11 4 9 13 10 7 5 2 5 15 11 8 3 2
Y = 6 - .4X
So in our example,
the slope is
significant, there is a
relationship in the
population, and 46%
of the variation in
number of children is
explained by income.
^

•We’ve talked about the
summary of the
relationship, but not
about strength of
association.
•How strong is the
association between our
variables?
•For this we need
correlation.

Y
X
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15
1
2
3
4
5
6
7
8
9
10
Case: 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25
Children (Y): 2 5 1 9 6 3 1 0 3 7 7 2 4 2 1 0 1 2 4 3 0 1 2 5 7
Income 1=$10K (X): 3 4 9 5 4 12 14 10 1 4 3 11 4 9 13 10 7 5 2 5 15 11 8 3 2
So our equation is:
Y = 6 - .4X
The slope tells us
direction of
association… How
strong is that?
^

• To find the strength of the relationship between two variables, we
need correlation.
• The correlation is the standardized slope… it refers to the standard
deviation change inY when you go up a standard deviation in X.

1
2
3
4
5
6
7
8
9
10
Example of Low Negative Correlation

1
2
3
4
5
6
7
8
9
10
Example of High Negative Correlation

• The correlation is the standardized slope… it refers to the standard
deviation change inY when you go up a standard deviation in X.
(X - X)2
• Recall that s.d. of x, Sx = n - 1
(Y -Y)2
• and the s.d. of y, Sy = n - 1
Sx
• Pearson correlation, r = Sy b

• The PearsonCorrelation, r:
• tells the direction and strength of the relationship between continuous
variables
• ranges from -1 to +1
• is + when the relationship is positive and - when the relationship is
negative
• the higher the absolute value of r, the stronger the association
• a standard deviation change in x corresponds with r standard deviation
change inY

• The PearsonCorrelation, r:
• The pearson correlation is a statistic that is an inferential statistic too.
r - (null = 0)
• tn-2 = (1-r2) (n-2)
• When it is significant, there is a linear relationship between the two variables in
the population—it is not non-existent!

u
a
M
1
R
q u
s t
u
E
t
m
P
a
O
b
4
1
4
3
0 a
6
3
8
0
4
R
R
T
M
1
m
u a
d
e
u
F
S i
P
a
D
b
i c
a
4
0
0
4
9
0
0
(
I N
M
1
B
E
d a
i c
e t
d
e
f
s
t
i g
D
a
Our data’s correlation is .679. How strong is that?
Correlation, r,
is significant.

If you were to use the “correlate, bivariate” command, you’d get this ouput…
Correlation, r, is significant.

Correlation_and_Regression-3.ppt

Recommended

More Related Content

Similar to Correlation_and_Regression-3.ppt (20)

Recently uploaded (20)

Correlation_and_Regression-3.ppt