Békés-Kézdi: Data Analysis, Chapter 09: Generalizing

regression result

Data Analysis for Business, Economics,

and Policy
Gábor Békés (Central European University)
Gábor Kézdi (University of Michigan)
Cambridge University Press, 2021
gabors-data-analysis.com

Central European University

Version: v3.1 License: CC BY-NC 4.0
Any comments or suggestions:
[email protected]
Generalizing Results Testing, p-values Intervals for Predicted Values External validity CS:B1 CS:A1 CS:A2 CS:A3 CS:A4

Regression and statistical inference

Data Analysis for Business, Economics, and Policy 2 / 47 Gábor Békés (Central European University)
Generalizing Results Testing, p-values Intervals for Predicted Values External validity CS:B1 CS:A1 CS:A2 CS:A3 CS:A4

Generalizing: reminder
I We have uncovered some pattern in our data. We are interested in generalize the
results.

I Question: Is the pattern we see in our data

I True in general ?
I or is it just a special case what we see?
I Need to specify the situation
I to what we want to generalize
I Inference - the act of generalizing results
I From a particular dataset to other situations or datasets.

I From a sample to population/ general pattern = statistical inference

I Beyond (other dates, countries, people, firms) = external validity
Data Analysis for Business, Economics, and Policy 3 / 47 Gábor Békés (Central European University)
Generalizing Results Testing, p-values Intervals for Predicted Values External validity CS:B1 CS:A1 CS:A2 CS:A3 CS:A4

Generalizing Linear Regression Coefficients from a Dataset

I We estimated the linear model

I β̂ is the average difference in y in the dataset between observations that are
different in terms of x by one unit.
I ŷi best guess for the expected value (average) of the dependent variable for
observation i with value xi for the explanatory variable in the dataset.
I Sometimes all we care about are patterns, predicted values, or residuals, in the
data we have.
I Often interested in patterns and predicted values in situations that are not limited
to the dataset we analyze.
I To what extent predictions / patterns uncovered in the data generalize to a situation
we care about.

Data Analysis for Business, Economics, and Policy 4 / 47 Gábor Békés (Central European University)
Generalizing Results Testing, p-values Intervals for Predicted Values External validity CS:B1 CS:A1 CS:A2 CS:A3 CS:A4

Statistical Inference: Confidence Interval

I The 95% CI of the slope coefficient of a linear regression

I similar to estimating a 95% CI of any other statistic.
h i
CI (β̂)95% = β̂ − 2SE (β̂), β̂ + 2SE (β̂)

I Formally: 1.96 instead of 2. (computer uses 1.96 – mentally use 2)

I The standard error (SE) of the slope coefficient
I is conceptually the same as the SE of any statistic.
I measures the spread of the values of the statistic across hypothetical repeated
samples drawn from the same population (or general pattern) that our data
represents

Data Analysis for Business, Economics, and Policy 5 / 47 Gábor Békés (Central European University)
Generalizing Results Testing, p-values Intervals for Predicted Values External validity CS:B1 CS:A1 CS:A2 CS:A3 CS:A4

Standard Error of the Slope

The simple SE formula of the slope is I A smaller standard error translates into
I narrower confidence interval,
Std [e] I estimate of slope coefficient with more
SE (β̂) = √
nStd [x] precision.
I Where: I More precision if
I Residual: e = y − α̂ − β̂x I smaller the standard deviation of the
I Std[e], the standard deviation of the residual – better fit, smaller errors.
regression residual, I larger the standard deviation of the
I Std[x], the standard deviation of the explanatory variable – more variation in x
explanatory
√ variable, is good.
I n the square root of the number of I more observations are in the data.
observations in the data.
√
I Smaller sample – may use n − 2.
I This formula is correct assuming
homoskedasticity
Data Analysis for Business, Economics, and Policy 6 / 47 Gábor Békés (Central European University)
Generalizing Results Testing, p-values Intervals for Predicted Values External validity CS:B1 CS:A1 CS:A2 CS:A3 CS:A4

Heteroskedasticity Robust SE
I Simple SE formula is not correct in general.
I Homoskedasticity assumption: the fit of the regression line is the same across the
entire range of the x variable
I In general this is not true
I Heteroskedasticity: the fit may differ at different values of x so that the spread of
actual y around the regression is different for different values of x

I Heteroskedastic-robust SE formula (White or Huber ) is correct in both cases

I Same properties as the simple formula: smaller when Std[e] is small, Std[x] is large
and n is large
I E.g. White formula uses the estimated errors’ square from the model and weight the
observations when calculating the SE [β̂]
I Note: there are many heteroskedastic-robust formula, which uses different weighting
techniques. Usually referred as ‘HC0’, ‘HC1’, ... , ‘HC4’.
I R’s default ‘HC2’ is perfect.

Data Analysis for Business, Economics, and Policy 7 / 47 Gábor Békés (Central European University)
Generalizing Results Testing, p-values Intervals for Predicted Values External validity CS:B1 CS:A1 CS:A2 CS:A3 CS:A4

The CI Formula in Action

I Run linear regression

I Compute endpoints of CI using SE
I 95% CI of slope and intercept
 
I β̂ ± 2SE β̂ ; α̂ ± 2SE (α̂)

I In regression, as default, use robust SE.

I Sometimes: homoskedastic and heteroskedasticity robust SEs are similar.
I Sometimes: heteroskedasticity robust SE is larger – and rightly so.
I Coefficient estimates, R 2 etc. are remain the same.

Data Analysis for Business, Economics, and Policy 8 / 47 Gábor Békés (Central European University)
Generalizing Results Testing, p-values Intervals for Predicted Values External validity CS:B1 CS:A1 CS:A2 CS:A3 CS:A4

Testing if (true) beta is zero

I Testing hypotheses: decide if a statement about a general pattern is true.

I Most often: Dependent variable and the explanatory variable are related at all?
I The null and the alternative:

H0 : βtrue = 0, HA : βtrue 6= 0

I The t-statistic is:

β̂ − 0
t=
SE (β̂)
I Often t = 2 (1.96) is the critical value, which corresponds to 95% CI.
(t = 2.6 → 99%)

Data Analysis for Business, Economics, and Policy 9 / 47 Gábor Békés (Central European University)
Generalizing Results Testing, p-values Intervals for Predicted Values External validity CS:B1 CS:A1 CS:A2 CS:A3 CS:A4

Testing if (true) beta is zero

Practical guidance:
I Choose a critical value.
I p-vale, the probability of a false positive in our dataset
I Balancing act: false positive (FP) and negative (FN)
I Higher critical value
I FP: less likely (less likely rejection of the null).
I FN: more likely (high risk of not rejecting a null even though it’s false)

Data Analysis for Business, Economics, and Policy 10 / 47 Gábor Békés (Central European University)
Generalizing Results Testing, p-values Intervals for Predicted Values External validity CS:B1 CS:A1 CS:A2 CS:A3 CS:A4

Language: significance of regression coefficients

I A coefficient is said to be “statistically significant”

I If its confidence interval does not contain zero
I So true value unlikely to be zero
I Level of significance refers to what % confidence interval
I Language uses the complement of the CI
I Most common: 5%, 1%
I Significant at 5%
I Zero is not in 95% CI, Often denoted p < 0.05
I Significant at 1%
I Zero is not in 99% CI, (p < 0.01)

Data Analysis for Business, Economics, and Policy 11 / 47 Gábor Békés (Central European University)
Generalizing Results Testing, p-values Intervals for Predicted Values External validity CS:B1 CS:A1 CS:A2 CS:A3 CS:A4

Ohh, that p=5% cutoff

I When testing, you start with a critical value first

I Often the standard to publish a result is to have a p value below 5%.
I Arbitrary, but... [major discussion]
I Some fun: here (+R code)
I If you find a result that cannot be told apart from 0 at 1% (max 5%), you should
say that explicitly.

Data Analysis for Business, Economics, and Policy 12 / 47 Gábor Békés (Central European University)
Generalizing Results Testing, p-values Intervals for Predicted Values External validity CS:B1 CS:A1 CS:A2 CS:A3 CS:A4

Dealing with 5-10%

I Sometimes regression result will not be significant at 5% but will be at 10%.

I What not to do? Avoid language like...
I "a barely detectable statistically significant difference" (p=0.073)
I "a margin at the edge of significance" (p=0.0608)
I "not significant in the normally accepted statistical sense" (p=0.064)
I "slight tendency toward significance" (p=0.086)
I "slightly missed the conventional level of significance" (p=0.061)
I More here

Data Analysis for Business, Economics, and Policy 13 / 47 Gábor Békés (Central European University)
Generalizing Results Testing, p-values Intervals for Predicted Values External validity CS:B1 CS:A1 CS:A2 CS:A3 CS:A4

Dealing with 5-10%

I Sometimes regression result will not be significant at 1% (5%) but will be at 10%.
I What to take? It depends. (our view...)
I Sometimes you work on a proposal. Proof of concept.
I To be lenient is okay.
I Say the point estimate and note the 95% confidence interval.

I Sometimes looking for a proof. Beyond reasonable doubt.

I Gender equality to be defended for a judge.
I Here you wanna be below 1%
I If not, say the p-value and note that at 1% you cannot reject the null of no
difference.
I Publish the p-value. Be honest...

Data Analysis for Business, Economics, and Policy 14 / 47 Gábor Békés (Central European University)
Generalizing Results Testing, p-values Intervals for Predicted Values External validity CS:B1 CS:A1 CS:A2 CS:A3 CS:A4

p-Hacking
I Very often many steps lead to a regression analysis
I Many: arbitrary decisions
I Often we work with a bias: looking to reinforce expectations
I Show a "significant" result.

I p-hacking = do many versions, only showing results significant at, say, 5%

I Danger: methods are fine, but only not everything is presented

I Present your most conservative result first

I Example: if uncertain, keep extreme values in.
I Show robustness checks: many additional regressions with different decisions
I May add that keeping extreme values weakens findings
I p-Hacking is linked to publication bias
I More: Ch06.9
Data Analysis for Business, Economics, and Policy 15 / 47 Gábor Békés (Central European University)
Generalizing Results Testing, p-values Intervals for Predicted Values External validity CS:B1 CS:A1 CS:A2 CS:A3 CS:A4

Chance Events And Size of Data

I Finding patterns by chance may go away with more observations

I Individual observations may be less influential
I Effects of idiosyncratic events may average out
I E.g.: more dates
I Specificities to a single dataset may be less important if more sources
I E.g.: more hotels
I More observations help only if
I Errors and idiosyncrasies affect some observations but not all
I Additional observations are from appropriate source
I If worried about specificities of Vienna more observations from Vienna would not help

Data Analysis for Business, Economics, and Policy 16 / 47 Gábor Békés (Central European University)
Generalizing Results Testing, p-values Intervals for Predicted Values External validity CS:B1 CS:A1 CS:A2 CS:A3 CS:A4

Prediction uncertainty
I Goal: predicting the value of y for observations outside the dataset, when only the
value of x is known.
I We predict y based on coefficient estimates, which are relevant in the general
pattern/population. With linear regression you have a simple model:

yi = α̂ + β̂xi + i
I The estimated statistic here is a predicted value for a particular observation ŷj . For
an observation j with known value xj this is

ŷj = α̂ + β̂xj
I Two kinds of intervals:
I Confidence interval for the predicted value/regression line - uncertainty about α̂, β̂
I Prediction interval - uncertainty about α̂, β̂ and i
Data Analysis for Business, Economics, and Policy 17 / 47 Gábor Békés (Central European University)
Generalizing Results Testing, p-values Intervals for Predicted Values External validity CS:B1 CS:A1 CS:A2 CS:A3 CS:A4

Confidence interval of the regression line I.

I Confidence interval (CI) of the predicted value = the CI of the regression line.
I The predicted value ŷj is based on α̂ and β̂ only.
I The CI of the predicted value combines the CI for α̂ and the CI for β̂.
I What value to expect if we know the value of xj and we have estimates of
coefficients α̂ and β̂ from the data.
I The 95% CI of the predicted value - 95%CI (ŷj ) is
I the value estimated from the sample
I plus and minus its standard error.

Data Analysis for Business, Economics, and Policy 18 / 47 Gábor Békés (Central European University)
Generalizing Results Testing, p-values Intervals for Predicted Values External validity CS:B1 CS:A1 CS:A2 CS:A3 CS:A4

Confidence interval of the regression line II.

I Predicted average y has a standard error (homoskedastic case)

95%CI (ŷj ) = ŷ ± 2SE (ŷj )

s
1 (xj − x̄)2
SE (ŷj ) = Std[e] +
n nVar [x]

I Based on formula for regression coefficients, it is small if:

I coefficient SEs are small (depends on Std[e] and Std[x]).
I Particular xj is close to the mean of x
I We have many observations n
I The role of n (sample size), here is even larger.
I Use robust SE formula in practice, but a simple formula is instructive

Data Analysis for Business, Economics, and Policy 19 / 47 Gábor Békés (Central European University)
Generalizing Results Testing, p-values Intervals for Predicted Values External validity CS:B1 CS:A1 CS:A2 CS:A3 CS:A4

Confidence interval of the regression line - use

I Can be used for any model

I Spline, polynomial
I The way it is computed is different for different kinds of regressions (usually
implemented in R packages)
I always true that the CI is narrower
I the smaller Std[e],
I the larger n and
I the larger Std[x]
I In general, the CI for the predicted value is an interval that tells where to expect
average y given the value of x in the population, or general pattern, represented by
the data.

Data Analysis for Business, Economics, and Policy 20 / 47 Gábor Békés (Central European University)
Generalizing Results Testing, p-values Intervals for Predicted Values External validity CS:B1 CS:A1 CS:A2 CS:A3 CS:A4

Prediction interval

I Prediction interval answers:

I Where to expect the particular yj value if we know the corresponding xj value and
the estimates of the regression coefficients from the data.
I Difference between CI and PI.
I The CI of the predicted value is about ŷj : where to expect the average value of the
dependent variable if we know xj .
I The PI (prediction interval) is about yj itself not its average value: where to expect
the actual value of yj if we know xj .
I So PI starts with CI. But adds additional uncertainty (Std[i ]) that actual yj will
be around its conditional.
I What shall we expect in graphs?

Data Analysis for Business, Economics, and Policy 21 / 47 Gábor Békés (Central European University)
Generalizing Results Testing, p-values Intervals for Predicted Values External validity CS:B1 CS:A1 CS:A2 CS:A3 CS:A4

Confidence vs Prediction interval

Confidence interval Prediction interval

Data Analysis for Business, Economics, and Policy 22 / 47 Gábor Békés (Central European University)
Generalizing Results Testing, p-values Intervals for Predicted Values External validity CS:B1 CS:A1 CS:A2 CS:A3 CS:A4

More on prediction interval

I Summarizes the additional uncertainty:
the actual yj value is expected to be
I The formula for the 95% prediction spread around its average value.
interval is I The magnitude of this spread is best
estimated by the standard deviation
95%PI (ŷj ) = ŷ ± 2SPE (ŷj ) of the residual.
s
1 (xj − x̄)2 I With SPE, no matter how large the
SPE (ŷj ) = Std[e] 1 + + sample we can always expect actual y
n nVar [x]
values to be spread around their
I SPE – Standard Prediction Error (SE average values.
I In the formula, all elements get very
of prediction)
I It does matter here which kind of SE small if n gets large, except for the
new element.
you use!

Data Analysis for Business, Economics, and Policy 23 / 47 Gábor Békés (Central European University)
Generalizing Results Testing, p-values Intervals for Predicted Values External validity CS:B1 CS:A1 CS:A2 CS:A3 CS:A4

External validty

Data Analysis for Business, Economics, and Policy 24 / 47 Gábor Békés (Central European University)
Generalizing Results Testing, p-values Intervals for Predicted Values External validity CS:B1 CS:A1 CS:A2 CS:A3 CS:A4

External validity

I Statistical inference helps us generalize to the population or general pattern

I Is this true beyond (other dates, countries, people, firms)?

I As external validity is about generalizing beyond what our data represents, we
can’t assess it using our data.
I We’ll never really know. Only think, investigate, make assumption, and hope...

Data Analysis for Business, Economics, and Policy 25 / 47 Gábor Békés (Central European University)
Generalizing Results Testing, p-values Intervals for Predicted Values External validity CS:B1 CS:A1 CS:A2 CS:A3 CS:A4

Data analysis to help assess external validity

I Analyzing other data can help!

I Focus on β, the slope coefficient on x.
I The three common dimensions of generalization are time, space, and other groups.

I To learn about external validity, we always need additional data, on say, other
countries or time periods.
I We can then repeat regression and see if slope is similar!

Data Analysis for Business, Economics, and Policy 26 / 47 Gábor Békés (Central European University)
Generalizing Results Testing, p-values Intervals for Predicted Values External validity CS:B1 CS:A1 CS:A2 CS:A3 CS:A4

Stability of hotel prices - idea

I Here we ask different questions: whether we can infer something about the
price–distance pattern for situations outside the data:
I Is the slope coefficient close to what we have in Vienna, November, weekday:
I Other dates (focus in class)
I Other cities
I Other type of accommodation: apartments
I Compare them to our benchmark model result
I Learn about uncertainty when using model to some types of external validity.

Data Analysis for Business, Economics, and Policy 27 / 47 Gábor Békés (Central European University)
Generalizing Results Testing, p-values Intervals for Predicted Values External validity CS:B1 CS:A1 CS:A2 CS:A3 CS:A4

Why carrying out such analysis?

I Such a speculation may be relevant:

I Find a good deal in the future without estimating a new regression but taking the
results of this regression and computing residuals accordingly.
I Be able to generalize to other groups, date and places.

Data Analysis for Business, Economics, and Policy 28 / 47 Gábor Békés (Central European University)
Generalizing Results Testing, p-values Intervals for Predicted Values External validity CS:B1 CS:A1 CS:A2 CS:A3 CS:A4

Benchmark model
The benchmark model is a spline with a knot at 2 miles.

ln(y )E = α1 + β1 x1x<2m + (α2 + β2 x)1x≥2m

Data is restricted to 2017, November weekday in Vienna, 3-4 star hotels, within 8 miles.

I Model has three output variables: α = 5.02, β1 = −0.31, β2 = 0.02

I α: Hotel prices are on average 151.41 euro (exp(5.02)) at the city center
I β1 : hotels that are within 2 miles from the city center, prices are 0.31 log units or
36% (exp(0.31) − 1) cheaper, on average, for hotels that are 1 mile farther away
from the city center.
I β2 : hotels in the data that are beyond 2 miles from the city center, prices are 2%
higher, on average, for hotels that are 1 mile farther away from the city center.
Data Analysis for Business, Economics, and Policy 29 / 47 Gábor Békés (Central European University)
 Generalizing Results Testing, p-values Intervals for Predicted Values External validity CS:B1 CS:A1 CS:A2 CS:A3 CS:A4

Comparing dates

(1) (2) (3) (4)

VARIABLES 2017-NOV-weekday 2017-NOV-weekend 2017-DEC-holiday 2018-JUNE-weekend

dist_0_2 -0.31** -0.44** -0.36** -0.31**

(0.038) (0.052) (0.041) (0.037)
dist_2_7 0.02 -0.00 0.07 0.04
(0.033) (0.036) (0.050) (0.039)
Constant 5.02** 5.51** 5.13** 5.16**
(0.042) (0.067) (0.048) (0.050)

Observations 207 125 189 181

R-squared 0.314 0.430 0.382 0.306
Note: Robust standard errors in parentheses *** p<0.01, ** p<0.05, * p<0.1
Source: hotels-europe data. Vienna, reservation price for November and December 2017, June in 2018

Data Analysis for Business, Economics, and Policy 30 / 47 Gábor Békés (Central European University)
Generalizing Results Testing, p-values Intervals for Predicted Values External validity CS:B1 CS:A1 CS:A2 CS:A3 CS:A4

Comparing dates - interpretation

I November weekday and the June weekend: β̂1 = 0.31

I Estimate is similar for December (-0.36 log units)
I Different for the November weekend: they are 0.44 log units or 55% (exp(0.44) − 1)
cheaper during the November weekend.
I The corresponding 95% confidence intervals overlap somewhat: they are [-0.39,-0.23]
and [-0.54,-0.34].
I Thus we cannot say for sure that the price–distance patterns are different during the
weekday and weekend in November.

Data Analysis for Business, Economics, and Policy 31 / 47 Gábor Békés (Central European University)
Generalizing Results Testing, p-values Intervals for Predicted Values External validity CS:B1 CS:A1 CS:A2 CS:A3 CS:A4

Stability of hotel prices - takeaway

I Fairly stable overtime but uncertainty is larger

I For more, read the case study B in Chapter 09

I Evidence of some external validity in Vienna

I External validity – if model applied beyond data, there is additional uncertainty!

Data Analysis for Business, Economics, and Policy 32 / 47 Gábor Békés (Central European University)
Generalizing Results Testing, p-values Intervals for Predicted Values External validity CS:B1 CS:A1 CS:A2 CS:A3 CS:A4

Case study A: Going through the process

Data Analysis for Business, Economics, and Policy 33 / 47 Gábor Békés (Central European University)
Generalizing Results Testing, p-values Intervals for Predicted Values External validity CS:B1 CS:A1 CS:A2 CS:A3 CS:A4

Case Study: Gender gap in earnings?

I Earning determined by many factor

I The idea of gender gap:
I Is there a systematic wage differences between male and female workers?

Data Analysis for Business, Economics, and Policy 34 / 47 Gábor Békés (Central European University)
Generalizing Results Testing, p-values Intervals for Predicted Values External validity CS:B1 CS:A1 CS:A2 CS:A3 CS:A4

Case Study: Gender gap - How data is born?

I Current Population Survey (CPS) of the U.S.
I Administrative data
I Large sample of households
I Monthly interviews
I Rotating panel structure: interviewed in 4 consecutive months, then not interviewed
for 8 months, then interviewed again in 4 consecutive months
I Weekly earnings asked in the “outgoing rotation group”
I In the last month of each 4-month period
I See more on MORG: “Merged outgoing rotation group”
I Sample restrictions used:
I Sample includes individuals of age 16-65
I Employed (has earnings)
I Self-employed excluded

Data Analysis for Business, Economics, and Policy 35 / 47 Gábor Békés (Central European University)
Generalizing Results Testing, p-values Intervals for Predicted Values External validity CS:B1 CS:A1 CS:A2 CS:A3 CS:A4

Case Study: Gender gap - the data

I Download data for 2014 (316,408 observations) with implemented restrictions

N = 149, 316
I Weekly earnings in CPS
I Before tax
I Top-coded very high earnings
I at $2,884.6 (top code adjusted for inflation, 2.5% of earnings in 2014)
I Would be great to measure other benefits, too (yearly bonuses, non-wage benefits).
But we don’t measure those.
I Need to control for hours
I Women may work systematically different in hours than men.
I Divide weekly earnings by ‘usual’ weekly hours (part of questionnaire)

Data Analysis for Business, Economics, and Policy 36 / 47 Gábor Békés (Central European University)
Generalizing Results Testing, p-values Intervals for Predicted Values External validity CS:B1 CS:A1 CS:A2 CS:A3 CS:A4

Case Study: Gender gap - conditional descriptives

Gender mean p25 p50 p75 p90 p95

Male $ 24 13 19 30 45 55
Female $ 20 11 16 24 36 45
% gap -17% -16% -18% -20% -20% -18%

I 17% difference on average in per hour earnings between men and women
I For linear regression analysis, we will use ln wage to compare relative difference.

Data Analysis for Business, Economics, and Policy 37 / 47 Gábor Békés (Central European University)
Generalizing Results Testing, p-values Intervals for Predicted Values External validity CS:B1 CS:A1 CS:A2 CS:A3 CS:A4

Case Study: Gender gap in comp science occupation - Analysis

I One key reason for gap could be women being sectors / occupations that pay less.
Focus on a single one: Computer science occupations, N = 4, 740

ln(w )E = α + β × Gfemale

I We regressed log earnings per hour on G binary variable that is one if the
individual is female and zero if male.
I The log-level regression estimate is β̂ = −0.1475
I female computer science field employee earns 14.7 percent less, on average, than
male with the same occupation in this dataset.
I Statistical inference based on 2014 data.
I SE: .0177; 95% CI: [-.182 -.112]
I Simple vs robust SE - Here no practical difference.

Data Analysis for Business, Economics, and Policy 38 / 47 Gábor Békés (Central European University)
Generalizing Results Testing, p-values Intervals for Predicted Values External validity CS:B1 CS:A1 CS:A2 CS:A3 CS:A4

Case Study: Gender gap in comp science occupation - Generalizing

I In 2014 in the U.S.

I the population represented by the data
I we can be 95% confident that the average difference between hourly earnings of
female CS employee versus a male one was -18.2% to -11.2%.
I This confidence interval does not include zero.
I Thus we can rule out with a 95% confidence that their average earnings are the
same.
I We can rule this out at 99% confidence as well

Data Analysis for Business, Economics, and Policy 39 / 47 Gábor Békés (Central European University)
Generalizing Results Testing, p-values Intervals for Predicted Values External validity CS:B1 CS:A1 CS:A2 CS:A3 CS:A4

Case Study: Gender gap in market analyst occupation

I Market research analysts and marketing specialists, N = 281, where females are
61%.
I Average hourly wage is $29 (sd:14.7)
I The regression estimate is β̂ = −0.113:
I Female market research analyst employee earns 11.3 percent less, on average, than
men with the same occupation in this dataset.
I Generalization:
I SE [β̂]: .061; 95% CI: [-.23 +0.01]
I We can be 95% confident that the average difference between hourly earnings of
female CS employee versus a male one was -23% to +1% in the total US population
I This confidence interval does include zero. Thus, we can not rule out with a 95%
confidence that their average earnings are the same. (p = 0.068)
I More likely, though, female market analysts earn less.
I we can rule out with a 90% confidence that their average earnings are the same

Data Analysis for Business, Economics, and Policy 40 / 47 Gábor Békés (Central European University)
Generalizing Results Testing, p-values Intervals for Predicted Values External validity CS:B1 CS:A1 CS:A2 CS:A3 CS:A4

Our two samples. What is the source of difference in CI?

I Computer and Mathematical Occupations
I 4740 employees, Female: 27.5%
I The regression estimate of slope: -0.1475 ; 95% CI: [-.1823 -.1128]
I Market research analysts and marketing specialists
I 281 employees, Female: 61%
I The regression estimate of slope is -0.113; 95% CI: [-.23 +0.01]
I Why the difference?
I True difference: gender gap is higher in CS.
I Statistical error: sample size issue −→ in small samples we may find more variety of
estimates. (Why? Remember the SE formula.)
I Which explanation is true?
I We do not know!
I Need to collect more data in CS industry.

Data Analysis for Business, Economics, and Policy 41 / 47 Gábor Békés (Central European University)
Generalizing Results Testing, p-values Intervals for Predicted Values External validity CS:B1 CS:A1 CS:A2 CS:A3 CS:A4

Confidence vs Prediction interval

Confidence interval Prediction interval

Data Analysis for Business, Economics, and Policy 42 / 47 Gábor Békés (Central European University)
Generalizing Results Testing, p-values Intervals for Predicted Values External validity CS:B1 CS:A1 CS:A2 CS:A3 CS:A4

Capture non-linearity with functional form

Lowess and scatterplot Various functional forms overlaid

Data Analysis for Business, Economics, and Policy 43 / 47 Gábor Békés (Central European University)
Generalizing Results Testing, p-values Intervals for Predicted Values External validity CS:B1 CS:A1 CS:A2 CS:A3 CS:A4

Average log earnings and age: regressions with CI

Data Analysis for Business, Economics, and Policy 44 / 47 Gábor Békés (Central European University)
Generalizing Results Testing, p-values Intervals for Predicted Values External validity CS:B1 CS:A1 CS:A2 CS:A3 CS:A4

Testing hypotheses about regression coefficients

I Recall that the coefficient estimate was 0.11

I To formally test whether average earnings are the same by gender, we simply test
if the coefficient on the binary variable is zero
I against the alternative that it is not zero.
I Corresponding t-statistic is 1.8.
I The critical values for a 5% significance level are ±2, and 1.8 falls within the
critical values not outside of them.
I Thus, we cannot reject the null of equal average earnings at a 5% level of
significance.
I Same result with the p-value: p = 0.07 > 0.05

Data Analysis for Business, Economics, and Policy 45 / 47 Gábor Békés (Central European University)
Generalizing Results Testing, p-values Intervals for Predicted Values External validity CS:B1 CS:A1 CS:A2 CS:A3 CS:A4

Case Study: Earnings and age - presenting regression table

Model:
Variables ln wage
I ln wage = α + βfemale
I Only one industry: market Female -0.11
analysts, N = 281 (0.062)
I Robust standard errors in Constant 3.31**
(0.049)
parentheses *** p<0.01, **
p<0.05, * p<0.1. Observations 281
R-squared 0.012

Data Analysis for Business, Economics, and Policy 46 / 47 Gábor Békés (Central European University)
Generalizing Results Testing, p-values Intervals for Predicted Values External validity CS:B1 CS:A1 CS:A2 CS:A3 CS:A4

Case Study: Earnings and age - presenting regression table

Model:
VARIABLES ln wage
I ln wage = α + f (age)
age 0.014**
I Only one industry: (0.003)
market analysts, Constant 2.732**
N = 281 (0.101)
I Robust standard
Observations 281
errors in parentheses R-squared 0.098
*** p<0.01, **
p<0.05, * p<0.1.

Data Analysis for Business, Economics, and Policy 47 / 47 Gábor Békés (Central European University)