Analyzing data in a multivariable, probabilistic world

Statistical control
Falsifiability: Informative comparative scenarios

Javier M. Rodríguez, Ph.D.

Mary Toepelt Nicolai and George S. Blair Assistant Professor
Public Policy, Department of Politics and Government
Inequality and Policy Research Center, co-Director
Statistical Control
 Scenario: Median income increase between 1990 and 2017.

 What is the message of the table?

 Let’s imagine that our research question wants to address the

fact that the increase in income, 1990-2017, may be
associated with gender.
 Let’s, then, look at the same table disaggregated by gender.
o Q: Why would we like to see this table disaggregated by
gender?
o Q: What do we mean, exactly, by “disaggregated”?
Statistical Control

 What is the main message of the table?

o Q: Would you like to see any other type of disaggregation?
Why?
o Q: What is it that you are attempting to do by
“disaggregating”?
Statistical Control
 What conclusions do the data suggest about
median income increase, 1990-2017?

 What pattern would we expect the data to display

if there was a median income increase between
1990 and 2017?

 The underlying discovered association between

gender, education, and income was only possible
because of statistical control.
 In this case, to explain the change in median
income between 1990 and 2017, we
controlled for gender and education.
 By controlling for gender and education, we
were able to identify the independent
contributions of gender and education over
the income change between 1990 and 2017.
 Statistical control is a tool for explanation.
Statistical Control
 Q: What’s the paradox?
 Overall income increased while it either
stagnated or decreased in all educational groups.
 Q: What explains the observed changes in median
income between 1990 and 2017?
 The educational profile of workers have changed
drastically 1990-2017.
 And there are income differences across
educational levels: higher income for the
highly educated.
 This educational change occurred along gender
lines: A higher share of the workforce with high
education are females in 2017 relative to 1990.
 But females make less money than males.
➤ Thus, overall income increases because there is
more high-education people working, but it
decreases within group because it is the group
(females) that makes less money the group
gaining higher education.
 Statistical control is what regression does
• When we include education as a “control variable”—also known as a “confounder”
(because it is confounded with the association between income and gender)—in a
regression model, we are making sure to correctly assign to gender the effect on median
income that belongs to gender alone (and not to the portion of educational changes that
belong to differential educational attainment between the genders, 1990-2017).
• We would say, then, that the effect of gender on income is independent from education
and, likewise, that the effect of education on income is independent from gender.
• That is, the differences in income between males and females, 1990-2017, are
irrespective of the changes in education between the genders, 1990-2017.
• If we do not control for education, we will attribute to gender what belongs to both gender
and education (to the degree to which they are confounded in relation to income).
• That is why we choose one independent variable of interest (here, gender), and a set of
“control variables” or “confounders”—which will allow us to approximate the true size of
the effect of gender on income.
Statistical Control
 Some morals of the median-income increase story:

 Q: If income is actually decreasing across subgroups, is “really” income increasing?

➰ Data rarely “speak for themselves.”

• It is easy to jump to false conclusions, even when “sticking to the facts.”
• Do not confuse the what with the why or the how.

➰ Good analysis usually boils down to generating enlightening comparisons.

• Always ask: “What comparisons are necessary to address the question before us?”

➰ Technique is important, but insight and careful thinking are more important.
• Learning to do good analysis cannot be reduced to some programmed algorithm.

➰ The social world is highly multivariate. Bivariate relationships can be greatly misleading in a
multivariate world.
• … and, for example, nevertheless, much (most?) political debate and ideological rhetoric
tends to be bivariate in character. The death penalty will reduce crime. Lower taxes will
increase economic growth. More stringent testing will cause schools to perform better…
Statistical Control
 Some morals of the median-income increase story:

➰ To understand how (not “why”) X influences Y in a multivariate world requires that we

“control” for lurking influences on Y that are “confounded” with X.
• The ideas of control and confounding are fundamental: They play a central role in any
sophisticated interpretation of evidence.

➰ Some technical advice:

• Always ask: “What contingency table will illuminate the question at hand?”
• Regression analysis is a statistical tool to study [many] contingency tables
[simultaneously].
• It estimates weighted averages of Y (using all the percentages of the categories
of the independent variables [as weights]).
• Test for “no effect” of X on Y.
• Note that, in regression, we model the uncertainty of “what is not” and not the
certainty of “what is”—two different worldviews!
• Close to the idea of falsifiability: A sound research design should incorporate a
mechanism to demonstrate that results can be wrong.
Comparative scenarios
Falsifiability test

Offers proof scenarios in which the estimated relationship of

interest (i.e., the coefficient of the independent variable of
interest) can be proven to be wrong.

⁍ The null hypothesis—i.e., the possibility that there is no

relationship between the IV of interest and the DV—is the
default in scientific inquiry.
• You are testing the null hypothesis: You are testing that
there is no relationship in the first place.
• The critical level of statistical significance—i.e., the
level of certainty about the relationship—is quite high.
• (Just think, how many times, in how many aspects of
life, you have been at leas 95% certain of
something?)

⁍ Statistical control is a second source of falsifiability: Each

additional control variable you add to the model is an
Comparative scenarios
Falsifiability test

⁍ Statistical control…
• The hypothesis you test under different model specifications is
a different one, every time.
• It tests how sensitive is your explanatory theory to
different scenarios in which the proposed theory could be
wrong.

⁍ The “placebo test”: A scenario in which a group of interest is

subjected to the “fake” influence of the process under study.
• Imagine you tell all students in a community college that they
are now participants of a program (inserted in a 1-year course
A) that will help them be more academically successful. But
half of students will “officially” start the program now and the
other half in 6 months. During the first 6 months of the
program, the students who have not officially started the
program comprise the placebo group—which is different from a
comparison group that has not been told that they are
participants of the program.
Comparative scenarios
Falsifiability test

⁍ “Negative” comparative scenarios (do NOT confuse with placebo test)

✤ Negative comparison:
 One comparison group in which you do not expect to see
differences in the DV.
 Ex: Imagine you are studying graduation rates in community
colleges (CC), and one CC of interest is located in the same
district of a high school. You can use that high school as a
negative comparison—and you should expect to see no
detectable increase differences in graduation rates in that
school compared to those of the CC.
 Eve though that school shares the same contextual
characteristics and serves the same population of the
CC, this is a high school, not a CC, and it is unrelated to
CC-specific processes and programs.
 If you find similar graduation rate fluctuations in the HS
and the CC, it is a sign that findings were driven by
shared contextual characteristics, and not due to
Comparative scenarios
Falsifiability test

✤ Negative treatment (or negative explanatory variable):

• One treatment (policy/program/intervention) or explanatory variable
that should not produce the same effects on the DV of the
policy/program/intervention or independent variable of interest.
• Ex 1: Imagine we are studying the GPA effects of pregnancy as a
biological process among high school girls. Imagine we identify the
respective high school boys who will become fathers. We, then, assign
to pregnant girls the GPA of their boyfriends. Whatever happens to the
“pregnancy” variable after adding the GPA of boyfriends to the model,
is due to the psychological/social stress shared by girlfriend-boyfriend
couples. Yet, boyfriends are not carrying a fetus inside their bodies,
thus their GPA cannot produce GPA-declines in their girlfriend GPAs
related to their biological pregnancy process.
• Ex 2: Imagine some students in a CC are exposed to a program to
improve calculus grades, and some others to improve singing skills.
Belonging to the singing program, or singing grades, should not
predict calculus grades. If they do, then changes in calculus grades
are not the effect of the program, but of factors shared by students in
Comparative scenarios
Falsifiability test

✤ Negative outcome:
• One dependent variable that should not be affected by the
policy/program or the independent variable of interest under study.
• Ex 1: Change the DV for singing grades (in the past example)—
using belonging or not to the calculus program as your IV of
interest.
• Ex 2: A discrimination policy against immigrants should not have
effects on their dental health (but it should on their mental
health). Use dental health as DV.
• Ex 3: Pregnancy rates in schools should not predict dropout rates
in only-boys high schools or Catholic seminaries (where only
males attend). Use dropout rates of only-boys institutions as DV.
• Ex 4: Pregnancy rates should not predict the GPA of same-grade
girls in comparable schools with low pregnancy rates (use these
schools’ GPA as DV).
• Ex 5: CC’s retention rate under a retention-rate program should
 Falsification (or falsifiability) test example

 The Political Realignment of Health

Context: The U.S. two-party system was transformed in the 1960s, when the Democratic Party
abandoned its Jim Crow protectionism to incorporate the policy agenda fostered by the Civil Rights
Movement and the Republican Party redirected its policy platform from a more liberal on civil rights
issues toward economic and social conservatism. We argue that the policy agendas that the parties
promote through presidents have been critical to codify a racially patterned access to resources and
power that is detrimental to the health of all.

Methods: We apply time series analysis methods to data from the National Center for Health Statistics,
1915-2017, and hypothesize that fluctuations in overall and race-specific infant (IMR) and maternal
(MMR) mortality rates shift between the parties in power before and after the Political Realignment of
the 1960s.

Findings: We find that, net of trend, overall infant, neonatal, and postneonatal mortality rates, white
and black IMR, and overall and race-specific MMR were higher during Democratic administrations
compared to Republican ones before the Political Realignment. This pattern reversed after the Political
Realignment, with Republican administrations underperforming Democratic ones.

Conclusions: The political parties and their diverging agendas have been instrumental for devising an
institutional racism that permeates the social determinants of health, affecting health outcomes for all
races in the U.S. during the past century.

Source: The Political Realignment of Health: How Partisan Power Shaped Infant and Maternal Health in
the U.S., 1915-2017.
 Falsification (or falsifiability) test example

Source: The Political Realignment of Health: How Partisan Power Shaped Infant and Maternal Health in
the U.S., 1915-2017.
 Falsification (or falsifiability) test example
Figure 1. Predicted values of overall and race-specific infant-related mortality rates by
presidential party before and after the Political Realignment, 1915-2017
Ln IMR Ln NMR Ln PMR
Predicted Values

Pre-PR Post-PR Pre-PR Post-PR Pre-PR Post-PR

Ln WIMR Ln BIMR
Predicted Values

Pre-PR Post-PR Pre-PR Post-PR

Democratic Presidents Republican Presidents

Note: The 95% CIs of predicted values are from standard errors calculated using the delta method.

Source: The Political Realignment of Health: How Partisan Power Shaped Infant and Maternal Health in
the U.S., 1915-2017.
 Falsification (or falsifiability) test example

Figure 2. Predicted values of overall and race-specific maternal mortality rates by presidential party
before and after the Political Realignment, 1915-2007
Ln MMR Ln WMMR Ln BMMR
Predicted Values

Pre-PR Post-PR Pre-PR Post-PR Pre-PR Post-PR

Democratic Presidents Republican Presidents

Note: The 95% CIs of predicted values are from standard errors calculated using the delta method.

Source: The Political Realignment of Health: How Partisan Power Shaped Infant and Maternal Health in
the U.S., 1915-2017.
 Falsification (or falsifiability) test example

Falsification tests:

“We also carry out a set of falsification tests, contingent on data availability. If the
detected effects are unrelated to the short-term ideological direction of policy
injected by the parties and executed by their presidents, but artifact of
unobserved factors or cumulative aspects of health policy and governance that
correlate with the president’s party before and after the PR, then we would
expect to observe analogous patterns described in Hypothesis 1 on (a)
cardiovascular disease (CVD) death rates, (b) heart disease death rates, and (c)
cancer death rates—conditions known to also significantly affect the working class
but to be driven by cumulative high-effort coping and stress in response to
enduring hardship (Geronimus et al. 2019).”

Source: The Political Realignment of Health: How Partisan Power Shaped Infant and Maternal Health in
the U.S., 1915-2017.
 Falsification (or falsifiability) test example
Figure 5A. Predicted values for heart disease and cardiovascular disease death rates, and
gender-specific cancer death rates by presidential party before and after the Political Realignment
Ln heart disease Ln cardiovascular disease

Predicted Values

Pre-PR Post-PR Pre-PR Post-PR

Ln cancer (males) Ln cancer (females)

Predicted Values

Pre-PR Post-PR Pre-PR Post-PR

Democratic Presidents Republican Presidents

Note: The 95% CIs of predicted values are from standard errors calculated using the delta method.

Source: The Political Realignment of Health: How Partisan Power Shaped Infant and Maternal Health in
the U.S., 1915-2017.
 Group activity!

✣ Meet with your project team members and, for your project,
propose at least two different falsifiability strategies (even if
hypothetical) from the following options:
✤ Placebo test
✤ Negative comparison
✤ Negative treatment
✤ Negative outcome
• Please:
• Define the variable and group at the center of the falsifiability
scenario.
• Explain why/how is it useful?
• Describe the falsifiability scenario (see examples in past
slides).
 Midterm exam

✣ We will post an announcement with instructions.

✣ You will be able to download the exam from Canvas at 12:20 PM
and will be able to upload it to Canvas until 3:10 PM.
✣ We will hold a 15-minute Q&A session at 2:00 PM. Just get connected via
Zoom using the class link. Bring your questions!
✣ The midterm is a book-open exam.
✣ The exam will be comprised of a conceptual and an analytic component.