Impact Evaluation-1:

Difference-in-differences (DiD)
Deepak Singhania
HS 649
17-10-2024
Main reference: World Bank (2016). Impact Evaluation in Practice. Link
Objective
To understand the difference-in-differences method, and application.
Before And After
What is before and after comparison?
- Pre-assessment and post-assessment

Sounds like the

Then I started using new machine
You know what, I this new weight- helped! I’ll buy Well, I also started
used to have reducing machine one. drinking lot of water,
excess weight. from the market. and it was also my
job market year!
Assumptions underlying Before And After
During the time that the intervention took place, the
outcome would not have changed in absence of the
intervention

Before After
intervention intervention
Connected to intervention
and outcome
INTERVENTION
 An example of the Before And After Approach
You started a remedial education program for % of Children who
school-going children in Std 2 to improve their can read grade 1
reading skills of grade 1 level text. level text
53%
The remedial education classes are run after
school hours. 38%
The outcome of interest is the ability to read
grade 1 level text
You know that there are no other remedial
education programs/after-school support
running in village. Before After
Cross-Sectional comparison
Comparing two groups at same time. What is the underlying assumption for
causal identification?

The assumption underlying cross-sectional comparison

Groups being compared are similar, on average, on all factors affecting the
outcome of interest other than the intervention

For example, comparing development in Northeastern

states with other similar states
Cross-Section comparisons example
A district conducts an exam to determine the eligibility to extra
learning support to students (a remedial education program).
The cut off is 50% - All those who receive 50% (or more) are
provided the support while those who received less than 50% are
not.
After 2 months, it conducts another exam to determine impact of
the support
The district compares average scores on the second exam between
those who received the support and those who did not.
Is comparing the difference between average scores of the two groups
a reasonable measure of program effect? If yes, why? If no, why not?
Selection Bias
Selection Bias: Often, groups that get/participate in intervention are
systematically different from groups that do not on factors affecting
the outcome of interest
Are there likely to be any systematic differences between groups that received
program and those that did not on other factors affecting the outcome of interest ?

Children going to private schools are systematically different from

children going to government schools on factors such as household
income, parents’ education.

Counterfactual!
Difference-in-differences
It is a mix of “cross-sectional” and
“before and after” comparison. One of
the most widely used methods.
Includes:
Before After Difference
• cross-sectional comparison MEANS
Treatment Treatment
• before and after comparison
Treated A B A-B
B-A
Control C D C-D
D-C
D-n-D basically compares changes in Difference A-C B-D (A-B)
(B-A) – (C-D)
(D-C)
outcomes over time for one group (B-D) – (A-C)
with changes in over time for the
other group. In other words it This is acting as counterfactual. The only
difference is that treated got treatment
compares the trends. Look at the table.
Difference-in-Difference Approach, with example

Learning Levels
Key Identifying Assumption
Before-after
Impact D-n-D In the absence of the intervention,
Impact the outcome levels would have
Cross-sectional increased by the same amounts in
Treated school treatment and control group.
Control school
Parallel trends assumption: it is
Before After
important to show parallel trends to
prove the impact through this method.
With an untreated group, we are closing both the time and
the cross-sectional back doors.
1. Isolate the within variation for both the treated group and untreated group.
Because we have isolated within variation, we are controlling for group
differences and closing the back door through Group (the “differences’’)

2. Compare the within variation in the treated group to the within variation in
the untreated group. Because the within variation in the untreated group is
affected by time, doing this comparison controls for time differences and
closes the back door through Time (the “difference” in those differences)
Example: Snow 1855
Most famous and oldest example of DiD
Snow 1855
▪ demonstrated to the world that cholera was spread by fecally-contaminated
water and not via the air

▪ Many theory on germ spread. In 19th century “miasma theory” – disease

spread through air coming from rotting material.
▪ Including cholera, a routine outbreak in Europe

▪ But, John Snow believed it was through dirty drinking water

Two dimensions
▪ “before” and “after” periods were 1849 and 1854
▪ Thames river, main source of water:
1. upstream,
2. downstream (contained everything that Londoners dumped in the river, including
plenty of fecal matter from people infected with cholera)
▪ A policy between 1849 and 1854: the Lambeth Company (water treatment) was required by
an Act of Parliament to move their water intake upstream of London.

▪ So, what’s the second dimension?

▪ the Treated group: anyone in an area where the water came from the Lambeth
company, and an Untreated group: anyone in an area without Lambeth
▪ did areas getting water from Lambeth see their Cholera numbers go down from 1849 to 1854
relative to areas getting no water from Lambeth?
So then the question is:
did areas getting water from Lambeth see their Cholera numbers go down
from 1849 to 1854 relative to areas getting no water from Lambeth?

What is the DiD estimate?

A Modern Example
To understand how DiD functions
Kessler and Roth (2014)
▪ studies the rate at which people sign up to be organ donors

▪ In U.S., you opt-in to be organ donor (otherwise, assumed to be opted out

unlike some other countries). One can opt-in while filling up a driving license
form.
▪ An alternative is “active choice”. Neither opt-out nor opt-in is assumed.
▪ In July 2011, the state of California switched from opt-in to active choice.
▪ compare California against the twenty-five states
Organ donation rates in California and
other states
What DiD does?

Take out the 1.4-

percentage-point
reduction from the
untreated group
and we see a DID
effect of -2.2
percentage points
of the active-choice
phrasing on organ
donor rates.
The Key Identifying
Assumption
PARALLEL TRENDS
Parallel trends assumption
if no treatment had occurred, the difference between the treated
group and the untreated group would have stayed the same in
the post-treatment period as it was in the pre-treatment period.

the entire plan behind a difference-in-differences design is to use the change in the
untreated group to represent all non-treatment changes in the treated group

So, you can’t have a control / untreated group that is having it’s own
kind of changes or trend that is specifically different for that group
in mathematical terms…
• The difference between pre-treatment and post-treatment in the treated group is
Effect of Treatment + Other Treated Group Changes

• The difference between pre-treatment and post-treatment in the untreated group is

Other Untreated group changes
• Difference-in-difference subtracts one from the other, giving us Effect of Treatment +
Other Treated Group Changes – Other Untreated Group changes

• So, essentially, for DID to be correctly identify the last two terms in the final
expression about should exactly cancel each other out.
So what to look for?
1.There’s no particular reason to believe the untreated group
would suddenly change around the time of treatment.

2.The treated group and untreated groups are generally

similar in many ways.

3.The treated group and untreated groups had similar

trajectories for the dependent variable before treatment.
How can we check if our untreated
group is appropriate? Prior trends
How can we check if our untreated
group is appropriate? Placebo test
❑Giving a false treatment, before the treatment period.
❑Somewhat similar to the prior trends
Empirical specification

𝑌𝑖𝑡 = ∝ + 𝛽1 𝑃𝑜𝑠𝑡𝑡 + 𝛽2 𝑇𝑟𝑒𝑎𝑡𝑖 + 𝛽3 (𝑃𝑜𝑠𝑡 ∗ 𝑇𝑟𝑒𝑎𝑡)𝑖𝑡

What is the variable of interest?

What does each coefficient capture?
What do the subscripts tell?

Can anyone connect β2Treati to what we have studied in the past?

Linking the Empirical specification to
the D-n-D table shown earlier
𝑌𝑖𝑡 = ∝ + 𝛽1 𝑃𝑜𝑠𝑡𝑡 + 𝛽2 𝑇𝑟𝑒𝑎𝑡𝑖 + 𝛽3 (𝑃𝑜𝑠𝑡 ∗ 𝑇𝑟𝑒𝑎𝑡)𝑖𝑡

Before After Difference

Treatment Treatment
𝛽1 + 𝛽3
Treated A B B-A
Control C D D-C
𝛽1
Difference A-C B-D (B-A) – (D-C)
(B-D) – (A-C)

𝛽2 𝛽3
𝛽2 + 𝛽3
Some key aspects of D-n-D to remember
❑ The effect of a treatment is on the change in outcome and not on the overall level of outcomes.

❑ Showing a parallel trends is a must to argue that you have clearly identified the effect of x on y,
or that your analysis is causal.

❑ You need at least three period of data. Why?

❑ The parallel trends need to be shown only on outcomes.

❑ Baseline differences, in levels, could be a cause of concern, but if you show valid enough
parallel trends then you don’t need to worry about it.

❑ If you have more many period data, you need to be careful about running a regular diff-n-diff
regression. (Synthesis of D-n-D method progression)
Exercise
o T and C are treatment and control groups.

o Assume that the treatment is training for employment.

o In the baseline (i.e. t=0), average unemployment rate is 10% for T, and in the
endline (i.e. t=1) it is 12%
o For the control group these nos. are 8% in the baseline and 10% in endline

o Did the treatment help the treatment group?

o What would be the difference-in-differences estimate?
One of the most popular examples of D-n-D
o New Jersey increased minimum wage from $4.25 an hr to $5.05 an hr after 1992.
o How do the employers respond to this?
▪ Economic predictions are the employment will reduce and it is not a good policy.
o But, empirical evidence is mixed.
o Card and Krueger (1992) used data from restaurants to analyze this.
▪ NJ and Pennsylvania similar in terms of their restaurant industry.
Parallel trends

Card and Krueger (2000)

My paper
Timeline of Decentralization Reforms in Indonesia

Moratorium on splitting and the timing of elections were plausibly exogenous (Burgess
et al. 2011, Bazzi and Gudgeon 2014) and hence provide robust causal estimate
Simpler specification (Effect of
Splitting)
𝑌𝑖𝑑𝑡 = ∝ + 𝛽1 𝑃𝑜𝑠𝑡𝑡 + 𝛽2 𝑆𝑝𝑙𝑖𝑡𝑑 + 𝛽1 (𝑆𝑝𝑙𝑖𝑡 ∗ 𝑃𝑜𝑠𝑡)𝑖𝑑𝑡
Full Specification (in my paper)
D-n-D in Stata
reg elec_hh_pc year jst_splt_dmy splt_post_dmy, cluster(knkab)

In the above code:

year==1 for post treatment, and ==0 for pre-treatment
jst_splt_dmy==1 for districts that were split, otherwise 0
splt_post_dmy is an interaction of the two.
Parallel trends
Some final points
Why regressions, instead of table?
❑ Controls
❑ Multiple time periods
❑ Group-specific time trends
❑ Heterogeneity

Source: Besley and Burgess 2004

A note on the two dimensions
Note that for diff-n-diff to work, you don’t necessarily need the data over time
period. If you can show that the treatment and control group are similar on
average, then the two dimensions can be anything.
With Without Difference
Treated A B A-B
Control C D C-D
Difference A-C B-D (A-B) – (C-D)

For example: Hanushek and Woessmann (2006); tracking/nontracking and

primary/secondary school
Triple difference
Treatment (min. wage) Control
Before After diff Before After Diff
High lockdown A C diff1T A’ C’ diff1C
Low lockdown B D diff2T B’ D’ diff2C
Diff. for Treated diff1T– diff2T Diff. for Control diff1C– diff2C
Difference-in-difference-in-difference = (diff1T– diff2T) – (diff1C– diff2C)

Learn more here.

Another Example: Chetty et al paper

Quadruple difference: Cycling to school, Muralidharan and

Prakash, AEJ, 2017
Potential data-sets that can be used to try
out diff-n-diff kind of exercise in the
Indian context
• Any kind of household/individual survey:
• Multiple rounds of NSSO
• Multiple rounds of NFHS
• Multiple rounds of CPHS
• Multiple rounds of IHDS

• Sub-national data over multiple time periods

Careful with diff-n-diff more than
simple two-period
Synthesis of D-n-D method progression

Simple and interesting read on diff-n-diff with two time periods and
multiple time periods
Next
Impact of MGNREGA using diff-n-diff method.