Econometrics - Difference in Differences

hello in this video we're going to be talking about difference in differences which is probably the most widely applied research design in econometrics at least right now and so what difference and difference is is we're continuing in the theme of trying to control for stuff that we can't actually measure which is a common problem in the social sciences now difference in differences doesn't apply everywhere but it does apply in particular situations where the treatment that we are trying to get the effect of is something that was not and then was so something that went into effect at a particular time so there was a before and an after time before the policy went into effect and after the policy went into effect so uh let's take as just an example so a number of years back in the united states we passed obamacare right or the america the affordable care act and this did a number of things uh and of course there was a before and after now in addition to all the other things that the affordable care act did one of them was the medicaid expansion which expanded the amount of funds available to states to support their medicaid medical programs and this was something that was taken up by some states and not taken up by other states so let's say that we're interested in trying to find the effect of this medicaid expansion on some sort of outcome maybe health outcomes or or mortality or something else and so what's a natural thing that we might try to do the first thing we might try to do is something called an event study now what an event study is is that you just take a place where a policy was enacted and you look before the policy and you look after the policy and you see what changed and statistically there's a couple more fancy things that you can do but that's the basic idea of it you're comparing before and after uh for a policy that went into effect uh now this can be a problem right we might have a problem with with believing that this particular approach identifies the effect that we are interested in at least in certain circumstances like certainly in the case of the medicaid expansion uh when the expansion went into effect uh plenty of other stuff changed too right the economy was changing in terms of how healthy it was and you know the population was aging and lots of other things were changing at the same time so we have a back door between treatment and outcome and that back door is all the sorts of stuff that we can just sort of lump together and call time things were changing over time at the same time that the policy went from not being in effect to being in effect so we can't just compare before and after for a lot of policies because well we're going to mix up the effect of the policy with the effect of everything else that was changing at the same time so this is what our diagram might look like right we have a treatment that affects an outcome and we have a back door through time we can get from treatment to outcome in two ways we can go straight from treatment to outcome that's the arrow that we're interested in or we can walk along the back door path from treatment to time to outcome which is an alternate explanation of why we might see an effect so that's a problem so what can we do well one thing we might want to do is control for time like we can just add time as a variable in our regression and we're good to go except that we can't we can't control for time here because the policy used to not be in effect and then it was and so all of the variation in the treatment can be explained by time and so if we control for time we are going to control away the treatment there's no treatment left after we control for time and so we can't possibly control for time and still get the effect of treatment so what are we going to do well what we have to do is we need to add another group that was not treated okay so we've been talking about a policy that was not in effect and then was an effect but let's go a step further let's say a policy that was not in effect and then was an effect for some people but not for others and then what we're going to do is that we're going to be able to compare the change before and after just like the event study i mentioned where we compare before and after but we're going to compare it differently for the treated group the group that actually ends up getting the policy or the treatment against that other group that never got the policy now remember i mentioned that the medicaid expansion some states took it other states didn't and so we have a policy that didn't exist and then it did and only for some state so we can say okay we're going to take these states that got the policy we're going to compare them before and after and then we're going to take the other states that never got the policy and they never got the policy nothing changed for them now if that happens this becomes what our diagram looks like now we have added a new part to it we've added group so now to get the treatment it's not just the effect of time right which was important right because that was our problem before the fact that time was the only thing that determined treatment that was the thing that got us a problem we couldn't control for time but now treatment is determined by two things it's determined by time and group it's not just going from before to after that gives you the policy it's going from before to after and being in the group that actually got treated which means that now we can control for time we can control for time because we have some variation in treatment that is not just driven by time it's also during my group now of course this introduces a new problem that we have another backdoor through group the states that got the medicaid expansion might just be different in some fundamental way than the states that didn't but we can control for group now we can control for group because we have variation within each group right so that's what we're gonna do we are going to bring in a group that never got treated so we can compare the group that did get treated how they changed against the group that didn't this is a use of within variation like we talked about in the last video right what's that event study approach doing it's using within variation it's saying here's the people before here's those same people after we're comparing before and after that is a within variation we're comparing within the same group of people but now we're doing a second within comparison of the group that never got treated how did that group that never got treated happened before how are they behaving after and we compare their before to after that is also within variation and what we're basically saying is this that if the treatment had never occurred the amount of within variation would have been the same for both groups okay so let's say in the medicaid expansion let's say for example the medicaid expansion raised lowered cancer rates by two percent okay let's say that that's what it did i made that number up entirely so if we saw that it it lowered cancer rates by two percent in the group that got medicaid expansion but it lo but over that same time period cancer rates dropped by one percent in the group that never got the medicaid expansion well that would tell us something it would say okay well yeah great you dropped by two percent but probably not all of that was the medicaid expansion some of it was probably just that things were changing over time and so we could say well how much were things just changing over time that's our backdoor that's our alternate explanation well i can say well okay among this group that never got treated they dropped by one percent so probably one percent of the effect was just time we're gonna take that out we're gonna take that two percent we're going to take out the one percent and leave ourselves one percent and then our effect would be one percent that's how difference and differences works here's how it works graphically so imagine that we have a pill we take some twins uh and we give them a pit one of them a pill that's supposed to make them grow taller okay uh now we measure them before they took the pill and then we come back a year or two later and we see how tall they are after the pill now there's an obvious problem here in that if we're giving this pill to kids kids just grown taller naturally over time so if we just took the kid that we gave the pill to adelaide and we looked we measured her height before and we measured her right after well of course she grew taller she's a kid she grows taller you know we don't need a pill to make her grow taller but maybe the pillar made her grow taller than she would have grown without the pill and for that we can compare her to her twin bella so if bella was let's say four inches shorter before and then afterwards she's seven inches shorter well the gap between them has grown by three inches telling us that maybe that pill made adelaide three inches taller okay that's the idea we have a difference before we have a difference after and then we take the difference in the differences to get our difference in difference effect that's the idea so we're basically seeing how much more effect was there among the treated group than among the non-treated group right the effect the change from before to after the non-treated group that's the effect of time the effect on the treated group is the effect of both time and also the treatment and so if we subtract out the time effect that we found in the untreated group what we're left with is just the treatment effect that's the idea so we see what the before difference was we subtract that out and then whatever's left over the additional increase that is our difference in difference effect how can we actually enact this well it turns out to be pretty straightforward to do in ordinary least squares because all we're doing is think back to our diagram we're just controlling for time and we're controlling for group that's all we got to do right and then we've identified the effect because those are our only two back doors and so all we got to do is we have to control for being after that's our time control are you before the treatment or after the treatment and we control for are you in the treated group or not that's our create our control for group right here's basically a fixed effect for time and a fixed effect for group and then here we have a variable that indicates whether you are currently being treated or not which only occurs for the people who are in the treated group after the treatment goes into place so we can multiply together after times treatment to get a treated indicator if you have you you can have more than two time periods and or more than two groups uh and in which case you just make this be are they are you currently being treated or not that's the variable and then this would give us our difference in difference estimate uh specifically this beta 3 will be our difference in difference estimate that is the effect of treatment because this variable indicates that you are actually currently being treated what's the effect of actually currently being treated controlling for the time period and controlling for the group and that that gives us our effect let's see a quick example here's an example uh using the earned income tax credit their income tax credit is a welfare program uh and you and so if we there was a change there was a change in how generous it was in 1994 uh and specifically this this thing made things more generous for working mothers as opposed to working non-mothers and so the working mothers saw this change in their treatment uh the working non-mothers did not see a change and so we can compare how both of them changed over time to see what is the effect of this extra generosity the first thing we can do is that comparison of means just like we did with adelaide and bella and so we can get the different means by whether it's after or before and whether you're treated or not and then we just compare the means so this is the after versus before comparison for the working mothers here is the after versus before comparison for the working non-mothers and then we subtract one from the other to get how much more change was there for the working mothers than for the working non-mothers and that gives us our difference in difference effect and we get an effect of 0.047 it increased the uh labor participation rate for uh the working mothers by 4.7 percentage points now let's run the ols version uh where we have we regress whether you're working or not on the interaction between being after and treated so after times treated will include after by itself treated by itself and then the interaction term which tells you that they are currently being treated uh so what do we have here so first of all this is saying that the uh the coefficient on after is negative 0.002 which is not very big telling us that there was not really that much of a change for the non-working mothers right so this is uh for treated equals zero so we're talking about working non-mothers uh there's no effect there which tells us the working non-mothers didn't really see a change from before to after the generosity increase here we have a negative 0.129 on the coefficient for treated so that's it after is equal to zero so what is the difference between the working mothers and the working non-mothers before this is telling us that the working mothers worked a lot less before uh the increased generosity then we have the interaction term of 0.047 that is the difference in the uh before and after change how much bigger that before and after change is for the working mothers and for the working non-mothers and so if we think that the before to after change was 4.7 percentage points bigger for the working mothers that tells us that the effect of the increased generosity which did not affect the working non-mothers but did affect the working mothers was 4.7 percentage points now there is one very important thing to keep in mind uh when we are doing this and this is the parallel trends assumption so for this all to work we need to make the assumption that the only thing changing about the difference between the treated and untreated groups at that period of time is the treatment itself so there's plenty of things changing over time that was the whole point of doing this in the first place right we know that things change over time and it's okay for difference in differences that things change over time what is not okay is if the gap between the two would have changed by itself over time so let's go back to the medicaid example so if it just so happened that the states that chose not to expand medicaid already had declining cancer rates and the group that did expand medicaid had increasing cancer rates right that would tell us that even if we had never done a medicaid expansion there would have been a change right from before to after that the before to after difference for this group would have been negative and the before after difference for this group would have been positive okay and so that would tell us that there's something happening here even though no treatment was put into place so we need to assume that the before and after difference for both the treated and untreated groups would have been the same that the within variation would have been the same uh for both groups that's parallel trends right the before to after trend would have been the same now we can't actually see this we can't test this because we don't know what would have happened to the treated group if they hadn't gotten treated there are some ways in which we can sort of you know convince ourselves that maybe we believe it so first of all we can just ask was there anything going on at the time right was there anything going on uh in indifferently in the states that expanded medicaid versus the states that didn't at the time that medicaid was expanded we can think about that you want to think deeply okay well you know what was going on at the time what was in the news what kind of political things were happening is there any reason why those different different groups of states would have pulled together or apart on their cancer rates other than the medicaid expansion itself and if there's anything else that's changing the gap between them at the sa at that time we can't use difference in differences doesn't work one thing we can also do is we can check the prior trends so we can look at whether the two groups were trending together or trending apart before the treatment went into place which would sort of give us a hint right if if they are already trending apart that tells us that hey maybe they would have continued to trend apart and so we have a difference before and after that is not that their treatment is not responsible for or if they're trending together same thing uh and so we want to see in the prior trends before the treatment goes to effect that the gap between them is pretty much the same in that eitc example it looks pretty good right you can see that things are going up and down but before the treatment goes into effect the gap between them stays relatively constant that's that's good evidence in favor of our difference in difference design it does not actually prove parallel trends but if this failed considerably we might be pretty suspicious of it all right that's the basics of difference in difference we are looking at a policy that goes from before not being implemented to after being implemented we have two groups in order to control for time and group we compare a group that did get the treatment against a group that did not get the treatment over those same before and after periods uh that means that we can see how much more the treated group changed than the untreated group the amount of the untreated group change that's the effect of time we subtract that out from the tree the treated groups change so that we just are left with the treatment effect itself we can implement this in ordinarily squares by putting in a set of fixed effects for the groups as well as fixed effects for the time period and then putting an indicator for you currently being treated the coefficient on currently being treated will then be our difference in difference effect for all this to work we need parallel trends to hold which means that the only thing changing the gap from before and after is the treatment itself and we there are some ways in which we can sort of get a gut check on whether that makes makes sense or not which we should do before using difference in difference all right that's it thank you

Channel: Nick Huntington-Klein

Views: 3,016

Rating: 5 out of 5

Keywords:

Id: 4M9Wzko84aE

Length: 16min 29sec (989 seconds)

Published: Tue Aug 25 2020