Difference in Differences and Matching techniques for Impact Evaluation

hey guys I hope you're well so we're gonna continue our march down this impact evaluation highway where we're trying to think about how can I legitimate lien validly pull or disentangle causality from some sort of program or treatment that I put in place for individuals in a country and we started remember with randomized control trials where it's much like an FDA drug test trial where we put in a treatment group and a control group and we give the treatment group the the policy or the procedure of the treatment from the government control group doesn't get it and then we compare the average the average value of the variable of interest okay that's kind of the gold standard that's kind of the best and then we switch to hey what happens if I don't have all the nests needed elements for a randomized control trial what can I do and then we talked about regression discontinuity so don't forget regression discontinuity is where there's some population and they're only eligible for the treatment at us after a certain cut point and so right around the cut point we can calculate the average value of the outcome variable of interest on one side other people who got it and the other side didn't get it we get this thing called the local average treatment effect and it's kind of like an RCT right on either side of the cut line all right now we're going to move to two more different types of procedures policies to get some causal impact estimates and one of them is called differences in difference in differences and the other is called matching so let's go first into difference in differences okay you're like that's your name differences and differences in differences and differences what on earth does that mean well let's let's jump in and I'll tell you what it means okay so let's review the before/after estimate this is really terrible remember you look at the outcome variable Y for somebody before they got it and then and that after the outcome variable Y after they got the treatment and then you say hey look any growth in the Y had to do with the treatment the government policy well that is not right okay this does not give us an unbiased estimate because the before version of ourselves is not about counterfactual for the after version why because there are many other things that may have changed simultaneously with the program so that only observed differences the before and the after you cannot attribute them solely to the program right if we see that somebody who took part in a job training program had a higher salary afterwards we don't know what other things are happening in the macroeconomic context maybe the GDP of the whole country is rising and every is getting richer right so we have these emitted variables okay we have a minute variables and we have what's called ovb and many variables biased so before after estimate no bueno does not work okay now the with/without estimate remember that one when we compare the people without the program to the people with the program that's a good idea right no wrong right remember the people without the program are not valid counterfactuals for the people with the program because for most of these programs the groups sign up voluntarily and so the people who signed up are systematically different maybe they're higher quality maybe they're better informed maybe you know how to bribe government officials I don't know but probably they systematically different from the people who do not have the program right so the differences between the two groups cannot be attributed solely to the program or treatment okay remember is this thing called selection bias they're different people who are selecting into the program or treatment are different than the people who are not selecting in so we have this issue called selection bias all right so remember that but when we put those together actually it's sort of works alright so if we can obtain the average before estimate from a group of people who got the treatment these are with and then compare this estimate to the average before/after estimate of a group who did not get the agreement the treatment we can recover something I say sort of causal and so here's why it works okay remember the problem with the before/after is that other things in the world might have changed in addition to the treatment okay and the problem with the before/after is that you know somebody who have a treatment before and then after other things in the world life might have changed but what if we look at the before and after data for somebody who didn't get the treatment as well and then maybe it's just so possible you can kind of tell a story and it's believable that if the economy is also going up and making this person's salary rise it will also make this person's salary rise so if I look at how much this person's salary rise raised who did not get the treatment and this guy is raised a little tiny bit more this additional difference in differences is actually kind of sort of a causal impact estimate right so if I subtract off the before/after of the withouts from the before/after of the width the emitted variables are kind of cleaned away why do I say they're cleaned away well whatever is hitting the person who had treatment and making their salary higher those same effects are also hitting the person who did not have the job training hit the program and so if I take away what how much theirs theirs their income raised then then I take off all of the other we call them committed variables right and so what I'm left with is something is sort of like a causal impact estimate all right so it cleans off the emitted variables it actually also cleans off the selection bias remember what's the selection bias and that is the problem with the width without estimator right it cleans off the selection bias to talk about more on that later so here's how I calculate it so alpha remember is the variable for our causal impact estimate and here's what I do I take Y remember Y is the thing we're interested in I take the Y in the time period after the treatment gets in place - the before so this is the before after and I do that for the widths and then I do the difference that before after for the without so now you're like hating without why do they have a before and after they never got the treatment yeah that's true but just in the same time period right if the people who did get the treatment they got it in too some 10 and we have data on 2005 hearing the same thing 2010 - 2005 for the withouts okay so we have a before after and a before after and then I have a width without estimate look it all mixes in together there's two differences I DIF this guy first and then I difference here that's the double differencing that's what we call it difference in differences I mean the difference here between these two differences right so the difference don't forget is the answer to a subtraction problem okay so it looks graphically kind of like this this is the way I like to think about it okay forget about this upper line this is a picot comparison group for right now there's a treatment group right and your zero there outcome of interest at Point a it's gonna be 0.6 right and then a year later we we measure them and we're like hey point seven four it went up right okay so hey our outcome variable of interest Y went up 14 100's and this if we just did that that would be a before after but the problem is right that that point 1 4 between point 6 and point seven for the plan form also includes the effect of emitted variables so during from years 0 to year 1 a lot of things happen they got the treatment but also I don't know macroeconomic conditions change corona virus pandemic slept through the town I don't really know what happened but probably a lot of other things happen during that time so what I also need to do is say look how did everybody else change and if you look here's another comparison group these are the withouts and i'm measure their Y before and after and I see that everybody else went up what is that 0.3 let's see that sorry 0.03 or 3/100 so these people went at 14 the people who got the treatment hundreds these people went up three hundreds and so let's go ahead and subtract off the three hundreds from the from the 14 okay so I have 14 minus the 300 and I get 1100s point 1 1 so the point 1 1 is cleaned off of all the other emitted variables why do I say it's clean off because whatever happened to the treatment people these mitad variables between Year Zero and near one also happen to the comparison group so if I say hey they would have moved point zero three anyway but they moved to point one four then I know point eleven is the amount of the change that's causally attributable directly to the treatment and only the treatment nothing else all right so 0.1 one is our estimate there okay so notice that the wind outs are not providing an estimate of the counterfactual for the whist because that would be the width without what we're doing is we are calculating the change in the outcome for the without is providing an estimate of the counterfactual for the change in the outcome of the whist all right so in the differences in differences okay and thus the comparison and the treatment groups can be somewhat different actually and if you know let me go back a slide real quick you'll see the comparison group they started off better off it was the poor or not quite as healthy or whatever his outcome is that we're measuring why they started off not not so great but what we're looking at it's not that they're the comparison group is being a counterfactual for the treatment we're looking at the change the 0.3 and we're comparing it to the change the 0.14 for the other group okay so it's the the change that is being the counterfactual for the change for the outline all right and then another benefit is that we clean off or get rid of the selection bias - well how is that well remember what's the selection bias it says well that some people are just I don't know more in the know higher quality they have better access to resources and they're always gonna have a better outcome well notice that I'm subtracting the before - to the after - before for the widths and I am essentially subtracting off any sort of selection bias they have right for whatever reason that makes this group you that got the treatment different than the comparison since I'm doing a difference between the two right I'm not saying hey these people are point six and point seven four no I'm saying the difference is 0.14 1400s then I'm actually subtracting off the selection bias because both of those though why before and the wife they're both the people with the higher or the lower what or different qualities okay and so the both the why after and the why before have the selection biases we subtract them off and we get rid of selection bias student right so we get rid of the OPP we'll get rid of the selection bias and actually theoretically this is sort of kind of causal alright notice however that the survey data required for this technique is a lot right here's what you need you need both before and after measurements for wimps and without and it's often easy to get survey data for Wis because other people who participated in the program maybe the government has a job training program right the government almost always has data on people how they were before they start in the program and after they started the program oh you know what they don't have is data on just random other people before the program started and after the program started but you need this because you need to track how they would have changed in the absence of such treatment okay and so there's a lot of survey requirement notice that in our Jeanne we talked about this last video you actually don't technically need a baseline or before measurement at all okay it's nice to have it to make sure that in fact the the people on either side of the cut point are in fact good analogues for each other or counterfactuals for each other but you don't in theory mean it you just need to look at the after within the RCT also you don't technically need a follow-up or after survey there either remember the only need for a baseline or before survey is to see if we trusted the randomization the randomization went properly right we compared the average age of the two groups before they got the treatment the average with a proportion of gender male or female is in there the average education etc just to make sure randomization that actually happened okay so you need a whole lot more data like four times more data because you need before and after four wits and without okay and so another assumption that it makes is that everyone responds to the same to the treatment or to the policy whatever it is they they we need to assume that both control and treatment groups respond the same to the treatment and that's kind of like a sketchy assumption sometimes right we call it equal trends assumption so you have to have equal trends for the widths and without okay in other words in order for the differences and differences technique to work we must assume that the emitted variables affect the treatment and the control groups equally but there's no way to absolutely check for this at all so you just kind of have to know the the conditions and the people whom you aren't interviewing and be able to verify that this this is true let me give you a graphical example of why I mean when the equal trends violation is violated okay so here was our original picture and we see that in the comparison group they went from point seven eight two point eight one and we said hey this is zero point zero three that's kind of like our counterfactual that's what everyone would have gone up in in absence of any treatment but let's just say that the true counterfactual so this is the the treatment group right so we assumed that they would go from here up to up there because it matched this one you see that it matches this guy this is the 0.03 increase but what if in true they would have gone all the way up to here right that are true counterfactual maybe they want to bun up and said blowing up 0.03 maybe they would about 0.06 and so then the impact is only this amount right here which is much less than 0.1 one right so when the world is represented by this graph you see that the impact estimate is biased because the change here is not the same as the change in the treatment group it is not a valid comparison for the change in the winds okay and so how can we check for equal trends well there's no real way to verify it but here there's a couple of different ways and we'll check we'll talk about this right now the first way that we can check is we can monitor the changes in the outcomes in both groups at two points before the treatment is applied and both of these would both occur during the baseline right and so if they're moving together before the treatment is likely they would have continued together if treatment hadn't occurred so what do I mean by that so come over here whoops and let's say that you surveyed them again so you can see the amount of survey that you're going to need for this so you survey both groups here and then you come back a year later and you survey both groups here and you see if the difference from here to here the difference from here to here in a time when nobody got any sort of treatment at all this is all on the baseline that before if they're moving the same and you see here in this example they're not actually moving quite the same so you might say oh there's equal trends if they're not moving the same amount over here back in the beforetime land then in the aftertime land they're probably not moving at the same amount either if on the other hand they're tracking together and you're like oh these two groups are looking the same before time you know for a while and it's probably true in the after time they would have tracked the same also okay but once again this is a ridiculous amount of surveying you have to do here you have to survey these people at another point in time or maybe you can two points in time so this is four more surveys right one two three or one more surveys before the program even starts so it's unlikely that anyone's really going to be interested in paying that much money to survey that many people all right here's another way you can do it you can run quote placebo test with another fake treatment group that didn't actually get the treatment and so what you do is if you can find another I'm gonna go back to site you can find another group of people okay and then you can do the same sort of a test you can do the before and the after on some other group of people something like this okay and why is that gonna work what you want to make sure that there's zero impact between the control and a fake treatment group so if you want to if you just can get some third group of people again even more surveying and if you can see that this random third group of people moves the same as this comparison group here and that they both have 0.03 right and so over after you subtract them from each other it's going to be zero if there's zero difference between the comparison group and the fake treatment group here then you're like you know what any group by would get fined including this treatment group is going to be this same counterfactual is that compare same group so it's another kind of way to think about it and whether or not that's a legitimate sort of test is up to you some people use it and especially you really have to know the the people in the situation of the country that you are serving in to realize if that it's gonna make sense or not know that each requires even more surveying right so you need two additional baseline surveys for technique number one and a whole other fake treatment group for technique number two right and you have to survey the trick with a treatment group at both baseline and follow-up right so let me go back to that graph again you need either two more surveys here or you need either four more surveys here in order to test for the equal trends assumption so the equal trend assumption is a kind of killer if you can to verify that that's true in the difference-in-differences approach all right so let me give you an example so this comes from a very famous economics journal the QJ eat a quarterly Journal of economics and they were wondering how does labor supply change in response to the Earned Income Tax Credit so labor supply is why basically what I mean by oversupply if people are working or not if they have jobs or not and then we give people changing it Earned Income Tax Credit so the Earned Income Tax Credit if you don't remember is a tax credit implemented by the United States and basically it's kind of a revision to the welfare system and poor families get it but it's the opposite of welfare usually you you get welfare when you don't have a job the Earned Income Tax Credit actually says hey if you make a thousand dollars we're going to give you an extra five hundred if you make two thousand we're going to give me an extra thousand dollars so the idea here is that it in said it's the opposite of welfare it actually incentivizes people to work and then of course once you get above forty fifty thousand dollars in the Earned Income Tax Credit goes away all right and so here we have the some data before the treatment and after the treatment so the legislation changed and it granted that Earned Income Tax Credit now if we look at the treatment group we're going to use the treatment group that unmarried women with children they're eligible for the EITC because you have to have children also to get the Earned Income Tax Credit so if you look at unmarried women with children hey look before the Earned Income Tax Credit seventy two point nine percent of them work after the income tax the Earned Income Tax Credit seventy five point three percent of them work that's a change of 2.4% an increase in two point four percent Hey two point four percent more people worked that must be a causal impact estimate right no remember that's just a before after estimate we don't know what else is happening in the economy maybe the economy is getting better and more jobs are being offered so more people work so what you have to do is you have to create a counterfactual change and so it's kind of nice we can look for a counterfactual we can look at people Gordon didn't get the EITC but basically are the exact same so unmarried women without children that's what the authors thought we're good counterfactual so you don't have children you don't get the EITC but you're still living through the exact same time periods as the people who are getting the EITC so if anything is changing in the world then it's going to change in this group too and if you look they were to ninety-five point two percent before and ninety five point two percent afterwards actually there's no change look at that the difference was zero so the differences in differences is two point four minus zero is two point four so we actually can say now that this is a causal impact estimate the increase in two point four percent of people working that's their labor supply why is and causally attributed to the Earned Income Tax Credit because we check to make sure the trend did not change for other people okay let me show you another example this is kind of a cool technique as you can probably see so here we are asking about labor supply once again is ry ok this is in the jpe the journal of political economy another famous economics journal and we want to know what's changing or the treatment is a disability insurance benefits and so this is from Canada we've actually looked at the Quebec pension program but fuller you know our our our deep topic but the Quebec pension program is the is the kind of the controller the control group the Canada Pension program is the treatment group so the people in the Canada Pension program as you can see so this top half right here is not the differences in differences this is just to show you that something changed okay so Canada pension program you see they were getting fifty one hundred dollars a month and they then they went up to seven thousand about dollars money so that's an increase of twenty six hundred dollars and we want to say hey when their disability insurance benefits go up what happens to their labor supply and so then now we're going to say hey the control group they went up a little bit but they went up nine seventy six there's a 1666 difference in differences this is not the causal impact estimate this is just to show you how in the Canada Pension program they their disability insurance benefit did go up now this part down here is in fact the impact estimate so let's look at percentage of men who were not employed and so in the treatment group the percentage of men not employed went from 20% to 21.7% and they went up 1.7 percent so we can say oh oh when you get twenty-six hundred more dollars right here and 1.7 percent more people are unemployed now this makes sense right if you get paid more money for being unemployed you're probably gonna stay unemployed more right so the unemployment is going on but this just looking at this alone would be a before/after estimate we don't know what would have happened in the world where they didn't get an increase so what we do is we say hey what happened to the Quebec pension program people they they got a little more money too actually but what happened to their labor supply the amount of unemployed people went down actually from 25 to 24 it went down the whole one percent so you know what in the absence of these people in the Canada Pension program getting more money we might actually assume that they would have gone down one percent instead they went up one point seven percent so this is actually gonna we need to make this bigger than 1.7 percent if I got one point seven minus a negative one that's one point seven plus one we are actually going to get two point seven percent and so you see the causal impact estimate of the Canada Pension program increase is that an increased unemployment by two point seven percent one point seven percent all by itself but otherwise we would have assumed that it actually would have gone down one percent okay and so you see that the causal impact estimate actually is bigger and the before/after estimate is clearly biased so this is something that's a little bit more believable and of course it got published in and did journal so usually a couple people look through it and make sure that it's a legitimate study all right so I want you to kind of think about how to do this yourself perhaps and I'll give you a little example I'll let you pause it here in for a second after I explain how to do this but this is a hypothetical so let's suppose that we want to know the average hours of worked week in a big-box store so we're talking about like I don't know Best Buy those the Walmart big-box stores after North Dakota increased its minimum wage so we know that probably if you increase your minimum wage what do we think maybe people work more yeah it's probably work more it's worth more now okay so let's see what what people did so in North Dakota this is going to be the control group because North Dakota is the one that increases minimum wage we can't just do a four and after we have a decrease of what is that about fourteen or something like that so people are working less'n North Dakota before and after but this is just the before after comparison we have to say what would they have done if we didn't change the law so come over here we can use South Dakota as a counterfactual so you see in South Dakota the hours went down two hours and nothing happened there so we would assume that North Dakota would have gone to down two hours but they went down 14 hours total so you take the difference in the differences find the differences here and then subtract them from each other and figure out what the difference is in differences estimator is okay so hopefully you kind of looked at that and you got a something that's around 12 now if you have other questions you can check out these two idea videos I'll post them in the comments below give you a little more introduction Doug McKee's actually he's pretty pretty smart guy I actually met him he came in University last semester i sat down with him and talk to him but yeah look at either of these I'll post them also in the comments now we're gonna move on to our next type of non randomized controlled trial way of disentangling causal impact estimates and it's called matching that's right the same or gift different game that my daughter's love to play but for adults so how does this work so recall that the key to finding a good estimate of the counterfactual is trying to find someone who is nearly identical to this other person but just didn't get the treatment right we need a clone okay well is there a way to find a clone maybe if you have enough data you can look at let's see here okay look at let's look at this data so I have data on people who got the treatment I have data on people who did not get the treatment and look at this it just so happens that here's somebody's 19 genders one that's usually one for males zero for female unemployed months three secondary diploma no secondary diploma two can graduate from secondary school so look look at this there's a person who's exactly the same almost on these four dimensions do you see that so based on these four dimensions we were able to find a clone for him also for her also for him right if you have three months but most of these other people don't have clones but we were able to find three fronts okay so we were able to find three clones we could look at the average the difference in the average treatment effect between the treated and the untreated okay now to think about this in order to get an even better clone you want to match on even more dimensions you want them to be not only be the same age and gender and be unemployed for the same amount of time and not have a secondary education but also maybe be the same ethnicity and live in the same town you want all kinds of similarities in order to find a perfect clone but think about this what's the likely the more restrictions or the more dimensions of matching that you put the likelihood of finding a perfect clone is decreasing it's very unlikely that you're gonna be able to find any matches if your dimension is four or five things that you're trying to match on this is known as the curse of dimensionality right so the more exactly try to make our clone the less likely they are to actually find a match okay so luckily statisticians have found a solution to this curse of dimensionality and what you do is you use all of the dimensions that you have all of the data you have you you use software to estimate the probability that some individual signs up for a program and this is known as their propensity score why propensity well propensity means like kind of like the probability and the likelihood of so their propensity to sign up for or enroll for the treatment okay so you can its regression analysis we're going to get into exactly how you do it but I want you to be aware of it because you will see some matching algorithms okay and you know what's interesting is surprising we can compare any two people with the same or nearly the same propensity score and it turns out that they're valid counterfactuals for the month for the other person all right this last point is proven with some complicated math we'll leave that out until you get to grad school but take a ticket my word for it if you can find out use some all the data on someone to find out their probability of being in the treatment group and you compare them to someone else with the problem same probability of being in the treatment group but is not in the treatment group then you can actually use them those people for counterfactual for each other right so matching tries to create a randomized control group but the control group is a fake constructed group alright and so it these people who are in the fake control group are people with the same propensity score as somebody who was in the treatment group but just they didn't get the treatment all right and so some practical considerations is the more dimensions meaning the more data the more information on each individual we have the better the matching process and so you need really detailed data for that it also requires a baseline survey for matching as well now why do you need a baseline survey it's because we must ensure that the outcome y is the same pretreatment for both the treated and the fake control group right if this really is the summary of the treated and the fake control group they got to be you know equal before so you need to check for appropriate matching so you need to come up with a baseline survey all right there's matching where you're doing actual matching of people then there's another one called synthetic matching and I bring this up only because there have been a couple of really good popular journal articles lately on synthetic matching and so here's an interesting one that I think is is it should be in the news right now because basically they're looking at how a universal basic income affects people's labor response right and then and the run-up to the presidential election we've been hearing a lot about universal basic income and that it giving everybody cash diminishes their desire to work okay and so they looked at Alaska and I don't know if you get Alaska big this thing called the Alaska Permanent Fund it's kind of like a universal basic income that everybody in Alaska gets they get the profits from drilling for oil in Antarctica excuse me in Alaska gets distributed to all of the people in Alaska right so it's kind of like a universal basic income and so let's compare the amount that people in Alaska work who are getting me Alaska Permanent Fund to the people who live in Alaska but don't get the Alaska and it fun well there is no such place so what they did is they created a synthetic Alaska or they got a bunch of different states and they put them together in different proportions so that it exactly matched what Alaska looks like so it matches them exactly in the same sort of age average age average education a gender composition ethnicity race all that sort of stuff they got a much different other states and so then they asked how much the does a labor force participation rate which is how much you work the difference between Alaska and this synthetic Alaska and we want to ask how much labor force participation decreased in response to an income increase right so we call intermediate micro theory right when your labor supply changes when your income change so if you get universal basic income how does it change your labor supply so we have actual Alaska that I'm getting this synthetic excuse me that are getting the universal basic income from a Permanent Fund we have synthetic Alaska over here then it's not getting it because it's composed of a bunch of different states then don't get the Alaska Permanent Fund and just a little um kind of summary of it they found no decrease in an employment in the non-tradable sector and a slight decrease in the tradable sector so remember what we talked about tradable sort of are the goods that can be traded across County Village State international borders or something like that all right and so the consensus is that increased in consumption drove employers to demand labor at a higher rate and that's offset the negative income effect okay so we have an income a factor says hey I'm getting free money I don't need to pay I don't need to work but the fact that everybody has more income they're buying more stuff so employers are more desperate to hire people so they're willing to pay a higher wage and so they work more right so actually it's double whammy good for the people right making extra money and they get a higher pay so it's there's a summary which I will also post in the comments it's really interesting it's popular for people who not aware of these techniques so it's accessible to all you should kind of check it out all right so in general ranking the techniques in robust or like how good something is at giving a ballad um consul estimate here's what we have randomized offering is number one randomized promotion is number two these are both types of RCTs remember we talked about these are the two ways to our duties occur in the world why is offering better than a promotion remember to offer something is actually to give it to some people and not give it to other people promotion means you just like encourage some people which does not encourage other people and then we have regression discontinuity right here we have differences and differences and finally match okay and so it's very common actually studies to use a combination of several of these techniques to check for robustness so maybe they were able to do an r d-- but then they also ran a matching to check to make sure they got the same answer and so that's pretty common and it helps you know that the impact estimate is truly valid because you have stuff that's agreeing from more than one study agreeing all right so those are the main types of disentangle and causality and all the ones that we'll study in this class if we're trying to get a causal impact estimate alright thanks guys

Channel: Justin Jarvis

Views: 3,052

Rating: 4.875 out of 5

Keywords: Economics

Id: TArnJZDiL4E

Length: 36min 16sec (2176 seconds)

Published: Wed Mar 25 2020