(humming tone) >>Hi again students. This is our last class topic
and it's one of my favorites. We're going to explore the
method of synthetic control. This topic is not in our textbook, but I've made a separate reading available to go with this video. Many times we have questions
about a single policy. For example, what was the economic cost to West Germany reunification? Or, what is the effect
of California's high cigarette tax on smoking? Or, what is the effect of
Oklahoma's Right to Work law on wages or employment? To answer these kinds of questions we need to know what would have happened without the policy intervention. Synthetic control does this
by creating an artificial version of the policy in
question and comparing actual outcomes to the outcomes predicted by this estimated synthetic control. The technique brings a lot
of credibility and rigor to case studies and is
being widely adopted in finance, economics, and political science. The method seeks to create a control that mimics the behavior of
the outcome being studied in the pretreatment period and also matches important characteristics of the policy under study. We will be more specific about that later. For now, suppose we want
to estimate the effect of Oklahoma's 2001 Right to Work law on state output by a synthetic control. We would first choose a set
of potential control units. In our example, states
without Right to Work laws would be a good choice. Then, we would choose a
set of characteristics that we think effect state output. For example, education
levels, population density, and investment. Then the method uses linear programming to choose a weighted average
of some of the potential control units to create
a synthetic Oklahoma from say 1980 to 2000, the
year before the law passed. That weighted average
is the synthetic control and is chosen both to
track actual Oklahoma state output levels and have a similar profile for education, population
density, and investment. To tell if we have a good control, we can compare how well
it tracks actual Oklahoma and how closely it matches actual Oklahoma on the characteristics we have chosen. In a way, this is like matching except we have two
criteria and we're looking to match over a relatively
long period of time instead of at one
particular point in time. Once we have a good synthetic control, we simply compare the path
of Oklahoma after 2001 with a path of synthetic Oklahoma
generated by our control. Assuming the only
difference between the two is due to the Right to Work
law, our estimated treatment effect of the law is simply the difference in the two post 2001 time paths. This is somewhat similar to a difference in difference approach. So to use this technique,
we need a relatively long pre-treatment period, a set of reasonable potential control units,
a set of characteristics we think are important in
determining the outcome we are studying, and data on the outcome, and characteristics for the treated unit and all the potential control units. In the pre-treatment period, we estimate the synthetic control unit as described above using
only pre-treatment date. Then, since the control
is just weighted average of the other units, we
can use those weights to create our counterfactual outcome in the treatment period. The estimated treatment
effect is the difference between the behavior of the
treated unit and that of it's synthetic control after
the treatment is realized. Let's consider an example
from the literature. In 2015, Abadie, Diamond, and Hainmueller published paper in the American
Journal of Political Science that used synthetic control
to estimate the effect reunification had on West Germany. Their pre-treatment
period was 1960 to 1989. Their potential control
units were Australia, Austria, Belgium, Denmark, France, Greece, Italy, Japan, the
Netherlands, New Zealand, Norway, Portugal, Spain, Switzerland, the United Kingdom, and the United States. The outcome variable was real per-capita gross domestic product. The characteristics they chose to match on were inflation, industry
share of value added, investment, schooling, and
a measure of trade openness. Their estimated synthetic
West Germany was 42% Austria, 16% Japan, 10% Netherlands,
10% Switzerland, and 22% the USA. Here's a graph showing the results. As can be seen, the
pre-treatment fit is very close. The synthetic control
tracks actual West Germany extremely well. After reunification though,
West Germany underperforms the control by about 12%
by the end of the period. Here is the table checking for matching on the economic characteristics. By comparing the first two columns it becomes clear that
the synthetic control matches West Germany quite well on the chosen economic characteristics. So in this paper, we
see a terrific example of what we want our
synthetic control to do which is closely track the
actual outcome variable pre-treatment and closely
match on the chosen characteristic variables. If we have that, our work is done and the treatment effect is what it is. In the case of West Germany, we see the country was
around 12% poorer in 2003 than it would have been
without reunification. Like propensity score matching, synthetic control matches on observables meaning that time varying unobservables could confound the results. However, if the control is
able to mimic the actual unit over a long pre-treatment period, then unobservables are not a
big problem during that period. Which means maybe they
aren't a big problem in the treatment period either. As described, the method does not provide a significance test on
the treatment effect. This isn't necessarily a problem as case studies generally
do not provide such tests. However, we can create
a significance level via what is know as a permutation test. To do this test, we estimate
a series of placebos. That is we pretend that
the treatment occurred in one of the control units, create a synthetic for that
unit in the pretreatment period, and then compare the fit
of the actual and synthetic in the treatment period. We repeat this process for
each of the control units. Obviously, since the
treatment didn't happen in the control countries,
we shouldn't see an effect. So if the bogus treatment
effects are estimated in the control units are
the same size or larger than the one we estimated
for the treated units, we would infer that the
estimated treatment effect is not really significant. It looks just like the noise generated by the placebo treatments. Under the assumption that
the treatment is randomly assigned to a unit, we can
use the rank of the size of our estimated treatment effect compared to the placebo effects as a significance test. In the paper on German reunification, they show the estimated
effect for West Germany is larger than any of the
estimated placebo effects. The probability of that
happening is one 1/17th. Or 17 is equal to the total
number of countries used, namely West Germany, and the 16 controls. 1/17th is equal to about 0.06 and that could be taken
as the significance level of estimate of the cost to West Germany of the reunification. Case studies often lack rigor and synthetic control is
a good way to make them more systematic and believable. To run a good synthetic
control experiment, we need a long pre-treatment time period to establish the control. We need a set of potential control units that are similar to the treated units, and we need to ensure
these potential controls do not receive the same
treatment as our treated unit. We also need a good set of variables that help to explain the
outcome we are studying. If the synthetic control outcome closely matches the treated unit's outcome during the pre-treatment period and matches the treated unit on the values on the predictor variables chosen, then we have a good
model and can simply take the difference of outcomes
between the synthetic and the actual treated unit
during the treatment period as our estimated treatment effect. If we have our heart set on
having a significance test, we can get one by performing
a series of placebo test and creating the significance
level for the estimated treatment effect based
on the rank ordering of the sizes of the estimated effects, both actual and placebo. Well, that's about it class. I hope you've enjoyed and
learned from this short course on quasi experimental methods. Always remember, friends
don't let friends run OLS regressions to
estimate treatment effects. (humming tones)
