Welcome to Impact Factor, your weekly commentary
on a new medical study. I’m Dr. F. Perry Wilson of the Yale School
of Medicine. Despite COVID-19 death rates holding relatively
steady, there is a palpable sense of pandemic optimism in the US and abroad. The Omicron wave – the largest of the pandemic
– has, in most of the world peaked, and is now receding. There is broad hope that the immunity engendered
by that wave, combined with that engendered by vaccines, may bring us to the end of the
pandemic. And with that hope, a slew of changes in governmental
policies regarding COVID – with even liberal coastal governors like my own Ned Lamont in
Connecticut starting to rescind mask mandates and other mitigation policies. As the pandemic ebbs, it is natural and appropriate
to look back on the decisions we made along the way and ask – were they correct? Did we do the right thing? A new “working paper” made a huge splash
last week by claiming that, when it comes to lockdowns – we made a terrible mistake. The paper comes from three Johns Hopkins Economists
and is a meta-analysis of other studies examining the effect of lockdown policies on mortality. Meta-analyses are useful tools – combining
the estimates of various studies into one overall estimate – allowing the vagaries
of study design and the vicissitudes of chance to wash out via averaging and, ideally, produce
an estimate of effect that is something closer to the truth. So is their overall finding – that lockdowns
do essentially nothing to reduce deaths from COVID – close to the truth? It was good enough for the media. Fox News ran multiple segments on the study,
on the news shows as well as on Hannity. It garnered an Editorial in the New York Post
and Newsweek as well. If true, the implications are two-fold. First, the individuals who were pushing hard
against government restrictions during COVID would get to drink a big glass of I-told-you-so,
but more importantly, when the next pandemic occurs – and it will – we can handle it
differently. But is it true? Is it possible that government restrictions
have no effect on COVID mortality? That would go against the whole pandemic playbook
– not just this pandemic – the sort of basic tenets of infectious disease public
health. So there may be a problem here. But before we get too deep, we need to talk
about this term – “lockdown”. What does that even mean? At first blush, when I think “lockdown”,
I think “shelter in place” – all but essential businesses closed and all of the
population told to remain in their homes. This is not what the authors mean when they
use the term lockdown. They state “Lockdowns are defined as the
imposition of at least one compulsory, non-pharmaceutical intervention”. So – closing schools? Lockdown. International travel ban? Lockdown. Mask mandate? Lockdown. Seems a bit hyperbolic to me, but there it
is. OK back to why this meta-analysis might not
be telling us the whole truth. The magic of a meta-analysis arises from the
studies you put into it, so let’s start with the included studies. Figuring out the impact of lockdowns on covid
mortality is not straight-forward. There are a myriad of approaches to try to
answer the question, but they fall into a few categories. First, you can do a modeling study. The idea here is to take the epidemic curve
you observe before lockdown and predict where it would go assuming the conditions on the
ground didn’t change. Then you observe where the curve actually
did go after the lockdown was instituted. The difference between predicted and observed
is an estimate of lockdown effectiveness. This study in Nature showing the effect of
lockdowns during the first Wave in Europe used that approach. You can see here how the predicted deaths
assuming no lockdown (in blue) was substantially higher than those observed once lockdowns
were put in place. This study does not appear in the meta-analysis. In fact, all studies that used a modeling
approach were excluded. You can try to get at the lockdown : mortality
effect by using a before-after study. Here, you simply look at the death rate before
lockdown and after lockdown and compare them. Of course, you need to account for other stuff
that changes with time, including the epidemic curve, but good epidemiologists are pretty
good at that. This is a PLOS Medicine paper that looks at
COVID cases and mortality before and after lockdowns by state in the US. Even after adjusting for things like the size
of the pandemic when the restrictions were put in place, the results show a significant
decrease in death rates once lockdown occurs. This study does not appear in the meta-analysis. In fact, all studies using a before/after
approach were excluded. You can take what’s known as a difference-in-difference
approach. Look at two, hopefully similar, locations
over time where one institutes some sort of lockdown and the other doesn’t. Then watch their respective death rates. This is the only type of study allowed in
this meta-analysis. There are some good difference-in-difference
studies, and some bad ones. Epidemiology is hard. In any case – after poring through 18,590
potentially eligible studies, the authors narrow the field to 34 studies for review. 24 of those were included in the meta-analysis. Of those 24, 12 are pre-prints. Some of the studies looked at the effect of
shelter-in-place orders – the only thing that really says “lockdown” to me. Overall, the authors estimate that shelter
in place orders decreased COVID mortality by 2.9%. That works out to around 27,000 people in
the US, which isn’t nothing, but would not be described as a pandemic game changer. The problem is that there are plenty of studies
that estimate that shelter in place orders have a much stronger effect. The meta-analysis contains studies, like the
Sears study here, that show a greater than 30% reduction in deaths, but they are given
very little weight in the analysis compared to one study – Alderman - which shows a
1% reduction in death rate. The authors describe how they create the weights
for each study but they don’t seem appropriate to me. In fact, the Sears study, which showed the
30% reduction in deaths, is an analysis of data from all 50 states. So is the Alderman study. Why is the Alderman study given 28 times the
weight in the meta-analysis? You see the same effect of weighting in another
analysis looking at studies that used the Oxford “stringency index”. The stringency index is an ordinal scale where
higher values mean tighter restrictions – from recommendations to social distance, to full
shelter-in-place orders. One study in that analysis, from the CDC,
showed that higher stringency was associated with lower death rates. The authors report that the CDC study implies
a 35% reduction in mortality when comparing a stringency level similar to what the US
had with a lower level of stringency more consistent with advice to be careful. Now compare the weight of that CDC study,
11, with the weight of this study by Dr. Carolyn Chisadza – 7390. That huge weight, assigned to one study, drives
the entire stringency score-based analysis. Given that many of these studies use national
and even international databases, it seems unlikely that one study should be given so
much higher importance. And, just to mention it, the author of that
study – apparently – disagreed with the meta-analysts interpretation saying: “They think that lockdown had no effect
on mortality, and that’s what they set out to show in their paper”
Turning back to the shelter-in-place order analysis, studies in the meta-analysis are
also given a quality score. The authors show that higher-quality studies
show less effect of shelter-in-place on mortality. For example, the four studies with only 2
out of 4 quality points show a 34% reduction in deaths, whereas the 4 studies that got
all 4 quality points show just a 1% decrease in deaths. There are several good instruments for scoring
the quality of medical studies. The four point system the meta-analysis authors
used is not one of them. Rather, you get a point if your paper was
peer-reviewed, a point if your study ended after May 31st, a point if your study shows
no effect on mortality in the first 14 days, and a point if your study is written by a
social scientist as opposed to an epidemiologist or other scientist. What? With the exception of the peer-review thing,
these quality metrics do not make sense. In fact, they seem designed to penalize studies
that show a mortality effect of lockdown. Now I’ve been focusing a lot on shelter-in-place
orders, but remember the authors define lockdown as any non-pharmacological mandate. And they do provide an analysis of mortality
prevention stratified by various mandates. It looks like this. As you can see, these authors find that mask
mandates reduce covid deaths by more than 20%, and that closing bars and restaurants
reduce deaths by more than 10%. These findings were not reported by the media. The narrative around this study solidified
firmly around “lockdowns have no effect”. Ok here’s the thing. Determining the effect of government policies
on COVID mortality is really hard. There have been a lot of papers – a lot
of attempts at figuring this out – so many, in fact, that you could write a meta-analysis
like the one we’re discussing today that says there are no real effects, or like this
one which argues the effects are quite profound. Personally, I think the clearest argument
for the efficacy of government mandates, at least in terms of curbing COVID mortality,
is China, which has experienced 3.5 COVID deaths per million population compared to
2800 in the US. Obviously, I would never accept the extremity
of lockdown that China’s authoritarian regime has imposed – and I doubt my fellow American
citizens would either. But from a purely scientific standpoint, it’s
hard to argue that lockdown – in the extreme at least – doesn’t work. The problem in the liberal democracies is
that the infection rate, government mandates, and personal choices are all interrelated. People might be more careful about masking
or social distancing because case rates are high. They might be more careful because the mere
fact that a government mandate exists makes them take the virus more seriously. And of course, they might be careful because
they are forced to be careful. It can be hard to tease that all apart. When the next pandemic occurs, this question
will come up again. How much should governments do to curb the
spread? It’s another one of those public health
outbreak tenets that you would rather do too much than too little. The problem, of course, is that if you do
too much – no one will ever know how bad it could have been. And if you do too little, well, people die. It’s a lose-lose from a PR standpoint. Criticism is inevitable. Of course, that’s all part of the job. For Medscape, I'm Perry Wilson.
