In 2011 an article was published in the
reputable "Journal of Personality and Social Psychology". It was called "Feeling the
Future: Experimental Evidence for Anomalous Retroactive Influences on
Cognition and Affect" or, in other words, proof that people can see into the
future. The paper reported on nine experiments. In one, participants were
shown two curtains on a computer screen and asked to predict which one had an
image behind it, the other just covered a blank wall. Once the participant made
their selection the computer randomly positioned an image behind one of the
curtains, then the selected curtain was pulled back to show either the image or
the blank wall the images were randomly selected from
one of three categories: neutral, negative, or erotic. If participants selected the
curtain covering the image this was considered a hit. Now with there being
two curtains and the images positions randomly behind one of them, you would
expect the hit rate to be about fifty percent. And that is exactly what the
researchers found, at least for negative neutral images however for erotic images the hit rate
was fifty-three percent. Does that mean that we can see into the future? Is that
slight deviation significant? Well to assess significance scientists usually
turn to p-values, a statistic that tells you how likely a result, at least this
extreme, is if the null hypothesis is true. In this case the null hypothesis
would just be that people couldn't actually see into the future and the
53-percent result was due to lucky guesses. For this study the p-value was .01 meaning there was just a one-percent chance of getting a hit rate of
fifty-three percent or higher from simple luck. p-values less than .05 are generally considered significant and worthy of publication but you might
want to use a higher bar before you accept that humans can accurately
perceive the future and, say, invite the study's author on your news program; but
hey, it's your choice. After all, the .05 threshold was arbitrarily selected by
Ronald Fisher in a book he published in 1925. But this raises the question: how much
of the published research literature is actually false? The intuitive answer
seems to be five percent. I mean if everyone is using p less than .05
as a cut-off for statistical significance, you would expect five of
every hundred results to be false positives but that unfortunately grossly
underestimates the problem and here's why. Imagine you're a researcher in a field
where there are a thousand hypotheses currently being investigated. Let's assume that ten percent of them
reflect true relationships and the rest are false, but no one of course knows
which are which, that's the whole point of doing the research. Now, assuming the
experiments are pretty well designed, they should correctly identify around
say 80 of the hundred true relationships this is known as a statistical power of
eighty percent, so 20 results are false negatives, perhaps the sample size was
too small or the measurements were not sensitive enough. Now considered that
from those 900 false hypotheses using a p-value of .05, forty-five false
hypotheses will be incorrectly considered true. As for the rest, they
will be correctly identified as false but most journals rarely published no
results: they make up just ten to thirty percent of papers depending on the field,
which means that the papers that eventually get published will include 80
true positive results: 45 false positive results and maybe 20
true negative results. Nearly a third of published results will be wrong even with the system working normally,
things get even worse if studies are underpowered, and analysis shows they
typically are, if there is a higher ratio of false-to-true hypotheses being tested
or if the researchers are biased. All of this was pointed out in 2005 paper
entitled "Why most published research is false". So, recently, researchers in a
number of fields have attempted to quantify the problem by replicating some
prominent past results. The Reproducibility Project repeated a hundred
psychology studies but found only thirty-six percent had a statistically
significant result the second time around and the strength of measured
relationships were on average half those of the original studies. An attempted
verification of 53 studies considered landmarks in the basic science of cancer
only managed to reproduce six even working closely with the original
study's authors these results are even worse than i just calculated the reason
for this is nicely illustrated by a 2015 study showing that eating a bar of
chocolate every day can help you lose weight faster. In this case the
participants were randomly allocated to one of three treatment groups: one went on a low-carb diet, another one on
the same low carb diet plus a 1.5 ounce bar of chocolate per day and the
third group was the control, instructed just to maintain their regular eating
habits at the end of three weeks the control group had neither lost nor
gained weight but both low carb groups had lost an average of five pounds per
person the group that a chocolate however lost
weight ten percent faster than the non-chocolate eaters the finding was statistically
significant with a p-value less than .05 As you might expect this news
spread like wildfire, to the front page of Bild, the most widely
circulated daily newspaper in Europe and into the Daily Star, the Irish Examiner,
to Huffington Post and even Shape Magazine unfortunately the whole thing had been
faked, kind of. I mean researchers did perform the experiment exactly as they
described, but they intentionally designed it to increase the likelihood
of false positives: the sample size was incredibly small, just five people per
treatment group, and for each person 18 different measurements were tracked
including: weight, cholesterol, sodium, blood protein levels, sleep quality,
well-being, and so on; so if weight loss didn't show a significant difference
there were plenty of other factors that might have. So the headline could have
been "chocolate lowers cholesterol" or "increases sleep quality" or... something. The point is: a p-value is only
really valid for a single measure once you're comparing a whole slew of
variables the probability that at least one of them gives you a false positive
goes way up, and this is known as "p-hacking". Researchers can make a lot of
decisions about their analysis that can decrease the p-value, for example let's
say you analyze your data and you find it nearly reaches statistical
significance, so you decide to collect just a few more data points to be sure then if the p-value drops below .05 you
stop collecting data, confident that these additional data points could only
have made the result more significant if there were really a true relationship
there, but numerical simulations show that relationships can cross the
significance threshold by adding more data points even though a much larger
sample would show that there really is no relationship. In fact, there are a
great number of ways to increase the likelihood of significant results like:
having two dependent variables, adding more observations, controlling for gender,
or dropping one of three conditions combining all three of these strategies
together increases the likelihood of a false-positive to over sixty
percent, and that is using p less than .05 Now if you think this is
just a problem for psychology neuroscience or medicine, consider the
pentaquark, an exotic particle made up of five quarks, as opposed to the
regular three for protons or neutrons. Particle physics employs particularly
stringent requirements for statistical significance referred to as 5-sigma or
one chance in 3.5 million of getting a false positive, but in 2002 a Japanese
experiment found evidence for the Theta-plus pentaquark, and in the two years
that followed 11 other independent experiments then looked for and found
evidence of that same pentaquark with very high levels of statistical
significance. From July 2003 to May 2004 a theoretical paper on pentaquarks
was published on average every other day, but alas, it was a false
discovery for their experimental attempts to confirm that theta-plus
pentaquark using greater statistical power failed to find any trace of its
existence. The problem was those first scientists
weren't blind to the data, they knew how the numbers were generated and what
answer they expected to get, and the way the data was cut and analyzed, or p-hacked,
produced the false finding. Now most scientists aren't p-hacking
maliciously, there are legitimate decisions to be made about how to collect, analyze and
report data, and these decisions impact on the statistical significance of
results. For example, 29 different research groups were given the same data
and asked to determine if dark-skinned soccer players are more likely to be
given red cards; using identical data some groups found there was no
significant effect while others concluded dark-skinned players were
three times as likely to receive a red card. The point is that data doesn't speak for
itself, it must be interpreted. Looking at those results it seems that dark skinned players are
more likely to get red carded but certainly not three times as likely;
consensus helps in this case but for most results only one research group
provides the analysis and therein lies the problem of incentives: scientists
have huge incentives to publish papers, in fact their careers depend on it; as
one scientist Brian Nosek puts it: "There is no cost to getting things wrong,
the cost is not getting them published". Journals are far more likely to publish results that reach statistical
significance so if a method of data analysis results in a p-value less than
.05 then you're likely to go with that method, publication's also more
likely if the result is novel and unexpected, this encourages researchers
to investigate more and more unlikely hypotheses which further decreases the
ratio of true to spurious relationships that are tested; now what about
replication? Isn't science meant to self-correct by having other scientists
replicate the findings of an initial discovery? In theory yes but in practice
it's more complicated, like take the precognition study from the start of
this video: three researchers attempted to replicate one of those experiments,
and what did they find? well, surprise surprise, the hit rate they
obtained was not significantly different from chance. When they tried to publish
their findings in the same journal as the original paper they were rejected.
The reason? The journal refuses to publish replication studies. So if you're
a scientist the successful strategy is clear and don't even attempt replication
studies because few journals will publish them, and there is a very good
chance that your results won't be statistically significant any way in
which case instead of being able to convince colleagues of the lack of
reproducibility of an effect you will be accused of just not doing it right. So a far better approach is to test
novel and unexpected hypotheses and then p-hack your way to a statistically
significant result. Now I don't want to be too cynical about this because over
the past 10 years things have started changing for the better. Many scientists acknowledge the problems
i've outlined and are starting to take steps to correct them: there are more
large-scale replication studies undertaken in the last 10 years, plus
there's a site, Retraction Watch, dedicated to publicizing papers that
have been withdrawn, there are online repositories for unpublished negative
results and there is a move towards submitting hypotheses and methods for
peer review before conducting experiments with the guarantee that
research will be published regardless of results so long as the procedure is
followed. This eliminates publication bias, promotes higher powered studies and
lessens the incentive for p-hacking. The thing I find most striking about the
reproducibility crisis in science is not the prevalence of incorrect information
in published scientific journals after all getting to the truth we know is hard
and mathematically not everything that is published can be correct. What gets me is the thought that even
trying our best to figure out what's true, using our most sophisticated and
rigorous mathematical tools: peer review, and the standards of practice, we still
get it wrong so often; so how frequently do we delude ourselves when we're not
using the scientific method? As flawed as our science may be, it is far away more
reliable than any other way of knowing that we have. This episode of veritasium
was supported in part by these fine people on Patreon and by Audible.com, the
leading provider of audiobooks online with hundreds of thousands of titles in
all areas of literature including: fiction, nonfiction and periodicals,
Audible offers a free 30-day trial to anyone who watches this channel, just go
to audible.com/veritasium so they know i sent you. A book i'd recommend is
called "The Invention of Nature" by Andrea Wolf which is a biography of Alexander
von Humboldt, an adventurer and naturalist who actually inspired Darwin to
board the Beagle; you can download that book or any other of your choosing for a
one month free trial at audible.com/veritasium so as always i want to thank
Audible for supporting me and I really want to thank you for watching.
I greatly enjoyed this video. The presentation is top notch. The narrator has a professional voice, no annoying incessant jump cuts, the message is delivered concisely, the background music is soothing and appropriate. Loved it!
Thank you for this high quality post!
So they're using published research to prove most published research is fake? Awesome
This video ignores the benefit of p-hacking, that being that researchers can and do find significant and surprising results that are incidental to their hypotheses. P-hacking is mainly the product of the pressure to publish, which is not useful or beneficial in itself, but it results in scientists learning to analyze data unconventionally, which is an inherently useful skill to have.
I'm reminded of astronomy and physics phD students I know who end up leaving academia but end up working in industry as "data analysts" with the skills they've developed.
Relevant XKCD
I kind of wish that in order to be published, you had to submit your experiment for replication by a third party. But no one would want to fund that, so...