A plane plummets out of the sky, a speed runner inexplicably
jumps to a higher platform. What the? What the?! And an election
recount is triggered. All because of the same
invisible phenomenon that permeates the universe. On May 18th, 2003, voters in
Belgium went to the polls. In many regions, voting
was done on a computer, something the Belgians had
been experimenting with for over a decade. But the system had a backup. Each voter would insert a
magnetic card into the machine and make their selection on screen. Their vote was saved both to the computer and the magnetic card, which they dropped into
a box for redundancy. Late that night, as the
votes were being tabulated, one of the election
officials detected a problem with the results from Schaerbeek, a municipality in central Brussels. Maria Vindevogel, a little known candidate with her own party received more votes than was mathematically possible. They knew this because of the way the preferential voting system works. So they took out the magnetic
cards and started a recount. One by one, they fed each of them through the machines again, and after several hours,
the recount was complete. The vote totals for every candidate were exactly the same as before, except for Maria Vindevogel. In her case, the recounted number of votes was less than the original by 4,096. So what went wrong? How had her original tally been inflated by over 4,000 votes? Computer experts were brought
in to run extensive tests on the software. They combed through the
code, but could find no bugs. They got the computer that had made the initial erroneous tally and tested the hardware again and again, but they could not replicate the error. Everything about the hardware seemed to be in perfect working order. And this left only one
possible explanation and it is seriously weird. The clue comes from the
excess number of votes Vindevogel received. 4,096. Computers work using binary,
strings of zeros and ones, each corresponding to a power of two. So somewhere inside the computer
tabulating all the votes was a string of bits representing the number
of votes Maria received. It started the day all zeros, and then as each vote for her came in, it would increment by one. Physically, this is done by
turning on a transistor for one and turning it off for zero. What's remarkable about the number 4,096 is that it is exactly a power of two. Two to the power of 12. That is the 13th bit. So for Maria Vindevogel to
receive an extra 4,096 votes, only one thing needed to happen. The 13th bit had to flip
from a zero to a one. But why would that happen? Computers work precisely
because bits don't flip unless we want them to, or do they? Looking into the problem, Belgian investigators found
reports of similar issues from big computer companies
starting in the 1970s. In 1978, Intel reported some
strange errors popping up in their 16 kilobit dynamic
random access memory or DRAM. Ones would spontaneously flip to zeros with no apparent cause. The problem turned out to
be the ceramic packaging the chip was encased in. With the demand for semiconductor
packaging skyrocketing in the 1970s, a new manufacturing
plant was constructed on the Green River in Colorado. Unfortunately, this site
happened to be just downstream of an old uranium mill. Radioactive atoms made
their way into the river and then into the ceramic
packaging for Intel's microchips. Intel scientists investigating the problem found that even trace amounts
of uranium and thorium in the ceramic were
sufficient to cause problems. In their DRAM, memory was
stored as the presence or absence of electrons
in a semiconductor well. The alpha particles emitted
by uranium and thorium were energetic and ionizing enough to create electron hole
pairs in the silicone. If an alpha particle is struck
in just the right place, it could create a large
number of free charge carriers causing electrons to
accumulate in the well flipping a one to a zero. This is known as a single event
upset, a type of soft error. The error is soft because the
device hasn't been damaged. The bit has changed, but you could erase it and
rewrite it with no problems. Investigators exposed the
chips to alpha emitters with different levels of activity. And just as you'd expect, they found the number of bit
flips directly correlated with the number of alpha particles the chip had been exposed to. The reason this problem
was identified in the 1970s was because chip components
had been miniaturized to the point where a single alpha particle could produce enough charge to flip a bit. Immediately, these findings
attracted a lot of attention. Before the paper was published, it was widely circulated in the industry. And as a result, chip manufacturers were a lot more careful to
avoid radioactive materials when producing their
microchips and packaging. Therefore, the bit flip
that gave Maria Vindevogel 4,096 extra votes wasn't caused by natural
radioactivity in the computer. So where did it come from? After Henri Becquerel
discovered radioactivity with uranium in 1896, scientists sought a way to measure it. How radioactive were different materials? And one way to do this is
with a gold leaf electrometer. When it's charged, the leaf is repelled and you can measure the amount of charge by the angle of the leaf. Now, if ionizing radiation
enters the chamber, it rips electrons off air molecules, creating positive and negative charges. Opposite charges are
attracted to the leaf, discharging it over time. The higher the level
of ionizing radiation, the faster the device discharges. In 1910, Theodore Wolf
took his electrometer to the top of the Eiffel Tower. Since radioactivity was found in the soil and rocks of earth, he
expected that 300 meters up, the radiation would be just a few percent of the ground radiation. Instead he found only a slight decrease. In 1911, Austrian physicist Victor Hess decided to take this experiment further. Literally. He loaded electric scopes into the basket of a hydrogen balloon. In his first two flights, he observed the same thing as Wolf. Up to an altitude of 1100 meters, both trips revealed no
fundamental change in radiation compared to the values
observed on the ground. But the next year, he
conducted seven balloon flights up to an altitude of 5,200 meters. And here he discovered
something remarkable. While there was an
initial drop in radiation for the first several hundred meters, above one kilometer or
so the level increased with increasing altitude. At his maximum height, the level of radiation
was several times greater than it was on the ground. The radiation appeared to be
coming, not from the earth, but from the sky. Hess scheduled one of his balloon flights during a solar eclipse. And as the moon passed
in front of the sun, he carefully watched his instruments. But the readings were unaffected. Even with the sun half covered, the level of radiation was the same. Since no influence of the eclipse on the penetrating radiation was noted, we conclude that even if
a part of the radiation should be of cosmic origin, it
hardly emanates from the sun. Victor Hess had discovered cosmic rays. High energy radiation from space. But what were these rays exactly and where were they coming from? Well, today we know they
aren't electromagnetic rays as many suspected, but particles. Around 90% are protons, 9% are helium nuclei, and
1% are heavier nuclei. Some of them are from the sun, but they have comparatively low energy. High energy cosmic rays moving very close to the speed of light come
from exploding stars, supernova in our own galaxy and in others. And the highest energy particles are thought to come from black holes, including the super massive black holes at the centers of galaxies. But it's hard to pin down exactly where cosmic rays come from because as charged particles, they're deflected by
magnetic fields in space. So they can wind their
way through the universe for billions of years. A cosmic ray detected
on October 15th, 1991 had an energy of 51 joules. That is a single subatomic particle with the energy of a baseball going a hundred kilometers per hour. For obvious reasons, it was
dubbed the OMG particle. But primary cosmic rays like these don't make it down to earth's surface. Instead, they collide with air molecules around 25 kilometers above the ground and create new particles like pions. These collide and decay into
other particles like neutrons, protons, muons, electrons,
positrons and photons, which in turn collide
with other air molecules in one long cascade. So from a single primary cosmic ray, comes a shower of particles
streaming towards the earth. It is one of these
particles that investigators suspect struck a transistor
in a computer in Belgium, flipping the 13th bit from a zero to a one and giving Maria Vindevogel
4,096 extra votes. But how often do things like this happen? In 1911, Charles Wilson made it possible to see the cosmic rays all around us when he perfected his cloud chamber, an enclosure with super
saturated water or alcohol vapor. When a cosmic ray passes
through the chamber, it ionizes the gas
causing vapor to condense into tiny droplets on the ions, revealing the path of the particle. Alpha particles, helium nuclei, leave short thick tracks
while beta particles, electrons leave long skinny trails. In 1932, Carl Anderson identified a trail that looked like it was
made by an electron, but in the applied magnetic field, it curved in the wrong direction. Implying it had a positive charge. Anderson had found the
anti electron or positron. It was the first confirmed
sighting of anti-matter. Four years later, also
using a cloud chamber, he discovered the muon,
again in cosmic rays. For his discovery of the positron, Anderson was awarded the Nobel
Prize in Physics in 1936. He shared the prize with Victor Hess, the man who discovered cosmic
rays in the first place, invisible particles that
affect our lives in ways most of us are oblivious to. This is possibly the rarest thing to ever happen in a video game. In 2013, user DOTA_Teabag was
speed running Super Mario 64 on the console. In the level Tik Tok Clock, he suddenly up warps
onto a higher platform. - [DOTA_Teabag] What the? - [Player] Did you get invisible wall? What?
- Please say you got the--- - [DOTA_Teabag] No, that
was the craziest thing I've ever seen though. - [Narrator] The move
shaves off a few seconds and it seems like a newly
discovered glitch in the game that could give speed runners and edge. User PenandCook12 put out a $1,000 bounty for anyone who could
replicate the up warp. But so far, after six years, no one has been able to. The best explanation
anyone can come up with is that a cosmic ray caused the glitch. It's been shown that a single bit flipped in the first bite of
Mario's height co-ordinate could have caused the effect. On the main level, the
bite was 1 1 0 0 0 1 0 1. But if you flip the last one to a zero, it changes his vertical position. And just by chance, this new height coincides
with the higher floor. PenandCook12 wrote a script
to manually flip the bit at the right moment
and was able to achieve the same up warp. This is a particularly visible bit flip, but the truth is cosmic rays
are triggering bit flips all the time. - An upset there, transient
there can alter the function of these devices and we
call that a single event functional interrupt. So an entire process can hang. So a blue screen of death that you get might actually have been
a neutron or whatnot. - [Derek] When people see
the blue screen of death, could that be caused by a cosmic ray? - Absolutely. - These days, there are a number of ways computer chips are made resilient
in the face of bit flips like error correction code or ECC memory. But you can't totally prevent
bit flips from happening. In 1996, IBM estimated that
for each 256 megabytes of RAM, one bit flip occurs per month. And the main culprit seems to be neutrons created in the shower of
particles from cosmic rays. Starting in 2009, Toyota recalled millions of vehicles due to the problem of
unintended acceleration. - We were in the fast
lane driving at about 70, and he said that the car
was continuing to accelerate and he couldn't bring it to a stop. - Many speculated that
cosmic ray induced bit flips in the electronic control
system were the cause. So much so that NASA was called in to help with the investigation. But it turns out, cosmic rays
were probably not the culprit. NASA identified as the main causes, sticky accelerator pedals, poorly fitted floor
mats and most commonly, drivers pushing on the accelerator thinking it was the brake. But cosmic rays have caused
crashes of supercomputers, especially at higher elevations. Los Alamos National Labs located
2200 meters above sea level is constantly dealing with neutron induced supercomputer crashes. So the software auto saves
frequently and neutron detectors have been installed
throughout the facility. If you go even higher, like climbing up to cruising
altitude in a plane, you can see the radiation
from cosmic rays increasing on a Geiger counter. To .5 microsieverts per
hour at 18,000 feet. Up to one microsieverts
per hour at 23,000 feet. Over two microsieverts
per hour at 33,000 feet. And over three microsieverts per hour at even higher altitudes
and towards the poles. At cruising altitude, this increases the chance
of a single event upset by 10 to 30 times. This isn't critical if it
happens in your laptop, but what if it occurs
in the flight computer? On October 7th, 2008, an Airbus A330 took off
from Singapore to Perth. Just over three hours into the flight, the plane suddenly pitched down diving 200 meters in 20 seconds. Inside the plane, everyone
experienced negative 0.8 Gs of acceleration. It would have felt like
the plane had flipped over. - The G-Force was enough, even with our three point harness to lift us both out of the seat and push us forward as well. - Minutes later, the plane
dropped another 120 meters. 119 people on board were injured, many from bumping their
heads into the ceiling. So the pilots decided to
make an emergency landing in Learmonth. In the investigation that
followed, the fault seem to occur with the first air data
inertial reference unit, or ADIRU for short. This computer supplies
critical data like airspeed, angle of attack and altitude. The way it supplies each of
these pieces of information is in a 32 bit binary word. The first eight bits identify
the type of information and bits 11 to 29 encode the actual data. What seems to have happened is that a bit flip in the first eight bits meant altitude information was mislabeled as angle of attack information. Inside the cockpit, alarms went on for over speed and stall simultaneously, something that should be impossible. But the plane nose down sharply
to correct what it thought was a stall, throwing passengers
and crew into the ceiling. Investigators looked into software bugs, software corruption, a hardware fail, physical environment factors and electromagnetic interference. But each of these possibilities
was found to be unlikely based on multiple sources of evidence. The other potential triggering event was a single event effect resulting from a high
energy atmospheric particle striking one of the integrated circuits within the CPU module. One of the challenges
with single event upsets is that they are soft errors. They don't leave a trace, but interestingly, the
Airbus A330 was built in 1992 when there were no specific regulatory or aircraft manufacturer requirements for airborne systems to be resilient to single event effects. With the space shuttle, redundancy was built in from the start. For navigation and control,
there were four computers simultaneously running identical software. If one computer had a soft error, the other three would overrule it. And using this setup, they could also track the
frequency of bit flips. On one five day mission, STS 48, there were 161 separate bit flips. Above the atmosphere,
cosmic rays are so energetic sometimes you can even see them. - Once in a while you
have your eyes closed and you're not asleep yet. And if you wait a little while, you occasionally will
see a flash of light. And we think it is heavy particles or individual bursts of
energy coming from radiation that are either going
through the eyeball itself or going through the optic nerve. And they, the way that your body registers
radiation going through it is amazingly enough by
showing you a little flash in one of your eyes, just to remind you that you
are subject to the radiation of not only our sun, but
every star of the universe, that is radiating at you. I picture back to the first astronauts who must have closed their
eyes and seen that radiation and gone, "I'm not going
to tell anybody about this "because no one's told me about it. "I'm not talking." I can just imagine the
first two guys that said, "Hey, I am, "sometimes I see flashes of light. "Do you see flashes of light?" And then it's, "Oh, we
all see flashes of light." "Oh, okay, well, that's all right then." (narrator laughing) - [Narrator] For missions
to other planets, protecting electronics is critical. - If a single bit of information controls a critical
function on your spacecraft, let's say your thrusters, and that goes from a one to
is zero, from a on to off, or vice versa, you could lose the mission. - [Narrator] That's why the computer on the Perseverance Rover
that just landed on Mars is 20 years old. It's a power PC launched in 2001 with only 256 megabytes of Ram and two gigabytes of flash storage. But it is radiation hardened, meaning the design, materials,
circuits and software are built to withstand
40 times the radiation of an ordinary computer. It's been used on over
a dozen space missions going back to 2005. - In fact, when we first started
doing the power PC testing years ago, the way we did it, we just simply stuck an
operating system processor in a beam line where we
generate these particles on the planet and look
for blue screens of death. You can kind of figure
out what's going wrong and try to undo that. So you don't get to the
blue screen of death because a spacecraft
that gets into that mode is basically unrecoverable. - As the Voyager 1 spacecraft
departed the solar system, one of the ways we could
tell was by an increase in the flux of cosmic rays it experienced. Although on earth, the
particles streaming from the sun and the solar wind are
a source of radiation, further out these same particles
maintain protective bubble. The heliosphere. It helps deflect and slow cosmic rays from interstellar space protecting the solar system
from ionizing radiation. But the sun has an 11 year activity cycle. So this protection fluctuates. The flux of cosmic rays
on earth is much lower when the sun is active
than when it's dormant. In the history of our planet, cosmic rays may have
played an even larger role flipping bits, not in electronics, but in the genetic codes
of living organisms, providing some of the variation on which natural selection acts. Maria Vindevogel is now a member of the Belgium Chamber of Representatives, elected by people, not a particle, but her story is a reminder
of the zillions of particles winding their way through the universe for millions or billions of years. One of which might at any moment, change your life by passing
through a tiny transistor and cra.... (electronic disturbance) Discovering the origin of cosmic rays or where 4,096 extra votes came from or how to protect the
Mars Rover from radiation requires problem solving,
which is a perishable skill. And the best way to keep your
problem solving skills sharp is to solve a diverse range of problems, which you can do with the
sponsor of this video, Brilliant. Brilliant is a website and app that teaches you STEM concepts
in an interactive way. They have courses on topics
from applied computer science to special relativity. Two courses I would recommend
if you enjoyed this video, are computer memory and
algorithm fundamentals. Brilliant have really upped
their interactivity recently. So instead of reading about algorithms, you're taught how to
write algorithms yourself. Brilliant engages you in active learning. Instead of telling you the right answer, they allow you to figure
it out for yourself with plenty of helpful hints
and guidance along the way. And for viewers of this channel, Brilliant is offering 20%
off an annual subscription to the first 200 people to sign up. Just go to brilliant.org/veritasium. I'll put that link down
in the description. So I want to thank Brilliant
for supporting Veritasium and I want to thank you for watching.
This is actually part of what I do for work: how to understand and mitigate single event effects like bit flips.
Edit: if you wanna hear more stories, RadioLab did an episode on this a few years ago: https://www.wnycstudios.org/podcasts/radiolab/articles/bit-flip
I work in IT, I'm gonna see if I can use this as an excuse the next time something happens that I can't explain. I mean, it's theoretically possible
So that's why my code crashed that one time! I knew it had absolutely nothing to do with my code
Back in the day (late 90s) I looked after a number of clients for HP. We had a V-class (big stuff then) crash due to a memory error, taking down a client's inventory management system and warehouse. Huge disruption.
The offending modules were duly sent back to the lab for analysis. The root cause? "Cosmic Rays".
2nd best root cause ever. The best was a client having a logic error in a script that cleaned up the file system after a backup. At one point it moved itself into superuser and then recursively deleted everything in the directory. Only problem was there was no error checking, so when it was in the wrong directory it didn't detect it and kept running. It was in the root directory so it blew away everything. Actually ran for another hour or so but with no operating system weird stuff happened. I worked 27 hours straight that day....
Veritasium's video on Gödels incompleteness theorems is also 10/10 (though not directly space-related)
I love Veritasium but god damn he always changes up his titles and thumbnails. I saw the video when it first came out and this is the 3rd time he's switched up the title and thumbnail.
What a great channel. I found out about him last year, ended up watching most of his content, and am always excited at his new uploads. This one doesn't disappoint!
Although the video doesn't mention it, cosmic rays almost destroyed Google Search in the early days. https://www.newyorker.com/magazine/2018/12/10/the-friendship-that-made-google-huge
Oh heck yeah, my favorite topic! I took a geiger counter on a plane once and graphed the results. Very fun.