History's Worst Software Error

katie yarborough woke up on a warm clear june day in 1985 and prepared for her 12th cancer treatment the 61 year old manicurist got dressed and drove herself to the kennestone regional oncology center in marietta georgia where a state-of-the-art linear accelerator called the therac-25 would direct high-energy electrons and or x-rays into her lymph nodes as it had done for patients in the area thousands of times before the therac would need only a few seconds to painlessly deliver around 200 rads to her upper left chest but that day something went wrong yarborough felt a red-hot sensation instead of nothing you burned me she told the technician who quickly assured her that this wasn't possible over the next few weeks she would need one breast fully removed and her left arm would become completely paralyzed but her useless arm didn't stop her from living her life or from driving she died five years later when her car was struck by a truck on a georgia highway katie yarborough was the first victim of what would be later called some of the worst software caused accidents in history this is the true story of the therak-25 how do you treat an insidious and deep-seated disease like cancer and its many forms without invasive and dangerous surgeries one answer in use for over a hundred years now is radiation or radiotherapy the concept is simpler than its name suggests radiation in the form of high energy particles and photons can ionize or otherwise change atoms and molecules in a sensitive structure like dna enough of this damage can lead to the death of a cell a disease like cancer progresses through the unchecked division of cells so why not try blasting these mutants with radiation that can by its very nature pass invisibly through body tissue what began in earnest only after world war ii and the first nuclear reactors is now a highly sophisticated field that uses three-dimensional body imaging and targeted beams of radiation from linear particle accelerators to prevent halt and otherwise destroy cancer cells in the best cases radiotherapy is considered an effective weapon in 4 out of 10 cancers no scalpel required in 1976 aecl medical a division of atomic energy of canada limited developed a revolutionary double pass accelerator which streamlined linear accelerator designs by using electromagnets to send beams through a target twice instead of once the therak-25 was one such double pass machine 7 feet high and 12 feet wide smaller than previous accelerators also unlike the accelerators of old the therac-25 was run principally by software instead of hardware lines of code instead of interdependent physical mechanisms in 1983 aecl performed a safety analysis on the new machine and started selling the therak25 to excited customers this state-of-the-art device was in high demand however left out of that 1983 analysis was any interrogation of the software that ran these complicated devices of the code based on the older therak-20 model and written by a single person a coding hobbyist who left the company in 1986. he remains unidentified to this day two weeks after katie yarborough told her technician that she felt a burning sensation during her cancer treatment there was a red mark the size of a dime on her chest and directly opposite that mark a larger disc on her back tim still the medical physicist at kennestone examined her that looks like the exit dose made by an electron beam he said it looked nothing like what could be created by her prescribed 200 rad dose the physicist later estimated what actually hit yarborough was closer to 20 000 rats hundreds of times more than what you'd receive standing inside a failed reactor at fukushima daiichi but dr still wasn't able to recreate a beam of that strength with the machine himself so he contacted a professional organization to tell them what had happened he quickly got a call from the aecl in response telling him to stop making these claims without any proof they assured him that such an overdose simply wasn't possible over the next few weeks the dime-sized red circle on yarborough's chest became a hole skin grafts failed as any new tissue simply rotted away her left breast recently cancer-free had to be entirely removed her left arm was now immobile many sources report it was though a slow-motion gunshot wound had gone through her chest and out of her back yarburo would hire a lawyer and sue the hospital and aecl in the october of 1985 but she wouldn't live to find out the reason why nanoscopic bullets had done this to her seven weeks later concerningly similar to katie yarbrough a 40 year old woman with cervical cancer arrived for her most recent therak-25 treatment at the hamilton regional cancer center in ontario canada she too was hit with a slow-motion bullet complaining of tingling electric shocks during treatment it would be later estimated that what the therak operator had mistakenly irradiated her hip width several times was a total of 17 000 rats a larger dose than what harry dogly and junior or louis slaughten received from the demon core the aecl was informed immediately and had an engineer dispatch to examine the unit the micro switches that controlled the position of the unit's turntable were deemed faulty and a software change to constantly check the turntable position was introduced aecl would later claim in a september letter to customers that this change had increased the safety of the therac-25 by five orders of magnitude but the cervical cancer patient died a month before this pronouncement on november 3rd her official cause of death was her cervical cancer though an autopsy revealed that if she had lived her hip obliterated by high energy radiation would have to have been entirely replaced five days later a letter from the canadian radiation protection bureau begged aeco for hardware fail-safes and additional software changes but nothing came of it a month after the heavily irradiated cervical cancer patient died it happened again a therak-25 unit at the yakima valley memorial hospital in washington state supposed to be now 9 million percent safer hit another cervical cancer patient in the hip with more radiation than what cecil kelly endured when a whirlpool of plutonium went prompt critical in his face thankfully the woman ultimately suffered only minor disability and scarring the more impactful outcome was that doctors and therak-25 operators in the u.s and canada were now talking to each other something was going on that the aecl obviously wasn't addressing or didn't care to two months later aecl declared that after careful consideration we are of the opinion that this damage could not have been produced by a malfunction of the therac-25 or by any operator error end quote however over the next 12 months therap-25 malfunction and operator error would kill three cancer patients the therac-25 software likely had around 100 000 lines of code small by today's standards but complicated nonetheless and error-prone operators would later testify that they encountered as many as 4 serious error messages a day many of those errors would simply read malfunction with a number from 1 to 64. these numbers were not explained not by aecl not in any manual operators also admitted that they became accustomed to this ambiguity rather than fearful of it they could and did simply press p to proceed without knowing whether or not an error code was benign or potentially deadly it was part of the job to keep an expensive and sought after machine like the therak running malfunction 54 one of these mysterious undefined errors would turn out to be the one you couldn't skip past but it would take three catastrophes before anyone figured out what they were allowing to happen [Music] voyn ray cox lay beneath the therak-25 unit at the east texas cancer center in tyler texas for his ninth cancer treatment a technician set his dose at 180 rads then she noticed a mistake she had selected x for x-ray instead of e for electron beam she quickly moved the cursor made the change and activated the machine malfunction 54. used to this by now she hit proceed anyway mr cox then felt a powerful shock according to reporting done by barbara wade rose cox tried to get up but because the intercom for the room just happened to be broken that day the technician didn't see him struggling or hear him screaming so she hit him again another shock ripped through mr cox the technician only stopped when she heard cox slamming the door she was behind with his fists he was examined by physicians and sent home told to return if anything changed a few weeks later he returned to the hospital spitting up blood [Music] after the accident no one can reproduce malfunction 54. aecl told the hospital that an overdose was impossible suggesting maybe it was indeed an electric shock that produced the sensations mr cox felt but ruled that out too the company claimed it knew of no other similar accidents so the use of the therak unit resumed 17 days later voin ray cox died the following august after receiving a calculated dose higher than the worst dose a liquidator would receive when the chernobyl nuclear power plant exploded just a month later only four days after the therac at the east texas cancer center was back online 66 year old bus driver vernon kidd walked through the lobby on the way to his scheduled treatment he was to have a therak-25 aimed at the skin cancer on his face malfunction 54 treatment proceeded a loud noise brought the technician back into the room to find mr kidd writhing in pain confused he said it had felt like something hit him on the side of the face he saw a flash of light heard an intense sizzling sound the therak-25 unit at the center was shut down until the cause could be determined verdan kidd died a month later from radiation-induced damage to his brain and brain stem his death four months before the death of voyn ray cox was the first recorded fatality from radiation treatment in medical history the therac technician on site that day and physicist dr fritz hager stayed the weekend after the kid accident attempting to recreate malfunction 54 the malfunction that aecl said wasn't possible they changed the machine's modes moved the cursor quickly up and down and typed in different treatment instructions for hours upon hours and then suddenly they did it dr hager telephoned to aecl immediately the fda already investigating the accidents declared the therak-25 defective and demanded a corrective action plan or cap from the company a letter soon went out from the aecl to all therac-25 users quote effective immediately and until further notice the key used for moving the cursor back through the prescription sequence must not be used for editing or any other purpose end quote it was something that everyone had missed and it was finally going to be fixed but even after everyone knew what malfunction 54 was and how to fix it the accidents continued you don't think of software as something being able to fail once working code is in a computer how could it bend like a steel beam or break like a pane of glass but like any machine there's a difference between how it's supposed to be used in theory and how it's actually used in practice the therac-25 used magnets to filter and control powerful beams of radiation magnets that after an input was received physically took eight seconds to move everything into position fine in theory what dr hager had figured out was that if an operator set radiation levels and then made a change to those levels within the eight seconds it took for the magnets to move the change was not detected the magnets were already in motion this could and did allow powerful unfiltered beams of radiation to strike patients this animation reproduced in spanish shows the sequence of events needed to produce malfunction 54. in theory an operator would make a change and then wait for the magnets in the machine to move but an experienced operator working with the therac every single day encountering multiple error messages a shift is in practice more than likely of making a change like changing a beam from x-ray to electron within 8 seconds there was no code in place to check whether the prescribed input on the monitor match what the machine was actually set up to do the therac-25's reliance on software and not hardware interlocks like previous models also meant that input errors didn't have mechanical fail-safes that wouldn't listen to the mistakes that ended up fatally irradiating people with malfunction 54 finally identified aecl sent the first corrective action plan to the fda part of which were changes to the therax software to tell the machine where the cursor actually was the cap was revised twice by the fda over the next few months and tharac25s were back in use before the end of the year six weeks later a therak unit killed again on january 17th 1987 glenn dodd 65 walked into the yakima valley memorial hospital for treatment of a carcinoma his disease was to be flooded with 86 rats the therak instead bombarded the man's chest with ten thousand he died from acute radiation poisoning three months later the staff at yakima reportedly stopped using the machine altogether after this accident they were paranoid they thought this had been fixed this was safe what did they miss what was going on eventually it was discovered that the therac25 had another invisible software error inside the code was a so-called housekeeper task that would constantly check whether or not the machine's turntable was in the correct position make adjustments if necessary and then revert to zero anything other than a zero in the code therefore was an error and the machine would not proceed with treatment again good in theory however like your car's odometer this code ticked up checks only until a certain value in this case one byte of memory 256 after that it would tick over to zero out of necessity if a technician entered an erroneous treatment at this precise moment just an instant before the next check the computer would read a zero no errors and proceed this is called arithmetic overflow and it's what killed glenn dodd in february 1987 the fda again declared the therak 25 defective recommended that all units be taken out of service until corrective action could be taken finally the accident stopped after months of revisions aecl told customers that the fda had accepted a final corrective action plan it included 23 software changes and six hardware safety features the largest of which was a dose per pulse monitor affixed to the machine that would shut down dangerous doses even if all the software safety checks failed however unsurprisingly after high profile and unprecedented accidents the therak was no more in 1988 aecl dissolved their medical division and lawsuits from families were settled out of court at the time of this recording with a cursory search i was unable to find any hospital actively using a therap-25 even if they are which isn't unlikely the machines are probably under a different name or company even without any further incidents to point to today no hospital wants to draw active comparison to disaster today the therac25 is more a staple of ethics and computer science class required readings than it is of medicine a unique case study of what can go wrong when new technology is trusted implicitly and when ethical decision making malfunctions aecl assumed that the software for the therac-25 written by a single unidentified hobbyist and imported from the older therak-20 model did not have residual software errors to be tested it didn't consider how the machine was being used in practice never envisioning something like malfunction 54 the company repeatedly denied knowing of any accidents and believed that overdoses were impossible it made proclamations like five orders of magnitude increases in safety that were physically impossible today the fda requires documentation on all software for new medical products which the therac-25 didn't have that can be investigated independently ten years after the deadliest software errors in history reporter barbara wade rose asked bill bird the lawyer for the first therac victim katie yarborough to comment on the events quote the thing that amazes me is that the people who develop these machines are surely some of the most brilliant people in the world this machine was unbelievably sophisticated and yet nobody would have gotten hurt if somebody had just used common sense until next time [Music] [Music] [Music] [Music] you

Channel: Kyle Hill

Views: 5,175,028

Keywords: because science, engineering, kyle hill, learning, math, physics, science, stem, the facility, chernobyl, nuclear, therac, therac-25 incident, therac-25 failure, therac-25 software bug, therac 25, therac-25, half-life histories, documentary, history

Id: Ap0orGCiou8

Length: 21min 4sec (1264 seconds)

Published: Wed Aug 31 2022