Nickolas Means: Who Destroyed Three Mile Island at Stretch Conference 2020

our last speakers he's here from Austin Texas and actually he just told us that he's been recently promoted to director of engineering so please give him a round of applause for that yes and also if you cannot get enough of him now on stage he does have a podcast called managing God show so if you really like to follow him you can still listen to him when he is done speaking for you today let me give you an eClass means [Applause] thanks so much for that intro do you know my computer's being cantankerous here give me just a second all right should be good now it is such an honor to be here with you this afternoon I hope you've all had a wonderful day there's been so much good content today I'm sure your heads are all a little bit swimming with everything you've learned so far I know mine is like Edina said I'm a director of engineering at github I work on our security and compliance products so that means not keeping github secure and compliant but helping our users keep their code bases secure in compliance but I'm also an aficionado of stories of engineering triumph and engineering disaster I've been fascinated with this particular story the story of the Three Mile Island nuclear accident as long as I can remember actually and I think it's got some really important lessons to teach us now before I get started I'm curious how many of you feel like you have a really good grasp on how nuclear power generation works that's about what I would figure not very many so let's start there let's start with the fundamentals of nuclear power generation when I was a kid my parents gave me this for volume set of books called how things work my dad's a mechanical engineer by training and so I spent lots of time asking him how various complicated pieces of the world worked and he would give me these very long very patient explanations and so at some point my parents decided that they needed to teach me to answer some of these questions for myself and so they gave me this set of books I don't look at them very often these days thanks to the wonders of the internet and Wikipedia but they still have a treasured place on my bookcase because they're a big part of what made me who I am today I distinctly remember turning to them when it was all over the news that a new nuclear power station had opened near the town where I grew up when I was a kid I had no idea how a nuclear reactor made electricity and I wanted to know and I think that same diagram I looked at back then is a great place for us to start today it turns out the basic mechanics of a nuclear power plant really aren't that different from any other steam-driven power plant there's a heat source that heats up water now in a normal steam power plant that would be burning coal or burning natural gas or burning oil but in this case it's a carefully controlled nuclear chain reaction fueled by uranium high pressure water circulating through the reactor carries that heat from the reactor core to a steam generator where it's used to boil water converting in at a steam and then that when water turns to steam it expands and that expansion drives a turbine which is basically a giant fan in a tube the turbine turns a generator and that's where you get your electricity then the steam is pumped into a condenser which turned back into water for another trip around the loop now there are two primary families of nuclear power reactors operating the United States the boiling water reactor and the pressurized water reactor we're looking at a pressurized water reactor because that's the kind that was operating at Three Mile Island they have the accident so what is it about this design that makes it a pressurized water reactor well the component that I just walked you through are on two separate loops the primary loop and orange consists of the water that flows through the reactor vessel gathering heat passing through the steam generator and boiling water in the secondary loop passing that heat off and then going back into the reactor itself the water in the secondary loop in blue flows from the steam generator where it boils to steam through the turbine through the generator back through the condenser to turn back into water and round and round and and water from the two loops never only advantages of this is that you have the nuclear portion of the plant really tightly contained in the primary loop and the secondary loop should never have any exposure to nuclear energy the thing that makes this design a pressurized water reactor is that the water in the primary coolant loop has held at about 2,000 pounds per square inch a boiling water reactor has to have a very large reactor vessel to allow the water the room to boil and expand in the reactor vessel but a reactor vessel the pressurized water reactor can be much smaller and the reason for this is that pressurizing water raises the boiling point and it raises the boiling point high enough that the water in the primary loop will never boil even at the plants operating temperature about 315 degrees Celsius or at least it shouldn't which brings us to March 28 1979 Three Mile Island nuclear generating station is a two unit nuclear power plant in Londonderry Township Pennsylvania it's built on a three-mile long sandbar imaginably named Three Mile Island in the Susquehanna River about ten miles south of the capital of Pennsylvania Harrisburg unit two is a nine hundred and six megawatt electric pressurized water reactor designed by Babcock & Wilcox that went at a commercial operation on December 30th 1978 early in the morning of March 28 1979 it's running at 97% capacity and it has been since it went online it's running hot straight and normal as they say in the nuclear power industry these four men are at the controls of Three Mile Island unit two for the overnight shift on March 28 Bill's II we was the shift supervisor for units 1 & 2 he's the most senior person on site overnight Fred Shima was the shift foreman for unit 2 easy.we second-in-command and directly responsible for running unit 2 an ED Frederic and Craig Foust were the control room operators on duty they were the ones that were actually sitting at the control desk directly operating the reactor that night everything the plant was normal that night everything was running exactly as it should except for a small problem in one of the condensate polishers that the previous shift wasn't able to solve these are condensate polishers obviously not the ones actually from Three Mile Island it turns out it's really hard to get a have a specific component of a specific nuclear power plant as you might expect but this is what they look like these are condensate polishers and what these are are tanks that are filled with resin beads and through my Ireland has eight of them and the purpose of these resin beads is to filter the water coming out of the cut out of the condenser before it goes back into the steam generator the steam generators are made up of thousands of tiny tubes and the tiniest speck of rust or dirt or anything in that coolant loop can clog the steam generator and eventually you would have to replace the steam generator when it gets clogged off so to combat this they have these condensate polishers the water flows through them any impurities in the water stick to these resin beads the only problem is the condensate polishers like to get stuck because there's they like to get clogged because of all of this stuff that they're collecting and so every so often one of these condensate polishers would get clogged and they have to backwash it essentially reverse the flow of water through the filter to rinse all of that stuff out and into the into the waste tank this particular night the number-7 tank is completely clogged and the swing shift had tried to unclog it they'd used the backwash system they'd even hooked up an auxilary air system to try to force this clog through and get this tank moving again so at 3:59 in the morning fred Shaymin is down in the basement of the turbine hall to see how things are coming with with this clog he's climbed up on the side of the number-7 tank and he's peering in the viewing port to see if if there's any water flowing through it when things get incredibly silent now he's in a room where millions of gallons of water per hour flow through so things getting quiet is not a good sign and he barely jumps free before a water hammer comes through and knocks the feed line from its moorings what's happened is over the course of about ten hours or so water has leaked back up this pressurized air line through a leaking one-way check valve because the air supply that they were using to clear this clog in the condensate polishers was the same air supply that controlled the valves for the condensate polishers and these are failsafe valves so when the water finally got up to the manifold that supplied air to all of these valves it blocked the air supply and all eight valves shut simultaneously obviously this is not good but to help us understand why here's a schematic of Three Mile Island unit two now I know this looks a whole lot more complicated than the diagram I showed you a minute ago but it's got all the same parts so let me take a second get you oriented for starters you can see the primary cooling coolant loop here in orange and the secondary loop in shades of blue here in the center is the reactor vessel where the nuclear chain reaction creates heat next to it are the two steam generators where heat from the primary loop boils water in the secondary loop to create steam this is a two loop pressurized water reactor and so that means it's got two redundant steam generators the steam is then piped to the turbine building to turn the turbine and generator and then here's the condenser where the steam gets cooled and turns back into water and then right after the condenser is the condensate polisher and it's completely blocked now because the condensate polisher is blocked there's no water to be pumped through the secondary cooling loop so the main feedwater pump strip offline it's now 36 seconds past 4:00 in the morning the official start of the accident two seconds after the main feedwater pumps trip the turbine senses it's not going to be getting any more steam and so the turbine and generator trip offline as well and this causes the plant's main safeties to open so all the steam that's built up in the system gets vented into the night sky at 4:00 in the morning and they said you could hear the rumble for miles in the control room at Frederick and Craig Foust are getting their first indications that something has gone awry an alarm horn announcing the turbines turbine trip starts going off and several alarm indicators start to flash a few seconds after the turbine and generator alarms go off the pressure in the reactor vessel begins to climb rapidly but this pressure spike is expected because without the secondary cooling loop to remove heat the primary loop climbs in temperature and when water climbs and temperature it expands and this raises the pressure of the primary loop the good news is the plant is designed for exactly this situation and it was taking action to resolve the situation automatically as soon as those first alarms went off the reactors pressure control system is the first to jump in action now there are two components to the system and both of them are important to the accident sequence first we're going to talk about is the pressurizer now the pressurizer in a pressurized nuclear power plant serves a few different functions first it regulates system pressure it's the highest point and a closed system so raising the pressure of the pressurizer raises the pressure of the whole system lowering it lowers the pressure second it measures the water level when they designed this plant they made a decision to save money to not put any water level instrumentation directly in the reactor vessel and the reason they did this is because it's a very expensive design instrumentation that's going to be accurate in such a high nuclear activity field and so they said because we've got this pressurizer sitting here we can measure the water level in the reactor vessel just by checking the pressurizer as long as there's water in the pressurizer there has to be water on the reactor vessel and third because steam is more compressible than liquid water it serves as a shock absorber the steam in the top of the pressurizer absorbs any pressure transients in the system just like the one unit two is experiencing right now so this name of the pressurizer absorbs the initial shock but the pressurizer itself is really only designed for small pressure adjustments right now the primary cooling loop is at 2250 psi and climbing about 100 psi higher than it should be and it would take the pressurizer several minutes to make this large of an adjustment so what does the system do about big pressure changes like this one well that's where the pilot operated relief valve comes in and if you've ever heard anything at all about Three Mile Island unit two this is the component that you'll remember the name of because this is the most important component in the accident sequence in the event of a big pressure spiked the pilot-operated relief valve will open and release coolant into a drain tank on the containment building floor this is to immediately in quickly release pressure in the system the pilot operator relief valve opens four seconds after the turbine and generator trip off line the reactor control system is doing exactly what it's supposed to to keep the reactor in a safe operational state a few seconds later the computer senses that reactor pressure is still continuing to rise despite the pilot-operated relief valve being open so it takes another defensive action it's crams the reactor now to understand what a scram is we have to know a little bit about what's going on in the core of a nuclear reactor how it generates heat so the core of a nuclear reactor is full of uranium and there are a bunch of neutrons flying around and every time one of those neutrons hits a uranium atom it causes it to split when that uranium atom splits it releases more neutrons and a copious amount of heat with the neutrons that are released there they go and bump into other uranium atoms and that's why it's a nuclear chain reaction well the primary means of controlling the intensity intensity of this reaction is a set of cadmium control rods that can be gently raised or lowered into the the core of the reactor an academy and control rods absorbs some amount of those neutrons that are flying around and slow down the reaction a scram is the reactors emergency stop instead of being gradually put in the core of the reactor the control rods are released from the mechanism that raises and lowers them and are allowed to fall by gravity into the core of the reactor and once they're all the way in they stop the chain reaction almost instantly the thing you have to know though is that even after the scram the reactor core is still producing some amount of heat even though there's no ongoing chain reaction it's still producing about 6 and 1/2 percent of the heat that it was producing running at full tilt that listens sharply over the first hour but it's critical that the chorus kept cool for this first hour to avoid damaging the core a few seconds later back in the control room alight on the console turns from red to green to indicate that the pilot-operated relief valve has been signaled to close now at this point everything feels very much under control reactor and turbine trips aren't an everyday occurrence otherwise the plant couldn't generate electricity but there an event that the reactor crew is trained for they know exactly how to respond to it this is not out of their expertise the alarm horns are blaring alarm indicators are flashing that the system is behaving exactly as it's designed that feeling of control would last exactly two minutes because two minutes later at Frederick and Craig's Foust's world is thrown into chaos when the automatic emergency core cooling system activates specifically the high-pressure injection system and when it turns on it begins dumping a thousand per minute of cold water directly into the reactor core now this was very unexpected and confusing to busy we in a screw the plant had gone from a state they understood to one they very much didn't as soon as high-pressure injection kicked in and here's why it was so confusing to them they were watching the pressurizer and water level in the pressurizer was rising seeing the water level on the pressurizer rise told them that there was plenty of water in the system so they didn't understand why high pressure injection would need to be adding more water to the reactor core and so Fred Shaymin made the call to turn off the emergency core cooling system to the activate high pressure injection after only been running for two and a half minutes had he not had he left high pressure injection running misread this accident would've been completely avoided the plant would have been saved but he didn't we're now five minutes into the accident there's something at this point that's perplexing bill Z we though the water level and pressurizer is continuing to rise there's plenty of water in the system but the pressure of the primary coolant loop continues to fall now this is a problem because if the pressure and the primary loop falls low enough it'll begin to boil and if it begins to boil it won't be able to effectively cool the core he has a hunch about why this might be happening he suspects that the pilot-operated relief valve might be stuck open not having closed after the automated system opened it earlier and that's why the system is having trouble maintaining pressure and so he double-checks the pilot-operated relief valve indicator on the control panel and it shows closed just like you would expect just to make sure he has one of the operators in the control room check the outlet temperature of the pilot-operated relief valve outlet now to do this the reactor has the the operator has to get up from the control desk and walk around to the back of that instrument panel you see at the back of the room because this gauge is on the back side of that panel the operator reads out 228 degrees Fahrenheit about 108 Celsius and so bill Z moves on there's a problem with this decision though the plant operation manual indicates that any reading over 200 degrees Fahrenheit about 93 Celsius indicates an open pilot-operated relief valve and requires that the man block valve in front of it should be closed now at bill Z we closed the block valve head he followed procedure in this incident would have been a minor inconvenience the accident sequence wouldn't have continued but he doesn't he leaves the block valve open we're now six minutes in five minutes later at 4:11 in the morning another alarm goes off this one indicates that the sump is filling up and the sump is a pit in the the floor of the control room that collects any water that might leak or be vented anywhere in the system there's potential that any water vented from nuclear reactor obviously has some nuclear contamination so they have to check it and clean it before they can dispose of this water in this case what's happening is that so much water has been released from the primary coolant loop that it is filled the drain tank on the floor of the containment building the drain tank has started overflowing into the sump now enough water in the sump to fill it up is a very clear indication that the system has a substantial leak but the operating crew ignore it the core is in serious trouble at this point but the operators are not done just after 5:00 in the morning the floor of the control room starts to rumble it's subtle at first but it quickly becomes impossible to ignore what's happening is the primary coolant pumps of the reactor are vibrating because they're pushing around steam in addition to the water that they were designed to pump and that steam is causing significant turbulence and vibration now they know what their training says to do when this happens in order to keep the very large very expensive pumps from vibrating themselves to pieces or causing a coolant leak they're supposed to shut them down now they hold off doing this as long as they can but 15 minutes in bills Ely and his crew can stand it no longer so they shut off the first set of pumps this helps for a while but 30 minutes later the vibration grows so intense that they close the other set it's now 544 in the morning and a nuclear reactor that was running at 97 percent of capacity less than two hours earlier now has no coolant at all moving through its core doesn't take very long for the effects of no circulation to make themselves known about six morning for cisely two hours into the accident a radiation alarm like this one in the containment building goes off now radiation alarm going off tells us a couple of things first you need to know that the uranium fuel in the reactor is contained in sealed zirconium rods preventing radioactive material from being absorbed in the water in the primary coolant loop so for radiation alarm to be going off one or more of the sealed fuel rods has to have been damaged and second if a fuel rod has been damaged it's almost certain that the water level on the reactor vessel has dropped below the top of the core now about this point Gary Miller the site manager of Three Mile Island essentially the the CEO of the plant finally makes it in hit the ground running in full crisis mode and almost as soon as he walks in the door he joins a conference call with Leland Rogers the site rep for reactor designer Babcock & Wilcox it's Leland Rogers job to know the design of the plant inside it out he has the schematics memorized you know he knows what valve is where he knows what each pipe does and as they're talking through what they know about the state of the plant Rodgers says they closed the block valve right the block valve the valve that bill Z we decided not to close earlier George Condor who was on the call from the control room of Three Mile Island unit to yells to someone else in the control room to ask is the block valve shut and the answer comes back yeah we shut it and so at 6:20 2:00 in the morning the block valve was finally closed thanks to Leland Rogers question the leak in the system sealed now this would been the right thing to do 20 minutes into the accident sequence but doing it now actually made things worse with all of the coolant pumps turned off closing the block valve eliminated the only source of cooling that the poor reactor had left boiling coolant out of the block valve it's now a completely closed completely sealed system and in this state the heat in the core intensified very rapidly it took 8 minutes for the top of the core to collapse subsequent calculations was show that by 7:00 in the morning the core was 2/3 uncovered and temperatures in the hottest part of the core were 4,000 degrees Fahrenheit about 2,200 degrees Celsius hot enough not only to melt the cladding around the fuel but the uranium in the fuel itself at 7:20 in the morning the radiation alarm in the top of the containment building goes off indicating a reading of 800 rym per hour they give me some sense of scale of this if a if a Three Mile Island worker had been standing in an 800 rim per hour radiation field they would have gotten their full legal yearly dose of radiation exposure in 20 seconds so this is an intense radiation alarm the crew had largely been did not in denial about core damage after the first radiation alarm went off but this was the big one they couldn't ignore this one immediately after this alarm they finally try to turn the high-pressure injection pumps back on but they turn them off 18 minutes later because they're making the pressurizer level go back up it wasn't until 826 in the morning after the situation continued to worsen that they finally reenable high-pressure injection for good largely out of desperation I'm not sure what else to do it would take until 10:30 in the morning for the Corps to finally be covered with water again ending the accident sequence over the next few days there would be continued worry about a nuclear release at the plan and so they keep monitoring the situation on the ground and by flying helicopters with radiation monitors overhead but the redundant containment designed into the plant did its job and no significant radiation release would ever occur there would be public concern about a potential hydrogen explosion in the containment building from hydrogen released as the fuel rod cladding melted but that turned out to be overblown as well on Sunday April 1st four days after the accident US President Jimmy Carter and his wife Rosalynn would visit the plant to reassure the American public about the safety of nuclear power President Carter actually knew a thing or two about this because he was trained as a reactor operator by the US Navy he would later convene an investigatory Commission that would result in this report on the accident where a lot of the facts for this story came from Three Mile Island unit two would be written off as a total loss around 20 tons of melted uranium ended up in the bottom of the reactor vessel and another 10 tons blob together in the middle of the core this is what they found when they began the initial cleanup in 1983 you're looking at severed melted fuel rods that mended up at the bottom of the Rio the final cost of that initial cleanup was just over 1 billion dollars and it took 14 years through our island unit to is still standing in the middle of the susquehanna d-- fueled but not dismantled final cleanup couldn't take place until unit 1 was shut down and decommissioned something that finally happened just last year so what happened how did these four men miss so many signs along the way that their reactor was in the midst of a loss-of-coolant accident that's literally the primary accident type that reactor operators trained for and they missed it why didn't they just leave the emergency cool core cooling system on when it activated why didn't they close the block valve sooner were they not adequately trained did they not know what they were doing maybe we're looking at this the wrong way Sydney Decker's wonderful book the field guide to understanding human error notice the scare quotes there is an in-depth guide to investigating and understanding what happened when things go wrong in it he introduces the concept of first stories and second stories and the story I just told you of Three Mile Island is very much a first story of the Three Mile Island accident first stories focus on the humans in the story the decisions they made and what they should have done differently my first story will almost always lay the blame for anything that happens at the feet of the humans that made the decisions there's a couple of problems with us though in the form of biases that we all have the first is hindsight bias this is the phenomenon where when you review an event after its occurred and you know the outcome you exaggerate your own ability to have predicted and prevented the outcome sometimes called the I knew it all along effect a great example in the Three Mile Island story is well all that water in the sump had to be coming from somewhere I don't know anything about operating a nuclear reactor but I could have figured out there was a leak somewhere the second one is outcome bias once you know the full outcome of a situation you carry that full weight with you into evaluating every decision that led to it it makes you more willing to judge those decisions and more likely to judge them harsher good example here is that turning off the emergency core cooling system early in the accident is obviously a stupid decision when you know that the outcome is a partial meltdown but Fred seaman didn't know that when he made that decision focusing on what he did know is the first step to finding a second story in a second story human error is seen as the effect of systemic vulnerabilities deeper inside the organization not a result of bad decision-making or a failure to follow instructions so how do we get to a second story well first we need to work from the participants reality we need to dig into decisions from the perspective of the people that made those decisions we need to work to consider the messy reality that they were facing when they made it not the clean room conditions that we get from our hindsight second we need to assume positive intent we need to have the belief that everybody involved made the best decisions that they can make given the information that they had and so let's see if we can let's see if we can find some second stories from Three Mile Island let's start early in the accident sequence why would Fred Shaymin make the decision to turn off the emergency core cooling system five minutes into the accident and we'll find our answer in the pressurizer in his deposition of the presidential inquiry Fred Siemens says that he turned off the emergency core cooling system because it was causing the water level in the pressurizer to rise and he was afraid that the pressurizer was going to go solid now what does that mean what does it mean for the pressurizer to go solid we'll remember that one of the pressurizes purposes is to absorb pressure shocks in the system via the bubble of steam at the top letting the pressurizer go solid is to allow the pressurizer to fill all the way up with water eliminating its shock absorbing capability so obviously I'm he's concerned about losing that shock absorbing capability but what about the reactor core that's melting down why would he be concerned about shock absorption when there's a bigger problem happening the answer to that question goes all the way back to Admiral Hyman rickover and the nuclear Navy because it turns out that bill ze Fred Shima edy Frederick and Craig Faust were all trained in the operation of nuclear reactors by the US Navy and the naval reactor training created by Admiral rickover had drilled into these men that keeping the pressurizer from going solid was the single most important thing for a reactor operator to focus on and in a sixty zero submarine that made a lot of sense a 60s era submarine reactor produced 12 megawatts of thermal energy Three Mile Island unit two in order to produce its 906 megawatts of electricity has to produce two thousand eight hundred and forty-one megawatts of thermal energy simply cuz of inefficiencies and loss in the system like I mentioned earlier when you scram a reactor the primary heat production stops immediately but they're still decay heat being produced and it takes a while for that decay heat to go away in a submarine reactor that energy is trivial around 780 kilowatts not enough to cause any problems not enough to melt the fuel a power reactor however is still producing a hundred and eighty-five megawatts of heat at shutdown more than enough to melt the fuel in a submarine a water hammer with no shock absorption is literally the worst case scenario because it can result in a loss of propulsion and because of that a lost ship carrying that mentality into the operation of a power reactor were far worse things are possible it turns out to be a huge systemic vulnerability one that was completely unreal eyes before Three Mile Island nobody realized how much of their nuclear training the u.s. reactor operators are relying on from the Navy and the knowledge that they carried over there the assumption that they had carried over from their naval training and so Fred Shaymin faced with a rising pressurizer inferred that the system was clearly full of water already and allowing the emergency core cooling system to continue high pressure injection would overfill the system risking a full pressurizer and so in an effort to keep the reactors safe he turned off high-pressure injection it's a decision that makes perfect sense when you consider where he was coming from let's look at another decision why did bill Z we not close the block valve when he first checked the pilot-operated relief valve if you remember the reported outlet temperature was 228 degrees Fahrenheit and procedures called for the block valve to be closed for any reading over 200 degrees but it turns out that the pilot operator relief valve at Three Mile Island two had been leaking ever so slightly since the day the reactor went into operation and they had made the decision because it was a slight leak and because it was on the nuclear side of the plant not on the secondary loop that they wouldn't fix it until the first refueling shut down several years in the future that leak led to regularly observed temperatures over 200 degrees Fahrenheit at the pilot-operated relief valve and so in this split-second that bill Z we had to make that decision key reason that given the baseline the temperature was always over 200 degrees Fahrenheit as far as he could remember and that the pilot-operated relief valve had recently been open venting very hot water 228 degrees Fahrenheit wasn't all that unreasonable of a temperature once more bill ZB was looking at a clear indicator on the control panel of the pilot operator relief valve position and it turned red when the valve open and green again when the valve closed the only problem with that is that that indicator wasn't actually an indicator of valve position it was an indicator of computer commands the light turning red indicated that the computer had commanded the valve to open turning green that the computer had commanded the valve to close there was no indication on the control panel of the actual position of the valve the only way to know was to infer from the temperature of the outlet and so bill zealey assimilating all the information he had at his disposal and considering the almost full pressurizer left the block valve open so that the pilot operated relief valve could still respond in case there was another significant pressure spike he left the block valve open to try to keep the reactor safe again makes perfect sense when you consider where he was working from one more and this was a quick one why did the crew not respond when the sump alarm went off how could they not know that they had a coolant leak it seems obvious right well answer that one's really simple they just never got the alarm the control room relays alarms to the reactor operators in two ways one is this bank of lights on the operating panel of the reactor and there's another one just like it on the other side of the room that you can't see in this photo there's a few problems with these lights number one they're noisy there's 600 of them in total and when the and when the reactor is operating normally when everything's going just fine 40 to 50 of these lights will be lit up that's not like anybody's exception monitoring system second there's no rhyme or reason to their placement one of the most important alarm lights the alarm for coolant pressure is right next to a light that indicates that an elevator is stuck in the containment building and third they don't indicate chronology you can't tell by looking at them the order that they went off or what's new since the last time you looked at the lights that one they actually had an answer for they had an alarm printer every time the alarm goes off it gets sent to the printer so there's a log of alarms the only problem is that that printer runs on a 300 baud serial connection that's really really slow less than an hour into the accident more than 100 alarm lights are illuminated exponentially more alarms than that have actually gone off there have been multiple alarms for the same lines and it would take this humble little printer two-and-a-half hours to get caught up on printing alarms there is no way that the operators can comprehend the flood of information that's being directed at them so they just missed the sump alarm they never see it the operating crew at Three Mile Island looks up quite a bit different now that we've dug into the second story is buying their decisions right so how do we implement this idea of first and second stories in our own teams well dr. Decker has some helpful advice for us first and we're trying to figure out why something went wrong we agree on a baseline rule that human error is never the cause human error is always a symptom of some other lot underlying systemic problem and problems so blaming an issue on human error just keeps us from figuring out what actually went wrong a good way of helping to frame the conversation in these terms is to ask what is responsible for an outcome not whose fault it is second understand why it made sense the people on your team don't come to work intending to do a bad job chances are when they make a decision that you don't understand or they miss something obvious there's a good reason they did what they did take the time to understand why it made sense because if it made sense to them it'll make sense to somebody else later and finally and probably most importantly seek forward accountability not backwards our instinct when things go wrong is often to find who is responsible and punish them when we try to move our organizations away from blame and towards finding second stories one of the most common objections is but what about holding people accountable if you've rolled out blameless post-mortems in your organization you've almost certainly run into this objection but one of the reasons that blameless post-mortems are so important is that removing punishment actually frees people up to candidly share their stories of what happened so you can learn from them instead of sweeping them under the rug for word accountability though goes even deeper than this it turns out that the act of telling the story of what happened giving their account and owning their part in it is generally all of the accountability that a well-intentioned person needs to improve give them the opportunity to learn from what just happened backward accountability looks to blame somebody for past events and forward accountability seeks to help people focus on the work necessary for change and improvement going forward beauty of this technique is that it's so broadly applicable there's always a second story if you're willing to do the work to find it it works when somebody drops the production database when the team misses an important deadline when a key team member chooses to leave or even when sales misses their quarterly target it requires honesty and building trust that's worth it because finding the second story is such a powerful way for your team to grow and improve and it allows you to treat your teammates with the humanity that they deserve it turns out who destroyed Three Mile Island isn't a fair question at all what destroyed Three Mile Island is much more helpful and thankfully that's the question that the President's Commission asked notice the subtitle of their report this report is full of second stories and the second stories revealed weaknesses in reactor design and operator training around the world by getting past human error to the real causes the President's Commission made the world a safer place if you take the time to find the second stories for everything not just your outages you'll not only address the things that affects your reliability quality and delivery speed you'll make your organization a safer place for the people who work there and that safety will empower them to do their best work best of luck Wow thorough thorough look at Three Mile Island it's great we have a few questions and we're running a little behind in time so that's great top of the line Three Mile Island was designed in the 60s and a half a century has passed we learned lessons so why do fatal design mistakes like that of the three 737 max still happen in your opinion interestingly enough the cause of the seven three-minute 737 max issue is a 60 year old design decision it dates back to the same eras Three Mile Island you're kidding nope the MKS system that everybody talks about just exacerbates this design flaw but the 737 has always had a flaw where the elevator can get stuck because of aerodynamic load I mean that's so frustrating so disappointing the counterpoint to this is if you look at the Boeing triple7 which is a modern airliner it's yet to have a loss of life accident so I think we probably have learned these lessons and the 737 max is a very complicated story hmm okay well if you'd like to talk more on that with Nikolas the party indeed up next what would be your one and most important message from this story for people as leaders create safety on your teams it's it's really easy to get pulled into blame culture especially when you have an executive team breathing down your neck wanting to know why a certain outage happened you need to create a safe environment for your team to make mistakes and learn from them otherwise they'll start sweeping those mistakes under the rug and your system will be less resilient because of it okay great isn't it assuming everyone having done their and idealistic and utopistic approach what about people not learning from mistakes or not paying attention that's different absolutely it is it's it comes from a place of assuming that you have done good hiring and you've done adequate performance management and so the people that aren't coming to work every day are not still at your company if you're allowing those people to stay around it's like Sean was talking about earlier your whole team can spot that you've got people on the team that aren't coming to work wanting to do their best work every day so I think that's actually on you as a manager to make sure that that statement is true it's a bit of a different issue it's yeah and finally was the moderator matters separate like graphite or the same as the cooling material like heavy water I'm smiling because somebody knows they're nuclear reactors Three Mile Island was a water moderated reactor and so the cadmium control rods were not the moderator it was separate so if you're coming from Chernobyl where you've got a graphite moderator and graphite control rods it's a different situation through Milan actually has a negative reactivity coefficient so if you drop all of the water out of the reactor reactivity will go down Chernobyl when you drop all the water out of the reactor it goes up I can't I can't pretend that this is not too technical for me but I'm blown away by your answer and I think the laughter was more about the last question which is can I steal your presentation to encourage this at my company and there will be videos of this available and I would love it if you show it to people at your company and if you do I would love to hear what comes from it there you go well thank you so much thanks so much for having me I have this for you wonderful thank you [Applause]

Channel: CraftHub Events

Views: 1,538

Rating: 4.8571429 out of 5

Keywords: stretchcon, leadership, psychological safety

Id: YoeUSzDkVXc

Length: 41min 13sec (2473 seconds)

Published: Wed Feb 19 2020