Don't Look Back in Anger: Wildman Whitehouse and the Great Failure of 1858

all right thank you every no yes okay thank you everybody for joining us for this final presentation before we move into conference closing mode Lily Ryan our speaker is a historian systems engineer and privacy activist she consults on identity and access management systems security architecture and software delivery gives public lectures on ethics and the security mindset and provides what she describes as paranoia as a service she's also always happy to talk about greyhounds but first up Lily's going to talk to us about Wildmon white house and the great failure of 1858 hi everyone how well can you hear me yep lots of thumbs up cool firstly I would like to acknowledge the gadigal of the eora nation they're the traditional custodians of this land and I would like to pay my respects to their elders both past and present so thank you for coming to my talk I can promise you lots of drama and backstabbing and poetry and lessons but before I do that I wanted to talk about my slide background which is I know it's a little hard to see but I want to do anyway can everyone up the back seat still kind of swirly yeah a bit annoying all right it'll look better in post so this this is a reproduction of some authentic Victorian era wallpaper that would probably have been in a rich person's house around the time that my story takes place and what I really like about this pattern is that if you entered a house that had it your chances of dying would increase exponentially and this is because this pattern and a lot of others like it were covered in arsenic arsenic would give you really vibrant colors so a lot of rich people who could afford vibrant colors plastered their homes in it and I've color altered this one for safety reasons so this sample is from a book called the shadows of the former walls of death which pretty cool it's published in 1874 and there are only four copies of this book left because most of them were destroyed any interest of public health and I really hope that people who digitized this copy are okay anyway this doesn't really have anything much to do with this story that we're going to hear today but if you're wondering about a lot of the behavior that went on during this period just remember that a lot of people were literally inhaling poison with their breakfast okay moving on my name is Lily Ryan and my career so far has existed at various points on the spectrum between historian and hacker and I really like to talk about all the weird things that went down in the past so that we can learn something from them and I think that being able to reflect on the failures and mistakes that we've made in the past helps us to build better teams in the future and to do better work and the story that I'm going to tell you today is one of the biggest failures in tech history I think that successful reflection comes in two halves firstly we want to understand how we got to where we are and secondly we want to apply those lessons to where we are going next the problem is when a team only does the first half of this which is what can often happen you think about what you did don't take any lessons from it and then do it again and you can often get away with this quite a long time if things are actually going alright if but when things go wrong applying these lessons to future actions is what makes it really important and it's definitely the only thing that can pull you out of a bad cycle unless you're very very lucky and it's usually when we fail that our projects are thrust into this spotlight just at that moment where we really want to hide but let's talk about success for a bit one of my favorite things about the successful release of a product is the go live celebrations it doesn't matter if this comes in the form of an office party or if the team slink to a bar after work or if they even get to have one at all because they're not just sitting there hoping and praying that the bill doesn't fail probably one of the biggest go live parties on record happened in 1858 after the completion of the first transatlantic Telegraph cable and I wanted to pause for a little bit to talk about what a telegraph is because it is so old that it is not vulnerable to meltdown or Spectre so most people here probably never used one so that's a picture of a telegraph receiver basically Telegraph's were the first Internet in the Victorian era people started to send messages to each other along copper wire using basic electrical pulses and mostly they'd use Morse code long and short pulses indicating in different combinations different letters so you'd write your message down on paper you take it to the telegraph office they would dip that in in Morse code for you to the direction you wanted it to go somebody would listen to it at the other end dots and dashes write it all down on paper again put that in an envelope and send it to your actual recipient that's how that worked it was better than normal mail because electricity can travel much faster than trains or horses which is good so telegraph messages got to their recipients a lot faster and by 1858 both North America and Britain had extensive internal Telegraph networks this was how most people did business but to send a message from Britain to North America or vice versa during this time would take several weeks because you had to write it down and physically put it on a ship to wait for that ship to sail across the sea and that was why when people heard that they were thinking about building a telegraph cable that went across the ocean connecting the two continents pretty much everyone went absolutely wild for it so the 1858 Telegraph cables completion heralded one of the most spectacular go-live parties in history both sides of the Atlantic people fired cannon and they let off fireworks and they danced in the streets and they wrote reams and reams of some of the worst poetry I have ever here is tis done the angry sea consents the nations stand no more apart with class bed hands the continents feel the throbbing of each other's hearts speed speed the cable let it run a loving girdle round the earth till all the nations meet the Sun shall be as brothers of one half I did not write that myself there is heaps more where that came from people really really love this cable like a lot they loved it so much that they were still celebrating a week later but the weeks apart II hit a snag when the cool new Telegraph cable went completely dead so these days when a big release goes bad we usually have some strategies in place to work around this we don't always assume that things are going to work first time we hope they will but we plan for the worst we might have thought of a way to roll the changes back how we plan a soft launch just to make sure that things are really working but the telegraph guys hadn't thought of anything except success there was no fallback plan so when this failed there was just a whole lot of rage we all like to think that the software that we build the stuff we build is going to change the world and quite often it does we change people's lives all the time by rolling out new products to make sure that the people who use and trust us with our business can do what they need to do so if this is they need to make an insurance claim faster when it really matters or they want to split bill payments at a restaurant or get live updates when there's bad weather coming all those kinds of things actually do make a difference but the transatlantic telegraph cable changed the world in a really huge history-making way like the invention of the car like the Internet it was that kind of scale this kind of makes all the fireworks understandable from the Victorian perspective I mean I don't find that poetry understandable but we can put that part aside I also hope though that we'll help you understand just how dramatic things got with this amazing cable just totally died and when your entire you know user base is the complete population of North America and Europe and the first person who wants to use it is Queen Victoria you really feel the heat when things go wrong it's always a hard time when a project fails and a lot of us have experienced this at one time or another and if you haven't you almost certainly will know you will no matter how excellent your team is or how good your idea is not everything's going to go smoothly but failures or alts are really valuable and for me they've been the places where I've learned the most about myself I've learned where my strengths are as a team player my technical strengths the ability of me and my team to bounce back when things go wrong ways that I can improve I think that reflecting on your work is often richer after a failure asking yourself in your team what did we learn gives us really useful fuel for moving on unfortunately for the transatlantic Telegraph cable team 1858 was only 15 years after the first computer program was written and it was 45 years before the first computer was built and it was a time when the phrase agile retrospective would only ever be used to describe someone looking over their shoulder over there running away from the bad guys so they really didn't have any models to learn from but this doesn't mean that we still can't learn something from the project today because in many ways this is the cornerstone of every modern delivery project and the key person we're going to learn all this from is this guy this is dr. Edmund orange Wildmon Whitehouse sideburns yeah so this guy in addition to having like the best name and the wildest facial hair on his entire team he was the chief electrician of this project which meant that he was basically the tech lead he was also the main architect of its demise White House had a history as a doctor and and an inventor now when I say inventor when I was doing the research for this I found a pattern that he'd lodged for a new kind of roller skate and he had also dabbled in telegraphy as one of his many many side projects and he'd patented a number of deviations on Telegraph technology which was basically just like forking other people's repos and changing a line and committing that and saying that the whole thing was your work anyway all of this his github profile everything this had impressed this guy whose name was Cyrus field he was an American businessman who was providing all the money for this Telegraph cable thing so despite having no experience with actually implementing any kind of telegraph system he got appointed as chief electrician to the Atlantic Telegraph Company and the main problem with the transatlantic Telegraph cable was that nobody was really sure if it could actually be done we've had Telegraph's for a while but not never anything that went underwater for such a long way this is a cable that had to be over 3,000 kilometers long it's a long cable this kind of felt about as impossible as building a space elevator to the moon and at the time scientists were still discovering some of the really basic laws of physics that we take for granted today so it wasn't just we know this is technically achievable we didn't even really know how electricity worked at all electric currents were being used for a lot of stuff in 1858 but it still wasn't really well understood and this meant that a lot of the ideas that people had about how to run a current along a cable that was 3,000 kilometers long under the sea will literally just guesses and the only way to prove or disprove that guess was to build the thing which was going to be extremely expensive fortunately Cyrus field with very rich and he had absolutely no idea how impossible it was this was all supposed to be so he went ahead and kind of Elon Musk Ashleigh just hired a bunch of people and got it done one of the other people that field hired was this guy this was a young man his name William Thompson he had fewer cool names and fewer credentials than Whitehouse but he was also a keen physicist he had a lot of interesting ideas and theories about how electrical currents worked and field had discovered Thompson after a few papers and proposals that he'd written for how to build this telegraph cable and he liked his spirit so he hired him and when White House met Thompson it was rivalry at first sight they hated each other White House was used to being listened to and Thompson was younger than him and questioning his authority and his he had no measure of respect with the board his facial hair wasn't nearly as majestic I mean look later in life he would grow a beard that was so magnificent that it would make a hipster weep to see it but at the time anyway the main point of contention at this point was about how the cables themselves should be designed so that they could hold up to the distance that they had to cover and still carry a signal Whitehouse argued that the copper that was used to conduct a signal should have a small diameter but the voltage that was used to push the signal along or carry a single across the ocean should be really strong kind of like yelling the current between the distance of the two continents and he also decided that the cable they need to be really heavy so that it would sink to the ocean floor Thompson disagreed with him about pretty much every aspect of the cables design but Whitehouse was confident he was right and he had a better beard and he refused to hear any of Thompson's ideas and there also wasn't much time to persuade anybody that Thompson's ideas had merit because field and the board wanted this project to be delivered yesterday he'd gone and gotten the investors all excited so they wanted to see it England and America were really excited and wanted to see it there are a lot of people waiting so they'd had to get it done they rushed out the cable they used White House's design they stuck the two halves of it on two different ships and one of these ships was going to spool the cable out behind it from Newfoundland and the other ship was going to do the same from Western Ireland and the idea was they meet in the middle and join up the cable and hey there's a working telegraph line this picture shows how they loaded the cable onto the ship's they basically made a little bridge to roll them along because the cables weighed about an imperial ton per mile so they couldn't just be carried really heavy and unfortunately Whitehouse's health wasn't great at the time that this project was going on so that meant that it was up to Thompson to actually get on the ship that was going to carry Whitehouse's cable out across the sea and spool it out across the Atlantic so the younger man had to actually oversee the project while Whitehouse remained on shore and waited and probably schemed and did this a lot I don't know the first attempt out at sea was a total disaster after about three days this incredibly expensive high-tech cable snapped under its own weight and sank to the bottom of the ocean I was lost whoops so field went and made his excuses to all the investors and it got some more money to make a replacement cable and this is what they're doing in this picture there covering the cable with layers of gutta-percha which is a choice that is tree sap it comes from a tree it was molded soft and then it hardened like plastic over time and this was before plastic had been invented so the Victorians used it for everything to the point where the tree itself nearly went extinct but then plastic was invented so trees were saved but then plastic anyway anyway after about a year of all this very expensive cable making the new cable was ready and they loaded it onto the ships and they set out to see again the idea the second time around was that the two ships would start in the middle of the Atlantic and then go towards the coast from there it took them three more tries meeting in the middle and sailing out again and every single time the cable snapped it's costing a lot of money in taking a lot of time and in one of their attempts the cable was almost completely destroyed by a passing whale which you can see illustrated here I like to call this the original fail whale [Applause] but by the fourth attempt they had made it the whole way across the ocean without the cable breaking at all and it was connected to the local Telegraph network at both ends and it was declared ready for business and this was the point where the entire and Western world went completely wild and they kicked off that go live party that we talked about earlier and there was all the poetry that was then so that this was also where wildmen Whitehouse sealed his own fate he and Thompson had argued of how the cable should be designed and now they fought over how it should be operated as I said earlier Whitehouse had designed the system to generate a very strong electrical current along the entire length of the line and Thompson had managed to get him to put some brakes on the voltage but and the system wasn't generating nearly as much power as it was able to do but as the week's went on the demand for telegraph you screw and grew messages backed off at both ends of the line so White House secretly increased the voltage as high as he could make it in prod his disaster but his expectation was that the stronger the current the faster and clearer the messages would get to the other line yeah the reality was that this amount of voltage completely fried the extremely expensive cables and he had actually overwhelmed them with power which meant they shorted out into the sea and some of the nearby sharks probably had boiled fish for dinner so the upshot of White House's secret tinkering with the system meant that this famous cable was completely dead within about three weeks of going live and the world was very angry and so was Cyrus field and so was the entire board of the Atlantic Telegraph Company because they had literally one job it was in their name and they haven't managed to do it this is pretty much as big a disaster as if the entire internet had gone down and they didn't know if they were going to get it back so when a project fails this spectacularly and when this much money is involved there's usually a lot of public head scratching and soul-searching about what went wrong and where it went wrong and questions get asked and as you probably expect the best companies will usually take some time to step back and reflect on what's happened to try and see what lessons they can learn to make it better and while most of the board of the Atlantic Telegraph company were busy doing this the one man who wasn't was Edmund orange wildmen Whitehouse he had skipped right across this reflection stage and straight into the deep pit Oh blame which is unfortunately very easy to fall into he didn't want to reflect because he knew whose fault it was literally everyone else's except him the public was so angry with the failure of this cable that it demanded and it got a public inquiry into the whole affair and it was one of the first project retrospectives ever to take place and it took place in front of an entire country during this inquiry William Thompson testified to all of this advice that had been ignored by the chief electrician and other stepped in to testify to the fact that they had seen the chief electrician messing with the voltage levels and the cable in the weeks after its completion which was completely against what the board had decided to do and against all known scientific advice so getting feedback like that it's a little bit of a blow especially when you thought you were doing a good job and getting feedback like that publicly is even harder and it's natural to feel defensive when this happens and honestly public feedback about a specific individual is generally not a great idea but that's what happened and it's usually not a great idea because it often leads to situations like what happened after this so White House hadn't entered into this inquiry with an open mind but honestly he hadn't entered into the entire project with an open mind he was a self-righteous Victorian era gentleman scientist and a doctor and a member of the Royal Society and he was used to being treated as though this meant he could do whatever he wanted unfortunately for White House there was no software development community in 1858 to suggest to him how he might tactfully handle this kind of situation so he did what an angry self-righteous Victorian era gentleman scientist would do he wrote a pamphlet and he sent it to all the papers and he had it published in them this pamphlet was called the Atlantic Telegraph but a better title might have been everyone is wrong except me the best scientists in the world I could describe it but I'm gonna let White House speak for himself here because it's better hmm the charges leveled against my ignorant on suspicion are three three of the most drama tree and detrimental not merely to the fame of a public but to the character of private man in the one case however error is but human in the other disgrace from the first my advice and wishes as projector had been disregarded and overruled and as an officer I had constantly been thought and obstructed in my operations I do not shrink therefore from the avowal that accustomed to such treatment and aware of the incompetence and division of counsel existing in the board I determined to do my best on my own responsibility to save the enterprise from destruction I make no appeal add Misericordia m-- I seek for no sympathy on scientific grounds sufficient for me that I have been identified and from the first with that prodigy of this age which may become a new starting point in history till the end of time a great responsibility rests upon those who have in any way contributed to the failure of this enterprise but for my own part I can safely say that neither zeal labor caution more anxiety was wanting upon the part of Edmund Orange woman white house well institution now the monster Eden it may surprise you to learn that publishing this defensive pamphlet did not really convince it anyway that the failure of the Telegraph cable was not his fault at any rate he convinced a lot of people that regardless of whether oh no it was his fault they didn't want to work with that guy and after this he was um asked to be chief electrician of the Atlantic Telegraph Company which was fine by him because he didn't want to work with them anyway any fold his arms and he stormed off to his room or something unfortunately I've seen many projects even in this day and age and this way and I'm sure look if you have to and this is a real shame because in 1858 it was already obvious that this was a really bad way to run a team there are a few things that I want to highlight out of this whole telegraph cable affair that I think that they're in really really really important mostly open-mindedness among your team is crucial to success but it is also crucial to good failure constructive reflection cannot happen if there are members of your team who aren't open to being challenged or having their minds changed even by people who are considered juju to them secondly if the project gets to the point that things are really bad feedback should be given sensitively with tact and most importantly privately people are naturally going to get defensive if they think that they're being attacked and especially if they think that they are having their value challenged in public it's not a good time thirdly in order to be effective retrospective should always follow the prime directive and she's here regardless of what we discover we understand and truly believe that everyone did the best job they could given what they knew at the time their skills and abilities the resources available and the situation at hand having a framework like this creates a safe space for reflection what people can openly discuss failures without fear of blame or without fear of being blamed it's also a good reminder for everybody there that blame is not appropriate so you can even like you can even print it out and put it on the wall before you do one of these I found that really helpful finally there's no room for hero behavior on a strong team I think that the story of the first transatlantic Telegraph cable would probably have been very different if that team did not include Wildmon Whitehouse while there were definitely problems with the way that the inquiry after the failure of the cable was held was handled they they may not have even needed an inquiry if he hadn't been involved in the first place and they'd appointed someone else one of the strongest findings of this inquiry was that White House's behavior was a real problem most particularly his inability to listen to anybody else and his tendency to run off on his own and make crucial changes to the project without telling anyone else so we've never met anybody like that in a development team occasionally we'll get people in teams who want to be seen as heroes because our culture often romanticizes this idea of someone who swoops in and saves the world all by themselves and this is really cool if you're Batman but if you've ever watched a Batman movie you you see how many innocent people get their cars smashed with big chunks of concrete and have their legs broken when he's chasing after a bad guy they're just standing there and the same thing happens with heroes on projects they might get some personal glory but team members who ended up with bits of smashed building crushing their parked cars aren't going to remember that person fondly and it probably won't want to work with them again so this makes for a terrible team environment and we all know that unhappy teams don't do good work here is a real quick recap of all of that in case you missed it open-mindedness feedback sensitive remember the prime directive and there's no room for Heroes so with the moral of this story over you may be wondering what happened to the Telegraph cable project after the dramatic pamphlet incident you'll be pleased to learn that the Atlantic Telegraph Company learned something about teamwork from their first iteration they point in William Thompson is the chief electrician next this was great because in addition to having far more solid and reasonable ideas about physics than his predecessor he was also delightful to work with and he knew how to treat his team with respect and at this point the Western world's demand for a transatlantic Telegraph cable was so large and the Atlantic Howe and the Atlantic Telegraph Company had so much experience in it like laying the cable at this point that they didn't actually have that much trouble getting money to try to try again and this time they built the cables the exact opposite of white Hatteras they were large in diameter not quite so heavy so that there could be a little flexible and the sea currents and they hopefully wouldn't cause themselves to snap and they used a comparatively tiny amount of voltage to power them and when they went out to sea the first time the cable snapped again but because they've been through this so many times by now Thompson and his crew knew enough that they could actually go and retrieve it from the ocean floor and repair it and they went back for a second cable and by the time that second cable had reached both coasts both coasts they had hooked up the repaired one too so now there were two working telegraph lines across the Atlantic and they both work and after they had tested it to make sure that it really worked the whole Western world had another wild go-live party and eventually they laid on to see telephone lines and even more eventually they were undersea fiber-optic cables and we all lived happily after in an interconnected world the end I'm quickly as an aside in a shameless self-promotion if you're interested in getting a scarf or a travel mug with this Ossining wallpaper pattern on it this is a link to that pattern on my red bubble shop if you're interested you don't have to go back to Twitter yeah hi sorry where the bananas I hate them it was the longest I did that you think they would have lost it that long I could have I didn't bring my makey makey sorry for those who don't know last time I gave this talk I wired two bananas up to my computer and use them as my slide click is instead of this thing as you do anyone else no yes I think so but not as much as the first time I think that one of those things they did upon reflection was not right that again there's still bad poetry out there but um you can look that up on your own time yeah lots of like round repeating verses you could sing it over the top of one another um yes um oh good question [Music] okay what year was it complete someone's googled bethe 1865 and 1866 for the two yep okay but this was 1858 yeah so I don't know if it was actually do you mean like complete the first time or the second time the actual second one yes okay sure all right anyone else yes do I know if one of the first Telegraph's that was ever sent was to sent to instruct the Americans to arrest someone who was fleeing England is that true good question I don't know but I have a whole other thing about the first time they ever caught a criminal who was trying to escape another city on Telegraph inside England that was really cool because when they arrested the guy at the other end and he'd left Scotland or something he'd murdered someone back in the day used to be able to just change your clothes and get on a train and then get off in London and no one would know who you were and they'd never find you um but anyway they realized the cops it's gotten realized that they could actually send a message that was faster than the Train so they messaged the police in London and said hey there's this guy he's done this thing kind of looks like this probably changed into these clothes he's on this train and the cops waiting for him at the other end and he's just like completely unfair to that guy did not factor that into his motor plan at all but yeah that was the first time that happened yes it's the actual arsenic in the pattern yeah but the books the books had actual samples in them yeah yeah the books were actually sampled and there are pictures of them digitizing this one well they're all like gloves and not just the normal archivist gloves but it's like hazmat suit and gloves and the whole thing like I don't know how they did this and got such clean images because I would have been shaking like yeah anyway they're really cool if you want to you know buy one yeah oh you actually do okay sure anyone else [Music] um they were the first to that work yeah Oh have you seen those maps of the undersea fiber optic cables oh they're really cool I'll find one and tweet it out later but um yeah they're amazing just to see how many there are going across the sea sorry which geography publishes those tana okay I'll find a map find a link to the map afterwards too yeah work that out but it's really cool see how many there are that's them that's the public ones and not the secret ones that the government doesn't want you know yeah okay I think we're out of time and there's often noon tea and so you should race and get scones and cupcakes before everyone else gets them [Music] [Applause]

Channel: LinuxConfAu 2018 - Sydney, Australia

Views: 929

Rating: 4.7714286 out of 5

Keywords: lca, lca2018, #linux.conf.au#linux#foss#opensource, LillyRyan

Id: xkRuzrIxvO0

Length: 36min 8sec (2168 seconds)

Published: Fri Jan 26 2018