The Error of Our Ways • Kevlin Henney • GOTO 2016

[Music] right thank you good afternoon so my parents gave me a conveniently internet unique name do this for your children okay I did it for my children yes I did google their names when they were born just to check that the first last name combination was unique and it has paid off so my older boy now has his own domain name you know so you know that moment was like 14 years in the making now normally what I talk about I worked myself I talk about kind of proper curly braket nerdy stuff normally talk about software architecture design patterns development practices very code focused stuff talked a lot about test-driven development talked with a lot about programming paradigms so I'm not going to talk about any of those today well actually I am a little bit but one of the other things that I seem to be known for is a fondness for pictures of software failures this is one of the kind of classics for me this is Madrid Airport terminal 1 2006 October 2006 and it's it's one of these things that it's fascinated me for a while we have and software developers we are the producers of the largest amount of installation art on the planet and so you know with this one I remember specifically with this one that's why I remember the dates my youngest son was only nine months old at this point nine ten months two-year-old and I pretty much threw him at my wife while I got my camera out to take a photograph of this because it's a huge error you know it's it's large enough I have to take two photographs of it because it goes all the way across and so you know I'm looking at this as a piece of art as something provocative this you know you sit there and try and work out the operating system no it's not windows because windows does not use the language of kernel except in one particular error message so this is most likely a Linux distro and you know I'm looking at this game all this is really interesting everybody else is focusing on the words free because that's what humans do five free packets I don't know what they are but I need to claim them how do I get them so you know so this is a constant source of fascination to it now there's an incidental feature because we live in political times we always have but we live in political times European Union only I'm going to come back to that one we get it on our screens we get these failures I take screenshots I'm going to come back to Facebook later as well but I love this your feedback will be used to improve Facebook thanks for taking the time travel to make a report I mean it's just like yeah I mean I did not just take the time I really took the time I took it all the way back if you look at that this starts telling you something about the nature of failure because look at that 31st of December 1969 yes I was I was alive but I probably wasn't programming much but if you look at that what's significant about that date it is the day before time began yeah the 1st of January in 1970 that is officially that when the universe began according to Unix so and one of my other favorites that you can take this one goes back about 10 years and tell it I'm using Mozilla at this point it's the Dutch rail website and there's something interesting about this java dot lang got a nullpointerexception because i was after a friend of mine lives in lived and eivin directs and i thought i'm going to go and visit him and i would like to get an earlier train so i will look for an earlier train there are no earlier trains there is only nullpointerexception you will look into the void and it will look straight back at you now what I find fascinating about this and this starts telling us something about the theme I want to explore in this talk which is the relationship of failure to people of software failures to people and here we start seeing something here even if you had not guessed by the completely uninhabited URL this is a java application when an vacation fails all of the efforts of the software architects the developers the testers to encapsulate their software all of those encapsulation decisions are rendered useless it's like when you drop something you see how it breaks you look at the fracture lines this breaks and it shows you what's really under the hood you've lost all encapsulation at this point we can tell how it's built importantly we can also tell how it's built because this is a really simple error to trap what we're learning here is something about the development practices it's really easy to test for this case so what does this tell us about the team were they under a lot of pressure or did they not know well they're not familiar with conventional practice you know big try block around everything do not let the user see this kind of thing you know give them a message in Dutch or in English saying I'm sorry things don't appear to be working right now but don't give it to them in Java and so there's a sort of an interesting thing here we're learning about the team what pressures were they under what knowledge do they have do they have an incentive to improve are they aware of these practices so it starts telling us something about the organization that creates it so from this tiny fracture we can start asking some questions it has now reached the point though that I no longer have to take photographs people now send me photographs I have I have 800 and this is this was said to be and another beautiful example when something breaks you see how it is built here it is using PHP we can see the object model you know it fractures on the floor and so we start learning about these things it has actually got to the point and things are retweeted to me so often but apparently these are now I'm now an adjective it's a careful in heavy screen and a week and a half ago at agile in the city Inn in Bristol where I live one of the guys one of the organizers John Clapham said Kevin there's one of those screens here in the in the venue I think we should take a picture of you standing in front of it I will tweet it and then you can retweet it so we can have this kind of like cyclic effect so yeah I'm absolutely fascinated by all of these things now some of these are obviously kind of just general configuration errors but they make a point about the impact all of these that I've just shown are it's some level minor inconvenience and we don't just see it out there in the world or on our browser we see it in our apps and sometimes it deals with money this is actually fascinating you cannot pay by paypal for orders over 0 pounds that's not very useful so this is the interesting thing these are all small they represent personal inconvenience the bugs that we have somebody laughs at them they take a photograph they tweet it to me and they represent a minor blip in somebody's daily experience but what happens if it gets bigger so I want to take you back to 2012 here because I apparently I can time travel according to Facebook which is a really neat skill I want to take you two back to 2012 I want to take you back to the 31st of July 2012 and the new york stock exchange new york stock exchange it was introducing a new system and a particular company night capital group decided to upgrade their servers in the week before in anticipation of the changeover on the 1st of august 9:30 Eastern Time Eastern summer time actually has daylight savings time they the market opened and they started trading nightcap group realized a 460 million u.s. dollar loss in 45 minutes not bad 10 million dollars a minute that's skill that's algorithmic trading doing the wrong thing at high speed okay it'sjust but actually the story is worse than that in some ways I mean let's just put it this way night capital is no longer with us in its form because it only had assets of about 400 million but they actually lost more in that day this is the amount that was outstanding after everything had been rolled back there's a kind of a agreement that software is you know if something arises through a software you roll things back this is as far as they were able to roll it back they lost billions in 45 minutes outstanding now there's a really good blog I'm going to quote from I've read a number of reports about this but I think Doug sevens won a couple of years back he takes a DevOps perspective he's looking at it from the perspective of how the system was configured but as a couple of points I want to make not only does he summarize it well there's a couple of other points he says the updates Mars was intended the system in question was intended to replace old unused code referred to as power peg functionality that night hadn't used in eight years when I'm talking about software architecture and software quality and the effect of this on agility my normal pitch is that dead code has a problem it is the dark matter of your system it shapes your system even though you can't really see it you don't notice it it is still there and it has a very strong exerts a very strong force on your system and the problem is there is no code that is truly dead okay it turns out all you need to do is make a small assumption a change of an assumption and then suddenly it's no longer dead it's zombie code it has come back to life and the zombie apocalypse costs money now by the way if you're looking for a safe haven the world is changing place if you're looking for a safe haven so I said in Bristol where I live the City Council actually has a zombie apocalypse plan okay you can google it zombie apocalypse plan City Council Bristol should probably get you there yeah it's a 2 3 page description of you know it's we actually have plans so I feel safe um but not too safe against this because this is zombie code so there's a point here there's dead code here in the system now this is the interesting thing dug 7 then observes why code that had been dead for eight years was still present in the code base is a mystery but that's not the point actually it is because what we're dealing with here is the the eight servers that were supposed to be upgraded there was an incorrect deployment one of them was not upgraded just for those of you who are interested in this it was manually they were manually updated okay so this is yeah this is okay so this should have been automated but seven out of eight were updated but not the eighth one so it's running the old code not the new code now an important point here is he says that's not the point no it is because when things like this happen they don't happen because of one thing or another they happen because of one thing and another had it just been an incorrect deployment and there been no dead code to bring back to life this problem would have mad that wouldn't have manifested itself had it been a correct deployment and the dead code had still been there there wouldn't have been a problem so the point is that these events occur because of a perfect storm of things so that is part of the point there these subtle things they're very difficult to recreate it's an assumption combined with another assumption combined with another so what was the assumption well I'm sure nobody's ever done this I have the code that was updated repurposed an old flag that was used to activate the power peg functionality you recycle an old flag you recycle an old bit of memory something to save you having to change the format of something that is established within an organization we can't change that that group says that we can't have more bytes in our headers and all the rest of it so something happens like this and that was the issue and then at that point boom first 45 minutes the market was open handled loads of parent orders four million transactions and against nearly 400 million shares and goodbye night capital so it turns out that this little coincidence had very very big knock-on effects but if you want something really explosive then one of my other favorite examples was the maiden flight of Ariane 5 in 1996 4th of June I remember correctly I mean Ariane 5 is a magnificent launcher it's a very powerful launcher for putting satellites into low to high orbit and its maiden flight looked a bit like this one of the world's most expensive fireworks that's 370 million dollars it was estimated that and that was about 38 39 seconds into flight so 10 million dollars a second yeah so yeah beat that night capital so the point here is they scattered pieces of the cluster mission over the forest of French Guiana which I'm pretty sure did not need that kind of topping and ultimately it turns out I dug through this it's kind of interesting I found the original Cove yeah I read about it in 1996 but I didn't find the original cones about 5 or 6 years ago my co-author Frank Bushman showed me something he said Kevin have you seen the code from the you know the Arianne thing and I said yeah I sometimes talk about it have you seen the code and he showed me a little piece of code and I said that's not the code and he said how can you be sure I said I can read it the guys who wrote this code they they don't read stuff like this in fact if you want to guess the language as you know some people are guessing but I the most perfect description came to me from Linda Riesling when I'd use this slider a couple of years back and it's a de Tran in other words it's a de but spoken with a very strong Fortran accent yeah okay and use it makes no use of any of the facilities of Ada whatsoever and and what you can see is even without knowing the language there are some things that might if you're if you're a software developer there's some things that might arouse your suspicions it's like hard coded numbers the same hard-coded numbers that obviously have some kind of 16-bit significance and also that's the same number in hexadecimal so not only they're hard coded but the programmer got so bored that they switched base just to make it more interesting okay the same structure the same checking what we're doing here is with Thresh holding we're checking that something's not too great if it's too large then we cut it at an upper limit it's saturated arithmetic what they're doing just for a bit of guidance here is V and H that's vertical and horizontal and what they're doing here is their threshold encoder flow to have integer to floating-point overflow in this case so this one is unguarded that one is unguarded it's not possible for Ariane four to exceed this now if you've been paying attention and I know it's the end of the day you might not have been you'll notice I'm talking about ariane for now this is reuse this is ariane for code sitting in ariane 5 this code doesn't even have a purpose in ariane 5 it's dead code it doesn't even have a role and the funniest thing is at 40 seconds it shuts off but at 37 seconds it overflows and causes a very spectacular spiral of the rocket and that point a really neat piece of software kicks in it's called the self-destruct system that's what the self-destruct system does you see up until 1996 I was used to think that self-destruct was a thing they did in Star Trek and other films but actually no real Rockets it turns out have a self-destruct system because it turns out their missiles and you really don't want those going out of control so at least the self-destruct system works so if you were the author of that code well done nice one how do you know your code works yeah so what we see here is there are some really interesting impacts what we're dealing with at this point is this had profound repercussions not simply a business not simply some small company it's not a ripple in the market this actually had some major implications for aerospace the owners of the cluster mission and a lot of stuff how easy is it to stop this stuff how can we address this or it turns out a recent paper makes a very very simple point we know we should be testing but do we have evidence simple testing can prevent most critical failures almost all catastrophic failure in an analysis of certain open source distributed system software almost all catastrophic failures are the result of incorrect handling of non fatal errors explicitly signaled in software in other words what we're dealing with is oh no something's gone wrong and the code that handles the oh no something's gone wrong goes wrong these are very untested paths I remember first coming across this phenomenon about 15 years ago there was a paper that explored the Linux kernel and the defects in the Linux kernel it turns out that most of the defects are on the error handling paths in other words oh no it's gone wrong but we've never tested it because they're unfamiliar paths what was interesting was one of the other results came out of this paper a majority of the production failures as in 3/4 can be reproduced by a unit test I mean that's that's that's really high people often say oh we don't really want to deal with the unit tests and stuff like that we want to test the whole system but a unit test it's very very modest it's very simple and what's interesting is that when we start taking this perspective and I realized as I was putting this talk together this fits in with a class that dates back to 1980 guy called Maya layman Maya mani layman he came up this classification and his focus was not on correctness his focus was on evolution technical debt as we would now consider it and what happens to code over time and he classified programs into three groupings s P and E s programs at s program one whose functionality is formally defined by and derived more from a specification if you do any coding carters or you look at most of the kind of training examples or teaching examples in computer science they mostly fall into this small problems that are well-defined and sometimes we dismiss them as toy problems except that how do you build a large software system you build a large software system out of small bits of software that's the point we build large things out of smaller things and smaller things out of even smaller things Turtles all the way down and that's the important idea is that if we can compose and that was his thesis we can compose more complex systems out of S programs that whose maintenance and lifecycle we can control but importantly that are unit testable that's the additional point P programs are larger programs what we're talking about is although we can define the problem the acceptability of a solution is determined by the environment in which it is embedded it turns out there are some things where P is for real-world problem in other words although I can characterize it to put it into software is is difficult the classic example of this is climate modeling and weather forecasting all I can show you I can tell you what the equations are actually I can't I'd have to look them up but we can look at what the equations are you can't reproduce those faithfully in a digital computer so therefore you have to approximate and if you want it done within the lifetime of this universe and preferably by next week you really have to approximate so therefore although we know the nature of the problem we are unable to state it exactly as it occurs because there's so many different variables we actually find this in a lot of social systems which leads us to the next one e programs that mechanize a human or societal activity in other words ease environment the probe has become a part of world models it is embedded in it it is embedded in the environment and the output of the software affects something in the real world that ultimately becomes its input in a very obvious example of this a particular scale which involves a few devices is an air traffic control system an air traffic control system whatever comes out of that goes through a human to another human that then guides a plane based on that that then has feedback through that in other words at this point we now start modeling a surprising number of the systems that we've created today and they dominate the world today we are in a world of assistance so what are the cons what are the consequences what can we learn from this well one of the things we can learn is that by the way christmas is coming this is a really good book ok 101 things I learned in architecture school it's got nothing to do with software architecture but it does make you look really smart ok if you have this book and people know you have this book they will look at you and they go you're smarter you're more intelligent more educated ok you're not just a software person it's also a really nice book aesthetically but about half the advice in it applies to any design or creative discipline always design a thing by considering it in its next larger context and that means even the small stuff and here's how s stuff if you get it wrong can have a really interesting effect on the East earth ok what if we stay what did we just deploy a piece of code anybody can use it what how much of any one earlier this year because it was known as the left pad incident as the headline the register how one developer just broke node Babel and so on in 11 lines of JavaScript this is what they describe the author of a little little feature left pad pulled his code from a public repository for reasons that were entirely entirely justifiable and to do with rights but he pulled this the Django metaphor is perfect he pulled this and lots of other things came down let's look at that 11 lines of code ok so I think this is a beautiful beautiful one from the if you not come across the oh really books or they're really joke covers co-written by some stranger on the internet is always perfect taking on needless dependencies fragile development guide it's a perfect illustration of this so that eleven lines of code what does it do well it's not even very well written this is the eleven lines of code it's what it does is it pads out a string and you can typically Pat it out using spaces or zeros these are the normal things so you you pad it out to a particular size I mean this is really clumsy way of doing it and most programming languages have a better way of doing it in this since that was pulled something else has taken over maintenance they've ended up with this this isn't a talk on development practice but honestly really it's not what it's let me put it this way it's not an improvement it real and it does all those terrible things I'm going to optimize something actually it doesn't optimize them and oh it's shocking so I had a go at doing it and I ended up with this now I'm not asking you to read that I'm just asking you to count the number of lines I'm also asking you to realize I'm not a JavaScript programmer but it's a case of like well yeah this is solvable it's a programming problem how do we know that it works well I decided given I wasn't a JavaScript programmer to write a simple set of tests and I wrote a test framework for it just to prove that so this is all easily doable these are all everything I've shown you here is s program classically s program when you run it against my code it all passes it's all green what's funny is when you actually run it against the published versions of Left pad it's got three bugs it doesn't actually do what it's supposed to and I'm sure people have got workarounds out there that deal with this now apparently as a result of this one of the future versions of Java Script or ACMA script is going to have a little function called pad start to avoid this I mean quite frankly if you do if you're using a proper language yes I am going to have language Wars in this session you thought you were just come into IT in society then you know if you're doing this in person you do that or that that's it one line well there is no problem the Internet does not collapse as a result of this but it's a valid illustration regardless of the language of Anderson's law I get to see any problem however complicated which when you looked at in the right way did not become still more complicated this is Paul Anderson his science fiction author and he has hit the hole of software development on the head with this quote this is the nature of it and we've known that this is not news this is not news in fact we can go back to communications of the ACM Edition in 2000 essica Dijkstra just before he died wrote a very simple piece but he said I therefore like to posit the computing central challenge how not to make a mess of it has not been met most of our systems are much more complicated because it healthy and are too messy and chaotic to be used in comfort and confidence it's that were their confidence where does our confidence derive from what are the consequences what we find is that we might look at these as isolated things oh I'm a developer I've just made a mistake or maybe I can skip that or I'm an organization that does development and yeah our our customers report back these things when they have bugs but the implications are not merely between you and a social responsibility or a kind of professional responsibility it goes much much further because clearly that professional responsibility goes to people you will never ever meet or interact with so a few months ago the swedish research has published a paper titled the paper software faults raise questions about the validity of brain studies or title one of the headlines turns out there's a lot of functional MRI activity that is actually not activity there's a whole series of algorithms that I use to classify in an MRI scan whether or not a voxel whether or not a visual element a volumetric element is actually valid and is it active and then above this is another set of algorithms there identify clusters of these so we can say oh look that part of the brain lit up that part of the brain didn't and it turns out that the the voxel identification is absolutely fine it's quite conservative but the cluster identification is very very poor so what they found is they frequently assigned activity to a region where none is likely to be present giving one of the classic results that a dead salmon apparently has high brain activity so this is how much that's that's pretty worrying you can be getting false positives and negatives at this level and indeed when they they when they ran these tests they also found a bug they've been sitting there for a decade and a half yeah so these are not insignificant this is affecting medical advice this is not a case of the Machine crashes I mean that's certainly inconvenient but it's one of those ideas that are well at least it crashed now we know this is subtle we are progressively and continuously misinforming people through errors through inappropriate or incorrect things things that we did not know were broken and this kind of scales up but there is a kind of a wonderful assumption here that was originally positioned Charles Babbage yeah these days famous for the analytical and the difference engine in the 19th century one of my favorite one of my favorite quotes of his he says on two occasions I have been asked pray mr. Babbage if you put into the machine wrong figures will the right answers come out I am NOT able rightly to apprehend the kind of confusion of ideas that could provoke such a question and yet and yet we live in a world where this appears to happen all the time what am I talking about one of the most common things spreadsheets so here if you think oh yeah functional programming is the world's next greatest thing I've got news for you everybody's using Excel more people have been doing functional programming than there are software developers interested in it and unfortunately it's not always functional sometimes it malfunctions really rather badly so we have a number of examples so going back there's this is this observation from Steven Levy people tend to forget even the most elegantly crafted spreadsheet is a house of cards ready to collapse at the first erroneous assumption he wrote this in 1984 so there wasn't even an axle this was just the early days this was Lotus 1-2-3 and VisiCalc in August 1984 Wall Street Journal reported a texas-based oil and gas company had fired several executives after the firm lost millions of dollars because of errors traced to a faulty financial analysis spreadsheet model in other words this is 1984 we this was already known as being an emerging problem and it's reported like old news at that point and these days we've seen this scaled up paper earlier this year Jean a mare is a widespread in the scientific literature so a lot of people in there in the genetics base and in by and as basically pharmaceuticals biochemistry all of this kind of space they use a lot of Excel spreadsheets as well as crazily enough a lot of Perl I'm just a little bit concerned about that although some of them are moving to Python which is good it turns out that they're they run foul of the defaults spreadsheet software Excel when used with default settings is known to convert gene names to dates and floating-point numbers which is not the kind of conversion that you expect for your genetics it's not going to give you any kind of x-men powers either a programmatic scan of leading genomics journals reveals that approximately one-fifth of papers with supplementary Excel gene lists contain erroneous gene name conversions so there are companies making decisions based on these papers about how they invest and what products are going to create and you know basically the future of genetics but we also get just bad science ok Christopher Essex and Ross McKittrick their climate change deniers in the number of years ago they published a thing basically that try to say oh look if you take the right kind of average you can end up with you can actually demonstrate the data shows a global cooling and a science blogger Tim Lambert made this observation back in 2004 he said Essex and McKittrick present an example that purports to show that whether you use the arithmetic or some other me can affect whether or not you find a warming trend when they calculated the trend they found an overall cooling trend of naught point - naught point one seven degrees and I would say they are they found a cooling trend now this is interesting they were doing a thing that it's known as a root mean square which I let me put it this way that's just bad physics that's not how you average temperatures they were trying to take an average average the temperature around the world so they are using root mean square that's not the correct way to average temperatures so first of all bad thermodynamics then it turns out that some of the data is missing and so they're actually using bad statistics it turns out that they are making averaging assumptions based on missing data and Tim Lambert spotted another thing I looked at the graphs and something seemed wrong to me the thing that seemed wrong he observes the missing values and he says when calculating the root mean square they treated the missing value as if they were measurements of zero degrees this is a newbie Excel error okay this is a really I was interested so I contacted Tim Lambert because unfortunately spreadsheets are no longer available is quite an old blog but he sent me their spreadsheets I'm not a spreadsheet person it was obvious when you looked at it it's just like really you overlook that this is you know anybody who's been using Excel for more than a few hours knows about what the question of empty cells and if you're doing proper science you don't use Excel so that's a that's another issue so the point here is that what you see here is nothing this is an empty cell okay if you use the average function then Excel ignores the field this is correct if you're taking the arithmetic mean it will ignore the field it just means that they were doing bad statistics as opposed to programming errors okay so bad statistics and bad thermodynamics wait a minute if you want to do bad thing with dynamics well and have a programming error you then end up with this it turns out that root in square is not supported by by Excel so you have to hand code it which they did and at that point empty cells take on a different meaning very basic error so this is also by coincidence the number of peer-reviewed papers that one of the authors Christo asacs has had in terms of climate science the other author Rosberg Ettrick he's an economist and economists they're rubbish right Harvard University economist Carmen Reinhart and Kenneth Rogoff acknowledged making a spreadsheet calculation in their classic 2010 paper growth in a time of debt also not a peer-reviewed paper it turns out um this is important this paper because it has been used as the basis of justifying austerity the austerity economics that has been used in the Western world since around 2010 has been justified by this and this is this is where we hit the next level this is not just bad science that informs say a niche niche group of climate change deniers this is also not simply a case of misclassifying gene names or putting a company out of business we've now got to a point where whole economies are effective what happened is they left out a few rows it turns out if you leave out a few rows in your spreadsheet and you want to average things up and add things up it turns out there's a significant result it turns out that if you put the wrong things in you don't get the right answer out by this much instead of shrinking the economy it actually still grows that's a really big difference if you're going to inform political policy so it turns out that these spreadsheet errors Steven Levy observes with spreadsheets the danger is not so much the incorrect figures fed into them is that garbage can be embedded in the models themselves the little little formulae are untested they are ad hoc they are also reliable truth people take a spreadsheet as a particular source of truth but we need to remember one of the oldest sayings in programming is going garbage in garbage out and that's what we get now the problem is that these days we have ways of making this faster I'm going to go back to trading so I don't know if you've been following the fate of the pound as a currency but hang on let me just show you something the pound has not been doing so well due to a little thing in summer and I can just find it dig through here we go so this is roughly the new 20 pound note you might not see it but it's 20 euros that's how much a pound or 20 pounds is worth almost it's pretty much close to that it is dropped massively but not as much as on the 6th and 7th of October where it really did if you look at this cliff edge here Wolfe against the dollar by the way the pound has recovered quite nicely in the last week against the dollar what happened here this is not market confidence this is not this is not somebody announcing a policy that pushes it one way or another this is not a politician saying something this is an automatic trading system that has a slight bug in it and as I said earlier you can really if you want to do something wrong you want to amplify an effect it says that computers can do the wrong thing faster way way faster so Elvina he came of Venus laws he's a he was an avionics safety critical expert and his summary of the challenges and the paradox of automation says digital devices tune out small errors while creating opportunities for large errors they amplify these effects and they can run away before a human being has had time to say wait a minute that's not right and if you have a whole system that set up of devices that are running the software humans aren't really involved anymore it turns out that not only can we affect an economy by misinforming through economic policy it turns out we can have a radical effect even faster at this level by getting just computers to do the whole thing slight misunderstanding as to the context the larger context the next larger context I was talking about earlier it also turns out that human beings are part of the system it's not just we're affected by the economy it turns out that when you put a constraint on people people respond my uber driver describing how drivers coordinate to engineer surge pricing say a big event finishes drivers know this so they shut off apps in unison they're not available so therefore if there are no other drivers available then driver availability is a scarce resource so therefore if you are going to be a driver you want to be the only driver or one of the few drives at that point the pricing goes up at that point they all sign back in they created an artificial shift in demand in other words this is a pure esis 'tom you've now actually and the problem is it's not a software to software system it's a software to human it starts exploiting the defects in us we start gaming the system this is also as a point it's one of those things I've always felt uncomfortable with SEO search engine optimization what we're doing is we're getting people to effectively game a system professionally and that system is something like Google so there is an effect here I want to talk about democracy so let's talk about democracy and show you how good the British public are with democracy some of you may be familiar with the boaty mcboatface incident earlier this year that um I mean I you know it was the will of the people I'm surprised that it's not it's not actually now law that this is the case but what happened is back in March this research vessel was going to be named and it's put out to the public and somebody proposed boaty mcboatface and then via the joy of social media this effect happened disappointingly they didn't call it that right but I think that it was the people's will we should have gone with it and this obviously has broader effects so once you then get the next effect we start looking at how this works with people this is a famous saying it's an actually quite an old saying when it's made more famous by Facebook it's a general engineering saying move fast and break things does that include democracy well number of points been made today about the effect that we've got here but this is interesting because what we have is normal groupings of people naturally have a filter bubble that they create they naturally create an echo chamber this is known within social networks and I don't mean social media I mean ordinary social networks of human beings just talking associating and so on and it turns out you there's already an exponential reinforcement that may be reduced or increased depending on the diversity of people you come into contact with the problem is that the algorithms that people like Facebook use are also exponential so therefore it creates a higher power experience which is basically a vertical wall you cannot hear the effect of things now I said I live in Bristol we do a lot of street art in Bristol this disappeared in May fortunately we managed to get rid of the fellow on the right well only sort of Boris Johnson but sadly the one on the left somehow made it in but there's this really important observation here that we can see this effect brilliant tweet the other day Facebook is now in the awkward position having to explain why they think they drive purchase decisions but not voting decisions in other words the purpose of advertising is to influence people to buy something so therefore you're selling advertising space on this and there's a lot of research that goes into doing this and there's a reinforcement boundary that they create I don't think they necessarily don't maliciously but it's normally people doing lots of small things without realising the bigger picture there it's not obvious they're doing this except that actually we can see that they do do this it was an experiment in 2012 that only came to light two years ago three years ago this is the abstract from the paper that was based it was a huge great social experiment on Facebook we show Vyra massive N equals six hundred and eighty nine thousand and three in other words that's the number of people that did not know they participated in an experiment on Facebook that emotional states can be transferred to others via emotional contagion in other words if we make some people slightly let's see more unhappy posts do they themselves become unhappy and post things that are unhappy can we change people's moods in case you're wondering what the answer is the answer is yes that's the whole significance of n so there is a there's something else going on here that some level there is an awareness now sometimes people say well that's just Facebook That's not me I'm afraid if you do a B testing you need to find out where your line is because you are changing by choosing between a product a and product B you're not simply is a better or be better if you are using a B testing to influence people's product decisions Hey look on a we got more click-throughs and more people spent money you are changing their social behavior you are changing their bank balance when does that start changing mood when does that start changing other effects a B testing is not without a moral side it turns out they just happen to do it on a really big scale at Facebook and we have the filter bubble effect that reinforcement so this was posted the day after the UK referendum I am actively searching through Facebook for people celebrating the brexit leave victory but the filter ball is so strong and extends so far into Facebook's custom search I cannot find anyone who is happy despite the fact that half the country is clearly jubilant today and despite the fact I'm actively looking it turns out that we've ended up we've shifted from the past where we actually had relatively little information and by this I mean when going back a couple of hundred years to a time where we have loads of information but what we lack is not quantity but variety and we have accidentally created a system that reinforces that so there is this idea they're trying to keep us happy we're trying to please people we're trying to please all of the people all the time and finally we have the opportunity to do that so we created algorithms to keep all of the people happy all of the time including it turns out the advertisers the point here is we don't discriminate between true or not true just big or small just popular and unpopular there's a difference here and we're also there's that also into this Facebook is making editorial decisions effectively and it's not doing those consciously the fact that they are being done by algorithms rather than human editors doesn't make Facebook any less responsible this is the kind of classic effect you may remember this from not very long ago at a couple of months ago this is one of the iconic images that is seen to have changed our reporting and our concept of war in the 20th century and this was posted in a Norwegian paper which had an outlet on Facebook and Facebook decided automatically no we're going to take that down and in fail to reinstate it at the appropriate point because well you know it's child pornography no it isn't that's a that is a young child screaming she has a napalm burning on her background is also one of most important images of the 20th century even when it was pointed out to them they did it and they also managed to when the Norwegian Prime Minister also put it on her page they blocked her blocking heads of state is generally considered to be a bad thing okay so we've now ended up with a massive escalation here an accidental escalation and nobody's actually set out to do anything malicious but what it does is it highlights a sort of a social responsibility that we may not have been aware of a broader effect so I'm going to close with a kind of observation I run a page on Facebook word Friday where I talk about linguistics and where I post things are about listening linguistics like every Friday I post an unusual word and it's definition I recently posted this word mecca NOC recei mecca NOC recei government or control society by machines or a state or other association of people run in such a way now of course this kind of sounds very ominous it sounds a bit rise of the machines terminator but the idea of machine rule brings to my robots but the term refers more broadly to the wide scale to my automation of governance and social management through software we are in that era now and it turns out that it's not quite the happy story we thought it was going to be it has engagement it offers engagement it's great there's a whole load of stuff that we we don't want to give up because it really is very good there are a lot of things that are massively improved it brings us into contact with people we would not normally have contact with it also unfortunately has some side effects and we seem to have a blind eye to those side effects the problem here is not that you know it's it's it's realizing the social consequence of that that we may accidentally without meaning to end up with a situation where we as we're part of the feedback loop we end up squeezing ourselves you know through this desire for mekin ah cracy for more efficiency for better business for a neater algorithm or just making a piece of code available but not worry about testing it we may accidentally end up receiving the wrong feedback and I followed I partner a tweeted this recently there's there's a kind of classic Churchill quote we shape our buildings and afterwards our buildings shape us and there is this idea that we have now become part of the system and we are responding in kind thank you very much you

Channel: GOTO Conferences

Views: 34,497

Rating: 4.8608694 out of 5

Keywords: GOTO, GOTOcon, GOTO Conference, GOTO (Software Conference), Videos for Developers, Computer Science, GOTOber, GOTO Berlin, Kevlin Henney, Software Industry, Software Development, Software Engineering, IT & Society, Society, Philosophy, Software Failures

Id: IiGXq3yY70o

Length: 49min 45sec (2985 seconds)

Published: Fri Nov 25 2016