Small Is Beautiful • Kevlin Henney • GOTO 2016

right good morning so there's a bunch of things that people don't generally say as you left me with the legacy systems thing I do there is the question why do systems become legacy what are the properties of a legacy system and this is actually something I run in workshops I ask people what what characterizes legacy and I get people to list off properties of legacy systems because there are many things that qualify as legacy we actually hide behind the word legacy to sometimes mean very very different things to people using that term can mean quite different things we can be talking about a system that is just using old technology that we don't like we can use we can talk about the word legacy to mean code that somebody else wrote that was not me that's one of the most popular definitions but one of the things people one of the things that people sort of invariably end up with is at some point there's a kind of sense of size and I'm going to pick on small today just of all the many things that we could talk about I'm going to talk about size issue sometimes that relates to simplicity not always so I'm pre-empting the fact that sometimes people come up to me and say yeah but you didn't talk about simplicity or you didn't talk that's fine that's you have when you do a talk you're supposed to stick to kind of one thing but I will wander around quite a lot physically and intellectually hopefully but people don't generally say oh yeah that legacy system it's so small they don't complain of like oh yeah the legacy system whose code base is about the right size that's that there's never a sense there is always a sense of there's too much stuff and that's an enduring idea and so there's a question of sustainability so I've shamed lessly stolen the title in fact the inspiration ultimately for this talk comes from this book here and small is beautiful published in the early 70s 97 three ants schumacher a an economist who effectively created what we now think of as a more responsible form of economics the idea that you don't operate in a vacuum and that perhaps there's a question of sustainability and so and it's a it's a lovely old book and I went back to it again recently it's kind of interesting seeing the view from the 1970s now that we're in the 21st century and how much he got wrong but also how much he got right and there's a subtitle here a study of economics as if people mattered so I can steal that I'm going to go with small is beautiful and you know by the way if you don't have kids yet give them an Internet unique name they will thank you at some point okay my parents managed to do this before the internet existed which is a major act of pressings on their part but you have no excuse I've done this for my children okay I did this when they were born I googled just to make sure that there were no hits on first name last name and you know when my older son joined Twitter dad what name shall I use I said just try a first name last name like I do he says oh it's available I said yeah that moment was thirteen years in the making so you you can offer this service to your children okay however a talk on codes that people mattered this is an interesting one one of the most amazing things that we've discovered in the last decade or so it turns out software developers are human beings I know I mean how did that happen but what that means is that human beings have a particular set of properties comfort how they interact with one another that gets reflected in the code we often try to think of the code of somehow separated from the people that work on it and this also tells us about why things get out of control and get bit but there's also another aspect talk on codes of economics mattered because there is this idea that people are always chasing a deadline there's always a reason why they're doing something the way that it is well sometimes it's an excuse not a reason but it's worth actually kind of pulling this back and saying hang on does this actually make sense is this the right way to invest in development and do things so that's the kind of the bigger picture now I'm out of all this I've mentioned the word sustainable development and kind of Schumacher's work ultimately led to this observation in the Brundtland report headed by the former Norwegian Prime Minister NGO harlem brundtland in the late 80s that was looking at sustainable agriculture sustainable economies and so on and I'm doing something far less interesting here I'm talking about software but the definition of sustainable development is an interesting one because it's something we can appropriate so when eventually the words agile and lean lose steam then perhaps we can use sustainable as a way of kind of saying the same thing but giving it a new label sustainable development this development that meets the needs of the present without compromising the ability of future generations to meet their own needs so they'll immediately we see that the challenge of development is to strike this balance if you never have to worry about future generations then that's that's fine you can just put out anything and you're done but normally we do have to worry about future generations even if it's not ourselves there's normally somebody else who has to pick up the code there's normally a release - what's that going to look like and that becomes the question it's not but people are so focused on initial release and first release and then the next release becomes another first release without realizing there's a whole procession of releases and maybe that this is a system of releases emini there's a better way to approach this on the other hands we end up with sometimes the over overcompensating big architecture the big enterprise projects the global projects I went through a phase of noticing and visiting companies either workshops or even contract work when I noticed that the word global kept appearing in their project names or enterprise and when a project acquires that name you know it's doomed okay it's just like this is going to be No so that's the point is that that's the future that will never arrive there's no present there so this is balance and the balance is the bit that's hard so let's explore this a little bit that's first of all start out with the idea that not everything is going to be small but that everything has an appropriate scale there is a right size for certain things the question is are we able to get to that size this is ultimately an optimization problem here is the system you would like and here's the system you've got how different are those not just in terms of the physical sizing of things like the source code which we're going to come to as being something that you need to put in your head but actually in terms of the development effort um so we can get a real sense of this with a bit of Shakespeare and C soon as we're in Denmark I will choose Hamlet Prince of Denmark and so since this is fairly classic to be or not to be that is the question this is fairly classic how would we approach this if we were writing this in marketing speak or business speak okay well we have an answer to this question Tom Burton and his book long words bother me how'd it go continuing existence or cessation of existence those of the scenarios is it more empowering mentally to work towards an accommodation of the downsizings and negative outcomes of fzero circumstance or would it be a greater etc etc and you know what a lot of developers write like this as well actually they write like this people have some really really crazy ideas about how to lay out their code you know it's just says oh look I've got an IMAX screen so I will push everything out over here and I will use really long names to be really helpful as we see the bigger the word it's not necessarily more impressive you end up with homeopathic naming where you add more and more to the identifier names more and more you're diluting the meaning further and further into there's absolutely nothing left so let's go back to this question of constraint and stick with Shakespeare for a moment this is the monstrosity and Lovelady that the will is infinite the execution can find that the desire is boundless in the actor slave to limit so there is this idea that human imagination is pretty cool we can imagine all kinds of things the problem is that we are eventually confined to what we can deploy what we can write what importantly we can actually hold in our heads that turns out to be the bandwidth limitation what you can actually reasonably in hold so there is this concept that a good architecture a good code base a good piece of code is something that at some level you can hold in your head and reason about comfortably the minute it overflows the harder it becomes to reason about and suddenly you slow down you're more likely to introduce defects you're more likely to introduce technical debt you're more likely not to touch anything at all because you're just a little bit cautious you're not entirely 100% sure you understand it and that is how you create a little as one of the ways in which you create a large system is that kind of hesitancy that a fact that you cannot reason about it this does not mean we do not build large systems it just means that we have created a system that feels large whereas you want to create a system that wherever you are it feels local you can grasp the concepts around you so it turns out that this is your bandwidth now obviously we can go to the extreme of saying well let's talk about ultimate smallness an ultimate constraint my favorite example which I've actually got an update for is this piece of code start the day with a bit of assembler I think some of you are still booting up so something fairly low-level like this should probably suit you this is z80 code it runs on a zx81 runs present-tense I guess ran it's written in the early 80s 1982-83 by a guy called David Horne this was referred to by games developers today as frog Lee as being the greatest program ever written which is quite a claim but it's actually got a fairly reasonable to consider that this is a chess program I mean it's not exactly you know deep blue but this is a chess program and it's done to fit in a 1k memory space okay that appreciation of one kilobyte in fact to be precise this doesn't fit in 1k if it's it's 672 bytes because the runtime has to take some space this is LX 81 so 672 bytes that's 4.8 tweets you can't even describe the rules of chess in 4.8 tweets yet this game this I mean as I say you might go to a few deep learning sessions it's not exactly going to compete with those but it can recognize a draw and a win it can play the basic rules of chess so I don't think it has I'll pass on basic bits of pieces like that and it can play against you that's just impressive obviously somebody forgot to tell David Horne this was not possible so he went ahead and did it now why is this why is this even remotely relevant well one it's kind of fun to I'm not going to claim that this code is brilliant but the chances are is that the reasoning behind it the very challenge that he has taken up is the thing that we're interested in is when somebody says how you can't do that there's a very good chance that you can and that the very constraint that you said you can't do that with will allow you to overcome this so there's a very good case for this being exactly what it is now I've used this example a number of times before or being in a remarkable example of cool code but actually recently a couple years back indeed last year Olivier Pooh dad managed to create boot chess which fits in the boot sector says 512 bytes so that's even smaller which is really impressive so that's that's that's half a K yeah now why is this relevant why is this relevant at all because obviously I'm not going to say we should all write obscure code we're already doing that anyway it's called code for a reason and obviously there's a particular aspect here such a thing is a work of art in one sense is a unique result of unique temperament but it is just a reminder that ultimately when you are creating things as the agile test david evans observed your customers do not buy your software by the line if they did that would be great for some people and we just type more just keep on typing because you look at some systems and you think is this are they actually incorporating and encoding code is a word that we can play with you are codifying business knowledge you're codifying technical knowledge you're bringing it all together and that's sometimes you look at a piece of code and you're thinking this is a really long hand way of doing this you look at an interface and you think this is a ridiculously overbearing and heavyweight interface there's so much here and yet for so little functionality so there's these questions that come to mind but the challenge should not be around this there's there's something deeper we want to kind of reason about and we have these lovely examples about constraints one of my sort of favorite classics and sense of how people have created and reason about these things is this book by the late John Lyons who's a professor at the University of New South Wales were correctly in the 1970s he wanted to write he wanted to run at a university course on operating systems and he wanted to run a real operating system he wanted something authentic that was not designed for students so he wanted an example from the real world and he so he managed to get his hands on the recently released code and you know there was no internet to speak of but the the code at the code from the six edition of UNIX and and then annotate it he wrote it up and described it as you can see before as she got published as a book it was passed around by samizdat people would basically photocopy the notes here is your whole operating system and annotations now for those of you who are a little young a photocopier is a kind of socialized scanning okay you'd have yeah it's you don't scan alone you scan other people you kind of hang out in the university library hoping that somebody you know and want to talk to will come by and yes as a social experience so that this was that this was the premise here but the key thing here is that the the colonel was under 10,000 lines of code 95% of which was written in C was written portably only 5% of which was written in assembler and this importantly so at the time this was absolutely one this is it's something you can't do you cannot write a whole operating system that is retarget able in a language that is retarget able this was a basic article of faith it always has to be assembler or most of it has to be assembler but here we have the very kernel not simply the shell being expressed in a way that is portal and we look back at the code and you look through any otherwise fairly antiquated C style it's an area of very different conventions and what was defined to be good and yet at the same time they did things like write code in separate files that 10,000 lines is not all in one place and yet I see classes that have this kind of line count if not larger there are classes out there in systems that people are paid good money for there are classes out there that have their own gravitational field and we know about the planet is more theory of planet formation you have something there and it has a particular size other things are drawn towards it because of the gravitational field and it becomes larger and larger until you've got one hot Jupiter of a legacy system ok you've ended up with this thing it just grew nobody planned it that way I mean people don't generally out this I don't believe this happens but if you have anything to contradict this people don't sit down and say right ok we're going to build a new system how we're going to do it I think we should have really large classes that with lots of duplication absolutely should we make it testable absolutely not okay you know I think we should you know I think we should you know if we're struggling for a name for things I think we should use names like manager and and and processor and controller and stuff like that and scatter the responsibilities around the code which would cause more duplication confusion and I think she put a you know shed load of people on the project as well and we're going to make this really obscure nobody will understand it okay make it so that's not how architectures are created I hope but there's a key point here that there is a challenge here that was met that there are under 10,000 lines they've been able to create an operating system whereas sometimes you look at something and what have you managed to do we've managed to take a message and process it into a different format I had that with one client it's like that's 20,000 lines good god I mean who is this not a solved problem so we have this observation this observation was made fairly early on in UNIX is life the late Dennis Ritchie and Ken Thompson 71-72 paper on UNIX there have always been fairly severe size constraints on the UNIX operating system and its software given the partially antagonistic desires for reasonable efficiency and expressive power the size constraint has encouraged not only economy but a certain elegance of design and this is the interesting thing is that are actually working with the constraint not simply against it and we often assume that a constraint is a negative thing to have but I want to try and get back to the point of why I showed David horns program it's not because I'm saying you should write obscure code and chess plan you know and encode chess programs into small bits of your enterprise system what I'm actually saying is that you can use seemingly arbitrary constraints as a benefit the human imagination is indeed infinite the word but the will is indeed confined if you can find that constraint you will probably find something of value to help drive the architecture and that thing a value may change but once you have that you have a driver and we see this time and again I mean Donald Knuth observed one rather curious thing I've noticed about aesthetic satisfaction is our pleasure is significantly enhanced when we accomplish something with limited tools okay and you know as he says yeah makes a person feel like a real virtuoso to achieve something under such severe restrictions but he's not the only person to make this observation we're going to talk about virtuosos to about Stravinsky the most the more constraints one imposes the more unfreeze oneself and the arbitrariness of the constraint serves only to obtain precision of execution you need to find the constraints in your system they're not always obvious at the beginning this is this is so this has becomes a hillwalking problem what you're trying to do is find the constraints that characterize your system and find the spaces that allow you your degrees of freedom and once you found that then you think to work against that will give you a more natural sizing you can reveal the natural sizing of a system so what is it that in that case stands against our assumptions well it's kind of a nice observation from a program I know Chris Allwood isn't a netbook screen a bit small know if the code doesn't fit on it it's too complicated and there's a big case for I want to go back to this observation I meant made earlier on the way that sometimes people respond with how they structure their code they structure it according to their available screen size and these days for developers that's quasi a quasar IMAX you know they're kind of sitting there in front of like triple-headed 4k resolution and they're like yeah that's fine I've got no problem with a name that is 120 characters long because I've got the screen for it and it's like you're solving the wrong problem the problem here is that there is a human being turns out we are humans much as our screen sizes have evolved it turns out that the visual system has yet to catch up and when you actually have to turn your head to read the end of an identifier you know that's a bad sign when our source line of code has its own microclimate that's different at the end of the line to the one at the beginning that's not a good sign so you did this with a client couple of years ago we're just going through the code and actually I'm going to say but their code was it was within 8080 columns actually except every now and then something would just go off it was kind of kind of funny looking like a looking at a like a heartbeat something you know this is a bit I'm goth screen a bit complex algorithm a bit what's that formula doing so we went in search of the right-hand side you know we we got our blunderbuss guns and we got our appropriate hunting gear and we went out into the wild we found a line that was 360 columns long you know we kind of almost wanted to make it it put an extra five characters in there so we have one column for every day of the year and the point is that that's not human that doesn't work with humans and similarly the way that sometimes people will hit 160 column limit and say are 80 columns that's just so 1970s it's like no no that's to do with human visual perception only if you actually care about how people read it turns out I'm going to give you bad news 60 columns so by saying 80 to 100 we've been quite generous it turns out that we're much more comfortable around the 50 to 60 look at how web pages are organized yeah in other words your code defines your code source code actually defines an experience for a user that user is a developer and how does their visual system work how are they able to chunk and reason about things so there's a sort of a call here some kind of balance and reasoning but also this says something about our ability to break a problem down and to reason about it clearly the sizing thing does matter is how easily you can grasp stuff I recall a visit to a company a couple years back where they were complaining about the technical debt on day one we were talking about technical debt as well as well I was there to talk about and then there was this discussion in the evening I went out for a meal and I said I'd really like to see some code as we talked about we've drawn diagrams on whiteboards and you know talked about things but I really like to see what code is like so I then paired the following warning with one of the developers and you know just sort of tweaked a couple of tests looked at the code and code looked pretty good to me I have to say and then the moment that will always stay with me is he pulled up on screen a method and it fitted on the screen and he said this Carolyn this is what I'm talking about this is the problem yeah he said look at that it's such a mess you know it's so large it's like right okay thank you you have calibrated your system for me if this is the worst you can show me you do not have a technical debt problem okay I said yeah it's there are people that were kill for methods that fit it on a screen this is the worst he had to show and yet you could reason through it yeah it was a bit tricky in places but that was actually a reflection of the nature of the problem not a reflection of source code chaos it was not a reflection of muddled thinking that it started down that road of like I'm not sure what's going on here let me add something rather than modify what's in place and so there's that idea that these seemingly arbitrary constraints can help us to reason about things obviously you have to spawn respond in a reasonably rational way don't do what I didn't meet this guy but it was at a place that I used to work and it's far enough back that when people talked about the size of procedures or methods in code they would talk about printed pages and the idea is that the routine should not go over a printed page so this guy responded in the way that many developers would with ingenuity and he worked around what he saw as an arbitrary constraint so he'd write the code as he wanted printed all out and then just mark off the boundaries and then break all of the procedures of those levels yeah so you have an arbitrary point here and called version the thing to at this point that did not have the desired effect that's not the kind of constraint we're talking about but we can take this further I've talked about code locally what about the idea of creating a whole product based on this kind of thinking what about the idea of actually introducing an idea to the world this is this is interesting that John O'Dell former editor of bite magazine a couple years back he was reflecting on Ward Cunningham's wiki and he said you know here's the original email 19 years ago I received this email from Ward Cunningham John so what do you think of wiki I put it up a year and a half ago when HTML authoring tools were in short supply I think it has held up pretty well that I suppose it's days are numbered it turns out the numbers quite large still there was something about it we call wiki nature that is in short supply on the net and in computers in general and that is still true 20 years on there's something really profound what Ward did was you know this is the top and tail of one of the early versions of the wiki and I used to hang out on the original c2 comm wiki it was quite a little community there but it actually turned out to be quite useful and it was there was an ease of use about it that came from the introduction of arbitrary constraints you couldn't just put HTML into it it was not an HTML authoring tool and it had a very simple model of links simplistic even in fact it's so naive that it's amazing it got popular at all but that's the whole idea there's an idea here I want to pick up at the end about the simplicity of the approach that led to something that was easy to discard we don't talk enough about discardable code we sometimes talk about throwaway code and that never gets thrown away have you noticed that we use we use metaphors so badly it's just like yeah I've just written this throwaway code now what normally happens is that throw a bit code becomes your system or in a good case you might have just actually just best about with something throwaway code your colleague comes to you six months later and says oh you know that's throwaway code you wrote six months ago to do blah blah blah and you say yes could I have a look at it yeah absolutely no problem well I want you to pause for a moment and think about the words throw away I threw it away but here it is six months later it did not fall into a time vortex to reappear at this point in time what happens is we didn't throw it way there's a very simple concept in the real world of what throwaway means it means when I throw it away I can't get it back we don't throw these things away we generally hold on to them but there is an idea here of a property that we don't value enough discard ability the ability to say there's a piece of code and that kind of worked out fine for us but now I've got a better idea and I'm happy and free to throw that away just purely from a human point of view the larger that thing is the less likely you are to throw it away okay the bigger something is this is this leads to the sunk cost fallacy the bigger something is the more effort it acquires around it the less likely we are to throw it away the less likely we are to experiment with it the less free we feel because in truth we don't really throw things away we just have different versions of things that's one of the fascinations of software development and probably one of the reasons quite a lot of people in software development enjoy science fiction in one way or another it's the fact that you get to play with alternative realities on a daily basis you get to play with constructed worlds you get to time-travel I mean you know that's that's quite that's quite a big deal yeah there's this whole idea that in other disciplines they don't get to time-travel whereas we can go all I didn't mean to do that let's go back in time step into the TARDIS unfortunately we don't have that kind of ceremony that makes it sound exciting but it would be kind of cool if every time you did an undo you kind of got the kind of warping sound of the TARDIS yeah just to remind you that what you're doing is actually quite interesting you are travelling back in time you can go into your repo and go back in time and navigate into an alternative reality we don't we don't play enough with that idea it's a fundamental advantage it allows us to create small things and throw them away so this is the top and tail of the original Perl file for the wiki so the wiki was made in Perl I mean good grief it's my least favorite length I'm not even sure it is a programming language so friend of mine once described it as executable line noise and I think he's still right but it's 150 lines of code that's the top and the tail of it and yet I want you to imagine if you given that to a team in an enterprise ii organization what we need is an interactive editable web thing ok we'll get back to you in eighteen months and then they overrun by six months this is just so small it's so ridiculous that it's a it's an idea that surely could never work and yet it does and it has the opportunity to grow because of this so Hemingway kind of had it right s you the monumental shun the epic all the guys who can paint great big pictures can paint great small ones so this is this problem that what we've ended up with is a sizing thing today we suffer from an almost universal idolatry of giantism it is therefore necessary to insist in on the virtues of smallest where this applies but the problem is it probably applies in more cases than we respect it and that sometimes when you get a conversation with people what you'll end up with is you know I do I do a session like this in a conference or I say something at a workshop at a company and somebody go yeah here we're saying but our system is special it's ten million lines it's half a million lines it's it's whatever and we're special so therefore we can't do that trust me everybody is a special snowflake that's the whole point of software software is about software development exists to create differentiation it's not a manufacturing process so everybody is ultimately going to be different and everybody is working against interesting constraints but is the problem is people weigh in and say our system is special without ever really asking why is it so specially large why is it that you have had to organize armies to do the work which kind of leads to the question of why do these systems get large they are we looked at a couple of reasons but let's take a take a different tack I'm going to borrow from friend of mine Hanlon Kelly who's been it's been making a very strong case for this understanding of the economics of software and the economics of team sizing and team effort and he says software development does not have economies of scale if anything development has economies of scale and this is it this is this is quite this is quite difficult to reason about because from a very early age we are ingrained with the idea of economies of scale it's kind of part of the language and the example he uses is this idea of if you get a he uses milk cartons I'm more likely to use wine glasses okay but there is this idea that if you ever buy a bottle of wine in a restaurant if you were to buy the equivalent number of glasses the cost of the glasses would be far more than the bottle of wine there's a natural economy of scale I'll sell you ten for the price of eight for something you know my youngest son had to do this exercise at school where they you know went away and did this little project and invested money and then try to sell things to other kids and parents and all the rest of it this is what happens when you do brexit you know it's a you've suddenly got to figure out where the hell's the money gonna come from so now we're we yeah child labor but if you want any pencils you know my youngest son can do a really good deal on them because that was one of the things that he and his best friend came up with was you know yeah what we'll do is we'll split do a special offer we'll give you five pencils for the price of three it's not Wow it was pretty good he's understood economies of scale that's before he turned ten it's ingrained into us very very deeply so there's this idea that when somebody says we need this software system it's just that right we need bigger architecture we need bigger teams we need more people we don't even know we're doing it because it's so hardwired into the spine of our thinking it's not even happening up here okay it's a much it's brewing down in the reptile brain so there's this challenge that we need to kind of pause and think maybe this is not the way we do maybe what we're doing is we are thinking with or through ideas and what we are call thinking is generally the application of pre-existing ideas to a given situation or set of facts we look at a situation through a lens of experience and habit and perhaps we need to challenge that and there are never underestimate how much force the situation of your existing ideas and also the organization around you exert now it has become traditional to refer to this paper but so therefore I do it slightly differently this is actually Melvin Conway's original paper how do committees invent in 1968 so it's not some kind of retro look so I'm after here this is actually genuine retro before the moon landings just to put this in perspective and a Melvin Conway made this observation that has been observed casually but also since verified a number of times yeah the basic thesis is that organizations which design systems are constrained to produce designs which are copies of the communication structure of these organizations in other words the way that your system will come out is to do with the way that you arrange your people and how the people communicate with one another and this is an important idea because it is how people actually communicate that he's saying here the problem is that we have an idealized view of organizations we think that the way that people are organized is follow something like an organic Ram a hierarchical decomposition yeah you have that you have Department you have the route of the company and you have department heads and then you have people who actually to the work and it's all beautiful and in some companies this this you know this appears on PowerPoint files and other companies it appears plaster on the walls you know if you wanted to know I just referred to brexit the greatest achievement of the brexit Department yes we have a department apparently over the whole summer was to produce an organic Ram of who was who they had hired that was it they hadn't done anything else but it was a very pretty diagram it was the what they were doing was offering the illusion that there was some reality here that this was the reality organizations do this there's a problem though that's the organization as you would like to perceive it according to pay grade line management and all the rest of it but let's take a slightly different observation it's taken from a book designed after modernism which has a slightly unusual cover to say the least by john zacchara's collection of essays is a rather interesting essay by Christopher Alexander a city is not a tree this gives us a different insight into sometimes how we can think about architecture when we talk about architecture we normally think about a single earth space within an urban space we use the metaphor to refer to buildings and we talk about buildings an awful lot and indeed this is the same Christopher Alexander that was responsible for this idea timeless we're building patterns this is he came he's the guy responsible for that it's the same guy and he's interested in systems of thinking overlapping thinking how one design idea can lead to another or support or negate another idea and that we're not thinking just in terms of modular parts we're thinking about how a system actually overlaps and as he observes it says the tree of my title as a city is not a tree it's not a green tree with leaves it is the name of an abstract structure at which point the whole room suddenly becomes comfortable ok yeah we can do that yeah we're good because as as you and I know trees their roots are at the top their leaves are at the bottom that's the true way and if trees have color then it's red and black okay this whole green trees with roots at the bottom and leaves at the top that's some kind of real-world aberration that we just target with CG with CGI yeah but the real trees are these they are abstract structures and he wants to contrast it with the semi lattice their ways of thinking about how a large collection of many small systems go to make up a large and complex system and as he then observes well look this is this is actually what goes on in a real organization people are not they don't sit within a hierarchy that's not how they live and breathe the system it's not how they communicate with people it's also not how the code grows because the paths of communication will emerge from that and you have a choice here you can either acknowledge that this is what happens and therefore you can organize your architectural effort around that or you can be surprised by it every single time it turns out that where people sit who people know who people trust who has certain expertise in some things versus others who is easy to communicate with all of these kinds of things we'll ultimately start shaping the system and we haven't even started talking about the requirements and how you choose to deliver so there's this idea of complexity which is why people shy away from it semi-lattice is potentially a much more complex and subtle structure than a tree the reality of today's structure is thick with overlap the subsidy the systems of friends and acquaintances form a semi lattice not a tree which means that although we may envisage a system and you know decompose it rather elegantly into smaller and smaller parts and if you look at every single paradigm of development they're always about decomposition and somewhere in there is the concept of a tree and we find it valuable it informs the syntax of our languages and informs the way that we indent code it's certainly valuable on a local basis the problem is the reality is a lot Messier so you've got a choice you can either say I deny reality and we're going to force it this way you can say no realities like this it is a bit messy how do we organize the messiness and yet still get localized order that's non-trivial that is exactly this Christopher Alexander says subtle because if you don't respect it you will end up with some surprises you will end up with the surprises that that occur when you don't respect this dynamism if you like we've seen that this fact has important implications for the management of system design a design effort should be organized according to the need for communication if we don't then our code will reflect this communication it will reflect the mismatched messy communication if you what is communication it is the transmission of knowledge mmm no that's good communication there's an awful lot of communication in organizations not about the transmission of knowledge it's to compensate for the organization's inefficiencies and attempt to be hierarchical and actually it isn't quite hierarchical so there's this idea that that sounds like just pure business speak but what I'm saying is it trickles down into the code as was observed in this paper in 2007 the MIT Sloan management review is a website it has a website and you can kind of go to it and read one article if you're going to read one article for free this is the article you want to read it's still remarkably relevant avoiding the alignment trap NIT they make this observation having looked at a number of large companies I t's effort to satisfy its various and sometimes conflicting business constituencies created a set of Byzantine overlapping systems that might satisfy individual units for a while but did not advance the company's business as a whole what they're looking at is the fact that an organization organizations often ended up with alignment that was local but also against other groups and they say we have a strategy we are business aligned the thing you need to remember is that in an organization an organization is made up of people and we we have a nasty habit of oversimplifying it's part of the abstraction nature of humans we sometimes talk about the customer as if the customer was a single entity in most cases the customer is an organization filled with individuals who don't all agree with each other and that that gets quite interesting if you say we are going to follow the organization's lead we have a vision for the organization it turns out everybody is pulling in different directions which leads do this which also starts telling us how systems can get messy or tells us again how systems can get messy obviously at this time but down not gonna present anything more complex in a quadrant diagram because you know quadrant diagrams are easy they divide the universe into four and normally there's one very very good quadrant or perhaps there's one very very bad quadrant in this diagram we have both okay and there is a simple idea we're going to based on a survey of over 500 companies that have some kind of software development and IT capacity within them as opposed to say software houses or consultancies two axes alignment with the organization is your IT aligned with the organization strongly or weakly and effectiveness how good are your teams how good are their practices this is not a judgment on how good are the individuals because as we know as we know you can get a lot of people together collectively and they can actually act dumber than any of the individuals in Britain we call this a referendum for example and also there's the England Football team individually there are actually quite good players collectively they can be beaten by Iceland okay so the point there is that it is quite possible to have a group of people be far less effective than the individual capacity suggests and this is it so this is about coordination about the necessary communication that you get and so the observation here is one element of the observation is unsurprising most teams 3/4 of them are in the bottom left-hand corner they're not particularly aligned they're not particularly effective but turns out from a financial point of view that's not particularly bad they're not really gaining money they're not really losing money in fact they're probably in equilibrium with an equally inefficient set of other departments in the organization everybody's reached a kind of comfortable equilibrium they're all kind of they will suck equally but they're happy with it that's good this is where most companies are what people want is the top right-hand corner which is one tenth of the size ok in other words these guys are they getting cheaper and they're generating more revenue this is fantastic everybody wants to go up there now this there's another reason I chose a chess program near the beginning this is a bit like chess except you don't get to be a bishop you get to be a rook you can only move horizontally or vertically if you're in the bottom left-hand quadrant would you like to become more aligned with the organization or would you like to become more effective at software development the standard the standard answer from many consultants is become more aligned because everything else will follow I'm sorry that's the alignment trap you've just walked into it it turns out that you have taken a group of people who are not necessarily coordinated as a whole who don't necessarily have the best development practices and you've just thrown them to the Lions and they've got eaten that's what those that's what that's what those numbers are what happens is that they're never able to recover they're always worrying about delivering in the present and they're never able to kind of say you know what we could step outside this vicious cycle that we've got ourselves in and there's a different way to approach the architecture there's a way of approaching technical debt we can improve the testing we can improve the deployment we can improve our skills we can view but we don't have enough time so none of that happens and it gets worse you've become an amplifier for the bad practices you haven't yeah sometimes people describe certain development processes this is a popular one in the scrum world a process that shows you the dysfunction of your organization that's one of the most frustrating phrases you can ever throw at a group you know they're they're struggling we're trying to do stuff well scrum is revealing your inefficiencies to you yes we know but that doesn't tell us how to solve them there's a frustration there that all we've done is said yeah you see that problem you thought you had you have it it's like thank you whereas it turns out that if you decide to become better at what you do it turns out that has its own benefits and you can that doesn't necessarily mean that you're you know not delivering useful values to the organization but you're more likely to be able to do that and this goes back to the release one versus release two verse thinking but not everything is released one the idea here is imagine simple thought experiment a system that has been developed by top left-hand and bottom right-hand teams they've developed the same functionality whose do you think is going to be most likely to meet the requirements quite possibly these guys because they've been aligned to the business in fact the business won't shut up so they really know what is required these guys might miss they might miss some key requirements and assumptions okay who's got better code what's these guys better deployment practices better quality better management and explicitness of all the things that are invisible in software development these guys and so what's their code look like those guys over there are gonna have a nightmare if you want to then join the team join this team because when they say you know what we realized we missed some key requirements oh that's easy we've got the test we've got the code it's all good it's small it fits in our head we know how to change it and you're done or as though these guys over here they've got this Jenga tower yes that is now a style of architecture they've got this Jenga tower of architecture of like don't go near it I need to fix a bug don't go near it because if you fix the bug the whole thing falls down and certainly don't add any functionality to it so there is this idea that the thing on the that side over there it's going to start getting a larger and Messier it's just going to become a challenge so the sizing aspect is what we're what we've walked into how else do you create a big system we've talked about the idea of communication we've talked about the idea that sometimes the code artifacts themselves start attracting stuff but there's other ways we can do this and this again goes back to questions of economies of scale how do we reason about how we organize an effort so let's just imagine very idealized example this is how much time is going to take one person to complete all the work all the work I'm going to include in this all aspects of a system including reasoning about the requirements and all this kind of stuff deployment testing the works so we then apply classic division of labor we divide by n we've got n people on the project and most people stop there problem is that's not how it works not everything can be parallelized not everything when you put more people on it turns out they start stepping on each other's toes sometimes that's a physical sometimes it's in the code sometimes it's how they work you end up with this which starts looking a bit scary when you start showing this to managers it's like Oh equations it turns out though that's quite important how much can you actually do in parallel how many weight states are there how much flow can you get out of the system it turns out that Kanban people were right you can do this from first principles in fact we already knew this this is am dars law but I've got an robustly applied it to people it turns out that this is going to reflect on your architecture but it's also going to start creating is going to start exerting a force back on your architecture people will continue working even when they're waiting they will start creating things they will start creating defenses in fact it gets more interesting because there's more equation they will start creating defenses against the question of communication because it turns out you have to talk to other people I know who thought that was going to be a consequence you got to talk to other people that's the number of connections you have the more people you have the more connections you have in it shoots up and there is a cost to communication this roof is reflected in the code code ultimately becomes a form of communication code is a form of communication you're transmitting knowledge you're not merely telling a machine how to execute you are organizing your thoughts it's a collective effort to create a belief that executes this is how we believe this technology will run and answer a business need and the problem is it interferes with the nature of humanity if we are not careful you end up with a nice curve like that which kind of fits our intuition because it turns out them you know if you have zero people on a project it will take infinitely long to develop but hey no bugs but then as you start adding people you find some kind of optimal size that actually you have got the right number of people to do this work to do this undertaking the problem is that it's not a beauty very idealized people are not all the same they don't all have the same abilities and you can't see the curve and it's not perfectly smooth so you don't exactly know when you're here and you may find yourself wandering up the wrong side and this is how you create large systems this is a really easy way lots of lots of consultancies hang on quick check over here the sponsor list to see the vine whether or not I'm insulting anybody actually I might be but I'll keep them quiet a lot of consultancies operate on the right hand side explicitly in other words we've you took your small problem we've made it a large problem by adding enough people to make it a large problem I bet I could make this room develop a HelloWorld application the like of which the world has never seen before and would never want to see simply by scaling the problem up it's not a problem wants to be scaled up it's a small problem and it needs to be met with a small team and a small technology and the point there is we're very bad at doing this sizing but once you start running up the right-hand side then it becomes a self filling prophecy because eventually that system becomes the 60 or 70 person system and you can say well Hangul if we were to rewrite it how many people will we need well we currently got 60 or 70 people on this so maybe we need 60 or 70 people it's like no you need probably about 10 and we have actually seen these experiments run out in practice accidentally sometimes and actually seeing them in the real world where two companies or in one case one company that I turned a job down for because I thought there is no way these guys are ever going to deliver anything sane or sensible and I don't want to I don't to work for them and then nine months later a friend of mine went for a job interview there and so I had a chat with him filled him in I said this is what they were doing this is probably what they're doing now he so he knew he was walking into the Lions Den but for him it was practical they were 10 min 10 or 15 minutes walk from his house so it's actually for his point of view that's what he values being close to home ok so some time later he told me yeah yeah we've got about 90 to 100 people developing this system and we've just discovered that in the north of England the same company because remember companies are not single unified entities they have no idea what they're doing the left and the right hand are very very different in the north of England another development team had developed functionality or a system with exactly the same functionality with 10 people in about six months these guys had spent 3 years and 100 people and there are loads of examples like that over the industry they had created by staffing it up to 100 people they had fulfilled their own prophecy it did become a system that needed 100 people to create and once you are there it's very difficult to come back down again I also had the benefit there the joy of working for somebody in my many years ago um who actually knew this he took a team that was late and half the team size and they went back on schedule yeah you hear about these things but I can say first hand it's real I've seen it happen he was hated by the rest of the management yeah because everybody else was going by the the easy bit of the equation T 1 divided by n but he actually said now wait a minute I think we've created the size the size is not a reflection so there's a point here this size is not a reflection of the functionality once you reach a particular size anything beyond that is no longer a reflection of the functionality and there's this idea that a larger system has more functionality actually we've seen time and again this is not the case what we've seen is that above a particular size the rest of that size is there to compensate for something else it there to solve a problem of organization so it came across this one last year the facebook iOS app has over 18,000 objective-c classes 18,000 classes not 18,000 lines 18,000 classes and in a single week 429 people contributing to her now I'm going to take a wild guess I don't run on iOS a phone I've got an Android phone I'm going to take a wild guess that it's not radically different on Android but if any iOS users want to tell me what the extra few thousand classes are doing that I'm somehow missing out on please let me know because that is just insane I wished earlier on we looked at an operating system I know some trivial thing called Unix whose original kernel was less than 10,000 lines in fact the fifth edition it was 5,000 lines and I'm not saying everything can be made trivial but there is a kind of a sense of perspective that you need to get this does not only have its own gravitational field this is actually at the heart of a galaxy this is one of those black holes okay there are hundreds of millions of stars orbiting it as a result and that's you're just thinking about what so they doing I mean it's one of those things I don't want to make it sound too trivial in the sense that what are you doing well I'm looking at friends pictures and you know pictures of cats and goats and things like that and I'm posting some things and I'm liking a few things I mean how am I that I'm there's something really special going on there and I don't think that's 17,000 worth 17,000 classes worth of NSA stuff okay so we have this kind of interesting challenge that we are looking at a number of interesting practices there's organizational things but there's also how we approach things our habits are non-thinking where we started this cargo cult programming style of computer programming characterized by the ritual inclusion of code or program structures that serve no real purpose cargo cult program can also refer to the results of applying a design pattern or coding style blindly without understanding the reasons behind that design principle and suddenly without realizing it you've turned a one-line problem oh that's too easy you've turned fizzbuzz into fizzbuzz Enterprise Edition one of my favorite sites I did actually have to point out so I will point out to you this is a joke I mean it's got real code it does compile but it's not intended to be real you know there's tongue-in-cheek enterprise software marks a special high grade class of software that makes careful use of relevant software architecture design principles this project is an example of how popular fizzbuzz game might be built where it's subject to the high quality standards of enterprise software the whole thing is in quotes okay you have to read it like that because you end up just looking at one of the areas this is just some of the interfaces in one corner of the system yeah we're in the factories interfaces section and we got such fundamentally important plugins as integer string returner Factory and string string returner Factory oh yeah there are factory factories as well Andersen's law observes there are two aspects to this law I have yet to see any problem however complicated which when you looked at in the right way did not become still more complicated one aspect of this is that it's Turtles all the way down some problems genuinely are deep but the other is a self-fulfillment we can make a simple thing hard we have we have decades of experience doing this in fact we're not alone in this one the American architect building architect Philip Johnson observes architecture is the art of how to waste space and you know sometimes you look at something at it yeah actually that might might be the that might be a good description of what's going on that there's a very simple idea here Paul McMahon kind of observed all code is technical there some code just has a higher interest rate that's what you're interested in when people sub sale how much technical debt if you've got that turns out not to be the real issue it's it's it's it's part of the issue it's if you are dealing with a piece of code that has high technical debt or other sorry if you have a piece of code that you never touch then its technical debt is low or its interest rate is low because you never interact with it it does not cause you pain on a daily basis if you constantly interact with something that has high technical debt that has a higher interest rate because you it's causing you pain on a daily basis you're having to pay out extra every single day but that code that's not doing anything is it really not doing anything does that not mean that is it technical debt that we have a problem with there it or is it something else so I want to close with a sort of consideration of a example that one of the other problems that we have when we talk about the dead past the dark matter of our system okay in any large system will be the galaxies are not simply shaped by the black hole in the heart there in their heart they are shaped by dark matter stuff you cannot see code you do not understand but also code that is not executed Knight Capital Group 2012 experienced a slight blip I think would be an understatement to say on the first of August going from 31st of July to thirst of August on the New York Stock Exchange and there's a very good blog but dug seven wrote this blog from a DevOps perspective night capital group realized a 460 million US dollar loss in 45 minutes ten million dollars a minute excellent yeah that's the great thing about computers yeah they can amplify basic human human error it's very difficult for an individual human to actually lose that okay Donald Trump spent years losing twice as much as this okay but you you what you can do is you can you can actually optimize it computers can let you do the wrong thing faster ridiculous speed they realized something was going wrong like quick pull the plug 45 minutes later they did actually they actually lost more money this is how much was left this is how much the loss was after everything had been rolled back that could reasonably be rolled back and what happened the update too smart a system I think they had eight service was intended to replace old unused code refer to as power peg functionality night hadn't used in eight years so there's dead code this is the problem is that they had eight machines seven updates were correct one of them was not so it ran old code and what they did is they recycled effectively a piece of protocol we're going to reuse that because nobody's using it yeah because it's safe it's easier than adding something new we'll just recycle that problem as it has meaning to the old system and it not the right meaning yeah we just recycle old ideas but the problem is we left the old code in place now what I think is interesting and where I disagree with this assessment and what tells us a little more about the grasp ability the in your head ability of this why code that had been dead for eight years we're still present in the code base is still a mystery but that's not the point actually it is how did you end up with this dead code this is not dead well it's a zombie code basically if you think you have dead code it's probably zombie code what would cause this code to resurrect and come back to life and a particular set of circumstances caused this so it is actually the point because the idea is that this would not have happened it was not just a bad update it was a bad update and the presence of the code the point here is this was a perfect storm it was not this or that it was this and that what happens is in systems which we have effectively lost intellectual control over is you increase the probability of cases like this that's what's going on and then boom you get a big surprise so ultimately just a demonstrate there is no new idea Under the Sun I'm gonna go back to the 1990s I mentioned patents before which Gabriel key member of the patents community wrote this book no longer available PDF is available very interesting series of essays one of which he talks about his approach from 26 years ago worse is better I proposed a theory called worse is better why software would be more likely to succeed if it was developed with minimal invention problem is with the phrasing worse is better you immediately think well worse I mean it's the wrong name for it but when somebody says it is far better to have an under featured product there's rock-solid fast and small than one that covers what an expert will consider the complete requirements he's talking about something fundamentally different he's talking about a philosophy that fits the wiki way for UNIX way and indeed where a lot of people now think of as lean startup he focuses on implementation characteristics because that's where developers live developers will ultimately produce a system of course the customer experience is important but ultimately somebody needs to develop it and you need to make that amenable impossible it should be fast it should be small because then you can throw it away you can understand it and think of better ways of doing it it should interoperate with programs and tools the expected users already using minimal invention he's saying something very modest wiki the wiki way for example did not define a fundamentally new browser concept or paradigm but it helped us understand what we might want from one it should be bug free this is what worse means worse means fewer bugs slightly counterintuitive but there's that idea of if you want to make it smaller reduce feature set that's easier to manage and don't get carried away with abstractions very simple idea so I want to close with this kind of observation that people we talk about incremental development a lot I'm very interested in an idea called decremental development okay we should we should learn more about how to throw things out more about that was our first attempt now how do I size it down to its natural sizing and I'm sure over time it will get bigger the problem is that what we do is we start in the wrong place and I'll give Schumacher the last word thank you very much you

Channel: GOTO Conferences

Views: 66,889

Keywords: GOTO, GOTOcon, GOTO Conference, GOTO (Software Conference), Videos for Developers, Computer Science, Kevlin Henney, Keynote, GOTO Copenhagen 2016, GOTO Copenhagen, scaling, development, Software, legacy system

Id: B3b4tremI5o

Length: 67min 3sec (4023 seconds)

Published: Fri Dec 16 2016

Reddit Comments

[u/goto-con]

FYI, here's the talk Abstract

Systems get bigger, technologies reach further, practices mature, advice changes... or at least some of it does. Some guidance remains unaffected by the passing of paradigms, the evolution of technology or the scaling of development: break your software into small, cohesive parts defined by clear interfaces and sound implementations, all the while striving for simplicity. The excuses for not doing so are as predictable as they are problematic. But if you want to create legacy code, make sure code is arbitrarily partitioned into large, incoherent parts characterised by unwieldy interfaces and incomprehensible implementations.

While there are many definitions of what a legacy system is, common to almost all perceptions of a legacy system is the concept of size — specifically, lots of it. The phrase 'small legacy system' can be considered an oxymoron. This talk sets out to re-assess and reassert the importance of thinking and implementing small as an essential property of software systems — yes, even the large ones.