Don't Walk Away from Complexity, Run • Venkat Subramaniam • GOTO 2018

[Music] [Applause] thank you very much thank you for having me here thanks for the conference organizers for having me here I want to thank you all for being here this morning early so I want to talk about something that affects every single one of us involved in developing software our goal in in our work is to deliver value to our clients and we go to work with a good ambition of doing exactly that but a lot of times what happens is that things kind of become really hard for us to manage when we have to deal with a lot of different issues including one that I felt with quite a bit is complexity well I want to start by saying that when I was a child I used to tell a lot of scary stories now I'm grown-up I just talk about programming and that is a scarier scarier than what I used to say before and I'm a consultant I travel around the world I work with companies in helping them deliver software and what I'm gonna talk about here for the next about 15 minutes or so is purely things I've observed personally things have come to realize based on what my own clients and various other companies I've worked with have actually seen and have been dealing with and part of the reason why I want to talk about these things is if we can identify some of these issues when we go back to work we can maybe avoid or at least work towards improving these things some of these are familiar to us but a lot of times just because something is familiar doesn't mean we act upon it gentle reminder can help us a great deal to relate not just towards the right direction and so that's what I want to spend time on today is talk about things we can do and improve what we do in our work well first of all of course we all heard this statements over and over and over the only constant is change well I work with a lot of companies like I said and when I go to companies one of the things they want to tell me where he quickly is they say when could be our agile and I always tell them I'm so glad we got that out of the way now let's talk about what you actually do and what I want to really ask the question is what does it really mean to be agile and I'm gonna say the single most important thing for me when it comes to agile development is are we really being adaptive that's the most critical question to ask it's not about standard meeting it's not about scrum master it's not about a lot of things we do the question we need to ask is are we really adaptive if we are not adaptive then it doesn't really make any sense at all to say that we are agile and I'm gonna be saying the problem a little ashamed of the tweet I posted but it turned out to be the most important for stood because it probably got the most number of retweets ever - to the surprise of everyone that knew when I posted this and and the tweet is this and and my tweet was kind of random because I was actually having food with my children in a restaurant and suddenly this thought came to me and I tweeted this and and here's what I said and I said I've set the wedding date of not as they're out yet this is how projects are managed and if you really think about it this is so true about how software projects are really done isn't it you're working on this project and they tell me we're decided on the scope we decided when the delivery of the project is going to be and then I look at them and ask them then what's there to be agile there is absolutely no sense in saying we are agile when we have decided all of these things our friend and yet we want to really be proud to say we're agile I think agile is just a rewording of waterfall it just makes it feel better to say it so the question really is if we really want to be agile and I want to say one of the words I really emphasize when it comes to agile development is the word sustainability are we able to sustain the pace of developing software especially when we have to constantly change when the requirements change when the needs of the customers change when the environment changes when the libraries we depend on change everything around us changes how capable are we to sustain the pace of development that's the question we should really ask ask ourselves so from that point of view I'm gonna say complexity makes it really hard to adapt to change there's a lot of things that really make it hard for us to adapt to change but one of the things I've noticed consistently is that complexity makes it really hard for us to to change but let's not be naive about it I'm gonna say system design is complicated by nature when you're working on the banking application when you're working on that medical application when you're working on the tax application when you're working on that application that is monitoring airplanes that are in flight none of those is really things that are easy to do so we are dealing with enormous amount of complexity and complicated things is the nature of our life's to deal with we are in a profession where computer programming and computer software is is probably the most vital aspect of any any human society today because we use software in every single aspect of our our lives and and in fact we're getting to the point where we're doing more and more critical things with software where the cost of failure could become really high as time goes on so as a result we need to be appreciated with the fact that we are building things that are absolutely complicated but unfortunately though we take what's already complicated and we add complexity to that in and make things even worse so what is complicated becomes even worse when we add complexity to it and if we can find a way to remove that complexity I think we can do a lot better with it so in terms of complexity I want to talk about two types of complexity that is something we need to identify as developers one is inherent complexity the inherent complexity is the complexity that comes from the problem domain it's a complexity of the application the nature of the science and engineering that we have to deal with in the problem domain itself we cannot remove inherent complexity from the problem itself we can try to manage it but unfortunately as developers we don't deal with that most of the time what really hurts us is the accidental complexity so while inherent complexity is the nature of the problem we deal with it's the accidental complexity that really hurts but where does accidental complicity come from the accidental complexity comes from the solution we are using to saw problems and when we use a solution the complexity the solution introduces what we typically do we bring in more solutions to deal with that problem and those solutions in turn bring in accidental complexity and we get dragged into this vicious cycle and all of a sudden we are spending all of our effort dealing with this complexity that we kind of introduced to the problem then it being in there in to begin with so what I'm gonna talk about is how can we deal with them well we are the wick terms of our own complexity and if we can remove that complexity we can do a lot better and I'm gonna argue that we the people the programmers are they are the right people to do it because we are the technical people we understand these kinds of complexities and if we don't take the time and effort to remove them there is absolutely no hope in the world to make these manageable so it's our professional responsibility to be able to handle this so I'm gonna talk about things I've observed in my experience where does complexity really come from and what really makes software systems complex so the first thing I want to talk about is the first complexity I've seen is too many moving parts now this is one of the things I'm sure every one of us face every single day in our experience too many moving parts and if you look around your own application and identify what are those moving parts and I think this is only getting worse by the year that we are working with well as a beautiful code by Leslie Lamport he says a distributed system is one in which a failure of a computer you didn't even know existed can't render your own computer unusable if you really think about the world we live in today you're on your mobile devices and what the minute you click on the devices surface to the time you get a response literally there are thousands of failure points between those two points in time so when you click on the button and the time you see the response it's a miracle that actually things even work because there are so many things that are in there this is even getting worse today with the micro services where we live in because there are literally hundreds if not thousands services that are in the background that are just crying to fail at any given time and the more the moving parts the more complexities to maintain so one of the things I would challenge developers to ask the question is ask if that is the right technology for the problem at hand because your solution should never be more complex than the problem it's trying to solve and if we go down the path of creating more complexity we are really leaving things behind for other people to deal with I was at a client site recently and it occurred to me and I tweeted these words and when what I said was a technical debt is the gift a developer leaves for the generation of programmers that come after them and that is something we have to ask the question your actions have a very deep ramification for people who come after you for developing your systems and maintaining it and how many times have you sat down to curse the people that have gone before you and I would rather be curse less in my experience so if I can do a little bit to minimize that I think we're done a better job well one of the things that that I came across was a pretty shocking experience I was working with the client where they came into a market where they really had they say pathway to hell is created on good intentions they came with a really good intention and the intention was they wanted to customize and deliver a software for their experience experience and in doing so when I when they called me to do a code review and help them to improve their software I was in a bit of a shock because when I started looking at their code I'm not even kidding I could see so much code deliberately making decisions based on who the customer is that the product is deployed to and then when I asked them a few more questions they took me to a configuration file they created a software so that it can be customized so much that you won't even recognize it's the same software running on two different clients machines when I ask them very curiously how this works they showed me XML configuration file and you cannot imagine every single thing you can ever think of can be configured in that XML file and then they told me that they have a team of people who spend four months customizing the XML file so they can deliver the software to their customers this came to a point after ten years in the field every single change would cost them a lot of time and money and suddenly where they were really agile and fast now they were extremely slow and error-prone they just could not keep up with it and what I noticed was excessive configuration really came to kill their product and their ability to deliver software over time unnecessary layers and unnecessary complexity and components really made it hard for them and and when I looked at their configurations it was unbelievable and I realize configuration XML at least in their project I'm sure and other projects do a configuration XML is configuration from hell and this was really hard to work with and when I started working on it I came to a realization that XML is you know XML has been around for a while and I I was working in the field before the time of XML and when XML was introduced it looked like a bright idea for a very short amount of time and then you realize I realized something that X this is my analysis of what XML is XML is like humans they're very cute when they are small becomes really annoying when they get bigger and this becomes absolutely unmanageable and it becomes horrible for us to deal with and this is one of the things I noticed is that the more we end up getting dragged into this the harder it is for us to really sustain the agility of development so when it comes to configuring we should really ask the question is this really worth configuring or should we not even go over there so in other words when you look at an application ask the question what is the cost of developing this software if during development you find it hard to maintain the code if you're during development it becomes expensive to identify and fix problems if during development you find it hard to deploy these are early signs of warning there is going to become a total mess in the in the in the later James for the entire organization to support this software where the software becomes really brutal it becomes error-prone it's really hard to deploy hard to test and hard to reason and that is one of the things I want to really emphasize is this idea of hard to test and over and over and over I begin to realize this particular concern among among teams so we all work in in world where we use euphemisms and sometimes we don't tell the real nasty truth we say polite words so when I work with teams I often hear people say this software is hard to test now I know what they really mean when they say it what they really say is the design of well what they say when they say the software is hard to test is they say the design of this software sucks that's what they really mean so usually it's a really nice reflection when a system is really hard to test it's truly is a reflection of poor quality of design in it and and so as a result I always ask people I don't want a softer to be testable well that's an after-the-fact I want it to be tested and I don't want just to say in theory it's just able I want them to test it thoroughly using automated means that if you cannot do it that's a early sign of warning the design is really bad and we need to recover from that very quickly so from that point of view we want to really focus on reducing the complexity by avoiding the moving parts testability is also one reflection because the more of the moving parts it is they're expensive it is to test you or software as well that's a good reflection to deal with it very quickly the next thing I want to talk about here is something I'm still very angry about and and the reason I'm very angry about is there are two things I value the most in this world one is my time and the other is other people's time and anything that wastes our time is is really the worst thing we could do I think well invisible changes and I came across this and I'm still bitter about it like I said well a team came to me and they said they had written code in C++ and they wanted me to read in and this code using modern tools and and because I was aware of some of these things they said can you help us with this and I shouldn't have said yes but I really wanted to help I said sure and the next thing you know they sent me the C++ code just to set the stage I am NOT new to C++ at all in fact C++ is the one language I would say I probably work but the longest in my life I started programming in C++ a good 35 years ago I spent a good 15 years teaching C++ in the university as well my corporate life have programmed C+ was extensively in I've kept up with the changes in C++ I know what's going on in C plus was eleven twelve eleven seven fourteen and seventeen and I'm so pretty aware of C++ I can read the code but yet I read this code they had sent me and for the life of me I could not understand what was going on and the more I read it the more stupid I I felt and I got really frustrated because I could not understand what the code was doing and then I started doing something ridiculous I started stepping through the debugger and I'm here in the constructor as I'm walking through the constructor there is a function that has a name called get something and to me get something means it's gonna fit some data to you and I step over that I step over that one function and a boatload of State gets loaded into this class I'm like whoa how did that happen now I'm stepping through the rabbit hole stepping in the function and then another function very soon I'm lost where I am and as I was doing this I came to realize that I would never be able to understand this code and then I wrote to them and said do me a favor is this code written based on some kind of a research paper or algorithm that's published and they said of course and they pointed me to the research paper where the algorithm is described I abandoned the code read the algorithm I'm not even kidding with you read the paper wrote the code got it working Center to them and said never talk to me again and and this is how angry I was because it wasted so much my time trying to understand what the code was doing and then I realize when I you know we want to take your anger and convert it to some understanding and this is my understanding after the fact there are two kinds of code that frustrate me the most the first kind of code is the one that won't work and I'm staring at the same why won't you work the second piece of code is the one that works but shouldn't and both of these are horrible and I'm just absolutely looking at the scores like why does this even work and I'm not sure exactly why it's working and and then I realized that people who write code that makes invisible change or absolutely cruel to other programmers because it's a cost of your time your money and you were effort and we always say the code should be transparent why because you want to walk the reader and set the stage for they understand and anticipate what the code is doing and the further we go away from it the more expensive it becomes to develop software so invisible changes are yet another complexity to software so when we start down the write the code we have to ask ourselves are we increasing the accidental complexity by making these changes invisible or are we removing that complexity from code that's something we have to really ask the question and so don't sneak around and start changing state because that becomes really hard for us to deal with and this results in enormous amount of time wasted on the on the code as well and we can really have an economic impact by reducing that kind of complexity in code the next thing I want to talk about here is about uncontrolled mutability now think about this for a little bit we all know mutability is not the most pleasant thing to do but if you go to any Java programmer c-sharp programmer see forceps programmer almost any other language that we use in the mainstream and you tell them don't do mutability they kind of look at you strange and say what are you talking about we live in the world of mutability so I'm gonna say mutability is okay we all do this but what about share while sharing is a good thing remember what mom told us you have to share that's that's that's what you should do has a nice person so mutability is okay sharing is good but shared mutability is devil's work and the minute you bring shared mutability all kinds of things fall apart it becomes a nightmare to deal with so what can we do to reduce one kind of complexity is to minimize mutability as much as we can but when it comes to mutability i'm gonna say there are certain things in life that are predictably irrational i want you to spend a minute thinking about it well when java was introduced what did we have in java we had a thread class i know we don't use thread class today but we use executor services and other thread pools but that didn't change one thing fundamental what did we send to the thread class we often sent a runnable and today with executor services we still use runnable quite a bit but what do we do with runnable well think about the runnable interface if you will what is the method of the runnable interface it is public void and then it says run now just tear at that for a minute what does that interface tell you I will not take anything from you and I won't give anything back to you how rude if you really think about it you sometimes work with people like that isn't it they won't listen to you and they won't talk to you either and how do you communicate when you have a piece of code this is unbelievable because the only way you can effectively use runnable is by adding shared mutability now think about that for a minute the only way to use this API is to do the very damn thing you shouldn't be doing in the first place now how do we deal with how do you cope with logically systems like this so it comes out that from the get-go this really makes our life a lot harder to deal with when it comes to this of course I'm gonna say mutability needs company it often hangs around with bugs so the more mutability we introduce in code the more difficult it is to maintain the code the harder reason with that and the code becomes error-prone as well so any effort we can put in to reduce mutability is something we can actually benefit from quite a bit and I'm gonna go to the extent of saying that state transition causes brain damage and and the more we deal with it becomes really really hard now you know one statement that we often hear from people people say programmers are weird programmers are antisocial and things like that I completely disagree with that I think programmers are definitely socio but among the right kind of people in my opinion and it turns out one of the reasons why we appear like that is we spend a lot of time thinking a lot of what we do is in our head and and I'm sure you have felt this quite a number of times in your own experience you're sitting at work and you are coding away maybe there's some music playing around you doesn't matter but you are coding and you're focused on it sometimes you're looking at the floor you looking at the ceiling and you're just thinking and what you've been doing at this point is in your mind you have been constructing this state in your mind in your mind you know this object is talking to this other object which in turn is talking to this third object which is about to send message to the fourth object and you are in this beautiful state of unbelievable understanding of how the system works and act that fateful moment somebody opens the door and says have you had lunch and you just stare at them you're looking at them and you're not speaking and they're like I'm sorry I asked you have you had lunch and you're like what and they look at you and say you're weird and they walk away and then you're like very angry at this point and then they open the door and say are you okay and you're like no and you have to put this all back in your head now because all this model you created has been shattered and it's on the floor and only programmers understand this you could have two people sitting and working together and they have two shared model in their mind and the third person interrupts and asks something unrelated and they both get angry at this person now this is something I've learned over the time arm or I should say my family has learned over the time and my wife to it there's nobody else in the world I respect more than my wife and the reason I have such a respect for her is she is the most patient person I ever know she didn't know this she thought she married me but then she realized she married a programmer and this was a distinction that she had to really go through when we were married knew there was one time she came to me and said I'm gonna go shopping and I told her I would be done with this in five minutes if you want to wait and then I saw her pick up the key and leave and I was kind of little bitter and when she came back in the current said I'm sorry sweetie I'm little upset I told you I'll be with you in five minutes but I you just kind of left I was a little you know angry upset about it and she put her hands on my shoulder and said sweetie let me tell you this once because I'm gonna say they say this only once to you you don't understand this when you say five minutes that is usually seven hours in the real world and that's what I understood programmers have a different personal perception of time how many times have you told your boss I'm almost done and the boss is like having the frustration of life and saying what do you mean you're almost done you say this every single time and you're like seriously I'm almost done we have this absolute you know perception of reality that's skewed from what the world is looking at but I'll tell you the best money that I ever paid for in my life was to go watch the social network movie I'm gonna play an audio here so let's make sure the audio is up on the computer so I'm gonna play a little audio but this one right in the middle of the scene my wife looked really screamed so I'll play this and I'll then talk about it so here we go I didn't know we had it or done and you get the door now he's wired in he's wired into security buzzes boys I'm Sean Parker oh he's wired in that's what I'm talking about so right then when Sean Parker I heard he's wired and my wife literally screamed and is like are you okay and she's like now I understand when you're working you're wired and I'm like take my money this is worth it and a week goes by I was working in the kitchen when one of my kids came around and asked a question and my wife said don't talk to that right now he's wired in I'm like yes and and this is one of the most important things I realized is when people understand how programmers work and I'm gonna say happiness is when the world understands programmers because then I think we can just live in harmony and when you sit there and write your code and you are in absolute sync with your code and only another programmer can truly understand and appreciate that but when the family begins to realize that the family knows what really means to you as a programmer and they're absolutely supportive to us so so one of the things is that this mutable state we have two Lord in our mind and all that complexity really introduces what we call as cognitive load and the more the cognitive load is the more expensive it is to develop software so if there is a way we can reduce this cognitive load I think we can do a lot better when it comes to developing software and making it easier to maintain and in the process making it a little less expensive the next thing I want to talk about here is lack of cohesion so what is lack of cohesion let me ask you the question how many of you believe long methods are a good idea raise your hand if you think so not a single person raises the hand let me ask you a different question entirely how many of you have seen long methods at work now the hands go up right away isn't it this is called cognitive dissonance but I got a good news for you those long methods I'm here tell you you didn't write them I mean you just told me a minute ago you don't believe in long methods but the bad news is those people who believe long methods are a good idea or at work today making those methods longer as we speak well the point really is that's an example of lack of cohesion you look at a long method and somebody asked the question what does this method do my question always is what does it not do because it is doing so many things after all it becomes really hard to maintain such a piece of code so we want to really focus on cohesion now cohesion is where a piece of code is narrow focused and does one thing and one thing really well and I remember this experience working with the client and this particular company has a software they've been maintaining for 30 plus years and and I'm absolutely humbled when I walk into their building because they have people working there who wrote the very first line of code on their software and I would go sit down next to them and they would look at me and say you know I wrote the first line of code in that software and it's very humbling and these people are still working they're trying to decide if they would retire first or die first and and this is just amazing to see software in one version control for past 30 years and one of the developers called me and said let me show you something Bank and he took me to this computer and he takes this version control shows me a function 15 years ago this was the day the function was born I could see this beautiful function its entire body in front of me and such a small little fingers cute as you can think of very small only two parameters and I was like wow Thomas mccabe's cyclomatic complexity value of 10 is considered to be high this function only had a complexity value of 5 and I was like tracing this function the day it was born and then he looked at me and said don't get carried away venket and then he perhaps the timeline and moves it to the current date and as he was doing it in front of me I saw that beautiful little function literally turned into a monster and don't even ask me how many parameters this function took today and I could not see its bottom anymore and then he showed me the cyclomatic complexity value of that function a value of 864 and then he said that's what we do to code in this company I'm like where's the nearest door I want to exit it is kind of scary if we don't leave factor the code and let it grow into a monster over all that time well one of the things to really think about is how can we really make the code reduce the codes complexity if you will and how do we really reduce the complexity of code and what is the reason for it one reason is it helps us to reduce the frequency of change of the code and that's one of the reasons to do it and so if we can create modular code and whether the modularity is in a function or a class or a component or a micro service whatever that layer of granularity is if we can make it modular I think we can do a lot better and keep it very cohesive that's one thing we can do to reduce complexity but the next thing I want to talk about is excessive dependencies and every single one of us have to deal with it every single day now dependencies are natural we have to live with dependencies but we also have to govern it quite a bit as well because dependencies can become really hard to maintain it well I'm gonna say when I started programming a few decades ago where I lived in a different world than we do today but there is consistently something I've noticed in the past about 30 plus years I've been writing software and that is I've held I've had a one hell of a programming career from a DLL hell to jar hell through assembly hell - and p.m. hell what is next is the question I want to ask but NPM of course is special NPM will the minute you call it will download the entire internet with vengeance I was on the flight the other day and I had my laptop turned off I was just sitting and having a good time and the captain said folks we have a very smooth flight I'm turning off the seatbelt and if you want to get around please feel free to and I shouldn't have done this I opened my laptop and the only thing I did was I typed NPM install turbulence immediately and the lighter seatbelt glare sign comes on I'm like sorry and I turned off the laptop never to open it again on that flight and it's kind of scary when you see things like this but the other day I was working with the client of mine and he said oh we used to run all the stairs but we quit running them recently I said why would you quit running your tests he said because we upgraded some of our dependencies and the tests started breaking we don't have time to fix it I looked at the and this was a JavaScript project and I said gosh how many dependencies do you have without missing a beat he said oh we got about 150 dependencies I said for this small project he kind of shrugged and said aha and I said why are you using 150 dependencies on this project he kind of said matter of the faculty he said because we found them and that is such a true thing isn't it you are out to lunch with your team you're having food in the next table somebody talked about this X Y Z package and you're like have you heard about it 30 minutes later it's in production code right and this is the world we live in we drag dependency like we are crazy well the point really is I was mentioning this the other day and a developer told me oh we don't have this problem I was like surprises that really how do you deal with dependencies then if you don't have this problem and he said oh we use maven he said I had to remind him nobody uses mavin mavin uses you so the point really is and of course to be fair it's not the mavens problem it's the dependencies problem dependency uses you through maven and you get dragged in you get sucked in you get beaten down and that's what happens to you when you deal with dependencies so the point really is we have to manage dependencies that's what we have to do otherwise it becomes absolutely ridiculous to deal with so what can we do about it well dependencies are hard to maintain and they become obsolete very quickly and they become incompatible so we have to be very careful and dragging in dependencies it takes maybe $1 to bring in a dependency but I bet it takes a lot more to take it out it is easy to drag things and very difficult to rip them apart and a lot of times the people were trying to rip them apart or not the people who brought them in this is something I've noticed as a consultant over and over and over anytime I point to a problem and ask the question why is this like this the team's immediate responses event if we want you to know the people who wrote that no longer work here this is how this conversation always starts isn't it because it's somebody else who has left us all this legacy for us to deal with and now it's our problem whatever it comes to this I want to mention about one other thing that I think we have to be careful about and I call this as technology infatuation I'm not here to tell you the past was better than today but in certain ways it was different and we were able to manage things maybe a little better so in is the technology the right choice for you you have to ask the question and one of the questions I will always start with is reversibility how reversible is the software decision you are gonna make a design decision you're gonna make how can you reverse out of it if it is easier to reverse out when you realize this was not the smartest decision to make if you can reverse out of it we are in a better shape we can make the decision a little lightly if it is hard to reverse out we have to take more time to commit to this decision for example the ability to back out of the design or architectural decisions now in that regard think about libraries and think about frameworks now libraries essentially are a little bit more manageable the use of a library is easier to reverse than the use of a framework well that's true but we have to be a bit careful I would ask the question about the surface area surface area for example if this is a library and if my code depends on it like this my surface area is quite big and if I want to rip this library out there's a lot of ripping I have to do because there's a bigger surface area connected to it on the other hand if I plan a little ahead I can keep my surface area relatively small towards this library so tomorrow if I decide to modify this and rip it out and replace it it's a lot easier for me to replace it because a surface area is relatively small so when it comes to libraries we can make reversibility a lot easier by managing the surface area unfortunately though frameworks are very different from libraries you call into a library but the framework grows around you and when the framework grows around you it becomes a lot harder because by definition or by their nature the surface area is a lot bigger in that regard here's a dumb analogy if you really think about it using the library is like dating using a framework is like entering into marriage be very careful in making this decision one you can flirt with and you can make it change your mind the other one is going to be really hurtful and expensive so we need to really think about this is this what we really mean is this what we really wanted and then we have to make the decision not to lightly though so that is one thing to think about in terms of reversibility one is more reversible or easier diverse about than the other one and we have to be a bit careful in how we make the decision moving forward so when I was young I used to there was there was no internet and this is always fun to talk to children about they always look at me what do you mean you don't have internet when you were a child well if I wanted to use something I would read about them in magazines and then you go to the procurement department ask for the permission to acquire it and then when they eventually deliver it you had a tape drive you have to go to and install it and inspiron it through it when a tape drive is involved between you and the software you don't download and you know you don't use software so quickly today we live in the world of downloading in a click of a button we can have all the software we need and so it is a lot more easier to integrate and bring things in but we also have to ask the question are we really increasing the cost of doing things this leads to what I would like to call as a resume driven development and a lot of times when you look at people you ask them why are you using this one oh because it's great to have this on my resume a so we have to ask the question is this needed for the product we are developing well I always say keep the learning separate from your production code we have to continuously learn and I would create small projects pet projects and site projects I would learn the most with new technology and have the wisdom of not throwing them in production until I find out that it is really the right choice so we have to kind of keep a two-prong approach taking time to learn separately and then be very prudent in bringing technology for adoption into our own projects but I'll share with you once a story that that breaks my heart and I call this as the tale of infatuation this was a time when the economy was really bad in the US and and we were really having a hard time hiring people for our project as well at that time we were interviewing candidates and we came across one candidate who was absolutely amazing and we were on a project for a very small company this was not a mega billion corporation it was a small company and as was working with them we wanted to really hire somebody we had a product to release in the next six months and when we interviewed this candidate I was blown away by this candidate and my boss came to came to me and said hey what do you think of this candidate I said oh don't waste your time talking to me hire him he's the best candidate I've ever seen and then my boss said yeah yeah but what do you think of the framework is being developing I said that's the best frame you ever seen but don't waste your time talking to me go hire the person and then the boss said what do you think of us using the framework I said you know sit down for a minute don't even talk about it we don't want the framework hire the person for what he can deliver let's get the project moving let's finish this project six months later you want to do anything he warned I'll give you the full support we'll work together to do any of those he said benka did you notice how this framework gave us a persistent messaging I said you never used the two words together in a sentence still today why are you saying this now and he went on to say many more things and I kept saying you never sent this is that this is one sentence don't do this don't talk about the framework hard the person a week went by I remember very clearly Wednesday afternoon I'm sitting and coding at my desk my boss comes and sits on my desk and says I got a good news for you banquette I said what is it we hired the person he has agreed to come and work with us I said I can't be happier I'm so excited because I'm gonna learn so much from him I'm absolutely selfish I'm here to learn and we will be gonna have fun times I can't tell you how happy I am and my boss said oh as part of the hiring contract we decided to use the framework I said I please tell me you are kidding it's like no no no this has been committed already I said well if that's the case I just quit he said oh I know you're gonna say this I already told my boss that you're gonna quit the minute I say this to you I said at least I'm predictable I'm happy for that well good luck with that I packed my stuff in five minutes I was out I said if there's any other project called me I would like to come and work with you guys but this is not the right project for me I don't want to be here telling you every day you made a mistake and this is one of the few days I went home on time my wife of course asked as the person she is look at me and said you are home early I said yes I quit and I and she said what does that mean you quit I said I don't have a job that's what it means and she said in that case I got some dishes for you in the kitchen and that's basically how we did this for a couple of weeks I was home doing dishes and then I found a job and I moved on but 16 months later I'm not even kidding with you 16 months later we were supposed to finish this in six months sorry 18 months later a year and a half later my phone rings hello hey man cat hey how's it going go ahead and say it I said what do you mean go ahead and say it go ahead and say it I told you so I said I'm really sorry didn't want to hear this what happened oh we spent the past you don't have writing more code for the framework remember this framework we didn't need in the first place and we have not still released the product I said I'm really sorry to hear it but what can I do for you why'd you call me I was wondering if you'll come and help us I said no there's a reason I quit there's no reason for me to come now when it's in a bigger failure so sorry and and and six more months spent by two years after I quit the phone rings again and he says come over just one day write a report and leave I said wow you guys are done oh yeah okay well what do you want me to do well very well pay for the day just one day gig comment write a report and leave and I went there aid in the morning and I sit in the front the computer and there's a little instructional to use it I entered this little key I click the button 25 seconds later the window popped up and I climb it again I profile it and I said look that clicked when he took 25 seconds he's like uh-huh and then I did a profile and I said that click resulted in a creation of 1863 objects he said its object oriented and I said database don't talk about it this is the fastest you will get processing power don't talk about it is the fastest you will get in production it's all much slower and then I said where's the team he wouldn't answer I said no tell me where's the team and that's when I knew they fired the entire team so here's the software that purely performs and nobody to work on it because they were so angry and then I looked at him and said you didn't call me to write a report did you why did you call me here he said fix it I'm like sorry I'm not a magician I'm just a mortal programmer and I had to escape from the building but the point really is it's very scary when people get infatuated and so don't build what you can't download but please don't download what you really don't need and this is one of the problems we are living with today is that infatuation to technology and I always ask the people is what kind of decision you made how much did you put effort into deciding this is the right software to use and if we didn't take the time to decide we are going to become a prey for the technology infatuation and that's gonna hurt us a lot so I'm gonna talk about accidental complexity let's take a quick look at one example here we look at a problem and said to ourselves I can use a pool of threads now we have a pool of problems to deal with now here's an example a boolean variable done and it's false to begin with I have a loop right here as you can see which is just looping through and waiting for them to turn into true hey this is all great and eventually after two-second delay I'm setting true to done to true if I run this code it should really work isn't it well that's actually called the hope if you will so let me go ahead and run the code you can see the program starting up in Maine it's running it's set the done to true but it never completes after that now why didn't it complete and I will add insult to injury here just go back to this code and say try right here and a catch exception and to add insult to injury simply says over here a thread dart sleep zero how about that just to sleep zero and then of course when I run the code after a thread sleep zero you will see that the program actually quit why would a zero off asleep actually make a difference well the reason for this is the concurrency model in Java was built based on the Java memory model a lot of developers did not understand the Java memory model when they actually started using it and this can be really problematic and as a result the complexity of concurrency is so enormous that most of us cannot really comprehend it and I would argue that the JDK concurrency model is the assembly language of concurrent programming how many of you hear programming assembly not not as well a few people I would thank the people who raised the hand you are doing it so the rest of us don't have to do it and and that is the whole point concurrency should be exactly that a few people have to soldier through it so the rest of us can have a peaceful life after that so a key design skill we need to develop is the ability to discern accidental complexity from inherent complexity in the same token I'm gonna say imperative style of programming is more complex than functional and decorative style of programming and so imperative style is packed with action little complexity but that's what been doing for a good 30 years as mainstream programming is doing imperative style of programming and if we can really go towards decorative and functional style we can ease the pain quite a bit and decorated really reduces that accidental complexity the last thing I want to talk about here is called employment and this is something a lot of us fall into a trap employment is where we take two ideas that do not belong with each other belong to each other with each other and they mix them together now we make this mistake but the folks who were behind the JDK made this mistake as well here's an example a resource class uses an external resource you want to clean it up properly maybe a file a database connection a socket whatever how do we do that in the past we write the finalize method but the problem is the finalize method is not called on pill the garbage collector kicks in the garbage culture will not kick in unless it needs the memory now we are tying an automatic garbage collector which is not deterministic to external resources this is a terrible idea and in fact it's so terrible that JDK 9 deprecated Java 9 deprecated finalized in fact rich Hickey puts us in a nice words he says completing things he's the source of complexity he says there's a new word he invented called complexing but it really stands for employment when you're taking two things that don't belong together and you mix them together really bad things happen and we see this a lot in our own programming if you ask me when could have you ever done this in your code I will give you examples oh yes I've absolutely done this a problem error in my code and I've suffered the consequences of it and and we see this a lot in a lot of different areas we definitely want to remove this kind of complexity from code by removing employment and in the case of Java they realized it and they got rid of it in Java 9 but they've been working on it for several years so we talk about these kinds of problems when it comes to complexity too many moving parts in possible changes uncontrolled immutability lack of cohesion expense a very excessive amount of dependencies infatuation towards technology without determining the real need a low level of concurrency usage imperative style coding too much of that and then of course enplanement I'm gonna finally say we should learn to deal with complexity but we should also have the wisdom to minimize it and this is a responsibility of every single one of us war calling our sis technical because only technical people truly understand these and only we are capable of really reducing and dealing with this so we should learn to deal with complexity and have the wisdom to minimize it maintainable code I'm gonna say is a gift we give ourselves for the future hope that was useful thank you you

Channel: GOTO Conferences

Views: 317,047

Rating: 4.3633442 out of 5

Keywords: GOTO, GOTOcon, GOTO Conference, GOTO (Software Conference), Videos for Developers, Computer Science, Programming, GOTOcph, GOTO Copenhagen, Venkat Subramaniam, Agile Developer Inc., agile, complexity

Id: 4MEKu2TcEHM

Length: 52min 4sec (3124 seconds)

Published: Fri May 17 2019

Reddit Comments

[u/craigbrownphd]

Amen. I have had clients propose complex solutions for not so complex problems. Things can look easy during the early phases and I have had my share of Ph.D. level proposals that in theory would work. Include your developers and engineers in these discussions so that they can chime in on what is realistic and what is not.

[u/mto96]

Check out this 50 minute talk on Complexity from GOTO Copenhagen 2018 by Venkat Subramaniam - Award-winning author, founder of Agile Developer, Inc. I've pasted in the talk abstract below for a read before you dive into watching it.

​

We constantly hear that change should be affordable and cost effective. True, but, in reality, that's easily said than done. Complexity makes change hard. We can't shy away from the hard problems posed by domains and business needs. So, how can we solve complicated problems without getting dragged into the quagmire of what appears to be an inevitable complexity? In this keynote, an award winning author and software practitioner will share experiences and observations from working on multiple software projects, about what leads to complexities, the traps developers and organizations fall into, and what we can do to effectively deal with these common, recurring issues we see across domains and products.

[u/zyxzevn]

Those who designed C++ should have watched this.