Do not walk away from Complexity, Run - Venkat Subramaniam

thank you thank you very much for having me here the wonderful folks who run the conference I really appreciate it thanks to you all for being here I want to talk about something we deal with every single day in our work what really makes our lives really difficult I want to talk about complexity but before I do that well when I was a child I used to tell a lot of scary stories now I just talk about programming and that does it and it turns out that we have a very challenging life to live we deal with complexity every single day at work and this is what we have to tackle and we have all the good efforts we put in we want to be successful we want to deliver applications but the things that we create today ends up really affecting what we can do tomorrow and that's what I want to talk about today is what are the things that lead to complexity and maybe how we can actually minimize such complexity in our programming life that we do and what I want to do is to identify certain things I've come to experience this is more of a sharing of my experience through my project that I worked on for clients over the years and and if you ask me what do I do for a living I'll be honest about it I actually do more of code removal than code adding these days I go to clients and I find that they have written way too much code than they really need and often times when I leave they actually end up having lot less code than when I walked in and one of the things I fear the most is unnecessary complexity and accidental complexity that people bring into the applications well one of the things we often hear about constantly is that the only constant is change but we live in a world of agile development these days so I want to ask the question what is really agile development well agile development really agility is all about our ability to adapt to change that's what we really are looking for and the apt ability is really the key in this case well if adaptability is really important we want to do things that make it easy for us to adapt but what do things really make it hard for us to adapt is the question we need to really ask and it turns out complexity makes it really hard to adapt to change and so if we really want to adapt in the future one of the things we have to really look for is to find and remove these kinds of complexity that make it really hard to adapt as we move forward because the change is the only constant well then we want to really be able to afford the change embrace the change even late in the game but how do we do that well it turns out we cannot be naive about it we need to really understand that the world we live in is not something that's going to be trivial we work with applications that are significant we work with applications that are in the banking industry we work in applications that are dealing with satellites we work with applications that deal with transportation imagine every single aspect of human life is now touched by computer programs I don't think I ever thought about it that way when I started programming but to realize one day that what you do affects every single aspect of human life on earth is just pretty amazing for us to be in this field but it turns out that system design is complicated by nature well there are things that are going to be complicated no matter what we try when you deal with your business the things they want your application to do and the minute you think you've got a grasp of it they come back and tell you that's not it and they tell you more that you have to deal with and you have to constantly deal with this this is complicated business we have to deal with but notice that the applications we develop are already complicated but unfortunately really sadly though that the complexity that we create really is what we add to this what is already complicated so we don't want to really add complexity to what is already complicated we want to really minimize that so what I want to talk about here today is how can we really you know maybe do that so I want to say that we are wickham's off of our own complexity so complexity often comes from the solution we create and I want to separate between the complicated systems we deal with and the complexity that we introduce and if we can be smart about it and eliminate that complexity we can do a lot better so I want to talk about the question what makes software complex and and every single one of this I'm going to talk about today is what I've observed in my experience working with different companies and and these are things I've kind of charted down in memory lane over the years and I kind of captured that in this particular talk so let's talk about the very first thing I've taken note of and that is moving parts do many moving parts make things really complex now when it comes to applications this is only getting worse by the day everyone wants to build micro-services today whether they want it or not that means more moving parts to what we are building for example and the question is are we really up for that complexity is that complexity really worth it is it what we really want to go after does it provide value is something we have to ask and there's a beautiful saying by Leslie Lamport he said a distributed system is one in which the failure of a computer you didn't even know exists can render your own computer unusable and this is so true in the world we live in today he wrote this a few decades ago but this is so true today as much as any other time and and that is one of the things we have to deal with and I like I said with more complexity we build in this only becomes reality but part of what we also do is we have fire to build applications that are extensible and I remember one particular client that I worked with that resonated really well with this problem they wanted to build an application that was extensible their desire was to build this application that they can deploy for any customer and it would look exactly what that customer wants so in their aspiration to do this they started configuring every single thing you can imagine there was nothing in this application that could not be configured so this was excessive configuration in this application and they had way too many components and way too many layers not a single person could really explain how things actually work and when I went to work for them and I asked them how do they use this application and I found out that when they want to deploy this application they would send a team of engineers who would spend their life configuring this application they spend more time deploying than actually developing because it was so much configuration and how did they configure you probably know this already right it was configuration in XML now we know one thing about configuring an XML can freaking in XML is configuration from hell because that's what we have to deal with is there anyone here who loves XML not a single hand went up here's my understanding of what XML s and this is my observation from my own use of XML over the years and I've come to realize that XML is like human it's really cute when it's small and gets very annoying as it gets bigger and this is something we are to deal with and can you imagine in this particular application there XML configuration was so big that nobody survived it and they have to spend months at the end on end to configure this to deploy it and eventually this resulted in the demise of the company actually because when they could be really responding to change early on it came to a point where they couldn't respond to change anymore so freaky zhilie because they're engineers got sucked into this XML never came out alive and and that is one of the things we have to keep in mind as we don't want to get into a configuration hell we want extensibility but we have to really look at the cost of how much effort it takes to really make things extensible that becomes really important as well so the application becomes brittle it becomes really error-prone you change something here and something breaks over there it's really hard to deploy very hard to reason when you ask people how does it work they're not able to describe to you how it works and one of the things I often look for is testability here's an early sign of warning if an application is really hard to test that's a disaster in the making in general so it always look for how easy it is to test this application during development and that's a sign during production as well so one of the things to really ask for is what are you moving parts more than moving parts in your application the harder it is to maintain the application as well the next thing I want to talk about here is invisible changes and I'm a little bit bitter about it even today because when I came across this was a team contacted me and said we need your help could you please take the C++ code and rewrite it using you know spark for us the logical thing I should have said is no but unfortunately I said yes and then they sent me the C++ code and I started reading the C++ code just before I say that let me say this I had spent 15 years teaching C++ I have spent several years working on C+ project I am current with the latest standards of C++ I still teach courses on C++ it's not a language that's alien to me but I looked at this code they sent me and to be fair it was not a very large piece of code only a few several hundred lines of code and I read through it and I write through it and I read through it again and every read it I felt more stupid and I'm like why don't they get this haven't gotten really old for this stuff and I'm reading it again and something is completely illogical because the code is saying one thing and the application is doing something completely different and I was trying to figure out why does this go do this and I came to realize something over time there are two kinds of code that frustrates me a lot the first piece of code that frustrates and frustrates me is the one that doesn't work and it's very annoying but even more annoying is the one that works but shouldn't and this is very painful and this is exactly what this code was and I'm looking at him like why are you working and I don't understand why this code actually works and I'm reading it and reading it and reading it finally yeah I open up the debugger and I start stepping through the code that's called an act of desperation isn't it and I come to the state of the object and there is this innocent method which does some get something and I step over that method and suddenly this object turned into a monster with a lot of state populated and like damn how did that happen and now I'm stepping into that function and you know how that goes then you step into the other function and then beep down like seven levels later I'm looking at that code and saying rascals why did you do this here and they had completely manipulated the state in a place where it would least expect and the problem here is when you read a code like that you feel so cheated as a programmer because you're like why didn't you just make this obvious to me because I spent the last few hours trying to understand how this code is working and if this was obvious I wouldn't have spent this much time and that is one of the things to really ask the question so my my observation at the point is don't sneak around and change state so I want us to be extremely obvious to the reader one of the things we need to keep in mind is we write code once but we read it several times it's no point in optimizing the code for writing when the real cost isn't reading the code so I always ask the question what is the reader of the code what is their mindset what are they looking at what is their chain of thought and we want to be able to lead the chain of thought with the reader of the code that is absolutely important so if we sneak around and change the code the and the state in the code it becomes really hard to maintain it so and what is the result of that the result of it is last time last time is lost productivity last time is lost money for projects and so what did I do in this particular project I'm not even kidding with you I started reading the code again and again and again and then I realized I will never be smart to read this code so I wrote them an email and said dear do me a favor this code you sent me is that a research paper that this was based on and they said oh of course and they sent me a link to the research paper I deleted the code honestly and read the research paper wrote the code in sport sent them to this and said here you go it works in sport and don't ever call me again and the point really is it's not enough waste of time and this becomes really frustrating we have to really ask the question how much time time and money are we wasting on stuff like this so so let's avoid invisible changes so do many moving parts invisible changes we talked about the next thing I want to talk about here is uncontrolled mutability now we all have done mutability in code in Java it was the way of life for us to do mutability but unfortunately as years go by I realize the more we mutate the really harder it is to understand what the code is actually doing and this becomes really hard to explain and reason with the code I remember this day when a programmer walked into my office and said hey here's a function I wrote it doesn't work and you help me and I quickly eyeball the code and said if you send this in petition tree view this output is that yeah I know right it doesn't so we both are reading through it like crazy people and after a few minutes I looked at them say hey did you mean to change the input variable in line number seven he got it for a minute stare at me and he said I'm stupid and he walked away and this is really hard because it's really hard to even think that you're modifying a variable at that point in the code so mutability needs company it often hangs around with bugs and so there is a correlation between the amount of error we introduce in the code and amount of mutability that we end up having in the code in fact I'm going to go to the extreme and and say that mutability actually causes brain damage and and this is something it's going to be really hard for us to work with as time goes on on projects so the less we do mutability the better it is for all of us but this is one thing that we all realize let's for a minute think about the life of a programmer if you will this is what we do isn't it when your code has all the state transition in it what do you normally do you're sitting there and staring at your code but when you're doing this you're built this entire model and the current state in your mind and as you're walking through the next code you are detaching this and attaching to this other object and you're in this harmony at this point and when you are doing this suddenly somebody opens the door and ask you the question you want to go for lunch and then you just stare at them and you're like what and they're asking you you want to go for lunch and you're like sorry and they're like you're weird and they walk away and then they come back and say are you okay you're like no look what you did and now you're putting this back together because they completely shattered the state and now it's gonna take you 30 minutes to build that state back in your mind only programmers understand this isn't it because when you're looking through the window you're not looking through the window you are just adjusting the state in your mind it's so fragile and anybody ever ask you a question it's completely shattered and then they look at us and say you people are antisocial they just don't get it and I'll tell you the best money that I ever paid in my life ever hands down was when I rented from my wife to watch the movie social network this was worth it I didn't know this guy coming I was right there sitting in the theater watching the movie and that was one scene she literally screamed right in the middle of the movie and said oh my gosh I get it I'm like gosh what did you kick just like I get this and here's the scene that she got really she got it in this scene so I'll show her shares our scene here with you hopefully the audio works so this is the scene right in the middle of the movie so here we go the audio is not working can we try the audio game please that audio is working let me try this nope it's through here oh I need to change the audio settings you know I'll tell you what every programmer says at this point it worked before I totally did so let's see it says headphone I'll change it to that and see if that works alright let's try this again here we go alright it doesn't want to work let's try this one more time otherwise I'll recite for you what it's doing so that seems to be out output yes yeah it isn't HDMI let's change to a headphone and see let's try that again there you go that's awesome isn't it ok alright so let's do this again [Music] I'm Sean Parker oh he's wire it is that's what I'm talking about so that's what I'm talking about that particular scene is when she screamed and said oh my gosh I get it I'm like what are you getting and she said I know what you do this when I talk to you you're wired in I'm like take my money this is amazing when the family gets it and a week later I was working in the kitchen and one of my kids came to me and wanted I have a question so daddy and my wife said don't talk to the hurry right now he's wired in and I'm like yes this is amazing and that's one thing I really you know realized that when people understand when you're staring at a computer or staring through the window it doesn't mean you're staring you're not doing anything you are managing your scale in your mind and and this is one thing I would say is I've been a programmer longer than I've been married and when I got married I was real fearful because I feared I have to choose between being a programmer and being married thankfully I didn't have to do that for two reasons one my wife is very kind second she understands programming I didn't say she understands me she understands programming and this is one thing I learned very early on in my marriage there was one day she came to me and she said I want to go out do you want to come with me I said oh I'll be with you in about five minutes and she picked up the key and left and she came back after a little bit and I told her I'm really sorry but I told you I'll be with you in five minutes I'm a little surprised you didn't care and you just left and she came to me and said sweetie don't know this let me tell you this for once when you say five minutes it is seven hours in human time and that's what I understood that we have a different concept of time remember when the boss comes to you and says how's it going it's almost done we said this all the time isn't it it's almost done so we have a very different perception of what time really is I was here in Francois and and the usergroup folks invited me and I had a wonderful time speaking there and then they said hey here's a little church there's a staircase that takes you up there do you want to walk with me and I said sure but how many stairs are there oh only 90 and I started walking you know what it was not 90 and I almost after that I said how many more just a little bit almost there they said and as I kept walking I stopped and said damn you guys are programmers how did I come to accept your estimation and this is the problem right so I'm very careful these days but one thing I realized from being up you know in the family is this to me this is one of my aha moments is that happiness is when the world understand programmers when your family understands what you do and I tell my family always I'm sitting here in the kitchen I'm working you can talk all you want just don't expect me to answer back and and that's when you really know because all this is kind of getting deflected and your thought process is just churning through it and so I realized that mutability makes it really hard for us to reason and and we need today to really reduce mutability as much as we can because it becomes really expensive for us to maintain the code as well and that's the next thing I would really like to do is to reduce mutability in our code well moving a little forward lack of cohesion is another problem so complexity comes from lack of cohesion what is cohesion cohesion is where a piece of code is narrow focused and does one thing and one thing really well but a code is not cohesive that means it does several things and when a code there's several things it's really more complex for us to understand and that becomes really hard to maintain that's complexity that we have to deal with well when it comes to cohesion like things are together unlike things are apart and of course the code is focused reason for this of course is we are minimizing the frequency of change so as much as we can we should really try to make the code more cohesive I was in Canada for speaking at a conference entirely run by students who go to universities over there and they had a competition that year where they wanted the students to build different things with Legos and and the options were almost endless and I began to realize how beautiful this is in terms of modularity and abstraction and what if the software we create can as well be so concise and cohesive how what difference that would that make is something to really think about the next thing I want to talk about is what we deal with every single day excessive dependencies dependencies dragged us in and beat us down now dependency is unfortunately a necessary evil we cannot develop without dependencies but we cannot also develop with dependencies I was talking to a Dean recently they were working on a JavaScript project and I asked the developer how many libraries do you use on this application you know I was working with them I know what the application is doing and in my mind I have an estimation of what they should do and without missing missing a beat he said oh we use a hundred fifty libraries in this I said 150 he opens a dependency file and shows it to me sure enough more than one hundred fifty and let me look at him and said why are you using that many he kind of shrugged and said because we found him and this is so true isn't it and and because we found them sure I guess it's being honest and this happens a lot I've been programming for about 35 years when I started my programming career I left through their days called DLL hell and this is one of the things I tell my children I tell that tell them the stories of my life how terrible it was to work back then and and you would have to deal with the DLL hell and then the middle of the day somebody would say your core doesn't work well that was very helpful thank you now what doesn't work it doesn't work well that's even better and then you go to them and say are you using the right Dearlove yes where's your DLL here and then after 30 minutes of wasted effort you find out they change the path on their machine and that's using the wrong version of the DLL well then I was so happy to move away from the DLL hell then I started programming in Java the good news in Java was we dealt with jar hell then came along c-sharp I worked with assembly hell but those things got really better as time went on so I've had a one hell of a programming career from DLL hell to jar hell to assembly hell but now npm hell well this is the story I'll share with you I was on the flight the other day and the captain came on the on the speaker and said folks we reached the recruiting altitude of 36,000 feet everything is going smooth I'm turning off the seatbelt I shouldn't have done this but I opened my laptop and all I did was npm install turbulence right away I'm like sorry I didn't mean to do this they never let me on the Wi-Fi anymore and the point really is this becomes horrible because it'll go with vengeance and download the entire internet for you I was mentioning this the other day and a developer said oh I don't have this problem I said really how come how do you manage your dependencies and he said I use maven how do I tell him this you don't use maven may even use this you and the point really is that we get sucked into these dependencies and that becomes a dependency hell that we have to live with and this becomes horrible and this is one of the things we fight every day constantly at work he's dealing with all these dependencies now I'm not going to tell you that the past was better than the current or the future but I've just shared with you the past that I lived through years ago or even decades ago I would say and and that is dependencies usually are really hard to maintain and we struggle with it and dependence has become obsolete over time and they become incompatible but back in time when I wanted to use a library or a framework what would I do I would have to go get them well one of the things to consider in this case is when it comes to these libraries and framework you need to be very careful how you bring them in and and related to that I'll talk about one other thing that causes these problems like the developer who said we found them well this leads to what is called technology infatuation this is a problem that plagues us a lot you hear about a certain technology and the next thing you know you're like oh my gosh I could use it and this is a really big problem because we don't know enough about it but we feel strong that we have to have that on our project and that becomes really a problem for us to deal with over time so the question to really ask is is this the right technology for our project and a word I would want every one of us to really think about is this word called a reversibility so what is reversibility the reversibility is our ability to back out of design decisions so you make a decision today and in three months you realize you know what that's really not a good idea and you want to back out of the design decision you just made and if you want a back o to the design decision how easy or how hard is it to really make the decision and if something is easy to reverse I'm all care to make the decision a little lightly if something is really hard to reverse I want to spend more time to make that decision as well from that point of view I will think a little bit about library versus framework well there are libraries we use and there are frameworks that we use what is the fundamental difference between these two when it comes to reversibility and the cost of using them and in terms of complexity so I would say in the in the case of framework versus library the use of a library is easier to reverse than the use of a framework if you're using the library you can more easily back out of it and say I don't want to use this library anymore and you can change to a different library there's a cause but it's not an overwhelming cost when it comes to a framework the framework calls into your code and in turn your code calls into a framework so what is the way to really think about it well here's a silly analogy I would use to compare libraries and framework I'll say using a library is like dating using a framework is like a marriage you don't want to be too easily getting committed to it well the other day a developer came to me and said but but but the framework I'm using I didn't choose it the Enterprise Architect beside it for my project that's like arranged marriage the point really is you still get dragged into it and you want to really ask the question is this the right choice and one requires a greater commitment than the other so we have to be very careful making the decision as well so in general I am a little bit more averse to frame work I would spend more time thinking about it and say is this really what I should be committed to because it's a long haul it's not something you can change so easily in personal experience the projects I have the ones I fear the most are the ones where I'm using frameworks more than the ones where I'm using libraries because that becomes really hard to change in the long run so from that point of view when I was young when I wanted to use a library this was time before the way we know Internet as it is today I know this was really hard for a lot of people to even think about it especially my children they look at me and say what do you mean there was no internet when you were a kid that's always hard to explain to them in fact the other day somebody was asking you know hey why don't you google for it well there was no Google for us to use back then so in other words if I wanted to really buy it use a product I would read about it come to know about it somewhere and then you place an order for the product and then you wait like it's eternity and one day honestly there will be a tape drive that arrives at your desk when there is a tape drive a tape arrives at your desk and what is the tape drive between you and the product you want to use you don't use that so lightly and I spent my youth installing products using tape drives but today you're out at a lunch at lunch and somebody mentions a particular package or a library by the time you come back to your office one of your colleagues is already downloaded it and it's already in production on your product and you're like wow how did you get to use that so quickly and you're so proud about and this is dangerous presidents we are setting and now everybody wants to develop microservices I'm like is this your project that's really requiring it everybody wants to use other libraries everybody wants to use something and the question is is this the right solution for us the other day I was talking to a developer and he said our project is gonna use it I said how do you know you need to use this and then he looked at me and said do me a favor please don't come talk to my boss I said why is because we convinced him we cannot use develop the product without this particular framework there's a name for it it's called RDD which stands for resume driven development and that's what we do quite often is so we can put it on the resuming this is scary stuff because when I look at resumes I often look at this and say what is not here because the list is so overwhelming they got every single library and product listed and and it directly goes to trashcan because what does that say when somebody has used 300 different libraries that tells me they're gonna come and slow my project down as well I don't want that kind of commitment on my project I want to deliver application pragmatically using what is necessary not just to say here are things so how do we really balance this out well sure if I want to really learn about something and I do I want to learn about it on a site project I'll spend my time developing that on a site project and once they learned enough about it I can then decide if that really makes sense or not and if it does only then would I bring it to my project not otherwise and there are so many things we should be really playing with but not putting into production and there should be a forced select few things that actually go into production as a dependency in our application but I'll share with you a really sad story related to this and I call this the tale of infatuation and and this is really hard for me because this is something you don't want to see happen to to things that are around you you want things around you to succeed so this was back in time as years ago and the economy was not really that good in the US but I was really happy I was working on a project as a consultant and we wanted to really hire more people so we were interviewing people to hire on the project and as we did we came across one developer who was just absolutely phenomenal and my jaws were on the floor looking at this person I'm like wow this guy's so knowledgeable I really want to work with him I was very selfish I said I want to work with him on and learn everything from him and so the the person in charge of the project the VP of the company came to me and said hey what do you think of this guy I said don't waste your time please go hire him he's the best person we have come across he's amazing his knowledge is wonderful I really want to work with him go hire him he said yeah I understand that what do you think of the framework he has been developing I said that's the best framework I've ever seen look at all the beautiful features it contains it's just amazing but of course it's not surprising the guy is really that good no surprise he developed this framework I'm all for you know what he has done but anyway don't waste your time go hire him yeah yeah yeah but what do you think of us using the framework I said oh well we would step back for a minute we got a product to release in six months you don't need any frameworks right now so can we keep our focus please on the project we are developing hire the guy let him come on board we'll work with them let's develop this let's deploy this six months later you do whatever you want to do he said no no no but but did you notice venket that this product had a distributed persistent data store and I said you never used those words in one sentence ever before why are you saying this he said do you did you see how it has persistent messaging with fault-tolerance built into it I said look you never used those words also in a sentence before you are infatuated please don't do this hire him put him to work on the project here's what I'll tell you you've finished this and release in six months you use any framework for the next version I will be with you let's do this well two weeks went by I remember it very clearly Wednesday afternoon I'm sitting and coding away on in my on my desk and he literally comes over sits on my desk I can I'd remember that like it's yesterday sits on the desk and says hey Lincoln I got a good news for you and I was sitting here looked at him and said what's what's up oh we have hired him and I just leaned back and said I can't tell you how happy I am this is the best news ever in my opinion we're gonna have fun this is gonna be great and the guy is amazing and like I said I really want to work with him cause I want to learn everything from him this is gonna be really fun so thanks for hiring him and then he said oh as part of the hiring contract we have agreed to use this framework on the product I said tell me you are just joking with me tell me you're saying this to make me angry he said no this is the truth we have agreed to use his framework I said well then I just resigned I just quit he said oh I know this is coming I already told my boss the minute I tell you you're gonna quit I said at least unpredictable I'm so happy for that well I closed my laptop I plugged my stuff out and I said if you have any other project please call me I'll come and work with you but this one it's doomed but you don't need me for this you should run with it how long you are gonna take to finish this oh six months will be still done with it I said awesome good luck with you and I this is one of the very few days I actually went home early I ran home my wife is like hey you're early I said yes I quit and she said you quit it's like yeah I quit well in that case I got some dishes for you to do that's the way she is and it was wonderful - doing dishes the rest of the day and the rest of the weekend another two weeks when I had no job and then of course eventually something kind of you know caught my eyes I moved on and I'm not kidding with you eighteen months later eighteen months later my phone rings hello I said hey Rankin hey how's it going go ahead and say it I'm sorry what do you mean go ahead and say it no I didn't say it I told you so I said I know what happened well things have not gone well we are behind schedule by at least a year we're still trying to finish this and after a little bit of talk I said so why are you calling me well I was wondering if you want to come and help us do turn this project around I said there's a reason I quit and why would I want to come today I I quit when the ship was on the shore now you're sunk in why would I want to jump in and rescue you now so it's never mind sorry not interested six more months later this is two years after I quit the phone rings hey what can I do for you I know I know you don't want to work on the project I'm not asking you to come and work with this all I want you to do is come in and look at the product write an export report on it and you're done you get paid for a day that's what I can do that so you guys are done absolutely I'm like this is awesome and I'll come and write a report for you no problems - one day kick I'll be happy and I showed up I'm not kidding with you 8:00 a.m. and it takes me into this room closes the door and there's a computer with a few instructions next to it and I open up the application and I enter a little information there click on a search button and I'm waiting and I'm waiting and 25 seconds later something popped up on the window and I did that again and then I measured the time and said it's taking 25 seconds is that uh-huh I said the speed of the computer oh don't even talk about it this one is many times fast than what we can get in the field okay database don't even talk about it this one is local it won't be local in production I'm trying different things and then I started profiling the application of not kidding when I profiled it that one click one click resulted in a creation of 1864 objects I said did you notice that one clay created 1864 objects he said its object oriented and I'm like okay and then he looked at on instead I got a question for you where's the team and he kind of swallowed a little deeply and he said so what do you think of the program I said no no I'm asking the question worse the team and then he says they fired the entire team well after two years deliver something that doesn't even work and they fired the entire team and then I asked him what did you ask me to come here for he said fix it I said I'm sorry I'm a mere mortal I'm not a magician I want to go home I'm not kidding with you I was there till 10:30 in the night he wouldn't let me go he wanted to know what options are there to fix it and when I left I said today I'm gonna leave and I will never send you invoice for today because I like doing surgery this is post mortem I'm not there for that and I walked away never to ever go back and and I went with a broken heart because these are good people they were struggling to develop a product to survive in the business and they want to develop an application technical people should have the wisdom to tell them don't do this and if we technical people don't have that wisdom the business tends zero chance and we always like to complain about management but a lot of times we have to stand up for responsibility and say we want to tell about this application for you and we can't go this way because it's not going to happen and that was a very really sad experience for me because here's product we were working really hard and it completely was dead because they were infatuated so we want to really avoid the kind of infatuation so don't build what you can download sure but don't download what you do not need so I'm very reluctant when it comes to using libraries I'll say that very honestly people will ask me why don't you want to use this it's there and a lot of this ask the question there's the cost of not using it but there's a bigger cost of using it and I want to know what's the cost of using it as much as the cost of not using it because if I end up using the wrong stuff it cost me more in the future it is so much easier to put that in it is so darn difficult to take it out so I want to be very reluctant for a good reason because I care about technology but I care about economics of Technology even more and the more I you know mature in my life I more think about what's the cost of doing it more than how cool it is to do this the next thing I want to talk about here is about a few accidental complexities one of them is low level of concurrency we see this a lot we look at a problem we say hey we can solve this front of the pool of threads now we have pool of problems to deal with let's look at an example of this just to get a feel for it so here's a little piece of code if you will and and in this piece of code all I have here is a little nice little beautiful variable called done which is boolean and notice it's a static variable right there and I set it to false in the main I create a new thread and I say that the thread is running then I have a loop where I say not done i increment the count right here and then I look through and whenever the done is true I will get out of the loop and I'll print exiting thread then after 2 second delay I print out setting done in main and I said - done - true so logically if you eyeball the code you're saying the got a loop for two seconds at the end of two seconds is gonna print setting done to true and then they're done is set to true it should bail out of this loop and she'd say exists eggs exiting thread well when I go back and run this code this time though notice that unfortunately in this case it is going to I need to save the code so in this case when I run this code notice it is going to say in main running two seconds later it said setting down to true but we're gonna wait for a very long time isn't it how long for eternity so why doesn't this go really work in fact this is a beautiful code that you can torture people with because you can go back to this code and ask the question if I say thread dot sleep zero can you even believe it at sleep of zero you're not even sleeping but you just dropped that one-liner right in here that's all you're doing asleep zero you go back and run exactly the same code in this particular case what is it going to do at this point okay it wants to deal with the exception let's go ahead and deal with that real quick so we'll go ahead and say in this case our try and then we'll put a little catch right here and of course in this case exception and we'll just put a little sleep zero that's all we are providing run the code this time and the very exact code now is terminating now how do you explain this to a developer why putting a sleep zero works whereas when you don't put asleep it doesn't really terminate well this is the beauty of the concurrency library provided in Java um remember watching Harry Potter what was so wonderful about Harry Potter what is wonderful of Harry Potter was there were seven horror crooks you don't know where walled more put it and you and Harry had to PI find it what's even worse is Harry says I don't know what we are looking for but look for it that's what made you sit in the edge and watch this movie can you imagine Harry Potter if they started the movie saying here's a list of documented horror crooks and here's exactly where they are placed you know whether you would have left the movie early right what fun is that well Java has seven horror crooks these horror crooks cross the memory barrier one of them is the sleep when you put the sleep the code behaves very different it doesn't matter whether you really sleep or not because the sleep makes the memory pick cross the memory barrier similarly when you do synchronize you cross a memory barrier twice when use a volatile variable but you read it or write it across the memory barrier and when you call a method with a join when you call a method with the wait and so these are built into the language but we didn't really say hey these are the methods explicitly to cross the memory barrier you just have to know it you have to play that Harry Potter and that's what makes it really hard for us because we have to deal with that kind of complexity in in programming and that makes life really hard so a key design skill we need to develop is the ability to discern accidental complexity from inherent complexity so if as programmers we can sit down and say that is an inherent complexity that one is accidental don't do it I think we can come out winning really by doing that well the next thing I'll talk about here is another complexity accidental complexity which is imperative style of programming most of us have done infinity style of programming I spent my youth programming an imperative style that's the one I'm most familiar with but I realized over time that it is also more complex so I'm gonna say in here imperative code is packed with accidental complexity on the other hand declarative code has less complexity but let's look at an example of the code just to see how that is true so here's a little code I have and in this code notice is trying tells us whether a number is prime or not the main simply has a random value N and K but I want to call the compute method what is a compute method do the compute method says return the value for N and K well where what is the value I'm going to return find the total of the square root of the first K prime numbers that start with n let's write this code in imperative style just to get a feel for it so what would you do double result is equal to 0 that's a very first step return the result this is called hope isn't it then what do I do int index is equal to n and then you say in count is equal to 0 so what did we don't do so far nothing useful right all we did was we create a bunch of garbage variables then you say wild count is less than K and at this point you stop everything and ask the question is it less than or less than or equal to do you ever ask that question every single time when new programmers come around and ask me what is the symbol I tell them that's the International symbol for confusion because everybody stops and sees her and we don't know and then what do you do then you say in this case you say if the index well in this case if is prime of index if it is really a prime number what am I gonna do the result plus equal to math dot and I'm gonna say in this case square root of the value index well okay is that done really you say where it no no no you need to increment the count so increment the count or is it done well you need to increment the index ok increment of the index is this correct no what's wrong you say it should be outside so I'll move it outside is it correct now you said no no this must be inside how do you feel about that right one should be inside one should be outside and then when you run the code it produces some result but look at the complexity of this code well okay this is a code you can write but it is a lot more effort to read it because you gotta read through and hold all that state change in your mind and that is a lot of effort if you're dealing with the code like this what's gonna happen when you're staring at this code without going to lunch your business analyst comes to you and says hey how's it going your response normally is that's what I'm trying to figure out you shut up and that's how we feel about code like this right it makes us very irritable well thankfully we don't have to do it this way what if we can say return stream dot iterate and I'm gonna say starting with the value n let's create an infinite stream on this and we'll do a filter and in this case we will simply call the sample this cutter current class and we'll call the method called East prime but we will perform a map to double on this and we will ask you to find the math square root of those values but then we say dart limit and we simply ask it to limit it to K values and then you perform the sum operation in the end and then you return the result of that particular call and if you notice over here this code begins to read like the problem statement you didn't have to write that much code and you can logically read the code from top to bottom one single pass and say given all the numbers that start from n give me all the prime numbers but give me the square root of them and limit them and return the sum that becomes a lot easier for us to work with so decorative style really helps us to focus on the problem instead of losing track of what we're doing by focusing on the how the last thing I want to talk about here is called employment employment everybody falls into this trap even the people who wrote the JDK did so what is entwine Minh employment is where you take two concepts that are orthogonal and not related to each other and you put them together a really good example of this is in how we dealt with garbage collection well in the case of Java garbage collection is automatic but we know that garbage collection is not instantaneous when God job introduced garbage collection as being automatic we were like yeah this is great but how do we deal with external resources what unfortunately though external resources need to be cleaned up properly and promptly well their initial sedition was hey why don't we provide you a finalized method that was a bad idea why because garbage collection is automatic built with main memory it is something that will happen on its own pace and when the garbage collection happens finalize is called when the finalize is called we will clean up external resources wait external resource cleanup is completely independent of memory cleanup you can clean up the memory anytime you want to clean up but I want to manage the external resources more deterministically so this is a really good example of employment in fact this hurts us a lot you know we're own applications we do this quite often and get into trouble with this in fact it's so prevalent rich Hickey kinda turned for it he causes complected he says completing things is the source of complexity he says and we do this a lot of this in our own code we take two different things and put them together and we suffer the consequence of the employment and we should really have the wisdom to say these are independent and keep them away from each other so we talked about these nine different things that caused complexity into our applications too many moving parts invisible changes uncontrolled mutability lack of cohesion excessive dependencies technology infatuation low level of concurrency and of course a concurrency low-level use of concurrency affair should say an imperative style of programming and employment and and we fall trapped to any of these are naturally unfortunately more of these and we deal with them I would say one of the things we should really do as we mature as developers is we should learn to deal with complexity and have the wisdom to minimize it and this is what I do when I go to projects I try to identify complexity and say can we get rid of these because these are hurting you on your project these are not things you need to be carrying with you so I'm gonna finally say the maintainable code is a gift we can give ourselves for the future and and when we create a Maitre bill code our future will come and thank us and so we should really strive really hard to reduce the complexity so we can make the code more maintainable in the future hope that was useful thank you you

Channel: Devoxx

Views: 27,642

Rating: 4.9444447 out of 5

Keywords: DVXPL18

Id: nZcLHkORdHE

Length: 57min 3sec (3423 seconds)

Published: Sun Jul 15 2018