I Bet You’re Overengineering Your Software

I am to recommend and approach the software development that is very highly quality focused I think that this quality mindset infuses nearly all of my work and thinking on the topic but qualities sometimes a tricky idea to get our heads around one person's quality work is another person's over engineering so where do we draw the line what is good quality work and where does that cross the line into over engineering oh and and what should we do to avoid the over engineering trap [Music] hi I'm Dave Farley of continuous delivery welcome to my channel if you haven't been here before please do hit subscribe and if you enjoy the content today hit like as well I have a very specific take on what quality means in the context of software in which is different to what you'll usually read if you search for definitions of software quality I think Wikipedia says that there are two aspects to software quality functional fit and non-functional fit to requirements I really dislike this description mainly because this is looking at quality in a kind of rear view mirror the reality of software development is that we almost never know the functional nor the non-functional requirements of a system before we build it but even if our initial guesses are good ones our software inhabits the real world and the world changes so in the unlikely events that we did start off with perfect guesses how long before the world has moved on and invalidated them let me take a moment now to thank our sponsors we are fortunate to be sponsored by equal experts trisentis transfig sloof and Ice panel all of these companies offer products and services that are well aligned with the topics that we discuss on this channel every week so if you're looking for excellence in continuous delivery and software engineering then click on the links in the description below to show your support and to check them out thank you even if our guesses for non-functional requirements like security scalability resilience and so on are good ones you'll almost certainly need to over engineer to some degree at the start because the needs of a new system with few users are very different to the needs of a popular system with Millions so if you start out by building for Millions you'll waste an awful lot of time and money if those Millions never show up and now maybe the reasons that they didn't show up was because you spent too much time making the system robust well someone else beat you to something less fantastic but good enough none of this is simple so looking back rather smugly as the Wikipedia description seems to imply thinking yes I built a quality system because I was very smart and guess right about all of those functional and non-functional things seems like at best highly unlikely to happen and at worst no help at all in making it happen I don't think that engineering let alone software engineering is meant to work like that engineering is the stuff that works the things that help us to do a better job so if our goal is to build high quality systems then shouldn't we adopt a definition of quality that helps us to achieve that rather than only look looking back and evaluating it with the benefit of hindsight I guess that the next question then is why should we care about building high quality systems at all it may or may not seem obvious to you that high quality is better than low sure we want to take pride in our work but the usual assumption is that quality comes at a cost so we're always playing some kind of trade-off or optimization game there's a very old idea called The Iron Triangle in project management circles the triangle is fast good and cheap but you can only pick two this is pretty widely and strongly held View the Dora metrics measure speed and quality in software projects specifically in terms of stability and throughput and their finding has profound implications in my view because it says the iron triangle is rubbish there is no trade-off if you want quality you must move quickly in this context quickly means in many small steps so that you can observe the impact of each small step if you want speed you must build with quality because otherwise you'll go slower because of all the time you spend fixing the things that you broke or dealing with the mess that you created if you compromise either of these to reduce costs or increase one at the expense of the other you end up building worse software more slowly and spend more money doing so actually the research doesn't say that last one explicitly but only implicitly the Dora research says that companies with good scores instability and throughput are better able to achieve commercial goals that is make more money and that teams with great scores spend 44 percent more of their time on new features so saving money by producing things more efficiently so there is no trade-off here this means that if you're not working to create software of high quality you're wasting time effort money and you're lowering motivation because people like to take pride in their work and work more effectively when they do I think that this Assumption of the iron triangle is one place where the often very strong reaction against some forms of engineering comes from because nearly everyone assumes that good work is too expensive and so assumes that poor quality is a valid trade-off and maybe good enough I don't believe that this is ever true that isn't all that there is to the idea of over engineering though software development teams and software developers are always at risk of falling into this trap it's very easily done I think that this kind of gets us really to the heart of design and what design is really for and to the heart of what quality in software really means too I think that technically quality and software is all about our ability to change it I know that this may sound weird surely our software needs to deliver services to its users and be scalable and resilient and secure yes but we can only get those things if our software is easy to change this definition works as a tool that we can use and gives us a subjective but concrete definition that we can use to guide our decision making on the route to better outcomes for every other aspect of our software that matters to us of course we want more than only this but without the ability to change our systems we can't achieve these other things we certainly can't maintain them as the context in which our software operates changes so if our goal is to maintain our ability to change our software easily and with confidence then certainly some level of engineering matters so what counts as too much when do we cross the line from good design to over engineering there are two common causes for doing too much and there are some ways to defend against both the first is kind of technical masturbation where we have a Grand Vision a dream that we'd like to fulfill and we take that too far we lose ourselves in the technicalities and enjoy ourselves so much that we never end up with anything sensibly usable does your system have features that no one ever uses for example this is really just playing though it may not feel like it at the time actually there's nothing wrong with playing in the right context so this may be fine but it does depend on what and maybe who you are doing this for ultimately who's paying you for your playing I suppose playing with ideas and Tech is a great way to learn but it's often not well directed in the early pass of my career as a developer I got really interested in computer graphics and I spent a lot of my time playing with them I built some rather cool things full-blown Ray tracing modeling refraction in transparencies in 3D scenes obsessed me for a little while but I was always focused on the stuff that interested me most not on producing anything other than core graphics and the code to make them the UI on my tool sucked because at the time it was the code and the pictures that were my primary focus this was fine because I did it in my spare time for me so it wasn't my it was only my time that I was spending and that was my choice later my employer at the time wanted something to help them to sell their computers at a trade show graphics on PCS then were pretty new and pretty basic but my stuff showed them up quite nicely so when they asked my team if we had any ideas to show off our computers I volunteered some of my code and experience with the graphics now we had a specific Target a real use case in mind so I spent some time writing some simple presentation Graphics tools for my team and between us we created some rather impressive for the time graphics for the show this stuff wasn't a sophisticated graphically as my home experiments but the tools were now usable enough for other people to use and build things with this is the primary defense against the first form of over engineering have real a real use case a real Target iterate to get that to that build what people want not just your dream the best way to do this is to build the system incrementally and to validate your guesses and charges after each small step working in small steps like this is essential because you'll make mistakes and getting your code out and into in use by other people is the fastest way for you to find those mistakes and to learn what to do better and what people want of your system having users in real use cases will focus your work and structure your playing with the technology for fun in the direction of more productive outcomes the problem here is that the playing often doesn't feel like playing the Communist version of over engineering of this form that I encounter is people building platforms or Frameworks in the abstract without any clear use case or at least without any real world use case they may have imaginary clearing clear use cases but these aren't good enough I've reached our experiments like these rarely end very well in my experience the best software comes from Team solving real problems rather than addressing imaginary needs have a real user find out what they really need and give them that solve the problem in front of you rather than the problem that you imagine that you may have one day that gets us to the second form of over engineering future proofing this happens when devs or orgs don't have a very clear focus on the problem and instead worry too much about all of the things that may happen in future and so then go on to build software to cope with all of those hypothetical things instead of focusing on what's really in front of them tiny startup with five users building their system with kubernetes or the team spending more time making their system cloud provider neutral than they do building features that help their users do something that matters to them this is in many ways the more Insidious lure for the technologist I think fundamentally I think that this stems from a lack of confidence a lack of confidence in their design and in their own ability to modify it when they learn something new this is one reason why I value the maintenance of our ability to change our code quite so highly more highly than anything else in fact at least in terms of guiding principles if we have a system that's easy to change then when we find that our system needs to scale we can change it to be more scalable if we have a confidence to change our system then when we discover that we chose the wrong cloud provider we can change that too sure it will be work but it will be possible because we've designed the system to be changeable the confidence to change our system allows us to the freedom to make mistakes to correct them and to adapt to new things as they crop up and so the freedom to steer our software in New Directions that we haven't thought of yet the point of software is that it's soft we should be able to change it and ideally change it fairly easily this takes a lot of things in my book I talk about the techniques of managing complexity and I think that this can be one area that may lead to confusion between good designs that manage complexity and poor designs that over engineer I think that these two things are actually quite distinct but the second is common enough and the first rare enough that people sometimes even experts confuse the two in my recent conversation with Kent Beck you can see the point at which he becomes nervous that I'm about to advise people to over engineer I think a good design is expressing my best understanding of the problem that I'm solving right now and avoiding designing for future features that I don't yet clearly understand or don't need right now but my design choices now will always be to the best of my ability aiming for my code to be modular cohesive and having a good separation of concerns particularly in trying to keep the essential complexity of my system somewhat abstracted from The Accidental and ideally Loosely coupled from it too here's a slightly more concrete example when we started work on building one of the world's highest performance Financial exchange into lmax there was an enormous amount of stuff that we didn't know we knew it needed to be extremely fast but at that stage we were only guessing quite about how fast we thought that it would need to be scalable but we didn't know how scalable really we assumed that one day we need to support markets for trading in a wide variety of different kinds of assets and we assumed that we'd need to be able to cope with tens of thousands possibly millions of online users one day and deliver them a great fast accurate Market data and a great trading experience to be honest this is the kind of stuff that you talk about in the pub or over dinner when planning to start work on a startup Imagining the system that you might be building nearly all but not quite all of this turned out to be wrong though fortunately we didn't start by assuming that it was all right in the first place but it was more the context that informed our mental model and then we updated so instead of building the system to do all that we imagined we started with what we were sure of and picked the simplest case that we could think of and we decided to build this as a collection of services each clearly focused on one part of the problem those weren't micro Services they were a bit beefier than that but one represented the matching of Trades another the positions held by Traders another their registrations and the details of their accounts and so on each service did one job or one part of the job and was only allowed to communicate via messages with other services I and others on the team had some experience of designs a bit like this so it was a reasonable first guess at an architecture that might work for us we knew that at some point we need to optimize all of this but at the start our main focus was on relatively simple relatively low volume trading that we could use to try out our thinking and explore the problem in a bit more detail I had some code at the time that I'd intended to release as an open source project that allowed Java Based Services to talk to one another asynchronously and hid nearly all of the detail of how that happened the code that I had was built on top of XML over http now we knew at the outset that this was never going to be anywhere close to fast enough for what we needed for our Exchange but it worked and importantly it hid The Accidental complexity of the technicalities of the communications from the essential complexity of the services so we could write as services and test out the business level conversations between them without worrying too much about the the technology right now for the communications we'd worry about making it fast later hopefully without needing to discard too much of the code in the services this worked out extremely well eventually after lots of experimentation we replaced the XML over hc2 with a blisteringly fast proprietary binary protocol and the fastest messaging system on the planet at the time and kept the bulk of our system in place but the key here is that I don't think that we ever over and engineered at any point at every stage in the development the software was an accurate representation of only the problem that it was focused on solving up to that point but it was designed with great modularity cohesion and separation of concerns with good lines of abstraction between components and services and the infrastructure and managed appropriate levels of coupling at every point in the design this meant that the complexity in one place was always hidden from other parts of the code when we did learn more and needed to enhance some part of the system any part of the system really we could do so and keep nearly everything else working and unchanged our system ended up doing things in ways that we had never foreseen and being used in ways that we hadn't designed it for at the outset but at each stage in its Evolution it did that in a way that kept the doors open for it to change without us knowing exactly what kinds of change we might be expecting to cope with in the future I think there's a big difference between high quality design that keeps the doors open to change and over engineering that attempts to predict exactly what changes may need to happen in future so apply the techniques to manage complexity if you come to a decision in your design always prefer the choice that maximizes these things and when you find a mistake or a new feature that you haven't thought of even a cross-cutting non-functional concern then while it may still be work to change the code it won't be as much work as if you didn't do this or if you've got out your over engineering crystal ball and guest rung thank you very much for watching and if you enjoy our stuff on on the continuous delivery Channel please consider supporting our work by joining our patreon community you can join for as little as two pounds thank you [Music]

Channel: Continuous Delivery

Views: 23,722

Keywords: overengineering, overengineered software, stop overengineering, overengineered, when to stop programming, programming, computer science, software engineering, software development, Dave Farley, continuous delivery, software overengineering

Id: FLe5dvqV6xs

Length: 19min 58sec (1198 seconds)

Published: Wed May 24 2023