5 Steps To Fast & Reliable Software Development

what does it take to create and deliver great software efficiently into the hands of our users with a minimum of fuss and effort I think there are five things and those five things also happen to be the defining characteristics of a deployment Pipeline and that's not really a coincidence so that's our topic for [Music] today hi I'm Dave Farley of continuous delivery welcome to my Channel if you haven't been here before please do hit subscribe and if you enjoy the content here today please do hit like as well it is common for ideas that become popular to become devalued misinterpreted through repetition over time some people call this effect semantic diffusion meaning that the meaning of the idea becomes more diffuse as it spreads one of my ideas that became very popular indeed was the idea of a deployment pipeline it has certainly suffered from semantic diffusion over the years but this idea remains at the heart of an effective approach to software development in my mind I think that a deployment pipeline is much more specific though than lots of people imagine the commonest semantic diffusion is that a deployment pipeline is just the workflow of steps that you go through to build your software this is a very common misconception but it misses some so much about what a deployment pipeline really is and what it's for that it's close to useless in helping us to do a better job the sequence of steps that you go through to build your software is usually captured in a build script not a deployment pipeline a deployment pipeline is meant to define the releasability of your software if the pipeline says it's all good then we can release this change into production with no further thought and no more work so if you need to get a sign off from someone else once the deployment pipeline's finished or run some integration tests against another service or component of the system before it's ready to release then by definition you don't have a deployment pipeline perhaps the easiest way to visualize this is to look at the high level structure of a deployment Pipeline and this identifies the first three of my five things the pipeline can be thought of as being divided into three zones or phases Each of which is important to the process of developing and delivering software each is focused on a different part of the problem of software development though let me pause there for a moment and say thank you to our sponsors we're fortunate to be sponsored by Eco experts tricentis and transic all of these companies offer products and services that are very well aligned with the topics that we discuss here every week so if you're looking for excellence in continuous delivery and software engineering do click on the links in the description below and check them out phase one of the deployment pipeline is the commit phase this is focused on providing great feedback on our design choices to the development team does this code do what I think it does does this keep my code in a state that means that changing it later will be easy is my code nice to use and nice to understand and so on fundamentally the feedback that we're looking for at this point is does it work and is it good does it work means that have we solved the problem that we meant to and by is it good I mean does this code exhibit the properties of good code that make it easy to change have we successfully managed to manage the complexity of the problem in our solution the next phase in the pipeline is the acceptance phase does it work and is it good are very nice things to know and they're part of what we need to know to do a good good job but they don't tell us enough to know that these changes are ready to release yet that's the job of the acceptance phase to answer that question is it releasable the scope of the commit phase is pretty generic the best way to know if our code works and if it's good is through test room development so the vast majority of the testing in the commit phase is all about test room and development which helps us to design better systems and gives us great feed feedback if we break anything acceptance though can be a bit more varied but one part that doesn't really change is answering the question does it do what our users want it to do for this we need in general bdd style acceptance testing that verifies that the software is Deployable configured correctly and meets our users's needs I talk more about this in detail in this video then after we've got that assurance that this our software does what our users want it to do we want to evaluate it to see that it does everything else that determines the releasability for our software is it secure is it resilient scalable enough usable regulatory compliant and so on everything that you use to decide whether or not the software is ready to go into production is fair game for the pipeline and should be automated if a change has passed all of our testing in the acceptance phase and the commit phase what's left to do well release it into production so the final step of the pipeline is the production phase this may be as simple as deploying and getting up and running a Docker image or it may be sophisticated and complex using release strategies techniques like Canary releasing blue green deployment AB testing and monitoring and evaluating the success or otherwise of our changes through observability and other site site reliability engineering techniques these three things are essential to producing great software efficiently however you organize your work you can think of the commit and acceptance phases as rehearsing the use of the software and the production phase as putting the software on stage a deployment pipeline is simply the most efficient way to organize these pieces of work and rehearsals and to more clearly focus our minds on these vital aspects of software development if you'd like to learn more about deployment pipelines I currently have a special offer on my anatomy of a deployment pipeline training course you can get1 pounds off if you buy it by the end of January check the links in the description below this video to find out more the deployment pipeline gives us feedback on the design choices that we make does it work is it good easy to change feedback on the releasability is it ready to release with no more work repeatability if the pipeline says it's ready to go then I can release it easily and with confidence whenever I like essentially at the push of a button the next two important ideas are foundational to this way of working but are more about how to put these capabilities to work effectively the first of these is speed if we want feedback that helps us to design better quality code we need this feedback to be ready for us when we need it which really means all the time which is what the continuous bit of continuous delivery and continuous integration is really all about if we organize our work to proceed in many smaller simpler steps everything becomes easier so it's the job of the deployment pipeline to not get in the way and to continuously deliver this feedback to us fast people in continuous delivery circles talk about shifting left but I think there's a more direct way of describing our goal if there's something wrong I want to find out about it as soon as we can we want to fail fast I'd prefer to spot mistakes while I'm coding in fractions of a second not minutes hours or days later I can do this through design where I can use design choices to reject inappropriate uses of my code with types for example but we can also do this by using tools like integrated development environments that can give me instant feedback on on some kinds of mistakes that I might make or maybe AI assistants that give me this kind of feedback too literally as I'm typing but where I can't get this kind of instant feedback I still want to be as fast as possible so I want to be able to run tests on the code that I'm working on and get responses in a second or two no more once I commit I'm relying on continuous integration to show me that my changes work alongside everybody else's but again if I have to wait for too long for that feedback its value diminishes exponentially knowing that my change doesn't work alongside yours within a few minutes is a dramatically more powerful experience and dramatically more effective approach to development than only finding out a few days later because some QA team found the problem during manual testing so we need to optimize for fast feedback we can get fast clearer feedback when we make progress in smaller steps but also once again all of this pushes Us in the direction of test driven development test driven development promotes working in small attestable steps I generally recommend that we aim to get feedback from the commit phase of our pipeline remember that's running our test room development tests in under five minutes optimiz is like crazy to make this happen if you need to it's one of the quickest wins out there so it's worth investing time money and effort into achieving it speed of feedback matters a great deal I once worked for a company that built a complex trading systems part of their challenge was a very large C++ build which took 9 and 1 half hours to run including running all of the tests so they organize their work into a nightly build each night the build would kick off and they'd inspect the results in the morning then they'd release the modules that passed all of the tests and hold back those that failed this worked fine except when one some of the passing modules depending on some of the failing modules which did happen sometimes the problem with overnight builds like this is the feedback loop is too long if I commit a change that breaks something I don't get to find out until the following day if I know immediately what I did wrong and commit a fix that doesn't correct things until the day after that so the time to detect and fix even the simplest problem is now 2 days after the commitment caused it that's way too slow this means that my code is not releasable for all of that time teams with overnight builds tend to find it extremely difficult to achieve a green all passing build because there's always the random small fraction of the tests that are failing and awaiting a fix so the results aren't clear release or not we can't tell this team had been working for this way for 3 years and during that period they'd only had three days when all of the tests passed so they couldn't rely on their tests for determining releasability they had to do something else my team which was tasked with trying to address this problem did loads of work to optimize this process we rewrote the build bought new hardware optimized the running of the tests in a variety of different ways we ended up with a commit stage of 12 minutes still a bit too long but better than before and an acceptance stage of 40 minutes acceptable the first two we period following the roll out of this new faster build with no other changes to team technology or process or the problems that we were fixing we had two completely green builds and then for the rest of my time in that organization we had at least one green build every single day meaning that we could now change our release approach and simply pick the newest green build and release it every day into production feedback was improved defects in production went down developers got a clear a picture of where they were and releases took less work than before all by simply focusing only on improving the speed of feedback the last of my list of five things is controlling the variables I use this phrase quite a lot and I think it's applicable in lots of different places but fundamentally what I mean here is if my tests pass can I rely on them to be telling me the truth to get to the point where we can be confident in our tests we need to be able to trust them and we get to trust them when they're consistent if a test sometimes passes and sometimes fails when we run it we can't trust it and the result of that is that we often end up adding other things to our process to try and hedge our bets on what the test is really telling us we might add on extra sign offs from other people additional manual testing steps and so on I view all of these things as sticking plaster solutions they don't solve the real problem they at best just cover it up my if my test fails then passes when I run it again it's lying to me one way or the other if I decide the pass matters more than the failure I'm just guessing if I ask somebody else what they think to sign off on whether to release or not they're just guessing two manual confirmation of the result of that specific test may help in that single case but relying on manual testing in general provides slow poor quality feedback and unless it is specifically targeted on a specific failure which means even more buau accuracy and even going even slower and more delayed results it isn't a solution either because the manual testing might miss the problem that the intermittently see is highlighting the best way to solve this problem is to prevent the test from lying to us and we do that by exerting more control over the variables we need to stabilize the environment in which we test our code to make our test deterministic and repeatable if we test the same code with the same data every time we should get the same results we will get the same results so strengthening our control over the variables is how we fix this problem permanently and at source rather than attempting to patch It Up by guessing further Downstream I talk about the elimination of flaky tests using some of these techniques in this video so my five things are answer the question does it work and is it good constantly continuously using fast commit phase testing then answer the question is it releasable multiple times per day with fast acceptance phase testing make the release process itself reliable and repeatable through Automation and Version Control then optimize all of these things for Speed to improve the speed of feedback and increase the determinism of all of your tests and your code by controlling the variables with Version Control and automation thank you very much for watching thanks again and if you enjoy our stuff here on the continuous delivery Channel please do consider supporting our work by joining our patreon community there's links to that in the description below too thanks [Music] again

Channel: Continuous Delivery

Views: 15,925

Keywords: software release, software release management process, release management, software release management, software release process, software delivery, sdlc, software deployment, deployment pipeline, continuous deployment pipeline, continuous delivery, dave farley, Dependence On Manual Software Releases, improving software process, software engineering, software development

Id: 3Q19qgHgQJo

Length: 15min 49sec (949 seconds)

Published: Wed Jan 24 2024