Continuous Delivery Pipelines: How to Build Better Software Faster • Dave Farley • GOTO 2021

so [Applause] thank you i hope you've already had a good conference um i i'm probably best known my name names associated with continuous delivery and so i thought it'd be interesting to talk about that so i want to ask you a question what is it that you think of when you hear the phrase continuous delivery maybe being being egotistical for a minute maybe it's my book it might be maybe it's the state of devops report which analyzed the practices of continuous delivery and measured their impact on software development delivery i think the state of devops report is now up to something around 45 46 000 different surveys covering probably tens of thousands of different software software projects and we've got data like we've never really had before about that says things important about our industry and the science behind capturing that data and collecting it is uh captured in this book accelerate the science between lean and behind lean and devops uh principally written by nicole fusgren uh with jess humble and gene chemist gene kim as co-authors um one of the things that the state of devops stuff is built upon is is this model of measurement they measure the performance of software teams based on throughput and stability throughput is a measurement of efficiency we could say it's a measurement of speed it's based on the lead time how long it takes to go from commit to something releasable and the frequency with which you deploy changes into production it's a combination of those two measures and stability is a measure of quality and it measures the change failure rate how often do you introduce a defect into production or wherever it is that you're measuring it and how long does it take you to recover to that point in the process when you find a mistake so these are two measures that essentially move measure the efficiency with which we create software and the quality with which we we create it and what they found is kind of dramatic the difference between high performance and low low performers is really quite dramatic we're not talking about small percentages improvements here we're talking about sometimes orders of magnitude and better improvements in performance in these on these measures high performers um uh are are um [Music] 208 times um more frequent faster in their lead time they're 106 times faster deployment frequency 2 600 times fewer defects overall when you start measuring these things these are massive numbers and the the big idea that comes out of these is that these are measures of speed and quality and this is kind of counter-intuitive to some degree but there's no trade-off between speed and quality in fact kind of the reverse if you want to go fast you need high quality if you want to do high quality you need to go fast to get the feedback good enough and that's kind of a profound idea i think there's no trade-off that's that's a big deal because we've assumed for a long time that there was i want to explore that idea a little bit and i came across this this is from stephen bungay's book the uh um i forgot what it's called um i know i haven't got my glasses on stephen bunga is excellent the art of action that's the name of the book um thank you for whoever is about to shout it out but so this is this model goes from outcomes plans and actions we want we want to achieve some kind of outcome so we're going to form a plan to achieve it then we're going to carry out the actions based on that plan and hopefully that's going to give us an outcome the trouble is that we're faced with these gaps there's the knowledge gap the disdifference between what we know and what we need to know to come up with the perfect plan and then there's the alignment gap the the difference between what we say we're going to do and what we actually do and then there's the outcome gap the effects gap between what actually the outcome is and what we expected the outcome to be traditional organizations responds to these in ways that we're all familiar with so the knowledge gap what we like to know versus what we know typically a traditional organization will plan harder um they will analyze more um they will do more detailed requirements be more specific about what it is that you want to do to be able to fix this problem what this means is that you go slower and we've just said there's this link between speed and quality then there's the gap between plans and actions the alignment gap what we want people to do and what they actually do when we want them to do it and the the response to this in a typical organization is to micromanage people more closely to try and make sure they do the right things to implement stricter process control and to add bureaucracy to the process there's loads more rules to try and force you into doing the right things guess what makes you go slower and then there's the the effects gap this is the gap between the expected outcome when we anticipated when we made our plan and the one that we actually get the response to this typically we tend to do expectation management we tend to try and under promise and over deliver but we sometimes do things like watermelon status reporting if you haven't come across that idea before it's red on the inside and green on the outside it's very common in big organizations and then we increase the program program management or project management rigor to make sure that we're going to do to do the things we're going to get the outcomes that we want and again all of this makes us go slower the trouble is it's worse than that because as i said before this relationship between speed and quality it's not just that we can have our cake and eat it we can have both speed and quality but in order to get quality we need to move quickly and in small steps so that we can see the changes evaluate them understand them and and that's the way that we build high quality systems in order to get speed we need quality so that when we do make a change we're not going to go back chasing defects and trying to figure out why this thing breaks in production and it didn't break anywhere else those all come at a cost so if we want to get speed we need to do work with high quality if we want to get quality we have to move fast so if we slow ourselves down if we go slower what we get is we get worse software slower not better software because we're moving more slowly so how can we tackle this because these we can't close these gaps they don't go away they're always true there's always a gap there's a there's no perfection here there's no way of removing those gaps altogether so the best way to improve the impact or reduce the impact i should say of those gaps is to reduce them and we do that by going faster we speed up we make the gaps between our actions we work make work in smaller steps we observe an outcome we do a short a short time horizon plan we act on that plan and then we observe the results and then we go around that loop much more quickly that's how we that's how we really impact this so one take on my book is working smaller steps so that our software is always in a releasable state continuous delivery is widely misunderstood i think it's not about not necessarily about deployment automation not necessarily about test automation it's certainly not about deploying into production frequently necessarily now all of those things are good ideas but that's not what really continuous delivery is about what it's really about is working so our software working in small steps so our software is permanently always as close as we can get to always in a releasable state and that has a profound impact on the way in which we work and the speed with which we work and the quality which we're able to produce so what determines releasability in continuous delivery and at the core of the book that jez and i wrote together a few years ago is this this is the deployment pipeline this is a machine that is intended to be the vehicle for releasability it's meant to be the one the one channel to production this is the definitive statement on the releasability of our system in continuous delivery i think it's tended to get marginalized a little bit as an idea over the years as it's become popular often i talk to organizations say yes we're doing continuous delivery what they mean is somewhere somewhere or other there's a server running jenkins that's not the same thing working so your software is always in a releasable state takes a lot of things it changes the way in which we organize ourselves the way in which we architect the systems that we're working on the way in which we test them the way in which we deploy them all of these things are modified when we're trying to just speed up this cycle and make it more efficient in order to be able to figure out whether our software is releasable there's a bunch of questions that we probably want to answer we probably want to run some tests locally because we want to make sure that our software is always releasable so we don't want to waste time by just committing rubbish into our version control system and evaluating that so we would like some level of confidence that it's okay so we're probably going to check that things were working locally before we make a commit and then we'd like feedback on whether our code works with everybody else's and whether it does what it is that we expect it to do we want that feedback to be fast and efficient and effective to give us some direction in terms of the design and the quality of our choices at that point in the system then we'd like to know that the software does what our users want it to do is it functionally appropriate is it configured correctly can i deploy it is it is it configured correctly so that so the deployment mechanism works and then it behaves appropriately when i simulate realistic scenarios and then we'd like to know whether it's nice to use is it effective is it pleasant does it paint pictures in my head that allow me to extrapolate and understand where i'm going to go next is the system fast enough is the system scalable enough is it secure resilient compliant enough all of these things might impact on whether our software is releasable or not and what we'd like to do is that we'd like to get from our deployment pipeline a definitive statement on the releasability of our changes if all of these things are checked and approved then there's no more work to do at the point at which we've transited the deployment pipeline if the pipeline says this is good it's releasable by definition pretty much so a deployment pipeline goes from commit generates a release candidate that we can evaluate and describes whether the the outcome is releasable or not if one test fails anywhere in the deployment pipeline probably means our software is not ready for release this allows us to be confident in the changes that we've made by testing our system thoroughly from a different a range of different dimensions it allows using automated testing it allows us to really have confidence to release our changes because we've we're as we checked it as thoroughly as we as we've been able to think of up to that point i think of this as a genuine engineering discipline for software if i can work this way if i can create automated tests on the whole there is a place for manual testing but if mostly for regression testing wholly for regression testing i'm going to automate all of those if all of those tests pass i can i'm happy to release the change into production the downside of this is that it comes with some consequences this is a this is a kind of interesting idea when i stand up here and i explain it to you it sounds fairly straightforward but there are some consequences to this if we want to make changes in small steps and every step is releasable and we want feedback on those frequently and i'm going to show you some times shortly about how frequently i think we ought to get those changes then the trouble is now we've got to start organizing our work to be able to achieve that and that starts leading us in the direction of challenging ideas like trunk based development and eliminating feature branching and pull requests from our development process to a large degree if we genuinely want to embrace this excuse me we want to try and i if anybody was in my talk earlier earlier in the conference i talked about engineering and i've just said that i think this is an engineering discipline i'm just going to quickly whiz past the ideas that i talked about in my earlier presentation if we want to be engineers then we need to optimize for learning the deployment pipeline is evidently an iterative machine we're going to put a change through we're going to flow through the pipeline that's an iteration of the evaluation of our system we're going to gather feedback in the form of tests test results from from executing in our pipeline and we're going to learn from that we're going to make changes incrementally growing the features of our system step by step along the way and we're going to it works as a fantastic experimental platform if we want to know with our systems fast enough we can put some performance tests in there run those in simulation and evaluate the performance of our system if you want to know whether it's resilient we can kind of start breaking pieces of the system and figure out whether uh whether our system's as resilient as we need it to be and so on it gives us this fantastic resilient platform in terms of the empiricism of our system we can by automating all of these things by the time we get out into production we can gather telemetry from there and if there is a defect that sneaks out into production we can very quickly put it through the pipeline again and get it out and recovery or an experiment that we carry out in production or any other kind of investigation that we want to explore if you want to manage complexity the testability of our system is going to drive the modularity of our system it's going to encourage us to build systems that are architected to be more abstract have those lines of separation within them that allow us to make changes in one place and not in another it's going to encourage a separation of concerns because that's what testable code looks like and it encourages to architect things for loose coupling and higher levels of cohesion so let's have a little look in a bit more detail about the the way in which we're going to organize some of those things we've got some developers here i've drawn the drawn this as a pair because that eliminates the need for separate code reviews because we could code review as a pair to start with commit a change and that's going to trigger the the commit cycle the commit cycle as i said earlier is about evaluating changes from the perspective of a developer we want very fast feedback um to the developers on whether the code does what we what they think it should do there was an academic study a few years ago of production defects and what they found was that 58 of production defects were the kinds of mistakes that all developers in all languages put into code all of the time so off by one errors conditionals the wrong way around scoping problems those sorts of mistakes that we're all familiar with that's the that's at the root of 58 of production defects according to this study and those are the kinds of things that test driven developments and the kind of unit testing i'm talking about here essentially eliminate that's one of the ways in which we end up being more efficient because we don't get those sorts of bugs so often so we're going to get these fast feedback on these things the deployment pipeline was originally also built to cope with the different rates at which we need to evaluate ideas because some of these evaluations are more costly in terms of time than others if we want to run a unit test then that's probably going to take microseconds of do you know some in-memory evaluation of a small piece of code if you want to do a performance test we've probably got to start up some significant chunk of the system initialize it with data and pump it with w with cases which is going to be take significantly longer so the idea of the deployment pipeline the other idea of the deployment pipeline is that we're going to try and optimize to fail fast we want to do the stuff that's straightforward and simple and get results back really quickly and then for the slower stuff we're going to allow the slowest stuff time to be able to evaluate and we can parallelize it and do all sorts of clever engineering things to get feedback as quickly as possible but we give ourselves the freedom to you know deploy the whole system and try it out you don't really want to be deploying your system in the fast feedback cycle because that's going to slow you down too much so this is the commit cycle and as i say this is largely focused on supporting development if you follow my advice and practice test driven development one of the other bits of feedback that you get from working this way is feedback on the quality of your design if you do test first test driven development you write a test and the test is hard to write because the setup is complicated what that's telling you is that the design of your code is rubbish and you need to redesign the code i don't know much else that gives you that kind of feedback that early in the process on the quality of your designs and that's one of the reasons why i value it quite so highly the commit stage is going to evaluate from the from the point of view of the of the of the developers is the code doing what i think it's doing primarily that's what you're really the question is really answering and then there's the acceptance cycle and the acceptance cycle is is the code releasable is this change safe to put into production is it fast enough resilient enough secure enough compliant enough all of those things whatever it is that determines the releasability for your software we evaluate within that within that part of the deployment pipeline and then there's a final pass the final phase which is the production phase and this is about getting this out into production and that may be simple or complicated maybe you've got production deployment with canary deployments and blue-green deployments in multiple ways and failovers and multi-data centers and all that kind of stuff and all of that stuff will be implemented as part of the production phase using infrastructure as code techniques and so on so we reduce the variables in the whole process so that we're sure of that stuff as well so let's look at a little bit more detail of the kinds of things that these different parts of the system are likely to be doing the commit stage the typical kind of model for the commit stage is something like this if we're using a compiled language you're going to do so we're going to compile the code we're going to run some unit tests maybe do some static analysis of some kind i quite like to automate the coding standards for for the team so that if you commit some code that doesn't meet your coding standards it rejects the change but you can do all the kinds of static analysis that's useful one of the projects i worked on we did an analysis of any input field and we did a sql injection attack on any input field on an automated basis and just looked at the outputs to see whether any sequels had snuck through so there's smart things you can do to give you that fast technical feedback at this point what we're looking for at this point as i've already said is very fast feedback and ideally um we want that to be in the order of minutes now we're not going to be able to evaluate every use case of the system in that way and we and there are different levels of testing that are important here the testing that i'm describing is kind of developer focused technical testing we also want to have something that's more product focused it does the product actually do does our software actually do what we want it to do what our users want it to do and that's the role of acceptance testing for acceptance tests what we'd really like is that we'd like to look at the software from the perspective of an external user of the system does it do what they need it to do whatever that might be and we'd like to be able to evaluate the software in life like scenarios from the external users perspective and we'd like to do that in a production-like test environment my preferred approach is to deploy the software using exactly the same tools techniques and as close to the production configuration as i can possibly achieve into a test environment and then simulate user interactions with it this is best done with the use of what are sometimes called executable specifications we build a development process around the creation of these executable specifications that describe the behaviors the outcomes that we would like from our system and we'd like that to be in the language of the problem domain this might be sounding a bit like behaviour driven development because it is one form of behavior-driven development but primarily what we're trying to achieve here is a separation between what we want the system to do from how it actually does it we'd like to be able to come up with a behavioral specification of what the system should do and the outcomes that we expect from it when it does those things from how the system actually works and when you get this right you can do that to the degree that you could throw away the system and replace it with an entirely different one working on a completely different tech stack and the test would still be correct that sounds like an extravagant claim but i've literally worked with clients that have done that using these kinds of techniques usually what we do is that we create a domain specific language that allows us to create they create these test cases in some detail we're going to deploy the system into this lifelike environment so we're going to configure the environment to start with using infrastructure as tech code techniques to make sure that those variables are controlled then we're going to deploy our release candidate the sequence of bytes that will end up in production as accurately as possible we don't want that to change we're going to evaluate the sequence of bytes that ends up in production a jar a xc dll a table a docker image whatever your deployable thing is that's what we want to deploy and evaluate then we're going to smoke test and health check start it up make sure it's up and running then we're going to run all of these scenarios these these these acceptance test cases this is a fantastic way of organizing the development and he has a lot of knock-on effects because it tends to push the behavior all the way up the stack so that you going to prefer requirements that also say only what the system should do and not how it does it so he's going to start establishing this ubiquitous language that describes the problems of your system that it tackles and that's going to automatically lead you into creating these executable specifications that define that in the context of the feature that you're working on and then you can do tdd tdd underneath it until you've got something that passes all of those tests and you kind of got an automated definition of done it's a really nice way of working i'm not going to talk any more about acceptance testing if you'll forgive me advertising i have a youtube channel and there's lots of videos that talk in more depth about different aspects of creating these dsls and working with them you can see some pictures of some of the videos there in general what we're looking for in terms of feedback i said we're looking for fast feedback my recommendation is to optimize for that speed remember there's no trade-off between speed and quality and if you want to do high quality work we want to increase the speed of feedback we can use that as a tool to drive us in the direction of better behaviors so my advice is that from the commit stage for the technical feedback from your deployment pipeline what you're looking for is feedback in under about five minutes the reason for that is because you want to be able to commit a change wait for the results check that it's okay and then move on to the next piece of work in confidence that it's likely to be okay we also want to get quite a high level of confidence that that's okay though that means we want good test coverage at that stage so we're going to want a lot of tests we want to run a lot of tests at this point and these are best generated as the fruits of test driven development in my opinion for a variety of reasons you can read some of my stuff and watch some of my youtube videos to find out why i think that um for the whole pipeline my advice is to aim for feedback in under an hour and again that sounds kind of extravagant if you've got if you're working with a big chunky system that might sound very very difficult to achieve and it might be difficult to achieve but striving to improve the situation i believe will kind of generically push you in the direction of better behaviors better engineering practices along the way why do i pick an hour it's kind of pragmatic in some to some degree my observation is that the longer the feedback of the duration of the feedback for your deployment pipeline the more difficult it is to keep all of the tests passing um i worked with a comp um a a bunch of teams there were about 2 000 developers working on a a bunch of trading systems in this company and they had their primary build was a big c plus plus build um lots of different discrete components and parts of the system in that build services that were deployed separately from one another sometimes but over but that but it was a large build it took nine and a half hours to do the build and to run all of the tests overnight they'd been running that process for about three years when i i started working with the team and we started looking into it and just we didn't change anything else except start looking at improving the speed of that feedback we worked to we tried all sorts of different things we experimented with all sorts of different different techniques but to cut a long story short we ended up bringing the build time down to the the commit stage build time down to 12 minutes we couldn't get it down to five and the the rest of the build down to 40 minutes what i was told in the three years preceding me joining the team they'd only ever seen two occasions where all of the tests passed in the first two weeks after we made this change to the build only speeding up changing nothing else we had two bit two builds that passed all of the tests and in the following two week periods forever after as long as i was with at that company there was at least one green build every single day so we could just pick the results of that build and deploy into production much more safely than what was upping operating before so the feedback has a dramatic change on the effectiveness of the work that we can do the quality of the work that we can do just by increasing feedback there's kind of an obvious reason to think why when you think about it if i am working with an overnight build i commit a change that creates a failure i don't find out about the failure until the following day i immediately realized oh well i know what i did wrong i commit the fix immediately it's going to be broken for a whole working day it's not going to be fixed until the following day the day after that so it's going to spend half the time broken so the faster the cycle the easier it is to stay on top of failures if this were all real engineering it would improve the quality and the efficiency of our work i want to remind you that the results of the state of devops report i think these numbers point to something important i think these numbers point to the fact that what we're talking about here is a genuine application of scientific rationalism to solving problems in software i think we'd call that engineering i think we'd call that software engineering and if we were to come up with something like a genuine engineering discipline for software we would expect to get better results and we do that's what the data says so i think that this is a way of creating better software faster and i think that's as a result of us applying engineering style thinking to solving practical problems in software development so i've rushed through this to try and give us a little bit of time therefore questions but my advice to you is to treat your deployment pipeline as an important thing make it definitive in terms of the releasability of your system that means that the scope of evaluation of your deployment pipeline should be a releasable thing if you've got to if you transit your deployment pipeline and then you've got to go and evaluate it with some more stuff before you're ready to release it's not really a deployment pipeline it might be useful but it's not really a deployment pipeline a deployment pipeline goes from commit to releasable outcome so that says something about the deployability of our system the size of evaluation of our software and the efficiency with which we can evaluate it if we start optimizing to get these definitive results we start to see these remarkable outcomes in terms of the performance of our software development approach and process thank you very much for listening to my talk i have whizzed through that rather quickly and i'm very happy to take questions i think we've probably got about 10 or 15 minutes for questions now [Applause] you

Channel: GOTO Conferences

Views: 24,090

Keywords: GOTO, GOTOcon, GOTO Conference, GOTO (Software Conference), Videos for Developers, Computer Science, Programming, Software Engineering, GOTOpia, GOTOcph, GOTO Copenhagen, Dave Farley, Continuous Delivery, CD, DevOps, Feature Branch

Id: MYVrLXKJp0Y

Length: 33min 21sec (2001 seconds)

Published: Wed Dec 15 2021