PAINFUL Software Release Cycles Are NO JOKE

do you work on a project that is a delight where you can easily make changes quickly and with no fear or is it a bit trickier than that do you have a long production process with lots of quality control Hoops to jump through and demands for more and more features being added all of the time but as your software is still getting harder and harder to work on making integration and releasing is more and more complicated as it grows do you get lots of bugs in production and only find out weeks or months after a change that it broke your software do you have the freedom to improve things or do you have to wait for permission from someone after every change to your software or Improvement to your development process if you work in an organization like that how on Earth did it get there and how do you get out of this seemingly impossible position or does it all just seem too hopeless in this video I'll share my experience of working with just such an organization and how they found a route out of this mess with continuous delivery moving from not being able to release software at all to releasing changes every week if you want to learn how to make these changes where you work or stop them from happening in the first place then keep listening I've got a special for offer for you later on [Music] hi I'm Dave Farley of continuous delivery welcome to my channel if you haven't been here before please do hit subscribe and if you enjoy the content today hit like as well let me pause here to say thanks to our sponsors we're fortunate to be sponsored by equal experts trisentis transfig and Ice panel all of these companies offer products and services that are well aligned with the topics that we discuss on this channel here every week so if you're looking for excellence in continuous delivery in software engineering click on the links in the description below to check them out at the start of a new piece of work it's an unavoidable fact that we know the least that we're ever going to know about it after all it's a problem that we haven't solved yet we don't even know any of the things that we need to know to create a plan at this stage the big looming snake pit of a trap here is the illusion of rigor the idea that we can come up with a good plan that will Define the work and how we will approach it this is where to my eyes a form of Madness seems to creep in at the point when we know the lease we will ever know we create a plan for how to do everything well at this stage I already don't fancy our chances much do you we can start with the best will in the world we may know that plans never survive survive first contact with the Enemy but there are lots of quotes about planning and you can pick the one that fits your thinking my impression of most organizations is that they prefer this one failing to prepare you're preparing to fail but preparation is not the same thing as deciding what Alice or Bob will be doing in six months time deciding on things that you'd like to learn before you progress further is also good preparation here's another project management quote give me six hours to Chuck down a tree and I'll spend the first four sharpening the ax preparation is about doing things to make the job easier not planning each cut of the tree in detail there's also this from The Hobbit it does not do to leave a live dragon out of your calculations if you live near to one in software we live pretty close to some fairly grumpy dragons this may sound like I'm talking about the start of a grand new development project but all of this is just as true for a new feature if that feature is more than a few days of work and certainly true of the annual or semi-annual budget rents that are common in big companies it seems naive to me to expect plans like these to be accurate we don't and can't know all that we need to do to come up with an accurate plan for things like this on the other hand some sense of direction is good but we don't necessarily need or can know even that with any degree of accuracy from the start so leaving our options open so that we can change our approach our priorities or even our Target destination is not only wise but in reality essential for solving complicated problems where the steps aren't obvious and are always likely to change as we learn more doing work to allow us to learn and change our minds is a wise course of action checking our progress after each small step and the continued validity of our current preferred destination is also very sensible so now if we have fallen into the snake pit we'll have come up with a grand if illusory plan now we're on the path to all kinds of trouble our product ideas are almost certainly wrong and the bigger the plan the more the pressure to skim over the details and the more the likelihood that we'll think of changes to the plan as bad things instead of as the inevitable consequence of learning which they represent we will probably be a bit lazy too and sketch out our vague wishes for product ideas then ask the development teams for accurate estimates and hold them to it this is sadly more common than seems at all sensible to me every book on project management that I've ever seen says that you can't fix time and scope and yet the vast majority of organizations that I've ever seen try to fix time and scope let me remind you of where we are we've just fixed time and scope and we haven't done any real work yet at all on the new idea we don't know if the technology will work we don't know if the design makes sense we may not know if the team is organized properly we don't know if our ideas make commercial sense or if our customers or users actually care about them or will find them useful at all but we do have a plan and that now feels hard to change the problem with plans like this is that it's human nature to gloss over complexity and to ignore the dragons until they start biting us that is so our plan is over optimistic always every time you'd think that we'd learn from this but we often don't seem to estimates are problematic because all of the pressure real and imagined is on us to underestimate early on we have a sketchy overly simplistic view of the problem oh yeah that's easy later as we get further and further in we start finding out all of the things that we didn't think of some of them might be dragons now there's lots more work to do that we never allowed for in our estimates or our plans when someone asks us how long will it take we all know that what they'd really like to hear from us is no time at all so even if we are grizzled veterans of lots of failed estimates we still like to be liked or like to get the job or like to be seen as one of the good team players so as well as not seeing the complexity we're also a bit tempted to downplay it but we aren't completely stupid so we also like our plans to be vaguely real at least most experienced people that I know say that they usually double as to the estimates that they first think of the problem is that the data says that estimates at this stage of a project before we've actually started doing any work are out by a factor of four so doubling only gets you halfway there and if you did multiply by four you'd never get the job in the first place and you'd be treated as the grumpy person in the corner the other ludicrous pressure at this point is for people to move on the people specifying the need are often busy and important to people so they can't spend all of their valuable time agonizing over every Last Detail the trouble with working in small steps is that it requires constant involvement of people with a vision for the product extreme programming called this customer on the team you can't realistically throw a meaningful product division over a wall because as I said that vision is going to change and if it doesn't change as we learn more things we aren't doing a good enough job of learning so back in the irrational world what these busy experts really want is to define the vision quickly with the least amount of effort and then move on to something new and presumably more important then they'd like to come back later at the agreed date and collect the perfectly realized implementation of their perfect vision and take all the plaudids the problem here is that in the real world their Vision was flawed not detailed enough missed some dragons and we misunderstood it in the irrational world then it makes sense to organize these projects as bigger things so our experts can feel justified in providing just the rough sketches and throwing them over the wall without feeling too guilty so when we talk about a plan what we really mean here is a plan for usually months or years of worth of work plans like these are expensive so now we need to do all sorts of extra work to sanity check them after all we don't want to be wasting Millions on some stupid idea so we need to do lots more work on the plan to make sure that it's more accurate and since they're never accurate let's make lots of bureaucratic rules and hurdles to try and make them more precise instead of being more accurate does this picture seem familiar yet there are lots of death spirals open to us now one of them is what my friend dantehurst North calls the infinite fractal Coastline of estimation the closer we look the more detail we see the bigger the project looks the scarier it gets the more process we add oh no here are more things we hadn't thought about and so on the other is the lure of bureaucracy as the project gets bigger and more expensive the moment we want to be able to confirm that we aren't taking too many silly risks so we put more gates into the process all of these things take more work and more people and as soon as we add more people there are overheads this is a law of ever diminishing returns at least for creative things like product and software development so now we're going to need more bureaucracy to coordinate the work of all of these people and inevitably there will be more work involved in this so the project gets bigger once again so now we've got this enormous months or years-long plan for our product there's an enormous amount of work and a very long way for anything to come out of the other end now our organization is structured around these enormous inflexible inefficient processes and we learn to adapt to them one of the ways that we adapt to them is to make sure that we load up every project with everything that we can think of everything that we can imagine ever possibly wanting every Bell and whistle because if we don't get everything in right now and we're not getting stuff very often we might as well get all as much as we can when we do get something right now the project grows again where this usually ends is in a tangled mess of software and organizational processes tactical hacks and ever increasing pressure to build more features the way to get out of a vicious circle like this is to break the circle there's only one solution here we have to simplify things we need to change things to allow us to work in smaller steps I once worked with a client that had been unable to release any software at all into production for well over three years I worked with them to help them adopt the practices of continuous delivery of working in small steps and optimizing for fast feedback and treating each small change as an experiment to learn from they went from not being able to release software at all to new releases every three months even for their most intractable pieces of software and much more frequent releases than that for the rest one developer told me I feel like we've got our future back is possible to make this change to go from protracted bureaucratic frustrating approaches that burn people out to a much more Dynamic productive creative way of working that engages people and produces better results if you're ready to learn how I have a course where you can learn about the continuous delivery approach to building better software faster and I've just added a new bonus section where you can write your own continuous delivery capability and get a free report from me we've targeted a device about how and where to improve your continuous delivery practice for a limited time you can also save a hundred pounds the details are in the description below now back to this company that made this change for real before they started with continuous delivery each change was fraught with risk and destabilized the work of lots of other people changing anything was scary and so slow so they'd attempted to reduce the risk by increasing their bureaucracy to the extent that one of the developers that I spoke to had been asked to put a bug that they'd found and fixed back into the code because it wasn't on the plan to fix it when they found it this is not an easy place to start from to improve things but the fundamentals are the fundamentals for a reason it's because they work whatever the place you're starting from I advise them to begin by focusing on improving feedback and to do that by improving things in small steps this wasn't at all easy there were technical organization and social barriers everywhere that we looked but this is another reason why it's so important to proceed in small steps because after each small step we can see the impact good or bad and we can show the Improvement to others and gain some more trust in others to help us and to continue making changes also now we're starting to work more experimentally trying out ideas to see if they work discarding the ideas that don't and doubling down on the ideas that do this organization though a big multinational firms started this activity with a very small group of people at the start there were only a handful of them working very closely together they were committed and so fed up with how bad things had become that they were they were committed to changing things and a bit feisty about it too they aimed to challenge the status quo and they did they weren't taking but this is always how we've done things for an answer I helped them to identify targets to aim for that would steer them towards success and the things that we've already mentioned what one of the main goals was to get feedback on the release ability of their software at least once per day they started with the simple things like adopting better ways of working for new Greenfield projects creating simple deployment pipelines establishing continuous integration and much better automated testing for those new pieces of work where it was easy to add them later tackling the huge barriers like overdependence on manual testing poor control of tests and production environments and very very slow Planning and Development cycles and so poor feedback for the bigger more difficult products as a result of these changes the teams on the Greenfield projects did better immediately but also the org was now gaining valuable real experience of this different way of working I helped with training and coaching and clarifying their vision and their goals a bit but they did all of the work there were some good wins and some big setbacks as they discovered lots of dragons along the way the more complex bits involved as ever difficult to deal with Legacy systems for some of these the builds alone took hours to complete and even then often failed production deployment was complex and sometimes took weeks to commission new versions of software in the field so the team driving the change did the technical work to stabilize the builds and optimize them and automate deployment this was hard graft and the benefits weren't immediately earth-shattering but they now had a better sense of direction and a way of working which meant that improvements were clearer even though small they were optimizing for fast repeatable feedback and they were doing this in small steps or by taking a more experimental approach to change they'd try out their best theory of how to make the next step forwards toward their goal whatever that might be and adapted to whatever it was that they learned from that small step this surfaced all sorts of challenges along the way so some of them they knew before they began some I helped them to avoid and others were traps that we fell into anyway and then had to find a route to climb out but using the continuous delivery principles as our guiding lights the team of change agents made real company-wide difference and never once took a step that reduced quality or efficiency as far as I ever saw first they started releasing things from the Greenfield teams sometimes peripheral products sometimes really useful additions that made real differences for their customers a small team of change agents grew in influence and size as a result with more and more people willingly joining the push to do better these teams were now a lot more product and user focused than before too because there was less bureaucracy and process between them and the real problem that they were working to solve the teams working on the more difficult Big Balls of mud inevitably took longer to gain traction they were nervous of being able to achieve the kinds of speed of feedback that I and the change agents were pushing them to aim for so they did a lot of work to decompose some of their bigger systems into smaller more tractable components and subsystems this is sometimes useful but always a slightly risky move because the danger is that you end up with a big dependency management problem after this change and they did suffer from this they certainly achieved better Insight better visibility and a faster pace of change locally by dividing the software into these smaller teams and smaller pieces but at the cost of losing visibility into the integration of the whole system not really any worse than before perhaps but now more clearly a real problem that they needed to tackle they came up with a tactical process hack to manage change between teams this got them to a significantly better place than they'd ever been before they still had to work very hard to get their first release in several years out solving lots and lots of organizational and Technical problems along the way but they made that first release the real test though was that six months later they released again and it was a lot less painful and three months later they released again and three months after that again and ever since again this company is today releasing new software all of the time the nature of their business means that they can't do that for all of their software they still have big complex Legacy systems but now they can work on them with much greater confidence and efficiency than they did before now even for the difficult safety critical Parts they make constant well understood observable progress and produce release candidates if not daily then at least weekly for most parts of the business this is not an example of beautiful theoretically perfect continuous delivery but then it often isn't after this kind of transition but this is a real company that regained control of their future through changes to their software development approach it's a bit grungy and less than ideal in some places but they are now working on improving these things too in a series of small steps optimizing for fast feedback and treating each change as an experiment they now produce new releasable software even for the grungier bits reliably and repeatably on a at most a three monthly basis and for most parts of the organization on a daily basis this is the impact of getting the model for software development right and continuous delivery defines the state of the art for being able to achieve that thank you for watching and if you enjoy the stuff on the continuous delivery Channel please do consider supporting us and joining our patreon community there are great conversations going on in Discord right now thank you bye foreign

Channel: Continuous Delivery

Views: 15,318

Keywords: release cycle, software release management, software release cycle, software release process, long release cycles, long release cycle problems, software release, software life cycle, software engineering, software development, dave farley, continuous delivery

Id: 4UeBH7rY4lY

Length: 21min 50sec (1310 seconds)

Published: Wed Jul 26 2023