It’s Not Continuous Delivery If You Can’t Deploy Right Now • Ken Mugrage • GOTO 2017

Video Statistics and Information

Video
Captions Word Cloud
Reddit Comments

"If you're doing feature development on branches, you aren't doing CI/CD."

That's a straw definition of CI/CD--so extreme, in fact, that nobody outside the startup world could reasonably do that. In fact, it's a philosophy that wouldn't work on any sufficiently large project.

There's no point in listening further. The fucking article is wrong, and the speaker needs to be ignored. Pull requests and code reviews exist for a reason.

👍︎︎ 126 👤︎︎ u/thephotoman 📅︎︎ Mar 20 2019 🗫︎ replies

Deploy what?

There are many things you can't deploy because of reasons that are well outside tech (business, marketing, etc). Within the realm of tech, there are important distinctions. For example, it's unwise the deploy the work of the new intern. Not so much the work of the team leader.

👍︎︎ 14 👤︎︎ u/ApprehensiveSet3 📅︎︎ Mar 20 2019 🗫︎ replies

FYI, here's the talk Abstract

I hear people say all the time that they're practicing continuous delivery. This declaration is often followed by something like, "I can let the security team know anytime", or "I just have to run the performance tests". If you can't push your software to production right now, you're not done with your continuous delivery journey.

In this talk, we’ll go over code management strategies, deployment patterns and types of continuous delivery pipelines you can use to make sure you can “deploy right now”.

👍︎︎ 11 👤︎︎ u/goto-con 📅︎︎ Mar 20 2019 🗫︎ replies

I don't need to deploy right now. I did deploy already automatically, I'm using CD.

👍︎︎ 6 👤︎︎ u/AngularBeginner 📅︎︎ Mar 20 2019 🗫︎ replies

This guy sounds like one of those scholastic (medieval) philosophers debating the essence of things. What is the "grassiness" of grass ? Is it still grass once it's withered ? At what point of the withering process does it, perhaps, cease to be grass and becomes hay ?

👍︎︎ 2 👤︎︎ u/sionescu 📅︎︎ Mar 21 2019 🗫︎ replies
Captions
[Music] okay so my talk it's not continuous delivery if you can't deploy right now basically the idea here is that you know words have meanings and might be doing really good automated building tests you know what have you but if there's not a system that you can log into and push a button right now to deploy to production then not quite continuous delivery yet okay so Who am I I'm a technology evangelist for thought works which just means that I get paid to fly around and do this which is kind of cool it's great job I do want to tell you that this is going to be pretty opinionated so I come from a company that's very opinionated some of the books written by current and former employees so I'm going to go into some things and say this is the way I think it should be done doesn't mean it's the only right answer there is no one right way I don't think but it's ways that have worked really really well for us okay so why this talk so I think we thought works a little over eight years and was on like our continuously read products for the entire time for the most part so I've seen a lot of deployment pipelines and CI and CD and so forth out in the market and I would hear quite often so yep we got a canoe slurry pipeline' no problem I just have to give it to the security team and then three months later I can deploy or what have you and so it was still very siloed it was still you know just better continuous integration not quite continuous delivery and so the point of this talk is there to show you it's going to be fairly high level because we're going to cover a lot of things but here's some types of testing that you might not be doing in your CD pipeline here's some code management strategies you might might thinking about some deployment strategies you might not be thinking about not going to go terribly in-depth on any um be around the whole two days we want to go more in depth but the idea is your takeaway is wow I didn't think that I should be doing that kind of testing as part of my automation you know because you really the purpose of a continuous delivery pipeline is to kill a release candidate okay it can't prove something is good but you can absolutely prove it's bad so if I code if I do a commit that's a release candidate and if all my tests pass be so confident my pipeline that if I wanted to I can push to play that's the purpose of a CD pipeline um and just because we want to do a Star Wars thing because I'm a geek it's real that's what I mean there's not a half way in it but there's also never done so you get to do these things it's an ongoing thing don't just try do it okay so why continuous delivery so before what I'm going to do why we do it a certain way why is it important to us lots people familiar with the agile manifesto allister's here a lot of people haven't read page two it's funny the term continuous delivery is commonly attributed to just humble and Dave Farley who wrote a book by the same title and in fact it comes from page 2 of the manifesto and so if we go all the way back to agile you know our goal was supposed to be continuous delivery of valuable software but we as an industry put in good building tests like CI practices and we had our release candidate and then it got handed to somebody else so we didn't actually deliver software to customers and so that's I think continuous delivery is kind of in you know the DevOps culture is kind of finishing what agile started and just because of that I pulled out a slide that I used to an agile training about 15 years ago why do CD because partially done might still be useful there might be functionality that's done that I can make use of and make money off of most of us are in those kinds of organizations and so this is literally a slide from agile training where you don't paint the Mona Lisa that way you paint it this way you continuously build rhe is the same thing if I can get features into customers hands at the very earliest times I can get real feedback on is this going the right way you want to have fun I didn't do a slide of it look up an x-ray of the blue man it's a famous painting but they didn't x-ray to show all the versions of the painting before it came what it is now it's one of those 20 million-dollar paintings but it did not start out to what it is so we want to get value sooner this look reason is hackers aren't going away you know it's getting worse and so if I have a system where I can go in and update a library if I can go in and update OpenSSL and be in production in 20 minutes that's a very good thing and you you have to make sure everything else is right in your pipeline before you can really say we do that when this when heartbleed came out a couple years ago a lot of companies oh my gosh they logged into production and they updated their SSL libraries and so now they're safe but their applications didn't work because the change didn't go through all the testing it's just with somebody logged into production thou shalt not have root and I'll go more into night capital who knows night capital yay only a couple I'll go to night capital in a minute as a reason why you should have this so I want to talk a little bit about continuous integration you cannot do continuous delivery if you're not doing really good continuous integration . so fix this first if you're not doing this yet we talked a lot about it's something we call CI theater this is where you know I got a CI server and I have a few unit tests and you know okay here and there but I'm not really doing all the things so I'm going to go into what the things are here in a little bit so you're probably not really doing advanced integration we actually did a study a couple couple years ago where we went out and interviewed a bunch of people and said are you doing CI yes okay let's talk about your process only 10% of people would even acknowledge you all I have a CI server I have you know go CDRs I have Jenkins I have bamboo I we're doing CI having a product and doing steams integration or two different things okay and so there's things that have to be done here I mean and you know goes back to extreme programming principles it's test-driven development pair programming it's those kinds of things to make it solid a couple of those practices anybody bring in raw fruit I hope not don't throw it at me I'm going to go into some opinionated stuff now if you're doing feature branching you are not doing continuous integration okay you might be doing really good automated build and test and that might work great for you and okay that's fine but continuous integration means building off trunk trunk base development master development what have you and it's important to contain its integration you'll see why are we getting deployment methodologies but I do want to go into this a little bit because this one's controversial so here we have Professor plum and Reverend green if professor plum is working on her code and she's doing some things and we're doing bug fixes on mainline and you bringing those in and that's working good the Reverend greens over there and he's working on his things he's bring it in bug requests and everything's good and then eventually one of them finishes first and so professor plum finishes merges back into mainline trunk master whatever you want to call it everything's good then Reverend green doesn't get pull and we had this massive merge conflict I like to joke the technical term for that the big ball of mud now we all know there are really good merge conflict tools out there I can find the text differences and say choose this one and that one you know etc but they don't look at intent they don't say this is why the code changed or you know that person's an American so they used feet and that person's from Europe so they use meters or what have you they don't talk about intent and so we strongly believe the right way to do this is to be pushing and pulling to master at least once a day so I work on my code I commit it to master trunk main line again whatever you want to call it the continuous integration includes delivery server is watching that not watching all the future branches necessarily it could but the branches yeah you might work on it for a few weeks but you're pushing your changes to master every single day we've stolen this term a lot but Benjy was first time I saw it it may have been used further so when everybody has their own branch in their own work and what-have-you we call that continuous isolation so you can run all the tests but it's really not continuous integration because the word integration means integrated code so from a CI perspective what I really want to encourage you to do is commit more often you I mean at least once a day and more and faster is better you know if I if I'm committing constantly and there's things we have to do in the code to make it so we can do that and the tests fail I have that much less code to go through to find why it failed something heard to do it more often again I we stressed trunk based development a lot I forgot to put the link up there but if you go to trunk based development com like that we didn't create but it goes very in a lot of detail about why and how you can do it and some of the benefits and strategies etc it's from a gentleman who used to work for us but he's done this with very large projects and it really works and then automate all the things so with that and it said fast moving no but I'll have questions at the end by the way I mentioned at the beginning that there's a lot of things that people say oh we don't consider it we just have to give it to this team or that team I live in Seattle in the United States and I was at a customer and I said yeah we deploy once a week and that's fast enough for us like cool so if you had to change in your code today it would be a week to get any production well know there's a six month hardening cycle so they deployed once a week but it was dev complete from six months ago so doesn't quite count who here could get code into production today if you had to awesome that's way that's going up that's great Mary poppendieck you started doing that question which is doing lean talks five or ten years ago and nobody could do it in one day they all had recruits I close and what have you so on your continuous integration teams delivery server there's some testing that you may not be doing that you may want to consider so first off is security how many people do security testing that's a part of their automation in CI CD yeah only what five percent maybe at that there's lots of things you can be doing now don't get me wrong there's I have a friend Jason who sits alone in a dark room and does pen testing and you don't want to replace Jason he's really really really good at that but there are a lot of things you can automate there's actually pre commit things there's an open-source product I think it's called talisman and I will publish all the references by the way that the developers plant on their own machine and you can do commits but when you do a git push it runs talisman against it and if you have an Amazon key or any of that stuff in there it's not getting pushed because if something touches github you're done that key has to be replaced if it's there for a millisecond you're done and so test for that put on the developers because mistakes happen oops went to one too far sorry so Sona type is a company that runs maven central so if there's any Java developers that use maven and your project they did a study that the most of the people that use maven so people use an open source java stuff was 106 components on average 24 of them had known vulnerabilities that had been fixed but when they setup maven they said get this library version 2 and get that library version 3 and meanwhile library version 12 has been out for years but they don't know that and so I mean you know what is that almost 20 22 % 20% it's scary I mean there's tools for these I'm blanking on the ones of Java but if you're using Ruby and gem files it's called break man if you do a pull request to the product I work on on github the CI tools going to run all unit tests you can watch all that but meanwhile we're running break man make sure you didn't use any gems that are that have this problem so you can do that sorry the formatting got a little bit changed but there's also a lot of tools now that can do you a based and they can do sequel injections into cross-site scripting and all that automated these tools there's open source ones the commercial tools aren't that expensive depend on your scale your definition of expensive these are all things that should be in your pipeline okay so every code commits gets security tested anybody doing performance testing in their pipeline even fewer than security yeah Amazon did a study and Amazon has 300,000 employees right now they're about to hit a market cap with a trillion dollars so they're kind of good at what they do but they figured out that half a second in response time makes a 50% cost of business so is that the you know website slows down that much people leave and they'll go buy it somewhere else performance is massive to your business success test it these are going to be really rudimentary for anyone that's done any kind of performance testing but I want to go over some of them real quick because some of them are easier to do on a pipeline than others so load testing is just what it is you know what you can have this many clients I do the access what happened you know you look at CPU percentage that's pretty that's one easy stress testing is a little bit harder stress testing is let's pretend a web-based app just because pretend so we have our app and I do ten clients 20 50 100 200 300 500 crash ok my stress test says I can do up to 550 so if I'm going to launch a new product in November is in the western part of the world I probably need to know that and have the pasady planning ready for that again that can all be automated I have never seen this in a pipeline outside of thought works soak test run something for two or three days and see if there's a memory leak test those kinds of things okay now if you're doing continuous deployment which is defined as I commit code a whole bunch of automated stiff runs and if everything passes it goes to production then you probably don't want to do a three-day soak test okay but if you're doing continuous delivery which is my code can be deployed at any time and a human makes the choice okay if you're an on-premise product a mobile product something like that we are not pushing it every day you can put soap tests it's not going to soak test every change because it's going to do a change it's going to join three days of testing and then the next whole group of changes you know etc you don't want to do first-in first-out you wanna grab all the things for the latest one but you can do this in a pipeline and then spike testing again depends on your industry this may not matter but you have ten clients you know you throw a thousand at it and see what happens now here's the thing we all say those of us who do you know DevOps continuous delivery talks fast feedback the whole point of this is to fail fast to get feedback very very fast and so people are like we just test ik days and this one takes at least hours and I'm going to slow my whole pipeline down you know I can't do that I'm not going to you just said can I get production into Kotov you know into code into production today not if I have a three-day spike test or soak that sorry you should be able to do this stuff in parallel okay so I do some unit test that builds my basic deployable unit and then on separate environments automatically put it on the functional test the load test the spike that's the fast things and I don't you can see the arrows very well but you know those are saying that these three all have to pass or my staging deployments not going to run okay but the purpose of a staging deployment is to test the deployment on production like hardware so if you're deploying to Solaris in a cluster using Oracle then your staging environment needs to be Solaris in a cluster using Oracle it might only be three servers instead of a thousand but it needs to look the same I can tell you that if that works had done this on a project I won't name in 2006 the continuous delivery server would still be booked excuse me would not exist we were building a Java project all the developers had Windows laptops didn't get Solaris machines which was the ultimate deployment target until six months in they got the machines deployed the software it didn't work I don't mean it had deployment problems it could not start turns out they used a way to access the file system that NFS and Solaris didn't play well with and so if they had had a staging server with Solaris on it they would have saved literally weeks worth of rework now this is a simplistic test our simplistic view of this this can get pretty complicated that's a screenshot of an actual pipeline you know there's eight source code reposing any number of other pipelines that have dependencies in between them this particular one is an open source product and so there's certain things that we do on every pull request but there's other things that only we control and hey we'll update the functional tests because those are harder to update and we get it and so if you do a PR for this product you can attest off the pass or we take go away but if those pass then and only then does it go any further and eventually it gets the functional test if those don't pass we'll fix them but we do that because in a separate source code repo using separate materials but this is all one product a funny thing on this one is if i zoom in to the end it's really important I believe to let your team's decide what's the best thing to do for them from a risk perspective this particular project has I don't know 12 or 15 thousand automated tests and so if we get a really bad performance thing it shows up there we have reporting and logging on how long it takes to run the functional test and it shows up there that this took way longer so they know we do the big performance tests after production okay that test cost six thousand dollars to run because it's using a whole lot of like Amazon machines that run for a very long time in etc and so this product releases once a month it's an on-prem product so more than that wouldn't be wouldn't be acceptable we've been watching the graphs of how long the functional tests are we're pretty confident in the performance and so we only run that test once a month that's the right risk profile for this okay if you're competing with Amazon that's not the right risk profile but you should be able to decide and I highly recommend you let your team's decide what's the best for them mobile app has different requirements than an internal CRM app for example also don't be afraid to do little hidden things okay that right there says security checks I mentioned security already a little bit but again this is a real open source product that's being built if you commit code it builds Linux and Windows and while that's going we're running break man to see if you put any gems in there that are going to cause no there's bigger security test later but this is something that for our architecture we could do quickly right at the beginning so think of things like that what are the things that you can do that are just a pass no pass if it breaks this know thou shalt not pass right at first you know there's if you look at like the exams delivery book or blogs or whatever they all say unit tests then functional tests then bla bla bla bla that first stage has a whole bunch of selenium functional tests in it they're fast they tell us what we need to know what the heck put them in the first stage you don't have to do it in the order of the book or whatever blog or whatever tells you to do order I tell you to do it do it in whatever order you want to so I talked about trunk based development you know deploying putting all the code in the codebase all the time but I also said you have to be able deploy right now well if I have a feature that's half done well how do I do that I can't deploy a feature that's half done there's lots of ways to do that and luckily these are becoming more and more common so I don't think much of this is going to be a huge shock so first off I want to differentiate between deploy and release they are two different things okay deploy means put the software on the servers or on the App Store or whatever release means make the software the feature available to the users or customers I can have software that's deployed with features turned off and so it's deployed but it's not released okay and then I can say on a big holiday I say offered for whatever reason I want to announce a thing June 10th that go to I can have the software deployed weeks ago but somebody could go in and turn it on and now with any big announcement believe me the Apple does not deploy their new store when they release an iPhone it's all ready to go they flip a router one of the more common ways to do that that is important if you're doing things like trunk based development is a concept called feature toggles and it's interesting because if you're if you're writing down URLs or taking pictures there's several slides on this but the URL changes because that comes from a a blog that Martin Fowler did in 2010 and basically what it says is I'm going to work on a new feature the first thing I do is create a config file that says it spits pet survey true or false and then in my code I say if it's true here's the stuff okay so if I if my config file says off and I deploy that code pet survey doesn't show up if it's on it turns on so I can be running this in CI I can have no parameters or however it is your CI system that says okay for these set it turn on and off for this set of tests may not want to do all of them but I'm checking the code every single time this can get very advanced guy by the name of Pete Hodgson did a follow-up article this year or late last year we talked about some of the ways that we use feature toggles on real projects and we do toggle routers so databases that'll do things they make decisions based on geography or browser literal or anything so you can get incredibly dynamic with these and they become not just a way to be able to do trunk based development but also a way to test your business so I can turn this feature on for some percentage of my clients and see if sales go up or down if they go up I keep working on feature as I go down I turn it off you know so we want to do a tie thing for the business and this is one way to do that there's lots of different things that we use toggles for pretty commonly so look sorry the dynamics and the longevity so lower left there is the release toggles so those are the ones I'm talking about we have a feature that's not done yet and when it is done we want to turn it on see if it makes sales go up or down sales go down we turn it off those don't live for very long and they're very dynamic now I want to be clear here this is technical debt it costs more to implement a feature turning on a limiting feature toggles than just doing the feature without them and you at this point once you decide yeah I'm selling more widgets you should go in and refactor and we move that code and so that feature toggle is not there in there anymore but it allows you to really test your assumptions and so forth and I'll just mention again night capital which I'll get back to notices one some experiments there that are very dynamic so again those can be based on geography those kinds of things I want to do an experiment and see how this plays in the Netherlands vs. Germany and do metrics and it's all in the code there might be permission toggles or operations toggles so everybody's Netflix big here I apologize I should know the answer to this and forever bring it up it's a Netflix streaming video what have you they have dynamic toggles for performance degradation so if they get a performance yet something happens they'll turn off features automatically one of the things the system does is when you log in it recommend hey you might like this movie recommendation engines the first thing it turns off you can log in and stream the thing that's already in your playlist but it's not going to do the computation required to recommend something and they do that automatically based on the load on the system so you can do those kinds of things and it's just a toggle there's even some permissions things up there so the load on the system is 96 nobody can even log in flip the permission toggle only people from Amsterdam are allowed to log in now you can get in I mean there's all kinds of things you can do for these things so feature toggles are in code if statements if on do this if off do that okay where that it comes from it's all another story so this is good for features but what about like architecture changes what about you know core libraries or what have you there's another concept and the title the name of the concept is really poorly named called branch by abstraction but it's not actually a branch so the idea is you have a component that you want to update so let's say it's the well here take one that we actually did on a project we have a project that uses Ruby on Rails it was rails 2.3 and we wanted to upgrade it to rails 4 that is a massive upgrade surely you can't do that on trunk except for we did so what you do is you have your consumers so your different parts of the application that are talking to the thing and you put an abstraction layer in the middle so instead of talking directly this thing they talk to this abstraction layer which then talks to the thing and at first it might be a one-to-one ok then you can add the new component in there and now new functionality has whatever logic to only talk the new component old functionality uses the old component and so you can over time do this the patate the product I was talking about was right in the middle of a rail - to Rails for upgrade which I don't there's any Ruby devs in here that's big lots of breaking changes right in the middle of that when a massive security I don't remember was heartbleed it was one of the OpenSSL ones came out huge they were able to get a new release out in minutes because they updated a configuration file that updated the version of a cell ran through the entire pipeline rails for was turned off and not it went if you are if you downloaded that particular product and change the configuration and there's a configuration file it wasn't even a router because it was on Prem product change the configurations are out how the rails for and restarted it would come up half the stuff was broken it wasn't done but it would come up and so you can do that we did same basic thing for a database layer on our different products these are ways that you can do things where you can only move what you have to move right now now I also want to encourage you to think about your risk profile okay so you know if you're doing a website where people going to do ratings on you know restaurants it's an important business but nothing's really going to happen if you know feature goes down or whatever so you have to think about your risk and so I want to people this people have we know about this but just do the bring it up okay so mean time between failures means I want to optimize to make it so I hardly ever fail mean time to recovery means I want to optimize so that when I do fail not if when I can recover very quickly people who are really Turin in continuous delivery almost never rollback because the changes are so small that I can roll forward so I do a deployment I have a problem I fix it which might mean actually fixing in it might be taking that code out but then I do a new release so I released version 12 it fails I don't go back to leaven I deploy 13 it might be code identical to 11 but it's a roll forward so now I have the history I have the test it failed I see what happened as an audit trail auditors love this those kinds of things now if you're working on aircraft systems as someone who flies well over a hundred thousand miles a year please do mean time between failure okay if you do in restaurant reviews meantime the Recovery's probably okay so it's something to keep in mind there's also a whole bunch of deployment patterns that we can do for this and again I know it's really fast but things you can look into do it anybody's ever heard of canary releasing cool so this one nobody would raise their hand three or four years ago but it was already really popular that's where you deploy some number of systems with the new verts new software test it and make sure it works and if it works you go to more right most people do this with the actual deployment so I'm going to deploy the new version to five percent of my servers run some metrics run some tests see once one works etc and then maybe deploy it to the rest you can also do this with feature toggles so I could deploy to all of them but do the canary through toggles it's a little bit safer a little bit easier and where you go dark launching is an interesting thing my favorite dark launching story is Facebook so a few years ago Facebook introduced a new feature where instant messaging where you can message back and forth to your friends etc and they had a problem how do you load test 200 million users texting each other it's kind of hard that's an expensive performance test the answer is you really can't so they got to a Minimum Viable Product and the feature was in every version of Facebook have you logged into Facebook before messenger was available for about a year almost it was actually there there's javascript in the browser that would execute and it would message X percentage of your friends and X percentage of them would answer you and so they started it by rolling it out in Silicon Valley to Facebook employees and then bigger and all through toggles by the way and so eventually for almost three months every single user on Facebook every time they logged in was messaging 30% of their friends or X percent I might be wrong in the percentage and some of those were answering so they were load testing in production when an anomaly would come up they'd roll it back they'd fix it they do it again they roll it back they fix it did it uh and so they did a huge flash and messenger is out and everybody globally now has facebook Messenger and it was flawless it works for everybody and a lot of people in IT we're how the heck did they do that they did it because you've been using it the whole time that's dark launching it's fun this one is relatively new so everybody's focus mode github right hopes to get probably the one of the most used features on get is get merge if I'm doing trunk lease development I what if I don't do a lot of code these days I have to admit but when I do the first thing I do is get checkout - be branch name but you just said trunk I'm going to push it back to master but I'm going to work on a branch and so when I got I going to merge it back in right so merge is something that that most developers use a lot but it was also really poorly written a huge percentage of github x' resources were dedicated solely to merge it was just not efficient at all so they needed to fix it but it's also something that a lot of us use a lot and so they can't do any degradation of service they can't make it so something that used to work doesn't work anymore that there's just no tolerance for that so what they did is they built version 2.0 or the library and there's a whole thing there on github engineering comm slash move - fast they build a different virtual library and they ran them in parallel now not you get that one and some people get this one every merge went through both what actually got merged was the old library what the new library did was capture a whole bunch of metrics and a bunch of tests and how much faster was it and did it work on this conflict and did it work on that conflict they actually found a whole bunch of edge cases in the original merged client they didn't know existed they tried to merge something with 512 merge conflicts talk about a long live branch but at any rate and it worked on the old client even though there was 512 conflicts the merge went through it turned out there was something in there than any base of 256 the old client wouldn't catch Irene's weird edge case but the point is that it's orders of magnitude faster you were all using it if you use github and when they were really done done then they just flipped them over and now your buddy got the new one and so again it's something that you have a business that's core to somebody somebody needs this to work every time they run it so not a high risk profile you can use to things like this I also want to encourage you when you're doing all these things to create useful feedback loops okay so a pipeline all of these things the point is to give the team the feedback they need to do their work this test failed this test didn't the business went up because you know etc that requires useful logging okay purchase failed is not useful it's you know the exception from the credit card handler you know it's cetera so yeah I saw one of the earlier talks talked about you know the right amount of logging etc that it has to be useful logging don't be afraid to run some of your tests in production if you're that kind of environment where you can do that if you're a web-based app or SAS or you know what have you after you do your deployment run some of your functional tests you know don't be careful creating data and you know that kind of stuff but there's a lot of things that you can test and production that you just you know dark launching that you just even the best staging environment you just really can't duplicate real world and so do that on par with the oops sorry time for either a new or an old clicker okay on par with logging also make sure your alerts are useful so I draw a strong differential between dev ops which I believe is culture I don't think you can do DevOps and continuous delivery which is the technical practices that you do I often say extreme programming is too agile as DevOps is to clean your starts controller its DevOps sorry part of the whole dead logs culture is you know you build it you run so now people that didn't used to get paged at 3:00 in the morning or getting paged at 3:00 in the morning if I get paged at 3:00 in the morning it says 5% of my production clusters down but performance is still where it's supposed to be that is a useless page I'm not going to go do anything about it right so we put logic in your alerting system also that says what are the things I actually need to wake somebody up for and one of the things that I don't useful night capital risk management right so I mentioned that one of the things about feature toggles is that it's tech debt and you got to go clean them up they didn't night capital is a trading firm that did a low number of high value trades so normally when you talk to a trading system if someone's trading on NASDAQ or the London Stock Exchange or whatever they're all about how many transactions per second can I do they didn't care okay their thing was we're doing a low number of transactions relatively speaking but they're a large value so they decided to build a function that would watch the stock that they were trading and say if it changes X percentage of time notify somebody so they can do I need to buy yourself make a decision based on the information the problem was six years ago they used the same toggle name for a feature that would automatically execute some orders they deployed the new software to seven out of their eight machines the eighth one started trading real stock 4 million execution 154 stocks 3 or 97 things in the time it took me to do this talk how much money you think they lost anybody been wondering what that number is the whole time in 45 minutes night capital lost four hundred and forty million dollars their market cap was four hundred go by other people came in and took the company over if they had an automated test that tested toggles never would have happened okay they had a risk profile that meant all their security tests all of their toggle tests everything needed to be run every single time and they didn't do it and a whole lot of people lost a whole lot of money one of the keynotes to say was about lugging vehicles in whatever money is bad somebody running into a tree because the car didn't turn that's worse okay code kills these days we have to do these things summarize if you don't have a system that you can go hit deploy literally right now not continuous delivery yet doesn't mean on a journey okay ci habits are absolutely fundamental you have to have them feature toggles are one way the other is etc
Info
Channel: GOTO Conferences
Views: 31,402
Rating: 4.8666668 out of 5
Keywords: GOTO, GOTOcon, GOTO Conference, GOTO (Software Conference), Videos for Developers, Computer Science, GOTOams, GOTO Amsterdam, Ken Mugrage, ThoughtWorks, Continuous Delivery, management strategies, deployment patterns, continuous delivery pipelines
Id: po712VIZZ7M
Channel Id: undefined
Length: 38min 39sec (2319 seconds)
Published: Fri Jun 23 2017
Related Videos
Note
Please note that this website is currently a work in progress! Lots of interesting data and statistics to come.