DevOps Demystified - An introduction to the ideas that are driving DevOps

are we good good are we good over there I'm pulling it i'm on a wheeeeel inch the answer is always yes do we need more beer yes always yeah Oh III hit the one okay all right thank you thank you I'm being pampered as a speaker they have special water for house it cover the bottle yes it's Fuji water which I feel a little bad about because apparently they're exploiting the few genes in order to get it we should thank del phix for the beer because this is leftover from ZFS day yes does anybody handle most days yeah it was good yeah speakers excluded all right I'm Ben ruffed I'm the director of systems engineering my other title is director cloud operations but I don't like that title because I fundamentally feed on the system engineer that's when I stick with a joint and we're not really going to talk about anything join related tonight if you have joint clawed questions or smarter less questions we're happy to entertain that and there are lots of people around it can't answer those but the topic of discussion tonight is DevOps DevOps is something I've been extremely interested in for a long time Patrick Dubois who's the father figure of DevOps and coined the term and I kind of found each other a long time ago and it's been a really interested this is sort of an advanced demos talking that this is sort of close for the perspective that you know what DevOps is and and I want to clarify some of the background where it came from so it's very few people involved in DevOps know where most of the ideas that they're discussing affecting if I want to shed that light I've been doing a pretty successful but that means that I need to kind of gauge where you all looks at so how many people know it develops you have to ask the embarrassing question if you don't raise your hand you look sad first to get people to raise um how many people would say that they're practicing demos okay that's good how many people have no idea what DevOps is really about you heard the term we're doing give up I think but how many people how many of you are here tonight think maybe you're doing DevOps what you're not sure so then we'll do the classical breakdown how many of you call yourselves predominantly developers okay and how many would call yourselves predominantly sysadmin my peoples okay so this is a slide that may or may not rhyme with you DevOps fundamentally it's about solving the problem where you you probably got this book but you should have gotten it with this book and you didn't you just got that one and ever since then you've been trying to figure out this is not about Titus anyone who's here who is a dev off be full of crap there is no one who is a demo it's not time son we do and in solving a fundamental problem which is that system administration isn't what it needs to be the way I describe this to max outside it's actually good analysis le just kind of described some people get surprised like I'm no longer seven I know a devil you're not there are lots of cooks right lots of people who become cooks and very good cooks and go to CIA become amazing culinary experts skills necessary to be a cook are important but they're their foundation right when you go to become a chef in a restaurant it all's different right you're not not a cook anymore the skill set required is a very different that's kind of what we're talking about fundamentally when we talk about DevOps it's just an administration I am still sysadmin and will always be so sad man until I die that's who I am what I do what I'm all about but the skills that I learned in this book did not prepare me to run cloud did not prepare me have thousands of customers with very different desires very different needs all running in concurrence this book would have really helped me but the only way you would have ever been given this textbook that you wouldn't took an MBA and if you can't take an MBA in your two years of training or whatever it is you would have taken probably one class and you may or may not have even a big Givens because most MBA programs today focus on policy rather than on the corner who's ever heard of operations management that's so much better than I'm used to operations management is is a study of operations and what is operations operations is doing what you do what your business does the class organization you sort of actually only have three branches of an organization you have executives which manage sales finance and operations so everything your business does whether it's making hot dogs or jumbo jets or showing movies that's your operations this is one thing that's happening in DevOps is people get a little too polarized with the idea of operations is just running the website in a classical MBA sense operations includes your development organization and it's part of the operations to give you a sense of kind of the ballpark we're in here let's now go back to the beginning so there are a lot of ideas that if all come together in what we call Devils all right general is this really kind of came out of strive towards agile operations that is lots of people and operations teams full system administrators looked at the agile transformation that really occurred in the last decade in thought we need to do that - let's do that - right scrum yeah let's use that putting everything in dip let's do that didn't really work so good for reasons that we'll cover later this movement of agile operations and trying to make it work and revision it over and over and over how do we become agile agile agile didn't work until Patrick Dubois who is one of the primary people thinking about how how to try and do this to the point that he actually rewrote the agile manifesto for operations you just place strikethrough bits and James terminology came up with this idea xx this term DevOps and all of a sudden it can gel and we had a word that codified who we were and what was most important was not necessarily what we were doing with the ideas that we had the things we were trying to accomplish what we wanted to do and we finally had this word that brought us all together to share these ideas and DevOps is very much a thing in flux ideas are changing all the time and it is growing and it has a very very long way to go my mission has been to try and bring a little bit of sanity in employing us back into the greater world of operations in the MBA sense of this it's this large amount of information that's out there that we can benefit from particularly things out of manufacturing great if you for anything about about DevOps at some point you see some analogy to manufacturing you know what the why are we talking about manufacturing once because they've had a lot of same problems that we have now they solve them a long time ago we should read it and Steel's much of it as possible okay so there are a number of different ways that people codify DevOps one of the most general overarching definition was made by having Jacobs who run to start an oxytocin chef which is the DevOps is a cultural and professional movement theory okay that goes back to what I was being the same before nobody is a dev off it's a cultural change in our profession okay that's it the other model came from John Willis and Damon Edwards who run the DevOps cafe podcast which is generally considered gospel if you haven't listened to that podcast go and listen to all of them right a lot of the ideas are there the great great thinkers and DevOps are all there and that's been a really important source of information for dell in general to listen to they had this idea called cams culture automation measurements and name service which is a good model they've kind of changed they've had a couple things into it this is the way I codify it is that it breaks down fundamentally into three things the collaboration of people convergence of process the creation and exploitation of tools okay and it's important is that we go top down not bottom up right a lot of people have heard about DevOps and want to hear tool tool tool tools tools right what is DevOps DevOps is puppet DevOps is chef I'm doing Jeff I'm therefore dev up we're going to get into that so in this talk we're going to go through these three areas we're going to talk about where they draw from what's important now important over arching concept there's one word you want to burn into your skull tattoo on your forehead so that you can see it every time you look the mirror is flow everything is about flow in a service organization we always want to be thinking about flow flow flow this is your possum these are the business people over here the create requirements what the market wants those requirements go into development develop creates a product they produce software they're producing it good right that's good is going to go into your operations team your operations team is going to deploy it into production right at that point the good becomes a service so services you're in customers right customers are going to consume it you have opinions they're going to buy it they're not going to buy it and all of those considerations are going to go back into your requirements your business people and you're going to go back and forth through this little of course this loop and reality is far more complexly right and it's not just your dev team and your office team as the name suggests right there's QA in there hopefully there's release management in there maybe you've got all kinds of marketing people and salespeople and executives and layers and this and that your split across multiple time points thank you very complicated fundamentally this is we're talking about right idea good service to market right flow we want to keep things flowing the the easier things that flow through there the more happy you are right when we have a shitty company we have bad bosses we have bad experience generally flows really bad right stuff gets bottled up here they're doing things that these guys don't want customers aren't sure about it there's you know and it gets all jacked up right flow we want to flow through okay so throughout this whole thing be thinking about flub encouraging flow so first the collaboration of people who's heard of simon Sinek did TED talk it was amazing you got it Ted Ted's website you look at the most popular talks out there I think last I checked he was number two which is odd because he was it more like TEDx something he wrote a book called starting with why and he's probably genius because he knows why is the most powerful question anyone can ever answer right you don't have to know what you're talking about what you're doing why are station why it's the smartest question you can ever ask kids are very good at this once a community what's a culture it's a group of people people with a common sense of values and beliefs when we're surrounded by people who believe what we believe say remarkable happens trust emergence it's an important subject when it comes to dental people Trust is something that happens among people who believe what we believe right Trust is not about whether somebody does what they say they're going to do Dutch means are reliable okay Trust is when somebody else believes like you believe chances are that the friends that you're around they believe what you believe right and these can happen all kinds of different strange circles right you never been in a you know in a in a foreign town and somebody says hey I'm from San Francisco you're like hey San Francisco there's a connection right there's this weird connection you know idea who they are if they weren't sailors just really um if you've ever gone to a foreign country you hear an accent on the train and you're like oh you're nagging you know where are you from I'm from Arizona em California and you're best friends right cuz you have something in common you believe something the same and it's strange how that works this is why people will gravitate around certain tribal instincts right you know a you like Ruby I like Ruby right we have an instant connection we instantly trust each other because we've demonstrated we believe something simple okay so how that works is really important because culture is really important but it's a very ambiguous sort of thing what its culture you can dissect that a whole lot of different ways so trying to think about culture and how ambiguous it is any like any of the definitions so when you don't like anything essence you go back to the Webster's okay and there are I think sixteen different definitions in Webster's for culture and this is the one that fits the best which is a culture as a set of shared attitudes values goals and practices that characterize this as an institutional okay what's interesting is is that these are tiered so we can rip these apart so here's our attributes shared values shared goals shared practices shared attitudes if you think about that you can actually start to do some extrapolations on that that helped us learn how to start hacking culture so how do you implement culture you determine what your core values are this is normally what your founder does right why there's normally a prolific founder at a company right and you may not like anything else about that company but that guy he believes what I believe we've got we've got something shares my core values right create goals which meet business needs that are consistent with your values right so all these goals what are we going to do what are we actually going to accomplish we're going to figure out what those are based on those values we're gonna have these goals so now we know what we believe we know we're what we want to do behind that we're going to create core practices which simplifies the process towards those goals how are we going to do that and consistent you defined sort of way so we're all working together and then you can use these attitudes tests all this with the attitudes everyone kind of gets together what's the actual attitude how do people feel are there are they anxious stressed out angry as Mystik those attitudes can tell you a lot about whether or not you actually share those core values whether or not you actually created sound goals that you'll believe in and that you actually have practices that you're following so then we can twist that we can alter our culture easy enough so we want to alter culture what do we want to do we want to listen carefully to those attitudes of our employees right everyone may say it's fine but if I huffing and puffing okay we can go ahead and do that for you it's a sign that Suns wrong determine with the value which values aren't being embraced right do you need to change the values do people not understand the values sounds a lot when you hire people over a long period of time the original stock get it because they probably spend a lot of time with founder people hired on later don't they hired him because their buddy was there or you know they were on Hacker News a lakh or something examine the current goals of the organization are the goals consistent have they changed what's going on there are they consistent with these and then adjust the practices so they're always driving towards that goal right nobody likes working for companies got lofty goals but never seemed to be moving towards them so there's some ways we can start the older culture Demming is very important that individual I won't belabor you with how awesome he is but he's insanely awesome and he has a list of seven deadly diseases of management the one that rings true with me and I think a lot in our industry it's number one lack of constancy of purpose as people don't even know either in business why are we fundamentally here why do we exist as an organization what the Frick is the point right because if you can't answer that it's really hard to get anyone behind you a little less get your customers engaged right and this is something that there's really been degraded by a lot of people over the last 15 years and as well you know if you're building product your customers just don't get it you need to pivot pivot you don't even know what you want to build and then you're going to just about we'll just build whatever they want and then everyone's gonna sue to stand behind that and get behind that and be engaged in that I think so simon Sinek came up with this idea of the Golden Circle there's a lot of interesting things that can be drawn with that they work in layers you start with why this is very emotional okay we move up how moving out what why we do it how we do it what we do and what Simon found is its Maps actually very much to the layers of our brain right why is something that's very very deep in our minds part of our mind that doesn't even have a language it has no way to rationally express ideas that's our emotional mind now here's our very logical money front it has facts and figures and can convince ourselves of certain things now there's all kinds of interesting things you can see about this company like Apple it's a company that's all about why the difference change the status quo you know it's not like yeah I get that you believe what I believe change it man make the best thing possible right we get very behind that you go with other companies like say it Dell we have 3.2 gigahertz Intel Xeon processors and that like okay they don't connect with you here they may have a what you need but you know they never really connects right companies that are really smart are the ones who focus on the why why they do it all this can fall behind it if you start out here with what you want to do have no idea why you try to build this backwards right figure out why you're doing it after you started doing it really really hard it's definitely hard to convey that message to your customers these are the companies we love this is why Apple produce a foam and there were lines around the block to get it right before we even knew what it was gotta have Apple did it it's got to be good right Dell and Acer and lots of other people make phones and TVs we're like what the hell just a part manufacturer why would hell would I buy a TV from Dell that's stupid I guarantee you've Apple put a flat-screen TV on the market tomorrow you like hell yeah I'm in line baby I'm getting me one of those right here are some of the 30 inches are pretty nice oh yeah they are um but unfortunately you're only going to get a very tiny small part of the market right okay so it's an important model that drives us in DevOps this is really important if you're just trying to if you're just reading the news about DevOps and you start implementing tools and things like this you're focusing on the what they're trying to do you're not focusing a lot and you're never going to get it never going to get it you're gonna be very frustrated in fact you've got to start with why and what's our fundamental lock that's your customer right I'm assuming all of us are in the web one way or another right this is our customer okay all of us sitting here in the room then we're all trying to impress and be cool to not our customers not our customers woman with a baby with somebody else screaming in the other room on the bed with a laptop that's probably your husband right she's the only other folks what is this anyone oh this looks like a con Edison or something yeah yeah this is the control room of a nuclear power plant it's very impressive I would love to tell you that my operations dashboard looks just like I'm not there yet but I'm getting there right Elijah we're getting it we're getting there kind of smells bad you could we're all gonna wear white lab coats down to it's gonna be part of joints philosophy why do we do this why don't we build it we built it so that she can vacuum our floor that's it right you don't build it for other nucular scientists you don't build it for anything more exciting that there should be a plug in somebody's house I can plug something like a vacuum cleaner into it back in therefore she needs a rule right and it's kind of depressing you're like we spent a lot of time and energy and science so that she can vacuum your floor but that's funny Moline what is it about right let's think bold the electoral devices in your house how many of them are any of the electoral devices in your house so impressive that it justified building that no but they built that so that you could use your toaster that's why it exists okay the sign we need to keep in mind when we build our operations when we build our products right we need to be thinking about the end end end user and not just our body local you know sick okay so that's people convergence of process and you know it's going to be exciting and when we start talking about process right I mean that seems more exciting the process you know who loves process people love process or generally people who are stuck in trapped under a really bad one alright Peter Drucker legendary management guru he he really is awesome honestly and facetiously read all of his books there maybe um efficiency is doing things writes effectiveness is doing the right things an interesting thing about this is what this tells us is that if we increase the effect of the efficiency of the wrong thing we're actually making things worse right the real core here is are you doing the right thing in the first place right we've all fought these battles right that you we've all had that moment where we stood up to the man cuz you're like yeah we're working really hard on it we're doing really good work for committing a lot of lines of code to the wrong damn thing nobody wants it it's not the right thing to do in the first place like now but we're going to make it the best thing in the world if I can usually the best piece of crap in the world right but we get efficiency some to get really hung up on right we're all kind of creatures of habit when you get running on Sun you just want to keep going going going via this perfectionist instinct we want to make it best that we can we continue to own it a lot of times we're not even doing the damn right thing in the first place this is really hopefully what our managers and what our founders and our board directors are ensuring that we're doing the right things in the first place so that we can make them more okay they're different thumbs now when it comes to DevOps there's a number of fields of study that influence where we're at with Devils it's not just like three guys on a blog who like had beers but it sort of was but there's a lot more that comes in behind it we start with agile right downloads obvi right agile school hassles awesome agile cause development to go off into a whole new sort of realm and sysadmin x' were just left there with nothing I'll get into what are nothing ones I think a little bit then there's other fields that we found like operations management this is generally what people think of with how to run a manufacturing plant right but it's more broad than that kind of talked about it before the areas of systems thinking and systems dynamics anybody go to MIT if you went to MIT we've spent a lot of time up the Theory of Constraints dr. Goldratt talk about that a little bit lene lene is the american name for the Toyota Production system which was influenced by Deming and couple of others what we talked about before we're going to it a little bit about what that's about and then this big monster this is the Deb's had agile we had IT Service Management right most of us only know of IT Service Management because we Lions it's just big nasty bureaucratic thing we all hate we didn't know what it is but it's awful so we'll talk about that and the most use formula IT Service Management ok ok so start with agile we've all read the agile manifesto they know if you have a tattooed to your back you get bonus points right but if you haven't seen it long while this is it now remember what it says is that things both things are important but the things on the left are more important to us than the things on the right we're going to prefer the things on the left over the things on the right so we're going to prefer individuals and interactions over processes and tools we're going to prefer working software over comprehensive documentation we're going to prefer customer collaboration over contract negotiation who doesn't look under and responding to change over following a play that's agile that's an agile manifesto everything we've done all these great things came out of these very four simple ideas right and I think we'd all agree agile has been huge right huge pletely transformed the industry very important very simple so six admins want to follow this right particularly a few were in operations teams to seven working with a bunch of developers who had now transformed everything from the waterfall project management styles into using agile you wanted to do it too right it was the and so we try to make a jalopy shion's work it didn't it was the idea that birth devops in the first place the term that came to replace an agile operations if you look back at velocity if you ever participate velocity and took those charts go back to the 2009 conference and look at the videos you won't hear anyone say DevOps you will hear people say agile operations look at the 2010 videos he point out saying DevOps and meaning the same thing scrum does not work for operations anybody tried scrum for ops how'd it go or yeah fun project repo you bastard everyone will be given a switch and we will we'll get you we could leave that poor man alone that the ideas are sound but they're incomplete and they tend to reinforce silos okay now why doesn't scrum work that's question why doesn't it work we know it doesn't work why doesn't it work Jean Kim everyone knows who Jean Kim is right hey yeah he's the man Jean Kim found it come to call trip wire wrote a very cool piece of software the 90s he then later wrote a number books most notably visible offs which turned ITIL and something for human beings and he's been working on a book that will be published I think in January called when IT fails that's the book when I teach failed it is a business model modeled on the goal written by full drat it is a novel Goldratt was going to write a book about manufacturing and operations management ideas but he knew that nobody actually reads books they skim them they look at the whole points and they kind of move on so I said screw you I will write a novel and you will have to read it and people did it was very good it's been hugely popular so Jean came is following that model in Rouen IT fails they're writing another book together with Patrick Dubois John Willis and orsa a couple others called the DevOps cookbook that will come out later IT revolution for us anyway in when IT fails there are a lot of interesting ideas but this this is the really important concept this is that there are four types of work that you do in an Operations team okay there are business projects so this is implementing a new service this is what your product managers are coming and telling you to do you're going to be doing business projects these things are going to service customers okay you have internal projects this is the stuff you wanted to do nobody else in the organization knows about and cares about generally but it seemed really good if you're an Adele development organization that might be something like you know building the Jenkins built server or something like that right it's not actually contributing but it's an internal project it would help you this might be you know implementing a new configuration management system switching over to puppet or chef or something right this is the project we generally want to do that nobody else gets then you've got planned changes right these are generally tickets right you got to take a cue that's thing you look at every morning as you've got X number of tickets in your queue these are planned changes if you have a full ite that ITIL system then you'll actually these would be the things you take to a board you know and then you've got onion client changes this is breakfast this is somebody coming on to the IRC channel or the jabber channel or whatever communication maybe there's a phone calling you going through the damage down dude it's slow you know it's something broken soon right it never makes it into you're like hey dude you need to open a ticket for that no it needs to be done right now right these are the four types of work that we do now why isn't scrum work scrum has a idea of a backlog right take all the things that need to be done and you put them onto the backlog then the scrum master prioritizes the backlog figures out which are the things that need to be done and puts them into a time boxed window in order to be done right this assumes that all your work is essentially the same the problem is not looking for different types of work right if you don't I understand you have four types of work you're trying to figure out why certain things don't happen because what happens when things on your backlog these things fast-track themselves out right these things that have to go to the front of the line all the time and there's always enough of them that they're always going to the front of line which means none of the rest of them are happening right you're hoping to get to these these are generally what you have a metric for the number of tickets you've done or whatever listen to make make you feel better really that's their thing it's like close to all right it never actually happens because you're doing with these there's no those things you want to do you never really get around to all right business projects and things you'll probably actually supposed to be getting done for the business right it works doesn't it this jives with what we do doesn't it when I read this book and saw this I was like holy that makes so many things make sense I always thought him as one thing I knew they weren't the same thing but I didn't know how to break them down this works and it helps us understand that's my scrum that's why scrum never worked never worked if you had four separate backlogs or something Kanban can work right but it doesn't break break these down so this is really really important when you understand this I find that it helps me a lot especially when I'm getting frustrated why certain things get pushed push push all the time right the things I want to do or these internal projects those are the things will make me feel better because you generally know if I can get my internal projects done I'll have less unplanned change right I can execute planning changes faster right but I can never get to those and somebody's always screaming at me about one of these right so this has a lot of very important kind of ramifications scrum being one of them okay so the book will be out in January I highly recommend you get it you should read it more importantly you should buy a copy and give it to your boss the real intention of this book was not to tell you what to do it's to give it to your boss so your boss can can come back to you and be like I'm so sorry your life sucks so bad and I never really understood why right that's what this book is for and hopefully will be a big success a lot of people will finally understand why I team is crushing their companies and why we all look like incompetent buffoons all the time right it's frustrating right what can't you get it done faster because you set me up to fail I don't know what you're talking about you have all the resources you need what am I not getting you done okay so the field of operations management this is line I stumbled upon I've been studying Deming I've been studying Goldratt and saying all sorts of things in life one day happened to cross and the strangest of all possible ways an operations management book my operations made a texture and I went that couldn't possibly have anything to do with what I did and I flipped through it and my mind was blown apparently people have been operating things for hundreds of years and they thought about it and they go they've come up with theories and and then ring them down believe it or not they've written them down um this is saying you only get to in an MBA program though it's a traditional study of management this is where you beacons maintenance where are you going to come with a date you know did local manager right people we hate no don't lint around it just be honest until you become one and then you're like oh yeah this is my theory about managers manager nobody ever wants to be a manager right everyone I'm never going to be a manager man I'm going to be a hardcore coder at 45-65 I'm going to code the rest of my life and I find that the way that you become a middle manager is at some point you're so tired of all your manager sucking you're like screw it I can do it myself and then you suck at it and then you are younger people and they do it to you and you're like oh that's why everyone hated me when I was 25 anyway generation will rule over some of the areas of study inside operations management is like scheduling mmm project management that could be useful process measurement Wow that would been handy to learn quality scheduling really are so right waiting line all these sorts of things what if you'd learn this as part of the CEA's program what if it not helpful right this is one of those things that I really think that CS management CS education is broken is it is one of things it has to do is it has to include this field of study and it should be like an accounting cause it can be the thing that you hate oh you do the homework and you forget about it but ten years later when you're trying to schedule something and you go hey wait there's a book on this I should probably pull it book back out again it would be really handy so previously focused predominantly on manufacturing today it's all about service and what is operations operations with service it also includes some things you may have heard of TOC and Theory of Constraints we're talking about a minute lean Toyota Production system things like Six Sigma that one stands out everyone's like kha'zix it's not black belts and things like that okay that's operations management it's important I make a big deal about it because if somebody told me about this ten years ago I'd be a much smarter dude than I am today all right systems thank you Systems thinking is a field of study it's something that's very important particularly of schools like MIT now is this something you what its system is all about a system as a whole that cannot be divided into interdependent parts okay and if you do the essential properties of system are those which none of its parts have okay you are a system right you cannot be divided into your parts right farcical your hand - no it's not detention 21 your hand can write right chop it off put it on the table and see what it does problem do much right the system only works with all of its components right and what's important is that the system the qualities the inherent qualities of a system are not the sum of its parts with some of its behaviors and a product of their interactions right what makes you is not your individual parts that's how all of your parts work together right a Ferrari is a fantastic car it is not a fantastic car because it has a great engine nor that it has great tires nor that it has a great leather seat but you put all those things together in the sum of their behaviour is what produces a really awesome system right we've all heard of silo mentality right most people they think of DevOps the first thing comes your - I notice right silo Devin the silo alums right if we don't use systems thinking these are two independent things what do we do we reinforce efficiency within each of those silos right one of the most efficient things that you could do for software development is to never give it to customers right these beats things up tremendously not real good when I talk about flow this is what we're talking about systems thinking you want to think about your entire organization which is why it's also not just about dev and ops it's about the entire organization you just optimize those two groups together you get another foster right we have to think of the entire system right here's a common kind of mental exercise that's used in systems thinking courses and things like that which is to imagine that your company just burned down everything you've got is gone this is all your backups are gone everything's closed okay and you have to completely start over from scratch 100% all you got is the people on what would you do differently how would you change it how would you build it better than it was before okay now here's the thing if you cannot answer that question you're screwed because if there's no constraints everything's gone you have to rebuild it anyway if there are no constraints and you don't know what to do how in the earth can you change anything with all the constraints of daily life right a lot of us have a hard time we like to micromanage and kind of push out little efficiencies all over the place right and say well what would you change in your company if you could change anything you want to we've all got lists 7,000 feet right um let's say it's all gone let's start over what are you going to do that if you have an answer and you don't that gets you into the realm of systems thinking holistically all right systems dynamic is a branch of systems thinking it's very big at MIT in couple of folks and it is really about the mathematical study of interactions within a system in this case it looks at all interactions is feedback loops cause-and-effect relationships all approximate organization and what it wants to do is it wants to mathematically work backwards from events things that happen turn those into patterns we see patterns emerge between certain interactions and then to look at how the entire system itself bangs dr. Jay Forrester was the one who really kicked us off and these are the kinds of people you can pay huge amounts of money to to come in and mathematically draw your entire organization how everything works and tell you why it's broken not really important to do all in systems dynamics except to know it's there if you guys have seen any of the recent work by Jim Kim and others so if that's going to be coming forth in the DevOps cookbook you'll notice they'll talk about the three ways the three ways are systems thinking systems dynamics that's really come from alright Theory of Constraints who's heard of toz ma ok Theory of Constraints was a was a theory put forth by dr. Goldratt 1982 or three on a book called the goal and if you watch or read much about DevOps at some point somebody will tell you you have to read the goal it changed my life uhm the Theory of Constraints that's laid out in the book and the book school because it is a novel walks you through and you sort of see how this thing thing comes about the book is all about a guy who's in a situation where his factory is got to be shut down in like three months unless you massively turns everything around and the guy goes crazy trying to figure how to rebuild it and as all these epiphanies it's a good book it's actually a quick read and you'll see yourself in it you're like yeah how you know exactly how that is but in theory constraints we have kind of five focusing steps we identify a systems construct okay we decide how to exploit the systems constraint once we've done that we try to subordinate everything into the above decision then we elevate the systems constraint and then finally as a result try not to allow inertia to set in status quo went over everyone's head right including money fundamental concept of TOC in those five steps really comes down to what's known as drum buffer room okay so here we have an assembly line exciting right I knew you're like totally I'm going to do I'm going to learn about assembly lines you are so here we have a three workstation okay the number of units that can be produced here is five and two and eight okay where's my constraint right there Bobby's right that's my constraint all right what are you right we all have constraints on our organizations a lot of this times this will happen in things like provisioning if you a lot of provisioning certain steps are really fast certain steps are really slow something requires a human being that's your constraint what do we do with just you constraints all over the plugs alright so we use drum buffer rope we use the focusing steps before see the first thing we need to do is identify the system skin strength we did that okay we know which ones are constrained now we need to decide how to exploit the systems constraint right well the important thing here is that we know that if we want to have efficient flow what do we need to do just Jam things into the first step as fast as possible no because what's going to happen a bottle up here right and this guy can never possibly do all the work that it needs to do because it's constrained right so first thing we need to do is agree that this is our bottleneck and everything needs to move at that speed if we only put 2 2 2 2 2 em everything's going to flow right through just nice and right ok so that becomes the drum okay bum-bum-bum - - - - everything dances right through the system okay now we need to subordinate everything this means that everything else cannot process more than two units at a time right everything else has to coincide with that and we need to see if we can elevate the systems can strength it's a fancy way of saying can you get another one of these machines so you can do four maybe you can you can get another one now that's four now we can move forward to time get three then it's not a constraint at all okay maybe you do maybe again and if you can't you can't let inertia set in because what happens particularly if you have elevated this constraint and it's no longer the bottleneck now it can process six units where's the bottle where's the constraint it moved one somewhere else this is one of the really important things that the big bull rat was really adamant about a lot of times people look for the problem constraint in your organization it works super hard to fix the kid to fix the constraint and they fix it they're like hurry we fixed the problem there's a new problem dude there's a new one it just moved now right so in the system drum buffer row so we set the tempo dum-dum-dum - everything marches through we create a pool based system with a rope we drag everything through the system right we pull things should only go into the system when it's going to come off the lining goes somewhere like - customer otherwise you end up producing things that never happen so what pulls work through the system should be orders right no orders don't do any work it's no point of doing work otherwise you just build inventory inventory is placed right and this idea of buffers this is an important thing here if we want to the most important thing in this system is to ensure that this thing is always working always always always always want the system can only move at its speed so this thing can never stop processing or we're screwed right what happens if this machine breaks down we're screwed everything just stopped right so what do we do how do we come back then we use buffers we build up a catch of work ahead of this workstation say eight units so that even if this guy stops this guy continued to process that's above okay and so this guy does work when when there's not enough work in the buffer right one of the important things that fell out of this and this is why it shook the entire world of manufacturing is this system tells us that if you've got a guy spanning here you should not be working as fast as he can because if he does he's screwing the entire system he's causing all kinds of problems like whoa you're saying the guy should stop working yes you should stop working dude he makes $14 an hour he's a work his ass off no he should only make enough parts to make sure that there's an adequate buffer for this and then he should stop this was a revolutionary friggin concept it completely blew the manufacturing world apart because they couldn't understand why would you pay someone to stand there and do nothing because they weren't employing systems thinking all they were thinking about is each worker these work as hard as they can for whatever hours they have and they were killing themselves u.s. card industry did this and this is incidentally by the Japanese kicked our ass they figured this stuff out because one of our guys Deming was mentioned earlier to Japan taught them all the tricks no one in America would listening to him they changed everything in particular Toyota which became known as the tornado production system which in 1980s when we were suffering because the Japanese were kicking our asses we went how the hell is Japan doing this they've got better quality whenever problems happen they bounce back faster how are they doing this and they found out oh one of our guys when taught them at it is we never listen to it and suddenly Demmings consulting business went through the roof so lean so in DevOps we try to learn a lot from the past since we're a lot of good ideas we try to draw on the principles of the Toyota Production system otherwise known as TPS the Toyota Production system was created by ona at Toyota but it did draw on the ideas of Deming Drucker Toyota mr. Toyota the founder of the company who noticed that D was changed to a te later Shingo sure forward except you want to learn about who all as people are watching my horrific lis boring and very popular strangely keynote at Lisa lean really focuses and that is to say the Toyota Production system really focuses on eliminating waste waste others in japanese muda trying to remove buddha and trying to create a pull based system so the way that the Japanese really came to kick our butts in the United States some of you probably some of you remember some of you don't and it's the oil prices nine to ten three right we produce cars and sign called planned obsolescence created by an MIT graduate believe it or not that she have in the 30s and so the oil crisis came people bought fewer cars and all these cars just spilled off assembly lines thanks to the mass production system created by Ford and all this inventory sat around in the next year guess what nobody wanted to buy the 1973 model Frank and chaos ensued the Japanese we're using a pull based system rank people will order cars it would pull things through the manufacturing lines in Japan when people stop ordering guess what happened so I'm making cars the next year who do you think had a better quarter right voted right and it all started to trickle out sadly lean concepts are a lot of concepts in lean that are really important one skies end continuous improvement right we have been constantly improving all the time this is what most of us refer to as the scientific method Kanban as society if you guys have heard of that Kanban in our circles you've heard of a perversion of Kanban Kanban is the just-in-time system that enables the pull signaling it's actually a card based system that is that when you are sitting on an assembly line you have a bunch of parts to help you make a brake rotor okay people the thing that the box of parts you assemble it and you pull up the card that's in the box and you put it in a stack and you make it and then you put that aside you pick up another one build the next one you put the card in the stack what happens boy comes around somebody comes around with a cart picks up all the stacks of cards for all the people take those back what are those cards there are orders for more kits so they go back to the storeroom they pick up all the kits and they bring them up to the assembly line and stack them right well when they do that all those boxes have guards in them they take those cards they go into a stack then somebody goes and orders all the parts and it goes all the way down the system so Toyota doesn't order a damn thing until it's actually used everything is pulled through and what's pulling it through orders orders are order dragging the assembly line the assembly line is driving the entire supply chain right that's what Kanban is all about it's really important and the interesting thing is you know where the idea for this whole system came from when a bunch of Japanese executives went to Chicago into a supermarket and notice that there isn't a giant box whole bread there's a couple loaves of bread on a rack and what happens you come in you take it off as soon as you take it off and walk down the aisle what happens somebody comes out it's another loaf in its spot anyway let's do that inspiration comes from the strangest places now other ideas judoka autonomous automation automation with a human touch believe that I won't tell you where that came from poka-yoke okay you're I am mistake proofing this is the idea of building things that can only be done the right way this is way of ensuring quality so you just design something such that only two parts can ever be put together in a certain way if you don't it doesn't fit it doesn't work that's the way you create single zero defects just don't make things that can be screwed up this is also things like in milling platforms and stuff they would have a little tray machine the part needs to slide it over to the side and it would if it fit it was done if it didn't fit it didn't work um this is used in manufacturing today they'll have all these blocks you'll see all around and you take the part you melt down and you slide it through and if it has two different gauges too big too small 5s sorting simplifying sweeping standardizing sustaining right a way of doing a kind of maintaining things 5y signs close to me and D traces heart root cause analysis anyone ever heard of 5y yeah it's really simple right the idea is to get to root cause analysis you have 5y times 5y 5 times so you say like you know we're out of bread why cuz I didn't feel like getting it fly because I was really busy at work why because I got this project that's coming why because people are stupid other the point is to not go to layers they break it shrine goes far down this is definitely fast hard rule and it should be fun with rigor but it is it is a good idea with DTrace this is a good idea right is DTrace is the tool that allows us what we do systems analysis to dig lot why does that happen ok well why is that why is that we just dig deeper deeper deeper down layers until we find science useful without stopping just because we're like oh we own the sole reason number three now keep going this idea we mention this before mood of waste removing all non-value add action this includes there's all types of different muda one of them is transportation moving you should never have people walking with things in a factory right put the machine right there slide no walking all kinds of you think about it in our organizations there's all kinds of muda the most common type of muda is is waiting waiting for something right somebody produces something the next person needs to take it this might be your development team releases a piece of software that your operations team needs to move it out so the customer is on the website and it just sits there it's going nowhere during that time the clock is ticking and no value is being added to it right in the in the TPS model every time anyone it it should be constantly be given a value added right because in a classical sense something goes into a bra material goes in one side of the assembly line value added mature up something that the whole time between you're adding value sounds forms okay a lot of stuff right you didn't expect that right so so creation exploitations of tools this is the one everyone loves so prepare to be excited sorry mark Guiness out there oh you can take our range you got it for you yeah but there's no you're drinking out of the language up okay these are the common DevOps tools right II did I miss anything what'd I miss cool sir Cronus is a new one I I use lockers right these are the DevOps tools man we need a chef puffer cfengine we've diagnosed ganglia we're using graphite steps be some moon in Splunk in it or I'm super cool and I'm log stashing into gray lon - oh yeah right DevOps tools right does everyone hurt my mind I guess Ram have you ever heard of the monitoring sucks movement hashtag monitoring sucks gender that one anybody agree with it monitoring sex subjects and it looks like ask people when they think monitoring sucks what monitoring tool they use nine times out of nine ten times out of ten you the Nagios figure it out nah no sucks I use Evans so I'm not a shield for zabbix but love yourself use them there are lots of good monitoring tools I'm not trying to say that Nagios sucks and you should use one other thing just not justice these are all great tools right they're all great tools but are they DevOps tools they're definitely hip they're tools okay are these demos tools you're missin SSH sssh is not a tool in the same way that air is not a resource you know what I mean it's like air is only like this really important to you you don't have it the same way that like if you go to a university and everything's like telnet Kerberos you're like dear Lord is giving SSH otherwise you don't think about is ubiquitous are these DevOps tools no what's the difference that was cool suburb 40 years old I if you don't think longer is cooler than audios I will arm-wrestle you to the hook the difference what makes something a DevOps tool is not who made it or how cool it is or how many times is mentioned in velocity it is how you use it notice that post-it notes is on there one of the most DevOps friendly tools there is go little friendly post-it note right where's the git repository here solicit or password to use anybody want to get a version yeah you know I don't know I'm doing the keyboard the important thing is not what tool you use it's how you use it it's how you use it right this is of course the best DevOps tool ever right your tool should be bringing people together enabling flow remember talking about flow flow flow flow okay beer tends to let people flow all right so what is it if ups with any tool that brings remember the three layers aids and the convergence of process and AIDS the collaboration of people and must support flow must support flow and what's so beer does beer aid in the convergence of process does it aid in the collaboration of people now put question marks here not yes yes because in some of your companies having everyone go out for a beer might not actually in in process or collaboration of people it may not right you may just cause a giant fest you know everyone's like I know I'm gonna smooth it over that guy we never seen eye to eye I got taken out to dinner and we'll bond and you know we'll finally get over it and then you're like nice it doesn't always work right your baby does right um maybe just going you know if you're having if you're not working well as a dev team and the operations team and QA team aren't working well together a lot of times this also happens we're like deaf people tend to have a lower opinion of office people obstacle didn't have a lower opinion of QA people you know what I mean there's all these things where people feel under other people just going to their damn meeting just listening don't talk just listen in there beans might be like what I didn't think you even cared what we do I do um that might be enough now metrics right I don't do memes so I don't have the measure all the things slide but metrics the graphs like this aid the convergence of process or the collaboration of people probably the most important thing is just show it to anybody if I create the world's greatest grasp of like old like user latency to our to our front-end website right this is this is customer experience if you don't show that to everyone else the organization it's hit really all that helpful no it's really fascinating for you but everyone else is screwed you got to unlock all those metrics so just graphy all the things may or may not be a dev op tool if you give that graft everyone you put in the dashboard says he's having screen pull you put some labels on it some P it's clear what people know and all of a sudden the ladies sing at the front desk at the reception desk knows how the business is doing in your CEO knows how the business is doing and you're devs know that's a tool that's bringing people together that's you taking your skills and doing some amazing things and using a tool to do it but if it's just sitting there on some back-end dashboard or just in sitting ganglia doing nothing it's not helping anybody okay and you can apply that same model to any of those tools you know you know even with not yet sir signing you or whatever right if you give your developers access to that they know in the sites down that's a dub tool if it's just you getting pages I don't know up tool so remember it's about customers it's about flow and an important thing is about pride of workmanship all of us had pretty much devoted our lives to this right it's not a lot of flakes in this business right you don't get in this because the money's good right there's too much frustration barriers too high you should actually enjoy what you do most of us would do this we're getting paid for in fact most of us didn't get paid far as one point we're doing it and we eventually found out somebody who pays money to them you should enjoy what you're doing and you should have pride in your work and you shouldn't have to struggle to do that most importantly there's the environment should allow you to have pride in your work mission if you're not having fun you're doing it all wrong thank you questions I guess they're all sort of feeding ideas but like there's a lot better but you mentioned sanity you like better like what why what is it you said it I didn't catch my look why is it it's not why I'm yeah does anybody else care about this I can tell you why um are you still videotaping yes - open up - I won't demonstrate um there's a couple of things um so now I guess there's big problems with configuration right configuration doesn't and it's fine when it's small right even when it's smaller like a lot of configuration but it's manageable over time it just balloons completely out of control a lot of people are doing the figuration management to manage their organs Nagios configuration it's just out of control to the point that if you ever lost the damn thing and had to rebuild it oh no right you have to definitely build a framework to rebuild the configuration based on of your notes you're screwed it's crazy I chose to use zabbix I chose Avex for a couple of reasons one it's distributed system that uses proxies so I have data centers all across the world and at one point I had a separate monitoring system in each one of those data centers and where's where's stuff having problems the one that you're not looking at right you'd have to go and look at literally you know all these different networks so I was one from I needed a tool it brought all that together into one I need to see the entire state of my business on one screen another one is adding those in August is impossible it is real right at scale at scale okay I'll give you a real pain now there so they can become a very fragile process zabbix has auto registration so node fires up so the agent starts and it contacts the server it's actually a contact of proxy the proxy system service services I've never seen you before it's oh it registers it automatically and it's now added to the system and it says there are rules you can assign to in actions you can assign to it and it says oh you came off this proxy okay that means you're an app data center I want you to associate these temples right and now all the checks that I want to sign to it or assign to it and it's now collecting all the data and it's all the topics another cool thing about zabbix is that every metric that you that you pull is inherently graphed this is one of those things where you're not going to build a dashboard based on these things right this is not a replacement for graphite or Rd tool or whatever you're using but what's handy about is there certain metrics that you don't care about the graph until you really wish you had it a good example would be temperatures temperatures on your on your planer right don't normally care about seeing the up-and-down of that until there is a massive spike across 50 different machines tapping our data center phones and also you're like under the temperature start pricing right having that graph and be like pots of six hours ago and start it up really hated how many needed there are a couple other things like it is zabbix I like because it is agent based the server never pulls the ADA pulls the server or the the exam Aksarben ever actually pulls the asian running on the machine of whether it's the other way around so the agent when it checks in is told here's what you're expected to do now there's all that stuff and sends it in regular intervals up string which means I can scale significantly without overwhelming to give a master because it's up to many machines for Polar's the rules are easier to write extension is really simple as this idea of user parameters where you could just give it like a key a key and a value that I use is just you know one line shell script shell line to go and pull symmetric so can add things super trivially and then I manage the whole thing through chef so a new node comes up I chef it it fires up shaft installs the agent and checks in everything's done as she's super super easy like says Amex is not the only system there are lots of other ones even a lot of the the you know was it in Cigna and a couple others that have tried to make it for the deficits of novios I think our questionably doing better jobs I'm just adding graphing to like some of them have done is not enough in and of itself but there are options is the thing I think a lot of people feel like they're stuck that not us is the only way right which is funny because Nadia's came around to unseat people who thought the HP openview was the only web right like that I can use Nagios or net sync before then remember Nessa yeah I don't have the great beer production so who monitors your sevens I um so I actually use veneers chair monitoring so that's another question so I use Pingdom so zabbix monitors my entire production environment and then I use Pingdom to actually back channel a lot of these other things give me redundant checks what's one of the other things I like doing in zabbix is actually monitor just about everything and Elijah has been huge elbows we monitor just about everything like two or three different ways so that we have come a Deadman checks and all these sorts of things so that we can catch weird scenarios of course one of the important things in the system like that where the agents are doing all the work is chugging it up is if something stops working how do you know that it stopped working right you never asked it how it was doing in the first place so we do that through we have certain checks that like if a note has not reported data in ten minutes we throw an alarm which tends to be really really helpful in odd situations where systems hang where it's up it's pinging you may even be able to log into the damn thing but processes stops spawning or something like they used some kind of odd condition all sudden agents not working it's not sending data up and guess what we just caught it well those are the situations like if the load average what's 10,000 how we've caught it that the ones where everything just stops those are really hard to catch and those are ones that we catch very easily just by the nature of our system so now I will say to be fair because I'm a sysadmin I hit everything to the one downside zabbix if you are going to implement it is that all the ways it does all these beautiful wonderful things is by storing everything in a database at some point that's up yeah database right and obviously of database scaling issues so that's something you need to deal with great way to deal with that kkona kkona makes it everything better we before we were running percona mic was terrible the other thing is is that lots of tools not yours and others have the ability to do escalations and things like that don't believe in that these pager to do so none of our tools do an escalation anymore everything just goes the patreon handle cents so Pingdom zabbix and pay to do this what why something like Apple that exactly does that well I mean this is the trick um this is this is a lot of people done a lot of great work on employment high companies change right what why company starts out and everyone's very passionate about overtime it ceases to be like that a lot of times it's because the founder becomes further and further away so why did why did I mean apples to apples our company that was founded had wild success and then tremendous failure and tremendous success and where is it going right now right I think we'd already everyone's gone you know cook comes out there and says I'm sorry that our navigation app isn't so great right like whoa there dog right actually within the pad those awning is awful bad good no I mean it anyway interesting example um this is this is what happens when the vision gets too far away and that's what happened to Apple it sit down and a lot of other companies right they have these tremendous ideas they had very passionate founders who were all about what they were trying to do why they were trying to do why it was passionately important them what happens is either that person leaves in the case of Apple right Steve leaves somebody comes in he doesn't care about the light he just cares about the one out it keeps making products so pretty good products I love utin those look fantastic product but right the passion wasn't there to sustain it because they lost their fundamental core reason for existing there why and a company slid until Jobs came back it was important this is the job this was always out there so we were as cut consumers we were connecting direct with Steve right there are a couple other people in the band around him right but ultimately was about Steve and we trusted Steve right he believed what we believed we bought into that vision we bought into that life right the real challenge for Apple now in the future is going to be can we still connect with Apple with the brand without him being there so I think a lot of us were connecting more with Steve Jobs than we were with Apple and that's going to play out in the next couple of years Dell is a great example Michael Dell's fantastic guy but he got pushed back back back back back back back right also he wasn't so much the fork facing person a company now my juices were where you get layers of customers right there are certain people care very much about Dell and they focus very they know where to look right there they're focusing on Michael Dell screw all the peons around him focusing on Michael Dell right and those people are gonna have a different opinion of Dell than people like perhaps myself who are just not that we don't believe so much we're not that engaged and we just sort of passively take whatever we get and his message is lost this is what happens in companies get really big and that's why it's really important to keep out there and to keep that vision with your customers questions on the street oh look I realized I was screamed yes I thought you were recording for youtubers I'm on I'm here with a camera what do you think is happening never questioned your girl ever somebody says her night home says if I haven't read any of the books he mentions which would he recommend first if you were to buy only one book I would recommend recommend buying and operations management textbook just to have it skim through it be impressed put it on your shelf forget about and then when you need it go back to into or first minute and if you want to get into all the other things get into all the other things there are lots of books to read I bought them all and read them all crazy but still yeah if you're going to buy like one by text there is also a hope that you will make the slides available so people can go buy things are already available actually gave this I actually wrote this talk for LA DevOps that I presented like three months ago so slides already I won't be possible and any other on the street though someone just says thanks Ben you're awesome yeah shout out to the stream people because I'm normally in my garage smoking watching the string because I can smoke at home when I watch on the stream rather mean impersonal yes this is all just part of the campaign to stop you smoking no pictures of you The Devil's ball right yes I have some shiny shiny other times I want to shoot him like that was my impression is I mean get yeah yeah yeah he is a parody he here's a very um it it is satyr so yeah I can do all things I measure all the things yeah hidden you first menu so you talk about a lot of monitoring codes what is what you use for been working this monitoring into the action evil instructions for somebody to resolve their quarterly about I only partially got here some of my modern ranges so what are you looking of the intuition that you are been wondering to actually label instructions to learn fix the problem a combination of two things one is honing exam explosive triggers it allows you a lot of flexibility and cracking that right because yeah one of the things that's really important in any response decent human being it needs to evoke an emotional response right you know a stack trace in your email just irritates you right like your your emotional reaction is going right you want something that that like makes you feel like it's broken and in folks this sort of like I just fix it I must I must go and make it ice right that's a difficult thing to do so so working on on the alerts that your system sends and making sure that both that they're actionable and that they're also very direct and you know something is down you know it's not you know some odd metric the other is to get very good at how you categorize them like a lot of like in my alerts are categorized I have disaster those are what go to page or duty and I've got other ones like hi you know all these things that are interesting to know I need to know about but I don't need to be woken up in the middle of the night but but I still want my radar the thing I need to go along with that is SOPs right so whoever's in your operations team or responding to pages needs to have access to a repository of procedures and those procedures need to be very very clear right they need to be step I am driving Elijah crazy it's got to be step one step two step three step four it's gotta have prerequisites right because a procedure that was written for somebody who's done the procedure 12 times before he's completely unuseful because the person who's never done it before which may be me and Mason um you know it's like logging in to the web server where is it where's the web server what's the address of the web server I don't know crap right it's like the procedure is now useless because you don't even know how to perform step so just being very concise I wrote a blog post about this while about co-write writing better SFPs I'm already very very clear procedures is very important making sure that everyone in the company especially anybody on phone response knows where they are can find them but it does take a lot of diligence to write them all do you do you see them being automated procedures being automated whoo you're taking me into a place where I can go for a long I was talking to Mark Burgess actually a cfengine Fame last night over dinner one of the one of the subjects that it's extremely near dear to my heart is knowledge management and how we store knowledge right all of us spend all of them all day long we're knowledge workers it's what we're called we're knowledgeable in what's called a third way and what do we do all day long we learn stuff for living that's we get paid to do right as employers we pay our employees to learn and do stuff right where does all I knowledge go it is a big bucket in your head right it just generally sees what if you could take all that knowledge that you're producing and store it so that the entire company could benefit it would be amazing how do you do that no freaking idea right the the best innovation that we came up with 10 years ago was the week and the only innovation to the wiki since the invention of the wiki was the hashtag so I'm dead serious in knowledge management academic circles that is the greatest innovation is the impact right um it's it's concerning so do automate it mark Burgess has very certain ideas about cfengine if you hurl it to cfengine is very complicated so this but it's all about embedding knowledge within your configuration and extreme inextricably linking them together so that you form a semantic knowledge web at the same time you configure all of your machines and devices to do what they need to do and it's it's combined and it's beautiful and most people don't get it including myself including myself I'm still coming around so if you're really interested in that sort of thinking very seriously look at cfengine cfengine 3:8 cfengine org they have a free trial and you'll be blown away by the source with it but but it doesn't require very in-depth if you're just the kind of person like most of us who are just like damaged and create 15 users on every single on my boxes it's not gonna happen I think you've every day my battery time go back to you on how you so there's a lot of stuff about library science by the French library she talked about out of these or knowledge I'm creepy but my question is um going are technology like inevitably like within your company you're going to have one piece knowledge all of these PDF is like hang out together they go out to lunch together they talk about stuff at lunch together and they're coming up with these ideas in the Alps people they all sit in the basement you know and they're doing their op C stuff and the dev scene or do is their dev stuff you know and so like but across the board like you need everybody to have this operational mindset and because you're delivering the product and so you've got you've got opinions like from Theo the sauce mangoes who said like f geniuses have an operational mindset and then you've got other opinions where it's like well why don't we just like take you know these Ops guys and embed them and death groups so you have like this embedded ops person but it's still like he's the specialist and he does all the op stuff in this dev team both are right look at it all depends on that on cynic you know are we doing the what or doing the why because if you're just saying like hey man we shouldn't forget it kumbaya I embed myself in your team why because it will improve our efficiency and we'll get along better right not going to work you're like how are you get the hell out of here yeah part of that is cultural right it's like we talked about before people believe what you believe right ops people believe very different things than deaf people do we have different fundamental I can get into you wanted that's a little arcane propositions and things so this is why I said it's really important to focus on that customer that woman if we're all thinking about her our end customer then all the idea why do you want to be in my dev team because of her cuz I want to make her experience better I want to make sure that when she hits the site it's fast its responsive I don't ever want to lose in transaction because she's sitting there it's got better things to do when she goes to clicked order whatever right whether it's a bank site or whatever we want her experience to be the best attending so she can get done what she wants to do and move home yeah right if that's your focus then all of it makes sense yeah come and be part of our group how do we improve our latency how can we make sure that if we do have an error in our code and it shows up on the website how we do a graceful way that redirects the customer so they get to where they need to go right all of us have examples of websites that we care a lot about right and if something breaks we're going to be very interested like github might be a good example right it throws an air we really hook back get out through an error right if we go to something like Eventbrite or take master to order a ticket and it doesn't work push it piece of crap I need my ticket I want to go see a concert I it's gonna sell out give me the ticket right there are such we care about using and carat size we have an emotional and we need to think of it that person who does not give a crap about our well-being as a company they just need a service and they want to do it I want to do efficiently we want to enable that and so if you have that mindset everything else will follow through what happens is we get distance away from that and the user is just a line analog you know and what gets a screwed because we don't have enough quarterly revenue at the end of the year and we're all not getting Christmas bonuses or when I guess I've strapped it away everything goes downhill and you don't care so much right we all like to be heroes right when we're working and striving we're doing the best that we can because because that end costumer we feel a lot better you go home at the end of the day you made somebody happy right or in the case of Pinterest you ruin people's marriages I actually saw them a velocity when I would like screw you I want my wife back you bastard they're like we have the same problem with our wives except today where they call us when it's slow damn it speed up the website I can't I can't tame it other questions so how hard do you find it in convincing people that DevOps is really not just about the tumors not extremely hard extremely hard but the tools are the easiest part to do right and honestly it's part of the waste assignments think developers do this too but sis admins do it more right this goes back to the fundamental value proposition out as and obviously different Lisa Simon's think differently right we'd love to figure it out fundamentally to our core that's what we do is we figure out we're the guys who don't read the instructions and get it right the first time anyway right that's where we get our value that's what we really makes us feel good our greatest joy is doing something with a piece of software or whatever that the creator never envisioned right like duty can do that like oh no could do that really hell yeah I'm a sysadmin man I made it do that like it's a printer I didn't know can make jello it can balloon I am a sysadmin um that's what we do so what happens when we go and see presentations the DevOps right we're scanning skated bla bla bla bla bla bla bla bla bla bla bla bla bla tool ah I'll use your tool and then I'm going to figure it out myself and implement it now the coolest shop now what was the most important thing most important message was culture not the tool you use you use the tool but that just it's an important thing for all of us is you know right you ever read a book and skip to the example right read a blog post and you see a shell you know example you know code block in the middle and you're just like blah blah blah yeah okay right Brendan Greg spends lots of time writing amazing books for system administrators and everyone skips to the examples and he wants to kill all of us you still are sir it never leaves you I want to parallel universe of pages and sample pages tjaka where you type example such-and-such this is one of the great yeah this is one of the things I want to because I start I'm old so I started on Solaris and then I like I like Linux came along and I'd like open a man page and go straight to the bottom feel like what no examples how does anyone know how to do anything they probably read these things or something O'Reilly sells a lot of books outside purpose oh yeah so it kind of getting into that right but Estes admins we have very certain odd needs dazzle of the built-ins yeah they like to take a bunch of code like look there was nothing there was a scraps of code and I assembled a cathedral behold my glory I've built it right it's generally nods but the guy who's like you built it but I'm gonna make a Mick gel right there's like damn it it was perfect you're like yeah yeah right I'm not gonna use it the way you intended I'm gonna do whatever I want to do with it right like hmm why are we at odds with each other I have no idea then do the QA guy in the middle is just like I can break stuff if you've ever met a really good QA person like you know the Deaf comes along is like it's perfect it's bulletproof go ahead and try it right and then you have the QA guys like oh I can smash it look stupid programmer right right of course ouch guys like yeah beat him up before comes to me don't forget marketing Oh marketing is I don't even understand market no this is this is an important thing particularly as we move towards DevOps and we're working closer together what makes us different you don't see many people who are who are sis admins become programmers and programmers becomes this advence it happens but generally I'm very early in your career right why does that why does that transition not happen right also I mean think CS purpose CS programs right if you want to go work with computers and you go to university where are you going to go you're in a one to two ways EEE is your own hardware your get a good CS was CS preusse programs generally really bad ones ok depends on it depends on the university right but but generally it produces very bad programmers where do all the SIS admins come from CS education the people who are like blah blah blah Buicks machine right right most of the best Greybeards of all time for physics guys they're like that's an interesting problem in physics except the supercomputer we built to solve the problems way more fun than the profits up that's where our best UNIX guys came it's all about that core motivation and that's an important thing for all of us as ask things change over the next 10 20 years some things become more more ambiguous between all these different groups is what what's really important to you what do you enjoy doing where is your core value come from is it from building something great like a sysadmin putting it all together building something wonderful or you know building software and and doing things that nobody has ever seen you know these things are important subtle but but drastic emails in the end the explanation of your role enjoy it oh man I'm talking about work so I'm I said I prefer the title by director system engineering because as you see them talk I believe seriously systems thinking and I engineer systems but I run the cloud I run the client operations so Elijah is my colleague and we handle the day-to-day operations of the giant cloud so and it knows when your machine crashes not that that happens it doesn't ever happen but if that would happen we would deal with that we deal with support escalations we build all the backend infrastructure systems that monitor and automates the joint cloud and all those sorts of things there's actually a video up right now I've been talking a great lengthen and great detail about exactly what he does enjoy it yeah so if you want to see sort of a taste of how we do things and join operations you to watch the LUMO stays on right now it's the top light of all this more no it's blog or then it's also the CFS DICOM site and you can see in there I go into things like why we use amethyst and how we do things and what we do with chef and I go into things like you know why LDAP is awesome and anyone who grants users through chef should be shot and you know are back and BSM and all those sorts of geeky sorts of things it was a great job I think in general I'm trying to be like our Lord omnipresent omnipotent good right I mean it's this the real trick right I'm I'm a human being and this is saying giddy like toddler circuit and stuff but I'm a human being non machine Tom I want machines to do work and come to me and say we need a human come over here and help us and so I spent a lot of time trying to build systems that can do that so I can be where I'm needed as a human being and the machines can do the wrestling and it's harder thing that it seems right the Lord's got angels I catch agents so so one of the themes that came out with people were asking questions for things like cutting decide like you should involve a human being how you determine what level detail for the process that needs to be implemented by something what the SOP should look like and it comes down to through the slide with the five lies on it so car going to process that Ben is going to follow it's five why's all the way down because otherwise there's going to be something in there he doesn't know cuz I'm a comedian we have too many moving water tongue everybody has too many moving parts it's not just it's not just me or Ben or what do you guys ever hear and ambiguities really hard ambiguity is the enemy information right it is it and this is the funny thing is like a theory right you know you can do thousand experiments to prove it but it only takes one failure to disprove it's like any procedure anything that we do right the thing that I'm most afraid of is the thing I don't know I need to know about that's the one thing that scared right and like I said in a lot of our cases the pragmatic example is somebody says go do something you're like I don't even know where to go I don't know where the thing is that's problem yeah get rid of ambiguity but that also means that you got to remove some of your pride right all of us like to hide things in order to make ourselves sort of thing dispensable I mean I know none of you do that I mean the other people somewhere fills it up revolve it about those things exactly exactly well we think you're indispensable bad so thank you very much for the time on the table out here somebody they get that that's why I can't figure that same take for you clothes or anything stacker our stickers please come pick them in tabs in the bathroom stall beer for all these places that might give people the wrong impression

Channel: Deirdré Straughan

Views: 48,267

Rating: 4.6745763 out of 5

Keywords: devops, Ben Rockwood

Id: h5E--QSBVBY

Length: 102min 43sec (6163 seconds)

Published: Wed Oct 31 2012