Devops is Terrible

a critique of how we fooled ourselves for years devops is ball how do you guys feel type one in the chat if you think Ops is [ __ ] okay we got a lot of ones in the chat okay let's find out over the years it has developed into an Unholy Beast of division and tunnel vision why did we stop dreaming bigger what happened to tearing down silos increasing engineering velocity and adding value remember the things devops was supposed to do those are also all the sentences I would expect to hear in TechCrunch what are you doing tearing down silos and we are increasing velocity adding value hell yeah uh but the reality is outside of fang and the most well-funded companies your team is probably doing one of the following you've got a devops team congrats that's not devops I'd wager most of what they're doing is using terraform and yaml to do menial tasks for engineering team pick is this true is this what you do pick s like no I used Haskell I no it's Haskell um it's not a lie damn it need a database file a ticket with devops need an I am roll file a ticket with devops before longing your massive team of Engineers has fully saturated your understaffed devops teams backlog this approach does not scale uh you are doing devops but it feels like [ __ ] I like this article uh devops the free for all where each engineer does it uh the necessary Ops to do their job best practices will invent them along the way sakir I'm too busy increasing conversion rates naming conventions nah okay that's not actually funny that's this is not funny okay there's nothing funny about that I don't think we should laugh about this because it's a real issue that we've all had it's just it's not okay that's not okay cost management deleting unused Cloud resources nope we've got AWS credits to burn the problem is most Engineers don't want to do operations work they want to build the product they want to add tangible value but organizations will force this upon them and Engineers will either work with what they've already got guess their way through some of the new cloud service or the devops task will slowly become the responsibility of a few individuals that become the devops team the worst part of the scenario is that deadlines rule the product stakeholders don't see Ops when was the last time you saw a product manager high five the Ops person say fast [ __ ] auto scaler an auto scalar [ __ ] auto scalar where the hell did it go where did Auto scale did I Just Lose It it's real no one high-fives the auto scaler guy it's very very true nobody cares about you I'm sorry slowly your team feels like a malleable a nature of devops stagnant into rigid infrastructure as teams stopped making the harder operation decisions and just bend their software around what they have in the end all infrastructure eventually becomes a platform how easy is yours to change very hard operations is a commodity let's act like it I've been on the operation side of engineering for a long time is this is this going to be a serverless article is this a serverless article have I been bamboozled I think I might be I think I might have been bamboozled right now called it serverless um I've been developing a ruby since uh since rails 1.0 okay well there's your first mistake before that I wrote some of the trashiest PHP in the world that has seen no it just sounds like you wrote PHP uh I've had the I had the opportunity to migrate from a data center to AWS ec2 when it was launched in 2006 and I've become more experienced in operations I was pitch and hold into the devops role to date I've deployed over 200 production grade kubernetes clusters across a handful of companies this man kubernetes okay this manned kubernetes want to know the secret all right let's do this you know what I'm talking about I've copied and pasta the same damn terraform module for every single one of them my job felt like a scam but companies pay me to forget to turn off alerts dang it dang it dang it Karen I need an intern yep got an intern roll coming up I need one I need it yeah I need it right now okay the response the responsibility you said um there's just this thing that happens every now and then Karen that might need a little bit of taken care of and I just figured you know maybe you know maybe an intern you know yeah it's about the yeah it's about it's about the alerts hold the copy pasta driven single user platform as a service and no one cared as so long nothing was broken you've made it this far so I'm just going to say it the company's building devops teams are going to uh in the right direction but they need to be moving away from infrastructure configuration management and towards platform engineering and enabling Dev developer self-service okay I like this this kind of what Netflix's way is the knowledge silos are good and silos are a feature not a bug expertise is a good thing devops is [ __ ] okay okay okay let's see continuous advancements through teamwork efficiency in communication elevated morale leaving the door open to suggestions for betterment sign me up right wait I'm going backwards aren't I test release deploy operate monitor plan code build test release poop I agree this model worked when a cloud was simple and made sense even back when we had a few virtual machines maybe an S3 bucket and a couple of cues this Loop was achievable what through wrench and everything was growing complexity of our system the operational burden of those systems and the compliance for many organizations operations teams inevitably became devops Gatekeepers at every phase of the loop planning planning today requires knowledge and trade-offs of the cloud services as the cloud has gotten more complex teams are left with a few options work with what they've got or sync with Ops to see what their backlog looks like yeah I I can feel this I can feel this we've had some similar things at Netflix where we've we've been gated by our ability to interact with other services this is this is real organizations that have invested in a platform engineering have either a golden path already identified or enabled teams to research and develop new golden paths rapidly this flexibility is key to a great platform we aren't uh building golden roads they are paths golden pass should provide Direction and guard rails but they ex but be acceptable when the business needs to change okay coding is becoming more and more of a hodgepodge of cloud apis this is a good thing we should run to run or we should want to write and run less code okay I agree with this yeah I think some of the problems is that like you know like I know we're talking about Golden pass and all this stuff like to even develop some of these things and make this stuff happen just requires such good ability to like foresee what your team needs that one man's golden path is another man's doo-doo path right it's just like I find this to be a very dip like I love the idea I'm just like Curious how well this actually works in realness oh am I saying that this yes the dyslexia is real shut up uh do you know what's worse than waiting uh wading through an Ops backlog during a planning phase missing your deadline and working late because you're waiting for someone on the Ops Team to update the IM policies and create KMS keys because you didn't realize the since topics were in different regions than your squares if you're not sniffing and squishing then you don't know [ __ ] um a great internal developer platform has conv conventions in place to handle a small tedious bits of the cloud like I am and KMS that take uh the most amount of time to address they focus on Engineers intentions not making Engineers worry about implementation details why can't they just change them jump in there and get some stuff done you know what I mean uh just teasing Brian you know I love you I know you do okay but don't make fun of my dyslexia uh yeah cicd that one term Dave talks about which I don't think he knows what it means but he says he does the fuzzy boundary between development and production these phases are so fraught with [ __ ] that they have become meme Worthy worked in uh worked fine in Dev Ops is uh Ops problem now this is great this is great but that's oh my can I just zoom out a little bit all right but it's uh an Ops problem is uh it's an entire organization problem I'm convinced that CI CD phase is the root of where frustration divisions grow between operations and Engineering teams so this is actually this is all one phase for each team right there is no the platform at Netflix provides like ways to monitor ways to scale ways to do a lot of stuff but like when we release it's not someone else's item right we actually have to take care of it we have to watch the whole dang thing you know how nervous I was when I was launching our first uh like next-gen Network stack and I put 50 000 people into a new network stack and I just watched it being like please keep playing Netflix and it like Works people kept playing Netflix and I'm like yes we've got it right like until then you know of course you put a bunch of people in there and you're just like yes and then you start monitoring logs and you're seeing what's going on then you realize no one actually got the next Network stack and then you go back to the [ __ ] your pants problem oh man like like this was just part of Netflix part of being on a team was when you're launching stuff you you did everything you did it from you do the whole souped nuts we just have platforms that are available to make things work right uh in production we are running in containers but developing developing in container is too slow so the team leans towards ASDF and a readme uh full of stuff to copy pasta and pray during a Sprint an engineer ads convert image magic to the mix to support manipulating images and forgets to update the docker file and then production goes down [Laughter] I mean that's how how porque Maria uh this is where trust between developers and operations begins to erode internal developer platforms should allow an engineer to run their applications where they want how they want Lambda and Docker sugarcades build uh build pack sure VMS and tarball sure one thing that they one thing that I noticed that they didn't say in here is canaries right like you wouldn't have really been able to take down production if you would have canaried the first 100 200 300 people go through your stuff will just be getting zeros right and you'd go oh wait a second we've royally screwed this up let's back it up people so canaries are a beautiful canaries always Canary unfortunately many of the platforms today Force developers to run on kubernetes a great developer platform meets Engineers where they are it should be flexible in architecture but opinionated and self-service uh operate and monitor the final stages of this perfect Loop model are arguably where Ops should be that'd be great except for now we've got threes to cover that huh if the devops team ships a postgres RDS instance it will run fine forever that is until the application starts using it all of a sudden a Cascade of n plus ones hit the CPU spikes and queries grind to a halt all of a sudden everyone's using the new react server components and nobody knows why is so chatty but all of a sudden you're making 900 requests just and you have no effing clue why it's so bad oh yeah I love I love it love it I cannot wait I cannot wait for the next year and a half when people come all the way around be like why'd we do that uh who uh who is woken up and why does this always happen at 2 A.M in this scenario there is nothing for operations Personnel to do yet here they are do you think devops is doing what it's supposed to do the expertise that operations and three teams have is critical to developing secure scalable systems but the old idea of devops and the hurdles our industry has turned it into is holding us back I really like the idea of this article which is really you should always be building for a platform you should always provide a we call it like a gray glove approach and agree just blame image magic always blame him image magic image magic is just the worst it's also the best at the exact same time but a gray glove is where it's not white glove but it's also it's not no glove right it's not black glove I'm not sure what the opposite of white Club is I've never even thought about that uh but it allows you to be able to just launch something and have that work or you could just build it yourself you know like you kind of get some options there show of hands how many people in your organization think CI CD is devops I think it's a mix and Netflix it's a little bit more unique in a sense I don't think anyone could say it's any one thing it's like a bunch of people's responses show of hands how many people in your organization think they don't need devops because they run serverless I knew it was a serverless article I knew it I knew it I knew it I knew it uh show of hands how many of you think above two interpretations are a problem yes uh to most the term devops have completely lost its meaning that's I mean it's true I actually don't even know what devops means at this point I just assume platform engineer if you're really really doing devops youth uh do you think Reinventing all of the commodity is really worth your engineering team's time is that good investment for the business you built it run it more like you build it but it takes longer than your Sprint estimates and you cut corners on the off bit that doesn't fit your definition of done isn't that just all engineering can we just apply this to every last bit of engineering ever for all time what'd you do I got some corners and then after cutting Corners I got some more corners you know what I mean know what I mean then I did these nuts joke on Twitter um Legend silo or a mountain Legend silos knowledge silos and expertise are two sides of the same coin from Full stack engineering to devop practitioner our industry loves to pretend everyone can do everything we're an industry of hobbyists we love to Tinker I don't know if we are fooling ourselves or if the industry has been exploiting our hobby driven nature but it's time for devops to get thrown out of the airlock damn pour one out for devops uh the growing Zeitgeist is that platform engineering is the future and given that I co-founded a project a product in the space I sure hope so unfortunately organizations can't get there by expecting the devops team to do it I'm sorry but copy pasting some terraform modules between your git repos is terrible platform your engineers don't want to deal with it and inevitably your Ops team will be on the hook for supporting it hell even if hashicorp is jumping on the no code provision train for their please contact sales plan huh seems like all these companies with Enterprise bucks are struggling too remember that uh so how does the average organization get to the promised land of platform engineering okay good I want to see some action here simple just hire some more front end and back-end Engineers to develop a great internal pass with all the golden pass your operation operations team is architecting while you are trying to build your actual product that runs on top of it it's that simple just pay a couple million okay obviously easier said than done to get there many organizations will need a reality check welcome welcome to reality [ __ ] uh for every operations person without software development skills there are 40 Engineers without cloud operation skills if you're going to build an internal platform uh you'll need expert experts with overlapping experience in both Fields working together you are also going to need a shitload of time and budget this isn't a hackathon project it isn't a pet project of the new CTO with a wild hair up their ass to plant their flag on the business [ __ ] you can give a cute Skunk Works name to the show everyone you have access to Wikipedia too but building an internal uh platform is a startup within your primary business building an internal development platform is like changing the tires and engine of a car while it is a hurtling toward a cliff stay agile stay thirsty my friends isn't that magic team called devops you will get that [ __ ] out of here okay buddy okay buddy okay buddy we're not gonna we're gonna do those around here okay I don't know if you know this but devops is [ __ ] uh migrate and auxiliary service to get it quickly wait wait I I dyslexia that one migrate an auxiliary service to it quickly get feedback from your engineering customers yes engineering customers engineering customers Boomer type they aren't your team anymore they are your business's second set of customers and if these customers aren't buying it you'll end up with morale problems engineering pinning for the old ways pining for the old ways and a boatload of debt and a bunch of wasted time and effort chin up you are the new cto's next team can get it done you hear that people chin up bootstraps pull them something about a cowboy cowboy up ride a horse wait Ride a Cowboy Save a Horse What's the phrase I can't remember uh anyways uh if you're planning to build a platform there are five core components that are required but in our experience there are a few more attributes that make for a great platform extendable with open source tooling teams and workloads are different and will have different golden paths if an IDP has one opinion I don't know what an IDP is uh has one opinion waited to run your workload it is no better than a pass an abstraction over kubernetes is not enough idps must work on all workload types whether containerized serverless or virtualized anti-lock in you should be able to walk away from a bad build or buy decision without risking production or having an arduous migration process I mean you're always locked in at some level uh security compliance and guardrails must be built in without them you are not enabling self-service yeah this is good you should really security and stuff should just be in there you shouldn't you should never have a Dev work on security do you know the time I worked on something that involved with security you know the time that I did I got I actually built something that got its own named attack after it okay I got my own named attack how many of you guys got that how many of you guys got the uh the what is it called the uh something grizzly attack uh gosh what is it now I can't even remember what it's called It's called The something grizzly attack uh Grizz the grizzly repulsive grizzly repulsive grisly denial of service yes I made this you're you're welcome I made this article possible okay I made that I programmed it it was I it was I who both discovered and made it oh NYX would not have helped this um but yeah I I made I I made the first one on Netflix I legitimately took down all of our non-production services with a bash while loop that is it that's it I took down everything I kept it down for hours it was simple I could have taken down entire production with a thousand requests that's it one thousand requests would have just blinked out Netflix forever I was pretty impressed with myself not gonna lie uh powerful building blocks don't worry it's gone now it's all gone now but uh you know I did that I I did that I thought blue screens were good yeah that's pretty much what I did uh let's see powerful building blocks uh to increase engineering velocity we have an industry wide shortage of expert expertise in the cloud space a great IDP I still don't know what IDP means should have safe Dependable building blocks for Designing cloud services quickly uh enabling experimentation through flexibility and extensibility if you have to reach for another tool or platform to see how your app would work in a serverless container or with a different Pub sub system you are going to receive pushback from your stakeholders in your IDP idps must enable experimentation so the right decision can be made for our applications I wonder how how loose can you really go like how many different Pub sub systems should you have how many different like I don't know that sounds very flexible ephemeral environments for applications and infrastructure uh must be supported our application is becoming so heavily dependent on cloud services if you can't provision dependencies like buckets queues or databases when you open a pull request you are really getting close to approximation to production configuring alerting and monitoring this is my this this is a sweet this is like a wheelhouse right here I love this for provision uh provision infrastructure and applications with good defaults if Engineers can deploy their own resources they must be monitored without requiring an extra tool to configure otherwise the likelihood that those resources have alerts configured will decrease significantly yeah there's like your base level alerts like how's the service doing but then there's also secondary level alerts like is your service actually operating correctly has one change to another actually started causing real issues and so there is always going to be a second level hand bespoke monitoring that must be required you know what I mean platform engineering is possible and it is the future our system are likely more complicated and higher scale earlier as more and more parts of the world get online we're creating a great new Engineers daily out of a out of boot camps but we aren't excelling in operational maturity as an industry we have a lot of data to protect we are stewards of personal information customers assume we have fiduciary responsibilities we but we mostly act like hobbyists we need to make sure platform engineering is the next [ __ ] buzzword I liked it I actually really liked this article yeah devops is [ __ ] what a crap article [Laughter] I mean what I like about the article is that they're put that they're putting forth this idea that you should be building a uh you should be building platforms which I totally agree with right I totally agree with you should be putting together platforms what you should not be doing is putting together like you should not have someone just always doing all the grunt work for everything at all times because that is truly unscalable right it is terrible um you know what I mean I mean that like it really makes a big difference what does he mean by a platform like the ability internal development platform okay it's like you need to be able to launch a service a micro service it needs to be able to have databases it needs to be able to have this that and the other how do you get that all done internally within your company well for the most part what you're going to be doing at any small companies that you're going to be just setting all that [ __ ] up yourself right you're gonna have your own bespoke version and the team next door to you is going to have their own bespoke version and kind of what the the thesis of this article is is that you just kind of copy pasta each other's configs and that's platform and so when something needs to change you have this like golden set of configs that everyone uses to launch all their crap but then you bespoke it out yourselves and then everything is hard to deal with there's no platform there's just configuration being copied around right which is total insanity but it does happen it completely happens uh uh it's not versioned absolutely definitely oh I mean it's version by get peace out Primus and have a great day after meetings hey have a good one buddy what is a platform a platform is something that does this for you right I need to launch a new service what are my language choices I have node or Java at Netflix that's pretty much what you have you also kind of have go starting to come up okay I'm gonna have go I need to be able to communicate with the database okay communicate with the database here's how you launch it I need to be able to do this okay you can do that this is how you launch it oh you need to be able to have logging okay this is how you do it this is how you launch it right like you kind of have to enable each one of the things and blah blah blah blah boom now you have yourself a service it's that simple but here's the deal people twitch.tv I need to go because I I just hear my my little phones is pinging yep that's like my fifth ping

Channel: ThePrimeTime

Views: 281,465

Keywords: programming, computer, software, software engineer, software engineering, program, development, developing, developer, developers, web design, web developer, web development, programmer humor, humor, memes, software memes, engineer, engineering, Regex, regexs, regexes, netflix, vscode, vscode engineer, vscode plugins, Lenovo, customer service

Id: qVEEpUvl0Kw

Length: 24min 35sec (1475 seconds)

Published: Thu Jun 15 2023