DevOps MUST Build Internal Developer Platform (IDP)

devops is not about containers and aws or azure or google and kubernetes and prometheus and all those things that's not what devops goal is the goal is to enable developers to be self-sufficient to be able to do whatever they need to do and to have freedom to do it on their own terms let's talk about how we should be doing devops and what should be the goal set in front of us [Music] what is the first association when you hear things like containers kubernetes cloud github ci cd logging monitoring infrastructures code troubleshooting security service mesh and so on and so forth for many of you you're probably thinking hey that's me that's devops that's what i do i manage those tools i do stuff with those tools but that's wrong that's not devops that's operations operations need those tools what you should be doing like operations person or essay or devops or whatever we call ourselves these days is not to give those tools to everybody because everybody is never going to figure out how kubernetes works all the details about it and all the details about aws or azure or google and how to troubleshoot how to monitor and so on and so forth we cannot shift left by giving those tools to everybody so they are reserved for operations one way or another but the question is what do we do with developers do they continue just writing javascript or python or java code of their applications and throw it over the wall to other people that are specialized in operations or cloud or security or should we enable developers to be able to do those things by themselves to manage their clusters to manage their servers infrastructure in general to deploy their applications to run ci and so on and so forth and if you do agree that devops is about combining development and operations and that the goal is to create self-sufficient themes we cannot say hey you just do it from now on here's an aws account create your kubernetes cluster define your applications running in kubernetes or serverless or whatever we're doing monitor yourself and so on and so forth just spend couple of decades learning the tools and then you will be self-sufficient that's not going to work that never worked and it will never work because it is impossible for everybody to know everything so here it goes devops is about creating services that others in our company can consume those services can and should use different tools to accomplish the mission but they cannot be exposed directly to everybody simply because it is impossible for everybody to learn everything so ops and in this case i'm using that term very broadly to describe all the people who are sitting on the right side that are in charge of cloud and kubernetes and monitoring and this and that all those should be creating services that developers can consume and that should enable them to do everything themselves everybody has a customer and your customers are developers and you need to make sure that they can self-serve their needs so here it goes devops is all about creating idp which stands for internal development platform it's internal because it is meant for internal usage it is developer because application developers are the consumers of that platform an internal developer platform or idp is a layer on top of all the technology and tooling that engineering team has in place or maybe that's not the case we're going to discuss that later for now what matters is that idp is a platform that enables everybody to do operations that they need it's about shifting left and doing it for real it's what operations and sres and devops and security teams create to enable developers to self-service themselves you can say that idp is a self-service platform for everything now you might be asking hey why would we do that why would we enable developers to be self-sufficient and there are quite a few reasons and the first one is that we want to free operations to do what matters and what matters is creating a service that others can consume instead of waiting for somebody to open a jira ticket like hey i want this application deployed there hey there's something wrong over here or hey i need a cluster it is much more efficient to create service for others to self-serve their needs than to wait until somebody tells us what to do and that somebody being blocked while waiting for us to do something there are numerous studies that show that idps increase productivity often drastically they reduce lean time they increase deployment frequency they decrease mean time to repair or empty tr they decrease change failure rate or cfr and so on and so forth so let's discuss how we should build an idp that is tailored made for a specific organization on a very high level the goal is to provide a service that enables full life cycle of everything that means change of desired states performing certain operations or actions converging the actual into the desired state and being able to observe the actual state of the system let's start by talking about the actual state the actual state are all the resources running in our system we can divide the textual state into providers like aws azure google cloud maybe on-prem data dog elastic splunk and so on and so forth so we have a number of providers that we are using for our system and on top of those providers or below those providers depending on how you look at it we have infrastructure that's servers and clusters and databases and so on and so forth and finally we have applications they can be our applications applications that we are developing or some third-party application we might choose to run database let's say postgres database ourselves instead of using a service from a provider and that would be an example of a third-party application we did not develop that application but we might need to run it nevertheless all that together is the actual state that's what is running somehow and serving certain business needs and to make the actual state state to make it run somehow we need to define the desired state the desired state is all about defining what the actual state should be we do it by writing code by writing some manifests or configuration files and so on and support the goal is to describe what we want and then have some processes or some services that convert those desires into reality or into the actual state and here we are facing a huge problem the manifest that define the desired state can be complex very complex for example if we want to define a kubernetes application we need to define an ingress and a service deployment or a stateful set maybe we are using service mesh in which case we will need to define a virtual service and then we need to figure out what to do about storage to mount it somehow into containers and we need to build container images and so on and so forth and that's only an application in kubernetes that application might be running as serverless somewhere else and then the manifest will be completely different or we might need to define a server or a cluster let's say a cluster eks clustering aws to define a cluster we need to figure out how to define eks itself and node group an internet gateway and a vpc and subnets and so on and so forth there are a lot of moving pieces that we need to gather together define them and put them into logical grooves and only then we have something meaningful now the problem is that all those things are typically building blocks we cannot really define an application in kubernetes we cannot define eks cluster as a cluster and so on and so forth we need to take many different pieces and put them together unless you're specialized in aws or kubernetes or serverless or this and that it would be very hard to imagine a person knowing how to do all those things so we need to simplify how those manifests are generated and we are going to get back to that later what matters for now is that the desired state is a collection of code and manifests and configs and so on and so forth and those manifest define the actual state a cluster a network an application and so on and so forth now we need to store that desired state somewhere and the only reasonable place to keep it is git because kit is where we keep code and code is something that can be interpreted by machines and that means that almost everything we do is code because almost everything we do is one way or another interpreted by machines now let's move into the area of tooling let's talk about tools because we cannot avoid having tools we need tools there are many different types of tools and typically we start with what we already have and that can be almost anything i will put them into different groups we have things like pipelines you know for cicd for automation of the steps you know a collection of steps executed whenever we make a change to a git repository we have githubs that is in charge of synchronizing the desired state with the actual state we have infrastructure tools that are making sure that our infrastructure is up and running we have our back for security authentication and so on and so forth and there are quite a few other types of tools however for idp those are the most important types of tools we need pipelines to automate stuff we need githubs to synchronize the states we need to manage infrastructure and we need to manage access through our back or something similar that does not exclude other types of tools we need more than that but those are the main areas that we should be thinking about when we think about idps and enabling self-service idp is what puts all that together and everybody interacts with idps through a ui so let's talk about ui so idp is a single pane of glass that unites everything it enables developers to change the desired states and observe the actual state now you might be asking hey what should be the form of the user interface that we use and that could be anything it really depends what you're comfortable with it could be a web ui or a cli or ide or more likely a combination of all those somebody might prefer to run things through a terminal somebody else might prefer to click things in a web ui and most of developers like working with ides like visual studio code so the user interface can have different forms now here's the important thing about user interfaces for idps they cannot be only links you cannot just create the interface that says hey here's a link to jenkins builds and here is a link to your terraform manifests and here's a link to this here's a link to that that does not enable people to be self-sufficient that does not enable self-service because the complexity is still the same if it was complicated to use terraform directly in the past it's not going to be less complicated if you put a link to terraform and the same thing goes for jenkins or circle ci or prometheus and graffana and so on and so forth now that's where almost everybody starts we use something like backstage and here's a link to this here's a link to that that's not helping much that's not what idps are about idps are about creating services that everybody can consume and to make that happen we need to make things easier to make things simpler and that very often means hiding the operational complexity hiding all those tools and the interfaces behind all those tools and creating new interfaces on top of them that will be tailor-made for what everybody can understand and use now for all that to work we need an api below that user interface we cannot create an interface that just executes cli commands because that's not enough for now everybody understands the importance of having an api that allows us to perform certain actions to query it to find some information and so on and so forth now the problem is that we cannot rely on many different apis if we have 75 different tools we would have 75 different apis that would never work we need a single api that the ui will talk to and the only candidate today that we have is kubernetes kubernetes is the only universal api that we can rely on everything else is fragmented or serves specific needs kubernetes api is the only universal api we have however when i say that we must standardize on a single api and that api should be kubernetes api or cube api i'm not saying that everything should be running in kubernetes that's not the goal kubernetes is not trying to be the only platform you should ever use to run your applications but it is trying to become the only api that matters we can use kubernetes not only to run containers inside of a cluster but to manage everything that's what it is designed for it was designed to be extensible and today we can extend it to interact with almost anything so let's talk about cube api and see why am i saying what i'm saying so why am i saying that we should use cube api for everything especially as the only api for idps is the only api that we should communicate with when using idp uis like web ui cli or id and so on and so forth there are quite a few reasons and the first one being that kubernetes api is on a path of becoming a universal api to manage everything it has continuous drive detection reconciliation meaning that it can take over part of operations that we normally do today it makes sure that things are always running and if there is a drift between what we want and what something is kubernetes will reconcile that and making sure that the actual state is always the same as the desired state it was designed from the very start to be extensible we can extend cube api to manage any type of resource by creating new custom resource definitions and finally this is important remember this kubernetes is not only about running containers kubernetes is a powerful universal api with a very good scheduler that can manage anything and containers are only the first wave of adoption they're just a small piece of what kubernetes can and should manage so let's talk about custom resource definitions and custom resources or crds and crs custom resource definitions kubernetes custom resource definitions or crds are the key in this story and to understand that i must ask a question who writes manifests that represent the desired state of something of an application or a cluster or a server or whatever that something is and if the answer is everybody should be able to define the desired state for the things that matter to them then we need to simplify those definitions those manifests and that's where crds come in kubernetes custom resource definitions allow us to define almost anything hey what does it mean for you to have a backend application this this and that excellent i can create a new custom resource definition specifically for that what does a database mean for you this this and that excellent i can create yet another custom resource definition i can create a crd to define whatever i want whatever we want we can create completely new definitions that are tailored made for specific audience and the level of knowledge and experience of that audience which in this case are developers so if resources currently available in kubernetes are building blocks crds are an opportunity to define what something is to define compositions and the relation between different building blocks and expose all that into a new definition that is tailored made for everybody and that means that we have an opportunity to simplify everything we have an opportunity for ops to combine building blocks into compositions and for developers to write manifests based on those custom resource definitions and convert those manifests into custom resources that will do whatever they need to do hey you need a cluster with networking and storage and monitoring excellent this is the resource definition that you define when you create an instance of custom resource based on that definition all those things will happen you do not need to worry about those building blocks you can focus on what matters now let's talk about the tools which tools should we use to put all this together i will ignore for a moment that you most likely already have a lot of tools and you cannot get rid of them you do not want to get rid of them and those tools can be anything but i will ignore that and imagine that we are starting from scratch how would we do things if we would start today and there are a couple of areas that we need to think about to begin with how would we define application and infrastructure specifications and that's something that i already covered previously we should define all those things as kubernetes custom resource definitions and then create manifests based on those crds and convert those manifest into custom resources that will do whatever needs to be done kubernetes crds and crs are the abstraction layer that we need to hide the details that do not matter for general audience and simplify everything in a way that everybody can self-serve their needs next we need pipelines that will perform certain actions like hey i need to run tests when this happens and i need to build some contender images when that happens and i need to do this and i need to do that and most of those actions being triggered as a result of some events like hey i just pushed something to get repository or this happened over there and so on and so forth now since we already standardized with kubernetes being the integration layer kubernetes api and scheduler then those pipeline tools which we typically call ci cicd should be based on kubernetes as well or to be more precise the way how we define those pipelines should also be based on custom resource definitions and then create as many custom resources as we have built in a system and so on and so forth so if you can choose i would probably go with argo workflows and tecton and if we cannot choose then use whatever you already have there is advantage in going kubernetes native but those solutions are often complicated very complex and if you cannot handle that complexity then any other solution should do like circular ci or roger devops does not matter most of the tools in that area are easy for developers to use because pipeline is just a set of actions or scripts that should be executed in certain order then we need to think about synchronization how do we synchronize the desired state and the actual state how do we make sure that the actual state is always the same as the desired state and that area today is very clear that's either argo cd or flux and potentially run your fleet in the near future if it matures but that's probably the easiest choice simply because everybody today uses argos editor flux if you're not using those you should you must next we need to figure out how are we going to orchestrate infrastructure how are we going to create obstructions on top of infrastructure for the same reasons why we are creating obstructions on top of everything and since we are standardizing on kubernetes that should also be based on kubernetes custom resource definitions and that means that we should be using crossplane or sdk from aws or any other infrastructures code tool that can run inside kubernetes and that is capable of extending itself through resource definitions the choices around application orchestration are even more limited if the goal is to tailor-made something for our own needs the options are narrowed down to open application model with cube vela or cross plane both of those allow us to create new custom resource definitions that define what an application is or anything else in an easy way and more importantly in a way that can be consumed by everybody and finally we have our back or access control which is relatively easy and straightforward if everything is stored in git and git contains the desired state then we always write things to git then git is the only one that has right access by humans and that means that we just need to make sure that correct people have correct permissions to access specific git repositories there is no need for majority of people to ever access directly kubernetes or cloud or whatsoever except in read-only mode we still need access to kubernetes that is sitting below idp simply because idp needs to observe the actual state so we humans need access to see what's going on with all the resources floating around wherever they are but that is really only access and that shouldn't be a big deal to set up now let's talk about the process how would that process look like how would people interact with idp and what should idp do everything we do is based on two things either we are changing the desired state or we are observing the actual state and if you focus on changes to the desired state it starts with a developer writing code and pushing it to a git repository either directly you know git commit git push or through idp ui whichever you are you are using it could be cli or id or web ui in any case if you go through idp the result should still be the same if you define something over there that something must and i repeat must be pushed to get as well it does not matter whether we interact with git directly or through a ui the end result must be pushed to get if you have a web ui of any sorts that is not pushing changes to git repository you should ditch that ui immediately unless you're using to observe something like you know monitoring tools if it's for observability that's okay if your intention is to change the actual state of something then your new desired state must be pushed to get that's where most web uis are failing big time now once we push things together directly or through idps pipelines are executed those are one-shot builds with steps that do things like hey i need to build an image i need to run some tests i need to push some artifacts to some registry and so on and so forth and pipelines themselves might also change the desire stating it if a pipeline is building a new container image and pushing it to registry and we want that image to be applied to an environment then pipeline itself should change the image tag in a manifest stored in a git repository and that manifest defines the desired state of an application apart from pushing application code you know java python script developers might be writing manifests or updating or modifying manifests and pushing that to git as well just as before it can be pushed to get directly or through web ui that has a form that allows you to create manifest in an easier way it does not really matter how we create those manifests and what does matter is that they are converted into code and by code that means something that can be interpreted by machines including yaml and push to a git repository those manifests must be based on custom resource definitions that define what something is this is an application this is a cluster and so on and so forth it must be based on custom resource definitions that's the only reasonable way for us to simplify things now independently of whether manifests were created directly or updated by us or by pipelines git of stools should detect that there is a drift between what manifests define which is the desired state and what the actual state of custom resources in kubernetes is we can call this the git of sexual state right that's the state of the resources in the kubernetes cluster and remember those custom resources might not be only containers with applications running in your cluster they could be aws resources they could be git repositories they could be databases and so on and so forth custom resources are tailor-made resources that describe the desired state of something now once one of the github's tools detects that there is a drift between what is a manifest and what is a custom resource in a kubernetes cluster and remember that custom resource is still not the real actual state that is coming later that's still not the database what matters is that git of stools are making sure that those two states converge what is in it and what is in a kubernetes cluster from there on the job of that custom resource is to create something something real which could be a cluster with vpcs and subnets and whatsoever or it could be a database server with certain databases inside and certain users configured or it could be an application that consists of ingress deployment and services and so on and so forth or it could be a serverless application running google cloud run or aws lambda and so on and so forth the job of custom resources is to make sure that the actual real resources physical resources are running somewhere so that is the process that's a high level architecture and now we can talk about user experience what do we actually do in idp how do we interact with internal developer platforms idps are in charge of three major areas to begin with they help us write manifests based on custom resource definitions defined by operators and they help us push those manifests to a git repository second idps help us with observing the desired state which is based on manifesting it it enables us to see what the system should be we can zoom in and out and see only a part of the system or a system as a whole but the point is that it helps us visualize the desired state which is based on what is stored in it and finally it helps us observe the actual state you know that's monitoring troubleshooting logs and so on and so forth it helps us collect all the information that we need and present that information in an easy to digest way what matters is that all those things are tailor made for specific developer needs idp is not about enabling operators to do the things that they already do idps are about enabling developers to do the things that they're not expert in doing and even if they are experts it makes things much easier much simpler and much more efficient and now you might be asking hey where do i get that where do i get an idp so let's talk about that so your first instinct might be hey i will just buy idp where do i buy idp and that's a good choice there are solutions that you can purchase and just use right away but they work well only for small companies that are starting from scratch and the reason why i'm saying that is that those solutions those that you can buy and start using right away like heroku for example are highly opinionated the only way to make something extremely simple from the get-go from day one is to make something extremely opinionated very very very opinionated and it's great when something is very opinionated because it's easy but the problem with that is that we need to adapt to the solution solution does not adapt to us and that works well if you're a small company or you're starting from scratch because you can say hey from day one i'm going to do things heroku way or whichever solution you chose but if you already have things if you already have the system you need to have a tool that adapts to you and that means that you need to build it yourself building an idp is the preferable way for bigger companies especially those that already have assets in place already have applications and systems and so on and so forth because if you build it yourself it can be tailor-made to your own needs and the solution will adapt to you instead of you adapting to the solution which let's face it will never happen if you're a big company it already has years and years and years of investment now when i say build it yourself i am in no way saying that you should build it from scratch and create a new file main.go and start and build it from scratch because that's insane that would never work because that's too much and we already have tools that can help us build our own solution but still based on existing tools that are made specifically for those use cases we need tools that help us build opinionated platforms instead of using off-the-shelf solutions which were great don't get me wrong but you need to be small or you need to be starting today and that solution that you're going to build must follow certain rules it must store changes to get so it doesn't matter if you go to web ui and fill in some form and click a button that button must push it to get the desired state must always always always go first to get you must be using git as the only source of truth of the desire state that solution must have a universal api an api that allows us to do everything and interact with everything no matter whether we go through one type of ui or another type of ui and that solution must be a control plane it cannot just blindly do things it must be acting as a control plane ensuring that things are happening in specific order under specific condition monitoring the drifts and reconciling the states and so on and so forth we must have a control plane so storing it use git as the only source of truth of the desired state have a universal api and have a universal control plane so what can we do today or should we wait for tomorrow let's let's talk about the present and the future so if i would start building an idp today not yesterday because yesterday we were doing those things differently and not tomorrow because tomorrow is still not today but today i would be using certain key ingredients the most important ingredients would be probably backstage that would give me the flexibility to create that ui that unifies everything cross-plain that would help me create custom resource definitions and custom resources and act as a control plane and so on and so forth argo cd or flux both are great for synchronization between the states and i need pipelines i'm not very opinionated about the pipelines you can use anything jenkins is fine circle ci is great azure devops is good and so on and so forth there are plenty of pipeline solutions use whatever you're familiar with but if i have to make a choice if i would be starting from scratch then that would be tucked on or argo workflows because they fit well into that kubernetes ecosystem and on top of all that you would have a bunch of other ingredients and that could be anything else that really differs from one organization to another so there's much more to it but the key ingredients would be backstage crossplane argosidior flux and tectonorargo workflows but how about tomorrow what is coming and i'm very excited about tomorrow because i believe that will be one of the next waves in 2022 and beyond we will see a huge surge in interest demand and solutions for idps that part of the market is going to explode for a very very very good reason idps are in a way evolution of kubernetes that's where kubernetes is going and remember i'm not advocating it you should run everything inside kubernetes i'm just saying that kubernetes will evolve into becoming a universal control plane a universal solution to manage everything not only what's inside that kubernetes cluster but outside as well the next steps is first for you to explore different types of tooling especially those presented in this video and then combine them all together and start building your own personal internal development platform or idp

Channel: DevOps Toolkit

Views: 53,631

Keywords: devops, devops toolkit, review, tutorial, viktor farcic, kuberentes, k8s, gitops, ci, cd, internal developer platform, internal development platform, idp, crossplane, backstage, argocd, argo cd, flux, flux cd, developer, ops, sre

Id: j5i00z3QXyU

Length: 36min 21sec (2181 seconds)

Published: Mon Feb 28 2022