Extending GitOps to the Enterprise

good morning good afternoon or good evening depending on where you are in the world my name is julia godinez and welcome to today's devops webinar extending get ops to the enterprise brought to you by harness we have a great webinar for you today but before we get started i need to go through some housekeeping announcements today's event is being recorded so if you miss any part of the webinar you will be able to watch it again we will be sending out a link to access the webinar on demand or you can visit devops.com webinar and it will be available for you as well we are taking questions from the audience throughout the presentation use your webinar interface to submit questions into the q a section and we will try to get to as many as possible finally stick around until the end because we are doing a drawing for four 25 amazon gift cards so stay tuned to see if you're a winner joining me today is srinivasa gravelly founding engineer and chief architect at harness and samarth wadwa senior product manager at harness i'm not going to put myself on mute turn off my camera and let you begin thank you julio good morning everyone uh i'm summerswag and i'm one of the product managers here at harness uh joining with me is srinivasa we'll be your guides today so we really want to get started and go over one what is get ops and also the rise of git ops what has really cost for everybody to move towards get ops that's what we're going to talk about then we'll briefly touch upon the get ops 101 the whole idea being why exactly good ops is becoming so famous and next we want to briefly touch upon the myths and facts uh there are a lot of myths about get offs and we really want to talk about what are the myths what are the facts and how exactly good ops you know uh can help solve certain things next the challenges with get ops a lot of people have seen different sets of challenges we'll briefly cover what those challenges are and the enterprise approach to solve these challenges for get ops and now i'm just going to go into the rise of get ops a decade ago if i were to start on srinivasa's team who is the architect right um i'm coming in as a application developer and then once i come in srinivas tells me hey summer go and provision a particular dev environment and make sure you have your vm set up you have your jboss setup and also you know using jproc now the other sets of environments that are being pre-provisioned is also being taken care of by my team but there are so many people involved in order to do this the first person involved is the systems engineer they help me with you know what the setup should look like they tell me what the vm configuration should be and so on and so forth next i have a networking engineer my networking engineer is going to tell me what the load balancing rules are going to be what different policies i need to apply and last not least i have a middleware engineer so all of these different things has been divided by different sets of people and everybody owns these different entities so i need to go to each and every engineer to make sure that i can get my vm provision and also i can make sure that you know i'm up and running within a few minutes but no that is not the case because it's going to take me about two to three weeks to get a new virtual machine next a load balancer change is going to take anywhere between two to four weeks and last not least to add a new app server note is going to take another week hence i'm sitting over there twiddling my thumbs for almost about a month and a half because i'm trying to understand what exactly i need to do when i get the virtual machine set up and also make all these different changes now think about it this way if i come on today in devices team what is this new setup going to look like right a new virtual machine can be up and running in 10 to 15 minutes a load balancer change now takes two to three minutes and a new app server note 5 to 10 minutes in order to be up and running so technically if i joined his team today within the next 30 minutes i should be able to get started and i should be able to start writing my first pull request that i can submit for the application next i'm going to briefly touch upon progressive versus declarative deployments also known as push-based deployments and pull-based deployments so thinking about progressive versus declarative if you look from a progressive standpoint these deployments are nothing but deployments that have a change being made to your manifest repository and you are pushing a particular change in this case what happens is that you have a particular artifact that you are writing the code for and then you make a push and as soon as that push happens you go ahead and you trigger either a workflow or a pipeline and that goes ahead and deploys that particular change to your cluster or the infrastructure of your choice moving to the declarative side of deployments it follows a pull-based model what does this mean it has the concept or the notion of a operator with this operator coming into the picture what it does is it looks at your current state of your application which is running on a particular kubernetes cluster and it also looks at the states that is defined in git the whole idea is to continuously compare and contrast what exactly is changing on your git repository and making sure that those changes get applied to your specific cluster now that's an important thing right because from a pull based model when you declare a particular change that change gets propagated across all your different clusters and it makes sure that the state of your application is pretty much in sync with the state of git making git as your source of truth next i'm going to talk about some common get ops tools that we see in the market if you think about it we've seen all these different tools such as argo jenkins x gitlab code fresh and flux now the important thing to consider is that all these different tools and providers in the market today they are solving some part of get ops but are they really solving all of git ops i think that's the question we really wanted to answer today as a part of this webinar because when we look at it when we look at it from an enterprise grade tool we want to make sure that we solve for a completely different use case and also make sure that github is able to solve the enterprise problem along with everything else the good and the bad now you know everything that comes in comes in with something good and something bad so obviously even get ops comes with the same thing the first thing saying that you know for your the good part of it is the source of truth is always git which is amazing right because we know that if anything should change if anything should have changed we know that we need to go to git and that will remain as a source of truth next is drift detection in order for anything to change from the state of your from the current state of your cluster and if anything is changing in git we have the ability to do drift detection which means that a drift is detected and we know what exactly has changed and how the reconciliation happens between git and the cluster is all taken care of by get ops the bad when it comes to non-declarative infrastructure that's where git ops does not really play a vital role now non-declarative infrastructure could be uh tools such as virtual machines they need certain steps that you need to deploy and then all of that happens in a non-declarative manner next is with regards to failures audit trails and role-based access control gear ops does certain sets of role-based access control and audit trails but everything at a cluster level but is it really good at handling failures i think that's always an open question to be honest last not least is the confidence building steps say you wanted to run some form of qa analysis or regression tests for that getups is not the best way to go because devops offers that capability where you can run a certain set of steps after your pipeline is running or as a part of your pipeline and make sure that everything is looking correct before you deploy it to certain other clusters or virtual machines all right moving on i'm going to spend some time on get ops 101. real quick we're going to talk about what is get ops well this was a term coined by v works in 2017 it was the concept that spoke about pretty much how an engineer was enacting a code change and you know git was becoming your source of truth and how exactly that was being propagated across different cluster and infrastructure changes so that exactly is get ops guys the whole idea for doing get ops from back in the day was to make sure that as changes get propagated it goes to the end users then and there and also we make sure that git is always your source of truth because let's be honest everybody including all developers and devops engineers and psychops love to be in git and won't get to be their source of truth next we're going to briefly touch upon the github's principles i know we've covered a part of this but the entire system is described declaratively which means that each and every entity that you perform and get think of it like a transaction maybe a banking transaction as well right everything that happens is a particular change that we do and that gets propagated to your cluster next the canonical desired system state is versioned say srinivas goes and makes a change and maybe we have another team member he also goes and makes a change and i go make a change everything will be version within git so that way we exactly know what has changed and how exactly we can handle these changes should something break and how do we go back to the previous commit from git next is the approved changes that can be automatically applied to the system what does this mean it basically means that as soon as you approve a particular change upon drift detection you will be alerted that hey the current state of your system has diverged or has changed and now what exactly happens is that as soon as a user approves this they can go ahead and they can apply that change to your clusters and last not least to the software agents to ensure correctness and alert on divergence it is nothing but the drift detection capability where when something does change we make sure that we are alerted on the divergence and then we can decide if we'd like to apply that change or not with this i'm going to pause for a brief second i'm going to turn it over to string of us to continue with the myths and facts of gitops hi good morning good evening to everybody thank you very much so much right so if you have any questions right now and feel free to put it in a chat now i would like to let us go over myths and facts so what is one of the myth is githubs is only for kids is it true you can answer it in the chart well the answer is no because it does not matter if the cluster runs runs indicates or not all github cares about is having an operator and the source has a gate and the destination could be any infrastructures vms or for a movement it could be any other services now the second fact is like is github's github only applies to cluster changes right as mentioned before githubs is not just for k8s as we see here there is an engine github engine with the predefined rules and configurations responsible for monitoring the git configurations and observing the current state of the cluster to take actions also in this case we have a clustered state which can al also can happen to be a vm state or any other other services right now okay github is centralized strategy people people assume that there is a centralized strategy like devops where we have a template that can be reused across different environments in the case of githubs this is not true because there is a one-to-one mapping between the cluster and the repositories so it does not follow the paradigm of centralized strategy which means users will have to maintain separate repositories of for different environments which is good however it comes with an overhead of maintenance consistency visibility we are going to talk about that in the in the coming slides now devops versus githubs right there is an assumption that github is subset of devops obviously which is not true the principle for devops is to break between uh break silo between dev and operation versus for github is to have intelligent automated ops now now we know that what is driving principle in the case of ci cd develops follows push-based model github follows pull-based model in case of deployment devops covers imperative approach github is like decal radio and when it comes to the monitoring the devops provides monitoring at all layers github's mainly concentrated on monitoring the status of the design versus current using operators we are going to talk about more in the coming slides so okay so far like we learned about the myths let's talk about the facts like one of the facts is like meetups is great as a continuous delivery solution where each commit results in a release candidate ready to be pushed into production continuous diplomats on other end is the full end-to-end journey of a commit straight into the production without any manual intervention what does it mean is like whenever the developer checks in the code he can go and happily sleep everything everything will be taken care by the uh github by the continuous deployment process where he doesn't need to worry about whether the where is the pr whether the pr is approved or not and like whether it is some so with someone or whether it's talking with the security team and everything right so gitobs requires a little bit human interaction it requires human intervention detox my githubs for powered by the pull request on a repository that contains our manifest and in most cases human needs to look at and approve these pull requests this means that practicing hitops involves at least some manual steps for handling these full requests in theory one could practice both githubs and and the continuous deployment continuous deployment while still adopting the githubs by fully automating the pull request but that is that is very difficult and there are challenges around it the another with like the another fact is github does not address promotion of release between environments as i talked pre as i mentioned earlier githubs is one of one to one mapping between uh cluster and the git report what does that mean is if you want your service to be propagated across qa and staging and the production you need to have at least three three uh applications which will be mapping to your git repository it could be one git repository or it could be multiple hit repositories or the single repository is uh differentiated by the branches whatever it is but you need to have three git repositories to map it to the three environments so that um that is a fact what what what we're going to lose we are going to cover on that and how to address this problem we are going to cover than that the another fact is that auditing is a problematic despite having all information get so one could argue that hey the git is very good in version history and the git history is that is that is good enough for enterprise auditing may not be because you always have like some reviewer or a proverb as privilege to do course fees post push once you do the post push you will lose all the history so is it like enterprise is going to like it yes but we need we need to have more than that the auditing should be preserved and there should be a visibility across uh days and uh we need to do some analytics on politics and uh follow the some security principles and all all in that now the last another fact is github covers only subset of the software cycle so github tools are sometimes marketed as the one size fits all solution that will solve your all release problems and this is simply not true first of all github requires that your deployment artifacts are already there that means in a whole devops cycle some part like compiling the code running unit 10 integration test security scan and status analysis this all is are outside of githubs okay now so we talked about the myths and facts and now let's go over some of the common challenges that every everyone faces today with the githubs the first challenge will be the governance so what do you mean by what is governance right as you know githubs is dependent on a pure pr process a pr approval based model the owners is on the pr improver you should be well aware of what is going on between securities what is going on between the um other teams and whether the developer is whatever the made he made changes whether it is right like for example a developer thinks that we can support the replicas 5 and he might have changed the replicas to 3 to five but it will be simple change but we don't know the fact like whether that is a valid change or whether this whether the company allows the policies to be changed to three to five this all it is not easy uh with the simple pr approval process right but enterprise have been looking for a ways to control what any users can do on the cluster and ways to ensure that clusters are in compliance with company policies these policies may be there to meet governments and legal requirements or to enforce best practices and organizational conventions auditability as i mentioned earlier default github's process is not fully auditable the key reason is that is post push force push essentially allows to remove any unwanted blocks of comments from get history on the central repository we are going to talk about that how we are going to solve this auditability problem in an enterprise world now the most important thing is like business approvals typical approval model used in github is via pull request developers make changes create a pr then approve approver may accept such pr which would be recorded and change would be deployed such a pr based approval model is based on a premises that everybody already uses a git and knows what pr is all this is great to the point where we need business approvals in the mix business approvals may very well not known gate not know not know what pr is and not understand what is happening so if we have a business approvals we have to go back and have to build a process over github's process which ultimately um trigger like increases the lead times and there is a manual intervention and all these things will come into picture okay now we talked about the myths we talked about the facts and summer explained about um what is a journey like a decade ago now i we talked about the there are common challenges now like is it github solved all the problems or this github solve all these problems no now let's talk about like how to do this at an enterprise level right so governance so like let's say we talked about what are the challenges with the governance now let's see how to address this governance there are many reasons why developers and platform engineers integrate policy as a code into their common in the current github process so nowadays like this ops suffix is coming to picture you have the configures code and like policy as code guitars which primarily covers the deployments and you are we are going to talk about ai ops all this right but when it comes to the governance should be if the government's like a policy as code should be integrated as part of the github process right so such that let's say if you're creating a pr which happens to change something at the same time this pr might require to make some changes to your check-ups your security policies right and for example you need to block some port or you need to open another port so now let's say if you think uh githubs and this policy has code to different silos then as a developer i i create the pr for the my manifest ranges and now i would be waiting on the other other security team to do the changes in the policy s code and then it is no more like continuous deployment right it is no more like continuous delivery so what if if you integrate the policy as code along with the githubs so for example we may not they may the developer may have no reason to know when deploying the load balance on to kubernetes in aws is are not not compliant right policy has code solved this problem automatically that means in the as a pr checks we run the policy us code checks if everything is good the pr will be merged and those the new changes will be deployed into the cluster right the ability to run policies during github's deployments and adhering to the policies as part of the deployment and upon any violations alert audit notify and auto roll back the changes from the git to the previous successful complaints state so that it remains in sync with the cluster okay let's talk about github's plus layouts so there is a notion that once github is completed which is good the source of the truth is gate and the changes are deployed into the cluster and we are happy right but the world is not like that there are many unknown reasons or unforeseen activities will come your application behaves or your infrastructure behaves normal during the normal load but all of sudden it behaves very different during of peak hours or like or there is a hack right or there is a compromise happened now so as part of github we can see the manifest change or infrastructure or artifact change but post deployment github's operator can instantly understand the changes in the service health and what caused those changes and automate the process of invasive investigating those changes using aiops and like user apply the machine learning like apply like canary analysis and do that canary analysis like the previous manifest changes and the previous build behavior behaving like this and that is the baseline now there is a newer version came and with a newer second changes newer manifest changes the new artifact changes then we should see how it is behaving from the previous build and as a you could write a policies or rules to detect is there any violation right is there anomalies and what is the threshold and let's say that github process should automatically automate that process while deploying let's say i would take one example we we usually do the canary deployment so if we do the canary reloads then once once you do the primary deployment we wanted to make sure that this all new nodes are good enough to uh solve to like add all the policies and they have behave very very properly in everywhere right now what you could do is we should run the analysis like aops against those um primary nodes and get the feedback loop and see if there is any anomaly then we should automatically roll back this automatic rollback should involve from like um and to towards the still the developer what does it mean is you have to uh revert to the commit the riveting the previous commit it and and merge that river could commit so that the newer version will be applied to the uh in order to like keep it uh live and restore to the previous state okay that's all like we need to integrate in the githubs and the apps now which is good we talked about uh github's like uh uh github solves the drift grip is that the drift is nothing but like as someone told it is uh what is the difference what what is the in the cluster and what is in the gate and then you can view the drift and you can uh you can view the drift you can see whether the changes are good or not in the in the pr right but let's say if i you but that is not enough enterprise customers would like to be elected whenever there is a drip in their repository and based on that customer can choose to approve or reject that drift he projected the major pr should be reverted to the previous state to ensure that git is in sync with the desired state of the cluster right so we have to build automatic like drip detection alert mechanism notifying a slack any emails or create a jira and automatically create an era and the ticket the ticket will go for an approval when automatically rule happens proceed with the pr merge right so yeah i think we can open it up for questions now i want to just make sure that you know if you do have any questions we'll be happy to take those now uh yes uh we do have questions but uh one thing i want to remind everybody is that you can submit questions by going to the q a tab next to chat and submitting your questions that way okay so one question we have here is do you have any suggestions on a get ops tool for infrastructure i've been using argo cd on kubernetes openshift but i would like to know if there is any similar git op solution for i believe that's iac using terraform for example [Music] yeah so yeah that is true uh there are like github tools that are available like argo cd and uh on the kubernetes and openshift uh there are some tools um i like to do the infrastructure as code but that is also not like um uh fully compliant to the enterprise like for example it is based on the telephone and everything but as we speak you might be familiar with ordnance uh ordnance is like uh customer like it's not a uh open source but it is one of the uh the best uh standard in the industry they are like working on like and as you speak i'm also part of harness we are working on uh the the ias tool like infrastructure as co code it's completely agnostic to either it is a terraform or any um anything like any other prisoners so i i we can get back to you but i don't have like any proper solution ready anyone is solving that is properly as of now okay well uh right now we don't have any other questions but uh we'll give it a minute just to give people time to submit any oh and somebody wants to know uh how do you spell the name of the solution you mentioned oh oddness also um is git ops only for container based no that's what we are saying is its githubs is mainly for kubernetes and it is saying like container but in reality it is not true so github means uh it can be like you can use against the infrastructures or any vms for a moment like for example we could extend the github process to intercommunicate between the two different services also for example in a middleware and everywhere if you want to if you you you have a service and you are producing some of the output and that should be consumed by and the services you can automate through this git process so the answer is not it can be for anything yeah i also feel i just want to add something here real quick guys uh for get offs only for container base right that seems to be like a myth for sure um a lot of times folks do get confused that everything happens to be anything get obsolete and happens to be only for containers but the answer is that you know it is not uh as srinivas mentioned it could also be a virtual machine or any other service for that matter where githubs can be extended uh it happens to be that get your remain as a source of truth but also there are different ways to solve it so you know that's that's something we want to be cognizant about and we want to make sure you know that that's how we can do it um i think it says it would be good to have an enterprise create tool for comprehensive kid ops on virtual machines and kubernetes how do you suggest using wavefront ignite it seems nice but it's still an alpha um i feel like we will have to do a little bit more deep type on what the tool does but it seems like if these guys are doing a form of comprehensive get ops right it might be similar to being able to see what your state of kit is and applying that on virtual machines and on kubernetes clusters but also it comes down to tying the enterprise great features right you want to make sure that they can solve for not just uh the drift detection but also the ability to do uh the entire approval based mechanism right you want to make sure that you have your roll based access control you want to make sure the audit trails that give you the capability to go and see who did what when and where those are the different enterprise great features you want to look for by deciding a tool that you want to use in order to do get ops how do you incorporate continuous testing as part of get ops is there any specific tool for testing in for a container based code should you know what you want to take this one how do you incorrupt continuously yeah so that's what like this when we are saying that extending enterprise githubs to the enterprise as of now there is no open source solution available to do this continuous testing but you know there are some vendors like cycle ci and everything they do as part of a pipeline process what our nas are trying to do is uh they are going to come up with a open source um they're going to come up with this like this solution to get through a pipeline so how to address this solution is like you you have a unified pipeline where you do the ci the build and then uh after the ci stays there is another stage called or another step called like test intelligence right or uh so testing you keep testing that uh in that stage and if you if you do if you makes you if you're sure that these are the test results are good then you could have a github stage where the stage does is it automatically creates the prs against the your targeted environments then you can attach the some of the policies to the stage if if those all policies are met it can be automatically approved that's where you have to extend the githubs that's the reason i was mentioning in my previous talk also gitops is the continuous delivery in order to achieve the continuous deployment or continuous testing we have to extend the githubs and introduce the concept of the pipeline where this all can be automated with more security time with approvals so yes that is one of the challenges in the vanilla githubs like you make changes to the dev environment but how do you in order to make the same similar changes to uh production environment you have to do the same manual process some part if you say argo cd it does solve a little bit by introducing the application set controller where that application set controller nothing but like very where you can automate this creating this applications in the prompt um by reading another repository which contains the configurations right but there should be more than that what what should be more than that that should be obviously that's not there that's what i was talking there should be more than that the environment propagation pipeline should be there what does that mean is you make changes to the dev and you have you need to have a verification step where you verify all those changes and with approvals then you are fine all those approvals and everything in the staging environment then once you approve it it should create a pr against the your production environment then after that github should take care of it i hope does that answer your question okay excellent um still have plenty of time for questions if anyone has anything but i did want to remind everybody that today's webinar has been recorded so if you missed any or all of the webinar you will be able to watch it again we will be sending an email with a link to access the webinar on demand it will also be available on devops.com so look in the on-demand section and in the webinar page and it should be there [Music] also i do want to make mention that the handouts uh the slide deck has been uploaded so if you'd like to download that you can go to the handouts page and uh download it for yourself and uh while we have a little bit of time left for questions i do i'll go ahead and read the 25 the ford uh amazon gift card winners our first winner today is onofrio l so congratulations our second winner is peter t congratulations peter our third winner is uh pedro a congratulations and our final winner is deepak r so congratulations to our winners we'll be reaching out to you via email with an instruction for claiming your amazon gift card so please check your inbox and if you don't see anything there check your spam folder and still not seeing any new questions so it looks like uh you all were very thorough with your presentation which is good to see all right all right then well going once for questions going twice okay well uh thank you very much to uh srinivasa and tamar for an excellent uh presentation uh really nice uh listening to you speak and uh thank you thanks again for the audience for joining us and for your questions oh looks like we might uh where can we find the presentation the presentation will be in the handouts it's right uh it's on the right next to q a and from there you can download it uh if not it will be available to you in the same email with the instructions to watch the on-demand so uh with that again thanks again to our speakers for an excellent webinar and thanks again for the audience um this is julie godinez signing off until next time be well thank you thank you

Channel: DevOpsTV

Views: 88

Rating: 5 out of 5

Keywords: devops.com, devops, devsecops, continuous delivery, microservices, containers, devopstv

Id: I1gSeHTYYcU

Length: 39min 46sec (2386 seconds)

Published: Thu Sep 30 2021