Flux CD v2 With GitOps Toolkit - Kubernetes Deployment And Sync Mechanism

this is going to be one of the deja vu videos we are going to review explore do some hands-on exercises with flex v 2 flex with devops toolkit and it is deja vu because i already did that last week and i did it terribly i misunderstood the documentation or documentation wasn't incomplete or my brain did not work well so i made some false claims in the previous flux 2 video which is now unlisted you cannot watch it anymore and so on and so forth so if you didn't watch it you will not be able to watch it if you watched it i apologize we are going to do that again we are going to explore flux v2 with devops toolkit and uh let me let me tell you right away it's awesome and we are going to go through it because we all like githubs if you don't know what github sees hey here's a link this explains what github sees and because flux is the main competitor of argo cd which is also awesome i explored it watch the video here so i will assume that you understand how it works today we are going to explore flux v2 again this is reviewed version new version uh and we are going to try to do that in 20 minutes or less [Applause] let's start with what i have and that's not much all i have all i created or did before recording this video is i created a cluster a kubernetes cluster k8s cluster that's it in this case it is mini cube cluster but it could be any cluster we can we can imagine any kubernetes cluster should do now the scenario that i want to go through is to use to have two environments in this cluster production and staging now this is for simplicity reasons i would normally have more i would have also preview environments maybe pre-production whatsoever i might have many environments and those environments might be split into multiple clusters or different name spaces inside of that same cluster and so on and so forth but for today's exercise i have a single cluster where i wanted to deploy staging and production but i want those two things to be separate simply because uh those environments are separate from access control reasons for repository organization reasons reasons and so on and so forth so two environments production namespace separated in different namespaces inside of the same cluster that's what i'm trying to accomplish today i'm going to start by creating repositories git repositories where i will store production manifests or references to manifests and staging so make there p flux production that will be the name of the repository and i will have the directory apps inside i will store the app manifests in the directory ups and then i'm going to go into the directory flux production i'm going to initialize kit uh i'm going to create repository it will be public but you normally would do private but for this access public and use the default values that's all okay cool and i'm going to create the readme file production just so that i have something to push and that will be readme.md and i'm going to add the files and commit git commit those files to the repo so i just created more or less uh empty repository it has a readme and not much more and i'm going to push that set up stream origin master to the remote repository to github in this case so just created repository nothing special happened so far so far and inside of the cluster i will create a namespace production so i have a repository for production i have a namespace for production now i'm going to do the same thing for staging make there flux staging and directory apps inside it i'm going to go inside of that directory and uh same thing initialize kit i'm going to create a repository okay public and default values i'm going to create commit i'm going to create readme and add the files commit the files and push the files setup stream master oh that was a mistake setup stream origin chain master cool and just like just as before with production i'm going to also create a namespace staging just as i did for production and go back so i have two repositories so far and the two namespaces in my cluster nothing else not more not less so i created a repository this is git get production and i created the repository a git repository in this case in github and that is staging and that's all i have so far let's continue let's see what else do i do i need next i will create a fleet repo and that might sound like a strange name but that is a repository where we will keep references to different environments production staging uh where or any other environment we might have and that is repository where we will keep the setup the installation the manifests of flux itself and while we're at it we're going to also install flux so and in flux terms from cli perspective that's called flux bootstrap so flux bootstrap the repository is github it could be gitlab and a few others owner is going to be me repository is going to be flux fleet branch will be main up oh that's typo the path where we will store the definitions the manifest will be ups remember we created a directory before and it is going to be a personal repository um it could be it could be organizational private company type of repository but this is my personal so personal should be okay so what is happening right now is that cli is connecting to github it created a repository flux fleet it created manifests pushed into that repository and now it is installing flux and this is absolutely awesome because it is really showing that flux is trying to apply github's principle everywhere including on itself like if you go to argo cd documentation you will see that there is a lot of mambo jumbo create this command create that command to install arco cd and only after argo cd is installed argos argo is really applying githubs for real but installation is finicky in case of flux it applies githubs from the very beginning and in a very guided tour i didn't create the repo i didn't create the manifest flux cli did all that for me it created all those that stuff pushed it to get to and then it re installed itself and from now on i can go to this repository flux fleet in this case and modify any aspect of flux and it will work it will update itself now let's take a quick look at what did we get cube cartel namespace flux system that's where flux installed itself and get pods and we can see that flux is up and running i will not go through the details of what each of those components means this is a quick overview let me see where i am now i'm going to clone the newly created repository and see what did we get over there and this is me and flux fleet free dot gift by the way i didn't mention before all the commands that i'm running will be available in a gist so go to the go to the comment section and you will find the gist with all the commands if you want to reproduce what i'm doing the commands are in the gist it's in the comments section cool so i cloned the repository if i go there flux fleet uh and if i output apps directory we can see that there is flux system subdirectory and if we output that file the directory subdirectory we can see geo tk component sync customization those are the manifests that do a couple of things but the most important in this context is that they define flux itself installation and maintenance of flux and they're telling flux and this is the most important thing that this is the rip we should monitor this is the source of the manifest we're going to create a couple of other sources very very soon actually right now let's do it so what do we need uh actually let me go back to drawing board to show you what we got so far so we got the third repository and this is also git repo and it is fleet repo and that repo contains manifests that are required for setup and maintenance of flux so some files were added here and then flux flux was installed inside of the cluster and from now on flux is monitoring this repository every time something changes in this repo it will be reflected in our cluster the next thing we need to do is to tell flux how to monitor the the staging repository and how to monitor production so those two repositories are also applied in the cluster that they define the desired state and we're going to do that by creating a source file we're going to create a source file he's put it in this repo and through that source file tell flux that there is staging repo and that there is production so sources will be referencing those two repos flux will pick up those source files and from there on flux will monitor those two repositories let's do that part so we're going to create use flux now cli and this is another awesome thing that i like about flux i don't have to create yummy files from scratch i don't need to copy and paste files from somewhere i can use the cli to create the file for me so let's do this create a create source it's going to be git we're going to call it staging and the url of that source will be github dot-com and v-farsic and flux staging so this is a this is this the repository that we created initially and we're going to specify a few more argument arguments like branch is going to be master interval is going to be 30 seconds so every 30 seconds this this repository will be synchronized or checked whether there are changes and we want to export the manifest created through this command this command will not change anything in cluster just create the manifest and we are going to store that in the apps directory and the file is going to be staging yaml so all that this command did was create this file it created this git repository definition that is now inside of the local copy of my flux field repository next we are going to do the same thing for staging so flux create it's going to be a customization this time we are going to create a file a customization the typical kubernetes customization actually not typical flux specific customization reference that will tell flux hey that staging source that staging repository here is how you convert that repository into manifest this is the format and the path and so on and so forth so customization and it's going to be called staging as well the source is staging now this is this is really important this is telling flux that the source is the one we created earlier the file actually that we created earlier the path is going to be the root anything in the root of that repository should be considered the manifest prune will be set to true meaning that whenever we delete something from that repo it will be removed from the cluster as well validation on the client side and what else can we say uh interval let's uh change the interval to 10 minutes for example and we want to export the manifest and just as before i'm going to store that into the file in the same staging file staging yaml so i'm going to upend this definition into the staging yaml file and this is the second definition now we got customization that tells flux how to how to treat the manifests in the source that we created earlier and this is important part almost everything in flux consists of a source that is a reference to a git repository and customization or helm release some form of release definition that tells flux say this is a customization these are customization files so these are hemp files and so on and so forth now i'm going to do the same thing for uh production and to simplify my life i'm going to copy and paste from my notes because the commands are the same they're just having diff reference to different git repository to production repository so it did exactly the same with production and there is one more thing missing one more important thing missing we are going to start thinking about the applications themselves so i have a demo application stored in repository prefarsig slash devops toolkit so i want to create a source uh to that repository reference to that repository i could do it inside of staging and production repos but it's better if i put it to the fleet repo simply because the same reference to the same up repository will be used over and over again releases will be different in staging and production but reference source is going to be the same so flux create source git i'm going to call it devops toolkit the url is going to be https and then github.com slash vr6 slash devops toolkit this is the existing repo branch is going to be master interval what should i put 30 seconds again and i want to export that and pipe it all into t to devops to the apps directory devops toolkit dot so what i did so far is just create ucli to create some yaml files all i did was create yaml files nothing else and those yaml files are stored now in the fleet repo in the local copy of the fleet repo so i'm going to push those files to the to github in this case and that is let me see git add i want to add those files i want to commit those files edit the environments and i'm going to push them now let's see what flux thinks about me pushing those files to the fleet repo i'm going to watch flux to retrieve sources of type git and we can see that it already picked them up there are four sources that it is monitoring for git repositories flux is monitoring there is devops toolkit this is the the application that i will soon deploy to staging and production flux system this is where flux and all the resources references is stored production defines production and staging defines staging so far for now those are only the sources those are only references to git repos sources are not going to deploy anything it's just source code without any actions now to see actions we need to uh watch for flux get cast authorizations and there are three and two of them are failing is this a disaster no not yet maybe it will be later right now flux detected three different customizations flux system this is the setup of flux itself and production and staging and it is complaining that there are no objects over there that's because we did not push any definitions to staging or production so it knows that it should monitor those repositories but it did not find any any helm releases or any customizations over there that's normal we didn't create them yet now let's deploy the first release and change this remove this error message now we're finished with the setup now we can do the now we can really go and do some githubs with our application let's start with staging so i'm going to go one directory back so i'm going to go through directory flux staging and i will create the first release the first read i'm going to express a desire to have a release of my application running staging but before we do that before i do that i will need to create values file because that release in staging will not use only the default values of my definition of that application it will have to have few overwrites and the new values file that i will create is this one this is a temporary file it will not exist for long uh so i'm specifying that hey i want to overwrite the tag of that release to be 2.9.9 i want that specific version running in staging and i want ingress host to be this address which is temporary domain of this cluster that i'm having cool so those are the values that i will overwrite it's it's a file that i will not need for long it's very temporary so let's create the first release the first helm release in staging plugs create helm release i'm going to call it devops toolkit and the source will be git repository and it will be devops toolkit this is the source that i created and stored in the fleet repo it doesn't matter where eventually in which rip you store stuff because flux picks that all up flux already knows that i have git repo that was toolkit because of the previous definition that i stored in fleet and what else do i need so that's the source devil's toolkit i will tell you tell it to use values from the values yaml file that i created earlier i will tell you that it is a chart helm meaning that that in that repository of that application there is a directory helm that it should use to find the benefits and the target namespace namespace will be staging i want to deploy this this release in staging and interval should be what 30 seconds again sounds reasonable and i want to export that manifest and then i'm going to pipe all that to a file that will be appended to apps devops toolkit now there is no need for to open this the first time i'm creating that file apps there works toolkit yaml so i created a file dev toolkit.yaml this time inside of the staging repositories repository dedicated to defining this desired state of the staging environment in this case a namespace it could be a whole cluster does not really matter and as you can see the values were upended to that file to the hem release definition so i don't really need the values file anymore i'm going to remove it i hope that the project will add set argument to flux create but so far i don't need this file anymore anyways as you see so far i'm not interacting with the cluster in any form of way at least not to change the state all i'm doing is interacting with git so i'm going to add those files i'm going to commit and it's going to be initial commit and i'm going to push those changes to i'm pushing it to the git repo to staging repository right now which as you already know flux is monitoring through the source file so watch flux get sources get actually already saw this there is no need to to watch the git sources i want to do something completely different and that's something completely different is i'm going to watch flux uh get the helm releases and you can see right now reconciliation is already in progress flux already detected that i pushed the change to the staging repository which is monitored by flux because i created the source and now it's finished it deployed my release to staging probably let's double check that cube cattle dash namespace staging getpods and here we are you can see that my application device toolkit is running in staging and all i did was push a change to git and flux is monitoring all the git repositories that i defined as sources now let's say that i want to change the release to deploy a new release to staging how would i do that let me think let me go and edit edit this file i'm going to edit this file and i'm going to change the tag to be 2.9.17 saving the file adding the file committing the file upgrade and pushing the file to get so all i'm doing is pushing the file to get now let's see what is happening i'm going to watch cube cattle dash dash namespace staging and get pots and you can see that the new pod is already what's creating now it's running the old release is already terminating it it is finishing the job come on come on terminated and there we go the new release is running the old one was destroyed uh it used deployment strategies was defined in this case rolling updates it could be canary deployments blue green does not matter what matters is that as soon as i changed my desired state in a git repo that is reflected in a cluster and the new release is running and i can let me show you i'm going to copy and paste this command because it is a bit long here this is the tag that is currently running in in staging 2.9.17 so it really did the right job now the only thing missing is for me to say hey you know what i want to promote that specific release to production let's say that i run the tests i reviewed the changes i did everything i need to do i'm confident that this is ready to be running in production how would i do that let's see cd flux production i'm going to the local copy of the production repository and i'm going to do the same thing as before for staging initially i'm going to create the file that will overwrite the values i'm going to create the hem release just as i did for staging the only difference is that it it is going to target the namespace production and that it is using uh and it is using actually the same source table toolkit that is defined in the fleet repo so i'm just telling now in a different repository that i want this specific release of this application to be running in the namespace production and i'm going to delete the values file i'm going to add those files remember now i'm working with the production repo not the staging that's the major difference git commit initial comment and git push now let me watch flux releases watch flux cap helm releases actually i'm not even gonna watch releases i'm going to watch i'm going to take a look at the pods namespace production get pods what is running in production nothing right now nothing is running in production let's actually watch it that's more interesting than getting the same command over and over again type there we are actually while i was figuring out my typo my application is already running in production that's about it i have stay now if i would like to change for example the release in production the logic would be the same i would go to the file and i would just change let's say the tag to be let's say that i want 18 right top save the file push to get and that's it change some definition push to get and let flux take care of it now let's let's go back to the drawing board so that i show you what we did uh what we really what happened in the meantime since the last time i was drawing so since then i created i created a repository actually already had a repository a git repository for my application app and i added in the fleet repo i did the source that is pointing to the trip repository and from there on flux started monitoring that triple and monitoring the repo as a source is not really doing anything the magic happened let's say that this is staging section of the cluster and that this is production state uh section of the cluster or namespaces now uh from there on what i did is that as soon as i defined something staging flux picked it up and deployed to staging the moment i changed that definition in staging flux picked it up again and again redeployed it then when i added something to production repo again flux picked it up figured out the changes and then deployed to to the production namespace in this case it could be cluster and so on and so forth so as soon as i did change some definitions in staging that's applied to the staging uh in this case staging namespace staging cluster staging whatever and same thing for production i could have additional repositories here git and here git and so on and so forth i could have as many or as few repositories as i want and as long as i tell flux through the as long as i add source that points to one of those repositories uh here and there uh flux will start monitoring those repos and from there on it's all about adding releases helm releases or customization files or whatsoever to apply those desired states into the cluster so that's it that was flux in 20 minutes or less actually i did not i did not track time i almost certainly went over the 20 minutes for for the demo if i did i apologize but uh flux has so many things that it is very tempting to go overboard and it is what can i say about it let me think i think it's absolutely awesome i think that what uh v works and the contributors to project did with flux 2 is absolutely amazing is it much better than argo cd no it's a very tight race it's a very close race but so i cannot say which one is better which one is worse they're similar in what they do but somehow different in how they do it uh argo cd has a single up definition that defines equivalent of flux v2 sources and releases and customization flux is split into different resources which might be more difficult to grasp learning curve is slightly higher but on the other hand the separation into sources and customizations and human releases and whatsoever allows us much much more possibilities much bigger number of permutations and potentially much more power i think that flux v2 is accomplishing something that very few tools managed to do and that is to be opinionated to guide us towards certain outcomes while still being extremely flexible very flexible it's the biggest difference between flux v2 and the previous one is that flexibility with version 2 you can do almost anything you want github's related at least related to kubernetes deployments and management of kubernetes resources it is absolutely awesome and i'm really really impressed what was done in the first iteration of the new major release we're going to see i will most certainly most likely make a new video maybe half a year from now and see how far the both projects flux and argo cd went but for now flux v2 is absolutely awesome i might i might even say that until recently uh my preference was for the argo cd now i think that i prefer flux v2 uh but for a very narrow margin it's a very close race i might change my opinion half a year four months from now but where flux speed tuna is now especially since it is the first iteration of the major release it's absolutely amazing the only thing i'm missing is uh ui open source version of the ui like cargo cd has so ui is a negative point or mis not having ui is a negative point at least not free version open source version part of the project on a positive side very flexible and potentially very very powerful it will be impressive to see where it will go from here if you're not subscribed to channel subscribe right now subscribe if you like this video thumbs up immediately if you want to support the channel buy one of the courses or the books uh courses are on udemy books are everywhere or nowhere i don't know uh the links to all those those things are below the gist with all the commands is also below in the comment section and what can i say uh flux i mean i'm really really impressed with flux speed too it took me a while i made a mistake i did not understand how it worked that's why i created the video now removed the previous video so there is some work pending to done to do on documentation side it is not really clear how it works but once you figure it out it's brilliant and what i like the most i forgot to say that i like that it follows guitar's principles from the get-go from the moment you decide to use flux it forces you in a way to use kit and with dargo cd you basically install argo cd in a finicky way without really following githu's principles and from there on your applications are all githubs and all that stuff bootstrapping in flux p2 is also amazing anyways i need to go uh this is already much longer than it's supposed to be hey see you next time cheers

Channel: DevOps Toolkit by Viktor Farcic

Views: 12,300

Rating: 4.978261 out of 5

Keywords: flux cd, flux, fluxcd, fluxcd helm, flux cd kubernetes, fluxcd github, fluxcd example, fluxcd tutorial, fluxcd exemple, GitOps, Kubernetes, k8s, deployment, git, GitHub, desired state, actual state, WeaveWorks, gitops kubernetes, gitops toolkit, gitops toolkit flux, gitops kubernetes flux, fluxcd 2, kubernetes tutorial, gitops flux, gitops best practices, gitops example, k8s tutorial, weaveworks flux, fluxcd getting started, weaveworks flux 2, weaveworks gitops flux

Id: R6OeIgb7lUI

Length: 37min 4sec (2224 seconds)

Published: Tue Nov 24 2020

Reddit Comments

[u/lukasmrtvy]

Wondering if is possible to include fluxv2 as argocd config management plugin, its definetly possible with fluxv1 ( https://github.com/alexmt/argocd-flux, it worked for me.. ). You will get best of both worlds ( argo ui, flux auto image update )

https://github.com/argoproj/gitops-engine is still not finished.