Mastering Argo CD Sync Waves: A Deep Dive into Effective GitOps Synchronization Strategies

kubernetes is awesome right we just need to define the desired State and if you figure out how to convert it into the actual State most of us are not even interacting with kubernetes API directly but by pushing manifest to git repositories and letting github's tools like Argo CD and flux do the synchronization everything is amazing until it isn't unfortunately we cannot always just push manifest to get repositories and let github's tools figure out everything sometimes we need to orchestrate orchestrators [Music] today I will focus on a few less known yet very very important features of Fargo CD that can help us orchestrate kubernetes resources and make sure that they are synchronized in the right order now when you hear the right order you might think that I lost my mind you might be thinking the cargo City will figure it all out so that you can watch Netflix while it does the work well that is actually true in some cases but false in others let me lay down a few problems to begin with argosity performs a dry run before it starts sinking resources that's great except when it's not there will be cases when dry run fails miserably because it thinks that resources were trying to synchronize do not have resource definitions a simple example would be to apply crosswind provider config before the provider itself is applied if you push both to get at the same time Argo CD will perform a dry run on the provider config and fail because the provider does not exist the second problem I want to discuss is the order resources are synchronized most of the time the order does not matter for example if you try to create a pods that references a secret that does not yet exist kubernetes will put it into the pending State and weight and weight and wait until the secret is created the Pod will eventually be created but if you try to upgrade a pod that contains a new release of an application it requires changes to the database schema we need to make sure that the database schema is updated before the Pod is upgraded with the new release otherwise the new release that expects the new schema will be deployed before the schema is updated and the application will fail actually it's your inverse application will not fail that would be good because then kubernetes will restart the application but the application would misbehave finally Argo CD relies heavily on specific Health signals from kubernetes resources but not all resources are emitting the same signals so we need to solve issues with dry run sync ordering and resource statuses we'll try to fix all those issues today and make your experience with Argos CD even better but before we move on I have a request for you forget everything I said so far clear your mind this is clear great let's start where most of us start let's deploy an application to kubernetes with Argo CD let's take a look at an application defined in silly demo yaml this is a simple Argo CD application definition that points to a specific repository where the manifests are stored there is nothing special here if you are familiar with argosity if you're not first of all be ashamed of yourself and was the shame wears off stop watching this video and watch those instead now let's take a look at the manifests of the application itself defined in appiamel this one starts as a very basic one with a Twist at the end there is a deployment a service and an Ingress further on we have a secret that contains a password to the database now don't judge me for storing that secret in unencrypted form in a git repo this is not a video about security so I did what no one should ever do finally we have something called SQL claim which is a Crosslink composition that will create and manage a database which in this case will be running in Google Cloud that's this with Argos CD I will assume that you are familiar with crosswind because it's awesome and you should be using it if that's not the case well this time there is no need to stop watching this video because understanding cross plane is not critical for understanding what we are going to explore today just write a note to yourself to watch those videos later so this is a stateful application or to be more precise a stateless application that depends on a database now databases at least SQL databases are useless without schemas so the final piece of the puzzle is the database schema defined in DB schema yaml in this case I'm using schema hero here to define the date of a schema I will not go through it either since I did already a video about that as well it's over there so far so good we have an application in kubernetes a database server in Google cloud and a database schema and all that is referenced through an argosity application so let's Supply the argosity application by executing cubecatal apply and see what will happen now if you go to argosity UI we can see some strange things happening some resources like the service the deployment the schema hero database Ingress and the table are not being synchronized why is that happening I mean I'm pretending I know the answer right isn't argosity supposed to sink all the resources at once now before I answer the question let's discuss what the order of synchronization should be application needs a database schema and the database schema needs a database server since database server is defined as cross-planet SQL claim I made sure that one is created before anything else I instructed Argo City to wait until the database server is ready before you start synchronizing other resources now to be clear I did not have to do any of that at least when the first release is concerned if I did not specify any order in Argo CD tried to sync them all at once some of those resources would fail the PODS of the application could not be created with the secret with the credential for the database and the secret cannot be created before the database server is created so some resources would fail and others would be the pending State until the dependent resources are created and that is normal everything would run eventually yet I did Define in which order resources should be synchronized the reason why I put extra effort will become evident when we get to deployment of the second and all other releases after that we'll get to those releases later for now let's take a look at how I defined the order Argo CD does its best to apply resources in order that makes sense for example if a namespace is defined it will create it before it creates namespace scope resources it will create custom resource definitions or share this early on and it will apply custom resources or CRS after the resource is baked into kubernetes what does matter is that it does not have the ability to guess in which order we want to synchronize resources Beyond some common rules it could not know that I wanted the database server to be up and running before it starts synchronizing other resources so I had to give it a hint let's take another look at a piano file over here we can see that the secret with the database password contains annotation sync wave set to -10 now to understand what that means we need to talk about phases and sync waves there are three phases in argosity pre-sync sync and postsync the default phase is sync but we can instruct argocity to sync specific resources before that will be Precinct and after the sync phase which would be passing I did not Define any phase so all resources are in the same sync phase now within each phase there are waves with numeric values resources are synced in the order of the Waves by default all resources are in the same wave zero if you would like to think something before other resources we can assign it a negative value and if you would like them to think after other resources in the same phase we can assign them a positive value in this case I set the sync wave annotation of the secret to -10 so that it is synced before other resources I did the same with SQL clay it also has the sync wave annotation set to -10 to summarize argosity all those resources first by phase then by wave followed by the kind of the resources and finally by the name of the resource so even though the order often does not matter it is predictable and we can influence some of it by declaring phases and sync waves I said that Argo CD Waits until all the resources in one group are ready before moving to the next one here's the problem Argo City cannot know what it means for a resource to be ready to be more precise it knows what it means to be ready for resources baked into kubernetes but not always for custom resources given that almost every kubernetes cluster has custom resources no matter whether we are aware of that or not we need to tell Argo City what it means for resource to be ready that's defined inarga City config map using Lua Scripts and that where pain begins let's take a look at what I have defined in values yaml file it's cam values file I used when I installed Argo CD over here I'm saying that I want to customize health checks for the API those toolkitseries.com and the kind escort claim further down I'm saying that if the condition type of the resource is ready and the condition status is false the status is degraded similarly if the type is ready and the status is true then the resource is healthy now unlike waves and phases that are easy to specify as annotations this madness with statuses is just silly to believe it many do not know Lua and even those who do might not be happy writing that whole script only to tell Largo city which field of the resource contains the status it's painful and silly and I hate it to be more precise I think it's okay to have low scripts as an escape hatch for things that are not baked into Argo CD but statuses are so important that I cannot see how that can be left to work around it will be much more elegant and user friendly if you could just specify the field as an annotation in resources just as we do with phases and sync waves it's bad but necessary so I had to do it unfortunately we are not finished yet there is one more piece of the puzzle missing I already mentioned the target CD performs a dry run before it starts synchronizing resources that's great since it can catch some errors before they happen however it can also fail when it should not imagine that you push the Manifest that will install Cloud native postgres or CMP and the Manifest that Phil created database using CMP those database manifests are custom resources Argo City Dry Run would fail because it would not be able to find the CMP crds it would say something like you cannot create those resources because they are not defined in your kubernetes cluster there are no crds synchronization that would create those crds would never happen because dry run would fail the solution to that problem is to tell Argo City to skip dry run for specific resources now I do not have an example for that in my application but I do have it in the manifests I used to install cross plane the providers and most importantly the configuration that brought the composition that we are using into the app let's take a look at config SQL file here I have an annotation that is extracting Argo CD to skip dry run for that resource the crd required for it to be applied will be installed eventually so I'm not worried about the order but I have to tell it to skip dry run so that the whole sync process starts otherwise nothing would be installed eventual consistency is a good thing only if synchronization happens without synchronization it's neither eventual nor consistent it's nothing let's get back to our application and see what happened and what we might need to do to make the second release successful it takes around 5 to 10 minutes for postgresql database in Google Cloud to be created I've been talking for a while now so I expect the first release of my application including the database server to be ready let's confirm that by taking a quick look at Target CD UI we can see that everything is green and that the last thing is okay I can confirm that everything is operational by sending a request to the application which in turn will try to communicate with the dataway so I will send a post request to insert some data to the database and the get request to retrieve the data everything looks great so I can congratulate myself on a job well done right well I think I was actually lucky schema hero worked it created the database schema but it could have failed to do that and I would not find out that there's something wrong at least not right away and not by looking at Argo CD UI I can explain that by showing you the spec of the schema hero table do you notice something strange right no do you there is no status neither kubernetes nor Argo City nor anything or anyone else can deduce whether it is ready or not whether it is okay or it failed or something else happened to it this is an example of a bad design made by someone who does not understand the basic principles of kubernetes and how resources should behave and by this I mean schema here itself now I was positive to schema hero so far because we did not have any other kubernetes native way to manage database schemas but that will change soon I found a better tool and you tell me true comments if you're interested in exploring it in one of the future videos for now I will ignore schema Hero's silliness and move on to the second release of my application for the second release of the application we need to apply schema changes if there are any before we deploy that new version of the application so let's instruct argosity to do just that by opening DB schema yaml file and adding a few annotations I could set a sync wave to some negative number however I will add a hook instead and set it to pre-sync that way I do not need to worry about the order of the waves and I can be sure that the schema will be applied before the application resources are upgraded since they are all in the default phase which is sync next I will copy the notation and add it to the table as well and finally I will make it a bit more realistic and add another table now that we have changes to the schema let's generate the manifests of the application itself instead of making some changes to one of the app manifests I will just execute the money to generate them for me based on the newer release by the way I have a video about team money as well so if you're not familiar with it and there it is you know what to do so the schema was changed and the deployment of the app now uses a new image so we are ready to go and that means git add git commit and git push the rest can be observed in argosity UI we can see that the new table something else was created more importantly for this story is that the tables and the database definitions from schema hero are now having the icon in the shape of an anchor that means that they have the pre-sync hook and that they will be synchronized before the application resources are upgraded to be honest that's not really the case because schema hero is not having statuses in Argo City cannot use it to deduce its Health to and so on and so forth anyway schema hero is silly but if it shouldn't be if it would work correctly Precinct would work well nevertheless that's separate problem to discuss I will show you a better way to manage database schema if you're interested in that in one of the next videos and now we're finished thank you for watching see you in the next one cheers foreign

Channel: DevOps Toolkit

Views: 8,007

Keywords: devops, devops toolkit, review, tutorial, viktor farcic, k8s, kubernetes, ArgoCD, argo cd, GitOps, Sync Waves, DryRun, Dry-Run, Statuses, Kubernetes, DevOps, Continuous Delivery, Deployment Automation, Container Orchestration, Infrastructure As Code, Infrastructure-As-Code, IaC, Application Management, K8s, Deployment Workflow, Efficient Sync, Application Synchronization, DevOps Tools, Container Deployment, CI/CD, CI, CD, Kube Config, Containerization, Deployment Strategies, Kubernetes Management

Id: LKuRtOTvlXk

Length: 18min 27sec (1107 seconds)

Published: Mon Aug 14 2023