What is GitOps, and what role argocd plays
in it? Keeping configurations separate from the source
code of an application and managing them through version control has been a best practice for
a long time. When I refer to configuration, I am talking
about various settings that an application requires to function correctly. These settings could be as simple as a database
URL. Suppose an application is operating in a staging
environment. In that case, it would store its data in a
separate staging database. The emergence of public clouds and the increased
adoption of automation has made it feasible to keep infrastructure configurations in Git
repositories, using tools like Terraform. With these tools, it is possible to declaratively
define the desired number of virtual machines to create, and then apply the Terraform configuration
files, which automates the process of provisioning those new virtual machines. As people started relying more on Git for
configuration management, it became the primary source of truth for infrastructure. This shift in thinking gave rise to new patterns
and approaches to managing infrastructure changes. Instead of directly modifying Terraform configurations
locally, the new approach involves creating a pull request for any changes to the infrastructure. After the team reviews and approves the request,
the changes get merged, and the corresponding CI/CD pipeline detects the update and applies
it automatically. The entire operations process revolves around
making changes in Git and letting automation handle the rest. Argo CD is a continuous delivery tool designed
to ensure that the Git state remains synchronized with the Kubernetes state. To deploy a new application using Argo CD,
you start by creating a pull request, just like with infrastructure changes. Once the pull request gets approved and merged,
Argo CD automatically applies the changes to the Kubernetes cluster. This process is relatively simple, with Argo
CD primarily cloning the Git repository and running kubectl apply at regular intervals. Most of the automation in this process is
built into Kubernetes and not Argo CD itself. Suppose you're using ArgoCD to manage your
application's deployments. In that case, the typical workflow would begin
with your build agent, like Jenkins, pulling the application's source code, running tests,
and building a Docker image. If this process happens repeatedly for every
change or commit to the application code, it's known as continuous integration. After building the Docker image, the build
agent would then release the image to a Docker registry like Docker Hub or any other repository
in use. Next, Jenkins would pull the GitOps repository
containing infrastructure configurations and update the image tag to the newly built image. It would then commit this change back to the
remote repository. On the other end of this pipeline, we have
Argo CD, which monitors the GitOps repository for changes, such as a new Docker image. When Argo CD detects a change, it clones the
repository and applies the updates, usually with kubectl apply or helm upgrade, depending
on what you're using. This process is referred to as continuous
delivery, where every change made to the application's source code results in a deployment in some
environment. Typically, continuous delivery is used in
development and staging environments. However, in production, manual approval is
still necessary before deploying any changes. First of all, we need a Kubernetes cluster. If you already have it, you can skip this
part, but I would highly recommend following along if you're a beginner. The easiest way to bootstrap Kubernetes is
to use Minikube. It allows you to create local Kubernetes clusters
in seconds. Recently they added support for docker; now,
you don't need to install a virtual box or any other hypervisors. They also have very friendly documentation
that you can follow along to install minikube. You can select any supported operating system
and architecture and get installation instructions. I use a mac with a new apple silicon which
is based on arm64 processor. Homebrew is a package manager for mac. You can copy command and run it in the terminal. It'll take a couple of minutes to install. I already have it installed. As I previously mentioned, the best way to
bootstrap Kubernetes is to use a docker driver. You need to install docker as well and make
sure that it's running. On a mac, you can simply search for docker
and start desktop docker application. Wait a few seconds and run docker again. To start minikube, you can simply run miniukube
start. But I would suggest using the latest supported
version. In my case, it's 1.26.1. If you run minikube without this flag, it'll
tell you what is the latest supported version. Also, I don't like to rely on defaults since
they tend to change from version to version. I want to explicitly specify the docker driver. This will allow minikube to create Kubernetes
nodes as docker containers. If you choose VirtualBox, for example, minikube
will spin up virtual machines as Kubernetes nodes. When you run it, it will pull docker images
and bootstrap the Kubernetes cluster locally and also configure kubectl to talk to your
new cluster. Make sure to install kubectl before you run
this command. To verify that you can access the Kubernetes
cluster, you can get Kubernetes nodes. We have a single Kubernetes node which is
also a control plane. Usually, you cannot run pods on the master,
but minikube removes the taints from the control plane, which prevents the scheduling of new
pods. Nowadays, the most common way to install open-source
projects to your Kubernetes clusters is to use provided helm charts. If you don't want to use helm, you can run
helm template command to generate yaml and apply it in that way. Let's go ahead and add argocd helm chart. Every time you add a new repo, you want to
update the index. Let's search for the argocd chart. We're going to use the latest version at the
moment, which is 3.35.4. I would recommend that you use the same version
and upgrade only after you go through this tutorial. Most of the time, we want to override at least
a few default variables. To get defaults, you can run helm show values
and specify the path to save this file. If you open it, there are a lot of stuff you
can find that you want to override. For example, argocd version, affinity, etc. Now, I'll show you how to install helm directly
and with terraform. I like terraform because it allows you to
declaratively define in the code what you want to install and where. And it's easier than you may think to actually
use terraform. Create terraform folder. First of all, we need to define a provider. It allows you to authenticate with Kubernetes
and provides resources that you can use to install helm charts. If you used minikube, you just need to point
to the default kube config. In case you used terraform to create EKS cluster,
for example, you can dynamically obtain a token to authenticate with a cluster. In the next file, we'll install argocd helm
chart. In case you don't want to use terraform, you
can just copy this command and run it in the terminal. But wait until we create values.yaml file
to override some variables. To install the helm chart, we use helm_release
terraform resource. The second argument is an arbitrary variable,
which you can call wherever you want, but you should follow the same snake case naming
conventions. The name property is a helm release name. I typically give it the same name as an application
that I'm deploying. Next, you need to point to the helm chart
repository. Then the chart name. In the case of terraform, you don't need to
prepend the repo name, just a chart name. Then, specify the namespace. You should use default argocd namespace for
the deployment. Otherwise, you would need to override some
variables in the chart. Then create a namespace if it does not exist. Version of the helm chart. Use the same one for now. There are a couple of ways to override variables. One way is to use set statements and to target
individual values. Or just create values.yaml file and specify
the path to that file, which is my preferred method. When you need to set up tolerations and affinity,
it's a lot of set statements, and it becomes not readable. Now let's create a values.yaml file. First of all, I want to use the latest argocd
versions. Then I don't want argocd to generate a self-cert
and redirect http to https. That's not how most people secure their endpoints. If you want to expose argocd outside, you
would use ingress and terminate https on the ingress level and then route plain http to
argocd. We'll set up the ingress later on in the tutorial. Before you can apply terraform, you need to
initialize it. It will download all providers and initialize
a local state. Then to deploy the helm chart, run terraform
apply. It will take a couple of minutes to install
it. In case it takes longer, you can open another
terminal window and run helm status argocd in argocd namespace. It will give you an error. Or you can install argocd helm chart without
terraform but make sure to clean up first. Try to get failed charts with helm list --pending
-A. Now let's verify that argocd is successfully
installed. Run kubectl get pods and make sure that all
pods are in a running state and not in a crash loop or pending. By default, this helm chart will generate
an admin password and store it in the Kubernetes secret, which is called initial admin secret
and is used only once during the starup. You can change it if you wish. To get the password let's get that secret
in yaml format. It will be encoded in base64. To deccode the secret, you can use echo and
pipe it to the base64 utility. The percent sign indicates the end of the
string; don't copy it. Now to access argocd, we can use port-forward
command. The username is admin, and the password is
the content of the Kubernetes secret. Now we're ready to create our first CD pipeline
using GitOps. In this part, we'll create a public github
repo and docker images just because it's easier to get started. Later we'll use private repos and images. First of all, let's go ahead and create a
public github repository. You can call it whatever you want. It will contain the Kubernetes yaml files
and helm charts. By modifying the content of this git repo,
we'll operate our Kubernetes cluster; that's why it's called GitOps. It should be a public repo, and also add a
readme file that we can clone it. Now let's copy the clone command. You can put it in any place on your workstation. I typically use devel folder. We'll come back to it later. We also need a docker hub account to store
our docker images. If you already have it, you can just sign
in, or you can register for free. A free docker hub account only allows you
to store a single private image and an unlimited number of public images; that's what we actually
want for now. You can create a repository first, but it's
optional and only required when you create a private image. To push a new public docker image, we don't
need to have a repo first. To simulate a CD pipeline, we need some kind
of image to play with. Let's search and pull nginx public docker
image. I'll use the latest version from the mainline,
which is 1.23.3. You can use any of them; it does not matter. To push images later, we need to authenticate
with the docker hub. Now let's pull the image. So far, we have a single image. To simulate CD pipeline, we would need to
increment image tags to deploy new versions. Let's tag this open-source image using your
personal username from the docker hub account. In my case, it's aputra. Now we have two images with exactly the same
image id, meaning they are identical. Use the docker push command to upload that
image to your personal docker hub account by using your own username. You can verify that you have uploaded your
image by going to my profile. As you can see, I have a single docker tag
for now, v0.1.0. Alright, next, we need to create Kubernetes
deployment for that new docker image. Let's go ahead and open public github repo
that we created earlier. Create a my-app folder. Now it is possible to manage namespaces and
metadata such as namespace labels and annotations using argocd itself, but I prefer to create
namespaces explicitly. Let's call it prod. Next is deployment. As I said, it's going to be based on the open-source
nginx image that we uploaded to the docker hub. Replace this image with yours. The rest of it does not really matter for
now. Since we are using GitOps, we need to add
this change to the git tree and commit the change with some meaningful message describing
your action. In this case, I want to deploy a new app to
Kubernetes. It will show up in the argocd UI. And finally, push the changes to the remote
github repo. Alright, we have the namespace and a deployment. But it's not all; we need to tell argocd to
watch this particular github repo and my-app path. Let's go back to the original workspace where
we deployed argocd. Create the first example folder. There are multiple ways to deploy apps; we'll
start with the basics and create application custom resource first. This Application kind is provided by the CRDs
that are created along with the rest of the deployments in the helm chart. Give it a name for the application, for example,
my-app. Next is a namespace to create an Application
object, don't confuse it with a target namespace for deployment. It is always must be the same argocd. Then the spec. Let's use a default project. Projects provide a logical grouping of applications,
which is useful when Argo CD is used by multiple teams. Then the source of the deployment objects. We need to point it to our public github repo
that we created. At this time, you can use https or ssh; it
does not matter since it's a public repo, but I would recommend using .git suffix since
argocd will not follow redirects. TargetRevesion. The head points to your main branch's latest
commit. You can use git branches, git tags, or even
primitive regex expressions. In the case of the helm, it should point to
the chart version. We'll touch on it later. Then the path to track inside your github
repo; if you followed along, you should have exactly the same path. The destination is useful when you use a single
argocd instance to deploy applications to multiple clusters. In our case, we'll deploy the app to the same
Kubernetes cluster where argocd is running, so we need to provide a path to the local
Kubernetes api server. That's the bare minimum that we need to get
started with argocd. We'll add more later. Now to instruct argocd to track our repo,
we need to manually, for now, apply that application. You can try to find the nginx in Kubernetes,
but it looks like it's not deployed yet. Alright, we have the new my-app argocd application,
but it has a warning that it's out of sync. By default, argocd will automatically refresh
and compare the state of Kubernetes and git, but it will not apply it. This is a default strategy for argocd and
can be useful in production environments if you want to be very careful. I don't like manual steps, so I tend to configure
it to automatically sync up the git state with Kubernetes. To actually deploy the app, we need to click
sync. For now, keep all defaults and synchronize. Now it's working, and argocd is deploying
nginx to Kubernetes. Here we have the argocd application resource,
then the namespace, deployment object, and replication set, which is managed by deployment,
and finally, the pod itself. And if you go back to the terminal, you'll
find a pod running in the prod namespace. Also, you'll notice that application is now
green, which means the git state matches the Kubernetes state. When I say the state, I mean all different
Kubernetes deployments and their configurations, including replica count, selectors, and others. Now let's simulate the CI/CD pipeline and
release a new version of our app. It will be using exactly the same default
nginx image, but this time we'll increment a tag to simulate the application upgrade. And don't forget to push that image to the
docker hub. Let's bring back the public GitHub repo that
we use for GitOps. And for now, manually increment the image
tag to v0.1.1. Commit the changes and push them to the remote
repo. Later we'll automate these steps in the script
that can be used by the build agents such as Jenkins, github actions, etc. Without setting up a webhook, it may take
up to 5 minutes for argocd to detect the change in the git. In the case of GitHub, if you reduce the time
for a refresh, for example, to 1 minute or less, Github has a rate limit and simply start
rejecting argocd api calls. Also, to set up a webhook means you need to
expose argocd to the internet, which is not feasible for most companies. If you host your own git, such as GitLabs,
inside your environment, you can totally set up a webhook and speed up the process. Now we can wait or just click refresh. Argocd detected the drift and now shows that
it's out of sync. With the default strategy, we're forced to
synchronize the state manually every time. Let's click sync. Argocd reapplied the deployment and released
a new version to our Kubernetes cluster. Next, let's remove that manual step that requires
us to synchronize every time manually. Go back to the application resource and add
syncPolicy. It's going to be automated. Set prune to true, enable self heal. It would allow synchronizing state if the
Kubernetes state was changed for some reason, maybe a manual change on the cluster by someone. Allow empty disables deleting all application
resources during automatic syncing. You can adjust those settings based on your
needs. Then we can modify sync behavior. Enables kubernetes validation; for example,
if the deployment has an unknown field, it will be rejected by argocd, which is actually
the default. Create a namespace if it does not exist, similar
to helm charts. I like to create namespaces explicitly. Propagation. And prune last. Apply the changes to the application resource. To verify that automatic sync works, let's
release a new version, 1.2. And push it to the docker hub. Make the appropriate change in the GitOps
repo by incrementing the git tag. Add the commit and push it as well to GitHub. This time I'll wait 5 minutes for argocd to
refresh the git state. You can wait or click refresh. Now argocd automatically release a new version
to the Kubentes. You can check it in the terminal as well. The typical workflow for CI/CD pipeline when
using argocd is the following. The build agent builds the app, releases a
new docker image, and modifies the GitOps repo by setting the image to the one it just
created. I want to show you how to create a simple
script that you can use on the build agent. Let's call it upgrade.sh. Exit on the first error that happens during
the execution of this script. Get the version tag from the script's first
argument. Usually, it matches the git tag. Print out the current version to the console. Release a new version of the docker image
using that new tag. Upload the docker image to the remote repository. Create a temporary folder where we are going
to clone the GitOps repo. Clone the repo. There are multiple ways to modify the content
of the deployments; the simplest one and most reliable is to use the built-in to Unix systems
sed tool. It traverses the file and replaces the tag. You'll need to replace it with your docker
hub username. Later I'll show you other strategies. Finally, commit the changes to the GitOps
repo and push it back to GitHub. Also, you want to delete that temporary folder
that we created for the GitOps repo. Make this script executable. Let's see what we have right now in the GitOps
repo. The current version is 1.2. To execute the script, we need to provide
a new tag, for example, 1.3. Okay, let's run it. It will release a new version and commit the
new tag to the GitOps repo. In GitHub, you can see that the script committed
a new version, 1.3. Go back to the argocd UI and check the current
version. Now let's refresh, and argocd will deploy
a new 1.3 nginx version to Kubernetes. In the terminal, you can also verify that
the new app was deployed. Now let me show you what happens when you
delete the application resource. It looks like argocd removed it from the UI,
but the app is still running. I actually want to delete the Kubernetes app
as well when I delete the application resource. To do that, we need to add a finalizer to
the metadata. It will force argocd to delete the Kubernetes
app first and then removes it from the argocd state. Let's create that application again in argocd. Okay, it's here. Now let's remove it and see what happens. Alright, it works as expected; it removed
it from Kubernetes and then from argocd, which is what we want most of the time. Especially helpful when we use App Of Apps
Pattern that we'll talk about later. When you have a lot of applications that you
want to deploy to Kubernetes, you don't want to create them manually. The most common approach that people use is
the app of apps pattern when you manage the creation and deletion of the apps using the
same GitOps repository. Let's go back to the GitOps repo and update
the namespace to foo for the first application. Next, clone this application and rename it
to the second-app. For this app, let's use bar namespace. You could, of course, to deploy them to the
same namespace as well and call it staging. Now, let's create a folder structure that
will allow us to manage multiple apps and environments from a single repository. Create a new staging folder and move those
apps to that environment. To deploy these applications automatically
to Kubernetes, create another folder called apps under staging. In this folder, we'll put all the applications
that we want to deploy in a staging environment. Here we'll create application resources; before,
we had to apply them manually using kubectl one by one, and now argocd will manage them
for us. This application will manage my-app, and don't
forget to include finalizers if you want to delete all of them using argocd as well. And create another application resource for
the second app. It's identical to the previous except it uses
a different path. As you can see, we have Kubernetes deployment
files and corresponding argocd application resources to register them with argocd. The workflow for helm and kustomize is similar,
except that you target helm charts. As always we need to add all those changes
to the git, commit and push to the GitHub. You can visit GitHub to verify that you have
uploaded all the required files. Now let's switch back to the main project
with examples. Let's go ahead and create a second example. This application resource will target apps
path and apply all of them on our behalf. If you open argocd right now, you should not
have any apps. Let's go ahead and apply the second example. Immediately you can notice that argocd will
create apps-staging first and then deploy my-app and the second app. You can verify that pods in the foo and bar
namespace are running. It's also much easier to delete all those
apps. Now you just need to delete the main apps
application. Argocd will remove all child apps and then
apps-staging itself. Since we have finalizers on each app, argocd
also removed them from Kubetnes. You can use this pattern for helm charts and
kustomize deployments. In the following section, I'll show you how
to use private git repositories and private docker images. Let's create a private docker repository. Let's call it nginx-private. It's going to be a slightly different approach
in AWS which I'll go through at the end of this tutorial. On the other hand, managing private git repositories
will be the same among all clouds. Let's create a private github repo and call
it lesson-158-private. First of all, we need to release a new docker
image and push it to the private repository. The push process is exactly the same as with
public images. Next, we need to clone the private github
repo. Now you could follow the same folder structure
with environments, but for simplicity, I'll keep it flat. Create a my-app folder. Then the namespace. And a deployment object. Here we need to use our private docker image. That's all for now for this GitOps repo. Let's commit the changes and push them to
GitHub. Create a new example 3 folder. And define the application resource. The main difference here is that we are not
using https anymore and also specifying a new private github repo. The rest of the application is the same. Let's try it out; apply the application. Let's see in the argocd ui if the application
was deployed. It looks like we have an issue. It's maybe not obvious from this message,
but it means argocd does not have access to the private github repository. Before we try to fix it, let's delete this
application. There are multiple ways to authenticate with
private repositories, including personal tokens and github apps. But the best option for CI/CD pipelines, in
my view, is to use ssh keys. Let's go ahead and generate an ssh key for
the argocd. We'll use an elliptic curve. On older machines and maybe windows, you still
need to use the rsa keys by using the following command. Step number one is to upload the public ssh
key to the github repository. Go to settings and deploy the new key. This key will grant access to only a single
repository. Call it argocd and paste the key. Now for the typical argocd workflow, we need
only read access. Later when we start using the image updater,
we'll update it to read and write permissions. The next step is to create a secret with the
private key in kubernetes and link it with that private repository. You must specify the same github repo that
you use in the application resource. Now let's copy the private key and paste it
into the secret. To test it, you need to create that secret
first; otherwise, you may have the same error, but if you refresh argocd, it will go away. Then apply the same application. This time it looks like argocd was able to
pull the github repo, but we have an issue with the pod. It's not running, and we have errors. I think you already know why. Let's get the pods in the foo namespace. And we have an error imagePullBackOff. If you describe that pod, you'll find that
minikube does not have permission to pull that image. It's not directly related to argocd, but we
still need to fix it. To grant access to minukube, we can either
use a personal password, but the better approach is to generate a read-only token. Go to security in the docker hub and generate
the token. The last step is to create Kubernetes secret
of type docker-registry in the same namespace where you deploy your application. I know it's not convenient to create the same
secret in all namespaces where you deploy your private image. Later in AWS, I'll show you another way. Use your token to create a secret. In the GitOps repo, we also need to explicitly
provide that secret to the deployment object. Just add the imagePullSecrets section with
the name of the secret. Commit the changes and push. Now in Argocd, you can wait a few minutes
or click refresh. Now it seems to work. Let's also verify in Kubernetes that the pod
is running. Alright, that's all for this section; let's
clean up and delete the deployment. In the next section, I want to show you how
to deploy Helm charts. As an example, we'll take metrics-server open-source
helm chart. Typically it is used for horizontal pod autoscaling
and also can be used directly by running the kubectl top command. As with any Helm charts, you need to know
what kind of variables you can override. The easiest and most accurate way is to use
helm show command with a specific version. Let's go ahead and add metrics-server helm
repo. Search for the metrics server. Keep a note of the current version. Next, save the default values. You can open that file and find all possible
variables that you may wish to override. Now we're ready for the deployment. Create a new example 4 folder with an application
resource. Specify the same helm repo under repoURL and
provide the helm version that you want to install. Chart name. Then you have the option to select a version
of the helm; most of the time, you want to use the latest v3 version. Release name for the chart. Since this chart is open to anyone, we don't
need to provide any credentials to access it. Now there are a few ways that you can use
to override default variables. One of them is to target individual values. Which is fine when you need to override something
like an image name but with tolerations and affinity, it becomes a mess. The following approach that I like more is
to provide a yaml with all variables that you want to override. When using minikube, you need to pass insecure-tls
flag to the metrics server. Otherwise, it won't work. Then the same destination section. By default, argocd will deploy this helm chart
o the argocd namespace. If you want to customize the namespace, you
can provide it here. You can find an example in my github repo. And the same sync policy. Now if you provide a custom namespace under
destination and it does not exist, you need to flip create namespace to true. That's pretty much all; let's go ahead and
apply this example. In the UI, you can see a bunch of resources
created by the hem chart. To verify that metrics-server is running,
let's get the CPU and memory usage of the pods in the kube-system namespace. You can also get the usage of the nodes. That's all for this example. Let's uninstall that chart. Another common and my prefered approch to
customize deployment objects is to use kustomize. It helps to keep your code dry and only override
parts of the application that are specific to that environment. Let's create my-app-base folder. Create a namespace. It can be anything; we will override it in
each environment. Then the deployment object. We don't need to specify a tag or a namespace
since it will be populated by kustomize. And you also need to include kustomize.yaml
file in the base folder, and specify what files can be customized. I have a full tutorial on how to use kustomize
that you can watch after we finish this example. Next, let's create the same folder structure
to hold different environments. I created a dev environment by I meant to
create staging. And then my-app folder. Here we only need to create a single kustomize
file and override some properties. Let's change the default namespace to staging. Then the image. This image tag will be updated from the build
agent, such as Jenkins, with a single kustomize command. And you also must specify the path to the
base folder that contains my-app. Let's commit all the changes and push them
to the remote. Now create a new example 5 folder with the
application resource. You can totally use the same app of apps pattern
and commit all those applications to the gitops repo. Since I deleted the previous secret to access
private repo, I need to copy it here from the previous example. From the application perspective, it looks
similar to how we deploy plain yaml files. You could kustomize some of the parts from
here, but I recommend using the gitops repo. Let's apply it. It created all the resources, including staging
namespace and deployment. Verify in the terminal that the pod is running. Looks good; let's tear it down. In the next video, we'll explore a slightly
different approach to continuous delivery using Argocd. In the following video, we'll use an image
updater to decouple continuous integration from continuous delivery.
