Kubernetes Daemonsets explained | Tutorial | Kubernetes Guide

kubernetes daemon sets what are they in kubernetes it all starts with a pod a pod can be a micro service a proxy a web service an application a cron job or a database now kubernetes provides different apis for running pods and this depends on our requirements for running web servers proxies or websites we can use what's called a deployment for databases we use stateful sets for jobs and scheduled tasks we use cron jobs so what is it damon said then in kubernetes we have nodes nodes are machines physical or virtual where pods can run as containers a diamond set allows us to ensure a pod runs on every node when we scale the cluster up the daemon set creates a new pod on that new node we can also have a daemon set run pods across multiple node groups or we can have one diamond set per group of nodes like one for windows nodes and one for linux nodes example use cases for daemon set pods are provisioning storage on each node deploying a monitoring agent per node or a log collector on each node in today's video we'll be taking a look at daemon sets in action we'll take a look at what we can do with the daemon set a couple of examples we'll deploy one and we'll take a look at some real world examples for how we can use diamond sets in the real world so without further ado let's go if we take a look at my get up repo at the roots i have a kubernetes folder and in the kubernetes folder i have a diamond set folder with a readme and in this readme is all the simple steps i'm going to be showing you guys today so be sure to check out the link down below to the source code so you can follow along now to demonstrate diamond sets with the first step is we're going to need a kubernetes cluster now for this i'd like to use a utility called kind which is a lightweight kubernetes cluster that i can run inside of a docker container for testing purposes now to run a daemon set in a kubernetes cluster we're going to need a cluster with multiple nodes so taking a look at the kubernetes daemon set folder i have a kind dot yaml and here's my cluster specification i'm going to run a cluster with one control plane node and three worker nodes so to create this cluster the first thing i'm going to do is change directory to the kubernetes daemon sets folder and then i'm going to say kind create cluster i'm going to call my cluster daemon sets i'm going to run kubernetes 1.20 and i'm going to pass in my kind.yaml file so i go ahead and copy paste that to the terminal and that'll create a lightweight kind kubernetes cluster in docker we can use for testing now once this command is complete we can test our cluster by saying cube ctl get nodes and that'll print out our control plane node as well as our three worker nodes so now we have a kubernetes cluster ready to go so the first thing we're going to be doing is taking a look at a basic domain set and how to run a pod on each node so if we take a look at the kubernetes daemon sets folder i have a daemonset.yaml and in here is a yaml file for what a basic daemon set looks like so we say kind damon said we can pass metadata such as a name and a namespace and labels so i'm going to call this one example daemon set and i'm going to deploy it to the default namespace and similar to a deployment in a stateful set we have a selector to select pods so here i'm just going to match by label and the label is going to be name example daemon set and then we have the pod spec and for this damon set i'm going to add a basic toleration which allows us to run the daemon set on master nodes if you need to but for that we just say key and the key we pass it is node roll kubernetes.i o master and the effect is no schedule if you don't want the pod to be running on masternodes you can just remove this toleration and then we're going to have a basic container and this is the same as a deployment pod spec so we have container we can give it a name we can give it an image i'm just going to run alpine over here we can pass in environment variables we can put volume mounts and secrets and things like that as well i'm just going to create an environment variable called node name and i'm going to use the kubernetes downwards api to get the node name and pass it into the environment variable then as a command for this container i'm just going to run a bash script i'm just going to say echo i'm going to say hello i am running on and then print out the node name and then sleep but with this we'll be able to get the logs of the pod and we'll see the pod running on each of the nodes i can also pass in resource limits and requests and a termination grace period now to deploy the daemon set it's very simple i just say cube ct i'll apply minus f and i'm going to apply it into the default namespace and then to see the pods running on each of the nodes i can do cube ctrl get pods and i can pass the o y black in and that will show us which node it's running on as well as the ip addresses we can see we're creating a pod on each of the nodes including the control plane master node now once the pods are up we can see they're running on each of the nodes we can see the ip addresses as well and to get the logs out of each pod we can say cube ctrl logs and the name of the pod and if i run that we can see it says hello i'm running on daemon set worker 2 so that is coming from this pod over here so you can see it's very simple to forcefully run a pod on each node using a diamond set now to clean up that diamond set all i need to do is say cube ctl delete diamond set or ds for short and the name of the daemon set so i go ahead and run that that'll delete the daemon set which will go ahead and delete all the pods from each of the nodes if we do cube ctrl get pods we can see they're all terminate at the same time now we've taken a look at a very simple example but more complex examples and typical use cases are running a cluster storage daemon on every node for provisioning storage running a log collector on every node to collect logs from each pod and running a node monitoring daemon on each of the nodes something like node exporter for prometheus we've also taken a look at how to write a simple daemon set spec so they go through the yaml file over here and it's also good to know how diamond set pods are scheduled damon said pods are scheduled slightly different to deployments and stateful sets and this is because with daemon sets we want to ensure that there's a pod on every node and sometimes these parts can be critical for monitoring log collection and alerting now normally with deployments and stateful sets the pods that the nodes run on is selected by the scheduler with daemon sets it's a little different and this is because daemon said pods have different requirements and different needs if we take a look at the taints and tolerations we can see daemon said pods will not be evicted when there are node problems such as network partition damon said parts tolerate disk pressure daemon says odds also tolerate memory pressure daman said pods also tolerate unscheduleable attributes by the default scheduler so we're a pod from a deployment or a stateful set may not be scheduled kubernetes may still go ahead and deploy the pod for a daemon set because daemon said pods might be more critical especially if you're using it for monitoring agents or log collection or storage provisioning the daemon sets may be scheduled regardless now another important aspect of daemon sets is how to connect to daemon set pods and how do we communicate with them let's say each of these pods are running a web server for example an endpoint that we can use to collect node metrics how do we talk to each of these nodes individually or if we're running web applications or proxies how do we load balance between these pods how does network communication work in general there are three ways to communicate with daemon set pods first wave is by using a cluster ip service which allows us to select diamond set pods using a label selector we can reach the dave and set pods using the service name and port kubernetes will load balance each request it to a pod at round robin fashion for public access the service can be of type load balancer the second way is by using host port if we know the ip address of each node we can set the host port in the pod spec and reach the pod by node ip and host port the third way is by using a headless service which gives each pod its own dns and allows us to reach each individual pod so in the kubernetes documentation they talk about how to communicate with daemon set pods so firstly you can avoid communication altogether if you're able to push data out from the pods so the first example is using a push method where you don't have any clients but you can just push data out of the pod this is not always possible the other three ways of communicating is by node ip and known ports we can use a dns by using a headless service as well as a kubernetes service so let's take a look at some examples so first example is by using a basic daemon set and having an http endpoint on each of the pods so for that i've created another yaml file called daemon set dash communication and if we take a look at that one it's a basic daemon set as we saw earlier but this time we're going to be running an nginx web server and expose an endpoint and a port kind as damon said i've got my basic metadata i'm going to call this one daemon set communicate i have a normal label selector and labels on each of the pods and here is my pod spec so again i have the toleration defined that i want to deploy this on the master node as well and i have an init container as well as a container let's take a look at the container first the name is called daemon set communicate i'm running an alpine flavor of engine x and i'm creating a small volume mount and this is just the volume out called nginx page and it's going to mount the default location where we can serve static html files from which is user share nginx html i then give it some basic resource limits and request values i expose a port which is port 80 i give it a termination grace period of 30 seconds and i define a volume called nginx page which is an empty directory so i'm mounting an empty folder into the location user share nginx html so there's no html page here this is where my init container comes in so before the pod starts what i'm going to do is run a small init container called create file that's gonna generate a small file inside of this user share nginx html directory so let's see how it does that if i go ahead and expand this i'm running an image called alpine an environment variable similar to before where i get the node name and i store that in an environment variable called node name and then i run a small shell script exactly like the one i did before where i say echo hello from node name but this time i pass that string to a file called index.html i'm basically generating a static html file on each of the pods so we'll be able to hit that endpoint in the browser and we'll be able to see the index.html page from each of these daemon set pods through http so to deploy this example i'm just going to say cube ctl apply minus f and i'm going to pass in the daemon set communication yaml file we create that and then i say cube ctrl get pods we can see each of the pods are creating the init container will run first create our static html page in that empty folder and nginx will start up and serve that file and now when we give it a couple of seconds we can see we've got a pod running on each of the nodes so now that we have a web server pod running on each of the nodes how do we reach and communicate with this pod this is where kubernetes services come in kubernetes services is a feature in kubernetes that allows us to perform communication between pods in a cluster if you're not familiar with kubernetes services check out my link down below to my kubernetes development guide where i covered deployments services config maps secrets and more now the first method i'm going to show you is how to communicate with the daemon set pods using a cluster ip service which is a private service within kubernetes and this will give us round robin load balancing over a private endpoint so to communicate with our daemon set from within the kubernetes cluster i'm going to deploy a pod to kubernetes that we can use to talk to our daemon set so for that in the daemon sets folder i have a pod.yaml and this is just a basic pod so you can see i'm running an alpine container and all i'm doing is installing bash and curl and then i say sleep for 60 minutes so this will give us a private part running in kubernetes that we can ssh to and we can test our communication so to deploy this part i'm just going to say cube ct i'll apply minus f pod.yaml and this will create that pod we can see that part is now being created and we can see it's now up and running now to access the pod it's very simple i can say cube ctrl exec minus it name of the pod which is called pod and i can pass in bash this will give me a bash terminal and i'm now inside of that pod and i can talk to my daemon set pods now before i can talk to my daemon set pods i need to expose a service so let's take a look at our github repo in the kubernetes folder in diamond sets folder i have a services folder with a couple of examples the first example is a cluster ip service which is a private service that allows us to load balance between pods in a cluster so take a look at cluster ip service and here i'm just going to create a service we say kind a service and the name is called daemon set service cluster ip and this is the important part we say type is cluster ip and then the other important part is we also have to use a selector to select the pods that are part of the server so because all our daemon set pods have a label called name where name is daemon set communicate we have this label on all pods and because this is part of the selector here kubernetes will look for each of the pods that has this label and that will become part of the back end of the service and then what we do is we expose a port so we say protocol tcp the port name is http and the port on this service is going to be port 80 and the target port on the pod is also 80. so to deploy the service what i'm going to do is just jump into a new terminal and then i'm going to change directory to the kubernetes daemon sets folder and then i'm going to say cubect i'll apply minus f services and i'm going to apply the cluster ipservice.yaml that'll create a new service i can then say cubectl getservice and we can see we have our service up and running notice that it doesn't have an external ip because it's a cluster ip service meaning it's private to test out the service to make sure it works i can say cube ctl describe service and the name of the service and notice we have a bunch of endpoints here you need to make sure that these endpoints exist otherwise your selector is probably wrong and you're not matching any pods on the back end so these are the ip addresses of each of the daemon set pods and notice that this service also has a private ip address and internally we can use the name of the service as the dns for the service so now that we have a service we can access all our damn set pods from within the cluster and to do that i'm going to jump back to my pod and what i'm going to do is run the simple while loop i'm going to say while true do curl and notice this is the service name so i'm going to call http the name of the service on port 80 and then i'm going to say sleep for one second and run this in a loop if i go ahead and paste this we can see that it's now contacting the service and load balancing between the pods so you can see this is round robin so this is how you would internally load balance pods if you wanted to expose this publicly you can use type load balancer so instead of saying cluster ip service and saying type as cluster ip you would say type is load balancer and when you apply this service kubernetes will give you a load balancer depending on the cloud provider you're running on if you're running an azure or aws you will get an azure load balancer or an elb if you're running on-prem kubernetes you'll have to make sure that whatever software you use to bootstrap your kubernetes cluster that you have load balancing capability you can also take a look at metal lb which is an open source load balancer now both the cluster ip as well as the load balancer type services will round robin between pods this is good for load balancing but it's not good for reaching specific pods so to reach a specific pod on a specific node we need to know the node ip where that part is running and then we can also provide a host port so we can use the node ip and host port to talk to specific pod on that node so we've covered services and this brings us now to node ip and known port so if we know the node ips we can use the node ip and a known port to talk to that pod in our daemon set so to do that if we take a look at the port specification we can add this thing called host port and pass a port on the host that we can expose so what i'm going to do is i'm going to jump back into my daemon set communication yaml file and i'm going to go down to the container section where i have the port exposed and what i'm going to do in here is pass in host port and pass in port 80. and then i can go ahead and apply that change by saying cube ct i'll apply minus f and that'll configure our daemon set we can see when we run cube ctrl get pods that kubernetes will terminate the pod before starting a new one and it'll do one node at a time so if we run cube ct i'll get pods again we can see it's now running on a second node it's doing one of them at a time until all of them are up and running so now that that change is rolled out we can now use the node eyepiece to talk to our daemon set pods so to get the node eyepiece i can say cube ctrl get nodes o wide and we can see we have all the internal ips of our nodes here and if i go ahead and grab one of them like o6 i can go back into my pod and i can say curl the node ip and the port and we can see we actually hit worker 2. i can change that ip address to another node and we hit worker change it to ip address 4 and we can hit worker 3. so this is a basic way of using the node ips to talk to our daemon set pot so this can be useful for exposing daemon set pods on each of your nodes on the cloud network level where you might have some other utilities that are running on the same network as your cloud virtual machines and they might know the virtual machine ip addresses so they can talk to these pods directly so we've covered load balancing as well as connecting via node ips what if you wanted to address each of the pods individually usually this is required for things like database clusters where databases have multiple instances that need to fear and form a cluster stateful sets uses this concept often and it's a service called headless service when we run a headless service kubernetes gives each of our pods its own dns without load balancing so we can address each part individually now to define a headless service all you need to do is go to a service and change the cluster ip equal to none so i have an example of this in my kubernetes daemon set folder under services i have a headless service defined so you can see here i say kinda service i give it a name called daemon set service headless cluster ip is equal to none and i also pass in a selector that's going to select all the pods i also give it a port as well as a target port and to deploy the headless surface i just say cube ct i apply minus f and i apply that yaml file and that will go ahead and create our headless service now that we have a headless service each one of the pods will get its own dns address it's also important to know that there are multiple ways to discover the dns for each of the pods in this example i'm going to show you two ways to do so one way to do so is to resolve and query the dns of the headless service which will respond with the endpoints of each of our pods or an alternative is if you know the private ips for each pod you can construct a fully qualified domain name for each of the pods so the first example is querying the dns of the headless server so you can see headless services without cluster ip are also assigned dns a and a records depending on the ib family of the service and it's generally in the form of service.the namespace.service.cluster domain.example so unlike normal services this resolves to the set of ips of the pods selected by the server so if we go and query this dns it will resolve the set of ips behind that service so to resolve the ip addresses for this headless service i'm going to install a utility called dig and i'm doing this inside of that private pod that we have in kubernetes and then what i'm going to do is i'm going to run the dig command on daemon set service headless dot default because it's in the default namespace dot service.cluster.local and if i run this notice in the answer section it resolves with the dns for each of the pods so we can see here we have the ip addresses for each of the pods behind it we can then use these ip addresses to call and address each of the pods individually another mechanism to resolve the ip addresses for each of the pods is to call the kubernetes api so we can say cube ctl describe endpoints for that headless service so if i run this we can now see all the addresses that comes back so this is another way of getting the ip addresses for each of the pods so we can call them individually so using both those two methods either using the kubernetes api or just resolving the dns we can get the ip address of the pods and then what we can do is we can get the a records for each of the pods by using the following formats so this is the podip address dot namespace.pod.clusterdomain.example. the example of this you can see i have public ip but instead of dots in the ip address i have dashes and then i have dot default because we're in the default namespace dot pod.cluster.local so i can go ahead and grab this first ip address i can copy it i can hop back into my container i can say curl http pass the ip address so i can either call the pod by its ip address or i can replace the dots with dashes and at the end i can say default.pod.cluster.local so this would give me a dns record that i can use instead of an ip address so that is basic network communication and how to talk to daemon set pods now what does this look like in the real world now if you're new to kubernetes and you want to see diamond sets and monitoring in action check out my link below to my prometheus kubernetes monitoring guide where i go through all the prometheus components in detail as well as node exporter which we're going to be taking a look at in this demo node exporter is a daemon set that will run a part on each node and provide metrics this is metrics like cpu memory network disk pressure and more now let's go ahead and deploy prometheus to our cluster deploy the daemon set for node exporter and see how prometheus communicates with each pod on each of the nodes now i'm going to run through this very quickly to show you a daemon set in action so what i'm going to do is deploy the prometheus operator stack to our cluster using the community repository called prometheus operator cube prometheus this provides us with all the components to monitor a kubernetes cluster with every version of kubernetes and the different release for every version so since i'm running kubernetes 1.20 i'm going to deploy release08 so what i'm going to do is i'm going to clone that repository locally saying git clone and the address of the git repo i'm going to change directory to that repo and then say git check out and check out that tag and this repo provides quick start instructions on how to deploy it but it's very straightforward i change directory to the manifests folder and then i apply all the manifests in the setup directory go ahead and run that that'll create the monitoring namespace as well as all the custom resource definitions and deployments for the prometheus operator if we take a look at the pods in that monitoring namespace we can see the prometheus operator is being created and give it a couple of seconds we can see it's now up and running and now i'm going to go ahead and apply all the rest of the components in the manifest directory saying cube ctrl create minus f and saying dot that'll go ahead and run the full prometheus stack inside of that monitoring namespace it'll give us a grafana dashboard it'll also deploy the node exporter daemon sets and a prometheus instance that'll scrape each of the daemon set pods and what we're interested in is see how these pods are running and how prometheus talks to them so if we do cube ctrl get pods in the monitoring namespace we can see we have alert manager we have cube state metrics grafana node exporter pods we have prometheus adapter and two prometheus instances as well as the operator running and the part that we're the most interested in is the daemon set so i'm going to say cube ctrl get ds in the monitoring namespace and we can see we have the node exported daemon set running and it's got four pods running at the moment if we do cube ctrl get pods in the monitoring namespace dash o wide we can see that we have the node exporter parts being created and they're running one pod on each of the nodes now after giving this a couple of minutes we can see now that we have all the pods up and running so it's all good to go so what i'm gonna do now is port forward to the prometheus instance and see how it connects to the daemon set so to port forward and access prometheus i'm going to say cubectl in the monitoring namespace port forward to the prometheus service on port 1990 and then i'm going to go to localhost 1990 in the browser and we can see we have our prometheus up and running i'm gonna head over to status and targets and this is what we're interested in we can see here that node exporter has automatically been picked up and you can see all of these pods are being picked up as well and if we do cubect i'll get nodes o wide we can see that the internal ip addresses for each of the nodes are listed here and if we go back to prometheus page we can see that the node ips are being used to talk to each of the pods behind the daemon set so prometheus in this case is using the node ip and known port method to talk to the pods on the cluster we can confirm this by going into the cube prometheus github repo that we cloned go to the manifests folder we can scroll down and look for the node exports at daemonset.yaml and we can see here we have kind as daemon sets name is node exporter it's going to the monitoring namespace and the important part that we're after if we scroll down to the port section we can see that it's exposing host port nine one double zero so prometheus just use the ip address of each of the nodes and talks to the host port defined here another example of daemon sets is blue and d fluendy uses the daemon set to run a pod on each node which runs a fluent d pod on each node and mounts in the volume of the host where every pod log is being written to this allows fluenty to pick up all the pod logs on every node in our cluster and send it to something like elasticsearch or some custom back end endpoint in my daemon set read me introduction right at the bottom i have a link to my monitoring and logging guide for fluentd so be sure to check that out if you're interested in how to run the flue indeed daemon sets so hopefully this video helps you understand what a demon set is how to run a basic demon set and define one yourself as well as giving you a couple of practical examples of how demon sets are used in the real world if you liked this video be sure to hit the like and subscribe button and hit the bell and also check out the links down below to the community page and if you want to support the channel even further be sure to hit the join button below and become a member and as always thanks for watching and until next time peace [Music]

Channel: That DevOps Guy

Views: 3,087

Rating: 5 out of 5

Keywords: devops, infrastructure, as, code, kubernetes, k8s, cloud, training, course, cloudnative, development, deployment, containers, web, services, tutorial, daemonsets, pods

Id: RGSeeN-o-kQ

Length: 26min 59sec (1619 seconds)

Published: Wed May 26 2021