Fluentd on Kubernetes: Log collection explained

[Music] so you're running all your applications and services on kubernetes and you want to seamlessly collect logs out of your application and send it to a central place where you can analyze it if you want to do this then fluentd is for you fluentd watches log files as log files are appended to a file fluentd reads it this is using the tail plugin fluendy uses a match plugin to select all the logs we want and send it anywhere like elasticsearch cloudwatch s3 or anywhere else in kubernetes we can have many servers running with many containers on them also known as pods the container host or docker stores these files on the host in a folder which it rotates from time to time to prevent the disk on the host from getting full now with multiple container pods writing logs kubernetes already keeps all these logs on the file system fluent can point to these files read it and send it elsewhere to get these logs from multiple kubernetes nodes we will run fluentd in a diamond set a diamond set allows us to run one pod on every machine in the cluster this will ensure we can grab the logs from every container on every machine in our entire cluster so today we're going to be taking a look at a kubernetes machine and where these logs are being stored we're going to use fluency to read these logs from the kubernetes node using the tail plugin and then we're gonna see how does log collection work and how do we take all of these logs and ship them out of the cluster we've got a jam-packed video today so without further ado let's go so if we go over to my github repo and we go to the monitoring and logging folder you'll see a readme and this has a reference to my logging introduction guide as well as my introduction to fluentd make sure to check out the link down below to my introduction to standardized and centralized logging as well as my introduction to fluentd but this video if we take a look at the monitoring logging fluency folder we also have a kubernetes folder with a readme and in this readme is the introduction to fluency prerequisite and all the steps that we're going to be taking a look at today so be sure to check out the link down below to the source code so you can follow along so the first thing we're going to need is a kubernetes cluster now when i do any testing i'd like to use a utility called kind which allows me to run a disposable kubernetes cluster that i can use for testing so to create a cluster all i'm going to do is say kind create cluster i'm going to call it fluentd and i'm going to run kubernetes 1.19 so if i copy that command paste it to the terminal this will go ahead and spin up a single node kubernetes cluster we can use for testing so once that's done we should have a kubernetes cluster up and running to test whether it's up i can do cube ctrl get nodes and we can see we have a one node kubernetes cluster up and running ready to go now before we start i'd highly recommend to check out the official fluentd repo for the flue indeed daemon set on the fluentd github page they have a fluentd kubernetes daemon set repo which has a lot of implementations you can see you can run fluentd as a diamond set and send logs to azure blob cloudwatch elasticsearch you can forward to other fluency instances you can use gcs greylock logley paper trails and syslog but there's a lot of implementations here as well as a docker image with the latest version of fluentd so in this video my yaml files and implementations are purely for demo purposes in this video we're going to be breaking down the yaml files and the docker files and break this entire process down so that we can simplify it and understand it so firstly to understand how fluentd is installed let's take a look at the fluency docker file so in my monitoring logging flue in the kubernetes folder i have a docker files folder with all the stuff required to build fluentd i have a docker file and this docker file describes how to get fluentd installed so in my dockerfile the first thing i say is from fluentd and i run fluendy1.11 i then go and set a few environment variables for ruby i also copy in a gem file which is basically a bunch of in the plugins for ruby and then we go ahead and install those dependencies we copy in our plugins folder into the image we copy an entry point in and we start up our entry point so this is a very basic example of running fluentd with its plugins if i take a look at the gem file over here we can see that these are all the plugins we're enabling in our fluendy image i also have a plugins folder with a few extra plugins i have a parser for kubernetes and i have a multi-line parser for kubernetes which is two ruby files here so this dockerfile will basically have fluency 1.11 and all the plugins we're gonna need to gather our logs and ship them out now it's important to understand that in the fluentd official github repo they have a docker image folder with different versions of fluency and if we go into 1.11 we can see that they have a docker file with the relevant plugins for each of these different scenarios so depending on where you want to send your logs to you may come and grab the appropriate docker file from here so the docker files i went and grabbed is in the debian elasticsearch 7 folder so you can see all these files here is what i grabbed into my github repo and the only real thing that's different is the plugins each one of these implementations will use slightly different plugins to send the logs to different places like blob cloudwatch elasticsearch and all these other areas so that's basically what the official docker image looks like and i highly recommend to use the official docker image as a source of reference if you need to build your own if you don't want to build your own you can also refer to docker hub and use the fluenty daemon set images directly from docker hub you can take a look at all the different tags that support it and you can pull and run those images instead now to get the image built it's very straightforward we can change directory to the monitoring logging flue in the kubernetes folder and in here we have the docker files folder if we change directory to that folder and we do alice we can see we have our docker file and all our files in here and then i can go ahead and build this docker file by saying docker build dot minus t and i can tag the image and then followed by pushing it i can say docker push and push it to my registry so i say docker build that'll go ahead and build the image as we can see and finally push it to my registry so i say docker push and that'll go and push it up to docker hub so now we have a custom fluency image that we can run in our kubernetes cluster to collect logs so now that we have a fluency image we want to run this image in a pod on every node in our cluster and we also want to look at how to collect the logs from each node in the cluster for doing this the configuration is key now in this readme i give a brief overview of the fluentd config map we have five config files in our config map the first config map is the fluent.conf this is our main config that includes all our other configuration files so we use the include statement to include other configuration files that we need this makes the fluent.conf very small and easy to read the second two configs are pods.fluen.conf so i have one for pods-kind.fluen.com and one called pods.fluent.com and both of these configurations are tail plug-in configurations that tells fluently to read container logs from a file on the node and the reason why i have a different one for kind is because kind cluster writes its log in a different format if you're running on aks or ecase or a cloud provider the logs will be in a different format so for that i have pods dash fluentd.conf and we'll take a look at these two in a second the next configuration i have is file-fluent.conf and this is a match block to capture the logs and write it to a local file just for testing purpose so in order to test whether our log collection is working we'll use a simple match block and collect all those files locally in the fluentd pod and once we're happy with that we'll enable the elasticsearch fluent.conf which is a match block that captures all the logs and sends it to elasticsearch this is highly customizable and you can include any type of configuration to push logs anywhere you like if you're not using elasticsearch so let's take a look at this config map so if we take a look at our monitoring logging fluentd kubernetes folder i have a fluentdashconfigmap.yaml that has those five files that i mentioned earlier so this is the fluent.conf this is the main fluency config file that fluently will load when it starts up and in this one you can see it's very short because we're just using this as a config to include other configs to make it easy to read so we can see here that we're including pods kind dot fluent.conf and we've disabled pods.fluent.com we're also enabling a file fluent.conf so we're collecting logs in a file locally and we've disabled the elastic search plugin so as i mentioned before for two examples we have two different fluency configs to read pod logs one is designed specifically for a kind cluster and the other one here is designed for flue indeed running in the cloud so if you're running fluentd in an aks or an eks cluster you would use this mechanism so let's take a look at that in more detail so we so what we're saying is we have a source block and type is tail so we're using the tail plugin to read from file so we're telling fluently to read from head so always read from the beginning of the file and we're telling fluently to tag each log with kubernetes.star and we're also giving it the path to read the logs from so on the kubernetes node the logs will be written in var log containers folder and it will have the container id followed by dot log we also give flue india position file and it and this is the file where it will record the line numbers of where it read up to so if fluency restarts it won't start reading all the logs from the beginning so it's important that we persist this file we also exclude a path so we don't read the flue indeed logs and then we're also including a specific parser to read the logs from the file and as i mentioned earlier kind cluster writes its log in a different format than a cluster in the cloud but we have a simple regex parser here to pick up the logs in this format so you can see the logs in a kind cluster starts with the time and it has the stream like standard out or standard error followed by a character and a message so to summarize this fluency configuration will tell fluentd where to get the container logs from and how to parse it and then we're going to use the kubernetes metadata plugin to add some important metadata to our logs so every log that gets read by fluentd will want things like the pod name the hostname and some kubernetes metadata so for that we have a filter block and we filter all the tags that matches the tag up here and we say type is kubernetes metadata this is the kubernetes metadata plugin that will give us metadata on each of our log entries and for this metadata plugin to work we need to provide it with a kubernetes url which you can see here we source from environment variables we also say things like verify ssl the ca files and what labels we want to skip and what this plugin will do is call the kubernetes api and get important information about the logs and attach it to each log so we'll be able to see which part is writing the logs and what name space it belongs to and what node it's running on so this is the configuration for a kind cluster if you're running kubernetes in the cloud i have a very very similar config called pods fluendy.conf and it's almost identical it has a similar source where we read from tail we read the logs from the same location we also have the same tag and we also include a position file the only thing difference here is we have a different parser so here we include a default password for kubernetes so it has type kubernetes and we give it a format that we want to write to which is defaulted to json and we also pass in a time format so this will allow this tail plugin to parse the json files on the host of each container in each pod and then exactly like the previous config we also use the kubernetes metadata plugin to add this little filter block to get important metadata like namespace pod name deployment name hostname of the kubernetes node and other important data to be attached to our log so that's for log collection once we've collected the logs we need to tell fluentd where to send it so for that i have two little config files here one is called file.fluen.com and this one is purely used for testing purposes in order to test whether these collection configurations work i create a small configuration called file.fluendy.com which matches all the logs being collected by fluentd we say type file and we simply dump it to attempt directory but this will allow us to run through indeed check this file to make sure logs are being collected and we can troubleshoot if the collection is not working once the collection is working we can disable this config and move on to a more realistic configuration like elasticsearch.fluen.com and in this one i basically say match match all the logs the type is elasticsearch so i give it a host port and an index name of where to write the logs to elasticsearch so this will be the elastic search address and the type name is fluentd so this is the minimum configuration that you will need to collect the pod locks from the container host and send it elsewhere so the first thing we need to do is create a fluency namespace so i'm just going to say cubectl creatensfluentd and then to deploy the config map all we need to do is say cubectl apply minus f and pass in the config mapped path so i go ahead and run that and that'll create a fluencyconfig map in our cluster so now that we have our config map in place let's go ahead and see what the kubernetes daemon sets for fluency looks like the diamond set will mount all these config files as well as the location where the logs are stored on the kubernetes node so in my git repo in my kubernetes folder under fluentd i have a fluency.yaml and this yaml file has been taken from the fluentdaemon set gitabrepo that i showed earlier we can see that kind as daemon set we have labels and we specify our container section where we say we want a container called fluentd we point to our image that we built earlier and here i'm going to set some environment variables because i'm using elasticsearch in this example i'm going to pass in the elasticsearch host and port address of the elasticsearch server if you're sending logs to a different location you may have to update these environment variables but if you want to send logs elsewhere make sure to check out the official fluency repo that i showcased earlier we specify some resource requests and limits and we have volume mounts that are very important so firstly we have a volume mount for our config map and the default location where fluendy reads configs is fluentd etc then we're mounting a path called var log as well as a path called var lib containers so these are two paths that are very important for reading container logs on the host if you take a look at the volumes here you'll see that both the var log as well as the var lib docker containers are both host paths so we're mounting in parts on the host into this fluency container so once we have this pod up and running i will go inside the pod and show you the file system now before we deploy our daemon set in order for fluentd to read the metadata of each pod and add that metadata to each log entry we have to deploy a flue in the rbac file and we need to give fluently access to be able to call the kubernetes api so to do that i have a fluent d r back dot yaml and here we just give it a cluster roll and roll binding and we allow it to basically get list and watch pods and namespaces in the cluster and we also create a small service account that we bind these roles to and to create the service account the cluster role and the cluster role binding i say cubect i'll apply minus f and i'm pass in the fluentdr back.yaml this will go ahead and create our service account the role and the role binding which we can use on the daemon set and if we take a look at the daemon set.yaml we're passing in specific service account name here which is fluentd so to deploy the daemon set it's really simple i say cubect i'll apply minus f and i point to the fluendy.yaml file i go ahead and run that and that'll create the daemon set and if i do cubectl in the fluendynamespace getpods we can see we now have a fluentd pod up and running so now that we have fluentd running let's deploy a small example application that writes logs that we can collect so to demonstrate that i have a small counter.yaml and what i have in here is just a pod called counter which runs a small busy box container image which runs in a loop and writes a date stamp out to standard out to deploy the simple counter app all we do is say cube ctl apply minus f and we pass in the counter.yaml file so i go ahead and run that and that'll create our counterpod if i say cubectl getpods we can see that pod is up and running and if i do cubectr logs for that pod we can see that that part is now writing logs so let's see how fluentd collects those logs so now that we have fluentd up and running it's very important to understand where the logs are stored on the kubernetes node that's the same place that we've just mounted into our fluency pod so let's go inside that fluently pod and look at the log files so to go inside that pod what i'm going to do is say cube ctl in the fluent namespace get pods i'm going to grab the pod name and i'm going to say cube ctl exec minus it the pod name followed by bash this will give me a bash terminal inside of the fluency pod and now that we are inside of the pod let's go to that folder that we're mounting in from the host which is var log if we change directory to that and we do ls we can see we have a folders called containers this is where the logs are stored on the host we also see that we have a fluentdcontainers.log.pos and this is the position file where fluendy keeps track of where it's read up to it's important to note that you have to mount this in from the hose so fluentd reboots it won't lose track of where it's read up to and it'll continue where it's left off we can then do ls in the containers folder and we can see that we have all these container folders over here we can see our counters here as well so there's our counterpod in the default namespace and if i do ls minus l to that location and to that specific log file we can see that that file under the containers folder is actually sim linked to var log pods so everything under var log containers is sim linked to var log pods and if i run cat on that var log pods folder and i do cat on that exact log file we can see that our raw logs are here on the node so the raw logs are being stored under var log pods so you can see the sim linking here gives linux the capability of writing like aliases for parts and if we go back to our fluency configuration file we can see that we always tell fluency to look at the var log containers folder so as a standard fluentd always looks at var log containers but depending on the type of kubernetes cluster you're running the container runtime might be writing its log in a different location and those kubernetes implementations will have different sim links to make sure that there is a sim link available so we can read logs from var log containers so if you're running kubernetes in a cloud provider every cloud provider might have a slightly different implementation every cloud provider might use a different operating system and a different container runtime some might use docker some might use container d or other container runtimes and if this is the case the cloud provider will provide a sim link to link that directory to var log containers as many cloud providers run docker that's why the fluentd daemon set under the volume mount section mounts var log as well as var lib docker containers so if you're running in like microsoft aks you might see that there is an additional sim link to sim link var lib containers to var log pod and in turn sim link var log pods to var log containers this helps us keep our fluency config map simple as we can always expect the logs to be in the var log containers directory so docker writes its container logs to var lib docker containers but not all kubernetes nodes use docker so to keep kubernetes consistent these folders may be sim sim-linked to var log containers to maintain consistency so now that we've confirmed that we can see the logs on the node and we've mounted those logs into our fluency container what should happen next is this file-fluent.conf configuration that we're using in fluentd should be matching all the logs that we're collecting and using the file plugin to write it to this location so now that i'm in the container here i can i can change directory to temp file test.log i can do ls and we can see that we have a log file over here and what i can do is i can print out that log file and we can see that we have all the logs being dumped here to file so this shows you that we're successfully reading all the logs from the host and we're able to collect it and write it to a file locally so we know our log collection is working and it's also important to notice that our logs have been parsed so they're now written as json format and we can also see our kubernetes metadata plugin has added extra kubernetes metadata onto this log so you can see we have this kubernetes section here and we can have we have the container name the namespace the pod name we also have the pod ids we have the hostname as well as labels to fluently is adding important metadata onto our logs so to summarize blue indeed looks at the path var log containers.log which is similar to var logs pods and if we cut out the raw file on the node we can see that our container is writing the raw logs onto the file system here and if we cut out that raw log file we can see that it is written to the disk over here this is where we have this custom ragex parser that will parse the logs into the format we expect all the logs that's picked up will be tagged as kubernetes.star and also extra metadata will be added onto each log and then if we scroll down the file fluentd conf will take all of the logs it matches and write it to a temp file on the temp directory and if we take a look at that temp directory we can see that the file plugin has written all the logs it's collected to a temp folder and we can also see all the kubernetes metadata on each log entry but it's important to notice that if you're running in the cloud to take a look at the raw logs by saying cat and cutting out the buffer and taking a look at the log format if you're running in the cloud you should see that this format might be different and it might be formatted as a json file instead so for that reason you may not want to use the kind fluent.conf and instead use the pods fluent.conf that i have here which is exactly the same the only difference is it has a kubernetes parser so it will pass the logs in json format by default now that we have log collection working we may want to send logs out of the system to a central place where we can analyze it for that we're going to be using a simple plugin and in this demo i will be using an elastic search plugin and send my logs to elasticsearch so for demo purposes inside my monitoring logging fluentd kubernetes folder i have an elastic folder with an elastic search demo and a kibana demo yaml file if we take a look at the elasticdemo.yaml it's just a deployment running a single instance of elasticsearch and we're running elasticsearch 7.9.1 we also have a simple service to expose elasticsearch and then if we take a look at the kibana demo.yaml we have a simple deployment with one replica running kibana 7.9.1 and we have an environment variable here pointing it to our elasticsearch instance we also have a service to expose kibana now to deploy this demo instance of elastic and kibana i'm going to create a new namespace called elastic-kibana and then what i'm going to do is i'm going to apply the demo file by saying cubectl in the elastic kibana namespace apply and i'm going to apply the elasticsearch demo file that'll go ahead and create a deployment with one node of elasticsearch as well as a service to expose it and to deploy kibana i'm going to do a similar thing saying cubectl apply in the elastic kibana namespace and i'm going to apply the kibana demo yaml file that'll go ahead and create a single node kibana instance with a service to expose it and finally if i do cube ctrl get pods i can see elasticsearch is up and running as well as kibana and we can access kibana by saying cubectl in the elastic kibana namespace port forward and we can port forward to the kibana service on port 5601 so if i copy that paste it in the terminal that'll go ahead and port forward and then i can open up my browser page on localhost 5601 and we can see i have access to the gemana dashboard so it's important to notice that we don't have any data yet as our fluency is not configured to send logs to elasticsearch yet so to enable that i go to my fluentdconfig map and i take a look at the fluent.conf and we can see we've disabled the elasticsearch and we're currently only writing logs to file so what i'm going to do is i'm going to go ahead and comment that block out and uncomment this line and now we'll pick up logs locally from the kind cluster and ship it out to elasticsearch so i'm going to roll out that change of that config map saying cubect i'll apply and i'm going to apply the config map back to the cluster i'm going to do cubect i'll get pods in the fluentd namespace and i'm going to delete that pod to force a restart and now we'll get a new fluency pod that'll access the new configmap value so i can do cube ctrl get pods we can see the parts terminating it went and created a new part and now it's back up and running so we should see logs now going to elasticsearch so if i hop back to kibana and give it some time we can hit the menu button go to kibana discover and we'll end up on the index patterns page and if i click create new index pattern we can see that we don't have any data yet so we need to be patient for the data to come to elasticsearch and after some time we can see now i can create an index pattern and we can see blue and d kubernetes has appeared here so i can go ahead and i can just create an index pattern for that guy go next and create it and the interesting thing we can see is that there's all these interesting fields attached to our logs so we have the docker container id we've got kubernetes container image we got container ids we got the host name that the container is running on any labels attached to that pod we got quite a lot of kubernetes metadata here that you can see a couple of pages of them which is very useful and if i go back to the menu and click on discover we can now see all our logs is our inelastic search and here we can see that our kubernetes pod name is counter so this is our counter application in the default namespace that's writing logs to standard out so hopefully you are now well equipped to understand the plumbing that's required to run fluentd on top of kubernetes and collect all your container logs and this also allows you to update your applications to simply write logs to standard out and you don't need applications to know about fluentd's existence applications also don't have to have any code to send logs to an external system they simply write it to standard out and fluentd will collect it so also be sure to check out my link below to my conceptual video of standardized and centralized logging that'll help you understand how to collect logs effectively in a distributed system that's scalable also check out my basic video on the basics introduction to fluentd it'll help you run fluentd locally understand its configuration and its plugin system also remember to like and subscribe and hit the bell and if you want to support the channel even further be sure to hit the join button down below and become a member and as always thanks for watching and until next time peace you

Channel: That DevOps Guy

Views: 14,632

Rating: 4.9906101 out of 5

Keywords: devops, infrastructure, as, code, azure, aks, kubernetes, k8s, cloud, training, course, cloudnative, az, github, development, deployment, containers, docker, rabbitmq, messagequeues, messagebroker, messge, broker, queues, servicebus, aws, amazon, web, services, google, gcp

Id: 6kmHvXdAzIM

Length: 27min 54sec (1674 seconds)

Published: Sat Dec 05 2020