How to configure Fluentbit to collect Logs for our K8S cluster ?

[Music] keep [Music] welcome everyone to another episode of is it observable it's uh the main objective of is it observable is to provide tutorial on how to observe a given technology today you are watching one of the episodes of the kubernetes series we already have several ones we had once dedicated to collect metrics in a community's environment the second one we were uh talking about logging so how to collect logs from a community's cluster and during this episode we're using loki and their default a log collector and for order named prom tail today episode uh because logging is a very huge topic we will dedicate this episode to one of the most popular agents of the market i'm referring to flynn bit correct all right so what are you going to learn out of this episode so first presentation of flynn bit and you will see that there are different things uh related to flame beat itself the configuration and also last before we jump to the troll i think it will be very beneficial for everyone that we see the differences between fluent d which is very popular it's the big brother of win bit and flame beat and then of course we will jump through to troll so in the previous ones we were we had already deployed a cluster with a application with prometheus and loki to get the lock so now we were gonna just simply replace uh the default agent prom tail by flynn bit and see how the logs are structured in grafana and then what is really important in this episode is the configuration of film bits you will see there is so many things you can achieve with flambeat so it makes sense that we spend more time today in this tutorial to to configure flambeat to be able to send not only log streams to loki but to dynatrace as well you'll see uh you have several things to uh to to take to consider to properly configure families all right so let's start with the first part of this episode flynn bit so let's see thumb bit the flambeat is a logging processor tool it's part of the big family of the log agents collector and forward so we already explained the value of logs in the observability world just to resume it it's extend our capacity to analyze our data and troubleshoot in the previous episodes we were mainly talking about how to collect application logs but there are also logs and solution that are also generating logs in our environment so we're using database so database with general logs there is system log as well from our operating systems we have the web server logs we have the access logs and yeah depending on the technology that you're dealing with there's a big chance that it will generate some logs as well so it could also make sense and beneficial beneficial for us to collect not only our application logs but also logs coming through those components so collecting all the logs and correlating them is very important to understand precisely what is going to happen happening in our environment with the right context remember that context is crucial because it will help us to crawl it the flame build of course will be able to collect logs from the resources from our pods like explained in a previous episode but and also from servers similar to prom tail flimbit can also collect metrics from our linux environment so and others as well by the way it could be used for in a traditional server could be also used in a vm environment containers and communities so like explained in the previous episode so we expect from fan beats to collect our logs to add the context to it label process and transform the log streams in a key value pair format and then once it's been transformed we can send it forward in fact to a lock storage solution so there are plenty of uh of a solution that will allow you to store your logs so elastic search blank kafka loki and dino choice as well the flynn bit is the most performant and lightweight log processor so what do you mean by performance think about it we are we are gonna use a solution that's going to stream collect logs from various sources transform it it could basically generate a lot of overrate there and more logs you have to deal with where more uh how overweight you can observe so flame beat has been basically created to high scale environment and to basically be the most lightweight log processor of the market the film bit is the little brother or sister of flindy so flindy is obviously the most imp popular solution of the market and because of the num the high number of plug-ins they are out there so like his big brother he took the same approach by utilizing a lot of plug-ins when you build a pipeline in a pipeline is the way of how you're going to ingest the logs and transform it you cannot utilize several plug-ins so first you have the notion of input so as an input you have input plugins that will help you to collect logs from various sources so it could be uh and also transform into a metric so you can have plugins to collect stats d metrics collect d cpu metrics there is disk i o docker metrics docker events http you can query an http endpoint pal which is very popular network and so on so forth then once you have collected you want to parse the data that you've you just collected so that's usually the second step of your pipeline there are also a lot of plug-in available out there so you have of course json because json is going to heavily used in logging you're going to have the regular expression to transform you have log format which is also a standard format for logging and you also can decode some values out of your log stream once you have parsed your logs you want to filter so filter will allow you to modify to unreach and drop information from your logs and similar to the other uh steps you have a lot of filter plugins available so you have expect you have goip filter you have grep very useful kubernetes uh you can also have the record modifier the re writer tags a throttle nest and modify so throttle by the way we're gonna see it during our tutorial and a lot of other uh filter uh plugins you will see because usually when you have some logs you have to transform it to the right format of your log storage keep in mind that there's another filter plugin that is very interesting uh juventus that never used it to be honest is the lua script so you can basically create a script in lua format and why can you build your own scripts simply because sometimes you want to anonymize the data so here is an example of other way script and this one will basically replace one of the records with another value so then you will analyze the sensitive data out of your log stream then you have the routine so routing i will call it output because basically once you have transform collected you want to send it to an output but what is just amazing with flip bits is you can plug several outputs to your pipeline so and we'll be in charge of storing the lock screw lock stream to various various data source and you could basically say all the logs coming from that tag will go here and the other one will go there so you can have a lot of advanced rules to decide where you want to send your logs to so similar to the other uh type of steps in your pipeline output has tons of output plugins available so you have low key you have azure blob blog you have azure log analytics you have google cloud bigqueries elasticsearch std out and you will see we use sdo by the way influx db and you can store it in a database as well stackdriver http and so on and so on so to review we have input plugins that will be in charge of collecting the data on a specific source and transform it then you have output plugin that will send the log stream on the desired format of the selected solutions not designed for it you will use of course parcels in the filter to build the desired format so third bit could be deployed with the help of a home charge this is how we're going to deploy it and technically behind the scene you will have of course some rolls and everything but you will have a daemon set uh and a configuration a file of them bits that will be stored in a config mac config map the configuration file or the configmap has of course several sections in it so you will basically define your pipeline so input uh the parser uh the filter and the output so basically you basically build your sequence of of steps that will define how you want to transform collect and send your data to your data source and by the way there is a very nice solution out there that i we're going to use for the tutorial and that it helps you to configure phone bits it's called caliptia and here is the url you can utilize it it will provide you a visualization of your configuration so basically you put your configuration file you can change it and that will help you to validate your syntax and visualize the pipeline that you have built and you have plenty of examples by the way so why is smailbit so powerful in a computer's environment well the cool feature is that you can directly add annotation on your deploying files that will allow you to to precise to film bits what type of parser you should use to collect the data from that particular pod the plugin architecture makes a little bit more powerful and more customizable i mean compared to promptel is a great solution but it's much more simpler to configure the transformation of the logs to be honest you can it supports also a lot of security concepts uh especially on the output plugins because at the end of the day you're gonna transform build collect a lot of sensitive metrics and then you wanna send it somewhere and you want to make sure that there is no security get lim issue when you're sending that data to your storage so what is the difference between flynn bit and fluently flynn bits advantage is the fact that it's way much lightweight compared to his big brother flynn d so when beats is about 60 60 150k kilobytes and flynn d is about 40 mix so flint b has been highly optimized at low cost to be basically be designed to run in a very high scale environment with a lot of log processing so it's been clearly designed for kubernetes where you know that you're gonna have lots of nodes potentially lots of pods running there lots of components that generate logs so it could generate tons of logs so it's been clearly a deep design for communities you can also combine the usage of flim bits and flandee for complex pipelines you can imagine that you use flame bit to transform most of your logs and you forward it back to fluent d and then flind will do the rest and maybe you don't want to build or complexify your process so you want to take advantage of a flind d plug-in so it makes sense to first do the job between bits and then just utilize three just to send the logs to your data source all right so we pretty much explained everything today so let's jump to the tutorial every tutorial of is it observable you will have a dedicated github repository for the episode so here we are looking at the episode related to kubernetes and fluent bits and you can basically follow this tutorial at home at your own pace and also it would be great to get your feedback out of those tutorials otherwise uh in this uh tutorial in the github repository uh there are a lot of steps that we are not going to do today because we already have done it on the previous tutorials so we already have a cluster with prometheus installed and i just freshly uh removed low-key with prom tail and now i just want to install low-key with flimbits so we can see the differences directly in grafana all right the first step let's open a terminal here it is and we are going to basically uh install loki so here in this uh in this installation i have set flame bit enabled true and problem tail false all right so let's run a a cube cuddle to figure out if we have all our pods running which is the fact we can see that the low key flint bit uh is running in fact it's it's a it's a version of a fluent beat that has uh the loki output plug-in pre-installed that's the what let's make the specific image for this part um so now we have it everything so let's jump into our uh grafana um and we don't have to reconfigure grafana because we already have the the datasource loki uh that we have defined the uh during the other our last tutorial so what i would like to do here is just to simply briefly explore to show you the differences that we had with the prom tail agent so here if we look at it at the first we have a bit less uh labels that was uh added by the flynn bit job but again those labels could be uh adapted depending on the configuration of flimbit so we are going still wants to look at the hipster shop which we do here so we have our logs and similar to the other day um we could also grab one of the logs that we'll be interested to collect um especially for example uh one with some information so here for example we have this the similar logged stream that we were looking at on the other session now we are looking it through uh the same stream but uh collected through with with filmbit so let's adapt the filters so here what i would be interested in is basically to grab the uh the front-end service so we're gonna add this filter here um and what is also interesting i didn't mention the other day uh is you also so now we have the the filter added and let's look at if the other any other filter that could help us to query the data that we would like to get out of this locule so first we have the severity so let's say let's do similar to the other filter we did the other day i'm going to add a pipe the pipe remember and equal means it says log stream contains a specific character a specific string or text so here i'm saying debug which is because i'm looking i'm interested in the debug like we mentioned let's run that query so you can see here debug has been filtered so now we have a bit more things related to what we're looking and you can see here there is some show context i didn't tell you the other day but here if you click on show context is going basically to select all the log stream related to that time frame uh so everything that happened before and after so then you can understand if any any events or any behavior uh that could happen from various services or various components of your application so here you can see that we have this uh the similar things that we can achieve so basically the the difference is uh you you don't see major difference between how the lock streams are being collected and stored in loki at the end of the day here we're using using low ql to display information so then you can basically play with all the various functions let's say here let's say i'm interested so let's say json and let's say we are interested in looking at the http http resp remember the json will convert the we will get the the various payload of your log stream uh that is stored in the json format and gets those uh and convert them into labels so all the dots will be replaced by underscore so resp and then let's say we want to look at the resp status and it's 500 so let's put 500 like this and let's run the query so now by doing this now i'm looking at only at the status 500 that flimbit is doing the same job it collects the log streams but then once you store it in low key then you can basically do whatever you want with we are basically going to remove it so let's do just a hum and install low key just name of my the name of the release i gave to to helm so now it's going to remove everything and now if i do it just get parts you'll see that all the various components related to loki would be removed now uh we are going to install flim bit and look at the various aspects of the configuration so first one thing that i'd like to highlight here is you have various way of installing fluent bits of course i would recommend to look at the documentation flowing bit where you can see all the steps and here in our case we would be basically interested in installing it into uh with the help of a home chart so let's open back the terminal and i just want to do is basically copy this one and install flimbits in this cluster so now let's have a look at uh the paws that we have so we can see that we have our our three pods that has been deployed through our daemon set so a bit is comfortable through the config map so let's look have a look at the config maps that we have here and the one we can see here we have flim bits so let's look at this one and now you can see our complete map and like i explained in this first section of this episode we will find here the various steps of our pipeline um we have a tail so he will basically grab the logs from the as desired folder where we have the container logs um and here it's also take take the systemd information and then we're gonna filter kubernetes by adding some labels there and then we have the an output defined at the moment which is uh by default it seems that we can see if i'm not wrong it's elastic search all right so now we the the the exercise that i would like to do here of course is to uh slightly change uh the uh the log stream that we are able to collect in order to send it to the log ingest api of the entries so the first thing i need to do in this uh in the steps here i want to show you that i'm to in order to debug uh your uh your pipeline i would recommend to do a cd out so sd out of your uh the the positive in beats you will see the various uh lock streams appearing so it will be a way for you to figure out if the transformation and that you have applied is do is doing like expected so here i'm gonna do a format so there are a few of several formats available json json lines is on streams and there's also log format but here in our example i i know that the data trace api wants some some json payload so i'm gonna use the format json then the key related to the timestamp because at the end day we're gonna send some send it on the desired format so let's remove this line here and copy paste i'm just going to paste the uh my my the key json date key here time stamp and the other one is which is could be also very pretty much relevant is the precision how the format of the date uh so here there's a you can put a for date formatter if you want uh there's uh you can look at the blogs of the the documentation of um bit but here date format which i'm gonna use is the 8601 here it we did the change uh but because i'm did the change on the configmat if you want to see uh the the changes lively then you need to basically uh reapply your uh your daemon sets all right so like if i do a ls on this folder where i have all the diverse files from my github repository so if i want to do this and just need to basically i can do i can basically do two things first i'm gonna qdl get the daemon set and we're gonna just delete it and re-apply it and he is going to relaunch the part and the parts will load the new settings of our offline bits all right now it's running so now let's grab one of those container and then we know those parts and i'm going to do a couple logs to get the logs of this this specific pod and we're going to look at what we have so here we can see that we have the logs stream that appeared um now the current format is timestamp you can see that now the date the date has been properly been renamed with the desired law of format we have a log field equals something and you have everything stored here in this log and you can see that we have kubernetes uh the kubernetes labels has been also added so we are pretty interest uh good here uh but let's um add a few things here so the first thing i want to do in this tutorial is basically modify so in dynatrace in the payload log is not a it's not a log format it's content so just for the sake of this exercise i like to rename the potential fields called log uh to content so i'm going to use one of those uh plugins a filter plugin called modify so let's edit again the config map and i'm going to add this one just before we add the labels here in the communities every field called message or log will be your name into a new field called content all right now let's save let's save this one all right now let's take one of the last of the log streams here you can see that now log has been renamed into contents exactly as we want it to the other thing that i'd like to do here i can you can see that the labels related kubernetes are in another level so you see the all the labels of communities are in another object is an object and i would be interested to basically to change the hierarchy so pod name i want to have it up in the upper to the main uh to the main json object so i'm going to use for this one another filter called nest nest help you to restructure with a building and a hierarchy in between your objects all right so in order to save a bit more a bit time here uh and avoid doing everything with vi i basically created a small file in this folder called a config mac demo a cm demo so if you do just a cat you can see that it's the same one that we had before but i just added here in my case this uh nest plug-in i'm using the nest and i'm i'm asking for a bit to say everything which is uh went through the label so because we all the logs coming from kubernetes has been adding a tag uh so there's you can see there's a you can add tag here hosts or in in the previous steps we added the tag for kubernetes which is here so all the logs coming from the docker containers they will be tagged kubernetes and here i'd like to modify only the log streams coming from this location this and they will create labels new attributes with a prefix which is communities underscore all right so let's have a look and now if you look at the new log stream let's take here so you can see it a bit better you can see here kubernetes underscore namespace kubernetes underscore pod so basically now the kubernetes object has disappeared but all those properties they has they are now part of the log stream i want to rename few of those kubernetes labels into a data trace format so then the interest can attach the kubernetes context to a cluster an information that is currently monitored by tenant trace so that's the first theme we use now the nest plugin now like i said uh i uh basically want to exclude few information so here i want to grab everything from my logs except uh the logs from flimbit i'm not interesting to collect those uh because i'll probably if i leave a sd out i will have a duplicated slot stream and it doesn't make sense so to do that again we are going to use again another uh plugin filter plugin which is the grep plugin as you can see here i have added this this filter here and i'm using grep so grep you can either include extra information include extra details or here in my case i said i want to exclude for the log streams do another cat and you can see here i use an again the filter modify to rename a few of those properties so here i'm i would like to rename kubernetes part name into kubernetes.pod.name all right so now you can see that there is some differences as you can see here so this is one of the log streams so let's just select the logs that i'm interested in for example this one so content we have the date it's an error and then you can see that the partner name has been renamed community's namespace is here so now i started i'm starting to have a format that uh is aligned with the uh the desired data trace login just api and you can see here i'm using the http output plugin uh where i'm using i'm gonna you have to refer to the active gate because the active gate of the netrace would be the one that has the lock stream component so if you want to send a api call to ingest some logs you have to go through an a in an active gate if you don't have one no worries you can install it in a in a linux server or any anything like this so if i do this what will happen is that um every uh the lock stream format that we just modify will be sent out to dana trace so if you don't have any data trace cluster no worries for the sake of the game site so you can start a free trial um and in once you have your dynatrace cluster available then you will have to deploy tanner trace so if i do start installation i will go for a wizard so if i want to install the communities operator in my cluster i could do it from here and in my case i want you can also deploy a fresh active gate if you look at the deployment status of my down trace um you can see that i have my active gate running here and you can see that i have the lock component enabled so i will be able to send uh lock streams to this active gate so what do you see the logs in dynatrace very easy so the log streams will be in of course in the observe and explore menu you have a log section and you can see all the logs appearing so similar to the lock ql language you have here a couple of filters you can select what type of process what type of events you can see the log streaming details here as you can see it will automatically attach it to a context which means in other screens i will have also the logs uh stream attached and the dyna dynatrace davies ai will basically utilize the logs as well to figure out if there is a problem or not otherwise you can also uh similar to the lock query like ql you can go to advanced by doing advanced uh instead of clicking here you can basically create your uh your rules i say for example do i have an example to show you i can say status equal error all right run that query it will basically show me only the logs with the errors and i can play and add some other filter using the end or the r operator is i wanted to show you the calypso there are a few examples already there so which is great uh but otherwise i can use the builder and the builder will guide me and say i want to have an input and you can click so in my case you remember we used we were using tail otherwise in my case i don't need it i already have it in my uh editor so i can just do this and i can validate and this will basically first of all look at the uh if there is no issue in the syntax of your configuration here it will basically visualize uh the your own pipeline so um i really like the the calypso in a sense where it's it's basically guide you in building your pipeline so if you are a bit lost and you don't know how how to start i think uh the calypso would be a good way uh to be guided at least in your in your process on building things all right that's it for today's episode so i thought i think we we learned quite a lot of things today uh we try i try to focus a bit more on how to build a lockstream pipeline with flint beats so you can see there is obviously a lot of different options and a lot of plugins to allow you to transform and add more labels to your lock streams uh with the help of those plugins so we took the example of sending the lock stream with the output http to uh send it to the uh log in just api of dyna trace but we could also have send it to the in the forwarder to fluently and funding could also have sent it out to danatrace all right so if you enjoyed this episode don't forget to like to subscribe to this channel and also if you want us to cover specific technology that yeah that you you like to have a tutorial don't uh don't hesitate to let us know i'm always happy to get new ideas from you guys alright so stay tuned for another episode see you soon

Channel: Is it Observable

Views: 1,646

Keywords: kubernetes, k8s, cloud, fluentbit, logs, observability

Id: KJlWV5-o8v0

Length: 36min 37sec (2197 seconds)

Published: Thu Aug 19 2021