Configure Telegraf, InfluxDB2 and Grafana using Docker Compose and Grafana Dashboards

Mo I want to show you how to configure what's known as a TIG TIG stack uh that stands for telegraph influx DB and grafana so together influx DB telegraphic and F are three open source tools that are used together for monitoring and visualizing time series data now before I get into the demo I want to walk through the presentation so we have a little background uh first what is influx DB so that's going to be the i in TIG all right so influx DB is a Time series database okay it's optimized for storing and quering large amounts of time stamped data so that's what makes a Time series uh it benefits being it's very high performant so it's designed to handle High WR and query uh loads for monitoring iot and realtime Analytics and it has a lot of flexible use cases suitable for a variety of use cases ranging from infrastructure monitoring and realtime analytics so infrastructure monitoring that's just going to you know say it's lend s well to Cloud applications all right now what is Telegraph uh Telegraph is a plug-in driven server agent so it's going to collect and send metrics and events from databases systems uh iot sensors for example and it's flexible it can collect data from a variety of different data sources and send it to influx DB for storage and Analysis and it's one of the more integral parts of the tig stack so it's what gets everything going so it's just generating the metrics for you know for the entire stack and finally what is grafana uh graan is well known it's a data visualization uh software um it allows us to visualize data from different data sources uh and the data source that we're going to be using grafana to configure is going to be inflex DB itself okay all right so let's kind of get right into uh the heart of this presentation so first things first I want to show you how you know I got everything set up all right I'm going to Docker Hub and you know reading the documentation and you know just kind of give you a little tip on how actually how to do this whenever you do you know set up a Docker container uh you definitely want to be looking at the you know the instructions for how to use it and so in this case for influx DB we have an automated setup okay so that means that there's like required uh these are environment variables uh these environment variables are required to actually get the database up and running okay so this is a Docker composed file so in this Docker composed file you know I'm basically you know providing everything that the document uh tells me I need to provide uh specifically these was it the you know four five environment variables okay so let's kind of take a look at these so the first thing is going to be the you know username second is going to be the password so we also need to configure what's know as what's known as an organization the name set to the systems initial organization that's going to be important in a moment and a Docker influx DB init bucket so it's a name to set for the system's initial bucket so as I can understand that initial bucket is just going to be you know a it's a container uh that's going to hold all the metrics that we're going to use you know for uh you know the system that we're going to be monitoring all right so that's that's why I configured all these so without these being set we would not be able to you know launch uh the application uh correctly okay all right so the next service here is going to be a telegraph okay again Telegraph is that agent that we that's going to collect you know metrics from you know whatever we tell it to excuse me whatever we tell it to it could be a system it could be like an endpoint that's giv off metric data um so in our case this Telegraph here an agent again for collecting metrics and writing them to influx DB or other outputs in our example we're going to monitor the docker engine host okay says like it says so one common use case for telegraph is to monitor the docker engine host from within the computer okay uh the recommended technique to mount the host file systems in the container and use the environment variables here is going to be here so we're just going to be monitoring you know the docker engine on my machine okay and visualizing it all right so this is going to give us like a a lot of metad into the performance of Docker as a service running on my machine and this right here it tells us how to actually Mount all of the volumes in addition to setting the correct environment variables so this needs to be translated over into uh dock compos I recommend using you know open AI for that or uh yeah chat GPT it's very helpful okay so that's that's how we got this right here okay so this is just going to allow us to monitor the docker uh engine host so it's just allow us to monitor the performance of Docker running on my machine okay um and it's going to be using what's known as a a telegraph.com so we also need to use this um telegraph.com uh to set that up so we can get it we'll get into the uh well let's get into it now um so whenever we configure This telegraph.com Here It Go says right here telegraph.com we need to be able to uh you know configure in like the inputs so it's just basically collecting you know the metrics and the intervals for the system that it's going to be writing uh into the database all right so that's why we need this Telegraph . okay all right so it tells us how to uh how to load this load this file uh as a volume so that it it's configured inside the container and you this command is also very interesting too because it also you know sets the permission so that this Docker socket can actually so that the docker in the container can actually you know read the socket on my machine okay so this is a very important statement too I can give you the uh the URL while I I actually got that to get this to work it this won't work without this either okay all right and finally we have grafana okay grafana that's is going to be a visualization software all right and so I'll show you how to configure grafana to use influx DB as a data source so without further Ado let's get everything spun up so I start I'm going to start from scratch okay so let me do a Docker compos up Docker compos up just basically launches all these containers so they can speak to each other after you configure it correctly so we got all these statements here influx DB Telegraph okay so that's fine all right so it's giving us a meage saying hey fail to write metric and I'm actually glad that we see this because I haven't configured I I previously had a recording I had to stop but uh that's talking about this token right here this so This token no longer exists anymore we're going to have to create a new token and we're going to see how that affects the these logging statements you see so right now it says unauthorized it gave us a 400 well that's good that means it's the okay we there's a problem on our end we need to configure the correct uh security credentials so let's do that right now so in order to access the uh the influx DB service we need to access access that on Port 8086 which is this okay just sign in so it's asking us for the username and password which of these environment variables I set username and password Cloud sheeper okay hit sign in okay so we have to uh I'll just kind of give you a uh you know just fast forward I had to go through and read the documentation on actually how to set all this up you know with influx uh you know documents and so in order to get the token that we need the security object that we need we need to first go to data and we're going to go to tokens here we want to go to generate tokens and we want to get a read write token so this just allows you know like it implies this allows any service to read and write data into influx DB okay and you know we configured these buckets so we want to say Hey listen with this token anybody can you know read from all buckets and they can write from all buckets okay we can just say this one is like you know you YT for YouTube all right so that's just a description there so again this is a security object it's giving the permissions for anyone who has this the ability to read and write from all the buckets so is it basically view all the data and you know update the data as it needs to which is what we need so token was created successfully let's go ahead and hit this guy and we get the security object okay this is the token summary of access permissions buckets read and write now if we go back to Telegraph we want to replace this one we want to replace this token which is from another demo with this one we updated it okay and while we're here we also want to go ahead and configure the URLs okay uh so basically the target where this is Telegraph this is the agent that's scraping this getting the data from the system we got to point it somewhere okay we want to point it is to influx DB remember it's called TIG so t for telegraph I for influx DB so we want to send it over to influx DB and you know because we're using Docker you know in Docker this is all in the same network and this Docker composed so the the docker container name that's going to be the DNS name that's how we actually reference these containers uh us in networ working in in Docker so we want to say uh we want to access influx DB through this URL all right which is the same one I used to actually uh well similar that I actually use to access that that user interface to log in okay so we have to configure the URL the token organization and the bucket remember so these are already configured so the bucket and the organization Okay so we've already configured those for the inflex DB okay so now that we have those in here we got to stop everything and we can start it back up see again we were seeing those unauthorized access because we had that wrong token but now let's hit dock compos up now that we have the right credentials in here okay and it's let's see Telegraph let's see some Telegraph statements here we just we should start to see those soon okay okay so that's just some kind of error in the plugin but that's not the same as the you 400 like access uh errors we were seeing before so let's just go ahead and see so now this is spun up let's go ahead and let's yeah we got to log it back in again all right so now that we do that we we see that it's actually writing to uh so this is Telegraph so Telegraph is actually writing to this to these buckets now so if we go to this bucket we can see we start to see all of these Docker related um metrics here okay okay so that's just because I tried to log in after I I I stopped the service all right so so we get to see all of these so I'm not an expert with using flux because this is um you know it's just a querying language that is brand new to me but I was doing this a little earlier the histogram actually will kind of show us some data uh and we just spun this up so we may have to give this a moment here so I tell you what what I'm going to do in the meantime is I want to go over to grafana and set that up you know until you know we start to see some metrics over here let me just uh see if I can get something here in the meantime but this is this is completely normal uh from what I just um saw let see here and because we just set this up it hasn't even been five minutes yet so let's just give us some time okay so what we're going to do now is we want to set up uh grafana so we accessed grafana on Port 3000 okay so Local Host 300 okay so in uh by default the username and password are admin so we're just going to enter those in for the first attempt let's get this out the way and we just want to enter that again just so we can you know get where we need to go all right so whenever you you log in to gra for the first time this is exactly what you'll see see some data sources so we can use influx DB as a data source to grafana okay which is this guy right here influx DB um now we're using influx db2 so that's that's just it actually says it right here supported for influx db2 that's the container excuse me so that's the container that we're using image is influx db2 so we need to be using influx what we need using flux here as a query language okay and to configure all the connection details we actually have most of that already over here because you know this is Telegraph it's already writing to influx DB so it's basically the same thing on the grafana end we just need to be able to tell her how to read and write or definitely how to how to read from um from influx DB okay so we have uh that URL we only have also the basic authentication let's go ahead and get that so remember the basic authentication that's just going to be the username and password I may need to look into this input. XC all right so we just C we just copied these from um Docker Hub so the documentation so if it's throwing off an error I need to look into that but yeah that's kind of where we're at right now now okay the cloud Shepherd the cloud Shepherd okay and we want to get the influx DB details okay so those details uh definitely going to start with let's start with the organization so these are all the same as the environment variables we set before I showed this so let's everything we need is right here in this Telegraph configuration file okay and again we got most of this stuff right from the docker uh Hub documentation here so so organization is that cloud.oracle.com um it's going to authorize us uh to access the database so we copy that in here the default bucket you know B I guess the bucket is just going to be an object that's going to store all the metrics that we're going to need to uh query in this time series database and so we have all these values here so now let's test and see if we have everything said correctly look at that so I like it I like grafana because it it just tells us right away uh in you know nice visual cues hey that this bucket is uh this data source is working so that's what's up so we save it all right so uh the next thing we want to do is just try to go ahead and configure a dashboard now with grafana like the way to do it I would tell you is just to uh you know you can actually get the pre-made dashboards uh so you know basically just you go to Google and say hey I need a a dashboard for influx db2 Telegraph Docker so I want to try this one to see if it works so I have you basically you copy the dashboard ID so create a dashboard you want to go down here to import a dashboard so you paste that ID in here and you hit load and it gives us so influx db2 Docker dashboard so let's hit import and we want to select the data source which is influx DB look at that that's it boom shaka laka that's that's our data and that's it that's how you do it so yeah so I mean that that's uh that's how you actually kind of get this up and running so uh I don't want to make this docu uh this YouTube very long so listen I want you to uh you know go ahead and you know if you haven't done so support my channel liking and subscribing please share uh are be making more of these videos depending on the feedback and uh if you have any questions again just leave them in the comment section and uh I look forward to uh you know making more content for everyone thank you and have a good day

Channel: The Cloud Shepherd

Views: 2,116

Keywords:

Id: SjwO5eCKPrA

Length: 20min 57sec (1257 seconds)

Published: Wed Nov 08 2023