Grafana Loki Promtail | Grafana Loki Setup And Configuration On CentOs

[Music] hello friends welcome back to my channel and today we are back with an exciting tutorial so there has been a lot of requests to me like to create a video on grafana loki so this will be a great uh interesting tutorial where we'll be going through complete end-to-end process of graffana loki so to get started you know we will see what is the grafana loki first so to give you some information so grafana loki is a horizontally scalable and highly available and multi-tenant log aggregation system and this is inspired by prometheus so this is a log aggregation system uh the loki which we are going to see in this tutorial and it's designed to be very cost effective and easy to operate so uh why you know some information like how grafana loki is working so because it's built around the idea of only indexing the labels or metadata of logs so you are not going to index the all logs so what is it does is like it only do indexing the meta data of logs leaving the original log messages unindexed that means logging a lock is cheaper to operate and can be order a magnitude of more efficient okay so uh this uh loki project was started by grafana labs in 2018 and this is uh open source or released under apache license so we will see more about grafana lockhees uh in the upcoming slides so before that i would request you like you know if you are new to my channel or if you haven't subscribed to my channel click on the subscribe button also like my video share and comment so let's uh see more about the benefits of locking so uh to give some details like as we discussed the loki uh only doing the indexing of meta data so not the full locks so that's the one of the benefit of using loki and uh loki is as a very cost effective because it's needed less usage of memory so you don't have to order for high memory systems so and the logs uh you know which can be stored on object stores like s3 or even like a file here so that's a very cost effective setup and also a lock is support multi-tenancy that means you know by using multi-tenant id each tenants have their data stored separately and also the mode of operation lock is you know optimized for running both locally and also for uh scaling horizontally so that these are some of the benefits of using locking so let's uh see more about uh you know the how the setup we are we will do in this uh grafana lockheed tutorial so there will be three components which you'll be seeing uh in this uh tutorial one is called prom tail so if you are not here about promptel promptel is a client which fetches and forwards the locks to locking okay so that's what uh promptil does and loki as we spoke about it's a log aggregation system and you know it's horizontally scalable highly available and multi-tenant and we'll have grafana you also also seen my grafana tutorial so it's a visualization tool uh which support loki as data source so grafana support other data so also like if you have seen a previous tutorial where we have used prometheus as data source right so in an architecture we like in this tutorial how what we will do is like we will have logs on uh you know which we want to in you know send it to loki so what we will have is promptel which will be doing this lock collection and which will be sending to loki right and from loki will connect it to grafana uh so loki will be added as a data source and we will have the grafana to show some charge or the data to be seen from grafana okay so that should be the complete architecture what we will be doing in this tutorial and let's see uh the step-by-step process what we will do as part of installation of and we'll be doing on a centos machine so there are other ways to do it we can do it as a docker container we can use docker compose we can use kubernetes services but we are going to start this tutorial with the centos setup which is the straightforward uh normal setup what we can do and you know uh going forward in the upcoming tutorials we will see how to set up on docker containers and also as docker compose or even as a kubernetes okay so what we will do is we will install and set up prom tail and that would be the one of the step and now we will also install and set up blocky and we'll also install and set up grafana and once we all have all those things set up what we will do is we will configure the data source in graphana uh you know as loki so the loki's data will be in you know available in grafana we'll be able to search it using referral okay okay then uh we are straight into the demo now we can go ahead and do the steps so i'm using a centos machine so you know i can do the setup here so first we will do the setup of grafana so if i go to this grafana download page there are different setups available you can use the m install you have binaries uh you know you have different kind of setup so we'll go with this standard standalone binaries so i'll just use this doubly get so if you don't have a duplicate install you need to install dublicate to download the grafana uh no binary so let's wait for this download to be completed okay great so we downloaded the grafana binary so you can see this over here to just to make it a little bit more uh you know easy to have all the files on one folder i will create a folder called loki and you know what i will do is i will move the grafana tar file to loki folder okay so if you see it's more to loki so i will go to that folder so you can keep all the files over here okay so now we need to enter this uh things or grafana binary so i will use this command so i will uh link all this uh tutorial uh you know into the video description so all the grafana links and loki and other links into the video description so you can get it from there okay so you don't need to write down these commands so you can see i have a folder called grafana 8.0.3 so i will just get into that and you can see there are different folders i will go to bin folder and you should have a grafana server so i will just start the grafana server okay so so that the grafana is will be starting up so now you can see grafana started on port 3000 so in order to access you need to use the ip address of this machine and the port 3000 so this is the ipad resin port 3000 looks like it's not loading because you know we need to do some uh port uh open also so we need to open the port for 3000 in the firewall so in order to do that i will just run this command so i will give this command in the video description and we need to reload the firewall so you need to reload then we can go and uh try accessing uh so now we can see the page is loaded so now we can log in with the username as that mean and password as admin as well so once you enter the username admin and password admin it will ask you to set a new password so you can set a new password so now we are inside uh the grafana login page right so you would have seen this multiple times in the previous grafana tutorial so this is the grafana page and now we need to have our you know data source loki and prom tail setup right so i will open a new session so i will just clear out the screen uh this is the same uh centos machine so now we need to set up uh loki and graph uh prompting so in order to download uh loki and promptel you can go to this github page where you'll have different versions so we will go for uh linux amd 64-bit uh this zip file okay don't go for this uh cli or other versions another rcl log cli but i know i i don't go for that go for loki linux amd64 and also uh prom tail linux amd64 okay so let me just download this zip file so in order to download i'm going to use the same uh loki folder so i can have everything inside that so i will use the same dublicate and the github location so you can get this github location from that link which i will link into the video description so let's just wait for this download to be completed so the loki is downloaded so if you can see the file is let's listed here let me download the prom tail as well so this is the link for the github link for prompted linux amd64.zip so let's download that as well so if you don't have duplicate you will have to install duplicate before uh downloading this okay so it's the same like we did for uh grafana so if you don't have duplicate in this machine you need to have yum install duplicate to get have it uh so i want this downloader we'll uh you know do the unzip of the uh this zip file so that we can get the loki and from tail setup okay great so promptly is also downloaded so let me clear out the screen so now you can see we have loki and from tail also available uh this zip file is not required anymore because we already extracted a graphana from it so now we need to unzip it so in order to unzip it i need the unzip so i'm going to install unzip first okay so let's just wait for that installation to be completed okay let me install uh unzip so let's wait for this installation to be completed then we can use that for unzipping that for files okay so the unzip is installed so let's go and unzip the loki and prom tail okay so not unzip it you have to use unzip in this file so now you can see you would be able to see like there is a file for loki and also you need to unzip promptly you can you should be able to see the file as prompted as well so let me remove those uh tar and zip file so that you know we don't have to use that so now you have two files loki and promptil and this is for grafana so in order to start loki and promptly we also need a configuration file so you know what i will do is i will use an existing uh standard configuration file so this is available in the grafana github so you know this is a standard configuration file you can edit it based on your requirement but i just want to do for this tutorial i will use this uh loki configuration file and also this is for the prompting so if you see in promptil what we are doing is we are pushing you know the locks uh you know to this localhost 33100 which is for loki ip address sorry the port number and also you know we are going to say in the scrapping like you know you need to use this log location so it will collect the logs from this location into loki okay so the port uh it says things 9080. so what we need is first we need to download this configuration file and also we need to open these two ports also on the machine so let me just download this uh files first ml files so i will use doubly get to download this ml file so you can see uh loki ml file is already available now okay let's download it for promptil as well so now you can see both loki and promptly ml files are available so what we also do is like we will open the ports for those 3100 and also 9080 so similar uh the command which we have used to add the ports we'll use it here as well so it's the command for 3100 and i will do for same for a 9080 as well now let's reload the firewall so these changes are applied so if you don't open this ports the connection will not happen between loki and uh prom tail so you will not be able to get the data on the grafana okay so now we can start the you know loki and from tail instance so we can start with loki and uh we cannot just run it uh just like that we also need to map this configuration file so what you need to do is we need to use the argument like config.file and we are going to map it to this uh file okay so if you are changing the file name to a different file name you need to use that file name here so let me run that so now you can see loki has started message right so lucky started now we need to start the prom tail as well so let me go to another session okay we need to start the uh prompted so i will go for prom till okay i think we need to go to loki folder and we need to call prom tail okay and we need to also use the config file here as well so the similar way how we set the config uh argument like config.file we are going to map the prompt a local ml file which is uh available here so let's just run that okay i think there is a small problem in the yaml file let me check that for some reason when we downloaded the yaml it did not uh it have more content so we don't need the whole thing so we'll just copy it from the github so i'll just create it manually so vi i'm going to call the file as same as that so let me just insert that i'll just goes to this github and i'll just copy this content from this one and i'll just go to the vi file and i'll just paste it so let me just save this and i will run the same uh prom tail uh with the config file now you can see it has started so it has started uh we'll just wait now you can see it has started you know this path which we have told so there are uh which we have around few files over here so one is uh for tally log another one is the last lock another one is boot lock and there is one for main log and there is a yam lock so i think these all files are sent to promptil and we should be able to see that to grafana as well so let's go to grafana okay so we have the grafana here so now we need to add the data source so you can go to search for loki just search for loki and here we need to enter the ip address so we'll use the same ip address with port 3100 so the change is only just 3100 and you can [Music] save and test you can see data source connector and labels found right so we are done with the the connection we can go back to the home page okay so you can see data source is connected let's go to explore okay so now here you can see loki from the drop down now we can do some search here so i can do some query here so i'll just use this file names equal to war log m dot log and if you see we got some output right so this is the uh unzip we just installed it right so we got some output here so this is for last one hour let's let me change it for 30 days maybe and i think there are nothing much installed here so maybe more than 30 days maybe two years yeah it looks this machine is looks like i don't have much things installed so maybe let's try with some other query okay so i'll just go for uh boot.log so let's try that so looks like nothing over here as well let's uh see what all thing we have it so let me go to dashboard let me create an empty dashboard and let me check the browser what all thing we have okay so in the file name looks like we have only m.log yeah so that's the problem i think we have only just one log indexed over here named job so i think we have only just m.log is what only we have uh you know available from that location which is sent here okay so that's why i think we have only m.log so you can see i just installed uh zip uh also it's showing up here so let me just try to make some other changes um [Music] install maybe kit let's see what it does over here it's not completed yet so let's see it's completed let's see whether we are able to see the lock here yeah you can now you can see it is showing uh quite some things right which got installed now so oh no it's uh you can see how we are trying to get the locks right now we are using the job here so similar way you can run for the final file name as well so basically you know what we are doing you can see this charts coming up now right so be using this content that you should be able to make the changes so one only thing what we need to know uh you know you need to make some changes is like you should know this uh promptly lock so here if you see we are not scrapping only this log file so if you want to make some changes what log file you need to set it to loki you need to make some changes in this uh you know configuration so it can send what uh necessary things into the loki system and that will be showing up on the grafana as well using this then you can make some charts or you know graphs as based on your request so but ultimately i just wanted to show you how to set up loki uh promptil and grafana so you should be able to get the data into the grafana so creating charts and everything it's you know it's more of a common thing you should be able to learn with my other tutorial on grafana where we have created a lot of graphs and data on that so in this tutorial we have seen how the locks can be you know uh sent through using prom tail to the locky right and from lucky we are connecting to grafana as a data source so you should be able to see all the data in grafana you can create the charts and everything with that so that's all for this tutorial i hope it's an informative tutorial for you so kindly click on the subscribe button for more videos and i'll keep supporting by liking the video sharing and comment

Channel: Thetips4you

Views: 4,137

Rating: 4.9452057 out of 5

Keywords: what is loki grafana, grafana loki promtail, grafana loki dashboard, grafana loki setup, grafana loki configuration, grafana loki installation, grafana loki tutorial, grafana loki install ubuntu, promtail loki, grafana loki demo, grafana loki architecture, grafana loki config, grafana loki docs, grafana loki getting started, grafana loki in production, grafana loki introduction, grafana loki logs, grafana loki prometheus, grafana loki windows, loki, grafana, promtail, thetips4you

Id: iqpLXUdJ0Ro

Length: 21min 15sec (1275 seconds)

Published: Sat Jun 19 2021