Meet Grafana LOKI, a Log Aggregation System for EVERYTHING

over the last couple of weeks i've been looking for a better logging system i've used many different logging systems either at work or at home in my home lab and having a mix of virtual machines other devices containers custom code has made it a little bit challenging to find the right one and while i thought my last logging system was the one i was going to settle on i decided to move to something else so i've recently switched over to grafana loki grafana loki is a scalable highly available multi-tenant log aggregation system like prometheus it's designed to be cost-effective and very easy to operate so there are a lot of similarities between loki and prometheus and that's why it's been coined prometheus but for logs one thing they share and one thing that's really awesome is that they both use grafana now grafana is a way to query and visualize almost anything which seems like a great fit for logs as well so if you're already using grafana or investigating it loki will fit right in so how does this work well there are three parts really to grifana loki first there's the agent the agent's name is prom tail and prom tail works in a push configuration and not a pole configuration so that means the agent is going to push logs into loki rather than query devices and pull it and so this one or many agents which is pushing logs will push them into loki itself and loki is where all of these are stored but once your logs and data is inside of loki you can then use grafana to get it out to query it and to visualize it i know this might sound a little complicated and hard to set up but actually it's surprisingly easy so that's what we're going to cover today so first we'll configure loki and get it up and running so we can accept some logs then we'll install our agent prom tail so we can push logs into loki and then we'll get grafana up and running and we'll configure the loki plugins so we can start querying logs and after we get all this working we'll explore a few different ways to get logs into our system we'll talk about scraping logs from a machine and then we'll talk about scraping logs from a docker host which is really awesome because all you have to do is set it up once and you can collect all the logs from all of the containers running on that machine and it only takes a small amount of config and then we'll touch briefly on how to set up a syslog endpoint so you can send logs from any device that supports this log and then we'll also touch on how to send logs from a whole entire kubernetes cluster but i'll go easy on the kubernetes i don't want you to fall asleep and if any of that sounds good to you don't forget to give this video a thumbs up and subscribe if you haven't already and like all of my videos i'll have a link in the description to my documentation site where you can copy and paste all of this config so what do you need to get started really you just need a machine that can run docker that can be a windows pc or a mac or a server or even a raspberry pi any of these are fine to run docker to be able to spin these containers up and if you need help setting up docker or docker compose that's also in my documentation so once you have daca running you'll just want to make sure it's working by running docker dash v see that you're good next make sure you're inside of a directory i don't know where else you would be but just make sure you're in the right one so we're going to create three directories one for each of our containers so first we're going to do a mkdir prom tail and that's our agent for our logs then we'll make a directory for loki then we'll make a directory for grafana next we'll want to create a docker compose file for all three of these services so we'll do a touch docker-compose.yml for yaml and then we can edit this file so what goes in the file inside of the file goes our loki stack so now when i say loki stack i'm referring to prom tail loki and grafana and here's the configuration that we're going to use so let's talk about it real quick so this is a typical docker compose file we're going to use docker composes api's version of three then we're going to create a network called loki now this is important because all three of these containers inside of the stack are going to share this network name so the first service is called loki and the image we're going to use is grafana loki and then we're pinning it to the latest version then we're going to create a volume for it so this volume can be anywhere on this machine that you're on but i'm saying that mine is in home slash server admin slash docker volume slash loki and then we're mapping that to the inside of that container that runs at c loki then we're exposing some ports 3100 on the outside 3100 on the inside and then we're saying restart unless stopped and this helps if the container starts crashing next is a command when this container starts up and we're saying dash config.file equals etsy loki loki dash config.yml now this is our locan config and we'll have this in a minute but if you notice this is going to map to the inside of slash loki here and that mounts to the host so that's why we're going to put a config file there here in a little bit but i'll show you that here in a second and we're saying hey this container belongs to this network called loki the next service in our stack is promtail now remember this is our logging agent so here we're saying use the image grafana prom tail and then a version and we're pinning it to that version then we're mounting a few volumes here so we're mounting bar log from our machine to var log within that container and then we're mounting slash home server admin docker volume slash prom tail on this machine to the containers at c slash prom tail and this is just for some persistence here and again we're saying restart unless dot and then again we're using the command of dash config file equals etsy prom tail prom tail dash config so same idea as griffon at loki up there we're going to specify a config file for this to use so when it mounts up and runs this command it can use that config file at runtime i'll show you that here in a minute next is grafana so this is actually going to spin up grafana this one we are pinning the latest because that's what grafana says to do next we're setting a user of 1000. now this helps with permission on the volume that grafana needs to write to now the way you can find this is just running id and you can see your id right here the uid of a thousand again that might be different from you but take that volume and plug it into the config then speaking of volumes and persistence this is where our volume is going to be stored for grafana so on the outside or the host it's server admin slash docker volumes slash grafana and then on the inside it's var lib slash grafana so this will persist grafana itself so that every time you reboot the container you don't lose your configuration for grafana and next is ports of 3000 on the outside and 3000 on the inside now i have a quick comment here that says you might need to change this and this applies to every port 3000 is a pretty common port so is 3100 so if you need to change those you would change it on the outside here on the left side this is the one that's exposed on your container and 3000 is a very common port for me because i run a lot of node stuff and they all use 3000 so i had a comment here specifically for myself but this applies to every port you see in the configuration here remember change the left side left side is your configuration right side is the containers configuration which you can't change unless you rebuild the container and then we see restart unless stopped again and then networks loki so this ties them all together so they're all using the loki network so they can all communicate to each other within this stack so let's take this configuration here and we'll copy it and we'll go back into our server for this docker compose file and we'll paste our configuration in there then we'll save it then if you do an ls you should see your folders there now you see a couple other folders that i have there but you should see grafana loki and prom tail there and then our docker compose file that we created but before we start it up we need a couple additional configs remember i mentioned earlier we need config for prom tail and we need config for loki and that's specified here so let's look at a file called loki-config.yaml here's your configuration for loki a hundred percent of this is the default setting that they say to use for loki it has some configurations for servers ports some paths within the container schema config and a whole bunch of other configuration now you shouldn't need to change anything in here unless you have a specialized setup or specialized needs for other systems that you're running so let's copy this configuration and then go inside of our loki folder and create a loki-config.yaml file once we create it then we can edit it and we're going to paste that configuration into there so let's save that and close and now we'll need to do the same thing for prom tail let's go one folder up then go into prom tail and let's do a touch tom tail dash config dot dml then we can edit it and here is our prom tail config and remember prom tail is our agent to scrape logs so if we look here some more configuration for the server what port it's listening on a positions key that points to a file name that points to a yaml file within the container then we'll have a client's url with loki at 3100 loki slash api slash v1 slash push so this is the endpoint where it's going to push logs and it could refer to it by loki because it's within this stack and we can refer to it by name and then we have some configuration for how to scrape logs now this should be a common but local machine logs we're going to configure first so we're going to scrape logs from this local machine and then push them into loki and it's saying the target is localhost and the job is called var logs and the path to var log and take all logs within var log and then some additional configuration commented out here that we'll talk about here in a little bit so let's copy this paste it into prom tail dash config and let's save it and if we do an ls within prom tail we should see our prom tail config and if we go up into loki do the same thing we should see our loki dash config if we go up into grafana and look in there we shouldn't see anything so let's go up one more level and now we should see our docker compose file that we created so now let's spin this whole entire stack up the way i typically do it is docker dash compose up dash d dash dash force recreate there's a million ways you can do it that's the way i do it here so if we run this command this will now create our whole entire stack so hold down loki pulling down prom tail and it's pulling down grafana now it's creating those containers done done done so it looks like we're running if we do a docker bs really quick just to make sure we should see all three of these running so prom tail started 10 seconds ago loki started 10 seconds ago and grafana started 10 seconds ago so we're looking good so how do we know it's working well we go to the ip address of this server and we can go to port 3100 metrics and you can see here i have some metrics already you can also go to slash ready to see if it's ready now obviously it should say ready if it's ready if it doesn't say ready it'll usually say it's waiting 15 seconds and checking the ingester again so just keep refreshing this page so this is a good sign our gesture is ready and we are getting metrics as shown in this endpoint so now we can actually log in to grafana so how do we do that well it's just port 3000 so 3000 is where grafana lives so if you haven't logged into grafana before the username and password is admin then you'll be prompted to create a new password and then once you sign in you have grafana so if you're not familiar with grafana this really isn't the tutorial for grafana i might have one in the future but you get a platform for visualizing logs metrics or really any data but we can also query that data and that's exactly what we're going to do with our loki stack so the first thing we need to do is actually set up our data source for loki so that we can query the logs so let's go into settings here then let's click add data source and we should see loki down here so let's select loki and in here very little configuration but the url of our endpoint to query now it's suggesting http logo host 3100 but if you remember in our configuration it's an http loki 3100 and we think either will work but i think they recommend to use loki and once we have that filled in we can actually save and test so now our data source is connected and labels are found so that's good so now let's explore this data our query our data in here so let's go to explore and you want to make sure that your data source is set to loki here now how do we query now teaching you log ql which is the log query language for grafana is kind of out of scope for this tutorial but i will give you some examples on how to query data and it's really straightforward at least for the simple queries because they suggest labels for you you get kind of intellisense so first we'll start with the curly bracket and right away it's saying file name or job so these are tags from prom tail that it's sending into loki and now loki and grafana are recognizing that so let's say job let's choose job what job did we have for our logs and var logs was in our prom tail configuration and then if we hit enter don't just hit enter but hit ctrl enter the enter is going to create a line break but just hit ctrl enter and right away we can query so these are the logs coming from this machine right here var logs we got logs from this machine doing almost nothing at all and so you can see all kinds of logs in here so you can see juno kernels juno is the name of this machine you can see there looks like some info messages here some warning messages here and a lot of logs that may or may not make sense to you depending on the type of logging you're used to looking at you can actually see one here there's an error here and so we can search for strings within here too but you'll always have to remember that you have to select first by a tag so i think the way that we search these logs for a loose string is the intuitive pipe equals and then a string in quotes so i see down there lots of juno's but i see docker let's search everything within here for docker and here we go so we just filter this now to docker you can see how fast we can query through these logs we can drill into these logs look at additional metadata we can include in our filter or exclude from our filter oh we just filtered out this entire log that we're in so we don't want to say not job there we go now if you're interested in doing some more advanced querying i highly recommend checking out their log ql syntax but really it's not that complex for me most of the time i'm looking for a needle in the haystack i'm looking for a specific phrase or keywords within my logs and then i'm able to narrow down and figure out maybe what the problem is so this is cool but this is only for this machine's logs which may or may not be helpful if you remember in prom tail we had some additional configuration commented out that was configuration for our docker containers this is where it gets really cool because currently the way that you check for docker logs is you do docker logs then choose your container we want to say heimdall we say docker logs heimdall we don't see any errors there so then we see docker logs and let's choose another one you can choose poor tanner see some errors or some logs there and then we can do docker logs up tankuma check out uptime kuma and it has lots of errors this isn't efficient or fun so to make this more fun and more efficient we can actually configure prom tail to log all of the standard out all of these logs that we see here to loki and one piece of configuration super cool so let's go back into prom tail and let's edit this config now i'll bring this up in vs code so you can see a little bit better but instead of scraping logs from the machine let's actually now scrape logs from docker so this is saying hey the job name is called docker there's a pipeline stage called docker and then we're going to create a label call docker and then we're going to grab all the logs from docker slash containers and then everything within here every subfolder that has the file name of dash json.log so this does exist but we need to make one more configuration but let's get this in place first so let's copy this let's get rid of our prom tail config or we could have uncommented that out and let's update it with our docker locks and we'll get rid of the host machine logs arguably you can keep them there totally up to you but let's save this close out of here so in order to log all of our containers logs to loki we actually need to install a docker driver or a plug-in for docker so that we can push all of those logs to loki and so there is a plug-in to do this and it's actually really cool and super simple to spin up so to do it we need to install the grafana loki docker driver client and to do that it's just as simple as running this command right here so docker plug-in install grafana loki docker driver latest so this is the container alias is loki and grant all permissions i know that sounds scary but that's how we get access to all of these logs to send them into loki so let's grab this go back into our machine and then let's run this so this pulled down the container and configured the driver for us there's one more thing we'll need to do first just check to make sure it's installed so docker plug-ins ls and we should see loki here so the loki logging driver and enabled is true next if we go back to where we configured the driver it mentions in here we need to change a couple of things so we're gonna have to configure the daemon config the daemon.json located in etsy docker on this machine so we can configure it in two ways there's a shorthand version and then there's the configuration with some additional options so there's the loki server and then there's the loki batch size so how big is the batch of logs to push and then for our configuration for our daemon.json for docker it's going to be log driver is loki log options and then inside of this object we have a property loki url which is the url of our server that we're on and here we can refer to it as localhost so you should be able to copy and paste this no problem and then the batch size of 400 again let's copy this and remember it said that we need to update the docker'sdaemon.js and etsy docker so on our server or machine we should be able to edit etsc docker daemon.json now this file may or may not exist so you'll just want to edit it and if it's not there it'll create it and here we're going to paste our options say this then it says after changing this file we'll need to restart the docker daemon for the changes to take effect all newly created containers from the host will then send logs to loki via the driver so keep that in mind newly created so we have some containers running so we might have to recreate them but that's super easy with a command but the most important piece here is that we need to restart the docker daemon so we should be able to run sudo system ctl restart you should be able to choose docker okay it looks like it restarted docker docker dash v just to make sure it's running and it is okay so let's go back to grafana now really quick so if we go back to grafana refresh to make sure it's working and make sure you're in explorer go back into our queries we don't see additional labels yet so we should have one here called docker and containers and a whole bunch but remember i said in the documentation it said only for newly created containers so let's actually recreate our containers that we're running so in order to recreate these at least for me i have all of my docker compose files within a folder so if i go into heimdall i should have my docker compose file and i can run docker dash compose horse recreate so recreate my heindall container and let's do the same for my uptime kuma and let's do the same for file browser and that should be good so let's do a docker ps to see what's running i have containers running i see that some are recreated and restarted so now let's go back in grafana we go back to grafana and we go into our query when we do a curly bracket now we see some additional labels so this is really cool so we can say container name equals and it will suggest one so container name equals heimdall and control enter and we can see the logs from heindel right here so this just happened really cool really cool okay so let's go back and if you hit control space at any time you'll get intellisense or suggestions just like you would in any ide or vs code let's choose file browser hit enter control enter i am hitting ctrl enter but control is broken on my keyboard i need a new keyboard keep telling myself that but ctrl enter the other control enter and we can see our logs from file browser awesome i have an error here but that's expected that was from my demo and then if we look and what was the last one uptime kuma let's see how he's doing control enter and we can see our logs here really cool and you can actually combine these logs too if you want to say uptight kuma and this doesn't make a whole bunch of sense but the host is juno which it is you can hit enter so if you were gathering logs from multiple docker hosts where they both had uptime kuma container names this would filter on juno which i only have one server running but if i hit filter on tests this will probably throw an error because i don't have one called test but again if i wanted to search for uptime kuma search and then if we wanted to do the pipe equals the most intuitive thing uh let's see uh k3s and enter now we just filtered all of our logs for the container name optim kuma that contain k3s in there so pretty pretty awesome pretty powerful stuff and so if you have 20 30 10 to however many containers running now all of those logs will be captured inside of loki just by editing the daemon jason and then having a config within prom tail it's a lot of words i know but it'll make sense i promise now if you wanted to save this query say in your favorites you could you could bookmark this or you could just go into query history look at the last query we just ran and then start and then to get back let's say that was closed go into query history go into star and then you can run this query and the really cool thing about this is it fits within your ecosystem if you're already using grafana so a couple more things i mentioned i'll touch on real quick assist logging so you can actually set this up to be a syslog target for any of your devices so let's say you're running systems that aren't containers let's say your network firewall or your virtual machine or your windows machine or your raspberry pi and you want to pipe those logs into loki as well well you would just set up prom tail to host a syslogging endpoint and so to set it up as a logging endpoint or a syslog receiver we would add this configuration right here so this configuration would actually go into prom tail i'll put it in there because it'll make a little more sense but if we were to add it to our prom tail configuration we could add it here and you could obviously comment this out or you can add two jobs but you can have multiple jobs i just commented these out but by adding this configuration we have a job now called syslog it's gonna listen on 0.0.0.0 on port 1514 and listen for syslogs to come in and so if you enable this then you're able to receive syslogs from any machine now a word of advice should either format those syslogs going out or format them before they come into here now it's not a big deal most of the time they'll look pretty good but depending on your device and how it formats syslogs you might not see them in the way you would expect to see them within grafana and another thing before i forget because i just noticed this port 1514 they'll have to open up on your prom tail docker convict but if you need help with configuring that leave a comment below and we'll try to work through it but that should be all you need to do so the last thing i mentioned was kubernetes not going to spend a ton of time on it but you can apply the same idea that we applied to docker to kubernetes so just think about it in kubernetes you have many containers running many pods running and you want to get all of those logs inside of loki well you have a couple of options one you can spin up prom tail there and dump all of those logs into this container or you could run your whole entire loki stack in kubernetes which is super easy there's a simple helm command you can do to run it and i'll actually have a blog post article in the documentation below of how to do it in kubernetes and literally it took me 10 minutes to get running if that a lot faster than the docker version because most of it's pre-configured with helm helm's just a package manager for kubernetes but anyways once you get all of your containers dumping their standard out and standard air into loki it's really powerful so here's my own home production version of loki that's actually running in kubernetes and it's scraping all of the logs from my kubernetes containers and dumping them into loki kind of inception but it works great so right away if i do curly bracket you can see i have lots of labels so these are labels that are specific to kubernetes that it already has and discovered you can see i can type in container name equals and i can see all of my containers that match this query then i can query this container and look at them so not much different from the docker version or any other version but i get all of these labels automatically as these logs come in so i could say the app kubernetes io instance is loki and see the logs for loki or i can say the chart is prometheus and look at the logs for that there's no logs there it's a weird label but you get the idea or i can say the specific pod name you can see my pods if i had multiple pods running is my technotem little link server you can see them here and so you can see how powerful this is i can log physical machines network devices docker containers kubernetes clusters really anything really fast and the nice part about it is it fits right into the grafana ecosystem it's always been compared to prometheus but for logs and that seems exactly what it does for me it takes that same idea of streams and labels and allows me to search and index and query the same way i would with prometheus and if you're familiar with grafana you can actually create dashboards as well based on some of these data just from these logs now maybe i'll save that for another tutorial or a grafana specific one but it's all definitely possible and if you're thinking about adding grafana or maybe prometheus maybe adding loki and logging will tip the scales for you so what do you think of grafana loki do you like the idea of using grafana to actually query your logs inside of loki do you like a different logging system altogether let me know what your favorite logging system is in the comments section below and remember if you found anything in this video helpful don't forget to like and subscribe thanks for watching and you guys are lightened up by my raspberry pi back there i don't think it stopped i don't think it stopped yet so thank you appreciate it everything is going to blow up no it's not going to blow up if it does i have automation in place to reboot it so thank you so much appreciate it

Channel: Techno Tim

Views: 45,501

Rating: 4.964602 out of 5

Keywords: loki, homelab, home lab, technotim, techno tim, prometheus for logs, kubernetes, docker, syslog, grafana loki, promtail, config, self-hosted, selfhosting, logging, logging server, elk stack, elk stack alternative, kubernetes logs, log all pods, containers, syslogging, kubernetes cluster logging, grafana, grafana loki tutorial, docker compose, log all the things, best logging system, grafana loki dashboard, grafana loki installation, grafana loki kubernetes, grafana loki vs elk

Id: h_GGd7HfKQ8

Length: 28min 13sec (1693 seconds)

Published: Sat Nov 20 2021