[ Kube 100 ] Getting started with Grafana Loki in Kubernetes

in this video we're going to take a look at a tool called loki which is a log aggregation tool it's similar to elasticsearch but it is slightly different the way it stores the logs and the way it indexes the logs so let's say you want to collect locks from all your parts all your containers gather them in one place so that it becomes easy for you to search for something and visualize it so loki is the right tool for you and we're going to look at loki in this video i've got my linux machine here i've got arch linux installed and i've got a kubernetes cluster and if i do cube cd i'll get notes i've got one master node two worker notes and this has been provisioned using my vagrant environment and all these are virtual machines running in virtualbox okay and let's go to the documentation i search for grafana loki alright so i'm gonna go to the loki documentation and going to go to the installation and i'm going to prefer the helm way of installing loki installed through helm all right the first thing we're going to do is we're going to add the chart repository for grafana loki so copy that and before that let me show you i've got helm repo list i've got nothing but one chart repository that's for the the nfs the dynamic nfs provisioning okay so i'm going to add this new repository and do a helmet recall update and now i can do a helmet repost search sorry helm search repo i always get this wrong helm search repo grafana and all these charts are provided by this repository and the one that we are interested in is loki stack loki's stack is a a set of components uh like you've got loki that stores the logs that's the log storage engine uh which also indexes the logs uh it doesn't index the logs but it indexes the metadata of the logs and then we have prompt tail so that's the agent that gets installed on all your worker nodes or all your kubernetes nodes that is responsible for collecting logs from all the ports running on the respective nodes and then it sends it to loki where it stores indexes and everything and then we have grafana as part of this loki stack and through grafana you can visualize the logs you can explore the logs you can search locks and things like that so we're going to be installing loki stack and the version of chart is going to be 2.4.1 and the app version is 2.1.0 but if you're watching this video in the future the versions might have been changed so i can't guarantee the installation process that i'm showing you today might be the same when you're watching this in the future but back this in mind for this video i'm using the chart version 2.4.1 and the app version is 2.1.0 right so we've added our helm repository and now we are good to install loki stack okay so there are different ways to install it comes with different components what i usually do is if you want to install something from helm i first download the values file so that i can understand what can be configured what can be changed and things like that all right so for that i'm going to run this command helm show values that's grafana is the chart repository and loki stack is the chart that we are trying to install and redirecting that to a temporary file so that is going to give us the values file that we can configure okay so if i edit that file temp loki stack values.here also these are the values that you can customize so i'm not interested in filebeat and i'm not interested in logstash so i'm going to delete all these okay i don't need prometheus right so we have loki that's enabled we have prom tail that's enabled we have fluent bit that's disabled i don't want fluid i'm gonna delete that instead of prompted you can use fluent bit if you like so frontal or fluid bit is the agent that's gonna get installed on all your kubernetes node that is responsible for collecting logs from all your parts and sending that to loki all right so i'm good with promptile and i'm also going to enable graphona which is disabled by default in this particular chart version so i'm going to say true all right so that's all you need loki prontail and grafana enable grafana and the grafana version is 7.5.0 okay so one thing to bear in mind at this point is if i show again the values.yaml file this one hasn't got any persistence enabled so we need to enable persistence for loki at least because loki is the part that stores the logs the locks that are collected from all your parts through prom tail is stored in loki so you need to enable if you are if you're just trying out loki then that's fine but if you are interested in running this in production make sure to enable persistence for loki otherwise when you restart the low key part all the data collected so far will be lost i've got dynamic volume provisioning installed in my class so if i show you cube ctl get storage class i've got a default storage class this is the the dynamic nfs provisioning so whenever i create a persistent volume claim a persistent volume will get created automatically for me and if you're searching youtube for just me dynamic nfs you'll have these two videos q23 is the one that i did two years ago that's not relevant anymore but cube 23.1 is the recent video that i did and i'm using that setup for this video okay so by default as i've shown in the values.yaml file persistence is not enabled so i'm going to show you how to enable persistence so for that i'm going to go back to the documentation and here it says deploy loki stack with persistent volume claim okay so you can either run the helm command by passing all the values that you find in this value.yaml file directly in the command line using minus minus set but i prefer downloading the values.yaml file and version control it so that i know what exactly has gone into each of the release okay so the things that are that i'm interested in are grafana they're enabling grafana which we did exactly here through the values file and then prometheus.enabled is true but i don't want prometheus so i've disabled it prometheus.alertmanager.persistentvolume so if you're using alert manager you make sure to enable persistent volume for alert manager as well but for me it's just the low key so this one here so i'm going to copy this paste it here okay so what i'm going to do now is okay let me turn off the syntax highlighting so that you can see better right here under low key and i need to have persistence colon and then enabled colon true and then we have storage class name is here it's set to standard so in my case the storage class is nfs dash client so that's my storage class but i don't have to specify this because that's the default storage class anyway so i'm not going to specify this but if you don't have any default storage class make sure to specify your storage class here and then finally the size size is set to 5 gigs set to whatever you like so i'm gonna set that to one gig so that's it and that's the line just for reference okay i'm deleting that line so that's all you need but as i said i'm going to delete the storage class name because i've got the default storage class that's going to get used okay so that's all i need in my values.yaml file all the changes that i have done is enabled grafana and then enabled persistence for loki all right so now let's install this and stack the local the loki stack okay the command is held installed loki stacks that's the name that you are giving it for this release it can be any name and then the chart repository the chart that you want to install from that chart repository and the values file that you want to use for this installation because you've changed something and then you if you want to install this on a specific namespace pass minus n low key so either you can create this loki namespace yourself or if you want helm to create the namespace for you just use this option minus minus create namespace all right so i'm gonna hit enter now okay so all these warnings should be gone when they update the chart it's just saying port security policy which is gonna be deprecated after 1.21 now i can do helm list you won't see anything because we've installed this helm release in a specific namespace so i can do helm list minus n low key yeah there we go so that's low key stack and if i do helm list list all the helm deployments on all the name spaces so we have loki stack and the nfs subdir external provisioner for my dynamic nfs provisioning okay cube ctl get name space i've got loki let's see what's being deployed in the loki name space cube cdl minus n loki get all right so we have the loki stack we have grafana we have promptel as you can see prom tail is a demon set it has to be demon set because as you add more notes to your cluster this prompter needs to be installed on all those nodes so it has to be a daemon set and we have a deployment of grafana with one replica and then we have loki stack which is a stateful set it's not ready yet let's give it some more time and i can do cube cdl minus n loki get pv come off pvc there we go so that's the storage persistent volume claim for the low key stack and the persistent volume that got created automatically and it's using the nfs client storage class okay let's do let's check all the pause again get all right so we've got everything running now we are good to log into grafana okay so cube cdl minus n low key get also if you look under service we have this loki stack grafana you can edit the service loki stack grafana and make it a load balancer type service so that you get a load balancer ip and if you're running this on a bare metal um or in a virtualization environment you can use metal lb for load balancing but for this time i'm just going to use a simple port forwarding but if you want you can set up an ingress for grafana or a simple load balancer service let's open up another terminal get all and i'm going to do a port forward low-key port forward so what i'm doing here is in the loki namespace i'm port forwarding the service loki stack grafana to port 3000 on my localhost so if i hit 3000 on my localhost now it will take me to port 80 of this service let's start our port forwarding service all right so that's port forwarding so now i can go here and go to localhost colon 3000. cool so that's our grafana welcome page and what's the credential the username is going to be admin and for password we need to do a bit of hack to find out what the password is so let me show you in the loki namespace if you do get secrets you have the loki stack grafana secret so that's the one that has the admin pass so let's describe that loki stack grafana minus oh yarmull if i scroll and pass the admin password is this one so that's the base64 encoded form of the password so let's copy that and i'm gonna echo and pipe that to base 64 minus d d code okay let me also do an echo so that we get the clear password so that's the password that we're gonna use okay copy and paste right logged in so that's our grafana dashboard let me increase the size of the font if i go to data sources the loki data source has already been added for me and if you look at the url it's loki stack colon 3100 and in here if you look in the service so that's the loki stack listening on three one double zero so the grafana pod has already been configured to connect to loki as the data source okay so now i can go to this explorer and here's where i can query the logs okay so let me show you how to to use this you need to know log ql so let me show here log qr loki query language it's an extensive query language which you need to learn by the way but i'll show you just some quick examples it will be that the format will be like this if you've used splunk or elasticsearch before you might know these syntaxes let's go back here and i'm going to run a simple query and uh there's also a visual help tool that you can use if you don't want to write the query yourself so if i click on log browser so make sure the first thing is to make sure that you've selected the right source here so if you see like grafana in here make sure to select the loki as the data source and then log browser okay so first if i click on the namespace so we have these namespaces so i'm going to filter by let's say cube system namespace and if i click show logs there we go so that's all the logs from the cube system namespace from all the parts running in the cube system namespace okay and if i click that again and if i say i want to have all the parts look at the logs of all the parts running in the loki namespace select that show logs and these are all the logs from fonts in the loki namespace and you can see here so equally you can write the query like this namespace is loki okay let's do something more additional so i want cube system and i also want there's four components 10 containers so if i click on component so i've already selected cube system as the name space so in the cube system name space i want logs from all the parts all the cube api server ports okay and click on show logs and that's it so that's all the logs from the qb api so if i expand one of them you can see the pod is cube api server name space is cube system okay right so that's the component here namespace this is how you can aggregate your query and let's say for example in the namespace cube system i want to look at all the logs of the qb api server where the log contains something okay so that's the actual useful query right i don't want to see all these uh rubbish thing i'm just particularly interested in a specific thing for example okay let's go ahead and do something else namespace is cube system component is hcd show logs okay this is better so let's say i want to filter all the all the log entries that contain mvcc what i need to do is type and add a equal to sign and then enter my search text here my search text is going to be mvcc and shift enter right so now we've got the query so all the odds all the hcd parts in the cube system namespace that contains mvcc in the log entry okay so if you want to do a reverse search or if you want to do a negate of this all the parts that all the log entries that doesn't contain mvc so you just need to replace the pipe symbol with an exclamation so that will give you all the locks that doesn't contain that don't contain mvcc okay so i'm just touching the tip of the iceberg this query language is so extensive here so if you want to deploy this in production i would advise you to go through these documentation the documentation is very helpful you've got very detailed documentation hope you enjoyed this video i think that's all for this video i'll see you all in my next video meanwhile if you've got any questions on this let me know i'll be happy to help until then keep learning and keep on learning bye

Channel: Just me and Opensource

Views: 9,174

Keywords: just me and opensource, loki kubernetes, kubernetes loki, loki log aggregation, deploy grafana loki in kubernetes, grafana loki k8s install, how to deploy grafana loki in kubernetes, kubernetes log aggregation using Loki, grafana loki deployment, log aggregation with grafana loki, install loki kubernetes cluster, just me loki, just me grafana loki, loki logs kubernetes, kubernetes cka ckad training

Id: UM8NiQLZ4K0

Length: 15min 39sec (939 seconds)

Published: Mon May 17 2021