OpenTelemetry Log Collector on Docker | How to Setup OTEL Logging locally with Docker | Loki Logs

hello everyone this is Bish welcome to my channel so thanks for tuning in back so today we'll be taking another interesting topic in open Telemetry uh this is in line with the previous session that we have it on open telary on you know Docker based environment so if you have not know look at the other videos you can simply go like And subscribe the channel then go to the uh open Telemetry playlist and then you can watch the other videos before we get into the detail of this today's session what we'll be doing we'll be doing a quick recap so what we have done so far we have done open Telemetry monitoring of Matrix locks and traces on kubernetes based envirment in my first video in the second video we have done the same thing on Docker B environment and today we'll be looking specifically how to monitor logs from a from a containerized application and send those logs specifically via open Telemetry or hotel collector to a to a locky receiver which will be ultimately visible onto the Gana so that is the ask for today so what we'll be doing uh to to make it you know as an accomplishment will be first of all installing uh Microsoft's app as a Docker container then we'll install open tetri collector as a container and then we'll install locky as a data source so that we can send the locks to the data source as a container and finally we'll see all the logs onto the grafana just to make sure that everything is coming properly okay let's quickly get started before we waste any more time so what we have done we have created a very small GitHub page for the same you can simply come onto this page there a read me logs open repository and this page will contain all the relevant steps that are needed to do the same thing onto your Docker based environment so let's quickly see what we are trying to do in this uh so open the same documentation on the uh vs code so yeah the are these are the definitely reference documentation so we'll be using this lck key receiver reference documentation which is actually helping us to create a oral ortal collector configuration file and you can just read more about it how uh the locks will be shipping from uh you know container to locky receiver and finally uh to collector and going to the uh locky database so let's I'm not going to explain the details of how Hotel collector was because that is already being explained in previous video but let's get quickly onto the Practical implementation of it okay so I'll simply okay let's quickly understand a little bit context or the theory about it so suppose you have a application that is successfully emitting metrix traces using oel collector and your logs is emitting and sending it via promil agent now what you need to do you need to just have open Telemetry collector as a as a you know imediator so that uh first of all prom will send to open Telemetry collector on port number 3500 and then it will be sent to the uh you know uh locky via locky receiver which will help you to do that via pipeline so we'll see all those configuration in a while so you can read more about this text and the you know Theory I'm not very keen to do theoretical things but yeah let's quickly clone this repository go to this folder cdel logs and then run this docup compos file I'm not running it directly first of all I'll go to this folder and then I'll explain what this Docker compos file contains so let me close everything so that it becomes cryst still clear now the same thing is being opened here so I'll go to the hotal logs I'm inside this folder now before I run that command I'll just show you what is happening so first of all this is the version networks is locky now we are actually initiating few Services basically if you see overall there are only five services or five contain that I'm bringing it up so first of all is my locky as a data source I'm bringing it up this is the latest IM and pulling is running 31004 and this is the configuration file of Loy which is being stored as a stored within the container now second service which I'm bringing it up or container which I'm sigana which is latest uh the image is 10.2 now these are the environment variable that I'm passing into my grafana contain that I'm making sure that uh disable login so I'm disabling the login prompt so that it should not it should not ask me to login bya username and password Anonymous enabled is true so that anyone can login directly and Anonymous enabled which is through is will get all the admin permission there is the admin rule so we will see this importance you know once we run this Docker compos file now these are the entry point now what what I'm doing additionally within the container I'm creating this folder and in this folder data source folder I'm creating D.L the content of ds. yl is your lucky data source and that will be added to your grafana the moment you run this file and grafana will definitely run on port number 3,000 which actually depends on this locky container first of all this will get up and then this will get up fine and then the promp tail is an agent which will actually uh read the logs from the container and then it will send to connect so just pulling this image uh this is the configuration file which I'm passing it from my local current directory to The Container as a read only so let me just show you this uh prompted local configuration file so this is straightforward I'm just saying that this is the server client this line is very important so I'm saying whatever logs that are being read by fromom tail agent and this is actually script configuration you can see it is reading the container name and just uh taking the uh uh uh container logs from this job name and then pushing it all everything to the hotel collector on this 3500 Port we'll see the total collector container in a while but this is what is the configuration that is pass for a prom fine and this is getting started as a configuration P now let's see FL fla is something but a small you know container as an application which is just running few commands Okay just you can consider this as as a application container now this the important container which is actually bringing up the hotel collector and open Telemetry collector conflict this is a version that I'm bringing it up and now what is the configuration file that I'm passing for collector this is that configuration f ig. is the same folder and this is gets inside into the Container this is running a let's see this otal config file now otal config file again have three important parameters if you remember my previous video receivers processors and exporters and finally the service which is the pipeline so receivers is nothing but your locky receivers which I was explaining it here this is a locky receiver which is first of all get the logs from this receiver uh the locky which has already been configured and then it will process it will do the uh you know labeling uh with the container name that the locky attribute labels will be renamed as a container attribute and then all the raw format logs will processed and then this is the exporter exporter is something where your end result will go so ultimately you know prom till is sending to collector from 3500 and this is the collector configuration so this is actually sending it to locky data source so this is how it works first of all uh application sends to promail fromom tail actually collects the log sends to collector on this particular Port 3500 and then finally the configuration send to lock dat so the beauty of is you know in future you can simply change this exporter to something else instead of lock you can have something else right so that is why we use aut collector and if you see the pipeline is nothing but doing all all these things in you know seral manner so I'm just running on only locks pipeline this time I'm not running you know the Matrix Andes pipeline so receivers is this one lock herey these are the processor and processing this attributes and finally the exporters is nothing but this I think this is Crystal Clear from understanding so basically we have only three files Docker compose which is actually bringing up five containers and then these two things let me get back to my documentation again let me run this command it will just bring up all these containers one by one okay I'm to this location okay so first of all it is it is created the network what locks lock and then all the five containers so let me see whether the containers are running or not clear Docker PS okay all my containers are up and running let me see whether the graffa is up and running or not so I'll just do Local Host so now I'll just explain the importance of those variables environment variable that we have passed so if you see come back here the uh do comos F the Gana so I have been this Au disable login so it does not ask me I kind of login user ID and password see that is disabled so that is the thing that we have done fine so so I'll just go to my Administration go to my databases as the databases lock which is gets added now how you may ask that how this lock has been added automatically uh while bringing up the container so that is already being defined in this piece right we are saying that please bring up add this data source into your um while you bring up the container of so this is fine now let me show you the explore section we we can explore the lock key select the lock key and then I'll just select uh the container and I can select any container and then run right I can get all the logs pertaining to this container now I'll see another label which is exporter which nothing but OTP so you can see OTP exporter this is the additional label that's get added when you send logs via you know o collector if you're sending directly via prom to Loy then you will not get uh you know this OT a label let's add the Json passer you see the some it's okay but you can see the Json related uh labels so you can explore lock in a better way you can do like this also so suppose the container I can okay let me show you one more thing so the moment I hit container label I can see all the containers so you can see all five containers are coming properly and then I can see the logs of any one particular container like this correct okay so you can see two labels are coming so we can have we can add more labels by following uh the the Orthodox documentation of L key in this so if you come down here so this is that locky format and yeah this is the one relabel config so we can have more Rel Lael configs and have more Target labels according to that for now I'm just taking the metadata Locker container name okay so that is that is what pretty much about in this session if you have any questions and cl clarifications what kind of videos you're looking for open Telemetry uh we can prepare that as well into the comment section but do not again do not forget to like And subscribe the channel because now we'll be doing lot of sessions or videos on open Telemetry and kubernetes and grafana so that it becomes you know easy for deployment and instrumentation everything is open S you can get all these documentation easily on your you know Google and everything since the beauty of openet it is it is you know available on this uh reference documentation so these are two reference documentation that has been used to uh to complete this video I mean we have successfully sent the logs from uh container to the perana using M and a locky receiver there are a number of ways to send locks but yeah we have chosen today the locky receiver so I think let me wrap up this video thanks for we'll see you soon in our next session thank you

Channel: Bhoopesh Sharma

Views: 891

Keywords: collector, grafana, how to use opentelemetry, loki, observability, observability aws, open telemetry, opentelemetry, opentelemetry .net, opentelemetry .net core 3.1, opentelemetry .net core example, opentelemetry .net example, opentelemetry asp.net core, opentelemetry c#, opentelemetry collector, opentelemetry custom metrics, opentelemetry distributed tracing, opentelemetry dotnet, opentelemetry logging, opentelemetry tutorial, otel, otel collector, what is opentelemetry

Id: vEGH8sH5xO4

Length: 14min 2sec (842 seconds)

Published: Wed Nov 22 2023