.NET | Log into Elasticsearch using Serilog and view logs in Kibana

all righty so in this video we're going to take a look at how you can set up your net cor application to log data into elastic search and use kibana to view that data my name is Vil Len and you're watching the architecture and detailed design Series where we are building a notification system from scratch using industry's best practices I already have a small video on how you can set up your donet core application to send logs into SE using serlog but I'll reiterate over what serlog is nonetheless it's basically one of the most popular logging libraries it's really easy to use Easy to install and has a wide Community Support basically because most of us are using it nowadays with lots of different nugget packages in forms of SNS and enrichers to get started with serock we're going to need a couple of nuget packages that I've already pre-installed over here so we're going to go to our directory packages props and we can see a couple of packages over here starting with serock hnet core serock do and reachers environment process thread exceptions settings configurations syns for console debug and elastic search here and a small explanation should be done about what is a serock sync and what is a serock enricher in short serock provides you this sync packages in order to let you log data into some kind of Provider easier basically you can just search for nouet over here and just type in Ser lo. syncs and you'll see a bunch of different providers over here the SEC that I mentioned previously HP data dos Flack Etc it's basically a lot of different places where you can send your logs into and enrichers are basically nothing else than ways to add additional data to our logs and our case we're using the thread the process environment and serock comes with a couple of prebuilt enrichers also now once we have all the nuget packages in place I can close the directory packages like here and go to our utilities folder and over here we have loging and loging installer and these three lines of code are basically everything you need to write in your C code well with the exception of the up settings but yeah C code related it's only these three lines of code that we're going to need and we basically say that we need the a new logger configuration we want to read it from the config ation that gets passed into the installer which is obviously our app settings file and want to create the logger this method is basically available with the installation of this one serock settings configuration nug package otherwise you're just simply not going to see this method over here if you try to install it in a class Library like I do here inside the shared class library now the next thing that we need to do is basically set up our app settings and inside app settings we can see this small Ser loog setup it looks a lot like the logging one since we can see the minimum level and basically I have a default minimum level and I'm overwriting Microsoft h. core to minimum level of warning if you want to have a course or a video series a long video series on logging and how to set up and what's here at all leave a comment down below other than that we can see that we have another field over here right to which is essentially us setting up our syns and basically over here we have two thingss first of all we have the console since we want to see the locks in our console and then we have another object over here which is basically our sync to elastic search with the name elastic search and then we can we should pass a couple of arguments over here the node U which right now is running in a Docker container on this port on my machine and then one of the more important things over here is index format why I say important because we're going to use it when we're going to set our kibana dashboard index format is notifications uncore system Dash and the current date the next thing that is interesting to us is this enrich part which basically tells us about the enrichers so I want to use the from Context with machine name with thread ID and reachers if you want to use additional syncs or additional en reachers you'll have first to install the nougat packages they won't work like you referencing here some richer or something that you have not installed via nuget package Into Your solution it doesn't work that way you first have to install the Nugget packages other than that we can also add additional properties that we want to our logs in our case it can be for example the application name can be some combination of application name for example node name or some form of maybe mention the environment where this is running obviously it's up to you what kind of properties you want to add to your locks and the last part that we need is basically the docker compos file which in our case is pretty simple we have two additional containers over here the first one is being elastic surch so you have the container name the docker image is basically from the docker elastic code the elastic search the ports is 9,200 then you'll also need to specify two additional environment properties the discovery type and XA security enable which you know our case is set to single note and false we will set a volume to keep our SL blocks intact even if we restart or delete the container in our case the elastic search and last we're going to set up kibana the container name is pretty obvious the image you can see it by yourself the ports 5601 it will depend on elastic search in our case so we do not start up the kibana without first starting up elastic search and last but not least the elastic search URL which in our case is basically indicates to the port of our elastic search container that's everything that we need to do and basically we can now just start up our application and with that I can go to Safari and basically double check two URLs first of all that our elastic search is working on Local Host 9,200 and basically you can get this kind of result from it that's fine and then we're going to go to our kibana which is on Port 561 and start adding our logs we will go over here to stack management and go to index management over here we can click on the index name and copy it from here or you can go to your app settings and copy the index name from there then go to discover we will be welcomed with this small message over here where we are asked to create data view so I just want to create data view name it notification system default and basically we can add over here an index pattern in our case it's play notification on underscore system and you can see that it doesn't match any existing index that we have since we add the date of the log to the index name however what happens if I want to see all the logs from all the dates we can basically just add a wild card over here and we can see that your index pattern matches two sources and yes I want to view and save this dashboard and as you can see we already have a lot of flow LS flying in over here from our system we can go over here and basically expand on it and see all the properties that we have in our structure log so basically we have the message the machine name um the command type the event ID itself the index notification ID and somewhere over here we can see and somewhere over here we should see the application name yeah is the property that we set in our app settings over there so you can for for example create some queries for specific applications if you have a single index where more than one application dumps its logs so with that basically your API already sends all the logs over to elastic search and you have kibana to create dashboard on top of the logs for searching for debugging purposes in general as I mentioned in my CQ video the approach that we took with the configurations part in the beginning with the three lines of code of c and most of the config ation coming from upsetting that's my preferred approach since it's easier for devops guys to configure logging in our system and basically they don't need our interaction on any small change over there if you like this kind of content leave a like comment down below subscribe to the channel and click the notification Bell it helps me a lot and until the next video comes out have a nice one

Channel: Vasilii Oleinic

Views: 2,499

Keywords: dotnet, dotnetcore, REST, REST API, MinimalAPI, Monolith, Microservices, Modular-Monolith, Architecture, Detailed-Design, Design, Logging, ElasticSearch, Serilog, Kibana, Docker

Id: B79IOUz77xo

Length: 8min 52sec (532 seconds)

Published: Sat Nov 11 2023