Structured Logging Using Serilog and Seq in .NET

in this video I'm going to teach you how to use the seek structured logging server for Real Time search and Analysis of your application logs we're going to use serilog to implement logging in our application and then we're going to connect it to the seek server to ingest the log data so that we can analyze it to detect any problems first of all let me introduce you to seek or seq it's a self-hosted structured logging server that allows you to perform search and Analysis on your structured logs and you can also set up real time alerting using slack or email if you encounter a spike in your application logs if you're familiar with application insights in Azure seek is a similar service to this except it's not hosted in the cloud you have to host it yourself on your server as far as pricing goes seek is free for individual use and development but if you're working in a larger team you have to buy a license so now that you have an idea of what seek is let's see how we're going to integrate it with ournet application I'm going to install several log to this application and I'm also going to add the seek sync which is a way for S log to write data to other services in this case we're going to be ingesting our application logs to seek so let me look for sarog and I'm going to install the Cog asp.net core package and then I'm going to look for seek and I will install the serilog syns seek Library which is how we're going to connect calog to our seek server with the nugit packages installed let's go ahead and configure s log in our application so I'm going to access the host Builder and I can call the use Ser log method I'm going to use the override that gives me access to the host Builder context and the logger configuration and I'm just going to say logger configuration read from and then I'll call the configuration method and I'm going to use the host Builder context to get an instance of the I configuration service so what I'm actually doing here is my preferred approach to configure sarog with an application and this is using the application settings that we have configured in this case those will live in our app settings Json file I'm going to show you the Json configuration required for sh log in just a moment but first let's go ahead head and Define our seek instance in the docker compost file so I'm going to add another service to my Docker compose file and let's call it eShop seek I'm going to Define which image to use and this will be the data last seek latest image then I'll give it a container name let's call it eShop seek which is following the convention that I already have in place now I'm going to Define an environment variable able to accept the end user license agreement so I'll say accept EA and give it a value of yes let's also expose the ports that we're going to need for the seek server so the default Port is 5341 so that's the port I'm going to expose so expose the port 5341 and I'm also going to expose an HTTP port for the dashboard that I'm going to show you in the browser and let's put it on Port 80 81 so I'll say expose Port 8081 using the internal Port of 80 and with this in place let's head over to my application settings development Json file where I'm going to add my Ser log configuration so I'm actually going to add s log on top of the existing logging section and I'm going to get rid of this section entirely a bit later but first of all let's configure the S log section I need to Define an array of these syncs that I want to use with sarog and I'm going to add two of them the first one will be the console sync so I'll say seral log syncs console and then I'm going to add the one that we actually are interested in and this is our SE server sync so this sync will take care of writing our application logs to the seek server automatically in the background then I'm going to define a section for the minimum log level and the default value will will be information so that I can get more details in my development environment so this section here basically replaces the logging section that I have here so I can get rid of it if I want to I can add an override section for a specific namespace for example I can say anything in the Microsoft namespace use the log level of warning but I'm also going to give it the value of the information log level so that I can get more details in my seek instance the next section that I need to Define is where I want to actually be writing my logs the first output is going to be our console so I'll give it the name of console and this will use the console sync to write our application logs and then I'm going to use the seek as our destination and give it the arguments that it needs to connect to the seek server so we pass that using the arcs parameter and we can give it the value that it's expecting which is is another object containing a server URL property which is going to point to our seek instance now because I'm using Docker compose I need to reference the container name that I have here and then use the port that I Exposed on this container instance so we're going to say HTTP and then use the name of the container and Connect using the port 534 1 lastly I'm just going to add a section to enrich our application logs using the Fred ID machine name and the log context with our Ser log set up in place we can start the application this configuration is going to be applied with our call to the read from configuration that we added in the program file and we're going to start ingesting log into our seek server if I start my application now and open up my Docker desktop instance you're going to see that my containers are spinning up and here is our seek server running our seek image and if we head to the 881 Port where the seek dashboard is exposed you're going to see that we already have some application logs that we got during the startup of our application you can see some database commands being executed by Ned framework some logs from our quartz Library setting up the background jobs and so on so now head over to postman and send a few requests to the customers endpoint which is used to register a new customer in the system so if I send this with an invalid email I'm going to get back a 400 bad request response telling me that the email address that is specified is not valid so I'm going to send this invalid request a few times and now I'm going to head over to the seek dashboard I'm going to refresh the logs that we have and you'll see that the error logs are now showing up with the validation exception so if I open up the error log here you'll see that I'm getting the exception message from seek the exception message is explaining what specifically went wrong and I also have a structured property here which is called message its value says validation failed and this is coming from my exception handling middleware so if I head over to my code for a moment and show you what's inside of the exception handling middleware and we're specifically interested in this line here which is logging the exception using the error log level now notice the exception message here saying exception occurred and then message and this value here is actually our property on the structured log its value is coming from the message of the exception and we can use this property on the structured log to do filtering in the seek server what that will look like is we can say the message and then we can filter on this property so let's say that it's equal to validation failed and if we run this query we're going to get all of the structured logs containing this property value and if you open up the structured log you'll see that this is true this is very practical if you're looking for specific errors in your application and seek allows you to filter the structured logs using the properties that are Exposed on the logs themselves so if I head back to the exception handling middleware I'm going to add an if statement here checking if the exception instance is a validation exception and I'm going to capture the validation exception into a variable and let's do an F else statement here so in the lse statement for any other exception I'm just going to log what we already had in place but for a validation exception I'm going to update the format of the message to add another property so first of all what I can do is add the errors that are present on the validation exception so I can specify those as errors and I want to serialize them into Json so I can add the add sign in front of this property and now I need to pass in a value for the errors property so I can say validation exception and then here is the errors property and you can even pass in the entire exception object if you wanted to by specifying an exception property and then you need to pass in the actual exception so let me start the application now and I'll show you what the structured log looks like I'm going to send one two three and four failure requests for example and then let's move over to seek we are going to run the same search query looking for a message property containing the validation failed value now you'll see that we are getting the old logs here and here are the new ones that are enriched with our specific values so if I open up here now you'll see that my structured log contains the errors property which is an array containing the validation error and you can see the specific error message that caused the validation exception then you also have the exception property which contains the raw exception value including the errors that we already have here which is a bit redundant but this is just an example to show you what you can do with seek and structured logging with sahog so seek allows you to inest your structured logs into the seek server and then filter them based on whatever condition that you want want to check additionally seek also gives you access to a monitoring dashboard where you can see some more details about your application for example you can see how many logs of which level you are getting then you can see the number of events over time I don't have too many requests so it's not showing a whole lot of details and you can filter this based on the time Horizon so if I look at the last 5 minutes I'll get just the requests that I sent in the last few minutes one seral loog feature Fe that you can add to your application is to use seral log request logging so you add it as just another middleware to the request pipeline by calling use seral log request logging and now this middleware will start writing structured logs for your API requests and then you can analyze these logs in seek so this is everything you have to do and I'm going to start the application and show you the result I'm going to send a few get requests to the get product endpoint which returns a list of products in this case I don't have any in the database but this isn't relevant for the example if I open up seek and update the events you're going to see the structured logs appearing for our get products request and you can see all of the properties that we have available for our API requests for example you can see what is the query string that was used for this specific request then you can see the complete URL in this log statement here and you also have a value in the elapsed milliseconds proper property telling you how long it took to complete this request so then I can use this property to look for any structured logs where the elapse milliseconds is let's say greater than 5 seconds and if I keep increasing this value eventually I'll filter out all of these slow logs for example I can use this to find any outliers let's say this one which took more than 200 milliseconds to complete and then I can further examine what was the issue that caused this specific query than the other ones in my application I hope you enjoyed this video about structured logging using seral log and seek and if you did then make sure to smash the like and the Subscribe buttons on your way out and until next time stay awesome

Channel: Milan Jovanović

Views: 12,620

Keywords: seq, seq c#, seq .net, seq .net core, seq asp.net, seq asp.net core, seq serilog, serilog seq, seq and serilog, serilog and seq, seq structured logging, seq structured logging server, seq server, seq license, seq serilog sink, serilog sinks seq, seq sink, seq logging, structured logging seq, structured logging, serilog, serilog sinks, serilog structured logging, asp.net core logging, .net structured logging, asp.net core structured logging, seq docker, seq docker-compose

Id: mT8ZkXafuZk

Length: 13min 16sec (796 seconds)

Published: Fri Oct 27 2023