Log Levels Explained (with practical examples!)

in this video we'll explore log levels which are a fundamental Concept in logging and in most logging packages and log levels are a vital concept to understand as you develop deploy and maintain applications so let's dive in and explore log levels and give some examples of their usage Now log levels underpin all good strategies for logging in applications and these log levels they have labels that indicate the severity or the urgency of particular event that are occurring in your applications and the typical levels that are used in modern software development are fatal error warning info debug and Trace so why do we need log levels when you have an application and when you're developing that application you need to record different information about different events of different severity that are occurring in the process and these events are not created equal some have great significance and could indicate a critical problem in your application whereas others are more informational messages that can help developers and maintainers as they try and understand the behavior of the application so not all events are equal here so to give an example of this on one hand you might have logs that record the status or the state at a given time in your application and an example of that might be the value of a variable or the response data from an API endpoint on the other hand you also need to log information about events that have greater significance and this could be about things like warnings and errors that are occurring in your application for example maybe some unexpected input from a user or perhaps a critical problem for example the server may be running out of disk space so you can create these levels to separate across these boundaries and in addition to that the use of levels also allows you to filter the logs as well so for example you can filter the logs that you have and only get back the logs with a particular level for example that could be the error level now now another benefit of levels is that you can send less important logs for example just informational logs to a different location than the more important logs and that allows frequent log rotation to occur on those less important messages and such a strategy will also cut down on storage costs on the other hand logs that contain critical messages or information about a failure in your application or the infrastructure on which it relies these will have a higher severity level and they can be sent to different handlers from those basic informational messages an example of that might be an email Handler that very quickly notifies developers and administrators about problems that are occurring in the application so the levels allow you to separate the logs and filter those logs and they can also enable analysis and alerting as well as anomaly detection so that's the introduction to log levels let's now move on we're going to look at a very brief history of log levels and then we're going to move on from that and we're going to show some practical examples of all of these level levels let's quickly explore the history of log levels we're going to reference this better stack article which is linked in the description and as it says here log levels have a history dating back to the 1980s and the concept of levels originated with the CIS log standard and these introduced the following log levels we had emergency alert critical error warning notice informational and debug now these map closely to some of the more modern log levels that are used in modern programming languages like python go and JavaScript and in fact if we scroll down a little bit here we can see that some logging Frameworks such as log for net and log 4J which are logging packages for net and Java these Frameworks recognize the significance of these severity levels and they further refined the concept so in most cases nowadays we have the following log levels and these are from most severe to least severe so we have fatal error warn info debug and trace and we're now going to see some examples of these log levels and practical examples of when you might want to use them for those examples we're going to use Python but this better stack guide has examples in different programming languages and also different Frameworks within those languages and we're going to start with the most severe log level and work our way down to the least severe which is the trace level so let's get started we're going to start by looking at the Fatal or critical log level now the critical or fatal log level this is reserved for the most severe issues that can occur in an application this level indicates a critical failure that will stop your application in its tracks so typically you cannot recover from this you're going to log the critical error and the application is then going to stop now some examples of critical errors include things like a loss of connectivity to an external source that your application depends on and that could be a database or an external API another example would be running out of disk space on on a server where your application is hosted configuration errors as well if your application relies on some configuration that is not available in the environment and also a security breach that could be considered a critical issue and you might want to log that out at the critical level now in order to demonstrate this I'm going to go to VSS code and we have a python application here that Imports the logging package and it creates a logger object sets the level to debug and it creates a formatter that's going to set the format of the output entries so we're going to have the time the level and the message in the output and then we create a stream Handler that's going to log to the console or The Terminal and we add that Handler to our logger so what I'm going to demonstrate here is something that might cause a critical error in your application what I'm going to do at the top is import the OS module and the CIS module from Python and then underneath the logger we're going to attempt to import some configuration from the environment and this is a standard practice in software you want crucial configuration to be stored in the operating system environment so in order to do that I'm going to paste this bit of code we're trying to retrieve a database URL by calling os. Environ doget and that's going to try and retrieve this database URL from uh value in the environment and what we can do in Python is we can check to see if the database URL has not been set with an if statement here and if we don't have a database URL what I'm going to do is I'm going to call logger do CRI IAL so when you want to log out a critical error or a critical event in the application you can use the logging packages critical level and as with all levels you give it a message and the message I'm going to pass here is that no database URL was available now when you have a critical event such as this your application is typically not going to be able to continue so what I'm going to do here just to demonstrate is call sis. exit and we pass a status code into that now on the terminal at the bottom I'm going to try and run this F file and the file is called app.py so let's try and run that and we're going to see that output here this is the log entry and it has the level critical here as you can see now what we can do if we run the script we can actually set that URL so what I'm going to do is set the database URL to a dummy value let's say XYZ and then after setting that we can then run python app.py we're not going to get the output in this case because that variable is actually set in the environment when we run the script so in an application that depends on some configuration if that does not exist you can log a critical message to indicate that to your administrators and of course if you send the logs to a service like better stack you can also configure alerts to be sent when you get these types of events let's now move on and we're going to look at the error log level now the error level is used to log errors that are unexpected but do not necessarily cause your application to crash and exit events at the error level they highlight issues that need need attention but that can possibly be handled and allow the application to continue running so what are some examples of error events we might have network communication errors or timeouts and other errors can include things like trying to decode Json data from an API but that's failing because the data is perhaps not in the right format another example of Errors might be a database constraint violation so imagine trying to insert data but you're trying to add a foreign key that doesn't exist that's going to cause a constraint violation and another example is when you try and access a file but you might not have the access rights for that file we're going to look at this exact example now so what I'm going to do is go back to VSS code and I'm going to remove the code from before with this database URL now on the left hand side we have the app.py file and underneath that I have a document. text that just contains some sample text if we execute the ls- LA command in the terminal you can see the document. text file now this doc document has an owner of root and that's a different user from the one with which I'm executing these python commands and running the script so what we're going to do is we're going to create a reference to that file name and we're going to try and open the file and we're going to try and read the contents of that file and print them to the terminal so the code we need to do that is here we open the file as F and then we call the file. read function let's save this and execute the script at the bottom and you can see the output that we're getting here we're getting permission denied error so we cannot open that document. text now let's see how we can handle this using Python and using the loging package so we're going to wrap this in a try accept block so let's do that just now so we're going to try and open this file but if we get any kind of exception here we're going to catch that exception and in that case what we can do is we can call logger do error so we're outputting a message with the level of error and let's pass through a message here permission denied when reading the file so hopefully if we expand the terminal here when we execute this again what we're going to get is that error message instead of the exception and that's what we get now now this might not necessarily cause your entire application to crash or make it unable to function what we can do here if we don't get the information from the file perhaps we can create a fallback and therefore this is not necessarily a critical error and we output at the error level but of course even though we can continue this is something that is probably going to need to be dealt with if you have a file on the server that you need to read but that file doesn't have the correct permissions assigned or the correct owner that's something that you're going to need to address at some point let's move on to the next level and that's the warning level now warning logs are used to indicate potential issues or issues that might lead to problems in the future and these warning events they don't necessarily stop your application from functioning but they are events that will require attention in the future and should be addressed as soon soon as possible and here are some examples of events that might be logged at the warning level you might have outdated configuration settings this might not prevent your application from running but you should log these kind of things at a warning level to let people know and that can be addressed as quickly as possible another example might be excessive resource consumption and warnings can occur due to user actions as well for example an excessive number of failed login attempts if a user is trying to log in to your site and they've tried that like 50 times or something you may want to log these things because it indicates a potential problem another example is if you're sending requests to an external service and the response is taking a long time and even things like expired tokens and certificates these can be logged at the warning level so let's Show an example now in Python of something that might trigger a warning and how to handle this and log it to our loggers let's open VSS code again and we have our file open here what I'm going to do is I'm going to import a package that's deprecated in Python's standard library and that's the tailet lib package tailet was a Communications protocol or it is a Communications protocol but it's not a secure protocol and it's considered to be outdated and shouldn't be used now if we run this python application you can see we get this deprecation warning the tailet lib package is deprecated and it's slated for removal in Python 3.13 now what I'm going to do here is I'm going to wrap this in a try catch just to show how we might catch a warning here and I'm going to use the accept statement to catch that deprecation warning now because we're importing tailet lib and we're aware that this is a deprecated package what I'm going to do is add a call to logger do warning and that's another log level method that's available on the python logging object and let's add a message here of tailet lib is deprecated if we execute the script at the bottom we get the warning output just below the message from python itself so that is going to send this war warning to all of the handlers that are configured so if your code has a module that's importing a deprecated package then you can use the logger warning to tell the administrators and tell the developers about that but again this is not going to break your application at the moment so let's move on now and look at the info log level info logs are used to capture general information about an application's operation and these informational logs they provide a high level overview of the behavior of your application and help developers and administrators understand the flow of the application how it's working and any changes in state that might be occurring and production systems are typically logging at this level and that's so that the developers and the people involved in the application can build up some knowledge of how the application is running and they can see the normal behavior of the app when they look at the logs so what kind of events might you want to log at info level one example might be a change in state in the application for example if you have an ordering system if an order was to change state from in progress to delivered that kind of change in state is something you might capture at the info level another example might be the successful completion of a scheduled task or when you've successfully completed a connection to some kind of component that the application works with for example a database or a cache or any other kind of system and of course you might want to also log general information about requests and responses and we mentioned a cache before you might also want to log information about when you have a cash hit or when things have been evicted from the cash or are not found in the cach so lots of examples of things that you might want to log at the info level let's provide a very simple example now and what we want to do here is we want to send an HTTP request to a service that's going to return some Json data at the top I'm going to import the requests package and then just below the login code I'm going to create a function called get users now in this function I'm going to reference this endpoint and this is a URL from the Json placeholder website and we're going to create a variable called response and we're going to use the requests.get function and pass that endpoint in now we don't know the status code of the response so I'm going to use a try accept block here and we're going to call a method called response. raise for status and if the response status is not beginning with 200 then this is going to raise an exception so we can catch that exception so I'm actually going to paste that in here it's actually a requests. HTTP error we're going to catch that and let's say we get an error we want to use the loger do error method that we saw earlier on and we can pass a message into this so I'm going to say request to endpoint fiiled with the given status code now at this stage we want to return none if we are unable to get a response from this endpoint but in the event that we do get a response we can log that out with the level of info and I'm going to pass another F string in here and say that the request to the endpoint was successful F and finally this is a function in Python so we can return the data if we know we have a response from that endpoint we can take the response and call the Json function so let's now call this function and then we can run python app.py at the bottom and we get the output with the level of info so the request to that endpoint was successful and we got a valid response there and converted it to adjacent let's change the endpoint here a little bit so that we get an invalid Response Code so I'm going to add some characters to the end of that when we execute that request this time we're getting back the log with the level of error and that indicates that the response was not valid so that's an example of how you can use info logs and error logs together in a function in Python and which one you use depends on the behavior of the function let's now move on to the next log level and that's the debug level now the debug level as the name implies is useful for debugging and for general information and very granular and detailed information as you develop applications it's not typically a level that's set on production when you deploy your apps but it's very useful during development to trace the flow of your application and get important information such as the value of variables at particular times now it's not normal to have debug in production as I just said and that's because this results in a lot of logs being sent to your log services and this can increase the burden on storage and dis IO especially if your application has a lot of users and a lot of requests and responses let's say so what might be logged at debug level an example might be database queries that are being sent to your database from the application layer we can also log details of API calls and response data at the debug level and when we're setting up the application and setting up new Services logging out configuration values can be helpful for debugging and for diagnosing any problems that might be occurring and finally you may also want to log out some timing information at the debug level and that can include includ some basic profiling of your application code so timing how long a block of code takes to execute let's now see an example of debug logs and I'm going to go back to VSS code for this example and we're actually going to extend the example that we had for the info level where we sent a request to adjacent placeholder so let's say we have the successful response here what I'm going to do is I'm going to convert that data to a python dictionary before I return it and we are interested in seeing what is this data what does it actually contain and again you don't want to have this in production because let's say you have a thousand users you don't want to be logging out the entire response data every time one of these responses is coming in so this is something that might be useful in development only so underneath the data I'm going to add a call to loger do debug and I'm going to add another F string in here and we're going to print out the response data from the endpoint so we're adding that data in here and we're going to log that at the debug level and finally we need to also return the data from the function so let's save this and we're going to go back to the terminal and I'm going to execute this command and we're getting a 404 not found here and that's because I changed the URL at the top let's change that back and go back here and try this again and this time you can see we're getting the debug logs that contain the data from the response if we scroll to the top of that you can see the log level of debug here and we also have the info log above that which comes from this statement here so again we don't want to be printing the response data every time this happens in our application and production so this is something we do in development when we need to do a bit of debugging we need to look at what is the response data in this case that's the purpose of logger do debug and that is this example that we've seen here so let's now move on we're going to look at the final level and that's the trace level now Trace logs are even more detailed than debug logs and Trace level logging is not as commonly used as logging at the debug level and as such it's not support orted by some loging packages and Frameworks and that includes the python standard Library its loging module does not include a loging do Trace level so why would you use Trace logs these are typically reserved for extremely detailed information and this information might be useful in diagnosing very specific or very intricate events that are occurring in your application now as I said the python standard Library doesn't include a logging do Trace method but there are external packages in Python such as log Guru which do offer this functionality now for more information about the trace level you can check out the better stack guide on log levels and there is an example on this site that uses the go zero log package and as you can see that particular package does include a trace method for the trace log level so that's all the main log levels we've covered some examples for each one of those let's finish by covering some additional information on log levels and for this I'm going to largely reference the better stack guide that we have here we're going to cover the final three points let's start with the one at the top here controlling the application's log volume so as it says here log levels they provide a means to regulate the amount of log data produced by your application and by setting the appropriate log level you can determine which log entries can be recorded and which should be ignored and this allows you to balance capturing essential information but also avoiding an overwhelming flood of logs now it's common as it says here to set the default level to info so when you're in production that's typically what you're going to use but in development you might want to set it to debug or trace and whatever you set your level to any level that is not as severe as that these logs are going to be discarded and that can help you control as it said earlier the amount of information that's coming to your logs and on the left hand side we have an example here of creating and using custom log levels so you might have an example if you're building a large application where you need to include a custom level and this is supported by some packages for example the p PJs package you can use PJs if you're doing logging in a nodejs application and we have a video on pjs that should be appearing on the screen now and in that video we have an example of how to create a custom log level for your application so if the default levels that we've covered in this video are not enough you can add your own custom levels for particular events now the final thing I want to cover is the use of log levels for monitoring and Analysis so once you've developed your application and it's in production and you've added logging statements with the correct levels to that application the question being asked here is how do we use those levels for proper analysis of the logs and this gives three examples the first one is filtering and searching the logs so I'm going to show a demo here on the better stack log tail I've generated a bunch of logs here and they have different levels you can see info error critical warning debug and so on now what I'm going to do is I'm going to try and filter these logs by a particular particular level so let's say we only wanted to find the errors let's click on error and we get back only the logs with that particular level so by adding proper levels we can filter the logs that we have in our services and we can find only the ones that we are actually interested in and each level is associated with a severity and you can see if we expand one of these logs on the better stack tail the level of error is associated with a severity level of or a severity number of four so what we can also do is we can do other searches here other filtering so we can look at logs that have a severity that are greater than or equal to four now if we execute this search you can see we get back not only the error logs but also the critical logs and that's because critical is the log level with the highest severity so we also get that back in the results you can see how easy it is to search and filter logs when we have these levels set up appropriately let's go back to the better stack article and what we can also do when we have levels properly set up is we can create alerts and do alerting in our applications so let's say you're sending your logs to better stack and when a log entry comes in with the level of error or critical you can trigger alerts to occur and send notifications to specific users when these events happen an example of that might be just sending an email to admins when you get a critical error in the application and this allows the relevant people to be notified immediately when these kind of issues occur in your production environment and the final example on the blog post is the ability to calculate various metrics now an example of this might be that after you deploy an update to your application you might see a significant increase in error or fatal SL critical logs and this may indicate that you have underlying issues or bugs that require attention if you have some analysis and dashboarding set up on better Stacks log service you can quickly detect when this is happening and again you can take steps to immediately resolve these issues so log levels have many uses you can see how they can be used for alerting filtering searching and also for analysis we've covered all of the main log levels in this video but we've also noted that you can create custom levels using many logging packages and that is all for this video if you've enjoyed this video please subscribe to the better stack Channel we have lots of exciting content coming up on topics like Docker and kubernetes testing task scheduling and also browser automation lots of things coming up please subscribe to the channel if you're enjoying this content and give the video a thumbs up and we'll see you in the next video

Channel: Better Stack

Views: 938

Keywords:

Id: myxlDVC2K74

Length: 25min 39sec (1539 seconds)

Published: Sat Jan 13 2024