Python logging tutorial: loggers, handlers, and formatters

logging it's something that nobody seems to like especially with the built-in logging module in Python I was recently recording this video for my fast API course and we were talking about just that how you can use loggers handlers and formatters and use the login module in your applications so I took this video from the course and I'm putting it here on YouTube for you guys to enjoy if you do like the video then please hit like And subscribe to the channel and also if you'd like me to post more videos from the course or from other courses here on YouTube then please let me know in the comments below I'll always post videos that are self-contained so they don't depend on other videos in the course but I think maybe there's something that we can do and putting content up in here as well so thank you guys let's get to the video [Music] here's a quick poem that I wrote with chat GPT that's actually pretty relevant in the realm of code logging's a Fickle dance Too Many logs May Leaf budgets as scans find the balance and debug you can but look too much you'll end up in a jam and this is really the problem with logging I find it's difficult to do well how much should you log how often should you log where should you put your logs once you've locked them in order so you can read them later on and that's where the budget line comes into play if you have a production application and you are logging a lot of data so that you can tell what the application is doing when you read the logs all of that adds up and you have to store them somewhere and that costs money so you can actually end up spending quite a lot of money in storage and in subscription plans for logging Services the other problem with logging is it takes up lines of code in your programs if you log too much most of your code ends up being logging calls and it can end up quite difficult to read so you have to find the balance and then logging is going to help you debug your application if something goes wrong when you deploy it and also it's really gonna help you identify what's going on in case your application grows so complex that you're maybe not 100 sure the path that the code is taking in all cases I thought there was an interesting thing chat GPT is a cool technology anyway let's move on the pros and cons of logging this is a bit of a guide as to what to expect logging mainly you use it to see what happened during a request to your web application because once your application is deployed you obviously can't step through it with the debugger at this not very easily so if a problem happens you want to be able to see what the user did in that request that led to the problem the problem with that is that it makes your code less readable as I mentioned earlier and you'll see what I mean when we start writing logs in our routers for our application the code ends up being a little bit less readable but if you find the balance it's not that bad also logs let you gain a historical context into your application because you can store your logs somewhere then you can search through the logs and you can use the logs to build big data applications or just to see what users tend to do with your applications that kind of information is something you can glean from your logs if you log well there is enough information in there to actually clean something from it but the Counterpoint to that is the more you log not only is your code less readable but you also have to spend more money once the logging system is set up with python the python logging module blogging is very straightforward you get a logger and you say logger send a message and then that's it the message goes to wherever you've told it to go when you set it up the initial setup is quite confusing there's obviously loggers handlers and formatters that we're talking about in this video there's also something called filters you've got to link these up to each other there's a few extensions as well that you can install to help with your logging then you've got logging services and Analysis services and all that kind of thing so it is a bit confusing once you've set it up it's fairly simple another benefit of logging is you can set up alerts and dashboards for example if you log an error message in your application you can set up an alert so that it will send you an email to your work email when that happens and you don't have to wait for users to complain that something's going wrong you can use the logs not just for seeing what happens during a request and gaining historical information but also for real-time information as well so logs are really useful and really you have to use logs if you are deploying an application and it's you know a serious application it's basically essential so let's continue a quick primer on the logging module in Python you've got loggers these schedule log information for output and each logger has one or more handlers and a Handler is responsible for getting that log information that you've scheduled and sending it to a particular destination I'll give you an example in a moment and each Handler has one formatter attached to it that defines how that log information will be displayed so what parts of the log information will be printed out to the console for example or put into a file and what parts will not be displayed let me give you an example you've got your logger and then you may have two handlers let's say a console Handler and a file Handler there's a few different handlers you can use and we'll go through them throughout the course of this section and then you've got a formatter that is set up to display the current time and the log message let's say that you use the logger to send or schedule a message that says this is my log message well first this goes out to the handlers the handlers each send it to the formatter or rather the handlers use the formatter and then the handlers in combination with the formatter will either print out the message to the console in case of the console Handler or put it into a file let's say store api.log in case of the file Handler and obviously you can customize the file name and various other parameters when you use a file Handler but this is the quick primer on how the login module Works you've got your logger that you use to schedule messages you've got handlers attached to the logger each Handler has a formatter and the Handler uses the formatter to display the log message in some way sending it to whatever you've told it to send it to such as the console or a file or indeed a third-party online service if you want to do that logging levels you can log messages in different levels and they have different meanings for example the critical level is a type of level that you should use only when there's an error that causes application failure for example you may try to connect to a database and you have a try accept Block in python in the accept block if you can't connect to the database maybe you will have a critical log message being scheduled then you've got error for example if a user tries to access an endpoint but they don't have permission to do so maybe you will log an error message then you've got warning info and debug in different environments such as When developing locally or when your application is in production you may only show on hide certain levels of message for example in production you may not want to show debug messages because you can often print out a lot of debug messages and that can take up a lot of space and therefore a lot of money in production and you're not going to be looking at them but when you're developing locally you may want to enable debug level messages so that you can see all that information when you are actively coding here's an application that uses the logging module to print out five different logging messages at five different criticality levels first you can see we have to import the login module then we set up the default handlers and formatters using logging Dot basicconfig and we pass the level at which we want to be able to log if we say logging.debug then that means that all messages will be shown if you pass logging dot error for example then only the error and critical messages would be shown so this is like a low pass filter that allows you to filter out certain types of message now once you've set up the login module you need to get a logger that you're going to use to actually schedule these messages so you do that with login.getlogger and you give the logger a name this initially sounds a bit weird that seems irrelevant it's going to become relevant later on in the next few videos and the logger name actually has some meaning that simplifies the logging setup so we'll get to that once you've got your logger you can simply do logger dot debug info warning error or critical and that schedules a message with the given text at that level of criticality so logger.debug would send a message at the debug level and again if you had configured the level here at logging dot info then this message would still be executed but it would be immediately filtered out and it would not get sent to the handlers and the default Handler by the way is a console Handler so if you do this you'll see your logs in the console and by the way this is what it looks like you would see the level so that's debug info warning error critical then you have a colon and the logger name you can see that's in my logger then another colon and then your message this is a default look of logs or this is what the default formatter displays logs as if you want to Define your own Handler so that you're not using the default Handler then you can do it like this just Define a variable make it equal to logging dot stream Handler and then this is going to send logs to the console now in basic config you can pass in a Handler's keyword argument with a list of handlers and you put your hander in there and now this does exactly the same thing but it sends it using this login.stream Handler this is what you can do if you wanted to Define multiple handlers so again Define your Handler and then pass it as a keyword argument if you wanted to Define two handlers for example you can do it like this you can add another Handler in here in this case it's logging dot file Handler which defines a file to which logs will be printed and then you can just add it to your Handler's list and now whenever you log these messages they would go to the console and to a file called file.log so this is what the logs look like as I mentioned earlier and when you want to display log messages using a formatter the formatter has access to certain variables that are defined within each log record a log record is the object that gets created when you actually schedule a log for output by the logger so the logger creates a log record then the Handler uses the formatter to display it and at that point the formatter has access to certain variables stored within the log record these are these variables here so you've got level name which is the criticality level you've got the name which is the logger name and you've got the message you can also notice that there's a percentage a pair of brackets and an S at the end this is the percent based string interpolation in Python and all this means is level name is a string but you have two percent brackets you put the variable name inside it and the S at the end means it's a string and this is how you tell the login module to display the level name variable just like with f strings we use curly braces with the login module you use this percent brackets and S at the end if you wanted to display a number instead of a string you would replace the S by a d but you can see that each one of these maps to one of these variables and the colons there are just for display purposes they don't mean anything it's just you want to actually show a colon there you could change it by a dash or a pipe or whatever you want now here's the same application that we had earlier but now we are defining the format string ourselves as well as level name name and message we also have ASC time for the current time and you've also got line number which is a number and that is the line number of the file where the log message was shown for example for this debug message when in line one two three four five six seven eight nine ten eleven twelve thirteen so line 13 would be shown in place of this line number variable this is what these messages would look like you can see the current time there that's the year month day hour minute second and millisecond then you've got the level the logger name a colon the line number and then our message if you compare this to the default look you can see there's a lot more information you've got the current date which is essential to be able to know when something happened and also you've got the line number which lets you find where the message was created if you have an error a log message you can go into the file and you know figure out how that was called what caused this log message to be created and from there figure out the problem hopefully but logging.bcconfig can make things easy but it's maybe not perfect it gives us some nice defaults but we don't have as much control and when you want to start doing things like send messages to a third-party logging service or maybe do some more advanced configuration it's just not something you can do with basic config plus basic config does a few things that are not ideal in terms of which loggers it configures you don't have to worry about that too much but it's better to set up our loggers handlers and formatters manually unfortunately it's pretty straightforward so let's do it we're going to create the logger handlers and formatters to the logger we're going to attach the handlers and to each Handler we're going to attach a formatter but the first thing to do would be to import the login module and then get a logger and set its level basic config was doing this for us but now you have to do it here at the logger level then create the handlers just as we did before when we were passing them to basic config you simply create as many handlers as you want your logger to use then you can create a formatter and you have a formatter per Handler but two handlers can use the same formatter if you want so here we're just creating one and then we're going to set formatter on each Handler so we'll do console which is the console Handler or the stream Handler we're going to set the formatter to be this formatter and for the file Handler we're going to set the formatter to be this formatter as well so that way when we use the logger to send the log records to these handlers they will use this formatter to display the logs then we basically do the same thing as we did with formatters and handlers but to handlers on logger so to our logger we will add the two handlers and then you can actually use the logger so that's basically it obviously it's not sustainable to do this in every file so whenever you want to log something you're not going to be doing this whole thing that's where logger inheritance is going to come into play we're going to configure a logger and then every other logger in our application is going to inherit from that logger and it's just going to use the login configuration that is set up all right that's everything for this video I hope you've enjoyed it and you've learned something new the logging module gets a little bit of lack for being a bit confusing and difficult to get started with but once you've got it set up it's actually fairly simple there's another video that I want to upload on the Thunder name variable in Python and how it works and how it can be used for logger inheritance so if you'd like me to publish that as well please let me know in the comments below all right thanks guys and I'll see you next time

Channel: teclado

Views: 5,533

Keywords: python logging tutorial, python logging, python tutorial, python logger, python handler, python formatter, logging handler, python logging handler, python logging logger, python logging formatter, logging logger, logging formatter, logging.getLogger, logging __name__, logging dunder name, logging getlogger name, Tecladocode, python handler example, python logging handler example

Id: b4Ms4wxJuPg

Length: 17min 5sec (1025 seconds)

Published: Wed May 17 2023