Logging in Python Crash Course - Security Levels, Log Files, Formatting

[Music] what's going on guys welcome back in today's video we're going to talk about logging in python so let us get right into it all right so let's talk about logging i think logging is one of those things that people oftentimes don't want to do a little bit like testing because it's a little bit boring it's not the most fancy thing not the most interesting thing that you can do in coding but it's one of those things that save you a lot of time and can help you tremendously if you need them now you hope you won't need them but if you need them you have a lot of data to analyze for example let's say your program crashes or your service doesn't work anymore you can look into the logs into the past logs into the current locks to see what is going on or what was going on what caused the problem what is the core of the problem and how can we find a solution so logging is something uh that is a long-term investment you want to lock everything especially if you have an application that's a little bit larger with many different classes that are interconnected you want to have the overview of what is happening and to some degree we already do logging all the time when we write scripts we constantly print messages that inform us about the state started to train the model model is now trained proceeding to the next step or maybe a warning this could go wrong because that component didn't load or this error happened or this component failed critically or maybe even even just a debug message that says the value of this variable at the moment is that um so we do some logging all the time but most of the time we just use print statements unless we're working on a a professional large large scale project but i want to show you today how you can do professional logging in python using the logging module and this is the go-to way to do logging you should not just be printing messages all the time with a print function you should not just open a with statement and dump some information into files you want to have professional log files especially if you're working on a real project so let's get started the first thing we need to do is we need to import the logging module itself it's just import logging part of the core python stack um and we can log right away with the logging module itself so we can say logging.log and then we can choose a security level by the way let's talk about security levels real quick we have five different security levels the first one is debug which is self-explanatory used for debugging then we have info then we have warning then we have era and then we have critical now i think all of them are pretty self-explanatory but basically debug is the lowest level which means that if you have your logging um if you have your logger on debug mode you're going to see all the messages if you have it on warning you're going to see warning error critical if you have it on critical you will only see critical messages so it has this you could say backward compatibility that if you have the lowest level you're gonna see everything if you have info you're going to see everything uh except for those below the level that you chose right so debug is just for debugging messages like this variable has this value at the moment info is more like informational this component started this model is now trained so just info that is not related to any problems a warning is not a problem yet but something that could go wrong because we have something that maybe you want to look at because maybe it will become an error or a critical message if you don't do anything about it an error is something that went wrong but it's not critical to the program it's not like you uh it's it's not like everything crashed and nothing's working anymore but maybe some connection didn't work some file was not found and so on and critical is like a critical component failed we need to stop but we need to do something right now because that is critical to the program so those are the five levels and we can just provide one by saying logging lock and then logging dot and we can choose the level debug info warning error and critical so we can go ahead and say for example info and then the message uh this is our first logging message like that so if we do that i'm not sure if we're going to see it right away because it depends on the logging level by default we're not going to see all the messages because we have a default logging level that we have if i change this to critical we will see it because critical is the highest level so there's no way we're not going to see any critical messages here but as you saw info is not displayed by default so if you want to change the basic config of the logging module you need to say logging.basicconfig and then you need to set the level to whatever level you want we're going to set it now to debug so that we see all the messages so we're going to say logging.debug like that and if i now run this with the info again [Music] we're going to see the info message as well so we can change this to the buck as well we can change this to critical we can change this one to warning [Music] warning there you go and you can see all of them are displayed now if i change this to info we will not see the debug message as you can see so it's gone because debug is one level below info so we're not we're not displaying it so this is one way to do logging you just use the module right away and you use the log function you can also use the module and say logging.info directly or logging.debug directly then you just have to provide the message hello so you can choose info you can choose warning warning [Music] and you can choose critical for example and let's add error as well so you can also do it like that if you want to and the other method in which you can lock or if you don't want to use the logging module itself you can also create a logger so we can say logger equals logging.getlogger and then we have a named logger so we can provide a name for this logger let's just call it neural logger like that and now if we say a logger dot info for example this is some information [Music] then you can see that i now have info and then neural logger as the logger name and then the message uh by the way one more thing that i want to mention is that sometimes this won't be enough i don't know when exactly it's not going to be enough i had some problems in the past sometimes you're not going to be able to set the level by just setting the basic config in such a case what you need to do is you need to say 4handler in logging.root.handlers and you want to remove all the handlers that we already have in the rootlogger now probably for this video you're not going to need it need this but if this for some reason doesn't work and you cannot choose your security level manually what you want to do is you want to remove all the handlers from the root logger and you can do it by just saying um for handler in logging root handlers and then logging root dot remove handler handler like that and then you want to set the basic config to info or whatever you want uh just some side information here because i struggled with that uh once already all right so we have this logger now and uh we can we can proceed to just you know do the same thing here we can say logger.log and provide a security level or we can say logger.info logger.critical and so on but that's all the basic command line logging the good thing about logging is that you can also uh log into files easily you can just add a file handler to the logger and then you can log on to the command line and at the same time into files so if you want to do that you create the logger and then you create the file handler so we can say file handler equals logging dot file handler and here we specify a file name we're just going to go with log file dot log for now so nothing too fancy yet we're going to talk about that in a second how we should name the files uh ideally so we're not going to stay with logfile.log and then what we do is we basically just say okay filehandler.setlevel because the filehandler can have a level itself let's say for example onto the command line we want to print everything so let's set this to debug but i only want to log into the uh into the file handler into the file i only want to lock all the messages that are warning or above so i can say logging dot warning so then i'm gonna print everything onto the screen i'm gonna log everything into the command line here but i will only log the warning messages error messages and critical messages into the actual log file and once i have the handler i just say logger.addhandler and filehandler like that then we can go ahead and lock some messages let's start with a debug message hey i am debug some info message hey i am info and then some warning as well there you go so if i run this now you will see that on the command line i get all three of them if i go into the just created log file you can see that i only see hey i am warning and you can also see that i only have uh the message itself i don't have any format like here i don't have a security level i don't have a name i don't have any timestamp i only have the pure message and this is something you want to change but before we change that let's talk a little bit about the file name that we should choose because the file name um it should not just be log file.log because then you have one big log file and there's um it's a little bit hard to look through all the logs if they're all in one log file so what i like to do personally is i like to have different log files for every day and how you can do that is you can just say in the beginning of the of the file you can say import date time as dt and then you can say today is dt datetime dot today so it gives you today's date and then you can craft a string so you can say file name equals and then we can create an f string and what we want to have is first we want to have the month the day and the year so what we can do is we say today dot day or actually we said month right today dot month is the first thing and i would like to add a formatting so that we have always two digits so basically if we have uh 12 it's already going to be two digits 11 as well but if we have nine i don't want to have nine i want to have zero nine if i have two i don't wanna have two i wanna have zero two and in order to get that formatting we just have to say colon 0 to d this is how we do that formatting then we can also say today dot day and it's going to do the same thing 0 to d and then the year which is today dot here here i don't need any formatting and then dot lock in the end so that is the file name and this is what we're going to pass to the actual file handler here so we're going to just pass file name and then depending on the day you start this logger it's going to have a different file name so we can run this here and you can see that i have the 0 8 19 20 21 log file because it's the 19th of august 2021 and this is why the log file is named like that if i run the same script tomorrow i'm gonna get a separate log file with 20 here instead of 19. all right so let's get to the formatting which is the final thing we're going to talk about today the formatting is very important because again we don't want to see a log file like that where i just have some message like something went wrong i don't even know the security level i don't know when it happened i don't know uh which logger is writing this so we want to change that and for this we have to create a formatter so what we do is we say up here formatter equals logging dot formatter and to this formatter we pass a string describing the um describing the format of the logging message now i personally always have to look this up so you can look into the documentation or you can just copy it from this video but the structure is the following for the timestamp that i want to have or for the logging structure that i want to have first a percent sign then asc time in parentheses followed by an s this is basically the timestamp so then we follow that up with a colon then we have percent and then level name which is the keyword for uh for the security level and then we have uh a dash and then we can go with message so percent messages so basically you can't see what it's doing you have a timestamp then a colon then the level name then a dash and then the message you can of course change these symbols here but these things need to stay the same these are just the keywords that we use to to get the certain things at the positions these are like placeholders for the individual pieces of information and once we have that we just say actually we need to do it before we add the handler but after we have the handler so we need to do it here we need to create the file handler and we need to create a formatter and then we're going to say filehandler dot set formatter formatter like that and then we add the handler so if we do this now let's add some messages here add a error message here and a critical message of course we change this to i am era and i am critical and now if i run this you can see here the format is the same because we only change the format for the file handler and inside of the log file you can see that we have a timestamp uh even with a time not not just the date with the time itself as well then the uh security level and then the error message itself so this is how you do professional logging uh you have to decide for yourself what you want to log how you want to log it but i would recommend whenever you have like something like try and accept for example you try something we're just going to add a pass here some code and then you have accept and something goes wrong i would always do logger.era or you can also say logger.warning there was a mistake we fixed it here is how we proceed because except basically means that your script uh encountered an error or exception but we handled it so it's actually no longer a problem you can choose what you want to log you can also if something fails critically you can just say okay critical we have to terminate the script and you need to do x y and z you can lock info messages whenever you do something so for example when you train neural networks you can say okay logger.info the neural network is now starting to train then logger.info the neural network is now trained you can start with the next step and so on uh it's always useful to have as much information as possible and yeah that is how you do logging in python so that's it for today's video hope you enjoyed hope you learned something if so let me know by hitting the like button leaving a comment in the comment section down below and of course don't forget to subscribe to this channel and hit the notification bell to not miss a single future video for free other than that thank you much for watching see you next video and bye [Music] you

Channel: NeuralNine

Views: 17,971

Keywords: logging, log, logs, log files, python logging, python logger, python log files, log into files, python, log formatting, formatter, formatting, how to log into files, python log, python logs, security levels, debug, info, warning, critical, error

Id: m08LtvC3jaY

Length: 16min 11sec (971 seconds)

Published: Sun Aug 22 2021