logging in python with log rotation and compression of rotated logs

hi friends welcome to time python Series in this video we're gonna talk about how to do logging in Python so before starting let's see what we're going to cover in this video we're going to cover minimal logging like logging in just one two lines and you can see logs are generated and then we are going to see how to use the logger object and logging Handler and how to change the date format of the logs and how to do logging into files just like this and how to show error data in logs and how to rotate clocks so that the log file doesn't becomes huge and how to rotate logs based upon the log file size and log file timestamps and how to rotate logs and zip the old log files so that you can save a lot of disk space something like this so you can see we are covering so you can see we are covering a lot of Topics in logging so I have created chapters for this video so that you can easily switch through the topics so please see the chapters for this video in the timeline alright let's get started you know why do logging logging is basically just something like printing into the console and why is it important logging can help to achieve observability of the performance and the security of the Python application errors can be easily debugged by looking at the logs you know changes in the application State can be easily traced by looking at the logs if they are logged properly and if there are some performance issues or security issues they can also be addressed easily by analyzing the logs which are generated by the application so logging is really important if you want to maintain an application and fortunately logging is really easy in Python there's a module called logging in Python which comes inbuilt with the python installation so you don't need to install logging module separately generally it comes along with the python installation all right there is something called a logging level so logging level is basically attaching a tag to each log there are six logging levels in the python logging module so these are the logging levels critical error warning info debug and not set the log with a level called critical is considered to be more important and the log with the level not set is considered to be least important so this is the decreasing level of the logs importance so this tag can be useful to easily filter out important and unimported loss so let's imagine we are creating some millions of logs the logs with the logging level critical might be only intense and the logs with the logging level warning might be only in hundreds so you can easily filter out the logs based upon the level of criticality that's why logging levels are important to easily sort out important logs and if you want to just debug what's application State you can just go through the debug logs or info logs so that's why logging levels are important and these six logging levels are present in the Python's login module all right let's get started with coding now I'll just show you how to do minimal logging that means logging with very less lines of code to create logs you just need to import the logging module right so in a python file I'm just going to write import login so I've got the logging module imported from Python and then I'm gonna just create a long logging dot info and let me name it hello world all right let's try to run this now you see I didn't get any log maybe this logging level is not qualified for printing let's try to change the log level from info to critical and now let's try to run this now and now you can see the log critical the log level the name of the user and the log message itself so this is the log which I just created in a single line of code but you know I just want to show info logs also I just don't want to show only critical locks in the console so let me try to write login.info and let me try to change the minimum logging level so how can I do that it's really simple just write login Dot basicconfig and using the basic config you can control which is the minimum level of logging so let me write level equal to login dot info all right now I've said that the minimum logging level is info so now my log should come to the console so let me try to save this and run it and here you can see the info log is also showing now so using basic config I'm controlling the minimum logging level of the root logger which is being used now so if I write logging dot debug the log should not come because the minimum logging level which I set is in for right debug is a log which is below info so if you just run this you can see there is no log created because debug is a logging level Which is less critical than info the next thing we're going to talk about is how to change the format of logging so here the default format was the level of logging the username which is logging and the message of the log itself so if I want to print the timestamp also how can I do that so you can do that very easily using the format input of this basic config function format equal to and it's a format string and here in the format string there are some keywords which you can use to specify the format for example to show the timestamp you can write something like this to show the logging level you can write something like this level name and to show the log message you can write message so now the log will be something like this the timestamp two colons the level name two columns and the message so this is going to be the log format now because I have set it in the basic config so let me try to save this and run this and now you can see I've got the timestamp the logging level and the log message so I have changed the default logging format using basic config so just using two lines of code now you can create as many logs as you want so limited copy paste this and write something like critical and the log would be this is a critical log and let's try to run this now so now you can and create as many logs as you want in this python code so this is how you can use just two lines of code to create logs in your python codes and you can even control the logging level and the logging format if you print logs into the console the logs may be lost when you just close the console right so if you want to purchase your logs to your text file it's really easy using the basic login config you just write file name input here file name equal to limited to create a file called test DOT log and now the logs will be pushed into this file called test.log so let me try to save this and run this you can see logs are not printed in the console and here I got a text.log file and in this logs are being generated as per our python code now let's try to run this again and let's just open this test.log again and you got the new locks appended here so now I got a record of my logs which are generated by my Python program so just by writing file I'm equal to path of the file you are able to create logs and here you can see the path is a relative path I just given the file name that means the log file will be in the same folder as the python code but if you want to give absolute part something like C users whatever the folder path test.log it will go there so you can use the absolute path also so that's how you can do very minimal logging using the basic config and the next important topic is logging with logger object and log Handler using this approach will give you more options and we're gonna see this in this example now so first let's try to create a logger object so I'm going to write logger equal to logging dot getlogger so get logger function you can create a logger object and you can give whatever the name of the logger you got name loggers now you can write my logger so now you have a logger with a name and then you can set the minimum logging level I can just write logger dot set level and I'm going to set the logging level as logging dot info and now let's see the first advantage of using logger object you can create multiple logger objects and you can use any of those log groups to create your logs but in the previous approach of minimal logging you just have one root logger and all the configuration will be in the same root logger that means now I can create another logger something like logger 2 and I can see the logging level of logger to something like critical and this can be having some other name like my logger 2. so this way using this approach you can create multiple logger objects all right now let's go to the log handlers so just creating a logger object wouldn't suffice to generate logs we have to create a log Handler which will just process the logs and send to the logging destination so let's try to create a Handler that can send logs to the console so I'm going to create a Handler called console logger so I'm going to create a Handler called console Handler and to create a console based logging Handler just try to create a Handler login.tream Handler so stream Handler is basically the logging Handler which actually sends logs to the console and now if you want to change the format of these logs you can have a log formatter actually instead of writing the format string directly you can create a format you can have a log formatted object actually so let's try to do that console formatter equal to login Dot formatter and here you can give the format string so this is my format string timestamp I have hyphen level name a hyphen and the message itself so now I've created a log formatted object and I have to assign it to the Handler right so I'm going to write console Handler dot set formatter this console formatter and now you can see the advantage of using the format or object you can use this formatter object in multiple log handlers and now you got the log Handler which can send logs to destinations but this Handler is still not mapped with the logger object right so just write logger dot add Handler and add this console Handler and that's it now you have your logging setup using the logger object and log Handler now let's try to create some logs logger dot info and here it's going to be something like an info log and let's try to create a logger something like logger dot warning and let's write this is a warning log and let's just run this now and here we got the console based logging using the logger and log Handler all right one more important thing is how can you show the error information in your locks something like extract trace or whatever so how can I do that I can write try and just try to simulate an error I'll just write 1 by 0 which is a divided by zero error and I'm gonna catch that error except exception as e now error is going to be in E and I'm going to write logger dot error and I will say that some error occur and now you can give extra data something like execution info equal to error so here you can use the error information the error object itself and that can be logged into the console so now let's try to run this and here you can see the error log is there some error occurred and the execution info which you have given the error object is also showing here and this can be used for us to trace the error and get insight on what happened currently this error is not being handled so if you want to handle there you can handle it or you can just throw the error again all right let's say to create logs again and here you can see even the fraction of seconds are also being displayed in the timestamp suppose if I don't want to show the fraction of seconds I just want to show date and time I don't I want to ignore this micro seconds how can I do that for that in the formatter you can give the date format using this input called date fmt and here you can specify the timestamp format so if you want my timestamp to be year month date hour minute seconds then only this format will be used while logging so limited to save this and run this and here you can see microseconds are gone now so if I just want to show time just remove this year month date and just run this only time is shown so this way using the date fmt input you can change the date format of the created logs all right now let's see how to use multiple handlers for a single logger so using log Handler has another Advantage you can log into multiple destinations At Once by creating multiple handles and adding them to logger it's really simple so let's try to do that now let's try to copy this again console Handler and just paste it here and now let's try to get another Handler called rotating file Handler from logging dot handlers import rotating file Handler so rotating file Handler is basically a log Handler which can send logs into files so I'm just going to copy this rotating file handle so I'm gonna write file Handler equal to rotating file Handler and you know we can reuse formatter sites I don't need to create another formatter and I'm going to say file Handler dot set for formatter and I can use the same console formatter because I am okay with this format and now you got to give some inputs to this rotating file Handler so let's try to go to the basic input which is the file name so I'm going to give the file name as test DOT log so basically all logs will be now logged into this test.log file so now I got another log Handler which is a file Handler and this can send logs into the text file called test.log but I have to add this Handler to the logger right so I'm just going to copy paste this line logger dot add Handler and I'm going to write file Handler so now a logger has two log handlers and the data will be sent into the console and the file simultaneously so let's try to see this I'm going to save this and run this and I got my console logs and let's try to see if I got my logs into the log file the logs are appended here you can see the timestamp format is also been imposed so from this line I got my logs generated in the file and in the console as well so using this log Handler and logger object approach you can create create logs simultaneously into multiple sources all right now let's talk about the rotating file Handler log Handler you know if you have something like a flash server or some script which will run on periodic basis for years your log file can get bloated and can convert into GBS and it can just crash your Python program So to avoid that you need to rotate locks and rotating locks is very good practice unfortunately it's really easy in rotating file Handler so suppose if you want to rotate logs for every 1 KB of log file size I can just write Max bytes equal to 1024 so that means one zero two four bytes is basically 1 KB so I'm telling for every one zero two four bytes if my log file crosses 1024 bytes just send the old logs to a new file and again create the logs freshly in this log file so that's basically log rotation and there's one more option called backup count so backup count what it does essentially is basically it retains only latest log files suppose you have some One lakh log files you don't want one lakh log files because it may bloat your storage right so if you want to retain only latest 100 log files if you specify backup count equal to 100 automatically the logger will delete the old log files so that you can have your storage less loaded so these are the two options for rotating file Handler Max bytes and backup count now my file handle will actually rotate logs when it reaches the threshold but you know I don't want to create logs of 1 KB so for testing purpose let me try it Max bytes equal to 1 that means if logs exceed one byte create new file so for testing purpose I am doing this I'm just running this again and here you can see I've got my logs rotated now test log test log 1 Test log 2 test log 3 because if the log file size crosses one byte which is obviously if you just write the simple line it will be more than one byte the logs are getting rotated to old files log dot one log.2 log.3 and logs are being freshly populated again in the test.log file so log rotation is really important if your logging continuously and your program is running period vertically days together and months together so always rotate logs and set the right thresholds and the right retention count all right now let's remove this console Handler and let's try to rename this console format as file formatter and now I've got my logs only logged into the file now let's go to another important log Handler which is time rotating file Handler so time rotating file Handler basically rotates logs based upon the timestamp of logs instead of the size of the log file that means if you want to rotate your log files daily it can be possible using time rotating file Handler so let's try to see that with an example now so my file Handler will be timed log file Adder so let me try to write time rotating log file Handler and the first input will be the file part and another input is the backup count which is basically how many log files you want to retain and the old log files will be deleted and then you got an option called when when equal to suppose I write when equal to D that means daily so for each 24 hours your log file will be rotated and obviously the interval interval equal to 1 that means for each 24 hours that means for each day your log will be rotated if you write interval equal to 2 for each 2 days the log file will be rotated so these are the options for the when parameter so when equal to d means for each 24 hours and there is an option called midnight so if I write 1 equal to midnight the logs will be rotated every 12 am in the night and then if you write 1 equal to S and write interval equal to something like 1 for each second your locks will be rotated when you write 1 equal to M and write interval equal to 15 for each 15 minutes your logs will be rotated if you write 1 equal to H that means your locks will be rotated for each 15 hours now and you can write d means days and if you want to rotate your logs every Monday suppose you can write w0 if you want your logs rotate to every Tuesday you can write w one so this way you can have a very flexible time based lock rotation using this one parameter so for our example let me try to rotate logs for every second actually so I'm going to write when equal to S and interval equal to 1. so let's try to save this now and now let's try to run this so here I got my log file with a timestamp let's set log again and now you got another log file with the timestamp and the latest log will be coming here and the previous logs will be sent into this time based log file so one very important advantage of using time based log rotation is you can easily trace the logs based upon time and now I have the timestamp and if I come and see these locks after 10 days I can know that okay this log is of 17 November all right so that's it about log rotation based upon file size and log time stamps all right one more thing that people would like to do is basically changing this extension at the end of the file so something like test DOT log dot timestamp instead of 31 text Dot timestamp.log you want DOT log to be at the end of the file extension because it's easy for code editors like vs code to interpret it as a log file now it's thinking it's a plain text but if you go to test.log it's interpreting it as a log file so how can I change the log file naming conventions while they are rotated there is a very handy function called Neymar so this is how you can customize the naming Convention of log rotation so I'm writing file Handler dot name or equal to a Lambda function basically it's a single line function in Python so I'm writing that if you want to change the name take the existing name and replace DOT log at the end so that I will get DOT log at the end of the file so this is a simple trick where I am moving the DOT log extension to the end of the file name so that my IDs can easily read the log files and interpret the log files I'm just going to save this and run this and now the new log file has DOT log extension at the end instead of having it at the first so if I just open this now my ID is able to interpret because the extension is DOT log so this is one more trick you can use to change the file naming conventions while log rotation all right now let's go to the last topic which is zipping the rotated block files so now you can see a lot of log files are being generated let's try to understand why zipping of rotated locks may be useful suppose in the production scenario I am creating logs and I am rotating logs every Monday that means each rotated log file will have logs of a single week this may be a huge file size or this can be a file size of something like MBS so if you zip each rotated log files you can save a lot of disk space through compressing the log files so let's see how can I compress my my rotated log files so first I'm going to define a rotator function and I need some module Imports for that so let's go through what the Rotator function does is so basically we are zipping the rotated log file and removing the original log file itself so this is the Rotator function which you have defined and then for a log Handler you can Define the Rotator function also so just by using this Rotator function you can just copy paste this I will give the link of my blog post in the description you can copy paste this Rotator function and just assign this Rotator function to your log Handler and then if you just run this now you have a zip file based log Rotator inbuilt in your logging system obviously you don't find any zip files here because the files will be rotated every Monday so let's just say that the log rotation will be each second and interval equal to 1. this is just for testing purposes now let's try to run this and here you got my log file you can see this is an unsupported log format which actually it's a zip file so one thing I have to do here is I have to change my name or function to replace DOT log extension with the dot zip extension when it is rotated because I'm actually zipping the log file side it's not a log file now anymore it's a zip file so I have to change my neighbor function to replace the DOT log with dot zip so let me try to save this and run this again and now I got a zip file so let's try to open the zip file in the file explorer and this is the zip file and here in the zip file I got my test.log and if I just open this log file you can see my logs so using this approach of zipping the locks will save you huge amount of memory if your log file is huge that means if you have some 1GB log file the zip version will be only something like 200 MB so this way you can substantially save disk space by zipping your old locks so that's it guys that is about logging in Python you can see I've created a blog post on how how to do logging in Python I've also given the source code so that you can copy paste and practice it in your own computer so be sure to check out the link of this blog post in the description of this video please share your valuable feedback or ask any questions in the comment section hope you like this video guys thank you for watching peace

Channel: Learning Software

Views: 6,664

Keywords: python, logging, log-rotation

Id: wrpu-Qr_Yvk

Length: 21min 58sec (1318 seconds)

Published: Sat Nov 19 2022