Python Logging EP 3: Rotating Log Handlers

hey guys and welcome back to another video looking at logging in Python 3 uh this session is going to be about the rotating log file Handler which is a really nice utility to use especially if you're working with a program that's going to be logging Around the Clock uh like many of this programs we use at work do this and you end up with some quite unsightly log files that tend to be gigantic to pass through them especially if you're trying to find some key bits of information so just to kick us off this is from the previous video and we've just defined a standard file log so we Define the logger's name the level it logs at we've got a format here we passed through it uh this is all from the previous video so I won't do too much on it uh so you can see you know if I was to run this a couple of times there we go like that we can see we got this my logger do out here so we can see it sticking in these log messages and that's absolutely fine now suppose you've got a program that logs maybe once every couple of hours um you know maybe it's only you reading the logs yeah this might be okay but if you're wanting to make your logs kind of more consistent uh make them more manageable then a rotating log file Handler is going to be a whole lot better for you so what we're going to do is clear this just get rid of all this here uh and just kind talk about what this does from scratch so the first thing it does is it defines a file which is pretty standard then we can basically tell python once the log file reaches a certain size go ahead and make me a new log file and keep doing that until we get to a certain amount uh and in this video we're going to explain how to effectively create this from scratch and we'll do a simple example just kind of showing this in practice so we'll see it creates some logs uh and then we're going to use a loop to make it add to the logs a lot then we'll see that log rotate here to see this in practice uh so let me get rid of my existing log file uh we're going to bring in a bit of code and we'll get this all done so the first thing we're going to need really simple is to import logging here and then we're also going to do from logging do handlers if I hover over the input here we should see there we go there's a base rotating Handler which actually this actually inherits from that uh you'll never really need this we can just do the rotating file Handler here uh and if I just do rotating file Handler and open this up here it takes quite a lot of different arguments uh so what I'm going to do first is just paste these in and we'll go through them kind of one by one to kind of explain what they need uh of course your first one we're going to need a file name of course um the next one is the mode if you don't give this it assumes a of course whenever you're dealing with files in Python a is to short for append so obviously we're going to be appending to our logs here uh and now the key ones I think this is our first key one here is the max byes so we can specify a size upon which if the log file becomes this big then a rotation occurs so say for example you do 1024 is your max size uh if as soon as that size is hit that file is shelv we then open a new file and we keep going until you know until we come here which I'll come to in just a second we can specify an encoding but if you don't give this it's going to assume the system default so probably utf8 uh and then this is this one here I don't use too much but we we'll talk about it briefly this is a delay so normally when you just go ahead and initialize the Handler it's going to make that log file as soon as you do it however if you pass in delay is true it will basically delay the creation of the log file until uh you do something like loging debug and then all you log to it so could be useful in some situations I don't use it personally but it's there and then finally we have the backup count this is how many log files to keep uh so in this example we we'll do three and then we'll have a go at doing five as well so without further Ado let's initialize our instance of a rotating file Handler and we'll get it all set up I'm going to define the Handler up here first so we'll say Handler is equal to rotating file Handler uh and let's pass in you I don't normally do it this way but you can just do like log. out uh but if we pass them in kind of like as these keyword arguments it might just keep it a little bit nicer um so what we'll do I've got a folder called logs here we'll put them inside there so we'll do logs um let's just do log. out like that so because I've got logs slash it's going to all stick them in here just to keep it nice and nice and uh nice and tidy for us that'll be nice uh the next one is mode so of course see how here it's predefined as a so we can leave one else it's going to assume that we're appending and the next one is going to be the max bite for this I think for for demonstration purposes it might be best just to keep this kind of small fun now if you make this really big we going to be logging to it for an awful long time before we see it actually do a rotation uh and then let's see what we can do let's do a let's do a backup count equal to fre so if I do backup count here equal to fre there we go that's a super simple instance here uh we'll change this to a bit of a bigger one later on uh but you could easily get away for a simple application of just doing this so we're basically saying in my logs folder create a file called log.out as soon as that single file reaches 64 bytes go ahead and make a new file and the maximum you're going to keep is free so once you got free files all filled up it's going to delete the oldest one and then kind of keep shifting it so you're always going to have the newest log file and then you'll also have two additional files as well uh now we've got that we can simply say take this Handler and just like we applied the file Handler to our logger um yeah we do exactly the same thing here so it's nice and simple so we'll say my logger is equal to logging dot let's do get logger and we'll keep it nice and simple call this my logger here and now we need to say for this logger go ahead and add this Handler to it so we'll do add Handler like this and just pass in Handler and then we're going to set a level as debug so we'll do logger do set level and let's do logging dot debug like this and there we go so that's that's a simple kind of use case done let's just hit run uh we've got logs here and see how already we've got log.out there we go nice and simple uh so what we could probably do now is have a go at kind of doing some logging inside of Loop uh to see what happens so what I'm going to do to make this uh make this nice import time just so we can kind of see stuff happening over time uh and then down here what we can do is we can do for blank in range uh let's do uh let's do 20 uh and we can make a statement here and we'll say in this statement as an F string um we'll say inside this F string the time is now blank like this and we're going to need a string to put in here uh so we'll do St Str of time. time like this and I'm just going to log this 20 times and see what happens so if we now do logger debug uh statement like this here okay so one more time we're saying for 20 times Define an F string so this is going to say the time is now time. time hence why I imported it here and then we're simply going to take this string and then log it actually before I run this I'm going to just to make it a bit more realistic I'm going to increase this to 10 24 bytes here and we're going to go ahead and hit run okay so we got a log file there we go right looks good okay so we can see see how this is logging now the time is now and it's you know it's kind of grad ually moving on through time and that's fine let's run this a couple more times okay now we've got log out. one so that was that's our first rotation there so this was the newest one and then what happened was that then got moved to here and now this becomes our newest one so what we should see is that kind of rotating as we run it like got do two keep going I keep going now see how I've run it quite a number of times but it didn't exceed the number of backups here that's obviously cuz my backup count is free so this is our main log file and these ones here just the backups so again yeah it's a super nice way of making sure that your log size kind of doesn't get out of control uh let's try it with backup count of five we'll go ahead and delete the main log file and all of its backups just move those to trash and this time here uh Let's do let's do 100 and let's do 2048 here okay so now we're going to run a much bigger Loop actually you know what let's do a th so we'll try a much bigger Loop here but we're also allowing for much bigger bite size uh I'll run main. P again there we go that's it so you can see the time kind of increasing as we go along here and to be honest without going into too much detail that's that's kind of effectively it uh let me show you uh how we can use that delay feature just for a second so if I run this here I'm going to quickly comment out this and let's do delay is equal to true so it'll be a bit different this time the actual file won't get initialized straight away yes even though I hit run there this didn't happen so this means because I put a delay on the the actual file is only going to be created as soon as we log to it which is quite nice and there we go just like that uh in terms of that the only things I would ever really have to use in this um sometimes encoding um but generally like it's always going to be utf8 on your system uh and again I'll reiterate the I think the two most important things in this your max byes and the backup count here so just something to bear in mind that if you're logging to a system which perhaps has like limited resources um this is a much better way of kind of managing your log files if you're just using the normal file Handler it's quite easy for your logs to just get really really big however if you use this here you know all you're doing is you're taking the maximum bytes multiply it by the backup count and that's that's kind of your max theoretical size for your logs and then we have it cheers for watching guys and see you all in the next session

Channel: James Clare

Views: 271

Keywords:

Id: s1aO_M_Vj3k

Length: 10min 45sec (645 seconds)

Published: Wed Jan 24 2024