Python Decorators: The Complete Guide

if you have been developing software for a while you'll notice that there are certain types of features that are really hard to integrate nicely into your code without it resulting in lots of duplication or lots of extra coupling for example logging so you have a function you want to log stuff so do you just add a bunch of lines that log everything and do you do that everywhere in your code resulting in lots of extra lines of code if you have a logger object do you import that and then use that everywhere leading to coupling or do you add an extra argument to every function that needs logging that is a logger object so you get extra arguments everywhere another example is authentication so you might have some endpoints in your code where you want to make sure that the user is actually allowed to call those endpoints so do you then just copy paste that authentication code to every function that needs it and then you have lots of code duplication or do you put it into a separate function that then you have to import and call everywhere which adds both coupling and duplication so this thing things are also called cross-cutting concerns because they basically end up all over your code and there's other examples of this next to the login authentication such as benchmarking or user Behavior tracking via events basically these things are a huge pain in the class so how do you solve it well a possible solution is using decorators that's the classic object oriented decorate the design pattern but in Python you actually shouldn't use that because it can solve the problem much more elegantly so today I'm going to show you how to do that in Python and I also have some thoughts about whether you should use decorators at all because well there are two problems you need to be aware of now before we start I have something for you if you want to become better at analyzing code and detecting where the issues are that you need to solve I've created a free code diagnosis Workshop you can join this Workshop by going to ion.codes slash diagnosis it's totally free it's a workshop that explains a three-factor framework to help you make better decisions about where and how you should improve your code to illustrate how to do this I take a look at some of the existing packages out there something you might actually use at the moment in your own project so ion dot codes diagnosis to join the free Workshop now let's dive into decorators the idea of the classic decorator design pattern is that you basically take an object and you want to add some Behavior to it I'm going to show you a simple example of how this works so I have a function here is prime that basically detects if a number is a prime number the algorithm is pretty basic if you have especially if you have larger prime numbers this is not the type of algorithm you want to use for this there are other better algorithms that I'm not going to touch on in this video but basically it checks if number is prime number and I'm going to use this function to show you how to decorate a pattern actually works now The Decorator pattern has two parts that are important there's a component which is basically the class that has some sort of behavior and then there are decorator objects that add Behavior to that component and can also add Behavior to other decorators so set up a really basic Cloud structure in this example to just quickly show you how this pattern works there is an abstract component class which is going to be the superclass of everything else and this has an X execute method and it returns an integer and gets an upward bounce and what I'm going to do in this execute method so that's where we get here in the concrete component is that it's going to determine for each number in the range up until the upper bound whether that is a prime number so it calls this is prime function and then just counts those numbers and that's what it returns as a result that's what execute does again this is not how you would say 11 python I just want to show you the basic way that the decorate pattern works so that's our component that's the behavior that it has and what The Decorator pattern does is that you then have that's what I have here an abstract decorator that's then a subclass of abstract component so that's this class here and this adds an initializer that itself gets as an argument an abstract component and it stores this in a decorated instance variable so a decorator has a reference to another component and since a decorator itself is also a component that reference can also be another decorator so that gives you like this chain of things that you can add and I've created one example of decoratory which is a benchmark decorator so of course it needs to have the execute method with the upper boundary turns in here but then what I'm doing is I use the birth counter function to record the start time then I compute the value by calling execute on the decorated object that we have a reference to I record the end time I determine what the runtime is and then I'm logging some information about that and then I return the value so what this does basically is it wraps around the original execute method from the self-talk decorated instance variable so that's Benchmark decorator and the way to then run this code is that well I've initialized some logging here but then I create that component so that's the component right here that has the execute method that determines how many prime numbers there are then I'm creating a benchmark decorator which gets the component as an argument and then finally I execute that Benchmark decorate I should probably not have called this component because that's now a bit confusing so let's rename this too Benchmark decorator like so and then here I'm also going to call execute on The Benchmark decorator so what's happening here is I call execute with a number 100K in this case so that's going to run Benchmark decorator dot execute is that records the start time that in turn calls self.decorator.xcute which does this and then it goes back and it locks the information at the end so when I run this then this is what we get we get the long information so apparently this took 0.08 seconds and we can also store a value and then let's also print the value so now we see that there are 9592 prime numbers below 100 000. what's interesting about the decorated design patterns that you can now add extra layers to this extra decorators for example let's say I want to create a class that's called logging decorator I want to do more logging we always want to do more logging and then that's also an abstract decorator and it's gonna have an execute method just like the other decorator right so this code is not what we want but what we want to do is well simply we're going to print some log information here like calling and let's say we want to call Self dots decorate it class name so that's what we want to print like so and I'm just gonna copy that line like so and now I'm going to write value equals self Dot decorate it dot execute and we're going to pass it the upper bounds like so and at the end of course we return the value and now what we can do is we go back to the main function we can add the logging to this sequence of components and decorators so if I add now a logging decorator which is a logging decorator object that gets the component like so and instead of passing the components to The Benchmark decorator I'm passing the logging decorator to The Benchmark decorator so now we have a component that's wrapped around by a locking decorator that's again wrapped around by a benchmark decorator so when I run this then you see we now get extra logging information here and you can also see that this is generated by our logging component and here we see that the Benchmark decorator then calls execute of the login decorator so that's a sequence that we get and apparently because of the extra class structure and login that we're doing that takes a bit longer namely 0.09 seconds and now it's also relatively easy to switch things around because maybe we don't want logging to be part of the Benchmark right so in this case what we're going to do is I'm going to copy this line and and move it over here delete it here and now what I'm going to do is that the Benchmark decorator is going to get the component and the logging decorator is going to get the Benchmark decorator so we're actually not measuring how much time it takes to lock something but we're actually measuring the execution time of determining whether something is a prime number and then what we do is we call logging decorates adults executes and then the number now when we run this we still get the 0.09 seconds that might have something to do with other tasks that the CPU is doing or there might be some other aspects that make it a bit slower like the extra depth in the call stack things like that but logging is now done in a different order first we're writing the log that then calls The Benchmark decorator that then executes the method of the concrete component so it's a bit different and what's interesting about the decorator pattern is that you can basically determine the order in any way that you like and that's also going to affect the behavior of your application now the way that I set this up is really purely following the classic version of The Decorator design pattern that's basically made for languages that don't have any support for functions that just accept classes and then this is how you have to do it because python has really great support for functions instead of having this complicated class structure we can simply wrap a function around another function in other words just have a function call another function so how does it actually work well let's first remove these class hierarchies that we had here and I'm going to change the execute method here into a function and let's also called that count Prime numbers like so and self of course is no longer needed so we have these very simple functions now so how would we decorate count prime numbers for example with benchmarking well what we could do is let me also remove this class is what we can do is change this Benchmark decorator also into a function so I'm going to remove the class name here and this is going to be a function that's called Benchmark Benchmark is getting an upper bounce it has exactly the same signature as count prime numbers but then what it does is it computes the start time and as a value it then does count prime numbers like so it computes the end time and then it prints the logging info and here we probably want to change this to count prime numbers like so lock and decorator I'm just going to comment out for now we'll add that back later and now if we want to use this what we need to do is value equals and then I simply call Benchmark well actually self we should also still delete here obviously and now in order to run this code what I need to do is call Benchmark and then I simply pass the upper bound and let's add back some logging like so and when I run this then this is what we get exactly what we had before oh we're back to 0.08 seconds as well now it's not so great is that currently Benchmark is directly dependent on count prime numbers I can't use the Benchmark function here to Benchmark other functions so that's not really nice right because now if I want to Benchmark another function I basically have to copy paste this code and then replace this function by the function that needs to be Benchmark so that's not so great so instead what we can do is make this a bit more generic and let Benchmark get a function that it's supposed to call but then we still have problem because Benchmark then needs to know what the arguments are of that function and then pass along those arguments but we'll come to that in a minute so instead of Benchmark directly relying on count prime numbers what we can do instead is that Benchmark gets a function that it's supposed to Benchmark right and let's say this is a callable and now we could do something like that it gets an ins and then it turns on ends right but then when we change count prime numbers to that function then we still need to supply somehow the upper bound and we don't have that so there's another way to make this a bit more generic what we can do instead is that Benchmark doesn't get a function and returns an integer but that it actually gets a function with a signature that we don't really care what it is so basically it's of type any and it's also going to return another function so that's also a callable so that has exactly the same signature so Benchmark gets a function modifies it in some way and then just returns that function and what does that function look like well that contains basically this code and let's call that a wrapper function and that wrapper function has arguments and it has keyword arguments so this could basically be any function in Python right and it returns anything any with a capital letter like so and then here in value well we actually call this function that we get as an argument and what we pass well we pass the arguments and we pass the keyword arguments like so and then we don't call this count prime numbers but we call this something like Funk Dot name because that's basically what we're doing right so we have our wrapper function that wraps around the original function that we pass and then we simply return that wrapper function so now Benchmark is generic it gets a function we Define internally a wrapper function that calls that function does logging and then in the end we return that wrapped function how does it work down here well here what we need to do is Benchmark that gets a returns a wrapper function let's call that wrapper here and we need to pass it the function that we want to wrap so that's count prime numbers and then the value is actually the wrapper function that we call and then we pass it the upper bound like so and now let's run this code and you see we get exactly the same result as before but now it's really nice is that Benchmark can basically accept any function that we like now it's really interesting in Python is that there is actually a simpler way of doing this by using the at notation so instead of calling Benchmark Supply the function get another function back and then calling that function there is an easier way to do it which is that we have our count prime numbers and I'm just going to move this down to make it a bit easier to follow and then what we do here is we simply write Benchmark and because we're adding this as a decorator to count prime numbers it means that all of this behavior that we're doing here with creating the wrapper function is not done for us automatically so what we can do instead of this is we can simply Now call Value is Count prime numbers and we don't need this code anymore and now when I run this we're going to get exactly the same result so this add sign in Python is basically syntactical sugar for wrapping Behavior around another function that's why this is called a decorator because in principle it's the same idea as The Decorator design pattern and we can do the same thing for logging so let me just copy over this code and I'm going to use that here and let's call that with logging and then in the wrapper function we're going to have this code that we have right here and of course there's no self dot decorator so we're simply calling Funk Dot and then the dollar name so it prints the name of the function and here we're also calling of course the function with the arguments and the keyword arguments we return the value as a result and then here this also becomes fungal dot name so that works in exactly the same way as The Benchmark function that we had here and now what we can do is we can add a second decorator here with logging and what's nice is that we simply add that here we don't have to change anything in the main function because now when we run this you see we're also logging this by the way if you are enjoying this video so far give it a like it's a nice way to decorate this video and it also helps me reach a wider audience on YouTube but now let's go back to the example and take a look at some of the problems because if we look at what logging actually does is that it logs the name of the function which is wrapper but that's not really what we want because wrapper is like an internal function name of a benchmark so we don't really care about that we simply wanted to write the actual function name so that we know what is happening that the log is actually useful right so how do we do that because we don't have that information here in with logging we simply know that there is a function that we get as an argument we don't have any other information so just wrapping the function like this is not working well fortunately that's a really easy solution and that comes from the funk tools package so let me import tools and Funk tools has a really nice decorator called wraps and that fixes that name issue and now to do that we need to go into our decorator into Benchmark and then we simply add Funk tool dot wraps and then we Supply it the function that it wraps around and let's also paste that here because here of course we have the same issue so now when I run this you see that it actually refers back to count prime numbers everywhere which of course makes a lot more sense I mentioned in the beginning that decorators can be a helpful tool to help with these cross-cutting concerns like logging or benchmarking so how does that actually work well you already kind of see it happening here if we scroll down that we have our count prime numbers function we didn't really have to add anything else to this function apart from this decorator with logging and decorator Benchmark so basically instead of having to copy paste benchmarking or login code everywhere we can simply decorate the function that's a single line and that already makes it a lot less work there might be an issue in that sometimes you might want to pass arguments to these decorators for example with logging we might want to supply it with a logging object or perhaps if we want to have more control over how benchmarking happens we might want to supply that with some settings as an example I'll show you how to do this with with logging with with logging and let's take a look at how that works well in principle this is simply a function call so what we can do instead of having this function that gets a function and returns another function we can add yet another layer on top of it so what I'm going to do is indent this and then rename this function to decorator because it is in principle a decorator and then we're going to create another function that's called with logging so now the only thing I'm doing is simply add another function layer around the original with logging function and then calling that layer above it also with logging and this is a regular function that gets an argument for example it gets a logger object and then this is going to return the decorator function so we need to add another line here return decorator and now instead of writing logging.info we're going to write logger.info so using the logger object here how do we use this well let's say at the top of the file we're going to define a logger which is logging Dot getslogger and let's call that my up and having a longer object by the way is sometimes useful if for example you need to add specific handlers which you can do with the basic logging setup so we have our logger object and now when we actually call the with logging decorator we simply pass it the logger and this code I can actually remove because we don't need that anymore and I see that it works in exactly the same way except now it uses the logger object and we can change things in that object so this is a nice way to supply some arguments to decorators without having to add a bunch of codes to each function that needs it there's not a slightly different way you can also do this in this case if we want to add logging we always need to supply the longer object which is maybe not what we really want to do all the time what you could do is use Funk tools parcel function application to create a decorator that already has the logger argument applied so how does that look like well we can create a variable let's call that with default logging and this is Funk tools top partial parcel application off which function that's with logging and we're going to pass it already the logger like so so what partial does is that it takes a function and returns a new function with some of the arguments already applied so you don't have to pass those arguments anymore when you call this so with default logging is now a function that doesn't get a logger object because you already applied it here and now instead of having to write this we can simply write with default logging like so and even though the IDE doesn't understand anymore that this is a decorator it still is a decorator even though the color looks wrong but now when you run this you see that there is actually an issue because this is a function and not a function call so this is one way to do it and then basically that works again but of course this is not really nice so there's one more thing we need to fix in order for this to work because with default logging is now a function that doesn't take any arguments and that doesn't work as a decorator right because a decorator needs to look like this so because we're using partial here we can actually simplify this again by removing this layer and then adding that back here and now I can de-indent this and let's also call this with logging and then we simply need to return the wrapper like so so now what's happened here is that with logging is again a simple decorator function we pass it another parameter and because of our partial function application this is where that happens and now I can simply write with default logging like so and when I run this code then we get again the same result what you could do now for example is move everything related to logging so the with default logging with logging the logger object to a separate file and simply import the with default logging decorator and then use that everywhere and then you don't have to care about that logger object because that's dealt with elsewhere you simply write with default logging now that's still coupling because we rely on with default logging whenever we want to add logging but it's already a lot simpler at least to add logging to function in this way so decorators are pretty cool solution to solving these cross-cutting concerns because they allow you to easily add benchmarking or logging in this way so should you use them all over the place you might be tempted to do so well there's also some problems with them that I think you need to be aware of in particular there are two problems the first is that it's possible that if you define your decorator that your code actually becomes harder to read how is it possible you might think because decorators really simplify things well the thing is that if you're using a decorator like here for example with default logging we have no idea as a user of this decorator what this actually does and how it wraps around the original function and same with Benchmark so if you use lots of decorators in lots of different combinations it might be hard to understand what is happening for example here do we log before we Benchmark or do we Benchmark after we log we might not know that unless we understand deeply how decorators work and how they are combined with regular function calls in Python so be careful with adding too many layers of decorators to function because that's going to affect readability a second potential problem with decorators is that they might modify the signature of the function in this case we're not doing that we're simply calling the function and returning the value so I'm not changing that but you could in principle write a decorator that returns a value of a different type or that has a wrapper function that has different arguments than the function that it wraps around and this potentially becomes very confusing I did a video while ago about the Hydra package I've put a link at the top that actually does this so Hydra is a configuration reading library that works a lot with decorators but what it does is that you supply a main function and you write at Hydra above it but that actually adds a config object to the main function but you still have to call the main function without the arguments somewhere else so that just really breaks the type system which is a really bad design practice so if you define a decorator my recommendation is to not change the function signature that it wraps around because it's going to lead to issues with type checking and the gamut leads to less readability overall I think decorators are a nice tool especially for these low level things like logging and benchmarking etc for more higher level things I tend to not use that operators all that much myself what I notice is that in many cases I simply rely on composing functions and objects in a logical way and for me that solution works very well in 99 of the cases you know especially if you have a more complex software design it can actually help to rely on more basic features alone to just make sure that the complexity of the language doesn't make your code harder to read by keeping things simple and sticking to more basic programming language features you actually make your code easier to read and maintain in the future especially if you wrote code a while ago that does decorators and then another software developer or intern has to take a look at it they also need to understand what a decorator is otherwise it's going to be hard for them to follow what the code is doing so in the end if you only use these more advanced features whenever it's really necessary that's just going to save you work of explaining that to other people in the future but I'm actually curious to hear your thoughts about this have you used decorators yourself in your code in what way have you been using them other tips you have for everybody simply post that in the comments below I've used a few Funk tools features in this video actually frontals is a really cool python package if you're not using it yet you should definitely give it a try you might want to watch this video next for a talk in more detail about the fun tools package and show you some of the cool things you can do with it thanks for watching and take care

Channel: ArjanCodes

Views: 120,648

Keywords: python decorator, python decorators, decorator python, decorator pattern 2023, decorator design pattern 2023, decorator design pattern tutorial, decorator design pattern, decorator pattern, decorator in python, python decorators explained, python decorators tutorial, decorators in python, decorators tutorial, python tutorials, python functions, python class decorators, decorator python function, decorator python package, functools python, hydra python package

Id: QH5fw9kxDQA

Length: 27min 59sec (1679 seconds)

Published: Fri Jan 27 2023