Python Standard Library: Functools

in today's video we're going to dig into the funk tools module Funk tools is an interesting assortment of utilities that alter functions or methods in some way we'll divide this video into four main sections hashing partials and reduce wrapping and finally dispatches and total ordering my name is Jake and this is a python standard Library we'll start today with what I think is the most useful part of funk tools hashing but you can also check the description for timestamps to each section now Funk tools offers three main cache decorators that's Cash Cash property and lru cache now I seen a decent amount of confusion between the three of these and when you would use them so that's what we're going to talk about today and the first thing we're going to look at is The lru cach Decorator lru standing for least recently used least Rec L used meaning that once we hit the max size that we specified as soon as something else gets added to the cash what would be the 129th entry to the cache whatever's the most stale item in the cach is removed to make room for the new item so we'll only get up to 128 items in the cash lru cach also adds some support for strict type checking for cached items another important note here is that the only things that can be cashed are things that can be hashable it's the same rule for keys to dictionaries you can check if something is hashable by running hash on it and here we see the hash of the string string has this hash value so let's see this an action so we'll clear that go back to our file and let's define a new function using our lru cache and since I wanted to demonstrate the functionality where it pushes cached entries out we're going to set a Max size equal to three and our cache function is simply just going to be add five which is going to take some number it's going to print that's it's adding five to that number and it's going to return that number + 5 easy enough so if we reload this file in IPython we now have that function available and If we were add five to say 15 we see adding five to 15 and it gives us 20 we call it again it doesn't print anything out however we keep adding entries in we get 12 go back to 15 doesn't need to recalculate it and then 27 recalculates that gives us 32 now if we were do one more in here here we'll do 30 it should have pushed something out and what it'll push out is the stalest item which since we called 15 after 12 it looks like 12 should have been pushed out so if we run it against 12 again we should see it print out the text adding 5 to 12 and adding that back in push out 15 so we've run 15 we see it recalculates there's some other cool things you can see from this cache method is that The Decorator also adds some information to the function itself so if you look here under the attributes available we see cash Clear Cash info and cache parameters first let's look at the cash info this tells us how many times it returned a cache value how many times it had to calculate a value the defined Max size and the current size since we're at three we're at our limit and if we look up in the output we should see our two hits here and those two hits in order are here at output two and here at output four everything else is part of those six misses now side cache info we have cache parameters which aside from the IPython warning just gives us the explicit and implicit arguments that were passed into our lru cache here you see we specified Max size of three and the default for typed is equal to false now more useful than that is Cash clear so if you want to invalidate the entire cache you can clear it and after being cleared we see it currently has zero hits zero misses current size is zero so if youall add five on 15 again it recalculates things not too bad so if at any point you decide you need to programmatically clear a cache you have that option available to you just by running cash clear on it and just like that those are the basics of how all of these cache methods are going to work in Python so what are the difference between lru cache and this other cachee method well it all comes down to the behavior that we see in lru Cache if we look at the dock string for lru cache we see something really important here if Max size is set to none the lru features are disabled and the cach can grow without bound what this means is it no longer has to do any checking to see the size of the cash and whether it needs to pop things out it'll grow as much as your system has memory for which is something you may need to keep in mind if you're going to be caching a lot now the authors of python decided that instead of having people specify Max size is equal to none all the time they would create an alas to this functionality here with the lru part disabled by simply letting you delete that parameter and delete the lru part and give you this cache decorator here which is literally an alias to lru Cache Max size equals none so if people ask you what the difference difference between the two is now you know the answer so let's really see this caching in action we'll give ourselves some more room and to demonstrate the caching we'll go the classic Fibonacci route with a really poorly implemented Fibonacci function and what I mean by this function being poor is that while it does work it's pretty inefficient and relies heavily on recursion but it's in this recursion that our cash really shines so we'll also make a hashed version of this literally calling it hashed fib and the way we're going to test this is with just a simple Benchmark called bench FIB we'll just have it calculate the Fibonacci number up to 38 because I think that will more than demonstrate our point and there we go we have a simple bench FIB function which is going to get start time it's going to run our Fibonacci function up to the goal it's going to print out our time taken without caching and then with caching so if we load this up again and run bench FIB you see it's taking some time to calculate it and look at that without caching we're up to 6.81 6 seconds but with caching we're all the way down to 0.0 03 seconds because each time that it had to do the recurr of calculation it was just looking up the value that it already calculated before so when we got really high up in the numbers like around 35 or so instead of recalculating 34 and 33 each of those requiring recursive calculations among themselves all the way back down to one it just returned the previous result giving us one heck of a speed up so just like that we see the power of caching as a matter of fact with this function even with a small cache we'll see a pretty sizable speed up so if we reload that and run bench FIB again you see that even with a Max size of just two we're substantially faster if we bump that back up to three and run it again we see that with a Max size of three we're already back to the same speed that we saw with an Unbound hash size and we also saved a bit more memory there too okay now let's look at our cached property for cach property we're going to do something a bit special because this has to be done within a class so we'll create a new class called mear and these are basically weather observations so I'll write out the class and bring you back when I'm done okay we're back back in our mear class has been written out what this is going to take is known as an IO code which are international codes for airports every recognized airport has them and they're a great way to look up weather information at the airports because the vast majority of airports around the world also have weather sensors there because they're extremely important for Aviation so what this class does is it has a number of properties associated with it it has this data property which is going to download the latest observation for the airport and then return the Json data and then some other properties so we can look at things like the raw observation the temperature dupoint and wind at the time now right now I haven't added any caching into this because I want you to see what it looks like before we add caching so we'll load up the file and we'll create a new instance of our Metar class passing in K RDU as our airport code now when we run krdu DOD dat which is our property it'll pull all the information down of course saying that it's getting the METAR for krdu and here we see all the information about the site and the observations at the specific time if we look at some of our properties here like raw we see again it's getting the mear for krdu it's having to download that data again and process it through the Json loader before it gives it back to us which are then accessing the items from the dictionary here this is especially bad for something like wind speed because we need to put two different fields together they're using self-d data twice so if you look at kd. wind it downloads the data twice obviously this isn't ideal not only because it takes time but if this were some type of production application we'd be hitting the API a lot and this particular API does have rate limiting So eventually they start rate limiting our requests but the way we get around this is really really easy since we already imported cach property from Funk tools we can just change this property into a hashed property and that's it this time instead of krdu we'll use rjaa and if you look at rja a. raw we see that's it's getting the mear for rjaa and gives us our observation now if you look at wind of course this is super basic so I didn't add any error checking in here we'll just strip out the type conversion save it and try again we get rja again Pall then let's look at that wind one more time so for some reason showing none there which we could look into the data and see why that is but the important thing is that we're not seeing we're getting the mear for rjaa again because we changed data into a cached property and that's all it takes to implement a cash property pretty easy right so for the amount of effort that it takes to add caching from Funk tools you really get a lot out of it okay let's talk about partials and reduce partials are definitely more commonly seen around python code than reduce is but they're both neat in their own way so where a partial is used is to build in an argument into an existing function in a way that you don't have to pass it again so we'll use URL open again and return back to a function that we saw in a different way in a previous video and that's going to be our get site status function so the way we would typically use this is we'd call get site status passing in the URL of some website say google.com and we have to do the same thing for any other website that we want you're getting the picture and anytime that we wanted to check the status of Google we do the same thing run get site status ask in google.com and go from there what partial lets us do is to make things a bit more simple moving forward say we wanted a new function called Google status dedicated solely to getting the status of Google well all we'd have to do is call partial we're going to be passing G site status into partial and the argument that we're going to bake into it as a partial method is google.com which is going to create a new function for us called Google status and we can do the same thing for Facebook here again we're going to call partial instead of calling get site status with that we'll just change the signature here called partial get site status with facebook.com we can repeat that a few more times to have specific site statuses so now we're starting to see the use of partial here baking an arguments to give us dedicated functions so if we load this up we should not only have our normal get site status function we should also have our dedicated ones like Google status which we just need to call as is and we get 200 same thing for Facebook status all as is 200 Red Hat status sure enough 200 and finally X which apparently this time didn't like being called X so let's check the site status for it's former name twitter.com and we get a 200 looks like they haven't fully made the change as of the time in this recording and with that you get the idea of how partials work so similar to partials are partial methods because if we look at a class and class methods create a new class here called VM manager which is going to have a single function in here called toggle power which is going to take an argument of self and to state which will then just do this toggling the VM on or off depending on the state that you pass in now if you were try to use the normal partial on this method here it really wouldn't work out too well because the first argument that goes in here is self and that's the instance variable the argument that we actually want to bake into a partial would be two state so we can't really use the normal partial here instead if we wanted to do partial versions of this we need to use partial method and that will look like this partial method we're going to make a partial method out of Tuggle power and what we're going to bake into this is on and we can store this as a new method on this class called power on copy that one more time change on to off and you guessed it we now have two new methods here that use toggle power but are simp simply called power on and power off if we load the file create a new instance of VM manager and say vm. power on we call that we see we're powering on the VM and same thing for power off we're powering off the VM of course at any time we could always call H power instead of explicitly to offer on however we want but with a partial methods here it gives us a way to make little help methods by baking in arguments to another method not too bad right okay let's change things up a little bit and look at reduce now I'll admit this one's a little bit weird and the uses of this are definitely much less common that we'd see partials in Python but I do think it's important to understand the behavior here because when you see reduce it's typically not immediately clear exactly what's going on so let's demonstrate this create a new function called multiply which is simply just going to take two numbers and you guessed it it's going to multiply them together so far pretty simple but let's play around with this I don't think we really need to test the behavior of multiply so we'll skip to just using reduce with it now the arguments to reduce seem a bit intimidating but it's very base level it's going to take in a function and some iterable so let's try it out with multiply that's going to be our function and our iterable is going to be a range from 1 to 5 we'll start simple and we see we get 24 but what's going on here well let's alter multiply a little bit and make it a little bit noisier then try that reduce one more time and now we're starting to see a little bit clearer what's happening here first we get multiplying num one by num two so it's going through our range and remember that evaluating this range will give us the numbers 1 through four so the first thing we see is it's multiplying 1 by two so it's going through iterable the next time through it's giving us two and three so we may think that it's moving forward within our range moving to the next pairing but if we look at this final statement here we see six and four well that's not what we thought right what's really happening here is it's taking the output of the previous run and feeding that into the next run as the first value so multiplies one and two together which gives us the value two it feeds two in and pulls the next value from our range here which is three multiplies two by three to get six then feed six in to the first number within multiply and passes in our last value of four let's try this again this time moving from five to 9 and here we see 5 * 6 is a 30 so it feeds 30 in as the first argument the next time around multiplies that by 7 gives us 210 which feeds into the first argument again and so on until it moves all the way through so this definitely can be a bit confusing right but there are some uses for this one use if it didn't exist in the python stand Library would be to construct a factorial function and we can actually combine this with partial so we'll make a partial out of reduce and we'll pass in the multiply function here which will create a new factorial function and with that factorial function we can pass in say a range of 1 through 20 and that computes the factorial all the way up to this ridiculously huge number which of course this already exists with within the math module so we can check our value there Max here going up to 19 we see we get the same value if you have any interesting uses of reduce be sure to let me know because again I've not seen it used too much but I think there's some interesting cases that could be done now for wraps and update wrapper these are almost exclusively going to be used when you're working with decorators I'm not really going to cover decorators in this video but if you're unsure what they are or want to brush up on them there will be a link in the top right now but for this example let's consider what we see on the screen here we have this decorator print time which is going to decorate some function and what's going to decorate is this perfect function here so if we load this file we can take a look at perfect function it's going to have some attributes to it like under doc saying this is a perfect doc string and it also has its own name which of course is perfect function function but if we were to apply this print time decorator manually overriding perfect function we now have a new decorated perfect function however this one isn't as perfect if we look at the doc string it's missing and even more concerning if we look at the name it's now wrapper that's because this decorator print time is actually replacing perfect function with this wrapper function here so it's clear that really Lo the file and this time let's actually store the decorated perfect function as a new function called decorated so this again is going to call Print time pass in perfect function and now we have decorated which as before has a Dame attribute of wrapper what we can do here is we can use update wrapper to update decorated with perfect function and what this does if we take a look at decorated now look at Dunder name we see perfect function additionally if we look under the doc string we see this is a perfect doc string so what this does is it copies over important attributes from the function that you're wrapping over to the function that is wrapping it now what specifically is being copied over well we can look at the help text for update wrapper and we see that has a few arguments to it the first being again the wrapper function in this example we literally had a wrapper function the function that is being wrapped this one being perfect function and under here we see an argument called assigned and these are the default attributes that are being copied over from the WRA function over to the wrapper function in here you see things like name and dock as well as some other important defaults like module fall name and annotations there's one additional thing in here that's also kind of interesting which is an updated field which will copy over the contents of the dunder dict from the original function now this isn't used nearly as much but it does have some uses and we can demonstrate this pretty easily so for you look at perfect function right now and specifically it's Dunder dict it doesn't really have anything to it but an interesting thing about functions in Python is that they are still objects you can assign random attributes to them so now under perfect function it has a something attribute which has the value test and this will now show up in its Thunder now decorated currently doesn't have that if you're try to access something on decorated we're going to get an attribute error but we can call update rapper again and now when we try to access something we get test because it copied over the entire contents of perfect functions dictionary okay so now we see how update rapper works but how do we use this in our code well one thing we could do here is we could call update WAP pass in the wrapper function as well as the function that's being decorated and this is perfectly valid in fact if we were to test this out ourselves doing another assignment of decorated manually applying that print time decorator to perfect function we'll see decorated. thunder name isn't rapper anymore it's now perfect function so we definitely could do that if we really wanted to however there's a cleaner way to do this and what we'll do is we'll move update wrapper to the top instead of calling it update wrapper we'll call it wraps and this will be a decorator and since it's a decorator that's decorating this wrapper function we no longer explicitly need to pass in wrapper here so we'll just delete that argument and now we have the wraps decorator passing in the function that we're decorating which is perfect function so if we apply this decorator we now have our solution and finally loading the file one last time we have perfect function we look at dock string this is exactly what we expected this is a perfect doc string if we look at its name it says perfect function so as far as we can tell nothing has really changed here but let's go ahead and run perfect function and here we see different Behavior to perfect function we see of course it printed out that it finished being perfect and then we see that it also printed that perfect function took 1. seconds to execute well of course it did because it's perfect and that's the beauty of wraps here is that it keeps all that metadata that someone expect from a decorated function in place instead of overwriting it when you're replacing it with a wrapper function now for the last things we'll cover in this video Single dispatch and total ordering these are some interesting helpers that you may not come across often but when you need them their functionality is definitely handy so single dispatch that sounds weird but what single dispatch allows you to do is to handle different types of arguments differently if you're familiar with some other languages that have operator overloading this is somewhat equivalent to that however it only considers the type of the first argument so let's try this out we're going to use single dispatch as a decorator and it's going to decorate in this example a single function which will be handle error now handle error is going to take in some type of error and we can either provide some type of default Behavior here or what we can do is we can raise a not implemented error as our default Behavior now just like this this doesn't seem pretty useful but it's what we're going to do next is where're going to really start to see some of the magic associated with single dispatch because decorating handle error here actually gives us a new decorator and that decorator Prov provides another decorator called register and in here we can register specific behaviors given a specific type so if we wanted to handle a type error we can create a new function that'll do just that now one common convention you'll see when dealing with single dispatches is not to actually name these additional registered functions is to just give them a single underscore because the name isn't really going to matter in this regard it's all going to be done under handle error again this will take in an error and we'll handle it so now our handle error function if it receives a type error it'll implement this Behavior we can copy this over a couple more times to handle things like value error zero division or any other type of error that we want so let's take a look at this we defined our single dispatch up here which is called handle error we then registered specific behaviors given the type of error that handle error receives so when it sees the type error it's going to execute this function and when it receives a zero division error it's going to execute this function and you can Define as many of these as you need now while this is definitely a much more common practice in other languages single dispatch definitely still has its use in Python if you have examples where you've used it or seen it before tell us about it in the comments now based on our Imports see we've already used single dis patch like we saw before for with caching we also have a method version of this so single dispatch method because remember with single dispatches it only checks the type of the first argument so if I were to create a new class called my num and write out a quick implementation for it it currently just has a single method add it which right now doesn't do anything but for my nums added what I wanted to do is to be able to handle three different things being added to our number their first will be a normal integer the second will be a string and then the last one is going to be a list and each of these needs to behave differently of course in our addit function we can always just do if checks to check the type and then Implement those different behaviors but that's not really exciting and that's not the example we're going to go with today because of course we want to use our single dispatch method and again we'll raise a non implemented error for our normal case and next we'll register our implementation methods we're going to register for an integer then we'll do another first string which is going to do a simple type conversion on another and then finally we'll do one for list which is going to iterate for every item in another and we'll call self. addit on that item that way our list doesn't just have to contain integers it could also contain strings or even other lists containing integers or strings okay let's save that then add another section down here to our scrip toied testing section it's going to test out that functionality and there we go we create an instance of my num called the num passing in an integer 5 then we're going to add in another integer 13 a string seven and then a list containing an integer one a string two and an integer three so let's save that clear it give ourselves a bit more room and run it and here we see under testing single dispatch we have five which is our original number 18 the result of adding 13 to it then 18 + 7 is 25 and then adding in our final three numbers brings us all the way up to 31 pretty good so all of course this could have been done with IFL statements handling each of the different cases here using single dispatch and registering specific m methods is also pretty nice and this is definitely useful when you're not doing something as simple as adding things together or even adding things together with different type conversions if you have much more complex behavior that needs to happen depending on the type of the value that's passed in single dispatches definitely become much more useful okay and then on to our last section here which is going to be total ordering now this is another thing that I think just like single dispatch isn't really named in a way that it's obvious what it does but basically what total ordering does is it helps to fill in missing comparison methods for your class meaning you really only need to Define ther equals and one other method in order to demonstrate total ordering I've created a class here called bad int and this is an integer class whose comparison is based solely on the length of the integer not the value of the integer so let's load this an i Python and take a look at it okay so if we create a new bat endt called five passing in the integer five to it we can start to check its equality statements so if we check to see if 5 is equal to 7 we see true because again remember it's checking just by size now we've also defined the less than operator so we can check to see if five is less than 15 which also is true if we' to reverse that and check to see if five is greater than 15 we get a type error saying that greater than is not supported between instances of bad int and int okay well that's not ideal but there's an easy fix because since we've already defined Dunder equal and less than all we have to do now is decorate bat inth with total ordering save it reload IPython recreate bad int and then redo that check to see if five is greater than 15 and now we see false it's because since we've implemented these two python will use them as the basis for the rest of the equalities so then greater than equal works less than equal Works Etc and we get that all for free just by decorating the class with total ordering and that wraps up this video and this stop on the python standard Library tour if you learn something new please please consider subscribing to the channel and sharing this video with your friends if you have any questions or comments please consider leaving them in the comments section below and as always thanks for watching

Channel: Jake Callahan

Views: 1,678

Keywords: python, python functions, functools, standard library, functools python, python decorators, python tutorials, learn python, python basics functools module, python tutorial, python 3, python programming, functions in python, python partial functions, use partial functions in python, how to create partial functions in python, learn about the functools module for python programming, python caching, python lru_cache, lru_cache vs cache, python singledispatch, python total_ordering

Id: FqCV-9vMXFk

Length: 33min 41sec (2021 seconds)

Published: Fri Oct 27 2023