Understanding Python's Functools Module

okay in this tutorial we will be covering the funk tools module this is a module that is built into python mainly for higher order functions we will go through um many of these functions many are um function decorators and we will go through some concrete examples on the most common use cases uh for this module okay the first function that we'll be covering is the reduce function so the reduce function uh basically it's also known as an accumulator so what it does is it uh Loops through everything inside of an iterable and then reduces it to a single value okay Switching to the editor here the first thing we need to do is import reduce and we'll just start off with a list of numbers and I'll create a new variable called total and the first item that we uh first argument is going to be a a function so we'll use a Lambda function here and our return value will be x + y and then we have to pass in our list um as the second uh argument okay so we'll go ahead and run that and we see our total is is 10 so basically this is just much shorter and more concise than setting like an initial um total value and then doing a four Loop and then adding each single time um so this just is a shortcut handy way of doing that so inside of here we notice we have a Lambda function um we can also use any sort of named function I'm going to go ahead and do that next we'll just call add it we take it in X and Y and then we'll do the same thing as we did before and then so we're just passing our function name only as the second argument in our list and print that out again and we get the same exact um total as we did before okay so there's a couple things to keep in mind with this um let's just say we're D somehow generating this list dynamically and then there's a chance that the list could be empty um that creates a real problem if you don't use an initial value so I'll go ahead and um try that and as my sequence I'm just passing in a list with nothing in it so I'm go ahead and save and run it and notice that we get this type error so empty iterable with no initial value so if there's a if that is the case and we know that it's a list of numbers we can just set an initial value to zero and then um we should get a it should run just fine just giving our um total of zero it looks like I made a mistake here pass initial value in but no it iterable so let's go ahead and fix that and then we see our total is zero there's one more case I wanted to cover uh with these examples um you can use it to find a max value it's probably not the best way because python already has that but we'll just do that for illustration purposes anyway so do the same thing as before Lambda function so return X if um X is greater than y else we return y so it should give us the max value and we see four is our max value okay so we can also do this with a list of strings so I'll just create a new file here and then import reduce so what we're going to do here is just concatenate everything so when we add two letters together it's just um in Python it be string concatenation so go and create a result variable and then so we have a single string value of ab C DF and that's just everything in this list concatenated together now we can also iterate through a single string so I'll just create a string value okay so this is different from above where every this is basically a list of characters and now we have a single string okay so we basically have two of the exact same things so what it's going so since a string can be an iterable it's going to iterate through each one of these characters the same way as if it was inside of a list um that is why we're getting um the exact same result as we did uh with both of these examples so that's something to keep in mind whether you have uh lists or strings when working with strings okay so the next uh thing we're going to cover is a list of dictionaries and I'm going to go ahead and paste a uh list of order items so each item is a dictionary it's got an ID a name an amount and a quantity so let's just say all we want to get is the total amounts of all of these so we want return X which is the previous Value Plus Y which is uh the amount and then our iterable as order items so with this we're going to run into an error I'll just show the error and then I'll go ahead and fix it so the problem is is that um since we have a dictionary and it's not just a single primitive type it has no way to know what data type and what what to infer is the initial value so it's thinking that the very first value is trying to add a float to a dictionary so the way uh we fix this is we have to set an initial value of zero since we know it's numeric and then um that will fix the problem otherwise it just has no way of just looking at this and say well what is there to what are we adding to on this and then we have an amount of 357 okay so moving on to the caching functions these are all going to be decorators the first uh method we'll cover is lru cache what this will do is it will save the result of a function every time you call it with every single distinct arguments that are passed into it at lru that is least recently used and um what that does here inside this Wikipedia article here's a visual demonstration of it so we basically have a cache with a capacity of four items and then we just come in here something gets filled in uh with each step and then by the time we get to step five with an index of four since this example zero based um so something has to be dropped so a was the most the last one that was used or not the most recent so that one gets dropped and then um here comes uh d5d is already here so that one uh becomes U more recent than any of these others and then um the way that this algorithm works is um it has to um maintain a fixed capacity okay so in this file we have a Fibonacci method um so the the way Fibonacci numbers work if you've ever worked with them in the past uh because of this recursion the even if you passed in like a modest number like 40 or 50 it's going to take a considerable time to run there's going to be a big lag because of all the recursion so let's just start off by actually running this and I'll just give it a value of 35 so even with 35 there should be a no noticeable lag here okay so we got our result back it's yeah 9.9 M 227,000 something and you notice that it took at least a couple seconds um in order to run this so what's going on here is we're basically calling the same function with passing in the same numbers like all the way up and down so um so we can fix the performance of this quite a bit um using a cachee so let's go ahead and import uh lru cache now the way this gets implemented we use a decorator with the at uh function so this is going to modify how our function works and we'll go ahead and set a Max size of 128 and then actually if you'll notice that um 128 is actually the default and then since we're looking at the document the actual um code here um want bring up is this types uh character so like I said every time you use this function so if you call it with a passing in an argument of three and you know it's going to save uh the result of that function is what's being cached um but if you have this uh typed set to true then um this is going to use up one slot in the cache and so is this so even though we have a 3.0 which is a float and three which is a an integer they technically are are um equal but um these are going to be treated as distinct calls so they're each going to take up one slot in that cache so that's just something to keep in mind um if you uh require strong typing okay so I'm going to go ahead and save this and then rerun it and we should notice the he considerable speed difference see this it just returned the result in a split second so yeah it's just a it's a huge difference um if you have a lot of since we have a lot of recursion inside this function and then one other thing I wanted to point out here was um this is basically using a wrapper around the function um then if we come down here to uh this uh cach wrapper we notice that there's a couple of extra things being added to it um so we're getting this uh me method called uh cach uh info and cach clear um and the cash info is our cash statistics so um this is very useful um we'll go ahead and uh use it as an example here print so then we want the name of our function and the uh method tied to that so cache info let's go ahead and run this again okay so we passed in notice we passed in a value of uh 35 but since this does a lot of recursion um so we we see that we have a number of hits and misses so this means we have uh 36 um items in the cache because it ran uh 36 times or 36 uh distinct times so basically the 36 uh means that the actual what's inside the function actually ran 36 times and then um a hit means um the cach value was used so so basically this repeated um a number of times where the code did not have to be rerun it just saved the result and then just gave us the result back to us and then since 36 is below our Max size of 128 um our cach is actually remained below the full capacity okay so the next uh decorator is actually just called uh cache and what this is is this is just a shorthand for this is actually using lru cache under the hood so if we go inside the implementation um it's basically using lru cache with a Max size of none so um your cach is going to be unbounded so there's no limit to at least theoretically how big the cash can get so that something to keep in mind if you use the um shorter syntax so all we have to do is uh change that and remove any arguments and then um we should get the same result here and notice everything is the same except for um max size in our case okay and the final caching function that we'll be covering here is the cached property which can be used for a method inside of a class so I'll just go ahead and create a new file I'll go ahead and just create a class called customer and we'll take in an argument called orders and this will be a list so I'll just set s so uh self underscore orders will be equal to orders and then we'll create a new property and we'll call this recent orders okay so want to do here is create a list called orders this is just going to give us some dummy data to use um for when we call the recent orders okay this is similar to the example that we were working with earlier um I made a couple differences to it just uh um basically had this order ID and we're going to assume that um the highest number is the most recent I do here is create a variable called orders and we're going to call the uh sorted function this is a built-in uh python function to um help us sort so we need to pass in our um itable which is uh what we have uh self underscore orders and then we need to set a key and we'll be using a Lambda function here and so what we want here is this order ID for every one of these uh dictionary items and then since um the highest number is considered the most recent um we have to pass in the reverse of arguments so um it knows to sort them in descending order and we'll just return the last uh three orders okay so in order to use this um let's create a new customer and we initialize a customer um we need a list of order so that is going to be this and then uh so we'll goe and print that method that we created that's uh property method and that was recent orders so if this is correct and if we go ahead and run that um we get a list back with our last three items so this is order IDE of 7 six and then five however if we run this multiple times so in the real world what what could what happens a lot is if um you call this this is making a call to another service it could be a database query or something like that and then you have two different places in the application where you're um accessing this property what we don't want to be doing is just making multiple queries uh just for the exact same data so what we could do is cash this um actually got the wrong import here and then we'll convert this to a cache property um so if we were to use this somewhere else in the application then this will load from the cache since it's the second time it'll save the um this return value right here so here we just printed it out twice but um that's really all you need to um use cache property inside of a class okay in this section we'll be covering the wraps function what this does is it allows you to um wrap different functions so basically with a wrapper you can um run pre-processing run the function then post-processing this does several things for you uh for example you can just encapsulate code that um any time this internal code changes you only have to change how the everything within the wrapper you don't have to change anything else within the code base another benefit is something that handles retries which is what we're going to which is going to be our second example where basically we're make we're using the requests module to make external API HTTP requests and then if we get back um any sort of an error we can set a delay and then then retry um as long as we're under the the retry limit that we had set out okay so now we can go ahead and import uh wraps from Funk tools and we'll do a really simple example we'll just call this function say hello we'll take name in it as an argument okay so this uh function is very uh concise and straightforward so I'm going go ahead and create a wrapper for this we'll just call it my decorator uh so the function is what we're going to be passing in as the argument now we need to use our at wraps decorator so we're going to create a new function inside of the uh decorator function so we're going to take in all arguments and keyword keyword arguments and actually need a double asterisk here so for pre and post-processing what we're going to be we're just going to use uh print statements to simulate that so uh we'll just print uh before executing function so right here we want to actually call our our our original function that got passed into into here um this is going to actually run the um contents of the function we originally set up and then just we want to just carry forward any arguments that were passed in so this is going to be our post-processing um so we'll just print after executing function and then so what we have to do to original function um so we need to add The Decorator so at my decorator and we save and run this okay looks like we got an error so see non type object is not callable oh and the problem is is that uh we need to return uh this wrapper function so return wrapper so now this should actually work and so we see our pre-processing the output of our actual function then our uh postprocessing so that is basically how um this all works U one thing to keep in mind here also is by using uh this wrapper we basically have access to the original um doc string and the original name of the function so let's just go ahead and add a doc string say say hello say name as a string and we use the double under name and then so we printing out the these double under methods for um our Target function here so since we used um the wraps uh function here we should actually get um the same functions um name and Doc so go ahead and save that and run it so here's our doc string right here and then here's the name of our function um so if I were to comment this out and run this again so we basically got nun as our doc string and then the wrapper function is what's actually tied to these double ender methods um so by including uh wraps um this actually preserves um all the uh all the double-ender data regarding um our original function okay so in this next example we're going to use a more realistic example we're going to be using uh requests to um make an API request so request is one of the most popular modules um in Python one of the most popular libraries so if you need you can install just with Pip install requests and then for testing this out we're going to be using um HTTP bin this is just a website that um you can send a request to and it'll post data back to you so it's just an easy way of simulating requests so if I were to go to this URL for example um it's going to give me some J on data back it's mostly the same data that I originally passed in plus um some headers information that um was actually in the original request so if I were to go ahead and run this as is um notice that everything here is just a synchronous um external request there's no U retries or anything like that um we're just using the library pretty much as is okay and just like in the browser basically got this whole um Json blob back as a result so the next thing we're going to do is use um wraps from Funk tools um what that's going to do um we can set up a means of retrying and then setting a some sort of a delay um between the retries um this what happens a lot with requests um especially external HTTP requests is you get you can have too many requests at the same time and they get throttled so um retrying them after some sort of a delay um when there's less uh traffic or less or fewer requests being made at the same time um then it works the second or third time around this is just putting in a more robust way of um managing having to make requests and getting data back okay so to implement our delay we're going to import the time module and so obviously for from we need to import reps okay I'm going and create some space here so we'll call our new function retry requests and so we'll take in a couple arguments uh the first one will be Max retries give it a default of three and then um we'll set a delay with also a default of three and for now I'll just put a to-do placeholder here so we'll go ahead and create a second function called Fetch data so here we need to take in two different sets of prems so we need a URL and prms and we'll give that a default value of none and we'll say return so we're running um basically just executing this request.get uh the same way we're doing down here except we're just parameterizing these items so we just go ahead and change this down here we'll just say so we'll just pass in our HTTP bin URL and then um set of prams and so since we're going to be using a DE to handle this we want to actually check and make sure we have the response so that's how we're going to be setting this all up um so up here we need to go ahead and um implement this so create a decorator function so the function is actually what's being passed in here if have remember to return the decorator okay so in here we're we're going basically three layers deep because what we're doing here is we have to manage two sets of prams we have to manage these Max retries and then we also have to manage this um delay along with um the original prams which is the URL and the uh get parameters uh from request this is more if we went with something less flexible we might be able to get rid of one of these layers but um in theory we should be able to set up sort of any other um function whether it's get post uh whatever and then be able to wrap this uh using the same code so this is actually the more uh flexible way so what's going to happen is if um this runs and fails it'll retry up to three times um adding a delay of three seconds between each one so next thing we have to do is actually Implement um handling um everything inside of our wrapper here so I'm going to go ahead and set up a for Loop call this for attempt in range so we're going to basically up until our number of Max retries so I'll just set up a try um catch block here and what we need to do is carry forward all of the arguments and this set of arguments is going to be um actually these right here so let's add our accept block request. request exception so what we want to do here is um check the number of retries that we had so we'll set an if block so if so if it tempt is less than uh Max retries minus one let's just go ahead and set up a print statement Source Mark request failed with error and we're basically just converting this error statement to a string so this should give us the actual error message um if this exception gets hit and we'll just print out that we're going to be retrying this so we say time. sleep so this is what's actually putting in the delay we say Max retry is reached so we just go ahead and raise re raise the except ception since there's just no more retries left we just will throw the exception and then we have to remember that if we get all the way through this Loop and then it still hasn't worked we need to return none okay so when we um actually make the API call um we'll just set a new print statement um saying that we're making the recall and then um what Response Code we got back so if we got a 200 if we get a 200 response back we know it went through so a 200 is a status code that means the attempt was successful if we on the other hand if we get like a 400 or 500 level Response Code that's how we know it failed so from the response we should be able to get the original URL so we'll just print that out here and then we'll print out the status code also okay so now if we like I said if we get the status code of 200 we're just returning the response and then this will be the end of our for Loop we should not hit this line of code down here which is why um if we we check for the response if this is not none then we know we actually got this whole response next one we'll be checking for is uh response of uh 429 the way that we can implement it through HTTP bin um we can just come down here to status codes so if we say um getstatus code so let's just try it in here I'll just copy that say status SL 429 so you see we got HTTP error 429 so yeah this actually worked as expected so um we'll go ahead and save this as um a way of testing our um error URL so let's go back to the code and then we'll just say just create a new variable and we'll just pass this in in order to basically test this whole thing out so what we're going to put in here we'll just put a new print statement or say rate limited uh so 429 usually means you hit some sort of a rate limit so we say waiting and retrying and then put in our delay and we'll just go ahead and do one more block we'll just say less than 600 to carry to catch anything that's um not a 200 but still less than 600 so if it's 500 something it's usually an internal server error we say server error and then if nothing catches any of this um we'll just uh raise an exception um so what request uh does um when they return a response so they have this raise for status method so if you do not um get a 200 uh back um it'll automatically raise it raise an exception so um we'll just go ahead and um um since since it's returning a code we're not anticipating we'll just go ahead and raise an exception here okay looks like we got an indenting problem here let's go ahead and fix these okay so if this was all put in correctly um there's a link to the source code in the description also um if this was a little bit hard to follow but um we just go ahead and retry this so This should work on the first try because we're still passing in the valid URL so this should return a 200 response back and then print our result so let's go ahead and run that one first okay so we got the same thing back as before um actually no what I forgot to put in the uh wrapper here so we need to add our decorator so retry request uh we'll just say two Max retries um and then a delay we'll set that to two also and we'll try rerunning this again okay so that's still went through as expected um let's go ahead and switch to the error URL so I'm going to go ahead and use that and since this is not going to return a 200 code this should actually run twice with a short delay and then ultimately return nothing okay so our first one went through and then here's our second one no response received so this is what we expected um the first one failed and it tried it again after a 2C DeLay So I mean this is just an approach that you can use in the real world um so say like you have a request that's intermittently it just sometimes it doesn't make it all the way through and then you can also set some sort of a exponential back off here so you can have some sort of multiplier where based on the attempt whether you're on attempt number depending on how many of these you have um just multiply these together so the first time it would be um a 3se second delay the second time you know 3 * 3 could be 9 for example um so you can you can also do it that way but um yeah this is just a simple way to um get this started here actually there's one more thing I wanted to cover like I said earlier um since we're wrapping this um we should be able to preserve the name and dock string so let's go ahead and do that say double under name and dog okay we got fetch data and none we don't have a doc string inside of fetch data so we just go ahead and add that okay let's go ahead and rerun this and then we have our original values here let's go ahead and then um just a test to make sure this works as we expected I'm going to comment out this of the wraps um decorator here let's uh rerun this again okay so it did that same thing as before wrapper and none so um by including this we actually even though we're uh multiple layers deep we still uh preserve the whole doc string in name of the originally targeted function so okay so the next function that we'll be covering is partial so um what we just implemented before using WS um what that's using under the hood is um partial so partial allows you to um create functions from other functions where you uh can freeze um some of the parameters one or more of the parameters in the original function so uh to illustrate how this works we'll use a uh doubling um function to do this so we just go ahead and import partial and I'm just going to create a simple function here called multiply okay so from that um we're just multiplying two values together whatever arguments that are passed in are going to be multiplied so if we want a function derived from this that only just doubles another function another just doubles a s a certain value uh we can do that uh using partial so I'll create a new uh variable called double it so the first thing we need to pass in is a function um that we're going to be um using partial on and then a value so uh what this does is it um basically freezing freezes this x value is two so no matter what number we pass in um it's going to multiply it by two so even though we decare this as a variable this is going to be a uh type of a function so let's go ahead and just print the type and then let's just say we want to double the number five we can do that so print and then just pass in a single argument of five um because X is frozen let's go ahead and run this okay so we have a a class of fun tools of partial um so it's actually a a WRA function and then our value we got the our answer here which is 10 and that's pretty much all uh partial does now there's some obviously there's some uh sophisticated ways to implement this um like for example This was um passing in um these constants so if we were to look at these this is why uh that metadata was able to uh be preserved when we were wrapping functions uh it's because of this okay so the final section will be single dispatch functions what this allows you to do is depending on the data type of the first argument um we could have multiple um function function implementations so so in this example here um we're basically what we have is a Json converter so we're just um simply converting it to a Json string and then uh this will run just fine as is however um if I were to create a new variable um let's make it a set okay so created a set here um we just go ahead and run this okay so we get this error here object of type set is not Json serializable so the way Json works is that um things like dictionaries uh lists and strings numbers those are all um serializable but it since there's no type of set supported in Json um so we have to do something to fix this problem if we want to convert this to Json so one thing we can do we can use single dispatch to um fix this problem so we just go ahead and import [Music] it so we have to add single dispatch as a decorator to our um this is is now going to be our default function so we'll create a new uh function so we need to add a original function as a decorator and then we need to register it so we registering it with a type of set so anything that is um has a set is this uh data argument will actually uh be executed by this function and then anything that's not a set will fall back to um the default since the uh default functions handling is already named we don't need to name our new function so we can just use an underscore as a placeholder so what we can do here is just put in a special implementation for sets so we'll just convert it to a list okay so now that we have that uh put in we go ahead and rerun this okay and then um so serialized is basically a list of one 2 three which before it was a set so um of course we could have just come in here to the original function and just check the data type and then you know use like an if condition um where you're going to run into this quite a bit is in in uh libraries especially so libraries have these really uh generic functions that can just take multiple um data types as arguments and then the difference between the original function and these other functions that extend it um they can become quite extensive the whole differences between them so um that's why you see them a lot more in libraries than you do um Regular applications but um when you're going through a library it's uh always good to understand um how all this is working and inside of class methods there's also a um single dispatch method decorator um we'll just run through a quick um example of how that works so I'll go ahead and create a new file for that so for uh class methods The Decorator is actually called single dispatch method which I'll go ahead and import and we'll just create a simple class called math and let's just go ahead and take in a value so let's go ahead and create a new method called add and then we return Value Plus item okay so this will work just fine if we set initial uh value as a number and then um every item was also a number um let's say we wanted to add strings and add lists um so we can use a single dispatch method to create different methods for each uh data type so we just need to add The Decorator and so we need to register any subsequent um methods that extended so we'll say string is our data type and we just need underscore uh for the method name we don't need a name so let's say example if this is a string let's just return an F string and for the example we just create one more we'll just uh take in a list okay so for the list um we'll just convert uh value into a list of one thing and then add another thing to the list um this is self-explanatory pretty self-explanatory so we'll just uh go through a very quick implementation here okay so we have our three different data types um we're just testing these three items out and we'll go ahead and run it okay so the first item we get basically just an in an integer value of 10 and then we have a string semic comma space lucky and then we have a new list of 79 and 11 um just as how they're implemented right here so um this is how you can use single dispatch method to use this whole uh single dispatch for multiple implementations of the same name function depending on the data type of uh the argument

Channel: kishstats

Views: 267

Keywords:

Id: qGE01LPxd2A

Length: 48min 37sec (2917 seconds)

Published: Sun Nov 05 2023