Exploring functools (reduce, partial, wraps, lru_cache, total_ordering, singledispatch, etc)

hey guys in this video we are gonna explode yet another interesting built a module in Python called funk tools the funk tools module is for higher-order functions now what I mean by that well actually higher-order functions treat the functions as objects they can take functions as arguments manipulate them and even return them as output now form tools is a kind of module which you do not necessarily need but you can use it to make your code look better and even efficient sometimes so in this video we're going to be exploring some of the important functions prod by from tools module so without any delay let's get started okay so let us start with a function which was not initially a part of the firm tools module I am talking about the reduced function the reduced function was originally an inbuilt function in Python but when python 3 was released then it was added to different tools module and the reason will be clear very soon so we have got from from tools imported use and not first of all let's say a very simple scenario where we might need reduce so let's say we have got a list we have got five elements like this so a very simple way of finding the sum of this list is that you start with the first two elements let us say 1 & 2 then you find the sum of them so it becomes 3 and you have your remaining elements still lying there and now you do your sum of 1 & 2 now you add the next remaining element of your list which is now 3 so 3 plus 3 becomes 6 and you still have 2 elements more and then 2 6 you add the next element which is 4 so it becomes 10 and 5 is remaining and finally you get 15 when you add 10 and 5 so this is a very simple way and this is called rolling computation and you're basically just reducing your problem space by taking two elements at a time just adding them up and then finally you reach a value which is your final answer that is exactly what reduce does and that's in the name actually right so let us see how reduce works so reduce takes two default arguments one is any kind of function and the other argument is any kind of a table your table means anything which can be iterated upon for example let's set or topple so the reason that reduce is a part of fun tools is now cleared because the first argument Telugu States is a function and since I told you before that from tools deals with higher-order functions the functions which deal with functions the function which takes functions as input manipulate functions and even return functions as output so that is what we are dealing with here so let me provide a simple lambda function here and reduce will be taking the functions which take two arguments as input because it has to work upon two arguments at a time for its role in the predation so we are going to do lambda X comma Y two arguments and we are going to just return the sum of them and now time provide provide my iterable which is let's say one two three four five so if I run this I get the output as 15 which is correct right you can do a few more things with this reduced function let's say we want to find the product overall product of your list then just replace plus with multiplication and you get the output because the logic remains same similarly you can find the maximum value in any given list by just replacing that with max of XY so basically in this floating permutation you will finally get the global maximum eventually so yeah so one more thing that you can do to reduce function is that you can provide an optional argument which is the initiator so by default the initiator is the first two values that you have in your list but if you provide an initializer here for example 10 then what will happen is that you will start with 10 and the first value of your list in this case which is 1 so first of all it will be 10 plus 1 11 then 11 plus 2 and so on so if I do it here I think we should get 25 right so yeah so we get 25 so in this way reduce works and I hope that's clear so let's move on to the next function so I hope you must be familiar with the magic functions in Python so the magic functions are basically the functions which you usually define in the in your class definition but you do not need to invoke them directly they are invoked internally when you take some kind of action over your class objects so for example you can define the Equality magic function which is whose name should be double underscore eqw underscore so if you define that function in your class definition and then you try to use the double the double equal to operator over the objects of your class then this function will actually can get invoked so in this similarly you have some other comparison magic functions as well you have less than you have less than equal to greater than n greater than equal to so here for example I have this class called car which takes which has two properties model and mileage and for equality it is checking if the man is same or not and for less than it is checking if mileage is less than the other mileage or not so if I just try to create two objects and if I try to see if c1 is equal to c2 or not I get false and if I try to do c1 is less than C 2 I get true so as simple as that but if I try to do c1 is less than equal to c2 then I get a type error which says that less than equal to is not supported right and it is actually true because I have not written any function here which is double underscore Lewa score which is less than equal to so what I have to do is that I have to just write that function as well so for that I will be just be doing something like this I'll be renaming the function now I'd be making it double underscore news4 putting this thing in here and then running it right so this is one way of adding more comparisons supporting your comparisons but there is one more way which fountains provides you which is called total ordering so total ordering is a function which is used as a decorator and first of all let me import it so from from tools import total ordering and now over our class definition we are gonna put that decorator so now what will happen is that total ordering will automatically generate the remaining comparison operator magic functions for me here so in the docs you can see here this is the total ordering so the minimum requirements which are needed for implementing or using total ordering is that your class must define one of the less than less than equal to greater than greater than equal to and also the class should have the equal to method as well so at least two methods where one will be always equals to and then the other will be one of these four that should already be defined so that according to that reference you can create the other comparison operators so once that is done then it can automatically generate the remaining comparisons right so I have put at the rate total ordering so let me just reload my class definition and now when I create my objects again and I just check the less than equal to it works and even though I have not written the code for that by myself right so it works in that way and if I just try to see greater then equal to that also works so yeah so this is what total order in does nothing too fancy it is quite simple and something to note here is that this decorator does come at the cost of slow execution your code might you might notice that it might be a bottleneck in an application because it slows down your application in some cases and also when some error happens then you might get some more complex stacktrace because of doing this right so keeping those things in mind you should decide if you want to use this or not but for simple cases it's quite handy and I think you should use it when you just don't want to repeat the code again again okay so before I talk about the next function of the form tools module let us talk a bit about the property decorator in Python so what property decorator does is that it makes you treat your class methods as actually class properties so let's take example of this class called mark sheet where you have a great property which were setting at the time of the running of a constructor function right so you also have two class methods which are total and average so these two have been decorated with the property decorator so what happens now is what we're going to see so if I do am dot total basically I am NOT doing Android total function I'm just doing am door total like a property of my class then you are actually running the function you are getting activity total and the value right so if I just run it again you are getting the value again and again the function is actually running right you are not referring to a particular object you are actually calling a real function again and again when you want to get the value of a particular property of your class right so similarly if I do am load average I will be getting this right so calculating average is printed first then tell tween total is printed and then I get the value as 95 so it's as simple as that and now one thing to think about here is that do you need to calculate or run these functions again and again even though you are using the property decorator but that doesn't mean that you have to calculate the total value again and again or the average again because that values 1a remains same so what you can do is that you can remove all this and just keep a property light here you know construct only with seven or total equal to sum of grades and something like that that is an option but let's say you want to stick with the property decorator only then in that case what you can do is that you can use some tools cached property decorator so cache property is actually new in version 3 point 8 if you are using Python 2.7 or less then you will you won't be able to use the cache for a decorator so if you really want to test it then you better use Python 78 so from some tools import cached property and now just replace the property decorator with cached prepared property decorator so cash property decorator is just the property decorator plus the cash in support so let us see how it works so I'm just creating the the market object again and now let me just do am door total so look at that first time I am getting the calculating total printed because the function ran of course but from the second time I'm never gonna see that printed again because I am not actually calling the function again the function is not running again and you are just getting the cash value same thing goes for average you will just see calculating average first and after that never so in this way we are cashing the property value and we are not during the running the function again and again so this is how you can use the cached property decorator from phone tools okay so for the use case of our next from tools function that we are going to look into let's see this Fibonacci function so we have defined a function called fib which takes n as input and it returns the NF number in the Phanatic series so the Fibonacci series goes like 0 1 1 2 3 5 8 and so on so the property of this series is that it starts with 0 and 1 and the next element is always the sum of the last two elements so that is the property of deformity series and we have written a recursive function for it here where we are trying to find the an eternity series number so what I've done here is that I have written a very simple loop kind of thing here where I am trying to print the first 10 Fibonacci series numbers so let me just run it so actually I am also printing like for what value of n I am calculating on deformity series number so if you look at the output that seems fine you have got 0 1 1 2 3 5 8 and so on but the problem that we have here is that we are calculating the Fibonacci series value for the same value of M a lot of time for example for two you have a lot of occurrences like to he's here 1 2 3 4 5 in a lot of places right so and same goes for other values as well so we are recalculating a lot and that is where we can use the concept of caching so caching is like in some temporary space you store some recently used values so that you can use them again and again instead of instead of recalculating it so a very nice module fun tools which we are talking about in this video that provides you something called LRU cache so an Lu cache is used as a decorator and all you have to do is you have to put it like this and provide the Mac size so Mac size is Mac size actually refers to for how many recent how many last recent calls do you want to save the information for example if you put max is equal to 2 then it will the input and the output of the last two calls and if you make it let's say 128 which is actually the default value then it will remember the output of the last 128 calls that were made to that function so if your input matches with any of those last 128 calls then it will return you that value without recalculating it again so that is what maxsize done maxsize does so if maxsize is equal to zero then that is of no use and even if it is one that might not be of much good use if you are not so yeah so if you put max L equal to one let's say what happens right so not a lot of change that you can see which means that you are not calculating flip to and again flip to consecutively so it is not getting used much right let me make the Mac sighs - okay so you can see that the size of the list has decreased quite a lot right so now let me make mac size three which means that you are remembering the last three calls information the input/output so look at that the list has decreased much and to make me exercise four then quite same okay so let us just make it 128 the default value and okay actually it will be printing those right two three four five six seven eight nine are gonna be printed because you have to calculate them at least once anyways so yeah so that is not gonna change okay so we are fine with it and one thing that you can also see here is v dot cash info so flipped out cash info will provide you the information about the caching that happened so it says that there were sixteen hits and ten misses so hits means that the value was found in the cache only and who didn't need to run the whole function again so that happened 16 times there were 10 misses because you didn't have that value already and that is understandable because 4 2 3 4 5 6 7 8 9 and also 1 you had to calculate that value at least once so that's why you are seeing 10 misses here and current size is 10 which means that you are holding up the information or the value of the last 10 calls and the maxsize as we defined is 128 so yeah so that's fine if you make maxsize none then it is like a store which has no limit it will be storing everything so that is also something that you mission that you should know and other than that there is another optional parameter which is called typed so type I think is a new in Python 3.8 actually yeah okay so no it is not new we have another thing which is new but pi type does actually added 3.3 only so what type does is that if you put type equal to false then it will treat the different types for example in tenth floor as two different values so 3 and 3.0 will be treated differently if you put type equal to true actually if you put type equal to 2 then they will be treated differently and if it is false then 3 and 3.0 will be treated as same so yeah so that is fine and the new thing in Python 3.8 is that you can use LRU cache without making it a parameterised decorator you do not need to put any kind of parameters here you can just keep it like this and in this case the value of max size will actually be 128 as we can see here and it will run just fine so yeah so this is how a lot of cache works it is least recently used it stores the most recently used calls up to the given limit for example by default it is 128 so yeah that's it okay so let's talk about a very simple scenario let's say we have got a function called ad which takes two parameters as input a and D and it just returns the sum of them and now I want to create another function which just returns we just increments the value of any given number for example you can just make it like ad 3 comma 1 so you're just incrementing the value of 3 right but you want to have a special function which takes only one argument as input so in that case you can do something like ad underscore one let's say the argument name is a then you put it in return you can just do add a comma one something like this right so you can do this and now the user can do ad one in which they just phosphite so that they get 6 right so it's as simple as that so the same thing that I did here by making a new function and all I can do that in a single line by using something called partial function defined in the front roots module so I'm just gonna remove this one and now I'm just gonna import partial function from form tools so from phone tools import partial so now I can do ad underscore 1 is equal to partial ad comma 1 so now if I do add 1 let's say 4 I get 5 as simple as that so what happened here is that the add function is now by default taking 1 as the value of a so let us just confirm that first so print a comma B and now let me just make the partial again yeah so the value of a is actually 1 and the remaining values now have to be provided by you so that is what you are doing by providing add 1 with value of 4 so that it becomes that becomes the value of B now let's say you want to provide the value of a as one let's see what to put the value of B as 1 by default so what you can do is you can say B equal to 1 so just write it like a keyword argument and here you will have 2 now if you now try to run this you will actually see that now a is getting passed as four and B is seems to define it as one that is getting back as one as simple as that but if you make a is equal to one here like this and now you're on it you are getting an L it is saying that ad called multiple values for argument a so why is this happening it's happening because you are providing the value of a as one and now you are in a way not providing the value of any positional argument from your partial function so when you are providing a positional value from here then it is being set as the value of a also so in a way two people are trying to set the value of a at the same time and that is causing the error so what you have to do is that you have to make this value the four as another keyword argument so I can do B equal to four and that should work so in this way you are basically removing the ambiguity and hence you are able to make it work properly so yeah so this is what partial does it just lets you create partial functions and those functions simply work okay now for the next function that we're going to discuss in the Franklin's module let us see this first like what example do we have a right now so I have actually created a decorator function which is called my logger and what it does is that basically takes your function as input and it has a wrapper it wraps your function with that wrapper function and it just returns that wrapper so this is basically a simple decorator that I've created here and it is doing nothing but it's just printing the name of a function it is like doing running the name function name and then it is actually running the function after that so it is a very simple logger that we have here and then I have put it as a decorator over my function called add and I've also given some dog string to my function add so it is as simple as that so now if I try to do add 1 comma 2 I get running ad and then the output which is fine now if I try to do add dot double underscore name double underscore that should return me the function name look let's see what do we get look at that we are getting wrapper was it happening it's happening because when the function was executing like right here it was inside the wrapper function so that's why we are getting the output as wrapper similarly if I try to do add dot double underscore doc double underscore I get nothing because even though I have the actual dog in my add function but wrapper function has no dog so that's why we are getting nothing so this is a problem because I should be able to get the name of the actual function that I want because the wrapper name doesn't mean much to me when I am actually running the add function right so to get rid of this problem from tools provide so something called traps so from func tools I'm gonna import wraps so wraps is a decorator which you just put over the function which is wrapping your actual function and you just pass the function the original function to it as a parameter it's a parameterize decorator and once you do that now if you just try to run your code again then that name will be fine it is ad and the talk string will also be fine so this is what wraps does it basically makes your original and the original identity of your actual function visible back to you again so that's what it does okay so consider this piece of code which has been written here it is called append underscore one this is a function and it takes an object as input and if the type of object is list then it just puts one at the end of the list if it's a set then it takes the union of the existing set object and give one as a set and if it's a string then it just append string one to that particular string which has been put as input so yeah and if it is nothing it is none of those types then it just prints unsupported type and does back the object without any change so it's a very simple function append one and let us try it out a bit so append one let's say we put one two three then we get one two three four and if we to append one with a set and then we get up and we get one two three back again because Union will return you the same thing here and if it is ABCDE then you get ABCD one so it's a very nice function which is doing this job for you right but now I'm gonna introduce you to the concept of single dispatch generate functions so first of all what is a gendered function a general function is a function which is composed of multiple functions which are implementing the same operation for different types and it is actually quite a lot like what is happening here but it is another thing that here you are doing everything in the same function but what I am calling a general function is a function which is composed of multiple different already existing functions which are trying to implement the same operation but for different types the different data types of the inputs so now which implementation or the which function of those would you want to call that depends upon the dispatching algorithm that you have and that can depend upon the type of the input that you are getting for example in this case if the type of object is list then let's say you want to this operation if the type of object is set then you want to do this operation and so on right and the only difference in what we are doing here and what I am talking about generate functions is that everything has been put together in the same function only but in case of generate function we call the function by the same name but actually the function internally is calling or using different functions which are working for different types basically that's what is happening and single dispatch simply means that in the case of single dispatch you choose which implementation to use on the basis of the type of a particular argument from your input and in our case we already have a single argument so that is going to be object only so on the basis of object we will decide which function to actually call so now let us see what all is happening what all is going to happen here so I'm gonna introduce you to something called single dispatch function in the funcles module so from from tools import single dispatch okay so now what I'm going to do is that I am going to define append 1 again and it is still gonna take the object obj as input and the right now I'm just gonna write test write something called pass so this function is going to do nothing but I'm gonna put a single dispatch over it so now what can I do is that I can create multiple implementations of this function and how is that possible you can simply just need to do append underscore one dot register and now you are registering this particular function to do something now you can also tell on basis of what type or in which case what do you want to run this particular function that you're gonna define here so let's say I want to do that then the type of my object is a list so if the type of my object is list in the append one function then I want to run this function so let me just call it append one list well actually the name of this function doesn't matter to me so I can just replace it with underscore only and the input will be obj so the input has to remain same for all the implementations here and now I'll be doing same thing which I was doing for the list in the original function so just wanna copy that here right and similarly I will write the implementation for the remaining ones so lists then we have got a set and finally we have got a strength so these are the three implementations that I have and what am i doing in each case so let me just replace it with the actual implementation and this will be object plus string one so yeah so this is done so now what we have actually done is that now if I do append one let's say one two three will it work or not so if there is no single dispatch here then it should not work because you are just doing pass so that will do nothing but right now if I run it look at that I'm getting one two three one so why did it happened it happened because single dispatch so first of all single dispatch made your append one function generic and then added the property that on the basis of the type of this object argument so by default it takes only the type it only considers the type of the first argument of a function so the type of object is seen and it is matched with these values and since it is a list then this function is run so now let me replace one two three list with some kind of set then you get the value for a set and replace it with something like ABCDE and you get output so is that simple as that and this thing actually looks much more cleaner right and if I just replace this with let's say an integer let's see what happens you are getting nothing so that is actually happening because the fallback when you are not able to match it with any of these the fallback is your original function so here I replace pass with let's say print the same thing that I had they have print unsupported type and return the object back that is what I'm just gonna paste here like this so now if I just do append one it is saying unsupported type and then just written one so I has simple as that so in this way single dispatch works it makes a general function for you and then it's upon you how you want to dispatch that particular any function call since 500 point seven there has been the support for type annotations so instead of doing all this putting lists here you can simply do object list and here as well you can simply replace it with type annotations in Python so these are used for type hunting and you can just put it like this so this will also work and looks a bit more cleaner so one two three it should work for you yeah so yeah so in this way we can use the single dispatch and it's quite cool when you have the case or a scenario pair on the basis of a particular argument you need to take different kind of actions and there are multiple cases so there you should use single dispatch so yeah so I think that was all that I had for some tools we discussed a lot of interesting functions and I hope you enjoyed it and if it's turned out you can put them in the comment section below that's it from this video thanks for watching

Channel: Indian Pythonista

Views: 2,837

Rating: 4.9310346 out of 5

Keywords:

Id: Jztj_yuFTlk

Length: 30min 56sec (1856 seconds)

Published: Sun Mar 29 2020