Mind-bending metaclasses - adding function overloads to Python

hello and welcome i'm james murphy and welcome to the mind-bending world of python metaclasses metaclasses let you break barriers and assumptions that you had about python consider the example of function overloads if you've spent a little bit of time with python then you know that python doesn't have function overloads here i try to define this function f three different times but actually the second definition just overwrites the first and the third overwrites the second so at the end here there's just one copy of f the last one you could see this by trying to call the function with just one argument if you do you end up with an error the typing module has a decorator named overload but that doesn't actually solve the problem so i would still get an error if i were to run this but what if i told you that with a meta class you can actually get true overloading just to show you how awesome metaclasses are that's exactly what i did now when i run the example if i call f with an int it calls the int overload if i call f with a string it calls the string overload and if i call it with multiple arguments it calls the two argument overload i know what you're thinking this is nothing short of mind bending face melting wizardry and you're not totally wrong before we can understand this example let's go back and understand just what is a metaclass in the first place oh and by the way i'm giving out some professional licenses to pycharm that's this editor that i always use in my videos be sure to stick around at the end to see how you can get a chance at winning a professional license for free every object in python has a type 42 is an int hello is a string the empty list is a list and an instance of my class a is an a classes allow you to make instances of that class but classes are also objects themselves so what's the type of a class in all of these examples the type of the class is a type which kind of makes sense and just like 42 is an instance of an int int is an instance of a type at runtime i can use int to create new integer objects and just the same at runtime i can use type to create new type objects by the way if you're wondering about what's the difference between a type and a class in python 3 there's no difference so if i say something is a type or something is a class i mean exactly the same thing as concepts of the language type and class are identical but literally in the language they mean different things class is a keyword which you use for constructing types it turns out that a class definition is actually just syntactic sugar for a specific way of calling the type constructor instead of using a normal class definition i could equally well call the type constructor directly doing something like this i just need the name of the type a tuple of bases that it's inheriting from and then i run the class body and then store those definitions from the class body in a dictionary called the namespace then i feed those things into the type constructor and out pops a new class a a slightly more accurate picture would look something like this though again i need the name and bases of the class then the namespace or class dictionary is created using this double under prepare method the body of the class just gets treated as code which gets executed directly in this new dictionary namespace metaclasses are a point of customization in this class creation process a metaclass is just a class that inherits from the type class that means just like type the job of a meta class is to create classes by defining our own meta class we'll be able to customize things like how the class namespace is created using prepare or things that happen in the init or new methods of the meta class the default metaclass if you don't specify is type so by changing the meta class of a class that means we're just going to substitute our metaclass in everywhere that you see type here here's the simplest possible example of metaclass here i just have a blank class as we've already seen if i create an instance of the class the type of a is capital a the class but the type of the class is type now let's change the meta class a metaclass is just a class that inherits from type you can change the metaclass of a class just by specifying it as a keyword argument in the class definition so the type of capital a is now my metaclass instead of type this gives another way to think about what a metaclass is a metaclass is just the type of a type or the class of a class if you want to know at runtime what's the meta class of a class then just call type on the actual class calling type on an instance little a tells you what class created the little a so little a was created by big a similarly the big a itself was created by the metaclass so type of big a is the metaclass here's a simple example loadtime meta automatically adds a new variable onto the class when it's created it just computes how many seconds after some initial base time the class was created and then stores that as a class variable whatever we do in new we do eventually need to make a supercall and pass all of the arguments up to type so that it can actually create the class and here we can see that we actually do have the class load time automatically added into the class if i didn't have load time meta as the metaclass then this would just crash because that wouldn't be defined another extremely important property of metaclasses is that they are inherited b inherits the meta class of a and therefore it automatically has its own class load time notice that the class load time is different than the one in a so it's not just looking up the value that was in a this is an important distinction between how meta classes are used and how something like a class decorator would be used you could use a class decorator to add something simple like this just a class variable onto the class but you wouldn't be able to make that property inheritable if you're not doing something that's supposed to apply to a whole hierarchy of classes then there's a good chance that it could just be done using class decorators so what's an example where you would want to modify an entire class hierarchy well how about a familiar example abstract base classes here's a stripped-down version of how abstract base classes in python actually work basically you just inherit from abc and then you mark some functions with abstract method as long as there is at least one function that's still marked abstract you're not allowed to create an instance of the class so i wouldn't be allowed to create an a because f and g are abstract but then i define class b that inherits from a and defines concrete implementations for f and g so i would be able to create an instance of b first this abstract method decorator well all it really does is set a secret attribute called is abstract equal to true on the function and then return it back then in the metaclass we override under call dunder call is called whenever you try to create an instance of the class the default implementation of dunder call is just going to call new to create the object and then initialize it using the init method in abc meta we just change that first we get a list of all the methods that are still marked abstract and then if there are any that are still marked abstract then we just refuse to create an instance otherwise we just use super to call the normal instantiation process another common thing that you'll see is creating basically a blank class that does nothing but inherit from the metaclass doing this allows your users to inherit from that class to change the meta class without even knowing it it's mostly just a convenience factor the abc example was kind of mundane all it did is prevent instantiation of a class it didn't really modify anything during class creation and that's how a lot of meta classes are they do just pretty mundane things meta classes alone are not this big scary thing that you have to be confused about so how did meta classes get their face melting gut-punching reputation there's no limit to what you can do with meta classes so naturally they start to harbor all of your most mind-bending ideas so let's get to it how do i add function overloads into python using a metaclass the key to all of this is the prepare method during the creation of a class like a the body of the class is executed and the local variables that are defined in the body are saved into a dictionary if you define a prepare method then its return value is used as that dictionary in a normal dictionary if i set f to be this function and then set it again to be this function and then set it again to be this function at the end of it it would just be this last function the value of f would just be this last thing but what if i created my own dictionary that when you set a value multiple times it doesn't just overwrite the previous value instead it creates a list of all the values that it's seen that's the basic idea of this overload dictionary so that's what i do first thing the overload decorator just sets the secret value under overload to true then in the overload dictionary if it sees something with thunder overload equal to true it's going to store those values in a list i actually define my own overload list which is just used to tell the difference between whether someone was just trying to store a normal list as a class variable versus if someone's trying to overload functions the first time that i try to set a value for a given key in the overload dictionary i either just put in the value as normal if it's not marked as overloaded or i put it in a singleton overload list if i try to set a value for a key that already exists in my overload dictionary and that key is associated with an overload list then i just append the value to the list so every time i define an overload function f it just gets appended onto this list if i try to set a value in a key that's not associated with an overload list then i just overwrite it like a normal dictionary does here's how you could use an overload dictionary here i define an overload function f and then i just set some values in the dictionary so i set the key a to be two different values but in the final dictionary it just has the last value because none of these values were marked as overload since f is marked as overload when i add it to the dictionary twice instead of overwriting the previous one it creates a list and keeps track of both of them so by changing a's metaclass to overload meta and marking the functions as overload now when i execute the class body all three of those definitions of functions are going to be stored so by using the overload dictionary i can keep track of all the functions but they're just stored in a list so at runtime how do i actually call one of the functions that's what new accomplishes the namespace that gets passed to new is going to be this overload dictionary that was created in the prepare method after the body of the class has executed what i can do is just look through the namespace and replace any overload lists with overload descriptors descriptors are what famously allow you to quote own the dot in python a descriptor is any class that defines a dunder get set or delete method if you set some value of a class equal to a descriptor it changes how gets and sets on that value work if i create an s and then execute s dot some value equals 5 it'll actually call the descriptor's set method with the instance being s and the value being 5. similarly s dot some val we'll call the get method if you've ever wondered about how app property works this is exactly what it's doing so a descriptor will let me intercept attribute access on a class so let's make an overload descriptor when you try to look up an overload descriptor it'll return to you a callable thing that knows how to pick one of the functions in the overload list the overload descriptor takes in the list of possible function overloads it also stores the signatures of all those functions using the inspect module when you try to access the overload descriptor i'm going to return some kind of object that's bound to the instance that knows about all of the overloads i have to split things up into another class because attribute access the time when you access this function doesn't have to be the same time when you call the function this acts exactly the same way that normal function lookup behaves here if i have a class a with a function f i can assign a variable func to a dot f but if little a is an instance of the class and i try to store funk equals a dot f then funk is going to be this bound method this bound method just remembers the little a so that when you call it it calls basically a dot f with whatever arguments you pass in when you construct a bound overload dispatcher it just remembers the instance and a bunch of other things like which overloads and their signatures when you go to actually call the dispatcher it uses some algorithm best match in order to decide which of the functions in the overload list is the best match then it goes ahead and calls that function with its remembered instance and the same args and keyword arcs i just did the simplest possible thing for finding the best match basically loop over all the possible overloads try to bind the arguments to the signature and the first one that passes and matches the types then i just call that the best match so let's see it in action a is a class with metaclass overload meta it has three overloads of the function f one takes an int one takes a string and one takes two arguments looking at capital a dot f we see that it's an overload with three functions in it but for little a dot f it returns a bound overload dispatcher so this object remembers little a and when you call it it will call one of the three functions with little a and whatever other parameters when i call little a dot f of one it calls the int overload a dot f of hello calls the string overload a dot f of one two calls the two argument overload and a dot f of none doesn't match any of the signatures so it raises a no matching overload exception well as you could probably guess using descriptors and checking signatures and type checking at runtime it's a big slowdown let's compare an overload class versus just a function that checks its arguments and does two different things based off of that in this case using this overload framework is basically causing a 1.5 times slowdown so that's not very good if i change the implementation to something quick like just returning 0 then it turns out to be 81 times slower so yeah doing all this signature inspecting and type checking at runtime is extremely expensive now i know the overload thing was the coolest part of the video but i can't let you go without letting you know about another application code generation this is not how data classes are actually implemented in python they're implemented using decorators but here's just an example of another way you could do it let's define a data class meta type when you construct a new class it looks for any annotations on the class then it makes an init function by that i mean it makes a string which is the string representation of the init function so i make the literal code text from this make init function based off of the annotations and then i just go ahead and execute that code that allows me to do something like this where i just define annotations on a function and then it defines the init function for me the possibilities with code generation and reflection are basically infinite so i'm not going to get much more into that but i just wanted to wet your appetite a little bit alrighty that's all i've got stick around if you want to hear the rules for the giveaway so i'm giving away two professional licenses for pycharm and two professional licenses for sea lion their cnc plus editor in order to have a chance to win you must like the video you must be a subscriber of mine and you must make a comment with either hashtag python sealion or both in the comment it must be a top level comment not a reply to someone else's comment you can post multiple comments but doing so isn't going to increase your chances of winning winners will be chosen randomly one week from the date this video is posted good luck

Channel: mCoding

Views: 12,362

Rating: 4.9847093 out of 5

Keywords: python, programming

Id: yWzMiaqnpkI

Length: 15min 44sec (944 seconds)

Published: Sat Oct 02 2021