"It's Pythons All The Way Down: Python Types & Metaclasses Made Simple" - Mark Smith (PyCon AU 2019)

hello everyone and welcome to the first morning session in this room for the day next up we've got a long technical deep dive so if you're not prepared for in 70 minutes 70 minutes of really interesting topic then get ready so we've got Mark Smith presenting its pythons all the way down python types and metaclasses made simple okay hello everybody thank you for coming to the talk and congratulations on dedicating yourself to fully understanding pythons kind of deeper magic this is this is maybe the start of a wonderful journey for you into building things that work in ways that people don't expect maybe I can help with that so in a moment I'm going to talk about Python types descriptors metaclasses I am going to hopefully make them simpler simple is a very bad word to put in a in a talk especially about a relatively arcane topic because it essentially means that people coming from different backgrounds some people will find it simple some people will find it immensely complex and both of those a point of view are valid so I'm kind of wishing I hadn't put that in there it's in there maybe maybe simpler is there a better word but before I start talking about descriptors metaclasses and deeper Python and magic first I'm going to talk about something more important which is me my name well I've been known as gt2k more or less everywhere online github Twitter etc but my real name is Mark Smith I'm a developer advocate for next MOU we are a sponsor of the conference I live in Edinburgh I run the local place the user group I'm also an organizer if you're a Python and PyCon UK and do a bunch of other stuff okay so you may not have heard of next mode could I just have a quick show of hands who has heard of next mo awesome that's why we're here we are a I won't talk this is a commercial break I won't talk about this for too long we are a set of we provide a set of web page web based a P is that allowed developers to write code that sends it and receives SMS messages makes voice calls does video streaming sends messages via whatsapp viber facebook Messenger and basically any type of medium between any type of platform that's the idea if that sounds interesting please do come and talk to me afterwards I have voucher codes that will get you set up and running with some credits but that's enough about me and next moment almost so this talk has a bit of a history goes back eight years so eight years ago when I had only been a professional Python developer for a very short time for some reason I decided to give a talk at Europe I Thurmond called a deep dive into Python classes but as part of the abstract I specifically said I won't be talking about meta classes because I didn't want people to be disappointed the main reason I didn't want to talk about meta classes was because although I had written a couple I didn't really know how they worked and I certainly wasn't confident talking to other people about them so it's now eight years later I've written a couple more I understand a little bit more about them and but I still I wanted to know the nitty-gritty and the best way to know the nitty-gritty about something is to to give a talk on it to actually do that research so this is this talk is basically the result of all that research so here we go the basic talk is divided into three sections first I'm going to talk about Python types which is a conceptual idea more than anything else I'm going to talk about Python classes which is pythons implementation of types and then I'm going to talk about quite a meta classes which are kind of an underlying layer beneath classes and how they work I will stop specifically for questions at the end of each one of those sections so if you have a sort of something that you haven't quite understood in that section like these you'll get a chance to ask a question it will also break up the talk a bit it's a very long session but if you have a question just while I'm in the middle of speaking if I'm not explaining something very clearly or you would just not quite managing to keep up with some of the concepts feel free to stick up your hand and ask me a question and I don't wait for a microphone I will repeat the question and then hopefully we can keep everybody kind of moving along for the whole session if you don't understand anything that I'm explain fully you will not be the only person in the room who doesn't understand it so really they build up their confidence to ask the question other people will appreciate it so starting with the the first of three Python types I am NOT a computer scientist I dropped out of my computer science degree and so the first thing I did when I wanted to talk about Python types was to Google typing in and I came up with this definition Wikipedia which kind of matches my own understanding in computer science and computer programming a data type or simply type is an attribute of data which tells the compiler or interpreter how the programmer intends to use the data a variable can take the type of data type constrains the values that an expression such as a variable or function might take the data type defines the operations that can be done on the data the meaning of the data and the way values of that type can be stored so I can summarize that essentially as a type in Python defines the capabilities of a piece of data or a group of related pieces of a data it constrains what can be done with that data so it tells you things that can't be done with that data and it describes a semantic meaning to that data so this is more a conceptual idea it's something that's useful for the developer more than maybe the interpreter it's I'll touch on that in a moment so just to go into those in a little bit more detail a capability is something you can do with some data and here are two intz and I'm sure everybody in this room knows that you can add two intz together and it returns you a new int this is a capability of the int type that you can use the addition operator with them if we look inside int we can see how that capability is defined so we use the dear function on the int and it gives us a bunch of stuff but which would just fill up the screen but the main thing that's important is that there's an attribute on the inter class called dunder at and Python uses this behind the scenes every time you call add it is calling the dunder add method on the left hand side there's there's some exceptions here as well but just this is the simplified view so this this is how the capability of numbers together is defined in Python this is what pythons do behind the scenes so constraints one of the constraints in Python 3 is that you cannot add an integer and a string together you get a type error it has tried to add these together and essentially that method has erased this error for us so this is something can't do the data because either the method isn't defined or the method has explicitly thrown an exception to say you can't do it and then the third part meaning as I say this is more conceptual idea and let's say we have a variable called D and we don't know what it represents but if we call type on it the type function will return us the class that is sitting behind D essentially the set of the meaning behind this this variable and we see that it's a date time which means that we know by reading the documentation about the date time class that D is a representation of a point in time and so we can do operations like move it forwards and time backwards in time and we can express it in different string forms things like that so this is meaning for the developer it's useful for debugging and just kind of tracking tracking the way data is moving through your program and what that data means alternatively we might have a variable P we don't know what that is so if we call type on it we can see that it's a functional method and assume you know this encapsulates some executable code a set of operations that we can pass some data to it will conduct some operations on that and potentially some of the global state as well and maybe return us a new value so this is again even this has meaning like a really low-level data type like a function has it has meaning to us we are meant to execute this constraints and meaning go hand-in-hand NUMA programs are programmers are sometimes upset that they can't for example just concatenate a string and an INT and other languages do allow this but you end up with this again with meaning like what does this mean are we trying to convert both of these two numbers and end up with 24 as we did before or should we convert them both the strings and end up with the string 1212 which I have in here somewhere so with the the literals at the top of this the first expression here it's really obvious maybe what we probably want from this but when you have two items that you say read from a database you don't necessarily know what types they are you probably don't want one of them to be coerced into another type and then the operation conducted because you don't know what the end result is going to be you don't know whether you're going to end up with a sensible expression or whether you've just got the wrong bit of data back and you've now like munching them together in some way that doesn't make sense so it's good that Python is strict about this in these cases at least because it doesn't know what decision to make it doesn't know what decision you wanted to make with these these random bits of data that are flowing through your program so it place plays it safe and tells you you did something that doesn't make sense so it's up to you to explicitly turn the string into an Internet or to turn the int into a string by by essentially passing it through this their string function so python has quite a few core types they're described in a document called the data model these are the ones that people tend to interact with day to day as eight of them a boolean expressions true or false values we've got into integers which are numbers without a fractional part with floating-point numbers which are numbers with a fractional parts complex numbers which are just like some sort of weird ad idiosyncrasy about python that it has a core type there represents complex numbers we've got byte strings which are an array of bytes we've got string which is an array of Unicode code points we have functions which as I said before encapsulate executable code and we've got class which is actually a bigger topic which will be we'll be covering later and class is essentially pythons expression of custom types they're your way of creating new types in Python and we'll cover some of the details of that in a bit if you call the type function on values of like instances of each of these types and it will hand you back the type essentially the class of that piece of data so you can see if we recall it on true we get back a class that although this is being expressed with like a less than greater than either side of a string but that's actually a class value there so you can do so you can introspect it as a class we've got the bull class flow class complex etc we don't really have expressions for functions so I've had to define that as a statement so I've defined a function called fun proposed fat to type again we got a class function so even even executable code has a class behind it so this is a fundamental building block of Python and then apologize for this I thought this was funny when I wrote it the class war so I'm gonna be using the word war a lot in the next couple of slides so if we call type one our war class the type of a type its type this is this is part of a Python slightly nasty way of dealing with meta classes and this is also our first glimpse of meta classes everybody say oh for the benefit of people watching the video they all said it so we've got the type of a type is a type of a class is type type itself is in fact a class so I've been calling a function but it's not a function it's a class and the type of type is itself type this is how mind-bending metaclasses get quite quickly which is why I've broken this slide this talk down into a 17 minute talk on on various different aspects that lead up to how metaclasses work but this is definitely like probably the single most mind-bending part of how type works in Python so yes I'm just going to show you another idiosyncratic thing about Python which I think expresses something that's useful to know up front so the type of our class war is type and if we put the type of type is type as I just mentioned on the previous slide so those have the same capabilities and constraints because those are expressed in types in Python so we should be able to do the same thing with them and classes are mutable which is kind of a fundamental thing about them that maybe I probably shouldn't be taking too much advantage of but I'm going to show you some ways to do that during this talk anyway so here we've I've just added an attribute to our class wall with a nonsense value and now if we look inside using the dear function we can see that our useful value has been added to our clothes so we can because it's got the same class as because it's got the same underlying type as type we can do the same thing with type except that we can't because type is not mutable and type is implemented in C it is a built-in or extension type so although it it looks like the same type it behaves in many ways the same it is not quite the same and this is really an another aspect of Python when you write a Python program every expression in Python is being executed by the Python interpreter which sounds really obvious but my point and you'll see various examples of this during the talk is that Python does things for you that you don't always know about if you don't know exactly how quite an execute code Python is always trying to help you out that's try it is much more powerful under the hood than you may first expect when you're learning how to write basic Python programs so that is the end of the types section which is the shortest section of the talk does anybody have any questions okay so there are more things we could cover about types as I mentioned during that this section and classes are pythons expression of types within the language and so it will definitely be covering some more of these concepts during this section so a useful question to ask ourselves when talking about classes is what is a class I don't I I found classes I learnt Java back in the 90s and I it took me weeks and weeks of writing programs with classes in because you can't write Java code without classes again in the gonna again just using them until one day they clicked and it was like oh I get what classes are for I get what instances I get how they relate to each other and how I might use them to build programs I'm not sure when you come to place and I think you see boilerplate code a lot and you don't have to work with classes immediately and I'm not sure everybody asked themselves this question up front so I'm asking it now so a class or and types themselves are different in dynamic dynamically typed languages and statically typed languages a dynamically typed language like Python tends to encapsulate a bunch of behaviors as I said earlier in a class or set of classes that relate to each other in statically typed languages they also tend to be a restriction you can also get this with type hints in Python where you have a function that will only accept a certain type or a type that implements a certain interface at that kind of behavior so it becomes a much more important thing how you structure your classes because if you want to call a certain function you will need an instance of a specific class so you may need to inherit from it etc python is much more flexible with duck typing as long as the type looks a bit like the thing that it is expected then that function should operate on that data as expected so in Python the class is just a structured to find used to define new types it's it's less of a kind of integral thing in the language it's really a set of instructions for creating an instance later so you will have a one-to-many relationship you'll have a class and then you'll have many instances of that class a class is a callable much like a function and it returns an instance so I don't know that everybody always thinks about that is that you class looks like a function and that's kind of important and on top of this as well as just being a function that returns an instance it also provides behavior the class hangs around so an instance maintains a reference to the class that created it it's used for looking up methods which provide behavior but they belong to the class and not the instance and this is because python is secretly a prototypical language and not a classical object-oriented language so it's yeah there's there's some conceptual stuff there that I'm not going to go into because we only have 70 minutes so to my mind a class is a thing for constructing instances I don't really like the word object because both classes and instances are objects in Python as I just mentioned if I couldn't get to metaclasses you'll see that the distinction between classes and instance is largely arbitrary and so the word object just means anything I will be using it from time to time but it is literally like the broadest Vegas term for a thing I use a word thing interchangeably with objects as well so a class is a thing for constructing instances so here's a simple example I had to find a class called car it has a initializer function which just takes a value and sets it on the instance which is self and then I've got a drive function which just simply prints out a message when you call it and then the last line instantiate set so it creates an instance of car and passes at the value read which will be stored away on that instance so then when we look at the instance if we just print it out we can see that hey we've got an it says object just imagine that says instance that and we can look inside the instance and we see a whole load of stuff there that we haven't defined that we've got for free but the important thing is at the end we've got color and drive the two things that we defined one the color is on the instance and drive is on the class so if we look inside the instance dictionary so unless you've implemented slots instances store all their attributes in a dictionary called dunder dict so in this case we've got this dictionary it contains the one item that we set on the instance which is color but where is our drive function well if we hope we can look up the class as I said instances maintain a reference to their class in the dunder class attributes and so this is now a class car if we look inside there that also stores its attributes inside a dictionary called dunder and if we look at that further there's two things to note first is that it returns a mapping proxy not a dictionary so it's a read-only dict I don't know why they did that guessing it's an optimization if you want to add stuff to a class you totally can by just using the full stop or a period to kind of to assign an attribute to that class and the second thing to note is that we have our drive function in there and I do want to highlight I'm using the word function very specifically here and I will come back to that later drive is a function called car drive not a method the instance sorry the class also stores the base classes that it inherits from I didn't specify any classes to inherit from so by default it just it's directly from object so you can see if I refer to the dunder basis attribute this is a tuple and it just contains one item which is our object class so this is our inheritance hierarchy or helps define our inheritance hierarchy for looking for things that are defined on instances and I'm going to get to that in a minute oh yes and here if we look in object because object is the thing we're inheriting from if we look inside that dict then we can see all the other stuff that was on our instance you remember when we did dir an instance it flattens everything that's in the instance and the entire class hierarchy into one dictionary so just for ease of access really and the other thing to know as I said is that usually classes are Singleton's so we'll have we've got object at the top there's one class called object there's one class called car and then we could instantiate that multiple times but they all refer to the same car so when you call drive there's only one drive function and it's defined on one instance of car so we haven't got look if you define a class with lots of behavior that's not duplicated if you create a thousand different instances of that class so looking briefly at how we define classes I suspect most of the people in the room really know this stuff so I'm not going to cover this in huge amounts of detail this is what a class definition looks like you give it a name and an optional list of parent classes direct parents you can define a dunder new function inside that class this is a class method it's a special case class method so you don't need to declare it as a class method so it takes the class as a first parameter the class has not been instantiated at this point it is the job of dunder new to instantiate the class to create an instance do whatever it wants to do with it and then return it by default you were calling objects dunder new function and it will return you a new instance of the class you provided it which is your class and it won't be initialized at this point so if new returns an instance of the current class or a subclass so that's it will then be call it were in it will net then be culture in it is the next function I'm going to cover if you return return anything you want from here if I'm defining a class I can return a string if I wanted to but if I return a string it's my job to initialize it it will not be initialized by calling the dunder init function dunder init as I said the initializer function so the clock the instance has already been created it has nothing set on self this is where you would set all the instance attributes that kind of thing so let's talk about inheritance this is going to take a while you may not expect it to take awhile but I think it's justified so what happens when you get an attribute certainly the diagram I have in my head looks something like this so we look on the instance if it's not on the instance we look on the class that was used to create the instance and if it's not there we go up the hierarchy of super classes and if it's not found on any of those then it raises an attribute error this is overly simplified so let's make it more complex so first there's dunder getattr so if you define this this acts as a fallback attribute accessor it's passed the instance and the name of the thing that you're asking for and you can do whatever you like in there so basically what happens is if it looks on the instance and doesn't find it and it then looks up the class hierarchy and doesn't find it just before it throws the attribute error it then calls done to get out which can do anything it can go across to go to the network it can load files it can do do whatever you want and if it can generate a value that makes sense to the user it can return that or it can raise the attribute error then there's get attribute which looks very similar but it is really quite different how you don't want to mix these up so this is called before anything is looked up on the instance it's defined on the class it goes here at the top so before it even looks himself to find something the first thing it will do is called under get attribute and then to get out to be can return whatever it wants to so this overrides essentially all the behavior of like looking on the instance going up the superclass and things like this you're actually calling this anyway because this is defined on object so this is your way to completely magically change the way attributes are defined the way they look it up on an object it's unusual to use it it's also really easy to write recursive version of this function that just calls itself until the interpreter crashes if you raise an attribute error it will jump straight to the guitar for your kind of fallback function so really it does kind of bypass all the other behavior in this in this sort of this cascading sequences of operations so I'm just going to take a brief break from this diagram it is going to get more complicated but first we've been talking about going up the class hierarchy and I think we will at least vaguely know how classes work in a hierarchy it gets complicated if you have not if you were using multiple inheritance to extend from two classes and they extend from the shared base and things like that but there's a there's a well described algorithm for how we traverse that that graph of super classes and get mro will essentially return you a flattened list of this is the order through the super classes that I will go to try and find data could be useful it can also be overridden and I'll show you how to do that in a bit super is a function that essentially provides you with an object that you can look up attributes on and it will look through the hierarchy using the get mro order to return you value so essentially it looks on the instance and then there's actually it doesn't look on your current instance or current class it looks on the superclass isn't above this used to require you to pass the current class and the current instance into it as parameters if anybody's done any Python too recently will easily remember it still does behind the scenes what Python does is when it finds a call to super it rewrites the bytecode to pass in those parameters there's a special value assigned to functions that allows it to find the class that a function was defined pon so really actually nothing's changed this is just they've done some semantic syntactic sugar to sort of change the way that this but you actually write this which is great because we've our Gillian complex before so now let's talk about descriptors so I just have a quick show of hands who's who knows how descriptors work okay that's that's good so you remember what I said earlier about Python doing things for you descriptors are kind of the ultimate example of that in my opinion so this is where we are so far with understanding how attributes are found on instances but in the Python data model there's a description of more rules that need to be assigned to this diagram called the descriptor protocol so it's essentially a set of a set of rules or a set of behaviors that are encountered it's kind of how Python works underneath the hood and I think is before I explain how they work I think it's useful to describe what they're for and three of the things that descriptors are for are to allow these decorators to work so if you use property class method or static method and they change the way that data is accessed on a class or an instance so property used to decorate a function and it means that when you then access that attribute it calls that function behind the scenes and then returns you the result and this sounds like sort of dark specific Python magic it's that maybe you couldn't implement your own property decorator if you wanted to but you totally can it's pure Python and I'm gonna kind of show you how how that works or at least the the way it works and in class method and static method if you think about it you'll you defining a method that either takes a class instead of self or doesn't take class or self in its parameter list but when you call it you are definitely passing in that self even though you don't see it you're calling it the same way as you call an instance method in many cases but somehow when it when the actual function gets called that date has been manipulated or removed so again this is though it's changing the attributes kind of creating a wrapper around that attribute the the method that when you call it does some stuff before actually calling the underlying function so now I've talked about well why we've got descriptors I'll just talk about how we do it so a descriptor is an object with binding behavior one whose attribute access has been overridden so this is the specific frame there's a way to override attribute access so we've seen a way to do it globally per class you should get at ER and get attribute but this is specifically per item so it's per attribute if the attribute defines dunder get dunder set or delete dunder delete and then it's defined as a descriptor it was its it is a descriptor and so Python will add will do different things when you access this on a class or an instance so here I've defined a descriptor it defines dunder get so it is a descriptor according to the previous definition and now I'm just defining a class just an ordinary class and I'm assigning a instance of that descriptor to the get me attribute and then here I'm creating an instance of the class and then I just access get me and what you'll see is that that string that's returned was generated by the dunder get method so we haven't just accessed something on the dict what we've done is we've called a method behind the scenes and this is how the rule works so basically it go it's it's where you'd kind of expect in the hierarchy so where we're traversing for data so basically if we find the attribute in the class definition or one of the super classes if it defines done to get then python won't return you that that value it will call them to get and return you what's returned from that so in many ways it's like a property if you think this is actually the underlying mechanism that property uses to define itself and so it wraps a function and calls the function returns you the data instead who knew this already I've got a few hands who knew that there are two different types of descriptors and they behave completely differently hand-up from the person who's also giving a meta class talking next few months so what we've just defined as a non-data descriptor if you define one of the mutation functions so set or delete then you have a data descriptor the terminology is not that important but the idea is now you have an attribute that you might want to you're overriding the sort of mutation behavior as well as just the accessing behavior and so let's define a data descriptor so here we've got another it's the same I've added a set method to it it doesn't go in the same place in this in this order of operations it goes just after done to get attribute so it's just before the instance value so this is actually accessed before looking up anything up on self and the question you might be asking yourself is why is this so complicated and the answer is relatively straightforward its methods so I've been very specific about the way I've been describing executable attributes defined on a class I've tried to be very specific anyway so we've been finding functions on the class and you'll see from the first line here that if we refer to drive on the class we get a function that's what I've been call it that's why I've been calling them functions but if we refer to it through the instance of car my car then we get a bound method and you can see there's actually two bits of data in that string it says it's car dot drive so that's the function and it's also bound to the car instance so what's essentially happened is you've passed it enough data that it's handed you back a wrapper object so a method is a wrapper object around a function that take that it doesn't require self self was bound to that object or just as you accessed it so if we look back at the original definition of get here it takes the instance and the owner and it's instantiating a method that contains the instance so that when you then call that you no longer need to pass in the instance that's where self comes from when you call a method if anybody's ever been confused by that this is this is where self goes it gets kind of stored away in the actual method you returned so that when you call it you don't need to pass it in so attribute lookup in Python is actually quite complicated you can see that it switches backwards and forwards between checking in the class hierarchy and checking the instance it's not a simple matter of looking at the instance and then falling back to this hierarchy of classes so I don't know about you but this is what goes in my head when I think about instances and classes it's like instances sit below classes somehow and then you kind of go up I mean you could flip it the other way around it doesn't really matter which direction this goes in but I always think of it as linear but this is not an ideal way of thinking about it especially as we move on to the next section the way I would like you to start thinking about this is as as two dimensions so we have the classes which are in a hierarchy we have instances off to the side there actually they actually sit slightly apart from the class hierarchy and this will be more useful as we move on through the next section of the sort so that's the end of the classes section of the torts anybody have any questions okay I've either lost all of you or I'm keeping your lung yes okay so actually it's not a tight sorry I'm probably mixing up the terminology that I'm using so their first well that's actually a class I'm sorry yes so the question was before I was talking about functions having a class of function whereas now I'm talking about the type of function as being a function is that right getting a very careful nod there so yeah no actually I'm not calling type on this so although I'm printing out car lot drive and it's just kind of saying yeah this is a function if I was to call type on car dot drive I would get the class function does that make sense yeah any more questions okay both metaclasses so again I would like to talk about why we define meta classes before I start talking about them because again they're added complexities why would you add extra complexity to a language so it's useful to know that why are we adding extra complexity to the language what does it allow us to do so there's a handful of things so there's probably more but these are things I've seen in the wild in real metaclasses used in real frameworks so we can register on definition so we've got some sort of central registry of a specific type of class so the metaclass just kind of means that every time you instantiate one of these classes it says to some central authority hey there's a new instance here you might be interested in Django uses this for registering models so then when you can run the Django command to say like create me a database and it will go through each of your model definitions and create a table for that model so this is useful we can initialize attributes so this is a common pattern for meta classes where it just looped through all the attributes that have been defined on the class and just tells each attribute what its name is because if you think about it when you're calling the function will go the class to define a thing it doesn't usually know its own name and it would be duplication if you kind of said ie D equals new ID field I am the same ID it's just oh sorry the ID string so yeah it's you have to do this kind of thing with named tuple for example inside classes you don't have to because you can use a meta class to set this for you so we can modify a class based on its definition we can add things we can remove things again you could you could go through the attributes and form an extra attribute that just contains a list of special types of attribute inside your class to make it easier to use you can ensure that a subclass is implemented correctly so after you've actually instantiated the class you can just go through it and say oh yeah they've defined all the things they needed to that could be useful too I haven't seen that used as much as I think it could be I think it could be really useful sometimes it can be tricky to kind of implement a subclass correctly and I think metaclasses could be a useful way of kind of ensuring that things are being implemented fully and you can totally mess with way a class behaves and I'm going to show you some of that magic too so before we start talking about how we really define meta classes I want to talk about type a bit more because we talked about the type function then we've talked about the type class and they're the same thing and we've seen the first version here already where we call we pass type an object a class or an instance and it tells you the type of that thing so if it's a value then it tells us the class and if it's a class and it tells us type essentially the class of the class but there's a second form which I've got up on the screen here the second one is it would create a new type it is a factory for new classes in Python so you call it with three values the first one is a string containing the name of the class that you want to create the second one is a list or tuple of the direct parents of the class so much as you this is basically equivalent to a class declaration if you think about it and then the third thing is the class dict which is a dictionary of the things you want inside the class so you can add functions you can add class data anything you want in there and it will essentially this is this is this is what class is syntactic sugar for so to give you an example this is equivalent to our car class that we had earlier but using type to do it dynamically so I've created an init phone call it dunder init because that would be a bit weird at the top level if I should start at the bottom really so the block at the bottom is creating this this car type it's passed three parameters as I said name of the class the direct parent is object we could pass it an empty to pool cause that just gets added in automatically and I've passed it in a dict with an init function and a drive function and obviously they don't I haven't defined the init function with the same name but it doesn't really matter it's the way it's looked up is on that that key not the value so this is identical to what we had before it's a lot uglier that there's a reason for having class syntax in Python but it is also really powerful and I have seen it used in the wild I'll show you an example in a bit so just imagine we had three mixin classes they're not related to each other none of these inherit from each other but each one provides behavior so in this case I've got a defines an a method be defined to be method etc imagine you needed for some reason to create combinations of each of these every possible combination of two of these so an a B and a c.b.c imagine you had six of them or twelve of them you need to create all those classes that would be horrendous code you would almost certainly if you really had to do I mean first you would say why why am i doing this thing so then you would probably write yourself some some Python code to generate Python code and then you would paste that into your project to be horrendous but Python is a dynamic language that we can do this dynamically and so we can do it in three lines of code after we've defined those classes you then use it a tools to go through all the the pairs of curd the combination pairs of those classes we can create a name dynamically out of those class names and then we can assign it to the global dictionary so the current modules dictionary and this is all a bit deep magic but hey you came to a class on metaclasses so what did you expect so that means that then I there's a class called a B suddenly in my namespace and I can look at the basis and I can see it inherits from a and B I can instantiate it and create a variable of that type I can call a on it it does what I expect I can call B on it and it does what it expects and I can call C on it which will fail because it doesn't inherit from C and so it doesn't have a C method and this is actually crazy powerful I can see some really interesting use cases that vaguely out there further creating matrices of classes out of because because inheritance is kind of limited in some ways and it's in its kind of pure form how you define it as being able to create a set of classes out of a matrix could could be really interesting so back to what is a meta class and now we've talked about why metaclasses a little bit it's a callable that returns a class and it literally is so meta class doesn't have to be a class yeah eva consistency it can be a function so here I got as I said stupid meta class in the in it's in the name and this just defers to the type function and returns the result but I can use this in the class definition I can say this is my meta class here's a function and that will get called to instantiate the resulting class the only problem is that when I this is now no longer the class of a class so stupid matically really that last line there should say class stupid meta class because that was what I defined in my class definition but because actually type was used to create this class I've got the wrong metaclass there so I've used it to create a class instance I could have done some weird stuff in that function it could be useful but fundamentally it's it's not related anymore like it's fire-and-forget way of creating a class so going like people say this jokingly like a meta class is the class of a class simple we all understand metaclasses now but it is genuinely useful if we've been talking a lot about inheritance and how classes relate to instances and if we go one level beyond and think of our class as being an instance of a meta class and just kind of blur our eyes a little bit it's the same pattern if we understand how instances relate to classes we can also understand how a meta classes relate to classes and I'll get into some more detail on that in a moment and but this is this is a really like again it's a stupid example but this is this is a sort of base implemented of a meta class so we have a class that extends type because its type is analogous to object so object is two classes as typist of meta classes and then we can use a new mine meta meta class in a class definition and then we can we can actually instantiate that class and this and I unfortunately I've really messed up the slide because in the previous slide I told you what the type of my class was and it's I saw told you it was wrong but in this light if we did type on my class what we would get back is my matter so they're actually related so as I said it's a class of a class so if we were to look at my class and doc Thunder class it would specifically point to my matter it's I'll show you a diagram that I think illustrates it at a moment but before we get to that I will talk in slightly more detail about how we defined a meta class so much like defining classes we use a class definition because it is just the class of a class where we inherit from type as I said before now if there's this slight complexity in the way that meta classes are initialized they're not quite as simple to define as classes so there's a three-step process so before new is called so still new and in it and I'll go into those in a minute but before those are called have done to prepare is called on our meta class if we've provided it and the only job that this has is to return a dictionary that will then be used to contain everything in the metaclass sorry to contain everything in the class this is our factory for creating method creating classes remember the reason for this is simply so you can pass back a special implementation of dict so if you want it if for example the order that your attributes were defined was important you could pass back a dict that as well as storing away the things that are added to it also recorded the order that they were added then that would then it allows for more expressiveness in what your meta class can do when creating the ultimate that the class ultimately normally you will ignore this unless as I said you care about the order that things are defined on your class then we have new so new is analogous to new on a on a normal class and but it takes more parameters so this is similar to what we just saw when we call type we've got name basis and class ticked and obviously it's called it's passed an instance of this meta class as well and then we can pass in optional keyword arguments as well I will get into where they come from in a moment so again this is the job of this is to create the class so usually this will call super new which is equivalent to calling type essentially and then but it might do some stuff to the class dict beforehand because the class dict gets passed in so you might then filter out things you don't want in your class or you might add things in that you want in your class essentially it's manipulating the input data to the class or when the class is returns you can then manipulate it in similar ways so that's what in it is for and it is for post-processing of your class if you don't really care how it's created but you want to remove a couple of things and this is where you would do it keyword arguments are interesting and I think that underused in when people do write metaclasses so meta class is normally don't take any parameters and they just do the same thing to any class that they're asked to create but in theory we could have we can pass parameters through so I'm gonna start at the bottom here you can see I've set the meta class and then I've set this magic private keyword argument and private is not special I could call that foo if I wanted to and all that happens is then that is passed through to new and in it when the class is constructed and it's up to the metaclass to decide what to do with that input data it's just like a parameter to a function right so means we can have a meta class for in this case maybe the metaclass strips out all the dunder attributes before creating the class and if you pass private equals true or maybe it doesn't if you don't pass probably equals true no the the you have many options here this is just another powerful feature of meta classes so I'm going to update the diagram that we were looking at before not the big complex one but the one where I was talking about how instances and classes relate to each other so here we have our classes and I've purposely colored these blue so they look a bit like blueprints so the classes of blueprints people think about classes that way but it gets more complex when you start thinking about meta classes maybe metaclasses just a pen and paper or something like that some sort of pre blueprint I'm not really sure but here we can see how type relates to object and car now this is in the case where I haven't defined a custom meta class for car so they both they are both instances of type type is the class of object and the class of car and then my car it's very car is then they class of my car if we define a custom meta class for car then it starts to look like this and you can see why we need this extra dimension right we've got two different class hierarchies here we've got the class hierarchy and then we've got the meta class hierarchy with type at the top so now let's talk about class attribute lookup because we would talk about instance attribute lookup before and I just want to go back to the big complex diagram but talk about how it works when you look up things directly on a class rather than going through an instance so this is basically the same diagram as before except everywhere it had class before it now says meta class and everywhere it said instance it now says class so we're just coming into that that diagram at a different point right we're just coming in one one level to the left and it works more or less the same with one caveat so where descripted so descriptors still work on the class there's a when you access access a descriptor on a class it still calls done to get for example but it passes in none as the instance so in this case when we act where as we did we never call descriptors on an instance they do get called on a class when you're doing class access for a descriptor it's just one kind of just one extra step is it's basically the only place where it's not consistent between like moving from looking things up on an instance and classes and looking things up on a class and meta classes but it's really important to note here that we are looking stuff up on the meta classes if you look meta classes are part of the inheritance chain so if you define a method or sorry a function on a meta class then you you can end up calling that if you you can cut you can access it via the class so it goes it looks at the it looks at the class which is our instance and it also looks at the meta class which is our class so it goes back to our definition of meta classes are the class of a class it's a horrible description but it is helpful I honestly I've been using meta classes for nearly a decade and I only realize that this year that I can actually call stuff on the metaclass they threw the class almost by accident so yeah you're actually affecting class behavior like just just how what you can do with a class by defining things on the meta class so if any of you are like me and and slightly evil what you're thinking to yourself is well when does this diagram stop it's like if if instances have classes and classes have classes what the meta classes have do we have meta meta classes and the answer is yes kind of so we have the type of a classes type a type of type is type and the type of a type of type is also type so that last diagram we were looking at actually looks like this I apologize that there's some overlaps there so you'll see type is an instance of itself and car meta is both both inherits from type and is also an instance of type which I know this really doesn't help with is the idea that I was supposed to be making this simple you can mostly forget this but it's also worth noting that you can kind of override some of these relationships if you want to if you want to change the way that car meta is created you can create a meta class for it I believe it will work I haven't tried it it there's only so far so much my mind will take but yet but fortunately because because of this weird magical relationship that type has to itself this diagram doesn't naturally go off to the left as far as you want it's not there's not a sort of meta type or anything like that because they've decided to kind of encapsulate it in this one type but it doesn't help with the confusion when you start talking about things being instances of themselves and inheriting from themselves and things like that it's it's a bit of a mess it also doesn't help that type plus two completely different behaviors right I meant to talk about this earlier as they really shouldn't have two completely different behaviors I I personally think that was a design mistake when they implemented new style classes in Python too but I I haven't had any discussions with anybody involved in those decisions so it's entirely possible and probable that I'm wrong so let's go back to MRO there's a method resolution order so this is what we were talking about before with the inspect function get MRO returns you a list of the order the sequence of classes that will be checked for the existence of an attribute one of the sort of grab bag of features that has been rolled into meta classes is that you can override this if you for some reason wanted a different resolution order I don't like the facts called method resolution order because fundamentally it's how you access attributes on a class hierarchy but if you wanted to change it here is an example of a meta class when applied to a class hierarchy it changes the inheritance order into an alphabetical inheritance order so it will call the class in your hierarchy with the name that's first in the alphabet first and then it will look to the next one second so it doesn't really matter your parent classes are now the way this works is that when the class is created this mro will be called and the result will be kind of stored in an attribute called under mro which is read-only so you can't mess with this at runtime basically once you created your class it has a even if you've given it a ridiculous order of method resolution and that's fixed and you can't mess with it but it's recalculated every time you update basis dunder basis on your class is mutable and if you update it even to set it to the same value that it already has it will then call mro again and create a new dunder mro value so you could actually have a function that every time you call it changes the the hierarchy of your class so that the next time you call it a calls a different version somewhere else in the class hierarchy and if any of you find a practical useful reason for doing this please tweet at me I desperately want to find a use for this so a common pattern that you will see in practice going back to some more practical stuff is that meta classes and it will usually have a base class associated with them people don't like setting metaclass equals it so it's why you tend not to do that in daily coding and so what happens instead is that you end up with a class called model so this is from Jango and what it doesn't really provide that much behavior you get the behavior from the meta class which is called model base so model has a meta class model base and by inheriting from model you get the same meta class so so that you get that behavior the same with ABC ABC we're going to cover this in a second but ABC is a core module in the Python library that provides us ABC class ABC claw the ABC class has a meta class of ABC meta and so you get that for free when you when you extend it so and this is generally how you kind of hide your magic from your user if you're going to be implementing meta classes you want to as much as possible hide the magic from the user you don't want to worry about the nitty-gritty or the fat that is a meta class you just explain in the documentation hey if you extend from model you will get this behavior so it all comes down to documenting the magic and trying actually to limit the magic as much as possible and again we'll get to that in a second so let's talk about metaclasses in the real world for a moment because I think that there's a general guidance around that hey if you don't know if you need metaclasses you probably don't not entirely certain that's true and all the other the other bit of wisdom that goes around is just never use metaclasses they're a bad idea again like both of these are partly true but I I'm standing here giving a talk to a roomful of people on how to use metaclasses so we were talking about the uses for metaclasses let's see a few of these in action so the first I'm going to concentrate on the ABC library in there comes with Python stands for abstract base class it's a way of defining classes that should never be instantiated they should be extended and then once you've defined all the abstract features in a subclass then that can be instantiated and so this is an example of a meta class that ensures that you have implemented a subclass correctly so here I've got a class called driveable which extends ABC as I said before ABC has a meta class of ABC meta and that works in combination with this decorator called abstract method so I've used that I've just declared drive as an abstract method which means that basically it's it's a placeholder right what I'm saying is anything that extends drivable has to implement drive if you want to create one so just below I've implemented I've extended drivable but I haven't implemented drive which means that when I then try to instantiate it I get it's a lovely error from the metaclass saying hey you haven't implemented the abstract method drive so this is still abstract so if I reimplemented at and actually implement drive in the drivable subclass then I can instantiate it and then I can call drive and it prints out driving and everybody's happy so and this is kind of useful and I have to admit I'd kind of forgotten this feature existed and I certainly didn't know how it worked I'm kind of pleased to see a practical use of meta classes in the core library moving a little bit further away from home I'm going to talk about Django for a moment Django uses metaclasses pretty extensively there are seven different meta classes in under the Django package in one place or another there's also at least one more in the testing framework that comes with it the one that people use most and I guess I'm most aware of is the model model class and the meta class that comes along with that so I'm gonna just roughly sketch out with some code examples what that does so for people who haven't worked with Django recently and this is the way it works you declare a class that really represents a table in the database you extend models model and you get lots of magic behavior the magic behavior works with fields so these field attributes that are defined on the class are actually descriptors so we've got one here called food name which is a character field we've got one called rating which is an integer field it's a rating food in case anybody hadn't works out what that was for and then what happens behind the scenes is that inside the metaclass after the class is created it goes through all of the fields well actually first it registers the model with a central database so it says hey we've got a new table here when they want to create the table when they want to run migrations this table exists so that's one thing that's done in the metaclass oh sorry oh yes and I've extracted out bits of model base here model basis the metaclass that we're talking about so also in you I've ripped up huge amounts of code there are over 200 lines of code just in that new function in model base one of the things it does is it goes through all the fields and it tells them their name so as I said before this is a really common use case so it just kind of says hey your name is rating and your name is food name so then when they come to deserialize data or serialize data they know what they're working with they know hey this is the type data that was passed to me in this constructor this is the name that was passed to me that during metaclass execution and then in model which has a metal cast meta class of model base it's adding some stuff to the class remember that is no idea where that is excuse me so I'm going to move a bit we are starting to run out of times I'm kind of skimming a little bit the other thing it does it manipulates the the resulting class by adding two classes to it so adds a does not exist class to the model class and it also adds a multiple objects return class to the model class and so these are specific for that table so if you've got a high-level exception capture for does not exist then you can specifically say but is it my model dot does not exist and then you know which table is missing the data at that point so it gives you these fine-grained exceptions that are dynamically created and they're actually created using our type class that I was talked about before this is another case of programmatically like looping through and using type to create classes at runtime what while we're talking about Django and metaclasses one of the things I want to really highlight here is that there's this convention I think is just because Jan goes all right but they they have this this like this concept of creating a class inside your model class and calling it meta and it does provide meta information about your class so how it should be ordered how it should be represented in different parts of the application and it is used by the metaclass on class construction but this itself is not a meta class it's just a class inside your other class and it just happens to be called meta which is not helpful when you're coming when you were talking about both these concepts at the same time and you should see the number of Stack Overflow questions that are confused about this so having seen a couple of meta classes and knowing how complex all this stuff is like inheritance generally in Python but also inheritance when it comes to class lookups and things like that let's talk about things that may be better than meta classes in some cases so have one update to the information I've been talking about before which is the descriptor protocol no longer just consists of dunder get dunder set and under delete it also now consists of set named thunder set name this if this is defined on your attribute then after class creation pythons default meta class we'll just call it with the instant with the it's been defined on and the name of the class so this removes from my experience about 50% of the need for meta classes in the wild it's like you just you no longer need a custom meta class you just define this on your field type or whatever it is and and suddenly you know your own name and yet you don't need to worry about that anymore then we have class decorators so we've probably used function decorators before I forget the version of Python but they also work with classes now function decorator is a sorry a class decorator is a function that takes a class and returns a class usually it will modify the class that's passed in and then return that same class in theory it can return whatever you like so I create a new class based on the class that's passed in and return that these only refer these only affect the class that they're applied to they don't affect don't affect subclasses so whereas with meta classes you they were inherited which I maybe didn't point out enough with class decorators they're not so you need to need to use them per class which could be a benefit right it's much more explicit and we like explicit in Python so I would like to talk about that with these two features together I would like to talk about how you might reimplementation or something like it using these two features so here I've got what I want to be able to implement so we're using a model decorator rather than inheriting from model and then I have to find my own fields that they just take a sequel parameter I'm really simplifying this example and what I want to be able to do is create a table from this so I want to generate sequel just based on this class definition and so by defining fields with the set name function I can I can get each field to just know its own name which is handy for the the sequel generation the decorator implementation is a little bit more complicated only because what I want to do is add this create method and this fields method to the resulting class so I'm essentially I'm modifying this class to add some functions to it so the the the actual decorator is this this block at the bottom that the actual model decorator takes the class and adds a create method and a fields method and then those are just functions that are defined above they look like methods or they look like class methods but they're defined as functions at the top level and then when you use it as I said this is what I wanted to end up with and what you get is this modified ranked food class with a new create method that I can then call below depending on the IDE you use this can be this can be awkward because this is no longer been defined using standard Python syntax so anything that's using doing static analysis of your code will not know that there's a create method on this class some of the more advanced ideas like IntelliJ do dynamic analysis of code that's executed sorry not IntelliJ Python to do analysis of code that's executed through it and they know what attributes are defined on classes sort of after they're realized and so they can often provide better sort of intelligence for when you're writing your code but there is a you know that I just wanted to point out there is a downside to every time you use this kind of magic you're no longer necessarily working with the systems that that help you edit code another option that may be better than metaclasses is code generation which sounds horrendous but you're almost certainly using it it this is a technique used in maine tuple ATA's data classes now so two of those are in the core library essentially the technique is you build a string containing the Python code you wish you had and then you run it through exec to create the class that you want it can be surprisingly intuitive especially if you hide it away so with it nobody knows that's what you're doing but ultimately what you end up with is just a class right you don't have a magic meta class behind the scenes it's just a class and you can do whatever you want to with it and that's one of the features that they really push with adders it's like you use these decorators to essentially create the class that you won't create a whole load of boilerplate code but once you've done that you've just got a class you can add methods to it the same way as you normally would and things like that and they're unaffected and so that can be pretty cool so just to wrap up I can't believe how close this came to time in conclusion inheritance is complicated probably more complicated than you thought metaclasses are really not that simple but they're also not that complicated once you understand how inheritance works descriptors are definitely more complicated than they appear and because of this complexity in all these areas that work together you should use all of these features that I've described in this talk sparingly or for fun so as I said like these features don't work well with code editors they also don't work well with our brains necessarily you know they are extra they work in different ways to other Python code that we might write and so there's a danger there that you're going to build something that works in such magic ways that when it doesn't do what the user wants they have no way of understanding why but the last point that I really the reason I wrote this talk in the first place because I really want people not to be afraid of this magic like Django has really shown how useful some of these techniques are just sometimes just a sprinkling of these kind of deeper magic techniques can make for a more intuitive framework or a more intuitive library so thank you very much I have some time for questions I think of three minutes [Applause] yeah feel free I'll repeat but can I ask one - one question at a time because I've got to repeat that anyway so the question was because metaclasses are inherited by subclasses what happens if you've got two metaclasses essentially that one's one class that inherits from another they have different meta classes so it's the same thing as you actually just seen basically which is that the lower meta class wins it's it's the most recent meta class in the hierarchy there are some rules about how metaclasses must inherit from each other there has to be a linear inheritance but I don't think I can explain it verbally but I can point you at a blog post that does a fantastic it does a fantastic job of explaining how that works so the second question so the question was have I ever found an example of metaclasses kind of extending from metaclasses extended from meta classes and it's a definitive no and so I think most developers are sensible enough to kind of minimize localize the magic as much as possible I think one more question anybody yeah yes I totally meant to have a slide in for that so that was a question about whether in its subclass removes the need for again more need for meta classes and that's absolutely true though these are relatively recent additions to the language that are taking use cases for meta classes and removing them so it's like more and more meta classes are becoming that's something you probably shouldn't do there's probably a better and more explicit way of doing what you want the answer your question thank you very much mark you

Channel: PyCon AU

Views: 5,778

Rating: 4.9285712 out of 5

Keywords: pyconau, pyconau2019, Python, PyCon, PyConAU, MarkSmith

Id: ZpV3tel0xtQ

Length: 69min 54sec (4194 seconds)

Published: Sat Aug 03 2019