Raymond Hettinger «Build powerful, new data structures with Python's abstract base classes»

[Music] thank you everyone for inviting me here we were here three years ago loved it here and it's a delight to be back and I'll see you all how many of you are ready for a technical talk to kick off the conference I mean a technical call with code and I'll give you all the code working code some of which you could start to use right away it's that better okay good now how many of you have already used pythons abstract base classes how many of you use the ones in the collection module that doesn't compute at all so that is one of the most important sets of abstract base classes and so since you're mostly familiar with them I'll do the beginning part of the talk where I go over the mechanics and do it somewhat quickly my understanding is that there will be a test afterwards and that they're giving a question out and I was supposed to supply an answer so make sure you pay attention to every detail fair enough are you ready to rock and roll all right here we go hi my name is Raymond Hettinger that's this part right here if you want to tweet right now and say that this is an enjoyable talk this is a place to do it if you want to watch some of my training videos they're mostly free to you online through Safari online there they are that's my company and here's what we're doing abstract base classes what are they all about so a B C stands for abstract base class and I'm guessing you already knew that what most people have a hard time articulating is what exactly are they for it is a form of communication between subclasses and parent classes why do we subclass the answers subclasses let us reuse the code from another class so they have to talk to each other so why do we need to abstract base classes we have three purposes the parent can communicate to a subclass here's the structure I demand of you secondly a class can identify itself as hey I meet those requirements I can work with this parent and there's an enforcement mechanism built inside to check and make sure that we meet those requirements so we'll start with a example ABC okay and this one is an entertainer class in order to build an ABC you import from the ABC module and inherit from ABC now this looks like regular inheritance but underneath the hood there's some meta class magic going on and the abstract method is a decorator to mark the abstract methods so to be an entertainer you need to be able to do two things you need to be able to sing and dance these are not concrete implementations they're abstract implementations which means that we've defined the code is passed it doesn't actually do anything so this essentially specifies what are the requirements of being an entertainer but it doesn't tell you how to be an entertainer I've got some instructions on the bottom for how to build an abstract base class but roughly it's this import this impair here from the ABC and take the abstract methods and mark them as such one of the nice things we can do with an existing ABC is ask it what are your abstract methods this is important if somebody else is giving you the ABC class and you need to know what is it I need to implement sir I'd like to become an entertainer what do I need to do well you need to be able to sing and dance so how do we identify a class as implementing the entertainer ABC one way is through registration so this class is a street performer it's a concrete class it actually knows how to do things and it actually knows how to sing and how to dance in order to mark it as an as fulfilling the requirements of Entertainer you can register it this has a really interesting effect in every other language I know is instance in its subclass checks to see if you're an instance or if you're a subclass but ours does something special it will also check to see if you're registered as a member of this class so if you claim to be an entertainer there's two ways to do it you can actually be an entertainer are you claimed to be an entertainer okay and here's how we make the claim this we will in a register register ourselves as a street performer The effect of this is if you make an instance of street performer it is now recognized as an entertainer even though it didn't subclass from that everybody got registration how do you make yourself an entertainer register yourself as such there's another way you can subclass entertainer this will also inform isinstance about the relationship and so if we want to know if the lounge performer is an entertainer it returns true in this case is instance is doing exactly what you expect it's identifying a member of a subclass the interesting contrast between the two ways is that subclasses have an enforcement mechanism and registration doesn't so here's a wannabe performer sure the wannabe performer knows how to sing but doesn't know how to dance it does not it's not a real entertainer so if we make an instance of wannabe performer it refuses to instantiate it says you can't be a performer because you don't know how to dance this is fantastic which raises the question what happens if we try the registration way instead of the sub-classing way so here's the wannabe performer not subclassing from entertainer it registers itself and then we go to make an instance this time it does make the instance not a good instance a broken instance we ask if it's an entertainer and it says that it actually is is it really an entertainer it doesn't know how to dance okay there you go just checking to make sure that you're all they dialed in remember there's a test afterwards so if we ask it to sing in fact it can sing an amateur version of the national anthem if we ask it to dance it can't dance even though it's registered as an entertainer in other words you get to do something very interesting with registration you can lie to is instance you don't actually have to be an entertainer to be registered so without sub-classing there's several differences the registration always succeeds even if you're not a real entertainer the instantiation succeeds always even if you're not a real entertainer and his instance will claim that you're an entertainer even if you're not and at runtime the method fails so there is some danger in the registration method you actually really want to make sure that you implement all the methods because it has no way to check for you who learned something new all right fantastic so here's some summary and comments why did we do abstract base classes at all is to formalize a relationship between the parent and the subclass everything that we've done so far could have been done without abstract base classes we wouldn't be able to do in his instance check later and we wouldn't be able to verify that all the proper methods are there so think of it as a way of formalizing the relationship you've learned how to do it which is inherit from ABC and mark the subclass contract with abstract method how do you make it recognizable by instance in subclass there are two ways you can subclass the ABC or you can register it the advantage of sub classing is instantiation is blocked if one of the methods is missing if you don't know how to dance you're not a real entertainer in other words the enforcement mechanism works you've also learned for registration there's no error checking and in fact you can lie to it the alternative to lying is you make a mistake and it doesn't point out the mistake and you've learned to introspect ABCs you can ask it what abstract methods do you require what is needed to become an entertainer singer and dancing and you've learned how they're implemented ABC is implemented with meta classes and abstract method with decorators here's an interesting a fact that is a widespread point of confusion ABC's are not recognizers people see this is instance and say is this lounge performer an actual entertainer and they think that more verification is going on than actually takes place and all it can do is check and see is something as a subclass or if it's uh is registered but if something's not a subclass and not registered even if it knows how to sing and dance it might not show up as an entertainer and so people are continually disappointed about this they make bug reports all the time and said I made an entertainer by a class that could sing and could dance and yet it didn't pass the isinstance check there are two ways to do it you can register it or subclass it but it cannot recognize an entertainer without having seen it first our ABCs recognizers no you have to register our subclass and one of the interesting things about the limitations of ABCs is they just check to see if the required methods are present it doesn't check their signature it doesn't check to make sure the methods work or that they do the right thing in other words it's a very limited form of checking congratulations if you learned everything in the summary points you are now masters of abstract base classes and I can consider myself done and goodbye I've got time left ok here we go strategies for using abstract base classes there are three ways to use them you can make a pure abstract class and this is what will be familiar to you if you come from a language like Java or C++ in those languages abstract base classes are fully abstract all the time and they do nothing else for you other than specify a formal relationship and so you don't get any extra firepower from that however in Python we use them in two other ways as well one of them is called the framework design pattern and the other is mixing classes and whatever the message I came to bring you today is we've got some mixing classes for you you don't have to write them yourself they're very powerful and they do a lot of work for you are these good characteristics things that are powerful do a lot of work for you are already tested and take very little work on your part so a pure abstract class is something like our entertainer class it's pure in the sense that there's only abstract methods and no other services so this is the kind of thing that you get in Java in other words we have a definition of what an entertainer is but we don't have an actual entertainer the more interesting one is the framework design pattern by the way I've got these sections at the end called extras they are working code for these frameworks that I'm about to show you the code that I've written myself but there are ready to use and they're pretty darn neat and they're fun to play with and they have practical uses as well so in a framework pattern that parent is smart it has all it manages the execution flow and the subclass is like a worker who reports to a boss that has some specific capabilities but the boss is in the parent class the parents rule the children do what they're supposed to and you put your children to work it's a very simple arrangement how do you like that arrangement those of you who are parents like it those are viewer children not so much all right so the parent class in this case is puzzle and the child class is stepper so what is a stepper have to do it defines a position it decide and defines a goal state in an iterator that shows possible moves that are forgetting they're an iterator takes the current state and generates a new state now this little bit right here what it does is shifts the 1:1 space to the left what this does is shift the code shift one space to the right so the one can move left and right this is a very simple puzzle if you all you can do is left and right how do you move the one over to the right and the answer is a series of our rightward moves the interesting thing is how we solve the puzzle the child defines the current state the goal state impossible moves the parent figures out how do we get to how do we figure out the solution so we call a solve method do you see a solved method in the stepper puzzle no where does it come from the parent which is smart this is called the framework design pattern in the framework design pattern parents are smart and children provide the concrete methods and they are the slaves the family as it ought to be simple thing there's another way the by the way this code for puzzle I put a simple toy puzzle here let me skip ahead to my puzzle framework it's got instructions for how to use it including canonical ization and symmetry arrangements this is the entire puzzle class including the smart parents solve method down below though I have lots of interesting puzzles a jug filling puzzle with nice cute chips that go along with it eight Queens puzzle triangular tea puzzle there's a lot of puzzles in here including some puzzles that are hard this is the first one that's hard I have never seen a human being ever able to solve this puzzle manually it's a famous puzzle in my family was at my grandfather's house so I called it paw puzzle and I saw many engineers come by and visit our house pick up the puzzle play with the puzzle for hours and not be able to solve it I know of only one person on this planet who was ever able to solve this puzzle who is it that was me how did I do it I wrote a program to do it I wrote it on a trs-80 in basic a long time ago but it is basically this program in Python and once I saw the solution I figured out how difficult it was what happens is the solution space gets very wide then very narrow and it's all a big circle but there's an exit door and if you recognize it you can go to another level another group that fans out and backs in and there are two thin little doors on the way to the solution and if you don't find the door you stay trapped in a space that has no solutions in it and you can play with it for hours so this is a somewhat hard puzzle but you get the idea is how do you make a puzzle solver you define the puzzle with an starting state an iterator that generates possible moves and some ability to recognize what a goal state is after that although the work is done by the parent this is called the framework design pattern so I've given you a framework that I wrote now mixing classes are somewhat a different in that case most of the intelligence is in the children they provide a almost complete working class however the parent provides extra capabilities that layer on top of what the children already know how to do so they augment your capabilities so here's an example a child class called calculator it does something very simple it stores a factor and then it multiplies a value by a factor it is a complete standalone class that would work fine by itself but we inherit from a mix in class the mix in class says we have an abstract method count Cal in what show does is it doesn't actually touch the underlying factor or data what show does instead is it calls that calc function so the characteristic of mixing classes is they depend on the child implementing some particular methods and if they're present it adds a new capability so here's an example of how to use it I make a calculator that can double results I call a calculator I'm 5 and it doubles it in shows 10 can you see the output from this line no it's invisible which makes it hard to debug there's a better way the better way is to call the show method does calculator have show it does not so it inherits it from the visible class method invisible calls calc and then shows the inputs and outputs improving your ability to debug this is a simplistic example but there's an enormous amount of power here the idea is that once you've written visible you can use it on any class that has a calc function on it and so we could end up with a small library of mix-ins and these mix-ins could take your existing classes and you can add new capabilities simply by naming the capabilities that you want so we've heard of object oriented oriented programming once you've made the basic functionality you just say I'd like to take this class and make it visible and persistent self-documenting and add in some other capabilities and you simply list the ones you want and the mix ends do all the work for you so this is a very high payoff activity especially if somebody has written the mix in classes for you so framework design pattern is pretty cool you extend an existing class and provide the details of a method you essentially customize it but this is the one that I really like the mixin methods they're wonderful because if you have a library of them you can get an enormous amount of work done simply by inheriting from those classes so let's put it together for something non-trivial the collections a PCS so this is a very rich group of mix-ins in the ABC module why do we like them it takes very little work on your part to implement a new collections class and then the mix in supplies lots and lots of other methods that are otherwise a pain to implement this is a huge time saver but more than a time saver these methods that you're bringing in have already been tested so when you add them you name the extra capabilities you want and you get a lot of firepower for free this is one of these tools that if you take the time to master it play with the code that I'm going to give you it'll amplify your capabilities enormous Lee you've heard of the myth of the 10x programmer it is in fact a bit of a myth of a 10x programmer that one person is always 10 times more productive than another on the other hand some programmers know about power tools like these things and they know how to just import the things they need and attach them rather than writing them from scratch and that gives you more than 10x I'll leverage also you get high reliability out of it so what is a sequence it's a generic old generic term for a resizable collection that's ordered indexed and sizable you saw before we can find out what the abstract methods are all you need to implement a sequence is a length and getitem all you need for a mutable sequence is length insert get item set item until item this is very little work on your part you want to make a sequence implement two methods what do you get for free from the mix in it writes all of these other methods for you and for a mutable sequence you automatically get all of these methods so you get about eleven of them for free which is fantastic likewise with sets what are sets they're like sets and mathematics and they support membership testing set to set operations Union intersection difference they're all about data analysis and we can see what their methods are only three of them are required to implement a set this is very little work and they automatically give you all of these other set methods for free that's where all of the interesting logic comes in and then mapping so mapping is a generic term for anything that's like a dictionary only three methods for a mapping and for a mutable mapping a total of five of them and then here's all the things that you get for free this is an enormous labor saver so here's an example a list based set regular sets are based on the hash tables and they require hashable objects but perhaps you don't want that perhaps you have unhackable objects well we can change the underlying data store from a hash table to a list so the list based set has an init that says save all of the elements in a new empty list now how do you get the elements out of a set you iterate over the underlying list how can you tell if something is in the set you look to see if it's in the underlying list how can you tell how big the set is you ask how big the underlying list is the interesting part of this the the part that makes it all work is when we put the data in to begin with we make sure that that data is not already there interestingly not in you just calls this contains function so one of the neat things about building this is if you have an underlying data structure with some rich message you can already use those methods here we're taking the advantage of the fact that list already implements most of the methods we need and this is close to being a straight pass through so here's how we use it make a list bit base that this one and this one that calls the init it eliminates the duplicate elements inside and stores it as a list the interesting thing is we get for free this thing the ampersand the and operator and it tells you exactly what members of each list are corresponding list members so it tells you there's an overlap of the letters D en F this is four lines of code five six seven to implement a complete set who thinks that's kind of awesome seven lines of code complete set all of the set methods and all of the interesting methods are already tested for you the only thing I don't like about this example is it doesn't suggest the full power of how far you can go with this so I built a few for you of things that I have used in production code I've simplified them a little bit just so they can fit in a short presentation but let me show you some cool things that can be built very very quickly using these tools so the first one is a bit set so regular sets are based on hash tables and hash tables are sparse and they're kind of big and they don't store the data in the hash table we have references from the hash table to the underlying data and so it can take a lot of space so this is our alternative it only works on integers that are in a fixed range but it stores them as an array of bits so what is it super power of this set it's much more compact than a regular set and so it will scale to fairly enormous sets this computer has 16 gigabytes of RAM a byte being 8-bit so 16 times e is the number of bits that's the size of a set I can store in RAM on this without going swapping out to a disk so that's a pretty massive capability so what's necessary here well we set a limit on there our range how big of an integer can you put in here and we make an underlying byte array if you want to store 800 bits well we need a byte array of size 1,000 in order to do a thousand times eight gives you the 8,000 bits this operation here just rounds up and says if I only want to store 20 bits it still takes three bytes to store it because we need some for the left over so how do we locate a bit it's with simple bitwise operations I used if mod eight that tells me if I want bit number 1000 then that's going to be at a by 1000 divided by eight and then the modulo 8 tells me which bit number it is so for this we'll locate the particular bit you want so if we want to know if a bit is in the array we find the byte number and bit number we look up the byte shift into place the relevant bit and it with 1 and it tells us whether the bit is stored addition is also a little bit flippin it takes a mask for the bit number and ours it on to the appropriate byte and discard does exactly the same thing but it ends with the inverse of a mask so this is bit flipping operations something that everybody used to know before the year 2000 it made you a real programmer after the year 2000 it's mostly an unnecessary skill except for this particular context building an iterator over it is not difficult and getting the size i just loop over our iterator and count the number of bits that are 1 oh not all of these methods are required by the abstract base class it doesn't require an it but if you want data in there you're going to need an it represent to look at it but these other ones are the required methods if you implement those 5 you get the cent logic for free now here's the interesting wrinkle in this along the way when you do set operations you have to build new sets and the abstract base class doesn't know how to do that so we provide this underscore method called from iterable and its job is to take anything you could loop over and build a new instance of this class and the reason that's important is there's no way the abstract base class can know that we have to pass in a limit okay pouty lips show some all right bit set let me switch over real quick kept the bit set so we have the code over here on the right side I'll try and make the type as small as possible so you have no chance of seeing it okay and from bits WordStar make a instance of a bit set and what does it need to know it needs to know a limit my maximum number will be 100 and iterables I'll pass in some data 20 30 40 60 75 are members of the set I can immediately ask questions like is 20 in the set that is true I can make another bit set T that has the same range and we'll put in 20 40 75 and 93 and 97 favorite prime numbers okay and we can compute the Union intersection and difference between the sets keep in mind I didn't write any of these methods or did I get them all from where did I get them from although that work was done by the immutable set the underlying data store here is in the data and that is a byte array if we list it out you'll see the numbers that are stored in the byte array these don't look like our original numbers because each one of those is a bitmap for instance I can take 16 and show it in a binary are 64 in binary and you can see one of the bits is set in that particular byte the important thing here is the size of that data when that is a set of up to 100 elements only stored in 13 bytes this is a big win because the size of an empty set and regular Python is already several hundred bytes okay so that's a huge win twenty to one space savings and it takes very little effort so this code I have used in production more than once I'm giving it to you I hope you have fun with it and I hope I've convinced you of the power of abstract base classes Michael they applauded if I was smart I'd get off the stage now all right your filesystem is a dictionary how do I mean by that in a and a directory you have file names these are keys and the contents of a file is a value so a problem with regular dictionaries is they're in memory which means you can't inspect them while they're running and if you turn off your Python process they die they're not persistent and they can't be shared with other programs and you can't look at them as a text file while they're running and inspect a program while it's running and you can't change data in a running program in a dictionary this is terrible there is in fact a better way a file dictionary and I don't have time to do all of a file dictionary so what am I going to get used to get the work to the work for me mutable mapping all right so when I create a file like a directory or a file dick I pass in a directory name and it builds a directory for me what does getitem do it says take the directory name and the key put them together and it looks for that as a file name it opens the file and whatever data is inside is the value now we transform file not found errors into cures because that's what dictionaries do said item looks for that file and writes it and writing in a W mode instead of an a mode it'll overwrite an existing value just like in a regular dictionary delete item is fairly simple we can do an OS remove but if the files not there we're a zakir length is pretty easy we list the directory and however big the list is is the entire size of the dictionary and this iterator all it does is iterate over the file names and the prepper just shows you all the items which is interesting this is called self dot items did I write self dot items no where did I get it from mutable mapping this is a hundred percent of the code necessary to write a file dictionary from scratch go ahead and create one from file dict import file dict and over on the right show the source code for it h cat file dict that's the code that we just looked at which is why it's small the dictionary will be Starks of whom there are not as many anymore okay and the directory for it will be Starks and then I can add in Starks Arya tomboy Starks Shaun's a bad taste in men okay and Starks that keys Starks values keep in mind I didn't write these methods I got them for free Stark's lookup Arya which raises the question how did all of this work there is a sub-directory now called Starks that has Arya and Sansa in it and if you look at the contents of them it says that uh Arya is a tomboy let's edit that outside of Python I want to show that this is a something that we can change and instead of a tomboy let's put out ice Kingslayer then over on the left is a still running process look up Arya and you see that it changed we'll want to clean up line endings and whatnot but you can have dozens of processes running all sharing this directory which means that they're sharing a dictionary you can look at the dictionaries while it's running and modify it while it's running and if I kill the Python process it's still there presto a persistent dictionary that is shareable and concurrent it thinks that's awesome who thinks it's kind of a hack of a high up file system you demand a better way Raymond I need something more official how about the underlying data store is SQL so this is exactly the same code except this time our Annette creates a database name and it creates a table with key value store any unique index so a set item doesn't insert into the dictionary getitem does the Select a value where the key matches a dil item does a delete link is a select count star in an iterator is select all keys from the dictionary I don't think I need to demonstrate this one because it works almost exactly the same way it has some really nice benefits for doing it this way one of the big benefits is it's stored in one file so after you this dictionary you can email this dictionary to someone else they can use it modify it and email it back you can share dictionaries as easily as you can share spreadsheets who thinks that's kind of Awesome all right am I getting the message across that abstract base classes are great they are fantastic and particularly the collections ABCs there is something that you really want I'd like to wrap up with one last framework this is a data validation framework this part is a subject of another talk descriptor talk and the idea is anytime we store a variable for a validator we're going to store a private name for it anytime we look up the variable we look up the private name no magic in any time we set a variable we store it to the private name the part that special is before we store it we validate the variable and the validation method is an abstract method this one does nothing so you can't instantiate this validator class so how do you use it you subclass it and so one of is one that a validator if we put in some options the options are valid values are win lose or draw and you store these as a set what does the validate method do it takes a value and it checks to see if it's one of the valid values and it will preclude you from accidentally storing something into the database that is incorrect and so I built several validators for you here a number validator can make sure something is an integer or a float check for a minimum value and a maximum value string validator can check for a minimum size a maximum file size some predicates like is it all uppercase or title case and it validates all these for you the interesting thing is making new validators is quite easy subclass from validator the abstract base class will check and make sure that you put in a validate method and the whole thing is fairly easy to use do I have a couple minutes left this is good C D each cat validators and let's pull up a one-of validator from validators import star you should never import star what did I just do import star do as I say not as I do okay so class speaker person's name is a string and the minimum size on the string is three and a person's age integer and four that's called an integer class that we called it number and we'll pass in a constraint which is let's say max value is 50 does this look a little bit like data classes and a little bit like typing name couple it does this is a very old fashioned technique the first software I've seen that did it was called trait le'ts and then ORM sees a somewhat similar strategy and topic will be constrained to there's very few things that you can talk about here and one of them is you can talk about Python or you can talk about computing or you can talk about having fun no other topics are allowed okay so you've got a speaker and we go to say stay the store the name is I am 54 years old good or bad instantly detected it expected 54 to be a string so we'll put in the name is Raymond hitting jure Raymond uh age 154 years old oh no more than 50 is allowed okay I'm actually not allowed here at all okay put in my younger self 36 years old that is my actual age in hex okay and it's uh validated and Raymond the topic I am here today to talk to you about Ruby not allowed it must be Python competing are something fun in Python so these are data validators they're very easy to use I've given you my abstract base class for it all you need to do is subclass it and give it an an it and a validate method and it is ready to extend and put to work right away it is a very very powerful tool for preventing data corruption errors who learn something new Wow did you have a good time I did too thank you again for inviting me it's a honor to be able to speak to this audience any questions please hear raise your hands thank you for the talk please explain why a registration method allows to register class that doesn't law that doesn't implement necessary methods why it doesn't throw not implemented a roll on registration so let me summarize the question and the answer briefly the question answered because all right his question was why doesn't registration do validation there is a very shallow but highly accurate answer to this question why is Python the way it is because Quito's said so so abstract base classes are implemented in a way that made sense to cuido at the at the time and he had a wide variety of use cases in mind when he designed it and so we try to not design in the abstract we try and design to match use cases in a particular use cases he had it was sometimes useful to be able to lie to is instance for example you have some class that is checking to see if a sequence is given into it collections deck at the time has sequence like behavior its indexable but it was missing some sequence methods like remove at the time however if you knew that a particular function didn't use the remove method it was okay to lie to that function and say I'm giving you a Dec treat this like a a sequence so when you know that at one of the required methods isn't going to be used at all lying is sometimes helpful along the way so I believe that was his rationale I think he had some specific real-world use cases for it and in particular we knew that there were lots of existing classes that of course didn't inherit from this is existing to and we're going to have to register and in some cases they didn't have a particular method there's sequence like behavior but it's missing insert or it's missing count which is a method people don't often need but is required by the API so I would say it's use case driven it's because cuido said so because it made sense at the time and what validates the decision until now no one ever has complained about it in over a decade out of millions of users which is a hint that cuido probably did the right thing great question next yes sir how I write madam great talk thank you very much and I'm a great fan of abstract base classes and particularly once in the collections module so thank you for that my question is kind of similar and and you might just say the same answer as before but I love a abstract base classes for the way they give us first-class interfaces in Python but there does seem to be some deficiencies compared to the concept from from other languages and the ones I particularly had in mind is that the validation doesn't do validation of function signatures so you can provide a broken implementation and it will still pass validation and the other question would be why validated in instantiation time why not validate a class creation time that Python provides the machinery to do that okay all right I'll stop there so let me summarize the first question what is the best way to connect a giraffes heart to his lungs is it to go directly or to go all the way up through its neck and all the way back down you might think direct is the right answer but in fact the giraffe has a artery that goes all the way through his neck and all the way back down what happened was at the time are early in the drafts evolution the direct path was the one that was used but it was well below the neck bone but the neck got larger and larger and larger over time in the early design decision persisted so in the case of abstract base classes when they were they had some somewhat limited goals we didn't try to solve all the world's problems all at once are the required methods present and by itself that was a useful thing it would have been nice to have extended it and say do the signatures match and do the types match did we have a good typing system at the time no we didn't get a good typing system until basically right now we've had typing for about three years but it hasn't gotten good until fairly recently so abstractive ace classes predate typing by a very long time we also have signature objects available in Python now they didn't exist at the time as well I think signature objects are on the order of six or seven years old whereas abstract base classes are on the order of twelve years old something like that and so the tools necessary didn't exist also when a person is building something like this they have a class of problems they want to solve they're not trying to solve everybody's problems and so this solved a particular set of problems cuido had at the time and seemed to work reasonably well at that particular task but over time our needs have grown so we've layered in a type system in addition sometimes you leave tool you try to not solve all the world's problems yourself because somebody else might do a better job at it than you and in fact at the time abstract base classes existed trait le'ts already existed trait le'ts are very much like the validator class I just showed you and they are dramatically more powerful than typing typing can tell you something as a list or a list of integers trait le'ts can tell verify is it a list of integers in a particular range that are all even numbers and so you can layer in logic that is not possible in a regular typing system in fact the world is moving toward types that are smarter now I think it's called appended types and has magic words like homotopy and words I haven't learned yet to describe it but the concept of traceless was around for a long time that was a dynamic runtime validation they could check a lot more than types it could check signatures it could check ranges etc and so perhaps this was a need that was not best met through abstract base classes so at the time it was made we didn't have the tooling for signatures we didn't have typing and we had a better tool trait le'ts that did a better job at that particular class of our problems which is not to say that if we had it all to do over again and knew all known use cases that there wouldn't be a unified approach that we wouldn't have separately validators trait --let's signatures abstract base classes they might become uncommon unified thing so like that to wrap example were we redesigning a draft for a scratch we might design a draft much differently than it's made so it's safe for a draft or drink water because the Lions know to wait for the draft to drink water and they come up take one swipe at the neck and then the giraffe is a goner and so we can exploit this weakness in the system our selves and your second question was why the cuido choose to do checking at the time of instantiation of the class and one practical reason is your subclass might not implement all of the required methods itself it might implement a handful of them and then you have a subclass to a subclass also Python being dynamic allows us to after the class is made inject methods after the class are after the class has been made some people feel that this is a little bit hacky but it gives Python superpowers occasionally it lets us do things that are very hard in other languages so you don't day-to-day monkey patch your own classes but once in a while it's a really wonderful technique in other words we can't know when the class is defined whether all the methods need to be present it is only when we instantiate by then they really need to be there otherwise the class itself is not functional this is particularly true because I believe given where cuido is working at the time they had dynamic class creation and so users would specify methods that they want and they would populate on the fly so I believe that was a basis with a design decision certain practical considerations that said if I were designing this module I probably would have designed it differently than cuido did on the other hand as I'd mentioned before this modules been around for a long time I watched the bug tracker every day every single day I look at the bug tracker and I can assure you since it's been created no one has ever complained about it and when you have millions of users and no one not complains about it either they're a not using your tool RB they're completely satisfied the truth is uh in fact somewhere in between and it just may be that there's a limitation the tool that in theory matters but in practice doesn't seem to come up a lot in other words it doesn't seem to be a real-world problem both excellent questions from Michael Ford who will be a speaker later today and somebody you really want to listen to somebody with really deep real-world experience more questions yes sir he needs a microphone oh I have microphone oh okay yeah oh you have a microphone yeah thank you for your talk actually sometimes user names extra base classes it's kind of overwhelming because you can achieve same behavior by just raising not implemented error you know a base class function so what do you think about that screaming I believe the question was can you do without abstract base classes simply by having a parent class to find a method that raises a not implemented error returning not implemented is a different thing that's done for comparison operators so raised and not implemented error and in fact this is something that was commonly done before there were abstract base classes there's a disadvantage from that it's not introspective all and you don't find out until you've called the method at which point it's too late and it creates a risk of a error in production for a method that did never got called until one day someone did something exotic but in fact that is a perfectly reasonable approach is design a method that will fail a hundred percent of the time when you call it also a problem with this approach and we also use document and dock strings for it as well weave right in you need to define a method that does so-and-so but these things weren't introspective all there was no way to programmatically go into a class and say are you an abstract base class what methods do you require so this defined an API so we can actually ask now what are the abstract base classes and we can say which ones of them have been filled out and which ones are left and give very useful error messages all of those things are difficult to do by returning on not implemented so like most things in computer science there is a simple answer to the problem that works practically a lot of the time but sometimes we throw a little bit more machinery at the problem and we get documentation we get enforcement we get the ability to lie to is instance if we need to get the ability to ask a whole bunch of classes which implement interface they implemented I'd like to do a a second photo onto a they part of the motivating use case for abstract base classes I was originally there was exactly one problem that uh that got us started on discussing abstract base classes and I had a very different solution in mind that was as low power as your solution and roughly the problem was this in Python you can make a mapping with a method called get item and if the get item fails it raises a kir or you can make a sequence that has a method under get item and if it fails it raises an index error and these two things behave very very differently than each other and we found all throughout our source code particularly in C Python source code it was often necessary to distinguish is this object a mapping or is it a sequence and if you check to see if getitem was present it didn't tell you anything you actually needed to know well what to get item are you so my proposed solution to the problem was to just put on a attribute that says I'm a mapping or an attribute that says I'm a sequence you know sequence is true you know how do you know if something is a sequence just to ask it it'll tell you how can you tell if there's life on Mars it's easy ask a Martian and even if they say no the answer is significant so I proposed an attribute cuido was driven more to this this style based on his experience as I believe in Java and at first I was opposed to it because I thought boy that's a lot of machinery just throw it just solving the problem are you a mapping or are you a sequence what made me really happy with the result though was we got more than an identification of witches which we got the mixing classes and that was a huge win the other win that we got out of it was prior to abstract base classes these words have been using like mapping and sequence thrown around randomly without precise meaning so when one person said sequence and another person said sequence we weren't necessarily talking about the same thing now when we say the word me sequence we mean precisely that it is a sizeable indexable ordered that it has a count method or removed method etc and so making these abstract notions precise was also a big win in a way it's the same is true of the typing module even if you never use my PI to validate the typing in your module the fact that we have a language now to describe this is a dictionary whose keys are a string and his values are a tuple that consists of a integer and a decimal was something that we used to say very loosely in comments and people would see it in different ways it was hard to identify we now have a precise language for describing types so abstract base classes were a win there too so even if you never use ABCs you get it you've gotten a small victory because we know precisely what is meant by a mapping by the way do you know the name of my training company sorry you missed that at the beginning and the opening slide it's called mutable minds what is the one thing I need to be able to teach people do I need a projector I do not do I need a computer I do not what's the one thing I need mutable minds and once in a while when I teach a class I get a room full of people with immutable minds okay so I've defined an abstract base class immutable Minds I've named my company out for the ABC and what do you need to be teachable a mind that can be mutated if you're open to new thoughts will do great did that answer your question sir yeah thank you can you thank you for a wonderful talk well I've seen several techniques to write a body to implement the body of method name actors abstracts to write write a string to return to race and non implemented terror or to write some some default and simple implementation just for demonstration what is your opinion what is the most preferable technique to do it can you take one more crack at the question I didn't catch all the parts ah well when you mock a method and as an abstract you need to write something in the body yeah okay so when you define something as an abstract method it cannot be called because you can't instantiate the class so in a way how will you define the method is irrelevant all that has to match is the function name interestingly you don't even have to have the word self in it for it to work because it will never be called that said Guido was now the creator of abstract base classes but also its first user and made our first examples so he set a pattern for us and the pattern has put the signature in of the actual function that you're going to use because you're essentially laying out a blueprint for somebody is going to implement this method what should it look like if this should return an integer he would put return zero that would be an archetype archetype of the answer if it returns a string he would return an empty string or he would return none if there was a particular seven exception that was characteristic of that function he would put in a raise there so for a get item method the abstract base class for it the function would be defined as raised he error now this function is never going to be called so all it is is a template that says when you write this function it should take a key self key and if a key is missing it should brace here in other words it should give you write this in a way that gives hints to the person that's implementing it dock strings are also fine for this purpose I would say there is no standard and best practice here there is cuido has made a model for us and to the extent that works for us it's great to the extent it's not working for us throw in all of your other tools but in a sample implementation but in type annotations put in dock strings do whatever you have to to communicate so I was a private pilot for a lot of years and it was interesting the guidance they gave to us and talking on the radio and roughly it was always have a strong preference for standard flight language so there's words like Roger and Wilco and immediate or words they have precise meaning to pilots and then you layer in other words along the way they're just technical details exit runway two-four left immediate however the regulations are very clear if communication is not occurring you can switch to any other thing that works and whatever you need to communicate work so the example for the take care of air traffic rollers is Cessna 385 exit taxiway 3/4 left immediate and then if you stay on the runway they switch language get off my TM runway right now you're about to be crushed by an Airbus a380 nice knowing you and so they will communicate in a very direct way likewise the same is true of documenting abstract base classes say whatever you need to to communicate to a person here's the method that you need to write fair enough alright great question thank you all so much again against the honor baker evan [Applause] by the way of giving a tutorial tonight the room holds 55 people I hope you all come and crush us all in there I'm doing two hours of lessons for you it'll be intensive you will learn a lot I hope to see you tonight

Channel: Видео с конференций IT-People

Views: 24,172

Rating: 4.9793458 out of 5

Keywords:

Id: S_ipdVNSFlo

Length: 62min 1sec (3721 seconds)

Published: Wed Jul 17 2019

Reddit Comments

[u/erez27]

Cute talk, but I'd say its title is a big exaggerated. A more sensible description would be "Here are a few classes to make it more convenient to do things you could have already done before".