James Powell: So you want to be a Python expert? | PyData Seattle 2017

so good morning everyone I'm James Powell as a session chair mentioned I've been doing PI data for a very long time I think this is my 2728 PI data conference I've attended believe this is the 50-something PI data talk I've given in the last four years I don't know that that makes me a Python expert I can tell you that oh we have a lot of material to cover so I won't go very deep into my own introduction other than to tell you that even though I attend many PI data events and I help out and organizing all of these I do this in the capacity of a volunteer there are a lot of volunteer opportunities available with mum and with PI data itself so if you're interested in running a PI data in your own city or getting more involved with open source data science please let me know or let any of the organizers of this conference know there's always room for more volunteers for those of you who've had the fortune or misfortune of seeing any of the talks that I've given previously at PI data event I often give talks about strange things that you can do with Python like embed Python interpreters within themselves or add features to Python at runtime or patch your Python so that you can load other Python interpreters using dynamic linker tricks and sometimes the subtext of all of these is you know these are the lifestyles of the Python experts what kind of crazy fun that we can have and so I decided maybe I should give a talk that might actually be useful to an audience as opposed to just showing off some crazy things that you can do and so one thing that might actually be useful is to really talk about what it means to be a Python expert I do a good amount of doing a couple of different things in my day job I work on a variety of different projects I can tell you if you are an employee at Microsoft and you're interested in Python something that I can talk to you about Python Microsoft is very invested in the future of Python in more ways than you might even know I also do some corporate training and in that corporate training this question of what's needed to become good at Python comes up very often so I'm asked to answer this question for you I can tell you that what it takes to be effective in Python it's pretty straightforward to be effective in Python that requires a certain understanding of a couple of core mental models of the language understanding of some of the things the language comes with like the built-in data types and the built-in functions a little bit of an understanding of what's available in the standard library this talk however is focused on the next step what does it take to be a little bit better than just effective what does it take to be good at Python or an expert of Python now I could spend the next two hours telling you in very philosophical terms what that means and I could tell you why Python experts do things in one way or the other we can even go very deeply into some of the meaning behind the Zen of Python but instead what I'd like to show you more specifically is a couple of what I call perhaps metaphors and programming in Python it turns out that as many of you know Python though it originated is something of kind of a scripting language is the convenience language to patch together or to orchestrate code that was written in C to write simple scripts has since grown into a full-fledged general-purpose programming language and along the way it's developed its own ways of thinking about certain core constructs in programming what I like to do in the course of this session is show you four features of Python and the ways that an expert would think about these features what I want to show you is not the syntax of these features the purpose of this presentation is not a feature parade it irritates me unending whenever I see somebody who's writing a book about advanced Python and the entirety of the book is chapter one here's a feature chapter two here's another feature chapter three here's a third feature and the entirety of the discussion of these features is merely its formal characteristics Oh a generator has a yield instead of a return Oh you can't send an on none value into it just started generating a lot of fundamental details of how these features work but very little talk about how we can conceptualize these features in the broader sense of what they mean for modeling core problems and so what I'd like to do is I'd like to talk about four of these features that you may have seen before and a couple of the core mental models for how you can think about them and how you can think about Python as a whole and wrap all of these together one thing that you'll notice is in addition to you can probably hear them a little bit selphie a little bit sick that's only going to exaggerate my inability to remember the syntax or the documentation for basically anything so you may see me stumble but part of this presentation is to see me stumble and to understand that in cases where you don't understand or you don't have the documentation memorized or some of the corner cases memorized what really is more important for all of these features is an understanding of what they are and what they mean and not so much all of the implementation details those implementation details are of course critically important when you're using a feature but that's information that you can look up and refresh when you're actually in the middle of using that feature itself before I get started I'd like to take a small survey of small informal survey of the audience so how many of you here are Microsoft employees almost everyone how many of you are Microsoft employees that use Python as part of your Microsoft work how many of you are just Microsoft employees that have seen Python used in various areas of the company but don't really have direct interaction with it yet how many of you consider yourself to be pretty okay at Python I mean if you consider yourself to be really good at Python a couple people how many people are already in the back of the room thinking I should be in the front of them getting a talk myself maybe one person maybe one person in the front okay maybe maybe we'll have to show you something new when it comes to those of you who've seen Python how many of you feel like you're pretty effective in Python I can give you a simple problem you can write some functions maybe even write a class or two you can solve the problem without too much sweat it's that most of you how many of your complete newbies you're like I have no idea how you can get started with Python okay for those of you who are complete newbies you may struggle a little bit part of this talk does presume a baseline of understanding of Python however I don't think that the core lessons that you'll pick up here our lessons which will be totally lost on you there are a couple of core lessons which even those of you who are somewhat effective in Python already may yet have internalized and we might we might have a little bit of time to repeat this what I want to start with I'll start with something really really really basic and I'm going to start a little bit more basic than what I plan to just because I thought the audience might be a little bit more advanced than that so what I'll talk about at the very beginning is something that we might call the data model now what you'll see is I have all of the materials for this prepared and memorize in my brain and unfortunately my brain is getting older and older as time goes on and so I've already forgotten exactly what I want to talk about and so you'll see a little bit of roughness here but part of the purpose of this presentation is to present you with just the information that matters without a lot of the distracting details and so I will actually rewrite all of my notes for this as we go going to be a little bit rough but hopefully it will be an opportunity for us to focus just on what really matters and for us to put aside a lot of the associated implementation details and things which frankly it matters when you care about them but it's not part of that core lesson of understanding Python I want to start off with something very simple and you can see I created a directory called data model and I'll just create a simple file here I want to create a class and we're going to do all of this everything in this course will be or everything in this session will be in Python 3 6 it'll be in Python 3 6 just for my own convenience just so that I don't have to worry about all the details that might differ between Python 2 and Python 3 there's no particular reason why I choose Python 3 versus Python 2 for the purposes of what I'm going to show you other than it's the Python that I happen to use and it's the Python that happens to be there's some political reasons behind why python 2 or 3 but even skipping that it's just for my own convenience just to simplify things so let's start with something very simple I have some class I want to do some kind of mathematical operation I have some piece of software I write and I have some class here that represents a polynomial some simple you know linear polynomial with various coefficients and I want to represent it as a Python object and so I start with some code that looks like this right class polynomial doesn't have anything interesting and I can create polynomials very simply right and I might say p1 has coefficients one two and three right so this would be x squared plus 2x plus 3 and p2 has coefficients three four and three so this would be 3x squared whoops plus 4x plus 3 okay and I want to be able to do some simple things that we did in math class with these I want to be able to add them I want to be able to subtract them I want to be able to you know figure out some details of them the very first thing for those of you were relatively effective Python programmers if you're going to look at this you're going to say why on earth did you write that in four lines when you could've written it in two line why did I write this in four lines here when I could have written it in two months how would I compact my code here so instead of constructing my two polynomials and then initializing the coefficients I could initialize them all together what would I do we're trying to get you to give me a little bit of feedback here so what would I do what am I missing here my class has nothing in its body what it's what is it missing to allow me to do this little bit more comedian a little bit louder for me I needed an it method and so the very first thing when some of you look at this as you say hold on you're wasting our time here you probably want to have internet method that looks something like this so you can write these lines much more hoarsely and you don't have to have four lines when two lines would do right so now we're happy what's so now we're happy now if I rub my coat and I'll split my screen shrink this a little bit and I'll maybe print out actually I'll do this so I'll run my code and I have P 1 and P 2 and then somebody else looks at them says oh my goodness print it out to the screen that looks so ugly what am I missing here let me see another method what am I missing I'm missing a method that corresponds to what happens when I call this function at the top level repper to figure out the representation of my Python object yes or no yes so what method do i implement up here I have some method called rep work right so I have some method here that returns a string that gives me the information gives me a printable representation of this class to the screen and so I'll make this very easy like that and so I'll run my code again and we'll see how there we go those are two polynomials pretty straightforward and we could write we could make this print out a little bit prettier we could see the x squared the X and the constant term if you wanted to but you can see this is a lot easier than just seeing the name of the class and it's memory location and we can continue down this exercise because the next thing I want to be able to do is I want to be able to add these together and when I add them together nothing happens Python gives me an error unsupported operand types 4 plus polynomial polynomial but that doesn't make any sense I know I can add polynomials I did it all the time when I was in high school how do I tell Python how to add polynomials I have another method called add and in my add method I might return a brand-new polynomial where I just give it all of the coasts there of course ride just zip together and add each of the individual items so when I run that in my script here p1 plus p2 it gives me a polynomial with 4x squared plus 6x plus 6 now what's the pattern that we see evolving here for those of you who are off the top of your head you didn't you didn't remember which of the underscore functions are for those of you who haven't ever seen these underscore functions before you've begun to see some pattern and what is that pattern I have some behavior that I want to implement and I write some underscore function some function that begins with 200 scores and ends with two underscores and for those of you who are Python programmers already you might be familiar with these underscore functions you might call them dunder methods or double underscore methods or if you find that word abhorrent as I do you might call it by a different name data model methods and you might say it's absolutely absurd thing to call this data model methods that's so many more syllables than dunder methods but there's a reason I call them data model methods which is I don't think I have internet access but maybe I do oh I do if you google for data model and python the very first or second link depending on whether you use Bing Google or DuckDuckGo will be a piece of documentation here that tells you all of these methods and what they do but the pattern is a little bit more interesting than that it's more than just having all of these methods on a single piece of documentation it is that whenever we want to implement some behavior of Python we want to tell Python for this arbitrary object do this behavior give me something that is a principle representation for this object perform some notion of addition for this object we always see this common pattern and the pattern is as follows there is a top-level function or some top-level syntax and there's a corresponding underscore function the exact arguments that underscore function takes will be determined by whatever you find on this documentation page here the name the default names of the arguments you can pick up off this documentation here so for ad I called it self another that's just the conventional names for this the what each of these actually do you can read from the documentation but there's something more fundamental here there's a pattern when I want to add two objects I implement ad when I want to initialize an object I implement in it when I want to tell Python what to do when you call repper on an object I implement wrapper we know in Python there's a function called Len it gives you some notion of the size of an object what would it make sense for the size of a polynomial to be what what like that notion be any thoughts the degree right the highest degree so we want to implement this notion of size on this polynomial how do we actually implement it what's the name of the underscore function we have to implement the one hint is there's usually a very close correspondence here so what would it be here Len if we wanted the exact arguments that any of the caveats for how this needs to be implemented with a look at documentation there isn't anything special in the case of Len and we just return something like land of coefficients so this will tell us the size of the polynomial in terms of its degree and you can see these are both degree three polynomial the pattern that we see is as follows the Python data model is a means by which you can implement protocols those protocols have some abstract meaning depending on the object itself in the case of a polynomial to add them means whatever that meant a math class in the case of a polynomial to find its size means whatever that would make sense for the case of mesfet so it may make sense for us know what decides this polynomial is in terms of its highest degree to get its representation its principle representation has meaning that's given to us by the documentation the representation of any Python object is typically whatever string we'd have to type at the consult to create another instance of that same object and so on in each case this protocol exists there is some underscore method that implements that protocol there is some top-level function like Len or repper or some top-level syntax like the plus sign or the multiplication sign that allows us to invoke that protocol and it all fits together in this hole we can think of this as a protocol in the sense that like many protocols when we implement something like Len we do that by delegating back to the protocol itself so notice Len is implemented in terms of lead being called on a constituent object ad is implemented by adding up some components namely the coefficients repper is implemented by calling wrapper on some component we see this is a very common pattern in the Python data model every time we want to implement some custom behavior on a Python object we do it by implementing an underscore function which ties to some top-level syntax or some top-level function and we implemented in terms of that thing itself so for example if we have some top-level syntax parentheses that coming after the name of an object and we might call that the call protocol we'd implement it by implementing the function called call we could look at the documentation to see any caveats or any special things we have to keep in mind when we implement that in the case of a polynomial this may not make a lot of sense perhaps to call it means something different maybe if we want to have some really horrible syntax where we say p1 p2 means multiplication we could do that I can't think of anything more ptosis than that but perhaps that's what we what we want but in any case we may implement this in terms of calling something else this is one of the core patterns that we see a Python this is probably one of the very first patterns that you'd see as a novice Python programmer in fact when you look at object orientation of Python there are three core patterns that you have to understand to really understand object orientation Python this is one of them the protocol view of Python the second one is the built-in inheritance protocol and how it works where you go on a Python object to look for things and the third one is some caveats around how object orientation of Python works that's where I wanted to start off with and for those of you who are just kind of on that level of being effective in Python these are the kind of patterns that I would encourage you to look out for as you learn Python I want to go much deeper than this and I want to jump into a very tricky metaphor using a feature which you may have heard of and you may have been warned against something you may have seen kind of along the margins when you're using some library if you do anything with web programming and you use Django you may have seen this in the ORM or you may have heard people whispering about this one feature and I want to bring up this feature for you I want to show you that it has a very clear and concise meaning just to illustrate how powerful these mental models are for understanding Python itself and for driving expertise so let's shelve this for a second let's show this idea of Python having a protocol orientated data model and let's look into a new feature I'll create a new directory called meta classes and we'll talk about meta classes for a bit now the gist of this act this tutorial is actually leading you towards three features which are far more common I'm for more interesting the meta classes namely decorators generators and context managers there is a reason I want to start you off with medical axis it's a feature that everybody is warned against using they're told it's magical it makes no sense it's totally the most complicated thing in the world when in reality it is the case of a feature where if you understand what the feature is really about it is very clear and very obvious for why and when you use it and you can it's something that you can kind of shelve away in your mind as oh this feature does this this is why I want to use it I don't need to use it all the time so let me show you what a meta class or what meta classes are really about and I'll show this to you for an example now here's an opportunity for you to try something because I think you'll really appreciate this example if you try let me do this in a different way I've read up P Y so I have two files here I want you to actually try this I want you to remember I want you to pretend that there are two groups working on some one piece of software one group is a core infrastructure group and they write library code and the other group is a developer group and they write user code the developer group takes the library code somebody else wrote and they use that library code to accomplish actual business objectives the core infrastructure group gets paid a lot of money to accomplish no actual business objectives but to provide library software that the rest of us can use isn't that the story of tech companies which which team would you rather work on nobody has any opinion whatsoever I'd rather work on the user team because I think that's where some of the more exciting work is and you're a lot closer where the money is actually being made so that can't hurt but some people like to be on the core infrastructure team because they think that the technical problems are often much more interesting than some particular business units problem around not getting data in the right format or who knows well let's say that this library provides classes and these classes are expected to be subclassed in the user code so I'll create a class here called base and I'll create a class here called derived that derives from base okay I've got five lines of code on the screen this is a basic pattern we see this pattern all over the place somebody writes some class in library code and somebody else has to use it I'm going to add two things one function on each side and then pose a question to you let's pretend you exist on the right-hand side of the screen you write user code you cannot write or alter or modify or any way and touch the code on the left-hand side of the screen you can use it but you can't change its library code you don't have source code access to it even if you did this is being deployed in some environment where any local changes you made wouldn't be preserved maybe in the deployment environment this is pulled down from NuGet or somewhere else so you have no way to change the code on the left-hand side of the screen your only control is code that you can write on the right hand side of the screen where could the code on the right-hand side of the screen break very simply where could this function break if there is no foo method exactly what can you do in your code to make sure that at least at the very minimum if the person who writes the library code removes that foo method your code fails not at runtime but at any point earlier than runtime one very simple thing you could do is you could write a test that just calls bar on your derives class very simple thing and make sure you run your tests before you deploy and you'll see the code fail sometime before it hits the runtime production of art that's one thing you can do anything simpler you can think of take a minute and just try and think is there anything simpler you could add to this code to just make sure that this code has the ability to fail before it hits the run time production environment doesn't have to there's no there's not really that same concept of compile time and Python as there is another languages but anything you can think of where you could just save yourself a little bit of effort if those core infrastructure meatheads accidentally screwed something up and keep you going you know I was I was on a team I I often worked in a vendor capacity working on very small very narrowly defined projects I was in a situation on Monday err there was a sev one issue and I was the only person on the team that was thought to be involved with it who was online somebody messaged me and asked me can you help me with that said one issue I told him I have no idea I work on one tiny very small Python specific very narrow area but apparently I was the only one between between the manager and the top of the company itself was online so I was sitting there trying to sell trying to tell people sorry I don't know anything I'm hopeless let's see if we can avoid those kind of issues let's see if we can avoid there's anything we could add to our code so that you don't have to message me and hear me complain to you that I don't know anything and then you see how much I billed per hour you're like well why are we paying you so much we don't know what to think what could you do any thoughts I'll give you something really easy real easy oops what is the student what's happening here here I check to see that that method exists if the method doesn't exist I assert the code will fail so all I have to do is make sure all of my modules can be imported which does not require a runtime environment I just import everything if for some reason they broke this this method here they gave it a different name they call it food instead I would see this code fail before this class is never defined and I get exactly what I wanted I have some early warning that this had broken what we can see in general is that what I'm trying to do here is enforce a constraint I'm having the user level and force some constraints on the library level in other words the derived class is enforcing constraint on the base class the drive class is saying hey the base class has to have these characteristics in order for me to run and be happy and if it doesn't have these characteristics in this particular case if it doesn't have this method implemented I'm going to fail loudly and I won't run so that's pretty straightforward actually that was that was pretty straightforward it turns out there's quite a few different ways we can do this this little approach is very simple very straightforward but let's flip the script a little bit let's say instead we have a case that looks like this let's say now you're the core infrastructure writer and you have to deal with those meatheads in the business unit actually using your code and abusing your code and misusing your code I have no idea what they're doing and you write your base class under the assumption that some responsible developer in the business unit will go and implement this bar method because if they don't everything falls apart now in this example you exist on the left-hand side of the screen not on the right-hand side of the screen you have no ability to change the code on the right-hand side of the screen even though you may be higher up in the technical organization you have no idea where this right-hand user code even sits your only ability is to put out library code for people to consume you have no idea who's even using your code you will even if you could see their code you might not be able to fix it let's ask a separate question but a very similar question to what we had before sitting on the left hand side of the screen how do you make sure that the user on the right-hand side of the screen doesn't screw up how do you make sure that in this case that bar method is in fact implemented because if it's not your code breaks and they're going to blame you they're going to call this foo method they're going to see in the trace back that foo method is in your file and they're going to blame you for getting this wrong so how do you work the other way how do you make sure they don't screw up we can't do the same thing we can't say assert has ADD or derived bar this doesn't make sense on the left-hand side of the screen we have to move that to the right-hand side of the screen we're not allowed to touch the code on the right-hand side of the screen so what do we do any thoughts turns out there are about three common answers to this one of them is metaclasses there's two other answers any thoughts where do we put the try-catch we could do try here that would be fine except this will only occur at runtime so if we try catch here it's not really any different than just running this function and seeing the error will still only see the error at runtime we won't be able to catch it before that code his production on July 3rd and they're calling me up to ask me to help out and like I have no clue about they're like you're on a team that's that's involved with as your security and I'll say I have no clue anything about Asian security true story really happens what can we do well if we think about that very first example we can see that our data model example actually goes much deeper than just that and the reason that we could call Python a protocol orientated language is not just because the Python data model the object model is protocol orientated but that the entire Python language itself has a notion of hooks and protocols and safety valves within it I want to show you something kind of interesting one thing that a lot of people who come to Python from other languages don't realize is that python is a much simpler language than you might appreciate for those of you who are originally C++ or Java programmers you have a mental model of your head in your head for how programming languages like those work and you have that mental model split between some compiled time steps and some run time steps and in that compile time step there's an enormous amount of complexity if you look at all the different optimizations that the visual studio compiler can perform on code there's an enormous amount of complexity there Python it turns out is a much much simpler language Python code runs from top to bottom linearly and almost everything that you see in Python almost every statement that you see in Python with the exception of two of them are actually executable runtime code in C++ or Java a class statement is not executable code in C++ it definitely is an executable code in C++ a class statement just says here's a bucket of bits here's how you name portions of that bucket of bits in Python a class statement is actually runtime executable code what does that mean well let's take a look what that means is that in Python I could do something like this define a class in a loop 10 times in a row totally the most useless thing I could ever do has no effect whatsoever but I can stick for loops outside of class bodies and redefine classes if I want has no use but it's it's meaningful in Python I can do the same thing the other way create a bar method under this foo object ten times only one of them will survive all the last one will call all the previous ones but Python accepts this as syntax because this class statement is fundamentally executable code now I told you and the direction where we started this was talking about Python as a protocol orientated language let's see something kind of interesting I'm just going to create a function and I only need this for wrapping purposes let me create my base class inside some function and let me use a function in a module in the Python standard library called dis dis stands for disassemble let's disassemble this and see what actually happens in the Python bytecode what we see is something kind of interesting we see that there's this thing in the Python bytecode called load build class so it's actual executable runtime instruction in the Python interpreter for creating a class well hold on a second in the very first example I showed you I said there typically tends to be in Python some correspondence between some top level syntax or function and some underscore method that implements that syntax or function there happens to be some top level mechanism here it's not explicitly syntax is not explicitly a function for building a class it turns out in Python there is a hook there is an underscore function that allows you to hook into and to do things with the process of building class and I bet you can guess what it's called because just like the correspondence between repper and double underscore repre Len and double underscore Len adding and double underscore add the same thing here there's a function of Python called Bill's class it sits on a module called the built-ins so if I equals IBC so if I define a function here called will do like this if I do this I'll capture the original build class I'll write my own build class I'll import from the built-ins and I'll swap them out here I can patch into what Python actually does when it creates classes and I won't do anything interesting in here I'll just print out all the arguments they're passed okay and so let's run our Python script one more time Python - I user duck py whoops and you can see here's the class that was created in fact you could even see I caught another class that was being built I was probably being built somewhere in the interactive interpreter itself but the most important one that you can see is I actually can catch can actually catch the building of this class you can see there's some interesting things that I'm passed and pass the name of the class and past its bases and I passed a function then pass no other arguments well what I can do now that I can see the class time building is I could hook into this and I could do something with it right I could add my assert here I could say if the actual arguments for this are this is where I can't remember them the function the name and the basis function the name and the basis and I could say if the base actually the base if the base is equal to this base here or if it's actually this actual base object I could check at the bar math is defined and when I run this you can see let's pass in any kW here as well which arduous would take one to help the third one is optional because not everything has to have a base class and [Applause] and who cares we this is a little this one you can see I struggle with this one a little bit because we never we don't typically do this we say if space is not none okay there we go you can see we were able to have a little place where we could check if that bar method was defined this isn't typically what we do but it's to show you this idea of Python being a protocol orientated language is actually quite a common pattern and quite a fundamental piece of Python almost everything that a Python language does in an execution context like building classes creating functions importing modules you can find a way to hook into that once you can find a way to hook into that you can start doing things that you want to do like check is my user code going to break from the perspective the library author now the reason that I struggle with writing this function here it's in part because first what's really the most important here is understanding this pattern exists knowing that there are options for solving this pattern and also knowing that this is not the option that you'd ever use for solving this better build class is available for you to use but it's not how people actually solve this problem there are two fundamental features in Python that people use to solve the problem of enforcing constraints from derived class to base classes the first one is the metaclass meta classes are merely classes that derive from type that have some special methods on them but you'll have to read the documentation to understand all the special methods for but fundamentally allow you to intercept the construction of derived types the method that you care about here is called new and it takes three or four arguments basis body and the name basis in the body and we'll see where it gives us an opportunity to actually perform our check a patient's body you can see it got called right here with our derived class you can see this last argument for the drive class is the body of that class it's a dictionary with all the methods of that class you can see in it the bar method and so simply to get the behavior I want I could simply do assert bar or if not bar in body raised type error bad user just like that this is the first answer to the core question in order to enforce the constraint the other way around if I can only touch the code in the left hand side of the screen and not the right-hand side of the screen I can't just have assert lines all over the place I have to do something fundamentally different and the thing I have to do fundamentally different is I have to find a way to intercept how classes are constructed of which there are three common approaches built class which is the least common approach but works on the idea that Python is a protocol based language has the safety valves and hooks even in things as fundamental as the construction of any class second meta classes which are just the mechanism by which classes are constructed that's the approach that you see here and I'll show you the third one the third one is just a slight variant on the metaclass approach when people tell you about meta classes typically they tell you this is a magical feature you'll never need to use this feature it's overly complicated and the reason that I like to start this this session with that despite its yes being a very rare feature feature you'll very rarely have to use is to show you that every feature of Python so far has at least one or two very clear and unambiguous mental models is applicable to one or two very clear and unambiguous metaphors for where it's actually useful in this case meta classes are how you can make sure that if you only control the code on the left-hand side of the screen you can still enforce constraints on the code on the right-hand side of the screen you can enforce constraints down the class hierarchy from a base class to a derived class this is one approach using a metal clasp that just intercepts the construction of the clasp and the second is even easier it turns out that in Python meta classes are have a bad reputation because even those people who understand where they're useful see that meta classes have a lot of complexity to them part of what I haven't told you at all is any of the syntax of the meta class or any of the arguments they take or any of the details for how the class hierarchy fits together because all of that doesn't really matter when you need to know that information you'll go look up and find that information but unless you know why you want to use the metaclass in the first place it doesn't matter how many details of the feature you need to know once you understand where and why you need to use the feature then the rest of it are just details you look up when you start looking up those details you'll see that in many of the features I show you the story isn't that easy there's always caveat there's always catches and one of the catches is this metaclass feature is very clumsy to use and so in Python 36 a new feature was used or a new feature was added called an it subclass and in its subclass I think it goes here it just gives you a method that allows you to hook into when a subclass of some class is being initialized and that's it and I think it goes I think it goes here let's find out if Brett were here you could tell you exactly what this let's remove the metaclass for now [Music] I will do one thing if will be that there okay so yes it got called the right place and then it's a PLAs okay yeah so this gets called with the self so the other approach we do is as follows this is being called on that and with the knit subclass we have other hooks for figuring out this one would come in in its subquest have to remember what arguments it takes but you can look into it really frankly it does not matter that much so this is my simple story about metaclasses metaclass is our feature that we're often told that are an expert-level feature you're not supposed to ever use them you should shy away from them you should cover your eyes before you read code that includes metaclasses and it simply isn't that it simply is that for all of these advanced features there's usually one or two very clear metaphors to understand what that features all about and once you understand what that features all about there's a lot of details you need to read through and documentation that's the approach I want to take for the remaining three features I want to show you the next feature is a very simple feature called decorators I'm sure some of you already see in this feature but I want to show you kind of where this feature comes from it's a very simple and very boring feature indeed and you've probably seen this feature before it's this little act that shows up on top of some function definition this little app with a dotted name so you see something like a dot B dot C and for those of you who are very familiar with this feature you might wonder why you can't do this or why there are limitations to what you can put in a decorator line the answer for it is a really stupid answer there is no good reason but there is a limitation for those of you haven't seen this feature before I want to show you something about Python something very cool and very fundamental about Python I told you when we talked about metaclasses that python is a live language there is no separate step that turns function definitions into bags of a set bits tagged and some elf binary or some PE binary somewhere that a function definition is actually a live thing it actually runs at runtime there's actually executable code associated with this def F okay so let me show you that in practice here's a function called add that adds two values okay let me create a little separation down here and let me run that and here I have my function add and I can add two values now notice the first thing I did in my terminal is I looked at the function itself and I got a return value you can see that the Python interpreter can actually tell me where physically a memory this function exists and in fact this add function is an object and I can ask it all sorts of things like what's your name what's your module where you were defined I can ask it let's give that a default value there and run this again I can say add what are your defaults and it can tell me what default values it had I can say what's your code like what code do you can do you contain this is actually the byte codes for this add function and you can see them right here I could ask it things like how many local variables do you have that's your name there's one more that's interesting for our names I can ask you what variable names do you do interact with one of the variable names of your signature x and y and do all sorts of things that interact live with this function well it's interesting that this function itself is a runtime object we kind of see this or you might be familiar with this for C or C++ programmers in C and C++ we can have function pointers but they're really just determined by the compiler they're just memory addresses the compiler knows for where it put the bag of bits that represents that function sits in Java for a Java programmer or a c-sharp programmer we might be familiar with runtime classes and functions under those runtime classes and Python it's closer to the latter but much simpler than that every Python structure that you interact with whether it's an object or a function or a generator has some runtime life has some runtime existence you can see it in memory you can ask it questions like what module UI were you defined in you can even ask it very useful questions like if you use the inspect module you can say what's your source code and I'll tell you the source code right back out so you can see you can ask the function to give it source but it'll give you a source code right back you can ask it what file are you in and it'll tell you what file is to find it you can ask it what mods are defined in or I think get line you can get get it ask what line number it was defined on get acid all sorts of important questions well there's something interesting that you can also do let's say that I have some code here that calls my add function maybe it's part of some testing code or maybe it's some code part of some application let's give it one more example so you can see I'll add both numbers and also add strings okay simple enough let's say that I want to change my code very slightly let's say for some reason this add function isn't quite doing what I want or maybe it's doing what I want but I need to do something like time how long it takes to run because for some reason some of these are much slower than other things it takes a lot of time to run I want to time it and so what are my options well I know in the time module there is a time function so I know I can say time and I can get the time right now and you do that again I can subtract them and figure out how many seconds it took and so how would I time these I'd say before equals time after equals time print time taken after - before like this and now I could kind of profile my code except this kind of sucks because every time I do this I need to add this in n places throughout my code after a turn all sorts of places throughout my code there's something missing here there's something wrong here when we look at this example and we think about decorators in the context of the problem we're trying to solve here we see another core and important pattern in Python the Python developers really don't want you to be here today because it's the 5th of July and they'd rather you be home with your family spending time with them I'm very happy that you're here today in this session and I hope you're enjoying yourself so far but what I mean by saying the Python core developers don't want you to be here is this given the option between writing and absolutely complete forward compatible absolutely perfect solution to a given problem and writing the simplest stupidest quickest thing that gets the job done the core developers or the Python language would rather that you write the simplest stupidest quickest thing to get the job done and go home and spend the rest of the day with your family and the reason that you know that that's true is that when you look throughout the Python programming language there are a number of examples of cases where you can write the simple and stupidest thing to get the job done and when the task at hand becomes harder or more complex or the requirements change you can change your code in a very simple fashion to linearly extend its functionality without having to rewrite everything from scratch this is a fundamental difference between writing code and Python and writing code in a language like C++ in C++ there is a lot of upfront design that you need to do in order to be effective and if you make a mistake and something in your upfront design in C++ you're going to be rewriting a lot of code down the line whereas in Python we'll see a number of different features are orientated around how do we write the simplest thing today and then when the problem gets a little bit harder we have an avenue for making our code a little bit more complex here we can see a bad example of that because in the case where I want the code to be a little bit more complex I wanted to capture the timings for how long this function took in addition to the function output I've made my code substantially more complex I've added a lot of garbage and I've rewritten a lot of stuff and you can think the ratio of library code here to user code here it's usually a lot more user code than library code the whole point of writing library code in the first place and so to have three times to have to have to add code in three different places instead of one place makes for something pretty pretty gnarly so the first thing that you might say as well I could add that code in just one place I could say something like this capture my return value and do the time before and the time after print it out here like that and you this would be definitely better than before definitely better than before I've added coded one place instead of three places but if my library looked like this and maybe I had examples my life my library looked like that then I'm still really not in the best situation because I have to add all this tracking code to two places so it's got to be a better way and in fact my library could still be very big and so it still means I have to write a lot of code and rewrite a lot of code so it must be a better way now remember what I tell you I told you that the pipe that Python programming language is a live language that everything has some runtime representation so both this add function and this sub function have runtime representations the sub doesn't work on strings so both add and sub are functions what can I do well maybe I could write my own function called time right or timer and I take a function and I take the X and the y arguments and I apply this function like that so blowing up right and I do the timing before and I do the timing after and I print the elapsed time here and I returned the return value okay now I would just have to change these lines of code slightly I have to change that from a parenthesis to a comma and so on as long as I could account for the default back I have to change each one of these lines as follows it's a little bit better not that much better but a little bit better I'll change the first three and we'll see it at work so you can see from time to time let's make this a little bit smaller so it all fits on the screen at the same time and you can see these three more times the rest part that's still not that much better I can still do better well remember that everything happens live in the Python programming language so couldn't I just create a new function here couldn't I define a new function here that takes two arguments like this and in this function do the timing call the function that was passed in and return this function here and you can see this here is a wrapper that calls the function that was passed in and wraps it with some time behavior before and after and printing something out when I do that I can define functions at runtime I can define functions anywhere and then instead of all this nasty that's changing all this user code I could just say ad equals timer of ad and sub equals timer of sub and you can see it added that behavior across everything and you can see what I do that what I'm doing is I'm taking one function I'm wrapping with some behavior I'm taking one aspect or one piece of code or common functionality and I'm wrapping a bunch of different functions in a really simple fashion and all I'm doing is I'm creating a new function that takes the original function and wraps it with a little bit of behavior before and after and that's it and it turns out that python wants to make this a little bit easier for you because this pattern of something equals call a function on that original thing that exists and that's what a decorator is in python a decorator is merely syntax that's equivalent to the line that says this sub equals timer of sub that's all it is it's syntax that then fits into the ability to dynamically construct a function to wrap this behavior so this is what our final example would look like we would have started with code that looked like this we would have had to add only the timer code and then to decorators and we be able to very easily slip in this extra functionality we want without having to rewrite all this user code ourselves that's the core of what a decorator in Python is at its core it is very simple syntax to just allow you to write an ugly pattern sub equals timer of sub and a slightly nicer way and a slightly nicer place at the top but fundamentally it's about allowing you to take this wrapping behavior for functions and to wrap wide swathes of functions in one fashion without having to rewrite a lot of user code or having to even perform a lot of turn on your library code that's what a decorators go ahead so one of the problems here I think you're identifying here I hard-coded the parameters right and I even hard-coded the default the real answer is this is where we use starts and start start quarks now this decorator works on any function that takes any arbitrary set of positional and keyword parameters and just forwards the one and so now this will work to wrap any function with any set of defaults in any fashion this is part of why starts and starts our quarks are so powerful they're there for arbitrary to write functions that can take any arbitrary parameter spec and forward it on in any fashion because that was at the core of what you're asking James [Music] so you want to do something an N number of times that's easy enough to so what you want to do so say you want to have the ad just run three times in a row and you want to have the sub run twice in a row what you do we'll take off the timer decorator so we want to have some function that runs n times right we have our wrapper function in the middle that takes any arguments here and here and the question is how do we identify how many times we're going to add one more level to this what we'll say it's we'll say that n times gives you a decorator so let's let's actually let's start this very simply let's assume that we always want n times to be twice okay we only want to run this function twice so I'll say four I'll just say n equals two here for da in range of N or V equals function like that okay so here's our rapper they'll just call the function twice capture the value twice and return it and or torn our rapper and then we can do we can apply this here and here now we won't see this really happen we won't see we won't see the results here but if we wanted to you could you could do this so you can see it ran ad twice here ad twice here sub twice there now you want to generalize this because you don't want to run it just two times you want to run at some number of times why can't we just wrap this another function that takes the end and just return this like that right so now this is a function it takes how many times these are what they call higher-order decorators really not that interesting or that important it's a fairly simplistic extension of the idea which is if you could return one if you can have a function that returns a function then you can have a function that returns a function where that inner function then returns another function ad infinitum going as deep as you want to solve whatever problem you want there is a very important core concept that is hidden in here which is what you might call the closure object duality it's not something that we have time to look at in this session but it is a very important duality in Python but it's not one of the topics that we can look at but just to show you this in practice you can see this got called twice this got called five times so you can write your own end times decorator very easily as well and the core of it is just the idea that you start with the wrapper so this is just a wrapper function what you're replacing it with the decorator syntax allows you to do that replacing very easily and then in order to get this programmatic behavior you just add one level a function outside that constructs the decorator the decorator then constructs the wrapper the wrapper and then wraps the function itself it is rare that you'll see a decorator deeper than this just because it's very rare that you have contacts that there's there's there's typically not an additional context where you need to feed information in that requires you to have decorators deeper than this so that's a decorator decorators are probably the least interesting of all the functions were to look at or one of the features were to look at the next feature I want to look at is called generators now some of you may already know what a generator is generators are very interesting and very powerful feature of Python they're also a feature that include where I feel that mental model is often very limited what people understand the mental model is often fairly limited there's more to it than just what you might call the eagerness versus laziness divide but let me start you off with something more fundamental than even that I told you before that there is always some top-level syntax or function and some underscore method that implements it so I told you if you have parentheses after something this is what's called the call protocol so there's some method call that implements it I want you to tell me what's the difference between this and this what's items to add one and add two and I'll even create a terminal down here and I'll add one adds two numbers and two adds two numbers from the outside what's the difference between these two you wouldn't unless you started peeking inside unless you started saying what are you add one oh you're a function what are you add two so they started peeking on the inside of your class you wouldn't be able to tell the difference functionally there's no way for you to distinguish between the two in fact add one behind-the-scenes looks very similar to add two it's just that syntactically it's a whole hell of a lot easier to write so from the perspective of actually writing the code you're always going to want our proof you're always going to want to you know do this way now there is a fundamental difference which is if you wanted to start adding some stateful behavior like this right you wanted to add that and clearly there's one way to do it for add for it for the adder there's another way to do it for the function so one way to do it for the classroom always do it for the function and this is the app for mention what I hinted at this object close your duality so actually a couple of different approaches you can think of for how you do this the core idea here is that fundamentally there is some very nice syntax and then some object model that everything kind of sits in and this is closer to what the object model itself looks like you can think that ad 1 is some object that kind of looks something like this although we'll remove the statefulness stuff just so that the two examples are at parity let's think about this in a different way let's think about some function that takes a lot of time to do something right maybe a function that goes off and performs that request and it needs to load data from a database what would that function look like well it might have their words the resulting rows and it might say you know while DB read rows that a pen something now this will be a little bit tricky for us to actually demo so we'll do a couple of simplifications just to make this code look you know kind of mimic what long-running computational code looks like but without actually having to use any fancy libraries so let's say what we have instead is we have some code and it just sleeps for a half a second every time it provides a return value and the return value will just be a number from 0 to 9 let me ask you a question we'll call this our we'll just call this compute whoops when I call compute how long does it take to run five seconds what if I only care about the first return value what if I only care about the first value how long does it take to run what if I care about the first three values how long does it take to run how much memory in terms of if we if we just make very simple assumptions how much memory does compute require 10 spaced 10 units of memory 10 integers in a list right what if I only care about the first time how much memory disney's same right what if I care about all the elements how much memory does it use say this is what we often mean by eager the idea that this function irrespective of what you actually care about in this computation always takes the exact same amount of memory and the exact same amount of time it easily gives you the entire result and you're sitting there waiting for the entire result so you can see it gives me the entire result all at once this is undesirable because if this happens to be not 0.5 seconds but 5 seconds but I have to wait 50 seconds before I can even start processing the very first entry and so I'm sitting there blocked waiting for 50 seconds when maybe I could process these one by one also if this is not ten entries but a million entries maybe I'll need to care it maybe I only want to look at them one by one I don't need to have the whole history of all the entries I'm now spending gigabytes of memory just so you can easily give me the entire result set what I only care to look at them one by one and the moment I look at one of them I can throw it away so you can see this is both wasteful from the perspective of the time that it takes and wasteful from the perspective of the amount of memory it takes let's think of there's a better way to do this and let's think about this in terms of our object model let me rewrite compute as a class so it doesn't take any arguments I'll just dump this stuff underneath nothing very interesting here but more or less does the exact same thing really no difference between the two versions so I haven't gained anything here I haven't gained anything by seeing into my object model that said I've kind of seen this pattern before I've seen this pattern of I only want one element at a time and I'm going to get each elements and as I get each element I'll process it and maybe throw it away so basic looping construct the fact I see it even here in this looping construct here I don't get the entire list of once I only get each one individual element of the list and one thing that we might know is that in Python we see top-level syntax or top-level functions has some corresponding underscore methods that implement it if I told you that there is a pattern there's some if I told you there's some syntax for X and XS and I told you that this syntax actually looks like this more or less under the covers you can probably guess what the underscore functions we have to implement would be they had correspond to these two top-level functions here and here they'd be in ER and they'd be next so what that means is we can take a class in Python we can add an inner and our next method and suddenly that class can be iterated over well in the process of that iteration what could we do well anything arbitrarily including perform a long computation so what we could do and what the inner method actually needs to return you'll find out when you read the documentation it's not that important will just return self the next method just gives you each next method that you iterate over well instead of putting the computation in this Igor call method what if I did this what if I have the inter method say the last number I looked at with zero and I do this itself that last plus equals one if self stopped last is greater than and will limit this loop back to ten just to keep our live sane it's greater than ten raise the operation otherwise we sleep for 0.5 seconds and we return self dot last and we'll actually do this will be slightly different so we'll do it like this okay so there this will give us the same thing so now slightly different order but you can see now our next method computes what was the last value we looked at captures it the increments the last value we looked at if there's too many it raises a stop iteration we won't worry about what that means just yet it'll sleep corresponding to our long-running computation and or return the value now notice something different between this compute method and this next method this compute method returned a list of all the values this next method returned a single value and the way that you'd use it is as follows you just put it out you loop over and print it out and we'll actually see that here in practice you can see it's computing each value one at a time and in this loop here you can do something you want with it I print it out it still takes 50 seconds in total to print out the whole thing but you only pay for or still takes five seconds of total to print out the whole thing but you only have to pay for one unit of computation before you can start using that value and it no longer requires 10 units of memory because there's no storage anymore notice here there's no storage and here there's no list there's no storage well this really looks ugly and this is really hard to read just like this looks really ugly and this is really hard to read and there's a much simpler way to write it a much simpler formal expression for how to write functions there's a much simpler way to write a function that operates in this fashion and that's what the generator syntax is a generator syntax is merely something that looks like this and all this means is this thing here compute is something that can be iterated over each time you iterate over it this is the value that it will yield the stop iteration naturally happens because if you try and iterate over this too many times there'll be nothing left to yield because you'll have fallen off the end of this block here if state is maintained internally it's just a function that kind of doesn't really return eagerly you'd never want to write this formulation this is just such a pain in the butt you'd instead want to take the original function formulation and write it in its generator formulation when you look at the function formulation the generator formulation you squint your eyes you can see all the same pieces in the same places except in one of them you have storage and you have eagerly eager computation of the entire result set and then the other one you have no storage and you're simply returning values as you see them this is the core concept behind the generator and the core mental model behind the generator that instead of eagerly computing values you give them to the user as they ask but there is another similarly important and critical core concept to how generators work beyond just us and that's what I want to show you leading up into our last example we have all seen API to look like this run this first we've all seen api's that look like this which have methods that indicate the order which you have to run functions if you don't run this first first everything blows up if you don't run this last last everything blows up and in the documentation they tell you please run this method then this method then this method because if you do it in any other order all hell breaks loose unfortunately this API is kind of lying to you because nothing stops you from being a dunderhead so i don't like to work dunder and doing this nothing physically stops you from putting these in the wrong order other than that you fail to read the documentation and you'll see a break well one interesting thing about the generator formulation is that if you look at the generator formulation you can see it performs some computation then not only does it yield the result back but it also yields control back to the caller you can see here in this formulation lower case this I perform one computation one long-running computation modeled by this sleep and then I give the value back to the user to do something with and then they ask the next computation give it back to the user to do something with so I'm kind of interleaving the user code in the library code in the ego formulation the library code ran to completion gave the entire result set back to the user then the user did whatever they want with it in the generator formulation I have a little bit of library code run a little bit of user code run a little bit of library code run over the user code run and I interleave them this is the actual core conceptualization behind what the generators are built upon the idea of co-routines subroutines we can think of as any piece of executable code that runs from some starting point to some ending point to completion they have they have one single entry way in the Hat sorry they have one single entry point they have one single exit point and that's it they run they're done if you look at how user code and library code interact if the subroutine is in library code it runs the completion and then the user code has to pickup for the generator or the co-routine you enter the generator and as you asked for values the generator runs but you can have this nice interleaving where some user user code runs as the generator for a value the generator runs it's code yields back to the user code the user code does River at once goes back to the generator astronaut of alyou and so on so this interleaving of the two pieces of your code the user code in the library code here in this API what we actually have is similarly when we use this API we'd ideally want to have some interleaving because we if we could run all three of these methods all at the same time would never provided as three methods if we didn't have to have some code between these three calls they would never be provided it's different calls if they can all be done at once we just have one method here called do it that just calls first second and last and make sure that we can't screw this up so the reason that the API gives us these three pieces split into three parts is because I specifically intend us to interleave code they just want to make sure we interleave code in a fixed sequencing in a fixed order well let's think about this if I wrote this as a generator we could do that interleaving while maintaining the sequencing and this I would say is maybe even a more critical mental model for generators than this II girlís eager vs lazy divided the idea that generators are a mechanism by which you can create code that can interleave with other code and also enforce sequencing notice here that what happens when this generator runs is it'll run up to this point yield no value but will yield control back to the caller then the caller can resume and the caller can resume in this particular formulation first second and last will always run in that order you can't guarantee the last full run but you never can guarantee that someone could pull the plug before that runs but here if the API are provided in this fashion you could guarantee that the last method was never called before the first of the second method the generator forces that sequencing on you and it forces that sequencing on you because the generator is ultimately a code tune that allows you that's interleaving between two types of code user code and library code let's remember that and we'll go into the final of the examples we want to look at the context manager context managers are a very very simple metaphor and a common metaphor you see all over the place if you're a C++ user you've seen this metaphor before you've probably called it resource allocation is initialization something like that if you're a Java programmer you see in this metaphor before but I don't know what you call it in Java I don't know if Java has a real name or concept for this it's the idea that there's always some desire to do some setup and teardown and you want to combine them if you're a Python programmer and you've done even very introductory stuff with Python you've seen a context manager before and it looks something like this the most basic context manager you've seen before you open some file and the reason use a context manager here is that there's some corresponding setup and teardown if you open the file you got to close the file this is especially true on Windows because on Linux file lots are a little bit less aggressive on Windows if you don't close the file you might not be able to delete it later this is especially true if perhaps this file is backed by some storage where it's not necessarily automatically flushed if there's some IO buffering because definitely if you open the file on your right to it you want to make sure those are flush to disk in case the program terminates at some point with unflushed IO buffers you might lose things that you intended to be written to disk but fundamentally there is this metaphor here this idea that we have some set up action and some teardown action some initial action and some final action we want to match them together let me show you an example of that it goes beyond files but it can go even to things like SQLite databases so in SQL in SQLite we have these these file back databases the connect method is actually a context manager itself so here I connected some database I have some coarser on that database and let me just do this and I could do some statement so I could do something like create table points X int y int create so I have some statement to execute a table maybe insert some things into this table X Y values 1 1 there we go and then I have some statement where I could select for this there we go and we could commit this so here we have just very simple use of a context manager wrapping some database operations and what is it's not Kerr dot to commit we don't need that actually and what we'll do is we also want to make sure that we drop our table when we're done there we go until you can see the code ran I'll do two selects just so you can see everything you can see our code ran now you can see that the idea of there being some setup and teardown action it's pretty clear with the opening and closing of the database right obviously if you open the database and we write to it you want to close it well there's another setup and teardown action that you might not see here see if you can spot if we assume that we did something we want to have some corresponding action to undo it what's the what's the pair that you see here yeah I create a table and I drop a table now naturally if we were to really properly do this we probably do in the transaction the transaction would give us the right semantics but let's assume that for SQLite we might not have transactional support so we have to be in charge of that pairing that create and that prop and naturally it's a very very natural pairing you create something you drop it you want to make sure the both of them get done irrespective of some error pops up in the middle so how do we actually write our own context manager and you'll see that this will bring together everything we've looked at so far as I told you it turns out fundamentally in Python there is always some top-level syntax or some function and some underscore methods that implement it if I told you let's write it like this to X as X if I told you that behind the scenes this actually looks like the following x equals c TX enter try finally if I told you that and I don't know what the are there's some arguments that capacitor I don't care what they are so if I told you that's what it looks like you even know exactly what the underscore methods are they're right in front of your face right enter an exit so that's how we write a context manager we implement enter an exit now here you're going to see me really fumble because I don't remember any of the arguments that capacitor let's try this so we're going to create a temp table context manager and we know we need to implement enter and exit right and this takes some other arguments I don't really care what they are and we know maybe this temp table needs to take you know just for this thing to work we'll need to initialize it with a cursor okay and then the enter just execute this statement and the exit just execute that statement except this is self dot cursor and this is self dot cursor and that's it we have some enter code into Mexico and I just move things around to fit it into the protocol and then the next thing I do is I just have these over and I say with temp table curie calls with that cursor fair enough there we go so let's see that in play seemed to work let's run that again seemed to work let's run that again seemed to work if the table weren't destroyed then I would get an error there saying table already exists so clearly the entering the exit being called I can even show you that more explicitly this seems really really uninteresting so where's he going with this I wonder see there's the answer the exit pretty straightforward now I wonder why I brought this one up because context managers have such a clear and unambiguous metaphor behind them why would I take almost you know a third or a fourth of our time to talk about context managers once you see something kind of interesting here kind of interesting here can the exit be called before the enter should it nope so the enter should always be called before the exit right what does that tell you there's some sequencing what does that suggest to you generator because we just saw generators or about sequencing maybe I can improve this example by using a generator because here you can see there's an enter and exit the answer has to be call before the exit there's some sequencing necessarily that's the generators are all about so let's do it like this let's move this up here let's move this down up here we'll put a yield in between the two and here we'll just say temp table we'll call this context manager let's say temp table salt cursor will create an instance of this and then we'll just do next self dot gen I think that'll work and I'm going to turn itself I wanted to contact manager we have to do another next in here and then we have to just do one tiny thing here that doesn't really matter that much so there we go we see everything still work so we'll put our we'll put our little enter and exit print statements just to see them still running enter up here or watch you make this a little bit better we'll say created table print drops table just add our little locking statements in there just so we can see them in our code create a table drop table so we can see how this conceptualization of generators as sequencing is very useful now this is not sufficiently general for my purposes because you can see I kind of hard code what the generator is so let me just do this I'll just have it initialize what I want the generator what generator I want to use the enter notice I can I have a callable function here so what I'll do is I'll just add in a call here so I can get some arguments which will be the cursor or maybe the args and the quarks I just capture those on the enter I'll call my generator with the arguments that I was passed and I'll get my generator instance and I'll forward it here and here so now I can say context manager of temp table I can call the cursor there and then I have something that'll work now let's see if that works and then this has to return itself and then this should be self kwargs okay there we go so we can see this still works now you might wonder why did I go through all this effort well I want to show you something kind of interesting this line here looks really ugly so here why don't I just do this just move my code around a little bit take that line here and I'll just say temp table equals context manager temp table you can see what I kind of did was I took my generator and I wrapped it in some object they just kind of sit edan the parts of the generator to the enter in the exit right that's all I did and now I can just say with temp table of course circ right this will kind of mean you probably seen this like a couple minutes ago never saw that pattern before I've seen something like that anyone seems really useful Python syntax so you don't have to write that you take some object that happens to be a function you pass it to another thing that gives you something else and that reminds it anyone seen this tonight before I can tab back to the panel it's going to back what is it well what we do here why would we not write this because we have a simpler syntax locally right easier here anybody decorator so you can see it works without the decorator for we'd probably just write this like that right oops still works well it turns out all this garbage up here we don't have to write it's already been written for us it's in a library called context Lib and it's just a decorator that turns it's just a decorator that turns a generator into a context manager and the only things that we need to make this work completely is that here in this example I want to show you something this is an example that combines three core features of Python together generators context managers and decorators it doesn't requiring that you know a little bit of the details of how each of these work but fundamentally what we can see is what we have here is three features with very clear conceptual meaning orthogonal conceptual meaning where each of those pieces of conceptual meaning kind of fit together a context manager is merely some piece of code that pairs setup and teardown actions so the teardown action always occurs if the setup action occurred a generator is merely some form of syntax that allows us to do things like enforce sequencing and interleaving notice the context manager requires interleaving because it's set up is interleaved with the actual action you do in the block you do the setup then the action in the block and the teardown at the end there is that sequencing where the set up the enter has to be done before the exit so it makes sense to have a generator here as well finally we need something to adapt the generator to this data model that we looked at at the very beginning we have these underscore methods and we have to find some way to take how the generator works and fit it into those underscore methods one of the things we need to do in order to do that is we need to take this generator object to wrap it in some fashion that wrapping is part of the core of how Python works it's easy to dynamics and construct functions and have those functions wrap other functions that's part of the core language itself it's not really something that we could even lift the level of being a feature it really is core to how Python itself is written however there does happen to be a feature called decorators that allows us a nice convenient syntax for doing that exactly so what you can see here is an example of everything except metaclasses and how they all fit together and how some of these core pieces of Python fit together to write what you might consider to be more nearly expert level code what I can tell you is that in Python expert level code is not code that uses every single feature it's in fact not code that even uses that many features of Python it's code that has a certain clarity to where and when a feature should be used it's code that doesn't waste the time of the person who's writing it because they say to themselves I have this pattern Python has this mechanism I fit them together and everything just seamlessly and and very smoothly works it's code that doesn't have a lot of additional mechanisms associated with it it doesn't have people creating their own protocols it doesn't have people creating their own frameworks where the language itself provides the core pieces that you need and you merely have to understand what those core pieces are what they need and how to assemble them so over the last 90 minutes I've shown you a couple of details of how Python itself is designed and for features that you've probably heard of that Python is known for for features that you might associate with features that are expert with no or at least a you go and interview for a Python job and you say I'm an expert Python programmer these are four features that you're going to be asked about more likely the question here is does anybody remember any of the syntax they looked at probably not even I don't remember and I'm giving the talk and I have cheat I have notes here I have like a cheat sheet the real answer here is the notes the details they do matter they may matter very much you can't write the code unless you know the syntax the code won't compile it won't run however what is far more important and then the names of the arguments that get pass to exit or the specific fashion in which different numeric underscore methods get dispatched whether you do our ad or or ad depending on which is the most derived class all of those details are actually secondary to a core conceptual understanding of what these feature mean what these features mean so what I can tell you as the real lessons from this session or as follows Python is a language orientated around protocols there's some behavior some syntax some bytecode or some top-level function and there's a way for you to tell Python how to implement that on an arbitrary object the underscore methods that exact correspondence is usually guessable but if you can't guess it Google Python data model and you'll find all the different methods and all the caveats of their use Python is a very simple simplistic language in terms of his execution model code runs from top to bottom and things which would not be executable statements in other languages like class statements and function definitions or generate definitions are actually executable code in Python because they're executable code not only can you do things like hook into them but you can also do things like define functions within functions based off of runtime data define classes within functions based off of some runtime information you have how these impact specific features metaclasses metaclass is merely some hook in a class construction process because classes are constructed at runtime you can hook code in there and be you can hook into the creation of subclasses you can ask the subclass certain questions like do you have these methods implemented implemented that's what they do that's the mechanism but the meeting behind it is quite simple you have library code and user code when you sit on the library author side how do you make sure the users don't screw up how do you enforce a good straight from the library code to the user code well all that it takes is some way talk into the process of how user code user classes are instantiated that's merely with the meta classes you find that place where you add that hook you add the check that you want Oh make sure that you have the bar method implemented and you solve the problem in fact in the Python standard library there are regularize solutions to problems like that in collections of ABC there is an ABC meta class it allows you to use decorators to mark certain methods and abstract methods so you don't have to write the metaclass yourself and you can think this is a pattern of what the meta class is doing in fact almost every example of a meta class that you'll see will be this pattern of some base class and some derived class the base class trying to enforce the constraint on the derived class it's almost every pattern of the meta class that you'll see where that meta classes are justified of course you can misuse them to do all sorts of crazy things you can use them in places where it doesn't make any sense but in the justified uses of the metaclass that's what it's all about for the decorator it really incidentally that interesting just a little bit of syntax to allow you to write one line of code a little bit nicer to take one line of code that you write after your function definition and put it above your function definition when it hooks into this idea in Python that everything gets clear at runtime so that when you combine that with the ability to define functions within functions you can use this in order to wrap sets of functions with some before and after behavior if you're been around long enough you might think you might recall this was the whole aspect orientated idea but the idea that can I take a function and wrap it with some behavior such as timing you know capturing the time before capture the time after or doing things like authentication or logging anything where you don't have to go it's the function itself you just wrap the behavior around it generators are merely a way to take a single computation that would otherwise run eagerly from the insert from the injection of its parameters to the final computation and interleaving with other code by adding yield points where you can yield the temp the intermediate result values or one small piece of the computation and also yield control back to the caller in that vein you can think of a generator as being a way to take one long piece of one long computation and break it up in small parts where the computation can run a small sub sub unit of that computation but your user click and step in and do whatever it wants maybe it will use a partial value or use one of the first return values if you're returning a sequence of return values you might use the each of the individual values and in that fashion you can have greater control over how much of that how much of that computation actually runs do you run it all the way to completion or do need to run it to gab the first five values you need you also have greater control over how much memory is used because the generator can just yield the value back to you and you can choose which of those return values you want to keep which of the do you want to throw away context managers are merely some structure for allowing you to tie two actions together a set up action and a teardown action and make sure they always happen in concordance with each other if the set up action occurs make sure that teardown action occurs even if some error happens in the middle now when you think about these features and you realize they're related but mostly orthogonal you can combine them in interesting ways like you see in this example here where you have a decorator a context manager and a generator all coexisting and serving exactly it's one purpose as I said at the very beginning the syntax doesn't matter the details won't matter until you need to use them what I want to leave you with especially since a lot of you are not already very fluent Python programmers I want to leave you with this idea remember what these features are about remember what they're for that's something that you should be able to remember longer than the exact details of the syntax or even to details in fact the implementation details the details on syntax will change over time some of the flaws of the ability ssin make it fixed other details will be fixed another fashion but the core meaning behind these is something that will last and will guide you towards writing expert level Python code far better than memorizing just a bunch of features ever will so that's my presentation I hope you enjoyed it thank you very much for being a part of it any questions in the back a little louder please yeah and can I use like when I use yield it basically continues to do the earlier processing yes so so the idea is what you've what you've immediately jumped on is this idea that you can interleave what you've I think it jumped on to is the idea that well while I'm well I've returned control back to the calling context can I have the original context keep keep working believe look a genetic is kind of sit in the background and do some work and then by the time I go back to the generator I can go and ask it for some value it doesn't have to do anything because it's already in the background done that well yes you can you could have the generator start its own thread and maybe do things in its own thread that's entirely possible and that wouldn't be part of the generator syntax itself the generator itself is not intended for interleaving in a concurrency mindset and so where you see generators used as part of this broader idea of concurrency and Python is that the generator mechanism and its syntax have been further refining the Python three six to this asynchronous co-routine syntax there are many reasons why strictly simultaneous running Python code is something that has been avoided for a very long time in Python you must have heard of the global interpreter lock of the limitations of global interpreter lock accepting that is a limitation the approaches that people take in concurrency in Python when they mean single in process concurrency are typically either threads and they ignore the global interpreter lock or they just allow for strict interleaving of code where you know it's all single process so it's all single threaded one thread one one greenlit sleeps while the other one runs and you don't have strictly concurrently running greenland's that's closer to your answer which is generally no the generator is not for this kind of multi-threaded equivalent approach it really is for this one runs and this one runs and this one runs this runs and that's considered to be the preferred approach for concurrency in Python where the concurrency is i/o bound click yeah so you're referring to the runtime chicks versus otherwise yeah so what would be the other one would be the point in time the other acids or validations would happen so the problem is runtime checks require you to have the code deployed in a runtime apart so you'll never know the problem until you hit that exact code pad the whole point of this is can we any sooner that when it hits that code path in deployment in a deployed environment catch the earth any kind of run because Python doesn't have a very strong compiler has a very small very that compiler doesn't really do a lot it's going to be very difficult however what we can think of is that module import time is an equivalent to compile time and Python so just being able to import a module it's something that we can do before we hit the run time compile context we can do it at our laptop and if the module imports normally then we can say ok this code will be safe in production all of the checks that we looked at there or checks we can run by just sitting at our laptop importing all of our modules if it worked we'd say good to go another thing that we could do is maybe run a unit test suite but that might be one level deeper than these checks because you can think the unit test suite is going to take a lot longer to run and it might have some mocking of data and moving things around so we might have is some checks in the file itself import everything ok good to go then run our unit tests ok good to go deploy to production and then hope that somebody's online who knows how to solve the problem when something breaks quick follow one the unit subclasses they're still run by motors it's they're all they're all they're minute subclass is a runtime check but everything that runs at the module level is run at runtime it's just that there are different runtime scopes in Python so in its subclass we'll operate in the same fashion as the metaclass it'll merely just another place that you can hook into the initialization of a subclass and you have to look into the details I thought it took different arguments than it does you'll have to look into the details to see what arguments it takes and how you actually use it for the for this purpose there's a lot of these details of these features unless you're really hopeful you're never you're not somebody who's constantly using them and once you're constantly using them the details are things that are easy to forget but if you know where to look or you know what to look for those are details that you can very quickly pick up and frankly and it's the questions only been around since 3.6 a lot of these details do change over time but the core models the core mental models have not changed since these features were first introduced even the extension of generators for co-routines and for asynchronous programming the core mental models behind and haven't really changed since they were first introduced in Python 2 6 2 whatever any last questions ok well I hope you enjoyed the session thank you very much

Channel: PyData

Views: 364,806

Rating: 4.8752842 out of 5

Keywords: Python, Tutorial, Education, NumFOCUS, PyData, Opensource, download, learn, syntax, software, python 3

Id: cKPlPJyQrt4

Length: 114min 11sec (6851 seconds)

Published: Mon Jul 24 2017

Reddit Comments

[u/NicolasGuacamole]

I realise the title isn’t very informative but this is a nice overview of a few python features along with really solid use cases for why you’d want and ‘need’ to use them. It shows some of the philosophy of python and how that influences how the language is designed and how you can use it to your advantage.