What Does It Take To Be An Expert At Python?

I can tell you that what it takes to be effective in python is pretty straightforward to be effective in Python that requires a certain understanding of a couple of core mental models of the language and understanding of some of the things the language comes with like the built-in data types and the built-in functions a little bit of an understanding of what's available in the standard library this talk however is focused on the next step what does it take to be a little bit better than just effective what does it take to be good at Python or an expert at Python now I could spend the next two hours telling you in very philosophical terms what that means and I could tell you why Python experts do things in one way or the other we can even go very deeply into some of the meaning behind the Zen of Python but instead what I'd like to show you more specifically is a couple of what I call perhaps metaphors and programming a Python it turns out that as many of you know Python though it originated as something of kind of a scripting language does the convenience language to patch together or to orchestrate code that was written in C to write simple scripts has since grown into a full-fledged general-purpose programming language and along the way it's developed its own ways of thinking about certain core constructs in programming what I like to do in the course of this session is show you four features of Python and the ways that an expert would think about these features what I want to show you is not the syntax of these features the purpose of this presentation is not a feature parade it irritates me on endlessly whenever I see somebody who's writing a book about advanced Python and the entirety of the book is chapter one here's a feature chapter two here's another feature chapter three here's a third feature and the entirety of the discussion of these features is merely its formal characteristics Oh a generator has a yield instead of a return oh you can't send an on none value into it just started generator a lot of fundamental details of how these features work but very little talk about how we can conceptualize these features in the broader sense of what they mean for modeling core problems and so what I'd like to do is I'd like to talk about four of these features that you may have seen before and a couple of the core mental models for how you can think about them and how you can think about Python as a whole and wrap all of these together one thing that you'll notice is in addition to you can probably hear them a little bit stuffy a little bit sick that's only going to exaggerate my inability to remember the syntax or the documentation for basically anything so you may see me stumble but part of this presentation is to see me stumble and to understand that in cases where you don't understand or you don't have the documentation memorized or some of the corner cases memorized what really is more important for all of these features is an understanding of what they are and what they mean and not so much all of the implementation details those implementation details are of course critically important when you're using a feature but that's information that you can look up and refresh when you're actually in the middle of using that feature itself before I get started I'd like to take a small survey of small informal survey of the audience so how many of you here are Microsoft employees almost everyone how many of you are Microsoft employees that use Python as part of your Microsoft work how many of you are just Microsoft employees that have seen Python used in various areas of the company but don't really have direct interaction with it yet how many of you consider yourselves to be pretty okay at Python how many of you consider yourself to be really good at Python a couple people how many people are already in the back of the room thinking I should be in the front of them giving a talk myself maybe one person maybe one person in the front okay maybe maybe we'll have to show you something new when it comes to those of you who've seen Python how many of you feel like you're pretty effective in Python I can give you a simple problem you can write some functions maybe even write a class or two and you could solve the problem without too much sweat is that most of you how many of you are complete newbies and you're like I have no idea how you can get started with Python okay for those of you who are complete newbies you may struggle a little bit part of this talk does presume a baseline of understanding of Python however I don't think that the core lessons that you'll pick up here our lessons which will be totally lost on you there are a couple of core lessons which even those of you who are somewhat effective in Python already may yet have internalized and we might we might have a little bit of time to repeat those what I want to start with is I'll start with something really really really basic and I'm gonna start a little bit more basic than what I plan to just because I thought the audience might be a little bit more advanced of that so what I'll talk about at the very beginning is something that we might call the data model now what you'll see is I have all of the materials for this prepared and memorized in my brain and unfortunately my brain is getting older and older as time goes on and so I've already forgotten exactly what I want to talk about and so you'll see a little bit of roughness here but part of the purpose of this presentation is to present you with just the information that matters without a lot of the distracting details and so I will actually rewrite all of my notes for this as we go it's gonna be a little bit rough but hopefully it'll be an opportunity for us to focus just on what really matters and for us to put aside a lot of the associated implementation details and things which frankly it matters when you care about them but it's not part of that core lesson of understanding Python I want to start off with something very simple and you can see I created a directory called data model and I'll just create a simple file here I want to create a class and we're gonna do all of this everything in this course will be or everything in this session will be in Python three six it'll be in Python three six just for my own convenience just so that I don't have to worry about all the details that might differ between Python 2 and Python 3 there's no particular reason why I choose Python 3 versus Python 2 for the purposes of what I'm going to show you other than it's the Python that I happen to use and it's the Python that happens to be there's some political reasons behind why Python 2 or 3 but even skipping that it's just for my own convenience just to simplify things so let's start with something very simple I have some class I want to do some kind of mathematical operation I have some piece of software I write and I have some class here that represents a polynomial some simple you know linear polynomial with various coefficients and I want to represent it as a Python object and so I start with some code that looks like this right class polynomial doesn't have anything interesting it and I can create two polynomials very simply right and I might say p1 has coefficients 1 2 & 3 right so this would be x squared plus 2x plus 3 and p2 has coefficients 3 4 & 3 so this would be 3x squared whoops plus 4x plus 3 okay and I want to be able to do some simple things that we did in math class with these I want to be able to add them I want to be able to subtract them I want to be able to you know figure out some details of them the very first thing for those of you were relatively effective Python programmers is you're gonna look at this I'm gonna say why on earth did you write that in four lines when he could have written it in too light why did I write this in four lines here when I could have written it in two lines how would I compact my code here so instead of constructing my two polynomials and then initializing the coefficients I could initialize them all together what would I do I'm trying to get you to give me a little bit of feedback here so what would I do what am I missing here my class has nothing in its body what it's what isn't missing to allow me to do this a little bit more completely a little bit louder for me I needed an it method and so the very first thing when some of you look at this is you say hold on you're wasting our time here you probably want to have internet method that looks something like this so you can write these lines much more hoarsely and you don't have to have four lines when two lines would do all right so now we're happy whoops so now we're happy now if I rub my coat and I'll split my screen shrink this a little bit and I'll maybe print out actually I'll do this so I'll run my code and I have P 1 and P 2 and then somebody else looks at this and says oh my goodness print it out to the screen that looks so ugly what am I missing here missing another method what am I missing I'm missing a method that corresponds to what happens when I call this function at the top level repper to figure out the representation of my Python object yes or no yes so what method do i implement up here I have some method called repper right so I have some method here that returns a string that gives me the information gives me a printable representation of this class to the screen and so I'll make this very easy like that and so I'll run my code again and we'll see oh there we go those are our two polynomials pretty straightforward and we could write we could make this print out a little bit prettier we could see the x squared and the X and the constant term if we wanted to but you can see this is a lot easier than just seeing the name of the class and it's memory location and we can continue down this exercise because the next thing I want to be able to do is I want to be able to add these together and when I add them together nothing happens Python gives me an error unsupported operand types 4 plus polynomial polynomial but that doesn't make any sense I know I can add polynomials I did it all the time when I was in high school how do I tell Python how to add polynomials I have another method called add and in my add method I might return a brand new polynomial where I just give it all of course they're not cross ride just zip together and add each of the individual items so when I run that in my script here p1 plus p2 it gives me a polynomial with 4x squared plus 6x plus 6 now what's the pattern that we see evolving here for those of you who are off the top of your head you didn't you didn't remember which of the underscore functions there are for those of you who haven't ever seen these underscore functions before you've begun to see some pattern and what is that pattern I have some behavior that I want to implement and I write some underscore function some function that begins with two underscores and ends with two hundred scores and for those of you who are Python programmers already you might be familiar with these underscore functions you might call them dunder methods or double underscore methods or if you find that word abhorrent as I do you might call it by a different name data model methods and you might say it's an absolutely absurd thing to call this data model methods that's so many more syllables than dunder methods but there's a reason I call them data model methods which is I don't think I have internet access but maybe I do oh I do if you google for data model and Python the very first or second link depending on whether you use Bing Google or DuckDuckGo will be a piece of documentation here that tells you all of these methods and what they do but the pattern is a little bit more interesting than that it's more than just having all of these methods on a single piece of documentation it is that whatever we want to implement some behavior of Python we want to tell Python for this arbitrary object do this behavior give me something that is a principle representation for this object perform some notion of addition for this object we always see this common pattern and the pattern is as follows there is a top-level function or some top-level syntax and there's a corresponding underscore function the exact arguments that underscore function takes will be determined by whatever you find on this documentation page here the name the default names of the arguments you can pick up off this documentation here so for ad I called it self another that's just the conventional names for this the what each of these actually do you can read from the documentation but there's something more fundamental here there's a pattern what I want to add two objects I implement ad when I want to initialize an object I implement in it when I want to tell Python what to do when you call repper on an object I implement wrapper we know in Python there's a function called Len it gives you some notion of the size of an object what would it make sense for the size of a polynomial to be what what might that know should be any thoughts the degree right the highest degree so we want to implement this notion of size on this polynomial how do we actually implement it what's the name of the underscore function we have to implement the one hint is there's usually a very close correspondence here so what would it be here Len if we wanted the exact arguments that any of the caveats for how this needs to be implemented we could look at documentation there isn't anything special in the case of Len and we just return something like land of coefficients so this will tell us the size of the polynomial in terms of its degree and you can see these are both degree three polynomial the pattern that we see is as follows the Python data model is a means by which you can implement protocols those protocols have some abstract meaning depending on the optic itself in the case of a polynomial to add them means whatever that meant a math class in the case of a polynomial to find its size means whatever that would make sense for the case of a math class so it may make sense for us to know what the size this polynomial is in terms of its highest degree to get its representation it's principal representation has meaning that's given to us by the documentation the representation of any Python object is typically whatever string we'd have to type at the console to create another instance of that same object and so on in each case this protocol exists there is some underscore method that implements that protocol there are some top-level function like Len or repper or some top-level syntax like the plus sign or the multiplication sign that allows us to invoke that protocol and it all fits together in this hole we can think of this as a protocol in the sense that like many protocols when we implement something like Len we do that by delegating back to the protocol itself so notice Len is implemented in terms of Len being called on a constituent object ad is implemented by adding up some components namely the coefficients repper is implemented by calling wrapper on some component we see this as a very common pattern in the Python data model every time we want to implement some custom behavior on a Python object we do it by implementing an underscore function which ties to some top-level syntax or some top-level function and we implemented in terms of that thing itself so for example if we have some top-level syntax parentheses that coming after the name of an object and we might call that the call protocol we'd implement it by implementing the function called call we could look at the documentation to see any caveats or any special things we have to keep in mind when we implement that in the case of a polynomial this may not make a lot of sense perhaps to call it means something different maybe if we want to have some really horrible syntax where we say p1 p2 means multiplication we could do that I can't think of anything more ptosis than that but perhaps that's what we what we want but in any case we may implement this in terms of calling something else this is one of the core patterns that we see in Python this is probably one of the very first patterns that you'd see as a novice Python programmer in fact when you look at object orientation of Python there are three core patterns that you'd have to understand to really understand object orientation of Python this is one of them the protocol view of Python the second one is the built-in inheritance protocol and how it works where you go on a Python object to look for things and the third one is some caveats around how object orientation of Python works that's where I wanted to start off with and for those of you who are just kind of on that level of being effective in Python these are the kind of patterns that I would encourage you to look out for as you learn Python I want to go much deeper than this and I want to jump into a very tricky metaphor using a feature which you may have heard of and you may have been warned against something you may have seen kind of along the margins when you're using some library if you do anything with web programming and you use Django you may have seen this in the ORM or you may have heard people whispering about this one feature and I want to bring up this feature for you I want to show you that it has a very clear and concise meaning just to illustrate how powerful these mental models are for understanding Python itself and for driving expertise so let's shelve this for a second let's show this idea of Python having a protocol orientated data model and let's look into a new feature I'll create a new directory called meta classes and we'll talk about meta classes for a bit now the gist of this act this tutorial is actually leading you towards three features which are far more common and far more interesting the meta classes namely decorators generators and context managers there is a reason I want to start you off with meta classes it's a feature that everybody is warned against using they're told it's magical it makes no sense it's totally the most complicated thing in the world when in reality it is the case of a feature where if you understand what the feature is really about it is very clear and very obvious for why and when you use it and you can it's something that you can kind of shelve away in your mind as oh this feature does this this is why I want to use it I don't need to use it all the time so let me show you what a matter class or what Mehta classes are really about and I'll show this to you for an example now here's an opportunity for you to try something because I think you'll really appreciate this example if you try she let me do this in a different way i braid up py so I have two files here and I want you to actually try this and I want you to remember I want you to pretend that there are two groups working on some one piece of software one group is a core infrastructure group and they write library code and the other group is a developer group and they write user code the developer group takes the library code somebody else wrote and they use that library code to accomplish actual business objectives the core infrastructure group gets paid a lot of money to accomplish no actual business objectives but to provide library software that the rest of us can use isn't that the story of tech companies which which team would you rather work on nobody has any opinion whatsoever I'd rather work on the user team because I think that's where some of the more exciting work is and you're a lot closer where the money is actually being made so that can't hurt but some people like to be on the core infrastructure team because they think that the technical problems are often much more interesting than some particular business units problem around not getting data in the right format or who knows well let's say that this library provides classes and these classes are expected to be subclassed in the user code so I'll create a class here called base and I'll create a class here called derived that derives from base okay I've got five lines of code on the screen this is the basic pattern we see this pattern all over the place somebody writes some class in library code and somebody else has to use it I'm going to add two things one function on each side and then pose a question to you let's pretend you exist on the right-hand side of the screen you write user code you cannot write or alter or modify or in any way touch the code on the left-hand side of the screen you can use it but you can't change it it's library code you don't have source code access to it even if you did this is being deployed in some environment where any local changes you made wouldn't be preserved maybe in the deployment environment this is pulled down from NuGet or somewhere else so you have no way to change the code on the left-hand side of the screen your only control is code that you can write on the right hand side of the screen where could the code on the right-hand side of the screen break very simply where could this function break if there is no foo method exactly what can you do in your code to make sure that at least at the very minimum if the person who writes a library code removes that foo method your code fails not at runtime but at any point earlier than runtime one very simple thing you could do is you could write a test that just calls bar on your derived class very simple thing and make sure you run your tests before you deploy and you'll see the code fail sometime before it hits the run time production environment that's one thing you can do anything simpler you can think of take a minute and just try and think is there anything simpler you could add to this code to just make sure that this code has the ability to fail before it hits the run time production environment doesn't have to there's no there's not really that same concept of compile time in Python as there is in other languages but anything you can think of where you could just save yourself a little bit of effort if those core infrastructure meatheads accidentally screw something up and keep you going you know I was I was on a team iiiiii I often work in a vendor capacity working on very small very narrowly defined projects I was in a situation on Monday where there was a sev one issue and I was the only person on the team that was thought to be involved with it who was online somebody messaged me and asked me can you help me with this f1 issue I told him I have no idea I work on one tiny very small Python specific very narrow area but apparently I was the only one between between the manager and the top of the company itself who was online so I was sitting there just kind of tell trying to tell people sorry I don't know anything I'm hopeless let's see if we can avoid those kind of issues let's see if we can avoid there's anything we could add to our code so that you don't have to message me and hear me complain to you that I don't know anything and then you see how much I billed per hour you're like well why are we paying you so much if you don't know anything what could you do any thoughts I'll give you something really easy real easy oops what is this different what what's happening here here I check to see that that method exists if the method doesn't exist I assert the code will fail so all I have to do is make sure all of my modules can be imported which does not require a runtime environment I just import everything if for some reason they broke this this method here they gave it a different name they called it food instead I would see this code fail before this class is ever defined and I'd get exactly what I wanted I have some early warning that this had broken what we can see in general is that what I'm trying to do here is enforce a constraint I'm having the user level and force some constraint on the library level in other words the derived class is enforcing a constraint on the base class the drive class is saying hey the base class has to have these characteristics in order for me to run and be happy and if it doesn't have these characteristics in this particular case if it doesn't have this method implemented I'm gonna fail loudly and I won't run so that's pretty straightforward actually that was that was pretty straightforward and it turns out there's quite a few different ways we can do this this little approach is very simple very straightforward but let's flip the script a little bit let's say instead we have a case that looks like this let's say now you're the core infrastructure writer and you have to deal with those meatheads in the business unit actually using your code and abusing your code and misusing your code I have no idea what they're doing and you write your base class under the assumption that some responsible developer in the business unit will go and implement this bar method because if they don't everything falls apart now in this example you exist on the left-hand side of the screen not on the right-hand side of the screen you have no ability to change the code on the right-hand side of the screen even though you may be higher up in the technical organization you have no idea where this right-hand user code even sits your only ability is to put out library code for people to consume you have no idea who's even using your code you you even if you could see their code you might not be able to fix it let's ask a separate question but a very similar question to what we had before sitting on the left hand side of the screen how do you make sure that the user on the right-hand side of the screen doesn't screw up how do you make sure that in this case that bar method is in fact implemented because if it's not your code breaks and they're gonna blame you they're gonna call this foo method they're gonna see in the trace back that foo meant that is in your file and they're gonna blame you for getting this wrong so how do you work the other way how do you make sure they don't screw up we can't do the same thing we can't say assert has ADD or derived bar this doesn't make sense on the left hand side of the screen we'd have to move that to the right-hand side of the screen but we're not allowed to touch the code on the right-hand side of the screen so what do we do any thoughts turns out there are about three common answers to this one of them is metaclasses there's two other answers any thoughts where do we put the try-catch we could do try here that would be fine except this will only occur at runtime so if we try catch here it's not really any different than just running this function and seeing the error will still only see the error at runtime we won't be able to catch it before that code has production on July 3rd and they're calling me up to ask me to help out and I'm like I have no clue about they're like you're you're on a team that's that's involved with as your security and I'll say I have no clue anything about agile security true story really happens what can we do well if we think about that very first example we can see that our data model example actually goes much deeper than just that and the reason that we could call python a protocol orientated language is not just because the Python data model the object model has protocol orientated but that the entire Python language itself has a notion of hooks and protocols and safety valves within it I want to show you something kind of interesting one thing that a lot of people who come to Python from other languages don't realize is that Python is a much simpler language than you might appreciate for those of you who are originally C++ or Java programmers you have a mental model of your head in your head for how programming languages like those work and you have that mental model split between some compiled time steps and some run time steps and in that compile time step there's an enormous amount of complexity if you look at all the different optimizations that the visual studio compiler can perform on code there's an enormous amount of complexity there Python it turns out is a much much simpler language Python code runs from top to bottom linearly and almost everything that you see in Python almost every statement that you see in Python with the exception of two of them are actually executable runtime code in C++ or Java a class statement is not executable code in C++ it definitely is an executable code in C++ a class statement just says here's a bucket of bits here's how you named portions of that bucket of bits in Python a class statement is actually runtime executable code what does that mean well let's take a look what that means is that in Python I could do something like this define a class in a loop 10 times in a row totally the most useless thing I could ever do has no effect whatsoever but I can stick for loops outside of class bodies and redefine classes if I want has no use but it's it's meaningful in Python I can do the same thing the other way create a bar method under this foo object ten times only one of them will survive although the last one will call all the previous ones but Python accepts this as syntax because this class statement is fundamentally executable code and I told you and the direction when we started this was talking about Python as a protocol orientated language let's see something kind of interesting I'm just gonna create a function and only need this for wrapping purposes let me create my base class inside some function and let me use a function and in a module in the Python standard library called dis dis stands for disassemble let's disassemble this and see what actually happens in a Python bytecode what we see is something kind of interesting we see that there's this thing in the python byte code called load build class it's an actual executable runtime instruction in the Python interpreter for creating a class well hold on a second in the very first example I showed you I said there typically tends to be in Python some correspondence between some top-level syntax or function and some underscore method that implements that syntax our function there happens to be some top-level mechanism here it's not explicitly syntax it's not explicitly a function for building a class it turns out in Python there is a hook there is an underscore function that allows you to hook into and to do things with the process of building classes and I bet you can guess what it's called because just like the correspondence between repper and double underscore rapport Len and double underscore Len adding and double underscore add it's the same thing here there's a function of Python called build class it sits on a module called the built-ins so if I equals IPC so if I define a function here called or to it like this do I do this I'll capture the original build class I'll write my own build class I'll import from the built-ins and I'll swap them out here I can patch into what Python actually does when it creates classes and I won't do anything interesting in here I'll just print out all the arguments that are passed okay and so let's run our Python script one more time Python - I use her py oops and you can see here's the class that was created in fact you could even see I caught another class that was being built I was probably being built somewhere in the interactive interpreter itself but the most important one that you can see is I actually can catch could actually catch the building of this class and you can see there's some interesting things that I'm passed and passed the name of the class and passed its basis and I'm passed a function and I'm passed no other arguments well what I can do now that I can see the class that I'm building is I could hook into this and I could do something with it right I could add my assert here I could say if the actual arguments for this are and this is where I can't remember them the function the name and the basis function the name and the basis and I could say if the base actually it's the base if the base is equal to this base here or if it's actually this actual base object I could check if the barmouth is defined and when I run this you can see let's pass in and kW here as well what target mrs. take one - nope that third one's optional because not everything has to have a base class and [Applause] and who cares really this is a little this one you can see I struggle with this one a little bit cuz we never we don't typically do this we say if base is not none okay there we go and you can see we were able to have a little place where we could check if that bar method was defined this isn't typically what we do but it's to show you this idea of Python being a protocol orientated language is actually quite a common pattern and quite a fundamental piece of Python almost everything that the Python language does in an execution context like building classes creating functions importing modules you can find a way to hook into that once you can find a way to hook into that you can start doing things that you want to do like check is my user code going to break from the perspective the library author now the reason that I struggle with writing this function here is in part because first what's really the most important here is understanding that this pattern exists knowing that there are options for solving this pattern and also knowing that this is not the option that you'd ever use for solving this better build class is available for you to use but it's not how people actually solve this problem there are two fundamental features in Python that people use to solve the problem of enforcing constraints from derived class to base classes the first one is the metaclass meta classes are merely classes that derive from type that have some special methods on them that you'll have to read the documentation to understand all the special methods for but fundamentally allow you to intercept the construction of derived types the method that you care about here is called new and it takes three or four arguments basis body and the name basis in the body and we'll see where it gives us an opportunity to actually perform our check the basin's body you can see it got called right here with our derived class you can see this last argument for the drive class is the body of that class it's a dictionary with all the methods of that class you can see in it the bar method and so simply to get the behavior I want I could simply do assert bar or if not bar in body raised type or her bad user just like that this is the first answer to the core question in order to enforce the constraint the other way around if I can only touch the code on the left hand side of the screen and not the right hand side of the screen I can't just have assert lines all over the place I have to do something fundamentally different and the thing I have to do fundamentally different is I have to find a way to intercept how classes are constructed of which there are three common approaches build class which is the least common approach but works on the idea that Python is a protocol based language has these safety valves and hooks even in things as fundamental as the construction of any class second meta classes which are just the mechanism by which classes are constructed that's the approach that you see here and I'll show you the third one and third one is just a slight variant on the metaclass approach when people tell you about meta classes typically they tell you this is a magical feature you'll never need to use this feature it's overly complicated and the reason that I like to start this this session with that despite its yes being a very rare feature feature you'll very rarely have to use it's to show you that every feature of Python so far has at least one or two very clear and unambiguous mental models is applicable to one or two very clear and unambiguous metaphors for where it's actually useful in this case meta classes or how you can make sure that if you only control the code on the left hand side of the screen you can still enforce constraints on the code on the right hand side of the screen you can enforce constraints down the class hierarchy from a base class to a derived class this is one approach using a metal clasp that just intercepts the construction of the clasp and the second is even easier it turns out that in Python meta classes are have a bad reputation because even those people who understand where they're useful see that meta classes have a lot of complexity to them part of what I haven't told you at all is any of the syntax of the meta class or any of the arguments they take or any of the details for how the class hierarchy fits together because all of that doesn't really matter when you need to know that information you'll go look up and find that information but unless you know why you want to use them at a class in the first place it doesn't matter how many details of the feature you need to know once you understand where and why you need to use the feature then the rest of it are just details you look up when you start looking up those details you'll see that in many of the features I show you the story isn't that easy there's always caveats there's always catches and one of the catches is this meta class feature is very clumsy to use and so in Python 3 6 a new feature was used or a new feature was added called an it subclass and in its subclass I think it goes here it just gives you a method that allows you to hook into when a subclass of some classes being initialized and that's it and I think it goes I think it goes here let's find out if Brett were here you could tell you exactly what goes let's remove the metaclass for now [Music] I will do one thing if always up there okay so yes it got cold in the right place and then it's a class okay yeah so this gets called with the self so the other approach we do is as follows this was being called on that and with the knit subclass we have other hooks for figuring out this one would come in a knit subclass I have to remember what arguments it takes but you can look into it really frankly it does not matter that much so this is my simple story about metaclasses metaclass is our feature that were often told that are an expert level feature you're not supposed to ever use them you should shy away from them you should cover your eyes before you read code that includes metaclasses and it simply isn't that it simply is that for all of these advanced features there's usually one or two very clear metaphors to understand what that features all about and once you understand what that figures all about there's a lot of details you need to read through a documentation that's the approach I want to take for the remaining three features I want to show you the next feature is a very simple feature called decorators I'm sure some of you already seen this feature but I want to show you kind of where this feature comes from it's a very simple and very boring feature indeed and you've probably seen this feature before it's this little act that shows up on top of some function definition this little app with the dotted name so you see something like a dot B dot C and for those of you who who are very familiar with this feature you might wonder why you can't do this or why there are limitations to what you can put in the decorator line the answer for it is a really stupid answer there is no good reason but there is a limitation for those of you haven't seen this feature before I want to show you something about Python something very core and very fundamental about Python I told you when we talked about metaclasses that python is a live language there is no separate step that turns function definitions into bags of a set bits tagged and some elf binary or some PE binary somewhere that a function definition is actually a live thing it actually runs at runtime there's actually executable code associated with this def F okay so let me show you that in practice here's a function called add that adds two values okay let me create a little separation down here and let me run that and here I have my function add and I can add two values now notice the first thing I did in my terminal is I looked at the function itself and I got a return value you can see that the part that interpret can actually tell me where physically a memory this function exists and in fact this add function is an object and I can ask it all sorts of things like what's your name what's your module where you did where to find I can ask it let's give that a default value there and run this again I can say add what are your defaults and it can tell me what default values it had I can say what's your code like what code do you can do you contain this is actually the byte codes for this add function and you can see them right here I could ask it things like how many local variables do you have that's your name there's one more that's interesting for our names I can ask you what variable names do you do interact with one of the variable names of your signature x and y and do all sorts of things that interact live with this function well it's interesting that this function itself is a runtime object we kind of see this or you might be familiar with this if we're C or C++ programmers in C and C++ we can have function pointers but they're really just determined by the compiler they're just memory addresses the compiler knows for where it put the bag of bits that represents that function sits in Java for a Java programmer or a c-sharp programmer we might be familiar with runtime classes and functions under those runtime classes and Python it's closer to the latter but much simpler than that every Python structure that you interact with whether it's an object or a function or a generator has some runtime life has some runtime existence you can see it in memory you can ask it questions like what module were you I did were you defined in you can even ask it very useful questions like if you use the inspect module you can say what's your source code and it'll tell you the source code right back out so you can see you can ask the function to give its source code it'll give you a source code right back you can ask it what file are you in and it'll tell you what file it was to find it you can ask it what module we defined in or I think get line you can get get let's go what line number it was defined on get ask it all sorts of important questions well there's something interesting that you could also do let's say that I have some code here that calls my add function maybe it's part of some testing code or maybe it's some code part of some application let's give it one more example so you can see I'll add both numbers and also add strings okay simple enough let's say that I want to change my code very slightly let's say for some reason this add function isn't quite doing what I want or maybe it's doing what I want but I need to do something like time how long it takes to run because for some reason some of these are much slower than other things it takes a lot of time to run I want to time it and so what are my options well I know the time module there's a time function so I know I can say time and I can get the time right now when you do that again I can subtract them and figure out how many seconds it took and so how would i time these I'd say before equals time after equals time print time taken after - before like this and now I could kind of profile my code except this kind of sucks because every time I need to do this I need to add this in n places throughout my code I have to add in all sorts of places throughout my code there's something missing here there's something wrong here when we look at this example and we think about decorators in the context of the problem we're trying to solve here we see another core and important pattern in Python the Python developers really don't want you to be here today because it's the 5th of July and they'd rather you be home with your family spending time with them I'm very happy that you're here today in this session and I hope you're enjoying yourself so far but what I mean by saying the Python core developers don't want you to be here is this given the option between writing and absolutely complete forward compatible absolutely perfect solution to a given problem and writing the simplest stupidest quickest thing that gets the job done the core developers or the Python language would rather that you write the simplest stupidest quickest thing to get the job done and go home and spend the rest of the day with your family and the reason that you know that that's true is that when you look throughout the Python programming language there are a number of examples of cases where you can write the simplest and stew to get the job done and when the task at hand becomes harder or more complex or the requirements change you can change your code in a very simple fashion to linearly extend its functionality without having to rewrite everything from scratch this is a fundamental difference between writing code and python and writing code in a language like C++ in C++ there is a lot of upfront design that you need to do in order to be effective and if you make a mistake and something in your upfront design in C++ you're gonna be rewriting a lot of code down the line whereas in Python we'll see a number of different features are orientated around how we write the simplest thing today and then when the problem gets a little bit harder we have an avenue for making our code a little bit more complex here we can see a bad example of that because in the case where I want the code to be a little bit more complex I wanted to capture the timings for how long this function took in addition to the function output I've made my code substantially more complex I've added a lot of garbage and I've rewritten a lot of stuff and you can think the ratio of library code here to user code here it's usually a lot more user code than library code it's the whole point of writing library code in the first place and so to have three times to have to have to add code in three different places instead of one place makes for something pretty pretty gnarly so the first thing that you might say is well I could add that code in just one place I could say something like this capture my return value and do the time before and the time after print it out here like that and you you this would be definitely better than before definitely better than before I've added co2 one place instead of three places but if my library looked like this and maybe I had examples my life my library looked like that then I'm still really not in the best situation because I have to add all this tracking code to two places so it's got to be a better way and in fact my library could still be very big and so that still means I have to write a lot of code and rewrite a lot of code so it must be a better way now remember what did I tell you I told you that the pipe that Python program language is a live language that everything has some runtime representation so both this add function and this sub function have runtime representations the sub doesn't work on strings so both add and sub are functions what can I do well maybe I could write my own function called time right or timer and I take a function and I take the X and the y arguments and I apply this function like that so pull it off right and I do the timing before and I do the timing after and I print the elapsed time here and I returned the return value okay now I would just have to change these lines of code slightly I have to change that from a parenthesis to a comma and so on as long as I could account for the default value but I have to change each one of these lines as follows it's a little bit better not that much better but a little bit better I'll change the first three and we'll see it at work so you can see from time to time let's make this a little bit smaller so it all fits on the screen at the same time and you can see these three were timed the rest weren't that's still not that much better I can still do better well remember that everything happens live in the Python programming language so couldn't I just create a new function here couldn't I define a new function here that takes two arguments like this and in this function do the timing call the function that was passed in and return this function here and you can see this here is a wrapper that calls the function that was passed in and wraps it with some timing behavior before and after and printing something out couldn't I do that I can define functions at runtime I can define functions anywhere and then instead of all this nastiness changing all this user code I could just say ad equals timer of AD and some equals timer of sub and you can see it added that behavior across everything and you can see what I do that what I'm doing is I'm taking one function I'm wrapping with some behavior I'm taking one aspect or one piece of core or common functionality and I'm wrapping a bunch of different functions in a really simple fashion and all I'm doing is I'm creating a new function that takes the original function and wraps it with a little bit of behavior before and after and that's it and it turns out that Python wants to make this a little bit easier for you because this pattern of something equals call a function on that original thing that exists and that's what a decorator is in Python a decorator is merely syntax that's equivalent to the line that says this sub equals timer of sub that's all it is it's syntax that then fits into the ability to dynamically construct a function to wrap this behavior so this is what our final example would look like we would have started with code that looked like this we would have had to add only the timer code and then to decorators and we'd be able to very easily slip in this extra functionality we want without having to rewrite all this user code ourselves that's the core of what a decorator in Python is at its core it is very simple syntax to just allow you to write an ugly pattern sub equals timer of sub in a slightly nicer way and a slightly nicer place at the top but fundamentally it's about allowing you to take this wrapping behavior for functions and to wrap wide swaths of functions in one fashion without having to rewrite a lot of user code or having to even perform a lot of churn on your library code that's what a decorator is go ahead so one of the problems here I think you're identifying here I hard-coded the parameters right and I even hard-coded the default the real answer is this is where we use starts and start start quarks now this decorator works on any function that takes any arbitrary set of positional and keyword parameters and just forwards the one and so now this will work to wrap any function with any set of defaults in any fashion this is part of why starts and start start quarks are so powerful they're there for arbitrary for to write functions that can take any arbitrary parameter spec and forward it on in any fashion was that was that the core of what you're asking [Music] so you want to do something an N number of times that's easy enough to so what you want to do so say you want to have the ad just run three times in a row and you want to have the sub run twice in a row what you do we'll take off the timer decorator so we want to have some function that runs end times right we have our wrapper function in the middle that takes any arguments here and here and the question is how do we identify how many times we're going to add one more level to this what we'll say is we'll say that n times gives you a decorator so what let's let's actually let's start this very simply let's assume that we always want n times to be twice okay we only want to run this function twice so I'll say four I'll just say n equals two here for da in range of n are V equals function like that okay so here's our rapper they'll just call a function twice capture the value twice and return it and or turn our rapper and then we can do we can apply this here and here now we won't see this really happen we won't see we won't see the results here but if we wanted to you could you could do this so you can see it ran ad twice here ad twice here sub twice there now you want to generalize this because you don't wanna run just two times you wanna run at some number of times well why can't we just wrap this another function that takes the end and just return this like that right so now this is a function that takes how many times these are what they call higher-order decorators really not that interesting or that important it's a fairly simplistic extension of the idea which is if you could return one if you can have a function that returns a function then you can have a function that returns a function where that inner function then returns another function ad infinitum going as deep as you want to solve whatever problem you want there is a very important core concept that is hidden in here which is what you might call the closure object duality it's not something that we have time to look at in this session but it is a very important duality in Python but it's not one of the topics that we can look at but just to show you this in practice you can see this got called twice this got called five times so you can write your own end times decorator very easily as well and the core of it is just the idea that you start with the wrapper so this is just a wrapper function what you're replacing it with the decorator syntax allows you to do that replacing very easily and then in order to get this programmatic behavior you just add one level a function outside that constructs the decorator the decorator thing constructs the wrapper the wrapper and then wraps the function itself it is rare that you'll see a decorator deeper than this just because it's very rare that you have context there's there's there's typically not an additional context where you need to feed information in that requires you to have decorators deeper than this so that's a decorator decorators are probably the least interesting of all the functions were going look at or one of the features were to look at the next feature I want to look at is called generators now some of you may already know what a generator is generators are very interesting and very powerful feature of Python they're also a feature that include where I feel that mental model is often very limited what people understand in the mental model is often fairly limited there's more to it than just what you might call the eagerness versus laziness divide but let me start you off with something more fundamental than even that I told you before that there's always some top-level syntax or function and some underscore method that implements it so I told you if you have parentheses after something this is what's called the call protocol so there's some method call that implements it I want you to tell me what's the difference between this and this what's the Romans who add one and add two and I'll even create a terminal down here and I'll add one adds two numbers add two adds two numbers from the outside what's the difference between these two you wouldn't unless you started peeking inside unless you started saying what are you add one well you're a function what are you add two started started peeking on the inside oh you're a class wouldn't be able to tell the difference functionally there's no way for you to distinguish between the two in fact add one behind the scenes looks very similar to add two it's just that syntactically it's a whole hell of a lot easier to write so from the perspective of actually writing the code you're always going to want our proof you're always gonna want to you know do it this way now there is a fundamental difference which is if you wanted to start adding some stateful behavior like this right if you wanted to add that and clearly there's one way to do it for add for it for the adder and there's another way to do it for the function it's one way to do it for the class and we'll wait here for the function and this is the Afra mention when I hinted at this object close your duality it's actually a couple of different approaches you can think of for how you do this the core idea here is that fundamentally there is some very nice syntax and then some object model that everything kind of sits in and this is closer to what the object model itself looks like you can think that ad 1 is some object that kind of looks something like this although we'll remove the statefulness stuff just so that the two examples are at parity let's think about this in a different way let's think about some function that takes a lot of time to do something right maybe a function that goes off and performs network requests and it needs to load data from a database what would that function look like well it might have their words the resulting rows and it might say you know while DB read rows dot append something now this will be a little bit tricky for us to actually demo so we'll do a couple of simplifications just to make this code look you know kind of mimic what long-running computational code looks like but without actually having to use any fancy libraries so let's say what we have instead it's we have some code and it just sleeps for a half a second every time it provides a return value and the return value will just be a number from 0 to 9 let me ask you a question I will call this our I'll just call this compute [Music] whoops what I call compute how long does it take to run five seconds what if I only care about the first return value what if I only care about the first value how long does it take to run what if I care about the first three values how long does it take to run how much memory in terms of if we if we just make very simple assumptions how much memory does compute require 10 spaced 10 units of memory 10 integers in a list right what if I only care about the first time how much memory does use same right what if I care about all the elements how much memory does it use say this is what we often mean by eager the idea that this function irrespective of what you actually care about in this computation always takes the exact same amount of memory and the exact same amount of time it Gurley gives you the entire result and you're sitting there waiting for the entire result so you can see it gives me the entire result all at once this is undesirable because if this happens to be not 0.5 seconds but 5 seconds then I have to wait 50 seconds before I can even start processing the very first entry and so I'm sitting there blocked waiting for 50 seconds when maybe I could process these one by one also if this is not ten entries but a million entries maybe only me to care it maybe only want to look at them one by one I don't need to have the whole history of all the entries I'm now spending gigabytes of memory just so you can equally give me the entire result set what I only care to look at them one by one and the moment I look at one of them I could throw it away so you can see this is both wasteful from the perspective of the time that it takes and wasteful from the perspective of the amount of memory it takes let's think if there's a better way to do this and let's think about this in terms of our object model let me rewrite compute as a class so it doesn't take any arguments and I'll just dump this stuff underneath nothing very interesting here but more or less does the exact same thing really no difference between the two versions so I haven't gained anything here I haven't gained anything by seeing into my object model that said I've kind of seen this pattern before I've seen this pattern of I only want one element at a time and I'm going to get each element and as I get each element I'll process it and maybe throw it away it's a basic looping construct in fact I see it even here in this looping construct here I don't get the entire list of once I only get each one individual element of the list and one thing that we might know is that in Python we see top-level syntax or top-level functions has some corresponding underscore methods that implement it if I told you that there's a pattern there's some if I told you that there's some syntax for X and XS and I told you that this syntax actually looks like this more or less under the covers you could probably guess what the underscore functions we have to implement would be they had correspond to these two top-level functions here and here they'd be in ER and they'd be next so what that means is we can take a class in Python we can add an inner and a next method and suddenly that class can be iterated over well in the process of that iteration what could we do well anything arbitrarily including perform a long computation so what we could do and what the inner method actually needs to return you'll find out when you read the documentation it's not that important will just return self the next method just gives you each next method that you iterate over well instead of putting the computation in this eager call method what if I did this what if I have the inner method say the last number I looked at was zero and I do this hey so soft outlast plus equals one if self stopped last is greater than and will limit this loop back to ten just to keep our live sane it's greater than 10 raise up iteration otherwise we sleep for 0.5 seconds and we return self dot last and we'll actually do this will be slightly different so we'll do it like this sure so there this will give us the same thing so now it's slightly different order but you can see now our next method computes what was the last value we looked at captures it it increments the last value we looked at if there's too many it raises to stop iteration we won't worry about what that means just yet it'll sleep corresponding to our long-running computation and or return the value now notice something different between this compute method and this next method this compute method returned a list of all the values this next method returned a single value and the way that you'd use it is as follows you just print it out you loop over and print it out and we'll actually see that here in practice you can see it's computing each value one at a time and in this loop here you can do something you want with it like print it out it still takes 50 seconds to total to print out the whole thing but you only pay for or still takes 5 seconds of totals to print out the whole thing but you only have to pay for one unit of computation before you can start using that value and it no longer requires 10 units of memory because there's no storage anymore notice here there's no storage and here there's no list there's no storage well this really looks ugly and this is really hard to read just like this looks really ugly and this is really hard to read and there's a much simpler way to write it a much simpler formal expression for how to write functions there's a much simpler way to write a function it operates in this fashion and that's what the generators in Texas that generator syntax is merely something that looks like this and all this means is this thing here compute is something that can be iterated over each time you iterate over it this is the value that it will yield the stop iteration naturally happens because if you try and iterate over this too many times there'll be nothing left to yield because you'll have fallen off the end of this block here it's state is maintained internally it's just a function that kind of doesn't really return eagerly you'd never want to write this formulation this is just such a pain in the butt you'd instead want to take the original function formulation and write it in its generator formulation when you look at the function formulation in the generator formulation and you squint your eyes you can see all the same pieces in the same places except in one of them you have storage and you have eagerly eager computation of the entire result set and then the other one you have no storage and you're simply returning values as you see them this is the core concept behind it generator and the core mental model behind a generator that instead of eagerly computing values you give them to the user as they ask for them but there is another similarly important and critical core concept to how generators work beyond just this and that's what I want to show you leading up into our last example we have all seen API to look like this run this first we've all seen api's that look like this which have methods that indicate the order which you have to run functions if you don't run this first first everything blows up if you don't run this last last everything blows up and in the documentation they tell you please run this method then this method then this method because if you do it in any other order all hell breaks loose unfortunately this API is kind of lying to you because nothing stops you from being a dunderhead so i don't like the word dunder and doing this nothing physically stops you from putting these in the wrong order other than that you failed to read the documentation and you'll see a break well one interesting thing about the generator formulation is that if you look at the generator formulation you can see it performs some computation then not only does it yield the result back but it also yields control back to the caller you can see here in this formulation it's lowercase this I perform one computation while in the long-running computation modeled by this sleep and then I give the value back to the user to do something with and then they ask for the next computation give it back to the user to do something with so I'm kind of interleaving the user code in the library code in the eager formulation the library code ran to completion gave the entire result sent back to the user then the user did whatever they want with it in the generator formulation I have a little bit of library code run a little bit of user code run then a little bit of library code run and over the user code run and I interleave them this is the actual core conceptualization behind what the generators are built upon the idea of co-routines subroutines we can think of as any piece of executable code that runs from some starting point to some ending point to completion they have they have one single entry way and they have sorry they have one single entry point and they have one single exit point and that's it they run they're done if you look at how user code a library code interact if the subroutine is in library code it runs the completion the user code has to pickup for the generator or the co-routine you enter the generator and as you asked for values the generator runs but you can have this nice interleaving where some user user code runs ask the generator for a value the generator runs its code yields back to the user code the user code does whatever at once it goes back to the generator astronaut of al you and so on so you have this interleaving of the two pieces of your code the user code and the library code here in this API what we actually have is similarly when we use this API we'd ideally want to have some interleaving because we we would if we could run all three of these methods all at the same time the API would never have provided as three methods if we didn't have to have some code between these three calls they would never be provided as different calls if they could all be done at once we just have one method here called do it that just calls first second and last and make sure that we can't screw this up so the reason that the API gives us these three pieces put into three parts is it because I specifically intend us to interleave code they just want to make sure we interleave code in a fixed sequencing in a fixed order well let's think about this if I wrote this as a generator we could do that interleaving while maintaining the sequencing and this I would say is maybe even a more critical mental model for generators than this eager list eager versus lazy divided the idea that generators are a mechanism by which you can create code that can interleave with other code and also enforce sequencing notice here that what happens when this generator runs is it'll run up to this point yield no value but allele control back to the caller then the caller can resume and the caller can resume in this particular formulation first second and last will always run in that order you can't guarantee the last two run but you never can guarantee that someone could pull the plug before that runs but here if the API were provided in this fashion you could guarantee that the last method was never called before the first of the second method the generator forces that sequencing on you and it forces that sequencing on you because the generator is ultimately a KO routine that allows you this interleaving between two types of code user code and library code let's remember that and we'll go into the final of the examples we want to look at the context manager context managers are a very very simple metaphor and a common metaphor you see all over the place if you're a C++ user you've seen this metaphor before you've probably called it resource allocation is initialization or something like that if you're a Java programmer you've seen this metaphor before but I don't know what you call it in Java I don't know if Java has a real name or concept for this it's the idea that there's always some desire to do some setup and teardown and then you want to combine them if you're a Python programmer and you've done even very introductory stuff with Python you've seen a context manager before and it looks something like this it's the most basic context manager that you've seen before you open some file and the reason use a context manager here is that there's some corresponding setup and teardown if you open the file you got to close the file this is especially true on Windows because on Linux file locks are a little bit less aggressive on Windows if you don't close the file you might not be able to delete it later this is especially true if perhaps this file is backed by some storage where it's not necessarily automatically flushed if there's some IO buffering because definitely if you open the file and you're right to it you want to make sure those are flushed to disk in case the program terminates at some point with unflushed io buffers you might lose things that you intend it to be written to disk but fundamentally there is this metaphor here this idea that we have some set up action and some teardown action some initial action and some final action and we want to match them together let me show you an example of that it goes beyond files but it can go even into things like SQLite databases so in SQL in in SQLite we have these these file back databases the connect method is actually a context manager itself so here I connected some database I have some cursor on that database and let me just do this and I could do some statement so I could do something like create table points X into y int create so I have some statement to execute a table maybe insert some things into this table X Y values 1 1 here we go and then I have some statement where I could select for this there we go and then we could commit this so here we have just a very simple use of a context manager wrapping some database operations and what is it's not corrupt to commit we don't need that actually and what we'll do is we also want to make sure that we drop our table when we're done there we go until you can see the code ran I'll do two selects just so you can see everything you can see our code ran now you can see that the idea of there being some setup and teardown action it's pretty clear with the opening and closing of the database right obviously if we open the database and we write to it we want to close it but there's another setup and teardown action that you might not see here see if you can spot it if we assumed that we did something and we want to have some corresponding action to undo it what's the what's the pair that you see here yeah I create a table and I drop a table now naturally if we were to really properly do this we'd probably do it in a transaction and the transaction would give us the right semantics but let's assume that for SQLite we might not have transactional support so we have to be in charge of that parent that create and that drop and naturally it's a very very natural pairing you create something you drop it you want to make sure that both of them get done irrespective of some error pops up in the middle so how do we actually write our own context manager and you'll see that this will bring together everything we've looked at so far as I told you it turns out fundamentally in Python there is always some top-level syntax or some function and some underscore methods that implement it if I told you let's write it like this GX as X if I told you that behind the scenes this actually looks like the falling x equals c TX enter try finally if I told you that and I don't know what the Ark there's some arguments that capacitor I don't care what they are but if I told you that's what it looks like you even know exactly what the underscore methods are they're right in front of your face right and turn exit so that's how we write a context manager we implement and turn exit now here you're gonna see me really fumble because I don't remember any of the arguments that get passed there let's try this so we're gonna create a temp table context manager and we know we need to implement enter and exit right and this takes some other arguments and I don't really care what they are and we know maybe this temp table needs to take you know just for this thing to work we'll need to initialize it with a cursor okay and then the enter just execute this statement and the exit just execute that statement except this is self dot cursor and this is self dot cursor and that's it we have some enter code and some exit code and I just move things around to fit it into the protocol and then the next thing I do is I just had these over and I say with temp table curry calls with that cursor fair enough there we go so let's see that in play seem to work let's run that again it seemed to work let's run that again seemed to work if the table weren't destroyed then I would get an error there saying table already exists so clearly the enter and the exit being called I can even show you that more explicitly this seems really really uninteresting so where's he going with this I wonder see there's the enter the exit pretty straightforward now I wonder why I brought this one up because context managers have such a clear unambiguous metaphor behind them why would I take almost you know a third or a fourth of our time to talk about context managers watch do you see something kind of interesting here kind of interesting here can the exit be called before the enter should it nope so the enter should always be called before the exit right what does that tell you there's some sequencing where does that suggest to you generator because we just saw generators or about sequencing maybe I can improve this example by using a generator because here you can see there's an enter and exit the answer has to be called before the exit there's some sequencing necessarily that two generators are all about so let's do it like this let's move this up here let's move this down up here we'll put it yield in between the two and here we'll just say temp table we'll call this a context manager almost a temp table self dot cursor will create an instance of this and then we'll just do next self dot gen I think that'll work and then we turn itself oh wait to do context manager we have two another next in here and then we have to just do one tiny thing here that doesn't really matter that much so there we go we see everything still work so we'll put our we'll put our little enter and exit print statements just to see them still running enter up here or watch you make this a little bit better we'll say created table drop it's a table just add our little locking statements in there just so we can see them in our car create a table drop table so we can see how this conceptualization of generators as sequencing is very useful now this is not sufficiently general for my purposes because you can see I kind of hard code what the generator is so let me just do this I'll just have it initialize what I want the generator what generator I want to use the enter notice I can I have a callable function here so what I'll do is I'll just add in a call here so I can get some arguments which will be the cursor or maybe the args and the quarks I just capture those on the enter I'll call my generator with the arguments that I was passed and I'll get my generator instance and I'll forward it here and here so now I can say context manager of temp table I can call the cursor there and then I have something that'll work now let's see if that works and then this asset return itself and then this should be self kwargs okay there we go so we can see this still works now you might wonder why did I go through all this effort well I want to show you something kind of interesting this line here looks really ugly so here why don't I just do this just move my code around a little bit take that line here and I'll just say temp table equals context manager of temp table you can see what I kind of did was I took my generator and I wrapped it in some object that just kind of fit eaten the parts of the generator to the enter in the exit right that's all I did and now I can just say with temp table of course sir right this will kind of probably seen this like a couple minutes ago never saw that pattern before yeah I've seen something like that anyone seen some really useful Python syntax so you don't have to write that you take some object that happens to be a function you pass it to another thing that gives you something else and that rebind zit anyone seen this too nice before I can tap back to the panel it's only two back what is it well would what we do here why would we not write this because we have a simpler syntax what we write easier here anybody decorator so you can see it works without the decorator but we'd probably just write this like that right oops it still works well it turns out all this garbage up here we don't have to write it's already been written for us it's in a library called context Lib and it's just a decorator the terms it's just a decorator that turns a generator into a context manager and the only things that we need to make this work completely is that here in this example I want to show you something this is an example that combines three core features of Python together generators context managers and decorators it doesn't requiring that you know a little bit of the details of how each of these work but fundamentally what we can see is what we have here is three features with very clear conceptual meaning orthogonal conceptual meaning where each of those pieces of conceptual meaning kind of fit together a context manager is merely some piece of code that pairs setup and teardown actions so that the teardown action always occurs if the set up action occurred a generator is merely some form of syntax that allows us to do things like enforce sequencing and interleaving notice the context manager requires interleaving because the setup is interleaved with the actual action you do in the block you do the setup then the action in the block then the teardown at the end there is that sequencing where the set up the enter has to be done before the exit so it makes sense to have a generator here as well finally we need something to adapt the generator to this data model that we looked at at the very beginning we have these underscore methods and we have to find some way to take how the generator works and fit it into those underscore methods one of the things we need to do in order to do that is we need to take this generator object and wrap it in some fashion that wrapping is part of the core of how Python works it's easy to dynamically construct functions and have those functions wrap other functions that's part of the core language itself it's not really something that we could even lift the level of being a feature it really is core to how Python itself is written however there does happen to be a feature called decorators that allows us a nice convenient syntax for doing that exactly so what you can see here is an example of everything except metaclasses and how they all fit together and how some of these core pieces of Python fit together to write what you might consider to be more nearly expert level code what I can tell you is that in Python expert level code is not code that uses every single feature it's in fact not code that even uses that many features of Python its code that has a certain clarity to where and when a feature should be used it's code that doesn't waste the time of the person who's writing it because they say to themselves I have this pattern Python has this mechanism I fit them together and everything just seamlessly and and and very smoothly works it's code that doesn't have a lot of additional mechanisms associated with it it doesn't have people creating their own protocols it doesn't have people creating their own frameworks where the language itself provides the core pieces that you need and you merely have to understand what those core pieces are what they mean and how to assemble them so over the last 90 minutes I've shown you a couple of details of how Python itself is designed and for features that you've probably heard of that python is known for for features that you might associate with features that an expert would know or at least if you go an interview for a Python job and you say I'm an expert Python programmer these are four features that you're going to be asked about more likely the question here is does anybody remember any of the syntax they looked at probably not even I don't remember it and I'm giving the talk and I have cheat I have notes here I have like a cheat sheet the real answer here is the notes the the details they do matter they matter very much you can't write the code unless you know the syntax the code won't won't compile it won't run however what is far more important and then the names of the arguments that get pass to exit or the specific fashion in which different numeric underscore methods get dispatched whether you do our ad or our ad depending on which is the most derived class all of those details are actually secondary to a core conceptual understanding of what these feature mean what these features mean so what I can tell you as the real lessons from this session or as follows Python is a language orientated around protocols there's some behavior some syntax some bytecode or some top-level function and there's a way for you to tell Python how to implement that on an arbitrary object via underscore methods that exact correspondence is usually guessable but if you can't guess it Google Python data model and you'll find all the different methods and all of the caveats of their use Python is a very simple simplistic language in terms of his execution model code runs from top to bottom and things which would not be executable statements and other languages like class statements and function definitions or general definitions are actually executable code in Python because they're executable code not only can you do things like hook into them but you can also do things like define functions within functions based off of runtime data define classes within functions based off of some runtime information you have how these impact specific features metaclasses metaclass is merely some hook into a class construction process because classes are constructed at runtime you can hook code in there and because you can hook into the creation of subclasses you can ask the subclass certain questions like do you have these methods simply do you have these methods implemented that's what they do that's the mechanism but the meaning behind it is quite simple you have library code and user code when you sit on the library author side how do you make sure the users don't screw up how do you enforce a good straight from the library code to the user code well all that it takes is some way talk into the process of how user code user classes are instantiated that's merely what the metaclass is you find that place where you add that hook you add the check that you want Oh make sure that you have the bar method implemented and you solve the problem in fact in the Python standard library there are regularize solutions to problems like that in collections that ABC there is an ABC meta class it allows you to use decorators to mark certain methods and abstract methods so you don't have to write them at a class yourself and you could think this is a pattern of what the meta class is doing in fact almost every example of a meta class that you'll see will be this pattern of some base class and some derived class the base class trying to enforce a constraint on the derived class it's almost every pattern of the meta class that you'll see where that meta class is it justified of course you can misuse them to do all sorts of crazy things you can use them in places where it doesn't make any sense but in the justified uses of the meta class that's what it's all about for the decorator it really isn't anything that interesting just a little bit of syntax to allow you to write one line of code a little bit nicer to take one line of code that you'd write after your function definition and put it above your function definition but it hooks into this idea in Python that everything gets constructor at runtime so that when you combine that with the ability to define functions within functions you can use this in order to wrap sets of functions with some before and after behavior if you're have been around long enough you might think you might recall this was the whole aspect orientated idea but the idea that can I take a function and wrap it with some behavior such as timing you know capturing the time before and capture the time after or doing things like authentication or logging anything where you don't have to go into the function itself you just wrap the behavior around generators are merely a way to take a single computation that would otherwise run eagerly from the insert from the injection of its parameters to the final computation and interleaving with other code by adding yield points where you can yield the tech the intermediate result values or one small piece of the computation and also yield control back to the caller in that vein you can think of a generator as being a way to take one long piece of one long computation and break it up in small parts where the computation can run a small sub subunit of that computation that your user code can step in and do whatever it wants maybe it will use a partial value or use one of the first return values if you're returning a sequence of return values it might use the each of the individual values and in that fashion you could have greater control over how much of that how much of that computation actually runs do you run it all the way to completion or the only need to run it to gab the first five values you need you also have greater control over how much memory is used because the generator can just yield the values back to you and you can choose which of those return values you want to keep which of them you want to throw away context managers are merely some structure for allowing you to tie two actions together a set up action and a teardown action and make sure they always happen in concordance with each other if the set up action occurs make sure that teardown action occurs even if some error happens in the middle and when you think about these features and you realize they're related but mostly orthogonal you can combine them in interesting ways like you see in this example here where you have a decorator a context manager and a generator all coexisting and serving exactly it's one purpose as I said at the very beginning the syntax doesn't matter the details won't matter until you need to use them what I want to leave you with especially since a lot of you are not already very fluent Python programmers I want to leave you with this idea remember what these features are about remember what they're for that's something that you should be able to remember longer than the exact details of the syntax or evil implement implementation details in fact the implementation the details of the syntax will change over time some of the flaws in the implementation may get fixed other details will be fixed another fashion but the core meaning behind these is something that will last and will guide you towards writing expert level Python code far better than memorizing just a bunch of features ever will so that's my presentation I hope you enjoyed it thank you very much for being a part of it any questions in the back a little louder please yeah and can I use like when I use yield it basically continues to do the earlier processing yes so so the idea is what you've what you've immediately jumped on is this idea that you can interleave what you've I think it jumped on to is the idea that well while I'm well I've returned control back to the calling context can I have the original context keep keep working please look in any kind of sit in the background and do some work and then by the time I go back to the generator I can go and ask it for some value it doesn't have to do anything cuz it's already in the background done that well yes you can you could have the generator start its own thread and maybe do things at its own thread that's entirely possible and that wouldn't be part of the generator syntax itself the generator itself is not intended for interleaving in a concurrency mindset and so where you see generators used as part of this broader idea of concurrency and Python is that the generator mechanism and its syntax have been further refined in Python 3 6 2 this asynchronous co-routine syntax there are many reasons why strictly simultaneous running Python code is something that has been avoided for a very long time at Python you must have heard of the global interpreter lock and some limitations of the global interpreter lock accepting that as a limitation the approaches that people take in concurrency in Python when they mean single in process concurrency are typically either threads and they ignore the global interpreter lock or they just allow for strict interleaving of code where you know it's all single process so it's all single threaded one thread one one greenlit sleeps while the other one runs and you don't have strictly concurrently running greenland's that's closer to your answer which is generally no the generator is not for this kind of multi-threaded equivalent approach it really is for this one runs then this one runs and this one runs this orleans and that's considered to be the preferred approach for concurrency in python where the concurrency is i/o bound yeah so you are referring to the runtime chicks versus otherwise yeah so what would be the other what would be the point in time the other asserts or validations would happen so the problem is runtime checks require you to have the code deployed in a runtime environment so you'll never know the problem until you hit that exact code path the whole point of this is can we any sooner than when it hits that code path in deployment in a deployed environment catch the error any kind of run because Python doesn't have a very strong compiler has a very small very that compiler doesn't really do a lot it's gonna be very difficult however what we can think of is that module import time is an equivalent to compile time and Python so just being able to import a module is something that we can do before we hit the run time compile context we can do it at our laptop and if the module imports normally then we can say okay this code will be safe in production all of the checks that we looked at there or checks which we can run by just sitting at our laptop importing all of our modules if it worked we'd say good to go another thing that we could do is maybe run a unit test suite but that might be one level deeper than these checks because you can think the unit test suite is gonna take a lot longer to run and it might have some mocking of data and moving things around so we might have is some checks in the file itself import everything okay good to go then run our unit tests okay good to go deploy to production and then hope that somebody's online who knows how to solve the problem when something breaks quick follow up on the unit subclasses that still run time or is it's a they're all they're all there in its subclass is a runtime check but everything that runs at the module level is run at runtime it's just that there are different runtime scopes in Python so in its subclass will operate in the same fashion as the metaclass it'll merely just another place that you can hook into the initialization of a subclass and you have to look into the details I thought it took different arguments than it does you'll have to look into the details to see what what arguments it takes and how you actually use it for the for this purpose because there's a lot of these details of these features unless you're really hopefully you're never you're not somebody who's constantly using them and unless you're constantly using them the details are things that are easy to forget but if you know where to look or you know what to look for those are details that you can very quickly pick up and frankly and it's the class has only been around since 3.6 a lot of these details do change over time but the core models the core mental models have not changed since these features were first introduced even the extension of generators for co-routines and for asynchronous programming the core mental models behind it haven't really changed since they were first introduced in python 2 6 2 whatever any last questions ok well I hope you enjoyed the session

Channel: Coding Tech

Views: 1,583,974

Rating: 4.9015183 out of 5

Keywords: python, expert, development, software development, machine learning, python3, engineering, programming, coding, learn python, python tutorial

Id: 7lmCu8wz8ro

Length: 112min 2sec (6722 seconds)

Published: Wed Aug 02 2017

Reddit Comments

[u/[deleted]]

Watched this some time back. It was good, and I liked how he hammered home the point that you aren't going to remember every function of every library. That's something that beginners need to be made to understand too.

Also, I learned that I had no mastery of my editor, compared to this guy. lol

[u/Anu2008]

I'm not an expert by any means but after doing dozens of Python-based projects ranging from web scraping, database (MySQL and Google Cloud services), data analytics, mini-games, and machine learning, I've definitely gained plenty insights on Python and its amazing technologies.

Basically, to become an expert, you just have to dive right into the language and do all kinds of projects that come to mind. As you get more complex with those projects, you'll find yourself easily sifting through the official Python docs and stackoverflow, where the community is very much so thriving and teeming with knowledge you didn't know you should be aware of.

Also, holding a job of some sort helps too, since you'll feel some pressure to do more and be better.

Apologies for the lack of links and external references. I'm currently on my phone and wanted to type this up immediately.

[u/[deleted]]

Can someone tl;dr this video?

[u/Spleeeee]

I think the threshold is feeling literate in how to go about using python to glue anything and everything together.

[u/bVector]

I remember watching this video and kinda disliking the presentation for being a tiny bit patronizing or 'brogrammer-y' (is that still a word that is used?). Very possible that I was just in a bad mood or something when I watched it.

( rewatched a few minutes of it and AGHH dashes in filenames /dies )

[u/mcstafford]

Except that he was clearly adjusting as he went, he starts at an oddly low-level, given the title of the talk.

[u/nate256]

This is just a video about syntax and how the language works. Knowing this doesn't make you an expert. Just because you read a design pattern book doesn't mean you know how to use design patterns, in practice it just means you know of design patterns and how to use them on toy examples. I went in hoping for some good real world examples but instead got someone trying to show off syntax. You could get the info in this video from reading the docs.