James Powell: Advanced Metaphors in Coding with Python | PyData Berlin 2017

so then to talk about metaphors and Python programming focusing on some advanced Python structures I'm James Powell we're at PI data Berlin it's Friday June 30th 2017 I don't know that we have enough time to get into any introductions but if you want to follow me on Twitter I'm don't use this code on Twitter you want to email me I'm James that don't use this code calm I've been doing enough corporate training recently that I probably should change my email address because that might be sending the wrong message I'll actually try and show you in this tutorial some things you really should actually do when the circumstance suits you what structure wise what I'm going to do is I'm going to be sitting here in the front so you don't have to see my goofy face you can just watch the screen up there and I'm going to try and lead you down a very specific path to get an understanding of four pretty advanced parts of the Python language as quickly as we can as I said before this was originally a three and a half hour tutorial at PI data Amsterdam and we're going to try and compress that into slightly over ninety minutes I have the three of the four topics that we'll cover for a really nice story to push this idea that there are some core metaphors some ways to really think about and to fundamentally understand what these features are about that leads you to a greater understanding of how they can be used in Python and give you a strong enough mental model that you can start making reasonable predictions about what happens when you mix and match them or what you might be able to do in terms of modeling complex programs using these features but in order to really drill home the idea that there are some metaphors in Python because a lot of people I said this before but a lot of people they get really far with data frames and endi arrays and simple functions and what they don't necessarily realize is that when you're dealing with simple functions there are some core metaphors there even the idea of modules and functions something that's relatively new in terms of computer science if you go looking you'll find a talk by I believe was Leslie Lamport about how in the early 70s people didn't really know modularization meant it was a topic that people were beginning to think about but the very clear idea we have in our heads of what it means for a function to be a function or what it means for a module to be a module it's something that had to be discovered it's something that all of us benefited from in our education because we could just take it for granted just assumed that it meant exactly what just be able to intuit what it means but it's something that somebody actually to sit down and think about and I want to show you these are other parts of Python that have that so I'm going to start with one of the most controversial one of the most ridiculously controversial features in Python I don't know that I'd call it a feature but it's a interesting way that Python does things the future I'll start with is called meta classes and I'm sure you've all heard of meta classes and I'm sure all of you have heard somebody say don't use them they don't make any sense they're magical they're scary they're pointless there's always a better way to do it and that was more or less the genesis of this talk genesis of this talk was people looking at that feature not really fundamentally understanding what the feature was and then saying well I don't see how it's useful I'll tell you meta classes aren't useful for anything and everything clearly if you have a fish and you need to debone it don't use the meta class use a paring knife I think I'm not sure but use some sort of knife don't use a medic lass there is one very common problem that has many different manifestations where a meta class is one of two one of only about two common ways to solve that problem let's talk about those two common ways in order to do that I'm going to give you a very simple challenge so let me open up a vim and I'll put a split I'm going to have two pieces of code here and I want you to imagine something in your mind when you're imagine a very common circumstance in your mind there is some let's actually do this with a different one it's like this as library UI and we'll do a V split with user dot py so we have two files here library dot py and user dot py and imagine the very common circumstance where there's some library code that you can't change and some user code that you can change because you're the user this happens all the time you know we were working with open source projects but it's not guaranteed that if there's a bug in the library we can just go change that code without necessarily upstreaming it because you know this code may be deployed we have some mechanism like PIP and our own personal changes won't be pulled down so in many cases we have to work around problems in the library until somebody can put in a bug request or a pull request and fix it upstream in other cases maybe there's an organizational reason where some code is written by somebody else they control that code and we can't change it it turns out it's common enough that there's some code that we can change and there's code that we can't change so for the purpose of the next for the purpose of this one little mental exercise I'm going to present you let's imagine that you're the user the only code that you can change is on the right-hand side of the screen user dot py you cannot change the code on the left-hand side of the screen let's say that in library py somebody created a class and they called that class foo and in that class they had some method bar and it doesn't really do anything interesting okay doesn't do anything interesting at all let's say in user dot py you want to make use of the code that they had cuz this wouldn't be very interesting exercise unless you made use of their code so you say from library import foo and you want to subclass it you want to create your own class so we'll call it cooks and we'll subclass from foo and we'll write our own method here that's what bad is going to do is it's just going to return Baz plus self dot bar okay this is not this is this this may be so simple that you don't even see how common it is and you might have a little bit of difficulty picking out the pieces that are important the two pieces that are important are as follows one we're using somebody else's code by sub classing from it and two we're calling a function that we didn't define that was defined on the left-hand side here's the challenge how can I in some Python code make sure that if I'm running user dot py I get an error the moment I try to define my class i'ma call my class something a little bit easier we'll call it who bet I will call it derived because it will make the example a little bit easier and I hate saying foo and bar all the time there we go how do we make sure that base has a bar method that's the only thing I'm asking you this is actually a very easy thing to do but let's take just a minute you have a little bit of code to your terminal I've only written seven lines of code on the screen how do you make sure self dot bar exists because you don't control library dot P why because you don't control library dot py whoever wrote that could accidentally remove the bar method they could delete it and your code would break so how do you and your code make sure that their code didn't screw up how do you do it take a minute and think about that try writing some code maybe give me about a minute or two there's many ways to do this in fact for both sides of the question I'm going to ask there are many ways to do this not see why it might be important to determine this constraint obviously you know I'm the library author you're the user code author if I change this to bar I just ruined your day all your code breaks right so you're going to make sure that I'm on the straight and narrow you need to keep me honest and you can't change my code you can't say you know can't add lines to library UI you can only add lines of code on the right hand side of the screen to use it up UI so does anybody have an idea what we could do a lot of choices in the back what will I do just as simply as possible what do you think we could do we could check out our constructor so we could say def in it self if not has attribute self VAR raise type error or something like that we could do it in simpler right couldn't we just do this if base doesn't have a bar attribute now you know this is not perfect because bar might come into existence through many different means but if base doesn't have a bar attribute this code breaks and it breaks immediately this assert will fail and you'll never whoops and you'll never run the drive quote they'll just break right and it'll break on your end so at worst you'll get that error immediately not when you actually call and use the drive class another simple way is you write a unit test and in that unit test you just do and this is probably how you might do it you might have some unit test that creates a derived instance and calls deep-ass and you say well i just run my unit test before I deploy the code of production if my unit test passes it means my class is correct my class is correct it means all the hidden constraints that it has from all of its base classes are correct and that's what I do but in either case you're able to ensure something is correct before it actually runs in some production environment either by writing a unit test that actually calls it or by simply assert at the top of your code here there's a very easy thing to do in Python let's try to understand what's happening in this example we have two pieces of code library dot py and user dot py there is a constraint that needs to be enforced user dot py needs to make sure library dot py is written in some fashion so what you could say is that the user code is enforcing a constraint on the library code the user code is making sure the library code is written in a particular fashion let's back this up a little bit let's do the following so now I change the code a little bit the library code calls the Baz method it does not define it itself it assumes that the only time you'll ever use this base class is when you create your own class deriving from it and it assumes that when you do this you have to implement bass now you could say that the authors here might just implement some dummy one here like Ray's not implemented error or return something but let's just assume that for some reason it doesn't make sense for them to give a dummy implementation that they necessarily require that you give your implementation of bass because the problem doesn't make sense unless you do let's say that the roles are switched you're no longer the user you're now the library author so you only control the code that's on the left-hand side of the screen you don't control the code that's on the right-hand side of the screen as the library author you're providing code to users to use you need to make sure they don't screw it up you need to make sure hey if you're going to use my class you implement this method bass so the first thing you might do is you might say hey if you do not implement bass then I hate you you can put a little comment right but that doesn't really help because nobody reads nobody reads comments they read comments even the read documentation first before they read comments and nobody reads documentation so that's not really going to help much at all here's the puzzle that I give to you you can only touch the code on the left-hand side of the screen you cannot touch the code on the right-hand side of the screen how do you on the left-hand side of the screen make sure that anybody who uses your class on the right-hand side of the screen necessarily implements the method pass in other words how is a library author do you enforce a constraint on user code in other words how do you enforce a constraint in Python from a base class to a derived class we saw the other way around what's trivial it's ten different ways to do it I gave you two already how do you do it the other way around like this you say in its self if not has attributes self as do something yeah but there's no there's no guarantee that they'll call your net right anybody else use an abstract class hold on to that for a second because that is an answer but it's an answer that's further down the chain from what the answer I'm driving at is the answer I'm driving at actually is how abstract classes are implemented so does any can anybody go a little bit closer to to the language features it's a good answer that's how you actually do it using part of the standard library we can you will get a little closer tool to the language features they don't see this is a fairly common problem you need to give code to other people and make sure they don't screw it up who here has coworkers you're like oh boy we better be careful if we give them this code they're going to screw it up nobody okay a couple people in the back are they here who has co-workers who are here who are that way you're like please come to PI data you need to know how to do it the right way a couple people tell people I told you in the kind of warm-up that there are protocols in Python it turns out that Python has a couple of dominating characteristics if you look at how it's designed one of the dominating characteristics is protocols one of the other dominating characteristics is that everything is dynamic there's a very rich runtime and because there's a very rich runtime there's ways to hook into different steps of that runtime what I mean by there being a very rich runtime is that if you're a C++ or C or Java programmer and you see a class statement what do you think in the back your mind you think the compiler reads through this does some stuff off to the side magically and then I can use it Python strictly evaluates code from top to bottom on a script and every statement in both of these scripts are executable code the class statement in a Python script is executable code what does it do well in the parsing step the Python interpreter grabs all the stuff under the class statement turns that into bytecode and then creates a certain bytecode called create class that will create this class and so what you can see if I give you a Python terminal here is if I say class if I have a function or just put this into a scope that creates some class I can create a class in a function and if I look at the bytecode that's generated we can see what actually happens when that code is run from top to bottom Python does a load build class or it's not create class its load build class using all that information which means there's no reason in Python why I can't do this other than it makes no sense whatsoever I can just tell Python instantiate you know this class object ten times in a row you never do this makes no sense it's pointless but you can do it because that class statement is actually executable code it's actually the Python interpreter going out and doing something and the thing that it's doing is creating an object called foo that has a method called call that creates instances of that object subject to a lot of the details of how this thing works internally that should give you a hint because python has hooks more or less anywhere in its runtime you could think well hold on a second something is happening in executable code to create derived there's actually some process inside the interpreter that goes in and creates that could I hook into that and it turns out you can there is a method in Python called build class and it's part of the built-ins and build class is just some function looking at underscores before and after and you see how it kind of matches up to the build class opcode it's just a function that's called whenever Python builds a class as simple as that I don't remember all the arguments that it takes I think it takes we'll just do this okay and we'll just see what it passes what it returns well print that yep and we'll call the original build class now we'll just pass these on so let's see if this I don't know if any of the code that I wrote will actually work but let's check and see if it does Python user dot py and then let's do this oh we have to actually have our import from library import base and build class might have and I think bill class doesn't have a underscore here and it's like that I don't remember all these details there honestly none of these details are that important to remember maybe you have to stick it on the built instead okay think it might be like that one five here this one no no it's yeah it's here oh you know what it is actually I did it I got it right the first time never second-guess yourself and then this is important built-ins there's always some mechanics to this there we go so let me make this a little bit smaller you can see the whole code I overrode this build class you can see it's printing something out every time every time I construct a class we have some hooked and we can do some stuff there right so obviously we could just do notice the arguments are the class you're creating the name of it and the base so those are the arguments it's the class the name and the base or the basis actually so here we could just do assert or if not has attribute class Baz raise typer just like that and we'll see clip we'll see if that works there and you'll see that this more or less should work but anyway you can you get the idea the idea here is that I think this might be a oh sorry it gives you it gives you the it gives you something slightly different build class give you something slightly different it doesn't matter it honestly does not matter how this feature actually works what really matters is when you need to use it you look it up this is not the right answer in terms of how we do this but it should get you thinking there's some protocol and Python so metaphor in Python everything that you can do that has some runtime effect probably has some hook somewhere the hook for building classes is called build class you get in there and you do anything magic you want the real answer for this and what I wanted to lead you to is the idea of a meta class I want to show you where meta classes are important the way of meta class works is when a class is actually constructed first build classes called you don't usually hook into that and then the thing that actually constructs derived classes is the meta class itself and the meta class itself is just another class that has methods like new and instead of creating a new instance this creates a new class name and I know the arguments for this space isn't body and then here you can see and this has to derive from type and you can see here again it's just printing things out to the screen once when I create the my base class and once when I create my derived class and here I would just say if Baz not in body because that's actually dictionary raise type error and throw my error here and the idea here is the metaclass is one of the mechanisms you have for enforcing constraints from a base class to a derived class and that's all it is and almost every time you've seen a meta class in production it's been exactly this somebody wrote code on the left hand side of the screen and they had to do something to force you on the right hand side of the screen to not screw it up what the meta class is is it inserts itself into the process of building python classes and allows you to do whatever you want in this case check that something was implemented so that what something is implemented that something exists in the collections dot ABC module there is a meta class called ABC meta all it does is it gives you the ability to systematically do this to make sure that certain methods are abstract they must be defined to systematically do all of what I showed you there without having to write all the code yourself but if you think about it where a meta classes used meta classes are used in ORM object models why because they need to make sure that the class that you're constructing the derived class for that Oh RM object satisfies certain constraints for example that it's fields have certain layout to them if you think about it a lot of problems that you're running to fit this model very easily one of the areas where it really dawned on me what a meta class was is that we had in a production environment a bunch of tests running and all of the tests derived from a single common test based class and the reason that was the case is that our single common test based class we connected to a test database so all our tests would use test data make sense right it's in a big in a big bank one of the problems was we needed to make sure that that connection was made every time you ran a test because accidentally some people were running their tests and they forgot to derive from the right place or they actually know they derive from the right place but I forgot to call a special method the set up method in your unit test test class they've got called a setup method to make the database connection and their tests were failing because they were either connecting to a default database didn't have any test data in it they weren't connect into the database at all the problem as it occurred to me looks like this we have a bunch of classes in production that people have written we have one base class that maybe one or two people have control over how do we make sure all these other pieces pieces of code satisfy some constraint determined in that base class there were two solutions proposed to it solution one write a meta class just make sure if you derive from this test class you satisfy certain constraints you do certain things in a certain fashion we know exactly what you're trying to create we know the class you're trying to create we know it's layout we know what's implemented there we can add any arbitrary lines of Python we want to do that solution to was write a script that grabs - everybody's code to look for problems and then send them an email I'll let you guess which solution one it wasn't the metaclass so in the in the end we actually ended up just grepping through everybody's code to look for errors there was a script that ran in the background and it was just an entire mess because everybody would get emails every day saying your code is out of variants and then they'd send an email back saying I don't know what you're talking about and some of would say no we can need to get your boss involved you didn't write the code the right way instead of writing a 20 line metaclass that just says look you're trying to create a derived class from this base class if you don't do this this and this your code won't even execute it won't execute without an error and that's all the metaclass is and that's where it's applicable now this is not tied into the full story I want to show you the full story I want to show you are for a lot of features of Python that are much easier than metaclasses much easier to understand but the core here is to see that what you have is a very other piece of Python it looks by all appearances to be very complicated and come inscrutable but when you get to the core of what that metaphor is namely we have base classes and derived classes sometimes one needs to make sure the other ones written in a certain way sometimes the other one needs to make sure the first one's written in a certain way you can see you have essentially one or two common ways to make sure that a base class sorry from a base class that a certain derived class satisfies certain constraints there's actually a brand new way to do that in Python three six I'll let you look it up on your spare time it's called in its subclass it's a new method that just hooks into when you create subclasses and the reason it was added is that meta classes have some implementation details are a little bit complicated but as I told you at the very beginning the implementation details simply do not matter for most of you these are features that you will need to reach for when you see a problem that matches this until you need to reach for this feature you don't have to really put a lot of thought to it the only thing you should remember is where is this feature what's the pattern for this which what's the metaphor that this feature implies and if you know that then when you look at the details you'll see well actually the implementation is not perfect there's some weird corner cases and that will lead you toward something like a knit subclass which gives you maybe a cleaner implementation but until you get to that point really the only thing that I would care that anybody remember is the core metaphor itself because you can see I don't even remember I do this stuff all the time I don't even remember the arguments I don't remember the syntax I don't remember what the types that are passed back and forth are I have kind of have to guess have to play around but I know fundamentally what does this feature mean and I know when to reach for it and I know when it's appropriate and when it's not appropriate with that said let's take a look at some easier features and you'll see that same pattern of there's a core way to think about this and if you have that way of thinking about this feature right then the rest is just details details you don't have to worry about until this is something that you need to reach for and use and we will by the end of the 90 minutes that we have a lot of figure out what appearing nice is so we'll be able to we'll be able to stretch that analogy a little bit further let me show you a much easier part of Python the feature called decorators I'm sure you all use decorators before and you've probably seen them before I want to show you what they mean what they fundamentally mean and they tie into another important characteristic of Python the language oftentimes when people look at this feature the watermark for whether a talk about decorators is an easy talk or a novice talk or an intermediate talk is whether they talk about higher-order decorators and the bar for if your interview if you're an advanced candidate or a junior candidate is well you can write a simple decorator can you write a decorator that you know three levels deep and what I want to tell you is all of those are really just simple details that follow from first principles if you understand what a decorator fundamentally is here I have a file called deck dot py I want to show you something I told you that in Python everything is executable code the Python execution model is incredibly simple in fact it's so simple that you might even say that one of the reasons that Python code is very difficult to optimize in practice it's because execution model is so simple and so dumb just executing lines from top to bottom of a script and that's it so you saw with the class with a class there's some execution right there's a build class bytecode that build class bytecode actually does this work of making that class available for you to use same thing happens with functions if I have a function called let's call it add that adds two things together there's a bytecode for this and the bytecode for this is called build build function I think let's find out in front of the by coders so I called make function see I can't remember any of this off the top I have two grips I do this stuff I'm in the Python interpreter all day long looking at byte codes all day long I can't remember the stuff up top my head so if you look at this knee like there's no way I'll remember that's more than 2 minutes after you finish talking and by the way I'm already kind of asleep already don't worry I was told by the way the first time I ever gave a talk in Berlin this is a side but the first time ever gave a talk in Berlin I tried to include some jokes and somebody came up to me after the presentation and said you know the German audience they don't really care for jokes they just want to see code so just let me know if I can stop the joking if the joking is really getting in your way of appreciating the pure code I just want to make sure I have the audience right does anyone here not like jokes one person do and you don't like any form of humor whatsoever just show me the code I'm wasting your time here okay I have a function called add right there's some bytecode for making a function we can't really intercept that bytecode for making a function that's not possible as I said pythons got a lot of hooks in a lot of places but they're not everywhere they're usually driven by specific need if you said we need a hook to hook into the creation of functions somebody might ask you well why and it would be kind of hard to come up with a really good use case for that for some reasons we'll see very shortly but that said where I told you was this thing here creates some actual exit this is some actual executable function that creates an actual object let's see that in practice what I'll do is I'll run my script here with a - I flag and the - I flag will drop me into an interactive console with the result of the script so I'll show you what that means I'll say Z equals 10 if I do this I can actually interact with the results of the script after it's run so I can see all the things that were created this is similar to if any of you have ever used Idol this is kind of similar to what happens in Idol well you can see is this add bytecode this this add thing is actually an object it's a very rich object in fact a lot of info important information for this object just like any other object can be queried directly from this live so sorry add is just some function you can even see it sits at this memory address in case that's at all useful you know if you look at this this is the code for these are all the byte code behind it you know you can see that it has a default argument hey look that's the default argument has a little dock string you can see it there I bet you it has a name oh that's the name other thing right if I say sub equals add you can see it's just a very sorry if I do sub equals add you can see the name is something that is inherent to the object when it was first created it's part of the protocol for creating these functions and binding it to a new variable name doesn't change anything about that underlying function so there's a real function here right now we know from writing simple functions that when we have a function we could pass anything as an argument right so I could write a function called log and maybe it has one argument called f I wonder what that's going to be and it has two arguments X and Y and I just print out F that name x and y and I just return f of x and y remember this syntax all it means is F is some object that implements some method call go find that method implement it execute it and pass it the arguments X and Y so when I run this code here you can see ad is just something that has call implemented the call on it just evaluate the code inside the body of the function so if I do log of ad 10 and 20 you can see it it logs the call so this ad function very simple all it does is it prints out the name of the function the arguments that were passed and then actually executes it for us let's seize that is the result very simple I'm just passing to this function log another function X now you might say well if I actually wanted to do this at some logging mechanism this is a real pain in the butt because I'd originally had some code that looks like this 1 plus 2 equals I originally had a bunch of lines of code that looks like this and I actually want to add in a little bit of logging information to this code so that I can when I run the script it actually tells me some other information like what how many times this add function was called you know maybe this function is a little bit more complicated than adding two numbers maybe it makes a database connection sort to log it and if I actually wanted to use this little thing I wrote here to log this I might even do this the return value is that I'll even log the return value if I wanted to actually use this I would be a real pain in the butt right I have to go and to change this user code and this is something that we rarely want to do here's another core lesson about Python Python is somewhat rare in terms of the programming languages you might use that has as clear this idea of escalation pathways or whatever you want to call it the idea being this in Python you're encouraged to write the dumbest simplest code possible that solves a problem in front of you the reason for this is if you ever talk to the core developers the Python core developers and I've talked to many of them they're all very nice people and I want you to spend as much time with your family as possible they don't want you to be at work until 11 p.m. trying to figure out how can I make this code robust for a problem that nobody's even asked me to solve how do I predict what's going to happen you know 10 months down the line because I know that I have to rewrite all my code as something small changed about my architecture instead the Python programmer said we want to have a world where now people are well-adjusted where kids grow up with their families and so what we'll do is we'll make I'm sorry for the joke sorry about that what we'll do is we'll write a language where it's okay to do the simplest stupidest thing possible as long as it solves a problem and then we'll make it easy to take that simple stupid code and make it slightly more complex when the need arises you see this is a common pattern throughout Python there's often a very simple way to do something then a slightly more complicated way to do that and what happens is the end user code doesn't need to get rid of you try to minimize how much code you have to rewrite to slot in that more advanced behavior I can show you some example to that but I think we'll run out of time so if you're interested in that I can show you a lot of examples of those and they fit together very nicely in some steppings I can tell you about three or four different steps where you say this is a dumbest way to write it then when you realize you need this you make this small change then you realize you need something else you make this small change and the most complicated version of that code requires you to rewrite as little as possible and makes it okay for you to write the dumbest version to start with one thing I love about Python which is the actual code you might see in production to solve a problem looks very similar to the stupid code that you saw in some book or some tutorial online and that if your problem is a little bit harder usually there's a slight extension to that it's implemented I never saw this in C++ I programs people SS for many years and the code you'd see in a book look nothing like the code anybody actually wrote in production yeah amen so here the problem is we want to want to add a little bit of logging to this add function and it kind of sucks to have to rewrite this so what we could say to ourselves is you know what I know Python is a dynamic language I know that this def statement something that can be executed well when I have it in a module like it's executed from top to bottom well couldn't I create a function inside another function I mean a function just execute code right so I could just create another mod function in here so I could call this logged add right oops actually I won't call a log that's kind of stupid so changes very minimally from what I had before there we go okay so here not anything too fancy I just create a brand new function in here all this function does is it computes the return value of the original function I passed in logs everything to the console and returns this and I just returned this log function here and instead of taking the arguments up here this just takes the original function okay and I do this what did I do I wrote my function normally then I said take that function pass it into this log function which dynamically creates a brand new function that calls the original thing I passed in logs everything - so irritating logs everything to the screen returns the original value remember that if I rewind a little bit my original log function just had those same lines in slightly different order this is what the original log function looks like my new function just passes this in that's what it looks like now slightly different and then down here I just say add equals log add and you can see I have my little log and behavior here and here another common thing you might see about Python is Python programmers want to make your life easy and so when they start to see very common patterns arise they're usually pretty amenable to giving you simple syntax to make this easy so a decorator all a decorator is is whenever you see the pattern of x equals some function called on X where this X here is either a class or a def statement you just replace it with yes this here is merely syntax for writing this at the end that's it there's nothing more to decorators people like to talk about decorators as being a function that takes a function returns a function really all a decorator is is an app that proceeds a death that just means this ad equals log n that's all there is now as a metaphor the decorator itself does not introduce any new metaphors to Python it does it's merely convenient syntax to access existing metaphors where that existing metaphor is the idea that everything is an object including functions everything is executed in this linear fashion therefore you can create a function within a function return it reassign the name and go along with your day such that this end user code does not have to change that's all there is to that feature it's very simple let's look at a slightly harder feature generators now a lot of you have used generators but I find that a lot of people don't quite realize exactly what a generator is all about I told you at the very beginning in the warm-up that there is a close correspondence between syntax like this and some underscore method so I'm curious if you could tell me what's the difference between this what syndromes going to add one and add two if you're the user and you're not going to poke around the internals so obviously you poke around the internals you can see and don't say well they're written differently obviously if you look at the source code they're written differently but from the perspective of the user I told you that when you do something here with parentheses after it it just calls this special method here while you think about that let me get this on the screen for you and let me just pull this up in a Python terminal add one 10 and 20 what's 10 plus 20 nobody it's 30 add to 100 plus 200 what's 100 plus 200 type error now it's 300 it's 300 well I can't tell the difference in the two of these but from from an end-user perspective what's the difference I mean they're both things that have parentheses after them the add in fact if you first wrote this and then you swapped it out with this I'd never know the difference remember how I told you there's a really easy way to write things in Python that doesn't solve every problem and then a slightly harder way to write it that solves a couple more problems this might be an example of just that in fact let me make this a little bit fancier I'll show you one way where you can immediately see a difference are these different yeah every time I call add to this secret little number behind the scenes gets stuck in to the results let's rerun that again hope this should be self dot I mmm that guy gets got in there something interesting is happening here it's actually a way you can do it and add one using a closure we won't talk about that we do not have enough time to talk about closure objects equivalents but the idea being here that maybe if you needed to have some local state for this function this might be an easy way to write it you need to have some little counter that kind of takes a function that looks like it doesn't take any arguments and slip in an extra argument in there that you can manipulate let me start over I told you that everything in Python I didn't say everything that's too much but I told you a lot of things in Python there's some syntax and there's some corresponding underscore methods well if I say this if I have some code looks like this X for our for X and XS do something with X what actually happens behind the scenes I'll tell you behind the scenes internally if you look at the bytecode Python does the following so not literally but in a say figurative way Python takes simple code for X and XS and turns it into some structure looks like this I say not literally because it doesn't actually literally go in there and perform this translation but what I'm trying to express is that if you look at what actually happens when you call this this is what happens behind the scenes code that looks like this well look there's a top-level function here and a top-level function here okay there's two top-level functions you think you can override what they do of course you can what are the names of the functions that you implement to override them well just the word inner with underscores before and the word next with underscores before so let's think about that that means and I don't know I don't know I don't know that all the arguments that capacity well actually these don't take any arguments I don't know all the arguments could pass all the functions actually we'll call this simple and we'll initialize this okay I know I can I can fit this into my iteration syntax because I know behind the scenes in order to be able to iterate over something I just have to be able to call it or next all right this is going somewhere don't worry this is going somewhere so I could say for X in foo print X and you'll see it starts putting numbers to the screen it just calls it err it's something to do anything interesting at all and it calls next which justices let's think about this for a second it's kind of interesting right because notice what could go under this next function could be anything here we kind of think about this for loop as oh you're iterating over the elements in something but anything could go under this next Clause right anything that I could put into a normal function unbounded amount of stuff what if I had some really slow computation that I put here and we'll just represent instead of writing out a sloka slow computational just represented by asleep it's my representation for a slow computation you notice every time you iterate over this foo object it's able to perform any arbitrary computation including a slow one but it only performs the computation when you ask for the next element for example let's save this if I say for index X in enumerate foo if index greater than five break out of the loop otherwise print X what will it do it'll go through my little object it'll try and grab the first five elements as part of grabbing each subsequent element it will perform my slow computation ie my sleep and you can see that's one lot slow computation two three four five but if you think about it you needed to need to get six in us remember the enumerate starts counting from zero so that's why we had six things showing here but if you think about it this could be any very long computation that can maintain some state because it's attached to some object so like in the previous example where I added in self is some state I would get incremented and mutate that cut or affect the computation every time I could have that under this next and I'm only paying for the computation as I do the iterations let's start over see if we can kind of mix these things together so what I said was I can have some class and that class can have some internal state something that it keeps track of right I can put a call method on it and I can make use of that state right here I'm making use of that today instead of putting a call method on it I could put an interim method on it and a next method and do exactly what I was going to do in that call method except I can't take arguments but I could still make use of that state the only difference between this next method in a call method is the call method I could pass arguments the next method I can't pass arguments there's no there's no way in the syntax to put that in there not quite we won't get into how you actually do that but not quite and this is more or less equivalent to my kind of function written using call that's all a generator is when I write some generator like this this is just some really simple syntax to write something that looks like this and let me and in fact if you really squint can't you see all the common pieces I started this off by saying if you want to write a function you could write it in both fashions you see how there's a correspondence between these these are both essentially simple syntax to implement this generator formulation and this function formulation just very simple syntax to fit into the Python protocols that's all a generator really is people talk about generators they're merely some syntax to fit into this structure however unlike the decorator where the decorator was purely syntax the generator brings with it certain metaphorical implications and the metaphorical implications are as follows let's think about this for a second this call function may perform a slow computation right but every time I call this call function it only has the ability to sorry this call function may perform some slow computation say it's returning multiple values at every time I call this call function I'm essentially restarting that computation notice here I restart the computation from the top every time in this generator notice I don't restart the computation every time I yield I yield control back to the user the computation doesn't get restarted the general has a couple of implications one of them is ignoring that behind the scenes it's just some object protocols that are set up behind the scenes if this generator wanted to return multiple values like a list of values and this function want to turn for a list of values and they both did this by making use of some very slow computational function the functional formulation would be eager and the generator function would be lazy let me show you this with a common example with generators one of the most common examples that people use for generators is a generator that gives you values off of the Fibonacci sequence and the reason they do that is because its implementation is fairly easy to read looks like this and when you look at this in correspondence to the functional implementation you begin to see some of the metaphorical implementation metaphorical in locations here and then we'd use this same right here let's take let's squint to take a look at the two the differences here well notice the Fibonacci in the in the functional formulation had to allocate storage for the return value is going generator didn't return didn't allocate any storage why because the syntax for the two is slightly different next let's look at this notice that the Fibonacci sequence had to tell you how many values you are pulling out here it's pulling out ten values whereas the generator formulation just runs forever why think about this syntax the syntax for calling says go off do this formula do this computation give me the result it's eager when we call this we have some result equals fib like that and it goes off it calls this function it does takes all the cost of going through this loop ten times maybe I'll simulate some hard function it pays for all that cost upfront and gives you the result in the generator formulation if I mock up some long-running or complex computation in there notice we don't pay the cost upfront because by virtue of the syntax each time we ask for another item out of this stream we then pay that cost and so if we do not ask for many items out of the stream if we for example only ask for the first five we only pay the cost for the first five computations you'll see with these two very simple formulations of essentially the same thing that first metaphorical implementation generators are lazy constructs generators do not provide or do not perform computations until you ask them to where as functions necessarily have one entrance and one exit and between that entrance up here and the exit wherever that return is they have to pay the cost of every computation they do because they have no chance to defer that computation anywhere else whereas the generator does not have to pay that cost you have one entrance point and then you yield a value but you don't close the generator you don't stop you don't destroy the state that's there you keep that state around and if you want the next value you just resume from where you left off it's one of the fundamental metaphorical implementations implications of a generator there's more to a generator that is very important and this is something that most people do not see about generators generators are not fundamentally just about laziness versus eagerness there's one other piece you've all seen code that looks like this right and I'll actually make this a little bit so you'll see this you've all seen some class where somewhere in the documentation somebody said here's three methods on this class call them in this order or else if you call this last one here first things blow up maybe this one sets up a database connection this one closes the database connection and this one here does something with a database so if you try and close the connection before you open it all hell breaks loose in user code somebody probably then did this because even though the names are very clear this class is fundamentally lying to the structure of the class says you could call these in any order the names tell you something different but I'm sure we've all had a situation where you know what was said was not what was meant where the generators come into this think about this here we have a computation that's split into three parts that can be run in any order whatsoever here we have a generator that has three computations and remember a generator only ever performs the computation when you ask it to so the way you'd use this if you'd say something like this that's the first computation the second computation the last computation right can you call these in the wrong order in the generator no so fundamental things about generator is that they enforce sequencing generators are not just laziness versus eagerness but they are about the enforcement of sequencing because they allow you to break up a single computation into specific parts normally that computation is done in sequence because that computation is building some structure like the elements from or performing some operation in a natural order like what's the next element out of this Fibonacci sequence but there's no reason why the generator campus is performing arbitrary functions in unenforced sequencing that brings us to the last topic I want to talk about hopefully we'll get through this context managers there is a very simple metaphor for context managers for generators I know a lot of you may have missed why that last metaphor was important in this example you'll see all of this come together and you'll see why these metaphors understanding these metaphors even if you never remember the syntax until you need it it's so important many of you have seen the context manager structure like this let's do this let's make this cg x wy yeah go ahead now it's a little bit late how about this that better yes no you want another joke I'll tell you no the joke because that helps you here we go we've all seen context managers before you have them you have something looks like this with open some file name let's even open ourselves to x2i as f for line and f print out the lines right and the reason that we were told context managers are important it's because in a simple structure like this we ensure that if you leave this block the file gets closed many of us have seen the context manager in the context of things like database connections so you can say with connect test DB as connection cursor equals connection cursor corrode execute and then some SQL here you've seen people write this the reason people write this is they want to say if you open the database you want to close the database the metaphor should jump out at you context managers are all about how do we perform some action such that there's some corresponding setup and teardown and if we do the setup we're sure to do the teardown and search through all of the problems that you've worked on where if you set it up you gotta tear it down if you start it you gotta clean up a lot of different things look like that right I want to show you an example using this SQLite example that brings together all of the metaphors one of the things that we might do an SQLite database like this is create a temporary table what do you think we probably want to do after we're done using that temporary table when I drop it well look there's a corresponding setup and teardown the setup is create the temporary table the teardown is destroy the temporary table we want to make sure the teardown always occurs if the set up occurred and we want to make sure there's no way for us to easily avoid that happening what we also know is that in Python for every piece of syntax like with X as with CTX as X there's probably some corresponding underscore methods I'll tell you what happens here this more or less turns into the following enter try whatever was there finally exit exit there's no top level function for these you can see quite quite clearly what you need to implement in order to build your own context manager some function called enter and some function called exit I warn you I never remember what the arguments for enter and exit are I always have to look the documentation I know a lot about context managers but knowing the argument themselves just simply doesn't matter doesn't matter that much what knowing what it's for matters far more and what it's for is I do some setup and I want to do some teardown and I need to keep those closely linked and I need to make sure I don't accidentally forget the teardown let me show you how to implement a context manager that would create a temporary table in a database so maybe we create a temporary table with this cursor and we'll just hard code what that temporary table looks like and we'll say four row incur dot select select star from temp will just print out will just print out the results of this okay well I know need to have some class in a class will be called temp table I know it needs to take some cursor because I'll need to use that cursor to actually execute stuff against the database so I'll just save that I know it needs to implement enter and it needs to implement exit I do not ever remember what the arguments were exit is and you'll see why I don't but I don't turns out I don't have to because are not that important what do i do on answer on enter I might just do Kurd execute kurta execute create table temp and the definition maybe we have X as an int and why isn't it that's right I think and then exit we might just do card execute drop table temp okay pretty simple let's see if this actually worked I'm not I'm not convinced I got that one right oh yes thank you someone's paying attention because I stop telling jokes that's why and you can see our color changing because never works out perfectly oh this should be Kurata execute yep there we go and we need to insert let's do a little bit of insert so we did this will say curd I've execute insert into temp X Y values 1 1 so we'll do this get it add data and we'll say for X Y in zip range 10 range 10 to 20 there we go so we'll insert a couple of values and we'll just select them out okay there we go you saw the values got entered selected and you see every time I run this if there's a little error in this statement here by virtue of using the context manager when I run this again uptempo already exists so we've got to do the exit didn't get called let's do the exit kind of failed to get called correctly but ideally the exit gets called and I think it's cleaned up after itself that's the idea sometimes it works sometimes it doesn't I want to show you something kind of interesting you see the enter and exit have a sequencing to them the enter always comes before the exit what structure in Python is good for enforcing sequencing that implies a core metaphor of sequencing generators that's kind of interesting right so maybe I could take this in this statement and stick them into a generator and maybe the generator will take the cursor assets as it's a argument and call this one in this one and put a yield in between right enforcing sequencing and now my temp table is now kind of fancy because what it will do is it might say self dot G equals an instance of this generator and it might just do next of my generator here and the next there and I might too actually it is a little bit you got to do it like this of course the details are always a little trickier but self talk to equals self dot equals g TT of course and then I actually have to do this these are all details that simply do not matter ok there we go and you can see this thing works and all I've done is now this thing just take some generator and it puts it into the context manager protocol and my generator here just enforces the sequencing of creating the table and tearing down well let me call it something better like that and I won't take this argument here I'll just move that into here and I'll leave the rest the same so the in it doesn't need to take anything actually I'll take the generator as my argument and I'll just do this I'll call it GI for generator instance haven't done anything too magical now this thing doesn't really reference it it's ever a temp table I'll call this thing up here temp table you can see it enforces the sequencing and then I'll just do this temp table equals context manager tempt able to turn my generator into something that fits into this protocol right I just pass this in here as its argument right I take the G I capture the generator that was passed in on enter I capture the cursor I advance through the first argument here on exit I advance the next argument and I'm done and we got a call to make sure we spelled these things right context manager and then this oh oh my god oh yeah there's a little trickier than that but you'll see let's do this let's stick it here little trick is always there's always a couple more details than you think but the details simply don't matter what matters here what fundamentally matters here always more details than you think but simply matters here is that what we have here is this context manager thing this generator that enforces its sequencing and this line here that turns our generator into a context manager right now let me fix the bug that I had in it originally and the bug is that this needs to take a cursor here so I want to show you something kind of interesting notice that this thing now has a call method because it's it's called so we'll bind the cursor and this call method here it's what we're missing and then this will work and will turn itself okay so now this still works now there's something really interesting here I'm just going to move this code down to the bottom do you see a pattern here ever seen this pattern before something equals call a function on something and that original thing was a function never seen that pattern before what was it called decorator now our code still works and it turns out my sloppy crappy version of this that I wrote we don't have to write it already exists in context Lib here we go here we have an example of a context manager a generator and a decorator all together in one place what you can see is each of these features has a fundamental metaphor behind it one of the fundamental metaphors of generators is enforcing sequences which is important because we have to do that create the table before we drop the table we fit this into a context manager the fundamental metaphor of context managers is some teardown some set up in some teardown that are paired together we have that here as well we got a create the temp table and if we create it we got to make sure we destroy it the last piece is a little bit of syntax because we want to fit this structure into the protocol a Python protocols enter an exit and the way that we do that is by calling a function that takes a function or a generator and we have our decorator syntax here do you see how it didn't really matter in order to get the code to actually work we asked to futz around a little bit we had to move lines around a little bit confusing probably would have helped if we had read through the documentation beforehand probably would have helped if maybe I warmed you up with a few more jokes I don't know but it doesn't matter if fundamentally doesn't matter when you need to know the syntax you'll go look it up what you might never be able to find from the documentation is what do these features actually mean what are they actually for who cares why do people care about them and I think you see in the course of last 90 minutes I showed you four core parts of Python and I told you each of them really fundamentally means let me review metaclasses they might mean many different things and they might be used or misused in many different places but fundamentally they are your core or primary way to make sure that if you have a class and someone's going to derive from that class that they do something you want them to do or don't do something you don't want them to do how do you do that will they give you an area where you can write arbitrary code to check their work when they try to create their derived class you have a little function that you can hook into and you can say hey you didn't implement this method I wanted this method I marked as abstract you didn't implement it this method that you wrote I'm going to wrap it and change its behavior slightly that's all it is meda meda class is just in that split-screen we're not on the right-hand side anymore we're on the left-hand side and we need to make sure people who use our code on the right-hand side do things in a certain way by forcing ourselves onto them by means of this hook all the meta classes decorator merely syntax that makes it convenient to use a core aspect of Python namely that everything is an object and everything is executed in this linear fashion from top to bottom including things like class statements or function or generator definitions because they're dynamic because they're executed at runtime it means we can create classes within functions create functions within functions because their executor means we can pass a function to other function as an argument and the decorator is merely a convenient syntax to take this common pattern we see in code make it easier to use the generator is merely a way to very easily write something that can be iterated over where the key part is that on each iteration what you do might be some interesting computation as opposed to just looking at the next element of a list or the next element here you can think fundamentally the easiest way to bridge the two is that usually when you think about it eration you think about getting the next item from some sequence what if that sequence is a row in a database getting the next item is a complex operation it's a network call you have to go on to the network send some information the database wait for it to come back a generator is not how those are typically implemented they're typically implemented in terms of itter and next that we saw before but you can think there's a similarity between what is done there and the core of what a generator provides you but one side effect of that or one consequence of that is that generators also fundamentally enforce the sequencing of operations because when you yield from a generator and you resume you can only resume from the point where you last left off finally a context manager as a simple metaphor the simplest of all do something to set up some state or some context and make sure you clean up after yourself afterwards if you open the cupboard to get a cup make sure you close it I never do that you come to my house now that we're all friends if you ever come to my house and I make you a cup of tea please remind me to close the cabinets because I seriously leave them open all the time and please also remind me to unplug the because it's really a fire hazard ever right next to my toaster only there were context managers for real life but they're not well I guess they are that would be like a housekeeper or something okay we're getting too far off track fundamentally a context manager is this pairing of setup Ontario and the example I show you here you can see all of them coming together the syntax for them of course you saw I struggled a little bit to get it right I struggle to remember all of the arguments but it doesn't matter when you see each of these independently you can now begin to think well I want my temporary table to be created before it's dropped that sounds like sequencing makes sense for that to be a generator I if I'm going to write as a generator I need to fit it into this protocol so I need to have some class in the middle in order to pass my generator into that class I need some syntax that looks really similar to the decorator syntax and there we go all the pieces fit together in one and they fit together fairly seamlessly I'll finish this example there's one last thing I have to do to make this correct and here this should stand as a complete and correct example of how to create a context manager using a decorator and a generator unfortunately I can't take a medical ass in here as well or I could but it would be gratuitous but you see there are some metaphors behind the scenes should you use these features all over the place no of course not but now I hope that all of you will understand that the details of these features are important when you need to use them and if you find that you need to use them you'll have to look at the details and say is this something I can really use or not but I hope that all of you especially those of you who aren't commonly using these parts of Python every day at least have a starting point where you can say hey this problem problem looks like something I've seen before it's probably a good use of a generator this problem looks like something I've seen before it's probably a good use of a meta class and if you can understand that and really fundamentally Intuit how Python was designed and what these features are really for the rest is just details what you look up when you need to so those are the four things I wanted to go over I'll open for questions I don't know how much time we have left since we're schedule is a little bit different but any questions we have Timmons that's good in the front first what's your name Stefan how you doing Stefan Stefan has enough friends he doesn't need another friend apparently oh I can be Stefan's friend fantastic I just wrap their function so when I got there derived class yeah so the question is I brought up an example where I use them at a class not to check something but to enforce some behavior the way it is in the meta class I get the class you're trying to construct right I can look at it and make sure that it satisfies some some requirements that I have but I can also change it and so I can just take any method you write and I can wrap it using the same trick I did with the decorator with my own method that does the things I want so that was my solution my solution was don't bother to even have them call the special database setup method we'll just take their method today right and wrap them ourselves and maybe behind the scenes check the environment to see if the task needs to be done or not there was another question what happens in this case the question is how can yield ever throw an exception what happens in this case is that when you when you you need to be able to capture that exception if it occurs because you might need to do something with it so what happens here is that generators are a lot richer than just next they have two other methods on them one of them is called send the other ones called throw send allows you to do something a little bit more sophisticated with generators and here just as a brand-new metaphor that brand new metaphors what's called a KO routine what we saw with a generator versus a function is that a function starts at one point runs to the end with a generator it starts at one point and then you can kind of enter and re-enter how do you stick information back in every time we enter that's a generalization from a subroutine that runs take some arguments and run to completion and a KO routine that takes some arguments run to some point pauses as you to pass in more arguments pauses allows you to pass in more arguments pauses by the way co-routines are very important part of asynchronous programming in Python the best asynchronous libraries in python namely curio and trio well ok the ones I like the most maybe not the best but the ones I like the most are built all around the idea of coroutines well that's what send is for what's throw for throw allows you to throw an exception back up into a generator what happens in this context manager adapter is that when an exception occurs inside the context the context manager knows it because those are what the arguments to exit were the ones I can't remember the name of it's like exception value exception type things like that it'll take that and we'll throw them back up into the yield and the exception will be percolated on line 10 here that's how a yield can actually raise an exception it's not being raised by the statement itself it's being reinfected into the generator from the outside to really understand that you'll have to look at how co-routines work because i think they're very closely tied other questions and you want to ever get anyone actually going to use any of these features when they get back to work who wasn't going to use them before nobody so everyone's going to use them before and they're going to keep using them so I didn't scare you away from the features that's good you just may be more confident in using I don't know but you can see they mean something much deeper than just oh you know here's some syntax this syntax is the least interesting part of them there's a question Oh is there not a question okay well that's that's what I wanted to show you this is something which I wanted a shortcut for you what I think takes a long time for people to build in their mind people use these features for years and years and years and they keep throwing mental models against the wall for how these things work and what they mean until one sticks what I wanted a shortcut for you is you use these things enough and you start to see patterns you try to figure out what are these patterns imply you just steal that and there you have a 90 minute pie data talk anyone can do it so there you go I hope you enjoyed that thank you very much you

Channel: PyData

Views: 41,107

Rating: 4.8611569 out of 5

Keywords: Python, Tutorial, Education, NumFOCUS, PyData, Opensource, download, learn, syntax, software, python 3, data scientist, data science, data analytics, analytics, coding, PyCon, Jupyter, Notebooks, data data analytics, big data

Id: R2ipPgrWypI

Length: 84min 36sec (5076 seconds)

Published: Wed Jul 26 2017